Check if the object is unreferenced by others before destroying it

Change-Id: I04680feab773528c5fb63db7510472d8e4f3d4ac