Only take the lock when the ref count hits zero
Fix race in ref count manipulation in slow-drop
Fix race in ref count manipulation harder