Revert to sweep checking a handle's reference count is zero inside the handle list lock.