Unregister handle from Dispose only. Remove Debug.Assert from RemoveHandle.