Make sure we call allReferencesCleared on the right thread.