Fix race in ref count manipulation harder