Fix race in ref count manipulation harder

r? @kmcallister