Fix race condition when removing things