Fix races

This fixes several races found by running the race detector on CI.
I'll change it. It's a habit from when I was writing a bunch of hot-path code a while ago, defers are pretty expensive.