Fix races

This fixes several races found by running the race detector on CI.