Fix race condition found by running go test -cpu=4 -race -v