Fix race condition when reloading config

This fix was previously removed due to lack of understanding. Add it
back in as I've seen this race condition manifest itself on some test
runs.
Bump version to 0.0.5