Fix race conditions in the config compiler.

Fixes #5335