Fix race condition in the config validator.

Fixes #5602