Serialize re-configuration, everything is going to go pear-shaped if
we reconfigure in multiple threads at the same time.