[director] Fix a race condition when changing network spec [fixes #53738423]