Put all the rename-related temp files in a directory together, and keep copies of the rewritten configs so we can roll back without having to re-edit them.