Just create a file which are the diffs between
their old config and our defaults