Move old configs instead of deleting them