Renaming a host will correctly clean up the old host dir