Rename files only after file is closed

Otherwise this will break on windows where write mode also means exclusive access to the file.