Make sure config listeners for required files are cleaned up properly