Fix bug where sources in FileMap were not being removed properly