Only remove files which are being uninstalled from the gitignore