Ignore the .swp file for a directory too