Added checking of string similarity and using this to remove string that
are potentially duplicates
changed break to continue, allows other files in, and changed tolerance to 0.4 removing dumb results, added some tests to locations