Modifying Extract.java in a way that it now identifies duplicated html pages in a directory and removes them
Merge branch 'master' of github.com:ViDA-NYU/memexcrawler
Modifying Extract.java in a way that it now removes duplicates and modifying concat_nltk.py to run it over different file formats
Merge branch 'master' of github.com:ViDA-NYU/memexcrawler