compare strings using the simmetrics library
strip html code from input
option to restrict crawling to certain domains
classifier to determine language of pages