Added Robot and Porter Stemming Alorithm

The Robot module will search if web page has a robot.txt, which
indicated they do not want to be crawled. The Porter Stemming Algorithm
will allow us to only index the stem of words