Moved robot.txt checking to the indexer module
*all links returned to the crawler when a page is indexed have been verified as robot.txt-okay
*docstrings have been updated for all functions in the indexer module