Temporarily broke the crawler. Added:
*robots.txt support
*improved schema for database records
*index on pages (search_term,url) to enable checking for duplicates

Removed:
*self.ever_crawled in BFS_crawl to enable pages to be recrawled (i.e. updated in cases where content has changed); especially relevant for news sites
Fixed bug that was crashing the crawler
Specifics: crawler encountered pages with keyword meta tags but no content, so now the keyerror returned in these cases is caught in the _get_keywords() method
Added query2 method to return results suitable for use with the Flask app
First version of the searching flask app