Update lynx_crawl.py

Crawler using Lynx and Beautiful soup.
Addition: Checks the robots.txt file for each site.