Added crawler.py interface. Extended simplespider.py to support more complex rules for crawling sites.