extensive changes to crawler_base.py