added to crawl class and cleaned up code