Documentation of code in crawler.py