Fixed a typo.
Added option for adding allowed and excluded URL prefixes.
Added storage backend to store and load crawler data.
Added a command line crawling script 'crawler.py'.