Fixed a bug where random.sample() doesn't work if it needs to sample all items in its sample space.
Created a script to parse the output of step3.sh and print a list of categories crawled.
Increased download delay to 2s and dropped the user agent portion - this seems to change the URLs returned by amazon to be relative from absolute which my code doesn't like.
Increased download delay.
Ignoring state preservation folder as well as some temp folder.
Dropped lambda function (doesn't work with state preservation) and used the meta parameter instead.
Converted debug print messages to scrapy log calls. Fixed a bug where an exception in an except block was not being caught.