Initial import of the Amazon category scraper.
The scraping script works well for depth=0. Not recursing likely due to an exception being silently caught. I need to tighten up my exception handling
This is my last change to the categoryScraper.py file. I'm moving to scrapy.
Adding vagrant and pyc files.
My vagrantfile for working with scrapy.
Ignoring JSON files - used for crawling output.
Initial import of my working scrapy Amazon category crawler. The crawler generates a total listing of the amazon book categories containing: name, number of items, URL and parent URL.
Many sub-categories are repeated, so I need to save the category tree with the name. Names now use '.' to identify different levels of hierarchy.
Removing a now defunct scraping tool I was coding from scratch - replaced by Scrapy.
Fixed an XPath bug where the cats of interest are the last() item not the second.
Adding the next step of the scraping pipeline - getting the leaf categories to crawl.
Adding the third step of my Amazon book price crawler - gtting books from categories.
Adding scripts to help me go through the different steps in this scraping pipelne.
Fixed a few bugs with reading the search listing page - mostly to do with category detection.
Accidentally left the depth limit on.
Added the fourth and final step - a spider to extract the book's prices
Shell script to run the fourth step.
Inverted the XPath test for categories as the original first test was inadvertently catching items destined for the original second test.
Fixed a bug where the regexp was designed with operating on the request URI rather than the link HREF.
Upped the minimum threshold since we may be unlucky when sampling pages and get a page with mostly out-of-print items. This new min is 3 pages of items, so hopefully lightning strikes but once.