fixed bugs, crawling. But because we are visiting one at a time, it is painfully slow. We visited 250 links before actually hitting a product page, and at that time we have 12K links to visit. we need to do some parallel stuff