Rewrote the spider using CrawlSpider. Now, we read the first page or items and crawl them and extract data from the items individual page directly. Still need to handle pagination and read further pages, though