use CrawlSpider,it provides a convenient mechanism for following links by defining a set of rules