CrawSpider: support process_links as generator

Adding tests for CrawlSpider's process_links
Merge pull request #555 from redapple/crawlspider-processlinks

CrawlSpider: support process_links as generator