scrapy/scrapy — pull request #721 by bigwind (2014-05-17 00:06:08+00:00)
Title: Remove duplicate URL

1.Improve performance: remove duplicate urls, because too much memory is used when crawl a large web site.
2.Users maybe want to extract links from a text response, so changes the
HtmlResponse to TextResponse.