Remove duplicate URL

1.Improve performance: remove duplicate urls, because too much memory is used when crawl a large web site.
2.Users maybe want to extract links from a text response, so changes the
HtmlResponse to TextResponse.
We already have a `LxmlLinkExtractor` as mentioned on #528, is this still needed?