Remove duplicate URL

1.Improve performance: remove duplicate urls, because too much memory is used when crawl a large web site.
2.Users maybe want to extract links from a text response, so changes the
HtmlResponse to TextResponse.