Remove duplicate URL

1.Don't need to crawl duplicate urls.
2.Users maybe want to extract links from a text response, so changes the
HtmlResponse to TextResponse.