"links to specific parts of external pages in which case you're more 
likely to want to scan it."
Can you give an example? I haven't come across data loss yet in my 
crawler.



-----Original Message-----
From: Ryan Merl <notifications@github.com>
To: theanti9/PyCrawler <PyCrawler@noreply.github.com>
Cc: John Bohne <John61590@aol.com>
Sent: Thu, Mar 13, 2014 2:13 am
Subject: Re: [PyCrawler] Removed anchor links from queue in ready_queue 
(#15)


In ready_queue.py:
> @@ -7,6 +7,9 @@ def ready_queue(address, html):
>  	links = linkregex.findall(html)
>  	queue = []
>  	for link in links:
> +		#no anchors i.e. video#title
> +		if "#" in link:
> +			continue

I think removing all links with anchors may be a bit aggressive and 
cause data loss. If it's purely an anchor link, i.e. the same page + an 
anchor, then sure, removing it is fine. However, occasionally you get 
links to specific parts of external pages in which case you're more 
likely to want to scan it.
So, maybe an improvement is to remove the anchor from a link, and ditch 
it if it's already queued or is the same page? Implement that logic and 
I'll gladly merge this.
—
Reply to this email directly or view it on GitHub.