gpupo/CamelSpider — pull request #12 by iampersistent (2012-06-06 08:41:37+00:00)
Title: add gathering rel in the link, so a nofollow can be implemented

I'm also looking at adding in Indexer::addLink() a test for nofollow. It could be hard coded, but I've thought about adding a LinkFilter class that basically would basically make sure the Link passed the test in the filter. For example, the SpiderAsserts::isDocumentLink would be put in a filter. The nofollow would go in a filter, custom rules for eliminating a link could go into a filter.

Here is how it would look https://gist.github.com/2880690

Let me know if you would like me to build that out.