oscardelben/rawler — pull request #28 by spenceralger (2012-11-26 23:03:54+00:00)
Title: Added Skip and iSkip options

Warning: I'm a Ruby noob.

Added a --skip and -s so that you can specify a pattern for urls that should be skipped. Also added --iskip and -i to make those patterns case-insensitive.

Personally used this to prevent rawler from digging into endless search results.