add '--exclude' option

patch which adds a '--exclude' option, consisting of comma-separated list of patterns to exclude from crawl. excluding stuff like ^/admin can significantly speed up the crawl.