navtis/xerxes — Issue #35: block robots and crawlers
Author: navtis
Created: 2012-03-08 15:25:49+00:00
State: open
Labels: Configuration, Feature request

robots.txt should deny all search requests, but not other pages. Need to have additional way to block crawlers not observing robots.txt