FEATURE: add ability to ignore robots by setting the setRobotsNoFollow() flag in CaptureSearchResult