looking at the history, we used to match crawler user agent here, so we should let the crawler through using agent + ip now for WEB-9040 +review WEBCR-393