If modified since

I would say this is something to think about. Consider we pull a page a.html that sets noindex, as a result the page is not passed to the handler. But later we go back and refetch the page with If-Modified-Since set, and the remote server returns 304: in this case the fetch result WILL be passed to the handler.