fix processing meta.robots=nofollow; add logging if url is disallowed by meta.robots