robots directive fix, domain pageQueue access fixed