(SCHOLAR-3.6) fix robots.txt directives array logic #2797

#2797