fix robots.txt directives array logic #2797

#2797