PPLAN-1493: Added robots.txt integration test