HYLIBRA-361: Modified robots.txt

The modified robots.txt should properly stop web crawlers from visiting
search pages while allowing access to the items in /catalog/ specified
by the sitemap file.