Added robots.txt and configured to block all crawling.