Add robot.txt
This is needed for security reasons - often people looking to exploit applications will search for something in a certain version of a web application and find many sites to exploit. This stops that from being so easy.