added robots file and updated config