modified routes to allow dynamic robots txt selection based on environment