Serve default robots.txt

closes #2062
- Server robots.txt from theme if available
- Serve default robots.txt from /core/shared/ otherwise
- Added tests for default robots.txt