Serve default robots.txt

closes #2062
- Server robots.txt from theme if available
- Serve default robots.txt from /core/shared/ otherwise
- Added tests for default robots.txt
Merge pull request #2370 from halfdan/2062-robots

Serve default robots.txt