Properly server shared files

refs #3992
- Don't use path.join for URLs
- Cache robots.txt for 1h instead of 1y