Properly server shared files

refs #3992
- Don't use path.join for URLs
- Cache robots.txt for 1h instead of 1y
Merge pull request #4160 from halfdan/3992-fix-shared

Properly serve shared files