Back-ported a Robots.txt handler with an improved configuration setup.