better handling of 404 errors in default handlers
added support for robots.txt