Rewrite and comment request processing

Replace some of the lengthy "else if" cascades of the request handling core
methods by a new, more comprehensible implementation. Document the source of
the new implementation.

Introduce a new define NO_FILES for server configurations that only use
callbacks and do not access any directory.