Features/rt94833

Addition of the following features submitted by patch in RT#94833 (and additions by PR1):

Feature: Option of ignoring routes disallowed in robots.txt - configurable via app config. 
               Works alongside in-code ignores.

Feature: Optional configuration of both HTML/XML sitemap routes along with the ability to disable
               one or both if required.

Thanks to David Pottage for the original patch for these features and Breno G. de Oliveira (@garu) for adding tests and code tweaking the new features.