Use the dirwalker to initialize the xmlwalker pipeline config
Replace URL opening and caching with  urlopener blueprint
Add xmlwalker support for skipping certain dirwalker items with glob patterns
Make xmlwalker redirects into Link objects
Only ignore xmlwalker 403s for directories
Better xmlwalker dropping of items based on patterns
Move an xmlwalker section where it is in the pipeling
Fall back to opening dirwalker files locally in xmlwalker
Ignore generated doc.txt