Initial web crawl

Added ability to ingest based on web crawl, refactored file processing code.  The code is not yet ready for production but this can initiate the merge with other changes.