added preliminary support for ingest via web crawl using crawler4j.  Refactored upload ingest code.