changes to ingestion, adding new threadpool interface, and splitting up output into multiple formats