Complete improved ingest framework