full integration of new ingestion