added incremental ingest support