added changes to work with new datasource ingest changes