updated ingestion script to deal with duplicates