add duplication removal and modify scripts accordingly