corrected generate_training for duplicate files