sort -u complete_file > sorted_file_CSV.sh

This script removes most of the duplicates in the merged csv files and writes them to sorted files. BUT! you have to give your merged files (with all your subcategories) specific names as you can gather from the script.