filter NAs and duplicates before merge. prevents a pandas index bug