Handle mysterious duplicates when adding rules