[Groups] Ensures that the conflict groups always get removed