Only allow switching from the larger team