Acclimatization aids racing performance at altitude, but in the elite, it may not give significant gains for sea level racing.
Swimmers gain altitude adaptation too.
All people doing athletic training at altitude have the potential to make gains. Running, bicycling and swimming, plus other endurance based sports get an endurance boost. The lower you endurance base when you arrive at altitude, the greater your altitude adaptation will be. Reaction to altitude, dehydration to thicken the blood, can be debilitating. The deeper breathing does help, red blood cell production does increase, but don't be concerned if you can't spend time at altitude.