Baseball is unique in the number of players who remain active for 20 years. A pro running back has accomplished something if he stays around for 10 years. If injuries take their toll careers can be cut even shorter. Witness the early exits of Sandy Koufax, Gayle Sayers, Bo Jackson, Bobby Orr, and Bill Walton. Size-ism is a necessary evil in sports.
Bigger players have advantages.
People of different sizes tend to have different talents and line up at positions where those talents are most likely to be maximized. The nature of baseball is such that playing catcher, second base, third base, or shortstop precludes lefthanders. So much of what players at these positions do requires throwing to the left, which can be executed far more smoothly by right-handers. Regardless of race, the rules of sports are the same. When Jackie Robinson came up to the Dodgers, he did not get four strikes while Ted Williams and Joe DiMaggio only got three.