The public is not the only one making observations of umpire work. An umpire that routinely makes bad calls will not go unnoticed.
Umpire performance is traced during each game, and both bad and good calls are reviewed after the game. Umpires who fail repeatedly can get suspended or even removed entirely from the field.