The violins fancy themselves the aristocracy of the orchestra, playing the main thematic melodies, with their feet planted firmly on the shoulders and faces of the other strings. The distinction is enormous. The baton is the most obvious manifestation of authority. With it, the conductor starts and stops the orchestra and helps it keep time. But the baton is, in reality, nothing but a tool.
Genuine authority comes from the eyes, the body.
It comes from the conductor's passion for and obvious knowledge of the music. It comes from his vision of the meaning of the music, and from his emotions. If the baton is a stick, then the conductor's charisma is the carrot. Great conductors don't just beat time. They inspire their players.