Once you're up to speed unless you're riding uphill, the main way panniers hold you back is through aerodynamic drag - they do stick out far more than a backpack or teardrop saddle bag. Uphill, too, there is a penalty, as there is when pulling away at lights.
Once you hit cruising speed they are only a little bit slower than the alternatives if you're carrying an equivalent load, and for commuting and touring the comfort, simplicity, and lack of bag-wobble more than make up for any speed deficit.