You weigh 250 pounds. The man across from you weighs 340. Try spending a day being hammered by the likes of a Jonathan Ogden or a Tony Boselli when you weigh 100 pounds less. But that's what you sacrifice to get the speed outside.
Speed is the reason why defenses as a whole have gotten smaller, especially at the end position.
Teams love the big, wide guys in the middle, but speed is what's needed on the outside. Missouri's Justin Smith weighs 268 pounds and is the top-rated defensive end. That's why ends have gotten smaller and the tackles are bigger. They complement each other. The Baltimore Ravens had a record-setting defense with that exact combination.