Their numbers stay good, because even relatively young, small predators will climb all over prey nearly their size. Bass barely longer than your hand have no problem slamming wide-mouthed into bullfrogs or rats, even though they may choke on those big meals. In fast-growing young largemouths, aggression overrides all other senses. Saltwater is no different. It isn't unusual to catch a speckled trout and redfish not much longer than a standard-issue lure.
Big fish eat big meals.
To catch them and them alone, quit throwing potato chips and start throwing baked potatoes. Everything a fish eats provides it with a measurable amount of nutrition, which it eventually converts to energy. That same fish must use its stored energy to capture and kill and eat and digest its food. As juveniles, bass and specks are highly aggressive feeders that chase down small prey by the dozens to meet high-volume energy needs. In maturity, those same fish become more deliberate, patient feeders.