Saying it "is not really a valid argument" isn't actually a valid argument. It might provide some wins, it might not, it might provide wins in some circumstances and actually slow things down in others. Without data and benchmarking we don't know.

Regardless, decoders are implemented the way they're implemented, currently, and it would be a significant breaking change to undo that. Not the biggest fish we have to fry, IMO. And certainly not one that should happen as a part of this already very large PR.