mad: robustify against corrupt byte streams (fixes #10726)

(cherry picked from commit 9cd0bbb7e80d34ba0c0f7e1c337243678bce45b1)