Fix too much union-ification in media_buf_t

Fixes #1917