avcodec: make vismv option as flag types

This allows for example -vismv pf+bf+bb instead of -vismv 7.
avcodec/mpegvideo: small indent fix in vismv code
transcoding: fix Doxygen file path

Signed-off-by: Timothy Gu <timothygu99@gmail.com>
Signed-off-by: Michael Niedermayer <michaelni@gmx.at>