vda: use the ffmpeg api to avoid memleaks.

This will not compile with libav. Libav needs to be fixed before we can
support it again here.
bin/mac: do not start playlist while quitting VLC

close #8353