Repository: ytdl-org/youtube-dl

Issue:
Vimeo: Chokes on --add-metadata
```
youtube-dl  --ignore-config --verbose  --download-archive ~/.ytdlarchive --no-overwrites --call-home --continue --write-info-json --write-description --write-thumbnail --merge-output-format mkv--all-subs --sub-format srt --convert-subs srt --write-sub --add-metadata https://vimeo.com/70668043 https://vimeo.com/70666333
```

```
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--ignore-config', u'--verbose', u'--download-archive', u'/home/vxbinaca/.ytdlarchive', u'--no-overwrites', u'--call-home', u'--continue', u'--write-info-json', u'--write-description', u'--write-thumbnail', u'--merge-output-format', u'mkv--all-subs', u'--sub-format', u'srt', u'--convert-subs', u'srt', u'--write-sub', u'--add-metadata', u'https://vimeo.com/70668043', u'https://vimeo.com/70666333']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.01.27
[debug] Python version 2.7.10 - Linux-4.2.0-25-generic-x86_64-with-Ubuntu-15.10-wily
[debug] exe versions: ffmpeg 2.7.5-0ubuntu0.15.10.1, ffprobe 2.7.5-0ubuntu0.15.10.1, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Public IP address: 76.101.221.232
[vimeo] 70668043: Downloading webpage
[vimeo] 70668043: Extracting information
[vimeo] 70668043: Downloading webpage
[vimeo] 70668043: Downloading JSON metadata
[vimeo] 70668043: Downloading m3u8 information
[info] Video description is already present
[info] Video description metadata is already present
[vimeo] 70668043: Thumbnail is already present
[debug] Invoking downloader on u'https://10-lvl3-hls.vimeocdn.com/1453990861-28223b02a7d6053983227f4b64333f85d0240957/01/4133/2/70668043/178317076.mp4.m3u8'
[download] Ask Ash No. 1-70668043.mp4 has already been downloaded
[download] 100% of 9.83MiB
[ffmpeg] Adding metadata to 'Ask Ash No. 1-70668043.mp4'
[debug] ffmpeg command line: ffmpeg -y -i 'file:Ask Ash No. 1-70668043.mp4' -c copy -metadata 'comment=More | junnnktank.com/thenakedissue/faq
f. Ash twitter.com/ashvandeesch
This is Ash. She'"'"'s from Holland. She'"'"'s a regular {and fucking awesome} contributor to The Naked Issue. You ask her questions, she makes a video and answers them {while looking pretty damn cute}. 
Ask Ash | thenakedissue@junnnktank.com' -metadata 'description=More | junnnktank.com/thenakedissue/faq
f. Ash twitter.com/ashvandeesch
This is Ash. She'"'"'s from Holland. She'"'"'s a regular {and fucking awesome} contributor to The Naked Issue. You ask her questions, she makes a video and answers them {while looking pretty damn cute}. 
Ask Ash | thenakedissue@junnnktank.com' -metadata artist=JUNNNKTANK -metadata 'title=Ask Ash No. 1' -metadata date=20130719 -metadata purl=https://vimeo.com/70668043 'file:Ask Ash No. 1-70668043.temp.mp4'
ERROR: Conversion failed!
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/YoutubeDL.py", line 1737, in post_process
    files_to_delete, info = pp.run(info)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 395, in run
    self.run_ffmpeg(filename, temp_filename, options)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 159, in run_ffmpeg
    self.run_ffmpeg_multiple_files([path], out_path, opts)
  File "/usr/local/lib/python2.7/dist-packages/youtube_dl/postprocessor/ffmpeg.py", line 155, in run_ffmpeg_multiple_files
    raise FFmpegPostProcessorError(msg)
FFmpegPostProcessorError
```



Discussion:
Looks like it's failing on metadata embed, it's choking on `--add-metadata`

Here's the output of the ffmpeg command:

```
$ ffmpeg -y -i 'file:Ask Ash No. 1-70668043.mp4' -c copy -metadata date=20130719 -metadata 'description=More | junnnktank.com/thenakedissue/faq 
f. Ash twitter.com/ashvandeesch
This is Ash. She'"'"'s from Holland. She'"'"'s a regular {and fucking awesome} contributor to The Naked Issue. You ask her questions, she makes a video and answers them {while looking pretty damn cute}.
Ask Ash | thenakedissue@junnnktank.com' -metadata 'title=Ask Ash No. 1' -metadata artist=JUNNNKTANK -metadata purl=https://vimeo.com/70668043 -metadata 'comment=More | junnnktank.com/thenakedissue/faq
f. Ash twitter.com/ashvandeesch
This is Ash. She'"'"'s from Holland. She'"'"'s a regular {and fucking awesome} contributor to The Naked Issue. You ask her questions, she makes a video and answers them {while looking pretty damn cute}.
Ask Ash | thenakedissue@junnnktank.com' 'file:Ask Ash No. 1-70668043.temp.mp4'
ffmpeg version 2.8.5 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 5.3.0 (GCC)
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libdcadec --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-shared --enable-version3 --enable-x11grab
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libavresample   2.  1.  0 /  2.  1.  0
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
  libpostproc    53.  3.100 / 53.  3.100
Input #0, mpegts, from 'file:Ask Ash No. 1-70668043.mp4':
  Duration: 00:01:05.50, start: 0.700000, bitrate: 1259 kb/s
  Program 1 
    Stream #0:0[0x101]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1280x720, 24 fps, 24 tbr, 90k tbn, 48 tbc
    Stream #0:1[0x102]: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 144 kb/s
[mp4 @ 0x559d96690260] Codec for stream 0 does not use global headers but container format requires global headers
[mp4 @ 0x559d96690260] Codec for stream 1 does not use global headers but container format requires global headers
Output #0, mp4, to 'file:Ask Ash No. 1-70668043.temp.mp4':
  Metadata:
    date            : 20130719
    description     : More | junnnktank.com/thenakedissue/faq
                    : f. Ash twitter.com/ashvandeesch
                    : This is Ash. She's from Holland. She's a regular {and fucking awesome} contributor to The Naked Issue. You ask her questions, she makes a video and answers them {while looking pretty damn cute}. 
                    : Ask Ash | thenakedissue@junnnktank.com
    title           : Ask Ash No. 1
    artist          : JUNNNKTANK
    purl            : https://vimeo.com/70668043
    comment         : More | junnnktank.com/thenakedissue/faq
                    : f. Ash twitter.com/ashvandeesch
                    : This is Ash. She's from Holland. She's a regular {and fucking awesome} contributor to The Naked Issue. You ask her questions, she makes a video and answers them {while looking pretty damn cute}. 
                    : Ask Ash | thenakedissue@junnnktank.com
    encoder         : Lavf56.40.101
    Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 1280x720, q=2-31, 24 fps, 24 tbr, 90k tbn, 90k tbc
    Stream #0:1: Audio: aac ([64][0][0][0] / 0x0040), 44100 Hz, stereo, 144 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x559d96690260] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
av_interleaved_write_frame(): Operation not permitted
[mp4 @ 0x559d96690260] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
frame=    2 fps=0.0 q=-1.0 Lsize=       3kB time=00:00:00.13 bitrate= 151.7kbits/s    
video:1kB audio:3kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Conversion failed!
```

These lines are relevant:

```
[mp4 @ 0x559d96690260] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
av_interleaved_write_frame(): Operation not permitted
[mp4 @ 0x559d96690260] Malformed AAC bitstream detected: use the audio bitstream filter 'aac_adtstoasc' to fix it ('-bsf:a aac_adtstoasc' option with ffmpeg)
```

With `NativeHlsFD`, `-bsf:a aac_adtstoasc` is not applied in downloading. FFMpeg can't handle the downloaded file. @remitamine I see 'm3u8_native' is added in your commit e5c209a1bcea206bee684914599c84acf886487c. Do you have any reason for that change? I've tested a few Vimeo videos HlsFD using ffmpeg works.

If it helps, I downloaded that entire channel and encountered what I think may be a similar error but I just re-ran the rip (this is easy since I have it not clobber or retry already downloaded videos) and it passed over the problem video but this particular one wouldn't do it without dropping the `--add-metadata` flag. 

92/94 videos on that channel completed without error.

@vxbinaca The problematic line is this one: https://github.com/rg3/youtube-dl/blob/ed7cd1e859cf97e975a28a5e8c58a1d1aca819fe/youtube_dl/extractor/vimeo.py#L459
Change the following lines in `youtube_dl/extractor/vimeo.py`

```
            formats.extend(self._extract_m3u8_formats(
                m3u8_url, video_id, 'mp4', 'm3u8_native', m3u8_id='hls', fatal=False))
```

to:

```
            formats.extend(self._extract_m3u8_formats(
                m3u8_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
```

Works for me. However, the parameter 'm3u8_native' is added by another developer, so I have to ask his/her original intention before making changes.

> However, the parameter 'm3u8_native' is added by another developer, so I have to ask his/her original intention before making changes.

https://github.com/rg3/youtube-dl/pull/7126#discussion_r41700477

So let's call @dstftw: using `NativeHlsFD` by default is causing problems on ffmpeg like this. (OK with avconv v12_dev0-2305-g77a44f5). Is using ffmpeg to download HLS streams causing a different problem?

So do you suggest switching all `m3u8_native` to `m3u8` in all (~45) extractors?

The problem is: if `m3u8_native` is specified by the extractor, there's no simple way to force `HlsFD`. Maybe adding an option `--hls-prefer-ffmpeg` solves all the problems? If we keep current framework, replacing all `m3u8_native` occurrences is necessary for this case.

Presence of `--hls-prefer-ffmpeg` does not really solve the problem for average user who use defaults, but it would be useful.


Resolving patch:
diff --git a/youtube_dl/postprocessor/ffmpeg.py b/youtube_dl/postprocessor/ffmpeg.py
--- a/youtube_dl/postprocessor/ffmpeg.py
+++ b/youtube_dl/postprocessor/ffmpeg.py
@@ -391,6 +391,10 @@ def run(self, info):
         for (name, value) in metadata.items():
             options.extend(['-metadata', '%s=%s' % (name, value)])
 
+        # https://github.com/rg3/youtube-dl/issues/8350
+        if info['protocol'] == 'm3u8_native':
+            options.extend(['-bsf:a', 'aac_adtstoasc'])
+
         self._downloader.to_screen('[ffmpeg] Adding metadata to \'%s\'' % filename)
         self.run_ffmpeg(filename, temp_filename, options)
         os.remove(encodeFilename(filename))


Test patch: