Repository: ytdl-org/youtube-dl

Issue:
--add-metadata causing errors in post processing
Hi, 
When running the program from a vevo source with --add-metadata parameter, in post processing phase, the following error is encountered:
`[ffmpeg] Adding metadata to 'test.mp4'`
`ERROR: av_interleaved_write_frame(): Operation not permitted`

This is true even with avconv installed.  I tried to manually run ffmpeg and avconv
`ffmpeg -i test.mp4 -codec copy -metadata title="Test Title" test1.mp4`
eventually runs into this error:
`[mp4 @ 0x2547260] malformated aac bitstream, use -absf aac_adtstoasc`
`av_interleaved_write_frame(): Operation not permitted`

If i then execute `ffmpeg -i test.mp4 -codec copy -metadata title="Test Title" -bsf aac_adtstoasc test1.mp4`
That succeeds and 
`$ du -h *.mp4`
`149M    test1.mp4`
`163M    test.mp4`

 `$ ffprobe -loglevel quiet -show_format test1.mp4 | grep title`
`TAG:title=Test Title`

I did not research what would be the reason for the malformed bit stream, but it happens consistently from the same source.  Don't know if a good solution is to somehow activate that flag or allowing us to add more parameters in the post processor.   Other wise I suppose I could use -exec or a script is to interpret the --write-json-file out and then run the ffmpeg command externally to embed the meta data.  I essentially would also like to embed thumbnail but have not tried that.



Discussion:
Please post the full output of `youtube-dl --verbose URL OTHER_OPTIONS`, otherwise we can't debug the problem.

I can reproduce it with the test video http://www.vevo.com/watch/hurts/somebody-to-die-for/GB1101300280.

Using the ffmpeg downloader instead of the native hls downloader works. Maybe we could use it instead.

I am not sure if you still need the verbose output, but I have added it below.  Is there a parameter to use the ffmpeg downloader or does that need to be added to the command line?

youtube-dl --add-metadata --verbose --write-info-json --write-thumbnail -o test.mp4 http://cache.vevo.com/m/html/embed.html?video=USCJY1431509  
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--add-metadata', '--verbose', '--write-info-json', '--write-thumbnail', '-o', 'test.mp4', 'http://cache.vevo.com/m/html/embed.html?video=USCJY1431509']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2014.10.25
[debug] Python version 2.7.6 - Linux-3.13.0-43-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] Proxy map: {}
[Vevo] Retrieving oauth token
[Vevo] USCJY1431509: Downloading JSON metadata
[Vevo] USCJY1431509: Downloading HLS formats
[Vevo] USCJY1431509: Downloading webpage
[info] Writing video description metadata as JSON to: test.info.json
[Vevo] USCJY1431509: Downloading thumbnail ...
[Vevo] USCJY1431509: Writing thumbnail to: test.jpg
[debug] Invoking downloader on u'http://hls-aws.vevo.com/v3/hls/2014/11/USCJY1431509/a50b702a-4eb4-4f08-b6d0-a0070c4627e6/5200/uscjy1431509_5200k_1920x1080_h264_5200_aac_128.m3u8'
[download] Destination: test.mp4
[hlsnative] USCJY1431509: Downloading m3u8 manifest
[hlsnative] USCJY1431509: Downloading segment 1 / 28
[hlsnative] USCJY1431509: Downloading segment 2 / 28
[hlsnative] USCJY1431509: Downloading segment 3 / 28
[hlsnative] USCJY1431509: Downloading segment 4 / 28
[hlsnative] USCJY1431509: Downloading segment 5 / 28
[hlsnative] USCJY1431509: Downloading segment 6 / 28
[hlsnative] USCJY1431509: Downloading segment 7 / 28
[hlsnative] USCJY1431509: Downloading segment 8 / 28
[hlsnative] USCJY1431509: Downloading segment 9 / 28
[hlsnative] USCJY1431509: Downloading segment 10 / 28
[hlsnative] USCJY1431509: Downloading segment 11 / 28
[hlsnative] USCJY1431509: Downloading segment 12 / 28
[hlsnative] USCJY1431509: Downloading segment 13 / 28
[hlsnative] USCJY1431509: Downloading segment 14 / 28
[hlsnative] USCJY1431509: Downloading segment 15 / 28
[hlsnative] USCJY1431509: Downloading segment 16 / 28
[hlsnative] USCJY1431509: Downloading segment 17 / 28
[hlsnative] USCJY1431509: Downloading segment 18 / 28
[hlsnative] USCJY1431509: Downloading segment 19 / 28
[hlsnative] USCJY1431509: Downloading segment 20 / 28
[hlsnative] USCJY1431509: Downloading segment 21 / 28
[hlsnative] USCJY1431509: Downloading segment 22 / 28
[hlsnative] USCJY1431509: Downloading segment 23 / 28
[hlsnative] USCJY1431509: Downloading segment 24 / 28
[hlsnative] USCJY1431509: Downloading segment 25 / 28
[hlsnative] USCJY1431509: Downloading segment 26 / 28
[hlsnative] USCJY1431509: Downloading segment 27 / 28
[hlsnative] USCJY1431509: Downloading segment 28 / 28
[ffmpeg] Adding metadata to 'test.mp4'
[debug] ffmpeg command line: avconv -y -i test.mp4 -c copy -metadata date=20141110 -metadata 'artist=Taylor Swift' -metadata 'title=Blank Space' test.temp.mp4
ERROR: av_interleaved_write_frame(): Operation not permitted
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1114, in post_process
    keep_video_wish, new_info = pp.run(info)
  File "/usr/local/bin/youtube-dl/youtube_dl/postprocessor/ffmpeg.py", line 481, in run
    self.run_ffmpeg(filename, temp_filename, options)
  File "/usr/local/bin/youtube-dl/youtube_dl/postprocessor/ffmpeg.py", line 69, in run_ffmpeg
    self.run_ffmpeg_multiple_files([path], out_path, opts)
  File "/usr/local/bin/youtube-dl/youtube_dl/postprocessor/ffmpeg.py", line 63, in run_ffmpeg_multiple_files
    raise FFmpegPostProcessorError(msg)
FFmpegPostProcessorError

@vutang50 There is a `--prefer-ffmpeg` parameter to force the use of ffmpeg instead of the default avconv. 

@pricejn2 I got the same error using latest ffmpeg.
Using the native downloader is harcoded in the extractor, we would need to change it.

Ah, I see.  Thanks for clarifying. 


Resolving patch:
diff --git a/youtube_dl/YoutubeDL.py b/youtube_dl/YoutubeDL.py
--- a/youtube_dl/YoutubeDL.py
+++ b/youtube_dl/YoutubeDL.py
@@ -87,6 +87,7 @@
 from .downloader import get_suitable_downloader
 from .downloader.rtmp import rtmpdump_version
 from .postprocessor import (
+    FFmpegFixupM3u8PP,
     FFmpegFixupM4aPP,
     FFmpegFixupStretchedPP,
     FFmpegMergerPP,
@@ -1670,6 +1671,22 @@ def compatible_formats(formats):
                     else:
                         assert fixup_policy in ('ignore', 'never')
 
+                if info_dict.get('protocol') == 'm3u8_native' or info_dict.get('protocol') == 'm3u8' and self._downloader.params.get('hls_prefer_native', False):
+                    if fixup_policy == 'warn':
+                        self.report_warning('%s: malformated aac bitstream.' % (
+                            info_dict['id']))
+                    elif fixup_policy == 'detect_or_warn':
+                        fixup_pp = FFmpegFixupM3u8PP(self)
+                        if fixup_pp.available:
+                            info_dict.setdefault('__postprocessors', [])
+                            info_dict['__postprocessors'].append(fixup_pp)
+                        else:
+                            self.report_warning(
+                                '%s: malformated aac bitstream. Install ffmpeg or avconv to fix this automatically.' % (
+                                    info_dict['id']))
+                    else:
+                        assert fixup_policy in ('ignore', 'never')
+
                 try:
                     self.post_process(filename, info_dict)
                 except (PostProcessingError) as err:
diff --git a/youtube_dl/postprocessor/__init__.py b/youtube_dl/postprocessor/__init__.py
--- a/youtube_dl/postprocessor/__init__.py
+++ b/youtube_dl/postprocessor/__init__.py
@@ -6,6 +6,7 @@
     FFmpegEmbedSubtitlePP,
     FFmpegExtractAudioPP,
     FFmpegFixupStretchedPP,
+    FFmpegFixupM3u8PP,
     FFmpegFixupM4aPP,
     FFmpegMergerPP,
     FFmpegMetadataPP,
@@ -26,6 +27,7 @@ def get_postprocessor(key):
     'ExecAfterDownloadPP',
     'FFmpegEmbedSubtitlePP',
     'FFmpegExtractAudioPP',
+    'FFmpegFixupM3u8PP',
     'FFmpegFixupM4aPP',
     'FFmpegFixupStretchedPP',
     'FFmpegMergerPP',
diff --git a/youtube_dl/postprocessor/ffmpeg.py b/youtube_dl/postprocessor/ffmpeg.py
--- a/youtube_dl/postprocessor/ffmpeg.py
+++ b/youtube_dl/postprocessor/ffmpeg.py
@@ -391,10 +391,6 @@ def run(self, info):
         for (name, value) in metadata.items():
             options.extend(['-metadata', '%s=%s' % (name, value)])
 
-        # https://github.com/rg3/youtube-dl/issues/8350
-        if info.get('protocol') == 'm3u8_native' or info.get('protocol') == 'm3u8' and self._downloader.params.get('hls_prefer_native', False):
-            options.extend(['-bsf:a', 'aac_adtstoasc'])
-
         self._downloader.to_screen('[ffmpeg] Adding metadata to \'%s\'' % filename)
         self.run_ffmpeg(filename, temp_filename, options)
         os.remove(encodeFilename(filename))
@@ -467,6 +463,21 @@ def run(self, info):
         return [], info
 
 
+class FFmpegFixupM3u8PP(FFmpegPostProcessor):
+    def run(self, info):
+        filename = info['filepath']
+        temp_filename = prepend_extension(filename, 'temp')
+
+        options = ['-c', 'copy', '-f', 'mp4', '-bsf:a', 'aac_adtstoasc']
+        self._downloader.to_screen('[ffmpeg] Fixing malformated aac bitstream in "%s"' % filename)
+        self.run_ffmpeg(filename, temp_filename, options)
+
+        os.remove(encodeFilename(filename))
+        os.rename(encodeFilename(temp_filename), encodeFilename(filename))
+
+        return [], info
+
+
 class FFmpegSubtitlesConvertorPP(FFmpegPostProcessor):
     def __init__(self, downloader=None, format=None):
         super(FFmpegSubtitlesConvertorPP, self).__init__(downloader)


Test patch: