internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: ytdl-org/youtube-dl

Issue:
"Erroneous output template" when using the upload date with The Escapist
```
youtube-dl -o "%(upload_date)s - %(title)s.%(ext)s" "http://www.escapistmagazine.com/videos/view/zero-punctuation/5394-Darkness-2"
[Escapist] 5394-Darkness-2: Extracting information
[Escapist] 5394-Darkness-2: Downloading webpage
[Escapist] 5394-Darkness-2: Downloading normal configuration
[Escapist] 5394-Darkness-2: Downloading hq configuration
ERROR: Erroneous output template
```

Usually, the upload date is replaced by "NA" when the extractor can't find it, but this seems to not be the case with The Escapist: instead, it's simply not defined and thus creates an error.

The problem can be solved by making the extractor for The Escapist return the upload_date key alongside everything else, even if it's set to None, but this seems like it could be a more broad problem. Shouldn't all values be set to NA if the extractor doesn't provide them, even if they are not defined?



Discussion:

Resolving patch:
diff --git a/youtube_dl/YoutubeDL.py b/youtube_dl/YoutubeDL.py
--- a/youtube_dl/YoutubeDL.py
+++ b/youtube_dl/YoutubeDL.py
@@ -3,6 +3,7 @@
 
 from __future__ import absolute_import
 
+import collections
 import errno
 import io
 import json
@@ -401,13 +402,11 @@ def prepare_filename(self, info_dict):
                 is_id=(k == u'id'))
             template_dict = dict((k, sanitize(k, v))
                                  for k, v in template_dict.items())
+            template_dict = collections.defaultdict(lambda: u'NA', template_dict)
 
             tmpl = os.path.expanduser(self.params['outtmpl'])
             filename = tmpl % template_dict
             return filename
-        except KeyError as err:
-            self.report_error(u'Erroneous output template')
-            return None
         except ValueError as err:
             self.report_error(u'Error in output template: ' + str(err) + u' (encoding: ' + repr(preferredencoding()) + ')')
             return None


Test patch:
diff --git a/test/test_YoutubeDL.py b/test/test_YoutubeDL.py
--- a/test/test_YoutubeDL.py
+++ b/test/test_YoutubeDL.py
@@ -7,6 +7,7 @@
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 from test.helper import FakeYDL
+from youtube_dl import YoutubeDL
 
 
 class YDL(FakeYDL):
@@ -140,6 +141,20 @@ def test_add_extra_info(self):
         self.assertEqual(test_dict['extractor'], 'Foo')
         self.assertEqual(test_dict['playlist'], 'funny videos')
 
+    def test_prepare_filename(self):
+        info = {
+            u'id': u'1234',
+            u'ext': u'mp4',
+            u'width': None,
+        }
+        def fname(templ):
+            ydl = YoutubeDL({'outtmpl': templ})
+            return ydl.prepare_filename(info)
+        self.assertEqual(fname(u'%(id)s.%(ext)s'), u'1234.mp4')
+        self.assertEqual(fname(u'%(id)s-%(width)s.%(ext)s'), u'1234-NA.mp4')
+        # Replace missing fields with 'NA'
+        self.assertEqual(fname(u'%(uploader_date)s-%(id)s.%(ext)s'), u'NA-1234.mp4')
+
 
 if __name__ == '__main__':
     unittest.main()
Repository: ytdl-org/youtube-dl

Issue:
Don't crash when website sends wrong charset information
Some (badly written) webpage might send wrong encoding information, for exemple with header Content-Type:   text/html; charset=ISO-8859
This generates this traceback :

Traceback (most recent call last): 
  File "./bin/youtube-dl", line 6, in <module> 
    youtube_dl.main() 
  File "/home/data/dev/youtube-dl/youtube_dl/**init**.py", line 837, in main 
    _real_main(argv) 
  File "/home/data/dev/youtube-dl/youtube_dl/__init__.py", line 827, in _real_main 
    retcode = ydl.download(all_urls) 
  File "/home/data/dev/youtube-dl/youtube_dl/YoutubeDL.py", line 1033, in download 
    self.extract_info(url) 
  File "/home/data/dev/youtube-dl/youtube_dl/YoutubeDL.py", line 511, in extract_info 
    ie_result = ie.extract(url) 
  File "/home/data/dev/youtube-dl/youtube_dl/extractor/common.py", line 161, in extract 
    return self._real_extract(url) 
  File "/home/data/dev/youtube-dl/youtube_dl/extractor/generic.py", line 377, in _real_extract 
    webpage = self._download_webpage(url, video_id) 
  File "/home/data/dev/youtube-dl/youtube_dl/extractor/common.py", line 270, in _download_webpage 
    res = self._download_webpage_handle(url_or_request, video_id, note, errnote, fatal) 
  File "/home/data/dev/youtube-dl/youtube_dl/extractor/common.py", line 254, in _download_webpage_handle 
    content = webpage_bytes.decode(encoding, 'replace') 
LookupError: unknown encoding: ISO-8859 

This could be fixed by falling back to ISO-8859-1 (HTTP 1.1's mandated behaviour), or (better IMO) to UTF-8.



Discussion:
Can you name an example URL? That would allow us to test this very easily.


Resolving patch:
diff --git a/youtube_dl/extractor/common.py b/youtube_dl/extractor/common.py
--- a/youtube_dl/extractor/common.py
+++ b/youtube_dl/extractor/common.py
@@ -251,7 +251,10 @@ def _download_webpage_handle(self, url_or_request, video_id, note=None, errnote=
             with open(filename, 'wb') as outf:
                 outf.write(webpage_bytes)
 
-        content = webpage_bytes.decode(encoding, 'replace')
+        try:
+            content = webpage_bytes.decode(encoding, 'replace')
+        except LookupError:
+            content = webpage_bytes.decode('utf-8', 'replace')
 
         if (u'<title>Access to this site is blocked</title>' in content and
                 u'Websense' in content[:512]):


Test patch:
Repository: ytdl-org/youtube-dl

Issue:
Bug in subtitle download error reporting
Exceptions during in subtitle downloading erroneously try to report errors using the description filename instead of the subtitle filename. Depending on whether `--write-description` is set or not, this causes either a misleading error message or a `UnboundLocalError` exception.



Discussion:

Resolving patch:
diff --git a/youtube_dl/YoutubeDL.py b/youtube_dl/YoutubeDL.py
old mode 100644
new mode 100755
--- a/youtube_dl/YoutubeDL.py
+++ b/youtube_dl/YoutubeDL.py
@@ -936,7 +936,7 @@ def process_info(self, info_dict):
                         with io.open(encodeFilename(sub_filename), 'w', encoding='utf-8') as subfile:
                                 subfile.write(sub)
                 except (OSError, IOError):
-                    self.report_error('Cannot write subtitles file ' + descfn)
+                    self.report_error('Cannot write subtitles file ' + sub_filename)
                     return
 
         if self.params.get('writeinfojson', False):


Test patch:
Repository: ytdl-org/youtube-dl

Issue:
AttributeError in ard module
With the command given below, I get the error message below. I'm using version 2014.06.09.

`youtube-dl http://www.ardmediathek.de/tv/Klassiker-der-Weltliteratur/Max-Frisch/BR-alpha/Video\?documentId\=19067308\&bcastId\=14913194`

```
[ARD] 19067308: Downloading webpage
[ARD] 19067308: Downloading JSON metadata
Traceback (most recent call last):
  File "/usr/bin/youtube-dl", line 9, in <module>
    load_entry_point('youtube-dl==2014.06.09', 'console_scripts', 'youtube-dl')()
  File "/usr/lib/python3.4/site-packages/youtube_dl/__init__.py", line 853, in main
    _real_main(argv)
  File "/usr/lib/python3.4/site-packages/youtube_dl/__init__.py", line 843, in _real_main
    retcode = ydl.download(all_urls)
  File "/usr/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 1050, in download
    self.extract_info(url)
  File "/usr/lib/python3.4/site-packages/youtube_dl/YoutubeDL.py", line 516, in extract_info
    ie_result = ie.extract(url)
  File "/usr/lib/python3.4/site-packages/youtube_dl/extractor/common.py", line 168, in extract
    return self._real_extract(url)
  File "/usr/lib/python3.4/site-packages/youtube_dl/extractor/ard.py", line 66, in _real_extract
    determine_ext(format['url']), format['quality'])
  File "/usr/lib/python3.4/site-packages/youtube_dl/utils.py", line 845, in determine_ext
    guess = url.partition(u'?')[0].rpartition(u'.')[2]
AttributeError: 'list' object has no attribute 'partition'
```



Discussion:

Resolving patch:
diff --git a/youtube_dl/extractor/ard.py b/youtube_dl/extractor/ard.py
--- a/youtube_dl/extractor/ard.py
+++ b/youtube_dl/extractor/ard.py
@@ -56,7 +56,18 @@ def _real_extract(self, url):
                 raise ExtractorError('This video is only available after 20:00')
 
         formats = []
+
         for s in streams:
+            if type(s['_stream']) == list:
+                for index, url in enumerate(s['_stream'][::-1]):
+                    quality = s['_quality'] + index
+                    formats.append({
+                        'quality': quality,
+                        'url': url,
+                        'format_id': '%s-%s' % (determine_ext(url), quality)
+                        })
+                continue
+
             format = {
                 'quality': s['_quality'],
                 'url': s['_stream'],


Test patch:
Repository: ytdl-org/youtube-dl

Issue:
vimeo: video_thumbnail ValueError: invalid literal for int() with base 10: 'base'
I have error with downloading one video from vimeo.
video link: player.vimeo.com/video/100139426

When I run Windows executable (v2014.08.02.1) I got:

```
Traceback (most recent call last):
  File "__main__.py", line 18, in <module>
  File "youtube_dl\__init__.pyo", line 889, in main
  File "youtube_dl\__init__.pyo", line 879, in _real_main
  File "youtube_dl\YoutubeDL.pyo", line 1052, in download
  File "youtube_dl\YoutubeDL.pyo", line 516, in extract_info
  File "youtube_dl\extractor\common.pyo", line 170, in extract
  File "youtube_dl\extractor\vimeo.pyo", line 266, in _real_extract
  File "youtube_dl\extractor\vimeo.pyo", line 266, in <genexpr>
ValueError: invalid literal for int() with base 10: 'base'
```

I installed with pip and edited vimeo.py file to print all returned thumbnails:

```
[(u'640', u'http://i.vimeocdn.com/video/481548617_640.jpg'), (u'base', u'http://i.vimeocdn.com/video/481548617'), (u'960',
u'http://i.vimeocdn.com/video/481548617_960.jpg'), (u'1280', u'http://i.vimeocdn.com/video/481548617_1280.jpg')]
Traceback (most recent call last):
  File "C:\Python27\Scripts\youtube-dl-script.py", line 9, in <module>
    load_entry_point('youtube-dl==2014.08.02.1', 'console_scripts', 'youtube-dl')()
  File "C:\Python27\lib\site-packages\youtube_dl\__init__.py", line 889, in main
    _real_main(argv)
  File "C:\Python27\lib\site-packages\youtube_dl\__init__.py", line 879, in _real_main
    retcode = ydl.download(all_urls)
  File "C:\Python27\lib\site-packages\youtube_dl\YoutubeDL.py", line 1052, in download
    self.extract_info(url)
  File "C:\Python27\lib\site-packages\youtube_dl\YoutubeDL.py", line 516, in extract_info
    ie_result = ie.extract(url)
  File "C:\Python27\lib\site-packages\youtube_dl\extractor\common.py", line 170, in extract
    return self._real_extract(url)
  File "C:\Python27\lib\site-packages\youtube_dl\extractor\vimeo.py", line 280, in _real_extract
    _, video_thumbnail = sorted((int(width), t_url) for (width, t_url) in video_thumbs.items())[-1]
  File "C:\Python27\lib\site-packages\youtube_dl\extractor\vimeo.py", line 280, in <genexpr>
    _, video_thumbnail = sorted((int(width), t_url) for (width, t_url) in video_thumbs.items())[-1]
ValueError: invalid literal for int() with base 10: 'base'
```

Looks like for some videos there are `base` thumbnail.



Discussion:

Resolving patch:
diff --git a/youtube_dl/extractor/vimeo.py b/youtube_dl/extractor/vimeo.py
--- a/youtube_dl/extractor/vimeo.py
+++ b/youtube_dl/extractor/vimeo.py
@@ -276,7 +276,7 @@ def _real_extract(self, url):
         if video_thumbnail is None:
             video_thumbs = config["video"].get("thumbs")
             if video_thumbs and isinstance(video_thumbs, dict):
-                _, video_thumbnail = sorted((int(width), t_url) for (width, t_url) in video_thumbs.items())[-1]
+                _, video_thumbnail = sorted((int(width if width.isdigit() else 0), t_url) for (width, t_url) in video_thumbs.items())[-1]
 
         # Extract video description
         video_description = None


Test patch:
Repository: ytdl-org/youtube-dl

Issue:
Support cloudy.ec videos
When downloading for example https://www.cloudy.ec/v/b8e25a358071f with youtube-dl 2014.08.29, the "generic information extractor" saves an HTML file.

Support cloudy.ec videos
When downloading for example https://www.cloudy.ec/v/b8e25a358071f with youtube-dl 2014.08.29, the "generic information extractor" saves an HTML file.



Discussion:

Resolving patch:
diff --git a/youtube_dl/extractor/__init__.py b/youtube_dl/extractor/__init__.py
--- a/youtube_dl/extractor/__init__.py
+++ b/youtube_dl/extractor/__init__.py
@@ -46,6 +46,7 @@
 from .clipfish import ClipfishIE
 from .cliphunter import CliphunterIE
 from .clipsyndicate import ClipsyndicateIE
+from .cloudy import CloudyIE
 from .clubic import ClubicIE
 from .cmt import CMTIE
 from .cnet import CNETIE
diff --git a/youtube_dl/extractor/cloudy.py b/youtube_dl/extractor/cloudy.py
new file mode 100644
--- /dev/null
+++ b/youtube_dl/extractor/cloudy.py
@@ -0,0 +1,67 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import re
+
+from .common import InfoExtractor
+from ..utils import (
+    ExtractorError,
+    compat_parse_qs,
+    compat_urllib_parse,
+)
+
+
+class CloudyIE(InfoExtractor):
+    _VALID_URL = r'''(?x)
+        https?://(?:www\.)?cloudy\.ec/
+        (?:v/|embed\.php\?id=)
+        (?P<id>[A-Za-z0-9]+)
+        '''
+    _API_URL = 'http://www.cloudy.ec/api/player.api.php?%s'
+    _TEST = {
+        'url': 'https://www.cloudy.ec/v/af511e2527aac',
+        'md5': '5cb253ace826a42f35b4740539bedf07',
+        'info_dict': {
+            'id': 'af511e2527aac',
+            'ext': 'flv',
+            'title': 'Funny Cats and Animals Compilation june 2013',
+        }
+    }
+
+    def _real_extract(self, url):
+        mobj = re.match(self._VALID_URL, url)
+        video_id = mobj.group('id')
+
+        url = 'http://www.cloudy.ec/embed.php?id=%s' % video_id
+        webpage = self._download_webpage(url, video_id)
+
+        file_key = self._search_regex(
+            r'filekey\s*=\s*"([^"]+)"', webpage, 'file_key')
+        data_url = self._API_URL % compat_urllib_parse.urlencode({
+            'file': video_id,
+            'key': file_key,
+        })
+        player_data = self._download_webpage(
+            data_url, video_id, 'Downloading player data')
+        data = compat_parse_qs(player_data)
+
+        if 'error' in data:
+            raise ExtractorError(
+                '%s error: %s' % (self.IE_NAME, ' '.join(data['error_msg'])),
+                expected=True)
+
+        title = data.get('title', [None])[0]
+        if title:
+            title = title.replace('&asdasdas', '').strip()
+
+        formats = []
+        formats.append({
+            'format_id': 'sd',
+            'url': data.get('url', [None])[0],
+        })
+
+        return {
+            'id': video_id,
+            'title': title,
+            'formats': formats,
+        }


Test patch:
Repository: ytdl-org/youtube-dl

Issue:
Site Support Request: thoughtworks@wistia
According to bug reports other sites using Wistia seeem to be supported by yt-dl, but the videos by Thoughtworks don't work yet, e.g. the one at http://thoughtworks.wistia.com/medias/uxjb0lwrcz : 

```
youtube-dl -v http://thoughtworks.wistia.com/medias/uxjb0lwrcz
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-v', 'http://thoughtworks.wistia.com/medias/uxjb0lwrcz']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2014.10.18
[debug] Python version 2.7.6 - Darwin-14.0.0-x86_64-i386-64bit
[debug] Proxy map: {}
[generic] uxjb0lwrcz: Requesting header
WARNING: Falling back on generic information extractor.
[generic] uxjb0lwrcz: Downloading webpage
[generic] uxjb0lwrcz: Extracting information
[Wistia] video: Downloading JSON metadata
ERROR: Error while getting the playlist
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 524, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 193, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/wistia.py", line 33, in _real_extract
    expected=True)
ExtractorError: Error while getting the playlist
```



Discussion:

Resolving patch:
diff --git a/youtube_dl/extractor/generic.py b/youtube_dl/extractor/generic.py
--- a/youtube_dl/extractor/generic.py
+++ b/youtube_dl/extractor/generic.py
@@ -380,6 +380,17 @@ class GenericIE(InfoExtractor):
                 'uploader': 'education-portal.com',
             },
         },
+        {
+            'url': 'http://thoughtworks.wistia.com/medias/uxjb0lwrcz',
+            'md5': 'baf49c2baa8a7de5f3fc145a8506dcd4',
+            'info_dict': {
+                'id': 'uxjb0lwrcz',
+                'ext': 'mp4',
+                'title': 'Conversation about Hexagonal Rails Part 1 - ThoughtWorks',
+                'duration': 1715.0,
+                'uploader': 'thoughtworks.wistia.com',
+            },   
+        },
     ]
 
     def report_following_redirect(self, new_url):
@@ -652,7 +663,7 @@ def _playlist_from_matches(matches, getter, ie=None):
 
         # Look for embedded Wistia player
         match = re.search(
-            r'<iframe[^>]+?src=(["\'])(?P<url>(?:https?:)?//(?:fast\.)?wistia\.net/embed/iframe/.+?)\1', webpage)
+            r'(?:<meta content|<iframe[^>]+?src)=(["\'])(?P<url>(?:https?:)?//(?:fast\.)?wistia\.net/embed/iframe/.+?)\1', webpage)
         if match:
             embed_url = self._proto_relative_url(
                 unescapeHTML(match.group('url')))
@@ -664,6 +675,7 @@ def _playlist_from_matches(matches, getter, ie=None):
                 'title': video_title,
                 'id': video_id,
             }
+            
         match = re.search(r'(?:id=["\']wistia_|data-wistia-?id=["\']|Wistia\.embed\(["\'])(?P<id>[^"\']+)', webpage)
         if match:
             return {


Test patch:
Repository: ytdl-org/youtube-dl

Issue:
motherless extractor - test 0 fails
```
$ python ~/projects/youtube-dl/test/test_download.py TestDownload.test_Motherless
[Motherless] AC3FFE1: Downloading webpage
[info] Writing video description metadata as JSON to: AC3FFE1.info.json
[debug] Invoking downloader on 'http://s17.motherlessmedia.com/dev386/0/572/287/0572287847.mp4/5cb6d38eccba71d7f6bb2ef260997c3d/544A96C0.mp4'
[download] Destination: AC3FFE1.mp4
[download] 100% of 10.00KiB in 00:00
F
======================================================================
FAIL: test_Motherless (__main__.TestDownload)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/crabman/projects/youtube-dl/test/test_download.py", line 170, in test_template
    self.assertTrue(os.path.exists(tc_filename), msg='Missing file ' + tc_filename)
AssertionError: False is not true : Missing file AC3FFE1.flv

----------------------------------------------------------------------
Ran 1 test in 1.690s

FAILED (failures=1)
```

Apparently that video's page no longer give you flv, but instead gives you mp4 file. I am not sure why it happened. Maybe motherless doesn't serve flv files anymore, maybe it still does for some videos - I don't know.



Discussion:

Resolving patch:
diff --git a/youtube_dl/extractor/motherless.py b/youtube_dl/extractor/motherless.py
--- a/youtube_dl/extractor/motherless.py
+++ b/youtube_dl/extractor/motherless.py
@@ -11,14 +11,14 @@
 
 
 class MotherlessIE(InfoExtractor):
-    _VALID_URL = r'http://(?:www\.)?motherless\.com/(?P<id>[A-Z0-9]+)'
+    _VALID_URL = r'http://(?:www\.)?motherless\.com/(?:g/[a-z0-9_]+/)?(?P<id>[A-Z0-9]+)'
     _TESTS = [
         {
             'url': 'http://motherless.com/AC3FFE1',
-            'md5': '5527fef81d2e529215dad3c2d744a7d9',
+            'md5': '310f62e325a9fafe64f68c0bccb6e75f',
             'info_dict': {
                 'id': 'AC3FFE1',
-                'ext': 'flv',
+                'ext': 'mp4',
                 'title': 'Fucked in the ass while playing PS3',
                 'categories': ['Gaming', 'anal', 'reluctant', 'rough', 'Wife'],
                 'upload_date': '20100913',
@@ -40,6 +40,20 @@ class MotherlessIE(InfoExtractor):
                 'thumbnail': 're:http://.*\.jpg',
                 'age_limit': 18,
             }
+        },
+        {
+            'url': 'http://motherless.com/g/cosplay/633979F',
+            'md5': '0b2a43f447a49c3e649c93ad1fafa4a0',
+            'info_dict': {
+                'id': '633979F',
+                'ext': 'mp4',
+                'title': 'Turtlette',
+                'categories': ['superheroine heroine  superher'],
+                'upload_date': '20140827',
+                'uploader_id': 'shade0230',
+                'thumbnail': 're:http://.*\.jpg',
+                'age_limit': 18,
+            }
         }
     ]
 


Test patch:
Repository: ytdl-org/youtube-dl

Issue:
Indicate "extraction" of direct video URL
It would be preferable when youtube-dl would somehow indicate if it did effectively no extraction because it was handed a direct URL. I suggest adding something like:

```
'_type': 'direct'
```

to the direct link return statement in [generic.py](https://github.com/rg3/youtube-dl/blob/master/youtube_dl/extractor/generic.py#L547)

Background: Since [we](https://github.com/rg3/youtube-dl/issues/4003) read stuff like the title from youtube-dl's output, it is important to know when we should better ignore the data from youtube-dl, because the player may be able to extract all relevant info from the file itself (example: title from an mkv file) instead of overriding it with youtube-dl's info.



Discussion:

Resolving patch:
diff --git a/youtube_dl/extractor/generic.py b/youtube_dl/extractor/generic.py
--- a/youtube_dl/extractor/generic.py
+++ b/youtube_dl/extractor/generic.py
@@ -559,6 +559,7 @@ def _real_extract(self, url):
             return {
                 'id': video_id,
                 'title': os.path.splitext(url_basename(url))[0],
+                'direct': True,
                 'formats': [{
                     'format_id': m.group('format_id'),
                     'url': url,


Test patch: