Repository: ytdl-org/youtube-dl

Issue:
[ARD:mediathek] RegexNotFoundError
## Please follow the guide below

- You will be asked some questions and requested to provide some information, please read them **carefully** and answer honestly
- Put an `x` into all the boxes [ ] relevant to your *issue* (like this: `[x]`)
- Use the *Preview* tab to see what your issue will actually look like

---

### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2018.11.23*. If it's not, read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
- [x] I've **verified** and **I assure** that I'm running youtube-dl **2018.11.23**

### Before submitting an *issue* make sure you have:
- [x] At least skimmed through the [README](https://github.com/rg3/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
- [x] [Searched](https://github.com/rg3/youtube-dl/search?type=Issues) the bugtracker for similar issues including closed ones
- [x] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

### What is the purpose of your *issue*?
- [x] Bug report (encountered problems with youtube-dl)
- [ ] Site support request (request for adding support for a new site)
- [ ] Feature request (request for a new functionality)
- [ ] Question
- [ ] Other

---

### The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your *issue*

---

### If the purpose of this *issue* is a *bug report*, *site support request* or you are not completely sure provide the full verbose output as follows:

Add the `-v` flag to **your command line** you run youtube-dl with (`youtube-dl -v <your command line>`), copy the **whole** output and insert it here. It should look similar to one below (replace it with **your** log inserted between triple ```):

```
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.ardmediathek.de/ard/player/Y3JpZDovL3dkci5kZS9CZWl0cmFnLTcyY2RlOTlkLTRhNmUtNGRjNi1hMmJkLTY2YTI3ZTBkZDUxYQ/planet-deutschland-300-millionen-jahre-der-grosse-crash-1-3']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.11.23
[debug] Python version 2.7.6 (CPython) - Linux-4.4.0-138-generic-x86_64-with-Ubuntu-14.04-trusty
[debug] exe versions: avconv 9.20-6, avprobe 9.20-6, rtmpdump 2.4
[debug] Proxy map: {}
[ARD:mediathek] planet-deutschland-300-millionen-jahre-der-grosse-crash-1-3: Downloading webpage
ERROR: Unable to extract title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 508, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/ard.py", line 177, in _real_extract
    webpage, 'title')
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 992, in _html_search_regex
    res = self._search_regex(pattern, string, name, default, fatal, flags, group)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 983, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
RegexNotFoundError: Unable to extract title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
...
<end of log>
```

---

### If the purpose of this *issue* is a *site support request* please provide all kinds of example URLs support for which should be included (replace following example URLs by **yours**):
- Single video: https://www.youtube.com/watch?v=BaW_jenozKc
- Single video: https://youtu.be/BaW_jenozKc
- Playlist: https://www.youtube.com/playlist?list=PL4lCao7KL_QFVb7Iudeipvc2BCavECqzc

Note that **youtube-dl does not support sites dedicated to [copyright infringement](https://github.com/rg3/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. In order for site support request to be accepted all provided example URLs should not violate any copyrights.

---

### Description of your *issue*, suggested solution and other information

Explanation of your *issue* in arbitrary form goes here. Please make sure the [description is worded well enough to be understood](https://github.com/rg3/youtube-dl#is-the-description-of-the-issue-itself-sufficient). Provide as much context and examples as possible.
If work on your *issue* requires account credentials please provide them or explain how one can obtain them.

ARD lauched their new website today and changed everything.




Discussion:
Since the mentioned video is not available anymore, here is another video, which currently fails:

- https://www.ardmediathek.de/tv/Dokumentarfilm/Das-Salz-der-Erde-Sebasti%C3%A3o-Salgado-im/SWR-Fernsehen/Video?bcastId=1105036&documentId=57927466

Fail log:
```
youtube-dl "https://www.ardmediathek.de/tv/Dokumentarfilm/Das-Salz-der-Erde-Sebasti%C3%A3o-Salgado-im/SWR-Fernsehen/Video?bcastId=1105036&documentId=57927466" -v
[debug] System config: []
[debug] User config: ['--netrc', '--retries', '30', '--mark-watched']
[debug] Custom config: []
[debug] Command-line args: ['https://www.ardmediathek.de/tv/Dokumentarfilm/Das-Salz-der-Erde-Sebasti%C3%A3o-Salgado-im/SWR-Fernsehen/Video?bcastId=1105036&documentId=57927466', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.12.03
[debug] Python version 3.7.1 (CPython) - Linux-4.19.4-arch1-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg n4.1, ffprobe n4.1, rtmpdump 2.4
[debug] Proxy map: {}
[ARD:mediathek] 57927466: Downloading webpage
ERROR: Unable to extract title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/tmp/youtube/venv/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "/tmp/youtube/venv/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 508, in extract
    ie_result = self._real_extract(url)
  File "/tmp/youtube/venv/lib/python3.7/site-packages/youtube_dl/extractor/ard.py", line 177, in _real_extract
    webpage, 'title')
  File "/tmp/youtube/venv/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 992, in _html_search_regex
    res = self._search_regex(pattern, string, name, default, fatal, flags, group)
  File "/tmp/youtube/venv/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 983, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
```

And here I found a completely strange behavior: This video (https://www.ardmediathek.de/tv/Menschen-hautnah/Liebe-ohne-Zukunft-Heimliche-Aff%C3%A4ren-un/WDR-Fernsehen/Video?bcastId=7535538&documentId=58092934) sometimes achieves to extract the title, and sometimes does not:

Success log:
```
youtube-dl "https://www.ardmediathek.de/tv/Menschen-hautnah/Liebe-ohne-Zukunft-Heimliche-Aff%C3%A4ren-un/WDR-Fernsehen/Video?bcastId=7535538&documentId=58092934" -F -v
[debug] System config: []
[debug] User config: ['--netrc', '--retries', '30', '--mark-watched']
[debug] Custom config: []
[debug] Command-line args: ['https://www.ardmediathek.de/tv/Menschen-hautnah/Liebe-ohne-Zukunft-Heimliche-Aff%C3%A4ren-un/WDR-Fernsehen/Video?bcastId=7535538&documentId=58092934', '-F', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.12.03
[debug] Python version 3.7.1 (CPython) - Linux-4.19.4-arch1-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg n4.1, ffprobe n4.1, rtmpdump 2.4
[debug] Proxy map: {}
[ARD:mediathek] 58092934: Downloading webpage
[ARD:mediathek] 58092934: Downloading media JSON
[ARD:mediathek] 58092934: Downloading m3u8 information
[info] Available formats for 58092934:
format code  extension  resolution note
hls-56       mp4        audio only   56k , mp4a.40.2
a0-mp4-0     mp4        unknown    
a0-mp4-1-0   mp4        unknown    
a0-mp4-1-1   mp4        unknown    
a0-mp4-1-2   mp4        unknown    
a0-mp4-2-0   mp4        unknown    
a0-mp4-2-1   mp4        unknown    
a0-mp4-3     mp4        unknown    
hls-184      mp4        320x180     184k , avc1.66.30, mp4a.40.2
hls-317      mp4        480x270     317k , avc1.66.30, mp4a.40.2
hls-552      mp4        512x288     552k , avc1.77.30, mp4a.40.2
hls-1103     mp4        640x360    1103k , avc1.77.30, mp4a.40.2
hls-1757     mp4        960x540    1757k , avc1.77.30, mp4a.40.2
hls-3329     mp4        1280x720   3329k , avc1.77.30, mp4a.40.2 (best)
```

Fail log:
```
youtube-dl "https://www.ardmediathek.de/tv/Menschen-hautnah/Liebe-ohne-Zukunft-Heimliche-Aff%C3%A4ren-un/WDR-Fernsehen/Video?bcastId=7535538&documentId=58092934" -F -v
[debug] System config: []
[debug] User config: ['--netrc', '--retries', '30', '--mark-watched']
[debug] Custom config: []
[debug] Command-line args: ['https://www.ardmediathek.de/tv/Menschen-hautnah/Liebe-ohne-Zukunft-Heimliche-Aff%C3%A4ren-un/WDR-Fernsehen/Video?bcastId=7535538&documentId=58092934', '-F', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2018.12.03
[debug] Python version 3.7.1 (CPython) - Linux-4.19.4-arch1-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg n4.1, ffprobe n4.1, rtmpdump 2.4
[debug] Proxy map: {}
[ARD:mediathek] 58092934: Downloading webpage
ERROR: Unable to extract title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/tmp/youtube/venv/lib/python3.7/site-packages/youtube_dl/YoutubeDL.py", line 792, in extract_info
    ie_result = ie.extract(url)
  File "/tmp/youtube/venv/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 508, in extract
    ie_result = self._real_extract(url)
  File "/tmp/youtube/venv/lib/python3.7/site-packages/youtube_dl/extractor/ard.py", line 177, in _real_extract
    webpage, 'title')
  File "/tmp/youtube/venv/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 992, in _html_search_regex
    res = self._search_regex(pattern, string, name, default, fatal, flags, group)
  File "/tmp/youtube/venv/lib/python3.7/site-packages/youtube_dl/extractor/common.py", line 983, in _search_regex
    raise RegexNotFoundError('Unable to extract %s' % _name)
youtube_dl.utils.RegexNotFoundError: Unable to extract title; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
```

I cannot explain this behavior, it seems as though youtube-dl gets different html websites served...

Resolving patch:
diff --git a/youtube_dl/extractor/ard.py b/youtube_dl/extractor/ard.py
--- a/youtube_dl/extractor/ard.py
+++ b/youtube_dl/extractor/ard.py
@@ -173,13 +173,18 @@ def _real_extract(self, url):
         title = self._html_search_regex(
             [r'<h1(?:\s+class="boxTopHeadline")?>(.*?)</h1>',
              r'<meta name="dcterms\.title" content="(.*?)"/>',
-             r'<h4 class="headline">(.*?)</h4>'],
+             r'<h4 class="headline">(.*?)</h4>',
+             r'<title[^>]*>(.*?)</title>'],
             webpage, 'title')
         description = self._html_search_meta(
             'dcterms.abstract', webpage, 'description', default=None)
         if description is None:
             description = self._html_search_meta(
-                'description', webpage, 'meta description')
+                'description', webpage, 'meta description', default=None)
+        if description is None:
+            description = self._html_search_regex(
+                r'<p\s+class="teasertext">(.+?)</p>',
+                webpage, 'teaser text', default=None)
 
         # Thumbnail is sometimes not present.
         # It is in the mobile version, but that seems to use a different URL


Test patch: