Repository: ytdl-org/youtube-dl

Issue:
Youtube description info truncates at 160 chars, started today

## Checklist
```
- [ x] I'm reporting a broken site support
- [x ] I've verified that I'm running youtube-dl version **2020.06.16.1**
- [x ] I've checked that all provided URLs are alive and playable in a browser
- [ x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [ x] I've searched the bugtracker for similar issues including closed ones
```

## Verbose log

```
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--restrict-filenames', '--write-description', '-o',
 '%(title)s.%(ext)s', '-f', '140', 'https://www.youtube.com/watch?v=WXYRvg8_H_w'
, '-v']
[debug] Encodings: locale cp1252, fs mbcs, out cp1252, pref cp1252
[debug] youtube-dl version 2020.06.16.1
[debug] Python version 3.4.4 (CPython) - Windows-7-6.1.7600-SP0
[debug] exe versions: ffmpeg git-2020-01-24-e931119, ffprobe git-2020-01-24-e931
119
[debug] Proxy map: {}
```

## Description

```
Youtube has apparently changed the way it passes video descriptions to youtube-dl, causing .description files to contain severely truncated data.

Descriptions used to be correctly formatted and complete, until today.  Now they run on a single unformatted line and are truncated at 160 characters, thus discarding much important information.

I compared description output of the same youtube download yesterday and today, using the same command line in batch file.  Yesterday's is correct; today's is truncated.  I tried this with 3 other youtubes, with the same problem.

Using the current version, checked with --update.

(Wild guess:  Has youtube changed the way its descriptions use end-of-line or linefeed characters?)

Entire truncated video description as downloaded today by youtube-dl, between dashed lines:
----------------------------------------------------------
Mario Castelnuovo-Tedesco (1895-1968): Antony and Cleopatra, Ouverture op.134 (1947). West Australian Symphony Orchestra diretta da Andrew Penny. Cover image...
----------------------------------------------------------

Correct video description as downloaded a week ago with the same command line:
----------------------------------------------------------
Mario Castelnuovo-Tedesco (1895-1968): Antony and Cleopatra, Ouverture op.134 (1947).

West Australian Symphony Orchestra diretta da Andrew Penny.

Cover image: painting by Harry Tresham.

***

The music published in our channel is exclusively dedicated to divulgation purposes and not commercial. This within a program shared to study classic educational music of the 1900's (mostly Italian) which involves thousands of people around the world. If someone, for any reason, would deem that a video appearing in this channel violates the copyright, please inform us immediately before you submit a claim to Youtube, and it will be our care to remove immediately the video accordingly.
----------------------------------------------------------


Hope this is  helpful.  Thanks very much!
```


Discussion:
`--write-info-json` is affected as well it seems
Strangely, descriptions still download correctly using JDownloader.  With Youtube settings to download descriptions, the result the the same complete text file as before.
Some things like `Artist` and `Album` for music videos rely on `\n` in the Regex. The new shortened description doesn't have any line breaks, which breaks detecting those.
I was able to fix this issue at least for my purposes in these two commits: https://github.com/randombyte-developer/youtube-dl/commits?author=randombyte-developer

I am using the description to extract details about music videos like title and artist. Since it uses the video description it should also have fixed this issue.

Here is a build for linux: https://github.com/randombyte-developer/youtube-dl/releases/tag/yt-fixed-description-1
Any idea when the main program will be updated with that fix? I'm in a hurry to download a channel before YouTube deletes it. 

Also even after the fix is applied, if in the future the description is truncated again, is there any way to get the program to display a warning letting me know that only a truncated description was downloaded?
Thanks
> 
> 
> Any idea when the main program will be updated with that fix? I'm in a hurry to download a channel before YouTube deletes it.
> 
> Also even after the fix is applied, if in the future the description is truncated again, is there any way to get the program to display a warning letting me know that only a truncated description was downloaded?
> Thanks

What you're asking for is if ytdl can implement sanity checks and possibly even probe the issue tracker for new bugs that may have a certain tag assigned like "data integrity" (just a suggestion), right?

I'd fancy that, not sure how much of a sanity check process is already integrated.  (if any)
@randombyte-developer:
How does a complete noob like myself implement your fix?
Thanks!
I figured out what's causing the problem with downloading descriptions .

For 
https://www.youtube.com/watch?v=-dac_2af0Ww

Use Chrome Inspect and look for instances of "description."    Three of them contain the truncated description, including this line:
<meta name="description" content="Arturo Toscanini  [etc]

However. one line begins like this:
{"@context":"https://schema.org","@type":"VideoObject","description":"Arturo Toscanini [etc]

This line contains the complete correct description.  The others only contain the truncated version.

I'm speculating the YTDL uses the Meta Name line above for extracting the description.  Youtube probably changed th way the meta name works.  

This looks like it'd be a small and easy change for YTDL to implement.  

How does one suggest this to their programmers?

(Thanks to all who've answered!)



Resolving patch:
diff --git a/youtube_dl/extractor/youtube.py b/youtube_dl/extractor/youtube.py
--- a/youtube_dl/extractor/youtube.py
+++ b/youtube_dl/extractor/youtube.py
@@ -1930,7 +1930,7 @@ def replace_url(m):
             ''', replace_url, video_description)
             video_description = clean_html(video_description)
         else:
-            video_description = self._html_search_meta('description', video_webpage) or video_details.get('shortDescription')
+            video_description = video_details.get('shortDescription') or self._html_search_meta('description', video_webpage)
 
         if not smuggled_data.get('force_singlefeed', False):
             if not self._downloader.params.get('noplaylist'):


Test patch: