Repository: yt-dlp/yt-dlp

Issue:
[bilibili.com] unable to --write-subtitles
### Checklist

- [X] I'm asking a question and **not** reporting a bug/feature request
- [X] I've looked through the [README](https://github.com/yt-dlp/yt-dlp#readme)
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar questions including closed ones

### Question

I've encountered unexpected behaviour I fail to explain. Could you look into this?

Video: https://www.bilibili.com/video/BV16v41137oJ/?uid=425631367634313133376F4A 

With `--ignore-config` the download works, but with my config file the download fails (gets stuck in an infinite retry loop).

I'm asking because both
- my config file doesn't give me any errors on other sites and
- the error encountered is somewhat strange and seems unrelated to any configuration issues, namely:
`Downloaded 311 bytes, expected 220 bytes.`

I can't fathom what configuration choices could lead to such an error, so I would be thankful if you could take a look. Thanks.

### Verbose log

```shell
yt-dlp --ignore-config -v https://www.bilibili.com/video/BV16v41137oJ/?uid=425631367634313133376F4A
[debug] Command-line config: ['--ignore-config', '-v', 'https://www.bilibili.com/video/BV16v41137oJ/?uid=425631367634313133376F4A']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, err utf-8, pref cp1252
[debug] yt-dlp version 2022.04.08 [7884ade] (win_exe)
[debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg N-105208-gb24f0c82b3-20220108 (setts), ffprobe N-105208-gb24f0c82b3-20220108
[debug] Optional libraries: brotli, certifi, Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] [BiliBili] Extracting URL: https://www.bilibili.com/video/BV16v41137oJ/?uid=425631367634313133376F4A
[BiliBili] 16v41137oJ: Grabbing original ID via API
[BiliBili] 250653958: Downloading webpage
[BiliBili] 250653958: Extracting videos in anthology
[BiliBili] Downloading anthology 250653958 - add --no-playlist to just download video
[download] Downloading playlist: 【纪录片】跳跃的原因 The Reason I Jump（2020）
[BiliBili] playlist 【纪录片】跳跃的原因 The Reason I Jump（2020）: Downloading 1 videos
[download] Downloading video 1 of 1
[debug] [BiliBili] Extracting URL: https://www.bilibili.com/video/BV16v41137oJ?p=1
[BiliBili] 16v41137oJ: Grabbing original ID via API
[BiliBili] 250653958: Downloading webpage
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[BiliBili] 250653958: Extracting videos in anthology
[BiliBili] 250653958: Downloading tags
[debug] Default format spec: bestvideo*+bestaudio/best
[info] 250653958_part1: Downloading 1 format(s): 5+2
[debug] Invoking downloader on "https://upos-hz-mirrorakam.akamaized.net/upgcxcode/74/82/415438274/415438274-1-30064.m4s?e=ig8euxZM2rNcNbdlhoNvNC8BqJIzNbfqXBvEqxTEto8BTrNvN0GvT90W5JZMkX_YN0MvXg8gNEV4NC8xNEV4N03eN0B5tZlqNxTEto8BTrNvNeZVuJ10Kj_g2UB02J0mN0B5tZlqNCNEto8BTrNvNC7MTX502C8f2jmMQJ6mqF2fka1mqx6gqj0eN0B599M=&uipk=5&nbs=1&deadline=1652833956&gen=playurlv2&os=akam&oi=1534581887&trid=900ed95b25f148058a7ab8740963740fu&platform=pc&upsig=ce3778498e2e8766bd0eae72fdff6de0&uparams=e,uipk,nbs,deadline,gen,os,oi,trid,platform&hdnts=exp=1652833956~hmac=f8fe67860023ef3bba6a0a8c27493147d16d12229f9cca43ee0a201c516dd36d&mid=0&bvc=vod&nettype=0&orderid=0,1&agrr=0&bw=190000&logo=80000000"
[debug] File locking is not supported on this platform. Proceeding without locking
[download] Destination: 【纪录片】跳跃的原因 The Reason I Jump（2020） [250653958_part1].f5.mp4
[download]   0.2% of 844.57MiB at 112.92KiB/s ETA 02:07:21
ERROR: Interrupted by user

yt-dlp https://www.bilibili.com/video/BV16v41137oJ/?uid=425631367634313133376F4A
[debug] Command-line config: ['https://www.bilibili.com/video/BV16v41137oJ/?uid=425631367634313133376F4A']
[debug] Portable config "C:\hp\-\setups\+downloaders\yt-dlp\appdir\yt-dlp.conf": ['--verbose', '--format-sort', 'hasvid,ie_pref,quality,res,hdr,codec,size,br,asr,ext,fps,hasaud,lang,source,proto,id', '-f', 'bestvideo*+bestaudio/best', '--paths', 'c:\\Users\\User\\Downloads\\yt', '-o', '.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\%(title)s [%(extractor)s.%(extractor_key)s.%(id)s] [%(format_id)s].%(ext)s', '-o', 'thumbnail:.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\thumbs\\thumb.%(ext)s', '-o', 'subtitle:.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\subs\\%(alt_title,title)s [%(extractor)s.%(extractor_key)s.%(id)s] [%(format_id)s].%(ext)s', '-o', 'infojson:.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\!nfo\\%(alt_title,title)s [%(extractor)s.%(extractor_key)s.%(id)s] [%(format_id)s].%(ext)s', '-o', 'link:.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\!nfo\\%(title)s [%(extractor)s.%(extractor_key)s.%(id)s] [%(format_id)s].%(ext)s', '-o', 'description:.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\!nfo\\%(title)s [%(extractor)s.%(extractor_key)s.%(id)s] [%(format_id)s].%(ext)s', '-o', 'annotation:.\\%(playlist|)s\\%(playlist_index|)02d%(playlist_index& = |)s%(release_date>%Y-%m-%d,release_timestamp>%Y-%m-%d,upload_date>%Y-%m-%d,timestamp>%Y-%m-%d|#nodate#)s %(alt_title,title)s\\!nfo\\%(title)s [%(extractor)s.%(extractor_key)s.%(id)s] [%(format_id)s].%(ext)s', '-o', 'pl_infojson:.\\%(playlist|)s\\playlist.%(ext)s', '-o', 'pl_thumbnail:.\\%(playlist|)s\\folder.%(ext)s', '-o', 'pl_description:.\\%(playlist|)s\\playlist description.%(ext)s', '--output-na-placeholder', '~~~', '--write-info-json', '--write-description', '--write-annotations', '--no-overwrites', '--no-post-overwrites', '--merge-output-format', 'mkv', '--remux-video', 'mkv', '--write-subs', '--write-auto-subs', '--all-subs', '--sub-langs', 'all', '--sub-format', 'all/ttml/vtt/best', '--write-thumbnail', '--write-all-thumbnails', '--add-metadata', '--embed-chapters', '--add-chapters', '--write-link', '--write-url-link', '--write-desktop-link', '--no-check-certificate', '--retries', 'infinite', '--geo-bypass', '--no-clean-infojson', '--write-comments', '--audio-multistreams', '--no-playlist']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, err utf-8, pref cp1252
[debug] yt-dlp version 2022.04.08 [7884ade] (win_exe)
[debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.19043-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] exe versions: ffmpeg N-105208-gb24f0c82b3-20220108 (setts), ffprobe N-105208-gb24f0c82b3-20220108
[debug] Optional libraries: brotli, certifi, Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
[debug] [BiliBili] Extracting URL: https://www.bilibili.com/video/BV16v41137oJ/?uid=425631367634313133376F4A
[BiliBili] 16v41137oJ: Grabbing original ID via API
[BiliBili] 250653958: Downloading webpage
[BiliBili] Downloading just video 250653958 because of --no-playlist
[debug] Sort order given by user: hasvid, ie_pref, quality, res, hdr, codec, size, br, asr, ext, fps, hasaud, lang, source, proto, id
[debug] Formats sorted by: hasvid, ie_pref, quality, res, hdr, vcodec, acodec, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, fps, hasaud, lang, source, proto, id
[BiliBili] 250653958: Downloading tags
[debug] Downloading subtitles: danmaku
[BiliBili] 250653958: Extracting comments from page 1
[BiliBili] Extracted 0 comments
[info] 250653958_part1: Downloading 1 format(s): 5+2
[info] Video description is already present
[info] Writing video subtitles to: c:\Users\User\Downloads\yt\2021-09-27 【纪录片】跳跃的原因 The Reason I Jump（2020）\【纪录片】跳跃的原因 The Reason I Jump（2020） [BiliBili.BiliBili.250653958_part1] [5+2].danmaku.xml
[debug] Invoking downloader on "https://comment.bilibili.com/415438274.xml"
[download] Resuming download at byte 311
[download] Unable to resume
[debug] File locking is not supported on this platform. Proceeding without locking
[download] Destination: c:\Users\User\Downloads\yt\2021-09-27 【纪录片】跳跃的原因 The Reason I Jump（2020）\【纪录片】跳跃的原因 The Reason I Jump（2020） [BiliBili.BiliBili.250653958_part1] [5+2].danmaku.xml
[download] 141.4% of 220.00B at Unknown speed ETA Unknown[download] Got server HTTP error: Downloaded 311 bytes, expected 220 bytes. Retrying (attempt 1 of inf) ...
[download] Unable to resume
[download] Destination: c:\Users\User\Downloads\yt\2021-09-27 【纪录片】跳跃的原因 The Reason I Jump（2020）\【纪录片】跳跃的原因 The Reason I Jump（2020） [BiliBili.BiliBili.250653958_part1] [5+2].danmaku.xml
[download] 141.4% of 220.00B at Unknown speed ETA Unknown[download] Got server HTTP error: Downloaded 311 bytes, expected 220 bytes. Retrying (attempt 2 of inf) ...
[download] Unable to resume
[download] Destination: c:\Users\User\Downloads\yt\2021-09-27 【纪录片】跳跃的原因 The Reason I Jump（2020）\【纪录片】跳跃的原因 The Reason I Jump（2020） [BiliBili.BiliBili.250653958_part1] [5+2].danmaku.xml
[download] 141.4% of 220.00B at 19.81KiB/s ETA 00:00[download] Got server HTTP error: Downloaded 311 bytes, expected 220 bytes. Retrying (attempt 3 of inf) ...
[download] Unable to resume
[download] Destination: c:\Users\User\Downloads\yt\2021-09-27 【纪录片】跳跃的原因 The Reason I Jump（2020）\【纪录片】跳跃的原因 The Reason I Jump（2020） [BiliBili.BiliBili.250653958_part1] [5+2].danmaku.xml
[download] 141.4% of 220.00B at Unknown speed ETA Unknown[download] Got server HTTP error: Downloaded 311 bytes, expected 220 bytes. Retrying (attempt 4 of inf) ...
[download] Unable to resume
[download] Destination: c:\Users\User\Downloads\yt\2021-09-27 【纪录片】跳跃的原因 The Reason I Jump（2020）\【纪录片】跳跃的原因 The Reason I Jump（2020） [BiliBili.BiliBili.250653958_part1] [5+2].danmaku.xml
[download] 141.4% of 220.00B at 19.82KiB/s ETA 00:00[download] Got server HTTP error: Downloaded 311 bytes, expected 220 bytes. Retrying (attempt 5 of inf) ...
[download] Unable to resume
[download] Destination: c:\Users\User\Downloads\yt\2021-09-27 【纪录片】跳跃的原因 The Reason I Jump（2020）\【纪录片】跳跃的原因 The Reason I Jump（2020） [BiliBili.BiliBili.250653958_part1] [5+2].danmaku.xml
[download] 141.4% of 220.00B at Unknown speed ETA Unknown[download] Got server HTTP error: Downloaded 311 bytes, expected 220 bytes. Retrying (attempt 6 of inf) ...
[download] Unable to resume
[download] Destination: c:\Users\User\Downloads\yt\2021-09-27 【纪录片】跳跃的原因 The Reason I Jump（2020）\【纪录片】跳跃的原因 The Reason I Jump（2020） [BiliBili.BiliBili.250653958_part1] [5+2].danmaku.xml
[download] 141.4% of 220.00B at 19.43KiB/s ETA 00:00[download] Got server HTTP error: Downloaded 311 bytes, expected 220 bytes. Retrying (attempt 7 of inf) ...
[download] Unable to resume
[download] Destination: c:\Users\User\Downloads\yt\2021-09-27 【纪录片】跳跃的原因 The Reason I Jump（2020）\【纪录片】跳跃的原因 The Reason I Jump（2020） [BiliBili.BiliBili.250653958_part1] [5+2].danmaku.xml
[download] 141.4% of 220.00B at Unknown speed ETA Unknown[download] Got server HTTP error: Downloaded 311 bytes, expected 220 bytes. Retrying (attempt 8 of inf) ...
[download] Unable to resume
[download] Destination: c:\Users\User\Downloads\yt\2021-09-27 【纪录片】跳跃的原因 The Reason I Jump（2020）\【纪录片】跳跃的原因 The Reason I Jump（2020） [BiliBili.BiliBili.250653958_part1] [5+2].danmaku.xml
[download] 141.4% of 220.00B at 19.48KiB/s ETA 00:00[download] Got server HTTP error: Downloaded 311 bytes, expected 220 bytes. Retrying (attempt 9 of inf) ...
```



Discussion:
The infinite loop is caused by `--retry infinite`. The error itself is due to `--write-subtitles`
Thank you. How can the problem be solved?
Isn’t this #637 all over again?
wdym?
…yes, it is. In a manner of speaking.

```sh-session
$ curl -v 'https://comment.bilibili.com/415438274.xml' -o /tmp/a
> GET /415438274.xml HTTP/2
> Host: comment.bilibili.com
> user-agent: curl/7.83.0
> accept: */*
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
} [5 bytes data]
< HTTP/2 200 
[…]
< content-length: 220
[…]
< content-encoding: deflate
```

Content-length measures the length of the deflated stream (i.e. after content-encoding), not the raw uncompressed data. Downloader code assumes the latter. It’s the same underlying error as #637, though in a different place.

Quoting that ticket:

> Other uses of the `data_len` variable may need to be audited as well, to ensure it’s not assumed to exactly correspond to the raw bytestream length elsewhere where it matters (mere uses in the UI may be tolerable).

It seems that audit is way overdue.
It works on https://github.com/yt-dlp/yt-dlp/pull/4221
I tested it on https://github.com/yt-dlp/yt-dlp/releases/tag/2022.09.01, it works now
I'm facing the same issue with 
`yt-dlp --write-subs --sub-format xml 'https://www.bilibili.com/video/BV1mU4y1K7b2/'` on version 2022.10.04
`[download] 135.6% of    225.00B at  Unknown B/s ETA Unknown[download] Got error: Downloaded 305 bytes, expected 225 bytes. Retrying (10/10)...`
same problem on https://www.bilibili.com/video/BV1SW411u7Go, version 2022.11.11
`[download] 202.8% of    1.76KiB at  290.63KiB/s ETA 00:00[download] Got error: Downloaded 3663 bytes, expected 1806 bytes. Retrying (1/10)...`
Is there a way to ignore this "Downloaded x bytes, expected x bytes" error? Not on bilibili but I'm having this exact same error when downloading subtitle from a m3u8 file (jwplayer)
Why is this an issue? Why can't yt-dlp just ignore its expectations? The downloaded subtitle file has no issues actually. So the error seems not necessary.
In case it's needed, here is another URL that fails with current yt-dlp version:
https://www.bilibili.com/video/BV1fs411h73o/?uid=42563166733431316837336F

Resolving patch:
diff --git a/yt_dlp/downloader/http.py b/yt_dlp/downloader/http.py
--- a/yt_dlp/downloader/http.py
+++ b/yt_dlp/downloader/http.py
@@ -211,7 +211,12 @@ def close_stream():
                 ctx.stream = None
 
         def download():
-            data_len = ctx.data.info().get('Content-length', None)
+            data_len = ctx.data.info().get('Content-length')
+
+            if ctx.data.info().get('Content-encoding'):
+                # Content-encoding is present, Content-length is not reliable anymore as we are
+                # doing auto decompression. (See: https://github.com/yt-dlp/yt-dlp/pull/6176)
+                data_len = None
 
             # Range HTTP header may be ignored/unsupported by a webserver
             # (e.g. extractor/scivee.py, extractor/bambuser.py).
diff --git a/yt_dlp/utils.py b/yt_dlp/utils.py
--- a/yt_dlp/utils.py
+++ b/yt_dlp/utils.py
@@ -1438,19 +1438,16 @@ def http_response(self, req, resp):
                     raise original_ioerror
             resp = urllib.request.addinfourl(uncompressed, old_resp.headers, old_resp.url, old_resp.code)
             resp.msg = old_resp.msg
-            del resp.headers['Content-encoding']
         # deflate
         if resp.headers.get('Content-encoding', '') == 'deflate':
             gz = io.BytesIO(self.deflate(resp.read()))
             resp = urllib.request.addinfourl(gz, old_resp.headers, old_resp.url, old_resp.code)
             resp.msg = old_resp.msg
-            del resp.headers['Content-encoding']
         # brotli
         if resp.headers.get('Content-encoding', '') == 'br':
             resp = urllib.request.addinfourl(
                 io.BytesIO(self.brotli(resp.read())), old_resp.headers, old_resp.url, old_resp.code)
             resp.msg = old_resp.msg
-            del resp.headers['Content-encoding']
         # Percent-encode redirect URL of Location HTTP header to satisfy RFC 3986 (see
         # https://github.com/ytdl-org/youtube-dl/issues/6457).
         if 300 <= resp.code < 400:


Test patch: