Repository: streamlink/streamlink

Issue:
ValueError: Padding is incorrect.
### Checklist

- [X] This is a plugin issue and not a different kind of issue
- [X] [I have read the contribution guidelines](https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#contributing-to-streamlink)
- [X] [I have checked the list of open and recently closed plugin issues](https://github.com/streamlink/streamlink/issues?q=is%3Aissue+label%3A%22plugin+issue%22)
- [X] [I have checked the commit log of the master branch](https://github.com/streamlink/streamlink/commits/master)

### Streamlink version

Latest stable release

### Description

`streamlink --http-header "User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" --http-header "Referer=https://www.log2base2.com/" "https://d2bzjah96pmxyc.cloudfront.net/C/Functions/Recursion/index.m3u8" best -l debug -o out1.ts`


`hls_audio_160k_v4.m3u8` has these `#EXT-X-BYTERANGE` lines.

#EXT-X-BYTERANGE:228432@0
#EXT-X-BYTERANGE:224288
#EXT-X-BYTERANGE:224864

...

And streamlink is setting these Range headers for HTTP GET requests.

Range: bytes=0-228431
Range: bytes=0-224287
Range: bytes=0-224863
...


Comment by:
@kikuyan

### Debug log

```text
streamlink --http-header "User-Agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36" --http-header "Referer=https://www.log2base2.com/" "https://d2bzjah96pmxyc.cloudfront.net/C/Functions/Recursion/index.m3u8" best -l debug -o out1.ts
[cli][debug] OS: Windows 10
[cli][debug] Python: 3.9.8
[cli][debug] Streamlink: 3.0.3
[cli][debug] Requests(2.26.0), Socks(1.7.1), Websocket(1.2.1)
[cli][debug] Arguments:
[cli][debug] url=https://d2bzjah96pmxyc.cloudfront.net/C/Functions/Recursion/index.m3u8
[cli][debug] stream=['best']
[cli][debug] --loglevel=debug
[cli][debug] --output=out1.ts
[cli][debug] --ffmpeg-ffmpeg=C:\Program Files (x86)\Streamlink\ffmpeg\ffmpeg.exe
[cli][debug] --http-header=[('User-Agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36'), ('Referer', 'https://www.log2base2.com/')]
[cli][info] Found matching plugin hls for URL https://d2bzjah96pmxyc.cloudfront.net/C/Functions/Recursion/index.m3u8
[plugins.hls][debug] URL=https://d2bzjah96pmxyc.cloudfront.net/C/Functions/Recursion/index.m3u8; params={}
[utils.l10n][debug] Language code: en_GB
[stream.hls][debug] Using external audio tracks for stream 1080p (language=None, name=Default)
[stream.hls][debug] Using external audio tracks for stream 1080p_alt (language=None, name=Default)
[stream.hls][debug] Using external audio tracks for stream 576p (language=None, name=Default)
[stream.hls][debug] Using external audio tracks for stream 360p (language=None, name=Default)
[stream.hls][debug] Using external audio tracks for stream 224p (language=None, name=Default)
[stream.hls][debug] Using external audio tracks for stream 73k (language=None, name=Default)
[cli][info] Available streams: 73k (worst), 224p, 360p, 576p, 1080p_alt, 1080p (best)
[cli][info] Opening stream: 1080p (hls-multi)
[stream.ffmpegmux][debug] Opening hls substream
[stream.hls][debug] Reloading playlist
[stream.ffmpegmux][debug] Opening hls substream
[stream.hls][debug] Reloading playlist
[utils.named_pipe][info] Creating pipe streamlinkpipe-6436-1-825
[utils.named_pipe][info] Creating pipe streamlinkpipe-6436-2-8256
[stream.ffmpegmux][debug] ffmpeg command: C:\Program Files (x86)\Streamlink\ffmpeg\ffmpeg.exe -nostats -y -i \.\pipe\streamlinkpipe-6436-1-825 -i \.\pipe\streamlinkpipe-6436-2-8256 -c:v copy -c:a copy -map 0:v? -map 0:a? -map 1:a -f mpegts pipe:1
[stream.ffmpegmux][debug] Starting copy to pipe: \.\pipe\streamlinkpipe-6436-1-825
[stream.ffmpegmux][debug] Starting copy to pipe: \.\pipe\streamlinkpipe-6436-2-8256
[cli][debug] Pre-buffering 8192 bytes
[stream.hls][debug] Segments in this playlist are encrypted
[stream.hls][debug] First Sequence: 0; Last Sequence: 24
[stream.hls][debug] Start offset: 0; Duration: None; Start Sequence: 0; End Sequence: 24
[stream.hls][debug] Adding segment 0 to queue
[stream.hls][debug] Adding segment 1 to queue
[stream.hls][debug] Adding segment 2 to queue
[stream.hls][debug] Adding segment 3 to queue
[stream.hls][debug] Adding segment 4 to queue
[stream.hls][debug] Adding segment 5 to queue
[stream.hls][debug] Adding segment 6 to queue
[stream.hls][debug] Adding segment 7 to queue
[stream.hls][debug] Adding segment 8 to queue
[stream.hls][debug] Adding segment 9 to queue
[stream.hls][debug] Adding segment 10 to queue
[stream.hls][debug] Adding segment 11 to queue
[stream.hls][debug] Adding segment 12 to queue
[stream.hls][debug] Adding segment 13 to queue
[stream.hls][debug] Adding segment 14 to queue
[stream.hls][debug] Adding segment 15 to queue
[stream.hls][debug] Adding segment 16 to queue
[stream.hls][debug] Adding segment 17 to queue
[stream.hls][debug] Adding segment 18 to queue
[stream.hls][debug] Adding segment 19 to queue
[stream.hls][debug] Adding segment 20 to queue
[stream.hls][debug] Adding segment 21 to queue
[stream.hls][debug] Segments in this playlist are encrypted
[stream.hls][debug] First Sequence: 0; Last Sequence: 25
[stream.hls][debug] Start offset: 0; Duration: None; Start Sequence: 0; End Sequence: 25
[stream.hls][debug] Adding segment 0 to queue
[stream.hls][debug] Adding segment 1 to queue
[stream.hls][debug] Adding segment 2 to queue
[stream.hls][debug] Adding segment 3 to queue
[stream.hls][debug] Adding segment 4 to queue
[stream.hls][debug] Adding segment 5 to queue
[stream.hls][debug] Adding segment 6 to queue
[stream.hls][debug] Adding segment 7 to queue
[stream.hls][debug] Adding segment 8 to queue
[stream.hls][debug] Adding segment 9 to queue
[stream.hls][debug] Adding segment 10 to queue
[stream.hls][debug] Adding segment 11 to queue
[stream.hls][debug] Adding segment 12 to queue
[stream.hls][debug] Adding segment 13 to queue
[stream.hls][debug] Adding segment 14 to queue
[stream.hls][debug] Adding segment 15 to queue
[stream.hls][debug] Adding segment 16 to queue
[stream.hls][debug] Adding segment 17 to queue
[stream.hls][debug] Adding segment 18 to queue
[stream.hls][debug] Adding segment 19 to queue
[stream.hls][debug] Adding segment 20 to queue
[stream.hls][debug] Adding segment 21 to queue
[stream.hls][debug] Segment 0 complete
[stream.hls][debug] Adding segment 22 to queue
[stream.hls][debug] Segment 0 complete
[stream.hls][debug] Adding segment 22 to queue
[cli][debug] Checking file output
[cli][debug] Writing stream to output
Exception in thread Thread-HLSStreamWriter:
Traceback (most recent call last):
File "threading.py", line 973, in _bootstrap_inner
File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\stream\segmented.py", line 200, in run
self.write(segment, result, *data)
File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\stream\hls.py", line 162, in write
return self._write(sequence, *args, **kwargs)
File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\stream\hls.py", line 192, in _write
chunk = unpad(decrypted_chunk, AES.block_size, style="pkcs7")
File "C:\Program Files (x86)\Streamlink\pkgs\Crypto\Util\Padding.py", line 92, in unpad
raise ValueError("Padding is incorrect.")
ValueError: Padding is incorrect.
Exception in thread Thread-HLSStreamWriter:
Traceback (most recent call last):
File "threading.py", line 973, in _bootstrap_inner
File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\stream\segmented.py", line 200, in run
self.write(segment, result, *data)
File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\stream\hls.py", line 162, in write
return self._write(sequence, *args, **kwargs)
File "C:\Program Files (x86)\Streamlink\pkgs\streamlink\stream\hls.py", line 192, in _write
chunk = unpad(decrypted_chunk, AES.block_size, style="pkcs7")
File "C:\Program Files (x86)\Streamlink\pkgs\Crypto\Util\Padding.py", line 92, in unpad
raise ValueError("Padding is incorrect.")
ValueError: Padding is incorrect.
```



Discussion:
There's a bug in the HLS playlist parser:
https://github.com/streamlink/streamlink/blob/3.0.3/src/streamlink/stream/hls_playlist.py#L217

The [BYTERANGE tag's offset value is optional](https://datatracker.ietf.org/doc/html/rfc8216#section-4.3.2.2) but the parser always sets the value to 0 if it doesn't exist, which is obviously wrong.

[The `create_request_params` code where the `Range` header gets set](https://github.com/streamlink/streamlink/blob/3.0.3/src/streamlink/stream/hls.py#L90-L106) stems from [the Livestreamer era in 2013](39008e9df12723e59b77c2da428a062ea7e1d857) and hasn't been changed since then apart from some minor stuff. Unfortunately nobody has written tests for HLS byteranges, so a side effect in the HLS playlist parser has broken the byterange requests.

However, there's another issue with how the `Range` header gets set. The `HLSStreamWriter` has an infinitely growing cache for offsets of previous segments (`byterange_offsets`). This cache depends on the segment's URL, which is incorrect. The HLS spec says that only the previous segment should be used for missing offset values, and this previous segment can have a different URL.

Another problem is that the `HLSStreamWriter.fetch` method (which calls `create_request_params`) is run by other threads from the writer's thread-pool, so a sequential order of requests is not guaranteed. The individual `Segment` objects don't have a reference to their previous segment, and the `Sequence` object which holds the `Segment` object and its sequence number currently doesn't get passed to the `create_request_params` method, so a single "previous segment byterange offset cache" value can't be used without changing that.

Let me quickly fix the HLS playlist parser first though before I take a look at the other issue.

Resolving patch:
diff --git a/src/streamlink/stream/hls_playlist.py b/src/streamlink/stream/hls_playlist.py
--- a/src/streamlink/stream/hls_playlist.py
+++ b/src/streamlink/stream/hls_playlist.py
@@ -27,7 +27,7 @@ class ExtInf(NamedTuple):
 # EXT-X-BYTERANGE
 class ByteRange(NamedTuple):  # version >= 4
     range: int
-    offset: int
+    offset: Optional[int]
 
 
 # EXT-X-DATERANGE
@@ -152,7 +152,7 @@ def is_date_in_daterange(cls, date: Segment.date, daterange: DateRange):
 class M3U8Parser:
     _extinf_re = re.compile(r"(?P<duration>\d+(\.\d+)?)(,(?P<title>.+))?")
     _attr_re = re.compile(r"([A-Z\-]+)=(\d+\.\d+|0x[0-9A-z]+|\d+x\d+|\d+|\"(.+?)\"|[0-9A-z\-]+)")
-    _range_re = re.compile(r"(?P<range>\d+)(@(?P<offset>.+))?")
+    _range_re = re.compile(r"(?P<range>\d+)(?:@(?P<offset>\d+))?")
     _tag_re = re.compile(r"#(?P<tag>[\w-]+)(:(?P<value>.+))?")
     _res_re = re.compile(r"(\d+)x(\d+)")
 
@@ -214,7 +214,10 @@ def parse_bool(value: str) -> bool:
 
     def parse_byterange(self, value: str) -> Optional[ByteRange]:
         match = self._range_re.match(value)
-        return None if match is None else ByteRange(int(match.group("range")), int(match.group("offset") or 0))
+        if match is None:
+            return None
+        _range, offset = match.groups()
+        return ByteRange(int(_range), int(offset) if offset is not None else None)
 
     def parse_extinf(self, value: str) -> Tuple[float, Optional[str]]:
         match = self._extinf_re.match(value)


Test patch: