Repository: streamlink/streamlink

Issue:
stream.hls: streamlink incorrectly decrypts media initialization section and gets error
### Checklist

- [X] This is a bug report and not [a different kind of issue](https://github.com/streamlink/streamlink/issues/new/choose)
- [X] [I have read the contribution guidelines](https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#contributing-to-streamlink)
- [X] [I have checked the list of open and recently closed bug reports](https://github.com/streamlink/streamlink/issues?q=is%3Aissue+label%3A%22bug%22)
- [X] [I have checked the commit log of the master branch](https://github.com/streamlink/streamlink/commits/master)

### Streamlink version

streamlink 6.6.2

### Description

I recently encountered an HLS playlist that streamlink incorrectly decrypts media initialization section and gets: `Error while decrypting segment 1: Data must be padded to 16 byte boundary in CBC mode`.

As shown in the `-l all` output (please see [this gist](https://gist.github.com/thisisgk/8c9e4a4c3ed1ccf9008a0eefd40c6c3b)), `#EXT-X-MAP:URI=...` is before `#EXT-X-KEY:METHOD=AES-128,URI=...`, so media initialization section is plain data.

Please note that this playlist might not be accessible from outside Japan and will expire in a day.


### Debug log

```text
$ streamlink -l debug --hls-duration 00:20 --http-cookie domand_bid=5e377f548bdfca20defd15f0b178b8b42e538ed14569c97f6338178cde548a49 'https://delivery.domand.nicovideo.jp/hlsbid/65d4695bd76609a88f8bfe85/playlists/media/audio-aac-192kbps.m3u8?session=8b75576e7a29181c14eb064f0359172c7d7896c56598baea0000000065da9fe6f06982f4871bf258&Expires=1708826598&Signature=r0s0nwOBDvaFFKTHFC22EuBanWTXQ0gSXQ2gDAn41bDDtAWsSb1k3yxKAO6-X96as8A2jH4pvoYiw8EKO4p704xTcRGijnmxy2fSBelLhSZ~C0KZNXnWNzgmF15zPKGc1BS8py2RF1UpYRP9y23hPbQR2eWh9c1-XEFnK~drvCltt5O8NgiR79OkZ5WzxR2mhvReS7XUoiNekR1mQsagRsTlZ-Tyktod8imWiMwPGQ6rx0zfohFliGh6Iwb~AJqLS7B4B3FWOxrsA2MTe4SxX97yoLReZ-sNZNrdwc-P5vZ8HSeiGERR4riGyNJpKEalsfG4OKHb-BoIGM2yWjjZeQ__&Key-Pair-Id=K11RB80NFXU134' best
[session][debug] Loading plugin: hls
[cli][debug] OS:         macOS 10.13.6
[cli][debug] Python:     3.12.2
[cli][debug] OpenSSL:    OpenSSL 1.1.1w  11 Sep 2023
[cli][debug] Streamlink: 6.6.2
[cli][debug] Dependencies:
[cli][debug]  certifi: 2024.2.2
[cli][debug]  isodate: 0.6.1
[cli][debug]  lxml: 5.1.0
[cli][debug]  pycountry: 23.12.11
[cli][debug]  pycryptodome: 3.20.0
[cli][debug]  PySocks: 1.7.1
[cli][debug]  requests: 2.31.0
[cli][debug]  trio: 0.24.0
[cli][debug]  trio-websocket: 0.11.1
[cli][debug]  typing-extensions: 4.9.0
[cli][debug]  urllib3: 2.2.1
[cli][debug]  websocket-client: 1.7.0
[cli][debug] Arguments:
[cli][debug]  url=https://delivery.domand.nicovideo.jp/hlsbid/65d4695bd76609a88f8bfe85/playlists/media/audio-aac-192kbps.m3u8?session=8b75576e7a29181c14eb064f0359172c7d7896c56598baea0000000065da9fe6f06982f4871bf258&Expires=1708826598&Signature=r0s0nwOBDvaFFKTHFC22EuBanWTXQ0gSXQ2gDAn41bDDtAWsSb1k3yxKAO6-X96as8A2jH4pvoYiw8EKO4p704xTcRGijnmxy2fSBelLhSZ~C0KZNXnWNzgmF15zPKGc1BS8py2RF1UpYRP9y23hPbQR2eWh9c1-XEFnK~drvCltt5O8NgiR79OkZ5WzxR2mhvReS7XUoiNekR1mQsagRsTlZ-Tyktod8imWiMwPGQ6rx0zfohFliGh6Iwb~AJqLS7B4B3FWOxrsA2MTe4SxX97yoLReZ-sNZNrdwc-P5vZ8HSeiGERR4riGyNJpKEalsfG4OKHb-BoIGM2yWjjZeQ__&Key-Pair-Id=K11RB80NFXU134
[cli][debug]  stream=['best']
[cli][debug]  --loglevel=debug
[cli][debug]  --player=mpv
[cli][debug]  --verbose-player=True
[cli][debug]  --player-no-close=True
[cli][debug]  --hls-duration=20.0
[cli][debug]  --http-cookie=[('domand_bid', '5e377f548bdfca20defd15f0b178b8b42e538ed14569c97f6338178cde548a49')]
[cli][info] Found matching plugin hls for URL https://delivery.domand.nicovideo.jp/hlsbid/65d4695bd76609a88f8bfe85/playlists/media/audio-aac-192kbps.m3u8?session=8b75576e7a29181c14eb064f0359172c7d7896c56598baea0000000065da9fe6f06982f4871bf258&Expires=1708826598&Signature=r0s0nwOBDvaFFKTHFC22EuBanWTXQ0gSXQ2gDAn41bDDtAWsSb1k3yxKAO6-X96as8A2jH4pvoYiw8EKO4p704xTcRGijnmxy2fSBelLhSZ~C0KZNXnWNzgmF15zPKGc1BS8py2RF1UpYRP9y23hPbQR2eWh9c1-XEFnK~drvCltt5O8NgiR79OkZ5WzxR2mhvReS7XUoiNekR1mQsagRsTlZ-Tyktod8imWiMwPGQ6rx0zfohFliGh6Iwb~AJqLS7B4B3FWOxrsA2MTe4SxX97yoLReZ-sNZNrdwc-P5vZ8HSeiGERR4riGyNJpKEalsfG4OKHb-BoIGM2yWjjZeQ__&Key-Pair-Id=K11RB80NFXU134
[plugins.hls][debug] URL=https://delivery.domand.nicovideo.jp/hlsbid/65d4695bd76609a88f8bfe85/playlists/media/audio-aac-192kbps.m3u8?session=8b75576e7a29181c14eb064f0359172c7d7896c56598baea0000000065da9fe6f06982f4871bf258&Expires=1708826598&Signature=r0s0nwOBDvaFFKTHFC22EuBanWTXQ0gSXQ2gDAn41bDDtAWsSb1k3yxKAO6-X96as8A2jH4pvoYiw8EKO4p704xTcRGijnmxy2fSBelLhSZ~C0KZNXnWNzgmF15zPKGc1BS8py2RF1UpYRP9y23hPbQR2eWh9c1-XEFnK~drvCltt5O8NgiR79OkZ5WzxR2mhvReS7XUoiNekR1mQsagRsTlZ-Tyktod8imWiMwPGQ6rx0zfohFliGh6Iwb~AJqLS7B4B3FWOxrsA2MTe4SxX97yoLReZ-sNZNrdwc-P5vZ8HSeiGERR4riGyNJpKEalsfG4OKHb-BoIGM2yWjjZeQ__&Key-Pair-Id=K11RB80NFXU134; params={}
[utils.l10n][debug] Language code: en_US
[cli][info] Available streams: live (worst, best)
[cli][info] Opening stream: live (hls)
[cli][info] Starting player: mpv
[stream.hls][debug] Reloading playlist
[cli][debug] Pre-buffering 8192 bytes
[stream.hls][debug] Segments in this playlist are encrypted
[stream.hls][debug] First Sequence: 1; Last Sequence: 29
[stream.hls][debug] Start offset: 0; Duration: 20; Start Sequence: 1; End Sequence: 29
[stream.hls][debug] Adding segment 1 to queue
[stream.hls][debug] Adding segment 2 to queue
[stream.hls][debug] Adding segment 3 to queue
[stream.hls][debug] Adding segment 4 to queue
[stream.hls][info] Stopping stream early after 20
[stream.segmented][debug] Closing worker thread
[stream.hls][debug] Writing segment 1 to output
[stream.hls][error] Error while decrypting segment 1: Data must be padded to 16 byte boundary in CBC mode
[stream.hls][debug] Writing segment 1 to output
[stream.hls][debug] Segment 1 complete
[stream.hls][debug] Writing segment 2 to output
[cli.output][debug] Opening subprocess: ['/usr/local/bin/mpv', '--force-media-title=https://delivery.domand.nicovideo.jp/hlsbid/65d4695bd76609a88f8bfe85/playlists/media/audio-aac-192kbps.m3u8?session=8b75576e7a29181c14eb064f0359172c7d7896c56598baea0000000065da9fe6f06982f4871bf258&Expires=1708826598&Signature=r0s0nwOBDvaFFKTHFC22EuBanWTXQ0gSXQ2gDAn41bDDtAWsSb1k3yxKAO6-X96as8A2jH4pvoYiw8EKO4p704xTcRGijnmxy2fSBelLhSZ~C0KZNXnWNzgmF15zPKGc1BS8py2RF1UpYRP9y23hPbQR2eWh9c1-XEFnK~drvCltt5O8NgiR79OkZ5WzxR2mhvReS7XUoiNekR1mQsagRsTlZ-Tyktod8imWiMwPGQ6rx0zfohFliGh6Iwb~AJqLS7B4B3FWOxrsA2MTe4SxX97yoLReZ-sNZNrdwc-P5vZ8HSeiGERR4riGyNJpKEalsfG4OKHb-BoIGM2yWjjZeQ__&Key-Pair-Id=K11RB80NFXU134', '-']
[stream.hls][debug] Segment 2 complete
[stream.hls][debug] Writing segment 3 to output
[stream.hls][debug] Segment 3 complete
[stream.hls][debug] Writing segment 4 to output
[stream.hls][debug] Segment 4 complete
[stream.segmented][debug] Closing writer thread
[cli][debug] Writing stream to output
[file] Reading from stdin...
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: could not find corresponding track id 1
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: could not find corresponding trex (id 1)
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: could not find corresponding track id 0
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: trun track id unknown, no tfhd was found
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: error reading header
[lavf] avformat_open_input() failed
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: could not find corresponding track id 1
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: could not find corresponding trex (id 1)
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: could not find corresponding track id 0
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: trun track id unknown, no tfhd was found
[ffmpeg/demuxer] mov,mp4,m4a,3gp,3g2,mj2: error reading header
[lavf] avformat_open_input() failed
Failed to recognize file format.

Exiting... (Errors when loading file)
[cli][info] Player closed
[cli][info] Stream ended
[cli][info] Closing currently open stream...
$
```



Discussion:
Thanks for the bug report.

The issue is this:

The data of both `EXT-X-MAP` and `EXT-X-KEY` are applied to each [`HLSSegment`](https://github.com/streamlink/streamlink/blob/6.6.2/src/streamlink/stream/hls/segment.py#L99-L106) item. This is obviously the wrong choice of data structure in this special case here where the initialization section is not encrypted. The `HLSStreamWorker` will [always assume that the initialization section is encrypted](https://github.com/streamlink/streamlink/blob/6.6.2/src/streamlink/stream/hls/hls.py#L245-L246), even if the decryption key was set after the initialization section.

Fixing this issue will very likely require a lot of refactoring of the `M3U8Parser`, the `HLSStreamWorker` and `HLSStreamWriter`.

The DASH implementation for example - while of course completely different - wouldn't have this kind of issue because [there's a clear distinction between initialization section segments and regular media segments](https://github.com/streamlink/streamlink/blob/6.6.2/src/streamlink/stream/dash/segment.py#L25). The HLS implementation never made this segment-type distinction and simply added the map attribute to segments.

The whole HLS media initialization section was also very much flawed in a different way until last year, because there was a bit of confusion with no clear answer when it was implemented first (due to MPEG-TS streams). Initialization sections were applied to each segment in the output, which is of course wrong. Demuxers however skipped this redundant data, so it was never a real problem apart from warning messages. As said, this was fixed last year, but the old implementation was pretty much kept. This issue here now justifies a full rewrite based on how the DASH implementation handles initialization sections, namely as separate segments that are yielded by the worker thread. This way, initialization sections can have independent `key` attributes which are not tied to the media segments that follow.

I don't see a reason to fix this issue here based on the current implementation, so a bug fix will have to wait until the rewrite has been completed, which as said, will be a lot of work.
Had a look at this again and decided to fix this with the current implementation. It's not a nice solution, but it'll work.
#5861

Feedback is probably not required, but in case you want to test the PR, see here:
https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#pull-request-feedback
Confirmed the PR works. Thanks.



Patch:
diff --git a/src/streamlink/stream/hls/hls.py b/src/streamlink/stream/hls/hls.py
index c342d7e4596..6c7c1c78f61 100644
--- a/src/streamlink/stream/hls/hls.py
+++ b/src/streamlink/stream/hls/hls.py
@@ -243,9 +243,12 @@ def write(self, segment: HLSSegment, result: Response, *data):
                 self.reader.pause()
 
     def _write(self, segment: HLSSegment, result: Response, is_map: bool):
-        if segment.key and segment.key.method != "NONE":
+        # TODO: Rewrite HLSSegment, HLSStreamWriter and HLSStreamWorker based on independent initialization section segments,
+        #       similar to the DASH implementation
+        key = segment.map.key if is_map and segment.map else segment.key
+        if key and key.method != "NONE":
             try:
-                decryptor = self.create_decryptor(segment.key, segment.num)
+                decryptor = self.create_decryptor(key, segment.num)
             except (StreamError, ValueError) as err:
                 log.error(f"Failed to create decryptor: {err}")
                 self.close()
diff --git a/src/streamlink/stream/hls/m3u8.py b/src/streamlink/stream/hls/m3u8.py
index 0895058922f..2c22db18c60 100644
--- a/src/streamlink/stream/hls/m3u8.py
+++ b/src/streamlink/stream/hls/m3u8.py
@@ -353,6 +353,7 @@ def parse_tag_ext_x_map(self, value: str) -> None:  # version >= 5
         byterange = self.parse_byterange(attr.get("BYTERANGE", ""))
         self._map = Map(
             uri=self.uri(uri),
+            key=self._key,
             byterange=byterange,
         )
 
diff --git a/src/streamlink/stream/hls/segment.py b/src/streamlink/stream/hls/segment.py
index 2c2e709103b..0e82812a6c8 100644
--- a/src/streamlink/stream/hls/segment.py
+++ b/src/streamlink/stream/hls/segment.py
@@ -46,6 +46,7 @@ class Key(NamedTuple):
 # EXT-X-MAP
 class Map(NamedTuple):
     uri: str
+    key: Optional[Key]
     byterange: Optional[ByteRange]
 
 


Test patch:
diff --git a/tests/stream/hls/test_hls.py b/tests/stream/hls/test_hls.py
index a1386930eb3..31b4347b44c 100644
--- a/tests/stream/hls/test_hls.py
+++ b/tests/stream/hls/test_hls.py
@@ -1,3 +1,4 @@
+import itertools
 import os
 import typing
 import unittest
@@ -59,7 +60,7 @@ class TagMapEnc(EncryptedBase, TagMap):
 
 
 class TagKey(Tag):
-    path = "encryption.key"
+    _id = itertools.count()
 
     def __init__(self, method="NONE", uri=None, iv=None, keyformat=None, keyformatversions=None):
         attrs = {"METHOD": method}
@@ -73,6 +74,7 @@ def __init__(self, method="NONE", uri=None, iv=None, keyformat=None, keyformatve
             attrs["KEYFORMATVERSIONS"] = self.val_quoted_string(keyformatversions)
         super().__init__("EXT-X-KEY", attrs)
         self.uri = uri
+        self.path = f"encryption{next(self._id)}.key"
 
     def url(self, namespace):
         return self.uri.format(namespace=namespace) if self.uri else super().url(namespace)
@@ -718,7 +720,6 @@ def test_hls_encrypted_aes128_with_map(self):
         self.mock("GET", self.url(map1), content=map1.content)
         self.mock("GET", self.url(map2), content=map2.content)
 
-        # noinspection PyTypeChecker
         segments = self.subject([
             Playlist(0, [key, map1] + [SegmentEnc(num, aesKey, aesIv) for num in range(2)]),
             Playlist(2, [key, map2] + [SegmentEnc(num, aesKey, aesIv) for num in range(2, 4)], end=True),
@@ -732,6 +733,52 @@ def test_hls_encrypted_aes128_with_map(self):
             map1, segments[0], segments[1], map2, segments[2], segments[3],
         ], prop="content_plain")
 
+    def test_hls_encrypted_aes128_with_differently_encrypted_map(self):
+        aesKey1, aesIv1, key1 = self.gen_key()  # init key
+        aesKey2, aesIv2, key2 = self.gen_key()  # media key
+        map1 = TagMapEnc(1, namespace=self.id(), key=aesKey1, iv=aesIv1)
+        map2 = TagMapEnc(2, namespace=self.id(), key=aesKey1, iv=aesIv1)
+        self.mock("GET", self.url(map1), content=map1.content)
+        self.mock("GET", self.url(map2), content=map2.content)
+
+        segments = self.subject([
+            Playlist(0, [key1, map1, key2] + [SegmentEnc(num, aesKey2, aesIv2) for num in range(2)]),
+            Playlist(2, [key1, map2, key2] + [SegmentEnc(num, aesKey2, aesIv2) for num in range(2, 4)], end=True),
+        ])
+
+        self.await_write(1 + 2 + 1 + 2)  # 1 map, 2 segments, 1 map, 2 segments
+        data = self.await_read(read_all=True)
+        self.await_close()
+
+        assert data == self.content([
+            map1, segments[0], segments[1], map2, segments[2], segments[3],
+        ], prop="content_plain")
+
+    def test_hls_encrypted_aes128_with_plaintext_map(self):
+        aesKey, aesIv, key = self.gen_key()
+        map1 = TagMap(1, namespace=self.id())
+        map2 = TagMap(2, namespace=self.id())
+        self.mock("GET", self.url(map1), content=map1.content)
+        self.mock("GET", self.url(map2), content=map2.content)
+
+        segments = self.subject([
+            Playlist(0, [map1, key] + [SegmentEnc(num, aesKey, aesIv) for num in range(2)]),
+            Playlist(2, [map2, key] + [SegmentEnc(num, aesKey, aesIv) for num in range(2, 4)], end=True),
+        ])
+
+        self.await_write(1 + 2 + 1 + 2)  # 1 map, 2 segments, 1 map, 2 segments
+        data = self.await_read(read_all=True)
+        self.await_close()
+
+        assert data == (
+            map1.content
+            + segments[0].content_plain
+            + segments[1].content_plain
+            + map2.content
+            + segments[2].content_plain
+            + segments[3].content_plain
+        )
+
     def test_hls_encrypted_aes128_key_uri_override(self):
         aesKey, aesIv, key = self.gen_key(uri="http://real-mocked/{namespace}/encryption.key?foo=bar")
         aesKeyInvalid = bytes(ord(aesKey[i:i + 1]) ^ 0xFF for i in range(16))