Repository: streamlink/streamlink

Issue:
stream.hls: stop stream early when EXT-X-ENDLIST is missing
When a live stream ends, streamlink retries 12 times and then exits. (12 is a strange number and I'm not sure where it comes from. `hls-segment-attempts` is default 3, so maybe the code that uses it is being done 4 times, for a total of 12. That's just stabbing in the dark though.)

I would like to limit that to only 1 retry, because I'm worried about hitting the server too much. I was once IP banned from a server for using an automated download program, and it now makes me worry about making too many attempts-- especially attempts which are clearly automated.

Ideally, when a live stream ends, I'd like to wait 60 seconds, then retry just once. Is there a way to do that?

The message I get when a stream ends is `[stream.hls][warning] Failed to reload playlist: Unable to open URL: ... (403 Client Error: Forbidden for url:..)`, which comes from hls.py.

This is streamlink 0.14.2 on both boxes I used (Ubuntu and Windows).

---

Update: I just tried this on my Windows box, testing by turning the internet off. The first time it made 8 attempts; the second time it made 12. This was while using `--hls-playlist-reload-attempts=1`.

Update2: I actually timed the time it took to finally die and it was 38 seconds. Combined with the dead airtime before the error messages start, that's getting close to 60 seconds. Is `hls-timeout` what I need?

Update3: It looks like the time it takes can be controlled with `hls-timeout`, and these 8-12 attempts I am seeing were actually just a loop of attempts until that time runs out. So it looks like I can prevent _any_ attempts by setting that to a low value, like 15.0. That's better than nothing, although I'd still rather have it pause for a minute, then exit if it gets another 403.


Discussion:
Since this was just mentioned in another thread and since I've refactored the HLS code a bit recently, what you've said regarding `--hls-timeout` (removed in favor of the generic `--stream-timeout` parameter) is correct. This parameter sets the read timeout for the output buffer in [`SegmentedStreamReader`](https://github.com/streamlink/streamlink/blob/db73cb2b00efaf54a13d6a6c642f49ea4777c6b2/src/streamlink/stream/segmented.py#L207-L236), which is 60 seconds by default. If no data was written to the buffer and there was a read attempt by the stream consumer, [this read attempt will time out](https://github.com/streamlink/streamlink/blob/master/src/streamlink/buffers.py#L104-L109) after these 60 seconds and an `OSError` will be raised with a "read timeout".

Another way of making Streamlink stop after a certain period of time when a stream has stopped but no [`#EXT-X-ENDLIST`](https://datatracker.ietf.org/doc/html/rfc8216#section-4.3.3.4) tag was set could be checking for new HLS segments to be queued. The HLSStreamWorker will continue refreshing the playlist when this tag is missing, but if there are no new segments (sequence number incrementation), this could be checked and the stream be ended this way when a certain time threshold has been reached. This would not be a graceful way of ending the stream, but the read timeout isn't either, and it's the responsibility of the server to include the endlist tag to signal the actual end of the stream.
I have a local branch here which will make Streamlink end the stream early when the HLS playlist didn't contain new segments for an X amount of time. This threshold is currently set to `TARGETDURATION * 2`.

That factor `2` is chosen very liberally, despite the HLS spec saying that the playlist's [`TARGETDURATION`](https://datatracker.ietf.org/doc/html/rfc8216#section-4.3.3.1) is the max duration of any segment, which means that when this time has passed, new segments MUST be available. The factor could maybe be reduced to 1.5 or so, but what I want to avoid are potential issues during stream discontinuities like embedded ads, where the targetduration could change to a different value that could result in the stream ending early by accident. The segment-queue timing threshold is independent of the playlist reload time since the reload time can be changed via session options and via custom stream implementations, like Twitch low latency streaming. I'm also not sure yet if I want to add a new session option that can change the value.

Since the changes depend on #5321 which is currently unmerged, I can't open a new PR for that just yet.

Resolving patch:
diff --git a/src/streamlink/stream/hls.py b/src/streamlink/stream/hls.py
--- a/src/streamlink/stream/hls.py
+++ b/src/streamlink/stream/hls.py
@@ -292,13 +292,17 @@ class HLSStreamWorker(SegmentedStreamWorker):
     writer: "HLSStreamWriter"
     stream: "HLSStream"
 
+    SEGMENT_QUEUE_TIMING_THRESHOLD_FACTOR = 2
+
     def __init__(self, *args, **kwargs) -> None:
         super().__init__(*args, **kwargs)
 
         self.playlist_changed = False
         self.playlist_end: Optional[int] = None
+        self.playlist_targetduration: float = 0
         self.playlist_sequence: int = -1
         self.playlist_sequences: List[Sequence] = []
+        self.playlist_sequences_last: datetime = now()
         self.playlist_reload_last: datetime = now()
         self.playlist_reload_time: float = 6
         self.playlist_reload_time_override = self.session.options.get("hls-playlist-reload-time")
@@ -353,6 +357,7 @@ def reload_playlist(self):
         sequences = [Sequence(media_sequence + i, s)
                      for i, s in enumerate(playlist.segments)]
 
+        self.playlist_targetduration = playlist.targetduration or 0
         self.playlist_reload_time = self._playlist_reload_time(playlist, sequences)
 
         if sequences:
@@ -365,8 +370,8 @@ def _playlist_reload_time(self, playlist: M3U8, sequences: List[Sequence]) -> fl
             return sum(s.segment.duration for s in sequences[-max(1, self.live_edge - 1):])
         if type(self.playlist_reload_time_override) is float and self.playlist_reload_time_override > 0:
             return self.playlist_reload_time_override
-        if playlist.target_duration:
-            return playlist.target_duration
+        if playlist.targetduration:
+            return playlist.targetduration
         if sequences:
             return sum(s.segment.duration for s in sequences[-max(1, self.live_edge - 1):])
 
@@ -398,6 +403,14 @@ def process_sequences(self, playlist: M3U8, sequences: List[Sequence]) -> None:
     def valid_sequence(self, sequence: Sequence) -> bool:
         return sequence.num >= self.playlist_sequence
 
+    def _segment_queue_timing_threshold_reached(self) -> bool:
+        threshold = self.playlist_targetduration * self.SEGMENT_QUEUE_TIMING_THRESHOLD_FACTOR
+        if now() <= self.playlist_sequences_last + timedelta(seconds=threshold):
+            return False
+
+        log.warning(f"No new segments in playlist for more than {threshold:.2f}s. Stopping...")
+        return True
+
     @staticmethod
     def duration_to_sequence(duration: float, sequences: List[Sequence]) -> int:
         d = 0.0
@@ -415,7 +428,10 @@ def duration_to_sequence(duration: float, sequences: List[Sequence]) -> int:
         return default
 
     def iter_segments(self):
-        self.playlist_reload_last = now()
+        self.playlist_reload_last \
+            = self.playlist_sequences_last \
+            = now()
+
         try:
             self.reload_playlist()
         except StreamError as err:
@@ -446,9 +462,12 @@ def iter_segments(self):
 
         total_duration = 0
         while not self.closed:
+            queued = False
             for sequence in filter(self.valid_sequence, self.playlist_sequences):
                 log.debug(f"Adding segment {sequence.num} to queue")
                 yield sequence
+                queued = True
+
                 total_duration += sequence.segment.duration
                 if self.duration_limit and total_duration >= self.duration_limit:
                     log.info(f"Stopping stream early after {self.duration_limit}")
@@ -461,6 +480,11 @@ def iter_segments(self):
 
                 self.playlist_sequence = sequence.num + 1
 
+            if queued:
+                self.playlist_sequences_last = now()
+            elif self._segment_queue_timing_threshold_reached():
+                return
+
             # Exclude playlist fetch+processing time from the overall playlist reload time
             # and reload playlist in a strict time interval
             time_completed = now()
diff --git a/src/streamlink/stream/hls_playlist.py b/src/streamlink/stream/hls_playlist.py
--- a/src/streamlink/stream/hls_playlist.py
+++ b/src/streamlink/stream/hls_playlist.py
@@ -129,7 +129,7 @@ def __init__(self, uri: Optional[str] = None):
         self.iframes_only: Optional[bool] = None  # version >= 4
         self.media_sequence: Optional[int] = None
         self.playlist_type: Optional[str] = None
-        self.target_duration: Optional[int] = None
+        self.targetduration: Optional[float] = None
         self.start: Optional[Start] = None
         self.version: Optional[int] = None
 
@@ -423,7 +423,7 @@ def parse_tag_ext_x_targetduration(self, value: str) -> None:
         EXT-X-TARGETDURATION
         https://datatracker.ietf.org/doc/html/rfc8216#section-4.3.3.1
         """
-        self.m3u8.target_duration = int(value)
+        self.m3u8.targetduration = float(value)
 
     def parse_tag_ext_x_media_sequence(self, value: str) -> None:
         """


Test patch:
diff --git a/tests/stream/test_hls.py b/tests/stream/test_hls.py
--- a/tests/stream/test_hls.py
+++ b/tests/stream/test_hls.py
@@ -202,6 +202,77 @@ def tearDown(self) -> None:
     def get_session(self, options=None, *args, **kwargs):
         return super().get_session({**self.OPTIONS, **(options or {})}, *args, **kwargs)
 
+    def test_segment_queue_timing_threshold_reached(self) -> None:
+        thread, segments = self.subject(
+            start=False,
+            playlists=[
+                Playlist(0, targetduration=5, segments=[Segment(0)]),
+                # no EXT-X-ENDLIST, last mocked playlist response will be repreated forever
+                Playlist(0, targetduration=5, segments=[Segment(0), Segment(1)]),
+            ],
+        )
+        worker: EventedHLSStreamWorker = thread.reader.worker
+        targetduration = ONE_SECOND * 5
+
+        with freezegun.freeze_time(EPOCH) as frozen_time, \
+             patch("streamlink.stream.hls.log") as mock_log:
+            self.start()
+
+            assert worker.handshake_reload.wait_ready(1), "Loads playlist for the first time"
+            assert worker.playlist_sequence == -1, "Initial sequence number"
+            assert worker.playlist_sequences_last == EPOCH, "Sets the initial last queue time"
+
+            # first playlist reload has taken one second
+            frozen_time.tick(ONE_SECOND)
+            self.await_playlist_reload(1)
+
+            assert worker.handshake_wait.wait_ready(1), "Arrives at first wait() call"
+            assert worker.playlist_sequence == 1, "Updates the sequence number"
+            assert worker.playlist_sequences_last == EPOCH + ONE_SECOND, "Updates the last queue time"
+            assert worker.playlist_targetduration == 5.0
+
+            # trigger next reload when the target duration has passed
+            frozen_time.tick(targetduration)
+            self.await_playlist_wait(1)
+            self.await_playlist_reload(1)
+
+            assert worker.handshake_wait.wait_ready(1), "Arrives at second wait() call"
+            assert worker.playlist_sequence == 2, "Updates the sequence number again"
+            assert worker.playlist_sequences_last == EPOCH + ONE_SECOND + targetduration, "Updates the last queue time again"
+            assert worker.playlist_targetduration == 5.0
+
+            # trigger next reload when the target duration has passed
+            frozen_time.tick(targetduration)
+            self.await_playlist_wait(1)
+            self.await_playlist_reload(1)
+
+            assert worker.handshake_wait.wait_ready(1), "Arrives at third wait() call"
+            assert worker.playlist_sequence == 2, "Sequence number is unchanged"
+            assert worker.playlist_sequences_last == EPOCH + ONE_SECOND + targetduration, "Last queue time is unchanged"
+            assert worker.playlist_targetduration == 5.0
+
+            # trigger next reload when the target duration has passed
+            frozen_time.tick(targetduration)
+            self.await_playlist_wait(1)
+            self.await_playlist_reload(1)
+
+            assert worker.handshake_wait.wait_ready(1), "Arrives at fourth wait() call"
+            assert worker.playlist_sequence == 2, "Sequence number is unchanged"
+            assert worker.playlist_sequences_last == EPOCH + ONE_SECOND + targetduration, "Last queue time is unchanged"
+            assert worker.playlist_targetduration == 5.0
+
+            assert mock_log.warning.call_args_list == []
+
+            # trigger next reload when the target duration has passed
+            frozen_time.tick(targetduration)
+            self.await_playlist_wait(1)
+            self.await_playlist_reload(1)
+
+            self.await_read(read_all=True)
+            self.await_close(1)
+
+            assert mock_log.warning.call_args_list == [call("No new segments in playlist for more than 10.00s. Stopping...")]
+
     def test_playlist_reload_offset(self) -> None:
         thread, segments = self.subject(
             start=False,
@@ -689,6 +760,7 @@ def test_hls_playlist_reload_time_no_data(self):
 
 @patch("streamlink.stream.hls.log")
 @patch("streamlink.stream.hls.HLSStreamWorker.wait", Mock(return_value=True))
+@patch("streamlink.stream.hls.HLSStreamWorker._segment_queue_timing_threshold_reached", Mock(return_value=False))
 class TestHlsPlaylistParseErrors(TestMixinStreamHLS, unittest.TestCase):
     __stream__ = EventedWriterHLSStream
 
diff --git a/tests/stream/test_hls_playlist.py b/tests/stream/test_hls_playlist.py
--- a/tests/stream/test_hls_playlist.py
+++ b/tests/stream/test_hls_playlist.py
@@ -381,7 +381,7 @@ def test_parse_date(self):
         delta_30 = timedelta(seconds=30, milliseconds=500)
         delta_60 = timedelta(seconds=60)
 
-        assert playlist.target_duration == 120
+        assert playlist.targetduration == 120
 
         assert list(playlist.dateranges) == [
             DateRange(