Repository: streamlink/streamlink

Issue:
HLS tests randomly failing on Windows due to threading issues
### Checklist

- [X] This is a bug report and not a different kind of issue
- [X] [I have read the contribution guidelines](https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#contributing-to-streamlink)
- [X] [I have checked the list of open and recently closed bug reports](https://github.com/streamlink/streamlink/issues?q=is%3Aissue+label%3A%22bug%22)
- [X] [I have checked the commit log of the master branch](https://github.com/streamlink/streamlink/commits/master)

### Streamlink version

Latest build from the master branch

### Description

This only started happening yesterday, and I still haven't understood why. I'm unable to reproduce this locally and it's only failing in the CI jobs. There haven't been any python or dependency upgrades prior to this, so this must be caused by an upgrade of the Windows CI environment, I think.
https://github.com/streamlink/streamlink/actions

----

For some reason, the following tests (so far) have been randomly failing due to the HLS stream's writer thread not writing the segments correctly. What is weird though is that these tests are supposed to be deterministic with lots of thread locks for writing and reading each segment, so there shouldn't be any timing issues between the threads.
- [`tests/streams/test_hls_filtered.py::TestFilteredHLSStream::test_filtered_closed`](https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/tests/streams/test_hls_filtered.py#L133-L167)
- [`tests/streams/test_hls_filtered.py::TestFilteredHLSStream::test_filtered_logging`](https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/tests/streams/test_hls_filtered.py#L57-L88)
- [`tests/streams/test_hls.py::TestHLSStreamEncrypted::test_hls_encrypted_aes128_incorrect_padding_length`](https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/tests/streams/test_hls.py#L243-L253)

----

Take a look at the [`test_filtered_closed`](https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/tests/streams/test_hls_filtered.py#L133-L167) integration test for example, which is the most simple one to understand. It sets up an HLS stream with only two segments which should all get filtered out. Ignore the additional `wait` mock, as it is not important for the issue.

The first assertion checks whether the reader thread's `filter_event` lock is initially set, [which is true](https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/src/streamlink/stream/hls.py#L377-L378), as it hasn't begun writing yet.

The next statement is an `await_write()` call, which makes the main thread that runs the test wait until the first `write()` call on the writer thread has finished. Since the first segment gets filtered out, [the reader's `filter_event` should be cleared](https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/src/streamlink/stream/hls.py#L172-L175) and the stream output be paused (so that read calls won't time out without data).

For some reason, and I absolutely don't understand why, the next assertion, which checks the state of the `filter_event` lock again, fails. See the log here from my debug branch:
https://github.com/bastimeyer/streamlink/runs/3067746756?check_suite_focus=true#step:6:51

How is this possible when `await_write()` ensures that `write()` on the writer thread has completed?
- https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/tests/mixins/stream_hls.py#L227-L235
- https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/tests/mixins/stream_hls.py#L85-L106

What is going on here?

### Debug log

```text
-
```

HLS tests randomly failing on Windows due to threading issues
### Checklist

- [X] This is a bug report and not a different kind of issue
- [X] [I have read the contribution guidelines](https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#contributing-to-streamlink)
- [X] [I have checked the list of open and recently closed bug reports](https://github.com/streamlink/streamlink/issues?q=is%3Aissue+label%3A%22bug%22)
- [X] [I have checked the commit log of the master branch](https://github.com/streamlink/streamlink/commits/master)

### Streamlink version

Latest build from the master branch

### Description

This only started happening yesterday, and I still haven't understood why. I'm unable to reproduce this locally and it's only failing in the CI jobs. There haven't been any python or dependency upgrades prior to this, so this must be caused by an upgrade of the Windows CI environment, I think.
https://github.com/streamlink/streamlink/actions

----

For some reason, the following tests (so far) have been randomly failing due to the HLS stream's writer thread not writing the segments correctly. What is weird though is that these tests are supposed to be deterministic with lots of thread locks for writing and reading each segment, so there shouldn't be any timing issues between the threads.
- [`tests/streams/test_hls_filtered.py::TestFilteredHLSStream::test_filtered_closed`](https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/tests/streams/test_hls_filtered.py#L133-L167)
- [`tests/streams/test_hls_filtered.py::TestFilteredHLSStream::test_filtered_logging`](https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/tests/streams/test_hls_filtered.py#L57-L88)
- [`tests/streams/test_hls.py::TestHLSStreamEncrypted::test_hls_encrypted_aes128_incorrect_padding_length`](https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/tests/streams/test_hls.py#L243-L253)

----

Take a look at the [`test_filtered_closed`](https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/tests/streams/test_hls_filtered.py#L133-L167) integration test for example, which is the most simple one to understand. It sets up an HLS stream with only two segments which should all get filtered out. Ignore the additional `wait` mock, as it is not important for the issue.

The first assertion checks whether the reader thread's `filter_event` lock is initially set, [which is true](https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/src/streamlink/stream/hls.py#L377-L378), as it hasn't begun writing yet.

The next statement is an `await_write()` call, which makes the main thread that runs the test wait until the first `write()` call on the writer thread has finished. Since the first segment gets filtered out, [the reader's `filter_event` should be cleared](https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/src/streamlink/stream/hls.py#L172-L175) and the stream output be paused (so that read calls won't time out without data).

For some reason, and I absolutely don't understand why, the next assertion, which checks the state of the `filter_event` lock again, fails. See the log here from my debug branch:
https://github.com/bastimeyer/streamlink/runs/3067746756?check_suite_focus=true#step:6:51

How is this possible when `await_write()` ensures that `write()` on the writer thread has completed?
- https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/tests/mixins/stream_hls.py#L227-L235
- https://github.com/streamlink/streamlink/blob/d04767fa5b812a4e971e454226cecb25b4d975ae/tests/mixins/stream_hls.py#L85-L106

What is going on here?

### Debug log

```text
-
```



Discussion:
I've been working on this issue for the past couple of days and was able to fix some of the problems with the EventedHLSStreamWriter related tests (and more). There's also some code refactoring of the HLS- and Segmented- classes. I still need to properly rebase my changes before I'm able to submit a PR.

I haven't been able to fully fix the issue yet though, and after already spending more time with debugging this than I would've liked, I'm still terribly confused. As I've said in another PR a couple of days ago, the tests are passing just fine while running the entire test suite, but they fail when running only a subset of the tests, eg. only `tests/streams`. Individual tests are also running just fine.

Just as a brief recap of what the `EventedHLSStreamWriter` class does: it subclasses the `HLSStreamWriter` and waits on each `write()` call for the main thread (which runs the test case) to instruct a write call, so that it can step through the stream write calls individually and make assertions about the HLS stream state. This `await_write()` method on the main thread does have a time limit, in case something goes wrong, so that the test can fail eventually, which means there is a certain time frame where the writer thread needs to make the instructed write call. Same for the main thread which needs to make the `await_write()` call in a certain time frame (the writer thread currently waits indefinitely in this case, but I have fixed this locally).

However, and I absolutely don't understand this, in some cases, the writer thread completely stands still or runs so slowly that it misses some of the time frames and the tests fail, but only when running a subset of all tests. This has nothing to do with thread locks, as this issue occurs when releasing the locks and waking up the writer thread. I have seen this lots of times on my local machine after adding log messages with timestamps all over the code, and I have no explanation for this. It doesn't make sense that there are gaps of several seconds between simple statements or lines of code. I suspect that there is an issue with the way the thread gets woken up.
Take a look at this test branch where I've downgraded requests from 2.26 to 2.25.1:
https://github.com/bastimeyer/streamlink/actions/runs/1056730773
The Windows tests are always passing. I've restarted the CI jobs several times now.
With requests 2.26 and without skipping the tests, at least one of the Windows CI jobs is always failing, every single time.

**This means that there must be an issue with requests-mock and requests 2.26** as we're not doing any real HTTP requests in the tests. And this perfectly makes sense since I've seen random requests-mock responses taking more than 5 seconds, even on my local system, which surely is not right, and this is the cause for the timing window misses in the EventedHLSStreamWriter tests, because either the writer's fetch() method takes too long for some segments to be written or the encryption key takes too long to load to decrypt a segment (or both), which is why the HLS encryption tests fail more often than the other ones.

This further means that even if the tests and test setups get improved and tweaked, there will be no guarantee that this will overall be fixed.

I haven't looked at the code changes of the requests 2.26 release yet, but if one wants to help, please do.

----

We've upgraded requests to `>=2.26.0,<3.0` in #3864 to fix the Windows installer build due to the new dependency tree which needs to be defined explicitly, but what we should actually do is force requests to 2.25.1 on Windows and continue like this until the 2.26 issue with requests-mock is fixed.
I've been working on this issue for the past couple of days and was able to fix some of the problems with the EventedHLSStreamWriter related tests (and more). There's also some code refactoring of the HLS- and Segmented- classes. I still need to properly rebase my changes before I'm able to submit a PR.

I haven't been able to fully fix the issue yet though, and after already spending more time with debugging this than I would've liked, I'm still terribly confused. As I've said in another PR a couple of days ago, the tests are passing just fine while running the entire test suite, but they fail when running only a subset of the tests, eg. only `tests/streams`. Individual tests are also running just fine.

Just as a brief recap of what the `EventedHLSStreamWriter` class does: it subclasses the `HLSStreamWriter` and waits on each `write()` call for the main thread (which runs the test case) to instruct a write call, so that it can step through the stream write calls individually and make assertions about the HLS stream state. This `await_write()` method on the main thread does have a time limit, in case something goes wrong, so that the test can fail eventually, which means there is a certain time frame where the writer thread needs to make the instructed write call. Same for the main thread which needs to make the `await_write()` call in a certain time frame (the writer thread currently waits indefinitely in this case, but I have fixed this locally).

However, and I absolutely don't understand this, in some cases, the writer thread completely stands still or runs so slowly that it misses some of the time frames and the tests fail, but only when running a subset of all tests. This has nothing to do with thread locks, as this issue occurs when releasing the locks and waking up the writer thread. I have seen this lots of times on my local machine after adding log messages with timestamps all over the code, and I have no explanation for this. It doesn't make sense that there are gaps of several seconds between simple statements or lines of code. I suspect that there is an issue with the way the thread gets woken up.
Take a look at this test branch where I've downgraded requests from 2.26 to 2.25.1:
https://github.com/bastimeyer/streamlink/actions/runs/1056730773
The Windows tests are always passing. I've restarted the CI jobs several times now.
With requests 2.26 and without skipping the tests, at least one of the Windows CI jobs is always failing, every single time.

**This means that there must be an issue with requests-mock and requests 2.26** as we're not doing any real HTTP requests in the tests. And this perfectly makes sense since I've seen random requests-mock responses taking more than 5 seconds, even on my local system, which surely is not right, and this is the cause for the timing window misses in the EventedHLSStreamWriter tests, because either the writer's fetch() method takes too long for some segments to be written or the encryption key takes too long to load to decrypt a segment (or both), which is why the HLS encryption tests fail more often than the other ones.

This further means that even if the tests and test setups get improved and tweaked, there will be no guarantee that this will overall be fixed.

I haven't looked at the code changes of the requests 2.26 release yet, but if one wants to help, please do.

----

We've upgraded requests to `>=2.26.0,<3.0` in #3864 to fix the Windows installer build due to the new dependency tree which needs to be defined explicitly, but what we should actually do is force requests to 2.25.1 on Windows and continue like this until the 2.26 issue with requests-mock is fixed.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -10,7 +10,10 @@
 
 data_files = []
 deps = [
-    "requests>=2.26.0,<3.0",
+    # Temporarily set requests to 2.25.1 on Windows to fix issues with randomly failing tests
+    # Don't force an older requests version on non-Windows systems due to packaging reasons
+    "requests>=2.26.0,<3.0 ; platform_system!='Windows'",
+    "requests==2.25.1      ; platform_system=='Windows'",
     "isodate",
     "websocket-client>=0.58.0",
     # Support for SOCKS proxies


Test patch:
diff --git a/tests/plugins/test_twitch.py b/tests/plugins/test_twitch.py
--- a/tests/plugins/test_twitch.py
+++ b/tests/plugins/test_twitch.py
@@ -5,7 +5,6 @@
 import requests_mock
 
 from streamlink import Streamlink
-from streamlink.compat import is_win32
 from streamlink.plugin import PluginError
 from streamlink.plugins.twitch import Twitch, TwitchHLSStream, TwitchHLSStreamReader, TwitchHLSStreamWriter
 from tests.mixins.stream_hls import EventedHLSStreamWriter, Playlist, Segment as _Segment, Tag, TestMixinStreamHLS
@@ -76,7 +75,6 @@ class _TwitchHLSStream(TwitchHLSStream):
     __reader__ = _TwitchHLSStreamReader
 
 
-@unittest.skipIf(is_win32, "temporarily skip EventedHLSStreamWriter related tests on Windows")
 @patch("streamlink.stream.hls.HLSStreamWorker.wait", MagicMock(return_value=True))
 class TestTwitchHLSStream(TestMixinStreamHLS, unittest.TestCase):
     __stream__ = _TwitchHLSStream
diff --git a/tests/streams/test_hls.py b/tests/streams/test_hls.py
--- a/tests/streams/test_hls.py
+++ b/tests/streams/test_hls.py
@@ -7,7 +7,6 @@
 from Crypto.Cipher import AES
 from Crypto.Util.Padding import pad
 
-from streamlink.compat import is_win32
 from streamlink.session import Streamlink
 from streamlink.stream import hls
 from tests.mixins.stream_hls import EventedHLSStreamWriter, Playlist, Segment, Tag, TestMixinStreamHLS
@@ -145,7 +144,6 @@ def test_map(self):
         self.assertTrue(self.called(map2, once=True), "Downloads second map only once")
 
 
-@unittest.skipIf(is_win32, "temporarily skip EventedHLSStreamWriter related tests on Windows")
 @patch("streamlink.stream.hls.HLSStreamWorker.wait", Mock(return_value=True))
 class TestHLSStreamEncrypted(TestMixinStreamHLS, unittest.TestCase):
     __stream__ = EventedHLSStream
diff --git a/tests/streams/test_hls_filtered.py b/tests/streams/test_hls_filtered.py
--- a/tests/streams/test_hls_filtered.py
+++ b/tests/streams/test_hls_filtered.py
@@ -2,7 +2,6 @@
 from threading import Event
 from unittest.mock import MagicMock, call, patch
 
-from streamlink.compat import is_win32
 from streamlink.stream.hls import HLSStream, HLSStreamReader
 from tests.mixins.stream_hls import EventedHLSStreamWriter, Playlist, Segment, TestMixinStreamHLS
 
@@ -24,7 +23,6 @@ class _TestSubjectHLSStream(HLSStream):
     __reader__ = _TestSubjectHLSReader
 
 
-@unittest.skipIf(is_win32, "temporarily skip EventedHLSStreamWriter related tests on Windows")
 @patch("streamlink.stream.hls.HLSStreamWorker.wait", MagicMock(return_value=True))
 class TestFilteredHLSStream(TestMixinStreamHLS, unittest.TestCase):
     __stream__ = _TestSubjectHLSStream