Repository: streamlink/streamlink

Issue:
stream.ffmpegmux: race condition when copying data to named pipe
### Checklist

- [X] This is a bug report and not a different kind of issue
- [X] [I have read the contribution guidelines](https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#contributing-to-streamlink)
- [X] [I have checked the list of open and recently closed bug reports](https://github.com/streamlink/streamlink/issues?q=is%3Aissue+label%3A%22bug%22)
- [X] [I have checked the commit log of the master branch](https://github.com/streamlink/streamlink/commits/master)

### Streamlink version

Latest build from the master branch

### Description

Opening this issue myself, because the person who messaged me doesn't want to create a GitHub account...

After having a look at an [example DASH stream from Vimeo that was sent to me](https://vimeo.com/channels/staffpicks/790905324), there appears to be a race condition within `FFMPEGMuxer.copy_to_pipe()`. This is where the stream data of each substream gets written to a named pipe where the FFmpeg process then reads from and muxes the streams. The read-write loop ends as soon as the stream's `closed` attribute is set (by its stream implementation in another thread):
https://github.com/streamlink/streamlink/blame/a0a5af60b0e99f3ee7987b1caad12497884a4062/src/streamlink/stream/ffmpegmux.py#L135

While rewriting the CLI's `StreamRunner` recently, I initially had the same condition in its read-write loop and then figured that this would be a bad idea because the buffer sometimes wouldn't get emptied at the end of the stream when all segments have finished downloading for example. This seems to be the case here, too.
https://github.com/streamlink/streamlink/blame/a0a5af60b0e99f3ee7987b1caad12497884a4062/src/streamlink_cli/streamrunner.py#L126

The following diff should work, but I will have to take a closer look first. Opening this issue thread as a reminder to check later in more detail.

```diff
diff --git a/src/streamlink/stream/ffmpegmux.py b/src/streamlink/stream/ffmpegmux.py
index 29a928aa..f48a7668 100644
--- a/src/streamlink/stream/ffmpegmux.py
+++ b/src/streamlink/stream/ffmpegmux.py
@@ -132,14 +132,16 @@ class FFMPEGMuxer(StreamIO):
     def copy_to_pipe(stream: StreamIO, pipe: NamedPipeBase):
         log.debug(f"Starting copy to pipe: {pipe.path}")
         pipe.open()
-        while not stream.closed:
+        while True:
             try:
                 data = stream.read(8192)
-                if len(data):
-                    pipe.write(data)
-                else:
+                if data == b"":
                     break
             except (OSError, ValueError):
+                break
+            try:
+                pipe.write(data)
+            except OSError:
                 log.error(f"Pipe copy aborted: {pipe.path}")
                 break
         try:
```

### Debug log

```text
$ streamlink -l debug -o /tmp/foo.mkv 'https://vimeo.com/channels/staffpicks/790905324' best
...
[stream.dash][debug] Download of segment: 0c433f4a.mp4 complete
[stream.dash][debug] Download of segment: 9ae75d32.mp4 complete
...
[stream.dash][debug] Download of segment: 0c433f4a.mp4 complete
[stream.dash][debug] Download of segment: 9ae75d32.mp4 complete
[stream.segmented][debug] Closing writer thread
[stream.ffmpegmux][debug] Pipe copy complete: /tmp/streamlinkpipe-812266-2-8643
[stream.dash][debug] Download of segment: 0c433f4a.mp4 complete
[stream.dash][debug] Download of segment: 0c433f4a.mp4 complete
...
[stream.dash][debug] Download of segment: 0c433f4a.mp4 complete
[stream.dash][debug] Download of segment: 0c433f4a.mp4 complete
[stream.segmented][debug] Closing writer thread
[stream.ffmpegmux][debug] Pipe copy complete: /tmp/streamlinkpipe-812266-1-2639
[stream.ffmpegmux][debug] Closing ffmpeg thread
[stream.ffmpegmux][debug] Closed all the substreams
[cli][info] Stream ended
[cli][info] Closing currently open stream...

$ ffmpeg -i /tmp/foo.mkv -c copy /tmp/foo.mp4
...

$ ffprobe -v error -of json -show_streams /tmp/foo.mp4 | jq -r .streams[].duration
906.832313
725.887667
```



Discussion:
The proposed patch should be good enough. I will have another look at this tomorrow, so this specific issue can get fixed at least.

After that, I'll give `FFMPEGMuxer` a proper rewrite, because it's really messy, has lots of other issues, and it doesn't have any tests apart from the is-usable stuff and options setup. Related: #4998

Resolving patch:
diff --git a/src/streamlink/stream/ffmpegmux.py b/src/streamlink/stream/ffmpegmux.py
--- a/src/streamlink/stream/ffmpegmux.py
+++ b/src/streamlink/stream/ffmpegmux.py
@@ -4,6 +4,7 @@
 import subprocess
 import sys
 import threading
+from contextlib import suppress
 from functools import lru_cache
 from pathlib import Path
 from shutil import which
@@ -131,22 +132,28 @@ def _resolve_command(cls, command: Optional[str] = None, validate: bool = True)
     @staticmethod
     def copy_to_pipe(stream: StreamIO, pipe: NamedPipeBase):
         log.debug(f"Starting copy to pipe: {pipe.path}")
+        # TODO: catch OSError when creating/opening pipe fails and close entire output stream
         pipe.open()
-        while not stream.closed:
+
+        while True:
             try:
                 data = stream.read(8192)
-                if len(data):
-                    pipe.write(data)
-                else:
-                    break
-            except (OSError, ValueError):
-                log.error(f"Pipe copy aborted: {pipe.path}")
+            except (OSError, ValueError) as err:
+                log.error(f"Error while reading from substream: {err}")
+                break
+
+            if data == b"":
+                log.debug(f"Pipe copy complete: {pipe.path}")
+                break
+
+            try:
+                pipe.write(data)
+            except OSError as err:
+                log.error(f"Error while writing to pipe {pipe.path}: {err}")
                 break
-        try:
+
+        with suppress(OSError):
             pipe.close()
-        except OSError:  # might fail closing, but that should be ok for the pipe
-            pass
-        log.debug(f"Pipe copy complete: {pipe.path}")
 
     def __init__(self, session, *streams, **options):
         if not self.is_usable(session):
@@ -223,17 +230,25 @@ def close(self):
             self.process.kill()
             self.process.stdout.close()
 
-            # close the streams
             executor = concurrent.futures.ThreadPoolExecutor()
+
+            # close the substreams
             futures = [
                 executor.submit(stream.close)
                 for stream in self.streams
                 if hasattr(stream, "close") and callable(stream.close)
             ]
-
             concurrent.futures.wait(futures, return_when=concurrent.futures.ALL_COMPLETED)
             log.debug("Closed all the substreams")
 
+            # wait for substream copy-to-pipe threads to terminate and clean up the opened pipes
+            timeout = self.session.options.get("stream-timeout")
+            futures = [
+                executor.submit(thread.join, timeout=timeout)
+                for thread in self.pipe_threads
+            ]
+            concurrent.futures.wait(futures, return_when=concurrent.futures.ALL_COMPLETED)
+
         if self.close_errorlog:
             self.errorlog.close()
             self.errorlog = None
diff --git a/src/streamlink/utils/named_pipe.py b/src/streamlink/utils/named_pipe.py
--- a/src/streamlink/utils/named_pipe.py
+++ b/src/streamlink/utils/named_pipe.py
@@ -4,6 +4,7 @@
 import random
 import tempfile
 import threading
+from contextlib import suppress
 from pathlib import Path
 
 from streamlink.compat import is_win32
@@ -65,11 +66,15 @@ def write(self, data):
         return self.fifo.write(data)
 
     def close(self):
-        if self.fifo:
-            self.fifo.close()
+        try:
+            if self.fifo is not None:
+                self.fifo.close()
+        except OSError:
+            raise
+        finally:
+            with suppress(OSError):
+                self.path.unlink()
             self.fifo = None
-        if self.path.is_fifo():
-            os.unlink(self.path)
 
 
 class NamedPipeWindows(NamedPipeBase):
@@ -118,9 +123,13 @@ def write(self, data):
         return written.value
 
     def close(self):
-        if self.pipe is not None:
-            windll.kernel32.DisconnectNamedPipe(self.pipe)
-            windll.kernel32.CloseHandle(self.pipe)
+        try:
+            if self.pipe is not None:
+                windll.kernel32.DisconnectNamedPipe(self.pipe)
+                windll.kernel32.CloseHandle(self.pipe)
+        except OSError:
+            raise
+        finally:
             self.pipe = None
 
 


Test patch: