Repository: scipy/scipy
Pull request: ENH: io.wavfile: read unseekable files
#### Reference issue
Closes gh-11328

#### What does this implement/fix?
Allow scipy.io.wavfile.read() to read non-seekable files.

#### Additional information
Allow scipy.io.wavfile.read() to read non-seekable files.

  * If passed stream is not seekable, wrap it in a SeekEmulatingReader, which tracks current position and implements tell() to report it.

Most seek()s in wavfile.read() simply seek forward from the current position by a fixed amount. To address these:

  * Support limited SeekEmulatingReader.seek() that emulates seek()s that can be supported by read()ing and discarding the result.

Befor returning, read() attempts to seek back to the start of the file. This is nice when possible, but if it isn't we shouldn't blow up. So:

  * Avoid final fid.seek(0) if underlying stream isn't seekable.

The remaining seek()s are in RF64 support, where chunk size is global and stored in the RIFF header rather than in the chunk header. _read_data_chunk() currently seeks back to a fixed offset from the file origin to grab the chunk size. To address this:

  * Modify RF64 support to parse and store the chunk size in _read_riff_chunk() so that _read_data_chunk() doesn't have to seek() back for it. This applies whether or not reading from a seekable stream.

To test this, I added test_streams(), which ensures that wavfile reads the same data no matter whether a path, seekable stream, or unseekable stream is passed to read().


Problem statement:
Scipy unable to read piped wav file
Scipy version 1.4.1

Repro:
```
import subprocess
import scipy
sp = subprocess.Popen(
    ['ffmpeg', '-i', 'test.mp3', '-f', 'wav', '-ac', '1', '-ar', '22050', '-'],
    stdout=subprocess.PIPE)
scipy.io.wavfile.read(sp.stdout)
```

Gives:
```
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/scipy/io/wavfile.py", line 273, in read
    while fid.tell() < file_size:
OSError: [Errno 29] Illegal seek
```

Hints from issue discussion:
I'm not an expert on this functionality---when was the last SciPy version when this worked for you? Or are you saying this has never worked?
I should have specified -- this is the first time I'm trying this and so I wouldn't be surprised if it has never worked with scipy. So I suppose this is a feature request.

I would note that the [wave](https://docs.python.org/3/library/wave.html) module is able to read non-seekable inputs fine.

Commit message: ENH: read unseekable files in scipy.io.wavfile.

Allow scipy.io.wavfile.read() to read non-seekable files.

  * If passed stream is not seekable, wrap it in a SeekEmulatingReader,
    which tracks current position and implements tell() to report it.

Most seek()s in wavfile.read() simply seek forward from the current
position by a fixed amount. To address these:

  * Support limited SeekEmulatingReader.seek() that emulates seek()s
    that can be supported by read()ing and discarding the result.

Befor returning, read() attempts to seek back to the start of the file.
This is nice when possible, but if it isn't we shouldn't blow up. So:

  * Avoid final fid.seek(0) if underlying stream isn't seekable.

The remaining seek()s are in RF64 support, where chunk size is global
and stored in the RIFF header rather than in the chunk header.
_read_data_chunk() currently seeks back to a fixed offset from the file
origin to grab the chunk size. To address this:

  * Modify RF64 support to parse and store the chunk size in
    _read_riff_chunk() so that _read_data_chunk() doesn't have to seek()
    back for it. This applies whether or not reading from a seekable
    stream.

To test this, I added test_streams(), which ensures that wavfile reads
the same data no matter whether a path, seekable stream, or unseekable
stream is passed to read().

Fixes #11328

Patch to review:
diff --git a/scipy/io/tests/test_wavfile.py b/scipy/io/tests/test_wavfile.py
index 8e0a545495a8..54ad33311467 100644
--- a/scipy/io/tests/test_wavfile.py
+++ b/scipy/io/tests/test_wavfile.py
@@ -334,6 +334,47 @@ def test_write_roundtrip_rf64(tmpdir):
     data2[0] = 0
 
 
+# Fake a non-seekable file-like object without resorting to subprocesses.
+class Nonseekable:
+    def __init__(self, fp):
+        self.fp = fp
+
+    def seekable(self):
+        return False
+    
+    def read(self, size=-1, /):
+        return self.fp.read(size)
+    
+    def close(self):
+        self.fp.close()
+
+
+def test_streams():
+    for filename in ['test-44100Hz-le-1ch-4bytes.wav',
+                     'test-8000Hz-le-2ch-1byteu.wav',
+                     'test-44100Hz-2ch-32bit-float-le.wav',
+                     'test-44100Hz-2ch-32bit-float-be.wav',
+                     'test-8000Hz-le-5ch-9S-5bit.wav',
+                     'test-8000Hz-le-4ch-9S-12bit.wav',
+                     'test-8000Hz-le-3ch-5S-24bit.wav',
+                     'test-1234Hz-le-1ch-10S-20bit-extra.wav',
+                     'test-8000Hz-le-3ch-5S-36bit.wav',
+                     'test-8000Hz-le-3ch-5S-45bit.wav',
+                     'test-8000Hz-le-3ch-5S-53bit.wav',
+                     'test-8000Hz-le-3ch-5S-64bit.wav',
+                     'test-44100Hz-be-1ch-4bytes.wav', # RIFX
+                     'test-44100Hz-le-1ch-4bytes-rf64.wav']:
+        dfname = datafile(filename)
+        with open(dfname, 'rb') as fp1, open(dfname, 'rb') as fp2:
+            rate1, data1 = wavfile.read(fp1)
+            rate2, data2 = wavfile.read(Nonseekable(fp2))
+            rate3, data3 = wavfile.read(dfname, mmap=False)
+            assert_array_equal(rate1, rate3)
+            assert_array_equal(rate2, rate3)
+            assert_array_equal(data1, data3)
+            assert_array_equal(data2, data3)
+
+
 def test_read_unknown_filetype_fail():
     # Not an RIFF
     for mmap in [False, True]:
diff --git a/scipy/io/wavfile.py b/scipy/io/wavfile.py
index b6978a1c461c..2e0e4481bd85 100644
--- a/scipy/io/wavfile.py
+++ b/scipy/io/wavfile.py
@@ -9,6 +9,7 @@
 
 """
 import io
+import os
 import sys
 import numpy as np
 import struct
@@ -27,6 +28,47 @@ class WavFileWarning(UserWarning):
     pass
 
 
+class SeekEmulatingReader:
+    """
+    Tracks stream position, provides tell(), and emulates only those
+    seeks that can be supported by reading forward. Other seeks raise
+    io.UnsupportedOperation. Note that this class implements only the
+    minimum necessary to keep wavfile.read() happy.
+    """
+    def __init__(self, reader):
+        self.reader = reader
+        self.pos = 0
+
+    def read(self, size=-1, /):
+        data = self.reader.read(size)
+        self.pos += len(data)
+        return data
+    
+    def seek(self, offset, whence=os.SEEK_SET, /):
+        match whence:
+            case os.SEEK_SET if offset >= self.pos:
+                self.read(offset - self.pos) # convert to relative
+            case os.SEEK_CUR if offset >= 0:
+                self.read(offset) # advance by offset
+            case os.SEEK_END if offset == 0:
+                self.read() # advance to end of stream
+            case _:
+                raise io.UnsupportedOperation("SeekEmulatingReader was asked to emulate"
+                                              " a seek operation it does not support.")
+        return self.pos
+    
+    def tell(self):
+        return self.pos
+    
+    def close(self):
+        self.reader.close()
+    
+    # np.fromfile expects to be able to call flush(), and _read_data_chunk
+    # expects to catch io.UnsupportedOperation if np.fromfile fails.
+    def flush(self):
+        raise io.UnsupportedOperation("SeekEmulatingReader can't flush.")
+
+
 class WAVE_FORMAT(IntEnum):
     """
     WAVE form wFormatTag IDs
@@ -397,7 +439,7 @@ def _read_fmt_chunk(fid, is_big_endian):
 
 
 def _read_data_chunk(fid, format_tag, channels, bit_depth, is_big_endian, is_rf64,
-                     block_align, mmap=False):
+                     block_align, mmap=False, rf64_chunk_size=None):
     """
     Notes
     -----
@@ -431,11 +473,8 @@ def _read_data_chunk(fid, format_tag, channels, bit_depth, is_big_endian, is_rf6
     if not is_rf64:
         size = struct.unpack(fmt+'I', fid.read(4))[0]
     else:
-        pos = fid.tell()
         # chunk size is stored in global file header for RF64
-        fid.seek(28)
-        size = struct.unpack('<Q', fid.read(8))[0]
-        fid.seek(pos)
+        size = rf64_chunk_size
         # skip data chunk size as it is 0xFFFFFFF
         fid.read(4)
 
@@ -538,6 +577,7 @@ def _read_riff_chunk(fid):
     # Size of entire file
     if not is_rf64:
         file_size = struct.unpack(fmt, fid.read(4))[0] + 8
+        rf64_chunk_size = None
         str2 = fid.read(4)
     else:
         # Skip 0xFFFFFFFF (-1) bytes
@@ -548,14 +588,15 @@ def _read_riff_chunk(fid):
             raise ValueError("Invalid RF64 file: ds64 chunk not found.")
         ds64_size = struct.unpack("<I", fid.read(4))[0]
         file_size = struct.unpack(fmt, fid.read(8))[0] + 8
+        rf64_chunk_size = struct.unpack('<Q', fid.read(8))[0]
         # Ignore additional attributes of ds64 chunk like sample count, tables, etc.
         # and just skip to the next chunk
-        fid.seek(ds64_size - 8, 1)
+        fid.seek(ds64_size - 16, 1)
 
     if str2 != b'WAVE':
         raise ValueError(f"Not a WAV file. RIFF form type is {repr(str2)}.")
 
-    return file_size, is_big_endian, is_rf64
+    return file_size, is_big_endian, is_rf64, rf64_chunk_size
 
 
 def _handle_pad_byte(fid, size):
@@ -672,9 +713,12 @@ def read(filename, mmap=False):
         mmap = False
     else:
         fid = open(filename, 'rb')
+    
+    if not (was_seekable := fid.seekable()):
+        fid = SeekEmulatingReader(fid)
 
     try:
-        file_size, is_big_endian, is_rf64 = _read_riff_chunk(fid)
+        file_size, is_big_endian, is_rf64, rf64_chunk_size = _read_riff_chunk(fid)
         fmt_chunk_received = False
         data_chunk_received = False
         while fid.tell() < file_size:
@@ -713,7 +757,8 @@ def read(filename, mmap=False):
                 if not fmt_chunk_received:
                     raise ValueError("No fmt chunk before data")
                 data = _read_data_chunk(fid, format_tag, channels, bit_depth,
-                                        is_big_endian, is_rf64, block_align, mmap)
+                                        is_big_endian, is_rf64, block_align,
+                                        mmap, rf64_chunk_size)
             elif chunk_id == b'LIST':
                 # Someday this could be handled properly but for now skip it
                 _skip_unknown_chunk(fid, is_big_endian)
@@ -727,7 +772,9 @@ def read(filename, mmap=False):
     finally:
         if not hasattr(filename, 'read'):
             fid.close()
-        else:
+        elif was_seekable:
+            # Rewind, if we are able, so that caller can do something
+            # else with the raw WAV stream.
             fid.seek(0)
 
     return fs, data


Code review comments:
diff_hunk: @@ -334,6 +334,47 @@ def test_write_roundtrip_rf64(tmpdir):
     data2[0] = 0
 
 
+# Fake a non-seekable file-like object without resorting to subprocesses.
+class Nonseekable:
+    def __init__(self, fp):
+        self.fp = fp
+
+    def seekable(self):
+        return False
+    
+    def read(self, size=-1, /):
+        return self.fp.read(size)
+    
+    def close(self):
+        self.fp.close()
+
+
+def test_streams():
+    for filename in ['test-44100Hz-le-1ch-4bytes.wav',
+                     'test-8000Hz-le-2ch-1byteu.wav',
+                     'test-44100Hz-2ch-32bit-float-le.wav',
+                     'test-44100Hz-2ch-32bit-float-be.wav',
+                     'test-8000Hz-le-5ch-9S-5bit.wav',
+                     'test-8000Hz-le-4ch-9S-12bit.wav',
+                     'test-8000Hz-le-3ch-5S-24bit.wav',
+                     'test-1234Hz-le-1ch-10S-20bit-extra.wav',
+                     'test-8000Hz-le-3ch-5S-36bit.wav',
+                     'test-8000Hz-le-3ch-5S-45bit.wav',
+                     'test-8000Hz-le-3ch-5S-53bit.wav',
+                     'test-8000Hz-le-3ch-5S-64bit.wav',
+                     'test-44100Hz-be-1ch-4bytes.wav', # RIFX
+                     'test-44100Hz-le-1ch-4bytes-rf64.wav']:
+        dfname = datafile(filename)
+        with open(dfname, 'rb') as fp1, open(dfname, 'rb') as fp2:
+            rate1, data1 = wavfile.read(fp1)
+            rate2, data2 = wavfile.read(Nonseekable(fp2))
+            rate3, data3 = wavfile.read(dfname, mmap=False)
+            assert_array_equal(rate1, rate3)
+            assert_array_equal(rate2, rate3)
+            assert_array_equal(data1, data3)
+            assert_array_equal(data2, data3)
path: scipy/io/tests/test_wavfile.py
text: @user1:
```suggestion
            assert_equal(rate1, rate3)
            assert_equal(rate2, rate3)
            assert_equal(data1, data3)
            assert_equal(data2, data3)
```