Repository: pyodide/pyodide

Issue:
`OSError: [Errno 9] Bad file descriptor` when trying to load `.npy` files, works with `.npz` file format
## 🐛 Bug

NumPy is unable to load arrays from `.npy` binaries, but it can read from compressed `.npz` archives.

### To Reproduce

I noticed this error when compiling PyWavelets (`pywt`) from source via the Emscripten toolchain.

In an activated virtual environment created by Pyodide, run the following:

```bash
git clone https://github.com/PyWavelets/pywt.git
pip install .
```

and then

```python
import pywt
import numpy as np

aero = pywt.data.aero()

ref = np.array([[178, 178, 179],
                    [170, 173, 171],
                    [185, 174, 171]])

np.testing.assert_allclose(aero[:3, :3], ref)
```

should fail. However, [after converting](https://github.com/PyWavelets/pywt/pull/701/files#diff-86b5b5c7cbe8cc8368f6991c914b7263019507351ce567543cbf2b627b91aa57) these `.npy` files to `.npz`, NumPy can safely load the arrays from the files as requested.

Here is an example of conversion from `.npy` to `.npz`:

```python
import numpy as np
ecg = np.load("pywt/data/ecg.npy")
np.savez(file="ecg.npz", data=ecg)
```

after which `ecg.npz` can be loaded as follows:

```python
import numpy as np
loaded = np.load("ecg.npz")
print(loaded["data"])
```

### Expected behavior

The Pyodide environment should be able to load the `.npy` file format stored in a directory, but [fails with multiple `OSError`s](https://github.com/agriyakhetarpal/pywt/actions/runs/7993252511/job/21828629911), possibly due to the lack of a server for filesystem access as the Pyodide documentation mentions – but this doesn't explain why `.npz` files work?

The expected behaviour should be that all file formats work.

### Environment

- Pyodide Version<!-- (e.g. 1.8.1) -->: `pyodide-build` version 0.25.0
- Browser version<!-- (e.g. Chrome 95.0.4638.54) -->:  N/A
- Any other relevant information:

<!-- If you are building Pyodide by yourself, please also include these information: -->

<!--
- Commit hash of Pyodide git repository:
- Build environment<!--(e.g. Ubuntu 18.04, pyodide/pyodide-env:19 docker)- ->:
-->

### Additional context

xref: https://github.com/PyWavelets/pywt/pull/701



Discussion:
Thanks for the report @agriyakhetarpal. So far I am not able to reproduce it. What is the traceback you are seeing?

I did the following steps:
```sh
mkdir tmp/pyodide-bug-4541
cd tmp/pyodide-bug-4541
python3.11 -m venv .venv
pip install pyodide-build
pyodide venv .venv-pyodide
git clone https://github.com/PyWavelets/pywt.git --depth 1
pyodide build pywt
.venv-pyodide/bin/pip install pywt/dist/*
```
Then I made a file with the contents you provided:
```py
import pywt
import numpy as np

aero = pywt.data.aero()

ref = np.array([[178, 178, 179],
                    [170, 173, 171],
                    [185, 174, 171]])

np.testing.assert_allclose(aero[:3, :3], ref)
```
Then I ran
```sh
.venv-pyodide/bin/python reproduction.py
```
and this executed the code without error.

> So far I am not able to reproduce it. What is the traceback you are seeing?

Ah, that may be a timing issue - I just merged the workaround (and a Pyodide CI job from @agriyakhetarpal!) at https://github.com/PyWavelets/pywt/pull/701.

With commit `e69b126c096` I think it'll reproduce.
Thanks for verifying, you might have to go back one commit in the `pywt` directory and re-run the `reproduction.py` script, because we had fixed this upstream by converting the `.npy` files to `.npz` before your message here: https://github.com/PyWavelets/pywt/pull/701

Moreover, the above example – it turns out, used a `.npz` file, so that was probably invalid.

Maybe accessing the files can be tried directly instead of the API that `pywt` provides?

```python
import numpy as np
loaded_data = np.load("pywt/data/'ecg.npy')
```

and then running array operations on the `loaded_data` array (it should fail at the time of loading, ideally)?

Edit: ah, @rgommers and I posted at the same time!
Okay, with commit `e69b126c096` and
```py
import numpy as np

loaded_data = np.load("pywt/pywt/data/ecg.npy")
```
I now reproduce the bug:
```py
Traceback (most recent call last):
  File "/tmp/pyodide-bug-4541/repro.py", line 11, in <module>
  File "/tmp/pyodide-bug-4541/.venv-pyodide/lib/python3.11/site-packages/numpy/lib/npyio.py", line 422, in load
  File "/lib/python311.zip/contextlib.py", line 589, in __exit__
  File "/lib/python311.zip/contextlib.py", line 574, in __exit__
OSError: [Errno 8] Bad file descriptor
```

Thanks @agriyakhetarpal and @rgommers!
A slightly simpler reproducer:
```py
from numpy.lib import format

with open("pywt/pywt/data/ecg.npy", "rb") as fid:
    format.read_array(fid)
```
And another layer in:
```py
import numpy as np

with open("pywt/pywt/data/ecg.npy", "rb") as fid:
    np.fromfile(fid, dtype=np.int32, count=1024)
```
And inside of `array_fromfile` we encounter our nemesis, `dup2`:
https://github.com/numpy/numpy/blob/main/numpy/_core/src/multiarray/multiarraymodule.c?plain=1#L2348
Can you tell me @rgommers why it is that the function called `npy_PyFile_Dup2` calls `os.dup` and not `os.dup2`?
https://github.com/numpy/numpy/blob/main/numpy/_core/include/numpy/npy_3kcompat.h?plain=1#L246

:laughing: 
Great question. Git blame tells me that that line of code has not been touched in 15 years. Given the name, it's not unlikely that it was a simple error that didn't matter on other platforms. From your comments I assume you've run into `dup`/`dup2` before? 

Both are documented as "not available for WASI": https://docs.python.org/3/library/os.html#os.dup. No statement on WASM.
From the numpy 1.8.1 release notes:

_"The utility function `npy_PyFile_Dup` and `npy_PyFile_DupClose` are broken by the internal buffering python 3 applies to its file objects. To fix this two new functions `npy_PyFile_Dup2` and `npy_PyFile_DupClose2` are declared in `npy_3kcompat.h` and the old functions are deprecated. Due to the fragile nature of these functions it is recommended to instead use the python API when possible."_

So the `Dup2` here may not have meant the Python-level `dup2`.
> From your comments I assume you've run into dup/dup2 before?

Yeah there have been at least five other bugs involving dup. At this point it's the first thing I expect when I see a bug report like this.
https://github.com/emscripten-core/emscripten/pulls?q=is%3Apr+author%3Ahoodmane+dup

> So the Dup2 here may not have meant the Python-level dup2.

As you say, it seems not.
Anyways I found the bug. Emscripten's filesystem maintains separate file descriptors from the linux host, so it has a mapping from the emscripten file descriptor to the linux file descriptor. In the reproduction, the file is opened and linux assigns it descriptor 27, Emscripten assigns it descriptor 3. When we `dup` the descriptor, Emscripten makes a new descriptor 4, also pointing to linux descriptor 27. Then when we close emscripten descriptor 4 which closes linux descriptor 27 and finally when emscripten descriptor 3 is closed it tries to close linux descriptor 27 again which raises `EBADF: close() on bad file descriptor` which propagates into the OS error you see. We either need to also do a native `dup` on the linux descriptor or reference count it.
Wow, that was a quick fix! Thanks a lot @hoodmane.

This issue can be closed, right?
I'll make a pr backporting the patch and ading a test case and close when that is merged.

Resolving patch:
diff --git a/conftest.py b/conftest.py
--- a/conftest.py
+++ b/conftest.py
@@ -54,6 +54,9 @@
 only_node = pytest.mark.xfail_browsers(
     chrome="node only", firefox="node only", safari="node only"
 )
+only_chrome = pytest.mark.xfail_browsers(
+    node="chrome only", firefox="chrome only", safari="chrome only"
+)
 
 
 def pytest_addoption(parser):


Test patch:
diff --git a/src/tests/test_filesystem.py b/src/tests/test_filesystem.py
--- a/src/tests/test_filesystem.py
+++ b/src/tests/test_filesystem.py
@@ -4,6 +4,9 @@
 """
 
 import pytest
+from pytest_pyodide import run_in_pyodide
+
+from conftest import only_chrome
 
 
 @pytest.mark.skip_refcount_check
@@ -22,7 +25,7 @@ def test_idbfs_persist_code(selenium_standalone):
         f"""
         let mountDir = '{mount_dir}';
         pyodide.FS.mkdir(mountDir);
-        pyodide.FS.mount(pyodide.FS.filesystems.{fstype}, {{root : "."}}, "{mount_dir}");
+        pyodide.FS.mount(pyodide.FS.filesystems.{fstype}, {{root : "."}}, mountDir);
         """
     )
     # create file in mount
@@ -109,9 +112,7 @@ def test_idbfs_persist_code(selenium_standalone):
 
 
 @pytest.mark.requires_dynamic_linking
-@pytest.mark.xfail_browsers(
-    node="Not available", firefox="Not available", safari="Not available"
-)
+@only_chrome
 def test_nativefs_dir(request, selenium_standalone):
     # Note: Using *real* native file system requires
     # user interaction so it is not available in headless mode.
@@ -254,3 +255,78 @@ def test_nativefs_dir(request, selenium_standalone):
         pyodide.FS.unmount("/mnt/nativefs");
         """
     )
+
+
+@pytest.fixture
+def browser(selenium):
+    return selenium.browser
+
+
+@pytest.fixture
+def runner(request):
+    return request.config.option.runner
+
+
+@run_in_pyodide
+def test_fs_dup(selenium, browser):
+    from os import close, dup
+    from pathlib import Path
+
+    from pyodide.code import run_js
+
+    if browser == "node":
+        fstype = "NODEFS"
+    else:
+        fstype = "IDBFS"
+
+    mount_dir = Path("/mount_test")
+    mount_dir.mkdir(exist_ok=True)
+    run_js(
+        """
+        (fstype, mountDir) =>
+            pyodide.FS.mount(pyodide.FS.filesystems[fstype], {root : "."}, mountDir);
+        """
+    )(fstype, str(mount_dir))
+
+    file = open("/mount_test/a.txt", "w")
+    fd2 = dup(file.fileno())
+    close(fd2)
+    file.write("abcd")
+    file.close()
+
+
+@pytest.mark.requires_dynamic_linking
+@only_chrome
+@run_in_pyodide
+async def test_nativefs_dup(selenium, runner):
+    from os import close, dup
+
+    import pytest
+
+    from pyodide.code import run_js
+
+    # Note: Using *real* native file system requires
+    # user interaction so it is not available in headless mode.
+    # So in this test we use OPFS (Origin Private File System)
+    # which is part of File System Access API but uses indexDB as a backend.
+
+    if runner == "playwright":
+        pytest.xfail("Playwright doesn't support file system access APIs")
+
+    await run_js(
+        """
+        async () => {
+            root = await navigator.storage.getDirectory();
+            testFileHandle = await root.getFileHandle('test_read', { create: true });
+            writable = await testFileHandle.createWritable();
+            await writable.write("hello_read");
+            await writable.close();
+            await pyodide.mountNativeFS("/mnt/nativefs", root);
+        }
+        """
+    )()
+    file = open("/mnt/nativefs/test_read")
+    fd2 = dup(file.fileno())
+    close(fd2)
+    assert file.read() == "hello_read"
+    file.close()