Repository: h5py/h5py
Pull request #2517 (2024-10-09 12:26:19)

Add File.in_memory() alternative constructor
From discussion on #2514, this adds a high-level way of working with 'files' in memory.

```python
# Start a new HDF5 file in memory
f = h5py.File.in_memory()
f['a'] = [1, 2, 3]

# Get the file data as bytes, e.g. to send over the network
f.flush()
hdf_data = f.id.get_file_image()

# Turn the bytes back into an h5py File object
f2 = h5py.File.in_memory(hdf_data)
```

Questions:

1. Looking at [the code in the core driver](https://github.com/HDFGroup/hdf5/blob/bcc795fcf66f5ce72b76dab4b4df5136f2558303/src/H5FDcore.c#L752-L754), there's an annoying check that the given filename doesn't exist on the filesystem. We bypass this when creating a new 'file', but I can't see a way to avoid it when we're passing in a file image.

```c
    if ((file_image_info.buffer != NULL) && !(H5F_ACC_CREAT & flags)) {
        if ((fd = HDopen(name, o_flags, H5_POSIX_CREATE_MODE_RW)) >= 0) {
            HDclose(fd);
            HGOTO_ERROR(H5E_FILE, H5E_FILEEXISTS, NULL, "file already exists");
        }
```

For now I just use a filename that's unlikely to exist - `h5py_in_memory_nonfile_<N>`. We could create a temporary directory to almost guarantee that we pass a filename that doesn't exist. But the aim is to avoid touching the filesystem, so that doesn't seeem ideal.

2. I found after writing this that there's an `H5LTopen_file_image()` function in the high-level library, exposed in h5py's low-level API as `h5py.h5f.open_file_image()`. This has options to avoid unnecessary copying of the file data, but these options are not yet exposed, and I didn't fancy playing with reference counts to ensure the image data is freed at the right time. Without that option, it looks like it does essentially the same as this code.
3. I haven't included a read-only option; in-memory files are always writable. It wouldn't be hard to add read-only mode if we want to.


Patch:
diff --git a/.github/workflows/build_wheels.yml b/.github/workflows/build_wheels.yml
index 72699246..9fdbaa7b 100644
--- a/.github/workflows/build_wheels.yml
+++ b/.github/workflows/build_wheels.yml
@@ -56,7 +56,7 @@ jobs:
           - "3.10"
           - "3.11"
           - "3.12"
-          - "3.13"
+          - "3.13-dev"
         include:
           - os: ubuntu-latest
             arch: x86_64
diff --git a/ci/triage_build.sh b/ci/triage_build.sh
index a5b97cdb..1a839c46 100755
--- a/ci/triage_build.sh
+++ b/ci/triage_build.sh
@@ -23,7 +23,7 @@ if [[ "$ARCH" == "aarch64" ]]; then
     echo "TOX_TEST_LIMITED=1" | tee -a $GITHUB_ENV
     if [[ "$GITHUB_EVENT_NAME" == "pull_request" ]] && [[ "$MSG" != *'[aarch64]'* ]]; then
         echo "Building limited $KIND wheels for PR with commit message: $MSG"
-        if [[ "$PYTHON" != "3.13" ]]; then
+        if [[ "$PYTHON" != "3.12" ]]; then
             echo "skip=1" >> $GITHUB_OUTPUT
         fi
         REPORTED=1
diff --git a/docs/high/file.rst b/docs/high/file.rst
index a9f3c266..df7cbda4 100644
--- a/docs/high/file.rst
+++ b/docs/high/file.rst
@@ -126,6 +126,30 @@ of supported drivers and their options:
            with ros3. Alternatively, use the :ref:`file-like object
            <file_fileobj>` support with a package like s3fs.
 
+.. _file_in_memory:
+
+In-memory 'files'
+-----------------
+
+HDF5 can make a file in memory, without reading or writing a real file.
+To do this with h5py, use the :meth:`.File.in_memory` class method::
+
+    # Start a new HDF5 file in memory
+    f = h5py.File.in_memory()
+    f['a'] = [1, 2, 3]
+
+    # Get the file data as bytes, e.g. to send over the network
+    f.flush()
+    hdf_data = f.id.get_file_image()
+
+    # Turn the bytes back into an h5py File object
+    f2 = h5py.File.in_memory(hdf_data)
+
+This uses HDF5's "core" :ref:`file driver <file_driver>`, which is likely to
+cause fewer odd problems than asking HDF5 to call back into a Python ``BytesIO``
+object (:ref:`described below <file_fileobj>`).
+
+.. versionadded:: 3.13
 
 .. _file_fileobj:
 
@@ -506,6 +530,15 @@ Reference
             new metadata block allocations. See :ref:`file_meta_block_size`.
     :param kwds:    Driver-specific keywords; see :ref:`file_driver`.
 
+    .. classmethod:: in_memory(file_image=None, block_size=64*1024, **kwargs)
+
+        :param file_image: The initial file contents as bytes (or anything that
+            supports the Python buffer interface). HDF5 takes a copy of this data.
+        :param block_size: Chunk size for new memory alloactions (default 64 KiB).
+
+        Other keyword arguments are like :class:`File`, although name, mode,
+        driver and locking can't be passed.
+
     .. method:: __bool__()
 
         Check that the file descriptor is valid and the file open:
diff --git a/h5py/_hl/files.py b/h5py/_hl/files.py
index 1a6d5f81..cad6e74e 100644
--- a/h5py/_hl/files.py
+++ b/h5py/_hl/files.py
@@ -11,8 +11,9 @@
     Implements high-level support for HDF5 file objects.
 """
 
-import sys
+import inspect
 import os
+import sys
 from warnings import warn
 
 from .compat import filename_decode, filename_encode
@@ -108,10 +109,12 @@ def registered_drivers():
     return frozenset(_drivers)
 
 
-def make_fapl(driver, libver, rdcc_nslots, rdcc_nbytes, rdcc_w0, locking,
-              page_buf_size, min_meta_keep, min_raw_keep,
-              alignment_threshold, alignment_interval, meta_block_size,
-              **kwds):
+def make_fapl(
+    driver, libver=None, rdcc_nslots=None, rdcc_nbytes=None, rdcc_w0=None,
+    locking=None, page_buf_size=None, min_meta_keep=0, min_raw_keep=0,
+    alignment_threshold=1, alignment_interval=1, meta_block_size=None,
+    **kwds
+):
     """ Set up a file access property list """
     plist = h5p.create(h5p.FILE_ACCESS)
 
@@ -567,6 +570,53 @@ class File(Group):
 
         super().__init__(fid)
 
+    _in_memory_file_counter = 0
+
+    @classmethod
+    @with_phil
+    def in_memory(cls, file_image=None, **kwargs):
+        """Create an HDF5 file in memory, without an underlying file
+
+        file_image
+            The initial file contents as bytes (or anything that supports the
+            Python buffer interface). HDF5 takes a copy of this data.
+        block_size
+            Chunk size for new memory alloactions (default 64 KiB).
+
+        Other keyword arguments are like File(), although name, mode,
+        driver and locking can't be passed.
+        """
+        for k in ('driver', 'locking', 'backing_store'):
+            if k in kwargs:
+                raise TypeError(
+                    f"File.in_memory() got an unexpected keyword argument {k!r}"
+                )
+        fcpl_kwargs = {}
+        for k in inspect.signature(make_fcpl).parameters:
+            if k in kwargs:
+                fcpl_kwargs[k] = kwargs.pop(k)
+        fcpl = make_fcpl(**fcpl_kwargs)
+
+        fapl = make_fapl(driver="core", backing_store=False, **kwargs)
+        if file_image:
+            if fcpl_kwargs:
+                kw = ', '.join(fcpl_kwargs)
+                raise TypeError(f"{kw} parameters cannot be used with file_image")
+            fapl.set_file_image(file_image)
+
+        # We have to give HDF5 a filename, but it should never use it.
+        # This is a hint both in memory, and in case a bug ever creates a file.
+        # The name also needs to be different from any other open file;
+        # we use a simple counter (protected by the 'phil' lock) for this.
+        name = b"h5py_in_memory_nonfile_%d"  % cls._in_memory_file_counter
+        cls._in_memory_file_counter += 1
+
+        if file_image:
+            fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)
+        else:
+            fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl)
+        return cls(fid)
+
     def close(self):
         """ Close the file.  All open objects become invalid """
         with phil:
diff --git a/h5py/h5p.pyx b/h5py/h5p.pyx
index 58134187..5fb8ec8a 100644
--- a/h5py/h5p.pyx
+++ b/h5py/h5p.pyx
@@ -1017,8 +1017,8 @@ cdef class PropFAID(PropInstanceID):
 
         Use the h5fd.CORE (memory-resident) file driver.
 
-        increment
-            Chunk size for new memory requests (default 1 meg)
+        block_size
+            Chunk size for new memory requests (default 64 KiB)
 
         backing_store
             If True (default), memory contents are associated with an
diff --git a/news/in-memory.rst b/news/in-memory.rst
new file mode 100644
index 00000000..b978a74b
--- /dev/null
+++ b/news/in-memory.rst
@@ -0,0 +1,30 @@
+New features
+------------
+
+* New :meth:`.File.in_memory` constructor to conveniently build an HDF5 file
+  structure in memory.
+
+Deprecations
+------------
+
+* <news item>
+
+Exposing HDF5 functions
+-----------------------
+
+* <news item>
+
+Bug fixes
+---------
+
+* <news item>
+
+Building h5py
+-------------
+
+* <news item>
+
+Development
+-----------
+
+* <news item>


Test patch:
diff --git a/h5py/tests/test_file_image.py b/h5py/tests/test_file_image.py
index 2ffe77b3..d36731bf 100644
--- a/h5py/tests/test_file_image.py
+++ b/h5py/tests/test_file_image.py
@@ -1,3 +1,5 @@
+import numpy as np
+
 import h5py
 from h5py import h5f, h5p
 
@@ -33,3 +35,20 @@ class TestFileImage(TestCase):
         f = h5py.File(fid)
 
         self.assertTrue('test' in f)
+
+
+def test_in_memory():
+    arr = np.arange(10)
+    # Passing one fcpl & one fapl parameter to exercise the code splitting them:
+    with h5py.File.in_memory(track_order=True, rdcc_nbytes=2_000_000) as f1:
+        f1['a'] = arr
+        f1.flush()
+        img = f1.id.get_file_image()
+
+        # Open while f1 is still open
+        with h5py.File.in_memory(img) as f2:
+            np.testing.assert_array_equal(f2['a'][:], arr)
+
+    # Reuse image now that previous files are closed
+    with h5py.File.in_memory(img) as f3:
+        np.testing.assert_array_equal(f3['a'][:], arr)