Repository: h5py/h5py
Pull request #2557: Native support for Numpy 2's variable-width strings
Author: login: crusaderky
Created: 2025-02-11T18:09:51Z | Merged: 2025-05-23T13:27:20Z

Closes #2551

- [x] main implementation
- [x] performance benchmarks
- [x] no breaking changes (as discussed in #2551)
- [x] don't break compilation on numpy <2.3
- [x] don't break execution on numpy 1
- [x] compile on numpy 2.0
- [x] unit tests
- [x] documentation

# Smoke test
[npystrings_demo.py](https://github.com/crusaderky/h5py/blob/npystrings_demo/npystrings_demo.py) 

# Performance test
[npystrings_bench.py](https://github.com/crusaderky/h5py/blob/npystrings_demo/npystrings_bench.py) 

1_000_000 strings of random width 0~100 characters
(82 MiB dataset) on local NVMe

object dtype (master)
    write: 0.463s
    read : 0.529s

StringDType()
    write: 0.252s
    read : 0.390s

On-disk format is identical between the two.


Linked issue #2551: Numpy 2 variable-width strings
I'm working on a PR to support reading and writing [numpy 2's variable-width strings](https://numpy.org/devdocs/user/basics.strings.html#variable-width-strings) without passing through object dtype.

How should the UI look like?

For `__getitem__`:

1. if numpy 2 is installed and `H5Tget_class(tid) == H5T_STRING and H5Tis_variable_str(tid)` and the meta says it's unicode, the dataset dtype will automatically change to `np.dtypes.StringDType()`. `ds.asstr` becomes a no-op which also returns StringDType.
This is the most natural interface IMHO, but it's also a breaking change.

2. `ds[:]` still returns a numpy array of `byte` objects. `ds.asstr` detects if numpy 2 is installed and automatically returns a StringDType if the encoding is UTF-8. This is also a breaking change.

3. Nothing changes by default. `ds.asstr` gains a new parameter, `dtype=np.dtype(object)`. If the user explicitly sets `np.dtypes.StringDType()` or an alias for it (`dtype="T"`), then the returned array is of the desired type.

4. as (3), but the default is `dtype=None`, which issues a FutureWarning that the returned type is object but in the future will change to StringDType. At a later date, remove the warning and switch the default to StringDType.

5. As (1), but with some sort of opt-in global flag that is later flipped to opt-out.

For `create_dataset`:
1. `h5py.string_dtype()` continues returning `np.dtype('O', metadata={'vlen': str})`. When someone calls `__setitem__` on such an array, there is magic happening that bypasses the conversion through object type if the parameter's type is StringDType.

2. `h5py.string_dtype()` returns StringDType. This would be cleaner but a breaking change.

3. As (1), but with some sort of opt-in global flag that is later flipped to opt-out.

Linked issue comments: user: login: crusaderky
created_at: 2025-02-05T11:22:43Z
body: IMHO the cleanest option is to add a opt-in global setting with a FutureWarning, which will later turn to opt-out, which after support for Numpy 1 is dropped will become the only option available.
author_association: CONTRIBUTOR
user: login: takluyver
created_at: 2025-02-05T17:43:29Z
body: I would lean towards the fully backwards compatible options for now - i.e. option 3 for `__getitem__` and option 1 for `create_dataset()`. I generally dislike global flags for this kind of thing - h5py might be used in multiple places within one process, and a global flag will affect all of them.

I could see us making breaking changes in this direction when/if we make h5py 4.0 - and perhaps if the benefits are big enough, it might even be the push to do that. But there's no particular plan for 4.0 at present.
author_association: MEMBER
user: login: aragilar
created_at: 2025-02-11T12:00:47Z
body: I agree with @takluyver about backwards compatibility.

I think the main thing that needs to be worked out is can the new numpy string type be used directly with HDF5, or does it have to go through some intermediate state (given the numpy buffer apparently only contains metadata)? Having some internal helper functions to handle going between the various types would also help (so any transition we may do only involves changing which helper to call).
author_association: MEMBER
user: login: takluyver
created_at: 2025-02-11T12:15:25Z
body: I think we could convert directly between HDF5 vlen strings and Numpy strings in Cython - similar to the existing `conv_vlen2str` function which converts to Python bytes objects.

https://github.com/h5py/h5py/blob/738d0e018427a0e122442738d0d13c111d8a0bed/h5py/_conv.pyx#L202-L226

The Numpy C functions for that are described in: https://numpy.org/doc/stable/reference/c-api/strings.html
author_association: MEMBER

Diff:
diff --git a/.gitignore b/.gitignore
index e5f3da885..e32206745 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,6 +1,7 @@
 h5py/h5*.c
 h5py/utils.c
 h5py/_conv.c
+h5py/_npystrings.c
 h5py/_proxy.c
 h5py/_objects.c
 h5py/_errors.c
diff --git a/docs/high/dataset.rst b/docs/high/dataset.rst
index debc45bad..10e06f680 100644
--- a/docs/high/dataset.rst
+++ b/docs/high/dataset.rst
@@ -504,14 +504,26 @@ Reference
 
     .. method:: astype(dtype)
 
-        Return a wrapper allowing you to read data as a particular
+        Return a read-only view allowing you to read data as a particular
         type.  Conversion is handled by HDF5 directly, on the fly::
 
             >>> dset = f.create_dataset("bigint", (1000,), dtype='int64')
-            >>> out = dset.astype('int16')[:]
+            >>> out = dset.astype('int16')[:500]
             >>> out.dtype
             dtype('int16')
 
+        This can be faster than reading the data and then
+        converting it with NumPy:
+
+            >>> out = dset[:500].astype('int16')  # Typically slower
+
+        In case of variable-width strings, calling ``.astype('T')``
+        (NumPy's native variable-width strings) is more efficient than reading
+        the data into an object-type array; read more at :ref:`npystrings`.
+
+        .. versionchanged:: 3.14
+           Added support for NumPy variable-width strings (``dtype='T'``).
+
         .. versionchanged:: 3.9
            :meth:`astype` can no longer be used as a context manager.
 
@@ -526,6 +538,13 @@ Reference
        encoding is defined by the datatype - ASCII or UTF-8.
        This is not guaranteed to be correct.
 
+       .. note::
+          If you don't require backwards compatibility with NumPy 1.x or
+          h5py <3.14, you should consider reading into NumPy native strings
+          instead, which can be much faster, with
+          :meth:`.astype('T')<astype>`.
+          Read more at :ref:`npystrings`.
+
        .. versionadded:: 3.0
 
     .. method:: fields(names)
diff --git a/docs/special.rst b/docs/special.rst
index 294769522..6b55530cb 100644
--- a/docs/special.rst
+++ b/docs/special.rst
@@ -11,9 +11,9 @@ types.  As of version 2.3, h5py fully supports HDF5 enums and VL types.
 How special types are represented
 ---------------------------------
 
-Since there is no direct NumPy dtype for variable-length strings, enums or
-references, h5py extends the dtype system slightly to let HDF5 know how to
-store these types.  Each type is represented by a native NumPy dtype, with a
+Since there is no direct NumPy dtype for enums or references (and, in NumPy 1.x, for
+variable-length strings), h5py extends the dtype system slightly to let HDF5 know how
+to store these types.  Each type is represented by a native NumPy dtype, with a
 small amount of metadata attached.  NumPy routines ignore the metadata, but
 h5py can use it to determine how to store the data.
 
@@ -21,14 +21,18 @@ The metadata h5py attaches to dtypes is not part of the public API,
 so it may change between versions.
 Use the functions described below to create and check for these types.
 
-Variable-length strings
------------------------
+Variable-length strings in NumPy 1.x
+------------------------------------
 
 .. seealso:: :ref:`strings`
 
+.. note::
+   Starting from h5py 3.14 + NumPy 2.0, you can use native NumPy variable-width
+   strings, a.k.a. NpyStrings or StringDType. See :ref:`npystrings`.
+
 In HDF5, data in VL format is stored as arbitrary-length vectors of a base
 type.  In particular, strings are stored C-style in null-terminated buffers.
-NumPy has no native mechanism to support this.  Unfortunately, this is the
+NumPy 1.x has no native mechanism to support this.  Unfortunately, this is the
 de facto standard for representing strings in the HDF5 C API, and in many
 HDF5 applications.
 
diff --git a/docs/strings.rst b/docs/strings.rst
index b6be98303..70a3ceecc 100644
--- a/docs/strings.rst
+++ b/docs/strings.rst
@@ -13,19 +13,25 @@ Reading strings
 ---------------
 
 String data in HDF5 datasets is read as bytes by default: ``bytes`` objects
-for variable-length strings, or numpy bytes arrays (``'S'`` dtypes) for
-fixed-length strings. Use :meth:`.Dataset.asstr` to retrieve ``str`` objects.
+for variable-length strings, or NumPy bytes arrays (``'S'`` dtypes) for
+fixed-length strings. On NumPy >=2.0, use :meth:`astype('T')<.astype>`
+to read into an array of native variable-width NumPy strings.
+If you need backwards compatibility with NumPy 1.x or h5py <3.14, you should instead
+call :meth:`.asstr` to retrieve an array of ``str`` objects.
 
 Variable-length strings in attributes are read as ``str`` objects. These are
 decoded as UTF-8 with surrogate escaping for unrecognised bytes. Fixed-length
-strings are read as numpy bytes arrays, the same as for datasets.
+strings are read as NumPy bytes arrays, the same as for datasets.
 
 Storing strings
 ---------------
 
 When creating a new dataset or attribute, Python ``str`` or ``bytes`` objects
 will be treated as variable-length strings, marked as UTF-8 and ASCII respectively.
-Numpy bytes arrays (``'S'`` dtypes) make fixed-length strings.
+NumPy bytes arrays (``'S'`` dtypes) make fixed-length strings.
+NumPy NpyStrings arrays (``StringDType`` or ``'T'`` dtypes) make native variable-length
+strings.
+
 You can use :func:`.string_dtype` to explicitly specify any HDF5 string datatype,
 as shown in the examples below::
 
@@ -49,22 +55,75 @@ When writing data to an existing dataset or attribute, data passed as bytes is
 written without checking the encoding. Data passed as Python ``str`` objects
 is encoded as either ASCII or UTF-8, based on the HDF5 datatype.
 In either case, null bytes (``'\x00'``) in the data will cause an error.
+Data passed as NpyStrings is always encoded as UTF-8.
 
 .. warning::
 
    Fixed-length string datasets will silently truncate longer strings which
-   are written to them. Numpy byte string arrays do the same thing.
+   are written to them. NumPy byte string arrays do the same thing.
 
    Fixed-length strings in HDF5 hold a set number of bytes.
    It may take multiple bytes to store one character.
 
+.. _npystrings:
+
+NumPy variable-width strings
+----------------------------
+
+Starting with NumPy 2.0 and h5py 3.14, you can also use ``np.dtypes.StringDType()``,
+or ``np.dtype('T')`` for short, to specify native NumPy variable-width string dtypes,
+a.k.a. NpyStrings.
+
+However, note that when you open a dataset that you created as a StringDType, its dtype
+will be object dtype with ``bytes`` elements. Likewise, :meth:`.create_dataset`
+with parameter ``dtype='T'``, or with ``data=`` set to a NumPy array with StringDType,
+will create a dataset with object dtype.
+This is because the two data types are identical on the HDF5 file; this design is to
+ensure that NumPy 1.x and 2.x behave in the same way unless the user explicitly requests
+native strings.
+
+In order to efficiently write NpyStrings to a dataset, simply assign a NumPy array with
+StringDType to it, either through ``__setitem__`` or :meth:`.create_dataset`.
+To read NpyStrings out of any string-type dataset, use
+:meth:`astype('T')<.astype>`. In both cases, there will be no
+intermediate conversion to object-type array, even if the dataset's dtype is object::
+
+    # These three syntaxes are equivalent:
+    >>> ds = f.create_dataset('x', shape=(2, ), dtype=h5py.string_dtype())
+    >>> ds = f.create_dataset('x', shape=(2, ), dtype=np.dtypes.StringDType())
+    >>> ds = f.create_dataset('x', shape=(2, ), dtype='T')
+
+    # They all return a dataset with object dtype:
+    >>> ds
+    <HDF5 dataset "x": shape (2,), type "|O">
+
+    # When you open an already existing dataset, you also get object dtype:
+    >>> f['x']
+    <HDF5 dataset "x": shape (2,), type "|O">
+
+    # You can write a NumPy array with StringDType to it:
+    >>> ds[:] = np.asarray(['hello', 'world'], dtype='T')
+
+    # Reading back, you will get an object-type array of bytes by default:
+    >>> ds[:]
+    array([b'hello', b'world'], dtype=object)
+
+    # To read it as a native NumPy variable-width string array, use .astype('T'):
+    >>> ds.astype('T')[:]
+    array(['hello', 'world'], dtype=StringDType())
+
+    # The above is much faster than converting after reading:
+    >>> ds[:].astype('T')  # Slower; don't do this!
+    array(['hello', 'world'], dtype=StringDType())
+
+
 What about NumPy's ``U`` type?
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-NumPy also has a Unicode type, a UTF-32 fixed-width format (4-byte characters).
-HDF5 has no support for wide characters.  Rather than trying to hack around
-this and "pretend" to support it, h5py will raise an error if you try to store
-data of this type.
+NumPy also has a Unicode fixed-width type, a UTF-32 fixed-width format
+(4-byte characters). HDF5 has no support for wide characters.
+Rather than trying to hack around this and "pretend" to support it,
+h5py will raise an error if you try to store data of this type.
 
 .. _str_binary:
 
diff --git a/h5py/_hl/base.py b/h5py/_hl/base.py
index 935748766..1abed0f4a 100644
--- a/h5py/_hl/base.py
+++ b/h5py/_hl/base.py
@@ -102,6 +102,9 @@ def is_float16_dtype(dt):
 def array_for_new_object(data, specified_dtype=None):
     """Prepare an array from data used to create a new dataset or attribute"""
 
+    if not isinstance(specified_dtype, (np.dtype, type(None))):
+        specified_dtype = np.dtype(specified_dtype)
+
     # We mostly let HDF5 convert data as necessary when it's written.
     # But if we are going to a float16 datatype, pre-convert in python
     # to workaround a bug in the conversion.
diff --git a/h5py/_hl/dataset.py b/h5py/_hl/dataset.py
index 5604625d9..2ede29f20 100644
--- a/h5py/_hl/dataset.py
+++ b/h5py/_hl/dataset.py
@@ -254,7 +254,7 @@ class AsTypeView(AbstractView):
     """
     def __init__(self, dset, dtype):
         super().__init__(dset)
-        self._dtype = numpy.dtype(dtype)
+        self._dtype = dtype
 
     @property
     def dtype(self):
@@ -437,6 +437,18 @@ def astype(self, dtype):
 
         >>> double_precision = dataset.astype('f8')[0:100:2]
         """
+        dtype = numpy.dtype(dtype)
+        if dtype == self.dtype:
+            return self
+
+        if dtype.kind == "T":
+            string_info = h5t.check_string_dtype(self.dtype)
+            if string_info is None:
+                raise TypeError(
+                    f"dset.astype({dtype}) can only be used on datasets with "
+                    "an HDF5 string datatype"
+                )
+
         return AsTypeView(self, dtype)
 
     def asstr(self, encoding=None, errors='strict'):
@@ -447,6 +459,12 @@ def asstr(self, encoding=None, errors='strict'):
         The parameters have the same meaning as in ``bytes.decode()``.
         If ``encoding`` is unspecified, it will use the encoding in the HDF5
         datatype (either ascii or utf-8).
+
+        .. note::
+           On NumPy 2.0 and later, it is recommended to use native NumPy
+           variable-width strings instead:
+
+           >>> str_array = dataset.astype('T')[:]
         """
         string_info = h5t.check_string_dtype(self.dtype)
         if string_info is None:
diff --git a/h5py/_npystrings.pyx b/h5py/_npystrings.pyx
new file mode 100644
index 000000000..781348605
--- /dev/null
+++ b/h5py/_npystrings.pyx
@@ -0,0 +1,258 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from .utils cimport emalloc
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from NumPy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of NumPy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to NumPy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct npystrings_pack_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct npystrings_unpack_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t npystrings_pack_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef npystrings_pack_t* info = <npystrings_pack_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the NumPy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def npystrings_pack(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+
+    This function defines a pure-python API interface to _proxy.pyx.
+    This is necessary to allow this module to be conditionally imported,
+    allowing NumPy 1.x to continue working everywhere else.
+    """
+    _npystrings_pack(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _npystrings_pack(hid_t space, void *contig, void *noncontig,
+                           PyArray_Descr *descr):
+    """Cython API interface of npystrings_pack"""
+    cdef npystrings_pack_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # H5Diterate needs a tid of the correct size.
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct of arbitrary size)
+    # and not HDF5 variable length strings (char*).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for NumPy
+    H5Diterate(noncontig, tid, space, npystrings_pack_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t npystrings_unpack_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef npystrings_unpack_t *info = <npystrings_unpack_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    # res == -1 if unpacking the string fails, 1 if packed_string is the null string,
+    # and 0 otherwise.
+    return -1 if res == -1 else 0
+
+
+def npystrings_unpack(hid_t space, size_t contig, size_t noncontig, size_t descr,
+                      size_t npoints):
+    """Convert a NpyString array (NumPy's native variable-width strings dtype)
+    to a zero-terminated char**, which is the in-memory representation for a
+    HDF5 variable-width string dataset.
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the input non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Output buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Input buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Returns
+    -------
+    char *
+        A temporary buffer containing the zero-terminated strings.
+        This buffer must be freed after writing to disk.
+
+        Note that this always the same as contig[0] when this
+        function returns, but H5Tconvert may change the contents
+        of contig[0] in place later on.
+
+    Notes
+    -----
+    This function defines a pure-python API interface to _proxy.pyx.
+    This is necessary to allow this module to be conditionally imported,
+    allowing NumPy 1.x to continue working everywhere else.
+    """
+    return <size_t>_npystrings_unpack(space, <void *>contig, <void *>noncontig,
+                                      <PyArray_Descr *>descr, npoints)
+
+
+cdef char * _npystrings_unpack(hid_t space, void *contig, void *noncontig,
+                               PyArray_Descr *descr, size_t npoints):
+    """Cython API interface of npystrings_unpack"""
+    cdef npystrings_unpack_t info
+    cdef size_t total_size
+    cdef size_t cur_size
+    cdef char *zero_terminated = NULL
+    cdef char *zero_terminated_cur = NULL
+    info.unpacked = NULL
+    info.i = 0
+
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+    try:
+        # Multiple steps needed:
+        # 1. Read npy_packed_static_string[] from NumPy and unpack
+        #    to npy_static_string[]; which is
+        #    {const char* buf, size_t size}[] - NOT zero-terminated
+        info.unpacked = <npy_static_string*>emalloc(
+            npoints * sizeof(npy_static_string)
+        )
+        H5Diterate(noncontig, tid, space, npystrings_unpack_cb, &info)
+        assert info.i == npoints
+
+        # 2. Calculate total size of strings with zero termination
+        total_size = npoints  # zero termination characters
+        for i in range(npoints):
+            total_size += info.unpacked[i].size
+
+        # 3. Copy to temporary buffer which is a concatenation of
+        #    zero-terminated char* and point to it from the
+        #    output char*[] for h5py
+        zero_terminated_buf = <char *>emalloc(total_size)
+        zero_terminated_cur = zero_terminated_buf
+        for i in range(npoints):
+            cur_size = info.unpacked[i].size
+            memcpy(zero_terminated_cur, info.unpacked[i].buf, cur_size)
+            zero_terminated_cur[cur_size] = 0
+            (<char **>contig)[i] = zero_terminated_cur
+            zero_terminated_cur += cur_size + 1
+
+        return zero_terminated_buf
+
+    finally:
+        free(info.unpacked)
+        NpyString_release_allocator(info.allocator)
+        H5Tclose(tid)
+        # after H5Dwrite, user must free(zero_terminated_buf)
diff --git a/h5py/_proxy.pxd b/h5py/_proxy.pxd
index 5fd2cc685..364e8cf99 100644
--- a/h5py/_proxy.pxd
+++ b/h5py/_proxy.pxd
@@ -9,10 +9,15 @@
 #           and contributor agreement.
 
 from .defs cimport *
+from numpy cimport PyArray_Descr
 
 cdef herr_t attr_rw(hid_t attr, hid_t mtype, void *progbuf, int read) except -1
 
 cdef herr_t dset_rw(hid_t dset, hid_t mtype, hid_t mspace, hid_t fspace,
                     hid_t dxpl, void* progbuf, int read) except -1
 
+cdef herr_t dset_rw_vlen_strings(hid_t dset, hid_t mspace, hid_t fspace,
+                    hid_t dxpl, void* progbuf, PyArray_Descr* descr,
+                    int read) except -1
+
 cdef htri_t needs_bkg_buffer(hid_t src, hid_t dst) except -1
diff --git a/h5py/_proxy.pyx b/h5py/_proxy.pyx
index e40504f53..5af9cc0f9 100644
--- a/h5py/_proxy.pyx
+++ b/h5py/_proxy.pyx
@@ -15,6 +15,12 @@
 
 include "config.pxi"
 
+from .h5t import NUMPY_GE2
+if NUMPY_GE2:
+    # Numpy native variable-width strings
+    # This fails to import on NumPy < 2.0
+    from ._npystrings import npystrings_pack, npystrings_unpack
+
 cdef enum copy_dir:
     H5PY_SCATTER = 0,
     H5PY_GATHER
@@ -123,6 +129,8 @@ cdef herr_t dset_rw(hid_t dset, hid_t mtype, hid_t mspace, hid_t fspace,
                 fspace = mspace = dspace = H5Dget_space(dset)
 
             npoints = H5Sget_select_npoints(mspace)
+            if npoints == 0:
+                return 0
             cspace = H5Screate_simple(1, &npoints, NULL)
 
             conv_buf = create_buffer(H5Tget_size(dstype), H5Tget_size(mtype), npoints)
@@ -161,6 +169,69 @@ cdef herr_t dset_rw(hid_t dset, hid_t mtype, hid_t mspace, hid_t fspace,
     return 0
 
 
+cdef herr_t dset_rw_vlen_strings(
+    hid_t dset, hid_t mspace, hid_t fspace, hid_t dxpl,
+    void* progbuf, PyArray_Descr* descr, int read) except -1:
+    """Variant of dset_rw for variable-width NumPy strings.
+    Note: doesn't support compound types.
+    """
+
+    cdef hid_t dstype = -1      # Dataset datatype
+    cdef hid_t h5_vlen_string = -1
+    cdef hid_t dspace = -1      # Dataset dataspace
+    cdef hid_t cspace = -1      # Temporary contiguous dataspaces
+
+    cdef void* conv_buf = NULL
+    cdef char* zero_terminated_buf = NULL
+    cdef hsize_t npoints
+
+    assert NUMPY_GE2
+
+    h5_vlen_string = H5Tcopy(H5T_C_S1)
+    H5Tset_size(h5_vlen_string, H5T_VARIABLE)
+    H5Tset_cset(h5_vlen_string, H5T_CSET_UTF8)
+
+    try:
+        dstype = H5Dget_type(dset)
+
+        if mspace == H5S_ALL and fspace != H5S_ALL:
+            mspace = fspace
+        elif mspace != H5S_ALL and fspace == H5S_ALL:
+            fspace = mspace
+        elif mspace == H5S_ALL and fspace == H5S_ALL:
+            fspace = mspace = dspace = H5Dget_space(dset)
+
+        npoints = H5Sget_select_npoints(mspace)
+        if npoints == 0:
+            return 0
+        cspace = H5Screate_simple(1, &npoints, NULL)
+
+        conv_buf = create_buffer(H5Tget_size(dstype), H5Tget_size(h5_vlen_string), npoints)
+
+        if read:
+            H5Dread(dset, h5_vlen_string, cspace, fspace, dxpl, conv_buf)
+            # Convert contiguous char** to discontiguous NpyStrings.
+            npystrings_pack(mspace, <size_t> conv_buf, <size_t> progbuf, <size_t> descr)
+            H5Dvlen_reclaim(dstype, cspace, H5P_DEFAULT, conv_buf)
+        else:
+            # Convert discontiguous NpyStrings to contiguous char**.
+            zero_terminated_buf = <char *> <size_t> npystrings_unpack(
+                mspace, <size_t> conv_buf, <size_t> progbuf, <size_t> descr,
+                npoints)
+            H5Dwrite(dset, h5_vlen_string, cspace, fspace, dxpl, conv_buf)
+    finally:
+        free(zero_terminated_buf)
+        free(conv_buf)
+        if dstype > 0:
+            H5Tclose(dstype)
+        if dspace > 0:
+            H5Sclose(dspace)
+        if cspace > 0:
+            H5Sclose(cspace)
+
+    return 0
+
+
 cdef hid_t make_reduced_type(hid_t mtype, hid_t dstype):
     # Go through dstype, pick out the fields which also appear in mtype, and
     # return a new compound type with the fields packed together
@@ -264,8 +335,8 @@ cdef herr_t h5py_gather_cb(void* elem, hid_t type_id, unsigned ndim,
 
 # Copy between a contiguous and non-contiguous buffer, with the layout
 # of the latter specified by a dataspace selection.
-cdef herr_t h5py_copy(hid_t tid, hid_t space, void* contig, void* noncontig,
-                 copy_dir op) except -1:
+cdef void h5py_copy(hid_t tid, hid_t space, void* contig, void* noncontig,
+                    copy_dir op):
 
     cdef h5py_scatter_t info
     cdef hsize_t elsize
@@ -281,9 +352,7 @@ cdef herr_t h5py_copy(hid_t tid, hid_t space, void* contig, void* noncontig,
     elif op == H5PY_GATHER:
         H5Diterate(noncontig, tid, space, h5py_gather_cb, &info)
     else:
-        raise RuntimeError("Illegal direction")
-
-    return 0
+        raise AssertionError("unreachable")
 
 # =============================================================================
 # VLEN support routines
diff --git a/h5py/h5d.pyx b/h5py/h5d.pyx
index 12676364e..c9eb21771 100644
--- a/h5py/h5d.pyx
+++ b/h5py/h5d.pyx
@@ -14,18 +14,18 @@
 include "config.pxi"
 
 # Compile-time imports
-
-from collections import namedtuple
 cimport cython
+from libc.string cimport strcmp
 from ._objects cimport pdefault
-from numpy cimport ndarray, import_array, PyArray_DATA
+from numpy cimport ndarray, import_array, PyArray_DATA, PyArray_Descr, PyArray_DESCR
 from .utils cimport  check_numpy_read, check_numpy_write, \
                      convert_tuple, convert_dims, emalloc, efree
-from .h5t cimport TypeID, typewrap, py_create
+from .h5t cimport TypeID, typewrap, py_create, H5PY_NUMPY_STRING_TAG
 from .h5s cimport SpaceID
 from .h5p cimport PropID, propwrap
-from ._proxy cimport dset_rw
+from ._proxy cimport dset_rw, dset_rw_vlen_strings
 
+from collections import namedtuple
 from ._objects import phil, with_phil
 from cpython cimport PyBUF_ANY_CONTIGUOUS, \
                      PyBuffer_Release, \
@@ -150,7 +150,23 @@ def open(ObjectID loc not None, char* name, PropID dapl=None):
     """
     return DatasetID(H5Dopen(loc.id, name, pdefault(dapl)))
 
-# --- Proxy functions for safe(r) threading -----------------------------------
+
+cdef bint _is_numpy_vlen_string(hid_t obj):
+    # Check for HDF5 datatype representing numpy variable-length strings
+    # This complexity is needed to ensure:
+    #   1) That ctag is freed
+    #   2) We don't segfault (for some reason a try-finally statement is needed,
+    #   even if we do (what I think are) the right steps in copying and freeing.
+    cdef char* ctag = NULL
+    try:
+        if H5Tget_class(obj) == H5T_OPAQUE:
+            ctag = H5Tget_tag(obj)
+            if ctag != NULL:
+                if strcmp(ctag, H5PY_NUMPY_STRING_TAG) == 0:
+                    return True
+        return False
+    finally:
+        H5free_memory(ctag)
 
 
 cdef class DatasetID(ObjectID):
@@ -225,6 +241,7 @@ cdef class DatasetID(ObjectID):
         """
         cdef hid_t self_id, mtype_id, mspace_id, fspace_id, plist_id
         cdef void* data
+        cdef PyArray_Descr* descr
         cdef int oldflags
 
         if mtype is None:
@@ -238,7 +255,11 @@ cdef class DatasetID(ObjectID):
         plist_id = pdefault(dxpl)
         data = PyArray_DATA(arr_obj)
 
-        dset_rw(self_id, mtype_id, mspace_id, fspace_id, plist_id, data, 1)
+        if _is_numpy_vlen_string(mtype_id):
+            descr = PyArray_DESCR(arr_obj)
+            dset_rw_vlen_strings(self_id, mspace_id, fspace_id, plist_id, data, descr, 1)
+        else:
+            dset_rw(self_id, mtype_id, mspace_id, fspace_id, plist_id, data, 1)
 
 
     @with_phil
@@ -265,6 +286,7 @@ cdef class DatasetID(ObjectID):
         """
         cdef hid_t self_id, mtype_id, mspace_id, fspace_id, plist_id
         cdef void* data
+        cdef PyArray_Descr* descr
         cdef int oldflags
 
         if mtype is None:
@@ -278,7 +300,11 @@ cdef class DatasetID(ObjectID):
         plist_id = pdefault(dxpl)
         data = PyArray_DATA(arr_obj)
 
-        dset_rw(self_id, mtype_id, mspace_id, fspace_id, plist_id, data, 0)
+        if _is_numpy_vlen_string(mtype_id):
+            descr = PyArray_DESCR(arr_obj)
+            dset_rw_vlen_strings(self_id, mspace_id, fspace_id, plist_id, data, descr, 0)
+        else:
+            dset_rw(self_id, mtype_id, mspace_id, fspace_id, plist_id, data, 0)
 
 
     @with_phil
diff --git a/h5py/h5t.pxd b/h5py/h5t.pxd
index 8e9bec014..be88cc4ee 100644
--- a/h5py/h5t.pxd
+++ b/h5py/h5t.pxd
@@ -69,4 +69,5 @@ cdef class TypeCompoundID(TypeCompositeID):
 cpdef TypeID typewrap(hid_t id_)
 cdef hid_t H5PY_OBJ
 cdef char* H5PY_PYTHON_OPAQUE_TAG
+cdef char* H5PY_NUMPY_STRING_TAG
 cpdef TypeID py_create(object dtype, bint logical=*, bint aligned=*)
diff --git a/h5py/h5t.pyx b/h5py/h5t.pyx
index 1d5d57d35..1db626568 100644
--- a/h5py/h5t.pyx
+++ b/h5py/h5t.pyx
@@ -43,6 +43,9 @@ _IS_PPC64 = _UNAME_MACHINE == "ppc64"
 _IS_PPC64LE = _UNAME_MACHINE == "ppc64le"
 
 cdef char* H5PY_PYTHON_OPAQUE_TAG = "PYTHON:OBJECT"
+cdef char* H5PY_NUMPY_STRING_TAG = "NUMPY:STRING"
+
+NUMPY_GE2 = int(np.__version__.split('.')[0]) >= 2
 
 # === Custom C API ============================================================
 
@@ -266,6 +269,14 @@ H5Tlock(H5PY_OBJ)
 
 PYTHON_OBJECT = lockid(H5PY_OBJ)
 
+# HDF5 datatype to represent Numpy (>=2) variable-length strings
+# Should never be directly stored in files.
+cdef hid_t H5PY_NPYSTR = H5Tcreate(H5T_OPAQUE, 2 * sizeof(size_t))
+H5Tset_tag(H5PY_NPYSTR, H5PY_NUMPY_STRING_TAG)
+H5Tlock(H5PY_NPYSTR)
+
+NUMPY_VLEN_STRING = lockid(H5PY_NPYSTR)
+
 # Translation tables for HDF5 -> NumPy dtype conversion
 cdef dict _order_map = { H5T_ORDER_NONE: '|', H5T_ORDER_LE: '<', H5T_ORDER_BE: '>'}
 cdef dict _sign_map  = { H5T_SGN_NONE: 'u', H5T_SGN_2: 'i' }
@@ -1480,7 +1491,7 @@ cdef TypeEnumID _c_enum(cnp.dtype dt, dict vals):
         if isinstance(name, bytes):
             bname = name
         else:
-            bname = unicode(name).encode('utf8')
+            bname = str(name).encode('utf8')
         out.enum_insert(bname, vals[name])
     return out
 
@@ -1618,7 +1629,7 @@ cdef TypeCompoundID _c_compound(cnp.dtype dt, int logical, int aligned):
     # (i.e. the position of the member in the struct)
     for name in sorted(dt.names, key=(lambda n: dt.fields[n][1])):
         field = dt.fields[name]
-        h5_name = name.encode('utf8') if isinstance(name, unicode) else name
+        h5_name = name.encode('utf8') if isinstance(name, str) else name
 
         # Get HDF5 data types and set the offset for each member
         member_dt = field[0]
@@ -1734,6 +1745,13 @@ cpdef TypeID py_create(object dtype_in, bint logical=0, bint aligned=0):
         elif kind == c'b':
             return _c_bool(dt)
 
+        # numpy.dtypes.StringDType
+        elif kind == c'T':
+            if logical:
+                return _c_vlen_unicode()
+
+            return NUMPY_VLEN_STRING
+
         # Object types (including those with vlen hints)
         elif kind == c'O':
 
@@ -1741,7 +1759,7 @@ cpdef TypeID py_create(object dtype_in, bint logical=0, bint aligned=0):
                 vlen = check_vlen_dtype(dt)
                 if vlen is bytes:
                     return _c_vlen_str()
-                elif vlen is unicode:
+                elif vlen is str:
                     return _c_vlen_unicode()
                 elif vlen is not None:
                     return vlen_create(py_create(vlen, logical))
@@ -1775,7 +1793,11 @@ def string_dtype(encoding='utf-8', length=None):
     not unicode code points.
 
     For variable length strings, the data should be passed as Python str objects
-    (unicode in Python 2) if the encoding is 'utf-8', and bytes if it is 'ascii'.
+    if the encoding is 'utf-8', and bytes if it is 'ascii'.
+    Starting from NumPy 2.0, it is preferred to pass utf-8 data as variable-width
+    NumPy strings (``dtype=numpy.dtypes.StringDType()`` or ``dtype='T'``
+    instead of ``dtype=h5py.string_dtype()``).
+
     For fixed length strings, the data should be numpy fixed length *bytes*
     arrays, regardless of the encoding. Fixed length unicode data is not
     supported.
@@ -1794,7 +1816,8 @@ def string_dtype(encoding='utf-8', length=None):
         # Fixed length string
         return np.dtype("|S" + str(length), metadata={'h5py_encoding': encoding})
     elif length is None:
-        vlen = unicode if (encoding == 'utf-8') else bytes
+        # Variable-width Python object string
+        vlen = str if (encoding == 'utf-8') else bytes
         return np.dtype('O', metadata={'vlen': vlen})
     else:
         raise TypeError("length must be integer or None (got %r)" % length)
@@ -1885,6 +1908,10 @@ def check_vlen_dtype(dt):
 
     Returns None if the dtype does not represent an HDF5 vlen.
     """
+    # StringDType (NumPy >=2.0)
+    if dt.kind == 'T':
+        return str
+
     try:
         return dt.metadata.get('vlen', None)
     except AttributeError:
@@ -1902,7 +1929,7 @@ def check_string_dtype(dt):
     Returns None if the dtype does not represent an HDF5 string.
     """
     vlen_kind = check_vlen_dtype(dt)
-    if vlen_kind is unicode:
+    if vlen_kind is str:
         return string_info('utf-8', None)
     elif vlen_kind is bytes:
         return string_info('ascii', None)
@@ -1971,6 +1998,10 @@ def check_dtype(**kwds):
     if name not in ('vlen', 'enum', 'ref'):
         raise TypeError('Unknown special type "%s"' % name)
 
+    # StringDType (numpy >=2.0)
+    if name == "vlen" and dt.kind == "T":
+        return str
+
     try:
         return dt.metadata[name]
     except TypeError:
diff --git a/h5py/tests/test_npystrings.py b/h5py/tests/test_npystrings.py
new file mode 100644
index 000000000..4291b0874
--- /dev/null
+++ b/h5py/tests/test_npystrings.py
@@ -0,0 +1,163 @@
+import numpy as np
+import pytest
+
+import h5py
+
+NUMPY_GE2 = int(np.__version__.split(".")[0]) >= 2
+pytestmark = pytest.mark.skipif(not NUMPY_GE2, reason="requires numpy >=2.0")
+
+
+def test_create_with_dtype_T(writable_file):
+    ds = writable_file.create_dataset("x", shape=(2, 2), dtype="T")
+    data = [["foo", "bar"], ["hello world", ""]]
+    ds[:] = data
+    a = ds.asstr()[:]
+    np.testing.assert_array_equal(a, data)
+
+    ds = writable_file["x"]
+    assert ds.dtype == object
+    np.testing.assert_array_equal(ds.asstr()[:], data)
+
+    ds[0, 0] = "baz"
+    data[0][0] = "baz"
+    a = ds.astype("T")[:]
+    assert a.dtype.kind == "T"
+    np.testing.assert_array_equal(a, data)
+
+    ds[0, 0] = np.asarray("123", dtype="O")
+    data[0][0] = "123"
+    np.testing.assert_array_equal(ds.asstr()[:], data)
+
+
+def test_fromdata(writable_file):
+    data = [["foo", "bar"]]
+    np_data = np.asarray(data, dtype="T")
+    x = writable_file.create_dataset("x", data=data, dtype="T")
+    y = writable_file.create_dataset("y", data=data, dtype=np.dtypes.StringDType())
+    z = writable_file.create_dataset("z", data=np_data)
+
+    for ds in (x, y, z):
+        assert ds.dtype.kind == "O"
+        np.testing.assert_array_equal(ds.astype("T")[:], np_data)
+    for name in ("x", "y", "z"):
+        ds = writable_file[name]
+        assert ds.dtype == object
+        np.testing.assert_array_equal(ds.asstr()[:], data)
+        ds = ds.astype("T")
+        assert ds.dtype.kind == "T"
+        a = ds[:]
+        assert a.dtype.kind == "T"
+        np.testing.assert_array_equal(a, data)
+
+
+def test_fixed_to_variable_width(writable_file):
+    data = ["foo", "longer than 8 bytes"]
+    x = writable_file.create_dataset(
+        "x", data=data, dtype=h5py.string_dtype(length=20)
+    )
+    assert x.dtype == "S20"
+
+    # read T <- S
+    y = x.astype("T")
+    assert y.dtype.kind == "T"
+    assert y[:].dtype.kind == "T"
+    np.testing.assert_array_equal(y[:], data)
+
+    # write T -> S
+    x[0] = np.asarray("1234", dtype="T")
+    data[0] = "1234"
+    np.testing.assert_array_equal(y[:], data)
+
+
+def test_fixed_to_variable_width_too_short(writable_file):
+    # Note: this test triggers calls to H5Tconvert which are otherwise skipped.
+
+    data = ["foo", "bar"]
+    x = writable_file.create_dataset(
+        "x", data=data, dtype=h5py.string_dtype(length=3)
+    )
+    assert x.dtype == "S3"
+
+    # write T -> S
+    x[0] = np.asarray("1234", dtype="T")
+    np.testing.assert_array_equal(x[:], [b"123", b"bar"])
+
+
+def test_variable_to_fixed_width(writable_file):
+    data = ["foo", "longer than 8 bytes"]
+    bdata = [b"foo", b"longer than 8 bytes"]
+    x = writable_file.create_dataset("x", data=data, dtype="T")
+
+    # read S <- T
+    y = x.astype("S20")
+    assert y.dtype == "S20"
+    assert y[:].dtype == "S20"
+    np.testing.assert_array_equal(y[:], bdata)
+
+    y = x.astype("S3")
+    assert y.dtype == "S3"
+    assert y[:].dtype == "S3"
+    np.testing.assert_array_equal(y[:], [b"foo", b"lon"])
+
+    # write S -> T
+    x[0] = np.asarray(b"1234", dtype="S5")
+    bdata[0] = b"1234"
+    np.testing.assert_array_equal(x[:], bdata)
+
+
+def test_write_object_into_npystrings(writable_file):
+    x = writable_file.create_dataset("x", data=["foo"], dtype="T")
+    x[0] = np.asarray("1234", dtype="O")
+    np.testing.assert_array_equal(x[:], b"1234")
+
+
+def test_write_npystrings_into_object(writable_file):
+    x = writable_file.create_dataset(
+        "x", data=["foo"], dtype=h5py.string_dtype()
+    )
+    assert x.dtype == object
+    x[0] = np.asarray("1234", dtype="T")
+    np.testing.assert_array_equal(x[:], b"1234")
+
+    # Test with HDF5 variable-length strings with ASCII character set
+    xa = writable_file.create_dataset(
+        "xa", shape=(1,), dtype=h5py.string_dtype('ascii')
+    )
+    xa[0] = np.asarray("2345", dtype="T")
+    np.testing.assert_array_equal(xa[:], b"2345")
+
+
+def test_fillvalue(writable_file):
+    # Create as NpyString dtype
+    x = writable_file.create_dataset("x", shape=(2,), dtype="T", fillvalue="foo")
+    assert isinstance(x.fillvalue, bytes)
+    assert x.fillvalue == b"foo"
+    assert x[0] == b"foo"
+
+    # Create as object dtype
+    y = writable_file.create_dataset(
+        "y", shape=(2,), dtype=h5py.string_dtype(), fillvalue=b"foo"
+    )
+    assert isinstance(y.fillvalue, bytes)
+    assert y.fillvalue == b"foo"
+    assert y[0] == b"foo"
+    # Convert object dtype to NpyString
+    y = y.astype("T")
+    assert y[0] == "foo"
+
+
+def test_empty_string(writable_file):
+    data = np.array(["", "a", "b"], dtype="T")
+    x = writable_file.create_dataset("x", data=data)
+    np.testing.assert_array_equal(x[:], [b"", b"a", b"b"])
+    np.testing.assert_array_equal(x.astype("T")[:], data)
+    data[:2] = ["c", ""]
+    x[:2] = data[:2]
+    np.testing.assert_array_equal(x[:], [b"c", b"", b"b"])
+    np.testing.assert_array_equal(x.astype("T")[:], data)
+
+
+def test_astype_nonstring(writable_file):
+    x = writable_file.create_dataset("x", shape=(2, ), dtype="i8")
+    with pytest.raises(TypeError, match="HDF5 string datatype"):
+        x.astype("T")
diff --git a/news/npystrings.rst b/news/npystrings.rst
new file mode 100644
index 000000000..046c33c40
--- /dev/null
+++ b/news/npystrings.rst
@@ -0,0 +1,34 @@
+New features
+------------
+
+* On NumPy 2.x, it is now possible to read and write native NumPy variable-width
+  strings, a.k.a. NpyStrings (``dtype=np.dtypes.StringDType()`` or ``dtype='T'``
+  for short), which are much faster than object strings. For the sake of
+  interoperability with NumPy 1.x, users need to explicitly opt in.
+  See :ref:`npystrings`.
+
+Deprecations
+------------
+
+* <news item>
+
+Exposing HDF5 functions
+-----------------------
+
+* <news item>
+
+Bug fixes
+---------
+
+* <news item>
+
+Building h5py
+-------------
+
+* NpyStrings introduce no changes in the build process: you need NumPy 2.x to build
+  (as before), and the same binary wheels are backwards compatible with NumPy 1.x.
+
+Development
+-----------
+
+* <news item>
diff --git a/setup_build.py b/setup_build.py
index 853216e2a..9b4766e03 100644
--- a/setup_build.py
+++ b/setup_build.py
@@ -10,6 +10,7 @@
 except ImportError:
     from distutils.extension import Extension
 from distutils.command.build_ext import build_ext
+import copy
 import sys
 import os
 import os.path as op
@@ -23,6 +24,7 @@ def localpath(*args):
     return op.abspath(op.join(op.dirname(__file__), *args))
 
 
+MODULES_NUMPY2 = ['_npystrings']
 MODULES = ['defs', '_errors', '_objects', '_proxy', 'h5fd', 'h5z',
             'h5', 'h5i', 'h5r', 'utils', '_selector',
             '_conv', 'h5t', 'h5s',
@@ -30,7 +32,7 @@ def localpath(*args):
             'h5d', 'h5a', 'h5f', 'h5g',
             'h5l', 'h5o',
             'h5ds', 'h5ac',
-            'h5pl']
+            'h5pl'] + MODULES_NUMPY2
 
 COMPILER_SETTINGS = {
    'libraries'      : ['hdf5', 'hdf5_hl'],
@@ -78,8 +80,8 @@ class h5py_build_ext(build_ext):
         NumPy being present in the main body of the setup script.
     """
 
-    @staticmethod
-    def _make_extensions(config):
+    @classmethod
+    def _make_extensions(cls, config):
         """ Produce a list of Extension instances which can be passed to
         cythonize().
 
@@ -115,12 +117,23 @@ def _make_extensions(config):
         if os.name != 'nt':
             settings['runtime_library_dirs'] = settings['library_dirs']
 
-        def make_extension(module):
-            sources = [localpath('h5py', module + '.pyx')] + EXTRA_SRC.get(module, [])
-            settings['libraries'] += EXTRA_LIBRARIES.get(module, [])
-            return Extension('h5py.' + module, sources, **settings)
+        return [cls._make_extension(m, settings) for m in MODULES]
+
+    @staticmethod
+    def _make_extension(module, settings):
+        import numpy
+
+        sources = [localpath('h5py', module + '.pyx')] + EXTRA_SRC.get(module, [])
+        settings = copy.deepcopy(settings)
+        settings['libraries'] += EXTRA_LIBRARIES.get(module, [])
+
+        assert numpy.__version__ >= '2.0'  # See build dependencies in pyproject.toml
+        if module in MODULES_NUMPY2:
+            # Enable NumPy 2.0 C API for modules that require it.
+            # These modules will not be importable when NumPy 1.x is installed.
+            settings['define_macros'].append(('NPY_TARGET_VERSION', 0x00000012))
 
-        return [make_extension(m) for m in MODULES]
+        return Extension('h5py.' + module, sources, **settings)
 
     def run(self):
         """ Distutils calls this method to run the command """


Reviews:
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T15:58:09Z
user: login: ngoldbaum
body: Neat, seems like a clean design given that the copies are necessary.

As you see from the comments, it also makes me wonder if there are things we can do in NumPy to make this easier. 

I think probably storing NpyString as null terminated buffers in the arena might be the most useful, to avoid the bookkeeping outside of NumPy. The extra memory cost is probably worth it because many real-world cases need the null termination.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T16:02:36Z
user: login: takluyver
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-07T16:25:57Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T10:33:19Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T10:43:26Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T10:46:41Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T10:48:07Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T10:58:04Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T11:50:09Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T11:52:11Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T11:53:06Z
user: login: neutrinoceros
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T11:55:45Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T11:57:46Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T12:02:31Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T12:05:13Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T12:07:41Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T12:10:42Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T13:04:43Z
user: login: ngoldbaum
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T14:26:24Z
user: login: ngoldbaum
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T14:27:02Z
user: login: ngoldbaum
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T14:38:10Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T10:36:02Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T10:42:22Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T15:45:42Z
user: login: aragilar
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-07T13:39:54Z
user: login: takluyver
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-09T15:38:31Z
user: login: takluyver
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-09T15:56:14Z
user: login: takluyver
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-09T16:16:09Z
user: login: takluyver
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-09T16:20:45Z
user: login: takluyver
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-09T16:39:40Z
user: login: takluyver
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-09T16:50:10Z
user: login: takluyver
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-09T17:19:34Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T11:50:57Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T12:13:10Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T12:23:53Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T12:33:35Z
user: login: crusaderky
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-16T15:01:11Z
user: login: takluyver
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T11:17:57Z
user: login: takluyver
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T11:47:07Z
user: login: takluyver
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-21T11:51:45Z

Review comments:
diff_hunk: @@ -138,7 +148,31 @@ cdef herr_t dset_rw(hid_t dset, hid_t mtype, hid_t mspace, hid_t fspace,
                 if read:
                     h5py_copy(mtype, mspace, back_buf, progbuf, H5PY_GATHER)
 
-            if read:
+            if H5Tis_variable_str(mtype):  # numpy.dtypes.StringDType
+                assert NUMPY_GE2
+                if read:
+                    H5Dread(dset, dstype, cspace, fspace, dxpl, conv_buf)
+                    if not H5Tis_variable_str(dstype):
+                        # Convert fixed-width bytes to char** in place.
+                        # When dstype is already char**, H5Tconvert is a very expensive no-op.
+                        H5Tconvert(dstype, mtype, npoints, conv_buf, back_buf, dxpl)
path: h5py/_proxy.pyx
user: login: crusaderky
body: One could achieve a substantial speedup by writing a specialized variant of vstrings_scatter instead. I chose not to as I believe this should be a rather niche use case so I preferred to avoid code duplication.
created_at: 2025-03-10T15:58:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,254 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct *typically* 16 bytes per point),
path: h5py/_npystrings.pyx
user: login: ngoldbaum
body: It's 16 bytes per string in the array buffer, not codepoint. Unless "per point" means something different here.

It's also only 16 bytes on 64 bit architectures and 8 and 32 bit, not sure if that matters from h5py's perspective.

I would also not use "typically" here, since it's always the same size on the same numpy version and architecture. We tried to design it so we could change the size in the future (e.g. make it smaller).
created_at: 2025-04-07T15:27:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,254 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct *typically* 16 bytes per point),
+    # and not HDF5 variable length strings (char*; 8 bytes per point).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
path: h5py/_npystrings.pyx
user: login: ngoldbaum
body: I'm a little surprised it's necessary to create an HDF5 type. Are you storing the packed strings directly in HDF5 files or is this needed for something else?
created_at: 2025-04-07T15:30:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,254 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct *typically* 16 bytes per point),
+    # and not HDF5 variable length strings (char*; 8 bytes per point).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for numpy
+    H5Diterate(noncontig, tid, space, vstrings_scatter_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t vstrings_gather_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_gather_t *info = <vstrings_gather_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    return res
+
+def vstrings_gather(hid_t space, size_t contig, size_t noncontig, size_t descr,
+                    size_t npoints):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    return <size_t>_vstrings_gather(space, <void *>contig, <void *>noncontig,
+                                    <PyArray_Descr *>descr, npoints)
+
+
+cdef char * _vstrings_gather(hid_t space, void *contig, void *noncontig,
+                             PyArray_Descr *descr, size_t npoints):
+    """Convert a NpyString array (NumPy's native variable-width strings dtype)
+    to a zero-terminated char**, which is the in-memory representation for a
+    HDF5 variable-width string dataset.
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the input non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Output buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Input buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Returns
+    -------
+    char *
+        A temporary buffer containing the zero-terminated strings.
+        This buffer must be freed after writing to disk.
+
+        Note that this always the same as contig[0] when this
+        function returns, but H5Tconvert may change the contents
+        of contig[0] in place later on.
+    """
+    cdef vstrings_gather_t info
+    cdef size_t total_size
+    cdef size_t cur_size
+    cdef char *zero_terminated = NULL
+    cdef char *zero_terminated_cur = NULL
+    info.unpacked = NULL
+    info.i = 0
+
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+    try:
+        # Multiple steps needed:
+        # 1. Read npy_packed_static_string[] from numpy and unpack
+        #    to npy_static_string[]; which is
+        #    {const char* buf, size_t size}[] - NOT zero-terminated
path: h5py/_npystrings.pyx
user: login: ngoldbaum
body: Would it avoid copies and be generally helpful here if NumPy strings were null-terminated? It would make it more complicated to support older NumPy versions here, of course. We wanted to hear about use-cases before deciding to do that.

I also wonder if it makes sense to have a NumPy C API function that does this conversion to a `char**` buffer, there might be optimizations that are possible if you have access to the NpyString internals.
created_at: 2025-04-07T15:33:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -508,10 +508,23 @@ Reference
         type.  Conversion is handled by HDF5 directly, on the fly::
 
             >>> dset = f.create_dataset("bigint", (1000,), dtype='int64')
-            >>> out = dset.astype('int16')[:]
+            >>> out = dset.astype('int16')[:500]
             >>> out.dtype
             dtype('int16')
 
+        This is typically faster than reading the data and then
+        converting it with NumPy:
path: docs/high/dataset.rst
user: login: takluyver
body: Have you done timings on this? I would guess that it can be more memory efficient in at least some scenarios, by converting chunks of data piece by piece, but I don't know if it's actually faster in general. And it wouldn't surprise me to learn that there are some cases where it's massively slower. My experience is that HDF5 is more likely to drop off a performance cliff than numpy when things aren't set up the optimal way.
created_at: 2025-04-07T15:36:15Z
author_association: MEMBER
diff_hunk: @@ -650,6 +693,12 @@ def maxshape(self):
     @with_phil
     def fillvalue(self):
         """Fill value for this dataset (0 by default)"""
+        if self.dtype.kind == "T":
+            # For the sake of simplicity, pass through object strings
+            arr = numpy.zeros((1,), dtype=self.astype(object).dtype)
path: h5py/_hl/dataset.py
user: login: ngoldbaum
body: Why not create a `StringDType` array here?
created_at: 2025-04-07T15:45:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -650,6 +693,12 @@ def maxshape(self):
     @with_phil
     def fillvalue(self):
         """Fill value for this dataset (0 by default)"""
+        if self.dtype.kind == "T":
+            # For the sake of simplicity, pass through object strings
+            arr = numpy.zeros((1,), dtype=self.astype(object).dtype)
path: h5py/_hl/dataset.py
user: login: ngoldbaum
body: Or I guess, more precisely, why is it simpler to not use StringDType here?
created_at: 2025-04-07T15:45:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,13 +63,35 @@ In either case, null bytes (``'\x00'``) in the data will cause an error.
    Fixed-length strings in HDF5 hold a set number of bytes.
    It may take multiple bytes to store one character.
 
+.. _npystrings:
+
+NumPy variable-width strings
+----------------------------
+
+Starting with NumPy 2.0 and h5py 3.14, you can also use ``dtype('T')`` to specify
+native NumPy variable-width strings. However, note that on a round-trip you will
+read object arrays of ``bytes`` again. This is to ensure that NumPy 1.x and 2.x
path: docs/strings.rst
user: login: takluyver
body: I think this could be a bit clearer - it's not specifically about round-tripping, just the default format for reading vlen string data.
created_at: 2025-04-07T15:47:41Z
author_association: MEMBER
diff_hunk: @@ -58,13 +63,35 @@ In either case, null bytes (``'\x00'``) in the data will cause an error.
    Fixed-length strings in HDF5 hold a set number of bytes.
    It may take multiple bytes to store one character.
 
+.. _npystrings:
+
+NumPy variable-width strings
+----------------------------
+
+Starting with NumPy 2.0 and h5py 3.14, you can also use ``dtype('T')`` to specify
+native NumPy variable-width strings. However, note that on a round-trip you will
+read object arrays of ``bytes`` again. This is to ensure that NumPy 1.x and 2.x
+behave in the same way unless the user explicitly requests native strings::
+
+    # NpyStrings (implicit)
+    npdata = np.asarray(string_data, dtype='T')
+    ds = f.create_dataset('npystrings1', data=npdata)
+
+    # NpyStrings (explicit)
+    ds = f.create_dataset('npystrings2', shape=4, dtype='T')
+    ds[:] = string_data
+
+    # On a round-trip, you're back to bytes by default.
+    # Calling .astype('T') creates a writeable view with NpyStrings dtype.
+    ds = f['npystrings1'].astype('T')
path: docs/strings.rst
user: login: takluyver
body: ```suggestion
    string_arr = f['npystrings1'].astype('T')[:]
```

Or similar - let's show actually reading them. Also, if possible, let's avoid calling the view 'ds' (dataset) in examples.
created_at: 2025-04-07T15:48:56Z
author_association: MEMBER
diff_hunk: @@ -124,10 +124,12 @@ def make_new_dset(parent, shape=None, dtype=None, data=None, name=None,
         if string_info is not None:
             # fake vlen dtype for fixed len string fillvalue
             # to not trigger unwanted encoding
-            dtype = h5t.string_dtype(string_info.encoding)
-            fillvalue = numpy.array(fillvalue, dtype=dtype)
+            fv_dtype = h5t.string_dtype(string_info.encoding)
+        elif dtype.kind == "T":
path: h5py/_hl/dataset.py
user: login: takluyver
body: I think this branch can't be taken, because `check_string_dtype()` will recognise your "T" dtype, so the branch above is taken instead.
created_at: 2025-04-07T15:54:11Z
author_association: MEMBER
diff_hunk: @@ -1115,17 +1164,20 @@ def __array__(self, dtype=None, copy=None):
     @with_phil
     def __repr__(self):
         if not self:
-            r = '<Closed HDF5 dataset>'
+            return "<Closed HDF5 dataset>"
+
+        if self.name is None:
+            namestr = "(anonymous)"
         else:
-            if self.name is None:
-                namestr = '("anonymous")'
-            else:
-                name = pp.basename(pp.normpath(self.name))
-                namestr = '"%s"' % (name if name != '' else '/')
-            r = '<HDF5 dataset %s: shape %s, type "%s">' % (
-                namestr, self.shape, self.dtype.str
-            )
-        return r
+            name = pp.basename(pp.normpath(self.name))
+            namestr = f'"{name}"' if name else "/"
+
+        if self.dtype.kind == "T":
+            typestr = str(self.dtype)  # StringDType()
+        else:
+            typestr = f'"{self.dtype.str}"'
path: h5py/_hl/dataset.py
user: login: ngoldbaum
body: Any chance you can open a NumPy bug for this? IMO `dtype.str` for StringDType should just be the same as the string conversion so you don't need to do stuff like this.
created_at: 2025-04-07T15:55:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,254 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct *typically* 16 bytes per point),
+    # and not HDF5 variable length strings (char*; 8 bytes per point).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for numpy
+    H5Diterate(noncontig, tid, space, vstrings_scatter_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t vstrings_gather_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_gather_t *info = <vstrings_gather_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    return res
+
+def vstrings_gather(hid_t space, size_t contig, size_t noncontig, size_t descr,
+                    size_t npoints):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    return <size_t>_vstrings_gather(space, <void *>contig, <void *>noncontig,
+                                    <PyArray_Descr *>descr, npoints)
+
+
+cdef char * _vstrings_gather(hid_t space, void *contig, void *noncontig,
+                             PyArray_Descr *descr, size_t npoints):
+    """Convert a NpyString array (NumPy's native variable-width strings dtype)
+    to a zero-terminated char**, which is the in-memory representation for a
+    HDF5 variable-width string dataset.
path: h5py/_npystrings.pyx
user: login: ngoldbaum
body: It occurs to me that this is the same operation as asking NumPy to do a cast to a fixed-width byte array (which outputs UTF-8 data).

I wonder if that would be more maintainable and/or faster than trying to do the copies yourself.
created_at: 2025-04-07T16:00:29Z
author_association: CONTRIBUTOR
diff_hunk: @@ -173,6 +175,9 @@ def make_new_dset(parent, shape=None, dtype=None, data=None, name=None,
         sid = h5s.create_simple(shape, maxshape)
 
     dset_id = h5d.create(parent.id, name, tid, sid, dcpl=dcpl, dapl=dapl)
+    # Multiple NumPy dtypes can map to the same hdf5 type; e.g.
+    # object strings or NpyStrings map to vlen strings.
+    dset_id._dtype = dtype
path: h5py/_hl/dataset.py
user: login: takluyver
body: Let's avoid this - a newly created dataset should not behave any differently from one retrieved from an existing file. I.e. it should not know that it was created with the new string dtype.
created_at: 2025-04-07T16:03:28Z
author_association: MEMBER
diff_hunk: @@ -508,10 +508,23 @@ Reference
         type.  Conversion is handled by HDF5 directly, on the fly::
 
             >>> dset = f.create_dataset("bigint", (1000,), dtype='int64')
-            >>> out = dset.astype('int16')[:]
+            >>> out = dset.astype('int16')[:500]
             >>> out.dtype
             dtype('int16')
 
+        This is typically faster than reading the data and then
+        converting it with NumPy:
+
+            >>> out = dset[:500].astype('int16')  # slower
+
+        Calling ``.astype('T')`` (NumPy's native variable-width strings)
+        on a dataset returns a writable dataset with the specified dtype;
+        read more at :ref:`npystrings`.
+        For all other dtypes, this method returns a read-only view.
path: docs/high/dataset.rst
user: login: takluyver
body: Why is a writable view needed for this case? I think writing an array with "T" dtype into a variable length string dataset should work without needing a view.
created_at: 2025-04-07T16:09:47Z
author_association: MEMBER
diff_hunk: @@ -436,7 +441,36 @@ def astype(self, dtype):
         different destination type, e.g.:
 
         >>> double_precision = dataset.astype('f8')[0:100:2]
+
+        Returns
+        -------
+        For dtype='T' (NumPy's native variable-length strings),
+        a writable dataset with the specified dtype.
+
+        For all other dtypes, a read-only view of the dataset
+        with the specified dtype.
         """
+        dtype = numpy.dtype(dtype)
+        if dtype == self.dtype:
+            return self
+
+        if dtype.kind == "T":
+            string_info = h5t.check_string_dtype(self.dtype)
+            if string_info is None:
+                raise TypeError(
+                    f"dset.astype({dtype}) can only be used on datasets with "
+                    "an HDF5 string datatype"
+                )
+            out = self.parent[self.name]
+            assert out.id is not self.id
+            out.id._dtype = dtype
+            return out
+        elif dtype == object and self.dtype.kind == "T":
+            out = self.parent[self.name]
+            assert out.id is not self.id
+            out.id._dtype = None
+            return out
path: h5py/_hl/dataset.py
user: login: takluyver
body: This seems rather convoluted, and I think it's in service of having writable `.astype('T')` views, which it's not clear to me are actually important. I think if we can get away from switching the dtype of dataset objects, quite a few of the later changes in this file can be avoided as well.
created_at: 2025-04-07T16:13:08Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,254 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct *typically* 16 bytes per point),
+    # and not HDF5 variable length strings (char*; 8 bytes per point).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
path: h5py/_npystrings.pyx
user: login: crusaderky
body: The tid is needed by H5TIterate a few lines later. I've added a comment.
created_at: 2025-04-08T10:33:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,254 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct *typically* 16 bytes per point),
+    # and not HDF5 variable length strings (char*; 8 bytes per point).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for numpy
+    H5Diterate(noncontig, tid, space, vstrings_scatter_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t vstrings_gather_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_gather_t *info = <vstrings_gather_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    return res
+
+def vstrings_gather(hid_t space, size_t contig, size_t noncontig, size_t descr,
+                    size_t npoints):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    return <size_t>_vstrings_gather(space, <void *>contig, <void *>noncontig,
+                                    <PyArray_Descr *>descr, npoints)
+
+
+cdef char * _vstrings_gather(hid_t space, void *contig, void *noncontig,
+                             PyArray_Descr *descr, size_t npoints):
+    """Convert a NpyString array (NumPy's native variable-width strings dtype)
+    to a zero-terminated char**, which is the in-memory representation for a
+    HDF5 variable-width string dataset.
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the input non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Output buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Input buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Returns
+    -------
+    char *
+        A temporary buffer containing the zero-terminated strings.
+        This buffer must be freed after writing to disk.
+
+        Note that this always the same as contig[0] when this
+        function returns, but H5Tconvert may change the contents
+        of contig[0] in place later on.
+    """
+    cdef vstrings_gather_t info
+    cdef size_t total_size
+    cdef size_t cur_size
+    cdef char *zero_terminated = NULL
+    cdef char *zero_terminated_cur = NULL
+    info.unpacked = NULL
+    info.i = 0
+
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+    try:
+        # Multiple steps needed:
+        # 1. Read npy_packed_static_string[] from numpy and unpack
+        #    to npy_static_string[]; which is
+        #    {const char* buf, size_t size}[] - NOT zero-terminated
path: h5py/_npystrings.pyx
user: login: crusaderky
body: > Would it avoid copies and be generally helpful here if NumPy strings were null-terminated? It would make it more complicated to support older NumPy versions here, of course. We wanted to hear about use-cases before deciding to do that.

Yes, it would avoid a copy of the whole thing into the temporary buffer.

> I also wonder if it makes sense to have a NumPy C API function that does this conversion to a `char**` buffer, there might be optimizations that are possible if you have access to the NpyString internals.

It would probably be useful. From my understanding of NpyStrings internal structure, NumPy could add a null termination at no cost for all strings shorter than 15 bytes* (the maximum size of an embedded string).

*7 bytes on 32 bit architectures?


created_at: 2025-04-08T10:43:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -650,6 +693,12 @@ def maxshape(self):
     @with_phil
     def fillvalue(self):
         """Fill value for this dataset (0 by default)"""
+        if self.dtype.kind == "T":
+            # For the sake of simplicity, pass through object strings
+            arr = numpy.zeros((1,), dtype=self.astype(object).dtype)
path: h5py/_hl/dataset.py
user: login: crusaderky
body: I would have needed to replicate everything I did for datasets (or make it somehow generic and probably much less legible) around `dcpl.get_fill_value` -> `H5Pget_fill_value`.
created_at: 2025-04-08T10:46:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -650,6 +693,12 @@ def maxshape(self):
     @with_phil
     def fillvalue(self):
         """Fill value for this dataset (0 by default)"""
+        if self.dtype.kind == "T":
+            # For the sake of simplicity, pass through object strings
+            arr = numpy.zeros((1,), dtype=self.astype(object).dtype)
path: h5py/_hl/dataset.py
user: login: crusaderky
body: `attrs` have the same issue - they would probably benefit from being read into a StringDType array and then converted to list instead of being read into a object array of bytes and then converted with `bytes.decode`. But performance wise they're inconsequential so I left them untouched.
created_at: 2025-04-08T10:48:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1115,17 +1164,20 @@ def __array__(self, dtype=None, copy=None):
     @with_phil
     def __repr__(self):
         if not self:
-            r = '<Closed HDF5 dataset>'
+            return "<Closed HDF5 dataset>"
+
+        if self.name is None:
+            namestr = "(anonymous)"
         else:
-            if self.name is None:
-                namestr = '("anonymous")'
-            else:
-                name = pp.basename(pp.normpath(self.name))
-                namestr = '"%s"' % (name if name != '' else '/')
-            r = '<HDF5 dataset %s: shape %s, type "%s">' % (
-                namestr, self.shape, self.dtype.str
-            )
-        return r
+            name = pp.basename(pp.normpath(self.name))
+            namestr = f'"{name}"' if name else "/"
+
+        if self.dtype.kind == "T":
+            typestr = str(self.dtype)  # StringDType()
+        else:
+            typestr = f'"{self.dtype.str}"'
path: h5py/_hl/dataset.py
user: login: crusaderky
body: https://github.com/numpy/numpy/issues/28670
created_at: 2025-04-08T10:58:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,254 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct *typically* 16 bytes per point),
+    # and not HDF5 variable length strings (char*; 8 bytes per point).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for numpy
+    H5Diterate(noncontig, tid, space, vstrings_scatter_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t vstrings_gather_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_gather_t *info = <vstrings_gather_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    return res
+
+def vstrings_gather(hid_t space, size_t contig, size_t noncontig, size_t descr,
+                    size_t npoints):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    return <size_t>_vstrings_gather(space, <void *>contig, <void *>noncontig,
+                                    <PyArray_Descr *>descr, npoints)
+
+
+cdef char * _vstrings_gather(hid_t space, void *contig, void *noncontig,
+                             PyArray_Descr *descr, size_t npoints):
+    """Convert a NpyString array (NumPy's native variable-width strings dtype)
+    to a zero-terminated char**, which is the in-memory representation for a
+    HDF5 variable-width string dataset.
path: h5py/_npystrings.pyx
user: login: crusaderky
body: I don't understand. Isn't a fixed-witdth array padded with NULLs? That would potentially cause a very high peak in memory usage?
created_at: 2025-04-08T11:50:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -508,10 +508,23 @@ Reference
         type.  Conversion is handled by HDF5 directly, on the fly::
 
             >>> dset = f.create_dataset("bigint", (1000,), dtype='int64')
-            >>> out = dset.astype('int16')[:]
+            >>> out = dset.astype('int16')[:500]
             >>> out.dtype
             dtype('int16')
 
+        This is typically faster than reading the data and then
path: docs/high/dataset.rst
user: login: crusaderky
body: ```suggestion
        This can be faster than reading the data and then
```
created_at: 2025-04-08T11:52:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -508,10 +508,23 @@ Reference
         type.  Conversion is handled by HDF5 directly, on the fly::
 
             >>> dset = f.create_dataset("bigint", (1000,), dtype='int64')
-            >>> out = dset.astype('int16')[:]
+            >>> out = dset.astype('int16')[:500]
             >>> out.dtype
             dtype('int16')
 
+        This is typically faster than reading the data and then
+        converting it with NumPy:
path: docs/high/dataset.rst
user: login: crusaderky
body: I have not done timings for the numeric use case. It is _definitely_ much faster in the specific case of NpyStrings though (benchmark in the initial comment of this PR).
created_at: 2025-04-08T11:53:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1115,17 +1164,20 @@ def __array__(self, dtype=None, copy=None):
     @with_phil
     def __repr__(self):
         if not self:
-            r = '<Closed HDF5 dataset>'
+            return "<Closed HDF5 dataset>"
+
+        if self.name is None:
+            namestr = "(anonymous)"
         else:
-            if self.name is None:
-                namestr = '("anonymous")'
-            else:
-                name = pp.basename(pp.normpath(self.name))
-                namestr = '"%s"' % (name if name != '' else '/')
-            r = '<HDF5 dataset %s: shape %s, type "%s">' % (
-                namestr, self.shape, self.dtype.str
-            )
-        return r
+            name = pp.basename(pp.normpath(self.name))
+            namestr = f'"{name}"' if name else "/"
+
+        if self.dtype.kind == "T":
+            typestr = str(self.dtype)  # StringDType()
+        else:
+            typestr = f'"{self.dtype.str}"'
path: h5py/_hl/dataset.py
user: login: neutrinoceros
body: Thanks @crusaderky ! Can you also link the issue in an inline comment so it's easier to decide when to clean up in the future ?
created_at: 2025-04-08T11:55:45Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,13 +63,35 @@ In either case, null bytes (``'\x00'``) in the data will cause an error.
    Fixed-length strings in HDF5 hold a set number of bytes.
    It may take multiple bytes to store one character.
 
+.. _npystrings:
+
+NumPy variable-width strings
+----------------------------
+
+Starting with NumPy 2.0 and h5py 3.14, you can also use ``dtype('T')`` to specify
+native NumPy variable-width strings. However, note that on a round-trip you will
+read object arrays of ``bytes`` again. This is to ensure that NumPy 1.x and 2.x
+behave in the same way unless the user explicitly requests native strings::
+
+    # NpyStrings (implicit)
+    npdata = np.asarray(string_data, dtype='T')
+    ds = f.create_dataset('npystrings1', data=npdata)
+
+    # NpyStrings (explicit)
+    ds = f.create_dataset('npystrings2', shape=4, dtype='T')
+    ds[:] = string_data
+
+    # On a round-trip, you're back to bytes by default.
+    # Calling .astype('T') creates a writeable view with NpyStrings dtype.
+    ds = f['npystrings1'].astype('T')
path: docs/strings.rst
user: login: crusaderky
body: but... it's not a `AsStrView` or a `AsTypeView`. it's an actual `h5py.Dataset` object.
created_at: 2025-04-08T11:57:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -58,13 +63,35 @@ In either case, null bytes (``'\x00'``) in the data will cause an error.
    Fixed-length strings in HDF5 hold a set number of bytes.
    It may take multiple bytes to store one character.
 
+.. _npystrings:
+
+NumPy variable-width strings
+----------------------------
+
+Starting with NumPy 2.0 and h5py 3.14, you can also use ``dtype('T')`` to specify
+native NumPy variable-width strings. However, note that on a round-trip you will
+read object arrays of ``bytes`` again. This is to ensure that NumPy 1.x and 2.x
path: docs/strings.rst
user: login: crusaderky
body: I've reworked the paragraph.
created_at: 2025-04-08T12:02:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -124,10 +124,12 @@ def make_new_dset(parent, shape=None, dtype=None, data=None, name=None,
         if string_info is not None:
             # fake vlen dtype for fixed len string fillvalue
             # to not trigger unwanted encoding
-            dtype = h5t.string_dtype(string_info.encoding)
-            fillvalue = numpy.array(fillvalue, dtype=dtype)
+            fv_dtype = h5t.string_dtype(string_info.encoding)
+        elif dtype.kind == "T":
path: h5py/_hl/dataset.py
user: login: crusaderky
body: `h5t.check_string_dtype` and `h5t.string_dtype` are strictly for vlen dtypes - not for StringDType.
created_at: 2025-04-08T12:05:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -173,6 +175,9 @@ def make_new_dset(parent, shape=None, dtype=None, data=None, name=None,
         sid = h5s.create_simple(shape, maxshape)
 
     dset_id = h5d.create(parent.id, name, tid, sid, dcpl=dcpl, dapl=dapl)
+    # Multiple NumPy dtypes can map to the same hdf5 type; e.g.
+    # object strings or NpyStrings map to vlen strings.
+    dset_id._dtype = dtype
path: h5py/_hl/dataset.py
user: login: crusaderky
body: This is what makes it possible to write
```python
ds = f.create_dataset("x", data=["foo"], dtype="T")
```
or
```python
ds = f.create_dataset("x", shape=(1, ), dtype="T")
```
or
```python
data = np.asarray(["foo"], dtype="T")
ds = f.create_dataset("x", data=data)
```
otherwise, the dtype information would be immediately forgotten and I think you'd agree it would be mightily confusing for end users to have to call `.astype("T")` immediately afterwards.
created_at: 2025-04-08T12:07:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -508,10 +508,23 @@ Reference
         type.  Conversion is handled by HDF5 directly, on the fly::
 
             >>> dset = f.create_dataset("bigint", (1000,), dtype='int64')
-            >>> out = dset.astype('int16')[:]
+            >>> out = dset.astype('int16')[:500]
             >>> out.dtype
             dtype('int16')
 
+        This is typically faster than reading the data and then
+        converting it with NumPy:
+
+            >>> out = dset[:500].astype('int16')  # slower
+
+        Calling ``.astype('T')`` (NumPy's native variable-width strings)
+        on a dataset returns a writable dataset with the specified dtype;
+        read more at :ref:`npystrings`.
+        For all other dtypes, this method returns a read-only view.
path: docs/high/dataset.rst
user: login: crusaderky
body: Yes, writing a StringDType array into a vlen Dataset works.
However, if the user fancies _reading_ back from it, they would be surprised to get an object array.
Particularly (see my previous comment) if they just explicitly called `create_dataset(..., dtype="T")`.

created_at: 2025-04-08T12:10:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -436,7 +441,36 @@ def astype(self, dtype):
         different destination type, e.g.:
 
         >>> double_precision = dataset.astype('f8')[0:100:2]
+
+        Returns
+        -------
+        For dtype='T' (NumPy's native variable-length strings),
+        a writable dataset with the specified dtype.
+
+        For all other dtypes, a read-only view of the dataset
+        with the specified dtype.
         """
+        dtype = numpy.dtype(dtype)
+        if dtype == self.dtype:
+            return self
+
+        if dtype.kind == "T":
+            string_info = h5t.check_string_dtype(self.dtype)
+            if string_info is None:
+                raise TypeError(
+                    f"dset.astype({dtype}) can only be used on datasets with "
+                    "an HDF5 string datatype"
+                )
+            out = self.parent[self.name]
+            assert out.id is not self.id
+            out.id._dtype = dtype
+            return out
+        elif dtype == object and self.dtype.kind == "T":
+            out = self.parent[self.name]
+            assert out.id is not self.id
+            out.id._dtype = None
+            return out
path: h5py/_hl/dataset.py
user: login: crusaderky
body: Are you saying that `create_dataset(..., dtype="T")` should be disallowed?
Could you write a mockup of the end-user API that you have in mind?
created_at: 2025-04-08T13:04:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -650,6 +693,12 @@ def maxshape(self):
     @with_phil
     def fillvalue(self):
         """Fill value for this dataset (0 by default)"""
+        if self.dtype.kind == "T":
+            # For the sake of simplicity, pass through object strings
+            arr = numpy.zeros((1,), dtype=self.astype(object).dtype)
path: h5py/_hl/dataset.py
user: login: ngoldbaum
body: Maybe leave a comment explaining?
created_at: 2025-04-08T14:26:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,254 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct *typically* 16 bytes per point),
+    # and not HDF5 variable length strings (char*; 8 bytes per point).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for numpy
+    H5Diterate(noncontig, tid, space, vstrings_scatter_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t vstrings_gather_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_gather_t *info = <vstrings_gather_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    return res
+
+def vstrings_gather(hid_t space, size_t contig, size_t noncontig, size_t descr,
+                    size_t npoints):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    return <size_t>_vstrings_gather(space, <void *>contig, <void *>noncontig,
+                                    <PyArray_Descr *>descr, npoints)
+
+
+cdef char * _vstrings_gather(hid_t space, void *contig, void *noncontig,
+                             PyArray_Descr *descr, size_t npoints):
+    """Convert a NpyString array (NumPy's native variable-width strings dtype)
+    to a zero-terminated char**, which is the in-memory representation for a
+    HDF5 variable-width string dataset.
path: h5py/_npystrings.pyx
user: login: ngoldbaum
body: Sorry of course you're right. Not the first time I've made this mistake...
created_at: 2025-04-08T14:27:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,254 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct *typically* 16 bytes per point),
+    # and not HDF5 variable length strings (char*; 8 bytes per point).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for numpy
+    H5Diterate(noncontig, tid, space, vstrings_scatter_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t vstrings_gather_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_gather_t *info = <vstrings_gather_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    return res
+
+def vstrings_gather(hid_t space, size_t contig, size_t noncontig, size_t descr,
+                    size_t npoints):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    return <size_t>_vstrings_gather(space, <void *>contig, <void *>noncontig,
+                                    <PyArray_Descr *>descr, npoints)
+
+
+cdef char * _vstrings_gather(hid_t space, void *contig, void *noncontig,
+                             PyArray_Descr *descr, size_t npoints):
+    """Convert a NpyString array (NumPy's native variable-width strings dtype)
+    to a zero-terminated char**, which is the in-memory representation for a
+    HDF5 variable-width string dataset.
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the input non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Output buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Input buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Returns
+    -------
+    char *
+        A temporary buffer containing the zero-terminated strings.
+        This buffer must be freed after writing to disk.
+
+        Note that this always the same as contig[0] when this
+        function returns, but H5Tconvert may change the contents
+        of contig[0] in place later on.
+    """
+    cdef vstrings_gather_t info
+    cdef size_t total_size
+    cdef size_t cur_size
+    cdef char *zero_terminated = NULL
+    cdef char *zero_terminated_cur = NULL
+    info.unpacked = NULL
+    info.i = 0
+
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+    try:
+        # Multiple steps needed:
+        # 1. Read npy_packed_static_string[] from numpy and unpack
+        #    to npy_static_string[]; which is
+        #    {const char* buf, size_t size}[] - NOT zero-terminated
path: h5py/_npystrings.pyx
user: login: ngoldbaum
body: I brought this up over on the StringDType tracking issue: https://github.com/numpy/numpy/issues/25693#issuecomment-2786673501
created_at: 2025-04-08T14:38:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1115,17 +1164,20 @@ def __array__(self, dtype=None, copy=None):
     @with_phil
     def __repr__(self):
         if not self:
-            r = '<Closed HDF5 dataset>'
+            return "<Closed HDF5 dataset>"
+
+        if self.name is None:
+            namestr = "(anonymous)"
         else:
-            if self.name is None:
-                namestr = '("anonymous")'
-            else:
-                name = pp.basename(pp.normpath(self.name))
-                namestr = '"%s"' % (name if name != '' else '/')
-            r = '<HDF5 dataset %s: shape %s, type "%s">' % (
-                namestr, self.shape, self.dtype.str
-            )
-        return r
+            name = pp.basename(pp.normpath(self.name))
+            namestr = f'"{name}"' if name else "/"
+
+        if self.dtype.kind == "T":
+            typestr = str(self.dtype)  # StringDType()
+        else:
+            typestr = f'"{self.dtype.str}"'
path: h5py/_hl/dataset.py
user: login: crusaderky
body: Done already :)
created_at: 2025-04-09T10:36:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -650,6 +693,12 @@ def maxshape(self):
     @with_phil
     def fillvalue(self):
         """Fill value for this dataset (0 by default)"""
+        if self.dtype.kind == "T":
+            # For the sake of simplicity, pass through object strings
+            arr = numpy.zeros((1,), dtype=self.astype(object).dtype)
path: h5py/_hl/dataset.py
user: login: crusaderky
body: Done
created_at: 2025-04-09T10:42:22Z
author_association: CONTRIBUTOR
diff_hunk: @@ -508,10 +508,23 @@ Reference
         type.  Conversion is handled by HDF5 directly, on the fly::
 
             >>> dset = f.create_dataset("bigint", (1000,), dtype='int64')
-            >>> out = dset.astype('int16')[:]
+            >>> out = dset.astype('int16')[:500]
             >>> out.dtype
             dtype('int16')
 
+        This is typically faster than reading the data and then
+        converting it with NumPy:
path: docs/high/dataset.rst
user: login: crusaderky
body: Changed wording from _is typically faster_ to _can be faster_.
created_at: 2025-04-14T15:45:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -120,7 +120,7 @@ def array_for_new_object(data, specified_dtype=None):
     # In most cases, this does nothing. But if data was already an array,
     # and as_dtype is a tagged h5py dtype (e.g. for an object array of strings),
     # asarray() doesn't replace its dtype object. This gives it the tagged dtype:
-    if as_dtype is not None:
+    if as_dtype is not None and data.dtype.kind != "T":
path: h5py/_hl/base.py
user: login: aragilar
body: Maybe I missed it, but why are we special-casing the variable width strings here?
created_at: 2025-05-07T13:39:53Z
author_association: MEMBER
diff_hunk: @@ -508,10 +508,23 @@ Reference
         type.  Conversion is handled by HDF5 directly, on the fly::
 
             >>> dset = f.create_dataset("bigint", (1000,), dtype='int64')
-            >>> out = dset.astype('int16')[:]
+            >>> out = dset.astype('int16')[:500]
             >>> out.dtype
             dtype('int16')
 
+        This is typically faster than reading the data and then
+        converting it with NumPy:
+
+            >>> out = dset[:500].astype('int16')  # slower
+
+        Calling ``.astype('T')`` (NumPy's native variable-width strings)
+        on a dataset returns a writable dataset with the specified dtype;
+        read more at :ref:`npystrings`.
+        For all other dtypes, this method returns a read-only view.
path: docs/high/dataset.rst
user: login: takluyver
body: I missed changing the docs, but since my changes that you merged into this PR, `.astype('T')` is now more like other `.as*` methods, giving you a read-only view, while you can write without needing astype.
created_at: 2025-05-09T15:38:31Z
author_association: MEMBER
diff_hunk: @@ -266,6 +269,14 @@ H5Tlock(H5PY_OBJ)
 
 PYTHON_OBJECT = lockid(H5PY_OBJ)
 
+# HDF5 datatype to represent Numpy (>=2) variable-length strings
+# Should never be directly stored in files.
+cdef hid_t H5PY_NPYSTR = H5Tcreate(H5T_OPAQUE, 2 * sizeof(size_t))
path: h5py/h5t.pyx
user: login: takluyver
body: Just to note, the size I specified here kind of relies on the in-memory format of the numpy string datatype, which is [described in NEP 55](https://numpy.org/neps/nep-0055-string_dtype.html#memory-layout-and-managing-heap-allocations), but it's not part of the API. I don't *think* it should actually matter if it's wrong, since we're just using the datatype as a marker, and handling all the conversion ourselves, but I haven't checked that.

I guess there's an argument that we should set it to a totally incorrect value, like 1 byte, so that nothing can rely on the size. :shrug: 
created_at: 2025-05-09T15:56:14Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,255 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
path: h5py/_npystrings.pyx
user: login: takluyver
body: It may just be me, but I'm not finding the names 'scatter' and 'gather' here easy to grasp at all - I can see where it comes from, but the core of 'scatter' is calling a 'pack' function, which as a word is kind of the opposite.

How about basing the names on something like pack/unpack, pack/load (following Numpy's API), or even something like `hdf5_strings_to_numpy` / `numpy_strings_to_hdf5`?
created_at: 2025-05-09T16:16:09Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,255 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
path: h5py/_npystrings.pyx
user: login: takluyver
body: This is useful explanation, but I think this more public function should have the longer API docstring.
created_at: 2025-05-09T16:20:45Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,255 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # H5Diterate needs a tid of the correct size.
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct of arbitrary size)
+    # and not HDF5 variable length strings (char*).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for numpy
+    H5Diterate(noncontig, tid, space, vstrings_scatter_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t vstrings_gather_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_gather_t *info = <vstrings_gather_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    return res
path: h5py/_npystrings.pyx
user: login: takluyver
body: The NEP says that NpyString_load is allowed to return 1 to indicate a null string, and the [H5Diterate docs](https://support.hdfgroup.org/documentation/hdf5/latest/group___h5_d.html#ga71421c684884ab49765748720fe938db) imply that returning a non-zero value exits the loop, although this isn't actually specified. I couldn't trigger a problem in a brief test, but do we need any special handling around this?
created_at: 2025-05-09T16:39:40Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,255 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # H5Diterate needs a tid of the correct size.
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct of arbitrary size)
+    # and not HDF5 variable length strings (char*).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for numpy
+    H5Diterate(noncontig, tid, space, vstrings_scatter_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t vstrings_gather_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_gather_t *info = <vstrings_gather_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    return res
+
+def vstrings_gather(hid_t space, size_t contig, size_t noncontig, size_t descr,
+                    size_t npoints):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    return <size_t>_vstrings_gather(space, <void *>contig, <void *>noncontig,
+                                    <PyArray_Descr *>descr, npoints)
+
+
+cdef char * _vstrings_gather(hid_t space, void *contig, void *noncontig,
+                             PyArray_Descr *descr, size_t npoints):
+    """Convert a NpyString array (NumPy's native variable-width strings dtype)
+    to a zero-terminated char**, which is the in-memory representation for a
+    HDF5 variable-width string dataset.
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the input non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Output buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Input buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Returns
+    -------
+    char *
+        A temporary buffer containing the zero-terminated strings.
+        This buffer must be freed after writing to disk.
+
+        Note that this always the same as contig[0] when this
+        function returns, but H5Tconvert may change the contents
+        of contig[0] in place later on.
+    """
+    cdef vstrings_gather_t info
+    cdef size_t total_size
+    cdef size_t cur_size
+    cdef char *zero_terminated = NULL
+    cdef char *zero_terminated_cur = NULL
+    info.unpacked = NULL
+    info.i = 0
+
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+    try:
+        # Multiple steps needed:
+        # 1. Read npy_packed_static_string[] from numpy and unpack
+        #    to npy_static_string[]; which is
+        #    {const char* buf, size_t size}[] - NOT zero-terminated
+        info.unpacked = <npy_static_string*>malloc(
+            npoints * sizeof(npy_static_string)
+        )
+        if info.unpacked is NULL:
+            raise MemoryError("Failed to allocate memory")
+
+        H5Diterate(noncontig, tid, space, vstrings_gather_cb, &info)
+        assert info.i == npoints
+
+        # 2. Calculate total size of strings with zero termination
+        total_size = npoints  # zero termination characters
+        for i in range(npoints):
+            total_size += info.unpacked[i].size
+
+        # 3. Copy to temporary buffer which is a concatenation of
+        #    zero-terminated char* and point to it from the
+        #    output char*[] for h5py
+        zero_terminated_buf = <char *>malloc(total_size)
+        if zero_terminated_buf is NULL:
+            raise MemoryError("Failed to allocate memory")
+        zero_terminated_cur = zero_terminated_buf
+        for i in range(npoints):
+            cur_size = info.unpacked[i].size
+            memcpy(zero_terminated_cur, info.unpacked[i].buf, cur_size)
path: h5py/_npystrings.pyx
user: login: takluyver
body: We unpack the strings, so `info.unpacked` holds (size, pointer) pairs, then we copy the string data into `zero_terminated_buf`, then we free `info.unpacked`, but does anything free the string data all its pointers were pointing to?
created_at: 2025-05-09T16:50:10Z
author_association: MEMBER
diff_hunk: @@ -58,13 +63,38 @@ In either case, null bytes (``'\x00'``) in the data will cause an error.
    Fixed-length strings in HDF5 hold a set number of bytes.
    It may take multiple bytes to store one character.
 
+.. _npystrings:
+
+NumPy variable-width strings
path: docs/strings.rst
user: login: takluyver
body: This section also needs an update - datasets created with 'T' dtype are now regular variable-length string datasets, and reopening them doesn't doesn't matter.
created_at: 2025-05-09T17:19:34Z
author_association: MEMBER
diff_hunk: @@ -120,7 +120,7 @@ def array_for_new_object(data, specified_dtype=None):
     # In most cases, this does nothing. But if data was already an array,
     # and as_dtype is a tagged h5py dtype (e.g. for an object array of strings),
     # asarray() doesn't replace its dtype object. This gives it the tagged dtype:
-    if as_dtype is not None:
+    if as_dtype is not None and data.dtype.kind != "T":
path: h5py/_hl/base.py
user: login: crusaderky
body: There's some weird behaviour with literal string `'T'` dtype. I reworked it by ensuring it's a np.dtype object.
created_at: 2025-05-16T11:50:56Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,255 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
path: h5py/_npystrings.pyx
user: login: crusaderky
body: renamed to `npystrings_pack` / `npystrings_unpack`
created_at: 2025-05-16T12:13:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,255 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # H5Diterate needs a tid of the correct size.
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct of arbitrary size)
+    # and not HDF5 variable length strings (char*).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for numpy
+    H5Diterate(noncontig, tid, space, vstrings_scatter_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t vstrings_gather_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_gather_t *info = <vstrings_gather_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    return res
+
+def vstrings_gather(hid_t space, size_t contig, size_t noncontig, size_t descr,
+                    size_t npoints):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    return <size_t>_vstrings_gather(space, <void *>contig, <void *>noncontig,
+                                    <PyArray_Descr *>descr, npoints)
+
+
+cdef char * _vstrings_gather(hid_t space, void *contig, void *noncontig,
+                             PyArray_Descr *descr, size_t npoints):
+    """Convert a NpyString array (NumPy's native variable-width strings dtype)
+    to a zero-terminated char**, which is the in-memory representation for a
+    HDF5 variable-width string dataset.
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the input non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Output buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Input buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Returns
+    -------
+    char *
+        A temporary buffer containing the zero-terminated strings.
+        This buffer must be freed after writing to disk.
+
+        Note that this always the same as contig[0] when this
+        function returns, but H5Tconvert may change the contents
+        of contig[0] in place later on.
+    """
+    cdef vstrings_gather_t info
+    cdef size_t total_size
+    cdef size_t cur_size
+    cdef char *zero_terminated = NULL
+    cdef char *zero_terminated_cur = NULL
+    info.unpacked = NULL
+    info.i = 0
+
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+    try:
+        # Multiple steps needed:
+        # 1. Read npy_packed_static_string[] from numpy and unpack
+        #    to npy_static_string[]; which is
+        #    {const char* buf, size_t size}[] - NOT zero-terminated
+        info.unpacked = <npy_static_string*>malloc(
+            npoints * sizeof(npy_static_string)
+        )
+        if info.unpacked is NULL:
+            raise MemoryError("Failed to allocate memory")
+
+        H5Diterate(noncontig, tid, space, vstrings_gather_cb, &info)
+        assert info.i == npoints
+
+        # 2. Calculate total size of strings with zero termination
+        total_size = npoints  # zero termination characters
+        for i in range(npoints):
+            total_size += info.unpacked[i].size
+
+        # 3. Copy to temporary buffer which is a concatenation of
+        #    zero-terminated char* and point to it from the
+        #    output char*[] for h5py
+        zero_terminated_buf = <char *>malloc(total_size)
+        if zero_terminated_buf is NULL:
+            raise MemoryError("Failed to allocate memory")
+        zero_terminated_cur = zero_terminated_buf
+        for i in range(npoints):
+            cur_size = info.unpacked[i].size
+            memcpy(zero_terminated_cur, info.unpacked[i].buf, cur_size)
path: h5py/_npystrings.pyx
user: login: crusaderky
body: No need: `NpyString_load` only creates a view to the packed data.
created_at: 2025-05-16T12:23:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,255 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # H5Diterate needs a tid of the correct size.
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct of arbitrary size)
+    # and not HDF5 variable length strings (char*).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for numpy
+    H5Diterate(noncontig, tid, space, vstrings_scatter_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t vstrings_gather_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_gather_t *info = <vstrings_gather_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    return res
path: h5py/_npystrings.pyx
user: login: crusaderky
body: Couldn't reproduce it either. For the sake of caution, I've added special handling for it and added a unit test.
created_at: 2025-05-16T12:33:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -102,6 +102,9 @@ def is_float16_dtype(dt):
 def array_for_new_object(data, specified_dtype=None):
     """Prepare an array from data used to create a new dataset or attribute"""
 
+    if not isinstance(specified_dtype, (np.dtype, type(None))):
+        specified_dtype = np.dtype(specified_dtype)
path: h5py/_hl/base.py
user: login: crusaderky
body: literal `'T'` caused weird behaviour in `data.view` below
created_at: 2025-05-16T15:01:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -504,14 +504,27 @@ Reference
 
     .. method:: astype(dtype)
 
-        Return a wrapper allowing you to read data as a particular
+        Return a read-only view allowing you to read data as a particular
         type.  Conversion is handled by HDF5 directly, on the fly::
 
             >>> dset = f.create_dataset("bigint", (1000,), dtype='int64')
-            >>> out = dset.astype('int16')[:]
+            >>> out = dset.astype('int16')[:500]
             >>> out.dtype
             dtype('int16')
 
+        This can be faster than reading the data and then
+        converting it with NumPy:
+
+            >>> out = dset[:500].astype('int16')  # Typically slower
+
+        In case of variable-width strings, calling ``.astype('T')``
+        (NumPy's native variable-width strings) is zero cost and
path: docs/high/dataset.rst
user: login: takluyver
body: I'm not sure what 'zero cost' means here - clearly there is still a cost to reading and repacking data. Maybe we can just say it's more efficient than making Python string objects?
created_at: 2025-05-21T11:17:56Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,255 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from Numpy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of numpy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to numpy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct vstrings_scatter_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct vstrings_gather_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t vstrings_scatter_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_scatter_t* info = <vstrings_scatter_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the numpy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def vstrings_scatter(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    _vstrings_scatter(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _vstrings_scatter(hid_t space, void *contig, void *noncontig,
+                            PyArray_Descr *descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+    """
+    cdef vstrings_scatter_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # H5Diterate needs a tid of the correct size.
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct of arbitrary size)
+    # and not HDF5 variable length strings (char*).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for numpy
+    H5Diterate(noncontig, tid, space, vstrings_scatter_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t vstrings_gather_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef vstrings_gather_t *info = <vstrings_gather_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    return res
+
+def vstrings_gather(hid_t space, size_t contig, size_t noncontig, size_t descr,
+                    size_t npoints):
+    """Pure-python API interface to _proxy.pyx. This is necessary to
+    allow this module to be conditionally imported, allowing numpy 1.x to
+    continue working everywhere else.
+    """
+    return <size_t>_vstrings_gather(space, <void *>contig, <void *>noncontig,
+                                    <PyArray_Descr *>descr, npoints)
+
+
+cdef char * _vstrings_gather(hid_t space, void *contig, void *noncontig,
+                             PyArray_Descr *descr, size_t npoints):
+    """Convert a NpyString array (NumPy's native variable-width strings dtype)
+    to a zero-terminated char**, which is the in-memory representation for a
+    HDF5 variable-width string dataset.
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the input non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Output buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Input buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Returns
+    -------
+    char *
+        A temporary buffer containing the zero-terminated strings.
+        This buffer must be freed after writing to disk.
+
+        Note that this always the same as contig[0] when this
+        function returns, but H5Tconvert may change the contents
+        of contig[0] in place later on.
+    """
+    cdef vstrings_gather_t info
+    cdef size_t total_size
+    cdef size_t cur_size
+    cdef char *zero_terminated = NULL
+    cdef char *zero_terminated_cur = NULL
+    info.unpacked = NULL
+    info.i = 0
+
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+    try:
+        # Multiple steps needed:
+        # 1. Read npy_packed_static_string[] from numpy and unpack
+        #    to npy_static_string[]; which is
+        #    {const char* buf, size_t size}[] - NOT zero-terminated
+        info.unpacked = <npy_static_string*>malloc(
+            npoints * sizeof(npy_static_string)
+        )
+        if info.unpacked is NULL:
+            raise MemoryError("Failed to allocate memory")
+
+        H5Diterate(noncontig, tid, space, vstrings_gather_cb, &info)
+        assert info.i == npoints
+
+        # 2. Calculate total size of strings with zero termination
+        total_size = npoints  # zero termination characters
+        for i in range(npoints):
+            total_size += info.unpacked[i].size
+
+        # 3. Copy to temporary buffer which is a concatenation of
+        #    zero-terminated char* and point to it from the
+        #    output char*[] for h5py
+        zero_terminated_buf = <char *>malloc(total_size)
+        if zero_terminated_buf is NULL:
+            raise MemoryError("Failed to allocate memory")
+        zero_terminated_cur = zero_terminated_buf
+        for i in range(npoints):
+            cur_size = info.unpacked[i].size
+            memcpy(zero_terminated_cur, info.unpacked[i].buf, cur_size)
path: h5py/_npystrings.pyx
user: login: takluyver
body: Aha, that makes sense, thank-you!
created_at: 2025-05-21T11:47:07Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,262 @@
+"""Support for NpyStrings - NumPy's native variable-width strings.
+
+This module requires NumPy >=2.0 to be imported. To allow the rest of h5py
+to work with NumPy 1.x installed, this module must be imported conditionally
+and can't be cimport'ed.
+"""
+include "config.pxi"
+
+from .defs cimport *
+from numpy cimport PyArray_Descr
+import numpy as np
+
+assert NUMPY_BUILD_VERSION >= '2.0'  # See pyproject.toml
+
+# =========================================================================
+# Scatter/gather routines for vlen strings to/from NumPy StringDType arrays
+# See also: _proxy.pyx::h5py_copy, h5py_scatter_cb, h5py_gather_cb
+
+IF NUMPY_BUILD_VERSION < '2.3':
+    # Backport NpyStrings from 2.3 to 2.0
+    # The C API was available since 2.0, but the Cython API was added in 2.3.
+    # This is a copy-paste from numpy/__init__.pxd of NumPy 2.3
+    cdef extern from "numpy/ndarraytypes.h":
+        ctypedef struct npy_string_allocator:
+            pass
+
+        ctypedef struct npy_packed_static_string:
+            pass
+
+        ctypedef struct npy_static_string:
+            size_t size
+            const char *buf
+
+        ctypedef struct PyArray_StringDTypeObject:
+            PyArray_Descr base
+            PyObject *na_object
+            char coerce
+            char has_nan_na
+            char has_string_na
+            char array_owned
+            npy_static_string default_string
+            npy_static_string na_name
+            npy_string_allocator *allocator
+
+    cdef extern from "numpy/arrayobject.h":
+        npy_string_allocator *NpyString_acquire_allocator(const PyArray_StringDTypeObject *descr)
+        void NpyString_acquire_allocators(size_t n_descriptors, PyArray_Descr *const descrs[], npy_string_allocator *allocators[])
+        void NpyString_release_allocator(npy_string_allocator *allocator)
+        void NpyString_release_allocators(size_t length, npy_string_allocator *allocators[])
+        int NpyString_load(npy_string_allocator *allocator, const npy_packed_static_string *packed_string, npy_static_string *unpacked_string)
+        int NpyString_pack_null(npy_string_allocator *allocator, npy_packed_static_string *packed_string)
+        int NpyString_pack(npy_string_allocator *allocator, npy_packed_static_string *packed_string, const char *buf, size_t size)
+
+ELSE:
+    from numpy cimport (
+        npy_string_allocator,
+        npy_static_string,
+        npy_packed_static_string,
+        NpyString_pack,
+        NpyString_load,
+        NpyString_acquire_allocator,
+        NpyString_release_allocator,
+        PyArray_StringDTypeObject,
+    )
+
+# Can't call sizeof(npy_packed_static_string) because the struct is
+# internal to NumPy
+cdef size_t SIZEOF_NPY_PACKED_STATIC_STRING = np.dtype("T").itemsize
+
+ctypedef struct npystrings_pack_t:
+    size_t i
+    npy_string_allocator *allocator
+    const char **contig
+
+ctypedef struct npystrings_unpack_t:
+    size_t i
+    npy_string_allocator *allocator
+    npy_static_string *unpacked
+
+
+cdef herr_t npystrings_pack_cb(
+    void* elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef npystrings_pack_t* info = <npystrings_pack_t*>operator_data
+    cdef const char* buf = info[0].contig[info[0].i]
+    # Deep copy char* from h5py into the NumPy array
+    res = NpyString_pack(
+        info[0].allocator,
+        <npy_packed_static_string*>elem,
+        buf,
+        strlen(buf),
+    )
+    info[0].i += 1
+    return res
+
+
+def npystrings_pack(hid_t space, size_t contig, size_t noncontig, size_t descr):
+    """Convert a zero-terminated char**, which is the in-memory representation
+    for a HDF5 variable-width string dataset, into a NpyString array
+    (NumPy's native variable-width strings dtype).
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the output non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Input buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Output buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Notes
+    -----
+    This function deep-copies the input zero-terminated strings.
+    Memory management for the outputs is handled by NumPy.
+
+    This function defines a pure-python API interface to _proxy.pyx.
+    This is necessary to allow this module to be conditionally imported,
+    allowing NumPy 1.x to continue working everywhere else.
+    """
+    _npystrings_pack(space, <void *>contig, <void *>noncontig, <PyArray_Descr *>descr)
+
+
+cdef void _npystrings_pack(hid_t space, void *contig, void *noncontig,
+                           PyArray_Descr *descr):
+    """Cython API interface of npystrings_pack"""
+    cdef npystrings_pack_t info
+    info.i = 0
+    info.contig = <const char**>contig
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    # H5Diterate needs a tid of the correct size.
+    # Disregard mtype from _proxy::dset_rw.
+    # The memory type is actually npy_packed_static_string
+    # (an opaque struct of arbitrary size)
+    # and not HDF5 variable length strings (char*).
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+
+    # Read char*[] (zero-terminated) from h5py
+    # and deep-copy to npy_packed_static_string[] for NumPy
+    H5Diterate(noncontig, tid, space, npystrings_pack_cb, &info)
+    NpyString_release_allocator(info.allocator)
+    H5Tclose(tid)
+
+
+cdef herr_t npystrings_unpack_cb(
+    void *elem, hid_t type_id, unsigned ndim,
+    const hsize_t *point, void *operator_data
+) except -1:
+    cdef npystrings_unpack_t *info = <npystrings_unpack_t *>operator_data
+    cdef npy_static_string *unpacked = info[0].unpacked + info[0].i
+    # Obtain a reference to the string (NOT zero-terminated) and its size
+    res = NpyString_load(info[0].allocator, <npy_packed_static_string*>elem, unpacked)
+    info[0].i += 1
+    # res == -1 if unpacking the string fails, 1 if packed_string is the null string,
+    # and 0 otherwise.
+    return -1 if res == -1 else 0
+
+
+def npystrings_unpack(hid_t space, size_t contig, size_t noncontig, size_t descr,
+                      size_t npoints):
+    """Convert a NpyString array (NumPy's native variable-width strings dtype)
+    to a zero-terminated char**, which is the in-memory representation for a
+    HDF5 variable-width string dataset.
+
+    Parameters
+    ----------
+    space : hid_t
+        HDF5 dataspace ID for the input non-contiguous buffer
+        containing packed NpyStrings
+    contig : void * (actually a char **)
+        Output buffer for the contiguous zero-terminated char **
+    noncontig : void *
+        Input buffer for the non-contiguous packed NpyStrings
+    descr : PyArray_Descr * (actually a PyArray_StringDTypeObject *)
+        NumPy dtype descriptor for the NpyStrings
+    npoints : size_t
+        Number of points to copy
+
+    Returns
+    -------
+    char *
+        A temporary buffer containing the zero-terminated strings.
+        This buffer must be freed after writing to disk.
+
+        Note that this always the same as contig[0] when this
+        function returns, but H5Tconvert may change the contents
+        of contig[0] in place later on.
+
+    Notes
+    -----
+    This function defines a pure-python API interface to _proxy.pyx.
+    This is necessary to allow this module to be conditionally imported,
+    allowing NumPy 1.x to continue working everywhere else.
+    """
+    return <size_t>_npystrings_unpack(space, <void *>contig, <void *>noncontig,
+                                      <PyArray_Descr *>descr, npoints)
+
+
+cdef char * _npystrings_unpack(hid_t space, void *contig, void *noncontig,
+                               PyArray_Descr *descr, size_t npoints):
+    """Cython API interface of npystrings_unpack"""
+    cdef npystrings_unpack_t info
+    cdef size_t total_size
+    cdef size_t cur_size
+    cdef char *zero_terminated = NULL
+    cdef char *zero_terminated_cur = NULL
+    info.unpacked = NULL
+    info.i = 0
+
+    info.allocator = NpyString_acquire_allocator(<PyArray_StringDTypeObject *>descr)
+    if info.allocator is NULL:
+        raise RuntimeError("Failed to acquire string allocator")
+
+    tid = H5Tcreate(H5T_OPAQUE, SIZEOF_NPY_PACKED_STATIC_STRING)
+    try:
+        # Multiple steps needed:
+        # 1. Read npy_packed_static_string[] from NumPy and unpack
+        #    to npy_static_string[]; which is
+        #    {const char* buf, size_t size}[] - NOT zero-terminated
+        info.unpacked = <npy_static_string*>malloc(
+            npoints * sizeof(npy_static_string)
+        )
+        if info.unpacked is NULL:
+            raise MemoryError("Failed to allocate memory")
path: h5py/_npystrings.pyx
user: login: takluyver
body: We have an `emalloc()` wrapper in `utils.pyx` that calls includes the error handling, if you want to save a few lines.
created_at: 2025-05-21T11:51:45Z
author_association: MEMBER

PR comments:
user: login: codecov[bot]
created_at: 2025-02-11T18:23:06Z
body: ## [Codecov](https://app.codecov.io/gh/h5py/h5py/pull/2557?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=h5py) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 90.39%. Comparing base ([`1e28b50`](https://app.codecov.io/gh/h5py/h5py/commit/1e28b507a31d73fc1ed020174503454aece51229?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=h5py)) to head ([`2e07239`](https://app.codecov.io/gh/h5py/h5py/commit/2e07239a1bfaa8e1bab7962f4313523d5b6ab56b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=h5py)).
:warning: Report is 136 commits behind head on master.

<details><summary>Additional details and impacted files</summary>



```diff
@@            Coverage Diff             @@
##           master    #2557      +/-   ##
==========================================
+ Coverage   90.36%   90.39%   +0.03%     
==========================================
  Files          16       16              
  Lines        2438     2447       +9     
==========================================
+ Hits         2203     2212       +9     
  Misses        235      235              
```
</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/h5py/h5py/pull/2557?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=h5py).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=h5py).
<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: crusaderky
created_at: 2025-03-07T16:46:51Z
body: All CI failures are codecov crashes and unrelated with this PR.
@takluyver this is ready for review.
author_association: CONTRIBUTOR
user: login: crusaderky
created_at: 2025-03-26T14:59:10Z
body: @takluyver gentle ping 
author_association: CONTRIBUTOR
user: login: ngoldbaum
created_at: 2025-04-04T17:08:13Z
body: Hi all, I'm going to try to give this PR a thorough once-over sometime next week. I'm actually just learning today that Guido has been working on this and I'm really excited to see `StringDType` getting integrated into h5py.
author_association: CONTRIBUTOR
user: login: takluyver
created_at: 2025-04-07T10:52:16Z
body: Thanks @crusaderky,  I'm just starting to look at this.

First question, I was expecting that this would add conversion machinery in `_conv.pyx`. E.g. this is where we already have conversions between HDF5 variable-length data and numpy object arrays. Did you explore this possibility and decide that there was some reason it couldn't work?

The `py_create()` function to convert a numpy dtype to an HDF5 datatype has a parameter to distinguish between exact and 'logical' conversion:

> when this flag is set, instead of returning a byte-for-byte identical representation of the type, the function returns the closest logically appropriate HDF5 type.  For example, in the case of a "hinted" dtype of kind "O" representing a string, it would return an HDF5 variable-length string type.

If logical is not set, for object arrays it will always return the special `PYTHON_OBJECT` datatype. This should never* appear in an actual file, so it always needs a conversion on read/write. I would hope it's possible to work with Numpy strings in a similar fashion by making a new `NUMPY_STRINGS` opaque datatype object.

(*it's probably not impossible to make a dataset with this datatype and write useless pointers directly into it. But hopefully it will only happen if you really try to make that happen)
author_association: MEMBER
user: login: crusaderky
created_at: 2025-04-07T13:32:04Z
body: > Thanks @crusaderky, I'm just starting to look at this.
> 
> First question, I was expecting that this would add conversion machinery in `_conv.pyx`. E.g. this is where we already have conversions between HDF5 variable-length data and numpy object arrays. Did you explore this possibility and decide that there was some reason it couldn't work?

The NpyStrings machinery must be in its own module, separated by a pure-Python API interface, otherwise you will no longer be able to import h5py when NumPy 1.x is installed.

I haven't tried writing a bespoke clone of `_conv.pyx` instead of attaching to `_proxy.pyx`.
I would expect some pretty major snags though. Namely, I'm reading `ndarray2vlen` and observing that it calls `emalloc` once _per point_ being written. That's very inefficient for temporary objects that are going to be released immediately after the write. I coded the NpyStrings write in a way that allocates a single temporary buffer (see `_proxy.pyx:164:173`).

author_association: CONTRIBUTOR
user: login: crusaderky
created_at: 2025-04-07T13:41:34Z
body: Another major snag (possibly a showstopper?) is that NpyStrings need to access the `PyArray_Descr` from the input/output ndarray. By the time we're deep inside HDF5 machinery I don't think it's retrievable?
author_association: CONTRIBUTOR
user: login: takluyver
created_at: 2025-04-07T14:15:11Z
body: > otherwise you will no longer be able to import h5py when NumPy 1.x is installed.

How much easier would this work be without that requirement? All else being equal, I would probably keep supporting Numpy 1.x for a while, but I also have the impression that the Numpy 2.x release has gone pretty well. If this can be substantially simplified by dropping 1.x support - or demoting it to 'if you build from source' - then we can weigh that up. We will probably be supporting this for Numpy 2.x for much longer than we're trying to support 1.x and 2.x.

> I'm reading ndarray2vlen and observing that it calls emalloc once per point being written

I don't think that one is insurmountable; HDF5 calls the conversion function for multiple elements at a time.

> NpyStrings need to access the PyArray_Descr from the input/output ndarray. By the time we're deep inside HDF5 machinery I don't think it's retrievable?

That one might be insurmountable, though. I'll take a look.

In any case, I think that `py_create` with `logical=False` shouldn't say that the new numpy string dtype is equivalent to HDF5 variable length strings. That is meant to mean that the data type is "byte-for-byte memory-compatible", i.e. no conversion is needed. We can either have an opaque type like we do for object arrays, or simply raise an error that there's no matching HDF5 datatype.
author_association: MEMBER
user: login: crusaderky
created_at: 2025-04-07T14:39:47Z
body: > How much easier would this work be without that requirement? All else being equal, I would probably keep supporting Numpy 1.x for a while, but I also have the impression that the Numpy 2.x release has gone pretty well. If this can be substantially simplified by dropping 1.x support - or demoting it to 'if you build from source' - then we can weigh that up. We will probably be supporting this for Numpy 2.x for much longer than we're trying to support 1.x and 2.x.

While SPEC0 states that NumPy 1 should exit is maintenance window in Q3 2025, I expect that *a lot* of people will become irate if you upgrade your minimum dependencies that quickly.
On the other hand, this is a killer feature for anybody working with strings, and limiting it to only people that build from sources (which is not an easy task particularly for less-than-expert users on windows) would severely impact its availability.

All in all I don't think that the solution I came up with is terrible in terms of readability/maintainability. Things would _not_ become particularly simpler if I could use the C interface.

I need to add, cleaning up the pure-Python API interface e.g. ~1 year from now would be a very straightforward exercise.

author_association: CONTRIBUTOR
user: login: crusaderky
created_at: 2025-04-22T16:45:52Z
body: @takluyver can I have some feedback please?
The points you are dissatisfied with are critical to the end user's UI, so I need to know if (and how _specifically_) you want the end-user UI to change in order to accommodate the technical changes you are asking for.

- **Option 1:** as it is now
- **Option 2:** `create_dataset(..., dtype="T")` becomes disallowed; the user needs to call `create_dataset(..., dtype=h5py.stringdtype()).astype("T")` if they want to _read_ from a freshly created dataset. `AsTypeView` becomes writeable.
- **Option 3:** `create_dataset(..., dtype="T")` returns a `AsTypeView`. `AsTypeView` becomes writeable.
- **Option 4:** ?

author_association: CONTRIBUTOR
user: login: crusaderky
created_at: 2025-04-28T09:26:11Z
body: @takluyver gentle ping
author_association: CONTRIBUTOR
user: login: takluyver
created_at: 2025-04-30T10:16:28Z
body: Sorry for the delay. I would like to avoid h5py's Dataset object having mutable state (except what is stored in the file, of course). I don't think it's a good design for a dataset newly created by specifying dtype "T" to behave differently to that same dataset when reopening the file.

I think logically the writing behaviour should not be an issue, because Numpy's variable length string dtype can only map to 1 HDF5 datatype, so any time we're trying to write data with this dtype, we should convert it to HDF5 variable-length strings. I don't believe we need to mutate the Dataset object to enable this.

Reading is a bit more complicated, because we want object arrays to remain the default (for now), but it would defeat the point to construct object arrays and then convert them to dtype "T" afterwards. But I think this is still possible without needing mutable state.

I'm going to check this branch out and play around a bit.
author_association: MEMBER
user: login: takluyver
created_at: 2025-04-30T14:14:29Z
body: I have opened https://github.com/crusaderky/h5py/pull/1 building on this in the direction that I'm describing. I've also opened a draft PR here (#2574) from the same branch in order to run the CI.
author_association: MEMBER
user: login: takluyver
created_at: 2025-05-09T17:16:41Z
body: I think we wouldn't be able to use a compound dtype containing T strings, because this is outside the more general conversion machinery, but I don't think there's a good way to solve this, given the need for the dtype object to pack & load strings. Numpy doesn't allow T within a compound dtype either, at least for now:

```text
In [35]: dt = np.dtype([('num', 'u2'), ('text', 'T')])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[35], line 1
----> 1 dt = np.dtype([('num', 'u2'), ('text', 'T')])

TypeError: StringDType is not currently supported for structured dtype fields.
```

We could also support reading & writing 'T' dtype data to attributes - I don't think there's any particular challenge for that, just some additional code to write. But attributes are also meant to be small - often just a single string - so it shouldn't be a big problem to convert to & from object arrays. So I think we can get the new functionality for datasets in & released and see how it goes in practice.
author_association: MEMBER
user: login: crusaderky
created_at: 2025-05-16T16:12:34Z
body: @takluyver I went through the documentation and everything else once more.
This is ready for final review and merge if there are no further comments.

author_association: CONTRIBUTOR
user: login: takluyver
created_at: 2025-05-21T12:00:24Z
body: Thanks Guido! I think this is looking good - once we adjust the bit saying 'zero cost' in the docs I'm happy to merge it. I'll aim to do a new release fairly soon so people can easily start trying this.
author_association: MEMBER
user: login: ngoldbaum
created_at: 2025-05-21T12:12:25Z
body: Thanks to you both for handling this. It's really awesome to see the ecosystem taking up support for StringDType. This is especially cool because StringDType is a "new-style" dtype that uses the infrastructure @seberg, @stefanv, and others got funded, implemented, and eventually shipped in NumPy 2.0.

If you notice bug reports related to this feature that look like bugs in NumPy or issues related to the use of new-style dtypes please reach out.
author_association: CONTRIBUTOR
user: login: crusaderky
created_at: 2025-05-23T08:04:24Z
body: @takluyver implemented suggestions
author_association: CONTRIBUTOR
user: login: takluyver
created_at: 2025-05-23T13:27:15Z
body: Thank you!
author_association: MEMBER
user: login: takluyver
created_at: 2025-06-06T14:38:41Z
body: This is now out in 3.14, so it's easy for people to start trying it.
author_association: MEMBER