Repository: single-cell-data/TileDB-SOMA (language: python)

Issue:
[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now



Discussion / hints:
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.

Fix patch:
diff --git a/libtiledbsoma/src/utils/arrow_adapter.cc b/libtiledbsoma/src/utils/arrow_adapter.cc
index f456b94647..17bda5582f 100644
--- a/libtiledbsoma/src/utils/arrow_adapter.cc
+++ b/libtiledbsoma/src/utils/arrow_adapter.cc
@@ -1252,4 +1252,61 @@ enum ArrowType ArrowAdapter::to_nanoarrow_type(std::string_view sv) {
             fmt::format("ArrowAdapter: Unsupported Arrow format: {} ", sv));
 }
 
+std::unique_ptr<ArrowSchema> ArrowAdapter::make_arrow_schema(
+    const std::vector<std::string>& names,
+    const std::vector<tiledb_datatype_t>& tiledb_datatypes) {
+    auto num_names = names.size();
+    auto num_types = tiledb_datatypes.size();
+
+    if (num_names != num_types) {
+        throw TileDBSOMAError(fmt::format(
+            "ArrowAdapter::make_arrow_schema: internal coding error {} != {}",
+            num_names,
+            num_types));
+    }
+
+    auto arrow_schema = std::make_unique<ArrowSchema>();
+    arrow_schema->format = "+s";           // structure, i.e. non-leaf node
+    arrow_schema->n_children = num_names;  // non-leaf node
+    arrow_schema->dictionary = nullptr;
+    arrow_schema->release = &ArrowAdapter::release_schema;
+    arrow_schema->children = new ArrowSchema*[arrow_schema->n_children];
+
+    for (int i = 0; i < (int)num_names; i++) {
+        ArrowSchema* dim_schema = new ArrowSchema;
+        dim_schema->name = strdup(names[i].c_str());
+        auto arrow_type_name = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatypes[i]);
+        dim_schema->format = strdup(arrow_type_name.c_str());
+        dim_schema->n_children = 0;  // leaf node
+        dim_schema->dictionary = nullptr;
+        dim_schema->release = &ArrowAdapter::release_schema;
+        arrow_schema->children[i] = dim_schema;
+    }
+
+    return arrow_schema;
+}
+
+std::unique_ptr<ArrowArray> ArrowAdapter::make_arrow_array_parent(
+    int num_columns) {
+    auto arrow_array = std::make_unique<ArrowArray>();
+
+    // All zero/null since this is a parent ArrowArray, and each
+    // column/child is also of type ArrowArray.
+    arrow_array->length = 0;
+    arrow_array->null_count = 0;
+    arrow_array->offset = 0;
+    arrow_array->n_buffers = 0;
+    arrow_array->buffers = nullptr;
+
+    arrow_array->n_children = num_columns;
+    arrow_array->release = &ArrowAdapter::release_array;
+    arrow_array->children = new ArrowArray*[num_columns];
+    for (int i = 0; i < num_columns; i++) {
+        arrow_array->children[i] = nullptr;
+    }
+
+    return arrow_array;
+}
+
 }  // namespace tiledbsoma
diff --git a/libtiledbsoma/src/utils/arrow_adapter.h b/libtiledbsoma/src/utils/arrow_adapter.h
index 6802f0dc39..bff81ba920 100644
--- a/libtiledbsoma/src/utils/arrow_adapter.h
+++ b/libtiledbsoma/src/utils/arrow_adapter.h
@@ -164,6 +164,21 @@ class PlatformConfig {
     bool consolidate_and_vacuum = false;
 };
 
+/**
+ * This is our application-specific wrapper around nanoarrow.
+ *
+ * Nominal use-cases include full dataframe-wide schemas from Python/R for
+ * DataFrame::create, domain/extent/etc information from Python/R for
+ * setting core dimensions, and readback to Python/R of core domain, current
+ * domain, and non-empty domain.
+ *
+ * nanoarrow is crucial to our code-centralization efforts in libtiledbsoma.
+ * Python and R can trivially do heterogenous lists like ["abc", 123, true]
+ * or list("abc", 123, TRUE); C++ cannot -- at least not so compactly. The
+ * combination of ArrowSchema (for the types) and ArrowArray (for the data) is
+ * key for Python/R interop with C++ libtiledbsoma.
+ */
+
 class ArrowAdapter {
    public:
     static void release_schema(struct ArrowSchema* schema);
@@ -229,6 +244,95 @@ class ArrowAdapter {
 
     static enum ArrowType to_nanoarrow_type(std::string_view sv);
 
+    /**
+     * @brief This is a keystroke-saver.
+     */
+    static std::string tdb_to_arrow_type(tiledb_datatype_t tiledb_datatype) {
+        return std::string(to_arrow_format(tiledb_datatype));
+    }
+
+    /**
+     * @brief Creates a nanoarrow ArrowSchema given the names and TileDB
+     * datatypes.
+     *
+     * Note that the parents and children in nanoarrow are both of type
+     * ArrowSchema. This constructs the parent and the children.
+     */
+    static std::unique_ptr<ArrowSchema> make_arrow_schema(
+        const std::vector<std::string>& names,
+        const std::vector<tiledb_datatype_t>& tiledb_datatypes);
+
+    /**
+     * @brief Creates a nanoarrow ArrowArray which accommodates
+     * a varying number of columns.
+     *
+     * Note that the parents and children in nanoarrow are both of type
+     * ArrowArray. This constructs the parent and not the children.
+     */
+    static std::unique_ptr<ArrowArray> make_arrow_array_parent(int num_columns);
+
+    /**
+     * @brief Creates a nanoarrow ArrowArray for a single column.
+     *
+     * Note that the parents and children in nanoarrow are both of type
+     * ArrowArray. This constructs not the parent but rather a child.
+     */
+    template <typename T>
+    static ArrowArray* make_arrow_array_child(const std::pair<T, T>& pair) {
+        std::vector<T> v({pair.first, pair.second});
+        return make_arrow_array_child<T>(v);
+    };
+
+    template <typename T>
+    static ArrowArray* make_arrow_array_child(const std::vector<T>& v) {
+        // Use new here, not malloc, to match ArrowAdapter::release_array
+        auto arrow_array = new ArrowArray;
+
+        size_t n = v.size();
+
+        arrow_array->length = n;
+        arrow_array->null_count = 0;
+        arrow_array->offset = 0;
+        arrow_array->n_buffers = 2;
+        arrow_array->release = &ArrowAdapter::release_array;
+        arrow_array->buffers = new const void*[2];
+        arrow_array->n_children = 0;  // leaf/child node
+
+        arrow_array->buffers[0] = nullptr;
+        // Use malloc here, not new, to match ArrowAdapter::release_array
+        T* dest = (T*)malloc(n * sizeof(T));
+        for (size_t i = 0; i < n; i++) {
+            dest[i] = v[i];
+        }
+        arrow_array->buffers[1] = (void*)dest;
+
+        return arrow_array;
+    };
+
+    static ArrowArray* make_arrow_array_child(
+        const std::vector<std::string>& v) {
+        // Use new here, not malloc, to match ArrowAdapter::release_array
+        auto arrow_array = new ArrowArray;
+
+        size_t n = v.size();
+
+        arrow_array->length = n;
+        arrow_array->null_count = 0;
+        arrow_array->offset = 0;
+        arrow_array->n_buffers = 2;
+        arrow_array->release = &ArrowAdapter::release_array;
+        arrow_array->buffers = new const void*[2];
+        arrow_array->n_children = 0;  // leaf/child node
+
+        // For core domain, these are always nullptr for strings and cannot be
+        // anything else. More general use of this class is WIP on
+        // https://github.com/single-cell-data/TileDB-SOMA/issues/2407
+        arrow_array->buffers[0] = nullptr;
+        arrow_array->buffers[1] = nullptr;
+
+        return arrow_array;
+    };
+
    private:
     static std::pair<const void*, std::size_t> _get_data_and_length(
         Enumeration& enmr, const void* dst);
@@ -290,7 +394,8 @@ class ArrowAdapter {
         Filter filter, std::string option_name, json value);
 
     static tiledb_layout_t _get_order(std::string order);
-};
-};  // namespace tiledbsoma
 
+};  // class ArrowAdapter
+
+};  // namespace tiledbsoma
 #endif


Test patch:
diff --git a/libtiledbsoma/test/common.cc b/libtiledbsoma/test/common.cc
index 3fdecfc76f..8e291662ef 100644
--- a/libtiledbsoma/test/common.cc
+++ b/libtiledbsoma/test/common.cc
@@ -47,8 +47,6 @@ static std::unique_ptr<ArrowSchema> _create_index_cols_info_schema(
 static std::unique_ptr<ArrowArray> _create_index_cols_info_array(
     const std::vector<DimInfo>& dim_infos);
 
-static std::string tdb_to_arrow_type(tiledb_datatype_t tiledb_dtype);
-
 // Notes:
 //
 // * This is multi-purpose code used for generic SOMASparseNDArray,
@@ -91,36 +89,22 @@ create_arrow_schema_and_index_columns(
     const std::vector<AttrInfo>& attr_infos) {
     int ndim = dim_infos.size();
     int nattr = attr_infos.size();
-
-    auto arrow_schema = std::make_unique<ArrowSchema>();
-    arrow_schema->format = "+s";
-    arrow_schema->n_children = ndim + nattr;  // non-leaf node
-    arrow_schema->dictionary = nullptr;
-    arrow_schema->release = &ArrowAdapter::release_schema;
-    arrow_schema->children = new ArrowSchema*[arrow_schema->n_children];
-
-    for (int i = 0; i < ndim; i++) {
-        const DimInfo& info = dim_infos[i];
-        ArrowSchema* dim = new ArrowSchema;
-        dim->name = strdup(info.name.c_str());
-        dim->format = strdup(tdb_to_arrow_type(info.tiledb_datatype).c_str());
-        dim->n_children = 0;  // leaf node
-        dim->dictionary = nullptr;
-        dim->release = &ArrowAdapter::release_schema;
-        arrow_schema->children[i] = dim;
+    int nfield = ndim + nattr;
+
+    std::vector<std::string> names(nfield);
+    std::vector<tiledb_datatype_t> tiledb_datatypes(nfield);
+    for (int i = 0; i < (int)ndim; i++) {
+        const DimInfo& dim_info = dim_infos[i];
+        names[i] = dim_info.name;
+        tiledb_datatypes[i] = dim_info.tiledb_datatype;
     }
-
-    for (int i = 0; i < nattr; i++) {
-        const AttrInfo& info = attr_infos[i];
-        ArrowSchema* attr = new ArrowSchema;
-        attr->name = strdup(info.name.c_str());
-        attr->format = strdup(tdb_to_arrow_type(info.tiledb_datatype).c_str());
-        attr->n_children = 0;  // leaf node
-        attr->flags = 0;
-        attr->dictionary = nullptr;
-        attr->release = &ArrowAdapter::release_schema;
-        arrow_schema->children[ndim + i] = attr;
+    for (int i = 0; i < (int)nattr; i++) {
+        const AttrInfo& attr_info = attr_infos[i];
+        names[ndim + i] = attr_info.name;
+        tiledb_datatypes[ndim + i] = attr_info.tiledb_datatype;
     }
+    auto arrow_schema = ArrowAdapter::make_arrow_schema(
+        names, tiledb_datatypes);
 
     auto index_cols_info_schema = _create_index_cols_info_schema(dim_infos);
     auto index_cols_info_array = _create_index_cols_info_array(dim_infos);
@@ -152,119 +136,85 @@ ArrowTable create_column_index_info(const std::vector<DimInfo>& dim_infos) {
 
 static std::unique_ptr<ArrowSchema> _create_index_cols_info_schema(
     const std::vector<DimInfo>& dim_infos) {
-    int ndim = dim_infos.size();
+    auto ndim = dim_infos.size();
 
-    auto index_cols_info_schema = std::make_unique<ArrowSchema>();
-    index_cols_info_schema->format = "+s";
-    index_cols_info_schema->n_children = ndim;  // non-leaf node
-    index_cols_info_schema->dictionary = nullptr;
-    index_cols_info_schema->release = &ArrowAdapter::release_schema;
-    index_cols_info_schema
-        ->children = new ArrowSchema*[index_cols_info_schema->n_children];
+    std::vector<std::string> names(ndim);
+    std::vector<tiledb_datatype_t> tiledb_datatypes(ndim);
 
-    for (int i = 0; i < ndim; i++) {
-        const DimInfo& info = dim_infos[i];
-        ArrowSchema* dim_schema = new ArrowSchema;
-        dim_schema->format = strdup(
-            tdb_to_arrow_type(info.tiledb_datatype).c_str());
-        dim_schema->name = strdup(info.name.c_str());
-        dim_schema->n_children = 0;  // leaf node
-        dim_schema->dictionary = nullptr;
-        dim_schema->release = &ArrowAdapter::release_schema;
-        index_cols_info_schema->children[i] = dim_schema;
+    for (int i = 0; i < (int)ndim; i++) {
+        const DimInfo& dim_info = dim_infos[i];
+        names[i] = dim_info.name;
+        tiledb_datatypes[i] = dim_info.tiledb_datatype;
     }
 
-    return index_cols_info_schema;
+    return ArrowAdapter::make_arrow_schema(names, tiledb_datatypes);
 }
 
 static std::unique_ptr<ArrowArray> _create_index_cols_info_array(
     const std::vector<DimInfo>& dim_infos) {
     int ndim = dim_infos.size();
 
-    auto index_cols_info_array = std::make_unique<ArrowArray>();
-    index_cols_info_array->length = 0;
-    index_cols_info_array->null_count = 0;
-    index_cols_info_array->offset = 0;
-    index_cols_info_array->n_buffers = 0;
-    index_cols_info_array->buffers = nullptr;
-    index_cols_info_array->n_children = ndim;
-    index_cols_info_array->release = &ArrowAdapter::release_array;
-    index_cols_info_array->children = new ArrowArray*[ndim];
+    auto index_cols_info_array = ArrowAdapter::make_arrow_array_parent(ndim);
 
     for (int i = 0; i < ndim; i++) {
         const DimInfo& info = dim_infos[i];
-
-        int n = info.use_current_domain ? 5 : 3;
-
-        auto dim_array = new ArrowArray;
-        dim_array->length = n;
-        dim_array->null_count = 0;
-        dim_array->offset = 0;
-        dim_array->n_buffers = 2;
-        dim_array->release = &ArrowAdapter::release_array;
-        dim_array->buffers = new const void*[2];
-        dim_array->buffers[0] = nullptr;
-        dim_array->n_children = 0;  // leaf node
-        index_cols_info_array->children[i] = dim_array;
+        ArrowArray* dim_array = nullptr;
 
         // The full user-level SOMA API supports many more index types.
         // Here we support enough types to verify we've got variant-indexed
         // SOMADataFrame objects baseline-tested in C++, then defer exhaustive
         // loop-over-all-datatypes handling to Python and R.
         if (info.tiledb_datatype == TILEDB_INT64) {
-            size_t nbytes = n * sizeof(int64_t);
-            dim_array->buffers[1] = malloc(nbytes);
             if (info.use_current_domain) {
                 // domain big; current_domain small
-                int64_t dom[] = {0, CORE_DOMAIN_MAX, 1, 0, info.dim_max};
-                void* vsrc = (void*)&dom[0];
-                std::memcpy((void*)dim_array->buffers[1], vsrc, nbytes);
+                std::vector<int64_t> dom(
+                    {0, CORE_DOMAIN_MAX, 1, 0, info.dim_max});
+                dim_array = ArrowAdapter::make_arrow_array_child(dom);
             } else {
                 // domain small; current_domain feature not being used
-                int64_t dom[] = {0, info.dim_max, 1};
-                void* vsrc = (void*)&dom[0];
-                std::memcpy((void*)dim_array->buffers[1], vsrc, nbytes);
+                std::vector<int64_t> dom({0, info.dim_max, 1});
+                dim_array = ArrowAdapter::make_arrow_array_child(dom);
             }
 
         } else if (info.tiledb_datatype == TILEDB_UINT32) {
-            size_t nbytes = n * sizeof(uint32_t);
-            dim_array->buffers[1] = malloc(nbytes);
             if (info.use_current_domain) {
                 // domain big; current_domain small
-                uint32_t dom[] = {
-                    0, (uint32_t)CORE_DOMAIN_MAX, 1, 0, (uint32_t)info.dim_max};
-                void* vsrc = (void*)&dom[0];
-                std::memcpy((void*)dim_array->buffers[1], vsrc, nbytes);
+                std::vector<uint32_t> dom(
+                    {0,
+                     (uint32_t)CORE_DOMAIN_MAX,
+                     1,
+                     0,
+                     (uint32_t)info.dim_max});
+                dim_array = ArrowAdapter::make_arrow_array_child(dom);
             } else {
                 // domain small; current_domain feature not being used
-                uint32_t dom[] = {0, (uint32_t)info.dim_max, 1};
-                void* vsrc = (void*)&dom[0];
-                std::memcpy((void*)dim_array->buffers[1], vsrc, nbytes);
+                std::vector<uint32_t> dom({0, (uint32_t)info.dim_max, 1});
+                dim_array = ArrowAdapter::make_arrow_array_child(dom);
             }
 
         } else if (info.tiledb_datatype == TILEDB_STRING_ASCII) {
             // Domain specification for strings is not supported in core. See
             // arrow_adapter for more info. We rely on arrow_adapter to also
             // handle this case.
-            dim_array->buffers[1] = nullptr;
+            if (info.use_current_domain) {
+                std::vector<std::string> dom({"", "", "", "", ""});
+                dim_array = ArrowAdapter::make_arrow_array_child(dom);
+            } else {
+                std::vector<std::string> dom({"", "", ""});
+                dim_array = ArrowAdapter::make_arrow_array_child(dom);
+            }
+        }
 
-        } else {
+        if (dim_array == nullptr) {
             throw TileDBSOMAError(
                 "Unsupported datatype encountered in unit test. You can add a "
                 "new type if you like!");
         }
+
+        index_cols_info_array->children[i] = dim_array;
     }
 
     return index_cols_info_array;
 }
 
-// Just a keystroke-saver
-static std::string tdb_to_arrow_type(tiledb_datatype_t tiledb_datatype) {
-    return std::string(ArrowAdapter::to_arrow_format(tiledb_datatype));
-}
-
-std::string to_arrow_format(tiledb_datatype_t tiledb_datatype) {
-    return std::string(ArrowAdapter::to_arrow_format(tiledb_datatype));
-}
-
 }  // namespace helper
diff --git a/libtiledbsoma/test/unit_soma_array.cc b/libtiledbsoma/test/unit_soma_array.cc
index 29ee295596..6901210d41 100644
--- a/libtiledbsoma/test/unit_soma_array.cc
+++ b/libtiledbsoma/test/unit_soma_array.cc
@@ -529,7 +529,8 @@ TEST_CASE("SOMAArray: Write and read back Boolean") {
     arrow_schema->release = &ArrowAdapter::release_schema;
     arrow_schema->children = new ArrowSchema*[arrow_schema->n_children];
     ArrowSchema* arrow_dim = arrow_schema->children[0] = new ArrowSchema;
-    arrow_dim->format = strdup(helper::to_arrow_format(TILEDB_INT64).c_str());
+    arrow_dim->format = strdup(
+        ArrowAdapter::tdb_to_arrow_type(TILEDB_INT64).c_str());
     arrow_dim->name = dim_name;
     arrow_dim->n_children = 0;
     arrow_dim->dictionary = nullptr;
diff --git a/libtiledbsoma/test/unit_soma_collection.cc b/libtiledbsoma/test/unit_soma_collection.cc
index 42309e1d7b..f0097ae794 100644
--- a/libtiledbsoma/test/unit_soma_collection.cc
+++ b/libtiledbsoma/test/unit_soma_collection.cc
@@ -62,7 +62,8 @@ TEST_CASE("SOMACollection: add SOMASparseNDArray") {
         std::string sub_uri = "mem://unit-test-add-sparse-ndarray/sub";
         std::string dim_name = "soma_dim_0";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         SOMACollection::create(base_uri, ctx, ts);
 
@@ -114,7 +115,7 @@ TEST_CASE("SOMACollection: add SOMADenseNDArray") {
     std::string sub_uri = "mem://unit-test-add-dense-ndarray/sub";
     std::string dim_name = "soma_dim_0";
     tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-    std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+    std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(tiledb_datatype);
 
     SOMACollection::create(base_uri, ctx, ts);
     // TODO: add support for current domain in dense arrays once we have that
@@ -169,7 +170,8 @@ TEST_CASE("SOMACollection: add SOMADataFrame") {
         std::string dim_name = "d0";
         std::string attr_name = "a0";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         SOMACollection::create(base_uri, ctx, ts);
 
@@ -227,7 +229,8 @@ TEST_CASE("SOMACollection: add SOMACollection") {
         std::string base_uri = "mem://unit-test-add-collection";
         std::string sub_uri = "mem://unit-test-add-collection/sub";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         SOMACollection::create(base_uri, ctx);
 
@@ -261,7 +264,8 @@ TEST_CASE("SOMACollection: add SOMAExperiment") {
         std::string dim_name = "d0";
         std::string attr_name = "a0";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         SOMACollection::create(base_uri, ctx);
 
@@ -314,7 +318,8 @@ TEST_CASE("SOMACollection: add SOMAMeasurement") {
         std::string dim_name = "d0";
         std::string attr_name = "a0";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         SOMACollection::create(base_uri, ctx);
 
@@ -428,7 +433,8 @@ TEST_CASE("SOMAExperiment: metadata") {
         std::string dim_name = "soma_dim_0";
         std::string attr_name = "soma_data";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         std::vector<helper::DimInfo> dim_infos(
             {{.name = dim_name,
@@ -519,7 +525,8 @@ TEST_CASE("SOMAMeasurement: metadata") {
         std::string dim_name = "soma_dim_0";
         std::string attr_name = "soma_data";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         std::vector<helper::DimInfo> dim_infos(
             {{.name = dim_name,
diff --git a/libtiledbsoma/test/unit_soma_dataframe.cc b/libtiledbsoma/test/unit_soma_dataframe.cc
index 64d998d6b3..1c1388f7a9 100644
--- a/libtiledbsoma/test/unit_soma_dataframe.cc
+++ b/libtiledbsoma/test/unit_soma_dataframe.cc
@@ -67,9 +67,12 @@ struct VariouslyIndexedDataFrameFixture {
     tiledb_datatype_t u32_datatype = TILEDB_UINT32;
     tiledb_datatype_t str_datatype = TILEDB_STRING_ASCII;
 
-    std::string i64_arrow_format = helper::to_arrow_format(i64_datatype);
-    std::string u32_arrow_format = helper::to_arrow_format(u32_datatype);
-    std::string attr_1_arrow_format = helper::to_arrow_format(str_datatype);
+    std::string i64_arrow_format = ArrowAdapter::tdb_to_arrow_type(
+        i64_datatype);
+    std::string u32_arrow_format = ArrowAdapter::tdb_to_arrow_type(
+        u32_datatype);
+    std::string attr_1_arrow_format = ArrowAdapter::tdb_to_arrow_type(
+        str_datatype);
 
     helper::DimInfo i64_dim_info(bool use_current_domain) {
         return helper::DimInfo(
diff --git a/libtiledbsoma/test/unit_soma_dense_ndarray.cc b/libtiledbsoma/test/unit_soma_dense_ndarray.cc
index 9e8d011740..150e6464d6 100644
--- a/libtiledbsoma/test/unit_soma_dense_ndarray.cc
+++ b/libtiledbsoma/test/unit_soma_dense_ndarray.cc
@@ -47,7 +47,8 @@ TEST_CASE("SOMADenseNDArray: basic", "[SOMADenseNDArray]") {
         std::string uri = "mem://unit-test-dense-ndarray-basic";
         std::string dim_name = "soma_dim_0";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         REQUIRE(!SOMADenseNDArray::exists(uri, ctx));
 
@@ -157,7 +158,8 @@ TEST_CASE("SOMADenseNDArray: platform_config", "[SOMADenseNDArray]") {
         std::string uri = "mem://unit-test-dense-ndarray-platform-config";
         std::string dim_name = "soma_dim_0";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         PlatformConfig platform_config;
         platform_config.dense_nd_array_dim_zstd_level = 6;
@@ -221,7 +223,8 @@ TEST_CASE("SOMADenseNDArray: metadata", "[SOMADenseNDArray]") {
         std::string uri = "mem://unit-test-dense-ndarray";
         std::string dim_name = "soma_dim_0";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         std::vector<helper::DimInfo> dim_infos(
             {{.name = dim_name,
diff --git a/libtiledbsoma/test/unit_soma_sparse_ndarray.cc b/libtiledbsoma/test/unit_soma_sparse_ndarray.cc
index 65e1a94005..df66f1f42c 100644
--- a/libtiledbsoma/test/unit_soma_sparse_ndarray.cc
+++ b/libtiledbsoma/test/unit_soma_sparse_ndarray.cc
@@ -47,7 +47,8 @@ TEST_CASE("SOMASparseNDArray: basic", "[SOMASparseNDArray]") {
         std::string dim_name = "soma_dim_0";
         std::string attr_name = "soma_data";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         REQUIRE(!SOMASparseNDArray::exists(uri, ctx));
 
@@ -182,7 +183,8 @@ TEST_CASE("SOMASparseNDArray: platform_config", "[SOMASparseNDArray]") {
         std::string uri = "mem://unit-test-dataframe-platform-config";
         std::string dim_name = "soma_dim_0";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         PlatformConfig platform_config;
         platform_config.sparse_nd_array_dim_zstd_level = 6;
@@ -231,7 +233,8 @@ TEST_CASE("SOMASparseNDArray: metadata", "[SOMASparseNDArray]") {
         std::string uri = "mem://unit-test-sparse-ndarray";
         std::string dim_name = "soma_dim_0";
         tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        std::string arrow_format = ArrowAdapter::tdb_to_arrow_type(
+            tiledb_datatype);
 
         std::vector<helper::DimInfo> dim_infos(
             {{.name = dim_name,