Repository: single-cell-data/TileDB-SOMA (language: python)

Issue:
[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[r] Implement missing `domain` argument to `DataFrame` `create`
**Issue and/or context:** As tracked on issue #2407 / [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048). Fixes #2967 / [[sc-55657]](https://app.shortcut.com/tiledb-inc/story/55657/use-domain-in-somadataframecreate).

**Changes:**

As a legacy R/Python-parity issue, the `domain` argument to `SOMADataFrame`'s `create` -- from the [SOMA spec](https://github.com/single-cell-data/SOMA/blob/main/abstract_specification.md) -- was never implemented in R. Here we remedy that.

**Notes for Reviewer:**

This PR is a work in progress. It is not ready for review.

Unit-test cases are not yet implemented.


Discussion / hints:
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.


Fix patch:
diff --git a/libtiledbsoma/src/utils/arrow_adapter.cc b/libtiledbsoma/src/utils/arrow_adapter.cc
index 6b11a7232c..ac2ea971e1 100644
--- a/libtiledbsoma/src/utils/arrow_adapter.cc
+++ b/libtiledbsoma/src/utils/arrow_adapter.cc
@@ -928,7 +928,40 @@ ArraySchema ArrowAdapter::tiledb_schema_from_arrow_schema(
                     //
                     // Fortunately, these are ASCII dims and we can range
                     // these accordingly.
-                    ndrect.set_range(col_name, "", "\xff");
+
+                    ArrowArray* child_array = index_column_array->children[i];
+                    ArrowSchema* child_schema = index_column_schema
+                                                    ->children[i];
+
+                    std::vector<std::string>
+                        strings = ArrowAdapter::get_array_string_column(
+                            child_array, child_schema);
+                    if (strings.size() != 5) {
+                        throw TileDBSOMAError(fmt::format(
+                            "ArrowAdapter::tiledb_schema_from_arrow_schema: "
+                            "internal error: "
+                            "expected 5 strings, got {}",
+                            strings.size()));
+                    }
+
+                    std::string lo = strings[3];
+                    std::string hi = strings[4];
+                    if (lo == "" && hi == "") {
+                        // These mean "I the caller don't care, you
+                        // libtiledbsoma make it as big as possible"
+                        ndrect.set_range(col_name, "", "\xff");
+                    } else {
+                        ndrect.set_range(col_name, lo, hi);
+                        LOG_DEBUG(fmt::format(
+                            "[ArrowAdapter] index_column_info nbuf {}",
+                            index_column_array->children[i]->n_buffers));
+                    }
+
+                    LOG_DEBUG(fmt::format(
+                        "[ArrowAdapter] current domain {} \"{}\"-\"{}\"",
+                        child_schema->name,
+                        lo,
+                        hi));
                 } else {
                     const void* buff = index_column_array->children[i]
                                            ->buffers[1];
diff --git a/libtiledbsoma/src/utils/arrow_adapter.h b/libtiledbsoma/src/utils/arrow_adapter.h
index 2bebffbbab..0220668b4a 100644
--- a/libtiledbsoma/src/utils/arrow_adapter.h
+++ b/libtiledbsoma/src/utils/arrow_adapter.h
@@ -299,6 +299,15 @@ class ArrowAdapter {
 
     template <typename T>
     static ArrowArray* make_arrow_array_child(const std::vector<T>& v) {
+        // We're aware of template-specialization wherein we can
+        // make a separate `make_arrow_array_child` with `template <>` (not
+        // `make_arrow_array_child_string`, as we've done). However,
+        // we find it a bit fiddly across different compilers to force
+        // the compiler to find the string variant. It's far more intuitive
+        // for the non-expert developers (and maybe even for the experts),
+        // and far more robust for any future maintainers, to explicitly
+        // (a) have a separate `..._string` variant; (b) throw here
+        // if callsites don't use t.
         if (std::is_same_v<T, std::string>) {
             throw std::runtime_error(
                 "ArrowAdapter::make_arrow_array_child: template-specialization "
@@ -414,10 +423,10 @@ class ArrowAdapter {
     // for keystroke-reduction in unit-test cases.
 
     template <typename T>
-    static std::vector<T> get_table_column_by_name(
+    static std::vector<T> get_table_non_string_column_by_name(
         const ArrowTable& arrow_table, std::string column_name) {
         int64_t index = _get_column_index_from_name(arrow_table, column_name);
-        return get_table_column_by_index<T>(arrow_table, index);
+        return get_table_non_string_column_by_index<T>(arrow_table, index);
     }
 
     static std::vector<std::string> get_table_string_column_by_name(
@@ -426,8 +435,12 @@ class ArrowAdapter {
         return get_table_string_column_by_index(arrow_table, index);
     }
 
+    /**
+     * Returns a copy of the data in a specified non-string column of an
+     * ArrowTable as a standard/non-Arrow C++ object.
+     */
     template <typename T>
-    static std::vector<T> get_table_column_by_index(
+    static std::vector<T> get_table_non_string_column_by_index(
         const ArrowTable& arrow_table, int64_t column_index) {
         ArrowArray* arrow_array = arrow_table.first.get();
         ArrowSchema* arrow_schema = arrow_table.second.get();
@@ -435,75 +448,173 @@ class ArrowAdapter {
 
         if (std::is_same_v<T, std::string>) {
             throw std::runtime_error(
-                "SOMAArray::_core_domain_slot: template-specialization "
+                "SOMAArray::get_table_non_string_column_by_index: "
+                "template-specialization failure.");
+        }
+
+        ArrowArray* child_array = _get_and_check_column(
+            arrow_table, column_index, 2);
+
+        return get_array_non_string_column<T>(child_array);
+    }
+
+    /**
+     * Returns a copy of the data in a specified string column of an
+     * ArrowTable as a standard/non-Arrow C++ object.
+     */
+    static std::vector<std::string> get_table_string_column_by_index(
+        const ArrowTable& arrow_table, int64_t column_index) {
+        ArrowArray* arrow_array = arrow_table.first.get();
+        ArrowSchema* arrow_schema = arrow_table.second.get();
+        _check_shapes(arrow_array, arrow_schema);
+
+        ArrowArray* child_array = _get_and_check_column(
+            arrow_table, column_index, 3);
+
+        const ArrowSchema* child_schema = arrow_schema->children[column_index];
+
+        return get_array_string_column(child_array, child_schema);
+    }
+
+    /**
+     * Returns a copy of the data in a specified non-string column of an
+     * ArrowArray as a standard/non-Arrow C++ object. There is no ArrowSchema*
+     * argument, as the caller must have determined the Arrow type, inferred
+     * a C++ type, and have invoked this method with the appropriate C++ type.
+     *
+     * This is a helper for get_table_non_string_column_by_index; also exposed
+     * for callsites which have access to child objects which are not top-level
+     * ArrowTables.
+     */
+    template <typename T>
+    static std::vector<T> get_array_non_string_column(
+        const ArrowArray* arrow_array) {
+        if (arrow_array->n_children != 0) {
+            throw std::runtime_error(
+                "ArrowAdapter::get_array_non_string_column: expected leaf "
+                "node");
+        }
+        if (arrow_array->n_buffers != 2) {
+            throw std::runtime_error(
+                "ArrowAdapter::get_array_non_string_column: expected two "
+                "buffers");
+        }
+
+        if (std::is_same_v<T, std::string>) {
+            throw std::runtime_error(
+                "SOMAArray::get_array_non_string_column: "
+                "template-specialization "
                 "failure.");
         }
 
-        ArrowArray* child = _get_and_check_column(arrow_table, column_index, 2);
+        // Two-buffer model for non-string data:
+        // * Slot 0 is the Arrow validity buffer which we leave null
+        // * Slot 1 is data, void* but will be derefrenced as T*
+        // * There is no offset information
 
         // For our purposes -- reporting domains, etc. -- we don't use the Arrow
         // validity buffers. If this class needs to be extended someday to
         // support arrow-nulls, we can work on that.
-        if (child->buffers[0] != nullptr) {
+        if (arrow_array->buffers[0] != nullptr) {
             throw std::runtime_error(
-                "ArrowAdapter::get_table_column_by_index: validity buffer "
+                "ArrowAdapter::get_array_non_string_column: validity buffer "
                 "unsupported here");
         }
+        if (arrow_array->buffers[1] == nullptr) {
+            throw std::runtime_error(
+                "ArrowAdapter::get_array_non_string_column: null data buffer");
+        }
 
-        const void* vdata = child->buffers[1];
+        const void* vdata = arrow_array->buffers[1];
         if (vdata == nullptr) {
             throw std::runtime_error(
-                "ArrowAdapter::get_table_column_by_index: null data buffer");
+                "ArrowAdapter::get_array_non_string_column: null data buffer");
         }
 
         const T* data = (T*)vdata;
 
-        std::vector<T> retval(child->length);
-        for (auto i = 0; i < child->length; i++) {
+        std::vector<T> retval(arrow_array->length);
+        for (auto i = 0; i < arrow_array->length; i++) {
             retval[i] = data[i];
         }
         return retval;
     }
 
-    static std::vector<std::string> get_table_string_column_by_index(
-        const ArrowTable& arrow_table, int64_t column_index) {
-        ArrowArray* arrow_array = arrow_table.first.get();
-        ArrowSchema* arrow_schema = arrow_table.second.get();
-        _check_shapes(arrow_array, arrow_schema);
+    /**
+     * Returns a copy of the data in a specified non-string column of an
+     * ArrowArray as a standard/non-Arrow C++ object.
+     * Helper for get_table_string_column_by_index. Also exposed for
+     * callsites which have access to child objects which are not top-level
+     * ArrowTables.
+     */
+    static std::vector<std::string> get_array_string_column(
+        const ArrowArray* arrow_array, const ArrowSchema* arrow_schema) {
+        if (arrow_array->n_children != 0 || arrow_schema->n_children != 0) {
+            throw std::runtime_error(
+                "ArrowAdapter::get_array_string_column: expected leaf "
+                "node");
+        }
+        if (arrow_array->n_buffers != 3) {
+            throw std::runtime_error(
+                "ArrowAdapter::get_array_string_column: expected three "
+                "buffers");
+        }
 
-        ArrowArray* child = _get_and_check_column(arrow_table, column_index, 3);
+        // Three-buffer model for string data:
+        // * Slot 0 is the Arrow uint8_t* validity buffer
+        // * Slot 1 is the Arrow offsets buffer: uint32_t* for Arrow string
+        //   or uint64_t* for Arrow large_string
+        // * Slot 2 is data, void* but will be derefrenced as T*
 
         // For our purposes -- reporting domains, etc. -- we don't use the Arrow
         // validity buffers. If this class needs to be extended someday to
         // support arrow-nulls, we can work on that.
-        if (child->buffers[0] != nullptr) {
+        if (arrow_array->buffers[0] != nullptr) {
             throw std::runtime_error(
-                "ArrowAdapter::get_table_column_by_index: validity buffer "
+                "ArrowAdapter::get_array_string_column: validity buffer "
                 "unsupported here");
         }
-
-        const char* data = (char*)child->buffers[2];
-
-        if (data == nullptr) {
+        if (arrow_array->buffers[1] == nullptr) {
             throw std::runtime_error(
-                "ArrowAdapter::get_table_column_by_index: null data buffer");
+                "ArrowAdapter::get_array_string_column: null "
+                "offsets buffer");
         }
-
-        if (strcmp(arrow_schema->children[column_index]->format, "U") != 0) {
+        if (arrow_array->buffers[2] == nullptr) {
             throw std::runtime_error(
-                "ArrowAdapter::get_table_column_by_index: expected Arrow "
-                "large_string");
+                "ArrowAdapter::get_array_string_column: null data "
+                "buffer");
         }
-        uint64_t* offsets = (uint64_t*)child->buffers[1];
 
-        int num_cells = (int)child->length;
-        std::vector<std::string> retval(num_cells);
-        for (int j = 0; j < num_cells; j++) {
-            std::string e(&data[offsets[j]], &data[offsets[j + 1]]);
-            retval[j] = e;
+        const char* data = (char*)arrow_array->buffers[2];
+
+        if (strcmp(arrow_schema->format, "u") == 0 ||
+            strcmp(arrow_schema->format, "z") == 0) {
+            uint32_t* offsets = (uint32_t*)arrow_array->buffers[1];
+            int num_cells = (int)arrow_array->length;
+            std::vector<std::string> retval(num_cells);
+            for (int j = 0; j < num_cells; j++) {
+                std::string e(&data[offsets[j]], &data[offsets[j + 1]]);
+                retval[j] = e;
+            }
+            return retval;
+
+        } else if (
+            strcmp(arrow_schema->format, "U") == 0 ||
+            strcmp(arrow_schema->format, "Z") == 0) {
+            uint64_t* offsets = (uint64_t*)arrow_array->buffers[1];
+            int num_cells = (int)arrow_array->length;
+            std::vector<std::string> retval(num_cells);
+            for (int j = 0; j < num_cells; j++) {
+                std::string e(&data[offsets[j]], &data[offsets[j + 1]]);
+                retval[j] = e;
+            }
+            return retval;
+
+        } else {
+            throw std::runtime_error(
+                "ArrowAdapter::get_array_string_column: expected "
+                "Arrow string, large_string, binary, or large_binary");
         }
-
-        return retval;
     }
 
    private:


Test patch:
diff --git a/libtiledbsoma/test/common.cc b/libtiledbsoma/test/common.cc
index f19d16ea19..5d9e4e9535 100644
--- a/libtiledbsoma/test/common.cc
+++ b/libtiledbsoma/test/common.cc
@@ -197,7 +197,8 @@ static std::unique_ptr<ArrowArray> _create_index_cols_info_array(
             // arrow_adapter for more info. We rely on arrow_adapter to also
             // handle this case.
             if (info.use_current_domain) {
-                std::vector<std::string> dom({"", "", "", "", ""});
+                std::vector<std::string> dom(
+                    {"", "", "", info.string_lo, info.string_hi});
                 dim_array = ArrowAdapter::make_arrow_array_child_string(dom);
             } else {
                 std::vector<std::string> dom({"", "", ""});
diff --git a/libtiledbsoma/test/common.h b/libtiledbsoma/test/common.h
index e4ad5f8476..e8e838724f 100644
--- a/libtiledbsoma/test/common.h
+++ b/libtiledbsoma/test/common.h
@@ -67,6 +67,8 @@ struct DimInfo {
     std::string name;
     tiledb_datatype_t tiledb_datatype;
     int64_t dim_max;
+    std::string string_lo;  // For custom/restricted DataFrame domains
+    std::string string_hi;  // For custom/restricted DataFrame domains
     bool use_current_domain;
 };
 
diff --git a/libtiledbsoma/test/unit_arrow_adapter.cc b/libtiledbsoma/test/unit_arrow_adapter.cc
index 22868dd3a6..573f0abf74 100644
--- a/libtiledbsoma/test/unit_arrow_adapter.cc
+++ b/libtiledbsoma/test/unit_arrow_adapter.cc
@@ -73,20 +73,20 @@ TEST_CASE("name", "[pattern]") {
 
     ArrowTable arrow_table(std::move(arrow_array), std::move(arrow_schema));
 
-    std::vector<int64_t>
-        outputs_int64 = ArrowAdapter::get_table_column_by_name<int64_t>(
+    std::vector<int64_t> outputs_int64 =
+        ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
             arrow_table, "int64");
-    std::vector<uint8_t>
-        outputs_uint8 = ArrowAdapter::get_table_column_by_name<uint8_t>(
+    std::vector<uint8_t> outputs_uint8 =
+        ArrowAdapter::get_table_non_string_column_by_name<uint8_t>(
             arrow_table, "uint8");
-    std::vector<double>
-        outputs_float64 = ArrowAdapter::get_table_column_by_name<double>(
+    std::vector<double> outputs_float64 =
+        ArrowAdapter::get_table_non_string_column_by_name<double>(
             arrow_table, "float64");
-    std::vector<float>
-        outputs_float32 = ArrowAdapter::get_table_column_by_name<float>(
+    std::vector<float> outputs_float32 =
+        ArrowAdapter::get_table_non_string_column_by_name<float>(
             arrow_table, "float32");
     std::vector<bool>
-        outputs_bool = ArrowAdapter::get_table_column_by_name<bool>(
+        outputs_bool = ArrowAdapter::get_table_non_string_column_by_name<bool>(
             arrow_table, "bool");
     std::vector<std::string>
         outputs_string = ArrowAdapter::get_table_string_column_by_name(
diff --git a/libtiledbsoma/test/unit_soma_collection.cc b/libtiledbsoma/test/unit_soma_collection.cc
index 48bb782f9a..b6f3e6eff7 100644
--- a/libtiledbsoma/test/unit_soma_collection.cc
+++ b/libtiledbsoma/test/unit_soma_collection.cc
@@ -71,6 +71,8 @@ TEST_CASE("SOMACollection: add SOMASparseNDArray") {
             {{.name = dim_name,
               .tiledb_datatype = tiledb_datatype,
               .dim_max = DIM_MAX,
+              .string_lo = "N/A",
+              .string_hi = "N/A",
               .use_current_domain = use_current_domain}});
 
         auto index_columns = helper::create_column_index_info(dim_infos);
@@ -125,6 +127,8 @@ TEST_CASE("SOMACollection: add SOMADenseNDArray") {
         {{.name = dim_name,
           .tiledb_datatype = tiledb_datatype,
           .dim_max = DIM_MAX,
+          .string_lo = "N/A",
+          .string_hi = "N/A",
           .use_current_domain = false}});
     auto index_columns = helper::create_column_index_info(dim_infos);
 
@@ -179,6 +183,8 @@ TEST_CASE("SOMACollection: add SOMADataFrame") {
             {{.name = dim_name,
               .tiledb_datatype = tiledb_datatype,
               .dim_max = DIM_MAX,
+              .string_lo = "N/A",
+              .string_hi = "N/A",
               .use_current_domain = use_current_domain}});
         std::vector<helper::AttrInfo> attr_infos(
             {{.name = attr_name, .tiledb_datatype = tiledb_datatype}});
@@ -273,6 +279,8 @@ TEST_CASE("SOMACollection: add SOMAExperiment") {
             {{.name = dim_name,
               .tiledb_datatype = tiledb_datatype,
               .dim_max = DIM_MAX,
+              .string_lo = "N/A",
+              .string_hi = "N/A",
               .use_current_domain = use_current_domain}});
         std::vector<helper::AttrInfo> attr_infos(
             {{.name = attr_name, .tiledb_datatype = tiledb_datatype}});
@@ -327,6 +335,8 @@ TEST_CASE("SOMACollection: add SOMAMeasurement") {
             {{.name = dim_name,
               .tiledb_datatype = tiledb_datatype,
               .dim_max = DIM_MAX,
+              .string_lo = "N/A",
+              .string_hi = "N/A",
               .use_current_domain = use_current_domain}});
         std::vector<helper::AttrInfo> attr_infos(
             {{.name = attr_name, .tiledb_datatype = tiledb_datatype}});
@@ -440,6 +450,8 @@ TEST_CASE("SOMAExperiment: metadata") {
             {{.name = dim_name,
               .tiledb_datatype = tiledb_datatype,
               .dim_max = DIM_MAX,
+              .string_lo = "N/A",
+              .string_hi = "N/A",
               .use_current_domain = use_current_domain}});
         std::vector<helper::AttrInfo> attr_infos(
             {{.name = attr_name, .tiledb_datatype = tiledb_datatype}});
@@ -532,6 +544,8 @@ TEST_CASE("SOMAMeasurement: metadata") {
             {{.name = dim_name,
               .tiledb_datatype = tiledb_datatype,
               .dim_max = DIM_MAX,
+              .string_lo = "N/A",
+              .string_hi = "N/A",
               .use_current_domain = use_current_domain}});
         std::vector<helper::AttrInfo> attr_infos(
             {{.name = attr_name, .tiledb_datatype = tiledb_datatype}});
diff --git a/libtiledbsoma/test/unit_soma_dataframe.cc b/libtiledbsoma/test/unit_soma_dataframe.cc
index beb815f50d..33089f6ea9 100644
--- a/libtiledbsoma/test/unit_soma_dataframe.cc
+++ b/libtiledbsoma/test/unit_soma_dataframe.cc
@@ -79,6 +79,8 @@ struct VariouslyIndexedDataFrameFixture {
             {.name = i64_name,
              .tiledb_datatype = i64_datatype,
              .dim_max = i64_dim_max,
+             .string_lo = "N/A",
+             .string_hi = "N/A",
              .use_current_domain = use_current_domain});
     }
     helper::DimInfo u32_dim_info(bool use_current_domain) {
@@ -86,13 +88,18 @@ struct VariouslyIndexedDataFrameFixture {
             {.name = u32_name,
              .tiledb_datatype = u32_datatype,
              .dim_max = u32_dim_max,
+             .string_lo = "N/A",
+             .string_hi = "N/A",
              .use_current_domain = use_current_domain});
     }
-    helper::DimInfo str_dim_info(bool use_current_domain) {
+    helper::DimInfo str_dim_info(
+        bool use_current_domain, std::string string_lo, std::string string_hi) {
         return helper::DimInfo(
             {.name = str_name,
              .tiledb_datatype = str_datatype,
              .dim_max = str_dim_max,
+             .string_lo = string_lo,
+             .string_hi = string_hi,
              .use_current_domain = use_current_domain});
     }
 
@@ -548,18 +555,18 @@ TEST_CASE_METHOD(
 
         // Check domainish accessors before resize
         ArrowTable non_empty_domain = soma_dataframe->get_non_empty_domain();
-        std::vector<int64_t>
-            ned_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        std::vector<int64_t> ned_sjid =
+            ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
                 non_empty_domain, "soma_joinid");
 
         ArrowTable soma_domain = soma_dataframe->get_soma_domain();
-        std::vector<int64_t>
-            dom_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        std::vector<int64_t> dom_sjid =
+            ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
                 soma_domain, "soma_joinid");
 
         ArrowTable soma_maxdomain = soma_dataframe->get_soma_maxdomain();
-        std::vector<int64_t>
-            maxdom_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        std::vector<int64_t> maxdom_sjid =
+            ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
                 soma_maxdomain, "soma_joinid");
 
         REQUIRE(ned_sjid == std::vector<int64_t>({1, 2}));
@@ -634,16 +641,16 @@ TEST_CASE_METHOD(
         soma_dataframe->open(OpenMode::read);
 
         non_empty_domain = soma_dataframe->get_non_empty_domain();
-        ned_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        ned_sjid = ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
             non_empty_domain, "soma_joinid");
 
         soma_domain = soma_dataframe->get_soma_domain();
-        dom_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        dom_sjid = ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
             soma_domain, "soma_joinid");
 
         soma_maxdomain = soma_dataframe->get_soma_maxdomain();
-        maxdom_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
-            soma_maxdomain, "soma_joinid");
+        maxdom_sjid = ArrowAdapter::get_table_non_string_column_by_name<
+            int64_t>(soma_maxdomain, "soma_joinid");
 
         if (!use_current_domain) {
             REQUIRE(ned_sjid == std::vector<int64_t>({1, 10}));
@@ -727,27 +734,27 @@ TEST_CASE_METHOD(
         soma_dataframe->open(OpenMode::read);
 
         ArrowTable non_empty_domain = soma_dataframe->get_non_empty_domain();
-        std::vector<int64_t>
-            ned_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        std::vector<int64_t> ned_sjid =
+            ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
                 non_empty_domain, "soma_joinid");
-        std::vector<uint32_t>
-            ned_u32 = ArrowAdapter::get_table_column_by_name<uint32_t>(
+        std::vector<uint32_t> ned_u32 =
+            ArrowAdapter::get_table_non_string_column_by_name<uint32_t>(
                 non_empty_domain, "myuint32");
 
         ArrowTable soma_domain = soma_dataframe->get_soma_domain();
-        std::vector<int64_t>
-            dom_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        std::vector<int64_t> dom_sjid =
+            ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
                 soma_domain, "soma_joinid");
-        std::vector<uint32_t>
-            dom_u32 = ArrowAdapter::get_table_column_by_name<uint32_t>(
+        std::vector<uint32_t> dom_u32 =
+            ArrowAdapter::get_table_non_string_column_by_name<uint32_t>(
                 soma_domain, "myuint32");
 
         ArrowTable soma_maxdomain = soma_dataframe->get_soma_maxdomain();
-        std::vector<int64_t>
-            maxdom_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        std::vector<int64_t> maxdom_sjid =
+            ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
                 soma_maxdomain, "soma_joinid");
-        std::vector<uint32_t>
-            maxdom_u32 = ArrowAdapter::get_table_column_by_name<uint32_t>(
+        std::vector<uint32_t> maxdom_u32 =
+            ArrowAdapter::get_table_non_string_column_by_name<uint32_t>(
                 soma_maxdomain, "myuint32");
 
         REQUIRE(ned_sjid == std::vector<int64_t>({1, 10}));
@@ -828,22 +835,22 @@ TEST_CASE_METHOD(
         soma_dataframe->open(OpenMode::read);
 
         non_empty_domain = soma_dataframe->get_non_empty_domain();
-        ned_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        ned_sjid = ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
             non_empty_domain, "soma_joinid");
-        ned_u32 = ArrowAdapter::get_table_column_by_name<uint32_t>(
+        ned_u32 = ArrowAdapter::get_table_non_string_column_by_name<uint32_t>(
             non_empty_domain, "myuint32");
 
         soma_domain = soma_dataframe->get_soma_domain();
-        dom_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        dom_sjid = ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
             soma_domain, "soma_joinid");
-        dom_u32 = ArrowAdapter::get_table_column_by_name<uint32_t>(
+        dom_u32 = ArrowAdapter::get_table_non_string_column_by_name<uint32_t>(
             soma_domain, "myuint32");
 
         soma_maxdomain = soma_dataframe->get_soma_maxdomain();
-        maxdom_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
-            soma_maxdomain, "soma_joinid");
-        maxdom_u32 = ArrowAdapter::get_table_column_by_name<uint32_t>(
-            soma_maxdomain, "myuint32");
+        maxdom_sjid = ArrowAdapter::get_table_non_string_column_by_name<
+            int64_t>(soma_maxdomain, "soma_joinid");
+        maxdom_u32 = ArrowAdapter::get_table_non_string_column_by_name<
+            uint32_t>(soma_maxdomain, "myuint32");
 
         if (!use_current_domain) {
             REQUIRE(ned_sjid == std::vector<int64_t>({1, 10}));
@@ -883,14 +890,22 @@ TEST_CASE_METHOD(
     std::ostringstream section;
     section << "- use_current_domain=" << use_current_domain;
     SECTION(section.str()) {
-        std::string suffix = use_current_domain ? "true" : "false";
+        auto specify_domain = GENERATE(false, true);
+        std::ostringstream section2;
+        section2 << "- specify_domain=" << specify_domain;
+
+        std::string suffix1 = use_current_domain ? "true" : "false";
+        std::string suffix2 = specify_domain ? "true" : "false";
         set_up(
             std::make_shared<SOMAContext>(),
-            "mem://unit-test-variant-indexed-dataframe-3-" + suffix);
+            "mem://unit-test-variant-indexed-dataframe-3-" + suffix1 + "-" +
+                suffix2);
 
+        std::string string_lo = specify_domain ? "apple" : "";
+        std::string string_hi = specify_domain ? "zebra" : "";
         std::vector<helper::DimInfo> dim_infos(
             {i64_dim_info(use_current_domain),
-             str_dim_info(use_current_domain)});
+             str_dim_info(use_current_domain, string_lo, string_hi)});
         std::vector<helper::AttrInfo> attr_infos({u32_attr_info()});
 
         // Create
@@ -915,12 +930,17 @@ TEST_CASE_METHOD(
 
             std::array<std::string, 2> str_range = ndrect.range<std::string>(
                 dim_infos[1].name);
-            // Can we write empty strings in this range?
-            REQUIRE(str_range[0] <= "");
-            REQUIRE(str_range[1] >= "");
-            // Can we write ASCII values in this range?
-            REQUIRE(str_range[0] < " ");
-            REQUIRE(str_range[1] > "~");
+            if (specify_domain) {
+                REQUIRE(str_range[0] == dim_infos[1].string_lo);
+                REQUIRE(str_range[1] == dim_infos[1].string_hi);
+            } else {
+                // Can we write empty strings in this range?
+                REQUIRE(str_range[0] <= "");
+                REQUIRE(str_range[1] >= "");
+                // Can we write ASCII values in this range?
+                REQUIRE(str_range[0] < " ");
+                REQUIRE(str_range[1] > "~");
+            }
         }
 
         // Check shape before write
@@ -949,24 +969,24 @@ TEST_CASE_METHOD(
 
         // Check domainish accessors before resize
         ArrowTable non_empty_domain = soma_dataframe->get_non_empty_domain();
-        std::vector<int64_t>
-            ned_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        std::vector<int64_t> ned_sjid =
+            ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
                 non_empty_domain, "soma_joinid");
         std::vector<std::string>
             ned_str = ArrowAdapter::get_table_string_column_by_name(
                 non_empty_domain, "mystring");
 
         ArrowTable soma_domain = soma_dataframe->get_soma_domain();
-        std::vector<int64_t>
-            dom_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        std::vector<int64_t> dom_sjid =
+            ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
                 soma_domain, "soma_joinid");
         std::vector<std::string>
             dom_str = ArrowAdapter::get_table_string_column_by_name(
                 soma_domain, "mystring");
 
         ArrowTable soma_maxdomain = soma_dataframe->get_soma_maxdomain();
-        std::vector<int64_t>
-            maxdom_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        std::vector<int64_t> maxdom_sjid =
+            ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
                 soma_maxdomain, "soma_joinid");
         std::vector<std::string>
             maxdom_str = ArrowAdapter::get_table_string_column_by_name(
@@ -976,11 +996,18 @@ TEST_CASE_METHOD(
         REQUIRE(ned_str == std::vector<std::string>({"apple", "bat"}));
 
         REQUIRE(dom_sjid == std::vector<int64_t>({0, 99}));
-        REQUIRE(dom_str == std::vector<std::string>({"", ""}));
 
         if (!use_current_domain) {
             REQUIRE(maxdom_sjid == std::vector<int64_t>({0, 99}));
         } else {
+            if (specify_domain) {
+                REQUIRE(dom_str[0] == dim_infos[1].string_lo);
+                REQUIRE(dom_str[1] == dim_infos[1].string_hi);
+            } else {
+                REQUIRE(dom_str[0] == "");
+                REQUIRE(dom_str[1] == "");
+            }
+
             REQUIRE(maxdom_sjid[0] == 0);
             REQUIRE(maxdom_sjid[1] > 2000000000);
         }
@@ -1040,20 +1067,20 @@ TEST_CASE_METHOD(
         soma_dataframe->open(OpenMode::read, TimestampRange(0, 2));
 
         non_empty_domain = soma_dataframe->get_non_empty_domain();
-        ned_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        ned_sjid = ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
             non_empty_domain, "soma_joinid");
         ned_str = ArrowAdapter::get_table_string_column_by_name(
             non_empty_domain, "mystring");
 
         soma_domain = soma_dataframe->get_soma_domain();
-        dom_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
+        dom_sjid = ArrowAdapter::get_table_non_string_column_by_name<int64_t>(
             soma_domain, "soma_joinid");
         dom_str = ArrowAdapter::get_table_string_column_by_name(
             soma_domain, "mystring");
 
         soma_maxdomain = soma_dataframe->get_soma_maxdomain();
-        maxdom_sjid = ArrowAdapter::get_table_column_by_name<int64_t>(
-            soma_maxdomain, "soma_joinid");
+        maxdom_sjid = ArrowAdapter::get_table_non_string_column_by_name<
+            int64_t>(soma_maxdomain, "soma_joinid");
         maxdom_str = ArrowAdapter::get_table_string_column_by_name(
             soma_maxdomain, "mystring");
 
@@ -1061,11 +1088,18 @@ TEST_CASE_METHOD(
         REQUIRE(ned_str == std::vector<std::string>({"", ""}));
 
         REQUIRE(dom_sjid == std::vector<int64_t>({0, 99}));
-        REQUIRE(dom_str == std::vector<std::string>({"", ""}));
 
         if (!use_current_domain) {
             REQUIRE(maxdom_sjid == std::vector<int64_t>({0, 99}));
+            REQUIRE(dom_str == std::vector<std::string>({"", ""}));
         } else {
+            if (specify_domain) {
+                REQUIRE(dom_str[0] == dim_infos[1].string_lo);
+                REQUIRE(dom_str[1] == dim_infos[1].string_hi);
+            } else {
+                REQUIRE(dom_str == std::vector<std::string>({"", ""}));
+            }
+
             REQUIRE(maxdom_sjid[0] == 0);
             REQUIRE(maxdom_sjid[1] > 2000000000);
         }
@@ -1073,7 +1107,6 @@ TEST_CASE_METHOD(
         REQUIRE(maxdom_str == std::vector<std::string>({"", ""}));
 
         REQUIRE(ned_str == std::vector<std::string>({"", ""}));
-        REQUIRE(dom_str == std::vector<std::string>({"", ""}));
 
         soma_dataframe->close();
     }
@@ -1087,13 +1120,21 @@ TEST_CASE_METHOD(
     std::ostringstream section;
     section << "- use_current_domain=" << use_current_domain;
     SECTION(section.str()) {
-        std::string suffix = use_current_domain ? "true" : "false";
+        auto specify_domain = GENERATE(false, true);
+        std::ostringstream section2;
+        section2 << "- specify_domain=" << specify_domain;
+
+        std::string suffix1 = use_current_domain ? "true" : "false";
+        std::string suffix2 = specify_domain ? "true" : "false";
         set_up(
             std::make_shared<SOMAContext>(),
-            "mem://unit-test-variant-indexed-dataframe-4-" + suffix);
+            "mem://unit-test-variant-indexed-dataframe-4-" + suffix1 + "-" +
+                suffix2);
 
+        std::string string_lo = specify_domain ? "apple" : "";
+        std::string string_hi = specify_domain ? "zebra" : "";
         std::vector<helper::DimInfo> dim_infos(
-            {str_dim_info(use_current_domain),
+            {str_dim_info(use_current_domain, string_lo, string_hi),
              u32_dim_info(use_current_domain)});
         std::vector<helper::AttrInfo> attr_infos({i64_attr_info()});
 
@@ -1114,12 +1155,17 @@ TEST_CASE_METHOD(
 
             std::array<std::string, 2> str_range = ndrect.range<std::string>(
                 dim_infos[0].name);
-            // Can we write empty strings in this range?
-            REQUIRE(str_range[0] <= "");
-            REQUIRE(str_range[1] >= "");
-            // Can we write ASCII values in this range?
-            REQUIRE(str_range[0] < " ");
-            REQUIRE(str_range[1] > "~");
+            if (specify_domain) {
+                REQUIRE(str_range[0] == dim_infos[0].string_lo);
+                REQUIRE(str_range[1] == dim_infos[0].string_hi);
+            } else {
+                // Can we write empty strings in this range?
+                REQUIRE(str_range[0] <= "");
+                REQUIRE(str_range[1] >= "");
+                // Can we write ASCII values in this range?
+                REQUIRE(str_range[0] < " ");
+                REQUIRE(str_range[1] > "~");
+            }
 
             std::array<uint32_t, 2> u32_range = ndrect.range<uint32_t>(
                 dim_infos[1].name);
@@ -1154,7 +1200,12 @@ TEST_CASE_METHOD(
             REQUIRE(dom_str == std::vector<std::string>({"", ""}));
             REQUIRE(maxdom_str == std::vector<std::string>({"", ""}));
         } else {
-            REQUIRE(dom_str == std::vector<std::string>({"", ""}));
+            if (specify_domain) {
+                REQUIRE(dom_str[0] == dim_infos[0].string_lo);
+                REQUIRE(dom_str[1] == dim_infos[0].string_hi);
+            } else {
+                REQUIRE(dom_str == std::vector<std::string>({"", ""}));
+            }
             REQUIRE(maxdom_str == std::vector<std::string>({"", ""}));
         }
 
@@ -1239,7 +1290,12 @@ TEST_CASE_METHOD(
             REQUIRE(dom_str == std::vector<std::string>({"", ""}));
             REQUIRE(maxdom_str == std::vector<std::string>({"", ""}));
         } else {
-            REQUIRE(dom_str == std::vector<std::string>({"", ""}));
+            if (specify_domain) {
+                REQUIRE(dom_str[0] == dim_infos[0].string_lo);
+                REQUIRE(dom_str[1] == dim_infos[0].string_hi);
+            } else {
+                REQUIRE(dom_str == std::vector<std::string>({"", ""}));
+            }
             REQUIRE(maxdom_str == std::vector<std::string>({"", ""}));
         }
 
diff --git a/libtiledbsoma/test/unit_soma_dense_ndarray.cc b/libtiledbsoma/test/unit_soma_dense_ndarray.cc
index 3f22ea7727..d9aaca6790 100644
--- a/libtiledbsoma/test/unit_soma_dense_ndarray.cc
+++ b/libtiledbsoma/test/unit_soma_dense_ndarray.cc
@@ -62,6 +62,8 @@ TEST_CASE("SOMADenseNDArray: basic", "[SOMADenseNDArray]") {
             {{.name = dim_name,
               .tiledb_datatype = dim_tiledb_datatype,
               .dim_max = dim_max,
+              .string_lo = "N/A",
+              .string_hi = "N/A",
               .use_current_domain = use_current_domain}});
 
         auto index_columns = helper::create_column_index_info(dim_infos);
@@ -174,6 +176,8 @@ TEST_CASE("SOMADenseNDArray: platform_config", "[SOMADenseNDArray]") {
             {{.name = dim_name,
               .tiledb_datatype = tiledb_datatype,
               .dim_max = dim_max,
+              .string_lo = "N/A",
+              .string_hi = "N/A",
               .use_current_domain = use_current_domain}});
 
         auto index_columns = helper::create_column_index_info(dim_infos);
@@ -236,6 +240,8 @@ TEST_CASE("SOMADenseNDArray: metadata", "[SOMADenseNDArray]") {
             {{.name = dim_name,
               .tiledb_datatype = tiledb_datatype,
               .dim_max = dim_max,
+              .string_lo = "N/A",
+              .string_hi = "N/A",
               .use_current_domain = use_current_domain}});
 
         auto index_columns = helper::create_column_index_info(dim_infos);
diff --git a/libtiledbsoma/test/unit_soma_sparse_ndarray.cc b/libtiledbsoma/test/unit_soma_sparse_ndarray.cc
index 07e6cd34c7..bb6b2decc4 100644
--- a/libtiledbsoma/test/unit_soma_sparse_ndarray.cc
+++ b/libtiledbsoma/test/unit_soma_sparse_ndarray.cc
@@ -63,6 +63,8 @@ TEST_CASE("SOMASparseNDArray: basic", "[SOMASparseNDArray]") {
             {{.name = dim_name,
               .tiledb_datatype = dim_tiledb_datatype,
               .dim_max = dim_max,
+              .string_lo = "N/A",
+              .string_hi = "N/A",
               .use_current_domain = use_current_domain}});
 
         auto index_columns = helper::create_column_index_info(dim_infos);
@@ -214,6 +216,8 @@ TEST_CASE("SOMASparseNDArray: platform_config", "[SOMASparseNDArray]") {
             {{.name = dim_name,
               .tiledb_datatype = dim_tiledb_datatype,
               .dim_max = dim_max,
+              .string_lo = "N/A",
+              .string_hi = "N/A",
               .use_current_domain = use_current_domain}});
 
         auto index_columns = helper::create_column_index_info(dim_infos);
@@ -264,6 +268,8 @@ TEST_CASE("SOMASparseNDArray: metadata", "[SOMASparseNDArray]") {
             {{.name = dim_name,
               .tiledb_datatype = dim_tiledb_datatype,
               .dim_max = dim_max,
+              .string_lo = "N/A",
+              .string_hi = "N/A",
               .use_current_domain = use_current_domain}});
 
         auto index_columns = helper::create_column_index_info(dim_infos);