Repository: single-cell-data/TileDB-SOMA (language: python)

Issue:
[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[c++] Parameterize schema-creation unit-test helpers
**Issue and/or context:** This is a line-count-reducing under-diff following on #2936 for issue #2407 / [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

The intended Python and R API changes are all agreed on and finalized as described in #2407.

**Changes:**

Some prep for having variant-indexed dataframe objects in libtiledbsoma test so that #2916 and #2917 can be unit-tested.

This will get put to work for other datatypes on #2916 and #2917.

**Notes for Reviewer:**

#2785 is unreviewable as-is: besides being a WIP, it's an all-in-one experimental area. Smaller PRs such as this one are being offered for manageable review.
[c++] Make a Catch2 test fixture for dataframes
**Issue and/or context:** This is an underdiff split beneath #2944 for issue #2407 / [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048). 

The intended Python and R API changes are all agreed on and finalized as described in #2407.

**Changes:**

While I made a test fixture on #2944 for the new unit-test cases there, I realized the same fixture applied to existing test-cases.

This PR separates out from #2944 the fixture itself, and the keystroke-reduction for existing unit tests. This allows the reviewer to focus solely on the fixture and the refactor without being distracted by any new cases.

**Notes for Reviewer:**

[c++] Resize for variant-indexed `DataFrame`
**Issue and/or context:** This follows on #2917 for issue #2407 / [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048). Here we implement `resize`.

Note that the intended Python and R API changes are all agreed on and finalized as described in #2407.

**Changes:**

While `SOMASparseNDArray` and `SOMADenseNDArray` must always and only have `int64` dims within the [SOMA data model](https://github.com/single-cell-data/SOMA/blob/main/abstract_specification.md), `SOMADataFrame` is different. The default behavior -- which almost everyone uses -- has a single `soma_joinid` dim which is indeed of type `int64`. (Also note that `exp.obs.shape` _does_ exist within TileDB-SOMA-Py, and people _do_ call it.) However, the spec only requires that `soma_joinid` exist as a dim or an attr: it can be a dim along with others, or it can not be a dim at all. Here we do the right thing, without and with current-domain support, to allow people to resize the `soma_joinid` dim, and only that, whether it's the sole dimension, a dimension but not the sole one, or not a dim at all.

**Notes for Reviewer:**



Discussion / hints:
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.
## [Codecov](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/2939?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 90.02%. Comparing base [(`c8d6ca2`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/c8d6ca2b2d3945e627d521dea4ecb9d10eadd02b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) to head [(`f577de6`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/f577de6a78197c87464876141bbc64a914902076?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data).
> Report is 1 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2939      +/-   ##
==========================================
+ Coverage   89.87%   90.02%   +0.15%     
==========================================
  Files          38       38              
  Lines        3999     3999              
==========================================
+ Hits         3594     3600       +6     
+ Misses        405      399       -6     
```

| [Flag](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/2939/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/2939/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `90.02% <ø> (+0.15%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Components](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/2939/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python_api](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/2939/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `90.02% <ø> (+0.15%)` | :arrow_up: |
| [libtiledbsoma](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/2939/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `∅ <ø> (∅)` | |

</details>
Leaving this in draft until I address unit-test failures



Fix patch:
diff --git a/libtiledbsoma/src/soma/soma_array.h b/libtiledbsoma/src/soma/soma_array.h
index 8e87f3b04a..790c8594bf 100644
--- a/libtiledbsoma/src/soma/soma_array.h
+++ b/libtiledbsoma/src/soma/soma_array.h
@@ -796,6 +796,13 @@ class SOMAArray : public SOMAObject {
      */
     std::optional<TimestampRange> timestamp();
 
+    /**
+     * Exposed for testing purposes.
+     */
+    CurrentDomain get_current_domain() {
+        return _get_current_domain();
+    }
+
    private:
     //===================================================================
     //= private non-static
diff --git a/libtiledbsoma/src/utils/arrow_adapter.cc b/libtiledbsoma/src/utils/arrow_adapter.cc
index 5a450e3e26..f456b94647 100644
--- a/libtiledbsoma/src/utils/arrow_adapter.cc
+++ b/libtiledbsoma/src/utils/arrow_adapter.cc
@@ -868,9 +868,8 @@ ArraySchema ArrowAdapter::tiledb_schema_from_arrow_schema(
                     //   nullptr)
                     //
                     // Fortunately, these are ASCII dims and we can range
-                    // these accordingly. These are minimum and maximum
-                    // values, avoiding the extremes 0x00 and 0xff.
-                    ndrect.set_range(col_name, "\x01", "\xfe");
+                    // these accordingly.
+                    ndrect.set_range(col_name, "", "\xff");
                 } else {
                     const void* buff = index_column_array->children[i]
                                            ->buffers[1];


Test patch:
diff --git a/libtiledbsoma/test/common.cc b/libtiledbsoma/test/common.cc
index f8a3b5cdaa..3fdecfc76f 100644
--- a/libtiledbsoma/test/common.cc
+++ b/libtiledbsoma/test/common.cc
@@ -237,7 +237,7 @@ static std::unique_ptr<ArrowArray> _create_index_cols_info_array(
                 std::memcpy((void*)dim_array->buffers[1], vsrc, nbytes);
             } else {
                 // domain small; current_domain feature not being used
-                int64_t dom[] = {0, info.dim_max, 1};
+                uint32_t dom[] = {0, (uint32_t)info.dim_max, 1};
                 void* vsrc = (void*)&dom[0];
                 std::memcpy((void*)dim_array->buffers[1], vsrc, nbytes);
             }
diff --git a/libtiledbsoma/test/unit_soma_dataframe.cc b/libtiledbsoma/test/unit_soma_dataframe.cc
index bd75350caa..540020cb91 100644
--- a/libtiledbsoma/test/unit_soma_dataframe.cc
+++ b/libtiledbsoma/test/unit_soma_dataframe.cc
@@ -32,56 +32,187 @@
 
 #include "common.h"
 
-#define DIM_MAX 1000
+// This is a keystroke-reduction fixture for some similar unit-test cases For
+// convenience there are dims/attrs of type int64, uint32, and string. (Feel
+// free to add more types.) The main value-adds of this fixture are (a) simple
+// keystroke-reduction; (b) you get to pick which ones are the dim(s) and which
+// are the attr(s).
+struct VariouslyIndexedDataFrameFixture {
+    std::shared_ptr<SOMAContext> ctx_;
+    std::string uri_;
+    bool use_current_domain_;
+
+    //  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+    // Using Catch2's TEST_CASE_METHOD we can't pass constructor args.
+    // This is a call-after-construction method.
+    void set_up(std::shared_ptr<SOMAContext> ctx, std::string uri) {
+        ctx_ = ctx;
+        uri_ = uri;
+    }
 
-TEST_CASE("SOMADataFrame: basic") {
-    int64_t dim_max = 1000;
-    auto use_current_domain = GENERATE(false, true);
-    // TODO this could be formatted with fmt::format which is part of internal
-    // header spd/log/fmt/fmt.h and should not be used. In C++20, this can be
-    // replaced with std::format.
-    std::ostringstream section;
-    section << "- use_current_domain=" << use_current_domain;
-    SECTION(section.str()) {
-        auto ctx = std::make_shared<SOMAContext>();
-        std::string uri = "mem://unit-test-dataframe-basic";
-        std::string dim_name = "d0";
-        std::string attr_name = "a0";
-        tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+    //  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+    // Helpers for setting up dim/attr configs and data
+    static const inline int64_t i64_dim_max = 100;
+    static const inline int64_t u32_dim_max = 10000;
+    static const inline int64_t str_dim_max = 0;  // not used for string dims
+
+    static const inline std::string i64_name = "soma_joinid";
+    static const inline std::string u32_name = "myuint32";
+    static const inline std::string str_name = "mystring";
+
+    tiledb_datatype_t i64_datatype = TILEDB_INT64;
+    tiledb_datatype_t u32_datatype = TILEDB_UINT32;
+    tiledb_datatype_t str_datatype = TILEDB_STRING_ASCII;
+
+    std::string i64_arrow_format = helper::to_arrow_format(i64_datatype);
+    std::string u32_arrow_format = helper::to_arrow_format(u32_datatype);
+    std::string attr_1_arrow_format = helper::to_arrow_format(str_datatype);
+
+    helper::DimInfo i64_dim_info(bool use_current_domain) {
+        return helper::DimInfo(
+            {.name = i64_name,
+             .tiledb_datatype = i64_datatype,
+             .dim_max = i64_dim_max,
+             .use_current_domain = use_current_domain});
+    }
+    helper::DimInfo u32_dim_info(bool use_current_domain) {
+        return helper::DimInfo(
+            {.name = u32_name,
+             .tiledb_datatype = u32_datatype,
+             .dim_max = u32_dim_max,
+             .use_current_domain = use_current_domain});
+    }
+    helper::DimInfo str_dim_info(bool use_current_domain) {
+        return helper::DimInfo(
+            {.name = str_name,
+             .tiledb_datatype = str_datatype,
+             .dim_max = str_dim_max,
+             .use_current_domain = use_current_domain});
+    }
 
-        REQUIRE(!SOMADataFrame::exists(uri, ctx));
+    helper::AttrInfo i64_attr_info(std::string name = i64_name) {
+        return helper::AttrInfo(
+            {.name = name, .tiledb_datatype = i64_datatype});
+    }
+    helper::AttrInfo u32_attr_info() {
+        return helper::AttrInfo(
+            {.name = u32_name, .tiledb_datatype = u32_datatype});
+    }
+    helper::AttrInfo str_attr_info() {
+        return helper::AttrInfo(
+            {.name = str_name, .tiledb_datatype = str_datatype});
+    }
 
-        std::vector<helper::DimInfo> dim_infos(
-            {{.name = dim_name,
-              .tiledb_datatype = tiledb_datatype,
-              .dim_max = dim_max,
-              .use_current_domain = use_current_domain}});
+    std::vector<int64_t> make_i64_data() {
+        return std::vector<int64_t>({1, 2});
+    }
+    std::vector<uint32_t> make_u32_data() {
+        return std::vector<uint32_t>({1234, 5678});
+    }
+    std::vector<std::string> make_str_data() {
+        return std::vector<std::string>({"apple", "bat"});
+    }
 
-        std::vector<helper::AttrInfo> attr_infos(
-            {{.name = attr_name, .tiledb_datatype = tiledb_datatype}});
+    //  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
+    // Helper methods for create/open/write/etc.
 
+    void create(
+        const std::vector<helper::DimInfo>& dim_infos,
+        const std::vector<helper::AttrInfo>& attr_infos) {
         auto [schema, index_columns] =
             helper::create_arrow_schema_and_index_columns(
                 dim_infos, attr_infos);
+        SOMADataFrame::create(
+            uri_,
+            std::move(schema),
+            ArrowTable(
+                std::move(index_columns.first),
+                std::move(index_columns.second)),
+            ctx_);
+    }
 
+    void create(
+        const std::vector<helper::DimInfo>& dim_infos,
+        const std::vector<helper::AttrInfo>& attr_infos,
+        const PlatformConfig& platform_config,
+        std::optional<TimestampRange> timestamp_range = std::nullopt) {
+        auto [schema, index_columns] =
+            helper::create_arrow_schema_and_index_columns(
+                dim_infos, attr_infos);
         SOMADataFrame::create(
-            uri,
+            uri_,
             std::move(schema),
             ArrowTable(
                 std::move(index_columns.first),
                 std::move(index_columns.second)),
-            ctx);
+            ctx_,
+            platform_config,
+            timestamp_range);
+    }
 
-        REQUIRE(SOMADataFrame::exists(uri, ctx));
-        REQUIRE(!SOMASparseNDArray::exists(uri, ctx));
-        REQUIRE(!SOMADenseNDArray::exists(uri, ctx));
+    std::unique_ptr<SOMADataFrame> open(
+        OpenMode mode,
+        ResultOrder result_order = ResultOrder::automatic,
+        std::optional<TimestampRange> timestamp_range = std::nullopt) {
+        return SOMADataFrame::open(
+            uri_,
+            mode,
+            ctx_,
+            {},  // column_names
+            result_order,
+            timestamp_range);
+    }
+
+    void write_generic_data() {
+        // No arguments -- for now.
+        // In a subsequent PR we'll vary writing in-bounds vs out of bounds.
+        auto soma_dataframe = SOMADataFrame::open(uri_, OpenMode::write, ctx_);
+
+        auto i64_data = make_i64_data();
+        auto u32_data = make_u32_data();
+        auto str_data = make_str_data();
+
+        soma_dataframe->set_column_data(
+            i64_name, i64_data.size(), i64_data.data());
+        soma_dataframe->set_column_data(
+            str_name, str_data.size(), str_data.data());
+        soma_dataframe->set_column_data(
+            u32_name, u32_data.size(), u32_data.data());
+        soma_dataframe->write();
+
+        soma_dataframe->close();
+    }
+};
+
+TEST_CASE_METHOD(VariouslyIndexedDataFrameFixture, "SOMADataFrame: basic") {
+    auto use_current_domain = GENERATE(false, true);
+    // TODO this could be formatted with fmt::format which is part of internal
+    // header spd/log/fmt/fmt.h and should not be used. In C++20, this can be
+    // replaced with std::format.
+    std::ostringstream section;
+    section << "- use_current_domain=" << use_current_domain;
+    SECTION(section.str()) {
+        set_up(
+            std::make_shared<SOMAContext>(), "mem://unit-test-dataframe-basic");
 
-        auto soma_dataframe = SOMADataFrame::open(uri, OpenMode::read, ctx);
-        REQUIRE(soma_dataframe->uri() == uri);
-        REQUIRE(soma_dataframe->ctx() == ctx);
+        std::vector<helper::DimInfo> dim_infos(
+            {i64_dim_info(use_current_domain)});
+        std::vector<helper::AttrInfo> attr_infos({u32_attr_info()});
+
+        REQUIRE(!SOMADataFrame::exists(uri_, ctx_));
+
+        create(dim_infos, attr_infos);
+
+        REQUIRE(SOMADataFrame::exists(uri_, ctx_));
+        REQUIRE(!SOMASparseNDArray::exists(uri_, ctx_));
+        REQUIRE(!SOMADenseNDArray::exists(uri_, ctx_));
+
+        auto soma_dataframe = open(OpenMode::read);
+        REQUIRE(soma_dataframe->uri() == uri_);
+        REQUIRE(soma_dataframe->ctx() == ctx_);
         REQUIRE(soma_dataframe->type() == "SOMADataFrame");
-        std::vector<std::string> expected_index_column_names = {dim_name};
+        std::vector<std::string> expected_index_column_names = {
+            dim_infos[0].name};
         REQUIRE(
             soma_dataframe->index_column_names() ==
             expected_index_column_names);
@@ -93,30 +224,33 @@ TEST_CASE("SOMADataFrame: basic") {
             d0[j] = j;
         std::vector<int> a0(10, 1);
 
-        soma_dataframe = SOMADataFrame::open(uri, OpenMode::write, ctx);
-        soma_dataframe->set_column_data(attr_name, a0.size(), a0.data());
-        soma_dataframe->set_column_data(dim_name, d0.size(), d0.data());
+        soma_dataframe = open(OpenMode::write);
+        soma_dataframe->set_column_data(
+            dim_infos[0].name, d0.size(), d0.data());
+        soma_dataframe->set_column_data(
+            attr_infos[0].name, a0.size(), a0.data());
         soma_dataframe->write();
         soma_dataframe->close();
 
-        soma_dataframe = SOMADataFrame::open(uri, OpenMode::read, ctx);
+        soma_dataframe = open(OpenMode::read);
         while (auto batch = soma_dataframe->read_next()) {
             auto arrbuf = batch.value();
-            auto d0span = arrbuf->at(dim_name)->data<int64_t>();
-            auto a0span = arrbuf->at(attr_name)->data<int>();
+            auto d0span = arrbuf->at(dim_infos[0].name)->data<int64_t>();
+            auto a0span = arrbuf->at(attr_infos[0].name)->data<int>();
             REQUIRE(d0 == std::vector<int64_t>(d0span.begin(), d0span.end()));
             REQUIRE(a0 == std::vector<int>(a0span.begin(), a0span.end()));
         }
         soma_dataframe->close();
 
-        auto soma_object = SOMAObject::open(uri, OpenMode::read, ctx);
-        REQUIRE(soma_object->uri() == uri);
+        auto soma_object = SOMAObject::open(uri_, OpenMode::read, ctx_);
+        REQUIRE(soma_object->uri() == uri_);
         REQUIRE(soma_object->type() == "SOMADataFrame");
         soma_object->close();
     }
 }
 
-TEST_CASE("SOMADataFrame: platform_config") {
+TEST_CASE_METHOD(
+    VariouslyIndexedDataFrameFixture, "SOMADataFrame: platform_config") {
     std::pair<std::string, tiledb_filter_type_t> filter = GENERATE(
         std::make_pair(
             R"({"name": "GZIP", "COMPRESSION_LEVEL": 3})", TILEDB_FILTER_GZIP),
@@ -158,26 +292,20 @@ TEST_CASE("SOMADataFrame: platform_config") {
         std::make_pair(R"("BYTESHUFFLE")", TILEDB_FILTER_BYTESHUFFLE),
         std::make_pair(R"("NOOP")", TILEDB_FILTER_NONE));
 
-    // TODO this use to be formatted with fmt::format which is part of internal
+    // TODO this used to be formatted with fmt::format which is part of internal
     // header spd/log/fmt/fmt.h and should not be used. In C++20, this can be
     // replaced with std::format.
     std::ostringstream section;
     section << "- filter=" << filter.first;
 
     SECTION(section.str()) {
-        int64_t dim_max = 1000;
         auto use_current_domain = GENERATE(false, true);
-        // TODO this could be formatted with fmt::format which is part of
-        // internal header spd/log/fmt/fmt.h and should not be used. In C++20,
-        // this can be replaced with std::format.
         std::ostringstream section2;
         section2 << "- use_current_domain=" << use_current_domain;
         SECTION(section2.str()) {
-            auto ctx = std::make_shared<SOMAContext>();
-            std::string uri = "mem://unit-test-dataframe-platform-config";
-            std::string dim_name = "d0";
-            std::string attr_name = "a0";
-            tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
+            set_up(
+                std::make_shared<SOMAContext>(),
+                "mem://unit-test-dataframe-platform-config");
 
             PlatformConfig platform_config;
             platform_config.dataframe_dim_zstd_level = 6;
@@ -189,28 +317,14 @@ TEST_CASE("SOMADataFrame: platform_config") {
             }
 
             std::vector<helper::DimInfo> dim_infos(
-                {{.name = dim_name,
-                  .tiledb_datatype = tiledb_datatype,
-                  .dim_max = dim_max,
-                  .use_current_domain = use_current_domain}});
-
-            std::vector<helper::AttrInfo> attr_infos(
-                {{.name = attr_name, .tiledb_datatype = tiledb_datatype}});
-
-            auto [schema, index_columns] =
-                helper::create_arrow_schema_and_index_columns(
-                    dim_infos, attr_infos);
-
-            SOMADataFrame::create(
-                uri,
-                std::move(schema),
-                ArrowTable(
-                    std::move(index_columns.first),
-                    std::move(index_columns.second)),
-                ctx,
-                platform_config);
-
-            auto soma_dataframe = SOMADataFrame::open(uri, OpenMode::read, ctx);
+                {i64_dim_info(use_current_domain)});
+            std::vector<helper::AttrInfo> attr_infos({i64_attr_info("a0")});
+
+            REQUIRE(!SOMADataFrame::exists(uri_, ctx_));
+
+            create(dim_infos, attr_infos, platform_config);
+
+            auto soma_dataframe = open(OpenMode::read);
             auto sch = soma_dataframe->tiledb_schema();
             REQUIRE(
                 sch->offsets_filter_list().filter(0).filter_type() ==
@@ -221,7 +335,7 @@ TEST_CASE("SOMADataFrame: platform_config") {
                 filter.second);
 
             auto dim_filter = sch->domain()
-                                  .dimension(dim_name)
+                                  .dimension(dim_infos[0].name)
                                   .filter_list()
                                   .filter(0);
             REQUIRE(dim_filter.filter_type() == TILEDB_FILTER_ZSTD);
@@ -230,7 +344,7 @@ TEST_CASE("SOMADataFrame: platform_config") {
 
             if (filter.second != TILEDB_FILTER_WEBP) {
                 REQUIRE(
-                    sch->attribute(attr_name)
+                    sch->attribute(attr_infos[0].name)
                         .filter_list()
                         .filter(0)
                         .filter_type() == filter.second);
@@ -240,8 +354,7 @@ TEST_CASE("SOMADataFrame: platform_config") {
     }
 }
 
-TEST_CASE("SOMADataFrame: metadata") {
-    int64_t dim_max = 1000;
+TEST_CASE_METHOD(VariouslyIndexedDataFrameFixture, "SOMADataFrame: metadata") {
     auto use_current_domain = GENERATE(false, true);
     // TODO this could be formatted with fmt::format which is part of internal
     // header spd/log/fmt/fmt.h and should not be used. In C++20, this can be
@@ -249,43 +362,18 @@ TEST_CASE("SOMADataFrame: metadata") {
     std::ostringstream section;
     section << "- use_current_domain=" << use_current_domain;
     SECTION(section.str()) {
-        auto ctx = std::make_shared<SOMAContext>();
-        std::string uri = "mem://unit-test-collection";
-        std::string dim_name = "d0";
-        std::string attr_name = "a0";
-        tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-        std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
+        set_up(std::make_shared<SOMAContext>(), "mem://unit-test-collection");
 
         std::vector<helper::DimInfo> dim_infos(
-            {{.name = dim_name,
-              .tiledb_datatype = tiledb_datatype,
-              .dim_max = dim_max,
-              .use_current_domain = use_current_domain}});
+            {i64_dim_info(use_current_domain)});
+        std::vector<helper::AttrInfo> attr_infos({u32_attr_info()});
 
-        std::vector<helper::AttrInfo> attr_infos(
-            {{.name = attr_name, .tiledb_datatype = tiledb_datatype}});
+        REQUIRE(!SOMADataFrame::exists(uri_, ctx_));
 
-        auto [schema, index_columns] =
-            helper::create_arrow_schema_and_index_columns(
-                dim_infos, attr_infos);
+        create(dim_infos, attr_infos, PlatformConfig(), TimestampRange(0, 2));
 
-        SOMADataFrame::create(
-            uri,
-            std::move(schema),
-            ArrowTable(
-                std::move(index_columns.first),
-                std::move(index_columns.second)),
-            ctx,
-            PlatformConfig(),
-            TimestampRange(0, 2));
-
-        auto soma_dataframe = SOMADataFrame::open(
-            uri,
-            OpenMode::write,
-            ctx,
-            {},
-            ResultOrder::automatic,
-            TimestampRange(1, 1));
+        auto soma_dataframe = open(
+            OpenMode::write, ResultOrder::automatic, TimestampRange(1, 1));
 
         int32_t val = 100;
         soma_dataframe->set_metadata("md", TILEDB_INT32, 1, &val);
@@ -336,56 +424,234 @@ TEST_CASE("SOMADataFrame: metadata") {
     }
 }
 
-TEST_CASE("SOMADataFrame: bounds-checking") {
+TEST_CASE_METHOD(
+    VariouslyIndexedDataFrameFixture, "SOMADataFrame: bounds-checking") {
     bool use_current_domain = true;
     int old_max = 100;
     int new_max = 200;
 
-    auto ctx = std::make_shared<SOMAContext>();
-    std::string uri = "mem://unit-test-bounds-checking";
-    std::string dim_name = "d0";
-    std::string attr_name = "a0";
-    tiledb_datatype_t tiledb_datatype = TILEDB_INT64;
-    std::string arrow_format = helper::to_arrow_format(tiledb_datatype);
-
-    std::vector<helper::DimInfo> dim_infos(
-        {{.name = dim_name,
-          .tiledb_datatype = tiledb_datatype,
-          .dim_max = old_max,
-          .use_current_domain = use_current_domain}});
+    set_up(std::make_shared<SOMAContext>(), "mem://unit-test-bounds-checking");
 
-    std::vector<helper::AttrInfo> attr_infos(
-        {{.name = attr_name, .tiledb_datatype = tiledb_datatype}});
+    std::vector<helper::DimInfo> dim_infos({i64_dim_info(use_current_domain)});
+    std::vector<helper::AttrInfo> attr_infos({u32_attr_info()});
 
-    auto [schema, index_columns] =
-        helper::create_arrow_schema_and_index_columns(dim_infos, attr_infos);
+    REQUIRE(!SOMADataFrame::exists(uri_, ctx_));
 
-    SOMADataFrame::create(
-        uri,
-        std::move(schema),
-        ArrowTable(
-            std::move(index_columns.first), std::move(index_columns.second)),
-        ctx);
+    create(dim_infos, attr_infos);
 
-    auto soma_dataframe = SOMADataFrame::open(uri, OpenMode::write, ctx);
+    auto soma_dataframe = open(OpenMode::write);
 
     std::vector<int64_t> d0({old_max + 1, old_max + 2});
     std::vector<double> a0({1.5, 2.5});
-    soma_dataframe->set_column_data(dim_name, d0.size(), d0.data());
-    soma_dataframe->set_column_data(attr_name, a0.size(), a0.data());
+    soma_dataframe->set_column_data(dim_infos[0].name, d0.size(), d0.data());
+    soma_dataframe->set_column_data(attr_infos[0].name, a0.size(), a0.data());
     // Writing outside the current domain should fail
     REQUIRE_THROWS(soma_dataframe->write());
     soma_dataframe->close();
 
-    soma_dataframe = SOMADataFrame::open(uri, OpenMode::write, ctx);
+    soma_dataframe = open(OpenMode::write);
     soma_dataframe->resize(std::vector<int64_t>({new_max}));
     soma_dataframe->close();
 
-    soma_dataframe = SOMADataFrame::open(uri, OpenMode::write, ctx);
-    soma_dataframe->set_column_data(dim_name, d0.size(), d0.data());
-    soma_dataframe->set_column_data(attr_name, a0.size(), a0.data());
+    soma_dataframe = open(OpenMode::write);
+    soma_dataframe->set_column_data(dim_infos[0].name, d0.size(), d0.data());
+    soma_dataframe->set_column_data(attr_infos[0].name, a0.size(), a0.data());
     // Writing after resize should succeed
     soma_dataframe->write();
 
     soma_dataframe->close();
 }
+
+TEST_CASE_METHOD(
+    VariouslyIndexedDataFrameFixture,
+    "SOMADataFrame: variant-indexed dataframe dim:sjid attr:str,u32") {
+    auto use_current_domain = GENERATE(false, true);
+    std::ostringstream section;
+    section << "- use_current_domain=" << use_current_domain;
+    SECTION(section.str()) {
+        set_up(
+            std::make_shared<SOMAContext>(),
+            "mem://unit-test-variant-indexed-dataframe-1");
+
+        std::vector<helper::DimInfo> dim_infos(
+            {i64_dim_info(use_current_domain)});
+        std::vector<helper::AttrInfo> attr_infos(
+            {str_attr_info(), u32_attr_info()});
+
+        // Create
+        create(dim_infos, attr_infos);
+
+        // Check current domain
+        auto soma_dataframe = open(OpenMode::read);
+
+        CurrentDomain current_domain = soma_dataframe->get_current_domain();
+        if (!use_current_domain) {
+            REQUIRE(current_domain.is_empty());
+        } else {
+            REQUIRE(!current_domain.is_empty());
+            REQUIRE(current_domain.type() == TILEDB_NDRECTANGLE);
+            NDRectangle ndrect = current_domain.ndrectangle();
+
+            std::array<int64_t, 2> i64_range = ndrect.range<int64_t>(
+                dim_infos[0].name);
+            REQUIRE(i64_range[0] == (int64_t)0);
+            REQUIRE(i64_range[1] == (int64_t)dim_infos[0].dim_max);
+        }
+
+        soma_dataframe->close();
+
+        write_generic_data();
+    }
+}
+
+TEST_CASE_METHOD(
+    VariouslyIndexedDataFrameFixture,
+    "SOMADataFrame: variant-indexed dataframe dim:sjid,u32 attr:str") {
+    auto use_current_domain = GENERATE(false, true);
+    std::ostringstream section;
+    section << "- use_current_domain=" << use_current_domain;
+    SECTION(section.str()) {
+        set_up(
+            std::make_shared<SOMAContext>(),
+            "mem://unit-test-variant-indexed-dataframe-2");
+
+        std::vector<helper::DimInfo> dim_infos(
+            {i64_dim_info(use_current_domain),
+             u32_dim_info(use_current_domain)});
+        std::vector<helper::AttrInfo> attr_infos({str_attr_info()});
+
+        // Create
+        create(dim_infos, attr_infos);
+
+        // Check current domain
+        auto soma_dataframe = open(OpenMode::read);
+
+        CurrentDomain current_domain = soma_dataframe->get_current_domain();
+        if (!use_current_domain) {
+            REQUIRE(current_domain.is_empty());
+        } else {
+            REQUIRE(!current_domain.is_empty());
+            REQUIRE(current_domain.type() == TILEDB_NDRECTANGLE);
+            NDRectangle ndrect = current_domain.ndrectangle();
+
+            std::array<int64_t, 2> i64_range = ndrect.range<int64_t>(
+                dim_infos[0].name);
+            REQUIRE(i64_range[0] == (int64_t)0);
+            REQUIRE(i64_range[1] == (int64_t)dim_infos[0].dim_max);
+
+            std::array<uint32_t, 2> u32_range = ndrect.range<uint32_t>(
+                dim_infos[0].name);
+            REQUIRE(u32_range[0] == (uint32_t)0);
+            REQUIRE(u32_range[1] == (uint32_t)dim_infos[0].dim_max);
+        }
+
+        soma_dataframe->close();
+
+        // Write
+        write_generic_data();
+    }
+}
+
+TEST_CASE_METHOD(
+    VariouslyIndexedDataFrameFixture,
+    "SOMADataFrame: variant-indexed dataframe dim:sjid,str attr:u32") {
+    auto use_current_domain = GENERATE(false, true);
+    std::ostringstream section;
+    section << "- use_current_domain=" << use_current_domain;
+    SECTION(section.str()) {
+        set_up(
+            std::make_shared<SOMAContext>(),
+            "mem://unit-test-variant-indexed-dataframe-3");
+
+        std::vector<helper::DimInfo> dim_infos(
+            {i64_dim_info(use_current_domain),
+             str_dim_info(use_current_domain)});
+        std::vector<helper::AttrInfo> attr_infos({u32_attr_info()});
+
+        // Create
+        create(dim_infos, attr_infos);
+
+        // Check current domain
+        auto soma_dataframe = open(OpenMode::read);
+
+        CurrentDomain current_domain = soma_dataframe->get_current_domain();
+        if (!use_current_domain) {
+            REQUIRE(current_domain.is_empty());
+        } else {
+            REQUIRE(!current_domain.is_empty());
+            REQUIRE(current_domain.type() == TILEDB_NDRECTANGLE);
+            NDRectangle ndrect = current_domain.ndrectangle();
+
+            std::array<int64_t, 2> i64_range = ndrect.range<int64_t>(
+                dim_infos[0].name);
+            REQUIRE(i64_range[0] == (int64_t)0);
+            REQUIRE(i64_range[1] == (int64_t)dim_infos[0].dim_max);
+
+            std::array<std::string, 2> str_range = ndrect.range<std::string>(
+                dim_infos[1].name);
+            // Can we write empty strings in this range?
+            REQUIRE(str_range[0] <= "");
+            REQUIRE(str_range[1] >= "");
+            // Can we write ASCII values in this range?
+            REQUIRE(str_range[0] < " ");
+            REQUIRE(str_range[1] > "~");
+        }
+
+        soma_dataframe->close();
+
+        // Write
+        write_generic_data();
+    }
+}
+
+TEST_CASE_METHOD(
+    VariouslyIndexedDataFrameFixture,
+    "SOMADataFrame: variant-indexed dataframe dim:str,u32 attr:sjid") {
+    auto use_current_domain = GENERATE(false, true);
+    std::ostringstream section;
+    section << "- use_current_domain=" << use_current_domain;
+    SECTION(section.str()) {
+        set_up(
+            std::make_shared<SOMAContext>(),
+            "mem://unit-test-variant-indexed-dataframe-4");
+
+        std::vector<helper::DimInfo> dim_infos(
+            {str_dim_info(use_current_domain),
+             u32_dim_info(use_current_domain)});
+        std::vector<helper::AttrInfo> attr_infos({i64_attr_info()});
+
+        // Create
+        create(dim_infos, attr_infos);
+
+        // Check current domain
+        auto soma_dataframe = open(OpenMode::read);
+
+        CurrentDomain current_domain = soma_dataframe->get_current_domain();
+        if (!use_current_domain) {
+            REQUIRE(current_domain.is_empty());
+        } else {
+            REQUIRE(!current_domain.is_empty());
+            REQUIRE(current_domain.type() == TILEDB_NDRECTANGLE);
+            NDRectangle ndrect = current_domain.ndrectangle();
+
+            std::array<std::string, 2> str_range = ndrect.range<std::string>(
+                dim_infos[0].name);
+            // Can we write empty strings in this range?
+            REQUIRE(str_range[0] <= "");
+            REQUIRE(str_range[1] >= "");
+            // Can we write ASCII values in this range?
+            REQUIRE(str_range[0] < " ");
+            REQUIRE(str_range[1] > "~");
+
+            std::array<uint32_t, 2> u32_range = ndrect.range<uint32_t>(
+                dim_infos[1].name);
+            REQUIRE(u32_range[0] == (uint32_t)0);
+            REQUIRE(u32_range[1] == (uint32_t)dim_infos[1].dim_max);
+        }
+
+        soma_dataframe->close();
+
+        // Write
+        write_generic_data();
+    }
+}