Repository: single-cell-data/TileDB-SOMA (language: python)

Issue:
[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now



Discussion / hints:
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.

Fix patch:
diff --git a/libtiledbsoma/src/soma/soma_array.cc b/libtiledbsoma/src/soma/soma_array.cc
index b294b9c326..0275430bad 100644
--- a/libtiledbsoma/src/soma/soma_array.cc
+++ b/libtiledbsoma/src/soma/soma_array.cc
@@ -1446,9 +1446,9 @@ std::vector<int64_t> SOMAArray::maxshape() {
     return _tiledb_domain();
 }
 
-// This is a helper for can_upgrade_domain and can_resize, which have
+// This is a helper for can_upgrade_shape and can_resize, which have
 // much overlap.
-std::pair<bool, std::string> SOMAArray::_can_set_shape_helper(
+StatusAndReason SOMAArray::_can_set_shape_helper(
     const std::vector<int64_t>& newshape,
     bool is_resize,
     std::string function_name_for_messages) {
@@ -1466,7 +1466,7 @@ std::pair<bool, std::string> SOMAArray::_can_set_shape_helper(
                 array_ndim));
     }
 
-    // Enforce the semantics that tiledbsoma_upgrade_domain must be called
+    // Enforce the semantics that tiledbsoma_upgrade_shape must be called
     // only on arrays that don't have a shape set, and resize must be called
     // only on arrays that do.
     bool has_shape = has_current_domain();
@@ -1525,7 +1525,7 @@ std::pair<bool, std::string> SOMAArray::_can_set_shape_helper(
 // This is a helper for _can_set_shape_helper: it's used for comparing
 // the user's requested shape against the core current domain or core (max)
 // domain.
-std::pair<bool, std::string> SOMAArray::_can_set_shape_domainish_subhelper(
+StatusAndReason SOMAArray::_can_set_shape_domainish_subhelper(
     const std::vector<int64_t>& newshape,
     bool check_current_domain,
     std::string function_name_for_messages) {
@@ -1585,7 +1585,7 @@ std::pair<bool, std::string> SOMAArray::_can_set_shape_domainish_subhelper(
     return std::pair(true, "");
 }
 
-std::pair<bool, std::string> SOMAArray::_can_set_soma_joinid_shape_helper(
+StatusAndReason SOMAArray::_can_set_soma_joinid_shape_helper(
     int64_t newshape, bool is_resize, std::string function_name_for_messages) {
     // Fail if the array doesn't already have a shape yet (they should upgrade
     // first).
@@ -1882,6 +1882,189 @@ void SOMAArray::_set_soma_joinid_shape_helper(
     schema_evolution.array_evolve(uri_);
 }
 
+StatusAndReason SOMAArray::can_upgrade_domain(
+    const ArrowTable& newdomain, std::string function_name_for_messages) {
+    // Enforce the semantics that tiledbsoma_upgrade_domain must be called
+    // only on arrays that don't have a shape set, and resize must be called
+    // only on arrays that do.
+    if (has_current_domain()) {
+        return std::pair(
+            false,
+            fmt::format(
+                "{}: dataframe already has a domain",
+                function_name_for_messages));
+    }
+
+    // * For old-style dataframe without shape: core domain (soma maxdomain) may
+    //   be small (like 100) or big (like 2 billionish).
+    // * For new-style dataframe with shape: core current domain (soma domain)
+    //   will probably be small and core domain (soma maxdomain) will be huge.
+    //
+    // In either case, we need to check that the user's requested soma domain
+    // isn't outside the core domain, which is immutable. For old-style
+    // dataframes, if the requested domain fits in the array's core domain, it's
+    // good to go as a new soma domain.
+    auto domain_check = _can_set_dataframe_domainish_subhelper(
+        newdomain, false, function_name_for_messages);
+    if (!domain_check.first) {
+        return domain_check;
+    }
+
+    // For new-style dataframes, we need to additionally that the the requested
+    // soma domain (core current domain) isn't a downsize of the current
+    // one.
+    if (has_current_domain()) {
+        auto current_domain_check = _can_set_dataframe_domainish_subhelper(
+            newdomain, true, function_name_for_messages);
+        if (!current_domain_check.first) {
+            return current_domain_check;
+        }
+    }
+
+    return std::pair(true, "");
+}
+
+// This is a helper for can_upgrade_domain: it's used for comparing
+// the user's requested soma domain against the core current domain or core
+// (max) domain.
+StatusAndReason SOMAArray::_can_set_dataframe_domainish_subhelper(
+    const ArrowTable& newdomain,
+    bool check_current_domain,
+    std::string function_name_for_messages) {
+    Domain domain = arr_->schema().domain();
+
+    ArrowArray* new_domain_array = newdomain.first.get();
+    ArrowSchema* new_domain_schema = newdomain.second.get();
+
+    if (new_domain_schema->n_children != domain.ndim()) {
+        return std::pair(
+            false,
+            fmt::format(
+                "{}: requested domain has ndim={} but the dataframe has "
+                "ndim={}",
+                function_name_for_messages,
+                new_domain_schema->n_children,
+                domain.ndim()));
+    }
+
+    if (new_domain_schema->n_children != new_domain_array->n_children) {
+        return std::pair(
+            false,
+            fmt::format(
+                "{}: internal coding error", function_name_for_messages));
+    }
+
+    for (unsigned i = 0; i < domain.ndim(); i++) {
+        const auto& dim = domain.dimension(i);
+
+        StatusAndReason status_and_reason;
+
+        switch (dim.type()) {
+            case TILEDB_STRING_ASCII:
+            case TILEDB_STRING_UTF8:
+            case TILEDB_CHAR:
+                status_and_reason =
+                    _can_set_dataframe_domainish_slot_checker_string(
+                        check_current_domain, newdomain, dim.name());
+                break;
+            case TILEDB_BOOL:
+                status_and_reason =
+                    _can_set_dataframe_domainish_slot_checker_non_string<bool>(
+                        check_current_domain, newdomain, dim.name());
+                break;
+            case TILEDB_INT8:
+                status_and_reason =
+                    _can_set_dataframe_domainish_slot_checker_non_string<
+                        int8_t>(check_current_domain, newdomain, dim.name());
+                break;
+            case TILEDB_UINT8:
+                status_and_reason =
+                    _can_set_dataframe_domainish_slot_checker_non_string<
+                        uint8_t>(check_current_domain, newdomain, dim.name());
+                break;
+            case TILEDB_INT16:
+                status_and_reason =
+                    _can_set_dataframe_domainish_slot_checker_non_string<
+                        int16_t>(check_current_domain, newdomain, dim.name());
+                break;
+            case TILEDB_UINT16:
+                status_and_reason =
+                    _can_set_dataframe_domainish_slot_checker_non_string<
+                        uint16_t>(check_current_domain, newdomain, dim.name());
+                break;
+            case TILEDB_INT32:
+                status_and_reason =
+                    _can_set_dataframe_domainish_slot_checker_non_string<
+                        int32_t>(check_current_domain, newdomain, dim.name());
+                break;
+            case TILEDB_UINT32:
+                status_and_reason =
+                    _can_set_dataframe_domainish_slot_checker_non_string<
+                        uint32_t>(check_current_domain, newdomain, dim.name());
+                break;
+            case TILEDB_INT64:
+            case TILEDB_DATETIME_YEAR:
+            case TILEDB_DATETIME_MONTH:
+            case TILEDB_DATETIME_WEEK:
+            case TILEDB_DATETIME_DAY:
+            case TILEDB_DATETIME_HR:
+            case TILEDB_DATETIME_MIN:
+            case TILEDB_DATETIME_SEC:
+            case TILEDB_DATETIME_MS:
+            case TILEDB_DATETIME_US:
+            case TILEDB_DATETIME_NS:
+            case TILEDB_DATETIME_PS:
+            case TILEDB_DATETIME_FS:
+            case TILEDB_DATETIME_AS:
+            case TILEDB_TIME_HR:
+            case TILEDB_TIME_MIN:
+            case TILEDB_TIME_SEC:
+            case TILEDB_TIME_MS:
+            case TILEDB_TIME_US:
+            case TILEDB_TIME_NS:
+            case TILEDB_TIME_PS:
+            case TILEDB_TIME_FS:
+            case TILEDB_TIME_AS:
+                status_and_reason =
+                    _can_set_dataframe_domainish_slot_checker_non_string<
+                        int64_t>(check_current_domain, newdomain, dim.name());
+                break;
+            case TILEDB_UINT64:
+                status_and_reason =
+                    _can_set_dataframe_domainish_slot_checker_non_string<
+                        uint64_t>(check_current_domain, newdomain, dim.name());
+                break;
+            case TILEDB_FLOAT32:
+                status_and_reason =
+                    _can_set_dataframe_domainish_slot_checker_non_string<float>(
+                        check_current_domain, newdomain, dim.name());
+                break;
+            case TILEDB_FLOAT64:
+                status_and_reason =
+                    _can_set_dataframe_domainish_slot_checker_non_string<
+                        double>(check_current_domain, newdomain, dim.name());
+                break;
+            default:
+                throw TileDBSOMAError(fmt::format(
+                    "{}: saw invalid TileDB type when attempting to cast "
+                    "domain information: {}",
+                    function_name_for_messages,
+                    tiledb::impl::type_to_str(dim.type())));
+        }
+
+        if (status_and_reason.first == false) {
+            return std::pair(
+                false,
+                fmt::format(
+                    "{} for {}: {}",
+                    function_name_for_messages,
+                    dim.name(),
+                    status_and_reason.second));
+        }
+    }
+    return std::pair(true, "");
+}
+
 std::vector<int64_t> SOMAArray::_tiledb_current_domain() {
     // Variant-indexed dataframes must use a separate path
     _check_dims_are_int64();
diff --git a/libtiledbsoma/src/soma/soma_array.h b/libtiledbsoma/src/soma/soma_array.h
index 1840b8ffbe..ed4619baeb 100644
--- a/libtiledbsoma/src/soma/soma_array.h
+++ b/libtiledbsoma/src/soma/soma_array.h
@@ -45,9 +45,49 @@
 #include "managed_query.h"
 #include "soma_object.h"
 
+// ================================================================
+// Some general developer notes:
+//
+// ----------------------------------------------------------------
+// In several places we have
+//
+//     template <typename T>
+//     static sometype foo(T arg) {
+//         if (std::is_same_v<T, std::string>) {
+//             throw std::runtime_error(...);
+//         }
+//         }D...
+//     }
+//
+//     static sometype foo_string(std::string arg) { ... }
+//
+// -- with explicit `_string` suffix -- rather than
+//
+//     template <typename T>
+//     static sometype foo(T arg) ...
+//
+//     template <>
+//     static sometype foo(std::string arg) ...
+//
+// We're aware of the former but we've found it a bit fiddly across systems and
+// compiler versions -- namely, with the latter we find it tricky to always
+// avoid the <typename T> variant being templated with std::string. It's simple,
+// explicit, and robust to go the `_string` suffix route, and it's friendlier to
+// current and future maintainers of this code.
+//
+// ----------------------------------------------------------------
+// These are several methods here for use nominally by SOMADataFrame. These
+// could be moved in their entirety to SOMADataFrame, but that would entail
+// moving several SOMAArray attributes from private to protected, which has
+// knock-on effects on the order of constructor initializers, etc.: in total
+// it's simplest to place these here and have SOMADataFrame invoke them.
+// ================================================================
+
 namespace tiledbsoma {
 using namespace tiledb;
 
+using StatusAndReason = std::pair<bool, std::string>;
+
 // This enables some code deduplication between core domain, core current
 // domain, and core non-empty domain.
 enum class Domainish {
@@ -1106,7 +1146,7 @@ class SOMAArray : public SOMAObject {
      * upgrade_shape), or the requested shape doesn't fit within the array's
      * existing core domain.
      */
-    std::pair<bool, std::string> can_resize(
+    StatusAndReason can_resize(
         const std::vector<int64_t>& newshape,
         std::string function_name_for_messages) {
         return _can_set_shape_helper(
@@ -1130,7 +1170,7 @@ class SOMAArray : public SOMAObject {
      * the requested shape is a downsize of the array's existing core current
      * domain.
      */
-    std::pair<bool, std::string> can_upgrade_shape(
+    StatusAndReason can_upgrade_shape(
         const std::vector<int64_t>& newshape,
         std::string function_name_for_messages) {
         return _can_set_shape_helper(
@@ -1141,7 +1181,7 @@ class SOMAArray : public SOMAObject {
      * This is similar to can_upgrade_shape, but it's a can-we call
      * for resize_soma_joinid_shape.
      */
-    std::pair<bool, std::string> can_resize_soma_joinid_shape(
+    StatusAndReason can_resize_soma_joinid_shape(
         int64_t newshape, std::string function_name_for_messages) {
         return _can_set_soma_joinid_shape_helper(
             newshape, true, function_name_for_messages);
@@ -1151,12 +1191,18 @@ class SOMAArray : public SOMAObject {
      * This is similar to can_upgrade_shape, but it's a can-we call
      * for upgrade_soma_joinid_shape.
      */
-    std::pair<bool, std::string> can_upgrade_soma_joinid_shape(
+    StatusAndReason can_upgrade_soma_joinid_shape(
         int64_t newshape, std::string function_name_for_messages) {
         return _can_set_soma_joinid_shape_helper(
             newshape, false, function_name_for_messages);
     }
 
+    /**
+     * This is for SOMADataFrame.
+     */
+    StatusAndReason can_upgrade_domain(
+        const ArrowTable& newdomain, std::string function_name_for_messages);
+
     /**
      * @brief Resize the shape (what core calls "current domain") up to the
      * maxshape (what core calls "domain").
@@ -1226,14 +1272,17 @@ class SOMAArray : public SOMAObject {
             newshape, false, function_name_for_messages);
     }
 
+    /**
+     * This is for SOMADataFrame.
+     */
+    void upgrade_domain(
+        const ArrowTable& newdomain, std::string function_name_for_messages);
+
    protected:
-    // These two are for use nominally by SOMADataFrame. This could be moved in
-    // its entirety to SOMADataFrame, but it would entail moving several
-    // SOMAArray attributes from private to protected, which has knock-on
-    // effects on the order of constructor initializers, etc.: in total it's
-    // simplest to place this here and have SOMADataFrame invoke it.
+    // See top-of-file notes regarding methods for SOMADataFrame being
+    // defined in this file.
     //
-    // They return the shape and maxshape slots for the soma_joinid dim, if
+    // These return the shape and maxshape slots for the soma_joinid dim, if
     // the array has one. These are important test-points and dev-internal
     // access-points, in particular, for the tiledbsoma-io experiment-level
     // resizer.
@@ -1283,9 +1332,9 @@ class SOMAArray : public SOMAObject {
     }
 
     /**
-     * This is a code-dedupe helper for can_resize and can_upgrade_domain.
+     * This is a code-dedupe helper for can_resize and can_upgrade_shape.
      */
-    std::pair<bool, std::string> _can_set_shape_helper(
+    StatusAndReason _can_set_shape_helper(
         const std::vector<int64_t>& newshape,
         bool is_resize,
         std::string function_name_for_messages);
@@ -1293,16 +1342,24 @@ class SOMAArray : public SOMAObject {
     /**
      * This is a second-level code-dedupe helper for _can_set_shape_helper.
      */
-    std::pair<bool, std::string> _can_set_shape_domainish_subhelper(
+    StatusAndReason _can_set_shape_domainish_subhelper(
         const std::vector<int64_t>& newshape,
         bool check_current_domain,
         std::string function_name_for_messages);
 
+    /**
+     * This is a code-dedupe helper for can_upgrade_domain.
+     */
+    StatusAndReason _can_set_dataframe_domainish_subhelper(
+        const ArrowTable& newdomain,
+        bool check_current_domain,
+        std::string function_name_for_messages);
+
     /**
      * This is a code-dedupe helper for can_resize_soma_joinid_shape and
      * can_upgrade_domain_soma_joinid_shape.
      */
-    std::pair<bool, std::string> _can_set_soma_joinid_shape_helper(
+    StatusAndReason _can_set_soma_joinid_shape_helper(
         int64_t newshape,
         bool is_resize,
         std::string function_name_for_messages);
@@ -1324,6 +1381,96 @@ class SOMAArray : public SOMAObject {
         bool is_resize,
         std::string function_name_for_messages);
 
+    /**
+     * This is a helper for can_upgrade_domain.
+     */
+    template <typename T>
+    StatusAndReason _can_set_dataframe_domainish_slot_checker_non_string(
+        bool check_current_domain,
+        const ArrowTable& domain_table,
+        std::string dim_name) {
+        std::pair<T, T> old_lo_hi = check_current_domain ?
+                                        _core_current_domain_slot<T>(dim_name) :
+                                        _core_domain_slot<T>(dim_name);
+        std::vector<T>
+            new_lo_hi = ArrowAdapter::get_table_non_string_column_by_name<T>(
+                domain_table, dim_name);
+        if (new_lo_hi.size() != 2) {
+            throw TileDBSOMAError(
+                "internal coding error detected at "
+                "_can_set_dataframe_domainish_slot_checker");
+        }
+
+        const T& old_lo = old_lo_hi.first;
+        const T& old_hi = old_lo_hi.second;
+        const T& new_lo = new_lo_hi[0];
+        const T& new_hi = new_lo_hi[1];
+
+        // It's difficult to use fmt::format within a header file since the
+        // include path to logger.h 'moves around' depending on which source
+        // file included us.
+        //
+        // TODO: once we're on C++ 20, just use std::format here and include
+        // things like "old ({}, {}) new ({}, {})".
+        if (new_lo > new_hi) {
+            return std::pair(false, "new lower > new upper");
+        }
+        if (new_lo > old_lo) {
+            return std::pair(
+                false, "new lower > old lower (downsize is unsupported)");
+        }
+        if (new_hi < old_hi) {
+            return std::pair(
+                false, "new upper < old upper (downsize is unsupported)");
+        }
+        return std::pair(true, "");
+    }
+
+    /**
+     * This is a helper for can_upgrade_domain.
+     */
+    StatusAndReason _can_set_dataframe_domainish_slot_checker_string(
+        bool check_current_domain,
+        const ArrowTable& domain_table,
+        std::string dim_name) {
+        std::pair<std::string, std::string>
+            old_lo_hi = check_current_domain ?
+                            _core_current_domain_slot_string(dim_name) :
+                            _core_domain_slot_string(dim_name);
+        std::vector<std::string>
+            new_lo_hi = ArrowAdapter::get_table_string_column_by_name(
+                domain_table, dim_name);
+        if (new_lo_hi.size() != 2) {
+            throw TileDBSOMAError(
+                "internal coding error detected at "
+                "_can_set_dataframe_domainish_slot_checker");
+        }
+
+        const std::string& old_lo = old_lo_hi.first;
+        const std::string& old_hi = old_lo_hi.second;
+        const std::string& new_lo = new_lo_hi[0];
+        const std::string& new_hi = new_lo_hi[1];
+
+        // It's difficult to use fmt::format within a header file since the
+        // include path to logger.h 'moves around' depending on which source
+        // file included us.
+        //
+        // TODO: once we're on C++ 20, just use std::format here and include
+        // things like "old ({}, {}) new ({}, {})".
+        if (new_lo > new_hi) {
+            return std::pair(false, "new lower > new upper");
+        }
+        if (new_lo > old_lo) {
+            return std::pair(
+                false, "new lower > old lower (downsize is unsupported)");
+        }
+        if (new_hi < old_hi) {
+            return std::pair(
+                false, "new upper < old upper (downsize is unsupported)");
+        }
+        return std::pair(true, "");
+    }
+
     /**
      * While SparseNDArray, DenseNDArray, and default-indexed DataFrame
      * have int64 dims, variant-indexed DataFrames do not. This helper


Test patch:
diff --git a/libtiledbsoma/test/common.cc b/libtiledbsoma/test/common.cc
index c3f0e60197..831b503ee5 100644
--- a/libtiledbsoma/test/common.cc
+++ b/libtiledbsoma/test/common.cc
@@ -42,8 +42,6 @@ namespace helper {
 //   array-creation error.)
 const int CORE_DOMAIN_MAX = 2147483646;
 
-static std::unique_ptr<ArrowSchema> _create_index_cols_info_schema(
-    const std::vector<DimInfo>& dim_infos);
 static std::unique_ptr<ArrowArray> _create_index_cols_info_array(
     const std::vector<DimInfo>& dim_infos);
 
@@ -112,7 +110,7 @@ create_arrow_schema_and_index_columns(
     auto arrow_schema = ArrowAdapter::make_arrow_schema(
         names, tiledb_datatypes);
 
-    auto index_cols_info_schema = _create_index_cols_info_schema(dim_infos);
+    auto index_cols_info_schema = create_index_cols_info_schema(dim_infos);
     auto index_cols_info_array = _create_index_cols_info_array(dim_infos);
 
     return std::pair(
@@ -133,14 +131,14 @@ ArrowTable create_column_index_info(const std::vector<DimInfo>& dim_infos) {
             info.use_current_domain));
     }
 
-    auto index_cols_info_schema = _create_index_cols_info_schema(dim_infos);
+    auto index_cols_info_schema = create_index_cols_info_schema(dim_infos);
     auto index_cols_info_array = _create_index_cols_info_array(dim_infos);
 
     return ArrowTable(
         std::move(index_cols_info_array), std::move(index_cols_info_schema));
 }
 
-static std::unique_ptr<ArrowSchema> _create_index_cols_info_schema(
+std::unique_ptr<ArrowSchema> create_index_cols_info_schema(
     const std::vector<DimInfo>& dim_infos) {
     auto ndim = dim_infos.size();
 
diff --git a/libtiledbsoma/test/common.h b/libtiledbsoma/test/common.h
index 7df4a6cc2b..ef01ada426 100644
--- a/libtiledbsoma/test/common.h
+++ b/libtiledbsoma/test/common.h
@@ -90,5 +90,8 @@ std::string to_arrow_format(tiledb_datatype_t tiledb_datatype);
 // Core PR: https://github.com/TileDB-Inc/TileDB/pull/5303
 bool have_dense_current_domain_support();
 
+std::unique_ptr<ArrowSchema> create_index_cols_info_schema(
+    const std::vector<DimInfo>& dim_infos);
+
 }  // namespace helper
 #endif
diff --git a/libtiledbsoma/test/unit_soma_dataframe.cc b/libtiledbsoma/test/unit_soma_dataframe.cc
index d481a86310..131fa405a4 100644
--- a/libtiledbsoma/test/unit_soma_dataframe.cc
+++ b/libtiledbsoma/test/unit_soma_dataframe.cc
@@ -601,16 +601,46 @@ TEST_CASE_METHOD(
         write_sjid_u32_str_data_from(8);
         REQUIRE(sdf->nnz() == 4);
 
-        // Check can_upgrade_soma_joinid_shape
         sdf->open(OpenMode::read);
+
+        // Check can_upgrade_soma_joinid_shape
         if (!use_current_domain) {
-            std::pair<bool, std::string>
-                check = sdf->can_upgrade_soma_joinid_shape(1, "testing");
+            StatusAndReason check = sdf->can_upgrade_soma_joinid_shape(
+                1, "testing");
             REQUIRE(check.first == true);
             REQUIRE(check.second == "");
         } else {
-            std::pair<bool, std::string>
-                check = sdf->can_upgrade_soma_joinid_shape(1, "testing");
+            StatusAndReason check = sdf->can_upgrade_soma_joinid_shape(
+                1, "testing");
+            // Must fail since this is too small.
+            REQUIRE(check.first == false);
+            REQUIRE(
+                check.second ==
+                "testing: dataframe already has its domain set.");
+        }
+        sdf->close();
+
+        // Check can_upgrade_domain
+        sdf->open(OpenMode::read);
+        std::unique_ptr<ArrowSchema>
+            domain_schema = create_index_cols_info_schema(dim_infos);
+        auto domain_array = ArrowAdapter::make_arrow_array_parent(
+            dim_infos.size());
+        domain_array->children[0] = ArrowAdapter::make_arrow_array_child(
+            std::vector<int64_t>({0, 0}));
+        auto domain_table = ArrowTable(
+            std::move(domain_array), std::move(domain_schema));
+        if (!use_current_domain) {
+            StatusAndReason check = sdf->can_upgrade_domain(
+                domain_table, "testing");
+            REQUIRE(check.first == false);
+            REQUIRE(
+                check.second ==
+                "testing for soma_joinid: new upper < old upper (downsize is "
+                "unsupported)");
+        } else {
+            StatusAndReason check = sdf->can_upgrade_soma_joinid_shape(
+                1, "testing");
             // Must fail since this is too small.
             REQUIRE(check.first == false);
             REQUIRE(
@@ -699,8 +729,7 @@ TEST_CASE_METHOD(
         }
 
         // Check can_resize_soma_joinid_shape
-        std::pair<bool, std::string> check = sdf->can_resize_soma_joinid_shape(
-            1, "testing");
+        StatusAndReason check = sdf->can_resize_soma_joinid_shape(1, "testing");
         if (!use_current_domain) {
             REQUIRE(check.first == false);
             REQUIRE(
@@ -735,6 +764,22 @@ TEST_CASE_METHOD(
         }
 
         sdf->close();
+
+        // Check can_upgrade_domain
+        sdf->open(OpenMode::read);
+        domain_schema = create_index_cols_info_schema(dim_infos);
+        domain_array = ArrowAdapter::make_arrow_array_parent(dim_infos.size());
+        domain_array->children[0] = ArrowAdapter::make_arrow_array_child(
+            std::vector<int64_t>({0, 0}));
+        domain_table = ArrowTable(
+            std::move(domain_array), std::move(domain_schema));
+        // The dataframe now has a shape
+        check = sdf->can_upgrade_soma_joinid_shape(1, "testing");
+        // Must fail since this is too small.
+        REQUIRE(check.first == false);
+        REQUIRE(
+            check.second == "testing: dataframe already has its domain set.");
+        sdf->close();
     }
 }
 
@@ -849,6 +894,53 @@ TEST_CASE_METHOD(
         actual = sdf->maybe_soma_joinid_shape();
         REQUIRE(actual.has_value());
         REQUIRE(actual.value() == expect);
+
+        // Check can_upgrade_soma_joinid_shape
+        if (!use_current_domain) {
+            StatusAndReason check = sdf->can_upgrade_soma_joinid_shape(
+                1, "testing");
+            REQUIRE(check.first == true);
+            REQUIRE(check.second == "");
+        } else {
+            StatusAndReason check = sdf->can_upgrade_soma_joinid_shape(
+                1, "testing");
+            // Must fail since this is too small.
+            REQUIRE(check.first == false);
+            REQUIRE(
+                check.second ==
+                "testing: dataframe already has its domain set.");
+        }
+
+        // Check can_upgrade_domain
+        std::unique_ptr<ArrowSchema>
+            domain_schema = create_index_cols_info_schema(dim_infos);
+        auto domain_array = ArrowAdapter::make_arrow_array_parent(
+            dim_infos.size());
+        domain_array->children[0] = ArrowAdapter::make_arrow_array_child(
+            std::vector<uint32_t>({0, 0}));
+        domain_array->children[1] = ArrowAdapter::make_arrow_array_child(
+            std::vector<int64_t>({0, 0}));
+        auto domain_table = ArrowTable(
+            std::move(domain_array), std::move(domain_schema));
+
+        if (!use_current_domain) {
+            StatusAndReason check = sdf->can_upgrade_domain(
+                domain_table, "testing");
+            REQUIRE(check.first == false);
+            REQUIRE(
+                check.second ==
+                "testing for myuint32: new upper < old upper (downsize is "
+                "unsupported)");
+        } else {
+            StatusAndReason check = sdf->can_upgrade_soma_joinid_shape(
+                1, "testing");
+            // Must fail since this is too small.
+            REQUIRE(check.first == false);
+            REQUIRE(
+                check.second ==
+                "testing: dataframe already has its domain set.");
+        }
+
         sdf->close();
 
         // Resize
@@ -946,8 +1038,7 @@ TEST_CASE_METHOD(
         }
 
         // Check can_resize_soma_joinid_shape
-        std::pair<bool, std::string> check = sdf->can_resize_soma_joinid_shape(
-            1, "testing");
+        StatusAndReason check = sdf->can_resize_soma_joinid_shape(1, "testing");
         if (!use_current_domain) {
             REQUIRE(check.first == false);
             REQUIRE(
@@ -1101,6 +1192,37 @@ TEST_CASE_METHOD(
 
         sdf->close();
 
+        // Check can_upgrade_domain
+        sdf = open(OpenMode::read);
+        std::unique_ptr<ArrowSchema>
+            domain_schema = create_index_cols_info_schema(dim_infos);
+        auto domain_array = ArrowAdapter::make_arrow_array_parent(
+            dim_infos.size());
+        domain_array->children[0] = ArrowAdapter::make_arrow_array_child(
+            std::vector<int64_t>({0, 0}));
+        domain_array->children[1] = ArrowAdapter::make_arrow_array_child_string(
+            std::vector<std::string>({"a", "z"}));
+        auto domain_table = ArrowTable(
+            std::move(domain_array), std::move(domain_schema));
+        if (!use_current_domain) {
+            StatusAndReason check = sdf->can_upgrade_domain(
+                domain_table, "testing");
+            REQUIRE(check.first == false);
+            REQUIRE(
+                check.second ==
+                "testing for soma_joinid: new upper < old upper (downsize is "
+                "unsupported)");
+        } else {
+            StatusAndReason check = sdf->can_upgrade_soma_joinid_shape(
+                1, "testing");
+            // Must fail since this is too small.
+            REQUIRE(check.first == false);
+            REQUIRE(
+                check.second ==
+                "testing: dataframe already has its domain set.");
+        }
+        sdf->close();
+
         // Resize
         auto new_shape = int64_t{SOMA_JOINID_RESIZE_DIM_MAX + 1};
 
@@ -1194,8 +1316,7 @@ TEST_CASE_METHOD(
         REQUIRE(ned_str == std::vector<std::string>({"", ""}));
 
         // Check can_resize_soma_joinid_shape
-        std::pair<bool, std::string> check = sdf->can_resize_soma_joinid_shape(
-            1, "testing");
+        StatusAndReason check = sdf->can_resize_soma_joinid_shape(1, "testing");
         if (!use_current_domain) {
             REQUIRE(check.first == false);
             REQUIRE(
@@ -1332,6 +1453,37 @@ TEST_CASE_METHOD(
         REQUIRE(!actual.has_value());
         sdf->close();
 
+        // Check can_upgrade_domain
+        sdf = open(OpenMode::read);
+        std::unique_ptr<ArrowSchema>
+            domain_schema = create_index_cols_info_schema(dim_infos);
+        auto domain_array = ArrowAdapter::make_arrow_array_parent(
+            dim_infos.size());
+        domain_array->children[0] = ArrowAdapter::make_arrow_array_child_string(
+            std::vector<std::string>({"a", "z"}));
+        domain_array->children[1] = ArrowAdapter::make_arrow_array_child(
+            std::vector<uint32_t>({0, 0}));
+        auto domain_table = ArrowTable(
+            std::move(domain_array), std::move(domain_schema));
+        if (!use_current_domain) {
+            StatusAndReason check = sdf->can_upgrade_domain(
+                domain_table, "testing");
+            REQUIRE(check.first == false);
+            REQUIRE(
+                check.second ==
+                "testing for mystring: new lower > old lower (downsize is "
+                "unsupported)");
+        } else {
+            StatusAndReason check = sdf->can_upgrade_soma_joinid_shape(
+                1, "testing");
+            // Must fail since this is too small.
+            REQUIRE(check.first == false);
+            REQUIRE(
+                check.second ==
+                "testing: dataframe already has its domain set.");
+        }
+        sdf->close();
+
         // Resize
         auto new_shape = int64_t{SOMA_JOINID_RESIZE_DIM_MAX + 1};
 
@@ -1405,8 +1557,7 @@ TEST_CASE_METHOD(
         }
 
         // Check can_resize_soma_joinid_shape
-        std::pair<bool, std::string> check = sdf->can_resize_soma_joinid_shape(
-            0, "testing");
+        StatusAndReason check = sdf->can_resize_soma_joinid_shape(0, "testing");
         if (!use_current_domain) {
             REQUIRE(check.first == false);
             REQUIRE(