Repository: single-cell-data/TileDB-SOMA (language: python)

Issue:
[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[c++] `upgrade_domain` for `DataFrame`
**Issue and/or context:** As tracked on issue #2407 / [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

Note that the intended Python and R API changes are all agreed on and finalized as described in #2407.

**Changes:**

* This is `upgrade_domain` for `SOMADataFrame` at the C++ level

**Notes for Reviewer:**

This PR is a work in progress. It is not ready for review.




Discussion / hints:
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.


Fix patch:
diff --git a/apis/python/src/tiledbsoma/_dataframe.py b/apis/python/src/tiledbsoma/_dataframe.py
index fb59049922..8831dc4ae2 100644
--- a/apis/python/src/tiledbsoma/_dataframe.py
+++ b/apis/python/src/tiledbsoma/_dataframe.py
@@ -7,7 +7,17 @@
 Implementation of a SOMA DataFrame
 """
 import inspect
-from typing import Any, List, Optional, Sequence, Tuple, Type, Union, cast
+from typing import (
+    Any,
+    Dict,
+    List,
+    Optional,
+    Sequence,
+    Tuple,
+    Type,
+    Union,
+    cast,
+)
 
 import numpy as np
 import pyarrow as pa
@@ -484,6 +494,103 @@ def tiledbsoma_upgrade_soma_joinid_shape(
             )
             return (True, "")
 
+    def _upgrade_or_change_domain_helper(
+        self, newdomain: Domain, function_name_for_messages: str
+    ) -> Any:
+        """Converts the user-level tuple of low/high pairs into a pyarrow table suitable for calling libtiledbsoma."""
+
+        # Check user-provided domain against dataframe domain.
+        dim_names = self._tiledb_dim_names()
+        if len(dim_names) != len(newdomain):
+            raise ValueError(
+                f"{function_name_for_messages}: requested domain has length {len(dim_names)} but the dataframe's schema has index-column count {len(newdomain)}"
+            )
+
+        if any([slot is not None and len(slot) != 2 for slot in newdomain]):
+            raise ValueError(
+                f"{function_name_for_messages}: requested domain must have low,high pairs, or `None`, in each slot"
+            )
+
+        # From the dataframe's schema, extract the subschema for only index columns (TileDB dimensions).
+        full_schema = self.schema
+        dim_schema_list = []
+        for dim_name in dim_names:
+            dim_schema_list.append(full_schema.field(dim_name))
+        dim_schema = pa.schema(dim_schema_list)
+
+        # Convert the user's tuple of low/high pairs into a dict keyed by index-column name.
+        new_domain_dict: Dict[str, Domain] = {}
+        for dim_name, new_dom in zip(dim_names, newdomain):
+            # Domain can't be specified for strings (core constraint) so let them keystroke that easily.
+            if (
+                dim_schema.field(dim_name).type
+                in [
+                    pa.string(),
+                    pa.large_string(),
+                    pa.binary(),
+                    pa.large_binary(),
+                ]
+                and new_dom is None
+            ):
+                new_domain_dict[dim_name] = ("", "")  # type: ignore
+            else:
+                new_domain_dict[dim_name] = tuple(new_dom)  # type: ignore
+
+        # Return this as a pyarrow table. This has n columns where n is the number of
+        # index columns, and two rows: one row for the low values and one for the high values.
+        return pa.RecordBatch.from_pydict(new_domain_dict, schema=dim_schema)
+
+    def tiledbsoma_upgrade_domain(
+        self, newdomain: Domain, check_only: bool = False
+    ) -> StatusAndReason:
+        """Allows you to set the domain of a SOMA :class:`DataFrame``, when the
+        ``DataFrame`` does not have a domain set yet.  The argument must be a
+        tuple of pairs of low/high values for the desired domain, one pair per
+        index column. For string index columns, you must offer the low/high pair
+        as `("", "")`.  If ``check_only`` is ``True``, returns whether the
+        operation would succeed if attempted, and a reason why it would not.
+        """
+        pyarrow_domain_table = self._upgrade_or_change_domain_helper(
+            newdomain, "tiledbsoma_upgrade_domain"
+        )
+
+        if check_only:
+            return cast(
+                StatusAndReason,
+                self._handle._handle.can_upgrade_domain(
+                    pyarrow_domain_table, "tiledbsoma_upgrade_domain"
+                ),
+            )
+        else:
+            self._handle._handle.upgrade_domain(
+                pyarrow_domain_table, "tiledbsoma_upgrade_domain"
+            )
+            return (True, "")
+
+    def change_domain(
+        self, newdomain: Domain, check_only: bool = False
+    ) -> StatusAndReason:
+        """Allows you to enlarge the domain of a SOMA :class:`DataFrame``, when
+        the ``DataFrame`` already has a domain.  The argument must be a tuple of
+        pairs of low/high values for the desired domain, one pair per index
+        column. For string index columns, you must offer the low/high pair as
+        `("", "")`.  If ``check_only`` is ``True``, returns whether the
+        operation would succeed if attempted, and a reason why it would not.
+        """
+        pyarrow_domain_table = self._upgrade_or_change_domain_helper(
+            newdomain, "change_domain"
+        )
+        if check_only:
+            return cast(
+                StatusAndReason,
+                self._handle._handle.can_change_domain(
+                    pyarrow_domain_table, "change_domain"
+                ),
+            )
+        else:
+            self._handle._handle.change_domain(pyarrow_domain_table, "change_domain")
+            return (True, "")
+
     def __len__(self) -> int:
         """Returns the number of rows in the dataframe. Same as ``df.count``."""
         return self.count
diff --git a/apis/python/src/tiledbsoma/_tdb_handles.py b/apis/python/src/tiledbsoma/_tdb_handles.py
index aadbc11ef2..80bda19881 100644
--- a/apis/python/src/tiledbsoma/_tdb_handles.py
+++ b/apis/python/src/tiledbsoma/_tdb_handles.py
@@ -505,6 +505,28 @@ def can_upgrade_soma_joinid_shape(
         """Only implemented for DataFrame."""
         raise NotImplementedError
 
+    def upgrade_domain(
+        self, newdomain: Domain, function_name_for_messages: str
+    ) -> None:
+        """Only implemented for DataFrame."""
+        raise NotImplementedError
+
+    def can_upgrade_domain(
+        self, newdomain: Domain, function_name_for_messages: str
+    ) -> StatusAndReason:
+        """Only implemented for DataFrame."""
+        raise NotImplementedError
+
+    def change_domain(self, newdomain: Domain, function_name_for_messages: str) -> None:
+        """Only implemented for DataFrame."""
+        raise NotImplementedError
+
+    def can_change_domain(
+        self, newdomain: Domain, function_name_for_messages: str
+    ) -> StatusAndReason:
+        """Only implemented for DataFrame."""
+        raise NotImplementedError
+
 
 class DataFrameWrapper(SOMAArrayWrapper[clib.SOMADataFrame]):
     """Wrapper around a Pybind11 SOMADataFrame handle."""
@@ -567,6 +589,34 @@ def can_upgrade_soma_joinid_shape(
             ),
         )
 
+    def upgrade_domain(
+        self, newdomain: Domain, function_name_for_messages: str
+    ) -> None:
+        """Wrapper-class internals"""
+        self._handle.upgrade_domain(newdomain, function_name_for_messages)
+
+    def can_upgrade_domain(
+        self, newdomain: Domain, function_name_for_messages: str
+    ) -> StatusAndReason:
+        """Wrapper-class internals"""
+        return cast(
+            StatusAndReason,
+            self._handle.can_upgrade_domain(newdomain, function_name_for_messages),
+        )
+
+    def change_domain(self, newdomain: Domain, function_name_for_messages: str) -> None:
+        """Wrapper-class internals"""
+        self._handle.change_domain(newdomain, function_name_for_messages)
+
+    def can_change_domain(
+        self, newdomain: Domain, function_name_for_messages: str
+    ) -> StatusAndReason:
+        """Wrapper-class internals"""
+        return cast(
+            StatusAndReason,
+            self._handle.can_change_domain(newdomain, function_name_for_messages),
+        )
+
 
 class PointCloudDataFrameWrapper(SOMAArrayWrapper[clib.SOMAPointCloudDataFrame]):
     """Wrapper around a Pybind11 SOMAPointCloudDataFrame handle."""
diff --git a/apis/python/src/tiledbsoma/soma_dataframe.cc b/apis/python/src/tiledbsoma/soma_dataframe.cc
index c7075ce564..d55a2ad3d2 100644
--- a/apis/python/src/tiledbsoma/soma_dataframe.cc
+++ b/apis/python/src/tiledbsoma/soma_dataframe.cc
@@ -216,6 +216,110 @@ void load_soma_dataframe(py::module& m) {
                 }
             },
             "newshape"_a,
+            "function_name_for_messages"_a)
+
+        .def(
+            "upgrade_domain",
+            [](SOMADataFrame& sdf,
+               py::object pyarrow_domain_table,
+               std::string function_name_for_messages) {
+                ArrowArray pyarrow_domain_array;
+                ArrowSchema pyarrow_domain_schema;
+                uintptr_t nanoarrow_domain_array_ptr =
+                    (uintptr_t)(&pyarrow_domain_array);
+                uintptr_t nanoarrow_domain_schema_ptr =
+                    (uintptr_t)(&pyarrow_domain_schema);
+                pyarrow_domain_table.attr("_export_to_c")(
+                    nanoarrow_domain_array_ptr, nanoarrow_domain_schema_ptr);
+                ArrowTable nanoarrow_domain_table(
+                    std::make_unique<ArrowArray>(pyarrow_domain_array),
+                    std::make_unique<ArrowSchema>(pyarrow_domain_schema));
+                try {
+                    sdf.upgrade_domain(
+                        nanoarrow_domain_table, function_name_for_messages);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            },
+            "pyarrow_domain_table"_a,
+            "function_name_for_messages"_a)
+
+        .def(
+            "can_upgrade_domain",
+            [](SOMADataFrame& sdf,
+               py::object pyarrow_domain_table,
+               std::string function_name_for_messages) {
+                ArrowArray pyarrow_domain_array;
+                ArrowSchema pyarrow_domain_schema;
+                uintptr_t nanoarrow_domain_array_ptr =
+                    (uintptr_t)(&pyarrow_domain_array);
+                uintptr_t nanoarrow_domain_schema_ptr =
+                    (uintptr_t)(&pyarrow_domain_schema);
+                pyarrow_domain_table.attr("_export_to_c")(
+                    nanoarrow_domain_array_ptr, nanoarrow_domain_schema_ptr);
+                ArrowTable nanoarrow_domain_table(
+                    std::make_unique<ArrowArray>(pyarrow_domain_array),
+                    std::make_unique<ArrowSchema>(pyarrow_domain_schema));
+                try {
+                    return sdf.can_upgrade_domain(
+                        nanoarrow_domain_table, function_name_for_messages);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            },
+            "pyarrow_domain_table"_a,
+            "function_name_for_messages"_a)
+
+        .def(
+            "change_domain",
+            [](SOMADataFrame& sdf,
+               py::object pyarrow_domain_table,
+               std::string function_name_for_messages) {
+                ArrowArray pyarrow_domain_array;
+                ArrowSchema pyarrow_domain_schema;
+                uintptr_t nanoarrow_domain_array_ptr =
+                    (uintptr_t)(&pyarrow_domain_array);
+                uintptr_t nanoarrow_domain_schema_ptr =
+                    (uintptr_t)(&pyarrow_domain_schema);
+                pyarrow_domain_table.attr("_export_to_c")(
+                    nanoarrow_domain_array_ptr, nanoarrow_domain_schema_ptr);
+                ArrowTable nanoarrow_domain_table(
+                    std::make_unique<ArrowArray>(pyarrow_domain_array),
+                    std::make_unique<ArrowSchema>(pyarrow_domain_schema));
+                try {
+                    sdf.change_domain(
+                        nanoarrow_domain_table, function_name_for_messages);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            },
+            "pyarrow_domain_table"_a,
+            "function_name_for_messages"_a)
+
+        .def(
+            "can_change_domain",
+            [](SOMADataFrame& sdf,
+               py::object pyarrow_domain_table,
+               std::string function_name_for_messages) {
+                ArrowArray pyarrow_domain_array;
+                ArrowSchema pyarrow_domain_schema;
+                uintptr_t nanoarrow_domain_array_ptr =
+                    (uintptr_t)(&pyarrow_domain_array);
+                uintptr_t nanoarrow_domain_schema_ptr =
+                    (uintptr_t)(&pyarrow_domain_schema);
+                pyarrow_domain_table.attr("_export_to_c")(
+                    nanoarrow_domain_array_ptr, nanoarrow_domain_schema_ptr);
+                ArrowTable nanoarrow_domain_table(
+                    std::make_unique<ArrowArray>(pyarrow_domain_array),
+                    std::make_unique<ArrowSchema>(pyarrow_domain_schema));
+                try {
+                    return sdf.can_change_domain(
+                        nanoarrow_domain_table, function_name_for_messages);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            },
+            "pyarrow_domain_table"_a,
             "function_name_for_messages"_a);
 }
 
diff --git a/libtiledbsoma/src/soma/soma_array.h b/libtiledbsoma/src/soma/soma_array.h
index 3bae58a912..77834ffd3b 100644
--- a/libtiledbsoma/src/soma/soma_array.h
+++ b/libtiledbsoma/src/soma/soma_array.h
@@ -1285,21 +1285,27 @@ class SOMAArray : public SOMAObject {
     }
 
     /**
-     * This is for SOMADataFrame.
-     * XXX comment more
+     * This is for SOMADataFrame.  While resize_soma_joinid_shape allows the
+     * user to do up the soma_joinid domain slot, without needing to specify
+     * the rest (which is the common operation for experiment-level resize)
+     * this allows the full-generality resize-every-index-column case
+     * (which only applies to variant-indexed/non-standard dataframes).
      */
-    void upgrade_domain(
+    void change_domain(
         const ArrowTable& newdomain, std::string function_name_for_messages) {
-        _set_domain_helper(newdomain, false, function_name_for_messages);
+        _set_domain_helper(newdomain, true, function_name_for_messages);
     }
 
     /**
-     * This is for SOMADataFrame.
-     * XXX comment more
+     * This is for SOMADataFrame.  While upgrade_soma_joinid_shape allows the
+     * user to do up the soma_joinid domain slot, without needing to specify
+     * the rest (which is the common operation for experiment-level resize)
+     * this allows the full-generality resize-every-index-column case
+     * (which only applies to variant-indexed/non-standard dataframes).
      */
-    void change_domain(
+    void upgrade_domain(
         const ArrowTable& newdomain, std::string function_name_for_messages) {
-        _set_domain_helper(newdomain, true, function_name_for_messages);
+        _set_domain_helper(newdomain, false, function_name_for_messages);
     }
 
    protected:
@@ -1447,22 +1453,51 @@ class SOMAArray : public SOMAObject {
         const T& new_lo = new_lo_hi[0];
         const T& new_hi = new_lo_hi[1];
 
-        // It's difficult to use fmt::format within a header file since the
-        // include path to logger.h 'moves around' depending on which source
+        // If we're checking against the core current domain: the user-provided
+        // domain must contain the core current domain.
+        //
+        // If we're checking against the core (max) domain: the user-provided
+        // domain must be contained within the core (max) domain.
+
+        // Note: It's difficult to use fmt::format within a header file since
+        // the include path to logger.h 'moves around' depending on which source
         // file included us.
         //
         // TODO: once we're on C++ 20, just use std::format here and include
         // things like "old ({}, {}) new ({}, {})".
+
         if (new_lo > new_hi) {
-            return std::pair(false, "new lower > new upper");
-        }
-        if (new_lo > old_lo) {
             return std::pair(
-                false, "new lower > old lower (downsize is unsupported)");
+                false,
+                "index-column name " + dim_name + ": new lower > new upper");
         }
-        if (new_hi < old_hi) {
-            return std::pair(
-                false, "new upper < old upper (downsize is unsupported)");
+
+        if (check_current_domain) {
+            if (new_lo > old_lo) {
+                return std::pair(
+                    false,
+                    "index-column name " + dim_name +
+                        ": new lower > old lower (downsize is unsupported)");
+            }
+            if (new_hi < old_hi) {
+                return std::pair(
+                    false,
+                    "index-column name " + dim_name +
+                        ": new upper < old upper (downsize is unsupported)");
+            }
+        } else {
+            if (new_lo < old_lo) {
+                return std::pair(
+                    false,
+                    "index-column name " + dim_name +
+                        ": new lower < limit lower");
+            }
+            if (new_hi > old_hi) {
+                return std::pair(
+                    false,
+                    "index-column name " + dim_name +
+                        ": new upper > limit upper");
+            }
         }
         return std::pair(true, "");
     }


Test patch:
diff --git a/apis/python/tests/test_shape.py b/apis/python/tests/test_shape.py
index d96bde6ab9..dbfc651bff 100644
--- a/apis/python/tests/test_shape.py
+++ b/apis/python/tests/test_shape.py
@@ -358,6 +358,103 @@ def test_dataframe_basics(tmp_path, soma_joinid_domain, index_column_names):
             sdf.write(data)
 
 
+def test_domain_mods(tmp_path):
+    if not tiledbsoma._flags.NEW_SHAPE_FEATURE_FLAG_ENABLED:
+        return
+
+    uri = tmp_path.as_posix()
+
+    schema = pa.schema(
+        [
+            ("soma_joinid", pa.int64()),
+            ("mystring", pa.string()),
+            ("myint", pa.int16()),
+            ("myfloat", pa.float32()),
+            ("mybool", pa.bool_()),  # not supported as an index type
+        ]
+    )
+    index_column_names = ["soma_joinid", "mystring", "myint", "myfloat"]
+
+    domain_for_create = [
+        [0, 3],
+        None,
+        [20, 50],
+        [0.0, 6.0],
+    ]
+
+    data_dict = {
+        "soma_joinid": [0, 1, 2, 3],
+        "mystring": ["a", "b", "a", "b"],
+        "myint": [20, 30, 40, 50],
+        "myfloat": [1.0, 2.5, 4.0, 5.5],
+        "mybool": [True, False, True, True],
+    }
+
+    data = pa.Table.from_pydict(data_dict)
+
+    with tiledbsoma.DataFrame.create(
+        uri,
+        schema=schema,
+        index_column_names=index_column_names,
+        domain=domain_for_create,
+    ) as sdf:
+        sdf.write(data)
+
+    # Check "expand" to same
+    with tiledbsoma.DataFrame.open(uri, "w") as sdf:
+        newdomain = [[0, 3], None, [20, 50], [0.0, 6.0]]
+        ok, msg = sdf.change_domain(newdomain, check_only=True)
+        assert ok
+        assert msg == ""
+
+    # Shrink
+    with tiledbsoma.DataFrame.open(uri, "w") as sdf:
+        newdomain = [[0, 2], None, [20, 50], [0.0, 6.0]]
+        ok, msg = sdf.change_domain(newdomain, check_only=True)
+        assert not ok
+        assert "downsize is unsupported" in msg
+
+    with tiledbsoma.DataFrame.open(uri, "w") as sdf:
+        newdomain = [[0, 3], None, [20, 40], [0.0, 6.0]]
+        ok, msg = sdf.change_domain(newdomain, check_only=True)
+        assert not ok
+        assert "downsize is unsupported" in msg
+
+    with tiledbsoma.DataFrame.open(uri, "w") as sdf:
+        newdomain = [[0, 3], None, [20, 50], [1.0, 6.0]]
+        ok, msg = sdf.change_domain(newdomain, check_only=True)
+        assert not ok
+        assert "downsize is unsupported" in msg
+
+    # String domain cannot be specified
+    with tiledbsoma.DataFrame.open(uri, "w") as sdf:
+        newdomain = [
+            [0, 3],
+            ["a", "z"],
+            [20, 50],
+            [0.0, 6.0],
+        ]
+        ok, msg = sdf.change_domain(newdomain, check_only=True)
+        assert not ok
+        assert "domain cannot be set for string index columns" in msg
+
+    # All clear
+    with tiledbsoma.DataFrame.open(uri, "w") as sdf:
+        newdomain = [[0, 9], None, [0, 100], [-10.0, 10.0]]
+        ok, msg = sdf.change_domain(newdomain, check_only=True)
+        assert ok
+        assert msg == ""
+        sdf.change_domain(newdomain)
+
+    # Check for success
+    with tiledbsoma.DataFrame.open(uri, "r") as sdf:
+        dom = sdf.domain
+        assert dom[0] == (0, 9)
+        assert dom[1] == ("", "")
+        assert dom[2] == (0, 100)
+        assert dom[3] == (-10.0, 10.0)
+
+
 @pytest.mark.parametrize("has_shapes", [False, True])
 def test_canned_experiments(tmp_path, has_shapes):
     uri = tmp_path.as_posix()
@@ -437,6 +534,28 @@ def _check_ndarray(ndarray, has_shapes, expected_shape):
     assert "[SparseNDArray] ms/RNA/obsp/distances" in body
     assert "ms/RNA/obsm/X_draw_graph_fr" in body
 
+    # Check upgrade_domain for dataframes
+    with tiledbsoma.Experiment.open(uri, "w") as exp:
+
+        ok, msg = exp.obs.tiledbsoma_upgrade_domain([[10, 4]], check_only=True)
+        if has_shapes:
+            assert not ok
+            assert "dataframe already has a domain" in msg
+        else:
+            assert not ok
+            assert "new lower > new upper" in msg
+
+        ok, msg = exp.obs.tiledbsoma_upgrade_domain([[0, 1]], check_only=True)
+        if has_shapes:
+            assert not ok
+            assert "dataframe already has a domain" in msg
+        else:
+            assert ok
+            assert msg == ""
+
+        with pytest.raises(ValueError):
+            exp.obs.tiledbsoma_upgrade_domain([[0, 1, 2]], check_only=True)
+
     # Check dry run of tiledbsoma.io.upgrade_experiment_shapes
     handle = io.StringIO()
     upgradeable = tiledbsoma.io.upgrade_experiment_shapes(
diff --git a/libtiledbsoma/test/unit_soma_dataframe.cc b/libtiledbsoma/test/unit_soma_dataframe.cc
index 17272aafeb..2d5b27e23a 100644
--- a/libtiledbsoma/test/unit_soma_dataframe.cc
+++ b/libtiledbsoma/test/unit_soma_dataframe.cc
@@ -623,6 +623,7 @@ TEST_CASE_METHOD(
                 domain_schema = create_index_cols_info_schema(dim_infos);
             auto domain_array = ArrowAdapter::make_arrow_array_parent(
                 dim_infos.size());
+            // OK since there currently is no shape set:
             domain_array->children[0] = ArrowAdapter::make_arrow_array_child(
                 std::vector<int64_t>({0, 0}));
             auto domain_table = ArrowTable(
@@ -630,13 +631,8 @@ TEST_CASE_METHOD(
             if (!use_current_domain) {
                 StatusAndReason check = sdf->can_upgrade_domain(
                     domain_table, "testing");
-                REQUIRE(check.first == false);
-                REQUIRE(
-                    check.second ==
-                    "testing for soma_joinid: new upper < old upper "
-                    "(downsize "
-                    "is "
-                    "unsupported)");
+                REQUIRE(check.first == true);
+                REQUIRE(check.second == "");
             } else {
                 StatusAndReason check = sdf->can_upgrade_soma_joinid_shape(
                     1, "testing");
@@ -999,6 +995,7 @@ TEST_CASE_METHOD(
                 domain_schema = create_index_cols_info_schema(dim_infos);
             auto domain_array = ArrowAdapter::make_arrow_array_parent(
                 dim_infos.size());
+            // OK since there currently is no shape set:
             domain_array->children[0] = ArrowAdapter::make_arrow_array_child(
                 std::vector<uint32_t>({0, 0}));
             domain_array->children[1] = ArrowAdapter::make_arrow_array_child(
@@ -1009,12 +1006,8 @@ TEST_CASE_METHOD(
             if (!use_current_domain) {
                 StatusAndReason check = sdf->can_upgrade_domain(
                     domain_table, "testing");
-                REQUIRE(check.first == false);
-                REQUIRE(
-                    check.second ==
-                    "testing for myuint32: new upper < old upper (downsize "
-                    "is "
-                    "unsupported)");
+                REQUIRE(check.first == true);
+                REQUIRE(check.second == "");
             } else {
                 StatusAndReason check = sdf->can_upgrade_soma_joinid_shape(
                     1, "testing");
@@ -1356,9 +1349,8 @@ TEST_CASE_METHOD(
                     REQUIRE(check.first == false);
                     REQUIRE(
                         check.second ==
-                        "testing for soma_joinid: new upper < old upper "
-                        "(downsize is "
-                        "unsupported)");
+                        "testing for mystring: domain cannot be set for string "
+                        "index columns: please use (\"\", \"\")");
                 } else {
                     StatusAndReason check = sdf->can_upgrade_soma_joinid_shape(
                         1, "testing");