Repository: single-cell-data/TileDB-SOMA (language: python)

Issue:
[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[python] New-shape testing for `tiledbsoma.io` [WIP]
**Issue and/or context:** As tracked on issue #2407 / [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048). Atop #2950.

Note that the intended Python and R API changes are all agreed on and finalized as described in #2407.

**Changes:**

Tracks the things needing changed for `tiledbsoma.io` wherein explicit resize will play a key role.

**Notes for Reviewer:**

This PR is a work in progress. It is not ready for review.


[c++] Simplify internal API for dataframe-resizer
**Issue and/or context:** As tracked on issue #2407 / [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

Note that the intended Python and R API changes are all agreed on and finalized as described in #2407.

**Changes:**

While the full-generality `upgrade_domain` is tasked out on #2407, the foremost use-case for append mode, and `tests/testthat/test-write-soma-resume.R` for PR #3089, will need a simpller keystroke-saver which is already implemented in C++.

What this existing method does in C++ is resize the `soma_joinid` dim, if it's a dim, and leave the other dims as-is; if `soma_joinid` is not a dim, leave it as-is. This exists in C++, it's unit-tested in C++, and it's ready (and it's time) to be plumbed up to Python and R as well.

I realize now that having C++ `maybe_resize_joinid` take a length-1 vector, and assert that it's length-1, is silly -- it should simply take a single `int64_t`. This PR does just that. It also removes the confusing prefix `maybe_`.

**Notes for Reviewer:**



Discussion / hints:
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.



Fix patch:
diff --git a/apis/python/src/tiledbsoma/_dataframe.py b/apis/python/src/tiledbsoma/_dataframe.py
index b76f5fa920..cfd9f26a4e 100644
--- a/apis/python/src/tiledbsoma/_dataframe.py
+++ b/apis/python/src/tiledbsoma/_dataframe.py
@@ -417,6 +417,18 @@ def tiledbsoma_has_upgraded_domain(self) -> bool:
         """
         return self._handle.tiledbsoma_has_upgraded_domain
 
+    def resize_soma_joinid(self, newshape: int) -> None:
+        """Increases the shape of the dataframe on the ``soma_joinid`` index
+        column, if it indeed is an index column, leaving all other index columns
+        as-is. If the ``soma_joinid`` is not an index column, no change is made.
+        This is a special case of ``upgrade_domain`` (WIP for 1.15), but simpler
+        to keystroke, and handles the most common case for dataframe domain
+        expansion.  Raises an error if the dataframe doesn't already have a
+        domain: in that case please call ``tiledbsoma_upgrade_domain`` (WIP for
+        1.15).
+        """
+        self._handle._handle.resize_soma_joinid(newshape)
+
     def __len__(self) -> int:
         """Returns the number of rows in the dataframe. Same as ``df.count``."""
         return self.count
diff --git a/apis/python/src/tiledbsoma/_tdb_handles.py b/apis/python/src/tiledbsoma/_tdb_handles.py
index 8e581227e5..decaab86c3 100644
--- a/apis/python/src/tiledbsoma/_tdb_handles.py
+++ b/apis/python/src/tiledbsoma/_tdb_handles.py
@@ -450,6 +450,10 @@ def tiledbsoma_upgrade_shape(self, newshape: Sequence[Union[int, None]]) -> None
         """Not implemented for DataFrame."""
         raise NotImplementedError
 
+    def resize_soma_joinid(self, newshape: int) -> None:
+        """Only implemented for DataFrame."""
+        raise NotImplementedError
+
 
 class DataFrameWrapper(SOMAArrayWrapper[clib.SOMADataFrame]):
     """Wrapper around a Pybind11 SOMADataFrame handle."""
@@ -496,6 +500,18 @@ def tiledbsoma_has_upgraded_domain(self) -> bool:
         """
         return cast(bool, self._handle.tiledbsoma_has_upgraded_domain)
 
+    def resize_soma_joinid(self, newshape: int) -> None:
+        """Increases the shape of the dataframe on the ``soma_joinid`` index
+        column, if it indeed is an index column, leaving all other index columns
+        as-is. If the ``soma_joinid`` is not an index column, no change is made.
+        This is a special case of ``upgrade_domain`` (WIP for 1.15), but simpler
+        to keystroke, and handles the most common case for dataframe domain
+        expansion.  Raises an error if the dataframe doesn't already have a
+        domain: in that case please call ``tiledbsoma_upgrade_domain`` (WIP for
+        1.15).
+        """
+        self._handle.resize_soma_joinid(newshape)
+
 
 class DenseNDArrayWrapper(SOMAArrayWrapper[clib.SOMADenseNDArray]):
     """Wrapper around a Pybind11 DenseNDArrayWrapper handle."""
diff --git a/apis/python/src/tiledbsoma/soma_dataframe.cc b/apis/python/src/tiledbsoma/soma_dataframe.cc
index e7f7ad84d9..f4275fc77a 100644
--- a/apis/python/src/tiledbsoma/soma_dataframe.cc
+++ b/apis/python/src/tiledbsoma/soma_dataframe.cc
@@ -155,6 +155,17 @@ void load_soma_dataframe(py::module& m) {
             "maybe_soma_joinid_maxshape",
             &SOMADataFrame::maybe_soma_joinid_maxshape)
         .def_property_readonly(
-            "tiledbsoma_has_upgraded_domain", &SOMAArray::has_current_domain);
+            "tiledbsoma_has_upgraded_domain", &SOMAArray::has_current_domain)
+
+        .def(
+            "resize_soma_joinid",
+            [](SOMADataFrame& sdf, int64_t newshape) {
+                try {
+                    sdf.resize_soma_joinid(newshape);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            },
+            "newshape"_a);
 }
 }  // namespace libtiledbsomacpp
diff --git a/apis/r/R/RcppExports.R b/apis/r/R/RcppExports.R
index 326c898385..c1d74177a6 100644
--- a/apis/r/R/RcppExports.R
+++ b/apis/r/R/RcppExports.R
@@ -218,6 +218,10 @@ resize <- function(uri, new_shape, ctxxp) {
     invisible(.Call(`_tiledbsoma_resize`, uri, new_shape, ctxxp))
 }
 
+resize_soma_joinid <- function(uri, new_shape, ctxxp) {
+    invisible(.Call(`_tiledbsoma_resize_soma_joinid`, uri, new_shape, ctxxp))
+}
+
 tiledbsoma_upgrade_shape <- function(uri, new_shape, ctxxp) {
     invisible(.Call(`_tiledbsoma_tiledbsoma_upgrade_shape`, uri, new_shape, ctxxp))
 }
diff --git a/apis/r/R/SOMADataFrame.R b/apis/r/R/SOMADataFrame.R
index 35dfabef62..0a8c669c99 100644
--- a/apis/r/R/SOMADataFrame.R
+++ b/apis/r/R/SOMADataFrame.R
@@ -395,6 +395,26 @@ SOMADataFrame <- R6::R6Class(
     #' @return Logical
     tiledbsoma_has_upgraded_domain = function() {
       has_current_domain(self$uri, private$.soma_context)
+    },
+
+    #' @description Increases the shape of the dataframe on the ``soma_joinid``
+    #' index column, if it indeed is an index column, leaving all other index
+    #' columns as-is. If the ``soma_joinid`` is not an index column, no change is
+    #' made.  This is a special case of ``upgrade_domain`` (WIP for 1.15), but
+    #' simpler to keystroke, and handles the most common case for dataframe
+    #' domain expansion.  Raises an error if the dataframe doesn't already have a
+    #' domain: in that case please call ``tiledbsoma_upgrade_domain`` (WIP for
+    #' 1.15).
+    #' @param new_shape An integer, greater than or equal to 1 + the
+    #' `soma_joinid` domain slot.
+    #' @return No return value
+    resize_soma_joinid = function(new_shape) {
+
+      stopifnot("'new_shape' must be an integer" = rlang::is_integerish(new_shape, n = 1) ||
+        (bit64::is.integer64(new_shape) && length(new_shape) == 1)
+      )
+      # Checking slotwise new shape >= old shape, and <= max_shape, is already done in libtiledbsoma
+      invisible(resize_soma_joinid(self$uri, new_shape, private$.soma_context))
     }
 
   ),
diff --git a/apis/r/src/RcppExports.cpp b/apis/r/src/RcppExports.cpp
index ad953fe658..832bdb4199 100644
--- a/apis/r/src/RcppExports.cpp
+++ b/apis/r/src/RcppExports.cpp
@@ -488,6 +488,18 @@ BEGIN_RCPP
     return R_NilValue;
 END_RCPP
 }
+// resize_soma_joinid
+void resize_soma_joinid(const std::string& uri, Rcpp::NumericVector new_shape, Rcpp::XPtr<somactx_wrap_t> ctxxp);
+RcppExport SEXP _tiledbsoma_resize_soma_joinid(SEXP uriSEXP, SEXP new_shapeSEXP, SEXP ctxxpSEXP) {
+BEGIN_RCPP
+    Rcpp::RNGScope rcpp_rngScope_gen;
+    Rcpp::traits::input_parameter< const std::string& >::type uri(uriSEXP);
+    Rcpp::traits::input_parameter< Rcpp::NumericVector >::type new_shape(new_shapeSEXP);
+    Rcpp::traits::input_parameter< Rcpp::XPtr<somactx_wrap_t> >::type ctxxp(ctxxpSEXP);
+    resize_soma_joinid(uri, new_shape, ctxxp);
+    return R_NilValue;
+END_RCPP
+}
 // tiledbsoma_upgrade_shape
 void tiledbsoma_upgrade_shape(const std::string& uri, Rcpp::NumericVector new_shape, Rcpp::XPtr<somactx_wrap_t> ctxxp);
 RcppExport SEXP _tiledbsoma_tiledbsoma_upgrade_shape(SEXP uriSEXP, SEXP new_shapeSEXP, SEXP ctxxpSEXP) {
@@ -708,6 +720,7 @@ static const R_CallMethodDef CallEntries[] = {
     {"_tiledbsoma_ndim", (DL_FUNC) &_tiledbsoma_ndim, 2},
     {"_tiledbsoma_c_dimnames", (DL_FUNC) &_tiledbsoma_c_dimnames, 2},
     {"_tiledbsoma_resize", (DL_FUNC) &_tiledbsoma_resize, 3},
+    {"_tiledbsoma_resize_soma_joinid", (DL_FUNC) &_tiledbsoma_resize_soma_joinid, 3},
     {"_tiledbsoma_tiledbsoma_upgrade_shape", (DL_FUNC) &_tiledbsoma_tiledbsoma_upgrade_shape, 3},
     {"_tiledbsoma_sr_setup", (DL_FUNC) &_tiledbsoma_sr_setup, 10},
     {"_tiledbsoma_sr_complete", (DL_FUNC) &_tiledbsoma_sr_complete, 1},
diff --git a/apis/r/src/rinterface.cpp b/apis/r/src/rinterface.cpp
index 12eb0f3d6f..2386db3324 100644
--- a/apis/r/src/rinterface.cpp
+++ b/apis/r/src/rinterface.cpp
@@ -389,6 +389,18 @@ void resize(
     sr->close();
 }
 
+// [[Rcpp::export]]
+void resize_soma_joinid(
+    const std::string& uri,
+    Rcpp::NumericVector new_shape,
+    Rcpp::XPtr<somactx_wrap_t> ctxxp) {
+    // This function is solely for SOMADataFrame.
+    auto sr = tdbs::SOMADataFrame::open(uri, OpenMode::write, ctxxp->ctxptr);
+    std::vector<int64_t> new_shape_i64 = i64_from_rcpp_numeric(new_shape);
+    sr->resize_soma_joinid(new_shape_i64[0]);
+    sr->close();
+}
+
 // [[Rcpp::export]]
 void tiledbsoma_upgrade_shape(
     const std::string& uri,
diff --git a/libtiledbsoma/src/soma/soma_array.cc b/libtiledbsoma/src/soma/soma_array.cc
index d341a30e19..fbcd1695f2 100644
--- a/libtiledbsoma/src/soma/soma_array.cc
+++ b/libtiledbsoma/src/soma/soma_array.cc
@@ -1471,7 +1471,7 @@ void SOMAArray::_set_current_domain_from_shape(
     schema_evolution.array_evolve(uri_);
 }
 
-void SOMAArray::maybe_resize_soma_joinid(const std::vector<int64_t>& newshape) {
+void SOMAArray::resize_soma_joinid(int64_t newshape) {
     if (mq_->query_type() != TILEDB_WRITE) {
         throw TileDBSOMAError(
             "[SOMAArray::resize] array must be opened in write mode");
@@ -1480,12 +1480,6 @@ void SOMAArray::maybe_resize_soma_joinid(const std::vector<int64_t>& newshape) {
     ArraySchema schema = arr_->schema();
     Domain domain = schema.domain();
     unsigned ndim = domain.ndim();
-    if (newshape.size() != 1) {
-        throw TileDBSOMAError(fmt::format(
-            "[SOMAArray::resize]: newshape has dimension count {}; needed 1",
-            newshape.size(),
-            ndim));
-    }
 
     auto tctx = ctx_->tiledb_ctx();
     CurrentDomain old_current_domain = ArraySchemaExperimental::current_domain(
@@ -1498,7 +1492,7 @@ void SOMAArray::maybe_resize_soma_joinid(const std::vector<int64_t>& newshape) {
     for (unsigned i = 0; i < ndim; i++) {
         if (domain.dimension(i).name() == "soma_joinid") {
             ndrect.set_range<int64_t>(
-                domain.dimension(i).name(), 0, newshape[0] - 1);
+                domain.dimension(i).name(), 0, newshape - 1);
         }
     }
 
diff --git a/libtiledbsoma/src/soma/soma_array.h b/libtiledbsoma/src/soma/soma_array.h
index 5af4d0422a..67903bbeb6 100644
--- a/libtiledbsoma/src/soma/soma_array.h
+++ b/libtiledbsoma/src/soma/soma_array.h
@@ -1066,7 +1066,7 @@ class SOMAArray : public SOMAObject {
      * @return Throws if the requested shape exceeds the array's create-time
      * maxshape. Throws if the array does not have current-domain support.
      */
-    void maybe_resize_soma_joinid(const std::vector<int64_t>& newshape);
+    void resize_soma_joinid(int64_t newshape);
 
    protected:
     // These two are for use nominally by SOMADataFrame. This could be moved in


Test patch:
diff --git a/apis/python/tests/test_shape.py b/apis/python/tests/test_shape.py
index 80b7262c60..e02c543650 100644
--- a/apis/python/tests/test_shape.py
+++ b/apis/python/tests/test_shape.py
@@ -134,7 +134,7 @@ def test_sparse_nd_array_basics(
                 table = pa.Table.from_pydict(dikt)
                 snda.write(table)
 
-        # Test reasonable resize
+        # Test resize
         new_shape = tuple([arg_shape[i] + 50 for i in range(ndim)])
         with tiledbsoma.SparseNDArray.open(uri, "w") as snda:
             snda.resize(new_shape)
@@ -216,14 +216,14 @@ def test_dataframe_basics(tmp_path, soma_joinid_domain, index_column_names):
         ]
     )
 
-    data = pa.Table.from_pydict(
-        {
-            "soma_joinid": [0, 1, 2, 3],
-            "mystring": ["a", "b", "a", "b"],
-            "myint": [20, 30, 40, 50],
-            "myfloat": [1.0, 2.5, 4.0, 5.5],
-        }
-    )
+    data_dict = {
+        "soma_joinid": [0, 1, 2, 3],
+        "mystring": ["a", "b", "a", "b"],
+        "myint": [20, 30, 40, 50],
+        "myfloat": [1.0, 2.5, 4.0, 5.5],
+    }
+
+    data = pa.Table.from_pydict(data_dict)
 
     domain_slots = {
         "soma_joinid": soma_joinid_domain,
@@ -232,6 +232,8 @@ def test_dataframe_basics(tmp_path, soma_joinid_domain, index_column_names):
         "myfloat": (-999.5, 999.5),
     }
 
+    has_soma_joinid_dim = "soma_joinid" in index_column_names
+
     domain = tuple([domain_slots[name] for name in index_column_names])
 
     soma_joinid_coords = data["soma_joinid"]
@@ -265,3 +267,43 @@ def test_dataframe_basics(tmp_path, soma_joinid_domain, index_column_names):
                 assert sdf._maybe_soma_joinid_maxshape is None
 
         assert len(sdf.non_empty_domain()) == len(index_column_names)
+
+        # This may be None if soma_joinid is not an index column
+        shape_at_create = sdf._maybe_soma_joinid_shape
+
+    if tiledbsoma._flags.NEW_SHAPE_FEATURE_FLAG_ENABLED:
+
+        # Test resize down
+        new_shape = 0
+        with tiledbsoma.DataFrame.open(uri, "w") as sdf:
+            if has_soma_joinid_dim:
+                # TODO: check draft spec
+                # with pytest.raises(ValueError):
+                with pytest.raises(tiledbsoma.SOMAError):
+                    sdf.resize_soma_joinid(new_shape)
+            else:
+                sdf.resize_soma_joinid(new_shape)
+
+        with tiledbsoma.DataFrame.open(uri) as sdf:
+            assert sdf._maybe_soma_joinid_shape == shape_at_create
+
+        # Test writes out of bounds, before resize
+        offset = shape_at_create if has_soma_joinid_dim else 100
+        data_dict["soma_joinid"] = [e + offset for e in data_dict["soma_joinid"]]
+        data = pa.Table.from_pydict(data_dict)
+
+        with tiledbsoma.DataFrame.open(uri, "w") as sdf:
+            if has_soma_joinid_dim:
+                with pytest.raises(tiledbsoma.SOMAError):
+                    sdf.write(data)
+            else:
+                sdf.write(data)
+
+        # Test resize
+        new_shape = 0 if shape_at_create is None else shape_at_create + 100
+        with tiledbsoma.DataFrame.open(uri, "w") as sdf:
+            sdf.resize_soma_joinid(new_shape)
+
+        # Test writes out of old bounds, within new bounds, after resize
+        with tiledbsoma.DataFrame.open(uri, "w") as sdf:
+            sdf.write(data)
diff --git a/apis/r/tests/testthat/test-shape.R b/apis/r/tests/testthat/test-shape.R
index e4ed2d93ed..f4959f18d8 100644
--- a/apis/r/tests/testthat/test-shape.R
+++ b/apis/r/tests/testthat/test-shape.R
@@ -10,9 +10,9 @@ test_that("SOMADataFrame shape", {
   )
 
   domain_at_create_choices  = list(
-    list(soma_joinid = c(0, 1000)),
-    list(soma_joinid = c(0, 1000), int_column = c(-10000, 10000)),
-    list(soma_joinid = c(0, 1000), string_column = NULL),
+    list(soma_joinid = c(0, 999)),
+    list(soma_joinid = c(0, 999), int_column = c(-10000, 10000)),
+    list(soma_joinid = c(0, 999), string_column = NULL),
     list(string_column = NULL, int_column = c(-10000, 10000)),
     list(string_column = c("apple", "zebra"), int_column = c(-10000, 10000))
   )
@@ -186,8 +186,8 @@ test_that("SOMADataFrame shape", {
             expect_equal(int_mxd, int_dfc)
           }
         } else {
-            expect_true(int_mxd[[1]] < -2000000000)
-            expect_true(int_mxd[[2]] > 2000000000)
+          expect_true(int_mxd[[1]] < -2000000000)
+          expect_true(int_mxd[[2]] > 2000000000)
         }
       }
 
@@ -205,7 +205,7 @@ test_that("SOMADataFrame shape", {
             expect_equal(str_dom, c("", ""))
           } else {
             if (is.null(str_dfc)) {
-                expect_equal(str_dom, c("", ""))
+              expect_equal(str_dom, c("", ""))
             } else {
               expect_equal(str_dom, str_dfc)
             }
@@ -216,6 +216,69 @@ test_that("SOMADataFrame shape", {
 
       sdf$close()
 
+      # Test resize for dataframes (more general upgrade_domain to be tested
+      # separately -- see https://github.com/single-cell-data/TileDB-SOMA/issues/2407)
+      if (.new_shape_feature_flag_is_enabled() && use_domain_at_create) {
+        has_soma_joinid_dim <- "soma_joinid" %in% index_column_names
+        sjid_dfc <- domain_for_create[["soma_joinid"]]
+
+        # Test resize down
+        new_shape <- 0
+        sdf <- SOMADataFrameOpen(uri, "WRITE")
+        if (has_soma_joinid_dim) {
+          # It's an error to downsize
+          expect_error(sdf$resize_soma_joinid(new_shape))
+        } else {
+          # There is no problem when soma_joinid is not a dim --
+          # sdf$resize_soma_joinid is a no-op in that case
+          expect_no_condition(sdf$resize_soma_joinid(new_shape))
+        }
+        sdf$close()
+
+        # Make sure the failed resize really didn't change the shape
+        if (has_soma_joinid_dim) {
+          sdf <- SOMADataFrameOpen(uri, "READ")
+          expect_equal(sdf$domain()[["soma_joinid"]], sjid_dfc)
+          sdf$close()
+        }
+
+        # Test writes out of bounds, before resize
+        old_shape <- 100
+        if (has_soma_joinid_dim) {
+          old_shape <- domain_for_create[["soma_joinid"]][[2]] + 1 + 100
+        }
+        new_shape <- old_shape + 100
+
+        tbl1 <- arrow::arrow_table(
+          int_column = 5L:8L,
+          soma_joinid = (old_shape+1L):(old_shape+4L),
+          float_column = 5.1:8.1,
+          string_column = c("egg", "flag", "geese", "hay"),
+          schema = asch)
+
+        sdf <- SOMADataFrameOpen(uri, "WRITE")
+        if (has_soma_joinid_dim) {
+          expect_error(sdf$write(tbl1))
+        } else {
+          expect_no_condition(sdf$write(tbl1))
+        }
+        sdf$close()
+
+        # Test resize
+        sdf <- SOMADataFrameOpen(uri, "WRITE")
+        sdf$resize_soma_joinid(new_shape)
+        sdf$close();
+
+        # Test writes out of old bounds, within new bounds, after resize
+        sdf <- SOMADataFrameOpen(uri, "WRITE")
+        expect_no_condition(sdf$write(tbl1))
+        sdf$close();
+
+        # To do: test readback
+
+        rm(tbl1)
+      }
+
       rm(sdf, tbl0)
 
       gc()
diff --git a/libtiledbsoma/test/unit_soma_dataframe.cc b/libtiledbsoma/test/unit_soma_dataframe.cc
index 33089f6ea9..ae20d2eda3 100644
--- a/libtiledbsoma/test/unit_soma_dataframe.cc
+++ b/libtiledbsoma/test/unit_soma_dataframe.cc
@@ -478,7 +478,7 @@ TEST_CASE_METHOD(
     soma_dataframe->close();
 
     soma_dataframe = open(OpenMode::write);
-    soma_dataframe->maybe_resize_soma_joinid(std::vector<int64_t>({new_max}));
+    soma_dataframe->resize_soma_joinid(int64_t{new_max});
     soma_dataframe->close();
 
     soma_dataframe = open(OpenMode::write);
@@ -588,7 +588,7 @@ TEST_CASE_METHOD(
         REQUIRE(soma_dataframe->nnz() == 4);
 
         // Resize
-        auto new_shape = std::vector<int64_t>({SOMA_JOINID_RESIZE_DIM_MAX + 1});
+        auto new_shape = int64_t{SOMA_JOINID_RESIZE_DIM_MAX + 1};
 
         if (!use_current_domain) {
             // Domain is already set. The domain (not current domain but domain)
@@ -599,7 +599,7 @@ TEST_CASE_METHOD(
 
             soma_dataframe = open(OpenMode::write);
             // Array not resizeable if it has not already been sized
-            REQUIRE_THROWS(soma_dataframe->maybe_resize_soma_joinid(new_shape));
+            REQUIRE_THROWS(soma_dataframe->resize_soma_joinid(new_shape));
             soma_dataframe->close();
 
         } else {
@@ -617,11 +617,11 @@ TEST_CASE_METHOD(
             soma_dataframe->close();
 
             soma_dataframe = open(OpenMode::read);
-            REQUIRE_THROWS(soma_dataframe->maybe_resize_soma_joinid(new_shape));
+            REQUIRE_THROWS(soma_dataframe->resize_soma_joinid(new_shape));
             soma_dataframe->close();
 
             soma_dataframe = open(OpenMode::write);
-            soma_dataframe->maybe_resize_soma_joinid(new_shape);
+            soma_dataframe->resize_soma_joinid(new_shape);
             soma_dataframe->close();
 
             // Check shape after resize
@@ -784,7 +784,7 @@ TEST_CASE_METHOD(
         soma_dataframe->close();
 
         // Resize
-        auto new_shape = std::vector<int64_t>({SOMA_JOINID_RESIZE_DIM_MAX + 1});
+        auto new_shape = int64_t{SOMA_JOINID_RESIZE_DIM_MAX + 1};
 
         if (!use_current_domain) {
             // Domain is already set. The domain (not current domain but domain)
@@ -795,7 +795,7 @@ TEST_CASE_METHOD(
 
             soma_dataframe = open(OpenMode::write);
             // Array not resizeable if it has not already been sized
-            REQUIRE_THROWS(soma_dataframe->maybe_resize_soma_joinid(new_shape));
+            REQUIRE_THROWS(soma_dataframe->resize_soma_joinid(new_shape));
             soma_dataframe->close();
 
         } else {
@@ -812,11 +812,11 @@ TEST_CASE_METHOD(
             soma_dataframe->close();
 
             soma_dataframe = open(OpenMode::read);
-            REQUIRE_THROWS(soma_dataframe->maybe_resize_soma_joinid(new_shape));
+            REQUIRE_THROWS(soma_dataframe->resize_soma_joinid(new_shape));
             soma_dataframe->close();
 
             soma_dataframe = open(OpenMode::write);
-            soma_dataframe->maybe_resize_soma_joinid(new_shape);
+            soma_dataframe->resize_soma_joinid(new_shape);
             soma_dataframe->close();
 
             // Check shape after resize
@@ -1016,7 +1016,7 @@ TEST_CASE_METHOD(
         soma_dataframe->close();
 
         // Resize
-        auto new_shape = std::vector<int64_t>({SOMA_JOINID_RESIZE_DIM_MAX + 1});
+        auto new_shape = int64_t{SOMA_JOINID_RESIZE_DIM_MAX + 1};
 
         if (!use_current_domain) {
             // Domain is already set. The domain (not current domain but domain)
@@ -1027,7 +1027,7 @@ TEST_CASE_METHOD(
 
             soma_dataframe = open(OpenMode::write);
             // Array not resizeable if it has not already been sized
-            REQUIRE_THROWS(soma_dataframe->maybe_resize_soma_joinid(new_shape));
+            REQUIRE_THROWS(soma_dataframe->resize_soma_joinid(new_shape));
             soma_dataframe->close();
 
         } else {
@@ -1044,11 +1044,11 @@ TEST_CASE_METHOD(
             soma_dataframe->close();
 
             soma_dataframe = open(OpenMode::read);
-            REQUIRE_THROWS(soma_dataframe->maybe_resize_soma_joinid(new_shape));
+            REQUIRE_THROWS(soma_dataframe->resize_soma_joinid(new_shape));
             soma_dataframe->close();
 
             soma_dataframe = open(OpenMode::write);
-            soma_dataframe->maybe_resize_soma_joinid(new_shape);
+            soma_dataframe->resize_soma_joinid(new_shape);
             soma_dataframe->close();
 
             // Check shape after resize
@@ -1229,7 +1229,7 @@ TEST_CASE_METHOD(
         soma_dataframe->close();
 
         // Resize
-        auto new_shape = std::vector<int64_t>({SOMA_JOINID_RESIZE_DIM_MAX + 1});
+        auto new_shape = int64_t{SOMA_JOINID_RESIZE_DIM_MAX + 1};
 
         if (!use_current_domain) {
             // Domain is already set. The domain (not current domain but domain)
@@ -1240,7 +1240,7 @@ TEST_CASE_METHOD(
 
             soma_dataframe = open(OpenMode::write);
             // Array not resizeable if it has not already been sized
-            REQUIRE_THROWS(soma_dataframe->maybe_resize_soma_joinid(new_shape));
+            REQUIRE_THROWS(soma_dataframe->resize_soma_joinid(new_shape));
             soma_dataframe->close();
 
         } else {
@@ -1252,11 +1252,11 @@ TEST_CASE_METHOD(
             soma_dataframe->close();
 
             soma_dataframe = open(OpenMode::read);
-            REQUIRE_THROWS(soma_dataframe->maybe_resize_soma_joinid(new_shape));
+            REQUIRE_THROWS(soma_dataframe->resize_soma_joinid(new_shape));
             soma_dataframe->close();
 
             soma_dataframe = open(OpenMode::write);
-            soma_dataframe->maybe_resize_soma_joinid(new_shape);
+            soma_dataframe->resize_soma_joinid(new_shape);
             soma_dataframe->close();
 
             // Check shape after resize -- noting soma_joinid is not a dim here