Repository: single-cell-data/TileDB-SOMA (language: python)

Issue:
[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[python] Domain-at-create unit-test PR 2/5
**Issue and/or context:** As tracked on issue #2407 / [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

Note that the intended Python and R API changes are all agreed on and finalized as described in #2407.

**Changes:**

Split out from #3189 as described there.
[python] Domain-at-create unit-test PR 3/5
**Issue and/or context:** As tracked on issue #2407 / [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

Note that the intended Python and R API changes are all agreed on and finalized as described in #2407.

**Changes:**

Split out from #3189 as described there.
[python] Domain-at-create unit-test PR 5/5
**Issue and/or context:** As tracked on issue #2407 / [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

Note that the intended Python and R API changes are all agreed on and finalized as described in #2407.

**Changes:**

Split out from #3189 as described there.


Discussion / hints:
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.
## [Codecov](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3190?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 83.29%. Comparing base [(`7683e1e`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/7683e1e7d2d10962a438a73d24181ba631af9ed9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) to head [(`003a669`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/003a669e89ddc2f07387124b9697c4839ca9f4f9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data).
> Report is 5 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #3190      +/-   ##
==========================================
+ Coverage   83.18%   83.29%   +0.10%     
==========================================
  Files          51       51              
  Lines        5483     5483              
==========================================
+ Hits         4561     4567       +6     
+ Misses        922      916       -6     
```

| [Flag](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3190/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3190/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `83.29% <ø> (+0.10%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Components](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3190/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python_api](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3190/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `83.29% <ø> (+0.10%)` | :arrow_up: |
| [libtiledbsoma](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3190/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `∅ <ø> (∅)` | |

</details>
## [Codecov](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3192?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 83.29%. Comparing base [(`7683e1e`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/7683e1e7d2d10962a438a73d24181ba631af9ed9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) to head [(`b1c64a1`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/b1c64a1d5dba359d66e027ca1b2bcc5f59dff97e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data).
> Report is 6 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #3192      +/-   ##
==========================================
+ Coverage   83.18%   83.29%   +0.10%     
==========================================
  Files          51       51              
  Lines        5483     5483              
==========================================
+ Hits         4561     4567       +6     
+ Misses        922      916       -6     
```

| [Flag](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3192/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3192/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `83.29% <ø> (+0.10%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Components](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3192/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python_api](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `83.29% <ø> (+0.10%)` | :arrow_up: |
| [libtiledbsoma](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3192/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `∅ <ø> (∅)` | |

</details>
## [Codecov](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3194?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 83.29%. Comparing base [(`7683e1e`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/7683e1e7d2d10962a438a73d24181ba631af9ed9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) to head [(`2135c2c`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/2135c2cb590ccc26efa3741d463a34a14a54ce77?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data).
> Report is 8 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #3194      +/-   ##
==========================================
+ Coverage   83.18%   83.29%   +0.10%     
==========================================
  Files          51       51              
  Lines        5483     5483              
==========================================
+ Hits         4561     4567       +6     
+ Misses        922      916       -6     
```

| [Flag](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3194/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3194/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `83.29% <ø> (+0.10%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Components](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3194/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python_api](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3194/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `83.29% <ø> (+0.10%)` | :arrow_up: |
| [libtiledbsoma](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3194/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `∅ <ø> (∅)` | |

</details>

Fix patch:
diff --git a/apis/r/DESCRIPTION b/apis/r/DESCRIPTION
index c77fb50e12..48056b0e0e 100644
--- a/apis/r/DESCRIPTION
+++ b/apis/r/DESCRIPTION
@@ -6,7 +6,7 @@ Description: Interface for working with 'TileDB'-based Stack of Matrices,
     like those commonly used for single cell data analysis. It is documented at
     <https://github.com/single-cell-data>; a formal specification available is at
     <https://github.com/single-cell-data/SOMA/blob/main/abstract_specification.md>.
-Version: 1.15.99.8
+Version: 1.15.99.9
 Authors@R: c(
     person(given = "Aaron", family = "Wolen",
            role = c("cre", "aut"), email = "aaron@tiledb.com",
diff --git a/apis/r/NEWS.md b/apis/r/NEWS.md
index 1b3869a217..9b2ef68912 100644
--- a/apis/r/NEWS.md
+++ b/apis/r/NEWS.md
@@ -16,6 +16,7 @@
 * Support for dense current domain with core 2.27 [#3180](https://github.com/single-cell-data/TileDB-SOMA/pull/3180)
 * Fix `is_named_list` bug for half-named lists [#3183](https://github.com/single-cell-data/TileDB-SOMA/pull/3183)
 * Expose block/random writer for sparse arrays [#3204](https://github.com/single-cell-data/TileDB-SOMA/pull/3204)
+* Min-sizing for dataframes/arrays with new shape feature [#3208](https://github.com/single-cell-data/TileDB-SOMA/pull/3208)
 
 # tiledbsoma 1.14.1
 
diff --git a/apis/r/R/SOMACollectionBase.R b/apis/r/R/SOMACollectionBase.R
index 116020351e..8e9902d782 100644
--- a/apis/r/R/SOMACollectionBase.R
+++ b/apis/r/R/SOMACollectionBase.R
@@ -94,10 +94,11 @@ SOMACollectionBase <- R6::R6Class(
     #' @param key The key to be added.
     #' @param schema Arrow schema argument passed on to DataFrame$create()
     #' @param index_column_names Index column names passed on to DataFrame$create()
+    #' @param domain As in ``SOMADataFrameCreate``.
     #' @template param-platform-config
-    add_new_dataframe = function(key, schema, index_column_names, platform_config = NULL) {
+    add_new_dataframe = function(key, schema, index_column_names, domain, platform_config = NULL) {
       ## TODO: Check argument validity
-      ndf <- SOMADataFrame$new(
+      sdf <- SOMADataFrame$new(
         uri = file_path(self$uri, key),
         platform_config = platform_config %||% private$.tiledb_platform_config,
         tiledbsoma_ctx = private$.tiledbsoma_ctx,
@@ -105,9 +106,9 @@ SOMACollectionBase <- R6::R6Class(
         internal_use_only = "allowed_use"
       )
 
-      ndf$create(schema, index_column_names, internal_use_only = "allowed_use")
-      super$set(ndf, key)
-      ndf
+      sdf$create(schema, index_column_names=index_column_names, domain=domain, internal_use_only = "allowed_use")
+      super$set(sdf, key)
+      sdf
     },
 
     #' @description Add a new SOMA DenseNdArray to this collection. (lifecycle: maturing)
diff --git a/apis/r/R/utils-arrow.R b/apis/r/R/utils-arrow.R
index 891c532b43..9ea924a0d9 100644
--- a/apis/r/R/utils-arrow.R
+++ b/apis/r/R/utils-arrow.R
@@ -488,7 +488,26 @@ get_domain_and_extent_dataframe <- function(tbl_schema, ind_col_names, domain =
 
         requested_slot <- domain[[ind_col_name]]
         ind_cur_dom <- if (is.null(requested_slot)) {
-          ind_max_dom
+          if (.new_shape_feature_flag_is_enabled()) {
+            # New shape: if the slot is null, make the size as small
+            # as possible since current domain can only be resized upward.
+            #
+            # Core current-domain semantics are (lo, hi) with both
+            # inclusive, with lo <= hi. This means smallest is (0, 0)
+            # which is shape 1, not 0.
+            if (bit64::is.integer64(ind_max_dom)) {
+              c(bit64::as.integer64(0), bit64::as.integer64(0))
+            } else if (is.integer(ind_max_dom)) {
+              c(0L, 0L)
+            } else {
+              c(0, 0)
+            }
+          } else {
+            # Old shape: if the slot is null, make the size as large
+            # as possible since there is not current domain, and the
+            # max domain is immutable.
+            ind_max_dom
+          }
         } else {
           requested_slot
         }


Test patch:
diff --git a/apis/r/tests/testthat/helper-test-data.R b/apis/r/tests/testthat/helper-test-data.R
index a3e8c6d692..efdd572de9 100644
--- a/apis/r/tests/testthat/helper-test-data.R
+++ b/apis/r/tests/testthat/helper-test-data.R
@@ -78,3 +78,15 @@ create_arrow_table <- function(nrows = 10L, factors = FALSE) {
       # schema = create_arrow_schema(false)
     )
 }
+
+domain_for_arrow_table <- function() {
+  return(
+    list(
+      int_column = c(0, 1000000),
+      soma_joinid = c(0, 1000000),
+      float_column = c(-1e6, 1e6),
+      string_column = NULL,
+      grp = NULL
+    )
+  )
+}
diff --git a/apis/r/tests/testthat/helper-test-soma-objects.R b/apis/r/tests/testthat/helper-test-soma-objects.R
index a2927e5abd..6bdf849328 100644
--- a/apis/r/tests/testthat/helper-test-soma-objects.R
+++ b/apis/r/tests/testthat/helper-test-soma-objects.R
@@ -9,10 +9,22 @@ create_and_populate_soma_dataframe <- function(
 ) {
   set.seed(seed)
 
-  # arrow_schema <- create_arrow_schema()
   tbl <- create_arrow_table(nrows = nrows, factors = factors)
 
-  sdf <- SOMADataFrameCreate(uri, tbl$schema, index_column_names = index_column_names)
+  full_domain <- domain_for_arrow_table()
+  # Pick out the index-column names actually being used in this case
+  domain <- list()
+  for (index_column in index_column_names) {
+    domain[[index_column]] <- full_domain[[index_column]]
+  }
+
+  sdf <- SOMADataFrameCreate(
+    uri,
+    tbl$schema,
+    index_column_names = index_column_names,
+    domain = domain
+  )
+
   sdf$write(tbl)
 
   if (is.null(mode)) {
@@ -67,11 +79,14 @@ create_and_populate_var <- function(
       rep_len("lvl2", length.out = floor(nrows / 2))
     ))
   }
+  domain <- list(
+    soma_joinid = c(0, nrows - 1L)
+  )
 
   dname <- dirname(uri)
   if (!dir.exists(dname)) dir.create(dname)
 
-  sdf <- SOMADataFrameCreate(uri, tbl$schema, index_column_names = "soma_joinid")
+  sdf <- SOMADataFrameCreate(uri, tbl$schema, index_column_names = "soma_joinid", domain = domain)
   sdf$write(tbl)
 
   if (is.null(mode)) {
diff --git a/apis/r/tests/testthat/test-Factory.R b/apis/r/tests/testthat/test-Factory.R
index b06c0a23c7..1611fad3be 100644
--- a/apis/r/tests/testthat/test-Factory.R
+++ b/apis/r/tests/testthat/test-Factory.R
@@ -8,9 +8,19 @@ test_that("DataFrame Factory", {
 
     # Check creation of a DF
     asch <- create_arrow_schema(foo_first=FALSE)
-    expect_silent(d2 <- SOMADataFrameCreate(uri, schema = asch))
-    tbl <- arrow::arrow_table(soma_joinid = 1L:10L, int_column = 1L:10L, float_column = sqrt(1:10),
-                              string_column = letters[1:10], schema = asch)
+
+    expect_silent(d2 <- SOMADataFrameCreate(
+      uri,
+      schema = asch,
+      domain = list(soma_joinid = c(0, 99))
+    ))
+
+    tbl <- arrow::arrow_table(
+      soma_joinid = 1L:10L,
+      int_column = 1L:10L,
+      float_column = sqrt(1:10),
+      string_column = letters[1:10],
+      schema = asch)
     d2$write(tbl)
 
     # Check opening to read
@@ -26,9 +36,21 @@ test_that("DataFrame Factory with specified index_column_names", {
     # Check creation of a DF
     asch <- create_arrow_schema()
     expect_error(d2 <- SOMADataFrameCreate(uri, index_column_names = "int_column")) # misses schema
-    expect_silent(d2 <- SOMADataFrameCreate(uri, schema = asch, index_column_names = "int_column"))
-    tbl <- arrow::arrow_table(int_column = 1L:10L, soma_joinid = 1L:10L, float_column = sqrt(1:10),
-                              string_column = letters[1:10], schema = asch)
+
+    expect_silent(d2 <- SOMADataFrameCreate(
+      uri,
+      schema = asch,
+      index_column_names = "int_column",
+      domain = list(int_column = c(1, 10))
+    ))
+
+    tbl <- arrow::arrow_table(
+      int_column = 1L:10L,
+      soma_joinid = 1L:10L,
+      float_column = sqrt(1:10),
+      string_column = letters[1:10],
+      schema = asch)
+
     d2$write(tbl)
 
     # Check opening to read
diff --git a/apis/r/tests/testthat/test-OrderedAndFactor.R b/apis/r/tests/testthat/test-OrderedAndFactor.R
index c05f7d3d8f..508481829d 100644
--- a/apis/r/tests/testthat/test-OrderedAndFactor.R
+++ b/apis/r/tests/testthat/test-OrderedAndFactor.R
@@ -97,7 +97,7 @@ test_that("SOMADataFrame round-trip with factor and ordered", {
     expect_equal(names(lvls), colnames(et))
 
     #sdf <- SOMADataFrameCreate(uri, sch)
-    sdf <- SOMADataFrameCreate(uri, att$schema)
+    sdf <- SOMADataFrameCreate(uri, att$schema, domain = list(soma_joinid = c(0, 999)))
     expect_true(inherits(sdf, "SOMADataFrame"))
 
     sdf$write(att)
diff --git a/apis/r/tests/testthat/test-SOMACollection.R b/apis/r/tests/testthat/test-SOMACollection.R
index e902aa891d..26d52534a4 100644
--- a/apis/r/tests/testthat/test-SOMACollection.R
+++ b/apis/r/tests/testthat/test-SOMACollection.R
@@ -43,7 +43,7 @@ test_that("SOMACollection basics", {
   subcollection$close()
 
   # Add another dataframe to the collection, this time using add_new_dataframe
-  collection$add_new_dataframe("new_df", create_arrow_schema(), "int_column")$close()
+  collection$add_new_dataframe("new_df", create_arrow_schema(), "int_column", domain = list(int_column = c(0, 999)))$close()
   df3 <- collection$get("new_df")
   df3 <- SOMADataFrameOpen(df3$uri)
   expect_true(df3$soma_type == "SOMADataFrame")
@@ -131,7 +131,7 @@ test_that("Platform config and context are respected by add_ methods", {
 
   # Add a dataframe element to the collection
   tbl <- create_arrow_table()
-  sdf1 <- collection$add_new_dataframe("sdf1", tbl$schema, "soma_joinid")
+  sdf1 <- collection$add_new_dataframe("sdf1", tbl$schema, "soma_joinid", domain = list(soma_joinid = c(0, 999)))
   sdf1$write(tbl)
   collection$close()
 
@@ -154,6 +154,7 @@ test_that("Platform config and context are respected by add_ methods", {
     key = "sdf2",
     schema = tbl$schema,
     index_column_names = "soma_joinid",
+    domain = list(soma_joinid = c(0, 999)),
     platform_config = cfg
   )
   sdf2$write(tbl)
diff --git a/apis/r/tests/testthat/test-SOMADataFrame.R b/apis/r/tests/testthat/test-SOMADataFrame.R
index 64a8334acd..1ca1439a9b 100644
--- a/apis/r/tests/testthat/test-SOMADataFrame.R
+++ b/apis/r/tests/testthat/test-SOMADataFrame.R
@@ -9,7 +9,7 @@ test_that("Basic mechanics", {
   )
   if (dir.exists(uri)) unlink(uri, recursive=TRUE)
 
-  sdf <- SOMADataFrameCreate(uri, asch, index_column_names = "int_column")
+  sdf <- SOMADataFrameCreate(uri, asch, index_column_names = "int_column", domain = list(int_column = c(1, 36)))
   expect_true(sdf$exists())
   expect_true(dir.exists(uri))
 
@@ -127,7 +127,7 @@ test_that("Basic mechanics with default index_column_names", {
   )
   if (dir.exists(uri)) unlink(uri, recursive=TRUE)
 
-  sdf$create(asch, internal_use_only = "allowed_use")
+  sdf$create(asch, domain = list(soma_joinid = c(0, 99)), internal_use_only = "allowed_use")
   expect_true(sdf$exists())
   expect_true(dir.exists(uri))
   expect_match(sdf$soma_type, "SOMADataFrame")
@@ -176,6 +176,15 @@ test_that("creation with all supported dimension data types", {
     arrow::field("string", arrow::utf8(), nullable = FALSE)
   )
 
+  domains <- list(
+    int8 = c(1L, 36L),
+    int16 = c(1L, 36L),
+    double = c(1.1, 36.1),
+    int = c(1L, 36L),
+    int64 = c(bit64::as.integer64(1L), bit64::as.integer64(36L)),
+    string = NULL
+  )
+
   tbl0 <- arrow::arrow_table(
     int8 = 1L:36L,
     int16 = 1:36L,
@@ -192,7 +201,7 @@ test_that("creation with all supported dimension data types", {
   for (dtype in tbl0$ColumnNames()) {
     uri <- tempfile(pattern=paste0("soma-dataframe-", dtype))
     expect_silent(
-      sdf <- SOMADataFrameCreate(uri, tbl0$schema, index_column_names = dtype)
+      sdf <- SOMADataFrameCreate(uri, tbl0$schema, index_column_names = dtype, domain = domains[dtype])
     )
     expect_true(sdf$exists())
     sdf$close()
@@ -211,7 +220,7 @@ test_that("int64 values are stored correctly", {
   )
   if (dir.exists(uri)) unlink(uri, recursive=TRUE)
 
-  sdf <- SOMADataFrameCreate(uri, asch, index_column_names = "int_column")
+  sdf <- SOMADataFrameCreate(uri, asch, index_column_names = "int_column", domain = list(int_column = c(1, 10)))
   tbl0 <- arrow::arrow_table(int_column = 1L:10L, soma_joinid = 1L:10L, schema = asch)
 
   orig_downcast_value <- getOption("arrow.int64_downcast")
@@ -245,7 +254,9 @@ test_that("creation with ordered factors", {
   tbl <- arrow::as_arrow_table(df)
   expect_true(tbl$schema$GetFieldByName("ord")$type$ordered)
   if (dir.exists(uri)) unlink(uri, recursive=TRUE)
-  expect_no_condition(sdf <- SOMADataFrameCreate(uri = uri, schema = tbl$schema))
+  expect_no_condition(
+    sdf <- SOMADataFrameCreate(uri = uri, schema = tbl$schema, domain = list(soma_joinid = c(0, n-1L)))
+  )
   expect_no_condition(sdf$write(values = tbl))
   expect_s3_class(sdf <- SOMADataFrameOpen(uri), "SOMADataFrame")
   expect_true(sdf$schema()$GetFieldByName("ord")$type$ordered)
@@ -270,7 +281,9 @@ test_that("explicit casting of ordered factors to regular factors", {
   if (dir.exists(uri)) unlink(uri, recursive=TRUE)
   tbl <- arrow::as_arrow_table(df)
   expect_true(tbl$schema$GetFieldByName("ord")$type$ordered)
-  expect_no_condition(sdf <- SOMADataFrameCreate(uri = uri, schema = tbl$schema,))
+  expect_no_condition(
+    sdf <- SOMADataFrameCreate(uri = uri, schema = tbl$schema, domain = list(soma_joinid = c(0, n-1L)))
+  )
   expect_no_condition(sdf$write(values = tbl))
   expect_s3_class(sdf <- SOMADataFrameOpen(uri), "SOMADataFrame")
   expect_true(sdf$schema()$GetFieldByName("ord")$type$ordered)
@@ -577,7 +590,7 @@ test_that("SOMADataFrame timestamped ops", {
                        arrow::field("valint", arrow::int32(), nullable=FALSE),
                        arrow::field("valdbl", arrow::float64(), nullable=FALSE))
   if (dir.exists(uri)) unlink(uri, recursive=TRUE)
-  sdf <- SOMADataFrameCreate(uri=uri, schema=sch)
+  sdf <- SOMADataFrameCreate(uri=uri, schema=sch, domain = list(soma_joinid = c(1, 100)))
   rb1 <- arrow::record_batch(soma_joinid = bit64::as.integer64(1L:3L),
                              valint = 1L:3L,
                              valdbl = 100*(1:3),
@@ -813,7 +826,7 @@ test_that("missing levels in enums", {
   # Create SOMADataFrame w/ missing enum levels
   if (dir.exists(uri)) unlink(uri, recursive=TRUE)
   tbl <- arrow::as_arrow_table(df)
-  sdf <- SOMADataFrameCreate(uri, tbl$schema)
+  sdf <- SOMADataFrameCreate(uri, tbl$schema, domain = list(soma_joinid = c(0, n-1)))
   on.exit(sdf$close())
   sdf$write(tbl)
   sdf$close()
@@ -874,7 +887,7 @@ test_that("factor levels can grow without overlap", {
                             arrow::field(name = "obs_col_like",
                                          type = arrow::dictionary(index_type = arrow::int8(), ordered = FALSE)))
 
-    sdf <- SOMADataFrameCreate(uri, schema)
+    sdf <- SOMADataFrameCreate(uri, schema, domain = list(soma_joinid = c(0, 5)))
 
     tbl_1 <- arrow::arrow_table(soma_joinid = bit64::as.integer64(c(0,1,2)),
                                 obs_col_like = factor(c("A", "B", "A")),
@@ -917,7 +930,7 @@ test_that("factor levels cannot extend beyond index limit", {
         df <- data.frame(soma_joinid = bit64::as.integer64(seq_len(65)),
                          obs = factor(paste0("elem", seq_len(65))))
         tbl <- arrow::as_arrow_table(df, schema = sch)
-        expect_silent(SOMADataFrameCreate(uri, sch)$write(tbl)$close())
+        expect_silent(SOMADataFrameCreate(uri, sch, domain = list(soma_joinid = c(0, 999)))$write(tbl)$close())
 
         df2 <- data.frame(soma_joinid =  bit64::as.integer64(65 + seq_len(65)),
                           obs = factor(paste0("elem_", 65 + seq_len(65))))
diff --git a/apis/r/tests/testthat/test-Timestamps.R b/apis/r/tests/testthat/test-Timestamps.R
index edfc8aa5a9..27d7b496ed 100644
--- a/apis/r/tests/testthat/test-Timestamps.R
+++ b/apis/r/tests/testthat/test-Timestamps.R
@@ -8,7 +8,7 @@ test_that("SOMADataFrame", {
 
     ## create at t = 1
     ts1 <- as.POSIXct(1, tz = "UTC", origin = "1970-01-01")
-    sdf <- tiledbsoma::SOMADataFrameCreate(uri, sch, tiledb_timestamp = ts1)
+    sdf <- tiledbsoma::SOMADataFrameCreate(uri, sch, tiledb_timestamp = ts1, domain = list(soma_joinid=c(0, 999)))
 
     ## write part1 at t = 2
     dat2 <- arrow::arrow_table(soma_joinid = bit64::as.integer64(1L:5L),
diff --git a/apis/r/tests/testthat/test-query-condition.R b/apis/r/tests/testthat/test-query-condition.R
index 2ae9703735..7331fe02a5 100644
--- a/apis/r/tests/testthat/test-query-condition.R
+++ b/apis/r/tests/testthat/test-query-condition.R
@@ -34,7 +34,13 @@ test_that("DataFrame Factory", {
       # arrow::field("datetime_day", arrow::date32())
     )
 
-    sdf <- SOMADataFrameCreate(uri, sch, index_column_names = "soma_joinid")
+    sdf <- SOMADataFrameCreate(
+      uri,
+      sch,
+      index_column_names = "soma_joinid",
+      domain = list(soma_joinid = c(0, 999))
+    )
+
     expect_true(sdf$exists())
     expect_true(dir.exists(uri))
 
diff --git a/apis/r/tests/testthat/test-reopen.R b/apis/r/tests/testthat/test-reopen.R
index a879471818..6f162b18d7 100644
--- a/apis/r/tests/testthat/test-reopen.R
+++ b/apis/r/tests/testthat/test-reopen.R
@@ -171,7 +171,8 @@ test_that("`reopen()` works on nested collections", {
       soma_joinid = bit64::integer64(),
       int = integer()
     )),
-    index_column_names = "soma_joinid"
+    index_column_names = "soma_joinid",
+    domain = list(soma_joinid = c(0, 999))
   )
   expect_length(col$names(), 4L)
 
diff --git a/apis/r/tests/testthat/test-shape.R b/apis/r/tests/testthat/test-shape.R
index 19f99569d3..e4a11cc9f0 100644
--- a/apis/r/tests/testthat/test-shape.R
+++ b/apis/r/tests/testthat/test-shape.R
@@ -24,266 +24,237 @@ test_that("SOMADataFrame shape", {
   for (i in seq_along(index_column_name_choices)) {
     index_column_names <- index_column_name_choices[[i]]
 
-    for (use_domain_at_create in c(FALSE, TRUE)) {
+    uri <- withr::local_tempdir("soma-dataframe-shape")
 
-      uri <- withr::local_tempdir("soma-dataframe-shape")
+    # Create
+    if (dir.exists(uri)) unlink(uri, recursive=TRUE)
 
-      # Create
-      if (dir.exists(uri)) unlink(uri, recursive=TRUE)
+    domain_for_create <- domain_at_create_choices[[i]]
 
-      domain_for_create <- NULL
-      if (use_domain_at_create) {
-        domain_for_create <- domain_at_create_choices[[i]]
-      }
+    sdf <- SOMADataFrameCreate(
+      uri,
+      asch,
+      index_column_names = index_column_names,
+      domain = domain_for_create)
 
-      sdf <- SOMADataFrameCreate(
-        uri,
-        asch,
-        index_column_names = index_column_names,
-        domain = domain_for_create)
+    expect_true(sdf$exists())
+    expect_true(dir.exists(uri))
 
-      expect_true(sdf$exists())
-      expect_true(dir.exists(uri))
+    # Write
+    tbl0 <- arrow::arrow_table(int_column = 1L:4L,
+                               soma_joinid = 1L:4L,
+                               float_column = 1.1:4.1,
+                               string_column = c("apple", "ball", "cat", "dog"),
+                               schema = asch)
 
-      # Write
-      tbl0 <- arrow::arrow_table(int_column = 1L:4L,
-                                 soma_joinid = 1L:4L,
-                                 float_column = 1.1:4.1,
-                                 string_column = c("apple", "ball", "cat", "dog"),
-                                 schema = asch)
+    sdf$write(tbl0)
+    sdf$close()
 
-      sdf$write(tbl0)
-      sdf$close()
+    sdf <- SOMADataFrameOpen(uri)
 
-      sdf <- SOMADataFrameOpen(uri)
+    # Check shape and maxshape et al.
+    if (!.new_shape_feature_flag_is_enabled()) {
+      expect_false(sdf$tiledbsoma_has_upgraded_domain())
+    } else {
+      expect_true(sdf$tiledbsoma_has_upgraded_domain())
+    }
+    expect_error(sdf$shape(), class = "notYetImplementedError")
+    expect_error(sdf$maxshape(), class = "notYetImplementedError")
+
+    # Not implemented this way per
+    # https://github.com/single-cell-data/TileDB-SOMA/pull/2953#discussion_r1746125089
+    # sjid_shape <- sdf$.maybe_soma_joinid_shape()
+    # sjid_maxshape <- sdf$.maybe_soma_joinid_maxshape()
+    soma_context <- soma_context()
+    sjid_shape <- maybe_soma_joinid_shape(sdf$uri, soma_context)
+    sjid_maxshape <- maybe_soma_joinid_maxshape(sdf$uri, soma_context)
+
+    if ("soma_joinid" %in% index_column_names) {
+      # More testing to come on
+      # https://github.com/single-cell-data/TileDB-SOMA/issues/2407
+      expect_false(rlang::is_na(sjid_shape))
+      expect_false(rlang::is_na(sjid_maxshape))
+    } else {
+      expect_true(rlang::is_na(sjid_shape))
+      expect_true(rlang::is_na(sjid_maxshape))
+    }
 
-      # Check shape and maxshape et al.
-      if (!.new_shape_feature_flag_is_enabled()) {
-        expect_false(sdf$tiledbsoma_has_upgraded_domain())
-      } else {
-        expect_true(sdf$tiledbsoma_has_upgraded_domain())
-      }
-      expect_error(sdf$shape(), class = "notYetImplementedError")
-      expect_error(sdf$maxshape(), class = "notYetImplementedError")
-
-      # Not implemented this way per
-      # https://github.com/single-cell-data/TileDB-SOMA/pull/2953#discussion_r1746125089
-      # sjid_shape <- sdf$.maybe_soma_joinid_shape()
-      # sjid_maxshape <- sdf$.maybe_soma_joinid_maxshape()
-      soma_context <- soma_context()
-      sjid_shape <- maybe_soma_joinid_shape(sdf$uri, soma_context)
-      sjid_maxshape <- maybe_soma_joinid_maxshape(sdf$uri, soma_context)
-
-      if ("soma_joinid" %in% index_column_names) {
-        # More testing to come on
-        # https://github.com/single-cell-data/TileDB-SOMA/issues/2407
-        expect_false(rlang::is_na(sjid_shape))
-        expect_false(rlang::is_na(sjid_maxshape))
-      } else {
-        expect_true(rlang::is_na(sjid_shape))
-        expect_true(rlang::is_na(sjid_maxshape))
-      }
+    # Check has_upgraded_domain
+    if (!.new_shape_feature_flag_is_enabled()) {
+      expect_false(sdf$tiledbsoma_has_upgraded_domain())
+    } else {
+      expect_true(sdf$tiledbsoma_has_upgraded_domain())
+    }
+
+    # Check domain and maxdomain
+    dom <- sdf$domain()
+    mxd <- sdf$maxdomain()
+
+    # First check names
+    expect_equal(names(dom), index_column_names)
+    expect_equal(names(mxd), index_column_names)
+
+    # Then check all slots are pairs
+    for (name in names(dom)) {
+      expect_length(dom[[name]], 2L)
+      expect_length(mxd[[name]], 2L)
+    }
+
+    # Then check contents
+
+    # Old shape/domainishes (without core current domain) for non-string dims:
+    # * There is no core current domain
+    # * Expect domain == maxdomain
+    # * If they asked for NULL: both should be huge (near min/max for datatype)
+    # * If they asked for something specific: they should get it
+    #
+    # New shape/domainishes (with core current domain) for non-string dims:
+    # * Maxdomain should be huge (near min/max for datatype)
+    # * If they asked for NULL: domain should be the same as maxdomain
+    # * If they asked for a specific domain: they should get it
+    #
+    # Old shape/domainishes (without core current domain) for string dims:
+    # * There is no core current domain
+    # * Expect domain == maxdomain
+    # * Core domain for strings is always ("", "")
+    #
+    # New shape/domainishes (with core current domain) for string dims:
+    # * Core domain (soma maxdomain) for strings is always ("", "")
+    # * Core current domain (soma domain) for strings:
+    #   o If they asked for NULL: expect ("", "")
+    #   o If they asked for something specific: they should get it
+
+    if ("soma_joinid" %in% index_column_names) {
+      sjid_dom <- dom[["soma_joinid"]]
+      sjid_mxd <- mxd[["soma_joinid"]]
+      sjid_dfc <- domain_for_create[["soma_joinid"]]
 
-      # Check has_upgraded_domain
       if (!.new_shape_feature_flag_is_enabled()) {
-        expect_false(sdf$tiledbsoma_has_upgraded_domain())
-      } else {
-        expect_true(sdf$tiledbsoma_has_upgraded_domain())
+        # Old behavior
+        expect_equal(sjid_dom, sjid_mxd)
       }
 
-      # Check domain and maxdomain
-      dom <- sdf$domain()
-      mxd <- sdf$maxdomain()
+      # Not: expect_equal(sjid_dom, bit64::as.integer64(sjid_dfc)) The
+      # soma_joinid dim is always of type int64.  Everything coming back
+      # from libtiledbsoma, through C nanoarrow, through the R arrow
+      # package, to Arrow RecordBatch, holds true to that. But the final
+      # as.list() converts the domain to regular integer. This is a feature
+      # TBH: suppressable with `op <- options(arrow.int64_downcast =
+      # FALSE)`. The maxdomainis likely to be in the 2**63 range
+      # but the domain is likely to be ordinary-sized numbers in the
+      # thousands or millions. Users are likely to prefer these
+      # being downcast to regular R integers.
+      expect_equal(sjid_dom, sjid_dfc)
+    }
 
-      # First check names
-      expect_equal(names(dom), index_column_names)
-      expect_equal(names(mxd), index_column_names)
+    if ("int_column" %in% index_column_names) {
+      int_dom <- dom[["int_column"]]
+      int_mxd <- mxd[["int_column"]]
+      int_dfc <- domain_for_create[["int_column"]]
 
-      # Then check all slots are pairs
-      for (name in names(dom)) {
-        expect_length(dom[[name]], 2L)
-        expect_length(mxd[[name]], 2L)
+      if (!.new_shape_feature_flag_is_enabled()) {
+        # Old behavior
+        expect_equal(int_dom, int_mxd)
       }
 
-      # Then check contents
-
-      # Old shape/domainishes (without core current domain) for non-string dims:
-      # * There is no core current domain
-      # * Expect domain == maxdomain
-      # * If they asked for NULL: both should be huge (near min/max for datatype)
-      # * If they asked for something specific: they should get it
-      #
-      # New shape/domainishes (with core current domain) for non-string dims:
-      # * Maxdomain should be huge (near min/max for datatype)
-      # * If they asked for NULL: domain should be the same as maxdomain
-      # * If they asked for a specific domain: they should get it
-      #
-      # Old shape/domainishes (without core current domain) for string dims:
-      # * There is no core current domain
-      # * Expect domain == maxdomain
-      # * Core domain for strings is always ("", "")
-      #
-      # New shape/domainishes (with core current domain) for string dims:
-      # * Core domain (soma maxdomain) for strings is always ("", "")
-      # * Core current domain (soma domain) for strings:
-      #   o If they asked for NULL: expect ("", "")
-      #   o If they asked for something specific: they should get it
-
-      if ("soma_joinid" %in% index_column_names) {
-        sjid_dom <- dom[["soma_joinid"]]
-        sjid_mxd <- mxd[["soma_joinid"]]
-        sjid_dfc <- domain_for_create[["soma_joinid"]]
-
-        if (!.new_shape_feature_flag_is_enabled()) {
-          # Old behavior
-          expect_equal(sjid_dom, sjid_mxd)
-        }
+      expect_equal(int_dom, int_dfc)
 
-        if (!use_domain_at_create) {
-          expect_equal(sjid_dom[[1]], 0)
-          expect_equal(sjid_mxd[[1]], 0)
-          # This is a really big number in the ballpark of 2**63; its exact
-          # value is unimportant.
-          expect_true(sjid_dom[[2]] > bit64::as.integer64(10000000000))
-          expect_true(sjid_mxd[[2]] > bit64::as.integer64(10000000000))
-        } else {
-          # Not: expect_equal(sjid_dom, bit64::as.integer64(sjid_dfc)) The
-          # soma_joinid dim is always of type int64.  Everything coming back
-          # from libtiledbsoma, through C nanoarrow, through the R arrow
-          # package, to Arrow RecordBatch, holds true to that. But the final
-          # as.list() converts the domain to regular integer. This is a feature
-          # TBH: suppressable with `op <- options(arrow.int64_downcast =
-          # FALSE)`. The maxdomainis likely to be in the 2**63 range
-          # but the domain is likely to be ordinary-sized numbers in the
-          # thousands or millions. Users are likely to prefer these
-          # being downcast to regular R integers.
-          expect_equal(sjid_dom, sjid_dfc)
-        }
+      if (!.new_shape_feature_flag_is_enabled()) {
+        expect_equal(int_mxd, int_dfc)
+      } else {
+        expect_true(int_mxd[[1]] < -2000000000)
+        expect_true(int_mxd[[2]] > 2000000000)
       }
+    }
 
-      if ("int_column" %in% index_column_names) {
-        int_dom <- dom[["int_column"]]
-        int_mxd <- mxd[["int_column"]]
-        int_dfc <- domain_for_create[["int_column"]]
-
-        if (!.new_shape_feature_flag_is_enabled()) {
-          # Old behavior
-          expect_equal(int_dom, int_mxd)
-        }
+    if ("string_column" %in% index_column_names) {
+      str_dom <- dom[["string_column"]]
+      str_mxd <- mxd[["string_column"]]
+      str_dfc <- domain_for_create[["string_column"]]
 
-        if (!use_domain_at_create) {
-          expect_true(int_dom[[1]] < -2000000000)
-          expect_true(int_dom[[2]] > 2000000000)
-        } else {
-          expect_equal(int_dom, int_dfc)
-        }
+      if (!.new_shape_feature_flag_is_enabled()) {
+        expect_equal(str_dom, c("", ""))
+        expect_equal(str_mxd, c("", ""))
 
-        if (!.new_shape_feature_flag_is_enabled()) {
-          if (!use_domain_at_create) {
-            expect_true(int_mxd[[1]] < -2000000000)
-            expect_true(int_mxd[[2]] > 2000000000)
-          } else {
-            expect_equal(int_mxd, int_dfc)
-          }
+      } else {
+        if (is.null(str_dfc)) {
+          expect_equal(str_dom, c("", ""))
         } else {
-          expect_true(int_mxd[[1]] < -2000000000)
-          expect_true(int_mxd[[2]] > 2000000000)
+          expect_equal(str_dom, str_dfc)
         }
+        expect_equal(str_mxd, c("", ""))
       }
+    }
 
-      if ("string_column" %in% index_column_names) {
-        str_dom <- dom[["string_column"]]
-        str_mxd <- mxd[["string_column"]]
-        str_dfc <- domain_for_create[["string_column"]]
+    sdf$close()
 
-        if (!.new_shape_feature_flag_is_enabled()) {
-          expect_equal(str_dom, c("", ""))
-          expect_equal(str_mxd, c("", ""))
+    # Test resize for dataframes (more general upgrade_domain to be tested
+    # separately -- see https://github.com/single-cell-data/TileDB-SOMA/issues/2407)
+    if (.new_shape_feature_flag_is_enabled()) {
+      has_soma_joinid_dim <- "soma_joinid" %in% index_column_names
+      sjid_dfc <- domain_for_create[["soma_joinid"]]
 
-        } else {
-          if (!use_domain_at_create) {
-            expect_equal(str_dom, c("", ""))
-          } else {
-            if (is.null(str_dfc)) {
-              expect_equal(str_dom, c("", ""))
-            } else {
-              expect_equal(str_dom, str_dfc)
-            }
-          }
-          expect_equal(str_mxd, c("", ""))
-        }
+      # Test resize down
+      new_shape <- 0
+      sdf <- SOMADataFrameOpen(uri, "WRITE")
+      if (has_soma_joinid_dim) {
+        # It's an error to downsize
+        expect_error(sdf$resize_soma_joinid_shape(new_shape))
+      } else {
+        # There is no problem when soma_joinid is not a dim --
+        # sdf$resize_soma_joinid_shape is a no-op in that case
+        expect_no_condition(sdf$resize_soma_joinid_shape(new_shape))
       }
-
       sdf$close()
 
-      # Test resize for dataframes (more general upgrade_domain to be tested
-      # separately -- see https://github.com/single-cell-data/TileDB-SOMA/issues/2407)
-      if (.new_shape_feature_flag_is_enabled() && use_domain_at_create) {
-        has_soma_joinid_dim <- "soma_joinid" %in% index_column_names
-        sjid_dfc <- domain_for_create[["soma_joinid"]]
-
-        # Test resize down
-        new_shape <- 0
-        sdf <- SOMADataFrameOpen(uri, "WRITE")
-        if (has_soma_joinid_dim) {
-          # It's an error to downsize
-          expect_error(sdf$resize_soma_joinid_shape(new_shape))
-        } else {
-          # There is no problem when soma_joinid is not a dim --
-          # sdf$resize_soma_joinid_shape is a no-op in that case
-          expect_no_condition(sdf$resize_soma_joinid_shape(new_shape))
-        }
+      # Make sure the failed resize really didn't change the shape
+      if (has_soma_joinid_dim) {
+        sdf <- SOMADataFrameOpen(uri, "READ")
+        expect_equal(sdf$domain()[["soma_joinid"]], sjid_dfc)
         sdf$close()
+      }
 
-        # Make sure the failed resize really didn't change the shape
-        if (has_soma_joinid_dim) {
-          sdf <- SOMADataFrameOpen(uri, "READ")
-          expect_equal(sdf$domain()[["soma_joinid"]], sjid_dfc)
-          sdf$close()
-        }
-
-        # Test writes out of bounds, before resize
-        old_shape <- 100
-        if (has_soma_joinid_dim) {
-          old_shape <- domain_for_create[["soma_joinid"]][[2]] + 1 + 100
-        }
-        new_shape <- old_shape + 100
-
-        tbl1 <- arrow::arrow_table(
-          int_column = 5L:8L,
-          soma_joinid = (old_shape+1L):(old_shape+4L),
-          float_column = 5.1:8.1,
-          string_column = c("egg", "flag", "geese", "hay"),
-          schema = asch)
-
-        sdf <- SOMADataFrameOpen(uri, "WRITE")
-        if (has_soma_joinid_dim) {
-          expect_error(sdf$write(tbl1))
-        } else {
-          expect_no_condition(sdf$write(tbl1))
-        }
-        sdf$close()
-
-        # Test resize
-        sdf <- SOMADataFrameOpen(uri, "WRITE")
-        sdf$resize_soma_joinid_shape(new_shape)
-        sdf$close();
-
-        # Test writes out of old bounds, within new bounds, after resize
-        sdf <- SOMADataFrameOpen(uri, "WRITE")
+      # Test writes out of bounds, before resize
+      old_shape <- 100
+      if (has_soma_joinid_dim) {
+        old_shape <- domain_for_create[["soma_joinid"]][[2]] + 1 + 100
+      }
+      new_shape <- old_shape + 100
+
+      tbl1 <- arrow::arrow_table(
+        int_column = 5L:8L,
+        soma_joinid = (old_shape+1L):(old_shape+4L),
+        float_column = 5.1:8.1,
+        string_column = c("egg", "flag", "geese", "hay"),
+        schema = asch)
+
+      sdf <- SOMADataFrameOpen(uri, "WRITE")
+      if (has_soma_joinid_dim) {
+        expect_error(sdf$write(tbl1))
+      } else {
         expect_no_condition(sdf$write(tbl1))
-        sdf$close();
+      }
+      sdf$close()
 
-        # To do: test readback
+      # Test resize
+      sdf <- SOMADataFrameOpen(uri, "WRITE")
+      sdf$resize_soma_joinid_shape(new_shape)
+      sdf$close();
 
-        rm(tbl1)
-      }
+      # Test writes out of old bounds, within new bounds, after resize
+      sdf <- SOMADataFrameOpen(uri, "WRITE")
+      expect_no_condition(sdf$write(tbl1))
+      sdf$close();
 
-      rm(sdf, tbl0)
+      # To do: test readback
 
-      gc()
+      rm(tbl1)
     }
-  }
+
+    rm(sdf, tbl0)
+
+    gc()
+}
   
   # Test `domain` assertions
   uri <- tempfile()