Repository: single-cell-data/TileDB-SOMA (language: python)

Issue:
[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[packaging] Core 2.27 release items [tracking]
This is just a central spot for tracking things related to the 2.27 release, including its preliminary release-candidate series.
[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now

[c++/python] Fixes for dense arrays and core 2.27/`dev`
**Issue and/or context:** As tracked on issue #2407 / [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048). See also #3364.

**Changes:**

Some findings from https://github.com/TileDB-Inc/centralized-tiledb-nightlies/issues/25. There is more to do though.

**Notes for Reviewer:**

## Python read-OOM repro
```
#!/usr/bin/env python

import tiledbsoma
import pyarrow as pa
import sys
import os, shutil

# Any dense 2D array with small current domain but huge (~ 2**63) core domain
uri = 'dense-dnda-227'

print("RD-DNDA URI", uri)

dnda = tiledbsoma.open(uri)
# This next line will slowly OOM -- watch htop
o = dnda.read()
```

Fixed as of this commit: https://github.com/single-cell-data/TileDB-SOMA/pull/3244/commits/0e6493a05ea64c58935c6216d579a671b3d95059

## R write-fail repro

```
$ cat wr-lldb.r
#!/usr/bin/env Rscript

library(tiledbsoma)
library(arrow)

uri <- "data-dnda-r"
element_type = arrow::float32()
shape = c(10, 20)

cat("CR-DNDA URI     ", uri, "\n")
cat("CR-DNDA SHAPE   ", shape, "\n")

if (dir.exists(uri)) unlink(uri, recursive=TRUE)
dnda <- tiledbsoma::SOMADenseNDArrayCreate(uri, type=element_type, shape=shape)

cat("WR-DNDA URI", uri, "\n")
dnda$write(matrix(1:12, nrow=3,ncol=4))

dnda$close()
```

```
$ ./wr-lldb.r
CR-DNDA URI      data-dnda-r
CR-DNDA SHAPE    10 20
WR-DNDA URI data-dnda-r
Error: WriterBase: Buffer sizes check failed; Invalid number of cells given for attribute 'soma_data' (12 != 18446744073709551615)
Execution halted
```

Fixed as of this commit: https://github.com/single-cell-data/TileDB-SOMA/pull/3244/commits/0e6493a05ea64c58935c6216d579a671b3d95059

[c++/python] Fixes for dense arrays and core 2.27/`dev`
**Issue and/or context:** As tracked on issue #2407 / [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048). See also #3364.

**Changes:**

Some findings from https://github.com/TileDB-Inc/centralized-tiledb-nightlies/issues/25. There is more to do though.

**Notes for Reviewer:**

## Python read-OOM repro
```
#!/usr/bin/env python

import tiledbsoma
import pyarrow as pa
import sys
import os, shutil

# Any dense 2D array with small current domain but huge (~ 2**63) core domain
uri = 'dense-dnda-227'

print("RD-DNDA URI", uri)

dnda = tiledbsoma.open(uri)
# This next line will slowly OOM -- watch htop
o = dnda.read()
```

Fixed as of this commit: https://github.com/single-cell-data/TileDB-SOMA/pull/3244/commits/0e6493a05ea64c58935c6216d579a671b3d95059

## R write-fail repro

```
$ cat wr-lldb.r
#!/usr/bin/env Rscript

library(tiledbsoma)
library(arrow)

uri <- "data-dnda-r"
element_type = arrow::float32()
shape = c(10, 20)

cat("CR-DNDA URI     ", uri, "\n")
cat("CR-DNDA SHAPE   ", shape, "\n")

if (dir.exists(uri)) unlink(uri, recursive=TRUE)
dnda <- tiledbsoma::SOMADenseNDArrayCreate(uri, type=element_type, shape=shape)

cat("WR-DNDA URI", uri, "\n")
dnda$write(matrix(1:12, nrow=3,ncol=4))

dnda$close()
```

```
$ ./wr-lldb.r
CR-DNDA URI      data-dnda-r
CR-DNDA SHAPE    10 20
WR-DNDA URI data-dnda-r
Error: WriterBase: Buffer sizes check failed; Invalid number of cells given for attribute 'soma_data' (12 != 18446744073709551615)
Execution halted
```

Fixed as of this commit: https://github.com/single-cell-data/TileDB-SOMA/pull/3244/commits/0e6493a05ea64c58935c6216d579a671b3d95059



Discussion / hints:
#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.

#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.
Investigation so far shows that Python dense-array write calls C++ `set_array_data` as well as `set_dim_points`, while R dense-array write calls only the former and not the latter.

cc @mojaveazure and your observation the other day that the dense-array writer doesn't handle coords. Furthermore, as presently coded, it can _only_ write the full core domain of the array. And now that with the new-shape feature flag enabled, the core domain (soma maxdomain) is huge and the core current domain (soma domain) is small.

Here is a repro that fails even with the new-shape feature flag disabled, proving that this bug is older than the new-shape mod:

```
#!/usr/bin/env Rscript

library(tiledbsoma)
library(arrow)

uri <- "data-dnda-r"
element_type = arrow::float32()
shape = c(10, 20)

cat("CR-DNDA URI     ", uri, "\n")
cat("CR-DNDA SHAPE   ", shape, "\n")

if (dir.exists(uri)) unlink(uri, recursive=TRUE)

dnda <- tiledbsoma::SOMADenseNDArrayCreate(uri, type=element_type, shape=shape)

cat("WR-DNDA URI", uri, "\n")

dnda$write(matrix(1:12, nrow=3,ncol=4))

dnda$close()
```

```
CR-DNDA URI      data-dnda-r 
CR-DNDA SHAPE    10 20 
WR-DNDA URI data-dnda-r 
Error: WriterBase: Buffer sizes check failed; Invalid number of cells given for attribute 'soma_data' (12 != 200)
Execution halted
```

Also fails with

```
...
m <- matrix(1:12, nrow=3,ncol=4)
coords <- list( 4:6, 7:10)
dnda$write(m, coords)
...
```

Same error message.
The typeguard-related failures are new and weird. I created #3245 to check.
CI run from commit 1 failed: https://github.com/single-cell-data/TileDB-SOMA/actions/runs/11542968229/job/32126790602?pr=3245
## [Codecov](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3244?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 83.92%. Comparing base [(`50c5d71`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/50c5d716ddca6d0388a6b90767b000e3e406b414?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) to head [(`0e6493a`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/0e6493a05ea64c58935c6216d579a671b3d95059?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data).
> Report is 2 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #3244      +/-   ##
==========================================
+ Coverage   83.79%   83.92%   +0.12%     
==========================================
  Files          51       51              
  Lines        5560     5566       +6     
==========================================
+ Hits         4659     4671      +12     
+ Misses        901      895       -6     
```

| [Flag](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3244/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3244/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `83.92% <100.00%> (+0.12%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Components](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3244/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python_api](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3244/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `83.92% <100.00%> (+0.12%)` | :arrow_up: |
| [libtiledbsoma](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3244/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `∅ <ø> (∅)` | |

</details>
@nguyenv this is ready for review, and has green CI for this repo (i.e. with core 2.26).

There is still an issue or two remaining for core `dev`, which I have installed on one laptop to mimic @jdblischak 's nightlies. So, there will be at least one more PR. Meanwhile everything here on this PR is already a strict improvment.
Moving back to draft, likely to abandon later on.

On discussion with @nguyenv : 

* The main context of this PR is:
  * For dense arrays only
  * If any dim's subarray range isn't specified, core uses the core domain
  * With new-shape feature-flag enabled, and with core 2.27 where dense arrays _can_ have current domain (they couldn't in 2.26), we are indeed using data-sized core current domain and huge core domain
  * Put those together and we get these problems
  * Note there are a few other issues on this PR, and a few _not_ yet included in this PR, other than that -- and those should be split out into separate PRs
* On this PR I've ben finding callsites, for write and for read, for Python and R, and touching up the ranges which end up being core subarrays on each dim slot
  * This is tedious and error-prone
* What we should do instead is:
  * In one place -- in `libtiledbsoma`, in `ManagedQuery`, before `submit_query`
  * If any dim's range has not been set up to this point, and if the array has a current domain, simply set the ranges on that dim to be the current domain's `(lo, hi)`
  * We know (for a fact) that core will go ahead and use the too-big domain anyway so this is safe
Investigation so far shows that Python dense-array write calls C++ `set_array_data` as well as `set_dim_points`, while R dense-array write calls only the former and not the latter.

cc @mojaveazure and your observation the other day that the dense-array writer doesn't handle coords. Furthermore, as presently coded, it can _only_ write the full core domain of the array. And now that with the new-shape feature flag enabled, the core domain (soma maxdomain) is huge and the core current domain (soma domain) is small.

Here is a repro that fails even with the new-shape feature flag disabled, proving that this bug is older than the new-shape mod:

```
#!/usr/bin/env Rscript

library(tiledbsoma)
library(arrow)

uri <- "data-dnda-r"
element_type = arrow::float32()
shape = c(10, 20)

cat("CR-DNDA URI     ", uri, "\n")
cat("CR-DNDA SHAPE   ", shape, "\n")

if (dir.exists(uri)) unlink(uri, recursive=TRUE)

dnda <- tiledbsoma::SOMADenseNDArrayCreate(uri, type=element_type, shape=shape)

cat("WR-DNDA URI", uri, "\n")

dnda$write(matrix(1:12, nrow=3,ncol=4))

dnda$close()
```

```
CR-DNDA URI      data-dnda-r 
CR-DNDA SHAPE    10 20 
WR-DNDA URI data-dnda-r 
Error: WriterBase: Buffer sizes check failed; Invalid number of cells given for attribute 'soma_data' (12 != 200)
Execution halted
```

Also fails with

```
...
m <- matrix(1:12, nrow=3,ncol=4)
coords <- list( 4:6, 7:10)
dnda$write(m, coords)
...
```

Same error message.
The typeguard-related failures are new and weird. I created #3245 to check.
CI run from commit 1 failed: https://github.com/single-cell-data/TileDB-SOMA/actions/runs/11542968229/job/32126790602?pr=3245
## [Codecov](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3244?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 83.92%. Comparing base [(`50c5d71`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/50c5d716ddca6d0388a6b90767b000e3e406b414?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) to head [(`0e6493a`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/0e6493a05ea64c58935c6216d579a671b3d95059?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data).
> Report is 2 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #3244      +/-   ##
==========================================
+ Coverage   83.79%   83.92%   +0.12%     
==========================================
  Files          51       51              
  Lines        5560     5566       +6     
==========================================
+ Hits         4659     4671      +12     
+ Misses        901      895       -6     
```

| [Flag](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3244/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3244/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `83.92% <100.00%> (+0.12%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Components](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3244/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python_api](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3244/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `83.92% <100.00%> (+0.12%)` | :arrow_up: |
| [libtiledbsoma](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/3244/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `∅ <ø> (∅)` | |

</details>
@nguyenv this is ready for review, and has green CI for this repo (i.e. with core 2.26).

There is still an issue or two remaining for core `dev`, which I have installed on one laptop to mimic @jdblischak 's nightlies. So, there will be at least one more PR. Meanwhile everything here on this PR is already a strict improvment.
Moving back to draft, likely to abandon later on.

On discussion with @nguyenv : 

* The main context of this PR is:
  * For dense arrays only
  * If any dim's subarray range isn't specified, core uses the core domain
  * With new-shape feature-flag enabled, and with core 2.27 where dense arrays _can_ have current domain (they couldn't in 2.26), we are indeed using data-sized core current domain and huge core domain
  * Put those together and we get these problems
  * Note there are a few other issues on this PR, and a few _not_ yet included in this PR, other than that -- and those should be split out into separate PRs
* On this PR I've ben finding callsites, for write and for read, for Python and R, and touching up the ranges which end up being core subarrays on each dim slot
  * This is tedious and error-prone
* What we should do instead is:
  * In one place -- in `libtiledbsoma`, in `ManagedQuery`, before `submit_query`
  * If any dim's range has not been set up to this point, and if the array has a current domain, simply set the ranges on that dim to be the current domain's `(lo, hi)`
  * We know (for a fact) that core will go ahead and use the too-big domain anyway so this is safe

Fix patch:
diff --git a/apis/python/src/tiledbsoma/_flags.py b/apis/python/src/tiledbsoma/_flags.py
index e59dd0e4fd..53a4bd7588 100644
--- a/apis/python/src/tiledbsoma/_flags.py
+++ b/apis/python/src/tiledbsoma/_flags.py
@@ -15,3 +15,9 @@
 NEW_SHAPE_FEATURE_FLAG_ENABLED = os.getenv("SOMA_PY_NEW_SHAPE") != "false"
 
 DENSE_ARRAYS_CAN_HAVE_CURRENT_DOMAIN = clib.embedded_version_triple() >= (2, 27, 0)
+
+# Temporary for # https://github.com/single-cell-data/TileDB-SOMA/issues/2407:
+# this allows testing dense + current domain on the same machine without
+# having to switch core builds (or switch machines).
+if os.getenv("SOMA_IGNORE_CORE_2_27") is not None:
+    DENSE_ARRAYS_CAN_HAVE_CURRENT_DOMAIN = False
diff --git a/apis/r/R/Init.R b/apis/r/R/Init.R
index 8735e8fa20..e26738f54a 100644
--- a/apis/r/R/Init.R
+++ b/apis/r/R/Init.R
@@ -36,7 +36,15 @@
   .pkgenv[["use_current_domain_transitional_internal_only"]]
 }
 
+# Temporary for # https://github.com/single-cell-data/TileDB-SOMA/issues/2407.
+# Once core 2.27 is released and we depend on it, this can go away.
 .dense_arrays_can_have_current_domain <- function() {
+  # This allows testing dense + current domain on the same machine without
+  # having to switch core builds (or switch machines).
+  if (Sys.getenv("SOMA_IGNORE_CORE_2_27") != "") {
+    return(FALSE)
+  }
+
   triple <- tiledb_embedded_version()
   return(triple[[1]] >= 2 && triple[[2]] >= 27)
 }
diff --git a/libtiledbsoma/src/soma/managed_query.cc b/libtiledbsoma/src/soma/managed_query.cc
index f3b6991eb9..bf036f68a9 100644
--- a/libtiledbsoma/src/soma/managed_query.cc
+++ b/libtiledbsoma/src/soma/managed_query.cc
@@ -62,7 +62,7 @@ void ManagedQuery::reset() {
     query_ = std::make_unique<Query>(*ctx_, *array_);
     subarray_ = std::make_unique<Subarray>(*ctx_, *array_);
 
-    subarray_range_set_ = false;
+    subarray_range_set_ = {};
     subarray_range_empty_ = {};
     columns_.clear();
     results_complete_ = true;
@@ -102,36 +102,10 @@ void ManagedQuery::setup_read() {
 
     auto schema = array_->schema();
 
+    _fill_in_subarrays_if_dense(true);
+
     // If the query is uninitialized, set the subarray for the query
     if (status == Query::Status::UNINITIALIZED) {
-        // Dense array must have a subarray set. If the array is dense and no
-        // ranges have been set, add a range for the array's entire non-empty
-        // domain on dimension 0. In the case that the non-empty domain does not
-        // exist (when the array has not been written to yet), use dimension 0's
-        // full domain
-        if (schema.array_type() == TILEDB_DENSE && !subarray_range_set_) {
-            // Check if the array has been written to by using the C API as
-            // there is no way to to check for an empty domain using the current
-            // CPP API
-            int32_t is_empty;
-            int64_t ned[2];
-            ctx_->handle_error(tiledb_array_get_non_empty_domain_from_index(
-                ctx_->ptr().get(), array_->ptr().get(), 0, &ned, &is_empty));
-
-            std::pair<int64_t, int64_t> array_shape;
-            if (is_empty == 1) {
-                array_shape = schema.domain().dimension(0).domain<int64_t>();
-            } else {
-                array_shape = std::make_pair(ned[0], ned[1]);
-            }
-
-            subarray_->add_range(0, array_shape.first, array_shape.second);
-            LOG_DEBUG(fmt::format(
-                "[ManagedQuery] Add full range to dense subarray = (0, {}, {})",
-                array_shape.first,
-                array_shape.second));
-        }
-
         // Set the subarray for range slicing
         query_->set_subarray(*subarray_);
     }
@@ -162,6 +136,8 @@ void ManagedQuery::setup_read() {
 }
 
 void ManagedQuery::submit_write(bool sort_coords) {
+    _fill_in_subarrays_if_dense(false);
+
     if (array_->schema().array_type() == TILEDB_DENSE) {
         query_->set_subarray(*subarray_);
     } else {
@@ -191,6 +167,130 @@ void ManagedQuery::submit_read() {
     });
 }
 
+// Please see the header-file comments for context.
+void ManagedQuery::_fill_in_subarrays_if_dense(bool is_read) {
+    LOG_TRACE("[ManagedQuery] _fill_in_subarrays enter");
+    // Don't do this on next-page etc.
+    if (query_->query_status() != Query::Status::UNINITIALIZED) {
+        LOG_TRACE("[ManagedQuery] _fill_in_subarrays exit: initialized");
+        return;
+    }
+    auto schema = array_->schema();
+
+    // Do this only for dense arrays.
+    if (schema.array_type() != TILEDB_DENSE) {
+        LOG_TRACE("[ManagedQuery] _fill_in_subarrays exit: non-dense");
+        return;
+    }
+
+    // Don't do this if the array doesn't have new shape AKA current domain.
+    auto current_domain = tiledb::ArraySchemaExperimental::current_domain(
+        *ctx_, schema);
+    if (current_domain.is_empty()) {
+        _fill_in_subarrays_if_dense_without_new_shape(is_read);
+    } else {
+        _fill_in_subarrays_if_dense_with_new_shape(current_domain);
+    }
+    LOG_TRACE("[ManagedQuery] _fill_in_subarrays exit");
+}
+
+void ManagedQuery::_fill_in_subarrays_if_dense_without_new_shape(bool is_read) {
+    LOG_TRACE(
+        "[ManagedQuery] _fill_in_subarrays_if_dense_without_new_shape enter");
+    // Dense array must have a subarray set for read. If the array is dense and
+    // no ranges have been set, add a range for the array's entire non-empty
+    // domain on dimension 0. In the case that the non-empty domain does not
+    // exist (when the array has not been written to yet), use dimension 0's
+    // full domain
+    if (_has_any_subarray_range_set()) {
+        return;
+    }
+
+    std::pair<int64_t, int64_t> array_shape;
+    auto schema = array_->schema();
+
+    if (!is_read && subarray_->range_num(0) > 0) {
+        LOG_TRACE(
+            "[ManagedQuery] _fill_in_subarrays_if_dense_without_new_shape "
+            "range 0 is set");
+        return;
+    }
+
+    if (is_read) {
+        // Check if the array has been written to by using the C API as
+        // there is no way to to check for an empty domain using the current
+        // C++ API.
+        int32_t is_empty;
+        int64_t ned[2];
+        ctx_->handle_error(tiledb_array_get_non_empty_domain_from_index(
+            ctx_->ptr().get(), array_->ptr().get(), 0, &ned, &is_empty));
+
+        if (is_empty == 1) {
+            array_shape = schema.domain().dimension(0).domain<int64_t>();
+        } else {
+            array_shape = std::make_pair(ned[0], ned[1]);
+        }
+    } else {
+        // Non-empty d0main is not avaiable for access at write time.
+        array_shape = schema.domain().dimension(0).domain<int64_t>();
+    }
+
+    subarray_->add_range(0, array_shape.first, array_shape.second);
+    LOG_TRACE(fmt::format(
+        "[ManagedQuery] Add full range to dense subarray dim0 = ({}, {})",
+        array_shape.first,
+        array_shape.second));
+
+    // Set the subarray for range slicing
+    query_->set_subarray(*subarray_);
+}
+
+void ManagedQuery::_fill_in_subarrays_if_dense_with_new_shape(
+    const CurrentDomain& current_domain) {
+    LOG_TRACE(
+        "[ManagedQuery] _fill_in_subarrays_if_dense_with_new_shape enter");
+    if (current_domain.type() != TILEDB_NDRECTANGLE) {
+        throw TileDBSOMAError("found non-rectangle current-domain type");
+    }
+    NDRectangle ndrect = current_domain.ndrectangle();
+
+    // Loop over dims and apply subarray ranges if not already done by the
+    // caller.
+    auto schema = array_->schema();
+    for (const auto& dim : schema.domain().dimensions()) {
+        std::string dim_name = dim.name();
+        if (subarray_range_set_[dim_name]) {
+            LOG_TRACE(fmt::format(
+                "[ManagedQuery] _fill_in_subarrays continue {}", dim_name));
+            continue;
+        }
+
+        // Dense arrays are (as of this writing in 1.15.0) all DenseNDArray.
+        // Per the spec DenseNDArray must only have dims named
+        // soma_dim_{i} with i=0,1,2,...,n-1, of type int64.
+        if (dim_name.rfind("soma_dim_", 0) != 0) {
+            throw TileDBSOMAError(fmt::format(
+                "found dense array with unexpected dim name {}", dim_name));
+        }
+        if (dim.type() != TILEDB_INT64) {
+            throw TileDBSOMAError(fmt::format(
+                "expected dense arrays to have int64 dims; got {} for {}",
+                tiledb::impl::to_str(dim.type()),
+                dim_name));
+        }
+
+        std::array<int64_t, 2> lo_hi_arr = ndrect.range<int64_t>(dim_name);
+        std::pair<int64_t, int64_t> lo_hi_pair(lo_hi_arr[0], lo_hi_arr[1]);
+        LOG_TRACE(fmt::format(
+            "[ManagedQuery] _fill_in_subarrays_if_dense_with_new_shape dim "
+            "name {} select ({}, {})",
+            dim_name,
+            lo_hi_pair.first,
+            lo_hi_pair.second));
+        select_ranges(dim_name, std::vector({lo_hi_pair}));
+    }
+}
+
 std::shared_ptr<ArrayBuffers> ManagedQuery::results() {
     if (is_empty_query()) {
         return buffers_;
diff --git a/libtiledbsoma/src/soma/managed_query.h b/libtiledbsoma/src/soma/managed_query.h
index 493c4a342b..a1cb26feff 100644
--- a/libtiledbsoma/src/soma/managed_query.h
+++ b/libtiledbsoma/src/soma/managed_query.h
@@ -151,7 +151,7 @@ class ManagedQuery {
     template <typename T>
     void select_ranges(
         const std::string& dim, const std::vector<std::pair<T, T>>& ranges) {
-        subarray_range_set_ = true;
+        subarray_range_set_[dim] = true;
         subarray_range_empty_[dim] = true;
         for (auto& [start, stop] : ranges) {
             subarray_->add_range(dim, start, stop);
@@ -168,7 +168,7 @@ class ManagedQuery {
      */
     template <typename T>
     void select_points(const std::string& dim, const std::vector<T>& points) {
-        subarray_range_set_ = true;
+        subarray_range_set_[dim] = true;
         subarray_range_empty_[dim] = true;
         for (auto& point : points) {
             subarray_->add_range(dim, point, point);
@@ -185,7 +185,7 @@ class ManagedQuery {
      */
     template <typename T>
     void select_points(const std::string& dim, const tcb::span<T> points) {
-        subarray_range_set_ = true;
+        subarray_range_set_[dim] = true;
         subarray_range_empty_[dim] = true;
         for (auto& point : points) {
             subarray_->add_range(dim, point, point);
@@ -203,7 +203,7 @@ class ManagedQuery {
     template <typename T>
     void select_point(const std::string& dim, const T& point) {
         subarray_->add_range(dim, point, point);
-        subarray_range_set_ = true;
+        subarray_range_set_[dim] = true;
         subarray_range_empty_[dim] = false;
     }
 
@@ -383,14 +383,7 @@ class ManagedQuery {
      * @return true if the query contains only empty ranges.
      */
     bool is_empty_query() {
-        bool has_empty = false;
-        for (auto subdim : subarray_range_empty_) {
-            if (subdim.second == true) {
-                has_empty = true;
-                break;
-            }
-        }
-        return subarray_range_set_ && has_empty;
+        return _has_any_empty_range() && _has_any_subarray_range_set();
     }
 
     /**
@@ -414,6 +407,53 @@ class ManagedQuery {
      */
     void check_column_name(const std::string& name);
 
+    // Helper for is_empty_query
+    bool _has_any_empty_range() {
+        for (auto subdim : subarray_range_empty_) {
+            if (subdim.second == true) {
+                return true;
+            }
+        }
+        return false;
+    }
+
+    // Helper for is_empty_query
+    bool _has_any_subarray_range_set() {
+        for (auto subdim : subarray_range_set_) {
+            if (subdim.second == true) {
+                return true;
+            }
+        }
+        return false;
+    }
+
+    /**
+     * This handles a few internals.
+     *
+     * One is that a dense array must have _at least one_
+     * dim's subarray set for a read query. Without that, reads fail immediately
+     * with the unambiguous
+     *
+     *   DenseReader: Cannot initialize reader; Dense reads must have a subarray
+     *   set
+     *
+     * The other is a combination of several things. Firstly, is current-domain
+     * support which we have for sparse arrays as of core 2.26, and for dense as
+     * of 2.27. Secondly, without current-domain support, we had small domains; with
+     * it, we have huge core domains (2^63-ish) which are immutable, and
+     * small current domains which are upward-mutable. (The soma domain and
+     * maxdomain, respectively, are core current domain and domain.) Thirdly,
+     * if a query doesn't have a subarray set on any
+     * particular dim, core will use the core domain on that dim. That was fine
+     * when core domains were small; not fine now that they are huge. In this
+     * routine, if the array is dense, for each dim without a subarray set,
+     * we set it to match the soma domain. This guarantees correct behavior.
+     */
+    void _fill_in_subarrays_if_dense(bool is_read);
+    void _fill_in_subarrays_if_dense_with_new_shape(
+        const CurrentDomain& current_domain);
+    void _fill_in_subarrays_if_dense_without_new_shape(bool is_read);
+
     // TileDB array being queried.
     std::shared_ptr<Array> array_;
 
@@ -433,7 +473,7 @@ class ManagedQuery {
     std::unique_ptr<Subarray> subarray_;
 
     // True if a range has been added to the subarray
-    bool subarray_range_set_ = false;
+    std::map<std::string, bool> subarray_range_set_ = {};
 
     // Map whether the dimension is empty (true) or not
     std::map<std::string, bool> subarray_range_empty_ = {};


Test patch:
diff --git a/libtiledbsoma/test/common.cc b/libtiledbsoma/test/common.cc
index 831b503ee5..4da1bb2994 100644
--- a/libtiledbsoma/test/common.cc
+++ b/libtiledbsoma/test/common.cc
@@ -45,7 +45,7 @@ const int CORE_DOMAIN_MAX = 2147483646;
 static std::unique_ptr<ArrowArray> _create_index_cols_info_array(
     const std::vector<DimInfo>& dim_infos);
 
-// Core PRP: https://github.com/TileDB-Inc/TileDB/pull/5303
+// Core PR: https://github.com/TileDB-Inc/TileDB/pull/5303
 bool have_dense_current_domain_support() {
     auto vers = tiledbsoma::version::embedded_version_triple();
     return std::get<0>(vers) >= 2 && std::get<1>(vers) >= 27;
diff --git a/libtiledbsoma/test/unit_soma_sparse_ndarray.cc b/libtiledbsoma/test/unit_soma_sparse_ndarray.cc
index e21edde284..89cd68abc4 100644
--- a/libtiledbsoma/test/unit_soma_sparse_ndarray.cc
+++ b/libtiledbsoma/test/unit_soma_sparse_ndarray.cc
@@ -340,8 +340,6 @@ TEST_CASE("SOMASparseNDArray: metadata", "[SOMASparseNDArray]") {
         REQUIRE(snda->metadata_num() == 2);
     }
 }
-void breakme() {
-}
 
 TEST_CASE(
     "SOMASparseNDArray: can_tiledbsoma_upgrade_shape", "[SOMASparseNDArray]") {
@@ -389,7 +387,6 @@ TEST_CASE(
     REQUIRE(dom.first == 0);
     REQUIRE(dom.second == dim_max);
 
-    breakme();
     std::vector<int64_t> newshape_wrong_dims({dim_max, 12});
     std::vector<int64_t> newshape_too_big({dim_max + 10});
     std::vector<int64_t> newshape_good({40});