Repository: single-cell-data/TileDB-SOMA (language: python)

Issue:
[Bug] Test failures from `test_shape.py` with libtiledb 2.26
**Describe the bug**
New nightly test failures that started last night. When building nightly SOMA against either nightly libtiledb (or 2.2.6.0-rc3), the following 12 tests from `test_shape.py` started failing:


```
=========================== short test summary info ============================
FAILED apis/python/tests/test_shape.py::test_sparse_nd_array_basics[shape_exc0-element_dtype0] - tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
FAILED apis/python/tests/test_shape.py::test_sparse_nd_array_basics[shape_exc0-element_dtype1] - tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
FAILED apis/python/tests/test_shape.py::test_sparse_nd_array_basics[shape_exc0-element_dtype2] - tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
FAILED apis/python/tests/test_shape.py::test_sparse_nd_array_basics[shape_exc0-element_dtype3] - tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
FAILED apis/python/tests/test_shape.py::test_sparse_nd_array_basics[shape_exc1-element_dtype0] - tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
FAILED apis/python/tests/test_shape.py::test_sparse_nd_array_basics[shape_exc1-element_dtype1] - tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
FAILED apis/python/tests/test_shape.py::test_sparse_nd_array_basics[shape_exc1-element_dtype2] - tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
FAILED apis/python/tests/test_shape.py::test_sparse_nd_array_basics[shape_exc1-element_dtype3] - tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
FAILED apis/python/tests/test_shape.py::test_sparse_nd_array_basics[shape_exc2-element_dtype0] - tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
FAILED apis/python/tests/test_shape.py::test_sparse_nd_array_basics[shape_exc2-element_dtype1] - tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
FAILED apis/python/tests/test_shape.py::test_sparse_nd_array_basics[shape_exc2-element_dtype2] - tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
FAILED apis/python/tests/test_shape.py::test_sparse_nd_array_basics[shape_exc2-element_dtype3] - tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
==== 12 failed, 1788 passed, 2 xfailed, 5022 warnings in 250.52s (0:04:10) =====
```


**To Reproduce**
* Install libtiledb 2.26.0-rc3 or build from source from the dev branch
* Install SOMA from main
* Run the Python tests

**Versions (please complete the following information):**
 - TileDB-SOMA version: nightly (https://github.com/single-cell-data/TileDB-SOMA/commit/796c168ae280e73020aecbd2110d4f2f3417dda2)
 - Language and language version (e.g. Python 3.8, R 4.2.2): Python 3.11.9
 - OS (e.g. MacOS, Ubuntu Linux): Ubuntu 22.04


**Additional context**

* Identified in https://github.com/jdblischak/centralized-tiledb-nightlies/issues/21
* Related to https://github.com/single-cell-data/TileDB-SOMA/issues/2407
[python] Errors raised as `RuntimeError` should raise `SOMAError`
Errors raised from inside SOMAReader (see example below) can't be caught as a `TileDBError` or `SOMAError`, as they are caught and rethrown as a `RuntimeError` inside the C++ bindings.

```
In [28]: sdf.schema
Out[28]: 
soma_joinid: int64
obs_id: large_string
n_genes: int64
percent_mito: float
n_counts: float
louvain: large_string
is_b_cell: bool

In [29]: sdf.read(value_filter="n_genes == b'abc'")
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In [29], line 1
----> 1 sdf.read(value_filter="n_genes == b'abc'")

File ~/TileDB-SOMA/apis/python/src/tiledbsoma/dataframe.py:250, in DataFrame.read(***failed resolving arguments***)
    247 if value_filter is not None:
    248     query_condition = QueryCondition(value_filter)
--> 250 sr = clib.SOMAReader(
    251     self._uri,
    252     name=self.__class__.__name__,
    253     schema=A.schema,  # query_condition needs this
    254     column_names=column_names,
    255     query_condition=query_condition,
    256     platform_config={} if self._ctx is None else self._ctx.config().dict(),
    257     result_order=result_order.value,
    258 )
    260 if coords is not None:
    261     if not isinstance(coords, (list, tuple)):

RuntimeError: TileDBError: Cannot cast `b'abc'` to int64.```


Ideally, catching these as `SOMAError` would be possible, or worst case, as `TileDBError`
[python/r/c++] Revisit `shape` for component arrays
## PRs

**Merged PRs:**

<details>

* #2909 `kerl/schevo-timestamp-methodize`
* #2913 `kerl/name-neaten`
* #2908 `kerl/ut-soma-exc-simplify`
* #2910 `kerl/test-common-parameterize`
* #2918 `kerl/cpp-test-deadstrip`
* #2919 `kerl/minor-unit-test-helper-mod`
* #2936 `kerl/cpp-ut-helper-neaten`
* #2938 `kerl/more-cur-dom-parameterize`
* #2915 `kerl/cpp-strict-int64-shape`
* #2911 `kerl/arrow-util-current-domain-optional`
* #2939 `kerl/step-two-temp`
* #2947 `kerl/cpp-ndarray-resize-testing`
* #2945 `kerl/dataframe-test-fixture`
* #2944 `kerl/cpp-variant-indexed-dataframes`
* #2916 `kerl/sdf-shape`
* #2917 `kerl/cpp-resizes`
* #2948 `kerl/upgrade-shape-int64`
* #2951 `kerl/sdf-test-accessors`
* #2953 `kerl/py-r-accessor-plumbing`
* #2957 `kerl/sdf-domain-accessors`
* #2960 `kerl/dense-link`
* #2963 `kerl/nightly-fix`
* #2970 `kerl/dense-writeable-after-create`
* #2968 `kerl/minor-trim`
* #2969 `kerl/more-py-domain-name-neaten`
* #2972 `kerl/libtiledbsoma-env-logging-level`
* #2962 `kerl/py-r-creation-paths`
* #2950 `kerl/py-r-test-2`
* #2994 `kerl/nanoarrow-helpers`
* #3011 `kerl/polydom3`
* #3017 `kerl/polydom5`
* #3018 `kerl/polydom6`
* #2990 `kerl/variant-nnz-bug`
* #3019 `kerl/index-swap`
* #3020 `kerl/ut-max-shape`
* #3012 `kerl/polydom4`
* #3025 `kerl/fix-3020-merge`
* #3026 `kerl/one-more-rename`
* #3028 `kerl/ff-not`
* #3029 `kerl/ut-vg`
* #3030 `kerl/table-utils-memory`
* #3067 `kerl/improve-sdf-test-field-names`
* #3068 `kerl/ut-generate`
* #3069 `kerl/cpp-sdf-domain-at-create`
* #3027 `kerl/hll-domainish`
* #3088 `kerl/max-domain-int64`
* #3090 `kerl/maybe-resize-soma-joinid-cpp-tweak`
* #3032 `kerl/sdf-domain-at-create` -- fixes #2967
* #3091 `kerl/maybe-resize-soma-joinid-py-r`
* #3095 `kerl/cpp-exp-resize-prep`
* #3089 `kerl/r-dataframe-shapeable`
* https://github.com/single-cell-data/SOMA/pull/233
* #3125 `kerl/cpp-ut-name-shortens`
* #3127 `kerl/helper-rename`
* #3130 `kerl/cpp-can-resizers-names`
* #3132 `kerl/cpp-dataframe-sizing-helpers`
* #3139 `kerl/cpp-dataframe-upgrade-test`
* #3140 `kerl/py-resizer-connects`
* #3151 `kerl/py-can-upgrade-shape`
* #3152 `kerl/registration-shape-acceessors`
* #3156 `kerl/py-exp-shaping`
* #3157 `kerl/py-exp-shaping2`
* #3148 `kerl/py-exp-resize`
* #3191 `kerl/py-domain-at-create-ut-1`
* #3190 `kerl/py-domain-at-create-ut-2`
* #3192 `kerl/py-domain-at-create-ut-3`
* #3193 `kerl/py-domain-at-create-ut-4`
* #3194 `kerl/py-domain-at-create-ut-5`
* #3203 `kerl/min-size-2`
* #3208 `kerl/r-min-sizing`
* #3211 `kerl/cpp-ugr-dom`
* #3232 `kerl/ff-interop`
* #3230 `kerl/ffon`
* #3234 `kerl/docstring-prune`
* #3236 `kerl/prefixing`
* #3241 `kerl/fix-bad-merge`
* #3235 `kerl/py-r-ugr-dom`
* #3237 `kerl/py-r-ugr-dom-2`
* #3238 `kerl/py-r-ugr-dom-3`
* #3253 `kerl/set-coords-rename`
* #3261 `kerl/pybind11-nda-sizing`
* #3265 `kerl/dense-227-a`
* #3263 `kerl/dense-range-trim`
* #3268 `kerl/dim-explosion`
* #3269 `kerl/python-227-dense-ned-read`
* #3270 `kerl/r-227-dense-fixes`
* #3280 `kerl/r-dense-227-more`
* #3286 `kerl/more-fn4m`
* #3283 `kerl/readthedocs-pre-1.15`
* #3288 `kerl/dense-ugrsh`
* #3289 `kerl/notebook-shape-upgrade`
* #3285 `kerl/new-shape-doc-updates`
* #3295 `kerl/notebook-data-refresh`
* #3290 `kerl/notebook-new-shape-refresh`
* #3301 `kerl/ffena`
* #3303 `kerl/r-data-refresh`
* #3300 `kerl/sdf-sjid-lower-zero`
* #3296 `kerl/dense-example-data-refresh`
* #3294 `kerl/new-shape-notebook-and-vignette`
* #3308 `kerl/upgrade-experiment-resources`
* #3309 `kerl/fix-notebook-merge`
* #3321 `kerl/more-use-shape`
* #3358 `kerl/revert-3300`
* #3368 `kerl/227a`
* #3369 `kerl/ucd1`
* #3370 `kerl/ucd2`
* #3371 `kerl/ucd3`
* #3372 `kerl/ucd4`
* #3396 `kerl/domain-at-create-docstrings`
* #3302 `kerl/new-shape-vignette`
* https://github.com/single-cell-data/SOMA/pull/250
* #3399 `kerl/new-shape-more-docstrings`
* #3400 `kerl/check-only-r`
* https://github.com/TileDB-Inc/somacore-feedstock/pull/27

</details>

**Closed/abandoned PRs:**

<details>

* #2785 -- This was only dogfooding for the core 2.25 release -- not to be merged
* #2952 `kerl/feature-flag-temp` -- folded into 2962
* #2995 `kerl/polydom`
* #2964 `kerl/tiledbsoma-io-test`
* #3189 `kerl/min-size`
* #3220 `kerl/cpp-ugr-dom-2`
* #3244 `kerl/dense-227-fixes`
* #3276 `kerl/dataframe-shape`

</details>

**Issues which are related but non-blocking:**

* https://github.com/single-cell-data/SOMA/pull/216
* https://github.com/TileDB-Inc/TileDB/pull/5303
* #2966
* #3081
* Note: R append mode does not exist yet -- see #1630 -- so an experiment-level resizer is not a priority in R
* #3271 
* #3272
* #3273

**See also:** [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Problem to be solved

Users want to know the `shape` of an array, in the SciPy sense:

* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`.
  * Some users need the ability to grow their datasets later, using either `tiledbsoma.io`'s append mode, or subsequent writes using the `tiledbsoma` API.
  * Note that the cellxgene census doesn't need this: eact week's published census has fixed shape, and any updates will happen in new storage, on a new week.

Using TileDB-SOMA up until the present:

* The TIleDB `domain` is immutable after array creation
  * This does bounds-checking for reads and writes, which is good
  * To leverage this to function as a `shape`, users would need to set the `domain` at array-creation time. However, users lose the ability to grow their datasets later.
* There is a `non_empty_domain` accessor
  * This only indicates min/max coordinates _at which data exists_. Consider an `X` array for 100 cells and 200 genes. If non-zero expression counts exist only for cell join IDs 2-17, then the `non_empty_domain` will indicate `(2,17)` along `soma_dim_0`.
  * Consider an `obms["X_pca"]` within the same experiment. This may be 100 cells by 50 PCA components: we need a placd to store the number 50.
  * Therefore users cannot leverage this to function as a `shape` accessor.
* We have offered a `used_shape` accessor since TileDB-SOMA 1.5.
  * This functions as a `shape` accessor, in the SciPy sense, but it is not multi-writer safe.

New feature for TileDB-SOMA 1.15:

* Arrays will have a  `shape`
* Reads and writes are bounds-checked against the shape
* This retains its value regardless of which values of a sparse array are or are not actually occupied
* Users can `resize`
* The `used_shape` accessor will be deprecated in TileDB-SOMA 1.13, and slated for removal in TileDB-SOMA 1.14.

Compatiblity:

This will now require users to do an explicit `resize` before appending/growing TileDB-SOMA Experiments. Guidance in the form of example notebooks will be provided.

##  Tracking

See also: [[sc-41074]](https://app.shortcut.com/tiledb-inc/story/41074) and [[sc-51048]](https://app.shortcut.com/tiledb-inc/story/51048).

## Scheduling

Support arrives in TileDB Core 2.25. Deprecations for TileDB-SOMA will be released with 1.13. Full support within TileDB-SOMA will be release in 1.14.

## Details

SOMA API mods as we've discussed in a Google doc are as follows.

### `SOMADataFrame`

* `create`: Retain the `domain` argument
  * Issue:
    * Core has a `(lo, hi)` tuple per dim, e.g. `(0,99)` or `(10,19)`
    * SOMA has count per dim, with 0 implicit: e.g. 100 or 20
    * For `SparseNDArray` and `DenseNDArray` core can have `(lo, hi)` and SOMA can have `count`
    * For `DataFrame` there can be multiple dims --- default is a single `soma_joinid`
    * That could be treated either in `(lo, hi)` fashion or `count` fashion
    * However additional dims (e.g. `cell_type`) can be on any type, including strings, floats, etc. where there is no implicit lo=0
    * Therefore we need to keep the current SOMA API wherein `DataFrame` takes a `domain` argument (in `(lo, hi)` fashion) and not a `shape` argument (in `count` fashion)

### `SparseNDArray and DenseNDArray`

* `create`
  * Have an optional shape argument which is of type `Tuple[Int,...]` where each element is the cell count of the corresponding dimension
    * If unsupplied, or if supplied but None in any slot: use the minimum 0 in each slot – nothing larger makes sense since we will not support downsize
  * User guidance should make clear that it will not be possible to create an ‘old’ style array with the ‘new style’ API. (See also the upgrade logic below.)

### All three of `SOMADataFrame`, `SparseNDArray`, `DenseNDArray`

* `write`
  * For new arrays, created with the new shape feature:
    * Core will bounds-check that coordinates provided at `write` time are within the current shape
    * Core will raise `tiledb.cc.TileDBError` to TileDB-SOMA, which will catch and raise `IndexError`, and R-standard behavior on the R side
  * For old arrays created before this feature:
    * Core will not bounds-check that coordinates provided at write time are within the current shape
* Existing `used_shape` accessor
  * TileDB-SOMA will deprecate this over a release cycle.
  * For new arrays: raise `NotImplementedError`
  * For old arrays: return what’s currently returned, with a deprecation warning.
  * Mechanism for determining old vs. new: `array.schema.version` (the core storage version).
* Existing `shape` accessor
  * For new arrays:
    * Have this return the new shape as proposed by core, no longer returning the TileDB domain.
  * For old arrays created before this feature:
    * Return the TileDB domain as now.
* Existing `non_empty_domain` accessor
  * Same behavior for old and new arrays (unaffected by this proposal).
  * Keep this accessor supported, but, with user notes that it’s generally non-useful
  * This should return None (or R equivalent) when there is a schema but no data have been written.
* New `maxshape` accessor
  * Maps the core-level `(lo, hi)` accessor for domain to count-style accessor hi+1. E.g. if the core domain is either `(0,99)` or `(50,99)` then TileDB-SOMA `maxshape` will say 100.
  * Same behavior for old and new arrays.
  * Let users query for what the TileDB domain is, with user notes that it’s the maximum that users can reshape to.
  * Issac suggests: maybe `domain` or `maxshape` (see h5py).
* New `resize` mutator
  * Note: `reshape` means something else in the community (numpy, zarr, h5py), e.g. a 5x20 (total 100 cells) being reinterpreted as 4x25 (still 100 cells). The standard name for changing cell-count is `resize`.
  * For old arrays created before this feature: raise `NotImplementedError`.
  * For new arrays:
    * Will raise `ValueError` if the new shape is smaller on any dim than currently in storage
    * Regardless of whether any data have been written whatsoever
    * Will raise `ValueError` if the new shape exceeds the TileDB domain from create time, which will serve TileDB-SOMA in a role of “max possible shape the user can reshape to”
    * Otherwise, any calls to write from this point will bounds-check writes within this new shape
    * **We don’t expect resize to be multi-writer safe with regard to write ; user notes must be clear on this point**
* New `tiledbsoma_upgrade_shape` method for SparseNDArray and DenseNDArray
  * This will leverage `array.schema.version` to see if an upgrade is needed
  * Leverage core support for storage-version updates
  * This will take a shape argument as in `create`
  * For arrays created with “just-right” size: this will succeed
  * For arrays created with “room-for-growth” / “two billion-ish” size: this will succeed
  * If the user passes a shape which exceeds the current TileDB domain: this will fail
* New `tiledbsoma_upgrade_domain method` for `DataFrame`
  * Same as for `SparseNDArray`/`DenseNDArray` except it will take a domain at the SOMA-API level just as `DataFrame`'s create method

### `tiledbsoma.io`

* The user-facing API has no shape arguments and thus won’t need changing.
* Internally to `tiledbsoma.io`, we’ll still ask the tiledbsoma API for the “big domain” (2 billionish)
* Append mode:
  * Will need a new `resize` method at the `Experiment` level
  * Users will need to:
    * Register as now
    * Call the experiment-level `resize`
      * Could be `exp.resize(...)`, or (better) this could be `tiledbsoma.io.reshape_experiment`
  * In either case: this method will take the new `obs` and `var` counts as inputs:
    * `exp.obs.reshape` to new `obs` count
    * `exp.ms[name].var.reshape` to new `var` count
    * `exp.ms[name].X[name].reshape` to new `obs` count x `var` count
    * `exp.ms[name].obsm[name].reshape` to new `obs` count x same width
    * `exp.ms[name].obsp[name].reshape` to new `obs` count x `obs` count
    * `exp.ms[name].varm[name].reshape` to new `var` count x same width
    * `exp.ms[name].varp[name].reshape` to new `var` count x `var` count
  * Do the individual append-mode writes as now



Discussion / hints:
On one system we get
```
RuntimeError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
```

On another we get
```
tiledb.cc.TileDBError: Subarray: Cannot add range to dimension 'soma_dim_0'; Range [110, 110] is out of domain bounds [0, 99]
```

Meanwhile
https://github.com/single-cell-data/TileDB-SOMA/blob/796c168ae280e73020aecbd2110d4f2f3417dda2/apis/python/tests/test_shape.py#L101-L103
-- as it so happens, this is an issue @nguyenv and I have been dealing with before, namely, ensuring all the throws get mapped to `TileDBSOMAError`

This is an instance related to https://github.com/single-cell-data/TileDB-SOMA/issues/783
Here is a second evaluation error which returns a different (still wrong IMHO) error.  Ideally it would raise a SOMAError, clearly indicating that the operation is unsupported.

There is code in query_condition.py that is trying to test for this (use of unsupported operator), but for some reason it is not catching the error at parse time.

```
In [37]: sdf.read(value_filter="n_genes > 1+1")
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In [37], line 1
----> 1 sdf.read(value_filter="n_genes > 1+1")

File ~/TileDB-SOMA/apis/python/src/tiledbsoma/dataframe.py:250, in DataFrame.read(***failed resolving arguments***)
    247 if value_filter is not None:
    248     query_condition = QueryCondition(value_filter)
--> 250 sr = clib.SOMAReader(
    251     self._uri,
    252     name=self.__class__.__name__,
    253     schema=A.schema,  # query_condition needs this
    254     column_names=column_names,
    255     query_condition=query_condition,
    256     platform_config={} if self._ctx is None else self._ctx.config().dict(),
    257     result_order=result_order.value,
    258 )
    260 if coords is not None:
    261     if not isinstance(coords, (list, tuple)):

RuntimeError: AttributeError: 'Constant' object has no attribute 'combine'
```


#2785 is a quick-and-dirty concept-prover -- its sole function is to flush out any API misunderstandings we might have, in prep for 2.25.0 core release.

Fix patch:
diff --git a/apis/python/src/tiledbsoma/pytiledbsoma.cc b/apis/python/src/tiledbsoma/pytiledbsoma.cc
index fced2214b2..f63ebde488 100644
--- a/apis/python/src/tiledbsoma/pytiledbsoma.cc
+++ b/apis/python/src/tiledbsoma/pytiledbsoma.cc
@@ -32,6 +32,13 @@ PYBIND11_MODULE(pytiledbsoma, m) {
 
     /* We need to make sure C++ TileDBSOMAError is translated to a
      * correctly-typed Python error
+     *
+     * We're aware of
+     * https://pybind11.readthedocs.io/en/stable/advanced/exceptions.html
+     * -- we find empirically that despite this translator, we still
+     * find it necessary to do explicit catch-and-rethrow within our
+     * pybind11 functions. See also
+     * https://github.com/single-cell-data/TileDB-SOMA/pull/2963
      */
     py::register_exception_translator([](std::exception_ptr p) {
         auto tiledb_soma_error = (py::object)py::module::import("tiledbsoma")
diff --git a/apis/python/src/tiledbsoma/soma_array.cc b/apis/python/src/tiledbsoma/soma_array.cc
index f89f7414f3..468be29894 100644
--- a/apis/python/src/tiledbsoma/soma_array.cc
+++ b/apis/python/src/tiledbsoma/soma_array.cc
@@ -247,62 +247,67 @@ void load_soma_array(py::module& m) {
 
                     auto coords = array_handle.attr("tolist")();
 
-                    if (!strcmp(arrow_schema.format, "l")) {
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<int64_t>>());
-                    } else if (!strcmp(arrow_schema.format, "i")) {
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<int32_t>>());
-                    } else if (!strcmp(arrow_schema.format, "s")) {
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<int16_t>>());
-                    } else if (!strcmp(arrow_schema.format, "c")) {
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<int8_t>>());
-                    } else if (!strcmp(arrow_schema.format, "L")) {
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<uint64_t>>());
-                    } else if (!strcmp(arrow_schema.format, "I")) {
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<uint32_t>>());
-                    } else if (!strcmp(arrow_schema.format, "S")) {
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<uint16_t>>());
-                    } else if (!strcmp(arrow_schema.format, "C")) {
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<uint8_t>>());
-                    } else if (!strcmp(arrow_schema.format, "f")) {
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<float>>());
-                    } else if (!strcmp(arrow_schema.format, "g")) {
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<double>>());
-                    } else if (
-                        !strcmp(arrow_schema.format, "u") ||
-                        !strcmp(arrow_schema.format, "z")) {
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<std::string>>());
-                    } else if (
-                        !strcmp(arrow_schema.format, "tss:") ||
-                        !strcmp(arrow_schema.format, "tsm:") ||
-                        !strcmp(arrow_schema.format, "tsu:") ||
-                        !strcmp(arrow_schema.format, "tsn:")) {
-                        // convert the Arrow Array to int64
-                        auto pa = py::module::import("pyarrow");
-                        coords = array_handle.attr("cast")(pa.attr("int64")())
-                                     .attr("tolist")();
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<int64_t>>());
-                    } else if (
-                        !strcmp(arrow_schema.format, "U") ||
-                        !strcmp(arrow_schema.format, "Z")) {
-                        array.set_dim_points(
-                            dim, coords.cast<std::vector<std::string>>());
-                    } else {
-                        TPY_ERROR_LOC(
-                            "[pytiledbsoma] set_dim_points: type={} not "
-                            "supported" +
-                            std::string(arrow_schema.format));
+                    try {
+                        if (!strcmp(arrow_schema.format, "l")) {
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<int64_t>>());
+                        } else if (!strcmp(arrow_schema.format, "i")) {
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<int32_t>>());
+                        } else if (!strcmp(arrow_schema.format, "s")) {
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<int16_t>>());
+                        } else if (!strcmp(arrow_schema.format, "c")) {
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<int8_t>>());
+                        } else if (!strcmp(arrow_schema.format, "L")) {
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<uint64_t>>());
+                        } else if (!strcmp(arrow_schema.format, "I")) {
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<uint32_t>>());
+                        } else if (!strcmp(arrow_schema.format, "S")) {
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<uint16_t>>());
+                        } else if (!strcmp(arrow_schema.format, "C")) {
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<uint8_t>>());
+                        } else if (!strcmp(arrow_schema.format, "f")) {
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<float>>());
+                        } else if (!strcmp(arrow_schema.format, "g")) {
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<double>>());
+                        } else if (
+                            !strcmp(arrow_schema.format, "u") ||
+                            !strcmp(arrow_schema.format, "z")) {
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<std::string>>());
+                        } else if (
+                            !strcmp(arrow_schema.format, "tss:") ||
+                            !strcmp(arrow_schema.format, "tsm:") ||
+                            !strcmp(arrow_schema.format, "tsu:") ||
+                            !strcmp(arrow_schema.format, "tsn:")) {
+                            // convert the Arrow Array to int64
+                            auto pa = py::module::import("pyarrow");
+                            coords = array_handle
+                                         .attr("cast")(pa.attr("int64")())
+                                         .attr("tolist")();
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<int64_t>>());
+                        } else if (
+                            !strcmp(arrow_schema.format, "U") ||
+                            !strcmp(arrow_schema.format, "Z")) {
+                            array.set_dim_points(
+                                dim, coords.cast<std::vector<std::string>>());
+                        } else {
+                            TPY_ERROR_LOC(
+                                "[pytiledbsoma] set_dim_points: type={} not "
+                                "supported" +
+                                std::string(arrow_schema.format));
+                        }
+                    } catch (const std::exception& e) {
+                        throw TileDBSOMAError(e.what());
                     }
 
                     // Release arrow schema
@@ -337,69 +342,135 @@ void load_soma_array(py::module& m) {
 
         .def(
             "set_dim_points_string_or_bytes",
-            static_cast<void (SOMAArray::*)(
-                const std::string&, const std::vector<std::string>&)>(
-                &SOMAArray::set_dim_points))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<std::string>& points) {
+                try {
+                    array.set_dim_points(dim, points);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_points_double",
-            static_cast<void (SOMAArray::*)(
-                const std::string&, const std::vector<double>&)>(
-                &SOMAArray::set_dim_points))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<double>& points) {
+                try {
+                    array.set_dim_points(dim, points);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_points_float",
-            static_cast<void (SOMAArray::*)(
-                const std::string&, const std::vector<float>&)>(
-                &SOMAArray::set_dim_points))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<float>& points) {
+                try {
+                    array.set_dim_points(dim, points);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_points_int64",
-            static_cast<void (SOMAArray::*)(
-                const std::string&, const std::vector<int64_t>&)>(
-                &SOMAArray::set_dim_points))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<int64_t>& points) {
+                try {
+                    array.set_dim_points(dim, points);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_points_int32",
-            static_cast<void (SOMAArray::*)(
-                const std::string&, const std::vector<int32_t>&)>(
-                &SOMAArray::set_dim_points))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<int32_t>& points) {
+                try {
+                    array.set_dim_points(dim, points);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_points_int16",
-            static_cast<void (SOMAArray::*)(
-                const std::string&, const std::vector<int16_t>&)>(
-                &SOMAArray::set_dim_points))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<int16_t>& points) {
+                try {
+                    array.set_dim_points(dim, points);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_points_int8",
-            static_cast<void (SOMAArray::*)(
-                const std::string&, const std::vector<int8_t>&)>(
-                &SOMAArray::set_dim_points))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<int8_t>& points) {
+                try {
+                    array.set_dim_points(dim, points);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_points_uint64",
-            static_cast<void (SOMAArray::*)(
-                const std::string&, const std::vector<uint64_t>&)>(
-                &SOMAArray::set_dim_points))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<uint64_t>& points) {
+                try {
+                    array.set_dim_points(dim, points);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_points_uint32",
-            static_cast<void (SOMAArray::*)(
-                const std::string&, const std::vector<uint32_t>&)>(
-                &SOMAArray::set_dim_points))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<uint32_t>& points) {
+                try {
+                    array.set_dim_points(dim, points);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_points_uint16",
-            static_cast<void (SOMAArray::*)(
-                const std::string&, const std::vector<uint16_t>&)>(
-                &SOMAArray::set_dim_points))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<uint16_t>& points) {
+                try {
+                    array.set_dim_points(dim, points);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_points_uint8",
-            static_cast<void (SOMAArray::*)(
-                const std::string&, const std::vector<uint8_t>&)>(
-                &SOMAArray::set_dim_points))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<uint8_t>& points) {
+                try {
+                    array.set_dim_points(dim, points);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         // In an initial version of this file we had `set_dim_ranges` relying
         // solely on type-overloading. This worked since we supported only int
@@ -416,80 +487,135 @@ void load_soma_array(py::module& m) {
 
         .def(
             "set_dim_ranges_string_or_bytes",
-            static_cast<void (SOMAArray::*)(
-                const std::string&,
-                const std::vector<std::pair<std::string, std::string>>&)>(
-                &SOMAArray::set_dim_ranges))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<std::pair<std::string, std::string>>& ranges) {
+                try {
+                    array.set_dim_ranges(dim, ranges);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
+
+        .def(
+            "set_dim_ranges_double",
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<std::pair<double, double>>& ranges) {
+                try {
+                    array.set_dim_ranges(dim, ranges);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
+
+        .def(
+            "set_dim_ranges_float",
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<std::pair<float, float>>& ranges) {
+                try {
+                    array.set_dim_ranges(dim, ranges);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_ranges_int64",
-            static_cast<void (SOMAArray::*)(
-                const std::string&,
-                const std::vector<std::pair<int64_t, int64_t>>&)>(
-                &SOMAArray::set_dim_ranges))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<std::pair<int64_t, int64_t>>& ranges) {
+                try {
+                    array.set_dim_ranges(dim, ranges);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_ranges_int32",
-            static_cast<void (SOMAArray::*)(
-                const std::string&,
-                const std::vector<std::pair<int32_t, int32_t>>&)>(
-                &SOMAArray::set_dim_ranges))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<std::pair<int32_t, int32_t>>& ranges) {
+                try {
+                    array.set_dim_ranges(dim, ranges);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_ranges_int16",
-            static_cast<void (SOMAArray::*)(
-                const std::string&,
-                const std::vector<std::pair<int16_t, int16_t>>&)>(
-                &SOMAArray::set_dim_ranges))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<std::pair<int16_t, int16_t>>& ranges) {
+                try {
+                    array.set_dim_ranges(dim, ranges);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_ranges_int8",
-            static_cast<void (SOMAArray::*)(
-                const std::string&,
-                const std::vector<std::pair<int8_t, int8_t>>&)>(
-                &SOMAArray::set_dim_ranges))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<std::pair<int8_t, int8_t>>& ranges) {
+                try {
+                    array.set_dim_ranges(dim, ranges);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_ranges_uint64",
-            static_cast<void (SOMAArray::*)(
-                const std::string&,
-                const std::vector<std::pair<uint64_t, uint64_t>>&)>(
-                &SOMAArray::set_dim_ranges))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<std::pair<uint64_t, uint64_t>>& ranges) {
+                try {
+                    array.set_dim_ranges(dim, ranges);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_ranges_uint32",
-            static_cast<void (SOMAArray::*)(
-                const std::string&,
-                const std::vector<std::pair<uint32_t, uint32_t>>&)>(
-                &SOMAArray::set_dim_ranges))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<std::pair<uint32_t, uint32_t>>& ranges) {
+                try {
+                    array.set_dim_ranges(dim, ranges);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_ranges_uint16",
-            static_cast<void (SOMAArray::*)(
-                const std::string&,
-                const std::vector<std::pair<uint16_t, uint16_t>>&)>(
-                &SOMAArray::set_dim_ranges))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<std::pair<uint16_t, uint16_t>>& ranges) {
+                try {
+                    array.set_dim_ranges(dim, ranges);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def(
             "set_dim_ranges_uint8",
-            static_cast<void (SOMAArray::*)(
-                const std::string&,
-                const std::vector<std::pair<uint8_t, uint8_t>>&)>(
-                &SOMAArray::set_dim_ranges))
-
-        .def(
-            "set_dim_ranges_double",
-            static_cast<void (SOMAArray::*)(
-                const std::string&,
-                const std::vector<std::pair<double, double>>&)>(
-                &SOMAArray::set_dim_ranges))
-
-        .def(
-            "set_dim_ranges_float",
-            static_cast<void (SOMAArray::*)(
-                const std::string&,
-                const std::vector<std::pair<float, float>>&)>(
-                &SOMAArray::set_dim_ranges))
+            [](SOMAArray& array,
+               const std::string& dim,
+               const std::vector<std::pair<uint8_t, uint8_t>>& ranges) {
+                try {
+                    array.set_dim_ranges(dim, ranges);
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
+            })
 
         .def("results_complete", &SOMAArray::results_complete)
 


Test patch:
diff --git a/apis/python/tests/test_dataframe.py b/apis/python/tests/test_dataframe.py
index edb5c6cfc8..1a53c7beb4 100644
--- a/apis/python/tests/test_dataframe.py
+++ b/apis/python/tests/test_dataframe.py
@@ -736,7 +736,7 @@ def make_multiply_indexed_dataframe(tmp_path, index_column_names: List[str]):
             "index_column_names": ["strings_aaa", "zero_one"],
             "coords": [[True], slice(None)],
             "A": None,
-            "throws": (RuntimeError, tiledb.cc.TileDBError, TypeError),
+            "throws": TypeError,
         },
         {
             "name": "2D index empty",
diff --git a/apis/python/tests/test_dataframe_index_columns.py b/apis/python/tests/test_dataframe_index_columns.py
index 50da7a0fd2..490039230a 100644
--- a/apis/python/tests/test_dataframe_index_columns.py
+++ b/apis/python/tests/test_dataframe_index_columns.py
@@ -3,7 +3,6 @@
 import pytest
 
 import tiledbsoma as soma
-import tiledb
 
 
 @pytest.fixture
@@ -1899,6 +1898,6 @@ def test_types_read_errors(
     with soma.DataFrame.open(uri, "w") as sdf:
         sdf.write(arrow_table)
 
-    with pytest.raises((RuntimeError, tiledb.cc.TileDBError)):
+    with pytest.raises((soma.SOMAError)):
         with soma.DataFrame.open(uri, "r") as sdf:
             sdf.read(coords=coords).concat()
diff --git a/apis/python/tests/test_dense_nd_array.py b/apis/python/tests/test_dense_nd_array.py
index 4a161b535d..3e4ffb5161 100644
--- a/apis/python/tests/test_dense_nd_array.py
+++ b/apis/python/tests/test_dense_nd_array.py
@@ -311,13 +311,13 @@ def test_dense_nd_array_slicing(tmp_path, io):
             "name": "negative",
             "shape": (10,),
             "coords": (-1,),
-            "throws": (RuntimeError, tiledb.cc.TileDBError),
+            "throws": (soma.SOMAError),
         },
         {
             "name": "12 in 10 domain",
             "shape": (10,),
             "coords": (12,),
-            "throws": (RuntimeError, tiledb.cc.TileDBError),
+            "throws": (soma.SOMAError),
         },
         {
             "name": "too many dims",
diff --git a/apis/python/tests/test_shape.py b/apis/python/tests/test_shape.py
index 42c27f2701..d96821bec6 100644
--- a/apis/python/tests/test_shape.py
+++ b/apis/python/tests/test_shape.py
@@ -98,9 +98,8 @@ def test_sparse_nd_array_basics(
 
     # Test reads out of bounds
     with tiledbsoma.SparseNDArray.open(uri) as snda:
-        # TODO: make sure this doesn't come through as RuntimeError
         # https://github.com/single-cell-data/TileDB-SOMA/issues/2407
-        with pytest.raises((RuntimeError, ValueError)):
+        with pytest.raises(tiledbsoma.SOMAError):
             coords = tuple(arg_shape[i] + 10 for i in range(ndim))
             snda.read(coords).tables().concat()
 
diff --git a/apis/python/tests/test_sparse_nd_array.py b/apis/python/tests/test_sparse_nd_array.py
index e043e925ba..ce7c859050 100644
--- a/apis/python/tests/test_sparse_nd_array.py
+++ b/apis/python/tests/test_sparse_nd_array.py
@@ -624,10 +624,7 @@ def test_csr_csc_2d_read(tmp_path, shape):
             "dims": {
                 "soma_dim_0": [2, 4],
             },
-            "throws": (
-                RuntimeError,
-                tiledb.cc.TileDBError,
-            ),
+            "throws": (soma.SOMAError),
         },
         {
             "name": "coords=[0,0]",