Repository: single-cell-data/TileDB-SOMA (language: python)

Issue:
[python] Fix nightly-build failure / `pybind11` exception-mapping
**Issue and/or context:** Resolves #2961 

**Changes:**

Do explicit try-catch. See also #783.

**Notes for Reviewer:**

It's not necessarily causal that this triggered on https://github.com/jdblischak/centralized-tiledb-nightlies/issues/21. It's true that the `apis/python/tests/test_shape.py` was recently introduced on the #2407 stack. However, whether un-rethrown errors (on implicit pybind body, as in the "before" parts of this PR) reach the Python level as `tiledbsoma.SOMAError` or `RuntimeError` or `tiledb.cc.TileDBError` has some element of randomness.

There's an attempt to globally register a rethrow:
https://github.com/single-cell-data/TileDB-SOMA/blob/1.13.1/apis/python/src/tiledbsoma/pytiledbsoma.cc#L36-L50

However, @nguyenv and I have long found that some things come through as `RuntimeError` on one machine, and `tiledb.cc.TileDBError` in another. We suspect that TileDB-Py's exception registrar has a race condition with our own.

Regardless, a safe practice is explicit catch-and-rethrow at the boundary between C++ and Python, which is what we do more of here.
[c++] Refine concurrency configuration (`soma.compute_concurrency_level`)
**Issue and/or context:**

Introduces direct control over the native thread pool concurrency level in SOMA. Previously, it was indirectly controlled by the TileDB Core `sm.compute_currency_control` configuration, and would create a thread pool that was 1/2 the size of that value.

**Changes:**

This PR introduces a new context configuration setting:  `soma.compute_currency_level`, which defaults to the host CPU count. It can be set with an numeric value,eg., in Python:

```python
context = tiledbsoma.SOMATileDBContext(tiledb_config={"soma.compute_currency_level": "20"})
```

**Notes for Reviewer:**

sc-54891

the PR also adds:
* improved error handling for malformed config values
* bound the value between 1 and 1024




Discussion / hints:
## [Codecov](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/2963?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 89.99%. Comparing base [(`796c168`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/796c168ae280e73020aecbd2110d4f2f3417dda2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) to head [(`f29a82c`)](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/commit/f29a82cbb769ca022ad9690e0d6bd67c646c1420?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data).
> Report is 4 commits behind head on main.

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main    #2963      +/-   ##
==========================================
+ Coverage   89.89%   89.99%   +0.09%     
==========================================
  Files          39       39              
  Lines        4049     4057       +8     
==========================================
+ Hits         3640     3651      +11     
+ Misses        409      406       -3     
```

| [Flag](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/2963/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/2963/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `89.99% <ø> (+0.09%)` | :arrow_up: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Components](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/2963/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | Coverage Δ | |
|---|---|---|
| [python_api](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/2963/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `89.99% <ø> (+0.09%)` | :arrow_up: |
| [libtiledbsoma](https://app.codecov.io/gh/single-cell-data/TileDB-SOMA/pull/2963/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=single-cell-data) | `∅ <ø> (∅)` | |

</details>


Fix patch:
diff --git a/apis/python/src/tiledbsoma/common.h b/apis/python/src/tiledbsoma/common.h
index 4b6632788f..e4b38e692c 100644
--- a/apis/python/src/tiledbsoma/common.h
+++ b/apis/python/src/tiledbsoma/common.h
@@ -13,22 +13,7 @@ using namespace std;
 using namespace tiledb;
 namespace py = pybind11;
 
-#define TPY_ERROR_LOC(m) throw TileDBSOMAPyError(m);
-
-class TileDBSOMAPyError : std::runtime_error {
-   public:
-    explicit TileDBSOMAPyError(const char* m)
-        : std::runtime_error(m) {
-    }
-    explicit TileDBSOMAPyError(std::string m)
-        : std::runtime_error(m.c_str()) {
-    }
-
-   public:
-    virtual const char* what() const noexcept override {
-        return std::runtime_error::what();
-    }
-};
+#define TPY_ERROR_LOC(m) throw TileDBSOMAError(m);
 
 namespace tiledbsoma {
 
diff --git a/apis/python/src/tiledbsoma/pytiledbsoma.cc b/apis/python/src/tiledbsoma/pytiledbsoma.cc
index f63ebde488..f9d56ff7fe 100644
--- a/apis/python/src/tiledbsoma/pytiledbsoma.cc
+++ b/apis/python/src/tiledbsoma/pytiledbsoma.cc
@@ -49,8 +49,6 @@ PYBIND11_MODULE(pytiledbsoma, m) {
                 std::rethrow_exception(p);
         } catch (const TileDBSOMAError& e) {
             PyErr_SetString(tiledb_soma_error.ptr(), e.what());
-        } catch (const TileDBSOMAPyError& e) {
-            PyErr_SetString(tiledb_soma_error.ptr(), e.what());
         } catch (py::builtin_exception& e) {
             throw;
         };
diff --git a/apis/python/src/tiledbsoma/reindexer.cc b/apis/python/src/tiledbsoma/reindexer.cc
index 100bf1231d..d22a0314b7 100644
--- a/apis/python/src/tiledbsoma/reindexer.cc
+++ b/apis/python/src/tiledbsoma/reindexer.cc
@@ -47,7 +47,7 @@ using namespace tiledbsoma;
  * @param lookups input values to be looked up
  * @return looked up values
  */
-py::array_t<int64_t> get_indexer_general(
+py::array_t<int64_t> get_indexer_general_aux(
     IntIndexer& indexer, py::array_t<int64_t> lookups) {
     auto input_buffer = lookups.request();
     int64_t* input_ptr = static_cast<int64_t*>(input_buffer.ptr);
@@ -58,6 +58,14 @@ py::array_t<int64_t> get_indexer_general(
     indexer.lookup(input_ptr, results_ptr, size);
     return results;
 }
+py::array_t<int64_t> get_indexer_general(
+    IntIndexer& indexer, py::array_t<int64_t> lookups) {
+    try {
+        return get_indexer_general_aux(indexer, lookups);
+    } catch (const std::exception& e) {
+        throw TileDBSOMAError(e.what());
+    }
+}
 
 /***
  * Helper function to provide data and schema for an arrow object
@@ -83,7 +91,7 @@ void extract_py_array_schema(
  * @py_arrow_array pyarrow inputs to be looked up
  * @return looked up values
  */
-py::array_t<int64_t> get_indexer_py_arrow(
+py::array_t<int64_t> get_indexer_py_arrow_aux(
     IntIndexer& indexer, py::object py_arrow_array) {
     // Check if it is not a pyarrow array or pyarrow chunked array
     if (!py::hasattr(py_arrow_array, "_export_to_c") &&
@@ -134,6 +142,15 @@ py::array_t<int64_t> get_indexer_py_arrow(
     return results;
 }
 
+py::array_t<int64_t> get_indexer_py_arrow(
+    IntIndexer& indexer, py::object py_arrow_array) {
+    try {
+        return get_indexer_py_arrow_aux(indexer, py_arrow_array);
+    } catch (const std::exception& e) {
+        throw TileDBSOMAError(e.what());
+    }
+}
+
 void load_reindexer(py::module& m) {
     // Efficient C++ re-indexing (aka hashing unique key values to an index
     // between 0 and number of keys - 1) based on khash
@@ -143,7 +160,11 @@ void load_reindexer(py::module& m) {
         .def(
             "map_locations",
             [](IntIndexer& indexer, py::array_t<int64_t> keys) {
-                indexer.map_locations(keys.data(), keys.size());
+                try {
+                    indexer.map_locations(keys.data(), keys.size());
+                } catch (const std::exception& e) {
+                    throw TileDBSOMAError(e.what());
+                }
             })
         // Perform lookup for a large input array of keys and writes the
         // looked up values into previously allocated array (works for the
diff --git a/apis/python/src/tiledbsoma/soma_array.cc b/apis/python/src/tiledbsoma/soma_array.cc
index f301f0947a..afad4bc758 100644
--- a/apis/python/src/tiledbsoma/soma_array.cc
+++ b/apis/python/src/tiledbsoma/soma_array.cc
@@ -46,9 +46,13 @@ void write(SOMAArray& array, py::handle py_batch, bool sort_coords = true) {
     uintptr_t arrow_array_ptr = (uintptr_t)(&arrow_array);
     py_batch.attr("_export_to_c")(arrow_array_ptr, arrow_schema_ptr);
 
-    array.set_array_data(
-        std::make_unique<ArrowSchema>(arrow_schema),
-        std::make_unique<ArrowArray>(arrow_array));
+    try {
+        array.set_array_data(
+            std::make_unique<ArrowSchema>(arrow_schema),
+            std::make_unique<ArrowArray>(arrow_array));
+    } catch (const std::exception& e) {
+        TPY_ERROR_LOC(e.what());
+    }
 
     try {
         array.write(sort_coords);


Test patch:
diff --git a/apis/python/tests/test_dataframe.py b/apis/python/tests/test_dataframe.py
index 1a53c7beb4..38cd2ef1f2 100644
--- a/apis/python/tests/test_dataframe.py
+++ b/apis/python/tests/test_dataframe.py
@@ -214,7 +214,7 @@ def test_dataframe_with_enumeration(tmp_path):
         data["soma_joinid"] = [0, 1, 2, 3, 4]
         data["foo"] = ["a", "bb", "ccc", "bb", "a"]
         data["bar"] = ["cat", "dog", "cat", "cat", "cat"]
-        with pytest.raises(ValueError):
+        with pytest.raises(soma.SOMAError):
             sdf.write(pa.Table.from_pydict(data))
 
         data["foo"] = pd.Categorical(["a", "bb", "ccc", "bb", "a"])
@@ -1458,7 +1458,7 @@ def test_enum_extend_past_numerical_limit(tmp_path):
 
     # cannot add additional categories as already maxed out earlier
     tbl = pa.Table.from_pandas(df2, preserve_index=False)
-    with pytest.raises((RuntimeError, soma.SOMAError)):
+    with pytest.raises(soma.SOMAError):
         with soma.open(uri, mode="w") as A:
             A.write(tbl)
 
diff --git a/apis/python/tests/test_indexer.py b/apis/python/tests/test_indexer.py
index b182bc66ac..09f94690ab 100644
--- a/apis/python/tests/test_indexer.py
+++ b/apis/python/tests/test_indexer.py
@@ -6,6 +6,7 @@
 import pyarrow as pa
 import pytest
 
+from tiledbsoma import SOMAError
 from tiledbsoma._indexer import IntIndexer
 from tiledbsoma.options import SOMATileDBContext
 from tiledbsoma.options._soma_tiledb_context import _validate_soma_tiledb_context
@@ -22,7 +23,7 @@ def test_duplicate_key_indexer_error(
     keys: Union[np.array, List[int]], lookups: np.array
 ):
     context = _validate_soma_tiledb_context(SOMATileDBContext())
-    with pytest.raises(RuntimeError, match="There are duplicate keys."):
+    with pytest.raises(SOMAError, match="There are duplicate keys."):
         IntIndexer(keys, context=context)
 
     pd_index = pd.Index(keys)