Repository: single-cell-data/TileDB-SOMA (language: python)

Issue:
[Feature request] Python 3.12 support
Add support for Python 3.12 (release Oct 2023).

TileDB-SOMA 1.14
Application-level blockers:

* [x] Conda pre-check following our [established procedure](https://github.com/single-cell-data/TileDB-SOMA/wiki/Branches-and-releases)
  * https://github.com/TileDB-Inc/tiledbsoma-feedstock/pull/199
* [x] Write leak
  * #2989
* [x] Propagation of R context
  * #2997
* [x] Index-remap corner case for enumerations:
  * #2897 
  * #2993
* [x] Update ETAs for new shape, and tiledb-py deprecation:
  * #2998
* [x] A post-deprecation removal
  * #2579
  * #3000
* [x] #3003
* [x] #3005
* [x] #3006
* [x] #3004
* [ ] Python 3.12 support
  * #1849
  * This will not be included in TileDB-SOMA 1.14.0 due to unexpected complications
    * https://github.com/single-cell-data/TileDB-SOMA/issues/1849#issuecomment-2355829037
    * https://github.com/single-cell-data/SOMA/pull/222
    * https://github.com/single-cell-data/TileDB-SOMA/pull/3001


Discussion / hints:
while we are at it, it would be good to remove 3.8 support. Lots of our upstream packages have already done so (e.g. numba, pandas, etc)
Status as of 2024-02-05: `numba` 0.58 is not yet ready for Python 3.12.

```
$ docker run --rm -it --entrypoint=/bin/bash python:3.12-bookworm
root@20d715ac5ab7:/# pip install numba==0.58
Collecting numba==0.58
  Downloading numba-0.58.0.tar.gz (2.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 32.5 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [8 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-417bk_qg/numba_1954fc6d020f4c6087042050c24f72c3/setup.py", line 51, in <module>
          _guard_py_ver()
        File "/tmp/pip-install-417bk_qg/numba_1954fc6d020f4c6087042050c24f72c3/setup.py", line 48, in _guard_py_ver
          raise RuntimeError(msg.format(cur_py, min_py, max_py))
      RuntimeError: Cannot install on Python version 3.12.1; only versions >=3.8,<3.12 are supported.
      [end of output]
```

```
$ docker run --rm -it --entrypoint=/bin/bash python:3.11-bookworm
root@d8f7153f52c6:/# pip install numba==0.58
Collecting numba==0.58
  Obtaining dependency information for numba==0.58 from https://files.pythonhosted.org/packages/f7/b6/8800604a3380aa94af830c30839db93077c7fe2812a48ce7d4400d75ba9c/numba-0.58.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata
...
Installing collected packages: numpy, llvmlite, numba
Successfully installed llvmlite-0.41.1 numba-0.58.0 numpy-1.25.2
```

Numba 0.59, with Python 3.12 support has shipped.  Pandas, Scipy and others also have support for it.  Worth getting on a sprint plan IMHO
@bkmartinjr let me re-try now what failed for me earlier as shown above ...
@bkmartinjr this succeeds for me! :)

```
ubuntu@anzo[prod][][~]$ docker run --rm -it --entrypoint=/bin/bash python:3.12-bookworm
root@62dc2a187e2d:/# pip install numba==0.59
...
```

I'll re-try my #2108 probe ...
Work items detailed on #2140; more info coming here momentarily
**Problem:**

* I put up PR #2108 and it was green so I merged it
* Now PRs #2119 #2136 and anything else new is failing MacPS Python 3.12 CI fails
  * Unable to install wheel for `pyarrow`
  * https://github.com/single-cell-data/TileDB-SOMA/actions/runs/7896650237/job/21550972931?pr=2138 for example
* Another issue: TileDB-Py doesn't have wheels for 3.12
  * https://pypi.org/project/tiledb/#files

**To do:**

* Leave 3.12 support in place -- don't entirely revert #1849 
* Don't fail CI for 3.12 until the above is resolved -- #2140
* Leave this issue open for tracking

See also: https://github.com/single-cell-data/TileDB-SOMA/pull/1938

It seems that maybe:

* We get a build-from-source for `pyarrow` < 13 on MacOS, which fails as above
* `pyarrow` >= 13 on MacOS led to #1938 -- we should re-try #1938 but for Python 3.12 ...
> while we are at it, it would be good to remove 3.8 support. Lots of our upstream packages have already done so (e.g. numba, pandas, etc)

@bkmartinjr I will file a separate tracking task (#2141) -- bringing up the leading edge and deprecating the trailing edge (for all stakeholders involved) are two very different problems
Please also see https://github.com/TileDB-Inc/TileDB-Py/issues/1865

[sc-37726]
@johnkerl - FYI, I believe tiledb-py is now supported on Python 3.12 (in reference to TileDB-Inc/TileDB-Py#1865).
@bkmartinjr thanks! On the discussion list for the next sprint :v:
Pending https://github.com/TileDB-Inc/TileDB-Py/issues/1865
Huge progress: https://github.com/TileDB-Inc/TileDB/pull/5223 is near completion in core. This should go into a core 2.26, which we'd include in a tiledbsoma 1.14.
Current unsolved challenges:

* Latest Python 3.12 is 3.12.5
  * As debug I tried Python 3.12.3 which worked as debug, but isn't a good option for shipping
* TileDB-SOMA's current `typeguard` version is 4.2.0
* Python 3.12.4 added a new required argument to `evaluate_forward_ref` within Python's built-in `typing` module which broke many packages including `typeguard`
* As a result, our TileDB-SOMA CI for [PR 3001](https://github.com/single-cell-data/TileDB-SOMA/pull/3001) fails [like this](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/10892424953/job/30225213572#step:13:17210)
  * `TypeError: ForwardRef._evaluate() missing 1 required keyword-only argument: 'recursive_guard'`
* `typeguard` 4.3.0 contains a fix for this problem
  * https://github.com/agronholm/typeguard/releases
* But this surfaces another problem which is:
```
E       typeguard.TypeCheckError: item 0 of argument "coords" (tuple) did not match any element in the union:
E         NoneType: is not an instance of NoneType
E         int: is not an instance of int
E         somacore.types.Slice[int]: is not compatible with the Slice protocol because it has no attribute named '__orig_bases__'

/home/ubuntu/.pyenv/versions/3.12.5/lib/python3.12/site-packages/typeguard/_checkers.py:433: TypeCheckError
```

https://github.com/single-cell-data/SOMA/pull/222

Once that works, we'll need to:

* Merge this PR
* Tag a release of this repo (`somacore`)
* Depend on 1.0.next of `somacore` within the TileDB-SOMA repo

I've currently got the following setup on an EC2 instance:

* `somacore` locally with `pip install -e` with this PR checked out
* `tiledbsoma` locally with `pip install -e` with https://github.com/single-cell-data/TileDB-SOMA/pull/3001 checked out

Blocked by https://github.com/single-cell-data/SOMA/pull/222

Fix patch:
diff --git a/.github/workflows/python-ci-full.yml b/.github/workflows/python-ci-full.yml
index fe41e3cbb5..59a3920955 100644
--- a/.github/workflows/python-ci-full.yml
+++ b/.github/workflows/python-ci-full.yml
@@ -25,7 +25,7 @@ jobs:
         # os: [ubuntu-latest, macos-latest, windows-2019]
         # TODO: add 3.12
         # https://github.com/single-cell-data/TileDB-SOMA/issues/1849
-        python-version: ['3.9', '3.10', '3.11']
+        python-version: ['3.9', '3.10', '3.11', '3.12']
         include:
           - runs-on: ubuntu-latest
             cc: gcc-11
diff --git a/.github/workflows/python-ci-minimal.yml b/.github/workflows/python-ci-minimal.yml
index 297f50e704..ab1b8c644b 100644
--- a/.github/workflows/python-ci-minimal.yml
+++ b/.github/workflows/python-ci-minimal.yml
@@ -29,7 +29,7 @@ jobs:
 
       matrix:
         os: [ubuntu-latest, macos-latest]
-        python-version: ['3.9', '3.11']
+        python-version: ['3.9', '3.12']
         include:
           - os: ubuntu-latest
             cc: gcc-11
@@ -43,6 +43,6 @@ jobs:
       python_version: ${{ matrix.python-version }}
       cc: ${{ matrix.cc }}
       cxx: ${{ matrix.cxx }}
-      report_codecov: ${{ matrix.python-version == '3.11' }}
-      run_lint: ${{ matrix.python-version == '3.11' }}
+      report_codecov: ${{ matrix.python-version == '3.12' }}
+      run_lint: ${{ matrix.python-version == '3.12' }}
     secrets: inherit
diff --git a/.github/workflows/python-ci-packaging.yml b/.github/workflows/python-ci-packaging.yml
index 3ff6d26ce1..7a9dba4fd0 100644
--- a/.github/workflows/python-ci-packaging.yml
+++ b/.github/workflows/python-ci-packaging.yml
@@ -102,7 +102,7 @@ jobs:
         run: |
           python --version
           python -m venv ./venv-soma
-          ./venv-soma/bin/pip install --prefer-binary pybind11-global typeguard sparse 'setuptools>=70.1' wheel
+          ./venv-soma/bin/pip install --prefer-binary pybind11-global typeguard sparse wheel 'setuptools>=70.1'
           ./venv-soma/bin/pip list
       - name: Build wheel
         run: |
@@ -333,7 +333,9 @@ jobs:
           otool -L ./venv-soma/lib/python*/site-packages/tiledbsoma/pytiledbsoma.*.so
           otool -l ./venv-soma/lib/python*/site-packages/tiledbsoma/pytiledbsoma.*.so
       - name: Install runtime dependencies
-        run: ./venv-soma/bin/pip install --prefer-binary `grep -v '^\[' apis/python/src/tiledbsoma.egg-info/requires.txt`
+        run: |
+          grep -v '^\[' apis/python/src/tiledbsoma.egg-info/requires.txt > runtime-reqs.txt
+          ./venv-soma/bin/pip install --prefer-binary -r runtime-reqs.txt
       - name: Runtime test
         run: ./venv-soma/bin/python -c "import tiledbsoma; print(tiledbsoma.pytiledbsoma.version())"
 
diff --git a/.github/workflows/python-packaging.yml b/.github/workflows/python-packaging.yml
index 7f30c30c91..acca4e1137 100644
--- a/.github/workflows/python-packaging.yml
+++ b/.github/workflows/python-packaging.yml
@@ -119,6 +119,8 @@ jobs:
             dotted-version: '3.10'
           - undotted-version: '311'
             dotted-version: '3.11'
+          - undotted-version: '312'
+            dotted-version: '3.12'
         wheel-name:
           - manylinux2014
           - macos-x86_64
diff --git a/apis/python/requirements_dev.txt b/apis/python/requirements_dev.txt
index 509e738cbd..7e1fe8532a 100644
--- a/apis/python/requirements_dev.txt
+++ b/apis/python/requirements_dev.txt
@@ -3,5 +3,8 @@ ruff
 pytest
 pytest-cov
 sparse
-typeguard==4.2.1
+# Python 3.12 support requires https://github.com/agronholm/typeguard/pull/490;
+# use Typeguard @ HEAD until that PR is included in a release. See also:
+# https://github.com/single-cell-data/TileDB-SOMA/issues/3216
+typeguard @ git+https://github.com/agronholm/typeguard
 types-setuptools


Test patch:
diff --git a/apis/python/tests/test_dataframe.py b/apis/python/tests/test_dataframe.py
index 8f0f0fc482..162d23865c 100644
--- a/apis/python/tests/test_dataframe.py
+++ b/apis/python/tests/test_dataframe.py
@@ -783,7 +783,9 @@ def make_multiply_indexed_dataframe(
             "domain": [[-1000, 1000]],
             "coords": [{"bogus": True}],
             "A": None,
-            "throws": TypeError,
+            # Disable Typeguard while asserting this error, otherwise a typeguard.TypeCheckError is
+            # raised (though that's not what would happen in production)
+            "throws": (TypeError, False),
         },
         {
             "name": "bad index type bool",
@@ -890,15 +892,30 @@ def test_read_indexing(tmp_path, io):
         read_kwargs.update(
             {k: io[k] for k in ("coords", "partitions", "value_filter") if k in io}
         )
-        if io.get("throws", None):
-            with pytest.raises(io["throws"]):
+
+        # `throws` can be `Type[Exception]`, or `(Type[Exception], bool)` indicating explicitly
+        # whether Typeguard should be enabled during the `with raises` check.
+        throws = io.get("throws", None)
+        if throws:
+            if isinstance(throws, tuple) and not throws[1]:
+                # Disable Typeguard, verify actual runtime error type (avoid
+                # `typeguard.TypeCheckError` short-circuit)
+                throws = throws[0]
+                throws_ctx = raises_no_typeguard
+            else:
+                throws_ctx = pytest.raises
+        else:
+            throws_ctx = None
+
+        if throws_ctx:
+            with throws_ctx(throws):
                 next(sdf.read(**read_kwargs))
         else:
             table = next(sdf.read(**read_kwargs))
             assert table["A"].to_pylist() == io["A"]
 
-        if io.get("throws", None):
-            with pytest.raises(io["throws"]):
+        if throws_ctx:
+            with throws_ctx(throws):
                 next(sdf.read(**read_kwargs)).to_pandas()
         else:
             table = next(sdf.read(**read_kwargs)).to_pandas()
diff --git a/apis/python/tests/test_sparse_nd_array.py b/apis/python/tests/test_sparse_nd_array.py
index 5472a887c8..fc45c64126 100644
--- a/apis/python/tests/test_sparse_nd_array.py
+++ b/apis/python/tests/test_sparse_nd_array.py
@@ -1036,7 +1036,7 @@ def test_sparse_nd_array_error_corners(tmp_path):
 
     with soma.SparseNDArray.open(tmp_path.as_posix()) as a:
         # other coord types are illegal
-        with pytest.raises(TypeError):
+        with raises_no_typeguard(TypeError):
             next(a.read("hi").tables())