Repository: swig/swig (language: cpp)

Issue:
__package__ will cease being set/used  in Python 3.15
The files generated by SWIG depend on `__package__` being present, but Python 3.14 will removing this in favor of `__spec__.parent`. The 3.13 release notes https://docs.python.org/3.13/whatsnew/3.13.html#pending-removal-in-python-3-14 say:

> [importlib](https://docs.python.org/3.13/library/importlib.html#module-importlib): `__package__` and `__cached__` will cease to be set or taken into consideration by the import system ([gh-97879](https://github.com/python/cpython/issues/97879)).

Both could be checked, or just `__spec__.parent` - I can verify that works back to 3.6, which is the oldest Python I had to check.


Discussion / hints:
Python 3.3 is the oldest SWIG currently supports and seems to lack `__spec__` entirely:

```
Python 3.3.5 (default, Mar 22 2014, 13:24:53) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information. 
>>> dir(__spec__)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name '__spec__' is not defined
>>> 
```
Added in 3.4: https://docs.python.org/3/reference/import.html#spec__
FWIW I think it would be fine to set 3.4 as the minimum required version, I think nobody uses anything less than 3.6 or maybe even later any more by now.
3.3 is _really_ old, by the way, and never was well supported - throughout 3.3's support window, 2.x was dramatically more popular. 3.5 was the first version with heavy use. Personally, 3.6 is still a _really_ old version of Python and a fine minimum; 3.6 is the oldest version you can build for via manylinux, for example. 3.8 is the oldest officially supported version of Python these days.
No arguments from me, but you'll need to convince wsfulton...
FWIW, I'd argue for the minimum Python3 version being 3.5 since that's the oldest we've actually been testing in CI since github dropped their ubuntu-18.04 image over a year ago.

Requiring CI testing for a version of a target language aligns with our requirements for supporting a target language version at all, and both seem reasonable requirements since without automated testing we won't detect breakages, and expecting developers to test by hand seems like the burden falling in the wrong place.  If someone wants to keep support for an older version, they get to sort out CI coverage for that version (by using packages from a PPA or using a prebuilt version or running the job in a docker image or whatever).
Supporting the latest version of python is more important than supporting older versions. It's easy enough to put in version specific code though. I agree with @ojwb though, that ideally we test the versions we state we support. I'm still using 3.6 though in places, and I doubt I am not the only one.

> Both could be checked, or just `__spec__.parent` - I can verify that works back to 3.6, which is the oldest Python I had to check

I don't see any `__spec__.parent` until python 3.13, eg for for 3.12:
```
$ python3.12
Python 3.12.4 (main, Jun  8 2024, 18:29:57) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> dir(__spec__)
['__bool__', '__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
>>> 
```
No parent, but there is in 3.13:
```
$ python3.13
Python 3.13.0b3 (main, Jul  8 2024, 01:23:25) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> dir(__spec__)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__firstlineno__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__static_attributes__', '__str__', '__subclasshook__', '__weakref__', '_cached', '_set_fileattr', '_uninitialized_submodules', 'cached', 'has_location', 'loader', 'loader_state', 'name', 'origin', 'parent', 'submodule_search_locations']
>>> 
```
How did you determine it works all the way back to python-3.6?

Adding @mromberg, who has done most of the advanced package support, for Python for comment.
It has to be in a file, the interpreter doesn’t have a `__spec__` until 3.13 (and only the new Python-based REPL).
> Supporting the latest version of python is more important than supporting older versions. It's easy enough to put in version specific code though. I agree with @ojwb though, that ideally we test the versions we state we support. I'm still using 3.6 though in places, and I doubt I am not the only one.

We still have CI testing for 3.6 (and 3.5) so enforcing a requirement for a version to be testable in CI wouldn't require dropping 3.6 at this point anyway.  It's only 3.3 and 3.4 that we claim to support without having any CI coverage to back that up (so unless anyone has been regularly manually testing with these older versions, really all we can say is that we haven't knowingly done anything that stops SWIG working Python 3.3 and 3.4, which to me isn't very reassuring either as a user or as a maintainer).
Looks like 3.14 will likely be released in just over a year, so we should aim to fix this for SWIG 4.4.0.
Note: it's important to support this at least by RC 1, since that's when CPython is ABI stable and packages are supposed to start shipping. The ".0" release is when normal users are supposed to start using it, packages should already have been built for 3.14. It would be a very good idea to support it by beta 1, as that's when most packages start experimenting with support.

Alphas are produced starting in October or so, and are available on GHA for _early_ testing. Anything can change, though.
Beta phase starts in May. This is fairly stable (feature freeze), _small_ chance of ABI changes though.
RC starts in August.
Final is October.

I believe more people will be using 3.14.0a1 than 3.3 and 3.4 combined, if that's helpful. Yesterday, pip was downloaded 10 times for 3.3 and 495 times for 3.4. For comparison, for 3.14, pip was downloaded 260 times, and that's just people building CPython from source. For 3.13, it was downloaded 12,609 (0.12%) times, which is slightly higher than 3.5's 11,389 (0.11%) times. The most popular Python version, 3.11, downloaded pip 2,605,209 times (25.2%).

So removing 3.3 and 3.4 removes 0.00% of Python users (who download pip, since that's the package I chose for a metric).

Here's the [download numbers](https://pypistats.org/packages/swig) for the pip `swig` package, by the way, which was downloaded 0 times for 3.3, 3.4, and 3.5:

![Screenshot 2024-09-19 at 12 48 16 AM](https://github.com/user-attachments/assets/65cd1757-ebb6-47e1-afae-7e9a9e2df90b)

If you scroll back a week or two, you can find one download for 3.5, on 9-9-2024.


We need an easy way to install python-3.14 with the `__package__` removal implemented to test in order to take this forward. Any news on 3.14 being ready for testing this would be good to know here. A patch with a proposed fix wouldn't go amiss either!
@henryiii SWIG is a volunteer project - if you want something to happen (and especially if you want it to happen in a particular time scale) the best way to achieve that is to submit a clean patch.  If you're wanting to get a patch in 4.3.0, then you have about a week before it gets released (which is a timeline driven primarily by the Python 3.13 release date).

(Even adding it to the 4.4 milestone doesn't guarantee it'll get done for 4.4.0, but it does at least ensure we don't forget about the issue completely.)

I don't **think** there's much argument against hard dropping support for 3.3 and 3.4 at this point if keeping it is blocking 3.14 support, though I'm not 100% clear on @wsfulton's position on this.  If he's wanting to keep conditional version-specific code for 3.3 and 3.4 that may make the task significantly harder because we don't have CI testing to tell us such code actually works.
> 3.3 is _really_ old, by the way, and never was well supported - throughout 3.3's support window, 2.x was dramatically more popular. 3.5 was the first version with heavy use. Personally, 3.6 is still a _really_ old version of Python and a fine minimum; 3.6 is the oldest version you can build for via manylinux, for example. 3.8 is the oldest officially supported version of Python these days.

3.6 is still part of SUSE Linux Enterprise Server 15, and it will be part of it for a long time (unfortunately, I am the maintainer).
Oh, sorry, missed this. This would be a lot easier if something like 3.5+ or 3.6+ became the minimum supported version, which I assume you'd not want on a short timescale anyway. It seem like 4.4 or later would be fine, as long as it's in by the beta series of 3.14, which is next year, something like March IIRC.

> 3.6 is still part of SUSE Linux Enterprise Server 15

Do people expect the latest version of SWIG to work with 3.6, though? All the rest of the core libraries for Python require 3.7+ or 3.8+ (and 3.8 is now EoL, so soon it will be 3.9+). I think it's fine to expect to need old versions of software on old OSs. Though from what I remember above even 3.5+ would be just fine for this specific thing.

FYI, you can't get older versions of Python (<3.8) via setup-python in GHA on macos-latest or ubuntu-latest. Apple Silicon support started in 3.8 and they didn't bother to build EoL Pythons for ubuntu-24.04.
I wasn't submitting a patch since I didn't want to raise the min version of Python, that seem like a project consensus type of a thing. Should I make a patch that just doesn't support <3.5 for this, then?
Making the minimum supported version as 3.5 doesn't seem an unreasonable compromise in order to add support for 3.14. Please submit your patch to support 3.5 and later and I'll take a look.
On Wed Oct 30, 2024 at 7:30 PM CET, Henry Schreiner wrote:
>> 3.6 is still part of SUSE Linux Enterprise Server 15
>
> Do people expect the latest version of SWIG to work with 3.6, though? All the rest of the core libraries for Python require 3.7+ or 3.8+ (and 3.8 is now EoL, so soon it will be 3.9+). I think it’s fine to expect to need old versions of software on old OSs. Though from what I remember above even 3.5+ would be just fine for this specific thing.

I didn’t mean that. Certainly we discourage users of our enterprise systems to use recent versions of software packages we provide in the system, moreover using most recent swig on the ancient system is beyond recklessness. However, as I believe SLE’s Python 3.6 is probably one of the oldest supported Pythons in the production use (I am not sure RHEL, but I think they don’t carry anything that ancient any more), it could be just one milestone where to stop with increasing the version of Python you support.

Manylinux also still supports 3.6 for a bit longer also (and because of that, it's still pretty easy to test on 3.6).

Fix patch:
diff --git a/.github/workflows/linux.yml b/.github/workflows/linux.yml
index 02aa3f0681..a7e8f84d96 100644
--- a/.github/workflows/linux.yml
+++ b/.github/workflows/linux.yml
@@ -132,6 +132,9 @@ jobs:
         - SWIGLANG: python
           VER: '3.13t' # no-gil testing
           CSTD: gnu99
+        - SWIGLANG: python
+          VER: '3.14'
+          CSTD: gnu99
         - SWIGLANG: python
           PY2: 2
           SWIG_FEATURES: -builtin
diff --git a/Doc/Manual/Python.html b/Doc/Manual/Python.html
index 23587e5dbc..01fc449a68 100644
--- a/Doc/Manual/Python.html
+++ b/Doc/Manual/Python.html
@@ -6552,7 +6552,7 @@ <H4><a name="Python_package_search_both_package_modules">33.11.6.1 Both modules
 
 <p>
 In this configuration, the pure Python module, foo.py, tries to load the C/C++ module, _foo, from the same package foo.py is
-located in.  The package name is determined from the <tt>__package__</tt>
+located in.  The package name is determined from the <tt>__spec__.parent</tt> (or <tt>__package__</tt> before Python 3.4)
 attribute if available, see <a href="https://www.python.org/dev/peps/pep-0366/">PEP 366</a>. Otherwise it is derived from the <tt>__name__</tt>
 attribute given to foo.py by the Python loader that imported foo.py.
 The interface file for this configuration would contain:
@@ -6675,7 +6675,7 @@ <H4><a name="Python_custom_module_import">33.11.6.4 More on customizing the modu
 
 <div class="targetlang">
 <pre>
-if __package__ or '.' in __name__:
+if getattr(__spec__, "parent", None) or '.' in __name__:
     from . import _foo
 else:
     import _foo
@@ -6760,7 +6760,7 @@ <H4><a name="Python_custom_module_import">33.11.6.4 More on customizing the modu
 
 <div class="targetlang">
 <pre>
-if __package__ or '.' in __name__:
+if getattr(__spec__, "parent", None) or '.' in __name__:
     from ._foo import *
 else:
     from _foo import *
diff --git a/Source/Modules/python.cxx b/Source/Modules/python.cxx
index 86daf131c8..3070a94fac 100644
--- a/Source/Modules/python.cxx
+++ b/Source/Modules/python.cxx
@@ -703,20 +703,22 @@ class PYTHON:public Language {
 	 * onwards (implicit relative imports raised a DeprecationWarning in 2.6,
 	 * and fail in 2.7 onwards).
 	 *
-	 * First check for __package__ which is available from 2.6 onwards, see PEP366.
+	 * First check for __spec__.parent which is available from 3.4 onwards,
+	 * see https://docs.python.org/3/reference/import.html#spec. If not,
+	 * check for __package__, which was set before 3.14.
 	 * Next try determine the shadow wrapper's package based on the __name__ it
 	 * was given by the importer that loaded it.
 	 * If the module is in a package, load the low-level C/C++ module from the
 	 * same package, otherwise load it as a global module.
 	 */
         Printv(default_import_code, "# Import the low-level C/C++ module\n", NULL);
-        Printv(default_import_code, "if __package__ or \".\" in __name__:\n", NULL);
+        Printv(default_import_code, "if getattr(globals().get(\"__spec__\"), \"parent\", None) or globals().get(\"__package__\") or \".\" in __name__:\n", NULL);
         Printv(default_import_code, tab4, "from . import ", module, "\n", NULL);
         Printv(default_import_code, "else:\n", NULL);
         Printv(default_import_code, tab4, "import ", module, "\n", NULL);
       } else {
         Printv(default_import_code, "# Pull in all the attributes from the low-level C/C++ module\n", NULL);
-        Printv(default_import_code, "if __package__ or \".\" in __name__:\n", NULL);
+        Printv(default_import_code, "if getattr(globals().get(\"__spec__\"), \"parent\", None) or globals().get(\"__package__\") or \".\" in __name__:\n", NULL);
         Printv(default_import_code, tab4, "from .", module, " import *\n", NULL);
         Printv(default_import_code, "else:\n", NULL);
         Printv(default_import_code, tab4, "from ", module, " import *\n", NULL);
@@ -4372,6 +4374,11 @@ class PYTHON:public Language {
     Printv(f, "#if PY_VERSION_HEX >= 0x030b0000\n", NIL);
     printSlot(f, getSlot(n, "feature:python:_ht_tpname"), "_ht_tpname", "char *");
 
+    // void *ht_token;
+    Printv(f, "#if PY_VERSION_HEX >= 0x030e0000\n", NIL);
+    printSlot(f, getSlot(n, "feature:python:ht_token"), "ht_token", "void *");
+    Printv(f, "#endif\n", NIL);
+
     // struct _specialization_cache _spec_cache;
     Printf(f, "  {\n");
     printSlot(f, getSlot(n, "feature:python:getitem"), "getitem", "PyObject *");


Test patch:
diff --git a/Examples/test-suite/python/python_annotations_variable_c_runme.py b/Examples/test-suite/python/python_annotations_variable_c_runme.py
index 153852d05e..d1f359bbbd 100644
--- a/Examples/test-suite/python/python_annotations_variable_c_runme.py
+++ b/Examples/test-suite/python/python_annotations_variable_c_runme.py
@@ -1,4 +1,17 @@
 import sys
+import inspect
+
+
+def get_annotations(cls):
+    # Python >=3.14 removed the __annotations__ attribute
+    # retrieve it via inspect (see also annotationlib)
+    if hasattr(inspect, "get_annotations"):
+        # Python >=3.10
+        return inspect.get_annotations(cls)
+    else:
+        # Python <3.10
+        return getattr(cls, "__annotations__", {})
+
 
 # Variable annotations for properties is only supported in python-3.6 and later (PEP 526)
 if sys.version_info[0:2] >= (3, 6):
@@ -8,17 +21,14 @@
     annotations_supported = not(is_python_builtin() or is_python_fastproxy())
 
     if annotations_supported:
-        ts = TemplateShort()
-        anno = ts.__annotations__
+        anno = get_annotations(TemplateShort)
         if anno != {'member_variable': 'int'}:
             raise RuntimeError("annotations mismatch: {}".format(anno))
 
-        ts = StructWithVar()
-        anno = ts.__annotations__
+        anno = get_annotations(StructWithVar)
         if anno != {'member_variable': 'int'}:
             raise RuntimeError("annotations mismatch: {}".format(anno))
 
-        ts = StructWithVarNotAnnotated()
-        if getattr(ts, "__annotations__", None) != None:
-            anno = ts.__annotations__
+        anno = get_annotations(StructWithVarNotAnnotated)
+        if anno != {}:
             raise RuntimeError("annotations mismatch: {}".format(anno))