Repository: pybind/pybind11 (language: python)

Issue:
[BUG]: retranslating local exception with custom data class hangs in free-threading cpython
### Required prerequisites

- [X] Make sure you've read the [documentation](https://pybind11.readthedocs.io). Your issue may be addressed there.
- [X] Search the [issue tracker](https://github.com/pybind/pybind11/issues) and [Discussions](https:/pybind/pybind11/discussions) to verify that this hasn't already been reported. +1 or comment there if it has.
- [X] Consider asking first in the [Gitter chat room](https://gitter.im/pybind/Lobby) or in a [Discussion](https:/pybind/pybind11/discussions/new).

### What version (or hash if on master) of pybind11 are you using?

2.13.5 and from source

### Problem description

I have a program stuck when retranslating local exception with custom data class using CPython with free-threading mode enabled.

This is due to `internals.mutex` being in the locked state when the callback is running. When using custom data, `all_type_info_get_cache` method is called internally which also tries to lock `internals.mutex`.

See the repro code below

### Reproducible example code

C++ test1.cpp

```c++
// https://pybind11.readthedocs.io/en/stable/basics.html
// clang++ -O3 -Wall -shared -std=c++11 -fPIC $(python -m pybind11 --includes) test1.cpp -o test1.so
// python -c "import test1; test1.check()"

#include <string>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>


namespace py = pybind11;

struct CustomData {
  CustomData(const std::string & a): a(a) {}
  std::string a;
};


struct CustomError {
  CustomError(const CustomData & message) : message(message) {}
  CustomData message;
};


PYBIND11_MODULE(test1, m, py::mod_gil_not_used()) {
  m.doc() = "test custom exception with free-threading";

  m.def("check", [](){
    auto d1 = CustomData("abc");
    throw CustomError(d1);
  });

  py::class_<CustomData>(m, "CustomData", py::module_local())
        .def(py::init<const std::string &>())
        .def_readwrite("a", &CustomData::a);
  py::register_local_exception_translator([](std::exception_ptr p) {
    try {
      if (p)
        std::rethrow_exception(p);
    } catch (const CustomError &e) {

      printf("Handle CustomError exception\n");
      auto mod = py::module_::import("exceptions1");
      py::object obj = mod.attr("CustomError");
      printf("Before the exception creation\n");

      // Here we can check that internals.mutex is locked: internals.mutex.mutex._bits == 1
      // obj(e.message) calls `all_type_info_get_cache` which would try to lock again internals

      // // If we unlock internals.mutex then obj(e.message) will work otherwise
      // // execution hangs.
      // auto &internals = py::detail::get_internals();
      // internals.mutex.unlock();

      py::object obj2 = obj(e.message);

      // We should lock again if we unlocked it
      // internals.mutex.lock();

      printf("After the exception creation\n");
      PyErr_SetObject(PyExc_Exception, obj2.ptr());

    }
  });
}
```

Python code: `exceptions1.py`
```python
class CustomError(Exception):
    def __init__(self, message):
        self.message = message
        super().__init__(message)

    def __str__(self):
        s = "[python]: " + self.message.a
        return s
```

Run:
```bash
python -c "import test1; test1.check()"
```
Output:
```bash
Handle CustomError exception
Before the exception creation
```


### Is this a regression? Put the last known working version here if it is.

Not a regression


Discussion / hints:
I think the easiest approach might be to use a different mutex for the exception translators than for the rest of the internals. Other than initialization and finalization, I think the accesses for exception translators and the rest of internals are separate

The alternative is to unlock before calling exception translators, but to do that safely I think you may need to copy the list of translators.
Thanks @colesbury let me check these approaches!

I tried another alternative doing the following, checking whether the mutex is already locked and lock it only if unlocked:
```
// Wrapper around PyMutex to provide BasicLockable semantics
class pymutex {
    PyMutex mutex;

public:
    pymutex() : mutex({}) {}
    void lock() { PyMutex_Lock(&mutex); }
    void unlock() { PyMutex_Unlock(&mutex); }
    bool has_lock() { return mutex._bits == 1; }
};


#ifdef Py_GIL_DISABLED
#    define PYBIND11_LOCK_INTERNALS(internals) std::unique_lock<pymutex> lock((internals).mutex, std::defer_lock); if ((internals).mutex.has_lock()) { lock.lock(); }
#else
#    define PYBIND11_LOCK_INTERNALS(internals)
#endif
``` 
What do you think? 
That looks like it would skip locking the mutex if it's held by another thread, which would not be safe. But also, your sample code only locks the mutex if it's already locked, so I don't think it will ever get locked.

Fix patch:
diff --git a/.github/workflows/emscripten.yaml b/.github/workflows/emscripten.yaml
index fe4cd821c6..18a1ad464a 100644
--- a/.github/workflows/emscripten.yaml
+++ b/.github/workflows/emscripten.yaml
@@ -23,7 +23,7 @@ jobs:
         submodules: true
         fetch-depth: 0
 
-    - uses: pypa/cibuildwheel@v2.20
+    - uses: pypa/cibuildwheel@v2.21
       env:
         PYODIDE_BUILD_EXPORTS: whole_archive
       with:
diff --git a/.github/workflows/pip.yml b/.github/workflows/pip.yml
index f241a83a73..77ecfb10bb 100644
--- a/.github/workflows/pip.yml
+++ b/.github/workflows/pip.yml
@@ -92,11 +92,12 @@ jobs:
     runs-on: ubuntu-latest
     if: github.event_name == 'release' && github.event.action == 'published'
     needs: [packaging]
-    environment: pypi
+    environment:
+      name: pypi
+      url: https://pypi.org/p/pybind11
     permissions:
       id-token: write
       attestations: write
-      contents: read
 
     steps:
     # Downloads all to directories matching the artifact names
@@ -111,8 +112,10 @@ jobs:
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
         packages-dir: standard/
+        attestations: true
 
     - name: Publish global package
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
         packages-dir: global/
+        attestations: true
diff --git a/docs/changelog.rst b/docs/changelog.rst
index 0145317743..a91082113f 100644
--- a/docs/changelog.rst
+++ b/docs/changelog.rst
@@ -31,6 +31,37 @@ New Features:
 * The ``array_caster`` in pybind11/stl.h was enhanced to support value types that are not default-constructible.
   `#5305 <https://github.com/pybind/pybind11/pull/5305>`_
 
+* Added ``py::warnings`` namespace with ``py::warnings::warn`` and ``py::warnings::new_warning_type`` that provides the interface for Python warnings.
+  `#5291 <https://github.com/pybind/pybind11/pull/5291>`_
+
+Version 2.13.6 (September 13, 2024)
+-----------------------------------
+
+New Features:
+
+* A new ``self._pybind11_conduit_v1_()`` method is automatically added to all
+  ``py::class_``-wrapped types, to enable type-safe interoperability between
+  different independent Python/C++ bindings systems, including pybind11
+  versions with different ``PYBIND11_INTERNALS_VERSION``'s. Supported on
+  pybind11 2.11.2, 2.12.1, and 2.13.6+.
+  `#5296 <https://github.com/pybind/pybind11/pull/5296>`_
+
+
+Bug fixes:
+
+* Using ``__cpp_nontype_template_args`` instead of ``__cpp_nontype_template_parameter_class``.
+  `#5330 <https://github.com/pybind/pybind11/pull/5330>`_
+
+* Properly translate C++ exception to Python exception when creating Python buffer from wrapped object.
+  `#5324 <https://github.com/pybind/pybind11/pull/5324>`_
+
+
+Documentation:
+
+* Adds an answer (FAQ) for "What is a highly conclusive and simple way to find memory leaks?".
+  `#5340 <https://github.com/pybind/pybind11/pull/5340>`_
+
+
 Version 2.13.5 (August 22, 2024)
 --------------------------------
 
@@ -238,6 +269,18 @@ Other:
 * Update docs and noxfile.
   `#5071 <https://github.com/pybind/pybind11/pull/5071>`_
 
+Version 2.12.1 (September 13, 2024)
+-----------------------------------
+
+New Features:
+
+* A new ``self._pybind11_conduit_v1_()`` method is automatically added to all
+  ``py::class_``-wrapped types, to enable type-safe interoperability between
+  different independent Python/C++ bindings systems, including pybind11
+  versions with different ``PYBIND11_INTERNALS_VERSION``'s. Supported on
+  pybind11 2.11.2, 2.12.1, and 2.13.6+.
+  `#5296 <https://github.com/pybind/pybind11/pull/5296>`_
+
 
 Version 2.12.0 (March 27, 2024)
 -------------------------------
@@ -413,6 +456,18 @@ Other:
 * An ``assert()`` was added to help Coverty avoid generating a false positive.
   `#4817 <https://github.com/pybind/pybind11/pull/4817>`_
 
+Version 2.11.2 (September 13, 2024)
+-----------------------------------
+
+New Features:
+
+* A new ``self._pybind11_conduit_v1_()`` method is automatically added to all
+  ``py::class_``-wrapped types, to enable type-safe interoperability between
+  different independent Python/C++ bindings systems, including pybind11
+  versions with different ``PYBIND11_INTERNALS_VERSION``'s. Supported on
+  pybind11 2.11.2, 2.12.1, and 2.13.6+.
+  `#5296 <https://github.com/pybind/pybind11/pull/5296>`_
+
 
 Version 2.11.1 (July 17, 2023)
 ------------------------------
diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h
index 9cc2dc684e..fdeac33f3e 100644
--- a/include/pybind11/cast.h
+++ b/include/pybind11/cast.h
@@ -1853,9 +1853,9 @@ class argument_loader {
     using indices = make_index_sequence<sizeof...(Args)>;
 
     template <typename Arg>
-    using argument_is_args = std::is_same<intrinsic_t<Arg>, args>;
+    using argument_is_args = std::is_base_of<args, intrinsic_t<Arg>>;
     template <typename Arg>
-    using argument_is_kwargs = std::is_same<intrinsic_t<Arg>, kwargs>;
+    using argument_is_kwargs = std::is_base_of<kwargs, intrinsic_t<Arg>>;
     // Get kwargs argument position, or -1 if not present:
     static constexpr auto kwargs_pos = constexpr_last<argument_is_kwargs, Args...>();
 
diff --git a/include/pybind11/detail/exception_translation.h b/include/pybind11/detail/exception_translation.h
index 2764180bb0..22ae8a1c94 100644
--- a/include/pybind11/detail/exception_translation.h
+++ b/include/pybind11/detail/exception_translation.h
@@ -50,17 +50,17 @@ inline void try_translate_exceptions() {
         - delegate translation to the next translator by throwing a new type of exception.
         */
 
-    bool handled = with_internals([&](internals &internals) {
-        auto &local_exception_translators = get_local_internals().registered_exception_translators;
-        if (detail::apply_exception_translators(local_exception_translators)) {
-            return true;
-        }
-        auto &exception_translators = internals.registered_exception_translators;
-        if (detail::apply_exception_translators(exception_translators)) {
-            return true;
-        }
-        return false;
-    });
+    bool handled = with_exception_translators(
+        [&](std::forward_list<ExceptionTranslator> &exception_translators,
+            std::forward_list<ExceptionTranslator> &local_exception_translators) {
+            if (detail::apply_exception_translators(local_exception_translators)) {
+                return true;
+            }
+            if (detail::apply_exception_translators(exception_translators)) {
+                return true;
+            }
+            return false;
+        });
 
     if (!handled) {
         set_error(PyExc_SystemError, "Exception escaped from default exception translator!");
diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h
index 72ed84ea39..8499c87cff 100644
--- a/include/pybind11/detail/internals.h
+++ b/include/pybind11/detail/internals.h
@@ -41,7 +41,11 @@
 #    elif PY_VERSION_HEX >= 0x030C0000 || defined(_MSC_VER)
 // Version bump for Python 3.12+, before first 3.12 beta release.
 // Version bump for MSVC piggy-backed on PR #4779. See comments there.
-#        define PYBIND11_INTERNALS_VERSION 5
+#        ifdef Py_GIL_DISABLED
+#            define PYBIND11_INTERNALS_VERSION 6
+#        else
+#            define PYBIND11_INTERNALS_VERSION 5
+#        endif
 #    else
 #        define PYBIND11_INTERNALS_VERSION 4
 #    endif
@@ -179,6 +183,7 @@ static_assert(sizeof(instance_map_shard) % 64 == 0,
 struct internals {
 #ifdef Py_GIL_DISABLED
     pymutex mutex;
+    pymutex exception_translator_mutex;
 #endif
     // std::type_index -> pybind11's type information
     type_map<type_info *> registered_types_cpp;
@@ -669,6 +674,19 @@ inline auto with_internals(const F &cb) -> decltype(cb(get_internals())) {
     return cb(internals);
 }
 
+template <typename F>
+inline auto with_exception_translators(const F &cb)
+    -> decltype(cb(get_internals().registered_exception_translators,
+                   get_local_internals().registered_exception_translators)) {
+    auto &internals = get_internals();
+#ifdef Py_GIL_DISABLED
+    std::unique_lock<pymutex> lock((internals).exception_translator_mutex);
+#endif
+    auto &local_internals = get_local_internals();
+    return cb(internals.registered_exception_translators,
+              local_internals.registered_exception_translators);
+}
+
 inline std::uint64_t mix64(std::uint64_t z) {
     // David Stafford's variant 13 of the MurmurHash3 finalizer popularized
     // by the SplitMix PRNG.
diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h
index 795894250c..dabf89e8a3 100644
--- a/include/pybind11/pybind11.h
+++ b/include/pybind11/pybind11.h
@@ -1387,7 +1387,17 @@ class generic_type : public object {
             } else {
                 internals.registered_types_cpp[tindex] = tinfo;
             }
+
+            PYBIND11_WARNING_PUSH
+#if defined(__GNUC__) && __GNUC__ == 12
+            // When using GCC 12 these warnings are disabled as they trigger
+            // false positive warnings.  Discussed here:
+            // https://gcc.gnu.org/bugzilla/show_bug.cgi?id=115824.
+            PYBIND11_WARNING_DISABLE_GCC("-Warray-bounds")
+            PYBIND11_WARNING_DISABLE_GCC("-Wstringop-overread")
+#endif
             internals.registered_types_py[(PyTypeObject *) m_ptr] = {tinfo};
+            PYBIND11_WARNING_POP
         });
 
         if (rec.bases.size() > 1 || rec.multiple_inheritance) {
@@ -2915,10 +2925,12 @@ void implicitly_convertible() {
 }
 
 inline void register_exception_translator(ExceptionTranslator &&translator) {
-    detail::with_internals([&](detail::internals &internals) {
-        internals.registered_exception_translators.push_front(
-            std::forward<ExceptionTranslator>(translator));
-    });
+    detail::with_exception_translators(
+        [&](std::forward_list<ExceptionTranslator> &exception_translators,
+            std::forward_list<ExceptionTranslator> &local_exception_translators) {
+            (void) local_exception_translators;
+            exception_translators.push_front(std::forward<ExceptionTranslator>(translator));
+        });
 }
 
 /**
@@ -2928,11 +2940,12 @@ inline void register_exception_translator(ExceptionTranslator &&translator) {
  * the exception.
  */
 inline void register_local_exception_translator(ExceptionTranslator &&translator) {
-    detail::with_internals([&](detail::internals &internals) {
-        (void) internals;
-        detail::get_local_internals().registered_exception_translators.push_front(
-            std::forward<ExceptionTranslator>(translator));
-    });
+    detail::with_exception_translators(
+        [&](std::forward_list<ExceptionTranslator> &exception_translators,
+            std::forward_list<ExceptionTranslator> &local_exception_translators) {
+            (void) exception_translators;
+            local_exception_translators.push_front(std::forward<ExceptionTranslator>(translator));
+        });
 }
 
 /**
diff --git a/tools/pybind11NewTools.cmake b/tools/pybind11NewTools.cmake
index d3e938fab0..8bac211e14 100644
--- a/tools/pybind11NewTools.cmake
+++ b/tools/pybind11NewTools.cmake
@@ -274,10 +274,6 @@ function(pybind11_add_module target_name)
     target_link_libraries(${target_name} PRIVATE pybind11::embed)
   endif()
 
-  if(MSVC)
-    target_link_libraries(${target_name} PRIVATE pybind11::windows_extras)
-  endif()
-
   # -fvisibility=hidden is required to allow multiple modules compiled against
   # different pybind versions to work properly, and for some features (e.g.
   # py::module_local).  We force it on everything inside the `pybind11`


Test patch:
diff --git a/tests/custom_exceptions.py b/tests/custom_exceptions.py
new file mode 100644
index 0000000000..b1a092d761
--- /dev/null
+++ b/tests/custom_exceptions.py
@@ -0,0 +1,10 @@
+from __future__ import annotations
+
+
+class PythonMyException7(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(message)
+
+    def __str__(self):
+        return "[PythonMyException7]: " + self.message.a
diff --git a/tests/test_exceptions.cpp b/tests/test_exceptions.cpp
index c1d05bb241..0a970065b4 100644
--- a/tests/test_exceptions.cpp
+++ b/tests/test_exceptions.cpp
@@ -111,6 +111,16 @@ struct PythonAlreadySetInDestructor {
     py::str s;
 };
 
+struct CustomData {
+    explicit CustomData(const std::string &a) : a(a) {}
+    std::string a;
+};
+
+struct MyException7 {
+    explicit MyException7(const CustomData &message) : message(message) {}
+    CustomData message;
+};
+
 TEST_SUBMODULE(exceptions, m) {
     m.def("throw_std_exception",
           []() { throw std::runtime_error("This exception was intentionally thrown."); });
@@ -385,4 +395,33 @@ TEST_SUBMODULE(exceptions, m) {
 
     // m.def("pass_exception_void", [](const py::exception<void>&) {}); // Does not compile.
     m.def("return_exception_void", []() { return py::exception<void>(); });
+
+    m.def("throws7", []() {
+        auto data = CustomData("abc");
+        throw MyException7(data);
+    });
+
+    py::class_<CustomData>(m, "CustomData", py::module_local())
+        .def(py::init<const std::string &>())
+        .def_readwrite("a", &CustomData::a);
+
+    PYBIND11_CONSTINIT static py::gil_safe_call_once_and_store<py::object>
+        PythonMyException7_storage;
+    PythonMyException7_storage.call_once_and_store_result([&]() {
+        auto mod = py::module_::import("custom_exceptions");
+        py::object obj = mod.attr("PythonMyException7");
+        return obj;
+    });
+
+    py::register_local_exception_translator([](std::exception_ptr p) {
+        try {
+            if (p) {
+                std::rethrow_exception(p);
+            }
+        } catch (const MyException7 &e) {
+            auto exc_type = PythonMyException7_storage.get_stored();
+            py::object exc_inst = exc_type(e.message);
+            PyErr_SetObject(PyExc_Exception, exc_inst.ptr());
+        }
+    });
 }
diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py
index ba5063a749..a8fd105ea0 100644
--- a/tests/test_exceptions.py
+++ b/tests/test_exceptions.py
@@ -3,6 +3,7 @@
 import sys
 
 import pytest
+from custom_exceptions import PythonMyException7
 
 import env
 import pybind11_cross_module_tests as cm
@@ -195,6 +196,10 @@ def test_custom(msg):
             raise RuntimeError("Exception error: caught child from parent") from err
     assert msg(excinfo.value) == "this is a helper-defined translated exception"
 
+    with pytest.raises(PythonMyException7) as excinfo:
+        m.throws7()
+    assert msg(excinfo.value) == "[PythonMyException7]: abc"
+
 
 def test_nested_throws(capture):
     """Tests nested (e.g. C++ -> Python -> C++) exception handling"""
diff --git a/tests/test_kwargs_and_defaults.cpp b/tests/test_kwargs_and_defaults.cpp
index bc76ec7c2d..09036ccd51 100644
--- a/tests/test_kwargs_and_defaults.cpp
+++ b/tests/test_kwargs_and_defaults.cpp
@@ -14,6 +14,26 @@
 
 #include <utility>
 
+// Classes needed for subclass test.
+class ArgsSubclass : public py::args {
+    using py::args::args;
+};
+class KWArgsSubclass : public py::kwargs {
+    using py::kwargs::kwargs;
+};
+namespace pybind11 {
+namespace detail {
+template <>
+struct handle_type_name<ArgsSubclass> {
+    static constexpr auto name = const_name("*Args");
+};
+template <>
+struct handle_type_name<KWArgsSubclass> {
+    static constexpr auto name = const_name("**KWArgs");
+};
+} // namespace detail
+} // namespace pybind11
+
 TEST_SUBMODULE(kwargs_and_defaults, m) {
     auto kw_func
         = [](int x, int y) { return "x=" + std::to_string(x) + ", y=" + std::to_string(y); };
@@ -322,4 +342,10 @@ TEST_SUBMODULE(kwargs_and_defaults, m) {
             py::pos_only{},
             py::arg("i"),
             py::arg("j"));
+
+    // Test support for args and kwargs subclasses
+    m.def("args_kwargs_subclass_function",
+          [](const ArgsSubclass &args, const KWArgsSubclass &kwargs) {
+              return py::make_tuple(args, kwargs);
+          });
 }
diff --git a/tests/test_kwargs_and_defaults.py b/tests/test_kwargs_and_defaults.py
index b9b1a7ea87..e3f758165c 100644
--- a/tests/test_kwargs_and_defaults.py
+++ b/tests/test_kwargs_and_defaults.py
@@ -17,6 +17,10 @@ def test_function_signatures(doc):
     assert (
         doc(m.args_kwargs_function) == "args_kwargs_function(*args, **kwargs) -> tuple"
     )
+    assert (
+        doc(m.args_kwargs_subclass_function)
+        == "args_kwargs_subclass_function(*Args, **KWArgs) -> tuple"
+    )
     assert (
         doc(m.KWClass.foo0)
         == "foo0(self: m.kwargs_and_defaults.KWClass, arg0: int, arg1: float) -> None"
@@ -98,6 +102,7 @@ def test_arg_and_kwargs():
     args = "a1", "a2"
     kwargs = {"arg3": "a3", "arg4": 4}
     assert m.args_kwargs_function(*args, **kwargs) == (args, kwargs)
+    assert m.args_kwargs_subclass_function(*args, **kwargs) == (args, kwargs)
 
 
 def test_mixed_args_and_kwargs(msg):
@@ -413,6 +418,12 @@ def test_args_refcount():
     )
     assert refcount(myval) == expected
 
+    assert m.args_kwargs_subclass_function(7, 8, myval, a=1, b=myval) == (
+        (7, 8, myval),
+        {"a": 1, "b": myval},
+    )
+    assert refcount(myval) == expected
+
     exp3 = refcount(myval, myval, myval)
     assert m.args_refcount(myval, myval, myval) == (exp3, exp3, exp3)
     assert refcount(myval) == expected