Repository: pybind/pybind11 (language: python)

Issue:
[BUG]: retranslating local exception with custom data class hangs in free-threading cpython
### Required prerequisites

- [X] Make sure you've read the [documentation](https://pybind11.readthedocs.io). Your issue may be addressed there.
- [X] Search the [issue tracker](https://github.com/pybind/pybind11/issues) and [Discussions](https:/pybind/pybind11/discussions) to verify that this hasn't already been reported. +1 or comment there if it has.
- [X] Consider asking first in the [Gitter chat room](https://gitter.im/pybind/Lobby) or in a [Discussion](https:/pybind/pybind11/discussions/new).

### What version (or hash if on master) of pybind11 are you using?

2.13.5 and from source

### Problem description

I have a program stuck when retranslating local exception with custom data class using CPython with free-threading mode enabled.

This is due to `internals.mutex` being in the locked state when the callback is running. When using custom data, `all_type_info_get_cache` method is called internally which also tries to lock `internals.mutex`.

See the repro code below

### Reproducible example code

C++ test1.cpp

```c++
// https://pybind11.readthedocs.io/en/stable/basics.html
// clang++ -O3 -Wall -shared -std=c++11 -fPIC $(python -m pybind11 --includes) test1.cpp -o test1.so
// python -c "import test1; test1.check()"

#include <string>
#include <pybind11/pybind11.h>
#include <pybind11/stl.h>


namespace py = pybind11;

struct CustomData {
  CustomData(const std::string & a): a(a) {}
  std::string a;
};


struct CustomError {
  CustomError(const CustomData & message) : message(message) {}
  CustomData message;
};


PYBIND11_MODULE(test1, m, py::mod_gil_not_used()) {
  m.doc() = "test custom exception with free-threading";

  m.def("check", [](){
    auto d1 = CustomData("abc");
    throw CustomError(d1);
  });

  py::class_<CustomData>(m, "CustomData", py::module_local())
        .def(py::init<const std::string &>())
        .def_readwrite("a", &CustomData::a);
  py::register_local_exception_translator([](std::exception_ptr p) {
    try {
      if (p)
        std::rethrow_exception(p);
    } catch (const CustomError &e) {

      printf("Handle CustomError exception\n");
      auto mod = py::module_::import("exceptions1");
      py::object obj = mod.attr("CustomError");
      printf("Before the exception creation\n");

      // Here we can check that internals.mutex is locked: internals.mutex.mutex._bits == 1
      // obj(e.message) calls `all_type_info_get_cache` which would try to lock again internals

      // // If we unlock internals.mutex then obj(e.message) will work otherwise
      // // execution hangs.
      // auto &internals = py::detail::get_internals();
      // internals.mutex.unlock();

      py::object obj2 = obj(e.message);

      // We should lock again if we unlocked it
      // internals.mutex.lock();

      printf("After the exception creation\n");
      PyErr_SetObject(PyExc_Exception, obj2.ptr());

    }
  });
}
```

Python code: `exceptions1.py`
```python
class CustomError(Exception):
    def __init__(self, message):
        self.message = message
        super().__init__(message)

    def __str__(self):
        s = "[python]: " + self.message.a
        return s
```

Run:
```bash
python -c "import test1; test1.check()"
```
Output:
```bash
Handle CustomError exception
Before the exception creation
```


### Is this a regression? Put the last known working version here if it is.

Not a regression


Discussion / hints:
I think the easiest approach might be to use a different mutex for the exception translators than for the rest of the internals. Other than initialization and finalization, I think the accesses for exception translators and the rest of internals are separate

The alternative is to unlock before calling exception translators, but to do that safely I think you may need to copy the list of translators.
Thanks @colesbury let me check these approaches!

I tried another alternative doing the following, checking whether the mutex is already locked and lock it only if unlocked:
```
// Wrapper around PyMutex to provide BasicLockable semantics
class pymutex {
    PyMutex mutex;

public:
    pymutex() : mutex({}) {}
    void lock() { PyMutex_Lock(&mutex); }
    void unlock() { PyMutex_Unlock(&mutex); }
    bool has_lock() { return mutex._bits == 1; }
};


#ifdef Py_GIL_DISABLED
#    define PYBIND11_LOCK_INTERNALS(internals) std::unique_lock<pymutex> lock((internals).mutex, std::defer_lock); if ((internals).mutex.has_lock()) { lock.lock(); }
#else
#    define PYBIND11_LOCK_INTERNALS(internals)
#endif
``` 
What do you think? 
That looks like it would skip locking the mutex if it's held by another thread, which would not be safe. But also, your sample code only locks the mutex if it's already locked, so I don't think it will ever get locked.

Fix patch:
diff --git a/include/pybind11/detail/exception_translation.h b/include/pybind11/detail/exception_translation.h
index 2764180bb0..22ae8a1c94 100644
--- a/include/pybind11/detail/exception_translation.h
+++ b/include/pybind11/detail/exception_translation.h
@@ -50,17 +50,17 @@ inline void try_translate_exceptions() {
         - delegate translation to the next translator by throwing a new type of exception.
         */
 
-    bool handled = with_internals([&](internals &internals) {
-        auto &local_exception_translators = get_local_internals().registered_exception_translators;
-        if (detail::apply_exception_translators(local_exception_translators)) {
-            return true;
-        }
-        auto &exception_translators = internals.registered_exception_translators;
-        if (detail::apply_exception_translators(exception_translators)) {
-            return true;
-        }
-        return false;
-    });
+    bool handled = with_exception_translators(
+        [&](std::forward_list<ExceptionTranslator> &exception_translators,
+            std::forward_list<ExceptionTranslator> &local_exception_translators) {
+            if (detail::apply_exception_translators(local_exception_translators)) {
+                return true;
+            }
+            if (detail::apply_exception_translators(exception_translators)) {
+                return true;
+            }
+            return false;
+        });
 
     if (!handled) {
         set_error(PyExc_SystemError, "Exception escaped from default exception translator!");
diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h
index 7cc6f53adf..fa224e0326 100644
--- a/include/pybind11/detail/internals.h
+++ b/include/pybind11/detail/internals.h
@@ -39,7 +39,11 @@
 #    if PY_VERSION_HEX >= 0x030C0000 || defined(_MSC_VER)
 // Version bump for Python 3.12+, before first 3.12 beta release.
 // Version bump for MSVC piggy-backed on PR #4779. See comments there.
-#        define PYBIND11_INTERNALS_VERSION 5
+#        ifdef Py_GIL_DISABLED
+#            define PYBIND11_INTERNALS_VERSION 6
+#        else
+#            define PYBIND11_INTERNALS_VERSION 5
+#        endif
 #    else
 #        define PYBIND11_INTERNALS_VERSION 4
 #    endif
@@ -177,6 +181,7 @@ static_assert(sizeof(instance_map_shard) % 64 == 0,
 struct internals {
 #ifdef Py_GIL_DISABLED
     pymutex mutex;
+    pymutex exception_translator_mutex;
 #endif
     // std::type_index -> pybind11's type information
     type_map<type_info *> registered_types_cpp;
@@ -643,6 +648,19 @@ inline auto with_internals(const F &cb) -> decltype(cb(get_internals())) {
     return cb(internals);
 }
 
+template <typename F>
+inline auto with_exception_translators(const F &cb)
+    -> decltype(cb(get_internals().registered_exception_translators,
+                   get_local_internals().registered_exception_translators)) {
+    auto &internals = get_internals();
+#ifdef Py_GIL_DISABLED
+    std::unique_lock<pymutex> lock((internals).exception_translator_mutex);
+#endif
+    auto &local_internals = get_local_internals();
+    return cb(internals.registered_exception_translators,
+              local_internals.registered_exception_translators);
+}
+
 inline std::uint64_t mix64(std::uint64_t z) {
     // David Stafford's variant 13 of the MurmurHash3 finalizer popularized
     // by the SplitMix PRNG.
diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h
index c52df969e6..bcdf641f43 100644
--- a/include/pybind11/pybind11.h
+++ b/include/pybind11/pybind11.h
@@ -2586,10 +2586,12 @@ void implicitly_convertible() {
 }
 
 inline void register_exception_translator(ExceptionTranslator &&translator) {
-    detail::with_internals([&](detail::internals &internals) {
-        internals.registered_exception_translators.push_front(
-            std::forward<ExceptionTranslator>(translator));
-    });
+    detail::with_exception_translators(
+        [&](std::forward_list<ExceptionTranslator> &exception_translators,
+            std::forward_list<ExceptionTranslator> &local_exception_translators) {
+            (void) local_exception_translators;
+            exception_translators.push_front(std::forward<ExceptionTranslator>(translator));
+        });
 }
 
 /**
@@ -2599,11 +2601,12 @@ inline void register_exception_translator(ExceptionTranslator &&translator) {
  * the exception.
  */
 inline void register_local_exception_translator(ExceptionTranslator &&translator) {
-    detail::with_internals([&](detail::internals &internals) {
-        (void) internals;
-        detail::get_local_internals().registered_exception_translators.push_front(
-            std::forward<ExceptionTranslator>(translator));
-    });
+    detail::with_exception_translators(
+        [&](std::forward_list<ExceptionTranslator> &exception_translators,
+            std::forward_list<ExceptionTranslator> &local_exception_translators) {
+            (void) exception_translators;
+            local_exception_translators.push_front(std::forward<ExceptionTranslator>(translator));
+        });
 }
 
 /**


Test patch:
diff --git a/tests/custom_exceptions.py b/tests/custom_exceptions.py
new file mode 100644
index 0000000000..b1a092d761
--- /dev/null
+++ b/tests/custom_exceptions.py
@@ -0,0 +1,10 @@
+from __future__ import annotations
+
+
+class PythonMyException7(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(message)
+
+    def __str__(self):
+        return "[PythonMyException7]: " + self.message.a
diff --git a/tests/test_exceptions.cpp b/tests/test_exceptions.cpp
index c1d05bb241..0a970065b4 100644
--- a/tests/test_exceptions.cpp
+++ b/tests/test_exceptions.cpp
@@ -111,6 +111,16 @@ struct PythonAlreadySetInDestructor {
     py::str s;
 };
 
+struct CustomData {
+    explicit CustomData(const std::string &a) : a(a) {}
+    std::string a;
+};
+
+struct MyException7 {
+    explicit MyException7(const CustomData &message) : message(message) {}
+    CustomData message;
+};
+
 TEST_SUBMODULE(exceptions, m) {
     m.def("throw_std_exception",
           []() { throw std::runtime_error("This exception was intentionally thrown."); });
@@ -385,4 +395,33 @@ TEST_SUBMODULE(exceptions, m) {
 
     // m.def("pass_exception_void", [](const py::exception<void>&) {}); // Does not compile.
     m.def("return_exception_void", []() { return py::exception<void>(); });
+
+    m.def("throws7", []() {
+        auto data = CustomData("abc");
+        throw MyException7(data);
+    });
+
+    py::class_<CustomData>(m, "CustomData", py::module_local())
+        .def(py::init<const std::string &>())
+        .def_readwrite("a", &CustomData::a);
+
+    PYBIND11_CONSTINIT static py::gil_safe_call_once_and_store<py::object>
+        PythonMyException7_storage;
+    PythonMyException7_storage.call_once_and_store_result([&]() {
+        auto mod = py::module_::import("custom_exceptions");
+        py::object obj = mod.attr("PythonMyException7");
+        return obj;
+    });
+
+    py::register_local_exception_translator([](std::exception_ptr p) {
+        try {
+            if (p) {
+                std::rethrow_exception(p);
+            }
+        } catch (const MyException7 &e) {
+            auto exc_type = PythonMyException7_storage.get_stored();
+            py::object exc_inst = exc_type(e.message);
+            PyErr_SetObject(PyExc_Exception, exc_inst.ptr());
+        }
+    });
 }
diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py
index ba5063a749..a8fd105ea0 100644
--- a/tests/test_exceptions.py
+++ b/tests/test_exceptions.py
@@ -3,6 +3,7 @@
 import sys
 
 import pytest
+from custom_exceptions import PythonMyException7
 
 import env
 import pybind11_cross_module_tests as cm
@@ -195,6 +196,10 @@ def test_custom(msg):
             raise RuntimeError("Exception error: caught child from parent") from err
     assert msg(excinfo.value) == "this is a helper-defined translated exception"
 
+    with pytest.raises(PythonMyException7) as excinfo:
+        m.throws7()
+    assert msg(excinfo.value) == "[PythonMyException7]: abc"
+
 
 def test_nested_throws(capture):
     """Tests nested (e.g. C++ -> Python -> C++) exception handling"""