Upgrade free-threading CI to run with pytest-freethreaded instead of pytest-xdist

There is a new kid on the block that should help us find out whether scikit-learn and its dependencies can be reliably considered free-threading compatible:

https://pypi.org/project/pytest-freethreaded/

Let's try to adopt it in scikit-learn.

Here is a possible plan:

- first run the tests locally a few times and see if they are tests (or a set of interacting tests) that cause a crash or a failure, open an issue for each of them, possibly upstream and then mark them as skipped under free-threading builds with a reference to the issue in the "reason" field;
- then upgrade our nightly free-threading scheduled CI run to use `pytest-freethreaded`.

Any comments @lesteve @jeremiedbb @ngoldbaum?

EDIT: anyone interested in getting hands on the first item can find this resource useful:

https://py-free-threading.github.io/

EDIT 2: there is also the [pytest-run-parallel](https://github.com/Quansight-Labs/pytest-run-parallel) plugin that can serve a similar purpose.

EDIT3: here is a TODO/plan for this problem:



- [x] resync and simplify #30041 to leverage https://github.com/Quansight-Labs/pytest-run-parallel/pull/19, https://github.com/Quansight-Labs/pytest-run-parallel/pull/33 and https://github.com/Quansight-Labs/pytest-run-parallel/pull/34
- [x] configure the `[free-threading]` CI to run with `pytest-run-parallel` #32023
- [x] enable the free-threading flag in Cython extension #31342
  - [x] investigate and fix crashes and test failures or open tracking issues and mark tests as `thread_unsafe` ;)
- [ ] benchmark the use of threading, in particular when we we expect nested parallelism between Python threads (e.g. in `GridSearchCV` with the "threading" backend of joblib) and BLAS or OpenMP native threading in the underlying estimators.
- [ ] communicate results on a blog post / pydata presentation / social media.

COMMENT:
Here is a somewhat simple test file that reproduces the issue. Note this happens on vanilla CPython 3.13 not only free-threaded CPython 3.13.

As you can tell from the output, the fixture is run in the main thread and the test function in another thread (threadpool created by the pytest plugin).

```py
# test_simple.py
import threading

import pytest

thread_local = threading.local()


@pytest.fixture
def change_thread_local():
    print("inside fixture", threading.current_thread())
    setattr(thread_local, "value", 1)
    yield


def test_simple(change_thread_local):
    print("inside function", threading.current_thread())
    assert getattr(thread_local, "value", None) == 1
```

```
pytest --threads 1 --iterations 1 test_simple.py
```

Output:
```
============================================================================================================= test session starts ==============================================================================================================
platform linux -- Python 3.13.0rc3, pytest-8.3.3, pluggy-1.5.0
rootdir: /tmp/proj
plugins: xdist-3.6.1, freethreaded-0.1.0
collected 1 item                                                                                                                                                                                                                               

test_simple.py F                                                                                                                                                                                                                         [100%]

=================================================================================================================== FAILURES ===================================================================================================================
_________________________________________________________________________________________________________________ test_simple __________________________________________________________________________________________________________________

change_thread_local = None

    def test_simple(change_thread_local):
        print("inside function", threading.current_thread())
>       assert getattr(thread_local, "value", None) == 1
E       AssertionError: assert None == 1
E        +  where None = getattr(<_thread._local object at 0x7b3843ff32e0>, 'value', None)

test_simple.py:17: AssertionError
------------------------------------------------------------------------------------------------------------ Captured stdout setup -------------------------------------------------------------------------------------------------------------
inside fixture <_MainThread(MainThread, started 135481613739584)>
------------------------------------------------------------------------------------------------------------- Captured stdout call -------------------------------------------------------------------------------------------------------------
inside function <Thread(ThreadPoolExecutor-0_0, started 135481585043136)>
=========================================================================================================== short test summary info ============================================================================================================
FAILED test_simple.py::test_simple - AssertionError: assert None == 1
============================================================================================================== 1 failed in 0.03s ===============================================================================================================
```

Labels: Build / CI, Needs Decision, module:test-suite, free-threading