Upgrade free-threading CI to run with pytest-freethreaded instead of pytest-xdist

There is a new kid on the block that should help us find out whether scikit-learn and its dependencies can be reliably considered free-threading compatible:

https://pypi.org/project/pytest-freethreaded/

Let's try to adopt it in scikit-learn.

Here is a possible plan:

- first run the tests locally a few times and see if they are tests (or a set of interacting tests) that cause a crash or a failure, open an issue for each of them, possibly upstream and then mark them as skipped under free-threading builds with a reference to the issue in the "reason" field;
- then upgrade our nightly free-threading scheduled CI run to use `pytest-freethreaded`.

Any comments @lesteve @jeremiedbb @ngoldbaum?

EDIT: anyone interested in getting hands on the first item can find this resource useful:

https://py-free-threading.github.io/

EDIT 2: there is also the [pytest-run-parallel](https://github.com/Quansight-Labs/pytest-run-parallel) plugin that can serve a similar purpose.

EDIT3: here is a TODO/plan for this problem:



- [x] resync and simplify #30041 to leverage https://github.com/Quansight-Labs/pytest-run-parallel/pull/19, https://github.com/Quansight-Labs/pytest-run-parallel/pull/33 and https://github.com/Quansight-Labs/pytest-run-parallel/pull/34
- [x] configure the `[free-threading]` CI to run with `pytest-run-parallel` #32023
- [x] enable the free-threading flag in Cython extension #31342
  - [x] investigate and fix crashes and test failures or open tracking issues and mark tests as `thread_unsafe` ;)
- [ ] benchmark the use of threading, in particular when we we expect nested parallelism between Python threads (e.g. in `GridSearchCV` with the "threading" backend of joblib) and BLAS or OpenMP native threading in the underlying estimators.
- [ ] communicate results on a blog post / pydata presentation / social media.

COMMENT:
I tried @lesteve's reproducer with `pytest-run-parallel` and I get exactly the same problem:

```
$ pytest --parallel-threads 2 test_thread_local_fixture.py -v
======================================= test session starts ========================================
platform darwin -- Python 3.12.5, pytest-8.3.2, pluggy-1.5.0 -- /Users/ogrisel/miniforge3/envs/dev/bin/python3.12
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/private/tmp/testfolder/.hypothesis/examples'))
rootdir: /private/tmp/testfolder
plugins: repeat-0.9.2, hypothesis-6.112.2, anyio-4.4.0, run-parallel-0.1.0, xdist-3.6.1
collected 1 item                                                                                   

test_thread_local_fixture.py::test_simple FAILED                                             [100%]

============================================= FAILURES =============================================
___________________________________________ test_simple ____________________________________________

change_thread_local = None

    def test_simple(change_thread_local):
        print("inside function", threading.current_thread())
>       assert getattr(thread_local, "value", None) == 1
E       AssertionError: assert None == 1
E        +  where None = getattr(<_thread._local object at 0x103d6aac0>, 'value', None)

test_thread_local_fixture.py:17: AssertionError
-------------------------------------- Captured stdout setup ---------------------------------------
inside fixture <_MainThread(MainThread, started 8595033088)>
--------------------------------------- Captured stdout call ---------------------------------------
inside function <Thread(Thread-2 (closure), started 6199488512)>
inside function <Thread(Thread-1 (closure), started 6182662144)>
===================================== short test summary info ======================================
FAILED test_thread_local_fixture.py::test_simple - AssertionError: assert None == 1
======================================== 1 failed in 0.14s =========================================
```

The function-fixture is executed in a different thread (the main thread) than the thread used to execute the test function.

Labels: Build / CI, Needs Decision, module:test-suite, free-threading