Upgrade free-threading CI to run with pytest-freethreaded instead of pytest-xdist

There is a new kid on the block that should help us find out whether scikit-learn and its dependencies can be reliably considered free-threading compatible:

https://pypi.org/project/pytest-freethreaded/

Let's try to adopt it in scikit-learn.

Here is a possible plan:

- first run the tests locally a few times and see if they are tests (or a set of interacting tests) that cause a crash or a failure, open an issue for each of them, possibly upstream and then mark them as skipped under free-threading builds with a reference to the issue in the "reason" field;
- then upgrade our nightly free-threading scheduled CI run to use `pytest-freethreaded`.

Any comments @lesteve @jeremiedbb @ngoldbaum?

EDIT: anyone interested in getting hands on the first item can find this resource useful:

https://py-free-threading.github.io/

EDIT 2: there is also the [pytest-run-parallel](https://github.com/Quansight-Labs/pytest-run-parallel) plugin that can serve a similar purpose.

EDIT3: here is a TODO/plan for this problem:



- [x] resync and simplify #30041 to leverage https://github.com/Quansight-Labs/pytest-run-parallel/pull/19, https://github.com/Quansight-Labs/pytest-run-parallel/pull/33 and https://github.com/Quansight-Labs/pytest-run-parallel/pull/34
- [x] configure the `[free-threading]` CI to run with `pytest-run-parallel` #32023
- [x] enable the free-threading flag in Cython extension #31342
  - [x] investigate and fix crashes and test failures or open tracking issues and mark tests as `thread_unsafe` ;)
- [ ] benchmark the use of threading, in particular when we we expect nested parallelism between Python threads (e.g. in `GridSearchCV` with the "threading" backend of joblib) and BLAS or OpenMP native threading in the underlying estimators.
- [ ] communicate results on a blog post / pydata presentation / social media.

COMMENT:
I think the issue at hand is that a separate threaded fixture is required here, which should be different from a global, pytest fixture. However, from a concurrency/distribution/parallelism point of view, each thread should get its own local atomic state, which should be independent from each other. As pytest fixtures are spawned globally from the main thread, the case exposed by @lesteve is the expected behaviour rather than a bug.

In order to fix this, there could be 2 different avenues:
1.  Extend the `@pytest.fixture` decorator to create a new `@threaded_fixture` that allows for the creation of per-thread fixtures. Challenges here could include interfacing with global pytest fixtures.
2. Explicitly initialize each thread state in the test function, this ensures that best practices regarding explicit local state when doing concurrent computing are followed. Right now, on https://github.com/scipy/scipy/pull/21496, many of the issues have been related to tests that were written using global state, which were useful when the GIL serialized code, but they obscure the execution workflow when brought into a concurrent stage. 

I support going for the second option rather than the first one, since it invites developers to think about concurrency, parallel and concurrent constructs explicitly, as opposed to having to rely on a separate machinery to (apparently) do the job without thinking about what is going on underneath. This is of uttermost importance if free-threaded is going to be the future of Python at some point.

Labels: Build / CI, Needs Decision, module:test-suite, free-threading