Upgrade free-threading CI to run with pytest-freethreaded instead of pytest-xdist

There is a new kid on the block that should help us find out whether scikit-learn and its dependencies can be reliably considered free-threading compatible:

https://pypi.org/project/pytest-freethreaded/

Let's try to adopt it in scikit-learn.

Here is a possible plan:

- first run the tests locally a few times and see if they are tests (or a set of interacting tests) that cause a crash or a failure, open an issue for each of them, possibly upstream and then mark them as skipped under free-threading builds with a reference to the issue in the "reason" field;
- then upgrade our nightly free-threading scheduled CI run to use `pytest-freethreaded`.

Any comments @lesteve @jeremiedbb @ngoldbaum?

EDIT: anyone interested in getting hands on the first item can find this resource useful:

https://py-free-threading.github.io/

EDIT 2: there is also the [pytest-run-parallel](https://github.com/Quansight-Labs/pytest-run-parallel) plugin that can serve a similar purpose.

EDIT3: here is a TODO/plan for this problem:



- [x] resync and simplify #30041 to leverage https://github.com/Quansight-Labs/pytest-run-parallel/pull/19, https://github.com/Quansight-Labs/pytest-run-parallel/pull/33 and https://github.com/Quansight-Labs/pytest-run-parallel/pull/34
- [x] configure the `[free-threading]` CI to run with `pytest-run-parallel` #32023
- [x] enable the free-threading flag in Cython extension #31342
  - [x] investigate and fix crashes and test failures or open tracking issues and mark tests as `thread_unsafe` ;)
- [ ] benchmark the use of threading, in particular when we we expect nested parallelism between Python threads (e.g. in `GridSearchCV` with the "threading" backend of joblib) and BLAS or OpenMP native threading in the underlying estimators.
- [ ] communicate results on a blog post / pydata presentation / social media.

COMMENT:
> For temporary directories, I think you should either migrate to a function that doesn't use temporary directories with a specific name, or prepend the name of the directory with e.g. a UUID to make the directory name unique.

But the point of the `tmpdir` fixture is to not have to deal with test isolation within the code of the test logic itself. Having to update the test code to make more verbose and complex would be a serious step back. It would be much better if to have a thread-local initialization of the function-scoped `tmpdir` fixture.

Granted, the `tmpdir` fixture has been deprecated in favor of the [`tmp_path` fixture](https://docs.pytest.org/en/latest/how-to/tmp_path.html#the-tmp-path-fixture) (to leverage the `pathlib.Path` API instead of a custom API). But the problem stays the same with the `tmp_path` fixture:

```python
import pytest


@pytest.mark.parametrize("value", list("abcd"))
def test_simple(tmp_path, value):
    file_path = tmp_path / "file.txt"
    assert not file_path.exists()
    file_path.write_text(value)
    read_value = file_path.read_text()
    assert read_value == value
```

```
$ pytest --parallel-threads=4 -x
=================================== test session starts ===================================
platform darwin -- Python 3.12.5, pytest-8.3.2, pluggy-1.5.0
rootdir: /private/tmp/testfolder
plugins: repeat-0.9.2, hypothesis-6.112.2, anyio-4.4.0, run-parallel-0.1.0, xdist-3.6.1
collected 4 items                                                                         

test_tmp_path.py F

======================================== FAILURES =========================================
_____________________________________ test_simple[a] ______________________________________

tmp_path = PosixPath('/private/var/folders/_y/lfnx34p13w3_sr2k12bjb05w0000gn/T/pytest-of-ogrisel/pytest-9/test_simple_a_0')
value = 'a'

    @pytest.mark.parametrize("value", list("abcd"))
    def test_simple(tmp_path, value):
        file_path = tmp_path / "file.txt"
>       assert not file_path.exists()
E       AssertionError: assert not True
E        +  where True = exists()
E        +    where exists = PosixPath('/private/var/folders/_y/lfnx34p13w3_sr2k12bjb05w0000gn/T/pytest-of-ogrisel/pytest-9/test_simple_a_0/file.txt').exists

test_tmp_path.py:7: AssertionError
================================= short test summary info =================================
FAILED test_tmp_path.py::test_simple[a] - AssertionError: assert not True
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================== 1 failed in 0.16s ====================================
```

Labels: Build / CI, Needs Decision, module:test-suite, free-threading