unit test failures on x64 darwin and scipy 1.12

### Describe the bug

after upgrading from scipy 1.11.4 -> 1.12 x64 darwin build fails with multiple unit test failures.
https://hydra.nixos.org/build/247540976
[raw log](https://cache.nixos.org/log/d449kn87qr1kcca138gd9pl14qg1nnxg-python3.11-scikit-learn-1.4.0.drv)

```console
=========================== short test summary info ============================
FAILED cluster/tests/test_mean_shift.py::test_parallel[float64] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED decomposition/tests/test_dict_learning.py::test_sparse_encode_shapes_omp - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED decomposition/tests/test_dict_learning.py::test_dict_learning_reconstruction_parallel - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED decomposition/tests/test_dict_learning.py::test_dict_learning_lassocd_readonly_data - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED decomposition/tests/test_dict_learning.py::test_sparse_coder_parallel_mmap - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED decomposition/tests/test_dict_learning.py::test_cd_work_on_joblib_memmapped_data - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED ensemble/tests/test_bagging.py::test_parallel_classification - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED ensemble/tests/test_bagging.py::test_parallel_regression - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED ensemble/tests/test_bagging.py::test_estimator - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED linear_model/tests/test_sgd.py::test_multi_core_gridsearch_and_early_stopping - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED ensemble/tests/test_forest.py::test_backend_respected - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED linear_model/tests/test_theil_sen.py::test_theil_sen_parallel - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED manifold/tests/test_mds.py::test_MDS - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED metrics/tests/test_classification.py::test_classification_metric_division_by_zero_nan_validaton[scoring0] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED metrics/tests/test_classification.py::test_classification_metric_division_by_zero_nan_validaton[scoring1] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED metrics/tests/test_classification.py::test_classification_metric_division_by_zero_nan_validaton[scoring2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED metrics/tests/test_classification.py::test_classification_metric_division_by_zero_nan_validaton[scoring3] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED ensemble/tests/test_voting.py::test_parallel_fit[42] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED metrics/tests/test_pairwise.py::test_sparse_manhattan_readonly_dataset[csr_matrix] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED metrics/tests/test_pairwise.py::test_sparse_manhattan_readonly_dataset[csr_array] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED feature_selection/tests/test_rfe.py::test_rfe_cv_n_jobs[42] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs[kernel0] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs[kernel1] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED model_selection/tests/test_search.py::test_random_search_bad_cv - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs[kernel2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED gaussian_process/tests/test_gpc.py::test_multi_class_n_jobs[kernel3] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED model_selection/tests/test_validation.py::test_cross_validate_many_jobs - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED model_selection/tests/test_validation.py::test_cross_validate_return_indices[42] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED neighbors/tests/test_kd_tree.py::test_kdtree_picklable_with_joblib[KDTree64] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED neighbors/tests/test_kd_tree.py::test_kdtree_picklable_with_joblib[KDTree32] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_permutation_importance_correlated_feature_regression[0.5-2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_permutation_importance_correlated_feature_regression[1.0-2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_robustness_to_high_cardinality_noisy_feature[0.5-2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_robustness_to_high_cardinality_noisy_feature[1.0-2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_permutation_importance_equivalence_sequential_parallel[500] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_permutation_importance_equivalence_sequential_parallel[1.0] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED inspection/tests/test_permutation_importance.py::test_permutation_importance_large_memmaped_data[array] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED neighbors/tests/test_neighbors.py::test_knn_forcing_backend[ball_tree-loky] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED neighbors/tests/test_neighbors.py::test_knn_forcing_backend[kd_tree-loky] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED neighbors/tests/test_neighbors.py::test_knn_forcing_backend[auto-loky] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_calibration.py::test_parallel_execution[True-sigmoid] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_calibration.py::test_parallel_execution[True-isotonic] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_calibration.py::test_parallel_execution[False-sigmoid] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_calibration.py::test_parallel_execution[False-isotonic] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED linear_model/tests/test_coordinate_descent.py::test_linear_models_cv_fit_with_loky[ElasticNetCV] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED linear_model/tests/test_coordinate_descent.py::test_linear_models_cv_fit_with_loky[LassoCV] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_config.py::test_config_threadsafe_joblib[loky] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_multioutput.py::test_multi_output_classification_partial_fit_parallelism - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED tests/test_pipeline.py::test_feature_union_parallel - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
FAILED utils/tests/test_parallel.py::test_configuration_passes_through_to_joblib[loky-2] - joblib.externals.loky.process_executor.BrokenProcessPool: A task has failed...
```

### Steps/Code to Reproduce

run unit tests on x64 darwin with scipy 1.12

### Expected Results

unit tests pass

### Actual Results

https://cache.nixos.org/log/d449kn87qr1kcca138gd9pl14qg1nnxg-python3.11-scikit-learn-1.4.0.drv

### Versions

```shell
$ python -c 'import sklearn; sklearn.show_versions()'
System:
    python: 3.11.7 (main, Dec  4 2023, 18:10:11) [Clang 16.0.6 ]
executable: /nix/store/7l7g8ypn5aa6n3nvqids669rlzq9yi93-python3-3.11.7-env/bin/python3.11
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.4.0
          pip: None
   setuptools: None
        numpy: 1.26.2
        scipy: 1.12.0
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.3.2
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /nix/store/qws8gsrmyb3fks94qm4n63rgg97xa3v8-openmp-16.0.6/lib/libomp.dylib
        version: None
    num_threads: 8

       user_api: blas
   internal_api: openblas
         prefix: libopenblas
       filepath: /nix/store/mxlhcj24il7ncsmy89hv5h1vjgy5k1nr-openblas-0.3.25/lib/libopenblasp-r0.3.25.dylib
        version: 0.3.25
threading_layer: pthreads
   architecture: Prescott
    num_threads: 8
```

COMMENT:
I tried to reproduce the error on a OSX VM, but so far no luck (I kind of tried to be close to the no-OpenMP CI build where we observed the same kind of errors and then tried some variations around it).

My version information in the latest variation:

```
System:
    python: 3.11.7 (main, Dec 15 2023, 12:09:04) [Clang 14.0.6 ]
executable: /Users/ci/mambaforge/envs/scikit-learn-dev-from-env/bin/python
   machine: macOS-10.15.4-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.5.dev0
          pip: 23.3.1
   setuptools: 68.2.2
        numpy: 1.26.3
        scipy: 1.12.0
       Cython: 3.0.8
       pandas: 2.1.4
   matplotlib: 3.8.0
       joblib: 1.2.0
threadpoolctl: 2.2.0

Built with OpenMP: False

threadpoolctl info:
       filepath: /Users/ci/mambaforge/envs/scikit-learn-dev-from-env/lib/libmkl_rt.2.dylib
         prefix: libmkl_rt
       user_api: blas
   internal_api: mkl
        version: 2023.1-Product
    num_threads: 2
threading_layer: intel

       filepath: /Users/ci/mambaforge/envs/scikit-learn-dev-from-env/lib/libiomp5.dylib
         prefix: libiomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 2
```

Labels: Bug, Needs Reproducible Code