Issue 1:
By default many model selection tools such as cross_validate, validation_curves and *SearchCV catch exceptions, raise a warning and score the model with nan. This can be a nice behavior, especially for *SearchCV estimators that can explore invalid hyper-parameter combinations.

However the warnings can be hidden when they are issued on the stderr of loky workers (python subprocesses), for instance in Jupyter interactive environments:

This is a big usability bug. I think we should refactor the model selection tools to raise the warning from the main process instead of the workers. This would make it possible to:

make it possible to display the warning on the main process stderr to avoid hiding those when parallel compution on several Python processes (e.g. with the default loky backend of joblib) or multi-machine (e.g. with the dask or ray cluster backends of joblib).

raise the warning only once (for the first case) to avoid a "wall of warnings"-effect when n_jobs=1

inform the user that can set error_score="raise" in the warning message if they want there code to raise an exception instead of gettting nan-valued scores.

This problem has been causing a lot of confusion to MOOC participants (INRIA/scikit-learn-mooc#377) so it's probably hurting scikit-learn usability significantly.

Issue 2:
Describe the bug

Hello, I am using a sklearn.tree.DecisionTreeRegressor for two tasks where one task is just a 'subsets' of the other and I expect to get the same tree, but I don't.

In the two tasks, the examples are identical, where task2 the feature matrix is duplicated horizontally, i.e. if the feature space in task1 is of dimension 9, then for task2 it is of dimension 18, where the first 9 features and the last 9 features are identical.

I am running :

stump = DecisionTreeRegressor(max_depth=1, random_state=42)

TASK 1 INPUT: 9 features

[[322. 0. 143. 265. 27. 0. 61. 32. 32.]
[116. 0. 50. 90. 12. 0. 22. 12. 12.]
[137. 0. 74. 125. 22. 0. 33. 12. 12.]
[ 97. 0. 46. 85. 10. 0. 16. 12. 12.]
[159. 0. 107. 156. 39. 0. 47. 12. 22.]
[268. 0. 145. 235. 39. 0. 70. 10. 28.]
[129. 0. 49. 80. 17. 0. 25. 6. 6.]
[122. 0. 42. 80. 8. 0. 20. 12. 12.]
[569. 0. 337. 520. 131. 0. 163. 0. 0.]
[ 76. 0. 30. 60. 4. 0. 13. 10. 10.]
[120. 0. 54. 100. 12. 0. 22. 12. 12.]
[245. 0. 74. 145. 10. 0. 36. 18. 21.]
[109. 0. 49. 90. 11. 0. 21. 10. 10.]
[126. 0. 80. 120. 28. 0. 37. 6. 14.]
[140. 0. 76. 120. 22. 0. 37. 6. 17.]]

TASK 2 INPUT: 18 features where the last 9 features are just duplications of the first 9

[[322. 0. 143. 265. 27. 0. 61. 32. 32. 322. 0. 143. 265. 27.
0. 61. 32. 32.]
[116. 0. 50. 90. 12. 0. 22. 12. 12. 116. 0. 50. 90. 12.
0. 22. 12. 12.]
[137. 0. 74. 125. 22. 0. 33. 12. 12. 137. 0. 74. 125. 22.
0. 33. 12. 12.]
[ 97. 0. 46. 85. 10. 0. 16. 12. 12. 97. 0. 46. 85. 10.
0. 16. 12. 12.]
[159. 0. 107. 156. 39. 0. 47. 12. 22. 159. 0. 107. 156. 39.
0. 47. 12. 22.]
[268. 0. 145. 235. 39. 0. 70. 10. 28. 268. 0. 145. 235. 39.
0. 70. 10. 28.]
[129. 0. 49. 80. 17. 0. 25. 6. 6. 129. 0. 49. 80. 17.
0. 25. 6. 6.]
[122. 0. 42. 80. 8. 0. 20. 12. 12. 122. 0. 42. 80. 8.
0. 20. 12. 12.]
[569. 0. 337. 520. 131. 0. 163. 0. 0. 569. 0. 337. 520. 131.
0. 163. 0. 0.]
[ 76. 0. 30. 60. 4. 0. 13. 10. 10. 76. 0. 30. 60. 4.
0. 13. 10. 10.]
[120. 0. 54. 100. 12. 0. 22. 12. 12. 120. 0. 54. 100. 12.
0. 22. 12. 12.]
[245. 0. 74. 145. 10. 0. 36. 18. 21. 245. 0. 74. 145. 10.
0. 36. 18. 21.]
[109. 0. 49. 90. 11. 0. 21. 10. 10. 109. 0. 49. 90. 11.
0. 21. 10. 10.]
[126. 0. 80. 120. 28. 0. 37. 6. 14. 126. 0. 80. 120. 28.
0. 37. 6. 14.]
[140. 0. 76. 120. 22. 0. 37. 6. 17. 140. 0. 76. 120. 22.
0. 37. 6. 17.]]

In the first task, the tree chooses feature number 2, with gain 1.1111111111111112, threshold 52.0.

The feature values across the examples in this case are:

[143. 50. 74. 46. 107. 145. 49. 42. 337. 30. 54. 74. 49. 80. 76.]

In the second task, the tree chooses feature number 12, with gain 1.1111111111111112, threshold 95.0.

The feature values across the examples in this case are:

[265. 90. 125. 85. 156. 235. 80. 80. 520. 60. 100. 145. 90. 120.
120.]

I understand that it is possible that there are multiple features that result in the same gain, but the feature number 12 that is selected in the second task, also exists in the first task as feature number 3, so I am trying to understand why it chooses different features for these two tasks, it doesn't make sense.

Is it possible to make this behavior deterministic (for example to make it choose the first feature that has max gain)

Thank you.

Steps/Code to Reproduce

data1 = np.array([[322., 0., 143., 265., 27., 0., 61., 32., 32.],
[116., 0., 50., 90., 12., 0., 22., 12., 12.],
[137., 0., 74., 125., 22., 0., 33., 12., 12.],
[ 97., 0., 46., 85., 10., 0., 16., 12., 12.],
[159., 0., 107., 156., 39., 0., 47., 12., 22.],
[268., 0., 145., 235., 39., 0., 70., 10., 28.],
[129., 0., 49., 80., 17., 0., 25., 6., 6.],
[122., 0., 42., 80., 8., 0., 20., 12., 12.],
[569., 0., 337., 520., 131., 0., 163., 0., 0.],
[ 76., 0., 30., 60., 4., 0., 13., 10., 10.],
[120., 0., 54., 100., 12., 0., 22., 12., 12.],
[245., 0., 74., 145., 10., 0., 36., 18., 21.],
[109., 0., 49., 90., 11., 0., 21., 10., 10.],
[126., 0., 80., 120., 28., 0., 37., 6., 14.],
[140., 0., 76., 120., 22., 0., 37., 6., 17.]])

labels = np.array([[0], [0], [1], [0], [0], [1], [0], [0], [1], [0], [1], [1], [0], [0], [0]])

stump1 = DecisionTreeRegressor(max_depth=1, random_state=42)
stump1.fit(data1, labels)
feature_index_1 = stump.tree_.feature[0]

data2 =np.array([[322., 0., 143., 265., 27., 0., 61., 32., 32., 322., 0.,
143., 265., 27., 0., 61., 32., 32.],
[116., 0., 50., 90., 12., 0., 22., 12., 12., 116., 0.,
50., 90., 12., 0., 22., 12., 12.],
[137., 0., 74., 125., 22., 0., 33., 12., 12., 137., 0.,
74., 125., 22., 0., 33., 12., 12.],
[ 97., 0., 46., 85., 10., 0., 16., 12., 12., 97., 0.,
46., 85., 10., 0., 16., 12., 12.],
[159., 0., 107., 156., 39., 0., 47., 12., 22., 159., 0.,
107., 156., 39., 0., 47., 12., 22.],
[268., 0., 145., 235., 39., 0., 70., 10., 28., 268., 0.,
145., 235., 39., 0., 70., 10., 28.],
[129., 0., 49., 80., 17., 0., 25., 6., 6., 129., 0.,
49., 80., 17., 0., 25., 6., 6.],
[122., 0., 42., 80., 8., 0., 20., 12., 12., 122., 0.,
42., 80., 8., 0., 20., 12., 12.],
[569., 0., 337., 520., 131., 0., 163., 0., 0., 569., 0.,
337., 520., 131., 0., 163., 0., 0.],
[ 76., 0., 30., 60., 4., 0., 13., 10., 10., 76., 0.,
30., 60., 4., 0., 13., 10., 10.],
[120., 0., 54., 100., 12., 0., 22., 12., 12., 120., 0.,
54., 100., 12., 0., 22., 12., 12.],
[245., 0., 74., 145., 10., 0., 36., 18., 21., 245., 0.,
74., 145., 10., 0., 36., 18., 21.],
[109., 0., 49., 90., 11., 0., 21., 10., 10., 109., 0.,
49., 90., 11., 0., 21., 10., 10.],
[126., 0., 80., 120., 28., 0., 37., 6., 14., 126., 0.,
80., 120., 28., 0., 37., 6., 14.],
[140., 0., 76., 120., 22., 0., 37., 6., 17., 140., 0.,
76., 120., 22., 0., 37., 6., 17.]])

labels = np.array([[0], [0], [1], [0], [0], [1], [0], [0], [1], [0], [1], [1], [0], [0], [0]])

stump2 = DecisionTreeRegressor(max_depth=1, random_state=42)
stump2.fit(data2, labels)
feature_index_2 = stump.tree_.feature[0]

Expected Results

I expect the model to choose the same feature over these two tasks. i.e.

feature_index_2 == feature_index_1

Actual Results

The model chooses different features:

feature_index_1 = 2

feature_index_2 = 12

Versions

Python dependencies:
sklearn: 1.1.1
pip: 22.1.2
setuptools: 63.4.1
numpy: 1.21.5
scipy: 1.7.3
Cython: None
pandas: 1.4.1
matplotlib: 3.5.1
joblib: 1.1.0
threadpoolctl: 2.2.0