Output redirection is broken

### Describe the bug

I'm trying to redirect output from a script that uses Scikit-Learn to a file, as follows:

`python myscript.py > mylog.log 2>&1`

This should redirect both stdout and stderr to _mylog.log_. Most of my code uses logging statements which, as expected, are written to stderr and appear in mylog.log. Here's an example:

`logging.warning(f'Writing dataframe to Excel file {excel_file}')`

This works fine.

What I'm trying to do is use `GridSearchCV` to tune hyperparameters for my model and (no surprise) this is taking a while. I've therefore turned verbosity up 4, as follows:

```
def tune_hyperparameters(training_features: np.ndarray, training_classes: np.ndarray, test_features: np.ndarray, test_classes: np.ndarray):
    parameters = {
        'n_estimators': [10, 50, 100, 200, 500],
        'max_depth': [None, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
        'min_samples_split': [2, 3, 4, 5, 6, 7, 8, 9, 10],
        'min_samples_leaf': [1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
        'max_features': ['auto', 'sqrt', 'log2', None],
        'bootstrap': [True, False],
        'criterion': ['gini', 'entropy']
    }

    estimator = ensemble.RandomForestClassifier(random_state=0)

    # NOTE THE verbose=4 HERE \/ \/ \/
    grid_search = model_selection.GridSearchCV(estimator, parameters, cv=5, n_jobs=-1, verbose=4)
    grid_search.fit(training_features, training_classes)
    best_parameters = grid_search.best_params_

    estimator = ensemble.RandomForestClassifier(**best_parameters, random_state=0)
    estimator.fit(training_features, training_classes)
    score = estimator.score(test_features, test_classes)

    return best_parameters, score
```

When output isn't redirected I get the expected output from GridSearchCV in the console as follows:

```
[CV 2/5] END bootstrap=True, criterion=gini, max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=2, n_estimators=200;, score=0.712 total time=  24.5s
[CV 5/5] END bootstrap=True, criterion=gini, max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=2, n_estimators=200;, score=0.678 total time=  25.3s
[CV 1/5] END bootstrap=True, criterion=gini, max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=2, n_estimators=200;, score=0.747 total time=  25.3s
[CV 4/5] END bootstrap=True, criterion=gini, max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=2, n_estimators=200;, score=0.828 total time=  25.9s
[CV 3/5] END bootstrap=True, criterion=gini, max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=2, n_estimators=200;, score=0.675 total time=  26.2s
[CV 2/5] END bootstrap=True, criterion=gini, max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=3, n_estimators=200;, score=0.714 total time=  27.9s
[CV 1/5] END bootstrap=True, criterion=gini, max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=3, n_estimators=200;, score=0.749 total time=  28.5s
[CV 3/5] END bootstrap=True, criterion=gini, max_depth=None, max_features=auto, min_samples_leaf=1, min_samples_split=3, n_estimators=200;, score=0.673 total time=  29.0s
```

But when output is redirected using `python myscript.py > mylog.log 2>&1` the above output from GridSearchCV does not appear in the log file as it should. It also doesn't appear in the console any more, so it's clearly being redirected somewhere, just not where it should be.

I've seen a number of solutions proposed on StackOverflow that involve adding Python code to write output for a file but these all seem like kludges to me: the standard output redirect mechanism built into the shell should work and it seems like misbehaviour on the part of the library that it doesn't. I also searched the issues here but didn't find anything related, which seems slightly odd given that several people _have_ reported it via StackOverflow.

I'm running on Mac OS using the terminal embedded in VS Code 1.76.0, though I've verified the same behaviour in Mac OS's terminal app using bash, and conda 23.1.0.

One final point: I also wondered if there might be some other config option that I need to tweak in order to get GridSearchCV to output as expected, but haven't found anything so far.

Could somebody advise as to what the issue is here, please? Any help gratefully received.

### Steps/Code to Reproduce

```
python myscript.py > mylog.log 2>&1
```

### Expected Results

Expect that output from GridSearchCV, etc., is written to log file.

### Actual Results

Please see my detailed description above but, basically, the output _isn't_ written to the log file as expected, and it also disappears from the console.

### Versions

```shell
>>> import sklearn; sklearn.show_versions()
/Users/bart/opt/anaconda3/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")

System:
    python: 3.9.13 (main, Aug 25 2022, 18:29:29)  [Clang 12.0.0 ]
executable: /Users/bart/opt/anaconda3/bin/python
   machine: macOS-10.15.7-x86_64-i386-64bit

Python dependencies:
          pip: 22.2.2
   setuptools: 63.4.1
      sklearn: 1.0.2
        numpy: 1.21.5
        scipy: 1.9.1
       Cython: 0.29.32
       pandas: 1.5.3
   matplotlib: 3.5.2
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True
```

COMMENT:
I cannot reproduce:

```python
from sklearn import svm, datasets
from sklearn.model_selection import GridSearchCV

iris = datasets.load_iris()
parameters = {"kernel": ("linear", "rbf"), "C": [1, 10]}
svc = svm.SVC()
clf = GridSearchCV(svc, parameters, verbose=4, n_jobs=-1)
clf.fit(iris.data, iris.target)
```

```
python myscript.py > mylog.log 2>&1
```

I get the following file:

```
Fitting 5 folds for each of 4 candidates, totalling 20 fits
[CV 1/5] END ................C=1, kernel=linear;, score=0.967 total time=   0.0s
[CV 4/5] END ...................C=1, kernel=rbf;, score=0.933 total time=   0.0s
[CV 5/5] END ...................C=1, kernel=rbf;, score=1.000 total time=   0.0s
[CV 1/5] END ...............C=10, kernel=linear;, score=1.000 total time=   0.0s
[CV 2/5] END ...............C=10, kernel=linear;, score=1.000 total time=   0.0s
[CV 3/5] END ...............C=10, kernel=linear;, score=0.900 total time=   0.0s
[CV 4/5] END ...............C=10, kernel=linear;, score=0.967 total time=   0.0s
[CV 5/5] END ...............C=10, kernel=linear;, score=1.000 total time=   0.0s
[CV 1/5] END ..................C=10, kernel=rbf;, score=0.967 total time=   0.0s
[CV 2/5] END ..................C=10, kernel=rbf;, score=1.000 total time=   0.0s
[CV 3/5] END ..................C=10, kernel=rbf;, score=0.967 total time=   0.0s
[CV 4/5] END ..................C=10, kernel=rbf;, score=0.967 total time=   0.0s
[CV 5/5] END ..................C=10, kernel=rbf;, score=1.000 total time=   0.0s
[CV 2/5] END ...................C=1, kernel=rbf;, score=0.967 total time=   0.0s
[CV 3/5] END ................C=1, kernel=linear;, score=0.967 total time=   0.0s
[CV 5/5] END ................C=1, kernel=linear;, score=1.000 total time=   0.0s
[CV 4/5] END ................C=1, kernel=linear;, score=0.967 total time=   0.0s
[CV 2/5] END ................C=1, kernel=linear;, score=1.000 total time=   0.0s
[CV 3/5] END ...................C=1, kernel=rbf;, score=0.967 total time=   0.0s
[CV 1/5] END ...................C=1, kernel=rbf;, score=0.967 total time=   0.0s
```

So I don't think this is a scikit-learn issue. We are only using `print()` statement.

Labels: Bug, Needs Triage