Issue 1:
Description

Passing any estimator to the init parameter in GradientBoostingClassifier (or GradientBoostingRegressor) causes errors when fitting. I'm including one example below as a MWE but I've fiddled with this issue for a while and get a number of different errors even if I can get past one. My guess is that this part of the code (the init estimator part) hasn't been shown any love in quite a while (since I've found similar issues that date back to 2014: #2691). A more complete exploration can be found here: https://github.com/stoddardg/sklearn_bug_example/blob/master/Gradient%2BBoosting%2Binit%2BMWE.ipynb

Steps/Code to Reproduce

from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegressionCV
from sklearn.ensemble import GradientBoostingClassifier

X, y = make_classification(n_samples=1000, n_features=50, n_informative=10,random_state=0)
init_clf = LogisticRegressionCV()
clf = GradientBoostingClassifier(init=init_clf)
clf.fit(X, y)

Actual Results

IndexError Traceback (most recent call last)
<ipython-input-5-f94c05274ee0> in <module>()
6 init_clf = LogisticRegressionCV()
7 clf = GradientBoostingClassifier(init=init_clf)
----> 8 clf.fit(train_X, train_y)

~/anaconda3/envs/eis_env/lib/python3.7/site-packages/sklearn/ensemble/gradient_boosting.py in fit(self, X, y, sample_weight, monitor)
1463 n_stages = self._fit_stages(X, y, y_pred, sample_weight, self._rng,
1464 X_val, y_val, sample_weight_val,
-> 1465 begin_at_stage, monitor, X_idx_sorted)
1466
1467 # change shape of arrays after fit (early-stopping or additional ests)

~/anaconda3/envs/eis_env/lib/python3.7/site-packages/sklearn/ensemble/gradient_boosting.py in _fit_stages(self, X, y, y_pred, sample_weight, random_state, X_val, y_val, sample_weight_val, begin_at_stage, monitor, X_idx_sorted)
1527 y_pred = self._fit_stage(i, X, y, y_pred, sample_weight,
1528 sample_mask, random_state, X_idx_sorted,
-> 1529 X_csc, X_csr)
1530
1531 # track deviance (= loss)

~/anaconda3/envs/eis_env/lib/python3.7/site-packages/sklearn/ensemble/gradient_boosting.py in _fit_stage(self, i, X, y, y_pred, sample_weight, sample_mask, random_state, X_idx_sorted, X_csc, X_csr)
1197 loss.update_terminal_regions(tree.tree_, X, y, residual, y_pred,
1198 sample_weight, sample_mask,
-> 1199 self.learning_rate, k=k)
1200
1201 # add tree to ensemble

~/anaconda3/envs/eis_env/lib/python3.7/site-packages/sklearn/ensemble/gradient_boosting.py in update_terminal_regions(self, tree, X, y, residual, y_pred, sample_weight, sample_mask, learning_rate, k)
396 self._update_terminal_region(tree, masked_terminal_regions,
397 leaf, X, y, residual,
--> 398 y_pred[:, k], sample_weight)
399
400 # update predictions (both in-bag and out-of-bag)

IndexError: too many indices for array

Versions

System
------
python: 3.7.0 (default, Jun 28 2018, 13:15:42) [GCC 7.2.0]
executable: /home/gstoddard/anaconda3/envs/eis_env/bin/python
machine: Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-redhat-7.5-Maipo

BLAS
----
macros: SCIPY_MKL_H=None, HAVE_CBLAS=None
lib_dirs: /home/gstoddard/anaconda3/envs/eis_env/lib
cblas_libs: mkl_rt, pthread

Python deps
-----------
pip: 18.0
setuptools: 39.2.0
sklearn: 0.20.0
numpy: 1.15.0
scipy: 1.1.0
Cython: None
pandas: 0.23.3

Issue 2:
here is a tiny script which reproduces the crash.

from sklearn.datasets import load_iris
from sklearn import ensemble
from sklearn.cross_validation import train_test_split

iris = load_iris()
X, y = iris.data, iris.target
X, y = X[y < 2], y[y < 2] # make it binary

X_train, X_test, y_train, y_test = train_test_split(X, y)

# Fit GBT init with RF
rf = ensemble.RandomForestClassifier()
clf = ensemble.GradientBoostingClassifier(init=rf)

clf.fit(X_train, y_train)
acc = clf.score(X_test, y_test)
print("Accuracy: {:.4f}".format(acc))

It also seems that the init param in GradientBoostingClassifier is

not really tested.

@pprett @glouppe @ogrisel