CalibratedClassifierCV gives a NotFittedError when accessing the underlying XGBoostClassifier feature_importances property

### Describe the bug

I am using CalibratedClassifierCV and XGBoost in a Pipeline and was able to train the model and use it to make predictions, etc. But I cannot access the underlying property of the XGBoost model. 


### Steps/Code to Reproduce

```
from sklearn.pipeline import Pipeline
import xgboost as xgb
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import make_classification

X, y = make_classification(n_features=2, n_redundant=0, n_informative=2, random_state=1, n_clusters_per_class=1)

xgb_model = xgb.XGBClassifier(n_jobs = -1,
                              objective = "binary:logistic", 
                              eval_metric = 'auc')

calibrated_xgb = CalibratedClassifierCV(base_estimator=xgb_model, cv=5, method='isotonic')

calibrated_xgb_pipeline = Pipeline(steps=[('model', calibrated_xgb)])

calibrated_xgb_pipeline.fit(X, y)

calibrated_xgb_pipeline.named_steps["model"].base_estimator.feature_importances_
```

### Expected Results

Feature importance should be returned

### Actual Results

```
---------------------------------------------------------------------------
NotFittedError                            Traceback (most recent call last)
Cell In[206], line 21
     17 calibrated_xgb_pipeline = Pipeline(steps=[('model', calibrated_xgb)])
     19 calibrated_xgb_pipeline.fit(X, y)
---> 21 calibrated_xgb_pipeline.named_steps["model"].base_estimator.feature_importances_

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/xgboost/sklearn.py:1278, in XGBModel.feature_importances_(self)
   1263 @property
   1264 def feature_importances_(self) -> np.ndarray:
   1265     """Feature importances property, return depends on `importance_type`
   1266     parameter. When model trained with multi-class/multi-label/multi-target dataset,
   1267     the feature importance is "averaged" over all targets. The "average" is defined
   (...)
   1276 
   1277     """
-> 1278     b: Booster = self.get_booster()
   1280     def dft() -> str:
   1281         return "weight" if self.booster == "gblinear" else "gain"

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/xgboost/sklearn.py:649, in XGBModel.get_booster(self)
    646 if not self.__sklearn_is_fitted__():
    647     from sklearn.exceptions import NotFittedError
--> 649     raise NotFittedError("need to call fit or load_model beforehand")
    650 return self._Booster

NotFittedError: need to call fit or load_model beforehand
```

### Versions

```shell
I am using sklearn 1.3.2 and xgboost 1.7.6.
```

COMMENT:
I have further tested this issue with sklearn's RandomForestClassifier and received the same error.
My code:
```
from sklearn.pipeline import Pipeline
import xgboost as xgb
from sklearn.ensemble import RandomForestClassifier
from sklearn.calibration import CalibratedClassifierCV
from sklearn.datasets import make_classification

X, y = make_classification(n_features=2, n_redundant=0, n_informative=2, random_state=1, n_clusters_per_class=1)

model = RandomForestClassifier()

calibrated_model = CalibratedClassifierCV(base_estimator=model, cv=5, method='isotonic')

calibrated_model_pipeline = Pipeline(steps=[('model', calibrated_model)])

calibrated_model_pipeline.fit(X, y)

calibrated_model_pipeline.named_steps["model"].base_estimator.feature_importances_
```
error:
```
---------------------------------------------------------------------------
NotFittedError                            Traceback (most recent call last)
Cell In[211], line 17
     13 calibrated_model_pipeline = Pipeline(steps=[('model', calibrated_model)])
     15 calibrated_model_pipeline.fit(X, y)
---> 17 calibrated_model_pipeline.named_steps["model"].base_estimator.feature_importances_

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sklearn/ensemble/_forest.py:625, in BaseForest.feature_importances_(self)
    604 @property
    605 def feature_importances_(self):
    606     """
    607     The impurity-based feature importances.
    608 
   (...)
    623         array of zeros.
    624     """
--> 625     check_is_fitted(self)
    627     all_importances = Parallel(n_jobs=self.n_jobs, prefer="threads")(
    628         delayed(getattr)(tree, "feature_importances_")
    629         for tree in self.estimators_
    630         if tree.tree_.node_count > 1
    631     )
    633     if not all_importances:

File ~/anaconda3/envs/python3/lib/python3.10/site-packages/sklearn/utils/validation.py:1461, in check_is_fitted(estimator, attributes, msg, all_or_any)
   1458     raise TypeError("%s is not an estimator instance." % (estimator))
   1460 if not _is_fitted(estimator, attributes, all_or_any):
-> 1461     raise NotFittedError(msg % {"name": type(estimator).__name__})

NotFittedError: This RandomForestClassifier instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
```

Labels: Bug, Needs Triage