Training final model with cross validation and using it to get unbiased probabilities

### Describe the workflow you want to enable

I want to use crossvalidation with let's say k=4 in order to get four models. That means that each sample in my dataset was used to train 3 of the four models. Thus, if I want to get a prediction for a given sample, I need to use the one model that was not trained with that particular sample. 

However, when I train my models and I get the four pickle files, I cannot pick the right model in an out-of-the-box way. I.e. there is no way to tell which model was not trained with that specific sample.

### Describe your proposed solution

The models could be loaded in some sort of model interface, whiich would take care of picking the right model for the prediction:

```python
df_feat = _get_features()
model_wrapper = load_models('/path/to/models/model*.pkl')
l_prob = mode_wrapper.predict_proba(df_feat)
```

This would mean that the model would have to somehow know that a given feature was used to train it, like:

```python
prob = None
for model in l_model:
    if model.used_feature(sr_feat):
        continue

    prob=model.predict_proba(sr_feat)
```

Which would mean that the model would get pretty large, unless each feature gets stored in a sort of short way within the model. E.g. storing some sort of hash for the whole training dataset as an attribute.

### Describe alternatives you've considered, if relevant

I can implement this myself using derived classes, a hash check. However it would be good to have it done upstream, or maybe it already exists?

### Additional context

The way I understand this is done in the real world is that cross validation is used to validate the model against overtraining. Once the model is validated, one trains the model again with the whole dataset, instead of e.g. 75% of it as in the example above. Thus one uses only one model instead of 4.

However in particle physics, people do not like to use a model trained with sample `S1` to make predictions for sample `S1`. That is regarded as introduced biases, and we need to reuse the datasets that were used for training also for analysis. Thus we need a more user-friendly way of doing this picking.

COMMENT:
> Sorry, some individual parts make sense but I am not 100% sure what you are trying to do ...
> 
> Can you share some code showing what you are currently doing with scikit-learn and which limitation you are hitting? Are you already using something like `sklearn.model_selection.cross_val_score` / `sklearn.model_selection.cross_validate` or `sklearn.model_selection.GridSearchCV`?

Hi, sure, thanks for checking my issue. The basic idea is to train 10 models from a given dataset, using cross validation. The models would know what sample was used to train them. Thus, when doing the probability prediction, the tool that does the prediction would switch between models, such that no sample would get a prediction from a model that was trained with that sample. 

I already created the derived class here:

https://github.com/acampove/dmu/blob/main/src/dmu/ml/cv_classifier.py

and the class that reads the probabilities here:

https://github.com/acampove/dmu/blob/main/src/dmu/ml/cv_predict.py

The unit tests are:

https://github.com/acampove/dmu/blob/main/tests/ml/test_cv_classifier.py
https://github.com/acampove/dmu/blob/main/tests/ml/test_cv_predict.py

and the models are trained with:

https://github.com/acampove/dmu/blob/main/src/dmu/ml/train_mva.py
https://github.com/acampove/dmu/blob/main/tests/ml/test_train_mva.py

Unfortunately, you might need ROOT to run all this. Feel free to take whatever you want/need from these files. Also let me know if you need anything else from me.

Labels: New Feature, Needs Info