Repository: andersbogsnes/ml_tooling

Issue:
BUG: When using the method Model.test_estimators with refit=True it's not using the specified metrics.
**Describe the bug**
When using the method Model.test_estimators with refit=True, it's not using the specified metrics. Instead it's using the default metric.

Additionally, wouldn't you except it to be refitting using CV as well if specified in the call?

**To Reproduce**
1) Run the code:
```python
from sklearn.dummy import DummyClassifier
from ml_tooling import Model
from ml_tooling.data.load_demo import load_demo_dataset

data = load_demo_dataset("breast_cancer")
data.create_train_test()

estimators = [DummyClassifier()]

clf, results = Model.test_estimators(data, estimators, metrics="roc_auc", refit=True)

assert clf.result.metrics.name == "accuracy"
clf.result
```
2) Verify that the metric is not roc_auc but accuracy.

**Expected behavior**
I'd expect it to keep evaluating with the previous chosen metrics and I'd expect it to keep using CV if chosen previously.

- [X] I have updated ML Tooling to the latest version
- [X] I have checked if this issue has been reported previously


Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752?src=pr&el=h1) Report
> Merging [#752](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752?src=pr&el=desc) (2568f88) into [develop](https://codecov.io/gh/andersbogsnes/ml_tooling/commit/18e5b480bea980347fde64ccbdb81b00dcc04232?el=desc) (18e5b48) will **not change** coverage.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752/graphs/tree.svg?width=650&height=150&src=pr&token=ucKtL3BlxO)](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           develop      #752    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           55        64     +9     
  Lines         1424      1709   +285     
  Branches       166       192    +26     
==========================================
+ Hits          1424      1709   +285     
```


| [Impacted Files](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/ml\_tooling/plots/utils.py](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752/diff?src=pr&el=tree#diff-c3JjL21sX3Rvb2xpbmcvcGxvdHMvdXRpbHMucHk=) | `100.00% <ø> (ø)` | |
| [src/ml\_tooling/transformers/binner.py](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752/diff?src=pr&el=tree#diff-c3JjL21sX3Rvb2xpbmcvdHJhbnNmb3JtZXJzL2Jpbm5lci5weQ==) | `100.00% <ø> (ø)` | |
| [src/ml\_tooling/transformers/feature\_union.py](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752/diff?src=pr&el=tree#diff-c3JjL21sX3Rvb2xpbmcvdHJhbnNmb3JtZXJzL2ZlYXR1cmVfdW5pb24ucHk=) | `100.00% <ø> (ø)` | |
| [src/ml\_tooling/transformers/renamer.py](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752/diff?src=pr&el=tree#diff-c3JjL21sX3Rvb2xpbmcvdHJhbnNmb3JtZXJzL3JlbmFtZXIucHk=) | `100.00% <ø> (ø)` | |
| [src/ml\_tooling/transformers/to\_categorical.py](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752/diff?src=pr&el=tree#diff-c3JjL21sX3Rvb2xpbmcvdHJhbnNmb3JtZXJzL3RvX2NhdGVnb3JpY2FsLnB5) | `100.00% <ø> (ø)` | |
| [src/ml\_tooling/\_\_init\_\_.py](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752/diff?src=pr&el=tree#diff-c3JjL21sX3Rvb2xpbmcvX19pbml0X18ucHk=) | `100.00% <100.00%> (ø)` | |
| [src/ml\_tooling/baseclass.py](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752/diff?src=pr&el=tree#diff-c3JjL21sX3Rvb2xpbmcvYmFzZWNsYXNzLnB5) | `100.00% <100.00%> (ø)` | |
| [src/ml\_tooling/config.py](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752/diff?src=pr&el=tree#diff-c3JjL21sX3Rvb2xpbmcvY29uZmlnLnB5) | `100.00% <100.00%> (ø)` | |
| [src/ml\_tooling/data/\_\_init\_\_.py](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752/diff?src=pr&el=tree#diff-c3JjL21sX3Rvb2xpbmcvZGF0YS9fX2luaXRfXy5weQ==) | `100.00% <100.00%> (ø)` | |
| [src/ml\_tooling/data/base\_data.py](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752/diff?src=pr&el=tree#diff-c3JjL21sX3Rvb2xpbmcvZGF0YS9iYXNlX2RhdGEucHk=) | `100.00% <100.00%> (ø)` | |
| ... and [51 more](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752?src=pr&el=footer). Last update [c84a00e...2568f88](https://codecov.io/gh/andersbogsnes/ml_tooling/pull/752?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

andersbogsnes: Code looks good - remember to update the CHANGELOG and maybe make a note in the docs?
Waiting for discussion on #751 to conclude before accepting
andersbogsnes: You can test if the scores are cross-validated by looking at the metrics `cross_validated_scores` attribute

Patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 6b95b1a..2645563 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,6 @@
 # v.0.12.1
 - Dataset features can now be easily accessed with the property dataset.features
+- `model.test_estimators` with CV will keep using CV if it's refitting the best estimator.
 
 # v0.12.0
 - Permutation importance and Feature importance are now two different plotting methods.
diff --git a/src/ml_tooling/baseclass.py b/src/ml_tooling/baseclass.py
index d1d2134..531c7ad 100644
--- a/src/ml_tooling/baseclass.py
+++ b/src/ml_tooling/baseclass.py
@@ -370,7 +370,7 @@ class Model:
             f"{results[0].metrics.name}: {results[0].metrics.score}"
         )
         if refit:
-            best_estimator.score_estimator(data)
+            best_estimator.score_estimator(data, metrics=metrics, cv=cv)
 
         return best_estimator, results
 


Test patch:
diff --git a/tests/test_baseclass.py b/tests/test_baseclass.py
index 3b47d11..0c10afc 100644
--- a/tests/test_baseclass.py
+++ b/tests/test_baseclass.py
@@ -587,6 +587,38 @@ class TestModelSelection:
         for result in results:
             assert "roc_auc" in result.metrics
 
+    def test_model_selection_with_nonstandard_metric_and_refitting_keeps_same_metric(
+        self, train_iris_dataset
+    ):
+        estimators = [
+            LogisticRegression(solver="liblinear"),
+            RandomForestClassifier(n_estimators=10),
+        ]
+
+        metrics = ["roc_auc", "f1_macro"]
+        best_estimator, results = Model.test_estimators(
+            train_iris_dataset, estimators, metrics=metrics, refit=True
+        )
+        assert all(metric in best_estimator.result.metrics for metric in metrics)
+
+    def test_model_selection_uses_cv_parameter_in_refitting(self, train_iris_dataset):
+        estimators = [
+            LogisticRegression(solver="liblinear"),
+            RandomForestClassifier(n_estimators=10),
+        ]
+
+        metrics = ["roc_auc", "f1_macro"]
+        cv = 2
+
+        best_estimator, results = Model.test_estimators(
+            train_iris_dataset, estimators, metrics=metrics, refit=True
+        )
+        best_estimator_cv, results = Model.test_estimators(
+            train_iris_dataset, estimators, metrics=metrics, refit=True, cv=cv
+        )
+        assert not best_estimator.result.metrics[0].cross_val_scores
+        assert len(best_estimator_cv.result.metrics[0].cross_val_scores) == cv
+
     def test_model_selection_with_pipeline_works_as_expected(
         self,
         pipeline_logistic: Pipeline,