Repository: automl/auto-sklearn

Issue:
[Question] How to find the corresponding cv_results_ model information by Model_id
[Question] How to find the corresponding cv_results_ model information by Model_id
automl.show_models() can output Ensemble models, and I want to get the CV evaluation scores for each model, but cv_results_ does not have an index for model_id, so how do I combine the information from show_models() and cv_results_?



Discussion:
Hi,

Thank you for your question.

Unfortunately, for now I can't think of a solution to combine the results of show_models and cv_results_, but I will look into this and come back with an answer soon. I would first have to take a look into how the model_ids are generated and why they are not tracked by cv_results_.
@aron-barm, just a heads up that what we export in `cv_results_` tries to corresponds to what scikit-learn advertise it does:
* https://scikit-learn.org/stable/modules/grid_search.html#analyzing-results-with-the-cv-results-attribute

To insert a `model_id`, I guess we can break their convention a bit and insert an extra column, inserting it into the column `params` makes no sense to me and doesn't make things any easier.

Might also make sense to have the function just construct a pandas dataframe upfront and finally convert to a dictionary before returning, easier to validate what's going on but I'll leave that up to you!
Hi,

I wanted to solve this issue last week but unfortunately I could only get onto it today. I hope it wasn't too urgent for you.
I will submit a PR today with the proposed chages. 
Once it's merged, the model_ids will be tracked by cv_results so you should be able to combine information from show_models and cv_results.

Resolving patch:
diff --git a/autosklearn/automl.py b/autosklearn/automl.py
--- a/autosklearn/automl.py
+++ b/autosklearn/automl.py
@@ -1921,15 +1921,17 @@ def cv_results_(self):
             metric_dict[metric.name] = []
             metric_mask[metric.name] = []
 
+        model_ids = []
         mean_fit_time = []
         params = []
         status = []
         budgets = []
 
-        for run_key in self.runhistory_.data:
-            run_value = self.runhistory_.data[run_key]
+        for run_key, run_value in self.runhistory_.data.items():
             config_id = run_key.config_id
             config = self.runhistory_.ids_config[config_id]
+            if run_value.additional_info and "num_run" in run_value.additional_info:
+                model_ids.append(run_value.additional_info["num_run"])
 
             s = run_value.status
             if s == StatusType.SUCCESS:
@@ -1990,6 +1992,8 @@ def cv_results_(self):
                 metric_dict[metric.name].append(metric_value)
                 metric_mask[metric.name].append(mask_value)
 
+        results["model_ids"] = model_ids
+
         if len(self._metrics) == 1:
             results["mean_test_score"] = np.array(metric_dict[self._metrics[0].name])
             rank_order = -1 * self._metrics[0]._sign * results["mean_test_score"]
@@ -2165,14 +2169,11 @@ def show_models(self) -> dict[int, Any]:
             warnings.warn("No ensemble found. Returning empty dictionary.")
             return ensemble_dict
 
-        def has_key(rv, key):
-            return rv.additional_info and key in rv.additional_info
-
         table_dict = {}
-        for run_key, run_val in self.runhistory_.data.items():
-            if has_key(run_val, "num_run"):
-                model_id = run_val.additional_info["num_run"]
-                table_dict[model_id] = {"model_id": model_id, "cost": run_val.cost}
+        for run_key, run_value in self.runhistory_.data.items():
+            if run_value.additional_info and "num_run" in run_value.additional_info:
+                model_id = run_value.additional_info["num_run"]
+                table_dict[model_id] = {"model_id": model_id, "cost": run_value.cost}
 
         # Checking if the dictionary is empty
         if not table_dict:


Test patch: