Repository: automl/auto-sklearn

Issue:
Feature: Seperate Cost Function and Test Metric for Runhistory
Hello,
I'm looking to implement a custom loss/score function to train models on but to also have a separate metric that assess the model during the run, specifically the data available accessible through `model.automl_.runhistory_.data` and used in the plotting example show [here](https://automl.github.io/auto-sklearn/master/examples/40_advanced/example_pandas_train_test.html#sphx-glr-examples-40-advanced-example-pandas-train-test-py). I'm not sure if this is possible through a custom SMAC configuration but a high level Python interface could be quite helpful .

## Use Case:
I'd like to use autosklearn's runhistory to be able to plot its score on the test set provided by `model.fit(X, y, X_Test, y_test)` but have this score metric be separate from the metric used to train and find suitable models.

At the moment, I am using autosklearn to predict a multiclass classification vector. For test scoring, the model must only get one of the positive labels correct to be sufficient i.e. 
```Python
import numpy as np

test_scorer = lambda y, y_pred: np.any(y & y_pred).astype(int)

label = np.asarray([0, 1, 1, 0])
pred = np.asarray([0, 1, 0, 0])

test_scorer(label, pred) # == 1
```

If I were to use this `test_scorer` as training metric, a model could achieve 100% of the possible score by simply predicting `[1,1,1,1]` for all samples. 

However, in training, I would still like the model to be trained using the default `f1_macro`  or using the current method for provided a `custom_scorer` as documented [here](https://automl.github.io/auto-sklearn/master/api.html#metrics).

```Python
from autosklearn.metrics import f1_macro

f1_macro(label, pred) # == 0.733...
```

Apologies if this is possible but I could not find how in either the documentation or the source code, feel free to close this if it is already possible.


Discussion:
Hi, so you want to use a different metric for training and testing, and have the test score logged while running? I'm afraid that this is currently not possible.

In the meantime, I can think of a very hacky way of achieving this. If you put an if-statement in your scorer, you can check whether it scores the training or the test set and change the behavior accordingly. 

Nevertheless, we should add tracking additional metrics at some point as I guess this something a lot of people would like to have.

Resolving patch:
diff --git a/autosklearn/evaluation/abstract_evaluator.py b/autosklearn/evaluation/abstract_evaluator.py
--- a/autosklearn/evaluation/abstract_evaluator.py
+++ b/autosklearn/evaluation/abstract_evaluator.py
@@ -19,7 +19,7 @@
 from autosklearn.pipeline.implementations.util import (
     convert_multioutput_multiclass_to_multilabel
 )
-from autosklearn.metrics import calculate_score, CLASSIFICATION_METRICS, REGRESSION_METRICS
+from autosklearn.metrics import calculate_score
 from autosklearn.util.logging_ import get_named_client_logger
 
 from ConfigSpace import Configuration
@@ -264,14 +264,9 @@ def _loss(self, y_true, y_hat, scoring_functions=None):
             scoring_functions=scoring_functions)
 
         if hasattr(score, '__len__'):
-            # TODO: instead of using self.metric, it should use all metrics given by key.
-            # But now this throws error...
-            if self.task_type in CLASSIFICATION_TASKS:
-                err = {key: metric._optimum - score[key] for key, metric in
-                       CLASSIFICATION_METRICS.items() if key in score}
-            else:
-                err = {key: metric._optimum - score[key] for key, metric in
-                       REGRESSION_METRICS.items() if key in score}
+            err = {metric.name: metric._optimum - score[metric.name]
+                   for metric in scoring_functions}
+            err[self.metric.name] = self.metric._optimum - score[self.metric.name]
         else:
             err = self.metric._optimum - score
 
diff --git a/examples/40_advanced/example_calc_multiple_metrics.py b/examples/40_advanced/example_calc_multiple_metrics.py
new file mode 100644
--- /dev/null
+++ b/examples/40_advanced/example_calc_multiple_metrics.py
@@ -0,0 +1,66 @@
+# -*- encoding: utf-8 -*-
+"""
+=======
+Metrics
+=======
+
+In *Auto-sklearn*, model is optimized over a metric, either built-in or
+custom metric. Moreover, it is also possible to calculate multiple metrics
+per run. The following examples show how to calculate metrics built-in
+and self-defined metrics for a classification problem.
+"""
+
+import autosklearn.classification
+import custom_metrics
+import pandas as pd
+import sklearn.datasets
+import sklearn.metrics
+from autosklearn.metrics import balanced_accuracy, precision, recall, f1
+
+
+def get_metric_result(cv_results):
+    results = pd.DataFrame.from_dict(cv_results)
+    results = results[results['status'] == "Success"]
+    cols = ['rank_test_scores', 'param_classifier:__choice__', 'mean_test_score']
+    cols.extend([key for key in cv_results.keys() if key.startswith('metric_')])
+    return results[cols]
+
+
+if __name__ == "__main__":
+    ############################################################################
+    # Data Loading
+    # ============
+
+    X, y = sklearn.datasets.load_breast_cancer(return_X_y=True)
+    X_train, X_test, y_train, y_test = \
+        sklearn.model_selection.train_test_split(X, y, random_state=1)
+
+    ############################################################################
+    # Build and fit a classifier
+    # ==========================
+
+    error_rate = autosklearn.metrics.make_scorer(
+        name='custom_error',
+        score_func=custom_metrics.error,
+        optimum=0,
+        greater_is_better=False,
+        needs_proba=False,
+        needs_threshold=False
+    )
+    cls = autosklearn.classification.AutoSklearnClassifier(
+        time_left_for_this_task=120,
+        per_run_time_limit=30,
+        scoring_functions=[balanced_accuracy, precision, recall, f1, error_rate]
+    )
+    cls.fit(X_train, y_train, X_test, y_test)
+
+    ###########################################################################
+    # Get the Score of the final ensemble
+    # ===================================
+
+    predictions = cls.predict(X_test)
+    print("Accuracy score", sklearn.metrics.accuracy_score(y_test, predictions))
+
+    print("#" * 80)
+    print("Metric results")
+    print(get_metric_result(cls.cv_results_).to_string(index=False))


Test patch: