Repository: autogluon/autogluon

Issue:
Is the calculation of the RMSE of test data correct?
Is the RMSE calculation in evaluate_prediction in autogluon version 0.0.11 correct?

From a prediction with the test data:
```
y_pred = predictor.predict(X_raw)
perf = predictor.evaluate_predictions(
        y_true=y_test, y_pred=y_pred, auxiliary_metrics=True
    )
```
I got:

```
Evaluation: root_mean_squared_error on test data: 1.9002197479504435
Evaluations on test data:
{
    "root_mean_squared_error": 1.9002197479504435,
    "mean_absolute_error": 0.2822201467381229,
    "explained_variance_score": 0.9041746987036652,
    "r2_score": 0.9041688612988515,
    "pearson_correlation": 0.9509362562666829,
    "mean_squared_error": 0.17589136840340483,
    "median_absolute_error": 0.18991703222021486
}
```

When the RMSE is calculated using
```
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

def rmse(y_pred, y_true):
    return np.sqrt(mean_squared_error(y_pred, y_true)) 

print(f'RMSE: {rmse(y_test, y_pred):.3f}')
print(f'MSE: {mean_squared_error(y_test, y_pred):.3f}')
print(f'r2 score: {r2_score(y_test, y_pred):.3f}')
```
the following values are returned:
```
RMSE: 0.419
MSE: 0.176
r2 score: 0.904
```



Discussion:
Interesting... Could you also do predictor.leaderboard(X_raw_with_labels)? That will indicate what is internally being used by AutoGluon.
This is the AutoGluon source code for rmse (Same as in https://stackoverflow.com/questions/17197492/is-there-a-library-function-for-root-mean-square-error-rmse-in-python):

```
def rmse_func(predictions, targets):
    return np.sqrt(((predictions - targets) ** 2).mean())
```
I've ran a test on this and it seems to be consistent (differing from your example). Could you double-check your code and ensure you ran everything in the same cell?

On 0.0.12 (No changes to RMSE):

```
predictor = task.fit(
    # time_limits=120,
    train_data=X,
    label=LABEL,
    eval_metric='root_mean_squared_error',
    hyperparameters={
        'CAT': {},
    },
)

path_test = path_prefix + 'test_data.csv'
X_test = load_pd.load(path_test)

predictor.leaderboard(X_test)

y_test = X_test[LABEL]
y_pred = predictor.predict(X_test)
perf = predictor.evaluate_predictions(y_true=y_test, y_pred=y_pred, auxiliary_metrics=True)
from sklearn.metrics import mean_squared_error, r2_score
import numpy as np

def rmse(y_pred, y_true):
    return np.sqrt(mean_squared_error(y_pred, y_true))

print(f'RMSE: {rmse(y_test, y_pred):.3f}')
print(f'MSE: {mean_squared_error(y_test, y_pred):.3f}')
print(f'r2 score: {r2_score(y_test, y_pred):.3f}')

```

Output:

```
                     model    score_test     score_val  pred_time_test  pred_time_val   fit_time  pred_time_test_marginal  pred_time_val_marginal  fit_time_marginal  stack_level  can_infer
0        CatboostRegressor -19219.966533 -26231.021279        0.071855       0.067640  77.554613                 0.071855                0.067640          77.554613            0       True
1  weighted_ensemble_k0_l1 -19219.966533 -26231.021279        0.075469       0.069419  77.559314                 0.003614                0.001779           0.004701            1       True
Evaluation: root_mean_squared_error on test data: 19219.966533285162
Evaluations on test data:
{
    "root_mean_squared_error": 19219.966533285162,
    "mean_absolute_error": 13203.561436971126,
    "explained_variance_score": 0.9259382784632942,
    "r2_score": 0.9256836316437639,
    "pearson_correlation": 0.96245422205405,
    "mean_squared_error": 369407113.5406017,
    "median_absolute_error": 9687.979841075066
}
RMSE: 19219.967
MSE: 369407113.541
r2 score: 0.926

```
See below for the output from the leaderboard.
```
predictor.leaderboard(test_data)

model  score_test  score_val  pred_time_test  pred_time_val      fit_time  pred_time_test_marginal  pred_time_val_marginal fit_time_marginal  stack_level  can_infer
0               weighted_ensemble_k0_l2   -0.419394  -0.419480     1371.710608     363.260315  85731.478442                 0.017222                0.001277           1.162603            2       True
1          LightGBMRegressor_STACKER_l1   -0.419805  -0.421002     1291.514437     346.914402  52674.690803                 5.264961                2.719512          49.868780            1       True
2    LightGBMRegressorCustom_STACKER_l1   -0.419831  -0.420828     1292.150959     347.095359  52745.572411                 5.901483                2.900468         120.750388            1       True
3          CatboostRegressor_STACKER_l1   -0.421231  -0.423362     1287.817158     345.872373  52695.846922                 1.567682                1.677483          71.024898            1       True
4   RandomForestRegressorMSE_STACKER_l1   -0.422732  -0.424340     1308.213786     349.527967  63250.733135                21.964311                5.333077       10625.911111            1       True
5     ExtraTreesRegressorMSE_STACKER_l1   -0.423172  -0.423621     1336.994950     350.628498  74862.760662                50.745474                6.433608       22237.938638            1       True
6               weighted_ensemble_k0_l1   -0.426322  -0.425883       57.560714      16.361131  16316.717391                 0.015704                0.001499           1.403781            1       True
7    LightGBMRegressorCustom_STACKER_l0   -0.428854  -0.431920       14.277234       5.369839   1295.655222                14.277234                5.369839        1295.655222            0       True
8          LightGBMRegressor_STACKER_l0   -0.432707  -0.440154        7.679609       3.904525    362.878872                 7.679609                3.904525         362.878872            0       True
9          CatboostRegressor_STACKER_l0   -0.432829  -0.437915        1.056855       1.666364   1155.844561                 1.056855                1.666364        1155.844561            0       True
10    ExtraTreesRegressorMSE_STACKER_l0   -0.469459  -0.467605       34.531311       5.418904  13500.934955                34.531311                5.418904       13500.934955            0       True
11  RandomForestRegressorMSE_STACKER_l0   -0.512019  -0.508367       35.587046       5.374662   7508.789899                35.587046                5.374662        7508.789899            0       True
12        NeuralNetRegressor_STACKER_l1   -0.635227  -1.591328     1582.314277     434.648647  83989.564337               296.064801               90.453757       31364.742313            1       True
13   KNeighborsRegressorDist_STACKER_l1   -0.753965  -0.768754     1818.558200     461.193521  52739.027242               532.308724              116.998631         114.205219            1       True
14   KNeighborsRegressorDist_STACKER_l0   -0.758689  -0.774370      365.134637     116.263620    113.223741               365.134637              116.263620         113.223741            0       True
15   KNeighborsRegressorUnif_STACKER_l1   -0.781376  -0.795930     1619.714213     461.372831  52739.213231               333.464738              117.177941         114.391208            1       True
16   KNeighborsRegressorUnif_STACKER_l0   -0.787333  -0.802268      348.599119     116.399790    113.495155               348.599119              116.399790         113.495155            0       True
17        NeuralNetRegressor_STACKER_l0   -0.884560  -3.940324      479.383664      89.797186  28573.999618               479.383664               89.797186       28573.999618            0       True
```
The output from `evaluate_predictions` is still:
`Evaluation: root_mean_squared_error on test data: 1.9002197479504435`

Two other observation:
A) I trained 14 regression models (training data had on average 24000 samples and 1243 features) using:
```
predictor = task.fit(
        train_data=train_data,
        label=label_column,
        output_directory=output_directory,
        auto_stack=True,
        feature_prune=True,
        verbosity=0,
    )
```
**NeuralNetRegressor_STACKER_l0 always had the highest RMSE.**

B) In one of my first runs the training dataset contained a couple of `Inf` and `-Inf` y-values, which crashed the training without giving a meaningful error message.
 
Thanks for the clarification! I believe I have narrowed down that this has to be a bug. I also suspect that `predictor.evaluate(test_data)` will produce the correct value, and that the bug purely lies in `predictor.evaluate_predictions` because it uses separate logic from what the rest of the code uses to calculate scores.

Contributions are welcome to try to refactor `predictor.evaluate_predictions` logic to re-use the other internal scoring logic. If nobody contributes a fix, I will try to find time eventually to resolve this.

@schinto To help the process along, could you provide a MWE of this defect in action? A dummy dataset generated through a few lines of code should suffice, along with the relevant calls to the defective logic + the correct logic. Also make sure to use v0.0.12 when constructing the example, it is possible it was fixed in 0.0.12 since you mentioned you were using 0.0.11. Regarding your other comments: A: This is a very deep topic regarding model behavior on unique datasets, and should exist in a separate issue if you believe that the neural network should be doing better on this problem (Along with ideally a NN example that fairs better than what AG produced). B: This is a valid defect that should be fixed, could you create a separate issue with a minimal example so we can work to resolve?
Minimum Working Example illustrating the problem with `root_mean_squared_error`:
```
import pandas as pd
import numpy as np
from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
import autogluon as ag
from autogluon import TabularPrediction as task
from sklearn.metrics import mean_squared_error, r2_score


print(f'Autogluon version: {ag.__version__}')

# Make simulated data for regression
X,y = make_regression(n_samples=100, n_features=6, n_informative=3,
n_targets=1, tail_strength=0.5, noise=0.02,
shuffle=True, coef=False, random_state=42)

xcols = [ f'x{i:04}' for i in range(X.shape[1]) ]
ycols = ['y']
df = pd.concat([pd.DataFrame(y, columns=ycols), pd.DataFrame(X, columns=xcols)], axis=1)

# Split dataframe in training and test data
train_data, test_data = train_test_split(df, train_size=0.75, random_state=42)
X_train, y_train = train_data.drop(columns=['y']), train_data['y']
X_test,  y_test  = test_data.drop(columns=['y']), test_data['y']

# Train model
label = 'y'
predictor = task.fit(
        time_limits=120,
        train_data=train_data,
        label=label,
        auto_stack=True,
        feature_prune=True,
        verbosity=0,
)

# predictor.leaderboard()
y_pred = predictor.predict(X_test)

perf = predictor.evaluate_predictions(
    y_true=y_test, y_pred=y_pred, auxiliary_metrics=True
)

print('Compare performance metrics:')
print('(A) Performance metrics from autogluon')
perf_df = pd.DataFrame(perf.values(), index=perf.keys(), columns=['value'])
print(perf_df.round(decimals=3))

# Calculate root mean square error from sklearn's mean_squared_error
def rmse(y_pred, y_true):
    return np.sqrt(mean_squared_error(y_pred, y_true))

print('\n(B) Performance metrics using sklearn function')
print(f'root_mean_squared_error: {rmse(y_test, y_pred):.3f}')
print(f'mean_squared_error: {mean_squared_error(y_test, y_pred):.3f}')
print(f'r2_score: {r2_score(y_test, y_pred):.3f}')

# lboard = predictor.leaderboard(test_data, silent=True)
# lboard.sort_values(by='score_val', ascending=False)
```
Output:
```
Autogluon version: 0.0.11
Compare performance metrics:
(A) Performance metrics from autogluon
                            value
root_mean_squared_error   108.738
mean_absolute_error        10.569
explained_variance_score    0.970
r2_score                    0.964
pearson_correlation         0.992
mean_squared_error        232.442
median_absolute_error       7.206

(B) Performance metrics using sklearn functions
root_mean_squared_error: 15.246
mean_squared_error: 232.442
r2_score: 0.964
```

`sqrt(232.442) = 15.246` => root_mean_squared_error returned by function `evaluate_predictions` is wrong.
@schinto Awesome, thanks!

Resolving patch:
diff --git a/autogluon/task/tabular_prediction/predictor.py b/autogluon/task/tabular_prediction/predictor.py
--- a/autogluon/task/tabular_prediction/predictor.py
+++ b/autogluon/task/tabular_prediction/predictor.py
@@ -134,6 +134,7 @@ def predict(self, dataset, model=None, as_pandas=False, use_pred_cache=False, ad
 
     def predict_proba(self, dataset, model=None, as_pandas=False, as_multiclass=False):
         """ Use trained models to produce predicted class probabilities rather than class-labels (if task is classification).
+            If `predictor.problem_type` is regression, this functions identically to `predict`, returning the same output.
 
             Parameters
             ----------
diff --git a/autogluon/utils/tabular/ml/learner/abstract_learner.py b/autogluon/utils/tabular/ml/learner/abstract_learner.py
--- a/autogluon/utils/tabular/ml/learner/abstract_learner.py
+++ b/autogluon/utils/tabular/ml/learner/abstract_learner.py
@@ -364,6 +364,10 @@ def _remove_missing_labels(self, y_true, y_pred):
                           f"You should carefully study why there are missing labels in your evaluation data.")
         return y_true, y_pred
 
+    # TODO: Refactor to be less brittle.
+    # TODO: Instead take y_pred_proba as input, convert to y_pred when necessary. Treat y_pred_proba as y_pred for regression.
+    #  This makes this function behave much nicer, currently confusion matrix is only able to be constructed when y_pred is supplied, and crashes when y_pred_proba is supplied.
+    #  Therefore, it is impossible to get confusion matrix when eval_metric is log_loss. This shouldn't be the case.
     def evaluate(self, y_true, y_pred, silent=False, auxiliary_metrics=False, detailed_report=True, high_always_good=False):
         """ Evaluate predictions.
             Args:
@@ -376,33 +380,16 @@ def evaluate(self, y_true, y_pred, silent=False, auxiliary_metrics=False, detail
             Otherwise returns dict where keys = metrics, values = performance along each metric.
         """
         assert isinstance(y_true, (np.ndarray, pd.Series))
-        assert isinstance(y_pred, (np.ndarray, pd.Series))
+        assert isinstance(y_pred, (np.ndarray, pd.Series))  # TODO: Enable DataFrame for y_pred_proba
 
         # TODO: Consider removing _remove_missing_labels, this creates an inconsistency between how .score, .score_debug, and .evaluate compute scores.
         y_true, y_pred = self._remove_missing_labels(y_true, y_pred)
-        trainer = self.load_trainer()
-        if trainer.eval_metric_expects_y_pred:
-            y_pred_cleaned = self.label_cleaner.transform(y_pred)
-            y_true_cleaned = self.label_cleaner.transform(y_true)
-            if self.problem_type == MULTICLASS:
-                y_true_cleaned = y_true_cleaned.fillna(-1)  # map unknown classes to -1
-            performance = self.eval_metric(y_true_cleaned, y_pred_cleaned)
-        else:
-            if self.problem_type == MULTICLASS:
-                y_true_cleaned = self.label_cleaner.transform(y_true)
-                y_true_cleaned = y_true_cleaned.fillna(-1)
-                if (not trainer.eval_metric_expects_y_pred) and (-1 in y_true_cleaned.unique()):
-                    # log_loss / pac_score
-                    raise ValueError(f'Multiclass scoring with eval_metric=\'{self.eval_metric.name}\' does not support unknown classes.')
-                performance = self.eval_metric(y_true_cleaned, y_pred)
-            else:
-                performance = self.eval_metric(y_true, y_pred)
+        performance = self.eval_metric(y_true, y_pred)
 
         metric = self.eval_metric.name
 
         if not high_always_good:
-            sign = self.eval_metric._sign
-            performance = performance * sign  # flip negative once again back to positive (so higher is no longer necessarily better)
+            performance = performance * self.eval_metric._sign  # flip negative once again back to positive (so higher is no longer necessarily better)
 
         if not silent:
             logger.log(20, f"Evaluation: {metric} on test data: {performance}")
@@ -427,9 +414,9 @@ def evaluate(self, y_true, y_pred, silent=False, auxiliary_metrics=False, detail
                 #   return auc(fpr, tpr)
                 f1micro_score = lambda y_true, y_pred: f1_score(y_true, y_pred, average='micro')
                 f1micro_score.__name__ = f1_score.__name__
-                auxiliary_metrics += [f1micro_score, (confusion_matrix, {'labels': self.label_cleaner.ordered_class_labels, 'output_format': 'pandas_dataframe' })]  # TODO: add auc?
-            elif self.problem_type == MULTICLASS:  # multiclass metrics
-                auxiliary_metrics += [(confusion_matrix, {'labels': self.label_cleaner.ordered_class_labels, 'output_format': 'pandas_dataframe' })]  # TODO: No multi-class specific metrics for now. Include top-5, top-10 accuracy here.
+                auxiliary_metrics += [f1micro_score]  # TODO: add auc?
+            # elif self.problem_type == MULTICLASS:  # multiclass metrics
+            #     auxiliary_metrics += []  # TODO: No multi-class specific metrics for now. Include top-5, top-10 accuracy here.
 
         performance_dict = OrderedDict({metric: performance})
         for metric_function in auxiliary_metrics:
@@ -439,14 +426,11 @@ def evaluate(self, y_true, y_pred, silent=False, auxiliary_metrics=False, detail
                 metric_kwargs = None
             metric_name = metric_function.__name__
             if metric_name not in performance_dict:
-                try: # only compute auxiliary metrics which do not error (y_pred = class-probabilities may cause some metrics to error)
+                try:  # only compute auxiliary metrics which do not error (y_pred = class-probabilities may cause some metrics to error)
                     if metric_kwargs:
                         performance_dict[metric_name] = metric_function(y_true, y_pred, **metric_kwargs)
                     else:
                         performance_dict[metric_name] = metric_function(y_true, y_pred)
-                    if metric_name == 'confusion_matrix':
-                        # Pandas Dataframe is not JSON Serializable. TODO: Standardize output format of each data type
-                        performance_dict[metric_name] = performance_dict[metric_name].to_json()
                 except ValueError:
                     pass
 
@@ -455,11 +439,16 @@ def evaluate(self, y_true, y_pred, silent=False, auxiliary_metrics=False, detail
             logger.log(20, json.dumps(performance_dict, indent=4))
 
         if detailed_report and (self.problem_type != REGRESSION):
+            # Construct confusion matrix
+            try:
+                performance_dict['confusion_matrix'] = confusion_matrix(y_true, y_pred, labels=self.label_cleaner.ordered_class_labels, output_format='pandas_dataframe')
+            except ValueError:
+                pass
             # One final set of metrics to report
             cl_metric = lambda y_true, y_pred: classification_report(y_true, y_pred, output_dict=True)
             metric_name = 'classification_report'
             if metric_name not in performance_dict:
-                try: # only compute auxiliary metrics which do not error (y_pred = class-probabilities may cause some metrics to error)
+                try:  # only compute auxiliary metrics which do not error (y_pred = class-probabilities may cause some metrics to error)
                     performance_dict[metric_name] = cl_metric(y_true, y_pred)
                 except ValueError:
                     pass


Test patch: