Repository: autogluon/autogluon
Pull request: Clarify that eval_metric has moved from fit() to Predictor() (#1483)
*Issue #, if available:*
1483

*Description of changes:*
Clarify that eval_metric has moved from fit() to Predictor() 

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.


Problem statement:
eval_metric applies to TabularPredictor() but logging and documentation says it's on fit()
This is the correct way to specify an eval_metric, on the TabularPredictor constructor:
```
predictor = TabularPredictor(label='label', eval_metric='root_mean_squared_error')
predictor.fit(dftrain)
```
But the documentation says to add it to the `fit()` call.  This is both within the code:
```
AutoGluon will gauge predictive performance using evaluation metric: 'root_mean_squared_error'
	To change this, specify the eval_metric argument of fit()
```

And also mentioned within the documentation:
- tabular-quickstart.html says in multiple places, things like:
> set the `eval_metric` argument of `fit()`
- tabular-indepth.md mentions eval_metric as a parameter to fit().

Hints from issue discussion:
You can assign this to me if you want, I will attack it at the weekend.
Good catch! eval_metric used to be in fit but was changed to init in v0.1.

Commit message: Clarify that eval_metric has moved from fit() to Predictor() (#1483)

Patch to review:
diff --git a/core/src/autogluon/core/trainer/abstract_trainer.py b/core/src/autogluon/core/trainer/abstract_trainer.py
index 22b114d7b1c7..2f723e09a463 100644
--- a/core/src/autogluon/core/trainer/abstract_trainer.py
+++ b/core/src/autogluon/core/trainer/abstract_trainer.py
@@ -60,7 +60,7 @@ def __init__(self, path: str, problem_type: str, eval_metric=None,
         if not (self.eval_metric.needs_pred or self.eval_metric.needs_quantile):
             logger.log(25, "\tThis metric expects predicted probabilities rather than predicted class labels, so you'll need to use predict_proba() instead of predict()")
 
-        logger.log(20, "\tTo change this, specify the eval_metric argument of fit()")
+        logger.log(20, "\tTo change this, specify the eval_metric argument of Predictor()")
         self.num_classes = num_classes
         self.quantile_levels = quantile_levels
         self.feature_prune = False  # will be set to True if feature-pruning is turned on.
diff --git a/docs/tutorials/tabular_prediction/tabular-indepth.md b/docs/tutorials/tabular_prediction/tabular-indepth.md
index a10ef2731557..1eeb37534698 100644
--- a/docs/tutorials/tabular_prediction/tabular-indepth.md
+++ b/docs/tutorials/tabular_prediction/tabular-indepth.md
@@ -3,7 +3,7 @@
 
 **Tip**: If you are new to AutoGluon, review :ref:`sec_tabularquick` to learn the basics of the AutoGluon API. To learn how to add your own custom models to the set that AutoGluon trains, tunes, and ensembles, review :ref:`sec_tabularcustommodel`.
 
-This tutorial describes how you can exert greater control when using AutoGluon's `fit()` or `predict()`. Recall that to maximize predictive performance, you should always first try `fit()` with all default arguments except `eval_metric` and `presets`, before you experiment with other arguments covered in this in-depth tutorial like `hyperparameter_tune_kwargs`, `hyperparameters`, `num_stack_levels`, `num_bag_folds`, `num_bag_sets`, etc.
+This tutorial describes how you can exert greater control when using AutoGluon's `fit()` or `predict()`. Recall that to maximize predictive performance, you should first try `TabularPredictor()` and `fit()` with all default arguments.  Then, consider non-default arguments for `TabularPredictor(eval_metric=...)`, and `fit(presets=...)`.  Later, you can experiment with other arguments to fit() covered in this in-depth tutorial like `hyperparameter_tune_kwargs`, `hyperparameters`, `num_stack_levels`, `num_bag_folds`, `num_bag_sets`, etc.
 
 Using the same census data table as in the :ref:`sec_tabularquick` tutorial, we'll now predict the `occupation` of an individual - a multiclass classification problem. Start by importing AutoGluon's TabularPredictor and TabularDataset, and loading the data.
 
diff --git a/docs/tutorials/tabular_prediction/tabular-quickstart.md b/docs/tutorials/tabular_prediction/tabular-quickstart.md
index 232fb4babef3..c006375c347d 100644
--- a/docs/tutorials/tabular_prediction/tabular-quickstart.md
+++ b/docs/tutorials/tabular_prediction/tabular-quickstart.md
@@ -67,8 +67,7 @@ from autogluon.tabular import TabularPredictor
 predictor = TabularPredictor(label=<variable-name>).fit(train_data=<file-name>)
 ```
 
-**Note:** This simple call to `fit()` is intended for your first prototype model. In a subsequent section, we'll demonstrate how to maximize predictive performance by additionally specifying two `fit()` arguments: `presets` and `eval_metric`.
-
+**Note:** This simple call to `fit()` is intended for your first prototype model. In a subsequent section, we'll demonstrate how to maximize predictive performance by additionally specifying the `presets` argument to `fit()` and the `eval_metric` argument to `TabularPredictor()`.
 
 ## Description of fit():
 
@@ -134,7 +133,7 @@ This command implements the following strategy to maximize accuracy:
 
 - Specify the argument `presets='best_quality'`, which allows AutoGluon to automatically construct powerful model ensembles based on [stacking/bagging](https://arxiv.org/abs/2003.06505), and will greatly improve the resulting predictions if granted sufficient training time. The default value of `presets` is `'medium_quality_faster_train'`, which produces *less* accurate models but facilitates faster prototyping. With `presets`, you can flexibly prioritize predictive accuracy vs. training/inference speed. For example, if you care less about predictive performance and want to quickly deploy a basic model, consider using: `presets=['good_quality_faster_inference_only_refit', 'optimize_for_deployment']`.
 
-- Provide the `eval_metric` if you know what metric will be used to evaluate predictions in your application. Some other non-default metrics you might use include things like: `'f1'` (for binary classification), `'roc_auc'` (for binary classification), `'log_loss'` (for classification), `'mean_absolute_error'` (for regression), `'median_absolute_error'` (for regression).  You can also define your own custom metric function, see examples in the folder: `autogluon/core/metrics/`
+- Provide the `eval_metric` to `TabularPredictor()` if you know what metric will be used to evaluate predictions in your application. Some other non-default metrics you might use include things like: `'f1'` (for binary classification), `'roc_auc'` (for binary classification), `'log_loss'` (for classification), `'mean_absolute_error'` (for regression), `'median_absolute_error'` (for regression).  You can also define your own custom metric function, see examples in the folder: `autogluon/core/metrics/`
 
 - Include all your data in `train_data` and do not provide `tuning_data` (AutoGluon will split the data more intelligently to fit its needs).
 
@@ -161,7 +160,7 @@ predictor_age = TabularPredictor(label=age_column, path="agModels-predictAge").f
 performance = predictor_age.evaluate(test_data)
 ```
 
-Note that we didn't need to tell AutoGluon this is a regression problem, it automatically inferred this from the data and reported the appropriate performance metric (RMSE by default). To specify a particular evaluation metric other than the default, set the `eval_metric` argument of `fit()` and AutoGluon will tailor its models to optimize your metric (e.g. `eval_metric = 'mean_absolute_error'`). For evaluation metrics where higher values are worse (like RMSE), AutoGluon may sometimes flip their sign and print them as negative values during training (as it internally assumes higher values are better).
+Note that we didn't need to tell AutoGluon this is a regression problem, it automatically inferred this from the data and reported the appropriate performance metric (RMSE by default). To specify a particular evaluation metric other than the default, set the `eval_metric` argument of `TabularPredictor()` and AutoGluon will tailor its models to optimize your metric (e.g. `eval_metric = 'mean_absolute_error'`). For evaluation metrics where higher values are worse (like RMSE), AutoGluon may sometimes flip their sign and print them as negative values during training (as it internally assumes higher values are better).
 
 We can call leaderboard to see the per-model performance:
 
diff --git a/examples/tabular/example_advanced_tabular.py b/examples/tabular/example_advanced_tabular.py
index d44f5b5b098a..dcc4717c63c4 100644
--- a/examples/tabular/example_advanced_tabular.py
+++ b/examples/tabular/example_advanced_tabular.py
@@ -1,6 +1,6 @@
 """ Example script for predicting columns of tables, demonstrating more advanced usage of fit().
     Note that all settings demonstrated here are just chosen for demonstration purposes (to minimize runtime), and do not represent wise choices to use in practice.
-    To maximize predictive accuracy, we recommend you do NOT specify `hyperparameters` or `hyperparameter_tune_kwargs`, and instead only specify the following fit() arguments: eval_metric=YOUR_METRIC, presets='best_quality'
+    To maximize predictive accuracy, we recommend you do NOT specify `hyperparameters` or `hyperparameter_tune_kwargs`, and instead only specify the arguments presets='best_quality' to fit() and eval_metric=YOUR_METRIC to TabularPredictor()
 """
 
 import autogluon.core as ag


Code review comments:
diff_hunk: @@ -1,6 +1,6 @@
 """ Example script for predicting columns of tables, demonstrating more advanced usage of fit().
     Note that all settings demonstrated here are just chosen for demonstration purposes (to minimize runtime), and do not represent wise choices to use in practice.
-    To maximize predictive accuracy, we recommend you do NOT specify `hyperparameters` or `hyperparameter_tune_kwargs`, and instead only specify the following fit() arguments: eval_metric=YOUR_METRIC, presets='best_quality'
+    To maximize predictive accuracy, we recommend you do NOT specify `hyperparameters` or `hyperparameter_tune_kwargs`, and instead only specify the arguments presets='best_quality' to fit() and eval_metric=YOUR_METRIC to TabularPredictor()
path: examples/tabular/example_advanced_tabular.py
text: @user1:
and instead specify `TabularPredictor(..., eval_metric=YOUR_METRIC).fit(..., presets='best_quality')`

@author:
Done

diff_hunk: @@ -60,7 +60,7 @@ def __init__(self, path: str, problem_type: str, eval_metric=None,
         if not (self.eval_metric.needs_pred or self.eval_metric.needs_quantile):
             logger.log(25, "\tThis metric expects predicted probabilities rather than predicted class labels, so you'll need to use predict_proba() instead of predict()")
 
-        logger.log(20, "\tTo change this, specify the eval_metric argument of fit()")
+        logger.log(20, "\tTo change this, specify the eval_metric argument of Predictor()")
path: core/src/autogluon/core/trainer/abstract_trainer.py
text: @user1:
argument -> parameter

@author:
Done