Repository: alteryx/evalml
Pull request #1452 (2020-11-21 00:26:34)

Update automl to use default of max_batches=1
We have 8 classification estimators and 7 regression estimators now! We should use all of them by default.

Docs changes visible [here](https://evalml.alteryx.com/en/ds_update_max_batches_default/start.html).

codecov[bot]: # [Codecov](https://codecov.io/gh/alteryx/evalml/pull/1452?src=pr&el=h1) Report
> Merging [#1452](https://codecov.io/gh/alteryx/evalml/pull/1452?src=pr&el=desc) (276defe) into [main](https://codecov.io/gh/alteryx/evalml/commit/e2b1030c0067c6d4a759e0634124d0ff8e1e27b6?el=desc) (e2b1030) will **decrease** coverage by `0.1%`.
> The diff coverage is `90.0%`.

[![Impacted file tree graph](https://codecov.io/gh/alteryx/evalml/pull/1452/graphs/tree.svg?width=650&height=150&src=pr&token=JDc0Ib7kYL)](https://codecov.io/gh/alteryx/evalml/pull/1452?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             main    #1452     +/-   ##
=========================================
- Coverage   100.0%   100.0%   -0.0%     
=========================================
  Files         223      223             
  Lines       14930    14933      +3     
=========================================
+ Hits        14923    14924      +1     
- Misses          7        9      +2     
```


| [Impacted Files](https://codecov.io/gh/alteryx/evalml/pull/1452?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [evalml/automl/automl\_search.py](https://codecov.io/gh/alteryx/evalml/pull/1452/diff?src=pr&el=tree#diff-ZXZhbG1sL2F1dG9tbC9hdXRvbWxfc2VhcmNoLnB5) | `99.3% <81.9%> (-0.3%)` | :arrow_down: |
| [evalml/tests/automl\_tests/test\_automl.py](https://codecov.io/gh/alteryx/evalml/pull/1452/diff?src=pr&el=tree#diff-ZXZhbG1sL3Rlc3RzL2F1dG9tbF90ZXN0cy90ZXN0X2F1dG9tbC5weQ==) | `100.0% <100.0%> (ø)` | |
| [.../automl\_tests/test\_automl\_search\_classification.py](https://codecov.io/gh/alteryx/evalml/pull/1452/diff?src=pr&el=tree#diff-ZXZhbG1sL3Rlc3RzL2F1dG9tbF90ZXN0cy90ZXN0X2F1dG9tbF9zZWFyY2hfY2xhc3NpZmljYXRpb24ucHk=) | `100.0% <100.0%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/alteryx/evalml/pull/1452?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/alteryx/evalml/pull/1452?src=pr&el=footer). Last update [e2b1030...276defe](https://codecov.io/gh/alteryx/evalml/pull/1452?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Patch:
diff --git a/docs/source/demos/fraud.ipynb b/docs/source/demos/fraud.ipynb
index 8094ce89..d2211027 100644
--- a/docs/source/demos/fraud.ipynb
+++ b/docs/source/demos/fraud.ipynb
@@ -107,7 +107,7 @@
     "automl = AutoMLSearch(problem_type='binary', \n",
     "                      objective=fraud_objective,\n",
     "                      additional_objectives=['auc', 'f1', 'precision'],\n",
-    "                      max_iterations=5,\n",
+    "                      max_batches=1,\n",
     "                      optimize_thresholds=True)\n",
     "\n",
     "automl.search(X_train, y_train)"
@@ -217,7 +217,7 @@
     "automl_auc = AutoMLSearch(problem_type='binary',\n",
     "                          objective='auc',\n",
     "                          additional_objectives=['f1', 'precision'],\n",
-    "                          max_iterations=5,\n",
+    "                          max_batches=1,\n",
     "                          optimize_thresholds=True)\n",
     "\n",
     "automl_auc.search(X_train, y_train)"
diff --git a/docs/source/demos/lead_scoring.ipynb b/docs/source/demos/lead_scoring.ipynb
index f0573ed4..202a4adf 100644
--- a/docs/source/demos/lead_scoring.ipynb
+++ b/docs/source/demos/lead_scoring.ipynb
@@ -129,7 +129,7 @@
     "automl = AutoMLSearch(problem_type='binary',\n",
     "                      objective=lead_scoring_objective,\n",
     "                      additional_objectives=['auc'],\n",
-    "                      max_iterations=5,\n",
+    "                      max_batches=1,\n",
     "                      optimize_thresholds=True)\n",
     "\n",
     "automl.search(X_train, y_train)"
@@ -239,7 +239,7 @@
     "automl_auc = evalml.AutoMLSearch(problem_type='binary',\n",
     "                                 objective='auc',\n",
     "                                 additional_objectives=[],\n",
-    "                                 max_iterations=5,\n",
+    "                                 max_batches=1,\n",
     "                                 optimize_thresholds=True)\n",
     "\n",
     "automl_auc.search(X_train, y_train)"
diff --git a/docs/source/demos/text_input.ipynb b/docs/source/demos/text_input.ipynb
index 4c5ea19e..f8a5448e 100644
--- a/docs/source/demos/text_input.ipynb
+++ b/docs/source/demos/text_input.ipynb
@@ -118,7 +118,7 @@
    "outputs": [],
    "source": [
     "automl = AutoMLSearch(problem_type='binary',\n",
-    "                      max_iterations=5,\n",
+    "                      max_batches=1,\n",
     "                      optimize_thresholds=True)\n",
     "\n",
     "automl.search(X_train_dt, y_train_dc)"
@@ -259,7 +259,7 @@
    "outputs": [],
    "source": [
     "automl_no_text = AutoMLSearch(problem_type='binary',\n",
-    "                      max_iterations=5,\n",
+    "                      max_batches=1,\n",
     "                      optimize_thresholds=True)\n",
     "\n",
     "automl_no_text.search(X_train_categorical, y_train_dc)"
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index a37de40a..cfc121f8 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -31,6 +31,7 @@ Release Notes
         * Reverted changes from :pr:`1337` :pr:`1409`
         * Updated data checks to return dictionary of warnings and errors instead of a list :pr:`1448`
         * Updated ``AutoMLSearch`` to pass ``Woodwork`` data structures to every pipeline (instead of pandas DataFrames) :pr:`1450`
+        * Update ``AutoMLSearch`` to default to ``max_batches=1`` instead of ``max_iterations=5`` :pr:`1452`
     * Documentation Changes
         * Added description of CLA to contributing guide, updated description of draft PRs :pr:`1402`
         * Updated documentation to include all data checks, ``DataChecks``, and usage of data checks in AutoML :pr:`1412`
diff --git a/docs/source/start.ipynb b/docs/source/start.ipynb
index c847f2fa..8d6c06b2 100644
--- a/docs/source/start.ipynb
+++ b/docs/source/start.ipynb
@@ -48,7 +48,7 @@
     "\n",
     "Below EvalML utilizes Bayesian optimization (EvalML's default optimizer) to search and find the best pipeline defined by the given objective.\n",
     "\n",
-    "EvalML provides a number of parameters to control the search process. `max_iterations` is one of the parameters which controls the stopping criterion for the AutoML search. It indicates the maximum number of pipelines to train and evaluate. In this example, `max_iterations` is set to 5.\n",
+    "EvalML provides a number of parameters to control the search process. `max_batches` is one of the parameters which controls the stopping criterion for the AutoML search. It indicates the maximum number of rounds of AutoML to evaluate, where each round may train and score a variable number of pipelines. In this example, `max_batches` is set to 1.\n",
     "\n",
     "** Graphing methods, like AutoMLSearch, on Jupyter Notebook and Jupyter Lab require [ipywidgets](https://ipywidgets.readthedocs.io/en/latest/user_install.html) to be installed.\n",
     "\n",
@@ -61,7 +61,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "automl = AutoMLSearch(problem_type=\"binary\", objective=\"f1\", max_iterations=5)"
+    "automl = AutoMLSearch(problem_type=\"binary\", objective=\"f1\", max_batches=1)"
    ]
   },
   {
diff --git a/docs/source/user_guide/automl.ipynb b/docs/source/user_guide/automl.ipynb
index ea42c12f..559c51c5 100644
--- a/docs/source/user_guide/automl.ipynb
+++ b/docs/source/user_guide/automl.ipynb
@@ -74,7 +74,7 @@
    "source": [
     "The AutoML search will log its progress, reporting each pipeline and parameter set evaluated during the search.\n",
     "\n",
-    "There are a number of mechanisms to control the AutoML search time. One way is to set the maximum number of candidate models to be evaluated during AutoML using `max_iterations`. By default, AutoML will search a fixed number of iterations and parameter pairs (`max_iterations=5`). The first pipeline to be evaluated will always be a baseline model representing a trivial solution. "
+    "There are a number of mechanisms to control the AutoML search time. One way is to set the `max_batches` parameter which controls the maximum number of rounds of AutoML to evaluate, where each round may train and score a variable number of pipelines. Another way is to set the `max_iterations` parameter which controls the maximum number of candidate models to be evaluated during AutoML. By default, AutoML will search for a single batch. The first pipeline to be evaluated will always be a baseline model representing a trivial solution. "
    ]
   },
   {
diff --git a/evalml/automl/automl_search.py b/evalml/automl/automl_search.py
index 850b330f..97344dbb 100644
--- a/evalml/automl/automl_search.py
+++ b/evalml/automl/automl_search.py
@@ -205,22 +205,21 @@ class AutoMLSearch:
         additional_objectives = [self._validate_objective(obj) for obj in additional_objectives]
         self.additional_objectives = additional_objectives
 
-        if max_time is None or isinstance(max_time, (int, float)):
-            self.max_time = max_time
-        elif isinstance(max_time, str):
-            self.max_time = convert_to_seconds(max_time)
-        else:
-            raise TypeError("max_time must be a float, int, or string. Received a {}.".format(type(max_time)))
-
-        if max_batches is not None and max_batches <= 0:
-            raise ValueError(f"Parameter max batches must be None or non-negative. Received {max_batches}.")
+        if not isinstance(max_time, (int, float, str, type(None))):
+            raise TypeError(f"Parameter max_time must be a float, int, string or None. Received {type(max_time)} with value {str(max_time)}..")
+        if isinstance(max_time, (int, float)) and max_time < 0:
+            raise ValueError(f"Parameter max_time must be None or non-negative. Received {max_time}.")
+        if max_batches is not None and max_batches < 0:
+            raise ValueError(f"Parameter max_batches must be None or non-negative. Received {max_batches}.")
+        if max_iterations is not None and max_iterations < 0:
+            raise ValueError(f"Parameter max_iterations must be None or non-negative. Received {max_iterations}.")
+        self.max_time = convert_to_seconds(max_time) if isinstance(max_time, str) else max_time
+        self.max_iterations = max_iterations
         self.max_batches = max_batches
         self._pipelines_per_batch = _pipelines_per_batch
-
-        self.max_iterations = max_iterations
         if not self.max_iterations and not self.max_time and not self.max_batches:
-            self.max_iterations = 5
-            logger.info("Using default limit of max_iterations=5.\n")
+            self.max_batches = 1
+            logger.info("Using default limit of max_batches=1.\n")
 
         if patience and (not isinstance(patience, int) or patience < 0):
             raise ValueError("patience value must be a positive integer. Received {} instead".format(patience))
@@ -287,6 +286,7 @@ class AutoMLSearch:
             f"Objective: {get_objective(self.objective).name}\n"
             f"Max Time: {self.max_time}\n"
             f"Max Iterations: {self.max_iterations}\n"
+            f"Max Batches: {self.max_batches}\n"
             f"Allowed Pipelines: \n{_print_list(self.allowed_pipelines or [])}\n"
             f"Patience: {self.patience}\n"
             f"Tolerance: {self.tolerance}\n"


Test patch:
diff --git a/evalml/tests/automl_tests/test_automl.py b/evalml/tests/automl_tests/test_automl.py
index 23b6aa36..2894b731 100644
--- a/evalml/tests/automl_tests/test_automl.py
+++ b/evalml/tests/automl_tests/test_automl.py
@@ -156,8 +156,9 @@ def test_pipeline_limits(mock_fit_binary, mock_score_binary,
     automl = AutoMLSearch(problem_type=automl_type)
     automl.search(X, y)
     out = caplog.text
-    assert "Using default limit of max_iterations=5." in out
-    assert len(automl.results['pipeline_results']) <= 5
+    assert "Using default limit of max_batches=1." in out
+    assert "Searching up to 1 batches for a total of" in out
+    assert len(automl.results['pipeline_results']) > 5
 
     caplog.clear()
     automl = AutoMLSearch(problem_type=automl_type, max_time=1e-16)
@@ -1324,7 +1325,7 @@ def test_max_batches_works(mock_pipeline_fit, mock_score, mock_regression_fit, m
     ensemble_nth_batch = len(automl.allowed_pipelines) + 1
 
     if max_batches is None:
-        n_results = 5
+        n_results = len(automl.allowed_pipelines) + 1
         max_batches = 1
         # _automl_algorithm will include all allowed_pipelines in the first batch even
         # if they are not searched over. That is why n_automl_pipelines does not equal
@@ -1402,7 +1403,7 @@ def test_max_batches_plays_nice_with_other_stopping_criteria(mock_fit, mock_scor
     # Use the old default when all are None
     automl = AutoMLSearch(problem_type="binary", objective="Log Loss Binary")
     automl.search(X, y, data_checks=None)
-    assert len(automl.results["pipeline_results"]) == 5
+    assert len(automl.results["pipeline_results"]) == len(get_estimators(problem_type='binary')) + 1
 
     # Use max_iterations when both max_iterations and max_batches are set
     automl = AutoMLSearch(problem_type="binary", objective="Log Loss Binary", max_batches=10,
@@ -1416,13 +1417,23 @@ def test_max_batches_plays_nice_with_other_stopping_criteria(mock_fit, mock_scor
     assert len(automl.results["pipeline_results"]) == 4
 
 
-@pytest.mark.parametrize("max_batches", [0, -1, -10, -np.inf])
+@pytest.mark.parametrize("max_batches", [-1, -10, -np.inf])
 def test_max_batches_must_be_non_negative(max_batches):
-
-    with pytest.raises(ValueError, match=f"Parameter max batches must be None or non-negative. Received {max_batches}."):
+    with pytest.raises(ValueError, match=f"Parameter max_batches must be None or non-negative. Received {max_batches}."):
         AutoMLSearch(problem_type="binary", max_batches=max_batches)
 
 
+def test_stopping_criterion_bad():
+    with pytest.raises(TypeError, match=r"Parameter max_time must be a float, int, string or None. Received <class 'tuple'> with value \('test',\)."):
+        AutoMLSearch(problem_type="binary", max_time=('test',))
+    with pytest.raises(ValueError, match=f"Parameter max_batches must be None or non-negative. Received -1."):
+        AutoMLSearch(problem_type="binary", max_batches=-1)
+    with pytest.raises(ValueError, match=f"Parameter max_time must be None or non-negative. Received -1."):
+        AutoMLSearch(problem_type="binary", max_time=-1)
+    with pytest.raises(ValueError, match=f"Parameter max_iterations must be None or non-negative. Received -1."):
+        AutoMLSearch(problem_type="binary", max_iterations=-1)
+
+
 def test_data_split_binary(X_y_binary):
     X, y = X_y_binary
     y[:] = 0
@@ -1704,7 +1715,8 @@ def test_automl_error_callback(mock_fit, mock_score, X_y_binary, caplog):
     automl.search(X, y)
     assert "AutoML search encountered an exception: all your model are belong to us" in caplog.text
     assert "fit" in caplog.text  # Check stack trace logged
-    assert len(automl._results['errors']) == 15  # 5 iterations, 3 folds each
+    # first automl batch, times 3 for 3-fold cross validation
+    assert len(automl._results['errors']) == (1 + len(get_estimators(problem_type='binary'))) * 3
     for e in automl._results['errors']:
         assert str(e) == msg
 
diff --git a/evalml/tests/automl_tests/test_automl_search_classification.py b/evalml/tests/automl_tests/test_automl_search_classification.py
index 52a7e2a9..0cc6fadc 100644
--- a/evalml/tests/automl_tests/test_automl_search_classification.py
+++ b/evalml/tests/automl_tests/test_automl_search_classification.py
@@ -297,7 +297,7 @@ def test_max_time_units():
     with pytest.raises(AssertionError, match="Invalid unit. Units must be hours, mins, or seconds. Received 'year'"):
         AutoMLSearch(problem_type='binary', objective='F1', max_time='30 years')
 
-    with pytest.raises(TypeError, match="max_time must be a float, int, or string. Received a <class 'tuple'>."):
+    with pytest.raises(TypeError, match="Parameter max_time must be a float, int, string or None. Received <class 'tuple'> with value \\(30, 'minutes'\\)."):
         AutoMLSearch(problem_type='binary', objective='F1', max_time=(30, 'minutes'))