Repository: alteryx/evalml (python)

Issue:
Clean up objective utils and add to API ref
The `get_objective` and `get_objectives` methods in `evalml/objectives/utils.py` aren't in the API ref yet. Let's add them, and make sure their docstrings make sense.

Also, let's clean up the API for working with the objective utils. I see we have a `print_all_objective_names` util. I'd like us to replace that with an `all_objective_names` method which returns a list. In fact we already have a private method `_all_objectives_dict()` which is a starting point.

So I'd propose we have the following public methods defined for working with objectives:
* `all_objective_names`
* `get_objective`
* `get_objectives`

Pull request description:
Clean up objective utils
### Pull Request Description
Fix #1214

- [x] Add all objective utils to the api reference
- [x] Add functions to display all objective names and all core objective names 
- [x] Rename `get_objectives` to `get_core_objectives` . Also changed the api to return instances as opposed to classes. The idea is that `get_core_objectives` should return a list of objectives that are ready-to-go for a given problem type. Returning classes was needed to return **all** objectives but it is more user friendly to exclude the ones that are domain-specific (`LeadScoring`, `RMSLE`, ...) rather than expecting the user to do it.
- [x] Add `get_default_primary_search_objective` function to make `_DEFAULT_OBJECTIVES` public.
- [x] Clean up fixture names.

-----
*After creating the pull request: in order to pass the **release_notes_updated** check you will need to update the "Future Release" section of* `docs/source/release_notes.rst` *to include this pull request by adding :pr:`123`.*


Patch:
diff --git a/docs/source/api_reference.rst b/docs/source/api_reference.rst
index 4dbeccf0..2e6cf045 100644
--- a/docs/source/api_reference.rst
+++ b/docs/source/api_reference.rst
@@ -51,6 +51,15 @@ AutoML Search Classes
     AutoMLSearch
 
 
+AutoML Utils
+~~~~~~~~~~~~
+.. autosummary::
+    :toctree: generated
+    :nosignatures:
+
+    get_default_primary_search_objective
+
+
 .. currentmodule:: evalml.automl.automl_algorithm
 
 AutoML Algorithm Classes
@@ -335,6 +344,20 @@ Regression Objectives
     RootMeanSquaredLogError
 
 
+Objective Utils
+~~~~~~~~~~~~~~~~~~~~~
+
+.. autosummary::
+    :toctree: generated
+    :nosignatures:
+
+    get_all_objective_names
+    get_core_objectives
+    get_core_objective_names
+    get_non_core_objectives
+    get_objective
+
+
 .. currentmodule:: evalml.problem_types
 
 Problem Types
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index dc8c1105..55b14854 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -16,6 +16,9 @@ Release Notes
         * `DataChecks` can now be parametrized by passing a list of `DataCheck` classes and a parameter dictionary :pr:`1167`
         * Added first CV fold score as validation score in `AutoMLSearch.rankings` :pr:`1221`
         * Updated `flake8` configuration to enable linting on `__init__.py` files :pr:`1234`
+        * Add `get_all_objective_names`, `get_core_objective_names`, and `get_non_core_objectives` util functions :pr:`1230`
+        * All functions in `evaml.objectives.utils` now show up in the api reference :pr:`1230`
+        * Added `get_default_primary_search_objective` to get the default objective for each problem type :pr:`1230`
     * Fixes
         * Updated GitHub URL after migration to Alteryx GitHub org :pr:`1207`
         * Changed Problem Type enum to be more similar to the string name :pr:`1208`
@@ -41,6 +44,7 @@ Release Notes
     **Breaking Changes**
         * DefaultDataChecks now accepts a problem_type parameter that must be specified :pr:`1167`
         * Pipeline's `._transform` method to evaluate all the preprocessing steps of a pipeline has been replaced with `.compute_estimator_features` :pr:`1231`
+        * `get_objectives` has been renamed to `get_core_objectives`. This function will now return a list of valid objective instances :pr:`1230`
 
 
 **v0.13.2 Sep. 17, 2020**
diff --git a/docs/source/user_guide/automl.ipynb b/docs/source/user_guide/automl.ipynb
index 7638ef9f..64a2fa4b 100644
--- a/docs/source/user_guide/automl.ipynb
+++ b/docs/source/user_guide/automl.ipynb
@@ -92,8 +92,8 @@
     "from evalml.problem_types import detect_problem_type\n",
     "\n",
     "y = pd.Series([0, 1, 1, 0, 1, 1])\n",
-    "detect_problem_type(y)"   
-	]
+    "detect_problem_type(y)"
+   ]
   },
   {
    "cell_type": "markdown",
@@ -103,7 +103,26 @@
     "\n",
     "AutoMLSearch takes in an `objective` parameter to determine which `objective` to optimize for. By default, this parameter is set to `auto`, which allows AutoML to choose `LogLossBinary` for binary classification problems, `LogLossMulticlass` for multiclass classification problems, and `R2` for regression problems.\n",
     "\n",
-    "It should be noted that the `objective` parameter is only used in ranking and helping choose the pipelines to iterate over, but is not used to optimize each individual pipeline during fit-time."
+    "It should be noted that the `objective` parameter is only used in ranking and helping choose the pipelines to iterate over, but is not used to optimize each individual pipeline during fit-time.\n",
+    "\n",
+    "To get the default objective for each problem type, you can use the `get_default_primary_search_objective` function."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from evalml.automl import get_default_primary_search_objective\n",
+    "\n",
+    "binary_objective = get_default_primary_search_objective(\"binary\")\n",
+    "multiclass_objective = get_default_primary_search_objective(\"multiclass\")\n",
+    "regression_objective = get_default_primary_search_objective(\"regression\")\n",
+    "\n",
+    "print(binary_objective.name)\n",
+    "print(multiclass_objective.name)\n",
+    "print(regression_objective.name)"
    ]
   },
   {
diff --git a/docs/source/user_guide/objectives.ipynb b/docs/source/user_guide/objectives.ipynb
index a41be0a1..7bf02e59 100644
--- a/docs/source/user_guide/objectives.ipynb
+++ b/docs/source/user_guide/objectives.ipynb
@@ -21,7 +21,7 @@
    "source": [
     "## Core Objectives\n",
     "\n",
-    "Use the `get_objectives` method to get a list of which objectives are included with EvalML for each problem type:"
+    "Use the `get_core_objectives` method to get a list of which objectives are included with EvalML for each problem type:"
    ]
   },
   {
@@ -30,10 +30,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from evalml.objectives import get_objectives\n",
+    "from evalml.objectives import get_core_objectives\n",
     "from evalml.problem_types import ProblemTypes\n",
     "\n",
-    "for objective in get_objectives(ProblemTypes.BINARY):\n",
+    "for objective in get_core_objectives(ProblemTypes.BINARY):\n",
     "    print(objective.name)"
    ]
   },
diff --git a/evalml/automl/__init__.py b/evalml/automl/__init__.py
index 27c93504..59449b6d 100644
--- a/evalml/automl/__init__.py
+++ b/evalml/automl/__init__.py
@@ -1,2 +1,3 @@
 from .automl_search import AutoMLSearch
+from .utils import get_default_primary_search_objective
 from .data_splitters import TrainingValidationSplit
diff --git a/evalml/automl/automl_search.py b/evalml/automl/automl_search.py
index 96c73c2f..b2e4c43b 100644
--- a/evalml/automl/automl_search.py
+++ b/evalml/automl/automl_search.py
@@ -17,6 +17,7 @@ from .pipeline_search_plots import PipelineSearchPlots
 
 from evalml.automl.automl_algorithm import IterativeAlgorithm
 from evalml.automl.data_splitters import TrainingValidationSplit
+from evalml.automl.utils import get_default_primary_search_objective
 from evalml.data_checks import (
     AutoMLDataChecks,
     DataChecks,
@@ -30,21 +31,9 @@ from evalml.exceptions import (
     PipelineScoreError
 )
 from evalml.objectives import (
-    CostBenefitMatrix,
-    FraudCost,
-    LeadScoring,
-    MeanSquaredLogError,
-    Recall,
-    RecallMacro,
-    RecallMicro,
-    RecallWeighted,
-    RootMeanSquaredLogError,
-    get_objective,
-    get_objectives
-)
-from evalml.objectives.utils import (
-    _all_objectives_dict,
-    _print_objectives_in_table
+    get_core_objectives,
+    get_non_core_objectives,
+    get_objective
 )
 from evalml.pipelines import (
     BinaryClassificationPipeline,
@@ -57,7 +46,6 @@ from evalml.pipelines.utils import make_pipeline
 from evalml.problem_types import ProblemTypes, handle_problem_types
 from evalml.tuners import SKOptTuner
 from evalml.utils import convert_to_seconds, get_random_state
-from evalml.utils.gen_utils import classproperty
 from evalml.utils.logger import (
     get_logger,
     log_subtitle,
@@ -77,10 +65,6 @@ class AutoMLSearch:
     # Necessary for "Plotting" documentation, since Sphinx does not work well with instance attributes.
     plot = PipelineSearchPlots
 
-    _DEFAULT_OBJECTIVES = {'binary': 'Log Loss Binary',
-                           'multiclass': 'Log Loss Multiclass',
-                           'regression': 'R2'}
-
     def __init__(self,
                  problem_type=None,
                  objective='auto',
@@ -174,7 +158,7 @@ class AutoMLSearch:
         self.verbose = verbose
         self.optimize_thresholds = optimize_thresholds
         if objective == 'auto':
-            objective = self._DEFAULT_OBJECTIVES[self.problem_type.value]
+            objective = get_default_primary_search_objective(self.problem_type.value)
         objective = get_objective(objective, return_instance=False)
         self.objective = self._validate_objective(objective)
         if self.data_split is not None and not issubclass(self.data_split.__class__, BaseCrossValidator):
@@ -182,7 +166,7 @@ class AutoMLSearch:
         if self.problem_type != self.objective.problem_type:
             raise ValueError("Given objective {} is not compatible with a {} problem.".format(self.objective.name, self.problem_type.value))
         if additional_objectives is None:
-            additional_objectives = [obj for obj in get_objectives(self.problem_type) if obj not in self._objectives_not_allowed_in_automl]
+            additional_objectives = get_core_objectives(self.problem_type)
             # if our main objective is part of default set of objectives for problem_type, remove it
             existing_main_objective = next((obj for obj in additional_objectives if obj.name == self.objective.name), None)
             if existing_main_objective is not None:
@@ -239,7 +223,7 @@ class AutoMLSearch:
         self._baseline_cv_score = None
 
         if _max_batches is not None and _max_batches <= 0:
-            raise ValueError("Parameter max batches must be None or non-negative. Received {max_batches}.")
+            raise ValueError(f"Parameter max batches must be None or non-negative. Received {_max_batches}.")
         self._max_batches = _max_batches
         # This is the default value for IterativeAlgorithm - setting this explicitly makes sure that
         # the behavior of max_batches does not break if IterativeAlgorithm is changed.
@@ -247,21 +231,12 @@ class AutoMLSearch:
 
         self._validate_problem_type()
 
-    @classproperty
-    def _objectives_not_allowed_in_automl(self):
-        return {CostBenefitMatrix, FraudCost, LeadScoring,
-                MeanSquaredLogError, Recall, RecallMacro, RecallMicro, RecallWeighted, RootMeanSquaredLogError}
-
-    @classmethod
-    def print_objective_names_allowed_in_automl(cls):
-        names = [name for name, value in _all_objectives_dict().items() if value not in cls._objectives_not_allowed_in_automl]
-        _print_objectives_in_table(names)
-
     def _validate_objective(self, objective):
+        non_core_objectives = get_non_core_objectives()
         if isinstance(objective, type):
-            if objective in self._objectives_not_allowed_in_automl:
-                raise ValueError(f"{objective.name} is not allowed in AutoML! "
-                                 "Use evalml.automl.AutoMLSearch.print_objective_names_allowed_in_automl() "
+            if objective in non_core_objectives:
+                raise ValueError(f"{objective.name.lower()} is not allowed in AutoML! "
+                                 "Use evalml.objectives.utils.get_core_objective_names()"
                                  "to get all objective names allowed in automl.")
             return objective()
         return objective
diff --git a/evalml/automl/utils.py b/evalml/automl/utils.py
new file mode 100644
index 00000000..996823a5
--- /dev/null
+++ b/evalml/automl/utils.py
@@ -0,0 +1,18 @@
+from evalml.objectives import get_objective
+from evalml.problem_types import handle_problem_types
+
+
+def get_default_primary_search_objective(problem_type):
+    """Get the default primary search objective for a problem type.
+
+    Arguments:
+        problem_type (str or ProblemType): problem type of interest.
+
+    Returns:
+        ObjectiveBase: primary objective instance for the problem type.
+    """
+    problem_type = handle_problem_types(problem_type)
+    objective_name = {'binary': 'Log Loss Binary',
+                      'multiclass': 'Log Loss Multiclass',
+                      'regression': 'R2'}[problem_type.value]
+    return get_objective(objective_name, return_instance=True)
diff --git a/evalml/objectives/__init__.py b/evalml/objectives/__init__.py
index a1af6771..58e479cd 100644
--- a/evalml/objectives/__init__.py
+++ b/evalml/objectives/__init__.py
@@ -42,4 +42,5 @@ from .standard_metrics import (
     RecallMicro,
     RecallWeighted
 )
-from .utils import get_objective, get_objectives, print_all_objective_names
+from .utils import get_objective, get_core_objectives, get_all_objective_names, get_non_core_objectives,\
+    get_core_objective_names
diff --git a/evalml/objectives/utils.py b/evalml/objectives/utils.py
index 9b8c6d13..2ff66454 100644
--- a/evalml/objectives/utils.py
+++ b/evalml/objectives/utils.py
@@ -1,12 +1,26 @@
-from texttable import Texttable
 
 from .objective_base import ObjectiveBase
 
+from evalml import objectives
 from evalml.exceptions import ObjectiveNotFoundError
 from evalml.problem_types import handle_problem_types
 from evalml.utils.gen_utils import _get_subclasses
 
 
+def get_non_core_objectives():
+    """Get non-core objective classes.
+
+    Non-core objectives are objectives that are domain-specific. Users typically need to configure these objectives
+    before using them in AutoMLSearch.
+
+    Returns:
+        List of ObjectiveBase classes
+    """
+    return [objectives.CostBenefitMatrix, objectives.FraudCost, objectives.LeadScoring,
+            objectives.MeanSquaredLogError, objectives.Recall, objectives.RecallMacro, objectives.RecallMicro,
+            objectives.RecallWeighted, objectives.RootMeanSquaredLogError]
+
+
 def _all_objectives_dict():
     all_objectives = _get_subclasses(ObjectiveBase)
     objectives_dict = {}
@@ -17,43 +31,35 @@ def _all_objectives_dict():
     return objectives_dict
 
 
-def _print_objectives_in_table(names):
-    """Print the list of objective names in a table.
+def get_all_objective_names():
+    """Get a list of the names of all objectives.
 
     Returns:
-         None
+        list (str): Objective names
     """
-    def iterate_in_batches(sequence, batch_size):
-        return [sequence[pos:pos + batch_size] for pos in range(0, len(sequence), batch_size)]
-    batch_size = 4
-    table = Texttable()
-    table.set_deco(Texttable.BORDER | Texttable.HLINES | Texttable.VLINES)
-    for row in iterate_in_batches(sorted(names), batch_size):
-        if len(row) < batch_size:
-            row += [""] * (batch_size - len(row))
-        table.add_row(row)
-    print(table.draw())
+    all_objectives_dict = _all_objectives_dict()
+    return list(all_objectives_dict.keys())
 
 
-def print_all_objective_names():
-    """Get all valid objective names in a table.
+def get_core_objective_names():
+    """Get a list of all valid core objectives.
 
     Returns:
-        None
+        list(str): Objective names.
     """
-    all_objectives_dict = _all_objectives_dict()
-    all_names = list(all_objectives_dict.keys())
-    _print_objectives_in_table(all_names)
+    all_objectives = _all_objectives_dict()
+    non_core = get_non_core_objectives()
+    return [name for name, class_name in all_objectives.items() if class_name not in non_core]
 
 
 def get_objective(objective, return_instance=False, **kwargs):
-    """Returns the Objective object of the given objective name
+    """Returns the Objective class corresponding to a given objective name.
 
     Arguments:
         objective (str or ObjectiveBase): Name or instance of the objective class.
         return_instance (bool): Whether to return an instance of the objective. This only applies if objective
             is of type str. Note that the instance will be initialized with default arguments.
-            **kwargs (Any): Any keyword arguments to pass into the objective. Only used when return_instance=True.
+        kwargs (Any): Any keyword arguments to pass into the objective. Only used when return_instance=True.
 
     Returns:
         ObjectiveBase if the parameter objective is of type ObjectiveBase. If objective is instead a valid
@@ -69,7 +75,7 @@ def get_objective(objective, return_instance=False, **kwargs):
         raise TypeError("If parameter objective is not a string, it must be an instance of ObjectiveBase!")
     if objective.lower() not in all_objectives_dict:
         raise ObjectiveNotFoundError(f"{objective} is not a valid Objective! "
-                                     "Use evalml.objectives.print_all_objective_names(allowed_in_automl=False)"
+                                     "Use evalml.objectives.get_all_objective_names()"
                                      "to get a list of all valid objective names. ")
 
     objective_class = all_objectives_dict[objective.lower()]
@@ -83,16 +89,18 @@ def get_objective(objective, return_instance=False, **kwargs):
     return objective_class
 
 
-def get_objectives(problem_type):
-    """Returns all objective classes associated with the given problem type.
+def get_core_objectives(problem_type):
+    """Returns all core objective instances associated with the given problem type.
+
+    Core objectives are designed to work out-of-the-box for any dataset.
 
     Arguments:
         problem_type (str/ProblemTypes): Type of problem
 
     Returns:
-        List of Objectives
+        List of ObjectiveBase instances
     """
     problem_type = handle_problem_types(problem_type)
     all_objectives_dict = _all_objectives_dict()
-    objectives = [obj for obj in all_objectives_dict.values() if obj.problem_type == problem_type]
+    objectives = [obj() for obj in all_objectives_dict.values() if obj.problem_type == problem_type and obj not in get_non_core_objectives()]
     return objectives


Test patch:
diff --git a/evalml/tests/automl_tests/test_automl.py b/evalml/tests/automl_tests/test_automl.py
index 365b335e..359050c5 100644
--- a/evalml/tests/automl_tests/test_automl.py
+++ b/evalml/tests/automl_tests/test_automl.py
@@ -9,7 +9,10 @@ import pytest
 from sklearn.model_selection import StratifiedKFold
 
 from evalml import AutoMLSearch
-from evalml.automl import TrainingValidationSplit
+from evalml.automl import (
+    TrainingValidationSplit,
+    get_default_primary_search_objective
+)
 from evalml.data_checks import (
     DataCheck,
     DataCheckError,
@@ -19,8 +22,14 @@ from evalml.data_checks import (
 from evalml.demos import load_breast_cancer, load_wine
 from evalml.exceptions import AutoMLSearchException, PipelineNotFoundError
 from evalml.model_family import ModelFamily
-from evalml.objectives import FraudCost
-from evalml.objectives.utils import _all_objectives_dict
+from evalml.objectives import (
+    R2,
+    CostBenefitMatrix,
+    FraudCost,
+    LogLossBinary,
+    LogLossMulticlass
+)
+from evalml.objectives.utils import get_core_objectives
 from evalml.pipelines import (
     BinaryClassificationPipeline,
     MulticlassClassificationPipeline,
@@ -964,11 +973,14 @@ def test_error_during_train_test_split(mock_fit, mock_score, mock_train_test_spl
         assert np.isnan(pipeline['score'])
 
 
-@pytest.mark.parametrize("objective_tuple,pipeline_scores,baseline_score",
-                         product(_all_objectives_dict().items(),
+all_objectives = get_core_objectives("binary") + get_core_objectives("multiclass") + get_core_objectives("regression")
+
+
+@pytest.mark.parametrize("objective,pipeline_scores,baseline_score",
+                         product(all_objectives + [CostBenefitMatrix],
                                  [(0.3, 0.4), (np.nan, 0.4), (0.3, np.nan), (np.nan, np.nan)],
                                  [0.1, np.nan]))
-def test_percent_better_than_baseline_in_rankings(objective_tuple, pipeline_scores, baseline_score,
+def test_percent_better_than_baseline_in_rankings(objective, pipeline_scores, baseline_score,
                                                   dummy_binary_pipeline_class, dummy_multiclass_pipeline_class,
                                                   dummy_regression_pipeline_class,
                                                   X_y_binary):
@@ -976,11 +988,6 @@ def test_percent_better_than_baseline_in_rankings(objective_tuple, pipeline_scor
     # Ok to only use binary labels since score and fit methods are mocked
     X, y = X_y_binary
 
-    name, objective = objective_tuple
-
-    if objective in AutoMLSearch._objectives_not_allowed_in_automl and name != "cost benefit matrix":
-        pytest.skip(f"Skipping because {name} is not allowed in automl as a string.")
-
     pipeline_class = {ProblemTypes.BINARY: dummy_binary_pipeline_class,
                       ProblemTypes.MULTICLASS: dummy_multiclass_pipeline_class,
                       ProblemTypes.REGRESSION: dummy_regression_pipeline_class}[objective.problem_type]
@@ -1006,12 +1013,12 @@ def test_percent_better_than_baseline_in_rankings(objective_tuple, pipeline_scor
     Pipeline1.score = mock_score_1
     Pipeline2.score = mock_score_2
 
-    if name == "cost benefit matrix":
+    if objective.name.lower() == "cost benefit matrix":
         automl = AutoMLSearch(problem_type=objective.problem_type, max_iterations=3,
                               allowed_pipelines=[Pipeline1, Pipeline2], objective=objective(0, 0, 0, 0))
     else:
         automl = AutoMLSearch(problem_type=objective.problem_type, max_iterations=3,
-                              allowed_pipelines=[Pipeline1, Pipeline2], objective=name)
+                              allowed_pipelines=[Pipeline1, Pipeline2], objective=objective)
 
     with patch(baseline_pipeline_class + ".score", return_value={objective.name: baseline_score}):
         automl.search(X, y, data_checks=None)
@@ -1079,14 +1086,10 @@ def test_max_batches_plays_nice_with_other_stopping_criteria(mock_fit, mock_scor
 @pytest.mark.parametrize("max_batches", [0, -1, -10, -np.inf])
 def test_max_batches_must_be_non_negative(max_batches):
 
-    with pytest.raises(ValueError, match="Parameter max batches must be None or non-negative. Received {max_batches}."):
+    with pytest.raises(ValueError, match=f"Parameter max batches must be None or non-negative. Received {max_batches}."):
         AutoMLSearch(problem_type="binary", _max_batches=max_batches)
 
 
-def test_can_print_out_automl_objective_names():
-    AutoMLSearch.print_objective_names_allowed_in_automl()
-
-
 def test_data_split_binary(X_y_binary):
     X, y = X_y_binary
     y[:] = 0
@@ -1139,3 +1142,14 @@ def test_data_split_multi(X_y_multi):
 
     y[5] = 0
     automl.search(X, y, data_checks="disabled")
+
+
+def test_get_default_primary_search_objective():
+    assert isinstance(get_default_primary_search_objective("binary"), LogLossBinary)
+    assert isinstance(get_default_primary_search_objective(ProblemTypes.BINARY), LogLossBinary)
+    assert isinstance(get_default_primary_search_objective("multiclass"), LogLossMulticlass)
+    assert isinstance(get_default_primary_search_objective(ProblemTypes.MULTICLASS), LogLossMulticlass)
+    assert isinstance(get_default_primary_search_objective("regression"), R2)
+    assert isinstance(get_default_primary_search_objective(ProblemTypes.REGRESSION), R2)
+    with pytest.raises(KeyError, match="Problem type 'auto' does not exist"):
+        get_default_primary_search_objective("auto")
diff --git a/evalml/tests/automl_tests/test_automl_search_classification.py b/evalml/tests/automl_tests/test_automl_search_classification.py
index 24b921c3..37c6ec9d 100644
--- a/evalml/tests/automl_tests/test_automl_search_classification.py
+++ b/evalml/tests/automl_tests/test_automl_search_classification.py
@@ -16,8 +16,7 @@ from evalml.objectives import (
     Precision,
     PrecisionMicro,
     Recall,
-    get_objective,
-    get_objectives
+    get_objective
 )
 from evalml.pipelines import (
     ModeBaselineBinaryPipeline,
@@ -100,7 +99,7 @@ def test_max_pipelines_deprecation(caplog):
 def test_recall_error(X_y_binary):
     X, y = X_y_binary
     # Recall is a valid objective but it's not allowed in AutoML so a ValueError is expected
-    error_msg = 'Recall is not allowed in AutoML!'
+    error_msg = 'recall is not allowed in AutoML!'
     with pytest.raises(ValueError, match=error_msg):
         AutoMLSearch(problem_type='binary', objective='recall', max_iterations=1)
 
@@ -124,7 +123,7 @@ def test_binary_auto(X_y_binary):
     assert len(np.unique(y_pred)) == 2
 
 
-def test_multi_auto(X_y_multi):
+def test_multi_auto(X_y_multi, multiclass_core_objectives):
     X, y = X_y_multi
     objective = PrecisionMicro()
     automl = AutoMLSearch(problem_type='multiclass', objective=objective, max_iterations=5)
@@ -134,11 +133,10 @@ def test_multi_auto(X_y_multi):
     y_pred = best_pipeline.predict(X)
     assert len(np.unique(y_pred)) == 3
 
-    expected_additional_objectives = [obj() for obj in get_objectives('multiclass') if obj not in automl._objectives_not_allowed_in_automl]
-    objective_in_additional_objectives = next((obj for obj in expected_additional_objectives if obj.name == objective.name), None)
-    expected_additional_objectives.remove(objective_in_additional_objectives)
+    objective_in_additional_objectives = next((obj for obj in multiclass_core_objectives if obj.name == objective.name), None)
+    multiclass_core_objectives.remove(objective_in_additional_objectives)
 
-    for expected, additional in zip(expected_additional_objectives, automl.additional_objectives):
+    for expected, additional in zip(multiclass_core_objectives, automl.additional_objectives):
         assert type(additional) is type(expected)
 
 
diff --git a/evalml/tests/conftest.py b/evalml/tests/conftest.py
index d8e58d37..3888a542 100644
--- a/evalml/tests/conftest.py
+++ b/evalml/tests/conftest.py
@@ -6,9 +6,8 @@ import pytest
 from sklearn import datasets
 from skopt.space import Integer, Real
 
-from evalml.automl import AutoMLSearch
 from evalml.model_family import ModelFamily
-from evalml.objectives.utils import get_objectives
+from evalml.objectives.utils import get_core_objectives
 from evalml.pipelines import (
     BinaryClassificationPipeline,
     MulticlassClassificationPipeline,
@@ -18,8 +17,6 @@ from evalml.pipelines.components import Estimator
 from evalml.pipelines.components.utils import _all_estimators
 from evalml.problem_types import ProblemTypes
 
-_not_allowed_in_automl = AutoMLSearch._objectives_not_allowed_in_automl
-
 
 def create_mock_pipeline(estimator, problem_type):
     if problem_type == ProblemTypes.BINARY:
@@ -245,15 +242,15 @@ def linear_regression_pipeline_class():
 
 
 @pytest.fixture
-def binary_objectives_allowed_in_automl():
-    return [obj() for obj in get_objectives(ProblemTypes.BINARY) if obj not in _not_allowed_in_automl]
+def binary_core_objectives():
+    return get_core_objectives(ProblemTypes.BINARY)
 
 
 @pytest.fixture
-def multiclass_objectives_allowed_in_automl():
-    return [obj() for obj in get_objectives(ProblemTypes.MULTICLASS) if obj not in _not_allowed_in_automl]
+def multiclass_core_objectives():
+    return get_core_objectives(ProblemTypes.MULTICLASS)
 
 
 @pytest.fixture
-def regression_objectives_allowed_in_automl():
-    return [obj() for obj in get_objectives(ProblemTypes.REGRESSION) if obj not in _not_allowed_in_automl]
+def regression_core_objectives():
+    return get_core_objectives(ProblemTypes.REGRESSION)
diff --git a/evalml/tests/model_understanding_tests/test_graphs.py b/evalml/tests/model_understanding_tests/test_graphs.py
index 709b8594..291af8e1 100644
--- a/evalml/tests/model_understanding_tests/test_graphs.py
+++ b/evalml/tests/model_understanding_tests/test_graphs.py
@@ -495,36 +495,36 @@ def test_get_permutation_importance_invalid_objective(X_y_regression, linear_reg
 
 @pytest.mark.parametrize("data_type", ['np', 'pd'])
 def test_get_permutation_importance_binary(X_y_binary, data_type, logistic_regression_binary_pipeline_class,
-                                           binary_objectives_allowed_in_automl):
+                                           binary_core_objectives):
     X, y = X_y_binary
     if data_type == 'pd':
         X = pd.DataFrame(X)
         y = pd.Series(y)
     pipeline = logistic_regression_binary_pipeline_class(parameters={}, random_state=np.random.RandomState(42))
     pipeline.fit(X, y)
-    for objective in binary_objectives_allowed_in_automl:
+    for objective in binary_core_objectives:
         permutation_importance = calculate_permutation_importance(pipeline, X, y, objective)
         assert list(permutation_importance.columns) == ["feature", "importance"]
         assert not permutation_importance.isnull().all().all()
 
 
 def test_get_permutation_importance_multiclass(X_y_multi, logistic_regression_multiclass_pipeline_class,
-                                               multiclass_objectives_allowed_in_automl):
+                                               multiclass_core_objectives):
     X, y = X_y_multi
     pipeline = logistic_regression_multiclass_pipeline_class(parameters={}, random_state=np.random.RandomState(42))
     pipeline.fit(X, y)
-    for objective in multiclass_objectives_allowed_in_automl:
+    for objective in multiclass_core_objectives:
         permutation_importance = calculate_permutation_importance(pipeline, X, y, objective)
         assert list(permutation_importance.columns) == ["feature", "importance"]
         assert not permutation_importance.isnull().all().all()
 
 
 def test_get_permutation_importance_regression(X_y_regression, linear_regression_pipeline_class,
-                                               regression_objectives_allowed_in_automl):
+                                               regression_core_objectives):
     X, y = X_y_regression
     pipeline = linear_regression_pipeline_class(parameters={}, random_state=np.random.RandomState(42))
     pipeline.fit(X, y)
-    for objective in regression_objectives_allowed_in_automl:
+    for objective in regression_core_objectives:
         permutation_importance = calculate_permutation_importance(pipeline, X, y, objective)
         assert list(permutation_importance.columns) == ["feature", "importance"]
         assert not permutation_importance.isnull().all().all()
diff --git a/evalml/tests/objective_tests/test_objectives.py b/evalml/tests/objective_tests/test_objectives.py
index 8a2e2644..6166f84e 100644
--- a/evalml/tests/objective_tests/test_objectives.py
+++ b/evalml/tests/objective_tests/test_objectives.py
@@ -8,9 +8,11 @@ from evalml.objectives import (
     CostBenefitMatrix,
     MulticlassClassificationObjective,
     RegressionObjective,
-    get_objective,
-    get_objectives,
-    print_all_objective_names
+    get_all_objective_names,
+    get_core_objective_names,
+    get_core_objectives,
+    get_non_core_objectives,
+    get_objective
 )
 from evalml.objectives.objective_base import ObjectiveBase
 from evalml.problem_types import ProblemTypes
@@ -78,20 +80,21 @@ def test_get_objective_kwargs():
     assert isinstance(obj, CostBenefitMatrix)
 
 
-def test_can_print_all_objective_names():
-    print_all_objective_names()
+def test_can_get_only_core_and_all_objective_names():
+    all_objective_names = get_all_objective_names()
+    core_objective_names = get_core_objective_names()
+    assert set(all_objective_names).difference(core_objective_names) == {c.name.lower() for c in get_non_core_objectives()}
 
 
-def test_get_objectives_types():
+def test_get_core_objectives_types():
+    assert len(get_core_objectives(ProblemTypes.MULTICLASS)) == 13
+    assert len(get_core_objectives(ProblemTypes.BINARY)) == 7
+    assert len(get_core_objectives(ProblemTypes.REGRESSION)) == 7
 
-    assert len(get_objectives(ProblemTypes.MULTICLASS)) == 16
-    assert len(get_objectives(ProblemTypes.BINARY)) == 11
-    assert len(get_objectives(ProblemTypes.REGRESSION)) == 9
 
-
-def test_objective_outputs(X_y_binary, X_y_multi, binary_objectives_allowed_in_automl,
-                           multiclass_objectives_allowed_in_automl,
-                           regression_objectives_allowed_in_automl):
+def test_objective_outputs(X_y_binary, X_y_multi, binary_core_objectives,
+                           multiclass_core_objectives,
+                           regression_core_objectives):
     _, y_binary_np = X_y_binary
     assert isinstance(y_binary_np, np.ndarray)
     _, y_multi_np = X_y_multi
@@ -102,7 +105,7 @@ def test_objective_outputs(X_y_binary, X_y_multi, binary_objectives_allowed_in_a
     classes = np.unique(y_multi_np)
     y_pred_proba_multi_np = np.concatenate([(y_multi_np == val).astype(float).reshape(-1, 1) for val in classes], axis=1)
 
-    all_objectives = binary_objectives_allowed_in_automl + regression_objectives_allowed_in_automl + multiclass_objectives_allowed_in_automl
+    all_objectives = binary_core_objectives + regression_core_objectives + multiclass_core_objectives
 
     for objective in all_objectives:
         print('Testing objective {}'.format(objective.name))
diff --git a/evalml/tests/objective_tests/test_standard_metrics.py b/evalml/tests/objective_tests/test_standard_metrics.py
index dfbd253f..4a7cd655 100644
--- a/evalml/tests/objective_tests/test_standard_metrics.py
+++ b/evalml/tests/objective_tests/test_standard_metrics.py
@@ -5,7 +5,6 @@ import pandas as pd
 import pytest
 from sklearn.metrics import matthews_corrcoef as sk_matthews_corrcoef
 
-from evalml.automl import AutoMLSearch
 from evalml.objectives import (
     F1,
     MSE,
@@ -33,11 +32,14 @@ from evalml.objectives import (
     RootMeanSquaredError,
     RootMeanSquaredLogError
 )
-from evalml.objectives.utils import _all_objectives_dict
+from evalml.objectives.utils import (
+    _all_objectives_dict,
+    get_non_core_objectives
+)
 
 EPS = 1e-5
 all_automl_objectives = _all_objectives_dict()
-all_automl_objectives = {name: class_() for name, class_ in all_automl_objectives.items() if class_ not in AutoMLSearch._objectives_not_allowed_in_automl}
+all_automl_objectives = {name: class_() for name, class_ in all_automl_objectives.items() if class_ not in get_non_core_objectives()}
 
 
 def test_input_contains_nan():