Repository: alteryx/evalml (python)

Issue:
Ability to freeze specific hyperparameters
**Description**
There are cases where we'll want to fix certain pipeline parameters to a predetermined value, thus excluding them from the search.

Today, in order to do this, we'd have to define a new pipeline subclass which updates `custom_hyperparameters` to set only one allowed value for the parameters being held constant. Then we'd have to exclude the original pipeline from automl and add the new subclass to automl. This should work fine but feels heavy-handed and hard to use.

**Options**
* Don't do anything; the way to do this is described above
* Same but add a convenience method to wrap the above
* A better way to support this would be to allow users to specify which parameters they want to freeze as an input to the automl search. We wouldn't need to create any subclasses or re-register any pipelines with automl. Instead, we could upgrade our use of the `Tuner` class to be able to exclude the specified parameters from the search space.

**Proposal**
The last option feels more sustainable. What would go into that:
* Add a `pipelines_freeze_parameters` parameter (name TBD) to `AutoSearchBase.__init__`. This could accept a map with partial parameters dicts for each pipeline:
```
{
    'Logistic Regression Pipeline': {
        'Simple Imputer': {
            'imputation_strategy': 'constant',
            'fill_value': 42
        }
    },
    ... # could include params from other pipelines
}
```
* When `AutoSearchBase` creates the tuners for each pipeline during init, it will pass that pipeline's frozen parameter dict to the tuner class
* Update `Tuner` to have an optional init arg `freeze_parameters`. If specified, when the tuner creates the search space, it will exclude the frozen parameters from the space. But when the tuner proposes a new set of parameters to evaluate, the proposal will include the frozen value(s).



Pull request:
Freeze Hyperparameters for AutoMLSearch
fix #767 

Right now, the implementation assumes that the user includes the 'default' values of the components in the hyperparameter dict. We catch the error that AutoML throws and warn the users if this default value isn't in the provided hyperparameter ranges. Hyperparameter validation would be tracked through [this issue](https://github.com/alteryx/evalml/issues/561). 

Since we have no hyperparameter validation as of yet, if a user creates a new pipeline with EvalML components, redefines a  hyperparameter range that doesn't include the default value, and calls `AutoMLSearch` on this pipeline, `AutoMLSearch` will raise the error that the default values aren't in the search space. This will ideally be addressed in the issue above. Additionally, since we don't do hyperparam validation, we don't catch/throw an error if a user misspells a component name, which will allow `AutoMLSearch` to run with no errors

```python
custom_hyperparameters = {
        # misspell `Imputer'
        "Inpute": {
            "numeric_impute_strategy": ["mean"]
        }
 }
# will run with no errors
make_pipelines(X, y, estimator, 'binary', custom_hyperparameters)
# will run and compute with original `Imputer` hyperparameters, and quietly return the results
AutoMLSearch(problem_type='binary', allowed_pipelines=[make_pipelines]).search(X, y) 
```

This can be fixed once hyperparameter validation is implemented in the before-mentioned issue. 

Docs [here](https://evalml.alteryx.com/en/bc_767_hyperparameters/user_guide/automl.html#Limiting-the-Hyperparameter-Search-Space)

Patch:
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index 6598ea40..0d29a656 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -3,6 +3,7 @@ Release Notes
 
 **Future Releases**
     * Enhancements
+        * Added ability to freeze hyperparameters for AutoMLSearch :pr:`1284`
     * Fixes
     * Changes
     * Documentation Changes
diff --git a/docs/source/user_guide/automl.ipynb b/docs/source/user_guide/automl.ipynb
index 93ef0d6c..2d7d9362 100644
--- a/docs/source/user_guide/automl.ipynb
+++ b/docs/source/user_guide/automl.ipynb
@@ -244,6 +244,52 @@
     "print(best_pipeline.parameters)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Limiting the AutoML Search Space\n",
+    "The AutoML search algorithm first trains each component in the pipeline with their default values. After the first iteration, it then tweaks the parameters of these components using the pre-defined hyperparameter ranges that these components have. To limit the search over certain hyperparameter ranges, use `make_pipeline` to define a pipeline with a custom hyperparameter range. Hyperparameter ranges can be found through the [API reference](https://evalml.alteryx.com/en/stable/api_reference.html) for each component. Note: The default value of the component must be included in any specified hyperparameter range for AutoMLSearch to succeed. Additionally, the parameter value must be specified as a list, even for just one value."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from evalml import AutoMLSearch\n",
+    "from evalml.demos import load_fraud\n",
+    "from evalml.pipelines.components.utils import get_estimators\n",
+    "from evalml.model_family import ModelFamily\n",
+    "from evalml.pipelines.utils import make_pipeline\n",
+    "import woodwork as ww\n",
+    "\n",
+    "X, y = load_fraud()\n",
+    "\n",
+    "# example of setting parameter to just one value\n",
+    "custom_hyperparameters = {'Imputer': {\n",
+    "    'numeric_impute_strategy': ['mean']\n",
+    "}}\n",
+    "\n",
+    "# limit the numeric impute strategy to include only `mean` and `median`\n",
+    "# `mean` is the default value for this argument, and it needs to be included in the specified hyperparameter range\n",
+    "custom_hyperparameters = {'Imputer': {\n",
+    "    'numeric_impute_strategy': ['mean', 'median']\n",
+    "}}\n",
+    "\n",
+    "# AutoMLSearch uses woodwork, so we want to use ww to convert the appropriate types when making pipelines\n",
+    "# ww changes the original X and y data, so we pass that instead of X_dt, y_dc\n",
+    "X_dt = ww.DataTable(X)\n",
+    "y_dc = ww.DataColumn(y)\n",
+    "\n",
+    "estimators = get_estimators('binary', [ModelFamily.EXTRA_TREES])\n",
+    "pipelines_with_custom_hyperparameters = [make_pipeline(X, y, estimator, 'binary', custom_hyperparameters) for estimator in estimators]\n",
+    "automl = AutoMLSearch(problem_type='binary', allowed_pipelines=pipelines_with_custom_hyperparameters)\n",
+    "automl.search(X, y)\n",
+    "automl.best_pipeline.hyperparameters"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
diff --git a/evalml/automl/automl_algorithm/iterative_algorithm.py b/evalml/automl/automl_algorithm/iterative_algorithm.py
index 5ded8793..a752f07e 100644
--- a/evalml/automl/automl_algorithm/iterative_algorithm.py
+++ b/evalml/automl/automl_algorithm/iterative_algorithm.py
@@ -95,7 +95,16 @@ class IterativeAlgorithm(AutoMLAlgorithm):
             pipeline (PipelineBase): The trained pipeline object which was used to compute the score.
         """
         if pipeline.model_family != ModelFamily.ENSEMBLE:
-            super().add_result(score_to_minimize, pipeline)
+            if self.batch_number == 1:
+                try:
+                    super().add_result(score_to_minimize, pipeline)
+                except ValueError as e:
+                    if 'is not within the bounds of the space' in str(e):
+                        raise ValueError("Default parameters for components in pipeline {} not in the hyperparameter ranges: {}".format(pipeline.name, e))
+                    else:
+                        raise(e)
+            else:
+                super().add_result(score_to_minimize, pipeline)
         if self.batch_number == 1:
             self._first_batch_results.append((score_to_minimize, pipeline.__class__))
 
diff --git a/evalml/pipelines/utils.py b/evalml/pipelines/utils.py
index 677658ea..0d59f0d0 100644
--- a/evalml/pipelines/utils.py
+++ b/evalml/pipelines/utils.py
@@ -81,7 +81,7 @@ def _get_pipeline_base_class(problem_type):
         return RegressionPipeline
 
 
-def make_pipeline(X, y, estimator, problem_type, text_columns=None):
+def make_pipeline(X, y, estimator, problem_type, custom_hyperparameters=None, text_columns=None):
     """Given input data, target data, an estimator class and the problem type,
         generates a pipeline class with a preprocessing chain which was recommended based on the inputs.
         The pipeline will be a subclass of the appropriate pipeline base class for the specified problem_type.
@@ -91,6 +91,8 @@ def make_pipeline(X, y, estimator, problem_type, text_columns=None):
         y (pd.Series): The target data of length [n_samples]
         estimator (Estimator): Estimator for pipeline
         problem_type (ProblemTypes or str): Problem type for pipeline to generate
+        custom_hyperparameters (dictionary): Dictionary of custom hyperparameters,
+            with component name as key and dictionary of parameters as the value
         text_columns (list): feature names which should be treated as text features. Defaults to None.
 
     Returns:
@@ -103,7 +105,10 @@ def make_pipeline(X, y, estimator, problem_type, text_columns=None):
     preprocessing_components = _get_preprocessing_components(X, y, problem_type, text_columns, estimator)
     complete_component_graph = preprocessing_components + [estimator]
 
-    hyperparameters = None
+    if custom_hyperparameters and not isinstance(custom_hyperparameters, dict):
+        raise ValueError(f"if custom_hyperparameters provided, must be dictionary. Received {type(custom_hyperparameters)}")
+
+    hyperparameters = custom_hyperparameters
     if not isinstance(X, pd.DataFrame):
         X = pd.DataFrame(X)
 


Test patch:
diff --git a/evalml/tests/automl_tests/test_automl.py b/evalml/tests/automl_tests/test_automl.py
index c61607d6..f6f6845d 100644
--- a/evalml/tests/automl_tests/test_automl.py
+++ b/evalml/tests/automl_tests/test_automl.py
@@ -1430,6 +1430,89 @@ def test_get_default_primary_search_objective():
         get_default_primary_search_objective("auto")
 
 
+@patch('evalml.tuners.skopt_tuner.SKOptTuner.add')
+def test_iterative_algorithm_pipeline_hyperparameters_make_pipeline_other_errors(mock_add, X_y_multi):
+    X, y = X_y_multi
+    custom_hyperparameters = {
+        "Imputer": {
+            "numeric_impute_strategy": ["most_frequent", "mean"]
+        }
+    }
+    estimators = get_estimators('multiclass', [ModelFamily.EXTRA_TREES])
+
+    pipelines = [make_pipeline(X, y, estimator, 'multiclass', custom_hyperparameters) for estimator in estimators]
+    automl = AutoMLSearch(problem_type='multiclass', allowed_pipelines=pipelines)
+
+    mock_add.side_effect = ValueError("Alternate error that can be thrown")
+    with pytest.raises(ValueError) as error:
+        automl.search(X, y)
+    assert "Alternate error that can be thrown" in str(error.value)
+    assert "Default parameters for components" not in str(error.value)
+
+
+@patch('evalml.pipelines.BinaryClassificationPipeline.score')
+@patch('evalml.pipelines.BinaryClassificationPipeline.fit')
+def test_iterative_algorithm_pipeline_hyperparameters_make_pipeline_errors(mock_fit, mock_score, X_y_multi):
+    X, y = X_y_multi
+    invalid_custom_hyperparameters = {
+        "Imputer": {
+            "numeric_impute_strategy": ["most_frequent", "median"]
+        }
+    }
+    larger_invalid = {
+        "Imputer": {
+            "numeric_impute_strategy": ["most_frequent", "mean"]
+        },
+        "Extra Trees Classifier": {
+            "max_depth": [4, 5, 6, 7],
+            "max_features": ["sqrt", "log2"]
+        }
+    }
+    estimators = get_estimators('multiclass', [ModelFamily.EXTRA_TREES])
+
+    invalid_pipelines = [make_pipeline(X, y, estimator, 'multiclass', invalid_custom_hyperparameters) for estimator in estimators]
+    automl = AutoMLSearch(problem_type='multiclass', allowed_pipelines=invalid_pipelines)
+    with pytest.raises(ValueError, match="Default parameters for components"):
+        automl.search(X, y)
+
+    invalid_pipelines = [make_pipeline(X, y, estimator, 'multiclass', larger_invalid) for estimator in estimators]
+    automl = AutoMLSearch(problem_type='multiclass', allowed_pipelines=invalid_pipelines)
+    with pytest.raises(ValueError, match="Default parameters for components"):
+        automl.search(X, y)
+
+
+@patch('evalml.pipelines.BinaryClassificationPipeline.score')
+@patch('evalml.pipelines.BinaryClassificationPipeline.fit')
+def test_iterative_algorithm_pipeline_hyperparameters_make_pipeline(mock_fit, mock_score, X_y_multi):
+    X, y = X_y_multi
+    custom_hyperparameters = {
+        "Imputer": {
+            "numeric_impute_strategy": ["mean"]
+        }
+    }
+    larger_custom = {
+        "Imputer": {
+            "numeric_impute_strategy": ["most_frequent", "mean"]
+        },
+        "Extra Trees Classifier": {
+            "max_depth": [4, 5, 6, 7],
+            "max_features": ["auto", "log2"]
+        }
+    }
+    estimators = get_estimators('multiclass', [ModelFamily.EXTRA_TREES])
+    pipelines = [make_pipeline(X, y, estimator, 'multiclass', custom_hyperparameters) for estimator in estimators]
+
+    automl = AutoMLSearch(problem_type='multiclass', allowed_pipelines=pipelines)
+    automl.search(X, y)
+    assert automl.best_pipeline.hyperparameters['Imputer']['numeric_impute_strategy'] == ["mean"]
+
+    invalid_pipelines = [make_pipeline(X, y, estimator, 'multiclass', larger_custom) for estimator in estimators]
+    automl = AutoMLSearch(problem_type='multiclass', allowed_pipelines=invalid_pipelines)
+    automl.search(X, y)
+
+    assert automl.best_pipeline.hyperparameters['Imputer']['numeric_impute_strategy'] == ["most_frequent", "mean"]
+
+
 @patch('evalml.pipelines.BinaryClassificationPipeline.score')
 @patch('evalml.pipelines.BinaryClassificationPipeline.fit')
 def test_automl_ensembling_false(mock_fit, mock_score, X_y_binary):
diff --git a/evalml/tests/pipeline_tests/test_pipelines.py b/evalml/tests/pipeline_tests/test_pipelines.py
index ddd6ab01..9a4e6049 100644
--- a/evalml/tests/pipeline_tests/test_pipelines.py
+++ b/evalml/tests/pipeline_tests/test_pipelines.py
@@ -1446,6 +1446,38 @@ def test_pipeline_repr(pipeline_class):
     assert repr(pipeline_with_nan_parameters) == expected_repr
 
 
+def test_make_pipeline_error():
+    X = pd.DataFrame([[0, 1], [1, 0]])
+    y = pd.Series([1, 0])
+    estimators = get_estimators(problem_type="binary")
+    custom_hyperparameters = [{"Imputer": {"numeric_imput_strategy": ["median"]}}, {"One Hot Encoder": {"value1": ["value2"]}}]
+
+    for estimator in estimators:
+        with pytest.raises(ValueError, match="if custom_hyperparameters provided, must be dictionary"):
+            make_pipeline(X, y, estimator, "binary", custom_hyperparameters)
+
+
+@pytest.mark.parametrize("problem_type", [ProblemTypes.BINARY, ProblemTypes.MULTICLASS, ProblemTypes.REGRESSION])
+def test_make_pipeline_custom_hyperparameters(problem_type):
+    X = pd.DataFrame({"all_null": [np.nan, np.nan, np.nan, np.nan, np.nan],
+                      "categorical": ["a", "b", "a", "c", "c"],
+                      "some dates": pd.date_range('2000-02-03', periods=5, freq='W')})
+    custom_hyperparameters = {'Imputer': {
+        'numeric_impute_strategy': ['median']
+    }}
+
+    y = pd.Series([0, 0, 1, 0, 0])
+    estimators = get_estimators(problem_type=problem_type)
+
+    for estimator_class in estimators:
+        for problem_type in estimator_class.supported_problem_types:
+            pipeline = make_pipeline(X, y, estimator_class, problem_type, custom_hyperparameters)
+            assert pipeline.custom_hyperparameters == custom_hyperparameters
+
+            pipeline2 = make_pipeline(X, y, estimator_class, problem_type)
+            assert not pipeline2.custom_hyperparameters
+
+
 def test_generate_code_pipeline_errors():
     class MockBinaryPipeline(BinaryClassificationPipeline):
         name = "Mock Binary Pipeline"