Repository: alteryx/evalml (python)

Issue:
AutoML: pass random_state to the created pipeline instances
**Problem**
I was digging through the code with @piguy314159265 and [we noticed that `IterativeAlgorithm.next_batch` doesn't initialize pipeline instances with a `random_state` value.](https://github.com/FeatureLabs/evalml/blob/main/evalml/automl/automl_algorithm/iterative_algorithm.py#L60)

**Symptom**
In short, artificially decreased variance in the modeling results.

Suppose we call automl search two times, the first time with the default `random_state` of 0, and the second time with a custom value, like 42. The two calls will generate different data splits, and the automl tuners (after the first batch) could select different pipelines and generate different parameters for those pipelines. That's fine. But, for the first batch at least, the pipelines which use default parameters will run *with the same random seed both times*.

Most users wouldn't notice this. However, it means some of our performance testing results may have higher variance than our results to date have reported. Our perf tests are supposed to run each trial with a different random seed. After this issue is fixed, the pipelines will also run with a different random seed.

**Fix**
[Update IterativeAlgorithm](https://github.com/FeatureLabs/evalml/blob/main/evalml/automl/automl_algorithm/iterative_algorithm.py#L60) to pass down the random seed when creating pipeline instances:
```python
pipeline_class(parameters=self._transform_parameters(pipeline_class, proposed_parameters), random_state=self._random_state)
```

Once we do this, we should do some performance testing to see what the impact is. My recommendation: 20 trials each on the small datasets yaml. Note we also have to resolve [a similar issue in the perf test code](https://github.com/FeatureLabs/evalml_looking_glass/issues/98) before we can test this.

Pull request:
Passing random state to pipelines created by IterativeAlgorithm next_batch
### Pull Request Description
Fixes #1181 . Also noticed a bug where `PipelineBase.clone` uses random_state = 0 as the default - which means the random state was unexpectedly being changed before fit/score.

-----
*After creating the pull request: in order to pass the **release_notes_updated** check you will need to update the "Future Release" section of* `docs/source/release_notes.rst` *to include this pull request by adding :pr:`123`.*


Patch:
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index 650f0996..65b266cd 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -27,6 +27,7 @@ Release Notes
         * Fixed LightGBM warning messages during AutoMLSearch :pr:`1342`
         * Fix warnings thrown during AutoMLSearch in ``HighVarianceCVDataCheck`` :pr:`1346`
         * Fixed bug where TrainingValidationSplit would return invalid location indices for dataframes with a custom index :pr:`1348`
+        * Fixed bug where the AutoMLSearch ``random_state`` was not being passed to the created pipelines :pr:`1321`
     * Changes
         * Allow ``add_to_rankings`` to be called before AutoMLSearch is called :pr:`1250`
         * Removed Graphviz from test-requirements to add to requirements.txt :pr:`1327`
diff --git a/evalml/automl/automl_algorithm/iterative_algorithm.py b/evalml/automl/automl_algorithm/iterative_algorithm.py
index e0a16ab1..7a067657 100644
--- a/evalml/automl/automl_algorithm/iterative_algorithm.py
+++ b/evalml/automl/automl_algorithm/iterative_algorithm.py
@@ -58,7 +58,7 @@ class IterativeAlgorithm(AutoMLAlgorithm):
 
         next_batch = []
         if self._batch_number == 0:
-            next_batch = [pipeline_class(parameters=self._transform_parameters(pipeline_class, {}))
+            next_batch = [pipeline_class(parameters=self._transform_parameters(pipeline_class, {}), random_state=self.random_state)
                           for pipeline_class in self.allowed_pipelines]
 
         # One after training all pipelines one round
@@ -79,7 +79,8 @@ class IterativeAlgorithm(AutoMLAlgorithm):
             pipeline_class = self._first_batch_results[idx][1]
             for i in range(self.pipelines_per_batch):
                 proposed_parameters = self._tuners[pipeline_class.name].propose()
-                next_batch.append(pipeline_class(parameters=self._transform_parameters(pipeline_class, proposed_parameters)))
+                pl_parameters = self._transform_parameters(pipeline_class, proposed_parameters)
+                next_batch.append(pipeline_class(parameters=pl_parameters, random_state=self.random_state))
         self._pipeline_number += len(next_batch)
         self._batch_number += 1
         return next_batch
diff --git a/evalml/automl/automl_search.py b/evalml/automl/automl_search.py
index 92e0a474..84bbd87e 100644
--- a/evalml/automl/automl_search.py
+++ b/evalml/automl/automl_search.py
@@ -651,7 +651,7 @@ class AutoMLSearch:
                 y_threshold_tuning = None
                 if self.optimize_thresholds and self.objective.problem_type == ProblemTypes.BINARY and self.objective.can_optimize_threshold:
                     X_train, X_threshold_tuning, y_train, y_threshold_tuning = train_test_split(X_train, y_train, test_size=0.2, random_state=self.random_state)
-                cv_pipeline = pipeline.clone()
+                cv_pipeline = pipeline.clone(pipeline.random_state)
                 logger.debug(f"\t\t\tFold {i}: starting training")
                 cv_pipeline.fit(X_train, y_train)
                 logger.debug(f"\t\t\tFold {i}: finished training")


Test patch:
diff --git a/evalml/tests/automl_tests/test_automl.py b/evalml/tests/automl_tests/test_automl.py
index c7c2b66d..3c4b0269 100644
--- a/evalml/tests/automl_tests/test_automl.py
+++ b/evalml/tests/automl_tests/test_automl.py
@@ -42,6 +42,8 @@ from evalml.problem_types import ProblemTypes, handle_problem_types
 from evalml.tuners import NoParamsException, RandomSearchTuner
 from evalml.utils.gen_utils import (
     categorical_dtypes,
+    check_random_state_equality,
+    get_random_state,
     numeric_and_boolean_dtypes
 )
 
@@ -1488,3 +1490,28 @@ def test_pipelines_per_batch(mock_fit, mock_score, X_y_binary):
     assert automl._pipelines_per_batch == 10
     assert automl._automl_algorithm.pipelines_per_batch == 10
     assert total_pipelines(automl, 2, 10) == len(automl.full_rankings)
+
+
+@patch('evalml.pipelines.BinaryClassificationPipeline.score', return_value={"Log Loss Binary": 0.8})
+@patch('evalml.pipelines.BinaryClassificationPipeline.fit')
+def test_automl_respects_random_state(mock_fit, mock_score, X_y_binary, dummy_classifier_estimator_class):
+
+    expected_random_state = get_random_state(42)
+    X, y = X_y_binary
+
+    class DummyPipeline(BinaryClassificationPipeline):
+        component_graph = [dummy_classifier_estimator_class]
+        num_pipelines_different_seed = 0
+        num_pipelines_init = 0
+
+        def __init__(self, parameters, random_state):
+            random_state = get_random_state(random_state)
+            is_diff_random_state = not check_random_state_equality(random_state, expected_random_state)
+            self.__class__.num_pipelines_init += 1
+            self.__class__.num_pipelines_different_seed += is_diff_random_state
+            super().__init__(parameters, random_state)
+
+    automl = AutoMLSearch(problem_type="binary", allowed_pipelines=[DummyPipeline],
+                          random_state=expected_random_state, max_iterations=10)
+    automl.search(X, y)
+    assert DummyPipeline.num_pipelines_different_seed == 0 and DummyPipeline.num_pipelines_init
diff --git a/evalml/tests/automl_tests/test_iterative_algorithm.py b/evalml/tests/automl_tests/test_iterative_algorithm.py
index c168e0c4..75ab7cbc 100644
--- a/evalml/tests/automl_tests/test_iterative_algorithm.py
+++ b/evalml/tests/automl_tests/test_iterative_algorithm.py
@@ -9,6 +9,7 @@ from evalml.model_family import ModelFamily
 from evalml.pipelines import BinaryClassificationPipeline
 from evalml.pipelines.components import Estimator
 from evalml.problem_types import ProblemTypes
+from evalml.utils import check_random_state_equality
 
 
 def test_iterative_algorithm_init_iterative():
@@ -106,6 +107,7 @@ def test_iterative_algorithm_results(ensembling_value, dummy_binary_pipeline_cla
             num_pipelines_classes = (len(dummy_binary_pipeline_classes) + 1) if ensembling_value else len(dummy_binary_pipeline_classes)
             cls = dummy_binary_pipeline_classes[(algo.batch_number - 2) % num_pipelines_classes]
             assert [p.__class__ for p in next_batch] == [cls] * len(next_batch)
+            assert all(check_random_state_equality(p.random_state, algo.random_state) for p in next_batch)
             assert algo.pipeline_number == last_pipeline_number + len(next_batch)
             last_pipeline_number = algo.pipeline_number
             assert algo.batch_number == last_batch_number + 1