Repository: alteryx/evalml (python)

Issue:
Add predicted vs actual plot (regression)
It would be helpful to add a predicted vs actual plot util method for regression pipelines.

Options:
* Simplest: scatter plot. This is where we should start
* Bin along each axis to produce a 2d heatmap
* In the future, this can be extended to order by datetime for timeseries regression problems

For basic regression this can be a scatter plot. Once we add timeseries support this can be extended to be ordered by time.

Others will want to use evalml who will be making this sort of plot, and rendering the results in separate UIs. If we separate the data generation functionality from the plotly plotting, we could recycle the data generation between those two teams. However, the requirements for those two use-cases may differ significantly. So my current plan for us here is simply add a plotly util, and once we've all learned and discussed mutual requirements, we can circle back and add a common data function.

Pull request:
Add Plot for Prediction vs Actual for Regression Problems
fix #772 

Added `outlier_threshold` for simple outlier detection

Updated Model_Understanding doc [here](https://evalml.alteryx.com/en/bc_772_plot/user_guide/model_understanding.html#Predicted-Vs-Actual-Values-Graph-for-Regression-Problems)

Regular documentation [here](https://evalml.alteryx.com/en/bc_772_plot/generated/evalml.model_understanding.graph_prediction_vs_actual.html#evalml.model_understanding.graph_prediction_vs_actual)

Patch:
diff --git a/docs/source/api_reference.rst b/docs/source/api_reference.rst
index ba1af6d6..1b811832 100644
--- a/docs/source/api_reference.rst
+++ b/docs/source/api_reference.rst
@@ -247,6 +247,7 @@ Graph Utils
     graph_permutation_importance
     binary_objective_vs_threshold
     graph_binary_objective_vs_threshold
+    graph_prediction_vs_actual
 
 .. currentmodule:: evalml.model_understanding.prediction_explanations
 
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index 6e18b4c7..d8d292a0 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -4,6 +4,8 @@ Release Notes
 **Future Releases**
     * Enhancements
         * Added stacked ensemble component classes (``StackedEnsembleClassifier``, ``StackedEnsembleRegressor``) :pr:`1134`
+        * Added ``graph_prediction_vs_actual`` in ``model_understanding`` for regression problems :pr:`1252`
+        * Added stacked ensemble component classes (StackedEnsembleClassifier, StackedEnsembleRegressor) :pr:`1134`
         * Added parameter to ``OneHotEncoder`` to enable filtering for features to encode for :pr:`1249`
         * Added percent-better-than-baseline for all objectives to automl.results :pr:`1244`
     * Fixes
diff --git a/docs/source/user_guide/model_understanding.ipynb b/docs/source/user_guide/model_understanding.ipynb
index 4d26e9e5..c398e4b7 100644
--- a/docs/source/user_guide/model_understanding.ipynb
+++ b/docs/source/user_guide/model_understanding.ipynb
@@ -271,6 +271,36 @@
     "graph_binary_objective_vs_threshold(pipeline, X, y, 'f1', steps=100)"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Predicted Vs Actual Values Graph for Regression Problems\n",
+    "\n",
+    "We can also create a scatterplot comparing predicted vs actual values for regression problems. We can specify an `outlier_threshold` to color values differently if the absolute difference between the actual and predicted values are outside of a given threshold. "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from evalml.model_understanding.graphs import graph_prediction_vs_actual\n",
+    "\n",
+    "class LinearRegressionPipeline(evalml.pipelines.RegressionPipeline):\n",
+    "    component_graph = ['One Hot Encoder', 'Linear Regressor']\n",
+    "\n",
+    "X_regress, y_regress = evalml.demos.load_diabetes()\n",
+    "X_train, X_test, y_train, y_test = evalml.preprocessing.split_data(X_regress, y_regress, regression=True)\n",
+    "\n",
+    "pipeline_regress = LinearRegressionPipeline({})\n",
+    "pipeline_regress.fit(X_train, y_train)\n",
+    "\n",
+    "y_pred = pipeline_regress.predict(X_test)\n",
+    "graph_prediction_vs_actual(y_test, y_pred, outlier_threshold=50)\n"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
diff --git a/evalml/model_understanding/__init__.py b/evalml/model_understanding/__init__.py
index 0890a0b5..014e0567 100644
--- a/evalml/model_understanding/__init__.py
+++ b/evalml/model_understanding/__init__.py
@@ -11,6 +11,7 @@ from .graphs import (
     binary_objective_vs_threshold,
     graph_binary_objective_vs_threshold,
     partial_dependence,
-    graph_partial_dependence
+    graph_partial_dependence,
+    graph_prediction_vs_actual
 )
 from .prediction_explanations import explain_prediction, explain_predictions_best_worst, explain_predictions
diff --git a/evalml/model_understanding/graphs.py b/evalml/model_understanding/graphs.py
index 53b04fc2..9792ed17 100644
--- a/evalml/model_understanding/graphs.py
+++ b/evalml/model_understanding/graphs.py
@@ -1,4 +1,3 @@
-
 import copy
 import warnings
 
@@ -487,3 +486,69 @@ def _calculate_axis_range(arr):
     min_value = arr.min()
     margins = abs(max_value - min_value) * 0.05
     return [min_value - margins, max_value + margins]
+
+
+def _get_prediction_vs_actual_data(y_true, y_pred, outlier_threshold):
+    """Helper method to help calculate the y_true and y_pred dataframe, with a column for outliers"""
+    predictions = y_pred.reset_index(drop=True)
+    actual = y_true.reset_index(drop=True)
+    data = pd.concat([pd.Series(predictions),
+                      pd.Series(actual)], axis=1)
+    data.columns = ['prediction', 'actual']
+    if outlier_threshold:
+        data['outlier'] = np.where((abs(data['prediction'] - data['actual']) >= outlier_threshold), "#ffff00", "#0000ff")
+    else:
+        data['outlier'] = '#0000ff'
+    return data
+
+
+def graph_prediction_vs_actual(y_true, y_pred, outlier_threshold=None):
+    """Generate a scatter plot comparing the true and predicted values. Used for regression plotting
+
+    Arguments:
+        y_true (pd.Series): The real target values of the data
+        y_pred (pd.Series): The predicted values outputted by the regression model.
+        outlier_threshold (int, float): A positive threshold for what is considered an outlier value. This value is compared to the absolute difference
+                                 between each value of y_true and y_pred. Values within this threshold will be blue, otherwise they will be yellow.
+                                 Defaults to None
+
+    Returns:
+        plotly.Figure representing the predicted vs. actual values graph
+
+    """
+    _go = import_or_raise("plotly.graph_objects", error_msg="Cannot find dependency plotly.graph_objects")
+    if jupyter_check():
+        import_or_raise("ipywidgets", warning=True)
+
+    if outlier_threshold and outlier_threshold <= 0:
+        raise ValueError(f"Threshold must be positive! Provided threshold is {outlier_threshold}")
+
+    if not isinstance(y_true, pd.Series):
+        y_true = pd.Series(y_true)
+    if not isinstance(y_pred, pd.Series):
+        y_pred = pd.Series(y_pred)
+
+    df = _get_prediction_vs_actual_data(y_true, y_pred, outlier_threshold)
+    data = []
+
+    x_axis = _calculate_axis_range(df['prediction'])
+    y_axis = _calculate_axis_range(df['actual'])
+    x_y_line = [min(x_axis[0], y_axis[0]), max(x_axis[1], y_axis[1])]
+    data.append(_go.Scatter(x=x_y_line, y=x_y_line, name="y = x line", line_color='grey'))
+
+    title = 'Predicted vs Actual Values Scatter Plot'
+    layout = _go.Layout(title={'text': title},
+                        xaxis={'title': 'Prediction', 'range': x_y_line},
+                        yaxis={'title': 'Actual', 'range': x_y_line})
+
+    for color, outlier_group in df.groupby('outlier'):
+        if outlier_threshold:
+            name = "< outlier_threshold" if color == "#0000ff" else ">= outlier_threshold"
+        else:
+            name = "Values"
+        data.append(_go.Scatter(x=outlier_group['prediction'],
+                                y=outlier_group['actual'],
+                                mode='markers',
+                                marker=_go.scatter.Marker(color=color),
+                                name=name))
+    return _go.Figure(layout=layout, data=data)


Test patch:
diff --git a/evalml/tests/model_understanding_tests/test_graphs.py b/evalml/tests/model_understanding_tests/test_graphs.py
index 291af8e1..3e21440d 100644
--- a/evalml/tests/model_understanding_tests/test_graphs.py
+++ b/evalml/tests/model_understanding_tests/test_graphs.py
@@ -19,6 +19,7 @@ from evalml.model_understanding.graphs import (
     graph_partial_dependence,
     graph_permutation_importance,
     graph_precision_recall_curve,
+    graph_prediction_vs_actual,
     graph_roc_curve,
     normalize_confusion_matrix,
     partial_dependence,
@@ -834,7 +835,7 @@ def test_graph_partial_dependence(test_pipeline):
 
 @patch('evalml.model_understanding.graphs.jupyter_check')
 @patch('evalml.model_understanding.graphs.import_or_raise')
-def test_jupyter_graph_check(import_check, jupyter_check, X_y_binary, test_pipeline):
+def test_jupyter_graph_check(import_check, jupyter_check, X_y_binary, X_y_regression, test_pipeline):
     pytest.importorskip('plotly.graph_objects', reason='Skipping plotting test because plotly not installed')
     X, y = X_y_binary
     clf = test_pipeline
@@ -877,3 +878,66 @@ def test_jupyter_graph_check(import_check, jupyter_check, X_y_binary, test_pipel
         graph_roc_curve(y, y_pred_proba)
         assert len(graph_valid) == 0
         import_check.assert_called_with('ipywidgets', warning=True)
+
+    Xr, yr = X_y_regression
+    with pytest.warns(None) as graph_valid:
+        rs = np.random.RandomState(42)
+        y_preds = yr * rs.random(yr.shape)
+        graph_prediction_vs_actual(yr, y_preds)
+        assert len(graph_valid) == 0
+        import_check.assert_called_with('ipywidgets', warning=True)
+
+
+def test_graph_prediction_vs_actual_default():
+    go = pytest.importorskip('plotly.graph_objects', reason='Skipping plotting test because plotly not installed')
+    y_true = [1, 2, 3000, 4, 5, 6, 7, 8, 9, 10, 11, 12]
+    y_pred = [5, 4, 2, 8, 6, 6, 5, 1, 7, 2, 1, 3000]
+
+    fig = graph_prediction_vs_actual(y_true, y_pred)
+    assert isinstance(fig, type(go.Figure()))
+    fig_dict = fig.to_dict()
+    assert fig_dict['layout']['title']['text'] == 'Predicted vs Actual Values Scatter Plot'
+    assert fig_dict['layout']['xaxis']['title']['text'] == 'Prediction'
+    assert fig_dict['layout']['yaxis']['title']['text'] == 'Actual'
+    assert len(fig_dict['data']) == 2
+    assert fig_dict['data'][0]['name'] == 'y = x line'
+    assert fig_dict['data'][0]['x'] == fig_dict['data'][0]['y']
+    assert len(fig_dict['data'][1]['x']) == len(y_true)
+    assert fig_dict['data'][1]['marker']['color'] == '#0000ff'
+    assert fig_dict['data'][1]['name'] == "Values"
+
+
+def test_graph_prediction_vs_actual():
+    go = pytest.importorskip('plotly.graph_objects', reason='Skipping plotting test because plotly not installed')
+    y_true = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]
+    y_pred = [5, 4, 3, 8, 6, 3, 5, 9, 7, 12, 1, 2]
+
+    with pytest.raises(ValueError, match="Threshold must be positive!"):
+        graph_prediction_vs_actual(y_true, y_pred, outlier_threshold=-1)
+
+    fig = graph_prediction_vs_actual(y_true, y_pred, outlier_threshold=100)
+    assert isinstance(fig, type(go.Figure()))
+    fig_dict = fig.to_dict()
+    assert fig_dict['layout']['title']['text'] == 'Predicted vs Actual Values Scatter Plot'
+    assert fig_dict['layout']['xaxis']['title']['text'] == 'Prediction'
+    assert fig_dict['layout']['yaxis']['title']['text'] == 'Actual'
+    assert len(fig_dict['data']) == 2
+    assert fig_dict['data'][1]['marker']['color'] == '#0000ff'
+
+    y_true = pd.Series(y_true)
+    y_pred = pd.Series(y_pred)
+    fig = graph_prediction_vs_actual(y_true, y_pred, outlier_threshold=6.1)
+    assert isinstance(fig, type(go.Figure()))
+    fig_dict = fig.to_dict()
+    assert fig_dict['layout']['title']['text'] == 'Predicted vs Actual Values Scatter Plot'
+    assert fig_dict['layout']['xaxis']['title']['text'] == 'Prediction'
+    assert fig_dict['layout']['yaxis']['title']['text'] == 'Actual'
+    assert len(fig_dict['data']) == 3
+    assert fig_dict['data'][1]['marker']['color'] == '#0000ff'
+    assert len(fig_dict['data'][1]['x']) == 10
+    assert len(fig_dict['data'][1]['y']) == 10
+    assert fig_dict['data'][1]['name'] == "< outlier_threshold"
+    assert fig_dict['data'][2]['marker']['color'] == '#ffff00'
+    assert len(fig_dict['data'][2]['x']) == 2
+    assert len(fig_dict['data'][2]['y']) == 2
+    assert fig_dict['data'][2]['name'] == ">= outlier_threshold"