Repository: alteryx/evalml (python)

Issue:
Prediction explanations: display Index ID for best/worst prediction explanations
**Goal**
If users intend to understand the best and worst predictions, the API should allow them to see the Index of the explanations. 

If the user can provide the index column (perhaps through DataTables), that index ID could be displayed in the reported outputted by `explain_predictions_best_worst`. 

If the user doesn't provide an index column, no index ID should be displayed in the report.  

**Proposal**
Add index ID to the prediction explanation, if the user provides an index column in X.

Note the **Index ID:**
```
        Best 1 of 2

                Predicted Probabilities: [benign: 0.0, malignant: 1.0]
                Predicted Value: malignant
                Target Value: malignant
                Cross Entropy: 0.0
                Index ID: 45


                     Feature Name         Feature Value        Contribution to        SHAP Value
                                                                 Prediction
                ================================================================================
                    worst perimeter          155.30                   +                  0.10
                     worst radius             23.14                   +                  0.08
                 worst concave points         0.17                    +                  0.08
                worst fractal dimension       0.09                    -                 -0.00
                   compactness error          0.04                    -                 -0.00
                    worst symmetry            0.22                    -                 -0.00


        Best 2 of 2

                Predicted Probabilities: [benign: 0.0, malignant: 1.0]
                Predicted Value: malignant
                Target Value: malignant
                Cross Entropy: 0.0
                Index ID: 2

                    Feature Name       Feature Value   Contribution to Prediction   SHAP Value
                ==============================================================================
                  worst perimeter         166.10                   +                   0.10
                    worst radius           25.45                   +                   0.08
                worst concave points       0.22                    +                   0.08
                 compactness error         0.03                    -                  -0.00
                 worst compactness         0.21                    -                  -0.00
                   worst symmetry          0.21                    -                  -0.00


        Worst 1 of 2

                Predicted Probabilities: [benign: 0.552, malignant: 0.448]
                Predicted Value: benign
                Target Value: malignant
                Cross Entropy: 0.802
                Index ID: 7

                    Feature Name       Feature Value   Contribution to Prediction   SHAP Value
                ==============================================================================
                  smoothness error         0.00                    +                   0.04
                    mean texture           21.58                   +                   0.03
                   worst texture           30.25                   +                   0.02
                worst concave points       0.11                    -                  -0.02
                    worst radius           15.93                   -                  -0.03
                mean concave points        0.02                    -                  -0.03

```


Pull request:
Adding the Index ID to explain_prediction_best_worst_output
### Pull Request Description

Fixes #1119.

FYI @RezaAkhtar

-----
*After creating the pull request: in order to pass the **release_notes_updated** check you will need to update the "Future Release" section of* `docs/source/release_notes.rst` *to include this pull request by adding :pr:`123`.*


Patch:
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index 6f61626d..a70129fc 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -4,6 +4,7 @@ Release Notes
 **Future Releases**
     * Enhancements
         * Added ability to freeze hyperparameters for AutoMLSearch :pr:`1284`
+        * Added the index id to the ``explain_predictions_best_worst`` output to help users identify which rows in their data are included :pr:`1365`
     * Fixes
         * Updated enum classes to show possible enum values as attributes :pr:`1391`
     * Changes
diff --git a/docs/source/user_guide/model_understanding.ipynb b/docs/source/user_guide/model_understanding.ipynb
index c398e4b7..7d500921 100644
--- a/docs/source/user_guide/model_understanding.ipynb
+++ b/docs/source/user_guide/model_understanding.ipynb
@@ -355,7 +355,7 @@
     "\n",
     "We can specify our own ranking function by passing in a function to the `metric` parameter. This function will be called on `y_true` and `y_pred`. By convention, lower scores are better.\n",
     "\n",
-    "At the top of each table, we can see the predicted probabilities, target value, and error on that prediction. For a regression problem, we would see the predicted value instead of predicted probabilities.\n"
+    "At the top of each table, we can see the predicted probabilities, target value, error, and row index for that prediction. For a regression problem, we would see the predicted value instead of predicted probabilities.\n"
    ]
   },
   {
diff --git a/evalml/model_understanding/prediction_explanations/_user_interface.py b/evalml/model_understanding/prediction_explanations/_user_interface.py
index 07b454d7..bb2ffe79 100644
--- a/evalml/model_understanding/prediction_explanations/_user_interface.py
+++ b/evalml/model_understanding/prediction_explanations/_user_interface.py
@@ -282,7 +282,7 @@ class _ClassificationPredictedValues(_SectionMaker):
         self.error_name = error_name
         self.predicted_values = y_pred_values
 
-    def make_text(self, index, y_pred, y_true, scores):
+    def make_text(self, index, y_pred, y_true, scores, dataframe_index):
         """Makes the predicted values section for classification problem best/worst reports formatted as text."""
         pred_value = [f"{col_name}: {pred}" for col_name, pred in
                       zip(y_pred.columns, round(y_pred.iloc[index], 3).tolist())]
@@ -292,9 +292,10 @@ class _ClassificationPredictedValues(_SectionMaker):
         return [f"\t\tPredicted Probabilities: {pred_value}\n",
                 f"\t\tPredicted Value: {self.predicted_values[index]}\n",
                 f"\t\tTarget Value: {true_value}\n",
-                f"\t\t{self.error_name}: {round(scores.iloc[index], 3)}\n\n"]
+                f"\t\t{self.error_name}: {round(scores.iloc[index], 3)}\n",
+                f"\t\tIndex ID: {dataframe_index.iloc[index]}\n\n"]
 
-    def make_dict(self, index, y_pred, y_true, scores):
+    def make_dict(self, index, y_pred, y_true, scores, dataframe_index):
         """Makes the predicted values section for classification problem best/worst reports formatted as dictionary."""
         pred_values = dict(zip(y_pred.columns, round(y_pred.iloc[index], 3).tolist()))
 
@@ -302,7 +303,8 @@ class _ClassificationPredictedValues(_SectionMaker):
                 "predicted_value": _make_json_serializable(self.predicted_values[index]),
                 "target_value": _make_json_serializable(y_true.iloc[index]),
                 "error_name": self.error_name,
-                "error_value": _make_json_serializable(scores.iloc[index])}
+                "error_value": _make_json_serializable(scores.iloc[index]),
+                "index_id": _make_json_serializable(dataframe_index.iloc[index])}
 
 
 class _RegressionPredictedValues(_SectionMaker):
@@ -312,17 +314,19 @@ class _RegressionPredictedValues(_SectionMaker):
             error_name = "Absolute Difference"
         self.error_name = error_name
 
-    def make_text(self, index, y_pred, y_true, scores):
+    def make_text(self, index, y_pred, y_true, scores, dataframe_index):
         """Makes the predicted values section for regression problem best/worst reports formatted as text."""
         return [f"\t\tPredicted Value: {round(y_pred.iloc[index], 3)}\n",
                 f"\t\tTarget Value: {round(y_true.iloc[index], 3)}\n",
-                f"\t\t{self.error_name}: {round(scores.iloc[index], 3)}\n\n"]
+                f"\t\t{self.error_name}: {round(scores.iloc[index], 3)}\n",
+                f"\t\tIndex ID: {dataframe_index.iloc[index]}\n\n"]
 
-    def make_dict(self, index, y_pred, y_true, scores):
+    def make_dict(self, index, y_pred, y_true, scores, dataframe_index):
         """Makes the predicted values section for regression problem best/worst reports formatted as a dictionary."""
         return {"probabilities": None, "predicted_value": round(y_pred.iloc[index], 3),
                 "target_value": round(y_true.iloc[index], 3), "error_name": self.error_name,
-                "error_value": round(scores.iloc[index], 3)}
+                "error_value": round(scores.iloc[index], 3),
+                "index_id": _make_json_serializable(dataframe_index.iloc[index])}
 
 
 class _SHAPTable(_SectionMaker):
@@ -390,7 +394,9 @@ class _ReportMaker:
         for rank, index in enumerate(data.index_list):
             report.extend(self.heading_maker.make_text(rank))
             if self.make_predicted_values_maker:
-                report.extend(self.make_predicted_values_maker.make_text(index, data.y_pred, data.y_true, data.errors))
+                report.extend(self.make_predicted_values_maker.make_text(index, data.y_pred, data.y_true,
+                                                                         data.errors,
+                                                                         pd.Series(data.input_features.index)))
             else:
                 report.extend([""])
             report.extend(self.table_maker.make_text(index, data.pipeline, data.input_features))
@@ -413,7 +419,8 @@ class _ReportMaker:
                 section["rank"] = self.heading_maker.make_dict(rank)
             if self.make_predicted_values_maker:
                 section["predicted_values"] = self.make_predicted_values_maker.make_dict(index, data.y_pred,
-                                                                                         data.y_true, data.errors)
+                                                                                         data.y_true, data.errors,
+                                                                                         pd.Series(data.input_features.index))
             section["explanations"] = self.table_maker.make_dict(index, data.pipeline,
                                                                  data.input_features)["explanations"]
             report.append(section)


Test patch:
diff --git a/evalml/tests/model_understanding_tests/prediction_explanations_tests/test_explainers.py b/evalml/tests/model_understanding_tests/prediction_explanations_tests/test_explainers.py
index d08362ba..e051acb7 100644
--- a/evalml/tests/model_understanding_tests/prediction_explanations_tests/test_explainers.py
+++ b/evalml/tests/model_understanding_tests/prediction_explanations_tests/test_explainers.py
@@ -229,6 +229,7 @@ regression_best_worst_answer = """Test Pipeline Name
                 Predicted Value: 1
                 Target Value: 2
                 Absolute Difference: 1
+                Index ID: {index_0}
 
                 table goes here
 
@@ -238,6 +239,7 @@ regression_best_worst_answer = """Test Pipeline Name
                 Predicted Value: 2
                 Target Value: 3
                 Absolute Difference: 4
+                Index ID: {index_1}
 
                 table goes here
 
@@ -290,6 +292,7 @@ binary_best_worst_answer = """Test Pipeline Name
                 Predicted Value: malignant
                 Target Value: malignant
                 Cross Entropy: 0.2
+                Index ID: {index_0}
 
                 table goes here
 
@@ -300,6 +303,7 @@ binary_best_worst_answer = """Test Pipeline Name
                 Predicted Value: malignant
                 Target Value: benign
                 Cross Entropy: 0.78
+                Index ID: {index_1}
 
                 table goes here
 
@@ -345,6 +349,7 @@ multiclass_best_worst_answer = """Test Pipeline Name
                 Predicted Value: setosa
                 Target Value: setosa
                 Cross Entropy: 0.15
+                Index ID: {{index_0}}
 
                 {multiclass_table}
 
@@ -355,6 +360,7 @@ multiclass_best_worst_answer = """Test Pipeline Name
                 Predicted Value: versicolor
                 Target Value: versicolor
                 Cross Entropy: 0.34
+                Index ID: {{index_1}}
 
                 {multiclass_table}
 
@@ -426,12 +432,23 @@ def test_explain_predictions_best_worst_and_explain_predictions(mock_make_table,
     pipeline.problem_type = problem_type
     pipeline.name = "Test Pipeline Name"
 
+    def _add_custom_index(answer, index_best, index_worst, output_format):
+
+        if output_format == "text":
+            answer = answer.format(index_0=index_best, index_1=index_worst)
+        else:
+            answer["explanations"][0]["predicted_values"]["index_id"] = index_best
+            answer["explanations"][1]["predicted_values"]["index_id"] = index_worst
+        return answer
+
     if problem_type == ProblemTypes.REGRESSION:
         abs_error_mock = MagicMock(__name__="abs_error")
         abs_error_mock.return_value = pd.Series([4, 1], dtype="int")
         mock_default_metrics.__getitem__.return_value = abs_error_mock
         pipeline.predict.return_value = pd.Series([2, 1])
         y_true = pd.Series([3, 2], index=custom_index)
+        answer = _add_custom_index(answer, index_best=custom_index[1],
+                                   index_worst=custom_index[0], output_format=output_format)
     elif problem_type == ProblemTypes.BINARY:
         pipeline.classes_.return_value = ["benign", "malignant"]
         cross_entropy_mock = MagicMock(__name__="cross_entropy")
@@ -440,6 +457,8 @@ def test_explain_predictions_best_worst_and_explain_predictions(mock_make_table,
         pipeline.predict_proba.return_value = pd.DataFrame({"benign": [0.05, 0.1], "malignant": [0.95, 0.9]})
         pipeline.predict.return_value = pd.Series(["malignant"] * 2)
         y_true = pd.Series(["malignant", "benign"], index=custom_index)
+        answer = _add_custom_index(answer, index_best=custom_index[0],
+                                   index_worst=custom_index[1], output_format=output_format)
     else:
         # Multiclass text output is formatted slightly different so need to account for that
         if output_format == "text":
@@ -452,6 +471,8 @@ def test_explain_predictions_best_worst_and_explain_predictions(mock_make_table,
                                                             "virginica": [0.1, 0.05]})
         pipeline.predict.return_value = ["setosa", "versicolor"]
         y_true = pd.Series(["setosa", "versicolor"], index=custom_index)
+        answer = _add_custom_index(answer, index_best=custom_index[0],
+                                   index_worst=custom_index[1], output_format=output_format)
 
     best_worst_report = explain_predictions_best_worst(pipeline, input_features, y_true=y_true,
                                                        num_to_explain=1, output_format=output_format)
@@ -514,6 +535,7 @@ regression_custom_metric_answer = """Test Pipeline Name
                 Predicted Value: 1
                 Target Value: 2
                 sum: 3
+                Index ID: 1
 
                 table goes here
 
@@ -523,6 +545,7 @@ regression_custom_metric_answer = """Test Pipeline Name
                 Predicted Value: 2
                 Target Value: 3
                 sum: 5
+                Index ID: 0
 
                 table goes here
 
@@ -533,11 +556,13 @@ regression_custom_metric_answer_dict = {
     "explanations": [
         {"rank": {"prefix": "best", "index": 1},
          "predicted_values": {"probabilities": None, "predicted_value": 1, "target_value": 2,
-                              "error_name": "sum", "error_value": 3},
+                              "error_name": "sum", "error_value": 3,
+                              "index_id": 1},
          "explanations": ["explanation_dictionary_goes_here"]},
         {"rank": {"prefix": "worst", "index": 1},
          "predicted_values": {"probabilities": None, "predicted_value": 2, "target_value": 3,
-                              "error_name": "sum", "error_value": 5},
+                              "error_name": "sum", "error_value": 5,
+                              "index_id": 0},
          "explanations": ["explanation_dictionary_goes_here"]}
     ]
 }