Repository: alteryx/evalml (python)

Issue:
For some datasets, percent_better_than_baseline_all_objectives is huge because baseline values are infinitesimal
I tested the `percent_better_than_baseline_all_objectives` values on a few datasets and found that `ExpVariance` on two of them was enormous. On `retail_data.csv` it's `1.4857769515459812e+18`.

Turns out this is because the cv objective scores for the baseline was infinitesimal:

```python
for scores in automl.results['pipeline_results'][0]['cv_data']:
    print(scores['all_objective_scores']['ExpVariance'])
```

```
0.0
0.0
1.1102230246251565e-16
```

With `evalml 0.13.2` these were all zero.

Now, obviously there's variability in this kind of thing, but it seems like `1.1102230246251565e-16` is a rounding error, not a real, valid value for this metric.

Here's the reproducer:

```python
import evalml
from evalml import AutoMLSearch
import pandas as pd

data = pd.read_csv('datasets/retail/retail_data.csv')
data = data.drop(['cancelled'], axis=1)

y = data['total']
X = data.drop(['total'], axis=1)

automl = AutoMLSearch(problem_type="regression", objective="Root Mean Squared Error", max_iterations=3)
automl.search(X, y)

# oh, I miss Scala
if automl.data_check_results is not None:
    for boom in automl.data_check_results:
        print(boom)

print(f'pipline 2 ExpVariance: {automl.results["pipeline_results"][2]["percent_better_than_baseline_all_objectives"]["ExpVariance"]}')


for scores in automl.results['pipeline_results'][0]['cv_data']:
    print(f'cv ExpVariance: {scores["all_objective_scores"]["ExpVariance"]}')

```


Pull request:
When baseline score is close to 0, set percent improved to nan
Fixes #1458 

**Problem**
In #1458 @rpeck found a dataset where, for the baseline model (which predicts the mean) `ExpVariance` came out to 0 on 2 of the 3 CV folds, but came out to ~1e-16 on the other CV fold. This resulted in the percent improvement score for `ExpVariance` for all pipelines being super big, because that computation divides by the baseline score for the objective in question.

**Solution**
Define a threshold (1e-10 seemed good) below which the percent improvement computation assumes the baseline objective score is essentially 0, and will then set the percent improvement for all other pipelines to nan.

Patch:
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index cfc121f8..b4fb0ea6 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -25,6 +25,7 @@ Release Notes
         * Updated enum classes to show possible enum values as attributes :pr:`1391`
         * Updated calls to ``Woodwork``'s ``to_pandas()`` to ``to_series()`` and ``to_dataframe()`` :pr:`1428`
         * Fixed bug in OHE where column names were not guaranteed to be unique :pr:`1349`
+        * Fixed bug with percent improvement of ``ExpVariance`` objective on data with highly skewed target :pr:`1467`
     * Changes
         * Changed ``OutliersDataCheck`` to return the list of columns, rather than rows, that contain outliers :pr:`1377`
         * Simplified and cleaned output for Code Generation :pr:`1371`
diff --git a/evalml/objectives/objective_base.py b/evalml/objectives/objective_base.py
index db028fb1..1d245a80 100644
--- a/evalml/objectives/objective_base.py
+++ b/evalml/objectives/objective_base.py
@@ -126,7 +126,7 @@ class ObjectiveBase(ABC):
         if pd.isna(score) or pd.isna(baseline_score):
             return np.nan
 
-        if baseline_score == 0:
+        if np.isclose(baseline_score, 0, atol=1e-10):
             return np.nan
 
         if baseline_score == score:


Test patch:
diff --git a/evalml/tests/objective_tests/test_standard_metrics.py b/evalml/tests/objective_tests/test_standard_metrics.py
index de36568e..c5b65577 100644
--- a/evalml/tests/objective_tests/test_standard_metrics.py
+++ b/evalml/tests/objective_tests/test_standard_metrics.py
@@ -14,6 +14,7 @@ from evalml.objectives import (
     BalancedAccuracyMulticlass,
     BinaryClassificationObjective,
     CostBenefitMatrix,
+    ExpVariance,
     F1Macro,
     F1Micro,
     F1Weighted,
@@ -473,3 +474,11 @@ def test_calculate_percent_difference_negative_and_equal_numbers():
     assert LogLossBinary.calculate_percent_difference(score=-10, baseline_score=-5) == 100
     assert LogLossBinary.calculate_percent_difference(score=-5, baseline_score=10) == 150
     assert LogLossBinary.calculate_percent_difference(score=10, baseline_score=-5) == -300
+
+
+def test_calculate_percent_difference_small():
+    expected_value = 100 * -1 * np.abs(1e-9 / (1e-9))
+    assert np.isclose(ExpVariance.calculate_percent_difference(score=0, baseline_score=1e-9), expected_value, atol=1e-8)
+    assert pd.isna(ExpVariance.calculate_percent_difference(score=0, baseline_score=1e-10))
+    assert pd.isna(ExpVariance.calculate_percent_difference(score=1e-9, baseline_score=0))
+    assert pd.isna(ExpVariance.calculate_percent_difference(score=0, baseline_score=0))