Repository: ludwig-ai/ludwig

Issue:
bug: test_visualization.py::test_visualization_calibration_1_vs_all_output_saved fails with IndexError in seaborn regplot() function
**Describe the bug**
Unit test 
```
tests/
  integration_tests/
    test_visualization.py::test_visualization_calibration_1_vs_all_output_saved
``` 
fails with `IndexError` exception in the travis ci run and locally for me in a docker container:
```
   for command, viz_pattern in zip(commands, vis_patterns):
            result = subprocess.run(command)
            figure_cnt = glob.glob(viz_pattern)
    
>           assert 0 == result.returncode
E           AssertionError: assert 0 == 1
E            +  where 1 = CompletedProcess(args=['python', '-m', 'ludwig.visualize', '--visualization', 'calibration_1_vs_all', '--metrics', 'ac...robabilities.npy', '--model_names', 'Model1', 'Model2', '--top_k', '6', '-od', 'results/experiment_run'], returncode=1).returncode
tests/integration_tests/test_visualization.py:1581: AssertionError
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/opt/python/3.6.7/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/python/3.6.7/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/travis/build/uber/ludwig/ludwig/visualize.py", line 3265, in <module>
    cli(sys.argv[1:])
  File "/home/travis/build/uber/ludwig/ludwig/visualize.py", line 3260, in cli
    vis_func(**vars(args))
  File "/home/travis/build/uber/ludwig/ludwig/visualize.py", line 623, in calibration_1_vs_all_cli
    calibration_1_vs_all(probabilities_per_model, gt, **kwargs)
  File "/home/travis/build/uber/ludwig/ludwig/visualize.py", line 2654, in calibration_1_vs_all
    filename=filename
  File "/home/travis/build/uber/ludwig/ludwig/utils/visualization_utils.py", line 856, in calibration_plot
    algorithm_names) else '')
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/seaborn/regression.py", line 810, in regplot
    x_jitter, y_jitter, color, label)
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/seaborn/regression.py", line 114, in __init__
    self.dropna("x", "y", "units", "x_partial", "y_partial")
  File "/home/travis/virtualenv/python3.6.7/lib/python3.6/site-packages/seaborn/regression.py", line 66, in dropna
    setattr(self, var, val[not_na])
IndexError: too many indices for array
```

**To Reproduce**
Steps to reproduce the behavior:
Any PR submitted after 24Jan2020 should exhibit the problem

**Expected behavior**
Unit test should not fail

**Screenshots**
See above

**Environment (please complete the following information):**
Failure occurs in two environments.  It fails in my local development environment.
 - OS: MacOS 10.15.2
 - Docker Desktop for Mac 2.2.0.0
- Python version: 3.6.8
- Ludwig version: 2.1

Failure also occurs in the [travis-ci environment](https://travis-ci.org/uber/ludwig/builds/642189471?utm_source=github_status&utm_medium=notification)

**Additional context**
After staring at this issue for a few days, it appears the error may be related to the version of the `seaborn` package.  When the unit test is run using `seaborn 0.9.0`, the test completes successfully.  However, when `seaborn 0.10.0` is installed, the test fails with the cited error message.  I've confirmed this observation through local testing, i.e.,
* `seaborn 0.9.0` => unit test completes successfully
* `seaborn 0.10.0` => unit test fails

Note:  unit test also fails with `seaborn 0.9.1`

`seaborn 0.10.0` was released on PyPi on 2020-01-24. `seaborn 0.9.0` was released 2018-07-16.  Prior to mid-January 2020, I did not encounter any issues in running the `pytest` suite of tests.

`seaborn 0.10.0` [release notes](https://seaborn.pydata.org/whatsnew.html) indicate changes were made to the `regplot()` function, which is the one failing in ludwig's unit test.  I can't tell if the issue we are seeing is due to a breaking api in the `seaborn` 0.10.0 api or a bug in 0.10.0 code base.  More analysis is required.

Since the requirements file used to build the travis-ci environment specifies `seaborn>=0.7`, this means `pip install` will use the most recent version of seaborn greater than 0.7.  This results in installing `seaborn 0.10.0`.   The implication is that any PR submitted from this point on will fail in the cited unit test.

A possible short-term work-around for the travis-ci run failures is to update `requirements_viz.txt` to replace the current `seaborn` specification from `seaborn>=0.7` with `seaborn>=0.7,<=0.9.0`.  If this is acceptable, I'll submit a PR to effect the change.



Discussion:
I doubt there'll be any be repercussions to changing to seaborn <= 0.9.0. So it should be okay. @w4nderlust 
I think it is worth fixing the specific function call and testing seaborn 0.10, we don't want to keep hooked to older versions.
@msaisumanth  @w4nderlust 
Initially I did try to figure out how to fix the function call but I got lost in understanding some of the internal data required data structures.  I'm still willing to try and fix the function call to support `seaborn 0.10.0`.  I may need some help in questions about the data structures for the 1_vs_all plot.

While I work on the fix to support `seaborn 0.10.0`, I'll submit the PR to cap the version of `seaborn` installed for the travis-ci to 0.9.0.  This way folks should get a clean run when submitting a PR for the near-term.   

Once the `seaborn 0.10.0` issues is fixed, the PR will backout the '<=0.9.0' cap on the `seaborn` version.
Makes sense @jimthompson5802 , thank you!
I actually added it myself. I'll keep the issue open anyway, because I prefer to adapt to whatever API change seaborn did in 0.10.0 as a more sustainable solution moving forward.
OK..I focus on the long-term fix to support `seaborn 0.10.0`
I've been working on this a bit, made some discoveries and pushed some code: https://github.com/uber/ludwig/commit/997c6bc02dffcc726efda7ddbb5978e425ffab96
The first discovery is that the number of classes was wrong, so I fixed it.
The second discovery is that there was no way to tell the plot function to draw a line, so now I set it to check how many points are actually there and and set the order ccordingly.
Third discovery: `calibration_curve` return lists of the length it wants even if you specify a specific number of bins, so now i add a 0,0 point at the beginning so that there's at least one point.
Fourth discovery: the API version of the test passes, but the CLI version does not, so there's something fishy there.
I wasn't able to fix the problem anyway, it goes deep in seaborn's linear algebra functions and it's pretty hard to figure out to be hones because of pretty arcane error:
```
ValueError: On entry to DLASCL parameter number 4 had an illegal value
```
It is probably related to the fact that there are nans in the matrix to invert, but I'm not 100 sure how nans end up there in the CLI case and not in the API case.
I pulled down your commit `997cbc` and can reproduce the symptoms.  Right now I'm focusing on understanding data flows to `seaborn`'s `reg_plot()` function. 
@w4nderlust   I instrumented the function `calibration_1_vs_all()` in `ludwig/visualize.py`.  

Re: your observation that the api test succeeds but the cli version fails with this error `ValueError: On entry to DLASCL parameter number 4 had an illegal value`.

From what I can tell the api and cli tests create the same synthetic data set for the tests.  

In case of the **cli** test the probabilities passed to `calibaration_1_vs_all()` function is 
<img width="443" alt="probablities_vis_cli_fails" src="https://user-images.githubusercontent.com/1425269/73410633-d9078180-42d0-11ea-85d5-1b6f835295ad.png">

In case of the **api** test the probabilities passed to `calibration_1_vs_all()` is this
<img width="458" alt="probabilities_vis_api_works" src="https://user-images.githubusercontent.com/1425269/73410674-f4728c80-42d0-11ea-8a31-b6c70db1f433.png">

I was expecting the structure of the probabilities matrix to be similar but they seem to be different.  What should the structure of the probabilities matrix?  Since the api version works, I'm thinking the first two columns contain the predicted probability for that particular class.  The 3rd column could be the probability for the predicted class.

Does this observation point to a direction to resolve this issue?
The structure is the same, although the distributions are very different. In the second case there's something weird as they don't look like probabilities, i.e. the rows don't sum up to 1... But that's the one that works surprisingly enough.
One potential weak point is the `calibration_curve` as sometimes it returns vectors of size 21 as requested (number of bins) sometimes it doesn't so maybe that function is doing something weird. Also it has a normalize parameter that if turned no gives some errors. Maybe this is a good starting point for investigation.
Finally, it seems like even with the modified version of seaborn there's still one failing test case. Will look into it.
I understand each row in the probabilities matrix should sum to one.  Although the **cli** version fails, it conforms to this requirement (within rounding).   The **api** version, which does not fail, does not conform to this requirement.

From what I can see the sample training data created for both the **api** and **cli** version create a two category output response variable:
```
output_features = [category_feature(vocab_size=2, reduce_input='sum')]
```
Excerpt from `description.json`:
```
       "output_features": [
            {
                "dependencies": [],
                "embedding_size": 5,
                "idx2str": [
                    "pvfdrgcrcl",
                    "ICznO"
                ],

        "preprocessing": {
            "category": {
                "fill_value": "<UNK>",
                "lowercase": false,
                "missing_value_strategy": "fill_with_const",
                "most_common": 10000
            },
````

I'm now trying to understand what probabilities are represented in each of the three columns.  If the output variable has two valid values, is the third probability--column 0--for the `<UNK>` missing value?
<img width="443" alt="probablities_vis_cli_fails" src="https://user-images.githubusercontent.com/1425269/73445590-02ec9280-4329-11ea-8759-177a89035f5e.png">

Assuming the three probabilities are as I described above, is the expected output the following, one set of charts for each of the three columns in the probability matrix?
<img width="413" alt="Screen Shot 2020-01-30 at 06 44 36" src="https://user-images.githubusercontent.com/1425269/73447142-2107c200-432c-11ea-9dec-785a89dcecb6.png">


@w4nderlust I have an explanation for this issue you pointed out re: the **api** test:

>  In the second case there's something weird as they don't look like probabilities, i.e. the rows don't sum up to 1... But that's the one that works surprisingly enough.

I believe the issue is due to an error in the `Experiment` class in `test_visualization_api.py` in this statement (line 85)
```
 self.probability = self.test_stats_full[0].iloc[:, 2:].values
```
<img width="721" alt="Experiment_code_collect_probabilities" src="https://user-images.githubusercontent.com/1425269/73505001-32d67d00-439f-11ea-8717-0cd6ea27b67d.png">

This is the `self.test_stats_full[0]` data structure.  The `.iloc[:,2:]` refers to the last three columns in the data structure.  
<img width="1149" alt="test_set_probabilities" src="https://user-images.githubusercontent.com/1425269/73505056-5ef1fe00-439f-11ea-8aff-1b044c1cd4ac.png">

For this particular, I believe the correct specification should have been `.iloc[:,1:4]`, which refers to the 2nd, 3rd and 4th column in the data structure.  From eye balling the numbers, these three number add up to 1, with rounding.

Unless I hear otherwise, I'll be make the change to refer to the correct columns.  I'll make sure that the modifications should handle any number of probabilities and not hard coded to three columns.

After making the change described above, the probabilities for the **api** test case now have the correct structure, i.e., the rows sum to one.
<img width="480" alt="Screen Shot 2020-01-30 at 21 08 08" src="https://user-images.githubusercontent.com/1425269/73506828-e130f100-43a4-11ea-9616-eed99803eabb.png">

With this change the **api** unit test now fails just like the **cli** unit test. 

I think this is progress because there is now a consistent error. :-)

Resolving patch:
diff --git a/ludwig/utils/visualization_utils.py b/ludwig/utils/visualization_utils.py
--- a/ludwig/utils/visualization_utils.py
+++ b/ludwig/utils/visualization_utils.py
@@ -849,7 +849,7 @@ def calibration_plot(
         # sns.tsplot(mean_predicted_values[i], fraction_positives[i], ax=ax1, color=colors[i])
 
         assert len(mean_predicted_values[i]) == len(fraction_positives[i])
-        order = min(3, len(mean_predicted_values[i] - 1))
+        order = min(3, len(mean_predicted_values[i]) - 1)
 
         sns.regplot(mean_predicted_values[i], fraction_positives[i],
                     order=order, x_estimator=np.mean, color=colors[i],


Test patch:
diff --git a/tests/integration_tests/test_visualization_api.py b/tests/integration_tests/test_visualization_api.py
--- a/tests/integration_tests/test_visualization_api.py
+++ b/tests/integration_tests/test_visualization_api.py
@@ -82,7 +82,8 @@ def __init__(self, csv_filename):
         # probabilities need to be list of lists containing each row data
         # from the probability columns
         # ref: https://uber.github.io/ludwig/api/#test - Return
-        self.probability = self.test_stats_full[0].iloc[:, 2:].values
+        num_probs = self.output_features[0]['vocab_size']
+        self.probability = self.test_stats_full[0].iloc[:, 1:(num_probs+2)].values
         self.ground_truth_metadata = self.model.train_set_metadata
         target_predictions = test_df[self.output_feature_name]
         self.ground_truth = np.asarray([