Repository: plotly/plotly.py

Issue:
Error when running the create_dendrogram example
I am running plotly.py version 4.8.2 in a jupyter notebook.
I ran the following code, copied from the [documentation page](https://plotly.com/python/dendrogram/):
```
import plotly.figure_factory as ff
import numpy as np
np.random.seed(1)

X = np.random.rand(10,10)
fig = ff.create_dendrogram(X)
fig.update_layout(width=800, height=500)
fig.show()
```
and getting the following error message:
```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-1-386e391a19fb> in <module>
      4 
      5 X = np.random.rand(10,10)
----> 6 fig = ff.create_dendrogram(X)
      7 fig.update_layout(width=800, height=500)
      8 fig.show()

~/miniconda3/envs/jupyterlab_new/lib/python3.8/site-packages/plotly/figure_factory/_dendrogram.py in create_dendrogram(X, orientation, labels, colorscale, distfun, linkagefun, hovertext, color_threshold)
     89         distfun = scs.distance.pdist
     90 
---> 91     dendrogram = _Dendrogram(
     92         X,
     93         orientation,

~/miniconda3/envs/jupyterlab_new/lib/python3.8/site-packages/plotly/figure_factory/_dendrogram.py in __init__(self, X, orientation, labels, colorscale, width, height, xaxis, yaxis, distfun, linkagefun, hovertext, color_threshold)
    143             distfun = scs.distance.pdist
    144 
--> 145         (dd_traces, xvals, yvals, ordered_labels, leaves) = self.get_dendrogram_traces(
    146             X, colorscale, distfun, linkagefun, hovertext, color_threshold
    147         )

~/miniconda3/envs/jupyterlab_new/lib/python3.8/site-packages/plotly/figure_factory/_dendrogram.py in get_dendrogram_traces(self, X, colorscale, distfun, linkagefun, hovertext, color_threshold)
    335                 y=np.multiply(self.sign[self.yaxis], ys),
    336                 mode="lines",
--> 337                 marker=dict(color=colors[color_key]),
    338                 text=hovertext_label,
    339                 hoverinfo="text",

KeyError: 'C1'
```
I've tried downgrading to 4.8.1, but got the same error.
My working env is created with miniconda3, using the yml file:
```
name: jupyterlab_new
channels:
  - conda-forge
  - bioconda
dependencies:
  - jupyterlab
  - pandas
  - numpy
  - seaborn
  - plotly
  - biopython
```

Thanks!


Discussion:
I can't replicate this in my local setup, so it seems to depend on a specific version of some dependency. Will keep digging.
@emmanuelle @nicholas-esterer @jonmmease are any of you able to replicate this error? It's showing up in CI here: https://circleci.com/gh/plotly/plotly.py/42585?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link
can't replicate with python 3.7.3
OK so it looks like `scipy` 1.5.0 breaks this. Works for me in 1.4.1 and not in 1.5.0, which was released Jun 21
See also https://community.plotly.com/t/error-in-create-dendrogram-example/42006/2
So, as @emmanuelle suggested, it is because the `color_list` returned by scipy now uses the cyclical color format from matplotlib: (see https://matplotlib.org/3.1.0/api/colors_api.html#module-matplotlib.colors, note about "CN" color spec).
However the new `color_list` is `['C1', 'C2', 'C0', 'C3', 'C4', 'C0', 'C0', 'C0', 'C0']` which while be isomorphic to the old `color_list` `['g', 'r', 'b', 'c', 'm', 'b', 'b', 'b', 'b']`, does not represent the same set of colors, nor do the actual color values match any of those in the original list
So my idea was to also add these "C1", "C2" ... keys to the `default_colors` list and have them map to some colors like was done with the single letter named colors. But how to choose this mapping?
This is the new color map if one uses the corresponding colors in `matplotlib.rcParams['axes.prop_cycle']`

![Screenshot from 2020-07-08 16-40-54](https://user-images.githubusercontent.com/64649603/86968286-f111e580-c139-11ea-97a9-9bba401d237c.png)

So do we want to replace these as well with different colors and if so, which ones? Because there are some colors that aren't in the list we already have (e.g., orange).
Is there an easy fix such that we keep the colors we used to have?
it would be mapping 'c1' to what 'g' was mapped to before, etc. but then what do we do for c5-c9?
what did we do before for more than 4 colors? Sorry, I'm not following the before vs after :)
Before the default `color_list` was ['g', 'r', 'b', 'c', 'm', 'b', 'b', 'b', 'b'] and now the default `color_list` is  `['C1', 'C2', 'C0', 'C3', 'C4', 'C0', 'C0', 'C0', 'C0']`. None of keys exist in the dictionary returned by default by `_dendrite.get_color_dict` ( dictionary that looks roughly like: `{ 'r': "rgb(255,65,54)", 'b': "rgb(0,116,217)", ... }`), hence the key error.
Anyhow if plotly.py doesn't support this "cyclic format" then we actually just need to add 'C1' to the `get_color_dict` dictionary, because the only case that the color 'C1' would be specified is when the user doesn't specify a color and so the default color is looked up, it would just probably be more robust to map all the colors so that if the `color_list` changes again (say it changes the order, or the set of colors), you still get something. But maybe not. 
By before I mean with `scipy===1.4.1` and now I mean `scipy===1.5.1`.

Resolving patch:
diff --git a/packages/python/plotly/plotly/figure_factory/_dendrogram.py b/packages/python/plotly/plotly/figure_factory/_dendrogram.py
--- a/packages/python/plotly/plotly/figure_factory/_dendrogram.py
+++ b/packages/python/plotly/plotly/figure_factory/_dendrogram.py
@@ -25,14 +25,21 @@ def create_dendrogram(
     color_threshold=None,
 ):
     """
-    Function that returns a dendrogram Plotly figure object.
+    Function that returns a dendrogram Plotly figure object. This is a thin
+    wrapper around scipy.cluster.hierarchy.dendrogram.
 
     See also https://dash.plot.ly/dash-bio/clustergram.
 
     :param (ndarray) X: Matrix of observations as array of arrays
     :param (str) orientation: 'top', 'right', 'bottom', or 'left'
     :param (list) labels: List of axis category labels(observation labels)
-    :param (list) colorscale: Optional colorscale for dendrogram tree
+    :param (list) colorscale: Optional colorscale for the dendrogram tree.
+                              Requires 8 colors to be specified, the 7th of
+                              which is ignored.  With scipy>=1.5.0, the 2nd, 3rd
+                              and 6th are used twice as often as the others.
+                              Given a shorter list, the missing values are
+                              replaced with defaults and with a longer list the
+                              extra values are ignored.
     :param (function) distfun: Function to compute the pairwise distance from
                                the observations
     :param (function) linkagefun: Function to compute the linkage matrix from
@@ -160,8 +167,8 @@ def __init__(
         if len(self.zero_vals) > len(yvals) + 1:
             # If the length of zero_vals is larger than the length of yvals,
             # it means that there are wrong vals because of the identicial samples.
-            # Three and more identicial samples will make the yvals of spliting center into 0 and it will \
-            # accidentally take it as leaves.
+            # Three and more identicial samples will make the yvals of spliting
+            # center into 0 and it will accidentally take it as leaves.
             l_border = int(min(self.zero_vals))
             r_border = int(max(self.zero_vals))
             correct_leaves_pos = range(
@@ -185,6 +192,9 @@ def get_color_dict(self, colorscale):
 
         # These are the color codes returned for dendrograms
         # We're replacing them with nicer colors
+        # This list is the colors that can be used by dendrogram, which were
+        # determined as the combination of the default above_threshold_color and
+        # the default color palette (see scipy/cluster/hierarchy.py)
         d = {
             "r": "red",
             "g": "green",
@@ -193,12 +203,14 @@ def get_color_dict(self, colorscale):
             "m": "magenta",
             "y": "yellow",
             "k": "black",
+            # TODO: 'w' doesn't seem to be in the default color
+            # palette in scipy/cluster/hierarchy.py
             "w": "white",
         }
         default_colors = OrderedDict(sorted(d.items(), key=lambda t: t[0]))
 
         if colorscale is None:
-            colorscale = [
+            rgb_colorscale = [
                 "rgb(0,116,217)",  # blue
                 "rgb(35,205,205)",  # cyan
                 "rgb(61,153,112)",  # green
@@ -206,13 +218,43 @@ def get_color_dict(self, colorscale):
                 "rgb(133,20,75)",  # magenta
                 "rgb(255,65,54)",  # red
                 "rgb(255,255,255)",  # white
-                "rgb(255,220,0)",
-            ]  # yellow
+                "rgb(255,220,0)",  # yellow
+            ]
+        else:
+            rgb_colorscale = colorscale
 
         for i in range(len(default_colors.keys())):
             k = list(default_colors.keys())[i]  # PY3 won't index keys
-            if i < len(colorscale):
-                default_colors[k] = colorscale[i]
+            if i < len(rgb_colorscale):
+                default_colors[k] = rgb_colorscale[i]
+
+        # add support for cyclic format colors as introduced in scipy===1.5.0
+        # before this, the colors were named 'r', 'b', 'y' etc., now they are
+        # named 'C0', 'C1', etc. To keep the colors consistent regardless of the
+        # scipy version, we try as much as possible to map the new colors to the
+        # old colors
+        # this mapping was found by inpecting scipy/cluster/hierarchy.py (see
+        # comment above).
+        new_old_color_map = [
+            ("C0", "b"),
+            ("C1", "g"),
+            ("C2", "r"),
+            ("C3", "c"),
+            ("C4", "m"),
+            ("C5", "y"),
+            ("C6", "k"),
+            ("C7", "g"),
+            ("C8", "r"),
+            ("C9", "c"),
+        ]
+        for nc, oc in new_old_color_map:
+            try:
+                default_colors[nc] = default_colors[oc]
+            except KeyError:
+                # it could happen that the old color isn't found (if a custom
+                # colorscale was specified), in this case we set it to an
+                # arbitrary default.
+                default_colors[n] = "rgb(0,116,217)"
 
         return default_colors
 


Test patch: