Repository: scipy/scipy

Issue:
Optimal leaf ordering in scipy.cluster.hierarchy.dendrogram
It might be useful to have an optional feature that allows for the dendrogram to be optimized in certain ways with regards to ordering. The cba and ReorderCluster (possibly others) in bioconductor do this in R, and optimalleaforder() function in MATLAB does it. The standard reference is http://bioinformatics.oxfordjournals.org/content/17/suppl_1/S22.full.pdf

I've written a working prototype in python, will be happy to help integrate it with scipy if you think it's of general use http://nbviewer.ipython.org/url/stanford.edu/~markak/2015-09-02%20-%20Optimal%20leaf%20ordering%20for%20hierarchical%20clustering.ipynb



Discussion:
I love optimal leaf ordering and would definitely use this.

@markak that would be a nice feature, PR very welcome!

Note that `dendrogram` is just reworked significantly in gh-5164, which should be merged asap. Make sure to work on top of that and not current master.

Something else that would be very welcome: a realistic example of a reasonable size from the bio world (none of the Scipy devs have that background), for the `dendrogram` docstring. The toy example that's there now isn't super informative.

I was looking for a way to do this, and since @markak's code isn't in scipy yet, I decided to reimplement it for myself.

After beating my head against it for a few days I think I've succeeded. I rewrote the algorithm to avoid recursion by taking advantage of the linkage matrix that scipy produces. This saves some time (on the order of 3x in very cursory testing). That's nice because this algorithm can be _really_ slow on large matrices.

I've got my code [on GitHub here](https://github.com/jamestwebber/jtw-utils/blob/master/optimal_leaf_ordering.py). It needs some editing to use standard scipy style. Also I'm not sure what the best way to introduce it to scipy.cluster.hierarchy would be. I could imagine sticking it in a few places. Let me know and I can prepare a PR.

@rgommers in terms of having a example from biology, I'm happy to find one but what is a reasonable size? One matrix I've used before (for intro-bioinformatics stuff) is a cleaned-up version of [Gasch et al. 2000](http://www.ncbi.nlm.nih.gov/pubmed/11102521), which as it happens I put on GitHub [in this repository](https://github.com/jamestwebber/bootcamp-programming/blob/master/bootcamp/data/experiment_data.txt). It's 4767 \* 32. 

I just realized that my code is reordering the heatmap but plotting the wrong dendrogram because it doesn't touch how that works at all. There's no option in dendrogram to specify the ordering of the tree, should there be? Alternatively it could be an option, reorder=True or something. But it wouldn't be good to force people to calculate the same thing twice, because this algorithm is rather slow on large data.

I would LOVE this feature. Thank you all for your efforts in making these tools even better.

As I said above I am happy to put together a PR but I would like some guidance/discussion on how to handle a few things:
- Just how should this be integrated into the module, anyway? A separate method, an option somewhere, or some combination?
- The algorithm is fairly slow (I _think_ this is unavoidable) and can take a very long time on large datasets
- It's unclear how it should be integrated with the dendrogram function, which currently doesn't provide any way to specify ordering. Given how long it takes to calculate an ordering, we shouldn't require anyone to do it twice on one tree.

If someone more knowledgeable than me could weigh in on this I will submit a PR as soon as I can get to it. I have the algorithm implemented already so it's mostly a matter of formatting and refactoring.

-bump-

I made @markak's code python3 compatible and put it inside a class namespace [here](https://gist.github.com/mlovci/dc23a1d30076746cf0d8). This is also not ready for scipy, but may help anyone who is up for doing that.

@jamestwebber I believe the way that Bar-Joseph originally created the program was to calculate linkage on its own, and then give that linkage to the optimal leaf ordering function. From a usability standpoint, this is rather annoying so it'd be much easier if it was an option in `scipy.cluster.hieararchy.linkage`, though it would also be very helpful if it was a separate function so it could be used with other linkage matrices as in from [`fastcluster`](http://danifold.net/fastcluster.html)

> @rgommers in terms of having a example from biology, I'm happy to find one but what is a reasonable size? One matrix I've used before (for intro-bioinformatics stuff) is a cleaned-up version of Gasch et al. 2000, which as it happens I put on GitHub in this repository. It's 4767 \* 32. 

Thanks @jamestwebber, that's useful. Apologies for taking ages to get back to looking at this. It doesn't take much effort to produce something reasonable-looking for this size data, which is what I wanted to verify:

![gasch_dendrogram](https://cloud.githubusercontent.com/assets/98330/18814557/4187486a-8373-11e6-8a1e-d8491ddd7a28.png)

Code:

```
import os

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.cluster import hierarchy

# Load linkage matrix from Gasch et al. (see
# https://github.com/scipy/scipy/issues/5238)
gasch_data = os.path.join(os.environ['HOME'], 'Code', 'tmp',
                          'bootcamp-programming', 'bootcamp', 'data',
                          'experiment_data.txt')

dists = pd.io.parsers.read_table(gasch_data, index_col=0)
dists = np.asarray(dists)

Z = hierarchy.linkage(dists, 'ward')

plt.figure()
dn = hierarchy.dendrogram(Z, truncate_mode='lastp', p=40, show_contracted=True,
                          orientation='right')
plt.show()
```

> it'd be much easier if it was an option in `scipy.cluster.hierarchy.linkage`, though it would also be very helpful if it was a separate function so it could be used with other linkage matrices as in from `fastcluster`

That sounds like a good way to implement this.

> The algorithm is fairly slow (I think this is unavoidable) and can take a very long time on large datasets

It's really slow, O(n^4) says the paper. It isn't clear to me from a look at your code whether you implemented the early termination of search paths that the paper discusses. If not, that would make a big difference.

The other thing to do is Cythonize the `optimal_scores` function. It's a triple for-loop, so should be much faster in Cython.

> It's unclear how it should be integrated with the dendrogram function, which currently doesn't provide any way to specify ordering. Given how long it takes to calculate an ordering, we shouldn't require anyone to do it twice on one tree.

If it's an option in `linkage` as suggested by @olgabot then there's nothing to be done inside `dendrogram`. `dendrogram` uses the provided linkage matrix as is, and it would be best to keep it that way I think.

In terms of the dendrogram you produced: I'm not sure what the labels on the leaves mean, so I can't say if it's reasonable. Is that a clustering of rows, with the number of rows in each cluster?

The file is a matrix of gene expression data--the columns are different experimental conditions (including replicates), and rows are different genes (in yeast). The simplest way to cluster it is to cluster the columns, which should group together replicates first and then similar conditions. I believe Ward clustering was able to do that when I tried it.

The column labels aren't meaningful because it was part of a project, here are real names: [gasch_randomized_labels.txt](https://github.com/scipy/scipy/files/491831/gasch_randomized_labels.txt)

Implementing it inside `linkage` makes sense, but will require some rewriting. I'm not sure how much it can be done on the fly, if at all.

> It's really slow, O(n^4) says the paper. It isn't clear to me from a look at your code whether you implemented the early termination of search paths that the paper discusses. If not, that would make a big difference.

I'm pretty sure I did this. It's quite possible I did something wrong somewhere, but it might just be that slow.

> The other thing to do is Cythonize the optimal_scores function. It's a triple for-loop, so should be much faster in Cython.

I have only written a tiny bit of Cython but I could try this. Hopefully I will have time to work on this soon.

> Implementing it inside linkage makes sense, but will require some rewriting. I'm not sure how much it can be done on the fly, if at all.

It would only be a few lines extra. You keep your standalone function, and then add a keyword to `linkage` (something like `optimal_order=False`) and then at the end of `linkage` add:

   if optimal_order is True:
        Z = optimal_order(Z)

> I have only written a tiny bit of Cython but I could try this. Hopefully I will have time to work on this soon.

If it takes you too much time, I think we can merge the Python version and do the Cython conversion later.

The only issue that my code doesn't return a new linkage matrix, just the ordering. So it will take a bit of rewriting to make that work.

> In terms of the dendrogram you produced: I'm not sure what the labels on the leaves mean, so I can't say if it's reasonable. Is that a clustering of rows, with the number of rows in each cluster?

Yes, that was row clustering. I meant it visually made sense, didn't know what was in the data. Here is the column clustering with labels, seems about right:

![gasch_rowclustering](https://cloud.githubusercontent.com/assets/98330/18826205/bd4d7bc6-8428-11e6-9163-f0178d4bd9f8.png)

```
import os

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy.cluster import hierarchy

# Load linkage matrix from Gasch et al. (see
# https://github.com/scipy/scipy/issues/5238)
basedir = os.path.join(os.environ['HOME'], 'Code', 'tmp',
                       'bootcamp-programming', 'bootcamp', 'data')
gasch_data = os.path.join(basedir, 'experiment_data.txt')
gasch_labels = os.path.join(basedir, 'gasch_randomized_labels.txt')

dists = pd.io.parsers.read_table(gasch_data, index_col=0)
colnames = pd.io.parsers.read_table(gasch_labels, index_col=1, names='abc')

Z = hierarchy.linkage(np.asarray(dists).T, 'ward')

plt.figure()
dn = hierarchy.dendrogram(Z, labels=np.asarray(colnames.index),
                          orientation='right')
plt.tight_layout()
plt.show()
```

I knew there was a reason @olgabot rang a bell...now that we will be co-workers I guess I should get around to writing up a PR.
(of course it still takes me a week and a half to start looking at this)

As is referenced above, since this issue opened it looks like @adrianveres has implemented a very nice Cython version of the algorithm. The code is MIT licensed so it should be straight-forward to incorporate it into Scipy with proper attribution. @olgabot has already done some of the work there to make it 2.7 compatible.

So, that is what I'm going to work on now. Happy to get advice from anyone else who wants to be involved.
Unfortunately I'm having trouble building and thus testing my code, lots of errors like:
```
/var/folders/y3/27clvgw94dxfmsj5qnz221qh0000gn/T//cc0AVEe0.s:1589533:2: error: ambiguous instructions require an explicit suffix (could be 'filds', or 'fildl')
error: Command "g++ -fno-strict-aliasing -I/Users/james/anaconda2/include -arch x86_64 -DNDEBUG -g -fwrapv -O3 -Wall -D__STDC_FORMAT_MACROS=1 -Iscipy/sparse/sparsetools -I/Users/james/anaconda2/lib/python2.7/site-packages/numpy/core/include -I/Users/james/anaconda2/include/python2.7 -c scipy/sparse/sparsetools/bsr.cxx -o build/temp.macosx-10.7-x86_64-2.7/scipy/sparse/sparsetools/bsr.o" failed with exit status 1
```
I suspect this is an issue with my build setup in some way or another but I'm not sure what will solve it. I'm running Anaconda 4.3.22 on OS X 10.12.5, with gcc 4.8.5 (just installed it).

Resolving patch:
diff --git a/scipy/cluster/hierarchy.py b/scipy/cluster/hierarchy.py
--- a/scipy/cluster/hierarchy.py
+++ b/scipy/cluster/hierarchy.py
@@ -60,6 +60,7 @@
    leaves_list
    to_tree
    cut_tree
+   optimal_leaf_ordering
 
 These are predicates for checking the validity of linkage and
 inconsistency matrices as well as for checking isomorphism of two
@@ -190,8 +191,9 @@
            'from_mlab_linkage', 'inconsistent', 'is_isomorphic',
            'is_monotonic', 'is_valid_im', 'is_valid_linkage', 'leaders',
            'leaves_list', 'linkage', 'maxRstat', 'maxdists', 'maxinconsts',
-           'median', 'num_obs_linkage', 'set_link_color_palette', 'single',
-           'to_mlab_linkage', 'to_tree', 'ward', 'weighted', 'distance']
+           'median', 'num_obs_linkage', 'optimal_leaf_ordering',
+           'set_link_color_palette', 'single', 'to_mlab_linkage', 'to_tree',
+           'ward', 'weighted', 'distance']
 
 
 class ClusterWarning(UserWarning):
@@ -481,7 +483,7 @@ def ward(y):
     return linkage(y, method='ward', metric='euclidean')
 
 
-def linkage(y, method='single', metric='euclidean'):
+def linkage(y, method='single', metric='euclidean', optimal_ordering=False):
     """
     Perform hierarchical/agglomerative clustering.
 
@@ -629,6 +631,14 @@ def linkage(y, method='single', metric='euclidean'):
         observation vectors; ignored otherwise. See the ``pdist``
         function for a list of valid distance metrics. A custom distance
         function can also be used.
+    optimal_ordering : bool, optional
+        If True, the linkage matrix will be reordered so that the distance
+        between successive leaves is minimal. This results in a more intuitive
+        tree structure when the data are visualized. defaults to False, because
+        this algorithm can be slow, particularly on large datasets [2]_. See 
+        also the `optimal_leaf_ordering` function.
+        
+        .. versionadded:: 1.0.0
 
     Returns
     -------
@@ -660,6 +670,9 @@ def linkage(y, method='single', metric='euclidean'):
     ----------
     .. [1] Daniel Mullner, "Modern hierarchical, agglomerative clustering
            algorithms", :arXiv:`1109.2378v1`.
+    .. [2] Ziv Bar-Joseph, David K. Gifford, Tommi S. Jaakkola, "Fast optimal
+           leaf ordering for hierarchical clustering", 2001. Bioinformatics
+           https://doi.org/10.1093/bioinformatics/17.suppl_1.S22
 
     Examples
     --------
@@ -703,12 +716,18 @@ def linkage(y, method='single', metric='euclidean'):
 
     n = int(distance.num_obs_y(y))
     method_code = _LINKAGE_METHODS[method]
+
     if method == 'single':
-        return _hierarchy.mst_single_linkage(y, n)
+        result = _hierarchy.mst_single_linkage(y, n)
     elif method in ['complete', 'average', 'weighted', 'ward']:
-        return _hierarchy.nn_chain(y, n, method_code)
+        result = _hierarchy.nn_chain(y, n, method_code)
+    else:
+        result = _hierarchy.fast_linkage(y, n, method_code)
+
+    if optimal_ordering:
+        return optimal_leaf_ordering(result, y)
     else:
-        return _hierarchy.fast_linkage(y, n, method_code)
+        return result
 
 
 class ClusterNode:
@@ -1122,6 +1141,71 @@ def to_tree(Z, rd=False):
         return nd
 
 
+def optimal_leaf_ordering(Z, y, metric='euclidean'):
+    """
+    Given a linkage matrix Z and distance, reorder the cut tree.
+
+    Parameters
+    ----------
+    Z : ndarray
+        The hierarchical clustering encoded as a linkage matrix. See
+        `linkage` for more information on the return structure and
+        algorithm.
+    y : ndarray
+        The condensed distance matrix from which Z was generated.
+        Alternatively, a collection of m observation vectors in n
+        dimensions may be passed as a m by n array.
+    metric : str or function, optional
+        The distance metric to use in the case that y is a collection of
+        observation vectors; ignored otherwise. See the ``pdist``
+        function for a list of valid distance metrics. A custom distance
+        function can also be used.
+    
+    Returns
+    -------
+    Z_ordered : ndarray
+        A copy of the linkage matrix Z, reordered to minimize the distance
+        between adjacent leaves.
+
+    Examples
+    --------
+    >>> from scipy.cluster import hierarchy
+    >>> np.random.seed(23)
+    >>> X = np.random.randn(10,10)
+    >>> Z = hierarchy.ward(X)
+    >>> hierarchy.leaves_list(Z)
+    array([0, 5, 3, 9, 6, 8, 1, 4, 2, 7], dtype=int32)
+    >>> hierarchy.leaves_list(hierarchy.optimal_leaf_ordering(Z, X))
+    array([3, 9, 0, 5, 8, 2, 7, 4, 1, 6], dtype=int32)
+    
+    """
+    from . import _optimal_leaf_ordering
+
+    Z = np.asarray(Z, order='c')
+    is_valid_linkage(Z, throw=True, name='Z')
+
+    y = _convert_to_double(np.asarray(y, order='c'))
+
+    if y.ndim == 1:
+        distance.is_valid_y(y, throw=True, name='y')
+        [y] = _copy_arrays_if_base_present([y])
+    elif y.ndim == 2:
+        if y.shape[0] == y.shape[1] and np.allclose(np.diag(y), 0):
+            if np.all(y >= 0) and np.allclose(y, y.T):
+                _warning('The symmetric non-negative hollow observation '
+                         'matrix looks suspiciously like an uncondensed '
+                         'distance matrix')
+        y = distance.pdist(y, metric)
+    else:
+        raise ValueError("`y` must be 1 or 2 dimensional.")
+
+    if not np.all(np.isfinite(y)):
+        raise ValueError("The condensed distance matrix must contain only "
+                         "finite values.")
+
+    return _optimal_leaf_ordering.optimal_leaf_ordering(Z, y)
+
+
 def _convert_to_bool(X):
     if X.dtype != bool:
         X = X.astype(bool)
diff --git a/scipy/cluster/setup.py b/scipy/cluster/setup.py
--- a/scipy/cluster/setup.py
+++ b/scipy/cluster/setup.py
@@ -27,6 +27,10 @@ def configuration(parent_package='', top_path=None):
         sources=[('_hierarchy.c')],
         include_dirs=[get_numpy_include_dirs()])
 
+    config.add_extension('_optimal_leaf_ordering',
+        sources=[('_optimal_leaf_ordering.c')],
+        include_dirs=[get_numpy_include_dirs()])
+
     return config
 
 


Test patch:
diff --git a/scipy/cluster/tests/hierarchy_test_data.py b/scipy/cluster/tests/hierarchy_test_data.py
--- a/scipy/cluster/tests/hierarchy_test_data.py
+++ b/scipy/cluster/tests/hierarchy_test_data.py
@@ -59,6 +59,13 @@
                                  [1., 6., 347.5, 3.],
                                  [8., 9., 670.125, 6.]])
 
+# the optimal leaf ordering of linkage_ytdist_single
+linkage_ytdist_single_olo = array([[5., 2., 138., 2.],
+                                   [4., 3., 219., 2.],
+                                   [7., 0., 255., 3.],
+                                   [1., 8., 268., 4.],
+                                   [6., 9., 295., 6.]])
+
 X = array([[1.43054825, -7.5693489],
            [6.95887839, 6.82293382],
            [2.87137846, -9.68248579],
@@ -84,6 +91,13 @@
                         [6., 8., 9.10208346, 4.],
                         [7., 9., 24.7784379, 6.]])
 
+# the optimal leaf ordering of linkage_X_ward
+linkage_X_ward_olo = array([[4., 3., 0.36265956, 2.],
+                            [5., 1., 1.77045373, 2.],
+                            [2., 0., 2.55760419, 2.],
+                            [6., 8., 9.10208346, 4.],
+                            [7., 9., 24.7784379, 6.]])
+
 inconsistent_ytdist = {
     1: array([[138., 0., 1., 0.],
               [219., 0., 1., 0.],
diff --git a/scipy/cluster/tests/test_hierarchy.py b/scipy/cluster/tests/test_hierarchy.py
--- a/scipy/cluster/tests/test_hierarchy.py
+++ b/scipy/cluster/tests/test_hierarchy.py
@@ -46,8 +46,8 @@
     is_isomorphic, single, leaders, complete, weighted, centroid,
     correspond, is_monotonic, maxdists, maxinconsts, maxRstat,
     is_valid_linkage, is_valid_im, to_tree, leaves_list, dendrogram,
-    set_link_color_palette, cut_tree, _order_cluster_tree,
-    _hierarchy, _LINKAGE_METHODS)
+    set_link_color_palette, cut_tree, optimal_leaf_ordering,
+    _order_cluster_tree, _hierarchy, _LINKAGE_METHODS)
 from scipy.spatial.distance import pdist
 from scipy.cluster._hierarchy import Heap
 
@@ -116,6 +116,11 @@ def test_compare_with_trivial(self):
             Z = linkage(d, method)
             assert_allclose(Z_trivial, Z, rtol=1e-14, atol=1e-15)
 
+    def test_optimal_leaf_ordering(self):
+        Z = linkage(hierarchy_test_data.ytdist, optimal_ordering=True)
+        expectedZ = getattr(hierarchy_test_data, 'linkage_ytdist_single_olo')
+        assert_allclose(Z, expectedZ, atol=1e-10)
+
 
 class TestLinkageTies(object):
     _expectations = {
@@ -1009,6 +1014,20 @@ def test_cut_tree():
                  cut_tree(Z, height=[10, 5]))
 
 
+def test_optimal_leaf_ordering():
+    # test with the distance vector y
+    Z = optimal_leaf_ordering(linkage(hierarchy_test_data.ytdist),
+                              hierarchy_test_data.ytdist)
+    expectedZ = hierarchy_test_data.linkage_ytdist_single_olo
+    assert_allclose(Z, expectedZ, atol=1e-10)
+
+    # test with the observation matrix X
+    Z = optimal_leaf_ordering(linkage(hierarchy_test_data.X, 'ward'),
+                              hierarchy_test_data.X)
+    expectedZ = hierarchy_test_data.linkage_X_ward_olo
+    assert_allclose(Z, expectedZ, atol=1e-06)
+
+
 def test_Heap():
     values = np.array([2, -1, 0, -1.5, 3])
     heap = Heap(values)