Repository: scipy/scipy

Issue:
linkage() function mistakes distance matrix as observation vectors
the document for scipy.cluster.hierarchy.linkage() function states that it accepts redundant distance matrix as input:

http://docs.scipy.org/doc/scipy/reference/generated/scipy.cluster.hierarchy.linkage.html
"y : ndarray
A condensed or redundant distance matrix. A condensed distance matrix is a flat array containing the upper triangular of the distance matrix. This is the form that pdist returns. Alternatively, a collection of m observation vectors in n dimensions may be passed as an m by n array."

This feature of accepting redundant distance matrix as input is also introduced by many popular books like "SciPy and NumPy: An Overview for Developers" By Eli Bressert:
http://books.google.co.uk/books?id=c-xzkDMDev0C&pg=PA48&dq=scipy+linkage+input&hl=en&sa=X&ei=4BzUUZqpNoSbPaj3gLgN&ved=0CDgQ6AEwAQ#v=onepage&q=scipy%20linkage%20input&f=false

However, when distance matrix is specified as input to the linkage() function, it is silently considered to be observation vectors. This can be seen by reading the source code :
https://github.com/scipy/scipy/blob/master/scipy/cluster/hierarchy.py

If the input is a 2-dimensional matrix, it is not checked by predicate scipy.spatial.distance.is_valid_dm(). Rather, it is simply taken as observation vectors for hierarchical clustering. It may be also seen from the following example:

``` python
arr = scipy.array([[0,5],[1,5],[-1,5],[0,-5],[1,-5],[-1,-5], [-1.1,-5]])
pd = pdist(arr)
dm = squareform(pd)
linkage(pd)
linkage(dm)  # result is different from the previous 
```

I think this issue has relatively serious consequence if users do not check their clustering results carefully. 

I suggest: either update the documentation for linkage() function to reflect the real functionality, or add a predicate check using scipy.spatial.distance.is_valid_dm() if two dimensional matrix is given as input so distance matrix is processed properly in the linkage() function.



Discussion:
@hongbo-zhu-cn    So how how is it in the end? I am also facing this problem. Very serious if one didn't look carefully. Now can I directly feed my distance matrix in or have to make it condensed first?

@perfectionming Yes the bug is indeed serious. I just wonder how much code is still around and how many results had been published using the code :-) It is not yet fixed, I believe. So you will have to take care of your own matrix before running linkage().

@hongbo-zhu-cn  LOL on "how many results had been published using the code". Ok so given the status quo, to be safe, I better pass the condensed version of it. Please help check whether my solution to this problem in StackOverflow (http://stackoverflow.com/questions/18952587/use-distance-matrix-in-scipy-cluster-hierarchy-linkage) is correct. Thanks! :+1:  Oh, BTW, 我们是同胞:)

I agree this is really misleading and that nearly everyone who uses this function probably uses distance matrices which the function treats as arrays of observations.


Resolving patch:
diff --git a/scipy/cluster/hierarchy.py b/scipy/cluster/hierarchy.py
--- a/scipy/cluster/hierarchy.py
+++ b/scipy/cluster/hierarchy.py
@@ -195,8 +195,12 @@
            'to_mlab_linkage', 'to_tree', 'ward', 'weighted', 'distance']
 
 
+class ClusterWarning(UserWarning):
+    pass
+
+
 def _warning(s):
-    warnings.warn('scipy.cluster: %s' % s, stacklevel=3)
+    warnings.warn('scipy.cluster: %s' % s, ClusterWarning, stacklevel=3)
 
 
 def _copy_array_if_base_present(a):
@@ -339,8 +343,8 @@ def centroid(y):
     """
     Performs centroid/UPGMC linkage.
 
-    See ``linkage`` for more information on the return structure
-    and algorithm.
+    See ``linkage`` for more information on the input matrix,
+    return structure, and algorithm.
 
     The following are common calling conventions:
 
@@ -359,7 +363,7 @@ def centroid(y):
     Parameters
     ----------
     y : ndarray
-        A condensed or redundant distance matrix. A condensed
+        A condensed distance matrix. A condensed
         distance matrix is a flat array containing the upper
         triangular of the distance matrix. This is the form that
         ``pdist`` returns. Alternatively, a collection of
@@ -405,7 +409,7 @@ def median(y):
     Parameters
     ----------
     y : ndarray
-        A condensed or redundant distance matrix. A condensed
+        A condensed distance matrix. A condensed
         distance matrix is a flat array containing the upper
         triangular of the distance matrix. This is the form that
         ``pdist`` returns. Alternatively, a collection of
@@ -427,7 +431,7 @@ def median(y):
 
 def ward(y):
     """
-    Performs Ward's linkage on a condensed or redundant distance matrix.
+    Performs Ward's linkage on a condensed distance matrix.
 
     See linkage for more information on the return structure
     and algorithm.
@@ -447,7 +451,7 @@ def ward(y):
     Parameters
     ----------
     y : ndarray
-        A condensed or redundant distance matrix. A condensed
+        A condensed distance matrix. A condensed
         distance matrix is a flat array containing the upper
         triangular of the distance matrix. This is the form that
         ``pdist`` returns. Alternatively, a collection of
@@ -469,15 +473,18 @@ def ward(y):
 
 def linkage(y, method='single', metric='euclidean'):
     """
-    Performs hierarchical/agglomerative clustering on the condensed
-    distance matrix y.
+    Performs hierarchical/agglomerative clustering.
+    
+    The input y may be either a 1d compressed distance matrix
+    or a 2d array of observation vectors.
 
-    y must be a :math:`{n \\choose 2}` sized
+    If y is a 1d compressed distance matrix,
+    then y must be a :math:`{n \\choose 2}` sized
     vector where n is the number of original observations paired
     in the distance matrix. The behavior of this function is very
     similar to the MATLAB linkage function.
 
-    An :math:`(n-1)` by 4  matrix ``Z`` is returned. At the
+    A :math:`(n-1)` by 4 matrix ``Z`` is returned. At the
     :math:`i`-th iteration, clusters with indices ``Z[i, 0]`` and
     ``Z[i, 1]`` are combined to form cluster :math:`n + i`. A
     cluster with an index less than :math:`n` corresponds to one of
@@ -598,7 +605,7 @@ def linkage(y, method='single', metric='euclidean'):
     Parameters
     ----------
     y : ndarray
-        A condensed or redundant distance matrix. A condensed distance matrix
+        A condensed distance matrix. A condensed distance matrix
         is a flat array containing the upper triangular of the distance matrix.
         This is the form that ``pdist`` returns. Alternatively, a collection of
         :math:`m` observation vectors in n dimensions may be passed as an
@@ -653,6 +660,11 @@ def linkage(y, method='single', metric='euclidean'):
         if method in _EUCLIDEAN_METHODS and metric != 'euclidean':
             raise ValueError("Method '{0}' requires the distance metric "
                              "to be Euclidean".format(method))
+        if y.shape[0] == y.shape[1] and np.allclose(np.diag(y), 0):
+            if np.all(y >= 0) and np.allclose(y, y.T):
+                _warning('The symmetric non-negative hollow observation '
+                         'matrix looks suspiciously like an uncondensed '
+                         'distance matrix')
         y = distance.pdist(y, metric)
     else:
         raise ValueError("`y` must be 1 or 2 dimensional.")


Test patch:
diff --git a/scipy/cluster/tests/test_hierarchy.py b/scipy/cluster/tests/test_hierarchy.py
--- a/scipy/cluster/tests/test_hierarchy.py
+++ b/scipy/cluster/tests/test_hierarchy.py
@@ -36,14 +36,15 @@
 
 import numpy as np
 from numpy.testing import (TestCase, run_module_suite, dec, assert_raises,
-                           assert_allclose, assert_equal, assert_)
+                           assert_allclose, assert_equal, assert_, assert_warns)
 
 from scipy._lib.six import xrange, u
 
 import scipy.cluster.hierarchy
 from scipy.cluster.hierarchy import (
-    linkage, from_mlab_linkage, to_mlab_linkage, num_obs_linkage, inconsistent,
-    cophenet, fclusterdata, fcluster, is_isomorphic, single, leaders,
+    ClusterWarning, linkage, from_mlab_linkage, to_mlab_linkage,
+    num_obs_linkage, inconsistent, cophenet, fclusterdata, fcluster,
+    is_isomorphic, single, leaders, complete, weighted, centroid,
     correspond, is_monotonic, maxdists, maxinconsts, maxRstat,
     is_valid_linkage, is_valid_im, to_tree, leaves_list, dendrogram,
     set_link_color_palette, cut_tree, _order_cluster_tree)
@@ -898,6 +899,14 @@ def calculate_maximum_inconsistencies(Z, R, k=3):
     return B
 
 
+def within_tol(a, b, tol):
+    return np.abs(a - b).max() < tol
+
+
+def test_unsupported_uncondensed_distance_matrix_linkage_warning():
+    assert_warns(ClusterWarning, linkage, [[0, 1], [1, 0]])
+
+
 def test_euclidean_linkage_value_error():
     for method in scipy.cluster.hierarchy._EUCLIDEAN_METHODS:
         assert_raises(ValueError, linkage, [[1, 1], [1, 1]],