Issue 1:
I'm trying to perform hierarchical clustering on sparse observation matrix. The matrix represents movie ratings for a number of users. However, running linkage([ndarray], method='single')

causes infinite recursion and linkage([ndarray], method='single', metric='cosine') crashes. ward works fine.

Cheers.

Reproducing code example:

Data source: https://www.librec.net/datasets/filmtrust.zip

Data format (ratings.dat):

[...]
36 6 4
36 250 3.5
36 219 3.5
36 234 3
36 211 4
[...]

import numpy as np

from scipy.sparse import dok_matrix
from scipy.cluster import hierarchy

def load_ratings(path):
ratings = {}
items = set([])

with open(path, 'r') as data:
for line in data.readlines():
rating = [e for e in line.replace('\n', '').split(' ')]
items.add(int(rating[1]))

if not int(rating[0]) in ratings:
ratings[int(rating[0])] = {}

ratings[int(rating[0])][int(rating[1])] = float(rating[2])

return (ratings, items)

ratings, items = load_ratings('ratings.txt')

R = dok_matrix((max(items) + 1, max(ratings) + 1), dtype=np.float32)

for user in ratings:
for item in ratings[user]:
R[item, user] = ratings[user][item]

R = R.transpose().toarray()
Z = hierarchy.linkage(R, method='single')

# This exceeds maximum recursion depth
hierarchy.dendrogram(Z)

Input matrix (R):

[[ 0. 0. 0. ..., 0. 0. 0. ]
[ 0. 2. 4. ..., 0. 0. 0. ]
[ 0. 0. 0. ..., 0. 0. 0. ]
...,
[ 0. 0. 0. ..., 0. 0. 0. ]
[ 0. 0. 4. ..., 0. 0. 0. ]
[ 0. 1.5 3. ..., 0. 0. 0. ]]

Error message 1

[...]
651, in _dendrogram_calculate_info
above_threshold_color=above_threshold_color)
File "/home/siegmeyer/.local/lib/python3.5/site-packages/scipy/cluster/hierarchy.py", line 2651, in _dendrogram_calculate_info
above_threshold_color=above_threshold_color)
File "/home/siegmeyer/.local/lib/python3.5/site-packages/scipy/cluster/hierarchy.py", line 2651, in _dendrogram_calculate_info
above_threshold_color=above_threshold_color)
File "/home/siegmeyer/.local/lib/python3.5/site-packages/scipy/cluster/hierarchy.py", line 2618, in _dendrogram_calculate_info
above_threshold_color=above_threshold_color)
File "/home/siegmeyer/.local/lib/python3.5/site-packages/scipy/cluster/hierarchy.py", line 2530, in _dendrogram_calculate_info
leaf_label_func, i, labels)
File "/home/siegmeyer/.local/lib/python3.5/site-packages/scipy/cluster/hierarchy.py", line 2403, in _append_singleton_leaf_node
ivl.append(str(int(i)))
RecursionError: maximum recursion depth exceeded while getting the str of an object

Error message 2 (with metric='cosine')

Traceback (most recent call last):
File "cosine_similarity.py", line 100, in <module>
Z = hierarchy.linkage(R, metric='cosine')
File "/home/siegmeyer/.local/lib/python3.5/site-packages/scipy/cluster/hierarchy.py", line 676, in linkage
raise ValueError("The condensed distance matrix must contain only finite values.")
ValueError: The condensed distance matrix must contain only finite values.

Scipy/Numpy/Python version information:

0.19.0 1.12.1 sys.version_info(major=3, minor=5, micro=2, releaselevel='final', serial=0)

Issue 2:
Original ticket http://projects.scipy.org/scipy/ticket/612 on 2008-02-26 by trac user rspringuel, assigned to @cournape.

The current cluster package for scipy implements only k-means clustering for a euclidean distance metric with centroids computed as the mean of the members of each cluster.

Pycluster implements 8 different distance metrics, three different centroid methods (mean, median, and medoid) and four different clustering algorithms (k-means, agglomerative hierarchical, SOM, and PCA) but contains non-BSD style license compatible elements and so cannot be incorporated into scipy.

Using Pycluster as a model, I have written a clustering package from scratch that duplicates most of its functionality and expands on it in certain areas as well.

I have also endevored to design the package to make future expansions easy and straight forward.

To date what I have written supports the following:

Distances:

euclidean

normalized euclidean

city block (aka manhattan)

normalized city block

hamming (aka simple mapping coefficient)

pearson

absolute pearson

uncentered pearson

arccosine of pearson

absolute uncentered pearson

spearman

kendall

modified simple matching coefficent of Rogers and Tanimoto

modified simple matching coefficent of Sokal and Sneath

jaccard coefficent

modified jaccard coefficent of Dice

modified jaccard coefficent of Sokal and Sneath

general Minkowski metric

Chebychev distance

Centroids:

arithmetic mean

median

absolute mean

geometric mean

harmonic mean

quadratic mean

mediod (using any of the above distances)

Clustering algorithms:

k-means

c-means (fuzzy clustering)

agglomerative hierarchical

Additional distances, centroid methods, and clustering algorithms may be added as my work requires them.

Since I have written all of this code from scratch, I control the license to it and have elected to release it under a BSD-style license so that it can be incorporated into scipy.

What I have written obviously greatly expands on the functionality of the current cluster package, but it is written entirely in python as so may be slower than what is currently present where the functionality overlaps (hence the quotation marks around "better" in the title of this ticket).

Note: All of the centroid methods are based on my statistical functions submitted in ticket gh-1131 and so the code would have to be revised should those stats functions not be incorporated into scipy.