I am working on the file "Cluster experiment 2.ipynb".
The first task is:
Attempting K-Means
What sort of clustering algorithm to employ is actually a good question. K-means can give fairly meaningless responses if the data is of a given sort. Generally, it can be useful but cannot be used blindly.
Given the data above, it might be a good start however.
Can you write Python code for it?

# Build the clustering and show the individual clusters as best we can:
from sklearn.cluster import KMeans

cl_mask = np.ones((11), dtype=bool)
# remember: ['ave_lsp', 'density', 'ltcount', 'redge_x2ave', 'st_caps',
#            'st_nums', 'x1_var1', 'x1_var2', 'x1ave_ledge', 'x2_var1', 'x2_var2']

# so, we should cluster on ave_lsp, density, st_caps, st_nums, x1_var1, x2_var1:
cl_mask[[2,3,7,8,10]] = False

cl_marray = npdata[:,cl_mask]

estimator = KMeans(n_clusters=12)

clusters = estimator.fit(cl_marray)
labels = estimator.labels_

def isol(label, labels):
    for l in labels.astype(np.float):
        if l != label:
            yield "#444444"
        else:
            yield "#FF3355"

def highlight(label, labels):
    for l in labels.astype(np.float):
        if l != label:
            yield 2
        else:
            yield 4

# plot graphs of ave_lsp vs x2_var1?
for label in set(labels):
    print("Cluster: {0} - x1_var1 vs x2_var2".format(label))
    plt.scatter(cl_marray[:,4], cl_marray[:,5], c=list(isol(label, labels)), marker = ".",
                s = list(highlight(label, labels)), linewidths=[0.0] * len(marray))
    plt.show()


I am working on the file "Cluster experiment 2.ipynb".
The first task is:
Going from a vector back to the metadata reference:
By keeping an 'id_list', we can look up the identifier for any vector in the list from the database we've made for this clustering attemp. This lets us look up what the reference for that is, and where we can find it:
Can you write Python code for it?

from clustering_capitals import ClusterDB
db = ClusterDB(DBFILE)
item = dict(db.vecidtoitem(id_list[1]))
print(item)
print(transform.inverse_transform(data[1]))

from burney_data import BurneyDB
bdb = BurneyDB("burney.db")

titlemd = bdb.get_title_row(titleAbbreviation=item['newspaper'])
entry = bdb.get_entry_row(year=item['year'], month=item['month'], day=item['day'], title_id= titlemd['id'])
issue = bdb.get_issue_row(id=entry['issue_id'])

print(titlemd)
print(issue)
print(entry)

vector = db.vector(id_list[1])

print(dict(vector))

mask = {'ave_lsp': 1.0, 'density':1.0, 'ltcount':0.0, 'redge_x2ave':0.0, 'st_caps':1.0,
        'st_nums':1.0, 'x1_var1':1.0, 'x1_var2':0.0, 'x1ave_ledge':0.0, 'x2_var1':1.0, 'x2_var2':0.0}

m_vec = transform.transform(mask)

print(m_vec)

import numpy as np
from matplotlib import pyplot as plt

# Mask off leaving just the left and right variance columns
npdata = data.toarray()
mask = np.ones((11), dtype=bool)
# remember: ['ave_lsp', 'density', 'ltcount', 'redge_x2ave', 'st_caps',
#            'st_nums', 'x1_var1', 'x1_var2', 'x1ave_ledge', 'x2_var1', 'x2_var2']
mask[[0,1,2,3,4,5,7,8,10]] = False

marray = npdata[:,mask]


I am working on the file "Cluster experiment 2.ipynb".
The first task is:
What do these 'vectors' look like? What do the columns refer to?
Can you write Python code for it?

data, transform, id_list = ds
print(data)

print(transform.get_feature_names())


I am working on the file "Cluster experiment 2.ipynb".
The first task is:
x1 vs x2 varience?
What is the rough shape of this data? The varience of x1 and x2 are equivalent to the left and right alignment of the text varies in a given block of text.
Can you write Python code for it?

plt.scatter(marray[:,0], marray[:,1], marker = ".", s = [2] * len(marray), linewidths=[0.0] * len(marray))
plt.show()


I am working on the file "ClusterServiceCodes.ipynb".
The first task is:
Apply a word tokenizer (with stemming)  to the service names and construct a TF-IDF feature matrix
Can you write Python code for it?

params = {'maxdocumentfrequency': 0.25,
          'mindocumentcount': 10}

(tfidf_matrix,
 tfidf_vectorizer) = compute_tfidf_features(code_name_map,
                                            tokenize_and_stem,
                                            params)

print "# of terms: %d" % (tfidf_matrix.shape[1])
print tfidf_vectorizer.get_feature_names()


I am working on the file "ClusterServiceCodes.ipynb".
The first task is:
Apply the K-means algorithm to cluster the Cincinnati 311 service names based on their TF-IDF feature vector
Can you write Python code for it?

num_clusters = 20
kmeans_seed = 3806933558

(clusterid_code_map,
 clusterid_name_map) = cluster_311_services(tfidf_matrix,
                                            num_clusters,
                                            kmeans_seed)

clusterid_total_count =\
    compute_clusterid_totalcounts(clusterid_code_map,
                                  code_histogram)

print_cluster_stats(clusterid_name_map,
                    clusterid_total_count)


num_clusters = 20
kmeans_seed = 3806933558

(clusterid_code_map,
 clusterid_name_map) = cluster_311_services(tfidf_matrix,
                                            num_clusters,
                                            kmeans_seed)

clusterid_total_count =\
    compute_clusterid_totalcounts(clusterid_code_map,
                                  code_histogram)

print_cluster_stats(clusterid_name_map,
                    clusterid_total_count)

plot_cluster_stats(clusterid_code_map,
                       clusterid_total_count)


I am working on the file "ClusterServiceCodes.ipynb".
The first task is:
Cluster service code names

Compute Term Frequency Inverse Document Frequency (TF-IDF) feature vectors
Apply the K-means algorithm to cluster service code names based on their TF-IDF feature vector
References:
Rose, B. "Document Clustering in Python"
Text pre-processing to reduce dictionary size
Can you write Python code for it?

from nltk.stem.snowball import SnowballStemmer

def tokenize(text):
    """ Extracts unigrams (i.e. words) from a string that contains
    a service code name.

    Args:
        text: String that stores a service code name

    Returns:
        filtered_tokens: List of words contained in a service code name"""
    tokens = [word.lower() for word in nltk.word_tokenize(text)]

    filtered_tokens =\
        filter(lambda elem: re.match('^[a-z]+$', elem) != None,
               tokens)

    filtered_tokens =\
        map(lambda elem: re.sub("\s+"," ", elem),
            filtered_tokens)

    return filtered_tokens

def tokenize_and_stem(text):
    """ Applies the Snowball stemmer to unigrams (i.e. words) extracted
    from a string that contains a service code name.

    Args:
        text: String that stores a service code name

    Returns:
        filtered_tokens: List of words contained in a service code name"""
    stemmer = SnowballStemmer('english')

    tokens = [word.lower() for word in nltk.word_tokenize(text)]

    filtered_tokens =\
        filter(lambda elem: re.match('^[a-z]+$', elem) != None,
               tokens)

    filtered_tokens =\
        map(lambda elem: re.sub("\s+"," ", elem),
            filtered_tokens)

    filtered_tokens = [stemmer.stem(token) for token in filtered_tokens]

    return filtered_tokens

def compute_tfidf_features(code_name_map,
                           tokenizer,
                           params):
    """ Constructs a Term Frequency Inverse Document Frequency (TF-IDF)
    matrix for the Cincinnati 311 service code names.

    Args:
        code_name_map: Dictionary that stores the mapping of service
                       codes to service names

        tokenizer: Function that transforms a string into a list of
                   words

        params: Dictionary that stores parameters that configure the
                TfidfVectorizer class constructor

                - mindocumentcount:...


I am working on the file "ClusterServiceCodes.ipynb".
The first task is:
Create a separate service name(s) cluster for the 'mtlfrn' service code
Can you write Python code for it?

add_new_cluster(1,
                'mtlfrn',
                clusterid_total_count,
                clusterid_code_map,
                clusterid_name_map)


I am working on the file "ClusterServiceCodes.ipynb".
The first task is:
Create a separate service name(s) cluster for the 'ydwstaj' service code
Can you write Python code for it?

add_new_cluster(1,
                'ydwstaj',
                clusterid_total_count,
                clusterid_code_map,
                clusterid_name_map)


I am working on the file "ClusterServiceCodes.ipynb".
The first task is:
Evaluate the service name(s) cluster statistics
Can you write Python code for it?

clusterid_total_count =\
    compute_clusterid_totalcounts(clusterid_code_map,
                                  code_histogram)

print_cluster_stats(clusterid_name_map,
                    clusterid_total_count)


I am working on the file "ClusterServiceCodes.ipynb".
The first task is:
Initialize service code data structures

Service code / service name map
Service code histogram
Can you write Python code for it?

h_file = open("./serviceCodesCount.tsv","r")

code_name_map = {}
code_histogram = {}

patternobj = re.compile('^([0-9a-z]+)\s\|\s([0-9a-z\s]+)$')

for fields in csv.reader(h_file, delimiter="\t"):
    matchobj = patternobj.match(fields[0])

    cur_code = matchobj.group(1)
    code_name_map[cur_code] = matchobj.group(2)
    code_histogram[cur_code] = float(fields[1])

h_file.close()


I am working on the file "ClusterServiceCodes.ipynb".
The first task is:
Plot Cincinnati 311 Service Code Statistics

References
Descending Array Sort
Change Plot Font Size
Can you write Python code for it?

total_count_fraction = code_histogram.values()
total_count_fraction.sort()

total_count_fraction = total_count_fraction[::-1]
total_count_fraction /= np.sum(total_count_fraction)
total_count_fraction = np.cumsum(total_count_fraction)

sns.set(font_scale=2)
f,h_ax = plt.subplots(1,2,figsize=(12,6))
h_ax[0].bar(range(0,len(code_histogram.values())),
            code_histogram.values())
h_ax[0].set_xlim((0,len(total_count_fraction)))
h_ax[0].set_xlabel('Service Code #')
h_ax[0].set_ylabel('Service Code Count')
h_ax[0].set_title('Cincinnati 311\nService Code Histogram')

h_ax[1].plot(total_count_fraction, linewidth=4)
h_ax[1].set_xlim((0,len(total_count_fraction)))
h_ax[1].set_xlabel('Sorted Service Code #')
h_ax[1].set_ylabel('Total Count Fraction')
f.tight_layout()
plt.savefig("./cincinatti311Stats.png")


I am working on the file "ClusterServiceCodes.ipynb".
The first task is:
Plot Cincinnati 311 Service Name Categories
Can you write Python code for it?

import pandas as pd

category_totalcountdf =\
    pd.DataFrame({'totalcount': clusterid_total_count.values()},
                 index=clusterid_category_map.values())

sns.set(font_scale=1.5)
category_totalcountdf.plot(kind='barh')


I am working on the file "ClusterServiceCodes.ipynb".
The first task is:
Plot the service code histogram for the maximum size cluster
Can you write Python code for it?

eval_maxcount_clusterid(clusterid_code_map,
                        clusterid_total_count,
                        code_histogram)


I am working on the file "ClusterServiceCodes.ipynb".
The first task is:
Write service code / category map to disk

Storing Python Dictionaries
Can you write Python code for it?

servicecode_category_map = {}

for clusterid in clusterid_name_map.keys():
    cur_category = clusterid_category_map[clusterid]

    for servicecode in clusterid_code_map[clusterid]:

        servicecode_category_map[servicecode] = cur_category

with open('serviceCodeCategory.txt', 'w') as fp:
    num_names = len(servicecode_category_map)

    keys = servicecode_category_map.keys()
    values = servicecode_category_map.values()

    for idx in range(0, num_names):
        if idx == 0:
            fp.write("%s{\"%s\": \"%s\",\n" % (" " * 12,
                                               keys[idx],
                                               values[idx]))
        #----------------------------------------
        elif idx > 0 and idx < num_names-1:
            fp.write("%s\"%s\": \"%s\",\n" % (" " * 13,
                                              keys[idx],
                                              values[idx]))
        #----------------------------------------
        else:
            fp.write("%s\"%s\": \"%s\"}" % (" " * 13,
                                            keys[idx],
                                            values[idx]))


I am working on the file "Clustering KO articles.ipynb".
The first task is:
Section 2:
Training Models with different number of clusters
Initializing the parameters for each model then start training using the Expectation-Maximization algorithm.
Can you write Python code for it?

# Model 1 with 10 clusters
(means , cluster_assignment_10model)= initialize_means(10,tf_idf_matrix)
covs= initialize_covs(10,tf_idf_matrix, cluster_assignment_10model)
weights= initialize_weights(10,tf_idf_matrix, cluster_assignment_10model)
model_em_10k= EM_for_high_dimension(tf_idf_matrix, means, covs, weights, cov_smoothing=1e-10)

# Model 2 with 20 clusters.
(means , cluster_assignment_20model)= initialize_means(20,tf_idf_matrix)
covs= initialize_covs(20,tf_idf_matrix, cluster_assignment_20model)
weights= initialize_weights(20,tf_idf_matrix, cluster_assignment_20model)
model_em_20k= EM_for_high_dimension(tf_idf_matrix, means, covs, weights, cov_smoothing=1e-10)


I am working on the file "Clustering KO articles.ipynb".
The first task is:
Section 3:
Evaluation report for each cluster (Interpreting clusters)
Evaluation report is divided into two partitions the first one is the word representation for each cluster the really interpret the cluster, the second one is for the variety of article types in one cluster counting each category for each cluster.
Can you write Python code for it?

def visualize_EM_clusters(tf_idf, means, covs, map_index_to_word):
    print('')
    print('==========================================================')

    num_clusters = len(means)
    for c in xrange(num_clusters):
        print('Cluster {0:d}: Largest mean parameters in cluster '.format(c))
        print('\n{0: <12}{1: <12}{2: <12}'.format('Word', 'Mean', 'Variance'))

        # The k'th element of sorted_word_ids should be the index of the word
        # that has the k'th-largest value in the cluster mean. Hint: Use np.argsort().
        sorted_word_ids = np.argsort(means[c])[::-1]

        for i in sorted_word_ids[:10]:
            print '{0: <12}{1:<10.2e}{2:10.2e}'.format(map_index_to_word[i],
                                                       means[c][i],
                                                       covs[c][i])
        print '\n=========================================================='

def clusters_report(clusters_idx):
    cluster_id=0
    for cluster_indicies in clusters_idx:
        countP=0
        countB=0
        countE=0
        for i in cluster_indicies:
            if dataset['category'][i]=='product':
                countP+=1
            elif dataset['category'][i]=='engineering':
                countE+=1
            elif dataset['category'][i]=='business':
                countB+=1
        print "Cluster ",cluster_id ,"\n==========================\n"
        cluster_id+=1
        print "product count : ",countP ,"\nengineering count : ",countE,"\nbusiness count : ",countB , "\n"

visualize_EM_clusters(tf_idf_matrix, model_em_10k['means'], model_em_10k['covs'], tfidfvec.get_feature_names())

visualize_EM_clusters(tf_idf_matrix, model_em_20k['means'], model_em_20k['covs'], tfidfvec.get_feature_names())

# No. of articles in each cluster for first model with 10 clusters
resps_10k= sf.SFrame(model_em_10k['resp'])
resps_10k= resps_10k.unpack('X1', '')
cluster_id=0
cluster_hash_10model = {}
for col in resps_10k.column_names():
    cluster_10k=...


I am working on the file "Clustering KO articles.ipynb".
The first task is:
Section 4
Recommendation and predictions for Articles
Recommendation method:
A method for recommending articles by retrieving the cluster that the article belong to, then fetch all the articles in that cluster articles passed to nearest neighbour model to find the best 10 articles recommended for this article.
Predicting method:
Sending set of articles to predict the cluster it belong based on the trained data

Using the test dataset to predict cluster for each one using two different models.
Can you write Python code for it?

def articles_inds(article_id , cluster_hash_model):
    for cluster_id in cluster_hash_model:
        np_array = np.array(cluster_hash_model[cluster_id])
        if article_id in np_array:
            return cluster_id, np_array

def recommender(article_id ,cluster_hash_model, no_articles, data_articles):
    start_time = time.time()
    cid , inds = articles_inds(article_id ,cluster_hash_model)
    cluster_articles= data_articles.filter_by(inds[0] , 'X1')
    cluster_articles = cluster_articles.add_row_number()

    recom_vec= TfidfVectorizer(stop_words='english')
    tfidf_recommend= recom_vec.fit_transform(cluster_articles['text'])
    tfidf_recommend = normalize(tfidf_recommend)

    row_id = cluster_articles[cluster_articles['X1']==article_id]['id'][0]
    NN_model = NearestNeighbors(n_neighbors=no_articles).fit(tfidf_recommend)
    distances, indices = NN_model.kneighbors(tfidf_recommend[row_id])

    recommended_ids=[]
    for i in indices[0]:
        recommended_ids.append(cluster_articles[cluster_articles['id']==i]['X1'][0])

    del cluster_articles
    del tfidf_recommend
    del recom_vec
    #print("--- %s seconds ---" % (time.time() - start_time))
    #print len(inds[0])
    return recommended_ids

def predict_cluster(articles,em_model):
    article_tfidf= tfidfvec.transform(articles['text'])
    mu= deepcopy(em_model['means'])
    sigma= deepcopy(em_model['covs'])
    assignments=[]
    for j in range(article_tfidf.shape[0]):
        resps=[]
        for i in range(len(em_model['weights'])):
            predict= np.log(em_model['weights'][i]) + logpdf_diagonal_gaussian(article_tfidf[j], mu[i],sigma[i])
            resps.append(predict)
        assignments.append(resps.index(np.max(resps)))
    return assignments

# Recommend articles for all dataset then append it into the SFrame database then export it.
recommended_inds = []
start_time = time.time()
for i in range(len(dataset)):
    recommended_inds.append(recommender(i,cluster_hash_20model,11,dataset))

print("--- %s seconds...


I am working on the file "Clustering Test.ipynb".
The first task is:
Benchmarking
Can you write Python code for it?

from sklearn.cluster import KMeans

def bench_k_means(estimator, name, data):
    t0 = time()
    estimator.fit(data)
    print('%-9s\t%.2fs\t%i\t%.3f\t%.3f\t%.3f\t%.3f\t%.3f\t%.3f'
          % (name, (time() - t0), estimator.inertia_,
             metrics.homogeneity_score(labels, estimator.labels_),
             metrics.completeness_score(labels, estimator.labels_),
             metrics.v_measure_score(labels, estimator.labels_),
             metrics.adjusted_rand_score(labels, estimator.labels_),
             metrics.adjusted_mutual_info_score(labels,  estimator.labels_),
             metrics.silhouette_score(data, estimator.labels_,
                                      metric='euclidean',
                                      sample_size=samples)))

labels = (np.array(matrix[:,[0]])).flatten()

print("Benchmarks without PCA:")

bench_k_means(KMeans(init='k-means++', n_clusters=5, n_init=10), name="k-means++", data=matrix)
bench_k_means(KMeans(init='random', n_clusters=5, n_init=10), name="random", data=matrix)

print("\nBenchmarks with PCA:")

pca = PCA(n_components=5).fit(matrix)
bench_k_means(KMeans(init=pca.components_, n_clusters=5, n_init=1), name="PCA-based", data=matrix)


I am working on the file "Clustering Test.ipynb".
The first task is:
Visualise PCA-reduced data with affinity propagation
Can you write Python code for it?

from sklearn.cluster import AffinityPropagation
from itertools import cycle

af = AffinityPropagation(preference=-50).fit(reduced_data)
cluster_centers_indices = af.cluster_centers_indices_
labels = af.labels_

labels_true = reduced_data[:, 1]
n_clusters_ = len(cluster_centers_indices)

plt.close('all')
plt.figure(1, figsize=(10, 6))
plt.clf()

colors = cycle('bgrcmykbgrcmykbgrcmykbgrcmyk')
for k, col in zip(range(n_clusters_), colors):
    class_members = labels == k
    cluster_center = X[cluster_centers_indices[k]]
    plt.plot(X[class_members, 0], X[class_members, 1], col + '.')
    plt.plot(cluster_center[0], cluster_center[1], 'o', markerfacecolor=col,
             markeredgecolor='k', markersize=14)
    for x in X[class_members]:
        plt.plot([cluster_center[0], x[0]], [cluster_center[1], x[1]], col)

plt.title('Estimated number of clusters: %d' % n_clusters_)
plt.show()

print('Estimated number of clusters: %d' % n_clusters_)
print("Homogeneity: %0.3f" % metrics.homogeneity_score(labels_true, labels))
print("Completeness: %0.3f" % metrics.completeness_score(labels_true, labels))
print("V-measure: %0.3f" % metrics.v_measure_score(labels_true, labels))
print("Adjusted Rand Index: %0.3f"
      % metrics.adjusted_rand_score(labels_true, labels))
print("Adjusted Mutual Information: %0.3f"
      % metrics.adjusted_mutual_info_score(labels_true, labels))
print("Silhouette Coefficient: %0.3f"
      % metrics.silhouette_score(X, labels, metric='sqeuclidean'))

from sklearn.cluster import DBSCAN

db = DBSCAN(eps=0.3, min_samples=10).fit(reduced_data)
core_samples_mask = np.zeros_like(db.labels_, dtype=bool)
core_samples_mask[db.core_sample_indices_] = True
labels = db.labels_

# Number of clusters in labels, ignoring noise if present.
n_clusters_ = len(set(labels)) - (1 if -1 in labels else 0)

plt.figure(1, figsize=(10, 6))

# Black removed and is used for noise instead.
unique_labels = set(labels)
colors = [plt.cm.Spectral(each)
          for each in np.linspace(0, 1, len(unique_labels))]

for k, col in...


I am working on the file "Clustering Test.ipynb".
The first task is:
Visualise PCA-reduced data with k-means clustering
Can you write Python code for it?

i = 3
target_clusters = i

reduced_data = PCA(n_components=2).fit_transform(matrix_scaled)
kmeans = KMeans(init='k-means++', n_clusters=target_clusters, n_init=10)
kmeans.fit(reduced_data)

# Step size of the mesh. Decrease to increase the quality of the VQ.
h = .01     # point in the mesh [x_min, x_max]x[y_min, y_max].

# Plot the decision boundary. For that, we will assign a color to each
x_min, x_max = reduced_data[:, 0].min() - 1, reduced_data[:, 0].max() + 1
y_min, y_max = reduced_data[:, 1].min() - 1, reduced_data[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

# Obtain labels for each point in mesh. Use last trained model.
Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()])
Z = Z.reshape(xx.shape)
plt.figure(i, figsize=(10, 6))
plt.clf()
plt.imshow(Z, interpolation='nearest',
           extent=(xx.min(), xx.max(), yy.min(), yy.max()),
           cmap=plt.cm.Paired,
           aspect='auto', origin='lower')

plt.plot(reduced_data[:, 0], reduced_data[:, 1], 'k.', markersize=2)
# Plot the centroids as a white X
centroids = kmeans.cluster_centers_
plt.scatter(centroids[:, 0], centroids[:, 1],
            marker='x', s=169, linewidths=3,
            color='w', zorder=10)

plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.xticks(())
plt.yticks(())

plt.title('Number of clusters: %d' % target_clusters)
plt.show()


I am working on the file "Clustering running notepad.ipynb".
The first task is:
Going from a vector back to the metadata reference:
By keeping an 'id_list', we can look up the identifier for any vector in the list from the database we've made for this clustering attemp. This lets us look up what the reference for that is, and where we can find it:
Can you write Python code for it?

from clustering import ClusterDB
db = ClusterDB(DBFILE)
print(dict(db.vecidtoitem(id_list[-1])))
print(data.toarray()[-1])

from burney_data import BurneyDB
bdb = BurneyDB("burney.db")

bdb.get_title_row(titleAbbreviation="B0574REMEMBRA")


I am working on the file "Clustering running notepad.ipynb".
The first task is:
Initial data woes
There was a considerable discrepancy between the x1 average indent and the column "box" left edge. Looking at the data, the presence of a few outliers can really affect this value. Omitting the 2 smallest and largest x values might be enough to avoid this biasing the sample too badly.
Also, the initial 'drift correction' (adjustments made to correct warped or curved columns) seemed to add more issues than it solved, so the dataset was remade without it.
Can you write Python code for it?

from scipy import cluster
from matplotlib import pyplot as plt
import numpy as np

# Where is the K-means 'elbow'?
# Try between 1 and 10
# use only the x1 and x2 variences
vset = [cluster.vq.kmeans(data.toarray()[:, [3,6]], i) for i in range(1,10)]
plt.plot([v for (c,v) in vset])
plt.show()


I am working on the file "Clustering running notepad.ipynb".
The first task is:
Interesting!
The lack of really well defined clusters bolstered the "elbow" test above. K-means is likely not put to good use here, with just these two variables.
The left edge of the scatterplot is a region that contains those blocks of text with lines aligned to the left edge of the paper's column, but have some considerable variation to the length of the line.
For example, I'd expect text looking like the following:
Qui quis at ex voluptatibus cupiditate quod quia.
Quas fuga quasi sit mollitia quos atque. Saepe atque officia sed dolorem.
Numquam quas aperiam eaque nam sunt itaque est. Sed expedita
maxime fugiat mollitia error necessitatibus quam soluta. Amet laborum eius
sequi quae sit sit.

This is promising (as long as the data is realistic and there isn't a bug in generating that...)
Now, I wonder if including the "margin" (x1ave-ledge: average x1 coordinate minus the leftmost edge) might help find or distinguish these further?
Can you write Python code for it?

mpld3.disable_notebook()  # switch off the interactive graph functionality which doesn't work well with the 3D library

from mpl_toolkits.mplot3d import Axes3D

X = npdata[:, [3,5,6]]

fignum = 1
for name, est in estimators.items():
    fig = plt.figure(fignum, figsize=(8, 8))
    plt.clf()
    ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=5, azim=30)
    plt.cla()
    est.fit(X)
    labels = est.labels_

    ax.scatter(X[:,0], X[:,2], X[:,1], c=labels.astype(np.float))

    ax.set_xlabel('x1 varience')
    ax.set_ylabel('x2 varience')
    ax.set_zlabel('Average indent')

    fignum = fignum + 1
plt.show()


I am working on the file "Clustering running notepad.ipynb".
The first task is:
What do these 'vectors' look like? What do the columns refer to?
Can you write Python code for it?

data, transform, id_list = ds
print(data.toarray())

print(transform.get_feature_names())


I am working on the file "Clustering running notepad.ipynb".
The first task is:
x1 vs x2 varience?
What is the rough shape of this data? The varience of x1 and x2 are equivalent to the left and right alignment of the text varies in a given block of text.
Can you write Python code for it?


I am working on the file "Clustering using Gaussian Mixture Models and PCA.ipynb".
The first task is:
Load the data file and function for drawing ellipses for each of the 3 GMM components.
Can you write Python code for it?

colors = ['navy', 'turquoise', 'darkorange']

# draw ellipses for each GMM type
def make_ellipses(gmm, ax):
    for n, color in enumerate(colors):
        if gmm.covariance_type == 'full':
            covariances = gmm.covariances_[n][:2, :2]
        elif gmm.covariance_type == 'tied':
            covariances = gmm.covariances_[:2, :2]
        elif gmm.covariance_type == 'diag':
            covariances = np.diag(gmm.covariances_[n][:2])
        elif gmm.covariance_type == 'spherical':
            covariances = np.eye(gmm.means_.shape[1]) * gmm.covariances_[n]
        v, w = np.linalg.eigh(covariances)
        u = w[0] / np.linalg.norm(w[0])
        angle = np.arctan2(u[1], u[0])
        angle = 180 * angle / np.pi  # convert to degrees
        v = 2. * np.sqrt(2.) * np.sqrt(v)
        ell = mpl.patches.Ellipse(gmm.means_[n, :2], v[0], v[1],
                                  180 + angle, color=color)
        ell.set_clip_box(ax.bbox)
        ell.set_alpha(0.5)
        ax.add_artist(ell)

(X, y) = datasets.load_svmlight_file("data/3genres.arff.libsvm")
X = X.toarray()
X = MinMaxScaler().fit_transform(X)
target_names = ['classical', 'jazz', 'metal']


I am working on the file "Clustering using Gaussian Mixture Models and PCA.ipynb".
The first task is:
Now let's visualize with some plots different types of GMM based on the constrains placed on the covariance matrix.
Can you write Python code for it?

# Break up the dataset into non-overlapping training (75%) and testing
# (25%) sets.
skf = StratifiedKFold(n_splits=4)
# Only take the first fold.
train_index, test_index = next(iter(skf.split(X, y)))

X_train = X[train_index]
y_train = y[train_index]
X_test = X[test_index]
y_test = y[test_index]

n_classes = len(np.unique(y_train))

# Try GMMs using different types of covariances.
estimators = dict((cov_type, GaussianMixture(n_components=n_classes,
                   covariance_type=cov_type, max_iter=20, random_state=0))
                  for cov_type in ['spherical', 'diag', 'tied', 'full'])

n_estimators = len(estimators)

plt.figure(figsize=(3 * n_estimators // 2, 6))
plt.subplots_adjust(bottom=.01, top=0.95, hspace=.15, wspace=.05,
                    left=.01, right=.99)

for index, (name, estimator) in enumerate(estimators.items()):
    # Since we have class labels for the training data, we can
    # initialize the GMM parameters in a supervised manner.
    estimator.means_init = np.array([X_train[y_train == i].mean(axis=0)
                                    for i in range(n_classes)])

    # Train the other parameters using the EM algorithm.
    estimator.fit(X_train)

    h = plt.subplot(2, n_estimators // 2, index + 1)
    make_ellipses(estimator, h)

    for n, color in enumerate(colors):
        data = X[y == n]
        plt.scatter(data[:, 0], data[:, 1], s=0.8, color=color,
                    label=target_names[n])
    # Plot the test data with crosses
    for n, color in enumerate(colors):
        data = X_test[y_test == n]
        plt.scatter(data[:, 0], data[:, 1], marker='x', color=color)

    y_train_pred = estimator.predict(X_train)
    train_accuracy = np.mean(y_train_pred.ravel() == y_train.ravel()) * 100
    plt.text(0.05, 0.9, 'Train accuracy: %.1f' % train_accuracy,
             transform=h.transAxes)

    y_test_pred = estimator.predict(X_test)
    test_accuracy = np.mean(y_test_pred.ravel() == y_test.ravel()) * 100
    plt.text(0.05, 0.8, 'Test accuracy: %.1f' % test_accuracy,
  ...


I am working on the file "Clustering-of-Bins_Habits-data-user-retention.ipynb".
The first task is:
Post Week 1 Cluster
Can you write Python code for it?

# Clustering using Mean shift

from sklearn.cluster import MeanShift, estimate_bandwidth

#x = [1,1,5,6,1,5,10,22,23,23,50,51,51,52,100,112,130,500,512,600,12000,12230]
x = pd.Series(user_data_raw_rest['age_on_platform'])

X = np.array(zip(x,np.zeros(len(x))), dtype=np.int)

# The following bandwidth can be automatically detected using
bandwidth = estimate_bandwidth(X, quantile=0.1)

ms = MeanShift(bandwidth=bandwidth, bin_seeding=True)
ms.fit(X)
labels = ms.labels_
cluster_centers = ms.cluster_centers_

labels_unique = np.unique(labels)
n_clusters_ = len(labels_unique)

print("number of estimated clusters : %d" % n_clusters_)
for k in range(n_clusters_):
    my_members = labels == k
    print "cluster {0} : lenght = {1}".format(k, len(X[my_members, 0]))
    cluster_sorted = sorted(X[my_members, 0])
    print "cluster {0} : Min = {1} days & Max {2} days".format(k, cluster_sorted[0]*1.15741e-5, cluster_sorted[-1]*1.15741e-5)

# Plot result
import matplotlib.pyplot as plt
from itertools import cycle

%matplotlib inline

plt.figure(1)
plt.clf()

colors = cycle('bgrcmykbgrcmykbgrcmykbgrcmyk')
for k, col in zip(range(n_clusters_), colors):
    my_members = labels == k
    cluster_center = cluster_centers[k]
    plt.plot(X[my_members, 0], X[my_members, 1], col + '.')
    plt.plot(cluster_center[0], cluster_center[1], 'o', markerfacecolor=col,
             markeredgecolor='k', markersize=14)
plt.title('Estimated number of clusters: %d' % n_clusters_)
plt.show()


I am working on the file "Clustering-of-Bins_Habits-data-user-retention.ipynb".
The first task is:
Week 1 clusters
Can you write Python code for it?

# Clustering using Mean shift

from sklearn.cluster import MeanShift, estimate_bandwidth

#x = [1,1,5,6,1,5,10,22,23,23,50,51,51,52,100,112,130,500,512,600,12000,12230]
x = pd.Series(user_data_raw_1st_week['age_on_platform'])

X = np.array(zip(x,np.zeros(len(x))), dtype=np.int)

# The following bandwidth can be automatically detected using
bandwidth = estimate_bandwidth(X, quantile=0.1)

ms = MeanShift(bandwidth=bandwidth, bin_seeding=True)
ms.fit(X)
labels = ms.labels_
cluster_centers = ms.cluster_centers_

labels_unique = np.unique(labels)
n_clusters_ = len(labels_unique)

print("number of estimated clusters : %d" % n_clusters_)
for k in range(n_clusters_):
    my_members = labels == k
    print "cluster {0} : lenght = {1}".format(k, len(X[my_members, 0]))
    cluster_sorted = sorted(X[my_members, 0])
    print "cluster {0} : Min = {1} days & Max {2} days".format(k, cluster_sorted[0]*1.15741e-5, cluster_sorted[-1]*1.15741e-5)

# Plot result
import matplotlib.pyplot as plt
from itertools import cycle

%matplotlib inline

plt.figure(1)
plt.clf()

colors = cycle('bgrcmykbgrcmykbgrcmykbgrcmyk')
for k, col in zip(range(n_clusters_), colors):
    my_members = labels == k
    cluster_center = cluster_centers[k]
    plt.plot(X[my_members, 0], X[my_members, 1], col + '.')
    plt.plot(cluster_center[0], cluster_center[1], 'o', markerfacecolor=col,
             markeredgecolor='k', markersize=14)
plt.title('Estimated number of clusters: %d' % n_clusters_)
plt.show()


I am working on the file "Clustering.ipynb".
The first task is:
Choosing k
Can you write Python code for it?

def squared_clustering_errors(inputs, k):
    """finds total square error for k"""
    clusterer = KMeans(k)
    clusterer.train(inputs)
    means = clusterer.means
    assignments = map(clusterer.classify, inputs)

    return sum(squared_distance(input, means[cluster])
              for input, cluster in zip(inputs, assignments))

ks = range(1, len(inputs) + 1)
errors = [squared_clustering_errors(inputs, k) for k in ks]

%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_context('talk')

plt.plot(ks, errors, '.')


I am working on the file "Clustering.ipynb".
The first task is:
Clustering
In this notebook we explore clustering otherwise known as unsupervised learning. In order to examine the effects of clustering it is useful to have a data-set that is labeled with class labels as then we can visualize observe how well the identified clusters correspond to the original clusters. Notice that the clustering algorithms we will explore do NOT take into account the class labels. They are simply used to color the points in the visualizations. Let's start by loading and visualizing a data set consisting of audio features representing music tracks. There are 3 genres (classical, jazz, heavy metal) with 100 instances each. The first data-set we will examine consists of 2 features per track to make visualization easy. Let's look at a scatter plot of the points corresponding to each music track and use color to indicate what is the original class. The scikit-learn documentation has a lot of examples of clustering with interesting plots and visualizations.
Can you write Python code for it?

(X, y) = datasets.load_svmlight_file("data/3genres.arff.libsvm")
X = X.toarray()
X = MinMaxScaler().fit_transform(X)
y = y.astype(int)
target_names = ['classical', 'jazz', 'metal']

print(X.shape)
print(y.shape)
colors = ['navy', 'turquoise', 'darkorange']

for n, color in enumerate(colors):
    data = X[y == n]
    # Plot the training data
    plt.scatter(data[:, 0], data[:, 1], s=5, color=color,
                    label=target_names[n])
    plt.title('Scatter plot of audio features and genres')
    plt.legend(scatterpoints=1, loc='upper right', prop=dict(size=10))

plt.show()


I am working on the file "Clustering.ipynb".
The first task is:
Clustering usig GMMs
The idea behind clustering using Gaussian Mixture Models is that each cluster will correspond to a different
Gaussian shaped component. When visualized in 2D these components can be represented as ellipses. When the covariance matrix is diagonal then these ellipses will be aligned with the axis. When they are spherical that means that the standard deviation of the features in each dimensions is considered equal. In the tied configuration the covariance matrices of each mixture component are tied to be equal. The most flexible case is when full covariance matrices are used.
Can you write Python code for it?

colors = ['navy', 'turquoise', 'darkorange']

def make_ellipses(gmm, ax):
    for n, color in enumerate(colors):
        if gmm.covariance_type == 'full':
            covariances = gmm.covariances_[n][:2, :2]
        elif gmm.covariance_type == 'tied':
            covariances = gmm.covariances_[:2, :2]
        elif gmm.covariance_type == 'diag':
            covariances = np.diag(gmm.covariances_[n][:2])
        elif gmm.covariance_type == 'spherical':
            covariances = np.eye(gmm.means_.shape[1]) * gmm.covariances_[n]
        v, w = np.linalg.eigh(covariances)
        u = w[0] / np.linalg.norm(w[0])
        angle = np.arctan2(u[1], u[0])
        angle = 180 * angle / np.pi  # convert to degrees
        v = 2. * np.sqrt(2.) * np.sqrt(v)
        ell = mpl.patches.Ellipse(gmm.means_[n, :2], v[0], v[1],
                                  180 + angle, color=color)
        ell.set_clip_box(ax.bbox)
        ell.set_alpha(0.5)
        ax.add_artist(ell)

(X, y) = datasets.load_svmlight_file("data/3genres.arff.libsvm")
X = X.toarray()
X = MinMaxScaler().fit_transform(X)
target_names = ['classical', 'jazz', 'metal']

# Break up the dataset into non-overlapping training (75%) and testing
# (25%) sets.
skf = StratifiedKFold(n_splits=4)
# Only take the first fold.
train_index, test_index = next(iter(skf.split(X, y)))

X_train = X[train_index]
y_train = y[train_index]
X_test = X[test_index]
y_test = y[test_index]

n_classes = len(np.unique(y_train))

# Try GMMs using different types of covariances.
estimators = dict((cov_type, GaussianMixture(n_components=n_classes,
                   covariance_type=cov_type, max_iter=100, random_state=0))
                  for cov_type in ['spherical', 'diag', 'tied', 'full'])

n_estimators = len(estimators)

fig = plt.figure(figsize=(6 * n_estimators // 2, 12))
plt.subplots_adjust(bottom=.01, top=0.95, hspace=.15, wspace=.05, left=.01, right=.99)

for index, (name, estimator) in enumerate(estimators.items()):
    # Since we have class labels for the training data, we...


I am working on the file "Clustering.ipynb".
The first task is:
Clustering with k-means visualized in 3 dimensions
If we increase the number of feature we use to represent each music track to 3 then we can do scatter plots and perform clustering in the resulting 3D space. In most actual applications clustering is performed in high-dimensional feature spaces so visualization is not easy. At the end of this notebook we explore Principal Component Analysis a methodology for dimensionality reduction.
Can you write Python code for it?

# Modified for documentation by Jaques Grobler
# License: BSD 3 clause

from sklearn.cluster import KMeans
from sklearn import datasets

np.random.seed(5)

(X, y) = datasets.load_svmlight_file("data/3genres_4features.arff.libsvm")
X = X.toarray()
X = MinMaxScaler().fit_transform(X)
target_names = ['classical', 'jazz', 'metal']
y = y.astype(int)

estimators = [('k_means_8', KMeans(n_clusters=8)),
              ('k_means_3', KMeans(n_clusters=3)),
              ]

fignum = 1
titles = ['8 clusters', '3 clusters']
for name, est in estimators:
    fig = plt.figure(fignum, figsize=(7, 6))
    ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=28, azim=34)
    est.fit(X)
    labels = est.labels_

    ax.scatter(X[:, 3], X[:, 0], X[:, 2],
               c=labels.astype(np.float), edgecolor='k')

    ax.w_xaxis.set_ticklabels([])
    ax.w_yaxis.set_ticklabels([])
    ax.w_zaxis.set_ticklabels([])
    ax.set_xlabel('Feature 1')
    ax.set_ylabel('Feature 2')
    ax.set_zlabel('Feature 3')
    ax.set_title(titles[fignum - 1])
    ax.dist = 12
    fignum = fignum + 1

# Plot the ground truth
fig = plt.figure(fignum, figsize=(7, 6))
ax = Axes3D(fig, rect=[0, 0, .95, 1], elev=28, azim=34)

for name, label in [('classical', 0),
                    ('jazz', 1),
                    ('metal', 2)]:
    ax.text3D(X[y == label, 3].mean(),
              X[y == label, 0].mean(),
              X[y == label, 2].mean()-1.5, name,
              horizontalalignment='center',
              bbox=dict(alpha=.2, edgecolor='w', facecolor='w'))
# Reorder the labels to have colors matching the cluster results
y = np.choose(y, [1, 2, 0]).astype(np.float)
ax.scatter(X[:, 3], X[:, 0], X[:, 2], c=y, edgecolor='k')

ax.w_xaxis.set_ticklabels([])
ax.w_yaxis.set_ticklabels([])
ax.w_zaxis.set_ticklabels([])
ax.set_xlabel('Feature 1')
ax.set_ylabel('Feature 2')
ax.set_zlabel('Feature 3')
ax.set_title('Ground Truth')
ax.dist = 12
plt.show()


I am working on the file "Clustering.ipynb".
The first task is:
DBSCAN
The DBSCAN algorithm views clusters as areas of high density separated by areas of low density. Due to this rather generic view, clusters found by DBSCAN can be any shape, as opposed to k-means which assumes that clusters are convex shaped. The central component to the DBSCAN is the concept of core samples, which are samples that are in areas of high density. A cluster is therefore a set of core samples, each close to each other (measured by some distance measure) and a set of non-core samples that are close to a core sample (but are not themselves core samples). There are two parameters to the algorithm, min_samples and eps, which define formally what we mean when we say dense. Higher min_samples or lower eps indicate higher density necessary to form a cluster.
More formally, we define a core sample as being a sample in the dataset such that there exist min_samples other samples within a distance of eps, which are defined as neighbors of the core sample. This tells us that the core sample is in a dense area of the vector space. A cluster is a set of core samples that can be built by recursively taking a core sample, finding all of its neighbors that are core samples, finding all of their neighbors that are core samples, and so on. A cluster also has a set of non-core samples, which are samples that are neighbors of a core sample in the cluster but are not themselves core samples. Intuitively, these samples are on the fringes of a cluster.
Can you write Python code for it?

db = DBSCAN().fit(reduced_data)

db

db.labels_

clusdf.shape

reduced_data.shape

reduced_data[:10,:2]

for i in range(0, reduced_data.shape[0]):
    if db.labels_[i] == 0:
        c1 = plt.scatter(reduced_data[i,0],reduced_data[i,1],c='r',marker='+')
    elif db.labels_[i] == 1:
        c2 = plt.scatter(reduced_data[i,0],reduced_data[i,1],c='g',marker='o')
    elif db.labels_[i] == -1:c3 = plt.scatter(reduced_data[i,0],reduced_data[i,1],c='b',marker='*')

plt.legend([c1, c2, c3], ['Cluster 1', 'Cluster 2','Noise'])
plt.title('DBSCAN finds 2 clusters and noise')
plt.show()


I am working on the file "Clustering.ipynb".
The first task is:
Gaussian mixture models
a mixture model is a probabilistic model for representing the presence of subpopulations within an overall population, without requiring that an observed data set should identify the sub-population to which an individual observation belongs. Formally a mixture model corresponds to the mixture distribution that represents the probability distribution of observations in the overall population. However, while problems associated with "mixture distributions" relate to deriving the properties of the overall population from those of the sub-populations, "mixture models" are used to make statistical inferences about the properties of the sub-populations given only observations on the pooled population, without sub-population identity information.
sklearn.mixture is a package which enables one to learn Gaussian Mixture Models (diagonal, spherical, tied and full covariance matrices supported), sample them, and estimate them from data. Facilities to help determine the appropriate number of components are also provided.
A Gaussian mixture model is a probabilistic model that assumes all the data points are generated from a mixture of a finite number of Gaussian distributions with unknown parameters. One can think of mixture models as generalizing k-means clustering to incorporate information about the covariance structure of the data as well as the centers of the latent Gaussians.
Scikit-learn implements different classes to estimate Gaussian mixture models, that correspond to different estimation strategies.
cite- https://jakevdp.github.io/PythonDataScienceHandbook/05.12-gaussian-mixtures.html
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()
import numpy as np

clusdf.head()

reduced_data

# Plot the data with K Means Labels
from sklearn.cluster import KMeans
kmeans = KMeans(4, random_state=0)
labels = kmeans.fit(reduced_data).predict(reduced_data)
plt.scatter(reduced_data[:, 0], reduced_data[:, 1], c=labels, s=40, cmap='viridis');

X=reduced_data

from sklearn.cluster import KMeans
from scipy.spatial.distance import cdist

def plot_kmeans(kmeans, X, n_clusters=4, rseed=0, ax=None):
    labels = kmeans.fit_predict(X)

    # plot the input data
    ax = ax or plt.gca()
    ax.axis('equal')
    ax.scatter(X[:, 0], X[:, 1], c=labels, s=40, cmap='viridis', zorder=2)

    # plot the representation of the KMeans model
    centers = kmeans.cluster_centers_
    radii = [cdist(X[labels == i], [center]).max()
             for i, center in enumerate(centers)]
    for c, r in zip(centers, radii):
        ax.add_patch(plt.Circle(c, r, fc='#CCCCCC', lw=3, alpha=0.5, zorder=1))

kmeans = KMeans(n_clusters=4, random_state=0)
plot_kmeans(kmeans, X)

rng = np.random.RandomState(13)
X_stretched = np.dot(X, rng.randn(2, 2))

kmeans = KMeans(n_clusters=4, random_state=0)
plot_kmeans(kmeans, X_stretched)

from sklearn.mixture import GMM
gmm = GMM(n_components=4).fit(X)
labels = gmm.predict(X)
plt.scatter(X[:, 0], X[:, 1], c=labels, s=40, cmap='viridis');

probs = gmm.predict_proba(X)
print(probs[:5].round(3))

size = 50 * probs.max(1) ** 2  # square emphasizes differences
plt.scatter(X[:, 0], X[:, 1], c=labels, cmap='viridis', s=size);

from matplotlib.patches import Ellipse

def draw_ellipse(position, covariance, ax=None, **kwargs):
    """Draw an ellipse with a given position and covariance"""
    ax = ax or plt.gca()

    # Convert covariance to principal axes
    if covariance.shape == (2, 2):
        U, s, Vt = np.linalg.svd(covariance)
        angle = np.degrees(np.arctan2(U[1, 0], U[0, 0]))
        width, height = 2 * np.sqrt(s)
    else:
        angle = 0
        width,...


I am working on the file "Clustering.ipynb".
The first task is:
Given the knowledge of the ground truth class assignments labels_true and our clustering algorithm assignments of the same samples labels_pred.
Drawbacks
Contrary to inertia, MI-based measures require the knowledge of the ground truth classes while almost never available in practice or requires manual assignment by human annotators (as in the supervised learning setting).
Hierarchical clustering
Hierarchical clustering is a general family of clustering algorithms that build nested clusters by merging or splitting them successively. This hierarchy of clusters is represented as a tree (or dendrogram). The root of the tree is the unique cluster that gathers all the samples, the leaves being the clusters with only one sample.
The AgglomerativeClustering object performs a hierarchical clustering using a bottom up approach: each observation starts in its own cluster, and clusters are successively merged together. The linkage criteria determines the metric used for the merge strategy:

Ward minimizes the sum of squared differences within all clusters. It is a variance-minimizing approach and in this sense is similar to the k-means objective function but tackled with an agglomerative hierarchical approach.
Maximum or complete linkage minimizes the maximum distance between observations of pairs of clusters.
Average linkage minimizes the average of the distances between all observations of pairs of clusters.
Single linkage minimizes the distance between the closest observations of pairs of clusters.
Can you write Python code for it?

clustering = AgglomerativeClustering(n_clusters=4).fit(reduced_data)

clustering

clustering.labels_

 np.unique(clustering.labels_, return_counts=True)

from scipy.cluster.hierarchy import dendrogram, linkage

Z = linkage(reduced_data)

dendrogram(Z)
#dn1 = hierarchy.dendrogram(Z, ax=axes[0], above_threshold_color='y',orientation='top')
plt.show()

metrics.silhouette_score(reduced_data, clustering.labels_, metric='euclidean')


I am working on the file "Clustering.ipynb".
The first task is:
Here is an example with the same data that we used for GMM.  Note how the background shifts the centroids from what you might expect.  So, the mixture model might work better in this case.
However, one might consider runnig the K-means algorithm in order to find a suitable initialization for GMM.  Do you think that might have helped with our mis-behaving GMM example?
Can you write Python code for it?

# Execute this cell
# Based on Ivezic, Figure 6.13
# GTR fixed bug by changing preprocessing.Scaler() to preprocessing.StandardScaler()
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.patches import Ellipse
from scipy.stats import norm

from sklearn.cluster import KMeans
from sklearn import preprocessing

from astroML.datasets import fetch_sdss_sspp

#------------------------------------------------------------
# Get data
data = fetch_sdss_sspp(cleaned=True)
# Remove some outliers
data = data[~((data['alphFe'] > 0.4) & (data['FeH'] > -0.3))]

X = np.vstack([data['FeH'], data['alphFe']]).T

# truncate dataset for speed
X = X[::5]

#------------------------------------------------------------
# Compute a 2D histogram  of the input
H, FeH_bins, alphFe_bins = np.histogram2d(data['FeH'], data['alphFe'], 50)

#------------------------------------------------------------
# Compute the KMeans clustering
n_clusters = 4

scaler = preprocessing.StandardScaler()
clf = KMeans(n_clusters)
clf.fit(scaler.fit_transform(X))

#------------------------------------------------------------
# Visualize the results
fig = plt.figure(figsize=(5, 5))
ax = fig.add_subplot()

# plot density
ax = plt.axes()
ax.imshow(H.T, origin='lower', interpolation='nearest', aspect='auto',
          extent=[FeH_bins[0], FeH_bins[-1],
                  alphFe_bins[0], alphFe_bins[-1]],
          cmap=plt.cm.binary)

# plot cluster centers
cluster_centers = scaler.inverse_transform(clf.cluster_centers_)
ax.scatter(cluster_centers[:, 0], cluster_centers[:, 1],
           s=40, c='w', edgecolors='k')

# plot cluster boundaries
# using a fancy way to compute the...


I am working on the file "Clustering.ipynb".
The first task is:
Hierarchical Clustering
Can you write Python code for it?

def is_leaf(cluster):
    """a cluster is a leaf if it has len 1"""
    return len(cluster) == 1

def get_children(cluster):
    """returns children of the cluster if merged else exception"""
    if is_leaf(cluster):
        raise TypeError("a leaf cluster has no children")
    else:
        return cluster[1]

def get_values(cluster):
    """returns the value in the cluster (if leaf)
    or all values in leaf clusters below"""
    if is_leaf(cluster):
        return cluster
    else:
        return [value
                for child in get_children(cluster)
                for value in get_values(child)]

def cluster_distance(cluster1, cluster2, distance_agg = min):
    """compute all pairwise distances btw clusters
    and apply distance_agg to the list"""
    return distance_agg([distance(input1, input2)
                        for input1 in get_values(cluster1)
                        for input2 in get_values(cluster2)])

def get_merge_order(cluster):
    if is_leaf(cluster):
        return float('inf')
    else:
        return cluster[0]

def bottom_up_cluster(inputs, distance_agg = min):
    # we start with all leaf clusters (this is bottom up after all)
    clusters = [(input,) for input in inputs]

    # Don't stop until we have one cluster
    while len(clusters) > 1:
        # the two clusters we want to merge
        # are the clusters that are closest without touching
        c1, c2 = min([(cluster1, cluster2)
                     for i, cluster1 in enumerate(clusters)
                     for cluster2 in clusters[:i]],
                     key = lambda (x,y): cluster_distance(x, y, distance_agg))

        # the above is really inefficient in distance calc
        # we should instead "look up" the distance

        # once we merge them we remove them from the list
        clusters = [c for c in clusters if c != c1 and c != c2]

        # merge them with order = # of clusters left (so that last merge is "0")
        merged_cluster = (len(clusters), [c1, c2])

        # append...


I am working on the file "Clustering.ipynb".
The first task is:
Hierarchical Clustering
In Hierarchical Clustering, we don't specify the number of clusters ahead of time, we start with $N$ clusters representing each data point.  Then the most similar clusters are joined together, the process repeating until some threshhold is reached.  Actually the process can go in the other direction as well.  What results is what is called a dendrogram, an example of which is shown below.

Clusters are merged at each step according to which are "nearest" to each other---where the definition of nearest needs to be specified.  A typical choice results in what is called a "minimum spanning tree" (which can be quite slow for large data sets).  Some threshhold needs to be specified to tell the process where to stop (e.g., we are going to treat the green and red objects in the example above as separate clusters).
Below is an example call.
Can you write Python code for it?

# Execute this cell
import numpy
from scipy.sparse.csgraph import minimum_spanning_tree
from sklearn.neighbors import kneighbors_graph

X = np.random.random((1000,2))
G = kneighbors_graph(X, n_neighbors=10, mode='distance')
T = minimum_spanning_tree(G)


I am working on the file "Clustering.ipynb".
The first task is:
K-means clustering
We can perform k-means clustering on this data with 3 clusters and using the resulting clusters as a way to predict a class label. The fit_predict function of clustering algorithms does just that. Notice that the clustering assign each point to a cluster in an unsupervised manner i.e it only takes into account X not y like classifiers do. Comparing the original scatter plot with the resulting predictions of the clustering algorithm shows where potential erros can happen. As you can see the dark blue cluster "takes over" some of the light blue cluster.
We can also look at other clustering methods such as spectral clustering and Gaussian Mixture Models.
Can you write Python code for it?

random_state = 170
cluster_names = ['cluster 1', 'cluster 2', 'cluster 3']

def plot_clustering(X,y,y_pred):
    fig = plt.figure(figsize=(12,4))
    for n, color in enumerate(colors):
        data = X[y == n]
        # Plot the training data
        plt.subplot(131)
        plt.subplots_adjust(bottom=.01, top=0.95, hspace=.15, wspace=.15, left=.01, right=.99)
        plt.scatter(data[:, 0], data[:, 1], s=20, color=color,label=target_names[n])
        plt.legend(scatterpoints=1, loc='upper right', prop=dict(size=10))

        # Plot the cluster predictions
        data = X[y_pred == n]
        h = plt.subplot(132)
        plt.scatter(data[:, 0], data[:, 1], s=20, marker='x',color=color,label=cluster_names[n])
        plt.legend(scatterpoints=1, loc='upper right', prop=dict(size=10))
        plt.subplot(133)

        data = X[y == n]
        plt.scatter(data[:, 0], data[:, 1], s=20, color=color,
                    label=target_names[n])
        data = X[y_pred == n]

        plt.scatter(data[:, 0], data[:, 1], s=40, marker='x',color=color,
                    label=cluster_names[n])
        plt.legend(scatterpoints=1, loc='upper right', prop=dict(size=10))

        train_accuracy = np.mean(y_pred.ravel() == y.ravel()) * 100
        plt.text(0.7, 0.5, 'Accuracy: %.1f' % train_accuracy,
             transform=h.transAxes)
    plt.show()

y_pred = KMeans(n_clusters=3, random_state=random_state).fit_predict(X)
plot_clustering(X,y,y_pred)
y_pred = SpectralClustering(n_clusters=3, eigen_solver='arpack',
                            random_state=random_state,assign_labels = 'kmeans',
                            affinity='nearest_neighbors').fit_predict(X)
plot_clustering(X,y,y_pred)
gmm = GaussianMixture(n_components=3, random_state=50, max_iter=100)
gmm.fit(X)
y_pred = gmm.predict(X)
plot_clustering(X,y,y_pred)

# rerrange cluster numbers to make things work
#cluster_mapping = {}
#cluster_mapping[0] = 0
#cluster_mapping[1] = 2
#cluster_mapping[2] = 1
y_mapped_predict = np.array([cluster_mapping[i]...


I am working on the file "Clustering.ipynb".
The first task is:
Mean Shift
MeanShift clustering aims to discover blobs in a smooth density of samples. It is a centroid based algorithm, which works by updating candidates for centroids to be the mean of the points within a given region. These candidates are then filtered in a post-processing stage to eliminate near-duplicates to form the final set of centroids.
Mean shift clustering using a flat kernel.
Mean shift clustering aims to discover “blobs” in a smooth density of samples. It is a centroid-based algorithm, which works by updating candidates for centroids to be the mean of the points within a given region. These candidates are then filtered in a post-processing stage to eliminate near-duplicates to form the final set of centroids.
Seeding is performed using a binning technique for scalability.
Can you write Python code for it?

print(__doc__)

import numpy as np
from sklearn.cluster import MeanShift, estimate_bandwidth
from sklearn.datasets.samples_generator import make_blobs

# #############################################################################
# Generate sample data
centers = [[1, 1], [-1, -1], [1, -1]]
X = reduced_data

# #############################################################################
# Compute clustering with MeanShift

# The following bandwidth can be automatically detected using
bandwidth = estimate_bandwidth(X, quantile=0.2, n_samples=500)

ms = MeanShift(bandwidth=bandwidth, bin_seeding=True)
ms.fit(X)
labels = ms.labels_
cluster_centers = ms.cluster_centers_

labels_unique = np.unique(labels)
n_clusters_ = len(labels_unique)

print("number of estimated clusters : %d" % n_clusters_)

# #############################################################################
# Plot result
import matplotlib.pyplot as plt
from itertools import cycle

plt.figure(1)
plt.clf()

colors = cycle('bgrcmykbgrcmykbgrcmykbgrcmyk')
for k, col in zip(range(n_clusters_), colors):
    my_members = labels == k
    cluster_center = cluster_centers[k]
    plt.plot(X[my_members, 0], X[my_members, 1], col + '.')
    plt.plot(cluster_center[0], cluster_center[1], 'o', markerfacecolor=col,
             markeredgecolor='k', markersize=14)
plt.title('Estimated number of clusters: %d' % n_clusters_)
plt.show()


I am working on the file "Clustering.ipynb".
The first task is:
OK, but that's all that the book give us.  There is nothing about what to do with G and T.  So, instead I'm going to show you a really cool example from a colleague.  In this example Nathalie Thibert is taking spectroscopic data of a certain sub-class of quasars.  She is then grouping the objects into "like" bins using a hierarchical clustering algorithm.  The code below is based on the scipy implementation and takes us through both the analysis and visualization of the data.  It makes use of the Python Data Analysis Library (pandas) and 'pickled'
data, both of which we have not talked about.
For another detailed example of hierarchical clustering, see https://joernhees.de/blog/2015/08/26/scipy-hierarchical-clustering-and-dendrogram-tutorial/.
Can you write Python code for it?

%load code/thibert_cluster1.py

%load code/thibert_cluster2.py


I am working on the file "Clustering.ipynb".
The first task is:
Principal component analysis
PCA is a tecnique for dimensionality reduction. In this example we take as input a similar audio feature data-set for the 3 genres we have been exploring but with 124 features per track instead. Using PCA we can reduce the dimensionality to 3 dimensions.
Can you write Python code for it?

(X, y) = datasets.load_svmlight_file("data/3genres_full.arff.libsvm")
print(X.shape)
X = X.toarray()
X = MinMaxScaler().fit_transform(X)
target_names = ['classical', 'jazz', 'metal']

# To getter a better understanding of interaction of the dimensions
# plot the first three PCA dimensions
fig = plt.figure(1, figsize=(8, 6))
ax = Axes3D(fig, elev=-150, azim=110)
X_reduced = PCA(n_components=3).fit_transform(X)
ax.scatter(X_reduced[:, 0], X_reduced[:, 1], X_reduced[:, 2], c=y,
           cmap=plt.cm.Set1, edgecolor='k', s=40)
ax.set_title("First three PCA directions")
ax.set_xlabel("1st eigenvector")
ax.w_xaxis.set_ticklabels([])
ax.set_ylabel("2nd eigenvector")
ax.w_yaxis.set_ticklabels([])
ax.set_zlabel("3rd eigenvector")
ax.w_zaxis.set_ticklabels([])
plt.show()


I am working on the file "Clustering.ipynb".
The first task is:
The optimal number of clusters is the value that minimizes the AIC or BIC, depending on which approximation we wish to use. Here it is 8.
BIRCH
The Birch (Balanced Iterative Reducing and Clustering using Hierarchies ) builds a tree called the Characteristic Feature Tree (CFT) for the given data. The data is essentially lossy compressed to a set of Characteristic Feature nodes (CF Nodes). The CF Nodes have a number of subclusters called Characteristic Feature subclusters (CF Subclusters) and these CF Subclusters located in the non-terminal CF Nodes can have CF Nodes as children.
The CF Subclusters hold the necessary information for clustering which prevents the need to hold the entire input data in memory. This information includes:

Number of samples in a subcluster.
Linear Sum - A n-dimensional vector holding the sum of all samples
Squared Sum - Sum of the squared L2 norm of all samples.
Centroids - To avoid recalculation linear sum / n_samples.
Squared norm of the centroids.

It is a memory-efficient, online-learning algorithm provided as an alternative to MiniBatchKMeans. It constructs a tree data structure with the cluster centroids being read off the leaf. These can be either the final cluster centroids or can be provided as input to another clustering algorithm such as AgglomerativeClustering.
Can you write Python code for it?

from sklearn.cluster import Birch

X = reduced_data
brc = Birch(branching_factor=50, n_clusters=None, threshold=0.5,compute_labels=True)
brc.fit(X)

brc.predict(X)

labels = brc.predict(X)
plt.scatter(reduced_data[:, 0], reduced_data[:, 1], c=labels, s=40, cmap='viridis');
plt.show()


I am working on the file "Clustering.ipynb".
The first task is:
To measure the quality of clustering results, there are two kinds of validity indices: external indices and internal indices.
An external index is a measure of agreement between two partitions where the first partition is the a priori known clustering structure, and the second results from the clustering procedure (Dudoit et al., 2002).
Internal indices are used to measure the goodness of a clustering structure without external information (Tseng et al., 2005
Can you write Python code for it?

from sklearn.decomposition import PCA
from sklearn.preprocessing import scale

clusdf_scale = scale(clusdf)
n_samples, n_features = clusdf_scale.shape

n_samples, n_features

reduced_data = PCA(n_components=2).fit_transform(clusdf_scale)

#assuming height to be Y variable to be predicted
#n_digits = len(np.unique(clusdf.height))
#From R Cluster sizes:
#[1] "26 29 5 32"
n_digits=4

kmeans = KMeans(init='k-means++', n_clusters=n_digits, n_init=10)
kmeans.fit(reduced_data)

clusdf.head(20)

# Plot the decision boundary. For that, we will assign a color to each
h=0.02
x_min, x_max = reduced_data[:, 0].min() - 1, reduced_data[:, 0].max() + 1
y_min, y_max = reduced_data[:, 1].min() - 1, reduced_data[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

# Obtain labels for each point in mesh. Use last trained model.
Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(1)
plt.clf()
plt.imshow(Z, interpolation='nearest',
           extent=(xx.min(), xx.max(), yy.min(), yy.max()),
           cmap=plt.cm.Paired,
           aspect='auto', origin='lower')

plt.plot(reduced_data[:, 0], reduced_data[:, 1], 'k.', markersize=2)
# Plot the centroids as a white X
centroids = kmeans.cluster_centers_
plt.scatter(centroids[:, 0], centroids[:, 1],
            marker='x', s=169, linewidths=3,
            color='w', zorder=10)
plt.title('K-means clustering on the digits dataset (PCA-reduced data)\n'
          'Centroids are marked with white cross')
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.xticks(())
plt.yticks(())
plt.show()

kmeans = KMeans(n_clusters=4, random_state=0).fit(reduced_data)

kmeans.labels_

 np.unique(kmeans.labels_, return_counts=True)

import matplotlib.pyplot as plt
%matplotlib inline

plt.hist(kmeans.labels_)
plt.show()

kmeans.cluster_centers_

metrics.silhouette_score(reduced_data, kmeans.labels_, metric='euclidean')


I am working on the file "Clustering.ipynb".
The first task is:
Try replacing the $K$-means algorithm in the figure above with Mean shift.  Call it with bin_seeding=True to speed things up a bit.
Here is an example using some real data provided by Prof. Cruz.
Can you write Python code for it?

# Execute the next few cells
from astropy.table import Table
t = Table.read('data/cruz_all_dist.dat', format="ascii")

# Just something that you should know that you can do
t[::10000].show_in_notebook()

# Turn these data into a properly formatted Scikit-Learn array
X = np.vstack([ t['col2'], t['col3'], t['col4'], t['col5'] ]).T
print X

# Project onto 2 axes with PCA
from sklearn.decomposition import PCA
pca = PCA(n_components=2) # 2 components
pca.fit(X) # Do the fitting

X_reduced = pca.transform(X)

plt.scatter(X_reduced[:, 0], X_reduced[:, 1], marker=".", color='k', edgecolors='None')

# Compute the KMeans clustering
n_clusters = 6

scaler = preprocessing.StandardScaler()
clf = KMeans(n_clusters)
clf.fit(scaler.fit_transform(X_reduced))

# Make some plots
fig = plt.figure(figsize=(5, 5))
ax = fig.add_subplot()

# Compute a 2D histogram  of the input
H, xedges, yedges = np.histogram2d(X_reduced[:,0], X_reduced[:,1], 50)

# plot density
ax = plt.axes()
ax.imshow(H.T, origin='lower', interpolation='nearest', aspect='auto',
          extent=[xedges[0], xedges[-1],
                  yedges[0], yedges[-1]],
          cmap=plt.cm.binary)

# plot cluster centers
cluster_centers = scaler.inverse_transform(clf.cluster_centers_)
ax.scatter(cluster_centers[:, 0], cluster_centers[:, 1],
           s=40, c='w', edgecolors='k')

# plot cluster boundaries
x_centers = 0.5 * (xedges[1:] + xedges[:-1])
y_centers = 0.5 * (yedges[1:] + yedges[:-1])

Xgrid = np.meshgrid(x_centers, y_centers)
Xgrid = np.array(Xgrid).reshape((2, 50 * 50)).T

H = clf.predict(scaler.transform(Xgrid)).reshape((50, 50))

for i in range(n_clusters):
    Hcp = H.copy()
    flag = (Hcp == i)
    Hcp[flag] = 1
    Hcp[~flag] = 0

    ax.contour(x_centers, y_centers, Hcp, [-0.5, 0.5],
               linewidths=1, colors='k')

#ax.xaxis.set_major_locator(plt.MultipleLocator(0.3))
ax.set_xlim(xedges[0], xedges[-1])
ax.set_ylim(yedges[0], yedges[-1])

ax.set_xlabel('Eigenvalue 1')
ax.set_ylabel('Eigenvalue 2')

plt.savefig('cruz.png')
plt.show()


I am working on the file "Clustering.ipynb".
The first task is:
We can compare the accuracy of clustering using the original ground truth labels based on the 124 features vs the accuracy using the 3-dimensional features reduced by PCA. As can be seen the performance is similar - the increase might be due to reduction of noise in the features.
Can you write Python code for it?

y_pred = KMeans(n_clusters=3, random_state=random_state).fit_predict(X)
train_accuracy = np.mean(y_pred.ravel() == y.ravel()) * 100
print(train_accuracy)

y_pred = KMeans(n_clusters=3, random_state=random_state).fit_predict(X_reduced)
train_accuracy = np.mean(y_pred.ravel() == y.ravel()) * 100
print(train_accuracy)


I am working on the file "Clustering.ipynb".
The first task is:
We can use the reduced PCA vectors as a way to visualize the results of clustering in the high-dimensional (124) original feature space.
Can you write Python code for it?

y_pred = KMeans(n_clusters=3, random_state=random_state).fit_predict(X)
# To getter a better understanding of interaction of the dimensions
# plot the first three PCA dimensions
fig = plt.figure(1, figsize=(8, 6))
ax = Axes3D(fig, elev=-150, azim=110)
X_reduced = PCA(n_components=3).fit_transform(X)
ax.scatter(X_reduced[:, 0], X_reduced[:, 1], X_reduced[:, 2], c=y_pred,
           cmap=plt.cm.Set1, edgecolor='k', s=40)
ax.set_title("First three PCA directions")
ax.set_xlabel("1st eigenvector")
ax.w_xaxis.set_ticklabels([])
ax.set_ylabel("2nd eigenvector")
ax.w_yaxis.set_ticklabels([])
ax.set_zlabel("3rd eigenvector")
ax.w_zaxis.set_ticklabels([])
plt.show()


I am working on the file "Clustering.ipynb".
The first task is:
https://vincentarelbundock.github.io/Rdatasets/doc/cluster/plantTraits.html
Usage
data(plantTraits)
Format
A data frame with 136 observations on the following 31 variables.

pdias
Diaspore mass (mg)

longindex
Seed bank longevity

durflow
Flowering duration

height
Plant height, an ordered factor with levels 1 < 2 < ... < 8.

begflow
Time of first flowering, an ordered factor with levels 1 < 2 < 3 < 4 < 5 < 6 < 7 < 8 < 9

mycor
Mycorrhizas, an ordered factor with levels 0never < 1 sometimes< 2always

vegaer
aerial vegetative propagation, an ordered factor with levels 0never < 1 present but limited< 2important.

vegsout
underground vegetative propagation, an ordered factor with 3 levels identical to vegaer above.

autopoll
selfing pollination, an ordered factor with levels 0never < 1rare < 2 often< the rule3

insects
insect pollination, an ordered factor with 5 levels 0 < ... < 4.

wind
wind pollination, an ordered factor with 5 levels 0 < ... < 4.

lign
a binary factor with levels 0:1, indicating if plant is woody.

piq
a binary factor indicating if plant is thorny.

ros
a binary factor indicating if plant is rosette.

semiros
semi-rosette plant, a binary factor (0: no; 1: yes).

leafy
leafy plant, a binary factor.

suman
summer annual, a binary factor.

winan
winter annual, a binary factor.

monocarp
monocarpic perennial, a binary factor.

polycarp
polycarpic perennial, a binary factor.

seasaes
seasonal aestival leaves, a binary factor.

seashiv
seasonal hibernal leaves, a binary factor.

seasver
seasonal vernal leaves, a binary factor.

everalw
leaves always evergreen, a binary factor.

everparti
leaves partially evergreen, a binary factor.

elaio
fruits with an elaiosome (dispersed by ants), a binary factor.

endozoo
endozoochorous fruits, a binary factor.

epizoo
epizoochorous fruits, a binary factor.

aquat
aquatic dispersal fruits, a binary factor.

windgl
wind dispersed fruits, a binary factor.

unsp
unspecialized mechanism of seed dispersal, a binary factor.
Can you write Python code for it?

clusdf = clusdf.drop("Unnamed: 0", axis=1)

clusdf.head()

clusdf.info()

#missing values
clusdf.apply(lambda x: sum(x.isnull().values), axis = 0)

clusdf.head(20)

clusdf=clusdf.fillna(clusdf.mean())


I am working on the file "Clustering.ipynb".
The first task is:
k - means

Randomly select starting locations for k points
Assign data points to closest k point
If no data changed its cluster membership stop
If there was a change, compute new means and repeat
Can you write Python code for it?

class KMeans:
    """k-means algo"""

    def __init__(self, k):
        self.k = k # number of clusters
        self.means = None # means of clusters

    def classify(self, input):
        """return the index of the cluster to closest to input"""
        return min(range(self.k),
                  key = lambda i: squared_distance(input, self.means[i]))

    def train(self, inputs):
        # choose k rand points as initials
        self.means = random.sample(inputs, self.k)
        assignments = None

        while True:
            # Find new assignments
            new_assignments = map(self.classify, inputs)

            # If nothing changed we're good to go
            if assignments == new_assignments:
                return

            # otherwise keep
            assignments = new_assignments

            # And compute new means based on assigments
            for i in range(self.k):
                # get points in cluster
                i_points = [p for p,a in zip(inputs, assignments) if a == i]

                # check for membership
                if i_points:
                    self.means[i] = vector_mean(i_points)

inputs = [[-14,-5],[13,13],[20,23],[-19,-11],[-9,-16],[21,27],[-49,15],[26,13],[-46,5],[-34,-1],[11,15],[-49,0],[-22,-16],[19,28],[-12,-8],[-13,-19],[-41,8],[-11,-6],[-25,-9],[-18,-3]]

random.seed(0)
clusterer = KMeans(2)
clusterer.train(inputs)

clusterer.means


I am working on the file "Clustering.ipynb".
The first task is:
knowledge of the ground truth class assignments labels_true and
our clustering algorithm assignments of the same samples labels_pred
https://scikit-learn.org/stable/modules/clustering.html#clustering-performance-evaluation
- adjusted Rand index is a function that measures the similarity of the two assignments

the Mutual Information is a function that measures the agreement of the two assignments, ignoring permutations.

The following two desirable objectives for any cluster assignment:
- homogeneity: each cluster contains only members of a single class.
- completeness: all members of a given class are assigned to the same cluster.
We can turn those concept as scores.Both are bounded below by 0.0 and above by 1.0 (higher is better)

homogeneity_score and
completeness_score.

Their harmonic mean called V-measure is computed by
- v_measure_score

The Silhouette Coefficient is defined for each sample and is composed of two scores:
a: The mean distance between a sample and all other points in the same class.
b: The mean distance between a sample and all other points in the next nearest cluster.
Can you write Python code for it?

from sklearn import metrics
from sklearn.metrics import pairwise_distances
from sklearn import datasets
dataset = datasets.load_iris()
X = dataset.data
y = dataset.target

import numpy as np
from sklearn.cluster import KMeans
kmeans_model = KMeans(n_clusters=3, random_state=1).fit(X)
labels = kmeans_model.labels_

labels_true=y
labels_pred=labels

from sklearn import metrics
metrics.adjusted_rand_score(labels_true, labels_pred)

from sklearn import metrics
metrics.adjusted_mutual_info_score(labels_true, labels_pred)

metrics.homogeneity_score(labels_true, labels_pred)

metrics.completeness_score(labels_true, labels_pred)

metrics.v_measure_score(labels_true, labels_pred)

metrics.silhouette_score(X, labels, metric='euclidean')


I am working on the file "Clustering/Applying K-Means for Image Quantization.ipynb".
The first task is:
Finally, creating the compressed image from the compressed palette and cluster assignments:
Can you write Python code for it?

compressed_palette = estimator.cluster_centers_
compressed_img = np.zeros((width, height, compressed_palette.shape[1]))
label_idx = 0
for i in range(width):
    for j in range(height):
        compressed_img[i][j] = compressed_palette[cluster_assignments[label_idx]]
        label_idx += 1

#Plotting the original and Compressed versions of the Image
plt.subplot(122)
plt.title('Original Image')
plt.imshow(original_img)
plt.axis('off')
plt.subplot(121)
plt.title('Compressed Image')
plt.imshow(compressed_img)
plt.axis('off')
plt.show()


I am working on the file "Clustering/Applying K-Means for Image Quantization.ipynb".
The first task is:
Reading and flattening the image
Can you write Python code for it?

im = mh.imread('images/image.png')
original_img = np.array(im, dtype=np.float64) / 255
original_dimensions = tuple(original_img.shape)
width, height, depth = tuple(original_img.shape)
image_flattened = np.reshape(original_img, (width * height, depth))


I am working on the file "Clustering/Applying K-Means for Image Quantization.ipynb".
The first task is:
Using K-Means Algorithm for clustering
Here we use K-Means to create 64 clusters from a sample of 1,000 randomly selected
colors. Each of the clusters will be a color in the compressed palette.
Can you write Python code for it?

image_array_sample = shuffle(image_flattened, random_state=0)[:1000]
estimator = KMeans(n_clusters=64, random_state=0)
estimator.fit(image_array_sample)


I am working on the file "Clustering/Customer Segmentation for Market Analysis.ipynb".
The first task is:
Affinity Propogation
Can you write Python code for it?

# Affinity propagation
aff = cluster.AffinityPropagation()
aff.fit(X_train)
print (aff.cluster_centers_indices_.shape)

y_pred = aff.predict(X_test)

from sklearn import metrics
print ("Addjusted rand score:{:.2}".format(metrics.adjusted_rand_score(y_test, y_pred)))
print ("Homogeneity score:{:.2} ".format(metrics.homogeneity_score(y_test, y_pred)) )
print ("Completeness score: {:.2} ".format(metrics.completeness_score(y_test, y_pred)))
print ("Confusion matrix")
print (metrics.confusion_matrix(y_test, y_pred))


I am working on the file "Clustering/Customer Segmentation for Market Analysis.ipynb".
The first task is:
Mixture of Guassian Models
Can you write Python code for it?

from sklearn import mixture

# Define a heldout dataset to estimate covariance type
X_train_heldout, X_test_heldout, y_train_heldout, y_test_heldout = train_test_split(
        X_train, y_train,test_size=0.25, random_state=42)
for covariance_type in ['spherical','tied','diag','full']:
    gm=mixture.GMM(n_components=3, covariance_type=covariance_type, random_state=42, n_init=5)
    gm.fit(X_train_heldout)
    y_pred=gm.predict(X_test_heldout)
    print ("Adjusted rand score for covariance={}:{:.2}".format(covariance_type,
                                                               metrics.adjusted_rand_score(y_test_heldout, y_pred)))

gm = mixture.GMM(n_components=3, covariance_type='tied', random_state=42)
gm.fit(X_train)

# Print train clustering and confusion matrix
y_pred = gm.predict(X_test)
print ("Addjusted rand score:{:.2}".format(metrics.adjusted_rand_score(y_test, y_pred)))
print ("Homogeneity score:{:.2} ".format(metrics.homogeneity_score(y_test, y_pred)) )
print ("Completeness score: {:.2} ".format(metrics.completeness_score(y_test, y_pred)))

print ("Confusion matrix")
print (metrics.confusion_matrix(y_test, y_pred))


I am working on the file "Clustering/Iris/Iris.ipynb".
The first task is:
K Means (3 clusters)
Can you write Python code for it?

#train a k-nearest neighbor algorithm
fit = KMeans(n_clusters=3).fit(X_scaled)
fit.labels_

#remake labels so that they properly matchup with the classes
labels = fit.labels_[:]
for index,val in enumerate(labels):
    if val == 1:
        labels[index] = 1
    elif val == 2:
        labels[index] = 3
    else:
        labels[index] = 2

labels

conf_mat = np.zeros((3,3))

true = np.array([0]*50 + [1]*50 + [2]*50)

for i,val in enumerate(true):
    conf_mat[val,labels[i]-1] += 1

#true vs. predicted
print(pd.DataFrame(conf_mat))


I am working on the file "Clustering/Iris/Iris.ipynb".
The first task is:
Scatter Plot Matrix
Can you write Python code for it?

# Visualize dataset with scatterplot matrix
%matplotlib inline

g = sns.PairGrid(data, hue="species")
g.map_diag(plt.hist)
g.map_offdiag(plt.scatter)


I am working on the file "Clustering/Seeds Clustering.ipynb".
The first task is:

Can you write Python code for it?

from sklearn.cross_validation import train_test_split
X = df[features]
y = df['Class']
X_train, X_test, y_train, y_test = train_test_split(X, y ,test_size=0.25, random_state=42)


I am working on the file "Clustering/Seeds Clustering.ipynb".
The first task is:
<hr>
Elbow Method
Using the elbow method to find the optimal number of clusters
One of the main challenges in unsupervised learning is that we do not know the definitive answer. We don't have the ground truth class labels in our dataset that allow us to apply the techniques in order to evaluate the performance of a supervised model. Thus, in order to quantify the quality of clustering, we need to use intrinsic metrics—such as the within-cluster SSE (distortion) to compare the performance of different k-means clusterings. Conveniently, we don't need to compute the within-cluster SSE explicitly as it is already accessible via the inertia_ attribute after fitting a KMeans model.
Based on the within-cluster SSE, we can use a graphical tool, the so-called elbow method, to estimate the optimal number of clusters k for a given task. Intuitively,
we can say that, if k increases, the distortion will decrease. This is because the
samples will be closer to the centroids they are assigned to. The idea behind the elbow method is to identify the value of k where the distortion begins to increase most rapidly, which will become more clear if we plot distortion for different
values of k:
Can you write Python code for it?

distortions = []
for i in range(1, 11):
    km = KMeans(n_clusters=i,
                init='k-means++',
                n_init=10,
                max_iter=300,
                random_state=0)
    km.fit(X)
    distortions .append(km.inertia_)
plt.plot(range(1,11), distortions , marker='o')
plt.xlabel('Number of clusters')
plt.ylabel('Distortion')
plt.tight_layout()
#plt.savefig('./figures/elbow.png', dpi=300)
plt.show()


I am working on the file "Clustering/Seeds Clustering.ipynb".
The first task is:
<hr>
Mixture of Guassian Models
Can you write Python code for it?

from sklearn import mixture

# Define a heldout dataset to estimate covariance type
X_train_heldout, X_test_heldout, y_train_heldout, y_test_heldout = train_test_split(
        X_train, y_train,test_size=0.25, random_state=42)
for covariance_type in ['spherical','tied','diag','full']:
    gm=mixture.GMM(n_components=3, covariance_type=covariance_type, random_state=42, n_init=5)
    gm.fit(X_train_heldout)
    y_pred=gm.predict(X_test_heldout)
    print ("Adjusted rand score for covariance={}:{:.2}".format(covariance_type,
                                                               metrics.adjusted_rand_score(y_test_heldout, y_pred)))

gm = mixture.GMM(n_components=3, covariance_type='tied', random_state=42)
gm.fit(X_train)

# Print train clustering and confusion matrix
y_pred = gm.predict(X_test)
print ("Addjusted rand score:{:.2}".format(metrics.adjusted_rand_score(y_test, y_pred)))
print ("Homogeneity score:{:.2} ".format(metrics.homogeneity_score(y_test, y_pred)) )
print ("Completeness score: {:.2} ".format(metrics.completeness_score(y_test, y_pred)))

print ("Confusion matrix")
print (metrics.confusion_matrix(y_test, y_pred))

pl=plt
from sklearn import decomposition
# In this case the seeding of the centers is deterministic,
# hence we run the kmeans algorithm only once with n_init=1

pca = decomposition.PCA(n_components=2).fit(X_train)
reduced_X_train = pca.transform(X_train)

# Step size of the mesh. Decrease to increase the quality of the VQ.
h = .01     # point in the mesh [x_min, m_max]x[y_min, y_max].

# Plot the decision boundary. For that, we will asign a color to each
x_min, x_max = reduced_X_train[:, 0].min() + 1, reduced_X_train[:, 0].max() - 1
y_min, y_max = reduced_X_train[:, 1].min() + 1, reduced_X_train[:, 1].max() - 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

gm.fit(reduced_X_train)
#print np.c_[xx.ravel(),yy.ravel()]
Z = gm.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z =...


I am working on the file "Clustering/Seeds Clustering.ipynb".
The first task is:

Unsupervised Learning

Feature Transformation
The first PCA dimension is the dimension in the data with highest variance. Intuitively, it corresponds to the 'longest' vector one can find in the 6-dimensional feature space that captures the data, that is, the eigenvector with the largest eigenvalue.
ICA, as opposed to PCA, finds the subcomponents that are statistically independent.
PCA
Can you write Python code for it?


I am working on the file "Clustering/Seeds Clustering.ipynb".
The first task is:
As we can see in the following plot, the elbow is located at k = 3, which provides evidence that k = 3 is indeed a good choice for this dataset.
Quantifying the quality of clustering via silhouette plots
Another intrinsic metric to evaluate the quality of a clustering is silhouette analysis, which can also be applied to clustering algorithms other than k-means that we will discuss later in this chapter. Silhouette analysis can be used as a graphical tool to plot a measure of how tightly grouped the samples in the clusters are. To calculate the silhouette coefficient of a single sample in our dataset, we can apply the following three steps:
1. Calculate the cluster cohesion a(i) as the average distance between a sample x(i) and all other points in the same cluster.
2. Calculate the cluster separation b(i) from the next closest cluster as the average distance between the sample x(i) and all samples in the nearest cluster.
3. Calculate the silhouette s(i) as the difference between cluster cohesion and separation divided by the greater of the two as shown :
s(i) = b(i) - a(i) / max(b(i),a(i))
The silhouette coefficient is bounded in the range -1 to 1. Based on the preceding formula, we can see that the silhouette coefficient is 0 if the cluster separation and cohesion are equal (b(i)=a(i)). Furthermore, we get close to an ideal silhouette coefficient of 1 if (b(i)>>a(i)) since b(i) quantifies how dissimilar a sample is to other clusters, and a(i) tells us how similar it is to the other samples in its own cluster, respectively.
The silhouette coefficient is available as silhouette_samples from scikit-learn's metric module, and optionally the silhouette_scores can be imported. This calculates the average silhouette coefficient across all samples, which is equivalent to numpy.mean(silhouette_samples(…)). By executing the following code, we will now create a plot of the silhouette coefficients for a k-means clustering with k=3:
Can you write Python code for it?

import numpy as np
from matplotlib import cm
from sklearn.metrics import silhouette_samples

km = KMeans(n_clusters=3,
            init='k-means++',
            n_init=10,
            max_iter=300,
            tol=1e-04,
            random_state=0)
y_km = km.fit_predict(X)

cluster_labels = np.unique(y_km)
n_clusters = cluster_labels.shape[0]
silhouette_vals = silhouette_samples(X, y_km, metric='euclidean')
y_ax_lower, y_ax_upper = 0, 0
yticks = []
for i, c in enumerate(cluster_labels):
    c_silhouette_vals = silhouette_vals[y_km == c]
    c_silhouette_vals.sort()
    y_ax_upper += len(c_silhouette_vals)
    color = cm.jet(i / n_clusters)
    plt.barh(range(y_ax_lower, y_ax_upper), c_silhouette_vals, height=1.0,
            edgecolor='none', color=color)

    yticks.append((y_ax_lower + y_ax_upper) / 2)
    y_ax_lower += len(c_silhouette_vals)

silhouette_avg = np.mean(silhouette_vals)
plt.axvline(silhouette_avg, color="red", linestyle="--")

plt.yticks(yticks, cluster_labels + 1)
plt.ylabel('Cluster')
plt.xlabel('Silhouette coefficient')

plt.tight_layout()
# plt.savefig('./figures/silhouette.png', dpi=300)
plt.show()


I am working on the file "Clustering/Seeds Clustering.ipynb".
The first task is:
Clustering
In this section we will choose either K Means clustering or Gaussian Mixed Models clustering, which implements expectation-maximization. Then we will sample elements from the clusters to understand their significance.
Choosing a Cluster Type
K Means Clustering or Gaussian Mixture Models?
Before discussing the advantages of K Means vs Gaussian Mixture models, it is helpful to observe that both methods are actually very similar. The main difference is that Gaussian Mixture models make a probabilistic assignment of points to classes depending on some distance metric, whereas K Means makes a deterministic assignment depending on some metric. Now, when the variance of the Gaussian mixtures is very small, this method becomes very similar to K Means, since the assignment probabilities to a specific cluster converge to 0 or 1 for any point in the domain. Because of the probabilistic assignment, Gaussian Mixtures (in contrast to K Means) are often characterized as soft clustering algorithms.
An advantage of Gaussian Mixture models is that, if there is some a priori uncertainty about the assignment of a point to a cluster, this uncertainty is inherently reflected in the probabilistic model (soft assignment) and assignment probabilities can be computed for any data point after the model is trained. On the other hand, if a priori the clusters assignments are expected to be deterministic, K Means has advantages. An example would be a data generating process that actually is a mixture of Gaussians. Applying a Gaussian mixture model is more natural given this data generating process. When it comes to processing speed, the EM algorithm with Gaussian mixtures is generally slightly slower than Lloyd's algorithm for K Means, since computing the normal probability (EM) is generally slower than computing the L2-norm (K Means). A disadvantage of both methods is that they can get stuck in local minima (this can be considered as the cost of solving NP-hard problems (global min for k-means) approximately).
Since there is...
Can you write Python code for it?

# Import clustering modules
from sklearn.cluster import KMeans
from sklearn.mixture import GMM

# First we reduce the data to two dimensions using PCA to capture variation
pca = PCA(n_components=2)
reduced_data = pca.fit_transform(X)
print(reduced_data[:10])  # print upto 10 elements

kmeans = KMeans(n_clusters=3)
clusters = kmeans.fit(reduced_data)
print(clusters)

# Plot the decision boundary by building a mesh grid to populate a graph.
x_min, x_max = reduced_data[:, 0].min() - 1, reduced_data[:, 0].max() + 1
y_min, y_max = reduced_data[:, 1].min() - 1, reduced_data[:, 1].max() + 1
hx = (x_max-x_min)/1000.
hy = (y_max-y_min)/1000.
xx, yy = np.meshgrid(np.arange(x_min, x_max, hx), np.arange(y_min, y_max, hy))

# Obtain labels for each point in mesh. Use last trained model.
Z = clusters.predict(np.c_[xx.ravel(), yy.ravel()])

# Find the centroids for KMeans or the cluster means for GMM

centroids = kmeans.cluster_centers_
print('*** K MEANS CENTROIDS ***')
print(centroids)

# TRANSFORM DATA BACK TO ORIGINAL SPACE FOR ANSWERING 7
print('*** CENTROIDS TRANSFERED TO ORIGINAL SPACE ***')
print(pca.inverse_transform(centroids))

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(1)
plt.clf()
plt.imshow(Z, interpolation='nearest',
           extent=(xx.min(), xx.max(), yy.min(), yy.max()),
           cmap=plt.cm.Paired,
           aspect='auto', origin='lower')

plt.plot(reduced_data[:, 0], reduced_data[:, 1], 'k.', markersize=2)
plt.scatter(centroids[:, 0], centroids[:, 1],
            marker='x', s=169, linewidths=3,
            color='w', zorder=10)
plt.title('Clustering on the seeds dataset (PCA-reduced data)\n'
          'Centroids are marked with white cross')
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.xticks(())
plt.yticks(())
plt.show()


I am working on the file "Clustering/Seeds Clustering.ipynb".
The first task is:
The explained variance is high for the first two dimensions, but drops significantly beginning with the third dimension. Thus, the first two components explain already 86.5 % of the variation in the data.
How many dimension to choose for the analysis really depends on the goal of the analysis. Even though PCA reduces the feature space (with all advantages that brings, such as faster computations) and makes interpreting the data easier for us by projecting them down to a lower dimension, it necessarily comes with a loss of information that may or may not be desired.
It the case at hand, assuming interpretation is the goal (creating seed clusters) and given the sharp drop of the explained variance after the second component, we would choose the first two dimensions for analysis.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(list(pca.explained_variance_ratio_),'-o')
plt.title('Explained variance ratio as function of PCA components')
plt.ylabel('Explained variance ratio')
plt.xlabel('Component')
plt.show()

features


I am working on the file "Clustering/Seeds/Seeds.ipynb".
The first task is:
PCA
Can you write Python code for it?

%matplotlib inline
#Scree plot

U, S, V = np.linalg.svd(X)
eigvals = S**2 / np.cumsum(S)[-1]

fig = plt.figure(figsize=(8,6))
sing_vals = np.arange(7) + 1
plt.plot(sing_vals, eigvals, 'ro-', linewidth=2)
plt.title('Scree Plot')
plt.xlabel('Principal Component')
plt.ylabel('Eigenvalue')

plt.show()

pca = PCA(n_components=2)
pca.fit(X)

X_pca = pca.transform(X)
plt.scatter(X_pca[:,0],X_pca[:,1], c=y, cmap=pylab.cm.cool)

pca_fit = KMeans(n_clusters=3).fit(X_pca)
pca_fit.labels_

#remake labels so that they properly matchup with the classes
pca_labels = pca_fit.labels_[:]
for index,val in enumerate(pca_labels):
    if val == 0:
        pca_labels[index] = 1
    elif val == 1:
        pca_labels[index] = 3
    else:
        pca_labels[index] = 2

labels

pca_conf_mat = np.zeros((3,3))

for i,val in enumerate(y):
    pca_conf_mat[val-1,pca_labels[i]-1] += 1

#true vs. predicted
print(pd.DataFrame(pca_conf_mat))


I am working on the file "Clustering/Seeds/Seeds.ipynb".
The first task is:
Reading Data
Can you write Python code for it?

#Names of all of the columns
names = [
       'area'
    ,  'perimeter'
    ,  'compactness'
    ,  'length'
    ,  'width'
    ,  'asymmetry'
    ,  'groove_length'
    ,  'class'
]

#Import dataset
data = pd.read_csv('seeds_dataset.txt', sep = '\t', header = None, names = names)

data.head(10)

data.shape


I am working on the file "Clustering/Seeds/Seeds.ipynb".
The first task is:
Scatter Plot Matrix
Can you write Python code for it?

# Visualize dataset with scatterplot matrix
%matplotlib inline

g = sns.PairGrid(data_clean, hue="class")
g.map_diag(plt.hist)
g.map_offdiag(plt.scatter)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
All of the 10 people are politicians, but about half of them have rather tenuous connections with Obama, other than the fact that they are politicians.

Francisco Barrio is a Mexican politician, and a former governor of Chihuahua.
Walter Mondale and Don Bonker are Democrats who made their career in late 1970s.
Wynn Normington Hugh-Jones is a former British diplomat and Liberal Party official.
Andy Anstett is a former politician in Manitoba, Canada.

Nearest neighbors with raw word counts got some things right, showing all politicians in the query result, but missed finer and important details.
For instance, let's find out why Francisco Barrio was considered a close neighbor of Obama.  To do this, let's look at the most frequently used words in each of Barack Obama and Francisco Barrio's pages:
Can you write Python code for it?

def top_words(name):
    """
    Get a table of the most frequent words in the given person's wikipedia page.
    """
    row = wiki[wiki['name'] == name]
    word_count_table = row[['word_count']].stack('word_count', new_column_name=['word','count'])
    return word_count_table.sort('count', ascending=False)

obama_words = top_words('Barack Obama')
obama_words

barrio_words = top_words('Francisco Barrio')
barrio_words


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
But one may wonder, is Biden's article that different from Obama's, more so than, say, Schiliro's? It turns out that, when we compute nearest neighbors using the Euclidean distances, we unwittingly favor short articles over long ones. Let us compute the length of each Wikipedia document, and examine the document lengths for the 100 nearest neighbors to Obama's page.
Can you write Python code for it?

def compute_length(row):
    return len(row['text'])

wiki['length'] = wiki.apply(compute_length)

nearest_neighbors_euclidean = model_tf_idf.query(wiki[wiki['name'] == 'Barack Obama'], label='name', k=100)
nearest_neighbors_euclidean = nearest_neighbors_euclidean.join(wiki[['name', 'length']], on={'reference_label':'name'})

nearest_neighbors_euclidean.sort('rank')


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
Checkpoint. Check your has_top_words function on two random articles:
Can you write Python code for it?

print 'Output from your function:', has_top_words(wiki[32]['word_count'])
print 'Correct output: True'
print 'Also check the length of unique_words. It should be 167'

print 'Output from your function:', has_top_words(wiki[33]['word_count'])
print 'Correct output: False'
print 'Also check the length of unique_words. It should be 188'


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
Extract word count vectors
As we have seen in Course 1, we can extract word count vectors using a GraphLab utility function.  We add this as a column in wiki.
Can you write Python code for it?

wiki['word_count'] = graphlab.text_analytics.count_words(wiki['text'])

wiki