I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Quiz Question. Among the words that appear in both Barack Obama and Francisco Barrio, take the 5 that appear most frequently in Obama. How many of the articles in the Wikipedia dataset contain all of those 5 words?
Hint:
* Refer to the previous paragraph for finding the words that appear in both articles. Sort the common words by their frequencies in Obama's article and take the largest five.
* Each word count vector is a Python dictionary. For each word count vector in SFrame, you'd have to check if the set of the 5 common words is a subset of the keys of the word count vector. Complete the function has_top_words to accomplish the task.
  - Convert the list of top 5 words into set using the syntax
set(common_words)
    where common_words is a Python list. See this link if you're curious about Python sets.
  - Extract the list of keys of the word count dictionary by calling the keys() method.
  - Convert the list of keys into a set as well.
  - Use issubset() method to check if all 5 words are among the keys.
* Now apply the has_top_words function on every row of the SFrame.
* Compute the sum of the result column to obtain the number of articles containing all the 5 top words.
Can you write Python code for it?

top_5_words = combined_words.sort('Obama', ascending=False)[0:5]['word']

common_words = set(top_5_words)

def has_top_words(word_count_vector):
    for word in common_words:
        if(word not in word_count_vector):
            return False
    return True  # YOUR CODE HERE

wiki['has_top_words'] = wiki['word_count'].apply(has_top_words)

# use has_top_words column to answer the quiz question
wiki['has_top_words'].sum() # YOUR CODE HERE


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
The first 10 words should say: Obama, law, democratic, Senate, presidential, president, policy, states, office, 2011.
Quiz Question. Among the words that appear in both Barack Obama and Phil Schiliro, take the 5 that have largest weights in Obama. How many of the articles in the Wikipedia dataset contain all of those 5 words?
Can you write Python code for it?

common_words = set(common_tf_idf['word'][0:5])  # YOUR CODE HERE

def has_top_words(word_count_vector):
    unique_words = set(word_count_vector.keys())
    return common_words <= unique_words   # YOUR CODE HERE

wiki['has_top_words'] = wiki['word_count'].apply(has_top_words)

# use has_top_words column to answer the quiz question
wiki['has_top_words'].sum() # YOUR CODE HERE


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
To see how these document lengths compare to the lengths of other documents in the corpus, let's make a histogram of the document lengths of Obama's 100 nearest neighbors and compare to a histogram of document lengths for all documents.
Can you write Python code for it?

plt.figure(figsize=(10.5,4.5))
plt.hist(wiki['length'], 50, color='k', edgecolor='None', histtype='stepfilled', normed=True,
         label='Entire Wikipedia', zorder=3, alpha=0.8)
plt.hist(nearest_neighbors_euclidean['length'], 50, color='r', edgecolor='None', histtype='stepfilled', normed=True,
         label='100 NNs of Obama (Euclidean)', zorder=10, alpha=0.8)
plt.axvline(x=wiki['length'][wiki['name'] == 'Barack Obama'][0], color='k', linestyle='--', linewidth=4,
           label='Length of Barack Obama', zorder=2)
plt.axvline(x=wiki['length'][wiki['name'] == 'Joe Biden'][0], color='g', linestyle='--', linewidth=4,
           label='Length of Joe Biden', zorder=1)
plt.axis([0, 1000, 0, 0.04])

plt.legend(loc='best', prop={'size':15})
plt.title('Distribution of document length')
plt.xlabel('# of words')
plt.ylabel('Percentage')
plt.rcParams.update({'font.size':16})
plt.tight_layout()


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Using the join operation we learned earlier, try your hands at computing the common words shared by Obama's and Schiliro's articles. Sort the common words by their TF-IDF weights in Obama's document.
Can you write Python code for it?

common_tf_idf = obama_tf_idf.join(schiliro_tf_idf, on='word').rename({'weight':'obama', 'weight.1':'schiliro'}).sort('obama', ascending=False)
common_tf_idf


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
All documents that obtain exactly this vector will be assigned to the same bin. We'd like to repeat the identical operation on all documents in the Wikipedia dataset and compute the corresponding bin indices. Again, we use matrix operations  so that no explicit loop is needed.
Can you write Python code for it?

corpus[0:2].dot(random_vectors) >= 0 # compute bit indices of first two documents

corpus.dot(random_vectors) >= 0 # compute bit indices of ALL documents


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Checkpoint.
Can you write Python code for it?

model = train_lsh(corpus, num_vector=16, seed=143)
table = model['table']
if   0 in table and table[0]   == [39583] and \
   143 in table and table[143] == [19693, 28277, 29776, 30399]:
    print 'Passed!'
else:
    print 'Check your code.'


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Checkpoint. Running the function with search_radius=0 should yield the list of documents belonging to the same bin as the query.
Can you write Python code for it?

obama_bin_index = model['bin_index_bits'][35817] # bin index of Barack Obama
candidate_set = search_nearby_bins(obama_bin_index, model['table'], search_radius=0)
if candidate_set == set([35817, 21426, 53937, 39426, 50261]):
    print 'Passed test'
else:
    print 'Check your code'
print 'List of documents in the same bin as Obama: 35817, 21426, 53937, 39426, 50261'


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Checkpoint: The following code block should return 'Check passed correctly', indicating that your matrix contains TF-IDF values for 59071 documents and 547979 unique words.  Otherwise, it will return Error.
Can you write Python code for it?

assert corpus.shape == (59071, 547979)
print 'Check passed correctly!'


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Compare the result with a former British diplomat, whose bin representation agrees with Obama's in only 8 out of 16 places.
Can you write Python code for it?

wiki[wiki['name']=='Wynn Normington Hugh-Jones']

print np.array(model['bin_index_bits'][22745], dtype=int) # list of 0/1's
print model['bin_indices'][22745] # integer format
model['bin_index_bits'][35817] == model['bin_index_bits'][22745]


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Extract TF-IDF matrix
We first use GraphLab Create to compute a TF-IDF representation for each document.
Can you write Python code for it?

wiki['tf_idf'] = graphlab.text_analytics.tf_idf(wiki['text'])
wiki


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
For this assignment, let us assign a unique ID to each document.
Can you write Python code for it?


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
How about the documents in the same bin as Barack Obama? Are they necessarily more similar to Obama than Biden?  Let's look at which documents are in the same bin as the Barack Obama article.
Can you write Python code for it?


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Let's try it out with Obama:
Can you write Python code for it?

query(corpus[35817,:], model, k=10, max_search_radius=3)


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Load in the Wikipedia dataset
Can you write Python code for it?


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Moral of the story. Similar data points will in general tend to fall into nearby bins, but that's all we can say about LSH. In a high-dimensional space such as text features, we often get unlucky with our selection of only a few random vectors such that dissimilar data points go into the same bin while similar data points fall into different bins. Given a query document, we must consider all documents in the nearby bins and sort them according to their actual distances from the query.
Query the LSH model
Let us first implement the logic for searching nearby neighbors, which goes like this:
1. Let L be the bit representation of the bin that contains the query documents.
2. Consider all documents in bin L.
3. Consider documents in the bins whose bit representation differs from L by 1 bit.
4. Consider documents in the bins whose bit representation differs from L by 2 bits.
...
To obtain candidate bins that differ from the query bin by some number of bits, we use itertools.combinations, which produces all possible subsets of a given list. See this documentation for details.
1. Decide on the search radius r. This will determine the number of different bits between the two vectors.
2. For each subset (n_1, n_2, ..., n_r) of the list [0, 1, 2, ..., num_vector-1], do the following:
   * Flip the bits (n_1, n_2, ..., n_r) of the query bin to produce a new bit vector.
   * Fetch the list of documents belonging to the bin indexed by the new bit vector.
   * Add those documents to the candidate set.
Each line of output from the following cell is a 3-tuple indicating where the candidate bin would differ from the query bin. For instance,
(0, 1, 3)
indicates that the candiate bin differs from the query bin in first, second, and fourth bits.
Can you write Python code for it?

from itertools import combinations

num_vector = 16
search_radius = 3

for diff in combinations(range(num_vector), search_radius):
    print diff


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Note. We will be using the model trained here in the following sections, unless otherwise indicated.
Inspect bins
Let us look at some documents and see which bins they fall into.
Can you write Python code for it?


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Quiz Question. What is the document id of Barack Obama's article? 35817
Quiz Question. Which bin contains Barack Obama's article? Enter its integer index. 50194
Can you write Python code for it?


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Recall from the previous assignment that Joe Biden was a close neighbor of Barack Obama.
Can you write Python code for it?


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Similarly, the second bit is computed as the sign of the dot product between the second random vector and the document vector.
Can you write Python code for it?

doc.dot(random_vectors[:, 1]) >= 0 # True if positive sign; False if negative sign


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Some observations:
* As we increase the search radius, we find more neighbors that are a smaller distance away.
* With increased search radius comes a greater number documents that have to be searched. Query time is higher as a consequence.
* With sufficiently high search radius, the results of LSH begin to resemble the results of brute-force search.
Quiz Question. What was the smallest search radius that yielded the correct nearest neighbor, namely Joe Biden?
Quiz Question. Suppose our goal was to produce 10 approximate nearest neighbors whose average distance from the query document is within 0.01 of the average for the true 10 nearest neighbors. For Barack Obama, the true 10 nearest neighbors are on average about 0.77. What was the smallest search radius for Barack Obama that produced an average distance of 0.78 or better?
Can you write Python code for it?


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
The conversion should take a few minutes to complete.
Can you write Python code for it?

start=time.time()
corpus, mapping = sframe_to_scipy(wiki['tf_idf'])
end=time.time()
print end-start


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
The following cell will run LSH with multiple search radii and compute the quality metrics for each run. Allow a few minutes to complete.
Can you write Python code for it?

max_radius = 17
precision = {i:[] for i in xrange(max_radius)}
average_distance  = {i:[] for i in xrange(max_radius)}
query_time  = {i:[] for i in xrange(max_radius)}

np.random.seed(0)
num_queries = 10
for i, ix in enumerate(np.random.choice(corpus.shape[0], num_queries, replace=False)):
    print('%s / %s' % (i, num_queries))
    ground_truth = set(brute_force_query(corpus[ix,:], corpus, k=25)['id'])
    # Get the set of 25 true nearest neighbors

    for r in xrange(1,max_radius):
        start = time.time()
        result, num_candidates = query(corpus[ix,:], model, k=10, max_search_radius=r)
        end = time.time()

        query_time[r].append(end-start)
        # precision = (# of neighbors both in result and ground_truth)/10.0
        precision[r].append(len(set(result['id']) & ground_truth)/10.0)
        average_distance[r].append(result['distance'][1:].mean())

plt.figure(figsize=(7,4.5))
plt.plot(range(1,17), [np.mean(average_distance[i]) for i in xrange(1,17)], linewidth=4, label='Average over 10 neighbors')
plt.xlabel('Search radius')
plt.ylabel('Cosine distance')
plt.legend(loc='best', prop={'size':15})
plt.rcParams.update({'font.size':16})
plt.tight_layout()

plt.figure(figsize=(7,4.5))
plt.plot(range(1,17), [np.mean(precision[i]) for i in xrange(1,17)], linewidth=4, label='Precison@10')
plt.xlabel('Search radius')
plt.ylabel('Precision')
plt.legend(loc='best', prop={'size':15})
plt.rcParams.update({'font.size':16})
plt.tight_layout()

plt.figure(figsize=(7,4.5))
plt.plot(range(1,17), [np.mean(query_time[i]) for i in xrange(1,17)], linewidth=4, label='Query time')
plt.xlabel('Search radius')
plt.ylabel('Query time (seconds)')
plt.legend(loc='best', prop={'size':15})
plt.rcParams.update({'font.size':16})
plt.tight_layout()


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
There are four other documents that belong to the same bin. Which documents are they?
Can you write Python code for it?

doc_ids = list(model['table'][model['bin_indices'][35817]])
doc_ids.remove(35817) # display documents other than Obama

docs = wiki.filter_by(values=doc_ids, column_name='id') # filter by id column
docs


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Train an LSH model
LSH performs an efficient neighbor search by randomly partitioning all reference data points into different bins. Today we will build a popular variant of LSH known as random binary projection, which approximates cosine distance. There are other variants we could use for other choices of distance metrics.
The first step is to generate a collection of random vectors from the standard Gaussian distribution.
Can you write Python code for it?

def generate_random_vectors(num_vector, dim):
    return np.random.randn(dim, num_vector)


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
We can compute all of the bin index bits at once as follows. Note the absence of the explicit for loop over the 16 vectors. Matrix operations let us batch dot-product computation in a highly efficent manner, unlike the for loop construction. Given the relative inefficiency of loops in Python, the advantage of matrix operations is even greater.
Can you write Python code for it?

doc.dot(random_vectors) >= 0 # should return an array of 16 True/False bits

np.array(doc.dot(random_vectors) >= 0, dtype=int) # display index bits in 0/1's


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
We have shown that we have a working LSH implementation!
Experimenting with your LSH implementation
In the following sections we have implemented a few experiments so that you can gain intuition for how your LSH implementation behaves in different situations. This will help you understand the effect of searching nearby bins and the performance of LSH versus computing nearest neighbors using a brute force search.
Effect of nearby bin search
How does nearby bin search affect the outcome of LSH? There are three variables that are affected by the search radius:
* Number of candidate documents considered
* Query time
* Distance of approximate neighbors from the query
Let us run LSH multiple times, each with different radii for nearby bin search. We will measure the three variables as discussed above.
Can you write Python code for it?

wiki[wiki['name']=='Barack Obama']

num_candidates_history = []
query_time_history = []
max_distance_from_query_history = []
min_distance_from_query_history = []
average_distance_from_query_history = []

for max_search_radius in xrange(17):
    start=time.time()
    result, num_candidates = query(corpus[35817,:], model, k=10,
                                   max_search_radius=max_search_radius)
    end=time.time()
    query_time = end-start

    print 'Radius:', max_search_radius
    print result.join(wiki[['id', 'name']], on='id').sort('distance')

    average_distance_from_query = result['distance'][1:].mean()
    max_distance_from_query = result['distance'][1:].max()
    min_distance_from_query = result['distance'][1:].min()

    num_candidates_history.append(num_candidates)
    query_time_history.append(query_time)
    average_distance_from_query_history.append(average_distance_from_query)
    max_distance_from_query_history.append(max_distance_from_query)
    min_distance_from_query_history.append(min_distance_from_query)


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
We now generate random vectors of the same dimensionality as our vocubulary size (547979).  Each vector can be used to compute one bit in the bin encoding.  We generate 16 vectors, leading to a 16-bit encoding of the bin index for each document.
Can you write Python code for it?

# Generate 16 random vectors of dimension 547979
np.random.seed(0)
random_vectors = generate_random_vectors(num_vector=16, dim=547979)
random_vectors.shape


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
We're almost done! To make it convenient to refer to individual bins, we convert each binary bin index into a single integer:
Bin index                      integer
[0,0,0,0,0,0,0,0,0,0,0,0]   => 0
[0,0,0,0,0,0,0,0,0,0,0,1]   => 1
[0,0,0,0,0,0,0,0,0,0,1,0]   => 2
[0,0,0,0,0,0,0,0,0,0,1,1]   => 3
...
[1,1,1,1,1,1,1,1,1,1,0,0]   => 65532
[1,1,1,1,1,1,1,1,1,1,0,1]   => 65533
[1,1,1,1,1,1,1,1,1,1,1,0]   => 65534
[1,1,1,1,1,1,1,1,1,1,1,1]   => 65535 (= 2^16-1)
By the rules of binary number representation, we just need to compute the dot product between the document vector and the vector consisting of powers of 2:
Can you write Python code for it?

doc = corpus[0, :]  # first document
index_bits = (doc.dot(random_vectors) >= 0)
powers_of_two = (1 << np.arange(15, -1, -1))
print index_bits
print powers_of_two
print index_bits.dot(powers_of_two)


I am working on the file "Clustering_1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
With this output in mind, implement the logic for nearby bin search:
Can you write Python code for it?

for x in combinations([1,2,3], 0):
    print 1

def search_nearby_bins(query_bin_bits, table, search_radius=2, initial_candidates=set()):
    """
    For a given query vector and trained LSH model, return all candidate neighbors for
    the query among all bins within the given search radius.

    Example usage
    -------------
    >>> model = train_lsh(corpus, num_vector=16, seed=143)
    >>> q = model['bin_index_bits'][0]  # vector for the first document

    >>> candidates = search_nearby_bins(q, model['table'])
    """
    num_vector = len(query_bin_bits)
    powers_of_two = 1 << np.arange(num_vector-1, -1, -1)

    # Allow the user to provide an initial set of candidates.
    candidate_set = copy(initial_candidates)

    for different_bits in combinations(range(num_vector), search_radius):
        # Flip the bits (n_1,n_2,...,n_r) of the query bin to produce a new bit vector.
        ## Hint: you can iterate over a tuple like a list
        alternate_bits = copy(query_bin_bits)
        for i in different_bits:
            alternate_bits[i] = 1 - alternate_bits[i] # YOUR CODE HERE

        # Convert the new bit vector to an integer index
        nearby_bin = alternate_bits.dot(powers_of_two)

        # Fetch the list of documents belonging to the bin indexed by the new bit vector.
        # Then add those documents to candidate_set
        # Make sure that the bin exists in the table!
        # Hint: update() method for sets lets you add an entire list to the set
        if nearby_bin in table:
            candidate_set |= set(table[nearby_bin]) # YOUR CODE HERE: Update candidate_set with the documents in this bin.

    return candidate_set


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
A few things to notice from the box plot:
* Random initialization results in a worse clustering than k-means++ on average.
* The best result of k-means++ is better than the best result of random initialization.
In general, you should run k-means at least a few times with different initializations and then return the run resulting in the lowest heterogeneity. Let us write a function that runs k-means multiple times and picks the best run that minimizes heterogeneity. The function accepts an optional list of seed values to be used for the multiple runs; if no such list is provided, the current UTC time is used as seed values.
Can you write Python code for it?

def kmeans_multiple_runs(data, k, maxiter, num_runs, seed_list=None, verbose=False):
    heterogeneity = {}

    min_heterogeneity_achieved = float('inf')
    best_seed = None
    final_centroids = None
    final_cluster_assignment = None

    for i in xrange(num_runs):

        # Use UTC time if no seeds are provided
        if seed_list is not None:
            seed = seed_list[i]
            np.random.seed(seed)
        else:
            seed = int(time.time())
            np.random.seed(seed)

        # Use k-means++ initialization
        # YOUR CODE HERE
        initial_centroids = smart_initialize(data, k, seed)

        # Run k-means
        # YOUR CODE HERE
        centroids, cluster_assignment = kmeans(data, k, initial_centroids, maxiter)

        # To save time, compute heterogeneity only once in the end
        # YOUR CODE HERE
        heterogeneity[seed] = compute_heterogeneity(data, k, centroids, cluster_assignment)

        if verbose:
            print('seed={0:06d}, heterogeneity={1:.5f}'.format(seed, heterogeneity[seed]))
            sys.stdout.flush()

        # if current measurement of heterogeneity is lower than previously seen,
        # update the minimum record of heterogeneity.
        if heterogeneity[seed] < min_heterogeneity_achieved:
            min_heterogeneity_achieved = heterogeneity[seed]
            best_seed = seed
            final_centroids = centroids
            final_cluster_assignment = cluster_assignment

    # Return the centroids and cluster assignments that minimize heterogeneity.
    return final_centroids, final_cluster_assignment


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
After initialization, the k-means algorithm iterates between the following two steps:
1. Assign each data point to the closest centroid.
$$
z_i \gets \mathrm{argmin}j \|\mu_j - \mathbf{x}_i\|^2
$$
2. Revise centroids as the mean of the assigned data points.
$$
\mu_j \gets \frac{1}{n_j}\sum{i:z_i=j} \mathbf{x}_i
$$
In pseudocode, we iteratively do the following:
cluster_assignment = assign_clusters(data, centroids)
centroids = revise_centroids(data, k, cluster_assignment)
Assigning clusters
How do we implement Step 1 of the main k-means loop above? First import pairwise_distances function from scikit-learn, which calculates Euclidean distances between rows of given arrays. See this documentation for more information.
For the sake of demonstration, let's look at documents 100 through 102 as query documents and compute the distances between each of these documents and every other document in the corpus. In the k-means algorithm, we will have to compute pairwise distances between the set of centroids and the set of documents.
Can you write Python code for it?

from sklearn.metrics import pairwise_distances

# Get the TF-IDF vectors for documents 100 through 102.
queries = tf_idf[100:102,:]

# Compute pairwise distances from every data point to each query vector.
dist = pairwise_distances(tf_idf, queries, metric='euclidean')

print dist


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Again, this makes sense since these values are each closer to [0.5 0.5 0] than to [0 -0.5 0].
Given all the data points in a cluster, it only remains to compute the mean. Use np.mean(). By default, the function averages all elements in a 2D array. To compute row-wise or column-wise means, add the axis argument. See the linked documentation for details.
Use this function to average the data points in cluster 0:
Can you write Python code for it?


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Assessing convergence
How can we tell if the k-means algorithm is converging? We can look at the cluster assignments and see if they stabilize over time. In fact, we'll be running the algorithm until the cluster assignments stop changing at all. To be extra safe, and to assess the clustering performance, we'll be looking at an additional criteria: the sum of all squared distances between data points and centroids. This is defined as
$$
J(\mathcal{Z},\mu) = \sum_{j=1}^k \sum_{i:z_i = j} \|\mathbf{x}_i - \mu_j\|^2.
$$
The smaller the distances, the more homogeneous the clusters are. In other words, we'd like to have "tight" clusters.
Can you write Python code for it?

def compute_heterogeneity(data, k, centroids, cluster_assignment):

    heterogeneity = 0.0
    for i in xrange(k):

        # Select all data points that belong to cluster i. Fill in the blank (RHS only)
        member_data_points = data[cluster_assignment==i, :]

        if member_data_points.shape[0] > 0: # check if i-th cluster is non-empty
            # Compute distances from centroid to data points (RHS only)
            distances = pairwise_distances(member_data_points, [centroids[i]], metric='euclidean')
            squared_distances = distances**2
            heterogeneity += np.sum(squared_distances)

    return heterogeneity


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Beware of local maxima
One weakness of k-means is that it tends to get stuck in a local minimum. To see this, let us run k-means multiple times, with different initial centroids created using different random seeds.
Note: Again, in practice, you should set different seeds for every run. We give you a list of seeds for this assignment so that everyone gets the same answer.
This may take several minutes to run.
Can you write Python code for it?

k = 10
heterogeneity = {}
import time
start = time.time()
for seed in [0, 20000, 40000, 60000, 80000, 100000, 120000]:
    initial_centroids = get_initial_centroids(tf_idf, k, seed)
    centroids, cluster_assignment = kmeans(tf_idf, k, initial_centroids, maxiter=400,
                                           record_heterogeneity=None, verbose=False)
    # To save time, compute heterogeneity only once in the end
    heterogeneity[seed] = compute_heterogeneity(tf_idf, k, centroids, cluster_assignment)
    print('seed={0:06d}, heterogeneity={1:.5f}'.format(seed, heterogeneity[seed]))
    cluster_size = np.bincount(cluster_assignment)
    biggest_cluster = np.argmax(cluster_size)
    print 'Biggest Cluster: {}, Size: {}'.format(biggest_cluster, cluster_size[biggest_cluster])
    sys.stdout.flush()
end = time.time()
print(end-start)

'{} {}'.format(1, 2)


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Checkpoint. For the last time, let us check if Step 1 was implemented correctly. With rows 0, 2, 4, and 6 of tf_idf as an initial set of centroids, we assign cluster labels to rows 0, 10, 20, ..., and 90 of tf_idf. The resulting cluster labels should be [0, 1, 1, 0, 0, 2, 0, 2, 2, 1].
Can you write Python code for it?

if np.allclose(assign_clusters(tf_idf[0:100:10], tf_idf[0:8:2]), np.array([0, 1, 1, 0, 0, 2, 0, 2, 2, 1])):
    print('Pass')
else:
    print('Check your code again.')


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Checkpoint: Let's put these steps together.  First, initialize three centroids with the first 3 rows of tf_idf. Then, compute distances from each of the centroids to all data points in tf_idf. Finally, use these distance calculations to compute cluster assignments and assign them to cluster_assignment.
Can you write Python code for it?

# Students should write code here
centroids = tf_idf[0:3, :]
distances = pairwise_distances(centroids, tf_idf, metric='euclidean')
cluster_assignment = np.argmin(distances, axis=0)
cluster_assignment

if len(cluster_assignment)==59071 and \
   np.array_equal(np.bincount(cluster_assignment), np.array([23061, 10086, 25924])):
    print('Pass') # count number of data points for each cluster
else:
    print('Check your code again.')


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Checkpoint: Next, given the pairwise distances, we take the minimum of the distances for each data point. Fittingly, NumPy provides an argmin function. See this documentation for details.
Read the documentation and write code to produce a 1D array whose i-th entry indicates the centroid that is the closest to the i-th data point. Use the list of distances from the previous checkpoint and save them as distances. The value 0 indicates closeness to the first centroid, 1 indicates closeness to the second centroid, and so forth. Save this array as closest_cluster.
Hint: the resulting array should be as long as the number of data points.
Can you write Python code for it?

# Students should write code here
closest_cluster = np.argmin(distances, axis=1)
closest_cluster

'''Test cell'''
reference = [list(row).index(min(row)) for row in distances]
if np.allclose(closest_cluster, reference):
    print('Pass')
else:
    print('Check your code again')


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Clusters 0 and 2 appear to be still mixed, but others are quite consistent in content.
* Cluster 0: artists, poets, writers, environmentalists
* Cluster 1: film directors
* Cluster 2: female figures from various fields
* Cluster 3: politicians
* Cluster 4: track and field athletes
* Cluster 5: composers, songwriters, singers, music producers
* Cluster 6: soccer (football) players
* Cluster 7: baseball players
* Cluster 8: professors, researchers, scholars
* Cluster 9: lawyers, judges, legal scholars
Clusters are now more pure, but some are qualitatively "bigger" than others. For instance, the category of scholars is more general than the category of baseball players. Increasing the number of clusters may split larger clusters. Another way to look at the size of the clusters is to count the number of articles in each cluster.
Can you write Python code for it?


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
How to choose K
Since we are measuring the tightness of the clusters, a higher value of K reduces the possible heterogeneity metric by definition.  For example, if we have N data points and set K=N clusters, then we could have 0 cluster heterogeneity by setting the N centroids equal to the values of the N data points. (Note: Not all runs for larger K will result in lower heterogeneity than a single run with smaller K due to local optima.)  Let's explore this general trend for ourselves by performing the following analysis.
Use the kmeans_multiple_runs function to run k-means with five different values of K.  For each K, use k-means++ and multiple runs to pick the best solution.  In what follows, we consider K=2,10,25,50,100 and 7 restarts for each setting.
IMPORTANT: The code block below will take about one hour to finish. We highly suggest that you use the arrays that we have computed for you.
Side note: In practice, a good implementation of k-means would utilize parallelism to run multiple runs of k-means at once. For an example, see scikit-learn's KMeans.
Can you write Python code for it?

#def plot_k_vs_heterogeneity(k_values, heterogeneity_values):
#    plt.figure(figsize=(7,4))
#    plt.plot(k_values, heterogeneity_values, linewidth=4)
#    plt.xlabel('K')
#    plt.ylabel('Heterogeneity')
#    plt.title('K vs. Heterogeneity')
#    plt.rcParams.update({'font.size': 16})
#    plt.tight_layout()

#start = time.time()
#centroids = {}
#cluster_assignment = {}
#heterogeneity_values = []
#k_list = [2, 10, 25, 50, 100]
#seed_list = [0, 20000, 40000, 60000, 80000, 100000, 120000]

#for k in k_list:
#    heterogeneity = []
#    centroids[k], cluster_assignment[k] = kmeans_multiple_runs(tf_idf, k, maxiter=400,
#                                                               num_runs=len(seed_list),
#                                                               seed_list=seed_list,
#                                                               verbose=True)
#    score = compute_heterogeneity(tf_idf, k, centroids[k], cluster_assignment[k])
#    heterogeneity_values.append(score)

#plot_k_vs_heterogeneity(k_list, heterogeneity_values)

#end = time.time()
#print(end-start)


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Implement k-means
Let us implement the k-means algorithm. First, we choose an initial set of centroids. A common practice is to choose randomly from the data points.
Note: We specify a seed here, so that everyone gets the same answer. In practice, we highly recommend to use different seeds every time (for instance, by using the current timestamp).
Can you write Python code for it?

def get_initial_centroids(data, k, seed=None):
    '''Randomly choose k data points as initial centroids'''
    if seed is not None: # useful for obtaining consistent results
        np.random.seed(seed)
    n = data.shape[0] # number of data points

    # Pick K indices from range [0, N).
    rand_indices = np.random.randint(0, n, k)

    # Keep centroids as dense format, as many entries will be nonzero due to averaging.
    # As long as at least one document in a cluster contains a word,
    # it will carry a nonzero weight in the TF-IDF vector of the centroid.
    centroids = data[rand_indices,:].toarray()

    return centroids


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
In lieu of indices, we can put in the list of Booleans to pick and choose rows. Only the rows that correspond to a True entry will be retained.
First, let's look at the data points (i.e., their values) assigned to cluster 1:
Can you write Python code for it?


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
In the above plot we show that heterogeneity goes down as we increase the number of clusters. Does this mean we should always favor a higher K? Not at all! As we will see in the following section, setting K too high may end up separating data points that are actually pretty alike. At the extreme, we can set individual data points to be their own clusters (K=N) and achieve zero heterogeneity, but separating each data point into its own cluster is hardly a desirable outcome. In the following section, we will learn how to detect a K set "too large".
Visualize clusters of documents
Let's start visualizing some clustering results to see if we think the clustering makes sense.  We can use such visualizations to help us assess whether we have set K too large or too small for a given application.  Following the theme of this course, we will judge whether the clustering makes sense in the context of document analysis.
What are we looking for in a good clustering of documents?
* Documents in the same cluster should be similar.
* Documents from different clusters should be less similar.
So a bad clustering exhibits either of two symptoms:
* Documents in a cluster have mixed content.
* Documents with similar content are divided up and put into different clusters.
To help visualize the clustering, we do the following:
* Fetch nearest neighbors of each centroid from the set of documents assigned to that cluster. We will consider these documents as being representative of the cluster.
* Print titles and first sentences of those nearest neighbors.
* Print top 5 words that have highest tf-idf weights in each centroid.
Can you write Python code for it?

def visualize_document_clusters(wiki, tf_idf, centroids, cluster_assignment, k, map_index_to_word, display_content=True):
    '''wiki: original dataframe
       tf_idf: data matrix, sparse matrix format
       map_index_to_word: SFrame specifying the mapping betweeen words and column indices
       display_content: if True, display 8 nearest neighbors of each centroid'''

    print('==========================================================')

    # Visualize each cluster c
    for c in xrange(k):
        # Cluster heading
        print('Cluster {0:d}    '.format(c)),
        # Print top 5 words with largest TF-IDF weights in the cluster
        idx = centroids[c].argsort()[::-1]
        for i in xrange(5): # Print each word along with the TF-IDF weight
            print('{0:s}:{1:.3f}'.format(map_index_to_word['category'][idx[i]], centroids[c,idx[i]])),
        print('')

        if display_content:
            # Compute distances from the centroid to all data points in the cluster,
            # and compute nearest neighbors of the centroids within the cluster.
            distances = pairwise_distances(tf_idf, centroids[c].reshape(1, -1), metric='euclidean').flatten()
            distances[cluster_assignment!=c] = float('inf') # remove non-members from consideration
            nearest_neighbors = distances.argsort()
            # For 8 nearest neighbors, print the title as well as first 180 characters of text.
            # Wrap the text at 80-character mark.
            for i in xrange(8):
                text = ' '.join(wiki[nearest_neighbors[i]]['text'].split(None, 25)[0:25])
                print('\n* {0:50s} {1:.5f}\n  {2:s}\n  {3:s}'.format(wiki[nearest_neighbors[i]]['name'],
                    distances[nearest_neighbors[i]], text[:90], text[90:180] if len(text) > 90 else ''))
        print('==========================================================')


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Let's assign these data points to the closest centroid.
Can you write Python code for it?

cluster_assignment = assign_clusters(data, centroids)
print cluster_assignment


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Let's now rerun k-means with 10 clusters using the same set of seeds, but always using k-means++ to initialize the algorithm.
This may take several minutes to run.
Can you write Python code for it?

k = 10
heterogeneity_smart = {}
start = time.time()
for seed in [0, 20000, 40000, 60000, 80000, 100000, 120000]:
    initial_centroids = smart_initialize(tf_idf, k, seed)
    centroids, cluster_assignment = kmeans(tf_idf, k, initial_centroids, maxiter=400,
                                           record_heterogeneity=None, verbose=False)
    # To save time, compute heterogeneity only once in the end
    heterogeneity_smart[seed] = compute_heterogeneity(tf_idf, k, centroids, cluster_assignment)
    print('seed={0:06d}, heterogeneity={1:.5f}'.format(seed, heterogeneity_smart[seed]))
    sys.stdout.flush()
end = time.time()
print(end-start)


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Likewise for cluster 0:
Can you write Python code for it?


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Looking at the representative examples and top words, we classify each cluster as follows. Notice the bolded items, which indicate the appearance of a new theme.
* Cluster 0: composers, songwriters, singers, music producers
* Cluster 1: poets
* Cluster 2: rugby players
* Cluster 3: baseball players
* Cluster 4: government officials
* Cluster 5: football players
* Cluster 6: radio hosts
* Cluster 7: actors, TV directors
* Cluster 8: professors, researchers, scholars
* Cluster 9: lawyers, judges, legal scholars
* Cluster 10: track and field athletes
* Cluster 11: (mixed; no clear theme)
* Cluster 12: car racers
* Cluster 13: priests, bishops, church leaders
* Cluster 14: painters, sculptors, artists
* Cluster 15: novelists
* Cluster 16: American football players
* Cluster 17: golfers
* Cluster 18: American politicians
* Cluster 19: basketball players
* Cluster 20: generals of U.S. Air Force
* Cluster 21: politicians
* Cluster 22: female figures of various fields
* Cluster 23: film directors
* Cluster 24: music directors, composers, conductors
Indeed, increasing K achieved the desired effect of breaking up large clusters.  Depending on the application, this may or may not be preferable to the K=10 analysis.
Let's take it to the extreme and set K=100. We have a suspicion that this value is too large. Let us look at the top words from each cluster:
Can you write Python code for it?

k=100
visualize_document_clusters(wiki, tf_idf, centroids[k](), cluster_assignment[k](), k,
                            map_index_to_word, display_content=False)
# turn off text for brevity -- turn it on if you are curious ;)


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
More formally, dist[i,j] is assigned the distance between the ith row of X (i.e., X[i,:]) and the jth row of Y (i.e., Y[j,:]).
Checkpoint: For a moment, suppose that we initialize three centroids with the first 3 rows of tf_idf. Write code to compute distances from each of the centroids to all data points in tf_idf. Then find the distance between row 430 of tf_idf and the second centroid and save it to dist.
Can you write Python code for it?

# Students should write code here
centroids = tf_idf[0:3, :]
distances = pairwise_distances(tf_idf, centroids, metric = 'euclidean')
dist = distances[430, 1]

'''Test cell'''
if np.allclose(dist, pairwise_distances(tf_idf[430,:], tf_idf[1,:])):
    print('Pass')
else:
    print('Check your code again')


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Now we are ready to fill in the blanks in this function:
Can you write Python code for it?

def assign_clusters(data, centroids):

    # Compute distances between each data point and the set of centroids:
    # Fill in the blank (RHS only)
    distances_from_centroids = pairwise_distances(centroids, data)

    # Compute cluster assignments for each data point:
    # Fill in the blank (RHS only)
    cluster_assignment = np.argmin(distances_from_centroids, axis=0)

    return cluster_assignment


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Plotting convergence metric
We can use the above function to plot the convergence metric across iterations.
Can you write Python code for it?

def plot_heterogeneity(heterogeneity, k):
    plt.figure(figsize=(7,4))
    plt.plot(heterogeneity, linewidth=4)
    plt.xlabel('# Iterations')
    plt.ylabel('Heterogeneity')
    plt.title('Heterogeneity of clustering over time, K={0:d}'.format(k))
    plt.rcParams.update({'font.size': 16})
    plt.tight_layout()


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Quiz Question. (True/False) The clustering objective (heterogeneity) is non-increasing for this example.
Quiz Question. Let's step back from this particular example. If the clustering objective (heterogeneity) would ever increase when running k-means, that would indicate: (choose one)

k-means algorithm got stuck in a bad local minimum
There is a bug in the k-means code
All data points consist of exact duplicates
Nothing is wrong. The objective should generally go down sooner or later.

Quiz Question. Which of the cluster contains the greatest number of data points in the end? Hint: Use np.bincount() to count occurrences of each cluster label.
 1. Cluster #0
 2. Cluster #1
 3. Cluster #2
Can you write Python code for it?


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
Quiz Question. Which of the 10 clusters above contains the greatest number of articles?

Cluster 0: artists, poets, writers, environmentalists
Cluster 4: track and field athletes
Cluster 5: composers, songwriters, singers, music producers
Cluster 7: baseball players
Cluster 9: lawyers, judges, legal scholars

Quiz Question. Which of the 10 clusters contains the least number of articles?

Cluster 1: film directors
Cluster 3: politicians
Cluster 6: soccer (football) players
Cluster 7: baseball players
Cluster 9: lawyers, judges, legal scholars

There appears to be at least some connection between the topical consistency of a cluster and the number of its member data points.
Let us visualize the case for K=25. For the sake of brevity, we do not print the content of documents. It turns out that the top words with highest TF-IDF weights in each cluster are representative of the cluster.
Can you write Python code for it?

visualize_document_clusters(wiki, tf_idf, centroids[25](), cluster_assignment[25](), 25,
                            map_index_to_word, display_content=False) # turn off text for brevity


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
The above matrix contains a TF-IDF score for each of the 59071 pages in the data set and each of the 547979 unique words.
Normalize all vectors
As discussed in the previous assignment, Euclidean distance can be a poor metric of similarity between documents, as it unfairly penalizes long articles. For a reasonable assessment of similarity, we should disregard the length information and use length-agnostic metrics, such as cosine distance.
The k-means algorithm does not directly work with cosine distance, so we take an alternative route to remove length information: we normalize all vectors to be unit length. It turns out that Euclidean distance closely mimics cosine distance when all vectors are unit length. In particular, the squared Euclidean distance between any two vectors of length one is directly proportional to their cosine distance.
We can prove this as follows. Let $\mathbf{x}$ and $\mathbf{y}$ be normalized vectors, i.e. unit vectors, so that $\|\mathbf{x}\|=\|\mathbf{y}\|=1$. Write the squared Euclidean distance as the dot product of $(\mathbf{x} - \mathbf{y})$ to itself:
\begin{align}
\|\mathbf{x} - \mathbf{y}\|^2 &= (\mathbf{x} - \mathbf{y})^T(\mathbf{x} - \mathbf{y})\
                              &= (\mathbf{x}^T \mathbf{x}) - 2(\mathbf{x}^T \mathbf{y}) + (\mathbf{y}^T \mathbf{y})\
                              &= \|\mathbf{x}\|^2 - 2(\mathbf{x}^T \mathbf{y}) + \|\mathbf{y}\|^2\
                              &= 2 - 2(\mathbf{x}^T \mathbf{y})\
                              &= 2(1 - (\mathbf{x}^T \mathbf{y}))\
                              &= 2\left(1 - \frac{\mathbf{x}^T \mathbf{y}}{\|\mathbf{x}\|\|\mathbf{y}\|}\right)\
                              &= 2\left[\text{cosine distance}\right]
\end{align}
This tells us that two unit vectors that are close in Euclidean distance are also close in cosine distance. Thus, the k-means algorithm (which naturally uses Euclidean distances) on normalized vectors will produce the same results as clustering using cosine distance as a distance metric.
We import...
Can you write Python code for it?

from sklearn.preprocessing import normalize
tf_idf = normalize(tf_idf)


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
The expression cluster_assignment==1 gives a list of Booleans that says whether each data point is assigned to cluster 1 or not:
Can you write Python code for it?


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
This makes sense since [0 0 0] is closer to [0 -0.5 0] than to [0.5 0.5 0].
Now let's look at the data points assigned to cluster 0:
Can you write Python code for it?


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
To use the pre-computed NumPy arrays, first download kmeans-arrays.npz as mentioned in the reading for this assignment and load them with the following code.  Make sure the downloaded file is in the same directory as this notebook.
Can you write Python code for it?

def plot_k_vs_heterogeneity(k_values, heterogeneity_values):
    plt.figure(figsize=(7,4))
    plt.plot(k_values, heterogeneity_values, linewidth=4)
    plt.xlabel('K')
    plt.ylabel('Heterogeneity')
    plt.title('K vs. Heterogeneity')
    plt.rcParams.update({'font.size': 16})
    plt.tight_layout()

filename = 'kmeans-arrays.npz'

heterogeneity_values = []
k_list = [2, 10, 25, 50, 100]

if os.path.exists(filename):
    arrays = np.load(filename)
    centroids = {}
    cluster_assignment = {}
    for k in k_list:
        print k
        sys.stdout.flush()
        '''To save memory space, do not load the arrays from the file right away. We use
           a technique known as lazy evaluation, where some expressions are not evaluated
           until later. Any expression appearing inside a lambda function doesn't get
           evaluated until the function is called.
           Lazy evaluation is extremely important in memory-constrained setting, such as
           an Amazon EC2 t2.micro instance.'''
        centroids[k] = lambda k=k: arrays['centroids_{0:d}'.format(k)]
        cluster_assignment[k] = lambda k=k: arrays['cluster_assignment_{0:d}'.format(k)]
        score = compute_heterogeneity(tf_idf, k, centroids[k](), cluster_assignment[k]())
        heterogeneity_values.append(score)

    plot_k_vs_heterogeneity(k_list, heterogeneity_values)

else:
    print('File not found. Skipping.')


I am working on the file "Clustering_2_kmeans-with-text-data_blank.ipynb".
The first task is:
We are now ready to complete this function:
Can you write Python code for it?

def revise_centroids(data, k, cluster_assignment):
    new_centroids = []
    for i in xrange(k):
        # Select all data points that belong to cluster i. Fill in the blank (RHS only)
        member_data_points = data[cluster_assignment == i]
        # Compute the mean of the data points. Fill in the blank (RHS only)
        centroid = member_data_points.mean(axis=0)

        # Convert numpy.matrix type to numpy.ndarray type
        centroid = centroid.A1
        new_centroids.append(centroid)
    new_centroids = np.array(new_centroids)

    return new_centroids


I am working on the file "Clustering_4_em-with-text-data_blank.ipynb".
The first task is:
EM in high dimensions
EM for high-dimensional data requires some special treatment:
 * E step and M step must be vectorized as much as possible, as explicit loops are dreadfully slow in Python.
 * All operations must be cast in terms of sparse matrix operations, to take advantage of computational savings enabled by sparsity of data.
 * Initially, some words may be entirely absent from a cluster, causing the M step to produce zero mean and variance for those words.  This means any data point with one of those words will have 0 probability of being assigned to that cluster since the cluster allows for no variability (0 variance) around that count being 0 (0 mean). Since there is a small chance for those words to later appear in the cluster, we instead assign a small positive variance (~1e-10). Doing so also prevents numerical overflow.
We provide the complete implementation for you in the file em_utilities.py. For those who are interested, you can read through the code to see how the sparse matrix implementation differs from the previous assignment.
You are expected to answer some quiz questions using the results of clustering.
Initializing mean parameters using k-means
Recall from the lectures that EM for Gaussian mixtures is very sensitive to the choice of initial means. With a bad initial set of means, EM may produce clusters that span a large area and are mostly overlapping. To eliminate such bad outcomes, we first produce a suitable set of initial means by using the cluster centers from running k-means.  That is, we first run k-means and then take the final set of means from the converged solution as the initial means in our EM algorithm.
Can you write Python code for it?

from sklearn.cluster import KMeans

np.random.seed(5)
num_clusters = 25

# Use scikit-learn's k-means to simplify workflow
kmeans_model = KMeans(n_clusters=num_clusters, n_init=5, max_iter=400, random_state=1, n_jobs=-1)
kmeans_model.fit(tf_idf)
centroids, cluster_assignment = kmeans_model.cluster_centers_, kmeans_model.labels_

means = [centroid for centroid in centroids]


I am working on the file "Clustering_4_em-with-text-data_blank.ipynb".
The first task is:
Initializing cluster weights
We will initialize each cluster weight to be the proportion of documents assigned to that cluster by k-means above.
Can you write Python code for it?

num_docs = tf_idf.shape[0]
weights = []
sum = 0
for i in xrange(num_clusters):
    # Compute the number of data points assigned to cluster i:
    num_assigned = (cluster_assignment == i).sum() # YOUR CODE HERE
    sum += num_assigned
    # print sum
    w = float(num_assigned) / num_docs
    weights.append(w)


I am working on the file "Clustering_4_em-with-text-data_blank.ipynb".
The first task is:
Initializing covariances
To initialize our covariance parameters, we compute $\hat{\sigma}{k, j}^2 = \sum{i=1}^{N}(x_{i,j} - \hat{\mu}_{k, j})^2$ for each feature $j$.  For features with really tiny variances, we assign 1e-8 instead to prevent numerical instability. We do this computation in a vectorized fashion in the following code block.
Can you write Python code for it?

covs = []
for i in xrange(num_clusters):
    member_rows = tf_idf[cluster_assignment==i]
    cov = (member_rows.power(2) - 2*member_rows.dot(diag(means[i]))).sum(axis=0).A1 / member_rows.shape[0] \
          + means[i]**2
    cov[cov < 1e-8] = 1e-8
    covs.append(cov)


I am working on the file "Clustering_4_em-with-text-data_blank.ipynb".
The first task is:
Load Wikipedia data and extract TF-IDF features
Load Wikipedia data and transform each of the first 5000 document into a TF-IDF representation.
Can you write Python code for it?

wiki = graphlab.SFrame('people_wiki.gl/').head(5000)
wiki['tf_idf'] = graphlab.text_analytics.tf_idf(wiki['text'])


I am working on the file "Clustering_4_em-with-text-data_blank.ipynb".
The first task is:
Quiz Question: For the above model, out_random_init, use the visualize_EM_clusters method you created above. Are the clusters more or less interpretable than the ones found after initializing using k-means?
Can you write Python code for it?

# YOUR CODE HERE. Use visualize_EM_clusters, which will require you to pass in tf_idf and map_index_to_word.
visualize_EM_clusters(tf_idf, out_random_init['means'], out_random_init['covs'], map_index_to_word)


I am working on the file "Clustering_4_em-with-text-data_blank.ipynb".
The first task is:
Quiz Question: Is the final loglikelihood larger or smaller than the final loglikelihood we obtained above when initializing EM with the results from running k-means?
Can you write Python code for it?


I am working on the file "Clustering_4_em-with-text-data_blank.ipynb".
The first task is:
Quiz Question: Try fitting EM with the random initial parameters you created above. (Use cov_smoothing=1e-5.) Store the result to out_random_init. What is the final loglikelihood that the algorithm converges to?
Can you write Python code for it?

out_random_init = EM_for_high_dimension(tf_idf, random_means, random_covs, random_weights, cov_smoothing=1e-5)


I am working on the file "Clustering_4_em-with-text-data_blank.ipynb".
The first task is:
Running EM
Now that we have initialized all of our parameters, run EM.
Can you write Python code for it?

out = EM_for_high_dimension(tf_idf, means, covs, weights, cov_smoothing=1e-10)

out['loglik']


I am working on the file "Clustering_4_em-with-text-data_blank.ipynb".
The first task is:
Using a utility we provide, we will create a sparse matrix representation of the documents. This is the same utility function you used during the previous assignment on k-means with text data.
Can you write Python code for it?

tf_idf, map_index_to_word = sframe_to_scipy(wiki, 'tf_idf')


I am working on the file "Clustering_4_em-with-text-data_blank.ipynb".
The first task is:
We can check that the length (Euclidean norm) of each row is now 1.0, as expected.
Can you write Python code for it?

for i in range(5):
    doc = tf_idf[i]
    print(np.linalg.norm(doc.todense()))


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
Changing the hyperparameter alpha
Since alpha is responsible for smoothing document preferences over topics, the impact of changing its value should be visible when we plot the distribution of topic weights for the same document under models fit with different alpha values. In the code below, we plot the (sorted) topic weights for the Wikipedia article on Barack Obama under models fit with high, original, and low settings of alpha.
Can you write Python code for it?

a = np.sort(tpm_low_alpha.predict(obama,output_type='probability')[0])[::-1]
b = np.sort(topic_model.predict(obama,output_type='probability')[0])[::-1]
c = np.sort(tpm_high_alpha.predict(obama,output_type='probability')[0])[::-1]
ind = np.arange(len(a))
width = 0.3

def param_bar_plot(a,b,c,ind,width,ylim,param,xlab,ylab):
    fig = plt.figure()
    ax = fig.add_subplot(111)

    b1 = ax.bar(ind, a, width, color='lightskyblue')
    b2 = ax.bar(ind+width, b, width, color='lightcoral')
    b3 = ax.bar(ind+(2*width), c, width, color='gold')

    ax.set_xticks(ind+width)
    ax.set_xticklabels(range(10))
    ax.set_ylabel(ylab)
    ax.set_xlabel(xlab)
    ax.set_ylim(0,ylim)
    ax.legend(handles = [b1,b2,b3],labels=['low '+param,'original model','high '+param])

    plt.tight_layout()

param_bar_plot(a,b,c,ind,width,ylim=1.0,param='alpha',
               xlab='Topics (sorted by weight of top 100 words)',ylab='Topic Probability for Obama Article')


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
Comparing LDA to nearest neighbors for document retrieval
So far we have found that our topic model has learned some coherent topics, we have explored these topics as probability distributions over a vocabulary, and we have seen how individual documents in our Wikipedia data set are assigned to these topics in a way that corresponds with our expectations.
In this section, we will use the predicted topic distribution as a representation of each document, similar to how we have previously represented documents by word count or TF-IDF. This gives us a way of computing distances between documents, so that we can run a nearest neighbors search for a given document based on its membership in the topics that we learned from LDA. We can contrast the results with those obtained by running nearest neighbors under the usual TF-IDF representation, an approach that we explored in a previous assignment.
We'll start by creating the LDA topic distribution representation for each document:
Can you write Python code for it?

wiki['lda'] = topic_model.predict(wiki_docs, output_type='probability')


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
For each of our two different document representations, we can use GraphLab Create to compute a brute-force nearest neighbors model:
Can you write Python code for it?

model_tf_idf = gl.nearest_neighbors.create(wiki, label='name', features=['tf_idf'],
                                           method='brute_force', distance='cosine')
model_lda_rep = gl.nearest_neighbors.create(wiki, label='name', features=['lda'],
                                            method='brute_force', distance='cosine')


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
From these two plots we can see that the low gamma model results in higher weight placed on the top words and lower weight placed on the bottom words for each topic, while the high gamma model places relatively less weight on the top words and more weight on the bottom words. Thus increasing gamma results in topics that have a smoother distribution of weight across all the words in the vocabulary.
Quiz Question: For each topic of the low gamma model, compute the number of words required to make a list with total probability 0.5. What is the average number of words required across all topics? (HINT: use the get_topics() function from GraphLab Create with the cdf_cutoff argument).
Can you write Python code for it?

words = tpm_low_gamma.get_topics(num_words=2000, cdf_cutoff=0.5)

words.num_rows() / 10.0


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
GraphLab provides a useful summary of the model we have fitted, including the hyperparameter settings for alpha, gamma (note that GraphLab Create calls this parameter beta), and K (the number of topics); the structure of the output data; and some useful methods for understanding the results.
Can you write Python code for it?


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
In the above plot, each line corresponds to one of our ten topics. Notice how for each topic, the weights drop off sharply as we move down the ranked list of most important words. This shows that the top 10-20 words in each topic are assigned a much greater weight than the remaining words - and remember from the summary of our topic model that our vocabulary has 547462 words in total!
Next we plot the total weight assigned by each topic to its top 10 words:
Can you write Python code for it?

top_probs = [sum(topic_model.get_topics(topic_ids=[i], num_words=10)['score']) for i in range(10)]

ind = np.arange(10)
width = 0.5

fig, ax = plt.subplots()

ax.bar(ind-(width/2),top_probs,width)
ax.set_xticks(ind)

plt.xlabel('Topic')
plt.ylabel('Probability')
plt.title('Total Probability of Top 10 Words in each Topic')
plt.xlim(-0.5,9.5)
plt.ylim(0,0.15)
plt.show()


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
It is certainly useful to have pre-implemented methods available for LDA, but as with our previous methods for clustering and retrieval, implementing and fitting the model gets us only halfway towards our objective. We now need to analyze the fitted model to understand what it has done with our data and whether it will be useful as a document classification system. This can be a challenging task in itself, particularly when the model that we use is complex. We will begin by outlining a sequence of objectives that will help us understand our model in detail. In particular, we will

get the top words in each topic and use these to identify topic themes
predict topic distributions for some example documents
compare the quality of LDA "nearest neighbors" to the NN output from the first assignment
understand the role of model hyperparameters alpha and gamma

Load a fitted topic model
The method used to fit the LDA model is a randomized algorithm, which means that it involves steps that are random; in this case, the randomness comes from Gibbs sampling, as discussed in the LDA video lectures. Because of these random steps, the algorithm will be expected to yield slighty different output for different runs on the same data - note that this is different from previously seen algorithms such as k-means or EM, which will always produce the same results given the same input and initialization.
It is important to understand that variation in the results is a fundamental feature of randomized methods. However, in the context of this assignment this variation makes it difficult to evaluate the correctness of your analysis, so we will load and analyze a pre-trained model.
We recommend that you spend some time exploring your own fitted topic model and compare our analysis of the pre-trained model to the same analysis applied to the model you trained above.
Can you write Python code for it?


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
Let's compare these nearest neighbor models by finding the nearest neighbors under each representation on an example document. For this example we'll use Paul Krugman, an American economist:
Can you write Python code for it?

model_tf_idf.query(wiki[wiki['name'] == 'Paul Krugman'], label='name', k=10)

model_lda_rep.query(wiki[wiki['name'] == 'Paul Krugman'], label='name', k=10)


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
Let's look at the top 10 words for each topic to see if we can identify any themes:
Can you write Python code for it?

[x['words'] for x in topic_model.get_topics(output_type='topic_words', num_words=10)]


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
Notice that that there is no overlap between the two sets of top 10 nearest neighbors. This doesn't necessarily mean that one representation is better or worse than the other, but rather that they are picking out different features of the documents.
With TF-IDF, documents are distinguished by the frequency of uncommon words. Since similarity is defined based on the specific words used in the document, documents that are "close" under TF-IDF tend to be similar in terms of specific details. This is what we see in the example: the top 10 nearest neighbors are all economists from the US, UK, or Canada.
Our LDA representation, on the other hand, defines similarity between documents in terms of their topic distributions. This means that documents can be "close" if they share similar themes, even though they may not share many of the same keywords. For the article on Paul Krugman, we expect the most important topics to be 'American college and politics' and 'science and research'. As a result, we see that the top 10 nearest neighbors are academics from a wide variety of fields, including literature, anthropology, and religious studies.
Quiz Question: Using the TF-IDF representation, compute the 5000 nearest neighbors for American baseball player Alex Rodriguez. For what value of k is Mariano Rivera the k-th nearest neighbor to Alex Rodriguez? (Hint: Once you have a list of the nearest neighbors, you can use mylist.index(value) to find the index of the first instance of value in mylist.)
Quiz Question: Using the LDA representation, compute the 5000 nearest neighbors for American baseball player Alex Rodriguez. For what value of k is Mariano Rivera the k-th nearest neighbor to Alex Rodriguez? (Hint: Once you have a list of the nearest neighbors, you can use mylist.index(value) to find the index of the first instance of value in mylist.)
Can you write Python code for it?

lda_neighbors = model_lda_rep.query(wiki[wiki['name'] == 'Alex Rodriguez'], label='name', k=5000)
tf_idf_neighbors = model_tf_idf.query(wiki[wiki['name'] == 'Alex Rodriguez'], label='name', k=5000)

print list(tf_idf_neighbors['reference_label']).index('Mariano Rivera')
print list(lda_neighbors['reference_label']).index('Mariano Rivera')


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
Quiz Question: For each topic of the high gamma model, compute the number of words required to make a list with total probability 0.5. What is the average number of words required across all topics? (HINT: use the get_topics() function from GraphLab Create with the cdf_cutoff argument).
Can you write Python code for it?

words = tpm_high_gamma.get_topics(num_words=2000, cdf_cutoff=0.5)

words.num_rows() / 10.0


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
To get a more robust estimate of the topics for each document, we can average a large number of predictions for the same document:
Can you write Python code for it?

def average_predictions(model, test_document, num_trials=100):
    avg_preds = np.zeros((model.num_topics))
    for i in range(num_trials):
        avg_preds += model.predict(test_document, output_type='probability')[0]
    avg_preds = avg_preds/num_trials
    result = gl.SFrame({'topics':themes, 'average predictions':avg_preds})
    result = result.sort('average predictions', ascending=False)
    return result

print average_predictions(topic_model, obama, 100)


I am working on the file "Clustering_5_lda_blank.ipynb".
The first task is:
We propose the following themes for each topic:

topic 0: Science and research
topic 2: Team sports
topic 3: Music, TV, and film
topic 4: American college and politics
topic 5: General politics
topic 6: Art and publishing
topic 7: Business
topic 8: International athletics
topic 9: Great Britain and Australia
topic 10: International music

We'll save these themes for later:
Can you write Python code for it?

themes = ['science and research','team sports','music, TV, and film','American college and politics','general politics', \
         'art and publishing','Business','international athletics','Great Britain and Australia','international music']


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
Bipartition the Wikipedia dataset using k-means
Recall our workflow for clustering text data with k-means:

Load the dataframe containing a dataset, such as the Wikipedia text dataset.
Extract the data matrix from the dataframe.
Run k-means on the data matrix with some value of k.
Visualize the clustering results using the centroids, cluster assignments, and the original dataframe. We keep the original dataframe around because the data matrix does not keep auxiliary information (in the case of the text dataset, the title of each article).

Let us modify the workflow to perform bipartitioning:

Load the dataframe containing a dataset, such as the Wikipedia text dataset.
Extract the data matrix from the dataframe.
Run k-means on the data matrix with k=2.
Divide the data matrix into two parts using the cluster assignments.
Divide the dataframe into two parts, again using the cluster assignments. This step is necessary to allow for visualization.
Visualize the bipartition of data.

We'd like to be able to repeat Steps 3-6 multiple times to produce a hierarchy of clusters such as the following:
(root)
                         |
            +------------+-------------+
            |                          |
         Cluster                    Cluster
     +------+-----+             +------+-----+
     |            |             |            |
   Cluster     Cluster       Cluster      Cluster
Each parent cluster is bipartitioned to produce two child clusters. At the very top is the root cluster, which consists of the entire dataset.
Now we write a wrapper function to bipartition a given cluster using k-means. There are three variables that together comprise the cluster:

dataframe: a subset of the original dataframe that correspond to member rows of the cluster
matrix: same set of rows, stored in sparse matrix format
centroid: the centroid of the cluster (not applicable for the root cluster)

Rather than passing around the three variables separately, we package them into a Python dictionary. The wrapper function...
Can you write Python code for it?

def bipartition(cluster, maxiter=400, num_runs=4, seed=None):
    '''cluster: should be a dictionary containing the following keys
                * dataframe: original dataframe
                * matrix:    same data, in matrix format
                * centroid:  centroid for this particular cluster'''

    data_matrix = cluster['matrix']
    dataframe   = cluster['dataframe']

    # Run k-means on the data matrix with k=2. We use scikit-learn here to simplify workflow.
    kmeans_model = KMeans(n_clusters=2, max_iter=maxiter, n_init=num_runs, random_state=seed, n_jobs=-1)
    kmeans_model.fit(data_matrix)
    centroids, cluster_assignment = kmeans_model.cluster_centers_, kmeans_model.labels_

    # Divide the data matrix into two parts using the cluster assignments.
    data_matrix_left_child, data_matrix_right_child = data_matrix[cluster_assignment==0], \
                                                      data_matrix[cluster_assignment==1]

    # Divide the dataframe into two parts, again using the cluster assignments.
    cluster_assignment_sa = graphlab.SArray(cluster_assignment) # minor format conversion
    dataframe_left_child, dataframe_right_child     = dataframe[cluster_assignment_sa==0], \
                                                      dataframe[cluster_assignment_sa==1]

    # Package relevant variables for the child clusters
    cluster_left_child  = {'matrix': data_matrix_left_child,
                           'dataframe': dataframe_left_child,
                           'centroid': centroids[0]}
    cluster_right_child = {'matrix': data_matrix_right_child,
                           'dataframe': dataframe_right_child,
                           'centroid': centroids[1]}

    return (cluster_left_child, cluster_right_child)


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
Caution. The granularity criteria is an imperfect heuristic and must be taken with a grain of salt. It takes a lot of manual intervention to obtain a good hierarchy of clusters.

If a cluster is highly mixed, the top articles and words may not convey the full picture of the cluster. Thus, we may be misled if we judge the purity of clusters solely by their top documents and words.
Many interesting topics are hidden somewhere inside the clusters but do not appear in the visualization. We may need to subdivide further to discover new topics. For instance, subdividing the ice_hockey_football cluster led to the appearance of golf.

Quiz Question. Which diagram best describes the hierarchy right after splitting the ice_hockey_football cluster? Refer to the quiz form for the diagrams.
Cluster of non-athletes
Now let us subdivide the cluster of non-athletes.
Can you write Python code for it?

# Bipartition the cluster of non-athletes
left_child_non_athletes, right_child_non_athletes = bipartition(non_athletes, maxiter=100, num_runs=8, seed=1)

display_single_tf_idf_cluster(left_child_non_athletes, map_index_to_word)

display_single_tf_idf_cluster(right_child_non_athletes, map_index_to_word)


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
Cluster of ice hockey players and football players
In answering the following quiz question, take a look at the topics represented in the top documents (those closest to the centroid), as well as the list of words with highest TF-IDF weights.
Quiz Question. Bipartition the cluster of ice hockey and football players. Which of the two child clusters should be futher subdivided?
Note. To achieve consistent results, use the arguments maxiter=100, num_runs=8, seed=1 when calling the bipartition function.

The left child cluster
The right child cluster
Can you write Python code for it?

left, right = bipartition(ice_hockey_football, maxiter=100, num_runs=8, seed=1)

display_single_tf_idf_cluster(left, map_index_to_word)

display_single_tf_idf_cluster(right, map_index_to_word)


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
Let's examine the contents of one of the two clusters, which we call the left_child, referring to the tree visualization above.
Can you write Python code for it?


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
Let's visualize the two child clusters:
Can you write Python code for it?

display_single_tf_idf_cluster(left_child, map_index_to_word)

display_single_tf_idf_cluster(right_child, map_index_to_word)


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
Load the Wikipedia dataset
Can you write Python code for it?


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
Note. Concerning use of "football"
The occurrences of the word "football" above refer to association football. This sports is also known as "soccer" in United States (to avoid confusion with American football). We will use "football" throughout when discussing topic representation.
Our hierarchy of clusters now looks like this:
Wikipedia
                                               +
                                               |
                    +--------------------------+--------------------+
                    |                                               |
                    +                                               +
                 Athletes                                      Non-athletes
                    +
                    |
        +-----------+--------+
        |                    |
        |                    +
        +                 football/
     baseball            ice hockey
Should we keep subdividing the clusters? If so, which cluster should we subdivide? To answer this question, we again think about our application. Since we organize our directory by topics, it would be nice to have topics that are about as coarse as each other. For instance, if one cluster is about baseball, we expect some other clusters about football, basketball, volleyball, and so forth. That is, we would like to achieve similar level of granularity for all clusters.
Notice that the right child cluster is more coarse than the left child cluster. The right cluster posseses a greater variety of topics than the left (ice hockey/football vs. baseball). So the right child cluster should be subdivided further to produce finer child clusters.
Let's give the clusters aliases as well:
Can you write Python code for it?

baseball            = left_child_athletes
ice_hockey_football = right_child_athletes


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
On the other hand, the right child cluster is a mix of football players and ice hockey players:
Can you write Python code for it?


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
Quiz Question. Let us bipartition the clusters scholars_politicians_etc and musicians_artists_etc. Which diagram best describes the resulting hierarchy of clusters for the non-athletes? Refer to the quiz for the diagrams.
Note. Use maxiter=100, num_runs=8, seed=1 for consistency of output.
Can you write Python code for it?

s_left, s_right = bipartition(scholars_politicians_etc, maxiter=100, num_runs=8, seed=1)

m_left, m_right = bipartition(musicians_artists_etc, maxiter=100, num_runs=8, seed=1)

display_single_tf_idf_cluster(s_left, map_index_to_word)

display_single_tf_idf_cluster(s_right, map_index_to_word)

display_single_tf_idf_cluster(m_left, map_index_to_word)

display_single_tf_idf_cluster(m_right, map_index_to_word)


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
The first cluster consists of scholars, politicians, and government officials whereas the second consists of musicians, artists, and actors. Run the following code cell to make convenient aliases for the clusters.
Can you write Python code for it?

scholars_politicians_etc = left_child_non_athletes
musicians_artists_etc = right_child_non_athletes


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
The left child cluster mainly consists of baseball players:
Can you write Python code for it?


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
The left cluster consists of athletes, whereas the right cluster consists of non-athletes. So far, we have a single-level hierarchy consisting of two clusters, as follows:
Wikipedia
                                               +
                                               |
                    +--------------------------+--------------------+
                    |                                               |
                    +                                               +
                 Athletes                                      Non-athletes
Is this hierarchy good enough? When building a hierarchy of clusters, we must keep our particular application in mind. For instance, we might want to build a directory for Wikipedia articles. A good directory would let you quickly narrow down your search to a small set of related articles. The categories of athletes and non-athletes are too general to facilitate efficient search. For this reason, we decide to build another level into our hierarchy of clusters with the goal of getting more specific cluster structure at the lower level. To that end, we subdivide both the athletes and non-athletes clusters.
Perform recursive bipartitioning
Cluster of athletes
To help identify the clusters we've built so far, let's give them easy-to-read aliases:
Can you write Python code for it?

athletes = left_child
non_athletes = right_child


I am working on the file "Clustering_6_hierarchical_clustering_blank.ipynb".
The first task is:
To run k-means on this dataset, we should convert the data matrix into a sparse matrix.
Can you write Python code for it?

from em_utilities import sframe_to_scipy # converter

# This will take about a minute or two.
tf_idf, map_index_to_word = sframe_to_scipy(wiki, 'tf_idf')


I am working on the file "CoNLL10 output with SpaCy.ipynb".
The first task is:
CoNLL10 output
SpaCy's output--in particular, its token IDs--takes some massaging in order to produce a well-formed CoNLL10 document.
The column layout is described here.
Can you write Python code for it?

for sent in doc.sents:
    # Create lookup dict for token IDs.
    ids = {}
    for i, token in enumerate(sent):
        ids[token.idx] = i+1

    for token in sent:
        # Clean up token attributes
        token_id = str(ids[token.idx]).strip()
        token_text = str(token).strip()
        lemma = str(token.lemma_).strip()
        pos_tag = str(token.tag_).strip()
        depend = str(token.dep_).strip()

        # Set head ID correctly for root of sentence.
        if token.dep_ == 'ROOT':
            head_id = str(0)
        else:
            head_id = str(ids[token.head.idx]).strip()

        # CoNLL10 output
        # Comments below are modified from https://corpling.uis.georgetown.edu/xrenner/doc/using.html#input-format
        print(token_id + '\t' +      # token ID w/in sentence
              token_text + '\t' +    # token text
              lemma + '\t' +         # lemmatized token
              pos_tag + '\t' +       # part of speech tag for token
              pos_tag + '\t' +       # part of speech tag for token
              '_' + '\t' +           # placeholder for morphological information
              head_id + '\t' +       # ID of head token
              depend + '\t' +        # dependency function
              '_' + '\t' + '_')      # two unused columns


I am working on the file "CoNLL10 output with SpaCy.ipynb".
The first task is:
Give spaCy some input text, then process it.
Note: SpaCy input has to be in Unicode.
Can you write Python code for it?

text = u'''1. Cato's family got its first lustre and fame from his great-grandfather Cato (a man whose virtue gained him the greatest reputation and influence among the Romans, as has been written in his Life), but the death of both parents left him an orphan, together with his brother Caepio and his sister Porcia. Cato had also a half-sister, Servilia, the daughter of his mother.1 All these children were brought up in the home of Livius Drusus, their uncle on the mother's side, who at that time was a leader in the conduct of public affairs; for he was a most powerful speaker, in general a man of the greatest discretion, and yielded to no Roman in dignity of purpose.
[2] We are told that from his very childhood Cato displayed, in speech, in countenance, and in his childish sports, a nature that was inflexible, imperturbable, and altogether steadfast. He set out to accomplish his purposes with a vigour beyond his years, and while he was harsh and repellent to those who would flatter him, he was still more masterful towards those who tried to frighten him. It was altogether difficult to make him laugh, although once in a while he relaxed his features so far as to smile; and he was not quickly nor easily moved to anger, though once angered he was inexorable.'''

doc = nlp(text)


I am working on the file "CoNLL10 output with SpaCy.ipynb".
The first task is:
Load the English tagger
Note: Loading the tagger is expensive. The documentation says it can take 10-20 seconds and 2-3 GB of RAM.
Can you write Python code for it?


I am working on the file "Code Snippets/daily_programmer/cliques/clique_detection.ipynb".
The first task is:
Since this bound is small enough, one could check all 7-membered or smaller subgraphs of the graph $G$ to see if they were complete and get a clique. But one can do better, presumably.
Can you write Python code for it?

clique_size = 7

valid_nodes = cd.find_possible_clique_member_nodes(clique_size, graph)

print(cd.get_cliques(clique_size, valid_nodes, graph))

print(cd.find_maximum_clique(graph))


I am working on the file "Code Snippets/daily_programmer/cliques/clique_detection.ipynb".
The first task is:
This loads the large dolphin social network into a graph $G$.
Can you write Python code for it?


I am working on the file "Code Snippets/daily_programmer/cliques/clique_detection.ipynb".
The first task is:
This was a terrible idea. Should have gone with Bronn-Kerbosch, which finds all maximal cliques. It works by starting off with 2-cliques, and expanding them, until no more cliques can be expanded. Why didn't I think of that?
Let's try Bronn-Kerbosch along with the naive bound on maximal clique size.
Can you write Python code for it?

num_nodes2, adjacency_list2 = cd.parse_file_into_adjacency_list("small.dat")
graph2 = cd.parse_graph(num_nodes2, adjacency_list2)

small_clique = (0,1)

print(cd.get_clique_extensions(small_clique, graph2))

print(cd.get_two_cliques(graph2))

cd.find_maximal_cliques(graph)

%timeit cd.find_maximal_cliques(graph)

%prun cd.find_maximal_cliques(graph)


I am working on the file "Code/.ipynb_checkpoints/AUP_LCA_evelynegroen-checkpoint.ipynb".
The first task is:
Step 1: Calculate partial derivatives
NB: this is a vectorized implementation of the MatLab code that was originally written by Reinout Heijungs & Sangwong Suh
Can you write Python code for it?

s = A_det.I * f                                 #scaling vector s: inv(A_det)*f
Lambda = B_det * A_det.I;                       #B_det*inv(A)

dgdA = -(s * Lambda).T                          #Partial derivatives A-matrix
Gamma_A = np.multiply((A_det/g_LCA), dgdA)      #For free: the multipliers of the A-matrix
print("The multipliers of the A-matrix are:")
print(Gamma_A)

dgdB = s.T                                      #Partial derivatives B-matrix
Gamma_B = np.multiply((B_det/g_LCA), dgdB)      #For free too: the multipliers of the B-matrix
print("The multipliers of the B-matrix are:")
print(Gamma_B)


I am working on the file "Code/.ipynb_checkpoints/AUP_LCA_evelynegroen-checkpoint.ipynb".
The first task is:
Step 2: Determine output variance
Can you write Python code for it?

CV = 0.05                             #Coefficient of variation set to 5% (CV = sigma/mu)
var_A = np.power(abs(CV*A_det),2)     #Variance of the A-matrix (var =sigma^2)
var_B = np.power(abs(CV*B_det),2)     #Variance of the B-matrix

P = np.concatenate((np.reshape(dgdA, 4), dgdB), axis=1)       #P contains partial derivatives of both A and B
var_P = np.concatenate((np.reshape(var_A, 4), var_B), axis=1) #var_P contains all variances of each parameter in A and B

var_g = sum(np.multiply(np.power(P, 2), var_P))               #Total output variance (first order Taylor)
var_g = var_g[0,0] + var_g[0,1] +var_g[0,2] + var_g[0,3] + var_g[0,4] + var_g[0,5]

print("The total output variance equals:", var_g)


I am working on the file "Code/1_data_ingestion.ipynb".
The first task is:
Azure Blob Storage Container
We will be storing intermediate results for use between these Jupyter notebooks in an Azure Blob Storage container. Instructions for setting up your Azure Storage account are available within this link (https://docs.microsoft.com/en-us/azure/storage/blobs/storage-python-how-to-use-blob-storage). You will need to copy your account name and account key from the Access Keys area in the portal into the following code block. These credentials will be reused in all four Jupyter notebooks.
We will handle creating the containers and writing the data to these containers for each notebook. Further instructions for using Azure Blob storage with AML Workbench are available
(https://github.com/Azure/ViennaDocs/blob/master/Documentation/UsingBlobForStorage.md).
You will need to enter the ACCOUNT_NAME as well as the ACCOUNT_KEY in order to access Azure Blob storage account you have created. This notebook will create and store all the resulting data files in a blob container under this account.
Can you write Python code for it?

# Enter your Azure blob storage details here
ACCOUNT_NAME = "<your blob storage account name>"

# You can find the account key under the _Access Keys_ link in the
# [Azure Portal](portal.azure.com) page for your Azure storage container.
ACCOUNT_KEY = "<your blob storage account key>"

#-------------------------------------------------------------------------------------------
# We will create this container to hold the results of executing this notebook.
# If this container name already exists, we will use that instead, however
# This notebook will ERASE ALL CONTENTS.
CONTAINER_NAME = "dataingestion"

# Connect to your blob service
az_blob_service = BlockBlobService(account_name=ACCOUNT_NAME, account_key=ACCOUNT_KEY)

# Create a new container if necessary, otherwise you can use an existing container.
# This command creates the container if it does not already exist. Else it does nothing.
az_blob_service.create_container(CONTAINER_NAME,
                                 fail_on_exist=False,
                                 public_access=PublicAccess.Container)


I am working on the file "Code/1_data_ingestion.ipynb".
The first task is:
Download simulated data sets
We will be reusing the raw simulated data files from another tutorial. The notebook automatically downloads these files stored at Microsoft/SQL-Server-R-Services-Samples GitHub site.
The five data files are:

machines.csv
maint.csv
errors.csv
telemetry.csv
failures.csv

To get an idea of what is contained in the data, we examine this machine schematic.

There are 1000 machines of four different models. Each machine contains four components of interest, and four sensors measuring voltage, pressure, vibration and rotation. A controller monitors the system and raises alerts for five different error conditions. Maintenance logs indicate when something is done to the machine which does not include a component replacement. A failure is defined by the replacement of a component.
This notebook does some preliminary data cleanup, creates summary graphics for each data set to verify the data downloaded correctly, and stores the resulting data sets in the Azure blob container created in the previous section.
Can you write Python code for it?

# The raw data is stored on GitHub here:
basedataurl = "http://media.githubusercontent.com/media/Microsoft/SQL-Server-R-Services-Samples/master/PredictiveMaintanenceModelingGuide/Data/"

# We will store each of these data sets in blob storage in an
# Azure Storage Container on your Azure subscription.
# See https://github.com/Azure/ViennaDocs/blob/master/Documentation/UsingBlobForStorage.md
# for details.

# These file names detail which blob each files is stored under.
MACH_DATA = 'machines_files.parquet'
MAINT_DATA = 'maint_files.parquet'
ERROR_DATA = 'errors_files.parquet'
TELEMETRY_DATA = 'telemetry_files.parquet'
FAILURE_DATA = 'failure_files.parquet'


I am working on the file "Code/1_data_ingestion.ipynb".
The first task is:
Errors  data set
The error log contains non-breaking errors recorded while the machine is still operational. These errors are not considered failures, though they may be predictive of a future failure event. The error datetime field is rounded to the closest hour since the telemetry data (loaded later) is collected on an hourly rate.
Can you write Python code for it?

# load raw data from the GitHub URL
datafile = "errors.csv"

# Download the file once, and only once.
if not os.path.isfile(datafile):
    urllib.request.urlretrieve(basedataurl+datafile, datafile)

# Read into pandas
errors = pd.read_csv(datafile, encoding='utf-8')

print(errors.count())
errors.head(10)


I am working on the file "Code/1_data_ingestion.ipynb".
The first task is:
Maintenance data set
The maintenance log contains both scheduled and unscheduled maintenance records. Scheduled maintenance corresponds with  regular inspection of components, unscheduled maintenance may arise from mechanical failure or other performance degradations. A failure record is generated for component replacement in the case  of either maintenance events. Because maintenance events can also be used to infer component life, the maintenance data has been collected over two years (2014, 2015) instead of only over the year of interest (2015).
Can you write Python code for it?

# load raw data from the GitHub URL
datafile = "maint.csv"

# Download the file once, and only once.
if not os.path.isfile(datafile):
    urllib.request.urlretrieve(basedataurl+datafile, datafile)

# Read into pandas
maint = pd.read_csv(datafile, encoding='utf-8')

print(maint.count())
maint.head(20)

# Quick plot to show structure
ggplot(aes(x="comp"), maint) + geom_bar(fill="blue", color="black")


I am working on the file "Code/1_data_ingestion.ipynb".
The first task is:
Now we write the spark dataframe to an Azure blob storage container for use in the remaining notebooks of this scenario.
Can you write Python code for it?

# Write the telemetry data set to intermediate storage
telemetry_spark.write.mode('overwrite').parquet(TELEMETRY_DATA)
for blob in az_blob_service.list_blobs(CONTAINER_NAME):
    if TELEMETRY_DATA in blob.name:
        az_blob_service.delete_blob(CONTAINER_NAME, blob.name)

# upload the entire folder into blob storage
for name in glob.iglob(TELEMETRY_DATA + '/*'):
    print(os.path.abspath(name))
    az_blob_service.create_blob_from_path(CONTAINER_NAME, name, name)

print("Telemetry files saved!")


I am working on the file "Code/1_data_ingestion.ipynb".
The first task is:
Telemetry data set
The telemetry time-series data consists of voltage, rotation, pressure, and vibration sensor measurements collected from each  machines in real time. The data is averaged over an hour and stored in the telemetry logs.
Can you write Python code for it?

# Github has been having some timeout issues. This should fix the problem for this dataset.
import socket
socket.setdefaulttimeout(30)

# load raw data from the GitHub URL
datafile = "telemetry.csv"

# Download the file once, and only once.
if not os.path.isfile(datafile):
    urllib.request.urlretrieve(basedataurl+datafile, datafile)

# Read into pandas
telemetry = pd.read_csv(datafile, encoding='utf-8')

# handle missing values
# define groups of features
features_datetime = ['datetime']
features_categorical = ['machineID']
features_numeric = list(set(telemetry.columns) - set(features_datetime) - set(features_categorical))

# Replace numeric NA with 0
telemetry[features_numeric] = telemetry[features_numeric].fillna(0)

# Replace categorical NA with 'Unknown'
telemetry[features_categorical]  = telemetry[features_categorical].fillna("Unknown")

# Counts...
print(telemetry.count())

# Examine 10 rowzs of data.
telemetry.head(10)

# Check the incoming schema, we want to convert datetime to the correct type.
# format datetime field which comes in as string
telemetry.dtypes


I am working on the file "Code/1_data_ingestion.ipynb".
The first task is:
The figure shows a histogram of component replacements divided into the four component types over the entire maintenance history. It looks like these four components are replaced at similar rates.
There are many ways we might want to look at this data including calculating how long each component type lasts, or the time history of component replacements within each machine. This will take some preprocess of the data, which we are delaying until we do the feature engineering steps in the next example notebook.
Next, we convert the errors data to a Spark dataframe, and verify the data types have converted correctly.
Can you write Python code for it?

# The data was read in using a Pandas data frame. We'll convert
# it to pyspark to ensure it is in a Spark usable form for later
# manipulations.
maint_spark = spark.createDataFrame(maint,
                              verifySchema=False)

# We no longer need the pandas dataframe, so we can release that memory.
del maint

# Check data type conversions.
maint_spark.printSchema()