I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
Find nearest neighbors
Let's start by finding the nearest neighbors of the Barack Obama page using the word count vectors to represent the articles and Euclidean distance to measure distance.  For this, again will we use a GraphLab Create implementation of nearest neighbor search.
Can you write Python code for it?

model = graphlab.nearest_neighbors.create(wiki, label='name', features=['word_count'],
                                          method='brute_force', distance='euclidean')


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
From a glance at the above table, things look better.  For example, we now see Joe Biden as Barack Obama's nearest neighbor!  We also see Hillary Clinton on the list.  This list looks even more plausible as nearest neighbors of Barack Obama.
Let's make a plot to better visualize the effect of having used cosine distance in place of Euclidean on our TF-IDF vectors.
Can you write Python code for it?

plt.figure(figsize=(10.5,4.5))
plt.figure(figsize=(10.5,4.5))
plt.hist(wiki['length'], 50, color='k', edgecolor='None', histtype='stepfilled', normed=True,
         label='Entire Wikipedia', zorder=3, alpha=0.8)
plt.hist(nearest_neighbors_euclidean['length'], 50, color='r', edgecolor='None', histtype='stepfilled', normed=True,
         label='100 NNs of Obama (Euclidean)', zorder=10, alpha=0.8)
plt.hist(nearest_neighbors_cosine['length'], 50, color='b', edgecolor='None', histtype='stepfilled', normed=True,
         label='100 NNs of Obama (cosine)', zorder=11, alpha=0.8)
plt.axvline(x=wiki['length'][wiki['name'] == 'Barack Obama'][0], color='k', linestyle='--', linewidth=4,
           label='Length of Barack Obama', zorder=2)
plt.axvline(x=wiki['length'][wiki['name'] == 'Joe Biden'][0], color='g', linestyle='--', linewidth=4,
           label='Length of Joe Biden', zorder=1)
plt.axis([1000, 5500, 0, 0.004])
plt.legend(loc='best', prop={'size':15})
plt.title('Distribution of document length')
plt.xlabel('# of words')
plt.ylabel('Percentage')
plt.rcParams.update({'font.size': 16})
plt.tight_layout()


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
Let's compare this distance to the distance between the Barack Obama article and all of its Wikipedia 10 nearest neighbors:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
Let's extract the list of most frequent words that appear in both Obama's and Barrio's documents. We've so far sorted all words from Obama and Barrio's articles by their word frequencies. We will now use a dataframe operation known as join. The join operation is very useful when it comes to playing around with data: it lets you combine the content of two tables using a shared column (in this case, the word column). See the documentation for more details.
For instance, running
obama_words.join(barrio_words, on='word')
will extract the rows from both tables that correspond to the common words.
Can you write Python code for it?

combined_words = obama_words.join(barrio_words, on='word')
combined_words


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
Let's look at the TF-IDF vectors for this tweet and for Barack Obama's Wikipedia entry, just to visually see their differences.
Can you write Python code for it?

tweet_tf_idf = sf[0]['tf_idf.word_count']
tweet_tf_idf

obama


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
Let's look at the top 10 nearest neighbors by performing the following query:
Can you write Python code for it?

model.query(wiki[wiki['name']=='Barack Obama'], label='name', k=10)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
Load Wikipedia dataset
We will be using the same dataset of Wikipedia pages that we used in the Machine Learning Foundations course (Course 1). Each element of the dataset consists of a link to the wikipedia article, the name of the person, and the text of the article (in lowercase).
Can you write Python code for it?

wiki = graphlab.SFrame('people_wiki.gl')

wiki


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
Note. The join operation does not enforce any particular ordering on the shared column. So to obtain, say, the five common words that appear most often in Obama's article, sort the combined table by the Obama column. Don't forget ascending=False to display largest counts first.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
Notice the huge difference in this calculation using TF-IDF scores instead  of raw word counts. We've eliminated noise arising from extremely common words.
Choosing metrics
You may wonder why Joe Biden, Obama's running mate in two presidential elections, is missing from the query results of model_tf_idf. Let's find out why. First, compute the distance between TF-IDF features of Obama and Biden.
Quiz Question. Compute the Euclidean distance between TF-IDF features of Obama and Biden. Hint: When using Boolean filter in SFrame/SArray, take the index 0 to access the first match.
The distance is larger than the distances we found for the 10 nearest neighbors, which we repeat here for readability:
Can you write Python code for it?

model_tf_idf.query(wiki[wiki['name'] == 'Barack Obama'], label='name', k=10)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
Now, compute the cosine distance between the Barack Obama article and this tweet:
Can you write Python code for it?

obama = wiki[wiki['name'] == 'Barack Obama']
obama_tf_idf = obama[0]['tf_idf']
graphlab.toolkits.distances.cosine(obama_tf_idf, tweet_tf_idf)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/.ipynb_checkpoints/0_nearest-neighbors-features-and-metrics_blank-checkpoint.ipynb".
The first task is:
Since both tables contained the column named count, SFrame automatically renamed one of them to prevent confusion. Let's rename the columns to tell which one is for which. By inspection, we see that the first column (count) is for Obama and the second (count.1) for Barrio.
Can you write Python code for it?

combined_words = combined_words.rename({'count':'Obama', 'count.1':'Barrio'})
combined_words


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
But one may wonder, is Biden's article that different from Obama's, more so than, say, Schiliro's? It turns out that, when we compute nearest neighbors using the Euclidean distances, we unwittingly favor short articles over long ones. Let us compute the length of each Wikipedia document, and examine the document lengths for the 100 nearest neighbors to Obama's page.
Can you write Python code for it?

def compute_length(row):
    return len(row['text'])

wiki['length'] = wiki.apply(compute_length)

nearest_neighbors_euclidean = model_tf_idf.query(wiki[wiki['name'] == 'Barack Obama'], label='name', k=100)
nearest_neighbors_euclidean = nearest_neighbors_euclidean.join(wiki[['name', 'length']], on={'reference_label':'name'})

nearest_neighbors_euclidean.sort('rank')


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Find nearest neighbors
Let's start by finding the nearest neighbors of the Barack Obama page using the word count vectors to represent the articles and Euclidean distance to measure distance.  For this, again will we use a GraphLab Create implementation of nearest neighbor search.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Indeed, the 100 nearest neighbors using cosine distance provide a sampling across the range of document lengths, rather than just short articles like Euclidean distance provided.
Moral of the story: In deciding the features and distance measures, check if they produce results that make sense for your particular application.
Problem with cosine distances: tweets vs. long articles
Happily ever after? Not so fast. Cosine distances ignore all document lengths, which may be great in certain situations but not in others. For instance, consider the following (admittedly contrived) example.
+--------------------------------------------------------+
|                                             +--------+ |
|  One that shall not be named                | Follow | |
|  @username                                  +--------+ |
|                                                        |
|  Democratic governments control law in response to     |
|  popular act.                                          |
|                                                        |
|  8:05 AM - 16 May 2016                                 |
|                                                        |
|  Reply   Retweet (1,332)   Like (300)                  |
|                                                        |
+--------------------------------------------------------+
How similar is this tweet to Barack Obama's Wikipedia article? Let's transform the tweet into TF-IDF features, using an encoder fit to the Wikipedia dataset.  (That is, let's treat this tweet as an article in our Wikipedia dataset and see what happens.)
Can you write Python code for it?

sf = graphlab.SFrame({'text': ['democratic governments control law in response to popular act']})
sf['word_count'] = graphlab.text_analytics.count_words(sf['text'])

encoder = graphlab.feature_engineering.TFIDF(features=['word_count'], output_column_prefix='tf_idf')
encoder.fit(wiki)
sf = encoder.transform(sf)
sf


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Let's compare this distance to the distance between the Barack Obama article and all of its Wikipedia 10 nearest neighbors:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Let's determine whether this list makes sense.
* With a notable exception of Roland Grossenbacher, the other 8 are all American politicians who are contemporaries of Barack Obama.
* Phil Schiliro, Jesse Lee, Samantha Power, and Eric Stern worked for Obama.
Clearly, the results are more plausible with the use of TF-IDF. Let's take a look at the word vector for Obama and Schilirio's pages. Notice that TF-IDF representation assigns a weight to each word. This weight captures relative importance of that word in the document. Let us sort the words in Obama's article by their TF-IDF weights; we do the same for Schiliro's article as well.
Can you write Python code for it?

def top_words_tf_idf(name):
    row = wiki[wiki['name'] == name]
    word_count_table = row[['tf_idf']].stack('tf_idf', new_column_name=['word','weight'])
    return word_count_table.sort('weight', ascending=False)

obama_tf_idf = top_words_tf_idf('Barack Obama')
obama_tf_idf

schiliro_tf_idf = top_words_tf_idf('Phil Schiliro')
schiliro_tf_idf


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Let's extract the list of most frequent words that appear in both Obama's and Barrio's documents. We've so far sorted all words from Obama and Barrio's articles by their word frequencies. We will now use a dataframe operation known as join. The join operation is very useful when it comes to playing around with data: it lets you combine the content of two tables using a shared column (in this case, the word column). See the documentation for more details.
For instance, running
obama_words.join(barrio_words, on='word')
will extract the rows from both tables that correspond to the common words.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Load Wikipedia dataset
We will be using the same dataset of Wikipedia pages that we used in the Machine Learning Foundations course (Course 1). Each element of the dataset consists of a link to the wikipedia article, the name of the person, and the text of the article (in lowercase).
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Note. Even though common words are swamping out important subtle differences, commonalities in rarer political words still matter on the margin. This is why politicians are being listed in the query result instead of musicians, for example. In the next subsection, we will introduce a different metric that will place greater emphasis on those rarer words.
TF-IDF to the rescue
Much of the perceived commonalities between Obama and Barrio were due to occurrences of extremely frequent words, such as "the", "and", and "his". So nearest neighbors is recommending plausible results sometimes for the wrong reasons.
To retrieve articles that are more relevant, we should focus more on rare words that don't happen in every article. TF-IDF (term frequency–inverse document frequency) is a feature representation that penalizes words that are too common.  Let's use GraphLab Create's implementation of TF-IDF and repeat the search for the 10 nearest neighbors of Barack Obama:
Can you write Python code for it?

wiki['tf_idf'] = graphlab.text_analytics.tf_idf(wiki['word_count'])

model_tf_idf = graphlab.nearest_neighbors.create(wiki, label='name', features=['tf_idf'],
                                                 method='brute_force', distance='euclidean')

model_tf_idf.query(wiki[wiki['name'] == 'Barack Obama'], label='name', k=10)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Note. The join operation does not enforce any particular ordering on the shared column. So to obtain, say, the five common words that appear most often in Obama's article, sort the combined table by the Obama column. Don't forget ascending=False to display largest counts first.
Can you write Python code for it?

sorted_combined = combined_words.sort('Obama', ascending=False)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Notice the huge difference in this calculation using TF-IDF scores instead  of raw word counts. We've eliminated noise arising from extremely common words.
Choosing metrics
You may wonder why Joe Biden, Obama's running mate in two presidential elections, is missing from the query results of model_tf_idf. Let's find out why. First, compute the distance between TF-IDF features of Obama and Biden.
Quiz Question. Compute the Euclidean distance between TF-IDF features of Obama and Biden. Hint: When using Boolean filter in SFrame/SArray, take the index 0 to access the first match.
Can you write Python code for it?

barack = wiki[wiki['name'] == 'Barack Obama']['tf_idf'][0]
biden = wiki[wiki['name'] == 'Joe Biden']['tf_idf'][0]
barack_biden = graphlab.toolkits.distances.euclidean(barack,biden)
barack_biden


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Now, compute the cosine distance between the Barack Obama article and this tweet:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Quiz Question. Among the words that appear in both Barack Obama and Francisco Barrio, take the 5 that appear most frequently in Obama. How many of the articles in the Wikipedia dataset contain all of those 5 words?
Hint:
* Refer to the previous paragraph for finding the words that appear in both articles. Sort the common words by their frequencies in Obama's article and take the largest five.
* Each word count vector is a Python dictionary. For each word count vector in SFrame, you'd have to check if the set of the 5 common words is a subset of the keys of the word count vector. Complete the function has_top_words to accomplish the task.
  - Convert the list of top 5 words into set using the syntax
set(common_words)
    where common_words is a Python list. See this link if you're curious about Python sets.
  - Extract the list of keys of the word count dictionary by calling the keys() method.
  - Convert the list of keys into a set as well.
  - Use issubset() method to check if all 5 words are among the keys.
* Now apply the has_top_words function on every row of the SFrame.
* Compute the sum of the result column to obtain the number of articles containing all the 5 top words.
Checkpoint. Check your has_top_words function on two random articles:
Can you write Python code for it?

common_words = sorted_combined['word'][ :5]
set(common_words)

common_words = sorted_combined['word'][ :5]  # YOUR CODE HERE
common_words_set = set(common_words)
def has_top_words(word_count_vector):
    # extract the keys of word_count_vector and convert it to a set
    unique_words = set(word_count_vector.keys())  # YOUR CODE HERE
    # return True if common_words is a subset of unique_words
    # return False otherwise
    return common_words_set.issubset(unique_words)  # YOUR CODE HERE

wiki['has_top_words'] = wiki['word_count'].apply(has_top_words)

# use has_top_words column to answer the quiz question
len(wiki[wiki['has_top_words']==1])

print 'Output from your function:', has_top_words(wiki[32]['word_count'])
print 'Correct output: True'
print 'Also check the length of unique_words. It should be 167'
print len(wiki[32]['word_count'].keys())

print 'Output from your function:', has_top_words(wiki[33]['word_count'])
print 'Correct output: False'
print 'Also check the length of unique_words. It should be 188'


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Quiz Question. Collect all words that appear both in Barack Obama and George W. Bush pages.  Out of those words, find the 10 words that show up most often in Obama's page.
Can you write Python code for it?

barack_words = top_words('Barack Obama')
bush_words = top_words('George W. Bush')
combined_words = barack_words.join(bush_words, on='word')
combined_words = combined_words.rename({'count':'Obama', 'count.1':'Bush'})
sorted_combined = combined_words.sort('Obama', ascending=False)
sorted_combined['word'][ :5]

barack_words


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Quiz Question. Measure the pairwise distance between the Wikipedia pages of Barack Obama, George W. Bush, and Joe Biden. Which of the three pairs has the smallest distance?
Hint: To compute the Euclidean distance between two dictionaries, use graphlab.toolkits.distances.euclidean. Refer to this link for usage.
Can you write Python code for it?

barack = wiki[wiki['name'] == 'Barack Obama']['word_count'][0]
bush = wiki[wiki['name'] == 'George W. Bush']['word_count'][0]
biden = wiki[wiki['name'] == 'Joe Biden']['word_count'][0]

barack_bush = graphlab.toolkits.distances.euclidean(barack,bush)
barack_bush

barack_biden = graphlab.toolkits.distances.euclidean(barack,biden)
barack_biden

bush_biden = graphlab.toolkits.distances.euclidean(bush,biden)
bush_biden


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Relative to the rest of Wikipedia, nearest neighbors of Obama are overwhemingly short, most of them being shorter than 2000 words. The bias towards short articles is not appropriate in this application as there is really no reason to  favor short articles over long articles (they are all Wikipedia articles, after all). Many Wikipedia articles are 2500 words or more, and both Obama and Biden are over 2500 words long.
Note: Both word-count features and TF-IDF are proportional to word frequencies. While TF-IDF penalizes very common words, longer articles tend to have longer TF-IDF vectors simply because they have more words in them.
To remove this bias, we turn to cosine distances:
$$
d(\mathbf{x},\mathbf{y}) = 1 - \frac{\mathbf{x}^T\mathbf{y}}{\|\mathbf{x}\| \|\mathbf{y}\|}
$$
Cosine distances let us compare word distributions of two articles of varying lengths.
Let us train a new nearest neighbor model, this time with cosine distances.  We then repeat the search for Obama's 100 nearest neighbors.
Can you write Python code for it?

model2_tf_idf = graphlab.nearest_neighbors.create(wiki, label='name', features=['tf_idf'],
                                                  method='brute_force', distance='cosine')

nearest_neighbors_cosine = model2_tf_idf.query(wiki[wiki['name'] == 'Barack Obama'], label='name', k=100)
nearest_neighbors_cosine = nearest_neighbors_cosine.join(wiki[['name', 'length']], on={'reference_label':'name'})

nearest_neighbors_cosine.sort('rank')


I am working on the file "Clustering_&_Retrieval/Week2/Assignment1/0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Since both tables contained the column named count, SFrame automatically renamed one of them to prevent confusion. Let's rename the columns to tell which one is for which. By inspection, we see that the first column (count) is for Obama and the second (count.1) for Barrio.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
All documents that obtain exactly this vector will be assigned to the same bin. We'd like to repeat the identical operation on all documents in the Wikipedia dataset and compute the corresponding bin indices. Again, we use matrix operations  so that no explicit loop is needed.
Can you write Python code for it?

corpus[0:2].dot(random_vectors) >= 0 # compute bit indices of first two documents

corpus.dot(random_vectors) >= 0 # compute bit indices of ALL documents


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
Checkpoint. Running the function with search_radius=0 should yield the list of documents belonging to the same bin as the query.
Can you write Python code for it?

obama_bin_index = model['bin_index_bits'][35817] # bin index of Barack Obama
candidate_set = search_nearby_bins(obama_bin_index, model['table'], search_radius=0)
if candidate_set == set([35817, 21426, 53937, 39426, 50261]):
    print 'Passed test'
else:
    print 'Check your code'
print 'List of documents in the same bin as Obama: 35817, 21426, 53937, 39426, 50261'


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
Checkpoint. Running the function with search_radius=1 adds more documents to the fore.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
Checkpoint: The following code block should return 'Check passed correctly', indicating that your matrix contains TF-IDF values for 59071 documents and 547979 unique words.  Otherwise, it will return Error.
Can you write Python code for it?

assert corpus.shape == (59071, 547979)
print 'Check passed correctly!'


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
For this assignment, let us assign a unique ID to each document.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
How about the documents in the same bin as Barack Obama? Are they necessarily more similar to Obama than Biden?  Let's look at which documents are in the same bin as the Barack Obama article.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
Load in the Wikipedia dataset
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
Next, we partition data points into bins. Instead of using explicit loops, we'd like to utilize matrix operations for greater efficiency. Let's walk through the construction step by step.
We'd like to decide which bin document 0 should go. Since 16 random vectors were generated in the previous cell, we have 16 bits to represent the bin index. The first bit is given by the sign of the dot product between the first random vector and the document's TF-IDF vector.
Can you write Python code for it?

doc = corpus[0, :] # vector of tf-idf values for document 0
doc.dot(random_vectors[:, 0]) >= 0 # True if positive sign; False if negative sign


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
Notice that the top 10 query results become more relevant as the search radius grows. Let's plot the three variables:
Can you write Python code for it?

plt.figure(figsize=(7,4.5))
plt.plot(num_candidates_history, linewidth=4)
plt.xlabel('Search radius')
plt.ylabel('# of documents searched')
plt.rcParams.update({'font.size':16})
plt.tight_layout()

plt.figure(figsize=(7,4.5))
plt.plot(query_time_history, linewidth=4)
plt.xlabel('Search radius')
plt.ylabel('Query time (seconds)')
plt.rcParams.update({'font.size':16})
plt.tight_layout()

plt.figure(figsize=(7,4.5))
plt.plot(average_distance_from_query_history, linewidth=4, label='Average of 10 neighbors')
plt.plot(max_distance_from_query_history, linewidth=4, label='Farthest of 10 neighbors')
plt.plot(min_distance_from_query_history, linewidth=4, label='Closest of 10 neighbors')
plt.xlabel('Search radius')
plt.ylabel('Cosine distance of neighbors')
plt.legend(loc='best', prop={'size':15})
plt.rcParams.update({'font.size':16})
plt.tight_layout()


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
Quiz Question. Examine the bit representations of the bins containing Barack Obama and Joe Biden. In how many places do they agree?

16 out of 16 places (Barack Obama and Joe Biden fall into the same bin)
14 out of 16 places
12 out of 16 places
10 out of 16 places
8 out of 16 places

Compare the result with a former British diplomat, whose bin representation agrees with Obama's in only 8 out of 16 places.
Can you write Python code for it?

wiki[wiki['name']=='Wynn Normington Hugh-Jones']

print np.array(model['bin_index_bits'][22745], dtype=int) # list of 0/1's
print model['bin_indices'][22745] # integer format
model['bin_index_bits'][35817] == model['bin_index_bits'][22745]


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
Quiz Question. What is the document id of Barack Obama's article?
Quiz Question. Which bin contains Barack Obama's article? Enter its integer index.
Recall from the previous assignment that Joe Biden was a close neighbor of Barack Obama.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
Similarly, the second bit is computed as the sign of the dot product between the second random vector and the document vector.
Can you write Python code for it?

doc.dot(random_vectors[:, 1]) >= 0 # True if positive sign; False if negative sign


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
Since it's the dot product again, we batch it with a matrix operation:
Can you write Python code for it?

index_bits = corpus.dot(random_vectors) >= 0
index_bits.dot(powers_of_two)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
The conversion should take a few minutes to complete.
Can you write Python code for it?

start=time.time()
corpus, mapping = sframe_to_scipy(wiki['tf_idf'])
end=time.time()
print end-start


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
There are four other documents that belong to the same bin. Which documents are they?
Can you write Python code for it?

doc_ids = list(model['table'][model['bin_indices'][35817]])
doc_ids.remove(35817) # display documents other than Obama

docs = wiki.filter_by(values=doc_ids, column_name='id') # filter by id column
docs


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
To identify the documents, it's helpful to join this table with the Wikipedia table:
Can you write Python code for it?

query(corpus[35817,:], model, k=10, max_search_radius=3)[0].join(wiki[['id', 'name']], on='id').sort('distance')


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
Train an LSH model
LSH performs an efficient neighbor search by randomly partitioning all reference data points into different bins. Today we will build a popular variant of LSH known as random binary projection, which approximates cosine distance. There are other variants we could use for other choices of distance metrics.
The first step is to generate a collection of random vectors from the standard Gaussian distribution.
Can you write Python code for it?

def generate_random_vectors(num_vector, dim):
    return np.random.randn(dim, num_vector)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
Upgrading to Scipy 0.16.0 or later. This assignment requires SciPy 0.16.0 or later. To upgrade, uncomment and run the following cell:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
We can compute all of the bin index bits at once as follows. Note the absence of the explicit for loop over the 16 vectors. Matrix operations let us batch dot-product computation in a highly efficent manner, unlike the for loop construction. Given the relative inefficiency of loops in Python, the advantage of matrix operations is even greater.
Can you write Python code for it?

doc.dot(random_vectors) >= 0 # should return an array of 16 True/False bits

np.array(doc.dot(random_vectors) >= 0, dtype=int) # display index bits in 0/1's


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
We have shown that we have a working LSH implementation!
Experimenting with your LSH implementation
In the following sections we have implemented a few experiments so that you can gain intuition for how your LSH implementation behaves in different situations. This will help you understand the effect of searching nearby bins and the performance of LSH versus computing nearest neighbors using a brute force search.
Effect of nearby bin search
How does nearby bin search affect the outcome of LSH? There are three variables that are affected by the search radius:
* Number of candidate documents considered
* Query time
* Distance of approximate neighbors from the query
Let us run LSH multiple times, each with different radii for nearby bin search. We will measure the three variables as discussed above.
Can you write Python code for it?

wiki[wiki['name']=='Barack Obama']

num_candidates_history = []
query_time_history = []
max_distance_from_query_history = []
min_distance_from_query_history = []
average_distance_from_query_history = []

for max_search_radius in xrange(17):
    start=time.time()
    result, num_candidates = query(corpus[35817,:], model, k=10,
                                   max_search_radius=max_search_radius)
    end=time.time()
    query_time = end-start

    print 'Radius:', max_search_radius
    print result.join(wiki[['id', 'name']], on='id').sort('distance')

    average_distance_from_query = result['distance'][1:].mean()
    max_distance_from_query = result['distance'][1:].max()
    min_distance_from_query = result['distance'][1:].min()

    num_candidates_history.append(num_candidates)
    query_time_history.append(query_time)
    average_distance_from_query_history.append(average_distance_from_query)
    max_distance_from_query_history.append(max_distance_from_query)
    min_distance_from_query_history.append(min_distance_from_query)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
We now generate random vectors of the same dimensionality as our vocubulary size (547979).  Each vector can be used to compute one bit in the bin encoding.  We generate 16 vectors, leading to a 16-bit encoding of the bin index for each document.
Can you write Python code for it?

# Generate 16 random vectors of dimension 547979
np.random.seed(0)
random_vectors = generate_random_vectors(num_vector=16, dim=547979)
random_vectors.shape


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/.ipynb_checkpoints/1_nearest-neighbors-lsh-implementation_blank-checkpoint.ipynb".
The first task is:
With this output in mind, implement the logic for nearby bin search:
Can you write Python code for it?

def search_nearby_bins(query_bin_bits, table, search_radius=2, initial_candidates=set()):
    """
    For a given query vector and trained LSH model, return all candidate neighbors for
    the query among all bins within the given search radius.

    Example usage
    -------------
    >>> model = train_lsh(corpus, num_vector=16, seed=143)
    >>> q = model['bin_index_bits'][0]  # vector for the first document

    >>> candidates = search_nearby_bins(q, model['table'])
    """
    num_vector = len(query_bin_bits)
    powers_of_two = 1 << np.arange(num_vector-1, -1, -1)

    # Allow the user to provide an initial set of candidates.
    candidate_set = copy(initial_candidates)

    for different_bits in combinations(range(num_vector), search_radius):
        # Flip the bits (n_1,n_2,...,n_r) of the query bin to produce a new bit vector.
        ## Hint: you can iterate over a tuple like a list
        alternate_bits = copy(query_bin_bits)
        for i in different_bits:
            alternate_bits[i] = ... # YOUR CODE HERE

        # Convert the new bit vector to an integer index
        nearby_bin = alternate_bits.dot(powers_of_two)

        # Fetch the list of documents belonging to the bin indexed by the new bit vector.
        # Then add those documents to candidate_set
        # Make sure that the bin exists in the table!
        # Hint: update() method for sets lets you add an entire list to the set
        if nearby_bin in table:
            ... # YOUR CODE HERE: Update candidate_set with the documents in this bin.

    return candidate_set


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
All documents that obtain exactly this vector will be assigned to the same bin. We'd like to repeat the identical operation on all documents in the Wikipedia dataset and compute the corresponding bin indices. Again, we use matrix operations  so that no explicit loop is needed.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Checkpoint.
Can you write Python code for it?

model = train_lsh(corpus, num_vector=16, seed=143)
table = model['table']
if   0 in table and table[0]   == [39583] and \
   143 in table and table[143] == [19693, 28277, 29776, 30399]:
    print 'Passed!'
else:
    print 'Check your code.'


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Checkpoint: The following code block should return 'Check passed correctly', indicating that your matrix contains TF-IDF values for 59071 documents and 547979 unique words.  Otherwise, it will return Error.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Extract TF-IDF matrix
We first use GraphLab Create to compute a TF-IDF representation for each document.
Can you write Python code for it?

wiki['tf_idf'] = graphlab.text_analytics.tf_idf(wiki['text'])
wiki


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
For the remainder of the assignment, we will use sparse matrices. Sparse matrices are [matrices](https://en.wikipedia.org/wiki/Matrix_(mathematics%29 ) that have a small number of nonzero entries. A good data structure for sparse matrices would only store the nonzero entries to save space and speed up computation. SciPy provides a highly-optimized library for sparse matrices. Many matrix operations available for NumPy arrays are also available for SciPy sparse matrices.
We first convert the TF-IDF column (in dictionary format) into the SciPy sparse matrix format.
Can you write Python code for it?

def sframe_to_scipy(column):
    """
    Convert a dict-typed SArray into a SciPy sparse matrix.

    Returns
    -------
        mat : a SciPy sparse matrix where mat[i, j] is the value of word j for document i.
        mapping : a dictionary where mapping[j] is the word whose values are in column j.
    """
    # Create triples of (row_id, feature_id, count).
    x = graphlab.SFrame({'X1':column})

    # 1. Add a row number.
    x = x.add_row_number()
    # 2. Stack will transform x to have a row for each unique (row, key) pair.
    x = x.stack('X1', ['feature', 'value'])

    # Map words into integers using a OneHotEncoder feature transformation.
    f = graphlab.feature_engineering.OneHotEncoder(features=['feature'])

    # We first fit the transformer using the above data.
    f.fit(x)

    # The transform method will add a new column that is the transformed version
    # of the 'word' column.
    x = f.transform(x)

    # Get the feature mapping.
    mapping = f['feature_encoding']

    # Get the actual word id.
    x['feature_id'] = x['encoded_features'].dict_keys().apply(lambda x: x[0])

    # Create numpy arrays that contain the data for the sparse matrix.
    i = np.array(x['id'])
    j = np.array(x['feature_id'])
    v = np.array(x['value'])
    width = x['id'].max() + 1
    height = x['feature_id'].max() + 1

    # Create a sparse matrix.
    mat = csr_matrix((v, (i, j)), shape=(width, height))

    return mat, mapping


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
For this assignment, let us assign a unique ID to each document.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
How about the documents in the same bin as Barack Obama? Are they necessarily more similar to Obama than Biden?  Let's look at which documents are in the same bin as the Barack Obama article.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
It turns out that Joe Biden is much closer to Barack Obama than any of the four documents, even though Biden's bin representation differs from Obama's by 2 bits.
Can you write Python code for it?

def cosine_distance(x, y):
    xy = x.dot(y.T)
    dist = xy/(norm(x)*norm(y))
    return 1-dist[0,0]

obama_tf_idf = corpus[35817,:]
biden_tf_idf = corpus[24478,:]

print '================= Cosine distance from Barack Obama'
print 'Barack Obama - {0:24s}: {1:f}'.format('Joe Biden',
                                             cosine_distance(obama_tf_idf, biden_tf_idf))
for doc_id in doc_ids:
    doc_tf_idf = corpus[doc_id,:]
    print 'Barack Obama - {0:24s}: {1:f}'.format(wiki[doc_id]['name'],
                                                 cosine_distance(obama_tf_idf, doc_tf_idf))


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Let's try it out with Obama:
Can you write Python code for it?

query(corpus[35817,:], model, k=10, max_search_radius=3)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Load in the Wikipedia dataset
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Moral of the story. Similar data points will in general tend to fall into nearby bins, but that's all we can say about LSH. In a high-dimensional space such as text features, we often get unlucky with our selection of only a few random vectors such that dissimilar data points go into the same bin while similar data points fall into different bins. Given a query document, we must consider all documents in the nearby bins and sort them according to their actual distances from the query.
Query the LSH model
Let us first implement the logic for searching nearby neighbors, which goes like this:
1. Let L be the bit representation of the bin that contains the query documents.
2. Consider all documents in bin L.
3. Consider documents in the bins whose bit representation differs from L by 1 bit.
4. Consider documents in the bins whose bit representation differs from L by 2 bits.
...
To obtain candidate bins that differ from the query bin by some number of bits, we use itertools.combinations, which produces all possible subsets of a given list. See this documentation for details.
1. Decide on the search radius r. This will determine the number of different bits between the two vectors.
2. For each subset (n_1, n_2, ..., n_r) of the list [0, 1, 2, ..., num_vector-1], do the following:
   * Flip the bits (n_1, n_2, ..., n_r) of the query bin to produce a new bit vector.
   * Fetch the list of documents belonging to the bin indexed by the new bit vector.
   * Add those documents to the candidate set.
Each line of output from the following cell is a 3-tuple indicating where the candidate bin would differ from the query bin. For instance,
(0, 1, 3)
indicates that the candiate bin differs from the query bin in first, second, and fourth bits.
Can you write Python code for it?

from itertools import combinations

num_vector = 16
search_radius = 3

for diff in combinations(range(num_vector), search_radius):
    print diff


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Next, we partition data points into bins. Instead of using explicit loops, we'd like to utilize matrix operations for greater efficiency. Let's walk through the construction step by step.
We'd like to decide which bin document 0 should go. Since 16 random vectors were generated in the previous cell, we have 16 bits to represent the bin index. The first bit is given by the sign of the dot product between the first random vector and the document's TF-IDF vector.
Can you write Python code for it?

doc = corpus[0, :] # vector of tf-idf values for document 0
doc.dot(random_vectors[:, 0]) >= 0 # True if positive sign; False if negative sign


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Note. We will be using the model trained here in the following sections, unless otherwise indicated.
Inspect bins
Let us look at some documents and see which bins they fall into.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Quiz Question. Examine the bit representations of the bins containing Barack Obama and Joe Biden. In how many places do they agree?

16 out of 16 places (Barack Obama and Joe Biden fall into the same bin)
14 out of 16 places
12 out of 16 places
10 out of 16 places
8 out of 16 places
Can you write Python code for it?

for data_list in model['table']:
    if 24478 in model['table'][data_list]:
        print data_list

print np.array(model['bin_index_bits'][24478], dtype=int) # list of 0/1's
print model['bin_indices'][24478] # integer format
model['bin_index_bits'][35817] == model['bin_index_bits'][24478]


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Recall from the previous assignment that Joe Biden was a close neighbor of Barack Obama.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Similarly, the second bit is computed as the sign of the dot product between the second random vector and the document vector.
Can you write Python code for it?

doc.dot(random_vectors[:, 1]) >= 0 # True if positive sign; False if negative sign


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Since it's the dot product again, we batch it with a matrix operation:
Can you write Python code for it?

index_bits = corpus.dot(random_vectors) >= 0
index_bits.dot(powers_of_two)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Some observations:
* As we increase the search radius, we find more neighbors that are a smaller distance away.
* With increased search radius comes a greater number documents that have to be searched. Query time is higher as a consequence.
* With sufficiently high search radius, the results of LSH begin to resemble the results of brute-force search.
Quiz Question. What was the smallest search radius that yielded the correct nearest neighbor, namely Joe Biden? 2
Quiz Question. Suppose our goal was to produce 10 approximate nearest neighbors whose average distance from the query document is within 0.01 of the average for the true 10 nearest neighbors. For Barack Obama, the true 10 nearest neighbors are on average about 0.77. What was the smallest search radius for Barack Obama that produced an average distance of 0.78 or better? 7
Quality metrics for neighbors
The above analysis is limited by the fact that it was run with a single query, namely Barack Obama. We should repeat the analysis for the entirety of data. Iterating over all documents would take a long time, so let us randomly choose 10 documents for our analysis.
For each document, we first compute the true 25 nearest neighbors, and then run LSH multiple times. We look at two metrics:

Precision@10: How many of the 10 neighbors given by LSH are among the true 25 nearest neighbors?
Average cosine distance of the neighbors from the query

Then we run LSH multiple times with different search radii.
Can you write Python code for it?

def brute_force_query(vec, data, k):
    num_data_points = data.shape[0]

    # Compute distances for ALL data points in training set
    nearest_neighbors = graphlab.SFrame({'id':range(num_data_points)})
    nearest_neighbors['distance'] = pairwise_distances(data, vec, metric='cosine').flatten()

    return nearest_neighbors.topk('distance', k, reverse=True)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
There are four other documents that belong to the same bin. Which documents are they?
Can you write Python code for it?

doc_ids = list(model['table'][model['bin_indices'][35817]])
doc_ids.remove(35817) # display documents other than Obama

docs = wiki.filter_by(values=doc_ids, column_name='id') # filter by id column
docs


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Train an LSH model
LSH performs an efficient neighbor search by randomly partitioning all reference data points into different bins. Today we will build a popular variant of LSH known as random binary projection, which approximates cosine distance. There are other variants we could use for other choices of distance metrics.
The first step is to generate a collection of random vectors from the standard Gaussian distribution.
Can you write Python code for it?

def generate_random_vectors(num_vector, dim):
    return np.random.randn(dim, num_vector)


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
Upgrading to Scipy 0.16.0 or later. This assignment requires SciPy 0.16.0 or later. To upgrade, uncomment and run the following cell:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
We can compute all of the bin index bits at once as follows. Note the absence of the explicit for loop over the 16 vectors. Matrix operations let us batch dot-product computation in a highly efficent manner, unlike the for loop construction. Given the relative inefficiency of loops in Python, the advantage of matrix operations is even greater.
Can you write Python code for it?

doc.dot(random_vectors) >= 0 # should return an array of 16 True/False bits

np.array(doc.dot(random_vectors) >= 0, dtype=int) # display index bits in 0/1's


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
We now generate random vectors of the same dimensionality as our vocubulary size (547979).  Each vector can be used to compute one bit in the bin encoding.  We generate 16 vectors, leading to a 16-bit encoding of the bin index for each document.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week2/Assignment2/1_nearest-neighbors-lsh-implementation_blank.ipynb".
The first task is:
We're almost done! To make it convenient to refer to individual bins, we convert each binary bin index into a single integer:
Bin index                      integer
[0,0,0,0,0,0,0,0,0,0,0,0]   => 0
[0,0,0,0,0,0,0,0,0,0,0,1]   => 1
[0,0,0,0,0,0,0,0,0,0,1,0]   => 2
[0,0,0,0,0,0,0,0,0,0,1,1]   => 3
...
[1,1,1,1,1,1,1,1,1,1,0,0]   => 65532
[1,1,1,1,1,1,1,1,1,1,0,1]   => 65533
[1,1,1,1,1,1,1,1,1,1,1,0]   => 65534
[1,1,1,1,1,1,1,1,1,1,1,1]   => 65535 (= 2^16-1)
By the rules of binary number representation, we just need to compute the dot product between the document vector and the vector consisting of powers of 2:
Can you write Python code for it?

doc = corpus[0, :]  # first document
index_bits = (doc.dot(random_vectors) >= 0)
powers_of_two = (1 << np.arange(15, -1, -1))
print index_bits
print powers_of_two
print index_bits.dot(powers_of_two)


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
After initialization, the k-means algorithm iterates between the following two steps:
1. Assign each data point to the closest centroid.
$$
z_i \gets \mathrm{argmin}j \|\mu_j - \mathbf{x}_i\|^2
$$
2. Revise centroids as the mean of the assigned data points.
$$
\mu_j \gets \frac{1}{n_j}\sum{i:z_i=j} \mathbf{x}_i
$$
In pseudocode, we iteratively do the following:
cluster_assignment = assign_clusters(data, centroids)
centroids = revise_centroids(data, k, cluster_assignment)
Assigning clusters
How do we implement Step 1 of the main k-means loop above? First import pairwise_distances function from scikit-learn, which calculates Euclidean distances between rows of given arrays. See this documentation for more information.
For the sake of demonstration, let's look at documents 100 through 102 as query documents and compute the distances between each of these documents and every other document in the corpus. In the k-means algorithm, we will have to compute pairwise distances between the set of centroids and the set of documents.
Can you write Python code for it?

from sklearn.metrics import pairwise_distances

# Get the TF-IDF vectors for documents 100 through 102.
queries = tf_idf[100:102,:]

# Compute pairwise distances from every data point to each query vector.
dist = pairwise_distances(tf_idf, queries, metric='euclidean')

print dist


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
Both clusters have mixed content, although cluster 1 is much purer than cluster 0:
* Cluster 0: artists, songwriters, professors, politicians, writers, etc.
* Cluster 1: baseball players, hockey players, football (soccer) players, etc.
Top words of cluster 1 are all related to sports, whereas top words of cluster 0 show no clear pattern.
Roughly speaking, the entire dataset was divided into atheletes and non-atheletes. It would be better if we sub-divided non-atheletes into more categories. So let us use more clusters. How about K=10?
Can you write Python code for it?

k = 10
visualize_document_clusters(wiki, tf_idf, centroids[k](), cluster_assignment[k](), k, map_index_to_word)


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
For the remainder of the assignment, we will use sparse matrices. Sparse matrices are matrices that have a small number of nonzero entries. A good data structure for sparse matrices would only store the nonzero entries to save space and speed up computation. SciPy provides a highly-optimized library for sparse matrices. Many matrix operations available for NumPy arrays are also available for SciPy sparse matrices.
We first convert the TF-IDF column (in dictionary format) into the SciPy sparse matrix format. We included plenty of comments for the curious; if you'd like, you may skip the next block and treat the function as a black box.
Can you write Python code for it?

def sframe_to_scipy(x, column_name):
    '''
    Convert a dictionary column of an SFrame into a sparse matrix format where
    each (row_id, column_id, value) triple corresponds to the value of
    x[row_id][column_id], where column_id is a key in the dictionary.

    Example
    >>> sparse_matrix, map_key_to_index = sframe_to_scipy(sframe, column_name)
    '''
    assert x[column_name].dtype() == dict, \
        'The chosen column must be dict type, representing sparse data.'

    # Create triples of (row_id, feature_id, count).
    # 1. Add a row number.
    x = x.add_row_number()
    # 2. Stack will transform x to have a row for each unique (row, key) pair.
    x = x.stack(column_name, ['feature', 'value'])

    # Map words into integers using a OneHotEncoder feature transformation.
    f = graphlab.feature_engineering.OneHotEncoder(features=['feature'])
    # 1. Fit the transformer using the above data.
    f.fit(x)
    # 2. The transform takes 'feature' column and adds a new column 'feature_encoding'.
    x = f.transform(x)
    # 3. Get the feature mapping.
    mapping = f['feature_encoding']
    # 4. Get the feature id to use for each key.
    x['feature_id'] = x['encoded_features'].dict_keys().apply(lambda x: x[0])

    # Create numpy arrays that contain the data for the sparse matrix.
    i = np.array(x['id'])
    j = np.array(x['feature_id'])
    v = np.array(x['value'])
    width = x['id'].max() + 1
    height = x['feature_id'].max() + 1

    # Create a sparse matrix.
    mat = csr_matrix((v, (i, j)), shape=(width, height))

    return mat, mapping

# The conversion will take about a minute or two.
tf_idf, map_index_to_word = sframe_to_scipy(wiki, 'tf_idf')

tf_idf


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
How to choose K
Since we are measuring the tightness of the clusters, a higher value of K reduces the possible heterogeneity metric by definition.  For example, if we have N data points and set K=N clusters, then we could have 0 cluster heterogeneity by setting the N centroids equal to the values of the N data points. (Note: Not all runs for larger K will result in lower heterogeneity than a single run with smaller K due to local optima.)  Let's see explore this general trend for ourselves by performing the following analysis.
Use the kmeans_multiple_runs function to run k-means with five different values of K.  For each K, use k-means++ and multiple runs to pick the best solution.  In what follows, we consider K=2,10,25,50,100 and 7 restarts for each setting.
IMPORTANT: The code block below will take about one hour to finish. We highly suggest that you use the arrays that we have computed for you.
Side note: In practice, a good implementation of k-means would utilize parallelism to run multiple runs of k-means at once. For an example, see scikit-learn's KMeans.
Can you write Python code for it?

#def plot_k_vs_heterogeneity(k_values, heterogeneity_values):
#    plt.figure(figsize=(7,4))
#    plt.plot(k_values, heterogeneity_values, linewidth=4)
#    plt.xlabel('K')
#    plt.ylabel('Heterogeneity')
#    plt.title('K vs. Heterogeneity')
#    plt.rcParams.update({'font.size': 16})
#    plt.tight_layout()

#start = time.time()
#centroids = {}
#cluster_assignment = {}
#heterogeneity_values = []
#k_list = [2, 10, 25, 50, 100]
#seed_list = [0, 20000, 40000, 60000, 80000, 100000, 120000]

#for k in k_list:
#    heterogeneity = []
#    centroids[k], cluster_assignment[k] = kmeans_multiple_runs(tf_idf, k, maxiter=400,
#                                                               num_runs=len(seed_list),
#                                                               seed_list=seed_list,
#                                                               verbose=True)
#    score = compute_heterogeneity(tf_idf, k, centroids[k], cluster_assignment[k])
#    heterogeneity_values.append(score)

#plot_k_vs_heterogeneity(k_list, heterogeneity_values)

#end = time.time()
#print(end-start)


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
In the above plot we show that heterogeneity goes down as we increase the number of clusters. Does this mean we should always favor a higher K? Not at all! As we will see in the following section, setting K too high may end up separating data points that are actually pretty alike. At the extreme, we can set individual data points to be their own clusters (K=N) and achieve zero heterogeneity, but separating each data point into its own cluster is hardly a desirable outcome. In the following section, we will learn how to detect a K set "too large".
Visualize clusters of documents
Let's start visualizing some clustering results to see if we think the clustering makes sense.  We can use such visualizations to help us assess whether we have set K too large or too small for a given application.  Following the theme of this course, we will judge whether the clustering makes sense in the context of document analysis.
What are we looking for in a good clustering of documents?
* Documents in the same cluster should be similar.
* Documents from different clusters should be less similar.
So a bad clustering exhibits either of two symptoms:
* Documents in a cluster have mixed content.
* Documents with similar content are divided up and put into different clusters.
To help visualize the clustering, we do the following:
* Fetch nearest neighbors of each centroid from the set of documents assigned to that cluster. We will consider these documents as being representative of the cluster.
* Print titles and first sentences of those nearest neighbors.
* Print top 5 words that have highest tf-idf weights in each centroid.
Can you write Python code for it?

def visualize_document_clusters(wiki, tf_idf, centroids, cluster_assignment, k, map_index_to_word, display_content=True):
    '''wiki: original dataframe
       tf_idf: data matrix, sparse matrix format
       map_index_to_word: SFrame specifying the mapping betweeen words and column indices
       display_content: if True, display 8 nearest neighbors of each centroid'''

    print('==========================================================')

    # Visualize each cluster c
    for c in xrange(k):
        # Cluster heading
        print('Cluster {0:d}    '.format(c)),
        # Print top 5 words with largest TF-IDF weights in the cluster
        idx = centroids[c].argsort()[::-1]
        for i in xrange(5): # Print each word along with the TF-IDF weight
            print('{0:s}:{1:.3f}'.format(map_index_to_word['category'][idx[i]], centroids[c,idx[i]])),
        print('')

        if display_content:
            # Compute distances from the centroid to all data points in the cluster,
            # and compute nearest neighbors of the centroids within the cluster.
            distances = pairwise_distances(tf_idf, centroids[c].reshape(1, -1), metric='euclidean').flatten()
            distances[cluster_assignment!=c] = float('inf') # remove non-members from consideration
            nearest_neighbors = distances.argsort()
            # For 8 nearest neighbors, print the title as well as first 180 characters of text.
            # Wrap the text at 80-character mark.
            for i in xrange(8):
                text = ' '.join(wiki[nearest_neighbors[i]]['text'].split(None, 25)[0:25])
                print('\n* {0:50s} {1:.5f}\n  {2:s}\n  {3:s}'.format(wiki[nearest_neighbors[i]]['name'],
                    distances[nearest_neighbors[i]], text[:90], text[90:180] if len(text) > 90 else ''))
        print('==========================================================')


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
Let us first look at the 2 cluster case (K=2).
Can you write Python code for it?

'''Notice the extra pairs of parentheses for centroids and cluster_assignment.
   The centroid and cluster_assignment are still inside the npz file,
   and we need to explicitly indicate when to load them into memory.'''
visualize_document_clusters(wiki, tf_idf, centroids[2](), cluster_assignment[2](), 2, map_index_to_word)


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
Let's assign these data points to the closest centroid.
Can you write Python code for it?

cluster_assignment = assign_clusters(data, centroids)
print cluster_assignment


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
Let's compare the set of cluster heterogeneities we got from our 7 restarts of k-means using random initialization compared to the 7 restarts of k-means using k-means++ as a smart initialization.
The following code produces a box plot for each of these methods, indicating the spread of values produced by each method.
Can you write Python code for it?

plt.figure(figsize=(8,5))
plt.boxplot([heterogeneity.values(), heterogeneity_smart.values()], vert=False)
plt.yticks([1, 2], ['k-means', 'k-means++'])
plt.rcParams.update({'font.size': 16})
plt.tight_layout()


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
Let's compute the cluster heterogeneity for the 2-cluster example we've been considering based on our current cluster assignments and centroids.
Can you write Python code for it?

compute_heterogeneity(data, 2, centroids, cluster_assignment)


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
Let's consider running k-means with K=3 clusters for a maximum of 400 iterations, recording cluster heterogeneity at every step.  Then, let's plot the heterogeneity over iterations using the plotting function above.
Can you write Python code for it?

k = 3
heterogeneity = []
initial_centroids = get_initial_centroids(tf_idf, k, seed=0)
centroids, cluster_assignment = kmeans(tf_idf, k, initial_centroids, maxiter=400,
                                       record_heterogeneity=heterogeneity, verbose=True)
plot_heterogeneity(heterogeneity, k)


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
Notice the variation in heterogeneity for different initializations. This indicates that k-means sometimes gets stuck at a bad local minimum.
Quiz Question. Another way to capture the effect of changing initialization is to look at the distribution of cluster assignments. Add a line to the code above to compute the size (# of member data points) of clusters for each run of k-means. Look at the size of the largest cluster (most # of member data points) across multiple runs, with seeds 0, 20000, ..., 120000. How much does this measure vary across the runs? What is the minimum and maximum values this quantity takes?
One effective way to counter this tendency is to use k-means++ to provide a smart initialization. This method tries to spread out the initial set of centroids so that they are not too close together. It is known to improve the quality of local optima and lower average runtime.
Can you write Python code for it?

def smart_initialize(data, k, seed=None):
    '''Use k-means++ to initialize a good set of centroids'''
    if seed is not None: # useful for obtaining consistent results
        np.random.seed(seed)
    centroids = np.zeros((k, data.shape[1]))

    # Randomly choose the first centroid.
    # Since we have no prior knowledge, choose uniformly at random
    idx = np.random.randint(data.shape[0])
    centroids[0] = data[idx,:].toarray()
    # Compute distances from the first centroid chosen to all the other data points
    distances = pairwise_distances(data, centroids[0:1], metric='euclidean').flatten()

    for i in xrange(1, k):
        # Choose the next centroid randomly, so that the probability for each data point to be chosen
        # is directly proportional to its squared distance from the nearest centroid.
        # Roughtly speaking, a new centroid should be as far as from ohter centroids as possible.
        idx = np.random.choice(data.shape[0], 1, p=distances/sum(distances))
        centroids[i] = data[idx,:].toarray()
        # Now compute distances from the centroids to all data points
        distances = np.min(pairwise_distances(data, centroids[0:i+1], metric='euclidean'),axis=1)

    return centroids


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
Now we are ready to fill in the blanks in this function:
Can you write Python code for it?

def assign_clusters(data, centroids):

    # Compute distances between each data point and the set of centroids:
    # Fill in the blank (RHS only)
    distances_from_centroids = ...

    # Compute cluster assignments for each data point:
    # Fill in the blank (RHS only)
    cluster_assignment = ...

    return cluster_assignment


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
Plotting convergence metric
We can use the above function to plot the convergence metric across iterations.
Can you write Python code for it?

def plot_heterogeneity(heterogeneity, k):
    plt.figure(figsize=(7,4))
    plt.plot(heterogeneity, linewidth=4)
    plt.xlabel('# Iterations')
    plt.ylabel('Heterogeneity')
    plt.title('Heterogeneity of clustering over time, K={0:d}'.format(k))
    plt.rcParams.update({'font.size': 16})
    plt.tight_layout()


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
Quiz Question. (True/False) The clustering objective (heterogeneity) is non-increasing for this example.
Quiz Question. Let's step back from this particular example. If the clustering objective (heterogeneity) would ever increase when running k-means, that would indicate: (choose one)

k-means algorithm got stuck in a bad local minimum
There is a bug in the k-means code
All data points consist of exact duplicates
Nothing is wrong. The objective should generally go down sooner or later.

Quiz Question. Which of the cluster contains the greatest number of data points in the end? Hint: Use np.bincount() to count occurrences of each cluster label.
 1. Cluster #0
 2. Cluster #1
 3. Cluster #2
Beware of local maxima
One weakness of k-means is that it tends to get stuck in a local minimum. To see this, let us run k-means multiple times, with different initial centroids created using different random seeds.
Note: Again, in practice, you should set different seeds for every run. We give you a list of seeds for this assignment so that everyone gets the same answer.
This may take several minutes to run.
Can you write Python code for it?

k = 10
heterogeneity = {}
import time
start = time.time()
for seed in [0, 20000, 40000, 60000, 80000, 100000, 120000]:
    initial_centroids = get_initial_centroids(tf_idf, k, seed)
    centroids, cluster_assignment = kmeans(tf_idf, k, initial_centroids, maxiter=400,
                                           record_heterogeneity=None, verbose=False)
    # To save time, compute heterogeneity only once in the end
    heterogeneity[seed] = compute_heterogeneity(tf_idf, k, centroids, cluster_assignment)
    print('seed={0:06d}, heterogeneity={1:.5f}'.format(seed, heterogeneity[seed]))
    sys.stdout.flush()
end = time.time()
print(end-start)


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
This makes sense since [0 0 0] is closer to [0 -0.5 0] than to [0.5 0.5 0].
Now let's look at the data points assigned to cluster 0:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week3/.ipynb_checkpoints/2_kmeans-with-text-data_blank-checkpoint.ipynb".
The first task is:
To use the pre-computed NumPy arrays, first download kmeans-arrays.npz as mentioned in the reading for this assignment and load them with the following code.  Make sure the downloaded file is in the same directory as this notebook.
Can you write Python code for it?

def plot_k_vs_heterogeneity(k_values, heterogeneity_values):
    plt.figure(figsize=(7,4))
    plt.plot(k_values, heterogeneity_values, linewidth=4)
    plt.xlabel('K')
    plt.ylabel('Heterogeneity')
    plt.title('K vs. Heterogeneity')
    plt.rcParams.update({'font.size': 16})
    plt.tight_layout()

filename = 'kmeans-arrays.npz'

heterogeneity_values = []
k_list = [2, 10, 25, 50, 100]

if os.path.exists(filename):
    arrays = np.load(filename)
    centroids = {}
    cluster_assignment = {}
    for k in k_list:
        print k
        sys.stdout.flush()
        '''To save memory space, do not load the arrays from the file right away. We use
           a technique known as lazy evaluation, where some expressions are not evaluated
           until later. Any expression appearing inside a lambda function doesn't get
           evaluated until the function is called.
           Lazy evaluation is extremely important in memory-constrained setting, such as
           an Amazon EC2 t2.micro instance.'''
        centroids[k] = lambda k=k: arrays['centroids_{0:d}'.format(k)]
        cluster_assignment[k] = lambda k=k: arrays['cluster_assignment_{0:d}'.format(k)]
        score = compute_heterogeneity(tf_idf, k, centroids[k](), cluster_assignment[k]())
        heterogeneity_values.append(score)

    plot_k_vs_heterogeneity(k_list, heterogeneity_values)

else:
    print('File not found. Skipping.')


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
A few things to notice from the box plot:
* Random initialization results in a worse clustering than k-means++ on average.
* The best result of k-means++ is better than the best result of random initialization.
In general, you should run k-means at least a few times with different initializations and then return the run resulting in the lowest heterogeneity. Let us write a function that runs k-means multiple times and picks the best run that minimizes heterogeneity. The function accepts an optional list of seed values to be used for the multiple runs; if no such list is provided, the current UTC time is used as seed values.
Can you write Python code for it?

def kmeans_multiple_runs(data, k, maxiter, num_runs, seed_list=None, verbose=False):
    heterogeneity = {}

    min_heterogeneity_achieved = float('inf')
    best_seed = None
    final_centroids = None
    final_cluster_assignment = None

    for i in xrange(num_runs):

        # Use UTC time if no seeds are provided
        if seed_list is not None:
            seed = seed_list[i]
            np.random.seed(seed)
        else:
            seed = int(time.time())
            np.random.seed(seed)

        # Use k-means++ initialization
        # YOUR CODE HERE
        initial_centroids = smart_initialize(data, k, seed)

        # Run k-means
        # YOUR CODE HERE
        centroids, cluster_assignment = kmeans(data, k, initial_centroids, maxiter, heterogeneity, verbose)

        # To save time, compute heterogeneity only once in the end
        # YOUR CODE HERE
        heterogeneity[seed] = compute_heterogeneity(data, k, centroids, cluster_assignment)

        if verbose:
            print('seed={0:06d}, heterogeneity={1:.5f}'.format(seed, heterogeneity[seed]))
            sys.stdout.flush()

        # if current measurement of heterogeneity is lower than previously seen,
        # update the minimum record of heterogeneity.
        if heterogeneity[seed] < min_heterogeneity_achieved:
            min_heterogeneity_achieved = heterogeneity[seed]
            best_seed = seed
            final_centroids = centroids
            final_cluster_assignment = cluster_assignment

    # Return the centroids and cluster assignments that minimize heterogeneity.
    return final_centroids, final_cluster_assignment


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
After initialization, the k-means algorithm iterates between the following two steps:
1. Assign each data point to the closest centroid.
$$
z_i \gets \mathrm{argmin}j \|\mu_j - \mathbf{x}_i\|^2
$$
2. Revise centroids as the mean of the assigned data points.
$$
\mu_j \gets \frac{1}{n_j}\sum{i:z_i=j} \mathbf{x}_i
$$
In pseudocode, we iteratively do the following:
cluster_assignment = assign_clusters(data, centroids)
centroids = revise_centroids(data, k, cluster_assignment)
Assigning clusters
How do we implement Step 1 of the main k-means loop above? First import pairwise_distances function from scikit-learn, which calculates Euclidean distances between rows of given arrays. See this documentation for more information.
For the sake of demonstration, let's look at documents 100 through 102 as query documents and compute the distances between each of these documents and every other document in the corpus. In the k-means algorithm, we will have to compute pairwise distances between the set of centroids and the set of documents.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Again, this makes sense since these values are each closer to [0.5 0.5 0] than to [0 -0.5 0].
Given all the data points in a cluster, it only remains to compute the mean. Use np.mean(). By default, the function averages all elements in a 2D array. To compute row-wise or column-wise means, add the axis argument. See the linked documentation for details.
Use this function to average the data points in cluster 0:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Beware of local maxima
One weakness of k-means is that it tends to get stuck in a local minimum. To see this, let us run k-means multiple times, with different initial centroids created using different random seeds.
Note: Again, in practice, you should set different seeds for every run. We give you a list of seeds for this assignment so that everyone gets the same answer.
This may take several minutes to run.
Can you write Python code for it?

k = 10
heterogeneity = {}
import time
start = time.time()
for seed in [0, 20000, 40000, 60000, 80000, 100000, 120000]:
    initial_centroids = get_initial_centroids(tf_idf, k, seed)
    centroids, cluster_assignment = kmeans(tf_idf, k, initial_centroids, maxiter=400,
                                           record_heterogeneity=None, verbose=False)
    print np.bincount(cluster_assignment)
    # To save time, compute heterogeneity only once in the end
    heterogeneity[seed] = compute_heterogeneity(tf_idf, k, centroids, cluster_assignment)
    print('seed={0:06d}, heterogeneity={1:.5f}'.format(seed, heterogeneity[seed]))
    sys.stdout.flush()
end = time.time()
print(end-start)


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Checkpoint. For the last time, let us check if Step 1 was implemented correctly. With rows 0, 2, 4, and 6 of tf_idf as an initial set of centroids, we assign cluster labels to rows 0, 10, 20, ..., and 90 of tf_idf. The resulting cluster labels should be [0, 1, 1, 0, 0, 2, 0, 2, 2, 1].
Can you write Python code for it?

if np.allclose(assign_clusters(tf_idf[0:100:10], tf_idf[0:8:2]), np.array([0, 1, 1, 0, 0, 2, 0, 2, 2, 1])):
    print('Pass')
else:
    print('Check your code again.')


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Checkpoint. Let's check our Step 2 implementation. Letting rows 0, 10, ..., 90 of tf_idf as the data points and the cluster labels [0, 1, 1, 0, 0, 2, 0, 2, 2, 1], we compute the next set of centroids. Each centroid is given by the average of all member data points in corresponding cluster.
Can you write Python code for it?

result = revise_centroids(tf_idf[0:100:10], 3, np.array([0, 1, 1, 0, 0, 2, 0, 2, 2, 1]))
if np.allclose(result[0], np.mean(tf_idf[[0,30,40,60]].toarray(), axis=0)) and \
   np.allclose(result[1], np.mean(tf_idf[[10,20,90]].toarray(), axis=0))   and \
   np.allclose(result[2], np.mean(tf_idf[[50,70,80]].toarray(), axis=0)):
    print('Pass')
else:
    print('Check your code')


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Checkpoint: Let's put these steps together.  First, initialize three centroids with the first 3 rows of tf_idf. Then, compute distances from each of the centroids to all data points in tf_idf. Finally, use these distance calculations to compute cluster assignments and assign them to cluster_assignment.
Can you write Python code for it?

# Students should write code here
cluster_assignment = closest_cluster

if len(cluster_assignment)==59071 and \
   np.array_equal(np.bincount(cluster_assignment), np.array([23061, 10086, 25924])):
    print('Pass') # count number of data points for each cluster
else:
    print('Check your code again.')


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Cluters 0 and 2 appear to be still mixed, but others are quite consistent in content.
* Cluster 0: artists, poets, writers, environmentalists
* Cluster 1: film directors
* Cluster 2: female figures from various fields
* Cluster 3: politicians
* Cluster 4: track and field athletes
* Cluster 5: composers, songwriters, singers, music producers
* Cluster 6: soccer (football) players
* Cluster 7: baseball players
* Cluster 8: professors, researchers, scholars
* Cluster 9: lawyers, judges, legal scholars
Clusters are now more pure, but some are qualitatively "bigger" than others. For instance, the category of scholars is more general than the category of baseball players. Increasing the number of clusters may split larger clusters. Another way to look at the size of cluster is to count the number of articles in each cluster.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Combining into a single function
Once the two k-means steps have been implemented, as well as our heterogeneity metric we wish to monitor, it is only a matter of putting these functions together to write a k-means algorithm that

Repeatedly performs Steps 1 and 2
Tracks convergence metrics
Stops if either no assignment changed or we reach a certain number of iterations.
Can you write Python code for it?

# Fill in the blanks
def kmeans(data, k, initial_centroids, maxiter, record_heterogeneity=None, verbose=False):
    '''This function runs k-means on given data and initial set of centroids.
       maxiter: maximum number of iterations to run.
       record_heterogeneity: (optional) a list, to store the history of heterogeneity as function of iterations
                             if None, do not store the history.
       verbose: if True, print how many data points changed their cluster labels in each iteration'''
    centroids = initial_centroids[:]
    prev_cluster_assignment = None

    for itr in xrange(maxiter):
        if verbose:
            print(itr)

        # 1. Make cluster assignments using nearest centroids
        # YOUR CODE HERE
        cluster_assignment = assign_clusters(data, centroids)

        # 2. Compute a new centroid for each of the k clusters, averaging all data points assigned to that cluster.
        # YOUR CODE HERE
        centroids = revise_centroids(data, k, cluster_assignment)

        # Check for convergence: if none of the assignments changed, stop
        if prev_cluster_assignment is not None and \
          (prev_cluster_assignment==cluster_assignment).all():
            break

        # Print number of new assignments
        if prev_cluster_assignment is not None:
            num_changed = sum(abs(prev_cluster_assignment-cluster_assignment))
            if verbose:
                print('    {0:5d} elements changed their cluster assignment.'.format(num_changed))

        # Record heterogeneity convergence metric
        if record_heterogeneity is not None:
            # YOUR CODE HERE
            score = compute_heterogeneity(data, k, centroids, cluster_assignment)
            record_heterogeneity.append(score)

        prev_cluster_assignment = cluster_assignment[:]

    return centroids, cluster_assignment


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
For the remainder of the assignment, we will use sparse matrices. Sparse matrices are matrices that have a small number of nonzero entries. A good data structure for sparse matrices would only store the nonzero entries to save space and speed up computation. SciPy provides a highly-optimized library for sparse matrices. Many matrix operations available for NumPy arrays are also available for SciPy sparse matrices.
We first convert the TF-IDF column (in dictionary format) into the SciPy sparse matrix format. We included plenty of comments for the curious; if you'd like, you may skip the next block and treat the function as a black box.
Can you write Python code for it?

def sframe_to_scipy(x, column_name):
    '''
    Convert a dictionary column of an SFrame into a sparse matrix format where
    each (row_id, column_id, value) triple corresponds to the value of
    x[row_id][column_id], where column_id is a key in the dictionary.

    Example
    >>> sparse_matrix, map_key_to_index = sframe_to_scipy(sframe, column_name)
    '''
    assert x[column_name].dtype() == dict, \
        'The chosen column must be dict type, representing sparse data.'

    # Create triples of (row_id, feature_id, count).
    # 1. Add a row number.
    x = x.add_row_number()
    # 2. Stack will transform x to have a row for each unique (row, key) pair.
    x = x.stack(column_name, ['feature', 'value'])

    # Map words into integers using a OneHotEncoder feature transformation.
    f = graphlab.feature_engineering.OneHotEncoder(features=['feature'])
    # 1. Fit the transformer using the above data.
    f.fit(x)
    # 2. The transform takes 'feature' column and adds a new column 'feature_encoding'.
    x = f.transform(x)
    # 3. Get the feature mapping.
    mapping = f['feature_encoding']
    # 4. Get the feature id to use for each key.
    x['feature_id'] = x['encoded_features'].dict_keys().apply(lambda x: x[0])

    # Create numpy arrays that contain the data for the sparse matrix.
    i = np.array(x['id'])
    j = np.array(x['feature_id'])
    v = np.array(x['value'])
    width = x['id'].max() + 1
    height = x['feature_id'].max() + 1

    # Create a sparse matrix.
    mat = csr_matrix((v, (i, j)), shape=(width, height))

    return mat, mapping

# The conversion will take about a minute or two.
tf_idf, map_index_to_word = sframe_to_scipy(wiki, 'tf_idf')

tf_idf


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
How to choose K
Since we are measuring the tightness of the clusters, a higher value of K reduces the possible heterogeneity metric by definition.  For example, if we have N data points and set K=N clusters, then we could have 0 cluster heterogeneity by setting the N centroids equal to the values of the N data points. (Note: Not all runs for larger K will result in lower heterogeneity than a single run with smaller K due to local optima.)  Let's see explore this general trend for ourselves by performing the following analysis.
Use the kmeans_multiple_runs function to run k-means with five different values of K.  For each K, use k-means++ and multiple runs to pick the best solution.  In what follows, we consider K=2,10,25,50,100 and 7 restarts for each setting.
IMPORTANT: The code block below will take about one hour to finish. We highly suggest that you use the arrays that we have computed for you.
Side note: In practice, a good implementation of k-means would utilize parallelism to run multiple runs of k-means at once. For an example, see scikit-learn's KMeans.
Can you write Python code for it?

#def plot_k_vs_heterogeneity(k_values, heterogeneity_values):
#    plt.figure(figsize=(7,4))
#    plt.plot(k_values, heterogeneity_values, linewidth=4)
#    plt.xlabel('K')
#    plt.ylabel('Heterogeneity')
#    plt.title('K vs. Heterogeneity')
#    plt.rcParams.update({'font.size': 16})
#    plt.tight_layout()

#start = time.time()
#centroids = {}
#cluster_assignment = {}
#heterogeneity_values = []
#k_list = [2, 10, 25, 50, 100]
#seed_list = [0, 20000, 40000, 60000, 80000, 100000, 120000]

#for k in k_list:
#    heterogeneity = []
#    centroids[k], cluster_assignment[k] = kmeans_multiple_runs(tf_idf, k, maxiter=400,
#                                                               num_runs=len(seed_list),
#                                                               seed_list=seed_list,
#                                                               verbose=True)
#    score = compute_heterogeneity(tf_idf, k, centroids[k], cluster_assignment[k])
#    heterogeneity_values.append(score)

#plot_k_vs_heterogeneity(k_list, heterogeneity_values)

#end = time.time()
#print(end-start)


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Implement k-means
Let us implement the k-means algorithm. First, we choose an initial set of centroids. A common practice is to choose randomly from the data points.
Note: We specify a seed here, so that everyone gets the same answer. In practice, we highly recommend to use different seeds every time (for instance, by using the current timestamp).
Can you write Python code for it?

def get_initial_centroids(data, k, seed=None):
    '''Randomly choose k data points as initial centroids'''
    if seed is not None: # useful for obtaining consistent results
        np.random.seed(seed)
    n = data.shape[0] # number of data points

    # Pick K indices from range [0, N).
    rand_indices = np.random.randint(0, n, k)

    # Keep centroids as dense format, as many entries will be nonzero due to averaging.
    # As long as at least one document in a cluster contains a word,
    # it will carry a nonzero weight in the TF-IDF vector of the centroid.
    centroids = data[rand_indices,:].toarray()

    return centroids


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
In lieu of indices, we can put in the list of Booleans to pick and choose rows. Only the rows that correspond to a True entry will be retained.
First, let's look at the data points (i.e., their values) assigned to cluster 1:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Let's compare the set of cluster heterogeneities we got from our 7 restarts of k-means using random initialization compared to the 7 restarts of k-means using k-means++ as a smart initialization.
The following code produces a box plot for each of these methods, indicating the spread of values produced by each method.
Can you write Python code for it?

plt.figure(figsize=(8,5))
plt.boxplot([heterogeneity.values(), heterogeneity_smart.values()], vert=False)
plt.yticks([1, 2], ['k-means', 'k-means++'])
plt.rcParams.update({'font.size': 16})
plt.tight_layout()


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Let's compute the cluster heterogeneity for the 2-cluster example we've been considering based on our current cluster assignments and centroids.
Can you write Python code for it?

compute_heterogeneity(data, 2, centroids, cluster_assignment)


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Let's now rerun k-means with 10 clusters using the same set of seeds, but always using k-means++ to initialize the algorithm.
This may take several minutes to run.
Can you write Python code for it?

k = 10
heterogeneity_smart = {}
start = time.time()
for seed in [0, 20000, 40000, 60000, 80000, 100000, 120000]:
    initial_centroids = smart_initialize(tf_idf, k, seed)
    centroids, cluster_assignment = kmeans(tf_idf, k, initial_centroids, maxiter=400,
                                           record_heterogeneity=None, verbose=False)
    #print np.bincount(cluster_assignment)
    # To save time, compute heterogeneity only once in the end
    heterogeneity_smart[seed] = compute_heterogeneity(tf_idf, k, centroids, cluster_assignment)
    print('seed={0:06d}, heterogeneity={1:.5f}'.format(seed, heterogeneity_smart[seed]))
    sys.stdout.flush()
end = time.time()
print(end-start)


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Likewise for cluster 0:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Load data, extract features
To work with text data, we must first convert the documents into numerical features. As in the first assignment, let's extract TF-IDF features for each article.
Can you write Python code for it?

wiki = graphlab.SFrame('people_wiki.gl/')

wiki['tf_idf'] = graphlab.text_analytics.tf_idf(wiki['text'])


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Looking at the representative examples and top words, we classify each cluster as follows. Notice the bolded items, which indicate the appearance of a new theme.
* Cluster 0: composers, songwriters, singers, music producers
* Cluster 1: poets
* Cluster 2: rugby players
* Cluster 3: baseball players
* Cluster 4: government officials
* Cluster 5: football players
* Cluster 6: radio hosts
* Cluster 7: actors, TV directors
* Cluster 8: professors, researchers, scholars
* Cluster 9: lawyers, judges, legal scholars
* Cluster 10: track and field athletes
* Cluster 11: (mixed; no clear theme)
* Cluster 12: car racers
* Cluster 13: priets, bishops, church leaders
* Cluster 14: painters, sculptors, artists
* Cluster 15: novelists
* Cluster 16: American football players
* Cluster 17: golfers
* Cluster 18: American politicians
* Cluster 19: basketball players
* Cluster 20: generals of U.S. Air Force
* Cluster 21: politicians
* Cluster 22: female figures of various fields
* Cluster 23: film directors
* Cluster 24: music directors, composers, conductors
Indeed, increasing K achieved the desired effect of breaking up large clusters.  Depending on the application, this may or may not be preferable to the K=10 analysis.
Let's take it to the extreme and set K=100. We have a suspicion that this value is too large. Let us look at the top words from each cluster:
Can you write Python code for it?

k=100
visualize_document_clusters(wiki, tf_idf, centroids[k](), cluster_assignment[k](), k,
                            map_index_to_word, display_content=False)
# turn off text for brevity -- turn it on if you are curious ;)


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
More formally, dist[i,j] is assigned the distance between the ith row of X (i.e., X[i,:]) and the jth row of Y (i.e., Y[j,:]).
Checkpoint: For a moment, suppose that we initialize three centroids with the first 3 rows of tf_idf. Write code to compute distances from each of the centroids to all data points in tf_idf. Then find the distance between row 430 of tf_idf and the second centroid and save it to dist.
Can you write Python code for it?

# Students should write code here
queries = tf_idf[0:3]

# Compute pairwise distances from every data point to each query vector.
distances = pairwise_distances(tf_idf, queries, metric='euclidean')
dist=pairwise_distances(tf_idf[430:431], queries[1], metric='euclidean')
print dist

'''Test cell'''
if np.allclose(dist, pairwise_distances(tf_idf[430,:], tf_idf[1,:])):
    print('Pass')
else:
    print('Check your code again')


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Now we are ready to fill in the blanks in this function:
Can you write Python code for it?

def assign_clusters(data, centroids):

    # Compute distances between each data point and the set of centroids:
    # Fill in the blank (RHS only)
    distances_from_centroids = pairwise_distances(data, centroids, metric='euclidean')

    # Compute cluster assignments for each data point:
    # Fill in the blank (RHS only)
    cluster_assignment = np.argmin(distances_from_centroids,axis=1)

    return cluster_assignment


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Plotting convergence metric
We can use the above function to plot the convergence metric across iterations.
Can you write Python code for it?

def plot_heterogeneity(heterogeneity, k):
    plt.figure(figsize=(7,4))
    plt.plot(heterogeneity, linewidth=4)
    plt.xlabel('# Iterations')
    plt.ylabel('Heterogeneity')
    plt.title('Heterogeneity of clustering over time, K={0:d}'.format(k))
    plt.rcParams.update({'font.size': 16})
    plt.tight_layout()


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Quiz Question. (True/False) The clustering objective (heterogeneity) is non-increasing for this example. True
Quiz Question. Let's step back from this particular example. If the clustering objective (heterogeneity) would ever increase when running k-means, that would indicate: (choose one)

k-means algorithm got stuck in a bad local minimum <-
There is a bug in the k-means code
All data points consist of exact duplicates
Nothing is wrong. The objective should generally go down sooner or later.

Quiz Question. Which of the cluster contains the greatest number of data points in the end? Hint: Use np.bincount() to count occurrences of each cluster label.
 1. Cluster #0
 2. Cluster #1
 3. Cluster #2
Can you write Python code for it?