I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
Revising clusters
Let's turn to Step 2, where we compute the new centroids given the cluster assignments.
SciPy and NumPy arrays allow for filtering via Boolean masks. For instance, we filter all data points that are assigned to cluster 0 by writing
data[cluster_assignment==0,:]
To develop intuition about filtering, let's look at a toy example consisting of 3 data points and 2 clusters.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
The above matrix contains a TF-IDF score for each of the 59071 pages in the data set and each of the 547979 unique words.
Normalize all vectors
As discussed in the previous assignment, Euclidean distance can be a poor metric of similarity between documents, as it unfairly penalizes long articles. For a reasonable assessment of similarity, we should disregard the length information and use length-agnostic metrics, such as cosine distance.
The k-means algorithm does not directly work with cosine distance, so we take an alternative route to remove length information: we normalize all vectors to be unit length. It turns out that Euclidean distance closely mimics cosine distance when all vectors are unit length. In particular, the squared Euclidean distance between any two vectors of length one is directly proportional to their cosine distance.
We can prove this as follows. Let $\mathbf{x}$ and $\mathbf{y}$ be normalized vectors, i.e. unit vectors, so that $\|\mathbf{x}\|=\|\mathbf{y}\|=1$. Write the squared Euclidean distance as the dot product of $(\mathbf{x} - \mathbf{y})$ to itself:
\begin{align}
\|\mathbf{x} - \mathbf{y}\|^2 &= (\mathbf{x} - \mathbf{y})^T(\mathbf{x} - \mathbf{y})\
                              &= (\mathbf{x}^T \mathbf{x}) - 2(\mathbf{x}^T \mathbf{y}) + (\mathbf{y}^T \mathbf{y})\
                              &= \|\mathbf{x}\|^2 - 2(\mathbf{x}^T \mathbf{y}) + \|\mathbf{y}\|^2\
                              &= 2 - 2(\mathbf{x}^T \mathbf{y})\
                              &= 2(1 - (\mathbf{x}^T \mathbf{y}))\
                              &= 2\left(1 - \frac{\mathbf{x}^T \mathbf{y}}{\|\mathbf{x}\|\|\mathbf{y}\|}\right)\
                              &= 2\left[\text{cosine distance}\right]
\end{align}
This tells us that two unit vectors that are close in Euclidean distance are also close in cosine distance. Thus, the k-means algorithm (which naturally uses Euclidean distances) on normalized vectors will produce the same results as clustering using cosine distance as a distance metric.
We import...
Can you write Python code for it?

from sklearn.preprocessing import normalize
tf_idf = normalize(tf_idf)


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
This makes sense since [0 0 0] is closer to [0 -0.5 0] than to [0.5 0.5 0].
Now let's look at the data points assigned to cluster 0:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week3/2_kmeans-with-text-data_blank.ipynb".
The first task is:
To use the pre-computed NumPy arrays, first download kmeans-arrays.npz as mentioned in the reading for this assignment and load them with the following code.  Make sure the downloaded file is in the same directory as this notebook.
Can you write Python code for it?

def plot_k_vs_heterogeneity(k_values, heterogeneity_values):
    plt.figure(figsize=(7,4))
    plt.plot(k_values, heterogeneity_values, linewidth=4)
    plt.xlabel('K')
    plt.ylabel('Heterogeneity')
    plt.title('K vs. Heterogeneity')
    plt.rcParams.update({'font.size': 16})
    plt.tight_layout()

filename = 'kmeans-arrays.npz'

heterogeneity_values = []
k_list = [2, 10, 25, 50, 100]

if os.path.exists(filename):
    arrays = np.load(filename)
    centroids = {}
    cluster_assignment = {}
    for k in k_list:
        print k
        sys.stdout.flush()
        '''To save memory space, do not load the arrays from the file right away. We use
           a technique known as lazy evaluation, where some expressions are not evaluated
           until later. Any expression appearing inside a lambda function doesn't get
           evaluated until the function is called.
           Lazy evaluation is extremely important in memory-constrained setting, such as
           an Amazon EC2 t2.micro instance.'''
        centroids[k] = lambda k=k: arrays['centroids_{0:d}'.format(k)]
        cluster_assignment[k] = lambda k=k: arrays['cluster_assignment_{0:d}'.format(k)]
        score = compute_heterogeneity(tf_idf, k, centroids[k](), cluster_assignment[k]())
        heterogeneity_values.append(score)

    plot_k_vs_heterogeneity(k_list, heterogeneity_values)

else:
    print('File not found. Skipping.')


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
Checkpoint. For this particular example, the EM algorithm is expected to terminate in 30 iterations. That is, the last line of the log should say "Iteration 29". If your function stopped too early or too late, you should re-visit your code.
Our algorithm returns a dictionary with five elements:
* 'loglik': a record of the log likelihood at each iteration
* 'resp': the final responsibility matrix
* 'means': a list of K means
* 'covs': a list of K covariance matrices
* 'weights': the weights corresponding to each model component
Quiz Question: What is the weight that EM assigns to the first component after running the above codeblock?
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
Checkpoint: To verify your implementation above, make sure the following code does not return an error.
Can you write Python code for it?

assert len(data) == 100
assert len(data[0]) == 2
print 'Checkpoint passed!'


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
Evaluating uncertainty
Next we'll explore the evolution of cluster assignment and uncertainty. Remember that the EM algorithm represents uncertainty about the cluster assignment of each data point through the responsibility matrix. Rather than making a 'hard' assignment of each data point to a single cluster, the algorithm computes the responsibility of each cluster for each data point, where the responsibility corresponds to our certainty that the observation came from that cluster.
We can track the evolution of the responsibilities across iterations to see how these 'soft' cluster assignments change as the algorithm fits the Gaussian mixture model to the data; one good way to do this is to plot the data and color each point according to its cluster responsibilities. Our data are three-dimensional, which can make visualization difficult, so to make things easier we will plot the data using only two dimensions, taking just the [R G], [G B] or [R B] values instead of the full [R G B] measurement for each observation.
Can you write Python code for it?

import colorsys
def plot_responsibilities_in_RB(img, resp, title):
    N, K = resp.shape

    HSV_tuples = [(x*1.0/K, 0.5, 0.9) for x in range(K)]
    RGB_tuples = map(lambda x: colorsys.hsv_to_rgb(*x), HSV_tuples)

    R = img['red']
    B = img['blue']
    resp_by_img_int = [[resp[n][k] for k in range(K)] for n in range(N)]
    cols = [tuple(np.dot(resp_by_img_int[n], np.array(RGB_tuples))) for n in range(N)]

    plt.figure()
    for n in range(len(R)):
        plt.plot(R[n], B[n], 'o', c=cols[n])
    plt.title(title)
    plt.xlabel('R value')
    plt.ylabel('B value')
    plt.rcParams.update({'font.size':16})
    plt.tight_layout()


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
Implementation
You will now complete an implementation that can run EM on the data you just created. It uses the loglikelihood function we provided above.
Fill in the places where you find ## YOUR CODE HERE. There are seven places in this function for you to fill in.
Hint: Some useful functions

multivariate_normal.pdf: lets you compute the likelihood of seeing a data point in a multivariate Gaussian distribution.
np.outer: comes in handy when estimating the covariance matrix from data.
Can you write Python code for it?

def EM(data, init_means, init_covariances, init_weights, maxiter=1000, thresh=1e-4):

    # Make copies of initial parameters, which we will update during each iteration
    means = init_means[:]
    covariances = init_covariances[:]
    weights = init_weights[:]

    # Infer dimensions of dataset and the number of clusters
    num_data = len(data)
    num_dim = len(data[0])
    num_clusters = len(means)

    # Initialize some useful variables
    resp = np.zeros((num_data, num_clusters))
    ll = loglikelihood(data, weights, means, covariances)
    ll_trace = [ll]

    for i in range(maxiter):
        if i % 5 == 0:
            print("Iteration %s" % i)

        # E-step: compute responsibilities
        # Update resp matrix so that resp[j, k] is the responsibility of cluster k for data point j.
        # Hint: To compute likelihood of seeing data point j given cluster k, use multivariate_normal.pdf.
        for j in range(num_data):
            for k in range(num_clusters):
                # YOUR CODE HERE
                resp[j, k] = multivariate_normal.pdf(data[j], means[k], covariances[k])
        row_sums = resp.sum(axis=1)[:, np.newaxis]
        resp = resp / row_sums # normalize over all possible cluster assignments

        # M-step
        # Compute the total responsibility assigned to each cluster, which will be useful when
        # implementing M-steps below. In the lectures this is called N^{soft}
        counts = np.sum(resp, axis=0)

        for k in range(num_clusters):
            Nsoft = resp[ ,k].sum()
            # Update the weight for cluster k using the M-step update rule for the cluster weight, \hat{\pi}_k.
            # YOUR CODE HERE
            weights[k] = Nsoft/num_data

            # Update means for cluster k using the M-step update rule for the mean variables.
            # This will assign the variable means[k] to be our estimate for \hat{\mu}_k.
            weighted_sum = 0
            for j in range(num_data):
                #...


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
Implementing the EM algorithm for Gaussian mixture models
In this section, you will implement the EM algorithm. We will take the following steps:

Create some synthetic data.
Provide a log likelihood function for this model.
Implement the EM algorithm.
Visualize the progress of the parameters during the course of running EM.
Visualize the convergence of the model.

Dataset
To help us develop and test our implementation, we will generate some observations from a mixture of Gaussians and then run our EM algorithm to discover the mixture components. We'll begin with a function to generate the data, and a quick plot to visualize its output for a 2-dimensional mixture of three Gaussians.
Now we will create a function to generate data from a mixture of Gaussians model.
Can you write Python code for it?

def generate_MoG_data(num_data, means, covariances, weights):
    """ Creates a list of data points """
    num_clusters = len(weights)
    data = []
    for i in range(num_data):
        #  Use np.random.choice and weights to pick a cluster id greater than or equal to 0 and less than num_clusters.
        k = np.random.choice(len(weights), 1, p=weights)[0]

        # Use np.random.multivariate_normal to create data from this cluster
        x = np.random.multivariate_normal(means[k], covariances[k])

        data.append(x)
    return data


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
Log likelihood
We provide a function to calculate log likelihood for mixture of Gaussians. The log likelihood quantifies the probability of observing a given set of data under a particular setting of the parameters in our model. We will use this to assess convergence of our EM algorithm; specifically, we will keep looping through EM update steps until the log likehood ceases to increase at a certain rate.
Can you write Python code for it?

def log_sum_exp(Z):
    """ Compute log(\sum_i exp(Z_i)) for some array Z."""
    return np.max(Z) + np.log(np.sum(np.exp(Z - np.max(Z))))

def loglikelihood(data, weights, means, covs):
    """ Compute the loglikelihood of the data for a Gaussian mixture model with the given parameters. """
    num_clusters = len(means)
    num_dim = len(data[0])

    ll = 0
    for d in data:

        Z = np.zeros(num_clusters)
        for k in range(num_clusters):

            # Compute (x-mu)^T * Sigma^{-1} * (x-mu)
            delta = np.array(d) - means[k]
            exponent_term = np.dot(delta.T, np.dot(np.linalg.inv(covs[k]), delta))

            # Compute loglikelihood contribution for this data point and this cluster
            Z[k] += np.log(weights[k])
            Z[k] -= 1/2. * (num_dim * np.log(2*np.pi) + np.log(np.linalg.det(covs[k])) + exponent_term)

        # Increment loglikelihood contribution of this data point across all clusters
        ll += log_sum_exp(Z)

    return ll


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
Plot progress of parameters
One useful feature of testing our implementation on low-dimensional simulated data is that we can easily visualize the results.
We will use the following plot_contours function to visualize the Gaussian components over the data at three different points in the algorithm's execution:

At initialization (using initial_mu, initial_cov, and initial_weights)
After running the algorithm to completion
After just 12 iterations (using parameters estimates returned when setting maxiter=12)
Can you write Python code for it?

import matplotlib.mlab as mlab
def plot_contours(data, means, covs, title):
    plt.figure()
    plt.plot([x[0] for x in data], [y[1] for y in data],'ko') # data

    delta = 0.025
    k = len(means)
    x = np.arange(-2.0, 7.0, delta)
    y = np.arange(-2.0, 7.0, delta)
    X, Y = np.meshgrid(x, y)
    col = ['green', 'red', 'indigo']
    for i in range(k):
        mean = means[i]
        cov = covs[i]
        sigmax = np.sqrt(cov[0][0])
        sigmay = np.sqrt(cov[1][1])
        sigmaxy = cov[0][1]/(sigmax*sigmay)
        Z = mlab.bivariate_normal(X, Y, sigmax, sigmay, mean[0], mean[1], sigmaxy)
        plt.contour(X, Y, Z, colors = col[i])
        plt.title(title)
    plt.rcParams.update({'font.size':16})
    plt.tight_layout()

# Parameters after initialization
plot_contours(data, initial_means, initial_covs, 'Initial clusters')

# Parameters after running EM to convergence
results = EM(data, initial_means, initial_covs, initial_weights)
plot_contours(data, results['means'], results['covs'], 'Final clusters')


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
Quiz Question: Plot the loglikelihood that is observed at each iteration. Is the loglikelihood plot monotonically increasing, monotonically decreasing, or neither [multiple choice]?
Can you write Python code for it?

results = EM(data, initial_means, initial_covs, initial_weights)

# YOUR CODE HERE
loglikelihoods = ...

plt.plot(range(len(loglikelihoods)), loglikelihoods, linewidth=4)
plt.xlabel('Iteration')
plt.ylabel('Log-likelihood')
plt.rcParams.update({'font.size':16})
plt.tight_layout()


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
Quiz Question: Using the same set of results, obtain the covariance that EM assigns the third component. What is the variance in the first dimension?
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
Quiz Question: Using the same set of results, obtain the mean that EM assigns the second component. What is the mean in the first dimension?
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
Testing the implementation on the simulated data
Now we'll fit a mixture of Gaussians to this data using our implementation of the EM algorithm. As with k-means, it is important to ask how we obtain an initial configuration of mixing weights and component parameters. In this simple case, we'll take three random points to be the initial cluster means, use the empirical covariance of the data to be the initial covariance in each cluster (a clear overestimate), and set the initial mixing weights to be uniform across clusters.
Can you write Python code for it?

np.random.seed(4)

# Initialization of parameters
chosen = np.random.choice(len(data), 3, replace=False)
initial_means = [data[x] for x in chosen]
initial_covs = [np.cov(data, rowvar=0)] * 3
initial_weights = [1/3.] * 3

# Run EM
results = EM(data, initial_means, initial_covs, initial_weights)


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
The following sections will evaluate the results by asking the following questions:

Convergence: How did the log likelihood change across iterations? Did the algorithm achieve convergence?
Uncertainty: How did cluster assignment and uncertainty evolve?
Interpretability: Can we view some example images from each cluster? Do these clusters correspond to known image categories?

Evaluating convergence
Let's start by plotting the log likelihood at each iteration - we know that the EM algorithm guarantees that the log likelihood can only increase (or stay the same) after each iteration, so if our implementation is correct then we should see an increasing function.
Can you write Python code for it?

ll = out['loglik']
plt.plot(range(len(ll)),ll,linewidth=4)
plt.xlabel('Iteration')
plt.ylabel('Log-likelihood')
plt.rcParams.update({'font.size':16})
plt.tight_layout()


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
The log likelihood increases so quickly on the first few iterations that we can barely see the plotted line. Let's plot the log likelihood after the first three iterations to get a clearer view of what's going on:
Can you write Python code for it?

plt.figure()
plt.plot(range(3,len(ll)),ll[3:],linewidth=4)
plt.xlabel('Iteration')
plt.ylabel('Log-likelihood')
plt.rcParams.update({'font.size':16})
plt.tight_layout()


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
To begin, we will visualize what happens when each data has random responsibilities.
Can you write Python code for it?

N, K = out['resp'].shape
random_resp = np.random.dirichlet(np.ones(K), N)
plot_responsibilities_in_RB(images, random_resp, 'Random responsibilities')


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
Use this function to show the top 5 images in each cluster.
Can you write Python code for it?

gl.canvas.set_target('ipynb')
for component_id in range(4):
    get_top_images(assignments, component_id).show()


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
We need to come up with initial estimates for the mixture weights and component parameters. Let's take three images to be our initial cluster centers, and let's initialize the covariance matrix of each cluster to be diagonal with each element equal to the sample variance from the full data. As in our test on simulated data, we'll start by assuming each mixture component has equal weight.
This may take a few minutes to run.
Can you write Python code for it?

np.random.seed(1)

# Initalize parameters
init_means = [images['rgb'][x] for x in np.random.choice(len(images), 4, replace=False)]
cov = np.diag([images['red'].var(), images['green'].var(), images['blue'].var()])
init_covariances = [cov, cov, cov, cov]
init_weights = [1/4., 1/4., 1/4., 1/4.]

# Convert rgb data to numpy arrays
img_data = [np.array(i) for i in images['rgb']]

# Run our EM algorithm on the image data using the above initializations.
# This should converge in about 125 iterations
out = EM(img_data, init_means, init_covariances, init_weights)


I am working on the file "Clustering_&_Retrieval/Week4/Assignment1/.ipynb_checkpoints/3_em-for-gmm_blank-checkpoint.ipynb".
The first task is:
We now use the above plotting function to visualize the responsibilites after 1 iteration.
Can you write Python code for it?

out = EM(img_data, init_means, init_covariances, init_weights, maxiter=1)
plot_responsibilities_in_RB(images, out['resp'], 'After 1 iteration')


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/.ipynb_checkpoints/4_em-with-text-data_blank-checkpoint.ipynb".
The first task is:
EM in high dimensions
EM for high-dimensional data requires some special treatment:
 * E step and M step must be vectorized as much as possible, as explicit loops are dreadfully slow in Python.
 * All operations must be cast in terms of sparse matrix operations, to take advantage of computational savings enabled by sparsity of data.
 * Initially, some words may be entirely absent from a cluster, causing the M step to produce zero mean and variance for those words.  This means any data point with one of those words will have 0 probability of being assigned to that cluster since the cluster allows for no variability (0 variance) around that count being 0 (0 mean). Since there is a small chance for those words to later appear in the cluster, we instead assign a small positive variance (~1e-10). Doing so also prevents numerical overflow.
We provide the complete implementation for you in the file em_utilities.py. For those who are interested, you can read through the code to see how the sparse matrix implementation differs from the previous assignment.
You are expected to answer some quiz questions using the results of clustering.
Initializing mean parameters using k-means
Recall from the lectures that EM for Gaussian mixtures is very sensitive to the choice of initial means. With a bad initial set of means, EM may produce clusters that span a large area and are mostly overlapping. To eliminate such bad outcomes, we first produce a suitable set of initial means by using the cluster centers from running k-means.  That is, we first run k-means and then take the final set of means from the converged solution as the initial means in our EM algorithm.
Can you write Python code for it?

from sklearn.cluster import KMeans

np.random.seed(5)
num_clusters = 25

# Use scikit-learn's k-means to simplify workflow
kmeans_model = KMeans(n_clusters=num_clusters, n_init=5, max_iter=400, random_state=1, n_jobs=-1)
kmeans_model.fit(tf_idf)
centroids, cluster_assignment = kmeans_model.cluster_centers_, kmeans_model.labels_

means = [centroid for centroid in centroids]


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/.ipynb_checkpoints/4_em-with-text-data_blank-checkpoint.ipynb".
The first task is:
Interpret clustering results
In contrast to k-means, EM is able to explicitly model clusters of varying sizes and proportions. The relative magnitude of variances in the word dimensions tell us much about the nature of the clusters.
Write yourself a cluster visualizer as follows.  Examining each cluster's mean vector, list the 5 words with the largest mean values (5 most common words in the cluster). For each word, also include the associated variance parameter (diagonal element of the covariance matrix).
A sample output may be:
```
==========================================================
Cluster 0: Largest mean parameters in cluster
Word        Mean        Variance
football    1.08e-01    8.64e-03
season      5.80e-02    2.93e-03
club        4.48e-02    1.99e-03
league      3.94e-02    1.08e-03
played      3.83e-02    8.45e-04
...
```
Can you write Python code for it?

# Fill in the blanks
def visualize_EM_clusters(tf_idf, means, covs, map_index_to_word):
    print('')
    print('==========================================================')

    num_clusters = len(means)
    for c in xrange(num_clusters):
        print('Cluster {0:d}: Largest mean parameters in cluster '.format(c))
        print('\n{0: <12}{1: <12}{2: <12}'.format('Word', 'Mean', 'Variance'))

        # The k'th element of sorted_word_ids should be the index of the word
        # that has the k'th-largest value in the cluster mean. Hint: Use np.argsort().
        sorted_word_ids = ...  # YOUR CODE HERE

        for i in sorted_word_ids[:5]:
            print '{0: <12}{1:<10.2e}{2:10.2e}'.format(map_index_to_word['category'][i],
                                                       means[c][i],
                                                       covs[c][i])
        print '\n=========================================================='

'''By EM'''
visualize_EM_clusters(tf_idf, out['means'], out['covs'], map_index_to_word)


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/.ipynb_checkpoints/4_em-with-text-data_blank-checkpoint.ipynb".
The first task is:
Load Wikipedia data and extract TF-IDF features
Load Wikipedia data and transform each of the first 5000 document into a TF-IDF representation.
Can you write Python code for it?

wiki = graphlab.SFrame('people_wiki.gl/').head(5000)
wiki['tf_idf'] = graphlab.text_analytics.tf_idf(wiki['text'])


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/.ipynb_checkpoints/4_em-with-text-data_blank-checkpoint.ipynb".
The first task is:
Quiz Question. Select all the topics that have a cluster in the model created above. [multiple choice]
Comparing to random initialization
Create variables for randomly initializing the EM algorithm. Complete the following code block.
Can you write Python code for it?

np.random.seed(5)
num_clusters = len(means)
num_docs, num_words = tf_idf.shape

random_means = []
random_covs = []
random_weights = []

for k in range(num_clusters):

    # Create a numpy array of length num_words with random normally distributed values.
    # Use the standard univariate normal distribution (mean 0, variance 1).
    # YOUR CODE HERE
    mean = ...

    # Create a numpy array of length num_words with random values uniformly distributed between 1 and 5.
    # YOUR CODE HERE
    cov = ...

    # Initially give each cluster equal weight.
    # YOUR CODE HERE
    weight = ...

    random_means.append(mean)
    random_covs.append(cov)
    random_weights.append(weight)


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/.ipynb_checkpoints/4_em-with-text-data_blank-checkpoint.ipynb".
The first task is:
Quiz Question: Try fitting EM with the random initial parameters you created above. (Use cov_smoothing=1e-5.) Store the result to out_random_init. What is the final loglikelihood that the algorithm converges to?
Quiz Question: Is the final loglikelihood larger or smaller than the final loglikelihood we obtained above when initializing EM with the results from running k-means?
Quiz Question: For the above model, out_random_init, use the visualize_EM_clusters method you created above. Are the clusters more or less interpretable than the ones found after initializing using k-means?
Can you write Python code for it?

# YOUR CODE HERE. Use visualize_EM_clusters, which will require you to pass in tf_idf and map_index_to_word.
...


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/.ipynb_checkpoints/4_em-with-text-data_blank-checkpoint.ipynb".
The first task is:
Using a utility we provide, we will create a sparse matrix representation of the documents. This is the same utility function you used during the previous assignment on k-means with text data.
Can you write Python code for it?

tf_idf, map_index_to_word = sframe_to_scipy(wiki, 'tf_idf')


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/.ipynb_checkpoints/4_em-with-text-data_blank-checkpoint.ipynb".
The first task is:
We can check that the length (Euclidean norm) of each row is now 1.0, as expected.
Can you write Python code for it?

for i in range(5):
    doc = tf_idf[i]
    print(np.linalg.norm(doc.todense()))


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/4_em-with-text-data_blank.ipynb".
The first task is:
As in the previous assignment, we will normalize each document's TF-IDF vector to be a unit vector.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/4_em-with-text-data_blank.ipynb".
The first task is:
EM in high dimensions
EM for high-dimensional data requires some special treatment:
 * E step and M step must be vectorized as much as possible, as explicit loops are dreadfully slow in Python.
 * All operations must be cast in terms of sparse matrix operations, to take advantage of computational savings enabled by sparsity of data.
 * Initially, some words may be entirely absent from a cluster, causing the M step to produce zero mean and variance for those words.  This means any data point with one of those words will have 0 probability of being assigned to that cluster since the cluster allows for no variability (0 variance) around that count being 0 (0 mean). Since there is a small chance for those words to later appear in the cluster, we instead assign a small positive variance (~1e-10). Doing so also prevents numerical overflow.
We provide the complete implementation for you in the file em_utilities.py. For those who are interested, you can read through the code to see how the sparse matrix implementation differs from the previous assignment.
You are expected to answer some quiz questions using the results of clustering.
Initializing mean parameters using k-means
Recall from the lectures that EM for Gaussian mixtures is very sensitive to the choice of initial means. With a bad initial set of means, EM may produce clusters that span a large area and are mostly overlapping. To eliminate such bad outcomes, we first produce a suitable set of initial means by using the cluster centers from running k-means.  That is, we first run k-means and then take the final set of means from the converged solution as the initial means in our EM algorithm.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/4_em-with-text-data_blank.ipynb".
The first task is:
Initializing covariances
To initialize our covariance parameters, we compute $\hat{\sigma}{k, j}^2 = \sum{i=1}^{N}(x_{i,j} - \hat{\mu}_{k, j})^2$ for each feature $j$.  For features with really tiny variances, we assign 1e-8 instead to prevent numerical instability. We do this computation in a vectorized fashion in the following code block.
Can you write Python code for it?

covs = []
for i in xrange(num_clusters):
    member_rows = tf_idf[cluster_assignment==i]
    cov = (member_rows.power(2) - 2*member_rows.dot(diag(means[i]))).sum(axis=0).A1 / member_rows.shape[0] \
          + means[i]**2
    cov[cov < 1e-8] = 1e-8
    covs.append(cov)


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/4_em-with-text-data_blank.ipynb".
The first task is:
Interpret clustering results
In contrast to k-means, EM is able to explicitly model clusters of varying sizes and proportions. The relative magnitude of variances in the word dimensions tell us much about the nature of the clusters.
Write yourself a cluster visualizer as follows.  Examining each cluster's mean vector, list the 5 words with the largest mean values (5 most common words in the cluster). For each word, also include the associated variance parameter (diagonal element of the covariance matrix).
A sample output may be:
```
==========================================================
Cluster 0: Largest mean parameters in cluster
Word        Mean        Variance
football    1.08e-01    8.64e-03
season      5.80e-02    2.93e-03
club        4.48e-02    1.99e-03
league      3.94e-02    1.08e-03
played      3.83e-02    8.45e-04
...
```
Can you write Python code for it?

# Fill in the blanks
def visualize_EM_clusters(tf_idf, means, covs, map_index_to_word):
    print('')
    print('==========================================================')

    num_clusters = len(means)
    for c in xrange(num_clusters):
        print('Cluster {0:d}: Largest mean parameters in cluster '.format(c))
        print('\n{0: <12}{1: <12}{2: <12}'.format('Word', 'Mean', 'Variance'))

        # The k'th element of sorted_word_ids should be the index of the word
        # that has the k'th-largest value in the cluster mean. Hint: Use np.argsort().
        sorted_word_ids = np.argsort(means[c])[::-1][:len(means[c])]  # YOUR CODE HERE

        for i in sorted_word_ids[:5]:
            print '{0: <12}{1:<10.2e}{2:10.2e}'.format(map_index_to_word['category'][i],
                                                       means[c][i],
                                                       covs[c][i])
        print '\n=========================================================='

'''By EM'''
visualize_EM_clusters(tf_idf, out['means'], out['covs'], map_index_to_word)


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/4_em-with-text-data_blank.ipynb".
The first task is:
Quiz Question. Select all the topics that have a cluster in the model created above. [multiple choice]
Comparing to random initialization
Create variables for randomly initializing the EM algorithm. Complete the following code block.
Can you write Python code for it?

np.random.seed(5)
num_clusters = len(means)
num_docs, num_words = tf_idf.shape

random_means = []
random_covs = []
random_weights = []

for k in range(num_clusters):

    # Create a numpy array of length num_words with random normally distributed values.
    # Use the standard univariate normal distribution (mean 0, variance 1).
    # YOUR CODE HERE
    mean = 1. * np.random.randn(num_words)

    # Create a numpy array of length num_words with random values uniformly distributed between 1 and 5.
    # YOUR CODE HERE
    cov = np.random.uniform(1.,5.,num_words)

    # Initially give each cluster equal weight.
    # YOUR CODE HERE
    weight = 1.

    random_means.append(mean)
    random_covs.append(cov)
    random_weights.append(weight)


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/4_em-with-text-data_blank.ipynb".
The first task is:
Quiz Question: Try fitting EM with the random initial parameters you created above. (Use cov_smoothing=1e-5.) Store the result to out_random_init. What is the final loglikelihood that the algorithm converges to?
Can you write Python code for it?

out_random_init = EM_for_high_dimension(tf_idf, random_means, random_covs, random_weights, cov_smoothing=1e-5)

out_random_init


I am working on the file "Clustering_&_Retrieval/Week4/Assignment2/4_em-with-text-data_blank.ipynb".
The first task is:
We can check that the length (Euclidean norm) of each row is now 1.0, as expected.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week5/.ipynb_checkpoints/5_lda_blank-checkpoint.ipynb".
The first task is:
For each of our two different document representations, we can use GraphLab Create to compute a brute-force nearest neighbors model:
Can you write Python code for it?

model_tf_idf = gl.nearest_neighbors.create(wiki, label='name', features=['tf_idf'],
                                           method='brute_force', distance='cosine')
model_lda_rep = gl.nearest_neighbors.create(wiki, label='name', features=['lda'],
                                            method='brute_force', distance='cosine')


I am working on the file "Clustering_&_Retrieval/Week5/.ipynb_checkpoints/5_lda_blank-checkpoint.ipynb".
The first task is:
Here we can clearly see the smoothing enforced by the alpha parameter - notice that when alpha is low most of the weight in the topic distribution for this article goes to a single topic, but when alpha is high the weight is much more evenly distributed across the topics.
Quiz Question: How many topics are assigned a weight greater than 0.3 or less than 0.05 for the article on Paul Krugman in the low alpha model?  Use the average results from 100 topic predictions.
Quiz Question: How many topics are assigned a weight greater than 0.3 or less than 0.05 for the article on Paul Krugman in the high alpha model? Use the average results from 100 topic predictions.
Changing the hyperparameter gamma
Just as we were able to see the effect of alpha by plotting topic weights for a document, we expect to be able to visualize the impact of changing gamma by plotting word weights for each topic. In this case, however, there are far too many words in our vocabulary to do this effectively. Instead, we'll plot the total weight of the top 100 words and bottom 1000 words for each topic. Below, we plot the (sorted) total weights of the top 100 words and bottom 1000 from each topic in the high, original, and low gamma models.
Now we will consider the following two models:
 - tpm_low_gamma, a model trained with gamma = 0.02 and default alpha
 - tpm_high_gamma, a model trained with gamma = 0.5 and default alpha
Can you write Python code for it?

del tpm_low_alpha
del tpm_high_alpha
tpm_low_gamma = gl.load_model('lda_low_gamma')
tpm_high_gamma = gl.load_model('lda_high_gamma')

a_top = np.sort([sum(tpm_low_gamma.get_topics(topic_ids=[i], num_words=100)['score']) for i in range(10)])[::-1]
b_top = np.sort([sum(topic_model.get_topics(topic_ids=[i], num_words=100)['score']) for i in range(10)])[::-1]
c_top = np.sort([sum(tpm_high_gamma.get_topics(topic_ids=[i], num_words=100)['score']) for i in range(10)])[::-1]

a_bot = np.sort([sum(tpm_low_gamma.get_topics(topic_ids=[i], num_words=547462)[-1000:]['score']) for i in range(10)])[::-1]
b_bot = np.sort([sum(topic_model.get_topics(topic_ids=[i], num_words=547462)[-1000:]['score']) for i in range(10)])[::-1]
c_bot = np.sort([sum(tpm_high_gamma.get_topics(topic_ids=[i], num_words=547462)[-1000:]['score']) for i in range(10)])[::-1]

ind = np.arange(len(a))
width = 0.3

param_bar_plot(a_top, b_top, c_top, ind, width, ylim=0.6, param='gamma',
               xlab='Topics (sorted by weight of top 100 words)',
               ylab='Total Probability of Top 100 Words')

param_bar_plot(a_bot, b_bot, c_bot, ind, width, ylim=0.0002, param='gamma',
               xlab='Topics (sorted by weight of bottom 1000 words)',
               ylab='Total Probability of Bottom 1000 Words')


I am working on the file "Clustering_&_Retrieval/Week5/.ipynb_checkpoints/5_lda_blank-checkpoint.ipynb".
The first task is:
Here we see that, for our topic model, the top 10 words only account for a small fraction (in this case, between 5% and 13%) of their topic's total probability mass. So while we can use the top words to identify broad themes for each topic, we should keep in mind that in reality these topics are more complex than a simple 10-word summary.
Finally, we observe that some 'junk' words appear highly rated in some topics despite our efforts to remove unhelpful words before fitting the model; for example, the word 'born' appears as a top 10 word in three different topics, but it doesn't help us describe these topics at all.
Topic distributions for some example documents
As we noted in the introduction to this assignment, LDA allows for mixed membership, which means that each document can partially belong to several different topics. For each document, topic membership is expressed as a vector of weights that sum to one; the magnitude of each weight indicates the degree to which the document represents that particular topic.
We'll explore this in our fitted model by looking at the topic distributions for a few example Wikipedia articles from our data set. We should find that these articles have the highest weights on the topics whose themes are most relevant to the subject of the article - for example, we'd expect an article on a politician to place relatively high weight on topics related to government, while an article about an athlete should place higher weight on topics related to sports or competition.
Topic distributions for documents can be obtained using GraphLab Create's predict() function. GraphLab Create uses a collapsed Gibbs sampler similar to the one described in the video lectures, where only the word assignments variables are sampled.  To get a document-specific topic proportion vector post-facto, predict() draws this vector from the conditional distribution given the sampled word assignments in the document.  Notice that, since these are draws from a distribution over topics that the model has...
Can you write Python code for it?

obama = gl.SArray([wiki_docs[int(np.where(wiki['name']=='Barack Obama')[0])]])
pred1 = topic_model.predict(obama, output_type='probability')
pred2 = topic_model.predict(obama, output_type='probability')
print(gl.SFrame({'topics':themes, 'predictions (first draw)':pred1[0], 'predictions (second draw)':pred2[0]}))


I am working on the file "Clustering_&_Retrieval/Week5/.ipynb_checkpoints/5_lda_blank-checkpoint.ipynb".
The first task is:
In the above plot, each line corresponds to one of our ten topics. Notice how for each topic, the weights drop off sharply as we move down the ranked list of most important words. This shows that the top 10-20 words in each topic are assigned a much greater weight than the remaining words - and remember from the summary of our topic model that our vocabulary has 547462 words in total!
Next we plot the total weight assigned by each topic to its top 10 words:
Can you write Python code for it?

top_probs = [sum(topic_model.get_topics(topic_ids=[i], num_words=10)['score']) for i in range(10)]

ind = np.arange(10)
width = 0.5

fig, ax = plt.subplots()

ax.bar(ind-(width/2),top_probs,width)
ax.set_xticks(ind)

plt.xlabel('Topic')
plt.ylabel('Probability')
plt.title('Total Probability of Top 10 Words in each Topic')
plt.xlim(-0.5,9.5)
plt.ylim(0,0.15)
plt.show()


I am working on the file "Clustering_&_Retrieval/Week5/.ipynb_checkpoints/5_lda_blank-checkpoint.ipynb".
The first task is:
In the original data, each Wikipedia article is represented by a URI, a name, and a string containing the entire text of the article. Recall from the video lectures that LDA requires documents to be represented as a bag of words, which ignores word ordering in the document but retains information on how many times each word appears. As we have seen in our previous encounters with text data, words such as 'the', 'a', or 'and' are by far the most frequent, but they appear so commonly in the English language that they tell us almost nothing about how similar or dissimilar two documents might be.
Therefore, before we train our LDA model, we will preprocess the Wikipedia data in two steps: first, we will create a bag of words representation for each article, and then we will remove the common words that don't help us to distinguish between documents. For both of these tasks we can use pre-implemented tools from GraphLab Create:
Can you write Python code for it?

wiki_docs = gl.text_analytics.count_words(wiki['text'])
wiki_docs = wiki_docs.dict_trim_by_keys(gl.text_analytics.stopwords(), exclude=True)


I am working on the file "Clustering_&_Retrieval/Week5/.ipynb_checkpoints/5_lda_blank-checkpoint.ipynb".
The first task is:
It is certainly useful to have pre-implemented methods available for LDA, but as with our previous methods for clustering and retrieval, implementing and fitting the model gets us only halfway towards our objective. We now need to analyze the fitted model to understand what it has done with our data and whether it will be useful as a document classification system. This can be a challenging task in itself, particularly when the model that we use is complex. We will begin by outlining a sequence of objectives that will help us understand our model in detail. In particular, we will

get the top words in each topic and use these to identify topic themes
predict topic distributions for some example documents
compare the quality of LDA "nearest neighbors" to the NN output from the first assignment
understand the role of model hyperparameters alpha and gamma

Load a fitted topic model
The method used to fit the LDA model is a randomized algorithm, which means that it involves steps that are random; in this case, the randomness comes from Gibbs sampling, as discussed in the LDA video lectures. Because of these random steps, the algorithm will be expected to yield slighty different output for different runs on the same data - note that this is different from previously seen algorithms such as k-means or EM, which will always produce the same results given the same input and initialization.
It is important to understand that variation in the results is a fundamental feature of randomized methods. However, in the context of this assignment this variation makes it difficult to evaluate the correctness of your analysis, so we will load and analyze a pre-trained model.
We recommend that you spend some time exploring your own fitted topic model and compare our analysis of the pre-trained model to the same analysis applied to the model you trained above.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week5/.ipynb_checkpoints/5_lda_blank-checkpoint.ipynb".
The first task is:
Measuring the importance of top words
We can learn more about topics by exploring how they place probability mass (which we can think of as a weight) on each of their top words.
We'll do this with two visualizations of the weights for the top words in each topic:
 - the weights of the top 100 words, sorted by the size
 - the total weight of the top 10 words
Here's a plot for the top 100 words by weight in each topic:
Can you write Python code for it?

for i in range(10):
    plt.plot(range(100), topic_model.get_topics(topic_ids=[i], num_words=100)['score'])
plt.xlabel('Word rank')
plt.ylabel('Probability')
plt.title('Probabilities of Top 100 Words in each Topic')


I am working on the file "Clustering_&_Retrieval/Week5/.ipynb_checkpoints/5_lda_blank-checkpoint.ipynb".
The first task is:
Next we add the TF-IDF document representations:
Can you write Python code for it?

wiki['word_count'] = gl.text_analytics.count_words(wiki['text'])
wiki['tf_idf'] = gl.text_analytics.tf_idf(wiki['word_count'])


I am working on the file "Clustering_&_Retrieval/Week5/.ipynb_checkpoints/5_lda_blank-checkpoint.ipynb".
The first task is:
Notice that that there is no overlap between the two sets of top 10 nearest neighbors. This doesn't necessarily mean that one representation is better or worse than the other, but rather that they are picking out different features of the documents.
With TF-IDF, documents are distinguished by the frequency of uncommon words. Since similarity is defined based on the specific words used in the document, documents that are "close" under TF-IDF tend to be similar in terms of specific details. This is what we see in the example: the top 10 nearest neighbors are all economists from the US, UK, or Canada.
Our LDA representation, on the other hand, defines similarity between documents in terms of their topic distributions. This means that documents can be "close" if they share similar themes, even though they may not share many of the same keywords. For the article on Paul Krugman, we expect the most important topics to be 'American college and politics' and 'science and research'. As a result, we see that the top 10 nearest neighbors are academics from a wide variety of fields, including literature, anthropology, and religious studies.
Quiz Question: Using the TF-IDF representation, compute the 5000 nearest neighbors for American baseball player Alex Rodriguez. For what value of k is Mariano Rivera the k-th nearest neighbor to Alex Rodriguez? (Hint: Once you have a list of the nearest neighbors, you can use mylist.index(value) to find the index of the first instance of value in mylist.)
Quiz Question: Using the LDA representation, compute the 5000 nearest neighbors for American baseball player Alex Rodriguez. For what value of k is Mariano Rivera the k-th nearest neighbor to Alex Rodriguez? (Hint: Once you have a list of the nearest neighbors, you can use mylist.index(value) to find the index of the first instance of value in mylist.)
Understanding the role of LDA model hyperparameters
Finally, we'll take a look at the effect of the LDA model hyperparameters alpha and gamma on the characteristics of our fitted...
Can you write Python code for it?

tpm_low_alpha = gl.load_model('lda_low_alpha')
tpm_high_alpha = gl.load_model('lda_high_alpha')


I am working on the file "Clustering_&_Retrieval/Week5/.ipynb_checkpoints/5_lda_blank-checkpoint.ipynb".
The first task is:
To get a more robust estimate of the topics for each document, we can average a large number of predictions for the same document:
Can you write Python code for it?

def average_predictions(model, test_document, num_trials=100):
    avg_preds = np.zeros((model.num_topics))
    for i in range(num_trials):
        avg_preds += model.predict(test_document, output_type='probability')[0]
    avg_preds = avg_preds/num_trials
    result = gl.SFrame({'topics':themes, 'average predictions':avg_preds})
    result = result.sort('average predictions', ascending=False)
    return result

print average_predictions(topic_model, obama, 100)


I am working on the file "Clustering_&_Retrieval/Week5/.ipynb_checkpoints/5_lda_blank-checkpoint.ipynb".
The first task is:
We propose the following themes for each topic:

topic 0: Science and research
topic 2: Team sports
topic 3: Music, TV, and film
topic 4: American college and politics
topic 5: General politics
topic 6: Art and publishing
topic 7: Business
topic 8: International athletics
topic 9: Great Britain and Australia
topic 10: International music

We'll save these themes for later:
Can you write Python code for it?

themes = ['science and research','team sports','music, TV, and film','American college and politics','general politics', \
         'art and publishing','Business','international athletics','Great Britain and Australia','international music']


I am working on the file "Clustering_&_Retrieval/Week5/5_lda_blank.ipynb".
The first task is:
Changing the hyperparameter alpha
Since alpha is responsible for smoothing document preferences over topics, the impact of changing its value should be visible when we plot the distribution of topic weights for the same document under models fit with different alpha values. In the code below, we plot the (sorted) topic weights for the Wikipedia article on Barack Obama under models fit with high, original, and low settings of alpha.
Can you write Python code for it?

a = np.sort(tpm_low_alpha.predict(obama,output_type='probability')[0])[::-1]
b = np.sort(topic_model.predict(obama,output_type='probability')[0])[::-1]
c = np.sort(tpm_high_alpha.predict(obama,output_type='probability')[0])[::-1]
ind = np.arange(len(a))
width = 0.3

def param_bar_plot(a,b,c,ind,width,ylim,param,xlab,ylab):
    fig = plt.figure()
    ax = fig.add_subplot(111)

    b1 = ax.bar(ind, a, width, color='lightskyblue')
    b2 = ax.bar(ind+width, b, width, color='lightcoral')
    b3 = ax.bar(ind+(2*width), c, width, color='gold')

    ax.set_xticks(ind+width)
    ax.set_xticklabels(range(10))
    ax.set_ylabel(ylab)
    ax.set_xlabel(xlab)
    ax.set_ylim(0,ylim)
    ax.legend(handles = [b1,b2,b3],labels=['low '+param,'original model','high '+param])

    plt.tight_layout()

param_bar_plot(a,b,c,ind,width,ylim=1.0,param='alpha',
               xlab='Topics (sorted by weight of top 100 words)',ylab='Topic Probability for Obama Article')


I am working on the file "Clustering_&_Retrieval/Week5/5_lda_blank.ipynb".
The first task is:
Here we can clearly see the smoothing enforced by the alpha parameter - notice that when alpha is low most of the weight in the topic distribution for this article goes to a single topic, but when alpha is high the weight is much more evenly distributed across the topics.
Quiz Question: How many topics are assigned a weight greater than 0.3 or less than 0.05 for the article on Paul Krugman in the low alpha model?  Use the average results from 100 topic predictions.
Quiz Question: How many topics are assigned a weight greater than 0.3 or less than 0.05 for the article on Paul Krugman in the high alpha model? Use the average results from 100 topic predictions.
Changing the hyperparameter gamma
Just as we were able to see the effect of alpha by plotting topic weights for a document, we expect to be able to visualize the impact of changing gamma by plotting word weights for each topic. In this case, however, there are far too many words in our vocabulary to do this effectively. Instead, we'll plot the total weight of the top 100 words and bottom 1000 words for each topic. Below, we plot the (sorted) total weights of the top 100 words and bottom 1000 from each topic in the high, original, and low gamma models.
Now we will consider the following two models:
 - tpm_low_gamma, a model trained with gamma = 0.02 and default alpha
 - tpm_high_gamma, a model trained with gamma = 0.5 and default alpha
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week5/5_lda_blank.ipynb".
The first task is:
Identifying topic themes by top words
We'll start by trying to identify the topics learned by our model with some major themes. As a preliminary check on the results of applying this method, it is reasonable to hope that the model has been able to learn topics that correspond to recognizable categories. In order to do this, we must first recall what exactly a 'topic' is in the context of LDA.
In the video lectures on LDA we learned that a topic is a probability distribution over words in the vocabulary; that is, each topic assigns a particular probability to every one of the unique words that appears in our data. Different topics will assign different probabilities to the same word: for instance, a topic that ends up describing science and technology articles might place more probability on the word 'university' than a topic that describes sports or politics. Looking at the highest probability words in each topic will thus give us a sense of its major themes. Ideally we would find that each topic is identifiable with some clear theme and that all the topics are relatively distinct.
We can use the GraphLab Create function get_topics() to view the top words (along with their associated probabilities) from each topic.
Quiz Question: Identify the top 3 most probable words for the first topic.
 Quiz Question: What is the sum of the probabilities assigned to the top 50 words in the 3rd topic?
Let's look at the top 10 words for each topic to see if we can identify any themes:
Can you write Python code for it?

topics = topic_model.get_topics()
topics_1 = topics[topics['topic']==0]
print topics_1[0:3]['word']
topics_2 = topics[topics['topic']==1]
# print topics_2
topics_3 = topics[topics['topic']==2]
print topics_3[0:50]['score'].sum()

topic_3

[x['words'] for x in topic_model.get_topics(output_type='topic_words', num_words=10)]


I am working on the file "Clustering_&_Retrieval/Week5/5_lda_blank.ipynb".
The first task is:
In the above plot, each line corresponds to one of our ten topics. Notice how for each topic, the weights drop off sharply as we move down the ranked list of most important words. This shows that the top 10-20 words in each topic are assigned a much greater weight than the remaining words - and remember from the summary of our topic model that our vocabulary has 547462 words in total!
Next we plot the total weight assigned by each topic to its top 10 words:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week5/5_lda_blank.ipynb".
The first task is:
In the original data, each Wikipedia article is represented by a URI, a name, and a string containing the entire text of the article. Recall from the video lectures that LDA requires documents to be represented as a bag of words, which ignores word ordering in the document but retains information on how many times each word appears. As we have seen in our previous encounters with text data, words such as 'the', 'a', or 'and' are by far the most frequent, but they appear so commonly in the English language that they tell us almost nothing about how similar or dissimilar two documents might be.
Therefore, before we train our LDA model, we will preprocess the Wikipedia data in two steps: first, we will create a bag of words representation for each article, and then we will remove the common words that don't help us to distinguish between documents. For both of these tasks we can use pre-implemented tools from GraphLab Create:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week5/5_lda_blank.ipynb".
The first task is:
It is certainly useful to have pre-implemented methods available for LDA, but as with our previous methods for clustering and retrieval, implementing and fitting the model gets us only halfway towards our objective. We now need to analyze the fitted model to understand what it has done with our data and whether it will be useful as a document classification system. This can be a challenging task in itself, particularly when the model that we use is complex. We will begin by outlining a sequence of objectives that will help us understand our model in detail. In particular, we will

get the top words in each topic and use these to identify topic themes
predict topic distributions for some example documents
compare the quality of LDA "nearest neighbors" to the NN output from the first assignment
understand the role of model hyperparameters alpha and gamma

Load a fitted topic model
The method used to fit the LDA model is a randomized algorithm, which means that it involves steps that are random; in this case, the randomness comes from Gibbs sampling, as discussed in the LDA video lectures. Because of these random steps, the algorithm will be expected to yield slighty different output for different runs on the same data - note that this is different from previously seen algorithms such as k-means or EM, which will always produce the same results given the same input and initialization.
It is important to understand that variation in the results is a fundamental feature of randomized methods. However, in the context of this assignment this variation makes it difficult to evaluate the correctness of your analysis, so we will load and analyze a pre-trained model.
We recommend that you spend some time exploring your own fitted topic model and compare our analysis of the pre-trained model to the same analysis applied to the model you trained above.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week5/5_lda_blank.ipynb".
The first task is:
Let's compare these nearest neighbor models by finding the nearest neighbors under each representation on an example document. For this example we'll use Paul Krugman, an American economist:
Can you write Python code for it?

model_tf_idf.query(wiki[wiki['name'] == 'Paul Krugman'], label='name', k=10)

model_lda_rep.query(wiki[wiki['name'] == 'Paul Krugman'], label='name', k=10)

Rodriguez_nearest = model_tf_idf.query(wiki[wiki['name'] == 'Alex Rodriguez'], label='name', k=5000)
print Rodriguez_nearest[Rodriguez_nearest['reference_label']=='Mariano Rivera']

Rodriguez_nearest_lda = model_lda_rep.query(wiki[wiki['name'] == 'Alex Rodriguez'], label='name', k=5000)
print Rodriguez_nearest_lda[Rodriguez_nearest_lda['reference_label']=='Mariano Rivera']

type(Rodriguez_nearest_lda['reference_label'])


I am working on the file "Clustering_&_Retrieval/Week5/5_lda_blank.ipynb".
The first task is:
Model fitting and interpretation
In the video lectures we saw that Gibbs sampling can be used to perform inference in the LDA model. In this assignment we will use a GraphLab Create method to learn the topic model for our Wikipedia data, and our main emphasis will be on interpreting the results. We'll begin by creating the topic model using create() from GraphLab Create's topic_model module.
Note: This may take several minutes to run.
Can you write Python code for it?

topic_model = gl.topic_model.create(wiki_docs, num_topics=10, num_iterations=200)


I am working on the file "Clustering_&_Retrieval/Week5/5_lda_blank.ipynb".
The first task is:
Next we add the TF-IDF document representations:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week5/5_lda_blank.ipynb".
The first task is:
Notice that that there is no overlap between the two sets of top 10 nearest neighbors. This doesn't necessarily mean that one representation is better or worse than the other, but rather that they are picking out different features of the documents.
With TF-IDF, documents are distinguished by the frequency of uncommon words. Since similarity is defined based on the specific words used in the document, documents that are "close" under TF-IDF tend to be similar in terms of specific details. This is what we see in the example: the top 10 nearest neighbors are all economists from the US, UK, or Canada.
Our LDA representation, on the other hand, defines similarity between documents in terms of their topic distributions. This means that documents can be "close" if they share similar themes, even though they may not share many of the same keywords. For the article on Paul Krugman, we expect the most important topics to be 'American college and politics' and 'science and research'. As a result, we see that the top 10 nearest neighbors are academics from a wide variety of fields, including literature, anthropology, and religious studies.
Quiz Question: Using the TF-IDF representation, compute the 5000 nearest neighbors for American baseball player Alex Rodriguez. For what value of k is Mariano Rivera the k-th nearest neighbor to Alex Rodriguez? (Hint: Once you have a list of the nearest neighbors, you can use mylist.index(value) to find the index of the first instance of value in mylist.)
Quiz Question: Using the LDA representation, compute the 5000 nearest neighbors for American baseball player Alex Rodriguez. For what value of k is Mariano Rivera the k-th nearest neighbor to Alex Rodriguez? (Hint: Once you have a list of the nearest neighbors, you can use mylist.index(value) to find the index of the first instance of value in mylist.)
Understanding the role of LDA model hyperparameters
Finally, we'll take a look at the effect of the LDA model hyperparameters alpha and gamma on the characteristics of our fitted...
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week5/5_lda_blank.ipynb".
The first task is:
Quiz Question: What is the topic most closely associated with the article about former US President George W. Bush? Use the average results from 100 topic predictions.
Quiz Question: What are the top 3 topics corresponding to the article about English football (soccer) player Steven Gerrard? Use the average results from 100 topic predictions.
Comparing LDA to nearest neighbors for document retrieval
So far we have found that our topic model has learned some coherent topics, we have explored these topics as probability distributions over a vocabulary, and we have seen how individual documents in our Wikipedia data set are assigned to these topics in a way that corresponds with our expectations.
In this section, we will use the predicted topic distribution as a representation of each document, similar to how we have previously represented documents by word count or TF-IDF. This gives us a way of computing distances between documents, so that we can run a nearest neighbors search for a given document based on its membership in the topics that we learned from LDA. We can contrast the results with those obtained by running nearest neighbors under the usual TF-IDF representation, an approach that we explored in a previous assignment.
We'll start by creating the LDA topic distribution representation for each document:
Can you write Python code for it?

wiki['lda'] = topic_model.predict(wiki_docs, output_type='probability')


I am working on the file "Clustering_&_Retrieval/Week5/5_lda_blank.ipynb".
The first task is:
We propose the following themes for each topic:

topic 0: Science and research
topic 2: Team sports
topic 3: Music, TV, and film
topic 4: American college and politics
topic 5: General politics
topic 6: Art and publishing
topic 7: Business
topic 8: International athletics
topic 9: Great Britain and Australia
topic 10: International music

We'll save these themes for later:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
And here is the content of the other cluster we named right_child.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
As we did in previous assignments, let's extract the TF-IDF features:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
Cluster of ice hockey players and football players
In answering the following quiz question, take a look at the topics represented in the top documents (those closest to the centroid), as well as the list of words with highest TF-IDF weights.
Quiz Question. Bipartition the cluster of ice hockey and football players. Which of the two child clusters should be futher subdivided?
Note. To achieve consistent results, use the arguments maxiter=100, num_runs=8, seed=1 when calling the bipartition function.

The left child cluster
The right child cluster

Caution. The granularity criteria is an imperfect heuristic and must be taken with a grain of salt. It takes a lot of manual intervention to obtain a good hierarchy of clusters.

If a cluster is highly mixed, the top articles and words may not convey the full picture of the cluster. Thus, we may be misled if we judge the purity of clusters solely by their top documents and words.
Many interesting topics are hidden somewhere inside the clusters but do not appear in the visualization. We may need to subdivide further to discover new topics. For instance, subdividing the ice_hockey_football cluster led to the appearance of golf.

Quiz Question. Which diagram best describes the hierarchy right after splitting the ice_hockey_football cluster? Refer to the quiz form for the diagrams.
Cluster of non-athletes
Now let us subdivide the cluster of non-athletes.
Can you write Python code for it?

# Bipartition the cluster of non-athletes
left_child_non_athletes, right_child_non_athletes = bipartition(non_athletes, maxiter=100, num_runs=8, seed=1)

display_single_tf_idf_cluster(left_child_non_athletes, map_index_to_word)

display_single_tf_idf_cluster(right_child_non_athletes, map_index_to_word)


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
Let's examine the contents of one of the two clusters, which we call the left_child, referring to the tree visualization above.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
Let's visualize the two child clusters:
Can you write Python code for it?

display_single_tf_idf_cluster(left_child, map_index_to_word)

display_single_tf_idf_cluster(right_child, map_index_to_word)


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
Load the Wikipedia dataset
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
On the other hand, the right child cluster is a mix of football players and ice hockey players:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
The first cluster consists of scholars, politicians, and government officials whereas the second consists of musicians, artists, and actors. Run the following code cell to make convenient aliases for the clusters.
Can you write Python code for it?

scholars_politicians_etc = left_child_non_athletes
musicians_artists_etc = right_child_non_athletes


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
The following cell performs bipartitioning of the Wikipedia dataset. Allow 20-60 seconds to finish.
Note. For the purpose of the assignment, we set an explicit seed (seed=1) to produce identical outputs for every run. In pratical applications, you might want to use different random seeds for all runs.
Can you write Python code for it?

wiki_data = {'matrix': tf_idf, 'dataframe': wiki} # no 'centroid' for the root cluster
left_child, right_child = bipartition(wiki_data, maxiter=100, num_runs=8, seed=1)


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
The left child cluster mainly consists of baseball players:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
The left cluster consists of athletes, whereas the right cluster consists of non-athletes. So far, we have a single-level hierarchy consisting of two clusters, as follows:
Wikipedia
                                               +
                                               |
                    +--------------------------+--------------------+
                    |                                               |
                    +                                               +
                 Athletes                                      Non-athletes
Is this hierarchy good enough? When building a hierarchy of clusters, we must keep our particular application in mind. For instance, we might want to build a directory for Wikipedia articles. A good directory would let you quickly narrow down your search to a small set of related articles. The categories of athletes and non-athletes are too general to facilitate efficient search. For this reason, we decide to build another level into our hierarchy of clusters with the goal of getting more specific cluster structure at the lower level. To that end, we subdivide both the athletes and non-athletes clusters.
Perform recursive bipartitioning
Cluster of athletes
To help identify the clusters we've built so far, let's give them easy-to-read aliases:
Can you write Python code for it?

athletes = left_child
non_athletes = right_child


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
To be consistent with the k-means assignment, let's normalize all vectors to have unit norm.
Can you write Python code for it?

from sklearn.preprocessing import normalize
tf_idf = normalize(tf_idf)


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
To run k-means on this dataset, we should convert the data matrix into a sparse matrix.
Can you write Python code for it?

from em_utilities import sframe_to_scipy # converter

# This will take about a minute or two.
tf_idf, map_index_to_word = sframe_to_scipy(wiki, 'tf_idf')


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
Using the bipartition function, we produce two child clusters of the athlete cluster:
Can you write Python code for it?

# Bipartition the cluster of athletes
left_child_athletes, right_child_athletes = bipartition(athletes, maxiter=100, num_runs=8, seed=1)


I am working on the file "Clustering_&_Retrieval/Week6/.ipynb_checkpoints/6_hierarchical_clustering_blank-checkpoint.ipynb".
The first task is:
Visualize the bipartition
We provide you with a modified version of the visualization function from the k-means assignment. For each cluster, we print the top 5 words with highest TF-IDF weights in the centroid and display excerpts for the 8 nearest neighbors of the centroid.
Can you write Python code for it?

def display_single_tf_idf_cluster(cluster, map_index_to_word):
    '''map_index_to_word: SFrame specifying the mapping betweeen words and column indices'''

    wiki_subset   = cluster['dataframe']
    tf_idf_subset = cluster['matrix']
    centroid      = cluster['centroid']

    # Print top 5 words with largest TF-IDF weights in the cluster
    idx = centroid.argsort()[::-1]
    for i in xrange(5):
        print('{0:s}:{1:.3f}'.format(map_index_to_word['category'][idx[i]], centroid[idx[i]])),
    print('')

    # Compute distances from the centroid to all data points in the cluster.
    distances = pairwise_distances(tf_idf_subset, [centroid], metric='euclidean').flatten()
    # compute nearest neighbors of the centroid within the cluster.
    nearest_neighbors = distances.argsort()
    # For 8 nearest neighbors, print the title as well as first 180 characters of text.
    # Wrap the text at 80-character mark.
    for i in xrange(8):
        text = ' '.join(wiki_subset[nearest_neighbors[i]]['text'].split(None, 25)[0:25])
        print('* {0:50s} {1:.5f}\n  {2:s}\n  {3:s}'.format(wiki_subset[nearest_neighbors[i]]['name'],
              distances[nearest_neighbors[i]], text[:90], text[90:180] if len(text) > 90 else ''))
    print('')


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
And here is the content of the other cluster we named right_child.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
As we did in previous assignments, let's extract the TF-IDF features:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
Bipartition the Wikipedia dataset using k-means
Recall our workflow for clustering text data with k-means:

Load the dataframe containing a dataset, such as the Wikipedia text dataset.
Extract the data matrix from the dataframe.
Run k-means on the data matrix with some value of k.
Visualize the clustering results using the centroids, cluster assignments, and the original dataframe. We keep the original dataframe around because the data matrix does not keep auxiliary information (in the case of the text dataset, the title of each article).

Let us modify the workflow to perform bipartitioning:

Load the dataframe containing a dataset, such as the Wikipedia text dataset.
Extract the data matrix from the dataframe.
Run k-means on the data matrix with k=2.
Divide the data matrix into two parts using the cluster assignments.
Divide the dataframe into two parts, again using the cluster assignments. This step is necessary to allow for visualization.
Visualize the bipartition of data.

We'd like to be able to repeat Steps 3-6 multiple times to produce a hierarchy of clusters such as the following:
(root)
                         |
            +------------+-------------+
            |                          |
         Cluster                    Cluster
     +------+-----+             +------+-----+
     |            |             |            |
   Cluster     Cluster       Cluster      Cluster
Each parent cluster is bipartitioned to produce two child clusters. At the very top is the root cluster, which consists of the entire dataset.
Now we write a wrapper function to bipartition a given cluster using k-means. There are three variables that together comprise the cluster:

dataframe: a subset of the original dataframe that correspond to member rows of the cluster
matrix: same set of rows, stored in sparse matrix format
centroid: the centroid of the cluster (not applicable for the root cluster)

Rather than passing around the three variables separately, we package them into a Python dictionary. The wrapper function...
Can you write Python code for it?

def bipartition(cluster, maxiter=400, num_runs=4, seed=None):
    '''cluster: should be a dictionary containing the following keys
                * dataframe: original dataframe
                * matrix:    same data, in matrix format
                * centroid:  centroid for this particular cluster'''

    data_matrix = cluster['matrix']
    dataframe   = cluster['dataframe']

    # Run k-means on the data matrix with k=2. We use scikit-learn here to simplify workflow.
    kmeans_model = KMeans(n_clusters=2, max_iter=maxiter, n_init=num_runs, random_state=seed, n_jobs=-1)
    kmeans_model.fit(data_matrix)
    centroids, cluster_assignment = kmeans_model.cluster_centers_, kmeans_model.labels_

    # Divide the data matrix into two parts using the cluster assignments.
    data_matrix_left_child, data_matrix_right_child = data_matrix[cluster_assignment==0], \
                                                      data_matrix[cluster_assignment==1]

    # Divide the dataframe into two parts, again using the cluster assignments.
    cluster_assignment_sa = graphlab.SArray(cluster_assignment) # minor format conversion
    dataframe_left_child, dataframe_right_child     = dataframe[cluster_assignment_sa==0], \
                                                      dataframe[cluster_assignment_sa==1]

    # Package relevant variables for the child clusters
    cluster_left_child  = {'matrix': data_matrix_left_child,
                           'dataframe': dataframe_left_child,
                           'centroid': centroids[0]}
    cluster_right_child = {'matrix': data_matrix_right_child,
                           'dataframe': dataframe_right_child,
                           'centroid': centroids[1]}

    return (cluster_left_child, cluster_right_child)


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
Cluster of ice hockey players and football players
In answering the following quiz question, take a look at the topics represented in the top documents (those closest to the centroid), as well as the list of words with highest TF-IDF weights.
Quiz Question. Bipartition the cluster of ice hockey and football players. Which of the two child clusters should be futher subdivided?
Note. To achieve consistent results, use the arguments maxiter=100, num_runs=8, seed=1 when calling the bipartition function.

The left child cluster
The right child cluster
Can you write Python code for it?

left_child_ice_hockey_football, right_child_ice_hockey_football = bipartition(ice_hockey_football, maxiter=100, num_runs=8, seed=1)

display_single_tf_idf_cluster(left_child_ice_hockey_football, map_index_to_word)
display_single_tf_idf_cluster(right_child_ice_hockey_football, map_index_to_word)


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
Let's examine the contents of one of the two clusters, which we call the left_child, referring to the tree visualization above.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
Let's visualize the two child clusters:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
Load the Wikipedia dataset
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
Note. Concerning use of "football"
The occurrences of the word "football" above refer to association football. This sports is also known as "soccer" in United States (to avoid confusion with American football). We will use "football" throughout when discussing topic representation.
Our hierarchy of clusters now looks like this:
Wikipedia
                                               +
                                               |
                    +--------------------------+--------------------+
                    |                                               |
                    +                                               +
                 Athletes                                      Non-athletes
                    +
                    |
        +-----------+--------+
        |                    |
        |                    +
        +                 football/
     baseball            ice hockey
Should we keep subdividing the clusters? If so, which cluster should we subdivide? To answer this question, we again think about our application. Since we organize our directory by topics, it would be nice to have topics that are about as coarse as each other. For instance, if one cluster is about baseball, we expect some other clusters about football, basketball, volleyball, and so forth. That is, we would like to achieve similar level of granularity for all clusters.
Notice that the right child cluster is more coarse than the left child cluster. The right cluster posseses a greater variety of topics than the left (ice hockey/football vs. baseball). So the right child cluster should be subdivided further to produce finer child clusters.
Let's give the clusters aliases as well:
Can you write Python code for it?

baseball            = left_child_athletes
ice_hockey_football = right_child_athletes


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
Quiz Question. Let us bipartition the clusters scholars_politicians_etc and musicians_artists_etc. Which diagram best describes the resulting hierarchy of clusters for the non-athletes? Refer to the quiz for the diagrams.
Note. Use maxiter=100, num_runs=8, seed=1 for consistency of output.
Can you write Python code for it?

left_child_scholars_politicians_etc, right_child_scholars_politicians_etc = bipartition(scholars_politicians_etc, maxiter=100, num_runs=8, seed=1)

display_single_tf_idf_cluster(left_child_scholars_politicians_etc, map_index_to_word)
display_single_tf_idf_cluster(right_child_scholars_politicians_etc, map_index_to_word)

left_child_musicians_artists_etc, right_child_musicians_artists_etc = bipartition(musicians_artists_etc, maxiter=100, num_runs=8, seed=1)

display_single_tf_idf_cluster(left_child_musicians_artists_etc, map_index_to_word)
display_single_tf_idf_cluster(right_child_musicians_artists_etc, map_index_to_word)


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
The first cluster consists of scholars, politicians, and government officials whereas the second consists of musicians, artists, and actors. Run the following code cell to make convenient aliases for the clusters.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
The following cell performs bipartitioning of the Wikipedia dataset. Allow 20-60 seconds to finish.
Note. For the purpose of the assignment, we set an explicit seed (seed=1) to produce identical outputs for every run. In pratical applications, you might want to use different random seeds for all runs.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
The left child cluster mainly consists of baseball players:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
The left cluster consists of athletes, whereas the right cluster consists of non-athletes. So far, we have a single-level hierarchy consisting of two clusters, as follows:
Wikipedia
                                               +
                                               |
                    +--------------------------+--------------------+
                    |                                               |
                    +                                               +
                 Athletes                                      Non-athletes
Is this hierarchy good enough? When building a hierarchy of clusters, we must keep our particular application in mind. For instance, we might want to build a directory for Wikipedia articles. A good directory would let you quickly narrow down your search to a small set of related articles. The categories of athletes and non-athletes are too general to facilitate efficient search. For this reason, we decide to build another level into our hierarchy of clusters with the goal of getting more specific cluster structure at the lower level. To that end, we subdivide both the athletes and non-athletes clusters.
Perform recursive bipartitioning
Cluster of athletes
To help identify the clusters we've built so far, let's give them easy-to-read aliases:
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
To be consistent with the k-means assignment, let's normalize all vectors to have unit norm.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
To run k-means on this dataset, we should convert the data matrix into a sparse matrix.
Can you write Python code for it?


I am working on the file "Clustering_&_Retrieval/Week6/6_hierarchical_clustering_blank.ipynb".
The first task is:
Visualize the bipartition
We provide you with a modified version of the visualization function from the k-means assignment. For each cluster, we print the top 5 words with highest TF-IDF weights in the centroid and display excerpts for the 8 nearest neighbors of the centroid.
Can you write Python code for it?


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
All of the 10 people are politicians, but about half of them have rather tenuous connections with Obama, other than the fact that they are politicians.

Francisco Barrio is a Mexican politician, and a former governor of Chihuahua.
Walter Mondale and Don Bonker are Democrats who made their career in late 1970s.
Wynn Normington Hugh-Jones is a former British diplomat and Liberal Party official.
Andy Anstett is a former politician in Manitoba, Canada.

Nearest neighbors with raw word counts got some things right, showing all politicians in the query result, but missed finer and important details.
For instance, let's find out why Francisco Barrio was considered a close neighbor of Obama.  To do this, let's look at the most frequently used words in each of Barack Obama and Francisco Barrio's pages:
Can you write Python code for it?

def top_words(name):
    """
    Get a table of the most frequent words in the given person's wikipedia page.
    """
    row = wiki[wiki['name'] == name]
    word_count_table = row[['word_count']].stack('word_count', new_column_name=['word','count'])
    return word_count_table.sort('count', ascending=False)

obama_words = top_words('Barack Obama')
obama_words

barrio_words = top_words('Francisco Barrio')
barrio_words


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
But one may wonder, is Biden's article that different from Obama's, more so than, say, Schiliro's? It turns out that, when we compute nearest neighbors using the Euclidean distances, we unwittingly favor short articles over long ones. Let us compute the length of each Wikipedia document, and examine the document lengths for the 100 nearest neighbors to Obama's page.
Can you write Python code for it?

def compute_length(row):
    return len(row['text'].split(' '))

wiki['length'] = wiki.apply(compute_length)

nearest_neighbors_euclidean = model_tf_idf.query(wiki[wiki['name'] == 'Barack Obama'], label='name', k=100)
nearest_neighbors_euclidean = nearest_neighbors_euclidean.join(wiki[['name', 'length']], on={'reference_label':'name'})

nearest_neighbors_euclidean.sort('rank')


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Extract word count vectors
As we have seen in Course 1, we can extract word count vectors using a GraphLab utility function.  We add this as a column in wiki.
Can you write Python code for it?

wiki['word_count'] = graphlab.text_analytics.count_words(wiki['text'])

wiki


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Find nearest neighbors
Let's start by finding the nearest neighbors of the Barack Obama page using the word count vectors to represent the articles and Euclidean distance to measure distance.  For this, again will we use a GraphLab Create implementation of nearest neighbor search.
Can you write Python code for it?

model = graphlab.nearest_neighbors.create(wiki, label='name', features=['word_count'],
                                          method='brute_force', distance='euclidean')


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
From a glance at the above table, things look better.  For example, we now see Joe Biden as Barack Obama's nearest neighbor!  We also see Hillary Clinton on the list.  This list looks even more plausible as nearest neighbors of Barack Obama.
Let's make a plot to better visualize the effect of having used cosine distance in place of Euclidean on our TF-IDF vectors.
Can you write Python code for it?

plt.figure(figsize=(10.5,4.5))
plt.figure(figsize=(10.5,4.5))
plt.hist(wiki['length'], 50, color='k', edgecolor='None', histtype='stepfilled', normed=True,
         label='Entire Wikipedia', zorder=3, alpha=0.8)
plt.hist(nearest_neighbors_euclidean['length'], 50, color='r', edgecolor='None', histtype='stepfilled', normed=True,
         label='100 NNs of Obama (Euclidean)', zorder=10, alpha=0.8)
plt.hist(nearest_neighbors_cosine['length'], 50, color='b', edgecolor='None', histtype='stepfilled', normed=True,
         label='100 NNs of Obama (cosine)', zorder=11, alpha=0.8)
plt.axvline(x=wiki['length'][wiki['name'] == 'Barack Obama'][0], color='k', linestyle='--', linewidth=4,
           label='Length of Barack Obama', zorder=2)
plt.axvline(x=wiki['length'][wiki['name'] == 'Joe Biden'][0], color='g', linestyle='--', linewidth=4,
           label='Length of Joe Biden', zorder=1)
plt.axis([0, 1000, 0, 0.04])
plt.legend(loc='best', prop={'size':15})
plt.title('Distribution of document length')
plt.xlabel('# of words')
plt.ylabel('Percentage')
plt.rcParams.update({'font.size': 16})
plt.tight_layout()


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Indeed, the 100 nearest neighbors using cosine distance provide a sampling across the range of document lengths, rather than just short articles like Euclidean distance provided.
Moral of the story: In deciding the features and distance measures, check if they produce results that make sense for your particular application.
Problem with cosine distances: tweets vs. long articles
Happily ever after? Not so fast. Cosine distances ignore all document lengths, which may be great in certain situations but not in others. For instance, consider the following (admittedly contrived) example.
+--------------------------------------------------------+
|                                             +--------+ |
|  One that shall not be named                | Follow | |
|  @username                                  +--------+ |
|                                                        |
|  Democratic governments control law in response to     |
|  popular act.                                          |
|                                                        |
|  8:05 AM - 16 May 2016                                 |
|                                                        |
|  Reply   Retweet (1,332)   Like (300)                  |
|                                                        |
+--------------------------------------------------------+
How similar is this tweet to Barack Obama's Wikipedia article? Let's transform the tweet into TF-IDF features, using an encoder fit to the Wikipedia dataset.  (That is, let's treat this tweet as an article in our Wikipedia dataset and see what happens.)
Can you write Python code for it?

sf = graphlab.SFrame({'text': ['democratic governments control law in response to popular act']})
sf['word_count'] = graphlab.text_analytics.count_words(sf['text'])

encoder = graphlab.feature_engineering.TFIDF(features=['word_count'], output_column_prefix='tf_idf')
encoder.fit(wiki)
sf = encoder.transform(sf)
sf


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Let's compare this distance to the distance between the Barack Obama article and all of its Wikipedia 10 nearest neighbors:
Can you write Python code for it?


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Let's determine whether this list makes sense.
* With a notable exception of Roland Grossenbacher, the other 8 are all American politicians who are contemporaries of Barack Obama.
* Phil Schiliro, Jesse Lee, Samantha Power, and Eric Stern worked for Obama.
Clearly, the results are more plausible with the use of TF-IDF. Let's take a look at the word vector for Obama and Schilirio's pages. Notice that TF-IDF representation assigns a weight to each word. This weight captures relative importance of that word in the document. Let us sort the words in Obama's article by their TF-IDF weights; we do the same for Schiliro's article as well.
Can you write Python code for it?

def top_words_tf_idf(name):
    row = wiki[wiki['name'] == name]
    word_count_table = row[['tf_idf']].stack('tf_idf', new_column_name=['word','weight'])
    return word_count_table.sort('weight', ascending=False)

obama_tf_idf = top_words_tf_idf('Barack Obama')
obama_tf_idf

schiliro_tf_idf = top_words_tf_idf('Phil Schiliro')
schiliro_tf_idf


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Let's extract the list of most frequent words that appear in both Obama's and Barrio's documents. We've so far sorted all words from Obama and Barrio's articles by their word frequencies. We will now use a dataframe operation known as join. The join operation is very useful when it comes to playing around with data: it lets you combine the content of two tables using a shared column (in this case, the word column). See the documentation for more details.
For instance, running
obama_words.join(barrio_words, on='word')
will extract the rows from both tables that correspond to the common words.
Can you write Python code for it?

combined_words = obama_words.join(barrio_words, on='word')
combined_words


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Note. The join operation does not enforce any particular ordering on the shared column. So to obtain, say, the five common words that appear most often in Obama's article, sort the combined table by the Obama column. Don't forget ascending=False to display largest counts first.
Can you write Python code for it?


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Notice the huge difference in this calculation using TF-IDF scores instead  of raw word counts. We've eliminated noise arising from extremely common words.
Choosing metrics
You may wonder why Joe Biden, Obama's running mate in two presidential elections, is missing from the query results of model_tf_idf. Let's find out why. First, compute the distance between TF-IDF features of Obama and Biden.
Quiz Question. Compute the Euclidean distance between TF-IDF features of Obama and Biden. Hint: When using Boolean filter in SFrame/SArray, take the index 0 to access the first match.
Can you write Python code for it?

obama_tfidf = wiki[wiki['name']=='Barack Obama']['tf_idf'][0]
bush_tfidf = wiki[wiki['name']=='George W. Bush']['tf_idf'][0]
biden_tfidf = wiki[wiki['name']=='Joe Biden']['tf_idf'][0]
from graphlab.toolkits.distances import euclidean
print euclidean(obama_tfidf, biden_tfidf)
print euclidean(bush_tfidf, biden_tfidf)
print euclidean(obama_tfidf, bush_tfidf)


I am working on the file "Clustering_0_nearest-neighbors-features-and-metrics_blank.ipynb".
The first task is:
Now, compute the cosine distance between the Barack Obama article and this tweet:
Can you write Python code for it?

obama_tf_idf = obama[0]['tf_idf']
graphlab.toolkits.distances.cosine(obama_tf_idf, tweet_tf_idf)