I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
3.2.2. LDA using python lda library
An alternative to gensim for LDA is the lda library from python. It requires a doc-frequency matrix as input
Can you write Python code for it?

# For testing LDA, you can use the reuters dataset
# X = lda.datasets.load_reuters()
# vocab = lda.datasets.load_reuters_vocab()
# titles = lda.datasets.load_reuters_titles()
X = np.int32(np.zeros((n_art, n_tokens)))
for n, art in enumerate(corpus_bow):
    for t in art:
        X[n, t[0]] = t[1]
print X.shape
print X.sum()

vocab = D.values()
titles = corpus_titles

# Default parameters:
# model = lda.LDA(n_topics, n_iter=2000, alpha=0.1, eta=0.01, random_state=None, refresh=10)
model = lda.LDA(n_topics=10, n_iter=1500, random_state=1)
model.fit(X)  # model.fit_transform(X) is also available
topic_word = model.topic_word_  # model.components_ also works

# Show topics...
n_top_words = 8
for i, topic_dist in enumerate(topic_word):
    topic_words = np.array(vocab)[np.argsort(topic_dist)][:-(n_top_words+1):-1]
    print('Topic {}: {}'.format(i, ' '.join(topic_words)))


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
Alternatively, we can apply lemmatization. For english texts, we can use the lemmatizer from NLTK, which is based on WordNet. If you have not used wordnet before, you will likely need to download it from nltk
Can you write Python code for it?

# You can comment this if the package is already available.
nltk.download("wordnet")


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
Before starting with the semantic analyisis, it is interesting to observe the token distribution for the given corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (I):
# Create a "flat" corpus with all tuples in a single list
corpus_bow_flat = [item for sublist in corpus_bow for item in sublist]

# Initialize a numpy array that we will use to count tokens.
# token_count[n] should store the number of ocurrences of the n-th token, D[n]
token_count = np.zeros(n_tokens)

# Count the number of occurrences of each token.
for x in corpus_bow_flat:
    # Update the proper element in token_count
    # scode: <FILL IN>
    token_count[x[0]] += x[1]

# Sort by decreasing number of occurences
ids_sorted = np.argsort(- token_count)
tf_sorted = token_count[ids_sorted]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
Exercise: There are usually many tokens that appear with very low frequency in the corpus. Count the number of tokens appearing only once, and what is the proportion of them in the token list.
Can you write Python code for it?

# scode: <WRITE YOUR CODE HERE>
# Example data
cold_tokens = [D[i] for i in ids_sorted if tf_sorted[i]==1]

print "There are {0} cold tokens, which represent {1}% of the total number of tokens in the dictionary".format(
    len(cold_tokens), float(len(cold_tokens))/n_tokens*100)


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
From LSI, we can check both the topic-tokens matrix and the document-topics matrix.
Now we can check the topics generated by LSI. An intuitive visualization is provided by the show_topics method.
Can you write Python code for it?

lsi.show_topics(num_topics=-1, num_words=10, log=False, formatted=True)


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
However, a more useful representation of topics is as a list of tuples (token, value). This is provided by the show_topic method.
Task: Represent the columns of the topic-token matrix as a series of bar diagrams (one per topic) with the top 25 tokens of each topic.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

n_bins = 25

# Example data
y_pos = range(n_bins-1, -1, -1)

# pylab.rcParams['figure.figsize'] = 16, 8  # Set figure size
plt.figure(figsize=(16, 8))

for i in range(n_topics):

    ### Plot top 25 tokens for topic i
    # Read i-thtopic
    # scode: <FILL IN>
    topic_i = lsi.show_topic(i, topn=n_bins)
    tokens = [t[0] for t in topic_i]
    weights = [t[1] for t in topic_i]

    # Plot
    # scode: <FILL IN>
    plt.subplot(1, n_topics, i+1)
    plt.barh(y_pos, weights, align='center', alpha=0.4)
    plt.yticks(y_pos, tokens)
    plt.xlabel('Top {0} topic weights'.format(n_bins))
    plt.title('Topic {0}'.format(i))

plt.show()
display()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
In the second step, let us create a numerical version of our corpus using the doc2bow method. In general, D.doc2bow(token_list) transform any list of tokens into a list of tuples (token_id, n), one per each token in token_list, where token_id is the token identifier (according to dictionary D) and n is the number of occurrences of such token in token_list.
 Task: Apply the doc2bow method from gensim dictionary D, to all tokens in every article in corpus_clean. The result must be a new list named corpus_bow where each element is a list of tuples (token_id, number_of_occurrences).
Can you write Python code for it?

# Transform token lists into sparse vectors on the D-space
# scode: corpus_bow = <FILL IN>
corpus_bow = [D.doc2bow(doc) for doc in corpus_clean]

Test.assertTrue(len(corpus_bow)==n_art, 'corpus_bow has not the appropriate size')


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
It allows incremental updates
3.2.2. LDA using Sci-kit Learn
The input matrix to the sklearn implementation of LDA contains the token-counts for all documents in the corpus.
sklearn contains a powerfull CountVectorizer method that can be used to construct the input matrix from the corpus_bow.
First, we will define an auxiliary function to print the top tokens in the model, that has been taken from the sklearn documentation.
Can you write Python code for it?

# Adapted from an example in sklearn site
# http://scikit-learn.org/dev/auto_examples/applications/topics_extraction_with_nmf_lda.html

# You can try also with the dataset provided by sklearn in
# from sklearn.datasets import fetch_20newsgroups
# dataset = fetch_20newsgroups(shuffle=True, random_state=1,
#                              remove=('headers', 'footers', 'quotes'))

def print_top_words(model, feature_names, n_top_words):
    for topic_idx, topic in enumerate(model.components_):
        print("Topic #%d:" % topic_idx)
        print(" ".join([feature_names[i]
                        for i in topic.argsort()[:-n_top_words - 1:-1]]))
    print()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
LSI approximates any document as a linear combination of the topic vectors. We can compute the topic weights for any input corpus entered as input to the lsi model.
Can you write Python code for it?

# On real corpora, target dimensionality of
# 200–500 is recommended as a “golden standard”
# Create a double wrapper over the original
# corpus bow  tfidf  fold-in-lsi
corpus_lsi = lsi[corpus_tfidf]
print corpus_lsi[0]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
Note that we can interpret each element of corpus_bow as a sparse_vector. For example, a list of tuples
[(0, 1), (3, 3), (5,2)]

for a dictionary of 10 elements can be represented as a vector, where any tuple (id, n) states that position id must take value n. The rest of positions must be zero.
[1, 0, 0, 3, 0, 2, 0, 0, 0, 0]

These sparse vectors will be the inputs to the topic modeling algorithms.
Note that, at this point, we have built a Dictionary containing
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
Now we are ready to compute the token counts.
Can you write Python code for it?

# Use tf (raw term count) features for LDA.
print("Extracting tf features for LDA...")
n_features = 1000
n_samples = 2000
tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                                max_features=n_features,
                                stop_words='english')

t0 = time()
tf = tf_vectorizer.fit_transform(data_samples)
print("done in %0.3fs." % (time() - t0))
print tf[0][0][0]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
Now we can apply the LDA algorithm.
Task: Create an LDA object with the following parameters:
    n_topics=n_topics, max_iter=5,
    learning_method='online',
    learning_offset=50.,
    random_state=0
Can you write Python code for it?

print("Fitting LDA models with tf features, "
      "n_samples=%d and n_features=%d..."
      % (n_samples, n_features))
# scode: lda = <FILL IN>
lda = LatentDirichletAllocation(n_topics=n_topics, max_iter=10,
                                learning_method='online', learning_offset=50., random_state=0)
#                                 doc_topic_prior= 1.0/n_topics, topic_word_prior= 1.0/n_topics)


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
Now, we have stored the whole text collection in two lists:

corpus_titles, which contains the titles of the selected articles
corpus_text, with the text content of the selected wikipedia articles

You can browse the content of the wikipedia articles to get some intuition about the kind of documents that will be processed.
Can you write Python code for it?

# n = 5
# print corpus_titles[n]
# print corpus_text[n]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
Or to apply a transformation to a whole corpus
Can you write Python code for it?

corpus_tfidf = tfidf[corpus_bow]
print corpus_tfidf[0][0:5]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
Task: Find the document with the largest positive weight for topic 0. Compare the document and the topic.
Can you write Python code for it?

# Extract weights from corpus_lsi
# scode weight0 = <FILL IN>
weight0 = [doc[0][1] if doc != [] else -np.inf for doc in corpus_lsi]

# Locate the maximum positive weight
nmax = np.argmax(weight0)
print nmax
print weight0[nmax]
print corpus_lsi[nmax]

# Get topic 0
# scode: topic_0 = <FILL IN>
topic_0 = lsi.show_topic(0, topn=n_bins)

# Compute a list of tuples (token, wordcount) for all tokens in topic_0, where wordcount is the number of
# occurences of the token in the article.
# scode: token_counts = <FILL IN>
token_counts = [(t[0], corpus_clean[nmax].count(t[0])) for t in topic_0]

print "Topic 0 is:"
print topic_0
print "Token counts:"
print token_counts


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
Task: Fit model lda with the token frequencies computed by tf_vectorizer.
Can you write Python code for it?

t0 = time()
corpus_lda = lda.fit_transform(tf)
print corpus_lda[10]/np.sum(corpus_lda[10])
print("done in %0.3fs." % (time() - t0))
print corpus_titles[10]
# print corpus_text[10]

print("\nTopics in LDA model:")
tf_feature_names = tf_vectorizer.get_feature_names()
print_top_words(lda, tf_feature_names, 20)

topics = lda.components_
topic_probs = [t/np.sum(t) for t in topics]
#print topic_probs[0]
print -np.sort(-topic_probs[0])


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
You can try with any other categories. Take into account that the behavior of topic modelling algorithms may depend on the amount of documents available for the analysis. Select a category with at least 100 articles. You can browse the wikipedia category tree here, https://en.wikipedia.org/wiki/Category:Contents, for instance.
We start downloading the text collection.
Can you write Python code for it?

# Loading category data. This may take a while
print "Loading category data. This may take a while..."
cat_data = category.Category(site, cat)

corpus_titles = []
corpus_text = []

for n, page in enumerate(cat_data.getAllMembersGen()):
    print "\r Loading article {0}".format(n + 1),
    corpus_titles.append(page.title)
    corpus_text.append(page.getWikiText())

n_art = len(corpus_titles)
print "\nLoaded " + str(n_art) + " articles from category " + cat


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
and a bow representation of a corpus with
Can you write Python code for it?

print "{0} Wikipedia articles".format(len(corpus_bow))


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
ids_sorted is a list of all token ids, sorted by decreasing number of occurrences in the whole corpus. For instance, the most frequent term is
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db.ipynb".
The first task is:
which appears
Can you write Python code for it?

print "{0} times in the whole corpus".format(tf_sorted[0])


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
2. Corpus Processing
Topic modelling algorithms process vectorized data. In order to apply them, we need to transform the raw text input data into a vector representation. To do so, we will remove irrelevant information from the text data and preserve as much relevant information as possible to capture the semantic content in the document collection.
Thus, we will proceed with the following steps:

Tokenization, filtering and cleaning
Homogeneization (stemming or lemmatization)
Vectorization

2.1. Tokenization, filtering and cleaning.
The first steps consists on the following:

Tokenization: convert text string into lists of tokens.
Filtering:
Removing capitalization: capital alphabetic characters will be transformed to their corresponding lowercase characters.
Removing non alphanumeric tokens (e.g. punktuation signs)
Cleaning: Removing stopwords, i.e., those words that are very common in language and do not carry out useful semantic content (articles, pronouns, etc).

To do so, we will need some packages from the Natural Language Toolkit.
Can you write Python code for it?

# You can comment this if the package is already available.
# nltk.download("punkt")
# nltk.download("stopwords")

stopwords_en = stopwords.words('english')
corpus_clean = []

for n, art in enumerate(corpus_text):
    print "\rProcessing article {0} out of {1}".format(n + 1, n_art),
    # This is to make sure that all characters have the appropriate encoding.
    art = art.decode('utf-8')

    # Tokenize each text entry.
    # scode: tokens = <FILL IN>
    token_list = word_tokenize(art)

    # Convert all tokens in token_list to lowercase, remove non alfanumeric tokens and stem.
    # Store the result in a new token list, clean_tokens.
    # scode: filtered_tokens = <FILL IN>
    filtered_tokens = [token.lower() for token in token_list if token.isalnum()]

    # Remove all tokens in the stopwords list and append the result to corpus_clean
    # scode: clean_tokens = <FILL IN>
    clean_tokens = [token for token in filtered_tokens if token not in stopwords_en]

    # scode: <FILL IN>
    corpus_clean.append(clean_tokens)
print "\nLet's check the first tokens from document 0 after processing:"
print corpus_clean[0][0:30]

Test.assertTrue(len(corpus_clean) == n_art, 'List corpus_clean does not contain the expected number of articles')
Test.assertTrue(len([c for c in corpus_clean[0] if c in stopwords_en])==0, 'Stopwords have not been removed')


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
2.2. Stemming vs Lemmatization
At this point, we can choose between applying a simple stemming or ussing lemmatization. We will try both to test their differences.
Task: Apply the .stem() method, from the stemmer object created in the first line, to corpus_filtered.
Can you write Python code for it?

# Select stemmer.
stemmer = nltk.stem.SnowballStemmer('english')
corpus_stemmed = []

for n, token_list in enumerate(corpus_clean):
    print "\rStemming article {0} out of {1}".format(n + 1, n_art),

    # Convert all tokens in token_list to lowercase, remove non alfanumeric tokens and stem.
    # Store the result in a new token list, clean_tokens.
    # scode: stemmed_tokens = <FILL IN>
    stemmed_tokens = [stemmer.stem(token) for token in token_list]

    # Add art to the stemmed corpus
    # scode: <FILL IN>
    corpus_stemmed.append(stemmed_tokens)

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_stemmed[0][0:30]

Test.assertTrue((len([c for c in corpus_stemmed[0] if c!=stemmer.stem(c)]) < 0.1*len(corpus_stemmed[0])),
                'It seems that stemming has not been applied properly')


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
3. Semantic Analysis
The dictionary D and the Bag of Words in corpus_bow are the key inputs to the topic model algorithms. In this section we will explore two algorithms:

Latent Semantic Indexing (LSI)
Latent Dirichlet Allocation (LDA)

The topic model algorithms in gensim assume that input documents are parameterized using the tf-idf model. This can be done using
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
3.1. Latent Semantic Indexing (LSI)
Now we are ready to apply a topic modeling algorithm. Latent Semantic Indexing is provided by LsiModel.
Task: Generate a LSI model with 5 topics for corpus_tfidf and dictionary D. You can check de sintaxis for gensim.models.LsiModel.
Can you write Python code for it?

# Initialize an LSI transformation
n_topics = 5

# scode: lsi = <FILL IN>


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
Alternatively, we can apply lemmatization. For english texts, we can use the lemmatizer from NLTK, which is based on WordNet. If you have not used wordnet before, you will likely need to download it from nltk
Can you write Python code for it?

# You can comment this if the package is already available.
# nltk.download("wordnet")


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
Before starting with the semantic analyisis, it is interesting to observe the token distribution for the given corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (I):
# Create a "flat" corpus with all tuples in a single list
corpus_bow_flat = [item for sublist in corpus_bow for item in sublist]

# Initialize a numpy array that we will use to count tokens.
# token_count[n] should store the number of ocurrences of the n-th token, D[n]
token_count = np.zeros(n_tokens)

# Count the number of occurrences of each token.
for x in corpus_bow_flat:
    # Update the proper element in token_count
    # scode: <FILL IN>
    token_count[x[0]] += x[1]

# Sort by decreasing number of occurences
ids_sorted = np.argsort(- token_count)
tf_sorted = token_count[ids_sorted]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
Document-topic distribution
Can you write Python code for it?

doc_topic = model.doc_topic_
for i in range(10):
    print("{} (top topic: {})".format(titles[i], doc_topic[i].argmax()))

# This is to apply the model to a new doc(s)
# doc_topic_test = model.transform(X_test)
# for title, topics in zip(titles_test, doc_topic_test):
#    print("{} (top topic: {})".format(title, topics.argmax()))


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
Exercise: Represent graphically those 20 tokens that appear in the highest number of articles. Note that you can use the code above (headed by # SORTED TOKEN FREQUENCIES) with a very minor modification.
Can you write Python code for it?

# scode: <WRITE YOUR CODE HERE>

# SORTED TOKEN FREQUENCIES (I):
# Count the number of occurrences of each token.
token_count2 = np.zeros(n_tokens)
for x in corpus_bow_flat:
    token_count2[x[0]] += (x[1]>0)

# Sort by decreasing number of occurences
ids_sorted2 = np.argsort(- token_count2)
tf_sorted2 = token_count2[ids_sorted2]

# SORTED TOKEN FREQUENCIES (II):
# Example data
n_bins = 25
hot_tokens2 = [D[i] for i in ids_sorted2[n_bins-1::-1]]
y_pos2 = np.arange(len(hot_tokens2))
z2 = tf_sorted2[n_bins-1::-1]/n_art

plt.figure()
plt.barh(y_pos2, z2, align='center', alpha=0.4)
plt.yticks(y_pos2, hot_tokens2)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()
display()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
Exercise: There are usually many tokens that appear with very low frequency in the corpus. Count the number of tokens appearing only once, and what is the proportion of them in the token list.
Can you write Python code for it?

# scode: <WRITE YOUR CODE HERE>
# Example data
cold_tokens = [D[i] for i in ids_sorted if tf_sorted[i]==1]

print "There are {0} cold tokens, which represent {1}% of the total number of tokens in the dictionary".format(
    len(cold_tokens), float(len(cold_tokens))/n_tokens*100)


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
From LSI, we can check both the topic-tokens matrix and the document-topics matrix.
Now we can check the topics generated by LSI. An intuitive visualization is provided by the show_topics method.
Can you write Python code for it?

lsi.show_topics(num_topics=-1, num_words=10, log=False, formatted=True)


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
From now on, tfidf can be used to convert any vector from the old representation (bow integer counts) to the new one (TfIdf real-valued weights):
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
However, a more useful representation of topics is as a list of tuples (token, value). This is provided by the show_topic method.
Task: Represent the columns of the topic-token matrix as a series of bar diagrams (one per topic) with the top 25 tokens of each topic.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

n_bins = 25

# Example data
y_pos = range(n_bins-1, -1, -1)

# pylab.rcParams['figure.figsize'] = 16, 8  # Set figure size
plt.figure(figsize=(16, 8))

for i in range(n_topics):

    ### Plot top 25 tokens for topic i
    # Read i-thtopic
    # topic_i =  <FILL IN>
    # tokens =  <FILL IN>
    # weights =  <FILL IN>

    # Plot
    plt.subplot(1, n_topics, i+1)
    plt.barh(# scode: <FILL IN>
    plt.yticks(y_pos, tokens)
    plt.xlabel('Top {0} topic weights'.format(n_bins))
    plt.title('Topic {0}'.format(i))

plt.show()
display()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
In the following we plot the most frequent terms in the corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

# Example data
n_bins = 25
hot_tokens = [D[i] for i in ids_sorted[n_bins-1::-1]]
y_pos = np.arange(len(hot_tokens))
z = tf_sorted[n_bins-1::-1]/n_art

plt.barh(y_pos, z, align='center', alpha=0.4)
plt.yticks(y_pos, hot_tokens)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()

# SORTED TOKEN FREQUENCIES:

# Example data
plt.semilogy(tf_sorted)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()
display()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
In the second step, let us create a numerical version of our corpus using the doc2bow method. In general, D.doc2bow(token_list) transform any list of tokens into a list of tuples (token_id, n), one per each token in token_list, where token_id is the token identifier (according to dictionary D) and n is the number of occurrences of such token in token_list.
 Task: Apply the doc2bow method from gensim dictionary D, to all tokens in every article in corpus_clean. The result must be a new list named corpus_bow where each element is a list of tuples (token_id, number_of_occurrences).
Can you write Python code for it?

# Transform token lists into sparse vectors on the D-space
# scode: corpus_bow = <FILL IN>
corpus_bow = [D.doc2bow(doc) for doc in corpus_clean]

Test.assertTrue(len(corpus_bow)==n_art, 'corpus_bow has not the appropriate size')


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
It allows incremental updates
3.2.2. LDA using Sci-kit Learn
The input matrix to the sklearn implementation of LDA contains the token-counts for all documents in the corpus.
sklearn contains a powerfull CountVectorizer method that can be used to construct the input matrix from the corpus_bow.
First, we will define an auxiliary function to print the top tokens in the model, that has been taken from the sklearn documentation.
Can you write Python code for it?

# Adapted from an example in sklearn site
# http://scikit-learn.org/dev/auto_examples/applications/topics_extraction_with_nmf_lda.html

# You can try also with the dataset provided by sklearn in
# from sklearn.datasets import fetch_20newsgroups
# dataset = fetch_20newsgroups(shuffle=True, random_state=1,
#                              remove=('headers', 'footers', 'quotes'))

def print_top_words(model, feature_names, n_top_words):
    for topic_idx, topic in enumerate(model.components_):
        print("Topic #%d:" % topic_idx)
        print(" ".join([feature_names[i]
                        for i in topic.argsort()[:-n_top_words - 1:-1]]))
    print()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
LSI approximates any document as a linear combination of the topic vectors. We can compute the topic weights for any input corpus entered as input to the lsi model.
Can you write Python code for it?

# On real corpora, target dimensionality of
# 200–500 is recommended as a “golden standard”
# Create a double wrapper over the original
# corpus bow  tfidf  fold-in-lsi
corpus_lsi = lsi[corpus_tfidf]
print corpus_lsi[0]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
Now we are ready to compute the token counts.
Can you write Python code for it?

# Use tf (raw term count) features for LDA.
print("Extracting tf features for LDA...")
n_features = 1000
n_samples = 2000
tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                                max_features=n_features,
                                stop_words='english')

t0 = time()
tf = tf_vectorizer.fit_transform(data_samples)
print("done in %0.3fs." % (time() - t0))
print tf[0][0][0]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
Now we can apply the LDA algorithm.
Task: Create an LDA object with the following parameters:
    n_topics=n_topics, max_iter=5,
    learning_method='online',
    learning_offset=50.,
    random_state=0
Can you write Python code for it?

print("Fitting LDA models with tf features, "
      "n_samples=%d and n_features=%d..."
      % (n_samples, n_features))
# scode: lda = <FILL IN>


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
Now, we need a dataset to feed the Count_Vectorizer object, by joining all tokens in corpus_clean in a single string, using a space ' ' as separator.
Can you write Python code for it?

print("Loading dataset...")
# scode: data_samples = <FILL IN>

print 'Document 0:'
print data_samples[0][0:200], '...'


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
Or to apply a transformation to a whole corpus
Can you write Python code for it?

corpus_tfidf = tfidf[corpus_bow]
print corpus_tfidf[0][0:5]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
Task: Find the document with the largest positive weight for topic 0. Compare the document and the topic.
Can you write Python code for it?

# Extract weights from corpus_lsi
# scode weight0 = <FILL IN>
weight0 = [doc[0][1] for doc in corpus_lsi]

# Locate the maximum positive weight
nmax = np.argmax(weight0)
print nmax
print weight0[nmax]
print corpus_lsi[nmax]

# Get topic 0
# scode: topic_0 = <FILL IN>

# Compute a list of tuples (token, wordcount) for all tokens in topic_0, where wordcount is the number of
# occurences of the token in the article.
# scode: token_counts = <FILL IN>

print "Topic 0 is:"
print topic_0
print "Token counts:"
print token_counts


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
Task: Fit model lda with the token frequencies computed by tf_vectorizer.
Can you write Python code for it?

t0 = # scode: <FILL IN>
corpus_lda = # scode: <FILL IN>
print corpus_lda[0]/np.sum(corpus_lda[0])
print("done in %0.3fs." % (time() - t0))

print("\nTopics in LDA model:")
tf_feature_names = # scode: <FILL IN>
print_top_words(lda, tf_feature_names, n_top_words)

topics = lda.components_
topic_probs = [t/np.sum(t) for t in topics]
# print topic_probs[0]
print -np.sort(-topic_probs[0])


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
and a bow representation of a corpus with
Can you write Python code for it?

print "{0} Wikipedia articles".format(len(corpus_bow))


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/databricks/TM2_Topic_Modeling_db_student.ipynb".
The first task is:
ids_sorted is a list of all token ids, sorted by decreasing number of occurrences in the whole corpus. For instance, the most frequent term is
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
2. Corpus Processing
Topic modelling algorithms process vectorized data. In order to apply them, we need to transform the raw text input data into a vector representation. To do so, we will remove irrelevant information from the text data and preserve as much relevant information as possible to capture the semantic content in the document collection.
Thus, we will proceed with the following steps:

Tokenization, filtering and cleaning
Homogeneization (stemming or lemmatization)
Vectorization

2.1. Tokenization, filtering and cleaning.
The first steps consists on the following:

Tokenization: convert text string into lists of tokens.
Filtering:
Removing capitalization: capital alphabetic characters will be transformed to their corresponding lowercase characters.
Removing non alphanumeric tokens (e.g. punktuation signs)
Cleaning: Removing stopwords, i.e., those words that are very common in language and do not carry out useful semantic content (articles, pronouns, etc).

To do so, we will need some packages from the Natural Language Toolkit.
Can you write Python code for it?

# You can comment this if the package is already available.
# Select option "d) Download", and identifier "punkt"
# Select option "d) Download", and identifier "stopwords"
# nltk.download()

stopwords_en = stopwords.words('english')
corpus_clean = []

for n, art in enumerate(corpus_text):
    print "\rProcessing article {0} out of {1}".format(n + 1, n_art),
    # This is to make sure that all characters have the appropriate encoding.
    art = art.decode('utf-8')

    # Tokenize each text entry.
    # scode: tokens = <FILL IN>
    token_list = word_tokenize(art)

    # Convert all tokens in token_list to lowercase, remove non alfanumeric tokens and stem.
    # Store the result in a new token list, clean_tokens.
    # scode: filtered_tokens = <FILL IN>
    filtered_tokens = [token.lower() for token in token_list if token.isalnum()]

    # Remove all tokens in the stopwords list and append the result to corpus_clean
    # scode: clean_tokens = <FILL IN>
    clean_tokens = [token for token in filtered_tokens if token not in stopwords_en]

    # scode: <FILL IN>
    corpus_clean.append(clean_tokens)
print "\nLet's check the first tokens from document 0 after processing:"
print corpus_clean[0][0:30]

Test.assertTrue(len(corpus_clean) == n_art, 'List corpus_clean does not contain the expected number of articles')
Test.assertTrue(len([c for c in corpus_clean[0] if c in stopwords_en])==0, 'Stopwords have not been removed')


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
2.2. Stemming vs Lemmatization
At this point, we can choose between applying a simple stemming or ussing lemmatization. We will try both to test their differences.
Task: Apply the .stem() method, from the stemmer object created in the first line, to corpus_filtered.
Can you write Python code for it?

# Select stemmer.
stemmer = nltk.stem.SnowballStemmer('english')
corpus_stemmed = []

for n, token_list in enumerate(corpus_clean):
    print "\rStemming article {0} out of {1}".format(n + 1, n_art),

    # Convert all tokens in token_list to lowercase, remove non alfanumeric tokens and stem.
    # Store the result in a new token list, clean_tokens.
    # scode: stemmed_tokens = <FILL IN>
    stemmed_tokens = [stemmer.stem(token) for token in token_list]

    # Add art to the stemmed corpus
    # scode: <FILL IN>
    corpus_stemmed.append(stemmed_tokens)

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_stemmed[0][0:30]

Test.assertTrue((len([c for c in corpus_stemmed[0] if c!=stemmer.stem(c)]) < 0.1*len(corpus_stemmed[0])),
                'It seems that stemming has not been applied properly')


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
3. Semantic Analysis
The dictionary D and the Bag of Words in corpus_bow are the key inputs to the topic model algorithms. In this section we will explore two algorithms:

Latent Semantic Indexing (LSI)
Latent Dirichlet Allocation (LDA)

The topic model algorithms in gensim assume that input documents are parameterized using the tf-idf model. This can be done using
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
3.2. Latent Dirichlet Allocation (LDA)
There are several implementations of the LDA topic model in python:

Python library lda.
Gensim module: gensim.models.ldamodel.LdaModel
Sci-kit Learn module: sklearn.decomposition

3.2.1. LDA using Gensim
The use of the LDA module in gensim is similar to LSI. Furthermore, it assumes that a tf-idf parametrization is used as an input, which is not in complete agreement with the theoretical model, which assumes documents represented as vectors of token-counts.
To use LDA in gensim, we must first create a lda model object.
Can you write Python code for it?

ldag = gensim.models.ldamodel.LdaModel(
    corpus=corpus_tfidf, id2word=D, num_topics=10, update_every=1, passes=10)

ldag.print_topics()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
3.2.2. LDA using python lda library
An alternative to gensim for LDA is the lda library from python. It requires a doc-frequency matrix as input
Can you write Python code for it?

# For testing LDA, you can use the reuters dataset
# X = lda.datasets.load_reuters()
# vocab = lda.datasets.load_reuters_vocab()
# titles = lda.datasets.load_reuters_titles()
X = np.int32(np.zeros((n_art, n_tokens)))
for n, art in enumerate(corpus_bow):
    for t in art:
        X[n, t[0]] = t[1]
print X.shape
print X.sum()

vocab = D.values()
titles = corpus_titles

# Default parameters:
# model = lda.LDA(n_topics, n_iter=2000, alpha=0.1, eta=0.01, random_state=None, refresh=10)
model = lda.LDA(n_topics=10, n_iter=1500, random_state=1)
model.fit(X)  # model.fit_transform(X) is also available
topic_word = model.topic_word_  # model.components_ also works

# Show topics...
n_top_words = 8
for i, topic_dist in enumerate(topic_word):
    topic_words = np.array(vocab)[np.argsort(topic_dist)][:-(n_top_words+1):-1]
    print('Topic {}: {}'.format(i, ' '.join(topic_words)))


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
Alternatively, we can apply lemmatization. For english texts, we can use the lemmatizer from NLTK, which is based on WordNet. If you have not used wordnet before, you will likely need to download it from nltk
Can you write Python code for it?

# You can comment this if the package is already available.
# Select option "d) Download", and identifier "wordnet"
# nltk.download()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
Before starting with the semantic analyisis, it is interesting to observe the token distribution for the given corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (I):
# Create a "flat" corpus with all tuples in a single list
corpus_bow_flat = [item for sublist in corpus_bow for item in sublist]

# Initialize a numpy array that we will use to count tokens.
# token_count[n] should store the number of ocurrences of the n-th token, D[n]
token_count = np.zeros(n_tokens)

# Count the number of occurrences of each token.
for x in corpus_bow_flat:
    # Update the proper element in token_count
    # scode: <FILL IN>
    token_count[x[0]] += x[1]

# Sort by decreasing number of occurences
ids_sorted = np.argsort(- token_count)
tf_sorted = token_count[ids_sorted]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
Document-topic distribution
Can you write Python code for it?

doc_topic = model.doc_topic_
for i in range(10):
    print("{} (top topic: {})".format(titles[i], doc_topic[i].argmax()))

# This is to apply the model to a new doc(s)
# doc_topic_test = model.transform(X_test)
# for title, topics in zip(titles_test, doc_topic_test):
#    print("{} (top topic: {})".format(title, topics.argmax()))


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
Exercise: Represent graphically those 20 tokens that appear in the highest number of articles. Note that you can use the code above (headed by # SORTED TOKEN FREQUENCIES) with a very minor modification.
Can you write Python code for it?

# scode: <WRITE YOUR CODE HERE>

# SORTED TOKEN FREQUENCIES (I):
# Count the number of occurrences of each token.
token_count2 = np.zeros(n_tokens)
for x in corpus_bow_flat:
    token_count2[x[0]] += (x[1]>0)

# Sort by decreasing number of occurences
ids_sorted2 = np.argsort(- token_count2)
tf_sorted2 = token_count2[ids_sorted2]

# SORTED TOKEN FREQUENCIES (II):
# Example data
n_bins = 25
hot_tokens2 = [D[i] for i in ids_sorted2[n_bins-1::-1]]
y_pos2 = np.arange(len(hot_tokens2))
z2 = tf_sorted2[n_bins-1::-1]/n_art

plt.barh(y_pos2, z2, align='center', alpha=0.4)
plt.yticks(y_pos2, hot_tokens2)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
Exercise: There are usually many tokens that appear with very low frequency in the corpus. Count the number of tokens appearing only once, and what is the proportion of them in the token list.
Can you write Python code for it?

# scode: <WRITE YOUR CODE HERE>
# Example data
cold_tokens = [D[i] for i in ids_sorted if tf_sorted[i]==1]

print "There are {0} cold tokens, which represent {1}% of the total number of tokens in the dictionary".format(
    len(cold_tokens), float(len(cold_tokens))/n_tokens*100)


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
From LSI, we can check both the topic-tokens matrix and the document-topics matrix.
Now we can check the topics generated by LSI. An intuitive visualization is provided by the show_topics method.
Can you write Python code for it?

lsi.show_topics(num_topics=-1, num_words=10, log=False, formatted=True)


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
From now on, tfidf can be used to convert any vector from the old representation (bow integer counts) to the new one (TfIdf real-valued weights):
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
However, a more useful representation of topics is as a list of tuples (token, value). This is provided by the show_topic method.
Task: Represent the columns of the topic-token matrix as a series of bar diagrams (one per topic) with the top 25 tokens of each topic.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

n_bins = 25

# Example data
y_pos = range(n_bins-1, -1, -1)

pylab.rcParams['figure.figsize'] = 16, 8  # Set figure size
for i in range(n_topics):

    ### Plot top 25 tokens for topic i
    # Read i-thtopic
    # scode: <FILL IN>
    topic_i = lsi.show_topic(i, topn=n_bins)
    tokens = [t[0] for t in topic_i]
    weights = [t[1] for t in topic_i]

    # Plot
    # scode: <FILL IN>
    plt.subplot(1, n_topics, i+1)
    plt.barh(y_pos, weights, align='center', alpha=0.4)
    plt.yticks(y_pos, tokens)
    plt.xlabel('Top {0} topic weights'.format(n_bins))
    plt.title('Topic {0}'.format(i))

plt.show()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
In the second step, let us create a numerical version of our corpus using the doc2bow method. In general, D.doc2bow(token_list) transform any list of tokens into a list of tuples (token_id, n), one per each token in token_list, where token_id is the token identifier (according to dictionary D) and n is the number of occurrences of such token in token_list.
 Task: Apply the doc2bow method from gensim dictionary D, to all tokens in every article in corpus_clean. The result must be a new list named corpus_bow where each element is a list of tuples (token_id, number_of_occurrences).
Can you write Python code for it?

# Transform token lists into sparse vectors on the D-space
corpus_bow = [D.doc2bow(doc) for doc in corpus_clean]

Test.assertTrue(len(corpus_bow)==n_art, 'corpus_bow has not the appropriate size')


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
It allows incremental updates
3.2.2. LDA using Sci-kit Learn
The input matrix to the sklearn implementation of LDA contains the token-counts for all documents in the corpus.
sklearn contains a powerfull CountVectorizer method that can be used to construct the input matrix from the corpus_bow.
First, we will define an auxiliary function to print the top tokens in the model, that has been taken from the sklearn documentation.
Can you write Python code for it?

# Adapted from an example in sklearn site
# http://scikit-learn.org/dev/auto_examples/applications/topics_extraction_with_nmf_lda.html

# You can try also with the dataset provided by sklearn in
# from sklearn.datasets import fetch_20newsgroups
# dataset = fetch_20newsgroups(shuffle=True, random_state=1,
#                              remove=('headers', 'footers', 'quotes'))

def print_top_words(model, feature_names, n_top_words):
    for topic_idx, topic in enumerate(model.components_):
        print("Topic #%d:" % topic_idx)
        print(" ".join([feature_names[i]
                        for i in topic.argsort()[:-n_top_words - 1:-1]]))
    print()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
Now we are ready to compute the token counts.
Can you write Python code for it?

# Use tf (raw term count) features for LDA.
print("Extracting tf features for LDA...")
n_features = 1000
n_samples = 2000
tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                                max_features=n_features,
                                stop_words='english')

t0 = time()
tf = tf_vectorizer.fit_transform(data_samples)
print("done in %0.3fs." % (time() - t0))
print tf[0][0][0]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
Now we can apply the LDA algorithm.
Task: Create an LDA object with the following parameters:
    n_topics=n_topics, max_iter=5,
    learning_method='online',
    learning_offset=50.,
    random_state=0
Can you write Python code for it?

print("Fitting LDA models with tf features, "
      "n_samples=%d and n_features=%d..."
      % (n_samples, n_features))
# scode: lda = <FILL IN>
lda = LatentDirichletAllocation(n_topics=n_topics, max_iter=10,
                                learning_method='online', learning_offset=50., random_state=0)
#                                 doc_topic_prior= 1.0/n_topics, topic_word_prior= 1.0/n_topics)


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
Now, we have stored the whole text collection in two lists:

corpus_titles, which contains the titles of the selected articles
corpus_text, with the text content of the selected wikipedia articles

You can browse the content of the wikipedia articles to get some intuition about the kind of documents that will be processed.
Can you write Python code for it?

# n = 5
# print corpus_titles[n]
# print corpus_text[n]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
Now, we need a dataset to feed the Count_Vectorizer object, by joining all tokens in corpus_clean in a single string, using a space ' ' as separator.
Can you write Python code for it?

print("Loading dataset...")
# scode: data_samples = <FILL IN>
print "*".join(['Esto', 'es', 'un', 'ejemplo'])
data_samples = [" ".join(c) for c in corpus_clean]
print 'Document 0:'
print data_samples[0][0:200], '...'


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
One of the advantages of the lemmatizer method is that the result of lemmmatization is still a true word, which is more advisable for the presentation of text processing results and lemmatization.
However, without using contextual information, lemmatize() does not remove grammatical differences. This is the reason why "is" or "are" are preserved and not replaced by infinitive "be".
As an alternative, we can apply .lemmatize(word, pos), where 'pos' is a string code specifying the part-of-speech (pos), i.e. the grammatical role of the words in its sentence. For instance, you can check the difference between wnl.lemmatize('is') and wnl.lemmatize('is, pos='v').
2.3. Vectorization
Up to this point, we have transformed the raw text collection of articles in a list of articles, where each article is a collection of the word roots that are most relevant for semantic analysis. Now, we need to convert these data (a list of token lists) into a numerical representation (a list of vectors, or a matrix). To do so, we will start using the tools provided by the gensim library.
As a first step, we create a dictionary containing all tokens in our text corpus, and assigning an integer identifier to each one of them.
Can you write Python code for it?

# Create dictionary of tokens
D = gensim.corpora.Dictionary(corpus_clean)
n_tokens = len(D)

print "The dictionary contains {0} tokens".format(n_tokens)
print "First tokens in the dictionary: "
for n in range(10):
    print str(n) + ": " + D[n]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
Or to apply a transformation to a whole corpus
Can you write Python code for it?

corpus_tfidf = tfidf[corpus_bow]
print corpus_tfidf[0][0:5]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
Task: Apply the .lemmatize() method, from the WordNetLemmatizer object created in the first line, to corpus_filtered.
Can you write Python code for it?

wnl = WordNetLemmatizer()

# Select stemmer.
corpus_lemmat = []

for n, token_list in enumerate(corpus_clean):
    print "\rLemmatizing article {0} out of {1}".format(n + 1, n_art),

    # scode: lemmat_tokens = <FILL IN>
    lemmat_tokens = [wnl.lemmatize(token) for token in token_list]

    # Add art to the stemmed corpus
    # scode: <FILL IN>
    corpus_lemmat.append(lemmat_tokens)

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_lemmat[0][0:30]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
Task: Fit model lda with the token frequencies computed by tf_vectorizer.
Can you write Python code for it?

t0 = time()
corpus_lda = lda.fit_transform(tf)
print corpus_lda[10]/np.sum(corpus_lda[10])
print("done in %0.3fs." % (time() - t0))
print corpus_titles[10]
# print corpus_text[10]

print("\nTopics in LDA model:")
tf_feature_names = tf_vectorizer.get_feature_names()
print_top_words(lda, tf_feature_names, 20)

topics = lda.components_
topic_probs = [t/np.sum(t) for t in topics]
# print topic_probs[0]
print -np.sort(-topic_probs[0])


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
ids_sorted is a list of all token ids, sorted by decreasing number of occurrences in the whole corpus. For instance, the most frequent term is
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_professor.ipynb".
The first task is:
which appears
Can you write Python code for it?

print "{0} times in the whole corpus".format(tf_sorted[0])


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
2.2. Stemming vs Lemmatization
At this point, we can choose between applying a simple stemming or ussing lemmatization. We will try both to test their differences.
Task: Apply the .stem() method, from the stemmer object created in the first line, to corpus_filtered.
Can you write Python code for it?

# Select stemmer.
stemmer = nltk.stem.SnowballStemmer('english')
corpus_stemmed = []

for n, token_list in enumerate(corpus_clean):
    print "\rStemming article {0} out of {1}".format(n + 1, n_art),

    # Convert all tokens in token_list to lowercase, remove non alfanumeric tokens and stem.
    # Store the result in a new token list, clean_tokens.
    # scode: stemmed_tokens = <FILL IN>

    # Add art to the stemmed corpus
    # scode: <FILL IN>

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_stemmed[0][0:30]

Test.assertTrue((len([c for c in corpus_stemmed[0] if c!=stemmer.stem(c)]) < 0.1*len(corpus_stemmed[0])),
                'It seems that stemming has not been applied properly')


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
3.1. Latent Semantic Indexing (LSI)
Now we are ready to apply a topic modeling algorithm. Latent Semantic Indexing is provided by LsiModel.
Task: Generate a LSI model with 5 topics for corpus_tfidf and dictionary D. You can check de sintaxis for gensim.models.LsiModel.
Can you write Python code for it?

# Initialize an LSI transformation
n_topics = 5

# scode: lsi = <FILL IN>


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
3.2. Latent Dirichlet Allocation (LDA)
There are several implementations of the LDA topic model in python:

Python library lda.
Gensim module: gensim.models.ldamodel.LdaModel
Sci-kit Learn module: sklearn.decomposition

3.2.1. LDA using Gensim
The use of the LDA module in gensim is similar to LSI. Furthermore, it assumes that a tf-idf parametrization is used as an input, which is not in complete agreement with the theoretical model, which assumes documents represented as vectors of token-counts.
To use LDA in gensim, we must first create a lda model object.
Can you write Python code for it?

ldag = gensim.models.ldamodel.LdaModel(
    corpus=corpus_tfidf, id2word=D, num_topics=10, update_every=1, passes=10)

ldag.print_topics()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
3.2.2. LDA using python lda library
An alternative to gensim for LDA is the lda library from python. It requires a doc-frequency matrix as input
Can you write Python code for it?

# For testing LDA, you can use the reuters dataset
# X = lda.datasets.load_reuters()
# vocab = lda.datasets.load_reuters_vocab()
# titles = lda.datasets.load_reuters_titles()
X = np.int32(np.zeros((n_art, n_tokens)))
for n, art in enumerate(corpus_bow):
    for t in art:
        X[n, t[0]] = t[1]
print X.shape
print X.sum()

vocab = D.values()
titles = corpus_titles

# Default parameters:
# model = lda.LDA(n_topics, n_iter=2000, alpha=0.1, eta=0.01, random_state=None, refresh=10)
model = lda.LDA(n_topics=10, n_iter=1500, random_state=1)
model.fit(X)  # model.fit_transform(X) is also available
topic_word = model.topic_word_  # model.components_ also works

# Show topics...
n_top_words = 8
for i, topic_dist in enumerate(topic_word):
    topic_words = np.array(vocab)[np.argsort(topic_dist)][:-(n_top_words+1):-1]
    print('Topic {}: {}'.format(i, ' '.join(topic_words)))


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
At this point, it is good to make sure to understand what has happened. In corpus_clean we had a list of token lists. With it, we have constructed a Dictionary, D, which assign an integer identifier to each token in the corpus.
After that, we have transformed each article (in corpus_clean) in a list tuples (id, n).
Can you write Python code for it?

print "Original article (after cleaning): "
print corpus_clean[0][0:30]
print "Sparse vector representation (first 30 components):"
print corpus_bow[0][0:30]
print "The first component, {0} from document 0, states that token 0 ({1}) appears {2} times".format(
    corpus_bow[0][0], D[0], corpus_bow[0][0][1])


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
Document-topic distribution
Can you write Python code for it?

doc_topic = model.doc_topic_
for i in range(10):
    print("{} (top topic: {})".format(titles[i], doc_topic[i].argmax()))

# This is to apply the model to a new doc(s)
# doc_topic_test = model.transform(X_test)
# for title, topics in zip(titles_test, doc_topic_test):
#    print("{} (top topic: {})".format(title, topics.argmax()))


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
Exercise: Represent graphically the topic distributions for the top 25 tokens with highest probability for each topic.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

n_bins = 50

# Example data
y_pos = range(n_bins-1, -1, -1)

pylab.rcParams['figure.figsize'] = 16, 8  # Set figure size

for i in range(n_topics):

    ### Plot top 25 tokens for topic i
    # Read i-thtopic
    # scode: <FILL IN>
    rank = np.argsort(- topic_i)[0:n_bins]

    tokens = [tf_feature_names[r] for r in rank]
    weights = [topic_i[r] for r in rank]

    # Plot
    # scode: <FILL IN>
    plt.barh(y_pos, weights, align='center', alpha=0.4)
    plt.yticks(y_pos, tokens)
    plt.xlabel('Top {0} topic weights'.format(n_bins))
    plt.title('Topic {0}'.format(i))

plt.show()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
Exercise: Represent graphically those 20 tokens that appear in the highest number of articles. Note that you can use the code above (headed by # SORTED TOKEN FREQUENCIES) with a very minor modification.
Can you write Python code for it?

# scode: <WRITE YOUR CODE HERE>

# SORTED TOKEN FREQUENCIES (I):
# Count the number of occurrences of each token.
token_count2 = np.zeros(n_tokens)
for x in corpus_bow_flat:
    token_count2[x[0]] += (x[1]>0)

# Sort by decreasing number of occurences
ids_sorted2 = np.argsort(- token_count2)
tf_sorted2 = token_count2[ids_sorted2]

# SORTED TOKEN FREQUENCIES (II):
# Example data
n_bins = 25
hot_tokens2 = [D[i] for i in ids_sorted2[n_bins-1::-1]]
y_pos2 = np.arange(len(hot_tokens2))
z2 = tf_sorted2[n_bins-1::-1]/n_art

plt.barh(y_pos2, z2, align='center', alpha=0.4)
plt.yticks(y_pos2, hot_tokens2)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
From LSI, we can check both the topic-tokens matrix and the document-topics matrix.
Now we can check the topics generated by LSI. An intuitive visualization is provided by the show_topics method.
Can you write Python code for it?

lsi.show_topics(num_topics=-1, num_words=10, log=False, formatted=True)


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
From now on, tfidf can be used to convert any vector from the old representation (bow integer counts) to the new one (TfIdf real-valued weights):
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
However, a more useful representation of topics is as a list of tuples (token, value). This is provided by the show_topic method.
Task: Represent the columns of the topic-token matrix as a series of bar diagrams (one per topic) with the top 25 tokens of each topic.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

n_bins = 25

# Example data
y_pos = range(n_bins-1, -1, -1)

pylab.rcParams['figure.figsize'] = 16, 8  # Set figure size
for i in range(n_topics):

    ### Plot top 25 tokens for topic i
    # Read i-thtopic
    # scode: <FILL IN>
    tokens = [t[0] for t in topic_i]
    weights = [t[1] for t in topic_i]

    # Plot
    # scode: <FILL IN>
    plt.barh(y_pos, weights, align='center', alpha=0.4)
    plt.yticks(y_pos, tokens)
    plt.xlabel('Top {0} topic weights'.format(n_bins))
    plt.title('Topic {0}'.format(i))

plt.show()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
In the following we plot the most frequent terms in the corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

# Example data
n_bins = 25
hot_tokens = [D[i] for i in ids_sorted[n_bins-1::-1]]
y_pos = np.arange(len(hot_tokens))
z = tf_sorted[n_bins-1::-1]/n_art

plt.barh(y_pos, z, align='center', alpha=0.4)
plt.yticks(y_pos, hot_tokens)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()

# SORTED TOKEN FREQUENCIES:

# Example data
plt.semilogy(tf_sorted)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
LSI approximates any document as a linear combination of the topic vectors. We can compute the topic weights for any input corpus entered as input to the lsi model.
Can you write Python code for it?

# On real corpora, target dimensionality of
# 200–500 is recommended as a “golden standard”
# Create a double wrapper over the original
# corpus bow  tfidf  fold-in-lsi
corpus_lsi = lsi[corpus_tfidf]
print corpus_lsi[0]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
Note that we can interpret each element of corpus_bow as a sparse_vector. For example, a list of tuples
[(0, 1), (3, 3), (5,2)]

for a dictionary of 10 elements can be represented as a vector, where any tuple (id, n) states that position id must take value n. The rest of positions must be zero.
[1, 0, 0, 3, 0, 2, 0, 0, 0, 0]

These sparse vectors will be the inputs to the topic modeling algorithms.
Note that, at this point, we have built a Dictionary containing
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
Now we are ready to compute the token counts.
Can you write Python code for it?

# Use tf (raw term count) features for LDA.
print("Extracting tf features for LDA...")
n_features = 1000
n_samples = 2000
tf_vectorizer = CountVectorizer(max_df=0.95, min_df=2,
                                max_features=n_features,
                                stop_words='english')

t0 = time()
tf = tf_vectorizer.fit_transform(data_samples)
print("done in %0.3fs." % (time() - t0))
print tf[0][0][0]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
Now, we have stored the whole text collection in two lists:

corpus_titles, which contains the titles of the selected articles
corpus_text, with the text content of the selected wikipedia articles

You can browse the content of the wikipedia articles to get some intuition about the kind of documents that will be processed.
Can you write Python code for it?

# n = 5
# print corpus_titles[n]
# print corpus_text[n]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
Now, we need a dataset to feed the Count_Vectorizer object, by joining all tokens in corpus_clean in a single string, using a space ' ' as separator.
Can you write Python code for it?

print("Loading dataset...")
# scode: data_samples = <FILL IN>
data_samples = [" ".join(c) for c in corpus_clean]
print 'Document 0:'
print data_samples[0][0:200], '...'


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
One of the advantages of the lemmatizer method is that the result of lemmmatization is still a true word, which is more advisable for the presentation of text processing results and lemmatization.
However, without using contextual information, lemmatize() does not remove grammatical differences. This is the reason why "is" or "are" are preserved and not replaced by infinitive "be".
As an alternative, we can apply .lemmatize(word, pos), where 'pos' is a string code specifying the part-of-speech (pos), i.e. the grammatical role of the words in its sentence. For instance, you can check the difference between wnl.lemmatize('is') and wnl.lemmatize('is, pos='v').
2.3. Vectorization
Up to this point, we have transformed the raw text collection of articles in a list of articles, where each article is a collection of the word roots that are most relevant for semantic analysis. Now, we need to convert these data (a list of token lists) into a numerical representation (a list of vectors, or a matrix). To do so, we will start using the tools provided by the gensim library.
As a first step, we create a dictionary containing all tokens in our text corpus, and assigning an integer identifier to each one of them.
Can you write Python code for it?

# Create dictionary of tokens
D = gensim.corpora.Dictionary(corpus_clean)
n_tokens = len(D)

print "The dictionary contains {0} tokens".format(n_tokens)
print "First tokens in the dictionary: "
for n in range(10):
    print str(n) + ": " + D[n]


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
Task: Find the document with the largest positive weight for topic 0. Compare the document and the topic.
Can you write Python code for it?

# Extract weights from corpus_lsi
# scode weight0 = <FILL IN>

# Locate the maximum positive weight
nmax = np.argmax(weight0)
print nmax
print weight0[nmax]
print corpus_lsi[nmax]

# Get topic 0
# scode: topic_0 = <FILL IN>

# Compute a list of tuples (token, wordcount) for all tokens in topic_0, where wordcount is the number of
# occurences of the token in the article.
# scode: token_counts = <FILL IN>

print "Topic 0 is:"
print topic_0
print "Token counts:"
print token_counts


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
and a bow representation of a corpus with
Can you write Python code for it?

print "{0} Wikipedia articles".format(len(corpus_bow))


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
ids_sorted is a list of all token ids, sorted by decreasing number of occurrences in the whole corpus. For instance, the most frequent term is
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py2_wikitools/notebooks/TM2_Topic_Modeling_student.ipynb".
The first task is:
which appears
Can you write Python code for it?

print "{0} times in the whole corpus".format(tf_sorted[0])


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
1. Corpus acquisition
In this block we will work with collections of text documents. The objectives will be:

Find the most important topics in the collection and assign documents to topics
Analyze the structure of the collection by means of graph analysis

We will work with a collection of research projects funded by the US National Science Foundation, that you can find under the ./data directory. These files are publicly available from the NSF website.
1.1. Exploring file structure
NSF project information is provided in XML files. Projects are yearly grouped in .zip files, and each project is saved in a different XML file. To explore the structure of such files, we will use the file 160057.xml. Parsing XML files in python is rather easy using the ElementTree module.
1.1.1. File format
To start with, you can have a look at the contents of the example file. We are interested on the following information of each project:

Project identifier
Project Title
Project Abstract
Budget
Starting Year (we will ignore project duration)
Institution (name, zipcode, and state)
Can you write Python code for it?

xmlfile = '../data/1600057.xml'

with open(xmlfile,'r') as fin:
    print(fin.read())


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
1.1.2. Parsing XML
XML is an inherently hierarchical data format, and the most natural way to represent it is with a tree. The ElementTree module has two classes for this purpose:

ElementTree represents the whole XML document as a tree
Element represents a single node in this tree

We can import XML data by reading an XML file:
Can you write Python code for it?

import xml.etree.ElementTree as ET
root = ET.fromstring(open(xmlfile,'r').read())


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
1.2. Building the dataset
Now, we will use the function you just implemented, to create a database that we will use throughout this module.
For simplicity, and given that the dataset is not too large, we will keep all projects in the RAM. The dataset will consist of a list containing the dictionaries associated to each of the considered projects in a time interval.
Can you write Python code for it?

# Construct an iterator (or a list) for the years you want to work with
years = [2015, 2016]
datafiles_path = '../data/'
NSF_data = []

for year in years:

    zpobj = zp.ZipFile(join(datafiles_path, str(year)+'.zip'))
    for fileinzip in zpobj.namelist():
        if fileinzip.endswith('xml'):

            #Some files seem to be incorrectly parsed
            try:
                project_dictio = parse_xmlproject(zpobj.read(fileinzip))
                if project_dictio['abstract']:
                    NSF_data.append(project_dictio)
            except:
                pass


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
2. Corpus Processing
Topic modelling algorithms process vectorized data. In order to apply them, we need to transform the raw text input data into a vector representation. To do so, we will remove irrelevant information from the text data and preserve as much relevant information as possible to capture the semantic content in the document collection.
Thus, we will proceed with the following steps:

Tokenization
Homogeneization, which includes:
Removing capitalization.
Removing non alphanumeric tokens (e.g. punktuation signs)
Stemming/Lemmatization.

Cleaning
Vectorization

For the first steps, we will use some of the powerful methods available from the Natural Language Toolkit. In order to use the word_tokenize method from nltk, you might need to get the appropriate libraries using nltk.download(). You must select option "d) Download", and identifier "punkt"
Can you write Python code for it?

from nltk import download
# You should comment this code fragment if the package is already available.
# download('punkt')
# download('stopwords')


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
2.1. Corpus Processing
We will create a list that contains just the abstracts in the dataset. As the order of the elements in a list is fixed, it will be later straightforward to match the processed abstracts to metadata associated to their corresponding projects.
Exercise 1: Generate a corpus of processed documents. For each document in corpus_raw complete the following steps:
1. Tokenize.
2. Remove capitalization and non-alphanumeric tokens.
3. Lemmatize
4. Remove the stopwords using the NLTK stopwords list.
Can you write Python code for it?

from nltk.tokenize import word_tokenize
from nltk.stem import SnowballStemmer, WordNetLemmatizer
from nltk.corpus import stopwords

# Maybe you can try the stemmer too.
# stemmer = SnowballStemmer('english')
wnl = WordNetLemmatizer()

stopwords_en = stopwords.words('english')

# Initialize ouput corpus
corpus_clean = []

ndocs = len(corpus_raw)
for n, text in enumerate(corpus_raw):
    if not n%100:
        print('\rTokenizing document', n, 'out of', ndocs, end='', flush=True)

    # Tokenize each text entry.
    # tokens = <FILL IN>
    # tokens_filtered = <FILL IN>
    # tokens_lemmatized = <FILL IN>
    # tokens_clean = <FILL IN>

    # Add the new token list as a new element to corpus_clean (that will be a list of lists)
    # corpus_clean.<FILL IN>

print('\n\n The corpus has been tokenized. Check the result for the first abstract:')
print(corpus_raw[0])
print(corpus_clean[0])


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
2.4. Vectorization
Up to this point, we have transformed the raw text collection in a list of documents, where each documen is a collection of the words that are most relevant for semantic analysis. Now, we need to convert these data (a list of token lists) into a numerical representation (a list of vectors, or a matrix). To do so, we will start using the tools provided by the gensim library.
As a first step, we create a dictionary containing all tokens in our text corpus, and assigning an integer identifier to each one of them.
Can you write Python code for it?

# Create dictionary of tokens
D = gensim.corpora.Dictionary(corpus_clean)
n_tokens = len(D)

print('The dictionary contains', n_tokens, 'terms')
print('First terms in the dictionary:')
for n in range(10):
    print(str(n), ':', D[n])


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
3. Topic Modeling
There are several implementations of the LDA topic model in python:

Python library lda.
Gensim module: gensim.models.ldamodel.LdaModel
Sci-kit Learn module: sklearn.decomposition

In the following sections we explore the use of gensim
3.1. Training a topic model using Gensim LDA
Since we already have computed the dictionary and documents BoW representation using Gensim, computing the topic model is straightforward using the LdaModel() function. Please, refer to Gensim API documentation for more information on the different parameters accepted by the function:
Exercise 3: Create an LDA model with the 50 topics using corpus_bow and the dictionary, D.
Can you write Python code for it?

num_topics = 50

# This might take some time...
# ldag = gensim.models.ldamodel.LdaModel(<FILL IN>)


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
3.2. LDA model visualization
Gensim provides a basic visualization of the obtained topics:
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
3.3. Gensim utility functions
In addition to visualization purposes, topic models are useful to obtain a semantic representation of documents that can later be used with some other purpose:

In document classification problems
In content-based recommendations systems

Essentially, the idea is that the topic model provides a (semantic) vector representation of documents, and use probability divergences to measure document similarity. The following functions of the LdaModel class will be useful in this context:

get_topic_terms(topic_id): Gets vector of the probability distribution among words for the indicated topic
get_document_topics(bow_vector): Gets (sparse) vector with the probability distribution among topics for the provided document

Exercise 4: Show the probability distribution over words for topic 0.
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
A more useful visualization is provided by the python LDA visualization library, pyLDAvis.
Before executing the next code fragment you might need to install pyLDAvis:
>> pip install (--user) pyLDAvis
Can you write Python code for it?

import pyLDAvis.gensim as gensimvis
import pyLDAvis

vis_data = gensimvis.prepare(ldag, corpus_bow, D)
pyLDAvis.display(vis_data)


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
An alternative to the use of the get_document_topics() function is to directly transform a dataset using the ldag object as follows. You can apply this transformation to several documents at once, but then the result is an iterator from which you can build the corresponding list if necessary
Can you write Python code for it?

print(ldag[corpus_bow[0]])

print('When applied to a dataset it will provide an iterator')
print(ldag[corpus_bow[:3]])

print('We can rebuild the list from the iterator with a one liner')
print([el for el in ldag[corpus_bow[:3]]])


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
At this point, it is good to make sure to understand what has happened. In clean_abstracts we had a list of token lists. With it, we have constructed a Dictionary, D, which assigns an integer identifier to each token in the corpus.
After that, we have transformed each article (in clean_abstracts) in a list tuples (id, n).
Can you write Python code for it?

print('Original document (after cleaning):')
print(corpus_clean[0])
print('Sparse vector representation (first 10 components):')
print(corpus_bow[0][:10])
print('Word counts for the first project (first 10 components):')
print(list(map(lambda x: (D[x[0]], x[1]), corpus_bow[0][:10])))


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
Exercise 5: Show the probability distribution over topics for document 0.
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
Exercise 6: Build a function that returns the most relevant projects for a given topic
Can you write Python code for it?

def most_relevant_projects(ldag, topicid, corpus_bow, nprojects=10):
    """This function returns the most relevant projects in corpus_bow

    : ldag: The trained topic model object provided by gensim
    : topicid: The topic for which we want to find the most relevant documents
    : corpus_bow: The BoW representation of documents in Gensim format
    : nprojects: Number of most relevant projects to identify

    : Returns: A list with the identifiers of the most relevant projects
    """

    print('Computing most relevant projects for Topic', topicid)
    print('Topic composition is:')
    print(ldag.show_topic(topicid))

    #<SOL>
    #</SOL>

#To test the function we will find the most relevant projects for a subset of the NSF dataset
project_id = most_relevant_projects(ldag, 17, corpus_bow[:10000])

#Print titles of selected projects
for idproject in project_id:
    print(NSF_data[idproject]['title'])


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
Exercise 7: Build a function that computes the semantic distance between two documents. For this, you can use the functions (or code fragments) provided in the library dist_utils.py.
Can you write Python code for it?

def pairwase_dist(doc1, doc2):
    """This function returns the Jensen-Shannon
    distance between the corresponding vectors of the documents

    : doc1: Semantic representation for the doc1 (a vector of length ntopics)
    : doc2: Semantic representation for the doc2 (a vector of length ntopics)

    : Returns: The JS distance between doc1 and doc2 (a number)
    """
    #<SOL>
    #</SOL>


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
Exercise 8: Explore the influence of the concentration parameters, $alpha$ and $eta$. In particular observe how do topic and document distributions change as these parameters increase.
 Exercise 9: Note that we have not used the terms in the article titles, though the can be expected to contain relevant words for the topic modeling. Include the title words in the analyisis. In order to give them a special relevante, insert them in the corpus several times, so as to make their words more significant.
4. Saving data.
The following function creates the Node CSV file that could be usefult for further processing. In particular, the output files could be used to visualize the document corpus as a graph using a visualization software like Gephi.
Can you write Python code for it?

#print(NSF_data[0].keys())
#print(NSF_data[0]['institution'])

def strNone(str_to_convert):
    if str_to_convert is None:
        return ''
    else:
        return str_to_convert

with open('NSF_nodes.csv','w') as fout:
    fout.write('Id;Title;Year;Budget;UnivName;UnivZIP;State\n')
    for project in NSF_data:
        fout.write(project['project_code']+';'+project['title']+';')
        fout.write(project['year']+';'+str(project['budget'])+';')
        fout.write(project['institution'][0]+';')
        fout.write(strNone(project['institution'][1])+';')
        fout.write(strNone(project['institution'][2])+'\n')


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
Finally, Gensim provides some useful functions to convert between formats, and to simplify interaction with numpy and scipy. The following code fragment converts a corpus in sparse format to a full numpy matrix
Can you write Python code for it?

reduced_corpus = [el for el in ldag[corpus_bow[:3]]]
reduced_corpus = gensim.matutils.corpus2dense(reduced_corpus, num_topics).T
print(reduced_corpus)


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
In the following we plot the most frequent terms in the corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
plt.rcdefaults()

# Example data
n_art = len(NSF_data)
n_bins = 25
hot_tokens = [D[i] for i in ids_sorted[n_bins-1::-1]]
y_pos = np.arange(len(hot_tokens))
z = tf_sorted[n_bins-1::-1]/n_art

plt.figure()
plt.barh(y_pos, z, align='center', alpha=0.4)
plt.yticks(y_pos, hot_tokens)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()

# SORTED TOKEN FREQUENCIES:

# Example data
plt.figure()
plt.semilogy(tf_sorted)
plt.ylabel('Total number of occurrences')
plt.xlabel('Token rank')
plt.title('Token occurrences')
plt.show()

# cold_tokens = <FILL IN>

print("There are {0} cold tokens, which represent {1}% of the total number of tokens in the dictionary".format(
    len(cold_tokens), float(len(cold_tokens))/n_tokens*100))


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
In the second step, let us create a numerical version of our corpus using the doc2bow method. In general, D.doc2bow(token_list) transforms any list of tokens into a list of tuples (token_id, n), one per each token in token_list, where token_id is the token identifier (according to dictionary D) and n is the number of occurrences of such token in token_list.
Exercise 2: Apply the doc2bow method from gensim dictionary D, to all tokens in every document in clean_abstracts. The result must be a new list named corpus_bow where each element is a list of tuples (token_id, number_of_occurrences).
Can you write Python code for it?


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
Note that we can interpret each element of corpus_bow as a sparse_vector. For example, a list of tuples
[(0, 1), (3, 3), (5,2)]

for a dictionary of 10 elements can be represented as a vector, where any tuple (id, n) states that position id must take value n. The rest of positions must be zero.
[1, 0, 0, 3, 0, 2, 0, 0, 0, 0]

These sparse vectors will be the inputs to the topic modeling algorithms.
As a summary, the following variables will be relevant for the next chapters:
   * D: A gensim dictionary. Term strings can be accessed using the numeric identifiers. For instance, D[0] contains the string corresponding to the first position in the BoW representation.
   * corpus_bow: BoW corpus. A list containing an entry per project in the dataset, and consisting of the (sparse) BoW representation for the abstract of that project.
   * NSF_data: A list containing an entry per project in the dataset, and consisting of metadata for the projects in the dataset
The way we have constructed the corpus_bow variable guarantees that the order is preserved, so that the projects are listed in the same order in the lists corpus_bow and NSF_data.
Before starting with the semantic analyisis, it is interesting to observe the token distribution for the given corpus.
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (I):
# Create a "flat" corpus with all tuples in a single list
corpus_bow_flat = [item for sublist in corpus_bow for item in sublist]

# Initialize a numpy array that we will use to count tokens.
# token_count[n] should store the number of ocurrences of the n-th token, D[n]
token_count = np.zeros(n_tokens)

# Count the number of occurrences of each token.
for x in corpus_bow_flat:
    # Update the proper element in token_count
    # scode: <FILL IN>

# Sort by decreasing number of occurences
ids_sorted = np.argsort(- token_count)
tf_sorted = token_count[ids_sorted]


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
The code below implements a function that parses the XML files and provides as its output a dictionary with fields:
project_code      (string)
title             (string)
abstract          (string)
budget            (float)
year              (string)
institution       (tuple with elements: name, zipcode, and statecode)
Can you write Python code for it?

def parse_xmlproject(xml_string):
    """This function processess the specified XML field,
    and outputs a dictionary with the desired project information

    :xml_string: String with XML content
    :Returns: Dictionary with indicated files
    """

    root = ET.fromstring(xml_string)
    dictio = {}

    for child in root[0]:
        if child.tag.lower() == 'awardtitle':
            dictio['title'] = child.text
        elif child.tag.lower() == 'awardeffectivedate':
            dictio['year'] = str(child.text[-4:])
        elif child.tag.lower() == 'awardamount':
            dictio['budget'] = float(child.text)
        elif child.tag.lower() == 'abstractnarration':
            dictio['abstract'] = child.text
        elif child.tag.lower() == 'awardid':
            dictio['project_code'] = child.text
        elif child.tag.lower() == 'institution':
            #For the institution we have to access the children elements
            #and search for the name, zipcode, and statecode only
            name = ''
            zipcode = ''
            statecode = ''
            for child2 in child:
                if child2.tag.lower() == 'name':
                    name = child2.text
                elif child2.tag.lower() == 'zipcode':
                    zipcode = child2.text
                elif child2.tag.lower() == 'statecode':
                    statecode = child2.text
            dictio['institution'] = (name, zipcode, statecode)

    return dictio

parse_xmlproject(open(xmlfile,'r').read())


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
We will extract some characteristics of the constructed dataset:
Can you write Python code for it?

print('Number of projects in dataset:', len(NSF_data))

# Budget
budget_data = list(map(lambda x: x['budget'], NSF_data))
print('Average budget of projects in dataset:', np.mean(budget_data))

# Institutions
insti_data = list(map(lambda x: x['institution'], NSF_data))
print('Number of unique institutions in dataset:', len(set(insti_data)))

# Counts per year
counts = dict()
for project in NSF_data:
    counts[project['year']] = counts.get(project['year'],0) + 1

print('Breakdown of projects by starting year:')
for el in counts:
    print(el, ':', counts[el])


I am working on the file "TM2.Topic_Models/TM_py3_NSF/notebook/TM2_TopicModels_student.ipynb".
The first task is:
which appears
Can you write Python code for it?

print("{0} times in the whole corpus".format(tf_sorted[0]))


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/.ipynb_checkpoints/TM_Exam2-checkpoint.ipynb".
The first task is:
A continuación cargaremos los datos en un RDD
Can you write Python code for it?

corpusRDD = sc.parallelize(corpus_text, 4)
print "\nRDD created with {0} elements".format(corpusRDD.count())


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/.ipynb_checkpoints/TM_Exam2-checkpoint.ipynb".
The first task is:
Para evitar problemas de sobrecarga de memoria, o de tiempo de procesado, puede reducir el tamaño el corpus, modificando el valor de la variable n_docs a continuación.
Can you write Python code for it?

n_docs = 500000

filenames = mycorpus.fileids()
fn_train = [f for f in filenames if f[0:5]=='train']

corpus_text = [mycorpus.raw(f) for f in fn_train]

# Reduced dataset:
n_docs = min(n_docs, len(corpus_text))
corpus_text = [corpus_text[n] for n in range(n_docs)]

print 'Loaded {0} files'.format(len(corpus_text))


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/.ipynb_checkpoints/TM_Exam_Solution-checkpoint.ipynb".
The first task is:
A continuación cargaremos los datos en un RDD
Can you write Python code for it?


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/.ipynb_checkpoints/TM_Exam_Solution-checkpoint.ipynb".
The first task is:
Aplique el algoritmo LDA con 4 tópicos sobre el corpus obtenido en corpus4lda, para un valor de topicConcentration = 2.0 y docConcentration = 3.0. (Tenga en cuenta que estos parámetros de entrada deben de ser tipo float).
Can you write Python code for it?

print "Training LDA: this might take a while..."
start = time.time()
n_topics = 4
ldaModel = LDA.train(corpus4lda, k=n_topics, topicConcentration=2.0, docConcentration=3.0)
print "Modelo LDA entrenado en: {0} segundos".format(time.time()-start)


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/.ipynb_checkpoints/TM_Exam_Solution-checkpoint.ipynb".
The first task is:
Ejercicio 10: Algoritmo LDA.
Para aplicar el algoritmo LDA, es necesario reemplazar las tuplas (token, valor) de wcRDD por tuplas del tipo (token_id, value), sustituyendo cada token por un identificador entero.
El código siguiente se encarga de completar este proceso:
Can you write Python code for it?

# Compute RDD replacing tokens by token_ids
corpus_sparseRDD = corpus_wcRDD2.map(lambda x: [(invD[t[0]], t[1]) for t in x])

# Convert list of tuplas into Vectors.sparse object.
corpus_sparseRDD = corpus_sparseRDD.map(lambda x: Vectors.sparse(n_tokens, x))
corpus4lda = corpus_sparseRDD.zipWithIndex().map(lambda x: [x[1], x[0]]).cache()


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/.ipynb_checkpoints/TM_Exam_Solution-checkpoint.ipynb".
The first task is:
Ejercicio 12: Pesos de un token.
Imprima el peso del token bank en cada tópico.
Can you write Python code for it?

# Output topics. Each is a distribution over words (matching word count vectors)
iBank = invD['bank']
topicMatrix = ldaModel.topicsMatrix()
print topicMatrix[iBank]


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/.ipynb_checkpoints/TM_Exam_Solution-checkpoint.ipynb".
The first task is:
Ejercicio 3: Vectorización
En este punto cada documento del corpus es una lista de tokens.
Calcule un nuevo RDD que contenga, para cada documento, una lista de tuplas. La clave (key) de cada lista será un token y su valor el número de repeticiones del mismo en el documento.
Imprima una muestra de 20 tuplas uno de los documentos del corpus.
Can you write Python code for it?

# corpus_wcRDD = <FILL IN>
corpus_wcRDD = (corpus_stemRDD
                .map(collections.Counter)
                .map(lambda x: [(t, x[t]) for t in x]))

print corpus_wcRDD.take(1)[0][0:20]


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/.ipynb_checkpoints/TM_Exam_Solution-checkpoint.ipynb".
The first task is:
Ejercicio 5: Número de tokens.
Determine el número total de tokens en el diccionario. Imprima el resultado.
Can you write Python code for it?


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/.ipynb_checkpoints/TM_Exam_Solution-checkpoint.ipynb".
The first task is:
Ejercicio 6: Términos demasiado frecuentes:
Determine los 5 tokens más frecuentes del corpus. Imprima el resultado.
Can you write Python code for it?

print wcRDD.takeOrdered(5, lambda x: -x[1])


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/.ipynb_checkpoints/TM_Exam_Solution-checkpoint.ipynb".
The first task is:
Ejercicio 7: Número de documentos del token más frecuente.
Determine en qué porcentaje de documentos aparece el token más frecuente.
Can you write Python code for it?

tokenmasf = 'said'
ndocs = corpus_stemRDD.filter(lambda x: tokenmasf in x).count()
print ndocs


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/.ipynb_checkpoints/TM_Exam_Solution-checkpoint.ipynb".
The first task is:
Ejercicio 8: Filtrado de términos.
Elimine del corpus los dós términos más frecuentes. Guarde el resultado en un nuevo RDD denominado corpus_wcRDD2, con la misma estructura que corpus_wcRDD (es decir, cada documento una lista de tuplas).
Can you write Python code for it?

corpus_wcRDD2 = corpus_wcRDD.map(lambda x: [tupla for tupla in x if tupla[0]
                                 not in ['said', 'mln']])

# scode: wcRDD = < FILL IN >
wcRDD2 = (corpus_wcRDD2
         .flatMap(lambda x: x)
         .reduceByKey(lambda x, y: x + y)
         .sortBy(lambda x: -x[1]))

print wcRDD2.takeOrdered(10, lambda x: -x[1])


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/.ipynb_checkpoints/TM_Exam_Solution-checkpoint.ipynb".
The first task is:
Para evitar problemas de sobrecarga de memoria, o de tiempo de procesado, puede reducir el tamaño el corpus, modificando el valor de la variable n_docs a continuación.
Can you write Python code for it?


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam.ipynb".
The first task is:
1. Ejercicios
Ejercicio 1: (0.6 ptos) Preprocesamiento de datos.
Prepare los datos para aplicar un algoritmo de modelado de tópicos en pyspark. Para ello, aplique los pasos siguientes:

Tokenización: convierta cada texto a utf-8, y transforme la cadena en una lista de tokens.
Homogeneización: pase todas las palabras a minúsculas y elimine todos los tokens no alfanuméricos.
Limpieza: Elimine todas las stopwords utilizando el fichero de stopwords disponible en NLTK para el idioma inglés.

Guarde el resultado en la variable corpus_tokensRDD
Ejercicio 2:  (0.6 ptos) Stemming
Aplique un procedimiento de stemming al corpus, utilizando el SnowballStemmer de NLTK. Guarde el resultado en corpus_stemRDD.
Ejercicio 3:  (0.6 ptos) Vectorización
En este punto cada documento del corpus es una lista de tokens.
Calcule un nuevo RDD que contenga, para cada documento, una lista de tuplas. La clave (key) de cada lista será un token y su valor el número de repeticiones del mismo en el documento.
Imprima una muestra de 20 tuplas uno de los documentos del corpus.
Ejercicio 4:  (0.8 ptos) Cálculo del diccionario de tokens
Construya, a partir de corpus_wcRDD, un nuevo diccionario con todos los tokens del corpus. El resultado será un diccionario python de nombre wcDict, cuyas entradas serán los tokens y sus valores el número de repeticiones del token en todo el corpus.
wcDict = {token1: valor1, token2, valor2, ...}
Imprima el número de repeticiones del token interpret
Ejercicio 5:  (0.6 ptos) Número de tokens.
Determine el número total de tokens en el diccionario. Imprima el resultado.
Ejercicio 6:  (0.8 ptos) Términos demasiado frecuentes:
Determine los 5 tokens más frecuentes del corpus. Imprima el resultado.
Ejercicio 7:  (0.8 ptos) Número de documentos del token más frecuente.
Determine en qué porcentaje de documentos aparece el token más frecuente.
Ejercicio 8:  (1 ptos) Filtrado de términos.
Elimine del corpus los dós términos más frecuentes. Guarde el resultado en un nuevo RDD denominado corpus_wcRDD2, con la misma...
Can you write Python code for it?


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
0. Adquisición de un corpus.
Descargue el contenido del corpus reuters de nltk.
import nltk
nltk.download()

Selecciona el identificador reuters.
Can you write Python code for it?

#nltk.download()
mycorpus = nltk.corpus.reuters


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
1. Ejercicios
Ejercicio 1: Preprocesamiento de datos.
Prepare los datos para aplicar un algoritmo de modelado de tópicos en pyspark. Para ello, aplique los pasos siguientes:

Tokenización: convierta cada texto a utf-8, y transforme la cadena en una lista de tokens.
Homogeneización: pase todas las palabras a minúsculas y elimine todos los tokens no alfanuméricos.
Limpieza: Elimine todas las stopwords utilizando el fichero de stopwords disponible en NLTK para el idioma inglés.

Guarde el resultado en la variable corpus_tokensRDD
Can you write Python code for it?

def getTokenList(doc, stopwords_en):

    # scode: tokens = <FILL IN>   # Tokenize docs
    tokens = word_tokenize(doc.decode('utf-8'))

    # scode: tokens = <FILL IN>   # Remove non-alphanumeric tokens and normalize to lowercase
    tokens = [t.lower() for t in tokens if t.isalnum()]
    # scode: tokens = <FILL IN>   # Remove stopwords
    tokens = [t for t in tokens if t not in stopwords_en]

    return tokens

stopwords_en = stopwords.words('english')
corpus_tokensRDD = (corpusRDD
                   .map(lambda x: getTokenList(x, stopwords_en))
                   .cache())

# print "\n Let's check tokens after cleaning:"
print corpus_tokensRDD.take(1)[0][0:30]


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
A continuación cargaremos los datos en un RDD
Can you write Python code for it?


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
Ejercicio 10: Algoritmo LDA.
Para aplicar el algoritmo LDA, es necesario reemplazar las tuplas (token, valor) de wcRDD por tuplas del tipo (token_id, value), sustituyendo cada token por un identificador entero.
El código siguiente se encarga de completar este proceso:
Can you write Python code for it?

# Compute RDD replacing tokens by token_ids
corpus_sparseRDD = corpus_wcRDD2.map(lambda x: [(invD[t[0]], t[1]) for t in x])

# Convert list of tuplas into Vectors.sparse object.
corpus_sparseRDD = corpus_sparseRDD.map(lambda x: Vectors.sparse(n_tokens, x))
corpus4lda = corpus_sparseRDD.zipWithIndex().map(lambda x: [x[1], x[0]]).cache()
print corpus4lda.take(1)