I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
Ejercicio 11: Tokens principales.
Imprima los dos tokens de mayor peso de cada tópico. (Debe imprimir el texto del token, no su índice).
Can you write Python code for it?

n_topics = 4
ldatopics = ldaModel.describeTopics(maxTermsPerTopic=2)
ldatopicnames = map(lambda x: x[0], ldatopics)
print ldatopicnames
for i in range(n_topics):
    print "Topic {0}: {1}, {2}".format(i, D[ldatopicnames[i][0]], D[ldatopicnames[i][1]])

print ldatopics


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
Ejercicio 12: Pesos de un token.
Imprima el peso del token bank en cada tópico.
Can you write Python code for it?

# Output topics. Each is a distribution over words (matching word count vectors)
iBank = invD['bank']
topicMatrix = ldaModel.topicsMatrix()
print topicMatrix[iBank]


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
Ejercicio 2: Stemming
Aplique un procedimiento de stemming al corpus, utilizando el SnowballStemmer de NLTK. Guarde el resultado en corpus_stemRDD.
Can you write Python code for it?

# Select stemmer.
stemmer = nltk.stem.SnowballStemmer('english')

# scode: corpus_stemRDD = <FILL IN>
corpus_stemRDD = corpus_tokensRDD.map(lambda x: [stemmer.stem(token) for token in x])

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_stemRDD.take(1)[0][0:30]


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
Ejercicio 3: Vectorización
En este punto cada documento del corpus es una lista de tokens.
Calcule un nuevo RDD que contenga, para cada documento, una lista de tuplas. La clave (key) de cada lista será un token y su valor el número de repeticiones del mismo en el documento.
Imprima una muestra de 20 tuplas uno de los documentos del corpus.
Can you write Python code for it?

# corpus_wcRDD = <FILL IN>
corpus_wcRDD = (corpus_stemRDD
                .map(collections.Counter)
                .map(lambda x: [(t, x[t]) for t in x]))

print corpus_wcRDD.take(1)[0][0:20]


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
Ejercicio 6: Términos demasiado frecuentes:
Determine los 5 tokens más frecuentes del corpus. Imprima el resultado.
Can you write Python code for it?

print wcRDD.takeOrdered(5, lambda x: -x[1])


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
Ejercicio 7: Número de documentos del token más frecuente.
Determine en qué porcentaje de documentos aparece el token más frecuente.
Can you write Python code for it?

tokenmasf = 'said'
ndocs = corpus_stemRDD.filter(lambda x: tokenmasf in x).count()

print 'El numerod de documentos es {0}, es decir, el {1} % del total de documentos'.format(
    ndocs, float(ndocs)/corpus_stemRDD.count()*100)


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
Ejercicio 8: Filtrado de términos.
Elimine del corpus los dós términos más frecuentes. Guarde el resultado en un nuevo RDD denominado corpus_wcRDD2, con la misma estructura que corpus_wcRDD (es decir, cada documento una lista de tuplas).
Can you write Python code for it?

corpus_wcRDD2 = corpus_wcRDD.map(lambda x: [tupla for tupla in x if tupla[0]
                                 not in ['said', 'mln']])

print corpus_wcRDD2.take(1)


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
Ejercicio 9: Lista de tokens y diccionario inverso.
Determine la lista de topicos de todo el corpus, y construya un dictionario inverso, cuyas entradas sean los números consecutivos de 0 al número total de tokens, y sus salidas cada uno de los tokens, es decir
invD = {0: token0, 1: token1, 2: token2, ...}
Can you write Python code for it?

# scode: wcRDD = < FILL IN >
wcRDD2 = (corpus_wcRDD2
         .flatMap(lambda x: x)
         .reduceByKey(lambda x, y: x + y)
         .sortBy(lambda x: -x[1]))

# Token Dictionary:
n_tokens = wcRDD2.count()
TD = wcRDD2.takeOrdered(n_tokens, lambda x: -x[1])

D = map(lambda x: x[0], TD)
token_count = map(lambda x: x[1], TD)

# Compute inverse dictionary
invD = dict(zip(D, xrange(n_tokens)))

print invD


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
Para evitar problemas de sobrecarga de memoria, o de tiempo de procesado, puede reducir el tamaño el corpus, modificando el valor de la variable n_docs a continuación.
Can you write Python code for it?

n_docs = 500000

filenames = mycorpus.fileids()
fn_train = [f for f in filenames if f[0:5]=='train']

corpus_text = [mycorpus.raw(f) for f in fn_train]

# Reduced dataset:
n_docs = min(n_docs, len(corpus_text))
corpus_text = [corpus_text[n] for n in range(n_docs)]

print 'Loaded {0} files'.format(len(corpus_text))


I am working on the file "TM3.Topic_Models_with_MLlib/ExB3_TopicModels/TM_Exam_Solution.ipynb".
The first task is:
Test 13: Indique cuáles de las siguientes afirmaciones se puede asegurar que son verdaderas:

En LSI, cada documento se asigna a un sólo tópico.
De acuerdo con el modelo LDA, todos los tokens de un documento han sido generados por el mismo tópico
LSI descompone la matriz de datos de entrada en el producto de 3 matrices cuadradas.
Si el rango de la matriz de entrada a un modelo LSI es igual al número de tópicos. La descomposición SVD del modelo LSI es exacta (no es una aproximación).

FFFV
Test 14: Indique cuáles de las siguientes afirmaciones se puede asegurar que son verdaderas:

En un modelo LDA, la distribución de Dirichlet se utiliza para generar distribuciones de probabilidad de tokens.
Si una palabra aparece en pocos documentos del corpus, su IDF es mayor.
El resultado de la lematización de una palabra es una palabra
El resultado del stemming de una palabra es una palabra
Can you write Python code for it?


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
1. Corpus acquisition.
In this notebook we will explore some tools for text processing and analysis and two topic modeling algorithms available from Python toolboxes.
To do so, we will explore and analyze collections of Wikipedia articles from a given category, using wikitools, that makes easy the capture of content from wikimedia sites.
(As a side note, there are many other available text collections to test topic modelling algorithm. In particular, the NLTK library has many examples, that can explore them using the nltk.download() tool.
import nltk
nltk.download()

for instance, you can take the gutemberg dataset
Mycorpus = nltk.corpus.gutenberg
text_name = Mycorpus.fileids()[0]
raw = Mycorpus.raw(text_name)
Words = Mycorpus.words(text_name)

Also, tools like Gensim or Sci-kit learn include text databases to work with).
In order to use Wikipedia data, we will select a single category of articles:
Can you write Python code for it?

site = wiki.Wiki("https://en.wikipedia.org/w/api.php")
# Select a category with a reasonable number of articles (>100)
cat = "Economics"
# cat = "Pseudoscience"
print cat


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
2.2. Stemming / Lemmatization
Now we will apply stemming and lemmatization to corpus_tokensRDD. We will test our topic models over the resulting RDDs, to test their differences.
Task: Apply stemming to all documents corpus_tokensRDD and save the result in a new RDD, corpus_stemmedRDD.
Can you write Python code for it?

# Select stemmer.
stemmer = nltk.stem.SnowballStemmer('english')

# scode: corpus_stemRDD = <FILL IN>
corpus_stemRDD = corpus_tokensRDD.map(lambda x: [stemmer.stem(token) for token in x])

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_stemRDD.take(1)[0][0:30]

Test.assertTrue((len([c for c in corpus_stemRDD.take(1)[0] if c!=stemmer.stem(c)])
                 < 0.1*len(corpus_stemRDD.take(1)[0])),
                'It seems that stemming has not been applied properly')


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
3. Latent Dirichlet Allocation
In order to apply the LDA algorithm, we need to represent the input documents in the format required by MLlib. More specifically. The input data should be an RDD where each element is a tuple
(doc_id, vector)

where doc_id is an integer document identifier, and vector can be a sparse or dense vector from class Vectors. We will use sparse vectors, which are more adequate for large vocabularies.
To compute the sparse vectors, we must first transform the lists of tuples (token, value) in wcRDD into a lists of (token_id, value), pairs, thus replacing each token by a numerical identifier.
We will proceed in two steps:

Compute an inverse dictionary, invD, transforming tokens into numbers.
Apply the inverse dictionary to compute a new RDD from wcRDD replacing each token by its token_id.

[ Task: complete the two steps outlined above.
Can you write Python code for it?

# INDICE INVERTIDO: EJEMPLO:
# D = ['token1', 'token2', 'token3', 'token4']
# D[1] = 'token2'
# invD = {'token1': 0, 'token2': 1, 'token3': 2, 'token4': 3}
# invD['token2'] = 1

# Compute inverse dictionary
# scode: invD = <FILL IN>
invD = dict(zip(D, xrange(n_tokens)))

### ALTERNATIVELY:
# invD_RDD = D_RDD.zipWithIndex()      ### Tuples (token, index)

# Compute RDD replacing tokens by token_ids
# scode: corpus_sparseRDD = <FILL IN>
corpus_sparseRDD = corpus_wcRDD.map(lambda x: [(invD[t[0]], t[1]) for t in x])

# Convert list of tuplas into Vectors.sparse object.
corpus_sparseRDD = corpus_sparseRDD.map(lambda x: Vectors.sparse(n_tokens, x))


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
Also, we need to load a list of english stopwords. Select now identifier "stopwords"
Can you write Python code for it?

# You can comment this if the package is already available.
# Select option "d) Download", and identifier "stopwords"
# nltk.download()


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
Alternatively, we can apply lemmatization. For english texts, we can use the lemmatizer from NLTK, which is based on WordNet. If you have not used wordnet before, you will likely need to download it from nltk
Can you write Python code for it?

# You can comment this if the package is already available.
# Select option "d) Download", and identifier "wordnet"
# nltk.download()


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
Alternatively, we can use the .describeTopics method that returns the most relevan terms for each topic, and it is more useful for a graphical plot.
Task: Represent the 25 most relevant terms for each topic using bar plots.
Can you write Python code for it?

n_bins = 25

# Example data
y_pos = range(n_bins-1, -1, -1)

pylab.rcParams['figure.figsize'] = 16, 8  # Set figure size
for i in range(3):

    topic = ldaModel.describeTopics(maxTermsPerTopic=n_bins)[i]
    tokens = [D[n] for n in topic[0]]
    weights = topic[1]

    plt.subplot(1, 3, i+1)
    plt.barh(y_pos, weights, align='center', alpha=0.4)
    plt.yticks(y_pos, tokens)
    plt.xlabel('Average number of occurrences per article')
    plt.title('Token distribution')


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
At this point, we have got a representation of documents as list of tuples (token, word_count) in corpus_wcRDD. From this RDD, we can compute a dictionary containing all tokens in the corpus as keys, and their respective number of occurrences as values.
Task: Using corpus_wcRDD compute a new RDD of (key, value) pairs, where keys are the tokens in the whole corpus and their respective values are the total number of occurences in the corpus.
Can you write Python code for it?

# scode: wcRDD = < FILL IN >
wcRDD = (corpus_wcRDD
         .flatMap(lambda x: x)
         .reduceByKey(lambda x, y: x + y))

print wcRDD.take(30)


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
Now, we have stored the whole text collection in two lists:

corpus_titles, which contains the titles of the selected articles
corpus_text, with the text content of the selected wikipedia articles

You can browse the content of the wikipedia articles to get some intuition about the kind of documents that will be processed.
Can you write Python code for it?

# n = 5
# print corpus_titles[n]
# print corpus_text[n]


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
Now, we will load the text collection into an RDD
Can you write Python code for it?

corpusRDD = sc.parallelize(corpus_text, 4)
print "\nRDD created with {0} elements".format(corpusRDD.count())

Test.assertTrue(corpusRDD.count() >= 100,
                "Your corpus_tokens has less than 100 articles. Consider using a larger dataset")


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
One of the advantages of the lemmatizer method is that the result of lemmatization is still a true word, which is more advisable for the presentation of text processing results and lemmatization.
However, without using contextual information, lemmatize() does not remove grammatical differences. This is the reason why "is" or "are" are preserved and not replaced by infinitive "be".
As an alternative, we can apply .lemmatize(word, pos), where 'pos' is a string code specifying the part-of-speech (pos), i.e. the grammatical role of the words in its sentence. For instance, you can check the difference between wnl.lemmatize('is') and wnl.lemmatize('is, pos='v').
2.4. Vectorization
Up to this point, we have transformed the raw text collection of articles in a list of articles, where each article is a collection of the word roots that are most relevant for semantic analysis. Now, we need to convert these data (a list of token lists) into a numerical representation (a list of vectors, or a matrix).
2.4.1. Word Count
As a first step, we compute the word count for every document in the corpus.
Task: Compute a new RDD from corpus_stemRDD where each element is a list of tuples related to a document. The key of each tuple is a token, and its value the number of occurrences of this token in the document. To do so, you can use method Counter from collections.
Can you write Python code for it?

# corpus_wcRDD = <FILL IN>
corpus_wcRDD = (corpus_stemRDD
                .map(collections.Counter)
                .map(lambda x: [(t, x[t]) for t in x]))

print corpus_wcRDD.take(1)[0][0:30]

Test.assertTrue(corpus_wcRDD.count() == n_art, 'List corpus_clean does not contain the expected number of articles')
Test.assertTrue(corpus_wcRDD.flatMap(lambda x: x).map(lambda x: x[1]).sum()== corpus_stemRDD.map(len).sum(),
                'The total token count in the output RDD is not consistent with the total number of input tokens')


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
Task: Create a method getTokenList with two inputs: a document (string) and a stopword list, and completes the first three steps of the corpus processing, as follows:

Tokenization: convert string to utf-8 and transform the string into a list of tokens, using word_tokenize from nltk.tokenize.
Homogeneization: transform capital letters to lowercase and remove non alphanumeric tokens.
Cleaning: remove stopwords

Return the result of cleaning (a list of tokens).
Can you write Python code for it?

def getTokenList(doc, stopwords_en):

    # scode: tokens = <FILL IN>   # Tokenize docs
    tokens = word_tokenize(doc.decode('utf-8'))

    # scode: tokens = <FILL IN>   # Remove non-alphanumeric tokens and normalize to lowercase
    tokens = [t.lower() for t in tokens if t.isalnum()]
    # scode: tokens = <FILL IN>   # Remove stopwords
    tokens = [t for t in tokens if t not in stopwords_en]

    return tokens

Test.assertEquals(getTokenList('The rain in spain stays mainly in the plane', stopwords_en),
                  [u'rain', u'spain', u'stays', u'mainly', u'plane'],
                  'getTokenList does not return the expected results')


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
Task: Lemmatize all documents corpus_tokensRDD using the .lemmatize() method, from the WordNetLemmatizer object created in the first line and save the result in a new RDD, corpus_lemRDD.
Can you write Python code for it?

wnl = WordNetLemmatizer()

# scode: corpus_lemmatRDD = <FILL IN>
corpus_lemmatRDD = (corpus_tokensRDD
                    .map(lambda x: [wnl.lemmatize(token) for token in x]))

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_lemmatRDD.take(1)[0][0:30]


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
The whole topics matrix can be computed using the .topicsMatrix() method.
Can you write Python code for it?

# Output topics. Each is a distribution over words (matching word count vectors)
print("Learned topics (as distributions over vocab of " + str(ldaModel.vocabSize()) + " words):")
topics = ldaModel.topicsMatrix()


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
We can visualize the token distribution using D and token_count, for the most frequent terms
Can you write Python code for it?

# SORTED TOKEN FREQUENCIES (II):
# plt.rcdefaults()

# Example data
n_bins = 25
y_pos = range(n_bins-1, -1, -1)
hot_tokens = D[0:n_bins]
z = [float(t)/n_art for t in token_count[0:n_bins]]

plt.barh(y_pos, z, align='center', alpha=0.4)
plt.yticks(y_pos, hot_tokens)
plt.xlabel('Average number of occurrences per article')
plt.title('Token distribution')
plt.show()


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
You can check the stopword list. This is a standard python list of strings. We could modify it by removing words or adding new ones if required.
Can you write Python code for it?

stopwords_en = stopwords.words('english')

print "The stopword list contains {0} elements: ".format(len(stopwords_en))
print stopwords_en


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib.ipynb".
The first task is:
You can try with any other categories. Take into account that the behavior of topic modelling algorithms may depend on the amount of documents available for the analysis. Select a category with at least 100 articles. You can browse the wikipedia category tree here, https://en.wikipedia.org/wiki/Category:Contents, for instance.
We start downloading the text collection.
Can you write Python code for it?

# Loading category data. This may take a while
print "Loading category data. This may take a while..."
cat_data = category.Category(site, cat)

corpus_titles = []
corpus_text = []

for n, page in enumerate(cat_data.getAllMembersGen()):
    print "\r Loading article {0}".format(n + 1),
    corpus_titles.append(page.title)
    corpus_text.append(page.getWikiText())

n_art = len(corpus_titles)
print "\nLoaded " + str(n_art) + " articles from category " + cat


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
1. Corpus acquisition.
In this notebook we will explore some tools for text processing and analysis and two topic modeling algorithms available from Python toolboxes.
To do so, we will explore and analyze collections of Wikipedia articles from a given category, using wikitools, that makes easy the capture of content from wikimedia sites.
(As a side note, there are many other available text collections to test topic modelling algorithm. In particular, the NLTK library has many examples, that can explore them using the nltk.download() tool.
import nltk
nltk.download()

for instance, you can take the gutemberg dataset
Mycorpus = nltk.corpus.gutenberg
text_name = Mycorpus.fileids()[0]
raw = Mycorpus.raw(text_name)
Words = Mycorpus.words(text_name)

Also, tools like Gensim or Sci-kit learn include text databases to work with).
In order to use Wikipedia data, we will select a single category of articles:
Can you write Python code for it?


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
2.2. Stemming / Lemmatization
Now we will apply stemming and lemmatization to corpus_tokensRDD. We will test our topic models over the resulting RDDs, to test their differences.
Task: Apply stemming to all documents corpus_tokensRDD and save the result in a new RDD, corpus_stemmedRDD.
Can you write Python code for it?

# Select stemmer.
stemmer = nltk.stem.SnowballStemmer('english')

# scode: corpus_stemRDD = <FILL IN>

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_stemRDD.take(1)[0][0:30]

Test.assertTrue((len([c for c in corpus_stemRDD.take(1)[0] if c!=stemmer.stem(c)])
                 < 0.1*len(corpus_stemRDD.take(1)[0])),
                'It seems that stemming has not been applied properly')


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
3. Latent Dirichlet Allocation
In order to apply the LDA algorithm, we need to represent the input documents in the format required by MLlib. More specifically. The input data should be an RDD where each element is a tuple
(doc_id, vector)

where doc_id is an integer document identifier, and vector can be a sparse or dense vector from class Vectors. We will use sparse vectors, which are more adequate for large vocabularies.
To compute the sparse vectors, we must first transform the lists of tuples (token, value) in wcRDD into a lists of (token_id, value), pairs, thus replacing each token by a numerical identifier.
We will proceed in two steps:

Compute an inverse dictionary, invD, transforming tokens into numbers.
Apply the inverse dictionary to compute a new RDD from wcRDD replacing each token by its token_id.

 Task: complete the two steps outlined above.
Can you write Python code for it?

# Compute inverse dictionary
# scode: invD = <FILL IN>

# Compute RDD replacing tokens by token_ids
# scode: corpus_sparseRDD = <FILL IN>

# Convert list of tuplas into Vectors.sparse object.


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
Alternatively, we can use the .describeTopics method that returns the most relevan terms for each topic, and it is more useful for a graphical plot.
Task: Represent the 25 most relevant terms for each topic using bar plots.
Can you write Python code for it?

n_bins = 25

# Example data
y_pos = range(n_bins-1, -1, -1)

pylab.rcParams['figure.figsize'] = 16, 8  # Set figure size
for i in range(3):

    # scode: topic = <FILL IN>  # Get 25 most relevant tokens for topic i

    tokens = [D[n] for n in topic[0]]
    weights = topic[1]

    plt.subplot(1, 3, i+1)
    plt.barh(y_pos, weights, align='center', alpha=0.4)
    plt.yticks(y_pos, tokens)
    plt.xlabel('Average number of occurrences per article')
    plt.title('Token distribution')


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
At this point, we have got a representation of documents as list of tuples (token, word_count) in corpus_wcRDD. From this RDD, we can compute a dictionary containing all tokens in the corpus as keys, and their respective number of occurrences as values.
Task: Using corpus_wcRDD compute a new RDD of (key, value) pairs, where keys are the tokens in the whole corpus and their respective values are the total number of occurences in the corpus.
Can you write Python code for it?

# scode: wcRDD = < FILL IN >

print wcRDD.take(30)


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
Now, we have stored the whole text collection in two lists:

corpus_titles, which contains the titles of the selected articles
corpus_text, with the text content of the selected wikipedia articles

You can browse the content of the wikipedia articles to get some intuition about the kind of documents that will be processed.
Can you write Python code for it?


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
One of the advantages of the lemmatizer method is that the result of lemmatization is still a true word, which is more advisable for the presentation of text processing results and lemmatization.
However, without using contextual information, lemmatize() does not remove grammatical differences. This is the reason why "is" or "are" are preserved and not replaced by infinitive "be".
As an alternative, we can apply .lemmatize(word, pos), where 'pos' is a string code specifying the part-of-speech (pos), i.e. the grammatical role of the words in its sentence. For instance, you can check the difference between wnl.lemmatize('is') and wnl.lemmatize('is, pos='v').
2.4. Vectorization
Up to this point, we have transformed the raw text collection of articles in a list of articles, where each article is a collection of the word roots that are most relevant for semantic analysis. Now, we need to convert these data (a list of token lists) into a numerical representation (a list of vectors, or a matrix).
2.4.1. Word Count
As a first step, we compute the word count for every document in the corpus.
Task: Compute a new RDD from corpus_stemRDD where each element is a list of tuples related to a document. The key of each tuple is a token, and its value the number of occurrences of this token in the document. To do so, you can use method Counter from collections.
Can you write Python code for it?

# corpus_wcRDD = <FILL IN>

print corpus_wcRDD.take(1)[0][0:30]

Test.assertTrue(corpus_wcRDD.count() == n_art, 'List corpus_clean does not contain the expected number of articles')
Test.assertTrue(corpus_wcRDD.flatMap(lambda x: x).map(lambda x: x[1]).sum()== corpus_stemRDD.map(len).sum(),
                'The total token count in the output RDD is not consistent with the total number of input tokens')


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
Task: Apply getTokenList to all documents in the corpus and save the result in a corpus_tokensRDD
Can you write Python code for it?

# scode: corpus_tokensRDD = <FILL IN>

# print "\n Let's check tokens after cleaning:"
print corpus_tokensRDD.take(1)[0][0:30]

Test.assertEquals(corpus_tokensRDD.count(), n_art,
                  "The number of documents in the original set does not correspond to the size of corpus_tokensRDD")
Test.assertTrue(all([c==c.lower() for c in corpus_tokensRDD.take(1)[0]]), 'Capital letters have not been removed')
Test.assertTrue(all([c.isalnum() for c in corpus_tokensRDD.take(1)[0]]),
                'Non alphanumeric characters have not been removed')
Test.assertTrue(len([c for c in corpus_tokensRDD.take(1)[0] if c in stopwords_en])==0,
                'Stopwords have not been removed')


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
Task: Create a method getTokenList with three inputs: a document (string), a stemmer object (created with nltk.stem) and a stopword list, and completes the first three steps of the corpus processing, as follows:

Tokenization: convert string to utf-8 and transform the string into a list of tokens, using word_tokenize from nltk.tokenize.
Homogeneization: transform capital letters to lowercase and remove non alphanumeric tokens.
Cleaning: remove stopwords

Return the result of cleaning (a list of tokens).
Can you write Python code for it?

def getTokenList(doc, stopwords_en):

    # scode: tokens = <FILL IN>   # Tokenize docs
    # scode: tokens = <FILL IN>   # Remove non-alphanumeric tokens and normalize to lowercase
    # scode: tokens = <FILL IN>   # Remove stopwords

    return tokens

Test.assertEquals(getTokenList('The rain in spain stays mainly in the plane', stopwords_en),
                  [u'rain', u'spain', u'stays', u'mainly', u'plane'],
                  'getTokenList does not return the expected results')


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
Task: Lemmatize all documents corpus_tokensRDD using the .lemmatize() method, from the WordNetLemmatizer object created in the first line and save the result in a new RDD, corpus_lemRDD.
Can you write Python code for it?

wnl = WordNetLemmatizer()

# scode: corpus_lemmatRDD = <FILL IN>

print "\nLet's check the first tokens from document 0 after stemming:"
print corpus_lemmatRDD.take(1)[0][0:30]


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
Task: Take all tuples in wcRDD in decreasing order of the number of token counts in variable TD and compute two lists:

token_count: a list of token counts, in decreasing order.
D: A list of tokens, in the same order.
Can you write Python code for it?

# Token Dictionary:
n_tokens = wcRDD.count()
# scode: TD = wcRDD.<FILL IN>
# scode: D = <FIll IN>    # Extract tokens from TD
# scode: token_count = <FILL IN>   # Extract token counts from TD


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
That's all. We can already call to the lda algorithm.'
Task: Train an LDA model with 3 topics and the corpus obtained in corpus4lda. Check the LDA documentation to find the appropriate command.
Can you write Python code for it?

print "Training LDA: this might take a while..."
# scode: ldaModel = LDA.<FILL IN>


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
The whole topics matrix can be computed using the .topicsMatrix() method.
Can you write Python code for it?


I am working on the file "TM3.Topic_Models_with_MLlib/TM3_TMwithMLlib_student.ipynb".
The first task is:
We can visualize the token distribution using D and token_count, for the most frequent terms
Can you write Python code for it?


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
1.1.2. Parsing XML
XML is an inherently hierarchical data format, and the most natural way to represent it is with a tree. The ElementTree module has two classes for this purpose:

ElementTree represents the whole XML document as a tree
Element represents a single node in this tree

We can import XLM data by reading an XML file:
Can you write Python code for it?

import xml.etree.ElementTree as ET
tree = ET.parse(xmlfile)
root = tree.getroot


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
2. Corpus Processing
Topic modelling algorithms process vectorized data. In order to apply them, we need to transform the raw text input data into a vector representation. To do so, we will remove irrelevant information from the text data and preserve as much relevant information as possible to capture the semantic content in the document collection.
Thus, we will proceed with the following steps:

Tokenization
Homogeneization
Cleaning
Vectorization

2.1. Tokenization
For the first steps, we will use some of the powerful methods available from the Natural Language Toolkit. In order to use the word_tokenize method from nltk, you might need to get the appropriate libraries using nltk.download(). You must select option "d) Download", and identifier "punkt"
Can you write Python code for it?

import nltk

# You should comment this code fragment if the package is already available.
# Select option "d) Download", and identifier "punkt"
# nltk.download()


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
2.2. Homogeneization
By looking at the tokenized corpus you may verify that there are many tokens that correspond to punktuation signs and other symbols that are not relevant to analyze the semantic content. They can be removed using the stemming or lemmatization tools from nltk.
The homogeneization process will consist of:

Removing capitalization: capital alphabetic characters will be transformed to their corresponding lowercase characters.
Removing non alphanumeric tokens (e.g. punktuation signs)
Stemming/Lemmatization: removing word terminations to preserve the root of the words and ignore grammatical information.

Exercise
Convert all tokens in tokenized_abstracts to lowercase (using the .lower() method) and remove non alphanumeric tokens (that you can detect with .isalnum() method). You can complete the following code fragment with a single line of code ...
Can you write Python code for it?

filtered_abstracts = []

for n, abstract in enumerate(tokenized_abstracts):
    if not n%100:
        print('\rFiltering abstract', n, 'out of', nprojects, end='', flush=True)

    #<SOL>
    #</SOL>

print('\n',filtered_abstracts[0])


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
2.2.1. Stemming vs Lemmatization
At this point, we can choose between applying a simple stemming or ussing lemmatization. We will try both to test their differences.
The lemmatizer from NLTK is based on WordNet. If you have not used wordnet before, you will likely need to download it from nltk (use the nltk.download() command)
Can you write Python code for it?

stemmer = nltk.stem.SnowballStemmer('english')
from nltk.stem import WordNetLemmatizer
wnl = WordNetLemmatizer()

print('Result for the first abstract in dataset applying stemming')
print([stemmer.stem(el) for el in filtered_abstracts[0]])

print('Result for the first abstract in the dataset applying lemmatization')
print([wnl.lemmatize(el) for el in filtered_abstracts[0]])


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
2.3. Cleaning
The third step consists of removing those words that are very common in language and do not carry out usefull semantic content (articles, pronouns, etc).
Once again, we might need to load the stopword files using the download tools from nltk
Exercise
In the second line below we read a list of common english stopwords. Clean lemmatized_abstracts by removing all tokens in the stopword list.
Can you write Python code for it?

from nltk.corpus import stopwords
stopwords_en = stopwords.words('english')

clean_abstracts = []

for n, abstract in enumerate(lemmatized_abstracts):
    if not n%100:
        print('\rCleaning abstract', n, 'out of', nprojects, end='', flush=True)

    # Remove all tokens in the stopwords list and append the result to clean_abstracts
    # <SOL>
    # </SOL>
    clean_abstracts.append(clean_tokens)

print('\n Let us check tokens after cleaning:')
print(clean_abstracts[0])


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
3.2. LDA model visualization
Gensim provides a basic visualization of the obtained topics:
Can you write Python code for it?


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
3.3. Gensim utility functions
In addition to visualization purposes, topic models are useful to obtain a semantic representation of documents that can later be used with some other purpose:

In document classification problems
In content-based recommendations systems

Essentially, the idea is that the topic model provides a (semantic) vector representation of documents, and use probability divergences to measure document similarity. The following functions of the LdaModel class will be useful in this context:

get_topic_terms(topic_id): Gets vector of the probability distribution among words for the indicated topic
get_document_topics(bow_vector): Gets (sparse) vector with the probability distribution among topics for the provided document
Can you write Python code for it?


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
A more useful visualization is provided by the python LDA visualization library, pyLDAvis.
Before executing the next code fragment you need to install pyLDAvis:
>> pip install (--user) pyLDAvis
Can you write Python code for it?

import pyLDAvis.gensim as gensimvis
import pyLDAvis

vis_data = gensimvis.prepare(ldag, corpus_bow, D)
pyLDAvis.display(vis_data)


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
An alternative to the use of the get_document_topics() function is to directly transform a dataset using the ldag object as follows. You can apply this transformation to several documents at once, but then the result is an iterator from which you can build the corresponding list if necessary
Can you write Python code for it?

print(ldag[corpus_bow[0]])

print('When applied to a dataset it will provide an iterator')
print(ldag[corpus_bow[:3]])

print('We can rebuild the list from the iterator with a one liner')
print([el for el in ldag[corpus_bow[:3]]])


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
At this point, it is good to make sure to understand what has happened. In clean_abstracts we had a list of token lists. With it, we have constructed a Dictionary, D, which assigns an integer identifier to each token in the corpus.
After that, we have transformed each article (in clean_abstracts) in a list tuples (id, n).
Can you write Python code for it?

print('Original article (after cleaning):')
print(clean_abstracts[0])
print('Sparse vector representation (first 10 components):')
print(corpus_bow[0][:10])
print('Word counts for the first project (first 10 components):')
print(list(map(lambda x: (D[x[0]], x[1]), corpus_bow[0][:10])))


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
Exercise
Build a function that computes the semantic distance between two documents. For this, you can use the functions (or code fragments) provided in the library dist_utils.py.
Can you write Python code for it?

def pairwase_dist(doc1, doc2):
    """This function returns the Jensen-Shannon
    distance between the corresponding vectors of the documents

    : doc1: Semantic representation for the doc1 (a vector of length ntopics)
    : doc2: Semantic representation for the doc2 (a vector of length ntopics)

    : Returns: The JS distance between doc1 and doc2 (a number)
    """
    #<SOL>
    #</SOL>


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
Exercise
Build a function that returns the most relevant projects for a given topic
Can you write Python code for it?

def most_relevant_projects(ldag, topicid, corpus_bow, nprojects=10):
    """This function returns the most relevant projects in corpus_bow

    : ldag: The trained topic model object provided by gensim
    : topicid: The topic for which we want to find the most relevant documents
    : corpus_bow: The BoW representation of documents in Gensim format
    : nprojects: Number of most relevant projects to identify

    : Returns: A list with the identifiers of the most relevant projects
    """

    print('Computing most relevant projects for Topic', topicid)
    print('Topic composition is:')
    print(ldag.show_topic(topicid))

    #<SOL>
    #</SOL>

#To test the function we will find the most relevant projects for a subset of the NSF dataset
project_id = most_relevant_projects(ldag, 17, corpus_bow[:10000])

#Print titles of selected projects
for idproject in project_id:
    print(NSF_data[idproject]['title'])


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
Exercise
Compute the average length of the abstracts of all projects in the dataset
Can you write Python code for it?


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
Finally, Gensim provides some useful functions to convert between formats, and to simplify interaction with numpy and scipy. The following code fragment converts a corpus in sparse format to a full numpy matrix
Can you write Python code for it?

reduced_corpus = [el for el in ldag[corpus_bow[:3]]]
reduced_corpus = gensim.matutils.corpus2dense(reduced_corpus, num_topics).T
print(reduced_corpus)


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
Function that creates the Node CSV file for Gephi
Can you write Python code for it?

#print(NSF_data[0].keys())
#print(NSF_data[0]['institution'])

def strNone(str_to_convert):
    if str_to_convert is None:
        return ''
    else:
        return str_to_convert

with open('NSF_nodes.csv','w') as fout:
    fout.write('Id;Title;Year;Budget;UnivName;UnivZIP;State\n')
    for project in NSF_data:
        fout.write(project['project_code']+';'+project['title']+';')
        fout.write(project['year']+';'+str(project['budget'])+';')
        fout.write(project['institution'][0]+';')
        fout.write(strNone(project['institution'][1])+';')
        fout.write(strNone(project['institution'][2])+'\n')


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
In the second step, let us create a numerical version of our corpus using the doc2bow method. In general, D.doc2bow(token_list) transforms any list of tokens into a list of tuples (token_id, n), one per each token in token_list, where token_id is the token identifier (according to dictionary D) and n is the number of occurrences of such token in token_list.
Can you write Python code for it?

corpus_bow = [D.doc2bow(doc) for doc in clean_abstracts]


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
One of the advantages of the lemmatizer method is that the result of lemmmatization is still a true word, which is more advisable for the presentation of text processing results and lemmatization.
However, without using contextual information, lemmatize() does not remove grammatical differences. This is the reason why "is" or "are" are preserved and not replaced by infinitive "be".
As an alternative, we can apply .lemmatize(word, pos), where 'pos' is a string code specifying the part-of-speech (pos), i.e. the grammatical role of the words in its sentence. For instance, you can check the difference between wnl.lemmatize('is') and wnl.lemmatize('is, pos='v').
Exercise
Complete the following code fragment to lemmatize all abstracts in the NSF dataset
Can you write Python code for it?

lemmatized_abstracts = []

for n, abstract in enumerate(filtered_abstracts):
    if not n%100:
        print('\rLemmatizing abstract', n, 'out of', nprojects, end='', flush=True)

    #<SOL>
    #</SOL>

print('Result for the first abstract in the dataset applying lemmatization')
print('\n',lemmatized_abstracts[0])


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
The presented classes and functions are all you need to solve the following exercise. However, there are many other interesting functions that can probably make it easier for you to work with XML files. For more information, please refer to the ElementTree API.
1.1.3. Exercise: Parsing the XML project files
Implement a function that parses the XML files and provides as its output a dictionary with fields:
project_code      (string)
title             (string)
abstract          (string)
budget            (float)
year              (string)
institution       (tuple with elements: name, zipcode, and statecode)
Can you write Python code for it?

def parse_xmlproject(xml_string):
    """This function processess the specified XML field,
    and outputs a dictionary with the desired project information

    :xml_string: String with XML content
    :Returns: Dictionary with indicated files
    """

    #<SOL>
    #</SOL>

parse_xmlproject(open(xmlfile,'r').read())


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
We will create a list that contains just the abstracts in the dataset. As the order of the elements in a list is fixed, it will be later straightforward to match the processed abstracts to metadata associated to their corresponding projects.
Can you write Python code for it?

from nltk.tokenize import word_tokenize

NSF_abstracts = list(map(lambda x: x['abstract'], NSF_data))

tokenized_abstracts = []
nprojects = len(NSF_abstracts)

for n, abstract in enumerate(NSF_abstracts):
    if not n%100:
        print('\rTokenizing abstract', n, 'out of', nprojects, end='', flush=True)
    tokenized_abstracts.append(word_tokenize(abstract))

print('\n\n The corpus has been tokenized. Check the result for the first abstract:')
print(NSF_abstracts[0])
print(tokenized_abstracts[0])


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
We will extract some characteristics of the constructed dataset:
Can you write Python code for it?

print('Number of projects in dataset:', len(NSF_data))

####
budget_data = list(map(lambda x: x['budget'], NSF_data))
print('Average budget of projects in dataset:', np.mean(budget_data))

####
insti_data = list(map(lambda x: x['institution'], NSF_data))
print('Number of unique institutions in dataset:', len(set(insti_data)))

####
counts = dict()
for project in NSF_data:
    counts[project['year']] = counts.get(project['year'],0) + 1

print('Breakdown of projects by starting year:')
for el in counts:
    print(el, ':', counts[el])


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
fromstring() parses XML from a string directly into an Element, which is the root element of the parsed tree. Other parsing functions may create an ElementTree, but we will not cover them here.
As an Element, root has a tag and a dictionary of attributes:
Can you write Python code for it?


I am working on the file "TM4.DTUCourse/notebook/DTU02901_student.ipynb".
The first task is:
or directly reading a string:
Can you write Python code for it?


I am working on the file "TOM.ipynb".
The first task is:
Estimating the optimal number of topics ($k$)
Non-negative matrix factorization approximates $A$, the document-term matrix, in the following way:
$$
A \approx HW
$$
where $H$ is a $n \times k$ matrix that describes the documents in terms of topics, and $W$ is a $k \times m$ matrix that describes topics in terms of words. More precisely, the coefficient $h_{i,j}$ defines the importance of topic $j$ in article $i$, and the coefficient $w_{i,j}$ defines the importance of word $j$ in topic $i$.
Determining an appropriate value of $k$ is critical to ensure a pertinent analysis of the EGC anthology. If $k$ is too small, then the discovered topics will be too vague; if $k$ is too large, then the discovered topics will be too narrow and may be redundant. To help us with this task, we compute two metrics implemented in TOM : the stability metric proposed by Greene et al. (2014) and the spectral metric proposed by Arun et al. (2010).
Can you write Python code for it?

from tom_lib.nlp.topic_model import NonNegativeMatrixFactorization

topic_model = NonNegativeMatrixFactorization(corpus)


I am working on the file "TOM.ipynb".
The first task is:
Guided by the two metrics described previously, we manually evaluate the quality of the topics identified with $k$ varying between 15 and 20. Eventually, we judge that the best results are achieved with NMF for $k=15$.
Can you write Python code for it?

k = 15
topic_model.infer_topics(num_topics=k)


I am working on the file "TOM.ipynb".
The first task is:
In the following, we leverage the discovered topics to highlight interesting particularities about the EGC society. To be able to analyze the topics, supplemented with information about the related papers, we partition the papers into 15 non-overlapping clusters, i.e. a cluster per topic. Each article $i \in [0;1-n]$ is assigned to the cluster $j$ that corresponds to the topic with the highest weight $w_{ij}$:
\begin{equation}
\text{cluster}i = \underset{j}{\mathrm{argmax}}(w{i,j})
\label{eq:cluster}
\end{equation}
Global topic proportions
Can you write Python code for it?

p = figure(x_range=[str(_) for _ in range(k)], plot_height=350, x_axis_label='topic', y_axis_label='proportion')
p.vbar(x=[str(_) for _ in range(k)], top=topic_model.topics_frequency(), width=0.7)
show(p)


I am working on the file "TOM.ipynb".
The first task is:
Shifting attention, evolving interests
Here we focus on topics topics 12 (social network analysis and mining) and 3 (association rule mining). The following figures describe these topics in terms of their respective top 10 words and top 3 documents.
Can you write Python code for it?

def plot_top_words(topic_id):
    words = [word for word, weight in topic_model.top_words(topic_id, 10)]
    weights = [weight for word, weight in topic_model.top_words(topic_id, 10)]

    p = figure(x_range=words, plot_height=300, plot_width=800, x_axis_label='word', y_axis_label='weight')
    p.vbar(x=words, top=weights, width=0.7)
    show(p)

def top_documents_df(topic_id):
    top_docs = topic_model.top_documents(topic_id, 3)
    d = {'Article title': [corpus.title(doc_id) for doc_id, weight in top_docs], 'Year': [int(corpus.date(doc_id)) for doc_id, weight in top_docs]}
    df = pd.DataFrame(data=d)
    return df


I am working on the file "TOM.ipynb".
The first task is:
Top 3 articles
Can you write Python code for it?


I am working on the file "TOM.ipynb".
The first task is:
Topic #12
Top 10 words
Can you write Python code for it?


I am working on the file "TOM.ipynb".
The first task is:
Topic #3
Top 10 words
Can you write Python code for it?


I am working on the file "TOM.ipynb".
The first task is:
Weighted Jaccard average stability
The figure below shows this metric for a number of topics varying between 5 and 50 (higher is better).
Can you write Python code for it?

from bokeh.io import show, output_notebook
from bokeh.plotting import figure
output_notebook()

p = figure(plot_height=250)
p.line(range(10, 51), topic_model.greene_metric(min_num_topics=10, step=1, max_num_topics=50, top_n_words=10, tao=10), line_width=2)
show(p)


I am working on the file "TOOLS/atlas-marsfrost.ipynb".
The first task is:
Carte en moyenne temporelle de $p_{sat}$ pour $CO_2$
Can you write Python code for it?

def psatco2(temp):
  return 1.382 * 1e12 * np.exp(-3182.48/temp)

temp = np.linspace(100,200,81)
plt.yscale('log')
plt.plot(temp,psatco2(temp))
plt.show()

# Paramètres utilisateurs -----------------------------------------
earthtopo = False # ajouter les traits de côte actuels
pco2atmo = 610. # CO2 pressure
vmin = 0.
vmax = 1.

# Code ------------------------------------------------------------
tsurfnc = data.variables['tsurf'][:,:,:]
dataplt = pco2atmo/psatco2(tsurfnc)

fig = plt.figure(figsize=(12,8))
ax = plt.axes(projection=ccrs.PlateCarree())
ax.set_global()
if (earthtopo): ax.coastlines(resolution="110m",linewidth=1)
gl = ax.gridlines(linestyle='--',color='black',
             draw_labels=True)
gl.xlabels_top = False
gl.ylabels_right = False
clevs = np.linspace(vmin,vmax,21)
plt.contourf(longitude, latitude, np.mean(dataplt[:,:,:],axis=0),
             clevs, transform=ccrs.PlateCarree(),cmap="jet")
plt.title(r"Saturation ratio", size=14)
cb = plt.colorbar(ax=ax, orientation="vertical", pad=0.02, aspect=16, shrink=0.8)
cb.set_label(r'Pa',size=12,rotation=0,labelpad=15)
cb.ax.tick_params(labelsize=10)
plt.show()


I am working on the file "TOOLS/atlas-marsfrost.ipynb".
The first task is:
Carte en moyenne temporelle de $p_{sat}$ pour $H_2O$
Can you write Python code for it?

def psatw(temp):
  # METHOD GOFF GRATCH (HygroLP) - OVER WATER
  # -----------------------------------------
  log10ew = -7.90298*(373.16/temp-1) \
    + 5.02808 * np.log10(373.16/temp) \
    - 1.3816e-7 * (10**(11.344 * (1-temp/373.16))-1) \
    + 8.1328e-3 * (10**(-3.49149 *(373.16/temp-1))-1) \
    + np.log10(1013.246)
  return 100 * (10**(log10ew))

def psati(temp):
  # METHOD GOFF GRATCH (HygroLP) - OVER ICE
  # ---------------------------------------
  log10ei = -9.09718*(273.16/temp-1) \
    - 3.56654*np.log10(273.16/temp) \
    + 0.876793*(1-temp/273.16) \
    + np.log10(6.1071)
  return  100 * (10**(log10ei))

tzero = 273.15
temp = np.linspace(-80+tzero,tzero,81)
plt.yscale('log')
plt.plot(temp,psatw(temp))
plt.plot(temp,psati(temp))
plt.show()

# Paramètres utilisateurs -----------------------------------------
earthtopo = False # ajouter les traits de côte actuels
ph2oatmo = 0.05e-2*610. # assumed mean water vapor partial pressure
vmin = 0.
vmax = 1.

# Code ------------------------------------------------------------
tsurfnc = data.variables['tsurf'][:,:,:]
dataplt = ph2oatmo/psati(tsurfnc)

fig = plt.figure(figsize=(12,8))
ax = plt.axes(projection=ccrs.PlateCarree())
ax.set_global()
if (earthtopo): ax.coastlines(resolution="110m",linewidth=1)
gl = ax.gridlines(linestyle='--',color='black',
             draw_labels=True)
gl.xlabels_top = False
gl.ylabels_right = False
clevs = np.linspace(vmin,vmax,21)
plt.contourf(longitude, latitude, np.mean(dataplt[:,:,:],axis=0),
             clevs, transform=ccrs.PlateCarree(),cmap="jet")
plt.title(r"Saturation ratio", size=14)
cb = plt.colorbar(ax=ax, orientation="vertical", pad=0.02, aspect=16, shrink=0.8)
cb.set_label(r'Pa',size=12,rotation=0,labelpad=15)
cb.ax.tick_params(labelsize=10)
plt.show()


I am working on the file "TOOLS/atlas-veget.ipynb".
The first task is:
Carte en moyenne temporelle de $\chi$
Can you write Python code for it?

def psatw(temp):
  # METHOD GOFF GRATCH (HygroLP) - OVER WATER
  # -----------------------------------------
  log10ew = -7.90298*(373.16/temp-1) \
    + 5.02808 * np.log10(373.16/temp) \
    - 1.3816e-7 * (10**(11.344 * (1-temp/373.16))-1) \
    + 8.1328e-3 * (10**(-3.49149 *(373.16/temp-1))-1) \
    + np.log10(1013.246)
  return 100 * (10**(log10ew))

def psati(temp):
  # METHOD GOFF GRATCH (HygroLP) - OVER ICE
  # ---------------------------------------
  log10ei = -9.09718*(273.16/temp-1) \
    - 3.56654*np.log10(273.16/temp) \
    + 0.876793*(1-temp/273.16) \
    + np.log10(6.1071)
  return  100 * (10**(log10ei))

def leafratio(temp,pvap,alti):
    # all inputs in ISU
    vpd = (psatw(temp) - pvap)/1e3 # in kPa
    xhi=(expit(0.0545*(temp-273.15-25.)-0.5*np.log(vpd)-0.0815*alti*1e-3+1.189))
    return xhi

# Paramètres utilisateurs -----------------------------------------
earthtopo = False # ajouter les traits de côte actuels
vmin = 0.
vmax = 1.

# Code ------------------------------------------------------------
psnc = data.variables['ps'][:,:,:]
tempnc = data.variables['temp'][:,:,:,:]
qvapnc = data.variables['h2o_vap'][:,:,:,:]
phisnc = data.variables['phisinit'][:,:]

pvap = qvapnc[:,0,:,:]*psnc[:,:,:]/0.622 # en Pa
alti = phisnc/9.81 # en metres

mask = alti
alti[alti <= 0] = np.nan
alti[alti > 0] = 1.

dataplt = np.zeros([len(Time),len(longitude),len(latitude)])
for itime in range(len(Time)):
    for ilon in range(len(longitude)):
        for ilat in range(len(latitude)):
            dataplt[itime,ilon,ilat] = leafratio(tempnc[itime,0,ilon,ilat],
                                                 pvap[itime,ilon,ilat],
                                                 alti[ilon,ilat])

fig = plt.figure(figsize=(12,8))
ax = plt.axes(projection=ccrs.PlateCarree())
ax.set_global()
if (earthtopo): ax.coastlines(resolution="110m",linewidth=1)
gl = ax.gridlines(linestyle='--',color='black',
             draw_labels=True)
gl.xlabels_top = False
gl.ylabels_right = False
clevs =...


I am working on the file "TOOLS/atlas-veget.ipynb".
The first task is:
Carte en moyenne temporelle sur la totalité de l'expérience
Can you write Python code for it?

# Paramètres utilisateurs -----------------------------------------
earthtopo = False # ajouter les traits de côte actuels
varname = 'tsurf'
vmin = 220
vmax = 350

# Code ------------------------------------------------------------
dataplt = data.variables[varname][:,:,:]

fig = plt.figure(figsize=(12,8))
ax = plt.axes(projection=ccrs.PlateCarree())
ax.set_global()
if (earthtopo): ax.coastlines(resolution="110m",linewidth=1)
gl = ax.gridlines(linestyle='--',color='black',
             draw_labels=True)
gl.xlabels_top = False
gl.ylabels_right = False
clevs = np.linspace(vmin,vmax,29)
plt.contourf(longitude, latitude, np.mean(dataplt[:,:,:],axis=0),
             clevs, transform=ccrs.PlateCarree(),cmap="jet")
plt.title(r"Température de surface moyenne", size=14)
cb = plt.colorbar(ax=ax, orientation="vertical", pad=0.02, aspect=16, shrink=0.8)
cb.set_label(r'K',size=12,rotation=0,labelpad=15)
cb.ax.tick_params(labelsize=10)
plt.show()


I am working on the file "TOOLS/atlas.ipynb".
The first task is:
$L_s$ est la longitude solaire, qui sert à désigner la position de la planète sur son orbite. Cumul des années pour les représentations graphiques :
Can you write Python code for it?

dafirst = Time[0]
daint = Time[1] - dafirst
dalast = dafirst + (len(Time)-1)*daint
year = 0.
add = np.linspace(dafirst,dalast,num=len(Time)) ; add[0] = 0.
for iii in range(1,len(Ls)):
    if Ls[iii] - Ls[iii-1] < 0: year = year+1.
    add[iii] = year*360.
Ls_true = add + Ls


I am working on the file "TOOLS/atlas.ipynb".
The first task is:
Bilan énergétique
Quatre figures donnant en fonction du temps $L_s$ et de la latitude :
1. le flux solaire incident au sommet $F_{vis}^{\downarrow_{TOA}}$,
2. le flux solaire absorbé par la surface $F_{vis}^{\downarrow_{Surf}}$,
3. $F_{IR}^{\uparrow_{TOA}}$ le flux infrarouge émis au sommet,
4. la température de surface $T_s$.
Can you write Python code for it?

ISR=data.variables['ISR'][:,:,:]
ASR=data.variables['ASR'][:,:,:]
OLR=data.variables['OLR'][:,:,:]
tsurf=data.variables['tsurf'][:,:,:]
ncontour = 20

fig, axs = plt.subplots(2, 2,figsize=(12,10))
im1 = axs[0, 0].contourf(Ls_true,latitude,np.mean(ISR[:,:,:],axis=2).transpose(),
                         ncontour,cmap='jet')
#axs[0, 0].cbar()
axs[0, 0].set_ylim([-90,90])
axs[0, 0].set_title('Incoming Solar Radiation')
fig.colorbar(im1,ax=axs[0, 0])

im2 = axs[0, 1].contourf(Ls_true,latitude,np.mean(ASR[:,:,:],axis=2).transpose(),
                         ncontour,cmap='jet')
axs[0, 1].set_title('Absorbed Solar Radiation')
fig.colorbar(im2,ax=axs[0, 1])

im3 = axs[1, 0].contourf(Ls_true,latitude,np.mean(OLR[:,:,:],axis=2).transpose(),
                         ncontour,cmap='jet')
axs[1, 0].set_title('OLR')
fig.colorbar(im3,ax=axs[1, 0])

im4 = axs[1, 1].contourf(Ls_true,latitude,(np.mean(tsurf[:,:,:],axis=2)-273).transpose(),
                         ncontour,cmap='bwr',vmin= -90,vmax=60)
axs[1, 1].set_title('Tsurf')
fig.colorbar(im4,ax=axs[1, 1])

for ax in axs.flat:
    ax.set(xlabel='Ls', ylabel='Latitude')

# Hide x labels and tick labels for top plots and y ticks for right plots.
for ax in axs.flat:
    ax.label_outer()


I am working on the file "TOOLS/atlas.ipynb".
The first task is:
Carte d'un champ à Ls et z donnés
Can you write Python code for it?

# Paramètres utilisateurs -----------------------------------------
alt_user = 1. # l'altitude a laquelle on veut regarder la variable 4D
year_user = 1 # année de simulation à regarder
Ls_user = 130.

varname = 'temp'
vmin = 180. # bornes de l'echelle de couleur, a ajuster pour
vmax = 360. # une meilleure visualisation

# Code ------------------------------------------------------------
alt = data.variables['altitude'][:]
alt_ind = np.where(abs(alt-alt_user)==
                  abs(alt-alt_user).min())[0]
print("La valeur la plus proche trouvée est altitude = "
      + str(alt[alt_ind]))

Ls_true_user = year_user*360. + Ls_user

Ls_ind = np.where(abs(Ls_true-Ls_true_user)==
                  abs(Ls_true-Ls_true_user).min())[0]
print("La valeur la plus proche trouvée est Ls = "
      + str(Ls_true[Ls_ind]-year_user*360.)
      + " pour l'année " + str(year_user))

var = data.variables[varname][:,:,:,:]
p = data.variables['p'][:,:,:,:]

fig, ax = plt.subplots(figsize=(14,6))
xplt=longitude
yplt = latitude
dataplt = np.squeeze(np.mean(var[Ls_ind,alt_ind,:,:],axis=0))
im = ax.contourf(xplt,yplt,dataplt,60,cmap='jet',
                 vmax=vmax, vmin=vmin)
ax.set_title("Carte de "+varname)
ax.set(xlabel='Longitude', ylabel='Latitude')
fig.colorbar(im,ax=ax)
# -----------------------------------------------------------------


I am working on the file "TOOLS/atlas.ipynb".
The first task is:
Carte de la valeur moyenne d'un champ dans un intervalle de $L_s$
Can you write Python code for it?

# Paramètres utilisateurs -----------------------------------------
earthtopo = False # ajouter les traits de côte actuels
year_user = 1 # année de simulation à regarder
Ls_user_min = 120. # longitude solaire choisie
Ls_user_max = 140. # longitude solaire choisie

varname = 'tsurf'
vmin = 180
vmax = 330

# Code ------------------------------------------------------------
Ls_true_user_min = year_user*360. + Ls_user_min
Ls_true_user_max = year_user*360. + Ls_user_max

Lsmin_ind = np.where(abs(Ls_true-Ls_true_user_min)==
                  abs(Ls_true-Ls_true_user_min).min())[0]
print("La valeur la plus proche trouvée est Ls_min = "
      + str(Ls_true[Lsmin_ind]-year_user*360.)
      + " pour l'année " + str(year_user))

Lsmax_ind = np.where(abs(Ls_true-Ls_true_user_max)==
                  abs(Ls_true-Ls_true_user_max).min())[0]
print("La valeur la plus proche trouvée est Ls_max = "
      + str(Ls_true[Lsmax_ind]-year_user*360.)
      + " pour l'année " + str(year_user))

# Code ------------------------------------------------------------
var = data.variables[varname][:,:,:]
vartmp = var[Lsmin_ind[0]:Lsmax_ind[0],:,:]
dataplt = np.mean(vartmp,axis=0)

fig = plt.figure(figsize=(12,8))
ax = plt.axes(projection=ccrs.PlateCarree())
ax.set_global()
if (earthtopo): ax.coastlines(resolution="110m",linewidth=1)
gl = ax.gridlines(linestyle='--',color='black',
             draw_labels=True)
gl.xlabels_top = False
gl.ylabels_right = False
clevs = np.linspace(vmin,vmax,29)
plt.contourf(longitude, latitude, dataplt,
             clevs, transform=ccrs.PlateCarree(),cmap="jet")
plt.title(r"Moyenne dans un intervalle de temps de "+varname, size=14)
cb = plt.colorbar(ax=ax, orientation="vertical", pad=0.02, aspect=16, shrink=0.8)
cb.set_label(r'K',size=12,rotation=0,labelpad=15)
cb.ax.tick_params(labelsize=10)
plt.show()


I am working on the file "TOOLS/atlas.ipynb".
The first task is:
Carte en moyenne temporelle sur la totalité de l'expérience
Can you write Python code for it?

# Paramètres utilisateurs -----------------------------------------
earthtopo = False # ajouter les traits de côte actuels
varname = 'tsurf'
vmin = 180
vmax = 330

# Code ------------------------------------------------------------
dataplt = data.variables[varname][:,:,:]

fig = plt.figure(figsize=(12,8))
ax = plt.axes(projection=ccrs.PlateCarree())
ax.set_global()
if (earthtopo): ax.coastlines(resolution="110m",linewidth=1)
gl = ax.gridlines(linestyle='--',color='black',
             draw_labels=True)
gl.xlabels_top = False
gl.ylabels_right = False
clevs = np.linspace(vmin,vmax,29)
plt.contourf(longitude, latitude, np.mean(dataplt[:,:,:],axis=0),
             clevs, transform=ccrs.PlateCarree(),cmap="jet")
plt.title(r"Température de surface moyenne", size=14)
cb = plt.colorbar(ax=ax, orientation="vertical", pad=0.02, aspect=16, shrink=0.8)
cb.set_label(r'K',size=12,rotation=0,labelpad=15)
cb.ax.tick_params(labelsize=10)
plt.show()


I am working on the file "TOOLS/atlas.ipynb".
The first task is:
Coupe de la valeur moyenne d'un champ dans un intervalle de $L_s$
Can you write Python code for it?

# Paramètres utilisateurs -----------------------------------------
year_user = 1 # année de simulation à regarder
Ls_user_min = 0. # longitude solaire choisie
Ls_user_max = 360. # longitude solaire choisie

varname = 'temp'
vmin = 200
vmax = 350

# Code ------------------------------------------------------------
Ls_true_user_min = year_user*360. + Ls_user_min
Ls_true_user_max = year_user*360. + Ls_user_max

Lsmin_ind = np.where(abs(Ls_true-Ls_true_user_min)==
                  abs(Ls_true-Ls_true_user_min).min())[0]
print("La valeur la plus proche trouvée est Ls_min = "
      + str(Ls_true[Lsmin_ind]-year_user*360.)
      + " pour l'année " + str(year_user))

Lsmax_ind = np.where(abs(Ls_true-Ls_true_user_max)==
                  abs(Ls_true-Ls_true_user_max).min())[0]
print("La valeur la plus proche trouvée est Ls_max = "
      + str(Ls_true[Lsmax_ind]-year_user*360.)
      + " pour l'année " + str(year_user))

# Code ------------------------------------------------------------
var = data.variables[varname][:,:,:,:]
p = data.variables['p'][:,:,:,:]

fig, ax = plt.subplots(figsize=(14,6))
xplt=latitude
yplt = np.squeeze(np.mean(np.mean(p[Ls_ind,:,:,:],axis=3),axis=2))
dataplt = np.squeeze(np.mean(np.mean(var[Lsmin_ind[0]:Lsmax_ind[0],:,:,:],axis=0),axis=2))
im = ax.contourf(xplt,yplt,dataplt,60,cmap='jet',vmax=vmax, vmin=vmin)
ax.set_title(varname)
ax.set_yscale('log')
ax.set(xlabel='Latitude', ylabel='p')
ax.invert_yaxis()
fig.colorbar(im,ax=ax)
# -----------------------------------------------------------------


I am working on the file "TOOLS/atlas.ipynb".
The first task is:
Coupe zonale à une longitude solaire $L_s$ donnée
Can you write Python code for it?

# Paramètres utilisateurs -----------------------------------------
year_user = 1 # année de simulation à regarder
Ls_user = 90. # longitude solaire choisie

varname = 'temp'
vmin = 180.
vmax = 360.

# Code ------------------------------------------------------------
Ls_true_user = year_user*360. + Ls_user

Ls_ind = np.where(abs(Ls_true-Ls_true_user)==
                  abs(Ls_true-Ls_true_user).min())[0]
print("La valeur la plus proche trouvée est Ls = "
      + str(Ls_true[Ls_ind]-year_user*360.)
      + " pour l'année " + str(year_user))

# Code ------------------------------------------------------------
var = data.variables[varname][:,:,:,:]
p = data.variables['p'][:,:,:,:]

fig, ax = plt.subplots(figsize=(14,6))
xplt=latitude
yplt = np.squeeze(np.mean(np.mean(p[Ls_ind,:,:,:],axis=3),axis=2))
dataplt = np.squeeze(np.mean(var[Ls_ind,:,:,:],axis=3))
im = ax.contourf(xplt,yplt,dataplt,60,cmap='jet',vmax=vmax, vmin=vmin)
#axs[0, 0].cbar()
#ax1.set_ylim([-90,90])
ax.set_title(varname)
ax.set_yscale('log')
ax.set(xlabel='Latitude', ylabel='p')
ax.invert_yaxis()
fig.colorbar(im,ax=ax)
# -----------------------------------------------------------------


I am working on the file "TOOLS/atlas.ipynb".
The first task is:
Evolution en moyenne globale
Can you write Python code for it?

# Paramètres utilisateurs -----------------------------------------
varname = 'tsurf' # nom de la variable
# Code ------------------------------------------------------------

var = data.variables[varname][:,:,:]

fig, ax = plt.subplots(figsize=(12,8))
area = data.variables['aire'][:]
totarea = np.sum(np.sum(area[:,:],axis=0),axis=0)
varmeanarea = np.mean(np.mean(var[:,:,:]*area,axis=1),axis=1)/totarea*area.size
ts1, = ax.plot(Ls_true, varmeanarea)
ax.set_xlabel('Solar longitude Ls')
ax.set_ylabel(varname)
ax.set_title("Evolution moyenne sur l'ensemble du globe")
ax.grid()


I am working on the file "TOOLS/atlas.ipynb".
The first task is:
Moyenne zonale d'un champ à une altitude donnée
Can you write Python code for it?

# Paramètres utilisateurs -----------------------------------------
alt_user = 1 # altitude choisie en km
vmin = 180. # bornes de l'echelle de couleur, a ajuster pour
vmax = 360. # une meilleure visualisation
varname = 'temp'

# Code
# -----------------------------------------------------------------
altitude = data.variables['altitude'][:]
alt = data.variables['altitude'][:]
alt_ind = np.where(abs(alt-alt_user)==
                  abs(alt-alt_user).min())[0]
print("La valeur la plus proche trouvée est altitude = "
      + str(alt[alt_ind]))

var = data.variables[varname][:,:,:,:]
p = data.variables['p'][:,:,:,:]

fig, ax = plt.subplots(figsize=(14,6))
xplt=Ls_true
yplt = latitude
dataplt = np.squeeze(np.mean(var[:,alt_ind,:,:],axis=3))
im = ax.contourf(xplt,yplt,dataplt.transpose(),60,cmap='jet',
                 vmax=vmax, vmin=vmin)
ax.set_title("Moyenne zonale de "+varname+" (alt="+str(alt[alt_ind])+")")
ax.set(xlabel='Solar longitude (Ls)', ylabel='Latitude')
fig.colorbar(im,ax=ax)
# -----------------------------------------------------------------


I am working on the file "TOOLS/atlas.ipynb".
The first task is:
Série temporelle en un point
Can you write Python code for it?

# Paramètres utilisateurs -----------------------------------------
longi_user = 160. # longitude considérée
lati_user = 90. # latitude considérée
alt_user = 1. # l'altitude a laquelle on veut regarder la variable 4D

varname = 'temp'
vmin = 180. # bornes de l'echelle de couleur, a ajuster pour
vmax = 360. # une meilleure visualisation

# On regarde le champ de température 4D "temp"
# -----------------------------------------------------------------
alt = data.variables['altitude'][:]
alt_ind = np.where(abs(alt-alt_user)==
                  abs(alt-alt_user).min())[0]
print("La valeur la plus proche trouvée est altitude = "
      + str(alt[alt_ind]))
longi = data.variables['longitude'][:]
lon_ind = np.where(abs(longi-longi_user)==
                  abs(longi-longi_user).min())[0]
print("La valeur la plus proche trouvée est longitude = "
      + str(longi[lon_ind]))
lati = data.variables['latitude'][:]
lat_ind = np.where(abs(lati-lati_user)==
                  abs(lati-lati_user).min())[0]
print("La valeur la plus proche trouvée est latitude = "
      + str(lati[lat_ind]))
# -----------------------------------------------------------------
var = data.variables[varname][:,:,:,:]

fig, ax = plt.subplots(figsize=(12,8))
dataplt = var[:,alt_ind,lat_ind,lon_ind]
plt1, = ax.plot(Ls_true, dataplt)
ax.set_xlabel('Solar longitude Ls')
ax.set_ylabel('Température (K)')
ax.set_title("Evolution de "+varname)
ax.grid()


I am working on the file "TOOLS/atlas.ipynb".
The first task is:
| Variable | Notation formelle usuelle | Nom dans resultat.nc | Unité |
|:---------|:-----------------|:-----------------------|:------|
| Température de surface | $T_s(x,y,t)$ | tsurf | K |
| Pression de surface | $p_s(x,y,t)$ | ps | Pa |
| Pression atmosphérique | $p(x,y,z,t)$ | p | Pa |
| Température de l'atmosphère | $T(x,y,z,t)$ | temp | K |
| Température potentielle | $\theta(x,y,z,t)$ | teta | K |
| Vent zonal | $u(x,y,z,t)$ | u | m s$^{-1}$ |
| Vent méridien | $v(x,y,z,t)$ | v | m s$^{-1}$ |
| Vent vertical | $w(x,y,z,t)$ | w | m s$^{-1}$ |
| Albédo | $A_b(x,y,t)$ | ALB | SU |
| Flux solaire incident au sommet | $F_{vis}^{\downarrow_{TOA}}(x,y,t)$ | ISR | W m$^{-2}$ |
| Flux solaire absorbé par la surface | $F_{vis}^{\downarrow_{Surf}}(x,y,t)$ | ASR | W m$^{-2}$ |
| Flux infrarouge émis au sommet | $F_{IR}^{\uparrow_{TOA}}(x,y,t)$ | OLR | W m$^{-2}$ |
| Taux de chauffage SW | $Q_{rad}^{SW}(x,y,z,t)$ | zdtsw | K s$^{-1}$ |
| Taux de chauffage LW | $Q_{rad}^{LW}(x,y,z,t)$ | zdtlw | K s$^{-1}$ |
| Taux de chauffage total | $Q_{rad}(x,y,z,t)$ | dtrad | K s$^{-1}$ |
| RMM en vapeur d'eau | $q_{vap}(x,y,z,t)$ | h2o_vap | kg kg$^{-1}$ |
| RMM en glace d'eau | $q_{glace}(x,y,z,t)$ | h2o_ice | kg kg$^{-1}$ |
| Masse de glace au sol par m$^2$ | $m_{glace}(x,y,t)$ | h2o_ice_surf | kg m$^{-2}$ |
Quelques remarques :
+ SW signifie shortwave, donc correspond au domaine des courtes longueurs d'onde (appelé aussi domaine "visible");
+ De la même façon LW signifie longwave, et correspond au domaine des grandes longueurs d'onde ou domaine infrarouge;
+ RMM signifie rapport de mélange massique, en kg de l'espèce considérée par kg d'air;
+ Le vent $\vec{v}$ s'écrit suivant les trois directions de l'espace $\vec{v}=u\ \hat{x}+v\ \hat{y}+w\ \hat{z}$.
Can you write Python code for it?

longitude=data.variables['longitude'][:]
latitude=data.variables['latitude'][:]
altitude=data.variables['altitude'][:]
Time = data.variables['Time'][:]
Ls = data.variables['Ls'][:]


I am working on the file "TP.ipynb".
The first task is:
Analizando los  años en Capital Federal se puede ver que los precios de las propiedades empezaron a disminuir en el 2013. Seguido de una leve recuperación recuperación en el 2015 y un crecimiento exponencial para  el 2017.
Mostramos como varia el precio mes a mes de Capital Federal
Can you write Python code for it?

CF.plot(x='date', y=('price_aprox_usd', 'mean') , kind='line', title="CapFed Prices")


I am working on the file "TP.ipynb".
The first task is:
Como se puede ver en el grafico para distancias pequeñas se puede ver como varia demasiado el precio, y que a partir de los 100000 m aproximadamente el precio empieza a bajar linealmente.
Grafico de la variacion del precio en dolares en funcion de la cercania a un hospital
Se espera que la relación del promedio de los precios en dólares de cada propiedad en función de la cercanía a los hospitales influirá en la suba del mismo, es decir,  que al estar mas cerca de un hospital, mas caro será la propiedad.
Can you write Python code for it?

hospitals = pd.read_csv("./extra/hospitales.csv", low_memory=False, \
                        sep=';', error_bad_lines=False)
BsAs = distanceAnalysis(df, hospitals, 'LAT', 'LNG', False)

# Filtro Las distancias lejanas porque si no la escala
# no se ve favorecida para graficarlos puntos calculados
BsAsPlot = BsAs[BsAs[('distances', '')] < 10**7]
BsAsPlot = BsAsPlot[BsAsPlot[('price_aprox_usd', 'mean')] < 10**7]
BsAsPlot.plot(x=('distances', ''), y=('price_aprox_usd', 'mean') , kind='line', \
              title="price depending on distance from hospitals")


I am working on the file "TP.ipynb".
The first task is:
Como se puede ver en el mapa en los distritos en el norte de Capital Federal es donde el promedio de los precios es el mayor. Que coincide con la zona de Palermo.
Análisis de la variación de precios en función de la cercania a distritos escolares
En este análisis se tuvo en cuenta como se dstrinuyen los distritos escolares en el mapa de buenos aires. De esta manera se nota  que la división en el territorio que se ve en el mapa es distinta a la división real del mapa de los barrios del mismo. Se mezclaron los archivos de geoDataFrame de los distritos escolares y de las propiedades de tal manera se agrego a esta ultima el poligono que representa la división de distritos.
Can you write Python code for it?

schoolDistricts = pd.read_csv("./extra/distritos-escolares.csv", \
                              low_memory=False, sep=',', error_bad_lines=False)
schoolDistricts['WKT'] = pd.Series(convertStringToPolygon(schoolDistricts['WKT']\
                                                          .tolist()))
schoolDistrictsMap = convertToGeoDataFrame(schoolDistricts)

NeiighbourgoodPrices = densityMapByPlaceName(df, 'Capital Federal')
geoDf = convertToGeoData(NeiighbourgoodPrices)
geoDf = belongToPolygon(geoDf, schoolDistricts)
geoDf.plot(legend=True, column = 'price_aprox_usd', cmap='OrRd', \
           scheme="Quantiles", figsize=(10,10))
plt.title('Densidad de precios en CABA en funcion de la cercania a  \
    los distritos escolares', fontsize = 20)


I am working on the file "TP.ipynb".
The first task is:
Filtrado  de datos de los archivos CSV de cada mes

Aquí declaramos algunas funciones útiles que usaremos para poder realizar un chequeo de consistencia sobre el cálculo de los precios de las propiedades en venta.

Can you write Python code for it?

# Filtro de propiedades con precio calculable. Devuelve 1 si es válido.
# De lo contrario, nan (Not A Number)
def filter(array):
    priceUSD, usdM2, surfaceTotal = array
    if np.isnan(surfaceTotal) or surfaceTotal == 0:
        return np.nan
    if not np.isnan(priceUSD) and not np.isnan(usdM2):
        price = usdM2 * surfaceTotal
        dif = abs(price - priceUSD)
        if ((dif / priceUSD) * 100) <= 10:
            return 1
    return 1

# Cálculo del precio aproximado de venta
def fillPrice(array):
    priceUSD, usdM2, surfaceTotal = array
    if np.isnan(priceUSD) and not np.isnan(usdM2):
        return (usdM2 * surfaceTotal)
    return priceUSD

# Cálculo del precio del metro cuadrado
def fillM2(array):
    priceUSD, usdM2, surfaceTotal = array
    if not np.isnan(priceUSD) and np.isnan(usdM2):
        return (priceUSD / surfaceTotal)
    return usdM2


I am working on the file "TP.ipynb".
The first task is:
Filtrado y concatenacion de todos los archivos CSV en un solo archivo
En esta parte del código se filtraron columnas de los archivos csv que no ibamos a a usar. Tambiénse filtraron propiedades que no pertenezcan a Capital Federal o Gran Buenos Aires, ya que solo vamos a acotar nuestro análisis a estos dos lugares. Tambien fué necesario renombrar la columna de superficie ya que algunos archivos CSV tenian nombres distintos y es necesario que todos refieran al mismo.
Luego agregamos la columna llamada Date la cual refiere a la fecha en la cualfue publicada cada propiedad. Se uso esta fecha ya que la columna 'created_on' no refleja laevolucion del precio de las propiedades en funcion del tiempo y distorciona el analisis de datos. En cambio la fecha de publicacion muestra la actualizacion a la fecha de los precios de cada propiedad.
Tambien se filtraron porpiedades en función de la validez de sus datos. Es decir, que si el valor en dolares por metro cuadradro multiplicado por la superficie no se encontraba en un rango menor al 10% respecto del precio en dolares, se descartaban.
Finalmente concatenamos todos los archivos abiertos los cuales fueron almacenados en un vector de DataFrames. Se tiene que tener en cuenta que al realizar la concatenación se crearon cien columnas vacias con nombre "unnmaed: 0" a  "unnmaed: 99" por lo cual se tuvieron que eliminar del dataframe concatenado.
Luego grabamos el DataFrame concatenado en un archivo CSV para trabajar de ahora en mas con este ultimo
Can you write Python code for it?

"""# Ruta de la carpeta con los archivos de datos modificados
root = "./properties/"
properties = []
indexAcum = 0
for archive in listdir(root):
    if ".csv" not in archive:
        continue
    df = pd.read_csv(root + archive, low_memory = False)
    df = df.loc[df.place_with_parent_names.str.contains('Capital Federal') \
        | df.place_with_parent_names.str.contains('Bs.As. G.B.A.'), :]

    # Durante la carga de datos, se eliminan ciertas columnas que nos
    #resultan irrelevantes para el trabajo como urls.
    if 'properati_url' in df:
        df.drop('properati_url', axis = 1, inplace = True)
    if 'geonames_id' in df:
        df.drop('geonames_id', axis = 1, inplace = True)
    if 'description' in df:
        df.drop('description', axis = 1, inplace = True)
    if 'image_thumbnail' in df:
        df.drop('image_thumbnail', axis = 1, inplace = True)
    if 'operation' in df:
        df.drop('operation', axis = 1, inplace = True)
    if 'created_on' in df:
        df.drop('created_on', axis = 1, inplace = True)
    if 'lat-lon' in df:
        df.drop('lat-lon', axis = 1, inplace = True)
    if 'currency' in df:
        df.drop('currency', axis = 1, inplace = True)
    if 'title' in df:
        df.drop('title', axis = 1, inplace = True)
    if 'id' in df:
        df.drop('id', axis = 1, inplace = True)
    if 'price_aprox_local_currency' in df:
        df.drop('price_aprox_local_currency', axis = 1, inplace = True)
    if 'price_aprox_usd' in df and 'price' in df:
        df.drop('price', axis = 1, inplace = True)
    if 'extra' in df and 'price' in df:
        df.drop('extra', axis = 1, inplace = True)

    # En algunos casos, es necesario renombrar algunas columnas
    if 'price_aprox_usd' not in df:
        df.rename(columns = {'price': 'price_aprox_usd'}, inplace = True)
    if 'surface_total_in_m2' not in df:
        df.rename(columns = {'surface_in_m2': 'surface_total_in_m2'}, \
            inplace = True)

    # Aquí reconvertimos algunas columnas a punto flotante
    df.loc[:,...


I am working on the file "TP.ipynb".
The first task is:
Grafico de la variacion del precio en dolares en funcion de la cercania a una parada de subte
Se espera que la relación del promedio de los precios en dólares de cada propiedad en función de la cercanía a cierta estación de subte influirá en la suba del mismo, es decir, que al estar mas cerca de la parada de un subte,  mas caro será la propiedad.
Can you write Python code for it?

subways = pd.read_csv('./extra/estaciones-de-subte.csv', low_memory=False)
CF = df.loc[df.place_with_parent_names.str.contains('Capital Federal'), :]
CF = CF[CF['place_name'] != "Capital Federal"]
CF = distanceAnalysis(df, subways, 'Y', 'X', False)

CFPlot.plot(x=('distances', ''), y=('price_aprox_usd', 'mean') , kind='line', \
            title="Price depending on distance in meters from subway")


I am working on the file "TP.ipynb".
The first task is:
Hacemos un rápido chequeo de los archivos para saber si alguno se descargó mal
Can you write Python code for it?

"""
# Ruta de la carpeta con los archivos de datos originales
rutaCarpetaOriginales = "./properties/"
for archive in listdir(rutaCarpetaOriginales):
    if ".csv" in archive:
        nombreArchivo = rutaCarpetaOriginales + archive
        try:
            print nombreArchivo
            df = pd.read_csv(nombreArchivo, low_memory = False)
            print('Ok')
        except ValueError:
            print("Error, archivo corrupto. Descargar de nuevo")"""


I am working on the file "TP.ipynb".
The first task is:
La relación de los precios de las propiedades con la distancia a los hospitales, como se esperaba para distancias pequeñas tiende a ser mayor a pesar de tener altas y bajas.
Grafico de la variacion del precio en dolares en funcion de la cercania a una parada de subte
la primer funcion recibe un string y cambia la coma por un punto. Esto se debe a que el dataFrame delas paradasde colectivo tiene las coordenadas de latitud y longitud en formato string y ademas el nuemro tiene separada la parte decimal por una coma, por lo cual no se lo puede convertir a float. Es necesario cambiar esa coma por un punto.
La segunda funcion cambia la lista de string que tiene numeros decimales, los cuales separan la parte entera de la decimalpor una coma, por una lista de numeros de tipo float, logrando antes cambiar esa coma por un punto para poder hacer la conversion
Se espera que la relación del promedio de los precios en dólares de cada propiedad en función de la cercanía a la parada de los colectivos influirá en la suba del mismo, es decir, que al estar mas cerca de la parada de un colectivo, mas caro será la propiedad.
Can you write Python code for it?

def changeStringCommaForPoint(string):
    point = "."
    split = string.split(',')
    return point.join(split)
def changeStringListCommaForPoint(stringList):
    aux = []
    for string in stringList:
        aux.append(changeStringCommaForPoint(string))
    return aux

busStops = pd.read_csv("./extra/paradas-de-colectivo.csv", \
                       low_memory=False, sep=';', error_bad_lines=False)
busStops['X'] = pd.Series(changeStringListCommaForPoint(busStops['X'].tolist()))
busStops['Y'] = pd.Series(changeStringListCommaForPoint(busStops['Y'].tolist()))
busStops.loc[:, 'X'] = busStops.loc[:, ['X']].apply(lambda x: float(x), axis = 1)
busStops.loc[:, 'Y'] = busStops.loc[:, ['Y']].apply(lambda x: float(x), axis = 1)

CF = df.loc[df.place_with_parent_names.str.contains('Capital Federal'), :]
CF = distanceAnalysis(CF, busStops, 'Y', 'X', True)

CFPlot.plot(x=('distances', ''), y=('price_aprox_usd', 'mean') , kind='line', \
        title="price depending on distance in kilometers from Bus stops")


I am working on the file "TP.ipynb".
The first task is:
Lo mismo pasa para Buenos Aires  que experimenta una caída del promedio en 2014, pero logro recuperarse y crecer al poco tiempo.
Mostramos como varia el precio mes a mes de Gran Buenos Aires
Can you write Python code for it?

GBA.plot(x='date', y=('price_aprox_usd', 'mean') , kind='line', title="GBA Prices")


I am working on the file "TP.ipynb".
The first task is:
Lo que se puede ver es que tiene una tendencia creciente hasta llegar a 12 donde encuentra su máximo y luego empieza a disminuir el promedio.
Grafico de densidades de los barrios de capital federal en función de el precio promedio  en dolares
Estas tres funciones conviertes el dataFrame a geoDataFrame.
La primer funcion reemplaza las columnas lat y lon por una columna necesaria para el dataFrame la cual se llama
geometry. Esta contiene la latitud y longitud en forma de objeto Point que representa un punto o coordenada del mapa.
La segunda función recibe un dataFrame que representa un mapa, que puede ser la división por barrio de la capital Federal o la division por distritos escolares o por comuna, etc, y reemplaza la columna WKT que en general contiene una lista de poligonos, que representa un area, la columna geometry.
La tercer funcion convierte una lista de string de poligonos a el objetos de lista de objetos poligonos. Esto es necesario ya que la mayoria de los DataFrame que prepresentan algun tipo de mapa, vienen con este tipo de inconvenientes.
Can you write Python code for it?

def convertToGeoData(df):
    geometry = [Point(xy) for xy in zip(df.lon, df.lat)]
    df = df.drop(['lat', 'lon'], axis = 1)
    crs = {'init': 'epsg:4326'}
    geoDf = gpd.GeoDataFrame(df, crs = crs, geometry = geometry)
    return geoDf
#--------------------------------------------------------------------
def convertToGeoDataFrame(df):
    geometry = [Polygon(x) for x in df['WKT'].tolist()]
    crs = {'init': 'epsg:4326'}
    geoDf = gpd.GeoDataFrame(df, crs = crs, geometry = geometry)
    return geoDf
#--------------------------------------------------------------------
def convertStringToPolygon(listOfPolygons):
    import re
    finalList = []
    pat = re.compile(r'''(-*\d+\.\d+ -*\d+\.\d+);*''')
    for s in listOfPolygons:
        matches = pat.findall(s)
        if matches:
            finalList.append([tuple(map(float, m.split())) for m in matches])
    return finalList


I am working on the file "TP.ipynb".
The first task is:
Para la variación de precios promedio respecto de las distancia a las paradas de colectivos es muy similar a la que tiene con la distancia a hospitales y estaciones de subte.
Grafico de la variacion del precio en dolares en funcion de la cercania a las universidades
Para este analisis se espera ver que si una propiedad es cercana a ana Univesidad su precio tiende a crecer.
Can you write Python code for it?

universities = pd.read_csv("./extra/universidades.csv", low_memory=False, \
                           sep=';', error_bad_lines=False)
universities.loc[:, 'LNG'] = universities.loc[:, ['LNG']]\
    .apply(lambda x: float(x), axis = 1)
universities.loc[:, 'LAT'] = universities.loc[:, ['LAT']]\
    .apply(lambda x: float(x), axis = 1)
universities = universities[~np.isnan(universities['LNG']) | \
                                ~np.isnan(universities['LAT'])]
BsAs = distanceAnalysis(df, universities, 'LAT', 'LNG', True)

#BSASPlot = BSAS[BSAS[('distances', '')] < 200000]
BSASPlot.plot(x=('distances', ''), y=('price_aprox_usd', 'mean') , kind='line', \
              title="price depending on distance in kilometers from Universities")


I am working on the file "TP.ipynb".
The first task is:
Se abre el archivo csv concatenado
Can you write Python code for it?

#Abrimos el dataframe unificado
df = pd.read_csv("propertiesConCat.csv", low_memory = False)


I am working on the file "TP/Python2/TP1GenerationRecursive.ipynb".
The first task is:
Dans les cas précédent, le générateur s'arrête de lui même au bout d'un certain temps. Cependant, il est aussi possible d'écrire des générateurs infinis. Dans ce cas, la responsabilité de l'arrêt revient à la l'appelant.
Can you write Python code for it?

def powers2():
    v = 1
    while True:
        yield v
        v*=2

for v in powers2():
    print v
    if v > 1000000:
        break


I am working on the file "TP/Python2/TP1GenerationRecursive.ipynb".
The first task is:
En vous inspirant des fonctions précédentes (mais sans les utiliser) ou en reprenant la fonction du cours, implantez de façon récursive la fonction suivante qui engendre l'ensemble des mots binaires d'une taille donnée.
Can you write Python code for it?

def binary_word_generator(n):
    """
    Return a generator on binary words of size n in lexicographic order

    Input :
        - n, the length of the words
    """
    # write code here

list(binary_word_generator(3))

# tests
import types
assert(type(binary_word_generator(0)) == types.GeneratorType)
assert(list(binary_word_generator(0)) == [''])
assert(list(binary_word_generator(1)) == ['0', '1'])
assert(list(binary_word_generator(2)) == ['00', '01', '10', '11'])
assert(list(binary_word_generator(3)) == ['000', '001', '010', '011', '100', '101', '110', '111'])
assert(len(list(binary_word_generator(4))) == 16)
assert(len(list(binary_word_generator(7))) == 128)


I am working on the file "TP/Python2/TP1GenerationRecursive.ipynb".
The first task is:
Exercice : implanter la fonction suivante dont le but est de générer les n premiers nombre de Fibonacci La suite de fibonacci est définie par :
$f_0 = 0$
$f_1 = 1$
$f_n = f_{n-1} + f_{n-2}$ pour $n \geq 2$.
Can you write Python code for it?

def first_fibonacci_generator(n):
    """
    Return a generator for the first ``n`` Fibonacci numbers
    """
    # write code here


I am working on the file "TP/Python2/TP1GenerationRecursive.ipynb".
The first task is:
Exercice: Implantez les fonctions suivantes
Can you write Python code for it?

def fibonacci_generator():
    """
    Return an infinite generator for Fibonacci numbers
    """
    # write code here

it = fibonacci_generator()

it.next()

def fibonacci_finder(n):
    """
    Return the first Fibonacci number greater than or equal to n
    """
    # write code here

assert(fibonacci_finder(10) == 13)
assert(fibonacci_finder(100) == 144)
assert(fibonacci_finder(1000) == 1597)
assert(fibonacci_finder(1000000) == 1346269)


I am working on the file "TP/Python2/TP1GenerationRecursive.ipynb".
The first task is:
Exécutez la ligne suivante et copiez la liste des nombres obentus dans Google.
Can you write Python code for it?

[len(set(dyck_prefix_generator(2*n, n))) for n in xrange(8)]


I am working on the file "TP/Python2/TP1GenerationRecursive.ipynb".
The first task is:
Le but de la fonction stupid_generator est de lister les entiers inférieurs à end. Cependant, elle ne retourne pas directement la liste mais un générateur sur cette liste. Comparez avec la fonction suivante.
Can you write Python code for it?

def stupid_list(end):
    i = 0
    result = []
    while i < end:
        result.append(i)
        i+=1
    return result

stupid_list(3)


I am working on the file "TP/Python2/TP1GenerationRecursive.ipynb".
The first task is:
Les fonctions suivantes génèrent les mots binaires de taille 0,1, et 2.
Can you write Python code for it?

def binary_word_generator0():
    yield ""

def binary_word_generator1():
    yield "0"
    yield "1"

def binary_word_generator2():
    for b in binary_word_generator1():
        yield b + "0"
        yield b + "1"

list(binary_word_generator2())


I am working on the file "TP/Python2/TP1GenerationRecursive.ipynb".
The first task is:
Sur le même modèle, implanter la fonction suivante. (un peu plus dur)
Posez-vous la question de cette façon : si j'ai un mot de taille $n$ qui termine par 0 et qui contient $k$ fois 1, combien de 1 contenait le mot taille $n-1$ à partir duquel il a été créé ? De même s'il termine par 1.
Remarque : l'ordre des mots n'est plus imposé
Can you write Python code for it?

def binary_kword_generator(n,k):
    """
    Return a generator on binary words of size n such that each word contains exacty k occurences of 1

    Input :

    - n, the size of the words
    - k, the number of 1
    """
    # write code here

list(binary_kword_generator(4,2))

# tests
import types
assert(type(binary_kword_generator(0,0)) == types.GeneratorType)
assert(list(binary_kword_generator(0,0)) == [''])
assert(list(binary_kword_generator(0,1)) == [])
assert(list(binary_kword_generator(1,1)) == ['1'])
assert(list(binary_kword_generator(4,4)) == ['1111'])
assert(list(binary_kword_generator(4,0)) == ['0000'])
assert(set(binary_kword_generator(4,2)) == set(['0011', '0101', '1001', '0110', '1010', '1100']))
assert(len(list(binary_kword_generator(7,3))) == 35)


I am working on the file "TP/Python2/TP1GenerationRecursive.ipynb".
The first task is:
Votre fonction doit passer les tests suivants :
Can you write Python code for it?

import types
assert(type(first_fibonacci_generator(3)) == types.GeneratorType)
assert(list(first_fibonacci_generator(0)) == [])
assert(list(first_fibonacci_generator(1)) == [0])
assert(list(first_fibonacci_generator(2)) == [0,1])
assert(list(first_fibonacci_generator(8)) == [0,1,1,2,3,5,8,13])