I am working on the file "Assignments-colab/ASSIGNMENT_4a.ipynb".
The first task is:
1c. Find the longest status updates
Define a function called get_longest_update() that has the following parameters:
* status_updates (positional parameter)
* length_type (keyword parameter with default string "tokens").
The function should find the longest update. By default, the fuction should find the status update that is the longest in terms of number of tokens. Also implement the options to find the longst status update in terms of characters or sentences in the message. These options should be carried out when length_type is changed to "sentences" or "characters"
The function should return the status message (called 'status_message' in the data structure) of the longest update as a string.
Attention: It is recommended to use NLTK for this exercise.
Can you write Python code for it?

def get_longest_update():
    # your code here

# test your function here


I am working on the file "Assignments-colab/ASSIGNMENT_4a.ipynb".
The first task is:
2a. Read the JSON files
We have already stored the data as the JSON files laureate.json and prize.json in the folder ../Data/json_data/NobelPrize. Open these JSON files and load them as the Python dictionaries dict_laureates and dict_prizes.
Can you write Python code for it?

# load laureates.json and prize.json here


I am working on the file "Assignments-colab/ASSIGNMENT_4a.ipynb".
The first task is:
2c. Get all prizes from affiliations
Create a function called get_affiliation_prizes() that takes one input parameters:

dict_laureates (positional parameter)

The function should find all affiliations that were involved in winning the Nobel Prize and provide information on the category and year of those Nobel Prizes. It should return a nested dictionary of the following format:
{
    "A.F. Ioffe Physico-Technical Institute": [
        {"category": "physics", "year": "2000"}
    ],
    "Aarhus University": [
        {"category": "chemistry", "year": "1997"},
        {"category": "economics","year": "2010"}
    ]
}
Tip: some of the entries will lack information (for example, there is no associated affiliation). Use if-statements to check if essential information is present.
General tip for working with data: If your code breaks, check whether your assumptions about the data hold (very often, they unfortunatelydo not). For instance, a dictionary key you thought was always present is missing from a couple of dictionaries, etc.
Can you write Python code for it?

def get_affiliation_prizes():
    # your code here

# test your function here


I am working on the file "Assignments-colab/ASSIGNMENT_4a.ipynb".
The first task is:
Assignment 4a: Data structures (CSV/TSV and JSON)
Deadline for Assignment 4a+b: Friday, October 15, 2021 (14.30) via Canvas (Assignment 4)

Please name your notebook with the following naming convention:
  ASSIGNMENT_4a_FIRSTNAME_LASTNAME.ipynb
Please submit your complete assignment (4a + 4b) by compressing all your material into a single .zip file following this naming convention: ASSIGNMENT_4_FIRSTNAME_LASTNAME.zip.

Please note that there is a BA and an MA version of Assignment 4b
In case you are not sure about creating a zip file from a folder, please refer to this guide (or any other guide you find online).
If you have questions about this chapter, please contact us at cltl.python.course@gmail.com. Questions and answers will be collected in this Q&A document, so please check if your question has already been answered.
In this block, we covered the following chapters about data formats:

Chapter 16 - Data Formats I (CSV/TSV)
Chapter 17 - Data Formats II (JSON)
Chapter 18 - Data Formats III (XML) only for master-level course

In this assignment, you will also have to apply your knowledge about containers. If you get stuck, you are likely to find solutions in the chapters about containers (Block 2).
Tip:
It could happen that your code throws a unicode error when you're trying to open one of the files used in this assignment. If this is the case, you can probably solve if by specifying the encoding when reading in the file:
python
with open(your/file/path, 'r', encoding = 'utf-8') as infile:
    #your code
Exercise 1: Trump's Facebook Status Updates (CSV/TSV)
In the folder ../Data/csv_data there is a TSV file called trump_facebook.tsv that contains Facebook status updates posted by Donald Trump. It was downloaded from here. Follow the instructions below to read the file and find specific status updates.
1a. Write your own function for reading CSV
Write a function called read_csv() that has two parameters:

input_file (positional parameter) and
delimiter (keyword parameter with default string ",")....
Can you write Python code for it?

def read_csv(input_file, delimiter=","):
    # your code here

# test your function here
filename = "../Data/csv_data/trump_facebook.tsv"
status_updates = read_csv(filename, delimiter="\t")
status_updates[0:2]


I am working on the file "Assignments-colab/ASSIGNMENT_4a.ipynb".
The first task is:
Exercise 2: Nobel Prize Winners (JSON)
There is a lot of interesting data online. For example, the Nobel Prize Organisaton provides the Nobel Prize API that allows you to download information about the prizes, the laureates and the countries.
The information is formatted in JSON. Have a look at the following URLs:
- http://api.nobelprize.org/v1/prize.json
- http://api.nobelprize.org/v1/laureate.json
- http://api.nobelprize.org/v1/country.json
For this exercise, we will only look at the prizes and the laureates.
We can download the data using the requests module. How this works is shown below.
Can you write Python code for it?

import requests

# Download data on prizes
api_url = "http://api.nobelprize.org/v1/prize.json"
r = requests.get(api_url)
dict_prizes = r.json()
# uncomment the line below if you'd like to see what's inside dict_prizes
#dict_prizes

# Download data on laureates
api_url = "http://api.nobelprize.org/v1/laureate.json"
r = requests.get(api_url)
dict_laureates = r.json()
# uncomment the line below if you'd like to see what's inside dict_prizes
#dict_laureates


I am working on the file "Assignments-colab/ASSIGNMENT_4a.ipynb".
The first task is:
In case you didn't manage to create the read_csv() function, run the following code using the DictReader() method from the csv module to get the data in the right format for the following exercises:
Can you write Python code for it?

import csv

filename = "../Data/csv_data/trump_facebook.tsv"
with open(filename, "r") as infile:
    status_updates = []
    csv_reader = csv.DictReader(infile, delimiter='\t')
    for row in csv_reader:
        status_updates.append(row)
status_updates[0:2]


I am working on the file "Assignments-colab/ASSIGNMENT_4b_BA.ipynb".
The first task is:
Assignment 4b-BA: Sentiment analysis using VADER
Due: Friday October 15, 2021, before 14:30
Please note that this is Assignment 4 for the Bachelor version of the Python course: Introduction to Python for Humanities and Social Sciences (L_AABAALG075)

Please submit your assignment (notebooks of parts 4a + 4b + JSON file) as a single .zip file.

Please name your zip file with the following naming convention: ASSIGNMENT_4_FIRSTNAME_LASTNAME.zip

Please submit your assignment on Canvas: Assignments --> Assignment 4

If you have questions about this chapter, please contact us at cltl.python.course@gmail.com. Questions and answers will be collected in this Q&A document, so please check if your question has already been answered.
Credits
The notebooks in this block have been originally created by Marten Postma and Isa Maks. Adaptations were made by Filip Ilievski.
Part I: VADER assignments
Preparation (nothing to submit):
To be able to answer the VADER questions you need to know how the tool works.
* Read more about the VADER tool in this blog.
* VADER provides 4 scores (positive, negative, neutral, compound). Be sure to understand what they mean and how they are calculated.
* VADER uses rules to handle linguistic phenomena such as negation and intensification. Be sure to understand which rules are used, how they work, and why they are important.
* VADER makes use of a sentiment lexicon. Have a look at the lexicon. Be sure to understand which information can be found there (lemma?, wordform?, part-of-speech?, polarity value?, word meaning?) What do all scores mean? You can inspect the README of the VADER system for more information.
Exercise 1
Consider the following sentences and their output as given by VADER. Analyze sentences 1 to 7, and explain the outcome for each sentence. Take into account both the rules applied by VADER and the lexicon that is used. You will find that some of the results are reasonable, but others are not. Explain what is going wrong or not when correct and incorrect results are...
Can you write Python code for it?

import json

my_tweets = json.load(open('my_tweets.json'))

for id_, tweet_info in my_tweets.items():
    print(id_, tweet_info)
    break


I am working on the file "Assignments-colab/ASSIGNMENT_4b_BA.ipynb".
The first task is:
Exercise 3
In this exercise, we are going to run VADER on our own tweets and evaluate it against the sentiment labels that we manually annotated for each tweet. We are going to make use of the following two functions:
Can you write Python code for it?

def run_vader(nlp,
              textual_unit,
              lemmatize=False,
              parts_of_speech_to_consider=set(),
              verbose=0):
    """
    Run VADER on a sentence from spacy

    :param str textual unit: a textual unit, e.g., sentence, sentences (one string)
    (by looping over doc.sents)
    :param bool lemmatize: If True, provide lemmas to VADER instead of words
    :param set parts_of_speech_to_consider:
    -empty set -> all parts of speech are provided
    -non-empty set: only these parts of speech are considered
    :param int verbose: if set to 1, information is printed
    about input and output

    :rtype: dict
    :return: vader output dict
    """
    doc = nlp(textual_unit)

    input_to_vader = []

    for sent in doc.sents:
        for token in sent:

            if verbose >= 2:
                print(token, token.pos_)

            to_add = token.text

            if lemmatize:
                to_add = token.lemma_

                if to_add == '-PRON-':
                    to_add = token.text

            if parts_of_speech_to_consider:
                if token.pos_ in parts_of_speech_to_consider:
                    input_to_vader.append(to_add)
            else:
                input_to_vader.append(to_add)

    scores = vader_model.polarity_scores(' '.join(input_to_vader))

    if verbose >= 1:
        print()
        print('INPUT SENTENCE', sent)
        print('INPUT TO VADER', input_to_vader)
        print('VADER OUTPUT', scores)

    return scores

def vader_output_to_label(vader_output):
    """
    map vader output e.g.,
    {'neg': 0.0, 'neu': 0.0, 'pos': 1.0, 'compound': 0.4215}
    to one of the following values:
    a) positive float -> 'positive'
    b) 0.0 -> 'neutral'
    c) negative float -> 'negative'

    :param dict vader_output: output dict from vader

    :rtype: str
    :return: 'negative' | 'neutral' | 'positive'
    """
    compound = vader_output['compound']

    if compound < 0:
        return...


I am working on the file "Assignments-colab/ASSIGNMENT_4b_MA.ipynb".
The first task is:
1.b Parallel documents
Before we can build anything, we have to understand how the data are strucutred. We start by looking at a single document.
Parallel documents are stored in the same document directory (d+number). The filenames indicate the language (e.g. en = English). The data we're interested in are stored in .xml format. Run the cell below to inspect the filepaths of a single document. Feel free to modify the path to inspect other documents.
Can you write Python code for it?

import glob

test_part = 'p27'
test_document = 'd0852'

test_doc_path = f'{path_pmb}/{test_part}/{test_document}/'
test_doc_files = glob.glob(f'{test_doc_path}*.xml')

for f in test_doc_files:
    print(f)


I am working on the file "Assignments-colab/ASSIGNMENT_4b_MA.ipynb".
The first task is:
2.b Get token and pos from a token element
Write a function which fulfills the following requirements:

Positional parameter: token element
Output: token (string) and part of speech tag (string) of the token element

An example token element is shown below. (You can use it for testing.)
Can you write Python code for it?

test_token_str = """
 <tagtoken xml:id="i1002">
   <tags>
     <tag type="verbnet" n="0">[]</tag>
     <tag type="tok">'m</tag>
     <tag type="sym">be</tag>
     <tag type="lemma">be</tag>
     <tag type="from">1</tag>
     <tag type="to">3</tag>
     <tag type="pos">VBP</tag>
     <tag type="sem">NOW</tag>
     <tag type="wordnet">O</tag>
   </tags>
 </tagtoken>
"""

test_token = etree.fromstring(test_token_str)
print(test_token)

def get_token_pos(token_element):
    pass

# Test your function using the first token
token, pos = get_token_pos(test_token)
assert token == "'m" and pos == 'VBP', 'token and pos not correct'


I am working on the file "Assignments-colab/ASSIGNMENT_4b_MA.ipynb".
The first task is:
2.c Get document text
Define a function with the following requirements:

Positional parameter: filepath of a document in a particular language (i.e. full, relativ filepath)
Output: the text of the document as a string

Hint:
There are two options to get the document text of a file:

Option 1: Access the comment indicated by <!--  -->. Look at the file above to find the comment. You will see that it contains the entire text represented in the xml file. You can access it by iterating over the child-elements of the root. Try this out in the notebook before defining your function. You can get started using the code below.

Option 2: Use the tokens. You can collect all the tokens in a document using the functions we have defined above. Once you have all tokens, you can use a string method to join them with whitespaces between them.

Only implement one of these options.
Can you write Python code for it?

# Code snippet for option 1

# use the test document
test_doc_path_en = test_doc_path
# load
doc_tree = etree.parse(test_doc_path_en)
# get root
root = doc_tree.getroot()
# iterate over child-elements
for ch in root.getchildren():
    print('tag', ch.tag)
    print('text', ch.text)

def get_doc_text(path_to_doc):
    #Your code here
    pass

# Test you function
test_part = 'p27'
test_document = 'd0852'
language = 'en'
test_doc_path = f'{path_pmb}/{test_part}/{test_document}/{language}.drs.xml'

text = get_doc_text(test_doc_path)

assert text == "I 'm not tired at~all .", 'doc text not correct'


I am working on the file "Assignments-colab/ASSIGNMENT_4b_MA.ipynb".
The first task is:
2.d Sort documents on languages
To explore the coverage of the corpus, it is convenient to store the documents as sets mapped to their language. We can then use set methods (i.e. intersection) to check which documents exist in a given language pair.
Write a function which fulfills the following criteria:

mandatory positional argument: path to the corpus (e.g. '../../../Data/PMB/pmb-2.1.0/data/gold')
output: a dictionary of the following format:
            {
              'language1': {'path_to_doc1', 'path_to_doc2', ...},
              'language2': {'path_to_doc1', 'path_to_doc4', ...},
              'language3': {'path_to_doc2', 'path_to_doc3', ...},
              }

Hints:

filepaths are strings; you can manipulate them using string methods (e.g. split on '/' or '.').
The os mudule has a convenient way of extracting the filename from a long path (i.e. the last bit of the path): os.path.basename(your_path)
Feel free to use defaultdict (with a set as the default value) (from collections import defaultdict)
Can you write Python code for it?

# Example for filepath manipulation:
import os

my_path = '../../some/dir/containing/a/file/with/an/interesting/name.txt'
f_name = os.path.basename(my_path)
print(f_name)
remaining_path = my_path.rstrip(f_name)
print(remaining_path)
name, extension = f_name.split('.')
print(name, extension)

def sort_documents(path_pmb):
    pass

# Test you function:
language_doc_dict = sort_documents(path_pmb)

n_en = len(language_doc_dict['en'])
n_it = len(language_doc_dict['it'])
n_de = len(language_doc_dict['de'])
n_nl = len(language_doc_dict['nl'])

assert n_en == 4555 and n_it == 635 and n_de == 1175 and n_nl == 586, 'sorting not correct'


I am working on the file "Assignments-colab/ASSIGNMENT_4b_MA.ipynb".
The first task is:
3.b Printing statistics for language pairs
We also want to explore the coverage of parallel data for the language pairs in the corpus. To do this, use an additional loop to iterate over all possible language pairs in the parallal meaning bank and print the number of documents which exist for both languages.
Use the function below to generate the language pairs. Simply copy-paste it into the script called utils.py and import it into explore_pmb.py. Use the cell below to explore how it works.
The list of language pair should look similar to this (and contain all possible pairs):
pairs = [(‘nl’, ‘en’), (‘it’, ‘de’), (‘en’, ‘it’)]
Print the following for each language pair (use f-strings):
Coverage for parallel data in [language_1] and [language_2]: [number of documents]
Hints:

You can unpack tuples in a loop.
Use a set method to get the document paths covered by both languages. Then simply count the paths.
You do not have to match the file-contents. Instead, use the information provided in the filepaths (in a previous step, you have sorted your corpus files according to language).  The file paths in the sets (representing the documents) are supposed to consist of the base names only (i.e. no directory paths). You can use set operations to get the overlap between two languages.
Can you write Python code for it?

def get_pairs(language_list):
    """Given a list, return a list of tuples of all element pairs."""
    pairs = []
    for l1 in language_list:
        for l2 in language_list:
            if l1 != l2:
                if (l1, l2) not in pairs and (l2, l1) not in pairs:
                    pairs.append((l1, l2))
    return pairs

language_list = ['a', 'b', 'c']
pairs = get_pairs(language_list)
print(pairs)

# Here's a start (feel free to modify this)

for lang1, lang2 in pairs:
    docs_lang1 = language_doc_dict[lang1]
    docs_lang2 = language_doc_dict[lang2]
    # you code here


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_A.ipynb".
The first task is:
Define a function called extract_statistics that has the following parameters:
* nlp: the result of calling spacy.load('en_core_web_sm')
* txt_path: path to a txt file, e.g., '../Data/Dreams/vickie8.txt'
The function:
* loads the content of the file using the function load_text
* processes the content of the file using nlp(content) (see tip 1 of this exercise)
The function returns a dictionary with five keys:
* num_sents: the number of sentences in the document
* num_chars: the number of characters in the document
* num_tokens: the number of words in the document
* longest_sent: the longest sentence in the document
    * Please make a list with all the sentences and call the function return_the_longest to retrieve the longest sentence
* longest_word: the longest word in the document
    * Please make a list with all the words and call the function return_the_longest to retrieve the longest word
Test the function on one of the files from Vickie's dreams.
Can you write Python code for it?

def extract_statistics(nlp, txt_path):
    """
    given a txt_path
    -use the load_text function to load the text
    -process the text using spaCy

    :param nlp: loaded spaCy model (result of calling spacy.load('en_core_web_sm'))
    :param str txt_path: path to txt file

    :rtype: dict
    :return: a dictionary with the following keys:
    -"num_sents" : the number of sentences
    -"num_chars" : the number of characters
    -"num_tokens" : the number of words
    -"longest_sent" : the longest sentence
    -"longest_word" : the longest word
    """

stats = extract_statistics(nlp, txt_path=paths[0])
stats


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_A.ipynb".
The first task is:
Exercise 3: return the longest
Define a function called return_the_longest that has the following parameter:
* list_of_strings: a list of strings
The function:
* returns the string with the highest number of characters. If multiple strings have the same length, return one of them.
Can you write Python code for it?

def return_the_longest(list_of_strings):
    """
    given a list of strings, return the longest string
    if multiple strings have the same length, return one of them.

    :param str list_of_strings: a list of strings

    """


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_A.ipynb".
The first task is:
Please test you function by running the following cell:
Can you write Python code for it?

a_list_of_strings = ["this", "is", "a", "sentence"]
longest_string = return_the_longest(a_list_of_strings)

error_message = f'the longest string should be "sentence", you provided {longest_string}'
assert longest_string == 'sentence', error_message


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_A.ipynb".
The first task is:
Please test your function using the following function call
Can you write Python code for it?

paths = get_paths(input_folder='../Data/Dreams')
print(paths)


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_A.ipynb".
The first task is:
Resit Assignment part A
Deadline: Friday, November 13, 2020 before 17:00

Please name your files:
ASSIGNMENT-RESIT-A.ipynb
utils.py (from part B)
raw_text_to_coll.py (from part B)

Please name your zip file as follows: RESIT-ASSIGNMENT.zip and upload it via Canvas (Resit Assignment).
- Please submit your assignment on Canvas: Resit Assignment
- If you have questions about this topic, please contact cltl.python.course@gmail.com.
Questions and answers will be collected in this Q&A document,
so please check if your question has already been answered.
All of the covered chapters are important to this assignment. However, please pay special attention to:
- Chapter 10 - Dictionaries
- Chapter 11 - Functions and scope
* Chapter 14 - Reading and writing text files
* Chapter 15 - Off to analyzing text
- Chapter 17 - Data Formats II (JSON)
- Chapter 19 - More about Natural Language Processing Tools (spaCy)
In this assignment:
* we are going to process the texts in ../Data/Dreams/*txt
* for each file, we are going to determine:
    * the number of characters
    * the number of sentences
    * the number of words
    * the longest word
    * the longest sentence
Note
This notebook should be placed in the same folder as the other Assignments!
Loading spaCy
Please make sure that spaCy is installed on your computer
Can you write Python code for it?


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_A.ipynb".
The first task is:
tip 2: the number of characters is the length of the document
Can you write Python code for it?

num_chars = len(doc.text)
print(num_chars)


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_A.ipynb".
The first task is:
tip 3: loop through the sentences of a document
Can you write Python code for it?

for sent in doc.sents:
    sent = sent.text
    print(sent)


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_A.ipynb".
The first task is:
tip 4: loop through the words of a document
Can you write Python code for it?

for token in doc:
    word = token.text
    print(word)


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_B.ipynb".
The first task is:
3. Building python modules to process files in a directory
In this exercise, you will write two python modules:

utils.py
texts_to_conll.py

The module texts_to_conll.py should do the following:

process all text files in a specified directory (we will use '../Data/Dreams')
write conll files representing these texts to another directory

Step 1: Preparation:

Create the two python modules in the same directory as this notebook
copy your function text_to_conll to the python module texts_to_conll.py
Move the function load_text you have defined in part A to utils.py and import it in text_to_conll.py
Move the function get_paths you have defined in part A to utils.py and import in it text_to_conll.py

Step 2: convert all text files in ../Data/Dreams:
Use your functions to convert all files in  ../Data/Dreams/. Please fulfill the following criteria:

The new files should be placed in a directory placed in the current directory called dreams_conll/
Each file should be named as follows: [original name without extension].tsv (e.g.vicky1.tsv)
The files should contain an index column

Tips:

Use a loop to iterate over the files in ../Data/Dreams.
Use string methods and slicing to create the new filename from the original filename (e.g. split on '/' and/or '.', use indices to extract certain substrings, etc.)
Look at the resulting files to check if your code works.

Step 3: Test and submit
Please test your code carefully. Them submitt all your files in a .zip file via Canvas.
Congratulations! You have completed your first file conversion exercise!
Can you write Python code for it?

# Files in '../Data/Dreams':
%ls ../Data/Dreams/


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_B.ipynb".
The first task is:
Assignment Resit - Part B
Deadline: Friday, November 13, 2020 before 17:00
This part of the assignment should be submitted as a zip file containing two python modules:

utils.py
texts_to_coll.py
ASSIGNMENT-RESIT-A.ipynb (notebook containing part A)

Please name your zip file as follows: RESIT-ASSIGNMENT.zip and upload it via Canvas (Resit Assignment).
Please submit your assignment on Canvas: Resit Assignment
If you have questions about this topic

If you have questions about this topic, please contact cltl.python.course@gmail.com.

Questions and answers will be collected in this Q&A document, so please check if your question has already been answered.
All of the covered chapters are important to this assignment. However, please pay special attention to:

Chapter 14 - Reading and writing text files
Chapter 15 - Off to analyzing text
Chapter 16 - Data formats I (CSV and TSV)
Chapter 19 - More about Natural Language Processing Tools (spaCy)

In this assignment, we are going to write code which conversts raw text to a structured format frequently used in Natural Lanugage Processing. No matter what field you will end up working in, you will always have to be able to convert data from format A to format B. You have already gained some experience with such conversions in Block 4.
The CoNLL format
Before you use the output of a text analysis system, you usually want to store the output in a structured format. One way of doing this is to use naf - a format using xml. In this assignment, we are going to look at CoNLL, which is a table-based format (i.e. it is similar to csv/tsv).
The format we are converting to is called CoNLL. CoNLL is the name of a conference (Conference on Natural Language Learning). Every year, the conference hosts a 'competion'. In this competition, participants have to build systems for a certain Natural Language Processing problem (usually referred to as 'task'). To compare results, participants have to stick to the CoNLL format. The format has become a popular format for storing the output...
Can you write Python code for it?

# your function

# test your function
text = 'This is an example text. The text mentions a former president of the United States, Barack Obama.'
basename = 'test_text.tsv'
output_dir = 'test_dir'
text_to_conll_simple(text,
                         nlp,
                         output_dir,
                         basename,
                         start_with_index = False,
                        overwrite_existing_conll_file = True)


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_B.ipynb".
The first task is:
Tip 1: Tokens, POS tags, and lemmas
Experiment with a small example to get the tokens and pos tags. Please refer to the chapter on SpaCy for an example on how to process text with spacy.
Spacy has different pos tags. For this exercise, it does not matter which one you use. Hint: To get a string (rather than a number, use the SpaCy attributes ending with '_').
You can use the code below to experiment:
Can you write Python code for it?

test = 'This is a test.'

doc = nlp(test)

tok = doc[0]
tok.text


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_B.ipynb".
The first task is:
Tip 2: Entities
Entity types
Entities are things (usually people/places/organizations/etc) that exist in the real world. SpaCy can tag texts with entity types. If an expression refers to an entity in the world, it will receive a lable indicating the type (for example, Barack Obama will be tagged as 'PERSON'. Since the expression 'Barack Obama' consists of two tokens, each token will receive such a label. Use dir() on a token object to find out how to get this information. Hint: Everything about entities starts with ('ent_')
Position of the entity token
An expression referring to an entity can consist of multiple tokens. To indicate that multiple tokens are part of the same/of different expressions, we often use the IOB system. In this system, we indicate whether a token is outside an entity mention, inside an entity mention or at the beginning of an entity mention.  In practice, most tokens of a text will thus be tagged as 'O'. 'Barack' will be tagged as 'B' and 'Obama' as 'I' (see example above). SpaCy can do this type of labeling. Use dir() on a token object to find out how to get this information.
Can you write Python code for it?

test = 'This is a test.'

doc = nlp(test)

tok = doc[0]
tok.text
dir(tok)


I am working on the file "Assignments-colab/ASSIGNMENT_RESIT_B.ipynb".
The first task is:
Tip 3: Dealing with directories and files
Use os to check if files or directories exist. You can also use os to make a directory if it does not exist yet.

os.path.isdir(path_to_dir) returns a boolean value. If the directory exists, it returns True. Else it returns False. You can use this to check if a directory exists. If it does not, you can make it.

os.path.isfile(path_to_file) returns a boolean value. If the file exists, it returns True. Else it returns False.

os.mkdir(path_to_dir) makes a new directory. Try it out and create a directory called 'test_dir' in the current directory.
Can you write Python code for it?

# Check if file exists

import os

a_path_to_a_file = '../Data/books/Macbeth.txt'

if os.path.isfile(a_path_to_a_file):
    print('File exists:', a_path_to_a_file)
else:
    print('File not found:', a_path_to_a_file)

another_path_to_a_file = '../Data/books/KingLear.txt'

if os.path.isfile(another_path_to_a_file):
    print('File exists:', another_path_to_a_file)
else:
    print('File not found:', another_path_to_a_file)

# check if directory exists

a_path_to_a_dir = '../Data/books/'

if os.path.isdir(a_path_to_a_dir):
    print('Directory exists:', a_path_to_a_dir)
else:
    print('Directory not found:', a_path_to_a_dir)

another_path_to_a_dir = '../Data/films/'

if os.path.isdir(another_path_to_a_dir):
    print('Directory exists:', another_path_to_a_dir)
else:
    print('Directory not found:', another_path_to_a_dir)


I am working on the file "Assignments/.ipynb_checkpoints/CP-400 - Assignment 1-checkpoint.ipynb".
The first task is:
Question 6 - Python dictionaries
For each item below, enter the code to complete the operation:
Can you write Python code for it?

# a  Add an additional value [c:30] to the following dictionary
tens = {a:10, b:20}

# b  Print the value of 'b'
tens = {a:10, b:20, c:30}

# c Merge the following two dictionaries in a single new dictionary
t1 = {'a': 100, 'b': 200}
t2 = {'c': 300, 'd': 400}


I am working on the file "Assignments/.ipynb_checkpoints/CP-400 - Assignment 1-checkpoint.ipynb".
The first task is:
Question 7 - Python loops
For each item below, enter the code to complete the operation:
Can you write Python code for it?

# a  Using the range() function, write a short for loop to print numbers from 1 to 10:

# b  Write a short while loop to create a list of numbers from 1 to 10
a = []
# Loop
print(len(a))

# c  Write a short loop from 1 to 100. Print 'fizz' if the number if evenly divisible by 3, print 'buzz'
#    if the number is evenly divisible by 5, and print 'fizzbuzz' if number is divisible by both 3 and 5.


I am working on the file "Assignments/01 - 08-27-2016/Assignment 01 - Plates and Shells.ipynb".
The first task is:

Generalized Hooke's Law

1) εx = $\frac{1}{E}[$σx - ν(σy + σz)$]$

2) εy = $\frac{1}{E}[$σy - ν(σz + σx)$]$

3) εz = $\frac{1}{E}[$σz - ν(σx + σy)$]$

where,

  E - Modulus of elasticity

  ν - poisson's ratio

Applying the following conditions to the above equations:

εz = 0

σz = 0

we get,

1) εx = $\frac{1}{E}[$σx - νσy$]$ ==> σx - νσy = E εx

2) εy = $\frac{1}{E}[$σy - νσx$]$ ==> σy - νσx = E εy

In matrix form we have:

$[ 1$  $-v ]$ $[$ σx $]$ = $[$ E εx $]$

$[ -v$  $1 ]$ $[$ σy $]$ = $[$ E εy $]$

or M * X = V

where: M - matrix, X - vector and V - vector

Now solving for σx and σy by using python and sympy
Can you write Python code for it?

M = sp.Matrix([[1, -v],
               [-v, 1]])        # Create the M matrix

V = sp.Matrix([E*EPx, E*EPy])   # Create the V vector

# Get the inverse of M matrix
Minv = M**-1

# Multiply inverse of M by V vector to solve for vector X
X = Minv * V
sp.simplify(X)


I am working on the file "Assignments/ASSIGNMENT-1.ipynb".
The first task is:
1b. Book prices
Calculate book prices for the following scenarios:
Suppose the price of a book is 24.95 EUR, but if the book is bought by a bookstore, they get a 30 percent discount (as opposed to customers buying from an online stores). Shipping costs 3 EUR for the first copy and 75 cents for each additional copy. Shipping costs always apply (the books also have to be shipped to the bookstore).
Write a program that can calculate the total costs for any number of copies for both bookstores and other customers. Use variables with clear names for your calculations and print the result using a full sentence.
The program should use variables which indicate whether the customer is a bookstore or not and how many books are bought. You can simply assign values to the variables in you code or use the input function (both is accepted).
Tip
Start small and add things in steps. For instance, start by calculating the price minus the discount. Then add the additional steps. Also, it helps to a start by simply assuming a specific number of books (start with 1 and make sure it works with any other number). Do not forget to test your program!
Can you write Python code for it?

# complete the code below

n_books =
customer_is_bookstore =

# you bookprice calculations here


I am working on the file "Assignments/ASSIGNMENT-1.ipynb".
The first task is:
3b. Colors
We start again with some repetition of the theory:
| Topic | Explanation |
|-----------|--------|
| a = b + c | if b and c are strings: concatenate b and c to form a new string a|
| a = b * c | if b is an integer and c is a string: c is repeated b times to form a new string a |
| a[0] |     the first character of string a  |
| len(a) |   the number of characters in string a |
| min(a) |   the smallest element in string a (alphabetically first) |
| max(a) |   the largest element in string a (alphabetically last) |
Please run the code snippet below and observe what happens:
Can you write Python code for it?

b = 'the'
c = 'cat'
d = ' is on the mat'
a = b + ' ' + c + d
print(a)
a = b * 5
print(a)
print('The first character of', c, 'is' , c[0])
print('The word c has,', len(c) ,'characters')


I am working on the file "Assignments/ASSIGNMENT-1.ipynb".
The first task is:
4b. Printing the lexicon
Have a good look at the internal representation of the string below. Use a combination of string methods (you will need at least 3 different ones and some will have to be used multiple times) in the correct order to remove punctuation and redundant whitespaces, and print each word in lowercase characters on a new line. The result should look like:
the
quick
brown
fox
jumps
etc.
Can you write Python code for it?

text = """           The quick, brown fox jumps over a lazy dog.\tDJs flock by when MTV ax quiz prog.
Junk MTV quiz graced by fox whelps.\tBawds jog, flick quartz, vex nymphs.
Waltz, bad nymph, for quick jigs vex!\tFox nymphs grab quick-jived waltz.
Brick quiz whangs jumpy veldt fox.       """

print(text)
print()
print(repr(text))

text = # your code here

print(text)


I am working on the file "Assignments/ASSIGNMENT-1.ipynb".
The first task is:
4c. Passwords
Write a program that asks a user for a password and checks some simple requirements of a password. If necessary, print out the following warnings (use if-statements):

Your password should contain at least 6 characters.
Your password should contain no more than 12 characters.
Your password only contains alphabetic characters! Please also use digits and/or special characters.
Your password only contains digits! Please also use alphabetic and/or special characters.
Your password should contain at least one special character.
Your password contains only lowercase letters! Please also use uppercase letters.
Your password contains only uppercase letters! Please also use lowercase letters.
Can you write Python code for it?


I am working on the file "Assignments/ASSIGNMENT-1.ipynb".
The first task is:
5b. Alarm clock
Write code to set you alarm clock! Given the day of the week and information about whether you are currently on vacation or not, your code should print the time you want to be woken up following these constraints:
Weekdays, the alarm should be "7:00" and on the weekend it should be "10:00". Unless we are on vacation -- then on weekdays it should be "10:00" and weekends it should be "off".
Encode the weeks days as ints in the following way: 0=Sun, 1=Mon, 2=Tue, ...6=Sat. Encode the vacation infromation as boolean. Your code should assign the correct time to a variable as a string (following this format: "7:00") and print it.
Note: Encoding the days as an integer helps you with defining conditions. You can check whether the week day is in a certain interval (instead of writing code for every single day).
Can you write Python code for it?


I am working on the file "Assignments/ASSIGNMENT-1.ipynb".
The first task is:
Exercise 2: Printing and user input
2a. Difference between "," and "+"
What is the difference between using + and , in a print statement? Illustrate by using both in each of the following:

calling the print() fuction with multiple strings
printing combinations of strings and integers
concatenating multiple strings and assign to one single variable
concatenating strings and integers and assign to one single variable

2b. Small Talk
Write a program to have a little conversation with someone. First ask them for their name and their age, and then say something about your own age compared to theirs. Your code should result in a conversation following this example:

Hello there! What is your name?
-- Emily.
Nice to meet you, Emily. How old are you?
-- 23
I'm 25 years old, so I'm 2 years older than you.

Also account for situations where the other person is older or the same age. You will need to use if-else-statements!
Can you write Python code for it?

name = input("Hello there! What is your name? ")
# finish this code


I am working on the file "Assignments/ASSIGNMENT-1.ipynb".
The first task is:
Exercise 3: String Art
3a. Drawing figures
We start with some repetition of the theory about strings:
| Topic | Explanation |
|-----------|--------|
| quotes |    A string is delimited by single quotes ('...') or double quotes ("...") |
| special characters |    Certain special characters can be used, such as "\n" (for newline) and "\t" (for a tab)     |
| printing special characters |   To print the special characters, they must be preceded by a backslash (\)   |
| continue on next line | A backslash (\) at the end of a line is used to continue a string on the next line      |
| multi-line strings |    A multi-line print statement should be enclosed by three double or three single quotes ("""...""" of '''...''')  |
Please run the code snippet below and observe what happens:
Can you write Python code for it?

print('hello\n')
print('To print a newline use \\n')
print('She said: \'hello\'')
print('\tThis is indented')
print('This is a very, very, very, very, very, very \
long print statement')
print('''
This is a multi-line print statement
First line
Second line
''')


I am working on the file "Assignments/ASSIGNMENT-2.ipynb".
The first task is:
Exercise 4a: preprocessing text
Before analyzing the two texts, we are first going to preprocess them.
Please use a particular string method multiple times to replace the following characters by empty strings in both a_story and another_story:
* newlines: '\n'
* commas: ','
* dots: '.'
* quotes: '"'
Please assign the processed texts to the variables cleaned_story and cleaned_another_story.
Can you write Python code for it?


I am working on the file "Assignments/ASSIGNMENT-2.ipynb".
The first task is:
Exercise 4b: from text to a list
For each text (cleaned_story and cleaned_another_story), please use a string method to convert cleaned_story and cleaned_another_story into lists by splitting using spaces. Please call the lists list_cleaned_story and list_cleaned_another_story.
Can you write Python code for it?


I am working on the file "Assignments/ASSIGNMENT-2.ipynb".
The first task is:
Exercise 4c: from a list to a vocabulary (a set)
Please create a set for the words in each text by adding each word to a set. In the end, you should have two variables vocab_a_story and vocab_another_story, each containing the unique words in each story. Please use the output of Exercise 4b as the input for this exercise.
Can you write Python code for it?

vocab_a_story = set()
for word in list_cleaned_story:
    # insert your code here


I am working on the file "Assignments/ASSIGNMENT-2.ipynb".
The first task is:
Exercise 4d: analyzing vocabularies
Please analyze the vocabularies by using set methods to determine:
* which words occur in both texts
* which words only occur in a_story
* which words only occur in another_story
Can you write Python code for it?


I am working on the file "Assignments/ASSIGNMENT-2.ipynb".
The first task is:
Please store this function in a file called assignment2_utils.py. Please import the function and call it in this notebook.
Exercise 6e: including and excluding words
Please write Exercise 5a as a function.
The function:
* is called exclude_and_count
* has one positional parameter words, which is a list of strings.
* creates a dictionary in which the key is a word and the value is the frequency of that word.
* words are excluded if they meet one of the following criteria:
    * end with the letter e
    * start with the letter t
    * start with the letter c and end with the letter w (both conditions must be met)
    * have six or more letters
* returns a dictionary in which the key is the word and the value is the frequency of the word.
Can you write Python code for it?


I am working on the file "Assignments/ASSIGNMENT-2.ipynb".
The first task is:
b.) Analyze your dicitionary by printing:
* how many keys it has
* what the highest word frequency is
* the sum of all values.
c.) In addition, print the frequencies of the following words using your dictionary (if the word does not occur in the dictionary, print 'WORD does not occur')
* up
* near
* together
* lion
* cow
Can you write Python code for it?

for word in ['up', 'near' , 'together', 'lion', 'cow']:
    # print frequency


I am working on the file "Assignments/ASSIGNMENT-2.ipynb".
The first task is:
do the same for the other text
Can you write Python code for it?


I am working on the file "Assignments/ASSIGNMENT-3a.ipynb".
The first task is:
Dealing with text files
Exercise 4
Playing with lyrics
a.) Write a function called load_text, which opens and reads a file and returns the text in the file. It should have the file path as a parameter. Test it by loading this file: ../Data/lyrics/walrus.txt

Hint: remember it is best practice to use a context manager
Hint: FileNotFoundError: This means that the path you provide does not lead to an existing file on your computer. Please carefully study Chapter 14. Please determine where the notebook or Python module that you are working with is located on your computer. Try to determine where Python is looking if you provide a path such as “../Data/lyrics/walrus.txt”. Try to go from your notebook to the location on your computer where Python is trying to find the file. One tip: if you did not store the Assignments notebooks 3a and 3b in the folder “Assignments”, you would get this error.

b.) Write a function called replace_walrus, which takes lyrics as input and replaces every instance of 'walrus' by 'hippo' (make sure to account for upper and lower case - it is fine to transform everything to lower case). The function should write the new version of the song to a file called 'walrus_hippo.txt and stored in ../Data/lyrics.
Don't forget to add docstrings to your functions.
Can you write Python code for it?


I am working on the file "Assignments/ASSIGNMENT-3a.ipynb".
The first task is:
Working with external modules
Exercise 2
NLTK offers a way of using WordNet in Python. Do some research (using google, because quite frankly, that's what we do very often) and see if you can find out how to import it. WordNet is a computational lexicon which organizes words according to their senses (collected in synsets). See if you can print all the synset definitions of the lemma dog.
Make sure you have run the following cell to make sure you have installed WordNet:
Can you write Python code for it?

import nltk
# uncomment the following line to download material including WordNet
# nltk.download('book')

# your code here


I am working on the file "Assignments/ASSIGNMENT-3a.ipynb".
The first task is:
Working with python scripts
Exercise  3
a.) Define a function called count, which determines how often each word occurs in a string. Do not use NLTK just yet. Find a way to test it.

Write a helper-function called preprocess, which removes the punctuation specified by the user, and returns the same string without the unwanted characters. You call the function preprocess inside the count function.

Remember that there are string methods that you can use to get rid of unwanted characters. Test the preprocess function using the following string 'this is a (tricky) test'.

Remember how we used dictionaries to count words? If not, have a look at Chapter 10 - Dictionaries.

make sure you split the string on a space character ' '. You loop over the list to count the words.

Test your function using an example string, which will tell you whether it fulfills the requirements (remove punctuation, split, count). You will get a point for good testing.

b.) Create a python script
Use your editor to create a Python script called count_words.py. Place the function definition of the count function in count_words.py. Also put a function call of the count function in this file to test it. Place your helper function definition, i.e., preprocess, in a separate script called utils_3a.py. Import your helper function preprocess into count_words.py. Test whether everything works as expected by calling the script count_words.py from the terminal.
The function preprocess preprocesses the text by removing characters that are unwanted by the user. preprocess is called within the count function and hence builds upon the output from the preprocess function and creates a dictionary in which the key is a word and the value is the frequency of the word.
Please submit these scripts together with the other notebooks.
Don't forget to add docstrings to your functions.
Can you write Python code for it?

# Feel free to use this cell to try out your code.


I am working on the file "Assignments/ASSIGNMENT-3b.ipynb".
The first task is:
2.b) Store the function in the Python module utils.py. Import it in analyze.py.
Edit analyze.py so that:
* you first call the function get_paths
* create an empty dictionary called book2stats, i.e., book2stats = {}
* Loop over the list of txt files (the output from get_paths) and call the function get_basic_stats on each file
* print the output of calling the function get_basic_stats on each file.
* update the dictionary book2stats with each iteration of the for loop.
Tip: book2stats is a dictionary mapping a book name (the key), e.g., ‘AnnaKarenina’, to a dictionary (the value) (the output from get_basic_stats)
Tip: please use the following code snippet to obtain the basename name of a file path:
Can you write Python code for it?

import os
basename = os.path.basename('../Data/books/HuckFinn.txt')
book = basename.strip('.txt')
print(book)


I am working on the file "Assignments/ASSIGNMENT-3b.ipynb".
The first task is:
Encoding issues with txt files
For Windows users, the file “AnnaKarenina.txt” gets the encoding cp1252.
In order to open the file, you have to add encoding='utf-8', i.e.,
python
a_path = 'some path on your computer.txt'
with open(a_path, mode='r', encoding='utf-8'):
    # process file
Exercise 1
Was the download successful? Let's start writing code! Please create the following two Python modules:

Python module analyze.py This module you will call from the command line
Python module utils.py This module will contain your helper functions

More precisely, please create two files, analyze.py and utils.py, which are both placed in the same directory as this notebook. The two files are empty at this stage of the assignment.
1.a) Write a function called get_paths and store it in the Python module utils.py
The function get_paths:
* takes one positional parameter called input_folder
* the function stores all paths to .txt files in the input_folder in a list
* the function returns a list of strings, i.e., each string is a file path
Once you've created the function and stored it in utils.py:
* Import the function into analyze.py, using from utils import get_paths
* Call the function inside analyze.py (input_folder="../Data/books")
* Assign the output of the function to a variable and print this variable.
* call analyze.py from the command line to test it
Exercise 2
2.a) Let's get a little bit of an overview of what we can find in each text. Write a function called get_basic_stats.
The function get_basic_stats:
* has one positional parameter called txt_path which is the path to a txt file
* reads the content of the txt file into a string
* Computes the following statistics:
    * The number of sentences
    * The number of tokens
    * The size of the vocabulary used (i.e. unique tokens)
    * the number of chapters/acts:
        * count occurrences of 'CHAPTER' in HuckFinn.txt
        * count occurrences of 'Chapter ' (with the space) in AnnaKarenina.txt
        * count occurrences of 'ACT' in Macbeth.txt
* return a...
Can you write Python code for it?

from nltk.tokenize import sent_tokenize, word_tokenize

text = 'Python is a programming language. It was created by Guido van Rossum.'
for sent in sent_tokenize(text):
    print('SENTENCE', sent)
    tokens = word_tokenize(sent)
    print('TOKENS', tokens)


I am working on the file "Assignments/ASSIGNMENT-3b.ipynb".
The first task is:
Please note that Exercises 3 and 4, respectively, are designed to be difficult. You will have to combine what you have learnt so far to complete these exercises.
Exercise 3
Let's compare the books based on the statistics. Create a dictionary stats2book_with_highest_value in analyze.py with four keys:
* num_sents
* num_tokens
* vocab_size
* num_chapters_or_acts
The values are not the frequencies, but the book that has the highest value for the statistic. Make use of the book2stats dictionary to accomplish this.
Exercise 4
4.a) The statistics above already provide some insights, but we want to know a bit more about what the books are about. To do this, we want to get the 30 most frequent tokens of each book. Edit the function get_basic_stats to add one more key:value pair:
* the key is top_30_tokens
* the value is a list of the 30 most frequent words in the text.
4.b) Write the top 30 tokens (one on each line) for each file to disk using the naming top_30_[FILENAME]:
* top_30_AnnaKarenina.txt
* top_30_HuckFinn.txt
* top_30_Macbeth.txt
Example of file (the and and may not be the most frequent tokens, these are just examples):
the
and
..
The following code snippet can help you with obtaining the top 30 occurring tokens. The goal is to call the function you updated in Exercise 4a, i.e., get_basic_stats, in the file analyze.py. This also makes it possible to write the top 30 tokens to files.
Can you write Python code for it?

import operator

token2freq = {'a': 1000, 'the': 100, 'cow' : 5}
for token, freq in sorted(token2freq.items(),
                          key=operator.itemgetter(1),
                          reverse=True):
    print(token, freq)


I am working on the file "Assignments/ASSIGNMENT-4a.ipynb".
The first task is:
1c. Find the longest status updates
Define a function called get_longest_update() that has the following parameters:
* status_updates (positional parameter)
* length_type (keyword parameter with default string "tokens").
The function should find the longest update. By default, the fuction should find the status update that is the longest in terms of number of tokens. Also implement the options to find the longst status update in terms of characters or sentences in the message. These options should be carried out when length_type is changed to "sentences" or "characters"
The function should return the status message (called 'status_message' in the data structure) of the longest update as a string.
Attention: It is recommended to use NLTK for this exercise.
Can you write Python code for it?

def get_longest_update():
    # your code here

# test your function here


I am working on the file "Assignments/ASSIGNMENT-4a.ipynb".
The first task is:
1d. Find the status updates containing specific keywords
Define a function called get_updates_with_keywords() that takes three input arguments:

status_updates (mandatory positional argument)
keywords (mandatory positional argument)
case_sensitive (keyword argument with default False)

The fuction should find the status updates that contain any of the keywords. The parameter case_sensitive should specify whether uppercase and lowercase characters must be treated as distinct.
The function should return filtered_status_updates, which is a list of dictioaries with all information about the status updates (same format as the input argument 'status_updates').
Attention: It is highly recommended to use NLTK for this exercise. Make sure that you tokenize the messages before you look for keywords.
Can you write Python code for it?

def get_update_with_keywords():
    # your code here

keywords = ["clinton", "obama"] # test with these keywords; also experiment with other keywords
# test your function here


I am working on the file "Assignments/ASSIGNMENT-4a.ipynb".
The first task is:
2a. Read the JSON files
We have already stored the data as the JSON files laureate.json and prize.json in the folder ../Data/json_data/NobelPrize. Open these JSON files and load them as the Python dictionaries dict_laureates and dict_prizes.
Can you write Python code for it?

# load laureates.json and prize.json here


I am working on the file "Assignments/ASSIGNMENT-4a.ipynb".
The first task is:
2b. Get all laureates from year and category
Create a function called get_laureates() that thas three parameters:

dict_prizes (positional parameter)
year (keyword parameter with default None)
category (keyword parameter with default None)

The function should find all laureates that received the Nobel Prize, optionally in a specific year and/or category (specified using the keywords year and category). It should return a list of the full names of the laureates.
For example, for the year 2018 and category "peace" it should return the list ['Denis Mukwege', 'Nadia Murad'].
Can you write Python code for it?

def get_laureates():
    # your code here

year = 2018
category = "peace"
# test your function here


I am working on the file "Assignments/ASSIGNMENT-4a.ipynb".
The first task is:
2d. Write to JSON
Next, write the dictionary created in the previous exercise to a JSON file using the following path:
../Data/json_data/NobelPrize/nobel_prizes_affiliations.json.
Can you write Python code for it?

# write the resulting dictionary to 'json_file'


I am working on the file "Assignments/ASSIGNMENT-4a.ipynb".
The first task is:
Exercise 2: Nobel Prize Winners (JSON)
There is a lot of interesting data online. For example, the Nobel Prize Organisaton provides the Nobel Prize API that allows you to download information about the prizes, the laureates and the countries.
The information is formatted in JSON. Have a look at the following URLs:
- http://api.nobelprize.org/v1/prize.json
- http://api.nobelprize.org/v1/laureate.json
- http://api.nobelprize.org/v1/country.json
For this exercise, we will only look at the prizes and the laureates.
We can download the data using the requests module. How this works is shown below.
Can you write Python code for it?

import requests

# Download data on prizes
api_url = "http://api.nobelprize.org/v1/prize.json"
r = requests.get(api_url)
dict_prizes = r.json()
# uncomment the line below if you'd like to see what's inside dict_prizes
#dict_prizes

# Download data on laureates
api_url = "http://api.nobelprize.org/v1/laureate.json"
r = requests.get(api_url)
dict_laureates = r.json()
# uncomment the line below if you'd like to see what's inside dict_prizes
#dict_laureates


I am working on the file "Assignments/ASSIGNMENT-4a.ipynb".
The first task is:
In case you didn't manage to create the read_csv() function, run the following code using the DictReader() method from the csv module to get the data in the right format for the following exercises:
Can you write Python code for it?

import csv

filename = "../Data/csv_data/trump_facebook.tsv"
with open(filename, "r") as infile:
    status_updates = []
    csv_reader = csv.DictReader(infile, delimiter='\t')
    for row in csv_reader:
        status_updates.append(row)
status_updates[0:2]


I am working on the file "Assignments/ASSIGNMENT-4b-BA.ipynb".
The first task is:
Exercise 3a
You will now run VADER on the tweets you've collected. You will process each tweet using the code we have shown you above. The goal is add information about each tweet (i.e. in every iteration of the loop) to each of the three lists listed below. We can use these lists to compare the Vader output to the correct labels you provided.

tweets: append your tweet
all_vader_output: append the vader_label: negative | neutral | positive
manual_annotation: append your annotation: negative | neutral | positive

You can use the code snippet below as a starting point.
Can you write Python code for it?

import json

my_tweets = json.load(open('my_tweets.json'))

tweets = []
all_vader_output = []
manual_annotation = []

for id_, tweet_info in my_tweets.items():
    the_tweet = tweet_info['text_of_tweet']
    vader_output = ''# run vader
    vader_label = ''# convert vader output to category

    tweets.append(the_tweet)
    all_vader_output.append(vader_label)
    manual_annotation.append(tweet_info['sentiment_label'])


I am working on the file "Assignments/ASSIGNMENT-4b-MA.ipynb".
The first task is:
1.c XML structure of a single document
Below, we access a single document and load the xml structure using lxml.etree. Run the cell to print the xml tree.
Explore the document structure and try to answer these questions:

Where can you find the full text of the document?
Where can you find information about each token in the text?
Can you write Python code for it?

from lxml import etree

test_doc_path_en = test_doc_path+'en.drs.xml'
doc_tree = etree.parse(test_doc_path_en)
doc_root = doc_tree.getroot()
etree.dump(doc_root, pretty_print=True)


I am working on the file "Assignments/ASSIGNMENT-4b-MA.ipynb".
The first task is:
2.b Get token and pos from a token element
Write a function which fulfills the following requirements:

Positional parameter: token element
Output: token (string) and part of speech tag (string) of the token element

An example token element is shown below. (You can use it for testing.)
Can you write Python code for it?

test_token_str = """
 <tagtoken xml:id="i1002">
   <tags>
     <tag type="verbnet" n="0">[]</tag>
     <tag type="tok">'m</tag>
     <tag type="sym">be</tag>
     <tag type="lemma">be</tag>
     <tag type="from">1</tag>
     <tag type="to">3</tag>
     <tag type="pos">VBP</tag>
     <tag type="sem">NOW</tag>
     <tag type="wordnet">O</tag>
   </tags>
 </tagtoken>
"""

test_token = etree.fromstring(test_token_str)
print(test_token)

def get_token_pos(token_element):
    pass

# Test your function using the first token
token, pos = get_token_pos(test_token)
assert token == "'m" and pos == 'VBP', 'token and pos not correct'


I am working on the file "Assignments/ASSIGNMENT-4b-MA.ipynb".
The first task is:
2.c Get document text
Define a function with the following requirements:

Positional parameter: filepath of a document in a particular language (i.e. full, relativ filepath)
Output: the text of the document as a string

Hint:
There are two options to get the document text of a file:

Option 1: Access the comment indicated by <!--  -->. Look at the file above to find the comment. You will see that it contains the entire text represented in the xml file. You can access it by iterating over the child-elements of the root. Try this out in the notebook before defining your function. You can get started using the code below.

Option 2: Use the tokens. You can collect all the tokens in a document using the functions we have defined above. Once you have all tokens, you can use a string method to join them with whitespaces between them.

Only implement one of these options.
Can you write Python code for it?

# Code snippet for option 1

# use the test document
test_doc_path_en = test_doc_path
# load
doc_tree = etree.parse(test_doc_path_en)
# get root
root = doc_tree.getroot()
# iterate over child-elements
for ch in root.getchildren():
    print('tag', ch.tag)
    print('text', ch.text)

def get_doc_text(path_to_doc):
    #Your code here
    pass

# Test you function
test_part = 'p27'
test_document = 'd0852'
language = 'en'
test_doc_path = f'{path_pmb}/{test_part}/{test_document}/{language}.drs.xml'

text = get_doc_text(test_doc_path)

assert text == "I 'm not tired at~all .", 'doc text not correct'


I am working on the file "Assignments/ASSIGNMENT-4b-MA.ipynb".
The first task is:
2.d Sort documents on languages
To explore the coverage of the corpus, it is convenient to store the documents as sets mapped to their language. We can then use set methods (i.e. intersection) to check which documents exist in a given language pair.
Write a function which fulfills the following criteria:

mandatory positional argument: path to the corpus (e.g. '../../../Data/PMB/pmb-2.1.0/data/gold')
output: a dictionary of the following format:
            {
              'language1': {'path_to_doc1', 'path_to_doc2', ...},
              'language2': {'path_to_doc1', 'path_to_doc4', ...},
              'language3': {'path_to_doc2', 'path_to_doc3', ...},
              }

Hints:

filepaths are strings; you can manipulate them using string methods (e.g. split on '/' or '.').
The os mudule has a convenient way of extracting the filename from a long path (i.e. the last bit of the path): os.path.basename(your_path)
Feel free to use defaultdict (with a set as the default value) (from collections import defaultdict)
Can you write Python code for it?

# Example for filepath manipulation:
import os

my_path = '../../some/dir/containing/a/file/with/an/interesting/name.txt'
f_name = os.path.basename(my_path)
print(f_name)
remaining_path = my_path.rstrip(f_name)
print(remaining_path)
name, extension = f_name.split('.')
print(name, extension)

def sort_documents(path_pmb):
    pass

# Test you function:
language_doc_dict = sort_documents(path_pmb)

n_en = len(language_doc_dict['en'])
n_it = len(language_doc_dict['it'])
n_de = len(language_doc_dict['de'])
n_nl = len(language_doc_dict['nl'])

assert n_en == 4555 and n_it == 635 and n_de == 1175 and n_nl == 586, 'sorting not correct'


I am working on the file "Assignments/ASSIGNMENT-4b-MA.ipynb".
The first task is:
3. Putting the tool together
Congratulations! You've written most of the code already!
From now on, we will mostly use the functions defined above and combine them in the file called explore_pmb.py.
Some code snippets are provided here to help you along the way.
3.a Printing statistics for all languages
Let's start by exploring the coverage of all languages individually. In explore_pmb.py, write the following code:

Import the functionsort_docs, call it and assign the output dictionary to a variable called language_doc_dict. Don't forget to define the path to the corpus, which you use as function input.
Create a list of all languages (hint: you can simply use the keys of language_doc_dict).
For each language, print the following:
    `[Language]: num docs: [number of documents], num tokens: [number of tokens]

Hints:

Each document is unique - you can simply count the elements in the sets to get the number of documents.
Use the function get_tokens to access the tokens. Then count them.
You will most likly use two nested loops: An outer loop for languages and an inner loop to access the tokens in the documents.
Use f-strings to print the results.
Can you write Python code for it?

# some code to help you along the way (you can also start from scratch)
languages = # your code here

for languagage in languages:
    n_docs = # your code here
    n_tokens = # your code here
    docs = # your code here
    # your code here
    for doc in docs:
        path_to_doc = f'{doc}/{language}.drs.xml'
        tokens = get_tokens(path_to_doc)
        # your code here
    print(f'{language}: num docs: {n_docs}, num tokens: {n_tokens}')


I am working on the file "Assignments/ASSIGNMENT-4b-MA.ipynb".
The first task is:
3.b Printing statistics for language pairs
We also want to explore the coverage of parallel data for the language pairs in the corpus. To do this, use an additional loop to iterate over all possible language pairs in the parallal meaning bank and print the number of documents which exist for both languages.
Use the function below to generate the language pairs. Simply copy-paste it into the script called utils.py and import it into explore_pmb.py. Use the cell below to explore how it works.
The list of language pair should look similar to this (and contain all possible pairs):
pairs = [(‘nl’, ‘en’), (‘it’, ‘de’), (‘en’, ‘it’)]
Print the following for each language pair (use f-strings):
Coverage for parallel data in [language_1] and [language_2]: [number of documents]
Hints:

You can unpack tuples in a loop.
Use a set method to get the document paths covered by both languages. Then simply count the paths.
You do not have to match the file-contents. Instead, use the information provided in the filepaths (in a previous step, you have sorted your corpus files according to language).  The file paths in the sets (representing the documents) are supposed to consist of the base names only (i.e. no directory paths). You can use set operations to get the overlap between two languages.
Can you write Python code for it?

def get_pairs(language_list):
    """Given a list, return a list of tuples of all element pairs."""
    pairs = []
    for l1 in language_list:
        for l2 in language_list:
            if l1 != l2:
                if (l1, l2) not in pairs and (l2, l1) not in pairs:
                    pairs.append((l1, l2))
    return pairs

language_list = ['a', 'b', 'c']
pairs = get_pairs(language_list)
print(pairs)

# Here's a start (feel free to modify this)

for lang1, lang2 in pairs:
    docs_lang1 = language_doc_dict[lang1]
    docs_lang2 = language_doc_dict[lang2]
    # you code here


I am working on the file "Assignments/ASSIGNMENT-RESIT-A.ipynb".
The first task is:
Define a function called process_all_txt_files that has the following parameters:
* nlp: the result of calling spacy.load('en_core_web_sm')
* input_folder: a string (we will test it using '../Data/Dreams')
The function:
* obtains a list of txt paths using the function get_paths with input_folder as an argument
* loops through the txt paths one by one
* for each iteration, the extract_statistics function is called with txt_path as an argument
The function returns a dictionary:
* the keys are the basenames of the txt files (see tip 1 of this exercise)
* the values are the output of calling the function extract_statistics for a specific file
Test your function using '../Data/Dreams' as a value for the parameter input_folder.
Can you write Python code for it?

def process_all_txt_files(nlp, input_folder):
    """
    given a list of txt_paths
    -process each with the extract_statistics function

    :param nlp: loaded spaCy model (result of calling spacy.load('en_core_web_sm'))
    :param list txt_paths: list of paths to txt files

    :rtype: dict
    :return: dictionary mapping:
    -basename -> output of extract_statistics function
    """

basename_to_stats = process_all_txt_files(nlp, input_folder='../Data/Dreams')
basename_to_stats


I am working on the file "Assignments/ASSIGNMENT-RESIT-A.ipynb".
The first task is:
Exercise 2: load text
Define a function called load_text that has the following parameter:
* txt_path: a string
The function:
* opens the txt_path for reading and loads the contents of the file as a string
* returns a string, i.e., the content of the file
Can you write Python code for it?


I am working on the file "Assignments/ASSIGNMENT-RESIT-A.ipynb".
The first task is:
Exercise 4: extract statistics
We are going to use spaCy to extract statistics from Vickie's dreams! Here are a few tips below about how to use spaCy:
tip 1: process text with spaCy
Can you write Python code for it?

a_text = 'this is one sentence. this is another.'
doc = nlp(a_text)


I am working on the file "Assignments/ASSIGNMENT-RESIT-A.ipynb".
The first task is:
Exercise 6: write to disk
In this exercise, you are going to write our results to our computer.
Please loop through basename_to_stats and create one JSON file for each dream.

the path is f'{basename}.json', i.e., 'vickie1.json', 'vickie2.json', etc. (please write them to the same folder as this notebook)
the content of each JSON file is each value of basename_to_stats
Can you write Python code for it?

import json

for basename, stats in basename_to_stats.items():
    pass


I am working on the file "Assignments/ASSIGNMENT-RESIT-A.ipynb".
The first task is:
Please test you function by running the following cell:
Can you write Python code for it?

a_list_of_strings = ["this", "is", "a", "sentence"]
longest_string = return_the_longest(a_list_of_strings)

error_message = f'the longest string should be "sentence", you provided {longest_string}'
assert longest_string == 'sentence', error_message


I am working on the file "Assignments/ASSIGNMENT-RESIT-A.ipynb".
The first task is:
Please test your function using the following function call
Can you write Python code for it?

paths = get_paths(input_folder='../Data/Dreams')
print(paths)


I am working on the file "Assignments/ASSIGNMENT-RESIT-A.ipynb".
The first task is:
tip 2: the number of characters is the length of the document
Can you write Python code for it?

num_chars = len(doc.text)
print(num_chars)


I am working on the file "Assignments/ASSIGNMENT-RESIT-A.ipynb".
The first task is:
tip 4: loop through the words of a document
Can you write Python code for it?

for token in doc:
    word = token.text
    print(word)


I am working on the file "Assignments/ASSIGNMENT-RESIT-B.ipynb".
The first task is:
3. Building python modules to process files in a directory
In this exercise, you will write two python modules:

utils.py
texts_to_conll.py

The module texts_to_conll.py should do the following:

process all text files in a specified directory (we will use '../Data/Dreams')
write conll files representing these texts to another directory

Step 1: Preparation:

Create the two python modules in the same directory as this notebook
copy your function text_to_conll to the python module texts_to_conll.py
Move the function load_text you have defined in part A to utils.py and import it in text_to_conll.py
Move the function get_paths you have defined in part A to utils.py and import in it text_to_conll.py

Step 2: convert all text files in ../Data/dreams:
Use your functions to convert all files in  ../Data/dreams/. Please fulfill the following criteria:

The new files should be placed in a directory placed in the current directory called dreams_conll/
Each file should be named as follows: [original name without extension].tsv (e.g.vicky1.tsv)
The files should contain an index column

Tips:

Use a loop to iterate over the files in ../Data/dreams.
Use string methods and slicing to create the new filename from the original filename (e.g. split on '/' and/or '.', use indices to extract certain substrings, etc.)
Look at the resulting files to check if your code works.

Step 3: Test and submit
Please test your code carefully. Them submitt all your files in a .zip file via Canvas.
Congratulations! You have completed your first file conversion exercise!
Can you write Python code for it?

# Files in '../Data/Dreams':
%ls ../Data/Dreams/


I am working on the file "Assignments/ASSIGNMENT-RESIT-B.ipynb".
The first task is:
Tip 0: Import spacy and load your model
(See part A and chapter on SpaCy for more information)
Can you write Python code for it?

import spacy
nlp = spacy.load('en_core_web_sm')


I am working on the file "Assignments/ASSIGNMENT-RESIT-B.ipynb".
The first task is:
Tip 1: Tokens, POS tags, and lemmas
Experiment with a small example to get the tokens and pos tags. Please refer to the chapter on SpaCy for an example on how to process text with spacy.
Spacy has different pos tags. For this exercise, it does not matter which one you use. Hint: To get a string (rather than a number, use the SpaCy attributes ending with '_').
You can use the code below to experiment:
Can you write Python code for it?

test = 'This is a test.'

doc = nlp(test)

tok = doc[0]
tok.text


I am working on the file "Assignments/ASSIGNMENT-RESIT-B.ipynb".
The first task is:
Tip 2: Entities
Entity types
Entities are things (usually people/places/organizations/etc) that exist in the real world. SpaCy can tag texts with entity types. If an expression refers to an entity in the world, it will receive a lable indicating the type (for example, Barack Obama will be tagged as 'PERSON'. Since the expression 'Barack Obama' consists of two tokens, each token will receive such a label. Use dir() on a token object to find out how to get this information. Hint: Everything about entities starts with ('ent_')
Position of the entity token
An expression referring to an entity can consist of multiple tokens. To indicate that multiple tokens are part of the same/of different expressions, we often use the IOB system. In this system, we indicate whether a token is outside an entity mention, inside an entity mention or at the beginning of an entity mention.  In practice, most tokens of a text will thus be tagged as 'O'. 'Barack' will be tagged as 'B' and 'Obama' as 'I' (see example above). SpaCy can do this type of labeling. Use dir() on a token object to find out how to get this information.
Can you write Python code for it?

test = 'This is a test.'

doc = nlp(test)

tok = doc[0]
tok.text
dir(tok)


I am working on the file "Assignments/ASSIGNMENT-RESIT-B.ipynb".
The first task is:
Tip 3: Dealing with directories and files
Use os to check if files or directories exist. You can also use os to make a directory if it does not exist yet.

os.path.isdir(path_to_dir) returns a boolean value. If the directory exists, it returns True. Else it returns False. You can use this to check if a directory exists. If it does not, you can make it.

os.path.isfile(path_to_file) returns a boolean value. If the file exists, it returns True. Else it returns False.

os.mkdir(path_to_dir) makes a new directory. Try it out and create a directory called 'test_dir' in the current directory.
Can you write Python code for it?

# Check if file exists

import os

a_path_to_a_file = '../Data/books/Macbeth.txt'

if os.path.isfile(a_path_to_a_file):
    print('File exists:', a_path_to_a_file)
else:
    print('File not found:', a_path_to_a_file)

another_path_to_a_file = '../Data/books/KingLear.txt'

if os.path.isfile(another_path_to_a_file):
    print('File exists:', another_path_to_a_file)
else:
    print('File not found:', another_path_to_a_file)

# check if directory exists

a_path_to_a_dir = '../Data/books/'

if os.path.isdir(a_path_to_a_dir):
    print('Directory exists:', a_path_to_a_dir)
else:
    print('Directory not found:', a_path_to_a_dir)

another_path_to_a_dir = '../Data/films/'

if os.path.isdir(another_path_to_a_dir):
    print('Directory exists:', another_path_to_a_dir)
else:
    print('Directory not found:', another_path_to_a_dir)


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
An alternative: use weights already provided in the dataset
You will find that many gridded datasets already provide a field that gives accurate area weighting.
In the case of the CESM output, the field is called gw
Can you write Python code for it?

weight_factor2 = atm_control.gw / atm_control.gw.mean(dim='lat')

#  Compare our two weights
print( (atm_control.FLNT * weight_factor).mean(dim=('time', 'lon', 'lat')))
print( (atm_control.FLNT * weight_factor2).mean(dim=('time', 'lon', 'lat')))


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
An important gotcha with xarray
Normally we can access a variable with the notation Dataset.variable_name
But in xarray (and also in numpy and other packages), the notation object.T actually represents the transpose operator:
Can you write Python code for it?

print( atm_control.FLNT.shape)
print( atm_control.FLNT.T.shape)


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
And here is some code to plot the average sounding (temperature as function of pressure) at a particular point in the month of January.
Can you write Python code for it?

plt.plot( atm_control.Ta[0,:,70,115], atm_control.lev )
plt.gca().invert_yaxis()
plt.ylabel('Pressure (hPa)')
plt.xlabel('Temperature (K)')


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
Another solution is just to rename the variable (here going from T to Ta):
Can you write Python code for it?

atm_control = atm_control.rename({'T': 'Ta'})
atm_control.Ta


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
HOMEWORK QUESTION 6:
Compute annual, global averages of the following four quantities.

Incoming solar radiation (or insolation)
Absorbed solar radiation
Planetary albedo
Outgoing longwave radiation

Compare your results briefly to the observations.

About all-sky versus clear-sky radiative fluxes
The GCM computes cloud cover prognostically as part of its simulation of current weather conditions. These clouds affect both longwave and shortwave radiative fluxes in important ways (as we will study later).
At each timestep, the CESM (like just about every other GCM) actually computes two kinds of radiative fluxes:

the actual flux, including the effects of current cloud cover
a clear-sky flux -- the flux that would occur if there were no clouds (or the flux that you would measure if you looked only at the clear-sky parts of the scene)

For example, there are two outgoing longwave radiation fields:
Can you write Python code for it?

for var in ['FLNT', 'FLNTC']:
    print( var)
    print( atm_control[var].long_name)
    print( atm_control[var].units)
    print( '')


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
If we want more control over the appearance of the plot, we can use features of matplotlib
Can you write Python code for it?

#  A filled contour plot of topography with contours every 500 m
lev = np.arange(0., 6., 0.5)
fig1, ax1 = plt.subplots(figsize=(8,4))
# Here we are masking the data to exclude points where the land fraction is zero (water only)
cax1 = ax1.contourf( height.lon, height.lat,
                    height.where(topo.LANDFRAC>0), levels=lev)
ax1.set_title('Topography (km) and land-sea mask in CESM')
ax1.set_xlabel('Longitude')
ax1.set_ylabel('Latitude')
cbar1 = fig1.colorbar(cax1)


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
Lots of different stuff! These are all the different quantities that are calculated as part of the model simulation. Every quantity represents a long-term average for a particular month.
Want to get more information about a particular variable?
Can you write Python code for it?


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
Making nicer maps
Notice that to make these plots we've just plotted the lat-lon array without using any map projection.
There are nice tools available to make better maps. We'll leave that as a topic for another day. But if you're keen to read ahead, check out:
http://matplotlib.org/basemap/
and
http://scitools.org.uk/cartopy/
(We will probably be using cartopy later in this course because it is
Ocean boundary conditions
Another important input file contains information about the slab ocean. You can see this file in the data catalog here:
http://ramadda.atmos.albany.edu:8080/repository/entry/show/Top/Users/BrianRose/CESM_runs/som_input/pop_frc.1x1d.090130.nc
Let's load it and take a look.
Can you write Python code for it?

som_input = xr.open_dataset( datapath + 'som_input/pop_frc.1x1d.090130.nc' + endstr, decode_times=False )
print( som_input)


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
Note that at 2º resolution we can see many smaller features (e.g. Pacific islands). The model is given a fractional land cover for each grid point.
Here let's plot the land-sea mask itself so we can see where there is at least "some" water:
Can you write Python code for it?

fig2, ax2 = plt.subplots()
cax2 = ax2.pcolormesh( topo.lon, topo.lat, topo.LANDFRAC )
ax2.set_title('Ocean mask in CESM')
ax2.set_xlabel('Longitude'); ax2.set_ylabel('Latitude')
cbar2 = fig2.colorbar(cax2);


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
Notice all the spatial structure here:

Lots of heat is going in to the oceans at the equator, particularly in the eastern Pacific Ocean.
The red hot spots show where lots of heat is coming out of the ocean.
Hot spots include the mid-latitudes off the eastern coasts of Asia and North America
And also the northern North Atlantic.

All this structure is determined by ocean circulation, which we are not modeling here. Instead, we are prescribing these heat flux patterns as an input to the atmosphere.
This pattern changes throughout the year. Recall that we just averaged over all months to make this plot. We might want to look at just one month:
Can you write Python code for it?

# select by month index (0 through 11)
som_input.qdp.isel(time=0)

#  select by array slicing (but for this you have to know the axis order!)
som_input.qdp[0,:,:]


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
Now make a nice plot of the annual mean q-flux.
Can you write Python code for it?

#  We can always set a non-standard size for our figure window
fig3, ax3 = plt.subplots(figsize=(10, 6))
lev = np.arange(-700., 750., 50.)
cax3 = ax3.contourf(som_input.xc, som_input.yc,
                    -som_input.qdp.mean(dim='time'),
                    levels=lev, cmap=plt.cm.bwr)
cbar3 = fig3.colorbar(cax3)
ax3.set_title( 'CESM: Prescribed heat flux out of ocean (W m$^{-2}$), annual mean',
              fontsize=14 )
ax3.set_xlabel('Longitude', fontsize=14)
ax3.set_ylabel('Latitude', fontsize=14)
ax3.text(65, 50, 'Annual', fontsize=16 )
ax3.contour(topo.lon, topo.lat, topo.LANDFRAC, levels=[0.5], colors='k');


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
Plotting the topography
We will now read the geopotential and make a plot of the topography of the Earth's surface as represented on the 2º grid. The code below makes a colorful plot of the topography. We also use the land-sea mask in order to plot nothing at grid points that are entirely ocean-covered.
Execute this code exactly as written first, and then play around with it to see how you might customize the graph.
Note that the function pcolormesh does most of the work here. It's a function that makes color 2D plots of an array.
Can you write Python code for it?

g = 9.8  # gravity in m/s2
meters_per_kilometer = 1E3
height = topo.PHIS / g / meters_per_kilometer  # in kilometers
#  Note that we have just created a new xarray.DataArray object that preserves the axis labels
#  Let's go ahead and give it some useful metadata:
height.attrs['units'] = 'km'
height.name = 'height'
height

height.name


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
The "pre-industrial" control run
Our control run is set up to simulate the climate of the "pre-industrial era", meaning before significant human-induced changes to the composition of the atmosphere, nominally the year 1850.
Output from the control run is available on the same data server as above. Look in the folder called som_1850_f19 (Here som stands for "slab ocean model", 1850 indicated pre-industrial conditions, and f19 is a code for the horizontal grid resolution).
There are climatology files for each active model component:

atmosphere,
sea ice
land surface

I created these files by averaging over the last 10 years of the simulation. Let's take a look at the atmosphere file. The file is called
som_1850_f19.cam.h0.clim.nc
(the file extension .nc is used to indicate NetCDF format).
Can you write Python code for it?

atm_control = xr.open_dataset( datapath + 'som_1850_f19/som_1850_f19.cam.h0.clim.nc' + endstr )

print( atm_control)


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
The Dataset object has several important attributes. Much of this should look familiar if you have worked with netCDF data before. The xarray package gives a very powerful and easy to use interface to the data.
We can access individual variables within the xarray.Dataset object as follows:
Can you write Python code for it?


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
The ocean / sea ice models exist on different grids than the atmosphere (1º instead of 2º resolution).
Now we are going to look at the annual mean heat flux out of the ocean, which is the prescribed 'q-flux' that we give to the slab ocean model.
It is stored in the field qdp in the input file.
The sign convention in CESM is that qdp > 0 where heat is going IN to the ocean. We will change the sign to plot heat going OUT of the ocean INTO the atmosphere (a more atmosphere-centric viewpoint).
Can you write Python code for it?


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
This field then represents the effects of clouds on the OLR.
We can of course do the same thing with the shortwave fluxes.
A few more tidbits
That's it for homework questions. But feel free to keep exploring the data.
Many other fields are four-dimensional (time, level, latitude, longitude).
For example, here is the shape of the array that hold the air temperature at every point and every month:
Can you write Python code for it?


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
This is the (prescribed) amount of CO2 in the atmosphere (about 285 parts per million by volume).
One nice thing about xarray.DataArray objects is that we can do simple arithmetic with them (already seen several examples of this in the notes above). For example, change the units of CO2 amount to ppm:
Can you write Python code for it?


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
Very close, the difference is insignificant.

HOMEWORK QUESTION 5:
Make sure you can take a global average, testing on surface temperature in the control run.
Surface temperature is called 'TS' in the control run data file.
Calculate annual, global average 'TS'
Verify that you get something close to 289.57
If it doesn't, try to find and fix the errors.

Finding the radiative fluxes in the model output
Now that you have a working function to take global averages, we can compare some energy budget values against observations.
The model output contains lots of diagnostics about the radiative fluxes. Here some CESM naming conventions to help you find the appropriate output fields:

All variables whose names being with 'F' are an energy flux of some kind.
Most have a four-letter code, e.g. 'FLNT'
'FL' means longwave flux (i.e. terrestrial)
'FS' means shortwave flux (i.e. solar)
The third letter indicates direction of the flux:
'U' = up
'D' = down
'N' = net

The fourth letter indicates the location of the flux:
'T' = top of atmosphere
'S' = surface

So 'FLNT' means 'net longwave flux at the top of atmosphere', i.e. the outgoing longwave radiation.

You wil see that these are all 12 x 96 x 144 -- i.e. a two-dimensional grid for every calendar month.
Can you write Python code for it?


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
When you execute the next cell, you should get a figure with a slider above it. Go ahead and play with it.
Can you write Python code for it?

#  This requires IPython 4 and above
from ipywidgets import interact
interact(sh, month=(0,11,1));


I am working on the file "Assignments/Assignment02 -- Introducing CESM.ipynb".
The first task is:
You are encouraged to experiment and tinker with all the code below.
We are going to use a package called xarray (abbreviated here as xr) to work with the datasets.
Boundary conditions: continents and topography
Here we are going to load the input topography file and take a look at what's inside.
We use the Dataset object from the netCDF4 module as our basic container for any netCDF data. Dataset() requires at least one argument telling it what file to open. This can either be a file on your local disk or a URL.
In this case we are passing it a URL to our online dataserver. We'll put the URL in a string variable called datapath to simplify things later on.
Can you write Python code for it?

datapath = "http://ramadda.atmos.albany.edu:8080/repository/opendap/latest/Top/Users/BrianRose/CESM_runs/"
endstr = "/entry.das"

#  Notice that in Python we can easily concatenate strings together just by `adding` them
fullURL = datapath + 'som_input/USGS-gtopo30_1.9x2.5_remap_c050602.nc' + endstr
print( fullURL)

#  Now we actually open the dataset
topo = xr.open_dataset( fullURL )
print( topo)


I am working on the file "Assignments/Assignment03 -- Energy budget in CESM.ipynb".
The first task is:
Homework questions: part A
Here we investigate differences between the control simulation and the 2xCO2 simulation (after it has reached its new, warmer equilibrium).
The two model output files you need are the control run:
som_1850_f19.cam.h0.clim.nc
and the doubled CO2 run:
som_1850_2xCO2.cam.h0.clim.nc

Calculate Equilibrium Climate Sensitivity (ECS) for the CESM slab ocean model.

Calculate the net TOA energy flux in the control run and in the equilibrated 2xCO2 run (time and global averages). Are they both close to zero?

What is the change in ASR and the change in OLR after doubling CO2?

What are the clear-sky and cloudy-sky components of those changes?

Make well-labeled maps of the change in the annual mean of these five quantities:

Surface temperature
ASR (total)
ASR (clear sky)
ASR (cloudy sky)
OLR (total)
OLR (clear sky)
OLR (cloud sky)

Comment on what you found in your maps.

Which regions warm more than others?
Are there any discernible spatial patterns in ASR and OLR changes?
What about the clear and cloudy sky components?
Comment on anything you find striking, interesting, or unexpected in these results.

Homework questions: part B
Here we investigate the transient adjustment to equilibrium.
For this, we will use the file
som_1850_2xCO2.cam.h0.global.nc
This file contains a monthly timeseries of the CESM model output from the 2xCO2 model run, which was initialized from the control run. Every variable in this file has already been averaged globally. We can use the timeseries to look at the adjustment of the global average temperature and energy budget to the new equilibrium.

Make a well-labeled graph of the timeseries of global mean surface temperature.
You will find that there is a well-defined annual cycle in this temperature. Offer a reasonable hypothesis to explain why such a cycle exists in the simulation.
Implement some kind of running average filter to smooth out the data. (There are many ways to do this... do whatever makes sense to you, but make sure your code is...
Can you write Python code for it?

import xarray as xr
url = "http://www.esrl.noaa.gov/psd/thredds/dodsC/Datasets/ncep.reanalysis.derived/surface_gauss/air.2m.mon.1981-2010.ltm.nc"
ncep_air2m = xr.open_dataset(url, decode_times=False)
## The NOAA ESRL server is shutdown! January 2019
#url = 'http://apdrc.soest.hawaii.edu:80/dods/public_data/Reanalysis_Data/NCEP/NCEP/clima/'
#ncep_air2m = xr.open_dataset(url + 'surface_gauss/air')

print( ncep_air2m)


I am working on the file "Assignments/Assignment03 -- Energy budget in CESM.ipynb".
The first task is:
The temperature data is called air. Take a look at the details:
Can you write Python code for it?


I am working on the file "Assignments/CP-400 - Assignment 1.ipynb".
The first task is:
Question 6 - Python dictionaries
For each item below, enter the code to complete the operation:
Can you write Python code for it?

# a  Add an additional value [c:30] to the following dictionary
tens = {a:10, b:20}

# b  Print the value of 'b'
tens = {a:10, b:20, c:30}

# c Merge the following two dictionaries in a single new dictionary
t1 = {'a': 100, 'b': 200}
t2 = {'c': 300, 'd': 400}


I am working on the file "Assignments/CP-400 - Assignment 1.ipynb".
The first task is:
Question 8 - Python date formatting
For each item below, enter the code to complete the operation:
Can you write Python code for it?

# a Print the current date in the format: "Month, Day Year"
from datetime import date

# b Calculate the number of days between the following two dates:
from datetime import date
d1 = date(2016, 10, 3)
d2 = date(2017, 10, 3)

# c Print current date and time in the format: 'YYYY-MM-DD HH:MM:SS'
import datetime


I am working on the file "Assignments/CP-400 Assignment 2.ipynb".
The first task is:
Question 6 - Regression Error Analysis
a. Review the following to estimate the mean squared error:
Can you write Python code for it?

# Calculate mean squared error
mse = np.mean((dataset.PRICE - lm.predict(x_param)) ** 2)
mse


I am working on the file "Assignments/CP-400 Assignment 2.ipynb".
The first task is:
b. Using Question 6 as an example, write the code for a simple regression error analysis.
Can you write Python code for it?

# Print Mean Squared Error

# Print Residuals


I am working on the file "Assignments/DV-assignment-01.ipynb".
The first task is:
Go to the documentation pages of matplotlib [http://matplotlib.org/contents.html] to find all the possible options for a plot and also to see more tutorials, videos and book chapters to help you along the way.
A few other nice tutorials:
* [http://www.labri.fr/perso/nrougier/teaching/matplotlib/]
Assignment
Recreate the above plot starting with the following lines. You will need to use this form a lot when you have plots that contains multiple subplots. Hint: you now need to change properties of the 'ax' object. Look on the internet for documentation.
Can you write Python code for it?

plt.figure()
fig, ax = plt.subplots()
ax.plot(X, Y, 'r--', linewidth=2);
ax.plot(X, Y/2, 'b-', linewidth=2);
ax.set_xlabel('X axis')
ax.set_ylabel('Y axis')
ax.set_title('Plot Title')
ax.set_xlim(-4, 4)
ax.set_ylim(-1.2, 1.2)
ax.legend(['red curve', 'blue curve'], loc='best');


I am working on the file "Assignments/DV-assignment-01.ipynb".
The first task is:
We will first create a simple plot of a mathematical function. We first create a numpy array of x-values. Then for each x-value we create the y-value, i.e. the function value. Plotting this function is as easy as giving it the x and y values.
Can you write Python code for it?

X = np.linspace(-np.pi, np.pi, 100) # define a NumPy array with 100 points in the range -Pi to Pi
Y = np.sin(X) # define the curve Y by the sine of X

plt.plot(X,Y); # use matplotlib to plot the function


I am working on the file "Assignments/DV-assignment-02.ipynb".
The first task is:
Assignment B
Now we want to plot the cities in The Netherlands only. Use a scatter plot again to plot the cities, but now vary the size of the marker and the color with the population of that city.
Use a colorbar to show how the color of the marker relates to its population.
Use sensible limits to your axes so that you show only mainland The Netherlands (and not the Dutch Antilles).
Can you write Python code for it?

dutch_cities = cities[ cities['Country'] =='nl' ]
plt.figure(figsize=[7,7]);

cm = plt.cm.get_cmap('YlOrRd')
y = dutch_cities.Latitude
x = dutch_cities.Longitude
pop = dutch_cities.Population
popsize = pop / 450

plt.xlim(3, 8)
plt.ylim(50.70, 53.6)

sc= plt.scatter(x,y,popsize,c=pop, cmap=cm, vmin=pop.min(), vmax=pop.max())

colobar = plt.colorbar(sc)