I am working on the file "Chapters/Chapter 13 - Working with Python files.ipynb".
The first task is:
Please run the output of the following command in the terminal:
Can you write Python code for it?

import sys
print(sys.executable + ' hello_world.py')


I am working on the file "Chapters/Chapter 13 - Working with Python files.ipynb".
The first task is:
You've succesfully run your first Python program!
4. Import your own functions
In Chapter 12, you've been introduced to importing modules and functions/methods.
You can see any python program that you create (so any .py file) as a module, which means that you can import it into another python program. Let's see how this works.
Please note that the following examples only work if all your python files are in the same directory. There are ways of importing python modules from other directories, but we will not discuss them here.
4.1 Importing your entire module
When importing your own functions from your own modules, several things are important. We have created two example scripts to illustrate them called the_program.py and utils.py. We recommend to open them to check the following things:

The extension .py is not used when importing modules. import utils will import the file utils.py [line 1 the_program.py]
We can use any function from the file. We can call the count_words function by typing utils.count_words [the_program.py line 6]
We can use any global variable declared in the imported module. E.g. utils.x and utils.python declared in utils.py can be used in the_program.py

4.2 Importing functions and variables individually
We can import specific functions using the syntax from MODULE import FUNCTION/VARIABLE
This can be seen in the file the_program_v2.py (lines 1-3). (Open the files the_program_v2.py and utils.py in an editor to check this).
4.3 Importing functions and variables to python notebooks
Please note that you can also import functions and variables from a python program while using notebooks. In this case, simply treat the notebook as the python files the_program.py and the_program_v2.py.
Can you write Python code for it?

from utils import count_words

words = ['how', 'often', 'does', 'each', 'string', 'occur', 'in', 'this', 'list', '?']

word2freq = count_words(words)
print('word2freq', word2freq)


I am working on the file "Chapters/Chapter 13 - Working with Python files.ipynb".
The first task is:
cd means 'change directory'. Here, you are using it to go to the directory we are currently working in. We use the os module to print the path to this directory (os.getcwd).
Please run the following command in the terminal:
python hello_world.py
You've succesfully run your first Python program!
B.) Running the program on Windows
Please use the terminal to navigate to the folder in which this notebook is placed by copying the output of the following cell in your terminal
Can you write Python code for it?

cwd = os.getcwd()
cwd_escaped_spaces = cwd.replace(' ', '^ ')
print('cd', cwd_escaped_spaces)


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
1.3.3 Readline()
The third operation readline() returns the next line of the file, returning the text up to and including the next newline character (\n, or \r\n on Windows). More simply put, this operation will read a file line-by-line. So if you call this operation again, it will return the next line in the file. Try it out below!
Can you write Python code for it?

filepath = "../Data/Charlie/charlie.txt"

infile = open(filepath, "r")
next_line = infile.readline()
print(next_line)

next_line = infile.readline()
print(next_line)

next_line = infile.readline()
print(next_line)
infile.close()


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
1.4.2 Using a context manager
There is actually an easier (and preferred) way to make sure that the file is closed as soon as you don't need it anymore, namely using what is called a context manager. Instead of using open() and close(), we use the  syntax shown below.
The main advantage of using the with-statement is that it automatically closes the file once you leave the local context defined by the indentation level. If you 'manually' open and close the file, you risk forgetting to close the file. Therefore, context managers are considered a best-practice, and we will use the with-statement in all of our following code.
From now on, we highly recommend using a context manager in your code.
Can you write Python code for it?

filepath = "../Data/Charlie/charlie.txt"

with open(filepath, "r") as infile:
    # the file is only open here
    # get content while file is open
    content = infile.read()

# the context manager took care of closing the file again
# we can now work with the content without having to worry about
# closing the file

print(content)


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
2 Manipulating file content
Once your file content is loaded in a Python variable, you can manipulate its content as you can manipulate any other variable. You can edit it, add/remove lines, count word occurrences, etc. Let's say we read the file content in a list of its lines, as shown below. Note that we can use all of the different methods for reading files in the context manager.
Can you write Python code for it?

filepath = "../Data/Charlie/charlie.txt"

with open(filepath, "r") as infile:
    lines = infile.readlines()

print(lines)


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
4.2 The os module
Another module that you will frequently see being used in examples is the os module. The os module has many features that can be very useful and which are not supported by the glob module. We will not go over each and every useful method here, but here's a list of some of the things that you can do (some of which we have seen above):

creating single or multiple directories: os.mkdir(), os.mkdirs();
removing single or multiple directories: os.rmdir(), os.rmdirs();
checking whether something is a file or a directory: os.path.isfile(), os.path.isdir();
split a path and return a tuple containing the directory and filename: os.path.split();
construct a pathname out of one or more partial pathnames: os.path.join()
split a filename and return a tuple containing the filename and the file extension: os.path.splitext()
get only the basename or the directory path: os.path.basename(), os.path.dirname().

Feel free to play around with these methods and figure out how they work yourself :-)
Can you write Python code for it?

# Start by importing the module:
import os

# let's use a filepath for testing it out:
filepath = "../Data/Charlie/charlie.txt"
os.path.basename(filepath)


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
A question mark (?) matches any single character in that position in the name. For example, the following code prints all filenames in the directory ../Data/dreams that start with 'vickie' followed by exactly 1 character and end with the extension .txt (note that this will not print vickie10.txt):
Can you write Python code for it?

for filename in glob.glob("../Data/Dreams/vickie?.txt"):
    print(filename)


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
Exercise 2:
Write a program that opens RedCircle.txt in the ../Data/RedCircle folder and prints a list containing all lines in the file:
Can you write Python code for it?


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
Exercise 3:
Create a counter dictionary like in block 2 (the dictionaries chapter), where you will count the number of occurences of each word in a file.
Can you write Python code for it?


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
Exercises
Exercise 1:
Write a program that opens RedCircle.txt in the ../Data/RedCircle folder and prints its content as a single string:
Can you write Python code for it?


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
If we only want to consider text files and ignore everything else (here a file called 'IGNORE_ME!'), we can specify this in our search by only looking for files with the extension .txt:
Can you write Python code for it?

for filename in glob.glob("../Data/Dreams/*.txt"):
    print(filename)


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
Note the last line of this code snippet: infile.close(). This closes our file, which is a very important operation. This prevents Python from keeping files that are unnecessary anymore still open. In the next subchapter, we will also see a more convenient way to ensure files get closed after their usage.
1.4. Closing the file
Here, we will introduce closing a file with the method close() and using a context manager to open and close files. After reading the contents of a file, the TextWrapper no longer needs to be open since we have stored the content as a variable. In fact, it is good practice to close the file as soon as you do not need it anymore.
1.4.1  close()
We do this by using the close() method as already shown several times above.
Can you write Python code for it?

filepath = "../Data/Charlie/charlie.txt"

# open file
infile = open(filepath , "r")

# assign content to a varialbe
content = infile.read()

# close file
infile.close()

# do whatever you want with the context, e.g. print it:

print(content)


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
Open the file charle_new.txt in the folder ../Data/Charlie in any text editor and read a personalized version of the story!
Note about append mode (a):
The third mode of opening a file is append ('a'). If the file 'charlie_new.txt' does not exist, then append and write act the same: they create this new file and fill it with content. The difference between write and append occurs when this file would exist. In that case, the write mode overwrites its content, while the append mode adds the new content at the end of the existing one.
4 Reading and writing multiple files
You will often have multiple files to work with. The folder ../Data/Dreams contains 10 text files describing dreams of Vickie, a 10-year-old girl. These texts are extracted from DreamBank.
To process multiple files, we often want to iterate over a list of files. These files are usually stored in one or multiple directories on your computer.
Instead of writing out every single file path, it is much more convenient to iterate over all the files in the directory ../Data/Dreams. So we need to find a way to tell Python: "I want to do something with all these files at this location!"
There are two modules which make dealing with multiple files a lot easier.

glob
os

We will introduce them below.
4.1 The glob module
The glob module is very useful to find all the pathnames matching a specified pattern according to the rules used by the Unix shell. You can use two wildcards: the asterisk (*) and the question mark (?). An asterisk matches zero or more characters in a segment of a name, while the question mark matches a single character in a segment of a name.
For example, the following code gives all filenames in the directory ../Data/dreams:
Can you write Python code for it?

import glob

for filename in glob.glob("../Data/Dreams/*"):
    print(filename)


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
Overview of possible mode arguments (the most important ones are 'r' and 'w'):
| Character | Meaning |
| --------- | ------- |
|'r' |  open for reading (default)|
|'w' |  open for writing, truncating the file first|
|'x' |  open for exclusive creation, failing if the file already exists|
|'a' |  open for writing, appending to the end of the file if it exists|
|'b' |  binary mode|
|'t' |  text mode (default)|
|'+' |  open a disk file for updating (reading and writing)|
|'U' |  universal newlines mode (deprecated)|
So far, we have opened the file. This, however, does not yet show us the file content. Try printing 'infile':
Can you write Python code for it?

infile = open("../Data/Charlie/charlie.txt" , "r")
print(infile)
infile.close()


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
Sometimes you see double dots at the beginning of the file path; this means 'the parent of the current directory'. When writing a file path, you can use the following:

/     means the root of the current drive;
./    means the current directory;
../   means the parent of the current directory.

Consider the directory tree below.

If you want to go from your current working directory (cwd) to the one directly above (dir3), your path is ../.
If you want to go to dir1, your path is ../../
If you want to go to dir5, your path is ../dir5/
If you want to go to dir2, your path is ../../dir2/

You will learn how to navigate your directory tree quite intuitively with a bit of practice. If you have any doubts, it is always a good idea to follow a quick tutorial on basic command-line operations.
<img src='images/directory_tree.png'>
Navigating your directory tree on Windows
Also, note that the formatting of file paths is different across operating systems. The file path, as specified above, should work on any UNIX platform (Linux, Mac). If you are using Windows, however, you might run into problems when formatting file paths in this way outside of this notebook, because Windows uses backslashes instead of forward slashes (Jupyter Notebook should already have taken care of these problems for you). In that case, it might be useful to have a look at this page about the differences between the file systems, and at this page about solving this problem in Python. In short, it's probably best if you use the code below (we will talk about the os module in more detail later today). This is very useful to know if you are a Windows user, and it will become relevant for the final assignment.
Can you write Python code for it?

# For windows:
import os
windows_file_path = os.path.normpath("C:/somePath/someFilename") # Use forward slashes


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
The code returns an empty list. To fix this, we have to open the file again:
Can you write Python code for it?

filepath = "../Data/Charlie/charlie.txt"

infile = open(filepath , "r")
content = infile.read()
infile = open(filepath, "r")
lines = infile.readlines()
print(content)
print(lines)
infile.close()


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
Then we can for instance preserve only the first 2 lines of the file, in a new variable:
Can you write Python code for it?


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
We can count the lines that are longer than 15 characters:
Can you write Python code for it?

counter=0
for line in lines:
    if len(line)>15:
        counter+=1
print(counter)


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
We can now save the manipulated content to a new file:
Can you write Python code for it?

filename = "../Data/Charlie/charlie_new.txt"
with open(filename, "w") as outfile:
    outfile.write(new_new_content)


I am working on the file "Chapters/Chapter 14 - Reading and writing text files.ipynb".
The first task is:
Which function to choose
For small files that you want to load entirely, you can use one of these three methods (readline, read, or readlines). Note, however, that we can also simply do the following to read a file line by line (this is recommended for larger files and when we are really only interested in a small portion of the file):
Can you write Python code for it?

infile = open(filename, "r")
for line in infile:
    print(line)
infile.close()


I am working on the file "Chapters/Chapter 15 - Off to analyzing text.ipynb".
The first task is:
2.3. Lemmatization
We can also use NLTK to lemmatize words.
The lemma of a word is the form of the word which is usually used in dictionary entries. This is useful for many NLP tasks, as it gives a better generalization than the strong a word appears in. To a computer, cat and cats are two completely different tokens, even though we know they are both forms of the same lemma.
2.3.1 The WordNet lemmatizer
We will use the WordNetLemmatizer for this using the lemmatize() function. In the code below, we loop through the list of verbs, lemmatize each of the verbs, and add them to a new list called verb_lemmas. Again, we show all the processing steps (consider the comments in the code below):
Can you write Python code for it?

#################################################################################
#### Process text as explained above ###

with open("../Data/Charlie/charlie.txt") as infile:
    content = infile.read()

tokens = nltk.word_tokenize(content)
tagged_tokens = nltk.pos_tag(tokens)

verb_tags = ["VBD", "VBG", "VBN", "VBP", "VBZ"]
verbs = []

for token, tag in tagged_tokens:
    if tag in verb_tags:
        verbs.append(token)

print(verbs)

#############################################################################
#### Use the list of verbs collected above to lemmatize all the verbs ###

# Instatiate a lemmatizer object
lmtzr = nltk.stem.wordnet.WordNetLemmatizer()

# Create list to collect all the verb lemmas:
verb_lemmas = []

for participle in verbs:
    # For this lemmatizer, we need to indicate the POS of the word (in this case, v = verb)
    lemma = lmtzr.lemmatize(participle, "v")
    verb_lemmas.append(lemma)
print(verb_lemmas)


I am working on the file "Chapters/Chapter 15 - Off to analyzing text.ipynb".
The first task is:
3 Nesting
So far, we typically used a single for-loop, or we were opening a single file at a time. In Python (and most programming languages), one can nest multiple loops or files in one another. For instance, we can use one (outer) for-loop to iterate through files, and then for each file iterate through all its sentences (internal for-loop). As we have learned above, glob is a convenient way of creating a list of files.
You might think: can we stretch this on more levels? Iterate through files, then iterate through the sentences in these files, then iterate through each word in these sentences, then iterate through each letter in these words, etc.  This is possible. Python (and most programming languages) allow you to perform nesting with (in theory) as many loops as you want. Keep in mind that nesting too much will eventually cause computational problems, but this also depends on the size of your data.
For the tasks we are treating here, a a couple of levels of nesting are fine.
In the code below, we want get an idea of the number and length of the sentences in the texts stored in the ../Data/dreams directory. We do this by creating two for loops: We iterate over all the files in the directory (loop 1), apply sentence tokenization and iterate over all the sentences in the file (loop 2).
Look at the code and comments below to figure out what is going on:
Can you write Python code for it?

import glob

### Loop 1 ####
# Loop1: iterate over all the files in the dreams directory
for filename in glob.glob("../Data/dreams/*.txt"):
    # read in the file and assign the content to a variable
    with open(filename, "r") as infile:
        content = infile.read()
    # split the content into sentences
    sentences = nltk.sent_tokenize(content)
    # Print the number of sentences in the file
    print(f"INFO: File {filename} has {len(sentences)} sentences")

    # For each file, assign a number to each sentence. Start with 0:
    counter=0

    #### Loop 2 ####
    # Loop 2: loop over all the sentences in a file:
    for sentence in sentences:
         # add 1 to the counter
        counter+=1
         # tokenize the sentence
        tokens=nltk.word_tokenize(sentence)
        # print the number of tokens per sentence
        print(f"Sentence {counter} has {len(tokens)} tokens")

    # print an empty line after each file (this belongs to loop 1)
    print()


I am working on the file "Chapters/Chapter 15 - Off to analyzing text.ipynb".
The first task is:
4 Putting it all together
In this section, we will use what we have learned above to write a small NLP program. We will go through all the steps and show how they can be put together. In the last chapters, we have already learned how to write functions. We will make use of this skill here.
Our goal is to collect all the nouns from Vickie's dream reports.
Before we write actual code, it is always good to consider which steps we need to carry out to reach the goal.
Important steps to remember:

create a list of all the files we want to process
open and read the files
tokenize the texts
perform pos-tagging
collect all the tokens analyzed as nouns

Remember, we first needed to import nltk to use it.
4.1 Writing a processing function for a single file
Since we want to carry out the same task for each of the files, it is very useful (and good practice!) to write a single function which can do the processing. The following function reads the specified file and returns the tokens with their POS tags:
Can you write Python code for it?

import nltk

def tag_tokens_file(filepath):
    """Read the contents of the file found at the location specified in
    FILEPATH and return a list of its tokens with their POS tags."""
    with open(filepath, "r") as infile:
        content = infile.read()
        tokens = nltk.word_tokenize(content)
        tagged_tokens = nltk.pos_tag(tokens)
    return tagged_tokens


I am working on the file "Chapters/Chapter 15 - Off to analyzing text.ipynb".
The first task is:
4.2 Iterating over all the files and applying the processing function
We can also do this for each of the files in the ../Data/dreams directory by using a for-loop:
Can you write Python code for it?

import glob

# Iterate over the `.txt` files in the directory and perform POS tagging on each of them
for filename in glob.glob("../Data/dreams/*.txt"):
    tagged_tokens = tag_tokens_file(filename)
    print(filename, "\n", tagged_tokens, "\n")


I am working on the file "Chapters/Chapter 15 - Off to analyzing text.ipynb".
The first task is:
Exercise 2:
The resulting list verb_lemmas above contains a lot of duplicates. Do you remember how you can get rid of these duplicates? Create a set in which each verb occurs only once and name it unique_verbs. Then print it.
Can you write Python code for it?

## the list is stored under the variable 'verb_lemmas'

# your code here

# Test your code here! If your code is correct, you should get a compliment :-)
assert len(unique_verbs) == 28
print("Well done!")


I am working on the file "Chapters/Chapter 15 - Off to analyzing text.ipynb".
The first task is:
Note about the wordnet lemmatizer:
We need to specify a POS tag to the WordNet lemmatizer, in a WordNet format ("n" for noun, "v" for verb, "a" for adjective). If we do not indicate the Part-of-Speech tag, the WordNet lemmatizer thinks it is a noun (this is the default value for its part-of-speech). See the examples below:
Can you write Python code for it?

test_nouns = ('building', 'applications', 'leafs')
for n in test_nouns:
    print(f"Noun in conjugated form: {n}")
    default_lemma=lmtzr.lemmatize(n) # default lemmatization, without specifying POS, n is interpretted as a noun!
    print(f"Default lemmatization: {default_lemma}")
    verb_lemma=lmtzr.lemmatize(n, 'v')
    print(f"Lemmatization as a verb: {verb_lemma}")
    noun_lemma=lmtzr.lemmatize(n, 'n')
    print(f"Lemmatization as a noun: {noun_lemma}")
    print()

test_verbs=('grew', 'standing', 'plays')
for v in test_verbs:
    print(f"Verb in conjugated form: {v}")
    default_lemma=lmtzr.lemmatize(v) # default lemmatization, without specifying POS, v is interpretted as a noun!
    print(f"Default lemmatization: {default_lemma}")
    verb_lemma=lmtzr.lemmatize(v, 'v')
    print(f"Lemmatization as a verb: {verb_lemma}")
    noun_lemma=lmtzr.lemmatize(v, 'n')
    print(f"Lemmatization as a noun: {noun_lemma}")
    print()


I am working on the file "Chapters/Chapter 15 - Off to analyzing text.ipynb".
The first task is:
Now that we have installed and downloaded NLTK, let's look at an example of a simple NLP pipeline. In the following cell, you can observe how we tokenize raw text into tokens and setnences, perform part of speech tagging and lemmatize some of the tokens. Don't worry about the details just yet - we will go trhough them step by step.
Can you write Python code for it?

text = "This example sentence is used for illustrating some basic NLP tasks. Language is awesome!"

# Tokenization
tokens = nltk.word_tokenize(text)

# Sentence splitting
sentences = nltk.sent_tokenize(text)

# POS tagging
tagged_tokens = nltk.pos_tag(tokens)

# Lemmatization
lmtzr = nltk.stem.wordnet.WordNetLemmatizer()
lemma=lmtzr.lemmatize(tokens[4], 'v')

# Printing all information
print(tokens)
print(sentences)
print(tagged_tokens)
print(lemma)


I am working on the file "Chapters/Chapter 15 - Off to analyzing text.ipynb".
The first task is:
Now we have an idea what Vickie dreams about!
Exercises
Exercise 1:
Try to collect all the present participles in the the text store in ../Data/Charlie/charlie.txt using the NLTK tokenizer and POS-tagger.
Can you write Python code for it?


I am working on the file "Chapters/Chapter 15 - Off to analyzing text.ipynb".
The first task is:
This looks good! We now have a list of words like boiling, sizzling, etc. However, we can see that there is one word in the list that actually is not a present participle (ceiling). Of course, also other words can end with -ing. So if we want to find all present participles, we have to come up with a smarter solution.
2.2. Part-of-speech (POS) tagging
Once again, NLTK comes to the rescue. Using the function pos_tag(), we can label each word in the text with its part of speech.
To do pos-tagging, you first need to tokenize the text. We have already done this above, but we will repeat the steps here, so you get a sense of what an NLP pipeline may look like.
2.2.1 pos_tag()
To see how pos_tag() can be used, we can (as always) look at the documentation by using the help() function. As we can see, pos_tag() takes a tokenized text as input and returns a list of tuples in which the first element corresponds to the token and the second to the assigned pos-tag.
Can you write Python code for it?

# As always, we can start by reading the documentation:
help(nltk.pos_tag)

# Open and read in file as a string, assign it to the variable `content`
with open("../Data/Charlie/charlie.txt") as infile:
    content = infile.read()

# Split up entire text into tokens using word_tokenize():
tokens = nltk.word_tokenize(content)

# Apply pos tagging to the tokenized text
tagged_tokens = nltk.pos_tag(tokens)

# Inspect pos tags
print(tagged_tokens)


I am working on the file "Chapters/Chapter 15 - Off to analyzing text.ipynb".
The first task is:
We can now do all sorts of cool things with these lists. For example, we can search for all words that have certain letters in them and add them to a list. Let's say we want to find all present participles in the text. We know that present participles end with -ing, so we can do something like this:
Can you write Python code for it?

# Open and read in file as a string, assign it to the variable `content`
with open("../Data/Charlie/charlie.txt") as infile:
    content = infile.read()

# Split up entire text into tokens using word_tokenize():
tokens = nltk.word_tokenize(content)

# create an empty list to collect all words having the present participle -ing:
present_participles = []

# looking through all tokens
for token in tokens:
    # checking if a token ends with the present parciciple -ing
    if token.endswith("ing"):
        # if the condition is met, add it to the list we created above (present_participles)
        present_participles.append(token)

# Print the list to inspect it
print(present_participles)


I am working on the file "Chapters/Chapter 15 - Off to analyzing text.ipynb".
The first task is:
You should get the following list:
['boiling', 'bubbling', 'hissing', 'sizzling', 'clanking', 'running', 'hopping', 'knowing', 'rubbing', 'cackling', 'going']
Can you write Python code for it?

# we can test our code using the assert statement (don't worry about this now,
# but if you want to use it, you can probably figure out how it works yourself :-)
# If our code is correct, we should get a compliment :-)
assert len(present_participles) == 11 and type(present_participles[0]) == str
print("Well done!")


I am working on the file "Chapters/Chapter 16 - Data formats I (CSV and TSV).ipynb".
The first task is:
2.1 Reading rows as lists
Now, let's see how we can get to the 'list of lists' representation. We can do that by iterating over each line of this file (as we have seen before), and then split each row into columns using the split() method:
Can you write Python code for it?

# Read the file and get all lines

# assign new line character to a variable
# (we are going to need this when processing the lines of the file)
#new line on linux/unix
newline_char = '\n'
#new line on windows (double-check this in an editor or use repr())
#newline_char = '\r\n'

# create empty list to collect the rows:
csv_data = []
# store path to file in variable
filename = "../Data/csv_data/baby_names/names_by_state/AK.csv"
# open file and read in line-by line
with open(filename, "r") as csvfile:
    for row in csvfile:
        # remove new-line charater -
        row = row.strip(newline_char)
        # separate the cells on comma (csv file)
        cells = row.split(",")   # split the line into cells
        csv_data.append(cells)

# Print only first 10 rows
print(csv_data[:10])
print()

# Iterate over first 10 rows - this is more readable
for row in csv_data[:10]:
    print(row)


I am working on the file "Chapters/Chapter 16 - Data formats I (CSV and TSV).ipynb".
The first task is:
2.2 Reading rows as dicts
We can also create a 'list of dicts'. We do this by first creating a dictionary for each row, and appending that dictionary to the list of rows. We can simply use the column headers as keys, so we can easily access the corresponding values later on.
Note: Do you see that we also convert the numerical values to int?
Can you write Python code for it?

# Read the file and get all lines

# again, create a list to collect the rows:
csv_data = []

filename = "../Data/csv_data/baby_names/names_by_state/AK.csv"
with open(filename, "r") as csvfile:
    for row in csvfile:
        row = row.strip("\n")      # remove all newlines
        columns = row.split(",")   # split the line into columns

        # Create a dictionary and add to list
        dict_row = {"state": columns[0],
                    "gender": columns[1],
                    "year": int(columns[2]),
                    "name": columns[3],
                    "frequency": int(columns[4])}
        csv_data.append(dict_row)

# First 10 rows
print(csv_data[:10])
print()

# Iterate over first 10 rows
for row in csv_data[:10]:
    print(row)


I am working on the file "Chapters/Chapter 16 - Data formats I (CSV and TSV).ipynb".
The first task is:
4. Dealing with column headers
Many csv/tsv files contain headers. This means that the first row contains the names of the columns. We easily read and write these kinds of files without having to type the headers manually.
4.1 Reading csv/tsv files with a header row
Consider the file called Concreteness_ratings_Brysbaert_et_al_BRM.txt (we're reading it in and printing the first 5 lines below).
The file has 10 columns, each of which have a column header. This means that all the values in the first row are not actual values. We need to treat them differently if we want to analyze the data (i.e. everything from the second row on). In addition, it would be convenient to use this information right away, without having to first inspect the file and manually type the headers as keys in dictionaries!
Can you write Python code for it?

with open('../Data/csv_data/Concreteness_ratings_Brysbaert_et_al_BRM.tsv') as infile:
    lines = infile.read().split('\n')

for line in lines[:5]:
    print(line)


I am working on the file "Chapters/Chapter 16 - Data formats I (CSV and TSV).ipynb".
The first task is:
4.2 Writing csv/tsv files with a header row
You can also write files with headers:
Can you write Python code for it?

row1 = {'name': 'Harry', 'house' : 'Gryffindor'}
row2 = {'name': 'Ron', 'house' : 'Gryffindor'}
row3 = {'name': 'Hannah', 'house': 'Hufflepuff'}
row4 = {'name': 'Leta', 'house' : 'Slytherin'}

data_dict_list = [row1, row2, row2, row4]

for d in data_dict_list:
    print(d)

header_row = ### your code here

with open('../Data/csv_data/hp_example.csv', 'w') as outfile:
    # write header
    for d in data_dict_list:
        row_list = d.values()
        row = ','.join(values)
        outfile.write(row+'\n')


I am working on the file "Chapters/Chapter 16 - Data formats I (CSV and TSV).ipynb".
The first task is:
A note on csv/tsv files containing a lot of textual data
If you're dealing with tables containing textual data (e.g. full sentences or tweets), you will realize that splitting on commas or tabs may result in confusion (as text often contains commas and may contain tabs). Consider for instances this file: ../Data/csv_data/debate.csv.
You may have ideas for a fix regarding this specific file and surely you will eventually find a good solution for it. However, for these more complex CSV files, we recommend that you make use of the csv module. Feel free to explore it yourself, but do not use it for the exercises and assignments in this course.
Exercises
Exercise 1:
Have another look at the code below. Can you predict what would happen if you skip some of the steps? How will the data be different?
Can you write Python code for it?

filename = "../Data/csv_data/baby_names/names_by_state/AK.csv"
with open(filename, "r") as csvfile:
    csv_data = []
    for row in csvfile:
        #row = row.strip("\n")      # what happens if you skip this step?
        columns = row.split(",")
        csv_data.append(columns)
print(csv_data[0:2])

filename = "../Data/csv_data/baby_names/names_by_state/AK.csv"
with open(filename, "r") as csvfile:
    csv_data = []
    for row in csvfile:
        row = row.strip("\n")
        #columns = row.split(",")   # what happes if you skip this step?
        csv_data.append(row)        # replaced columns by row
print(csv_data[0:2])


I am working on the file "Chapters/Chapter 16 - Data formats I (CSV and TSV).ipynb".
The first task is:
Again, we can now easily work with the data, but now we use the names of the columns instead of indices to access the cells. This can make the code more readable. Have a look at the following examples:
Can you write Python code for it?

# Example: print all information of the first 5 rows
for row in csv_data[:5]:
    for column_name, cell_value in row.items():
        print(column_name, "=", cell_value)
    print()

# Example: print all names given in 1912
for row in csv_data:
    if row["year"] == 1912:
        print(row["name"])


I am working on the file "Chapters/Chapter 16 - Data formats I (CSV and TSV).ipynb".
The first task is:
Can you complete the example below?
Can you write Python code for it?

data_dict_list = []

with open('../Data/csv_data/Concreteness_ratings_Brysbaert_et_al_BRM.tsv') as infile:
    lines = infile.read().split('\n')

# the first row is at index 0 and contains the headers
headers = lines[0]

# we consider everythin following the header row as data
for line in lines[1:3]:
    line_list = line.split('\t')
    header_list = headers.split('\t')
    row_dict = dict()

    # can you think of a way to fill the row dict?
    # hint: check out the zip function
    # zip for iterating over two lists at the same time

    ### YOUR CODE ####

    data_dict_list.append(row_dict)


I am working on the file "Chapters/Chapter 16 - Data formats I (CSV and TSV).ipynb".
The first task is:
Exercise 2:
Read the csv data in AK.csv and store it as a list of lists. Now print the following:
- all names that started with an M given in 1990
- all unique female names (hint: create a set)
- all names that were given more than 30 times in a certain year (print name+year)
- all unique names longer than 8 letters given between 1985 and 1990
Read the csv data in AK.csv and store it as a list of dicts. Now print the following:
- all names that started with an M given in 1990
- all unique female names (hint: create a set)
- all names that were given more than 30 times in a certain year (print name+year)
- all unique names longer than 8 letters given between 1985 and 1990
Exercise 3:
Can you think of a way to add a header to the TSV file below? Hint: make use of the dictionary keys.
Can you write Python code for it?

# Create list of dicts
address_book = [
    {"name":"John", "e-mail":"john@example.com", "phone":"555-1234"},
    {"name":"William", "e-mail":"william@example.com", "phone":"555-5678"},
    {"name":"Jane", "e-mail":"jane@example.com", "phone":"555-7777"}
]

# Write the list of dicts to a TSV file
outfilename = "../Data/csv_data/address_book.tsv"
with open(outfilename, "w") as outfile:

    # Write the header
    # your code here

    # Write the rows using the values of the dictionaries
    for row in address_book:
        column_values = row.values()
        line = "\t".join(column_values) + '\n'
        outfile.write(line)


I am working on the file "Chapters/Chapter 16 - Data formats I (CSV and TSV).ipynb".
The first task is:
Exercise 5:
Create a function load_tabular_data(filename, delimiter) that receives the filename and a delimiter as input parameters, and returns the file content as a list of lists. Then we can call this function for CSV (with a comma as an argument) and for TSV (with a tabulator as an argument). Also make sure that delimiter is a keyword parameter, with a defaul value ','.
Can you write Python code for it?

# Create your function here

# Now let's test the function
csv_filename = "../Data/csv_data/baby_names/names_by_state/AK.csv"


I am working on the file "Chapters/Chapter 16 - Data formats I (CSV and TSV).ipynb".
The first task is:
It does not really matter whether you choose for a 'list of lists' or a 'list of dicts'. Just use the one that you prefer.
3. Writing CSV files
Let's say now we have a table in Python stored as a 'list of lists' or as a 'list of dicts', and we want to store our result in a CSV file. This is basically the inverse process of reading a CSV file.
3.1 Writing rows as lists
In order to write a list of lists as a CSV file, we need to iterate over the rows and make a string out of them. Remember that we can concatenate strings in a list with any separator with the join() method:
Can you write Python code for it?

a_list = ["John", "john@example.nl", "555-1234"]
a_string = ",".join(a_list)
print(a_string)


I am working on the file "Chapters/Chapter 16 - Data formats I (CSV and TSV).ipynb".
The first task is:
Please also have a close look at the internal representation of the file. Do you see how the columns are separated by commas, and the rows by newline characters \n?
Can you write Python code for it?


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
2.2 Writing JSON to file or string
Let's first define our Python dictionary again:
Can you write Python code for it?

dict_doe_family = {
    "John": {
        "first name": "John",
        "last name": "Doe",
        "gender": "male",
        "age": 30,
        "favorite_animal": "panda",
        "married": True,
        "children": ["James", "Jennifer"],
        "hobbies": ["photography", "sky diving", "reading"]},
    "Jane": {
        "first name": "Jane",
        "last name": "Doe",
        "gender": "female",
        "age": 27,
        "favorite_animal": "zebra",
        "married": False,
        "children": None,
        "hobbies": ["cooking", "gaming", "tennis"]}}


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
3. Accessing data in a json dictionary
Note that the values in this dictionary can be containers themselves: Each key has another dictionary as a value. The keys 'children' and 'hobbies' have lists as values. Note that we can look at such a dictionary in terms of layers of nesting.
Consider the dict below. How many layers (i.e. containers within containers) can you identify?
Can you write Python code for it?


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
Exercise 2:
Working with JSON data thus means working with dictionaries. Let's practice a bit more with accessing the values of dictionaries by printing the following:
* all (parent) names
* the age of Jane
* the first hobby of John
* the favorite animal of each person (use a for-loop)
Can you write Python code for it?

# Example: This will print the gender of "John"
print(dict_doe_family["John"]["gender"])


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
Exercise 3:
Please add the following entry to dict_doe_family. Write the result to a JSON file.
Can you write Python code for it?

julia = {"Julia": {"first name": "Julia",
                   "last name": "Doe",
                   "age": 29,
                   "favorite_animal": "penguin",
                   "married": False,
                   "children": ["Jack"],
                   "hobbies": ["snowboarding", "hiking"]}}

# your code here


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
Exercise 4:
Let's have a look at another example. The JSON file ../Data/json_data/StrangerThings.json contains information about episodes from the TV show Stranger Things. Read the file and store the dictionary as tv_show.
Can you write Python code for it?


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
Exercises
Exercise 1:
What is the difference between str() and json.dumps() to convert a Python dictionary to a string? Try it out below.
Can you write Python code for it?

dict_doe_family = {
    "John": {
        "first name": "John",
        "last name": "Doe",
        "gender": "male",
        "age": 30,
        "favorite_animal": "panda",
        "married": True,
        "children": ["James", "Jennifer"],
        "hobbies": ["photography", "sky diving", "reading"]},
    "Jane": {
        "first name": "Jane",
        "last name": "Doe",
        "gender": "female",
        "age": 27,
        "favorite_animal": "zebra",
        "married": False,
        "children": None,
        "hobbies": ["cooking", "gaming", "tennis"]}}

str_doe_family = json.dumps(dict_doe_family)
print(str_doe_family)

str_doe_family = str(dict_doe_family)
print(str_doe_family)


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
Now let's extract the hobbies of all family members. Can you finish the code below?
Hint: If you are confused about how to access elements in the structure, the type() function is your friend! If you are having a hard time remembering how to deal with the different containers, please have a look at the chapters about containers. Also, remember that you can always get a list of all methods of a container by using the dir() function (+ help() to see what they are doing).
Can you write Python code for it?

# iterate over family dict by accessing
#the family members (keys) and their information (values):

all_hobbies = []
for member, info_dict in dict_doe_family.items():
    # check what we are accessing:
    print(member, type(info_dict))
    # access hobbies from info_dict
    hobbies = info_dict['hobbies']
    print(hobbies)
    # your code here


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
Often, we will need to extract information from such a nested structure. To do this, it is helpful to remember what you have learned in Block II about containers and looping.
Let's say we want to extract the hobbies of Doe family members. How could we approach this? There are various ways of working your way through the layers.
For instance, you can use dictionary keys:
Can you write Python code for it?

# access information about John
john_info = dict_doe_family['John']
print(john_info)
# access information about John's hobbies:
john_hobbies = john_info['hobbies']
print(john_hobbies)

# You can also do this in one go:
john_hobbies = dict_doe_family['John']['hobbies']


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
The dumps() method is used to convert a Python dictionary to a JSON formatted string:
Can you write Python code for it?

str_doe_family = json.dumps(dict_doe_family)
print(str_doe_family)


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
The json.dump() method is used to write a Python dictionary to a JSON encoded file:
Can you write Python code for it?

with open("../Data/json_data/Doe.json", "w") as outfile:
     json.dump(dict_doe_family, outfile)


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
The loads() method is used to load a JSON formatted string as a Python dictionary. This is useful if you want to create a json dictionary from a string. There are not many situations in which you will have to use it. You may come accross json structures stored as strings in when working with corpora or collecting human annotations with annotation software.
Can you write Python code for it?

str_doe_family = """
{
    "Jane": {
        "age": 27,
        "children": null,
        "favorite_animal": "zebra",
        "first name": "Jane",
        "gender": "female",
        "hobbies": [
            "cooking",
            "gaming",
            "tennis"
        ],
        "last name": "Doe",
        "married": false
    },
    "John": {
        "age": 30,
        "children": [
            "James",
            "Jennifer"
        ],
        "favorite_animal": "panda",
        "first name": "John",
        "gender": "male",
        "hobbies": [
            "photography",
            "sky diving",
            "reading"
        ],
        "last name": "Doe",
        "married": true
    }
}
"""

dict_doe_family = json.loads(str_doe_family)
print(type(dict_doe_family))
print(dict_doe_family)


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
To help you understand the structure a bit, first have a look at the following examples:
Can you write Python code for it?

print(tv_show.keys())

print(tv_show["_embedded"].keys())

print(tv_show["_embedded"]["episodes"])


I am working on the file "Chapters/Chapter 17 -  Data formats II (JSON).ipynb".
The first task is:
We will focus on the following methods:
* to read JSON: the methods json.load() and json.loads()
* to write JSON: the methods json.dump() and json.dumps()
The functions with an s take string arguments.
2.1 Loading JSON from file or string
The load() method is used to load a JSON encoded file as a Python dictionary:
Can you write Python code for it?

# open file just as you would open a txt file
with open("../Data/json_data/Doe.json", "r") as infile:
    # read in file content as dict using the json module
    dict_doe_family = json.load(infile)
    print(type(dict_doe_family))
    print(dict_doe_family)


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
3.2 Accessing elements
There are several ways of accessing XML elements. The find() method returns the first matching child.
Can you write Python code for it?

first_person_el = root.find('person')
# Printing the element itself again only shows a reference
print(first_person_el)
#instead, we use etree.dump:
etree.dump(first_person_el, pretty_print=True)


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
3.3 Accessing element information
We will now show how to access the attributes, text, and tag of an element.
The get() method is used to access the attribute of an element.
Attention: If an attribute does not exist, it will return None. Hence, there will not be an error.
Can you write Python code for it?

first_person_el = root.find('person')
role_first_person_el = first_person_el.get('role')
attribute_not_found = first_person_el.get('blabla')
print('role first person element:', role_first_person_el)
print('value if not found:', attribute_not_found)


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
4 How to deal with more than one layer
In our previous example, we had an XML with only one nested layer (person). However, XML can deal with many more.
Let's look at such an example and think about how you would access the first target element, i.e.
```xml
<target id="t1" />
```
```xml
<NAF xml:lang="en" version="v3">
    <terms>
        <term id="t1" type="open" lemma="Tom" pos="N" morphofeat="NNP">
        <term id="t2" type="open" lemma="Cruise" pos="N" morphofeat="NNP">
        <term id="t3" type="open" lemma="be" pos="V" morphofeat="VBZ">
        <term id="t4" type="open" lemma="an" pos="R" morphofeat="DT">
        <term id="t5" type="open" lemma="actor" pos="N" morphofeat="NN">
    </terms>
    <entities>
        <entity id="e3" type="PERSON">
              <references>
                  <span>
                      <target id="t1" />
                      <target id="t2" />
                  </span>
              </references>
        </entity>
    </entities>
</NAF>
```
Again, we use etree.fromstring() to load XML from a string:
Can you write Python code for it?

naf_string = """
<NAF xml:lang="en" version="v3">
    <text>
        <wf id="w1" offset="0" length="3" sent="1" para="1">tom</wf>
        <wf id="w2" offset="4" length="6" sent="1" para="1">cruise</wf>
        <wf id="w3" offset="11" length="2" sent="1" para="1">is</wf>
        <wf id="w4" offset="14" length="2" sent="1" para="1">an</wf>
        <wf id="w5" offset="17" length="5" sent="1" para="1">actor</wf>
    </text>
    <terms>
        <term id="t1" type="open" lemma="Tom" pos="N" morphofeat="NNP"/>
        <term id="t2" type="open" lemma="Cruise" pos="N" morphofeat="NNP"/>
        <term id="t3" type="open" lemma="be" pos="V" morphofeat="VBZ"/>
        <term id="t4" type="open" lemma="an" pos="R" morphofeat="DT"/>
        <term id="t5" type="open" lemma="actor" pos="N" morphofeat="NN"/>
    </terms>
    <entities>
        <entity id="e3" type="PERSON">
              <references>
                  <span>
                      <target id="t1" />
                      <target id="t2" />
                  </span>
              </references>
        </entity>
    </entities>
</NAF>
"""

naf = etree.fromstring(naf_string)
print(type(naf))
etree.dump(naf, pretty_print=True)


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
6. EXTRA: Creating your own XML
Please note that this section is optional, meaning that you don't need to understand this section in order to complete the assignment.
There are three main steps:
* Step a: Create an XML object with a root element
* Step b: Creating child elements and adding them
* Step c: Writing to a file
Step a: Create an XML object with a root element
You create a new XML object by:
* creating the root element -> using etree.Element
* creating the main XML object -> using etree.ElementTree
You do not have to fully understand how this works. Please make sure you can reuse this code snippet when you create your own XML.
Can you write Python code for it?

our_root = etree.Element('Course')
our_tree = etree.ElementTree(our_root)


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
As we have seen before, XML can have multiple nested layers. Creating these works the same way as adding child elements to the root, but now we specify one of the other elements as the parent (in this case, new_person_element).
Can you write Python code for it?

# Define tag, attributes and text of the new element
tag = 'pet'
attributes = {'role': 'joy'}
name_pet = 'Romeo'

# Add to new_person_element
new_pet_element = etree.SubElement(new_person_element, tag, attrib=attributes) # parent is new_person_element
new_pet_element.text = name_pet

# Inspect the current XML
etree.dump(our_root, pretty_print=True)


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
As you can see, etree.parse() returns an ElementTree, whereas etree.fromstring() returns an Element. One of the important differences is that the ElementTree class serialises as a complete document, as opposed to a single Element. This includes top-level processing instructions and comments, as well as a DOCTYPE and other DTD content in the document. For now, it's not too important that you know what these are; just remember that there is a difference btween ElementTree and Element.
3.1 Accessing root element
While etree.fromstring() gives you the root element right away, etree.parse() does not. In order to access the root element of ElementTree, we first need to use the getroot() method. Note that this does not show the XML element itself, but only a reference. In order to show the element itself, we can use the etree.dump() method.
Hint: etree.dump is helpful for inspecting (parts of) an xml structure you have loaded from a file. You will see examples of this later.
Can you write Python code for it?

root = tree.getroot()
print('root', type(root), root)
print()
print('etree.dump example')
etree.dump(root, pretty_print=True)


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
As you see, we created an XML object, containing only the root element Course.
Step b: Creating child elements and adding them
There are two ways to add child elements to the root element. The first is to create an element using the etree.Element() method and using append() to add it to the root:
Can you write Python code for it?

# Define tag, attributes and text of the new element
tag = 'person' # what the start and end tag will be
attributes = {'role': 'student'} # dictionary of attributes, can be more than one
name_student = 'Lee' # the text of the elements

# Create new Element
new_person_element = etree.Element(tag, attrib=attributes)
new_person_element.text = name_student

# Add to root
our_root.append(new_person_element)

# Inspect the current XML
etree.dump(our_root, pretty_print=True)


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
Exercises
Exercise 1:
Have another look at the XML below. Then print the following information:
* the names of all students
* the names of all instructors whose name starts with 'Van'
* all names containing a space
* the role of 'Rubber duck'
Can you write Python code for it?

xml_string = """
<Course>
    <person role="coordinator">Van der Vliet</person>
    <person role="instructor">Van Miltenburg</person>
    <person role="instructor">Van Son</person>
    <person role="instructor">Marten Postma</person>
    <person role="student">Baloche</person>
    <person role="student">De Boer</person>
    <animal role="student">Rubber duck</animal>
    <person role="student">Van Doorn</person>
    <person role="student">De Jager</person>
    <person role="student">King</person>
    <person role="student">Kingham</person>
    <person role="student">Mózes</person>
    <person role="student">Rübsaam</person>
    <person role="student">Torsi</person>
    <person role="student">Witteman</person>
    <person role="student">Wouterse</person>
    <person/>
</Course>
"""

tree = etree.fromstring(xml_string)
print(type(tree))


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
However, this is so common that there is a shorter and much more efficient way to do this: by using etree.SubElement(). It accepts the same arguments as the etree.Element() method, but additionally requires the parent as first argument:
Can you write Python code for it?

# Define tag, attributes and text of the new element
tag = 'person'
attributes = {'role': 'student'}
name_student = 'Pitt'

# Add to root
another_person_element = etree.SubElement(our_root, tag, attrib=attributes) # parent is our_root
another_person_element.text = name_student

# Inspect the current XML
etree.dump(our_root, pretty_print=True)


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
In order to get a list of all person children, we can use the findall() method.
Notice that this does not return the animal since we are looking for person elements.
Can you write Python code for it?

all_person_els = root.findall('person')
# Check how many we found
print(len(all_person_els))
all_person_els


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
Is there a better way? The answer is yes! The following way is an easier way to find our target element:
Can you write Python code for it?

target_el = naf.find('entities/entity/references/span/target')
etree.dump(target_el, pretty_print=True)


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
Please note that the structure is as follows:
* the NAF element is the parent of the elements text, terms, and entities
* the wf elements are children of the text element, which provides us information about the position of words in the text, e.g. that tom is the first word in the text (id="w1") and in the first sentence (sent="1")
* the term elements are children of the term elements, which provide us information about lemmatization and part of speech
* the entity element is a child of the entities element. We learn from the entity element that the terms t1 and t2 (e.g. Tom Cruise) form an entity of type person.
One way of accessing the first target element is by going one level at a time:
Can you write Python code for it?

entities_el = naf.find('entities')
entity_el = entities_el.find('entity')
references_el = entity_el.find('references')
span_el = references_el.find('span')
target_el = span_el.find('target')
etree.dump(target_el, pretty_print=True)


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
Step c: Writing to a file
This is how we can write our selfmade XML to a file. Please inspect ../Data/xml_data/selfmade.xml using a text editor to check if it worked.
Can you write Python code for it?

with open('../Data/xml_data/selfmade.xml', 'wb') as outfile:
    our_tree.write(outfile,
                   pretty_print=True,
                   xml_declaration=True,
                   encoding='utf-8')


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
The etree.parse() method is used to load XML files on your computer:
Can you write Python code for it?

tree = etree.parse('../Data/xml_data/course.xml')
print(tree)
print(type(tree))


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
We can inspect what we have created by using the etree.dump() method. As you can see, we only have the root node Course currently in our document.
Can you write Python code for it?


I am working on the file "Chapters/Chapter 18 - Data formats III (XML).ipynb".
The first task is:
We will focus on the following methods/attributes:
* to parse the XML from file or string: the methods etree.parse() and etree.fromstring()
* to access the root element: the methods getroot()
* to access elements: the methods find(), findall(), and getchildren()
* to access attributes: the method get()
* to access element information: the attributes tag and text
3.1 Parsing XML from file or string
The etree.fromstring() is used to parse XML from a string:
Can you write Python code for it?

xml_string = """
<Course>
    <person role="coordinator">Van der Vliet</person>
    <person role="instructor">Van Miltenburg</person>
    <person role="instructor">Van Son</person>
    <person role="instructor">Marten Postma</person>
    <person role="student">Baloche</person>
    <person role="student">De Boer</person>
    <animal role="student">Rubber duck</animal>
    <person role="student">Van Doorn</person>
    <person role="student">De Jager</person>
    <person role="student">King</person>
    <person role="student">Kingham</person>
    <person role="student">Mózes</person>
    <person role="student">Rübsaam</person>
    <person role="student">Torsi</person>
    <person role="student">Witteman</person>
    <person role="student">Wouterse</person>
    <person/>
</Course>
"""

tree = etree.fromstring(xml_string)
print(type(tree))

# printing the tree only shows a reference to the tree, but not the tree itself
# To access information, you will have to use the methods we introduce below.
print(tree)


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
4. NLTK vs. spaCy vs. CoreNLP
There might be different reasons why you want to use NLTK, spaCy or Stanford CoreNLP. There are differences in efficiency, quality, user friendliness, functionalities, output formats, etc. At this moment, we advise you to go with spaCy because of its ease in use and high quality performance.
Here's an example of both NLTK and spaCy in action.

The example text is a case in point. What goes wrong here?
Try experimenting with the text to see what the differences are.
Can you write Python code for it?

import nltk
import spacy

nlp = spacy.load('en_core_web_sm')

text = "I like cheese very much"

print("NLTK results:")
nltk_tagged = nltk.pos_tag(text.split())
print(nltk_tagged)

print()

print("spaCy results:")
doc = nlp(text)
spacy_tagged = []
for token in doc:
    tag_data = (token.orth_, token.tag_,)
    spacy_tagged.append(tag_data)
print(spacy_tagged)


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
Do you want to learn more about the differences between NLTK, spaCy and CoreNLP? Here are some links:
- Facts & Figures (spaCy)
- About speed (CoreNLP vs. spaCy)
- NLTK vs. spaCy: Natural Language Processing in Python
- What are the advantages of Spacy vs NLTK?
- 5 Heroic Python NLP Libraries
5. Some other useful modules for cleaning and preprocessing
Data is often messy, noisy or includes irrelevant information. Therefore, chances are big that you will need to do some cleaning before you can start with your analysis. This is especially true for social media texts, such as tweets, chats, and emails. Typically, these texts are informal and notoriously noisy. Normalising them to be able to process them with NLP tools is a NLP challenge in itself and fully discussing it goes beyond the scope of this course. However, you may find the following modules useful in your project:

tweet-preprocessor: This library makes it easy to clean, parse or tokenize the tweets. It supports cleaning, tokenizing and parsing of URLs, hashtags, reserved words, mentions, emojis and smileys.
emot: Emot is a python library to extract the emojis and emoticons from a text (string). All the emojis and emoticons are taken from a reliable source, i.e. Wikipedia.org.
autocorrect: Spelling corrector (Python 3).
html: Can be used to remove HTML tags.
chardet: Universal encoding detector for Python 2 and 3.
ftfy: Fixes broken unicode strings.

If you are interested in reading more about these topic, these papers discuss preprocessing and normalization:

Assessing the Consequences of Text Preprocessing Decisions (Denny & Spirling 2016). This paper is a bit long, but it provides a nice discussion of common preprocessing steps and their potential effects.
What to do about bad language on the internet (Eisenstein 2013). This is a quick read that we recommend everyone to at least look through.

And here is a nice blog about character encoding.
Exercises
Can you write Python code for it?

import spacy
nlp = spacy.load('en_core_web_sm')


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
Exercise 3:
Remember how we can use the os and glob modules to process multiple files? For example, we can read all .txt files in the dreams folder like this:
Can you write Python code for it?

import glob
filenames = glob.glob("../Data/dreams/*.txt")
print(filenames)


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
Exercise 4:
Create a function called get_sentences_with_keyword that takes one positional parameter filenames and one keyword parameter filenames with default value None. It should read in all filenames and return a list called sentences that contains all sentences (the complete texts) with the keyword.
Hints:
- It's best to check for the lemmas of each token
- Lowercase both your keyword and the lemma
Can you write Python code for it?

import glob
filenames = glob.glob("../Data/dreams/*.txt")
print(filenames)

def get_sentences_with_keyword(filenames, keyword=None):
    #your code here

# test your function here
sentences = get_sentences_with_keyword(filenames, keyword="toy")
print(sentences)
assert len(sentences) == 4 # if your code is correct, this should not raise an error


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
First of all, text simply gives you the whole document as a string:
Can you write Python code for it?


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
If you find this difficult to comprehend, you can also simply convert it to a list and then loop over the list. Remember that this is less efficient, though.
Can you write Python code for it?

# You can also store the sentences in a list and then loop over the list
sentences = list(doc.sents)
for sentence in sentences:
    print(sentence, type(sentence))


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
Next, you will want to define which annotators to use and which output format should be produced (text, json, xml, conll, conllu, serialized). Annotating the document then is very easy. Note that Stanford CoreNLP uses some large models that can take a long time to load. You can read more about it here.
Can you write Python code for it?

harry_potter = "Harry Potter is a series of fantasy novels written by British author J. K. Rowling.\
The novels chronicle the life of a young wizard, Harry Potter, and his friends Hermione Granger and Ron Weasley,\
all of whom are students at Hogwarts School of Witchcraft and Wizardry.\
The main story arc concerns Harry's struggle against Lord Voldemort, a dark wizard who intends to become immortal,\
overthrow the wizard governing body known as the Ministry of Magic, and subjugate all wizards and Muggles."

# Define annotators and output format
properties= {'annotators': 'tokenize, ssplit, pos, lemma, parse',
             'outputFormat': 'json'}

# Annotate the string with CoreNLP
doc = nlp.annotate(harry_potter, properties=properties)


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
Notice that some of the attributes end with an underscore. For example, tokens have both lemma and lemma_ attributes. The lemma attribute represents the id of the lemma (integer), while the lemma_ attribute represents the unicode string representation of the lemma. In practice, you will mostly use the lemma_ attribute.
Can you write Python code for it?

for token in doc:
    print(token.lemma, token.lemma_)


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
Now create a function called get_vocabulary that takes one positional parameter filenames. It should read in all filenames and return a set called unique_words, that contains all unique words in the files.
Can you write Python code for it?

def get_vocabulary(filenames):
    # your code here

# test your function here
unique_words = get_vocabulary(filenames)
print(unique_words, len(unique_words))
assert len(unique_words) == 415 # if your code is correct, this should not raise an error


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
Pretty cool, but what does NORP mean? Again, you can use spacy.explain() to find out:
3. EXTRA: Stanford CoreNLP
Another very popular NLP pipeline is Stanford CoreNLP. You can use the tool from the command line, but there are also some useful Python wrappers that make use of the Stanford CoreNLP API, such as pycorenlp. As you might want to use this in the future, we will provide you with a quick start guide. To use the code below, you will have to do the following:

Download Stanford CoreNLP here.
Install pycorenlp (run pip install pycorenlp in your terminal, or simply run the cell below).
Open a terminal and run the following commands (replace with the correct directory names):
cd LOCATION_OF_CORENLP/stanford-corenlp-full-2018-02-27
java -mx4g -cp "*" edu.stanford.nlp.pipeline.StanfordCoreNLPServer
   This step you will always have to do if you want to use the Stanford CoreNLP API.
Can you write Python code for it?

%%bash
pip install pycorenlp

from pycorenlp import StanfordCoreNLP
nlp = StanfordCoreNLP('http://localhost:9000')


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
Similarly, noun_chunks can be used to create a generator for all noun chunks in the text.
Can you write Python code for it?

# Get all the noun chunks as a generator
print(doc.noun_chunks, type(doc.noun_chunks))

# You can loop over a generator; each noun chunk is a span of tokens
for chunk in doc.noun_chunks:
    print(chunk, type(chunk))
    print()


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
Text, sentences and noun_chunks
If you call the dir() function on a Doc object, you will see that it has a range of methods and attributes. You can read more about them in the documentation. Below, we highlight three of them: text, sents and noun_chunks.
Can you write Python code for it?


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
The benefit of converting it to a list is that we can use indices to select certain sentences. For example, in the following we only print some information about the tokens in the second sentence.
Can you write Python code for it?

# Print some information about the tokens in the second sentence.
sentences = list(doc.sents)
for token in sentences[1]:
    data = '\t'.join([token.orth_,
                      token.lemma_,
                      token.pos_,
                      token.tag_,
                      str(token.i),    # Turn index into string
                      str(token.idx)]) # Turn index into string
    print(data)


I am working on the file "Chapters/Chapter 19 - More about Natural Language Processing Tools (spaCy).ipynb".
The first task is:
nlp is now a Python object representing the English NLP pipeline that we can use to process a text.
EXTRA: Larger models
For English, there are three models ranging from 'small' to 'large':

en_core_web_sm
en_core_web_md
en_core_web_lg

By default, the smallest one is loaded. Larger models should have a better accuracy, but take longer to load. If you like, you can use them instead. You will first need to download them.
Can you write Python code for it?

#%%bash
#python -m spacy download en_core_web_md

#%%bash
#python -m spacy download en_core_web_lg

# uncomment one of the lines below if you want to load the medium or large model instead of the small one
# nlp = spacy.load('en_core_web_md')
# nlp = spacy.load('en_core_web_lg')


I am working on the file "Chapters/Chapter 20 - Visualization and Statistics.ipynb".
The first task is:
2.3. Common plots
Example 1: Line plot Let's create our first (line) plot:
Can you write Python code for it?


I am working on the file "Chapters/Chapter 20 - Visualization and Statistics.ipynb".
The first task is:
5. Take home message: The steps of visualization
Now you've seen several different plots, hopefully the general pattern is becoming clear: visualization typically consists of three steps:

Load the data.
Organize the data in such a way that you can feed it to the visualization function.
Plot the data using the function of your choice.

There's also an optional fourth step: After plotting the data, tweak the plot until you're satisfied. Of these steps, the second and fourth are usually the most involved.
6. Optional: On your own
If you would like to practice, here is an exercise with data from Donald Trump's Facebook page. The relevant file is Data/Trump-Facebook/FacebookStatuses.tsv. Try to create a visualization that answers one of the following questions:

How does the number of responses to Trump's posts change over time?
What webpages does Donald Trump link to, and does this change over time? Which is the most popular? Are there any recent newcomers?
What entities does Trump talk about?
Starting March 2016 (when the emotional responses were introduced on Facebook), how have the emotional responses to Trumps messages developed?
[Question of your own.]

Try to at least think about what kind of visualization might be suitable to answer these questions, and we'll discuss this question in class on Monday. More specific questions:

What kind of preprocessing is necessary before you can start visualizing the data?
What kind of visualization is suitable for answering these questions?
What sort of chart would you choose?
How could you use color to improve your visualization?

What might be difficult about visualizing this data? How could you overcome those difficulties?
Can you write Python code for it?

# Open the data.

# Process the data so that it can be visualized.

# Plot the data.

# Modify the plot.


I am working on the file "Chapters/Chapter 20 - Visualization and Statistics.ipynb".
The first task is:
Because Dodds et al. collected data from several languages, we can plot the distributions for multiple languages and see whether they all have normally distributed scores. We will do this with a Kernal Density Estimation plot. Basically, such a plot shows you the probability distribution (the chance of getting a particular score) as a continuous line. Because it's a line rather than a set of bars, you can show many of them in the same graph.
Can you write Python code for it?

# This is necessary to get all the separate files.
import glob

# Get all the score files.
filenames = glob.glob('../Data/Dodds2014/data/labMTscores-*.csv')

# Showing the first 5, because else you can't keep track of all the lines.
for filename in filenames[:5]:
    # Read the language from the filename
    language = filename.split('-')[1]
    language = language.split('.')[0]
    with open(filename) as f:
        scores = [float(line.strip()) for line in f]
        scores_array = np.array(scores) # This is necessary because the kdeplot function only accepts arrays.
        sns.kdeplot(scores_array, label=language)

plt.legend()


I am working on the file "Chapters/Chapter 20 - Visualization and Statistics.ipynb".
The first task is:
If all went alright, you see a graph above this block. Try changing the numbers in the vals list to see how it affects the graph. Plotting is as simple as that!
Example 2: Column chart Now, let's try plotting some collected data. Suppose we did a survey to ask people for their favorite pizza. We store the result in a dictionary:
Can you write Python code for it?

counts = {
    'Calzone': 63,
    'Quattro Stagioni': 43,
    'Hawaii': 40,
    'Pepperoni': 58,
    'Diavolo': 63,
    'Frutti di Mare': 32,
    'Margarita': 55,
    'Quattro Formaggi': 10,
}


I am working on the file "Chapters/Chapter 20 - Visualization and Statistics.ipynb".
The first task is:
Look at all those unimodal distributions (with a single peak)!
4.2. A concreteness dataset
We'll work with another data file by Brysbaert and colleagues, consisting of concreteness ratings. I.e. how abstract or concrete participants think a given word is.
Can you write Python code for it?

import csv
# Let's load the data first.
concreteness_entries = []
with open('../Data/concreteness/Concreteness_ratings_Brysbaert_et_al_BRM.txt') as f:
    reader = csv.DictReader(f, delimiter='\t')
    for entry in reader:
        entry['Conc.M'] = float(entry['Conc.M'])
        concreteness_entries.append(entry)


I am working on the file "Chapters/Chapter 20 - Visualization and Statistics.ipynb".
The first task is:
Note on bar/column plots: while they're super useful, don't use them to visualize distributions. There was even a Kickstarter to raise money for sending T-shirts with a meme image to the editorial boards of big journals!
3. Correlation
Let's look at correlation between values in Python. We'll explore two measures: Pearson and Spearman correlation. Given two lists of numbers, Pearson looks whether there is any linear relation between those numbers. This is contrasted by the Spearman measure, which aims to see whether there is any monotonic relation. The difference between linear and monotonic is that the latter is typically less strict:

Monotonic: a constant relation between two lists of numbers.
if a number in one list increases, so does the number in the other list, or
if a number in one list increases, the number in the other list decreases.

Linear: similar to monotonic, but the increase or decrease can be modeled by a straight line.

Here is a small example to illustrate the difference.
Can you write Python code for it?

# Scipy offers many statistical functions, among which the Pearson and Spearman correlation measures.
from scipy.stats import pearsonr, spearmanr

# X is equal to [1,2,3,...,99,100]
x = list(range(100))

# Y is equal to [1^2, 2^2, 3^2, ..., 99^2, 100^2]
y = [i**2 for i in x]

# Z is equal to [100,200,300, ..., 9900, 10000]
z = [i*100 for i in x]

# Plot x and y.
plt.plot(x, y, label="X and Y")

# Plot y and z in the same plot.
plt.plot(x, z, label="X and Z")

# Add a legend.
plt.legend(loc='upper left')

correlation, significance = pearsonr(x,y)
print('The Pearson correlation between X and Y is:', correlation)

correlation, significance = spearmanr(x,y)
print('The Spearman correlation between X and Y is:', correlation)

print('----------------------------------------------------------')

correlation, significance = pearsonr(x,z)
print('The Pearson correlation between X and Z is:', correlation)

correlation, significance = spearmanr(x,z)
print('The Spearman correlation between X and Z is:', correlation)


I am working on the file "Chapters/Chapter 20 - Visualization and Statistics.ipynb".
The first task is:
This loop processes the dictionary into a format that's easy to send to matplotlib - a list of pizza names (for the labels on the bars) and a list of vote counts (for the actual graph.)
Can you write Python code for it?

names = []
votes = []
# Split the dictionary of names->votes into two lists, one holding names and the other holding vote counts
for pizza in counts:
    names.append(pizza)
    votes.append(counts[pizza])


I am working on the file "Chapters/Chapter 20 - Visualization and Statistics.ipynb".
The first task is:
We can use a simple command from another package, Seaborn, to make all Matplotlib plots look prettier!
This import and the next command change the Matplotlib defaults for styling.
Can you write Python code for it?

import seaborn as sns
sns.set_style("whitegrid")


I am working on the file "Chapters/Chapter 20 - Visualization and Statistics.ipynb".
The first task is:
We create a range of indexes for the X values in the graph, one entry for each entry in the "counts" dictionary (ie len(counts)), numbered 0,1,2,3,etc. This will spread out the graph bars evenly across the X axis on the plot.
np.arange is a NumPy function like the range() function in Python, only the result it produces is a "NumPy array". We'll see why this is useful in a second.
plt.bar() creates a column graph, using the "x" values as the X axis positions and the values in the votes array (ie the vote counts) as the height of each bar. Finally, we add the labels, rotated with a certain angle.
Can you write Python code for it?

import numpy as np

x = np.arange(len(counts))
print(x)
plt.bar(x, votes)
plt.xticks(x, names, rotation=60)
plt.yticks(votes)


I am working on the file "Chapters/Chapter 21 - Tables and Networks.ipynb".
The first task is:
1. Tables
There are (at least) two ways to output your data as a formatted table:

Using the tabulate package. (You might need to install it first, using conda install tabulate)
Using the pandas dataframe method df.to_latex(...), df.to_string(...), or even df.to_clipboard(...).

This is extremely useful if you're writing a paper. First version of the 'results' section: done!
Option 1: Tabulate
Can you write Python code for it?

from tabulate import tabulate

table = [["spam",42],["eggs",451],["bacon",0]]
headers = ["item", "qty"]

# Documentation: https://pypi.python.org/pypi/tabulate
print(tabulate(table, headers, tablefmt="latex_booktabs"))


I am working on the file "Chapters/Chapter 21 - Tables and Networks.ipynb".
The first task is:
Networks are made up out of edges: connections between nodes (also called vertices). To build a graph of the WordNet-taxonomy, we need to generate a set of edges. This is what the function below does.
Can you write Python code for it?

def hypernym_edges(synset):
    """
    Function that generates a set of edges
    based on the path between the synset and entity.n.01
    """
    edges = set()
    for path in synset.hypernym_paths():
        synset_names = [s.name() for s in path]
        # bigrams turns a list of arbitrary length into tuples: [(0,1),(1,2),(2,3),...]
        # edges.update adds novel edges to the set.
        edges.update(bigrams(synset_names))
    return edges

# Use the synset 'dog.n.01'
dog = wn.synset('dog.n.01')

# Generate a set of edges connecting the synset for 'dog' to the root node (entity.n.01)
edges = hypernym_edges(dog)

# Create a graph object.
G = nx.Graph()

# Add all the edges that we generated earlier.
G.add_edges_from(edges)


I am working on the file "Chapters/Chapter 21 - Tables and Networks.ipynb".
The first task is:
Now we can actually start drawing the graph. We'll increase the figure size, and use the draw_spring method (that implements the Fruchterman-Reingold layout algorithm).
Can you write Python code for it?

# Increasing figure size for better display of the graph.
from pylab import rcParams
rcParams['figure.figsize'] = 11, 11

# Draw the actual graph.
nx.draw_spring(G,with_labels=True)


I am working on the file "Chapters/Chapter 21 - Tables and Networks.ipynb".
The first task is:
Once you've produced your LaTeX table, it's almost ready to put in your paper. If you're writing an NLP paper and your table contains scores for different system outputs, you might want to make the best scores bold, so that they stand out from the other numbers in the table.
More to explore
The pandas library is really useful if you work with a lot of data (we'll also use it below). As Jake Vanderplas said in the State of the tools video, the pandas DataFrame is becoming the central format in the Python ecosystem. Here is a page with pandas tutorials.
2. Networks
Some data is best visualized as a network. There are several options out there for doing this. The easiest is to use the NetworkX library and either plot the network using Matplotlib, or export it to JSON or GEXF (Graph EXchange Format) and visualize the network using external tools.
Let's explore a bit of WordNet today. For this, we'll want to import the NetworkX library, as well as the WordNet module. We'll look at the first synset for dog: dog.n.01, and how it's positioned in the WordNet taxonomy. All credits for this idea go to this blog.
Can you write Python code for it?

import networkx as nx # You might need to install networkx first (conda install -c anaconda networkx)
from nltk.corpus import wordnet as wn
from nltk.util import bigrams # This is a useful function.


I am working on the file "Chapters/Chapter 21 - Tables and Networks.ipynb".
The first task is:
What is interesting about this is that there is a cycle in the graph! This is because dog has two hypernyms, and those hypernyms are both superseded (directly or indirectly) by animal.n.01.
What is not so good is that the graph looks pretty ugly: there are several crossing edges, which is totally unnecessary. There are better layouts implemented in NetworkX, but they do require you to install pygraphviz. Once you've done that, you can execute the next cell. (And if not, then just assume it looks much prettier!)
Can you write Python code for it?

# Install pygraphviz first: pip install pygraphviz
from networkx.drawing.nx_agraph import graphviz_layout

# Let's add 'cat' to the bunch as well.
cat = wn.synset('cat.n.01')
cat_edges = hypernym_edges(cat)
G.add_edges_from(cat_edges)

# Use the graphviz layout. First compute the node positions..
positioning = graphviz_layout(G)

# And then pass node positions to the drawing function.
nx.draw_networkx(G,pos=positioning)


I am working on the file "Chapters/Chapter 22 - Sentiment analysis with VADER.ipynb".
The first task is:
2. Load VADER model
The model can be loaded in the following way.
Can you write Python code for it?

from nltk.sentiment.vader import SentimentIntensityAnalyzer

vader_model = SentimentIntensityAnalyzer()


I am working on the file "Chapters/Chapter 22 - Sentiment analysis with VADER.ipynb".
The first task is:
The next for loop assigns a sentiment score from VADER to each sentence.
Can you write Python code for it?

for sent in sentences:
    scores = vader_model.polarity_scores(sent)
    print()
    print('INPUT SENTENCE', sent)
    print('VADER OUTPUT', scores)


I am working on the file "Chapters/Chapter 22 - Sentiment analysis with VADER.ipynb".
The first task is:
The next function defines an API to process texts (textual_unit) using different settings. This function operates on texts and assumes spaCy is loaded with the corresponding language model as we just did. Take a little time to analyze the function, which uses certain spaCy token properties to process the text in different ways and returns the VADER sentiment.
Can you write Python code for it?

def run_vader(nlp,
              textual_unit,
              lemmatize=False,
              parts_of_speech_to_consider=set(),
              verbose=0):
    """
    Run VADER on a sentence from spacy

    :param nlp: spaCy model
    :param str textual unit: a textual unit, e.g., sentence, sentences (one string)
    (by looping over doc.sents)
    :param bool lemmatize: If True, provide lemmas to VADER instead of words
    :param set parts_of_speech_to_consider:
    -empty set -> all parts of speech are provided
    -non-empty set: only these parts of speech are considered
    :param int verbose: if set to 1, information is printed
    about input and output

    :rtype: dict
    :return: vader output dict
    """
    doc = nlp(textual_unit)

    input_to_vader = []

    for sent in doc.sents:
        for token in sent:

            if verbose >= 2:
                print(token, token.pos_)

            to_add = token.text

            if lemmatize:
                to_add = token.lemma_

                if to_add == '-PRON-':
                    to_add = token.text

            if parts_of_speech_to_consider:
                if token.pos_ in parts_of_speech_to_consider:
                    input_to_vader.append(to_add)
            else:
                input_to_vader.append(to_add)

    scores = vader_model.polarity_scores(' '.join(input_to_vader))

    if verbose >= 1:
        print()
        print('INPUT SENTENCE', sent)
        print('INPUT TO VADER', input_to_vader)
        print('VADER OUTPUT', scores)

    return scores


I am working on the file "Chapters/Chapter 22 - Sentiment analysis with VADER.ipynb".
The first task is:
This is even more interesting. The part of speech label VERB is not applied to any of the tokens ('s is labeled as auxiliary and not with the label VERB). We have not provided VADER with input at all!
Let's also try adjectives:
Can you write Python code for it?

run_vader(nlp,
          sentences[1],
          lemmatize=True,
          parts_of_speech_to_consider={'ADJ'},
          verbose=1)


I am working on the file "Chapters/Chapter 22 - Sentiment analysis with VADER.ipynb".
The first task is:
To verify that the download was successful, you can run the following command.
Can you write Python code for it?


I am working on the file "Chapters/Chapter 22 - Sentiment analysis with VADER.ipynb".
The first task is:
VADER provides a dictionary containing four ratings, i.e., keys, for each sentence.
The sentence is rated on how negative (key neg), positive (key pos), and neutral (key neu), it is.
Also, there is a compound key that combines the values of the keys neg, pos, and neu into one single score, i.e., the compound key. The compound value ranges from -1, i.e., very negative, to 1, i.e., very positive. You can read more about the VADER system on this blog.
3. Using spaCy to manipulate the input to VADER
In the examples in Section 2, VADER always takes into account each token, i.e., word, in the sentence to arrive at its sentiment prediction. In this section,
we are going to use spaCy to manipulate the input to VADER. This is one way to gain insight into how language systems work, i.e., by manipulating the input and inspecting the output.
We use spaCy as a tool to manipulate the input to VADER.
Please first install spaCy by following the instructions from Chapter 19 - More about Natural Language Processing Tools (spaCy) -- Section 2.1 Installing and loading spaCy
Can you write Python code for it?

import spacy
nlp = spacy.load('en') # en_core_web_sm


I am working on the file "Chapters/Three/Feature Extraction.ipynb".
The first task is:
Data standardization
Can you write Python code for it?

from sklearn import preprocessing
import numpy as np

X = np.array()


I am working on the file "Chapters/Three/Feature Extraction.ipynb".
The first task is:
Extracting features from images
Extracting features from pixel intensities
A digital image is usually a raster, or pixmap, that maps colors to coordinates on
a grid. An image can be viewed as a matrix in which each element represents a color. A basic feature representation for an image can be constructed by reshaping the matrix into a vector by concatenating its rows together. Optical character recognition (OCR) is a canonical machine learning problem. Let's use this technique to create basic feature representations that could be used in an OCR application for recognizing hand-written digits in character-delimited forms.
The digits dataset included with scikit-learn contains grayscale images of more than 1,700 hand-written digits between zero and nine. Each image has eight pixels on a side. Each pixel is represented by an intensity value between zero and 16; white is the most intense and is indicated by zero, and black is the least intense and is indicated by 16. The following  gure is an image of a hand-written digit taken from the dataset:
Can you write Python code for it?

from sklearn import datasets

digits = datasets.load_digits()

print("Digit: ", digits.target[0])
print(digits.images[0])


I am working on the file "Chapters/Three/Feature Extraction.ipynb".
The first task is:
Extracting points of interest as features
The feature vector we created previously represents every pixel in the image; all of the informative attributes of the image are represented and all of the noisy attributes are represented too. After inspecting the training data, we can see that all of the images have a perimeter of white pixels; these pixels are not useful features. Humans can quickly recognize many objects without observing every attribute of the object. We can recognize a car from the contours of the hood without observing the rear-view mirrors, and we can recognize an image of a human face from a nose or mouth. This intuition is motivation to create representations of only the most informative attributes of an image. These informative attributes, or points of interest, are points that are surrounded by rich textures and can be reproduced despite perturbing the image. Edges and corners are two common types of points of interest. An edge is a boundary at which pixel intensity rapidly changes, and a corner is an intersection of two edges. Let's use scikit-image to extract points of interest from the following  gure:
Can you write Python code for it?

import numpy as np
import skimage

from skimage.feature import corner_harris, corner_peaks
from skimage.color import rgb2gray
import matplotlib.pyplot as plt
%matplotlib inline

import skimage.io as io

from skimage.exposure import equalize_hist

def show_corners(corners, image):
    fig = plt.figure()
    plt.gray()
    plt.imshow(image)
    y_corner, x_corner = zip(*corners)
    plt.plot(x_corner, y_corner, "or")
    plt.xlim(0, image.shape[1])
    plt.ylim(image.shape[0],0)
    fig.set_size_inches(np.array(fig.get_size_inches()) * 1.5)
    plt.show()

mandrill = io.imread('mandril.jpeg')

mandrill = equalize_hist(rgb2gray(mandrill))

corners = corner_peaks(corner_harris(mandrill), min_distance=2)
show_corners(corners, mandrill)


I am working on the file "Chapters/Three/Feature Extraction.ipynb".
The first task is:
This corpus contains eight unique words: UNC, played, Duke, in, basketball, lost, the, and game. The corpus's unique words comprise its vocabulary. The bag-of-words model uses a feature vector with an element for each of the words in the corpus's vocabulary to represent each document. Our corpus has eight unique words, so each document will be represented by a vector with eight elements. The number of elements that comprise a feature vector is called the vector's dimension. A dictionary maps the vocabulary to indices in the feature vector.
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer()
print(vectorizer.fit_transform(corpus).todense())
print(vectorizer.vocabulary_)

 corpus = [
       'UNC played Duke in basketball',
       'Duke lost the basketball game',
       'I ate a sandwich'
]

vectorizer = CountVectorizer()
print(vectorizer.fit_transform(corpus).todense())
print(vectorizer.vocabulary_)

from sklearn.metrics.pairwise import euclidean_distances
#Transform it to list
counts = vectorizer.fit_transform(corpus).todense().tolist()

#Check Euclidean distances
print("Distance between 1st and 2nd documents: ", euclidean_distances(counts[0],counts[1]))
print("Distance between 2st and 3nd documents: ", euclidean_distances(counts[1],counts[2]))
print("Distance between 1st and 3nd documents: ", euclidean_distances(counts[2],counts[0]))


I am working on the file "Chapters/Three/Feature Extraction.ipynb".
The first task is:
Use stop_words method to filter words with little meaning such as "the", "a", "an" etc
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
corpus = [
     'UNC played Duke in basketball',
     'Duke lost the basketball game',
     'I ate a sandwich'
]

vectorizer = CountVectorizer(stop_words='english')
print(vectorizer.fit_transform(corpus).todense())
print(vectorizer.vocabulary_)

counts = vectorizer.fit_transform(corpus).todense().tolist()

#Check Euclidean distances
print("Distance between 1st and 2nd documents: ", euclidean_distances(counts[0],counts[1]))
print("Distance between 2st and 3nd documents: ", euclidean_distances(counts[1],counts[2]))
print("Distance between 1st and 3nd documents: ", euclidean_distances(counts[2],counts[0]))


I am working on the file "Chapters/Three/Feature Extraction.ipynb".
The first task is:
Using the parts of speech of gathering, NLTK's WordNetLemmatizer correctly
lemmatizes the words in both documents as shown in the following example:
Can you write Python code for it?

lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize('gathering','v'))
print(lemmatizer.lemmatize('gathering','n'))


I am working on the file "Chapters/Two/Simple Linear Regression.ipynb".
The first task is:
Cross validation
Can you write Python code for it?

#Make cross validation
from sklearn.cross_validation import cross_val_score
scores = cross_val_score(regressor, X, y, cv = 5)

print(scores.mean(), scores)


I am working on the file "Chapters/Two/Simple Linear Regression.ipynb".
The first task is:
Evaluate fit using r square
Can you write Python code for it?

#Load another set
X_test = np.array([8,9,11,16,12])
Y_test = np.array([11,8.5,15,18,11])

model.fit(X_test.reshape(-1,1),Y_test)
model.predict(X_test.reshape(-1,1))

def total_sum_squares(Y):
    return np.sum((Y_test - np.mean(Y_test))**2)

#Residual sum of squares
def residual_sum_squares(Y):
    return np.sum( (Y_test - model.predict(X_test.reshape(-1,1)))**2)

#Get R square

1 - residual_sum_squares(Y_test)/total_sum_squares(Y_test)

#From sklearn
model.score(X_test.reshape(-1,1),Y_test)


I am working on the file "Chapters/Two/Simple Linear Regression.ipynb".
The first task is:
Evaluate fitness of model using cost (lost) function
Cost function for simple linear regression is the sum total of residuals, called RSS
Can you write Python code for it?

import numpy as np

print("Residual sum of squares: %.2f" %np.mean((model.predict(X)-Y)**2))