I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
1.8 Storing a function in a Python module
Since Python functions are nice blocks of code with a clear focus, wouldn't it be nice if we can store them in a file? By doing this, we make our code visually very appealing since we are only left with functions calls instead of function definitions.
Please open the file utils_chapter11.py (is in the same folder as the notebook you are now reading). In it, you will find three of the functions that we've shown so far in this notebook. So, how can we use those functions? We can import the function using the following syntax:
from NAME OF FILE WITHOUT .PY import function name
Can you write Python code for it?

from utils_chapter11 import happy_birthday

happy_birthday('George')

from utils_chapter11 import multiply

multiply(1,2)

from utils_chapter11 import is_even

is_it_even = is_even(5)
print(is_it_even)


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
Also consider this:
Can you write Python code for it?

x = 0
def setx():
    """Set the value of a variable to 1."""
    x = 1
setx()
print(x)


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
Chapter 11: Functions and scope
We use an example from this website to show you some of the basics of writing a function.
We use some materials from this other Python course.
We have seen that Python has several built-in functions (e.g. print() or max()). But you can also create a function. A function is a reusable block of code that performs a specific task. Once you have defined a function, you can use it at any place in your Python script. You can even import a function from an external module (as we will see in the next chapter). Therefore, they are beneficial for tasks that you will perform more often. Plus, functions are a convenient way to order your code and make it more readable!
At the end of this chapter, you will be able to:

write a function
work with function inputs
understand the difference between (keyword and positional) arguments and parameters
return zero, one, or multiple values
write function docstrings
understand the scope of variables
store your function in a Python module and call it
debug your functions

If you want to learn more about these topics, you might find the following link useful:

Tutorial: Defining Functions of your Own
The docstrings main formats
PEP 287 -- reStructured Docstring Format
Introduction to assert

Now let's get started!
If you have questions about this chapter, please contact us (cltl.python.course@gmail.com).
1. Writing a function
A function is an isolated chunk of code that has a name, gets zero or more parameters, and returns a value. In general, a function will do something for you based on the input parameters you pass it, and it will typically return a result. You are not limited to using functions available in the standard library or the ones provided by external parties. You can also write your own functions!
Whenever you are writing a function, you need to think of the following things:
* What is the purpose of the function?
* How should I name the function?
* What input does the function need?
* What output should the function generate?
1.1. Why...
Can you write Python code for it?

print("Happy Birthday to you!")
print("Happy Birthday to you!")
print("Happy Birthday, dear Emily.")
print("Happy Birthday to you!")


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
Exercise 2:
Add another keyword parameter message to the multiply function, which will allow a user to print a message. The default value of this keyword parameter should be an empty string. Test this with 2 messages of your choice. Also test it without specifying a value for the keyword argument when calling a function.
Can you write Python code for it?

# function to modify:

def multiply(x, y, third_number=1):
    """Multiply two or three numbers and print the result."""
    result=x*y*third_number
    print(result)


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
Exercise 6:
Complete this code to switch the values of two variables:
Can you write Python code for it?

def switch_two_values(x,y):
# your code here

a='orange'
b='apple'

a,b = switch_two_values(a,b) # `a` should contain "apple" after this call, and `b` should contain "orange"

print(a,b)


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
However, when the actual output is different from what we expected, we got an error. Let's say we made a mistake in writing the function.
Can you write Python code for it?

def is_even(p):
    """Check whether a number is even."""
    if p % 2 == 1:
        return False
    else:
        return False

input_value = 2
expected_output = True
actual_output = is_even(input_value)
assert actual_output == expected_output, f'expected {expected_output}, got {actual_output}'


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
If we assign the result to a variable, but do not use the return statement, the function cannot return it. Instead, it returns None (as you can try out below).
This is important to realize: even functions without a return statement do return a value, albeit a rather boring one. This value is called None (it’s a built-in name). You have seen this already with list methods - for example list.append(val) adds a value to a list, but does not return anything explicitly.
Can you write Python code for it?

def multiply_no_return(x, y):
    """Multiply two numbers and does not return the result."""
    result = x * y

is_this_a_result = multiply_no_return(2,3)
print(is_this_a_result)


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
If we do not specify a value for a positional parameter, the function call will fail (with a very helpful error message):
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
If we execute the code above, we don't get any output. That's because we only told Python: "Here's a function to do this, please remember it." If we actually want Python to execute everything inside this function, we have to call it:
1.3 How to call a function
It is important to distinguish between a function definition and a function call. We illustrate this in 1.3.1. You can also call functions from within other functions. This will become useful when you split up your code into small chunks that can be combined to solve a larger problem. This is illustrated in 1.3.2.
1.3.1) A simple function call
A function is defined once. After the definition, Python has remembered what this function does in its memory.
A function is executed/called as many times as we like. When calling a function, you should always use parenthesis.
Can you write Python code for it?

# function definition:

def happy_birthday_to_emily(): # Function definition
    """
    Print a birthday song to Emily.
    """
    print("Happy Birthday to you!")
    print("Happy Birthday to you!")
    print("Happy Birthday, dear Emily.")
    print("Happy Birthday to you!")

# function call:

print('Function call 1')

happy_birthday_to_emily()

print()
# We can call the function as many times as we want (but we define it only once)
print('Function call 2')

happy_birthday_to_emily()

print()

print('Function call 3')

happy_birthday_to_emily()

print()
# This will not call the function

print('This is not a function call')
happy_birthday_to_emily


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
In fact, this code has produced two completely unrelated x's!
So, you can not read a local variable outside of the local context. Nevertheless, it is possible to read a global variable from within a function, in a strictly read-only fashion.
Can you write Python code for it?

x = 1
def getx():
    """Print the value of a variable x."""
    print(x)

getx()


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
Returning multiple values
Similarly as the input, a function can also return multiple values as output. We call such a collection of values a tuple (does this term sound familiar ;-)?).
Can you write Python code for it?

def calculate(x,y):
    """Calculate product and sum of two numbers."""
    product = x * y
    summed = x + y

    #we return a tuple of values
    return product, summed

# the function returned a tuple and we unpack it to var1 and var2
var1, var2 = calculate(10,5)

print("product:",var1,"sum:",var2)


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
The help we get on a function will become more interesting once we learn about function inputs and outputs ;-)
1.4 Working with function input
1.4.1 Parameters and arguments
We use parameters and arguments to make a function execute a task depending on the input we provide. For instance, we can change the function above to input the name of a person and print a birthday song using this name. This results in a more generic function.
To understand how we use parameters and arguments, keep in mind the distinction between function definition and function call.
Parameter: The variable name in the function definition below is a parameter. Variables used in function definitions are called parameters.
Argument: The variable my_name in the function call below is a value for the parameter name at the time when the function is called. We refer to such variables as arguments. We use arguments so we can direct the function to do different kinds of work when we call it at different times.
Can you write Python code for it?

# function definition with using the parameter `name'
def happy_birthday(name):
    """
    Print a birthday song with the "name" of the person inserted.
    """
    print("Happy Birthday to you!")
    print("Happy Birthday to you!")
    print(f"Happy Birthday, dear {name}.")
    print("Happy Birthday to you!")

# function call using specifying the value of the argument
happy_birthday("James")


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
This could be the purpose of a function: to print the lines of a birthday song for Emily.
Now, we define a function to do this. Here is how you define a function:

write def;
the name you would like to call your function;
a set of parentheses containing the parameter(s) of your function;
a colon;
a docstring describing what your function does;
the function definition;
ending with a return statement

Statements must be indented so that Python knows what belongs in the function and what not. Functions are only executed when you call them. It is good practice to define your functions at the top of your program or in another Python module.
We give the function a clear name, happy_birthday_to_emily, and we define the function as shown below. Note that we specify what it does in the docstring at the beginning of the function:
Can you write Python code for it?

def happy_birthday_to_emily(): # Function definition
    """
    Print a birthday song to Emily.
    """
    print("Happy Birthday to you!")
    print("Happy Birthday to you!")
    print("Happy Birthday, dear Emily.")
    print("Happy Birthday to you!")


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
We can also store the name in a variable:
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
You can do the same tricks that we learnt to apply on the built-in functions, like asking for help or for a function type:
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_11_Functions_and_scope.ipynb".
The first task is:
You can see that this docstring describes the function goal, its parameters, its outputs, and the errors it raises.
It is a good practice to write a docstring for your functions, so we will always do this! For now we will stick with  single-sentence docstrings
You can read more about this topic here, here, and here.
1.7 Debugging a function
Sometimes, it can hard to write a function that works perfectly. A common practice in programming is to check whether the function performs as you expect it to do. The assert statement is one way of debugging your function. The syntax is as follows:
assert code == your expected output,message to show when code does not work as you'd expected
Let's try this on our simple function.
Can you write Python code for it?

def is_even(p):
    """Check whether a number is even."""
    if p % 2 == 1:
        return False
    else:
        return True


I am working on the file "Chapters-colab/Chapter_12_Importing_external_modules.ipynb".
The first task is:
3.2 Requests
Can you write Python code for it?

import requests
a=requests.get("https://tae898.github.io")
print(a.content)


I am working on the file "Chapters-colab/Chapter_12_Importing_external_modules.ipynb".
The first task is:
Chapter 12: Importing external modules
We use some materials from this other Python course.
Most of the functionality we have used thus far is built into the Python language itself. Now we have also learned how to write our own functions. Often, however, you need to use external modules in your code. A lot of external modules are already available in the Python Standard Library, for a wide variety of tasks. There are also countless third-party providers of Python modules.
At the end of this chapter, you will be able to:

import an entire module
import a specific function from a module
use dir and help to get information about a module or function
import a few commonly used modules: random, datetime, and requests

If you want to learn more about these topics, you might find the following links useful:
* Tutorial: The import statement
* Video: OS Module - Use Underlying Operating System Functionality
If you have questions about this chapter, please contact us (cltl.python.course@gmail.com).
1. Importing a module or a function
Python modules
A Python module is a Python file (i.e., a file ending in .py) which contains function definitions and statements. You can already write one yourself! (More about this in the next chapter.)
Some modules are part of the Python standard library and contain useful functions. You can import these modules and use the functions as explained below.
More on modules can be found here: https://docs.python.org/3/tutorial/modules.html
To use an external module in your code, you need to import it explicitely. Consider, for example, the module random from the standard library, which contains functions for generating random numbers:
Can you write Python code for it?

import random
print(random.randint(0, 100))


I am working on the file "Chapters-colab/Chapter_12_Importing_external_modules.ipynb".
The first task is:
Don't worry too much about the response: it is a format called HTML (which is basically behind all web pages) and one would need to write some more code to extract the useful information from it. But: it is really impressive that we can retrieve any webpage from the web with just three lines of code! If you want to have a quick look at how to extract content, feel free to check out the beautifulsoup module.
So, how does one find a useful module for a specific task? Answer: Google it ;-)
Exercises
Exercise 1:
Check which arguments should be supplied to the random.sample() method and what is their intention. Then call this function with different arguments.
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_12_Importing_external_modules.ipynb".
The first task is:
Exercise 2
Figure out what the os module does and think about tasks it could be used for.
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_12_Importing_external_modules.ipynb".
The first task is:
If we like the randint() method, we might also be interested which other methods are offered in the random package. We can use dir() for this:
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_12_Importing_external_modules.ipynb".
The first task is:
In this case we wouldn't have to specify where our machine should find the randint() function: note that we don't need to use the module random in the function call in this case.
2. Getting help
We might not understand how an external method works, or what input it needs, or what are the output values it returns. In that case, we can use help() (just as we did with functions and methods):
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_12_Importing_external_modules.ipynb".
The first task is:
Note the syntax used: using the dot, we indicate that our machine should look for the randint() method inside the random module we just imported. You can import an entire module or import only a specific function in the module. We could also have imported the (single) method we needed as follows:
Can you write Python code for it?

from random import randint
print(randint(0, 100))


I am working on the file "Chapters-colab/Chapter_13_Working_with_Python_files.ipynb".
The first task is:
Chapter 13 - Working with Python files
In the previous blocks, we've mainly used notebooks to develop and run our Python code. In this chapter, we'll introduce how to create Python modules (.py files) and how to run them. The most common way to work with Python is actually to use .py files, which is why it is important that you know how to work with them. You can see python files as one cell in a notebook without any markdown.
Before we write actual code in a .py file, we will explain the basics you need to know for doing this:

Choosing an editor
starting the terminal (from which you will run your .py files)

At the end of this chapter, you will be able to
* create python modules, i.e., .py files
* run python modules from the command line
If you have questions about this chapter, please contact us (cltl.python.course@gmail.com).
1. Editor
We first need to choose which editor we will use to develop our Python code.
There are two options.

You create the python modules in your browser. After opening Jupyter notebook, you can click File -> New and then Text file to start developing Python modules.

You install an editor.
Please take a look here to get an impression of which ones are out there.
We can highly recommend Atom (for macOS, Windows, Linux). Other options are BBEdit (for macOS) and Notepad++ (for Windows). A simple way to create a new .py file usually is to open a new file and save it as name_of_your_program.py (make sure to use indicative names).

Please choose between options 1 and 2.
2. Starting the terminal
To run a .py file we wrote in an editor, we need to start the terminal. This works differently for windows and Mac:

On Windows, please look at Anaconda Prompt
on OS X/macOS (Mac computer), please type terminal in Spotlight and start the terminal

It's a useful skill to know how to navigate through your computer (i.e., go from one directory to another, all the files and subdirectories in a directory, etc.) using the terminal.
For Windows users, this is a good tutorial.
For OS...
Can you write Python code for it?

import os
cwd = os.getcwd()
cwd_escaped_spaces = cwd.replace(' ', '\ ')
print('cd', cwd_escaped_spaces)


I am working on the file "Chapters-colab/Chapter_13_Working_with_Python_files.ipynb".
The first task is:
You've succesfully run your first Python program!
4. Import your own functions
In Chapter 12, you've been introduced to importing modules and functions/methods.
You can see any python program that you create (so any .py file) as a module, which means that you can import it into another python program. Let's see how this works.
Please note that the following examples only work if all your python files are in the same directory. There are ways of importing python modules from other directories, but we will not discuss them here.
4.1 Importing your entire module
When importing your own functions from your own modules, several things are important. We have created two example scripts to illustrate them called the_program.py and utils.py. We recommend to open them to check the following things:

The extension .py is not used when importing modules. import utils will import the file utils.py [line 1 the_program.py]
We can use any function from the file. We can call the count_words function by typing utils.count_words [the_program.py line 6]
We can use any global variable declared in the imported module. E.g. utils.x and utils.python declared in utils.py can be used in the_program.py

4.2 Importing functions and variables individually
We can import specific functions using the syntax from MODULE import FUNCTION/VARIABLE
This can be seen in the file the_program_v2.py (lines 1-3). (Open the files the_program_v2.py and utils.py in an editor to check this).
4.3 Importing functions and variables to python notebooks
Please note that you can also import functions and variables from a python program while using notebooks. In this case, simply treat the notebook as the python files the_program.py and the_program_v2.py.
Can you write Python code for it?

from utils import count_words

words = ['how', 'often', 'does', 'each', 'string', 'occur', 'in', 'this', 'list', '?']

word2freq = count_words(words)
print('word2freq', word2freq)


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
1.2 Opening a file
We can use the file path to tell Python which file to open by using the built-in function open(). The open() function does not return the actual text that is saved in the text file. It returns a 'file object' from which we can read the content using the .read() function (more on this later). We pass three arguments to the open() function:

the path to the file that you wish to open
the mode, a combination of characters explaining the purpose of the file opening (like read or write) and type of content stored in the file (like textual or binary format). For instance, if we are reading a plain text file, we can use the characters 'r' (represents read-mode) and 't' (represents plain text-mode).
the last argument, a keyword argument (encoding), specifies the encoding of the text file, but you can forget about this for now.

The most important mode arguments the open() function can take are:

r = Opens a file for reading only. The file pointer is placed at the beginning of the file.
w = Opens a file for writing only. Overwrites the file if the file exists. If the file does not exist, creates a new file for writing.
a = Opens a file for appending. The file pointer is at the end of the file if the file exists. If the file does not exist, it creates a new file for writing. Use it if you would like to add something to the end of a file

Then, to open the file 'charlie.txt' for reading purposes, we use the following:
Can you write Python code for it?

filepath = "../Data/Charlie/charlie.txt"
infile = open(filepath, "r") # 'r' stands for READ mode
# Do something with the file
infile.close() # Close the file (you can ignore this for now)


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
1.4.2 Using a context manager
There is actually an easier (and preferred) way to make sure that the file is closed as soon as you don't need it anymore, namely using what is called a context manager. Instead of using open() and close(), we use the  syntax shown below.
The main advantage of using the with-statement is that it automatically closes the file once you leave the local context defined by the indentation level. If you 'manually' open and close the file, you risk forgetting to close the file. Therefore, context managers are considered a best-practice, and we will use the with-statement in all of our following code.
From now on, we highly recommend using a context manager in your code.
Can you write Python code for it?

filepath = "../Data/Charlie/charlie.txt"

with open(filepath, "r") as infile:
    # the file is only open here
    # get content while file is open
    content = infile.read()

# the context manager took care of closing the file again
# we can now work with the content without having to worry about
# closing the file

print(content)


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
2 Manipulating file content
Once your file content is loaded in a Python variable, you can manipulate its content as you can manipulate any other variable. You can edit it, add/remove lines, count word occurrences, etc. Let's say we read the file content in a list of its lines, as shown below. Note that we can use all of the different methods for reading files in the context manager.
Can you write Python code for it?

filepath = "../Data/Charlie/charlie.txt"

with open(filepath, "r") as infile:
    lines = infile.readlines()

print(lines)


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
4.2 The os module
Another module that you will frequently see being used in examples is the os module. The os module has many features that can be very useful and which are not supported by the glob module. We will not go over each and every useful method here, but here's a list of some of the things that you can do (some of which we have seen above):

creating single or multiple directories: os.mkdir(), os.mkdirs();
removing single or multiple directories: os.rmdir(), os.rmdirs();
checking whether something is a file or a directory: os.path.isfile(), os.path.isdir();
split a path and return a tuple containing the directory and filename: os.path.split();
construct a pathname out of one or more partial pathnames: os.path.join()
split a filename and return a tuple containing the filename and the file extension: os.path.splitext()
get only the basename or the directory path: os.path.basename(), os.path.dirname().

Feel free to play around with these methods and figure out how they work yourself :-)
Can you write Python code for it?

# Start by importing the module:
import os

# let's use a filepath for testing it out:
filepath = "../Data/Charlie/charlie.txt"
os.path.basename(filepath)


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
A question mark (?) matches any single character in that position in the name. For example, the following code prints all filenames in the directory ../Data/dreams that start with 'vickie' followed by exactly 1 character and end with the extension .txt (note that this will not print vickie10.txt):
Can you write Python code for it?

for filename in glob.glob("../Data/Dreams/vickie?.txt"):
    print(filename)


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
Exercise 3:
Create a counter dictionary like in block 2 (the dictionaries chapter), where you will count the number of occurences of each word in a file.
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
Exercise 4:
The module os implements functions that allow us to work with the operating system (see folder contents, change directory, etc.). Use the function listdir from the module os to see the contents of the current directory. Then print all the items that do not start with a dot.
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
If we only want to consider text files and ignore everything else (here a file called 'IGNORE_ME!'), we can specify this in our search by only looking for files with the extension .txt:
Can you write Python code for it?

for filename in glob.glob("../Data/Dreams/*.txt"):
    print(filename)


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
Navigating your directory tree on Windows
Also, note that the formatting of file paths is different across operating systems. The file path, as specified above, should work on any UNIX platform (Linux, Mac). If you are using Windows, however, you might run into problems when formatting file paths in this way outside of this notebook, because Windows uses backslashes instead of forward slashes (Jupyter Notebook should already have taken care of these problems for you). In that case, it might be useful to have a look at this page about the differences between the file systems, and at this page about solving this problem in Python. In short, it's probably best if you use the code below (we will talk about the os module in more detail later today). This is very useful to know if you are a Windows user, and it will become relevant for the final assignment.
Can you write Python code for it?

# For windows:
import os
windows_file_path = os.path.normpath("C:/somePath/someFilename") # Use forward slashes


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
Note the last line of this code snippet: infile.close(). This closes our file, which is a very important operation. This prevents Python from keeping files that are unnecessary anymore still open. In the next subchapter, we will also see a more convenient way to ensure files get closed after their usage.
1.4. Closing the file
Here, we will introduce closing a file with the method close() and using a context manager to open and close files. After reading the contents of a file, the TextWrapper no longer needs to be open since we have stored the content as a variable. In fact, it is good practice to close the file as soon as you do not need it anymore.
1.4.1  close()
We do this by using the close() method as already shown several times above.
Can you write Python code for it?

filepath = "../Data/Charlie/charlie.txt"

# open file
infile = open(filepath , "r")

# assign content to a varialbe
content = infile.read()

# close file
infile.close()

# do whatever you want with the context, e.g. print it:

print(content)


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
Open the file charle_new.txt in the folder ../Data/Charlie in any text editor and read a personalized version of the story!
Note about append mode (a):
The third mode of opening a file is append ('a'). If the file 'charlie_new.txt' does not exist, then append and write act the same: they create this new file and fill it with content. The difference between write and append occurs when this file would exist. In that case, the write mode overwrites its content, while the append mode adds the new content at the end of the existing one.
4 Reading and writing multiple files
You will often have multiple files to work with. The folder ../Data/Dreams contains 10 text files describing dreams of Vickie, a 10-year-old girl. These texts are extracted from DreamBank.
To process multiple files, we often want to iterate over a list of files. These files are usually stored in one or multiple directories on your computer.
Instead of writing out every single file path, it is much more convenient to iterate over all the files in the directory ../Data/Dreams. So we need to find a way to tell Python: "I want to do something with all these files at this location!"
There are two modules which make dealing with multiple files a lot easier.

glob
os

We will introduce them below.
4.1 The glob module
The glob module is very useful to find all the pathnames matching a specified pattern according to the rules used by the Unix shell. You can use two wildcards: the asterisk (*) and the question mark (?). An asterisk matches zero or more characters in a segment of a name, while the question mark matches a single character in a segment of a name.
For example, the following code gives all filenames in the directory ../Data/dreams:
Can you write Python code for it?

import glob

for filename in glob.glob("../Data/Dreams/*"):
    print(filename)


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
This TextIOWrapper thing is Python's way of saying it has opened a connection to the file charlie.txt. To actually see its content, we need to tell python to read the file.
1.3 Reading a file
Here, we will discuss three ways of reading the contents of a file:

read()
readlines()
readline()

1.3.1 read()
The read() method is used to access the entire text in a file, which we can assign to a variable. Consider the code below.
The variable content now holds the entire content of the file charlie.txt as a single string, and we can access and manipulate it just like any other string. When we are done with accessing the file, we use the close() method to close the file.
Can you write Python code for it?

# Opening the file using the filepath and and the 'read' mode:
infile = open("../Data/Charlie/charlie.txt" , "r")
# Reading the file using the `read()` function and assigning it to the variable `content`
content = infile.read()
print(content)
print()
print('This function returns a', type(content))

# closing the file (more on this below)
infile.close()


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
We will soon see how to perform text processing once we have loaded the file, by using an external module in the next chapter. But let's first write store the modified text in a new file to preserve the changes.
3 Writing files
To write content to a file, we can open a new file and write the text to this file by using the write() method. Again, we can do this by using the context manager. Remember that we have to specify the mode using w.
Let's first slightly adapt our Charlie story by replacing the names in the text:
Can you write Python code for it?

filepath = "../Data/Charlie/charlie.txt"

# read in file and assign content to the variable content
with open(filepath, "r") as infile:
    content = infile.read()

# manipulate content

your_name = "x y" #type in your name
friends_name = "a b" #type in the name of a friend

# Replace all instances of Charlie Bucket with your name and save it in new_content
new_content = content.replace("Charlie Bucket", your_name)

# Replace all instancs of Mr Wonka with your friends name and save it in new_new_content
new_new_content = new_content.replace("Mr Wonka", friends_name)


I am working on the file "Chapters-colab/Chapter_14_Reading_and_writing_text_files.ipynb".
The first task is:
Which function to choose
For small files that you want to load entirely, you can use one of these three methods (readline, read, or readlines). Note, however, that we can also simply do the following to read a file line by line (this is recommended for larger files and when we are really only interested in a small portion of the file):
Can you write Python code for it?

infile = open(filename, "r")
for line in infile:
    print(line)
infile.close()


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
2.1 Tokenization and sentence splitting with NLTK
2.1.1 word_tokenize()
Now, let's try tokenizing our Charlie story! First, we will open and read the file again and assign the file contents to the variable content. Then, we can call the word_tokenize() function from the nltk module as follows:
Can you write Python code for it?

with open("../Data/Charlie/charlie.txt") as infile:
    content = infile.read()

tokens = nltk.word_tokenize(content)
print(type(tokens), len(tokens))
print(tokens)


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
2.2.2 Working with POS tags
As we saw above, pos_tag() returns a list of tuples: The first element is the token, the second element indicates the part of speech (POS) of the token.
This POS tagger uses the POS tag set of the Penn Treebank Project, which can be found here. For example, all tags starting with a V are used for verbs.
We can now use this, for example, to identify all the verbs in a text:
Can you write Python code for it?

# Open and read in file as a string, assign it to the variable `content`
with open("../Data/Charlie/charlie.txt") as infile:
    content = infile.read()

# Apply tokenization and POS tagging
tokens = nltk.word_tokenize(content)
tagged_tokens = nltk.pos_tag(tokens)

# List of verb tags (i.e. tags we are interested in)
verb_tags = ["VBD", "VBG", "VBN", "VBP", "VBZ"]

# Create an empty list to collect all verbs:
verbs = []

# Iterating over all tagged tokens
for token, tag in tagged_tokens:

    # Checking if the tag is any of the verb tags
    if tag in verb_tags:
        # if the condition is met, add it to the list we created above
        verbs.append(token)

# Print the list to inspect it
print(verbs)


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
4 Putting it all together
In this section, we will use what we have learned above to write a small NLP program. We will go through all the steps and show how they can be put together. In the last chapters, we have already learned how to write functions. We will make use of this skill here.
Our goal is to collect all the nouns from Vickie's dream reports.
Before we write actual code, it is always good to consider which steps we need to carry out to reach the goal.
Important steps to remember:

create a list of all the files we want to process
open and read the files
tokenize the texts
perform pos-tagging
collect all the tokens analyzed as nouns

Remember, we first needed to import nltk to use it.
4.1 Writing a processing function for a single file
Since we want to carry out the same task for each of the files, it is very useful (and good practice!) to write a single function which can do the processing. The following function reads the specified file and returns the tokens with their POS tags:
Can you write Python code for it?

import nltk

def tag_tokens_file(filepath):
    """Read the contents of the file found at the location specified in
    FILEPATH and return a list of its tokens with their POS tags."""
    with open(filepath, "r") as infile:
        content = infile.read()
        tokens = nltk.word_tokenize(content)
        tagged_tokens = nltk.pos_tag(tokens)
    return tagged_tokens


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
4.2 Iterating over all the files and applying the processing function
We can also do this for each of the files in the ../Data/Dreams directory by using a for-loop:
Can you write Python code for it?

import glob

# Iterate over the `.txt` files in the directory and perform POS tagging on each of them
for filename in glob.glob("../Data/Dreams/*.txt"):
    tagged_tokens = tag_tokens_file(filename)
    print(filename, "\n", tagged_tokens, "\n")


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
4.3 Collecting all the nouns
Now, we extend this code a bit so that we don't print all POS-tagged tokens of each file, but we get all (proper) nouns from the texts and add them to a list called nouns_in_dreams. Then, we print the set of nouns:
Can you write Python code for it?

# Create a list that will contain all nouns
nouns_in_dreams = []

# Iterate over the `.txt` files in the directory and perform POS tagging on each of them
for filename in glob.glob("../Data/Dreams/*.txt"):
    tagged_tokens = tag_tokens_file(filename)

    # Get all (proper) nouns in the text ("NN" and "NNP") and add them to the list
    for token, pos in tagged_tokens:
        if pos in ["NN", "NNP"]:
            nouns_in_dreams.append(token)

# Print the set of nouns in all dreams
print(set(nouns_in_dreams))


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
As you can see, we now have a list of all words in the text. The punctuation marks are also in the list, but as separate tokens.
2.1.2 sent_tokenize()
Another thing that NLTK can do for you is to split a text into sentences by using the sent_tokenize() function. We use it on the entire text (as a string):
Can you write Python code for it?

with open("../Data/Charlie/charlie.txt") as infile:
    content = infile.read()

sentences = nltk.sent_tokenize(content)

print(type(sentences), len(sentences))
print(sentences)


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
Exercise 3:
Now use a for-loop to count the number of times that each of these verb lemmas occurs in the text! For each verb in the list you just created, get the count of this verb in charlie.txt using the count() method. Create a dictionary that contains the lemmas of the verbs as keys, and the counts of these verbs as values. Refer to the notebook about Topic 1 if you forgot how to use the count() method or how to create dictionary entries!
Tip: you don't need to read in the file again, you can just use the list called verb_lemmas.
Can you write Python code for it?

verb_counts = {}

# Finish this for-loop
for verb in unique_verbs:
    # your code here

print(verb_counts)

# Test your code here! If your code is correct, you should get a compliment :-)
assert len(verb_counts) == 28 and verb_counts["bubble"] == 1 and verb_counts["be"] == 9
print("Well done!")


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
Note about the wordnet lemmatizer:
We need to specify a POS tag to the WordNet lemmatizer, in a WordNet format ("n" for noun, "v" for verb, "a" for adjective). If we do not indicate the Part-of-Speech tag, the WordNet lemmatizer thinks it is a noun (this is the default value for its part-of-speech). See the examples below:
Can you write Python code for it?

test_nouns = ('building', 'applications', 'leafs')
for n in test_nouns:
    print(f"Noun in conjugated form: {n}")
    default_lemma=lmtzr.lemmatize(n) # default lemmatization, without specifying POS, n is interpretted as a noun!
    print(f"Default lemmatization: {default_lemma}")
    verb_lemma=lmtzr.lemmatize(n, 'v')
    print(f"Lemmatization as a verb: {verb_lemma}")
    noun_lemma=lmtzr.lemmatize(n, 'n')
    print(f"Lemmatization as a noun: {noun_lemma}")
    print()

test_verbs=('grew', 'standing', 'plays')
for v in test_verbs:
    print(f"Verb in conjugated form: {v}")
    default_lemma=lmtzr.lemmatize(v) # default lemmatization, without specifying POS, v is interpretted as a noun!
    print(f"Default lemmatization: {default_lemma}")
    verb_lemma=lmtzr.lemmatize(v, 'v')
    print(f"Lemmatization as a verb: {verb_lemma}")
    noun_lemma=lmtzr.lemmatize(v, 'n')
    print(f"Lemmatization as a noun: {noun_lemma}")
    print()


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
Now that we have installed and downloaded NLTK, let's look at an example of a simple NLP pipeline. In the following cell, you can observe how we tokenize raw text into tokens and setnences, perform part of speech tagging and lemmatize some of the tokens. Don't worry about the details just yet - we will go trhough them step by step.
Can you write Python code for it?

text = "This example sentence is used for illustrating some basic NLP tasks. Language is awesome!"

# Tokenization
tokens = nltk.word_tokenize(text)

# Sentence splitting
sentences = nltk.sent_tokenize(text)

# POS tagging
tagged_tokens = nltk.pos_tag(tokens)

# Lemmatization
lmtzr = nltk.stem.wordnet.WordNetLemmatizer()
lemma=lmtzr.lemmatize(tokens[4], 'v')

# Printing all information
print(tokens)
print(sentences)
print(tagged_tokens)
print(lemma)


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
Now we have an idea what Vickie dreams about!
Exercises
Exercise 1:
Try to collect all the present participles in the the text store in ../Data/Charlie/charlie.txt using the NLTK tokenizer and POS-tagger.
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
Once you have downloaded the NLTK book, you do not need to run the download again. If you are using the NLTK again, it is sufficient to import it.
Can you write Python code for it?

# downloading nltk

import nltk
nltk.download('book')


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
This looks good! We now have a list of words like boiling, sizzling, etc. However, we can see that there is one word in the list that actually is not a present participle (ceiling). Of course, also other words can end with -ing. So if we want to find all present participles, we have to come up with a smarter solution.
2.2. Part-of-speech (POS) tagging
Once again, NLTK comes to the rescue. Using the function pos_tag(), we can label each word in the text with its part of speech.
To do pos-tagging, you first need to tokenize the text. We have already done this above, but we will repeat the steps here, so you get a sense of what an NLP pipeline may look like.
2.2.1 pos_tag()
To see how pos_tag() can be used, we can (as always) look at the documentation by using the help() function. As we can see, pos_tag() takes a tokenized text as input and returns a list of tuples in which the first element corresponds to the token and the second to the assigned pos-tag.
Can you write Python code for it?

# As always, we can start by reading the documentation:
help(nltk.pos_tag)

# Open and read in file as a string, assign it to the variable `content`
with open("../Data/Charlie/charlie.txt") as infile:
    content = infile.read()

# Split up entire text into tokens using word_tokenize():
tokens = nltk.word_tokenize(content)

# Apply pos tagging to the tokenized text
tagged_tokens = nltk.pos_tag(tokens)

# Inspect pos tags
print(tagged_tokens)


I am working on the file "Chapters-colab/Chapter_15_Off_to_analyzing_text.ipynb".
The first task is:
You should get the following list:
['boiling', 'bubbling', 'hissing', 'sizzling', 'clanking', 'running', 'hopping', 'knowing', 'rubbing', 'cackling', 'going']
Can you write Python code for it?

# we can test our code using the assert statement (don't worry about this now,
# but if you want to use it, you can probably figure out how it works yourself :-)
# If our code is correct, we should get a compliment :-)
assert len(present_participles) == 11 and type(present_participles[0]) == str
print("Well done!")


I am working on the file "Chapters-colab/Chapter_16_Data_formats_I_(CSV_and_TSV).ipynb".
The first task is:
4. Dealing with column headers
Many csv/tsv files contain headers. This means that the first row contains the names of the columns. We easily read and write these kinds of files without having to type the headers manually.
4.1 Reading csv/tsv files with a header row
Consider the file called Concreteness_ratings_Brysbaert_et_al_BRM.txt (we're reading it in and printing the first 5 lines below).
The file has 10 columns, each of which have a column header. This means that all the values in the first row are not actual values. We need to treat them differently if we want to analyze the data (i.e. everything from the second row on). In addition, it would be convenient to use this information right away, without having to first inspect the file and manually type the headers as keys in dictionaries!
Can you write Python code for it?

with open('../Data/csv_data/Concreteness_ratings_Brysbaert_et_al_BRM.tsv') as infile:
    lines = infile.read().split('\n')

for line in lines[:5]:
    print(line)


I am working on the file "Chapters-colab/Chapter_16_Data_formats_I_(CSV_and_TSV).ipynb".
The first task is:
4.2 Writing csv/tsv files with a header row
You can also write files with headers:
Can you write Python code for it?

row1 = {'name': 'Harry', 'house' : 'Gryffindor'}
row2 = {'name': 'Ron', 'house' : 'Gryffindor'}
row3 = {'name': 'Hannah', 'house': 'Hufflepuff'}
row4 = {'name': 'Leta', 'house' : 'Slytherin'}

data_dict_list = [row1, row2, row2, row4]

for d in data_dict_list:
    print(d)

header_row = ### your code here

with open('../Data/csv_data/hp_example.csv', 'w') as outfile:
    # write header
    for d in data_dict_list:
        row_list = d.values()
        row = ','.join(values)
        outfile.write(row+'\n')


I am working on the file "Chapters-colab/Chapter_16_Data_formats_I_(CSV_and_TSV).ipynb".
The first task is:
A note on csv/tsv files containing a lot of textual data
If you're dealing with tables containing textual data (e.g. full sentences or tweets), you will realize that splitting on commas or tabs may result in confusion (as text often contains commas and may contain tabs). Consider for instances this file: ../Data/csv_data/debate.csv.
You may have ideas for a fix regarding this specific file and surely you will eventually find a good solution for it. However, for these more complex CSV files, we recommend that you make use of the csv module. Feel free to explore it yourself, but do not use it for the exercises and assignments in this course.
Exercises
Exercise 1:
Have another look at the code below. Can you predict what would happen if you skip some of the steps? How will the data be different?
Can you write Python code for it?

filename = "../Data/csv_data/baby_names/names_by_state/AK.csv"
with open(filename, "r") as csvfile:
    csv_data = []
    for row in csvfile:
        #row = row.strip("\n")      # what happens if you skip this step?
        columns = row.split(",")
        csv_data.append(columns)
print(csv_data[0:2])

filename = "../Data/csv_data/baby_names/names_by_state/AK.csv"
with open(filename, "r") as csvfile:
    csv_data = []
    for row in csvfile:
        row = row.strip("\n")
        #columns = row.split(",")   # what happes if you skip this step?
        csv_data.append(row)        # replaced columns by row
print(csv_data[0:2])


I am working on the file "Chapters-colab/Chapter_16_Data_formats_I_(CSV_and_TSV).ipynb".
The first task is:
Again, we can now easily work with the data, but now we use the names of the columns instead of indices to access the cells. This can make the code more readable. Have a look at the following examples:
Can you write Python code for it?

# Example: print all information of the first 5 rows
for row in csv_data[:5]:
    for column_name, cell_value in row.items():
        print(column_name, "=", cell_value)
    print()

# Example: print all names given in 1912
for row in csv_data:
    if row["year"] == 1912:
        print(row["name"])


I am working on the file "Chapters-colab/Chapter_16_Data_formats_I_(CSV_and_TSV).ipynb".
The first task is:
Exercise 4:
Now append information about Jennifer and Justin (stored as a list) at the bottom of this address_book.tsv file (attention: make sure you do not overwrite the contents of the file).
Can you write Python code for it?

more_people = [
    ["Jennifer", "jennifer@example.com", "555-9876"],
    ["Justin", "justin@example.com", "555-5555"]
]

# Append the data to the `address_book.tsv` file


I am working on the file "Chapters-colab/Chapter_16_Data_formats_I_(CSV_and_TSV).ipynb".
The first task is:
Exercise 5:
Create a function load_tabular_data(filename, delimiter) that receives the filename and a delimiter as input parameters, and returns the file content as a list of lists. Then we can call this function for CSV (with a comma as an argument) and for TSV (with a tabulator as an argument). Also make sure that delimiter is a keyword parameter, with a defaul value ','.
Can you write Python code for it?

# Create your function here

# Now let's test the function
csv_filename = "../Data/csv_data/baby_names/names_by_state/AK.csv"


I am working on the file "Chapters-colab/Chapter_16_Data_formats_I_(CSV_and_TSV).ipynb".
The first task is:
In addition, we should use the newline character \n to write each row on a line.
Run the cell below and check whether it worked by opening the file ../Data/address_book.csv in a text editor. Is everything correct? Did you use the correct new-line character?
Can you write Python code for it?

# assign new line character to a variable
# (we are going to need this when processing the lines of the file)
#new line on linux/unix
newline_char = '\n'
#new line on windows (double-check this in an editor or use repr())
#newline_char = '\r\n'

# Create list of lists
address_book = [
    ["John", "john@example.nl", "555-1234"],
    ["William", "william@example.nl", "555-5678"],
    ["Jane", "jane@example.nl", "555-7777"]
]

# Write the list of lists to a CSV file
outfilename = "../Data/address_book.csv"
with open(outfilename, "w") as outfile:
    for row in address_book:
        line = f'{",".join(row)}{newline_char}'
        outfile.write(line)


I am working on the file "Chapters-colab/Chapter_16_Data_formats_I_(CSV_and_TSV).ipynb".
The first task is:
Please also have a close look at the internal representation of the file. Do you see how the columns are separated by commas, and the rows by newline characters \n?
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_16_Data_formats_I_(CSV_and_TSV).ipynb".
The first task is:
The file AK.csv contains a list of names given to children in the state of Alaska from 1910 to 2015 with their frequency. Each line in this file has five elements, which are separated by commas:

the state abbreviation (AK for Alaska)
gender (F/M)
year
name
frequency of that name in the given year and state

Below, we will work towards representing this data in Python as a list of lists (i.e., a nested list) or as a list of dicts. In both cases, the elements of the (first) list represent the complete rows. The individual rows, then, can be either represented as a list (without column names) or as a dictionary (with column names).
|      LIST OF LISTS        |  LIST OF DICTS            |
|:-------------------------:|:-------------------------:|
Can you write Python code for it?

from IPython.display import Image
display(Image('./images/list_of_lists.png'))
display(Image('./images/list_of_dicts.png'))


I am working on the file "Chapters-colab/Chapter_16_Data_formats_I_(CSV_and_TSV).ipynb".
The first task is:
You can find examples of CSV files in the folder ../Data/csv_data. For instance, look at the files in ../Data/csv_data/baby_names/names_by_state/. If you like, open them in a text editor (e.g., Atom, BBEdit, or Notepad++), or Excel (convert text to columns by using the comma as delimiter) to see their content.
For example, these are the first 10 rows of the CSV file AK.csv:
Can you write Python code for it?

from IPython.display import Image
Image('./images/csv.png')


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
3. Accessing data in a json dictionary
Note that the values in this dictionary can be containers themselves: Each key has another dictionary as a value. The keys 'children' and 'hobbies' have lists as values. Note that we can look at such a dictionary in terms of layers of nesting.
Consider the dict below. How many layers (i.e. containers within containers) can you identify?
Can you write Python code for it?

dict_doe_family = {
    "John": {
        "first name": "John",
        "last name": "Doe",
        "gender": "male",
        "age": 30,
        "favorite_animal": "panda",
        "married": True,
        "children": ["James", "Jennifer"],
        "hobbies": ["photography", "sky diving", "reading"]},
    "Jane": {
        "first name": "Jane",
        "last name": "Doe",
        "gender": "female",
        "age": 27,
        "favorite_animal": "zebra",
        "married": False,
        "children": None,
        "hobbies": ["cooking", "gaming", "tennis"]}}


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
Both dump() and dumps() use the same keyword parameters. You can check them out with help():
Hint: There are ways of formating the dictionary nicely. Can you find out how it works from reading the documentation?
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
Chapter 17: Data formats (JSON)
Let's have a look at another data format: JSON (JavaScript Object Notation). JSON is a lightweight data-interchange format that is easy for humans to read and write, and easy for machines to parse and generate. There is a good chance that at some point you will be gathering information through an API that is formatted in JSON (e.g. Twitter data).
At the end of this chapter, you will be able to:

read and write JSON dictionary files
deal with multiple layers of nesting within JSON dictionary structures

If you want to learn more about these topics, you might find the following links useful:

Video: Working With JSON
Tutorial: Working With JSON Data in Python

If you have questions about this chapter, please contact us at cltl.python.course@gmail.com.
1. Introduction to JSON
JSON is completely language independent. However, data formatted in JSON works just like a Python dictionary!
Can you write Python code for it?

from IPython.display import Image
Image('./images/python_json_conversion_table.png')


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
Exercise 2:
Working with JSON data thus means working with dictionaries. Let's practice a bit more with accessing the values of dictionaries by printing the following:
* all (parent) names
* the age of Jane
* the first hobby of John
* the favorite animal of each person (use a for-loop)
Can you write Python code for it?

# Example: This will print the gender of "John"
print(dict_doe_family["John"]["gender"])


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
Exercise 3:
Please add the following entry to dict_doe_family. Write the result to a JSON file.
Can you write Python code for it?

julia = {"Julia": {"first name": "Julia",
                   "last name": "Doe",
                   "age": 29,
                   "favorite_animal": "penguin",
                   "married": False,
                   "children": ["Jack"],
                   "hobbies": ["snowboarding", "hiking"]}}

# your code here


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
Exercise 4:
Let's have a look at another example. The JSON file ../Data/json_data/StrangerThings.json contains information about episodes from the TV show Stranger Things. Read the file and store the dictionary as tv_show.
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
Exercises
Exercise 1:
What is the difference between str() and json.dumps() to convert a Python dictionary to a string? Try it out below.
Can you write Python code for it?

dict_doe_family = {
    "John": {
        "first name": "John",
        "last name": "Doe",
        "gender": "male",
        "age": 30,
        "favorite_animal": "panda",
        "married": True,
        "children": ["James", "Jennifer"],
        "hobbies": ["photography", "sky diving", "reading"]},
    "Jane": {
        "first name": "Jane",
        "last name": "Doe",
        "gender": "female",
        "age": 27,
        "favorite_animal": "zebra",
        "married": False,
        "children": None,
        "hobbies": ["cooking", "gaming", "tennis"]}}

str_doe_family = json.dumps(dict_doe_family)
print(str_doe_family)

str_doe_family = str(dict_doe_family)
print(str_doe_family)


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
However, since it is structured to correspond well to Python objects, we use an existing mudule, the json library, which provides an easy way to encode and decode data in JSON.  Let's first import it:
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
The dumps() method is used to convert a Python dictionary to a JSON formatted string:
Can you write Python code for it?

str_doe_family = json.dumps(dict_doe_family)
print(str_doe_family)


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
The loads() method is used to load a JSON formatted string as a Python dictionary. This is useful if you want to create a json dictionary from a string. There are not many situations in which you will have to use it. You may come accross json structures stored as strings in when working with corpora or collecting human annotations with annotation software.
Can you write Python code for it?

str_doe_family = """
{
    "Jane": {
        "age": 27,
        "children": null,
        "favorite_animal": "zebra",
        "first name": "Jane",
        "gender": "female",
        "hobbies": [
            "cooking",
            "gaming",
            "tennis"
        ],
        "last name": "Doe",
        "married": false
    },
    "John": {
        "age": 30,
        "children": [
            "James",
            "Jennifer"
        ],
        "favorite_animal": "panda",
        "first name": "John",
        "gender": "male",
        "hobbies": [
            "photography",
            "sky diving",
            "reading"
        ],
        "last name": "Doe",
        "married": true
    }
}
"""

dict_doe_family = json.loads(str_doe_family)
print(type(dict_doe_family))
print(dict_doe_family)


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
To help you understand the structure a bit, first have a look at the following examples:
Can you write Python code for it?

print(tv_show.keys())

print(tv_show["_embedded"].keys())

print(tv_show["_embedded"]["episodes"])


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
Two useful keyword arguments are for example indent and sort_keys. They are illustrated below:
Can you write Python code for it?

# Create the JSON file
with open("../Data/json_data/Doe.json", "w") as outfile:
     json.dump(dict_doe_family,
               outfile,
               indent=4,
               sort_keys=True)

# Read in the JSON file again
with open("../Data/json_data/Doe.json", "r") as infile:
    json_string = infile.read()
    print(json_string)

str_doe_family = json.dumps(dict_doe_family,
                            indent=4,
                            sort_keys=True)
print(str_doe_family)


I am working on the file "Chapters-colab/Chapter_17_Data_formats_II_(JSON).ipynb".
The first task is:
We will focus on the following methods:
* to read JSON: the methods json.load() and json.loads()
* to write JSON: the methods json.dump() and json.dumps()
The functions with an s take string arguments.
2.1 Loading JSON from file or string
The load() method is used to load a JSON encoded file as a Python dictionary:
Can you write Python code for it?

# open file just as you would open a txt file
with open("../Data/json_data/Doe.json", "r") as infile:
    # read in file content as dict using the json module
    dict_doe_family = json.load(infile)
    print(type(dict_doe_family))
    print(dict_doe_family)


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
3.3 Accessing element information
We will now show how to access the attributes, text, and tag of an element.
The get() method is used to access the attribute of an element.
Attention: If an attribute does not exist, it will return None. Hence, there will not be an error.
Can you write Python code for it?

first_person_el = root.find('person')
role_first_person_el = first_person_el.get('role')
attribute_not_found = first_person_el.get('blabla')
print('role first person element:', role_first_person_el)
print('value if not found:', attribute_not_found)


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
4 How to deal with more than one layer
In our previous example, we had an XML with only one nested layer (person). However, XML can deal with many more.
Let's look at such an example and think about how you would access the first target element, i.e.
```xml
<target id="t1" />
```
```xml
<NAF xml:lang="en" version="v3">
    <terms>
        <term id="t1" type="open" lemma="Tom" pos="N" morphofeat="NNP">
        <term id="t2" type="open" lemma="Cruise" pos="N" morphofeat="NNP">
        <term id="t3" type="open" lemma="be" pos="V" morphofeat="VBZ">
        <term id="t4" type="open" lemma="an" pos="R" morphofeat="DT">
        <term id="t5" type="open" lemma="actor" pos="N" morphofeat="NN">
    </terms>
    <entities>
        <entity id="e3" type="PERSON">
              <references>
                  <span>
                      <target id="t1" />
                      <target id="t2" />
                  </span>
              </references>
        </entity>
    </entities>
</NAF>
```
Again, we use etree.fromstring() to load XML from a string:
Can you write Python code for it?

naf_string = """
<NAF xml:lang="en" version="v3">
    <text>
        <wf id="w1" offset="0" length="3" sent="1" para="1">tom</wf>
        <wf id="w2" offset="4" length="6" sent="1" para="1">cruise</wf>
        <wf id="w3" offset="11" length="2" sent="1" para="1">is</wf>
        <wf id="w4" offset="14" length="2" sent="1" para="1">an</wf>
        <wf id="w5" offset="17" length="5" sent="1" para="1">actor</wf>
    </text>
    <terms>
        <term id="t1" type="open" lemma="Tom" pos="N" morphofeat="NNP"/>
        <term id="t2" type="open" lemma="Cruise" pos="N" morphofeat="NNP"/>
        <term id="t3" type="open" lemma="be" pos="V" morphofeat="VBZ"/>
        <term id="t4" type="open" lemma="an" pos="R" morphofeat="DT"/>
        <term id="t5" type="open" lemma="actor" pos="N" morphofeat="NN"/>
    </terms>
    <entities>
        <entity id="e3" type="PERSON">
              <references>
                  <span>
                      <target id="t1" />
                      <target id="t2" />
                  </span>
              </references>
        </entity>
    </entities>
</NAF>
"""

naf = etree.fromstring(naf_string)
print(type(naf))
etree.dump(naf, pretty_print=True)


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
5. Extracting infromation from XML
Often, we want to extract information from an XML file, so we can analyze and possibly manipulate it in python. It can be very useful to use python containers for this. In the following example, we want to collect all tokens (i.e. words as they appear in text) of a part of speech.
To do this, we have to extract infromation from the token layer and combine it with infromation from the term layer.
Can you write Python code for it?

path_to_tokens = 'text/wf'
path_to_terms = 'terms/term'

# We define dictionaries to map identifiers to tokens ('word forms') and term tags including pos information.
# We can use the identifiers to map the tokens to the pos tags in the next step
tokens = naf.findall(path_to_tokens)
terms = naf.findall(path_to_terms)

id_token_dict = dict()
id_pos_dict = dict()

# map ids to tokens
for token in tokens:
    id_token_dict[token.get('id')] = token.text
#map ids to terms
for term in terms:
    id_pos_dict[term.get('id')] = term.get('pos')

 #use ids to map tokens to pos tags
for token_id, token in id_token_dict.items():
    # term identifiers start with a t, token identifiers with a w. The numbers correspond.
    term_id = token_id.replace('w', 't')
    pos = id_pos_dict[term_id]
    print(token, pos)


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
6. EXTRA: Creating your own XML
Please note that this section is optional, meaning that you don't need to understand this section in order to complete the assignment.
There are three main steps:
* Step a: Create an XML object with a root element
* Step b: Creating child elements and adding them
* Step c: Writing to a file
Step a: Create an XML object with a root element
You create a new XML object by:
* creating the root element -> using etree.Element
* creating the main XML object -> using etree.ElementTree
You do not have to fully understand how this works. Please make sure you can reuse this code snippet when you create your own XML.
Can you write Python code for it?

our_root = etree.Element('Course')
our_tree = etree.ElementTree(our_root)


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
As we have seen before, XML can have multiple nested layers. Creating these works the same way as adding child elements to the root, but now we specify one of the other elements as the parent (in this case, new_person_element).
Can you write Python code for it?

# Define tag, attributes and text of the new element
tag = 'pet'
attributes = {'role': 'joy'}
name_pet = 'Romeo'

# Add to new_person_element
new_pet_element = etree.SubElement(new_person_element, tag, attrib=attributes) # parent is new_person_element
new_pet_element.text = name_pet

# Inspect the current XML
etree.dump(our_root, pretty_print=True)


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
As you can see, etree.parse() returns an ElementTree, whereas etree.fromstring() returns an Element. One of the important differences is that the ElementTree class serialises as a complete document, as opposed to a single Element. This includes top-level processing instructions and comments, as well as a DOCTYPE and other DTD content in the document. For now, it's not too important that you know what these are; just remember that there is a difference btween ElementTree and Element.
3.1 Accessing root element
While etree.fromstring() gives you the root element right away, etree.parse() does not. In order to access the root element of ElementTree, we first need to use the getroot() method. Note that this does not show the XML element itself, but only a reference. In order to show the element itself, we can use the etree.dump() method.
Hint: etree.dump is helpful for inspecting (parts of) an xml structure you have loaded from a file. You will see examples of this later.
Can you write Python code for it?

root = tree.getroot()
print('root', type(root), root)
print()
print('etree.dump example')
etree.dump(root, pretty_print=True)


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
Chapter 18 - Data Formats III (XML)
In this chapter, we will learn how to work with one of the most popular structured data formats: XML. XML is used a lot in Natural Language Processing (NLP), and it is important that you know how to work with it. In theory, you could load XML data just as you read in a text file, but the structure is much too complicated to extract information manually. Therefore, we use an existing library.
At the end of this chapter, you will be able to

read an XML file using etree.parse
read XML from string using etree.fromstring
convert an XML element to a string using etree.tostring

use the following methods and attributes of an XML element (of type lxml.etree._Element):

to access elements: methods find, findall, and getchildren
to access attributes: method get
to access element information: attributes tag and text

[not needed for assignment] create your own XML and write it to a file

If you want to learn more about this chapter, you might find the following links useful:

XML
detailled XML introduction
NAF XML
Xpath
Other structured data formats: JSON-LD, MicroData, RDF

If you have questions about this chapter, please contact us at cltl.python.course@gmail.com.
1. Introduction to XML
Natural language processing (NLP) is all about text data. More specifically, we usually want to annotate (manually or automatically) textual data with information about:

part of speech
word senses
syntactic information (in particulay dependencies)
entities
semantic role labelling
Events
many many many more.....

How should we represent such annotated data? This is what you get from using NLTK tools:
Can you write Python code for it?

import nltk
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')

text = nltk.word_tokenize("Tom Cruise is an actor.")
text_pos_tagged = nltk.pos_tag(text)
print(text_pos_tagged)
print(type(text_pos_tagged), type(text_pos_tagged[0]))


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
However, this is so common that there is a shorter and much more efficient way to do this: by using etree.SubElement(). It accepts the same arguments as the etree.Element() method, but additionally requires the parent as first argument:
Can you write Python code for it?

# Define tag, attributes and text of the new element
tag = 'person'
attributes = {'role': 'student'}
name_student = 'Pitt'

# Add to root
another_person_element = etree.SubElement(our_root, tag, attrib=attributes) # parent is our_root
another_person_element.text = name_student

# Inspect the current XML
etree.dump(our_root, pretty_print=True)


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
In order to get a list of all person children, we can use the findall() method.
Notice that this does not return the animal since we are looking for person elements.
Can you write Python code for it?

all_person_els = root.findall('person')
# Check how many we found
print(len(all_person_els))
all_person_els


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
In this example, we see that the format is a list of tuples.  The first element of each tuple is the word and the second element is the part of speech tag. Great, so far, this works.
However, we usually want to store more information. For instance, we may want to indicate that Tom Cruise is a named entity. We could also represent syntactic information about this sentence. Now we start to run into difficulties because some annotations are for single words and some are for combinations of words. In addition, we have more than one annotation per token. Data structures such as CSV and TSV are not great at representing linguistic information. So is there a format that is better at it? The answer is yes and the format is XML.
2. Terminology
Let's look at an example (the line numbers are there for explanation purposes). On purpose, we start with a non-linguistic, hopefully intuitive example. In the folder ../Data/xml_data this XML is stored as the file course.xml. You can inspect this file using a text editor (e.g. Atom, BBEdit or Notepad++).
xml
1.  <Course>
2.      <person role="coordinator">Van der Vliet</person>
3.      <person role="instructor">Van Miltenburg</person>
4.      <person role="instructor">Van Son</person>
5.      <person role="instructor">Postma</person>
7.      <person role="student">Baloche</person>
8.      <person role="student">De Boer</person>
9.      <animal role="student">Rubber duck</animal>
10.     <person role="student">Van Doorn</person>
11.     <person role="student">De Jager</person>
12.     <person role="student">King</person>
13.     <person role="student">Kingham</person>
14.     <person role="student">Mózes</person>
15.     <person role="student">Rübsaam</person>
16.     <person role="student">Torsi</person>
17.     <person role="student">Witteman</person>
18.     <person...
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
Is there a better way? The answer is yes! The following way is an easier way to find our target element:
Can you write Python code for it?

target_el = naf.find('entities/entity/references/span/target')
etree.dump(target_el, pretty_print=True)


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
Please note that the structure is as follows:
* the NAF element is the parent of the elements text, terms, and entities
* the wf elements are children of the text element, which provides us information about the position of words in the text, e.g. that tom is the first word in the text (id="w1") and in the first sentence (sent="1")
* the term elements are children of the term elements, which provide us information about lemmatization and part of speech
* the entity element is a child of the entities element. We learn from the entity element that the terms t1 and t2 (e.g. Tom Cruise) form an entity of type person.
One way of accessing the first target element is by going one level at a time:
Can you write Python code for it?

entities_el = naf.find('entities')
entity_el = entities_el.find('entity')
references_el = entity_el.find('references')
span_el = references_el.find('span')
target_el = span_el.find('target')
etree.dump(target_el, pretty_print=True)


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
Sometimes, we simple want all the children, while ignoring the start tags. This can be achieved using the getchildren() method. The list created below will contain all elements under root (including the animal element)
Can you write Python code for it?

all_child_els = root.getchildren()
print(len(all_child_els))
all_child_els


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
Step c: Writing to a file
This is how we can write our selfmade XML to a file. Please inspect ../Data/xml_data/selfmade.xml using a text editor to check if it worked.
Can you write Python code for it?

with open('../Data/xml_data/selfmade.xml', 'wb') as outfile:
    our_tree.write(outfile,
                   pretty_print=True,
                   xml_declaration=True,
                   encoding='utf-8')


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
The etree.parse() method is used to load XML files on your computer:
Can you write Python code for it?

tree = etree.parse('../Data/xml_data/course.xml')
print(tree)
print(type(tree))


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
The tag of an element is found in the attribute tag:
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
The text of an element is found in the attribute text:
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_18_Data_formats_III_(XML).ipynb".
The first task is:
We will focus on the following methods/attributes:
* to parse the XML from file or string: the methods etree.parse() and etree.fromstring()
* to access the root element: the methods getroot()
* to access elements: the methods find(), findall(), and getchildren()
* to access attributes: the method get()
* to access element information: the attributes tag and text
3.1 Parsing XML from file or string
The etree.fromstring() is used to parse XML from a string:
Can you write Python code for it?

xml_string = """
<Course>
    <person role="coordinator">Van der Vliet</person>
    <person role="instructor">Van Miltenburg</person>
    <person role="instructor">Van Son</person>
    <person role="instructor">Marten Postma</person>
    <person role="student">Baloche</person>
    <person role="student">De Boer</person>
    <animal role="student">Rubber duck</animal>
    <person role="student">Van Doorn</person>
    <person role="student">De Jager</person>
    <person role="student">King</person>
    <person role="student">Kingham</person>
    <person role="student">Mózes</person>
    <person role="student">Rübsaam</person>
    <person role="student">Torsi</person>
    <person role="student">Witteman</person>
    <person role="student">Wouterse</person>
    <person/>
</Course>
"""

tree = etree.fromstring(xml_string)
print(type(tree))

# printing the tree only shows a reference to the tree, but not the tree itself
# To access information, you will have to use the methods we introduce below.
print(tree)


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
2.2 Using spaCy
Parsing a text with spaCy after loading a language model is as easy as follows:
Can you write Python code for it?

doc = nlp("I have an awesome cat. It's sitting on the mat that I bought yesterday.")


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
Chapter 19 - More about Natural Language Processing Tools (spaCy)
Text data is unstructured. But if you want to extract information from text, then you often need to process that data into a more structured representation. The common idea for all Natural Language Processing (NLP) tools is that they try to structure or transform text in some meaningful way. You have already learned about four basic NLP steps: sentence splitting, tokenization, POS-tagging and lemmatization. For all of these, we have used the NLTK library, which is widely used in the field of NLP. However, there are some competitors out there that are worthwhile to have a look at. One of them is spaCy, which is fast and accurate and supports multiple languages.
At the end of this chapter, you will be able to:
- work with spaCy
- find some additional NLP tools
1. The NLP pipeline
There are many tools and libraries designed to solve NLP problems. In Chapter 15, we have already seen the NLTK library for tokenization, sentence splitting, part-of-speech tagging and lemmatization. However, there are many more NLP tasks and off-the-shelf tools to perform them. These tasks often depend on each other and are therefore put into a sequence; such a sequence of NLP tasks is called an NLP pipeline. Some of the most common NLP tasks are:

Tokenization: splitting texts into individual words
Sentence splitting: splitting texts into sentences
Part-of-speech (POS) tagging: identifying the parts of speech of words in context (verbs, nouns, adjectives, etc.)
Morphological analysis: separating words into morphemes and identifying their classes (e.g. tense/aspect of verbs)
Stemming: identifying the stems of words in context by removing inflectional/derivational affixes, such as 'troubl' for 'trouble/troubling/troubled'
Lemmatization: identifying the lemmas (dictionary forms) of words in context, such as 'go' for 'go/goes/going/went'
Word Sense Disambiguation (WSD): assigning the correct meaning to words in context
Stop words recognition: identifying commonly used...
Can you write Python code for it?

!pip install -U spacy

%%bash
python -m spacy download en_core_web_sm


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
Do you want to learn more about the differences between NLTK, spaCy and CoreNLP? Here are some links:
- Facts & Figures (spaCy)
- About speed (CoreNLP vs. spaCy)
- NLTK vs. spaCy: Natural Language Processing in Python
- What are the advantages of Spacy vs NLTK?
- 5 Heroic Python NLP Libraries
5. Some other useful modules for cleaning and preprocessing
Data is often messy, noisy or includes irrelevant information. Therefore, chances are big that you will need to do some cleaning before you can start with your analysis. This is especially true for social media texts, such as tweets, chats, and emails. Typically, these texts are informal and notoriously noisy. Normalising them to be able to process them with NLP tools is a NLP challenge in itself and fully discussing it goes beyond the scope of this course. However, you may find the following modules useful in your project:

tweet-preprocessor: This library makes it easy to clean, parse or tokenize the tweets. It supports cleaning, tokenizing and parsing of URLs, hashtags, reserved words, mentions, emojis and smileys.
emot: Emot is a python library to extract the emojis and emoticons from a text (string). All the emojis and emoticons are taken from a reliable source, i.e. Wikipedia.org.
autocorrect: Spelling corrector (Python 3).
html: Can be used to remove HTML tags.
chardet: Universal encoding detector for Python 2 and 3.
ftfy: Fixes broken unicode strings.

If you are interested in reading more about these topic, these papers discuss preprocessing and normalization:

Assessing the Consequences of Text Preprocessing Decisions (Denny & Spirling 2016). This paper is a bit long, but it provides a nice discussion of common preprocessing steps and their potential effects.
What to do about bad language on the internet (Eisenstein 2013). This is a quick read that we recommend everyone to at least look through.

And here is a nice blog about character encoding.
Exercises
Can you write Python code for it?

import spacy
nlp = spacy.load('en_core_web_sm')


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
Exercise 1:

What is the difference between token.pos_ and token.tag_? Read the docs to find out.

What do the different labels mean? Use space.explain to inspect some of them. You can also refer to this page for a complete overview.
Can you write Python code for it?

doc = nlp("I have an awesome cat. It's sitting on the mat that I bought yesterday.")
for token in doc:
    print(token.pos_, token.tag_)

spacy.explain("PRON")


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
Exercise 4:
Create a function called get_sentences_with_keyword that takes one positional parameter filenames and one keyword parameter filenames with default value None. It should read in all filenames and return a list called sentences that contains all sentences (the complete texts) with the keyword.
Hints:
- It's best to check for the lemmas of each token
- Lowercase both your keyword and the lemma
Can you write Python code for it?

import glob
filenames = glob.glob("../Data/Dreams/*.txt")
print(filenames)

def get_sentences_with_keyword(filenames, keyword=None):
    #your code here

# test your function here
sentences = get_sentences_with_keyword(filenames, keyword="toy")
print(sentences)
assert len(sentences) == 4 # if your code is correct, this should not raise an error


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
If you find this difficult to comprehend, you can also simply convert it to a list and then loop over the list. Remember that this is less efficient, though.
Can you write Python code for it?

# You can also store the sentences in a list and then loop over the list
sentences = list(doc.sents)
for sentence in sentences:
    print(sentence, type(sentence))


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
Let's inspect some of the attributes of the tokens. Can you figure out what they mean? Feel free to try out a few more.
Can you write Python code for it?

# Print attributes of tokens
for token in doc:
    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_, token.shape_)


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
Notice that some of the attributes end with an underscore. For example, tokens have both lemma and lemma_ attributes. The lemma attribute represents the id of the lemma (integer), while the lemma_ attribute represents the unicode string representation of the lemma. In practice, you will mostly use the lemma_ attribute.
Can you write Python code for it?

for token in doc:
    print(token.lemma, token.lemma_)


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
Now create a function called get_vocabulary that takes one positional parameter filenames. It should read in all filenames and return a set called unique_words, that contains all unique words in the files.
Can you write Python code for it?

def get_vocabulary(filenames):
    # your code here

# test your function here
unique_words = get_vocabulary(filenames)
print(unique_words, len(unique_words))
assert len(unique_words) == 415 # if your code is correct, this should not raise an error


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
Please note that even though these look like strings, they are not:
Can you write Python code for it?

for token in doc:
    print(token, "\t", type(token))


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
Text, sentences and noun_chunks
If you call the dir() function on a Doc object, you will see that it has a range of methods and attributes. You can read more about them in the documentation. Below, we highlight three of them: text, sents and noun_chunks.
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
The benefit of converting it to a list is that we can use indices to select certain sentences. For example, in the following we only print some information about the tokens in the second sentence.
Can you write Python code for it?

# Print some information about the tokens in the second sentence.
sentences = list(doc.sents)
for token in sentences[1]:
    data = '\t'.join([token.orth_,
                      token.lemma_,
                      token.pos_,
                      token.tag_,
                      str(token.i),    # Turn index into string
                      str(token.idx)]) # Turn index into string
    print(data)


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
These Token objects have many useful methods and attributes, which we can list by using dir(). We haven't really talked about attributes during this course, but while methods are operations or activities performed by that object, attributes are 'static' features of the objects. Methods are called using parantheses (as we have seen with str.upper(), for instance), while attributes are indicated without parantheses. We will see some examples below.
You can find more detailed information about the token methods and attributes in the documentation.
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
You can create a Span object from the slice doc[start : end]. For instance, doc[2:5] produces a span consisting of tokens 2, 3 and 4. Stepped slices (e.g. doc[start : end : step]) are not supported, as Span objects must be contiguous (cannot have gaps). You can use negative indices and open-ended ranges, which have their normal Python semantics.
Can you write Python code for it?

# Create a Span
a_slice = doc[2:5]
print(a_slice, type(a_slice))

# Iterate over Span
for token in a_slice:
    print(token.lemma_, token.pos_)


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
nlp is now a Python object representing the English NLP pipeline that we can use to process a text.
EXTRA: Larger models
For English, there are three models ranging from 'small' to 'large':

en_core_web_sm
en_core_web_md
en_core_web_lg

By default, the smallest one is loaded. Larger models should have a better accuracy, but take longer to load. If you like, you can use them instead. You will first need to download them.
Can you write Python code for it?

#%%bash
#python -m spacy download en_core_web_md

#%%bash
#python -m spacy download en_core_web_lg

# uncomment one of the lines below if you want to load the medium or large model instead of the small one
# nlp = spacy.load('en_core_web_md')
# nlp = spacy.load('en_core_web_lg')


I am working on the file "Chapters-colab/Chapter_19_More_about_Natural_Language_Processing_Tools_(spaCy).ipynb".
The first task is:
sents can be used to get all the sentences. Notice that it will create a so-called 'generator'. For now, you don't have to understand exactly what a generator is (if you like, you can read more about them online). Just remember that we can use generators to iterate over an object in a fast and efficient way.
Can you write Python code for it?

# Get all the sentences as a generator
print(doc.sents, type(doc.sents))

# We can use the generator to loop over the sentences; each sentence is a span of tokens
for sentence in doc.sents:
    print(sentence, type(sentence))


I am working on the file "Chapters-colab/Chapter_20_Visualization_and_Statistics.ipynb".
The first task is:
.
.
.
.
Surprise! It doesn't. This is a typical bimodal distribution with two peaks. Going back to the original article, this is also mentioned in their discussion:

One concern, for instance, is that concreteness and abstractness may be not the two extremes of a quantitative continuum (reflecting the degree of sensory involvement, the degree to which words meanings are experience based, or the degree of contextual availability), but two qualitatively different characteristics. One argument for this view is that the distribution of concreteness ratings is bimodal, with separate peaks for concrete and abstract words, whereas ratings on a single, quantitative dimension usually are unimodal, with the majority of observations in the middle (Della Rosa et al., 2010; Ghio, Vaghi, & Tettamanti, 2013).

It is commonly known in the literature on concreteness that concreteness ratings are (negatively) correlated with word length: the longer a word, the more abstract it typically is. Let's try to visualize this relation. We can plot the data using a regression plot to verify this. In addition, we're using a Pandas DataFrame to plot the data. You could also just use sns.regplot(word_length, rating, x_jitter=0.4).
Can you write Python code for it?

# Create two lists of scores to correlate.
word_length = []
rating = []
for entry in concreteness_entries:
    word_length.append(len(entry['Word']))
    rating.append(entry['Conc.M'])

# Create a Pandas Dataframe.
# I am using this here, because Seaborn adds text to the axes if you use DataFrames.
# You could also use pd.read_csv(filename,delimiter='\t') if you have a file ready to plot.
df = pd.DataFrame.from_dict({"Word length": word_length, "Rating": rating})

# Plot a regression line and (by default) the scatterplot.
# We're adding some jitter because all the points fall on one line.
# This makes it difficult to see how densely 'populated' the area is.
# But with some random noise added to the scatterplot, you can see more clearly
# where there are many dots and where there are fewer dots.
sns.regplot('Word length', 'Rating', data=df, x_jitter=0.4);


I am working on the file "Chapters-colab/Chapter_20_Visualization_and_Statistics.ipynb".
The first task is:
2.3. Common plots
Example 1: Line plot Let's create our first (line) plot:
Can you write Python code for it?


I am working on the file "Chapters-colab/Chapter_20_Visualization_and_Statistics.ipynb".
The first task is:
5. Take home message: The steps of visualization
Now you've seen several different plots, hopefully the general pattern is becoming clear: visualization typically consists of three steps:

Load the data.
Organize the data in such a way that you can feed it to the visualization function.
Plot the data using the function of your choice.

There's also an optional fourth step: After plotting the data, tweak the plot until you're satisfied. Of these steps, the second and fourth are usually the most involved.
6. Optional: On your own
If you would like to practice, here is an exercise with data from Donald Trump's Facebook page. The relevant file is Data/Trump-Facebook/FacebookStatuses.tsv. Try to create a visualization that answers one of the following questions:

How does the number of responses to Trump's posts change over time?
What webpages does Donald Trump link to, and does this change over time? Which is the most popular? Are there any recent newcomers?
What entities does Trump talk about?
Starting March 2016 (when the emotional responses were introduced on Facebook), how have the emotional responses to Trumps messages developed?
[Question of your own.]

Try to at least think about what kind of visualization might be suitable to answer these questions, and we'll discuss this question in class on Monday. More specific questions:

What kind of preprocessing is necessary before you can start visualizing the data?
What kind of visualization is suitable for answering these questions?
What sort of chart would you choose?
How could you use color to improve your visualization?

What might be difficult about visualizing this data? How could you overcome those difficulties?
Can you write Python code for it?

# Open the data.

# Process the data so that it can be visualized.

# Plot the data.

# Modify the plot.


I am working on the file "Chapters-colab/Chapter_20_Visualization_and_Statistics.ipynb".
The first task is:
For any kind of ratings, you can typically expect the data to have a normal-ish distribution: most of the data in the middle, and increasingly fewer scores on the extreme ends of the scale. We can check whether the data matches our expectation using a histogram.
Can you write Python code for it?

scores = []
for entry in concreteness_entries:
    scores.append(entry['Conc.M'])

# Plot the distribution of the scores.
sns.distplot(scores, kde=False);


I am working on the file "Chapters-colab/Chapter_20_Visualization_and_Statistics.ipynb".
The first task is:
If all went alright, you see a graph above this block. Try changing the numbers in the vals list to see how it affects the graph. Plotting is as simple as that!
Example 2: Column chart Now, let's try plotting some collected data. Suppose we did a survey to ask people for their favorite pizza. We store the result in a dictionary:
Can you write Python code for it?

counts = {
    'Calzone': 63,
    'Quattro Stagioni': 43,
    'Hawaii': 40,
    'Pepperoni': 58,
    'Diavolo': 63,
    'Frutti di Mare': 32,
    'Margarita': 55,
    'Quattro Formaggi': 10,
}