I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
So far, we have only used a variable called x. Nevertheless, we are entirely free to change the names of our variables, as long as these names do not contain strange characters, such as spaces, numbers or punctuation marks. (Underscores, however, are allowed inside names!) In the following block, we assign the outcome of our calculation to a variable that has a more meaningful name than the abstract name x.
Can you write Python code for it?

seconds_in_seven_weeks = 70560
print(seconds_in_seven_weeks)


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
So, how many ways are there to comment on your code in Python?

What we have learnt
To finish this section, here is an overview of what we have learnt. Go through the list and make sure you understand all the concepts.

concatenation
index
slicing
zero-indexed numbering
len()
type casting: int() and str()
type()
code commenting via hashtags and triple double quotes

Final Exercises Chapter 1
Inspired by Think Python by Allen B. Downey (http://thinkpython.com), Introduction to Programming Using Python by Y. Liang (Pearson, 2013). Some exercises below have been taken from: http://www.ling.gu.se/~lager/python_exercises.html.

Ex. 1: Suppose the cover price of a book is 24.95 EUR, but bookstores get a 40 percent discount. Shipping costs 3 EUR for the first copy and 75 cents for each additional copy. What is the total wholesale cost for 60 copies? Print the result in a pretty fashion, using casting where necessary!
Can you write Python code for it?


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
Some of the arithmetic operators we saw earlier can also be used to do useful things with strings. Both the multiplication operator (*) and the addition operator (+) provide interesting functionality for dealing with strings, as the block below illustrates.
Can you write Python code for it?

original_string = "bla"
new_string = 2*original_string
print(new_string)
new_string = new_string+"h"
print(new_string)


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
Strings are called strings because they consist of a series (or 'string') of individual characters. We can access these individual characters in Python with the help of 'indexing', because each character in a string has a unique 'index'. To print the first letter of your name, you can type:
Can you write Python code for it?

first_letter = name[0]
print(first_letter)


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
Suppose that we buy a new book for the office today: we can now update our book count accordingly, by adding one to our previous count:
Can you write Python code for it?

number_of_books = number_of_books + 1
print(number_of_books)


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
Take a look at the string "Mr White". We use the index 0 to access the first character in the string. This might seem odd, but  remember that all indexes in Python start at zero. Whenever you count in Python, you start at 0 instead of 1. Note that the space character gets an index too, namely 2. This is something you will have to get used to!
Because you know the length of your name you can ask for the last letter of your name:
Can you write Python code for it?

last_letter = name[# fill in the last index of your name (tip indexes start at 0)]
print(last_letter)


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
The 0 index is optional, so we could just as well type in name[:2]. This says: take all characters of name until you reach index 2 (i.e. up to the third letter, but not including the third letter). We can also start at index 2 and leave the end index unspecified:
Can you write Python code for it?

without_first_two_letters = name[2:]
print(without_first_two_letters)


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
To access the last character in a string you have to use the index [-1]. Alternatively, there is the len() command which returns the length of a string:
Can you write Python code for it?


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
Updates like these happen a lot. Python therefore provides a shortcut and you can write the same thing using +=.
Can you write Python code for it?

number_of_books += 5
print(number_of_books)


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
Using the operators we have learned about above, we can change the variables in our code as many times as we want. We can assign new values to old variables, just like we can put new or more things in the boxes which we already had. Say, for instance, that yesterday we counted how many books we have in our office and that we stored this count in our code as follows:
Can you write Python code for it?


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
Variables are also case sensitive, accessing months is not the same as Months
Can you write Python code for it?


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
What we have learnt
To finish this section, here is an overview of the concepts you have learnt. Go through the list and make sure you understand all the concepts.

variable
value
assignment operator (=)
difference between variables and values
integers vs. floats
operators for multiplication (*), subtraction (-), addition (+), division (/)
the shortcut operators: +=, -=, *=, /=
print()

Text strings
So far, we have only worked with variables that contain numbers (integers like -5 or 72 or floats like 45.89 or -5.609). Note, however, that variables can also contain other things than numbers. Many disciplines within the humanities work on texts. Quite naturally, programming skills for the humanities will have to focus a lot on manipulating texts. Have a look at the code block below, for instance. Here we put text, namely the title of a book, as a value inside the variable book. Then, we print what is inside the book variable.
Can you write Python code for it?

book = "The Lord of the Flies"
print(book)


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
You will undoubtedly remember from your math classes in high school that there is something called 'operator precedence', meaning that multiplication, for instance, will always be executed before subtraction. In Python you can explicitly set the order in which arithmetic operations are executed, using round brackets. Compare the following lines of code:
Can you write Python code for it?


I am working on the file "Chapter 1 - Variables.ipynb".
The first task is:
You're starting to become a real expert in indexing strings. Now what if we would like to find out what the last two or three letters of our name are? In Python we can use so-called 'slice-indexes' or 'slices' for short. To find the first two letters of our name we type in:
Can you write Python code for it?

first_two_letters = name[0:2]
print(first_two_letters)


I am working on the file "Chapter 10 - Position & Momentum.ipynb".
The first task is:
Example 10.3
Can you write Python code for it?

p = Symbol('p', real=True)

# Solution
A = integrate(1/sqrt(2*pi*a*h)*exp(-I*p*x/h)*cos((pi*x)/(2*a)),(x,-a,a), conds='none')

# Solution
A

psi_p = sqrt(2*a*pi/h) * 2/(pi**2 - (2*p*a/h)**2) * cos(p*a/h)
psi_p

psi_p == sqrt(2*a*pi/h)*2/(pi**2 - (2*p*a/h)**2) * cos(p*a/h)


I am working on the file "Chapter 10 - Position & Momentum.ipynb".
The first task is:
So $\Delta x^2 = a^2 - 0^2$ therefore $\Delta x = a$
Problem 10.17:
Now find the momentum representation of the state from 10.3
Can you write Python code for it?

p = symbols('p', nonzero=True, real=True, positive=True)
B = integrate(sqrt(1/(2*pi*h))*exp(-I*p*x/h)*psi2,(x,-oo,oo))
B

B.simplify()


I am working on the file "Chapter 10 - Position & Momentum.ipynb".
The first task is:
So this tells us the normalization constant should be $c=\frac{1}{\sqrt{a}}$. Check that it is normalized if we do that:
Can you write Python code for it?

psi = 1/sqrt(a)*cos((pi*x)/(2.0*a))  # notice we can name the expression something useful.
B = psi**2
B.integrate( (x,-a,a), conds='none')


I am working on the file "Chapter 10 - Position & Momentum.ipynb".
The first task is:
There are two ways to use the integrate function. In one line, like integrate(x,(x,0,1)) or by naming an expression and then integrating it over a range:
A = (c*cos((pi*x)/(2.0*a)))**2
A.integrate((x,-a,a),conds='none')
We'll use both, at different times. For longer expressions, the second form can be easier to read and write.
First, just try the following, then we'll re-create some examples in the book.
Can you write Python code for it?


I am working on the file "Chapter 10 - Position & Momentum.ipynb".
The first task is:
This agrees with the book after we notice that we had to force $p$ to be positive in order to get the integral to converge. The book has $|p|$ in the argument of the exponent to reflect this constraint.
Problem 10.13
Can you write Python code for it?

psi = 1/sqrt(2*a)*sech(x/a)
dpsi = diff(psi,x)
dpsi

ddpsi = diff(dpsi,x)
ddpsi = ddpsi.simplify()
ddpsi

expect_p = integrate(psi*ddpsi,(x,-oo,oo))
expect_p


I am working on the file "Chapter 10 - Position & Momentum.ipynb".
The first task is:
Use the diff function to take a derivative of a symbolic expression. For example:
Can you write Python code for it?

diff(x**2, x)

# Solution
-1j*h*diff( 1/a*cos((pi*x)/(2*a)) ,x)

# Solution
B1 = (pi*h/(2*a))**2 * (cos((pi*x)/(2*a)))**2
B1.integrate( (x,-a,a), conds='none' )


I am working on the file "Chapter 10 - Position & Momentum_blank.ipynb".
The first task is:
Example 10.2
Can you write Python code for it?


I am working on the file "Chapter 10 - Position & Momentum_blank.ipynb".
The first task is:
The cell below will return an odd set of conditions on the result. This is because the solver doesn't want to assume anything about a and there is a special case where the answer would be different. If you look closely though, that special case isn't physically realistic so to igore these special conditions, we add conds='none'. The next cell down does what you'd expect. From here on out, just add this to the integrate function and we'll get what we expect.
Can you write Python code for it?


I am working on the file "Chapter 10 - Position & Momentum_blank.ipynb".
The first task is:
Use the diff function to take a derivative of a symbolic expression. For example:
Can you write Python code for it?

diff(x**2, x)

# Solution goes here

# Solution goes here


I am working on the file "Chapter 12 - SHO.ipynb".
The first task is:
And a thermal state:
Can you write Python code for it?

# A thermal state has decaying amplitudes
plot_fock_distribution(thermal_dm(20,2))


I am working on the file "Chapter 12 - SHO.ipynb".
The first task is:
Define a coherent state with $\alpha = 2:$
Can you write Python code for it?


I am working on the file "Chapter 12 - SHO.ipynb".
The first task is:
Expectation value $\langle n\rangle$:
Can you write Python code for it?

psi.dag()*n*psi

# Convert to a density matrix:
psi_dm = ket2dm(psi)

psi*psi.dag()

# Fock state distribution of the coherent state:
plot_fock_distribution(psi_dm)


I am working on the file "Chapter 13 - Hydrogen Solutions.ipynb".
The first task is:
Define some variables, radial, polar, azimuthal, time, and two frequencies:
Can you write Python code for it?


I am working on the file "Chapter 13 - Hydrogen Solutions.ipynb".
The first task is:
Look at a few of the radial equations and the spherical harmonics
Notice that instead of Ylm the name is Ynm... the arguments to the function are still the quantum numbers l and m
Can you write Python code for it?

R_nl(1, 0, r, 1)  # the n = 1, l = 0 radial function

Ynm(1,0,theta,phi).expand(func=True)  # the l = 0, m = 0 spherical harmonic


I am working on the file "Chapter 13 - Hydrogen Solutions.ipynb".
The first task is:
No surprise, the average z position of the electron in the hydrogen atom is 0.
Now for problem 13.21
find $\langle z \rangle(t)$. Use the same integral, but add a time-dependent piece to each term in the wavefunction, add them together and multiply by the complex conjugate.
Can you write Python code for it?

psi = 1/sqrt(2)*(psi100*exp(1j*w1*t) + psi210*exp(1j*w2*t))
psi_conj = 1/sqrt(2)*(psi100*exp(-1j*w1*t) + psi210*exp(-1j*w2*t))

expect2 = integrate(r**2*sin(theta)* (r*cos(theta)) * psi*psi_conj,(r,0,oo),(theta,0,pi),(phi,0,2*pi))

plot(re(expect2.subs({w2:2, w1:1})),(t,0,10))


I am working on the file "Chapter 13 - Hydrogen Solutions.ipynb".
The first task is:
Now do the $|210\rangle$ state:
Can you write Python code for it?

psi210 = R_nl(2, 1, r, 1)*Ynm(1,0,theta,phi).expand(func=True)

psi210  # check how it looks


I am working on the file "Chapter 13 - Hydrogen Solutions.ipynb".
The first task is:
Write the equation for the $|nlm\rangle = |100\rangle$ state. Use the sympy method .expand(func=True) to convert to the actual expression. To create this state, we combine the Radial function and the Ylm function. Make sure to set n, l, and m to the correct values. The fourth argument to R_nl is Z which we set to 1 since we are talking about a 1-proton nucleus.
The combination of R_nl and Ynm should look like the following (replace N, L, and M with the appropriate values):
R_nl(N, L, r, 1)*Ynm(L, M, theta, phi).expand(func=True)
Can you write Python code for it?

# this is the |100> state:
psi100 = R_nl(1, 0, r, 1)*Ynm(0,0,theta,phi).expand(func=True)

psi100  # check to see how it looks as an expression


I am working on the file "Chapter 13 - Hydrogen.ipynb".
The first task is:
Look at a few of the radial equations and the spherical harmonics
Notice that instead of Ylm the name is Ynm... the arguments to the function are still the quantum numbers l and m
Can you write Python code for it?

R_nl(1, 0, r, 1)  # the n = 1, l = 0 radial function

Ynm(0,0,theta,phi).expand(func=True)  # the l = 0, m = 0 spherical harmonic


I am working on the file "Chapter 13 - Hydrogen.ipynb".
The first task is:
Note, if you compare these to listed solutions (for example at http://hyperphysics.phy-astr.gsu.edu/hbase/quantum/hydwf.html#c3) you see that there are not any factors of $a_0$. This is because the R_nl function is defined in units of $a_0$. $a_0$ is the Bohr Radius: http://en.wikipedia.org/wiki/Bohr_radius
Can you write Python code for it?

psi211 = R_nl(2, 1, r, 1)*Ynm(1,1,theta,phi).expand(func=True)
psi211


I am working on the file "Chapter 13 - Hydrogen.ipynb".
The first task is:
Write the equation for the $|nlm\rangle = |100\rangle$ state. Use the sympy method .expand(func=True) to convert to the actual expression. To create this state, we combine the Radial function and the Ylm function. Make sure to set n, l, and m to the correct values. The fourth argument to R_nl is Z which we set to 1 since we are talking about a 1-proton nucleus.
The combination of R_nl and Ynm should look like the following (replace N, L, and M with the appropriate values):
R_nl(N, L, r, 1)*Ynm(L, M, theta, phi).expand(func=True)
Can you write Python code for it?


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
A list acts like some kind of container where we can store all kinds of information. We can access a list using indexes and slices. We can also add new items to a list. For that you use the function append(). Let's see how that works. Say we want to keep a list of all our good reads. We first declare an empty list using square brackets. Next, we add some good books to the list:
Can you write Python code for it?

# start with an empty list
good_reads = []
good_reads.append("The Hunger games")
print(good_reads)
good_reads.append("A Clockwork Orange")
print(good_reads)


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
DIY

Can you come up with your own sentence my_sentence and split it into words along spaces? Print the new list of words.
You can recognize functions because they are always followed by (round brackets). Apart from the split() function, we already encountered other functions, also in the previous chapter. Which ones? Can you describe their functionality? Are there any differences in terms of syntax when you compare these to split()?
Is there a difference in length between the variables sentence and words? (Use functions to find this out!)
Can you write Python code for it?


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
DIY
Update the good_reads collection with some of your own books and give them all a score and a publication year by nesting lists. Can you print out the score you gave to the first book in the list? And the publication year of the third item in your list? (Hint: you can pile up indexes using square brackets!)
Can you write Python code for it?

# insert your code here
print(good_reads[2])


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
DYI
Here's another small DIY! Add two new titles to the list of good_reads. Then, try to change the title of the second book in our good reads collection:
Can you write Python code for it?


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
Do you get the syntax that goes with the append() function? The list we wish to append the item to goes first and we join the append() function to this list using a dot (.). In between the round brackets that go with the function name, we place the actual string that we wish to add to the list. We call such a input value an 'argument' or a 'parameter' that we 'pass' to a function. Next, the function will return a 'return value'. Make sure that you are familiar with this terminology because you will often come across such terms when you look for help online!
Now, if for some reason we don't like a particular book anymore, we can change it as follows using the old item's index:
Can you write Python code for it?

good_reads[1] = "Pride and Prejudice"
print(good_reads)


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
Ex. 2: Consider the lookup dictionary below. The following letters are still missing from it: 'k':'kilo', 'l':'lima', 'm':'mike'. Add them to lookup! Could you spell the word "marvellous" in code language now? Collect these codes into the list object msg. Next, join the items in this list together with a comma and print the spelled out version!

lookup = {'a':'alfa', 'b':'bravo', 'c':'charlie', 'd':'delta', 'e':'echo', 'f':'foxtrot', 'g':'golf', 'h':'hotel', 'i':'india', 'j':'juliett', 'n':'november', 'o':'oscar', 'p':'papa', 'q':'quebec', 'r':'romeo', 's':'sierra', 't':'tango', 'u':'uniform', 'v':'victor', 'w':'whiskey', 'x':'x-ray', 'y':'yankee', 'z':'zulu'}
Can you write Python code for it?


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
Ex. 3: Collect the code terms in the lookup dict (alpha, bravo, ...) from the previous exercise into a list called code_words. Is this list alphabetically sorted? No? Then make sure that this list is sorted alphabetically. Now remove the items victor, india and papa. Append the words pigeon and potato at the end of this list. Combine this new list of items into a single string, using a semicolon as a delimiter and print this string.
Can you write Python code for it?


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
If we try to remove a book that is not in our collection, Python raises an error to signal that something is wrong.
Can you write Python code for it?


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
In a way, this is similar to what we have seen before when we altered our book list. There we indexed the list using a integer to access a particular book. Here we directly use the title of the book. Note that the keys in a dictionary must be unique: why would that be?
DIY
Update the new good reads datastructure with your own books. Try to print out the score your gave for one of the books which you added.
Can you write Python code for it?


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
In many ways, list variables are very similar to strings. We can for example access its components using indexes and we can use slice indexes to access parts of the list. Let's try this out.
Write a small program that defines a variable first_word and assign to it the first word of our word list words from above. Do the same for the fifth word, the last word and the last but one word. Also, try to extract a slice from words and isolate the string of words between derived and Flying (the slice should not include derived and Flying). Also, make a slice of words that is identical to the title of the television series in words.
Can you write Python code for it?


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
Just as with strings, we can concatenate two lists using the + operator. Here is an example:
Can you write Python code for it?

# first we specify two lists of strings:
good_reads = ["The Hunger games", "A Clockwork Orange",
              "Pride and Prejudice", "Water for Elephants",
              "The Shadow of the Wind", "Bel Canto"]
bad_reads = ["Fifty Shades of Grey", "Twilight", "The Hunger games"]

# then we combine them
all_reads = good_reads + bad_reads
print(all_reads)

good_reads += bad_reads
print(good_reads)


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
Lists are a really powerful way of dealing with your data in Python. Let's explore some other ways in which we can manipulate lists.
remove()
Let's assume our good read collection has grown a lot and we would like to remove some of the books from the list. Python provides the function remove() that you can call on a list and which takes as argument the item we would like to remove.
Can you write Python code for it?

good_reads = ["The Hunger games", "A Clockwork Orange",
             "Pride and Prejudice", "Water for Elephants", "Illias", "Water for Elephants", "Water for Elephants"]
print(good_reads)
good_reads.remove("Water for Elephants")
print(good_reads)
good_reads.remove("Water for Elephants")
print(good_reads)


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
Python has two functions for changing the case of a string. lower() converts a string to lowercase characters and upper() returns an uppercased version:
Can you write Python code for it?

my_string = "AllCaps"
print(my_string)
my_string_upper = my_string.upper()
print(my_string_upper)
my_string_lower = my_string.lower()
print(my_string_lower)
my_string_capped = my_string.capitalize()
print(my_string_capped)


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
Python only prints the first character of our sentence. (Think about this if you do not understand why.) We can transform our sentence into a list of words (represented by strings) using the split() function as follows:
Can you write Python code for it?


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
Similarly we can ask for the values:
Can you write Python code for it?


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
Take a close look at the new syntax. Notice the curly brackets and the colons. To look up the value of a given key, we `index' the dictionary using that key (again, between square brackets):
Can you write Python code for it?

description = my_dict["sword"]
print(description)


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
We can put this to use to enhance our good read collection with a score for every book we have. An entry in our collection will consist of a score within the range of 1 and 10 and the title of our book. The first element is the title; the second the score [title, score]. We initialize an empty list:
Can you write Python code for it?


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
What we have learnt
To finish this section, here is an overview of the new concepts you have learnt. Go through them and make sure you understand them all.

lists
nested lists
mutable versus immutable
.split() vs. .join()
.append()
.remove()
.sort()
.upper() vs. .lower()

Dictionaries
Our little good reads collection is starting to look quite impressive and we can perform all kinds of manipulations on it. Now, imagine that our list is large and we would like to look up the score we gave to a particular book. How are we going to find that book? For this purpose Python provides another more appropriate data structure, named dictionary. A dictionary is similar to the dictionaries you have at home. It consists of entries, or keys, that hold a value. Let's define one:
Can you write Python code for it?

my_dict = {"book": "physical objects consisting of a number of pages bound together",
           "sword": "a cutting or thrusting weapon that has a long metal blade",
           "pie": "dish baked in pastry-lined pan often with a pastry top"}
my_dict['word'] = "a cutting or thrusting weapon that has a long metal blade"
print(my_dict)


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
keys(), values()
To retrieve a list of all the books we have in our collection, we can ask the dictionary to return its keys as a list:
Can you write Python code for it?

vals = good_reads.values()
vals = list(vals)


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
nested lists
Until now our lists only consisted of strings. However, a list can contain all kinds of data types, such as integers and even lists! Do you understand what is happening in the following example? Have a close look at the square brackets used.
Can you write Python code for it?

nested_list = [[1, 2, 3, 4], [5, 6, 7, 8]]
print(nested_list[0])
print(nested_list[0][0])
print(nested_list[1][2])
print(nested_list[0][:-2])


I am working on the file "Chapter 2 - Collections.ipynb".
The first task is:
replace()
The replace() function is another function which can be called on a string. It will replace all occurrences of a specified substring with another string. Consider the lines in the code block below - and mind the order in which you pass the arguments to the function!
Can you write Python code for it?

text = "You can not compare apples and pears"
text = text.replace("pears", "apples")
text = text.replace("not ", "")
print(text)


I am working on the file "Chapter 2 - Data Processing.ipynb".
The first task is:
Converting categorical data to numerical features
Can you write Python code for it?

cat_data = array(['male', 'female', 'male', 'male', 'female', 'male', 'female', 'female'])

def cat_to_num(data):
    categories = unique(data)
    features = []
    for cat in categories:
        binary = (data == cat)
        features.append(binary.astype("int"))
    return features

cat_to_num(cat_data)


I am working on the file "Chapter 2 - Data Processing.ipynb".
The first task is:
Simple feature engineering of the Titanic dataset
Can you write Python code for it?

cabin_data = array(["C65", "", "E36", "C54", "B57 B59 B63 B66"])

def cabin_features(data):
    features = []
    for cabin in data:
        cabins = cabin.split(" ")
        n_cabins = len(cabins)
        # First char is the cabin_char
        try:
            cabin_char = cabins[0][0]
        except IndexError:
            cabin_char = "X"
            n_cabins = 0
        # The rest is the cabin number
        try:
            cabin_num = int(cabins[0][1:])
        except:
            cabin_num = -1
        # Add 3 features for each passanger
        features.append( [cabin_char, cabin_num, n_cabins] )
    return features

cabin_features(cabin_data)


I am working on the file "Chapter 2 - Iterables Consistency.ipynb".
The first task is:
As you can see, it doesn't generate the complete list like the listcomp above. Genexps don't produce entire lists in memory. You need to iterate over them, and then you will get the item one by one. No one is keeping the entire list!
Can you write Python code for it?

for article in couleurs_et_tailles:
    print(article)


I am working on the file "Chapter 2 - Iterables Consistency.ipynb".
The first task is:
Next, the chapter moves on to list.sort() vs. sorted(). I kinda like sorted because it works with everything including generators, and it returns a new list. Whereas list.sort() is more limited because it is an in-place sort which returns None.
Next is about bisect, which is interesting but one can read the documentation easily and experiment with it.
Also obviously important is knowing which data structure meets your needs. A bit like knowing the Java Collections Framework again. What I don't really like about Python here is that the nicer and supposedly basic things are not part of the standard library. For example, SortedContainers is by all accounts an excellent package. This is something we've been taking for granted in Java for example.
One notable thing from sorting that's different (and I agree with the author, is nicer) than in other languages is using the "key" argument instead of the usual "Comparator" function/object that returns the standard -1, 0, 1. Again, more examples:
Can you write Python code for it?

jours = ["lundi", "mardi", "mercredi", "jeudi", "vendredi", "samedi", "dimanche"] # already ordered

print(sorted(jours))
print(sorted(jours, reverse=True))
print(sorted(jours, key=len, reverse=True)) # by length of string, descend
print(sorted(jours, reverse=True, key=lambda x: jours.index(x))) # by actual order of week, descending


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
Apparently, the test vector's nearest neighbour can be found in first position; this is in fact a text by the author we where looking for:
Can you write Python code for it?

print('Closest text:', titles[nn_idx])
print('Closest author:', authors[nn_idx])


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
As is clear from the shape info, we have obtained a 9 by 9 matrix, which holds the distance between each pair of texts. Note that the distance from a text to itself is of course zero:
Can you write Python code for it?


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
As we can see, Jane Austen's novels form a tight and distinctive cloud; apparantly Dickens and Thackeray are more difficult to tell apart. The actual distance between nodes is hinted at on the horizontal length of the branches (i.e. the values on the x-axis in this plot). The previous code blocks used the Manhattan city block distance, a very simple distance metric which is also used in the calculation of Burrows's Delta. Note that we can easily switch to, for instance, the Euclidean distance:
Can you write Python code for it?

dm = squareform(pdist(X, 'euclidean'))
linkage_object = linkage(dm, method='ward')
d = dendrogram(Z=linkage_object, labels=authors, orientation='right')


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
As you can see we loaded a list of titles, authors, words and a frequency table - which is named capital X. These lists are perfectly matched: the authors and titles can for instance be easily zipped together:
Can you write Python code for it?

print('This dummy corpus holds:')
for title, author in zip(titles, authors):
    print('\t-', title, 'by', author)


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
As you can see, this yields a list of 8 values: the respective distances between our test_vector and all training items. Now, we can use some convenient numpy functions, to find out which training texts shows the minimal distance to our anonymous text:
Can you write Python code for it?

import numpy as np
nearest_dist = np.min(dists)
print('Smallest distance: ', nearest_dist)


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
At this point, however, we still don't know to which exact training item our test item is closest. For this purpose, we can the use the argmin() function. This will not return the actual minimal distance, but rather the index at which the smallest value can be found:
Can you write Python code for it?

nn_idx = np.argmin(dists) # index of the nearest neighbour
print('Index of nearest neighbour: ', nn_idx)


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
If you are interested in getting a version of this matrix which is easier to deal with, pandas is an interesting library. Basically, it wraps a lot of functionality around numpy matrixes, and makes it easier to access, for instance, columns using actual names, instead of less intuitive indices. Thus, it brings a lot of functionality to Python which you might know from e.g. R. Pandas is imported as pd conventionally:
Can you write Python code for it?


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
Loading the data
It is time to get practical! In the data directory in the repository for this course, I have included a corpus representing novels by three famous British authors from the Victorian era: Jane Austen, Charles Dickens, and William Thackeray. In the next code block, I load these texts and turn them into a vectorized matrix. You can simple execute the code block and ignore it for the time being. In the next chapter, we will deep deeper into the topic of vectorization.
Can you write Python code for it?

import pickle
titles, authors, words, X = pickle.load(open("dummy.p", "rb"))


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
Or with a list comprehension:
Can you write Python code for it?

dists = [manhattan(test_vector, v) for v in X_train]
print(dists)


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
The lists of words matches the names of the columns in our frequency table. To select the frequencies for the pronoun 'my' in each text, we could therefore do:
Can you write Python code for it?

idx_my = words.index('my')
freqs_my = X[:, idx_my]
print(freqs_my)


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
This command will construct a nice table out of our data matrix, which can be easily indexed. The example with 'my' above:
Can you write Python code for it?


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
This metric can be used to calculate the cityblock distance between any two vectors, as follows:
Can you write Python code for it?


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
Thus, we obtain a single test vector, consisting of 30 features, and a 8 by 30 matrix of training texts: i.e. the texts of our 'known' authors. We can now calculate to which training text our unknown test text is closest, using the Manhattan distance implemented in scipy:
Can you write Python code for it?

from scipy.spatial.distance import cityblock as manhattan


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
To turn X into a pandas DataFrame (which is the most important object in pandas), we could do this:
Can you write Python code for it?

df = pd.DataFrame(X, columns=words, index=titles)


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
Using the authors as labels is of course also a good idea:
Can you write Python code for it?

from scipy.cluster.hierarchy import dendrogram
linkage_object = linkage(dm, method='ward')
d = dendrogram(Z=linkage_object, labels=authors, orientation='right')


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
We can now draw our dendrogram. Note that we annotate the outer leaf nodes in our tree (i.e. the texts) using the labels argument. With the orientation argument, we make sure that our dendrogram can be easily read:
Can you write Python code for it?

from scipy.cluster.hierarchy import dendrogram
linkage_object = linkage(dm, method='ward')
d = dendrogram(Z=linkage_object, labels=titles, orientation='right')


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
We can now run this function on our corpus. To obtain a nice and clean matrix, we apply squareform() to pdist()'s: result: like that, we obtain a matrix which has a row as well as a column for each of our original texts. This representation is a bit superfluous, because matrix[i][j] will be identical to matrix[j][i]. This is because most distance metrics in stylometry are symmetric (such as the cityblock or Manhattan distance used below): the distance of document B to document A is equal to the distance from document A to document B.
Can you write Python code for it?

dm = squareform(pdist(X, 'cityblock'))
print(dm.shape)


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
We now proceed to calculate the distance between our anonymous text and all texts in our training set:
Can you write Python code for it?

dists = []
for v in X_train:
    dists.append(manhattan(test_vector, v))
print(dists)


I am working on the file "Chapter 2 - Stepping up with SciPy.ipynb".
The first task is:
With pdist('cityblock'), we can calculate the pairwise distances between all rows in a single matrix; using sp.spatial.distance.cityblock(), we can calculate the same Manhattan distance between two vectors. What can we do when we would like to calculate the distances between all the rows in one matrix, to all the rows in another matrix? This is a valid question, since often we would like to attribute more than one anonymous text to a series of candidate authors. This is where scipy's cdist() function comes in handy:
Can you write Python code for it?

from scipy.spatial.distance import cdist


I am working on the file "Chapter 2/EstimatorCV Objects.ipynb".
The first task is:
Efficient hyper-parameter selection for Lasso
Can you write Python code for it?

from sklearn.datasets import make_regression
X, y = make_regression(noise=60, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

from sklearn.linear_model import Lasso, LassoCV
lasso = Lasso().fit(X_train, y_train)
print("lasso score with default alpha: %f" % lasso.score(X_test, y_test))

lassocv =  LassoCV().fit(X_train, y_train)
print("lasso score with automatic alpha: %f" % lassocv.score(X_test, y_test))

grid_search = GridSearchCV(Lasso(), param_grid={'alpha': np.logspace(-5, 1, 20)})
grid_search.fit(X_train, y_train)
print("lasso score with grid-searched alpha: %f" % grid_search.score(X_test, y_test))

print("best alpha found by LassoCV: %f" % lassocv.alpha_)
print("best alpha found by GridSearchCV: %f" % grid_search.best_params_['alpha'])

%timeit Lasso().fit(X_train, y_train)

%timeit LassoCV().fit(X_train, y_train)

%timeit grid_search.fit(X_train, y_train)


I am working on the file "Chapter 2/EstimatorCV Objects.ipynb".
The first task is:
EstimatorCV Objects for Efficient Parameter Search
Recursive Feature Eliminiation
Can you write Python code for it?

from sklearn.datasets import load_iris
from sklearn.cross_validation import train_test_split

iris = load_iris()
X_train, X_test, y_train, y_test = train_test_split(iris.data, iris.target, random_state=0)

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

feature_elimination_lr = RFE(LogisticRegression(C=100), n_features_to_select=2)

feature_elimination_lr.fit(X_train, y_train)
feature_elimination_lr.score(X_test, y_test)

from sklearn.grid_search import GridSearchCV

param_grid = {'n_features_to_select': range(1, 5)}
grid_search = GridSearchCV(feature_elimination_lr, param_grid, cv=5)
grid_search.fit(X_train, y_train)
grid_search.score(X_test, y_test)

grid_search.best_params_

from sklearn.feature_selection import RFECV
rfecv = RFECV(LogisticRegression(C=100)).fit(X_train, y_train)
rfecv.score(X_test, y_test)

rfecv.n_features_


I am working on the file "Chapter 2/Linear models.ipynb".
The first task is:
Linear Regression
Can you write Python code for it?

from sklearn.linear_model import LinearRegression
linear_regression = LinearRegression().fit(X_train, y_train)
print("R^2 on training set: %f" % linear_regression.score(X_train, y_train))
print("R^2 on test set: %f" % linear_regression.score(X_test, y_test))

from sklearn.metrics import r2_score
print(r2_score(np.dot(X, true_coefficient), y))

plt.figure(figsize=(10, 5))
coefficient_sorting = np.argsort(true_coefficient)[::-1]
plt.plot(true_coefficient[coefficient_sorting], "o", label="true")
plt.plot(linear_regression.coef_[coefficient_sorting], "o", label="linear regression")

plt.legend()


I am working on the file "Chapter 2/Linear models.ipynb".
The first task is:
Linear models for classification
y_pred = x_test[0] * coef_[0] + ... + x_test[n_features-1] * coef_[n_features-1] + intercept_ > 0
The influence of C in LinearSVC
Can you write Python code for it?

from figures import plot_linear_svc_regularization
plot_linear_svc_regularization()


I am working on the file "Chapter 2/Linear models.ipynb".
The first task is:
Linear models for regression
y_pred = x_test[0] * coef_[0] + ... + x_test[n_features-1] * coef_[n_features-1] + intercept_
Can you write Python code for it?

from sklearn.datasets import make_regression
from sklearn.cross_validation import train_test_split

X, y, true_coefficient = make_regression(n_samples=80, n_features=30, n_informative=10, noise=100, coef=True, random_state=5)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=5)
print(X_train.shape)
print(y_train.shape)


I am working on the file "Chapter 2/Linear models.ipynb".
The first task is:
Multi-Class linear classification
Can you write Python code for it?

from sklearn.datasets import make_blobs

X, y = make_blobs(random_state=42)
plt.scatter(X[:, 0], X[:, 1], c=y)

from sklearn.svm import LinearSVC
linear_svm = LinearSVC().fit(X, y)
print(linear_svm.coef_.shape)
print(linear_svm.intercept_.shape)

plt.scatter(X[:, 0], X[:, 1], c=y)
line = np.linspace(-15, 15)
for coef, intercept in zip(linear_svm.coef_, linear_svm.intercept_):
    plt.plot(line, -(line * coef[0] + intercept) / coef[1])
plt.ylim(-10, 15)
plt.xlim(-10, 8)


I am working on the file "Chapter 2/Linear models.ipynb".
The first task is:
Ridge Regression (L2 penalty)
Can you write Python code for it?

from sklearn.linear_model import Ridge
ridge_models = {}
training_scores = []
test_scores = []

for alpha in [100, 10, 1, .01]:
    ridge = Ridge(alpha=alpha).fit(X_train, y_train)
    training_scores.append(ridge.score(X_train, y_train))
    test_scores.append(ridge.score(X_test, y_test))
    ridge_models[alpha] = ridge

plt.plot(training_scores, label="training scores")
plt.plot(test_scores, label="test scores")
plt.xticks(range(4), [100, 10, 1, .01])
plt.legend(loc="best")

plt.figure(figsize=(10, 5))
plt.plot(true_coefficient[coefficient_sorting], "o", label="true", c='b')

for i, alpha in enumerate([100, 10, 1, .01]):
    plt.plot(ridge_models[alpha].coef_[coefficient_sorting], "o", label="alpha = %.2f" % alpha, c=plt.cm.summer(i / 3.))

plt.legend(loc="best")


I am working on the file "Chapter 2/Support Vector Machines.ipynb".
The first task is:
Kernel SVMs
Predictions in a kernel-SVM are made using the formular
$$
\hat{y} = \alpha_0 + \alpha_1 y_1 k(\mathbf{x^{(1)}}, \mathbf{x}) + ... + \alpha_n y_n k(\mathbf{x^{(n)}}, \mathbf{x})> 0
$$
$$
0 \leq \alpha_i \leq C
$$
Radial basis function (Gaussian) kernel:
$$k(\mathbf{x}, \mathbf{x'}) = \exp(-\gamma ||\mathbf{x} - \mathbf{x'}||^2)$$
Can you write Python code for it?

from sklearn.metrics.pairwise import rbf_kernel
line = np.linspace(-3, 3, 100)[:, np.newaxis]
kernel_value = rbf_kernel([[0]], line, gamma=1)
plt.plot(line, kernel_value.T)

from figures import plot_svm_interactive
plot_svm_interactive()

svc = SVC().fit(X_train, y_train)
svc.score(X_test, y_test)

Cs = [0.001, 0.01, 0.1, 1, 10, 100]
gammas = [0.001, 0.01, 0.1, 1, 10, 100]

from sklearn.grid_search import GridSearchCV

param_grid = {'C': Cs, 'gamma' : gammas}
grid_search = GridSearchCV(SVC(), param_grid, cv=5)
grid_search.fit(X_train, y_train)

grid_search.score(X_test, y_test)

# We extract just the scores
scores = [x[1] for x in grid_search.grid_scores_]
scores = np.array(scores).reshape(6, 6)

plt.matshow(scores)
plt.xlabel('gamma')
plt.ylabel('C')
plt.colorbar()
plt.xticks(np.arange(6), param_grid['gamma'])
plt.yticks(np.arange(6), param_grid['C']);


I am working on the file "Chapter 2/Support Vector Machines.ipynb".
The first task is:
Support Vector Machines
Can you write Python code for it?

from sklearn.datasets import load_digits
from sklearn.cross_validation import train_test_split

digits = load_digits()
X_train, X_test, y_train, y_test = train_test_split(digits.data / 16., digits.target % 2, random_state=2)

from sklearn.svm import LinearSVC, SVC
linear_svc = LinearSVC(loss="hinge").fit(X_train, y_train)
svc = SVC(kernel="linear").fit(X_train, y_train)

np.mean(linear_svc.predict(X_test) == svc.predict(X_test))


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
Because our book is not in the collection, Python returns False. Let's do the same thing for a book that we know is in the collection:
Can you write Python code for it?


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
Can you do the same thing, but this time using the function len()?
Can you write Python code for it?


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
Can you spot the reasoning error in the following code?
Can you write Python code for it?

score = 98.0
if score >= 60.0:
    grade = 'D'
elif score >= 70.0:
    grade = 'C'
elif score >= 80.0:
    grade = 'B'
elif score >= 90.0:
    grade = 'A'
else:
    grade = 'F'
print(grade)


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
DIY

Can you write code that prints "This is an empty list" if the provided list does not contain any values?
Can you write Python code for it?

numbers = []
# insert your code here
if not numbers:
    print("Is an empty list")


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
First the if statement will be evaluated. Only if that statement turns out to be False the computer will proceed to evaluate the elif statement. If the elif statement in turn would prove to be False, the machine will proceed and execute the lines of code associated with the else statement. You can think of this coding structure as a decision tree! Remember: if somewhere along the tree, your machine comes across a logical expression which is true, it won't bother anymore to evaluate the remaining options!

DIY
Let's practice our new condition skills a little. Write a small program that defines a variable weight. If the weight is > 50 pounds, print "There is a $25 charge for luggage that heavy." If it is not, print: "Thank you for your business." If the weight is exactly 50, print: "Pfiew! The weight is just right!". Change the value of weight a couple of times to check whether your code works. (Tip: make use of the logical operators and if-elif-else tree! Make sure you use the correct indentation.)
Can you write Python code for it?


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
If one of the expressions evaluates to False, nothing will be printed:
Can you write Python code for it?

if ("a" in word) and ("z" in word):
    print("Both a and z are in " + word)


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
In the code block below, can you add an else statement that prints that none of the letters were found?
Can you write Python code for it?

if ("a" in word) and ("z" in word):
    print("a or z are in " + word)
else:
    print("None of these were found...")
# insert your code here


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
Indeed, it is in the collection! Back to our if statement. If the expression after if evaluates to True, our program will go on to the next line and print book + " is in the collection". Let's try that as well:
Can you write Python code for it?

if "Emma" in good_reads:
    print("Found it!")

if book in good_reads:
    print("Found it!")


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
Note how we can use round brackets to make the code more readable (but you can just as easily leave them out):
Can you write Python code for it?

word = "banana"
if ("a" in word) and ("b" in word):
    print("Both a and b are in " + word)


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
Notice that the last print statement is not executed. That is because the value we assigned to book is not in our collection and thus the part after if did not evaluate to True. In our little program above we used another statement besides if, namely else. It shouldn't be too hard to figure out what's going on here. The part after else will be evaluated if the if statement evaluated to False. In English: if the book is not in the collection, print that is is not.
Indentation!
Unlike other languages, Python does not make use of curly braces to mark the start and end of pieces of code, like if-statements. The only delimiter is a colon (:) and the indentation of the code (i.e. the use of whitespace). This indentation must be used consistently throughout your code. The convention is to use 4 spaces as indentation. This means that after you have used a colon (such as in our if statement) the next line should be indented by four spaces. (The shortcut for typing these 4 spaces in many editors is inserting a TAB.)
Sometimes we have various conditions that should all evaluate to something different. For that Python provides the elif statement. We use it similar to if and else. Note however that you can only use elif after an if statement! Above we asked whether a book was in the collection. We can do the same thing for parts of strings or for items in a list. For example we could test whether the letter a is in the word banana:
Can you write Python code for it?


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
Objects, such as strings or integers of lists are True, simply because they exist. Empty strings, lists, dictionaries etc on the other hand are False because in a way they do not exist -- an empty list is not really a list, right? This principle is often by programmers to, for example, only execute a piece of code if a certain list contains anything at all:
Can you write Python code for it?

numbers = [1, 2, 3, 4]
if numbers:
    print("I found some numbers!")


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
The relevant 'logical operators' that we used here are: <, <=, >,>=,==,!=. In Python-speak, we say that such logical expression gets 'evaluated' when you run the code. The outcome of such an evaluation is a 'binary value' or a so-called 'boolean' that can take only two possible values: True or False. You can assign such a boolean to a variable:
Can you write Python code for it?

greater = (5 > 2)
print(greater)
greater = 5 < 2
print(greater)
print(type(greater))


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
What we have learnt
To finish this section, here is an overview of the new functions, statements and concepts we have learnt. Go through them and make sure you understand what their purpose is and how they are used.

conditions
indentation
if
elif
else
True
False
empty objects are false
not
in
and
or
multiple conditions
==
<
>
!=
KeyError

Final Exercises Chapter 3
Inspired by Think Python by Allen B. Downey (http://thinkpython.com), Introduction to Programming Using Python by Y. Liang (Pearson, 2013). Some exercises below have been taken from: http://www.ling.gu.se/~lager/python_exercises.html.

Can you implement the following grading scheme in Python?
<img src="https://raw.githubusercontent.com/mikekestemont/python-course/master/images/grade.png">
Can you write Python code for it?


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
Write Python code that defines two numbers and prints the largest one of them. Use an if-then-else tree.
Can you write Python code for it?


I am working on the file "Chapter 3 - Conditions.ipynb".
The first task is:
if, elif and else
At the end of the previous chapter, we have talked about dictionaries, which are a kind of data structure which you will need a lot when writing Python. Recall our collection of good_reads from the previous chapter. Recall that we could use the key of an entry to retrieve the score of a book in our collection:
Can you write Python code for it?

good_reads = {"Emma":8, "Pride and Prejudice":10, "Sense and Sensibility":7, "Northanger Abbey":3}
score = good_reads["Sense and Sensibility"]
print(score)


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
After running the PCA algorithm, we can apply the reduction to our original data:
Can you write Python code for it?

X_bar = pca.transform(X)
print(X_bar.shape)


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
After running the clustering algorithm, we can now access the cluster labels, indicating to which of the two clusters each inidividual text has been assigned:
Can you write Python code for it?

cl_labels = clust.labels_
print(cl_labels)


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
Again, we first need to instantiate the clusters, before we can run it on our data. Note that we have the algorithm discern only the two most distinct clusters, using the n_clusters parameter.
Can you write Python code for it?

clust = AgglomerativeClustering(linkage='ward', n_clusters=2)
clust.fit(X_bar)


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
And now we add the text labels:
Can you write Python code for it?

fig, ax1 = sns.plt.subplots()
x1, x2 = X_bar[:,0], X_bar[:,1] # select coordinates via numpy indexing!
ax1.scatter(x1, x2, 100, edgecolors='none', facecolors='none')
for x, y, title in zip(x1, x2, titles):
    ax1.text(x, y, title, ha='center', va="center")


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
As you can see, a very similar structure arises from this analysis.
Adding loadings
One particular advantage of such techniques that they do not need to function as black boxes, but they can actually inform us a lot about as to why this specific structure arises. In PCA, this can be achieved by overlaying our plot with the so-called 'loadings' for the original words: these will show us how specific words contribute to the oppositions created in the plots above. The loadings can be retrieved from the PCA object as follows:
Can you write Python code for it?

pca = PCA(n_components=2)
X_bar = pca.fit_transform(X)
loadings = pca.components_
print(loadings.shape)


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
As you can see, the loadings_ property of the pca object returns a 2 x 100 matrix which holds for each of the 2 principal components a score for each word in our vocabulary of 100. These scores will tell us how important the contribution of specific words was, when it comes to the creation of the principal components. To be able to interact more intuitively with these loadings, we now take the transpose the matrix:
Can you write Python code for it?

loadings = loadings.transpose()
print(loadings.shape)


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
As you can see, this data matrix is fairly high-dimensional, since we work with 100 word frequencies: while we can easily plot texts in two, or perhaps three dimensions, it is much more difficult to think about data in 100 dimensions. PCA is one of the techniques for so-called 'dimension reduction' which is commonly used: it will attempt to convert our original 9 by 100 matrix, to a much lower-dimensional matrix, such as a 9 x 2 matrix. The newly created columns are called principal components and the idea is that they offer a summary of the original 9 x 100 matrix that is maximally faithful to the original structure. This 'column reduction' is easily achieved in sklearn:
Can you write Python code for it?

from sklearn.decomposition import PCA
pca = PCA(n_components=2)


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
As you can see, we see a canvas appear with a number of dots. In our case, however, it will be more informative to remove the dots, and annotate them with the titles corresponding with each dot. We therefore first plot the position labels without any dot:
Can you write Python code for it?

fig, ax1 = sns.plt.subplots()
x1, x2 = X_bar[:,0], X_bar[:,1] # select coordinates via numpy indexing!
ax1.scatter(x1, x2, 100, edgecolors='none', facecolors='none')


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
Here, we instantiate the method and then apply it to the data that we have (X) via the fit() method. After fitting our method on the data, we can use it to transform() our data to some new format. Conveniently, there is also a convenience method fit_transform():
Can you write Python code for it?

m = SomeMethod(arg1='foo', arg2='foo2')
X_bar = clf.fit_transform(X)


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
Let us now add these word loadings to our plots, and overlay our original graph with the word loadings in a lighter colour. To achieve this we actually need to add a second axis to the plot, because the word loadings have a scale that is different from the text's position. We can do that in a way that is precisely parallel to plotting the title labels:
Can you write Python code for it?

sns.set_style('white') # remove gridlines because of dual axes

# we repeat the previous bit:
fig, ax1 = sns.plt.subplots()
x1, x2 = X_bar[:,0], X_bar[:,1] # select coordinates via numpy indexing!
ax1.scatter(x1, x2, 100, edgecolors='none', facecolors='none')
for x, y, title in zip(x1, x2, titles):
    ax1.text(x, y, title, ha='center', va="center")

ax2 = ax1.twinx().twiny()
l1, l2 = loadings[:,0], loadings[:,1]
ax2.scatter(l1, l2, 100, edgecolors='none',
            facecolors='none') # first empty plot!
for x, y, l in zip(l1, l2, words):
    ax2.text(x, y, l ,ha='center', va="center",
             size=8, color="darkgrey")


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
Notice the underscore at the end of the .label_ property. If you scroll up, you can see that the .loadings_ property of the PCA object has it too. The fact that this property ends with an underscore, is a convention which sklearn uses to indicate that this is a property, which only becomes available after fitting the algorithm. We can now use these labels to obtain a more objective coloring of our samples:
Can you write Python code for it?

sns.set_style('white') # remove gridlines because of dual axes

clust_colors = ('r', 'b')

# we repeat the previous bit:
fig, ax1 = sns.plt.subplots()
x1, x2 = X_bar[:,0], X_bar[:,1] # select coordinates via numpy indexing!
ax1.scatter(x1, x2, 100, edgecolors='none', facecolors='none')
for x, y, title, c_idx in zip(x1, x2, titles, cl_labels):
    ax1.text(x, y, title, ha='center',
             va="center", color=clust_colors[c_idx])

# overlay the loadings:
ax2 = ax1.twinx().twiny()
l1, l2 = loadings[:,0], loadings[:,1]
ax2.scatter(l1, l2, 100, edgecolors='none',
            facecolors='none') # first empty plot!
for x, y, l in zip(l1, l2, words):
    ax2.text(x, y, l ,ha='center', va="center",
             size=8, color="darkgrey")


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
This is the standard procedure for supervised algorithms for unsupervised algorithms, such as Principal Components Analysis a similar procedure exists:
Can you write Python code for it?

m = SomeMethod(arg1='foo', arg2='foo2')
clf.fit(X)
X_bar = clf.transform(X)


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
This will probably look a bit abstract for now, so let's delve into Principal Components Analysis as a case study to introduce sklearn's powerful capabilities.
Principal Components Analysis
The basics
Principal Components Analysis or PCA is a commonly used exploratory technique in stylometry. Let us have a look at our Victorian data matrix again:
Can you write Python code for it?

import pickle
titles, authors, words, X = pickle.load(open("dummy.p", "rb"))
print(X.shape)


I am working on the file "Chapter 3 - First steps in sklearn.ipynb".
The first task is:
This yields the exact same result. Let us now have a look at X_bar:
Can you write Python code for it?


I am working on the file "Chapter 3 - Modeling and prediction.ipynb".
The first task is:
Classification with multiple classes: hand-written digits
We use the popular non-linear multi-class K-nearest neighbor algorithm to predict hand-written digits from the MNIST dataset.
Can you write Python code for it?

mnist = pandas.read_csv("data/mnist_small.csv")
mnist_train = mnist[:int(0.8*len(mnist))]
mnist_test = mnist[int(0.8*len(mnist)):]

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=10)
knn.fit(mnist_train.drop("label", axis=1), mnist_train['label'])

preds = knn.predict_proba(mnist_test.drop("label", axis=1))
pandas.DataFrame(preds[:5], index=["Digit %d"%(i+1) for i in range(5)])

knn.score(mnist_test.drop("label", axis=1), mnist_test['label'])


I am working on the file "Chapter 3 - Modeling and prediction.ipynb".
The first task is:
Predicting numerical values with a regression model
We use the the Linear Regression algorithm to predict miles-per-gallon of various automobiles.
Can you write Python code for it?

auto = pandas.read_csv("data/auto-mpg.csv")

# Convert origin to categorical variable
auto = auto.join(cat_to_num(auto['origin']))
auto = auto.drop('origin', axis=1)

# Split in train/test set
auto_train = auto[:int(0.8*len(auto))]
auto_test = auto[int(0.8*len(auto)):]

auto[:5]

from sklearn.linear_model import LinearRegression
reg = LinearRegression()
reg.fit(auto_train.drop('mpg', axis=1), auto_train["mpg"])

pred_mpg = reg.predict(auto_test.drop('mpg',axis=1))

plot(auto_test.mpg, pred_mpg, 'o')
x = linspace(10,40,5)
plot(x, x, '-');


I am working on the file "Chapter 3 - Modeling and prediction.ipynb".
The first task is:
Preparing the data
Can you write Python code for it?

# The categorical-to-numerical function from chapter 2
# Changed to automatically add column names
def cat_to_num(data):
    categories = unique(data)
    features = {}
    for cat in categories:
        binary = (data == cat)
        features["%s=%s" % (data.name, cat)] = binary.astype("int")
    return pandas.DataFrame(features)

def prepare_data(data):
    """Takes a dataframe of raw data and returns ML model features
    """

    # Initially, we build a model only on the available numerical values
    features = data.drop(["PassengerId", "Survived", "Fare", "Name", "Sex", "Ticket", "Cabin", "Embarked"], axis=1)

    # Setting missing age values to -1
    features["Age"] = data["Age"].fillna(-1)

    # Adding the sqrt of the fare feature
    features["sqrt_Fare"] = sqrt(data["Fare"])

    # Adding gender categorical value
    features = features.join( cat_to_num(data['Sex']) )

    # Adding Embarked categorical value
    features = features.join( cat_to_num(data['Embarked']) )

    return features


I am working on the file "Chapter 3 - Modeling and prediction.ipynb".
The first task is:
The Titanic dataset
We use the Pandas library to import the Titanic survival dataset.
Can you write Python code for it?

import pandas
data = pandas.read_csv("data/titanic.csv")
data[:5]

# We make a 80/20% train/test split of the data
data_train = data[:int(0.8*len(data))]
data_test = data[int(0.8*len(data)):]


I am working on the file "Chapter 4 - Burrows's Delta in sklearn.ipynb".
The first task is:
As we can see, the algorithm gets one of nine attributions wrong. But how does the classification stage work exactly in kNN? When confronted with a new test vector, the classifier will scan its memory for a 'nearest neighbour', or the training text, which looks most like the new test item. It determines this nearest neighbour using a distance calculation like the one we implemented ourselves in SciPy in a previous chapter. Next, it extrapolates the class label from this nearest neighbor and attaches the same authorship label to the unseen test instance. Therefore, the distance measure is crucial to the working of a robust kNN-classifier; it can be set in the constructor:
Can you write Python code for it?

knn = KNeighborsClassifier(metric='cityblock',
                           n_neighbors=1,
                           algorithm='brute')


I am working on the file "Chapter 4 - Burrows's Delta in sklearn.ipynb".
The first task is:
As we loop over the newly created loo object, we see that it returns a different test item in each iteration:
Can you write Python code for it?

for train, test in loo:
    print("%s %s" % (train, test))

knn = KNeighborsClassifier(metric='cityblock',
                           n_neighbors=1,
                           algorithm='brute')
preds = []
for train, test in loo:
    X_train, X_test = X[train], X[test]
    y_test = [authors[test]]
    y_train = [authors[i] for i in train]
    knn.fit(X_train, y_train)
    pred = knn.predict(X_test)
    preds.append(pred)

print('Accuracy after LOO:', accuracy_score(preds, authors))


I am working on the file "Chapter 4 - Burrows's Delta in sklearn.ipynb".
The first task is:
During the fitting stage, the StandardScaler will have calculated the per-column (i.e. per-word) standard deviations, which we can access as follows (mind the underscore!):
Can you write Python code for it?


I am working on the file "Chapter 4 - Burrows's Delta in sklearn.ipynb".
The first task is:
Here, we also set two other parameters, to make our implementation exactly parallel to Burrows's original Delta: i.e. we only look at an item's single nearest neighbour (instead of a larger 'neighbourhood') and we 'brutely' calculate the distance of the test items to all training items in memory. This 'brute' approach is less feasible for larger data sets, but which is fine for the smaller corpora we work with.
Burrows originally used the Manhattan cityblock distance, which we already came across in this course a couple of times. Importantly, however, Argamon showed that he would scale the relative frequencies of words in texts using the standard deviation of a word's frequency in the entire corpus. Thus, each simple relative frequency in our table needs to be replaced by the ratio of that frequency over the word's standard deviation in this corpus. This scaling method captures the following intuition: in authorship attribution, we are especially interested in high-frequency function words - as you will all know. Therefore, Delta will tune down the weight of words that show a relatively large standard deviation across - and which are probably not function words. Note that this operation is fully unsupervised: we do not need access to the author labels in our training data, to be able to scale our matrixes in this way. Again, in sklearn this is fairly easy to achieve, via the StandardScaler. We first import this Scaler and fit in on our data;
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler(with_mean=False)
scaler.fit(X)


I am working on the file "Chapter 4 - Burrows's Delta in sklearn.ipynb".
The first task is:
Indeed, we have obtained the standard deviation in the frequencies for each item in our 100-word vocabulary. Scaling the matrix, as Burrows proposed, is now a simple matter of applying the scaler:
Can you write Python code for it?

X_scaled = scaler.transform(X)
print(X_scaled.shape)


I am working on the file "Chapter 4 - Burrows's Delta in sklearn.ipynb".
The first task is:
Interestingly, the scaling seems to have helped, since now all test instances are correctly predicted, including the Thackeray book which was previously misattributed.
Evaluation
Until now, we have been making one major error, since we have been testing our classifier on the same data it was trained on. This is a bad mistake: to evaluate the performance of a classifier, we should really be testing it on out-of-training-data, and see how well it performs on such, previously unseen data. A number of evaluation strategies are available in this respect. One interesting technique is 'leave-one-out' validation: in this setup, we will loop over the available training data; each time we set aside one of the original training samples and we pretend we have never seen it. Then, we fit our classifier on the remaining instances and we apply it to the held-out item. After doing this for all instances, we now get an idea of how well our classifier would do, if it were to be applied to new, unseen documents. Sklearn ships with a lot of interesting functionality for such evaluation procedures and leave-onout validation is one of them:
Can you write Python code for it?

from sklearn.cross_validation import LeaveOneOut

nb = X_scaled.shape[0]
loo = LeaveOneOut(nb)


I am working on the file "Chapter 4 - Burrows's Delta in sklearn.ipynb".
The first task is:
Note that the shape of our data is not affected by this operation. We can now re-run our attribution algorithm:
Can you write Python code for it?

knn = KNeighborsClassifier(metric='cityblock',
                           n_neighbors=1,
                           algorithm='brute')
knn.fit(X_scaled, authors)
preds = knn.predict(X_scaled)
print(preds)
print('Accuracy: ', accuracy_score(preds, authors))


I am working on the file "Chapter 4 - Burrows's Delta in sklearn.ipynb".
The first task is:
Note that, here, we are dealing with supervised classification algorithms, so that fit() also takes a vector labels as input, which knn will store in its memory alongside the actual vectors. After fitting the classifier, we can apply it to new data, such as anonymous texts. The predict() which serves this purpose, now only takes the new, unseen text vectors as input - and not the labels, since it exactly its task to predict this. Let us apply this predict method to our training data:
Can you write Python code for it?

predictions = knn.predict(X)
print(predictions)


I am working on the file "Chapter 4 - Burrows's Delta in sklearn.ipynb".
The first task is:
The output above shows that predict() returns a label for each vector it was fed. In this case, we tested our classifier on the same data which it was trained on, so it should come as no surprise that it got a lot of the classification problems right. Using the metrics module, we can easily calculate the attribution accuracy which the algorithm obtains here:
Can you write Python code for it?

from sklearn.metrics import accuracy_score
accuracy_score(predictions, authors)


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
"99 Bottles of Beer" is a traditional song in the United States and Canada. It is popular to sing on long trips, as it has a very repetitive format which is easy to memorize, and can take a long time to sing. The song's simple lyrics are as follows: "99 bottles of beer on the wall, 99 bottles of beer. Take one down, pass it around, 98 bottles of beer on the wall." The same verse is repeated, each time with one fewer bottle. The song is completed when the singer or singers reach zero. Your task here is write a Python code block capable of generating all the verses of the song. Use a counter integer variable and a while loop. Make sure that your loop will come to an end and that the inflection of the word bottle is adapted to the counter!
Can you write Python code for it?


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
By now, you already know that Python has the len() function built-in, but can you write yourself a code block that prints the length of the string lengthy_word that you will define? First use a for loop; then try to achieve the same results with a while loop, but watch out that you don't get stuck an infinite loop!
Can you write Python code for it?


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
Congrats: you've reached the end of Chapter 4! Ignore the code block below; it's only here to make the page prettier.
Can you write Python code for it?

from IPython.core.display import HTML
def css_styling():
    styles = open("styles/custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
DIY

Compute the average of numbers:
Can you write Python code for it?


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
DIY

Given a list of words, output only the ones that are palindromes. As you know, a palindrome is a word that does not change when read backwards. If you need an extra challenge, try to print the palindromes in alphabetic order!
Can you write Python code for it?

words = ['bicycle', 'radar', 'origin', 'tie', 'level', 'poop', 'solar', 'nun']
# insert your code here

palindromes = []

for original_word in words:
    reversed_word = ''
    for character in reversed(original_word):
        reversed_word += character
    if original_word == reversed_word:
        palindromes.append(original_word)

print(palindromes)


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
DIY
The function len() returns the length of an iterable item:
Can you write Python code for it?


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
DIY
We have already covered a lot of ground. Now it is time to put some of the things that we have learned together. The following quiz might be quite hard and we would be very impressed if you get it right!
What we want you to do is write code that counts how often the letter a occurs in a text. You cannot do this, however, on the basis of the text itself. Instead, you need to do this on the basis of a frequency dictionary of the test. In this dictionary frequency_dictionary, keys are words and values are the frequencies. Assign your value to the variable number_of_words_as.
Can you write Python code for it?

frequency_dictionary = {"Beg": 1, "Goddard's": 1, "I": 3, "them": 2, "absent": 1, "already": 1,
                          "alteration": 1, "amazement": 2, "appeared": 1, "apprehensively": 1,
                          "associations": 1, 'clever': 1, 'clock': 1, 'composedly': 1,
                          'deeply': 7, 'do': 7, 'encouragement': 1, 'entrapped': 1,
                          'expressed': 1, 'flatterers': 1, 'following': 12, 'gone': 9,
                          'happening': 4, 'hero': 2, 'housekeeper': 1, 'ingratitude': 1,
                          'like': 1, 'marriage': 15, 'not': 25, 'opportunities': 1,
                          'outgrown': 1, 'playfully': 2, 'remain': 1, 'required': 2,
                          'ripening': 1, 'slippery': 1, 'touch': 1, 'twenty-five': 1,
                          'ungracious': 2, 'unwell': 1, 'verses': 1, 'yards': 5}
number_of_as = 0
print(number_of_as != 0)

# insert your code here

for word in frequency_dictionary:
    if 'a' in word:
        cnt = 0
        for character in word:
            if character == 'a':
                cnt += 1
        freq = frequency_dictionary[word]
        number_of_as += freq*cnt

print(number_of_as)

# if your code is correct, the following line should print True!
#print(number_of_as == 42)


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
Does notice a difference between the way Python prints lists, and the way it prints tuples to your screen? The following would have worked for lists, but does not for tuples (nor strings as we have seen before):
Can you write Python code for it?

#fruittuple[0] = 'orange' # will raise an error, tuples are immutable!
for fruit in fruittuple:
    print(fruit)


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
Have another look at the string variable lenghty_word that you defined in the previous exercices. Can you write a code block that fills a dictionary char_freqs containing the frequency of the different individual characters in length_word?
Can you write Python code for it?


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
Let's have yet another at lengthy_word. Can you write code that creates the dictionary next_char, holding for each first occurence of a character (key) the next character in the word as value. If the character is already in the dictionary, do nothing and if you're dealing with the last character in the word, add "Last word!" as value to the dictionary for this character.
Can you write Python code for it?


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
Now write a small program that iterates through the list colors and appends all colors that contain the letter r to the list colors_with_r. (Tip: use the append() function!)
Can you write Python code for it?

colors = ["yellow", "red", "green", "blue", "purple"]
colors_with_r = []
# insert you code here

for color in colors:
    l = len(color)
    colors_with_r.append(l)
    print(colors_with_r)


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
The code in the loop is executed as many times as their are letters, with a different value for the variable letter at each iteration. (Read the previous sentence again, until you fully understand it.)
Likewise, we can print all the items that are contained in a list:
Can you write Python code for it?

colors = ["yellow", "red", "green", "blue", "purple"]
for color in colors:
    print(color)


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
The third person singular verb form in English is distinguished by the suffix -s, which is added to the stem of the infinitive form: run -> runs. A simple set of rules can be given as follows: "If the verb ends in y, remove it and add ies. If the verb ends in o, ch, s, sh, x or z, add es. By default just add s." Your task in this exercise is to write a code block which given a verb in infinitive form, prints its third person singular form. Test your function with words like "try", "brush", "run" and "fix". Can you think of verbs for which your code doesn't work? Check out the string method .endswith() online!
Can you write Python code for it?


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
We can use this function to print the length of each word in the color list. Write your code in the box below:
Can you write Python code for it?

colors = ["yellow", "red", "green", "blue", "purple"]
# insert your code here


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
Write a code block that defines a list of integers and prints them as a histogram to the screen. For example, for histogram = [4, 9, 7, 2, 16, 8, 3], the code should print the following:

++++
+++++++++
+++++++
++
++++++++++++++++
++++++++
+++
Can you write Python code for it?


I am working on the file "Chapter 4 - Loops.ipynb".
The first task is:
min(), max() and sum()
When dealing with lists of numbers, there are three functions which come in handy:
Can you write Python code for it?

numbers = [1, 2, 3, 4, 5]
print(min(numbers))
print(max(numbers))
print(sum(numbers))


I am working on the file "Chapter 4 In-Class Problems-Solutions.ipynb".
The first task is:
4.11: Express $\hat{R}_p(\theta)$ in ±45 basis
Can you write Python code for it?

def Rp(theta):
    return Qobj([[cos(theta),-sin(theta)],[sin(theta),cos(theta)]]).tidyup()

Shv45 = sim_transform(H,V,P45,M45)


I am working on the file "Chapter 4 In-Class Problems-Solutions.ipynb".
The first task is:
The sim_transform function creates the matrix $\bar{\mathbf{S}}$ that can convert from one basis to another. As an example, it will create the tranform matrix to convert from HV to ±45 if you run:
Shv45 = sim_transform(H,V,P45,M45)    #  creates the matrix Shv45

Then you can convert a ket from HV to ±45 by applying the Shv45 matrix:
Shv45*H    #  will convert H from the HV basis to the ±45 basis

To convert operators, you have to sandwich the operator between $\bar{\mathbf{S}}$ and $\bar{\mathbf{S}}^\dagger$:
Shv45*Ph*Shv45.dag()     #  converts Ph from HV basis to the ±45 basis.
Can you write Python code for it?

def sim_transform(o_basis1, o_basis2, n_basis1, n_basis2):
    a = n_basis1.dag()*o_basis1
    b = n_basis1.dag()*o_basis2
    c = n_basis2.dag()*o_basis1
    d = n_basis2.dag()*o_basis2
    return Qobj([[a.data[0,0],b.data[0,0]],[c.data[0,0],d.data[0,0]]])


I am working on the file "Chapter 4 In-Class Problems.ipynb".
The first task is:
4.12:
Can you write Python code for it?

Rp45 = Shv45*Rp(pi/4)*Shv45.dag()

Rp45*Shv45*P45 == Shv45*V   # convert P45 to the ±45 basis

Rp45* Qobj([[1],[0]])

ShvLR = sim_transform(H,V,L,R)

ShvLR*Rp(pi/4)*ShvLR.dag()


I am working on the file "Chapter 4 In-Class Problems.ipynb".
The first task is:
Remember, these states are represented in the HV basis:
Can you write Python code for it?


I am working on the file "Chapter 4/Classification Metrics.ipynb".
The first task is:
Binary tasks
Can you write Python code for it?

y_even = y % 2
X_train, X_test, y_train, y_test = train_test_split(X, y_even, random_state=42)

classifier = LinearSVC().fit(X_train, y_train)
y_test_pred = classifier.predict(X_test)

confusion_matrix(y_test, y_test_pred)


I am working on the file "Chapter 4/Classification Metrics.ipynb".
The first task is:
Confusion matrices
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
confusion_matrix(y_test, y_test_pred)

plt.imshow(confusion_matrix(y_test, y_test_pred), cmap="coolwarm", interpolation="None")
plt.colorbar()
plt.xlabel("Predicted label")
plt.xticks(range(10))
plt.yticks(range(10))
plt.ylabel("True label")


I am working on the file "Chapter 4/Default metrics.ipynb".
The first task is:
Be mindful of default metrics
Regression
$$ R^2(y, \hat{y}) = 1 - \frac{\displaystyle \sum_{i=0}^{n - 1} (y_i - \hat{y}i)^2}{\displaystyle \sum{i=0}^{n - 1} (y_i - \bar{y})^2}$$
where $$\bar{y} =  \frac{1}{n} \sum_{i=0}^{n - 1} y_i$$
Can you write Python code for it?

rng = np.random.RandomState(42)
X = rng.uniform(size=(30, 1))
a = rng.normal(scale=10)
b = rng.normal()

y_clean = np.dot(X, a).ravel() + b
y = y_clean + rng.normal(size=len(y_clean))
plt.plot(X[:, 0], y, 'x')

from sklearn.metrics import mean_squared_error, r2_score
from sklearn.linear_model import LinearRegression

lr = LinearRegression().fit(X, y)
y_pred = lr.predict(X)
plt.plot(X[:, 0], y, 'x')
plt.plot(X[:, 0], y_pred)

print("training set R^2: %f" % r2_score(y, y_pred))  # same as lr.score(X, y)
print("training set MSE: %f" % mean_squared_error(y, y_pred))

print("training set R^2: %f" % r2_score(10 * y, 10 * y_pred))
print("training set MSE: %f" % mean_squared_error(10 * y, 10 * y_pred))

from sklearn.cross_validation import cross_val_score, LeaveOneOut
cv = LeaveOneOut(len(y))
cross_val_score(LinearRegression(), X, y, cv=cv)


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Also consider this:
Can you write Python code for it?

x = 0
def setx():
    x = 1
setx()
print(x)


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Bonus Exercise
Write a function textstats(filename) that, given a filename to load return a three-tuple with the following general statistics on the file: number of lines, number of words/tokens, number of characters. Don't worry about tokenisation: you can just split the words along whitespace.
Can you write Python code for it?


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Can you now try try to run main.py again, but now with the shorter call fib(upper=2000), witout explicitly mentioning the module from which the function originates? Does that work work out for you?
Now check out the files in your Desktop folder: you will notice that an additional file has been created, namely script.pyc. (If you can't see file, note that you can explicitly list all files in the current directory using the ls command in both Windows and Unix-like operating systems.) The extension of this new file stands for 'Compiled Python File'. Don't worry about this file -- you won't be able to inspect its contents using a text editor anyway. This file contains the numerical 'bytecode' that will be executed by your machine: it is this machine-readable version of your code that has actually been imported into the main.py module. You can safely ignore these files, but now you know what they are for. (By the way: note that there is no main.pyc file which has been created, because no functionality from this file has been imported into another module.)
It's always a good idea to distribute your code over a set of modules. In technical terms, your code should be as 'modular' as possible, meaning that similar functions should be grouped into the same module. This will help you keep your code organized especially when you are working on a larger project. If you have a set of functions that use you for loading and parsing files in Python, why not group under the same module? This way you can organize your own coding more efficiently, as well as share and document your code more easily.
Now you know how how to store and organize your code in separate files and modules. Still, a lot of programmers continue to explore their data using 'interactive Python' via a so-called 'interactive Python interpreter' that more or less resembles the IPython envirionment you have been working in so far. To launch such an interpreter, just type in python3 in your console and hit enter. This will launch a live Python session in your...
Can you write Python code for it?


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Check your output by viewing the words.txt file in a text editor such as Sublime Text 2. (Windows users: do not use Notepad!)

Importing external modules
Most of the functionality we have used thus far, is simply built into the Python language itself. Often however you need to use external modules in your code. A lot of external modules are already available in the Python Standard Library, for a wide variety of tasks. There are also countless third-party providers of Python modules.
To use an external module in your code you need to explicitly 'import' it. Consider for example the module random from the standard library, which contains functions for generating random numbers:
Can you write Python code for it?

import random
print(random.randint(0, 10))


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Create a new file called main.py in the same directory, namely your Desktop folder. Add the shebang line on top, as well as the following statement which will import the functionality from the script.py module. Note that the syntax is entirely the same as for importing one of the 'official' functions from the Python Standard Library! Instead of running script.py, now try to run main.py which will import the script.fib() function. Does this work out? You don't have to add the '.py' by the way: your computer will figure out this extension itself.
Can you write Python code for it?

#!/usr/bin/env python
import script
script.fib(upper=2000)


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Exercise
Write a function calculate_average(numbers), taking one parameter, a list (or other iterable) containing only numbers (you may assume that), and returning the average/mean:
Can you write Python code for it?

# write your function calculate_average here

# do not modify the code below, for testing only
numbers = [1, 2, 3, 4, 5]
av = calculate_average(numbers)
assert av == 3 # should be True if you did it correctly


# write your function calculate_average here

# do not modify the code below, for testing only
numbers = [1, 2, 3, 4, 5]
av = calculate_average(numbers)
print(av == 3) # should be True if you did it correctly


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
In fact, this code has produced two completely unrelated x's!
Nevertheless, it is possible to read a global variable from within a function, in a strictly read-only fashion. But as soon as you assign something, the variable will be a local copy:
Can you write Python code for it?


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
In the code block above in which you read the Austen file, the newline is still included with the original line that preceded it in the file: this is why you see all the extra empty lines in the output above! If you wish to remove all preceding and trailing whitespace in a string (newlines, spaces, but also tabs), you can use the strip() function:
Can you write Python code for it?


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
In this case we wouldn't have to specify where or machine should find the randint() function. You can also (temporarily) change the names of the functions you import:
Can you write Python code for it?

from random import randint as random_int
print(random_int(0, 100))


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
In this code block, we have automatically created a new file called testoutput.txt in the data directory. We then wrote a single line to this file and then we closed it. Note that the w in wt is a crucial addition: if you would have left this out, Python would have opened the file in 'readonly' mode and you wouldn't have been able to write to it! The 't' in the argument, again, signifies that we will be writing to this file in plain text mode.
If you want your data to be written on multiple lines, you need to take care to explicitly encode the newlines. Instead of:
Can you write Python code for it?

f = open('data/testouput.txt','wt', encoding='utf-8')
f.write("Hello world on the first line!")
f.write("Hello world on the second line!")
f.close()


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Instead of executing this code by hitting ctrl+enter as we have done in the IPython notebook so far, we will now learn how to execute our code differently. We have two ways for doing this: an easy one, and a difficult one. When you work with a code editor like Sublime Text, there is often an easy way to execute your code. In Sublime, for instance, you can first save your file with a ".py" extension, and then 'build' your code by hitting Ctrl+b (Windows, Unix) or Command+b (Mac OS X). You will now see that the output of your script will be written to your screen in Sublime.
For the second option, you can use a command line interface or prompt. Watch out: this can be pretty scary at first... In general, you should always watch out when you use a command line interface to your machine: only execute commands that you (more or less) understand! You typically have complete control over your machine via such an interface, so you need to watch out not to remove any important files. (You could e.g. unintentionally delete your entire operating system from your hard drive with a single command...).
First we will deal with instructions for doing this in Mac OS X and Linux-distributions such as Ubuntu. Mac OS X and Linux tend to behave similarly because they are both 'Unix-based' operating systems.

On a Mac,  you should open your 'Terminal'-application by clicking the relevant icon in the folder Applications > Utilities. Alternatively, click the magnifying glass in the top-right of your screen (shortcut: Command key+Spacebar). Next, type 'Terminal' in the box that appears and hit enter when your Mac has found the Terminal app.
On a Linux installation, first open a command line window by navigating to the Terminal via Applications Menu > Accesories > Terminal (Gnome) or Dash > More Apps > Accessories > Terminal (Unity). Under both Gnome and Unity, you should also be able to use the keyboard shortcut Ctrl+Alt+T to open a console window.

Now 'cd' (= Change Directory) into the director that contains your script.py file...
Can you write Python code for it?


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Next, add a line that actually calls the fib() function for upper=2000. (Don't forget to take care of the correct indentation!)
Can you write Python code for it?


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Note that we have to be explicit about where our Python interpreter should look for the fib() function using the syntax with a dot (module.function()). If we want to be able to use the shorter version of the function call, we should have used to following import statement:
Can you write Python code for it?


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Note that your functions don't have to return anything explicitly. If you run the example below, you will see that such a 'void' function in reality returns 'None'.
Can you write Python code for it?

def little_grey(doctor):
    print("Oh Dr", doctor, "!")

whatever = little_grey("Steamy")
print(whatever)


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Note the syntax used: using the dot, we indicate that our machine should look for the randint() function inside the random module we just imported. You can import an entire module or import only a specific function in the module. We could also have imported the (single) function we needed as follows:
Can you write Python code for it?

from random import randint
print(randint(0, 100))


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Now we mastered the art of reading files, let's move on to writing files, which follows a similar logic:
Can you write Python code for it?

f = open('data/testoutput.txt', 'wt', encoding='utf-8')
f.write("Hello world!")
f.close()


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Now, let's define a more advanced function. The following function count_articles() counts the number of articles (the, a, an) in a list of words. The words are passed to the function as a list of strings. Note that the function itself lowercases the words, so that you never have to take care of this again in the rest of your code! Can you change count_articles() in such a manner that it will now accept the entire string and splits it into tokens internally? That way, the user of a function wouldn't have to care about this!
Can you write Python code for it?

def count_articles(tokens):
    count = 0
    for token in tokens:
        if token.lower() in ('the', 'a', 'an'):
            count += 1
    return count

text = "A bit less trivial , this function counts the number of articles ( the , a , an ) in a list of words"
words = text.split()
print(count_articles(words))


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Now, try to adapt the code that read in the Austen file and have your code print each line without the preceding and trailing whitespace! Have the annoying "double lines" disappeared in your output?
Can you write Python code for it?

f = open('data/austen-emma-excerpt.txt','rt', encoding='utf-8')
for line in f:
    print(line)
f.close()


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Otherwise your file would have Hello world!Hello world! in it, i.e. without the newlines.
Besides 'read-mode' and 'write-mode' when dealing with text files, there is also the 'append-mode' in Python. Watch out: in 'write-mode', you will always overwrite the existing content of the file. However, if you've open a file in 'append-mode', everything you write to the file will be added at the end of the file, without deleting anything of the existing content in the file. In order to enable the append mode, you need to specify 'at' as your second parameter when you open files ('a' for append mode; 't' for text mode).
Exercise
Read the file data/austen-emma-excerpt-tokenised.txt, and write to a file words.txt all words occuring in this text (without duplicates!!), alphabetically ordered, one word per line. That way, you are really creating a lexicon or word list of the text. (Tip: you should use sets in this exercise!)
Can you write Python code for it?


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Quiz
Read the file data/austen-emma-excerpt.txt and compute the average length (in characters) of the lines.
Can you write Python code for it?

f = open('data/austen-emma-excerpt.txt', 'rt', encoding='utf-8')
# insert your code here
# important: always remember to properly close your files again!


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Rather than just printing, we can of course do whatever we want with this file's content. Let's count the number of lines (but note, that a line does not necessarily correspond to a sentence).
Can you write Python code for it?

count = 0
f = open('data/austen-emma-excerpt.txt', 'rt', encoding='utf-8')
for line in f:
    count += 1
f.close()
print(count)


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Scripts
Up until now, we have been using the interactive IPython software to write our Python code. We have only been writing really small bits and pieces of code, however, instead of writing longer scripts that can provide a more significant batch of functionality. Now, let us make our first independent Python script together. (Note that this way of working will also resemble more closely your future day-to-day coding practice.)
Open 'Sublime Text 2', a popular text editor which we will use in this course (http://www.sublimetext.com/). Create a new file -- this might have happened automatically when you opened the editor -- and save it as "script.py" in a convenient location (here, we will assume that you have saved it in your Desktop folder. Note that files containing Python code typically take the ".py" extension.
If you are working in a UNIX-like environment (Mac or Linux), you should now add the following code on the very first line of your script:
Can you write Python code for it?