I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
There exists a similar character to encode 'tab' characters, namely \t. You can use this character to play around with the indentation of your (e.g. hierarchically structured) output:
Can you write Python code for it?

s = "First line\n\t* Second line\n\t* Third line\n\t* Fourth line\nFifth line"
print(s)


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Variable scope
Any variables you declare in a function, as well as the parameters that are passed to a function will only exist within the 'scope' of that function, i.e. inside the function itself. The following code will produce an error, because the variable x does not exist outside the function:
Can you write Python code for it?


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
Watch out: the open() function doesn't return the actual text that is saved in the text file. It only returns a so-called 'file object' from which we can read the content using the .read() function. We passed three arguments to the open() function:

the name of the file that you wish to open
the mode, a combination of characters, 'r' represents read-mode, and 't' represent plain text-mode. This indicates we are reading a plain text file.
the last argument, a named argument (encoding), specifies the encoding of the text file.

UTF-8
You may wonder what an encoding is and what utf-8 is. For anyone working with texts and computers this is vital to know. Internally, a computer knows no characters whatsoever: every piece of information is represented as numbers (which in turn are represented in a binary format, as zeroes and ones). An encoding specifies which numbers represent which characters. A famous and long-standing encoding scheme is ASCII, in which for example the letter 'A' is encoded using the number 65. ASCII however only has a very limited alphabet and can not encode a lot of writing systems. A modern-day encoding supporting countless writing systems is unicode and utf-8 is a kind of unicode. This the type of encoding that you will want to use for your data whenever possible. Whenever you have a choice, you should use unicode!

Reading an entire file in one string is not always desirable, especially not with huge files. The following example reads up until a newline everytime, and returns one line at a time.
Can you write Python code for it?

f = open('data/austen-emma-excerpt.txt','rt', encoding='utf-8')
for line in f:
    print(line)
    print("n")
f.close()


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
We can also have a function return multiple values by combining them in a tuple (the type of ordered, immutable list we talked about in the previous chapter!). Python has a nice way of 'unpacking' such a tuple using assignment:
Can you write Python code for it?

def count(text):
    words = text.split(" ")
    word_count = len(words)
    character_count = len(text)
    return word_count, character_count

word_count, character_count = count("To be or not to be , that is the question .")
print(word_count)
print(character_count)


I am working on the file "Chapter 5 - Functions and Files.ipynb".
The first task is:
You need to write:
Can you write Python code for it?

f = open('data/testoutput.txt','wt', encoding='utf-8')
f.write("Hello world on the first line!\n")
f.write("Hello world on the second line!")
f.close()


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
As you can see we loop over the txt-files under the data/victorian_large directory and end up with three lists (authors, titles, and the actual texts) which can easily zipped together:
Can you write Python code for it?

for t, a in zip(titles, authors):
    print(t, 'by', a)


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
As you can see, the max_df takes us away from the high-frequence function words, with a lot of proper nouns coming through. By the way: make sure that you specify min_df and max_df as floats: it you specify them as integers, sklearn will interpret these number as the minimum or maximum number of individual documents in which a term should occur.
Finally, it is good to know that we can also manually specify vocabularies, through the vocabulary argument. This way, we can exercise a much tighther control over which words go into a procedure - and manually remove words from a previous analysis, if necessary.
Can you write Python code for it?

vec = TfidfVectorizer(vocabulary=('my', 'i', 'we'))
X = vec.fit_transform(texts)
print(vec.get_feature_names())


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
Having such a module is great, but it also hides a lot of the interfacing which is needed with sklearn. In the next paragraphs, we will have a lot at the preprocessing functionality which sklearn offers to deal with text.
Integer frequencies
Calculating absolute frequencies of, for instance, words in texts is something that is rarely done in stylometry. Because we often work with texts of unequal length, it is typically safer to back off to relative frequencies. Nevertheless, it is good to know that sklearn's supports the extraction of absolute counts with its CountVectorizer:
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
vec = CountVectorizer(max_features=100)
X = vec.fit_transform(texts)
print(X.shape)


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
Here, we have to watch out of course, because specifying such ranges will interfere with the max_features parameter. Because bigrams are much more frequent than tetragrams, for instance, the tetragrams might never make it to to frequency table, if the max_features paramater isn't high enough! Naturally we could gain more control over this extraction process, by running two independent vectorizers, and stacking their respective outcomes:
Can you write Python code for it?

vec = TfidfVectorizer(max_features=50,
                      analyzer='char',
                      ngram_range=(2, 2))
X1 = vec.fit_transform(texts).toarray()

vec = TfidfVectorizer(max_features=100,
                      analyzer='char',
                      ngram_range=(3, 3))
X2 = vec.fit_transform(texts).toarray()

import numpy as np
print(X1.shape)
print(X2.shape)
X = np.hstack((X1, X2))
print(X.shape)


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
It becomes clear, therefore, that sklearn is performing some sort of tokenization internally. Inconveniently, it also removes certain words: can you find out which?
To override this default behaviour, we need a little hack. One common solution is to create our own analyzer (i.e. tokenizer) function and pass that to our vectorizer:
Can you write Python code for it?

def identity(x):
    return x.split()

vec = TfidfVectorizer(max_features=10,
                      analyzer=identity,
                      use_idf=False)
X = vec.fit_transform(texts)
print(vec.get_feature_names())


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
Let us apply this new function:
Can you write Python code for it?

for i in range(len(texts)):
    texts[i] = preprocess(texts[i])


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
Let us start with some basic preprocessing. The function preprocess() below lowercases each text and only retains alphabetic characters (and whitespace). Additionally, to speed things up a bit, we truncate each document after the 200K first characters:
Can you write Python code for it?

def preprocess(text, max_len=200000):
    return ''.join([c for c in text.lower()
                        if c.isalpha() or c.isspace()])[:max_len]


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
Of course, the list of features extracted is not altered by changing this argument, but they values will have changed.
Feature types
So far, we have only considered word frequencies as stylometric style markers - where we naively define a word as a space-free string of alphabetic characters. Implicitly, we have been setting the analyzer argument to 'word':
Can you write Python code for it?

vec = TfidfVectorizer(max_features=10,
                      analyzer='word')
X = vec.fit_transform(texts)
print(vec.get_feature_names())


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
To create a vector space that simple has relative frequencies (which have not been normalized using IDF's), we can simple add the following parameter:
Can you write Python code for it?

vec = TfidfVectorizer(max_features=10,
                      use_idf=False)
X = vec.fit_transform(texts).toarray()
print(vec.get_feature_names())


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
We can now instantiate our vectorizer with some traditional settings; we will extract the 100 most frequent words and scale them using the per-column standard deviation. We limit this vectorizer to word unigrams (ngram_size=1) and we specify the 'culling' rule (cf. min_df) that words should be present in at least 70% of all texts:
Can you write Python code for it?

vectorizer = Vectorizer(mfi=100,
                        vector_space='tf_std',
                        ngram_type='words',
                        ngram_size=1,
                        min_df=0.7)


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
We can now use this object to vectorize our lists of documents:
Can you write Python code for it?

X = vectorizer.vectorize(texts)
print(X.shape)


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
We see that sklearn by default returns a so-called sparse matrix, which only explicitly stores non-zero values. While this very efficient for larger datasets, there are many methods which cannot deal with such sparse matrices - such as sklearn's PCAobject, to give but one example. If memory is not that much of an issue, it is always safer to convert the sparse matrix back to a 'dense' array:
Can you write Python code for it?


I am working on the file "Chapter 5 - Vectorization.ipynb".
The first task is:
When using only 50 top-frequent features, it is logical that these are present in most of our texts.
Real-valued Frequencies
Working with relative frequencies is much more common in stylometry. Although this is somewhat counter-intuitive, sklearn does not have a dedicated function for this vectorization strategy. Rather, one must work with the TfidfVectorizer, which can be imported as follows:
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfVectorizer


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
By the way, there exist more characters that you might have to escape using a backslash. This is because they are part of the syntax that use to define regular expressions: if you don't escape them, Python will not know that you are looking for an literal match. Characters that you typically might want to escape include: ( + ? . * ^ $ ( ) [ ] { } | \ ) ,. For example:
Can you write Python code for it?

s = "In principio [erat] verbum, et verbum erat apud Deum."
brackets_wrong = re.compile(r"[|]")
print(brackets_wrong.sub("X", s))
brackets_right = re.compile(r"(\[)|(\])")
print(brackets_right.sub("X", s))


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
If we want to correct the double spaces, we could now do:
Can you write Python code for it?

ds = re.compile(r" +")
for line in whitespace.split(s):
    print ds.sub(" ", line)


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
If you would have wanted to split on the linebreaks only (possible followed by e.g. spaces), you could have used the following pattern:
Can you write Python code for it?

s = """This is a text  on three   lines
with  multiple instances of
double spaces."""
whitespace = re.compile(r"\s*\n\s*")
print(whitespace.split(s))


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
Interestingly, you also use regular expression to search inside words. Can you explain why the following patterns (don't) match?
Can you write Python code for it?

candidates = ["good", "god", "gud", "gd"]
p = re.compile(r"go+d")
for c in candidates:
    print(p.match(c))


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
Next, we can call the sub() function from the re package on this pattern, in order to replace (or 'substitute') our pattern with another word, like this:
Can you write Python code for it?

text = pattern.sub("XXX", s)
print(text)


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
Note how our pattern allows for a special syntax: the pipe symbol which we used allows to express that one character OR another one is fine for the regular expression to match. Oops: the capital letter at the beginning of the sentence hasn't been replaced because we only included lowercase vowels in our pattern definition. Let's add the uppercase vowels to the regex:
Can you write Python code for it?

vowel_pattern = re.compile(r"a|A|e|E|o|O|u|U|i|I")
without_vowels = vowel_pattern.sub("X", s)
print(without_vowels)


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
Note the order of the arguments passed to sub(): first, the word we would like to replace our pattern with, and secondly our original string. We can just as easily get back our original string:
Can you write Python code for it?

pattern2 = re.compile(r"XXX")
text = pattern2.sub("verbum", s)
print(text)


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
One final feature we should mention is the [^...] syntax: this will match any character that is NOT between the brackets. Remember the vowel_pattern above? Using the caret symbol we can quickly 'invert' this pattern, so that it will match all consonants:
Can you write Python code for it?

s = "these are vowels and consonants"
consonants = re.compile(r"[^aeuoi]")
print(consonants.sub("X", s))


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
So far nothing special: we are simply replacing one word for another word. The smart ones among you will have noticed that we could have achieved the exact same result using the replace() function, which we came across in an earlier chapter. But now: say you would like to replace all vowels in a string. With regular expressions, this is a piece of cake:
Can you write Python code for it?

vowel_pattern = re.compile(r"a|e|o|u|i")
without_vowels = vowel_pattern.sub("X", s)
print(without_vowels)


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
Speaking of words: it might be interesting to know that you can use regular expressions for advanced string splitting. If you want to split a sentence across all whitespace characters for instance, you can use an espaced \s. This operator will match all whitespace characters, such as tabs, linebreaks, normal spaces etc. If you add a + sign, your pattern will match series of whitespace characters:
Can you write Python code for it?

s = """This is a text  on three   lines
with  multiple instances of
double spaces."""
whitespace = re.compile(r"\s+")
print(whitespace.split(s))


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
The next thing we define is the actual regular expression which we will use, or the string that we will use to search the sentence we defined above. We pass this string to the compile() function in the re package, which will allow fast searching later on. Note that we put an r in front of this string when we initialize it, which turns our string into a so-called 'raw string'. While this is not always necessary, it is a good idea to do this consistently when dealing with regular expressions.
Can you write Python code for it?


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
The syntax for regular expression includes a whole range of possibilities which we simply cannot all deal with it here. Because of that we will stick to a number of helpful examples. An interesting feature is that you can specify whether or not a character really has to occur. You can check whether the pattern occurs in a string using the match() function which will return None if it doesn't find the pattern in the string searched:
Can you write Python code for it?

pattern = re.compile(r"m{2,4}")
print(pattern.match(""))
print(pattern.match("m"))
print(pattern.match("mm"))
print(pattern.match("mmm"))
print(pattern.match("mmmm"))
print(pattern.match("mmmmm"))
print(pattern.match("mmmmmm"))
print(pattern.match("mmmmammm"))


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
There is also a syntax to match any character (except the newline):
Can you write Python code for it?

any_char = re.compile(r".")
print(any_char.sub("X", s))


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
There is in fact an easy way to match all lowercase and uppercase characters in a string, like this:
Can you write Python code for it?

ups = re.compile(r"[A-Z]")
lows = re.compile(r"[a-z]")
without_ups = ups.sub("X", s)
print(without_ups)
without_ups = lows.sub("X", s)
print(without_ups)


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
Using a plus sign you can indicate whether you want to match multiple occurrences of a character. A good example from the world of paper writing are double spaces, which can be hard to spot. In the code block below, we replace all multiple occurences of a whitespace character by a single whitespace character. Note that you can use the whitespace character just like any other character (you don't have to escape it). Multiple occurences of the whitespace character will be matched: it doesn't matter how many, as long as there is at least one:
Can you write Python code for it?

paper = "My thesis on  biology     contains a lot of  double spaces.   I will remove  them."
mult = re.compile(r" +")
print(mult.sub(" ", paper))


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
With the curly brackets, you indicate that you are only interested in the letter 'm' if it occurs 2,3 or 4 times in a row in the string you search. Because None is returned if not a single match was found, you can use the outcome of match()in an if-statement. The following example shows how you can also use the curly brackets to match an exact number of occurences (in this case three a's).
Can you write Python code for it?

pattern = re.compile(r"a{5}")
if pattern.match("aaaaa"):
    print("Found it!")
else:
    print("Nope...")
# or:
if pattern.match("aa"):
    print("Found it!")
else:
    print("Nope...")


I am working on the file "Chapter 6 - Regular Expressions.ipynb".
The first task is:
You can also look for more specific, as well as longer letter groups by arranging them with round brackets:
Can you write Python code for it?

p = re.compile(r"(ri)|(um)|(Th)")
print(vowel_pattern.sub("X", s))


I am working on the file "Chapter 6 - Spin.ipynb".
The first task is:
Example: determine $P(S_x = \frac{\hbar}{2} ||-y\rangle)$
Can you write Python code for it?


I am working on the file "Chapter 6 - Spin.ipynb".
The first task is:
This makes sense given that $S_x$ can be either $\frac{+\hbar}{2}$ or $\frac{-\hbar}{2}$ with equal probability. Similarly, if the state is $|\psi\rangle=|+x\rangle$.
Can you write Python code for it?


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
... or indeed use a call to range() and len():
Can you write Python code for it?

for i in range(len(words)):
    print(words[i], ": index", i)


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
Do you see what happened here? In fact, zip() really functions like a 'zipper' in the real-world: it zips together multiple lists, and return a list of mini-tuples, in which the correct authors, titles and dates will be combined with each other. Moreover, you can pass multiple sequences to at once zip():
Can you write Python code for it?


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
Enumerate
Of course, range() can also be used to iterate over the items in a list or tuple, typically in combination with calling len() to avoid IndexErrors:
Can you write Python code for it?

words = "Be yourself; everyone else is already taken".split()
for i in range(len(words)):
    print(words[i])


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
Finally, we should also mention that dictionaries and sets can also be filled in a one-liner using such comprehensions. For sets, the syntax runs entirely parallel to that of list and tuple comprehensions, but here, we use curly brackets to surround the expression:
Can you write Python code for it?

text = "This text contains a lot of different characters, but probably not all of them."
chars = {char.lower() for char in text if char not in string.punctuation}
print(chars)


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
For dictionaries, which consist of key-value pairs, the syntax is only slightly more complicated. Here, you have to make sure that you link the correct key to the correct value using a colon, in the very first part of the comprehension. The following example will make this clearer:
Can you write Python code for it?

counts = {word:len(word) for word in words}
print(counts)


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
Good programmers can do amazing things with comprehensions. With list comprehensions, it becomes really easy, for example, to create nested lists (lists that themselves consist of lists or tuples). Can you figure out what is happening in the following code block:
Can you write Python code for it?

nested_list = [[x,x+2] for x in range(10, 22, 3)]
print(nested_list)
print(type(nested_list))
print(type(nested_list[3]))


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
Great: you are starting to become a real pro at comprehensions! The following, very dense code block, however, might be more challenging: can you figure out what is going on?
Can you write Python code for it?

A = tuple([x-1,x+3] for x in range(10, 100, 3))
B = [(n*n, n+50) for n in range(10, 1000, 3) if n <= 100]
sums = sum(tuple(item_a[1]+item_b[0] for item_a, item_b in zip(A[:10], B[:10])))
print(sums)


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
Here -- as with range() -- we have to cast the result of enumerate() to e.g. a list before we can actually print it. Iterating over the result of enumerate(), on the other hand, is not a problem. Here, we print out each mini-tuple, consisting of an index and an time in a for-loop:
Can you write Python code for it?

for mini_tuple in enumerate(words):
    print(mini_tuple)


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
Here, range() will return a number of integers, starting from zero, up to (but not including) the number which we pass as an argument to the function. Using range() is of course much more convenient to generate such lists of numbers than writing e.g. a while-loop to achieve the same result. Note that we can pass more than one argument to range(), if we want to start counting from a number higher than zero (which will be the default when you only pass a single parameter to the function):
Can you write Python code for it?


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
Importantly, we can just as easily create a tuple using the same comprehension syntax, but this time calling tuple() on the comprehension, instead of using the squared brackets to create a normal list:
Can you write Python code for it?

tuple_word_lengths = tuple(len(word) for word in words if word not in string.punctuation)
print(tuple_word_lengths)
print(type(tuple_word_lengths))


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
In our for-loop example, we can apply the same kind of unpacking in each iteration:
Can you write Python code for it?

for item in enumerate(words):
    index, word = item
    print(index)
    print(word)
    print("=======")


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
In the comprehensions above, words is the only pre-existing input to our comprehension; all the other variables are created and manipulated inside the function. The new range() function which we saw at the beginning of this chapter is also often often as the input for a comprehension:
Can you write Python code for it?

square_numbers = [x*x for x in range(10)]
print(square_numbers)


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
In the first line above, we create a new list (nested_list) but we don't fill it with single numbers, but instead with mini-lists that contain two values. We could just as easily have done this with mini-tuples, by using round brackets. Can you spot the differences below?
Can you write Python code for it?

nested_tuple = [(x,x+2) for x in range(10, 22, 3)]
print(nested_tuple)
print(type(nested_tuple))
print(type(nested_tuple[3]))
nested_tuple = tuple((x,x+2) for x in range(10, 22, 3))
print(nested_tuple)
print(type(nested_tuple))
print(type(nested_tuple[3]))


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
Note that zip() can also be very useful in this respect, because you can unpack items inside the comprehension. Do you understand what is going in the following code block:
Can you write Python code for it?


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
This is very useful, especially if you can figure out why the following code block will generate an error...
Can you write Python code for it?

tuple_word_lengths = tuple()
for word in words:
    if word not in string.punctuation:
        tuple_word_lengths.append(len(word))
print(tuple_word_lengths)


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
We can even specify a 'step size' as a third argument, which controls how much a variable will increase with each step:
Can you write Python code for it?


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
You've reached the end of Chapter 7! Ignore the code below, it's only here to make the page pretty:
Can you write Python code for it?

from IPython.core.display import HTML
def css_styling():
    styles = open("styles/custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Chapter 7 - More on Loops.ipynb".
The first task is:
Zip
Obviously, enumerate() can be really useful when you're working lists or other kinds of data sequences. Another helpful function in this respect is zip(). Supposed that we have a small database of 5 books in the forms of three lists: the first list contains the titles of the books, the second the author, while the third list contains the dates of publication:
Can you write Python code for it?

titles = ["Emma", "Stoner", "Inferno", "1984", "Aeneid"]
authors = ["J. Austen", "J. Williams", "D. Alighieri", "G. Orwell", "P. Vergilius"]
dates = ["1815", "2006", "Ca. 1321", "1949", "before 19 BC"]


I am working on the file "Chapter 7 with solutions.ipynb".
The first task is:
7.10 A spin-1 particle is measured to have $S_y=\hbar$. What is the probability that a subsequent measurement will yield $S_z=0$? $S_z=\hbar$? $S_z=-\hbar$?
Can you write Python code for it?


I am working on the file "Chapter 7 with solutions.ipynb".
The first task is:
Define the spin-1 operators. We use J just to keep them apart. Could be S instead.
Can you write Python code for it?


I am working on the file "Chapter 7 with solutions.ipynb".
The first task is:
Doesn't look like example 7.4 because it's not simplified. How to check:
Can you write Python code for it?


I am working on the file "Chapter 7 with solutions.ipynb".
The first task is:
Our usual spin operators (spin-1/2)
Can you write Python code for it?

pz = basis(2,0)
mz = basis(2,1)
px = 1/sqrt(2)*(pz + mz)
mx = 1/sqrt(2)*(pz - mz)
py = 1/sqrt(2)*(pz + 1j*mz)
my = 1/sqrt(2)*(pz - 1j*mz)
Sx = 1/2.0*sigmax()
Sy = 1/2.0*sigmay()
Sz = 1/2.0*sigmaz()
Splus = Sx + 1j*Sy
Sminus = Sx - 1j*Sy


I am working on the file "Chapter 7 with solutions.ipynb".
The first task is:
Yes, this agrees.
Can also use inner product:
Can you write Python code for it?


I am working on the file "Chapter 8 - Alice and Bob.ipynb".
The first task is:
Create the projection operators for each of the angles, two for Alice, two for Bob
Can you write Python code for it?


I am working on the file "Chapter 8 - Alice and Bob.ipynb".
The first task is:
Create the state $\big|\psi\big\rangle = \sqrt{0.2} \big|H,H\big\rangle + \sqrt{0.8} \big|V,V\big\rangle$:
Can you write Python code for it?


I am working on the file "Chapter 8 - Alice and Bob.ipynb".
The first task is:
Find the conditional probability $P(\theta_{A2}|\theta_{B1}) = \frac{P(\theta_{A2},\theta_{B1})}{P(\theta_{B1})}$
Can you write Python code for it?

# A2 conditioned on B1 (expect 1)
# can do it all on one line:
expect(tensor(Pa2,Pb1),psi) / expect(tensor(qeye(2),Pb1),psi)

expect(tensor(Pa2,Pb2),psi)  # joint for A2, B2 (classically expect 0.09, QM says 0)


I am working on the file "Chapter 8 - Alice and Bob.ipynb".
The first task is:
Find the conditional probability $P(\theta_{B2}|\theta_{A1}) = \frac{P(\theta_{B2},\theta_{A1})}{P(\theta_{A1})}$
Can you write Python code for it?

# B2 conditioned on A1 (expect 1)
Prob_b2_a1 = expect(tensor(Pa1,Pb2),psi)
#(psi.dag()*tensor(Pa1,Pb2)*psi).data[0,0]   # the joint probability
Prob_a1 = expect(tensor(Pa1,qeye(2)),psi)
#(psi.dag()*tensor(Pa1,qeye(2))*psi).data[0,0]  # the singular probability
Prob_b2a1 = Prob_b2_a1 / Prob_a1  # the conditional probability
Prob_b2a1


I am working on the file "Chapter 8 - Alice and Bob.ipynb".
The first task is:
First define the projection operator for a state at angle $\theta$:
Can you write Python code for it?

def P(theta):
    """The projection operator for a state at angle theta"""
    theta_ket = cos(theta)*H + sin(theta)*V
    return theta_ket*theta_ket.dag()


I am working on the file "Chapter 8 - Alice and Bob.ipynb".
The first task is:
Make a list of the probability of joint measurements for a pair of angles:
Can you write Python code for it?

out = []
for r in rads:
    out.append(expect(tensor(P(-r),P(r)),psi))

plt.plot(angles,out,".") # plot in units of pi


I am working on the file "Chapter 8 - Alice and Bob.ipynb".
The first task is:
Now, find the joint probability that Alice measures A1 and Bob measures B1. We do this by finding the expectation value of the projection operator for the joint state $\big|\theta_{A1},\theta_{B1}\big\rangle$. This is formed as the tensor product of the two appropriate projection operators. In these tensor products, be sure to put Alice's operator first, then Bob's (just like we did for the signal and idler photons). Each operator acts on the photon corresponding to the order in the tensor() function.
Notice we'll be using a new function expect(). This is equivalent to putting the operator in between the state bra and ket:
Can you write Python code for it?

P1 = expect(tensor(Pa1,Pb1),psi)  # joint for A1, B1 (expect 0.09)
P2 = psi.dag()*tensor(Pa1,Pb1)*psi
P1 == P2.data[0,0]  # The only difference is that we have to pull out the value
                    # from the Qobj using the .data[0,0] method so we can compare it to result from `expect`

P1


I am working on the file "Chapter 8 - Alice and Bob.ipynb".
The first task is:
So only 19 and 35 work. Now, can you derive 19 and 35 given only the state $|\psi\rangle$? Try the first plot, i.e. calculate the joint probability $P(\theta_A,\theta_B)$
Solution
Using the state, write the projection operators for a two photon state with angles $\theta_A$ and $\theta_B$. First, recall $$\big|\theta_i\big\rangle = \cos\theta_i\big|H\big\rangle + \sin\theta_i\big|V\big\rangle.$$ Next, form the two-photon state: $$\big|\theta_A,\theta_B\big\rangle = \big|\theta_A\big\rangle \otimes \big|\theta_B\big\rangle = \left(\cos\theta_A\big|H\big\rangle + \sin\theta_A\big|V\big\rangle\right) \otimes \left(\cos\theta_B\big|H\big\rangle + \sin\theta_B\big|V\big\rangle\right)$$
which we can reduce to:
$$=\cos\theta_A\cos\theta_B\big|H,H\big\rangle + \cos\theta_A\sin\theta_B\big|H,V\big\rangle + \sin\theta_A\cos\theta_B\big|V,H\big\rangle + \sin\theta_A\sin\theta_B\big|V,V\big\rangle.$$
Find the probability of a joint measurement of polarizations $\theta_A$ and $\theta_B$:
$$P(\theta_A,\theta_B) = \big|\big\langle\psi\big|\theta_A,\theta_B\big\rangle\big|^2$$
Since $\big|\psi\big\rangle$ only has $\big|H,H\big\rangle$ and $\big|V,V\big\rangle$ terms, this probability only has two terms:
$$P(\theta_A,\theta_B) = \left|\sqrt{0.2}\cos\theta_A\cos\theta_B + \sqrt{0.8}\sin\theta_A\sin\theta_B\right|^2$$
Plot is shown below for $\theta_A = -\theta_B$ and it agrees perfectly with our model above.
Can you write Python code for it?

# Solution:
# For the first plot, we can show the joint probability for two angles is given by:

plt.plot(rad2deg(rads),(sqrt(0.2)*cos(-rads)*cos(rads) + sqrt(0.8)*sin(-rads)*sin(rads))**2)


I am working on the file "Chapter 8 - Alice and Bob.ipynb".
The first task is:
These all agree (as they should).
Explore the angles in more detail:
Why these angles, 19 and 35?
Can you write Python code for it?

psi=sqrt(0.2)*tensor(H,H) + sqrt(0.8)*tensor(V,V)

angles = arange(1,90,1)

rads = deg2rad(angles)


I am working on the file "Chapter 8 - Alice and Bob.ipynb".
The first task is:
This is harder to interpret, but we clearly have different probabilities. Finally, check if we had used a mixed state:
A mixed state instead of the pure (entangled state).
Here we have to use the density matrix (since a ket cannot describe a mixed state). First some background:
QuTiP has a function that gives the density matrix from a ket state: ket2dm.
Can you write Python code for it?

rho_mix = 0.2 * ket2dm(tensor(H,H)) + 0.8 * ket2dm(tensor(V,V))
rho_mix

# joint for A1, B1
expect(tensor(Pa1,Pb1),rho_mix)

# B2 conditioned on A1
expect(tensor(Pa1,Pb2),rho_mix) / expect(tensor(Pa1,qeye(2)),rho_mix)

# A2 conditioned on B1
expect(tensor(Pa2,Pb1),rho_mix) / expect(tensor(Pb1,qeye(2)),rho_mix)

# joint for A2, B2:
expect(tensor(Pa2,Pb2),rho_mix)


I am working on the file "Chapter 8 - Alice and Bob.ipynb".
The first task is:
This is what we described in class.
What if the state was just $|H,H\rangle$?
Can you write Python code for it?

psi2=tensor(H,H)

expect(tensor(Pa1,Pb1),psi2)  # joint for A1, B1 (expect 0.09)

# B2 conditioned on A1:
expect(tensor(Pa1,Pb2),psi2) / expect(tensor(Pa1,qeye(2)),psi2)

# A2 conditioned on B1
expect(tensor(Pa2,Pb1),psi2) / expect(tensor(qeye(2),Pb1),psi2)

# joint for A2, B2
expect(tensor(Pa2,Pb2),psi2)


I am working on the file "Chapter 8 - Alice and Bob.ipynb".
The first task is:
We see that the joint probabilities have a zero at 35˚. Now plug that in to one of the conditional probabilities and see what angle for the conditional probability gives 1:
Can you write Python code for it?

out = []
for r in rads:
    out.append(expect(tensor(P(r),P(deg2rad(35))),psi) / expect(tensor(P(r),qeye(2)),psi))

plt.plot(angles,out,".")


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
A hands-on case study: French plays
OK, it time to get your hands even more dirty. For textual analyses, there are a number of great datasets out there which have been encoded in rich XML. One excellent resource which we have recently worked with, can be found at theatre-classique.fr: this website holds an extensive collection of French plays from the time of the Classical and Enlightenment era in France. Some of the plays have been authored by some of France's finest authors such as Molière pr Pierre and Thomas Corneille. What is interesting about this resource, is that it provides a very rich XML markup: apart from extensive metadata on the play or a detailed descriptions of the actors involved, the actually lines have been encoded in such a manner, that we perfectly know which character uttered a particular line, or to which scene or act a line belongs. This allows us to perform much richer textual analyses than if we would only have a raw text version of the plays. We have collected a subset of these plays for you under the data/TEIdirectory:
Can you write Python code for it?

import os
dirname = "data/TEI/french_plays/"
for filename in os.listdir(dirname):
    if filename.endswith(".xml"):
        print(filename)


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
Adding whole elements is fairly easy too. Let's add a single dummy element (<break/>) to indicate a line break at the end of each line. Importantly, we have to create this element inside our loop, before we can add it:
Can you write Python code for it?

break_el = etree.Element("break")
break_el.attrib["author"] = "Mike"
print(etree.tostring(break_el).decode())


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
For the record, we should mention that there exist many other libraries in Python to parse XML, such as minidom or BeautifulSoup which is an interesting library, when you intend to scrape data from the web. While these might come with more advanced bells and whistles than lxml, they can also be more complex to use, which is why we stick to lxml in this course. Let us now import our sonnet in Python, which has been saved in the file sonnet18.xml:
Can you write Python code for it?

tree = etree.parse("data/TEI/sonnet18.xml")
print(tree)


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
If we have more complex data, it might also be to set the pretty_print parameter to True, to obtain a more beautifully formatted string, with Python taking care of indendation etc. In our example, it doesn't change much:
Can you write Python code for it?


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
If we wouldn't know which attributes were in fact available for a node, we could also retrieve the attribute names by calling keys() on the attributes property of a node, just like we would do with a regular dictionary:
Can you write Python code for it?

for key in root_node.attrib.keys():
    print(root_node.attrib[key])


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
Note that the search expression ("//rhyme") has two forward slashes before our actual search term. This is in fact XPath syntax, and the two slashes indicate that the search term can occur anywhere (e.g. not necessarily among a node's direct children). Unfortunately, printing the nodes themselves again isn't really insightful: in this way, we only get rather prosaic information of the Python objects holding our rhyme nodes. We can use the .tag property to print the tag's name:
Can you write Python code for it?

for node in tree.iterfind("//rhyme"):
    print(node.tag)


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
Note that we get an empty line at the volta, since there isn't any actual text associated with this empty tag.
Quiz!
Could you now write your own code, which iterates over the lines in our tree and prints out the line number based on the n attribute of the line element?
Can you write Python code for it?

for node in root_node:
    if node.tag == "line":
        print(node.attrib["n"])


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
Now let us start processing the contents of our file. Suppose that we are not really interested in the full hierarchical structure of our file, but just in the rhyme words occuring in it. The high-level function interfind() allows us to easily select all rhyme-element in our tree, regardless of where exactly they occur. Because this functions returns a list of nodes, we can simply loop over them:
Can you write Python code for it?

for node in tree.iterfind("//rhyme"):
    print(node)


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
OK: under this directory, we appear to have a bunch of XML-files, but their titles are just numbers, which doesn't tell us a lot. Let's have a look at what's the title and author tags in these files:
Can you write Python code for it?

for filename in os.listdir(dirname):
    if filename.endswith(".xml"):
        print("*****")
        print("\t-", filename)
        tree = etree.parse(dirname+filename)
        author_element = tree.find("//author") # find vs iterfind!
        print("\t-", author_element.text)
        title_element = tree.find("//title")
        print("\t-", title_element.text)


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
Our root node turns out to have 15 child nodes, which makes a lot of sense, since we have 14 line elements and the volta. We can actually loop over these children, just as we would loop over any other list:
Can you write Python code for it?

for node in root_node:
    print(node.tag)


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
Python for TEI
In Digital Humanities, you hear a lot about the TEI nowadays, or the Text Encoding Initiative (tei-c.org). The TEI refers to an initiative which has developed a highly influential "dialect" of XML for encoding texts in the Humanities. The beauty about XML is that tag names aren't predefined and you can invent your own tag and attributes. Our Shakepearean example could just have well have read:
<?xml version="1.0"?>
<poem writer="William Shakepeare" date="1609">
    <l nr="1">Shall I compare thee to a summer's <last>day</last>?</l>
    <l nr="2">Thou art more lovely and more <last>temperate</last>:</l>
    <l nr="3">Rough winds do shake the darling buds of <last>May</last>,</l>
    <l nr="4">And summer's lease hath all too short a <last>date</last>:</l>
    <l nr="5">Sometime too hot the eye of heaven <last>shines</last>,</l>
    <l nr="6">And often is his gold complexion <last>dimm'd</last>;</l>
    <l nr="7">And every fair from fair sometime <last>declines</last>,</l>
    <l nr="8">By chance, or nature's changing course, <last>untrimm'd</last>;</l>
    <break/>
    <l nr="9">But thy eternal summer shall not <last>fade</last></l>
    <l nr="10">Nor lose possession of that fair thou <last>ow'st</last>;</l>
    <l nr="11">Nor shall Death brag thou wander'st in his <last>shade</last>,</l>
    <l nr="12">When in eternal lines to time thou <last>grow'st</last>;</l>
    <l nr="13">So long as men can breathe or eyes can <last>see</last>,</l>
    <l nr="14">So long lives this, and this gives life to <last>thee</last>.</l>
</poem>
As you can see, all the tag and attribute names are different in this version, but the essential structure is still the same. You could therefore say that XML is a markup language which provides a syntax to talk about texts, but does not come with a default semantics. This freedom in choosing name tags etc. can also be a bit daunting: this is why the TEI provides Guidelines as how tag names etc. can be used to mark up specific phenomena in texts. The TEI therefore also refers...
Can you write Python code for it?

tree = etree.parse("data/TEI/sonnet17.xml")
print(etree.tostring(tree).decode())


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
Python has now read and parsed our xml-file via the etree.parse() function. We have stored our XML tree structure, which is returned by the parse() function, in the tree variable, so that we can access it later. If we print tree as such, we don't get a lot of useful information. To have a closer look at the XML in a printable text version, we need to call the tostring() method on the tree before printing it.
Can you write Python code for it?


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
Quiz
The <break/> element is still empty: could you add to it an n attribute, to which you assign the line number from the current <line> element?
Can you write Python code for it?

tree = etree.parse("data/TEI/sonnet18.xml")
root_node = tree.getroot()
for node in root_node:
    if node.tag == "line":
        v = node.attrib["n"]
        break_el = etree.Element("break")
        break_el.attrib["n"] = v
        node.append(break_el)
print(etree.tostring(tree).decode())


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
That looks better!
Just now, we have been iterating over our rhyme elements in simple order of appearance: we haven't been really been exploiting the hierarchy of our XML file yet. Let's see now how we can navigate our xml tree. Let's first select our root node: there's a function for that!
Can you write Python code for it?

root_node = tree.getroot()
print(root_node.tag)


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
To extract the actual rhyme word contained in the element, we can use the .text property of the nodes:
Can you write Python code for it?

for node in tree.iterfind("//rhyme"):
    print(node.text)


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
To extract the actual text in our lines, we need one additional for-loop which will allow us to iteratre over the pieces of text under each line:
Can you write Python code for it?

for node in root_node:
    if node.tag != "volta":
        line_text = ""
        for text in node.itertext():
            line_text = line_text + text
        print(line_text)
    else:
        print("=== Volta found! ===")


I am working on the file "Chapter 8 - Parsing XML.ipynb".
The first task is:
You'll notice that we actually get a string in a raw format: if we want to display it properly, we have to decode it:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
After passing our tokens twice to the counter, we see that the numbers double in size.
Quiz
Write code that makes a word frequency counter named vocab, which counts the cumulative frequencies of all words in the Arabian Nights. Which are the 15 most frequent words? Does that make sense?
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
And lo behold: the directory has disappeared again:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Apart from os.listdir() there are a number of other common ways to obtain directory listings in Python. Using the glob module for instance, we can easily access the full relative path leading to our Arabian Nights:
Can you write Python code for it?

import glob
filenames = glob.glob('data/arabian_nights/*')
print(filenames[:10])


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
As you can see, os.walk() allows you to efficiently loop over the entire tree. As always, don't forget that help is right around the corner in your notebooks. Using help(), you can quickly access the documentation of modules and their functions etc. (but only after you have imported the modules first!).
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
As you can see, this folder holds a number of plain text files, ending in the .txt extension. Let us open a random file:
Can you write Python code for it?

f = open('data/arabian_nights/848.txt', 'r')
text = f.read()
f.close()
print(text[:500])


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
As you can see, this simple command can be used to quickly obtain a visualization that shows interesting trends. On the y-axis, we plot absolute word counts for each of our nights. The x-axis is figured out automatically by matplotlib and adds an index on the horizontal x-axis. Implicitly, it interprets our command as follows:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Both os.sep and os.path.basename have the advantage that they know what separator is used for paths in the operating system, so you don't need to explicitly code it like in the first solution. Separators differ between Windows (backslash) and Mac/Linux (forward slash).
Finally, sometimes, you might be interested in all the subdirectories of a particular directory (and all the subdirectories of these subdirectories etc.). Parsing such deep directory structures can be tricky, especially if you do not know how deep a directory tree might run. You could of course try stacking multiple loops using os.listdir(), but a more convenient way is os.walk():
Can you write Python code for it?

for root, directory, filename in os.walk("data"):
    print(filename)


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Et voilà!
Closing Assignment
In this larger assignment, you will have to perform some basic text processing on the larger set of XML-encoded files under data/TEI/french_plays. For this assignment, there are several subtasks:
1. Each of these files represent a play written by a particular author (see the <author> element): count how many texts were written by each author in the entire corpus. Make use of a Counter.
2. Each play has a cast list (<castList>), with a role-element for every character in it. In this element, the civil-attribute encodes the gender of the character (M/F, or another charatcer ). Create for each individual author a barplot using matplotlib, showing the percentage of male, female and 'other' characters as a percentage. Pick beautiful colors.
3. Difficult: The information contained in the castList is priceless, because it allows us to determine for each word in the play by whom it is uttered, since the <sp> tag encodes which character in the cast list is speaking at a particular time. Parse play 156.xml (L'Amour à la mode) and calculate which of the characters has the highest vocabulary richness: divide the number of unique words in the speaker's utterances by the total number of words (s)he utters. Only consider speakers that utter at least 1000 tokens in the play.
Hint: If your run into encoding errors etc. when processing larger text collections, you can always use try/except constructions to catch these.

Ignore the following, it's just here to make the page pretty:
Can you write Python code for it?

from IPython.core.display import HTML
def css_styling():
    styles = open("styles/custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Here, we created a directory, wrote a new file to it (1001.txt), and removed it again. Using os.path.exists() we monitored at which point the file existed. Finally, the shutil module also ships with a useful copyfile() function which allows you to copy files from one location to another, possibly with another name. To copy night 66 to the present directory, for instance, we could do:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Here, we use the open() function to create a file object f, which we can use to access the actual text content of the file. Make sure that you do not pass the 'w' parameter ("write") to open(), instead of 'r' ("read"), since this would overwrite and thus erase the existing file.  After assigning the string returned by f.read() to the variable text, we print the 500 first characters of text to get an impression of what it contains, using simple string indexing ([:500]). Don't forget to close the file again after you have opened or strange things could happen to your file! One little trick which is commonly used to avoid having to explicitly open and close your file is a with block (mind the indentation):
Can you write Python code for it?

with open('data/arabian_nights/848.txt', 'r') as f:
    text = f.read()
print(text[:500])


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Here, we use the rmtree() command to remove the entire directory in a recursive way: even if the directory isn't empty and contains files and subfolders, we will remove all of them. The os module also comes with a rmdir() but this will not allow you to remove a directory which is not empty, as becomes clear in the OSError raised below:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
However, when using glob.glob(), you might sometimes want to be able to extract a file's base name again. There are several solutions to this:
Can you write Python code for it?

filenames = glob.glob('data/arabian_nights/*.txt')
fn = filenames[10]

# simple string splitting:
print(fn.split('/')[-1])

# using os.sep:
print(fn.split(os.sep)[-1])

# using os.path:
print(os.path.basename(fn))


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
If we would like to find which items are most frequent for instance, we could simply do:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Indeed, we have added an exact copy of night 66 to our present working directory:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Interestingly, the command in this code block will only load filenames that end in ".txt". This is interesting when we would like to ignore other sorts of junk files etc. that might be present in a directory. To replicate similar behaviour with os.listdir(), we would have needed a typical for-loop, such as:
Can you write Python code for it?

filenames = []
for fn in os.listdir('data/arabian_nights'):
    if fn.endswith('.txt'):
        filenames.append(fn)
print(filenames[:10])


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
It might therefore be convenient to check whether a directory actually exists in a given location:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Let us now finally visualize the frequencies of the 15 most frequent items using a standard barplot in matplotlib. This can be achieved as follows. We first split out the names and frequencies, since .mostcommon(n) returns a list of tuples, and we create indices:
Can you write Python code for it?

freqs = [f for _, f in vocab.most_common(15)]
words = [w for w, _ in vocab.most_common(15)] # note the use of underscores for 'throwaway' variables
idxs = range(1, len(freqs)+1)


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Note: There is a more elegant, but also slightly less trivial way to achieve the correct order in this case:
Can you write Python code for it?

for fn in sorted(os.listdir('data/arabian_nights/'),
                key=lambda nb: int(nb[:-4])):
    print(fn)


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Or for you stylish coders out there, you can show off with a list comprehension:
Can you write Python code for it?

filenames = [fn for fn in os.listdir('data/arabian_nights') if fn.endswith('.txt')]


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Or we use Python:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Parsing files
Using the code from the previous quiz, it is now trivial to sort our nights sequentially on the basis of their actual name (i.e. a string variable):
Can you write Python code for it?

for fn in sorted(os.listdir('data/1001')):
    print(fn)


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Paths
The paths we have used so far are 'relative' paths, in the sense that they are relative to the place on our machine from which we execute our Python code. Absolute paths can also be retrieved and will differ on each computer, because they typically include user names etc:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Python throws a FileNotFoundError, complaining that the file we wish to open does not exist. This situation stems from the fact that os.listdir() only returns the base name of a given file, and not an entire (absolute or relative) path to it. To properly access the file, we must therefore not forget to include the rest of the path again:
Can you write Python code for it?

filenames = os.listdir('data/arabian_nights')
random_filename = filenames[9]
with open('data/arabian_nights/'+ random_filename, 'r') as f:
    text = f.read()
print(text[:500])


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Quiz

Using axvline() you can add vertical lines to a plot, for instance at position:
Can you write Python code for it?

plt.plot(idxs, word_counts, color='r')
plt.xlabel('Word length')
plt.ylabel('# words (absolute counts)')
plt.title(r'The Arabian Nights')
plt.xlim(1, 1001)
plt.axvline(500, color='g')


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Quiz
Iterate over all the nights in 1001 in a sorted way. Open, preprocess and tokenize each text. Store in a list called word_counts how many words each story has.
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Removing directories is also easy, but PLEASE watch out, sometimes it is too easy: if you remove a wrong directory in Python, it will be gone forever... Unlike other applications, Python does not keep a copy of it in your Trash and it does not have a Ctrl-Z button. Please watch out with what you do, since with great power comes great responsiblity! Removing the entire directory which we just created can be done as follows:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Right now, we are visualizing texts, but we might also be interested in the vocabulary used in the story collection. Counting how often a word appears in a text is trivial for you right now with custom code, for instance:
Can you write Python code for it?

cnts = {}
for word in tokens:
    if word in cnts:
        cnts[word] += 1
    else:
        cnts[word] = 1
print(cnts)


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Should you be interested: here, we pass a key argument to sort, which specifies which operations should be applied to the filenames before actually sorting them. Here, we specify a so-called lambda function to key, which is less intuitive to read, but which allow you to specify a sort of 'mini-function' in a very condensed way: this lambda function chops off the last four characters from each filename and then converts (or 'casts') the results to a new data type using int(), namely an integer (a 'whole' number, as opposed to floating point numbers). Eventually, this leads to the same order.

More functions
So far, we have been using pre-existing, ready-made functions from Python's standard library, or the standard set of functionality which comes with the programming language. Importantly, there are two additional ways of using functions on your code, which we will cover below: (i) you can write your own functions, and (ii) you can use functions from other, external libraries, which have been developped by so-called 'third parties'. Below, we will for instance use plotting functions from matplotlib, which is a common visualization library for Python.
At this point, we have an efficient way of looping over the Arabian Nights sequentially. What we still lack, are functions to load and clean our data. As you could see above, our files still contain a lot of punctuation marks etc., which are perhaps less interesting from the point of view of textual analysis. Let us write a simple function that takes a string as input, and returns a cleaner version of it, where all characters are lowercased, and only alphabetic characters are kept:
Can you write Python code for it?

import re
def preprocess(in_str):
    out_str = ''
    for c in in_str.lower():
        if c.isalpha() or c.isspace():
            out_str += c
    whitespace = re.compile(r'\s+')
    out_str = whitespace.sub(' ', out_str)
    return out_str


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
The asterisk (*) in the argument passed to glob.glob() is worth noting here. Just like with regular expressions, this asterisk is a sort of wildcard which will match any series of characters (i.e. the filenames under arabian_nights). When we exploit this wildcard syntax, glob.glob() offers another distinct advantage: we can use it to easily filter out filenames which we are not interested in:
Can you write Python code for it?

filenames = glob.glob('data/arabian_nights/*.txt')
print(filenames[:10])


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
The function os.listdir() returns a list of strings, representing the filenames contained under a directory.
Quiz

In Burton's translation some of the 1001 nights are missing. How many?
Can you come up with a clever way to find out which nights are missing? Hint: a counting loop and some string casting might be useful here!
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
The second directory, naturally, does not exist and isdir() evaluates to False in this case. Creating a new (and thus empty) directory is also easy using os:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
The second line is needed to make sure that the plots will properly show up in our notebook. Let us start with a simple visualization:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
This Counter makes it much easier to write code for counting. Below you can see how this counter automatically creates a dictionary-like structure:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
This code block does exactly the same thing as the previous one but saves you some typing. In this chapter we would like to work with all the files in the arabian_nights directory. This is where loops come in handy of course, since what we really would like to do, is iterate over the contents of the directory. Accessing these contents in Python is easy, but requires importing some extra functionality. In this case, we need to import the os module, which contains all functionality related to the 'operating system' of your machine, such as directory information:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
This code reviews some of the materials from previous chapters, including the use of a regular expression, which converts all consecutive instances of whitespace (including line breaks, for instance) to a single space. After executing the previous code block, we can now test our function:
Can you write Python code for it?

old_str = 'This;     is -- a very    DIRTY string!'
new_str = preprocess(old_str)
print(new_str)


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Using the dot-syntax (os.xxx), we can now access all functions that come with this module, such as listdir(), which returns a list of the items which are included under a given directory
Can you write Python code for it?

filenames = os.listdir('data/arabian_nights')
print(len(filenames))
print(filenames[:20])


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Using the list comprehension, we make sure that we do not accidentally return empty strings as a token, for instance, at the beginning of a text which starts with a newline. Remember that anything in Python with a length of 0, will evaluate to False, which explains the if t in the comprehension: empty strings will fail this condition. We can start stacking our functions now:
Can you write Python code for it?

with open('data/1001/0007.txt', 'r') as f:
    in_str = f.read()
tokens = tokenize(preprocess(in_str))
print(tokens[:10])


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Using the old filenames, this was not possible directly, because of the way Python sorts strings of unequal lengths. Note that the number in the filenames are represented as strings, which are completely different from real numeric integers, and thus will be sorted differently:
Can you write Python code for it?

for fn in sorted(os.listdir('data/arabian_nights/')):
    print(fn)


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
We can also pass the Counter the tokens to count in multiple stages:
Can you write Python code for it?

cnt = Counter()
cnt.update(tokens)
cnt.update(tokens)
print(cnt.most_common(25))


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
We can now apply this function to the contents from a random night:
Can you write Python code for it?

with open('data/1001/0007.txt', 'r') as f:
    in_str = f.read()
print(preprocess(in_str))


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
We can now make our plot more truthful, and add some bells and whistles:
Can you write Python code for it?

plt.plot(idxs, word_counts, color='r')
plt.xlabel('Word length')
plt.ylabel('# words (absolute counts)')
plt.title('The Arabian Nights')
plt.xlim(1, 1001)


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
We can now start analyzing our nights. A good start would be to check the length of each night in words:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
We can safely remove it again:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
We can see that it lives in the present working directory now, by typing ls again:
Can you write Python code for it?


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
When plt.plot receives two flat lists as arguments, it plots the first along the x-axis, and the second along the y-axis. If it only receives one list, it plots it along the y-axis and uses the range we now (redundantly) specified here for the x-axis. This is in fact a subtoptimal plot, since the index of the first data point we plot is zero, although the name of the first night is '1.txt'. Additionally, we know that there are some nights missing in our data. To set this straight, we could pass in our own x-coordinates as follows:
Can you write Python code for it?

filenames = sorted(os.listdir('data/1001'))
idxs = [int(i[:-4]) for i in filenames]
print(idxs[:20])
print(min(idxs))
print(max(idxs))


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
While absolute paths are longer to type, they have the advantage that they can be used anywhere on your computer (i.e. irrespective of where you run your code from). Paths can be tricky. Suppose that we would like to open one of our filenames:
Can you write Python code for it?

filenames = os.listdir('data/arabian_nights')
random_filename = filenames[9]
with open(random_filename, 'r') as f:
    text = f.read()
print(text[:500])


I am working on the file "Chapter 9 - Text analysis.ipynb".
The first task is:
Write code that plots the position of the missing nights using this function (and blue lines).
Can you write Python code for it?


I am working on the file "Chapter-2-Regression.ipynb".
The first task is:
Exercise 2-3
Experiment with different values of alpha, theta, and iters. What do you find?
Results
Let's see how our optimal parameter values can be used to make predictions.
Can you write Python code for it?

x = np.linspace(data.Population.min(), data.Population.max(), 100)
f = theta_opt[0, 0] + (theta_opt[1, 0] * x)

fig, ax = plt.subplots(figsize=(8,5))
ax.plot(x, f, 'r', label='Prediction')
ax.scatter(data.Population, data.Profit, label='Training Data')
ax.legend(loc=2)
ax.set_xlabel('Population')
ax.set_ylabel('Profit')
ax.set_title('Predicted Profit vs. Population Size')


I am working on the file "Chapter-2-Regression.ipynb".
The first task is:
Let's calculate the cost over the entire dataset of getting it wrong for
$\begin{bmatrix}
\theta_{0} \
\theta_{1}
\end{bmatrix} =
\begin{bmatrix}
-10 \
1
\end{bmatrix}$
Can you write Python code for it?

# Get X and y into the right shapes for use in computeCost
# Add a first column of 1s to X
X.insert(0,'Ones', 1)

X.shape

# The new form of the inputs
X.head()

# Turn inputs, outputs and thetas into matrices to suit the costCompute function
X = np.matrix(X.values)
y = np.matrix(y.values)

# Choose values for theta
theta_test = np.matrix('-10;1')

c = computeCost(X,y,theta_test)
c


I am working on the file "Chapter-2-Regression.ipynb".
The first task is:
Step 1: Define the Input(s)
For this data, we have a single input -- town population. We have 97 different town populations in our dataset. That's 97 different values for our single input variable. Keep in mind that each value is in 10,000s. So multipy the value you see by 10,000 to get the actual value of the population.
Can you write Python code for it?

# Here are the input values
# Number of colums in our dataset
cols = data.shape[1]
# Inputs are in the first column - indexed as 0
X = data.iloc[:, 0:cols-1]
print "number of columns %d" %cols
print "sample inputs \n %s" %X.head()


I am working on the file "Chapter-2-Regression.ipynb".
The first task is:
Step 2: Define the Output
The output is annual restaurant profit. For each value of the input we have a value for the output. Keep in mind that each value is in \$10,000s. So multipy the value you see by \$10,000 to get the actual annual profit for the restaurant.
Let's look at some of these output values.
Can you write Python code for it?

# Here are the output vaues
# Outputs are in the second column - indexed as 1
y = data.iloc[:, cols-1:cols]

# See a sampe of the outputs
y.head()


I am working on the file "Chapter-2-Regression.ipynb".
The first task is:
The complete data set can be described using the traditional statistical descriptors:
Can you write Python code for it?


I am working on the file "Chapter-2-Regression.ipynb".
The first task is:
The first few rows of our data look like this:
Can you write Python code for it?


I am working on the file "Chapter-3-Regression-With-Many-Inputs.ipynb".
The first task is:
Rescale/Normalize the Data
Notice that the size of a house in square feet is about 1,000 times the number of bedrooms it has. Similarly, the price of a house is about 100 times the size of the house. This is quite common in many datasets, but when it happens, the iterative method of gradient descent (which we'll use again) becomes inefficient. This is simply a matter of making the computations efficient. To do so, we'll do something called feature normalization.
Can you write Python code for it?

# Normalize the dataset
data2Norm = (data2 - data2.mean()) / data2.std()
data2Norm.head()

# Let's keep track of the mean and standard deviation of the house sizes, number of bedrooms, and prices in the dataset.
# We'll need these values when we make predictions

# We can get them easily for size, bedrooms and price by using data2.mean()[0], ..., [2]
data2.mean()[0], data2.mean()[1], data2.mean()[2], data2.std()[0], data2.std()[1], data2.std()[2]


I am working on the file "Chapter-3-Regression-With-Many-Inputs.ipynb".
The first task is:
Results
It looks like a learning rate ($\alpha$ value) greater than 0.003 is good enough to get our iterative gradient descent to plunge down to arrive at the lowest cost value and stay there.
Let's make some predictions...What is our prediction for a house that is 5,000 square feet in size with 4 bedrooms? Let's plug these in to our model and use the optimal $\theta$ values we've calculated above.
Can you write Python code for it?

# Change size and num_bedrooms to make distinct predictions
size = 5000
num_bedrooms = 4

# Remember we've run the model using rescaled house sizes and number of bedrooms
# So we should scale the inputs down and then scale the prediction up when we're done
size_scaled = (size - data2.mean()[0])/data2.std()[0]
beds_scaled = (num_bedrooms - data2.mean()[1])/data2.std()[1]

# This is our model -- we're just using it here to make a calculation
pred_price = (theta_opt.item(0,0) * 1) + (theta_opt.item(1,0) * size_scaled) + (theta_opt.item(2,0) * beds_scaled)

# Now we re-re-scale the price to convert it back to the right order of magnitude
pred_price_real = (pred_price * data2.std()[2]) + data2.mean()[2]

# Format and print the result
print "Predicted Price: ", '{:20,.0f}'.format(math.ceil(pred_price_real))


I am working on the file "Chapter-3-Regression-With-Many-Inputs.ipynb".
The first task is:
Step 2: Define the Output
The output is the house price. Let's set it up as a variable y.
Can you write Python code for it?

# The output
y = data2Norm.iloc[:, cols-1:cols]

y.head()


I am working on the file "Chapter-3-Regression-With-Many-Inputs.ipynb".
The first task is:
Step 3: Define the Model
We're going to stick to a simple model -- exactly the same one as before but now with an additional input:
$$y = h_{\theta}(X) = \theta_{0} * x_{0}\ +\ \theta_{1} * x_{1} +\ \theta_{2} * x_{2}$$
It's still linear but now has more than 1 variable, hence the $x_{2}$.
Step 4: Define the Parameters of the Model
$\theta_{0}$, $\theta_{1}$, and $\theta_{2}$ are the parameters of the model. Unlike our example of the boiling water in Chapter 1, these parameters can each take on an infinite number of values. $\theta_{0}$ is called the bias value.
With this model, we know exactly how to transform an input into an output -- that is, once the values of the parameters are given.
Let's pick a value of X from the dataset, fix a specific value for $\theta_{0}$ and $\theta_{1}$, and see what we get for the value of y.
Specifically, let
$\begin{bmatrix}
\theta_{0} \
\theta_{1} \
\theta_{2}
\end{bmatrix} =
\begin{bmatrix}
-10 \
1 \
0
\end{bmatrix}$
This means $\theta_{0}$ is -10, $\theta_{1}$ is 1, and $\theta_{2}$ is 0.
Let's try out X * theta for the first few rows of X.
Can you write Python code for it?

# Outputs generated by our model for the first 5 inputs with the specific theta values below
theta_test = np.matrix('-10;5;1.2')
outputs = np.matrix(X.iloc[0:5, :]) * theta_test
outputs

# Compare these outputs to the actual values of y in the dataset
y.head()


I am working on the file "Chapter-3-Regression-With-Many-Inputs.ipynb".
The first task is:
The Business Problem: Predicting Housing Prices
What's the market value of a house? One way to determine the market price is to collect up the prices of houses based on a few characteristics such as size in square feet and number of bedrooms. Then think like a machine and "learn" what the price should be based on this data.
The data we have is housing prices in Portland, Oregon. Let's see what it looks like.
Load the Data
Can you write Python code for it?

import os
path = os.getcwd() + '/Data/ex1data2.txt'
data2 = pd.read_csv(path, header=None, names=['Size', 'Bedrooms', 'Price'])
data2.head()

data2.describe()

data2.Size.min()


I am working on the file "Chapter-3-Regression-With-Many-Inputs.ipynb".
The first task is:
We don't know yet if this is high or low -- we'll have to try out a whole bunch of $\theta$ values. Or better yet, we can use pick an iterative method and implement it.
Step 6: Pick an Iterative Method to Minimize the Cost of Getting it Wrong
Once again, the method that will "learn" the optimal values for $\theta$ is gradient descent. We don't have to do a thing to the function we wrote before for gradient descent. Let's use it to find the minimum cost and the values of $\theta$ that result in that minimum cost.
Can you write Python code for it?

theta_init = np.matrix('10;10;10')
# Run gradient descent for a number of different learning rates
alpha = [0.001, 0.003, 0.01, 0.03, 0.1]
iters = 500
theta_ag = np.matrix(np.zeros((3,5)))
cost_ag = np.zeros((5, iters))
for i in range(5):
    theta_opt, cost_min = gradientDescent(np.matrix(X.values), np.matrix(y.values), theta_init, alpha[i], iters)
    theta_ag[:,i] = theta_opt
    cost_ag[i,:] = cost_min

theta_ag[:, 4]

#cost_ag[4,:]

# This is the value of theta for the last iteration above -- hence for alpha = 0.1
theta_opt

# The last cost of the last iteration is the minimum that gradient descent was able to find
# See cost_ag[:,4] above to check the structure of this output
cost_min[-1]

fig, ax = plt.subplots(figsize=(8,5))
ax.plot(np.arange(iters), cost_ag[0,:], 'g', label="alpha = 0.001")
ax.plot(np.arange(iters), cost_ag[1,:], 'r', label="alpha = 0.003")
ax.plot(np.arange(iters), cost_ag[2,:], 'y', label="alpha = 0.01")
ax.plot(np.arange(iters), cost_ag[3,:], 'b', label="alpha = 0.03")
ax.plot(np.arange(iters), cost_ag[4,:], 'c', label="alpha = 0.1")
ax.set_xlabel('Iterations')
ax.set_ylabel('Cost')
ax.legend()
ax.set_title('How Costs Fall as Alpha is Changed')


I am working on the file "Chapter-3-Regression-With-Many-Inputs.ipynb".
The first task is:
We have information about 47 homes in this dataset. Each house is described by two characteristics or "features".
Visualize the Data
Can you write Python code for it?

data2.plot.scatter(x='Size', y='Price', figsize=(8,5))

data2.plot.scatter(x='Bedrooms', y='Price', figsize=(8,5))

data2.plot.scatter(x='Bedrooms', y='Size', figsize=(8,5))

# 3D scatter plot -- just because...
from mpl_toolkits.mplot3d import Axes3D
threedee = plt.figure(figsize=(10,8)).gca(projection='3d')
threedee.scatter(data2['Bedrooms'], data2['Size'], data2['Price'])
threedee.set_xlabel('Bedrooms')
threedee.set_ylabel('Size')
threedee.set_zlabel('Price')
plt.show()


I am working on the file "Chapter-4-Non-Linear-Regression.ipynb".
The first task is:
Exercise 4-1
Based on the visuals above, how would you describe the data? Write a short paragraph describing the data.
Use Size as the Key Variable
Paradoxically, to demostrate how multivariate non-linear regression works, we'll strip down our original dataset into one that just has Size and Price; the Bedrooms part of the data is removed. This simplifies things so that we can easily visualize what's going on.
So, to visualize things more easily, we're going to focus just on the sinlge variable -- the size of the house. We'll turn this into a multi-variable situation in just a bit.
Can you write Python code for it?

# Just checking on the type of object data2 is ... good to remind ourselves
type(data2)

# First drop the Bedrooms column from the data set -- we're not going to be using it for the rest of this notebook
data3 = data2.drop('Bedrooms', axis = 1)
data3.head()

# Visualize this simplified data set
data3.plot.scatter(x='Size', y='Price', figsize=(8,5))


I am working on the file "Chapter-4-Non-Linear-Regression.ipynb".
The first task is:
How Polynomials Fit the Data
Let's visualize the fit for various degrees of polynomial functions.
Can you write Python code for it?

# Because Price is about 100 times Size, first normalize the data
data3Norm = (data3 - data3.mean()) / data3.std()
data3Norm.head()

X = data3Norm['Size']
y = data3Norm['Price']

# fit the data with a 2nd degree polynomial
z2 = np.polyfit(X, y, 2)
p2 = np.poly1d(z2) # construct the polynomial (note: that's a one in "poly1d")

# fit the data with a 3rd degree polynomial
z3 = np.polyfit(X, y, 3)
p3 = np.poly1d(z3) # construct the polynomial

# fit the data with a 4th degree polynomial
z4 = np.polyfit(X, y, 4)
p4 = np.poly1d(z4) # construct the polynomial

# fit the data with a 8th degree polynomial - just for the heck of it :-)
z8 = np.polyfit(X, y, 8)
p8 = np.poly1d(z8) # construct the polynomial

# fit the data with a 16th degree polynomial - just for the heck of it :-)
z16 = np.polyfit(X, y, 16)
p16 = np.poly1d(z16) # construct the polynomial

xx = np.linspace(-2, 4, 100)
plt.figure(figsize=(8,5))
plt.plot(X, y, 'o', label='data')
plt.xlabel('Size')
plt.ylabel('Price')
plt.plot(xx, p2(xx), 'g-', label='2nd degree poly')
plt.plot(xx, p3(xx), 'y-', label='3rd degree poly')
#plt.plot(xx, p4(xx), 'r-', label='4th degree poly')
plt.plot(xx, p8(xx), 'c-', label='8th degree poly')
#plt.plot(xx, p16(xx), 'm-', label='16th degree poly')
plt.legend(loc=2)
plt.axis([-2,4,-1.5,3]) # Use for higher degrees of polynomials


I am working on the file "Chapter-4-Non-Linear-Regression.ipynb".
The first task is:
Step 8: The Results
Let's make some predictions based on the values of $\theta_{opt}$. We're using our 4th-order polynomial as the model.
Can you write Python code for it?

size = 2
size_nonnorm = (size * data3.std()[0]) + data3.mean()[0]
price = (theta_opt[0] * 1) + (theta_opt[1] * size) + (theta_opt[2] * np.power(size,2)) + (theta_opt[3] * np.power(size,3)) + (theta_opt[4] * np.power(size,4))

price[0,0]

# Transform the price into the real price (not normalized)
price_mean = data3.mean()

price_std = data3.std()[1]

price_pred = (price[0,0] * price_std) + price_mean

price_pred

size_nonnorm

data3.mean()[1]


I am working on the file "Chapter-4-Non-Linear-Regression.ipynb".
The first task is:
Steps 1 and 2: Define the Inputs and the Outputs
Can you write Python code for it?

# Add a column of 1s to the X input (keeps the notation simple)
data3Norm.insert(0,'x0',1)
data3Norm.head()

data3Norm.insert(2,'Size^2', np.power(data3Norm['Size'],2))
data3Norm.head()

data3Norm.insert(3,'Size^3', np.power(data3Norm['Size'],3))
data3Norm.head()

data3Norm.insert(4,'Size^4', np.power(data3Norm['Size'],4))
data3Norm.head()


I am working on the file "Chapter-4-Non-Linear-Regression.ipynb".
The first task is:
That's quite a bit off from the actual values; so we know that the values for $\theta$ in theta_test must be quite far from the optimal values for $\theta$ -- the values that will minimize the cost of getting it wrong.
Step 5: Define the Cost of Getting it Wrong
Our cost function is exactly the same as it was before for the single variable case.
The cost of getting it wrong is defined as a function $J(\theta)$:
$$J(\theta) = \frac{1}{2m} \sum_{i=1}^{m} (h_{\theta}x^{(i)}) - y^{(i)})^2$$
The only difference from what we had before is the addition of the various $\theta$s and $x$s
$$h_{\theta}(X) = \theta_{0} * x_{0}\ +\ \theta_{1} * x_{1} +\ \theta_{2} * x_{2} +\ \theta_{3} * x_{3} +\ \theta_{4} * x_{4}$$
where $x_{2} = x_{1}^{2}$, $x_{3} = x_{1}^{3}$, and $x_{4} = x_{1}^{4}$.
Can you write Python code for it?

# Compute the cost for a given set of theta values over the entire dataset
# Get X and y in to matrix form
computeCost(np.matrix(X3.values), np.matrix(y3.values), theta_test)


I am working on the file "Chapter-4-Non-Linear-Regression.ipynb".
The first task is:
Visualize the Data
We can visualize the entire dataset as follows.
Can you write Python code for it?

import seaborn as sns
sns.set(style='whitegrid', context='notebook')
cols = ['Size', 'Bedrooms', 'Price']
sns.pairplot(data2[cols], size=2.5)
plt.show()


I am working on the file "Chapter-4-Non-Linear-Regression.ipynb".
The first task is:
We don't know yet if this is high or low -- we'll have to try out a whole bunch of $\theta$ values. Or better yet, we can use pick an iterative method and implement it.
Steps 6 and 7: Pick an Iterative Method to Minimize the Cost of Getting it Wrong and Implement It
Once again, the method that will "learn" the optimal values for $\theta$ is gradient descent. We don't have to do a thing to the function we wrote before for gradient descent. Let's use it to find the minimum cost and the values of $\theta$ that result in that minimum cost.
Can you write Python code for it?

theta_init = np.matrix('-1;0;1;0;-1')
# Run gradient descent for a number of different learning rates
alpha = 0.00001
iters = 5000

theta_opt, cost_min = gradientDescent(np.matrix(X3.values), np.matrix(y3.values), theta_init, alpha, iters)

# This is the value of theta for the last iteration above -- hence for alpha = 0.1
theta_opt

# The minimum cost
cost_min[-1]


I am working on the file "Chapter-5-Logistic Regression.ipynb".
The first task is:
Because the output of our classification is not a real number (unlike in regression where it can take on a value within a certain range of numbers), we need a function to take exam scores (or similar inputs) and produce two categories of output (say 0 for Not Admitted, and 1 for Admitted). The way to do this is to use the sigmoid function.
What the sigmoid function does is take an expression like the one we're familiar with below
$$h_{\theta}(x) = \theta_{0} x_{0} + \theta_{1} x_{1} + \theta_{2} x_{2}$$
and convert it to the following expression:
$$g(h_{\theta}(x)) = \frac{1}{1 + e^{-h_{\theta}(x)}}$$
where $e$ is the natural log. So what does the sigmoid function or sigmoid transformation look like?
Can you write Python code for it?

# Define the sigmoid function or transformation
# NOTE: ALSO PUT INTO THE SharedFunctions notebook
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Plot the sigmoid function
# Generate the values to be plotted
x_vals = np.linspace(-10,10,100)
y_vals = [sigmoid(x) for x in x_vals]

# Plot the values
fig, ax = plt.subplots(figsize=(8,5))
ax.plot(x_vals, y_vals, 'blue')
# Draw some constant lines to aid visualization
plt.axvline(x=0, color='black')
plt.axhline(y=0.5, color='black')
plt.yticks(np.arange(0,1.1,0.1))
plt.xticks(np.arange(-10,11,1))
plt.xlabel(r'$h_{\theta}(x)$', fontsize=15)
plt.ylabel(r'$g(h_{\theta}(x))$', fontsize=15)
plt.title('The Sigmoid Transformation')
ax.plot


I am working on the file "Chapter-5-Logistic Regression.ipynb".
The first task is:
Right away we see that this doesn't even look like a regular regression problem -- there are two classes -- Admitted and Not Admitted -- and we'd like to separate them. We need a technique for learning what class a student falls into based on his or her exam scores.
We'll start by trying to draw a line through the plot above that optimally separates the candidates who have been admitted from those who have not been admitted. We'll think like a machine to find the parameter values that define this line. The line is called a decision boundary -- depending on which side of the line a point falls, its fate is decided.
Can you write Python code for it?

# A few examples of decision boundaries
#plot_decision_regions(data.iloc[:, 1:3].values, data.iloc[:,3].values, 'ppn')
## TO DO


I am working on the file "Chapter-5-Logistic Regression.ipynb".
The first task is:
Step 5: Define the Cost of Getting it Wrong
The cost of classfying things wrong is defined as
$$J(\theta) = \frac{1}{m} \sum_{i=1}^{m} [-y^{(i)} log(g_{\theta}(x^{(i)})) - (1-y^{(i)}) log(1-g_{\theta}(x^{(i)}))]$$
As always, the cost of getting it wrong is defined over the entire dataset. There are $m$ rows of data and each $x^{(i)}$ consists of the two scores in a row of the dataset.
Let's visualize this cost function. [[VISUALIZE COST FUNCTION]]
Can you write Python code for it?

# Visualize the cost function when y = 1 and y = 0
x_vals = np.linspace(0,1,100)
y_1_vals = -np.log(x_vals)
y_0_vals = -np.log(1 - x_vals)

fig, ax = plt.subplots(figsize=(8,5))
ax.plot(x_vals, y_1_vals, color='blue', linestyle='solid', label='y = 1')
ax.plot(x_vals, y_0_vals, color='yellow', linestyle='solid', label='y = 0')
plt.legend(loc='upper center')
plt.xlabel(r'$g_{\theta}(x)$', fontsize=15)
plt.ylabel(r'$J(\theta)$', fontsize=15)
ax.plot


I am working on the file "Chapter-5-Logistic Regression.ipynb".
The first task is:
Steps 6 and 7: Pick an Iterative Method to Minimize the Cost of Getting it Wrong and Implement It
Once again, the method that will "learn" the optimal values for $\theta$ is gradient descent. For logistic regression we have to change our existing gradientDescent function to account for the sigmoid transformation. Otherwise, the expression for gradient descent looks the same as it did before, namely:
$$\frac{\partial J(\theta)}{\partial \theta_{j}} = \frac{1}{m}\sum_{i=1}^{m}(g_{\theta}(x^{(i)}) - y^{(i)}) x_{j}^{(i)}$$
Let's implement this new gradient descent function for logistic regression now.
Can you write Python code for it?

# BEGIN scratch work
# Doing some scratch work to make sure the matrix multiplication works out
# matrix multiplication is great for computational efficiency
test1 = np.matrix([[1],[2],[3],[4]])

test1

test2 = np.matrix([[1,2,3,4],[1,2,3,4], [1,2,3,4], [1,2,3,4]])

test2

test2 * test1

(test1.T * test2).T

# Right, that's the result we need
# END scratch work

# Gradient Descent for Logistic Regression/Classification
def logisticGradDescent(X, y, theta, alpha, iters):
    # X is a m x n matrix (including a first column of 1s)
    # y is a m x 1 matrix
    # theta is an n x 1 matrix
    # alpha is the learning rate
    # iters is the number of iterations of gradient descent

    # Keep track of the evolution of theta and cost values
    theta_agg = theta # initial value of the aggregated array
    cost_vals = logisticCost(X,y,theta) # initial value of cost for the initial theta value

    m = len(X)

    # Initialize theta for the iter loop below
    theta_val = theta

    # Notice there's only 1 loop -- the one over iters -- in this implementation
    for i in range(iters):
        error = np.subtract(sigmoid(X * theta), y)
        sum_error = (error.T * X).T # using the test1, test2 example above
        # no need to explicitly sum the error because the matrix multiplication does it automatically
        # Multiply by alpha and divide by m to normalize the sum_error
        norm_sum_error = np.divide(np.multiply(sum_error, alpha),m)
        # norm_sum_error is an n x 1 matrix containing the correction values for each theta parameter

        # Update all the thetas simultaneously
        theta_val = np.subtract(theta_val, norm_sum_error)

        # keep track of the latest theta val
        theta_agg = np.c_[theta_agg, theta_val]

        # Calculate the cost for these parameter values
        cost_vals = np.c_[cost_vals, logisticCost(X, y, theta_val)]

    return theta_agg, cost_vals

# Test it out
theta_out, cost_out = logisticGradDescent(X, y, theta,...


I am working on the file "Chapter-5-Logistic Regression.ipynb".
The first task is:
The beauty of the sigmoid transoformation is it gives us a way to take a variable with continuous values and transform it into a variable with just two values -- 0 or 1. The sigmoid can never be less than zero; nor can it be greater than 1. When $h_{\theta}(x)$ is around 6 or greater, the sigmoid is, for all practical purposes, equal to 1. Similarly, when $h_{\theta}(x)$ is around -6 or smaller, the sigmoid is for all practical purposes 0.
Many other functions can do this kind of 1 or 0 transformation -- a simple step function will do. The advantage of using a sigmoid is the values can be read off as probabilities.
Steps 1 and 2: Define the Inputs and the Output
How do we find the line that separates the data? Let's begin by defining the inputs.
Can you write Python code for it?

# add a ones column to the inputs - this makes the matrix multiplication work out easier
data.insert(0, 'Ones', 1)

# set X (training data) and y (target variable)
cols = data.shape[1]
X = data.iloc[:,0:cols-1]
y = data.iloc[:,cols-1:cols]

# convert to matrices
X = np.matrix(X.values)
y = np.matrix(y.values)

X.shape

y.shape

X[0:5,:]


I am working on the file "Chapter-5-Logistic Regression.ipynb".
The first task is:
The cost function is designed to penalize misclassification. For each set of parameter values, there will be a cost value over the entire data set. The optimal parameter values are the ones that minimize this cost function. Remember, to think like a machine is to take a problem like this, turn it into a giant optimization problem, and then devise and implement a technique for finding the optimal paramter values (afterall, you can't try every possible combination of $\theta$ values, because that will take longer than the time for the heat death of the universe!).
Of course our technique for finding the right paramater values is going to be gradient descent. But before we get to that, let's implement the cost function.
Can you write Python code for it?

def logisticCost(X, y, theta):
    # Inputs must be matrices
    # X = m x n (including bias feature of 1 for x0)
    # y = m x 1
    # theta = n x 1

    # Number of data points in the dataset
    m = len(X)
    cost_first_term = np.multiply(-y,np.log(sigmoid(X * theta)))
    cost_second_term = np.multiply(1-y, np.log(1 - sigmoid(X * theta)))
    non_normal_term = np.sum(np.subtract(cost_first_term, cost_second_term))
    cost = non_normal_term / m

    return cost

# try it out for theta = zeros
logisticCost(X,y,theta)

# try it out for theta = 0.03
logisticCost(X, y, np.matrix([[0.03],[0.03],[0.03]]))


I am working on the file "Chapter-6-Non-Linear-Logistic-Regression.ipynb".
The first task is:
Load the Data
Can you write Python code for it?

import os
path = os.getcwd() + '/Data/ex2data2.txt'
data2 = pd.read_csv(path, header=None, names=['Test 1', 'Test 2', 'Accepted'])
data2.head()

data2.shape


I am working on the file "Chapter-6-Non-Linear-Logistic-Regression.ipynb".
The first task is:
Visualize the Training Data
Can you write Python code for it?

# Bring the training inputs and training outputs together
#X_train.shape
#y_train.shape
training_data = np.c_[X_train, y_train]

# Get all inputs that have positive outputs -- i.e., Accepted
pos_train_inputs = np.array([tr_d for tr_d in training_data if tr_d[2] == 1])[:,0:2]
#pos_train_inputs

# Get all the inputs that have negative outputs -- i.e., Rejected
neg_train_inputs = np.array([tr_d for tr_d in training_data if tr_d[2] == 0])[:,0:2]
#neg_train_inputs

fig, ax = plt.subplots(figsize=(8,5))
ax.scatter(pos_train_inputs[:,0], pos_train_inputs[:,1], s=30, c='b', marker='s', label='Accepted')
ax.scatter(neg_train_inputs[:,0], neg_train_inputs[:,1], s=30, c='r', marker='x', label='Rejected')
ax.legend()
ax.set_xlabel('Test 1 Score')
ax.set_ylabel('Test 2 Score')
ax.set_title('Training Dataset')


I am working on the file "Chapter-6-Non-Linear-Logistic-Regression.ipynb".
The first task is:
We've increased the number of features from 2 to 28 by using a 6th degree polynomial. Let's use logistic regression to solve the problem.
Steps 4 - 7: Set Up and Implement the Model
We've made the inputs very complicated -- so before we implement a logistic regression algorithm that depends on some form of gradient descent to find its solution, let's normalize our inputs so that the algorithm can operate efficiently.
Can you write Python code for it?

# first row of the training inputs
X_train_poly6[0]

# Implement logistic regression using Scikit Learn
# Large C means low regularization lambda; Small C means high regularization lambda
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
from sklearn.model_selection import cross_val_score

lr = LogisticRegression(solver='newton-cg', C=100)
lr.fit(X_train_poly6, y_train)

# Classification accuracy on the training data
lr.score(X_train_poly6, y_train)

# How did this classifier do on the test data?
predicted = lr.predict(X_test_poly6)
print metrics.accuracy_score(y_test, predicted)
print metrics.roc_auc_score(y_test, predicted)

lr.intercept_

# The optimal values of theta
lr.coef_

# Sanity checks
m = X_train_poly6.shape[0]
print 'Number of samples:', m
print 'lr.coef_ shape:', lr.coef_.shape
# Multiply the inputs by the coefficients to get the optimal cost
print 'X_train_poly6 shape:', X_train_poly6.shape
dotProd = sigmoid(np.matrix(X_train_poly6) * np.matrix(lr.coef_).T)
cost_first_term = np.multiply(-y_train.reshape(94,1), np.log(dotProd))
cost_second_term = np.multiply(np.subtract(1,y_train.reshape(94,1)),np.log(np.subtract(1, dotProd)))
non_reg_cost = np.divide(np.sum(np.subtract(cost_first_term, cost_second_term)), m)
print 'Optimal Non-Regularized Cost:', non_reg_cost

# Regularize the cost
Lambda = 100.
theta_squared = np.square(lr.coef_)
theta_squared[0,0] = 0
reg_theta = np.sum(theta_squared)
reg_term = np.divide(Lambda,np.multiply(2,m))
print 'Regularization Term:', reg_term
reg_cost = np.add(non_reg_cost, reg_term)
print 'Regularized Cost:', reg_cost

# Contour plot of the decision boundary
# Make grid values
xx1, xx2 = np.mgrid[-1:1.5:.02, -1:1.5:.02]

xx1

xx2

# Create the grid
grid = np.c_[xx1.ravel(), xx2.ravel()]
grid.shape

# Get the probabilities for each grid value
# Note: Make sure it's transformed so
probs = lr.predict_proba(poly.fit_transform(grid))
probs.shape

probs_bin = lr.predict(poly.fit_transform(grid))
probs_bin.shape

grid.shape

probs_to_binary =...


I am working on the file "Chapter-7-Neural-Networks.ipynb".
The first task is:
Note: When you see a digit misidentified as itself, what that means is that the classifier was unsure and made more than one guess, including a guess that it was the digit itself. Because the classifier was not sure, we count this as a misclassification.
The digits 0 and 1 are the easiest to recognize while 5 and 8 seem to be confused with a wide range of other digits. 9 is also hard to discern correctly.
Getting Under the Hood
Can you write Python code for it?

[clf.score(X_train, y_train), clf.score(X_test, y_test)]

clf.loss_


I am working on the file "Chapter-7-Neural-Networks.ipynb".
The first task is:
That's close to perfect, but just a small sample of the test dataset. But not bad for just using a pretty basic neural network. How does the classifier do over the entire dataset?
Can you write Python code for it?

# How well does the classifer perform?
print("Training set score: %f" % clf.score(X_train, y_train))
print("Test set score: %f" % clf.score(X_test, y_test))

# Misclassification of the test dataset
# Get all the results in terms of success or failure
results = [(clf.predict(X_test[i].reshape(1,-1)) == y_test[i]).all() for i in range(len(X_test))]
# Get the index numbers of all the failures
idx_failures = [i for i, x in enumerate(results) if x == False]

# At the same time, get the index numbers of all the successful predictions
idx_successes = [i for i, x in enumerate(results) if x == True]

# Quick check on test images that are not classified correctly
idx_failures[0:5]

# Let's check on the misclassification for the first few failures
for i in idx_failures[0:3]:
    print clf.predict(X_test[i].reshape(1,-1))
    plt.figure()
    mnist.display(X_test[i])