I am working on the file "Lesson_05_python_idioms_and_style.ipynb".
The first task is:
Observe the differences. In the first case, we always need to now what exactly we should check before we perform our operation (so we "ask for permission"). What if we don't know?
Imagine - you open a file, but first you "ask for permission" - you check if the file exists. It exists, you open it, but then an exception is raised, like "You do not have sufficient rights to read this file". Our program fails.
If you first perform an operation, and "ask for forgiveness", then you have a much greater control, and you communicate something with your code - that it should work most of the time, except some times it does not.
If you always "ask for permission", then you are wasting computation.
Some recommendations for the EAFP rule:
EAFP (Easier to Ask for Forgiveness than Permission)
IO operations (Hard drive and Networking)
Actions that will almost always be successful
Database operations (when dealing with transactions and can rollback)
Fast prototyping in a throw away environment
LBYL (Look Before You Leap):
Irrevocable actions, or anything that may have a side effect
Operation that may fail more times than succeed
When an exception that needs special attention could be easily caught beforehand
Idiomatic Python
First, I recommend watching this video:
https://www.youtube.com/watch?v=OSGv2VnC0go
"Transforming Code into Beautiful, Idiomatic Python"
Most of these examples come from that video.
Pythonic, idiomatic Python
It just means, that the code uses Python idioms, the Python features that make this programming language unique. The code will be more readable, expressive, will be able to do more things than you thought it can. Let's go through some examples.
Change many "if" into a dictionary
To avoid the infamous "if" ladders, it is much much easier to change this into a dictionary.
First examples shows how to change the argument of "print" function with this approach. Try to count how many less "checks" are performed by the system.
Can you write Python code for it?

# This is bad:
s = ["a",1,(2,2), 20.00]

for elem in s:
    if isinstance(elem, str):
        print("This is string")
    elif isinstance(elem, int):
        print("This is an integer")
    elif isinstance(elem, tuple):
        print("This is a tuple")
    else:
        print("This is something else. Details:" + str(type(elem)))

# This is good:
s = ["a", 1, (2,2), 20.00]

helper_dict = {
    str: "This is string",
    int: "This is integer",
    tuple: "This is a tuple"}

for elem in s:
    # Notice "asking for forgiveness" and not "permission"
    try:
        print(helper_dict[type(elem)])
    except Exception as e:
        print("This is something else. Details: " + str(e))

# Another example, but to store FUNCTIONS instead of VARIABLES
from datetime import datetime
helper_dict = {"amount": float, "counter": int, "date": datetime.strptime}

# Types references are also functions that convert variables between types.

some_dict = {"currency": "USD", "amount": "10000", "source": "Poland", "target": "Poland", "counter": "9298", "date": "20171102"}

for key, value in some_dict.items():
    try:
        converted = helper_dict[key](value)
    except Exception:
        converted = str(value)

    print(converted)
    print(type(converted))


I am working on the file "Lesson_05_python_idioms_and_style.ipynb".
The first task is:
Unpacking with wildcard "*"
Can you write Python code for it?

seq = ["a", "b", "c", "d", "e", "d"]
start, *middle, end = seq
print(start)
print(middle)
print(end)


I am working on the file "Lesson_05_python_idioms_and_style.ipynb".
The first task is:
Updating multiple variables at once
Can you write Python code for it?

# Bad fibonacci implementation

def fibonacci(n):
    x = 0
    y = 1
    for i in range(n):
        print(x)
        t = y
        y = x + y
        x = t

fibonacci(8)

# Simpler implementation

def fibonacci(n):
    x, y = 0, 1
    for i in range(n):
        print(x)
        x, y = y, x + y

fibonacci(8)

# Multiple updates at the same time
x, y, z, u = range(0,4)
print(x, y, z, u)

x, y, z, u = x + 1, y + z, u - x, z**2
print(x, y, z, u)


I am working on the file "Lesson_06_iterators_generators.ipynb".
The first task is:
.... and sometimes the condition changes?
Can you write Python code for it?

from random import randint

divisor = randint(1,500)

for i in range(1,500):
    if i%divisor==0:
        print("Got it: " + str(i))
        break


I am working on the file "Lesson_06_iterators_generators.ipynb".
The first task is:
Examples with time
Can you write Python code for it?

from time import sleep
from random import randint

def get_latest_customers():
    rv = []
    for i in range(10):
        sleep(.5)
        rv.append(randint(1,20))
    return rv

customer_id = 18

# We gather lot's of customer ID's and then check if our customer ID=19 is there.

print(customer_id in get_latest_customers())

def get_latest_customers_lazy(n):
    for i in range(n):
        sleep(.5)
        yield(randint(1,20))

iterator = get_latest_customers_lazy(2000000000)

for x in iterator:
    if customer_id is x:
        print("Found him!")
        break
    else:
        print("Still looking..." + str(x))


I am working on the file "Lesson_06_iterators_generators.ipynb".
The first task is:
Generator expressions
Can you write Python code for it?

laziest_shortest_range = (x for x in range(10))

next(laziest_shortest_range)

next(laziest_shortest_range)


I am working on the file "Lesson_06_iterators_generators.ipynb".
The first task is:
It seems we don't need to generate the big list at all. Imagine it took some time, like 0.5 seconds for each element to check. Do we need to have them all before we check?
We need a lazy evaluation, not eager.
Am I done yet? No? Then give me the next one. Just the one.
Iteration protocol
Can you write Python code for it?

# Normal method of using collections - just generate all at once
a = [1,2,3,4,5]

# Let's change the list into an ITERATOR
my_iterator = iter(a)
print(type(my_iterator))
print(my_iterator)

# Let's iterate the iterator
next(my_iterator)

# And some more:
next(my_iterator)
next(my_iterator)
next(my_iterator)
next(my_iterator)

# ... and more:
next(my_iterator)


I am working on the file "Lesson_06_iterators_generators.ipynb".
The first task is:
Meaning of 'yield'
Can you write Python code for it?

def semaphore():
    print("You go")
    yield
    print("No, you go")
    yield
    print("No, really, you go!")
    yield

a = semaphore()

next(a)

next(a)

next(a)


I am working on the file "Lesson_06_iterators_generators.ipynb".
The first task is:
Silly example
Let's say you need to stop your loop when you have a specific condition.
Can you write Python code for it?

for i in range(0,500):
    if i%376==0:
        print("Got it!")
        break


I am working on the file "Lesson_06_iterators_generators.ipynb".
The first task is:
That looks terrible. Is there an easy way?
Generators
Can you write Python code for it?

def laziest_range(N):
    i = 0
    while i<N:
        yield i
        i += 1

a = laziest_range(5)

next(a)

next(a)

for x in laziest_range(10):
    print(x)


I am working on the file "Lesson_06_iterators_generators.ipynb".
The first task is:
Woops! When iterator is done, it always throws a STOP sign.
A better approach is to just iterate over the iterator.
Can you write Python code for it?

some_list = [1,2,3,4,5,5]
some_iter = iter(some_list)

for x in some_iter:
    print(x)


I am working on the file "Lessons/Lesson 1/Lesson One.ipynb".
The first task is:
However, if we then reassign the variable later on, it will remember the latest assignment.
Can you write Python code for it?

name = "Bobby"
name = "Bambi"

print name


I am working on the file "Lessons/Lesson 1/Lesson One.ipynb".
The first task is:
Interfacing with the Raspberry Pi
Now we will see how we can use python to read sensors with the Raspberry pi. If you are unfamiliar with the raspberry pi please read the getting started with raspberry pi lesson here.
Once you have set up the PSoC board with the raspberry pi, follow instructions on how to connect the button with the pi.
Can you write Python code for it?

#import python library with buttonPressed() method
#b = new Button()


I am working on the file "Lessons/Lesson 1/Lesson One.ipynb".
The first task is:
One important concept/ability that programming can do is to remember past values using variables. We can them to remember past values, then call on them later on when we need their values. For example, the example below stores a name, then prints it out.
Can you write Python code for it?


I am working on the file "Lessons/Lesson 1/Lesson One.ipynb".
The first task is:
Remember how before we dealt with numbers such as 1,49,3 in the calculations section. Those are a class of values called integers, and the words, such as "Bambi", "Bobby", "Hi", are values known as strings. Here we will introduce another type of values called booleans. Simply put, booleans are either true or false and are denoted as "True" and "False". We can print them like we did with integers and strings.
Can you write Python code for it?

yes = True
no = False

print yes
print no


I am working on the file "Lessons/Lesson 1/Lesson One.ipynb".
The first task is:
We can also use the print system to print letters and words, such as the example below. In order to print a word, the word has to be surrounded by a pair of quotation marks (" ")
Can you write Python code for it?


I am working on the file "Lessons/Lesson 1/Lesson One.ipynb".
The first task is:
We set the variable "like_candy" to True and then pass it in the if statement. Since like_candy is true, we run the code following the if block.
Now we will see how we can use these tools to write a simple program that can interface with sensors. A button is a digital sensor, and one way we can see it is to call b.buttonPressed(). This is pretty complicated syntax, so don't worry about it for now. Just know that b.buttonPressed() returns a boolean value, corresponding to whether the button is pressed or not. If the button is pressed, b.buttonPressed() is True, if the button is not presesd, it is false.
Do not press the button and run the following code.
Can you write Python code for it?


I am working on the file "Lessons/Lesson 1/Lesson One.ipynb".
The first task is:
Write some code below to display "Hi, my name is (your name here)!".
Can you write Python code for it?

#Write your code here

#Solution
print "Hi, my name is Nom!"


I am working on the file "Lessons/Lesson 2/Lesson 2.ipynb".
The first task is:
Congratulations, you've made your first data logger! We could easily modify this code to take temperatures over an entire day, or a week, or a month, or even a year! Now we have to discuss a concept called sample rate. Simply put, a sample rate is how many samples we take a second. In the example above we had a sample rate of 1, because we took one sample every second. If we had taken 10 samples over 5 seconds, we would have a 0.5 second delay between each sample, and had a sample rate of 2, because we would have taken 2 samples every single second. In general we can see that $sample rate = \frac{total samples}{total time}$. In general a higher sample rate is good because it allows us to see more data points between points. If we have too low of a sample rate we won't be able to see any patterns/how the temperature changes, it'll just be telling us what the temperature is at certain times.
Once we have a list of our values, we can visualize it through python's plotter. The code below is written to view the temperature.
Can you write Python code for it?

import matplotlib.pyplot as plot
plot.plot(temperatures)
plot.show()


I am working on the file "Lessons/Lesson 2/Lesson 2.ipynb".
The first task is:
In the block below, write some code, using a while loop, that will print out every even number, starting from 2 and ending with 20. It should print out
2
4
6
8
10
12
14
16
18
20
Can you write Python code for it?

#WRITE YOUR CODE HERE

#SOlUTION
i = 2
while(i <= 20):
    print i
    i = i+2


I am working on the file "Lessons/Lesson 2/Lesson 2.ipynb".
The first task is:
This is great, because now we can remember values that our sensors returned at specific instances of time. There are many operations we can use to modify a list. One important one that we will use here is the append operation. Essentially what it lets us do is add an element to the end of the list. An example is shown below. Initially one_to_ten is empty, we want to see how we can fill it up. Initially we bind one_to_ten to an empty list, which is denoted by [].
Can you write Python code for it?

one_to_ten = []
one_to_ten.append(1)
print one_to_ten
one_to_ten.append(2)
print one_to_ten
one_to_ten.append(3)
print one_to_ten
one_to_ten.append(4)
print one_to_ten
one_to_ten.append(5)
print one_to_ten
one_to_ten.append(6)
print one_to_ten
one_to_ten.append(7)
print one_to_ten
one_to_ten.append(8)
print one_to_ten
one_to_ten.append(9)
print one_to_ten
one_to_ten.append(10)
print one_to_ten


I am working on the file "Lessons/Lesson 2/Lesson 2.ipynb".
The first task is:
We can see that every time we called one_to_ten.append(x), we appended x to the list. For more operations we can perform on lists, see the python documentation here.
But looking at the code above, this is quite cumbersome. It involves a lot of typing and a lot of code. Programmers are lazy, so they invented a nice little way to let us not write as much code by using loops. Loops are a construct that repeat a section of code a certain number of times, as defined by the programming. One example is shown below. This is the while loop
Can you write Python code for it?

i = 1
while(i <= 10):
    print i
    i = i+1


I am working on the file "Lessons/Lesson 3/Lesson 3.ipynb".
The first task is:
A function can also have multiple parameters, or no parameters at all. Remember, the parameters are the things that go in between the paranthesis. Suppose that we want a function that takes in zero parameters and always returns $1337$, we just leave the parameter section blank.
Can you write Python code for it?


I am working on the file "Lessons/Lesson 3/Lesson 3.ipynb".
The first task is:
Look at some sample outputs below
Can you write Python code for it?

print double(1);
print double(2);
print double(3);


I am working on the file "Lessons/Lesson 3/Lesson 3.ipynb".
The first task is:
Recall the example of you and your friend walking home from school. Now we will write a function that will calculate the time difference of arrival for an abstract sense. If your friend lives a distance $a$ miles away from you and you both decide to hang out at a location $x$ during the weekend, what will be the time difference of arrival of you and your friend getting home? Both of you walk with the same speed $s$. Answer the questions below to help find the TDOA.

How far do you have to walk home?
How long does it take for you to walk home?
How far does your friend have to walk home?
How long will it take him to walk home?
What is the difference in these two times?

Write a python function below that takes in $a$ (friends house), $x$ (hangout location) and $s$ (the speed you two walk) and returns the TDOA. The function declaration should be
python
def TDOA(a, x, s):
Can you write Python code for it?

#Write your function here

#Solution
def TDOA(a, x, s):
    return (a-2.0*x)/s;


I am working on the file "Lessons/Lesson 3/Lesson 3.ipynb".
The first task is:
What do you think will happen if we run
python
print leet();
Try it below!
Can you write Python code for it?

#Try what "print leet()" does.
#Enter your code here:


I am working on the file "Lessons/Lesson 3/Lesson 3.ipynb".
The first task is:
Whenever we call print_double, we ask the function to print $2x$ so we don't have to.
Can you write Python code for it?


I am working on the file "Lessons/Lesson 5/Lesson 5.ipynb".
The first task is:
Finally, we want to calculate the average of all the values in speeds. Do this by calculating the sum of all the values in speeds, then dividing it by ten (the number of elements in speeds). Set $s$ to be this value.
Can you write Python code for it?

#WRITE YOUR CODE HERE
s = 0
for newS in speeds:
    s = s + newS
s = s/10
print s


I am working on the file "Lessons/Lesson 5/Lesson 5.ipynb".
The first task is:
Now we can use this to make our mouse move to any pixel on the screen. We will first need to write a method that clicks to a position $(x,y)$. The way coordinates are defined here are not quite the same as cartesian coordinates. The upper left corner is defined as $(0,0)$. Increasing the first coordinate still makes the mouse move right, but increasing the second coordinate will make the mouse move down.
Run the code below to see what I mean
Can you write Python code for it?

mouse = PyMouse() #Making a new mouse
mouse.move(0,0) #Making the mouse move to the very top left corner
time.sleep(1)
mouse.move(10, 0) #Moving the mouse right
time.sleep(1)
mouse.move(20, 0)
time.sleep(1)
mouse.move(30, 0)
time.sleep(1)
mouse.move(30, 10)
time.sleep(1)
mouse.move(30, 20)
time.sleep(1)
mouse.move(30, 30)


I am working on the file "Lessons/Lesson 5/Lesson 5.ipynb".
The first task is:
Now we know $a$ and $s$! So whenever we get a $t$ by tapping the board, we should be able to calculate the position of where we've tapped!
We've written the majority of the meat for our code. However, displaying a texual format of where we've tapped isn't so satisfying. We want something visual and exciting! So instead of just printing out where we tapped, we move our mouse to the location. The effect of this is that when we project our screen onto the board, we can set the sensors to be at the left most edge and right most edge. Whenever we tap on the line in between them, our mouse will travel to where we've tapped, creating a tocuh screen!
Writing a module that will move our mouse from scratch is rather difficult, however, someone else has done this already! The library that moves the mouse is called PyMouse. We can import this library by calling
python
    from pymouse import PyMouse
Do this in the code block below.
Can you write Python code for it?

#from pymouse import PyMouse
import time


I am working on the file "Lessons/Lesson 5/Lesson 5.ipynb".
The first task is:
Take a brief moment to review what it does. It takes in three parameters, $a$ (position of friend's house), $t$ (time distance of arrival), and $s$ (the speed you and your friend travel at). Now instead of using the metaphor of you and your friend racing home, how can this be used to model the piTap?
The setup of piTap is that there are two sensors, connected by a horizontal line. We want to be able to tap anywhere in between the line and predict where we've tapped. This is similar to you and your friend running home, except now it is two waves that are racing towards the sensors. The sensors are analogous to the homes, the position of the tap is is analogous to the location of the hangout spot, and the circular wave front is equivalent to you and your friend running left and right.
We know what the value of $a$ is, just the distance beween the two sensors, which we can measure with a meter stick or ruler. We know what $t$ is since we can ask the raspberry Pi what the time difference between the sensors is. However, we do not yet know what $s$ is, and we will need some way of calculating it. From the previous lesson, we asked "What is the TDOA if you decide to hang out at a location LEFT of your house?". The answer to this is that it is simply the the time it takes your friend to run to his house from your house. Thus if we tap the board LEFT of the leftmost sensor, the time distance of arrival will be the time it takes the wave to propagate from one sensor to another. And since we know that the distance between the sensors, we can use the formula that $s = \frac{\Delta p}{t} = \frac{a}{t}$.
So we could calculate $s$ by tapping the board left of the left-most sensor, retrieve the TDOA and take the quotient of the length and the TDOA. However, since we've only conducted the experiment once, this result could be rather inaccurate. In order to get a more accurate value of $s$, we will want to conduct this experiment multiple times and then take an average of all the experiment results.
Below, write a program...
Can you write Python code for it?

#WRITE YOUR CODE HERE
#ANSWER
a = 1.0
speeds = []
i = 10
while(i > 0):
    i = i-1
    thisTime = 1 #piTap.getTDOA()
    speeds.append(a/thisTime)


I am working on the file "Lessons/Lesson 5/Lesson 5.ipynb".
The first task is:
We will now need to find the resolution of our monitor. The resolution tells us the dimensions of our display, pixels wide by pixels tall.
This is done for you below. Run the code below.
Can you write Python code for it?

import Tkinter as tk
root = tk.Tk()

widthPixels = root.winfo_screenwidth()
heightPixels = root.winfo_screenheight()

print (widthPixels, heightPixels)


I am working on the file "Lessons/Lesson 6/Lesson 6.ipynb".
The first task is:
Distance between two points
The pythagorean theorem lets us easily find the distance between any two cartesian points. This is because if we visualize two points on a cartesian grid, the distance between them is just the length of the hypotenuse created by connecting the two points! See the figure below:
<img src="distance.png" alt="Drawing" style="width: 500px;"/>
Since this is a right triangle, we know that $distance(P_1, P_2)^2 = |x_2-x_1|^2 + |y_2-y_1|^2 \rightarrow
distance(P_1, P_2) = \sqrt{|x_2-x_1|^2 + |y_2-y_1|^2}$
Write a function below called distance that takes in four arguments, x1, y1, x2, y2, corresponding to the points $p_1 = (x_1, y_1)$ and $p_2 = (x_2, y_2)$ and returns the distance between $p_1$ and $p_2$.
Can you write Python code for it?

#Write your function here

#Solution

def distance(x1,y1,x2,y2):
    return math.sqrt((x2-x1)*(x2-x1) + (y2-y1)*(y2-y1))


I am working on the file "Lessons/Lesson 6/Lesson 6.ipynb".
The first task is:
Finding two-dimensional TDOA
Now that we can find the distance between cartesian points, we can now find the TDOA of four sensors on a plane instead of the TDOA along just a single line with two sensors!
<img src="four_sensor_d.png" alt="Drawing" style="width: 750px;"/>
When we set up the sensors, they will be on the four corners of a rectangle with width $a$ and height $b$. Thus the positions of the sensors are at $(0,0)$, $(a,0)$, $(b,0)$, and $(a,b)$. Whenever we tap within this rectangular region, we will be creating a circular wave with a velocity $v$, that travels to the four corners. Since we don't know exactly where we tapped, have it be at a variable point $(x,y)$.
Write a function that takes in four arguments, $a$ (width of the rectangle), $b$ (height of the triangle), and $x,y$, the cartesian coordinates of the wave source and return an array/list of length four, where the first element is the distance from the wave source to the first sensor, the second element is the distance from the wave source to the second sensor, etc. Call it distance_to_sensor.
Hint: you may find it useful to use the distance function you defined above.
Can you write Python code for it?

#Write your function below

#Solution
def distance_to_sensor(a,b,x,y):
    distance_one = distance(0,0,x,y)
    distance_two = distance(a,0,x,y)
    distance_three = distance(0,b,x,y)
    distance_four = distance(a,b,x,y)

    return [distance_one, distance_two, distance_three, distance_four]


I am working on the file "Lessons/Lesson 6/Lesson 6.ipynb".
The first task is:
Now we have successfully written a method that can return the theoretical TDOA to our sensors of a wave source! However, what we would ideally like is a way that can identify what the position $(x,y)$ is given the time distance of arrivals. The implementation for this is rather difficut, so we will not go into it. However, it is the same method that is used by seismologists to detect an earthquake, and what a GPS might use to determine your location!
Some things to think about though. If we know which sensor has the least TDOA is (remember, some of them can be negative since we are subtracting the time it takes the wave to get to sensor one. If the time difference is negative, it simply means that the wave reached that sensor before it reached sensor one), we can figure out which quadrant (or identify an area) that the wave source must be from. Discuss how we can do this.
We've written an implementation that takes in the TDOA. You can import this and use it to create a 2D touch screen!
Extension Material
In this section we will talk about a simple way of determining where the source of a disturbance is given the values of the TDOA.
For simplicity, imagine that we place the sensors on a square of sidelength $a$. One way we could hypothetically find the location is to test every single possible point within the square, and look for the point that returns the correct TDOA values. However, this is unfeasible because there are an infinite number of points within the grid, and it would be impossible to check every single point. But, instead of looking at every single point, we can look a finite number of points and look for the one whose TDOA values are the closest to that of the ones we are given.
<img src="grid.png" alt="Drawing" style="width: 500px;"/>
In the picture above, we are dividing each side into 8 points, and checking a total of $8^2 = 64$ points for the one with the best TDOA values. However, to make it more accurate, we can easily divide each side into $100$ points, and check a total of $10000$...
Can you write Python code for it?

#definition of cost function

def err(TDOA1, TDOA2):
    ans = 0
    for i in range(1,4):
        ans += (TDOA1[i] - TDOA2[i])*(TDOA1[i] - TDOA2[i])
    return ans

#Write your method below

#Solution

def find_point(a, time_differences, v):
    currentX = -1
    currentY = -1
    min_error = 100000000

    x = 0.0
    while(x <= a):
        y = 0.0
        while(y <= a):
            error = err(time_differences, TDOA(a,a,x,y,v))
            if(error < min_error):
                min_error = error
                currentX = x
                currentY = y
            y = y+a/100.0
        x = x+a/100.0

    return (currentX,currentY)

print find_point(1, [0, 0, .618033, .618033], 1)
print TDOA(1,1, .5, 0, 1)


I am working on the file "Lessons/Lesson 6/Lesson 6.ipynb".
The first task is:
Now write a function that takes in the four parameters above, but an additional one called $v$ which is the velocity of the wave source. Have this function return an array/list of length four with the time it takes for the wave to reach of the sensors. Call it time_to_sensor.
Can you write Python code for it?

#Write your function below

#Soluton

def time_to_sensor(a,b,x,y,v):
    distances = distance_to_sensor(a,b,x,y)
    time_one = distances[0]/v;
    time_two = distances[1]/v;
    time_three = distances[2]/v;
    time_four = distances[3]/v;

    #return map(lambda x: x/v, distances)

    return [time_one, time_two, time_three, time_four]


I am working on the file "Lessons/Lesson 6/Lesson 6.ipynb".
The first task is:
We can categorize triangles into three categories based on the properties of their angles.

Acute Triangle: All of the triangle's angles are less than $90^\circ$

For example if a triangle had the angles of $15^\circ, 85^\circ, 80^\circ$, it would be acute.

Obtuse Triangle: One of the triangle's angles is larger than $90^\circ$

For example if a triangle had the angles of $15^\circ, 15^\circ, 150^\circ$, it would be obtuse.

Right Triangle: One of the triangle's angles is exactly $90^\circ$. It is called a right triangle because $90^\circ$ is called a right angle and is the angle you see on the corner of books, doors, and squares.

The pythagorean theorem deals with (and only applies) to right triangles. If we notice from the picture above, the right angle is opposite to the longest side, we call this the hypotenuse. We call the other two shorter sides the legs. We denote the legs with $A$, and $B$, and the hypotenuse with $C$, shown in the picture below.

The pythagorean statements simply states that $A^2 + B^2 = C^2$. Although this equation may look really simple, it is quite powerful. Conversly, if a triangle has three sides $A, B, C$ such that $A^2 + B^2 = C^2$, then it must be a right triangle.
The theorem is quite useful because if we know any the length of any two sides of a right triangle, we can figure out the length of the third.
For example, if we know the lengths of both the legs of a right triangle are $3$ and $4$, then the pythagorean theorem tells us that $3^2 + 4^2 = C^2 \rightarrow C = \sqrt{3^2+4^2} = \sqrt{9+16} = \sqrt{25} = 5$.
If we know the length of the hypotenuse and the length of one leg, we can figure out the length of the remaining leg. For example, if we know that a right triangle has a hypotenuse with length 13, and a...
Can you write Python code for it?

#Write your functions here
import math
#Solutions

def find_hypotenuse(a, b):
    return math.sqrt(a*a+b*b)

def find_leg(a,b):
    if(a > b):
        return math.sqrt(a*a-b*b)
    else:
        return math.sqrt(b*b-a*a)


I am working on the file "Lessons/Lesson_2/L1_Starter_Code.ipynb".
The first task is:
Comparing the Two Student Groups
Can you write Python code for it?

######################################
#                 12                 #
######################################

## Compute some metrics you're interested in and see how they differ for
## students who pass the subway project vs. students who don't. A good
## starting point would be the metrics we looked at earlier (minutes spent
## in the classroom, lessons completed, and days visited).


I am working on the file "Lessons/Lesson_2/L1_Starter_Code.ipynb".
The first task is:
Debugging Data Analysis Code
Can you write Python code for it?

#####################################
#                 8                 #
#####################################

## Go through a similar process as before to see if there is a problem.
## Locate at least one surprising piece of data, output it, and take a look at it.


I am working on the file "Lessons/Lesson_2/L1_Starter_Code.ipynb".
The first task is:
Getting Data from First Week
Can you write Python code for it?

# Takes a student's join date and the date of a specific engagement record,
# and returns True if that engagement record happened within one week
# of the student joining.
def within_one_week(join_date, engagement_date):
    time_delta = engagement_date - join_date
    return time_delta.days < 7

#####################################
#                 7                 #
#####################################

## Create a list of rows from the engagement table including only rows where
## the student is one of the paid students you just found, and the date is within
## one week of the student's join date.

paid_engagement_in_first_week = list()
for element in non_udacity_engagement:
    account = element['account_key']
    if account in paid_students:
        join_date = paid_students[account]
        engagement_record_date = element['utc_date']
        if within_one_week(join_date, engagement_record_date):
            paid_engagement_in_first_week.append(element)
len(paid_engagement_in_first_week)


I am working on the file "Lessons/Lesson_2/L1_Starter_Code.ipynb".
The first task is:
Improving Plots and Sharing Findings
Can you write Python code for it?

######################################
#                 14                 #
######################################

## Make a more polished version of at least one of your visualizations
## from earlier. Try importing the seaborn library to make the visualization
## look better, adding axis labels and a title, and changing one or more
## arguments to the hist() function.


I am working on the file "Lessons/Lesson_2/L1_Starter_Code.ipynb".
The first task is:
Lessons Completed in First Week
Can you write Python code for it?

#####################################
#                 9                 #
#####################################

## Adapt the code above to find the mean, standard deviation, minimum, and maximum for
## the number of lessons completed by each student during the first week. Try creating
## one or more functions to re-use the code above.


I am working on the file "Lessons/Lesson_2/L1_Starter_Code.ipynb".
The first task is:
Missing Engagement Records
Can you write Python code for it?

#####################################
#                 4                 #
#####################################

## Find any one student enrollments where the student is missing from the daily engagement table.
## Output that enrollment.
for enrollment in enrollments:
    student = enrollment['account_key']
    if student not in unique_engagement_students:
        print enrollment
        break
# We have canceled users!!


I am working on the file "Lessons/Lesson_2/L1_Starter_Code.ipynb".
The first task is:
Note when running the above cells that we are actively changing the contents of our data variables. If you try to run these cells multiple times in the same session, an error will occur.
Investigating the Data
Can you write Python code for it?

#####################################
#                 2                 #
#####################################

def get_unique(data, key_s):
    unique_students = set()
    for data_point in data:
        unique_students.add(data_point[key_s])
    return unique_students

## Find the total number of rows and the number of unique students (account keys)
## in each table.
len(enrollments)
unique_enrolled_students = get_unique(enrollments,'account_key')
len(unique_enrolled_students)

len(daily_engagement)
unique_engagement_students = get_unique(daily_engagement,'acct')
len(unique_engagement_students)

len(project_submissions)
unique_project_submitters = get_unique(project_submissions,'account_key')
len(unique_project_submitters)


I am working on the file "Lessons/Lesson_2/L1_Starter_Code.ipynb".
The first task is:
Problems in the Data
Can you write Python code for it?

#####################################
#                 3                 #
#####################################

## Rename the "acct" column in the daily_engagement table to "account_key".
for elem in daily_engagement:
    elem['account_key'] = elem['acct']
    del elem['acct']

len(get_unique(daily_engagement,'account_key'))
daily_engagement[0]


I am working on the file "Lessons/Lesson_2/L1_Starter_Code.ipynb".
The first task is:
Splitting out Passing Students
Can you write Python code for it?

######################################
#                 11                 #
######################################

## Create two lists of engagement data for paid students in the first week.
## The first list should contain data for students who eventually pass the
## subway project, and the second list should contain data for students
## who do not.

subway_project_lesson_keys = ['746169184', '3176718735']

passing_engagement =
non_passing_engagement =


I am working on the file "Lessons/PythonIntro.ipynb".
The first task is:
Another neat arithmetic trick is doing "operate-and-assign," which will likely become more important for loops.
Can you write Python code for it?


I am working on the file "Lessons/PythonIntro.ipynb".
The first task is:
Basic Cheat Sheet
Variable Types in Python

Numbers
int (integers, limited to 64bit representation), i.e. 10
float (floating point real values), i.e 6.2
long (long integers, limited only to available memory), i.e 10L
complex (complex numbers), i.e. 0.5j
String, i.e. "hello, world!"
List, i.e. [1,'star', 3, 'planet', 7]
Tuple, i.e. (1, 'star', 3, 'planet',7)
think of this as a "read-only" list
Dictionary, i.e. {'star': 'sun', 'planet': 'earth'}
think of dictionaries as "key-value" pairs

Arithmetic in Python

+ : addition
-: subtraction
/: division
*: multiplication
%: modulus (remainder)
**: exponentiation

Comparisons and Boolean Operators in Python

==, !=: equal to, not equal to
<, <=: less than, less than or equal to
>, >=: greater than, greater than or equal to
or, i.e A or B: true if either A or B or both are true
and, i.e. A and B: true only if both are true
not, i.e. not A: true only if A is false

Basic Data Types in Python
Can you write Python code for it?

# Beginning a line with "#" is a comment that is ignored by python
myint = 8 # assigns the integer value to the variable myint
print myint # shows the value assigned to myint on the screen
print type(myint) # shows the variable type of myint, note that we did not have to specify this! python got it.


I am working on the file "Lessons/PythonIntro.ipynb".
The first task is:
Comparison Operators and Boolean Variables
Can you write Python code for it?

mass_of_sun = 1.989*10**30 # mass of sun in kg
mass_of_earth = 5.972*10**24 # mass of earth in kg
mass_of_earth < mass_of_sun # here we are making a comparison that is evaluated to true

mass_of_sun == mass_of_earth # this comparison operator means "equal to," note that it is NOT the same as "=" which
                            # assigns a value to a variable

mass_of_sun != mass_of_earth # this comparison operator means "not equal to"


I am working on the file "Lessons/PythonIntro.ipynb".
The first task is:
Instead of doing all our calculations like this in Python, let's look at how we can make things easier using variable assignment.
Can you write Python code for it?

c = 3.0*10**5 # speed of light (c) in km/s
diameter_lyr = 120000 # diameter of mw in lyr
s_per_yr = 3.154e+7 # number of seconds in a year
diameter_km = diameter_lyr*c*s_per_yr # diameter of mw in km
print diameter_km


I am working on the file "Lessons/PythonIntro.ipynb".
The first task is:
Let's look at a more complicated way to slice arrays:
Can you write Python code for it?

bins = np.zeros(5) + np.arange(5) # this creates an array of zeros, and then uses another built in function of numpy
                               # called arange to populate each element of this array with different values
bins # notice how arange gives you the "range" of the input value 5, but starts again with zero.

bins = np.zeros(5) + np.arange(6) # why won't this work?

len(bins) # you can always check the size of your array using "len"

bins = np.arange(5) # note that there are multiple ways to create this array, all of which work fine
bins

bincenters = (bins[1:] + bins[:-1])*0.5 # slicing the array in this way gives me the "centers" of my previous values
bincenters

# let's try to see in detail how this slicing works, step-by-step:
print "first bin slice: {0}".format(str(bins[1:]))
print "second bin slice: {0}".format(str(bins[:-1]))
print "bin slices added: {0}".format(str(bins[1:] + bins[:-1]))
print "bin centers: {0}".format(str(bincenters))


I am working on the file "Lessons/PythonIntro.ipynb".
The first task is:
Note that you should be careful about doing comparisons on floating point values since these are stored in a
specific way.
Can you write Python code for it?

0.1 + 0.2 == 0.3 # this equality returns False, why?

print "{0:.20f}".format(0.1 + 0.2) # don't worry about the print statements now; they tell us how many decimals to print
print "{0:.20f}".format(0.3) # clearly these two are not equal! careful with floats.


I am working on the file "Lessons/PythonIntro.ipynb".
The first task is:
Now that we know more about indexing and slicing, we can even access individual elements of matrices!
Can you write Python code for it?

mymatrix # recall we made this 5x3 matrix earlier

mymatrix[0] = 6. # assigns the value 6 to first ROW of the matrix
mymatrix[0,0] = 10. # assigns the vlaue 10 to the first element of the first column in the first row
mymatrix

print mymatrix[:,0] # accesses first column of matrix
print mymatrix[0,:] # accesses first row of matrix


I am working on the file "Lessons/PythonIntro.ipynb".
The first task is:
Now that we've seen the "Boolean" variables True and False in Python let's take a look at logical operators
that can test these Boolean variables.
Can you write Python code for it?

(mass_of_moon < mass_of_earth) and (mass_of_earth < mass_of_sun) # both MUST be true for this to return true

(mass_of_moon > mass_of_earth) or (mass_of_earth < mass_of_sun) # only one must be true for this to return true

(mass_of_moon > mass_of_earth) or not (mass_of_earth > mass_of_sun) # what do you expect this to return?

(mass_of_moon > mass_of_earth) or (mass_of_earth > mass_of_sun) # what do you expect this to return?


I am working on the file "Lessons/PythonIntro.ipynb".
The first task is:
Sidebar: how are arrays different from lists that we have seen above? We have seen this lists can store heterogenous data (data that contains different types). Arrays, on the other hand, should store homogenous data (data of the same type), and are usually used for storing things that you want to perform fast mathematical operations on. Arrays can speed things up a lot in Python as we will see later. Arrays are NOT comma separated like a list, but the elements of
an array can be accessed just like the individual elements of a list can (again, as we will see later).
Can you write Python code for it?

myarray + 2 # we can do mathematical operations on arrays for the WHOLE array at once! very powerful.

myarray = myarray + 10 # note that the above cell did not change the value of the whole array because we didn't
                       # assign that operation to any variable. now we have done this

np.log10(myarray) # this is another built-in function from numpy that allows you to take the log
                  # of the whole array (in base 10)

help(np.zeros) # using this help function is much like "man" in bash. it will tell us more about this function


I am working on the file "Lessons/PythonIntro.ipynb".
The first task is:
You can also make arrays in Python which are not strictly 1-d. For example:
Can you write Python code for it?

mymatrix = np.zeros((5,3)) # python uses row-column notation, so this creates a matrix of five rows and three columns
print mymatrix


I am working on the file "Lessons/Python_Loops_Files_Functions.ipynb".
The first task is:
An important note: above we have passed integer value arguments to our functions. Simple variable types like int, float, and string are passed by VALUE to a function, while compound types like lists, dicts, tuples, etc are passed by REFERENCE. Let's see a concrete example:
Can you write Python code for it?

z = 5
print add_four(z) # we can pass z by reference to our add function
print(z) # but it's value won't be changed globally

def append_element(x):
    return x.append(4)

list = [1,2,3]
append_element(list) # pass the list by reference to our append_element function
print(list) # the values in the list are changed globally


I am working on the file "Lessons/Python_Loops_Files_Functions.ipynb".
The first task is:
Combining Control Flow Loops
Can you write Python code for it?

num_array = range(20)
for i in num_array:
    if i < 1: # for multiple if statements use if, elif, else.
        print("Zero")
    elif (i <= 1) & (x > 0): # you can combine comparison operators!
        print("One")
    elif i == 2:
        print("Two")
    elif (i > 2) & (i <= 3):
        pass # pass does nothing
    elif i > 3:
        break # break quits the loop

# use a for loop to pick out even numbers!
for num in range(2,10):
    if num % 2 == 0: # note the "==" this is for checking whether two values are equal, i.e. it is doing "?="
        print("Even number {}".format(num))
        continue # this tells the loop to continue, although it is not *required* that you use this
    elif num % 2 != 0:
        print("Odd number {}".format(num))
        break # try commenting out the break to see what happens!

# try it for yourself! write a loop that takes the following list and prints only the
# words that are longer than four letters.
# helpful hint: you will need a for loop and an if statement.
solar_system_objects = ['sun', 'moon', 'earth', 'mars', 'pluto', 'neptune', 'venus', 'mercury']


I am working on the file "Lessons/Python_Loops_Files_Functions.ipynb".
The first task is:
File I/O: Reading Files
Since we are already familiar with the built-in module "numpy" in python, let's use some of its functions to read and write data files. There are many other ways to read and write files in python; this is just a little something to get us started.
Can you write Python code for it?

import numpy as np
# numpy has a function called "loadtxt" which will read in text files. you just need to give it the path to the file!
# note that this will actually give you an error. why? because loadtxt doesn't like that there are different
# variable types in different columns, i.e. a mix of strings and floats; it wants to assume everything is float.
# there are a couple of ways around this
stars_table = np.loadtxt('data/BrightStars.dat')

# one option is to simply skip the first column, the star names, since these are the string values that are throwing
# loadtxt off. for this we use the loadtxt option "usecols" where we can feed it a list of column numbers we want to
# use. remember that python is zero indexing, so to have it skip the first column we omit the number zero.
# just like with the bash "sort" command loadtxt sees spaces as separating one column from a new column.
stars_table = np.loadtxt('data/BrightStars.dat', usecols=[1,2,3,4,5,6,7,8,9,10,11])

# the output of this is an array that contains 11 columns and 5016 lines.
type(stars_table), stars_table.shape

# another option is to explicitly tell it all the column types
# we will do this with dictionaries and lists! we make the "dtype" argument of
# loadtxt into a dictionary which has the keys "names" and "formats," each of which
# have tuples that make up their values.
stars_table = np.loadtxt('data/BrightStars.dat', dtype={'names': ('starname', 'ra_hr', 'ra_min', 'ra_s', 'dec_deg',
                                                                  'dec_arcmin', 'deg_arcsec', 'Vmag', 'B-V',
                                                                  'Parallax', 'uRA', 'uDec'),
                                                        'formats': ('|S15', np.float, np.float, np.float, np.float,
                                                                    np.float, np.float, np.float,np.float,np.float,
                                                                    np.float,np.float)})

# the neat thing about doing...


I am working on the file "Lessons/Python_Loops_Files_Functions.ipynb".
The first task is:
File I/O: Writing Files
Now that we have seen how to read in files, let's look at writing output to a different file; this will likely be how you are recording output from programs or simulations that you run. If you write data output to a file you can do data analysis on it later!
Can you write Python code for it?

# the function that we will be using to write output to a file is numpy's "savetxt"
# savetxt takes as arguments the filename (in quotes) and then the data (in array form) that you write to that filename
# the "fmt" argument is optional but specifying it allows us to choose the specific data type
# in this case we are choosing float values with two decimal places. try it without this to see what you get!
np.savetxt('five_brightest_table.dat', five_brightest_mag, fmt='%.2f')

%%bash
cat five_brightest_table.dat

# here we have only written one line, but what if we want to write a multi-column file?
# let's get the parallax values of the five brightest stars. remember that argsort gave us INDICES. let's reuse them.
five_brightest_dist = stars_table['Parallax'][sort_index[:5]]
five_brightest_dist

# if we want to save both these arrays, we need to feed both of them to "savetxt" in the following way.
# when we look at the output of this we'll notice that savetxt defaults to saving arrays as ROWS.
np.savetxt('five_brightest_table.dat', (five_brightest_mag,five_brightest_dist), fmt='%.2f')

%%bash
cat five_brightest_table.dat

# if we want each array to be a column instead we should use numpy's built-in transpose function
# let's look at what transpose does
print(five_brightest_mag, five_brightest_dist)
print np.transpose([five_brightest_mag,five_brightest_dist])

# to get a two-column data table we do the following:
np.savetxt('five_brightest_table.dat', np.transpose([five_brightest_mag, five_brightest_dist]), fmt=['%.2f', '%.2f'])

%%bash
cat five_brightest_table.dat


I am working on the file "Lessons/Python_Loops_Files_Functions.ipynb".
The first task is:
Iteration with For Loops
Can you write Python code for it?

# here we iterate over the items in a list
z = range(5) # make a list of integers using function range
for i in z: # for each value "i" in my array z...
    print(i) # print that value i

# this will take the value i and set it equal to the first value in the list. then will will
# execute the indentation block and set i equal to the NEXT value in the list, and so on.

# we could just as easily do the following, which is equivalent to the above:
for i in range(5):
    print(i)

# we can also combine range() and len() to iterate over the indices of a list or array
solar_system = ['sun', 'moon', 'earth', 'pluto']
for i in range(len(solar_system)):
    print(i, solar_system[i])

# what is this for loop doing?
for i in solar_system:
    print(i, len(i))

# perhaps a more useful example: what if i want to make everything in my list a float?
magnitudes = ['-1.46', '-0.04', '0.12', '0.5', '1.25']
for i in range(len(magnitudes)):
    magnitudes[i] = float(magnitudes[i])
print(magnitudes)


I am working on the file "Lessons/Python_Loops_Files_Functions.ipynb".
The first task is:
Now that I've written this module, we want to run it. In IPython you can simply do "run module.py" and the module will run. In this case, the calculate_distance module should write a file called "distance_parallax.dat," so with the bash
command below I can check to see if a) this file exists and b) it contains the correct contents given my module. The last few lines of the file define a special variable called "name." This block of codes basically means that whatever comes after the if statement will ONLY run if this module is called directly from the command line (i.e. it is RUN and not simply imported). If you were to just do
python
    import calculate_distance
the block of code would not execute. Instead you could do something like
python
    import calculate_distance
    calculate_distance.distance_in_lyr(argument)
Can you write Python code for it?

%run calculate_distance.py

%%bash
cat distance_parallax.dat


I am working on the file "Lessons/Python_Loops_Files_Functions.ipynb".
The first task is:
There is one wrinkle to using things like np.genfromtxt, np.savetxt, and np.loadtxt: these functions have a hard time dealing with mixed variable types. As you saw, when we read in the files we had to tell it the kind of data types we wanted for each column. It is likewise not simple to write columns of different data types out to a file using these functions. For this reason we'll import a new module astropy and the submodules as functions associated with it.
Can you write Python code for it?

from astropy.io import ascii
# we use the read function with the path to the file name. notice how we don't define data types
stars_table_astropy = ascii.read('data/BrightStars.dat')
print(stars_table_astropy)

print(stars_table_astropy['col1']) # we don't have names for our columns so astropy's read just names them in this way
print(type(stars_table_astropy['col1'][0])) # it knows that elements of the first column are strings!

# now let's see how easy it is to write a table using this module
ascii.write([stars_table_astropy['col1'], stars_table_astropy['col9']],'five_brightest_table_astropy.dat')

%%bash
cat five_brightest_table_astropy.dat | head -5

# that is a rather simple example of how powerful ascii.write() can be. let's see what else it can do
# here i can pass it a LIST of column names, and a DICTIONARY of data types for those columns
ascii.write([stars_table_astropy['col1'], stars_table_astropy['col9']], 'five_brightest_table_astropy.dat',
            names=['StarName', 'Vmag'], formats={'StarName': '%s', 'Vmag': '%.2f'})

%%bash
cat five_brightest_table_astropy.dat | head -5


I am working on the file "Lessons/Python_Plotting.ipynb".
The first task is:
SIDE NOTE: If you are running things in the IPython environment or from a script you would want to do something like the following to get your plots to show up in a new window:
python
plt.scatter(galaxy_table['col1'], galaxy_table['col2'])
plt.show()
In an IPython Notebook, you will see the plot outputs whether or not you call plt.show() because we've used the %matplotlib inline magic function.
Let's break down these basic examples:
- We are running functions called "plot" or "scatter" that take specific arguments.
- The most basic arguments that these functions take are in the form of (x,y) values for the plot, and we get these from a data table.
- We can use more specific arugments like 'o' to customize things like the plot symbol (marker) that we are using.
With plt.scatter() you can change things like point color, point size, point edge color and point type. The argument syntax for adding these options are as follows:

color = 'colorname'; could be 'b' for blue, 'k' for black, 'r' for red
s = number; changes marker size
markeredgecolor = None or 'colorname'
marker = 'symbolname', i.e. 's' for square, 'o' for circle, '+' for cross, 'x' for x, '*' for star, '^' for triangle, etc.

Let's do an example:
Can you write Python code for it?

plt.scatter(galaxy_table['col1'], galaxy_table['col2'],
            color='blue', s=1, edgecolor='None', marker='o')
plt.show()

# here would be the equivalent statement using plt.plot(), note that the syntax is a little different
plt.plot(galaxy_table['col1'], galaxy_table['col2'], 'o',
         color='blue', markersize=1, markeredgecolor='None')
plt.show()


I am working on the file "Lessons/Python_Plotting.ipynb".
The first task is:
Scatter Plots
Can you write Python code for it?

# first let's read in some data to use for plotting
galaxy_table = ascii.read('data/mygalaxy.dat')
galaxy_table[:5]

# simple scatter plot
plt.scatter(galaxy_table['col1'], galaxy_table['col2'])
plt.show()


I am working on the file "Lessons/Python_Plotting.ipynb".
The first task is:
Subplots
Subplots are a way put multiple plots in what amounts to the same figure; think of subplots like an array of plots! The following picture is helpful for understanding how matplotlib places subplots based on row, column, and figure number:
<img src="images/subplot-grid.png">
Can you write Python code for it?

# make two side by side plots

x1 = np.linspace(0.0, 5.0)
x2 = np.linspace(0.0, 2.0)

y1 = np.cos(2 * np.pi * x1) * np.exp(-x1)
y2 = np.cos(2 * np.pi * x2)

plt.figure(figsize=[15,3])
plt.subplot(1,2,1) # 1 row, 2 columns, 1st figure
plt.plot(x1,y1)
plt.xlabel('Xlabel')
plt.ylabel('Ylabel')

plt.subplot(1,2,2) # 1 row, 2 columsn, 2nd figure
plt.plot(x2,y2)
plt.xlabel('Xlabel')
plt.ylabel('Ylabel')
plt.show()

# stack two plots on top of one another

plt.subplot(2,1,1) # 1 row, 2 columns, 1st figure
plt.plot(x1,y1)
plt.xlabel('Xlabel')
plt.ylabel('Ylabel')

plt.subplot(2,1,2) # 1 row, 2 columsn, 2nd figure
plt.plot(x2,y2)
plt.xlabel('Xlabel')
plt.ylabel('Ylabel')
plt.show()


I am working on the file "Lessons/Python_Plotting.ipynb".
The first task is:
The axis labels are easy to read, but the numbers and tick marks on the axis are pretty small. We can tweak lots of little things about how the tick marks look, how they are spaced, and if we want to have a grid to guide the reader's eyes. I will give just a couple of examples here:
Can you write Python code for it?

plt.scatter(galaxy_table['col1'], galaxy_table['col2'],
            color='blue', s=1, edgecolor='None', marker='o')

# Labels
plt.xlabel('Galactic Longitude (degrees)',
           fontweight='bold', size=16)
plt.ylabel('Galactic Latitude (degrees)',
           fontweight='bold', size=16)

# Set limits
plt.xlim([-180,180])
plt.ylim([-90,90])

# Choose axis ticks
plt.xticks(range(-180,210,60), fontsize=16, fontweight='bold') # change tick spacing, font size and bold
plt.yticks(range(-90,120,30), fontsize=16, fontweight='bold')

# turn on minor tick marks
plt.minorticks_on()

plt.grid() # turn on a background grip to guide the eye
plt.show()


I am working on the file "Lessons/Python_Plotting.ipynb".
The first task is:
The last thing I'll mention here is how to put text on plots. This too is simple as long as you specify (x,y) coordinates for the text.
Can you write Python code for it?

plt.figure(figsize=(10,4)) # change figure size
plt.scatter(galaxy_table['col1'], galaxy_table['col2'],
            color='blue', s=1, edgecolor='None', marker='o')

# the next three lines put text on the figure at the specified coordinates
plt.text(-90, -50, 'LMC', fontsize=20)
plt.text(-60, -60, 'SMC', fontsize=20)
plt.text(0, -30, 'MW Bulge', fontsize=20)

plt.xlabel('Galactic Longitude (degrees)',
           fontweight='bold', size=16)
plt.ylabel('Galactic Latitude (degrees)',
           fontweight='bold', size=16)

plt.xlim([-180,180])
plt.ylim([-90,90])

plt.xticks(range(-180,210,60), fontsize=16, fontweight='bold') # change tick spacing, font size and bold
plt.yticks(range(-90,120,30), fontsize=16, fontweight='bold')
plt.minorticks_on() # turn on minor tick marks
plt.grid() # turn on a background grip to guide the eye
plt.show()


I am working on the file "Lessons/Python_Plotting.ipynb".
The first task is:
We can also change things like the axis limits with plt.xlim() and plt.ylim(). For these we just want to feed it a range of values for each axis:
Can you write Python code for it?

plt.scatter(galaxy_table['col1'], galaxy_table['col2'],
            color='blue', s=1, edgecolor='None', marker='o')

plt.xlabel('Galactic Longitude (degrees)',
           fontweight='bold', size=16)
plt.ylabel('Galactic Latitude (degrees)',
           fontweight='bold', size=16)

plt.xlim([-180,180])
plt.ylim([-90,90])
plt.show()


I am working on the file "Lessons/Python_Plotting.ipynb".
The first task is:
You can do fancier things with subplots like have different plots share the same axis, put smaller plots as insets to larger plots, etc. Again, take a look at things like the matplotlib library for examples of different plots.
Plotting Exoplanets
Let's try to make some plots with a new dataset. The file that we'll use is taken from exoplanets.eu.
Can you write Python code for it?

# don't worry about this way to read in files right now
import pandas as pd
exoplanets = pd.read_csv('data/exoplanet.eu_catalog_1022.csv')

# get rid of some rows with missing values to be safe
exoplanets = exoplanets[np.isfinite(exoplanets['orbital_period'])]

# let's see what the data table looks like
exoplanets.head()

# plot distance from host star versus mass (in jupiter masses) for each exoplanet
plt.loglog(exoplanets['semi_major_axis'], exoplanets['mass'],'.')
plt.annotate("Earth", xy=(1,1/317.), size=12)
plt.annotate("Jupiter", xy=(5,1), size=12)
plt.xlabel('Semi-Major Axis [AU]',size=20)
plt.ylabel('Mass [M$_{Jup}$]', size=20)

# let's try to find out if the blobs above separate out by detection type
import seaborn as sns; sns.set()
transits = exoplanets[exoplanets['detection_type'] == 'Primary Transit']
radial_vel = exoplanets[exoplanets['detection_type'] == 'Radial Velocity']
imaging = exoplanets[exoplanets['detection_type'] == 'Imaging']
ttv = exoplanets[exoplanets['detection_type'] == 'TTV']
plt.loglog(transits['semi_major_axis'], transits['mass'], '.', label='Transit',markersize=12)
plt.loglog(radial_vel['semi_major_axis'], radial_vel['mass'], '.', label='Radial Vel', markersize=12)
plt.loglog(imaging['semi_major_axis'], imaging['mass'], '.', label='Direct Imaging', markersize=16)
plt.loglog(ttv['semi_major_axis'], ttv['mass'], '.', label='TTV', color='cyan', markersize=16)
plt.annotate("Earth", xy=(1,1/317.), size=12)
plt.annotate("Jupiter", xy=(5,1), size=12)
plt.xlabel('Semi-Major Axis [AU]', size=20)
plt.ylabel('Mass [M$_{Jup}$]', size=20)
plt.legend(loc=4, prop={'size':16})

# and now just for fun an xkcd style plot!
plt.xkcd()
plt.scatter(exoplanets['discovered'], exoplanets['radius']*11)
plt.xlabel('Year Discovered')
plt.ylabel('Radius [R_Earth]')


I am working on the file "Lessons/Stethoscope/Piezo Stethoscope.ipynb".
The first task is:
Now we have to add to our display the channels that are connected. We can do this by the add_sensor_X("Name") method. For example, if we wanted to view channel four and call it "My Heartbeat", we would call .add_sensor_one("My Heartbeat"). The variable we call in the constructor ("My Heartbeat") will be name of the graph that is displayed. Write the code to view the sensors that you want to use below.
Can you write Python code for it?

plot.add_sensor_one("Person one")
plot.add_sensor_two("Person two")
plot.add_sensor_three("Person three")
plot.add_sensor_four("Person four")


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:

Character Frequencies
To count character frequencies, we'll use a character iterator within a word iterator, and turn it loose on the entire chapter. The list containing each word converted to lowercase, lowerlist, will come in handy for this purpose.
Can you write Python code for it?

# Make a dictionary that contains total count of each character
charcount = {}
tot = 0
for word in lowerlist:
    for ch in word:
        if ch in charcount.keys():
            charcount[ch] += 1
            tot += 1
        elif (re.match('^[A-Za-z]{1,}$', ch) is not None):
            charcount[ch] = 1
            tot += 1

# Make a dictionary that contains frequency of each character
charfrequencies = {}
keys = charcount.keys()
keys.sort()
for k in keys:
    f = charcount[k]/(1.0*tot)
    charfrequencies[k] = f*100

print "%s : %s : %s"%('char.','occurr.','freq.')
for k in charcount.keys():
    print "%s : %04d times : %0.2f %%"%(k,charcount[k],charfrequencies[k])


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:

n-grams
We can also look at some n-grams - combinations of n letters. Let's start by revisiting our search for two vowels, and modify the regular expression we used to look for three or more vowels:
Can you write Python code for it?

vowel_ngrams = nltk.FreqDist(vs for word in lowerlist for vs in re.findall(r'[aeiou]{3,}',word))
vowel_ngrams.most_common()[:20]


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:

Patterns in Parts of Speech
Things get more interesting when we expand the number of words we're looking at to include phrases. For example, the following loop will look for phrases in the form <POS1> <POS2> <POS3> and will print them out:
Can you write Python code for it?

def pos_combo(pos1,pos2,pos3):
    for i in range(len(words_tags)-2):
        if( words_tags[i][1]==pos1 ):
            if( words_tags[i+1][1]==pos2 ):
                if( words_tags[i+2][1]==pos3 ):
                    print ' '.join([words_tags[i][0],words_tags[i+1][0],words_tags[i+2][0]])

pos_combo('ADJ','ADJ','NOUN')

pos_combo('ADV','ADV','VERB')

pos_combo('ADP','VERB','NOUN')


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:

Words, words, words
NLTK also provides tools for exploring the text at the word level. Let's start by looking at word lengths:
Can you write Python code for it?

import numpy as np
sizes = [len(word) for word in lowerlist]
mean_size = np.mean(sizes)
var_size = np.std(sizes)
print "Mean word size: " + str(mean_size)
print "Std. dev. in word size: " + str(var_size)


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
An interesting observation - between the nouns, pepositions, conjunctions, and periods, you've got 45% of the chapter covered.
There is an implicit bias, in simple taggers, to tag unknown words as nouns, so that may be the cause for all the nouns. But Lestrygonians is a more earthy, organic, and sensory chapter, so it would make sense that Joyce focuses more on nouns, on things and surroundings, and that those would fill up the chapter.
We'll see later on that we can explore those parts of speech tags to see whether they were correct and where they went wrong, when we look at multi-word phrases.
By utilizing the built-in FreqDist object, we can explore the most common occurrences of various parts of speech. For example, if we create a FreqDist with a tagged version of Lestrygonians, we can filter the most common words by their part of speech:
Can you write Python code for it?

# Create a frequency distribution from POS tag data
words_tags_fd = nltk.FreqDist(words_tags)

most_common_verbs = [wt[0] for (wt, _) in words_tags_fd.most_common() if wt[1] == 'VERB']
most_common_verbs[:30]

most_common_nouns = [wt[0] for (wt, _) in words_tags_fd.most_common() if wt[1] == 'NOUN']
most_common_nouns[:20]


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
By far the most common vowel-vowel bigram in Chapter 8 is "ou", with 554 occurrences, followed by "ea" (366 occurrences) and "oo" (304 occurrences) in a distant second and third place.
Can you write Python code for it?

vc_bigrams = nltk.FreqDist(vs for word in lowerlist for vs in re.findall(r'[aeiou][^aeiou]',word))
sorted(vc_bigrams.items(), key=lambda tup: tup[1], reverse=True)[:20]

cc_bigrams = nltk.FreqDist(vs for word in lowerlist for vs in re.findall(r'[^aeiou]{2}',word))
cc_bigrams.most_common()[:20]


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
Hmmmm......
In any case - if we want to save this model, we can save it in a pickle file:
Can you write Python code for it?

from pickle import dump
output = open('lestrygonians_parser.pkl', 'wb')
dump(t2, output, -1)
output.close()

## To load:
# from pickle import load
# input = open('lestrygonians_parser.pkl', 'rb')
# tagger = load(input)
# input.close()

better_tags = t2.tag(tokens)

print "%40s\t\t%30s"%("original","better")
for z in range(110):
    print "%35s (%s)\t%35s (%s)"%(words_tags[z][0],words_tags[z][1],better_tags[z][0],better_tags[z][1])


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
If we revisit the most common trigrams, we can see which words cuased those trigrams to appear so often. Take the "you" trigram, which was the most common at 140 occurrences:
Can you write Python code for it?

cvc_ngram = [(cv,w) for w in lowerlist for cv in re.findall(r'[^aeiou][aeiou][aeiou]',w)]
cvc_index = nltk.Index(cvc_ngram)

print "Number of words containing trigram 'you': " + str( len(cvc_index['you']) )
print "Number of unique words containing trigram 'you': " + str( len(set(cvc_index['you'])) )
print "They are:"
for t in set(cvc_index['you']):
    print t


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
Joyce's Ulysses is filled with colors. If we have a list of colors, we can pass them to the concordence one at a time, like so:
Can you write Python code for it?

colors = ['blue','purple','red','green','white','yellow']#'indigo','violet']
for c in colors:
    p()
    text.concordance(c,width=65)


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
Let's revisit that text object from earlier. Recall we created it using the code:
text = nltk.Text(tokens)
The Text object has made a determination of what words are similar, based on their appearance near one another in the text. We can obtain this list using the similar() method. The results are interesting to consider:
Can you write Python code for it?


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
Notice the woman-night and man-day association, and the woman-bull and man-nannygoat association. Some curious things here...
Because of the way the Text class is implemented, these are printed directly to output, instead of being returned as a list or something convenient. See the NLTK source for details. But we can fix this by extending the Text class to make our own class, called the SaneText class, to behave more sanely:
Can you write Python code for it?

##################################################
# This block of code is extending an NLTK object
# to modify its behavior.
#
# This creates a SaneText object, which modifies the similar() method
# to behave in a sane way (return a list of words, instead of printing it
# to the console).
#
# Source: NLTK source code.
# Modified lines are indicated.
# http://www.nltk.org/_modules/nltk/text.html#ContextIndex.word_similarity_dict

from nltk.compat import Counter
from nltk.util import tokenwrap

class SaneText(nltk.Text):

    def similar(self, word, num=20):
        """
        This is copied and pasted from the NLTK source code,
        but with print statements replaced with return statements.
        """

        if '_word_context_index' not in self.__dict__:
            #print('Building word-context index...')
            self._word_context_index = nltk.ContextIndex(self.tokens,
                                                    filter=lambda x:x.isalpha(),
                                                    key=lambda s:s.lower())

#        words = self._word_context_index.similar_words(word, num)

        word = word.lower()
        wci = self._word_context_index._word_to_contexts
        if word in wci.conditions():
            contexts = set(wci[word])
            fd = Counter(w for w in wci.conditions() for c in wci[w]
                          if c in contexts and not w == word)
            words = [w for w, _ in fd.most_common(num)]

            #######
            # begin changed lines
            return tokenwrap(words)
        else:
            return u''
            # end changed lines
            ######
#
# Done defining custom class
################################################

sanetext = SaneText(tokens)

def similar_words(w):
    print "%32s : %s"%( w, sanetext.similar(w)...


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
Some curious and unexpected connections here - particularly, "God and "publichouse,"

Part of Speech
Let's tag each sentence of the text with its part of speech (POS), using NLTK's built-in method (trained on a large corpus of data).
NOTE: It's important to be very skeptical of the part of speech tagger's results. The following does not focus on whether the parts of speech being tagged are correct, as that leads to other topics involving multiple words. Here, we show how to analyze the results of a part of speech tagger, not how to train one to be more accurate.
We can tag the parts of speech with the nltk.pos_tag() method. This is a static method that results in a list of tuples. Each tuple represents a word and its part of speech tag. The sentence
I am Jane.
would be tagged:
I (PRON) am (VERB) Jane (NOUN) . (.)
and would be stored as the tuple:
[('I',    'PRON'),
 ('am',   'VERB'),
 ('Jane', 'NOUN'),
 ('.',    '.')]
Can you write Python code for it?


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
The adjective-pronoun-verb combination seems to happen when there is a change or transition in what the sentence is saying - the beginning of a new phrase.
On the other hand, if we look at the adjective-noun-adjective combination, which is similarly infrequent, it shows words that look like they were intended to cluster together:
Can you write Python code for it?

pos_combo('ADJ','NOUN','ADJ')

print sanetext
print dir(sanetext)


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
The fact that the pronoun-verb combination occurs frequently, but the adjective-pronoun-verb combination does not, indicates that the adjective-pronoun-verb combination is unlikely to be a sensible phrase. By analyzing a few specific sentences and identifying issues with tags, as we did above, it's possible to improve the part of speech tagger. It's also possible to use a hierarchical part of speech tagger - one that starts by looking at three or four neighboring words, and determing the part of speech based on their part of speech. If that fails, it looks at two neighboring words, or one neighboring word, until the simplest case, when it looks at a single word itself. Many of the part of speech tags that failed were fooled by suffixes of words, and ignored their context.

Improving Part of Speech Tagging
To improve the tagging of parts of speech, we can use n-gram tagging, which is the idea that when you're tagging a word with a part of speech, it can be helpful to look at neighboring words and their parts of speech. However, in order to do this, we'll need a set of training data, to train our part of speech tagger.
Start by importing a corpus. The brown corpus, described at the Brown Corpus wikipedia page, is approximately one million tagged English words, in a range of different categories. This is far more complete than the treebank corpus, another tagged (but partially complete) corpus from the University of Pennsylvania (the Treebank project's homepage gives a 404, but see Treebank-3 from the Linguistic Data Consortium).
We'll use two data sets: one training data set, to train the part of speech tagger, using tagged sentences; and one test data set, to which we apply our part of speech tagger, and compare to a known result (tagged versions of the same sentences), enabling a quantitative measure of accuracy.
Can you write Python code for it?

from nltk.corpus import brown
print dir(brown)

print brown.categories()

# Get tagged and untagged sentences of fiction
btagsent = brown.tagged_sents(categories='fiction')
bsent = brown.sents(categories='fiction')


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
The next step is to compute how large of a deviation from "normal" the frequency of this character is. We use the formula for percent difference. If the character frequency of a given character is given by $y$, and the character frequency of that character in the Lestrygonians chapter is denoted $y_L$ and the character frequency of that character in the English language is given by $y_{E}$, then the formula for percent difference is given by:
$$
\text{Pct Diff} = \dfrac{y_{L} - y_{E}}{y_{E}}
$$
Can you write Python code for it?

frequency_variation = {}
for k in common_keys:
    clf = charfrequencies[k]
    elf = EnglishLanguageLetterFrequency[k]
    pd = ((clf-elf)/elf)*100
    frequency_variation[k] = pd

for k in frequency_variation.keys():
    print "%s : %0.2f %%"%(k,frequency_variation[k])


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
The regular expression syntax '[aeiou]' will match any vowels, and the '{2}' syntax means, match exactly 2 occurrences of the pattern (two vowels in a row).
Can you write Python code for it?

sorted(vowel_bigrams.items(), key=lambda tup: tup[1], reverse=True)


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
The variable tokens is now a list containing all the word-level tokens that resulted from word_tokenize():
Can you write Python code for it?

print type(tokens)
print len(tokens)

print tokens[:21]


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
This also allows us to do things like compile lists of words that are either colors themselves, or that have color words in them. To find unique words, we use a set object, a built-in type for Python, and add words that contain color words ('blue','green', and so on). The color red tends to add lots of non-color words so is excluded.
Can you write Python code for it?

colors = ['orange','yellow','green','blue','indigo','rose','violet']#,'red']
mecolors = set()
_ = [mecolors.add(w.lower()) for c in colors for w in tokens if re.search(c,w.lower()) ]
print mecolors


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
To compose Lestrygonians, Joyce's choice of words tended toward words containing more harsh consonants like b's, g's, j's, and k's; shorter word lengths (in keeping with the peristaltic nature of the chapter's language, masticating words, passing them down the mental esophagus one bit at a time (2-6 letters).
What about the percentage of vowels and consonants occurring in the text?
This should be possible with a one liner. We'll have a nested for loop - one loop over all lowercase word tokens, and one over all letters in the word token. Then we'll count up the number of matches of exactly 1 instance of [aeiou]  and the number of matches of exactly 1 instance of letters that are not aeiou (note that [^aeiou] will also count punctuation, we need to be careful), and that will give us our letter count.
Can you write Python code for it?

# Count consonants and vowels
consonants = [re.findall('[b-df-hj-np-tv-z]',w) for w in lowerlist]
n_consonants = sum([len(e) for e in consonants])
vowels = [re.findall('[aeiou]',w) for w in lowerlist]
n_vowels = sum([len(e) for e in vowels])

print "Lestrygonians contains %d consonants and %d vowels, ratio of vowels:consonants is %0.2f"%(n_consonants,n_vowels,n_vowels/n_consonants)


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
We also see that interesting phenomena, whereby text is still partly or mostly comprehensible, even with all of the vowels removed:
Can you write Python code for it?

''.join([''.join(t)+" " for t in consonants[:145]])
#for t in consonants[:21]:
#    print ''.join(t)


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
We can also look for consonant-vowel-vowel or vowel-vowel-consonant combinations,
Can you write Python code for it?

cvv_ngrams = nltk.FreqDist(vs for word in lowerlist for vs in re.findall(r'[^aeiou][aeiou]{2}',word))
cvv_ngrams.most_common()[:20]

vvc_ngrams = nltk.FreqDist(vs for word in lowerlist for vs in re.findall(r'[aeiou]{2}[^aeiou]',word))
vvc_ngrams.most_common()[:20]

vcv_ngrams = nltk.FreqDist(vs for word in lowerlist for vs in re.findall(r'[aeiou][^aeiou][aeiou]',word))
vcv_ngrams.most_common()[:20]


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
We see (and would expect) that there is more variation among the less common letters (higher sensitivity to variation).
However, there are some meaningful trends - if we pick out the letters that have a large positive percent difference, meaning they occur more in this chapter than they usually do in English, we find more harsh sounds: there are more b's, g's, j's and k's in this chapter than would be expected. These sounds are more guttural, cutting, blubbering letters, and fit with the character of Lestrygonians. Joyce's word selection in this chapter comes through even at the character frequency level.

Bigrams
The next thing we can do is use regular expressions to analyze the bigrams that appear in the chapter. Regular expressions allow us to seek out pairs of letters matching a particular pattern. Combined with list comprehensions, we can do some very powerful analysis with just a little bit of code.
For example, this one-liner iterates over each word token, grabs bigrams matching a particular regular expression, and uses the result to initialize a frequency distribution object, which enables quick and easy access to useful statistical information about the bigrams:
Can you write Python code for it?

vowel_bigrams = nltk.FreqDist(vs for word in lowerlist for vs in re.findall(r'[aeiou]{2}',word))


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
We'll use NLTK's built-in word tokenizer to tokenize the text. There are many tokenizers available, both at the word and sentence level, but the word_tokenize() function is the no-hassle option.
Can you write Python code for it?

######################################
# Words, words, words.
# Start by splitting the text into word tokens.

# Create a word tokenizer object.
# This uses io.open() because that deals with unicode more gracefully.
tokens = nltk.word_tokenize(io.open('txt/08lestrygonians.txt','r').read())


I am working on the file "Lestrygonians Part 1.ipynb".
The first task is:
with sopping sippets above shows an example of a mislabeled word - sopping should be labeled as an adjective, but was mislabeled by a tagger that was indiscriminately labeling "ing" words as verbs (getting, writing, holding). Likewise, pudding was accidentally tagged as a verb for the same reason. There's also an -ed word, bloodhued, which was, for some inexplicable reason, tagged as a verb, instead of an adjective.
In any case, we see that there is room for improvement, but it's an interesting way to explore the text nevertheless.
We already explored what parts of speech are the most common in this text. But suppose we're interested, now, in what combinations of parts of speech are the most common. To do this, we can construct a conditional frequency distribution based on the parts of speech.
If we're thinking about three-letter phrases, then we want to tabulate the frequency of particular combinations of three parts of speech show up. We can think of this as a probability distribution of three random variables taking on values from a set of nominal values ('NOUN', 'VERB', etc.). This is equivalent to a three-dimensional space chopped up into bins, with different frequencies occurring in each bin based on the words appearing in the text.
However, because NLTK's built-in conditional frequency object is only designed to handle two-dimensional data, we'll have to be a bit careful. We'll start by picking the first part of speech (thus eliminating one variable). Then we'll tabulate the conditional frequencies the combinations of parts of speech for the remaining two words.
Can you write Python code for it?

def get_trigrams(tagged_words):
    for i in range(len(tagged_words)-2):
        yield (tagged_words[i],tagged_words[i+1],tagged_words[i+2])

trigram_freq = {}
for ((word1,tag1),(word2,tag2),(word3,tag3)) in get_trigrams(words_tags):
    if( tag1 in trigram_freq.keys() ):
        trigram_freq[tag1].append((tag2,tag3))
    else:
        trigram_freq[tag1] = [(tag2,tag3)]

adj_cf = nltk.ConditionalFreqDist(trigram_freq['ADJ'])
print adj_cf.tabulate()


I am working on the file "Lestrygonians Part 2.ipynb".
The first task is:

Chapter-by-Chapter Analysis
If we want to start analyzing Ulysses as a whole and look for connections across chapters, we'll need objects to store data about each chapter, objects that will encapsulate much of the functionality laid out in Part I and Part II of these notebooks.
To design such an object, a Lestrygonians object, we would first want to define a UlyssesChapter object. The constructor would take a text file representing the chapter. There would be a number of methods to get useful lists, dictionaries, or sets.
Useful lists:
* List of tokens representing words and word boundaries
* List of tokens representing all words turned into lowercase
* List of tokens representing only the lowercase words
* List of sentences
* If the chapter has a more complicated structure, such as 07 Aeolus or 15 Circe, then a special/custom parser would be needed and different methods available (list of headlines, or list of characters, or lists of dialogue)
Useful dictionaries:
* Word counts - all words
* Word counts - unique words (would count base words toward base) (eyes --> eye)
Useful sets:
* Unique words appearing in this chapter
Building wordlists:
* Use a set of words, and search for larger words containing them.
 * Example: ['orange','yellow','green','blue','indigo','rose','violet'] becomes [u'blue', u'greenhouses', u'greens', u'penrose', u'orangepeels', u'bluecoat', u'orangegroves', u'greeny', u'yellow', u'bluey', u'yellowgreen', u'green', u'rose', u'blues', u'greenwich']
 * Not perfect, but it can certainly be improved

Word and Subject Tagging
While it would be a long and labor-intensive process to build a grammar parser that could understand, even at a primitive level, the text of Ulysses, we can get some useful information from some very dumb tags.
In other words, by tagging various sentences, based on the words contained in them, we can find themes that run throughout the novel - from objects that appear and reappear (soap, eyes, bread) to people...
Can you write Python code for it?

words = nltk.word_tokenize(file_contents)
print len(words)

import pandas as pd

wdf = pd.DataFrame(columns=['words'],data=words[:50])
wdf['cream'] = wdf.apply( lambda x : len( re.findall('cream',x.values[0] ) ), axis=1)
print wdf


I am working on the file "Lestrygonians Part 2.ipynb".
The first task is:

Combining Context with Patterns
This is a useful function that we can combine with some other conditions - such as searching a wordlist for words matching a certain pattern. Then we can pass a pattern, and get back each word matching our pattern, with three sentences of context. We'll need a wordlist first, which we can obtain by tokenizing each of our sentences.
Can you write Python code for it?

wordlist = nltk.word_tokenize(file_contents)
wordlist = [w.lower() for w in wordlist]
english_words = [w for w in nltk.corpus.words.words('en') if w.islower()]
z1 = set(wordlist)
z2 = set(english_words)

print "Number of words in Chapter 8:",len(wordlist)
print "Number of unique words in Ch. 8:",len(z1)
print "Number of words in English dictionary:",len(z2)
print "Numer of words in Ch. 8 in English dictionary:",len( z1.intersection(z2) )

intersection = z1.intersection(z2)
non_dictionary_words = z1.symmetric_difference(intersection)
print len(non_dictionary_words)

non_dictionary_words = sorted(list(non_dictionary_words))
print non_dictionary_words[110:125]


I am working on the file "Lestrygonians Part 2.ipynb".
The first task is:
Now that we've tokenized the text by sentence, we can set to work. The first useful task we'll want to be able to do is to print out a sentence if it contains a given word. We can use a Text object and use the concordance('word') function, but this only prints out the context - and does not return the sentence or its context.

Words with Context
Suppose we want to search for a word, like "eye", and we want to return the sentence that contains it, along with two sentences of context (the sentence before, and the sentence after).
We can do this by looping through each sentence, breaking it apart using a word tokenizer, and searching for the word of interest. If we find it, we add the prior sentence, current sentence, and next sentence to the list of instances.
Can you write Python code for it?

small_sentences = sentences[:21]

def word_with_context(word,sentences):
    final_list = []
    for i,sentence in enumerate(sentences):
        if i>0 and i<(len(sentences)-1):
            words = nltk.word_tokenize(sentence)
            if word in words:
                final_list.append( [re.sub('\n',' ',sentences[i-1] ),
                                    re.sub('\n',' ',sentences[i]   ),
                                    re.sub('\n',' ',sentences[i+1] ) ]
                                  )
    return final_list

for i in word_with_context('eyes',sentences):
    p()
    print '\n'.join(i)


I am working on the file "Lestrygonians Part 2.ipynb".
The first task is:
Now, we are able to pass words and regular expressions, and get a few sentences of context back in return. We can use various techniques to identify keywords, or provide keywords from a file, or from a list. We could iterate through a file containing any of the following things:
* Beverages
* Food items
* Kitchen items
* Numbers
* Colors
* Animals
We can also look for particular phonetic sounds, which often occur in groups (as we can see from the word searches above, many of the sentences are repeated because the "ood" pattern often shows up repeatedly over a few sentences.
We can also look for patterns across the chapters - something we haven't done yet, since we've been focusing on Chapter 8 alone, as a smaller and more manageable body of text.
First, let's expand on that context function, to print out N sentences of context:
Can you write Python code for it?

def re_with_context(rex,sentences,n_sentences):
    final_list = []
    half = (int)(np.floor(n_sentences/2))
    for i,sentence in enumerate(sentences):
        if i>=half and i<(len(sentences) - half):
            words = nltk.word_tokenize(sentence)
            for word in words:
                if len(re.findall(rex,word))>0:
                    short_list = []
                    for s in sentences[i-half:i] + sentences[i:i+half+1]:
                        short_list.append( re.sub(r'[\n\t]',' ',s) )
                    final_list.append(short_list)
    return final_list

for group in re_with_context('eyes',sentences,5):
    p()
    print '\n'.join(group)


I am working on the file "Lestrygonians Part 2.ipynb".
The first task is:
The phrase "woman's breasts full" is reminiscent of Lady Macbeth's speech from Macbeth, Act 1 Scene 5, when she discovers Duncan is staying the night (it has a somewhat, uh, different tone):

Stop up the access and passage to remorse,

That no compunctious visitings of nature

Shake my fell purpose, nor keep peace between

The effect and it! Come to my woman’s breasts,

And take my milk for gall, you murd'ring ministers,

Wherever in your sightless substances

You wait on nature’s mischief.

- Macbeth, Act 1, Scene 5

If instead we wanted to search for words matching a regular expression, we could write a function that takes a regular expression, searches for words matching that expression, and passes them to the word_with_context() function.
Can you write Python code for it?

def re_with_context(rex,sentences):
    final_list = []
    for i,sentence in enumerate(sentences):
        if i>0 and i<(len(sentences)-1):
            words = nltk.word_tokenize(sentence)
            for word in words:
                if len(re.findall(rex,word))>0:
                    final_list.append( [re.sub('\n',' ',sentences[i-1] ),
                                        re.sub('\n',' ',sentences[i]   ),
                                        re.sub('\n',' ',sentences[i+1] ) ]
                                      )
    return final_list

for i,ss in enumerate(re_with_context(r'ood\b',sentences)):
    if i<25:
        p()
        print '\n'.join(ss)


I am working on the file "Lestrygonians Part 2.ipynb".
The first task is:
We now have a list of words that aren't found in an English dictionary provided by the NLTK corpus, so these have the potential to be interesting words. We'll use these results to print out some context for each word.
While we're at it, we can also get word counts of each of these words using a Text object:
Can you write Python code for it?

text = nltk.Text(wordlist)
print "Number of occurences of",non_dictionary_words[115],":",text.count(non_dictionary_words[113])

result = word_with_context(non_dictionary_words[115],sentences)
print '\n'.join(result[0])


I am working on the file "Lestrygonians Part 4.ipynb".
The first task is:

Acronyms
Let's take a look at some acronyms. For this application, it might be better to tokenize by sentence, and extract acronyms for sentences.
Can you write Python code for it?

with io.open(txtfile) as f:
    sentences = nltk.sent_tokenize(f.read())
print len(sentences)

acronyms = []
for s in sentences:
    s2 = re.sub('\n',' ',s)
    words = s2.split(" ")
    acronym = ''.join(w[0] for w in words if w<>u'')
    acronyms.append(acronym)

print len(acronyms)
print "-"*20
printlist(acronyms[:10])
print "-"*20
printlist(sentences[:10]) # <-- contains newlines, but removed to create acronyms

from nltk.corpus import words

acronyms[101:111]


I am working on the file "Lestrygonians Part 4.ipynb".
The first task is:

Functionalizing Bigram Searches
We can functionalize the search for patterns with a single and double character shared, i.e., dropping currants (the letter r).
Can you write Python code for it?

def double_letter_alliteration(c,tokens):
    """
    This function finds all occurrences of double-letter and single-letter
    occurrences of the character c.

    This function is called by all_double_letter_alliteration().
    """
    allall  = []
    for (i,j) in nltk.bigrams(tokens):
        if len( re.findall(c+c,i) )>0:
            if len( re.findall(c,j) )>0:
                lolly.append( ' '.join([i,j]) )
        elif len( re.findall(c+c,j) )>0:
            if len( re.findall(c,i) )>0:
                allall.append(' '.join([i,j]) )
    return allall


I am working on the file "Lestrygonians Part 4.ipynb".
The first task is:

Method 1: TokenSearcher Object
The first method for searching for regular expressions in a set of tokens is the TokenSearcher object. This can be fed a regular expression that searches across tokens, and it will search through each token. This provides a big advantage: we don't have to manually break all of our tokens into n-grams ourselves, we can just let the TokenSearcher do the hard work.
Here's an example of how to create and call that object:
Can you write Python code for it?

tsearch = nltk.TokenSearcher(tokens)
s_s_ = tsearch.findall(r'<s.*> <.*> <s.*> <.*> <.*>')
print len(s_s_)
for s in s_s_:
    print ' '.join(s)


I am working on the file "Lestrygonians Part 4.ipynb".
The first task is:
Let's look at the pattern taken one step further: we'll look for double letters in neighbor words.
Can you write Python code for it?

def match_double(aleph,tokens):
    matches = []
    for (i,j) in nltk.bigrams(tokens):
        if len( re.findall(aleph+aleph,i) )>0:
            if len( re.findall(aleph+aleph,j) )>0:
                matches.append(' '.join([i,j]))
    return matches

def double_double(tokens):
    dd = []
    alphabet = list(string.ascii_lowercase)
    for aleph in alphabet:
        results = match_double(aleph, tokens)
        print "Matching %s%s: %d"%(aleph,aleph,len(results))
        dd += results
    return dd

print "Neighbor words with double letters:"
dd = double_double(tokens)
printlist(dd)


I am working on the file "Lestrygonians Part 4.ipynb".
The first task is:
Now we can use this function to search for the single-double letter pattern individually, or we can define a function that will loop over all 26 letters to find all matching patterns.
Can you write Python code for it?

printlist(double_letter_alliteration('r',tokens))

printlist(double_letter_alliteration('o',tokens))

import string

def all_double_letter_alliteration(tokens):
    all_all = []
    alphabet = list(string.ascii_lowercase)
    for aleph in alphabet:
        results = double_letter_alliteration(aleph,tokens)
        print "Matching",aleph,":",len(results)
        all_all += results
    return all_all

allall = all_double_letter_alliteration(tokens)
print len(allall)


I am working on the file "Lestrygonians Part 4.ipynb".
The first task is:
That's a mouthful of alliteration! We can compare the number of words that matched this (one, single) search for examples of alliteration to the total number of words in the chapter:
Can you write Python code for it?


I am working on the file "Leukemia.ipynb".
The first task is:
Enrichment score calculations
We graphically present the calculation of ES.
Can you write Python code for it?


I am working on the file "Leukemia.ipynb".
The first task is:
GSEA analysis
Can you write Python code for it?

# use `n_jobs=-1` to use all cores
%time order, NES, p_values = gsea(D, C, gene_sets, n_jobs=-1)

from IPython.display import display, Markdown
s = "| geneset | NES | p-value | number of genes in geneset |\n |-------|---|---|---|\n "
for i in range(len(order)):
    s = s + "| **%s** | %.3f | %.7f | %d |\n" % (gene_set_names[order[i]][0], NES[i], p_values[i], len(gene_sets[order[i]]))
display(Markdown(s))


I am working on the file "Leukemia.ipynb".
The first task is:
Load data
Can you write Python code for it?

genes, D, C = read_expression_file("data/leukemia.txt")
gene_sets, gene_set_names = read_genesets_file("data/pathways.txt", genes)
gene_set_hash = {}
for i in range(len(gene_sets)):
    gene_set_hash[gene_set_names[i][0]] = {'indexes':gene_sets[i],'desc':gene_set_names[i][1]}

# verify that the dimensions make sense
len(genes),D.shape,len(C)


I am working on the file "Leukemia.ipynb".
The first task is:
Multiple Hypotesis testing
We present two example gene sets. One with a high NES and low p-value and one with a low NES and a high p-value. We plot the histograms of null distribution for ES.
Can you write Python code for it?

name = 'DNA_DAMAGE_SIGNALLING'
L,r = rank_genes(D,C)
plot_es_calculations(name, L, r)

n = 1000
S =  gene_set_hash[name]['indexes']
L, r = rank_genes(D,C)
ES = enrichment_score(L,r,S)
ES_pi = np.zeros(n)
for i in range(n):
    pi = np.array([np.random.randint(0,2) for i in range(k)])
    L, r = rank_genes(D,pi)
    ES_pi[i] = enrichment_score(L,r,S)

hist(ES_pi,bins=100)
plot([ES,ES],[0,20],'r-',label="ES(S)")
title("Histogram of ES vlues for random phenotype labels.\nRed line is ES for the selected gene set.")

name = 'tcrPathway'
L,r = rank_genes(D,C)
plot_es_calculations(name, L, r)

n = 1000
S =  gene_set_hash[name]['indexes']
L, r = rank_genes(D,C)
ES = enrichment_score(L,r,S)
ES_pi = np.zeros(n)
for i in range(n):
    pi = np.array([np.random.randint(0,2) for i in range(k)])
    L, r = rank_genes(D,pi)
    ES_pi[i] = enrichment_score(L,r,S)

hist(ES_pi,bins=100)
plot([ES,ES],[0,20],'r-',label="ES(S)")
title("Histogram of ES vlues for random phenotype labels.\nRed line is ES for the selected gene set.")


I am working on the file "Leukemia.ipynb".
The first task is:
Performance optimizations
Can you write Python code for it?

%timeit L,R = rank_genes(D,C)

%timeit ES = enrichment_score(L,r,S)

%prun order, NES, p_values = gsea(D, C, gene_sets)


I am working on the file "Level_01/Level_1.ipynb".
The first task is:
Bei der Benennung von Variablen sollte darauf geachtet werden, kurze aber verständliche Variablennamen zu benutzen, da so klar ist wozu die Variable benutzt wird. Auf keinen Fall sollten Variablennamen wie l, O oder I benutzt werden, da diese, je nach Schriftart, wie 0 oder 1 aussehen können.
Can you write Python code for it?

# den Typen eines Wertes können wir mit type() bestimmt werden:
print(type("a"))
print(type(2))
print(type(4.8))


I am working on the file "Level_01/Level_1.ipynb".
The first task is:
Eingabe
Die input()Funktion wird benutzt um Eingaben über die Standardeingabe zu erhalten. Dabei liefert die input()Funktion immer einen string.
Can you write Python code for it?

eingabe = input("Bitte etwas eingeben: ")
print(eingabe)
print(type(eingabe))


I am working on the file "Level_01/Level_1.ipynb".
The first task is:
Genauer betrachtet besteht die Zeile 4 + 34 aus zwei Literalen (4 und 34) und einem Operator (+), die kombiniert den Ausdruck ergeben. Ein Literal ist die direkte Darstellung eines Wertes. Operatoren verknüpfen Werte und geben Werte zurück.
Bei den Werten im obigen Beispiel handelt es sich um Werte vom Typ integer. Diese stellen ganze Zahlen dar.
Can you write Python code for it?

print(3 + 4)  # Addition
print(4 - 6)  # Subtraktion
print(3 * 7)  # Multiplikation
print(3 // 2) # Ganzzahlige Division
print(3 % 2)  # Division mit Rest
print(3 / 2)  # Division
print(2 ** 4) # Potenz, alternativ pow(2, 4)
print(4 << 1) # Bitshift nach links, alternativ 4 * (2 ** 1)
print(4 >> 1) # Bitshift nach rechts, alternativ 4 // (2 ** 1)
print(5 ^ 1)  # bitweises XOR