I am working on the file "Crash Course v0.62.ipynb".
The first task is:
3.8 Functional programming
Functional programming is a very broad subject. The idea is to have a series of functions, each of which generates a new data structure from an input, without changing the input structure at all. By not modifying the input structure (something that is called not having side effects), many guarantees can be made about how independent the processes are, which can help parallelization and guarantees of program accuracy. There is a Python Functional Programming HOWTO in the standard docs that goes into more details on functional programming. I just wanted to touch on a few of the most important ideas here.
There is an operator module that has function versions of most of the Python operators. For example:
Can you write Python code for it?

from operator import add, mul
add(1,2)

mul(3,4)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
3.9 Object Oriented Programming
We've seen a lot of examples of objects in Python. We create a string object with quote marks:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
A triangular well:
Can you write Python code for it?

def triangular(x,F=30): return F*x

tw = Schrod1d(triangular,m=10)
tw.plot()


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Although enumerate uses generators (see below) so that it doesn't have to create a big list, which makes it faster for really long sequenes.
3.4 Optional arguments of a function
You will recall that the linspace function can take either two arguments (for the starting and ending points):
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
And assign it to a function separately:
Can you write Python code for it?

another_doubler = lambda x: 2*x
another_doubler(19)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
As well as Jacobi, Laguerre, Hermite polynomials, Hypergeometric functions, and many others. There's a full listing at the Scipy Special Functions Page.
2.8 Least squares fitting
Very often we deal with some data that we want to fit to some sort of expected behavior. Say we have the following:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
But it can also print data types other than strings:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
By providing a little bit more information, you're less likely to make mistakes, like referring to your customer as "alien attack".
As a scientist, you're less likely to be sending bulk mailings to a bunch of customers. But these are great methods for generating and submitting lots of similar runs, say scanning a bunch of different structures to find the optimal configuration for something.
For example, you can use the following template for NWChem input files:
Can you write Python code for it?

nwchem_format = """
start %(jobname)s

title "%(thetitle)s"
charge %(charge)d

geometry units angstroms print xyz autosym
%(geometry)s
end

basis
  * library 6-31G**
end

dft
  xc %(dft_functional)s
  mult %(multiplicity)d
end

task dft %(jobtype)s
"""


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Dictionaries are an object called "mappings" or "associative arrays" in other languages. Whereas a list associates an integer index with a set of objects:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Everything in Python is an object, including functions. This means that functions can be returned by other functions. (They can also be passed into other functions, which is also useful, but a topic for another discussion.) In the gauss_maker example, the g function that is output "remembers" the A, a, x0 values it was constructed with, since they're all stored in the local memory space (this is what the lexical closure really refers to) of that function.
Factories are one of the more important of the Software Design Patterns, which are a set of guidelines to follow to make high-quality, portable, readable, stable software. It's beyond the scope of the current work to go more into either factories or design patterns, but I thought I would mention them for people interested in software design.
3.7 Serialization: Save it for later
Serialization refers to the process of outputting data (and occasionally functions) to a database or a regular file, for the purpose of using it later on. In the very early days of programming languages, this was normally done in regular text files. Python is excellent at text processing, and you probably already know enough to get started with this.
When accessing large amounts of data became important, people developed database software based around the Structured Query Language (SQL) standard. I'm not going to cover SQL here, but, if you're interested, I recommend using the sqlite3 module in the Python standard library.
As data interchange became important, the eXtensible Markup Language (XML) has emerged. XML makes data formats that are easy to write parsers for, greatly simplifying the ambiguity that sometimes arises in the process. Again, I'm not going to cover XML here, but if you're interested in learning more, look into Element Trees, now part of the Python standard library.
Python has a very general serialization format called pickle that can turn any Python object, even a function or a class, into a representation that can be written to a file and read in later. But, again,...
Can you write Python code for it?

# Data in a json format:
json_data = """\
{
    "a": [1,2,3],
    "b": [4,5,6],
    "greeting" : "Hello"
}"""
import json
loaded_json=json.loads(json_data)
loaded_json


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Finally, note that you can also string multiple comparisons together, which can result in very intuitive tests:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Here i%2 is the remainder when i is divided by 2, so that i%2==1 is true if the number is odd. Even though this is a relative new addition to the language, it is now fairly common since it's so convenient.
iterators are a way of making virtual sequence objects. Consider if we had the nested loop structure:
for i in range(1000000):
    for j in range(1000000):

Inside the main loop, we make a list of 1,000,000 integers, just to loop over them one at a time. We don't need any of the additional things that a lists gives us, like slicing or random access, we just need to go through the numbers one at a time. And we're making 1,000,000 of them.
iterators are a way around this. For example, the xrange function is the iterator version of range. This simply makes a counter that is looped through in sequence, so that the analogous loop structure would look like:
for i in xrange(1000000):
    for j in xrange(1000000):

Even though we've only added two characters, we've dramatically sped up the code, because we're not making 1,000,000 big lists.
We can define our own iterators using the yield statement:
Can you write Python code for it?

def evens_below(n):
    for i in range(n):
        if i%2 == 0:
            yield i
    return

for i in evens_below(9):
    print(i)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Here we did two things with tuples you haven't seen before. First, we unpacked an object into a set of named variables using tuple assignment:
>>> name,x,y = obj

We also returned multiple values (minx,miny), which were then assigned to two other variables (x,y), again by tuple assignment. This makes what would have been complicated code in C++ rather simple.
Tuple assignment is also a convenient way to swap variables:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Here's an interesting use of random numbers: compute $\pi$ by taking random numbers as x and y coordinates, and counting how many of them were in the unit circle. For example:
Can you write Python code for it?

npts = 5000
xs = 2*np.random.rand(npts)-1
ys = 2*np.random.rand(npts)-1
r = xs**2+ys**2
ninside = (r<1).sum()

plt.figure(figsize=(6,6)) # make the figure square
plt.title("Approximation to pi = %f" % (4*ninside/float(npts)))
plt.plot(xs[r<1],ys[r<1],'b.')
plt.plot(xs[r>1],ys[r>1],'r.')
plt.figure(figsize=(8,6)) # change the figsize back to standard size for the rest of the notebook


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
However, if we did want to write a function ourselves, we could do recursively by noting that
$$ n! = n(n-1)!$$
The program then looks something like:
Can you write Python code for it?

def fact(n):
    if n <= 0:
        return 1
    return n*fact(n-1)

fact(20)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
However, it's good (but not essential) to use lists for similar objects that are somehow logically connected. If you want to group different data types together into a composite data object, it's best to use tuples, which we will learn about below.
You can find out how long a list is using the len() command:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
However, tuples are immutable, you can't append to them or change the elements of them:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
If we evaluate it by itself, as we just did, we see that it returns a boolean value, False. The "==" operator performs equality testing. If the two items are equal, it returns True, otherwise it returns False. In this case, it is comparing two variables, the string "Sunday", and whatever is stored in the variable "day", which, in this case, is the other string "Saturday". Since the two strings are not equal to each other, the truth test has the false value.
The if statement that contains the truth test is followed by a code block (a colon followed by an indented block of code). If the boolean is true, it executes the code in that block. Since it is false in the above example, we don't see that code executed.
The first block of code is followed by an else statement, which is executed if nothing else in the above if statement is true. Since the value was false, this code is executed, which is why we see "Go to work".
Try setting the day equal to "Sunday" and then running the above if/else statement. Did it work as you thought it would?
You can compare any data types in Python:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
If we really want to get fancy, we can pass a third element into the slice, which specifies a step length (just like a third argument to the range() function specifies the step):
Can you write Python code for it?

numbers = list(range(0,21))
evens = numbers[6::2]
evens


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
If we want the list containing the first two elements of a list, we can do this via
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
If you have a lot of words to concatenate together, there are other, more efficient ways to do this. But this is fine for linking a few strings together.
1.3 Lists
Very often in a programming language, one wants to keep a group of similar items together. Python does this using a data type called lists.
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
If you try to access a variable that you haven't yet defined, you get an error:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
In conjunction with matplotlib, this is a nice way to plot things:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Just like the other two data objects we're familiar with (ints and floats), you can assign a string to a variable
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Let's compare the first function to our solution.
Can you write Python code for it?

plt.plot(x,ho_evec(x,0,1,1),label="Analytic")
plt.plot(x,-U[:,0]/np.sqrt(h),label="Numeric")
plt.xlabel('x (bohr)')
plt.ylabel(r'$\psi(x)$')
plt.title("Comparison of numeric and analytic solutions to the Harmonic Oscillator")
plt.legend()


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Let's define a finite well potential:
Can you write Python code for it?

def finite_well(x,V_left=1,V_well=0,V_right=1,d_left=10,d_well=10,d_right=10):
    V = np.zeros(x.size,'d')
    for i in range(x.size):
        if x[i] < d_left:
            V[i] = V_left
        elif x[i] > (d_left+d_well):
            V[i] = V_right
        else:
            V[i] = V_well
    return V

fw = Schrod1d(finite_well,start=0,end=30,npts=100)
fw.plot()


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Let's go through this line by line. First, we define the variable n, and set it to the integer 20. n is the length of the sequence we're going to form, and should probably have a better variable name. We then create a variable called sequence, and initialize it to the list with the integers 0 and 1 in it, the first two elements of the Fibonacci sequence. We have to create these elements "by hand", since the iterative part of the sequence requires two previous elements.
We then have a for loop over the list of integers from 2 (the next element of the list) to n (the length of the sequence). After the colon, we see a hash tag "#", and then a comment that if we had set n to some number less than 2 we would have a problem. Comments in Python start with #, and are good ways to make notes to yourself or to a user of your code explaining why you did what you did. Better than the comment here would be to test to make sure the value of n is valid, and to complain if it isn't; we'll try this later.
In the body of the loop, we append to the list an integer equal to the sum of the two previous elements of the list.
After exiting the loop (ending the indentation) we then print out the whole list. That's it!
1.8 Functions
We might want to use the Fibonacci snippet with different sequence lengths. We could cut an paste the code into another cell, changing the value of n, but it's easier and more useful to make a function out of the code. We do this with the def statement in Python:
Can you write Python code for it?

def fibonacci(sequence_length):
    "Return the Fibonacci sequence of length *sequence_length*"
    sequence = [0,1]
    if sequence_length < 1:
        print("Fibonacci sequence only defined for length 1 or greater")
        return
    if 0 < sequence_length < 3:
        return sequence[:sequence_length]
    for i in range(2,sequence_length):
        sequence.append(sequence[i-1]+sequence[i-2])
    return sequence


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Let's see whether this works for our sin example from above:
Can you write Python code for it?

x = np.linspace(0,2*np.pi)
dsin = nderiv(np.sin(x),x)
plt.plot(x,dsin,label='numerical')
plt.plot(x,np.cos(x),label='analytical')
plt.title("Comparison of numerical and analytical derivatives of sin(x)")
plt.legend()


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Next lets generate the factorials.
Can you write Python code for it?

facts = []
for i in range(10):
    facts.append(factorial(i))


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
No matter how experienced a programmer you are, these are words to meditate on.
2. Numpy and Scipy
Numpy contains core routines for doing fast vector, matrix, and linear algebra-type operations in Python. Scipy contains additional routines for optimization, special functions, and so on. Both contain modules written in C and Fortran so that they're as fast as possible. Together, they give Python roughly the same capability that the Matlab program offers. (In fact, if you're an experienced Matlab user, there a guide to Numpy for Matlab users just for you.)
First off, it is important to import the library. Again, we did this at the very beginning of this whole jupyter notebook, but here it is in case you've jumped straight here without running the first code line:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Note that range(n) starts at 0 and gives the sequential list of integers less than n. If you want to start at a different number, use range(start,stop)
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Note that we have defined a few new things you haven't seen before: a break statement, that allows us to exit a for loop if some conditions are met, and an exception statement, that causes the interpreter to exit with an error message. For example:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Notice in either case you are not choosing the ordering -- it is automagically grouped alphabetically.
The len() command works on both tuples and dictionaries:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Of course we can combine if statements with for loops, to make a snippet that is almost interesting:
Can you write Python code for it?

for day in days_of_the_week:
    statement = "On " + day + ":"
    print (statement)
    if day == "Sunday":
        print ("   Sleep in")
    elif day == "Saturday":
        print ("   Do chores")
    else:
        print ("   Go to work")


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Or we can combine the two, making something like a semiconductor quantum well with a top gate:
Can you write Python code for it?

def tri_finite(x): return finite_well(x)+triangular(x,F=0.025)

tfw = Schrod1d(tri_finite,start=0,end=30,npts=100)
tfw.plot()


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Or, more elegantly:
Can you write Python code for it?

rands = np.random.rand(100)

plt.plot(rands,'o')


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Python lists, like C, but unlike Fortran, use 0 as the index of the first element of a list. Thus, in this example, the 0 element is "Sunday", 1 is "Monday", and so on. If you need to access the nth element from the end of the list, you can use a negative index. For example, the -1 element of a list is the last element:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Recursion can be very elegant, and can lead to very simple programs.
1.10 Two More Data Structures: Tuples and Dictionaries
Before we end the Python overview, I wanted to touch on two more data structures that are very useful (and thus very common) in Python programs.
A tuple is a sequence object like a list or a string. It's constructed by grouping a sequence of objects together with commas, either without brackets, or with parentheses:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Since strings are sequences, you can also do this to them:
Can you write Python code for it?

day = "Saturday"
abbreviation = day[:3]
print(abbreviation)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Since the data is now in a list of lines, we can iterate over it, splitting up data where we see a comma:
Can you write Python code for it?

for line in lines:
    # do something with each line
    words = line.split(",")


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Something that confuses Matlab users is that the times (*) operator give element-wise multiplication rather than matrix multiplication:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Splitting is a big concept in text processing. We used splitlines() here, and next we'll use the more general .split(",") function below to split each line into comma-delimited words.
We now want to do three things:

Skip over the lines that don't carry any information
Break apart each line that does carry information and grab the pieces we want
Turn the resulting data into something that we can plot.

To break apart each line, we will use .split(","). Let's see what it does to one of the lines:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Table of Contents

0. Preliminary
0.1 Why Python?
0.2 What You Need to Install

1. Python Overview
1.1 Using Python as a Calculator
1.2 Strings
1.3 Lists
1.4 Iteration, Indentation, and Blocks
1.5 Slicing
1.6 Booleans and Truth Testing
1.7 Code Example: The Fibonacci Sequence
1.8 Functions
1.9 Recursion and Factorials
1.10 Two More Data Structures: Tuples and Dictionaries
1.11 Plotting with Matplotlib
1.12 Conclusion of the Python Overview

2. Numpy and Scipy
2.1 Making vectors and matrices
2.2 Linspace, matrix functions, and plotting
2.3 Matrix operations
2.4 Matrix Solvers
2.5 Example: Finite Differences
2.6 One-Dimensional Harmonic Oscillator using Finite Difference
2.7 Special Functions
2.8 Least squares fitting
2.9 Monte Carlo, random numbers, and computing $\pi$
2.10 Numerical Integration
2.11 Fast Fourier Transform and Signal Processing

3. Intermediate Python
3.1 Parsing data output
3.2 Reading in data files
3.3 More Sophisticated String Formatting and Processing
3.4 Optional arguments of a function
3.5 List Comprehensions and Generators
3.6 Factory Functions
3.7 Serialization: Save it for later
3.8 Functional programming
3.9 Object Oriented Programming

4. Speeding Python: Timeit, Profiling, Cython, SWIG, and PyPy
4.1 Timeit
4.2 Profiling
4.3 Other Ways to Speed Python
4.4 Fun: Finding Primes

5. References
6. Acknowledgements

0. Preliminary
0.1 Why Python?
Python is the programming language of choice for many scientists to a large degree because it offers a great deal of power to analyze and model scientific data with relatively little overhead in terms of learning, installation or development time. It is a language you can pick up in a weekend, and use for the rest of your life.
The Python Tutorial is a great place to start getting a feel for the language. To complement this material, I taught a Python Short Course years ago to a group of computational chemists during a time that I was worried the field was moving too much in the direction of using canned software rather than developing one's...
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
The Python Tutorial has more on using Python as an interactive shell. The IPython tutorial makes a nice complement to this, since IPython has a much more sophisticated iteractive shell.
1.2 Strings
Strings are lists of printable characters, and can be defined using either single quotes
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
The agreement is almost exact.
We can use the subplot command to put multiple comparisons in different panes on a single plot (run %matplotlib qt on a separate line first to plot in a separate window):
Can you write Python code for it?

phase_correction = [-1,1,1,-1,-1,1]
for i in range(6):
    plt.subplot(2,3,i+1)
    plt.plot(x,ho_evec(x,i,1,1),label="Analytic")
    plt.plot(x,phase_correction[i]*U[:,i]/np.sqrt(h),label="Numeric")


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
The factorial function grows much faster. In fact, you can't even see the Fibonacci sequence. It's not entirely surprising: a function where we multiply by n each iteration is bound to grow faster than one where we add (roughly) n each iteration.
Let's plot these on a semilog plot so we can see them both a little more clearly:
Can you write Python code for it?

plt.semilogy(facts,label="factorial")
plt.semilogy(fibs,label="Fibonacci")
plt.xlabel("n")
plt.legend()


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
The first argument is a tuple containing the shape of the matrix, and the second is the data type argument, which follows the same conventions as in the array command. Thus, you can make row vectors:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
The idea behind the program is that the ratio of the area of the unit circle to the square that inscribes it is $\pi/4$, so by counting the fraction of the random points in the square that are inside the circle, we get increasingly good estimates to $\pi$.
The above code uses some higher level Numpy tricks to compute the radius of each point in a single line, to count how many radii are below one in a single line, and to filter the x,y points based on their radii. To be honest, I rarely write code like this: I find some of these Numpy tricks a little too cute to remember them, and I'm more likely to use a list comprehension (see below) to filter the points I want, since I can remember that.
As methods of computing $\pi$ go, this is among the worst. A much better method is to use Leibniz's expansion of arctan(1):
$$\frac{\pi}{4} = \sum_k \frac{(-1)^k}{2*k+1}$$
Can you write Python code for it?

n = 100
total = 0
for k in range(n):
    total += pow(-1,k)/(2*k+1.0)
print(4*total)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
The index in a dictionary is called the key, and the corresponding dictionary entry is the value. A dictionary can use (almost) anything as the key. Whereas lists are formed with square brackets [], dictionaries use curly brackets {}:
Can you write Python code for it?

ages = {"Rick": 46, "Bob": 86, "Fred": 21}

print("Rick's age is ",ages["Rick"])


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
The init() function specifies what operations go on when the object is created. The self argument is the object itself, and we don't pass it in. The only required argument is the function that defines the QM potential. We can also specify additional arguments that define the numerical grid that we're going to use for the calculation.
For example, to do an infinite square well potential, we have a function that is 0 everywhere. We don't have to specify the barriers, since we'll only define the potential in the well, which means that it can't be defined anywhere else.
Can you write Python code for it?

square_well = Schrod1d(lambda x: 0*x,m=10)
square_well.plot(4,titlestring="Square Well Potential")


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
The lists created above with range have a step of 1 between elements. You can also give a fixed step size via a third command:
Can you write Python code for it?

evens = list(range(0,20,2))
evens

evens[3]


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
The little % sign that we have in front of the timeit call is an example of an IPython magic function, which we don't have time to go into here, but it's just some little extra mojo that IPython adds to the functions to make it run better in the IPython environment. You can read more about it in the IPython tutorial.
In any case, the timeit function runs 3 loops, and tells us that it took on the average of 583 ns to compute 20!. In contrast:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
The range() command is a convenient way to make sequential lists of numbers:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
The range() command is particularly useful with the for statement to execute loops of a specified length:
Can you write Python code for it?

for i in range(20):
    print("The square of ",i," is ",i*i)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
The real value in a language like Python is that it makes it easy to take additional steps to analyze data in this fashion, which means you are thinking more about your data, and are more likely to see important patterns.
3.2 Reading in data files
Let's take a look at a perhaps easier approach to a common problem -- you have a data file with some header info and comma-delimited values and you want the data so you can start doing stuff with it. Let's use numpy's genfromtxt()
Can you write Python code for it?

filename= 'DS0004.csv'
data = np.genfromtxt(filename,delimiter=',',skip_header=17 )
x_values = data[:,0]
y_values = data[:,1]

plt.plot(x_values, y_values)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
There are a number of routines to compute eigenvalues and eigenvectors

eigvals returns the eigenvalues of a matrix
eigvalsh returns the eigenvalues of a Hermitian matrix
eig returns the eigenvalues and eigenvectors of a matrix
eigh returns the eigenvalues and eigenvectors of a Hermitian matrix.
Can you write Python code for it?

A = np.array([[13,-4],[-4,7]],'d')
np.linalg.eigvalsh(A)

np.linalg.eigh(A)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
There are additional signal processing routines in Scipy (e.g. splines, filtering) that you can read about here.
3. Intermediate Python
3.1 Parsing data output
As more and more of our day-to-day work is being done on and through computers, we increasingly have output that one program writes, often in a text file, that we need to analyze in one way or another, and potentially feed that output into another file.
Suppose we have the following output in CSV (comma separated values) format, a format that originally came from Microsoft Excel, and is increasingly used as a data interchange format in big data applications. How would we parse that?
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
There are many more things you can do with Matplotlib. We'll be looking at some of them in the sections to come. In the meantime, if you want an idea of the different things you can do, look at the Matplotlib Gallery. Rob Johansson's IPython notebook Introduction to Matplotlib is also particularly good.
1.12 Conclusion of the Python Overview
There is, of course, much more to the language than we've covered here. I've tried to keep this brief enough so that you can jump in and start using Python to simplify your life and work. My own experience in learning new things is that the information doesn't "stick" unless you try and use it for something in real life.
You will no doubt need to learn more as you go. I've listed several other good references, including the Python Tutorial and Learn Python the Hard Way. Additionally, now is a good time to start familiarizing yourself with the Python Documentation, and, in particular, the Python Language Reference.
Tim Peters, one of the earliest and most prolific Python contributors, wrote the "Zen of Python", which can be accessed via the "import this" command:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
There's a lot of philosophy behind object oriented programming. Since I'm trying to focus on just the basics here, I won't go into them, but the internet is full of lots of resources on OO programming and theory. The best of this is contained in the Design Patterns book, which I highly recommend.
4. Speeding Python: Timeit, Profiling, Cython, SWIG, and PyPy
The first rule of speeding up your code is not to do it at all. As Donald Knuth said:

"We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil."

The second rule of speeding up your code is to only do it if you really think you need to do it. Python has two tools to help with this process: a timing program called timeit, and a very good code profiler. We will discuss both of these tools in this section, as well as techniques to use to speed up your code once you know it's too slow.
4.1 Timeit
timeit helps determine which of two similar routines is faster. Recall that some time ago we wrote a factorial routine, but also pointed out that Python had its own routine built into the math module. Is there any difference in the speed of the two? timeit helps us determine this. For example, timeit tells how long each method takes:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
There's also a convenient way to create dictionaries without having to quote the keys.
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
There's also a diag() function that takes a list or a vector and puts it along the diagonal of a square matrix.
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
These are useful building blocks for functional programming.
The lambda operator allows us to build anonymous functions, which are simply functions that aren't defined by a normal def statement with a name. For example, a function that doubles the input is:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
This code snippet goes through each element of the list called days_of_the_week and assigns it to the variable day. It then executes everything in the indented block (in this case only one line of code, the print statement) using those variable assignments. When the program has gone through every element of the list, it exists the block.
(Almost) every programming language defines blocks of code in some way. In Fortran, one uses END statements (ENDDO, ENDIF, etc.) to define code blocks. In C, C++, and Perl, one uses curly braces {} to define these blocks.
Python uses a colon (":"), followed by indentation level to define code blocks. Everything at a higher level of indentation is taken to be in the same block. In the above example the block was only a single line, but we could have had longer blocks as well:
Can you write Python code for it?

for day in days_of_the_week:
    statement = "Today is " + day
    print(statement)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
This data looks more Gaussian than exponential. If we wanted to, we could use polyfit for this as well, but let's use the curve_fit function from Scipy, which can fit to arbitrary functions. You can learn more using help(curve_fit).
First define a general Gaussian function to fit to.
Can you write Python code for it?

def gauss(x,A,a): return A*np.exp(a*x**2)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
This gives exactly the same result if we don't specify anything:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
This is a giant string. If we use splitlines(), we see that a list is created where line gets separated into a string:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
This is a very bad geometry for a water molecule, and it would be silly to run so many geometry optimizations of structures that are guaranteed to converge to the same single geometry, but you get the idea of how you can run vast numbers of simulations with a technique like this.
We used the enumerate function to loop over both the indices and the items of a sequence, which is valuable when you want a clean way of getting both. enumerate is roughly equivalent to:
Can you write Python code for it?

def my_enumerate(seq):
    l = []
    for i in range(len(seq)):
        l.append((i,seq[i]))
    return l
my_enumerate(oxygen_xy_coords)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
This is clearly what we want.
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
This is something of an advanced topic, but ordinary data types have boolean values associated with them, and, indeed, in early versions of Python there was not a separate boolean object. Essentially, anything that was a 0 value (the integer or floating point 0, an empty string "", or an empty list []) was False, and everything else was true. You can see the boolean value of any data object using the bool() function.
Can you write Python code for it?

bool(1)

bool(0)

bool(["This "," is "," a "," list"])


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
This is where profiling can be useful. Our code now runs 3x faster by making trivial changes. We wouldn't have thought to look in these places had we not had access to easy profiling. Imagine what you would find in more complicated programs.
4.3 Other Ways to Speed Python
When we compared the fact and factorial functions, above, we noted that C routines are often faster because they're more streamlined. Once we've determined that one routine is a bottleneck for the performance of a program, we can replace it with a faster version by writing it in C. This is called extending Python, and there's a good section in the standard documents. This can be a tedious process if you have many different routines to convert. Fortunately, there are several other options.
Swig (the simplified wrapper and interface generator) is a method to generate binding not only for Python but also for Matlab, Perl, Ruby, and other scripting languages. Swig can scan the header files of a C project and generate Python binding for it. Using Swig is substantially easier than writing the routines in C.
Cython is a C-extension language. You can start by compiling a Python routine into a shared object libraries that can be imported into faster versions of the routines. You can then add additional static typing and make other restrictions to further speed the code. Cython is generally easier than using Swig.
PyPy is the easiest way of obtaining fast code. PyPy compiles Python to a subset of the Python language called RPython that can be efficiently compiled and optimized. Over a wide range of tests, PyPy is roughly 6 times faster than the standard Python Distribution.
4.4 Fun: Finding Primes
Project Euler is a site where programming puzzles are posed that might have interested Euler. Problem 7 asks the question:

By listing the first six prime numbers: 2, 3, 5, 7, 11, and 13, we can see that the 6th prime is 13.
What is the 10,001st prime number?

To solve this we need a very long list of prime numbers. First we'll make a function that uses the...
Can you write Python code for it?

def primes(n):
    """\
    From python cookbook, returns a list of prime numbers from 2 to < n

    >>> primes(2)
    [2]
    >>> primes(10)
    [2, 3, 5, 7]
    """
    if n==2: return [2]
    elif n<2: return []
    s=list(range(3,n+2,2))
    mroot = n ** 0.5
    half=(n+1)/2-1
    i=0
    m=3
    while m <= mroot:
        if s[i]:
            j=int((m*m-3)/2)
            s[j]=0
            while j<half:
                s[j]=0
                j+=m
        i=i+1
        m=2*i+3
    return [2]+[x for x in s if x]

number_to_try = 1000000
list_of_primes = primes(number_to_try)
print(list_of_primes[10001])


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
To build matrices, you can either use the array command with lists of lists:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Tuples are useful anytime you want to group different pieces of data together in an object, but don't want to create a full-fledged class (see below) for them. For example, let's say you want the Cartesian coordinates of some objects in your program. Tuples are a good way to do this:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
We can check the distribution by using the histogram feature, as shown on the help page for numpy.random.normal:
Can you write Python code for it?

count, bins, ignored = plt.hist(s, 30, normed=True)

plt.plot(bins, 1/(sigma * np.sqrt(2 * np.pi)) *
             np.exp( - (bins - mu)**2 / (2 * sigma**2) ),
             linewidth=2, color='r')


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
We can similarly redefine the Harmonic Oscillator potential.
Can you write Python code for it?

ho = Schrod1d(lambda x: x**2,start=-3,end=3)
ho.plot(6,titlestring="Harmonic Oscillator")


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
We can use reduce to define an analogous prod function:
Can you write Python code for it?

from functools import reduce
def prod(l): return reduce(mul,l)
prod([1,2,3,4,5])


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
We've introduced a several new features here. First, note that the function itself is defined as a code block (a colon followed by an indented block). This is the standard way that Python delimits things. Next, note that the first line of the function is a single string. This is called a docstring, and is a special kind of comment that is often available to people using the function through the python command line:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
We've made a couple of hacks here to get the orbitals the way we want them. First, I inserted a -1 factor before the wave functions, to fix the phase of the lowest state. The phase (sign) of a quantum wave function doesn't hold any information, only the square of the wave function does, so this doesn't really change anything.
But the eigenfunctions as we generate them aren't properly normalized. The reason is that finite difference isn't a real basis in the quantum mechanical sense. It's a basis of Dirac δ functions at each point; we interpret the space betwen the points as being "filled" by the wave function, but the finite difference basis only has the solution being at the points themselves. We can fix this by dividing the eigenfunctions of our finite difference Hamiltonian by the square root of the spacing, and this gives properly normalized functions.
2.7 Special Functions
The solutions to the Harmonic Oscillator are supposed to be Hermite polynomials. The Wikipedia page has the HO states given by
$$\psi_n(x) = \frac{1}{\sqrt{2^n n!}}
\left(\frac{m\omega}{\pi\hbar}\right)^{1/4}
\exp\left(-\frac{m\omega x^2}{2\hbar}\right)
H_n\left(\sqrt{\frac{m\omega}{\hbar}}x\right)$$
Let's see whether they look like those. There are some special functions in the Numpy library, and some more in Scipy. Hermite Polynomials are in Numpy:
Can you write Python code for it?

from numpy.polynomial.hermite import Hermite
def ho_evec(x,n,m,ohm):
    vec = [0]*9
    vec[n] = 1
    Hn = Hermite(vec)
    return (1/np.sqrt(2**n*factorial(n)))*pow(m*ohm/np.pi,0.25)*np.exp(-0.5*m*ohm*x**2)*Hn(x*np.sqrt(m*ohm))


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
What the keyword argument construction does is to take any additional keyword arguments (i.e. arguments specified by name, like "endpoint=False"), and stick them into a dictionary called "kwargs" (you can call it anything you like, but it has to be preceded by two stars). You can then grab items out of the dictionary using the get command, which also lets you specify a default value. I realize it takes a little getting used to, but it is a common construction in Python code, and you should be able to recognize it.
There's an analogous *args that dumps any additional arguments into a list called "args". Think about the range function: it can take one (the endpoint), two (starting and ending points), or three (starting, ending, and step) arguments. How would we define this?
Can you write Python code for it?

def my_range(*args):
    start = 0
    step = 1
    if len(args) == 1:
        end = args[0]
    elif len(args) == 2:
        start,end = args
    elif len(args) == 3:
        start,end,step = args
    else:
        raise Exception("Unable to parse arguments")
    v = []
    value = start
    while True:
        v.append(value)
        value += step
        if value > end: break
    return v


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
You can access members of the list using the index of that item:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
You can add additional items to the list using the .append() command:
Can you write Python code for it?

languages = ["Fortran","C","C++"]
languages.append("Python")
print(languages)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
You can also pass in keywords to exclude the endpoint:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
You can also put some boolean testing into the construct:
Can you write Python code for it?

odds = [i for i in range(20) if i%2==1]
odds


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
You can define variables using the equals (=) sign:
Can you write Python code for it?

width = 20
length = 30
area = length*width
area


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
You can name a variable almost anything you want. It needs to start with an alphabetical character or "_", can contain alphanumeric charcters plus underscores ("_"). Certain words, however, are reserved for the language:
and, as, assert, break, class, continue, def, del, elif, else, except,
exec, finally, for, from, global, if, import, in, is, lambda, not, or,
pass, print, raise, return, try, while, with, yield

Trying to define a variable using one of these will result in a syntax error:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
Your data sits in something that looks like a Python dictionary, and in a single line of code, you can load it into a Python dictionary for use later.
In the same way, you can, with a single line of code, put a bunch of variables into a dictionary, and then output to a file using json:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
and we have a bunch of methods we can use on the object:
Can you write Python code for it?

mystring.split()

mystring.startswith('Hi')

len(mystring)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
and you need to define it:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
as well as a ones command.
2.2 Linspace, matrix functions, and plotting
The linspace command makes a linear array of points from a starting to an ending value.
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
as well as matrix-vector products.
There are determinant, inverse, and transpose functions that act as you would suppose. Transpose can be abbreviated with ".T" at the end of a matrix object:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
as well as when you add two matrices together. (However, the matrices have to be the same shape.)
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
dot can also do dot products (duh!):
Can you write Python code for it?

v = np.array([3,4],'d')
np.sqrt(np.dot(v,v))


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
lambda is particularly convenient (as we'll see below) in passing simple functions as arguments to other functions.
map is a way to repeatedly apply a function to a list:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
linspace is an easy way to make coordinates for plotting. Functions in the numpy library (all of which are imported into IPython notebook) can act on an entire vector (or even a matrix) of points at once. Thus,
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
np.random.random() uses the Mersenne Twister algorithm, which is a highly regarded pseudorandom number generator. There are also functions to generate random integers, to randomly shuffle a list, and functions to pick random numbers from a particular distribution, like the normal distribution:
It is generally more efficient to generate a list of random numbers all at once, particularly if you're drawing from a non-uniform distribution. Numpy has functions to generate vectors and matrices of particular types of random distributions:
Can you write Python code for it?

mu, sigma = 0, 0.1 # mean and standard deviation
s=np.random.normal(mu, sigma,1000)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
or column vectors:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
or double quotes
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
or you can simply import the math library itself
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
print even converts some arguments to strings for us:
Can you write Python code for it?

a,b,c = 1,2,3
print("The variables are ",1,2,3)


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
reduce is a way to repeatedly apply a function to the first two items of the list. There already is a sum function in Python that is a reduction:
Can you write Python code for it?


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
the factorial function we wrote is about a factor of 10 slower. This is because the built-in factorial function is written in C code and called from Python, and the version we wrote is written in plain old Python. A Python program has a lot of stuff in it that make it nice to interact with, but all that friendliness slows down the code. In contrast, the C code is less friendly but more efficient. If you want speed with as little effort as possible, write your code in an easy to program language like Python, but dump the slow parts into a faster language like C, and call it from Python. We'll go through some tricks to do this in this section.
4.2 Profiling
Profiling complements what timeit does by splitting the overall timing into the time spent in each function. It can give us a better understanding of what our program is really spending its time on.
Suppose we want to create a list of even numbers. Our first effort yields this:
Can you write Python code for it?

def evens(n):
    "Return a list of even numbers below n"
    l = []
    for x in range(n):
        if x % 2 == 0:
            l.append(x)
    return l


I am working on the file "Crash Course v0.62.ipynb".
The first task is:
which is somewhat logically consistent with negative indices accessing the last elements of the list.
You can do:
Can you write Python code for it?

workdays = days_of_the_week[1:5]
print(workdays)


I am working on the file "CrashCourse_ExerciseAudio.ipynb".
The first task is:
Exercise 1
Exercise: Produce a feature matrix for the sound files using sliding window, and a corresponding label vector.
Hint: you can use np.full(n, x) to generate a vector [x,x,x,x,...] and np.vstack(l) to stack a list of vectors into a matrix.
Make the window size and overlap part of params (window_length and overlap) and write a function features, labels = make_features(processed, params)
Can you write Python code for it?

params['window_overlap'] = -1024
params['window_length'] = 256
features, labels = make_features(processed, params)
print features.shape


I am working on the file "CrashCourse_ExerciseAudio.ipynb".
The first task is:
Feature transform
The raw audio data isn't a great feature for classification. We can apply transformations to the feature vectors to improve the features. The Fourier transform is one way of doing that. To avoid artifacts, each section must have a window function applied to taper off the signal at the ends and avoid a large discontinuity.
Can you write Python code for it?

def transform_features(data):
    # window features and compute magnitude spectrum
    # try different window functions (e.g. Hann, Blackman-Harris)
    window = sig.hamming(features.shape[1])
    fft_features = np.abs(np.fft.fft(features * window))
    return fft_features

print transform_features(features).shape


I am working on the file "CrashCourse_ExerciseAudio.ipynb".
The first task is:
Preprocessing steps
Two things we should do in the pre-processing step:
1. normalise the data to 0-1 range
2. apply bandpass filtering to select frequencies we are interested in
Can you write Python code for it?

def bandpass(x, low, high, sample_rate):
    # scipy.signal.filtfilt applies a linear filter to data (*without* phase distortion)
    # scipy.signal.butter will design a linear Butterworth filter
    nyquist = sample_rate / 2
    b,a = sig.butter(4, [low/float(nyquist), high/float(nyquist)], btype="band")
    return sig.filtfilt(b,a,x)

def pre_process(sound_files, params):
    processed = []
    for sound_file in sound_files:
        normalised = sound_file / 32768.0
        p = bandpass(normalised, params["low_cutoff"], params["high_cutoff"], params["sample_rate"])
        processed.append(p)
    return processed


I am working on the file "CrashCourse_ExerciseAudio.ipynb".
The first task is:
Since these are just plain wave files, we can listen to the data using aplay:
Can you write Python code for it?

# play the first five seconds of these two files
!aplay stane_2.wav -d 5
!aplay stane_4.wav -d 5

# load each of the files into sound_files
sound_files = []
for texture in "12345":
    # load the wavefile
    fname = "stane_%s.wav" % texture
    sr, data = wavfile.read(fname)
    print "Loaded %s, %s samples at %dHz (%f seconds)" % (fname, len(data), sr, len(data)/float(sr))
    sound_files.append(data)


I am working on the file "CrashCourse_ExerciseAudio.ipynb".
The first task is:
Step 1: Loading the data
The first thing we need to do is to load the data. The data is in datasets/stane/ and consists of five wave files from scratching five different surfaces, each 60 seconds in length, 4Khz, 16 bit PCM.
Can you write Python code for it?


I am working on the file "CrashCourse_ExerciseAudio.ipynb".
The first task is:
Testing pre-processing
We can test this and check it working by plotting the time series and spectrogram before and after. We can create a quick function to plot this:
Can you write Python code for it?

def plot_second(x, params):
    one_second = params["sample_rate"]
    plot_section = x[:one_second]
    # generate time indices
    timebase = np.arange(len(plot_section)) / float(params["sample_rate"])
    plt.figure()
    plt.plot(timebase, plot_section)
    plt.ylabel("Amplitude")
    plt.xlabel("Time (s)")
    plt.figure()
    _ = plt.specgram(plot_section, NFFT=256, Fs=params["sample_rate"], noverlap=128, cmap='gist_heat')
    plt.ylabel("Freq (Hz)")
    plt.xlabel("Time (s)")

# test the filtering; these are example values only
params["low_cutoff"]=100
params["high_cutoff"]=1500
processed = pre_process(sound_files, params)

# plot the results
plot_second(sound_files[0], params)
plot_second(processed[0], params)


I am working on the file "CrashCourse_ExerciseAudio.ipynb".
The first task is:
This has loaded each of the wave files into sound_files[], one for each of our 5 classes. We must process this into fixed length feature vectors which we can feed to a classifier. This is the major "engineering" of the machine learning process -- good feature selection is essential to getting good performance.
It's important that we can change the parameters of the feature extraction and learning and be able to rerun the entire process in one go. We define a dictionary called params which will hold every adjustable parameter and a function called run_pipeline() which will run our entire pipeline. For now, it does nothing.
Can you write Python code for it?

params = {'sample_rate':4096,
         }

def run_pipeline(sound_files, params):
    # this is the outline of our pipeline
    pre_processed = pre_process(sound_files, params)
    features, targets = feature_extract(pre_processed, params)
    train, validate, test = split_features(features, targets, params)
    classifier = train_classifier(features, targets, params)
    evaluate(classifier, features, targets, params)


I am working on the file "CrashCourse_IntroIpython.ipynb".
The first task is:
If you hit SHIFT-ENTER on the above, you should see a plot.
It looks pretty noisy; perhaps there is some periodic structure. We can use the FFT to look at this. np.fft.fft() computes the http://en.wikipedia.org/wiki/Fourier_transform; we can compute the magnitude spectrum $|f(x)|$ by taking the absolute value, and discarding the symmetric half:
Can you write Python code for it?

fftd = np.fft.fft(collatzed)
# take absolute value
real_magnitude = np.abs(fftd)
# trim off symmetric part (note the slice syntax)
real_magnitude = real_magnitude[1:len(fftd)/2] # note that we drop the 0th element (DC)
fig = plt.figure() # make a new figure
ax = fig.add_subplot(111) # this just creates a new single blank axis
ax.plot(real_magnitude)   # and plots onto it (we can create multi-panel plots using add_subplot)


I am working on the file "CrashCourse_IntroIpython.ipynb".
The first task is:
Now try it in the cell below (e.g. entering collatz(331) and pressing SHIFT-ENTER should print 24).
Can you write Python code for it?


I am working on the file "CrashCourse_IntroIpython.ipynb".
The first task is:
Now we create an array of integers 1:n and plot it. Note the use of arange to create an array of integers, and the list comprehension [collatz(n) for n in ns], which applies the collatz function to each element of ns.
Can you write Python code for it?

ns = np.arange(1,500)   # generate n = [1,2,3,4,...]
collatzed = np.array([collatz(n) for n in ns]) # apply collatz(n) to each value and put it in a numpy array
plt.plot(ns, collatzed)  # plot the result


I am working on the file "CrashCourse_IntroIpython.ipynb".
The first task is:
OK, let's plot the graph of this function for various $n$. We'll use numpy to manipulate vectors of values, and matplotlib to plot the graph. First we must import them. In future workbooks this will already be done at the start, but we do it explicitly here for clarity.
Can you write Python code for it?

import numpy as np # np is the conventional short name for numpy
import matplotlib.pyplot as plt # and plt is the conventional name for matplotlib
import seaborn # all this does (in this case) is restyle matplotlib to use better layouts


I am working on the file "CrashCourse_IntroIpython.ipynb".
The first task is:
OK, this is more interesting. Let's show a normal fit to the distribution, using maximum likelihood estimation. scipy.stats has the tools we need to do this.
Can you write Python code for it?

mean, std = np.mean(collatzed), np.std(collatzed)
import scipy.stats as stats # we must import scipy.stats, as we've not used it yet

# np.linspace() linearly spaces points on a range: here 200 points spanning the distribution
pdf_range = np.linspace(np.min(collatzed), np.max(collatzed), 200)

# scipy.stats has many distribution functions, including normal (norm)
pdf = stats.norm.pdf(pdf_range, mean, std)
plt.hist(collatzed, bins=50, normed=True)
plt.plot(pdf_range, pdf, 'g', linewidth=3) # plot using thick green line


I am working on the file "CrashCourse_IntroIpython.ipynb".
The first task is:
Obviously, this distribution is non-Gaussian, but let's test to make sure. scipy.stats provides many statistical tools, including normality testing. scipy.stats.normaltest gives us a combination of D’Agostino and Pearson’s test.
Note: to see the documentation for normaltest, try clicking at the end of normaltest and press SHIFT-TAB to see the tooltip. Hit the ^ symbol to bring up the full help in a pane below. This works for any function.
Can you write Python code for it?

import scipy.stats as stats # we must import scipy.stats as we've not used it yet
k2, p = stats.normaltest(collatzed)
print p # p-value, testing if the distribution differs from the normal. p<0.05 suggests it is


I am working on the file "CrashCourse_IntroIpython.ipynb".
The first task is:
Some interesting structure, with big spikes, but the frequency approach isn't revealing much. Let's investigate the distribution of the variable. We can get a histogram with plt.hist().
Can you write Python code for it?

# normed forces the frequency axis to sum to 1
plt.hist(collatzed, bins=50, normed=True);


I am working on the file "CrashCourse_IntroIpython.ipynb".
The first task is:
That looks pretty unstructured. Let's try more numbers, and make the fft plot a function we can reuse later.
Can you write Python code for it?

ns = np.arange(1,10000)
collatzed = [collatz(n) for n in ns]

def fft_plot(x):
    """Plot the magnitude spectrum of x, showing only the real, positive-frequency
    portion, and excluding component 0 (DC). """
    fftd = np.fft.fft(x)
    # get absolute (magnitude spectrum)
    real_magnitude = np.abs(fftd)
    # chop off symmetric part
    real_magnitude = real_magnitude[1:len(fftd)/2]
    fig = plt.figure() # make a new figure
    ax = fig.add_subplot(111)
    ax.plot(real_magnitude)

fft_plot(collatzed)


I am working on the file "CrashCourse_IntroIpython.ipynb".
The first task is:
You can run python scripts using %run
%run myscript.py

and edit scripts with %edit
%edit myscript.py

If you invoke %edit with no argument, it will open an editor with a blank temporary file -- if you save and exit, IPython will load the contents into a cell and execute it
You can get previous cell inputs using _i (the last input) and _ for the last output
Previous values can be accessed via the In[] and Out[] arrays (see the In[n] at the left of each code cell -- you can use this directly)
print In[1]

Basic python
Let's compute the length of a path in the Collatz sequence. That is, we take a number $n$ and halve it if $n \equiv 0 \mod 2$, otherwise replace it with $3n+1$. We count the number of steps until we reach 1.
Remember that to execute the cell, and thus to define the function, select the cell and press SHIFT-ENTER.
Can you write Python code for it?

def collatz(n, steps=0):
    """Compute the number of steps to reach 1 in the Collatz sequence.
    Note the use of triple quotes to specify a docstring.

    Also note the use of a default parameter (steps) to count the number
    of recursive calls."""
    if n==1:
        return steps
    if n%2==0:
        return collatz(n/2, steps+1)
    else:
        return collatz(3*n+1, steps+1)


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
As the function approximates the data better, the generalisation performance drops. If we split the data randomly into two portions $X_1$ and $X_2$, and learn the KDE using only $X_1$ (training set) and then compute how likely $X_2$ (test set) is given that learned distribution, we can see this loss of generalisation performance.
Can you write Python code for it?

def split_data(x):
    # our data here is random and uncorrelated, so we can just split the array into two
    l = len(x)//2
    return x[:l], x[l:]

def learn_kde(x, width):
    return stats.kde.gaussian_kde(x,width)

def evaluate_kde(x, kde):
    # we can compute the log-likelihood by summing the log pdf evaluated at x
    return np.sum(kde.logpdf(x))

def test_kde(x, width):
    # split the data into two parts, train on one, and then test it on both of the splits
    x1,x2 = split_data(x)
    kde = learn_kde(x1, width)
    # return the train log-likelihood and test log-likelihood
    return evaluate_kde(x1, kde), evaluate_kde(x2, kde)

def plot_kde_lik(x):
    # plot test and train log-likelihood as a function of 1/sigma
    widths = np.linspace(0.05, 20, 100)
    trains = []
    tests = []
    # test a bunch of widths
    for width in widths:
        train, test = test_kde(x,1.0/width)
        trains.append(train)
        tests.append(test)

    # plot and label
    plt.plot(widths, trains)
    plt.plot(widths, tests)
    plt.xlabel("$\sigma^{-1}$")
    plt.ylabel("Log-likelihood")
    plt.legend(["Training", "Test"])


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Baselines
Let's return to our classification example. What is good performance? Or more particularly, what is bad performance -- what's the baseline? A very simple test of whether the classifier is learning anything useful is to simply randomly permute the targets so they no longer correspond to the inputs.
This estimates the chance performance of the classification process. It might be that you have a binary classification problem with exactly balanced datasets, so the baseline is 50%; but often the data is unbalanced and there are multiple classes. Permuting the targets is a very quick way to test the random performance.
Can you write Python code for it?

print sklearn.metrics.classification_report(sonar_test_labels, sonar_predicted_labels)

# fit an SVM to the randomised label data
svm = sklearn.svm.SVC(C=15, gamma=0.12)
svm.fit(sonar_train_features, random_sonar_train_labels)
# this should be about 0.5
print "Score: %f" % svm.score(sonar_test_features, random_sonar_test_labels)


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Boosting
Boosting is an alternative ensemble method which trains a weak classifier on a dataset, identifies samples it is performing poorly on, and trains another classifier to learn the poor samples, identifies samples the ensemble is still performing poorly on, and trains a classifer to learn those and so on.
The selection of the weak samples is usually done by weighting the samples rather than a binary inclusion/exclusion. The AdaBoost algorithm is a well-known example of this class, and can combine weak learners such as decision stumps (a decision tree with only one decision!) into effective classifiers.
Can you write Python code for it?

dec = sklearn.tree.DecisionTreeClassifier(max_depth=1)
dec.fit(sonar_train_features, sonar_train_labels)
boost_scores = []
for estimators in range(50):
    boosted_dec = sklearn.ensemble.AdaBoostClassifier(dec, n_estimators=estimators+1, algorithm='SAMME')
    boosted_dec.fit(sonar_train_features, sonar_train_labels)
    boost_scores.append(boosted_dec.score(sonar_test_features, sonar_test_labels))
plt.axhline(dec.score(sonar_test_features, sonar_test_labels))
plt.plot(np.arange(len(boost_scores))+1, boost_scores)
plt.xlabel('No. estimators')
plt.ylabel('Accuracy')


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Confusion matrices
Confusion matrices are effective tools for communicating where classifiers are going wrong: which labels are being confused with which? A confusion matrix shows the distribution of predicted labels for each true label as a matrix of size $l \times l$ for $l$ labels.
Perfect classification results in a confusion matrix with a single diagonal of 1s (every test example predicts the label to be the true label). This matrix can reveal classes which are poorly separated in multi-class problems.
Can you write Python code for it?

# we can print the confusion matrix
confusion_matrix =  sklearn.metrics.confusion_matrix(sonar_test_labels, sonar_predicted_labels).astype(np.float64)
# normalise so that sum(row)=1
confusion_matrix = (confusion_matrix.T / np.sum(confusion_matrix, axis=1)).T
# ipy_table.make_table just pretty prints the resulting matrix
ipy_table.make_table(confusion_matrix)


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Dimensional reduction
A very common unsuperised learning task is dimensional reduction; taking a dataset with a dimension of $d$ and reducing to a dimension of $d^\prime$ which is smaller than $d$ but retains as much of the useful information as possible.
This can be thought of as a form of lossy compression -- finding a "simpler" representation of the data which captures its essential properties. This of course depends upon what the "essential properties" that we want to keep are, but generally we want to reject noise and keep non-random structure. We find a subspace that captures the meaningful variation of a dataset.
One way of viewing this process is finding latent variables; variables we did not directly observe, but which are simple explanations of the ones we did observe. For example, if we measure a large number of weather measurements (rainfall, pressure, humidity, windspeed), these might be a very redundant representation of a few simple variables (e.g. is there a storm?). If features correlate or cluster in the measured data we can learn this structure even without knowing training labels.
Visualisation
Dimensional reduction has an obvious role in visualisation, reducing datasets to 2D or 3D representations that humans can work with. It often possible to find a mapping from high to low dimensions which maintains salient structure; e.g. one in which different classes of data are visually separated.
Pre-training
Another important use for dimensional reduction is in learning a simplified feature vector for subsequent supervised learning. If much of the irrelevant variation can be "factored out" by the dimensional reduction process, then the decision problem for the supervised learning algorithm becomes easier. This is known as pre-training.
This process can also dramatically speed up some supervised learning algorithms which have a time complexity which scales badly with $d$ (e.g. $O(d^2)$ or even $(O(d^3)$).
Principal component analysis
One very simple method of dimensional reduction is principal...
Can you write Python code for it?

digits = sklearn.datasets.load_digits()
digit_data = digits.data

# plot a single digit data element
def show_digit(d):
    plt.imshow(d.reshape(8,8), cmap='gray', interpolation='nearest')
    plt.axis('off')
    plt.figure()

# show a couple of raw digits
show_digit(digit_data[0])
show_digit(digit_data[400])

# apply principal component analysis
pca = sklearn.decomposition.PCA(n_components=2).fit(digit_data)
digits_2d = pca.transform(digit_data)

# plot each digit with a different color
plt.scatter(digits_2d[:,0], digits_2d[:,1], c=digits.target, cmap='jet')


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Evaluating machine learning
We need to be able to quantify how well our learning algorithms perform on predicting unseen data given the model that has been learned. This involves testing on data that was not presented to the learning algorithm during the training phase.
Classifiers
One common problem is to measure the performance of a classifier; i.e. how well the learned function predicts labels for unseen data. An obvious metric is accuracy, the ratio of correctly classified examples to the total number of examples.
Can you write Python code for it?

%cd datasets/uci
%ls

# load the "Sonar" data set [https://archive.ics.uci.edu/ml/datasets/Connectionist+Bench+(Sonar,+Mines+vs.+Rocks)]
# This is a set of 60 sonar measurements, from bouncing sonar waves off either rocks ("R") or mines ("M")
# The problem is to tell the mines apart from the rocks
sonar_data = pd.read_csv("sonar.all-data")

# separate features
sonar_features = np.array(sonar_data)[:,0:60].astype(np.float64)

# we use label_binarize to convert "M" and "R" labels into {0,1}
# the ravel() just flattens the resulting 1D matrix into a vector
sonar_labels = sklearn.preprocessing.label_binarize(np.array(sonar_data)[:,60], classes=['M', 'R'])[:,0]

# split into a train and test section, holding out 20% (0.2) of the data for testing
sonar_train_features, sonar_test_features, sonar_train_labels, sonar_test_labels = sklearn.cross_validation.train_test_split(
    sonar_features, sonar_labels, test_size=0.3, random_state=0)

# fit an SVM with the default parameters
svm = sklearn.svm.SVC(C=1)
svm.fit(sonar_train_features, sonar_train_labels)
print "Score: %f" % svm.score(sonar_test_features, sonar_test_labels)

sonar_predicted_labels = svm.predict(sonar_test_features)


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Feature selection
Often the number of features in the "raw" measurements are very large (e.g. in EEG, we might measure time series at 64 electrodes, each with 32 spectral features). But this raw representation is very redundant, and the redundancy can both make training very slow and can cause the classifier to learn irrelevant variations in high-dimensional space that are essentially noise.
There are algorithms which can perform feature selection, identifying an optimal subset of features that maximise test performance. A simple algorithm is just to randomly drop features and test the resulting classifier, but this is not efficient. Binary search over features can be used for selection
Other algorithms can form linear combination of features, essentially performing dimensional reduction before supervised learning begins.
Can you write Python code for it?

# load the "Sonar" data set [https://archive.ics.uci.edu/ml/datasets/Connectionist+Bench+(Sonar,+Mines+vs.+Rocks)]
# This is a set of 60 sonar measurements, from bouncing sonar waves off either rocks ("R") or mines ("M")
# The problem is to tell the mines apart from the rocks
%cd datasets/uci

sonar_data = pd.read_csv("sonar.all-data")

# separate features
sonar_features = np.array(sonar_data)[:,0:60].astype(np.float64)

# we use label_binarize to convert "M" and "R" labels into {0,1}
# the ravel() just flattens the resulting 1D matrix into a vector
sonar_labels = sklearn.preprocessing.label_binarize(np.array(sonar_data)[:,60], classes=['M', 'R']).ravel()

# split into a train and test section, holding out 20% (0.2) of the data for testing
sonar_train_features, sonar_test_features, sonar_train_labels, sonar_test_labels = sklearn.cross_validation.train_test_split(
    sonar_features, sonar_labels, test_size=0.3, random_state=0)

# choose 10 best features
k_best = sklearn.feature_selection.SelectKBest(k=10)
sonar_k_features = k_best.fit_transform(sonar_train_features, sonar_train_labels)

# fit an SVM with the default parameters
svm = sklearn.svm.SVC(C=10, gamma=0.1)
svm.fit(sonar_k_features, sonar_train_labels)
print "10 feature score: %f" % svm.score(k_best.transform(sonar_test_features), sonar_test_labels)

svm = sklearn.svm.SVC(C=10, gamma=0.1)
svm.fit(sonar_train_features, sonar_train_labels)
print "60 feature score: %f" % svm.score(sonar_test_features, sonar_test_labels)


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Feature vectors
In almost all machine learning contexts, we predict outputs given a fixed length set of features; the input space a fixed dimension $d$. Each of those features is usually (but not always) continuous-valued.
This means that any data we want to process must be converted to fixed length real-valued vectors. Some data is already in this form (e.g. a set of weather measurements at a given time: pressure, humidity, temperature). In other cases, it is more awkward to find a space in which to represent the data.
For example, we often want to make predictions based on time series; a set of measurements of the same variable (or variables) made repeatedly over time. This includes problems like speech recognition, where we have acoustic measurements made over a period of time, or predicting the chance of a car's brakes failing given a the time history of the speed of the car and the reflectivity of the tires (as a measure water-loggedness).
In these cases we have regularly sampled data (i.e. time between measurements is constant).
Windowing
One general solution to this time series problem is to use a delay embedding -- a fixed length sequence of previous measurements. For example the measurements $[x_{t=t}, x_{t=t-1}, x_{t=t-2}, \dots, x_{t=t-d}]$ might make up the feature vector. A general delay embedding does not need to capture equally spaced delayed elements: it could capture $x_{t}, x_{t-5}, x_{t-10}, x_{t-20}, x_{t-40}$. Choosing the spacing of these captures can be tricky. If we just use $d$ consecutive measurements, this process is known as windowing, because we chop up the data into fixed length windows by "sliding" a time window along the data.
These windows can overlap, which increases the size of the training set, but excessive overlapping can capture lots of redundant features. Balancing the size of the windows (and thus the feature vector size $d$) and the amount of overlap is a matter of experimentation and domain knowledge.
Can you write Python code for it?

# windowing example

t = np.linspace(0,20,1000)
x = (1+0.1*t)*np.cos(t) + np.cos(t*8+0.2)*0.4 + np.cos(t*3+3)*0.1 + np.random.normal(0,0.1,t.shape)

# show windows
def plot_windows(t,x,window_size,overlap):
    plt.figure()
    plt.plot(t,x)
    end = window_size
    while end<len(t):
        plt.fill_between(t[end-window_size:end]+0.01, -4,4, facecolor='r', edgecolor='k',alpha=0.1)
        plt.fill_between(t[end-window_size:end]+0.01, -4,4, facecolor='none', edgecolor='k',alpha=0.5)

        end += window_size-overlap

# d = 100
plot_windows(t,x,100,0)
plt.title("$d=100$")
# d = 500
plot_windows(t,x,400,0)
plt.title("$d=500$")
# d = 100, overlap = 50
plot_windows(t,x,100,50)
plt.title("$d=100, k=50$")
# d = 400, overlap = 350
plot_windows(t,x,400,350)
plt.title("$d=400, k=350$")

# skipping
# d = 100, overlap = -100
plot_windows(t,x,100, -150)
plt.title("$d=100, k=-150$")


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Features, labels, data formats
We've worked so far with real valued data -- i.e. where the features $X$ take on continuous values. Regression involves learning real-valued output $y$ and classification deals with categorical outputs $y$.
Categorical data must be treated differently from real data because categorical variables are discrete and have no inherent ordering or scaling. Categorical variables encode distinct classes, e.g. "car", "bus", "train".
Ordinal variables are variables that have a distinct set of discrete values with an ordering (e.g. responses on a Likert scale on a questionnaire) but do not have a constant scaling (the difference between 1 and 2 might be totally different from 8 and 9).
There are specific algorithms to learn function with categorical and ordinal inputs and outputs. In general, algorithms for continuous data are not suitable directly.
One-hot-encoding
One way of converting a categorical value to a real valued vector is to use one hot encoding. This replaces each category from $k$ categories with a binary vector with a single 1, producing $k$ distinct vectors. For example, if we had categories: "bird", "mammal", "reptile", "amphibian", we might create target vectors:
$$[1,0,0,0],\
[0,1,0,0],\
[0,0,1,0],\
[0,0,0,1]$$
These lie at the vertices of a simplex (the high-dimensional analogue of a triangle) and are easy to separate. There are other approaches which use error-correcting codes to increase efficiency when $k$ is large, to avoid having very long sparse vectors.
Can you write Python code for it?

def one_hot_encode(categories):
    """Return a dictionary with binary one-hot vector for each category"""
    n = len(categories)
    one_hot = {}
    for i in range(n):
        binary_vector = [0 for k in range(n)]
        binary_vector[i] = 1
        one_hot[categories[i]] = binary_vector
    return one_hot

print one_hot_encode([1,2,3,4,5,6])
print one_hot_encode(["cat", "dog", "horse", "cow", "sheep"])


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Great! 95% accuracy!
Let's try it with random labels:
Can you write Python code for it?

wilt_svm = sklearn.svm.SVC(C=1)
wilt_svm.fit(wilt_train_features, np.random.permutation(wilt_train_labels))
print "Score: %f" % wilt_svm.score(wilt_test_features, np.random.permutation(wilt_test_labels))


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Grid search
One way of finding the best hyperparameters to use is grid search. A grid of every possible combination of hyperparameters in some range is created, and the performance of the algorithm is evaluated at each step. The best grid combination can then be used to train the final model.
Enhancements include adaptive grid search, where a coarse grid is searched, and then a finer grid is searched in the vicinity of the best point from that search, and so on. This works well when the hyperparameter space is smooth and performance doesn't vary much with small changes in hyperparameters. If the perfomance is not smooth, an adaptive search may miss narrow peaks in the search space.
Randomised grid search is another approach, where points are randomly selected from a distribution of hyperparameter values. This can be useful when it may be possible to terminate early -- random search may find good points while standard grid search spends enormous amounts of time in very poor parts of the space.
In general, grid search is only practical for small numbers of hyperparameters, as the increasing dimension of the hyperparameter space makes the search time grow exponentially.
Can you write Python code for it?

# let's search for the best parameters of C and gamma in an SVM
# we create a grid of parameter values, log-spaced across a wide range
C_vals = np.logspace(-2,8,10)
gamma_vals = np.logspace(-4,3,10)

# 10x10 values of C and gamma; 10 fold cross validation for each
grid_cv = sklearn.grid_search.GridSearchCV(svm, {'C':C_vals, 'gamma':gamma_vals}, cv=10)
grid_cv.fit(sonar_train_features, sonar_train_labels)

# print the optimal best results
print grid_cv.best_params_
print grid_cv.best_score_

# fit an SVM using the optimal parameters found
svm = sklearn.svm.SVC(C=grid_cv.best_params_['C'], gamma=grid_cv.best_params_['gamma'])
svm.fit(sonar_train_features, sonar_train_labels)
print "Score: %f" % svm.score(sonar_test_features, sonar_test_labels)


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
However, if the features are transformed in a way that "unravels" the data so that a hyperplane can separate them, then the model can learn the function correctly.
K Nearest Neighbors (KNN)
A very simple way of classifying data is to predict class labels by looking at the class labels of the datapoints nearest to the  point we are trying to predict. By finding the nearest $k$ neighbours in the original dataset $X$ of a new point $x^\prime$, we can predict a class label $y$ by majority voting from the class labels of those $k$ neighbours (i.e. the label which occurs most often among those neighbours).
kNN can deal with multi-class problems without any changes. It can also be used for regression tasks by interpolating the output of the neighbours (e.g. linearly) rather than using majority voting.
This is a very simple to implement algorithm, and it works relatively well for many cases. It is a good baseline for performance -- kNN has few parameters to tune, and is easy to apply, although can be computationally expensive for large datasets.  More sophisticated algorithms can do much better, but their behaviour can be hard to predict, and a sensible check if a more sophisticated algortithm is doing something useful is to compare it with the KNN performance.
Can you write Python code for it?

# k neareset neigbbours classification example, again with the Iris dataset

def plot_knn(knn):
    # plot the original data
    plt.figure()
    # predict output value across the space
    res = 80
    xm, ym = np.meshgrid(np.linspace(0,8,res), np.linspace(0,5,res))
    zm = knn.predict(np.c_[xm.ravel(), ym.ravel()])
    zm = zm.reshape(xm.shape)
    plt.contourf(xm,ym,zm, cmap='Paired')

    plt.scatter(iris_2d[:,0], iris_2d[:,1], c=iris.target, cmap='jet')

# Use Euclidean norm
knn = sklearn.neighbors.KNeighborsClassifier(5)
knn.fit(iris_2d, iris.target)
plot_knn(knn)

# use L1  / taxicab norm
knn = sklearn.neighbors.KNeighborsClassifier(5, p=1)
knn.fit(iris_2d, iris.target)
plot_knn(knn)


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
One example where PCA does badly is the "swiss roll dataset" -- a plane rolled up into a spiral in 3D. This has a very simple structure; a simple plane with some distortion. But PCA can never unravel the spiral to find this simple explanation because it cannot be unravelled via a linear transformation.
Can you write Python code for it?

swiss_pos, swiss_val = sklearn.datasets.make_swiss_roll(800, noise=0.0)
fig = plt.figure()
# make a 3D figure
ax = fig.add_subplot(111, projection="3d")
ax.scatter(swiss_pos[:,0], swiss_pos[:,1], swiss_pos[:,2], c=swiss_val, cmap='gist_heat', s=10)

# Apply PCA to learn this structure (which doesn't help much)
pca = sklearn.decomposition.PCA(2).fit(swiss_pos)
pca_pos = pca.transform(swiss_pos)
plt.scatter(pca_pos[:,0], pca_pos[:,1], c=swiss_val, cmap='gist_heat')


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
One useful property of PCA is that we compute exactly how "fat" each of these learned dimensions were. The ratio of explained variance tells us how much each of the original variation in the dataset is captured by each learned dimension.
If most of the variance is in the first couple of components, we know that a 2D representation will capture much of the original dataset. If the ratios of variance are spread out over many dimensions, we will need many dimensions to represent the data well.
Can you write Python code for it?

# We can see how many dimensions we need to represent the data well using the eigenspectrum
# here we show the first 32 components
pca = sklearn.decomposition.PCA(n_components=32).fit(digit_data)
plt.bar(np.arange(32), pca.explained_variance_ratio_)
plt.xlabel("Component")
plt.ylabel("Proportion of variance explained")


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Overfitting
Overfitting is the key issue with machine learning algorithms. It is trivially easy to devise a supervised learning algorithm that takes in input features and exactly predicts the corresponding output classes in the training data. A simple lookup table will do this.
To make useful predictions, a learning algorithm must predict values with features it has never seen. The problem is we can only optimise the performance based on data we have seen. The generalisation performance of an algorithm is the ablility to make predictions outside of the training data.
This means that we cannot optimise an algorithm by adjusting parameters to fit the training data better; this will lead to overfitting, where the predictive power of the algorithm decreases as it is exposed to additional data.
Instead, we must use a separate testing set which the training algorithm is never exposed to to evaluate performance. The importance of this can be shown in a few simple examples.
Polynomial overfitting
A simple example of overfitting can be seen when fitting a polynomial $y=a_0x^0 + a_1x^1 + \dots + a_qx^q$ to $n$ points. When $n=q$, the curve goes through all the points, but makes wildly inappropriate interpolations between them.
The example below shows this effect:
Can you write Python code for it?

# create some random points
n_points = 10
x = np.random.uniform(-1,1,(n_points,))
y = x*x + np.random.normal(0,0.1,(n_points,))

# plot the points
plt.scatter(x,y);


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Random forests
Random forests use ensembles of very simple classifiers -- decision trees -- to transform a weak and poorly generalising learning model into a much more robust model. Random forests often have very competitive performance in classification tasks.
Can you write Python code for it?

# Run the test 10 times
for trials in range(20):
    forest_scores = []
    # iterate over number of trees in the ensemble
    for estimators in range(20):
        # completely randomised tree ensemble classifiers
        forest = sklearn.ensemble.ExtraTreesClassifier(n_estimators=estimators+1)
        forest.fit(sonar_train_features, sonar_train_labels)
        forest_scores.append(forest.score(sonar_test_features, sonar_test_labels))
    plt.plot(np.arange(len(forest_scores))+1, forest_scores, alpha=0.2)
plt.xlabel('No. estimators')
plt.ylabel('Accuracy')


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Receiver-operator curves
A very useful tool for capturing binary classification performance is the receiver-operator curve (ROC curve). This works with any classifier that produces scores as outputs (e.g. continuous values in the range [0,1]). Classifiers that only produce discrete class labels cannot be used to generate a ROC curve.
To plot the curve, we iterate through a set of threshold values $\tau_1, \tau_2, \dots$, and plot the accuracy we would get if we thresholded the classifiers at $\tau_i$. A classifier with chance performance will have an ROC curve with $y=x$; a very good classifier will have the curve bent up towards the upper-left corner.
Can you write Python code for it?

# we can plot the receiver-operator curve: the graph of false positive rate against true positive rate
scores = svm.decision_function(sonar_test_features)
fpr, tpr, thresholds = sklearn.metrics.roc_curve(sonar_test_labels, scores)
plt.plot(fpr,tpr)
plt.plot([0,1], [0,1])
plt.plot([0,1], [1,0])
plt.xlabel("False positive rate")
plt.ylabel("True positive rate")
plt.legend(["ROC", "Chance"])

# We can randomly permute the label vectors using np.random.permutation
# Now they have no relation to the training features
random_sonar_train_labels = np.random.permutation(sonar_train_labels)
random_sonar_test_labels = np.random.permutation(sonar_test_labels)


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Regression
Regression metrics include absolute error (total difference between predicted values $y^\prime$ and true values $y$), $R^2$ coefficient, and the ratio of explained variance.
Some measures, like mean squared error (MSE) depend on the units of the data and are only relevant for relative comparisons. Others, like $R^2$ and explained variance are independent of the units of the data.
Can you write Python code for it?

# Airfoil self-noise data
# Given parameters of an airfoil shape, predict how loud it will be
airfoil_data = np.loadtxt('airfoil_self_noise.dat', delimiter='\t')
airfoil_features = airfoil_data[:,:-1]
airfoil_values = airfoil_data[:,-1]

# test train split
airfoil_train_features, airfoil_test_features, airfoil_train_values, airfoil_test_values = sklearn.cross_validation.train_test_split(
    airfoil_features, airfoil_values, test_size=0.3, random_state=0)

def summarise_regression(true, predicted):
    print "Explained variance: %.2f" % (sklearn.metrics.explained_variance_score(true, predicted))
    print "R^2 score: %.2f" % (sklearn.metrics.r2_score(true, predicted))
    print "Mean absolute error: %.2f" % (sklearn.metrics.mean_absolute_error(true, predicted))
    print "Mean squared error (MSE): %.2f" % (sklearn.metrics.mean_squared_error(true, predicted))

def test_linear_model(model):
    plt.figure()
    # fit a penalised linear model
    airfoil_ridge = model
    airfoil_ridge.fit(airfoil_train_features, airfoil_train_values)

    # plot the prediction against the true
    airfoil_predicted_values = airfoil_ridge.predict(airfoil_test_features)
    plt.scatter(airfoil_test_values, airfoil_predicted_values)
    plt.xlabel("True value (dB)")
    plt.xlabel("Predicted value (dB)")
    summarise_regression(airfoil_test_values, airfoil_predicted_values)

test_linear_model(sklearn.linear_model.Ridge(alpha=0.5))
print
test_linear_model(sklearn.linear_model.LinearRegression())


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Single-class classifiers
So-called "single class" classifiers operate on slightly different basis than conventional classifiers. They are essentially unsupervised learning algorithms which learn to predict whether a sample is similar to ones seen before; i.e. whether it follows the distribution of the training data.
This is anomaly detection -- detecting data that is outside the "normal parameters" without explicitly training on out-of-class data.
Can you write Python code for it?

from matplotlib.patches import Ellipse
# plot some data with a normal distribution, plus a few outliers that are not part of the model

def anomaly_plot():
    x = np.random.normal(0, 1, (200,2))
    tr = np.array([[0.5, -0.2], [0.9, 0.3]])
    x = np.dot(x,tr)
    plt.scatter(x[:,0], x[:,1])

    for i in range(3):
        scale = 0.5+i*0.2
        e = Ellipse(xy=[0,0], width=3*scale, height=8*scale, angle=100, alpha=0.2)
        plt.gca().add_artist(e)
    x2 = np.random.normal(4, 3, (5,2))
    plt.scatter(x2[:,0], x2[:,1], marker='x', c='r', s=40)

anomaly_plot()


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Supervised learning
Supervised learning involves learning a relationship between attribute variables and target variables; in other words learning a function which maps input measurements to target values. This can be in the context of making discrete decisions (is this image a car or not?) or learning continuous relationships (how loud will this aircraft wing be if I make the shape like this?).
Unsupervised learning
Unsupervised learning learns the structure of data without any explicit labeling of points. The key idea is that datasets may have a simple underlying or latent representation which can be determined simply by looking at the data itself.
Two common unsupervised learning tasks are clustering and dimensional reduction. Clustering can be thought of as the unsupervised analogue of classification -- finding discrete classes in data. Dimensional reduction can be thought of as the analogue of regression -- finding a small set of continuous variables which "explain" a higher dimensional set.
Reinforcement learning
There are other machine learning paradigms, such as reinforcement learning, which tries to learn a policy (what to do in specific circumstances) using only a reward function (was what the algorithm did good or bad?). This is important in learning control tasks, e.g. for autonomous vehicles. We won't cover reinforcement learning here.
Black box models
Some learning algorithms produce models which are essentially inscrutable. They are intricate deformations of the initial data and there is no way for a human to meaningfully understand the process by which decisions are produced; the models are black boxes that produce predictions. Other models, especially well-defined probabilistic models, have a clear and interpretable structure and it is easy to understand how predictions are arrived at. Generative models are probablistic models that can draw samples from the model, and can be easier to interpret that discriminative models which simply learn a function that discriminates between distinct...
Can you write Python code for it?

### Perceptron demo
iris = sklearn.datasets.load_iris()
# we just choose a 2D slice of this data
iris_2d = iris.data[:,0:2]

# plot the 2D data
plt.scatter(iris_2d[:,0], iris_2d[:,1], c=iris.target, cmap='jet')
plt.title("Original iris data")

# make binary targets (either class 0 or other class)
binary = np.where(iris.target==0, -1, 1)
plt.figure()
plt.scatter(iris_2d[:,0], iris_2d[:,1], c=binary, cmap='jet')
plt.title("Binary classes")

# find a separating plane
per = sklearn.linear_model.Perceptron( n_iter=5, eta0=1)
# "warm up" and fill in the coefficients
per.partial_fit(iris_2d, binary, [-1,1])

def fit_perceptron(x):
    # apply 20 steps
    for i in range(20):
        per.partial_fit(x, binary, [-1,1])

def plot_perceptron(x):
    # plot the original data
    plt.figure()
    # predict output value across the space
    res = 150

    # we generate a set of points covering the x,y space
    xm, ym = np.meshgrid(np.linspace(0,8,res), np.linspace(0,5,res))
    # then predict the perceptron output value at each position
    zm = per.predict(np.c_[xm.ravel(), ym.ravel()])
    zm = zm.reshape(xm.shape)
    # and plot it
    plt.contourf(xm,ym,zm, cmap='Paired')

    plt.scatter(x[:,0], x[:,1], c=binary, cmap='jet')

for i in range(5):
    fit_perceptron(iris_2d)
    plot_perceptron(iris_2d)


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
The SVM learned essentially nothing; the accuracy is very misleading.
Can you write Python code for it?

scores = wilt_svm.decision_function(wilt_test_features)
fpr, tpr, thresholds = sklearn.metrics.roc_curve(wilt_test_labels, scores)
plt.plot(fpr,tpr)
plt.plot([0,1], [0,1])
plt.xlabel("False positive rate")
plt.ylabel("True positive rate")
plt.legend(["ROC", "Chance"])


I am working on the file "CrashCourse_Lecture1.ipynb".
The first task is:
Try running the above code with increasing values for the order
Kernel density estimation example
We can see the same effect if we try to learn the distribution of data using kernel density estimation (KDE). KDE is effectively a smoothed histogram, which is created by "placing" smooth distributions on each observed data point and summing them (i.e. convolving them with some window function). This can be used to estimate an underlying smooth distribution from point samples.
The key parameter in KDE is the kernel width $\sigma$, which determines how wide each distribution will be and thus how smooth the overall distribution will be.
Can you write Python code for it?

# generate some random numbers -- the use of the  beta distribution isn't important, it just gives an interesting shape
x = np.random.beta(0.5, 2, 40);
# plot the data points
plt.figure()
# scatter plot showing the actual positions
plt.scatter(x,np.ones_like(x))
plt.figure()
# histogram of the data points
plt.hist(x, normed=True);

import scipy.stats as stats

# plot the kernel density estimate (Gaussian window) with the given bandwidth
def plot_kde(x, width):
    kde = stats.kde.gaussian_kde(x, bw_method=width)
    # evaluate kde estimate over range of x
    xs = np.linspace(np.min(x), np.max(x), 100)
    plt.figure()
    plt.plot(xs, kde(xs))
    plt.scatter(x, np.ones_like(x))

plot_kde(x, 2) # too smooth
plot_kde(x, 1) # a bit too smooth
plot_kde(x, 0.5) # good
plot_kde(x, 0.1) # too rough
plot_kde(x, 0.01) # just learning the data points