I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
Get used to throwing appropriate — and meaningful — exceptions in your code! It makes reading and debugging your code much, much easier.
More Advanced Exception Handling
There is also the possibility of adding else and finally clauses to your try statements.
You'll probably not need these often, but in case you encounter them some time, it's good to know what they do.
The behavior looks like this:
Can you write Python code for it?

try:
    print("doing something")
except:
    print("this only happens if it fails")
else:
    print("this only happens if it succeeds")

try:
    print("doing something")
    raise ValueError()
except:
    print("this only happens if it fails")
else:
    print("this only happens if it succeeds")


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
Looks OK, yes?  Let's try it out:
Can you write Python code for it?


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
Note that if your code contains even a single syntax error, none of it will run:
Can you write Python code for it?


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
Note that the finally clause is executed no matter what, even if the return statement has already executed!
This makes it useful for cleanup tasks, such as closing an open file, restoring a state, or something along those lines.
Handling Semantic Errors: Debugging
Here is the most difficult piece of this lecture: handling semantic errors.  This is the situation where your program runs, but doesn't produce the correct result.  These errors are commonly known as bugs, and the process of correcting the bugs is debugging.
There are three main methods commonly used for debugging Python code.  In order of increasing sophistication, they are:

Inserting print statements
Injecting an IPython interpreter
Using a line-by-line debugger like pdb

The easiest method: print statements
Say we're trying to compute the entropy of a set of probabilities.  The
form of the equation is
$$
H = -\sum_i p_i \log(p_i)
$$
We can write the function like this:
Can you write Python code for it?

def entropy(p):
    p = np.asarray(p)  # convert p to array if necessary
    items = p * np.log(p)
    return -np.sum(items)


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
Notice that the first block executes up until the point of the Runtime error.
Once the error is hit, the except block is executed.
One important note: the above clause catches any and all exceptions.  It is not
generally a good idea to catch-all.  Better is to name the precise exception you expect:
Can you write Python code for it?

def safe_divide(a, b):
    try:
        return a / b
    except:
        print("oops, dividing by zero. Returning None.")
        return None

print(safe_divide(15, 3))
print(safe_divide(1, 0))


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
Our program tells us we're dividing by zero, but we aren't!  This is one reason you should almost never use a catch-all try..except statement, but instead specify the errors you're trying to catch:
Can you write Python code for it?

def better_safe_divide(a, b):
    try:
        return a / b
    except ZeroDivisionError:
        print("oops, dividing by zero. Returning None.")
        return None

better_safe_divide(15, 0)

better_safe_divide(15, 'three')


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
Remember this lesson, and always specify your except statements!  I once spent an entire day tracing down a bug in my code which amounted to this.
Raising Your Own Exceptions
When you write your own code, it's good practice to use the raise keyword to create your own exceptions
when the situation calls for it:
Can you write Python code for it?

import os  # the "os" module has useful operating system stuff

def read_file(filename):
    if not os.path.exists(filename):
        raise ValueError("'{0}' does not exist".format(filename))
    f = open(filename)
    result = f.read()
    f.close()
    return result


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
Say these are our probabilities:
Can you write Python code for it?

p = np.arange(5.)
p /= p.sum()

entropy(p)


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
This also allows you to specify different behaviors for different exceptions:
Can you write Python code for it?

def even_better_safe_divide(a, b):
    try:
        return a / b
    except ZeroDivisionError:
        print("oops, dividing by zero. Returning None.")
        return None
    except TypeError:
        print("incompatible types.  Returning None")
        return None

even_better_safe_divide(15, 3)

even_better_safe_divide(15, 0)

even_better_safe_divide(15, 'three')


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
This can be a more convenient way to debug programs and step through the actual execution.
When you run this, you'll see the pdb prompt where you can enter one of several commands.  If you type h for "help", it will list the possible commands:
```
(Pdb) h
Documented commands (type help <topic>):
========================================
EOF    bt         cont      enable  jump  pp       run      unt
a      c          continue  exit    l     q        s        until
alias  cl         d         h       list  quit     step     up
args   clear      debug     help    n     r        tbreak   w
b      commands   disable   ignore  next  restart  u        whatis
break  condition  down      j       p     return   unalias  where
Miscellaneous help topics:
exec  pdb
Undocumented commands:
retval  rv
```
Type h collowed by a command to see the documentation of that command:
(Pdb) h n
n(ext)
Continue execution until the next line in the current function
is reached or it returns.
The most useful are probably the following:

q(uit): quit the debugger and the program.
c(ontinute): quit the debugger, continue in the program.
n(ext): go to the next step of the program.
list: show the current location in the file.
<enter>: repeat the previous command.
p(rint): print variables.
s(tep into): step into a subroutine.
r(eturn out): return out of a subroutine.
Arbitrary Python code: writing Python code at the (Pdb) will execute it at that point in the program.

We'll see more of this in the next section.
IPython Debugging
IPython also has some magic commands that allow you to debug scripts withing the notebook as soon as you see a failure. For example, imagine we have the following file:
Can you write Python code for it?


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
We get a type error. We can immediately open the debugger using IPython's %debug magic function. Remember to type q to quit!
Can you write Python code for it?


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
We get nan, which stands for "Not a Number".  What's going on here?
Often the first thing to try is to simply print things and see what's going on.
Within the file, you can add some print statements in key places:
Can you write Python code for it?

def entropy(p):
    p = np.asarray(p)  # convert p to array if necessary
    print(p)
    items = p * np.log(p)
    print(items)
    return -np.sum(items)

entropy(p)


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
We'll use IPython's %%file magic to quickly create a text file
Can you write Python code for it?

%%file tmp.txt
this is the contents of the file

read_file('tmp.txt')

read_file('file.which.does.not.exist')


I am working on the file "LectureNotes/ExceptionsDebugging.ipynb".
The first task is:
finally is generally used for some sort of cleanup (closing a file, etc.)  It might seem a bit redundant, though.  Why not write the following?
Can you write Python code for it?

try:
    print("do something")
except:
    print("this only happens if it fails")
else:
    print("this only happens if it succeeds")
print("this happens no matter what.")


I am working on the file "LectureNotes/ProceduralPython/Completed-ProceduralPython.ipynb".
The first task is:
(paste the above function in pronto_utils.py)
Can you write Python code for it?


I am working on the file "LectureNotes/ProceduralPython/Completed-ProceduralPython.ipynb".
The first task is:
Breakout: Add functionality
Working in pairs, do the following:

create a function that will plot an interesting aspect of this data
once you are happy with the function, copy it into the Python package you have created
Write a "smoke-test" – this is a test that calls the function, but doesn't necessarily validate the output. This can be useful for "testing" plotting functions, because it's generally difficult to programatically evaluate the plot output itself

For step 3, you'll have to tell matplotlib not to invoke the graphical backend, which you can do by putting the following at the top of the test file:
python
import matplotlib as mpl
mpl.use('Agg')  # Don't invoke graphical backend for plots
If you want to go farther with testing the output of your plot, matplotlib has some useful plot testing tools that you can use.
Can you write Python code for it?

%matplotlib inline

def plot_totals_by_birthyear():
    df = load_trip_data()
    totals_by_birthyear = df.birthyear.value_counts().sort_index()
    return totals_by_birthyear.plot(linestyle='steps')

plot_totals_by_birthyear()

def test_plot_totals():
    ax = plot_totals_by_birthyear()
    assert len(ax.lines) == 1

import numpy as np
import matplotlib as mpl

def test_plot_totals_by_birthyear():
    ax = plot_totals_by_birthyear()

    # Some tests of the output that dig into the
    # matplotlib internals
    assert len(ax.lines) == 1

    line = ax.lines[0]
    x, y = line.get_data()
    assert np.all((x > 1935) & (x < 2000))
    assert y.mean() == 1456

test_plot_totals_by_birthyear()

!py.test pronto_utils.py


I am working on the file "LectureNotes/ProceduralPython/Completed-ProceduralPython.ipynb".
The first task is:
Step 2: Make a Package
Now that this function works, let's create a Python package that we can import it from
(Use a text editor to edit pronto_utils.py)
Can you write Python code for it?

from pronto_utils import download_if_needed
download_if_needed(URL, 'open_data_year_one.zip')


I am working on the file "LectureNotes/ProceduralPython/Completed-ProceduralPython.ipynb".
The first task is:
Step 3: Write a Unit Test
Let's write a unit test to make sure our download script works properly. We will use pytest here.
Can you write Python code for it?

import pandas as pd
from pronto_utils import load_trip_data

def test_trip_data():
    df = load_trip_data()
    assert isinstance(df, pd.DataFrame)
    assert df.shape == (142846, 12)

test_trip_data()


I am working on the file "LectureNotes/ProceduralPython/Completed-ProceduralPython.ipynb".
The first task is:
Use Python to unzip and load the data:
Can you write Python code for it?

import zipfile
import pandas as pd

def load_trip_data(filename='open_data_year_one.zip'):
    """Load trip data from the zipfile; return as DataFrame"""
    download_if_needed(URL, filename)
    zf = zipfile.ZipFile(filename)
    return pd.read_csv(zf.open('2015_trip_data.csv'))

data = load_trip_data()
data.head()


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
And here is another one:
Can you write Python code for it?

x2 = -3   # oops maybe the choice of variable names is not optimal
y2 = 17


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
Careful with the amount of code inside a function (7$\pm$2 rule)!

Default arguments are pretty powerful!
However, there's a caveat here.
Default arguments are evaluated at definition-time and not at call-time.
What does that entail? Here is an example of a function that logs a message together with a time stamp:
Can you write Python code for it?

import datetime as dt     # What's that?

print dt.datetime.now()

def log_time(message, time=dt.datetime.now()):
    print("{0}: {1}".format(time.isoformat(), message))

log_time("message 1")

log_time("message 2")

log_time("message 3")


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
Functions seem to be a powerful programming tool.
It might be a good idea to go through Python and its function functionality.
More on Functions
Function Defintion
Can you write Python code for it?

def functionname( parameters ):
   "function_docstring"
   function_suite
   return [expression]


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
Hm, that's kind of annoying.
What we are doing here is called
 Imperative Programming

IP uses statements to change the state of a program.
Changes are here-and-now changes (see length calculation above).
IP goes back to the early days of computers when programming consisted of simple statements.
IP is very powerful when comes to data manipulation.
But...

Interlude: The Magical Number 7$\pm$2

George Miller
research on short-term memory capacity
1956: paper on the Magical Number 7$\pm$2.
7 $\pm$ 2 is the limit of storable information
But it's also a good rule of thumb when it comes to programming.

When you need more than 7 +/- 2 lines of code to describe something, it's very likely that you need to break down the problem in smaller chunks.
Back to our problem...

Imperative programming forced us to blow up our source code.
And we easily violated Miller's Rule!
Do we have to type the same thing over and over again?
Well, of course not!
We know that we can 'outsource' common program code in functions!

This is in short the idea of the  procedural programming paradigm.

We know how to write a function:
Can you write Python code for it?

def length(x, y):
    return sqrt(x* x + y * y)


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
Obviously this class is pretty boring. Nevertheless, we can now create objects of type Vehicle by instanciating:
Can you write Python code for it?


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
Procedural programming seems to be enourmos improvement over imperative programming.
code maintainance
debugging
code encapsulation (the user doesn't need to know about the implementation)
code sharing

It's natural to ask Can we extend the concept of procedural programming?
Yes, it's called <font, color='red'> Functional Programming</font>

What is Functional Programming?
Everything is a function!
Everything?
Everything!!!
Even loops?
Even loops!!!

Functional programming (FP) surely sounds like a radical approach.
But not for a software engineer or a mathematician ($\lambda$-calculus).
Especially mathematicians deal with expressions like this all the time: $\qquad f(g(h(x)))$
Mapping such a construct into a source code surely has it's advantages.
Python was never designed with FP in mind.
That being said, FP works quite well in Python.
However, it probably requires more than just an hour to dive into the details.

Let's go back to out little program...

We structured our code with the help of functions/methods/procedures.
In principle we can extend our code by adding new methods.
And we caneven share code by exchanging functions.
Now a colleague is interested in our little vector program.
He wants to do some vector calculations as well.
Unfortunately, he doesn't have vectors with cartesian coordinates but polar coordinates.

No problem! We write a function that converts all the polar coordinate tuples to cartesian coordinate tuples.

That works fine.

Until a third colleague comes along.
He now has both, vectors in cartesian coordinates and vectors in polar coordinates.
On top of the conversion function we need to test now what kind of vector we have.
It looks like that most of the code is now about conversion and book-keeping.
But this is really beside the actual goal: Vector Operations.

And you wonder, there should be a better way to organize the code...

In principle a vector is a vector is a vector.

Its behavior invariant under the choice of a coordinate system.
Can we actually map a vector (its...
Can you write Python code for it?


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
That was a crash course in OO programming.
We haven't touched topics like data encapsulation, Class and instance variables, ...
Unfortunately, this is beyond the scope of our lecture.
Let's compare the programming paradigms and finish with another OO example.

Comparing the four programming paradigms
Imperative: Computation is performed as a direct change to program state. This style is especially useful when manipulating data structures and produces elegant, but simple, code.
Can you write Python code for it?

my_list = [1, 2, 3, 4, 5]
sum = 0
for x in my_list:
    sum += x
print(sum)


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
The application of default arguments seem to be obvious.
But there are some not so obvious applications: <font, color='red'> method overriding</font>.
MO might be known from other languages:
Define multiple functions with the same name
Functions are then distinguished by their number and type of arguments.

Python can do something similar (and in most cases it's more elegant).
Here is a simple example $\Rightarrow$
Can you write Python code for it?

def vector_sum(vx1, vy1, vx2, vy2, vz1=None, vz2=None):
    if vz1 == None or vz2 == None:             # making sure that both values are set
        return vx1 + vx2, vy1 + vy2
    else:
        return vx1 + vx2, vy1 + vy2, vz1 + vz2

print vector_sum(1, 2, 3, 4)

print vector_sum(1, 2, 3, 4, 8,8)


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
The object-oriented coding style is all about increasing the ability of applications to reuse code and making code easier to understand. The encapsulation that object-orientation provides allows developers to treat code as a black box.
Summary

We have seen various approaches to structure source code.
Python leaves it up to you to choose and pick.
None of them is right or wrong.
The choice strongly depends on your problem.
You need something 'quick and dirty': Imperative Style
You are working on a larger project that maybe even involves several developers: OO
Another way to look at it, is how data is being manipulated.
You have a fixed set of operations on things: OO
You have a fixed set of things: Functional/Procedural Programming

Exercise

In this week's exercise you are supposed to get familiar with OOP.
We want to write the code for a linked list.
A linked list is a container like lists, arrays, dictionaries, etc.
The speciality of a linked list is that each element in a list (called node) contains also a pointer to the following element.

Here is the code for a node:
Can you write Python code for it?

class ListNode:
    def __init__(self, data):
        "constructor to initiate this object"

        # store data
        self.data = data

        # store reference (next item)
        self.next = None
        return

    def contains_value(self, value):
        "method to compare the value with the node data"
        if self.data == value:
            return True
        else:
            return False

node1 = ListNode('data point 1')
node2 = ListNode([15, 16 , 3.14])
node3 = ListNode(16.6)


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
The procedural style relies on procedure calls to create modularized code.
It seeks to simplify the application code by creating small pieces that a developer can view easily.
Even though the procedural coding style is an older form of application development, it’s still a viable approach when a task lends itself to step-by-step execution.

Functional: Any form of state or mutable data are avoided. The main advantage of this approach is that there aren’t any side effects to consider. In addition, this coding style lends itself well to parallel processing because there is no state to consider.
Can you write Python code for it?

import functools
my_list = [1, 2, 3, 4, 5]
def add(x, y):
    return (x + y)
sum = functools.reduce(add, my_list)
# calculates add(add(add(add(1, 2), 3), 4), 5)

print(sum)


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
Wait, there's more!
We found a way to map real objects in source code.
But so far we have ignored a possible hierarchical relation among objects.
For example: animal -> mammal -> cat
It looks like in such a hierarchy some properties are shared and some are not.
It's not too far-fetched to assume that this sort of hierarchies makes also sense in programming.
It's called inheritance and it works in Python like this:
Can you write Python code for it?

from math import cos, sin, sqrt

class MyVector():
    def __init__(self, x, y):
        self.x = x
        self.y = y

    def get_length(self):
        return sqrt(self.x * self.x + self.y * self.y)

    def scale_vector(self, a):
        self.x *= a
        self.y *= a

    def add_vector(self, vector):
        self.x += vector.x
        self.y += vector.y

    def __add__(self, other):
        x = self.x + other.x
        y = self.y + other.y
        return MyVector(x, y)

    def get_r(self):
        return self.get_length()

class MyPolarVector(MyVector):
    def __init__(self, r, phi):
        self.r = r
        MyVector.__init__(self, r * cos(phi), r * sin(phi))

    def get_length(self):
        print "inside MyPolarVector"
        return self.r

    def get_r(self):
        return self.r

v1 = MyVector(10, 20)
v2 = MyPolarVector(12, 2.7)

print (v1 + v2).get_r()

print v1.get_length()
print v2.get_length()
print (v1 + v2).get_length()


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
We call __init__ the constructor method.
self represents the instance of the class.
By using the self keyword we can access the attributes and methods of the class in python.
Let's add some more methods.
Can you write Python code for it?

class Vehicle():
    def __init__(self, number_of_wheels, number_of_seats, max_velocity):
        self.number_of_wheels = number_of_wheels
        self.number_of_seats = number_of_seats
        self.max_velocity = max_velocity

    def get_max_velocity(self):
        return self.max_velocity

    def make_noise(self):
        print("vrummmmm")

car = Vehicle(4, 5, 200.0)
print car.get_max_velocity()
print car.max_velocity
car.make_noise()


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
We have dealt with the actual function code.
We know about the return value(s).
But what about the parameters?

Function Arguments (Parameters)

required arguments
keyword arguments
default arguments
variable length arguments

required arguments
Can you write Python code for it?


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
We just created an object that we called car of type Vehicle.

Let's add some attributes and methods to our class.
In order to initialize an object with a set of attributes we use the __init__ method:
Can you write Python code for it?

class Vehicle():
    def __init__(self, number_of_wheels, number_of_seats, max_velocity):
        self.number_of_wheels = number_of_wheels
        self.number_of_seats = number_of_seats
        self.max_velocity = max_velocity

car = Vehicle(4, 5, 200.0)
bike = Vehicle(2, 2, 50.0)
print car.number_of_wheels
print bike.number_of_seats


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
What about other functions?
Can you write Python code for it?

def vector_sum(vx1, vy1, vx2, vy2):
    x = vx1 + vx2
    y = vy1 + vy2
    return x, y      #Nice, there's no restriction on the amount of return values.

def scalar_prod(vx1, vy1, vx2, vy2):
    return (vx1 * vx2 + vy2 * vy2)


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
keyword arguments
Keyword argument allow the caller to identify the arguments by their parameter names:
Can you write Python code for it?

def foo(name, age):
    print("name:", name)
    print("age: ", age)

foo('Alice', 35)
print('----')
foo(19, 'Bob')
print('----')
foo(age = 78, name = 'Dave')


I am working on the file "LectureProgrammingParadigms/Lecture Programming Paradigms.ipynb".
The first task is:
variable-length arguments
Can you write Python code for it?

def foo(arg1, *argtuple):
    print(arg1)
    for arg in argtuple:
        print(arg)

foo('Hello World')
foo('x', 1, 'hello', 3.1415, '20')
foo()


I am working on the file "Lecture_01.ipynb".
The first task is:
2. データベースへのデータの格納
データベースとは
データベースとは、色々なデータの目的ベースでの管理や、効率的なデータ参照／検索を可能にするものです。
データベースの中には複数のテーブルがあります。
テーブルはちょうどスプレッドシートのようになっていて、それぞれの列に名前があり、1行が1つのデータとなるイメージです。
データの格納
データの格納の流れは大まかに、
1. テーブルの作成
2. テーブルへのインサート
3. errorやwarningの確認
の3つのステップとなります。
kaggleのページにテーブルの定義が書いてあるので、ここでは、その通りに作成します。
まずは、user_listのテーブル作成クエリと実行です。
MySQLのCREATE TABLE構文については、http://dev.mysql.com/doc/refman/5.6/ja/create-table.html を参照ください。
Can you write Python code for it?

dbcur.execute('''DROP TABLE IF EXISTS user_list;''')  # チュートリアルの便宜上、一度削除します。
query = '''
CREATE TABLE IF NOT EXISTS user_list (
    reg_date DATETIME,
    sex_id VARCHAR(1),
    age INT,
    withdraw_date DATETIME,
    pref_name VARCHAR(15),
    user_id_hash VARCHAR(32),
    PRIMARY KEY(user_id_hash),
    INDEX(reg_date),
    INDEX(sex_id),
    INDEX(age),
    INDEX(withdraw_date),
    INDEX(pref_name)
) ENGINE=MyISAM DEFAULT CHARSET=utf8;
'''
dbcur.execute(query)


I am working on the file "Lecture_01.ipynb".
The first task is:
coupon_detail_train.csv
Can you write Python code for it?

### coupon_detail_train
csv_path = DATA_DIR + '/coupon_detail_train.csv'
dbcur.execute('''DROP TABLE IF EXISTS coupon_detail_train;''')
dbcur.execute('''
CREATE TABLE IF NOT EXISTS coupon_detail_train (
    item_count INT,
    i_date DATETIME,
    small_area_name VARCHAR(32),
    purchaseid_hash VARCHAR(32),
    user_id_hash VARCHAR(32),
    coupon_id_hash VARCHAR(32),
    INDEX(coupon_id_hash)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
;
''')

dbcur.execute('''
LOAD DATA LOCAL INFILE "''' + csv_path + '''"
INTO TABLE coupon_detail_train
CHARACTER SET utf8
FIELDS TERMINATED BY ','
IGNORE 1 LINES
(item_count, i_date, small_area_name, purchaseid_hash, user_id_hash, coupon_id_hash)
;
''')


I am working on the file "Lecture_01.ipynb".
The first task is:
coupon_list_train.csv, coupon_list_test.csv
2つともWarningが出ますが、
日時の値が正しくないために発生しているだけなので、無視します。
下記のクエリの
SQL
SET validperiod=IF(@validperiod = 'NA', Null, @validperiod)
を外すと、おそらくNULLが入ってほしい箇所に0が入ってしまうため、ここでは、NULLがはいるように変換します。
Can you write Python code for it?

### coupon_list_train
csv_path = DATA_DIR + '/coupon_list_train.csv'
dbcur.execute('''DROP TABLE IF EXISTS coupon_list_train;''')
dbcur.execute('''
CREATE TABLE IF NOT EXISTS coupon_list_train (
    capsule_text VARCHAR(20),
    genre_name VARCHAR(50),
    price_rate INT,
    catalog_price INT,
    discount_price INT,
    dispfrom DATETIME,
    dispend DATETIME,
    dispperiod INT,
    validfrom DATE,
    validend DATE,
    validperiod INT,
    usable_date_mon VARCHAR(7),
    usable_date_tue VARCHAR(7),
    usable_date_wed VARCHAR(7),
    usable_date_thu VARCHAR(7),
    usable_date_fri VARCHAR(7),
    usable_date_sat VARCHAR(7),
    usable_date_sun VARCHAR(7),
    usable_date_holiday VARCHAR(7),
    usable_date_before_holiday VARCHAR(7),
    large_area_name VARCHAR(30),
    ken_name VARCHAR(8),
    small_area_name VARCHAR(30),
    coupon_id_hash VARCHAR(32),
    PRIMARY KEY(coupon_id_hash),
    INDEX(ken_name),
    INDEX(genre_name)

) ENGINE=MyISAM DEFAULT CHARSET=utf8
;
''')

dbcur.execute('''
LOAD DATA LOCAL INFILE "''' + csv_path + '''"
INTO TABLE coupon_list_train
CHARACTER SET utf8
FIELDS TERMINATED BY ','
IGNORE 1 LINES
(capsule_text,genre_name,price_rate,catalog_price,discount_price,dispfrom,dispend,dispperiod,validfrom,validend,@validperiod,usable_date_mon,usable_date_tue,usable_date_wed,usable_date_thu,usable_date_fri,usable_date_sat,usable_date_sun,usable_date_holiday,usable_date_before_holiday,large_area_name,ken_name,small_area_name,coupon_id_hash)
SET validperiod=IF(@validperiod = 'NA', Null, @validperiod)
;
''')

### coupon_list_test
csv_path = DATA_DIR + '/coupon_list_test.csv'
dbcur.execute('''DROP TABLE IF EXISTS coupon_list_test;''')
dbcur.execute('''
CREATE TABLE IF NOT EXISTS coupon_list_test (
    capsule_text VARCHAR(20),
    genre_name VARCHAR(50),
    price_rate INT,
    catalog_price INT,
    discount_price INT,
    dispfrom DATETIME,
    dispend DATETIME,
    dispperiod INT,
    validfrom DATE,
    validend DATE,
    validperiod INT,
    usable_date_mon VARCHAR(7),
   ...


I am working on the file "Lecture_01.ipynb".
The first task is:
coupon_visit_train.csv
このファイルはレコード数が一番多く、インサートが完了するまで少し時間がかかります。
Can you write Python code for it?

### coupon_visit_train
csv_path = DATA_DIR + '/coupon_visit_train.csv'
dbcur.execute('''DROP TABLE IF EXISTS coupon_visit_train;''')
dbcur.execute('''
CREATE TABLE IF NOT EXISTS coupon_visit_train (
    purchase_flg INT,
    i_date DATETIME,
    page_serial INT,
    referrer_hash VARCHAR(128),
    view_coupon_id_hash VARCHAR(128),
    user_id_hash VARCHAR(32),
    session_id_hash VARCHAR(128),
    purchaseid_hash VARCHAR(32),
    INDEX(user_id_hash, i_date),
    INDEX(i_date, user_id_hash),
    INDEX(view_coupon_id_hash),
    INDEX(purchaseid_hash),
    INDEX(purchase_flg)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
;
''')

dbcur.execute('''
LOAD DATA LOCAL INFILE "''' + csv_path + '''"
INTO TABLE coupon_visit_train
CHARACTER SET utf8
FIELDS TERMINATED BY ','
IGNORE 1 LINES
(purchase_flg,i_date,page_serial,referrer_hash,view_coupon_id_hash,user_id_hash,session_id_hash,purchaseid_hash)
;
''')


I am working on the file "Lecture_01.ipynb".
The first task is:
テーブルの作成に利用したCREATE TABLE文には、
テーブルの型の定義ではなく、インデックスと呼ばれるものの定義も含まれています。
インデックスとはデータの検索を高速化するものです。
PRIMARY KEY
テーブル内でuniqueで、かつ、検索するカラムに付与する。
例えば、user_listテーブルのuser_id_hashは当該テーブルで、ユニークであり，かつ、ユーザの検索によく用いるため、PRIMARY KEYを付与しておいた方が良い。
INDEX
テーブル内でuniqueではないが、検索するカラムに付与する。例えば、ユーザを性別や年齢に応じて検索・集計して、割合を見たい場合には、sex_idやageなどのカラムに付与しておいた方が良い。
TODO
MYSQL関数などの説明の加筆。
Exercise
下記の他のファイルについても同様にテーブルを作成し、データをインサートしてください。
- prefecture_locations.csv
- coupon_area_train.csv, coupon_area_test.csv
- coupon_detail_train.csv
- coupon_visit_train.csv
- coupon_list_train.csv, coupon_list_test.csv
実装例

prefecture_locations.csv
Can you write Python code for it?

### prefecture_locations
csv_path = DATA_DIR + '/prefecture_locations.csv'
dbcur.execute('''DROP TABLE IF EXISTS prefecture_locations;''')
dbcur.execute('''
CREATE TABLE IF NOT EXISTS prefecture_locations (
    pref_name VARCHAR(15),
    PRIMARY KEY(pref_name),
    prefectual_office VARCHAR(15),
    latitude DOUBLE,
    longitude DOUBLE
) ENGINE=MyISAM DEFAULT CHARSET=utf8
;
''')

dbcur.execute('''
LOAD DATA LOCAL INFILE "''' + csv_path + '''"
INTO TABLE prefecture_locations
CHARACTER SET utf8
FIELDS TERMINATED BY ','
IGNORE 1 LINES
(pref_name, prefectual_office, latitude, longitude)
;
''')


I am working on the file "Lecture_01.ipynb".
The first task is:
実行すると、それぞれのレコードでWarningが発生しますが、
データベースに展開されたレコードのlongitudeを確認すると正しく展開されているため、ここではWarningは無視します。
(確認しておりませんが、行末の改行コードがWarningの原因かもしれません、、、)
- coupon_area_train.csv, coupon_area_test.csv
Can you write Python code for it?

### coupon_area_train
csv_path = DATA_DIR + '/coupon_area_train.csv'
dbcur.execute('''DROP TABLE IF EXISTS coupon_area_train;''')
dbcur.execute('''
CREATE TABLE IF NOT EXISTS coupon_area_train (
    small_area_name  VARCHAR(32),
    pref_name VARCHAR(15),
    coupon_id_hash VARCHAR(32),
    INDEX(coupon_id_hash),
     INDEX(pref_name)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
;
''')

dbcur.execute('''
LOAD DATA LOCAL INFILE "''' + csv_path + '''"
INTO TABLE coupon_area_train
CHARACTER SET utf8
FIELDS TERMINATED BY ','
IGNORE 1 LINES
(small_area_name,pref_name,coupon_id_hash)
;
''')

### coupon_area_test
csv_path = DATA_DIR + '/coupon_area_test.csv'
dbcur.execute('''DROP TABLE IF EXISTS coupon_area_test;''')
dbcur.execute('''
CREATE TABLE IF NOT EXISTS coupon_area_test (
    small_area_name  VARCHAR(32),
    pref_name VARCHAR(15),
    coupon_id_hash VARCHAR(32),
    INDEX(coupon_id_hash),
    INDEX(pref_name)
) ENGINE=MyISAM DEFAULT CHARSET=utf8
;
''')

dbcur.execute('''
LOAD DATA LOCAL INFILE "''' + csv_path + '''"
INTO TABLE coupon_area_test
CHARACTER SET utf8
FIELDS TERMINATED BY ','
IGNORE 1 LINES
(small_area_name,pref_name,coupon_id_hash)
;
''')


I am working on the file "Lecture_02.ipynb".
The first task is:
3. モデリング対象の設定
対象の明確化
このコンペティションの最終的なゴールは、各ユーザが将来どのクーポンを購入するかをより正確に予測することです。
モデリング(≒予測モデルの構築)は
- ユーザの属性データ(性別、年代、地域など)
- クーポンの属性データ(ジャンル、価格、地域など)
- いつどのユーザがどのクーポンを見たか、購買したか等のログデータ
の3つのデータを主に利用します（これは、クーポンに限らず他の多くの問題で共通しています)。
これらのデータを利用して将来ユーザがどのクーポンを購買するのかをモデリングするわけですが、
特に、このコンペティションでは、モデリングの良し悪しを評価する指標が与えられており、
参加者はこの指標のスコアを競っている形になっています。
https://www.kaggle.com/c/coupon-purchase-prediction/details/evaluation によれば、
このコンペティションでは、Mean Average Precision@10(MAP@10)という指標を最大化するような予測モデルを構築することが最終的なゴールです。
Map@10は、下記の数式で定義されています。
$$
MAP@10=\frac{1}{|U|}\sum_{u=1}^{|U|}\frac{1}{min(m, 10)} \sum_{k=1}^{min(n, 10)}P(k)
$$
$|U|$：ユーザ数
$P(k)$：k番目まで着目した時のPrecision(購買すると予測したクーポン中の実際に購買されたクーポンの割合)
$n$：購買すると予測されたクーポンの数
$m$：実際に購買されたクーポンの数(m=0の時は、Precision=0)
この指標を直接最大化するようにモデリングする方がコンペティションの最終的な評価は高いと考えられますが、
ここでは、簡単のため、下記のように問題を少し変更し別の指標で代替することにします。
- 全てのユーザは検証用の期間で必ずクーポンをちょうど1つ購入する。
- その1つのクーポンを訓練用の期間のデータから予測する。
このように問題を簡単にし、検証用の期間で購買される確率が最も高いクーポンを選択する問題にすることで、
正解率(Accuracy)でモデルの良し悪しを近似的に評価することが出来ます。
このように、近似した問題に対してモデルを構築することとして、以下は進めていきます。
モデル構築に於ける3つのデータ：training data, validation data, test data
データはまず大きく分けて、訓練用のデータとテスト用のデータ(test data)に大別されます。
特に、Kaggleのコンペティションのデータセットは明示的にそれらが分けられているものもあります。
通常、訓練用のデータで予測モデルを構築し、テスト用のデータで構築した予測モデルの精度を検証します。
<img src="files/data_split.png" width="600px;"/>
このチュートリアルでは、機械学習のアルゴリズムに基づいた予測モデルを構築しますが、
機械学習のアルゴリズムには一般にハイパーパラメタというものが存在します。
機械学習アルゴリズムには、内部パラメタとハイパーパラメタの2つのパラメタがあり、
内部パラメタは自動で学習しますが、ハイパーパラメタは予め設定する必要があります。
良い精度を出すためにはハイパーパラメタをこのうまく調整する必要があります。
実際に予測モデルを構築する際には、訓練用のデータをさらにtraining dataとvalidation dataに分けて、
あるハイパーパラメタでtraining dataから予測モデルを構築しvalidation dataで精度を検証し、
別のハイパーパラメタで予測モデルを構築しvalidation dataで精度を検証し、、、
ということを繰り返します。
最終的な精度評価に際しては、
validation dataへの検証実験で良い結果となったハイパーパラメタで構築された予測モデルを今度はtest dataに適用し、
実際に未知のデータに対して、どれくらいの精度で予測できるのかということ検証します。
ここでは、便宜的に訓練データの最後の1週間の期間(2012-06-17 00:00:00から2012-06-23 23:59:59)をvalidation dataとし、
残りの約1年の期間をtraining dataとして、データを分割します。
Can you write Python code for it?

validation_start = datetime.strptime('2012-06-17 00:00:00', '%Y-%m-%d %H:%M:%S')
validation_end = validation_start + timedelta(days=7)
dbcur.execute(''' DROP TABLE IF EXISTS coupon_visit_train_training;''')  # チュートリアルの便宜上一回削除します。
dbcur.execute(''' CREATE TABLE IF NOT EXISTS coupon_visit_train_training LIKE coupon_visit_train;''')
dbcur.execute('''
    INSERT INTO coupon_visit_train_training
        SELECT *
        FROM coupon_visit_train
        WHERE i_date >= "2011-07-01 00:00:00" AND i_date < %s
    ;
''', (validation_start, ))

dbcur.execute(''' DROP TABLE IF EXISTS coupon_visit_train_validation;''')  # チュートリアルの便宜上一回削除します。
dbcur.execute(''' CREATE TABLE IF NOT EXISTS coupon_visit_train_validation LIKE coupon_visit_train;''')
dbcur.execute('''
    INSERT INTO coupon_visit_train_validation
        SELECT *
        FROM coupon_visit_train
        WHERE i_date >= %s
    ;
''', (validation_start, ))


I am working on the file "Lecture_02.ipynb".
The first task is:
3. 実際に購買したクーポンと照らし合わせ、MAP@10を算出する。
Can you write Python code for it?

#  validation期間に購買したクーポンリストを抽出。
dbcur.execute('''
    SELECT
        user_id_hash, view_coupon_id_hash
        FROM coupon_visit_train_validation
        WHERE purchase_flg = 1
    ;
''')
user_pcoupon_true = {}
for row in dbcur.fetchall():
    if row[0] not in user_pcoupon_true:
        user_pcoupon_true[row[0]] = []
    user_pcoupon_true[row[0]].append(row[1])

#  ap10を算出する関数を定義。
def get_ap10(y_pred, y_true):
    ap10 = 0.
    y_true = set(y_true)
    for i in range(len(y_pred)):
        if y_pred[i] in y_true:
            c = set(y_pred[:i + 1])
            ap10 += len(y_true & c) / float(i + 1)
    ap10 /= min(len(y_true), 10)
    return ap10

map10 = 0.
n_purchased_user = 0.
for user_id in user_pcoupon_pred:
    if user_id not in user_pcoupon_true:
        # 当該ユーザがvalidation期間にcouponを買わなかった場合、
        # ap@10は0
        continue
    n_purchased_user += 1
    y_true = user_pcoupon_true[user_id]
    y_pred = user_pcoupon_pred[user_id]

    map10 += get_ap10(y_pred, y_true)

max_map10 = n_purchased_user / len(user_pcoupon_pred)
map10 /= len(user_pcoupon_pred)

print 'max_map@10: %.5f, map@10: %.5f' % (max_map10, map10)


I am working on the file "Lecture_02.ipynb".
The first task is:
Excercise
test dataに対して、同様の手法で予測・csvに結果を出力して、
コンペティションにサブミットしてください。
※1日に最大5回までしかサブミットできないので、注意してください。
4. 機械学習アルゴリズムによる予測モデルの構築
先の章では、モデリング対象や評価指標への理解を深めました。
具体的に先に挙げた、
- ユーザの属性データ(性別、年代、地域など)
- クーポンの属性データ(ジャンル、価格、地域など)
- いつどのユーザがどのクーポンを見たか、購買したか等のログデータ
のそれぞれの要素がどれぐらい将来の購買行動と関連があるのか、
を考える前に、まず簡単に機械学習アルゴリズムによる予測モデルを構築して、
アルゴリズムの内部がどうなっているのか、
アルゴリズムのinputとoutputがどうなっているのか、
見ていきましょう。
ここでは、下記の設定、
対象アルゴリズム：ロジスティック回帰
対象データ：ユーザの属性データとクーポンの属性データ
将来購買されるクーポンを予測するということをやってみましょう。
機械学習アルゴリズム
ロジスティック回帰は非常にシンプルなアルゴリズムで
下記の数式のように定義されています。
$$
v=Wx+b
$$
$$
y \simeq \tilde{y}=\frac{exp(v)}{\sum_{k=1}^{N}exp(v_k)}
$$
$x$：素性ベクトル(ユーザとクーポンの特徴を表現したベクトル)
$W$：素性ベクトルの各列をどれだけ評価するかのウェイトマトリックス(パラメタ)
$b$：バイアス項(パラメタ)
$N$：クラスの数(この場合、買われたか否か)
$v$：N次元ベクトル
$\tilde{y}$：各クラスが発生する確率
$y$：正解ベクトル
ロジスティック回帰は与えられた素性ベクトル$x$の各次元の重みを評価して、各クラスに所属する確率$\tilde{y}$を算出します。
モデルの学習の際には、$\tilde{y}$と$y$の誤差が最小となるように内部のパラメタ$W$と$b$を学習し、
予測の際には、入力$x$を計算式に当てはめ、出力結果$\tilde{y}$を見て、例えば、所属する確率が最大のクラスを予測結果とします。
つまり、学習の際は入力は$x$と$y$で、予測の際は入力は$x$のみです。
これは、他の多くの機械学習アルゴリズムに共通です。
機械学習ライブラリの1つであるscikit-learnは、
多くの機械学習アルゴリズムに対して共通のフレームワークで提供され内部がブラックボックス化されており、
中身が分からなくても簡単に利用できるものとなっています。
ここでは、scikit-learnで提供されているロジスティック回帰を利用していきます。
データ加工(素性/ラベル作成)
簡単のためログデータを用いずに、
ユーザの属性データとクーポンの属性データのみを用いて素性$x$を作成します。
特徴ベクトルを作成する際には、
1. 各次元のスケールを合わせる
2. 連続値で表現すべきでないところは、フラグにする
を意識しましょう。
ユーザの特徴ベクトル
sequel proでuser_listテーブルを見てみると、
素性に利用しやすそうな属性として、性別、年齢、場所があります。
性別と場所は連続値でないため、フラグで表現します。
年齢のような連続値は、場合によってはフラグとした方が良い時もありますが、
ここでは、連続値として扱います。
sequel proで年齢のカラムをソートしてみると、
最小値が15最大値が80なので素性を作る時はスケールを合わせるために、
年齢値から15引いて65で割ります。
このように、[0, 1]の値での表現のように最小値と最大値を整えるスケーリングをmini max scaling等と呼びます。
他には平均0分散1の標準正規分布に変換するstandard scalingがあります。
※いずれもscikit-learnのpreprocessingモジュール(http://scikit-learn.org/stable/modules/classes.html#module-sklearn.preprocessing )に実装されています。
Can you write Python code for it?

# ユニークな都道府県リストの取得
dbcur.execute(''' SELECT pref_name FROM prefecture_locations ORDER BY pref_name ; ''')
pref_data = []
for row in dbcur.fetchall():
    pref_data.append(row[0])

# ユーザの素性を作成。(ユーザの素性はtraining、validation, testで共通)
dbcur.execute('''
    SELECT
        t1.user_id_hash,
        IF(t1.sex_id = 'm', 1, 0),
        (t1.age-15)/65,
        ''' + ', '.join([u'IF(t1.pref_name = "' + p[0] + u'", 1, 0)' for i, p in enumerate(pref_data)]) + '''
    FROM user_list AS t1
''')

user_feature = {}  # ユーザの素性ベクトル
for row in dbcur.fetchall():
    user_feature[row[0]] = row[1:]


I am working on the file "Lecture_02.ipynb".
The first task is:
まず、同じ地域からの購買よりも，異なる地域からの購買の方が多いことが分かります。
また、same_rateとdiff_rateを比べると、同じ地域での閲覧に於ける購買の割合より、異なる地域での閲覧に於ける購買の割合の方がやや大きいことが分かります。
そもそも、異なる地域での閲覧が多いということ自体が先の仮定で意識したものとは異なっていた、ということもありますが、
検証したかった仮説はどうやら、正しくなさそうです。
2. ユーザは女性の方が男性より割引率の高いクーポンを購買する可能性が高い。
この仮説も、時間軸とは関係のなく、いわゆる分析的な集計である為、ここでは訓練データ全体で集計します。
Can you write Python code for it?

dbcur.execute('''
    SELECT
        t1.sex_id,
        AVG(t1.discount_rate_view),
        AVG(t1.discount_rate_purchase)
    FROM (
        SELECT
            t1.user_id_hash,
            t1.sex_id,
            AVG(100 - t3.price_rate) AS discount_rate_view,
            COALESCE(SUM(IF(t2.purchase_flg, 100 - t3.price_rate, 0)) / SUM(t2.purchase_flg), 0) AS discount_rate_purchase

        FROM user_list AS t1
        LEFT JOIN coupon_visit_train AS t2 ON t1.user_id_hash = t2.user_id_hash
        LEFT JOIN coupon_list_train AS t3 ON t2.view_coupon_id_hash = t3.coupon_id_hash
        GROUP BY t1.user_id_hash
    ) AS t1
    GROUP BY t1.sex_id
    ;
''')
data = []
for row in dbcur.fetchall():
    row = list(row)
    row[1] = float(row[1])
    row[2] = float(row[2])
    data.append(tuple(row))
for row in data:
    print 'sex_id: %s, discount_rate_view: %.2f, discount_rate_purchase: %.2f' % (row)


I am working on the file "Lecture_02.ipynb".
The first task is:
クーポンの特徴ベクトル
アイテムは、比較的用意に利用できそうなフラグのカラムが多いです。
ここでは、簡単のためフラグのみを、
学習に利用するクーポンはvalidation期間の前の7日で購買可能なデータ
を利用して特徴ベクトルを作っていきます。
※実際には、より多くのデータを使った方が精度は出る傾向にあります。
Can you write Python code for it?

training_start = validation_start - timedelta(days=7)  # 訓練開始日時を算出。

# カテゴリリストの取得
dbcur.execute(''' SELECT DISTINCT(capsule_text) FROM coupon_list_train ORDER BY capsule_text;''')
capsule_data = []
for row in dbcur.fetchall():
    capsule_data.append(row[0])

# ジャンルリストの取得
dbcur.execute(''' SELECT DISTINCT(genre_name) FROM coupon_list_train ORDER BY genre_name;''')
genre_data = []
for row in dbcur.fetchall():
    genre_data.append(row[0])

# 大エリアリストの取得
dbcur.execute(''' SELECT DISTINCT(large_area_name) FROM coupon_list_train ORDER BY large_area_name;''')
larea_data = []
for row in dbcur.fetchall():
    larea_data.append(row[0])

# 都道府県リストの取得
dbcur.execute(''' SELECT DISTINCT(ken_name) FROM coupon_list_train ORDER BY ken_name;''')
pref_data = []
for row in dbcur.fetchall():
    pref_data.append(row[0])

# 小エリアリストの取得
dbcur.execute(''' SELECT DISTINCT(small_area_name) FROM coupon_list_train ORDER BY small_area_name;''')
sarea_data = []
for row in dbcur.fetchall():
    sarea_data.append(row[0])

def get_item_feature(f_date, t_date):
    #  クーポンの素性を作成する関数。
    # @f_date：対象期間の開始日時
    # @t_date：対象期間の終了日時
    # テーブルが訓練用のテーブルとなっている為、training とvalidationのデータを作成する際にしか利用できない。
    dbcur.execute('''
        SELECT
            coupon_id_hash,
            ''' + ', '.join([u'IF(capsule_text = "' + p[0] + u'", 1, 0)' for i, p in enumerate(capsule_data)]) + ''',
            ''' + ', '.join([u'IF(genre_name = "' + p[0] + u'", 1, 0)' for i, p in enumerate(genre_data)]) + ''',
            COALESCE(CAST(usable_date_mon AS SIGNED), 0),
            COALESCE(CAST(usable_date_tue AS SIGNED), 0),
            COALESCE(CAST(usable_date_wed AS SIGNED), 0),
            COALESCE(CAST(usable_date_thu AS SIGNED), 0),
            COALESCE(CAST(usable_date_fri AS SIGNED), 0),
            COALESCE(CAST(usable_date_sat AS SIGNED), 0),
            COALESCE(CAST(usable_date_sun AS SIGNED), 0),
            COALESCE(CAST(usable_date_holiday AS SIGNED), 0),
            COALESCE(CAST(usable_date_before_holiday AS SIGNED), 0),
            ''' + ',...


I am working on the file "Lecture_02.ipynb".
The first task is:
ユーザ・クーポンの特徴ベクトルと正解ラベルの割当
ユーザ特徴ベクトルとクーポン特徴ベウトルの関係性を評価する方法は無数にあります。
ベクトルを横に結合するという単純な方法や、
単純な結合に加えてそれぞれの掛け合わせも加えるという方法(pair-wise)、
などが代表的な簡単な方法です。
素性が長くなれば、それだけ情報量が多くなりますが、一方で過学習(Over fitting)が起きやすくなります。
「過学習が起きる」とは、訓練データでモデルを学習させる際に、過剰に訓練データにフィッティングして、
validationデータやtest dataに対する汎化性能が低いモデルが出来てしまう現象で、
本来、未知のデータに対する汎化・予測性能のあるモデルを構築したいため、
過学習を防ぐ必要があります。
しかし、過学習を防ぐプロセスはやや煩雑であるため、
ここでは意識せず、単にベクトルを横に結合するという方法ですすめます。
Can you write Python code for it?

def get_purchased_coupons(f_date, t_date):
    # 実際に購買されるクーポンの取得
    # @f_date：対象期間の開始日時
    # @t_date：対象期間の終了日時
    dbcur.execute('''
        SELECT user_id_hash, view_coupon_id_hash
        FROM coupon_visit_train
        WHERE i_date >= %s AND i_date < %s AND purchase_flg = 1
        ORDER BY user_id_hash, view_coupon_id_hash
        ;
    ''', (f_date, t_date))

    purchased_items = {}  # 各ユーザがどのクーポン群を購入するかを辞書型で返す。
    for row in dbcur.fetchall():
        if row[0] not in purchased_items:
            purchased_items[row[0]] = set([])
        purchased_items[row[0]].add(row[1])

    return purchased_items

user_pcoupon_train = get_purchased_coupons(training_start, validation_start)  # training 期間に各ユーザが実際に買ったクーポン
user_pcoupon_valid = get_purchased_coupons(validation_start, validation_end)  # validation 期間に各ユーザが実際に買ったクーポン

n_pairs_train = len(user_feature) * len(item_feature_train)  # ユーザ数×trainingクーポン数
n_pairs_valid = len(user_feature) * len(item_feature_valid)  # ユーザ数×validation クーポン数
print 'n_train_datasets: %d, n_validation_datasets: %d, n_puser: %d' %(n_pairs_train, n_pairs_valid, len([1 for a in user_pcoupon_train if len(a) > 0]))


I am working on the file "Lecture_02.ipynb".
The first task is:
ランダム推定・MAP@10の評価
先にも触れましたが、本コンペティションではMAP@10というスコアを競い合っています。
提出のフォーマットは、https://www.kaggle.com/c/coupon-purchase-prediction/details/evaluation に
記載されています。
訓練データをtraining dataと validation dataに分割したので、
試しに、各ユーザが購買するクーポンをランダムに推定して、
・MAP@10の性質
・ランダム推定の精度
・サブミッションの形式
をざっくり把握していきましょう。

validation期間に購買されうるクーポンを抽出する。
抽出したクーポン群から各ユーザが購買するクーポンをランダムに10個選び、予測結果とする。
実際に購買したクーポンと照らし合わせ、MAP@10を算出する。
サブミッション用のcsvファイルを作成する。

の4ステップで見ていきます。
1. validation期間に購買されうるクーポンを抽出する。
Can you write Python code for it?

# validation 期間に購買されうるクーポンの抽出
dbcur.execute('''
    SELECT
        coupon_id_hash
        FROM coupon_list_train
        WHERE
            NOT (dispend <= %s OR dispfrom > %s)
    ;
''', (validation_start, validation_end))
coupon_ids = []
for row in dbcur.fetchall():
    coupon_ids.append(row[0])


I am working on the file "Lecture_02.ipynb".
The first task is:
先ほどの、ランダム予測よりだいぶ上がったようです。
今回の例では、ユーザの特徴とクーポンの特徴を単純に結合しましたが、
モデルの入力である素性は無数の形式が存在します。
機械学習アルゴリズムを利用する際、この素性を如何に設計するかは非常に重要です。
他に予測精度をあげるポイントとして、
・機械学習アルゴリズムを選ぶ。
・機械学習アルゴリズムのハイパーパラメタを調整する。
・前処理をちゃんとやる。
・目的関数を正しく選ぶ。
などがあります。
興味がある方は、適用を検討してみてください。
Excercise

同様の方法を test dataに適用、各ユーザの購買クーポンを予測し、submissionしてみてください。
学習に用いるデータ期間を7日から増やして、精度の変化を見てみてください。

5. データの概観把握・予測モデルの改善
TODO: もっと多くの実例を取り上げる。
このコンペティションの最終的なゴールは、各ユーザが将来どのクーポンを購入するかを予測することです。
先のように、ランダムに選ぶよりは、ユーザやクーポンの特徴をモデリングした方が将来の購買するクーポンを予測しやすいことが分かりました。
では、将来ユーザがどのクーポンを購入するかは、何と関係があるでしょうか。
その関係性を捉えることで、モデリングの際によりよい結果が得られることが多いです。
下記の2ステップで、購買確率の予測に寄与する指標が何かを探していきます。
1. 構築した予測モデルにおける各素性の予測への寄与度を評価する。
2. 過学習を理解し防ぐ。
3. 購買行動への仮説を立て、データを集計&可視化し検証する。
5-1. 構築した予測モデルにおける各素性の予測への寄与度を評価する。
機械学習のアルゴリズムの中には、
予測モデルの構築に利用した各素性($x_0, x_1,,, x_n$)がそれぞれ、
予測にどれだけ寄与したか、もしくは、どのように寄与したか
をパラメタとして保持しているものがあります。
先の4章で構築した予測モデルに利用したロジスティック回帰というアルゴリズムもそういったものの一つです。
ロジスティック回帰の場合は、
下記の数式の$W$が各素性($x_0, x_1,,, x_n$)の重みになっているわけです。
$$
v=Wx+b
$$
$$
y \simeq \tilde{y}=\frac{exp(v)}{\sum_{k=1}^{N}exp(v_k)}
$$
この重みを見ることで、例えば、
全く予測に寄与していない素性を見つけることや、
対象のデータに関する知見を獲得すること
ができます。
それでは、実際に見てみましょう。
scikit-learnのロジスティック回帰の各素性の重みの大きさの大きい順に10個、小さい順に10個
出力してみましょう。
Can you write Python code for it?

n = 10
#print ['sex', 'age'] + pref_data
label_names = (
    ['user_' + c for c in (['sex', 'age'] + pref_data)]
    + ['item_' + c for c in (
       capsule_data
       + genre_data
       + ['mon', 'tue', 'wed', 'thu', 'fri', 'sat', 'sun', 'holiday', 'before_holiday']
       + larea_data
       + pref_data
       + sarea_data

      )]
)
print "合計の素性数：%d" % (model.coef_.shape[1])
print "降順に10個"
for i in numpy.argsort(abs(model.coef_[0]))[-n:][::-1]:
    print 'index: %d, name: %s, %.4f' % (i, label_names[i], model.coef_[0][i])

print "昇順に10個"
for i in numpy.argsort(abs(model.coef_[0]))[:n]:
    print 'index: %d, name: %s, %.4f' % (i, label_names[i], model.coef_[0][i])


I am working on the file "Lecture_02.ipynb".
The first task is:
関係性についての仮説をたてる
どういったユーザがどういったクーポンを購買するかについての仮説を立てます。
今回のデータセットでは、例えば、下記のような仮説が立てられます。
1. ユーザは自分の地域と同じ地域を対象とするクーポンを購買する可能性が高い。
2. ユーザは女性の方が男性より割引率の高いクーポンを購買する可能性が高い。
3. ユーザは最後に購買したクーポンと同じジャンルのクーポンを、将来購買する可能性が高い。
4. (過去に購買したクーポンがない)ユーザは最後に閲覧したクーポンと同じジャンルのクーポンを、将来購買する可能性が高い。
仮説を立てる際に意識して頂きたいことは、
立てた仮説がどういったユーザがどういったクーポンを購買するかを具体的に表現していて、
そこから、どのようにデータを集計し、検証を行うかが具体的に分かるレベルで考えるということです。
たとえば、
「ユーザは過去の行動に応じて、将来に購買するクーポンを変える可能性が高い。」
という仮説をたてたとしましょう。しかし、この抽象度では具体的に何をすべきか分かりません。
従って、正しいかもしれないが、次のアクションに繋がらないという点で、役に立たない(≒価値がない)仮説だと言えます。
また、別の角度から見ると、立てた仮説はグレーであればあるほど、白黒はっきりさせた(検証した)あとの有用性が大きいことが多いです。
実際に、「データを分析する」際には、この「仮説を立てて、重要なところから取りかかる」ということが非常に重要です。
スキルは所詮スキルでしかなく、使いどころを誤るといたずらに時間を溶かす結果になりますので、是非意識してもらえればと思います。
仮説を検証するために、データを集計&可視化する
具体例として挙げた仮説を一つずつ検証いていきましょう。
1. ユーザは自分の地域と同じ地域を対象とするクーポンを購買する可能性が高い。
この仮説は、時間軸とは関係のなく、いわゆる分析的な集計である為、ここでは訓練データ全体で集計します。
Can you write Python code for it?

dbcur.execute('''
    SELECT
        AVG(same_pref_purchase_cnt),
        AVG(same_pref_view_cnt),
        AVG(same_pref_purchase_cnt / same_pref_view_cnt),
        AVG(diff_pref_purchase_cnt),
        AVG(diff_pref_view_cnt),
        AVG(diff_pref_purchase_cnt / diff_pref_view_cnt)
    FROM (
        SELECT
            t1.user_id_hash,
            SUM(t1.pref_name = t3.ken_name AND purchase_flg = 1) AS same_pref_purchase_cnt,
            SUM(t1.pref_name = t3.ken_name) AS same_pref_view_cnt,
            SUM(t1.pref_name != t3.ken_name AND purchase_flg = 1) AS diff_pref_purchase_cnt,
            SUM(t1.pref_name != t3.ken_name) AS diff_pref_view_cnt
        FROM user_list AS t1
        LEFT JOIN coupon_visit_train AS t2 ON t1.user_id_hash = t2.user_id_hash
        LEFT JOIN coupon_list_train AS t3 ON t2.view_coupon_id_hash = t3.coupon_id_hash
        WHERE t1.pref_name != ""
        GROUP BY t1.user_id_hash
    ) AS t1

    ;
''')
data = None
for row in dbcur.fetchall():
    data = row
print 'same_purchase: %.2f, same_view: %.2f, same_rate: %.2f, diff_purchase: %.2f, diff_view: %.2f, diff_rate: %.2f' % (data)


I am working on the file "Lecture_07.ipynb".
The first task is:
Could the game ever continue forever on to infinity?
Recall that we have the following solution to the difference equation for the Gambler's Ruin game:
\begin{align}
    P_i &=
        \begin{cases}
            \frac{1-\left(\frac{q}{p}\right)^i}{1-\left(\frac{q}{p}\right)^N}  & \quad \text{ if } p \neq q \
            \frac{i}{N}  & \quad \text{ if } p = q \
        \end{cases}
\end{align}
The only time you'd think the game could continue on to infinity is when $p=q$. But
\begin{align}
    P(\Omega) &= 1\
    &= P(\text{A wins all}) + P(\text{B wins all}) \
    &= P_i + P_{N-i} \
    &= \frac{i}{N} + \frac{N-i}{N}
\end{align}
The above implies that aside from the case where $A$ wins all, and the case where $B$ wins all, there is no other event in $\Omega$ to consider, hence the game can never continue on to infinity without either side winning.
This also means that unless $p=q$, you will lose your money, and the only question is how fast will you lose it.

Random Variables
Consider these statements:
\begin{align}
    x + 2 &= 9 \
    x &= 7
\end{align}
What is a variable?
* variable $x$ is a symbol that we use as a substitute for an arbitrary constant value.
What is a __random__ variable?

This is not a variable, but a function from the sample space $S$ to $\mathbb{R}$.
It is a "summary" of an aspect of the experiment (this is where the randomness comes from)

Here are a few of the most useful discrete random variables.

Bernoulli Distribution
Description
A probability distribution of a random variable that takes the value 1 in the case of a success with probability $p$; or takes the value 0 in case of a failure with probability $1-p$.
A most common example would be a coin toss, where heads might be considered a success with probability $p=0.5$ if the coin is a fair.
A random variable $x$ has the Bernoulli distribution if
- $x \in {0, 1}$
- $P(x=1) = p$
- $P(x=0) = 1-p$
Notation
$X \sim \operatorname{Bern}(p)$
Parameters
$0 < p < 1 \text{, } p \in \mathbb{R}$
Probability mass function
The probability mass...
Can you write Python code for it?

import matplotlib
import numpy as np
import matplotlib.pyplot as plt

from matplotlib.ticker import (MultipleLocator, FormatStrFormatter,
                               AutoMinorLocator)
from scipy.stats import binom

%matplotlib inline

plt.xkcd()

_, ax = plt.subplots(figsize=(12,8))

# a few Binomial parameters n and p
pop_sizes = [240, 120, 60, 24]
p_values = [0.2, 0.3, 0.4, 0.8]
params = list(zip(pop_sizes, p_values))

# colorblind-safe, qualitative color scheme
colors = ['#a6cee3','#1f78b4','#b2df8a','#33a02c']

for i,(n,p) in enumerate(params):
    x = np.arange(binom.ppf(0.01, n, p), binom.ppf(0.99, n, p))
    y = binom.pmf(x, n, p)
    ax.plot(x, y, 'o', ms=8, color=colors[i], label='n={}, p={}'.format(n,p))
    ax.vlines(x, 0, y, color=colors[i], alpha=0.3)

# legend styling
legend = ax.legend()
for label in legend.get_texts():
    label.set_fontsize('large')
for label in legend.get_lines():
    label.set_linewidth(1.5)

# y-axis
ax.set_ylim([0.0, 0.23])
ax.set_ylabel(r'$P(x=k)$')

# x-axis
ax.set_xlim([10, 65])
ax.set_xlabel('# of successes k out of n Bernoulli trials')

# x-axis tick formatting
majorLocator = MultipleLocator(5)
majorFormatter = FormatStrFormatter('%d')
minorLocator = MultipleLocator(1)
ax.xaxis.set_major_locator(majorLocator)
ax.xaxis.set_major_formatter(majorFormatter)

ax.grid(color='grey', linestyle='-', linewidth=0.3)

plt.suptitle(r'Binomial PMF: $P(x=k) = \binom{n}{k} p^k (1-p)^{n-k}$')

plt.show()


I am working on the file "Lecture_1.ipynb".
The first task is:
Alternatively you can put the specific function you want to load here.
Can you write Python code for it?


I am working on the file "Lecture_1.ipynb".
The first task is:
An asterisk in this context means ‘everything in the module’.
Can you write Python code for it?

from math import log10
a = log10(1000000)
print(str(a))


I am working on the file "Lecture_1.ipynb".
The first task is:
Getting user input
The input() function makes the program wait for the user to input something on the keyboard and hit enter before continuing
Can you write Python code for it?


I am working on the file "Lecture_1.ipynb".
The first task is:
Missing brackets
Can you write Python code for it?


I am working on the file "Lecture_1.ipynb".
The first task is:
Modules in Python

There is no need to re-invent the wheel - unless you’re learning!
Modules are pre-written pieces of code (such as useful functions) that are designed for specific tasks
You need to call these at the top of the script
Before you use a module you have to reference it in your script like this:
Can you write Python code for it?


I am working on the file "Lecture_1.ipynb".
The first task is:
Note: Knowing how to fix errors in the logic comes with practice, good commenting of your code and planning your code before you write it (e.g. Draw a flow diagram)
Can you write Python code for it?

x = 3
y = 7
average = (x + y) / 2
print(average)


I am working on the file "Lecture_1.ipynb".
The first task is:
Special characters in Strings
Strings can contain non-text characters with special meanings
Most useful are:
\n  newline (end of line character)
\t  tab
Can you write Python code for it?

print("Hello, World!\n")    # adds a new line at the end
print("Hello\tWorld!")       # puts a tab character in


I am working on the file "Lecture_1.ipynb".
The first task is:
These are incredibly useful (topic of a later lecture)
Way of matching and manipulating strings e.g. “Find and replace” operation

One example is sub()
Substitutes one pattern in a string for another
Can you write Python code for it?

from re import sub
string = "Al Darbi"
string_mod1 = sub('l','C',string)
string_mod2 = sub('i','y',string_mod1)
print(string)
print(string_mod2)


I am working on the file "Lecture_1.ipynb".
The first task is:
Variable name spelling or capitalisation
Indentation (covered in a later lecture)
Runtime errors
Can you write Python code for it?


I am working on the file "Lecture_1.ipynb".
The first task is:
Virtually everything in python is an Object.
Objects come in different types here are some examples:

bool (short for boolean) is a True or False
int (short for integer) is a whole number
str (short for string) is text
float is a number with a decimal point.

Python will automatically assign a type but you can and should override this when needed.
Can you write Python code for it?

Operations on variables

first_name = "Alistair"
last_name = "Darby"
fullname = first_name + last_name
print (fullname)


I am working on the file "Lecture_1.ipynb".
The first task is:
When used with strings the '+' operator will concatenate them
When used with numbers  '+' will behave as you would expect
Can you write Python code for it?

aphids = 5
beetles = 12
total_insects = aphids + beetles
print ("Number of insects = ", total_insects)

aphids = 5
beetles = 12
total_insects = Aphids + Beetles
total_insects_string = str(total_insects)
print("Number of insects = " + total_insects_string)


I am working on the file "Lecture_1.ipynb".
The first task is:
You can also add an input prompt
name = input("What is your name?")
print("Hello ", name, "!")
Incorporating input
Can you write Python code for it?

# This script calculates the number of cats you need
rooms = input("How many rooms do you have? ")
cats = input("How many cats fit in a room? ")
Total_cats = int(rooms) * int(cats)
print("You require " , Total_cats , " cats")


I am working on the file "Lecture_1.ipynb".
The first task is:
You will hear lots more about regular expressions later on, but there is just one example included here, in case you’re interested in reading more about them (might also be useful for the part of the assignment...)
Making errors / refining scripts

Scripts almost never work first time
Usually require fixing small mistakes:
Syntax (i.e. Python gives an error when you try to run)
Runtime (something goes wrong while it is running)
Logic (script doesn’t behave as you expect)

Often error messages are not very informative
Takes a bit of practice to work out what might be causing an error

Tip: One way to check for syntax errors is to read each line of code from the end to the beginning. This forces you to look at each line without attaching meaning.
Syntax Errors
Most syntax errors are due to incorrect formatting like:
Missing matching quotes
Can you write Python code for it?


I am working on the file "Lecture_1.ipynb".
The first task is:
math is the name of the module
An asterisk in this context means ‘everything in the module’.
Alternatively you can put the specific function you want to use here
Can you write Python code for it?

from math import *
a = log10(1000000)
print(str(a))


I am working on the file "Lecture_2.ipynb".
The first task is:
CONDITIONALS

Often you will want to add logic to your programs

This is done with simple statements (in nearly all languages):

if
else
elif (aka else if)

In Python, code within a conditional or a loop is denoted by a : followed by indentation (this will become clearer with examples)

Comparisons (logical operators)
An if statement is either true or false (synonymous with 1 or 0), these are known as Boolean values.
Can you write Python code for it?

x = 5
y = 5
if x == y :
    print("x and y are the same")


I am working on the file "Lecture_2.ipynb".
The first task is:
Examples of logical operators in action
Can you write Python code for it?

a = 5
b = 4

if a < b:
    print("a is less than b") #conditional block must be indented four spaces or a tab

else:
    print("a is not less than b")

name1 = "Alistair"
name2 = "Alastair"

if name1 == name2:
    print("names are the same")
else:
    print("names are not the same")


I am working on the file "Lecture_2.ipynb".
The first task is:
Finding the length of a list
Can you write Python code for it?

exam_scores = [67,78,94,45,55]
length = len(exam_scores)
print("number of scores:",length)


I am working on the file "Lecture_2.ipynb".
The first task is:
Note:  you can have make statements as nested and complicated as you need. BUT...
 1. Your indentation Must be correct
 2. Always work out your logic in advance and add comments to your code (otherwise your code won’t do what you want it to do)
Using elif
elif is very useful if you want to test lots of conditions in order
Can you write Python code for it?

module_code = "BIOL734"
if module_code == "BIOL007":
    module_name = "Statistics for Life Science"

elif module_code == "BIOL733":
    module_name = "Perl Programming"

elif module_code == "BIOL734":
    module_name = "Post-genomic technologies"

else:
    module_name = "Unknown module code"
print("The module is " + module_name + "\n")


I am working on the file "Lecture_2.ipynb".
The first task is:
Note: As with conditionals, indentation is crucial to denote the code that you want to run within the loop

for initiates the loop
x is a variable that represents the current loop item (in this case an integer representing the number of the loop)
range() creates a list of numbers between two given integers

Loop through a list
Can you write Python code for it?

exam_scores = [67,78,94,45,55]

counter = 1
for x in exam_scores:
    print("score " + str(counter) + ": " + str(x))
    counter +=1 # “+=“ is the same as counter = counter + 1  incrementing


I am working on the file "Lecture_2.ipynb".
The first task is:
Note: If you want to print all elements individually (and do something else with them), you should loop through them – later in lecture...
Adding to lists
You can add values to the end of a list using the append() method
Append is a method so it must be called on a list
Can you write Python code for it?

exam_scores = [67,78,94,45,55]
exam_scores.append(32) #add a variable to this list
print ("Here are the scores:",exam_scores)


I am working on the file "Lecture_2.ipynb".
The first task is:
Other useful list methods
Can you write Python code for it?

# extend(list) - appends another list

exam_scores = [67,78,94,45,55]
exam_scores2 = [54,73,65]
exam_scores.extend(exam_scores2)
print (exam_scores)

# insert(index,item)-insert an item at a given index

exam_scores = [67,78,94,45,55]
exam_scores.insert(4,90)
print (exam_scores)

exam_scores = [67,78,94,45,55]
exam_scores.pop(2)
print (exam_scores)


I am working on the file "Lecture_2.ipynb".
The first task is:
Slicing can also be used with strings in the same way. Just imagine that each character in the string is the same as a list element
Changing a list value
Changing a value in a specific position iss similar to defining a variable
Can you write Python code for it?

exam_scores = [67,78,94,45,55]
exam_scores[2] = 90
print("score: " ,exam_scores[2])


I am working on the file "Lecture_2.ipynb".
The first task is:
Sometimes we may want to capture the popped value.
Can you write Python code for it?

exam_scores = [67,78,94,45,55]
popped_value = exam_scores.pop(2)
print (exam_scores)
print("Popped value=", popped_value)

exam_scores = [67,78,94,45,55]
exam_scores.reverse()
print (exam_scores)


I am working on the file "Lecture_2.ipynb".
The first task is:
Sorting big to small
Can you write Python code for it?

values = [5, 7, 4, 6, 1, 2, 45, 12]
sorted_values = sorted(values,reverse=True)
print("Sorted values:", sorted_values)

values = [5, 7, 4, 6, 1, 2, 45, 12]
values.sort(reverse=True)
print("Sorted values:",values)


I am working on the file "Lecture_2.ipynb".
The first task is:
Sorting lists numerically
Can you write Python code for it?

values = [5, 7, 4, 6, 1, 2]
sorted_values = sorted(values)
print("Sorted values:", sorted_values)


I am working on the file "Lecture_2.ipynb".
The first task is:
This is wrong!
Statement really is:
“If the day is not Monday and the day is not Friday, I am researching”
Can you write Python code for it?

day = "Monday"

if day != "Monday" and day != "Friday":
    print("Alistair is researching\n")

else:
    print("Alistair is not researching\n")


I am working on the file "Lecture_2.ipynb".
The first task is:
Using or

Code to test if either mark is less than 50
Can you write Python code for it?

biol733_mark = 34
biol734_mark = 55

if biol733_mark <50  or biol734_mark < 50:
    print("You've failed a module :-( ")

else:
    print("You've passed your modules :-) ")


I am working on the file "Lecture_2.ipynb".
The first task is:
i.e. with negatives both conditions have to be true
Nested statements
Can you write Python code for it?

a = 4
b = 4
c = 6

if a == b:
    print("a equals b")

    if b >= c:
        print("and b is greater or equal to c")

    else:
        print("and b is less than c")

else:
    print("a doesn't equal b")


I am working on the file "Lecture_2.ipynb".
The first task is:
sort() – is a method and will replace the original list object
Can you write Python code for it?

names = ["James", "John", "Andy", "Ben", "Chris", "Thomas"]
names.sort()
print("Sorted names:",names)


I am working on the file "Lecture_3.ipynb".
The first task is:
A list needs to be created (a.k.a. initialised) before you can apply methods to it.
Initialising the list before running the loop is important. If we had tried to initialise within the loop structure the list would be reset after each iteration of the loop!
Dictionaries

Dictionaries are a lookup key-value system:

Very similar to a list except the index is replaced by a defined key
The keys and values can be anything you like.
However, keys must be unique.

Simple to declare:

A variable: Value = 10
A list: Values = [10,11,12,13]
A dictionary: Value_lookup = {"Apple" : "Pie", "Banana" : "split", 3 : 60}

You can retrieve elements from a list using their numerical index whereas a dictionary can use any value specified by the programmer.
Declaring a dictionary requires curly braces {} and a colon : between key and value
Dictionary example
Can you write Python code for it?

student_records = {20071213: "Alistair Darby"} # declaring a dictionary
student_records[20081423] = "John Smith"   # adding some more records
student_records[20096137] = "Jane Doe"
student_records[20109334] = "Fred Blogs"

print(student_records[20081423])           # printing a specific record


I am working on the file "Lecture_3.ipynb".
The first task is:
Here we setup a temporary key (called name) and then loop through every key, retrieving the corresponding value for each key
Manipulating strings
Slicing is very useful for manipulating strings and lists
Can you write Python code for it?

DNA = "ACTGATCGACTGATCGATCGA"

for index in range(0, len(DNA), 3): # Remember - range requires the arguments (start, stop,  step). Step has a default value of 1

    chunk = DNA[index:index+3] # Remember - you can slice a string by giving [start:stop] indexes
    print(chunk,end=" ")


I am working on the file "Lecture_3.ipynb".
The first task is:
Remember – the numbering starts from 0!

index() and slicing can also be used with lists
Can you write Python code for it?

codons = ["ATG", "GAC", "TTG"]
print("Index for TTG : ", codons.index( "TTG" ))
print("Index for ATG : ", codons.index( "ATG" ))


I am working on the file "Lecture_3.ipynb".
The first task is:
Small mistakes can be difficult to spot. You will avoid and spot these mistakes with practice.
Recap on lists

A list stores an ordered list of values
Loop through all elements using for

Useful list functions:

extend(list) - adds a list to the end of a list

append(value) - adds a value to the end of a list

Also insert(index, value) and pop(index)

sort

sorted_names = sorted(names)  #using the function
names.sort()                  #using the method

Recap on list methods
Can you write Python code for it?

values = [5, 7, 4, 6, 1, 2]

print (values)

print(values[3]) #By the position

for temp_value in values: #Using a loop
    print (temp_value)


I am working on the file "Lecture_3.ipynb".
The first task is:
The index method can be used to find the position of specific substrings
Can you write Python code for it?

DNA = "ACTGATCGACTGATCGATCGA"
print("Index for CGA : ", DNA.index( "CGA" ))
print("Index for GAT : ", DNA.index( "GAT" ))


I am working on the file "Lecture_3.ipynb".
The first task is:
The keys() method returns a view of the keys which can be looped through like a list or turned into a list
delete a record from a dictionary
Can you write Python code for it?

student_records = {20071213: "Andy Jones",
                   20081423 : "John Smith",
                   20096137 :"Jane Doe"}

del student_records[20081423]
print("Total number of students = " , len(student_records.keys()))

student_records = {20071213: "Andy Jones",
                    20081423 : "John Smith",
                    20096137 :"Jane Doe"}

for name in student_records.keys():
     print(name, ":", student_records[name])


I am working on the file "Lecture_3.ipynb".
The first task is:
The two print statements illustrate alternatives
When using multiple formatters, variables are provided in order within brackets
Be careful with multiple parentheses!
Get dictionary size – keys method
Can you write Python code for it?

student_records = {20071213: "Andy Jones"}
student_records[20081423] = "John Smith"
student_records[20096137] = "Jane Doe"

print("Total number of students =" , len(student_records.keys()))
print(student_records.keys())
list(student_records.keys())


I am working on the file "Lecture_3.ipynb".
The first task is:
Two common ways to access elements in an list:
By the position (a.k.a. index)  -  values(2)
or
Use a loop to grab every value and put it into a temporary variable (here called temp_value
Recap on list methods
Can you write Python code for it?

names = ["Andy", "Bob", "Chris"]

removed_name = names.pop(0)
print (removed_name)
print(names)


I am working on the file "Lecture_3.ipynb".
The first task is:
Vanilla use of print
Can you write Python code for it?

message = "Hello world"
print ("My 1st message is:", message)
print ("My 2nd message is: %s" % message)


I am working on the file "Lectures/Debugging Notebooks.ipynb".
The first task is:
A nice cheatsheet:

Advantages

No installation required
Dynamic
Mature:
More features
Documentation (Stackoverflow ftw)
Less bugs

Disadvantages:

Very scary
Learning curve

Additional resources:

Introduction to PDB
The Glory of pdb's set trace

Better Way Jupyter Notebooks - pixie_debugger
Can you write Python code for it?

import contextlib

with contextlib.redirect_stdout(None):
    import pixiedust

%%pixie_debugger
import random
def find_max (values):
    max = 0
    for val in values:
        if val > max:
            max = val
    return max
find_max(random.sample(range(100), 10))


I am working on the file "Lectures/Debugging Notebooks.ipynb".
The first task is:
Advantages

Easy
Dynamic

Disadvantages:

Additional installation required
Not mature:
Documentation
Supported evaluation
Working on Jupyter notebooks but not in jupyterlab

Setup:

pip install pixiedust

Additional resources:

blog post

Better Way Jupyter lab - xpython
Can you write Python code for it?

import random
def find_max (values):
    max = 0
    for val in values:
        if val > max:
            max = val
    return max
find_max(random.sample(range(100), 10))


I am working on the file "Lectures/Generators.ipynb".
The first task is:
Memory of Genrators Vs Iterators 😇

For the generator's work, you need to keep in memory the variables of the generator function.
But you don't have to keep the entire collection in memory, so usually this is EXACTLY the trade-off you want to make.
Can you write Python code for it?

%%memit
g = generators(range(10**8))
print(sum(g))

%%memit
i = iterators(range(10**8))
print(sum(i))


I am working on the file "Lectures/Generators.ipynb".
The first task is:
Performance of Genrators Vs Iterators 😃
Can you write Python code for it?

%%time
g = generators(range(10**8))
print(sum(i))

%%time
i = iterators(range(10**8))
print(sum(i))


I am working on the file "Lectures/Lecture 10 - Graph Algorithms/Python Graphs - networkx intro .ipynb".
The first task is:
Let's play with some algorithms in class:
https://networkx.github.io/documentation/latest/reference/algorithms.html
Social Graph analysis algorithms
Betweenness:
https://networkx.github.io/documentation/latest/reference/algorithms.centrality.html#betweenness
EigenVector centrality
https://networkx.github.io/documentation/latest/reference/algorithms.centrality.html#eigenvector
Clustering per node
https://networkx.github.io/documentation/latest/reference/generated/networkx.algorithms.cluster.clustering.html
All Shortest Paths
https://networkx.github.io/documentation/latest/reference/generated/networkx.algorithms.shortest_paths.generic.all_shortest_paths.html
Laplacian matrix
https://networkx.github.io/documentation/latest/reference/generated/networkx.linalg.laplacianmatrix.laplacian_matrix.html#networkx.linalg.laplacianmatrix.laplacian_matrix
Each of the students pair up and work on demonstrating a networkx algorithm
Create a new ipython page and submit it
Can you write Python code for it?

#Enumeration of all cliques
list(nx.enumerate_all_cliques(G1))

list(nx.cliques_containing_node(G1,'A'))

list(nx.cliques_containing_node(G1,'D'))


I am working on the file "Lectures/Lecture 10 - Graph Algorithms/Python Graphs - networkx intro .ipynb".
The first task is:
http://networkx.github.io/documentation/latest/install.html
pip install networkx
to install networkx into your python modules
pip install networkx  --upgrade
to upgrade you previously installed version
You may have to restart your ipython engine to pick up the new module
Test your installation by running the new box locally
A full tutorial is available at:
http://networkx.github.io/documentation/latest/tutorial/index.html
There are other packages for graph manipulation for python:
python-igraph, (http://igraph.org/python/#pydoc1)
Graph-tool, (https://graph-tool.skewed.de)
I picked networkx because it took little effort to install.
Can you write Python code for it?

import networkx as nx
G1 = nx.Graph()

G1.add_nodes_from(['A','B','C','D','E','F','G'])
G1.nodes()

G1.add_edges_from([('A','B'),('A','C')])
G1.add_edges_from([('B','C'),('B','D')])
G1.add_edges_from([('D','E'),('D','F'),('D','G')])
G1.add_edges_from([('E','F')])
G1.add_edges_from([('F','G')])

G1.number_of_edges()

G1.edges()

G1.neighbors('D')

import matplotlib.pyplot as plt
#drawing the graph
%matplotlib inline

nx.draw(G1)

pos=nx.spring_layout(G1)
nx.draw(G1,pos,node_color='y', edge_color='r', node_size=600, width=3.0)
nx.draw_networkx_labels(G1,pos,color='W',font_size=20,font_family='sans-serif')
#https://networkx.github.io/documentation/latest/reference/generated/networkx.drawing.nx_pylab.draw_networkx.html
#Some parameters to play with


I am working on the file "Lectures/Lecture 11 - Machine Learning/Lecture 11 - Machine learning - Intro Tutorial.ipynb".
The first task is:
Conventions
Can you write Python code for it?

import numpy as np
from sklearn import random_projection

# default type is float64 unless specified
rng = np.random.RandomState(0)
X = rng.rand(10, 2000)
X = np.array(X, dtype='float32')

transformer = random_projection.GaussianRandomProjection()
X_new = transformer.fit_transform(X)

print ( "Data types for X = " + str(X.dtype) + " and X_new = "+ str(X_new.dtype))

from sklearn import datasets
from sklearn.svm import SVC
iris = datasets.load_iris()
clf = SVC()
clf.fit(iris.data, iris.target)

list(clf.predict(iris.data[:3]))

clf.fit(iris.data, iris.target_names[iris.target])

list(clf.predict(iris.data[:3]))

# refitting and updating parameters

import numpy as np
from sklearn.svm import SVC

rng = np.random.RandomState(0)
X = rng.rand(100, 10)
y = rng.binomial(1, 0.5, 100)
X_test = rng.rand(5, 10)

clf = SVC()
clf.set_params(kernel='linear').fit(X, y)
print ("Predict x_test: "+str(clf.predict(X_test)))
clf.set_params(kernel='rbf').fit(X, y)
print ("Predict x_test after refitting: "+str(clf.predict(X_test)))


I am working on the file "Lectures/Lecture02 -- Solving the zero-dimensional EBM.ipynb".
The first task is:
And this equilibrium temperature is just slightly warmer than 288 K.  Why?
A climate change scenario
Suppose that, due to global warming (changes in atmospheric composition and subsequent changes in cloudiness):

The longwave transmissitivity decreases to $\tau = 0.57$
The planetary albedo increases to $\alpha = 0.32$

What is the new equilibrium temperature?
For this very simple model, we can work out the answer exactly:
Can you write Python code for it?

Teq_new = equilibrium_temperature(0.32,Q,0.57)
#  an example of formatted print output, limiting to two or one decimal places
print('The new equilibrium temperature is {:.2f} K.'.format(Teq_new))
print('The equilibrium temperature increased by about {:.1f} K.'.format(Teq_new-Teq_observed))


I am working on the file "Lectures/Lecture02 -- Solving the zero-dimensional EBM.ipynb".
The first task is:
Note how the temperature adjusts smoothly toward the equilibrium temperature, that is, the temperature at which
ASR = OLR.
If the planetary energy budget is out of balance, the temperature must change so that the OLR gets closer to the ASR!
The adjustment is actually an exponential decay process: The rate of adjustment slows as the temperature approaches equilibrium.
The temperature gets very very close to equilibrium but never reaches it exactly.
Python fact 13: We can easily make simple graphs with the function plt.plot(x,y), where x and y are arrays of the same size. But we must import it first.
This is actually not native Python, but uses a special graphics library called matplotlib.
Just about all of our notebooks will start with this:
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

7. Analytical solution of the Energy Balance Model: e-folding time and feedback parameter

Equilibrium solutions
We've already seen that the equilibrium solution of the model is
$$ T_{eq} = \left( \frac{(1-\alpha) Q}{\tau \sigma} \right)^\frac{1}{4} $$
and tuned the model parameter based on this relationship.
We are going to linearize the equation for small perturbations away from this equilibrium.
Let $T_s = T_{eq} + T_s^\prime$ and restrict our solution to $T_s^\prime << T_{eq}$.
Note this this is not a big restriction! For example, a 10 degree warming or cooling is just $\pm$3.4% of the absolute equilibrium temperature.
Linearizing the governing equation
Now use a first-order Taylor series expansion to write
$$ \text{OLR} = \tau \sigma T_s^4 $$
$$OLR = \tau \sigma T_s^4 = \tau \sigma \left( T_{eq} + T_s^\prime \right)^4  \approx \tau \sigma \left( T_{eq}^4 + 4 T_{eq}^3 T_s^\prime \right) $$
and the budget for the perturbation temperature thus becomes
$$C \frac{d T_s^\prime}{d t} = -\lambda_0 T_s^\prime$$
where we define
$$\lambda_0 = 4 \tau \sigma T_{eq}^3 $$
Putting in our observational values, we get
Can you write Python code for it?

lambda_0 = 4 * sigma * tau * Teq_observed**3
#  This is an example of formatted text output in Python
print( 'lambda_0 = {:.2f} W m-2 K-1'.format(lambda_0)  )


I am working on the file "Lectures/Lecture02 -- Solving the zero-dimensional EBM.ipynb".
The first task is:
Planetary albedo
Some of the incoming radiation is not absorbed at all but simply reflected back to space. Let's call this quantity $F_{reflected}$
From observations we have:
Can you write Python code for it?

Freflected = 101.9  #  reflected shortwave flux in W/m2


I am working on the file "Lectures/Lecture02 -- Solving the zero-dimensional EBM.ipynb".
The first task is:
Solving the energy balance model
This is a first-order Ordinary Differential Equation (ODE) for $T_s$ as a function of time. It is also our very first climate model.
To solve it (i.e. see how $T_s$ evolves from some specified initial condition) we have two choices:

Solve it analytically
Solve it numerically

Option 1 (analytical) will usually not be possible because the equations will typically be too complex and non-linear. This is why computers are our best friends in the world of climate modeling.
HOWEVER it is often useful and instructive to simplify a model down to something that is analytically solvable when possible. Why? Two reasons:

Analysis will often yield a deeper understanding of the behavior of the system
Gives us a benchmark against which to test the results of our numerical solutions.

5. Representing time derivatives on a computer

Recall that the derivative is the instantaneous rate of change. It is defined as
$$ \frac{dT}{dt} = \lim_{\Delta t\rightarrow 0} \frac{\Delta T}{\Delta t}$$

On the computer there is no such thing as an instantaneous change.
We are always dealing with discrete quantities.
So we approximate the derivative with $\Delta T/ \Delta t$.
So long as we take the time interval $\Delta t$ "small enough", the approximation is valid and useful.
(The meaning of "small enough" varies widely in practice. Let's not talk about it now)

So we write our model as
$$ C  \frac{\Delta T}{\Delta t} \approx \text{ASR} - \text{OLR}$$
where $\Delta T$ is the change in temperature predicted by our model over a short time interval $\Delta t$.
We can now use this to make a prediction:
Given a current temperature $T_1$ at time $t_1$, what is the temperature $T_2$ at a future time $t_2$?
We can write
$$ \Delta T = T_2-T_1 $$
$$ \Delta t = t_2-t_1 $$
and so our model says
$$ C  \frac{T_2-T_1}{\Delta t} = \text{ASR} - \text{OLR} $$
Which we can rearrange to solve for the future temperature:
$$ T_2 = T_1 + \frac{\Delta t}{C} \left( \text{ASR} - \text{OLR}(T_1) \right) ...
Can you write Python code for it?

dt = 60. * 60. * 24. * 365.   # one year expressed in seconds

# Try a single timestep, assuming we have working functions for ASR and OLR
T1 = 288.
T2 = T1 + dt / C * ( ASR(alpha=0.32) - OLR(T1, tau=0.57) )
print(T2)


I am working on the file "Lectures/Lecture02 -- Solving the zero-dimensional EBM.ipynb".
The first task is:
That is, about 30% of the incoming radiation is reflected back to space.
Absorbed Shortwave Radiation
The Absorbed Shortwave Radiation or ASR is the part of the incoming sunlight that is not reflected back to space, i.e. that part that is absorbed somewhere within the Earth system.
Mathematically we write
$$ \text{ASR} = Q - F_{reflected} = (1-\alpha) Q $$
From the observations:
Can you write Python code for it?

ASRobserved = Q - Freflected
print(ASRobserved)


I am working on the file "Lectures/Lecture02 -- Solving the zero-dimensional EBM.ipynb".
The first task is:
The planetary albedo is the fraction of $Q$ that is reflected.
We will denote the planetary albedo by $\alpha$.
From the observations:
Can you write Python code for it?


I am working on the file "Lectures/Lecture02 -- Solving the zero-dimensional EBM.ipynb".
The first task is:
This is a rather fast timescale relative to other processes that can affect the planetary energy budget.
But notice that the climate feedback parameter $\lambda$ is smaller, the timescale gets longer.  We will come back to this later.

8. Summary and take-away messages

We looked at the flows of energy in and out of the Earth system.
These are determined by radiation at the top of the Earth's atmosphere.
Any imbalance between shortwave absorption (ASR) and longwave emission (OLR) drives a change in temperature
Using this idea, we built a climate model!
This Zero-Dimensional Energy Balance Model solves for the global, annual mean surface temperature $T_s$
Two key assumptions:
Energy content of the Earth system varies proportionally to $T_s$
The OLR increases as $\tau \sigma T_s^4$ (our simple greenhouse model)

Earth (or any planet) has a well-defined equilibrium temperature at which ASR = OLR, because of the temperature dependence of the outgoing longwave radiation.
The system will tend to relax toward its equilibrium temperature on an $e$-folding timescale that depends on
(1) radiative feedback processes, and
(2) effective heat capacity.

In our estimate, this e-folding time is relatively short. In the absence of other processes that can either increase the heat capacity or lower (in absolute value) the feedback parameter, the Earth would never be very far out of energy balance.
We will quantify this statement more as the term progresses.

[Back to ATM 623 notebook home](../index.ipynb)

Version information
Can you write Python code for it?

%load_ext version_information
%version_information numpy, matplotlib


I am working on the file "Lectures/Lecture02 -- Solving the zero-dimensional EBM.ipynb".
The first task is:
This is actually our first estimate of what is often called the Planck feedback. It is the tendency for a warm surface to cool by increased longwave radiation to space.
It may also be refered to as the "no-feedback" climate response parameter. As we will see, $\lambda_0$ quantifies the sensitivity of the climate system in the absence of any actual feedback processes.
Solve the linear ODE
Now define
$$ t^* = \frac{C}{\lambda_0}  $$
This is a positive constant with dimensions of time (seconds). With these definitions the temperature evolves according to
$$ \frac{d T_s^\prime}{d t} = - \frac{T_s^\prime}{t^*}$$
This is one of the simplest ODEs. Hopefully it looks familiar to most of you. It is the equation for an exponential decay process.
We can easily solve for the temperature evolution by integrating from an initial condition $T_s^\prime(0)$:
$$ \int_{T_s^\prime(0)}^{T_s^\prime(t)} \frac{d T_s^\prime}{T_s^\prime} = -\int_0^t  \frac{dt}{t^*}$$
$$\ln \bigg( \frac{T_s^\prime(t)}{T_s^\prime(0)} \bigg) = -\frac{t}{t^*}$$
$$T_s^\prime(t) = T_s^\prime(0) \exp \bigg(-\frac{t}{t^*} \bigg)$$
I hope that the mathematics is straightforward for everyone in this class. If not, go through it carefully and make sure you understand each step.
e-folding time for relaxation of global mean temperature
Our model says that surface temperature will relax toward its equilibrium value over a characteristic time scale $t^$. This is an e-folding time* – the time it takes for the perturbation to decay by a factor $1/e = 0.37$
What should this timescale be for the climate system?
To estimate $t^*$ we need a value for the effective heat capacity $C$.
Our "quick and dirty" estimate above used 100 meters of water to set this heat capacity.
What is the right choice for water depth $H$?
That turns out to be an interesting and subtle question. It depends very much on the timescale of the problem

days?
years?
decades?
millenia?

We will revisit this question later in the course. For now, let’s just continue assuming $H = 100$ m (a bit deeper...
Can you write Python code for it?

tstar = C / lambda_0   #  Calculated value of relaxation time constant
seconds_per_year = 60.*60.*24.*365.
print( 'The e-folding time is {:1.2e} seconds or about {:1.0f} years.'.format(tstar, tstar / seconds_per_year))


I am working on the file "Lectures/Lecture02 -- Solving the zero-dimensional EBM.ipynb".
The first task is:
Try it out with an arbitrary temperature:
Can you write Python code for it?


I am working on the file "Lectures/Lecture02 -- Solving the zero-dimensional EBM.ipynb".
The first task is:
What did we just do?

Created an array of zeros
set the initial temperature to 288 K
repeated our time step 20 times.
Stored the results of each time step into the array.

Python fact 11: the for statement executes a statement (or series of statements) a specified number of times (a loop!)
Python fact 12: Use square bracket [ ] to refer to elements of an array or list. Use round parentheses ( ) for function arguments.
Plotting the result
Now let's draw a picture of our result!
Can you write Python code for it?

# a special instruction for the Jupyter notebook
#   Display all plots inline in the notebook
%matplotlib inline
#  import the plotting package
import matplotlib.pyplot as plt

plt.plot( Years, Tsteps )
plt.xlabel('Years')
plt.ylabel('Global mean temperature (K)');


I am working on the file "Lectures/Lecture04 -- Intro to CLIMLAB.ipynb".
The first task is:

2. Using climlab to implement the zero-dimensional energy balance model

Recall that we have worked with a zero-dimensional Energy Balance Model
$$ C \frac{dT_s}{dt} = (1-\alpha) Q - \tau \sigma T_s^4 $$
Here we are going to implement this exact model using climlab.
Yes, we have already written code to implement this model, but we are going to repeat this effort here as a way of learning how to use climlab.
There are tools within climlab to implement much more complicated models, but the basic interface will be the same.
Can you write Python code for it?

# create a zero-dimensional domain with a single surface temperature
state = climlab.surface_state(num_lat=1,  # a single point
                              water_depth = 100.,  # 100 meters slab of water (sets the heat capacity)
                             )
state


I am working on the file "Lectures/Lecture04 -- Intro to CLIMLAB.ipynb".
The first task is:

5. Further climlab resources

We will be using climlab extensively throughout this course. Lots of examples of more advanced usage are found here in the course notes. Here are some links to other resources:

The documentation is hosted at https://climlab.readthedocs.io/en/latest/
Source code (for both software and docs) are at https://github.com/brian-rose/climlab
A video of a talk I gave about climlab at the 2018 AMS Python symposium (January 2018)
Slides from a talk and demonstration that I gave in Febrary 2018 (The Apple Keynote version contains some animations that will not show up in the pdf version)

Version information
Can you write Python code for it?

%load_ext version_information
%version_information numpy, matplotlib, climlab


I am working on the file "Lectures/Lecture04 -- Intro to CLIMLAB.ipynb".
The first task is:
Automatic timestepping
Often we want to integrate a model forward in time to equilibrium without needing to store information about the transient state.
climlab offers convenience methods to do this easily:
Can you write Python code for it?

ebm3 = climlab.process_like(ebm2)

ebm3.integrate_years(50)

#  What is the current temperature?
ebm3.Ts

#  How close are we to energy balance?
ebm3.ASR - ebm3.OLR

#  We should be able to accomplish the exact same thing with explicit timestepping
for n in range(608):
    ebm2.step_forward()

ebm2.Ts

ebm2.ASR - ebm2.OLR


I am working on the file "Lectures/Lecture04 -- Intro to CLIMLAB.ipynb".
The first task is:
Contents

Introducing climlab
Using climlab to implement the zero-dimensional energy balance model
Run the zero-dimensional EBM out to equilibrium
A climate change scenario in the EBM
Further climlab resources

1. Introducing climlab

climlab is a python package for process-oriented climate modeling.
It is based on a very general concept of a model as a collection of individual,
interacting processes. climlab defines a base class called Process, which
can contain an arbitrarily complex tree of sub-processes (each also some
sub-class of Process). Every climate process (radiative, dynamical,
physical, turbulent, convective, chemical, etc.) can be simulated as a stand-alone
process model given appropriate input, or as a sub-process of a more complex model.
New classes of model can easily be defined and run interactively by putting together an
appropriate collection of sub-processes.
climlab is an open-source community project. The latest code can always be found on github:
https://github.com/brian-rose/climlab
You can install climlab by doing
conda install -c conda-forge climlab
Can you write Python code for it?

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import climlab


I am working on the file "Lectures/Lecture04 -- Intro to CLIMLAB.ipynb".
The first task is:
Here is some internal information about the timestep of the model:
Can you write Python code for it?


I am working on the file "Lectures/Lecture04 -- Intro to CLIMLAB.ipynb".
The first task is:
It is also possible to see this state dictionary as an xarray.Dataset object:
Can you write Python code for it?

climlab.to_xarray(state)

#  create the longwave radiation process
olr = climlab.radiation.Boltzmann(name='OutgoingLongwave',
                                  state=state,
                                  tau = 0.612,
                                  eps = 1.,
                                  timestep = 60*60*24*30.)
#  Look at what we just created
print(olr)

#  create the shortwave radiation process
asr = climlab.radiation.SimpleAbsorbedShortwave(name='AbsorbedShortwave',
                                                state=state,
                                                insolation=341.3,
                                                albedo=0.299,
                                                timestep = 60*60*24*30.)
#  Look at what we just created
print(asr)

#  couple them together into a single model
ebm = olr + asr
#  Give the parent process name
ebm.name = 'EnergyBalanceModel'
#  Examine the model object
print(ebm)


I am working on the file "Lectures/Lecture04 -- Intro to CLIMLAB.ipynb".
The first task is:
Shoud the model warm up or cool down?
Well, we can find out:
Can you write Python code for it?


I am working on the file "Lectures/Lecture04 -- Intro to CLIMLAB.ipynb".
The first task is:
So why did the model get colder in the first timestep?
What do you think will happen next?

3. Run the zero-dimensional EBM out to equilibrium

Let's look at how the model adjusts toward its equilibrium temperature.
Exercise:

Using a for loop, take 500 steps forward with this model
Store the current temperature at each step in an array
Make a graph of the temperature as a function of time

4. A climate change scenario

Suppose we want to investigate the effects of a small decrease in the transmissitivity of the atmosphere tau.
Previously we used the zero-dimensional model to investigate a hypothetical climate change scenario in which:
- the transmissitivity of the atmosphere tau decreases to 0.57
- the planetary albedo increases to 0.32
How would we do that using climlab?
Recall that the model is comprised of two sub-components:
Can you write Python code for it?

for name, process in ebm.subprocess.items():
    print(name)
    print(process)


I am working on the file "Lectures/Lecture04 -- Intro to CLIMLAB.ipynb".
The first task is:
The model got colder!
To see why, let's look at some useful diagnostics computed by this model:
Can you write Python code for it?


I am working on the file "Lectures/Lecture04 -- Intro to CLIMLAB.ipynb".
The first task is:
This is another dictionary, now with two items. They should make sense to you.
Just like the state variables, we can access these diagnostics variables as attributes:
Can you write Python code for it?


I am working on the file "Lectures/Lecture04 -- Intro to CLIMLAB.ipynb".
The first task is:
This says the timestep is 2592000 seconds (30 days!), and the model has taken 0 steps forward so far.
To take a single step forward:
Can you write Python code for it?


I am working on the file "Lectures/Lecture06 -- Radiation.ipynb".
The first task is:
Going from cool to warm:

total emission increases
maximum emission occurs at shorter wavelengths.

The integral of these curves over all wavelengths gives us our familiar $\sigma T^4$
Mathematically it turns out that
$$ λ_{max} T = \text{constant} $$
(known as Wien’s displacement law).
By fitting the observed solar emission to a blackbody curve, we can deduce that the emission temperature of the sun is about 6000 K.
Knowing this, and knowing that the solar spectrum peaks at 0.6 micrometers, we can calculate the wavelength of maximum terrestrial radiation as
$$ λ_{max}^{Earth} = 0.6 ~ \mu m \frac{6000}{255} = 14 ~ \mu m $$
This is in the far-infrared part of the spectrum.

3. Terrestrial Radiation and absorption spectra

Terrestrial versus solar wavelengths
Now let's look at normalized blackbody curves for Sun and Earth:
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(14,7))
wavelength_um = np.linspace(0.1, 200, 10000)
wavelength_meters = wavelength_um / 1E6

for T in [6000, 255]:
    ax.semilogx(wavelength_um,
         (Planck_wavelength(wavelength_meters, T) / T**4 * wavelength_meters),
           label=str(T) + ' K')
ax.legend(fontsize=label_size)
ax.set_xlabel('Wavelength (um)', fontsize=label_size)
ax.set_ylabel('Normalized spectral radiance (W sr$^{-1}$ m$^{-2}$ K$^{-4}$)', fontsize=label_size)
ax.set_title("Normalized blackbody emission spectra $T^{-4} \lambda B_{\lambda}$ for the sun ($T_e = 6000$ K) and Earth ($T_e = 255$ K)",
            fontsize=label_size);


I am working on the file "Lectures/Lecture06 -- Radiation.ipynb".
The first task is:
Let's make a better plot.
Here we're going to use a package called metpy to automate plotting this temperature profile in a way that's more familiar to meteorologists: a so-called skew-T plot.
Can you write Python code for it?

from metpy.plots import SkewT

fig = plt.figure(figsize=(9, 9))
skew = SkewT(fig, rotation=30)
skew.plot(Tglobal.level, Tglobal, color='black', linestyle='-', linewidth=2, label='Observations')
skew.ax.set_ylim(1050, 10)
skew.ax.set_xlim(-75, 45)
# Add the relevant special lines
skew.plot_dry_adiabats(linewidth=0.5)
skew.plot_moist_adiabats(linewidth=0.5)
#skew.plot_mixing_lines()
skew.ax.legend()
skew.ax.set_title('Global, annual mean sounding from NCEP Reanalysis',
             fontsize = 16)