I am working on the file "Notebooks/Intro/One Dimensional Data Worksheet-Python.ipynb".
The first task is:
Exercise 5
You are given a Series of IP Addresses and the goal is to limit this data to private IP addresses.  Python has an ipaddress module which provides the capability to create, manipulate and operate on IPv4 and IPv6 addresses and networks.  Complete documentation is available here: https://docs.python.org/3/library/ipaddress.html.
Here are some examples of how you might use this module:
```python
import ipaddress
myIP = ipaddress.ip_address( '192.168.0.1' )
myNetwork = ipaddress.ip_network( '192.168.0.0/28' )
Check membership in network
if myIP in myNetwork:  #This works
    print "Yay!"
Loop through CIDR blocks
for ip in myNetwork:
    print( ip )
192.168.0.0
192.168.0.1
…
…
192.168.0.13
192.168.0.14
192.168.0.15
Testing to see if an IP is private
if myIP.is_private:
    print( "This IP is private" )
else:
    print( "Routable IP" )
```

First, write a function which takes an IP address and returns true if the IP is private, false if it is public.  HINT:  use the ipaddress module.
Next, use this to create a Series of true/false values in the same sequence as your original Series.
Finally, use this to filter out the original Series so that it contains only private IP addresses.
Can you write Python code for it?


I am working on the file "Notebooks/Intro/Two Dimensional Data Worksheet - Python Answers.ipynb".
The first task is:
Exercise 1
Using the twitterData DataFrame and the commands we have learned thus far and create a Series called tweetCounts which contains the user name and how many tweets each user posted.  Next, output the top 10 "tweeters".
Can you write Python code for it?

tweetCounts = twitterData['Username'].value_counts()
tweetCounts.head(10)


I am working on the file "Notebooks/Intro/Two Dimensional Data Worksheet - Python Answers.ipynb".
The first task is:
Exercise 3
In the Data folder, there is a spreadsheet called studentData.csv consisting of students and test scores.  Write a script which calculates each students' average test score and adds that as a column to the DataFrame.  The first person to raise their hand and tell me which student has the highest average test score, and what it is wins something.
Can you write Python code for it?

studentData = pd.read_csv('../../data/studentData.csv')

studentData['average'] = studentData[['Test1', 'Test2', 'Test3', 'Test4', 'Test5']].mean(axis=1)

studentData.sort_values('average', axis=0, ascending=False )


I am working on the file "Notebooks/Introduction/1 - Introduction to Jupyter notebooks.ipynb".
The first task is:
Exercise cells
Some cells are intended to be used for class exercises and have a little plus sign which indicates you can expand the cell to see a worked example or answer to a question. These cells are not executed unless they are expanded and run individually. Try this one both ways.
Can you write Python code for it?

print "You can hide and you can't run !"


I am working on the file "Notebooks/Introduction/1 - Introduction to Jupyter notebooks.ipynb".
The first task is:
When you run a code cell, it is just the same as typing all the code into the interpreter. If you run a cell twice it is the same as if you typed everything twice ... the notebook remembers ! You can run some of a notebook or all of it, and you can run cells out of order. This is great for experimenting and getting things right, but be careful, this can break things easily.
Can you write Python code for it?


I am working on the file "Notebooks/Introduction/2 - Introduction to ipython.ipynb".
The first task is:
Another way to do this is to use the cell magic functionality to direct the notebook to change the cell to something different (here everything in the cell is interpreted as a unix shell )
Can you write Python code for it?


I am working on the file "Notebooks/Introduction/2 - Introduction to ipython.ipynb".
The first task is:
It works on modules to list the available methods and variables. Take the math module, for example:
Can you write Python code for it?

import math

# math.is  # Try completion on this

help(math.isinf)

# try math.isinf() and hit shift-tab while the cursor is between the parentheses
# you should see the same help pop up.

# math.isinf()


I am working on the file "Notebooks/Introduction/3 - Python data types.ipynb".
The first task is:
But one of the problems with strings as data structures is that they are immutable. To change anything, we need to make copies of the data
Can you write Python code for it?


I am working on the file "Notebooks/Introduction/3 - Python data types.ipynb".
The first task is:
Dictionaries
These are very useful data collections where the information can be looked up by name instead of a numerical index. This will come in handy as a lightweight database and is commonly something we need to use when using modules to read in data.
Can you write Python code for it?

d = { "item1": ['a','b','c'], "item2": ['c','d','e']}

print d["item1"]
print d["item1"][1]

d1 = {"Small Number":1.0, "Tiny Number":0.00000001, "Big Number": 100000000.0}

print d1["Small Number"] + d1["Tiny Number"]
print d1["Small Number"] + d1["Big Number"]

print d1.keys()

for k in d1.keys():
    print "{:>15s}".format(k)," --> ", d1[k]


I am working on the file "Notebooks/Introduction/3 - Python data types.ipynb".
The first task is:
Lists are more flexible than tuples, they can be assigned to, have items removed etc
Can you write Python code for it?

l  = [1.0, 2.0, 3.0]
ll = ['a', 'b', 'c']
lll = [1.0, 'a', (1,2,3), ['f','g', 'h']]

print l
print ll
print l[2], ll[2]
print 2*l
print l+l
print lll
print lll[3], " -- sub item 3 --> ", lll[3][1]

print 2.0*l

l[2] = 2.99
print l

l.append(3.0)
print l

ll += 'b'
print ll
ll.remove('b')  # removes the first one !
print ll

l += [5.0]
print "1 - ", l
l.remove(5.0)
print "2 - ", l
l.remove(3.0)
print "3 - ", l
l.remove(4.0)
print "4 - ", l


I am working on the file "Notebooks/Introduction/3 - Python data types.ipynb".
The first task is:
Sets are special list-like collections of unique items. NOTE that the elements are not ordered (no such thing as s[1]
Can you write Python code for it?


I am working on the file "Notebooks/Introduction/3 - Python data types.ipynb".
The first task is:
Strings
Can you write Python code for it?

s = 'hello'
print s[1]
print s[-1]
print len(s)
print s + ' world'

ss = "\t\t hello \n \t\t world\n  \t\t !!!\n\n "
print ss
print ss.partition(' ')

print s[-1]," ", s[0:-1]


I am working on the file "Notebooks/Introduction/3 - Python data types.ipynb".
The first task is:
The math module needs to be imported before you can use it.
Can you write Python code for it?

import math
math.sqrt(f)

math.sqrt(c)

math.sqrt(-1)

import cmath

print cmath.sqrt(f)
print cmath.sqrt(c)
print cmath.sqrt(-1)


I am working on the file "Notebooks/Introduction/3 - Python data types.ipynb".
The first task is:
numbers as objects
Virtually everything in python is an object. This means that it is a thing that can have multiple copies made (all of which behave independently) and which knows how to do certain operations on itself.
For example, a floating point number knows certain things that it can do as well as simply "being" a number:
Can you write Python code for it?

help(f)

print f.is_integer()  # Strange eh ?
print f.conjugate()
print c.conjugate()
print f.__div__(2.0)  # This looks odd, but it is the way that f / 2.0 is implemented underneath


I am working on the file "Notebooks/Introduction/3 - Python data types.ipynb".
The first task is:
tuples and lists and sets
Tuples are bundles of data in a structured form but they are not vectors ... and they are immutable
Can you write Python code for it?


I am working on the file "Notebooks/Introduction/4 - Python classes.ipynb".
The first task is:
And changing the definition of the class does not affect the instances that have already been defined. The class definition is copied into the object.

A better way to deal with this is to define some methods that also live in the class and use those to set values of data at initialisation time, and to define how the class behaves.
Let's look at a better way to initialise the class:
Can you write Python code for it?

class colour2(object):
    """
    An rgb colour object with a description of the colour
    """

    def __init__(self, rgb = (0.0,0.0,0.0), description="Black"):

        self.rgb = rgb
        self.description = description
        self.complement = ( 1.0 - rgb[0], 1.0 - rgb[1], 1.0-rgb[2] )

        return

new_c = colour2(rgb=(1.0,0.0,0.0), description="Red")
new_d = colour2(rgb=(0.0,1.0,0.0), description="Green")

print new_c.rgb
print new_c.complement

help(colour2)

class colour2(object):
    """
    An rgb colour object with a description of the colour
    """

    def __init__(self, rgb = (0.0,0.0,0.0), description="Black"):

        self.rgb = rgb
        self.description = description
        self.complement = ( 1.0 - rgb[0], 1.0 - rgb[1], 1.0-rgb[2] )

        return

    def mix(self, other):
        if type(self) != type(other):
            print "Cannot mix apples and oranges"
            return
        else:
            return ( 0.5*(self.rgb[0]+other.rgb[0]),
                     0.5*(self.rgb[1]+other.rgb[1]),
                     0.5*(self.rgb[2]+other.rgb[2]) )

    def mixmax(self, other):
        if type(self) != type(other):
            print "Cannot mix apples and oranges"
            return
        else:
            return ( min(self.rgb[0]+other.rgb[0],1.0),
                     min(self.rgb[1]+other.rgb[1],1.0),
                     min(self.rgb[2]+other.rgb[2],1.0) )

new_c = colour2(rgb=(1.0,0.0,0.0), description="Red")
new_d = colour2(rgb=(0.0,1.0,0.0), description="Green")

print new_c.mix(new_d)


I am working on the file "Notebooks/Introduction/4 - Python classes.ipynb".
The first task is:
That's all we really need to know about classes, but if you want to know more about that last example, you'll need to know about this page at the very least: Object Model: numeric types from the python documentation. This is mostly advanced stuff, but if you find yourself being given a class description when you ask for help on something, this information will help you.
Can you write Python code for it?

## 1. Another example would be to make sure to set complement any time rgb gets changed
## 2. Another example would be to show how to subclass colour and match types so anything with an rgb works for mixing


I am working on the file "Notebooks/Introduction/4 - Python classes.ipynb".
The first task is:
colour is a data container that standardises how we refer to colours. By default it is "Black" but we can always change the values. The only thing is, we can also do this:
Can you write Python code for it?

d.complement = (1.0,1.0,0.0)
print d.complement

print c.complement


I am working on the file "Notebooks/JavaScript Notebook Extensions.ipynb".
The first task is:
and custom js is in
Can you write Python code for it?

import os.path
custom_js_path = os.path.join(profile_dir,'profile_default','static','custom','custom.js')

#  my custom js
with open(custom_js_path) as f:
    for l in f:
        print l,


I am working on the file "Notebooks/JavaScript Notebook Extensions.ipynb".
The first task is:
custom.js
To inject Javascript we provide an entry point: custom.js that allow teh user to execute and load other resources into the notebook.
Javascript code in custom.js will be executed when the notebook app start and can then be used to customise almost anything in the UI and in the behavior of the notebook.
custom.js can be found in IPython profile dir, and so you can have different UI modification on a per profile basis, as well as share your modfication with others.
Because we like you....
You have been provided with an already existing profile folder with this tutorial...
start the notebook from the root of the tutorial directory with :
bash
$ ipython notebook --ProfileDir.location=./profile_euroscipy
but back to theory
Can you write Python code for it?

profile_dir = ! ipython locate
profile_dir = profile_dir[0]
profile_dir


I am working on the file "Notebooks/JobLib/Joblib_for_Daniel.ipynb".
The first task is:
Direct copy of Joblib memmaping example
Can you write Python code for it?

def sum_row(input, output, i):
    """Compute the sum of a row in input and store it in output"""
    sum_ = input[i, :].sum()
    print("[Worker %d] Sum for row %d is %f" % (os.getpid(), i, sum_))
    output[i] = sum_

if __name__ == "__main__":
    rng = np.random.RandomState(42)
    folder = tempfile.mkdtemp()
    samples_name = os.path.join(folder, 'samples')
    sums_name = os.path.join(folder, 'sums')
    try:
        # Generate some data and an allocate an output buffer
        samples = rng.normal(size=(10, int(1e6)))

        # Pre-allocate a writeable shared memory map as a container for the
        # results of the parallel computation
        sums = np.memmap(sums_name, dtype=samples.dtype,
                         shape=samples.shape[0], mode='w+')
        print("samples shape", samples.shape)
        # Dump the input data to disk to free the memory
        dump(samples, samples_name)

        # Release the reference on the original in memory array and replace it
        # by a reference to the memmap array so that the garbage collector can
        # release the memory before forking. gc.collect() is internally called
        # in Parallel just before forking.
        samples = load(samples_name, mmap_mode='r')

        # Fork the worker processes to perform computation concurrently
        Parallel(n_jobs=4)(delayed(sum_row)(samples, sums, i)
                           for i in range(samples.shape[0]))

        # Compare the results from the output buffer with the ground truth
        print("Expected sums computed in the parent process:")
        expected_result = samples.sum(axis=1)
        print(expected_result)

        print("Actual sums computed by the worker processes:")
        print(sums)

        assert np.allclose(expected_result, sums)
    finally:
        try:
            shutil.rmtree(folder)
        except:
            print("Failed to delete: " + folder)


I am working on the file "Notebooks/Logistic-Regression.ipynb".
The first task is:

Can you write Python code for it?

# Scatter of V2 versus V3
positive = data[data['Genuine'].isin([1])]
negative = data[data['Genuine'].isin([0])]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive['V2'], positive['V3'], s=30, c='b', marker='+', label='Genuine')
ax.scatter(negative['V2'], negative['V3'], s=30, c='r', marker='s', label='Forged')
ax.legend(loc='lower right')
ax.set_xlabel('V2')
ax.set_ylabel('V3')
plt.title('Bank Note Validation Based on Feature Values V2 and V3');


# Scatter of V3 versus V4
positive = data[data['Genuine'].isin([1])]
negative = data[data['Genuine'].isin([0])]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive['V3'], positive['V4'], s=30, c='b', marker='+', label='Genuine')
ax.scatter(negative['V3'], negative['V4'], s=30, c='r', marker='s', label='Forged')
ax.legend(loc='lower right')
ax.set_xlabel('V3')
ax.set_ylabel('V4')
plt.title('Bank Note Validation Based on Feature Values V3 and V4');


# Scatter of Skewness versus Entropy
positive = data[data['Genuine'].isin([1])]
negative = data[data['Genuine'].isin([0])]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive['V2'], positive['V4'], s=30, c='b', marker='+', label='Genuine')
ax.scatter(negative['V2'], negative['V4'], s=30, c='r', marker='s', label='Forged')
ax.legend(loc='lower right')
ax.set_xlabel('V2')
ax.set_ylabel('V4')
plt.title('Bank Note Validation Based on Feature Values V2 and V4');


I am working on the file "Notebooks/Logistic-Regression.ipynb".
The first task is:
Step 1: Visualize the Data
Can you write Python code for it?

# Scatter of V1 versus V2
positive = data[data['Genuine'].isin([1])]
negative = data[data['Genuine'].isin([0])]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive['V1'], positive['V2'], s=30, c='b', marker='.', label='Genuine')
ax.scatter(negative['V1'], negative['V2'], s=30, c='r', marker='.', label='Forged')
ax.legend(loc='lower right')
ax.set_xlabel('V1')
ax.set_ylabel('V2')
plt.title('Bank Note Validation Based on Feature Values 1 and 2');


I am working on the file "Notebooks/Logistic-Regression.ipynb".
The first task is:
Step 2b: Identify the Output/Target
The output or target we'd like to predict is the feature called "Genuine". It takes the value 1 when the banknote is real and 0 when the banknote is counterfeit.
Can you write Python code for it?

# First few rows of the output/target
output = data[['Genuine']]
output.head()


I am working on the file "Notebooks/Logistic-Regression.ipynb".
The first task is:
Step 3: Define the Model
Step 3a: Define the Features
We have 2 features: V1 and V2.
Step 3b: Transform the Inputs Into an Output
Although the task we now face is different from the regression task, we're going to start just as we did before.
$$\hat{y} = w_{0} * x_{0}\ +\ w_{1} * x_{1} +\ w_{2} * x_{2}$$
It looks like the form of a linear regression and that's exactly what it is.
But now a twist...
When we transform the inputs V1 and V2 using the expression
$$\hat{y} = w_{0} * x_{0}\ +\ w_{1} * x_{1} +\ w_{2} * x_{2}$$
we're going to end up with a numeric value. It might be 4.2 or -12.56 or whatever depending on the values you plug in for $w_{0}$, $w_{1}$, and $w_{2}$.
But what we need is an output of 0 or 1.
Question: How to go from a numeric (continuous) value like -12.56 to a categorical value like 0 or 1?
The Sigmoid
The way to transform a numerical value into a categorical value is through something called a sigmoid. Here's what it looks like.
Can you write Python code for it?

# Define the sigmoid function or transformation
# NOTE: ALSO PUT INTO THE SharedFunctions notebook
def sigmoid(z):
    return 1 / (1 + np.exp(-z))

# Plot the sigmoid function
# Generate the values to be plotted
x_vals = np.linspace(-10,10,1000)
y_vals = [sigmoid(x) for x in x_vals]

# Plot the values
fig, ax = plt.subplots(figsize=(12,6))
ax.plot(x_vals, y_vals, 'blue')
ax.grid()
# Draw some constant lines to aid visualization
plt.axvline(x=0, color='black')
plt.axhline(y=0.5, color='black')
plt.yticks(np.arange(0,1.1,0.1))
plt.xticks(np.arange(-10,11,1))
plt.xlabel(r'$\hat{y}$', fontsize=15)
plt.ylabel(r'$sigmoid(\hat{y})$', fontsize=15)
plt.title('The Sigmoid Transformation', fontsize=15)
ax.plot;


I am working on the file "Notebooks/Logistic-Regression.ipynb".
The first task is:
Step 6: Use the Model and Optimal Parameter Values to Make Predictions
Can you write Python code for it?

# Genuine or fake for the entire data set
y_pred = lr.predict(X_train)
print(y_pred)

# How do the predictions compare with the actual labels on the data set?
y_train == y_pred

# The probabilities of [Genuine = 0, Genuine = 1]
y_pred_probs = lr.predict_proba(X_train)
print(y_pred_probs)

# Where did the model misclassify banknotes?
errors = data[data['Genuine'] != y_pred]
#errors

# Following Sonya Sawtelle
# (https://sdsawtelle.github.io/blog/output/week3-andrew-ng-machine-learning-with-python.html)
# This is the classifier boundary line when z=0
x1 = np.linspace(-6,6,100)  # Array of exam1 value
x2 = (-w0/w2) - (w1/w2)*x1  # Corresponding V2 values along the line z=0

# Following Sonya Sawtelle
# (https://sdsawtelle.github.io/blog/output/week3-andrew-ng-machine-learning-with-python.html)

# Scatter of V1 versus V2
positive = data[data['Genuine'].isin([1])]
negative = data[data['Genuine'].isin([0])]

fig, ax = plt.subplots(figsize=(15,10))
#colors = ["r", "b"]
#la = ["Forged", "Genuine"]
#markers = [colors[gen] for gen in data['Genuine']] # this is a cool way to color the categories!
#labels = [la[gen] for gen in data['Genuine']]
#ax.scatter(data['V1'], data['V2'], color=markers, s=10, label=labels)

ax.scatter(positive['V1'], positive['V2'], s=30, c='b', marker='.', label='Genuine')
ax.scatter(negative['V1'], negative['V2'], s=30, c='r', marker='.', label='Forged')

ax.set_xlabel('V1')
ax.set_ylabel('V2')

# Now plot black circles around data points that were incorrectly predicted
ax.scatter(errors["V1"], errors["V2"], facecolors="none", edgecolors="m", s=80, label="Wrongly Classified")

# Finally plot the line which represents the decision boundary
ax.plot(x1, x2, color="green", linestyle="--", marker=None, label="boundary")

ax.legend(loc='upper right')

plt.title('Bank Note Validation Based on Feature Values 1 and 2');


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Define a function to combine two events
Can you write Python code for it?

def combine_prob(p1,p2):
    """Returns the probability that event 1 or 2 happens"""
    return 1. - (1-p1)*(1-p2)


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Hazard Maps
If we fix k then exceed_prob[:,:,k] is a two dimensional array giving the probability of exceedance at all points on the grid for a fixed exceedance level zeta[k].  We can plot this to obtain a hazard map showing probabilities for a given exceedance value.
Define contours and colors and a function to plot probability maps
prob_clines will be the probability levels to use in contour maps
prob_colors will define the color map to use.   This is a list of tuples (R,G,B) of red,green,blue values, chosen to go from light blue to red.
Note: The function plot_pmap defined in the cell below uses the exceedance probabilities exceed_prob computed above.  If you recompute these (e.g. by changing the set of events to include, or the probabilities of individual events), you must re-execute this cell to redefine plot_pmap before re-making the plots in later cells!
Can you write Python code for it?

prob_clines = [1e-5, 1e-4, 1e-3, 2e-3, 1e-2, 2e-2, 1e-1]
nlines = len(prob_clines)
n1 = int(floor((nlines-1)/2.))
n2 = nlines - 1 - n1

Green = hstack([linspace(1,1,n1),linspace(1,0,n2)])
Red = hstack([linspace(0,0.8,n1), ones(n2)])
Blue = hstack([linspace(1,0.2,n1), zeros(n2)])
prob_colors = list(zip(Red,Green,Blue))

color_offscale = (.5,0,0)  # color to use if above maximum
prob_colors.append(color_offscale)

# Choose the background for plots by uncommenting one line:
background = plot_CCmap
#background = plot_topo

def plot_pmap(k):
    fig = background()
    contourf(X,Y,exceed_prob[:,:,k], prob_clines, colors=prob_colors,alpha = 0.6, extend='max')
    title("Annual probability of flooding above %g meters" % zeta[k])
    colorbar()
    show()


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Interactive viewer for a range of probabilities:
Can you write Python code for it?


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Interactive viewer of all hazard maps:
Can you write Python code for it?


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Interactive viewer to move the point around:
Can you write Python code for it?


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Inundation maps for given probability:
A more commonly used map is obtained by fixing a probability (e.g. $p = 0.01$ for a "100-year" flood map) and plotting the maximum depth expected with this annual probability.
This requires determining, for each grid point (i,j), the largest value of k for which exceed_prob[k]  $\geq p$. Then the value zeta[k] is the largest exceedance value for which the probability is at least $p$.
Recall that zeta is defined to be maximum depth of inundation on shore, or maximum height above MHW offshore.
Note: The functions compute_zeta and plot_inundation_map defined in the cell below uses the exceedance probabilities exceed_prob computed above.  If you recompute these (e.g. by changing the set of events to include, or the probabilities of individual events), you must re-execute this cell to redefine the functions before re-making the plots in later cells!
Can you write Python code for it?

def compute_zeta(p):

    # create boolean array K with K[i,j,k] == True only where exceed_prob[i,j,k] > p:
    K = exceed_prob > p

    K[:,:,0] = True
    zeta_p = zeros(X.shape)
    for i in range(nx):
        for j in range(ny):
            zeta_p[i,j] = zeta[K[i,j,:]][-1]
    return zeta_p

# Set contour lines and colors for plotting zeta = inundation depth

zeta_clines = [1e-3] + list(linspace(0.5,4.5,9))
nlines = len(zeta_clines)
n1 = int(floor((nlines-1)/2.))
n2 = nlines - 1 - n1
Green = hstack([linspace(1,1,n1),linspace(1,0,n2)])
Red = hstack([linspace(0,0.8,n1), ones(n2)])
Blue = hstack([linspace(1,0.2,n1), zeros(n2)])
zeta_colors = list(zip(Red,Green,Blue))
color_offscale = (.5,0,0)  # color to use if above maximum
zeta_colors.append(color_offscale)

# Choose the background for plots by uncommenting one line:
background = plot_CCmap
#background = plot_topo

def plot_inundation_map(p):
    zeta_p = compute_zeta(p)
    fig =background()
    contourf(X,Y,zeta_p,zeta_clines, colors=zeta_colors, alpha = 0.6, extend='max')
    title("Depth of flooding for annual probability %g\nReturn time %5.0f years" % (p, (1./p)))
    colorbar()
    show();


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Plot a sample map:
Can you write Python code for it?


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Plot hazard curves
The array exceed_prob[i,j,:] (i.e. fixing i,j and letting the last index k vary from 0 to nzeta - 1) gives the probability of exceedance at the (i,j) grid point as we vary the exceedance value zeta[k].  Plotting this gives exactly the hazard curve at the (i,j) point.
The function plot_hcurve defined below plots this for a given (longitude, latitude) by first figuring out the index (i,j) for the nearest point on the grid covering Crescent City.
Can you write Python code for it?

dx = X[1,0] - X[0,0]
dy = Y[0,1] - Y[0,0]
nx, ny = X.shape
xmin = X.min(); xmax = X.max()
ymin = Y.min(); ymax = Y.max()

def plot_hcurve(longitude, latitude):
    i = int(round((longitude - X[0,0]) / dx))
    j = int(round((latitude - Y[0,0]) / dy))
    if (i<0) or (i>=nx) or (j<0) or (j>=ny):
        print("out of domain")
        return
    fig = figure(figsize=(12,5))
    subplot(1,2,1)
    p = maximum(exceed_prob[i,j,:], 1e-10)
    semilogy(zeta, p, 'b')
    ylim(1e-5,1)
    xlabel('zeta in meters')
    ylabel('annual probability')
    title('Hazard Curve')

    # Also plot the CC image with a red dot showing the location:
    ax = subplot(1,2,2)
    imshow(CCmap,extent=extent)
    CClatitude = 41.75  # to rescale longitude
    ax.set_aspect(1. / cos(pi*CClatitude/180.))
    ax.ticklabel_format(format='plain',useOffset=False)
    plot([longitude], [latitude], 'ro')
    xlim(xmin,xmax)
    ylim(ymin,ymax)
    title('Location')
    show()


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Plot the hazard curve for one location:
Can you write Python code for it?


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Read in image of Crescent City as background for plots
Can you write Python code for it?

CCmap = imread('%s/MapsTopo/CCimage.png' % data_dir)
extent = (235.79781, 235.82087, 41.739671,41.762726)   #small region

def plot_CCmap():
    fig = figure(figsize=(6,6))
    ax = axes()
    imshow(CCmap,extent=extent)
    CClatitude = 41.75  # to rescale longitude
    ax.set_aspect(1. / cos(pi*CClatitude/180.))
    ax.ticklabel_format(format='plain',useOffset=False)
    axis(extent)
    return fig


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Read in the topography data and define a function to make a contour plot:
Can you write Python code for it?

# Read in topography data:

fixed_grid_file = os.path.join(data_dir, 'MapsTopo', 'fixedgrid_xyB_small.npy')
d=load(fixed_grid_file)
x=d[:,0]
y=d[:,1]
B=d[:,2]
topo = reshape(B, (250,250), order='F')
X = reshape(x, (250,250), order='F')
Y = reshape(y, (250,250), order='F')

def plot_topo():
    fig = figure(figsize=(6,6))
    ax = axes()
    topo_clines = arange(0,20,2)
    contour(X,Y,topo,topo_clines,colors='k')
    CClatitude = 41.75  # to rescale longitude
    ax.set_aspect(1. / cos(pi*CClatitude/180.))
    ax.ticklabel_format(format='plain',useOffset=False)
    return fig


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Set the exceedance values
This should be a list or array of values $\zeta$ (zeta) representing depth of flooding on shore, or elevation above sea level offshore (in meters).  The hazard curves will be computed by determining the annual probability that the maximum $\zeta$ observed at each spatial point is above $\zeta_k$, for each value $\zeta_k$ in this list.
Can you write Python code for it?

# these levels were used in original study:
#zeta = hstack((arange(0,2.,.1), arange(2.0,12.5,.5)))

# you get nicer looking curves by using a denser set of exceedance values:
zeta = linspace(0,12,121)
nzeta = len(zeta)
print('%i exceedance values, \nzeta =  %s' % (nzeta,zeta))


I am working on the file "Notebooks/Make_Hazard_Curves_and_Maps.ipynb".
The first task is:
Specify the set of events to include in computing hazard curves:
Can you write Python code for it?

events = all_events

# Instead, to use a subset of the events, specify a list such as:
#events = ['AASZa', 'AASZb', 'AASZc']


I am working on the file "Notebooks/Make_Hazard_Curves_interactive.ipynb".
The first task is:
Define a function to combine two events
Can you write Python code for it?


I am working on the file "Notebooks/Make_Hazard_Curves_interactive.ipynb".
The first task is:
Interactive viewer to move the point around:
The interact_manual command used below is used in the same way as interact but does not create or refresh the plot until the Run plot_hcurve button is pressed.  This may be desirable if refreshing happens too slowly as the slider is dragged with interact.  (Try either in the cell below to compare the behavior.)
Can you write Python code for it?


I am working on the file "Notebooks/Make_Hazard_Curves_interactive.ipynb".
The first task is:
Plot the hazard curve for one location:
Can you write Python code for it?


I am working on the file "Notebooks/Make_Hazard_Curves_interactive.ipynb".
The first task is:
Read in image of Crescent City as background for plots
Can you write Python code for it?


I am working on the file "Notebooks/Make_Hazard_Curves_interactive.ipynb".
The first task is:
Set the exceedance values
This should be a list or array of values $\zeta$ (zeta) representing depth of flooding on shore, or elevation above sea level offshore (in meters).  The hazard curves will be computed by determining the annual probability that the maximum $\zeta$ observed at each spatial point is above $\zeta_k$, for each value $\zeta_k$ in this list.
Can you write Python code for it?


I am working on the file "Notebooks/Make_Textfile_Demo.ipynb".
The first task is:
Read in zeta for a single event:
Can you write Python code for it?

event = 'AASZb'
event_dir = os.path.join(events_dir, event)
zeta_file = os.path.join(event_dir, 'h_eta_small.npy')
zeta = load(zeta_file)


I am working on the file "Notebooks/Make_Textfile_Demo.ipynb".
The first task is:
Write out to text file
Either of the approaches above could be used, e.g.
Can you write Python code for it?

fname = '%s.txt' % event
data_array = vstack((x, y, zeta)).T
savetxt(fname, data_array, fmt='%24.10f')
print("Created file ", os.path.abspath(fname))


I am working on the file "Notebooks/Make_Textfile_Demo.ipynb".
The first task is:
Write out to text file
The file will have 3 columns containing x, y, B at each of the $250^2 = 62500$ grid points.
Can you write Python code for it?

fname = 'xyB.txt'
f = open(fname,'w')
for npts in range(len(B)):
    f.write('%24.10f %24.10f %24.10f\n' % (x[npts],y[npts],B[npts]))
f.close()
print("Created file ", os.path.abspath(fname))


I am working on the file "Notebooks/Make_Transects.ipynb".
The first task is:
Compute the combined probability of exceeding each exceedance value:
exceed_prob is computed as an array of shape
Can you write Python code for it?

nx, ny = X.shape  # note that X is a 2d array of longitude values at each point
exceed_prob = zeros((nx,ny,nzeta))  # initialize to zero

# loop over all events and update exceed_prob at each grid point by combining
# current value with the probability Pk of this event:

for event in events:
    event_dir = os.path.join(events_dir, event)
    hmax_file = os.path.join(event_dir, 'h_eta_small.npy')
    hmax = load(hmax_file)
    Hmax = hmax.reshape((nx,ny),order='F')
    for k in range(nzeta):
        Pk = exceed_prob[:,:,k]  # probabilities at all points for one exceedance value zeta_k
        exceed_prob[:,:,k] = where(Hmax > zeta[k], combine_prob(event_prob[event],Pk), Pk)

print("Computed exceedance probabilities.  \nMaximum over all grid points is %g" % exceed_prob.max())


I am working on the file "Notebooks/Make_Transects.ipynb".
The first task is:
Define a function to combine two events
Can you write Python code for it?


I am working on the file "Notebooks/Make_Transects.ipynb".
The first task is:
Define plot_transect function
We first define a function topo_func that allows us to evaluate the topography at any point (x,y).
The function plot_transect is then defined to take two points (x1,y1) and (x2,y2) and interpolate the topography onto a set of 1000 equally spaced points along the transect (straight line connecting the points).  The function also takes another argument, a z_func function that can also be evaluated at any point and is assumed to return a value of $\zeta$ at this point.  This might be defined to be the zeta for a particular single event, or it might be the zeta coming from a probabilistic analysis at some particular probability p.
Can you write Python code for it?

from scipy.interpolate import RegularGridInterpolator
topo_func = RegularGridInterpolator((X[:,0], Y[0,:]), topo)

def plot_transect(x1,y1, x2,y2, z_func):

    # points on transect:
    xi = linspace(x1, x2, 1000)
    yi = linspace(y1, y2, 1000)

    # plot the transect on top of topography:
    fig = plot_topo()
    plot([x1,x2], [y1,y2], 'r-', linewidth=3)
    title('Transect')

    # evaulate topo and zeta on transect:
    Bi = topo_func(array(list(zip(xi,yi))))
    zi = z_func(array(list(zip(xi,yi))))

    # define eta as zeta offshore or zeta + topo onshore:
    eta = where(Bi<0, zi, zi+Bi)

    # plot cross sections on topo
    # plot vs. longitude or latitude depending on orientation:

    if (abs(x2-x1) > 0.5*abs(y2-y1)):
        ti = xi
        xtext = 'longitude'
    else:
        ti = yi
        xtext = 'latitude'

    Bi0 = where(Bi<0, 0., Bi)

    figure(figsize=(15,4))
    fill_between(ti, Bi0, eta, color='b')
    plot(ti, Bi, 'g')

    xlabel(xtext)
    ylabel('meters')
    plt.title('Elevation vs. %s' % xtext)

    # choose limits of vertical axis to give nice plots:
    eta_wet_max = where(zi>0, zi+Bi, 0).max()
    ylim(-2, max(6,eta_wet_max))

    ticklabel_format(format='plain',useOffset=False)
    plt.xticks(rotation=20)


I am working on the file "Notebooks/Make_Transects.ipynb".
The first task is:
Next set the transect end points and plot using the plot_transect function defined above.
Can you write Python code for it?

zeta_event_func

x1 = 235.805;  y1 = 41.745
x2 = 235.800;  y2 = 41.762

print("Event: %s" % event)
plot_transect(x1,y1, x2,y2, zeta_event_func)


I am working on the file "Notebooks/Make_Transects.ipynb".
The first task is:
Read in image of Crescent City as background for plots
Can you write Python code for it?

CCmap = imread('%s/MapsTopo/CCimage.png' % data_dir)
extent = (235.79781, 235.82087, 41.739671,41.762726)   #small region

def plot_CCmap():
    fig = figure(figsize=(6,6))
    ax = axes()
    imshow(CCmap,extent=extent)
    CClatitude = 41.75  # to rescale longitude
    ax.set_aspect(1. / cos(pi*CClatitude/180.))
    ax.ticklabel_format(format='plain',useOffset=False)
    axis(extent)


I am working on the file "Notebooks/Make_Transects.ipynb".
The first task is:
Set the desired annual probability for each event
Note that we are only using 14 events for this workshop.  The probabilities have been adjusted accordingly.
event_prob is a Python dictionary. It is inialized to an empty dictionary and then we set event_prob[key] = value where the keys are the names of the hypothetical events and the associated value is the annual probability.
Can you write Python code for it?

all_events = ['AASZa', 'AASZb', 'AASZc', 'AASZd', 'CSZa', 'CSZb', 'CSZc', 'CSZd', 'CSZe', \
              'CSZf', 'KmSZa', 'KrSZa', 'SChSZa', 'TOHa']

event_prob = {}
event_prob['AASZa'] = 1./394.
event_prob['AASZb'] = 1./750.
event_prob['AASZc'] = 1./563.
event_prob['AASZd'] = 1./324.
event_prob['CSZa'] = 1./250. * .0125
event_prob['CSZb'] = 1./250. * .0125
event_prob['CSZc'] = 1./250. * .0750
event_prob['CSZd'] = 1./250. * .5000
event_prob['CSZe'] = 1./250. * .1750
event_prob['CSZf'] = 1./250. * .2250
event_prob['KmSZa'] = 1./50.
event_prob['KrSZa'] = 1./167.
event_prob['SChSZa'] = 1./300.
event_prob['TOHa'] = 1./103.

print("Annual probability of each event is set to:")
print(event_prob)


I am working on the file "Notebooks/Make_Transects.ipynb".
The first task is:
Transects
Now we can plot transects for a given probability $p$, e.g. $p=0.002$ for the "500-year flood".  Experiment with changing p or the transect...
Can you write Python code for it?

p = 0.002
zeta_p = compute_zeta(p)
zeta_p_func = RegularGridInterpolator((X[:,0], Y[0,:]), zeta_p)

# Set endpoints of transect:
x1 = 235.805;  y1 = 41.745
x2 = 235.800;  y2 = 41.762

print("Transect of zeta for given annual probability p = %g" % p)
plot_transect(x1,y1, x2,y2, zeta_p_func)


I am working on the file "Notebooks/Make_Transects_interactive.ipynb".
The first task is:
Define class to allow interactive plotting
The class DrawLine allows the reader to select a transect and will then interpolate the solution from the specified event to this transect.
We first define a function topo_func that allows us to evaluate the topography at any point (x,y).
The function plot_transect is then defined to take two points (x1,y1) and (x2,y2) and interpolate the topography onto a set of 1000 equally spaced points along the transect (straight line connecting the points).  The function also takes another argument, a z_func function that can also be evaluated at any point and is assumed to return a value of $\zeta$ at this point.  This might be defined to be the zeta for a particular single event, or it might be the zeta coming from a probabilistic analysis at some particular probability p.
Can you write Python code for it?

from scipy.interpolate import RegularGridInterpolator
topo_func = RegularGridInterpolator((X[:,0], Y[0,:]), topo)

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec

class DrawLine:
    def __init__(self, line,zeta,topo):
        self.line = line
        self.figure = line[0].figure
        self.xprev = 0
        self.yprev = 0
        self.xnew = 0
        self.ynew = 0
        self.press = None
        self.zeta = zeta
        self.topo = topo

    def connect(self):
        'connect to all the events we need'
        self.cidpress = self.figure.canvas.mpl_connect(
            'button_press_event', self.on_press)
        self.cidrelease = self.figure.canvas.mpl_connect(
            'button_release_event', self.on_release)

    def on_press(self, event):
        self.xprev = event.xdata
        self.yprev = event.ydata
        self.press = 1
        ax1.plot(self.xprev,self.yprev,'ro')
        self.figure.canvas.draw()
        #self.line[0].set_data([[event.xdata],[event.ydata]])
        #self.line[0].set_xdata = np.array([event.xdata, self.line[0].get_xdata()[1]])
        #self.line[0].set_ydata = np.array([event.ydata, self.line[0].get_ydata()[1]])

    def on_release(self, event):
        self.press = None
        self.xnew = event.xdata
        self.ynew = event.ydata
        #print('+++ xnew, ynew: ', xnew,ynew)
        #self.line[0].set_xdata = np.array([self.line[0].get_xdata()[0]], event.xdata)
        #self.line[0].set_ydata = np.array([self.line[0].get_ydata()[0]], event.ydata)
        ax1.cla()

        topo_clines = arange(0,20,2)
        ax1.contour(X,Y,topo,topo_clines,colors='k')
        CClatitude = 41.75  # to rescale longitude
        ax1.set_aspect(1. / cos(pi*CClatitude/180.))
        ax1.ticklabel_format(format='plain',useOffset=False)

        ax1.plot(self.xprev,self.yprev,'ro')
        ax1.plot([self.xprev,self.xnew],[self.yprev,self.ynew],'-r',lw=3)
        #ax1.set_xlim(-5,5)
        #ax1.set_ylim(-5,5)

       ...


I am working on the file "Notebooks/Mapping/2 - Images and GeoTIFFs.ipynb".
The first task is:
OK, that didn't look like the sample image that I claimed it should and the reason is that we didn't tell the plotting routines what the original projection for the data was. Here is the fix: tell the imshow command the transformation of the original data - (this can take a little while to process).
Can you write Python code for it?

fig = plt.figure(figsize=(12, 12), facecolor="none")
ax = plt.axes(projection=ccrs.Orthographic())
plt.imshow(globalmarble_img, zorder=0, transform=ccrs.PlateCarree())
ax.coastlines(color="Yellow", zorder=1)

plt.show()


I am working on the file "Notebooks/Mapping/2 - Images and GeoTIFFs.ipynb".
The first task is:
Projections
What if we want to use a different projection for the image. For example, we might want to use this as a background to plot some other information. How does this work ?
Let's try what we used before to plot with a different projection. Specifically, let's try an orthographic projection which should wrap the image around a sphere and show us the disk from a particular orientation. Here is one I made earlier ... the output should look like this:
<img src="../../Data/Reference/OrthographicProjectionBlueMarble.png" width=30%>
Can you write Python code for it?

fig = plt.figure(figsize=(12, 12), facecolor="none")
ax = plt.axes(projection=ccrs.Orthographic())
plt.imshow(globalmarble_img, zorder=0)
ax.coastlines(color="Yellow", zorder=1)
plt.show()


I am working on the file "Notebooks/Mapping/2 - Images and GeoTIFFs.ipynb".
The first task is:
This looks really nice but it is just the original image plotted in its original shape.
The gdal object can tell you the projection which applies to the original data and various other attributes. This might not seems particularly useful at this point, but it is helpful to realise that there is a lot of information being passed around behind the scenes when you use these tools.
Can you write Python code for it?

print "1 - ", globalmarble.GetProjection(), "\n"
print "2 - ", globalmarble.GetDescription(), "\n"
print "3 - ", globalmarble.GetMetadata(), "\n"
print "4 - ", globalmarble.GetGeoTransform()


I am working on the file "Notebooks/Mapping/2 - Images and GeoTIFFs.ipynb".
The first task is:
You can try other projections here, though I have found quite a few do not behave in quite the way you expect !
Feel free to play with these data which are global magnetic intensity, the global etopo database of topography and bathymetry in color format and a black/white (height only) version of the same thing, all of them have the base projection of PlateCarree. Note that we can define a variable which points to this function and pass it into the transform argument of imshow
Can you write Python code for it?

base_projection   = ccrs.PlateCarree()

globalmag         = gdal.Open("../../Data/Resources/EMAG2_image_V2.tif")
globalmag_img     = globalmag.ReadAsArray().transpose(1,2,0).astype(numpy.float16)
globalmag_img_s   = globalmag_img[::2,::2,::]
del(globalmag)

globaletopo       = gdal.Open("../../Data/Resources/color_etopo1_ice_low.tif")
globaletopo_img   = globaletopo.ReadAsArray().transpose(1,2,0).astype(numpy.float16)
del(globaletopo)

print globalmag_img.shape
print globalmag_img_s.shape
print globalmarble_img.shape

# Global pretty map ... can you make one centred on Australia and save it to an image ?

#projection = ccrs.Orthographic(central_longitude=80.0, central_latitude=30.0, globe=None)
this_projection = ccrs.PlateCarree()

global_extent     = [-180.0, 180.0, -90.0, 90.0]

fig = plt.figure(figsize=(12, 12), facecolor="none")
ax = plt.axes(projection=this_projection)
ax.imshow(globalmag_img_s, origin='upper', transform=base_projection, extent=global_extent)
ax.imshow(globalmarble_img, origin='upper', transform=base_projection, extent=global_extent, alpha=0.5)
ax.coastlines(color="yellow")

plt.show()


I am working on the file "Notebooks/Metropolis MCMC for coin fairness inference.ipynb".
The first task is:
Let's compare the posterior inference to the prior: have we learned anything about our coin?
Can you write Python code for it?

plt.rcParams["figure.figsize"] = fig_size

a = 2
b = 2
x2 = np.linspace(beta.ppf(beginning, a, b), beta.ppf(end, a, b), 100)
plt.plot(x2, beta.pdf(x2, a, b),'b-', lw=5, alpha=0.6, label='PRIOR: Beta pdf, a=2, b=2')
plt.hist(params, label='POSTERIOR', density=True, bins=200, color="lightgreen")
plt.legend(loc='best', frameon=False)
plt.xlabel("Parameter value")
plt.ylabel("Density/Frequency")
sns.kdeplot(np.array(params), bw=0.03, lw=5, color="green", shade=True)


I am working on the file "Notebooks/Metropolis MCMC for coin fairness inference.ipynb".
The first task is:
Our median estimate for the parameter is:
Can you write Python code for it?


I am working on the file "Notebooks/Metropolis MCMC for coin fairness inference.ipynb".
The first task is:
We build a probabilistic model of coin tossing.
All coin tosses are supposed to be independent tosses of the same coin, which always have the same probability of returning a head.
We want to perform Bayesian inference, therefore we need a prior.
For inference, we will be using Metropolis MCMC.
Defining a prior
We need to put some prior probability on the fairness of the coin. For this, a beta distribution seems appropriate, as it is a continuous distribution between 0 and 1.
Let's display a beta distribution with various parameter values.
Can you write Python code for it?

fig_size=[]
fig_size.append(15)
fig_size.append(9)
plt.rcParams["figure.figsize"] = fig_size

beginning = 0.0001
end = 0.9999

a = 1
b = 1
x = np.linspace(beta.ppf(beginning, a, b), beta.ppf(end, a, b), 100)
plt.plot(x, beta.pdf(x, a, b),'r-', lw=5, alpha=0.6, label='Beta pdf, a=1, b=1')
a = 2
b = 2
x2 = np.linspace(beta.ppf(beginning, a, b), beta.ppf(end, a, b), 100)
plt.plot(x2, beta.pdf(x2, a, b),'b-', lw=5, alpha=0.6, label='Beta pdf, a=2, b=2')
a = 0.8
b = 0.8
x3 = np.linspace(beta.ppf(beginning, a, b), beta.ppf(end, a, b), 100)
plt.plot(x3, beta.pdf(x3, a, b),'g-', lw=5, alpha=0.6, label='Beta pdf, a=0.8, b=0.8')
a = 2
b = 0.8
x4 = np.linspace(beta.ppf(beginning, a, b), beta.ppf(end, a, b), 100)
plt.plot(x4, beta.pdf(x4, a, b),'p-', lw=5, alpha=0.6, label='Beta pdf, a=2, b=0.8')
plt.legend(loc='best', frameon=False)
plt.xlabel("Parameter value")
plt.ylabel("Density/Frequency")


I am working on the file "Notebooks/Metropolis MCMC for coin fairness inference.ipynb".
The first task is:
We choose to use a=2, b=2. This is a weakly informative prior that the coin should be fair, given our past experience with coins.
Inference of the fairness of the coin using MCMC
We build a MCMC chain to estimate the probability of heads for this coin.
First we define the model, with the prior, the likelihood and the posterior probability, then we implement a Metropolis MCMC inference mechanism.
Building of the model
Can you write Python code for it?

# Function to compute the likelihood P(D|M)
def likelihood (data, parameter):
    p = 1.0
    for d in data:
        if d == 0:
            p *= 1-parameter
        else:
            p *= parameter
    return p

# Function to compute the prior P(M)
def prior (parameter):
    return beta.pdf(parameter, a=2, b=2)

# Function to compute the un-normalized posterior P(D|M) * P(M)
def unnormalized_posterior (data, parameter):
    return likelihood(data, parameter) * prior(parameter)


I am working on the file "Notebooks/MkvChains.ipynb".
The first task is:
Load the books and buildthe dictionaries, and run some simple test for proof of principle.
Can you write Python code for it?

mkv_chain, inv_chain = make_markov_chain(all_books)

#print(mkv_chain)
for i in range(20):
    print('\n',make_chain(mkv_chain,'He said'))


I am working on the file "Notebooks/Moment of Inertia of a Crane.ipynb".
The first task is:
Model the Links as Thin Rods:
$\left[\mathbb{I}{G_1}^{m_1}\right]\mathcal{B}=$
Can you write Python code for it?


I am working on the file "Notebooks/Moment of Inertia of a Crane.ipynb".
The first task is:
Rotate the MOI matrix of $m_1$ into frame $\mathcal{A}$:
$\left[\mathbb{I}{G_1}^{m_1}\right]\mathcal{A}= {}^{\mathcal A}{C}^\mathcal B\left[\mathbb{I}{G_1}^{m_1}\right]\mathcal{B}{}^{\mathcal B}{C}^\mathcal A$
Can you write Python code for it?


I am working on the file "Notebooks/Moment of Inertia of a Crane.ipynb".
The first task is:
Rotate the MOI matrix of $m_2$ into frame $\mathcal{A}$:
${}^{\mathcal A}{C}^\mathcal C = {}^{\mathcal A}{C}^\mathcal B {}^{\mathcal B}{C}^\mathcal c$
$\left[\mathbb{I}{G_2}^{m_2}\right]\mathcal{A}= {}^{\mathcal A}{C}^\mathcal C\left[\mathbb{I}{G_2}^{m_2}\right]\mathcal{C}{}^{\mathcal C}{C}^\mathcal A$
Can you write Python code for it?

bCc = Matrix(([cos(b),sin(b),0],[-sin(b),cos(b),0],[0,0,1])); bCc

aCc = simplify(aCb*bCc); aCc


I am working on the file "Notebooks/Moment of Inertia of a Crane.ipynb".
The first task is:
Simplify by defining $\theta_2 \triangleq \beta - \theta_1$
Can you write Python code for it?

aCc = aCc.subs(b-th1,th2); aCc

I_G2_m2_A = aCc*I_G2_m2_C*aCc.transpose(); I_G2_m2_A


I am working on the file "Notebooks/Moment of Inertia of a Crane.ipynb".
The first task is:
Total Moment of Inertia is the sum of the 3 Moment of Inertia matrices
$\left[\mathbb{I}{O}\right]\mathcal{A} =\left[\mathbb{I}{O}^{m_1}\right]\mathcal{A}+\left[\mathbb{I}{O}^{m_2}\right]\mathcal{A} + \left[\mathbb{I}{O}^{\textrm{cab}}\right]\mathcal{A}$
Can you write Python code for it?

I_O_A = simplify(I_O_cab + I_O_m1_A + I_O_m2_A); I_O_A


I am working on the file "Notebooks/NERC maps.ipynb".
The first task is:
Maps of 2001 and 2017 annual values
Can you write Python code for it?

title = '2001 US Average\n{:.0f} g $\mathregular{{CO_2 \ kWh^{{-1}}}}$'.format(usa_2001)
kwargs = dict(
    regions_lw = 1,
    regions_ec = '0.1',
    SERC_y = -1.5,
    SPP_y = 2.25
)

vmin = regions_albers.loc[:, [2001, 2017]].min().min()
vmax = regions_albers.loc[:, [2001, 2017]].max().max()
plot_nerc_annual(regions_proj=regions_albers, states_proj=states_albers, data_col=2001,
                 text_col='nerc', vmin=vmin, vmax=vmax, title=title,
                 cbar_title='g $\mathregular{CO_2 \ kWh^{-1}}$', **kwargs)

path = join(figure_path, 'NERC map_cividis_2001.pdf')
plt.savefig(path, bbox_inches='tight')#, dpi=350)

title = '2017 US Average\n{:.0f} g $\mathregular{{CO_2 \ kWh^{{-1}}}}$ (↓ 30%)'.format(usa_2017)
kwargs = dict(
    regions_lw = 1,
    regions_ec = '0.1',
    SERC_y = -1.5,
    SPP_y = 2.25,
    FRCC_x = 4.3
)

vmin = regions_albers.loc[:, [2001, 2017]].min().min()
vmax = regions_albers.loc[:, [2001, 2017]].max().max()
regions_albers['arrow reduction'] = '↓ ' + regions_albers['reduction']
plot_nerc_annual(regions_proj=regions_albers, states_proj=states_albers, data_col=2017,
                 text_col='arrow reduction', vmin=vmin, vmax=vmax, title=title,
                 cbar_title='g $\mathregular{CO_2 \ kWh^{-1}}$', **kwargs)
path = join(figure_path, 'NERC map_cividis_2017_change.pdf')
plt.savefig(path, bbox_inches='tight')#, dpi=350)


I am working on the file "Notebooks/NERC maps.ipynb".
The first task is:
Read NERC regions shapefile
Can you write Python code for it?

path = os.path.join(data_path, 'nercregions', 'NERCregions.shp')
regions_nerc = gpd.read_file(path)
regions_nerc['nerc'] = regions_nerc['NERCregion']

regions_nerc

regions_nerc.to_crs(epsg=2163).boundary.plot()


I am working on the file "Notebooks/NERC maps.ipynb".
The first task is:
Read states shapefile
Can you write Python code for it?

path = os.path.join(data_path, 'cb_2016_us_state_20m', 'cb_2016_us_state_20m.shp')
states = gpd.read_file(path)
states.crs

drop_states = ['Alaska', 'Hawaii', 'Puerto Rico']
states = states.loc[~states['NAME'].isin(drop_states)]

states.to_crs(epsg=2163).plot()

path = join(data_path, 'final NERC data',
            'Summary table {}.csv'.format(file_date))
index = pd.read_csv(path, index_col='nerc')
index

index.loc['USA', '2001']

# Add 2001, 2017, and % reduction values from summary table to geodataframe
for nerc in regions_nerc['nerc'].unique():
    try:
        val_2017 = index.loc[nerc, '2017']
        val_2001 = index.loc[nerc, '2001']
        reduce = index.loc[nerc, 'Percent Reduction']
        regions_nerc.loc[regions_nerc['nerc']==nerc, 2017] = val_2017
        regions_nerc.loc[regions_nerc['nerc']==nerc, 2001] = val_2001

        regions_nerc.loc[regions_nerc['nerc']==nerc, 'reduction'] = '{:.0%}'.format(reduce)
        regions_nerc.loc[regions_nerc['nerc']==nerc, 'reduction value'] = reduce
    except:
        pass

# Define national parameters for use in plot titles
usa_2001 = index.loc['USA', '2001']
usa_2017 = index.loc['USA', '2017']

regions_nerc


I am working on the file "Notebooks/NERC maps.ipynb".
The first task is:
Transform projection to Albers equal area
Can you write Python code for it?

regions_albers = regions_nerc.to_crs(epsg=2163)
states_albers = states.to_crs(epsg=2163)


I am working on the file "Notebooks/NVD/nvd_introduction.ipynb".
The first task is:
For the sake of clarity, let's consider just one of the dataframes in the list of dataframes to show how the percent is done.
Can you write Python code for it?

#Choose the first dataframe at position 0
nvd_df = nvd_dataframes[0]
#Parse the timestamp field turning it into a datetimeindex object, and then access the month attribute
nvd_df['month'] = pd.DatetimeIndex(nvd_df['timestamp']).month

#Now that we have a month column, we can 'group by' the table by the month column.
nvd_df = nvd_df.groupby(by=['month'])['cve_id','cwe_id'].count()
nvd_df

#All that is left is divide row-wise the number of cwe_ids, by the numter of cve_ids.
#Since the cwe_ids are never null, then they effectively represent the number of rows for the given month.
#cwe_id, instead, that can be null, will only counted when it occurs.
#Dividing one by the other, gives us the cwe_coverage we desire for the timeseries.
nvd_df['cwe_coverage'] = nvd_df['cwe_id']/nvd_df['cve_id']
nvd_df


I am working on the file "Notebooks/NVD/nvd_introduction.ipynb".
The first task is:
Histograms by Month
Can you write Python code for it?

#using panda
import pandas as pd
import csv

#We first load all file paths
import glob
nvd_filepaths = glob.glob("data/*.csv")
#Then we prepare a list, that will contain all the tables that exist in these file paths
nvd_dataframes = []
for nvd_filepath in nvd_filepaths:
    #the csvs do not contain headers, so are added here. TO-DO: Add headers to CSV when they are generated.
    nvd_dataframes.append(pd.read_csv(nvd_filepath,names=['cve_id', 'cwe_id','timestamp']))


I am working on the file "Notebooks/NVD/nvd_introduction.ipynb".
The first task is:
Now we generate the cwe coverage table for all our dataframes in our list.
Can you write Python code for it?

cwe_coverage_dfs = []
for nvd_df in nvd_dataframes:
    cwe_coverage_dfs.append(calculate_cwe_coverage(nvd_df))
#cwe coverage for the 3rd dataset.
print(cwe_coverage_dfs)


I am working on the file "Notebooks/NVD/nvd_introduction.ipynb".
The first task is:
We now have all the information needed to plot our timeseries.
CWE Coverage Timeseries
Can you write Python code for it?

#imports for histogram
import numpy as np
import pandas as pd
from bokeh.plotting import figure, show
from bokeh.models import Range1d
from bokeh.io import output_notebook
from bokeh.charts import Bar
import matplotlib.pyplot as plot
from datetime import datetime
output_notebook()

color_map = {
'2007': 'red',
'2008': 'green',
'2009': 'yellow',
'2010': 'violet',
'2011': 'indigo',
'2012': 'brown',
'2013': 'black',
'2014': 'blue',
'2015': 'orange',
'2016': 'olive',
'2017': 'navy',
}

def create_multi_line(vul):
    vul_plot = plot.subplot(111)
    map={}
    year = 2006;
    for frame in cwe_coverage_dfs:
        year+=1;
        map[year]= frame['cwe_coverage']
        vul_plot.plot(map[year], label = year )
    vul_plot.set_ylabel('cwe coverage')
    vul_plot.set_xlabel('month')
    vul_plot.legend(bbox_to_anchor=(1.05, 1),loc=2, borderaxespad=0.)
    vul_plot.set_title("Count of Vulnerability type per month")
    vul_plot.set_autoscaley_on(False)
    vul_plot.set_ylim([0,1])
    vul_plot.set_autoscalex_on(False)
    vul_plot.set_xlim([1,12])
    vul_plot.xaxis.set_ticks([1,2,3,4,5,6,7,8,9,10,11,12])
    plot.show()

create_multi_line(cwe_coverage_dfs)


I am working on the file "Notebooks/NoiseHunting/IncoherentSubtraction.ipynb".
The first task is:
Find the PSD of DARM, and the coherence with the aux channel.
Can you write Python code for it?

fftlen=32
overlap=24
coh=darm.coherence(aux,fftlen,overlap)
psd=darm.psd(fftlen,overlap)


I am working on the file "Notebooks/NoiseHunting/IncoherentSubtraction.ipynb".
The first task is:
Subtract the predicted effect of the aux channel on the PSD. Noises should add incoherently.
Can you write Python code for it?

coh_long=zeros(len(psd),dtype=coh.dtype)
coh_long[:len(coh)]=coh.value

psd_sub=(1.-coh_long)*psd

p1=psd.plot()
p1.gca().plot(psd_sub,label='Subtracting DHARD\_Y')
p1.set_xlim(10,100)
p1.set_ylim(1e-47,1e-38)

p1=(psd**0.5).plot()
p1.gca().plot(psd_sub**0.5,label='Subtracting DHARD\_Y')
p1.set_xlim(10,100)
p1.set_ylim(1e-24,1e-19)

def eff_range(my_psd,f_low,f_high):
    """ Calculate the sensemon range. This is just a quick
        estimation. It's mostly useful for comparing ranges
        with different PSDs or different frequency limits."""
    norm=1.8e-21 # Normalizing factor, eyeballed
    idx1=int(f_low/my_psd.df.value)
    idx2=int(f_high/my_psd.df.value)
    integrand=(my_psd.frequencies**(-7./3.)/my_psd).value
    return norm*sqrt(sum(integrand[idx1:idx2]))

eff_range(psd_sub,20,300)/eff_range(psd,20,300)


I am working on the file "Notebooks/Non-Linear-Logistic-Regression.ipynb".
The first task is:
...and this is what we saw last time for linear logistic regression
Can you write Python code for it?

# logistic regression - what we saw last time
# NOTE: The contours are probabilities that the bank note is genuine
errors, accuracy, conf_matrix, report, plot = poly_boundary_plot(X_train_v1_v2,
                    y_train,
                    degree=1,
                    show_contours=0)

# Which rows of the dataset are misclassfied?
errors

# Classification accuracy
accuracy

# Comfusion Matrix
print(conf_matrix)

# True negatives, false positives, false negatives, and true positives
tn, fp, fn, tp = conf_matrix.ravel()
tn, fp, fn, tp

# Precision, recall, f1-score
print(report)


I am working on the file "Notebooks/Non-Linear-Logistic-Regression.ipynb".
The first task is:
At this point, (just imagine that) we've:

visualized the data
defined the task we'd like to accomplish
defined the model
defined the penalty for the being wrong
used an iterative algorithm (like gradient descent) to find the optimal values of the parameters

(Can you picture all of this from the dataset point of view?)

Can you write Python code for it?

# These are the optimal values of w0, w1 and w2
w0 = lr.intercept_[0]
w1 = lr.coef_.squeeze()[0]
w2 = lr.coef_.squeeze()[1]
print("w0: {}\nw1: {}\nw2: {}".format(w0, w1, w2))

# Function for plotting class boundaries
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LogisticRegression
from sklearn import metrics
from sklearn.model_selection import cross_val_score

def poly_boundary_plot(XTrain, YTrain, degree, show_contours=0):

    # XTrain has to have exactly 2 features for this visualization to work

    # Transform the training inputs
    poly = PolynomialFeatures(degree)
    X_train_poly = poly.fit_transform(XTrain['data'])
    # NOTE: the poly function adds a bias value of 1 to each row of input data --
    #  default setting is include_bias=True

    # Set up the logistic regression model from SciKit Learn
    # Solvers that seem to work well are 'liblinear' and 'newton-cg"
    lr = LogisticRegression(C=100.0, random_state=0, solver='liblinear', verbose=2)

    # Fit the polynomial data to the simple linear logistic regression model we have
    lr.fit(X_train_poly, YTrain);

    # Create a grid of feature values

    # Find the min and max values of the two features
    # Make grid values
    GRID_INCREMENT = 0.02
    x1_min = np.array([XTrain['data'][i][0] for i in range(len(XTrain['data']))]).min()
    x1_max = np.array([XTrain['data'][i][0] for i in range(len(XTrain['data']))]).max()

    x2_min = np.array([XTrain['data'][i][1] for i in range(len(XTrain['data']))]).min()
    x2_max = np.array([XTrain['data'][i][1] for i in range(len(XTrain['data']))]).max()

    xx1, xx2 = np.mgrid[x1_min:x1_max:GRID_INCREMENT, x2_min:x2_max:GRID_INCREMENT]
    #xx1.shape, xx2.shape

    # Create the grid
    grid = np.c_[xx1.ravel(), xx2.ravel()]
    grid.shape

    # The predictions of the model
    preds_poly = lr.predict(poly.fit_transform(grid))
    preds_poly_probs = lr.predict_proba(poly.fit_transform(grid))
    preds_poly_probs_0 =...


I am working on the file "Notebooks/Non-Linear-Logistic-Regression.ipynb".
The first task is:
Non-Linear Logistic Regression
Can you write Python code for it?

# logistic regression
# NOTE: The contours are probabilities that the bank note is genuine
errors, accuracy, conf_matrix, report, plot = poly_boundary_plot(X_train_v1_v2,
                                                y_train,
                                                degree=5,
                                                show_contours=1)

# Which rows of the dataset are misclassfied?
errors

# Classification accuracy
accuracy

# Comfusion Matrix
print(conf_matrix)

# True negatives, false positives, false negatives, and true positives
tn, fp, fn, tp = conf_matrix.ravel()
tn, fp, fn, tp

# Precision, recall, f1-score
print(report)


I am working on the file "Notebooks/Non-Linear-Regression.ipynb".
The first task is:
And here's what the expression for the 8th-degree non-linear model looks like:
Can you write Python code for it?

# What does the 8th-degree polynomial expression look like?
print(p8)


I am working on the file "Notebooks/Non-Linear-Regression.ipynb".
The first task is:
Exercise 1
Based on the visuals above, how would you describe the data? Write a short paragraph describing the data.
Rescale/Normalize the Data
Notice that the size of a house in square feet is about 1,000 times the number of bedrooms it has. Similarly, the price of a house is about 100 times the size of the house. This is quite common in many datasets, but when it happens, the iterative method of gradient descent (which we'll use again) becomes inefficient. This is simply a matter of making the computations efficient. To do so, we'll do something called feature normalization.
Can you write Python code for it?

# Normalize the scales in the dataset
# NOTE: the output doesn't have to be rescaled but we've done that here anyway
# If you know statistics: What we're doing is rewriting each value in terms of standard deviation units
data2Norm = (data2 - data2.mean()) / data2.std()
data2Norm.head()

# In Orange use the Preprocessor widget


I am working on the file "Notebooks/Non-Linear-Regression.ipynb".
The first task is:
Non-Linear Regression in Orange - Demonstration
Can you write Python code for it?

# More complicated models -- what the dataset looks like


I am working on the file "Notebooks/Non-Linear-Regression.ipynb".
The first task is:
Step 1: Visualize the Data
Can you write Python code for it?

import seaborn as sns
sns.set(style='whitegrid', context='notebook')
cols = ['Size', 'Bedrooms', 'Price']
sns.pairplot(data2[cols], size=4)
plt.show()

# From https://matplotlib.org/mpl_toolkits/mplot3d/tutorial.html
# 3D scatter plot of size, bedrooms and price
from mpl_toolkits.mplot3d import Axes3D
threeD = plt.figure(figsize=(12,8)).gca(projection='3d')
threeD.scatter(data2['Bedrooms'], data2['Size'], data2['Price'])
threeD.set_xlabel('Bedrooms', fontsize=14)
threeD.set_ylabel('Size', fontsize=14)
threeD.set_zlabel('Price', fontsize=14)
plt.show()


I am working on the file "Notebooks/Non-Linear-Regression.ipynb".
The first task is:
Step 3: Define the Model
Step 3a: Define the Features
In this case our features are exactly the same as our inputs. We have 2 features: the size of a house in square feet and the number of bedrooms a house has. These are the features encoded in the variables $x_{1}$ and $x_{2}$.
Later we'll see models where the features and the inputs are not one and the same. In these cases the features are constructed by combining inputs in various ways.
Step 3b: Transform the Inputs Into an Output
When we were doing linear regression with multiple features our model -- the transformation of inputs into the output looked like this:
$$\hat{y} = w_{0} * x_{0}\ +\ w_{1} * x_{1} +\ w_{2} * x_{2}$$
This is linear because there are not squares or cubes or anything like that. But what if we wanted to say that the price of a house is linear with respect to its size but the square of the number of bedrooms? We'd write it like this:
$$\hat{y} = w_{0} * x_{0}\ +\ w_{1} * x_{1} +\ w_{2} * x_{2}^{2}$$
Rather than just writing an expression with multiplication and addition, we now have powers -- in this case, $x_{2}^{2}$.
Acutally, we're free to make the model anything we want. It depends on what we think is best for prediction. So, we can make it complicated:
$$\hat{y} = w_{0} * x_{0}\ +\ w_{1} * x_{1} +\ w_{2} * x_{1} * x_{2} +\ w_{3} * x_{1}^2 +\ w_{4} * x_{2}^{5}$$
That seems crazy, but the point is simply that this is our model to choose and we have lots of choices!
How Non-Linear Terms Fit the Data
To get a feel for how these non-linear models behave, let's look a simplified dataset -- the one we're using but with the number of bedrooms removed. We just have the square footage and the price. This allows us to visualize things in 2 dimensions.
Let's visualize the fit for various degrees of polynomial functions. This is how we're going to make the transformation from inputs to output non-linear.
Can you write Python code for it?

# First drop the Bedrooms column from the data set
# Notice -- this is the raw data -- no scaling yet.
data3 = data2.drop('Bedrooms', axis = 1)
data3.head()

# Getting a handle on the simplified dataset
data3.describe()

# Using non-linear (polynomial) models to "fit" the simplified dataset

X_p = data3['Size']
y_p = data3['Price']
xx = np.linspace(500, 5000, 100)
#print(xx)

# fit the data with a first degree polynomial
z1 = np.polyfit(X_p, y_p, 1)
p1 = np.poly1d(z1)

# fit the data with a 2nd degree polynomial
z2 = np.polyfit(X_p, y_p, 2)
p2 = np.poly1d(z2) # construct the polynomial (note: that's a one in "poly1d")

# fit the data with a 3rd degree polynomial
z3 = np.polyfit(X_p, y_p, 3)
p3 = np.poly1d(z3) # construct the polynomial

# fit the data with a 4th degree polynomial
z4 = np.polyfit(X_p, y_p, 4)
p4 = np.poly1d(z4) # construct the polynomial

# fit the data with a 8th degree polynomial - just for the heck of it :-)
z8 = np.polyfit(X_p, y_p, 8)
p8 = np.poly1d(z8) # construct the polynomial

# fit the data with a 16th degree polynomial - just for the heck of it :-)
z16 = np.polyfit(X_p, y_p, 16)
p16 = np.poly1d(z16) # construct the polynomial

plt.figure(figsize=(12,8))
plt.plot(X_p, y_p, 'o', label='data')
plt.title('Non-Linear Ways to Fit Housing Prices', fontsize=14)
plt.xlabel('Size')
plt.ylabel('Price')
plt.plot(xx, p1(xx), 'b-', label='linear')
#plt.plot(xx, p2(xx), 'g-', label='2nd degree')
#plt.plot(xx, p3(xx), 'y-', label='3rd degree')
#plt.plot(xx, p4(xx), 'r-', label='4th degree')
plt.plot(xx, p8(xx), 'c-', label='8th degree')
plt.plot(xx, p16(xx), 'm-', label='16th degree')
plt.legend(loc=2)
plt.axis([500,5000,150000,800000]); # Useful for higher degrees of polynomials


I am working on the file "Notebooks/Non-Linear-Regression.ipynb".
The first task is:
Step 3c: Clarify the Parameters of the Model
$w_{0}$, $w_{1}$, and $w_{2}$ are the parameters of the model. These parameters can each take on an infinite number of values. In other words they are continuous variables. $w_{0}$ is called the intercept or bias value.
With our model defined above, we know exactly how to transform an input into an output -- that is, once the values of the parameters are given.
Let's pick a value of X from the dataset, fix a specific value for $w_{0}$ and $w_{1}$, and see what we get for the value of y.
Specifically, let
$\begin{bmatrix}
w_{0} \
w_{1} \
w_{2}
\end{bmatrix} =
\begin{bmatrix}
-10 \
1 \
0
\end{bmatrix}$
This means $w_{0}$ is -10, $w_{1}$ is 1, and $w_{2}$ is 0.
Let's try out the scheme for transforming the first few rows of X_2.
Can you write Python code for it?

# X_2 * W for the first 5 rows of X_2 (more accurately: X_2 * W transpose)
df_addOnes(X_2.iloc[0:5]) * np.matrix('-10;1;0')

# Initialize the parameter values, the learning rate and the number of interations
W_init = [-1, 1.4, 0.5]
learning_rate = 0.005 # the learning rate
num_iters = 10 # number of iterations

# Run gradient descent
# Outputs generated by our model for the first 5 inputs with the specific w values below
W_opt, final_penalty, running_w, running_penalty = gradientDescent(X_2, y, W_init, num_iters, learning_rate)

# These are initial predictions
# Compare these outputs to the actual values of y in the dataset (after de-scaling)
(df_addOnes(X_2.iloc[0:5]) * np.matrix(W_opt))


I am working on the file "Notebooks/Non-Linear-Regression.ipynb".
The first task is:
Step 6: Use the Model and Optimal Parameter Values to Make Predictions
It looks like a learning rate greater than 0.003 is good enough to get our iterative gradient descent to plunge down to arrive at the lowest cost value and stay there.
Let's make some predictions...What is our prediction for a house that is 5,000 square feet in size with 4 bedrooms? Let's plug these in to our model and use the optimal $W$ values we've calculated above.
Can you write Python code for it?

# Change size and num_bedrooms to make distinct predictions
size = 30000
num_bedrooms = 5

# Remember we've run the model using rescaled house sizes and number of bedrooms
# So we should scale the inputs down and then scale the prediction up when we're done
size_scaled = (size - data2.mean()[0])/data2.std()[0]
beds_scaled = (num_bedrooms - data2.mean()[1])/data2.std()[1]

# This is our model -- we're just using it here to make a calculation
pred_price = (W_opt[0] * 1) + (W_opt[1] * size_scaled) + (W_opt[2] * beds_scaled)

# Get the optimal W values into the right form for display
W_opt_display = np.array(W_opt.squeeze()).squeeze()

# Format and print the result
print("Predicted Price: ", '${:8,.0f}'.format(math.ceil(pred_price)))
print("Optimal Parameter Values: {}".format(W_opt))


I am working on the file "Notebooks/Non-Linear-Regression.ipynb".
The first task is:
The Business Problem: Predicting Housing Prices
What's the market value of a house? One way to determine the market value is to collect up the prices of houses based on a few characteristics such as size in square feet and number of bedrooms. Then apply machine learning to "learn" what the price should be based on this data.
The data we have is a well know machine learning dataset of housing prices in Portland, Oregon. Let's refresh our memories of what this dataset looks like.
Load the Data
Can you write Python code for it?

import os
# Load the housing prices dataset
file_url = data_dir + os.sep + "portland-house-prices.txt"
# Load the data into a dataframe
data2 = pd.read_csv(file_url, header=None, names=['Size', 'Bedrooms', 'Price'])
data2.head()


I am working on the file "Notebooks/Non-Linear-Regression.ipynb".
The first task is:
We don't know yet if this is high or low -- we'll have to try out a whole bunch of $W$ values. Or better yet, we can use pick an iterative method and implement it.
Step 5: Find the Parameter Values that Minimize the Penalty
Once again, the method that will "learn" the optimal values for $W$ is gradient descent. Let's use it to find the minimum cost and the values of $W$ that result in that minimum cost.
Can you write Python code for it?

# Set hyper-parameters
num_iters = 2000 # number of iterations
learning_rate = 0.0001 # the learning rate

# Run gradient descent and capture the progression of cost values and the ultimate optimal W values
%time W_opt, final_penalty, running_w, running_penalty = gradientDescent(X_2, y, W_init, num_iters, learning_rate)
# Get the optimal W values and the last few W values and cost values
W_opt, final_penalty, running_w[-5:], running_penalty[-5:]

# How the cost changes as the number of iterations increase
fig, ax = plt.subplots(figsize=(8,5))
ax.plot(np.arange(num_iters), running_penalty, 'g')
ax.set_xlabel('Number of Iterations')
ax.set_ylabel('Cost')
plt.xlim(0,num_iters)
ax.set_title('Cost vs. Iterations Over the Dataset');

# Run gradient descent for a few different values of the learning rate
learning_rates = [0.0001, 0.003, 0.005, 0.01, 0.03, 0.1]
gdResults = [gradientDescent(X_2, y, W_init, num_iters, learning_rates[i]) for i in range(len(learning_rates))]
#gdResults

# For each learning rate, get the progression of costs
#  for each iteration
penalty_list = [gdResults[i][3] for i in range(len(gdResults))]
penalty_list[0]

# How the cost of the transformation varies with the learning rate
plot_color_list = ['g', 'r', 'c', 'b', 'm', 'y']
fig, ax = plt.subplots(figsize=(12,8))
[ax.plot(np.arange(num_iters), penalty_list[i], plot_color_list[i], label=learning_rates[i]) for i in range(len(learning_rates))]
ax.set_xlabel('Iterations')
ax.set_ylabel('Cost')
ax.legend()
plt.xlim(0,100)
ax.set_title('Cost vs. Learning Rate');


I am working on the file "Notebooks/Non-Linear-Regression.ipynb".
The first task is:
What does this prediction surface look like?
Can you write Python code for it?

'''
from https://matplotlib.org/mpl_toolkits/mplot3d/tutorial.html
======================
3D surface (color map)
======================

Demonstrates plotting a 3D surface colored with the coolwarm color map.
The surface is made opaque by using antialiased=False.

Also demonstrates using the LinearLocator and custom formatting for the
z axis tick labels.
'''

from mpl_toolkits.mplot3d import Axes3D
import matplotlib.pyplot as plt
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
import numpy as np

fig = plt.figure(figsize=(14,10))
ax = fig.gca(projection='3d')

# Make data.
X = np.arange(-5, 5, 1)
Y = np.arange(-5, 5, 1)
X, Y = np.meshgrid(X, Y)
#R = np.sqrt(X**2 + Y**2)
#Z = np.sin(R)
Z = W_opt_display[0] + W_opt_display[1] * X + W_opt_display[2] * Y

# Plot the surface.
surf = ax.plot_surface(X, Y, Z, cmap=cm.coolwarm,
                       linewidth=0, antialiased=True)

plt.xlabel('Size in Square Feet (Scaled)', fontsize=12)
plt.ylabel('# Bedrooms Squared (Scaled)', fontsize=12)
plt.title('Predicted Price', fontsize=14)

# Customize the z axis.
#ax.set_zlim(100000, 1000000)
ax.zaxis.set_major_locator(LinearLocator(10))
ax.zaxis.set_major_formatter(FormatStrFormatter('%.0f'))

# Add a color bar which maps values to colors.
fig.colorbar(surf, shrink=0.5, aspect=5)

plt.show()

'''
From https://matplotlib.org/mpl_toolkits/mplot3d/tutorial.html
=================
3D wireframe plot
=================

A very basic demonstration of a wireframe plot.
'''

from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(14,10))
ax = fig.add_subplot(111, projection='3d')

# Grab some test data.
#X, Y, Z = axes3d.get_test_data(0.05)
X = np.arange(-5, 5, 1)
Y = np.arange(-5, 5, 1)
Z = W_opt_display[0] + W_opt_display[1] * X + W_opt_display[2] * Y

# Plot a basic wireframe.
ax.plot_wireframe(X, Y, Z, rstride=1, cstride=1)

plt.show()


I am working on the file "Notebooks/Numpy+Scipy/1 - Introduction to Numpy.ipynb".
The first task is:
Arrays are compatible as long as each of their dimensions (shape) is either equal to the other or 1.
Thus, above, the multplication works when a.shape is (1,3) and b.shape is either (1,3) or (1,1)
(Actually, these are (3,) and (1,) in the examples above ...
Can you write Python code for it?

print a.shape
print b.shape
print (a*b).shape
print (a+b).shape

aa = a.reshape(1,3)
bb = b.reshape(1,1)

print aa.shape
print bb.shape
print (aa*bb).shape
print (aa+bb).shape


I am working on the file "Notebooks/Numpy+Scipy/1 - Introduction to Numpy.ipynb".
The first task is:
Broadcasting is a way of looping on arrays which have "compatible" but unequal sizes.
For example, the element-wise multiplication of 2 arrays
python
a = np.array([1.0, 2.0, 3.0])
b = np.array([2.0, 2.0, 2.0])
print a * b
has an equivalent:
python
a = np.array([1.0, 2.0, 3.0])
b = np.array([2.0])
print a * b
or
python
a = np.array([1.0, 2.0, 3.0])
b = 2.0
print a * b
in which the "appropriate" interpretation of b is made in each case to achieve the result.
Can you write Python code for it?


I am working on the file "Notebooks/Numpy+Scipy/1 - Introduction to Numpy.ipynb".
The first task is:
But not for
Can you write Python code for it?


I am working on the file "Notebooks/Numpy+Scipy/1 - Introduction to Numpy.ipynb".
The first task is:
Exercise: Try this again for a 10000x10000 array
Indexing / broadcasting
In numpy, we can index an array by explicitly specifying elements, by specifying slices, by supplying lists of indices (or arrays), we can also supply a boolean array of the same shape as the original array which will select / return an array of all those entries where True applies.
Although some of these might seem difficult to use, they are often the result of other numpy operations. For example np.where converts a truth array to a list of indices.
Can you write Python code for it?

AA = np.zeros((100,100))

AA[10,11] = 1.0
AA[99,1]  = 2.0

cond = np.where(AA >= 1.0)
print cond
print AA[cond]
print AA[ AA >= 1]


I am working on the file "Notebooks/Numpy+Scipy/1 - Introduction to Numpy.ipynb".
The first task is:
Exercise: find out how long it takes to compute the sin, sqrt, power of a 1000000 length vector (array). How does this speed compare to using the normal math functions element by element in the array ? What happens if X is actually a complex array ?
Hints: you might find it useful to know about:
  - np.linspace v np.arange
  - np.empty_like or np.zeros_like
  - the python enumerate function
  - how to write a table in markdown
| description     | time   | notes |
|-----------------|--------|-------|
| np.sin        | ?      |       |
| math.sin      | ?      |       |
|                 | ?      |  -    |
Can you write Python code for it?

X = np.linspace(0.0, 2.0*np.pi, 10000000)
print X.shape

# ...

%%timeit
S = np.sin(X)

%%timeit

S = np.empty_like(X)
for i, x in enumerate(X):
    S[i] = math.sin(x)

X = np.linspace(0.0, 2.0*np.pi, 10000000)
Xj = X + 1.0j
print Xj.shape, Xj.dtype

%%timeit

Sj = np.sin(Xj)

import cmath

%%timeit

Sj = np.empty_like(Xj)
for i, x in enumerate(Xj):
    Sj[i] = cmath.sin(x)


I am working on the file "Notebooks/Numpy+Scipy/1 - Introduction to Numpy.ipynb".
The first task is:
Exercise: look through the functions below from numpy, choose 3 of them and work out how to use them on arrays of data. Write a few lines to explain what you find.

np.max v. np.argmax
np.where
np.logical_and
np.fill_diagonal
np.count_nonzero
np.isinf and np.isnan

Here is an example:
np.concatenate takes a number of arrays and glues them together. For 1D arrays this is simple:
```python
A = np.array([1.0,1.0,1.0,1.0])
B = np.array([2.0,2.0,2.0,2.0])
C = np.array([3.0,3.0,3.0,3.0])
R = np.concatenate((A,B,C))
array([ 1.,  1.,  1.,  1.,  2.,  2.,  2.,  2.,  3.,  3.,  3.,  3.])
```
an equivalent statement is np.hstack((A,B,C)) but note the difference with np.vstack((A,B,C))
With higher dimensional arrays, the gluing takes place along one axis:
```python
A = np.array(([1.0,1.0,1.0,1.0],[2.0,2.0,2.0,2.0]))
B = np.array(([3.0,3.0,3.0,3.0],[4.0,4.0,4.0,4.0]))
C = np.array(([5.0,5.0,5.0,5.0],[6.0,6.0,6.0,6.0]))
R = np.concatenate((A,B,C))
print R
print
R = np.concatenate((A,B,C), axis=1)
print R
```
Can you write Python code for it?


I am working on the file "Notebooks/Numpy+Scipy/1 - Introduction to Numpy.ipynb".
The first task is:
Vector Operations
Can you write Python code for it?

X = np.arange(0.0, 2.0*np.pi, 0.0001)

print X[0:100]

import math

math.sin(X)

np.sin(X)

S = np.sin(X)
C = np.cos(X)

S2 =  S**2 + C**2
print S2

print S2 - 1.0

test = np.isclose(S2,1.0)
print test

print np.where(test == False)
print np.where(S2 == 0.0)


I am working on the file "Notebooks/Numpy+Scipy/1 - Introduction to Numpy.ipynb".
The first task is:
Views of the data (are free)
It costs very little to look at data in a different way (e.g. to view a 2D array as a 1D vector).
Making a copy is a different story
Can you write Python code for it?

print A.reshape((2,8))
print

print A.reshape((-1))
print A.ravel()
print

print A.reshape((1,-1))
print

%%timeit

A.reshape((1,-1))

%%timeit

elements = A.shape[0]*A.shape[1]
B = np.empty(elements)
B[...] = A[:,:].ravel()

%%timeit

elements = A.shape[0]*A.shape[1]
B = np.empty(elements)

for i in range(0,A.shape[0]):
    for j in range(0,A.shape[1]):
        B[i+j*A.shape[1]] = A[i,j]


I am working on the file "Notebooks/Numpy+Scipy/2 - Application - The Game of Life.ipynb".
The first task is:
Exercise:   Your task is to write a function that "runs" the game of life. This should be possible by filling out the two function templates below.

conditions: boundaries are always dead
Can you write Python code for it?

def get_neighbours(start):
"""
This function gets the number of live neighbours in the binary array start
start : np.ndarray
"""

def game_of_life(start, n):
    """
    this function runs the game of life for n steps...
    start : np.ndarray (0s and 1s)
    n: int number of steps
    """
    assert (1>= start.min() >= 0) and (1>= start.max() >= 0), "array must be ones and zeros"

    current = np.copy(start)

    while n:
        neighbours = get_neighbours(current)

        for index, value in np.ndenumerate(current):
            print(index, value)
            # Apply the rules to current
            if ...

            else ...

        n -= 1

    return current


I am working on the file "Notebooks/Numpy+Scipy/2 - Application - The Game of Life.ipynb".
The first task is:
We will talk more about plotting later, but for now we can use this without digging deeper:
Can you write Python code for it?

plt.imshow(start, interpolation='nearest', cmap="gray")

print start[4:8,4:8]  # neighbours of start[5,5]
print start[1:4,1:4]  # neighbours of start[2,2]
#print start[?:?]  # neighbours of start[1,1]
#print start[?:?] # neighbours of start[0,0]

live_neighbours = np.empty(start.shape)
for index, value in np.ndenumerate(start):
    #Need to add 2, becase the slicing works like 'up to but not including'
    x0 = max(0,(index[0]-1))
    x1 = max(0,(index[0]+2))
    y0 = max(0,(index[1]-1))
    y1 = max(0,(index[1]+2))
    subarray = start[x0:x1, y0:y1]
    live_neighbours[index] = subarray.sum() - value # need to subtract actual value at that cell...

live_neighbours


I am working on the file "Notebooks/Numpy+Scipy/5 - Scipy Interpolate.ipynb".
The first task is:
1D data
Can you write Python code for it?

from scipy.interpolate import interp1d

x = np.linspace(0, 10, num=11, endpoint=True)
y = np.cos(-x**2/9.0)
f = interp1d(x, y, kind='linear') # default if kind=None
f2 = interp1d(x, y, kind='cubic')
f3 = interp1d(x, y, kind='nearest')

xnew = np.linspace(0, 10, num=41, endpoint=True)
plt.plot(x, y, 'o', xnew, f(xnew), '-', xnew, f2(xnew), '--', xnew, f3(xnew), '.-')
plt.legend(['data', 'linear', 'cubic', 'nearest'], loc='best')
plt.show()


I am working on the file "Notebooks/Numpy+Scipy/5 - Scipy Interpolate.ipynb".
The first task is:
Splines
Which have the added benefit of giving smooth derivative information
Can you write Python code for it?

from scipy.interpolate import splrep, splev

x = np.arange(0, 2*np.pi+np.pi/4, 2*np.pi/8)
y = np.sin(x)
tck = splrep(x, y, s=0)
xnew = np.arange(0, 2*np.pi, np.pi/50)
ynew = splev(xnew, tck, der=0)
yder = splev(xnew, tck, der=1)

plt.figure()
plt.plot(x, y, 'x', xnew, ynew, xnew, np.sin(xnew), x, y, 'b')
plt.legend(['Linear', 'Cubic Spline', 'True'])
plt.axis([-0.05, 6.33, -1.05, 1.05])
plt.title('Cubic-spline interpolation')
plt.show()

plt.figure()
plt.plot(xnew, yder, xnew, np.cos(xnew),'--')
plt.legend(['Cubic Spline', 'True'])
plt.axis([-0.05, 6.33, -1.05, 1.05])
plt.title('Derivative estimation from spline')
plt.show()


I am working on the file "Notebooks/Numpy+Scipy/5 - Scipy Interpolate.ipynb".
The first task is:
nD data
There are fewer approaches to n-dimensional data, the evaluation for arbitrary dimensions is always for points on an n dimensional grid.
Can you write Python code for it?

from scipy.interpolate import griddata

def func(x, y):
    return x*(1-x)*np.cos(4*np.pi*x) * np.sin(4*np.pi*y**2)**2

# A regular grid array of x,y coordinates

grid_x, grid_y = np.mgrid[0:1:100j, 0:1:200j] # see np.info(np.mgrid) for an explanation of the 200j !!

np.info(np.mgrid)

# A random sampling within the same area

points = np.random.rand(1000, 2)
values = func(points[:,0], points[:,1])

# Resample from the values at these points onto the regular mesh

grid_z0 = griddata(points, values, (grid_x, grid_y), method='nearest')
grid_z1 = griddata(points, values, (grid_x, grid_y), method='linear')
grid_z2 = griddata(points, values, (grid_x, grid_y), method='cubic')

plt.subplot(221)
plt.imshow(func(grid_x, grid_y).T, extent=(0,1,0,1), origin='lower', cmap='jet')
plt.plot(points[:,0], points[:,1], 'k.', ms=1)
plt.title('Original')
plt.subplot(222)
plt.imshow(grid_z0.T, extent=(0,1,0,1), origin='lower', cmap='jet')
plt.title('Nearest')
plt.subplot(223)
plt.imshow(grid_z1.T, extent=(0,1,0,1), origin='lower', cmap='jet')
plt.title('Linear')
plt.subplot(224)
plt.imshow(grid_z2.T, extent=(0,1,0,1), origin='lower', cmap='jet')
plt.title('Cubic')
plt.gcf().set_size_inches(6, 6)
plt.show()


I am working on the file "Notebooks/Numpy+Scipy/6 - Scipy Spatial and Meshing.ipynb".
The first task is:
KDtree
Allows very fast point to point searches.
Can you write Python code for it?

from scipy.spatial import KDTree, cKDTree

tree = cKDTree(points)

print tree.data

%%timeit

tree.query((0.5,0.5))

test_points = np.random.rand(1000, 2)   # 1000 random points in 2-D

%%timeit

tree.query(test_points)

more_points = np.random.rand(10000, 2)   # 1000 random points in 2-D

big_tree = KDTree(more_points)

%%timeit

KDTree(more_points)

%%timeit

big_tree.query(test_points)


I am working on the file "Notebooks/Numpy+Scipy/8 - Numpy, Scipy Miscellany.ipynb".
The first task is:
netcdf
Many earth-science datasets are stored in the netcdf format. scipy provides a reader for this format
Can you write Python code for it?

nf = netcdf.netcdf_file(filename="../../Data/Reference/velocity_AU.nc")

from pprint import pprint # pretty printer for python objects

pprint( nf.dimensions )
pprint( nf.variables )

print nf.variables["lat"].data.shape
print nf.variables["lon"].data.shape
print nf.variables['ve'].data.shape
print nf.variables['vn'].data.shape


I am working on the file "Notebooks/Numpy+Scipy/8 - Numpy, Scipy Miscellany.ipynb".
The first task is:
text-based data files
Completely portable and human readable, text-based formats are common outputs from simple scripted programs, web searches, program logs etc. Reading and writing them is trivial, and it is easy to append information to a file. The only problem is that the conversion can be extremely slow.
This example is taken from our mapping exercise and shows the benefit of converting data to binary formats.

```python
Seafloor age data and global image - data from Earthbyters
The data come as ascii lon / lat / age tuples with NaN for no data.
This can be loaded with ...
age = np.loadtxt("global_age_data.3.6.xyz")
age_data = age.reshape(1801,3601,3)  # I looked at the data and figured out what numbers to use
age_img  = age_data[:,:,2]
But this is super slow, so I have just stored the Age data on the grid (1801 x 3601) which we can reconstruct easily
datasize = (1801, 3601, 3)
age_data = np.empty(datasize)
ages = np.load("global_age_data.3.6.z.npz")["ageData"]
lats = np.linspace(90, -90, datasize[0])
lons = np.linspace(-180.0,180.0, datasize[1])
arrlons,arrlats = np.meshgrid(lons, lats)
age_data[...,0] = arrlons[...]
age_data[...,1] = arrlats[...]
age_data[...,2] = ages[...]
```

The timing comparison is astonishing
On my laptop the numpy binary file is about a million times faster to read. I cut out the lat/lon values from this file to save some space, but this would add, at most, a factor of 3 to the npz timing.
Can you write Python code for it?

%%timeit

gadtxt = np.loadtxt("../../Data/Reference/global_age_data.3.6.xyz")

%%timeit

gadnpy = np.load("../../Data/Reference/global_age_data.3.6.z.npz")


I am working on the file "Notebooks/Numpy+Scipy/Incomplete/_Seasonal_averages.ipynb".
The first task is:
Creating our seasonal averages
Before operating on the whol data array, we will have a play around with the times array and see if we can find a way to manipulate our data so that we have, for instance, all of the Autumn data in one, all of the Summer data, amd so on. Let's have a look at how the times array looks:
Can you write Python code for it?

print(times)

#First split array of 36 months into 12 separate arrays, each an individual season
#We do this with the np.split fucntion (note that the output is actually a list of arrays)
seasons = np.array_split(times, 12)

#Next we use slicing on our list of arrays, with a 'step' parameter
#This gathers every fourth seasons, which is what we want
SON = seasons[::4]
DJF = seasons[1::4]
MAM = seasons[2::4]
JJA = seasons[3::4]
print(JJA, type(JJA))

#Now we apply exactly the same steps to the time axis of our temperature data
#Note that because array_split creates lists, we need to manually convert back to arrays
dseasons = np.array_split(tair[:, :, :], 12)
yearlySON = np.array(dseasons[::4])
yearlyDJF = np.array(dseasons[1::4])
yearlyMAM = np.array(dseasons[2::4])
yearlyJJA = np.array(dseasons[3::4])

#Now we can compute the (UNWEIGHTED) Seasonal means
mean_spring = yearlySON.mean(axis=(0,1))
mean_summer = yearlyDJF.mean(axis=(0,1))
mean_autumn = yearlyMAM.mean(axis=(0,1))
mean_winter = yearlyJJA.mean(axis=(0,1))


I am working on the file "Notebooks/Numpy+Scipy/Incomplete/_Seasonal_averages.ipynb".
The first task is:
Exercise:
So far we've computed a reasonable approximation to the the seasonal averages. The deficiency in our code is that our averages were unweighted with respect to the number of days in each month, which is variable.
Your task is to rectify this. We have provided nearly all of the tools to do this, you'll juts have to and combine things in a slightly different way.
So, you need to create some new arrays, (similar to e.g. mean_summer, mean_spring), but you need to apply a weighted mean. Once you've created your new arrays, try plotting the arrays for unweighted average , the weighted average, and the difference between the two. You shoud be able to reuse the plot setup from above, just change the name of teh array.
Hints: the np.average() function takes an array of weights as an optional argument.
Helper functions for the weighted means
Can you write Python code for it?

#Utility dictionary

dpm = {'noleap': [0, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31],
       '365_day': [0, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31],
       'standard': [0, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31],
       'gregorian': [0, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31],
       'proleptic_gregorian': [0, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31],
       'all_leap': [0, 31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31],
       '366_day': [0, 31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31],
       '360_day': [0, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30]}

def leap_year(year, calendar='standard'):
    """Determine if year is a leap year"""
    leap = False
    if ((calendar in ['standard', 'gregorian',
        'proleptic_gregorian', 'julian']) and
        (year % 4 == 0)):
        leap = True
        if ((calendar == 'proleptic_gregorian') and
            (year % 100 == 0) and
            (year % 400 != 0)):
            leap = False
        elif ((calendar in ['standard', 'gregorian']) and
                 (year % 100 == 0) and (year % 400 != 0) and
                 (year < 1583)):
            leap = False
    return leap

def get_dpm(time, calendar='standard'):
    """
    return a array of days per month corresponding to the months provided in `months`
    """
    month_length = np.zeros(len(time), dtype=np.int)

    cal_days = dpm[calendar]

    for i, dt in enumerate(times):
        #print i, dt
        month_length[i] = cal_days[dt.month]
        if leap_year(dt.year, calendar=calendar):
            month_length[i] += 1
    return month_length

##Example of how to use the get_dpm fucntion ("get days per month")
dayspermonth = get_dpm(times, calendar='standard')
dayspermonth


I am working on the file "Notebooks/Numpy+Scipy/Incomplete/_Seasonal_averages.ipynb".
The first task is:
Import Data
netCDF is a binary storage format popular in Earth Sciences. In this exercis, we simply want to corece the data into  Numpy array (2D). Becaue netCDF is a a more sophisticated data format that we're used to, we'll use a dedicated library netCDF4. The main point to remember is that we'll soon have a Numoy array - a familiar object that we know how to handlde.
Can you write Python code for it?

f = nc.Dataset('../../data/example_data.nc')
# We acess the data through a variables dictionary.
# Let's print th dictionary keys so we know what we're dealing with
print(f.variables.keys())

 # use the keys 'Tair' and 'time' to get netcdf data as Numpy arrays

tair = f.variables['Tair'][:] # temperature variable
time = f.variables['time'][:] #time variable

#Let's find our what shape our numpy array is:
tair.shape


I am working on the file "Notebooks/Numpy+Scipy/Incomplete/_Seasonal_averages.ipynb".
The first task is:
Okay, so that looks good. The netcdf variables dictionary (metadata) told us that the dimensions of this data set were: dimensions(sizes): time(36), y(205), x(275)
It also gave us a a nice summary of our data. The time axis has values that correspond to:
calendar: noleap
dimensions: 1
long_name: time
type_preferred: int
units: days since 0001-01-01
Let's use a slice to take a look at data for the first month:
Can you write Python code for it?

#Use our go-to 2D array plotting
month1 = tair[0,:,:]
plt.imshow(test, cmap='terrain')

#And as usual, let's check the type of the data:

type(month1)


I am working on the file "Notebooks/Numpy+Scipy/Incomplete/_Seasonal_averages.ipynb".
The first task is:
Remember the netcdf variable dictionary, which contained metadata, told us that out time had "units: days since 0001-01-01".
This is a great in one way, because days are a linear measurement of time, whereas months and years are not. We will, however, want to work with months and years, and this is where we use the nc.num2date function that we imported.
Can you write Python code for it?

times = nc.num2date(time, 'days since 0001-01-01', 'noleap')
times


I am working on the file "Notebooks/Numpy+Scipy/Incomplete/_Seasonal_averages.ipynb".
The first task is:
This might be somthing new to you. Masked arrays are arrays that may have missing or invalid entries. The numpy.ma module provides a nearly work-alike replacement for numpy that supports data arrays with masks.
In the lessons we dealt with some numpy arrays that contained NaNs. Masked arrays are nice because we use them almost exactly like we would an normal array. The masked parts of the array are simply invisible to the operations we perform.
Remeber that when our Numpy arrays contained NaNs, we had to resort to using the special methods like np.nanmean() fucntions. With masked arrays we just call regular methods:
Can you write Python code for it?

print('mean of month one is:', month1.mean())

np.nan


I am working on the file "Notebooks/Numpy+Scipy/Incomplete/_Seasonal_averages.ipynb".
The first task is:
What we have done is to transform those "days since 0001-01-01" into special object that originate in a module called datetime.
Can you write Python code for it?


I am working on the file "Notebooks/Numpy+Scipy/Incomplete/_Seasonal_averages.ipynb".
The first task is:
You don't need to know much about the datetime objects, only that they have some useful attributes and methods:
Can you write Python code for it?

print("day is:",times[0].day, "month is:",times[0].month, "year is:", times[0].year)


I am working on the file "Notebooks/Numpy+Scipy/Incomplete/_Seasonal_averages.ipynb".
The first task is:
get datetime objects
We're happy with the structure and behaviour of our spatial data. Now we should have a look at the time array we creaed.
Can you write Python code for it?