I am working on the file "00_Intro_Python_Jupyter_notebooks/2_Jupyter_strings_and_lists.ipynb".
The first task is:
Pay attention:

the for statement ends with a colon, :
the variable fruit is implicitly defined in the for statement
fruit takes the (string) value of each element of the list fruits, in order
the indented print() statement is executed for each value of fruit
once Python runs out of fruits, it stops
we don't need to know ahead of time how many items are in the list!

Challenge question:
— What is the value of the variable fruit after executing the for statement above? Discuss with your neighbor. (Confirm your guess in a code cell.)
A very useful function to use with for statements is enumerate(): it adds a counter that you can use as an index while your iteration runs. To use it, you implicitly define two variables in the for statement: the counter, and the value of the sequence being iterated on.
Study the following block of code:
Can you write Python code for it?

names = ['sam', 'zoe', 'naty', 'gil', 'tom']

for i, name in enumerate(names):
    names[i] = name.capitalize()
print(names)


I am working on the file "00_Intro_Python_Jupyter_notebooks/2_Jupyter_strings_and_lists.ipynb".
The first task is:
Play with data in Jupyter
This is the second lesson of our course in "Engineering Computations." In the first lesson, Interacting with Python, we used IPython, the interactive Python shell. It is really great to type single-line Python expressions and get the outputs, interactively. Yet, believe it or not, there are greater things!
In this lesson, you will continue playing with data using Python, but you will do so in a Jupyter notebook. This very lesson is written in a Jupyter notebook. Ready? You will love it.
What is Jupyter?
Jupyter is a set of open-source tools for interactive and exploratory computing. You work right on your browser, which becomes the user interface through which Jupyter gives you a file explorer (the dashboard) and a document format: the notebook.
A Jupyter notebook can contain: input and output of code, formatted text, images, videos, pretty math equations, and much more. The computer code is executable, which means that you can run the bits of code, right in the document, and get the output of that code displayed for you. This interactive way of computing, mixed with the multi-media narrative, allows you to tell a story (even to yourself) with extra powers!
Working in Jupyter
Several things will seem counter-intuitive to you at first. For example, most people are used to launching apps in their computers by clicking some icon: this is the first thing to "unlearn." Jupyter is launched from the command line (like when you launched IPython). Next, we have two types of content—code and markdown—that handle a bit differently. The fact that your browser is an interface to a compute engine (called "kernel") leads to some extra housekeeping (like shutting down the kernel). But you'll get used to it pretty quick!
Start Jupyter
The standard way to start Jupyter is to type the following in the command-line interface:
jupyter notebook
Hit enter and tadah!!
After a little set up time, your default browser will open with the Jupyter app. It should look like in the screenshot below, but you may...
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/2_Jupyter_strings_and_lists.ipynb".
The first task is:
Remember that we can concatenate strings ("add"), for example:
Can you write Python code for it?

new_string = str_1 + str_2
print(new_string)


I am working on the file "00_Intro_Python_Jupyter_notebooks/2_Jupyter_strings_and_lists.ipynb".
The first task is:
Suppose we want to replace the period ('.') by an exaclamation mark ('!'). Can we just modify this string element?
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/2_Jupyter_strings_and_lists.ipynb".
The first task is:
That is the last l in the string hello world. Python is so clever, it can count backwards!
Slicing strings
Sometimes, we want to grab more than one single element: we may want a section of the string. We do it using slicing notation in the square brackets. For example, we can use  [start:end], where start is the index to begin the slice, and end is the (non-inclusive) index to finish the slice.  For example, to grab the word hello from our string, we do:
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/2_Jupyter_strings_and_lists.ipynb".
The first task is:
The count() method gives the number of ocurrences of a substring in a range. The arguments for the range are optional.
Syntax:
str.count(substring, start, end)
Here, start and end are integers that indicate the indices where to start and end the count. For example, if we want to know how many letters 'e' we have in the whole string, we can do:
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/2_Jupyter_strings_and_lists.ipynb".
The first task is:
The last string method we'll mention is split(): it returns a list of all the words in a string. We can also define a separator and split our string according to that separator, and optionally we can limit the number of splits to num.
Syntax:
str.split(separator, num)
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/2_Jupyter_strings_and_lists.ipynb".
The first task is:
The output is False because the word is not capitalized! Upper-case and lower-case letters are distinct characters.
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/2_Jupyter_strings_and_lists.ipynb".
The first task is:
What if we want to add a space that separates hello from world? We directly add the string ' ' in the middle of the two variables. A space is a character!
Can you write Python code for it?

my_string = str_1 + ' ' + str_2
print(my_string)


I am working on the file "00_Intro_Python_Jupyter_notebooks/2_Jupyter_strings_and_lists.ipynb".
The first task is:
You can skip the start index, if you want to slice from the beginning of the string, and you can skip the end of a slice, indicating you want to go all the way to the end of the string. For example, if we want to grab the word 'world' from my_string, we could do the following:
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/2_Jupyter_strings_and_lists.ipynb".
The first task is:
You might have noticed that in the cell above we have a line before the code that starts with the # sign. That line seems to be ignored by Python: do you know why?
It is a comment: whenever you want to comment your Python code, you put a # in front of the comment. For example:
Can you write Python code for it?

my_string[1] #this is how we access the second element of a string


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
2D arrays can be added, subtracted, and multiplied:
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Another useful one: numpy.arange() gives an array of evenly spaced values in a defined interval.
Syntax:
numpy.arange(start, stop, step)
where start by default is zero, stop is not inclusive, and the default
for step is one.  Play with it!
Can you write Python code for it?

numpy.arange(4)

numpy.arange(2, 6)

numpy.arange(2, 6, 2)

numpy.arange(2, 6, 0.5)


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Array addition is defined element-wise, like when adding two vectors (or matrices). Array multiplication is also element-wise:
Can you write Python code for it?

mult_array = x_array * z_array
print(mult_array)


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Below, from the first matrix in our a_3D array, we will grab the two middle elements (5,6):
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Element-wise sum of NumPy arrays
In this case, we generate arrays with random integers using the NumPy function numpy.random.randint(). The arrays we generate with this function are not going to be like the lists: in this case we'll have 100 elements in the range [0, 100) but they can repeat. Our goal is to compare the time it takes to compute addition of a list or an array of numbers, so all that matters is that the arrays and the lists are of the same length and type (integers).
Can you write Python code for it?

arr_1 = numpy.random.randint(0, 100, size=100)
arr_2 = numpy.random.randint(0, 100, size=100)

#print first 10 elements
print(arr_1[0:10])
print(arr_2[0:10])


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Exercises:
From the X array:

Grab the 2nd column.
Grab the 2nd row.

Let's practice with a 3D array.
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Exercises:
From the X array:

Grab the 2nd element in the 1st column.
Grab the 2nd element in the 2nd column.

Play with slicing on this array:
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Exercises:
From the array named a_3D:

Grab the two middle elements (17, 18) from the second matrix.
Grab the last row from both matrices.
Grab the elements of the 1st matrix that exclude the first row and the first column.
Grab the elements of the 2nd matrix that exclude the last row and the last column.

NumPy == Fast and Clean!
When we are working with numbers, arrays are a better option because the NumPy library has built-in functions that are optimized, and therefore faster than vanilla Python. Especially if we have big arrays. Besides, using NumPy arrays and exploiting their properties makes our code more readable.
For example, if we wanted to add element-wise the elements of 2 lists, we need to do it with a for statement. If we want to add two NumPy arrays, we just use the addtion '+' symbol!
Below, we will add two lists and two arrays (with random elements) and we'll compare the time it takes to compute each addition.
Element-wise sum of a Python list
Using the Python library random, we will generate two lists with 100 pseudo-random elements in the range [0,100), with no numbers repeated.
Can you write Python code for it?

#import random library
import random

lst_1 = random.sample(range(100), 100)
lst_2 = random.sample(range(100), 100)

#print first 10 elements
print(lst_1[0:10])
print(lst_2[0:10])


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
If we want to grab the first column of both matrices in our a_3D array, we do:
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Now that we have different arrays x_array, y_array and z_array, we can do more computations, like add or multiply them. For example:
Can you write Python code for it?

add_array = x_array + y_array
print(add_array)


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Now that we've saved it with a variable name, we can do some computations with the array. E.g., take the square of every element of the array, in one go:
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Now we can use the %%time cell magic, again, to see how long it takes NumPy to compute the element-wise sum.
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Or equivalently we can use numpy.dot():
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Play with NumPy Arrays
Welcome to Lesson 4 of the first course module in "Engineering Computations." You have come a long way!
Remember, this course assumes no coding experience, so the first three lessons were focused on creating a foundation with Python programming constructs using essentially no mathematics. The previous lessons are:

Lesson 1: Interacting with Python
Lesson 2: Play with data in Jupyter
Lesson 3: Strings and lists in action

In engineering applications, most computing situations benefit from using arrays: they are sequences of data all of the same type. They behave a lot like lists, except for the constraint in the type of their elements. There is a huge efficiency advantage when you know that all elements of a sequence are of the same type—so equivalent methods for arrays execute a lot faster than those for lists.
The Python language is expanded for special applications, like scientific computing, with libraries. The most important library in science and engineering is NumPy, providing the n-dimensional array data structure (a.k.a, ndarray) and a wealth of functions, operations and algorithms for efficient linear-algebra computations.
In this lesson, you'll start playing with NumPy arrays and discover their power. You'll also meet another widely loved library: Matplotlib, for creating two-dimensional plots of data.
Importing libraries
First, a word on importing libraries to expand your running Python session. Because libraries are large collections of code and are for special purposes, they are not loaded automatically when you launch Python (or IPython, or Jupyter). You have to import a library using the import command. For example, to import NumPy, with all its linear-algebra goodness, we enter:
python
import numpy
Once you execute that command in a code cell, you can call any NumPy function using the dot notation, prepending the library name. For example, some commonly used functions are:

numpy.linspace()
numpy.ones()
numpy.zeros()
numpy.empty()
numpy.copy()

Follow the links to...
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
The command %matplotlib inline is there to get our plots inside the notebook (instead of a pop-up window, which is the default behavior of pyplot).
We'll use the pyplot.plot() function, specifying the line color ('k' for black) and line style ('-', '--' and ':' for continuous, dashed and dotted line), and giving each line a label. Note that the values for color, linestyle and label are given in quotes.
Can you write Python code for it?

#Plot x^2
pyplot.plot(xarray, pow2, color='k', linestyle='-', label='square')
#Plot x^3
pyplot.plot(xarray, pow3, color='k', linestyle='--', label='cube')
#Plot sqrt(x)
pyplot.plot(xarray, pow_half, color='k', linestyle=':', label='square root')
#Plot the legends in the best location
pyplot.legend(loc='best')


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
The line above is telling NumPy that we want:

first ':' : from the first dimension, grab all the elements (2 matrices).
second ':': from the second dimension, grab all the elements (all the rows).
'0'       : from the third dimension, grab the first element (first column).

If we want the first 2 elements of the first column of both matrices:
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
The multiplication using the '*' operator is element-wise. If we want to do matrix multiplication we use the '@' operator:
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
To illustrate other features, we will plot the same data, but varying the colors instead of the line style. We'll also use LaTeX syntax to write formulas in the labels. If you want to know more about LaTeX syntax, there is a quick guide to LaTeX available online.
Adding a semicolon (';') to the last line in the plotting code block prevents that ugly output, like <matplotlib.legend.Legend at 0x7f8c83cc7898>. Try it.
Can you write Python code for it?

#Plot x^2
pyplot.plot(xarray, pow2, color='red', linestyle='-', label='$x^2$')
#Plot x^3
pyplot.plot(xarray, pow3, color='green', linestyle='-', label='$x^3$')
#Plot sqrt(x)
pyplot.plot(xarray, pow_half, color='blue', linestyle='-', label='$\sqrt{x}$')
#Plot the legends in the best location
pyplot.legend(loc='best');


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
To plot the resulting arrays as a function of the orginal one (xarray) in the x-axis, we need to import the module pyplot from Matplotlib.
Can you write Python code for it?

from matplotlib import pyplot
%matplotlib inline


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
Visualizing the dimensions of the a_3D array can be tricky, so here is a diagram that will help you to understand how the dimensions are assigned: each dimension is shown as  a coordinate axis. For a 3D array, on the "x axis", we have the sub-arrays that themselves are two-dimensional (matrices). We have two of these 2D sub-arrays, in this case; each one has 3 rows and 4 columns. Study this sketch carefully, while comparing with how the array a_3D is printed out above.
<img src="images/3d_array_sketch.png" style="width: 400px;"/>
When we have multidimensional arrays, we can access slices of their elements by slicing on each dimension. This is one of the advantages of using arrays: we cannot do this with lists.
Let's access some elements of our 2D array called X.
Can you write Python code for it?

X

# Grab the element in the 1st row and 1st column
X[0, 0]

# Grab the element in the 1st row and 2nd column
X[0, 1]


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
We can also divide arrays, but you have to be careful not to divide by zero. This operation will result in a nan which stands for Not a Number. Python will still perform the division, but will tell us about the problem.
Let's see how this might look:
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
We can also take the square root of a positive array, using the numpy.sqrt() function:
Can you write Python code for it?

z_array = numpy.sqrt(y_array)
print(z_array)


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
We can check for the shape of a NumPy array using the function numpy.shape():
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
We need to write a for statement, appending the result of the element-wise sum into a new list we call result_lst.
For timing, we can use the IPython "magic" %%time. Writing at the beginning of the code cell the command %%time will give us the time it takes to execute all the code in that cell.
Can you write Python code for it?

%%time
res_lst = []
for i in range(100):
    res_lst.append(lst_1[i] + lst_2[i])

print(res_lst[0:10])


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
What if we try to multiply arrays using the '*'operator?
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
When we don't specify the start and/or end point in the slicing, the symbol ':' means "all". In the example above, we are telling NumPy that we want all the elements from the 0-th index in the second dimension (the first column).
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/4_NumPy_Arrays_and_Plotting.ipynb".
The first task is:
numpy.linspace() is similar to numpy.arange(), but uses number of samples instead of a step size. It returns an array with evenly spaced numbers over the specified interval.
Syntax:
numpy.linspace(start, stop, num)
stop is included by default (it can be removed, read the docs), and num by default is 50.
Can you write Python code for it?

numpy.linspace(2.0, 3.0)

len(numpy.linspace(2.0, 3.0))

numpy.linspace(2.0, 3.0, 6)

numpy.linspace(-1, 1, 9)


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
"Split regression"
If you look at the plot above, you might notice that around 1970 the temperature starts increasing faster that the previous trend. So maybe one single straight line does not give us a good-enough fit.
What if we break the data in two (before and after 1970) and do a linear regression in each segment?
To do that, we first need to find the position in our year array where the year 1970 is located. Thankfully, NumPy has a function called  numpy.where() that can help us. We pass a condition and numpy.where() tells us where in the array the condition is True.
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
Exercise
Write a function that computes the coefficients, call the function to compute them and compare the result with the values we obtained before. As a hint, we give you the structure that you should follow:
```python
def coefficients(x, y, x_mean, y_mean):
    """
    Write docstrings here
    """
a_1 =
a_0 =

return a_1, a_0

```
We now have the coefficients of a linear function that best fits our data. With them, we can compute the predicted values of temperature anomaly, according to our fit. Check again the equations above: the values we are going to compute are $f(x_i)$.
Let's call reg the array obtined from evaluating $f(x_i)$ for all years.
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
How would we go about understanding the trends from the data on global temperature?
The first step in analyzing unknown data is to generate some simple plots using Matplotlib. We are going to look at the temperature-anomaly history, contained in a file, and make our first plot to explore this data.
We are going to smooth the data and then we'll fit a line to it to find a trend, plotting along the way to see how it all looks.
Let's get started!
Step 1: Read a data file
We took the data from the NOAA (National Oceanic and Atmospheric Administration) webpage. Feel free to play around with the webpage and analyze data on your own, but for now, let's make sure we're working with the same dataset.
We have a file named land_global_temperature_anomaly-1880-2016.csv in our data folder. This file contains the year on the first column, and averages of land temperature anomaly listed sequentially on the second column, from the year 1880 to 2016. We will load the file, then make an initial plot to see what it looks like.
Note:
If you downloaded this notebook alone, rather than the full collection for this course, you may not have the data file on the location we assume below. In that case, you can download the data if you add a code cell, and execute the following code in it:
Python
from urllib.request import urlretrieve
URL = 'http://go.gwu.edu/engcomp1data5?accessType=DOWNLOAD'
urlretrieve(URL, 'land_global_temperature_anomaly-1880-2016.csv')
The data file will be downloaded to your working directory, and you will then need to remove the path information, i.e., the string '../../data/', from the definition of the variable fname below.
Let's start by importing NumPy.
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
Let's define a custom function that calculates the mean value of any array. Study the code below carefully.
Can you write Python code for it?

def mean_value(array):
    """ Calculate the mean value of an array

    Arguments
    ---------
    array: Numpy array

    Returns
    -------
    mean: mean value of the array
    """
    sum_elem = 0
    for element in array:
            sum_elem += element # this is the same as sum_elem = sum_elem + element

    mean = sum_elem / len(array)

    return mean


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
Linear regression with real data
Earth temperature over time
In this lesson, we will apply all that we've learned (and more) to analyze real data of Earth temperature over time.
Is global temperature rising? How much? This is a question of burning importance in today's world!
Data about global temperatures are available from several sources: NASA, the National Climatic Data Center (NCDC) and the University of East Anglia in the UK. Check out the University Corporation for Atmospheric Research (UCAR) for an in-depth discussion.
The NASA Goddard Space Flight Center is one of our sources of global climate data. They produced the video below showing a color map of the changing global surface temperature anomalies from 1880 to 2015.
The term global temperature anomaly means the difference in temperature with respect to a reference value or a long-term average. It is a very useful way of looking at the problem and in many ways better than absolute temperature. For example, a winter month may be colder than average in Washington DC, and also in Miami, but the absolute temperatures will be different in both places.
Can you write Python code for it?

from IPython.display import YouTubeVideo
YouTubeVideo('gGOzHVUQCw0')


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
Neat! You learned how to write a Python function, and we wrote one for computing the mean value of an array of numbers. We didn't have to, though, because NumPy has a built-in function to do just what we needed: numpy.mean().
Exercise
Calculate the mean of the year and temp_anomaly arrays using the NumPy built-in function, and compare the results with the ones obtained using our custom mean_value function.
Now that we have mean values, we can compute our coefficients by following equations (12). We first calculate $a_1$ and then use that value to calculate $a_0$.
Our coefficients are:
$$
    a_1 = \frac{ \sum_{i=0}^{n} y_{i} (x_i - \bar{x})}{\sum_{i=0}^{n} x_i (x_i - \bar{x})} \quad , \quad a_0  = \bar{y} - a_1\bar{x}
$$
We already calculated the mean values of the data arrays, but the formula requires two sums over new derived arrays. Guess what, NumPy has a built-in function for that: numpy.sum(). Study the code below.
Can you write Python code for it?

a_1 = numpy.sum(temp_anomaly*(year - year_mean)) / numpy.sum(year*(year - year_mean))

print(a_1)

a_0 = temp_anomaly_mean - a_1*year_mean

print(a_0)


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
Now we have a line plot, but if you see this plot without any information you would not be able to figure out what kind of data it is! We need labels on the axes, a title and why not a better color, font and size of the ticks.
Publication quality plots should always be your standard for plotting.
How you present your data will allow others (and probably you in the future) to better understand your work.
We can customize the style of our plots using Matplotlib's rcParams. It lets us set some style options that apply for all the plots we create in the current session.
Here, we'll make the font of a specific size and type. You can also customize other parameters like line width, color, and so on (check out the documentation).
Can you write Python code for it?

from matplotlib import rcParams
rcParams['font.family'] = 'serif'
rcParams['font.size'] = 16


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
Now we know how to split our data in two sets, to get two regression lines. We need two slices of the arrays year and temp_anomaly, which we'll save in new variable names below. After that, we complete two linear fits using the helpful NumPy functions we learned above.
Can you write Python code for it?

year_1 , temp_anomaly_1 = year[0:90], temp_anomaly[0:90]
year_2 , temp_anomaly_2 = year[90:], temp_anomaly[90:]

m1, b1 = numpy.polyfit(year_1, temp_anomaly_1, 1)
m2, b2 = numpy.polyfit(year_2, temp_anomaly_2, 1)

f_linear_1 = numpy.poly1d((m1, b1))
f_linear_2 = numpy.poly1d((m2, b2))

pyplot.figure(figsize=(10, 5))

pyplot.plot(year, temp_anomaly, color='#2929a3', linestyle='-', linewidth=1, alpha=0.5)
pyplot.plot(year_1, f_linear_1(year_1), 'g--', linewidth=2, label='1880-1969')
pyplot.plot(year_2, f_linear_2(year_2), 'r--', linewidth=2, label='1970-2016')

pyplot.xlabel('Year')
pyplot.ylabel('Land temperature anomaly [°C]')
pyplot.legend(loc='best', fontsize=15)
pyplot.grid();


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
Once you execute the cell above, the functionmean_value() becomes available to use on any argument of the correct type. This function works on arrays of any length. We can try it now with our data.
Can you write Python code for it?

year_mean = mean_value(year)
print(year_mean)

temp_anomaly_mean = mean_value(temp_anomaly)
print(temp_anomaly_mean)


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
The plot() function of the pyplot module makes simple line plots. We avoid that stuff that appeared on top of the figure, that Out[x]: [< ...>] ugliness, by adding a semicolon at the end of the plotting command.
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
To load our data from the file, we'll use the function numpy.loadtxt(), which lets us immediately save the data into NumPy arrays. (We encourage you to read the documentation for details on how the function works.) Here, we'll save the data into the arrays year and temp_anomaly.
Can you write Python code for it?

fname = 'data/land_global_temperature_anomaly-1880-2016.csv'

year, temp_anomaly = numpy.loadtxt(fname, delimiter=',', skiprows=5, unpack=True)


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
To split the data, we use the powerful instrument of slicing with the colon notation. Remember that a colon between two indices indicates a range of values from a start to an end. The rule is that [start:end] includes the element at index start but excludes the one at index end. For example, to grab the first 3 years in our year array, we do:
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
Using the help function instead:
Can you write Python code for it?


I am working on the file "00_Intro_Python_Jupyter_notebooks/5_Linear_Regression_with_Real_Data.ipynb".
The first task is:
We'll redo the same plot, but now we'll add a few things to make it prettier and publication quality. We'll add a title, label the axes and, show a background grid. Study the commands below and looks at the result!
Can you write Python code for it?

#You can set the size of the figure by doing:
pyplot.figure(figsize=(10,5))

#Plotting
pyplot.plot(year, temp_anomaly, color='#2929a3', linestyle='-', linewidth=1)
pyplot.title('Land global temperature anomalies. \n')
pyplot.xlabel('Year')
pyplot.ylabel('Land temperature anomaly [°C]')
pyplot.grid();


I am working on the file "00_Introduction_to_Jupyter_and_Python.ipynb".
The first task is:
Accessing entries:
Can you write Python code for it?

raclette = menu.get('Raclette')
print(raclette)

#accessing a non-existing key
burger = menu.get('Burger')
print(burger)

#listing keys
menu.keys()

#listing values
menu.values()


I am working on the file "00_Introduction_to_Jupyter_and_Python.ipynb".
The first task is:
Adding entries:
Can you write Python code for it?

menu = {} # Empty dictionary
menu['Raclette'] = 14.50 # Adding new key-value pair
print(menu['Raclette'])

menu['Cheese Fondue'] = 10.50
menu['Muesli'] = 13.50
menu['Quiche'] = 19.50
menu['Cervela'] = 17.50 # Your code here: Add some dish-price pairs to menu!

print("There are " + str(len(menu)) + " items on the menu.")
print(menu)


I am working on the file "00_Introduction_to_Jupyter_and_Python.ipynb".
The first task is:
Looping over the elements of a list:
Can you write Python code for it?

my_list = [1,9,3,8,5,7]

for number in my_list:
    print(2*number)
      # Your code here


I am working on the file "00_Introduction_to_Jupyter_and_Python.ipynb".
The first task is:
Slicing:
Can you write Python code for it?

suitcase = ["sunglasses", "hat", "passport", "laptop", "suit", "shoes"]

first  = suitcase[0:2]  # The first and second items (index zero and one)
middle = suitcase[2:4]   # Third and fourth items (index two and three)
last   = suitcase[4:6]  # The last two items (index four and five)

print(last)


I am working on the file "00_Introduction_to_Jupyter_and_Python.ipynb".
The first task is:
This is an equation formatted in LaTeX $y = \sin(x)$
double-click this cell to edit it
2. Jupyter Notebook App
The Jupyter Notebook App is a server-client application that allows editing and running notebook documents via a web browser.
3. kernel
A notebook kernel is a “computational engine” that executes the code contained in a Notebook document. The ipython kernel executes python code. Kernels for many other languages exist (official kernels).
4. Notebook Dashboard
The Notebook Dashboard is the component which is shown first when you launch Jupyter Notebook App. The Notebook Dashboard is mainly used to open notebook documents, and to manage the running kernels (visualize and shutdown).
5. References

Jupyter project
Jupyter documentation
Jupyter notebooks documentation

(Very) Short Python Introduction
In addition to "standard" data types such as integers, floats and strings, Python knows a number of compound data types, used to group together other values. We will briefly see * Lists and Dictionaries*
1. Lists
https://docs.python.org/3/tutorial/introduction.html#lists
Accessing by index:
Can you write Python code for it?

# define a new list
zoo_animals = ["pangolin", "cassowary", "sloth", "tiger"]

if len(zoo_animals) > 3:
    print("The first animal at the zoo is the " + zoo_animals[0])
    print("The second animal at the zoo is the " + zoo_animals[1])
    print("The third animal at the zoo is the " + zoo_animals[2])
    print("The fourth animal at the zoo is the " + zoo_animals[3])


I am working on the file "00_Miscellaneous/model_evaluation_pipeline/Tutorial - TF Model Evaluation Pipeline with Dataflow and BigQuery.ipynb".
The first task is:
3. Training on Cloud ML Engine
Can you write Python code for it?

%%bash

if ! gsutil ls | grep -q gs://${BUCKET}/; then
  gsutil mb -l ${REGION} gs://${BUCKET}
  # copy canonical set of preprocessed files if you didn't do previous notebook
  gsutil -m cp -R gs://cloud-training-demos/babyweight gs://${BUCKET}
fi

%%bash

OUTDIR=gs://${BUCKET}/babyweight/trained_model
JOBNAME=babyweight_$(date -u +%y%m%d_%H%M%S)
echo $OUTDIR $REGION $JOBNAME

gsutil -m rm -rf $OUTDIR

gcloud ml-engine jobs submit training $JOBNAME \
  --region=$REGION \
  --module-name=trainer.task \
  --package-path=$(pwd)/trainer \
  --job-dir=$OUTDIR \
  --staging-bucket=gs://$BUCKET \
  --scale-tier=STANDARD_1 \
  --runtime-version=1.4 \
  -- \
  --bucket=${BUCKET} \
  --output_dir=${OUTDIR} \
  --train_examples=200000


I am working on the file "00_Miscellaneous/model_evaluation_pipeline/Tutorial - TF Model Evaluation Pipeline with Dataflow and BigQuery.ipynb".
The first task is:
4. Testing an evaluation pipeline with DirectRunner
Configure a DirectRunner job
Can you write Python code for it?

import os
import datetime

os.environ['RUNNER'] = 'DirectRunner'
os.environ['JOB_NAME'] = 'evaluate-ml-model-{0}'.format(datetime.datetime.now().strftime('%y%m%d-%H%M%S'))
os.environ['BQ_DATASET'] = 'model_evaluation'
os.environ['BQ_TABLE'] = 'sample'
os.environ['OUTPUT_TABLE'] = '{}:{}.{}'.format(os.environ['PROJECT'], os.environ['BQ_DATASET'], os.environ['BQ_TABLE'])

%%bash

# Create BQ dataset if it doesn't exist.
if ! bq ls ${BQ_DATASET} > /dev/null; then
  bq mk ${BQ_DATASET}
fi


I am working on the file "00_Miscellaneous/model_evaluation_pipeline/Tutorial - TF Model Evaluation Pipeline with Dataflow and BigQuery.ipynb".
The first task is:
Execute a Dataflow job
Can you write Python code for it?

%%bash

cd $(pwd)/evaluator

python -m main \
  --datasize=1000000 \
  --year_from=1984 \
  --year_to=1988 \
  --saved_model_dir=${SAVED_MODEL_DIR} \
  --output_table=${OUTPUT_TABLE} \
  --project=${PROJECT} \
  --runner=${RUNNER} \
  --region=${REGION} \
  --staging_location=${STAGING_LOCATION} \
  --temp_location=${TEMP_LOCATION} \
  --job_name=${JOB_NAME} \
  --setup_file=$(pwd)/setup.py

import pandas
from pandas.io import gbq

query = """
SELECT * FROM `{0}` LIMIT 10
""".format(os.environ['OUTPUT_TABLE'].replace(':', '.'))

df = gbq.read_gbq(query=query, dialect='standard', project_id=os.environ['PROJECT'], verbose=False)
df.head()

query = """
SELECT
  model,
  testdata,
  SQRT(SUM(POW(weight_residual,2))/COUNT(*)) AS RMSE,
  AVG(time_inference) AS AVG_MSEC
FROM
  `{0}`
GROUP BY
  model, testdata
""".format(os.environ['OUTPUT_TABLE'].replace(':', '.'))

df = gbq.read_gbq(query=query, dialect='standard', project_id=os.environ['PROJECT'], verbose=False)
df


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
1. Upload the model artefacts to Google Cloud Storage bucket
Can you write Python code for it?

%%bash

gsutil -m rm -r gs://${BUCKET}/tf-model-optimisation

%%bash

saved_models_base=models/mnist/keras_classifier/export/
saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)

echo ${saved_model_dir}

gsutil -m cp -r ${saved_model_dir} gs://${BUCKET}/tf-model-optimisation/original

%%bash

saved_models_base=models/mnist/keras_classifier/export/
saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)/optimised

echo ${saved_model_dir}

gsutil -m cp -r ${saved_model_dir} gs://${BUCKET}/tf-model-optimisation


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
1.2 Estimator
1.2.1 Keras Model Function
Can you write Python code for it?

def keras_model_fn(params):

    inputs = tf.keras.layers.Input(shape=(28, 28), name='input_image')
    input_layer = tf.keras.layers.Reshape(target_shape=(28, 28, 1), name='reshape')(inputs)

    # convolutional layers
    conv_inputs = input_layer
    for i in range(params.num_conv_layers):

        filters = params.init_filters * (2**i)
        conv = tf.keras.layers.Conv2D(kernel_size=3, filters=filters, strides=1, padding='SAME', activation='relu')(conv_inputs)
        max_pool = tf.keras.layers.MaxPool2D(pool_size=2, strides=2, padding='SAME')(conv)
        batch_norm = tf.keras.layers.BatchNormalization()(max_pool)
        conv_inputs = batch_norm

    flatten = tf.keras.layers.Flatten(name='flatten')(conv_inputs)

    # fully-connected layers
    dense_inputs = flatten
    for i in range(len(params.hidden_units)):

        dense = tf.keras.layers.Dense(units=params.hidden_units[i], activation='relu')(dense_inputs)
        dropout = tf.keras.layers.Dropout(params.dropout)(dense)
        dense_inputs = dropout

    # softmax classifier
    logits = tf.keras.layers.Dense(units=NUM_CLASSES, name='logits')(dense_inputs)
    softmax = tf.keras.layers.Activation('softmax', name='softmax')(logits)

    # keras model
    model = tf.keras.models.Model(inputs, softmax)
    return model


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
1.2.2 Convert Keras model to Estimator
Can you write Python code for it?

def create_estimator(params, run_config):

    keras_model = keras_model_fn(params)
    print keras_model.summary()

    optimizer = tf.keras.optimizers.Adam(lr=params.learning_rate)
    keras_model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
    mnist_classifier = tf.keras.estimator.model_to_estimator(
        keras_model=keras_model,
        config=run_config
    )

    return mnist_classifier


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
10. Prediction with the Optimised SavedModel
Can you write Python code for it?

optimized_saved_model_dir = os.path.join(saved_model_dir,'optimised')
print(optimized_saved_model_dir)
inference_test(saved_model_dir=optimized_saved_model_dir, signature='serving_default', input_name='serving_input_image')


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
2. Deploy models to Cloud ML Engine
Don't forget to delete the model and the model version if they were previously deployed!
Can you write Python code for it?

%%bash

echo ${MODEL_NAME}

gcloud ml-engine models create ${MODEL_NAME} --regions=${REGION}


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
2. Inspect the Exported SavedModel
Can you write Python code for it?

%%bash

saved_models_base=models/mnist/keras_classifier/export/
saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)
echo ${saved_model_dir}
ls ${saved_model_dir}
saved_model_cli show --dir=${saved_model_dir} --all


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
3. Cloud ML Engine online predictions
Can you write Python code for it?

from googleapiclient import discovery
from oauth2client.client import GoogleCredentials

credentials = GoogleCredentials.get_application_default()
api = discovery.build(
    'ml', 'v1',
    credentials=credentials,
    discoveryServiceUrl='https://storage.googleapis.com/cloud-ml/discovery/ml_v1_discovery.json'
)

def predict(version, instances):

    request_data = {'instances': instances}

    model_url = 'projects/{}/models/{}/versions/{}'.format(PROJECT, MODEL_NAME, version)
    response = api.projects().predict(body=request_data, name=model_url).execute()

    class_ids = None

    try:
        class_ids = [item["class_ids"] for item in response["predictions"]]
    except:
        print response

    return class_ids

def inference_cmle(version, batch=100, repeat=10):

    instances = [
            {'input_image_3': [float(i) for i in list(eval_data[img])] }
        for img in range(batch)
    ]

    #warmup request
    predict(version, instances[0])
    print 'Warm up request performed!'
    print 'Timer started...'
    print ''

    time_start = datetime.utcnow()
    output = None

    for i in range(repeat):
        output = predict(version, instances)

    time_end = datetime.utcnow()

    time_elapsed_sec = (time_end - time_start).total_seconds()

    print "Inference elapsed time: {} seconds".format(time_elapsed_sec)
    print ""

    print "Prediction produced for {} instances batch, repeated {} times".format(len(output), repeat)
    print "Average latency per batch: {} seconds".format(time_elapsed_sec/repeat)
    print ""

    print "Prediction output for the last instance: {}".format(output[0])

version='v_org'
inference_cmle(version)

version='v_opt'
inference_cmle(version)


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
3. Test Prediction with SavedModel
Can you write Python code for it?

saved_model_dir = os.path.join(
    export_dir, [f for f in os.listdir(export_dir) if f.isdigit()][0])
print(saved_model_dir)
inference_test(saved_model_dir)


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
4. Describe the SavedModel Graph (before optimisation)
Load GraphDef from a SavedModel Directory
Can you write Python code for it?

def get_graph_def_from_saved_model(saved_model_dir):

    print saved_model_dir
    print ""

    from tensorflow.python.saved_model import tag_constants

    with tf.Session() as session:
        meta_graph_def = tf.saved_model.loader.load(
            session,
            tags=[tag_constants.SERVING],
            export_dir=saved_model_dir
        )

    return meta_graph_def.graph_def

describe_graph(get_graph_def_from_saved_model(saved_model_dir))


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
5. Freeze SavedModel
This function will convert the SavedModel into a GraphDef file (freezed_model.pb), and storing the variables as constrant to the freezed_model.pb
You need to define the graph output nodes for freezing. We are only interested in the output of softmax/Softmax node
Can you write Python code for it?

def freeze_graph(saved_model_dir):

    from tensorflow.python.tools import freeze_graph
    from tensorflow.python.saved_model import tag_constants

    output_graph_filename = os.path.join(saved_model_dir, "freezed_model.pb")
    output_node_names = "softmax/Softmax"
    initializer_nodes = ""

    freeze_graph.freeze_graph(
        input_saved_model_dir=saved_model_dir,
        output_graph=output_graph_filename,
        saved_model_tags = tag_constants.SERVING,
        output_node_names=output_node_names,
        initializer_nodes=initializer_nodes,

        input_graph=None,
        input_saver=False,
        input_binary=False,
        input_checkpoint=None,
        restore_op_name=None,
        filename_tensor_name=None,
        clear_devices=False,
        input_meta_graph=False,
    )

    print "SavedModel graph freezed!"

freeze_graph(saved_model_dir)

%%bash
saved_models_base=models/mnist/keras_classifier/export/
saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)
echo ${saved_model_dir}
ls ${saved_model_dir}


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
6. Describe the freezed_model.pb Graph (after freezing)
Load GraphDef from GraphDef File
Can you write Python code for it?

def get_graph_def_from_file(graph_filepath):

    print graph_filepath
    print ""

    from tensorflow.python import ops

    with ops.Graph().as_default():
        with tf.gfile.GFile(graph_filepath, "rb") as f:
            graph_def = tf.GraphDef()
            graph_def.ParseFromString(f.read())

            return graph_def

freezed_filepath=os.path.join(saved_model_dir,'freezed_model.pb')
describe_graph(get_graph_def_from_file(freezed_filepath))


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
8. Describe the Optimised Graph
Can you write Python code for it?


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
9. Convert Optimised graph (GraphDef) to SavedModel
Can you write Python code for it?

def convert_graph_def_to_saved_model(graph_filepath):

    from tensorflow.python import ops
    export_dir=os.path.join(saved_model_dir,'optimised')

    if tf.gfile.Exists(export_dir):
        tf.gfile.DeleteRecursively(export_dir)

    graph_def = get_graph_def_from_file(graph_filepath)

    with tf.Session(graph=tf.Graph()) as session:
        tf.import_graph_def(graph_def, name="")
        tf.saved_model.simple_save(session,
                export_dir,
                inputs={
                    node.name: session.graph.get_tensor_by_name("{}:0".format(node.name))
                    for node in graph_def.node if node.op=='Placeholder'},
                outputs={
                    "softmax": session.graph.get_tensor_by_name("softmax/Softmax:0"),
                }
            )

        print "Optimised graph converted to SavedModel!"

optimised_filepath=os.path.join(saved_model_dir,'optimised_model.pb')
convert_graph_def_to_saved_model(optimised_filepath)


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
Cloud ML Engine Deployment and Prediction
Can you write Python code for it?

PROJECT = 'ksalama-gcp-playground'
BUCKET = 'ksalama-gcs-cloudml'
REGION = 'europe-west1'
MODEL_NAME = 'mnist_classifier'

os.environ['BUCKET'] = BUCKET
os.environ['PROJECT'] = PROJECT
os.environ['REGION'] = REGION
os.environ['MODEL_NAME'] = MODEL_NAME


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
Describe GraphDef
Can you write Python code for it?

def describe_graph(graph_def, show_nodes=False):

    print 'Input Feature Nodes: {}'.format([node.name for node in graph_def.node if node.op=='Placeholder'])
    print ""
    print 'Unused Nodes: {}'.format([node.name for node in graph_def.node if 'unused'  in node.name])
    print ""
    print 'Output Nodes: {}'.format( [node.name for node in graph_def.node if 'softmax' in node.name])
    print ""
    print 'Quanitization Nodes: {}'.format( [node.name for node in graph_def.node if 'quant' in node.name])
    print ""
    print 'Constant Count: {}'.format( len([node for node in graph_def.node if node.op=='Const']))
    print ""
    print 'Variable Count: {}'.format( len([node for node in graph_def.node if 'Variable' in node.op]))
    print ""
    print 'Identity Count: {}'.format( len([node for node in graph_def.node if node.op=='Identity']))
    print ""
    print 'Total nodes: {}'.format( len(graph_def.node))
    print ''

    if show_nodes==True:
        for node in graph_def.node:
            print 'Op:{} - Name: {}'.format(node.op, node.name)


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
Get model size
Can you write Python code for it?

def get_size(model_dir):

    print model_dir
    print ""

    pb_size = os.path.getsize(os.path.join(model_dir,'saved_model.pb'))

    variables_size = 0
    if os.path.exists(os.path.join(model_dir,'variables/variables.data-00000-of-00001')):
        variables_size = os.path.getsize(os.path.join(model_dir,'variables/variables.data-00000-of-00001'))
        variables_size += os.path.getsize(os.path.join(model_dir,'variables/variables.index'))

    print "Model size: {} KB".format(round(pb_size/(1024.0),3))
    print "Variables size: {} KB".format(round( variables_size/(1024.0),3))
    print "Total Size: {} KB".format(round((pb_size + variables_size)/(1024.0),3))

get_size(saved_model_dir)


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
TensorFlow Graph
<img src="graph_keras.png" width="500"/>
1.3.3 Run Experiment
Can you write Python code for it?

if tf.gfile.Exists(model_dir):
    print("Removing previous artifacts...")
    tf.gfile.DeleteRecursively(model_dir)

os.makedirs(model_dir)

estimator = run_experiment(params, run_config)


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST with Keras.ipynb".
The first task is:
Version: v_org is the original SavedModel (before optimisation)
Can you write Python code for it?

%%bash

MODEL_VERSION='v_org'
MODEL_ORIGIN=gs://${BUCKET}/tf-model-optimisation/original

gcloud ml-engine versions create ${MODEL_VERSION}\
            --model=${MODEL_NAME} \
            --origin=${MODEL_ORIGIN} \
            --runtime-version=1.10


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
1. Train and Export a TensorFlow DNNClassifier
1.1 Import Data
Can you write Python code for it?

mnist = tf.contrib.learn.datasets.load_dataset("mnist")
train_data = mnist.train.images
train_labels = np.asarray(mnist.train.labels, dtype=np.int32)
eval_data = mnist.test.images
eval_labels = np.asarray(mnist.test.labels, dtype=np.int32)
NUM_CLASSES = 10

print "Train data shape: {}".format(train_data.shape)
print "Eval data shape: {}".format(eval_data.shape)


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
1.2 Estimator
1.2.1 Model Function
Can you write Python code for it?

def model_fn(features, labels, mode, params):

    is_training = True if mode == tf.estimator.ModeKeys.TRAIN else False

    # convolution layers
    def _cnn_layers(conv_inputs):

        for i in range(params.num_conv_layers):

            filters = params.init_filters * (2**i)

            conv = tf.keras.layers.Conv2D(kernel_size=3, filters=filters, strides=1, padding='SAME')(conv_inputs)
            pool = tf.keras.layers.MaxPool2D(pool_size=2, strides=2, padding='SAME')(conv)
            batch_norm = tf.keras.layers.BatchNormalization()(pool, training=is_training)
            conv_inputs = batch_norm

        outputs = conv_inputs
        return outputs

    # fully-connected layers
    def _fully_connected_layers(dense_inputs):

        for i in range(len(params.hidden_units)):
            dense = tf.keras.layers.Dense(params.hidden_units[i], activation='relu')(dense_inputs)
            dense_dropout = tf.keras.layers.Dropout(params.dropout)(dense, training=is_training)
            dense_inputs = dense_dropout

        outputs = dense_inputs
        return outputs

    # model body
    def _inference(features, mode, params):

        input_layer = tf.keras.layers.Reshape([28, 28, 1])(features["input_image"])
        convolutions = _cnn_layers(input_layer)
        flatten = tf.keras.layers.Flatten()(convolutions)
        fully_connected = _fully_connected_layers(flatten)

        # unused_layer
        unused_layers = tf.keras.layers.Dense(units=100, name='unused', activation=None)(flatten)

        logits = tf.keras.layers.Dense(units=NUM_CLASSES, name='logits', activation=None)(fully_connected)
        return logits

    # model head
    head = tf.contrib.estimator.multi_class_head(n_classes=NUM_CLASSES)

    return head.create_estimator_spec(
            features=features,
            mode=mode,
            logits=_inference(features, mode, params),
            labels=labels,
           ...


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
1.2.2 Create Custom Estimator
Can you write Python code for it?

def create_estimator(params, run_config):

    # evaluation metric_fn
    def _metric_fn(labels, predictions):

        metrics = {}
        pred_class = predictions['class_ids']
        metrics['micro_accuracy'] = tf.metrics.mean_per_class_accuracy(
            labels=labels, predictions=pred_class, num_classes=NUM_CLASSES
        )

        return metrics

    mnist_classifier = tf.estimator.Estimator(
        model_fn=model_fn, params=params, config=run_config)

    mnist_classifier = tf.contrib.estimator.add_metrics(
        estimator=mnist_classifier, metric_fn=_metric_fn)

    return mnist_classifier


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
2. Deploy models to Cloud ML Engine
Don't forget to delete the model and the model version if they were previously deployed!
Can you write Python code for it?


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
3. Cloud ML Engine online predictions
Can you write Python code for it?

from googleapiclient import discovery
from oauth2client.client import GoogleCredentials

credentials = GoogleCredentials.get_application_default()
api = discovery.build(
    'ml', 'v1',
    credentials=credentials,
    discoveryServiceUrl='https://storage.googleapis.com/cloud-ml/discovery/ml_v1_discovery.json'
)

def predict(version, instances):

    request_data = {'instances': instances}

    model_url = 'projects/{}/models/{}/versions/{}'.format(PROJECT, MODEL_NAME, version)
    response = api.projects().predict(body=request_data, name=model_url).execute()

    class_ids = None

    try:
        class_ids = [item["class_ids"] for item in response["predictions"]]
    except:
        print response

    return class_ids

def inference_cmle(version, batch=100, repeat=10):

    instances = [
            {'input_image': [float(i) for i in list(eval_data[img])] }
        for img in range(batch)
    ]

    #warmup request
    predict(version, instances[0])
    print 'Warm up request performed!'
    print 'Timer started...'
    print ''

    time_start = datetime.utcnow()
    output = None

    for i in range(repeat):
        output = predict(version, instances)

    time_end = datetime.utcnow()

    time_elapsed_sec = (time_end - time_start).total_seconds()

    print "Inference elapsed time: {} seconds".format(time_elapsed_sec)
    print ""

    print "Prediction produced for {} instances batch, repeated {} times".format(len(output), repeat)
    print "Average latency per batch: {} seconds".format(time_elapsed_sec/repeat)
    print ""

    print "Prediction output for the last instance: {}".format(output[0])

version='v_org'
inference_cmle(version)

version='v_opt'
inference_cmle(version)


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
3. Test Prediction with SavedModel
Can you write Python code for it?

saved_model_dir = os.path.join(export_dir, os.listdir(export_dir)[-1])
print(saved_model_dir)
inference_test(saved_model_dir)


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
4. Describe the SavedModel Graph (before optimisation)
Load GraphDef from a SavedModel Directory
Can you write Python code for it?


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
5. Freeze SavedModel
This function will convert the SavedModel into a GraphDef file (freezed_model.pb), and storing the variables as constrant to the freezed_model.pb
You need to define the graph output nodes for freezing. We are only interested in the class_id, which is produced by head/predictions/ExpandDims node
Can you write Python code for it?

def freeze_graph(saved_model_dir):

    from tensorflow.python.tools import freeze_graph
    from tensorflow.python.saved_model import tag_constants

    output_graph_filename = os.path.join(saved_model_dir, "freezed_model.pb")
    output_node_names = "head/predictions/ExpandDims"
    initializer_nodes = ""

    freeze_graph.freeze_graph(
        input_saved_model_dir=saved_model_dir,
        output_graph=output_graph_filename,
        saved_model_tags = tag_constants.SERVING,
        output_node_names=output_node_names,
        initializer_nodes=initializer_nodes,

        input_graph=None,
        input_saver=False,
        input_binary=False,
        input_checkpoint=None,
        restore_op_name=None,
        filename_tensor_name=None,
        clear_devices=False,
        input_meta_graph=False,
    )

    print "SavedModel graph freezed!"

freeze_graph(saved_model_dir)

%%bash
saved_models_base=models/mnist/cnn_classifier/export/
saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)
echo ${saved_model_dir}
ls ${saved_model_dir}


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
6. Describe the freezed_model.pb Graph (after freezing)
Load GraphDef from GraphDef File
Can you write Python code for it?


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
8. Describe the Optimised Graph
Can you write Python code for it?


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
Get model size
Can you write Python code for it?


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
Optimised SavedModel Size
Can you write Python code for it?

optimised_saved_model_dir = os.path.join(saved_model_dir,'optimised')
get_size(optimised_saved_model_dir)

%%bash

saved_models_base=models/mnist/cnn_classifier/export/
saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)/optimised
ls ${saved_model_dir}
saved_model_cli show --dir ${saved_model_dir} --all


I am working on the file "00_Miscellaneous/model_optimisation/Tutorial - TensorFlow Model Optimisation for Serving - MNIST.ipynb".
The first task is:
Version: v_opt is the optimised SavedModel (after optimisation)
Can you write Python code for it?

%%bash

MODEL_VERSION='v_opt'
MODEL_ORIGIN=gs://${BUCKET}/tf-model-optimisation/optimised

gcloud ml-engine versions create ${MODEL_VERSION}\
            --model=${MODEL_NAME} \
            --origin=${MODEL_ORIGIN} \
            --runtime-version=1.10


I am working on the file "00_Miscellaneous/text-similarity-analysis/bqml/classification_with_embeddings.ipynb".
The first task is:
Create Features
Can you write Python code for it?

def get_flattened_embeddings(input_column='content_embed', n=512):
  L = []
  for i in range(n):
    L.append("{0}[OFFSET({1})] AS {0}_{1}".format(input_column, i))
  return (',').join(L)

def get_one_hot_vectors(input_column='topics', categories=['acl','gas']):
  L = []
  for cat in categories:
    L.append(
      "CASE WHEN '{1}' in UNNEST(SPLIT({0})) THEN 1 ELSE 0 END as {1}".format(
        input_column, cat))
  return (',').join(L)

def get_number(input_column='topics', categories=['acl','gas']):
  L = []
  for cat in categories:
    L.append(
      "(CASE WHEN '{1}' in UNNEST(SPLIT({0})) THEN 1 ELSE 0 END)".format(
        input_column, cat))
  return ('+').join(L)

def get_topics_and_count(dataset_id, table_id):
  sql = """
  WITH topics_flatten AS
  (
  SELECT topic FROM
    (
    SELECT
      SPLIT(topics, ',') AS topic
    FROM
      `{0}.{1}`
    WHERE
      topics != ''
    ),
    unnest(topic) topic
  )

  SELECT
    REPLACE(topic, '-', '_') AS topic,
    count(*) AS count
  FROM
    topics_flatten
  GROUP BY
    topic
  """.format(dataset_id, table_id)
  return sql

df = client.query(get_topics_and_count(dataset_id, table_id)).to_dataframe()

df.hist(bins=100)

target_categories = list(df[df['count'] >= 100].topic)
target_categories[:2]

def create_dataset(
    dataset_id, table_id, categories=['acq'], embed='content_embed', n=512):
  sql = """
  CREATE OR REPLACE TABLE `{0}.input_data`
    AS
    (
      SELECT
        STRUCT( {2} ) AS label,
        STRUCT( {3} ) AS feature
      FROM
        `{0}.{1}`
      WHERE
        {4} > 0
    )
  """.format(
      dataset_id, table_id,
      get_one_hot_vectors(input_column='topics', categories=categories),
      get_flattened_embeddings(input_column=embed, n=n),
      get_number(input_column='topics', categories=categories))
  return sql

sql = create_dataset(
  dataset_id, table_id,
  categories=target_categories, embed='content_embed', n = 512)
client.query(sql)


I am working on the file "00_Miscellaneous/text-similarity-analysis/bqml/classification_with_embeddings.ipynb".
The first task is:
Create Models
Can you write Python code for it?

def create_model(dataset_id, target='acq'):
  sql = """
  CREATE OR REPLACE MODEL `{0}.model_{1}`
  OPTIONS (
      model_type='logistic_reg',
      input_label_cols=['label']) AS
  SELECT
      label.{1} as label,
      feature.*
  FROM
      `{0}.input_data`
  """.format(dataset_id, target)

  return sql

sql = create_model(dataset_id, target='acq')
client.query(sql)


I am working on the file "00_Miscellaneous/text-similarity-analysis/bqml/classification_with_embeddings.ipynb".
The first task is:
Multi-Class Logistic Regression
Can you write Python code for it?

for target_category in target_categories[:3]:
  print("Creating a model for {0}...".format(target_category))
  client.query(create_model(dataset_id, target=target_category))

d_cat_to_eval = {}

for target_category in target_categories[:3]:
  print("Evaluating a model for {0}...".format(target_category))
  sql = evaluate_model_roc(dataset_id, target=target_category)
  d_cat_to_eval[target_category] = client.query(sql).to_dataframe()

d_cat_to_eval['acq'].plot(x='false_positive_rate', y='recall', grid=True, title='roc curve')

d_cat_to_eval['bop'].plot(x='false_positive_rate', y='recall', grid=True, title='roc curve')

d_cat_to_eval['cpi'].plot(x='false_positive_rate', y='recall', grid=True, title='roc curve')


I am working on the file "00_Miscellaneous/text-similarity-analysis/bqml/classification_with_embeddings.ipynb".
The first task is:
Table Schema
Can you write Python code for it?

dataset_id = 'YOUR-DATASET-NAME'
table_id = 'YOUR-TABLE-NAME'

dataset_ref = client.dataset(dataset_id)
table_ref = dataset_ref.table(table_id)
table = client.get_table(table_ref)

# View table properties
for e in table.schema:
  print(e)


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - Optimising Learning Rate.ipynb".
The first task is:
3. Create estimator
Can you write Python code for it?

def create_estimator(params, run_config):

    feature_columns = create_feature_columns()

    estimator = tf.estimator.Estimator(
        model_fn,
        params=params,
        config=run_config
    )

    return estimator


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - Optimising Learning Rate.ipynb".
The first task is:
4. Data Input Function
Can you write Python code for it?

def make_input_fn(file_pattern, batch_size, num_epochs,
                  mode=tf.estimator.ModeKeys.EVAL):

    def _input_fn():
        dataset = tf.data.experimental.make_csv_dataset(
            file_pattern=file_pattern,
            batch_size=batch_size,
            column_names=HEADER,
            column_defaults=HEADER_DEFAULTS,
            label_name=TARGET_NAME,
            field_delim=',',
            use_quote_delim=True,
            header=False,
            num_epochs=num_epochs,
            shuffle=(mode==tf.estimator.ModeKeys.TRAIN)
        )

        iterator = dataset.make_one_shot_iterator()
        features, target = iterator.get_next()
        return features, target

    return _input_fn


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - Optimising Learning Rate.ipynb".
The first task is:
5. Experiment Definition
Can you write Python code for it?

def train_and_evaluate_experiment(params, run_config):

    # TrainSpec ####################################
    train_input_fn = make_input_fn(
        TRAIN_DATA_FILE,
        batch_size=params.batch_size,
        num_epochs=None,
        mode=tf.estimator.ModeKeys.TRAIN
    )

    train_spec = tf.estimator.TrainSpec(
        input_fn = train_input_fn,
        max_steps=params.traning_steps
    )
    ###############################################

    # EvalSpec ####################################
    eval_input_fn = make_input_fn(
        EVAL_DATA_FILE,
        num_epochs=1,
        batch_size=params.batch_size,
    )

    eval_spec = tf.estimator.EvalSpec(
        name=datetime.utcnow().strftime("%H%M%S"),
        input_fn = eval_input_fn,
        steps=None,
        start_delay_secs=0,
        throttle_secs=params.eval_throttle_secs
    )
    ###############################################

    tf.logging.set_verbosity(tf.logging.INFO)

    if tf.gfile.Exists(run_config.model_dir):
        print("Removing previous artefacts...")
        tf.gfile.DeleteRecursively(run_config.model_dir)

    print ''
    estimator = create_estimator(params, run_config)
    print ''

    time_start = datetime.utcnow()
    print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
    print(".......................................")

#     tf.estimator.train_and_evaluate(
#         estimator=estimator,
#         train_spec=train_spec,
#         eval_spec=eval_spec
#     )

    estimator.train(train_input_fn, steps=params.traning_steps)

    time_end = datetime.utcnow()
    print(".......................................")
    print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
    print("")
    time_elapsed = time_end - time_start
    print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - Optimising Learning Rate.ipynb".
The first task is:
6. Run Experiment with Parameters
Can you write Python code for it?

MODELS_LOCATION = 'models/census'
MODEL_NAME = 'dnn_classifier-01'
model_dir = os.path.join(MODELS_LOCATION, MODEL_NAME)

BATCH_SIZE = 64
NUM_EPOCHS = 10
steps_per_epoch = int(math.ceil((TRAIN_DATA_SIZE / BATCH_SIZE)))
training_steps = int(steps_per_epoch * NUM_EPOCHS)

print("Training data size: {}".format(TRAIN_DATA_SIZE))
print("Btach data size: {}".format(BATCH_SIZE))
print("Steps per epoch: {}".format(steps_per_epoch))
print("Traing epochs: {}".format(NUM_EPOCHS))
print("Training steps: {}".format(training_steps))

params  = tf.contrib.training.HParams(
    batch_size=BATCH_SIZE,
    traning_steps=training_steps,
    hidden_units=[64, 32],
    learning_rate=1.e-3,
    cycle_length=500,
    dropout_prob=0.1,
    eval_throttle_secs=0,
    lr_search=False
)

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    save_checkpoints_steps=steps_per_epoch,
    log_step_count_steps=100,
    save_summary_steps=1,
    keep_checkpoint_max=3,
    model_dir=model_dir,
)

train_and_evaluate_experiment(params, run_config)


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - Optimising Learning Rate.ipynb".
The first task is:
Dataset Metadata
Can you write Python code for it?

HEADER = ['age', 'workclass', 'fnlwgt', 'education', 'education_num',
               'marital_status', 'occupation', 'relationship', 'race', 'gender',
               'capital_gain', 'capital_loss', 'hours_per_week',
               'native_country', 'income_bracket']

HEADER_DEFAULTS = [[0], [''], [0], [''], [0], [''], [''], [''], [''], [''],
                       [0], [0], [0], [''], ['']]

NUMERIC_FEATURE_NAMES = ['age', 'education_num', 'capital_gain', 'capital_loss', 'hours_per_week']
CATEGORICAL_FEATURE_NAMES = ['gender', 'race', 'education', 'marital_status', 'relationship',
                             'workclass', 'occupation', 'native_country']

FEATURE_NAMES = NUMERIC_FEATURE_NAMES + CATEGORICAL_FEATURE_NAMES
TARGET_NAME = 'income_bracket'
TARGET_LABELS = [' <=50K', ' >50K']
WEIGHT_COLUMN_NAME = 'fnlwgt'
NUM_CLASSES = len(TARGET_LABELS)

def get_categorical_features_vocabolary():
    data = pd.read_csv(TRAIN_DATA_FILE, names=HEADER)
    return {
        column: list(data[column].unique())
        for column in data.columns if column in CATEGORICAL_FEATURE_NAMES
    }

feature_vocabolary = get_categorical_features_vocabolary()
print(feature_vocabolary)


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
As expected, the training steps, given 3 epochs (for training data of 32561 records and batch size of 200 records), is 32561, which correspods to:  ceiling of (TRAIN_DATA_SIZE / BATCH_SIZE) * num_epochs)
Note that, if both num_epochs (in the train_input_fn) and steps (in estimator.train) are supplied, the model will stop on the earlier criteria.
Can you write Python code for it?

train_input_fn = make_input_fn(
    TRAIN_DATA_FILE,
    batch_size=BATCH_SIZE,
    num_epochs=1000,
    shuffle=True,
)

train_experiment(
    training_steps=10, # the model will train for only 10 steps, ignoring the 1000 epochs
    clean_start=True,
    incremental=True,
    run_config=run_config
)

train_input_fn = make_input_fn(
    TRAIN_DATA_FILE,
    batch_size=BATCH_SIZE,
    num_epochs=1, # the model will train for only 1 epoch (55 steps), ignoring the 1000 steps
    shuffle=True,
)

train_experiment(
    training_steps=1000,
    clean_start=True,
    incremental=True,
    run_config=run_config
)


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
As shown, since the training (1000 iterasion) finished in less than 600 seconds (default value for **save_checkpoint_sec), only 2 checkpoints where saved: the initial one, and the final one.
Now let's set save_checkpoints_steps in the run_config to 200, so that in 1000 steps, you produce 5 checkpoints
Can you write Python code for it?

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    model_dir=model_dir,
    save_checkpoints_steps=200, ## so in 1000 steps, you produce 5 checkpoints
    save_checkpoints_secs=None
)

estimator=train_experiment(
    training_steps=1000,
    clean_start=True,
    incremental=True,
    run_config=run_config
)

%%bash

ls ${MODEL_DIR}


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
As shown, the total number of training steps is 2000, starting from step 1000 (from the previous run)
Train: Steps vs Epochs
While the steps refers to how many data batchs are needed for training, the epochs refers to how many times the whole training data needs to be used for training.
While using epochs to define the number of training iteration is a conventional practice in machine learninig, however, when working with very large datasets to train Deep Learning models, batch-level training steps (rather than the whole-training-data-level epochs) are more practical.
Can you write Python code for it?

num_epochs=3

train_input_fn = make_input_fn(
    TRAIN_DATA_FILE,
    batch_size=BATCH_SIZE,
    num_epochs=num_epochs,
    shuffle=True,
)

expected_training_steps = math.ceil(TRAIN_DATA_SIZE/float(BATCH_SIZE))*num_epochs

print('Training data size: {}'.format(TRAIN_DATA_SIZE))
print('Batch size: {}'.format(BATCH_SIZE) )
print('Number of epochs (supplied): {}'.format(num_epochs))
print('Number of training steps (expected): {}'.format(expected_training_steps))
print('')

train_experiment(
    training_steps=None,
    clean_start=True,
    incremental=True,
    run_config=run_config
)


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
Best exporter
This runs everytime when the new model is better than any exsiting model.
It uses the evaluation events stored under the eval folder.
You need to set the name of the subfolder in the EvalSpec, and set the event_file_pattern in the BestExporter to point to this folder and perform the evalution comparesions.
Can you write Python code for it?

eval_name=datetime.utcnow().strftime("%H%M%S")

exporter = tf.estimator.BestExporter(
    event_file_pattern='eval_{}/*.tfevents.*'.format(eval_name),
    name="estimate",
    serving_input_receiver_fn=make_serving_input_receiver_fn(),
    exports_to_keep=1
)

params = tf.contrib.training.HParams(
    batch_size=BATCH_SIZE,
    traning_steps=10000,
    eval_throttle_secs=0,
    exporter_type='best',
    clean_start=True,
    eval_name=eval_name
)

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    save_checkpoints_steps=500,
    model_dir=model_dir
)

train_evaluate_export_experiment(params, run_config, exporter)

%%bash

saved_models_base=${MODEL_DIR}/export/estimate/
echo 'exported model folders:'
ls ${saved_models_base}
echo ''

saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)
echo 'last exported model: '${saved_model_dir}
ls ${saved_model_dir}
saved_model_cli show --dir=${saved_model_dir} --all


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
Create Estimator
Note that, the purpose of the tutorial is not to create the state-of-the-art model for this dataset. The purpose is to show the mechanisms of training and evaluating a TensorFlow estimator, regardless of the sophistication of the model or its predictive power. Thus, we use a premade tf.estimator.DNNClassifier for our examples. The ideas discussed in this tutorial applies to more complex custom estimators.
Can you write Python code for it?

import math

def create_feature_columns():

    feature_columns = []

    for column in NUMERIC_FEATURE_NAMES:
        feature_column = tf.feature_column.numeric_column(column)
        feature_columns.append(feature_column)

    for column in CATEGORICAL_FEATURE_NAMES:
        vocabolary = feature_vocabolary[column]
        embed_size = int(math.sqrt(len(vocabolary)))
        feature_column = tf.feature_column.embedding_column(
            tf.feature_column.categorical_column_with_vocabulary_list(column, vocabolary),
            embed_size)
        feature_columns.append(feature_column)

    return feature_columns

def create_estimator(run_config):

    feature_columns = create_feature_columns()

    estimator = tf.estimator.DNNClassifier(
        feature_columns=feature_columns,
        n_classes=len(TARGET_LABELS),
        label_vocabulary=TARGET_LABELS,
        weight_column=WEIGHT_COLUMN_NAME,
        hidden_units=[100, 70, 50] ,
        dropout=0.2,
        batch_norm=True,
        config=run_config
    )

    return estimator

MODELS_LOCATION = 'models/census'
MODEL_NAME = 'dnn_classifier'
model_dir = os.path.join(MODELS_LOCATION, MODEL_NAME)

print(model_dir)

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    model_dir=model_dir
)


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
Download Data
Can you write Python code for it?

DATA_DIR='data'
!mkdir $DATA_DIR
!gsutil cp gs://cloud-samples-data/ml-engine/census/data/adult.data.csv $DATA_DIR
!gsutil cp gs://cloud-samples-data/ml-engine/census/data/adult.test.csv $DATA_DIR

TRAIN_DATA_FILE = os.path.join(DATA_DIR, 'adult.data.csv')
EVAL_DATA_FILE = os.path.join(DATA_DIR, 'adult.test.csv')

!wc -l $TRAIN_DATA_FILE
!wc -l $EVAL_DATA_FILE


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
Each checkpoint is labelled by the step number it was saved in.
Evaluate: Epochs vs Steps

batch_size is set (which can be bigger than batch size of training, of the batch fits in memory)
num_epochs is usually set to 1 (as you want to evaluate your model on the entire evaluation data once)
Can you write Python code for it?

eval_input_fn = make_input_fn(
    EVAL_DATA_FILE,
    batch_size=BATCH_SIZE,
    num_epochs=1,
    shuffle=False,
)


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
For evaluation, if you set epochs to be 1, you can ignore the steps param (set it you None).
By default, the latest checkpoint is evaluated
Can you write Python code for it?


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
In order to train the model for num_epochs, you need to do the following:
* the training data size needs to be known before training
* compute the training steps as: (TRAIN_DATA_SIZE / BATCH_SIZE) * num_epochs
* In TrainSpec, max_step to the computed value
* set num_epochs in the train_input_fn to None
Can you write Python code for it?

num_epochs = 5
computed_traning_steps = int(math.ceil((TRAIN_DATA_SIZE/float(BATCH_SIZE)))*num_epochs)

print('Training data size: {}'.format(TRAIN_DATA_SIZE))
print('Batch size: {}'.format(BATCH_SIZE))
print('Number of epochs (supplied): {}'.format(BATCH_SIZE))
print('Number of training steps (computed): {}'.format(computed_traning_steps))
print('')

params = tf.contrib.training.HParams(
    batch_size=BATCH_SIZE,
    traning_steps=computed_traning_steps,
    eval_throttle_secs=0,
    clean_start=True,
)

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    save_checkpoints_steps=200,
    model_dir=model_dir
)

train_and_evaluate_experiment(params, run_config)


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
Intertwining Training & Evalution

Use TrainSpec & EvalSpec with tf.estimator.train_and_evaluate()
In TrainSpec:
num_epochs in the train_input_fn is ignored. (Set it to None)
You need to set max_steps param, otherwise it will train forever

In EvalSpec (to evaluate the model using the whole evaluation data once):
num_epochs is set to 1 in the eval_input_fn
steps param is set to None

Evaluation occurs when a new checkpoint is saved
Checkpoints saving frequency is configures in run_config (using save_checkpoints_steps or save_checkpoints_secs)
You can set minimum amount of time between two evaluation, using throttle_secs in EvalSpec. For example, if throttle_secs is set to 60sec, this means that the following evaluation will only occure after 60sec from the previous evaluation, even if save_checkpoins_sec is set to 10.
If throttle_secs is set to 0, then evaluation will occure each time a checkpoint is saved, regardless the time difference between two consequtive checkpoints
Can you write Python code for it?

def train_and_evaluate_experiment(params, run_config):

    # TrainSpec ####################################
    train_input_fn = make_input_fn(
        TRAIN_DATA_FILE,
        batch_size=BATCH_SIZE,
        num_epochs=None,
        shuffle=True
    )

    train_spec = tf.estimator.TrainSpec(
        input_fn = train_input_fn,
        max_steps=params.traning_steps
    )
    ###############################################

    # EvalSpec ####################################
    eval_input_fn = make_input_fn(
        EVAL_DATA_FILE,
        batch_size=BATCH_SIZE,
        num_epochs=1,
        shuffle=False
    )

    eval_spec = tf.estimator.EvalSpec(
        name=datetime.utcnow().strftime("%H%M%S"),
        input_fn = eval_input_fn,
        steps=None,
        start_delay_secs=0,
        throttle_secs=params.eval_throttle_secs
    )

    ###############################################

    tf.logging.set_verbosity(tf.logging.INFO)

    if params.clean_start:
        if tf.gfile.Exists(run_config.model_dir):
            print("Removing previous artefacts...")
            tf.gfile.DeleteRecursively(run_config.model_dir)

    print("")
    estimator = create_estimator(run_config)
    print("")

    time_start = datetime.utcnow()
    print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
    print(".......................................")

    tf.estimator.train_and_evaluate(
        estimator=estimator,
        train_spec=train_spec,
        eval_spec=eval_spec
    )

    time_end = datetime.utcnow()
    print(".......................................")
    print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
    print("")
    time_elapsed = time_end - time_start
    print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))

    return estimator


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
Latest exporter
This exports a model after each evaluation.
You can specify the maximum number of exported models to keep using exports_to_keep param
Can you write Python code for it?

exporter = tf.estimator.LatestExporter(
            name="estimate",
            serving_input_receiver_fn=make_serving_input_receiver_fn(),
            exports_to_keep=3,
)

params = tf.contrib.training.HParams(
    batch_size=BATCH_SIZE,
    traning_steps=computed_traning_steps,
    eval_throttle_secs=0,
    clean_start=True,
    eval_name=datetime.utcnow().strftime("%H%M%S")
)

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    save_checkpoints_steps=200,
    model_dir=model_dir
)

train_evaluate_export_experiment(params, run_config, exporter)

%%bash

saved_models_base=${MODEL_DIR}/export/estimate/
echo 'exported model folders:'
ls ${saved_models_base}
echo ''

saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)
echo 'last exported model: '${saved_model_dir}
ls ${saved_model_dir}
saved_model_cli show --dir=${saved_model_dir} --all


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
Note that the checkpoint at the beginning of the training is removed, since keep_checkpoint_max is set to 5 (default value).
Train, Evaluate, and Export
Can you write Python code for it?

def train_evaluate_export_experiment(params, run_config, exporter):

    # TrainSpec ####################################
    train_input_fn = make_input_fn(
        TRAIN_DATA_FILE,
        batch_size=BATCH_SIZE,
        num_epochs=None,
        shuffle=True
    )

    train_spec = tf.estimator.TrainSpec(
        input_fn = train_input_fn,
        max_steps=params.traning_steps
    )
    ###############################################

    # EvalSpec ####################################
    eval_input_fn = make_input_fn(
        EVAL_DATA_FILE,
        batch_size=BATCH_SIZE,
        num_epochs=1,
        shuffle=False
    )

    eval_spec = tf.estimator.EvalSpec(
        name=params.eval_name,
        input_fn=eval_input_fn,
        exporters=[exporter],
        steps=None,
        start_delay_secs=0,
        throttle_secs=params.eval_throttle_secs
    )
    ###############################################

    tf.logging.set_verbosity(tf.logging.INFO)

    if params.clean_start:
        if tf.gfile.Exists(run_config.model_dir):
            print("Removing previous artefacts...")
            tf.gfile.DeleteRecursively(run_config.model_dir)

    print("")
    estimator = create_estimator(run_config)
    print("")

    time_start = datetime.utcnow()
    print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
    print(".......................................")

    tf.estimator.train_and_evaluate(
        estimator=estimator,
        train_spec=train_spec,
        eval_spec=eval_spec
    )

    time_end = datetime.utcnow()
    print(".......................................")
    print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
    print("")
    time_elapsed = time_end - time_start
    print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))

    return estimator


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
Note that, since we set save_checkpoint_steps to 200, and we have 815 steps, we get 5 checkpoints (and corresponding evaluations):

at step 200
at step 400
at step 600
at step 800
in the end (at step 815)
Can you write Python code for it?


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
The training data includes 32561 records, while the evaluation data includes 16278 records. We will fix the batch size to 200
Can you write Python code for it?

TRAIN_DATA_SIZE = 32561
EVAL_DATA_SIZE = 16278
BATCH_SIZE = 200

print("Training data size:{}".format(TRAIN_DATA_SIZE))
print("Number of batches in training data: {}".format(TRAIN_DATA_SIZE/float(BATCH_SIZE)))
print("Evaluation data size:{}".format(EVAL_DATA_SIZE))
print("Number of batches in evaluation data: {}".format(EVAL_DATA_SIZE/float(BATCH_SIZE)))


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
This is equivalent to setting epochs to None and setting the steps to the number of batches in the dataset
Export: Serving Input Receiver Function
Can you write Python code for it?

def make_serving_input_receiver_fn():

    inputs = {}
    for feature_name in FEATURE_NAMES:
        dtype = tf.float32 if feature_name in NUMERIC_FEATURE_NAMES else tf.string
        inputs[feature_name] = tf.placeholder(shape=[None], dtype=dtype)

    return tf.estimator.export.build_raw_serving_input_receiver_fn(inputs)

export_dir = os.path.join(model_dir, 'export')

if tf.gfile.Exists(export_dir):
    tf.gfile.DeleteRecursively(export_dir)

estimator.export_savedmodel(
    export_dir_base=export_dir,
    serving_input_receiver_fn=make_serving_input_receiver_fn()
)

%%bash

saved_models_base=${MODEL_DIR}/export/
saved_model_dir=${saved_models_base}$(ls ${saved_models_base} | tail -n 1)
echo ${saved_model_dir}
ls ${saved_model_dir}
saved_model_cli show --dir=${saved_model_dir} --all


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
Total number of steps 1000.
Lets run this again, with max_steps, without deleting the previous model.
Can you write Python code for it?

train_experiment(
    training_steps=1000,
    clean_start=False,
    incremental=False,
    run_config=run_config,
)


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
Train: Checkpoints
By default, a checkpoint is saved every 600 secs (10mins). This behaviour is configured in the run_config passed to the estimator, using only one of the following parameters:

save_checkpoints_secs: Save checkpoints every this many seconds.
save_checkpoints_steps: Save checkpoints every this many steps.

In addition, you can specify the number of the checkpoints to keep using keep_checkpoint_max  Defaults to 5 (that is, the 5 most recent checkpoint files are kept.)
The following code trains the model for 1000 steps...
Can you write Python code for it?

os.environ['MODEL_DIR'] = model_dir

train_input_fn = make_input_fn(
    TRAIN_DATA_FILE,
    batch_size=BATCH_SIZE,
    num_epochs=None,
    shuffle=True,
)

train_experiment(
    training_steps=1000,
    clean_start=True,
    incremental=True,
    run_config=run_config # using the default checkpoints param values
)

%%bash

ls ${MODEL_DIR}


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow Estimator Train, Evaluate, Export, Explained.ipynb".
The first task is:
Train: Input Function

Batch size is set
Epochs is ignored (set to None)

Later we are going to see how to use epochs for training.
Can you write Python code for it?

train_input_fn = make_input_fn(
    TRAIN_DATA_FILE,
    batch_size=BATCH_SIZE,
    num_epochs=None,
    shuffle=True
)


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow from Estimators to Keras.ipynb".
The first task is:
2. Create model_fn

Use feature columns to create input_layer
Use tf.keras.layers to define the model architecutre and output
Use binary_classification_head for create EstimatorSpec
Can you write Python code for it?

def model_fn(features, labels, mode, params):

    is_training = True if mode == tf.estimator.ModeKeys.TRAIN else False

    # model body
    def _inference(features, mode, params):

        feature_columns = create_feature_columns()
        input_layer = tf.feature_column.input_layer(features=features, feature_columns=feature_columns)
        dense_inputs = input_layer
        for i in range(len(params.hidden_units)):
            dense = tf.keras.layers.Dense(params.hidden_units[i], activation='relu')(dense_inputs)
            dense_dropout = tf.keras.layers.Dropout(params.dropout_prob)(dense, training=is_training)
            dense_inputs = dense_dropout
        fully_connected = dense_inputs
        logits = tf.keras.layers.Dense(units=1, name='logits', activation=None)(fully_connected)
        return logits

    # model head
    head = tf.contrib.estimator.binary_classification_head(
        label_vocabulary=TARGET_LABELS,
        weight_column=WEIGHT_COLUMN_NAME
    )

    return head.create_estimator_spec(
        features=features,
        mode=mode,
        logits=_inference(features, mode, params),
        labels=labels,
        optimizer=tf.train.AdamOptimizer(params.learning_rate)
    )


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow from Estimators to Keras.ipynb".
The first task is:
3. Create estimator
Can you write Python code for it?

def create_estimator(params, run_config):

    feature_columns = create_feature_columns()

    estimator = tf.estimator.Estimator(
        model_fn,
        params=params,
        config=run_config
    )

    return estimator


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow from Estimators to Keras.ipynb".
The first task is:
4. Data Input Function
Can you write Python code for it?

def make_input_fn(file_pattern, batch_size, num_epochs,
                  mode=tf.estimator.ModeKeys.EVAL):

    def _input_fn():
        dataset = tf.data.experimental.make_csv_dataset(
            file_pattern=file_pattern,
            batch_size=batch_size,
            column_names=HEADER,
            column_defaults=HEADER_DEFAULTS,
            label_name=TARGET_NAME,
            field_delim=',',
            use_quote_delim=True,
            header=False,
            num_epochs=num_epochs,
            shuffle= (mode==tf.estimator.ModeKeys.TRAIN)
        )
        return dataset

    return _input_fn


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow from Estimators to Keras.ipynb".
The first task is:
4. Run experiment
When using out-of-memory dataset, that is, reading data chuck by chuck from file(s) and feeding it to the model (using the tf.data.Dataset APIs), you usually do not know the size of the dataset. Thus, beside the number of epochs required to train the model for, you need to specify how many step is considered as an epoch.
This is not required when use an in-memory (numpy array) dataset, since the size of the dataset is know to the model, hence how many steps here are in the epoch.
In our experiment, we know the size of our dataset, thus we compute the steps_per_epoch as: training data size /batch size
Can you write Python code for it?

train_data = make_keras_input_fn(
    TRAIN_DATA_FILE,
    batch_size=params.batch_size,
    mode=tf.estimator.ModeKeys.TRAIN
)()

valid_data = make_keras_input_fn(
    EVAL_DATA_FILE,
    batch_size=params.batch_size,
    mode=tf.estimator.ModeKeys.EVAL
)()

steps_per_epoch = int(math.ceil(TRAIN_DATA_SIZE/float(params.batch_size)))
model.fit(
    train_data,
    epochs=5,
    steps_per_epoch=steps_per_epoch,
    validation_data=valid_data,
    validation_steps=steps_per_epoch,
    callbacks=callbacks
)

!ls $model_dir/checkpoints


I am working on the file "00_Miscellaneous/tf_train_eval_export/Tutorial - TensorFlow from Estimators to Keras.ipynb".
The first task is:
6. Run Experiment with Parameters
Can you write Python code for it?

MODELS_LOCATION = 'models/census'
MODEL_NAME = 'dnn_classifier'
model_dir = os.path.join(MODELS_LOCATION, MODEL_NAME)

params  = tf.contrib.training.HParams(
    batch_size=200,
    traning_steps=1000,
    hidden_units=[100, 70, 50],
    learning_rate=0.01,
    dropout_prob=0.2,
    eval_throttle_secs=0,
)

strategy = None
num_gpus = len([device_name for device_name in tf.contrib.eager.list_devices()
                if '/device:GPU' in device_name])

if num_gpus > 1:
    strategy = tf.contrib.distribute.MirroredStrategy()
    params.batch_size = int(math.ceil(params.batch_size / num_gpus))

run_config = tf.estimator.RunConfig(
    tf_random_seed=SEED,
    save_checkpoints_steps=200,
    keep_checkpoint_max=3,
    model_dir=model_dir,
    train_distribute=strategy
)

train_and_evaluate_experiment(params, run_config)