I am working on the file "03-clustering.ipynb".
The first task is:
(b) Complete the following Python function.

Compute a background image using a median filtering for each colour channel with a square kernel of size 30.
Subtract each background channel to the original channel.
Normalise each channel using min-max normalisation.
Can you write Python code for it?

# Import morpho element
from skimage.morphology import square
# Import the median filtering
from skimage.filter.rank import median

# Function to pre process the images
def PreProcessing(rgb_image):
    output = np.zeros(np.shape(rgb_image))

    # Obtain the background image for each channel through median filtering
    background_im_r = ...
    background_im_g = ...
    background_im_b = ...

    # Remove the background to the original channels
    output[:, :, 0] = ...
    output[:, :, 1] = ...
    output[:, :, 2] = ...

    # Normalise the image
    output[:, :, 0] = normalise_im(...)
    output[:, :, 1] = normalise_im(...)
    output[:, :, 2] = normalise_im(...)

    return output

# Function to apply min-max normalisation
def normalise_im(im_2d):
    return ...


I am working on the file "03-clustering.ipynb".
The first task is:
(b) Plot the cluster centers and the data labelled by the k-means fitting.
Can you write Python code for it?

plt.plot(centers_k_means[0][0],centers_k_means[0][1],'ob', label='Centers of clusters 1')
plt.plot(centers_k_means[1][0],centers_k_means[1][1],'or', label='Centers of clusters 2')

plt.legend()
# Show the figure
plt.show()


I am working on the file "03-clustering.ipynb".
The first task is:
(c) Complete the following function to compute the misclassification rate.
Can you write Python code for it?

# Compute the misclassification rate
def compute_error_rate(k_means_labels, gt_labels):
    ### Use the function nonzero()
    return float(np.size(np.nonzero(np.squeeze(k_means_labels != gt_labels)))) / float(np.size(gt_labels)) * 100.


I am working on the file "03-clustering.ipynb".
The first task is:
(c) Plot in each data point to the most probable cluster to which it will belongs. Plot also the centroids.
Can you write Python code for it?


I am working on the file "03-clustering.ipynb".
The first task is:
(d) Extract the characteristic features from the pre-processed image.
Can you write Python code for it?

# Extraction of the data
### You can use np.reshape()
data = ...


I am working on the file "03-clustering.ipynb".
The first task is:
(d) What is the misclassification for the current fitting? Highlight inside a plot the element which have been misclasified.
Hint: think at swapping the label if the error rate is really high. The label affected is performed in an unsupervised manner.
Can you write Python code for it?

# Show the misclassification rate
print 'The error rate is {} %'.format(compute_error_rate(k_means_cluster.labels_, gt))

# Plot the misclassified samples
# Find the samples
idx_wellclass = np.ravel(np.nonzero(np.squeeze(k_means_cluster.labels_==gt)))
idx_misclass = np.ravel(np.nonzero(np.squeeze(k_means_cluster.labels_!=gt)))

# Maybe we have to swap the cluster
if (np.size(idx_misclass) > np.size(idx_wellclass)):
    tmp = idx_wellclass[:]
    idx_wellclass = idx_misclass[:]
    idx_misclass = tmp[:]
    del tmp

# Get the data
data_wellclass = data[idx_wellclass,:]
data_misclass = data[idx_misclass,:]

# Make the plot
fig = plt.figure()
# Find the indexes of the first cluster
legend_tptn = plt.plot(data_wellclass[:,0],data_wellclass[:,1],'+r')

legend_fpfn = plt.plot(data_misclass[:,0],data_misclass[:,1],'ob')
plt.legend([legend_tptn[0], legend_fpfn[0]], ["TP & TN", "FP & FN"])
# Show the figure
plt.show()


I am working on the file "03-clustering.ipynb".
The first task is:
(e) Run k-means with 10 iterations and k-means++ as initialisation of the cluster.
Can you write Python code for it?


I am working on the file "03-clustering.ipynb".
The first task is:
(g) Run fuzzy c-means.
Can you write Python code for it?


I am working on the file "03-clustering.ipynb".
The first task is:
(h) Plot the degree of membership for each cluster to depict the segmentation.
Can you write Python code for it?


I am working on the file "03-clustering.ipynb".
The first task is:
Assuming that the image can be clustered with four classes:

One cluster with artefacts at the edges of the image
One cluster with the optic nerve and other artefacts
One cluster with noise across the image
One cluster with the vessels
Can you write Python code for it?


I am working on the file "03-clustering.ipynb".
The first task is:
Clustering via fuzzy c-means
(a) Use fuzzy c-means clustering method to find the cluster centers for $c=2$. Check the following link for an example:
https://github.com/scikit-fuzzy/scikit-fuzzy/blob/master/skfuzzy/cluster/tests/test_cmeans.py
Can you write Python code for it?

# Define the number of clusters
c = 2
# Exponentiation parameter
m = 2.

# Run the fuzzy c-means - need to transpose the data
fuzz(np.transpose(data))


I am working on the file "03-clustering.ipynb".
The first task is:
Clustering via k-means
(a) Use k-means clustering method to find the cluster centers for $k=2$. To do so, you will:

Call the constructor KMeans(),
Use the function predict of the object build in order to apply the clustering,
Get the centers of each cluster,
Display these centers.
Can you write Python code for it?

# Define the number of clusters k
k = 2

# Define the parameters of k-means
### use init 'random' and only one try
k_means_cluster = KMeans(k, n_init=1,init='random') # ,n_jobs=4
# number of iterations
# Run k-means
### Use the function predict()
k_means_cluster.fit(data)
labels = k_means_cluster.predict(data)
# Get the centers of k-means
centers_k_means = k_means_cluster.cluster_centers_

print 'The centers found by k-means are \n {}'.format(centers_k_means)


I am working on the file "03-clustering.ipynb".
The first task is:
Import the library to perform the clustering with k-means and fuzzy c-means.
Can you write Python code for it?

# Import k-means clustering method from scikit-learn
from sklearn.cluster import KMeans
# Import fuzzy c-means from scikit-fuzzy
import skfuzzy as fuzz


I am working on the file "03-clustering.ipynb".
The first task is:
Retina segmentation using k-means and fuzzy c-means
Can you write Python code for it?

# Import scikit-image for input-output manipulation
from skimage import io
from skimage import img_as_float


I am working on the file "03-example_iris.ipynb".
The first task is:
Each value we are predicting is the response (also known as: target, outcome, label, dependent variable)
Classification is supervised learning in which the response is categorical
Regression is supervised learning in which the response is ordered and continuous

Requirements for working with data in scikit-learn

Features and response are separate objects
Features and response should be numeric
Features and response should be NumPy arrays
Features and response should have specific shapes
Can you write Python code for it?

# check the types of the features and response
print(type(iris.data))
print(type(iris.target))

# check the shape of the features (first dimension = number of observations, second dimensions = number of features)
print(iris.data.shape)

# check the shape of the response (single dimension matching the number of observations)
print(iris.target.shape)

# store feature matrix in "X"
X = iris.data

# store response vector in "y"
y = iris.target


I am working on the file "03-example_iris.ipynb".
The first task is:
Machine learning on the iris dataset

Framed as a supervised learning problem: Predict the species of an iris using the measurements
Famous dataset for machine learning because prediction is easy
Learn more about the iris dataset: UCI Machine Learning Repository

Loading the iris dataset into scikit-learn
Can you write Python code for it?

# import load_iris function from datasets module
from sklearn.datasets import load_iris

# save "bunch" object containing iris dataset and its attributes
iris = load_iris()
type(iris)

# print the iris data
print(iris.data)


I am working on the file "03-example_iris.ipynb".
The first task is:
Machine learning terminology

Each row is an observation (also known as: sample, example, instance, record)
Each column is a feature (also known as: predictor, attribute, independent variable, input, regressor, covariate)
Can you write Python code for it?

# print the names of the four features
print(iris.feature_names)

# print integers representing the species of each observation
print(iris.target)

# print the encoding scheme for species: 0 = setosa, 1 = versicolor, 2 = virginica
print(iris.target_names)


I am working on the file "03-exploratory-analysis_Woodruff_FinalCopy.ipynb".
The first task is:
Build a model
Can you write Python code for it?

#create linear regression object
lm = linear_model.LinearRegression()

#train the model using the training data
lm.fit(X_train,y_train)


I am working on the file "03-exploratory-analysis_Woodruff_FinalCopy.ipynb".
The first task is:
Check and convert all data types to numerical
Can you write Python code for it?

# Check data types
df_train.dtypes

#these are all numerical but are not continuous values and therefore don't have relative significant to one another, except for week
#however, creating dummy variables for all these is too memory intensive. as such, might have to explore using a random forest model
#in addition to the linear regression model


I am working on the file "03-exploratory-analysis_Woodruff_FinalCopy.ipynb".
The first task is:
Check for missing values and drop or impute
Can you write Python code for it?

# Check counts for missing values in each column
print "train dataset missing values:"
print df_train2.isnull().sum()
print
print "test dataset missing values:"
print df_test2.isnull().sum()


I am working on the file "03-exploratory-analysis_Woodruff_FinalCopy.ipynb".
The first task is:
Evaluate the model
Can you write Python code for it?

# Check R^2 on test set
print "R^2: %0.3f" % lm.score(X_test,y_test)

# Check MSE on test set
#http://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics
print "MSE: %0.3f" % metrics.mean_squared_error(y_test, lm.predict(X_test))
#QUESTION - should i check this on train set?
print "MSE: %0.3f" % metrics.mean_squared_error(y_train, lm.predict(X_train))


I am working on the file "03-exploratory-analysis_Woodruff_FinalCopy.ipynb".
The first task is:
Kaggle score : 0.75682
Can you write Python code for it?

#notes
#want to try to use a classifier like random forest or logistic regression


I am working on the file "03-exploratory-analysis_Woodruff_FinalCopy.ipynb".
The first task is:
Part 1. Identify the Problem
Problem: Given various sales/client/product data, we want to predict demand for each product at each store on a weekly basis. Per the train dataset, the average demand for a product at a store per week is 7.2 units. However, this does not factor in cases in which store managers under-predict demand for a product which we can see when returns=0 for that week. There are 74,180,464 records in the train data, of which 71,636,003 records have returns=0 or approx 96%. This generally means that managers probably often under predict product demand (unless that are exact on the money, which seems unlikely).
Goals: The goal is to predict demand for each product at each store on a weekly basis while avoiding under-predicting demand.
Hypothesis: As stated previously, the average product demand at a store per week is 7.2 units per the train data. However, given the likelihood of managers underpredicint product demand, I hypothesize a good model should return a number higher than 7.2 units to more accurately predict demand.

Part 2. Acquire the Data
Kaggle has provided five files for this dataset:
train.csv: Use for building a model (contains target variable "Demanda_uni_equil")
test.csv: Use for submission file (fill in for target variable "Demanda_uni_equil")
cliente_tabla.csv: Contains client names (can be joined with train/test on Cliente_ID)
producto_tabla.csv: Contains product names (can be join with train/test on Producto_ID)
town_state.csv: Contains town and state (can be join with train/test on Agencia_ID)
Notes: I will further split train.csv to generate my own cross validation set. However, I will use all of train.csv to train my final model since Kaggle has already supplied a test dataset. Additionally, I am only using a random 10% of the train data given to me for EDA and model development. Using the entire train dataset proved to be too time consuming for the quick iternations needed for initial modeling building and EDA efforts. I plan to use 100% of the train dataset once I...
Can you write Python code for it?

# Load train data
# Given size of training data, I chose to use only 10% for speed reasons
# QUESTION - how can i randomize with python? i used sql to create the random sample below.
df_train = pd.read_csv("train_random10percent.csv")

# Check head
df_train.head()

# Load test data
df_test = pd.read_csv("test.csv")

# Check head. I noticed that I will have to drop certain columns so that test and train sets have the same features.
df_test.head()

#given that i cannot use a significant amount of variables in train data, i created additoinal features using the mean
#i grouped on product id since i will ultimately be predicting demand for each product

df_train_mean = df_train.groupby('Producto_ID').mean().add_suffix('_mean').reset_index()
df_train_mean.head()

#from above, adding 2 additional features, the average sales units and the average demand
df_train2 = df_train.merge(df_train_mean[['Producto_ID','Venta_uni_hoy_mean', 'Demanda_uni_equil_mean']],how='inner',on='Producto_ID')
df_train2.sample(5)

# Adding features to the test set in order to match train set
df_test2 = df_test.merge(df_train_mean[['Producto_ID','Venta_uni_hoy_mean', 'Demanda_uni_equil_mean']],how='left',on='Producto_ID')
df_test2.head()


I am working on the file "03-exploratory-analysis_Woodruff_FinalCopy.ipynb".
The first task is:
Part 3. Parse, Mine, and Refine the data
Perform exploratory data analysis and verify the quality of the data.
Check columns and counts to drop any non-generic or near-empty columns
Can you write Python code for it?

# Check columns
print "train dataset columns:"
print df_train2.columns.values
print
print "test dataset columns:"
print df_test2.columns.values

# Check counts
print "train dataset counts:"
print df_train2.count()
print
print "test dataset counts:"
print df_test2.count()


I am working on the file "03-exploratory-analysis_Woodruff_FinalCopy.ipynb".
The first task is:
Wrangle the data to address any issues from above checks
Can you write Python code for it?

# Drop columns not included in test dataset
df_train2 = df_train2.drop(['Venta_uni_hoy', 'Venta_hoy', 'Dev_uni_proxima', 'Dev_proxima'], axis=1)

# Check data
df_train2.head()

# Drop blank values in test set and replace with mean

# Replace missing values for venta_uni_hoy_mean using mean
df_test2.loc[(df_test2['Venta_uni_hoy_mean'].isnull()), 'Venta_uni_hoy_mean'] = df_test2['Venta_uni_hoy_mean'].dropna().mean()

# Replace missing values for demand using mean
df_test2.loc[(df_test2['Demanda_uni_equil_mean'].isnull()), 'Demanda_uni_equil_mean'] = df_test2['Demanda_uni_equil_mean'].dropna().mean()

print "test dataset missing values:"
print df_test2.isnull().sum()


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Additional entries can be appended onto the end of a list:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
But it might be that the input is just wrong: there's a typo, and the caller should be warned, and maybe the input completely rejected. We can re-write the program to reflect that.
Can you write Python code for it?

from math import pi

def degrees_to_radians_normalized(theta_d):
    """
    Convert an angle from degrees to radians, normalizing the output.

    Parameters
    ----------

    theta_d : float
        The angle in degrees.

    Returns
    -------

    theta_r : float
        The angle in radians. Will lie in [0, 2 pi]
    """
    if theta_d > 360.0:
        print("Input angle greater than 360 degrees. Did you mean this?")
    if theta_d < 0.0:
        print("Input angle less than 0 degrees. Did you mean this?")
    theta_r = pi / 180.0 * (theta_d % 360.0)
    return theta_r

theta_d = 5134.6
print(degrees_to_radians_normalized(theta_d))
theta_d = -52.3
print(degrees_to_radians_normalized(theta_d))


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
But it might be that the input is just wrong: there's a typo, and the caller should be warned, and maybe the input completely rejected. We can write a different function to check that.
Can you write Python code for it?

from math import pi

def check_angle_normalized(theta_d):
    """
    Check that an angle lies within [0, 360] degrees.

    Parameters
    ----------

    theta_d : float
        The angle in degrees.

    Returns
    -------

    normalized : Boolean
        Whether the angle lies within the range
    """

    normalized = True
    if theta_d > 360.0:
        normalized = False
        print("Input angle greater than 360 degrees. Did you mean this?")
    if theta_d < 0.0:
        normalized = False
        print("Input angle less than 0 degrees. Did you mean this?")
    return normalized

theta_d = 5134.6
print(check_angle_normalized(theta_d))
theta_d = -52.3
print(check_angle_normalized(theta_d))


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
But we can get much more general than that. The different things in the set don't have to have the same type:
Can you write Python code for it?

things = 1, 2.3, True, degrees_to_radians
for thing in things:
    print(thing)


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
By using a negative step we can reverse the order (as shown in the final example), but then we need to be careful with the <start> and <end>.
This <start>:<end>:<step> notation varies between programming languages: some use <start>:<step>:<end>.
Second, we can give an index that counts from the end, where the final entry is -1:
Can you write Python code for it?

print(list1[-1])
print(list1[-2])
print(list1[2:-2])
print(list1[-4:-2])


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Debugging
Earlier we saw how to read error messages to debug single statements. When we start including loops and functions it may be more complex and the information from the error message alone, whilst useful, may not be enough.
In these more complex cases the reason for the error depends on the calculations inside the code, and the steps through the code need inspecting in detail. This is where a debugger is useful. It allows you to run the code, pause at specific points or conditions, step through it as it runs line-by-line, and inspect all the values as you go. There are a number of Python debuggers - pdb and ipdb being the most basic. However, spyder has a debugger built in, and learning to use it will make your life considerably easier.
Breakpoints
The main use of the debugger is to inspect the internal state of a code whilst it is running. To do that we have to stop the execution of the code somewhere. This is typically done using breakpoints.
Copy the following function into a file named breakpoints.py:
```python
def test_sequence(N):
    """
    Compute the infinite sum of 2^{-n} starting from n = 0, truncating
    at n = N, returning the value of 2^{-n} and the truncated sum.
Parameters
----------

N : int
    Positive integer, giving the number of terms in the sum

Returns
-------

limit : float
    The value of 2^{-N}
partial_sum : float
    The value of the truncated sum

Notes
-----

The limiting value should be zero, and the value of the sum should
converge to 2.
"""

# Start sum from zero, so give zeroth term
limit = 1.0
partial_sum = 1.0

# At each step, increment sum and change summand
for n in range(1, N+1):
    partial_sum = partial_sum + limit
    limit = limit / 2.0

return limit, partial_sum

if name == 'main':
    print(test_sequence(50))
```
This computes the value $2^{-N}$ and the partial sum $\sum_{n=0}^N 2^{-n}$. The limit as $N\to\infty$ of the value should be zero, and of the sum should be two.
The final two lines ensures that, if the file is run as a python script, the...
Can you write Python code for it?

import breakpoints
print(breakpoints.test_sequence(10))
print(breakpoints.test_sequence(100))
print(breakpoints.test_sequence(1000))


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Debugging
Earlier we saw how to read error messages to debug single statements. When we start including loops and functions it may be more complex and the information from the error message alone, whilst useful, may not be enough.
In these more complex cases the reason for the error depends on the calculations inside the code, and the steps through the code need inspecting in detail. This is where a debugger is useful. It allows you to run the code, pause at specific points or conditions, step through it as it runs line-by-line, and inspect all the values as you go. There are a number of Python debuggers - pdb and ipdb being the simplest. However, spyder has a debugger built in, and learning to use it will make you life considerably easier.
Breakpoints
The main use of the debugger is to inspect the internal state of a code whilst it is running. Do do that we have to stop the execution of the code somewhere. This is typically done using breakpoints.
Copy the following function into a file named breakpoints.py:
```python
def test_sequence(N):
    """
    Compute the infinite sum of 2^{-n} starting from n = 0, truncating
    at n = N, returning the value of 2^{-n} and the truncated sum.
Parameters
----------

N : int
    Positive integer, giving the number of terms in the sum

Returns
-------

limit : float
    The value of 2^{-N}
partial_sum : float
    The value of the truncated sum

Notes
-----

The limiting value should be zero, and the value of the sum should
converge to 2.
"""

# Start sum from zero, so give zeroth term
limit = 1.0
partial_sum = 1.0

# At each step, increment sum and change summand
for n in range(1, N+1):
    partial_sum += limit
    limit /= 2.0

return limit, partial_sum

if name == 'main':
    print(test_sequence(50))
```
This computes the value $2^{-N}$ and the partial sum $\sum_{n=0}^N 2^{-n}$. The limit as $N\to\infty$ of the value should be zero, and of the sum should be two.
The final two lines ensures that, if the file is run as a python script, the function will be called (with...
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Depending on the version of Python you are using, this might either give a list or an iterator.
When iterating over a dictionary (for k in dict:) the key is returned, as if we had said for key in dict.keys():. This is most useful in a loop, such as the following. Think carefully about this code, and make sure you understand what is happening!
Can you write Python code for it?

for name in functions:
    print("The result of {}(1) is {}.".format(name, functions[name](1.0)))


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Entries can be removed (popped) from the end of a list:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
However, if a member of a tuple can itself be modified (for example, it's a list, as tuple4[1] is), then that entry can be modified:
Can you write Python code for it?

print(tuple4[1])
tuple4[1][1] = 33
print(tuple4[1])


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
In some programming languages this is where the discussion of a for loop would end: the "loop counter" must be an integer. In Python, a loop is just iterating over a set of values, and these can be much more general. An alternative way (using floats) to do the same loop would be
Can you write Python code for it?

angles = 15.0, 30.0, 45.0, 60.0, 75.0, 90.0
for angle in angles:
    print(degrees_to_radians(angle))


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Let's examine this code line by line. It first defines a set of numbers, steps, which contains the integers from 1 to 6 (we will make this more precise later when we discuss lists and tuples). We then define the loop using the for command. This looks at the set of numbers steps and picks an entry out one at a time, setting the variable n to be  the value of that member of the set. So, the first time through the loop n=1. The next, n=2. Once it has iterated through all members of the set steps, it stops.
The colon : at the end of the line defines the code block that each iteration of the loop should perform. Exactly as when we defined a function, the code block is indented by four spaces or one tab. In each iteration through the loop, the commands indented by this amount will be run. In this case, only one line (the print... line) will be run. On each iteration the value of n changes, leading to the different angle.
Writing out a long list of integers is a bad idea. A better approach is the use the range function. This compresses the code to:
Can you write Python code for it?

for n in range(1,7):
    print(degrees_to_radians(15*n))


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Let's examine this line by line. The first line defines the angle in degrees, theta_d. We start from $\theta_d=0$.
The next line defines the loop. This has similarities to our definition of a function. We use the keyword while to say that what follows is going to be a loop. We then define a logical condition that will be either True or False. Whilst the condition is True, the statements in the loop will be executed. The colon : at the end of the line ends the logical condition and says that what follows will be the statements inside the loop. As with the function, the code block with the statements inside the loop is indented by four spaces or one tab.
The code block contains two lines, both of which will be executed. The first prints the converted angle to the screen. The second increases the angle in degrees by 15. At the end of the code block, Python will check the logical condition theta_d <= 90.0 again. If it is True the statements inside the loop are executed again. If it is False, the code moves on to the next line after the loop.
There is another way to write a loop that we can use:
for loops
Can you write Python code for it?

steps = 1, 2, 3, 4, 5, 6
for n in steps:
    print(degrees_to_radians(15*n))


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Lists are defined by square brackets, []. Objects in the list are separated by commas. A list can be empty (list5 above). A list can contain other lists (list4 above). The objects in the list don't have to have the same type (list3 and list4 above).
We can access a member of a list by giving its name, square brackets, and the index of the member (starting from 0!):
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Lists are probably the most used container, but there's a closely related container that we've already used: the tuple.
Tuples
Tuples are ordered collections of objects that, once created, cannot be modified. For example:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Note
There is a big divide between programming languages that index containers (or vectors, or lists) starting from 0 and those that index starting from 1. There is no consensus on which is better, so as you move between languages, get used to checking which is used.
Entries in a list can be modified:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Note that the order it prints out need not match the order we entered the values in. In fact, the order could change if we used a different machine, or entered the values again. This emphasizes the unordered nature of dictionaries.
To access an individual value, we use its key:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Slicing
Accessing and manipulating multiple entries of a list at once is an efficient and effective way of coding: it shows up a lot in, for example, linear algebra. This is where slicing comes in.
We have seen that we can access a single element of a list using square brackets and an index. We can use similar notation to access multiple elements:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
So this does exactly the same thing as the previous loop, and most of the code is the same. However, rather than accessing the dictionary each time (using functions[name]), the value in the dictionary has been returned at the start. What is happening is that the items function is returning both the key and the value as a tuple on each iteration through the loop. The name, function notation then uses unpacking to appropriately set the variables. This form is "more pythonic" (ie, is shorter, clearer to many people, and faster).
Above we have always set the key to be a string. This is not necessary - it can be an integer, or a float, or any constant object. We have also set the values to have the same type. As with lists this is not necessary.
Dictionaries are very useful for adding simple structure to your code, and allow you to pass around complex sets of parameters easily.
Control flow
Not every algorithm can be expressed as a single mathematical formula in the manner used so far. Alternatively, it may make the algorithm appear considerably simpler if it isn't expressed in one complex formula but in multiple simpler forms. This is where the computer has to be able to make choices, to control when and if a particular formula is used.
As a simple example, if we used our degrees_to_radians calculation as previously given, then if the angle $\theta_d$ is outside the standard $[0, 360]^{\circ}$ interval then the converted angle in radians $\theta_r$ will be outside the $[0, 2\pi]$ interval.
Suppose we want to "normalize" all our angles to lie within the $[0, 2\pi]$ interval. We could use modular arithmetic using the % operator:
Can you write Python code for it?

theta_d = 5134.6
theta_d_normalized = theta_d % 360.0
print(theta_d_normalized)


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
The control flow here uses the if statement. As with loops such as the for and while loops we have a condition which is checked which, if satisfied, leads to the indented code block after the colon being executed. The logical statements theta_d > 360.0 and theta_d < 0.0 are evaluated and return either True or False (which is how Python represents boolean values). If True, then the statement is executed.
We could use only a single logical statement to check if $\theta_d$ lies in an acceptable range by using logical relations. For example, we could replace the two if statements by the single statement
python
if (theta_d > 360.0) or (theta_d < 0.0):
    normalized = False
    print("Input angle outside [0, 360] degrees. Did you mean this?")
The logical statement (theta_d > 360.0) or (theta_d < 0.0) is either True or False as above. In addition to the logical or statement, Python also has the logical and and logical not statements, from which more complex statements can be generated.
Often we want to do one thing if a condition is true, and another if the condition is false. A full example of this would be to rewrite the whole function as:
Can you write Python code for it?

from math import pi

def check_angle_normalized(theta_d):
    """
    Check that an angle lies within [0, 360] degrees.

    Parameters
    ----------

    theta_d : float
        The angle in degrees.

    Returns
    -------

    normalized : Boolean
        Whether the angle lies within the range
    """

    normalized = True
    if theta_d > 360.0:
        normalized = False
        print("Input angle greater than 360 degrees. Did you mean this?")
    elif theta_d < 0.0:
        normalized = False
        print("Input angle less than 0 degrees. Did you mean this?")
    else:
        print("Input angle in range [0, 360] degrees. Good.")
    return normalized


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
The elif statement allows another condition to be checked - it is how Python represents "else if", or "all previous checks have been false; let's check this statement as well". Multiple elif blocks can be included to check more conditions. The else statement contains no logical check: this code block will always be executed if all previous statements were false.
For example:
Can you write Python code for it?

theta_d = 543.2
print(check_angle_normalized(theta_d))
theta_d = -123.4
print(check_angle_normalized(theta_d))
theta_d = 89.12
print(check_angle_normalized(theta_d))


theta_d = 543.2
print(degrees_to_radians_normalized(theta_d))
theta_d = -123.4
print(degrees_to_radians_normalized(theta_d))
theta_d = 89.12
print(degrees_to_radians_normalized(theta_d))


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
The last example is particularly important, as this chained example (6 < 7 < 8) is equivalent to (6 < 7) and (7 < 8), which is of course true (and mathematically what you would expect). However, many programming languages would not interpret it this way, but would instead interpret it as (6 < 7) < 8, which is equivalent to True < 8, which is nonsense. Chaining operations in this way is useful in Python, but don't expect it to always work in other languages.
Containers and Sequences
When talking about loops we informally introduced the collection of objects 1, 2, 3, 4, 5, 6, and assigned it to a single variable steps. This is one of many types of container: a single object that contains other objects. If the objects in the container have an order then the container is often called a sequence: an object that contains an ordered sequence of other objects. These sorts of objects are everywhere in mathematics: sets, groups, vectors, matrices, equivalence classes, categories, ... Programming languages also implement a large number of them. Python has four essential containers, the most important of which for our purposes are lists, tuples, and dictionaries.
Lists
A list is an ordered collection of objects. For example:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
The last example is particularly important, as this chained example (6 < 7 < 8) is equivalent to (6 < 7) and (7 < 8). This checks both inequalities - checking that $6 < x < 8$ when $x=7$, by checking that both $6 < x$ and $x < 8$ is true when $x=7$, which is true (and mathematically what you would expect). However, many programming languages would not interpret it this way, but would instead interpret it as (6 < 7) < 8, which is equivalent to True < 8, which is nonsense. Chaining operations in this way is useful in Python, but don't expect it to always work in other languages.
Containers and Sequences
When talking about loops we informally introduced the collection of objects 1, 2, 3, 4, 5, 6, and assigned it to a single variable steps. This is one of many types of container: a single object that contains other objects. If the objects in the container have an order then the container is often called a sequence: an object that contains an ordered sequence of other objects. These sorts of objects are everywhere in mathematics: sets, groups, vectors, matrices, equivalence classes, categories, ... Programming languages also implement a large number of them. Python has four essential containers, the most important of which for our purposes are lists, tuples, and dictionaries.
Lists
A list is an ordered collection of objects. For example:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
The length of a list can be found:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
The number of entries on both sides must match.
Dictionaries
All the containers we have seen so far have had an order - lists and tuples are sequences, and we access the objects within them using list[0] or tuple[3], for example. A dictionary is our first unordered container. These are useful for collections of objects with meaningful names, but where the order of the objects has no importance.
Dictionaries are defined using curly braces. The "name" of each entry is given first (usually called its key), followed by a :, and then after the colon comes its value. Multiple entries are separated by commas. For example:
Can you write Python code for it?

from math import sin, cos, exp, log

functions = {"sine" : sin,
             "cosine" : cos,
             "exponential" : exp,
             "logarithm" : log}

print(functions)


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
The range function takes the input arguments <start> and <end>, and the optional input argument <step>, to produce the integers from the start up to, but not including, the end. If the <start> is not given it defaults to 0, and if the <step> is not given it defaults to 1.
(Strictly, range does not return the full list in one go. It generates the results one at a time. This is much faster and more efficient. In the for loop this is all you need. To actually view what range generates all together, convert it to a list, as list(range(...))).
Check this against examples such as:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
The slicing notation [<start>:<end>] returns all entries from the <start> to the entry before the <end>. So:

list1[1:3] returns all entries from the second (index 1) to the third (the entry before index 3).
if <end> is not given all entries up to the end are returned, so list1[2:] returns all entries from the third (index 2) to the end.
if <start> is not given all entries from the start are returned, so list1[:4] returns all entries from the start until the fourth (the entry before index 4).

There are a number of other ways that slicing can be used. First, we can specify the step:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
This can be used to directly swap two variables, for example:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
This can be used to write very efficient code, but is a feature that isn't always available in other programming languages.
When should we use for loops and when while loops? In most cases either will work. Different algorithms have different conventions, so where possible follow the convention. The advantage of the for loop is that it is clearer how much work will be done by the loop (as, in principle, we know how many times the loop block will be executed). However, sometimes you need to perform a repetitive operation but don't know in advance how often you'll need to do it. For example, to the nearest degree, what is the largest angle $\theta_d$ that when converted to radians is less than 4? (For the picky, we're restricting $\theta_d$ so that $0^{\circ} \le \theta_d < 360^{\circ}$).
Rather than doing the sensible thing and doing the analytic calculation, we can do the following:

Set $\theta_d = 0^{\circ}$.
Calculate the angle in radians $\theta_r$.

If $\theta_r < 4$:

Increase $\theta_d$ by $1^{\circ}$;
Repeat from step 2.

We can reproduce this algorithm using a while loop:
Can you write Python code for it?

theta_d = 0.0

while degrees_to_radians(theta_d) < 4.0:
    theta_d = theta_d + 1.0


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
This could be done in a for loop, but not so straightforwardly.
To summarize:
The structure of the while loop is similar to the for loop. The loop is defined by a keyword (while or for) and the end of the line defining the loop condition is given by a colon. With each iteration of the loop the indented code is executed. The difference is in how the code decides when to stop looping, and what changes with each iteration.
In the for loop the code iterates over the objects in a set, and some variable is modified with each iteration based on the new object. Once all objects in the set have been iterated over, the loop stops.
In a while loop some condition is checked; while it is true the loop continues, and as soon as it is false the loop stops. Here we are checking if $\theta_r$, given by degrees_to_radians(theta_d), is still less than 4. However, nothing in the definition of the loop actually changes: it is the statement within the loop that actually changes the angle $\theta_d$.
We quickly check that the answer given makes sense:
Can you write Python code for it?

print(theta_d - 1.0)
print(theta_d)
print(degrees_to_radians(theta_d-1.0) / 4.0)
print(degrees_to_radians(theta_d) / 4.0)


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
This is related to a very useful Python feature: unpacking. Normally we have assigned a single variable to a single value (although that value might be a container such as a list). However, we can assign multiple values in one go:
Can you write Python code for it?

a, b, c = list1[3:]
print(a)
print(b)
print(c)


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
To explain: The first line says that we are going to iterate over each entry in the dictionary by assigning the value of the key (which is the name of the function in functions) to the variable name. The next line extracts the function associated with that name using functions[name] and applies that function to the value 1 using functions[name](1.0). The full line prints out the result in a human readable form.
But Python has other ways of iterating over dictionaries that can make life even easier, such as the items function:
Can you write Python code for it?

for name, function in functions.items():
    print("The result of {}(1) is {}.".format(name, function(1.0)))


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
To find all the keys or values we can use dictionary methods:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Tuples appear a lot when using functions, either when passing in parameters, or when returning results. They can often be treated like lists, and there are functions that convert lists to tuples and vice versa:
Can you write Python code for it?

converted_list1 = list(tuple1)
converted_tuple1 = tuple(list1)


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Tuples are defined by the commas separating the entries. The round brackets () surrounding the entries are conventional, useful for clarity, and for grouping. If you want to create an empty tuple (tuple5) the round brackets are necessary. A tuple containing a single entry (tuple6) must have a trailing comma.
Tuples can be accessed in the same ways as lists, and their length found with len in the same way. But they cannot be modified, so we cannot add additional entries, or remove them, or alter any:
Can you write Python code for it?


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
Unpacking
Slicing is often seen as part of assignment. For example
Can you write Python code for it?

list_slice = [0, 0, 0, 0, 0, 0, 0, 0]
list_slice[1:4] = list1[3:]
print(list_slice)


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
We can nest statements as deep as we like, nesting loops and control flow statements as we go. We have to ensure that the indentation level is consistent. Here is a silly example.
Can you write Python code for it?

angles = [-123.4, 543.2, 89.12, 0.67, 5143.6, 30.0, 270.0]

# We run through all the angles, but only print those that are
# - in the range [0, 360], and
# - if sin^2(angle) < 0.5

from math import sin

for angle in angles:
    print("Input angle in degrees:", angle)
    if (check_angle_normalized(angle)):
        angle_r = degrees_to_radians(angle)
        if (sin(angle_r)**2 < 0.5):
            print("Valid angle in radians:", angle_r)


angles = [-123.4, 543.2, 89.12, 0.67, 5143.6, 30.0, 270.0]

# We run through all the angles, but only print those that are
# - in the range [0, 360], and
# - if sin^2(angle) < 0.5

from math import sin

for angle in angles:
    print("Input angle in degrees:", angle)
    if (angle > 0.0) and (angle < 360.0):
        angle_r = degrees_to_radians_normalized(angle)
        if (sin(angle_r)**2 < 0.5):
            print("Valid angle in radians:", angle_r)


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
We noted above that what the code is doing is printing the angles $(n \pi)/ 12$ for $n = 1, 2, \dots, 6$. It does this by calling the degrees_to_radians function on the angles $15 n$ degrees for $n = 1, 2, \dots, 6$. So this is the formula we want to reproduce in code. To do that we write a loop.
Here is a standard way to do it:
Can you write Python code for it?

theta_d = 0.0
while theta_d <= 90.0:
    print(degrees_to_radians(theta_d))
    theta_d = theta_d + 15.0


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
We see that the answer is $229^{\circ}$.
Logical statements
We noted above that whether or not a while loop executes depends on the truth (or not) of a particular statement. In programming these logical statements take Boolean values (either true or false). In Python, the values of a Boolean statement are either True or False, which are the keywords used to refer to them. Multiple statements can be chained together using the logical operators and, or, not. For example:
Can you write Python code for it?

print(True)
print(6 < 7 and 10 > 9)
print(1 < 2 or 1 < 0)
print(not (6 < 7) and 10 > 9)
print(6 < 7 < 8)


I am working on the file "03-loops-control-flow.ipynb".
The first task is:
We showed above how to use this code to print the angles $(n \pi)/ 12$ for $n = 1, 2, \dots, 6$. We did this by calling the degrees_to_radians function on the angles $15 n$ degrees for $n = 1, 2, \dots, 6$. So this is the formula we want to reproduce in code. To do that we write a loop.
Here is a standard way to do it:
Can you write Python code for it?


I am working on the file "03-matplotlib.ipynb".
The first task is:
A maior parte do tempo você vai trabalhar com o objeto Axes.  A documentação
é bem detalhada e recomendo a leitura, principalmente do método plot:

http://matplotlib.org/api/pyplot_summary.html
http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.plot
http://matplotlib.org/api/axes_api.html?#matplotlib.axes.Axes.plot
Can you write Python code for it?


I am working on the file "03-matplotlib.ipynb".
The first task is:
Exemplo com dados reais
Can you write Python code for it?

import numpy as np
import numpy.ma as ma

data = np.loadtxt("./data/dados_pirata.csv", skiprows=1,
                  usecols=range(2, 16), delimiter=',')

data = ma.masked_less_equal(data, -9999)

plt.pcolormesh(np.flipud(data.T))
plt.colorbar();

fig, ax = plt.subplots(figsize=(12.5, 5))

cs = ax.pcolormesh(np.flipud(data.T))

fig.colorbar(cs);

z = [1, 10, 100, 120, 13, 140, 180, 20, 300, 40, 5, 500, 60, 80]

fig, ax = plt.subplots()
ax.plot(data[42, :], z, 'ko')
ax.invert_yaxis()

fig, ax = plt.subplots(figsize=(9, 3))

ax.plot(data[:, 0])

ax.set_xlabel("Tempo")
ax.set_ylabel("Temperatura")
ax.set_title(u"Série-temporal PIRATA");

def plot_ts(data):
    fig, ax = plt.subplots(figsize=(9, 3))

    ax.plot(data)
    ax.set_xlabel("Tempo")
    ax.set_ylabel("Temperatura")
    ax.set_title(u"Série-temporal PIRATA")

plot_ts(data)

from matplotlib import style

style.available

with style.context('dark_background'):
    plot_ts(data[:, 0])

with style.context('ggplot'):
    plot_ts(data[:, 0])

style.use('ggplot')

n, bins, pathes = plt.hist(data[0, :])

style.use('ggplot')

idx = 0
mask = data.mask[:, 0]

n, bins, pathes = plt.hist(data[:, 0].data[~mask], bins=10)

idx = 12

mask = data.mask[:, idx]

n = plt.hist(data[:, idx].data[~mask], bins=30)

def make_wave():
    twopi = 2 * np.pi
    n = 2
    A = 2
    w = twopi / 10
    k = twopi / 200
    phi = np.deg2rad(180)
    t = np.arange(0, 10 * n, 0.01)
    y = A * np.cos(w * t - k * 0 + phi)

    fig, ax = plt.subplots(figsize=(8, 4))
    ax.plot(t, y, linewidth='2', color='#006633')
    ax.set_frame_on(False)
    ax.axis([-0.1, 20.1, -3, 3])
    ax.get_xaxis().set_ticks([])
    ax.get_yaxis().set_ticks([])

    # Box.
    kw = dict(linestyle='--', color='grey', zorder=90)
    ax.axhline(y=2, xmin=0.15, xmax=0.85, **kw)
    ax.axhline(y=-2, xmin=0.15, xmax=0.85, **kw)
    ax.axvline(x=5, ymin=-0.15, ymax=1.85, **kw)
    ax.axvline(x=15, ymin=-0.15, ymax=1.85, **kw)

    # Zero line.
    _ = ax.annotate('', xy=(20, 0), xycoords='data',
               ...


I am working on the file "03-matplotlib.ipynb".
The first task is:
O coração de todos os gráficos é o objeto figure.  O objeto Figure
está no pode ser desenhado em qualquer formato de saída ou apenas a tela.
Can you write Python code for it?


I am working on the file "03-matplotlib.ipynb".
The first task is:
Outros tipos de gráficos
Can you write Python code for it?

import numpy as np

x = np.linspace(-180, 180, 60)
y = np.linspace(-90, 90, 30)
x2d, y2d = np.meshgrid(x, y)

data = np.cos(3 * np.deg2rad(x2d)) + np.sin(2 * np.deg2rad(y2d))

plt.contourf(x, y, data);

plt.imshow(data, extent=[-180, 180, -90, 90],
           interpolation='nearest', origin='lower');

plt.pcolormesh(x, y, data);

plt.scatter(x2d, y2d, c=data, s=15);

plt.bar(x, data.sum(axis=0), width=np.diff(x)[0]);

plt.plot(x, data.sum(axis=0),
         linestyle='--',
         marker='d',
         markersize=10,
         color='red');


I am working on the file "03-matplotlib.ipynb".
The first task is:
Sozinho o objeto Figure não é interessante.  Temos que "desenhar" algo nele.
Os objetos de desenho do matplotlib são chamados de Artists.  De longe o
Artists mais útil é o Axes artist.  O Axes artist representa o
espaço dos dados (x, y) ou em caso de coordenadas polares (r, $\theta$).
Não há limite no número de Artists que podemos adicionar em uma figura.
Vamos ver um exemplo.
Can you write Python code for it?


I am working on the file "03-model-inference-hugectr.ipynb".
The first task is:
1. Exporting Triton ensemble model
A Triton ensemble model represents a pipeline of one or more models and the connection of input and output tensors between these models. Ensemble models are intended to encapsulate inference pipelines that involves multiple steps, each performed by a different model. For example, a common  "data preprocessing -> inference -> data postprocessing" pattern. Using ensemble models for this purpose can avoid the overhead of transferring intermediate tensors between client and serving endpoints and minimize the number of requests that must be sent to Triton.
In our case, an inference pipeline comprises two steps: input preprocessing using a fitted NVTabular workflow and generating predictions using a HugeCTR ranking model.
An ensemble model is not an actual serialized model. There are no addtional model artifacts created when an ensemble is defined. It is a configuration that specifies which actual models comprise the ensemble, the execution flow when processing an inference request and the flow of data between inputs and outputs of the component models. This configuration is defined using the same protocol buffer based configuration format as used for serving other model types in Triton. Refer to Triton Inference Server Model Configuration guide for detailed information about configuring models and model ensembles.
You can create an ensemble model manually by arranging the component models into the prescribed folder structure and editing the required configuration files. For ensemble models that utilize the "NVTabular workflow -> Inference Model" processing pattern you can utilize a set of utility functions provided by the nvtabular.inference.triton module. Specifically to create a "NVTabular workflow -> HugeCTR model" ensemble, as utilized in this notebook, you can use the nvtabular.inference.triton.export_hugectr_ensemble function.
We have encapsulated the ensemble export logic in the src.serving.export_ensemble function. In addition to calling...
Can you write Python code for it?

if os.path.isdir(LOCAL_WORKSPACE):
    shutil.rmtree(LOCAL_WORKSPACE)
os.makedirs(LOCAL_WORKSPACE)

!gsutil -m cp -r {WORKFLOW_MODEL_PATH} {LOCAL_WORKSPACE}
!gsutil -m cp -r {HUGECTR_MODEL_PATH} {LOCAL_WORKSPACE}


I am working on the file "03-model-inference-hugectr.ipynb".
The first task is:
2. Building a custom serving container
The custom serving container is derived from the NVIDIA NGC Merlin inference container. It adds Google Cloud SDK and an entrypoint script that executes the tasks described in detail in the overview.
Can you write Python code for it?


I am working on the file "03-model-inference-hugectr.ipynb".
The first task is:
3. Uploading the model and its metadata to Vertex Models.
In the following cell you will register (upload) the ensemble model as a Vertex AI Prediction Model resource.
Refer to Use a custom container for prediction guide for detailed information about creating Vertex AI Prediction Model resources.
Notice that the value of  model_repository_path that was used when exporting the ensemble is passed as a command line parameter to the serving container. The entrypoint script in the container will copy the ensemble artifacts to this location when the container starts. This ensures that the locations of the artifacts in the container's local file system and the paths in the ps.json and other configuration files used by Triton match.
Can you write Python code for it?

serving_container_args = [model_repository_path]

model = vertex_ai.Model.upload(
    display_name=MODEL_DISPLAY_NAME,
    description=MODEL_DESCRIPTION,
    serving_container_image_uri=IMAGE_URI,
    artifact_uri=gcs_ensemble_path,
    serving_container_args=serving_container_args,
    sync=True
)

model.resource_name


I am working on the file "03-model-inference-hugectr.ipynb".
The first task is:
4. Deploying the model to Vertex AI Prediction.
Deploying a Vertex AI Prediction Model is a two step process. First you create an endpoint that will expose an external interface to clients consuming the model. After the endpoint is ready you can deploy multiple versions of a model to the endpoint.
Refer to Deploy a model using the Vertex AI API guide for more information about the APIs used in the following cells.
Create the Vertex Endpoint
Before deploying the ensemble model you need to create a Vertex AI Prediction endpoint.
Can you write Python code for it?

endpoint = vertex_ai.Endpoint.create(
    display_name=ENDPOINT_DISPLAY_NAME
)


I am working on the file "03-model-inference-hugectr.ipynb".
The first task is:
5. Invoking the model
To invoke the ensemble through Vertex AI Prediction endpoint you need to format your request using a standard Inference Request JSON Object or a Inference Request JSON Object with a binary extension and submit a request to Vertex AI Prediction REST rawPredict endpoint. You need to use the rawPredict rather than predict endpoint because inference request formats used by Triton are not compatible with the Vertex AI Prediction standard input format.
The below cell shows a sample request body formatted as a standard Inference Request JSON Object. The request encapsulates a batch of three records from the Criteo dataset.
Can you write Python code for it?


I am working on the file "03-model-inference-hugectr.ipynb".
The first task is:
Change the following variables ONLY if necessary.
You can leave the default variables.
Can you write Python code for it?

MODEL_ARTIFACTS_REPOSITORY = f'gs://{STAGING_BUCKET}/recsys-models'

MODEL_NAME = 'deepfm'
MODEL_VERSION = 'v01'
MODEL_DISPLAY_NAME = f'criteo-hugectr-{MODEL_NAME}-{MODEL_VERSION}'
MODEL_DESCRIPTION = 'HugeCTR DeepFM model'
ENDPOINT_DISPLAY_NAME = f'hugectr-{MODEL_NAME}-{MODEL_VERSION}'

LOCAL_WORKSPACE = '/home/jupyter/serving_notebook'
IMAGE_NAME = 'triton-serving'
IMAGE_URI = f'gcr.io/{PROJECT_ID}/{IMAGE_NAME}'
DOCKERNAME = 'triton'


I am working on the file "03-model-inference-hugectr.ipynb".
The first task is:
Change the following variables according to your definitions.
Can you write Python code for it?

# Project definitions
PROJECT_ID = '<YOUR PROJECT ID>' # Change to your project id.
REGION = '<LOCATION OF RESOURCES>'  # Change to your region.

# Bucket definitions
STAGING_BUCKET = '<YOUR BUCKET NAME>' # Change to your bucket.
WORKFLOW_MODEL_PATH = "gs://..." # Change to GCS path of the nvt workflow
HUGECTR_MODEL_PATH = "gs://..." # Change to GCS path of the hugectr trained model


I am working on the file "03-model-inference-hugectr.ipynb".
The first task is:
The deepfm folder contains artifacts and configurations for the HugeCTR model. The deepfm_ens folder contains a configuration for the ensemble model. And the deepfm_nvt contains artifacts and configurations for the NVTabular preprocessing workflow. The ps.json file contains information required by the Triton's HugeCTR backend.
Notice that the file paths in ps.json  use the value from model_repository_path.
Can you write Python code for it?


I am working on the file "03-model-inference-hugectr.ipynb".
The first task is:
The previous cell created the following local folder structure
Can you write Python code for it?


I am working on the file "03-model-inference-hugectr.ipynb".
The first task is:
Upload the ensemble to GCS
In the later steps you will register the exported ensemble model as a Vertex AI Prediction model resource. Before doing that we need to move the ensemble to GCS.
Can you write Python code for it?

gcs_ensemble_path = '{}/{}'.format(MODEL_ARTIFACTS_REPOSITORY, Path(local_ensemble_path).parts[-1])

!gsutil -m cp -r {local_ensemble_path}/* {gcs_ensemble_path}/


I am working on the file "03-model-inference-hugectr.ipynb".
The first task is:
You use Cloud Build to build the serving container and push it to your projects Container Registry.
Can you write Python code for it?

FILE_LOCATION = './src'
! gcloud builds submit --config src/cloudbuild.yaml --substitutions _DOCKERNAME=$DOCKERNAME,_IMAGE_URI=$IMAGE_URI,_FILE_LOCATION=$FILE_LOCATION --timeout=2h --machine-type=e2-highcpu-8


I am working on the file "03-neural-networks/lecture.ipynb".
The first task is:
Comparing a fully connected layer and a Conv layer
Can you write Python code for it?

from keras.layers import Conv2D

fc = Sequential()
fc.add(Dense(units=4, input_dim=32*32))
fc.add(Activation('relu'))

fc.summary()

nodes = 4
32*32*nodes + nodes

cnn = Sequential()
cnn.add(Conv2D(4, (3, 3), input_shape=(32, 32, 1))) # 32x32 picture with one channel
# 4 -> filters: is the number of filters
# 3 -> strides: An integer or tuple/list of 2 integers, specifying the strides of the convolution along the width and height.
cnn.add(Activation('relu'))

cnn.summary()

n_filters = 4
n_filters * 3*3 + n_filters


I am working on the file "03-neural-networks/lecture.ipynb".
The first task is:
For any serious Neural Networking use keras
https://keras.io/ is a library that implements all the cool and useful layers and optimizers
that are used in today's deep learning. Importantly it uses code written in C instead of
python so it is fast (and if you have a GPU it will run on that which is even faster).
Can you write Python code for it?

## world's simplest NN with keras
from keras.models import Sequential
from keras.losses import mean_squared_error

from keras.layers import Dense, Activation
from keras.optimizers import SGD
from keras.initializers import RandomUniform

# construct a model as close as possible to the one scikit-learn uses
model = Sequential()

model.add(Dense(units=1, input_dim=1,
                bias_initializer=RandomUniform(minval=-3**0.5, maxval=3**0.5)
               ))
model.add(Activation('tanh'))
# for regression the last layer in sklearn is the identity
model.add(Dense(units=1))

model.compile(loss=mean_squared_error,
              optimizer=SGD(lr=0.001))

np.random.seed(12345)
X = np.random.normal(scale=2, size=(1600,1))
y = X[:, 0]

history = model.fit(X, y, epochs=3200, batch_size=200, validation_split=0.2, verbose=False)

print('minimum loss:', np.min(history.history['val_loss']))
plt.plot(history.history['loss'], label='loss')
plt.plot(history.history['val_loss'], label='validation loss');
plt.legend(loc='best');

model.predict([[0.], [1.5], [-1.4]])


I am working on the file "03-neural-networks/lecture.ipynb".
The first task is:
Ingredients for backpropagation
Implement two simple modules that know how to compute their "local" gradients. Then build up
a more complicated module that contains those two. We can treat each successive module
as a black-box that just "magically" knows how to analytically compute the gradients.
Below we implement $f(x, y, z) = (x+y)z$ and compute the gradient.
Can you write Python code for it?

class Multiply:
    def forward(self, x, y):
        self.x = x
        self.y = y
        return x * y

    def backward(self, dLdz):
        dzdx = self.y
        dLdx = dLdz * dzdx

        dzdy = self.x
        dLdy = dLdz * dzdy
        return [dLdx, dLdy]

class Add:
    def forward(self, x, y):
        self.x = x
        self.y = y
        return x + y

    def backward(self, dLdz):
        dzdy = 1
        dzdx = 1
        return [dLdz * dzdy, dLdz * dzdx]

def f_with_gradients(x, y, z):
    # create our operators
    q = Add()
    f = Multiply()

    # feed inputs into the summer first, then do multiplication
    # this builds our computational graph
    q_out = q.forward(x, y)
    f_out = f.forward(q_out, z)

    # this one is somehow weird ... but hey.
    # step backwards through our graph to compute the gradients
    grad_f = f.backward(1.)
    grad_q = q.backward(grad_f[0])

    # sort our gradients so we have [df/dx, df/dy, df/dz]
    gradients = [grad_q[0], grad_q[1], grad_f[1]]
    return f_out, gradients

f_with_gradients(-2, 5, -4)


I am working on the file "03-neural-networks/lecture.ipynb".
The first task is:
We can now zoom out, make a "f module", that contains some differentiable magic on the inside and stop caring how it works:
Can you write Python code for it?

class F:
    def forward(self, x, y, z):
        self.q = Add()
        self.f = Multiply()

        self.q_out = self.q.forward(x, y)
        self.f_out = self.f.forward(self.q_out, z)
        return self.f_out

    def backward(self, dfdz):
        grad_f = self.f.backward(dfdz)
        grad_q = self.q.backward(grad_f[0])

        return [grad_q[0], grad_q[1], grad_f[1]]

f = F()
print('f(x, y, z) = ', f.forward(-2, 5, -4))

print('[df/dx, df/dy, df/dz] = ', f.backward(1))


I am working on the file "03-numba.ipynb".
The first task is:
For numba, we use exactly the same implementations of code as the Python case, we just decorate the functions with the JIT. First try running with only decorating the calcTotalEnergy function.
Can you write Python code for it?

from math import cos,pi,sqrt
import numba

def potentialEnergyFunk1(r,width=1.0,height=10.0):
    '''
    Calculates the (soft) potential energy between two atoms

    Parameters
    ----------
    r: float
        separation distance between two atoms
    height: float
        breadth of the potential i.e. where the potential goes to zero
    width: float
        strength/height of the potential
    '''
    if r<width:
        return 0.5 * height * (1 + cos(pi*r/width))
    else:
        return 0

from math import sqrt

@numba.jit()
def calcTotalEnergy1(x,y,z,lx,ly,lz):
    '''
    Parameters
    ----------
    x,y,z: lists of floats
        1-D lists of cartesian coordinate positions

    lx,ly,lz: floats
        simulation box dimensions
    '''

    #sanity check
    assert len(x) == len(y) == len(z)

    # store half box lengths for minimum image convention
    lx2 = lx/2.0
    ly2 = ly/2.0
    lz2 = lz/2.0

    U = 0

    #The next two loops simply loop over every element in the x, y, and z arrays
    for i,(x1,y1,z1) in enumerate(zip(x,y,z)):
        for j,(x2,y2,z2) in enumerate(zip(x,y,z)):

            # We only need to consider each pair once
            if i>=j:
                continue

            dx = abs(x1 - x2)
            dy = abs(y1 - y2)
            dz = abs(z1 - z2)

            # The next few lines take care of applying the minimum image convention
            # This is neccesary because many/most molecular simulations use periodic boundary conditions
            if dx > lx2:
                dx -= lx
            if dy > ly2:
                dy -= ly
            if dz > lz2:
                dz -= lz

            dist = sqrt(dx*dx + dy*dy + dz*dz)

            U += potentialEnergyFunk1(dist)

    return U


I am working on the file "03-numba.ipynb".
The first task is:
Memory profiling!
Can you write Python code for it?

memprof = %memit -o calcTotalEnergy3(pos,box)

usage = memprof.mem_usage[0]
incr = memprof.mem_usage[0] - memprof.baseline
with open('prof/numba3.memprof','w') as f:
    f.write('{}\n{}\n'.format(usage,incr))


I am working on the file "03-numba.ipynb".
The first task is:
Runtime profiling!
Can you write Python code for it?

%%prun -D prof/numba2.prof
energy = calcTotalEnergy2(x,y,z,lx,ly,lz)

with open('energy/numba2.dat','w') as f:
    f.write('{}\n'.format(energy))


%%prun -D prof/numba1.prof
energy = calcTotalEnergy1(x,y,z,lx,ly,lz)

with open('energy/numba1.dat','w') as f:
    f.write('{}\n'.format(energy))


I am working on the file "03-numba.ipynb".
The first task is:
We load in the position and box information created in the intro notebook. If you haven't run that notebook, this line will not work! (You don't have to read the wall of text, just run the cells...)
Can you write Python code for it?

import numpy as np
pos = np.loadtxt('data/positions.dat')
box = np.loadtxt('data/box.dat')

x,y,z = map(list,pos.T)
lx,ly,lz = box

print('Read {:d} positions.'.format(pos.shape[0]))
print('x min/max: {:+4.2f}/{:+4.2f}'.format(pos.min(0)[0],pos.max(0)[0]))
print('y min/max: {:+4.2f}/{:+4.2f}'.format(pos.min(0)[1],pos.max(0)[1]))
print('z min/max: {:+4.2f}/{:+4.2f}'.format(pos.min(0)[2],pos.max(0)[2]))


I am working on the file "03-plotting-data.ipynb".
The first task is:
A better solution here is to fill all NaN values with the mean weight value:
Can you write Python code for it?

nan_mean = survey.copy()
nan_mean.weight.fillna(survey.weight.mean(), inplace=True)
print(survey.weight.mean(), nan_mean.weight.mean())
nan_mean.head()


I am working on the file "03-plotting-data.ipynb".
The first task is:
And then produce a line plot:
Can you write Python code for it?

from matplotlib import pyplot
pyplot.plot(x, y1)
pyplot.show()


I am working on the file "03-plotting-data.ipynb".
The first task is:
Data across multiple files
Now let's look at a slightly more complex example where the data is spread across multiple files and contains many different fields of different datatypes.
Spotify provide a web API which can be used to download data about its music. This data includes the audio features of a track, a set of measures including 'acousticness', 'danceability', 'speechiness' and 'valence':

A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).

We can download this data using a library such as spotipy. In the folder spotify_data, you shall find a few .csv files containing data downloaded for tracks from playlists of several different musical genres.
Let's begin by importing our data.
Can you write Python code for it?

dfs = {'indie': pandas.read_csv('spotify_data/indie.csv'), 'pop': pandas.read_csv('spotify_data/pop.csv'),
       'country': pandas.read_csv('spotify_data/country.csv'), 'metal': pandas.read_csv('spotify_data/metal.csv'),
       'house': pandas.read_csv('spotify_data/house.csv'), 'rap': pandas.read_csv('spotify_data/rap.csv')}


I am working on the file "03-plotting-data.ipynb".
The first task is:
Exercises

Create a histogram of the SepalWidth for each of the species groups in the iris dataset
Plot acousticness against liveness for the music dataset. Use a for loop to add the different datasets to the plot (i.e. rather than typing each out by hand, as done above).
Can you write Python code for it?

for name, df in iris.groupby('Name'):
    # create a new figure
    pyplot.figure()
    # plot histogram of sepalwidth
    df['SepalWidth'].plot.hist()
    # add title
    pyplot.title(name)


I am working on the file "03-plotting-data.ipynb".
The first task is:
Exercises

Load the Excel spreadsheet from https://github.com/IanHawke/msc-or-week0/blob/master/excel_data/dice-roll-example.xlsx?raw=true which generates and displays three dice rolls. Work out how to isolate the data for just the dice rolls (eg, use skiprows when loading, and drop unused columns) and compute summary data for each.
Create a histogram of the SepalWidth for each of the species groups in the iris dataset
Plot acousticness against liveness for the music dataset. Use a for loop to add the different datasets to the plot (i.e. rather than typing each out by hand, as done above).
Can you write Python code for it?

dice = pandas.read_excel('https://github.com/IanHawke/msc-or-week0/blob/master/excel_data/dice-roll-example.xlsx?raw=true', skiprows=5)
print(dice.columns)
dice = dice[['# 1', '# 2', '# 3']]
print(dice.columns)

dice.describe()

for name, df in iris.groupby('Name'):
    # create a new figure
    pyplot.figure()
    # plot histogram of sepalwidth
    df['SepalWidth'].plot.hist()
    # add title
    pyplot.title(name)


I am working on the file "03-plotting-data.ipynb".
The first task is:
Finally, let's try printing the first few records in our dataframe:
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
For our dataset, this is not the best choice as it will change the mean of our data:
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
From this alone we can get a lot of information: house tracks are on average almost twice as long as tracks from the other genres, over 97% of rap tracks contain explicit lyrics, metal tracks are the most energetic but tend to be sadder (lower valence) than country or indie. Let's try sorting our data to find the saddest tracks in each genre.
We do this by sorting the data by valence (sort_values('valence')), grouping by genre (groupby('genre')) then by taking the first value of each group (head(1)).
Can you write Python code for it?

data.sort_values('valence')[['album', 'artists', 'name', 'genre', 'valence']].groupby('genre').head(1)


I am working on the file "03-plotting-data.ipynb".
The first task is:
In the solution below for the music genre exercise, we've included a few extra steps in order to format the plot and make it more readable (e.g. changing the axis limits, increasing the figure size and fontsize).
Can you write Python code for it?

# create a new axis
fig, axis = pyplot.subplots()

# create a dictionary of colours
colours = {'indie': 'red', 'pop': 'blue',
           'country': 'green', 'metal': 'black',
           'house': 'orange', 'rap': 'pink'}
# create a dictionary of markers
markers = {'indie': '+', 'pop': 'x',
           'country': 'o', 'metal': 'd',
           'house': 's', 'rap': '*'}

for name, df in data.groupby('genre'):
    df.plot.scatter('acousticness', 'liveness', label=name, s=30, color=colours[name], marker=markers[name],
                    ax=axis, figsize=(10,8), fontsize=16)

# set limits of x and y axes so that they are between 0 and 1
axis.set_xlim([0,1.0])
axis.set_ylim([0,1.0])

# set the font size of the axis labels
axis.xaxis.label.set_fontsize(16)
axis.yaxis.label.set_fontsize(16)
pyplot.show()


I am working on the file "03-plotting-data.ipynb".
The first task is:
It may be that it's more appropriate for us to set all missing data with a certain value. For example, let's set all missing weights to 0:
Can you write Python code for it?

nan_zeros = survey.copy() # make a copy so we don't overwrite original dataframe
nan_zeros.weight.fillna(0, inplace=True)
nan_zeros.head()


I am working on the file "03-plotting-data.ipynb".
The first task is:
Let's get some information about the file we just read in. First, let's see what data fields our dataset has:
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
Matplotlib will allow you to generate and place axes pretty much wherever you like, to use logarithmic scales, to do different types of plot, and so on. Check the examples and gallery for details.
Data sets
If the information is not in numpy arrays but in a spreadsheet-like format, Matplotlib may not be the best approach.
For handling large data sets, the standard Python library is pandas. It keeps the data in a dataframe which keeps the rectangular data together with its labels.
Let's load the standard Iris data set, which we can get from GitHub, in:
Can you write Python code for it?

import pandas

iris = pandas.read_csv('https://raw.githubusercontent.com/pandas-dev/pandas/master/pandas/tests/data/iris.csv')


I am working on the file "03-plotting-data.ipynb".
The first task is:
Matplotlib will allow you to generate and place axes pretty much wherever you like, to use logarithmic scales, to do different types of plot, and so on. Check the examples and gallery for details.
Exercise
The logistic map builds a sequence of numbers ${ x_n }$ using the relation
$$
  x_{n+1} = r x_n \left( 1 - x_n \right),
$$
where $0 \le x_0 \le 1$.

Write a program that calculates the first $N$ members of the sequence, given as input $x_0$ and $r$ (and, of course, $N$).
Fix $x_0=0.5$. Calculate the first 2,000 members of the sequence for $r=1.5$ and $r=3.5$ Plot the last 100 members of the sequence in both cases. What does this suggest about the long-term behaviour of the sequence?
Fix $x_0 = 0.5$. For each value of $r$ between $1$ and $4$, in steps of $0.01$, calculate the first 2,000 members of the sequence. Plot the last 1,000 members of the sequence on a plot where the $x$-axis is the value of $r$ and the $y$-axis is the values in the sequence. Do not plot lines - just plot markers (e.g., use the 'k.' plotting style).
Can you write Python code for it?

def logistic(x0, r, N = 1000):
    sequence = [x0]
    xn = x0
    for n in range(N):
        xnew = r*xn*(1.0-xn)
        sequence.append(xnew)
        xn = xnew
    return sequence

x0 = 0.5
N = 2000
sequence1 = logistic(x0, 1.5, N)
sequence2 = logistic(x0, 3.5, N)
pyplot.plot(sequence1[-100:], 'b-', label = r'$r=1.5$')
pyplot.plot(sequence2[-100:], 'k-', label = r'$r=3.5$')
pyplot.xlabel(r'$n$')
pyplot.ylabel(r'$x$')
pyplot.show()


I am working on the file "03-plotting-data.ipynb".
The first task is:
Note that pandas can read Excel files (using pandas.read_excel), and takes as arguments either the URL (as here) or the filename on the local machine.
Once we have the data, <dataframe>.plot gives us lots of options to plot the result. Let's plot a histogram of the Petal Length:
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
Now let's see what datatype (i.e. integer, boolean, string, float,...) the data in each field is:
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
Now onto some analysis. Let's first look at some statistics for each of our genres.
Can you write Python code for it?

data[['duration_ms', 'explicit', 'popularity', 'acousticness', 'danceability', 'energy', 'instrumentalness',
     'liveness', 'loudness', 'speechiness', 'tempo', 'valence', 'genre']].groupby('genre').mean()


I am working on the file "03-plotting-data.ipynb".
The first task is:
Sucess! Before we do anything else, let's write our single combined dataset to file.
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
The above command is only needed if you are plotting in a Jupyter notebook.
We now construct some data:
Can you write Python code for it?

import numpy

x = numpy.linspace(0, 1)
y1 = numpy.sin(numpy.pi * x) + 0.1 * numpy.random.rand(50)
y2 = numpy.cos(3.0 * numpy.pi * x) + 0.2 * numpy.random.rand(50)


I am working on the file "03-plotting-data.ipynb".
The first task is:
The axes variable contains all of the separate axes that you may want. This makes it easy to construct many subplots using a loop:
Can you write Python code for it?

data = []
for nx in range(2,5):
    for ny in range(2,5):
        data.append(numpy.sin(nx * numpy.pi * x) + numpy.cos(ny * numpy.pi * x))

fig, axes = pyplot.subplots(nrows=3, ncols=3, figsize=(10,10))
for nrow in range(3):
    for ncol in range(3):
        ndata = ncol + 3 * nrow
        axes[nrow, ncol].plot(x, data[ndata])
        axes[nrow, ncol].set_xlabel(r'$x$')
        axes[nrow, ncol].set_ylabel(r'$\sin({} \pi x) + \cos({} \pi x)$'.format(nrow+2, ncol+2))
fig.tight_layout()
pyplot.show()


I am working on the file "03-plotting-data.ipynb".
The first task is:
There are several different ways we can deal with NaNs - which we choose depends on the individual dataset.
It may be that missing data is due to e.g. the machine reading the data in malfunctioning, and the best practice is just to discard all records containing missing data. We can do that with the dropna function.
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
This has given us a fairly sizeable dataframe with 513 rows and 32 columns. However, if you look closely at the index column you'll notice something dodgey has happened - combining our dataframes has meant that the index field is no longer unique (multiple records share the same index).
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
This is not good. Looking at the printout of the dataframe above, we see that the last record is LOYALTY. by Kendrick Lamar and has index 46. However, if we try to access the record with index 46, we instead get Rebellion (Lies) by Arcade Fire.
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
This suggests that, for $r=1.5$, the sequence has settled down to a fixed point. In the $r=3.5$ case it seems to be moving between four points repeatedly.
Can you write Python code for it?

# This is the "best" way of doing it, but we may not have much numpy yet
# r_values = numpy.arange(1.0, 4.0, 0.01)
# This way only uses lists
r_values = []
for i in range(302):
    r_values.append(1.0 + 0.01 * i)
x0 = 0.5
N = 2000
for r in r_values:
    sequence = logistic(x0, r, N)
    pyplot.plot(r*numpy.ones_like(sequence[1000:]), sequence[1000:], 'k.')
pyplot.xlabel(r'$r$')
pyplot.ylabel(r'$x$')
pyplot.show()


I am working on the file "03-plotting-data.ipynb".
The first task is:
To compare the data from these different datasets, it will help if we first combine them into a single dataframe. Before we do this, we'll add an extra field to each of our dataframes describing the musical genre so that we do not lose this information when the dataframes are combined.
Can you write Python code for it?

# add genre field to each dataframe
for name, df in dfs.items():
    df['genre'] = name

# combine into single dataframe
data = pandas.concat(dfs.values())
data


I am working on the file "03-plotting-data.ipynb".
The first task is:
Unfortunately, there are still duplicates where the same track appears in multiple playlists. Let's remove these duplicates, keeping only the first instance.
Can you write Python code for it?

data = data[~data.index.duplicated(keep='first')]
data.index.is_unique


I am working on the file "03-plotting-data.ipynb".
The first task is:
We can add labels and titles:
Can you write Python code for it?

pyplot.plot(x, y1)
pyplot.xlabel('x')
pyplot.ylabel('y')
pyplot.title('A single line plot')
pyplot.show()


I am working on the file "03-plotting-data.ipynb".
The first task is:
We can also combine criteria like so:
Can you write Python code for it?

iris[(iris.Name == 'Iris-setosa') & (iris.PetalWidth < 0.3)].head()


I am working on the file "03-plotting-data.ipynb".
The first task is:
We can also extract specific metrics:
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
We can change the plotting style, and use LaTeX style notation where needed:
Can you write Python code for it?

pyplot.plot(x, y1, linestyle='--', color='black', linewidth=3)
pyplot.xlabel(r'$x$')
pyplot.ylabel(r'$y$')
pyplot.title(r'A single line plot, roughly $\sin(\pi x)$')
pyplot.show()


I am working on the file "03-plotting-data.ipynb".
The first task is:
We can plot two lines at once, and add a legend, which we can position:
Can you write Python code for it?

pyplot.plot(x, y1, label=r'$y_1$')
pyplot.plot(x, y2, label=r'$y_2$')
pyplot.xlabel(r'$x$')
pyplot.ylabel(r'$y$')
pyplot.title('Two line plots')
pyplot.legend(loc='lower left')
pyplot.show()


I am working on the file "03-plotting-data.ipynb".
The first task is:
We can remedy this by reindexing. Looking at the fields available, it looks like the tracks' id would be a good choice for a unique index.
Can you write Python code for it?

data.set_index('id', inplace=True)

data.index.is_unique


I am working on the file "03-plotting-data.ipynb".
The first task is:
We can see the underlying library is Matplotlib, but it's far easier to plot large data sets.
We can get some basic statistics for our data using describe():
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
We can select subsets of our data using criteria. For example, we can select all records with PetalLength greater than 5:
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
We can visualise our data by plotting the various characteristics against each other. In the plot below, we compare the energy and danceability of country, metal and house music. The data from the three different genres separates into three pretty distinct clusters.
Can you write Python code for it?

colours = ['red', 'blue', 'green', 'orange', 'pink', 'purple']

ax = data[data.genre == 'country'].plot.scatter('danceability', 'energy', c=colours[0], label='country', figsize=(10,10))
data[data.genre == 'metal'].plot.scatter('danceability', 'energy', c=colours[1], marker='x', label='metal', ax=ax)
data[data.genre == 'house'].plot.scatter('danceability', 'energy', c=colours[2], marker='+', label='house', ax=ax)


I am working on the file "03-plotting-data.ipynb".
The first task is:
We may just wish to discard records with NaNs in a particular column (e.g. if we wish to deal with NaNs in other columns in a different way). We can discard all the records with NaNs in the weight column like so:
Can you write Python code for it?


I am working on the file "03-plotting-data.ipynb".
The first task is:
We would probably prefer to use subplots. At this point we have to leave the simple interface, and start building the plot using its individual components, figures and axes, which are objects to manipulate:
Can you write Python code for it?

fig, axes = pyplot.subplots(nrows=1, ncols=2, figsize=(10,6))
axis1 = axes[0]
axis1.plot(x, y1)
axis1.set_xlabel(r'$x$')
axis1.set_ylabel(r'$y_1$')
axis2 = axes[1]
axis2.plot(x, y2)
axis2.set_xlabel(r'$x$')
axis2.set_ylabel(r'$y_2$')
fig.tight_layout()
pyplot.show()


I am working on the file "03-python-cmdline-instructors.ipynb".
The first task is:

A file-checker

Write a program called check.py that takes the names of one or more inflammation data files as arguments and checks that all the files have the same number of rows and columns. What is the best way to test your program?

Can you write Python code for it?

from __future__ import division, print_function
import sys
import numpy as np

def main():
    script = sys.argv[0]
    filenames = sys.argv[1:]
    print filenames[0]
    shape0 = np.loadtxt(filenames[0]).shape
    for file in filenames[1:]:
        assert np.loadtxt(file).shape == shape0, 'Shape of {} does not match'.format(file)

main()

from __future__ import division, print_function
import sys
import numpy as np

def main():
    script = sys.argv[0]
    filenames = sys.argv[1:]
    shape0 = np.loadtxt(filenames[0], delimiter=',').shape
    for file in filenames[1:]:
        assert np.loadtxt(file, delimiter=',').shape == shape0, 'Shape of {} does not match'.format(file)

main()


I am working on the file "03-python-cmdline-instructors.ipynb".
The first task is:

Adding a default action

Separately, modify readings.py so that if no action is given it displays the means of the data.

Can you write Python code for it?

def main():
    script = sys.argv[0]
    if len(sys.argv) > 1:
        action = sys.argv[1]
    else:
        action = '--mean'
    filenames = sys.argv[2:]
    assert action in ['--min', '--mean', '--max'], \
           'Action is not one of --min, --mean, or --max: ' + action
    if len(filenames) == 0:
        process(sys.stdin, action)
    else:
        for f in filenames:
            process(f, action)


I am working on the file "03-python-cmdline-instructors.ipynb".
The first task is:

Changing flags

Rewrite readings.py so that it uses -n, -m, and -x instead of --min, --mean, and --max respectively. Is the code easier to read? Is the program easier to understand?

Adding a help message

Separately, modify readings.py so that if no parameters are given (i.e., no action is specified and no filenames are given), it prints a message explaining how it should be used.

Can you write Python code for it?

from __future__ import division, print_function
import sys

def main():
    if len(sys.argv) < 2:
        print("""This program should be called with an action and a filename or list of filenames, like so:
              $ python readings.py [action] [filename(s)]
              In the above, action is either '--min', '--mean', or '--max'.
              filename(s) is a filename or several filenames, or a file in standard input.""")
    script = sys.argv[0]
    action = sys.argv[1]
    filenames = sys.argv[2:]
    assert action in ['--min', '--mean', '--max'], \
           'Action is not one of --min, --mean, or --max: ' + action
    if len(filenames) == 0:
        process(sys.stdin, action)
    else:
        for f in filenames:
            process(f, action)

def process(filename, action):
    data = numpy.loadtxt(filename, delimiter=',')

    if action == '--min':
        values = data.min(axis=1)
    elif action == '--mean':
        values = data.mean(axis=1)
    elif action == '--max':
        values = data.max(axis=1)

    for m in values:
        print(m)

main()


I am working on the file "03-python-cmdline-instructors.ipynb".
The first task is:

Finding particular files

Using the glob module introduced earlier, write a simple version of ls that shows files in the current directory with a particular suffix. A call to this script should look like this:

$ python my_ls.py py

left.py
right.py
zero.py

Can you write Python code for it?

from __future__ import division, print_function
import sys
import glob

def main():
    suffix = sys.argv[1]
    files = glob.glob('*.' + suffix)
    for file in files:
        print(file)

main()


I am working on the file "03-python-cmdline-instructors.ipynb".
The first task is:
and run that:

The Right Way to Do It

If our programs can take complex parameters or multiple filenames, we shouldn’t handle sys.argv directly. Instead, we should use Python’s argparse library, which handles common cases in a systematic way, and also makes it easy for us to provide sensible error messages for our users.

Handling Multiple Files
The next step is to teach our program how to handle multiple files. Since 60 lines of output per file is a lot to page through, we’ll start by using three smaller files, each of which has three days of data for two patients:
Using small data files as input also allows us to check our results more easily: here, for example, we can see that our program is calculating the mean correctly for each line, whereas we were really taking it on faith before. This is yet another rule of programming: test the simple things first.
We want our program to process each file separately, so we need a loop that executes once for each filename. If we specify the files on the command line, the filenames will be in sys.argv, but we need to be careful: sys.argv[0] will always be the name of our script, rather than the name of a file. We also need to handle an unknown number of filenames, since our program could be run for any number of files.
The solution to both problems is to loop over the contents of sys.argv[1:]. The ‘1’ tells Python to start the slice at location 1, so the program’s name isn’t included; since we’ve left off the upper bound, the slice runs to the end of the list, and includes all the filenames. Here’s our changed program readings-03.py:
and here it is in action:

The Right Way to Do It

<div...
Can you write Python code for it?

from __future__ import division, print_function
import sys

def main():
    action = sys.argv[1]
    number1 = int(sys.argv[2])
    number2 = int(sys.argv[3])

    assert action in ['add', 'subtract']

    if action == 'add':
        print(number1 + number2)
    else:
        print(number1 - number2)

main()


I am working on the file "03-python2-defense-instructors.ipynb".
The first task is:
(Take a moment to think about why we use max to raise lowest and min to lower highest). We’d now like to re-run our tests, but they’re scattered across three different cells. To make running them easier, let’s put them all in a function:
Can you write Python code for it?


I am working on the file "03-python2-defense-instructors.ipynb".
The first task is:
Again, we get an error because we haven’t written our function, but we’re now ready to do so:
Can you write Python code for it?

def range_overlap(ranges):
    '''Return common overlap among a set of [low, high] ranges.'''
    lowest = 0.0
    highest = 1.0
    for (low, high) in ranges:
        lowest = max(lowest, low)
        highest = min(highest, high)
    return (lowest, highest)


I am working on the file "03-python2-defense-instructors.ipynb".
The first task is:
Programs like the Firefox browser are full of assertions: 10-20% of the code they contain are there to check that the other 80-90% are working correctly. Broadly speaking, assertions fall into three categories:

A precondition is something that must be true at the start of a function in order for it to work correctly.
A postcondition is something that the function guarantees is true when it finishes.
An invariant is something that is always true at a particular point inside a piece of code.

For example, suppose we are representing rectangles using a tuple of four coordinates (x0, y0, x1, y1), representing the lower left and upper right corners of the rectangle. In order to do some calculations, we need to normalize the rectangle so that the lower left corner is at the origin and the longest side is 1.0 units long. This function does that, but checks that its input is correctly formatted and that its result makes sense:
Can you write Python code for it?

def normalize_rectangle(rect):
    '''Normalizes a rectangle so that it is at the origin and 1.0 units long on its longest axis.'''
    # Some preconditions to catch invalid inputs
    assert len(rect) == 4, 'Rectangles must contain 4 coordinates'
    x0, y0, x1, y1 = rect
    assert x0 < x1, 'Invalid X coordinates'
    assert y0 < y1, 'Invalid Y coordinates'

    # Define width of rectangle in either direction
    dx = x1 - x0
    dy = y1 - y0
    # If x is longer, scale to x
    if dx > dy:
        scaled = dx / dy
        upper_x, upper_y = 1.0, scaled
    # Otherwise scale to y
    else:
        scaled = dx / dy
        upper_x, upper_y = scaled, 1.0

    # Reality check on results
    assert 0 < upper_x <= 1.0, 'Calculated upper X coordinate invalid'
    assert 0 < upper_y <= 1.0, 'Calculated upper Y coordinate invalid'

    return (0, 0, upper_x, upper_y)


I am working on the file "03-python2-defense-instructors.ipynb".
The first task is:
Re-reading our function, we realize that line 10 should divide dy by dx rather than dx by dy. (You can display line numbers by typing Ctrl-M, then L.) If we had left out the assertion at the end of the function, we would have created and returned something that had the right shape as a valid answer, but wasn’t. Detecting and debugging that would almost certainly have taken more time in the long run than writing the assertion.
But assertions aren’t just about catching errors: they also help people understand programs. Each assertion gives the person reading the program a chance to check (consciously or otherwise) that their understanding matches what the code is doing.
Most good programmers follow two rules when adding assertions to their code. The first is, fail early, fail often. The greater the distance between when and where an error occurs and when it’s noticed, the harder the error will be to debug, so good code catches mistakes as early as possible.
The second rule is, turn bugs into assertions or tests. Whenever you fix a bug, write an assertion that catches the mistake should you make it again. If you made a mistake in a piece of code, the odds are good that you have made other mistakes nearby, or will make the same mistake (or a related one) the next time you change it. Writing assertions to check that you haven’t regressed (i.e., haven’t re-introduced an old problem) can save a lot of time in the long run, and helps to warn people who are reading the code (including your future self) that this bit is tricky.
Test-Driven Development
An assertion checks that something is true at a particular point in the program. The next step is to check the overall behavior of a piece of code, i.e., to make sure that it produces the right output when it’s given a particular input. For example, suppose we need to find where two or more time series overlap. The range of each time series is represented as a pair of numbers, which are the time the interval started and ended. The output is the largest range that they all...
Can you write Python code for it?


I am working on the file "03-python2-defense-instructors.ipynb".
The first task is:
The error is actually reassuring: we haven’t written range_overlap yet, so if the tests passed, it would be a sign that someone else had and that we were accidentally using their function.
And as a bonus of writing these tests, we’ve implicitly defined what our input and output look like: we expect a list of pairs as input, and produce a single pair as output.
Something important is missing, though. We don’t have any tests for the case where the ranges don’t overlap at all:
What should range_overlap do in this case: fail with an error message, produce a special value like (0.0, 0.0) to signal that there’s no overlap, or something else? Any actual implementation of the function will do one of these things; writing the tests first helps us figure out which is best before we’re emotionally invested in whatever we happened to write before we realized there was an issue.
And what about this case?
Do two segments that touch at their endpoints overlap or not? Mathematicians usually say "yes", but engineers usually say "no". The best answer is "whatever is most useful in the rest of our program", but again, any actual implementation of range_overlap is going to do something, and whatever it is ought to be consistent with what it does when there’s no overlap at all.
Since we’re planning to use the range this function returns as the X axis in a time series chart, we decide that:

every overlap has to have non-zero width, and
we will return the special value None when there’s no overlap.

None is built into Python, and means “nothing here”. (Other languages often call the equivalent value null or nil). With that decision made, we can finish writing our last two tests:
Can you write Python code for it?


I am working on the file "03-python2-defense-instructors.ipynb".
The first task is:
The first test that was supposed to produce None fails, so we know something is wrong with our function. We don’t know whether the other tests passed or failed because Python halted the program as soon as it spotted the first error. Still, some information is better than none, and if we trace the behavior of the function with that input, we realize that we’re initializing lowest and highest to 0.0 and 1.0 respectively, regardless of the input values. This violates another important rule of programming: always initialize from data.

Pre- and post-conditions

Suppose you are writing a function called average that calculates the average of the numbers in a list. What pre-conditions and post-conditions would you write for it? Compare your answer to your neighbor’s: can you think of a function that will pass your tests but not hers or vice versa?

Testing assertions

Given a sequence of values, the function running returns a list containing the running totals at each index.

running([1, 2, 3, 4])

[1, 3, 6, 10]

running('abc')

['a', 'ab', 'abc']

Explain in words what the assertions in this function check, and for each one, give an example of input that will make that assertion fail.

def running(values):
    <span...
Can you write Python code for it?

# assert len(values) > 0 tests that you haven't passed an empty sequence in
running([])
running ('')

# assert result[-1] >= 0 checks that the last item in the results list is greater than or equal to 0
running([-1, 0, 1])

# assert result[-1] >= result[0] checks that the last item in the results list is greater than or equal to the first item
running([1, -1, 0])


I am working on the file "03-python2-defense-instructors.ipynb".
The first task is:
The post-conditions help us catch bugs by telling us when our calculations cannot have been correct. For example, if we normalize a rectangle that is taller than it is wide everything seems OK:
Can you write Python code for it?

print(normalize_rectangle( (0.0, 0.0, 1.0, 5.0) ))


I am working on the file "03-python2-defense-instructors.ipynb".
The first task is:
The preconditions on lines 2, 4, and 5 catch invalid inputs:
Can you write Python code for it?

print(normalize_rectangle( (0.0, 1.0, 2.0) )) # missing the fourth coordinate

print(normalize_rectangle( (4.0, 2.0, 1.0, 5.0) )) # X axis inverted