I am working on the file "0209_CrossValidation.ipynb".
The first task is:
Here's a short overview of the discussed functions and how they relate to each other:
|No. of Categories | Applicable Functions |
|:----------------:|:---------------------|
| 2 | pd.factorize(), sklearn.preprocessing.LabelEncoder() |
| > 2 | pd.get_dummies(), sklearn.preprocessing.OneHotEncoder()|

Incorrect class labeling is one of the more common mistake made in machine learning. The crux of the matter is that even if we forget to One-Hot-Encode our features, ML algorithms might still yield good results. Yet, as we know now, such results would be flawed. Therefore it is important to correctly preprocess the data set before applying any ML algorithms.

Partitioning a Data Set Into Train, Test Sets
In section 4.5 of the script we have briefly touched upon the importance of randomly splitting our data into a training set (to train/calibrate our ML algorithms) and a test set (to evaluate the accuracy of our model). As Pedregosa et al. (2011) put it in the scikit-learn documentation: "Learning the parameters of a prediction function and testing it on the same data is a methodological mistake: a model that would just repeat the labels of the samples that it has just seen would have a perfect score but would fail to predict anything useful on yet-unseen data. This situation is called overfitting. To avoid it, it is common practice when performing a (supervised) machine learning experiment to hold out part of the available data as a test set X_test, y_test." The split ratio is somewhat arbitrary but literature suggests a range between 60:40 (train:test) and 80:20 for smaller samples and 90:10 to 99:1 for large sets (several thousands of observations) (Raschka (2015)).
In its model_selection submodule scikit-learn offers a convenient function called train_test_split that randomly splits a data set into separate train and holdout sets. To show how this function is applied we first load the publicly available 'adult' data set. It includes 14 features from the 1994 US census that measure an individual's...
Can you write Python code for it?

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split

df = pd.read_csv('Data/adult.csv', sep=',')
df.head(3)


I am working on the file "0209_CrossValidation.ipynb".
The first task is:
It is important to point out that we fit the MinMaxScaler only once on the training data, not the test data. On the test data we only run the .transform() method. The parameters we get out of the training set (i.e. mms.data_min_ and mms.data_max_) are then used to transform any new data (such as the holdout set).

Why aren't we applying the scaling on the full data set, before we split the data into train and test set? This would not be a good idea because of one particular reason: we would "contaminate" our model with information about the test set. If we were to do it we run the risk of having a very good model for our train AND test data but this model would perform poorly on new data that previously was not part of any train or test set. The effect of such an approach would be poor generalization. By the way, the same is true for feature selection. This too should not be done on the full set but only on a training set.

Standardization
When we need values in a bounded interval, normalization via min-max scaling is a useful technique. Yet for many machine learning algorithms standardization is a more practical approach. The rationale behind it is that most linear models (such as logistic regression, LDA or support vector machines) initialize coefficients/parameters/weights to small random values close to 0. By standardizing our features, we center the values at mean 0 with standard deviation of 1. This makes the computations easier to learn the coefficients/parameters/weights. Beyond that, standardization does not distort useful statistical information about outliers and with that makes the algorithm less sensitive to them in contrast to min-max scaling (Raschka (2015)). We express the process of standarizing a feature column by the following equation:
$$\begin{equation}
X_{i}^{\text{std}} = \frac{X_i - \bar{X}i}{\sigma{X_i}}
\end{equation}$$
Here we follow the common notation that $\bar{X}i$ is the mean of vector $X_i$ and $\sigma{X_i}$ the vector's standard deviation.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler

# Apply StandardScaler on continuous columns only
stdsc = StandardScaler()
X_train_std = stdsc.fit_transform(X_train[cols_scl])  # fit & transform
X_test_std  = stdsc.transform(X_test[cols_scl])  # ONLY transform


I am working on the file "0209_CrossValidation.ipynb".
The first task is:
LabelEncoder()
The alternatives in Scikit-learn are called LabelEncoder (equivalent to pd.facorize()) and OneHotEncoder() (similar to pd.get_dummies()). Their setup is a bit more abstract and cumbersome, but it also has advantages compared to Pandas' solution. For example the OneHotEncoder() can be used in Scikit-learn pipelines and it returns a sparse matrix which is highly efficient computationally. But for the sake of brefity we will not go into too much detail here but simply show for reference how these Scikit-learn functions are applied. Ultimately it is at this stage a question of preference what functions you want to use for your task. Important is just that this step is done at the very beginning of the process - even before a train-test split is applied. That way you make sure the mapping is the same for both train and test split.
Can you write Python code for it?

from sklearn import preprocessing as pp

# Factorize 'fab' column (similar to pd.factorize())
le = pp.LabelEncoder()
data_le = le.fit_transform(data['fab'])
data_le


I am working on the file "0209_CrossValidation.ipynb".
The first task is:
Noteworthy is one last function: cross_validate. This allows us to define a list of measures that we can pass on to the function as scoring parameter. The output is a dictionary with fit_time and score_time (time elapsed to fit model/calculate scores), test_accuracy and train_accuracy (accuracy for the test set and training set, the output on the latter though only if return_train_score is manually set to True), etc.
Can you write Python code for it?

from sklearn.model_selection import cross_validate

# Calculate return
measures = ['accuracy', 'recall', 'roc_auc']
scores = cross_validate(logReg, X_train_std, y_train, cv=kFold,
                        scoring=measures, return_train_score=True, n_jobs=2)
scores

print('Train set accuracy (CV=5):    ', scores['train_accuracy'].mean())
print('Validation set scores (CV=5): ', scores['test_accuracy'].mean())
print('Test set accuracy:            ', logReg.fit(X_test_std, y_test).score(X_test_std, y_test))


I am working on the file "0209_CrossValidation.ipynb".
The first task is:
Now we are ready to perform the train-test split. For this example we use a test set size of 30%. The parameter random_state=0 fixes the random split in a way such that results are reproducible.
Can you write Python code for it?

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)


I am working on the file "0209_CrossValidation.ipynb".
The first task is:
Only the column 'fab', which contained strings, was converted. To make get_dummies() bend to our will, we have to convert the district values into strings first. Then we can apply the conversion.
Can you write Python code for it?

data['dist'] = data['dist'].astype(str)
pd.get_dummies(data)


I am working on the file "0209_CrossValidation.ipynb".
The first task is:
The first six columns are numeric values. The remaining columns are categorical information and thus need to be converted.
Can you write Python code for it?

# Get column names
cols = df.columns.values[6:]

# Factorize 'sex' and 'income' column (both binary)
df[cols[-2:]] = df[cols[-2:]].apply(lambda x: pd.factorize(x)[0])
df.head(3)

# Assign response to y
y = df[cols[-1]]

# Factorize categorical values, assign output to X
X = pd.get_dummies(df.iloc[:, :-1])
X.head()

X.shape


I am working on the file "0209_CrossValidation.ipynb".
The first task is:
The output here is a sparse matrix. Computationally and from a data storage perspective this is highly efficient because the matrix contains only integers as entries. However, the print-out of this sparse matrix is less indicative. All column labels are empty. To make sense of it we have to understand the setup of this sparse matrix. Each column is a binary representation of a value. The first column indicates which of the rows had a dist=12 (only row index 2), second dist=2 (only row index 1), third dist=5, fourth dist=9, fifth fab=bricks etc.
Beyond the convoluted output a further problem of the above approach is that this still leaves us with the task to combine numeric ('price', 'rooms') and categorical output (the aforementioned sparse matrix) into one feature matrix X. To simplify this task, there is a wrapper function ColumnTransformer that facilitates this in a simple way.
Can you write Python code for it?

from sklearn.compose import ColumnTransformer

# Intatiate OneHotEncoder
ohe = pp.OneHotEncoder()

# create list of columns that should be transformed,
# i.e. columns 2 and 3 are categorical and should be
# one-hot-encoded
trnsfrms_list = [('cat', ohe, [2, 3])]

# Make use of the ColumnTransformer function,
# fit/transpose df and display output
trnsfrms = ColumnTransformer(transformers=trnsfrms_list, remainder='passthrough')
X_ohe = trnsfrms.fit_transform(data)
X_ohe


I am working on the file "0209_CrossValidation.ipynb".
The first task is:
Well, if we would follow through with this approach and feed these values into a Scikit-learn ML function, the model would make the fundamental assumption that bricks > concrete > wood. Furthermore, we have numerical values for dist (the districts of Zurich) where we know precisely that these represent categorical values (2 = Wollishofen, 5 = Industrie, 9 = Altstetten, 12 = Schwamendingen). And this - geographic or demographic jokes aside - would not make much sense (VanderPlas (2016)). Houston we've got a problem!

NOTE:
* In the binary case with $X_i \in {0, 1}$ ordering is not an issue in scikit-learn.
* Ordering is relevant for features. Class labels for response $y$ are not ordinal for the ML algorithm we discuss in this course. Therefore it doesn't matter what number we assign them, and factorization is still valid (e.g. $y \in {\text{Product X}=0, \text{Product Y}=1, \text{Product Z}=2}$.

As soon as we have more than 2 categories, factorizing is in most cases no longer the appropriate solution. Instead, we should make use of a method called one-hot encoding, which effectively creates extra columns with dummy variables (booleans) indicating the presence of a category with a value of 1 or 0, respectively. There are numerous ways of encoding categorical values. In this chapter we will briefly touch upon Pandas' and Scikit-Learn's tools. For those looking for a thorough tutorial please refer to Chris Moffit's (2017) *Guide to Encoding Categorical Values in Python *.
get_dummies()
We start with Pandas' pd.get_dummies() function, which works of course seamlessly with DataFrames and is truly easy to work with. The two caveats it has is that (a) if a column contains numbers, it will not transform them into categorical values and (b) this approach does not work in scikit-learn piepelines (Pipelines are a ML workflow management tool that we will discuss in more detail in later chapters.). Regarding (a) here's what is meant:
Can you write Python code for it?


I am working on the file "0211_SVM.ipynb".
The first task is:
Applying SVM
Having assigned the data to X and y we are now ready to divide the dataset into separate training and test sets.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y,
                                                    test_size=0.2,
                                                    random_state=0,
                                                    stratify=y)


I am working on the file "0211_SVM.ipynb".
The first task is:
As for the other missing values we are left to decide whether we want to remove the corresponding observations (rows) or apply a filling method. The problem with dropping all rows with missing values is that we might lose a lot of valuable information. Therefore in this case we prefer to use a common interpolation technique and impute NaN values with the corresponging feature mean. Alternatively we could use 'median' or 'most_frequent' as strategy. A convenient way to achieve this imputation is to use the Imputer class from sklearn.
Notice that as of sklearn version 0.20.2 Scikit-learn has relabeled function Imputer we use below to SimpleImputer(). Furthermore, with version 0.22.1 other imputers were introduced, such as a KNNImputer or a (yet still experimental) IterativeImputer. It is up to the reader to familiarize her-/himself with the available options for imputing. See Scikit-learn's guide for details. To check for the Sklearn version you currently run hit !pip list in your shell.
Can you write Python code for it?

from sklearn.impute import SimpleImputer

# Impute missing values by mean (axis=0 --> along columns;
# Notice that argument 'axis=' has been removed as of version 0.20.2)
ipr = SimpleImputer(missing_values=np.nan, strategy='mean')
ipr = ipr.fit(df.values)
imputed_data = ipr.transform(df.values)

# Assign imputed values to 'df' and check for 'NaN' values
df = pd.DataFrame(imputed_data, columns=df.columns)
df.isnull().sum().sum()


I am working on the file "0211_SVM.ipynb".
The first task is:
As it looks linear and polynomial SVM yield similar results. What is clearly unsatisfactory is the number of true defaults that the SVM missed to detect. Both linear as well as non linear SVM miss to label $\geq$ 80 defaults [sic]. From a financial perspective, this is unacceptable and raises questions regarding
* Class imbalance
* Hyperparameter fine-tuning through cross validation and grid search
* Feature selection
* Noise & dimension reduction
which we want to address in the next section.
Dealing with Class Imbalance
When we deal with default data sets we observe that the ratio of non-default to default records is heavily skewed towards non-default. This is a common problem in real-world data set: Samples from one class or multiple classes dominate the data set. For the present data set we are talking 93% non-defaults vs. 7% defaults. Having an algorithm that predicts non-default 100 out of a 100 times is right in 93% of the cases. Therefore, training a model on such a data set that achieves the same 93% test accuracy (as our SVM above) means nothing else than our model hasn't learned anything informative from the features provided in this data set. Thus, when assessing a classifier on an imbalanced data set we have learned that other metrics such as precision, recall, ROC curve etc. might be more informative.
Having said that, what we have to consider is that a class imbalance might influences a learning algorithm during the model fitting itself. Machine learning algorithms typically optimize a reward or cost function. This means that an algorithm implicitly learns the model that optimizes the predictions based on the most abundant class in the dataset in order to minimize the cost or maximize the reward during the training phase. And this in turn might yield skewed results in case of imbalanced data sets.
There are several options to deal with class imbalance, we will discuss two of them. The first option is to set the class_weight parameter to class_weight='balanced'. Most classifier hae this option...
Can you write Python code for it?

# Initiate and fit a polynomial SVM to training set
svm_poly = SVC(kernel='poly', random_state=1, class_weight='balanced')
svm_poly.fit(X_train_std, y_train)

# Predict classes and print results
y_pred = svm_poly.predict(X_test_std)
print(metrics.classification_report(y_test, y_pred))
print(metrics.confusion_matrix(y_test, y_pred))
print("Test score: {:.2f}".format(svm_poly.score(X_test_std, y_test)))


I am working on the file "0211_SVM.ipynb".
The first task is:
Downsampling works in similar fashion.
Can you write Python code for it?

# Downsampling
X_dnsampled, y_dnsampled = resample(X[y==0], y[y==0],
                                    replace=False,
                                    n_samples=X[y==1].shape[0],
                                    random_state=1)


I am working on the file "0211_SVM.ipynb".
The first task is:
From the above output we see that the SVC yields the best accuracy.
Feature Selection and Dimensionality Reduction
Complexity and the Curse of Overfitting
If we observe that a model performs much better on training than on test data, we have an indication that the model suffers from overfitting. The reason for the overfitting is most probably that our model is too complex for the given training data. Common solutions to reduce the generalization error are (Raschka (2015)):
* Collect more (training) data
* Introduce a penalty for complexity via regularization
* Choose a simpler model with fewer parameter
* Reduce the dimensionality of the data
Collecting more data is self explanatory but often not applicable. Regularization via a complexity penalty term is a technique that is primarily applicable to regression settings (e.g. logistic regression). We will not discuss it here but the interested reader will easily find helpful information in e.g. James et al. (2013) chapter 6 or Raschka (2015) chapter 4. Here we will look at one commonly used solution to reduce overfitting: dimensionality reduction via feature selection.
Feature Selection
A useful approach to select relevant features from a data set is to use information from the random forest algorithm we introduced in the previous chapter. There we elaborated how decision trees rank the feature importance based on a impurity decrease. Conveniently, we can access this feature importance rank directly from the RandomForestClassifier object. By executing below code - following the example in Raschka (2015) - we will train a random forest model on the balanced default data set (from before) and rank the features by their respective importance measure.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

# Extract feature labels
feat_labels = df.columns[:-1]

# Create Random Forest object, fit data and
# extract feature importance attributes
forest = RandomForestClassifier(random_state=1)
forest.fit(X_train_bal, y_train_bal)
importances = forest.feature_importances_

# Sort output (by relative importance) and
# print top 15 features
indices = np.argsort(importances)[::-1]
n = 15
for i in range(n):
    print('{0:2d}) {1:7s} {2:6.4f}'.format(i + 1,
                                           feat_labels[indices[i]],
                                           importances[indices[i]]))


I am working on the file "0211_SVM.ipynb".
The first task is:
In the same way we can run a Kernel SVM on the data. We have four Kernel options: one linear as introduced above and three non-linear. All of them have hyperparameter available. If these are not specified, default values are taken. Check the documentation for details.

linear: linear SVM as shown above with C as hyperparameter
rbf: Radial basis function Kernel with C, gamma as hyperparameter
poly: Polynomial Kernel with C, degree, gamma, coef0 as hyperparameter
sigmoid: Sigmoid Kernel with C, gamma, coef0 as hyperparameter

Let us apply a polynomial Kernel as example.
Can you write Python code for it?

svm_poly = SVC(kernel='poly', random_state=1)
svm_poly


I am working on the file "0211_SVM.ipynb".
The first task is:
Next we define a parameter grid to search over and construct a GridSearchCV from the pipeline and the parameter grid. Notice that we have to specify for each parameter which step of the pipeline it belongs to. This is done by calling the name we gave this step, followed by a double underscore and the parameter name. For the present example, let us compare different degrees, and C values.
Can you write Python code for it?

# Define parameter grid
param_grid = {'svm_poly__C': [0.1, 1, 10, 100],
              'svm_poly__degree': [1, 2, 3, 5, 7]}


I am working on the file "0211_SVM.ipynb".
The first task is:
Not having specified hyperparameter C, degree, gamma, and coef0 we see that the algorithm has taken default values. For C it is equal to 1, default degree is 3, gamma=auto means that the value will be calculated as $1/n_{\text{features}}$, and coef0 is set to 0 as default.
Can you write Python code for it?

# Fit polynomial SVM to standardized training set
svm_poly.fit(X_train_std, y_train)

# Print results
print("Observed probability of non-default: {:.2f}".format(np.count_nonzero(y_train==0) / len(y_train)))
print("Train score: {:.2f}".format(svm_poly.score(X_train_std, y_train)))
print("Test score:  {:.2f}".format(svm_poly.score(X_test_std, y_test)))

# Predict classes
y_pred = svm_poly.predict(X_test_std)

# Manual confusion matrix as pandas DataFrame
confm = pd.DataFrame({'Predicted': y_pred,
                      'True': y_test})
confm.replace(to_replace={0:'Non-Default', 1:'Default'}, inplace=True)
print(confm.groupby(['True','Predicted'], sort=False).size().unstack('Predicted'))


I am working on the file "0211_SVM.ipynb".
The first task is:
Running the SVM algorighm on the balanced dataset works now as you would expect:
Can you write Python code for it?

# Combine datasets
X_bal = np.vstack((X[y==1], X_dnsampled))
y_bal = np.hstack((y[y==1], y_dnsampled))

# Train test split
X_train_bal, X_test_bal, y_train_bal, y_test_bal = \
    train_test_split(X_bal, y_bal,
                     test_size=0.2,
                     random_state=0,
                     stratify=y_bal)

# Standardize features; equal results as if done in two
# separate steps (first .fit() and then .transform())
X_train_bal_std = sc.fit_transform(X_train_bal)

# Transform test set
X_test_bal_std = sc.transform(X_test_bal)

# Initiate and fit a polynomial SVM to training set
svm_poly_bal = SVC(kernel='poly', random_state=1)
svm_poly_bal.fit(X_train_bal_std, y_train_bal)

# Predict classes and print results
y_pred_bal = svm_poly_bal.predict(X_test_bal_std)
print(metrics.classification_report(y_test_bal, y_pred_bal))
print(metrics.confusion_matrix(y_test_bal, y_pred_bal))
print("Test score: {:.2f}".format(svm_poly_bal.score(X_test_bal_std, y_test_bal)))


I am working on the file "0211_SVM.ipynb".
The first task is:
The second option we want to discuss is up- & downsampling of the minority/majority class. Both up- and downsampling are implemented in Scikit-learn through the resample function and depending on the data and given the task at hand, one might be better suited than the other. For the upsampling, scikit-learn will apply a bootstrapping to draw new samples from the datasets with replacement. This means that the function will repeatedly draw new samples from the minority class until it contains the number of samples we define. Here's a code example:
Can you write Python code for it?

pd.DataFrame(X[y==1]).shape

X[y==0].shape

from sklearn.utils import resample

# Upsampling: define which rows you want to upsample
# (i.e. all columns of X where value in corresponding y vector is equal to 1: X[y==1],
# and similar for y[y==1]. Then define how many samples should be generated through
# bootstrapping (here: X[y==0].shape[0] = 5'500))
X_upsampled, y_upsampled = resample(X[y==1], y[y==1],
                                    replace=True,
                                    n_samples=X[y==0].shape[0],
                                    random_state=1)
print('No. of default samples BEFORE upsampling:  {:.0f}'.format(y.sum()))
print('No. of default samples AFTER upsampling:  {:.0f}'.format(y_upsampled.sum()))


I am working on the file "0211_SVM.ipynb".
The first task is:
The value in decimal is the relative importance for the respective feature. We can also plot this result to have a better overview. Below code shows one way of doing it.
Can you write Python code for it?

# Get cumsum of the n most important features
feat_imp = np.sort(importances)[::-1]
sum_feat_imp = np.cumsum(feat_imp)[:n]

# Plot Feature Importance (both cumul., individual)
plt.figure(figsize=(12, 8))
plt.bar(range(n), importances[indices[:n]], align='center')
plt.xticks(range(n), feat_labels[indices[:n]], rotation=90)
plt.xlim([-1, n])
plt.xlabel('Feature')
plt.ylabel('Rel. Feature Importance')
plt.step(range(n), sum_feat_imp, where='mid',
         label='Cumulative importance')
plt.tight_layout();


I am working on the file "0211_SVM.ipynb".
The first task is:
To close, one last code snippet is provided. Running it will visualize the cumulative explained variance ratio as a function of the number of components. (Mathematically, the explained variance ratio is the ratio of the eigenvalue of principal component $i$ to the sum of the eigenvalues, $\frac{\lambda_i}{\sum_{i}^p \lambda_i}$. See the appendix in the script to better understand the meaning of eigenvalues in this context.) In practice, this might be helpful in deciding on the number of principal components $q$ to use.
Can you write Python code for it?

# Run PCA for all possible PCs
pca = PCA().fit(X_train_bal)

# Define max no. of PC
q = X_train_bal.shape[1]

# Get cumsum of the PC 1-q
expl_var = pca.explained_variance_ratio_
sum_expl_var = np.cumsum(expl_var)[:q]

# Plot Feature Importance (both cumul., individual)
plt.figure(figsize=(12, 6))
plt.bar(range(1, q + 1), expl_var, align='center')
plt.xticks(range(1, q + 1, 5))
plt.xlim([0, q + 1])
plt.xlabel('Principal Components')
plt.ylabel('Explained Variance Ratio')
plt.step(range(1, 1 + q), sum_expl_var, where='mid')
plt.tight_layout();


I am working on the file "0211_SVM.ipynb".
The first task is:
Unlike e.g. decision tree algorithms SVM are sensitive to the magnitude the data. Therefore scaling our data is recommended.
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler

# Create StandardScaler object
sc = StandardScaler()

# Standardize features; equal results as if done in two
# separate steps (first .fit() and then .transform())
X_train_std = sc.fit_transform(X_train)

# Transform test set
X_test_std = sc.transform(X_test)


I am working on the file "0211_SVM.ipynb".
The first task is:
With that we can run a GridSearchCV as usual.
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV

# Run grid search
grid = GridSearchCV(pipe, param_grid=param_grid, cv=5, n_jobs=-1)
grid.fit(X_train_bal, y_train_bal)

# Print results
print('Best CV accuracy: {:.2f}'.format(grid.best_score_))
print('Test score:       {:.2f}'.format(grid.score(X_test_bal, y_test_bal)))
print('Best parameters: {}'.format(grid.best_params_))


I am working on the file "0211_SVM.ipynb".
The first task is:
With the data standardized, we can finally apply a SVM on the data. We import the SVC (for Support Vector Classifier) from the Scikit-learn toolbox and create a svm_linear object that represents a linear SVM with C=1. Recall that C helps us control the penalty for misclassification. Large values of C correspond to large error penalties and vice-versa. More parameter can be specified. Details are best explained in the function's documentation page.
Can you write Python code for it?

from sklearn.svm import SVC
from sklearn import metrics
import matplotlib.pyplot as plt

# Create object
svm_linear = SVC(kernel='linear', C=1.0)
svm_linear


I am working on the file "02A_TensorFlow-Slim.ipynb".
The first task is:
Importantly, notice that we didn't have to retrain the model. Instead, the graph and all variable values were loaded directly from our checkpoint files. In this example, this probably takes just as long, but for more complex models, the utility of saving/restoring is immense.
TF-Slim Model Zoo
One of the biggest and most surprising unintended benefits of the ImageNet competition was deep networks' transfer learning properties: CNNs trained on ImageNet classification could be re-used as general purpose feature extractors for other tasks, such as object detection. Training on ImageNet is very intensive and expensive in both time and computation, and requires a good deal of set-up. As such, the availability of weights already pre-trained on ImageNet has significantly accelerated and democratized deep learning research.
Pre-trained models of several famous architectures are listed in the TF Slim portion of the TensorFlow repository. Also included are the papers that proposed them and their respective performances on ImageNet. Side note: remember though that accuracy is not the only consideration when picking a network; memory and speed are important to keep in mind as well.
Each entry has a link that allows you to download the checkpoint file of the pre-trained network. Alternatively, you can download the weights as part of your program. A tutorial can be found here, but the general idea:
Can you write Python code for it?

from datasets import dataset_utils
import tensorflow as tf

url = "http://download.tensorflow.org/models/vgg_16_2016_08_28.tar.gz"
checkpoints_dir = './checkpoints'

if not tf.gfile.Exists(checkpoints_dir):
    tf.gfile.MakeDirs(checkpoints_dir)

dataset_utils.download_and_uncompress_tarball(url, checkpoints_dir)

import os
import tensorflow as tf
from nets import vgg

slim = tf.contrib.slim

# Load images
images = ...

# Pre-process
processed_images = ...

# Create the model, use the default arg scope to configure the batch norm parameters.
with slim.arg_scope(vgg.vgg_arg_scope()):
    logits, _ = vgg.vgg_16(processed_images, num_classes=1000, is_training=False)

probabilities = tf.nn.softmax(logits)

# Load checkpoint values
init_fn = slim.assign_from_checkpoint_fn(
    os.path.join(checkpoints_dir, 'vgg_16.ckpt'),
    slim.get_model_variables('vgg_16'))


I am working on the file "02A_TensorFlow-Slim.ipynb".
The first task is:
Understanding every line of this model isn't important. The main point to notice is how much space this takes up. Several of the above lines (conv2d, bias_add, relu, maxpool) can obviously be combined to cut down on the size a bit, and you could also try to compress the code with some clever for looping, but all at the cost of sacrificing readability. With this much code, there is high potential for bugs or typos (to be honest, there are probably a few up there^), and modifying or refactoring the code becomes a huge pain.
By the way, although VGG-16's paper was titled "Very Deep Convolutional Networks for Large-Scale Image Recognition", it isn't even considered a particularly deep network by today's standards. Residual Networks (2015) started beating state-of-the-art results with 50, 101, and 152 layers in their first incarnation, before really going off the deep end and getting up to 1001 layers and beyond. I'll spare you from me typing out the uncompressed TensorFlow Core code for that.
TF-Slim
Enter TF-Slim. The same VGG-16 model can be expressed as follows:
Can you write Python code for it?

import tensorflow as tf

slim = tf.contrib.slim

# Set up the data loading:
images, labels = ...

# Define the model:
with slim.arg_scope([slim.conv2d, slim.fully_connected],
                    activation_fn=tf.nn.relu,
                    weights_initializer=tf.truncated_normal_initializer(0.0, 0.01),
                    weights_regularizer=slim.l2_regularizer(0.0005)):
    net = slim.repeat(images, 2, slim.conv2d, 64, [3, 3], scope='conv1')
    net = slim.max_pool2d(net, [2, 2], scope='pool1')
    net = slim.repeat(net, 2, slim.conv2d, 128, [3, 3], scope='conv2')
    net = slim.max_pool2d(net, [2, 2], scope='pool2')
    net = slim.repeat(net, 3, slim.conv2d, 256, [3, 3], scope='conv3')
    net = slim.max_pool2d(net, [2, 2], scope='pool3')
    net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv4')
    net = slim.max_pool2d(net, [2, 2], scope='pool4')
    net = slim.repeat(net, 3, slim.conv2d, 512, [3, 3], scope='conv5')
    net = slim.max_pool2d(net, [2, 2], scope='pool5')
    net = slim.fully_connected(net, 4096, scope='fc6')
    net = slim.dropout(net, 0.5, scope='dropout6')
    net = slim.fully_connected(net, 4096, scope='fc7')
    net = slim.dropout(net, 0.5, scope='dropout7')
    net = slim.fully_connected(net, 1000, activation_fn=None, scope='fc8')

predictions = net


I am working on the file "02_Analysis_of_Twitter_Social_Network.ipynb".
The first task is:
Exercise: do the same for the Graph comprising only retweet, replies, quote and mentions
Percolation of the Giant Component
Can you write Python code for it?

import random

def getGCsize(G):
    """ returns the size of the largest component of G"""

    comps = nx.connected_components(G)
    return max([len(comp) for comp in comps])


I am working on the file "02_Analysis_of_Twitter_Social_Network.ipynb".
The first task is:
Exercise: implement the High-Degree Adaptative (HDA) attack where at each step the node with the highest degree of the remaining graph is removed.
Save the graph to a GraphML file:
GraphML is file format based on XML useful for exchanging files between softwares.
http://graphml.graphdrawing.org/
Can you write Python code for it?


I am working on the file "02_Analysis_of_Twitter_Social_Network.ipynb".
The first task is:
Let's build the network
Can you write Python code for it?

import networkx as nx

# define an empty Directed Graph
# A directed graph is a graph where edges have a direction
# in our case the edges goes from user that sent the tweet to
# the user with whom they interacted (retweeted, mentioned or quoted)
G = nx.DiGraph()

# loop over all the tweets and add edges if the tweet include some interactions
for tweet in tweet_list:
    # find all influencers in the tweet
    tweeter, interactions = getAllInteractions(tweet)
    tweeter_id, tweeter_name = tweeter

    # add an edge to the Graph for each influencer
    for interaction in interactions:
        interact_id, interact_name = interaction

        # add edges between the two user ids
        # this will create new nodes if the nodes are not already in the network
        G.add_edge(tweeter_id, interact_id)

        # add name as a property to each node
        # with networkX each node is a dictionary
        G.node[tweeter_id]['name'] = tweeter_name
        G.node[interact_id]['name'] = interact_name

# The graph's node are contained in a dictionary
print(type(G.node))

#print(G.node.keys())
# the keys are the user_id
print(G.node[tweeter_id])

# each node is itself a dictionary with node attributes as key,value pairs
print(type(G.node[tweeter_id]))

# edges are also contained in a dictionary
print(type(G.edge))

# we can see all the edges going out of this node
# each edge is a dictionary inside this dictionary with a key
# corresponding to the target user_id
print(G.edge[tweeter_id])

# so we can access the edge using the source user_id and the target user_id
G.edge[tweeter_id][interact_id]


I am working on the file "02_Analysis_of_Twitter_Social_Network.ipynb".
The first task is:
Let's look at the informations contained in a tweet
Can you write Python code for it?

# take the first tweet of the list
tweet = tweet_list[2]

# each tweet is a python dictionary
type(tweet)

# all the 'entries' of the dictionary
tweet.keys()


I am working on the file "02_Analysis_of_Twitter_Social_Network.ipynb".
The first task is:
Network components
For directed graphs we can define two types of components:
- Weakly connected components
- Strongly connected components
Weakly connected component (WCC): maximal set of nodes where there exists a path in at least one direction between each pair of nodes.
Strongly connected component (SCC): maximal set of nodes where there exists a path in both directions between each pair of nodes.
Weakly connected giant (largest) component (WCGC): Largest WCC
Strongly connected giant (largest) component (SCGC): Largest SCC
<img src="network_components.svg" style="width: 250px;"/>
Can you write Python code for it?

# this returns a list of set of nodes belonging to the
# different (weakly) connected components
components = list(nx.weakly_connected_components(G))

# sort the component according to their size
components = list(sorted(components, key=lambda x:len(x), reverse=True))

# make a list with the size of each component
comp_sizes = []
for comp in components:
    comp_sizes.append(len(comp))

# plot the histogram of component sizes
hist = plt.hist(comp_sizes, bins=100)

# histogram with logarithmic y scale
hist = plt.hist(comp_sizes, bins=100, log=True)
plt.xlabel('component size')
plt.ylabel('number of components')

# sizes of the ten largest components
comp_sizes[:10]

# let's make a new graph which is the subgraph of G corresponding to
# the largest connected component
# let's find the largest component
largest_comp = components[0]
LCC = G.subgraph(largest_comp)

G.number_of_nodes()

LCC.number_of_nodes()

# let's plot the degree distribution inside the LCC
degrees = nx.degree(LCC)
degrees

degree_array = np.array(list(degrees.values()))
hist = plt.hist(degree_array, bins=100)

# using logarithmic scales
hist = plt.hist(degree_array, bins=100, log=True)
plt.xscale('log')

# logarithmic scale with logarithmic bins
N, bins, patches = plt.hist(degree_array, bins=np.logspace(0,np.log10(degree_array.max()+1), 20), log=True)
plt.xscale('log')
plt.xlabel('k - degree')
plt.ylabel('number of nodes')

# Degree probability distribution (P(k))

# since we have logarithmic bins, we need to
# take into account the fact that the bins
# have different lenghts when normalizing
bin_lengths = np.diff(bins) # lenght of each bin

summ = np.sum(N*bin_lengths)
normalized_degree_dist = N/summ

# check normalization:
print(np.sum(normalized_degree_dist*bin_lengths))

hist = plt.bar(bins[:-1], normalized_degree_dist, width=np.diff(bins))
plt.xscale('log')
plt.yscale('log')
plt.xlabel('k (degree)')
plt.ylabel('P(k)')


I am working on the file "02_Analysis_of_Twitter_Social_Network.ipynb".
The first task is:
Some basic properties of the Network:
Can you write Python code for it?

G.number_of_nodes()

G.number_of_edges()

# listing all nodes
node_list = G.nodes()
node_list[:3]

# degree of a node
print(G.degree(node_list[2]))
print(G.in_degree(node_list[2]))
print(G.out_degree(node_list[2]))

# dictionary with the degree of all nodes
all_degrees = G.degree(node_list) # this is the degree for undirected edges
in_degrees = G.in_degree(node_list)
out_degrees = G.in_degree(node_list)

# average degree
2*G.number_of_edges()/G.number_of_nodes()

import numpy as np
np.array(list(all_degrees.values())).mean()

np.array(list(in_degrees.values())).mean()

np.array(list(out_degrees.values())).mean()

# maximum degree
max(all_degrees.values())

# we want to make a list with (user_id, username, degree) for all nodes
degree_node_list = []
for node in G.nodes_iter():
    degree_node_list.append((node, G.node[node]['name'], G.degree(node)))

print('Unordered user, degree list')
print(degree_node_list[:10])

# sort the list according the degree in descinding order
degree_node_list = sorted(degree_node_list, key=lambda x:x[2], reverse=True)
print('Ordered user, degree list')
print(degree_node_list[:10])

# we need to import matplolib for making plots
# and numpy for numerical computations
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "02_Analysis_of_Twitter_Social_Network.ipynb".
The first task is:
you can find a description of the fields in the Twitter API documentation: https://dev.twitter.com/overview/api/tweets
Can you write Python code for it?

#creation time
tweet['created_at']

# text of the tweet
print(tweet['text'])

# user info
tweet['user']

# user is itslef a dict
print(type(tweet['user']))

tweet['user']['name']

# unique id of the user
tweet['user']['id']

#is the tweet a retweet?
'retweeted_status' in tweet

if 'retweeted_status' in tweet:
    print(tweet['retweeted_status'])
# the `retweeted_status` is also a tweet dictionary

# user id and name of the retweeted user?
if 'retweeted_status' in tweet:
    print(tweet['retweeted_status']['user']['id'])
    print(tweet['retweeted_status']['user']['name'])

# is the tweet a reply?
'in_reply_to_user_id' in tweet and tweet['in_reply_to_user_id'] is not None

# 'entities' contains the hashtags, urls and usernames used in the tweet
tweet['entities']

# user id of the mentioned users
for  mention in tweet['entities']['user_mentions']:
    print(mention['id'])

# is the tweet a quote?
'quoted_status' in tweet


I am working on the file "02_BinarySearch.ipynb".
The first task is:
Follow up for "Search in Rotated Sorted Array":

What if duplicates are allowed?
Would this affect the run-time complexity? How and why?
Can you write Python code for it?

class Solution(object):
    def search(self, nums, target):
        """
        :type nums: List[int]
        :type target: int
        :rtype: bool
        """
        if nums is None or len(nums) == 0:
            return False

        start, end = 0, len(nums)-1
        while start + 1 < end:
            mid = start + (end - start) / 2
            if nums[mid] == target:
                return True
            if nums[mid] == nums[end]:
                end -= 1
            elif nums[mid] > nums[end]:
                if nums[end] < target < nums[mid]:
                    end = mid - 1
                else:
                    start = mid
            else:
                if nums[mid] < target <= nums[end]:
                    start = mid + 1
                else:
                    end = mid

        return True if nums[start] == target or nums[end] == target else False


I am working on the file "02_BinarySearch.ipynb".
The first task is:
Recover Rotated Sorted Array

Given a rotated sorted array, recover it to sorted array in-place.
Example
[4, 5, 1, 2, 3] -> [1, 2, 3, 4, 5]
Can you write Python code for it?

class Solution:
    """
    @param nums: The rotated sorted array
    @return: nothing
    """
    def recoverRotatedSortedArray(self, nums):
        # write your code here
        def rev(nums, start, end):
            for i in range((end - start) / 2 + 1):
                nums[start+i], nums[end-i] = nums[end-i], nums[start+i]

        n = len(nums)
        for i in range(n-1):
            if nums[i] > nums[i+1]:
                rev(nums, 0, i)
                rev(nums, i+1, n-1)
                rev(nums, 0, n-1)
                break


I am working on the file "02_BinarySearch.ipynb".
The first task is:
Related: Find Peak Element

https://leetcode.com/problems/find-peak-element/#/description
Can you write Python code for it?

class Solution(object):
    def findPeakElement(self, nums):
        """
        :type nums: List[int]
        :rtype: int
        """
        if nums is None or len(nums) == 0:
            return -1
        n = len(nums)
        if n == 1:
            return 0

        start, end = 0, n-1
        while start + 1 < end:
            mid = start + (end - start) / 2
            if nums[mid] > nums[mid-1] and nums[mid] > nums[mid+1]:
                return mid
            elif nums[mid] < nums[mid+1]:
                start = mid
            else:
                end = mid

        return start if nums[start] > nums[end] else end


I am working on the file "02_BinarySearch.ipynb".
The first task is:
Rotate String

Given a string and an offset, rotate string by offset. (rotate from left to right)
Given "abcdefg".
offset=0 => "abcdefg"
offset=1 => "gabcdef"
offset=2 => "fgabcde"
offset=3 => "efgabcd"
Can you write Python code for it?

class Solution:
    # @param s: a list of char
    # @param offset: an integer
    # @return: nothing
    def rotateString(self, s, offset):
        # write you code here
        def rev(s, start, end):
            for i in range((end - start) / 2 + 1):
                s[start+i], s[end-i] = s[end-i], s[start+i]

        if s is not None and len(s) > 0:
            n = len(s)
            offset %= n
            if 0 < offset < n-1:
                rev(s, 0, n - offset - 1)
                rev(s, n - offset, n - 1)
                rev(s, 0, n-1)
            elif offset == n-1:
                rev(s, 0 , n-1)


I am working on the file "02_BinarySearch.ipynb".
The first task is:
Search a 2D Matrix
Write an efficient algorithm with O(m + n) that searches for a value in an m x n matrix. This matrix has the following properties:
Integers in each row are sorted in ascending from left to right.
Integers in each column are sorted in ascending from top to bottom.
Can you write Python code for it?

class Solution(object):
    def searchMatrix(self, matrix, target):
        """
        :type matrix: List[List[int]]
        :type target: int
        :rtype: bool
        """
        if not matrix or len(matrix) == 0:
            return False
        if len(matrix[0]) == 0:
            return False

        m, n = len(matrix), len(matrix[0])

        # O(m + n) brilliant
        r, c = 0, n-1
        while r <= m-1 and c >= 0:
            if matrix[r][c] == target:
                return True
            elif matrix[r][c] > target:
                c -= 1
            else:
                r += 1

        return False

        # O(mlogn), straight forward
        for i in range(m):
            if matrix[i][0] <= target <= matrix[i][-1]:
                start, end = 0, n-1
                while start + 1 < end:
                    mid = start + (end - start) / 2
                    if matrix[i][mid] == target:
                        return True
                    elif matrix[i][mid] < target:
                        start  = mid + 1
                    else:
                        end = mid - 1
                if matrix[i][start] == target or matrix[i][end] == target:
                    return True
        return False


I am working on the file "02_BinarySearch.ipynb".
The first task is:
Search in Rotated Sorted Array

You are given a target value to search. If found in the array return its index, otherwise return -1.
https://leetcode.com/problems/search-in-rotated-sorted-array/#/description
Can you write Python code for it?

# trival way
class Solution(object):
    def search(self, nums, target):
        """
        :type nums: List[int]
        :type target: int
        :rtype: int
        """
        n = len(nums)
        if nums is None or n == 0:
            return -1

        # find the minimun index
        start, end = 0, n-1
        while start < end:
            mid = start + (end - start) / 2
            if nums[mid] > nums[end]:
                start = mid + 1
            else:
                end = mid
        mark = start

        # binary search
        start, end = 0, n-1
        while start + 1 < end:
            mid = start + (end - start) / 2
            realmid = (mid + mark) % n
            if nums[realmid] == target:
                return realmid    ## important !!
            elif nums[realmid] < target:
                start = mid + 1
            else:
                end = mid - 1

        if nums[(start + mark) % n] == target:
            return (start + mark) % n
        if nums[(end + mark) % n] == target:
            return (end + mark) % n
        return -1

# good way: beats 96.13%  key of bisection: shirink the interval !!
class Solution(object):
    def search(self, nums, target):
        """
        :type nums: List[int]
        :type target: int
        :rtype: int
        """
        n = len(nums)
        if nums is None or n == 0:
            return -1

        # binary search
        start, end = 0, n-1
        while start + 1 < end:
            mid = start + (end - start) / 2
            if nums[mid] == target:
                return mid
            elif nums[mid] > nums[end]:
                if nums[end] < target < nums[mid]:
                    end = mid - 1
                else:
                    start = mid
            else:
                if nums[mid] < target <= nums[end]:
                    start = mid + 1
                else:
                    end = mid

        if nums[start] == target:
            return start
        if...


I am working on the file "02_BinarySearch.ipynb".
The first task is:
See more at:
https://leetcode.com/tag/binary-search/
http://www.jiuzhang.com/qa/974/
Summary
Four steps:
• start + 1 < end
• start + (end - start) / 2
• A[mid] ==, <, >
• A[start] A[end] ? target
Corner cases:
if nums is None or len(nums) == 0: return -1
Tricks:
compare with nums[start] or nums[end] for the rotated arrays
by setting mid = start + (end - start) / 2, mid + 1 should be in the range
Can you write Python code for it?

# reverse
def rev(nums, start, end):
    for i in range((end - start) / 2 + 1):
        nums[start+i], nums[end-i] = nums[end-i], nums[start+i]


I am working on the file "02_BinarySearch.ipynb".
The first task is:
Smallest Rectangle Enclosing Black Pixels
Can you write Python code for it?

class Solution(object):
    def minArea(self, image, x, y):
        """
        :type image: List[List[str]]
        :type x: int
        :type y: int
        :rtype: int
        """

        def isBlackCol(image, col):
            for i in range(len(image)):
                if image[i][col] == '1':
                    return True
            return False

        def isBlackRow(image, row):
            for j in range(len(image[row])):
                if image[row][j] == '1':
                    return True
            return False

        if image is None or len(image) == 0:
            return 0

        m, n = len(image), len(image[0])

        # left
        start, end = 0, y
        while start + 1 < end:
            mid = start + (end - start) / 2
            if isBlackCol(image, mid):
                end = mid
            else:
                start = mid

        if isBlackCol(image, start):
            left = start
        else:
            left = end

        # right
        start, end = y, n-1
        while start + 1 < end:
            mid = start + (end - start) / 2
            if isBlackCol(image, mid):
                start = mid
            else:
                end = mid

        if isBlackCol(image, end):
            right = end
        else:
            right = start

        # up
        start, end = 0, x
        while start + 1 < end:
            mid = start + (end - start) / 2
            if isBlackRow(image, mid):
                end = mid
            else:
                start = mid

        if isBlackRow(image, start):
            up = start
        else:
            up = end

        # bottom
        start, end = x, m-1
        while start + 1 < end:
            mid = start + (end - start) / 2
            if isBlackRow(image, mid):
                start = mid
            else:
                end = mid

        if isBlackRow(image, end):
            bottom = end
        else:
           ...


I am working on the file "02_Classification/03.0 - TF Classification Model - DNN Wide & Deep + Train_And_Evaluate + Dataset + TFRecords.ipynb".
The first task is:
3. Define Feature Columns
The input numeric columns are assumed to be normalized (or have the same scale). Otherise, a normlizer_fn, along with the normlisation params (mean, stdv) should be passed to tf.feature_column.numeric_column() constructor
Can you write Python code for it?

def extend_feature_columns(feature_columns, hparams):

    num_buckets = hparams.num_buckets
    embedding_size = hparams.embedding_size

    buckets = np.linspace(-3, 3, num_buckets).tolist()

    alpha_X_beta = tf.feature_column.crossed_column(
            [feature_columns['alpha'], feature_columns['beta']], 4)

    x_bucketized = tf.feature_column.bucketized_column(
            feature_columns['x'], boundaries=buckets)

    y_bucketized = tf.feature_column.bucketized_column(
            feature_columns['y'], boundaries=buckets)

    x_bucketized_X_y_bucketized = tf.feature_column.crossed_column(
           [x_bucketized, y_bucketized], num_buckets**2)

    x_bucketized_X_y_bucketized_embedded = tf.feature_column.embedding_column(
            x_bucketized_X_y_bucketized, dimension=embedding_size)

    feature_columns['alpha_X_beta'] = alpha_X_beta
    feature_columns['x_bucketized_X_y_bucketized'] = x_bucketized_X_y_bucketized
    feature_columns['x_bucketized_X_y_bucketized_embedded'] = x_bucketized_X_y_bucketized_embedded

    return feature_columns

def get_feature_columns(hparams):

    CONSTRUCTED_NUMERIC_FEATURES_NAMES = ['x_2', 'y_2', 'xy', 'dist_xy']
    all_numeric_feature_names = NUMERIC_FEATURE_NAMES.copy()

    if PROCESS_FEATURES:
        all_numeric_feature_names += CONSTRUCTED_NUMERIC_FEATURES_NAMES

    numeric_columns = {feature_name: tf.feature_column.numeric_column(feature_name)
                       for feature_name in all_numeric_feature_names}

    categorical_column_with_vocabulary = \
        {item[0]: tf.feature_column.categorical_column_with_vocabulary_list(item[0], item[1])
         for item in CATEGORICAL_FEATURE_NAMES_WITH_VOCABULARY.items()}

    feature_columns = {}

    if numeric_columns is not None:
        feature_columns.update(numeric_columns)

    if categorical_column_with_vocabulary is not None:
        feature_columns.update(categorical_column_with_vocabulary)

    if EXTEND_FEATURE_COLUMNS:
        feature_columns =...


I am working on the file "02_Classification/03.0 - TF Classification Model - DNN Wide & Deep + Train_And_Evaluate + Dataset + TFRecords.ipynb".
The first task is:
4. Define an Estimator Creation Function
a. Get wide and deep feature columns

dense columns = numeric columns + embedding columns
categorical columns = vocabolary list columns + bucketized columns
sparse columns = hashed categorical columns + crossed columns
categorical columns => indicator columns
deep columns = dense columns + indicator columns
wide columns = categorical columns + sparse columns
Can you write Python code for it?

def get_wide_deep_columns():

    feature_columns = list(get_feature_columns(hparams).values())

    dense_columns = list(
        filter(lambda column: isinstance(column, feature_column._NumericColumn) |
                              isinstance(column, feature_column._EmbeddingColumn),
               feature_columns
        )
    )

    categorical_columns = list(
        filter(lambda column: isinstance(column, feature_column._VocabularyListCategoricalColumn) |
                              isinstance(column, feature_column._BucketizedColumn),
                   feature_columns)
    )

    sparse_columns = list(
        filter(lambda column: isinstance(column,feature_column._HashedCategoricalColumn) |
                              isinstance(column, feature_column._CrossedColumn),
               feature_columns)
    )

    indicator_columns = list(
            map(lambda column: tf.feature_column.indicator_column(column),
                categorical_columns)
    )

    deep_feature_columns = dense_columns + indicator_columns
    wide_feature_columns = categorical_columns + sparse_columns

    return wide_feature_columns, deep_feature_columns


I am working on the file "02_Classification/03.0 - TF Classification Model - DNN Wide & Deep + Train_And_Evaluate + Dataset + TFRecords.ipynb".
The first task is:
5. Run Experiment
a. Set HParam and RunConfig
Can you write Python code for it?

TRAIN_SIZE = 12000
NUM_EPOCHS = 1000
BATCH_SIZE = 500
NUM_EVAL = 10
TOTAL_STEPS = (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS
CHECKPOINT_STEPS = int((TRAIN_SIZE/BATCH_SIZE) * (NUM_EPOCHS/NUM_EVAL))

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    hidden_units=[16, 12, 8],
    num_buckets = 6,
    embedding_size = 3,
    max_steps = TOTAL_STEPS,
    dropout_prob = 0.001)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.contrib.learn.RunConfig(
    save_checkpoints_steps=CHECKPOINT_STEPS,
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", TOTAL_STEPS)
print("Required Evaluation Steps:", NUM_EVAL)
print("That is 1 evaluation step after each",NUM_EPOCHS/NUM_EVAL," epochs")
print("Save Checkpoint After",CHECKPOINT_STEPS,"steps")


I am working on the file "02_Classification/03.0 - TF Classification Model - DNN Wide & Deep + Train_And_Evaluate + Dataset + TFRecords.ipynb".
The first task is:
6. Evaluate the Model
Can you write Python code for it?


I am working on the file "02_Classification/03.0 - TF Classification Model - DNN Wide & Deep + Train_And_Evaluate + Dataset + TFRecords.ipynb".
The first task is:
Serving via the Saved Model
Can you write Python code for it?

import os

export_dir = model_dir +"/export/predict/"

saved_model_dir = export_dir + "/" + os.listdir(path=export_dir)[-1]

print(saved_model_dir)

predictor_fn = tf.contrib.predictor.from_saved_model(
    export_dir = saved_model_dir,
    signature_def_key="predict"
)

output = predictor_fn({'csv_rows': ["0.5,1,ax01,bx02", "-0.5,-1,ax02,bx02"]})
print(output)


I am working on the file "02_Classification/03.0 - TF Classification Model - DNN Wide & Deep + Train_And_Evaluate + Dataset + TFRecords.ipynb".
The first task is:
Steps to use the TF Estimator (Train_And_Evaluate) APIs

Define dataset metadata
Define data input function to read the data from .tfrecord files + feature processing
Create TF feature columns based on metadata + extended feature columns
Define an estimator (LinearCombinedDNNClassifier) with the required feature columns (wide/deep) & parameters
Run an experiment using the estimator train_and_evaluate function to train, evaluate, and export the model
Evaluate the model using test data
Perform predictions & serving the exported model
Can you write Python code for it?

MODEL_NAME = 'class-model-01'

TRAIN_DATA_FILES_PATTERN = 'data/train-*.tfrecords'
VALID_DATA_FILES_PATTERN = 'data/valid-*.tfrecords'
TEST_DATA_FILES_PATTERN = 'data/test-*.tfrecords'

RESUME_TRAINING = False
PROCESS_FEATURES = True
EXTEND_FEATURE_COLUMNS = True
MULTI_THREADING = True


I am working on the file "02_Classification/03.0 - TF Classification Model - DNN Wide & Deep + Train_And_Evaluate + Dataset + TFRecords.ipynb".
The first task is:
b. Data pipeline input function
Can you write Python code for it?

def tfrecods_input_fn(files_name_pattern, mode=tf.estimator.ModeKeys.EVAL,
                 num_epochs=None,
                 batch_size=200):

    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False

    print("")
    print("* data input_fn:")
    print("================")
    print("Input file(s): {}".format(files_name_pattern))
    print("Batch size: {}".format(batch_size))
    print("Epoch Count: {}".format(num_epochs))
    print("Mode: {}".format(mode))
    print("Shuffle: {}".format(shuffle))
    print("================")
    print("")

    file_names = tf.matching_files(files_name_pattern)
    dataset = data.TFRecordDataset(filenames=file_names)

    if shuffle:
        dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)

    dataset = dataset.batch(batch_size)
    dataset = dataset.map(lambda tf_example: parse_tf_example(tf_example))

    if PROCESS_FEATURES:
        dataset = dataset.map(lambda features, target: (process_features(features), target))

    dataset = dataset.repeat(num_epochs)
    iterator = dataset.make_one_shot_iterator()

    features, target = iterator.get_next()
    return features, target

features, target = tfrecods_input_fn(files_name_pattern="")
print("Feature read from TFRecords: {}".format(list(features.keys())))
print("Target read from TFRecords: {}".format(target))


I am working on the file "02_Classification/03.0 - TF Classification Model - DNN Wide & Deep + Train_And_Evaluate + Dataset + TFRecords.ipynb".
The first task is:
b. Define serving function
Can you write Python code for it?

def csv_serving_input_fn():

    SERVING_HEADER = ['x','y','alpha','beta']
    SERVING_HEADER_DEFAULTS = [[0.0], [0.0], ['NA'], ['NA']]

    rows_string_tensor = tf.placeholder(dtype=tf.string,
                                         shape=[None],
                                         name='csv_rows')

    receiver_tensor = {'csv_rows': rows_string_tensor}

    row_columns = tf.expand_dims(rows_string_tensor, -1)
    columns = tf.decode_csv(row_columns, record_defaults=SERVING_HEADER_DEFAULTS)
    features = dict(zip(SERVING_HEADER, columns))

    return tf.estimator.export.ServingInputReceiver(
        process_features(features), receiver_tensor)


I am working on the file "02_Classification/03.0 - TF Classification Model - DNN Wide & Deep + Train_And_Evaluate + Dataset + TFRecords.ipynb".
The first task is:
b. Define the estimator
Can you write Python code for it?

def create_estimator(run_config, hparams, print_desc=False):

    wide_feature_columns, deep_feature_columns = get_wide_deep_columns()

    estimator = tf.estimator.DNNLinearCombinedClassifier(

        n_classes= len(TARGET_LABELS),
        label_vocabulary=TARGET_LABELS,

        dnn_feature_columns = deep_feature_columns,
        linear_feature_columns = wide_feature_columns,

        dnn_hidden_units= hparams.hidden_units,

        dnn_optimizer= tf.train.AdamOptimizer(),

        dnn_activation_fn= tf.nn.elu,
        dnn_dropout= hparams.dropout_prob,

        config= run_config
    )

    if print_desc:
        print("")
        print("*Estimator Type:")
        print("================")
        print(type(estimator))
        print("")
        print("*deep columns:")
        print("==============")
        print(deep_feature_columns)
        print("")
        print("wide columns:")
        print("=============")
        print(wide_feature_columns)
        print("")

    return estimator


I am working on the file "02_Classification/03.0 - TF Classification Model - DNN Wide & Deep + Train_And_Evaluate + Dataset + TFRecords.ipynb".
The first task is:
c. Define TrainSpec and EvaluSpec
Can you write Python code for it?

train_spec = tf.estimator.TrainSpec(
    input_fn = lambda: tfrecods_input_fn(
        TRAIN_DATA_FILES_PATTERN,
        mode = tf.estimator.ModeKeys.TRAIN,
        num_epochs=hparams.num_epochs,
        batch_size=hparams.batch_size
    ),
    max_steps=hparams.max_steps,
    hooks=None
)

eval_spec = tf.estimator.EvalSpec(
    input_fn = lambda: tfrecods_input_fn(
        VALID_DATA_FILES_PATTERN,
        mode=tf.estimator.ModeKeys.EVAL,
        num_epochs=1,
        batch_size=hparams.batch_size
    ),
    exporters=[tf.estimator.LatestExporter(
        name="predict", # the name of the folder in which the model will be exported to under export
        serving_input_receiver_fn=csv_serving_input_fn,
        exports_to_keep=1,
        as_text=True)],
    steps=None,
    hooks=None
)


I am working on the file "02_Classification/04.0 - TF Classification Model - Custom Estimator + Experiment + Dataset + CSV.ipynb".
The first task is:
3. Define Feature Columns
Can you write Python code for it?


I am working on the file "02_Classification/04.0 - TF Classification Model - Custom Estimator + Experiment + Dataset + CSV.ipynb".
The first task is:
4. Define Model Function
Can you write Python code for it?

def get_input_layer_feature_columns(hparams):

    feature_columns = list(get_feature_columns(hparams).values())

    dense_columns = list(
        filter(lambda column: isinstance(column, feature_column._NumericColumn) |
                              isinstance(column, feature_column._EmbeddingColumn),
               feature_columns
        )
    )

    categorical_columns = list(
        filter(lambda column: isinstance(column, feature_column._VocabularyListCategoricalColumn) |
                              isinstance(column, feature_column._BucketizedColumn),
                   feature_columns)
    )

    indicator_columns = list(
            map(lambda column: tf.feature_column.indicator_column(column),
                categorical_columns)
    )

    return dense_columns+indicator_columns

def classification_model_fn(features, labels, mode, params):

    hidden_units = params.hidden_units
    output_layer_size = len(TARGET_LABELS)

    feature_columns = get_input_layer_feature_columns(hparams)

    # Create the input layers from the feature columns
    input_layer = tf.feature_column.input_layer(features= features,
                                                feature_columns=feature_columns)

    # Create a fully-connected layer-stack based on the hidden_units in the params
    hidden_layers = tf.contrib.layers.stack(inputs= input_layer,
                                            layer= tf.contrib.layers.fully_connected,
                                            stack_args= hidden_units)

    # Connect the output layer (logits) to the hidden layer (no activation fn)
    logits = tf.layers.dense(inputs=hidden_layers,
                             units=output_layer_size)

    # Reshape output layer to 1-dim Tensor to return predictions
    output = tf.squeeze(logits)

    # Provide an estimator spec for `ModeKeys.PREDICT`.
    if mode == tf.estimator.ModeKeys.PREDICT:
        probabilities = tf.nn.softmax(logits)
        predicted_indices = tf.argmax(probabilities, 1)

        #...


I am working on the file "02_Classification/04.0 - TF Classification Model - Custom Estimator + Experiment + Dataset + CSV.ipynb".
The first task is:
6. Run Experiment
a. Define experiment function
Can you write Python code for it?

def generate_experiment_fn(**experiment_args):

    def _experiment_fn(run_config, hparams):

        train_input_fn = lambda: csv_input_fn(
            TRAIN_DATA_FILES_PATTERN,
            mode = tf.estimator.ModeKeys.TRAIN,
            num_epochs=hparams.num_epochs,
            batch_size=hparams.batch_size
        )

        eval_input_fn = lambda: csv_input_fn(
            VALID_DATA_FILES_PATTERN,
            mode=tf.estimator.ModeKeys.EVAL,
            num_epochs=1,
            batch_size=hparams.batch_size
        )

        estimator = create_estimator(run_config, hparams)

        return tf.contrib.learn.Experiment(
            estimator,
            train_input_fn=train_input_fn,
            eval_input_fn=eval_input_fn,
            eval_steps=None,
            **experiment_args
        )

    return _experiment_fn


I am working on the file "02_Classification/04.0 - TF Classification Model - Custom Estimator + Experiment + Dataset + CSV.ipynb".
The first task is:
Steps to use the TF Experiment APIs

Define dataset metadata
Define data input function to read the data from .tfrecord files + feature processing
Create TF feature columns based on metadata + extended feature columns
Define an a model function with the required feature columns, EstimatorSpecs, & parameters
Run an Experiment with learn_runner to train, evaluate, and export the model
Evaluate the model using test data
Perform predictions & serving the exported model (using CSV/JSON input)
Can you write Python code for it?

MODEL_NAME = 'class-model-02'

TRAIN_DATA_FILES_PATTERN = 'data/train-*.csv'
VALID_DATA_FILES_PATTERN = 'data/valid-*.csv'
TEST_DATA_FILES_PATTERN = 'data/test-*.csv'

RESUME_TRAINING = False
PROCESS_FEATURES = True
EXTEND_FEATURE_COLUMNS = True
MULTI_THREADING = True


I am working on the file "02_Classification/04.0 - TF Classification Model - Custom Estimator + Experiment + Dataset + CSV.ipynb".
The first task is:
d. Run the Experiment via learn_runner
Can you write Python code for it?

if not RESUME_TRAINING:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...")

tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow()
print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................")

learn_runner.run(
    experiment_fn=generate_experiment_fn(

        export_strategies=[
            make_export_strategy(
            json_serving_input_fn,
            exports_to_keep=1,
            as_text=True
            )
        ]
    ),
    run_config=run_config,
    schedule="train_and_evaluate",
    hparams=hparams
)

time_end = datetime.utcnow()
print(".......................................")
print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
And these are the results of applying the filter-weights to the second image.
Can you write Python code for it?


plot_conv_layer(layer=layer_conv2, image=image2)

#plot_conv_layer(layer=layer_conv3, image=image1)

#plot_conv_layer(layer=layer_conv3, image=image2)


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Applying each of these convolutional filters to the first input image gives the following output images, which are then used as input to the second convolutional layer. Note that these images are down-sampled to 14 x 14 pixels which is half the resolution of the original input image.
Can you write Python code for it?


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Check that the output of the fully-connected layer is a tensor with shape (?, 128) where the ? means there is an arbitrary number of images and fc_size == 128.
Can you write Python code for it?

layer_fc1

layer_dropout1 = tf.nn.dropout(layer_fc1, keep_prob)

#layer_fc2 = new_fc_layer(input=layer_dropout1,
#                         num_inputs=fc_size,
#                         num_outputs=fc_size,
#                         use_relu=True)

#layer_dropout2 = tf.nn.dropout(layer_fc2, keep_prob)


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Check that the tensors now have shape (?, 1764) which means there's an arbitrary number of images which have been flattened to vectors of length 1764 each. Note that 1764 = 7 x 7 x 36.
Can you write Python code for it?


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Check the shape of the tensor that will be output by the convolutional layer. It is (?, 14, 14, 16) which means that there is an arbitrary number of images (this is the ?), each image is 14 pixels wide and 14 pixels high, and there are 16 different channels, one channel for each of the filters.
Can you write Python code for it?


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Check the shape of the tensor that will be output from this convolutional layer. The shape is (?, 7, 7, 36) where the ? again means that there is an arbitrary number of images, with each image having width and height of 7 pixels, and there are 36 channels, one for each filter.
Can you write Python code for it?


layer_conv2

#layer_conv3, weights_conv3 = \
#    new_conv_layer(input=layer_conv2,
#                   num_input_channels=num_filters2,
#                   filter_size=filter_size3,
#                   num_filters=num_filters3,
#                   use_pooling=True)

#layer_conv3


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Configuration of Neural Network
The configuration of the Convolutional Neural Network is defined here for convenience, so you can easily find and change these numbers and re-run the Notebook.
Can you write Python code for it?

# Convolutional Layer 1.
filter_size1 = 5          # Convolution filters are 5 x 5 pixels.
num_filters1 = 16         # There are 16 of these filters.

# Convolutional Layer 2.
filter_size2 = 5          # Convolution filters are 5 x 5 pixels.
num_filters2 = 36         # There are 36 of these filters.

# Fully-connected layer.
fc_size = 128             # Number of neurons in fully-connected layer.


# Convolutional Layer 1.
filter_size1 = 5          # Convolution filters are 5 x 5 pixels.
num_filters1 = 9          # There are 16 of these filters.

# Convolutional Layer 2.
filter_size2 = 5          # Convolution filters are 5 x 5 pixels.
num_filters2 = 9          # There are 36 of these filters.

# Convolutional Layer 2.
#filter_size3 = 5          # Convolution filters are 5 x 5 pixels.
#num_filters3 = 25         # There are 25 of these filters.

# Fully-connected layer.
fc_size = 32               # Number of neurons in fully-connected layer.


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Convolution Layer 1
Now plot the filter-weights for the first convolutional layer.
Note that positive weights are red and negative weights are blue.
Can you write Python code for it?


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Convolutional Layer 2
Create the second convolutional layer, which takes as input the output from the first convolutional layer. The number of input channels corresponds to the number of filters in the first convolutional layer.
Can you write Python code for it?

layer_conv2, weights_conv2 = \
    new_conv_layer(input=layer_conv1,
                   num_input_channels=num_filters1,
                   filter_size=filter_size2,
                   num_filters=num_filters2,
                   use_pooling=True)


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Cost-function to be optimized
To make the model better at classifying the input images, we must somehow change the variables for all the network layers. To do this we first need to know how well the model currently performs by comparing the predicted output of the model y_pred to the desired output y_true.
The cross-entropy is a performance measure used in classification. The cross-entropy is a continuous function that is always positive and if the predicted output of the model exactly matches the desired output then the cross-entropy equals zero. The goal of optimization is therefore to minimize the cross-entropy so it gets as close to zero as possible by changing the variables of the network layers.
TensorFlow has a built-in function for calculating the cross-entropy. Note that the function calculates the softmax internally so we must use the output of layer_fc2 directly rather than y_pred which has already had the softmax applied.
Can you write Python code for it?

cross_entropy = tf.nn.softmax_cross_entropy_with_logits(logits=layer_fc3,
                                                        labels=y_true)


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Create placeholder var for dropout
Can you write Python code for it?


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Data Dimensions
The data dimensions are used in several places in the source-code below. They are defined once so we can use these variables instead of numbers throughout the source-code below.
Can you write Python code for it?

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
img_shape = (img_size, img_size)

# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = 1

# Number of classes, one class for each of 10 digits.
num_classes = 10


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Flatten Layer
The convolutional layers output 4-dim tensors. We now wish to use these as input in a fully-connected network, which requires for the tensors to be reshaped or flattened to 2-dim tensors.
Can you write Python code for it?

layer_flat, num_features = flatten_layer(layer_conv2)


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
From these images, it looks like the second convolutional layer might detect lines and patterns in the input images, which are less sensitive to local variations in the original input images.
These images are then flattened and input to the fully-connected layer, but that is not shown here.
Close TensorFlow Session
We are now done using TensorFlow, so we close the session to release its resources.
Can you write Python code for it?

# This has been commented out in case you want to modify and experiment
# with the Notebook without having to restart it.
# session.close()


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Fully-Connected Layer 2
Add another fully-connected layer that outputs vectors of length 10 for determining which of the 10 classes the input image belongs to. Note that ReLU is not used in this layer.
Can you write Python code for it?

layer_fc3 = new_fc_layer(input=layer_dropout1,
                         num_inputs=fc_size,
                         num_outputs=num_classes,
                         use_relu=False)

layer_fc3


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Function for performing a number of optimization iterations so as to gradually improve the variables of the network layers. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples.  The progress is printed every 100 iterations.
Can you write Python code for it?

print(data.train)

# Counter for total number of iterations performed so far.
total_iterations = 0

def optimize(num_iterations):
    # Ensure we update the global variable rather than a local copy.
    global total_iterations

    # Start-time used for printing time-usage below.
    start_time = time.time()

    for i in range(total_iterations,
                   total_iterations + num_iterations):

        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = data.train.next_batch(train_batch_size)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch,
                           keep_prob: 0.5}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)

        # Print status every 100 iterations.
        if i % 100 == 0:
            # Calculate the accuracy on the training-set.
            acc = session.run(accuracy, feed_dict=feed_dict_train)

            # Message for printing.
            msg = "Optimization Iteration: {0:>6}, Training Accuracy: {1:>6.1%}"

            # Print it.
            print(msg.format(i + 1, acc))

    # Update the total number of iterations performed.
    total_iterations += num_iterations

    # Ending time.
    end_time = time.time()

    # Difference between start and end-times.
    time_dif = end_time - start_time

    # Print the time-usage.
    print("Time usage: " + str(timedelta(seconds=int(round(time_dif)))))


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function for creating a new Convolutional Layer
This function creates a new convolutional layer in the computational graph for TensorFlow. Nothing is actually calculated here, we are just adding the mathematical formulas to the TensorFlow graph.
It is assumed that the input is a 4-dim tensor with the following dimensions:

Image number.
Y-axis of each image.
X-axis of each image.
Channels of each image.

Note that the input channels may either be colour-channels, or it may be filter-channels if the input is produced from a previous convolutional layer.
The output is another 4-dim tensor with the following dimensions:

Image number, same as input.
Y-axis of each image. If 2x2 pooling is used, then the height and width of the input images is divided by 2.
X-axis of each image. Ditto.
Channels produced by the convolutional filters.
Can you write Python code for it?

def new_conv_layer(input,              # The previous layer.
                   num_input_channels, # Num. channels in prev. layer.
                   filter_size,        # Width and height of each filter.
                   num_filters,        # Number of filters.
                   use_pooling=True):  # Use 2x2 max-pooling.

    # Shape of the filter-weights for the convolution.
    # This format is determined by the TensorFlow API.
    shape = [filter_size, filter_size, num_input_channels, num_filters]

    # Create new weights aka. filters with the given shape.
    weights = new_weights(shape=shape)

    # Create new biases, one for each filter.
    biases = new_biases(length=num_filters)

    # Create the TensorFlow operation for convolution.
    # Note the strides are set to 1 in all dimensions.
    # The first and last stride must always be 1,
    # because the first is for the image-number and
    # the last is for the input-channel.
    # But e.g. strides=[1, 2, 2, 1] would mean that the filter
    # is moved 2 pixels across the x- and y-axis of the image.
    # The padding is set to 'SAME' which means the input image
    # is padded with zeroes so the size of the output is the same.
    layer = tf.nn.conv2d(input=input,
                         filter=weights,
                         strides=[1, 1, 1, 1],
                         padding='SAME')

    # Add the biases to the results of the convolution.
    # A bias-value is added to each filter-channel.
    layer += biases

    # Use pooling to down-sample the image resolution?
    if use_pooling:
        # This is 2x2 max-pooling, which means that we
        # consider 2x2 windows and select the largest value
        # in each window. Then we move 2 pixels to the next window.
        layer = tf.nn.max_pool(value=layer,
                               ksize=[1, 2, 2, 1],
                               strides=[1, 2, 2, 1],
                               padding='SAME')

    # Rectified Linear Unit (ReLU).
    # It calculates max(x, 0) for each input...


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function for creating a new Fully-Connected Layer
This function creates a new fully-connected layer in the computational graph for TensorFlow. Nothing is actually calculated here, we are just adding the mathematical formulas to the TensorFlow graph.
It is assumed that the input is a 2-dim tensor of shape [num_images, num_inputs]. The output is a 2-dim tensor of shape [num_images, num_outputs].
Can you write Python code for it?

def new_fc_layer(input,          # The previous layer.
                 num_inputs,     # Num. inputs from prev. layer.
                 num_outputs,    # Num. outputs.
                 use_relu=True): # Use Rectified Linear Unit (ReLU)?

    # Create new weights and biases.
    weights = new_weights(shape=[num_inputs, num_outputs])
    biases = new_biases(length=num_outputs)

    # Calculate the layer as the matrix multiplication of
    # the input and weights, and then add the bias-values.
    layer = tf.matmul(input, weights) + biases

    # Use ReLU?
    if use_relu:
        layer = tf.nn.relu(layer)

    return layer


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function for flattening a layer
A convolutional layer produces an output tensor with 4 dimensions. We will add fully-connected layers after the convolution layers, so we need to reduce the 4-dim tensor to 2-dim which can be used as input to the fully-connected layer.
Can you write Python code for it?

def flatten_layer(layer):
    # Get the shape of the input layer.
    layer_shape = layer.get_shape()

    # The shape of the input layer is assumed to be:
    # layer_shape == [num_images, img_height, img_width, num_channels]

    # The number of features is: img_height * img_width * num_channels
    # We can use a function from TensorFlow to calculate this.
    num_features = layer_shape[1:4].num_elements()

    # Reshape the layer to [num_images, num_features].
    # Note that we just set the size of the second dimension
    # to num_features and the size of the first dimension to -1
    # which means the size in that dimension is calculated
    # so the total size of the tensor is unchanged from the reshaping.
    layer_flat = tf.reshape(layer, [-1, num_features])

    # The shape of the flattened layer is now:
    # [num_images, img_height * img_width * num_channels]

    # Return both the flattened layer and the number of features.
    return layer_flat, num_features


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function for plotting images
Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function for plotting the output of a convolutional layer
Can you write Python code for it?

def plot_conv_layer(layer, image):
    # Assume layer is a TensorFlow op that outputs a 4-dim tensor
    # which is the output of a convolutional layer,
    # e.g. layer_conv1 or layer_conv2.

    # Create a feed-dict containing just one image.
    # Note that we don't need to feed y_true because it is
    # not used in this calculation.
    feed_dict = {x: [image]}

    # Calculate and retrieve the output values of the layer
    # when inputting that image.
    values = session.run(layer, feed_dict=feed_dict)

    # Number of filters used in the conv. layer.
    num_filters = values.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot the output images of all the filters.
    for i, ax in enumerate(axes.flat):
        # Only plot the images for valid filters.
        if i<num_filters:
            # Get the output image of using the i'th filter.
            # See new_conv_layer() for details on the format
            # of this 4-dim tensor.
            img = values[0, :, :, i]

            # Plot image.
            ax.imshow(img, interpolation='nearest', cmap='binary')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


def plot_conv_layer(layer, image):
    # Assume layer is a TensorFlow op that outputs a 4-dim tensor
    # which is the output of a convolutional layer,
    # e.g. layer_conv1 or layer_conv2.

    # Create a feed-dict containing just one image.
    # Note that we don't need to feed y_true because it is
    # not used in this calculation.
    feed_dict = {x: [image],
                 keep_prob: 1}

    # Calculate and retrieve the output values of the layer
    # when inputting that image.
    values = session.run(layer, feed_dict=feed_dict)

    # Number of filters used in the conv. layer.
    num_filters = values.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot the output images of all the filters.
    for i, ax in enumerate(axes.flat):
        # Only plot the images for valid filters.
        if i<num_filters:
            # Get the output image of using the i'th filter.
            # See new_conv_layer() for details on the format
            # of this 4-dim tensor.
            img = values[0, :, :, i]

            # Plot image.
            ax.imshow(img, interpolation='nearest', cmap='binary')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function to perform optimization iterations
There are 55,000 images in the training-set. It takes a long time to calculate the gradient of the model using all these images. We therefore only use a small batch of images in each iteration of the optimizer.
If your computer crashes or becomes very slow because you run out of RAM, then you may try and lower this number, but you may then need to perform more optimization iterations.
Can you write Python code for it?


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function to plot confusion matrix
Can you write Python code for it?

def plot_confusion_matrix(cls_pred):
    # This is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Get the true classifications for the test-set.
    cls_true = data.y_test_cls

    # Get the confusion matrix using sklearn.
    cm = confusion_matrix(y_true=cls_true,
                          y_pred=cls_pred)

    # Print the confusion matrix as text.
    print(cm)

    # Plot the confusion matrix as an image.
    plt.matshow(cm)

    # Make various adjustments to the plot.
    plt.colorbar()
    tick_marks = np.arange(num_classes)
    plt.xticks(tick_marks, range(num_classes))
    plt.yticks(tick_marks, range(num_classes))
    plt.xlabel('Predicted')
    plt.ylabel('True')

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Helper-function to plot example errors
Function for plotting examples of images from the test-set that have been mis-classified.
Can you write Python code for it?

def plot_example_errors(cls_pred, correct):
    # This function is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # correct is a boolean array whether the predicted class
    # is equal to the true class for each image in the test-set.

    # Negate the boolean array.
    incorrect = (correct == False)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.test.images[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.test.cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])


def plot_example_errors(cls_pred, correct):
    # This function is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # correct is a boolean array whether the predicted class
    # is equal to the true class for each image in the test-set.

    # Negate the boolean array.
    incorrect = (correct == False)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.x_test[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.y_test_cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Input Images
Helper-function for plotting an image.
Can you write Python code for it?

def plot_image(image):
    plt.imshow(image.reshape(img_shape),
               interpolation='nearest',
               cmap='binary')

    plt.show()


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
It can be difficult to understand and keep track of how these filters are applied because of the high dimensionality.
Applying these convolutional filters to the images that were ouput from the first conv-layer gives the following images.
Note that these are down-sampled yet again to 7 x 7 pixels which is half the resolution of the images from the first conv-layer.
Can you write Python code for it?


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
It is difficult to see from these images what the purpose of the convolutional filters might be. It appears that they have merely created several variations of the input image, as if light was shining from different angles and casting shadows in the image.
Convolution Layer 2
Now plot the filter-weights for the second convolutional layer.
There are 16 output channels from the first conv-layer, which means there are 16 input channels to the second conv-layer. The second conv-layer has a set of filter-weights for each of its input channels. We start by plotting the filter-weigths for the first channel.
Note again that positive weights are red and negative weights are blue.
Can you write Python code for it?


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Optimization Method
Now that we have a cost measure that must be minimized, we can then create an optimizer. In this case it is the AdamOptimizer which is an advanced form of Gradient Descent.
Note that optimization is not performed at this point. In fact, nothing is calculated at all, we just add the optimizer-object to the TensorFlow graph for later execution.
Can you write Python code for it?


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Performance Measures
We need a few more performance measures to display the progress to the user.
This is a vector of booleans whether the predicted class equals the true class of each image.
Can you write Python code for it?

correct_prediction = tf.equal(y_pred_cls, y_true_cls)


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Performance after 1 optimization iteration
The classification accuracy does not improve much from just 1 optimization iteration, because the learning-rate for the optimizer is set very low.
Can you write Python code for it?


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Performance after 10,000 optimization iterations
After 10,000 optimization iterations, the model has a classification accuracy on the test-set of about 99%.
Can you write Python code for it?

optimize(num_iterations=9000) # We performed 1000 iterations above.

print_test_accuracy(show_example_errors=True,
                    show_confusion_matrix=True)


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Performance after 100 optimization iterations
After 100 optimization iterations, the model has significantly improved its classification accuracy.
Can you write Python code for it?

optimize(num_iterations=99) # We already performed 1 iteration above.

print_test_accuracy(show_example_errors=True)


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Performance before any optimization
The accuracy on the test-set is very low because the model variables have only been initialized and not optimized at all, so it just classifies the images randomly.
Can you write Python code for it?


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Placeholder variables
Placeholder variables serve as the input to the TensorFlow computational graph that we may change each time we execute the graph. We call this feeding the placeholder variables and it is demonstrated further below.
First we define the placeholder variable for the input images. This allows us to change the images that are input to the TensorFlow graph. This is a so-called tensor, which just means that it is a multi-dimensional vector or matrix. The data-type is set to float32 and the shape is set to [None, img_size_flat], where None means that the tensor may hold an arbitrary number of images with each image being a vector of length img_size_flat.
Can you write Python code for it?

x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Plot an image from the test-set which will be used as an example below.
Can you write Python code for it?

image1 = data.test.images[0]
plot_image(image1)


image1 = data.x_test[0]
plot_image(image1)


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
Predicted Class
The second fully-connected layer estimates how likely it is that the input image belongs to each of the 10 classes. However, these estimates are a bit rough and difficult to interpret because the numbers may be very small or large, so we want to normalize them so that each element is limited between zero and one and the 10 elements sum to one. This is calculated using the so-called softmax function and the result is stored in y_pred.
Can you write Python code for it?


I am working on the file "02_Convolutional_Neural_Network.ipynb".
The first task is:
TensorFlow Run
Create TensorFlow session
Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph.
Can you write Python code for it?