I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
3.1. A first predictor
For a span of days we are interested in knoweledge, which flight we should take. Therefore let's choose the popular date for Christmas returning flights 21.12.2015 as example. As we want to use historical data, let's get first clear what data we have. Is it possible to compare flights over the years?
Can you write Python code for it?

query_day = 21
query_month = 12

# how many flights do exist in all years?
flights = []
flightvalues = []
for y in years:
    query = list(bigdf[(bigdf.YEAR == int(y)) & (bigdf.MONTH == query_month) & (bigdf.DAY_OF_MONTH == query_day)].FL_NUM.astype(int).unique())
    flights.append(query)
    flightvalues += query

# build a matrix
data_matrix = np.zeros((len(flightvalues), len(years)))
# build dict
flightdict = dict(zip(flightvalues, np.arange(0, len(flightvalues))))

# fill datamatrix
for i in xrange(len(years)):
    for j in flights[i]:
        data_matrix[flightdict[j], i] = 1.

# plot matrix
plt.figure(figsize=(10, 6))
plt.imshow(data_matrix, extent=[0,data_matrix.shape[0],0,data_matrix.shape[1] * 5], \
           interpolation='none', cmap='gray')


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Adjusting numerical data
Let's change teh format of all time entries. Instead of having hour : minute, we convert everything in minutes.
 WARNING  \
RUN THIS CELL ONLY ONCE!
Can you write Python code for it?

%%time
ti = lambda x : x/100 + x % 100
dexample['CRS_ARR_TIME_COR'] = dexample.CRS_ARR_TIME.map(ti)
dexample['CRS_DEP_TIME_COR'] = dexample.CRS_DEP_TIME.map(ti)
dexample.drop(['CRS_DEP_TIME', 'CRS_ARR_TIME'], axis = 1, inplace = True)


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Analysis
Let's now look at the importance coefficients, that the average usage of each coefficients in the random forest. As we can see, the main factors in the delay of a flight are the age of the aircraft, the time of the departure and the time of the origin. The fact that the departure and arrival time confirm what we observed in the explorative analysis. However, the role of the age of the aircraft was not very marked in the explorative analysis. Lastly, the weekday has a huge influence.
Notice that the airline and the aircraft model play a very limited role in the delay according to this random forest classifier.
Can you write Python code for it?

features = dexample_class.drop('ARR_DELAY_COR', axis =1).columns
coeffs_class = randomForest2014_class.classifier[len(randomForest2014_class)-1].feature_importances_
u = pd.Series(coeffs_class, index=features)
u.sort(inplace=True, ascending=False)
print u


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Classification task
We look at various paramters for the classifier. Note that there are two loss functions available for the random forest classifier, a Gini loss and an entropy loss. Simulations show they behave similarly!
Can you write Python code for it?

nb_trees = [25, 50, 75, 100, 150, 200, 300, 400, 500, 750, 1000, 2000, 5000]
nb_features = ['sqrt', 'log2']
test_size = 0.4

%%time
randomForest2014_class =  classify_random_forest_class(dexample, ['ARR_DELAY_COR'], 'ARR_DELAY_COR', test_size=test_size, nb_trees=nb_trees, nb_features=nb_features)

randomForest2014_class.head(9)

# save file to /data/ folder
file_path = "cache/predictionData/randomForest2014_class.csv"
randomForest2014_class.to_csv(path_or_buf= file_path)


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Cleaning the data
We will focus on flights between New York and Chicago. When cleaning the data set, we have to remove the following entries:

flights that have been cancelled or diverted. We focus on predicting the delay. As a result, we also remove the columns associated with diverted flights.
colmuns that give the answer. This is the case of many colmuns related to the arrival of the plane
rows where a value is missing
flights whose destination or origin do not correspond to our study aim

Note that data points have to be cleaned in this order because most flights have empty entries for the 'diverted' columns.
Can you write Python code for it?

#entries to be dropped in the analysis
columns_dropped = ['index', 'TAIL_NUM', 'FL_NUM', 'DEP_TIME', 'DEP_DELAY', 'TAXI_OUT', 'WHEELS_OFF', \
                   'WHEELS_ON', 'TAXI_IN', 'ARR_TIME', 'CANCELLED', 'CANCELLATION_CODE', 'AIR_TIME', \
                   'CARRIER_DELAY', 'WEATHER_DELAY', 'NAS_DELAY', 'SECURITY_DELAY', 'LATE_AIRCRAFT_DELAY']

def clean(data, list_col):
    '''
    Creates a dataset by excluding undesirable columns contained in list_col

    Parameters:
    -----------

    data: pandas.DataFrame
       Flight dataframe

    list_col: <list 'string'>
        Comumns to exclude from the data set
    '''

    data.drop(data[data.CANCELLED == 0].index, inplace=True)
    data.drop(list_col, axis=1, inplace=True)
    data.dropna(axis = 0, inplace = True)
    return

%%time
data2014 = clean(rawData2014.copy(), columns_dropped)
print data2014.columns


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Comparing the RMSEs and Mean absolute error we see that the linear regression model over the whole data clearly outperformed the baseline predictor. Also, more data did not really tremendously improve the mean absolute error but we can see a clear variance reduction in the RMSE. To further improve the model, additional variables should be included. One of the next ideas is weather data. As formatted historic weather data is unfortunately not for free available and scraping it for 100Ks entries per year is not feasible, the variables could not be added to the model. Another interesting idea might be to include some more meta data. I.e. an variable measuring the effect of holidays and variables accounting for winds which could be based by geographic location (i.e. a flight in western direction is longer than one in eastern direction b/c of the passat wind.
3.3 Application: Predicting the best flight for a given date and route
Can you write Python code for it?

from datetime import timedelta, datetime, tzinfo
from pytz import timezone
import pytz

def convertToUTC(naive, zonestring="America/New_York"):
    local = pytz.timezone (zonestring)
    local_dt = local.localize(naive, is_dst=None)
    return local_dt.astimezone (pytz.utc)


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Encode categorical variables
Many of the features are categorical - for example the origin airport feature contains strings which represent the code of each airport. To make a classification/regression possible, we need to encode these features as series of indicators. More specifically, if the origin airport feature has n possibles values A1, A2, ... , An, we create n indicator functions I(A1), ..., I(An). Indicator function I(Ak) takes the value 1 if the feature origin airport has  value Ak.
Can you write Python code for it?

encoded_list = ['UNIQUE_CARRIER', 'ORIGIN', 'DEST', 'AIRCRAFT_MFR', 'MONTH','DAY']
dexample = pd.get_dummies(dexample, columns=encoded_list)

dexample.head(3)


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Extracting month and day
The date is given as a string in the format "Year-Month-Day". We exctract the month and the weekday in the following.
 WARNING  \
RUN THIS CELL ONLY ONCE! Because airports are linked to one another,
Can you write Python code for it?

from time import strptime
days = {0:"Mon", 1:"Tues", 2:"Wed", 3:"Thurs", 4:"Fri", 5:"Sat", 6:"Sun"}
months = {1:"Jan", 2:"Feb", 3:"Mar", 4:"Apr", 5:"May", 6:"June", 7:"July", 8:"Aug", 9:"Sep", \
          10:"Oct", 11:"Nov", 12:"Dec"}

def adjust_time(data):
    '''
    Return the month list and the weekday list out of a DataFrame.

    Parameters:
    -----------

    data: pandas.DataFrame
       dataframe that contains a colum of flight dates
    '''

    monlist = np.empty(len(data), dtype = str)
    daylist = np.empty(len(data), dtype = str)

    for i in xrange(len(data)):
        date= strptime(data.FL_DATE.iloc[i], "%Y-%M-%d")
        monlist[i] = months[date.tm_min]
        daylist[i] = days[date.tm_wday]

    return monlist, daylist

%%time
# now add the weekday and the month columns created by the adjust_time
monlist, daylist = adjust_time(dexample)
print "OK"
dexample['MONTH'] = pd.Series(monlist, index=dexample.index)
dexample['DAY'] = pd.Series(daylist, index=dexample.index)
if 'FL_DATE' in dexample.columns:
    dexample.drop('FL_DATE', axis = 1, inplace= True)
print dexample.columns


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
First, let us again establish a baseline which has to be beaten by our model. To get a feeling for a good baseline we pick flights from from New York(all airports) to Chicago(all airports).
Can you write Python code for it?

# first step is to load the actual data and exclude rows that are unnecessary
# a script that produces the csv file can be found in the src folder
# (it might take some while to run, on my macbook up to one hour)
print('loading data...')
bigdf = pd.read_csv('cache/Big5FlightTable.csv')

years = ['2010', '2011', '2012', '2013', '2014']

# specify here which cities should be investigated
city_from = 'New York, NY'
city_to = 'Chicago, IL'

# filter for cities
bigdf = bigdf[(bigdf.ORIGIN_CITY_NAME == city_from) & (bigdf.DEST_CITY_NAME == city_to)]


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
How good did it perform comparing the actual arrival delay?
Can you write Python code for it?

def rmse(y, y_pred):
    return np.sqrt(((y - y_pred)**2).mean())

def mas(y, y_pred):
    return (np.abs(y - y_pred)).mean()

MAS_base = mas(y_test, y_pred)
RMSE_base = rmse(y_test, y_pred)
RMSE_base


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
In a first model, we use OLS to get a feeling of what is achievable using a reduced dataset. The motivation for this is mainly to see whether this might be used to speedup the whole process. As the RMSE reveals, we did not do better than the base classifier. What happened?
Can you write Python code for it?


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
In the next step, before fitting the actual model categorical variables need to be encoded as binary features (they have no order!) and numerical features shall be normalized.
Can you write Python code for it?

# split data into numerical and categorical features
numericalFeat = bigdf[['DISTANCE', 'AIRCRAFT_AGE']].astype('float')
categoricalFeat = bigdf[['MONTH', 'DAY_OF_MONTH', 'ORIGIN', 'DEST', \
                         'HOUR_OF_ARR', 'HOUR_OF_DEP', 'UNIQUE_CARRIER', \
                         'DAY_OF_WEEK', 'AIRCRAFT_MFR']]


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Looking at the RMSE again is a bit disappointing. We did worse than OLS and the base classifier! This means, it is now time to tackle the unavoidable: Regressing the whole dataset!
As the data becomes rapidly huge (for 1 year ~ 700MB uncompressed CSV, 5 years ~ 3.5GB, 10 years ~ 7.2GB) the code to perform the actual regression has been developed first in an IPython notebook and then run separately. It can be found in the src folder.
Regression over the whole data
In the following 2 models based on 1, 5 years of data are used (saved in BigFlightTable.csv, Big5FlightTable.csv).
Can you write Python code for it?

import json
import seaborn as sns
import matplotlib.ticker as ticker

def load_model(filename):
    mdl = None
    with open(filename, 'r') as f:
        mdl = json.load(f)
    return mdl

# load model results
mdl1 = load_model('results/models/2014model.json') # model with 2014 data (1 year)
mdl5 = load_model('results/models/2010_2014model.json') # model with 2010-2014 data ( 5 years)

labels = np.array(['base predictor', '1 year LM', '5 year LM'])
RMSEs = np.array([RMSE_base, mdl1['RMSE'], mdl5['RMSE']])
MASs = np.array([MAS_base, mdl1['MAS'], mdl5['MAS']])

width = .35
xx = np.arange(len(RMSEs))
plt.bar(xx-width, RMSEs, width, label='RMSE', color=sns.color_palette()[0])
plt.bar(xx , MASs, width, label='MAE', color=sns.color_palette()[2])
plt.legend(loc='best')

# Hide major tick labels
ax = plt.gca()
ax.xaxis.set_major_formatter(ticker.NullFormatter())

# Customize minor tick labels
ax.xaxis.set_minor_locator(ticker.FixedLocator([i for i in xx]))
ax.xaxis.set_minor_formatter(ticker.FixedFormatter([labels[i] for i in xx]))


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Looking at the number of variables (74 here), it might be that either the fit is not good enough or we are ignoring the dependency of flight delays within different routes.
Ridge regression
Can you write Python code for it?

%%time
# Use ridge regression (i.e. Gaussian prior) and vary the lambda parameter using Grid search
from sklearn.linear_model import SGDRegressor
from sklearn.grid_search import GridSearchCV

SGD_params = {'alpha': 10.0 ** -np.arange(-2,8)}
SGD_model = GridSearchCV(SGDRegressor(random_state = 42), \
                         SGD_params, scoring = 'mean_absolute_error', cv = 6) # cross validate 6 times

# train the model, this might take some time...
SGD_model.fit(X_train, y_train)

y_pred = SGD_model.predict(X_test)
rmse(y_test, y_pred)


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
MSE by size of the forest
Can you write Python code for it?

import matplotlib.pyplot as plt
from matplotlib.figure import Figure

# MSE on test set
fig = plt.gcf()
fig.set_size_inches(8, 5)

for key, co in zip(['auto', 'sqrt', 'log2'], ['b', 'g', 'y']):
    x = randomForest2014[randomForest2014.nb_features == key].nb_trees.values
    y = randomForest2014[randomForest2014.nb_features == key].test_score.values
    plt.plot(x, y, alpha = 0.5, c =co, label = key)

plt.ylabel("Mean Square Error")
plt.xlabel("Number of features")
plt.title("MSE on test set by nb of features and nb of trees")
plt.legend(loc = 1)

plt.show()

# MSE on train set
fig = plt.gcf()
fig.set_size_inches(8, 5)

for key, co in zip(['auto', 'sqrt', 'log2'], ['b', 'g', 'y']):
    x = randomForest2014[randomForest2014.nb_features == key].nb_trees.values
    y = randomForest2014[randomForest2014.nb_features == key].train_score.values
    plt.plot(x, y, alpha = 0.5, c =co, label = key)

plt.ylabel("Mean Square Error")
plt.xlabel("Number of features")
plt.title("MSE on train set by nb of features and nb of trees")
plt.legend(loc = 1)

plt.show()


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
On what day is it best to fly in the evening?
Analogously, we now want our model to give us the answer which is the best flight during the busy evening hours (17:00-20:00)
Can you write Python code for it?

db3 = db[(db.DEP_TIME > 1700) & (db.DEP_TIME < 2000)]
dbres3 = db2.sort('PREDICTED_FLIGHT_TIME')
dbres3.head()


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Ordinal least squares regression
Can you write Python code for it?

clf = sklearn.linear_model.LinearRegression()

# fit the model!
clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)
rmse(y_test, y_pred)


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Parameter dashboard and run cell
We test the following parameters.

the number of trees in the forest
the number of features retained in each tree
the test size
Can you write Python code for it?

nb_trees = [25, 50, 75, 100, 150, 200, 300, 400, 500, 600, 700, 800, 900, 1000]
nb_features = ['auto', 'log2', 'sqrt']
test_size = 0.4

%%time
randomForest2014 =  classify_random_forest(dexample, ['ARR_DELAY'], 'ARR_DELAY', test_size=test_size, nb_trees=nb_trees, nb_features=nb_features)

randomForest2014.head()

# save file to /data/ folder
file_path = "cache/predictionData/randomForest2014.csv"
randomForest2014.to_csv(path_or_buf= file_path)


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Random forest functions for a classification task
Can you write Python code for it?

def split(data, list_drop, target, test_size):
    '''
    Splits the data into a training and a test set
    Separates the training and test sets according to a feature set and a target set
    Balance the features sets by retaining only fraction of its points

    Parameters:
    -----------

    data: pandas.DataFrame
       Flight dataframe

    list_drop: <list 'string'>
        List of columns to exclude from the features set

    target: string
        target column along whch we make the target set

    test_size: float
        size of the test set

    '''

    #split the dataset into a training set and a test set
    dtrain, dtest = train_test_split(data, test_size = 0.3)

    Xtrain = dtrain.drop(list_drop, axis=1).values
    ytrain = dtrain[target].values
    Xtest = dtest.drop(list_drop, axis=1).values
    ytest = dtest[target].values

    return Xtrain, ytrain, Xtest, ytest

def score_random_forest_classifier(Xtrain, ytrain, Xtest, ytest, n_trees=10, max_features='auto'):
    '''
    Fits a random forest with (Xtrain ,ytrain)
    Computes the score on (Xtest, ytest)

    Parameters:
    -----------

    Xtrain: numpy 2D array
       Feature training set

    ytrain: numpy 1D array
        Target training set

    Xtest: numpy 2D array
       Feature test set

    ytest: numpy 1D array
        Target test set

    n_trees: int
        number of trees in the forest

    max_features: string or int
        number of features used for every tree

    Outputs:
    --------

    score_train: float
        score on the train set

    score_test: float
        score on the test set

    clf.feature_importances_
        weights of each feature as used by the classifier

    '''

    clf= RandomForestClassifier(n_estimators=n_trees, max_features= max_features)
    clf.fit(Xtrain, ytrain)

    score_train = clf.score(Xtrain, ytrain)
    score_test = clf.score(Xtest, ytest)

    return  score_train, score_test, clf

def...


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Recombining categorical and numerical features allows to start the usual Machine Learning routine (split into test/train, normalize, train/cross-validate, test)
Can you write Python code for it?

numericals_sparse = sparse.csr_matrix(numericalFeat)
# get data matrix & response variable
X_all = sparse.hstack((numericals_sparse, categoricals_encoded))
y_all = bigdf['ARR_DELAY'].values

X_train, X_test, y_train, y_test = train_test_split(X_all, y_all, test_size = 0.2, random_state = 42)

# we have 2 numerical features
X_train_numericals = X_train[:, 0:3].toarray()
X_test_numericals = X_test[:, 0:3].toarray()

# use sklearn tools to standardize numerical features...
scaler = StandardScaler()
scaler.fit(X_train_numericals) # get std/mean from train set
X_train_numericals = sparse.csr_matrix(scaler.transform(X_train_numericals))
X_test_numericals = sparse.csr_matrix(scaler.transform(X_test_numericals))

# update sets
X_train[:, 0:3] = X_train_numericals
X_test[:, 0:3] = X_test_numericals


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Restricting the dataset: Case study NYC - Chicago flights
The dataset has more than 4 millions entries, which makes any data manipulation extremely costly - let alone model fitting. We will therefore make some restrictions on the airports and the airlines considered.
One way to do it is to make a case study, an example that we will follow all along. Here, the example looks at a flight from New York to Chicago.
 Comment
: it is possible to work on a different data set or a different example. However, due to the high complexity of the data set - both by a large number of points and by a large number of high-dimensional categorical variables, it is easier to work on a shrinked version of it.
Can you write Python code for it?

dexample = data2014[data2014.DEST == 'ORD'][(data2014.ORIGIN == 'JFK') | (data2014.ORIGIN == 'LGA') | (data2014.ORIGIN == 'EWR')].copy()
print len(dexample)
dexample.head(3)


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Score by size of the forest
Can you write Python code for it?

import matplotlib.pyplot as plt
from matplotlib.figure import Figure

# Score on test set
fig = plt.gcf()
fig.set_size_inches(8, 5)

for key, co in zip(['sqrt', 'log2'], ['b', 'g']):
    x = randomForest2014_class[randomForest2014_class.nb_features == key].nb_trees.values
    y = randomForest2014_class[randomForest2014_class.nb_features == key].test_score.values
    plt.plot(x, y, alpha = 0.5, c =co, label = key)

plt.ylabel("Score")
plt.xlabel("Number of features")
plt.title("Score on test set by nb of features and nb of trees")
plt.legend(loc = 4)

plt.show()

# Score on train set
fig = plt.gcf()
fig.set_size_inches(8, 5)

for key, co in zip(['sqrt', 'log2'], ['b', 'g']):
    x = randomForest2014_class[randomForest2014_class.nb_features == key].nb_trees.values
    y = randomForest2014_class[randomForest2014_class.nb_features == key].train_score.values
    plt.plot(x, y, alpha = 0.5, c =co, label = key)

plt.ylim(0.935, 0.95)
plt.ylabel("Score")
plt.xlabel("Number of features")
plt.title("Score on train set by nb of features and nb of trees")
plt.legend(loc = 4)

plt.show()


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Split data into training/test sets
First, let's split the data set into a training set and a test set.
Can you write Python code for it?

def split(data, list_drop, target, test_size):
    '''
    Splits the data into a training and a test set
    Separates the training and test sets according to a feature set and a target set
    Balance the features sets by retaining only fraction of its points

    Parameters:
    -----------

    data: pandas.DataFrame
       Flight dataframe

    list_drop: <list 'string'>
        List of columns to exclude from the features set

    target: string
        target column along whch we make the target set

    test_size: float
        size of the test set

    '''

    #split the dataset into a training set and a test set
    dtrain, dtest = train_test_split(data, test_size = 0.3)

    Xtrain = dtrain.drop(list_drop, axis=1).values
    ytrain = dtrain[target].values
    Xtest = dtest.drop(list_drop, axis=1).values
    ytest = dtest[target].values

    return Xtrain, ytrain, Xtest, ytest


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Useful functions
The functions below satisfy various tasks. The first trains a classifier and estimates its scores on the test set and the training set. The second tries different parameter values for the regressor. The last one wraps everything.
Can you write Python code for it?

def score_random_forest(Xtrain, ytrain, Xtest, ytest, n_trees=10, max_features='auto'):
    '''
    Fits a random forest with (Xtrain ,ytrain)
    Computes the score on (Xtest, ytest)

    Parameters:
    -----------

    Xtrain: numpy 2D array
       Feature training set

    ytrain: numpy 1D array
        Target training set

    Xtest: numpy 2D array
       Feature test set

    ytest: numpy 1D array
        Target test set

    n_trees: int
        number of trees in the forest

    max_features: string or int
        number of features used for every tree

    Outputs:
    --------

    score_train: float
        score on the train set

    score_test: float
        score on the test set

    clf.feature_importances_
        weights of each feature as used by the classifier

    '''

    clf= RandomForestRegressor(n_estimators=n_trees, max_features= max_features)
    clf.fit(Xtrain, ytrain)

    score_train = mean_squared_error(clf.predict(Xtrain), ytrain)
    score_test = mean_squared_error(clf.predict(Xtest), ytest)

    return  score_train, score_test, clf

def best_parameters(Xtrain, ytrain, Xtest, ytest, nb_trees, nb_features):
    '''
    Fits sequentially random forest classifiers
    Adds each test score in a pandas.DataFrame with the number of trees, the loss function, the train score,
    and the importance of each features
    Returns a DataFrame with all scores

    Parameters:
    -----------

    Xtrain: numpy 2D array
       Feature training set

    ytrain: numpy 1D array
        Target training set

    Xtest: numpy 2D array
       Feature test set

    ytest: numpy 1D array
        Target test set

    n_trees: <list int>
        list of numbers of trees in the forest

    nb_features: <list int>
        list of number of features in the forest

    Outputs:
    --------

    score_tab: pandas.DataFrame
        DataFrame of scores with associated parameters

    '''

    score_tab =...


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
Using the root mean squared error as one (of many possible) measures, any model that we develop should beat the benchmark of 44.
3.2 Building a linear regression model for prediction
As flight delay changes over the time of day like explored in the data exploration part, we introduce a new feature which models in a categorical variable 10 minute time windows. I.e. for each window we introduce a latent variable that captures some sort of delay influence of this frame. This is done for both the departure and the arrival time. The model here is first developed for the reduced dataset containing the flights between New York / Chicago only.
Can you write Python code for it?

%%time
bigdf['HOUR_OF_ARR'] = 0
bigdf['HOUR_OF_DEP'] = 0

for index, row in bigdf.iterrows():
    bigdf.set_value(index, 'HOUR_OF_ARR', int(row['ARR_TIME']) / 10)
    bigdf.set_value(index, 'HOUR_OF_DEP', int(row['DEP_TIME']) / 10)


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
We are now ready to predict the delay time on our flights!
Can you write Python code for it?

# create for each day info
db['PREDICTED_DELAY'] = 0.
db['FLIGHT_TIME'] = 0
db['PREDICTED_FLIGHT_TIME'] = 0
for index, row in db.iterrows():
    print 'processing {idx}'.format(idx=index)
    y_pred = predictDelayTime(row, mdl)
    db.set_value(index, 'PREDICTED_DELAY', y_pred)

    arr_time = datetime(year=2015, month=row['MONTH'], day=row['DAY_OF_MONTH'], \
                        hour= int(row['ARR_TIME'] / 100), minute=int(row['ARR_TIME'] % 100))
    dep_time = datetime(year=2015, month=row['MONTH'], day=row['DAY_OF_MONTH'], \
                        hour= int(row['DEP_TIME'] / 100), minute=int(row['DEP_TIME'] % 100))

    flight_time_in_min =  (convertToUTC(arr_time) - convertToUTC(dep_time))
    flight_time_in_min = int(flight_time_in_min.total_seconds() / 60)

    db.set_value(index, 'FLIGHT_TIME', flight_time_in_min)
    db.set_value(index, 'PREDICTED_FLIGHT_TIME', y_pred + flight_time_in_min)


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
We memorize the cleaned dataset. This will save us some processing time. The two boxes below save the dataset and recover it from a 'cache/predictionData/' folder.
Can you write Python code for it?

#%%time
# save the data to avoid computing them again
#file_path = "cache/predictionData/predictionData2014.csv"
#data2014.to_csv(path_or_buf= file_path)

#%%time
# recover data2014 from cache/predictionData folder
#file_path = "cache/predictionData/predictionData2014.csv"
#data2014 = pd.read_csv(file_path)
#data2014.drop('Unnamed: 0', axis= 1, inplace = True)
#data2014.head(3)

# test that clean did the job
print "size of raw data set: ", len(rawData2014)
print "number of cancelled: ", len(rawData2014[(rawData2014.CANCELLED == 1)])
print "size of data set: ", len(data2014)


I am working on the file "04_PredictiveAnalytics.ipynb".
The first task is:
We need to center and normalize all continuous data for better results. The idea is that if a feature X has a range of variation that is considerably higher than a feature Y, the variations of X may completely mask the variations of Y and therefore make Y useless.
Can you write Python code for it?

def normalize(array):
    mean = np.mean(array)
    std = np.std(array)
    return [(x - mean)/std for x in array]

def normalize_data(data, feature_list):
    '''
    Normalize data.

    Parameters:
    -----------

    data: pandas.DataFrame
       dataframe

    feature_list: <list 'string'>
        List of features to be normalized
    '''

    for feature in feature_list:
        if feature in data.columns:
            data[feature + '_NOR'] = normalize(data[feature].values)
            data.drop(feature, axis =1, inplace=True)
    return

dexample.drop(dexample[dexample.AIRCRAFT_YEAR =='    '].index, inplace = True)
dexample['AIRCRAFT_YEAR_COR'] = dexample.AIRCRAFT_YEAR.map(lambda x: int(x))
dexample.drop('AIRCRAFT_YEAR', axis = 1, inplace = True)

normalize_data(dexample, ['DISTANCE', 'LAT', 'LONG', 'CRS_ARR_TIME_COR', 'CRS_DEP_TIME_COR', 'AIRCRAFT_YEAR_COR'])

dexample.head(3)


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Bonus Topic - Symbolic Mathematics (SymPy)
Can you write Python code for it?

import sympy as sp

sp.init_printing()     # Turns on pretty printing

np.sqrt(8)

sp.sqrt(8)


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Control Flow
Like all computer languages, Python supports the standard types of control flows including:

IF statements
WHILE loops
FOR loops
Can you write Python code for it?

x = -1

if x > 0:
    print("{0} is a positive number".format(x))
else:
    print("{0} is not a positive number".format(x))

x = 0

if x > 0:
    print("x is positive")
elif x == 0:
    print("x is zero")
else:
    print("x is negative")

y = 0

while y < 12:
    print(s, end=" ")      # specify what charater to print at the end of output
    if y > 6:
        print(e, end=" * ")
    y += 1


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Emoji are unicode characters, so you can use them a well
Can you write Python code for it?

pumpkin = "\U0001F383"
radio_telescope = "\U0001F4E1"

print(pumpkin + radio_telescope)


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Find and Replace
Can you write Python code for it?


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
For loops are different in python.
You do not need to specify the beginning and end values of the loop
Can you write Python code for it?

T = QTable.read('Planets.csv', format='ascii.csv')
T[0:2]

for Value in T['Name']:
    print(Value)

for Idx,Val in enumerate(T['Name']):
    print(Idx,Val)

for Idx,Val in enumerate(T['Name']):

    a = T['a'][Idx] * u.AU

    if (a < 3.0 * u.AU):
        Place = "Inner"

    else:
        Place = "Outer"

    S = "The planet {0:s}, at a distance of {1:.1f}, is in the {2:s} solar system".format(Val,a,Place)

    print(S)


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Formatting works with units
Can you write Python code for it?

NH_D = 34.47 * u.AU

"The New Horizons spacecraft is {0:.1f} from the Sun".format(NH_D)

"The New Horizons spacecraft is at a distance of {0.value:.1f} in the units of {0.unit:s} from the Sun".format(NH_D)


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Line Formatting
Can you write Python code for it?

anotherline = "mY hoVErCRaft iS fUlL oF eEELS"

anotherline.upper()

anotherline.lower()

anotherline.title()

anotherline.capitalize()

anotherline.swapcase()


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Loops are slow in Python. Do not use them if you do not have to!
Can you write Python code for it?

np.random.seed(42)
BigZ = np.random.random(10000)    # 10,000 value array
BigZ[:10]

# This is slow!

for Idx,Val in enumerate(BigZ):
    if (Val > 0.5):
        BigZ[Idx] = 0

BigZ[:10]

%%timeit

for Idx,Val in enumerate(BigZ):
    if (Val > 0.5):
        BigZ[Idx] = 0

# Masks are MUCH faster

mask = np.where(BigZ>0.5)
BigZ[mask] = 0

BigZ[:10]

%%timeit -o

mask = np.where(BigZ>0.5)
BigZ[mask] = 0


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Python supports Unicode characters
You can enter unicode characters directly from the keyboard (depends on your operating system), or you can use the ASCII encoding.
A list of ASCII encoding can be found here.
For example the ASCII ecoding for the greek capital omega is U+03A9, so you can create the character with \U000003A9
Can you write Python code for it?

print("This resistor has a value of 100 k\U000003A9")

Ω = 1e3

Ω + np.pi


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Splitting and Joining
Can you write Python code for it?

line.split()

'_*_'.join(line.split())

' '.join(line.split()[::-1])


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Strings
Strings are just arrays of characters
Can you write Python code for it?


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Use explicit formatting to avoid these errors
Python string formatting has the form:
{Variable Index: Format Type}  .format(Variable)
Can you write Python code for it?

A = 42
B = 1.23456
C = 1.23456e10
D = 'Forty Two'

"I like the number {0:d}".format(A)

"I like the number {0:s}".format(D)

"The number {0:f} is fine, but not a cool as {1:d}".format(B,A)

"The number {0:.3f} is fine, but not a cool as {1:d}".format(C,A)       # 3 places after decimal

"The number {0:.3e} is fine, but not a cool as {1:d}".format(C,A)       # sci notation

"{0:g} and {1:g} are the same format but different results".format(B,C)


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Watch out for variable types!
Can you write Python code for it?

n = 4

print("I would like " + n + " orders of spam")

print("I would like " + str(n) + " orders of spam")


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
Working with strings
Can you write Python code for it?


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
You can compare strings
Can you write Python code for it?

"spam" == "good"

"spam" != "good"

"spam" == "spam"

"sp" < "spam"

"spam" < "eggs"


I am working on the file "04_Python_LoopsAndStrings.ipynb".
The first task is:
You have to explicitly tell SymPy what symbols you want to use
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Calculate the classification accuracy on the validation-set.
Can you write Python code for it?

def validation_accuracy():
    # Get the array of booleans whether the classifications are correct
    # for the validation-set.
    # The function returns two values but we only need the first.
    correct, _ = predict_cls_validation()

    # Calculate the classification accuracy and return it.
    return cls_accuracy(correct)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Calculate the predicted class for the test-set.
Can you write Python code for it?

def predict_cls_test():
    return predict_cls(images = data.test.images,
                       labels = data.test.labels,
                       cls_true = data.test.cls)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Calculate the predicted class for the validation-set.
Can you write Python code for it?

def predict_cls_validation():
    return predict_cls(images = data.validation.images,
                       labels = data.validation.labels,
                       cls_true = data.validation.cls)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Close TensorFlow Session
We are now done using TensorFlow, so we close the session to release its resources.
Can you write Python code for it?

# This has been commented out in case you want to modify and experiment
# with the Notebook without having to restart it.
# session.close()


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Create the directory if it does not exist.
Can you write Python code for it?

if not os.path.exists(save_dir):
    os.makedirs(save_dir)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Data Dimensions
The data dimensions are used in several places in the source-code below. They are defined once so we can use these variables instead of numbers throughout the source-code below.
Can you write Python code for it?

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
img_shape = (img_size, img_size)

# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = 1

# Number of classes, one class for each of 10 digits.
num_classes = 10


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Execute the function now to initialize the variables.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Function for performing a number of optimization iterations so as to gradually improve the variables of the network layers. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples.  The progress is printed every 100 iterations where the validation accuracy is also calculated and saved to a file if it is an improvement.
Can you write Python code for it?

# Counter for total number of iterations performed so far.
total_iterations = 0

def optimize(num_iterations):
    # Ensure we update the global variables rather than local copies.
    global total_iterations
    global best_validation_accuracy
    global last_improvement

    # Start-time used for printing time-usage below.
    start_time = time.time()

    for i in range(num_iterations):

        # Increase the total number of iterations performed.
        # It is easier to update it in each iteration because
        # we need this number several times in the following.
        total_iterations += 1

        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = data.train.next_batch(train_batch_size)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)

        # Print status every 100 iterations and after last iteration.
        if (total_iterations % 100 == 0) or (i == (num_iterations - 1)):

            # Calculate the accuracy on the training-batch.
            acc_train = session.run(accuracy, feed_dict=feed_dict_train)

            # Calculate the accuracy on the validation-set.
            # The function returns 2 values but we only need the first.
            acc_validation, _ = validation_accuracy()

            # If validation accuracy is an improvement over best-known.
            if acc_validation > best_validation_accuracy:
                # Update the best-known validation accuracy.
                best_validation_accuracy = acc_validation

               ...


print(global_step )

# Counter for total number of iterations performed so far.
total_iterations = 0

def optimize(num_iterations):
    # Ensure we update the global variables rather than local copies.
    global total_iterations
    global best_validation_accuracy
    global last_improvement

    # Start-time used for printing time-usage below.
    start_time = time.time()

    for i in range(num_iterations):

        # Increase the total number of iterations performed.
        # It is easier to update it in each iteration because
        # we need this number several times in the following.
        total_iterations += 1

        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = data.train.next_batch(train_batch_size)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)

        # Print status every 100 iterations and after last iteration.
        if (total_iterations % 100 == 0) or (i == (num_iterations - 1)):

            # Calculate the accuracy on the training-batch.
            acc_train = session.run(accuracy, feed_dict=feed_dict_train)

            # Calculate the accuracy on the validation-set.
            # The function returns 2 values but we only need the first.
            acc_validation, _ = validation_accuracy()

            # If validation accuracy is an improvement over best-known.
            if acc_validation > best_validation_accuracy: #* 0.98:
                # Update the best-known validation accuracy.
                best_validation_accuracy = acc_validation...


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Getting the Weights
Further below, we want to plot the weights of the neural network. When the network is constructed using Pretty Tensor, all the variables of the layers are created indirectly by Pretty Tensor. We therefore have to retrieve the variables from TensorFlow.
We used the names layer_conv1 and layer_conv2 for the two convolutional layers. These are also called variable scopes (not to be confused with defaults_scope as described above). Pretty Tensor automatically gives names to the variables it creates for each layer, so we can retrieve the weights for a layer using the layer's scope-name and the variable-name.
The implementation is somewhat awkward because we have to use the TensorFlow function get_variable() which was designed for another purpose; either creating a new variable or re-using an existing variable. The easiest thing is to make the following helper-function.
Can you write Python code for it?

def get_weights_variable(layer_name):
    # Retrieve an existing variable named 'weights' in the scope
    # with the given layer_name.
    # This is awkward because the TensorFlow function was
    # really intended for another purpose.

    with tf.variable_scope(layer_name, reuse=True):
        variable = tf.get_variable('weights')

    return variable


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Helper-function for plotting convolutional weights
Can you write Python code for it?

def plot_conv_weights(weights, input_channel=0):
    # Assume weights are TensorFlow ops for 4-dim variables
    # e.g. weights_conv1 or weights_conv2.

    # Retrieve the values of the weight-variables from TensorFlow.
    # A feed-dict is not necessary because nothing is calculated.
    w = session.run(weights)

    # Print mean and standard deviation.
    print("Mean: {0:.5f}, Stdev: {1:.5f}".format(w.mean(), w.std()))

    # Get the lowest and highest values for the weights.
    # This is used to correct the colour intensity across
    # the images so they can be compared with each other.
    w_min = np.min(w)
    w_max = np.max(w)

    # Number of filters used in the conv. layer.
    num_filters = w.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot all the filter-weights.
    for i, ax in enumerate(axes.flat):
        # Only plot the valid filter-weights.
        if i<num_filters:
            # Get the weights for the i'th filter of the input channel.
            # The format of this 4-dim tensor is determined by the
            # TensorFlow API. See Tutorial #02 for more details.
            img = w[:, :, input_channel, i]

            # Plot image.
            ax.imshow(img, vmin=w_min, vmax=w_max,
                      interpolation='nearest', cmap='seismic')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Helper-function for plotting images
Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Helper-function for showing the performance
Function for printing the classification accuracy on the test-set.
It takes a while to compute the classification for all the images in the test-set, that's why the results are re-used by calling the above functions directly from this function, so the classifications don't have to be recalculated by each function.
Can you write Python code for it?

def print_test_accuracy(show_example_errors=False,
                        show_confusion_matrix=False):

    # For all the images in the test-set,
    # calculate the predicted classes and whether they are correct.
    correct, cls_pred = predict_cls_test()

    # Classification accuracy and the number of correct classifications.
    acc, num_correct = cls_accuracy(correct)

    # Number of images being classified.
    num_images = len(correct)

    # Print the accuracy.
    msg = "Accuracy on Test-Set: {0:.1%} ({1} / {2})"
    print(msg.format(acc, num_correct, num_images))

    # Plot some examples of mis-classifications, if desired.
    if show_example_errors:
        print("Example errors:")
        plot_example_errors(cls_pred=cls_pred, correct=correct)

    # Plot the confusion matrix, if desired.
    if show_confusion_matrix:
        print("Confusion Matrix:")
        plot_confusion_matrix(cls_pred=cls_pred)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Helper-function to perform optimization iterations
There are 55,000 images in the training-set. It takes a long time to calculate the gradient of the model using all these images. We therefore only use a small batch of images in each iteration of the optimizer.
If your computer crashes or becomes very slow because you run out of RAM, then you may try and lower this number, but you may then need to perform more optimization iterations.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Helper-function to plot confusion matrix
Can you write Python code for it?

def plot_confusion_matrix(cls_pred):
    # This is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Get the true classifications for the test-set.
    cls_true = data.test.cls

    # Get the confusion matrix using sklearn.
    cm = confusion_matrix(y_true=cls_true,
                          y_pred=cls_pred)

    # Print the confusion matrix as text.
    print(cm)

    # Plot the confusion matrix as an image.
    plt.matshow(cm)

    # Make various adjustments to the plot.
    plt.colorbar()
    tick_marks = np.arange(num_classes)
    plt.xticks(tick_marks, range(num_classes))
    plt.yticks(tick_marks, range(num_classes))
    plt.xlabel('Predicted')
    plt.ylabel('True')

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Helper-function to plot example errors
Function for plotting examples of images from the test-set that have been mis-classified.
Can you write Python code for it?

def plot_example_errors(cls_pred, correct):
    # This function is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # correct is a boolean array whether the predicted class
    # is equal to the true class for each image in the test-set.

    # Negate the boolean array.
    incorrect = (correct == False)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.test.images[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.test.cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Helper-functions for calculating classifications
This function calculates the predicted classes of images and also returns a boolean array whether the classification of each image is correct.
The calculation is done in batches because it might use too much RAM otherwise. If your computer crashes then you can try and lower the batch-size.
Can you write Python code for it?

# Split the data-set in batches of this size to limit RAM usage.
batch_size = 256

def predict_cls(images, labels, cls_true):
    # Number of images.
    num_images = len(images)

    # Allocate an array for the predicted classes which
    # will be calculated in batches and filled into this array.
    cls_pred = np.zeros(shape=num_images, dtype=np.int)

    # Now calculate the predicted classes for the batches.
    # We will just iterate through all the batches.
    # There might be a more clever and Pythonic way of doing this.

    # The starting index for the next batch is denoted i.
    i = 0

    while i < num_images:
        # The ending index for the next batch is denoted j.
        j = min(i + batch_size, num_images)

        # Create a feed-dict with the images and labels
        # between index i and j.
        feed_dict = {x: images[i:j, :],
                     y_true: labels[i:j, :]}

        # Calculate the predicted class using TensorFlow.
        cls_pred[i:j] = session.run(y_pred_cls, feed_dict=feed_dict)

        # Set the start-index for the next batch to the
        # end-index of the current batch.
        i = j

    # Create a boolean array whether each image is correctly classified.
    correct = (cls_true == cls_pred)

    return correct, cls_pred


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Helper-functions for the classification accuracy
This function calculates the classification accuracy given a boolean array whether each image was correctly classified. E.g. cls_accuracy([True, True, False, False, False]) = 2/5 = 0.4
Can you write Python code for it?

def cls_accuracy(correct):
    # Calculate the number of correctly classified images.
    # When summing a boolean array, False means 0 and True means 1.
    correct_sum = correct.sum()

    # Classification accuracy is the number of correctly classified
    # images divided by the total number of images in the test-set.
    acc = float(correct_sum) / len(correct)

    return acc, correct_sum


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Imports
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
from sklearn.metrics import confusion_matrix
import time
from datetime import timedelta
import math
import os

# Use PrettyTensor to simplify Neural Network construction.
import prettytensor as pt


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Initialize Variables Again
Re-initialize all the variables of the neural network with random values.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Initialize variables
The variables for weights and biases must be initialized before we start optimizing them. We make a simple wrapper-function for this, because we will call it again below.
Can you write Python code for it?

def init_variables():
    session.run(tf.initialize_all_variables())


def init_variables():
    session.run(tf.global_variables_initializer())


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Neural Network
This section implements the Convolutional Neural Network using Pretty Tensor, which is much simpler than a direct implementation in TensorFlow, see Tutorial #03.
The basic idea is to wrap the input tensor x_image in a Pretty Tensor object which has helper-functions for adding new computational layers so as to create an entire neural network. Pretty Tensor takes care of the variable allocation, etc.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Next we have the placeholder variable for the true labels associated with the images that were input in the placeholder variable x. The shape of this placeholder variable is [None, num_classes] which means it may hold an arbitrary number of labels and each label is a vector of length num_classes which is 10 in this case.
Can you write Python code for it?

y_true = tf.placeholder(tf.float32, shape=[None, 10], name='y_true')


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Now that we have wrapped the input image in a Pretty Tensor object, we can add the convolutional and fully-connected layers in just a few lines of source-code.
Note that pt.defaults_scope(activation_fn=tf.nn.relu) makes activation_fn=tf.nn.relu an argument for each of the layers constructed inside the with-block, so that Rectified Linear Units (ReLU) are used for each of these layers. The defaults_scope makes it easy to change arguments for all of the layers.
Can you write Python code for it?

with pt.defaults_scope(activation_fn=tf.nn.relu):
    y_pred, loss = x_pretty.\
        conv2d(kernel=5, depth=16, name='layer_conv1').\
        max_pool(kernel=2, stride=2).\
        conv2d(kernel=5, depth=36, name='layer_conv2').\
        max_pool(kernel=2, stride=2).\
        flatten().\
        fully_connected(size=128, name='layer_fc1').\
        softmax_classifier(num_classes=num_classes, labels=y_true)


with pt.defaults_scope(activation_fn=tf.nn.relu):
    y_pred, loss = x_pretty.\
        conv2d(kernel=5, depth=16, name='layer_conv1').\
        max_pool(kernel=2, stride=2).\
        conv2d(kernel=5, depth=25, name='layer_conv2').\
        max_pool(kernel=2, stride=2).\
        flatten().\
        fully_connected(size=128, name='layer_fc1').\
        softmax_classifier(class_count=10, labels=y_true)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Optimization Method
Pretty Tensor gave us the predicted class-label (y_pred) as well as a loss-measure that must be minimized, so as to improve the ability of the neural network to classify the input images.
It is unclear from the documentation for Pretty Tensor whether the loss-measure is cross-entropy or something else. But we now use the AdamOptimizer to minimize the loss.
Note that optimization is not performed at this point. In fact, nothing is calculated at all, we just add the optimizer-object to the TensorFlow graph for later execution.
Can you write Python code for it?

optimizer = tf.train.AdamOptimizer(learning_rate=1e-4).minimize(loss,  global_step=global_step)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Perform 10,000 optimization iterations
We now perform 10,000 optimization iterations and abort the optimization if no improvement is found on the validation-set in 1000 iterations.
An asterisk * is shown if the classification accuracy on the validation-set is an improvement.
Can you write Python code for it?

optimize(num_iterations=10000)

print_test_accuracy(show_example_errors=True,
                    show_confusion_matrix=True)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Performance Measures
We need a few more performance measures to display the progress to the user.
First we calculate the predicted class number from the output of the neural network y_pred, which is a vector with 10 elements. The class number is the index of the largest element.
Can you write Python code for it?

y_pred_cls = tf.argmax(y_pred, dimension=1)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Performance before any optimization
The accuracy on the test-set is very low because the model variables have only been initialized and not optimized at all, so it just classifies the images randomly.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
PrettyTensor version:
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Restore Best Variables
Re-load all the variables that were saved to file during optimization.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
TensorFlow Graph
The entire purpose of TensorFlow is to have a so-called computational graph that can be executed much more efficiently than if the same calculations were to be performed directly in Python. TensorFlow can be more efficient than NumPy because TensorFlow knows the entire computation graph that must be executed, while NumPy only knows the computation of a single mathematical operation at a time.
TensorFlow can also automatically calculate the gradients that are needed to optimize the variables of the graph so as to make the model perform better. This is because the graph is a combination of simple mathematical expressions so the gradient of the entire graph can be calculated using the chain-rule for derivatives.
TensorFlow can also take advantage of multi-core CPUs as well as GPUs - and Google has even built special chips just for TensorFlow which are called TPUs (Tensor Processing Units) and are even faster than GPUs.
A TensorFlow graph consists of the following parts which will be detailed below:

Placeholder variables used for inputting data to the graph.
Variables that are going to be optimized so as to make the convolutional network perform better.
The mathematical formulas for the convolutional network.
A loss measure that can be used to guide the optimization of the variables.
An optimization method which updates the variables.

In addition, the TensorFlow graph may also contain various debugging statements e.g. for logging data to be displayed using TensorBoard, which is not covered in this tutorial.
Placeholder variables
Placeholder variables serve as the input to the TensorFlow computational graph that we may change each time we execute the graph. We call this feeding the placeholder variables and it is demonstrated further below.
First we define the placeholder variable for the input images. This allows us to change the images that are input to the TensorFlow graph. This is a so-called tensor, which just means that it is a multi-dimensional array. The data-type is set to float32 and...
Can you write Python code for it?

x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')


I am working on the file "04_Save_Restore.ipynb".
The first task is:
TensorFlow Run
Create TensorFlow session
Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
The class-labels are One-Hot encoded, which means that each label is a vector with 10 elements, all of which are zero except for one element. The index of this one element is the class-number, that is, the digit shown in the associated image. We also need the class-numbers as integers for the test- and validation-sets, so we calculate them now.
Can you write Python code for it?

data.test.cls = np.argmax(data.test.labels, axis=1)
data.validation.cls = np.argmax(data.validation.labels, axis=1)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
The classification accuracy for the validation-set will be calculated for every 100 iterations of the optimization function below. The optimization will be stopped if the validation accuracy has not been improved in 1000 iterations. We need a few variables to keep track of this.
Can you write Python code for it?

# Best validation accuracy seen so far.
best_validation_accuracy = 0.0

# Iteration-number for last improvement to validation accuracy.
last_improvement = 0

# Stop optimization if no improvement found in this many iterations.
require_improvement = 1000


I am working on the file "04_Save_Restore.ipynb".
The first task is:
The classification accuracy is calculated by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then taking the average of these numbers.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "04_Save_Restore.ipynb".
The first task is:
The classification accuracy is high again when using the variables that were previously saved.
Note that the classification accuracy may be slightly higher or lower than that reported above, because the variables in the file were chosen to maximize the classification accuracy on the validation-set, but the optimization actually continued for another 1000 iterations after saving those variables, so we are reporting the results for two slightly different sets of variables. Sometimes this leads to slightly better or worse performance on the test-set.
Can you write Python code for it?

print_test_accuracy(show_example_errors=True,
                    show_confusion_matrix=True)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
The convolutional layers expect x to be encoded as a 4-dim tensor so we have to reshape it so its shape is instead [num_images, img_height, img_width, num_channels]. Note that img_height == img_width == img_size and num_images can be inferred automatically by using -1 for the size of the first dimension. So the reshape operation is:
Can you write Python code for it?

x_image = tf.reshape(x, [-1, img_size, img_size, num_channels])


I am working on the file "04_Save_Restore.ipynb".
The first task is:
The convolutional weights are random, but it can be difficult to see any difference from the optimized weights that are shown below. The mean and standard deviation is shown so we can see whether there is a difference.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
The convolutional weights have now been optimized. Compare these to the random weights shown above. They appear to be almost identical. In fact, I first thought there was a bug in the program because the weights look identical before and after optimization.
But try and save the images and compare them side-by-side (you can just right-click the image to save it). You will notice very small differences before and after optimization.
The mean and standard deviation has also changed slightly, so the optimized weights must be different.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
The convolutional weights should be nearly identical to those shown above, although not completely identical because the weights shown above had 1000 optimization iterations more.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
The convolutional weights should now be different from the weights shown above.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
The saved files are often called checkpoints because they may be written at regular intervals during optimization.
This is the directory used for saving and retrieving the data.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Then we create a vector of booleans telling us whether the predicted class equals the true class of each image.
Can you write Python code for it?

correct_prediction = tf.equal(y_pred_cls, y_true_cls)


I am working on the file "04_Save_Restore.ipynb".
The first task is:
This is the path for the checkpoint-file.
Can you write Python code for it?

save_path = os.path.join(save_dir, 'best_validation')


I am working on the file "04_Save_Restore.ipynb".
The first task is:
This means the neural network classifies the images completely randomly again, so the classification accuracy is very poor because it is like random guesses.
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
This was developed using Python 3.5.2 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "04_Save_Restore.ipynb".
The first task is:
Using this helper-function we can retrieve the variables. These are TensorFlow objects. In order to get the contents of the variables, you must do something like: contents = session.run(weights_conv1) as demonstrated further below.
Can you write Python code for it?

weights_conv1 = get_weights_variable(layer_name='layer_conv1')
weights_conv2 = get_weights_variable(layer_name='layer_conv2')


I am working on the file "04_Save_Restore.ipynb".
The first task is:
We could also have a placeholder variable for the class-number, but we will instead calculate it using argmax. Note that this is a TensorFlow operator so nothing is calculated at this point.
Can you write Python code for it?

y_true_cls = tf.argmax(y_true, dimension=1)


I am working on the file "04_Times_Series/01.0 - TF ARRegressor - Estimator + Numpy.ipynb".
The first task is:
2. Define a Data Input Function
Can you write Python code for it?

def generate_input_fn(file_name, mode, header_lines=1, batch_size = None, windows_size = None, tail_count=None):

    dataframe = pd.read_csv(file_name, names=HEADER, skiprows=header_lines)

    if tail_count is not None:
        dataframe = dataframe.tail(tail_count)

    print("Dataset Size: {}".format(len(dataframe)))
    print("")

    data = {
        ts.TrainEvalFeatures.TIMES: dataframe[TIME_INDEX_FEATURE_NAME],
        ts.TrainEvalFeatures.VALUES: dataframe[VALUE_FEATURE_NAMES],
    }

    reader = ts.NumpyReader(data)

    num_threads = multiprocessing.cpu_count() if MULTI_THREADING else 1

    if mode == tf.estimator.ModeKeys.TRAIN:
        input_fn = tf.contrib.timeseries.RandomWindowInputFn(
            reader,
            batch_size=batch_size,
            window_size=windows_size,
            num_threads= num_threads
        )

    elif mode == tf.estimator.ModeKeys.EVAL:
        input_fn = tf.contrib.timeseries.WholeDatasetInputFn(reader)

    return input_fn


I am working on the file "04_Times_Series/01.0 - TF ARRegressor - Estimator + Numpy.ipynb".
The first task is:
3. Define a Create Estimator Function
Can you write Python code for it?

def create_estimator(run_config, hparams):

    estimator = ts.ARRegressor(
        periodicities= hparams.periodicities,
        input_window_size= hparams.input_window_size,
        output_window_size= hparams.output_window_size,
        num_features=len(VALUE_FEATURE_NAMES),
        loss=hparams.loss,
        hidden_layer_sizes = hparams.hidden_units,
        optimizer = tf.train.AdagradOptimizer(learning_rate=hparams.learning_rate),
        config=run_config
    )

    print("")
    print("Estimator Type: {}".format(type(estimator)))
    print("")

    return estimator


I am working on the file "04_Times_Series/01.0 - TF ARRegressor - Estimator + Numpy.ipynb".
The first task is:
4. Instantiate and Estimator with the Required Parameters
Can you write Python code for it?

CHECKPOINT_STEPS=1000

hparams  = tf.contrib.training.HParams(
    training_steps = 10000,
    periodicities = [200],
    input_window_size = 40,
    output_window_size=10,
    batch_size = 15,
    loss = tf.contrib.timeseries.ARModel.NORMAL_LIKELIHOOD_LOSS, # NORMAL_LIKELIHOOD_LOSS | SQUARED_LOSS
    hidden_units = None,
    learning_rate = 0.1

)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.estimator.RunConfig().replace(
    save_checkpoints_steps=CHECKPOINT_STEPS,
    tf_random_seed=19830610,
    model_dir=model_dir
)

print("Model directory: {}".format(run_config.model_dir))
print("Hyper-parameters: {}".format(hparams))
print("")

train_input_fn = generate_input_fn(
    file_name=TRAIN_DATA_FILE,
    mode = tf.estimator.ModeKeys.TRAIN,
    batch_size=hparams.batch_size,
    windows_size = hparams.input_window_size + hparams.output_window_size
)

estimator = create_estimator(run_config, hparams)


I am working on the file "04_Times_Series/01.0 - TF ARRegressor - Estimator + Numpy.ipynb".
The first task is:
5. Train the Estimator
Can you write Python code for it?

if not RESUME_TRAINING:
    shutil.rmtree(model_dir, ignore_errors=True)

tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow()
print("Estimator training started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................")

estimator.train(input_fn=train_input_fn, steps=hparams.training_steps)

time_end = datetime.utcnow()
print(".......................................")
print("Estimator training finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Estimator training elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "04_Times_Series/01.0 - TF ARRegressor - Estimator + Numpy.ipynb".
The first task is:
6. Evalute the Estimator
Can you write Python code for it?

hparams  = tf.contrib.training.HParams(
    training_steps = 10000,
    periodicities = [200],
    input_window_size = 40,
    output_window_size=10,
    batch_size = 15,
    loss = tf.contrib.timeseries.ARModel.SQUARED_LOSS, # NORMAL_LIKELIHOOD_LOSS | SQUARED_LOSS
    hidden_units = None,
    learning_rate = 0.1

)

estimator = create_estimator(run_config, hparams)

eval_input_fn = generate_input_fn(
    file_name=TRAIN_DATA_FILE,
    mode = tf.estimator.ModeKeys.EVAL,
)

tf.logging.set_verbosity(tf.logging.WARN)
evaluation = estimator.evaluate(input_fn=eval_input_fn, steps=1)
print("")
print(evaluation.keys())
print("")
print("Evaluation Loss ({}) : {}".format(hparams.loss, evaluation['loss']))

def compute_rmse(a, b):
    rmse =  np.sqrt(np.sum(np.square(a - b)) / len(a))
    return round(rmse,5)

def compute_mae(a, b):
    mae =  np.sqrt(np.sum(np.abs(a - b)) / len(a))
    return round(mae,5)

x_current = evaluation['times'][0]
y_current_actual = evaluation['observed'][0].reshape(-1)
y_current_estimated = evaluation['mean'][0].reshape(-1)

rmse = compute_rmse(y_current_actual, y_current_estimated)
mae = compute_mae(y_current_actual, y_current_estimated)
print("Evaluation RMSE {}".format(rmse))
print("Evaluation MAE {}".format(mae))

plt.figure(figsize=(20, 10))

plt.title("Time Series Data")
plt.plot(x_current, y_current_actual, label='actual')
plt.plot(x_current, y_current_estimated, label='estimated')
plt.xlabel("Time Index")
plt.ylabel("Value")
plt.legend(loc=2)
plt.show()


I am working on the file "04_Times_Series/01.0 - TF ARRegressor - Estimator + Numpy.ipynb".
The first task is:
Steps to use the ARRegressor Estimator API

Define the metadata
Define a data (Numpy) input function
Define a create Estimator function
Instantiate an esimator with the run_config and required params
Train the estimaor
Evaluate the estimator
Predict using the estimator

1. Define Metadata
Can you write Python code for it?

HEADER = ['time_index','value']
TIME_INDEX_FEATURE_NAME = 'time_index'
VALUE_FEATURE_NAMES = ['value']


I am working on the file "04_Times_Series/02.0 - TF ARRegressor - Experiment + CSV.ipynb".
The first task is:
4. Run Experiment
a. Set hyper-params values
Can you write Python code for it?

CHECKPOINT_STEPS=1000

hparams  = tf.contrib.training.HParams(
    training_steps = 10000,
    periodicities = [200],
    input_window_size = 40,
    output_window_size=10,
    batch_size = 15,
    anomaly_prob = 0.5,
    anomaly_dist = "gaussian",
    loss = tf.contrib.timeseries.ARModel.NORMAL_LIKELIHOOD_LOSS, # NORMAL_LIKELIHOOD_LOSS | SQUARED_LOSS
    hidden_units = None,
    learning_rate = 0.1

)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.contrib.learn.RunConfig(
    save_checkpoints_steps=CHECKPOINT_STEPS,
    tf_random_seed=19831060,
    model_dir=model_dir
)

print("Model directory: {}".format(run_config.model_dir))
print("Hyper-parameters: {}".format(hparams))
print("")


I am working on the file "04_Times_Series/02.0 - TF ARRegressor - Experiment + CSV.ipynb".
The first task is:
5. Run the Experiment
Can you write Python code for it?

if not RESUME_TRAINING:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...")

tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow()
print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................")

learn_runner.run(
    experiment_fn=generate_experiment_fn(),
    run_config=run_config,
    schedule="train", #"train_and_evaluate"
    hparams=hparams
)

time_end = datetime.utcnow()
print(".......................................")
print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "04_Times_Series/02.0 - TF ARRegressor - Experiment + CSV.ipynb".
The first task is:
Steps to use the ARRegressor + Experiment API

Define the metadata
Define a data (csv) input function
Define a create Estimator function
Run an Experiment with learn_runner to train, evaluate, and export the model
Predict using the estimator
Serve the saved model

1. Define Metadata
Can you write Python code for it?

TIME_INDEX_FEATURE_NAME = 'time_index'
VALUE_FEATURE_NAMES = ['value']


I am working on the file "04_Times_Series/02.0 - TF ARRegressor - Experiment + CSV.ipynb".
The first task is:
b. Define Experiment Function
Can you write Python code for it?

def generate_experiment_fn(**experiment_args):

    train_input_fn = generate_input_fn(
        TRAIN_DATA_FILES,
        skip_header_lines=1,
        mode = tf.estimator.ModeKeys.TRAIN,
        batch_size=hparams.batch_size,
        windows_size = hparams.input_window_size + hparams.output_window_size
    )

    eval_input_fn = generate_input_fn(
        TRAIN_DATA_FILES,
        skip_header_lines=1,
        mode = tf.estimator.ModeKeys.EVAL,
        windows_size = hparams.input_window_size + hparams.output_window_size
    )

    def _experiment_fn(run_config, hparams):

        estimator = create_estimator(run_config, hparams)

        return tf.contrib.learn.Experiment(
            estimator,
            train_steps=hparams.training_steps,
            eval_steps=1,
            train_input_fn=train_input_fn,
            eval_input_fn=eval_input_fn,
            **experiment_args
        )

    return _experiment_fn


I am working on the file "04_TransportProblem.ipynb".
The first task is:
A third, powerful way to initialize a parameter is using a user-defined function.
This function will be automatically called by pyomo with any possible (i,j) set. In this case pyomo will actually call c_init() six times in order to initialize the model.c parameter.
Can you write Python code for it?

#  Parameter c(i,j)  transport cost in thousands of dollars per case ;
#            c(i,j) = f * d(i,j) / 1000 ;
def c_init(model, i, j):
    return model.f * model.d[i,j] / 1000

model.c = Param(model.i, model.j, initialize=c_init, doc='Transport cost in thousands of dollar per case')


I am working on the file "04_TransportProblem.ipynb".
The first task is:
As we are here looping over two distinct sets, we can see how list comprehension really simplifies the code. The objective function could have being written without list comprehension as:
Can you write Python code for it?

def objective_rule(model):
    obj = 0.0
    for ki in model.i:
        for kj in model.j:
            obj += model.c[ki,kj]*model.x[ki,kj]
    return obj


I am working on the file "04_TransportProblem.ipynb".
The first task is:
By default, the optimization results are stored in the file results.json:
Can you write Python code for it?


I am working on the file "04_TransportProblem.ipynb".
The first task is:
Complete script
Here is the complete script:
Can you write Python code for it?


I am working on the file "04_TransportProblem.ipynb".
The first task is:
Constrains
At this point, it should not be a surprise that constrains are again defined as model objects with the required information passed as parameter in the constructor function.
Can you write Python code for it?

## Define contrains ##
# supply(i)   observe supply limit at plant i
# supply(i) .. sum (j, x(i,j)) =l= a(i)
def supply_rule(model, i):
    return sum(model.x[i,j] for j in model.j) <= model.a[i]

model.supply = Constraint(model.i, rule=supply_rule, doc='Observe supply limit at plant i')
# demand(j)   satisfy demand at market j ;
# demand(j) .. sum(i, x(i,j)) =g= b(j);
def demand_rule(model, j):
    return sum(model.x[i,j] for i in model.i) >= model.b[j]

model.demand = Constraint(model.j, rule=demand_rule, doc='Satisfy demand at market j')


I am working on the file "04_TransportProblem.ipynb".
The first task is:
Finally, if you are very lazy and want to run the script with just ./transport.py (and you are in Linux) add the following lines at the top:
Can you write Python code for it?

#!/usr/bin/env python
# -*- coding: utf-8 -*-


I am working on the file "04_TransportProblem.ipynb".
The first task is:
Parameters
Parameter objects are created specifying the sets over which they are defined and are initialised with either a python dictionary or a scalar:
Can you write Python code for it?

## Define parameters ##
#   Parameters
#       a(i)  capacity of plant i in cases
#         /    seattle     350
#              san-diego   600  /
#       b(j)  demand at market j in cases
#         /    new-york    325
#              chicago     300
#              topeka      275  / ;
model.a = Param(model.i, initialize={'seattle':350,'san-diego':600}, doc='Capacity of plant i in cases')
model.b = Param(model.j, initialize={'new-york':325,'chicago':300,'topeka':275}, doc='Demand at market j in cases')
#  Table d(i,j)  distance in thousands of miles
#                    new-york       chicago      topeka
#      seattle          2.5           1.7          1.8
#      san-diego        2.5           1.8          1.4  ;
dtab = {
    ('seattle',  'new-york') : 2.5,
    ('seattle',  'chicago')  : 1.7,
    ('seattle',  'topeka')   : 1.8,
    ('san-diego','new-york') : 2.5,
    ('san-diego','chicago')  : 1.8,
    ('san-diego','topeka')   : 1.4,
    }
model.d = Param(model.i, model.j, initialize=dtab, doc='Distance in thousands of miles')
#  Scalar f  freight in dollars per case per thousand miles  /90/ ;
model.f = Param(initialize=90, doc='Freight in dollars per case per thousand miles')


I am working on the file "04_TransportProblem.ipynb".
The first task is:
Retrieving the Output
We use the pyomo_postprocess() function to retrieve the output and do something with it.  For example, we could display solution values (see below), plot a graph with matplotlib or save it in a csv file.
This function is called by pyomo after the solver has finished.
Can you write Python code for it?

## Display of the output ##
# Display x.l, x.m ;
def pyomo_postprocess(options=None, instance=None, results=None):
    model.x.display()


I am working on the file "04_TransportProblem.ipynb".
The first task is:
Set Definitions
Sets are created as attributes object of the main model objects and all the information is given as parameter in the constructor function. Specifically, we are passing to the constructor the initial elements of the set and a documentation string to keep track on what our set represents:
Can you write Python code for it?

## Define sets ##
#  Sets
#       i   canning plants   / seattle, san-diego /
#       j   markets          / new-york, chicago, topeka / ;
model.i = Set(initialize=['seattle','san-diego'], doc='Canning plans')
model.j = Set(initialize=['new-york','chicago', 'topeka'], doc='Markets')


I am working on the file "04_TransportProblem.ipynb".
The first task is:
The above code take advantage of list comprehensions, a powerful feature of the python language that provides a concise way to loop over a list. If we take the supply_rule as example, this is actually called two times by pyomo (once for each of the elements of i). Without list comprehensions we would have had to write our function using a for loop, like:
Can you write Python code for it?

def supply_rule(model, i):
    supply = 0.0
    for j in model.j:
        supply += model.x[i,j]
    return supply <= model.a[i]


I am working on the file "04_TransportProblem.ipynb".
The first task is:
Variables
Similar to parameters, variables are created specifying their domain(s). For variables we can also specify the upper/lower bounds in the constructor.
Differently from GAMS, we don't need to define the variable that is on the left hand side of the objective function.
Can you write Python code for it?

## Define variables ##
#  Variables
#       x(i,j)  shipment quantities in cases
#       z       total transportation costs in thousands of dollars ;
#  Positive Variable x ;
model.x = Var(model.i, model.j, bounds=(0.0,None), doc='Shipment quantities in case')


I am working on the file "04_TransportProblem.ipynb".
The first task is:
We can print model structure information with model.pprint() (“pprint” stand for “pretty print”).
Results are also by default saved in a results.json file or, if PyYAML is installed in the system, in results.yml.
Editing and Running the Script
Differently from GAMS, you can use whatever editor environment you wish to code a pyomo script. If you don't need debugging features, a simple text editor like Notepad++ (in windows), gedit or kate (in Linux) will suffice. They already have syntax highlight for python.
If you want advanced features and debugging capabilities you can use a dedicated Python IDE, like e.g. Spyder.
You will normally run the script as pyomo solve –solver=glpk transport.py. You can output solver specific output adding the option –stream-output. If you want to run the script as python transport.py add the following lines at the end:
Can you write Python code for it?

# This emulates what the pyomo command-line tools does
from pyomo.opt import SolverFactory
import pyomo.environ

opt = SolverFactory("glpk")
results = opt.solve(model)

# sends results to stdout
results.write()

print("\nDisplaying Solution\n" + '-'*60)
pyomo_postprocess(None, None, results)


I am working on the file "04_Twitter_Sentiment_Analysis.ipynb".
The first task is:
2) Define the features that will represent the tweet
We will use the occurrence of words and pair of words (bigrams) as features.
This corresponds to a bag-of-words representation (https://en.wikipedia.org/wiki/Bag-of-words_model): we just count each words (or n-grams) without taking account their order. For document classification, the frequency of occurence of each words is usually taken as a feature. In the case of tweets, they are so short that we can just count each words once.
Using pair of words allows to capture some of the context in which each words appear. This helps capturing the correct meaning of words.
Can you write Python code for it?

from TwSentiment import bag_of_words_and_bigrams

# this will return a dictionary of features,
# we just list the features present in this tweet
bag_of_words_and_bigrams(tokenizer.tokenize(tweet_df.text.iloc[0]))


I am working on the file "04_Twitter_Sentiment_Analysis.ipynb".
The first task is:
Download the logistic regression classifier
https://www.dropbox.com/s/09rw6a85f7ezk31/sklearn_SGDLogReg_.pickle.zip?dl=1
I trained this classifier on this dataset: http://help.sentiment140.com/for-students/, following the approach from this paper: http://cs.stanford.edu/people/alecmgo/papers/TwitterDistantSupervision09.pdf
This is a set of 14 million tweets with emoticons. Tweets containing "sad" emoticons (7 million) are considered negative and tweets with "happy" emoticons (7 million) are considered positive.
I used a Logistic Regression classifier with L2 regularization that I optimized with a 10 fold cross-validation using $F_1$ score as a metric.
Can you write Python code for it?

# the classifier is saved in a "pickle" file
import pickle

with open('sklearn_SGDLogReg_.pickle', 'rb') as fopen:
    classifier_dict = pickle.load(fopen)

# classifier_dict contain the classifier and label mappers
# that I added so that we remember how the classes are
# encoded
classifier_dict


I am working on the file "04_Twitter_Sentiment_Analysis.ipynb".
The first task is:
Extracting features from the tweets
1) Tokenize the tweet in a list of words
This part uses concepts from Naltural Langage Processing.
We will use a tweet tokenizer I built based on TweetTokenizer from NLTK (http://www.nltk.org/).
You can see how it works by opening the file TwSentiment.py. The goal is to process any tweets and extract a list of words taking into account usernames, hashtags, urls, emoticons and all the informal text we can find in tweets. We also want to reduce the number of features by doing some transformations such as putting all the words in lower cases.
Can you write Python code for it?

from TwSentiment import CustomTweetTokenizer

tokenizer = CustomTweetTokenizer(preserve_case=False, # keep Upper cases
                                 reduce_len=True, # reduce repetition of letter to a maximum of three
                                 strip_handles=False, # remove usernames (@mentions)
                                 normalize_usernames=True, # replace all mentions to "@USER"
                                 normalize_urls=True, # replace all urls to "URL"
                                 keep_allupper=True) # keep upercase for words that are all in uppercase

# example
tweet_df.text.iloc[0]

tokenizer.tokenize(tweet_df.text.iloc[0])

# other examples
tokenizer.tokenize('Hey! This is SO cooooooooooooooooool! :)')

tokenizer.tokenize('Hey! This is so cooooooool! :)')


I am working on the file "04_Twitter_Sentiment_Analysis.ipynb".
The first task is:
In our case we have two steps:

Vectorize the textual features (using http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.DictVectorizer.html)
Classify the vectorized features (using http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html)
Can you write Python code for it?

pipline.steps

# this the step that will transform a list of textual features to a vector of zeros and ones
dict_vect = pipline.steps[0][1]

dict_vect.feature_names_

# number of features
len(dict_vect.feature_names_)

# a little example
text = 'Hi all, I am very happy today'
# first tokenize
tokens = tokenizer.tokenize(text)
print(tokens)

# list features
features = bag_of_words_and_bigrams(tokens)
print(features)

# vectorize features
X = dict_vect.transform(features)

print(X.shape)

# X is a special kind of numpy array. beacause it is extremely sparse
# it can be encoded to take less space in memory
# if we want to see it fully, we can use .toarray()

# number of non-zero values in X:
X.toarray().sum()


I am working on the file "04_Twitter_Sentiment_Analysis.ipynb".
The first task is:
Putting it all together:
We will use the class TweetClassifier from TwSentiment.py that puts together this process for us:
Can you write Python code for it?

from TwSentiment import TweetClassifier

twClassifier = TweetClassifier(pipline,
                              tokenizer=tokenizer,
                              feature_extractor=bag_of_words_and_bigrams)

# example
text = 'Hi all, I am very happy today'
twClassifier.classify_text(text)

# the classify text method also accepts a list of text as input
twClassifier.classify_text(['great day today!', 'bad day today...'])
# the classify text method also accepts a list of text as input
# twClassifier.classify_text(['not sad', 'not happy'])


I am working on the file "04_Twitter_Sentiment_Analysis.ipynb".
The first task is:
The classifier is in fact contained in a pipeline.
A sklearn pipeline allows to assemble several transformation of your data (http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html)
Can you write Python code for it?


I am working on the file "04_Twitter_Sentiment_Analysis.ipynb".
The first task is:
The mapping between the list of features and the vector of zeros and ones is done when you train the pipeline with its .fit method.
Classifing the tweet
Now that we have vector representing the presence of features in a tweet, we can apply our logistic regression classifier to compute the probability that a tweet belong to the "sad" or "happy" category
Can you write Python code for it?

classifier = pipline.steps[1][1]

classifier

# access the weights of the logistic regression
classifier.coef_

# we have as many weights as features
classifier.coef_.shape

# plus the intrecept
classifier.intercept_

# let's check the weight associated with a given feature
x = dict_vect.transform({'bad': True})
_, ind = np.where(x.todense())
classifier.coef_[0,ind]

# find the probability for a specific tweet
classifier.predict_proba(X)


I am working on the file "04_Twitter_Sentiment_Analysis.ipynb".
The first task is:
We can now classify our tweets:
Can you write Python code for it?

emo_clas, prob = twClassifier.classify_text(tweet_df.text.tolist())

# add the result to the dataframe

tweet_df['pos_class'] = (emo_clas == 'pos')
tweet_df['pos_prob'] = prob[:,1]

tweet_df.head()

# plot the distribution of probability
import matplotlib.pyplot as plt
%matplotlib inline
h = plt.hist(tweet_df.pos_prob, bins=50)


I am working on the file "04_Twitter_Sentiment_Analysis.ipynb".
The first task is:
We see to numbers, the first one is the probability of the tweet being sad, the second one is the probability of the tweet being happy.
Can you write Python code for it?

# note that:
pipline.predict_proba(features).sum()


I am working on the file "04_Twitter_Sentiment_Analysis.ipynb".
The first task is:
We want to classify users based on the class of their tweets.
Pandas allows to easily group tweets per users using the groupy method of DataFrames:
Can you write Python code for it?

user_group = tweet_df.groupby('user_id')

print(type(user_group))

# let's look at one of the group
groups = user_group.groups
uid = list(groups.keys())[5]
user_group.get_group(uid)

# we need to make a function that takes the dataframe of tweets grouped by users and return the class of the users
def get_user_emo(group):
    num_pos = group.pos_class.sum()
    num_tweets = group.pos_class.size
    if num_pos/num_tweets > 0.5:
        return 'pos'
    elif num_pos/num_tweets < 0.5:
        return 'neg'
    else:
        return 'NA'

# apply the function to each group
user_df = user_group.apply(get_user_emo)

# This is a pandas Series where the index are the user_id
user_df.head(10)


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
At example problem: How long would it take to go 100 km at velocities in velocity?
Can you write Python code for it?

distance = 100 * u.km

time_2 = distance / velocity

time_2


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
Constants
The Astropy package also includes a whole bunch of built-in constants to make your life easier.
A complete list of the built-in constants can be found here.
Can you write Python code for it?


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
Dimentionless Units
Can you write Python code for it?

dimless_y = (1 * u.m) / (1 * u.km)

dimless_y

dimless_y.unit

dimless_y.decompose()   # returns the scale of the dimentionless quanity


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
Functions and Units - (Last week's homework)
Can you write Python code for it?

def find_diameter(H,A):
    result = (1329 / np.sqrt(A)) * (10 ** (-0.2 * H))
    return result * u.km

H = 3.34
A = 0.09

asteroid_diameter = find_diameter(H,A)
asteroid_diameter

def find_mass(D):
    p = 3000 * (u.kg / u.m**3)
    result = p * (1/6) * np.pi * D**3
    return result

asteroid_mass = find_mass(asteroid_diameter)
asteroid_mass

asteroid_mass.decompose()

moon_mass = u.def_unit('Lunar\ Masses', 7.34767309e22 * u.kg)

asteroid_mass.to(moon_mass)


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
Note: because we imported the units package as u, you cannot use u as a variable name.
You can use the units by themselves:
Can you write Python code for it?

u.m    # The unit of meters

u.s    # The unit of seconds

u.m / u.s    # combine them into a composite unit


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
Notice that the units are a bit strange. We can simplify this using .decompose()
Can you write Python code for it?


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
Or they expect the correct type of unit!
Can you write Python code for it?


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
Some math functions only make sense with dimentionless quanities
Can you write Python code for it?


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
Tables and units
Astropy tables are constructed to use units
Can you write Python code for it?

planet_table = QTable.read('Planets.csv', format='ascii.csv')

planet_table[0:3]

planet_table['a'].unit = u.AU
planet_table[0:3]

planet_table['a'].to(u.km)

planet_table['a'].to(u.km)[2]


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
The units package is much more useful when you combine it with scalars or arrays to create Quantities
Can you write Python code for it?

time_1 = 0.25 * u.s

time_1

position = np.arange(1,6,1) * u.m    # np.arange(x,y,z) - create an array of numbers between x and y in steps z

position

velocity = position / time_1

velocity


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
Unit conversion is really easy!
Can you write Python code for it?

time_2.to(u.h)

velocity.to(u.cm / u.h)

velocity.to(imperial.mi / u.h)

velocity.si                     # quick conversion to SI units

velocity.cgs                    # quick conversion to CGS units

density = 3000 * (u.kg / u.m**3)     # From last week's homework

density.to(u.kg / u.km**3)


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
Units make math with Time easy!
Can you write Python code for it?

u.s.find_equivalent_units()

time_3 = 29 * u.day + 7 * u.h + 56 * u.min + 12 * u.s
time_4 = 2 * u.fortnight + 1.33 * u.day

time_3 - time_4

(time_3 - time_4).to(u.min)


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
You can access the number and unit part of the Quantity separately:
Can you write Python code for it?


I am working on the file "04_Units_In_Python.ipynb".
The first task is:
You can define your own units
Can you write Python code for it?

ringo = u.def_unit('Ringos', 3.712 * imperial.yd)

position.to(ringo)

velocity.to(ringo / u.s)


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Altair
Un module intéressant, mais peu mature.
Can you write Python code for it?

import altair as alt

alt.Chart(iris).mark_circle().encode(
    x='petal_width:Q',
    y='petal_length:Q',
    row='species:N',
    color='species:N'
)


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Ce graphique pourrait ressembler à une distribution binormale, ou un coup de 12 dans une porte de grange. Mais on aperçoit des données allignées, parfois de manière rectiligne, parfois en forme d'ellipse. Le tableau datasaurus a une colonne d'information supplémentaire. Utilisons-la comme catégorie pour générer des couleurs différente.
Can you write Python code for it?

sns.FacetGrid(data=datasaurus, hue='dataset', size=5).\
    map(plt.plot, 'x', 'y', marker='o', linestyle='None', alpha=0.5)


I am working on the file "04_Visualisation/4_Visualisation.ipynb".
The first task is:
Ce n'est pas vraiment plus clair. Il y a toutefois des formes qui se dégage, comme des ellipse et des lignes. Et si je regarde bien, j'y vois une étoile. La catégorisation pourrait-elle être mieux utilisée si on segmetnait par colonne au lieu de des couleurs?
Can you write Python code for it?

sns.FacetGrid(data=datasaurus, col='dataset', col_wrap=4).\
    map(plt.plot, 'x', 'y', marker='o', linestyle='None')