I am working on the file "10_model_interpretability/src/loco.ipynb".
The first task is:
Load and prepare data for modeling
Can you write Python code for it?

# load clean data
path = '../../03_regression/data/train.csv'
frame = h2o.import_file(path=path)

# assign target and inputs
y = 'SalePrice'
X = [name for name in frame.columns if name not in [y, 'Id']]


I am working on the file "10_model_interpretability/src/loco.ipynb".
The first task is:
Negative mean reason codes
Can you write Python code for it?

median_mean_loco = loco_ensemble['Mean Local Importance'].sort_values()[:5]
_ = median_mean_loco.plot(kind='bar',
                          title='Negative Mean Reason Codes for the Median of Predicted Sale Price\n',
                          legend=False)


I am working on the file "10_model_interpretability/src/loco.ipynb".
The first task is:
Plot some reason codes for a representative row
Can you write Python code for it?

%matplotlib inline

median_loco = preds[preds['Id'] == int(quantile_dict[50]), :].as_data_frame().drop(['Id', 'predict'], axis=1)
median_loco = median_loco.T.sort_values(by=0)[:5]
_ = median_loco.plot(kind='bar',
                     title='Negative Reason Codes for the Median of Predicted Sale Price\n',
                     legend=False)

median_loco = preds[preds['Id'] == int(quantile_dict[50]), :].as_data_frame().drop(['Id', 'predict'], axis=1)
median_loco = median_loco.T.sort_values(by=0, ascending=False)[:5]
_ = median_loco.plot(kind='bar',
                     title='Positive Reason Codes for the Median of Predicted Sale Price\n',
                     color='r',
                     legend=False)


I am working on the file "10_model_interpretability/src/loco.ipynb".
The first task is:
Positive mean reason codes
Can you write Python code for it?

median_mean_loco = loco_ensemble['Mean Local Importance'].sort_values(ascending=False)[:5]
_ = median_mean_loco.plot(kind='bar',
                          title='Positive Mean Reason Codes for the Median of Predicted Sale Price\n',
                          color='r',
                          legend=False)


I am working on the file "10_model_interpretability/src/loco.ipynb".
The first task is:
Rescore predictive model

Each time leaving one input (covariate) out by setting it to missing
To generate local feature importance values for each decision
Can you write Python code for it?

h2o.no_progress()

for k, i in enumerate(X_reals_decorr):

    # train and predict with Xi set to missing
    valid_loco = h2o.deep_copy(valid, 'valid_loco')
    valid_loco[i] = np.nan
    preds_loco = model.predict(valid_loco)

    # create a new, named column for the LOCO prediction
    preds_loco.columns = [i]
    preds = preds.cbind(preds_loco)

    # subtract the LOCO prediction from
    preds[i] = preds[i] - preds['predict']

    print('LOCO Progress: ' + i + ' (' + str(k+1) + '/' + str(len(X_reals_decorr)) + ') ...')

print('Done.')

preds.head()


I am working on the file "10_model_interpretability/src/loco.ipynb".
The first task is:
Shutdown H2O
Can you write Python code for it?


I am working on the file "10_model_interpretability/src/loco.ipynb".
The first task is:
The numeric values in each column are an estimate of how much each variable contributed to each decision. These values can tell you how a variable and it's values were weighted in any given decision by the model. These values are crucially important for machine learning interpretability and are often to referred to "local feature importance", "reason codes", or "turn-down codes." The latter phrases are borrowed from credit scoring. Credit lenders must provide reasons for turning down a credit application, even for automated decisions. Reason codes can be easily extracted from LOCO local feature importance values, by simply ranking the variables that played the largest role in any given decision.
Helper function for finding quantile indices
Can you write Python code for it?

def get_quantile_dict(y, id_, frame):

    """ Returns the percentiles of a column y as the indices for another column id_.

    Args:
        y: Column in which to find percentiles.
        id_: Id column that stores indices for percentiles of y.
        frame: H2OFrame containing y and id_.

    Returns:
        Dictionary of percentile values and index column values.

    """

    quantiles_df = frame.as_data_frame()
    quantiles_df.sort_values(y, inplace=True)
    quantiles_df.reset_index(inplace=True)

    percentiles_dict = {}
    percentiles_dict[0] = quantiles_df.loc[0, id_]
    percentiles_dict[99] = quantiles_df.loc[quantiles_df.shape[0]-1, id_]
    inc = quantiles_df.shape[0]//10

    for i in range(1, 10):
        percentiles_dict[i * 10] = quantiles_df.loc[i * inc,  id_]

    return percentiles_dict

quantile_dict = get_quantile_dict('predict', 'Id', preds)
print(quantile_dict)


I am working on the file "10_model_interpretability/src/loco.ipynb".
The first task is:
Train a predictive model
Can you write Python code for it?

# train GBM model
model = H2OGradientBoostingEstimator(ntrees=100,
                                     max_depth=10,
                                     distribution='huber',
                                     learn_rate=0.1,
                                     stopping_rounds=5,
                                     seed=12345)

model.train(y=y, x=X_reals_decorr, training_frame=train, validation_frame=valid)

preds = valid['Id'].cbind(model.predict(valid))


I am working on the file "10_model_interpretability/src/mono_xgboost.ipynb".
The first task is:
Calculate partial dependence for 3 important variables
Can you write Python code for it?

par_dep_OverallCond = par_dep('OverallCond', valid[reals], xgb_model1)
par_dep_GrLivArea = par_dep('GrLivArea', valid[reals], xgb_model1)
par_dep_LotArea = par_dep('LotArea', valid[reals], xgb_model1)


I am working on the file "10_model_interpretability/src/mono_xgboost.ipynb".
The first task is:
Calculate values for ICE
Can you write Python code for it?

bins_OverallCond = list(par_dep_OverallCond['OverallCond'])
bins_GrLivArea = list(par_dep_GrLivArea['GrLivArea'])
bins_LotArea = list(par_dep_LotArea['LotArea'])

for i in sorted(quantile_dict.keys()):

    col_name = 'Percentile_' + str(i)

    par_dep_OverallCond[col_name] = par_dep('OverallCond',
                                            valid[valid['Id'] == int(quantile_dict[i])][reals],
                                            xgb_model1,
                                            bins=bins_OverallCond)['partial_dependence']

    par_dep_GrLivArea[col_name] = par_dep('GrLivArea',
                                          valid[valid['Id'] == int(quantile_dict[i])][reals],
                                          xgb_model1,
                                          bins=bins_GrLivArea)['partial_dependence']

    par_dep_LotArea[col_name] = par_dep('LotArea',
                                         valid[valid['Id'] == int(quantile_dict[i])][reals],
                                         xgb_model1,
                                         bins=bins_LotArea)['partial_dependence']


I am working on the file "10_model_interpretability/src/mono_xgboost.ipynb".
The first task is:
Examine monotonic behavior with partial dependence and ICE

Partial dependence is used to view the global, average behavior of a variable under the monotonic model.
ICE is used to view the local behavior of a single instance and single variable under the monotonic model.
Overlaying partial dependence onto ICE in a plot is a convenient way to validate and understand both global and local monotonic behavior.

Helper function for calculating partial dependence
Can you write Python code for it?

def par_dep(xs, frame, model, resolution=20, bins=None):

    """ Creates Pandas dataframe containing partial dependence for a single variable.

    Args:
        xs: Variable for which to calculate partial dependence.
        frame: H2OFrame for which to calculate partial dependence.
        model: XGBoost model for which to calculate partial dependence.
        resolution: The number of points across the domain of xs for which to calculate partial dependence.

    Returns:
        Pandas dataframe containing partial dependence values.

    """

    # don't show progress bars for parse
    h2o.no_progress()

    # init empty Pandas frame w/ correct col names
    par_dep_frame = pd.DataFrame(columns=[xs, 'partial_dependence'])

    # cache original data
    col_cache = h2o.deep_copy(frame[xs], xid='col_cache')

    # determine values at which to calculate partial dependency
    if bins == None:
        min_ = frame[xs].min()
        max_ = frame[xs].max()
        by = (max_ - min_)/resolution
        bins = np.arange(min_, max_, by)

    # calculate partial dependency
    # by setting column of interest to constant
    for j in bins:
        frame[xs] = j
        dframe = xgb.DMatrix(frame.as_data_frame(),)
        par_dep_i = h2o.H2OFrame(model.predict(dframe).tolist())
        par_dep_j = par_dep_i.mean()[0]
        par_dep_frame = par_dep_frame.append({xs:j,
                                              'partial_dependence': par_dep_j},
                                              ignore_index=True)

    # return input frame to original cached state
    frame[xs] = h2o.get_frame('col_cache')

    return par_dep_frame


I am working on the file "10_model_interpretability/src/mono_xgboost.ipynb".
The first task is:
Helper function for finding decile indices
Can you write Python code for it?

def get_quantile_dict(y, id_, frame):

    """ Returns the percentiles of a column y as the indices for another column id_.

    Args:
        y: Column in which to find percentiles.
        id_: Id column that stores indices for percentiles of y.
        frame: H2OFrame containing y and id_.

    Returns:
        Dictionary of percentile values and index column values.

    """

    quantiles_df = frame.as_data_frame()
    quantiles_df.sort_values(y, inplace=True)
    quantiles_df.reset_index(inplace=True)

    percentiles_dict = {}
    percentiles_dict[0] = quantiles_df.loc[0, id_]
    percentiles_dict[99] = quantiles_df.loc[quantiles_df.shape[0]-1, id_]
    inc = quantiles_df.shape[0]//10

    for i in range(1, 10):
        percentiles_dict[i * 10] = quantiles_df.loc[i * inc,  id_]

    return percentiles_dict


I am working on the file "10_model_interpretability/src/mono_xgboost.ipynb".
The first task is:
Load and prepare data for modeling
Can you write Python code for it?


I am working on the file "10_model_interpretability/src/mono_xgboost.ipynb".
The first task is:
Plot variable importance
Can you write Python code for it?


I am working on the file "10_model_interpretability/src/mono_xgboost.ipynb".
The first task is:
Shutdown H2O
Can you write Python code for it?


I am working on the file "10_model_interpretability/src/mono_xgboost.ipynb".
The first task is:
Train XGBoost with monotonicity Constraints
Can you write Python code for it?

ave_y = train['SalePrice'].mean()[0]

# XGBoost uses SVMLight data structure, not Numpy arrays or Pandas data frames
dtrain = xgb.DMatrix(train.as_data_frame()[reals],
                      train.as_data_frame()['SalePrice'])
dvalid = xgb.DMatrix(valid.as_data_frame()[reals],
                      valid.as_data_frame()['SalePrice'])

# tuning parameters
params = {
    'objective': 'reg:linear',
    'booster': 'gbtree',
    'eval_metric': 'rmse',
    'eta': 0.005,
    'subsample': 0.1,
    'colsample_bytree': 0.8,
    'max_depth': 5,
    'reg_alpha' : 0.01,
    'reg_lambda' : 0.0,
    'monotone_constraints':mono_constraints,
    'base_score': ave_y,
    'silent': 0,
    'seed': 12345,
}

# watchlist is used for early stopping
watchlist = [(dtrain, 'train'), (dvalid, 'eval')]

# train model
xgb_model1 = xgb.train(params,
                       dtrain,
                       1000,
                       evals=watchlist,
                       early_stopping_rounds=50,
                       verbose_eval=True)


I am working on the file "10_model_interpretability/src/pdp_ice.ipynb".
The first task is:
From this partial dependence and ICE plot, it can be seen that most individual percentiles of predicted sale price are well-represented by partial dependence, except for the most expensive houses, above the 90th percentile. Pricing for these homes appears to behave differently than the average behavior of other homes under the model.
Shutdown H2O
Can you write Python code for it?


I am working on the file "10_model_interpretability/src/pdp_ice.ipynb".
The first task is:
ICE Plots
Helper function for finding quantile indices
Can you write Python code for it?

def get_quantile_dict(y, id_, frame):

    """ Returns the percentiles of a column y as the indices for another column id_.

    Args:
        y: Column in which to find percentiles.
        id_: Id column that stores indices for percentiles of y.
        frame: H2OFrame containing y and id_.

    Returns:
        Dictionary of percentile values and index column values.

    """

    quantiles_df = frame.as_data_frame()
    quantiles_df.sort_values(y, inplace=True)
    quantiles_df.reset_index(inplace=True)

    percentiles_dict = {}
    percentiles_dict[0] = quantiles_df.loc[0, id_]
    percentiles_dict[99] = quantiles_df.loc[quantiles_df.shape[0]-1, id_]
    inc = quantiles_df.shape[0]//10

    for i in range(1, 10):
        percentiles_dict[i * 10] = quantiles_df.loc[i * inc,  id_]

    return percentiles_dict

quantile_dict = get_quantile_dict('SalePrice', 'Id', valid)


I am working on the file "10_model_interpretability/src/pdp_ice.ipynb".
The first task is:
Load and prepare data for modeling
Can you write Python code for it?

# load clean data
path = '../../03_regression/data/train.csv'
frame = h2o.import_file(path=path)

# assign target and inputs
y = 'SalePrice'
X = [name for name in frame.columns if name not in [y, 'Id']]

# determine column types
# impute
reals, enums = [], []
for key, val in frame.types.items():
    if key in X:
        if val == 'enum':
            enums.append(key)
        else:
            reals.append(key)

_ = frame[reals].impute(method='median')
_ = frame[enums].impute(method='mode')

# split into training an validation, and 30% test
train, valid = frame.split_frame([0.7])


I am working on the file "10_model_interpretability/src/pdp_ice.ipynb".
The first task is:
Simple function for two-dimensional partial dependence

Using H2O for speed and scalability
Can you write Python code for it?

# manually calculate 2-D partial dependence

def par_dep_2d(xs1, xs2, frame, model, resolution=20):

    """ Creates Pandas dataframe containing partial dependence for a two variables.

    Args:
        xs1: First variable for which to calculate partial dependence.
        xs2: Second variable for which to calculate partial dependence.
        frame: Data for which to calculate partial dependence.
        model: Model for which to calculate partial dependence.
        resolution: The number of points across the domain of xs for which to calculate partial dependence.

    Returns:
        Pandas dataframe containing partial dependence values.

    """

    # init empty Pandas frame w/ correct col names
    par_dep_frame = pd.DataFrame(columns=[xs1, xs2, 'partial_dependence'])

    # cache original data
    col_cache1 = frame[xs1]
    col_cache2 = frame[xs2]

    # determine values at which to calculate partial dependency
    # for xs1
    min1_ = frame[xs1].min()
    max1_ = frame[xs1].max()
    by1 = float((max1_ - min1_)/resolution)
    range1 = np.arange(min1_, max1_, by1)

    # determine values at which to calculate partial dependency
    # for xs2
    min2_ = frame[xs2].min()
    max2_ = frame[xs2].max()
    by2 = float((max2_ - min2_)/resolution)
    range2 = np.arange(min2_, max2_, by2)

    # calculate partial dependency
    for j in range1:
        for k in range2:
            frame[xs1] = j
            frame[xs2] = k
            par_dep_i = model.predict(frame)
            par_dep_j = par_dep_i.mean()[0]
            std_j = model.predict(frame).sd()[0]
            pos_std, neg_std = par_dep_j + std_j, par_dep_j - std_j
            par_dep_frame = par_dep_frame.append({xs1:j,
                                                  xs2:k,
                                                  'partial_dependence': par_dep_j},
                                                  ignore_index=True)

    # return input frame to original cached state
    frame[xs1] =...


I am working on the file "10_model_interpretability/src/pdp_ice.ipynb".
The first task is:
Train a predictive model
Can you write Python code for it?

# train GBM model
model = H2OGradientBoostingEstimator(ntrees=100,
                                     max_depth=10,
                                     distribution='huber',
                                     learn_rate=0.1,
                                     stopping_rounds=5,
                                     seed=12345)

model.train(y=y, x=X, training_frame=train, validation_frame=valid)

model.varimp_plot()


I am working on the file "10_model_interpretability/src/pdp_ice.ipynb".
The first task is:
Use matplotlib to plot two-dimensional partial dependence
Can you write Python code for it?

# create 2-D partial dependence plot

# imports
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter

# create 3-D grid
new_shape = (resolution, resolution)
x = np.asarray(par_dep_OverallQual_v_GrLivArea['OverallQual']).reshape(new_shape)
y = np.asarray(par_dep_OverallQual_v_GrLivArea['GrLivArea']).reshape(new_shape)
z = np.asarray(par_dep_OverallQual_v_GrLivArea['partial_dependence']).reshape(new_shape)

fig = plt.figure(figsize=(8,6))
ax = plt.axes(projection='3d')

# set axes labels
ax.set_title('Partial Dependence for Sale Price')
ax.set_xlabel('OverallQual')
ax.set_ylabel('GrLivArea')
ax.set_zlabel('\nSale Price')

# axis decorators/details
#ax.zaxis.set_major_locator(LinearLocator(10))
#ax.zaxis.set_major_formatter(FormatStrFormatter('%.1f'))

# surface
surf = ax.plot_surface(x, y, z,
                       cmap=cm.coolwarm,
                       linewidth=0.05,
                       rstride=1,
                       cstride=1,
                       antialiased=True)
plt.tight_layout()

_ = plt.show()


I am working on the file "10_model_interpretability/src/sensitivity_analysis.ipynb".
The first task is:
Helper function for finding quantile indices
Can you write Python code for it?

def get_quantile_dict(y, id_, frame):

    """ Returns the percentiles of a column y as the indices for another column id_.

    Args:
        y: Column in which to find percentiles.
        id_: Id column that stores indices for percentiles of y.
        frame: H2OFrame containing y and id_.

    Returns:
        Dictionary of percentile values and index column values.

    """

    quantiles_df = frame.as_data_frame()
    quantiles_df.sort_values(y, inplace=True)
    quantiles_df.reset_index(inplace=True)

    percentiles_dict = {}
    percentiles_dict[0] = quantiles_df.loc[0, id_]
    percentiles_dict[99] = quantiles_df.loc[quantiles_df.shape[0]-1, id_]
    inc = quantiles_df.shape[0]//10

    for i in range(1, 10):
        percentiles_dict[i * 10] = quantiles_df.loc[i * inc,  id_]

    return percentiles_dict

sale_quantile_dict = get_quantile_dict('SalePrice', 'Id', preds)
pred_quantile_dict = get_quantile_dict('predict', 'Id', preds)

print('SalePrice quantiles:\n', sale_quantile_dict)
print()
print('prediction quantiles:\n',pred_quantile_dict)


I am working on the file "10_model_interpretability/src/sensitivity_analysis.ipynb".
The first task is:
Impute missing values
Can you write Python code for it?

# determine column types
# impute
reals, enums = [], []
for key, val in frame.types.items():
    if key in X:
        if val == 'enum':
            enums.append(key)
        else:
            reals.append(key)

_ = frame[reals].impute(method='median')
_ = frame[enums].impute(method='mode')

# split into training and validation
train, valid = frame.split_frame([0.7], seed=12345)


I am working on the file "10_model_interpretability/src/sensitivity_analysis.ipynb".
The first task is:
Load and prepare data for modeling
Can you write Python code for it?

# load data
path = '../../03_regression/data/train.csv'
frame = h2o.import_file(path=path)

# assign target and inputs
y = 'SalePrice'
X = [name for name in frame.columns if name not in [y, 'Id']]


I am working on the file "10_model_interpretability/src/sensitivity_analysis.ipynb".
The first task is:
This result may point to unstable predictions for the higher end of SalesPrice.
Shutdown H2O
Can you write Python code for it?


I am working on the file "10_model_interpretability/src/sensitivity_analysis.ipynb".
The first task is:
Train a predictive model
Can you write Python code for it?

# train GBM model
model = H2OGradientBoostingEstimator(ntrees=100,
                                     max_depth=10,
                                     distribution='huber',
                                     learn_rate=0.1,
                                     stopping_rounds=5,
                                     seed=12345)

model.train(y=y, x=X, training_frame=train, validation_frame=valid)

preds = valid.cbind(model.predict(valid))


I am working on the file "10_nuances_de_Fibonacci.ipynb".
The first task is:
III Fibonacci : approche de bas en haut
Can you write Python code for it?

# Approche de bas en haut

def FiboMonte(n) :
    fib=[0 for _ in range(n+1)]
    fib[1]=fib[2]=1
    for i in range(3,n+1) :
        fib[i] =fib[i-1]+fib[i-2]

    return fib[n]

from timeit import timeit
import matplotlib.pyplot as plt
abcisse = [i for i in range(1000,22000,3000)]
ordonneeMonte=[0 for _ in range(len(abcisse))]

ordonneeMonte=[0 for _ in range(len(abcisse))]
for i in range(len(abcisse)) :
    ordonneeMonte[i] = timeit(stmt="FiboMonte(abcisse[i])", number=50, globals=globals())

# Graphique
plt.plot(abcisse, ordonnee, "ro-") # en rouge
plt.plot(abcisse, ordonneeMonte, "go-") # en vert

plt.show()
plt.close()


I am working on the file "10_nuances_de_Fibonacci.ipynb".
The first task is:
Bien sûr, on peut encore écrire du code Cython plus efficace, cf la doc.
Et avec Cython on utilise les types C  donc les entiers bornés, d'où mon utilisation de unsigned long long, le type natif d'entirs C le plus grand possible.
-- Partie rédigée par Lilian Besson (@Naereen)
Graphiques avec plein de comparaison
Can you write Python code for it?

globals().update({'ok': 'ok'})

from timeit import timeit
import numpy as np
import matplotlib.pyplot as plt
import sys

sys.setrecursionlimit(90000)

exp_min, exp_max, nb_tailles = 1, 4, 10
number_timeit = 1000

tailles = np.array(np.ceil(np.logspace(exp_min, exp_max, nb_tailles)), dtype=int)
print(f"Tailles = {tailles}")

fonctions = {
    "Linéaire (for)": fibo1,
    "Linéaire (while)": fibo2,
    "Linéaire (Cython)": fibo3,
}
temps = {}

for nom, fonction in fonctions.items():
    temps[nom] = np.zeros_like(tailles)
    print(f"Estimation des temps pour {nom} ({fonction})...")
    for i, taille in enumerate(tailles):
        temps[nom][i] = timeit(stmt="fonctions[nom](taille)",
                               number=number_timeit,
                               globals={'fonction': fonction, 'taille': taille, 'fonctions': fonctions, 'nom': nom}
                               )
        #print(f"Calling %timeit {fonction}({taille}): ")
        #%timeit fonction(taille)

# Graphique
plt.figure(figsize=(12, 8), dpi=300)
plt.title("Comparaison de trois fonctions calculant Fibo(n) en complexité linéaire ?")

markers = ['+', 'o', 'd']
for i, (nom, fonction) in enumerate(fonctions.items()):
    plt.plot(tailles, temps[nom], label=nom, marker=markers[i % len(fonctions)])

plt.legend()
plt.show()


I am working on the file "10_nuances_de_Fibonacci.ipynb".
The first task is:
D'emblée... c'est mieux non ?
Voyons le graphique :
Can you write Python code for it?

from timeit import timeit
import matplotlib.pyplot as plt
import sys

sys.setrecursionlimit(90000)

abcisse = [i for i in range(1000,22000,3000)]
ordonnee=[0 for _ in range(len(abcisse))]
for i in range(len(abcisse)) :
    ordonnee[i] = timeit(stmt="Fibomem(abcisse[i])", number=100, globals=globals())

# Graphique
plt.plot(abcisse, ordonnee, "ro-") # en rouge

plt.show()
plt.close()

# Variante :

def Fibomem2(n,memo=None) :
    if memo == None : memo={1:1,2:1}
    if n in memo.keys() : return memo[n]
    memo[n] = Fibomem2(n-1,memo)+Fibomem2(n-2,memo)
    return memo[n]
Fibomem2(100)

from timeit import timeit
import matplotlib.pyplot as plt
import sys

sys.setrecursionlimit(90000)

abcisse2 = [i for i in range(1000,15000,3000)]
ordonnee2=[0 for _ in range(len(abcisse2))]
for i in range(len(abcisse2)) :
    ordonnee2[i] = timeit(stmt="Fibomem2(abcisse2[i])", number=100, globals=globals())

# Graphique
plt.plot(abcisse, ordonnee, "ro-") # en rouge
plt.plot(abcisse2, ordonnee2, "go-") # en vert

plt.show()
plt.close()


I am working on the file "10_nuances_de_Fibonacci.ipynb".
The first task is:
Et avec une boucle for sans mémoire ?
Can you write Python code for it?

%%time

def fibo1(n: int) -> int:
    fi, fip1 = 0, 1
    for i in range(n):
        fi, fip1 = fip1, fi + fip1
    return fi

for n in range(100, 110):
    print(f"fibo1({n}) = {fibo1(n)}")

for n in [10, 100, 1000, 10000]:
    print(f"Pour n = {n}, fibo1 a pris ce temps : ", end='', flush=True)
    %timeit fibo1(n)


I am working on the file "10_nuances_de_Fibonacci.ipynb".
The first task is:
Et avec une fonction Cython ?
On va tester de compiler la fonction en C avec Cython pour voir si on obtient des performances "vraiment linéaires" :
Can you write Python code for it?

%load_ext cython

%%cython

def fibo3(unsigned long long n):
    cdef unsigned long long fi = 0
    cdef unsigned long long fip1 = 1
    cdef unsigned long long fip2 = 2
    cdef unsigned long long i = 1
    while i <= n:
        fip2 = fi + fip2
        i += 1
        fi = fip1
        fip1 = fip2
    return fi

%%time
for n in range(100, 110):
    print(f"fibo3({n}) = {fibo3(n)}")

for n in [10, 100, 1000, 10000]:
    print(f"Pour n = {n}, fibo3 a pris ce temps :", end='', flush=True)
    %timeit fibo3(n)


I am working on the file "10_nuances_de_Fibonacci.ipynb".
The first task is:
Que constatez vous ???
Can you write Python code for it?

from timeit import timeit
import matplotlib.pyplot as plt

abcisse = [5,10,15,20,25,28,30]
ordonnee=[0 for _ in range(len(abcisse))]
for i in range(len(abcisse)) :
    ordonnee[i] = timeit("FiboRec(abcisse[i])", number=10, globals=globals())

# Graphique pour le tri sélection en rouge
plt.plot(abcisse, ordonnee, "ro-") # en rouge

plt.show()
plt.close()


I am working on the file "10_nuances_de_Fibonacci.ipynb".
The first task is:
hmmm.... pas fameux !!!
Mais il fallait s'y attendre, la complexité est exponentielle.
II Fibonacci récursif + mémoïsation
Can you write Python code for it?

# Codez la version avec mémoïsation

def Fibo(n,memo) :
    if n in memo.keys() : return memo[n]
    #if n<=2 : return 1
    memo[n] = Fibo(n-1,memo)+Fibo(n-2,memo)
    #memo[n]=1+1
    return memo[n]
def Fibomem(n):
    memo={1:1,2:1}
    return Fibo(n,memo)

for i in range(1,40) : print(i,Fibo(i,{1:1,2:1}))


I am working on the file "10_training_deep_nets_ex9.ipynb".
The first task is:
9.3)
Can you write Python code for it?

n_epochs = 20
batch_size = 50

with tf.Session(graph=graph) as sess:
    init.run()
    for epoch in range(n_epochs):
        for iteration in range(mnist.train.num_examples // batch_size):
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch, training: True}) # <= CHANGED
        acc_train = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
        acc_val = accuracy.eval(feed_dict={X: mnist.validation.images, y: mnist.validation.labels})
        print(epoch, "Train accuracy:", acc_train, "Validation accuracy:", acc_val)

    save_path = saver.save(sess, "./my_mnist_model")


I am working on the file "10_training_deep_nets_ex9.ipynb".
The first task is:
9.3) Update the following training code to feed the value of the training placeholder, where appropriate, then run the code and see if the model performs better than without dropout.
Can you write Python code for it?

n_epochs = 20
batch_size = 50

with tf.Session(graph=graph) as sess:
    init.run()
    for epoch in range(n_epochs):
        for iteration in range(mnist.train.num_examples // batch_size):
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
        acc_train = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
        acc_val = accuracy.eval(feed_dict={X: mnist.validation.images, y: mnist.validation.labels})
        print(epoch, "Train accuracy:", acc_train, "Validation accuracy:", acc_val)

    save_path = saver.save(sess, "./my_mnist_model")


I am working on the file "10_training_deep_nets_ex9.ipynb".
The first task is:
Early Stopping
Can you write Python code for it?

n_epochs = 1000
batch_size = 50

best_acc_val = 0
check_interval = 100
checks_since_last_progress = 0
max_checks_without_progress = 100

with tf.Session(graph=graph) as sess:
    init.run()
    for epoch in range(n_epochs):
        for iteration in range(mnist.train.num_examples // batch_size):
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch, training: True})
            if iteration % check_interval == 0:
                acc_val = accuracy.eval(feed_dict={X: mnist.validation.images[:2000], y: mnist.validation.labels[:2000]})
                if acc_val > best_acc_val:
                    best_acc_val = acc_val
                    checks_since_last_progress = 0
                    saver.save(sess, "./my_best_model_so_far")
                else:
                    checks_since_last_progress += 1
        acc_train = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
        acc_val = accuracy.eval(feed_dict={X: mnist.validation.images[2000:], y: mnist.validation.labels[2000:]})
        print(epoch, "Train accuracy:", acc_train, "Validation accuracy:", acc_val, "Best validation accuracy:", best_acc_val)
        if checks_since_last_progress > max_checks_without_progress:
            print("Early stopping!")
            saver.restore(sess, "./my_best_model_so_far")
            break

    acc_test = accuracy.eval(feed_dict={X: mnist.test.images[2000:], y: mnist.test.labels[2000:]})
    print("Final accuracy on test set:", acc_test)
    save_path = saver.save(sess, "./my_mnist_model")


I am working on the file "10_training_deep_nets_ex9.ipynb".
The first task is:
Saving the model to disk so often slows down training. Let's save to RAM instead:
Can you write Python code for it?

def get_model_params():
    gvars = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES)
    return {gvar.op.name: value for gvar, value in zip(gvars, tf.get_default_session().run(gvars))}

def restore_model_params(model_params):
    gvar_names = list(model_params.keys())
    assign_ops = {gvar_name: tf.get_default_graph().get_operation_by_name(gvar_name + "/Assign")
                  for gvar_name in gvar_names}
    init_values = {gvar_name: assign_op.inputs[1] for gvar_name, assign_op in assign_ops.items()}
    feed_dict = {init_values[gvar_name]: model_params[gvar_name] for gvar_name in gvar_names}
    tf.get_default_session().run(assign_ops, feed_dict=feed_dict)

n_epochs = 1000
batch_size = 50

best_acc_val = 0
check_interval = 100
checks_since_last_progress = 0
max_checks_without_progress = 100
best_model_params = None

with tf.Session(graph=graph) as sess:
    init.run()
    for epoch in range(n_epochs):
        for iteration in range(mnist.train.num_examples // batch_size):
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch, training: True})
            if iteration % check_interval == 0:
                acc_val = accuracy.eval(feed_dict={X: mnist.validation.images[:2000], y: mnist.validation.labels[:2000]})
                if acc_val > best_acc_val:
                    best_acc_val = acc_val
                    checks_since_last_progress = 0
                    best_model_params = get_model_params()
                else:
                    checks_since_last_progress += 1
        acc_train = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
        acc_val = accuracy.eval(feed_dict={X: mnist.validation.images[2000:], y: mnist.validation.labels[2000:]})
        print(epoch, "Train accuracy:", acc_train, "Validation accuracy:", acc_val, "Best validation accuracy:", best_acc_val)
        if checks_since_last_progress > max_checks_without_progress:
            print("Early stopping!")
            break

    if...


I am working on the file "10_training_deep_nets_ex9.ipynb".
The first task is:
Techniques for Training Deep Nets
Using He initialization and the ELU activation function (with the help of a partial()):
Can you write Python code for it?

from functools import partial

n_inputs = 28 * 28
n_hidden1 = 100
n_hidden2 = 100
n_outputs = 10

graph = tf.Graph()
with graph.as_default():
    with tf.name_scope("inputs"):
        X = tf.placeholder(tf.float32, shape=[None, n_inputs], name="X")
        y = tf.placeholder(tf.int32, shape=[None], name="y")

    he_init = tf.contrib.layers.variance_scaling_initializer()

    dense_layer = partial(tf.layers.dense,
                          kernel_initializer=he_init,
                          activation=tf.nn.elu)
    hidden1 = dense_layer(X, n_hidden1, name="hidden1")
    hidden2 = dense_layer(hidden1, n_hidden2, name="hidden2")
    logits = dense_layer(hidden2, n_outputs, activation=None, name="output")
    Y_proba = tf.nn.softmax(logits)

    with tf.name_scope("train"):
        xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=y)
        loss = tf.reduce_mean(xentropy)
        optimizer = tf.train.AdamOptimizer()
        training_op = optimizer.minimize(loss)

    with tf.name_scope("eval"):
        correct = tf.nn.in_top_k(logits, y, 1)
        accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))

    with tf.name_scope("init_and_save"):
        init = tf.global_variables_initializer()
        saver = tf.train.Saver()

n_epochs = 20
batch_size = 50

with tf.Session(graph=graph) as sess:
    init.run()
    for epoch in range(n_epochs):
        for iteration in range(mnist.train.num_examples // batch_size):
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
        acc_train = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
        acc_val = accuracy.eval(feed_dict={X: mnist.validation.images, y: mnist.validation.labels})
        print(epoch, "Train accuracy:", acc_train, "Validation accuracy:", acc_val)

    save_path = saver.save(sess, "./my_mnist_model")


I am working on the file "10b--csv-to-db.ipynb".
The first task is:
Connect to an SQL data source
Can you write Python code for it?

disk_engine = create_engine ('sqlite:///%s' % DB_FILE)


I am working on the file "10b--csv-to-db.ipynb".
The first task is:
Convert CSV to SQLite DB

Load the CSV, chunk-by-chunk, into a DataFrame
Process each chunk by doing some minimal data normalization and stripping out uninteresting columns
Append each chunk to the SQLite database
Can you write Python code for it?

# Convert .csv to .db

CHUNKSIZE = 25000 # Number of rows to read at a time

# List of columns to keep
KEEP_COLS = ['Agency', 'CreatedDate', 'ClosedDate', 'ComplaintType',
             'Descriptor', 'CreatedDate', 'ClosedDate',
             'TimeToCompletion', 'City']

start = dt.datetime.now () # start timer

j = 0
index_start = 1
for df in pd.read_csv(CSV_FILE, chunksize=CHUNKSIZE, iterator=True, encoding='utf-8'):

    # Remove spaces from columns
    df = df.rename(columns={c: c.replace(' ', '') for c in df.columns})

    # Convert to proper date+timestamps
    df['CreatedDate'] = pd.to_datetime(df['CreatedDate'])
    df['ClosedDate'] = pd.to_datetime(df['ClosedDate'])

    df.index += index_start

    for c in df.columns:
        if c not in KEEP_COLS:
            df = df.drop (c, axis=1)

    j += 1
    print ('{} seconds: completed {} rows'.format((dt.datetime.now() - start).seconds, j*chunksize))

    df.to_sql('data', disk_engine, if_exists='append')
    index_start = df.index[-1] + 1


I am working on the file "10b--csv-to-db.ipynb".
The first task is:
Determine output (SQLite DB) filename from the input filename
Can you write Python code for it?

assert CSV_FILE

import re
CSV_BASES = re.findall (r'(.*)\.csv$', CSV_FILE, re.I)

assert len (CSV_BASES) >= 1
CSV_BASE = CSV_BASES[0]

DB_FILE = "%s.db" % CSV_BASE

print ("Converting: %s to %s (an SQLite DB) ..." % (CSV_FILE, DB_FILE))


I am working on the file "10b--csv-to-db.ipynb".
The first task is:
Input (CSV) filename
Can you write Python code for it?

#CSV_FILE = 'NYC-311-2M.csv'
CSV_FILE = None


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
4- Clustering con sklearn
Exploración de datos
Can you write Python code for it?

data.columns

data["class"].value_counts()

data.describe(include="all")


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
4- Clustering con sklearn
Exploración gráfica de datos
Can you write Python code for it?

from matplotlib import pyplot as plt
data.hist(figsize=(12,20))
plt.show()

from matplotlib import pyplot as plt
#pd.scatter_matrix(data, figsize=(12,12), range_padding=0.2)
#plt.show()


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
4- Clustering con sklearn
Graficando lo anterior, obtenemos
Can you write Python code for it?

from matplotlib import pyplot as plt
fig = plt.figure(figsize=(16,8))
plt.plot(clusters, total_distance, 'rs')
plt.xlim(min(clusters)-1, max(clusters)+1)
plt.ylim(0, max(total_distance)*1.1)
plt.show()


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
4- Clustering con sklearn
Normalizacion de datos
Resulta conveniente escalar los datos, para que el algoritmo de clustering funcione mejor
Can you write Python code for it?

from sklearn import preprocessing
X_scaled = preprocessing.scale(X)

print(X_scaled.mean())

print(X_scaled.std())


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
4- Clustering con sklearn
Regla del codo
En todos los casos hemos utilizado que el número de clusters es igual a 3. En caso que no conociéramos este dato, deberíamos graficar la suma de las distancias a los clusters para cada punto, en función del número de clusters.
Can you write Python code for it?

from sklearn.cluster import KMeans

clusters = range(2,20)
total_distance = []
for n_clusters in clusters:
    kmeans = KMeans(n_clusters)
    kmeans.fit(X_scaled)
    pred_labels = kmeans.labels_
    centroids = kmeans.cluster_centers_
    # Get the distances
    distance_for_n = 0
    for k in range(n_clusters):
        points = X_scaled[pred_labels==k]
        aux = (points - centroids[k,:])**2
        distance_for_n += (aux.sum(axis=1)**0.5).sum()
    total_distance.append(distance_for_n)


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
4- Clustering con sklearn
Wine Dataset
Los datos del Wine Dataset son un conjunto de datos clásicos para verificar los algoritmos de clustering.
<img src="images/wine.jpg" alt="" width="600px" align="middle"/>
Los datos corresponden a 3 cultivos diferentes de vinos de la misma región de Italia, y que han sido identificados con las etiquetas 1, 2 y 3.
4- Clustering con sklearn
Wine Dataset
Para cada tipo de vino se realizado 13 análisis químicos:

Alcohol
Malic acid
Ash
Alcalinity of ash
Magnesium
Total phenols
Flavanoids
Nonflavanoid phenols
Proanthocyanins
Color intensity
Hue
OD280/OD315 of diluted wines
Proline

La base de datos contiene 178 muestras distintas en total.
Can you write Python code for it?


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
4- Custering
Magnitudes de los datos
Can you write Python code for it?


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
5- Clasificación
Análisis del error (cont. de)
El siguiente código muestra el error de clasificación, permitiendo verificar que números son confundibles
Can you write Python code for it?

from sklearn.metrics import confusion_matrix as cm
cm = cm(Y_test, Y_pred)
print(cm)

# As in http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html
def plot_confusion_matrix(cm, title='Confusion matrix', cmap=plt.cm.jet):
    plt.figure(figsize=(10,10))
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(10)
    plt.xticks(tick_marks, tick_marks)
    plt.yticks(tick_marks, tick_marks)
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
    plt.show()
    return None

# Compute confusion matrix
plt.figure()
plot_confusion_matrix(cm)

# Normalize the confusion matrix by row (i.e by the number of samples in each class)
cm_normalized = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
plot_confusion_matrix(cm_normalized, title='Normalized confusion matrix')


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
5- Clasificación
Podemos visualizar los datos anteriores utilizando el siguiente código, que requiere que sd_error_for k y mean_error_for_k hayan sido apropiadamente definidos.
Can you write Python code for it?

mean = np.array(mean_error_for_k)
std = np.array(std_error_for_k)
plt.figure(figsize=(12,8))
plt.plot(k_range, mean - std, "k:")
plt.plot(k_range, mean , "r.-")
plt.plot(k_range, mean + std, "k:")
plt.xlabel("Numero de vecinos k")
plt.ylabel("Error de clasificacion")
plt.show()


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
5- Clasificación
Predicción en testing dataset
Ahora que el modelo kNN ha sido completamente entrenado, calcularemos el error de predicción en un set de datos completamente nuevo: el set de testing.
Can you write Python code for it?

# Cargando el archivo data/optdigits.tes
XY_test = np.loadtxt("data/optdigits.test", delimiter=",")
X_test = XY_test[:,:64]
Y_test = XY_test[:, 64]
# Predicción de etiquetas
Y_pred = kNN.predict(X_test)


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
5- Clasificación
Reconocimiento de dígitos
Los datos se encuentran en 2 archivos, data/optdigits.train y data/optdigits.test.
Como su nombre lo indica, el set data/optdigits.train contiene los ejemplos que deben ser usados para entrenar el modelo, mientras que el set data/optdigits.test se utilizará para obtener una estimación del error de predicción.
Ambos archivos comparten el mismo formato: cada línea contiene 65 valores. Los 64 primeros corresponden a la representación de la imagen en escala de grises (0-blanco, 255-negro), y el valor 65 corresponde al dígito de la imágen (0-9).
5- Clasificación
Cargando los datos
Para cargar los datos, utilizamos np.loadtxt con los parámetros extra delimiter (para indicar que el separador será en esta ocasión una coma) y con el dype np.int8 (para que su representación en memoria sea la mínima posible, 8 bits en vez de 32/64 bits para un float).
Can you write Python code for it?

import numpy as np

XY_tv = np.loadtxt("data/optdigits.train", delimiter=",", dtype=np.int8)
print(XY_tv)
X_tv = XY_tv[:,:64]
Y_tv = XY_tv[:, 64]

print(X_tv.shape)
print(Y_tv.shape)
print(X_tv[0,:])
print(X_tv[0,:].reshape(8,8))
print(Y_tv[0])


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
5- Clasificación
Visualización de etiquetas incorrectas
Más interesante que el gráfico anterior, resulta considerar los casos donde los dígitos han sido incorrectamente etiquetados.
Can you write Python code for it?

from matplotlib import pyplot as plt

# Mostrar los datos correctos
mask = (Y_pred!=Y_test)
X_aux = X_test[mask]
Y_aux_true = Y_test[mask]
Y_aux_pred = Y_pred[mask]

# We'll plot the first 100 examples, randomly choosen
nx, ny = 5, 5
fig, ax = plt.subplots(nx, ny, figsize=(12,12))
for i in range(nx):
    for j in range(ny):
        index = j+ny*i
        data  = X_aux[index,:].reshape(8,8)
        label_pred = str(int(Y_aux_pred[index]))
        label_true = str(int(Y_aux_true[index]))
        ax[i][j].imshow(data, interpolation='nearest', cmap=plt.get_cmap('gray_r'))
        ax[i][j].text(0, 0, label_pred, horizontalalignment='center',
                verticalalignment='center', fontsize=10, color='red')
        ax[i][j].text(7, 0, label_true, horizontalalignment='center',
                verticalalignment='center', fontsize=10, color='blue')
        ax[i][j].get_xaxis().set_visible(False)
        ax[i][j].get_yaxis().set_visible(False)
plt.show()


I am working on the file "10b_libreria_sklearn/sklearn.ipynb".
The first task is:
5- Clasificación
Visualizando los datos
Para visualizar los datos utilizaremos el método imshow de pyplot. Resulta necesario convertir el arreglo desde las dimensiones (1,64)  a (8,8) para que la imagen sea cuadrada y pueda distinguirse el dígito. Superpondremos además el label correspondiente al dígito, mediante el método text. Realizaremos lo anterior para los primeros 25 datos del archivo.
Can you write Python code for it?

from matplotlib import pyplot as plt

# Well plot the first nx*ny examples
nx, ny = 5, 5
fig, ax = plt.subplots(nx, ny, figsize=(12,12))
for i in range(nx):
    for j in range(ny):
        index = j+ny*i
        data  = X_tv[index,:].reshape(8,8)
        label = Y_tv[index]
        ax[i][j].imshow(data, interpolation='nearest', cmap=plt.get_cmap('gray_r'))
        ax[i][j].text(7, 0, str(int(label)), horizontalalignment='center',
                verticalalignment='center', fontsize=10, color='blue')
        ax[i][j].get_xaxis().set_visible(False)
        ax[i][j].get_yaxis().set_visible(False)
plt.show()


I am working on the file "11 - Gotchas and Optimization.ipynb".
The first task is:
Floating point errors: loss of precision
Some numbers simply can't be represented perfectly, so when two numbers of very different sizes are used, precision may be lost:
Can you write Python code for it?

var1 = 1/3*10**18  # Start with a large number
var2 = 1
var3 = var1+var2 # Add a small number

print(var3 - var1)# When subtracting a large number from a similarly large number, the smaller number is ignored


I am working on the file "11 - Gotchas and Optimization.ipynb".
The first task is:
If we add one to the sum, we can get pretty different results when using the Python scientific stack (numpy, pandas):
Can you write Python code for it?

print("The python sum is:  ", x+1)
print("The numpy sum is:   ", x_np+1)
print("The pandas sum is:  ", x_pd[0]+1)


I am working on the file "11 - Gotchas and Optimization.ipynb".
The first task is:
Multiply these values by large matricies, and it becomes a bit more clear why some data types might be more practical than others for a given application.
Unavoidable errors
Floating point errors: small remainders
Frequently, the inablility to store a perfect number (i.e. 1/3, which has 0.333 repeating) leads to small bits leftover from calculations:
Can you write Python code for it?


I am working on the file "11 - Gotchas and Optimization.ipynb".
The first task is:
Python integers have arbitrary precision, so the integer sum comes out as expected. In the scientific python stack, the C-style fixed-precision integers actually wrap around to negative numbers when increased.
Optimization
Vectorization
For calculations that involve many steps, often a specific step is repeated. Often, this can be reformulated in a way that computers can do far more efficiently - via matrix math. Reformulating a computation from element-wise operations to array operations is known as vectorization.
Can you write Python code for it?

randArr = np.random.rand(10000)

def scaleLoop(array): # For loop goes through each element and adds one
    for i in range(len(array)):
        array[i] = array[i]+1
    return array

%timeit forloopsum = scaleLoop(randArr)

%timeit vectorsum = randArr + 1 # Numpy was written to perform vectorized operations


I am working on the file "11 - Gotchas and Optimization.ipynb".
The first task is:
The subtraction of the two similar numbers gives a zero result, resulting in a significant loss in precision (i.e. absolute error of infinity).
Integer Overflow
When using integers in the Python scientific stack (numpy, pandas), it is possible for a simple arithmetic to cause the values to 'wrap around' or become negative.
Can you write Python code for it?

x = 2**63 - 1       # Python uses 'int'
x_np = np.array(x)  # numpy automatically assigns to 'int64'
x_pd = pd.Series(x) # pandas automatically assigns to 'int64'
print(x)
print(x_np)
print(x_pd[0])


I am working on the file "11 - Gotchas and Optimization.ipynb".
The first task is:
This comes up frequently when applying logic to floating point arithmetic:
Can you write Python code for it?


I am working on the file "11 Persistence.ipynb".
The first task is:
Because the query result is a sequence, we can use Python’s powerful sequence and iteration tools to process it.
For instance, to select just the name column values, we can run a more specific SQL query and get a list of tuples
Can you write Python code for it?

curs.execute('select name from people')
names = curs.fetchall()
names


I am working on the file "11 Persistence.ipynb".
The first task is:
Exercise no. 2
Load Data from Text files: load table from comma-delimited text file.
Can you write Python code for it?


I am working on the file "11 Persistence.ipynb".
The first task is:
Exercise:
Convert list of row tuples to list of row dicts with field name keys.
Can you write Python code for it?

# %load files/makedicts.py
"""
convert list of row tuples to list of row dicts with field name keys
"""
import sqlite3

def makedicts(cursor, query, params=()):
    cursor.execute(query, params)
    colnames = [desc[0] for desc in cursor.description]
    rowdicts = [dict(zip(colnames, row)) for row in cursor]
    return rowdicts

conn = sqlite3.connect('data/dbase1')
cursor = conn.cursor()
query  = 'select name, pay from people where pay < ?'
lowpay = makedicts(cursor, query, [70000])
for rec in lowpay:
    print(rec)


I am working on the file "11 Persistence.ipynb".
The first task is:
More sophisticated Example
Can you write Python code for it?

curs.description

curs.execute('select * from people')
colnames = [desc[0] for desc in curs.description]
for row in curs:
    print('-' * 30)
    for (name, value) in zip(colnames, row):
        print('%s => %s' % (name, value))


I am working on the file "11 Persistence.ipynb".
The first task is:
The fetchmany call returns a sequence of rows from the result, but not the entire table; you can specify how many rows to grab each time with a parameter or rely on the default as given by the cursor’s arraysize attribute.
Each call gets at most that many more rows from the result or an empty sequence at the end of the table:
Can you write Python code for it?

curs.execute('select * from people')
while True:
    rows = curs.fetchmany()  # size=N optional argument
    if not rows:
        break
    for row in rows:
        print(row)


I am working on the file "11 Persistence.ipynb".
The first task is:
The last command here creates the table called “people” within the database; the name, job, and pay information specifies the columns in this table, as well as their datatypes, using a “type(size)” syntax — two strings and an integer.
Datatypes can be more sophisticated than ours, but we’ll ignore such details here.
In SQLite, the file is the database, so there’s no notion of creating or using a specific database within it, as there is in some systems.
Adding Records
There are three basic statement-based approaches we can use here:

inserting one row at a time;
inserting multiple rows with a single call statement;
using a Python loop.

Here is the simple case
Can you write Python code for it?

curs.execute('insert into people values (?, ?, ?)', ('Bob', 'dev', 50000))
curs.rowcount

sqlite3.paramstyle


I am working on the file "11 Persistence.ipynb".
The first task is:
fetchall vs fetchone
The fetchall call we’ve used so far fetches the entire query result table all at once, as a single sequence (an empty sequence comes back, if the result is empty).
That’s convenient, but it may be slow enough to block the caller temporarily for large result tables or generate substantial network traffic if the server is running remotely (something could easily require a parallel thread in GUI).
To avoid such a bottleneck, we can also grab just one row, or a bunch of rows, at a time with fetchone and fetchmany.
The fetchone call returns the next result row or a None false value at the end of the table:
Can you write Python code for it?

curs.execute('select * from people')
while True:
    row = curs.fetchone()
    if not row:
        break
    print(row)

curs.execute('select * from people')
for row in curs:
    print(row)


I am working on the file "11-InfraredMeasurementAndAnalysis.ipynb".
The first task is:
Calculating a practical instrument function
Load the calibration data, presumably done with a blackbody and at short range.
Can you write Python code for it?

calData = ryptw.JadeCalibrationData('./lwir100mm150us10ND20090910.xml', '.')


I am working on the file "11-InfraredMeasurementAndAnalysis.ipynb".
The first task is:
Finally, print out the calibration data and instrument function:
Can you write Python code for it?


I am working on the file "11-InfraredMeasurementAndAnalysis.ipynb".
The first task is:
Instrument function and calibration
Can you write Python code for it?


I am working on the file "11-InfraredMeasurementAndAnalysis.ipynb".
The first task is:
Instrument function applied for temperature measurement (unity emissivity)
The following measurement observed a large-area blackbody in the lab.
The blackbody set point was 150 $^\circ$C and a recording made. The Cedip Altair s/w calculated 157 $^\circ$C in the centre area of the source.   The ryptw module calculates a temperature of 149 $^\circ$C.
In this analysis we use the calibration file that was analysed above. The instrument (housing) temperature for the relevant frame is extracted from the header frame.  Once the instrument internal temperature is known, it is used to interpolate a curve between the two extreme instrument functions.  With the instrument function known at the operational temperature, the source temperature is calculated from the digital levels in the ptw file.
The ptw file may have nill data for the FPA and sensor4 temperatures, from the file description document it appears that a nill value means that it is 'disabled' (what ever that may mean).  The example below has nill values.
Can you write Python code for it?

import os.path

#load the calibration file and calculate the instrument functon
calData = ryptw.JadeCalibrationData('./lwir100mm150us10ND20090910.xml', '.')

#load the ptw file header
ptwfile  = 'LWIR-BBref-150C-150us.ptw'
header = ryptw.readPTWHeader(ptwfile)
# showHeader(header)

dfpts = pd.DataFrame()

#loading sequence of frames
framesToLoad = [1,2]
for frame in framesToLoad:
    data,fheader = ryptw.getPTWFrame (header, frame)
    strTime ='{}:{}:{}'.format(fheader.h_frameHour,
            fheader.h_frameMinute,fheader.h_frameSecond)
    columns=['Frame','Time','CamT4 C','CamT1 C','FPA K','Row','Col','DL','T deg C','T K','L W/(m2.sr)']

#     print('Sensor Temperature4 {} K '.format(fheader.h_sensorTemp4))
#     print('FPA Temperature {} K '.format(fheader.h_detectorTemp))
#     print('Time {}:{}:{} '.format(fheader.h_frameHour,
#             fheader.h_frameMinute,fheader.h_frameSecond))

    #get the internal temperature from the header and use here
    tempIm = calData.LU.LookupDLTemp(data, header.h_HousingTemperature1-273.15)
    # get a radiance image
    radIm = calData.LU.LookupDLRad(data, header.h_HousingTemperature1-273.15)

#     print('Instrument temperature during measurement was {:.2f} C'.format(header.h_HousingTemperature1-273.15))

#     print('Temperature at ({},{})={} C'.format(160,120,tempIm[160,120]-273.15))
#     print('Temperature at ({},{})={} C'.format(140,110,tempIm[140,110]-273.15))

    for coords in [np.argwhere(np.min(radIm)==radIm)[0],[160,120],[140,110],np.argwhere(np.max(radIm)==radIm)[0]]:
        df2 = pd.DataFrame([[frame,strTime,
                             round(fheader.h_sensorTemp4-273.15,1),
                             round(header.h_HousingTemperature1-273.15,1),
                             round(fheader.h_detectorTemp,1),
                             coords[0],coords[1],int(data[coords[0],coords[1]]),
                             round(tempIm[coords[0],coords[1]]-273.15,1),
                            ...


I am working on the file "11-InfraredMeasurementAndAnalysis.ipynb".
The first task is:
Load single frames and concatenate into a numpy array.  The first index into the array is the frame dimension and the second and third indices are row and columns in the image.  First load the 0'th element to create the data array and then append the additional frames to the original data array.
Can you write Python code for it?

# loading single frames
framesToLoad = list(range(1, 101, 1))
frames = len(framesToLoad)
data, fheaders = ryptw.getPTWFrame (header, framesToLoad[0])
for frame in framesToLoad[1:]:
    f, fheaders  = ryptw.getPTWFrame (header, frame)
    data = np.concatenate((data, f))

rows = header.h_Rows
cols = header.h_Cols
img = data.reshape(frames, rows ,cols)
print(img.shape)


I am working on the file "11-InfraredMeasurementAndAnalysis.ipynb".
The first task is:
Next load a number of frames from the file: select which frames by listing the required frames (zero-based) in the list of frames:
Can you write Python code for it?

#loading sequence of frames
framesToLoad = [3,4,10]
data, fheaders = ryptw.getPTWFrames (header, framesToLoad)
print(data.shape)
print(ptwfile)
print('(1) Data shape={}'.format(data.shape))
ifrm = 0
print('Frame {} summary data:'.format(ifrm))
print('Image data:\n{}'.format(data[ifrm]))
print('Sensor Temperature4 {} K '.format(fheaders[ifrm].h_sensorTemp4))
print('FPA Temperature {} K '.format(fheaders[ifrm].h_detectorTemp))
print('Time {}:{}:{} '.format(fheaders[ifrm].h_frameHour,
        fheaders[ifrm].h_frameMinute,fheaders[ifrm].h_frameSecond))


I am working on the file "11-InfraredMeasurementAndAnalysis.ipynb".
The first task is:
Plot the spectral radiance data for the temperatures used in the calibration process:
Can you write Python code for it?

calData.LU.PlotCalSpecRadiance()
# HTML('<img src="images/lwir100mm0150us10ND20090910-CalRadiance.png" width=800/>')
# display(Image(filename='lwir100mm0150us10ND20090910-CalRadiance.png'))


I am working on the file "11-InfraredMeasurementAndAnalysis.ipynb".
The first task is:
The calibration file contains all the information required to calculate the instrument function of the camera.  It lists the filenames for the various spectral parameters (sensor response, optics transmittance, source emissivity, and atmospheric transmittance).  The file also contains key sensor parameters such as detector pitch and fill factor and focal length.  The most important information is  the list of camera digital levels versus source temperature, for one or two instrument temperatures. By convention, the temperature in this file must be in degrees celcius.  An example file contents is shown below.
Two elements in this file are not measured but must be determined as part of the data analysis process:
JadeCalibration/Caldatas/Caldata@DlFloor
JadeCalibration/Caldatas/Caldata@Power

<JadeCalibration Name="lwir100mm0150us10ND20090910" ID="lwir100mm10ND" Version="" Summary="" >
    <SensorResponse Filename="LWIRsensor.txt"/>
    <OpticsTransmittance Filename="LW100mmLens.txt"/>
    <Filter Filename="LWND10.txt"/>
    <SourceEmis Filename="Unity.txt"/>
    <AtmoTau Filename="Unity.txt"/>
    <DetectorPitch Value="30e-6"/>
    <FillFactor Value="0.7"/>
    <Focallength Value="0.1"/>
    <IntegrationTime Value="150e-6"/>
    <Fnumber Value="2"/>
    <Nu Min="700" Max="1665" Inc="5" />
    <Caldatas>
        <Caldata InstrTemperature="17.1" DlFloor="3625" Power="10" >
            <CalPoint Temp="50" DL="4571" Comment="temp in deg C" />
            <CalPoint Temp="100" DL="5132" />
            <CalPoint Temp="150" DL="5906" />
            <CalPoint Temp="200" DL="6887" />
            <CalPoint Temp="250" DL="8034" />
            <CalPoint Temp="300" DL="9338" />
            <CalPoint Temp="350" DL="10834" />
            <CalPoint Temp="400" DL="12386" />
            <CalPoint Temp="450" DL="14042" />
            <!--...
Can you write Python code for it?

print(calData.Info())

calData.LU.PlotSpectrals()
# HTML('<img src="images/lwir100mm0150us10ND20090910-spectrals.png" width=800 />')
# display(Image(filename='lwir100mm0150us10ND20090910-spectrals.png'))


I am working on the file "11-InfraredMeasurementAndAnalysis.ipynb".
The first task is:
The following graph plots the source radiance versus source temperature, for the instrument spectral response, as defined in the input files.
Can you write Python code for it?

calData.LU.PlotCalTempRadiance()
# HTML('<img src="images/lwir100mm0150us10ND20090910-CalTempRadiance.png" width=800 />')
# display(Image(filename='lwir100mm0150us10ND20090910-CalTempRadiance.png'))


I am working on the file "11-InfraredMeasurementAndAnalysis.ipynb".
The first task is:
The instrument function is nominally a straight line, but flattens out at the low and high ends. At the low end, the (constant) radiance in the optics and instrument dominates the incoming flux on the detector, resulting in an asymptotic lower limit in digital level.  At the top end, the detector or digitiser saturates, resulting in an asymptotic line at the maximum recordable level.  In practice, the top-end asymptotic line is avoided simply by not using the instrument at these signal levels.  The low-end asymptotic line is always present and must always be accounted for.
The low-end asymptotic line is defined by a set of two parameters in the input file (one set each for each instrument temperature):
JadeCalibration/Caldatas/Caldata@DlFloor
JadeCalibration/Caldatas/Caldata@Power

These two parameters are not directly measurable and must be determined by an iterative process.    This is done by entering estimates into the data file and then plotting the two graphs below, changing the values until you are satisfied with the graph.  The DlFloor parameter determines the asymptotic limit, and the Power value determines the shape of the transition between the asymptotic line and the linear portion of the instrument function.
Can you write Python code for it?

calData.LU.PlotCalDLRadiance()
# HTML('')
# display(Image(filename='lwir100mm0150us10ND20090910-CaldlRadiance.png'))

calData.LU.PlotCalDLTemp()
# HTML('')
# display(Image(filename='lwir100mm0150us10ND20090910-CalDLTemp.png'))

calData.LU.PlotCalTempDL()
# HTML('')
# display(Image(filename='lwir100mm0150us10ND20090910-CalDLTemp.png'))


I am working on the file "11-InfraredMeasurementAndAnalysis.ipynb".
The first task is:
The radiance caused by the hot optics in the spectral band is as follows:
Can you write Python code for it?

calData.LU.PlotCalTintRad()
# HTML('<img src="images/lwir100mm0150us10ND20090910-CalInternal.png" width=800/>')
# display(Image(filename='lwir100mm0150us10ND20090910-CalInternal.png'))


I am working on the file "11-InfraredMeasurementAndAnalysis.ipynb".
The first task is:
This information is summarised from (Sec 9.8) of my book.
The imaging radiometers and spectrometer were calibrated (or characterized) against laboratory blackbody sources. In most cases the calibration is an elaborate  process covering several instrument settings, filters, and environmental conditions, but it can be summarized as follows. Calibration entails measuring the instrument output (voltage or digital levels) versus source temperature for a series of source temperatures (ideally over the full dynamic range of the instrument). The set of measurements form a temperature calibration curve (see figure below). When using the instrument, the curve is read 'backward' such that a given instrument voltage returns the appropriate source temperature.  If the test object's emissivity is the same as the laboratory source emissivity, the object's temperature will be equal to the source temperature.
Accurate measurement work requires that the spectral sensor response must be known.  In this case, the calibration source temperature, together with the spectral sensor response, can be used to calculate the inband source radiance.  A new curve is now constructed to relate signal voltage with inband radiance for an extended source completely filling the pixels. This curve can now be used to determine the radiance field incident on the instrument during a measurement.  This radiance value lends itself to further analysis.
The figure below shows typical calibration data.  The top curve relates source temperature and instrument voltage (digital level). The bottom curve relates source irradiance on the sensor entrance aperture to instrument voltage.  For an extended source, the source apparent radiance is related to irradiance by $E=L\omega$, where $\omega$ is the pixel FOV. In this case the instrument was calibrated for three different gain settings (integration times of 30, 120, and 500 $\mu$s) and two different sensor internal temperature conditions (16 and 42.6 $^\circ$C).  Observe the importance of calibration at...
Can you write Python code for it?


I am working on the file "11-Similarity-Based-Learning/SimilarityBased.ipynb".
The first task is:
Nearest-neighbor clustering
Can you write Python code for it?

# Let's fit a nearest-neighbor model to our data, using Euclidean distance...
nearest_neighbor_model = NearestNeighbors(1, metric='euclidean').fit(df[['Speed','Agility']], df['Draft'])

# OK, now let's find the nearest neighbors for some new data points!
samples = np.array([[7,7],[5,5]]) # samples to classify, in [speed, agility] format

fig, ax = plt.subplots()
ax.margins(0.05)
groups = df.groupby('Draft')
for name, group in groups:
    ax.plot(group.Speed, group.Agility, marker='o', linestyle='', ms=12, label=name);
ax.legend(numpoints=1, loc='lower right');
ax.plot(samples[:,0],samples[:,1], marker='o', linestyle='', ms=12, c='red');

nearest_neighbor_model.kneighbors(samples, return_distance=True)

df.Draft.iloc[nearest_neighbor_model.kneighbors(samples, return_distance=False).ravel()] # the kneighbors method returns the index of the
                                                                                # nearest neighbors....

nearest_neighbor_model.kneighbors([[7,7],[5,4]], return_distance=False).ravel()


I am working on the file "11-Similarity-Based-Learning/SimilarityBased.ipynb".
The first task is:
Scaling the Axes
The Euclidean distance can be written in (suggestive) vector notation as:
$d^2(X,Y) = (X-Y)^T I_{n \times n} (X-Y)$
Instead of the $n \times n$ identity matrix, we could use and positive definite matrix.
A positive definite matrix is defined as a matrix $M$ for which $z^T M z \geq 0$ for all real vectors $z$, with equality only if $z$ is the vector of all zeros.
We can use this matrix to appropriately rescale the axes, for example to correct for high variance along a given dimension in our feature space: this gives us the Mahalanobis metric,
$d_M(X,Y) = (X-Y)^T \Sigma^{-1} (X-Y)$,
where $\Sigma$ is the covariance matrix of your data points.
Additional reading:
Distances between words (taking into account the context):
http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/
Let's try some clustering
Can you write Python code for it?

import pandas as pd
from sklearn.neighbors import NearestNeighbors

# read in the basketball draft dataset
df = pd.read_csv('./Table5-2.csv', names=['ID','Speed','Agility','Draft'], skiprows=[0])

df.head()

fig, ax = plt.subplots()
ax.margins(0.05)
groups = df.groupby('Draft')
for name, group in groups:
    ax.plot(group.Speed, group.Agility, marker='o', linestyle='', ms=12, label=name);
ax.legend(numpoints=1, loc='lower right');


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
And the negative:
Can you write Python code for it?


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
And the opposite:
Can you write Python code for it?


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
Embeddings
We can return the actual high-dimensional vector by simply indexing the model with the word as the key:
Can you write Python code for it?


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
First get the sentences:
Can you write Python code for it?

sentences = [sentence for novel in novel_list for sentence in sent_tokenize(novel)]


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
How does this differ from our novels model?

Homework
Heuser's blog post explores an analogy in eighteenth-century thought that Riches are to Virtue what Learning is to Genius. How true is this in the ECCO-trained Word2Vec model? Is it true in the one we trained?
How might we compare word2vec models more generally?

Alternative features for a classification model
This is really cool but what implications does this have for our model of language? Well, word embeddings are simply more precise features of what we've been trying to get at already. That means we can use them in the machine learning models we've been building.
Recall our DTM bag of words classifier:
Can you write Python code for it?

from nltk.corpus import movie_reviews
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, TfidfTransformer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score, train_test_split
from sklearn.utils import shuffle

reviews = [movie_reviews.raw(fileid) for fileid in movie_reviews.fileids()]
judgements = [movie_reviews.categories(fileid)[0] for fileid in movie_reviews.fileids()]

np.random.seed(0)

X, y = shuffle(reviews, judgements, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=50)

# get tfidf values
tfidf = TfidfVectorizer()
tfidf.fit(X)
X_train_transformed = tfidf.transform(X_train)
X_test_transformed = tfidf.transform(X_test)

# build and test logit
logit_class = LogisticRegression(penalty='l2', C=1000)
logit_model = logit_class.fit(X_train_transformed, y_train)
logit_model.score(X_test_transformed, y_test)


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
Now the words:
Can you write Python code for it?

words_by_sentence = [fast_tokenize(sentence.lower()) for sentence in sentences]


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
Pre-Processing
Word2Vec learns about the relationships among words by observing them in context. We'll need to tokenize the words in our corpus while retaining sentence boundaries. Since novels were imported as single strings, we'll first use <i>sent_tokenize</i> to divide them into sentences, and second, we'll split each sentence into its own list of words.
We'll use nltk's sentence tokenizer:
Can you write Python code for it?


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
Schmidt looked at words associated with male and female pronouns to investigate gender. Let's try take all the female pronouns and subtracting the male pronouns:
Can you write Python code for it?


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
So how can we use word embeddings as features? Believe it or not, one of the most effective ways is to simply average each dimension of our embedding across all the words for a given document. Recall our w2v model for novels was trained for 100 dimensions. Creating the features for a specific document would entail first extracting the 100 dimensions for each word, then average each dimension across all words:
Can you write Python code for it?

np.mean([model[w] for w in fast_tokenize(X[0]) if w in model], axis=0)


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
Some fancy matplotlib code...
Can you write Python code for it?

_, ax = plt.subplots(figsize=(10,10))
ax.scatter(embeddings[:,0], embeddings[:,1], alpha=0)
for i in range(len(vectors)):
    ax.annotate(her_tokens[i], ((embeddings[i,0], embeddings[i,1])))


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
This gives us a set X array with 100 features. We can write a function to do this for us for any given string:
Can you write Python code for it?

def w2v_featurize(document, model):
    return np.mean([model[w] for w in fast_tokenize(document) if w in model], axis=0)


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
We can also find cosine distance between two clusters of word vectors. Each cluster is measured as the mean of its words:
Can you write Python code for it?


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
We can find words that don't belong with doesnt_match. It finds the mean vector of the words in the list, and identifies the furthest away:
Can you write Python code for it?


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
We can fit and score the machine learning modle just as before:
Can you write Python code for it?

logit_class = LogisticRegression(random_state=0, penalty='l2', C=1000)
logit_model = logit_class.fit(X_train_w2v, y_train)
logit_model.score(X_test_w2v, y_test)


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
We can then calculate pairwise the cosine distance:
Can you write Python code for it?

from sklearn.metrics import pairwise
dist_matrix = pairwise.pairwise_distances(vectors, metric='cosine')


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
We can then featurize all of our documents:
Can you write Python code for it?

X_train_w2v = [w2v_featurize(d, model) for d in X_train]
X_test_w2v = [w2v_featurize(d, model) for d in X_test]


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
We should now have a list of lists with sentences and words:
Can you write Python code for it?


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
We'll double check that we don't have any empty sentences:
Can you write Python code for it?

words_by_sentence = [sentence for sentence in words_by_sentence if sentence != []]


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
What about Heuser's model?
Can you write Python code for it?

X_train_w2v = [w2v_featurize(d, ecco_model) for d in X_train]
X_test_w2v = [w2v_featurize(d, ecco_model) for d in X_test]
logit_class = LogisticRegression(random_state=0, penalty='l2', C=1000)
logit_model = logit_class.fit(X_train_w2v, y_train)
logit_model.score(X_test_w2v, y_test)


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
What kinds of semantic relationships exist in the diagram above? Are there any words that seem out of place? How do you think they go there?

Saving/Loading Models
We can save the model as a .txt file with the save_word2vec_format method:
Can you write Python code for it?


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
Word Embedding
This lesson is designed to explore features of word embeddings described by Ben Schmidt in his blog post "Rejecting the Gender Binary".
The primary corpus we use consists of the 150 English-language novels made available by the .txtLab at McGill. We also look at a Word2Vec model trained on the ECCO-TCP corpus of 2,350 eighteenth-century literary texts made available by Ryan Heuser. (I have shortened the number of terms in the model by half in order to conserve memory.)
For background on Word2Vec's mechanics, I suggest this brief tutorial by Google, especially the sections "Motivation," "Skip-Gram Model," and "Visualizing."
We'll read in Andrew Piper's corpus we used in our Topic Modeling notebook:
Can you write Python code for it?

metadata_tb = Table.read_table('../09-Topic-Modeling/data/txtlab_Novel150_English.csv')

fiction_path = '../09-Topic-Modeling/data/txtlab_Novel150_English/'

novel_list = []

# Iterate through filenames in metadata table
for filename in metadata_tb['filename']:

    # Read in novel text as single string, make lowercase
    with open(fiction_path + filename, 'r') as file_in:
        novel = file_in.read()

    # Add novel text as single string to master list
    novel_list.append(novel)


I am working on the file "11-Word-Embeddings/01-Word-Embeddings.ipynb".
The first task is:
gensim comes with some handy methods to analyze word relationships. similarity will give us a number from 0-1 based on how similar two words are. If this sounds like cosine similarity for words, you'd be right! It just takes the cosine similarity of these high dimensional vectors:
Can you write Python code for it?


I am working on the file "11-analisis.ipynb".
The first task is:
Análisis de sensibilidad
Se realiza variando una de las variables del problema para determinar el efecto en el indicador utilizado.
Ejemplo.-- Se tiene un proyecto con la siguiente información:

Años de operación: 10
Años de construcción: 1
Producción: 1000 unidades por año
Precio de venta: $ 10 por unidad
Costos operativos: $ 220 por año
Costos de inversión: $ 2000 depreciables a 10 años.
MARR = 10%
Impuesto de renta: 30%

Construya la gráfica de sensibilidad del npv a la producción, el precio de venta y la inversión.
Can you write Python code for it?

## se construye una función que recibe la información
## relevante y retorn el npv

def project(marr,
            produccion,
            precio,
            costo,
            inversion):

    # ingresos operativos
    ingre = cf.cashflow(const_value=[0] + [precio*produccion]*10, start=2018)

    # costos operativos
    opera = cf.cashflow(const_value=[0] + [costo]*10, start=2018)

    # inversión
    inver = cf.cashflow(const_value=[inversion] + [0]*10, start=2018)

    # vida útil de la inversión
    life = cf.cashflow(const_value=[10] + [0]*10, start = 2018)

    # depreciación
    dep = cf.depreciation_sl(costs=inver, life=life)['Depr']

    # utilidad antes de impuestos
    antes = ingre - opera - inver - dep

    tax_rate = cf.interest_rate(const_value=[30]*11, start=2018)

    impue = cf.after_tax_cashflow(antes, tax_rate=tax_rate)

    neto = antes + impue

    npv = cf.timevalue(cflo=neto, prate=cf.interest_rate([marr]*11, start=2018))

    return npv

project(marr=10, produccion=100, precio=10, costo=220, inversion=2000)

## resultados para diferentes valores de la MARR
x=[]
for i in [8, 10, 12]:
    x.append(project( i, 100, 10, 220, 2000))
x

## resultados para diferentes valores de la inversión
[project(10, 100, 10, 220, x) for x in [1600, 1800, 2000, 2200, 2400]]

## resultados para diferentes valores del precio
[project(10, 100, x, 220, 2000) for x in [8, 9, 10, 11, 12]]

import matplotlib.pyplot as plt
%matplotlib inline

precio = [8, 9, 10, 11, 12]
y = [project(0.10, 100, x, 220, 2000) for x in precio]

plt.plot(precio, y)


I am working on the file "11-analisis.ipynb".
The first task is:
Tasa interna de retorno (IRR)
$r^*$ -- tasa interna de retorno (irr). Es la tasa que hace el valor presente igual a cero.
$$PV(r^) = \sum_{t=0}^N \frac{F_t}{(1+r^)^t} ~=~0$$
Ejemplo.-- Calcule la irr para el ejemplo anterior.
Can you write Python code for it?

import cashflows as cf
cflo = cf.cashflow(const_value=[-1000, 400, 360, 320, 280, 240], start=2000)
cf.irr(cflo)

cf.irr([cflo, cflo, cflo])


I am working on the file "11-intro-to-hadoop-03.ipynb".
The first task is:
How different are the number of shuffle bytes between the two jobs?
2.2.2 Optimization through combiner function
Can you write Python code for it?

!hdfs dfs -ls /repository/

!yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar \
    -input /repository/complete-shakespeare.txt \
    -output intro-to-hadoop/output-wordcount-01 \
    -file ./codes/wordcountMapper.py \
    -mapper wordcountMapper.py \
    -file ./codes/wordcountReducer.py \
    -reducer wordcountReducer.py

!yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar \
    -input /repository/complete-shakespeare.txt \
    -output intro-to-hadoop/output-wordcount-02 \
    -file ./codes/wordcountMapper.py \
    -mapper wordcountMapper.py \
    -file ./codes/wordcountReducer.py \
    -reducer wordcountReducer.py \
    -combiner wordcountReducer.py

%%writefile codes/avgGenreCombiner.py
#!/usr/bin/env python

import sys
import csv
import json

genreList = {}

for line in sys.stdin:
    line = line.strip()
    genre, ratingString = line.split("\t", 1)
    ratingInfo = json.loads(ratingString)

    if genre in genreList:
        genreList[genre]["total_rating"] += ratingInfo["total_rating"]
        genreList[genre]["total_count"] += ratingInfo["total_count"]
    else:
        genreList[genre] = {}
        genreList[genre]["total_rating"] = ratingInfo["total_rating"]
        genreList[genre]["total_count"] = 1

for genre in genreList:
    print ("%s\t%s" % (genre, json.dumps(genreList[genre])))

!hdfs dfs -rm -r intro-to-hadoop/output-movielens-06
!yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar \
    -input /repository/movielens/ratings.csv \
    -output intro-to-hadoop/output-movielens-06 \
    -file ./codes/avgGenreMapper02.py \
    -mapper avgGenreMapper02.py \
    -file ./codes/avgGenreReducer02.py \
    -reducer avgGenreReducer02.py \
    -file ./codes/avgGenreCombiner.py \
    -combiner avgGenreCombiner.py \
    -file ./movielens/movies.csv


I am working on the file "11-intro-to-hadoop-03.ipynb".
The first task is:
What is being passed from Map to Reduce?
Can reducer do the same thing as mapper, that is, to load in external data?
If we load external data on the reduce side, do we need to do so on the map side?
Can you write Python code for it?

%%writefile codes/avgRatingReducer02.py
#!/usr/bin/env python
import sys
import csv

movieFile = "./movies.csv"
movieList = {}

with open(movieFile, mode = 'r') as infile:
    reader = csv.reader(infile)
    for row in reader:
        movieList[row[0]] = {}
        movieList[row[0]]["title"] = row[1]
        movieList[row[0]]["genre"] = row[2]

current_movie = None
current_rating_sum = 0
current_rating_count = 0

for line in sys.stdin:
    line = line.strip()
    movie, rating = line.split("\t", 1)
    try:
        rating = float(rating)
    except ValueError:
        continue

    if current_movie == movie:
        current_rating_sum += rating
        current_rating_count += 1
    else:
        if current_movie:
            rating_average = current_rating_sum / current_rating_count
            movieTitle = movieList[current_movie]["title"]
            movieGenres = movieList[current_movie]["genre"]
            print ("%s\t%s\t%s" % (movieTitle, rating_average, movieGenres))
        current_movie = movie
        current_rating_sum = rating
        current_rating_count = 1

if current_movie == movie:
    rating_average = current_rating_sum / current_rating_count
    movieTitle = movieList[current_movie]["title"]
    movieGenres = movieList[current_movie]["genre"]
    print ("%s\t%s\t%s" % (movieTitle, rating_average, movieGenres))

!hdfs dfs -rm -r intro-to-hadoop/output-movielens-03
!yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-streaming.jar \
    -input /repository/movielens/ratings.csv \
    -output intro-to-hadoop/output-movielens-03 \
    -file ./codes/avgRatingMapper02.py \
    -mapper avgRatingMapper02.py \
    -file ./codes/avgRatingReducer02.py \
    -reducer avgRatingReducer02.py \
    -file ./movielens/movies.csv

!hdfs dfs -ls intro-to-hadoop/output-movielens-02
!hdfs dfs -ls intro-to-hadoop/output-movielens-03

!hdfs dfs -cat intro-to-hadoop/output-movielens-03/part-00000 \
    2>/dev/null | head -n 10


I am working on the file "11-intro-to-hadoop-04.ipynb".
The first task is:
<center> Final Cleanup </center>
Executing the cell below will clean up all HDFS output directories created as a result of previous MapReduce programs.
Can you write Python code for it?

!hdfs dfs -ls intro-to-hadoop

!hdfs dfs -rm -r intro-to-hadoop/
!rm -Rf codes/
!rm -Rf movielens/


I am working on the file "11-more-classes.ipynb".
The first task is:
And, as Monomial variables are Polynomials, we can multiply them together to get a Polynomial:
Can you write Python code for it?


I am working on the file "11-more-classes.ipynb".
The first task is:
In fact, we can be a bit smarter than this. Note that the __init__ function of the Monomial class is identical to that of the Polynomial class, just with the leading_term set explicitly to 1. Rather than duplicating the code and modifying a single value, we can call the __init__ function of the Polynomial class directly. This is because the Monomial class is built on the Polynomial class, so knows about it. We regenerate the class, but only change the __init__ function:
Can you write Python code for it?

class Monomial(Polynomial):
    """Representing a monomial, which is a polynomial with leading term 1."""
    explanation = "I am a monomial"

    def __init__(self, roots):
        Polynomial.__init__(self, roots, 1)

    def __repr__(self):
        string = ""
        for root in self.roots:
            if root == 0:
                string = string + "x"
            elif root > 0:
                string = string + "(x - {})".format(root)
            else:
                string = string + "(x + {})".format(-root)
        return string

v = Monomial((2, -3))
v.explain_to("Fred")
print(v)


I am working on the file "11-more-classes.ipynb".
The first task is:
The final special function we'll look at (although there are many more, many of which may be useful) is __mul__. This allows Python to multiply two variables together. We did this before using the multiply method, but by using the __mul__ method we can multiply together two polynomials using the standard * operator. With this we can take the product of two polynomials:
Can you write Python code for it?

class Polynomial(object):
    """Representing a polynomial."""
    explanation = "I am a polynomial"

    def __init__(self, roots, leading_term):
        self.roots = roots
        self.leading_term = leading_term
        self.order = len(roots)

    def __repr__(self):
        string = str(self.leading_term)
        for root in self.roots:
            if root == 0:
                string = string + "x"
            elif root > 0:
                string = string + "(x - {})".format(root)
            else:
                string = string + "(x + {})".format(-root)
        return string

    def __mul__(self, other):
        roots = self.roots + other.roots
        leading_term = self.leading_term * other.leading_term
        return Polynomial(roots, leading_term)

    def explain_to(self, caller):
        print("Hello, {}. {}.".format(caller,self.explanation))
        print("My roots are {}.".format(self.roots))

p = Polynomial(p_roots, p_leading_term)
q = Polynomial((1,1,0,-2), -1)
r = p*q
print(r)


I am working on the file "11-more-classes.ipynb".
The first task is:
Variables of the  Monomial class are also variables of the Polynomial class, so can use all the methods and functions from the Polynomial class automatically:
Can you write Python code for it?

m = Monomial((-1, 4, 9))
m.explain_to("Caroline")
print(m)


I am working on the file "11-more-classes.ipynb".
The first task is:
We note that these functions, methods and variables may not be exactly right, as they are given for the general Polynomial class, not by the specific Monomial class. If we redefine these functions and variables inside the Monomial class, they will override those defined in the Polynomial class. We do not have to override all the functions and variables, just the parts we want to change:
Can you write Python code for it?

class Monomial(Polynomial):
    """Representing a monomial, which is a polynomial with leading term 1."""
    explanation = "I am a monomial"

    def __init__(self, roots):
        self.roots = roots
        self.leading_term = 1
        self.order = len(roots)

    def __repr__(self):
        string = ""
        for root in self.roots:
            if root == 0:
                string = string + "x"
            elif root > 0:
                string = string + "(x - {})".format(root)
            else:
                string = string + "(x + {})".format(-root)
        return string

m = Monomial((-1, 4, 9))
m.explain_to("Caroline")
print(m)


I am working on the file "11-more-classes.ipynb".
The first task is:
We now have a simple class that can represent polynomials and multiply them together, whilst printing out a simple string form representing itself. This can obviously be extended to be much more useful.
Inheritance
As we can see above, building a complete class from scratch can be lengthy and tedious. If there is another class that does much of what we want, we can build on top of that. This is the idea behind inheritance.
In the case of the Polynomial we declared that it started from the object class in the first line defining the class: class Polynomial(object). But we can build on any class, by replacing object with something else. Here we will build on the Polynomial class that we've started with.
A monomial is a polynomial whose leading term is simply 1. A monomial is a polynomial, and could be represented as such. However, we could build a class that knows that the leading term is always 1: there may be cases where we can take advantage of this additional simplicity.
We build a new monomial class as follows:
Can you write Python code for it?

class Monomial(Polynomial):
    """Representing a monomial, which is a polynomial with leading term 1."""

    def __init__(self, roots):
        self.roots = roots
        self.leading_term = 1
        self.order = len(roots)


I am working on the file "11-more-classes.ipynb".
The first task is:
allowed polynomials to be created, displayed, and multiplied together. However, the language is a little cumbersome. We can take advantage of a number of useful features of Python, many of which carry over to other programming languages, to make it easier to use the results.
Remember that the __init__ function is called when a variable is created. There are a number of special class functions, each of which has two underscores before and after the name. This is another Python convention that is effectively a rule: functions surrounded by two underscores have special effects, and will be called by other Python functions internally. So now we can create a variable that represents a specific polynomial by storing its roots and the leading term:
Can you write Python code for it?

p_roots = (1, 2, -3)
p_leading_term = 2
p = Polynomial(p_roots, p_leading_term)
p.explain_to("Alice")
q = Polynomial((1,1,0,-2), -1)
q.explain_to("Bob")


I am working on the file "11-notebook-after-class.ipynb".
The first task is:
Aufgabe 2     Informationsextraktion per Syntaxanalyse
Gegenstand dieser Aufgabe ist eine anwendungsnahe Möglichkeit, Ergebnisse einer Syntaxanalyse weiterzuverarbeiten. Aus den syntaktischen Abhängigkeiten eines Textes soll (unter Zuhilfenahme einiger Normalisierungsschritte) eine semantische Repräsentation der im Text enthaltenen Informationen gewonnen werden.
Für die syntaktische Analyse soll der DependencyParser der Stanford CoreNLP Suite verwendet werden. Die semantische Repräsentation eines Satzes sei ein zweistelliges, logisches Prädikat, dessen Argumente durch Subjekt und Objekt gefüllt sind. (Bei Fehlen eines der beiden Elemente soll None geschrieben werden.)
Folgendes Beispiel illustriert das gewünschte Ergebnis:
Eingabe:
I shot an elephant in my pajamas.
The elephant was seen by a giraffe in the desert.
The bird I need is a raven.
The man who saw the raven laughed out loud.

Ausgabe:
shot(I, elephant)
seen(giraffe, elephant)
need(I, bird)
raven(bird, None)
saw(man, raven)
laughed(man, None)

Beachten Sie, dass PATH_TO_CORE in folgender Code-Zelle Ihrem System entsprechend angepasst werden muss!
Can you write Python code for it?

from nltk.parse.stanford import StanfordDependencyParser

PATH_TO_CORE = "/pfad/zu/stanford-corenlp-full-2017-06-09"
jar = PATH_TO_CORE + '/' + "stanford-corenlp-3.8.0.jar"
model = PATH_TO_CORE + '/' + "stanford-corenlp-3.8.0-models.jar"

dep_parser = StanfordDependencyParser(
    jar, model,
    model_path="edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz"
)

from collections import defaultdict

def generate_predicates_for_sentence(sentence):
    verbs = set()
    sbj = {}
    obj = {}
    sbj_candidates = defaultdict(list)
    case = {}
    relcl_triples = []
    for result in dep_parser.raw_parse(sentence):
        for triple in result.triples():
            # print(*triple)

            if triple[1] == "nsubj":
                # whenever we find a subject, its head can be called verb
                # if something is added twice it does not matter --> sets
                # so it is better to add too often than not enough !
                # remember that nouns can be "verbs" in that sense together with copula
                verbs.add(triple[0])
                sbj[triple[0]] = triple[2]

            if triple[1] == "dobj" or triple[1] == "nsubjpass":
                # everything that has a direct object should be called a verb as well
                verbs.add(triple[0])
                obj[triple[0]] = triple[2]

            if triple[0][1].startswith('V'):
                # everything with a 'verb' as part of speech can be called a verb
                verbs.add(triple[0])
                if triple[1] == "nmod":
                    sbj_candidates[triple[0]].append(triple[2])

            if triple[1] == "case":
                case[triple[0]] = triple[2][0]

            if triple[1] == "acl:relcl":
                relcl_triples.append(triple)

    for triple in relcl_triples:
        if triple[2] not in sbj or sbj[triple[2]][1] in ["WP", "WDT"]:
            sbj[triple[2]] = triple[0]
        else:
           ...


I am working on the file "11-notebook-after-class.ipynb".
The first task is:
Aufgabe 4     Mehr Semantik für IE
Zusätzlich zu den in Aufgabe 2 behandelten Konstruktionen sollen jetzt auch negierte und komplexe Sätze mit Konjunktionen sinnvoll verarbeitet werden.
Eingabe:
I see an elephant.
You didn't see the elephant.
Peter saw the elephant and drank wine.

Gewünschte Ausgabe:
see(I, elephant)
not_see(You, elephant)
saw(Peter, elephant)
drank(Peter, wine)

Kopieren Sie am besten Ihren aktuellen Stand von oben herunter und fügen Sie Ihre Erweiterungen dann hier ein.
Can you write Python code for it?

def generate_predicates_for_sentence(sentence):
    pass
def generate_predicates_for_text(text):
    pass

text = """
I see an elephant.
You didn't see the elephant.
Peter saw the elephant and drank wine.
"""


I am working on the file "11-notebook-after-class.ipynb".
The first task is:
Hausaufgaben
Aufgabe 3     Parent Annotation
Parent Annotation kann die Performanz einer CFG wesentlich verbessern. Schreiben Sie eine Funktion, die einen gegebenen Syntaxbaum dieser Optimierung unterzieht. Auf diese Art und Weise transformierte Bäume können dann wiederum zur Grammatikinduktion verwendet werden.
parentHistory soll dabei die Anzahl der Vorgänger sein, die zusätzlich zum direkten Elternknoten berücksichtigt werden. (Kann bei der Lösung der Aufgabe auch ignoriert werden.)
parentChar soll ein Trennzeichen sein, das bei den neuen Knotenlabels zwischen dem ursprünglichen Knotenlabel und der Liste von Vorgängern eingefügt wird.
Can you write Python code for it?

def parent_annotation(tree, parentHistory=0, parentChar="^"):
    pass

test_tree = nltk.Tree(
    "S",
    [
        nltk.Tree("NP", [
            nltk.Tree("DET", []),
            nltk.Tree("N", [])
        ]),
        nltk.Tree("VP", [
            nltk.Tree("V", []),
            nltk.Tree("NP", [
                nltk.Tree("DET", []),
                nltk.Tree("N", [])
            ])
        ])
    ]
)

parent_annotation(
   test_tree
)