I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
Correlated Underlyings
The second example case is exactly the same but now with a highly positive correlation between the two relevant risk factors. Correlations are to be provided as a list  of list objects using the risk factor model names to reference them.
Can you write Python code for it?


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
Except from now passing this new object, the application and usage remains the same.
Can you write Python code for it?

port = derivatives_portfolio(
            name='portfolio',
            positions=positions,
            val_env=val_env,
            risk_factors=risk_factors,
            correlations=correlations,
            fixed_seed=True,
            parallel=False)

port.get_statistics()


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
For the instantiation, we pass all the elements to the portfolio class.
Can you write Python code for it?

port = derivatives_portfolio(
            name='portfolio',  # name
            positions=positions,  # derivatives positions
            val_env=val_env,  # valuation environment
            risk_factors=risk_factors, # relevant risk factors
            correlations=False,  # correlation between risk factors
            fixed_seed=False,  # fixed seed for randon number generation
            parallel=False)  # parallel valuation of portfolio positions


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
In this case, the payoff is that of a regular, plain vanilla European call option. If $T$ is the maturity date, $S_T$ the value of the relevant risk factor at that date and $K$ the strike price, the payoff $h_T$ at maturity of such an option is given by
$$
h_T = \max[S_T - K, 0]
$$
maturity_value represents the value vector of the risk factor at maturity. Any other "sensible" payoff definition is possible. For instance, the following works as well:
Can you write Python code for it?

payoff = 'np.maximum(np.minimum(maturity_value) * 2 - 50, 0)'


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
Let us pick two specific simulated paths, one for each risk factor, and let us visualize these.
Can you write Python code for it?

path_no = 0
paths1 = port.underlying_objects['gbm'].get_instrument_values()[:, path_no]
paths2 = port.underlying_objects['jd'].get_instrument_values()[:, path_no]


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
Once instantiated, the method get_statistics provides major portfolio statistics like position values, position deltas ans position vegas.
Can you write Python code for it?

%%time
stats = port.get_statistics()

stats


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
Other standardized payoff elemenets include mean_value, max_value and min_value representing maturity value vectors with the pathwise means, maxima and minima. Using these payoff elements allows the easy definition of options with Asian features.
Having instantiated the valuation class, the present_value method returns the present value Monte Carlo estimator for the call option.
Can you write Python code for it?


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
Our relevant market now takes on the following form (defined a dictionary objects):
Can you write Python code for it?

risk_factors = {'gbm': me, 'jd' : me_jump}
positions = {'put' : put, 'call_jump' : call_jump}


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
Portfolio Valuation
In general, market players (asset managers, investment banks, hedge funds, insurance companies, etc.) have to value not only single derivatvies instruments but rather portfolios composed of several derivatives instruments. A consistent derivatives portfolio valuation is particularly important when there are multiple derivatives written on the same risk factor and/or correlations between different risk factors.
These are the classes availble for a consistent portfolio valuation:

derivatives_position to model a portfolio position
derivatives_portfolio to model a derivatives portfolio

derivatives_position
We work with the market_environment object from before and add information about the risk factor model we are using.
Can you write Python code for it?


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
Since DX Analytics relies on Monte Carlo simulation and other numerical methods, the calculation of the delta and vega of such an option is identical to the European exercise case.
Can you write Python code for it?

put_ame.delta()

put_ame.vega()

%%time
k_list = np.arange(26., 46.1, 2.)
pv = []; de = []; ve = []
for k in k_list:
    put_ame.update(strike=k)
    pv.append(put_ame.present_value())
    de.append(put_ame.delta(.5))
    ve.append(put_ame.vega(0.2))

plot_option_stats(k_list, pv, de, ve)


put_ame.delta()

put_ame.vega()

%%time
s_list = np.arange(34., 46.1, 2.)
pv = []; de = []; ve = []
for s in s_list:
    put_ame.update(s)
    pv.append(put_ame.present_value())
    de.append(put_ame.delta(.5))
    ve.append(put_ame.vega(0.2))

plot_option_stats(s_list, pv, de, ve)


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
The following single risk factor valuation classes are available:

valuation_mcs_european_single for derivatives with European exercise
valuation_mcs_american_single for derivatives with American/Bermudan exercise

Modeling the Risk Factor
Before moving on to the valuation classes, we need to model an instantiate an underlying risk factor, in this case a geometric_brownian_motion object. Background information is provided in the respective part of the documentation about model classes.
Can you write Python code for it?

r = constant_short_rate('r', 0.06)

me = market_environment('me', dt.datetime(2015, 1, 1))

me.add_constant('initial_value', 36.)
me.add_constant('volatility', 0.2)
me.add_constant('final_date', dt.datetime(2015, 12, 31))
me.add_constant('currency', 'EUR')
me.add_constant('frequency', 'W')
me.add_constant('paths', 25000)

me.add_curve('discount_curve', r)

gbm = geometric_brownian_motion('gbm', me)


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
The method get_info prints an overview of the all relevant information stored for the respective derivatives_position object.
Can you write Python code for it?


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
The method get_positions provides detailed information about the single derivatives positions of the derivatives_portfolio object.
Can you write Python code for it?


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
The method returns a standard pandas DataFrame object with which you can work as you are used to.
Can you write Python code for it?


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
The metod get_values only calculates the present values of the derivatives instruments and positions and is therefore a bit less compute and time intensive.
Can you write Python code for it?


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
The only difference to consider here is that for American options where exercise can take place at any time before maturity, the inner value of the option (payoff of immediate exercise) is relevant over the whole set of dates. Therefore, maturity_value needs to be replaced by instrument_values in the definition of the payoff function.
Can you write Python code for it?


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
The plot illustrates that the two paths are indeed highly positively correlated. However, in this case a large jump occurs for the jump_diffusion object.
Can you write Python code for it?

plt.figure(figsize=(10, 6))
plt.plot(port.time_grid, paths1, 'r', label='gbm')
plt.plot(port.time_grid, paths2, 'b', label='jd')
plt.gcf().autofmt_xdate()
plt.legend(loc=0); plt.grid(True)
# highly correlated underlyings
# -- with a large jump for one risk factor


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
There is a little plot helper function available to plot these statistics conveniently.
Can you write Python code for it?

plot_option_stats_full(k_list, pv, de, ga, ve, th, rh)


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
This approach allows to work with such a valuation object similar to an analytical valuation formula like the one of Black-Scholes-Merton (1973). For example, you can estimate and plot present values, deltas and vegas for a range of different initial values of the risk factor.
Can you write Python code for it?

%%time
s_list = np.arange(34., 46.1, 2.)
pv = []; de = []; ve = []
for s in s_list:
    call_eur.update(s)
    pv.append(call_eur.present_value())
    de.append(call_eur.delta(.5))
    ve.append(call_eur.vega(0.2))

%matplotlib inline


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
This approach allows to work with such a valuation object similar to an analytical valuation formula like the one of Black-Scholes-Merton (1973). For example, you can estimate and plot present values, deltas, gammas, vegas, thetas and rhos for a range of different initial values of the risk factor.
Can you write Python code for it?

%%time
k_list = np.arange(26., 46.1, 2)
pv = []; de = []; ve = []; th = []; rh = []; ga = []
for k in k_list:
    call_eur.update(strike=k)
    pv.append(call_eur.present_value())
    de.append(call_eur.delta(0.5))
    ve.append(call_eur.vega(0.2))
    th.append(call_eur.theta())
    rh.append(call_eur.rho())
    ga.append(call_eur.gamma())

%matplotlib inline


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
To instantiate a the valuation_mcs_european_single class, the following information/data is to be provided:

name as a string object
instance of a model class
market environment
payoff of the instrument a string object and containing "regular" Python/NumPy code
Can you write Python code for it?

call_eur = valuation_mcs_european_single(
                name='call_eur',
                underlying=gbm,
                mar_env=me,
                payoff_func='np.maximum(maturity_value - strike, 0)')


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
To instantiate the derivatives_portfolio class, a valuation environment (instance of market_environment class) is needed.
Can you write Python code for it?

val_env = market_environment('general', dt.datetime(2015, 1, 1))
val_env.add_constant('frequency', 'M')
val_env.add_constant('paths', 50000)
val_env.add_constant('starting_date', val_env.pricing_date)
val_env.add_constant('final_date', val_env.pricing_date)
val_env.add_curve('discount_curve', r)


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
derivatives_portfolio
The derivatives_portfolio class implements the core portfolio valuation tasks. This sub-section illustrates to cases, one with uncorrelated underlyings and another one with correlated underlyings
Uncorrelated Underlyings
The first example is based on a portfolio with two single-risk factor instruments on two different risk factors which are not correlated. In addition to the gbm object, we define a jump_diffusion object.
Can you write Python code for it?

me_jump = market_environment('me_jump', dt.datetime(2015, 1, 1))

me_jump.add_environment(me)
me_jump.add_constant('lambda', 0.8)
me_jump.add_constant('mu', -0.8)
me_jump.add_constant('delta', 0.1)
me_jump.add_constant('model', 'jd')


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
valuation_mcs_american_single
The modeling and valuation of derivatives with American/Bermudan exercise is almost completely the same as in the more simple case of European exercise.
Can you write Python code for it?

me.add_constant('initial_value', 36.)
  # reset initial_value

put_ame = valuation_mcs_american_single(
                name='put_eur',
                underlying=gbm,
                mar_env=me,
                payoff_func='np.maximum(strike - instrument_values, 0)')


I am working on the file "03_dx_valuation_single_risk.ipynb".
The first task is:
valuation_mcs_european_single
The first instrument we value is a European call option written on the single relevant risk factor as embodied by the gbm model object. To this end, we add a maturity date to the market environment and a strike price.
Can you write Python code for it?

me.add_constant('maturity', dt.datetime(2015, 12, 31))
me.add_constant('strike', 40.)


I am working on the file "03_embed_classlabels.ipynb".
The first task is:
"Kernel PCA" and Ridge Regression vs. SimEc
To get an idea of how a perfect similarity preserving embedding would look like when computing similarities from class labels, we can embed the data by performing an eigendecomposition of the similarity matrix (i.e. performing kernel PCA). However, since in a real setting we would be unable to compute the similarities of the test samples to the training samples (since we don't know their class labels), to map the test samples into the embedding space we additionally need to train a (ridge) regression model to map from the original input space to the embedding space.
A SimEc with multiple hidden layers starts to get close to the eigendecomposition solution.
Can you write Python code for it?

D, V = np.linalg.eig(S)
# as a comparison: regular kpca embedding: take largest EV
D1, V1 = D[np.argsort(D)[::-1]], V[:,np.argsort(D)[::-1]]
X_embed = np.dot(V1.real, np.diag(np.sqrt(np.abs(D1.real))))
plot_mnist(X_embed[:,:2], y, title='MNIST (train) - largest 2 EV')
print("similarity approximation  2D - mse: %f" % check_similarity_match(X_embed[:,:2], S)[0])
print("similarity approximation  5D - mse: %f" % check_similarity_match(X_embed[:,:5], S)[0])
print("similarity approximation  7D - mse: %f" % check_similarity_match(X_embed[:,:7], S)[0])
print("similarity approximation 10D - mse: %f" % check_similarity_match(X_embed[:,:10], S)[0])
print("similarity approximation 25D - mse: %f" % check_similarity_match(X_embed[:,:25], S)[0])

n_targets = 2000
# get good alpha for RR model
m = Ridge()
rrm = GridSearchCV(m, {'alpha': [0.000001, 0.00001, 0.0001, 0.001, 0.01, 0.1, 0.25, 0.5, 0.75, 1., 2.5, 5., 7.5, 10., 25., 50., 75., 100., 250., 500., 750., 1000.]})
rrm.fit(X, X_embed[:,:8])
alpha = rrm.best_params_["alpha"]
print("Ridge Regression with alpha: %r" % alpha)
mse_ev, mse_rr, mse_rr_test = [], [], []
mse_simec, mse_simec_test = [], []
mse_simec_hl, mse_simec_hl_test = [], []
e_dims = [2, 3, 4, 5, 6, 7, 8, 9, 10, 15]
for e_dim in e_dims:
    print(e_dim)
    # eigenvalue based embedding
    mse = check_similarity_match(X_embed[:,:e_dim], S)[0]
    mse_ev.append(mse)
    # train a linear ridge regression model to learn the mapping from X to Y
    model = Ridge(alpha=alpha)
    model.fit(X, X_embed[:,:e_dim])
    X_embed_r = model.predict(X)
    X_embed_test_r = model.predict(X_test)
    mse = check_similarity_match(X_embed_r, S)[0]
    mse_rr.append(mse)
    mse = check_similarity_match(X_embed_test_r, S_test)[0]
    mse_rr_test.append(mse)
    # simec - linear
    simec = SimilarityEncoder(X.shape[1], e_dim, n_targets, s_ll_reg=0.5, S_ll=S[:n_targets,:n_targets],
                              orth_reg=0.001 if e_dim > 8 else 0., l2_reg_emb=0.00001,
                              l2_reg_out=0.0000001,...


I am working on the file "03_embed_classlabels.ipynb".
The first task is:
Great, we already see some clusters separating from the rest! What if we add more layers?
We can examine how the embedding changes during training: first some clusters separate, then it starts to look like the eigenvalue based embedding with the clusters of several numbers pulled together.
Can you write Python code for it?

# similarity encoder with similarities relying on class information - 1 hidden layer
n_targets = 1000
simec = SimilarityEncoder(X.shape[1], 2, n_targets, hidden_layers=[(100, 'tanh')],
                          l2_reg=0.00000001, l2_reg_emb=0.00001, l2_reg_out=0.0000001,
                          s_ll_reg=0.5, S_ll=S[:n_targets,:n_targets], opt=keras.optimizers.Adamax(lr=0.01))
e_total = 0
for e in [5, 10, 10, 10, 15, 25, 25]:
    e_total += e
    print(e_total)
    simec.fit(X, S[:,:n_targets], epochs=e)
    X_embed = simec.transform(X)
    clf = KNN(n_neighbors=10)
    clf.fit(X_embed[:1000], y[:1000])
    acc = clf.score(X_embed[1000:], y[1000:])
    print("knn accuracy: %f" % acc)
    print("similarity approximation - mse: %f" % check_similarity_match(X_embed, S)[0])
    plot_digits(X_embed, digits, title='SimEc after %i epochs; accuracy: %.1f' % (e_total, 100*acc) , plot_box=False)


I am working on the file "03_embed_classlabels.ipynb".
The first task is:
Lets first try a simple linear SimEc.
Can you write Python code for it?

# similarity encoder with similarities relying on class information - linear
simec = SimilarityEncoder(X.shape[1], 2, n_targets, l2_reg_emb=0.00001, l2_reg_out=0.0000001,
                          s_ll_reg=0.5, S_ll=S[:n_targets,:n_targets], opt=keras.optimizers.Adamax(lr=0.005))
simec.fit(X, S[:,:n_targets])
X_embed = simec.transform(X)
plot_digits(X_embed, digits, title='Digits - SimEc (class sim, linear)')
# of course we're overfitting here quite a bit since we used all samples for training
# even if we didn't use the corresponding similarities...but this is only a toy example anyways
clf = KNN(n_neighbors=10)
clf.fit(X_embed[:n_targets], y[:n_targets])
print("knn accuracy: %f" % clf.score(X_embed[n_targets:], y[n_targets:]))
print("similarity approximation - mse: %f" % check_similarity_match(X_embed, S)[0])


I am working on the file "03_embed_classlabels.ipynb".
The first task is:
MNIST Dataset
Embedding the regular 28x28 pixel MNIST digits
Can you write Python code for it?

# load digits
mnist = fetch_mldata('MNIST original', data_home='data')
X = mnist.data/255.  # normalize to 0-1
y = np.array(mnist.target, dtype=int)
# subsample 10000 random data points
np.random.seed(42)
n_samples = 10000
n_test = 2000
n_targets = 1000
rnd_idx = np.random.permutation(X.shape[0])[:n_samples]
X_test, y_test = X[rnd_idx[:n_test],:], y[rnd_idx[:n_test]]
X, y = X[rnd_idx[n_test:],:], y[rnd_idx[n_test:]]
# scale
ss = StandardScaler(with_std=False)
X = ss.fit_transform(X)
X_test = ss.transform(X_test)
n_train, n_features = X.shape

# compute similarity matrix based on class labels
Y = np.tile(y, (len(y), 1))
S = center_K(np.array(Y==Y.T, dtype=int))
Y = np.tile(y_test, (len(y_test), 1))
S_test = center_K(np.array(Y==Y.T, dtype=int))


I am working on the file "03_embed_classlabels.ipynb".
The first task is:
SimEc based on class labels
We've seen that SimEcs can reach the same solutions as traditional spectral methods such as kPCA and isomap. However, these methods have the limitation that you can only embed new data points if you can compute their kernel map, i.e. the similarity to the training examples. But what if the similarity matrix used as targets during training was generated by an unknown process such as human similarity judgments?
To show how we can use SimEc in such a scenario, we construct the similarity matrix from the class labels assigned by human annotators (1=same class, 0=different class).
Can you write Python code for it?

Y = np.tile(y, (len(y), 1))
S = center_K(np.array(Y==Y.T, dtype=int))
# take only some of the samples as targets to speed it all up
n_targets = 1000

# knn accuracy using all original feature dimensions
clf = KNN(n_neighbors=10)
clf.fit(X[:n_targets], y[:n_targets])
print("knn accuracy: %f" % clf.score(X[n_targets:], y[n_targets:]))

# PCA
pca = PCA(n_components=2)
X_embedp = pca.fit_transform(X)
plot_digits(X_embedp, digits, title='Digits embedded with PCA')
clf = KNN(n_neighbors=10)
clf.fit(X_embedp[:n_targets], y[:n_targets])
print("knn accuracy: %f" % clf.score(X_embedp[n_targets:], y[n_targets:]))

# check how many relevant dimensions there are
eigenvals = np.linalg.eigvalsh(S)[::-1]
plt.figure();
plt.plot(list(range(1, S.shape[0]+1)), eigenvals, '-o', markersize=3);
plt.plot([1, S.shape[0]],[0,0], 'k--', linewidth=0.5);
plt.xlim(1, X.shape[1]+1);
plt.title('Eigenvalue spectrum of S (based on class labels)');

D, V = np.linalg.eig(S)
# regular kpca embedding: take largest EV
D1, V1 = D[np.argsort(D)[::-1]], V[:,np.argsort(D)[::-1]]
X_embed = np.dot(V1.real, np.diag(np.sqrt(np.abs(D1.real))))
plot_digits(X_embed[:,:2], digits, title='Digits embedded based on first 2 components', plot_box=False)
clf = KNN(n_neighbors=10)
clf.fit(X_embed[:n_targets,:2], y[:n_targets])
print("knn accuracy: %f" % clf.score(X_embed[n_targets:,:2], y[n_targets:]))
print("similarity approximation - mse: %f" % check_similarity_match(X_embed[:,:2], S)[0])


I am working on the file "03_getting_started_with_iris.ipynb".
The first task is:
Each value we are predicting is the response (also known as: target, outcome, label, dependent variable)
Classification is supervised learning in which the response is categorical
Regression is supervised learning in which the response is ordered and continuous

Requirements for working with data in scikit-learn

Features and response are separate objects
Features and response should be numeric
Features and response should be NumPy arrays
Features and response should have specific shapes
Can you write Python code for it?

# check the types of the features and response
print(type(iris.data))
print(type(iris.target))

# check the shape of the features (first dimension = number of observations, second dimensions = number of features)
print(iris.data.shape)

# check the shape of the response (single dimension matching the number of observations)
print(iris.target.shape)

# store feature matrix in "X"
X = iris.data

# store response vector in "y"
y = iris.target


I am working on the file "03_getting_started_with_iris.ipynb".
The first task is:
Machine learning on the iris dataset

Framed as a supervised learning problem: Predict the species of an iris using the measurements
Famous dataset for machine learning because prediction is easy
Learn more about the iris dataset: UCI Machine Learning Repository

Loading the iris dataset into scikit-learn
Can you write Python code for it?

# import load_iris function from datasets module
from sklearn.datasets import load_iris

# save "bunch" object containing iris dataset and its attributes
iris = load_iris()
type(iris)

# print the iris data
print(iris.data)


I am working on the file "03_getting_started_with_iris.ipynb".
The first task is:
Resources

scikit-learn documentation: Dataset loading utilities
Jake VanderPlas: Fast Numerical Computing with NumPy (slides, video)
Scott Shell: An Introduction to NumPy (PDF)

Comments or Questions?

Email: kevin@dataschool.io
Website: http://dataschool.io
Twitter: @justmarkham
Can you write Python code for it?

from IPython.core.display import HTML
def css_styling():
    styles = open("styles/custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "03_image_models/03b_finetune_MOBILENETV2_flowers5.ipynb".
The first task is:
Configuration
Can you write Python code for it?

GCS_DS_PATH = "gs://practical-ml-vision-book/flowers_5_tfr"

# Settings for TPUv3. When running on hardware with less memory such as a TPUv2 (Colab)
# or a GPU, you might have to use lower BATCH_SIZE and/or IMAGE_SIZE values.

IMAGE_SIZE = [224, 224] # available image sizes in flowers104 dataset: 512x512, 331x331, 224x224, 192,192
EPOCHS = 13

# Learning rate schedule for fine-tuning: use trainable=True (best validation accuracy 0.91)
#BATCH_SIZE = 16 * strategy.num_replicas_in_sync
#LR_START = 0.00001
#LR_MAX = 0.000025 * strategy.num_replicas_in_sync
#LR_MIN = 0.00001
#LR_RAMPUP_EPOCHS = 3
#LR_SUSTAIN_EPOCHS = 1
#LR_EXP_DECAY = .8

# Learning rate schedule for fine-tuning with AdamW: use trainable=True (best validation accuracy 0.92)
BATCH_SIZE = 8 * strategy.num_replicas_in_sync
LR_START = 0.00001
LR_MAX = 0.0001 * strategy.num_replicas_in_sync
LR_MIN = 0.00001
LR_RAMPUP_EPOCHS = 3
LR_SUSTAIN_EPOCHS = 0
LR_EXP_DECAY = .8

# learning rate schedule for transfer learning: use trainable=False (best validation accuracy 0.90)
#BATCH_SIZE = 16 * strategy.num_replicas_in_sync
#LR_START = 0.00001
#LR_MAX = 0.00075 * strategy.num_replicas_in_sync #(Note: 0.00007 with trainable=True to replicate graph in book)
#LR_MIN = 0.00001
#LR_RAMPUP_EPOCHS = 0
#LR_SUSTAIN_EPOCHS = 0
#LR_EXP_DECAY = .8

GCS_PATH_SELECT = { # available image sizes
    192: GCS_DS_PATH + '/tfrecords-jpeg-192x192',
    224: GCS_DS_PATH + '/tfrecords-jpeg-224x224',
    331: GCS_DS_PATH + '/tfrecords-jpeg-331x331',
    512: GCS_DS_PATH + '/tfrecords-jpeg-512x512'
}
GCS_PATH = GCS_PATH_SELECT[IMAGE_SIZE[0]] + '/*.tfrec'
filenames = tf.io.gfile.glob(GCS_PATH)
validation_split = 0.19
split = len(filenames) - int(len(filenames) * validation_split)
TRAINING_FILENAMES = filenames[:split]
VALIDATION_FILENAMES = filenames[split:]

CLASSES = ['daisy', 'dandelion', 'roses', 'sunflowers', 'tulips']

def lrfn(epoch):
    if epoch < LR_RAMPUP_EPOCHS:
        lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START
    elif epoch < LR_RAMPUP_EPOCHS +...


I am working on the file "03_image_models/03b_finetune_MOBILENETV2_flowers5.ipynb".
The first task is:
Dataset visualizations (5 flowers)
Can you write Python code for it?

# data dump
print("Training data shapes:")
for image, label in get_training_dataset().take(3):
    print(image.numpy().shape, label.numpy().shape)
print("Training data label examples:", label.numpy())

# Peek at training data
training_dataset = get_training_dataset()
training_dataset = training_dataset.unbatch().batch(20)
train_batch = iter(training_dataset)

# run this cell again for next set of images
display_batch_of_images(next(train_batch))


I am working on the file "03_image_models/03b_finetune_MOBILENETV2_flowers5.ipynb".
The first task is:
Datasets
Can you write Python code for it?

def decode_image(image_data):
    image = tf.image.decode_jpeg(image_data, channels=3) # decoded inamge in uint8 format range [0,255]
    image = tf.reshape(image, [*IMAGE_SIZE, 3]) # explicit size needed for TPU
    return image

def read_tfrecord(example):
    TFREC_FORMAT = {
        "image": tf.io.FixedLenFeature([], tf.string), # tf.string means bytestring
        "class": tf.io.FixedLenFeature([], tf.int64),  # shape [] means single element
    }
    example = tf.io.parse_single_example(example, TFREC_FORMAT)
    image = decode_image(example['image'])
    label = tf.cast(example['class'], tf.int32)
    return image, label # returns a dataset of (image, label) pairs

def load_dataset(filenames, ordered=False):
    # Read from TFRecords. For optimal performance, reading from multiple files at once and
    # disregarding data order. Order does not matter since we will be shuffling the data anyway.

    ignore_order = tf.data.Options()
    if not ordered:
        ignore_order.experimental_deterministic = False # disable order, increase speed

    dataset = tf.data.TFRecordDataset(filenames, num_parallel_reads=AUTO) # automatically interleaves reads from multiple files
    dataset = dataset.with_options(ignore_order) # uses data as soon as it streams in, rather than in its original order
    dataset = dataset.map(read_tfrecord, num_parallel_calls=AUTO)
    # returns a dataset of (image, label) pairs
    return dataset

def data_augment(image, label):
    # data augmentation. Thanks to the dataset.prefetch(AUTO) statement in the next function (below),
    # this happens essentially for free on TPU. Data pipeline code is executed on the "CPU" part
    # of the TPU while the TPU itself is computing gradients.
    image = tf.image.random_flip_left_right(image)
    #image = tf.image.random_saturation(image, 0, 2)
    return image, label

def get_training_dataset():
    dataset = load_dataset(TRAINING_FILENAMES)
    dataset = dataset.map(data_augment, num_parallel_calls=AUTO)
    dataset = dataset.repeat() #...


I am working on the file "03_image_models/03b_finetune_MOBILENETV2_flowers5.ipynb".
The first task is:
Model
Can you write Python code for it?

with strategy.scope():
    pretrained_model = tf.keras.applications.MobileNetV2(
        weights='imagenet',
        include_top=False,
        input_shape=[*IMAGE_SIZE, 3])

    pretrained_model.trainable = True # fine-tuning

    model = tf.keras.Sequential([
        tf.keras.layers.Lambda( # convert image format from int [0,255] to the format expected by this model
            lambda data: tf.keras.applications.mobilenet.preprocess_input(
                tf.cast(data, tf.float32)), input_shape=[*IMAGE_SIZE, 3]),
        pretrained_model,
        tf.keras.layers.GlobalAveragePooling2D(),
        tf.keras.layers.Dense(16, activation='relu', name='flower_dense'),
        tf.keras.layers.Dense(len(CLASSES), activation='softmax', name='flower_prob')
    ])

    mult = 0.4 # for pretrained layers
    mult_by_layer={
            # Clasification head
            'flower_prob': 1.0,
            'flower_dense': 1.0,
            # Pretrained layers
            'block_1_': 0.02 * mult,
            'block_2_': 0.04 * mult,
            'block_3_': 0.06 * mult,
            'block_4_': 0.08 * mult,
            'block_5_': 0.1 * mult,
            'block_6_': 0.15 * mult,
            'block_7_': 0.2 * mult,
            'block_8_': 0.25 * mult,
            'block_9_': 0.3 * mult,
            'block_10_': 0.35 * mult,
            'block_11_': 0.4 * mult,
            'block_12_': 0.5 * mult,
            'block_13_': 0.6 * mult,
            'block_14_': 0.7 * mult,
            'block_15_': 0.8 * mult,
            'block_16_': 0.9 * mult,
            # these layers do not have stable identifiers in tf.keras.applications.MobileNetV2
            'conv': 0.5 * mult,
            'Conv': 0.5 * mult
    }

    optimizer = AdamW(lr=LR_MAX, model=model, lr_multipliers=mult_by_layer)

model.compile(
    #optimizer='adam',
    optimizer=optimizer,
    loss = 'sparse_categorical_crossentropy',
    metrics=['sparse_categorical_accuracy'],
    steps_per_execution=8
)
model.summary()


I am working on the file "03_image_models/03b_finetune_MOBILENETV2_flowers5.ipynb".
The first task is:
TPU or GPU detection
Can you write Python code for it?

try: # detect TPUs
    tpu = tf.distribute.cluster_resolver.TPUClusterResolver.connect()
    strategy = tf.distribute.TPUStrategy(tpu)
except ValueError: # detect GPUs or multi-GPU machines
    strategy = tf.distribute.MirroredStrategy()

print("REPLICAS: ", strategy.num_replicas_in_sync)


I am working on the file "03_image_models/03b_finetune_MOBILENETV2_flowers5.ipynb".
The first task is:
Visual validation
Can you write Python code for it?

dataset = get_validation_dataset()
dataset = dataset.unbatch().batch(20)
batch = iter(dataset)

# run this cell again for next set of images
images, labels = next(batch)
probabilities = model.predict(images)
predictions = np.argmax(probabilities, axis=-1)
display_batch_of_images((images, labels), predictions)


I am working on the file "03_image_models/03b_finetune_MOBILENETV2_flowers5.ipynb".
The first task is:
Visualization utilities
data -> pixels, nothing of much interest for the machine learning practitioner in this section.
Can you write Python code for it?

# numpy and matplotlib defaults
np.set_printoptions(threshold=15, linewidth=80)

def batch_to_numpy_images_and_labels(data):
    images, labels = data
    numpy_images = images.numpy()
    numpy_labels = labels.numpy()
    if numpy_labels.dtype == object: # binary string in this case, these are image ID strings
        numpy_labels = [None for _ in enumerate(numpy_images)]
    # If no labels, only image IDs, return None for labels (this is the case for test data)
    return numpy_images, numpy_labels

def title_from_label_and_target(label, correct_label):
    if correct_label is None:
        return CLASSES[label], True
    correct = (label == correct_label)
    return "{} [{}{}{}]".format(CLASSES[label], 'OK' if correct else 'NO', u"\u2192" if not correct else '',
                                CLASSES[correct_label] if not correct else ''), correct

def display_one_flower(image, title, subplot, red=False, titlesize=16):
    plt.subplot(*subplot)
    plt.axis('off')
    plt.imshow(image)
    if len(title) > 0:
        plt.title(title, fontsize=int(titlesize) if not red else int(titlesize/1.2), color='red' if red else 'black', fontdict={'verticalalignment':'center'}, pad=int(titlesize/1.5))
    return (subplot[0], subplot[1], subplot[2]+1)

def display_batch_of_images(databatch, predictions=None):
    """This will work with:
    display_batch_of_images(images)
    display_batch_of_images(images, predictions)
    display_batch_of_images((images, labels))
    display_batch_of_images((images, labels), predictions)
    """
    # data
    images, labels = batch_to_numpy_images_and_labels(databatch)
    if labels is None:
        labels = [None for _ in enumerate(images)]

    # auto-squaring: this will drop data that does not fit into square or square-ish rectangle
    rows = int(math.sqrt(len(images)))
    cols = len(images)//rows

    # size and spacing
    FIGSIZE = 13.0
    SPACING = 0.1
    subplot=(rows,cols,1)
    if rows < cols:
        plt.figure(figsize=(FIGSIZE,FIGSIZE/cols*rows))
 ...


I am working on the file "03_improving.ipynb".
The first task is:
Answer: [Write your answer text here]
Task 3
Create a function tfidf(t,d) that returns the tf-idf score of term t in document d by using the tf(t,d), df(t) and num_documents() functions we defined above. The tf-idf formula can be found on the lecture slides. Use tf-idf with plain (non-logarithmic) term frequency, as applied by scoring variant ntn. Test your function with the examples shown below.
You can use our old index for this task and the tasks below: You do not need to include the results from above with the smarter tokenization and preprocessing functions.
You can use the log10(n) function to calculate the base 10 logarithm.
Can you write Python code for it?

#def tfidf(t,d):
#    ...

#print(tfidf('network', 24130474))
#print(tfidf('var', 24130474))
#print(tfidf('surface', 24130474))


I am working on the file "03_improving.ipynb".
The first task is:
Come up with a few example queries to run, and include the output here. Do the results make sense? Why (not)?
Can you write Python code for it?


I am working on the file "03_improving.ipynb".
The first task is:
Create a function query(query_string), which accepts as input a single query string that could consist of one or more words, and returns or prints a list of (up to) 10 best matching documents, along with their score.

You should use tf-idf to calculate document scores based on the query, and the results should be ordered by score in descending order.
Hint: Start by copying your or_query function from mini-assignment 2, then expand that to rank the results, making use of the tfidf(t,d) function you created earlier.
Use the provided display_summary(id,extra_text) function to make the output a bit more 'search engine'-like.
Can you write Python code for it?


I am working on the file "03_improving.ipynb".
The first task is:
Let's test these functions with some examples:
Can you write Python code for it?

print(tf('network', 24130474))
print(df('network'))
print(num_documents())
print(length_tf(24130474))


I am working on the file "03_improving.ipynb".
The first task is:
Now implement the smarter_and_query function based on the two functions above. You can start from the code for and_query from the last assignment.
Can you write Python code for it?

# Smarter and_query based on the smarter tokenize and preprocess functions

# Your code here:

#def smarter_and_query(query_string):
#  ...


I am working on the file "03_improving.ipynb".
The first task is:
Now we can make a smarter index based on these functions. For practical purposes, the code below generates the smarter index on a subset of the data, as generating an index with stemming on the entire set would take too much time. (You don't need to change or add anything here.)
Can you write Python code for it?

# Below, we create our smarter index (based on a subset of the documents for demonstration purposes)
smarter_index = defaultdict(set)

# Here we define the subset (somewhat arbitrary):
subset_of_ids = set(key for key in Abstracts.keys() if 24100000 <= key < 24200000)
subset_of_abstracts = ((key, Abstracts[key]) for key in subset_of_ids)

# Uncomment this line to process the whole corpus (might take a long time):
#subset_of_abstracts = Abstracts.items()

# Building our smarter index:
for (id, abstract) in subset_of_abstracts:
    for term in smarter_preprocess(smarter_tokenize(abstract)):
        smarter_index[term].add(id)


I am working on the file "03_improving.ipynb".
The first task is:
Ranking
Another way to improve our search results is to rank them. A possible way to do this is to calculate a score for each document based on the matching terms from the query. One such scoring method is tf-idf, which comes with several variants, as explained in the lecture slides.
In order to quickly calculate the scores for a term/document combination, we'll need quick access to a couple of things:

tf(t,d): How often does a term occur in a document
df(t): In how many documents does a term occur
N: The number of documents in our index
length_tf(d): The length of the document vector (with vectors of plain tf values)
Can you write Python code for it?

tf_matrix = defaultdict(Counter)
length_values = defaultdict(int)

for (doc_id, abstract) in Abstracts.items():
    tokens = preprocess(tokenize(abstract))
    tf_matrix[doc_id] = Counter(tokens)
    l = 0
    for t in tf_matrix[doc_id].keys():
        l += tf_matrix[doc_id][t] ** 2
    length_values[doc_id] = sqrt(l)

def tf(t,d):
    return float(tf_matrix[d][t])

def df(t):
    return float(len(inverted_index[t]))

def num_documents():
    return float(len(Abstracts))

def length_tf(d):
    return length_values[d]


I am working on the file "03_improving.ipynb".
The first task is:
Task 4
Create a function query_ntn(query_string), which accepts as input a single query string that could consist of one or more words, and returns or prints a list of (up to) 10 best matching documents, along with their score. Use tf-idf to calculate document scores based on the query, applying variant ntn, as above (see the formula for score_ntn on the lecture slides). Use an auxiliary function score_ntn to calculate the score. The results should be shown in descending order by score.
You can start by copying your or_query function from mini-assignment 2, then expand that to rank the results, making use of the tfidf(t,d) function you created above.
Can you write Python code for it?

#score_ntn(query_words, doc_id)
#    ...

#query_ntn(query_string)
#    ...


I am working on the file "03_improving.ipynb".
The first task is:
Using these three helper functions, we can now easily calculate the tf-idf weights of a term in a document by implementing the weighting formula from the slides, which you will do in the assignments below.

Assignments
Your name: ...
Task 1
Implement in the code block below the smarter_tokenize function using NLTK's function for tokenization, and the smarter_preprocess function to perform stemming in addition to case normalization.
Can you write Python code for it?

# Smarter linguistic processing

# Your code here:

#def smarter_tokenize(text):
#  ...

#def smarter_preprocess(tokens):
#  ...

# To test it:
print(smarter_preprocess(smarter_tokenize("Mary had a little group of processes worth $5.4")))


I am working on the file "03_improving.ipynb".
The first task is:
Using these three helper functions, we can now easily calculate the tf-idf weights of a term in a document by implementing the weighting formula from the slides, which you will do in the assignments below.
Assignments
Your name: ...

Change (in the code cell below) the smarter_tokenize function to use NLTK's word_tokenize function for tokenization, and the smarter_preprocess function to perform stemming in addition to case normalization. Does smarter_and_query("air sample") return the paper 26488732? Why (not)?

Note: We are generating this index on a subset of the data, as generating an index with stemming on the entire set would take up to half an hour.
Can you write Python code for it?

# Change this code according to the task above:

from functools import reduce

def smarter_tokenize(text):
    # Change this
    return text.split(' ')

def smarter_preprocess(tokens):
    result = []
    for token in tokens:
        # Change this
        result.append(token.lower())
    return result

def smarter_and_query(query): # Regular and_query using smarter_tokenize and smarter_preprocess
    return reduce(lambda a, e: a.intersection(e), [smarter_index[term] for term in smarter_preprocess(smarter_tokenize(query))])

smarter_index = defaultdict(set)
# The code below creates an inverted index based on a subset of the documents
subset = set(Abstracts.keys()).intersection(set(range(26400000,26500000)))

for (id, abstract) in ((k, Abstracts[k]) for k in subset):
    for term in smarter_preprocess(smarter_tokenize(abstract)):
        smarter_index[term].add(id)


I am working on the file "03_improving.ipynb".
The first task is:
[Write your answer text here]

Create a function tfidf(t,d) that returns the tf.idf score of term t in document d by using the tf(t,d), df(t) and num_documents() functions we defined above. The tf-idf formula can be found on the lecture slides. Test your function with the examples shown below.

You can use our old index for this task and the tasks below: You do not need to include the results from above with the smarter tokenization and preprocessing functions.
You can use the log10(n) function to calculate log10(n).
Can you write Python code for it?

# Add your code here

#print(tfidf('air', 26488732))
#print(tfidf('samples', 26488732))
#print(tfidf('monkey', 26488732))


I am working on the file "03_intro-to-tflearn/TFLearn_Digit_Recognition.ipynb".
The first task is:
Building the network
TFLearn lets you build the network by defining the layers in that network.
For this example, you'll define:

The input layer, which tells the network the number of inputs it should expect for each piece of MNIST data.
Hidden layers, which recognize patterns in data and connect the input to the output layer, and
The output layer, which defines how the network learns and outputs a label for a given image.

Let's start with the input layer; to define the input layer, you'll define the type of data that the network expects. For example,
net = tflearn.input_data([None, 100])
would create a network with 100 inputs. The number of inputs to your network needs to match the size of your data. For this example, we're using 784 element long vectors to encode our input data, so we need 784 input units.
Adding layers
To add new hidden layers, you use
net = tflearn.fully_connected(net, n_units, activation='ReLU')
This adds a fully connected layer where every unit (or node) in the previous layer is connected to every unit in this layer. The first argument net is the network you created in the tflearn.input_data call, it designates the input to the hidden layer. You can set the number of units in the layer with n_units, and set the activation function with the activation keyword. You can keep adding layers to your network by repeated calling tflearn.fully_connected(net, n_units).
Then, to set how you train the network, use:
net = tflearn.regression(net, optimizer='sgd', learning_rate=0.1, loss='categorical_crossentropy')
Again, this is passing in the network you've been building. The keywords:

optimizer sets the training method, here stochastic gradient descent
learning_rate is the learning rate
loss determines how the network error is calculated. In this example, with categorical cross-entropy.

Finally, you put all this together to create the model with tflearn.DNN(net).
Exercise: Below in the build_model() function, you'll put together the network using TFLearn. You get to choose how many layers...
Can you write Python code for it?

# Define the neural network
def build_model():
    # This resets all parameters and variables, leave this here
    tf.reset_default_graph()

    #### Your code ####
    net = tflearn.input_data([None, 28*28])
    net = tflearn.fully_connected(net, 128, activation='relu')   # Hidden
    net = tflearn.fully_connected(net, 32, activation='relu')   # Hidden
    net = tflearn.fully_connected(net, 10, activation='softmax')   # Output
    net = tflearn.regression(net, optimizer='sgd',
                             learning_rate=0.1,
                             loss='categorical_crossentropy')

    # This model assumes that your network is named "net"
    model = tflearn.DNN(net)
    return model

# Build the model
model = build_model()


I am working on the file "03_intro-to-tflearn/TFLearn_Digit_Recognition.ipynb".
The first task is:
Testing
After you're satisified with the training output and accuracy, you can then run the network on the test data set to measure it's performance! Remember, only do this after you've done the training and are satisfied with the results.
A good result will be higher than 95% accuracy. Some simple models have been known to get up to 99.7% accuracy!
Can you write Python code for it?

# Compare the labels that our model predicts with the actual labels

# Find the indices of the most confident prediction for each item. That tells us the predicted digit for that sample.
predictions = np.array(model.predict(testX)).argmax(axis=1)

# Calculate the accuracy, which is the percentage of times the predicated labels matched the actual labels
actual = testY.argmax(axis=1)
test_accuracy = np.mean(predictions == actual, axis=0)

# Print out the result
print("Test accuracy: ", test_accuracy)


I am working on the file "03_intro-to-tflearn/TFLearn_Digit_Recognition.ipynb".
The first task is:
Training the network
Now that we've constructed the network, saved as the variable model, we can fit it to the data. Here we use the model.fit method. You pass in the training features trainX and the training targets trainY. Below I set validation_set=0.1 which reserves 10% of the data set as the validation set. You can also set the batch size and number of epochs with the batch_size and n_epoch keywords, respectively.
Too few epochs don't effectively train your network, and too many take a long time to execute. Choose wisely!
Can you write Python code for it?

# Training
with tf.device('/gpu:0'):
    model.fit(trainX, trainY, validation_set=0.1, show_metric=True, batch_size=100, n_epoch=10)

from tensorflow.python.client import device_lib
device_lib.list_local_devices()


I am working on the file "03_intro-to-tflearn/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Counting word frequency
To start off we'll need to count how often each word appears in the data. We'll use this count to create a vocabulary we'll use to encode the review data. This resulting count is known as a bag of words. We'll use it to select our vocabulary and build the word vectors. You should have seen how to do this in Andrew's lesson. Try to implement it here using the Counter class.

Exercise: Create the bag of words from the reviews data and assign it to total_counts. The reviews are stores in the reviews Pandas DataFrame. If you want the reviews as a Numpy array, use reviews.values. You can iterate through the rows in the DataFrame with for idx, row in reviews.iterrows(): (documentation). When you break up the reviews into words, use .split(' ') instead of .split() so your results match ours.
Can you write Python code for it?

reviews[0][1]

from collections import Counter

total_counts = Counter()
for _, review in reviews.iterrows():
    total_counts.update([s.lower() for s in review[0].split(' ')])
print("Total words in data set: ", len(total_counts))


I am working on the file "03_intro-to-tflearn/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
If you do this right, the following code should return
```
text_to_vector('The tea is for a party to celebrate '
               'the movie so she has no time for a cake')[:65]
array([0, 1, 0, 0, 2, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 1, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0])
```
Can you write Python code for it?


I am working on the file "03_intro-to-tflearn/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Intializing the model
Next we need to call the build_model() function to actually build the model. In my solution I haven't included any arguments to the function, but you can add arguments so you can change parameters in the model if you want.

Note: You might get a bunch of warnings here. TFLearn uses a lot of deprecated code in TensorFlow. Hopefully it gets updated to the new TensorFlow version soon.
Can you write Python code for it?


I am working on the file "03_intro-to-tflearn/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Now, run through our entire review data set and convert each review to a word vector.
Can you write Python code for it?

word_vectors = np.zeros((len(reviews), len(vocab)), dtype=np.int_)
for ii, (_, text) in enumerate(reviews.iterrows()):
    word_vectors[ii] = text_to_vector(text[0])

# Printing out the first 5 word vectors
word_vectors[:5, :23]


I am working on the file "03_intro-to-tflearn/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Preparing the data
Following along with Andrew, our goal here is to convert our reviews into word vectors. The word vectors will have elements representing words in the total vocabulary. If the second position represents the word 'the', for each review we'll count up the number of times 'the' appears in the text and set the second position to that count. I'll show you examples as we build the input data from the reviews data. Check out Andrew's notebook and video for more about this.
Read the data
Use the pandas library to read the reviews and postive/negative labels from comma-separated files. The data we're using has already been preprocessed a bit and we know it uses only lower case characters. If we were working from raw data, where we didn't know it was all lower case, we would want to add a step here to convert it. That's so we treat different variations of the same word, like The, the, and THE, all the same way.
Can you write Python code for it?

reviews = pd.read_csv('reviews.txt', header=None)
labels = pd.read_csv('labels.txt', header=None)


I am working on the file "03_intro-to-tflearn/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Testing
After you're satisified with your hyperparameters, you can run the network on the test set to measure its performance. Remember, only do this after finalizing the hyperparameters.
Can you write Python code for it?

predictions = (np.array(model.predict(testX))[:,0] >= 0.5).astype(np.int_)
test_accuracy = np.mean(predictions == testY[:,0], axis=0)
print("Test accuracy: ", test_accuracy)


I am working on the file "03_intro-to-tflearn/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Text to vector function
Now we can write a function that converts a some text to a word vector. The function will take a string of words as input and return a vector with the words counted up. Here's the general algorithm to do this:

Initialize the word vector with np.zeros, it should be the length of the vocabulary.
Split the input string of text into a list of words with .split(' '). Again, if you call .split() instead, you'll get slightly different results than what we show here.
For each word in that list, increment the element in the index associated with that word, which you get from word2idx.

Note: Since all words aren't in the vocab dictionary, you'll get a key error if you run into one of those words. You can use the .get method of the word2idx dictionary to specify a default returned value when you make a key error. For example, word2idx.get(word, None) returns None if word doesn't exist in the dictionary.
Can you write Python code for it?

def text_to_vector(text):
    vector = np.zeros(len(word2idx))
    idxs = [word2idx[word.lower()] for word in text.split(' ') if word in word2idx.keys()]
    for i in idxs:
        vector[i] += 1
    return vector


I am working on the file "03_intro-to-tflearn/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Train, Validation, Test sets
Now that we have the word_vectors, we're ready to split our data into train, validation, and test sets. Remember that we train on the train data, use the validation data to set the hyperparameters, and at the very end measure the network performance on the test data. Here we're using the function to_categorical from TFLearn to reshape the target data so that we'll have two output units and can classify with a softmax activation function. We actually won't be creating the validation set here, TFLearn will do that for us later.
Can you write Python code for it?

Y = (labels=='positive').astype(np.int_)
records = len(labels)

shuffle = np.arange(records)
np.random.shuffle(shuffle)
test_fraction = 0.9

train_split, test_split = shuffle[:int(records*test_fraction)], shuffle[int(records*test_fraction):]
trainX, trainY = word_vectors[train_split,:], np.array([to_categorical(y, 2).reshape(2) for y in Y.values[train_split]])
testX, testY = word_vectors[test_split,:], np.array([to_categorical(y, 2).reshape(2) for y in Y.values[test_split]])


I am working on the file "03_intro-to-tflearn/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Training the network
Now that we've constructed the network, saved as the variable model, we can fit it to the data. Here we use the model.fit method. You pass in the training features trainX and the training targets trainY. Below I set validation_set=0.1 which reserves 10% of the data set as the validation set. You can also set the batch size and number of epochs with the batch_size and n_epoch keywords, respectively. Below is the code to fit our the network to our word vectors.
You can rerun model.fit to train the network further if you think you can increase the validation accuracy. Remember, all hyperparameter adjustments must be done using the validation set. Only use the test set after you're completely done training the network.
Can you write Python code for it?

# Training
model.fit(trainX, trainY, validation_set=0.1, show_metric=True, batch_size=128, n_epoch=50)


I am working on the file "03_intro-to-tflearn/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
Try out your own text!
Can you write Python code for it?

# Helper function that uses your model to predict sentiment
def test_sentence(sentence):
    positive_prob = model.predict([text_to_vector(sentence.lower())])[0][1]
    print('Sentence: {}'.format(sentence))
    print('P(positive) = {:.3f} :'.format(positive_prob),
          'Positive' if positive_prob > 0.5 else 'Negative')

sentence = "Moonlight is by far the best movie of 2016."
test_sentence(sentence)

sentence = "It's amazing anyone could be talented enough to make something this spectacularly awful"
test_sentence(sentence)

rev = "Wonder Woman is a superhero movie, and it fulfills the heroic and mythic demands of that genre, but it's also an entry in the genre of wisdom literature that shares hard-won insights and long-pondered paradoxes of the past with a sincere intimacy."
test_sentence(rev)

test_sentence("quite nice movie to watch after reading the book. good music and actors play. starts a little bit slow but then the viewer gets involved. well presented story about italian mafia in new york after 2nd world war. ")

rev = "this movie was not very good. The actors play was far from terrific. I would consider this film mediocre performance."
test_sentence(rev)

rev = "this movie was very good. The actors play was terrific. I would consider this film far from not mediocre performance."
test_sentence(rev)


I am working on the file "03_intro-to-tflearn/TFLearn_Sentiment_Analysis.ipynb".
The first task is:
What's the last word in our vocabulary? We can use this to judge if 10000 is too few. If the last word is pretty common, we probably need to keep more words.
Can you write Python code for it?

print(vocab[-1], ': ', total_counts[vocab[-1]])


I am working on the file "03_memmap/03_memmap.ipynb".
The first task is:
3.- Now, we flush memory changes to disk by deleting the object:
Can you write Python code for it?


I am working on the file "03_memmap/03_memmap.ipynb".
The first task is:
4.- Reading a memory-mapped array from disk involves the same memmap function. The data type and the shape need to be specified again, as this information is not stored in the file:
Can you write Python code for it?

%%memit
f = np.memmap('memmapped.dat', dtype=np.float32, shape=(nrows, ncols))

%%memit
np.array_equal(f[-1,:],x)

print(type(f[-1,:]))


I am working on the file "03_memmap/03_memmap.ipynb".
The first task is:
<div id='hdf5' />
3.- HDF5 and h5py
Why HD5 instead of native NumPy's memmap:

HDF5 (Hierarchical Data Format version 5) is an open format specification, i.e, language independet. Can be used with Python, Matlab, R, C, Java, etc.
HDF5 not only implement memory mappings but also provides a POSIX-like hierarchy to organize different arrays.
It is more versatile than NumPy's memmap. The latter (numpy.memmap) stores the array in binary form on the hard-disk using contiguous blocks of memory. The first (hdf5) work with chunks (which are atomic objects) organized in a B-tree structure. The content of a chunk is contiguously stored in the hard-disk.

There are two libraries to work with HDF5 in Python: PyTables and h5py. We use the second one since it is more lightweight and more adapted than PyTablesin some situations.
1.- Let's create a new empty HDF5 file:
Can you write Python code for it?


I am working on the file "03_memmap/03_memmap.ipynb".
The first task is:

1.- Very Large Arrays
numpy.arrays are meant to live in memory. If you want to work with matrices larger than your RAM, you have to work around that. There are at least two approaches you can follow:

Try a more efficient matrix representation that exploits any special structure that your matrices have. For example, there are efficient data structures for sparse matrices (matrices with lots of zeros), like scipy.sparse.csc_matrix.
Modify your algorithm to work on submatrices. You can read from disk only the matrix blocks that are currently being used in computations. Algorithms designed to run on clusters usually work blockwise, since the data is scatted across different computers, and passed by only when needed.

2.- NumPy's Memory Mappings
Some important features of Memory Mappings:

Memory mapping lets you work with huge arrays almost as if they were regular arrays.
Python code that accepts a NumPy array as input will also accept a memmap array.
We need to ensure that the array is used efficiently, i.e, the array is never loaded as a whole.
This method is not the most adapted for long-term storage of data and data sharing (HDF5 is it!).

1.- Let's create a memory-mapped array:
Can you write Python code for it?

nrows, ncols = 100, 1000000
f = np.memmap('memmapped.dat', dtype=np.float32, mode='w+', shape=(nrows, ncols))

print(type(f))


I am working on the file "03_memmap/03_memmap.ipynb".
The first task is:
A practical example: Huge matrix multiplication.
Supose we want to perform the dot product between $A \in \mathbb{R}^{m\times n}$ and $b \in \mathbb{R}^n$, such that $A$ doesn't fit in the available RAM memory. How can we solve it? R: Memory Mappings.
Can you write Python code for it?

f.close()

# We first create the hdf5 file, and add a dataset/array
nrows = 1000
ncols = 100000

f = h5py.File("test.h5", "w")
dset = f.create_dataset("array", (nrows,ncols), dtype='f')

%%memit
tmp = np.random.random((nrows,ncols))


I am working on the file "03_memmap/03_memmap.ipynb".
The first task is:
And perform dot product without loading $A$ (completely) into main memory:
Can you write Python code for it?

%%memit
for i in range(nrows):
    c[i] = np.dot(dset[i,:],b)

%%timeit
for i in range(nrows):
    c[i] = np.dot(dset[i,:],b)


I am working on the file "03_memmap/03_memmap.ipynb".
The first task is:
And we can visualize the matrix structure with matplotlib.pyplot.matshow function:
Can you write Python code for it?

plt.figure(figsize=(15,15))
plt.matshow(B, cmap=plt.cm.gray)
plt.show()


I am working on the file "03_memmap/03_memmap.ipynb".
The first task is:
How sparse matrices are stored in memory?
Answer: With linkded lists! (We just store the needed values in such lists). There are seven available sparse matrix types/formats:

csc_matrix: Compressed Sparse Column format
csr_matrix: Compressed Sparse Row format
bsr_matrix: Block Sparse Row format
lil_matrix: List of Lists format
dok_matrix: Dictionary of Keys format
coo_matrix: COOrdinate format (aka IJV, triplet format)
dia_matrix: DIAgonal format

CSR example
<img src='data/csr.png' style="width: 600px;">
For more info see Scipy Documentation
Important note: Despite their similarity to NumPy arrays, it is strongly discouraged to use NumPy functions directly on these matrices because NumPy may not properly convert them for computations, leading to unexpected (and incorrect) results.
A practical example again...
Can you write Python code for it?

nrows = 1000
ncols = 100000
A = sp.sparse.random(nrows, ncols, density=0.1, format='csr')
B = A.toarray()
b = np.random.random(ncols)

print("Size of A:",get_size(A),"Bytes")
print("Size of B:",get_size(B),"Bytes")

%timeit A.dot(b)

%timeit B.dot(b)


I am working on the file "03_memmap/03_memmap.ipynb".
The first task is:
Then we create the $b$ array and a $c$ empty array to store the results:
Can you write Python code for it?

b = np.random.random(ncols)
c = np.empty(ncols)


I am working on the file "03_memmap/03_memmap.ipynb".
The first task is:
We fill the array with random values, without loading the whole array
Can you write Python code for it?

%%memit -r 2
for i in range(nrows):
    tmp = np.random.random(ncols)
    dset[i,:] = tmp
    del tmp


I am working on the file "03_memmap/03_memmap.ipynb".
The first task is:
We save memory, but pay with overhead in time (The time needed to retreive each row in dset into main memory):
Can you write Python code for it?

A = np.random.random((nrows,ncols))

%%timeit
np.dot(A,b)


I am working on the file "03_memmap/03_memmap.ipynb".
The first task is:
We save the last column of the array:
Can you write Python code for it?


I am working on the file "03_numpy_io_matlab.ipynb".
The first task is:
DYI
Let's play a bit with the data loaded st_temperatures to combine fancy indexing (i.e. defining conditions to get subset of data) and very simple statistics.
For example:
Can you write Python code for it?

st_temperatures[:10, ]

st_temperatures.dtype

## Calculate which and how many years we have in our data
years = np.unique(st_temperatures[:, 0]).astype(np.int)
years, len(years)

years.min(), years.max()

!head stockholm_td_adj.dat

mask_year = st_temperatures[:, 0] == 1984

mask_feb = st_temperatures[:, 1] == 2

mask_feb.shape

mask_year.dtype

type(mask_year)

## Calculate the mean temperature of mid-days on February in 1984
feb_noon_temps = st_temperatures[(mask_year & mask_feb), 4]

type(feb_noon_temps)

feb_noon_temps.dtype

feb_noon_temps.mean()

## ....


I am working on the file "03_numpy_io_matlab.ipynb".
The first task is:
If we try to add, subtract or multiply objects with incomplatible shapes we get an error:
Can you write Python code for it?

v_incompat = matrix(list(range(1, 7))).T

M.shape, v_incompat.shape

M * v_incompat


I am working on the file "03_numpy_io_matlab.ipynb".
The first task is:
MATLAB $\mapsto$ NumPy: scipy.io.loadmat
Can you write Python code for it?

data_dictionary = spio.loadmat('numpy_to.mat')

list(data_dictionary.keys())

data_dictionary['A']

A_load = data_dictionary['A']

np.all(A == A_load)

type(A_load)


I am working on the file "03_numpy_io_matlab.ipynb".
The first task is:
Numpy's native file format

Useful when storing and reading back numpy array data.

Use the functions np.save and np.load:

np.save
Can you write Python code for it?


I am working on the file "03_numpy_io_matlab.ipynb".
The first task is:
See also the related functions: inner, outer, cross, kron, tensordot.
Try for example help(inner).

Loading and Saving .mat file
Let's create a numpy.ndarray object
Can you write Python code for it?

A = np.random.rand(10000, 300, 50)  # note: this may take a while

A


I am working on the file "03_numpy_io_matlab.ipynb".
The first task is:
See also:

np.savez : save several NumPy arrays into one single file
np.savez_compressed
np.savetxt

np.load
Can you write Python code for it?

T = np.load("st_temperatures.npy")
print(T.shape, T.dtype)


I am working on the file "03_pcf.ipynb".
The first task is:
Compute!
Can you write Python code for it?

A = 'O'
B = 'O'
df = pcf(A, B, a, twobody)


I am working on the file "03_pcf.ipynb".
The first task is:
Load the twobody data
Can you write Python code for it?

xyz = pd.read_hdf('xyz.hdf5', 'xyz')
twobody = pd.read_hdf('twobody.hdf5', 'twobody')


I am working on the file "03_pcf.ipynb".
The first task is:
Plot!
Can you write Python code for it?

import seaborn as sns
sns.set_context('poster', font_scale=1.3)
sns.set_style('white')
sns.set_palette('colorblind')

# Lets modify a copy of the data for plotting
plotdf = df.set_index('r')
plotdf.columns = ['PCF', 'Pair Count']

# Generate the plot
ax = plotdf.plot(secondary_y='Pair Count')
ax.set_ylabel('Pair Correlation Function ({0}, {1})'.format(A, B))
ax.right_ax.set_ylabel('Pair Count ({0}, {1})'.format(A, B))
ax.set_xlabel('Distance ($\AA$)')
patches, labels = ax.get_legend_handles_labels()
patches2, labels2 = ax.right_ax.get_legend_handles_labels()
legend = ax.legend(patches+patches2, labels+labels2, loc='upper center', frameon=True)
frame = legend.get_frame()
frame.set_facecolor('white')
frame.set_edgecolor('black')


I am working on the file "03_pcf.ipynb".
The first task is:
Save the everything for later
First generate a beautiful graph...
Can you write Python code for it?

df1 = pcf('O', 'O', a, twobody)
df2 = pcf('O', 'H', a, twobody)
df3 = pcf('H', 'H', a, twobody)
df = pd.concat((df1, df2, df3), axis=1)
df.columns = ['$g_{OO}$', '$n_{OO}$', '$r$', '$g_{OH}$', '$n_{OH}$', 'del1', '$g_{HH}$', '$n_{HH}$', 'del2']
del df['del1']
del df['del2']
df.set_index('$r$', inplace=True)
ax = df.plot(secondary_y=['$n_{OO}$', '$n_{OH}$', '$n_{HH}$'])
ax.set_ylabel('Pair Correlation Function ($g_{AB}$)')
ax.right_ax.set_ylabel('Pairwise Count ($n_{AB}$)')
ax.set_xlabel('Distance ($\AA$)')
ax.set_ylim(0, 5)
ax.right_ax.set_ylim(0, 20)
patches, labels = ax.get_legend_handles_labels()
patches2, labels2 = ax.right_ax.get_legend_handles_labels()
legend = ax.legend(patches+patches2, labels+labels2, loc='upper right', frameon=True)
frame = legend.get_frame()
frame.set_facecolor('white')
frame.set_edgecolor('black')

# Save the figure
fig = ax.get_figure()
fig.savefig('pcf.pdf')

# Save the pcf data
store = pd.HDFStore('pcf.hdf5', mode='w')
store.put('pcf', df)
store.close()


I am working on the file "03_performance.ipynb".
The first task is:
Can we make the Operator run quicker? Yes !
What we are missing so far is performance tuning. Take loop blocking, for example. This is a parametric loop transformation: its impact will vary depending on block shape, size and scheduling. In the literature, over the years, dozens of different loop blocking strategies have been studied! Even though we used the simplest loop blocking scheme on Earth, we would need at least to come up with a block size fitting in some level of cache. Obviously, this is such a low level detail... and we don't want users to waste any time on thinking about these matters.
Like other frameworks, Devito can automatically detect a "sufficiently good" block size through an auto-tuning engine. Let's try it out; in particular, we tell Devito to be "aggressive" in the search for block sizes. Being "aggressive" means that more time will be spent on auto-tuning, but there's a greater chance of retrieving a better candidate. We can either do this by setting an environment variable (export DEVITO_AUTOTUNING=aggressive) or directly through the configuration dictionary.
Can you write Python code for it?

configuration['autotuning'] = 'aggressive'

%run $benchmark run -P acoustic -so 8 -d 256 256 256 --tn 50 -dse advanced -dle advanced -a


I am working on the file "03_performance.ipynb".
The first task is:
Compared to the previous runs, do you note any change ...

in the Devito output reports?
in execution times? why?

And why, from a performance analysis point of view, is the DSE useful even though no changes in execution times are observed?
DLE - Devito Loop Engine
We know how to switch on parallelism and how to leverage the DSE to reduce the flop count of our stencil equations. What's still missing are SIMD vectorization and optimizations for data locality. We won't be able to reach a significant fraction of the attainable machine peak without aggressive loop transformations. Clearly, Devito users don't "see" loops -- in fact, they only write maths in Python! -- but the generated code is nothing more than classic C with loop nests for grid function updates. So how do these "low-level loops" get optimized? Quoting from the documentation:

Loop optimizations - Examples include SIMD vectorization and loop blocking. These are performed by the Devito Loop Engine (DLE), a sub-module consisting of a sequence of compiler passes manipulating abstract syntax trees [...]

In other words, the Devito compiler, through the DLE module, automatically applies loop transformations. The how it does that(i.e., manipulation of an intermediate representation), here, is irrelevant; we rather want to understand what the DLE can do, how to use it and what kind of tuning is required to maximize the performance of an Operator. As we shall see, using and tuning the DLE will be as simple as switching on some special environment variables!
So here's a (non complete) list of transformations that the DLE will automatically apply for you:

SIMD Vectorization
Loop blocking
Optimization of so called "remainder" loops
Creation of elemental functions

OK, let's run the same problem we ran above, but this time with the DLE at maximum level (it's gonna apply all optimizations listed above). Can you guess whether the performance will be substantially better? Why?
Can you write Python code for it?

%run $benchmark run -P acoustic -so 8 -d 256 256 256 --tn 50 -dse advanced -dle advanced


I am working on the file "03_performance.ipynb".
The first task is:
If one didn't have access to the Intel compiler, it would still be possible to enable thread pinning through analogous mechanisms provided by OpenMP 4.5, namely the OMP_PLACES and OMP_PROC_BIND variables.
Can you write Python code for it?

# Uncomment if necessary; note that the available GCC version must support OpenMP 4.5 for the following to have effect
# %env OMP_PROC_BIND=close
# %env OMP_PLACES=cores
# %env DEVITO_ARCH=gcc


I am working on the file "03_performance.ipynb".
The first task is:
Let's see how threading and pinning impact the Operator performance.
We run the isotropic acoustic forward operator again, but this time with a much larger grid, a 256x256x256 cube, and a more realistic space order, 8. We also shorten the duration by deliberately choosing a very small simulation end time (50 ms).
Can you write Python code for it?

for i in range(3):
    print ("Run %d" % i)
    %run $benchmark run -P acoustic -so 8 -d 256 256 256 --tn 50


I am working on the file "03_performance.ipynb".
The first task is:
Multiple threads will now be used when running an Operator. But how many? And how efficiently? We may be running on a multi-socket node -- how should we treat it, as a "flatten system" or what?
Devito aims to use distributed-memory parallelism over multi-socket nodes; that is, it allocates one MPI process per socket, and each MPI process should spawn as many OpenMP threads as the number of cores on the socket. Users don't get all this for free, however; a minimal configuration effort is required. But don't worry: as you shall see, it's much simpler than it sounds!
For this tutorial, we forget about MPI; we rather focus on enabling OpenMP on a single socket. So, first of all, we want to restrain execution to a single socket -- we want threads to stay on that socket without ever migrating to other cores of the system due to OS scheduling. Are we really on a multi-socket node? And how many physical cores does a socket have? Let's find out. We shall use a very standard tool such as lscpu on Linux systems.
Can you write Python code for it?


I am working on the file "03_performance.ipynb".
The first task is:
Now we can experiment a little with YASK. Watch out, however, as this is work in progress: the performance may still be quite far from the attainable peaks (e.g., Devito cannot still leverage YASK's auto-tuning system) and some Operators are still unsupported, although our running example should just work out-of-the-box.
Can you write Python code for it?

%run $benchmark run -P acoustic -so 8 -d 256 256 256 --tn 50 -dse advanced


I am working on the file "03_performance.ipynb".
The first task is:
That was simple. Of course, we may want to run the same simulation on a bigger grid, with different grid point spacing or space order, and so on. And yes, we'll do this later on in the tutorial. But before scaling up in problem size, we shall take a look at what sort of performance optimizations we'll be able to apply to speed it up.
In essence, there are four knobs we can play with to maximize the Operator performance (or to see how the performace varies when adding or removing specific transformations):

parallelism,
the Devito Symbolic Engine (DSE) optimization level,
the Devito Loop Engine (DLE) optimization level,
loop blocking auto-tuning.

Shared-memory parallelism
We start with the most obvious -- parallelism. Devito implements shared-memory parallelism via OpenMP. To enable it, we would usually simply set the environment variable DEVITO_OPENMP=1. However, since we are running in a Jupyter notebook, we can change the relevant configuration option directly:
Can you write Python code for it?

from devito import configuration
configuration['openmp'] = True


I am working on the file "03_performance.ipynb".
The first task is:
This exercise asks you to repeat the same study we did for acoustic, playing with different DSE and DLE levels. Oh, and don't forget, there's another DSE level that you can try besides advanced, namely aggressive; just give it a go. How does the performance impact of DSE/DLE vary w.r.t. isotropic acoustic operator? And why?
A sneak peek at the YASK backend
YASK -- Yet Another Stencil Kernel -- is, as described on its GitHub page

a framework to facilitate exploration of the HPC stencil-performance design space.

The fundamental point is that YASK operates at a level of abstraction lower than that of Devito; for example, besides being a static system written in C++, no symbolic language is offered. We've been working on integrating YASK, meaning that, under the hood, Devito may "offload" (some of) the Operator stencil updates onto YASK. This is totally transparent to users; no changes to existing user code are required to exploit YASK! The goal is to create a synergy between the simplicity of use of Devito (i.e., the high-level abstractions) and a powerful optimization system such as YASK, specifically conceived to optimize stencil codes.
So, how do we try out YASK? Again, it's as difficult as setting one more environment variable: DEVITO_BACKEND=yask. In a notebook, we simply run the following:
Can you write Python code for it?

configuration['backend'] = 'yask'
# Decrease Devito logger verbosity, as currently YASK produces lots of debug messages
configuration['log_level'] = 'INFO'


I am working on the file "03_performance.ipynb".
The first task is:
Thus, 16 threads will be spawned each time an Operator is run. They will be killed as soon as the Operator has done its job.
We also want to bind them to the physical cores of socket 0; that is, we want to prevent OS-induced migration. This is known as thread pinning. One may use a program such as numactl or, alternatively, exploit other OpenMP environment variables. If the Intel compiler is at our disposal, we can enforce pinning through the following two-step procedure:

We tell Devito to use the Intel compiler through the special DEVITO_ARCH environment variable;
We set the Intel-specific KMP_HW_SUBSET and KMP_AFFINITY environment variables.
Can you write Python code for it?

# Thread pinning
%env KMP_HW_SUBSET=16c,1t
%env KMP_AFFINITY=compact
# Tell Devito to use the Intel compiler
%env DEVITO_ARCH=intel
# Or, analogously, using the configuration dictionary
configuration['compiler'] = 'intel'


I am working on the file "03_planck_distribution.ipynb".
The first task is:
SciPy has a quite extensive list of physical constants, fundamental or otherwise, that you can use.
For example, Avogadro's constants is simply:
Can you write Python code for it?


I am working on the file "03_planck_distribution.ipynb".
The first task is:
The conversion factor between Joule and electronvolt can also be obtained. You can either search in the physical_constants dictionary or call the value function:
Can you write Python code for it?

print("Joule to electronvolt:", physical_constants['joule-electron volt relationship'])
print("Joule to electronvolt:", value('joule-electron volt relationship'))


I am working on the file "03_planck_distribution.ipynb".
The first task is:
What's the difference between the two? physical_constants is a dictionary, meaning that it is a list of values that can be searched via a string. Much like a phone book! By looking at the key joule-electron volt relationship we can retrieve the value (6.241509126e+18, 'eV', 38000000000.0).
This is called a tuple and contains the following information:

the numeric value of the conversion factor (6.241509126e+18)
the unit of measure ('eV')
the uncertainty (38000000000.0)

The value function is a convenience function that takes the key to be searched in the physical_constants dictionary and returns just the numeric value, instead of the whole tuple.
You can find the entire list of keys in the physical_constants dictionary here.
Exercise 1: The visible spectrum
The following picture shows the visible part of the electromagnetic spectrum. The wavelength $\lambda$ associated with each colour in the spectrum is reported in nanometers.
<figure>
  <IMG SRC="gfx/visible.jpg" WIDTH=800>
</figure>

Your task is to convert each of the reported wavelengths into $\mathrm{A}$, $\mathrm{Hz}$ and $\mathrm{J}$.
You have to write a Python function for each conversion. The Python function will use the SciPy module to get the correct conversion factor and will have to return to converted value:
Python
def convert_to_angstrom(lambda):
    import scipy
    # Which conversion factor?
    # Which operation?
    angstrom = lambda * ...
    # Return converted value
    return angstrom
Remember! You have to take into account the unit of measure of the incoming value lambda!
The wavelength to frequency and frequency to energy conversions are given by:
\begin{equation}
  c = \nu\lambda \quad\quad\quad\quad E = h\nu
\end{equation}
It is more common to find energies reported in another unit: the electronvolt (symbol eV). Convert the energies from Joule to electronvolt, using the conversion factor given by SciPy. Define another function for this task.
One additional unit that is quite often used in spectroscopy is the...
Can you write Python code for it?

import matplotlib.pyplot as plt
import numpy as np
import scipy as sp
%matplotlib inline

x = np.linspace(-4, 4, 10)
y = 10 * x**3 * np.exp(-x**2) + np.sin(x**5) / np.cos(x**3)

plt.plot(x, y)


I am working on the file "03_problem_representation/ensemble_methods.ipynb".
The first task is:
Check that our tf.data dataset:
Can you write Python code for it?

def show_batch(dataset):
    for batch, label in dataset.take(1):
        for key, value in batch.items():
            print("{:20s}: {}".format(key,value.numpy()))

show_batch(train_data)


I am working on the file "03_problem_representation/ensemble_methods.ipynb".
The first task is:
Create our feature columns
Can you write Python code for it?

numeric_columns = [fc.numeric_column("mother_age"),
                  fc.numeric_column("gestation_weeks")]

CATEGORIES = {
    'plurality': ["Single(1)", "Twins(2)", "Triplets(3)",
                  "Quadruplets(4)", "Quintuplets(5)", "Multiple(2+)"],
    'is_male' : ["True", "False", "Unknown"],
    'mother_race': [str(_) for _ in df.mother_race.unique()]
}

categorical_columns = []
for feature, vocab in CATEGORIES.items():
  cat_col = fc.categorical_column_with_vocabulary_list(
        key=feature, vocabulary_list=vocab)
  categorical_columns.append(fc.indicator_column(cat_col))


I am working on the file "03_problem_representation/ensemble_methods.ipynb".
The first task is:
Create our tf.data input pipeline
Can you write Python code for it?

# Determine CSV, label, and key columns
# Create list of string column headers, make sure order matches.
CSV_COLUMNS = ["weight_pounds",
               "is_male",
               "mother_age",
               "plurality",
               "gestation_weeks",
               "mother_race"]

# Add string name for label column
LABEL_COLUMN = "weight_pounds"

# Set default values for each CSV column as a list of lists.
# Treat is_male and plurality as strings.
DEFAULTS = [[0.0], ["null"], [0.0], ["null"], [0.0], ["0"]]

def get_dataset(file_path):
    dataset = tf.data.experimental.make_csv_dataset(
        file_path,
        batch_size=15, # Artificially small to make examples easier to show.
        label_name=LABEL_COLUMN,
        select_columns=CSV_COLUMNS,
        column_defaults=DEFAULTS,
        num_epochs=1,
        ignore_errors=True)
    return dataset

train_data = get_dataset("./data/babyweight_train.csv")
test_data = get_dataset("./data/babyweight_eval.csv")


I am working on the file "03_problem_representation/ensemble_methods.ipynb".
The first task is:
Lastly, we will evaluate our Stacked Ensemble against the test set.
Can you write Python code for it?

val_loss, val_mse = ensemble_model.evaluate(test_data.map(stack_features))

print("Validation RMSE: {}".format(val_mse**0.5))


I am working on the file "03_problem_representation/ensemble_methods.ipynb".
The first task is:
Lastly, we'll create our Stacked Ensemble model. It is also a neural network. We'll use the Functional Keras API.
Can you write Python code for it?

member_inputs = [model.input for model in members]

# concatenate merge output from each model
member_outputs = [model.output for model in members]
merge = layers.concatenate(member_outputs)
h1 = layers.Dense(30, activation='relu')(merge)
h2 = layers.Dense(20, activation='relu')(h1)
h3 = layers.Dense(10, activation='relu')(h2)
h4 = layers.Dense(5, activation='relu')(h2)
ensemble_output = layers.Dense(1, activation='relu')(h3)
ensemble_model = Model(inputs=member_inputs, outputs=ensemble_output)

# plot graph of ensemble
tf.keras.utils.plot_model(ensemble_model, show_shapes=True, to_file='ensemble_graph.png')

# compile
ensemble_model.compile(loss='mse', optimizer='adam', metrics=['mse'])


I am working on the file "03_problem_representation/ensemble_methods.ipynb".
The first task is:
We will need to freeze the layers of the pre-trained models since we won't train these models any further. The Stacked Ensemble will the trainable and learn how to best combine the results of the ensemble members.
Can you write Python code for it?

# update all layers in all models to not be trainable
for i in range(n_members):
    model = members[i]
    for layer in model.layers:
        # make not trainable
        layer.trainable = False
        # rename to avoid 'unique layer name' issue
        layer._name = 'ensemble_' + str(i+1) + '_' + layer.name


I am working on the file "03_problem_representation/multilabel.ipynb".
The first task is:
Parsing sigmoid results
Unlike softmax output, we can't simply take the argmax of the output probability array. We need to consider our thresholds for each class. In this case, we'll say that a tag is associated with a question if our model is more than 70% confident.
Below we'll print the original question along with our model's predicted tags.
Can you write Python code for it?

# Get some test predictions
predictions = model.predict(body_test[:3])

classes = tag_encoder.classes_

for q_idx, probabilities in enumerate(predictions):
  print(test_qs[q_idx])
  for idx, tag_prob in enumerate(probabilities):
    if tag_prob > 0.7:
      print(classes[idx], round(tag_prob * 100, 2), '%')
  print('')


I am working on the file "03_problem_representation/multilabel.ipynb".
The first task is:
Sigmoid output for binary classification
Typically, binary classification is the only type of multilabel classification (each input has only one class) where you'd want to use sigmoid output. In this case, a 2-element softmax output is redundant and can increase training time.
To demonstrate this we'll build a model on the UCI mushroom dataset to determine whether a mushroom is edible or poisonous.
Can you write Python code for it?

# First, download the data. We've made it publicly available in Google Cloud Storage
!gsutil cp gs://ml-design-patterns/mushrooms.csv .

mushroom_data = pd.read_csv('mushrooms.csv')
mushroom_data.head()


I am working on the file "03_problem_representation/multilabel.ipynb".
The first task is:
To keep things simple, we'll first convert the label column to numeric and then
use pd.get_dummies() to covert the data to numeric.
Can you write Python code for it?

# 1 = edible, 0 = poisonous
mushroom_data.loc[mushroom_data['class'] == 'p', 'class'] = 0
mushroom_data.loc[mushroom_data['class'] == 'e', 'class'] = 1

labels = mushroom_data.pop('class')

dummy_data = pd.get_dummies(mushroom_data)

# Split the data
train_size = int(len(mushroom_data) * .8)

train_data = dummy_data[:train_size]
test_data = dummy_data[train_size:]

train_labels = labels[:train_size]
test_labels = labels[train_size:]

model = keras.Sequential([
    keras.layers.Dense(32, input_shape=(len(dummy_data.iloc[0]),), activation='relu'),
    keras.layers.Dense(8, activation='relu'),
    keras.layers.Dense(1, activation='sigmoid')
])

model.summary()

# Since we're using sigmoid output, we use binary_crossentropy for our loss function
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(train_data.values.tolist(), train_labels.values.tolist())

model.evaluate(test_data.values.tolist(), test_labels.values.tolist())


I am working on the file "03_problem_representation/multilabel.ipynb".
The first task is:
🥑🥑🥑
We've pre-processed this dataset to remove any uses of the tag within a question and replaced it with the word "avocado". For example, the question: "How do i feed a pandas dataframe to a keras model?" would become "How do I feed a avocado dataframe to a avocado model?" This will help the model learn more nuanced patterns throughout the data, rather than just learning to associate the occurrence of the tag itself in a question.
Can you write Python code for it?

data = pd.read_csv('so_data.csv', names=['tags', 'original_tags', 'text'], header=0)
data = data.drop(columns=['original_tags'])
data = data.dropna()

data = shuffle(data, random_state=22)
data.head()

# Encode top tags to multi-hot
tags_split = [tags.split(',') for tags in data['tags'].values]
print(tags_split[0])

tag_encoder = MultiLabelBinarizer()
tags_encoded = tag_encoder.fit_transform(tags_split)
num_tags = len(tags_encoded[0])
print(data['text'].values[0][:110])
print(tag_encoder.classes_)
print(tags_encoded[0])

# Split our data into train and test sets
train_size = int(len(data) * .8)
print ("Train size: %d" % train_size)
print ("Test size: %d" % (len(data) - train_size))

# Split our labels into train and test sets
train_tags = tags_encoded[:train_size]
test_tags = tags_encoded[train_size:]

train_qs = data['text'].values[:train_size]
test_qs = data['text'].values[train_size:]

from tensorflow.keras.preprocessing import text

VOCAB_SIZE=400 # This is a hyperparameter, try out different values for your dataset

tokenizer = text.Tokenizer(num_words=VOCAB_SIZE)
tokenizer.fit_on_texts(train_qs)

body_train = tokenizer.texts_to_matrix(train_qs)
body_test = tokenizer.texts_to_matrix(test_qs)

# Note we're using sigmoid output with binary_crossentropy loss
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(50, input_shape=(VOCAB_SIZE,), activation='relu'))
model.add(tf.keras.layers.Dense(25, activation='relu'))
model.add(tf.keras.layers.Dense(num_tags, activation='sigmoid'))

model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.summary()

# Train and evaluate the model
model.fit(body_train, train_tags, epochs=3, batch_size=128, validation_split=0.1)
print('Eval loss/accuracy:{}'.format(
  model.evaluate(body_test, test_tags, batch_size=128)))


I am working on the file "03_problem_representation/neutral.ipynb".
The first task is:
On the Natality data
Let's do this on real data.
A baby with an Apgar score of 10 is healthy and one with an Apgar score of <= 7 requires some medical attention.
What about babies with scores of 8-9? They are neither perfectly healthy, nor do they need serious medical intervention.
Let's see how the model does with a 2-class model and with a 3-class model that includes a Neutral class.
First, without the Neutral class
Can you write Python code for it?

%%bigquery
CREATE OR REPLACE MODEL mlpatterns.neutral_2classes
OPTIONS(model_type='logistic_reg', input_label_cols=['health']) AS

SELECT
  IF(apgar_1min >= 9, 'Healthy', 'NeedsAttention') AS health,
  plurality,
  mother_age,
  gestation_weeks,
  ever_born
FROM `bigquery-public-data.samples.natality`
WHERE apgar_1min <= 10

%%bigquery
SELECT * FROM ML.EVALUATE(MODEL mlpatterns.neutral_2classes)


I am working on the file "03_problem_representation/neutral.ipynb".
The first task is:
With 3 classes (including a neutral class)
Can you write Python code for it?

%%bigquery
CREATE OR REPLACE MODEL mlpatterns.neutral_3classes
OPTIONS(model_type='logistic_reg', input_label_cols=['health']) AS

SELECT
  IF(apgar_1min = 10, 'Healthy', IF(apgar_1min >= 8, 'Neutral', 'NeedsAttention')) AS health,
  plurality,
  mother_age,
  gestation_weeks,
  ever_born
FROM `bigquery-public-data.samples.natality`
WHERE apgar_1min <= 10

%%bigquery
SELECT * FROM ML.EVALUATE(MODEL mlpatterns.neutral_3classes)


I am working on the file "03_problem_representation/rebalancing.ipynb".
The first task is:
Downsampling
To demonstrate downsampling, we'll be using this synthetic fraud detection dataset from Kaggle. We've made a version of it available in a public Cloud Storage bucket.
Can you write Python code for it?

# Download the data and preview
!gsutil cp gs://ml-design-patterns/fraud_data_kaggle.csv .

fraud_data = pd.read_csv('fraud_data_kaggle.csv')
fraud_data.head()

# Drop a few columns we won't use for this demo
fraud_data = fraud_data.drop(columns=['nameOrig', 'nameDest', 'isFlaggedFraud'])
fraud_data = pd.get_dummies(fraud_data)

# Split into separate dataframes
fraud = fraud_data[fraud_data['isFraud'] == 1]
not_fraud = fraud_data[fraud_data['isFraud'] == 0]

# Take a random sample of non-fraud data
# The .005 frac will give us around an 80/20 split of not-fraud/fraud samples
# You can experiment with this value
not_fraud_sample = not_fraud.sample(random_state=2, frac=.005)

# Put the data back together and shuffle
fraud_data = pd.concat([not_fraud_sample,fraud])
fraud_data = shuffle(fraud_data, random_state=2)

# Look at our data balance after downsampling
fraud_data['isFraud'].value_counts()

train_test_split = int(len(fraud_data) * .8)

train_data = fraud_data[:train_test_split]
test_data = fraud_data[train_test_split:]

train_labels = train_data.pop('isFraud')
test_labels = test_data.pop('isFraud')

model = xgb.XGBRegressor(
    objective='reg:linear'
)

model.fit(train_data.values, train_labels)

# Get some test predictions
y_pred = model.predict(test_data.values)

# To build a confusion matrix using the scikit utility, we'll need the values as ints
y_pred_formatted = []

for i in y_pred:
  y_pred_formatted.append(int(round(i)))

cm = confusion_matrix(test_labels.values, y_pred_formatted)
print(cm)

# This is from the sklearn docs
# https://scikit-learn.org/0.18/auto_examples/model_selection/plot_confusion_matrix.html
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
   ...


I am working on the file "03_problem_representation/rebalancing.ipynb".
The first task is:
First, let's look at the cluster prediction results for an "average" example from our dataset.
Can you write Python code for it?

%%bigquery average_pred --project sara-cloud-ml
SELECT
  *
FROM
  ML.PREDICT (MODEL `sara-cloud-ml.natality.baby_weight_clusters`,
    (
    SELECT
      7.0 as weight_pounds,
      28 as mother_age,
      40 as gestation_weeks
     )
  )

average_pred


I am working on the file "03_problem_representation/rebalancing.ipynb".
The first task is:
Here there's a high distance from each cluster, which we can use to conclude that this might be an anomaly.
Can you write Python code for it?


I am working on the file "03_problem_representation/rebalancing.ipynb".
The first task is:
Here, it's fairly obvious that this datapoint should be put in cluster 1, given the short distance from that cluster.
Can you write Python code for it?

# Print the resulting cluster distances
df['NEAREST_CENTROIDS_DISTANCE'].iloc[0]


I am working on the file "03_problem_representation/rebalancing.ipynb".
The first task is:
Let's compare this with a cluster prediction for an outlier baby weight.
Can you write Python code for it?

%%bigquery outlier_pred --project sara-cloud-ml
SELECT
  *
FROM
  ML.PREDICT (MODEL `sara-cloud-ml.natality.baby_weight_clusters`,
    (
    SELECT
      3.0 as weight_pounds,
      20 as mother_age,
      27 as gestation_weeks
     )
  )

outlier_pred


I am working on the file "03_problem_representation/rebalancing.ipynb".
The first task is:
Now let's try with weighted classes and add a bias initializer to our output layer. First, calculate the class weights.
Can you write Python code for it?

# Get number of examples for each class from the training set
num_minority = train_labels.value_counts()[1]
num_majority = train_labels.value_counts()[0]

minority_class_weight = 1 / (num_minority / len(train_data)) / 2
majority_class_weight = 1 / (num_majority / len(train_data)) / 2

# Pass the weights to Keras in a dict
# The key is the index of each class
keras_class_weights = {0: majority_class_weight, 1: minority_class_weight}
print(keras_class_weights)

# Calculate output bias
output_bias = math.log(num_minority / num_majority)
print(output_bias)

fraud_model = keras.Sequential([
    keras.layers.Dense(16, input_shape=(len(train_data.iloc[0]),), activation='relu'),
    keras.layers.Dropout(0.25),
    keras.layers.Dense(16, activation='relu'),
    keras.layers.Dense(1, activation='sigmoid', bias_initializer=tf.keras.initializers.Constant(output_bias))
])

metrics = [
      tf.keras.metrics.BinaryAccuracy(name='accuracy'),
      tf.keras.metrics.Precision(name='precision'),
      tf.keras.metrics.Recall(name='recall'),
      tf.keras.metrics.AUC(name='roc_auc'),
]

fraud_model.compile(loss='binary_crossentropy', optimizer='adam', metrics=metrics)

fraud_model.fit(train_dataset, validation_data=test_dataset, epochs=10, class_weight=keras_class_weights)


I am working on the file "03_problem_representation/rebalancing.ipynb".
The first task is:
We'll take all of the fraud examples from this dataset, and a subset of non-fraud. Then we'll shuffle and combine and look at the number of examples we have for each class.
Can you write Python code for it?

%%bigquery fraud_df --project sara-cloud-ml
SELECT
  *
FROM
  `bigquery-public-data.ml_datasets.ulb_fraud_detection`
WHERE Class = 1

# This query will take a a minute to run
%%bigquery nonfraud_df --project sara-cloud-ml
SELECT
  *
FROM
  `bigquery-public-data.ml_datasets.ulb_fraud_detection`
WHERE Class = 0
AND RAND() < 0.05

bq_fraud_data = pd.concat([fraud_df, nonfraud_df])
bq_fraud_data.sort_values(by=['Time'])
# bq_fraud_data = shuffle(bq_fraud_data, random_state=22)

# Scale time and amount values
time_scaler = MinMaxScaler()
amt_scaler = MinMaxScaler()

bq_fraud_data['Time'] = time_scaler.fit_transform(bq_fraud_data['Time'].values.reshape(-1,1))
bq_fraud_data['Amount'] = amt_scaler.fit_transform(bq_fraud_data['Amount'].values.reshape(-1,1))

# See data balance
bq_fraud_data['Class'].value_counts()

train_test_split = int(len(bq_fraud_data) * .8)

train_data = bq_fraud_data[:train_test_split]
test_data = bq_fraud_data[train_test_split:]

train_labels = train_data.pop('Class')
test_labels = test_data.pop('Class')

# Create a tf dataset
train_dataset = tf.data.Dataset.from_tensor_slices((train_data.values, train_labels))
train_dataset = train_dataset.shuffle(len(train_data)).batch(1024)

test_dataset = tf.data.Dataset.from_tensor_slices((test_data.values, test_labels))
test_dataset = test_dataset.shuffle(len(test_data)).batch(1)


I am working on the file "03_problem_representation/rebalancing.ipynb".
The first task is:
Weighted classes and output bias
To demonstrate weighted classes, we'll use a different fraud detection dataset in BigQuery. This one has far fewer minority class examples than the one used in the example above.
Can you write Python code for it?

# To access BigQuery, you'll need to authenticate to your Cloud account
from google.colab import auth
auth.authenticate_user()