I am working on the file "09/homework-9-schuetz.ipynb".
The first task is:
PART THREE: Doing it in bulk
Read in the csv of the past 30 days of 1.0+ earthquke activity from http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/1.0_month.csv (tip: read_csv works with URLs!)
<p>Because we haven't covered looping through pandas, use the following code to convert a pandas DataFrame into a list of dictionaries that you can loop through.
Can you write Python code for it?

earthquakes_df = pd.read_csv("http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/1.0_month.csv")
earthquakes = earthquakes_df.to_dict('records')

earthquakes[1]
# eq_to_sentence(earthquakes)


I am working on the file "09_02_random_distributions.ipynb".
The first task is:
Random walk methods: the Metropolis algorithm
Suppose that we want to generate random variables according to an
arbitrary probability density $P(x)$. The Metropolis algorithm produces
a “random walk” of points ${x_i}$ whose asymptotic probability
approaches $P(x)$ after a large number of steps. The random walk is
defined by a “transition probability” $w(x_i \rightarrow x_j)$ for one
value $x_i$ to another $x_j$ in order that the distribution of points
$x_0$, $x_1$, $x_2$, ... converges to $P(x)$. In can be shown that it is
sufficient (but not necessary) to satisfy the “detailed balance”
condition $$p(x_i)w(x_i \rightarrow x_j) = p(x_j)w(x_j \rightarrow x_i).
$$ This relation dos not specify $w(x_i \rightarrow x_j)$
uniquely. A simple choice is
$$w(x_i \rightarrow x_j)=\min{\left[ 1,\frac{P(x_j)}{P(x_i)} \right] }.$$
This choice can be described by the following steps. Suppose that the
“random walker” is a position $x_n$. To generate $x_{n+1}$ we

choose a trial position $x_t=x_n+\delta _n$ , where the $\delta _n$
    is a random number in the interval $[-\delta ,\delta]$.

Calculate $w=P(x_t)/P(x_n)$.

If $w \geq 1$ we accept the change and let $x_{n+1}=x_t$.

If $w \leq 1$, generate a random number $r$.

If $r \leq w$, accept the change and let $x_{n+1} = x_t$.

If the trial change is not accepted, the let $x_{n+1}=x_n$.

It is necessary to sample a number of points of the random walk before
the asymptotic probability $P(x)$ is attained. How do we choose the
“step size” $\delta$? If $\delta$ is too large, only a small fraction of
changes will be accepted and the sampling will be inefficient. If
$\delta$ is too small, a large number will be accepted, but it would
take too long to sample $P(x)$ over the whole interval of interest.
Ideally, we want at least 1/3-1/2 of the trial steps to be accepted. We
also want to choose $x_0$ such that the distribution ${x_i}$ converges
to $P(x)$ as quickly as possible. An obvious choice is to begin the
random walk at the point where $P(x)$ is maximum.
Exercise 9.1:...
Can you write Python code for it?

N = 100000

x = np.zeros(N)
delta = 2.
sigma = 10.
sigma2 = sigma**2

def metropolis(xold):
    xtrial = np.random.random()
    xtrial = xold+(2*xtrial-1)*delta
    weight = np.exp(-0.5*(xtrial**2-xold**2)/sigma2)
    xnew = xold
    if(weight >= 1): #Accept
        xnew = xtrial
    else:
        r = np.random.random()
        if(r <= weight): #Accept
            xnew = xtrial
    return xnew

xwalker = 20.
Nwarmup = 100000
for i in range(Nwarmup):
    xwalker = metropolis(xwalker)

x[0] = xwalker
tot = x[0]
for i in range(1,N):
    x[i] = metropolis(x[i-1])

binwidth=sigma/10
pyplot.hist(x,bins=np.arange(-50,50., binwidth),normed=True);

norm = 1./(sigma*np.sqrt(2*np.pi))
pyplot.plot(np.arange(-50.,50.,binwidth),norm*np.exp(-0.5*np.arange(-50.,50.,binwidth)**2/sigma2),ls='-',c='red',lw=3);


I am working on the file "09_Flowers_tranfer_learning.ipynb".
The first task is:
1. extract feature & save feature data
model를 생성하고 predict하여 이미지로 부터 transfer values를 생성할 것입니다.
Can you write Python code for it?

import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
from torchvision import datasets, transforms
from torch.autograd import Variable
import matplotlib.pyplot as plt
import numpy as np

is_cuda = torch.cuda.is_available() # cuda 사용가능시, True
traindir = '/tmp/flower_photos'

normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

batch_size = 1
train_loader = torch.utils.data.DataLoader(
    datasets.ImageFolder(traindir,
                         transforms.Compose([
                             transforms.RandomSizedCrop(224),
                             transforms.RandomHorizontalFlip(),
                             transforms.ToTensor(),
                             normalize,])),
    batch_size=batch_size,
    shuffle=True)

cls_num = len(datasets.folder.find_classes(traindir)[0])

model = torchvision.models.resnet152(pretrained = True)
# remove last fully-connected layer
model = nn.Sequential(*list(model.children())[:-1])

model.eval()
features = []
targets = []
for image, target in train_loader:
    image, target = Variable(image, volatile=True), Variable(target, volatile=True)
    if is_cuda :  image, target = image.cuda(), target.cuda()
    feature = model(image).data
    features.append(feature)
    targets.append(target.data.squeeze())

features = torch.cat(features, 0).squeeze()
targets = torch.cat(targets, 0)
torch.save(features, 'flower_feature.pth')
torch.save(targets, 'flower_label.pth')


I am working on the file "09_Flowers_tranfer_learning.ipynb".
The first task is:
2. 사전 설정

model

2048개의 특징을 받아서 5개의 꽃 종류로 표시하는 fully connect layer를 만듭니다.

loss
opimizer
Can you write Python code for it?

# remove last fully-connected layer
fcmodel = nn.Linear(2048, cls_num)

loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(fcmodel.parameters(), lr=1e-4, weight_decay=1e-4)

if is_cuda : model.cuda(), loss_fn.cuda()


I am working on the file "09_Flowers_tranfer_learning.ipynb".
The first task is:
3. Trainning loop
* (입력 생성)
* model 생성
* loss 생성
* zeroGrad
* backpropagation
* optimizer step (update model parameter)

```
Can you write Python code for it?

# trainning
fcmodel.train()
train_loss = []
train_accu = []
i = 0
for epoch in range(1000):
    for feature, target in feature_loader:
        feature, target = Variable(feature), Variable(target.squeeze()) # 입력image Target 설정
        if is_cuda :  feature, target = feature.cuda(), target.cuda()
        output = fcmodel(feature) # model 생성
        loss = loss_fn(output, target) #loss 생성
        optimizer.zero_grad() # zero_grad
        loss.backward() # calc backward grad
        optimizer.step() # update parameter

        pred = output.data.max(1)[1]
        accuracy = pred.eq(target.data).sum()/batch_size

        train_loss.append(loss.data[0])
        train_accu.append(accuracy)

        if i % 300 == 0:
            print(i, loss.data[0])
        i += 1

plt.plot(train_accu)

plt.plot(train_loss)

checkpoint_filename = 'flowerfeature_resnet152.ckpt'

## save a parameter
torch.save(fcmodel.state_dict(), checkpoint_filename)


I am working on the file "09_Flowers_tranfer_learning.ipynb".
The first task is:
4. Predict & Evaluate
이제 학습된 마지막단 fully connected layter를 기존의 모델에 연결하여, prediction을 수행하여 봅니다. batch size를 더 크게 할 수 있어서 그러진 8장의 예제보다 좀 더 정확한  93%정도 수준의 정확도를 보여줍니다.
Can you write Python code for it?

import torch
import torchvision
import torch.nn as nn
from torchvision import datasets, transforms
from torch.autograd import Variable

traindir = '/tmp/flower_photos'
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
cls_num = len(datasets.folder.find_classes(traindir)[0])

test_loader = torch.utils.data.DataLoader(
    datasets.ImageFolder(traindir,
                         transforms.Compose([
                             transforms.RandomSizedCrop(224),
                             transforms.RandomHorizontalFlip(),
                             transforms.ToTensor(),
                             normalize,])),
    shuffle=True)

model = torchvision.models.resnet152(pretrained = True)
# make new fully connected layer
fcmodel = nn.Linear(2048, cls_num)
# load saved parameter into fc layer
checkpoint_filename = 'flowerfeature_resnet152.ckpt'
checkpoint = torch.load(checkpoint_filename)
fcmodel.load_state_dict(checkpoint)
#connect fc layer to resnet152
model.fc = fcmodel

model.eval()
correct = 0

from itertools import islice
for image, target in islice(test_loader, 100):
    if is_cuda :  image, target = image.cuda(), target.cuda()
    image, target = Variable(image, volatile=True), Variable(target)
    output = model(image)
    prediction = output.data.max(1)[1]
    correct += prediction.eq(target.data).sum()

#print('\nTest set: Accuracy: {:.2f}%'.format(100. * correct / len(test_loader.dataset)))
print('\nTest set: Accuracy: {:.2f}%'.format(100. * correct / 100))


I am working on the file "09_PSV_Rayleigh_wave_modelling/1_2D_PSV_hom_block_model.ipynb".
The first task is:
Update stresses
Can you write Python code for it?

@jit(nopython=True) # use JIT for C-performance
def update_s(vx, vy, sxx, syy, sxy, nx, ny, dtdx, lam, mu):

    for j in range(1, ny-1):
        for i in range(1, nx-1):

            # calculate spatial derivatives
            vxx = vx[j][i] - vx[j][i-1]
            vyy = vy[j][i] - vy[j-1][i]
            vyx = vy[j][i+1] - vy[j][i]
            vxy = vx[j+1][i] - vx[j][i]

            # update stresses
            sxx[j, i] = sxx[j, i] + dtdx * ( lam * (vxx + vyy) + 2.0 * mu * vxx )
            syy[j, i] = syy[j, i] + dtdx * ( lam * (vxx + vyy) + 2.0 * mu * vyy )
            sxy[j, i] = sxy[j, i] + dtdx * (  mu * (vyx + vxy) )

    return sxx, syy, sxy


I am working on the file "09_Python_LaTeX.ipynb".
The first task is:
# Symbolic Mathematics (SymPy)
Can you write Python code for it?

sp.init_printing()     # Turns on pretty printing

np.sqrt(8)

sp.sqrt(8)


I am working on the file "09_Python_LaTeX.ipynb".
The first task is:
SymPy can do your calculus homework.
Can you write Python code for it?

yet_another_equation

sp.diff(yet_another_equation,x)

sp.diff(yet_another_equation,x,2)

sp.integrate(yet_another_equation,x)

sp.integrate(yet_another_equation,(x,0,5))   # limits x = 0 to 5


I am working on the file "09_Python_LaTeX.ipynb".
The first task is:
SymPy has all sorts of ways to manipulates symbolic equations
Can you write Python code for it?

sp.simplify(my_equation / x)

another_equation = (x + 2) * (x - 3)
another_equation

sp.expand(another_equation)

yet_another_equation = 2 * x**2 + 5 * x + 3

sp.factor(yet_another_equation)

sp.solve(yet_another_equation,x)

long_equation = 2*y*x**3 + 12*x**2 - x + 3 - 8*x**2 + 4*x + x**3 + 5 + 2*y*x**2 + x*y
long_equation

sp.collect(long_equation,x)

sp.collect(long_equation,y)


I am working on the file "09_Python_LaTeX.ipynb".
The first task is:
You have to explicitly tell SymPy what symbols you want to use
Can you write Python code for it?

x, y, z = sp.symbols('x y z')

my_equation = 2 * x + y
my_equation

my_equation + 3

my_equation - x

my_equation / x


I am working on the file "09_bqml/image_embeddings.ipynb".
The first task is:
Embedding model for images
We're going to use the EfficientNets model trained on ImageNet. It is compact and trained on a large variety of real-world images.
Can you write Python code for it?

import tensorflow as tf
import tensorflow_hub as tfhub
import os

model = tf.keras.Sequential()
model.add(tf.keras.Input(shape=[None,None,3]))
model.add(tfhub.KerasLayer("https://tfhub.dev/google/efficientnet/b4/feature-vector/1", name='image_embeddings'))
model.summary()


I am working on the file "09_bqml/image_embeddings.ipynb".
The first task is:
The model on TensorFlow Hub expects images of a certain size, and provided as normalized arrays.
So, we'll define a serving function that carries out the necessary reading and preprocessing of the images.
Can you write Python code for it?

@tf.function(input_signature=[tf.TensorSpec(shape=[None], dtype=tf.string)])
def serve(filename):
    img = tf.io.read_file(filename[0])
    img = tf.io.decode_image(img, channels=3)
    img = tf.cast(img, tf.float32) / 255.0
    #img = tf.image.resize(img, [380, 380])
    return model(img)

path='gs://{}/effnet_image_embedding'.format(BUCKET)
tf.saved_model.save(model, path, signatures={'serving_default': serve})

!saved_model_cli show --all --dir gs://$BUCKET/effnet_image_embedding


I am working on the file "09_bqml/text_embeddings.ipynb".
The first task is:
Document clustering
We can use the embeddings as input to a K-Means clustering model. To make things interesting, let's also include the day and location.
K-Means at present doesn't accept arrays as input, so I'm defining a function to make it a struct with named parameters.
Can you write Python code for it?

%%bigquery
CREATE TEMPORARY FUNCTION arr_to_input_20(arr ARRAY<FLOAT64>)
RETURNS
STRUCT<p1 FLOAT64, p2 FLOAT64, p3 FLOAT64, p4 FLOAT64,
       p5 FLOAT64, p6 FLOAT64, p7 FLOAT64, p8 FLOAT64,
       p9 FLOAT64, p10 FLOAT64, p11 FLOAT64, p12 FLOAT64,
       p13 FLOAT64, p14 FLOAT64, p15 FLOAT64, p16 FLOAT64,
       p17 FLOAT64, p18 FLOAT64, p19 FLOAT64, p20 FLOAT64>

AS (
STRUCT(
    arr[OFFSET(0)]
    , arr[OFFSET(1)]
    , arr[OFFSET(2)]
    , arr[OFFSET(3)]
    , arr[OFFSET(4)]
    , arr[OFFSET(5)]
    , arr[OFFSET(6)]
    , arr[OFFSET(7)]
    , arr[OFFSET(8)]
    , arr[OFFSET(9)]
    , arr[OFFSET(10)]
    , arr[OFFSET(11)]
    , arr[OFFSET(12)]
    , arr[OFFSET(13)]
    , arr[OFFSET(14)]
    , arr[OFFSET(15)]
    , arr[OFFSET(16)]
    , arr[OFFSET(17)]
    , arr[OFFSET(18)]
    , arr[OFFSET(19)]
));

CREATE OR REPLACE MODEL advdata.storm_reports_clustering
OPTIONS(model_type='kmeans', NUM_CLUSTERS=10) AS

SELECT
  arr_to_input_20(output_0) AS comments_embed,
  EXTRACT(DAYOFYEAR from timestamp) AS julian_day,
  longitude, latitude
FROM ML.PREDICT(MODEL advdata.swivel_text_embed,(
  SELECT timestamp, longitude, latitude, LOWER(comments) AS sentences
  FROM `bigquery-public-data.noaa_preliminary_severe_storms.wind_reports`
  WHERE EXTRACT(YEAR from timestamp) = 2019
))


I am working on the file "09_bqml/text_embeddings.ipynb".
The first task is:
Let's load the model into a BigQuery dataset named advdata (create it if necessary)
Can you write Python code for it?

%%bigquery
CREATE OR REPLACE MODEL advdata.swivel_text_embed
OPTIONS(model_type='tensorflow', model_path='gs://ai-analytics-solutions-kfpdemo/swivel/*')


I am working on the file "09_bqml/text_embeddings.ipynb".
The first task is:
Loading model into BigQuery
The Swivel model above is already available in SavedModel format. But we need it on Google Cloud Storage before we can load it into BigQuery.
Can you write Python code for it?

%%bash
BUCKET=ai-analytics-solutions-kfpdemo   # CHANGE AS NEEDED

rm -rf tmp
mkdir tmp
FILE=swivel.tar.gz
wget --quiet -O tmp/swivel.tar.gz  https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1?tf-hub-format=compressed
cd tmp
tar xvfz swivel.tar.gz
cd ..
mv tmp swivel
gsutil -m cp -R swivel gs://${BUCKET}/swivel
rm -rf swivel

echo "Model artifacts are now at gs://${BUCKET}/swivel/*"


I am working on the file "09_deploying/09a_inmemory.ipynb".
The first task is:
If we pass in a list of filenames, the entire list is sent to the model.
This is useful if we do a GroupBy for example
For efficiency, it can be helpful to generate a random key, group by it.
Can you write Python code for it?

with beam.Pipeline() as p:

    shared_handle = Shared()

    (p
     | 'input' >> beam.Create([
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9818247_e2eac18894.jpg',
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9853885425_4a82356f1d_m.jpg',
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/9299302012_958c70564c_n.jpg',
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8733586143_3139db6e9e_n.jpg',
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8713397358_0505cc0176_n.jpg'
     ])
     | 'addlabel' >> beam.Map(lambda filename: (filename.split('/')[5], filename))
     | 'groupbykey' >> beam.GroupByKey()  # (daisy, [daisyfiles])
     | 'addpred' >> beam.Map(lambda x: ModelPredict(shared_handle, MODEL_LOCATION)(x[1]))
     | 'write' >> beam.io.textio.WriteToText('/tmp/out.txt', num_shards=1)
    )

!cat /tmp/out.txt*

with beam.Pipeline() as p:

    shared_handle = Shared()

    (p
     | 'input' >> beam.Create([
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9818247_e2eac18894.jpg',
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9853885425_4a82356f1d_m.jpg',
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/9299302012_958c70564c_n.jpg',
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8733586143_3139db6e9e_n.jpg',
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8713397358_0505cc0176_n.jpg'
     ])
     | 'batch' >> beam.BatchElements(min_batch_size=2, max_batch_size=3)
     | 'addpred' >> beam.Map( ModelPredict(shared_handle, MODEL_LOCATION) )
     | 'write' >> beam.Map(print)
    )


I am working on the file "09_deploying/09a_inmemory.ipynb".
The first task is:
In memory model (Beam pipeline)
Make sure to share the model so that it is not loaded for each element.
There will be a different model on different worker machines.
Can you write Python code for it?

import apache_beam as beam
from apache_beam.utils.shared import Shared
import tensorflow as tf

class ModelPredict:
    def __init__(self, shared_handle, model_location):
        self._shared_handle = shared_handle
        self._model_location = model_location

    def __call__(self, filenames):
        def initialize_model():
            print('Loading Keras model from ' + self._model_location)
            return (tf.keras.models.load_model(self._model_location)
                    .signatures['serving_default'])

        serving_fn = self._shared_handle.acquire(initialize_model)
        if isinstance(filenames, str):
            # only one element, put it into a batch of 1
            result = serving_fn(tf.convert_to_tensor([filenames]))
        else:
            # a list
            result = serving_fn(tf.convert_to_tensor(filenames))
        return {
            'filenames': filenames,
            'probability': result['probability'].numpy(),
            'pred_label': result['flower_type_str'].numpy()
        }

with beam.Pipeline() as p:

    shared_handle = Shared()

    (p
     | 'input' >> beam.Create([
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9818247_e2eac18894.jpg',
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9853885425_4a82356f1d_m.jpg',
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/9299302012_958c70564c_n.jpg',
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8733586143_3139db6e9e_n.jpg',
        'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8713397358_0505cc0176_n.jpg'
     ])
     | 'pred' >> beam.Map(ModelPredict(shared_handle, MODEL_LOCATION))
     | 'write' >> beam.io.textio.WriteToText('/tmp/out.txt', num_shards=1)
    )

!cat /tmp/out.txt*


I am working on the file "09_deploying/09a_inmemory.ipynb".
The first task is:
In-memory model (Python program)
Can you write Python code for it?

import tensorflow as tf
serving_fn = tf.keras.models.load_model(MODEL_LOCATION).signatures['serving_default']
filenames = [
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9818247_e2eac18894.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9853885425_4a82356f1d_m.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/9299302012_958c70564c_n.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8733586143_3139db6e9e_n.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8713397358_0505cc0176_n.jpg'
]
pred = serving_fn(tf.convert_to_tensor(filenames))
print(pred)
print('******')
print(pred['flower_type_str'].numpy())

import matplotlib.pyplot as plt
f, ax = plt.subplots(1, 5, figsize=(15,15))
for idx, (filename, prob, pred_label) in enumerate(
        zip(filenames, pred['probability'].numpy(), pred['flower_type_str'].numpy())):

    img = tf.io.read_file(filename)
    img = tf.image.decode_jpeg(img, channels=3)
    ax[idx].imshow((img.numpy()));

    ax[idx].set_title('{} ({:.2f})'.format(pred_label.decode('utf-8'), prob))
    ax[idx].axis('off')


I am working on the file "09_deploying/09b_rest.ipynb".
The first task is:
IMPORTANT: CHANGE THIS CELL
Note the endpoint ID and deployed model ID above. Set it in the cell below.
Can you write Python code for it?

# CHANGE THESE TO REFLECT WHERE YOU DEPLOYED THE MODEL
import os
os.environ['ENDPOINT_ID'] = '4327589805996113920' # CHANGE
os.environ['MODEL_ID'] = '1963683786742824960' # CHANGE
os.environ['PROJECT'] = 'ai-analytics-solutions' # CHANGE
os.environ['BUCKET'] = 'ai-analytics-solutions-mlvisionbook' # CHANGE
os.environ['REGION'] = 'us-central1' # CHANGE


I am working on the file "09_deploying/09b_rest.ipynb".
The first task is:
Sending over HTTP Post
Can you write Python code for it?

# Invoke from Python.
import json
from oauth2client.client import GoogleCredentials
import requests

PROJECT = os.environ['PROJECT']
REGION = os.environ['REGION']
ENDPOINT_ID = os.environ['ENDPOINT_ID']

token = GoogleCredentials.get_application_default().get_access_token().access_token
api = "https://{}-aiplatform.googleapis.com/v1/projects/{}/locations/{}/endpoints/{}:predict".format(
    REGION, PROJECT, REGION, ENDPOINT_ID)
headers = {"Authorization": "Bearer " + token }
data = {
    "instances": [
        {
            "filenames": "gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9818247_e2eac18894.jpg"
        },
        {
            "filenames": "gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9853885425_4a82356f1d_m.jpg"
        },
        {
            "filenames": "gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/9299302012_958c70564c_n.jpg"
        },
        {
            "filenames": "gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8733586143_3139db6e9e_n.jpg"
        },
        {
            "filenames": "gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8713397358_0505cc0176_n.jpg"
        }
    ]
}
response = requests.post(api, json=data, headers=headers)
print(response.content)


I am working on the file "09_deploying/09b_rest.ipynb".
The first task is:
[Optional] CAIP Batch prediction
Can you write Python code for it?

%%writefile batchinputs.jsonl
{"filenames": "gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9818247_e2eac18894.jpg"}
{"filenames": "gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9853885425_4a82356f1d_m.jpg"}
{"filenames": "gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/9299302012_958c70564c_n.jpg"}
{"filenames": "gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8733586143_3139db6e9e_n.jpg"}
{"filenames": "gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8713397358_0505cc0176_n.jpg"}

%%bash
gsutil cp batchinputs.jsonl gs://BUCKET

# Invoke from Python.
import json
from oauth2client.client import GoogleCredentials
import requests

PROJECT = os.environ['PROJECT']
REGION = os.environ['REGION']
ENDPOINT_ID = os.environ['ENDPOINT_ID']
MODEL_ID = os.environ['MODEL_ID']
BUCKET = os.environ['BUCKET'] # used for staging

BATCH_JOB_NAME = "batch_pred_job"
INPUT_FORMAT = "jsonl"
INPUT_URI = "gs://{}/batchinputs.jsonl".format(BUCKET)
OUTPUT_DIRECTORY = "gs://{}/batch_predictions".format(BUCKET)
MACHINE_TYPE = "n1-standard-2"
STARTING_REPLICA_COUNT = 1
BATCH_SIZE = 64

token = GoogleCredentials.get_application_default().get_access_token().access_token
api = "https://{}-aiplatform.googleapis.com/v1/projects/{}/locations/{}/batchPredictionJobs".format(
    REGION, PROJECT, REGION
)
headers = {"Authorization": "Bearer " + token}
data = {
    "displayName": BATCH_JOB_NAME,
    "model": "projects/{}/locations/{}/models/{}".format(
        PROJECT, REGION, MODEL_ID
    ),
    "inputConfig": {
        "instancesFormat": INPUT_FORMAT,
        "gcsSource": {
            "uris": [INPUT_URI],
        },
    },
    "outputConfig": {
        "predictionsFormat": "jsonl",
        "gcsDestination": {
            "outputUriPrefix": OUTPUT_DIRECTORY,
        },
    },
    "dedicatedResources" : {
        "machineSpec" : {
            "machineType": MACHINE_TYPE
        },
        "startingReplicaCount": STARTING_REPLICA_COUNT
    },
   ...


I am working on the file "09_deploying/09c_changesig.ipynb".
The first task is:
Changing signatures of exported model
In this notebook, we start from an already trained and saved model (as in Chapter 7).
For convenience, we have put this model in a public bucket in gs://practical-ml-vision-book/flowers_5_trained
Enable GPU and set up helper functions
This notebook and pretty much every other notebook in this repository
will run faster if you are using a GPU.
On Colab:
- Navigate to Edit→Notebook Settings
- Select GPU from the Hardware Accelerator drop-down
On Cloud AI Platform Notebooks:
- Navigate to https://console.cloud.google.com/ai-platform/notebooks
- Create an instance with a GPU or select your instance and add a GPU
Next, we'll confirm that we can connect to the GPU with tensorflow:
Can you write Python code for it?

import tensorflow as tf
print('TensorFlow version' + tf.version.VERSION)
print('Built with GPU support? ' + ('Yes!' if tf.test.is_built_with_cuda() else 'Noooo!'))
print('There are {} GPUs'.format(len(tf.config.experimental.list_physical_devices("GPU"))))
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
    raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))


I am working on the file "09_deploying/09c_changesig.ipynb".
The first task is:
Exported model
We start from a trained and saved model from Chapter 7.
<pre>
  model.save(...)
</pre>
Can you write Python code for it?

MODEL_LOCATION='gs://practical-ml-vision-book/flowers_5_trained'

!gsutil ls {MODEL_LOCATION}

!saved_model_cli show --tag_set serve --signature_def serving_default --dir {MODEL_LOCATION}


I am working on the file "09_deploying/09c_changesig.ipynb".
The first task is:
Multiple signatures
Can you write Python code for it?

import os
import shutil
import tensorflow as tf
model = tf.keras.models.load_model(MODEL_LOCATION)
old_fn = model.signatures['serving_default']

@tf.function(input_signature=[tf.TensorSpec([None,], dtype=tf.string)])
def pass_through_input(filenames):
    result = old_fn(filenames) # has flower_type_int etc.
    result['filename'] = filenames
    return result

shutil.rmtree('export', ignore_errors=True)
os.mkdir('export')
model.save('export/flowers_model2',
          signatures={
              'serving_default': old_fn,
              'input_pass_through': pass_through_input
          })

!saved_model_cli show --tag_set serve --dir export/flowers_model2

!saved_model_cli show --tag_set serve --dir export/flowers_model2 --signature_def serving_default

!saved_model_cli show --tag_set serve --dir export/flowers_model2 --signature_def input_pass_through


I am working on the file "09_deploying/09d_bytes.ipynb".
The first task is:
Export signature that will handle bytes from client
Can you write Python code for it?

@tf.function(input_signature=[tf.TensorSpec([None,], dtype=tf.string)])
def predict_bytes(img_bytes):
    input_images = tf.map_fn(
        preprocess,
        img_bytes,
        fn_output_signature=tf.float32
    )
    batch_pred = model(input_images) # same as model.predict()
    top_prob = tf.math.reduce_max(batch_pred, axis=[1])
    pred_label_index = tf.math.argmax(batch_pred, axis=1)
    pred_label = tf.gather(tf.convert_to_tensor(CLASS_NAMES), pred_label_index)
    return {
        'probability': top_prob,
        'flower_type_int': pred_label_index,
        'flower_type_str': pred_label
    }

@tf.function(input_signature=[tf.TensorSpec([None,], dtype=tf.string)])
def predict_filename(filenames):
    img_bytes = tf.map_fn(
        tf.io.read_file,
        filenames
    )
    result = predict_bytes(img_bytes)
    result['filename'] = filenames
    return result

shutil.rmtree('export', ignore_errors=True)
os.mkdir('export')
model.save('export/flowers_model3',
          signatures={
              'serving_default': predict_filename,
              'from_bytes': predict_bytes
          })

!saved_model_cli show --tag_set serve --dir export/flowers_model3

!saved_model_cli show --tag_set serve --dir export/flowers_model3 --signature_def serving_default

!saved_model_cli show --tag_set serve --dir export/flowers_model3 --signature_def from_bytes


I am working on the file "09_deploying/09d_bytes.ipynb".
The first task is:
Note how we pass the base-64 encoded data
Can you write Python code for it?

# Invoke from Python.
import base64
import json
from oauth2client.client import GoogleCredentials
import requests

PROJECT = "ai-analytics-solutions"  # CHANGE
REGION = "us-central1"  # make sure you have GPU/TPU quota in this region
ENDPOINT_ID = "7318683646011899904"

def b64encode(filename):
    with open(filename, 'rb') as ifp:
        img_bytes = ifp.read()
        return base64.b64encode(img_bytes)

token = GoogleCredentials.get_application_default().get_access_token().access_token
api = "https://{}-aiplatform.googleapis.com/v1/projects/{}/locations/{}/endpoints/{}:predict".format(
    REGION, PROJECT, REGION, ENDPOINT_ID)
headers = {"Authorization": "Bearer " + token }
data = {
    "signature_name": "from_bytes",  # currently bugged
    "instances": [
        {
            "img_bytes": {"b64": b64encode('/tmp/test1.jpg')}
        },
        {
            "img_bytes": {"b64": b64encode('/tmp/test2.jpg')}
        },
    ]
}
response = requests.post(api, json=data, headers=headers)
print(response.content)


I am working on the file "09_deploying/09d_bytes.ipynb".
The first task is:
Read from checkpoints.
We start from the checkpoints not the saved model because we want the full model
not just the signatures.
Can you write Python code for it?

import os
import shutil
import tensorflow as tf

CHECK_POINT_DIR='gs://practical-ml-vision-book/flowers_5_trained/chkpts'
model = tf.keras.models.load_model(CHECK_POINT_DIR)
print(model.summary())

IMG_HEIGHT = 345
IMG_WIDTH = 345
IMG_CHANNELS = 3
CLASS_NAMES = 'daisy dandelion roses sunflowers tulips'.split()

def read_from_jpegfile(filename):
    img_bytes = tf.io.read_file(filename)
    return img_bytes

def preprocess(img_bytes):
    img = tf.image.decode_jpeg(img_bytes, channels=IMG_CHANNELS)
    img = tf.image.convert_image_dtype(img, tf.float32)
    return tf.image.resize_with_pad(img, IMG_HEIGHT, IMG_WIDTH)

filenames = [
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9818247_e2eac18894.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/dandelion/9853885425_4a82356f1d_m.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/9299302012_958c70564c_n.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8733586143_3139db6e9e_n.jpg',
    'gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/tulips/8713397358_0505cc0176_n.jpg'
]
for filename in filenames:
    img_bytes = read_from_jpegfile(filename)
    img = preprocess(img_bytes)
    img = tf.expand_dims(img, axis=0)
    pred = model.predict(img)
    print(pred)


I am working on the file "09_deploying/09d_bytes.ipynb".
The first task is:
Send img bytes over the wire
No need for intermediate file on GCS. Note that we are simply using Python's file reading method.
Can you write Python code for it?

!gsutil cp gs://practical-ml-vision-book/flowers_5_jpeg/flower_photos/daisy/9299302012_958c70564c_n.jpg /tmp/test.jpg

with open('/tmp/test.jpg', 'rb') as ifp:
    img_bytes = ifp.read()
    serving_fn = tf.keras.models.load_model('./export/flowers_model3').signatures['from_bytes']
    pred = serving_fn(tf.convert_to_tensor([img_bytes]))
    print(pred)


I am working on the file "09_deploying/09e_tflite.ipynb".
The first task is:
Convert to TFLite
This will take a while to do the conversion
Can you write Python code for it?

import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(MODEL_LOCATION)
tflite_model = converter.convert()

with open('export/model.tflite', 'wb') as ofp:
    ofp.write(tflite_model)

!ls -lh export/model.tflite


I am working on the file "09_discontinuos_galerkin/dg_hetero_1d.ipynb".
The first task is:
1. Initialization of setup
Can you write Python code for it?

# Initialization of setup
# --------------------------------------------------------------------------
tmax   = 2.5          # Length of seismogram [s]
xmax   = 10000        # Length of domain [m]
vs0    = 2500         # Advection velocity
rho0   = 2500         # Density [kg/m^3]
mu0    = rho0*vs0**2  # shear modulus
N      = 2            # Order of Lagrange polynomials
ne     = 200          # Number of elements
sig    = 100          # width of Gaussian initial condition
x0     = 4000         # x locartion of Gauss
eps    = 0.2          # Courant criterion
iplot  = 20           # Plotting frequency
imethod = 'RK'        # 'Euler', 'RK'

nx = ne*N + 1
dx = xmax/(nx-1)      # space increment
#--------------------------------------------------------------------

# Initialization of GLL points integration weights
[xi,w] = gll(N)     # xi, N+1 coordinates [-1 1] of GLL points
                    # w Integration weights at GLL locations
# Space domain
le = xmax/ne       # Length of elements, here equidistent
ng = ne*N + 1

# Vector with GLL points
k = 0
xg = np.zeros((N+1)*ne)
for i in range(0, ne):
    for j in range(0, N+1):
        k += 1
        xg[k-1] = i*le + .5*(xi[j] + 1)*le

x = np.reshape(xg, (N+1, ne), order='F').T

# Calculation of time step acoording to Courant criterion
dxmin = np.min(np.diff(xg[1:N+1]))
dt = eps*dxmin/vs0 # Global time step
nt = int(np.floor(tmax/dt))

# Mapping - Jacobian
J = le/2  # Jacobian
Ji = 1/J  # Inverse Jacobian

# 1st derivative of Lagrange polynomials
l1d = lagrange1st(N)


I am working on the file "09_discontinuos_galerkin/dg_hetero_1d.ipynb".
The first task is:
3. Flux Matrices
The main difference in the heterogeneous case with respect the homogeneous one is found in the definition of fluxes. As in the case of finite volumes when we solve the 1D elastic wave equation, we allow the coefficients of matrix A to vary inside the element.
\begin{equation}
\mathbf{A}=
     \begin{pmatrix}
       0 & -\mu_i \
       -1/\rho_i & 0
     \end{pmatrix}
\end{equation}
Now we need to diagonalize $\mathbf{A}$. Introducing the seismic impedance $Z_i = \rho_i c_i$, we have
\begin{equation}
\mathbf{A} = \mathbf{R}^{-1}\mathbf{\Lambda}\mathbf{R}
\qquad\text{,}\qquad
\mathbf{\Lambda}=
     \begin{pmatrix}
       -c_i & 0 \
        0 & c_i
     \end{pmatrix}
\qquad\text{,}\qquad
\mathbf{R} =
     \begin{pmatrix}
       Z_i & -Z_i \
       1 &  1
     \end{pmatrix}
\qquad\text{and}\qquad
\mathbf{R}^{-1} = \frac{1}{2Z_i}
     \begin{pmatrix}
       1 & Z_i \
      -1 & Z_i
     \end{pmatrix}
\end{equation}
We decompose the solution into right propagating $\mathbf{\Lambda}^{+}$ and left propagating eigenvalues $\mathbf{\Lambda}^{-}$ where
\begin{equation}
\mathbf{\Lambda}^{+}=
     \begin{pmatrix}
       -c_i & 0 \
        0 & 0
     \end{pmatrix}
\qquad\text{,}\qquad
\mathbf{\Lambda}^{-}=
     \begin{pmatrix}
        0 & 0 \
        0 & c_i
     \end{pmatrix}
\qquad\text{and}\qquad
\mathbf{A}^{\pm} = \mathbf{R}^{-1}\mathbf{\Lambda}^{\pm}\mathbf{R}
\end{equation}
This strategy allows us to formulate the Flux term in the discontinuous Galerkin method. The following cell initializes all flux related matrices
Can you write Python code for it?

# Inialize Flux relates matrices
# ---------------------------------------------------------------

# initialize heterogeneous A
Ap  = np.zeros((ne,2,2))
Am  = np.zeros((ne,2,2))
Z   = np.zeros(ne)
rho = np.zeros(ne)
mu  = np.zeros(ne)

# initialize c, rho, mu, and Z
rho = rho + rho0
rho[int(ne/2):ne] = .25 * rho[int(ne/2):ne] # Introduce discontinuity
mu = mu + mu0
c = np.sqrt(mu/rho)
Z = rho * c

# Initialize flux matrices
for i in range(1,ne-1):
    # Left side positive direction
    R = np.array([[Z[i], -Z[i]], [1, 1]])
    Lp = np.array([[0, 0], [0, c[i]]])
    Ap[i,:,:] = R @ Lp @ np.linalg.inv(R)

    # Right side negative direction
    R = np.array([[Z[i], -Z[i]], [1, 1]])
    Lm = np.array([[-c[i], 0 ], [0, 0]])
    Am[i,:,:] = R @ Lm @ np.linalg.inv(R)


I am working on the file "09_discontinuos_galerkin/dg_homo_1d.ipynb".
The first task is:
1. Initialization of setup
Can you write Python code for it?

# Initialization of setup
# --------------------------------------------------------------------------
c      = 2500          # acoustic velocity [m/s]
tmax   = 2.0           # Length of seismogram [s]
xmax   = 10000         # Length of domain [m]
vs     = 2500          # Advection velocity
rho    = 2500          # Density [kg/m^3]
mu     = rho*vs**2     # shear modulus
N      = 4             # Order of Lagrange polynomials
ne     = 200           # Number of elements
sig    = 200           # Gaussian width
x0     = 5000          # x location of Gaussian
eps    = 0.4           # Courant criterion
iplot  = 20            # Plotting frequency
imethod = 'RK'      # 'Euler', 'RK'
#--------------------------------------------------------------------

# GLL points and integration weights
[xi,w] = gll(N)     # xi, N+1 coordinates [-1 1] of GLL points
                    # w Integration weights at GLL locations
# Space domain
le = xmax/ne        # Length of elements
ng = ne*N + 1

# Vector with GLL points
k=0
xg = np.zeros((N+1)*ne)
for i in range(0, ne):
    for j in range(0, N+1):
        k += 1
        xg[k-1] = i*le + .5*(xi[j]+1)*le

x = np.reshape(xg, (N+1, ne), order='F').T

# Calculation of time step acoording to Courant criterion
dxmin = np.min(np.diff(xg[1:N+1]))
dt = eps*dxmin/vs    # Global time step
nt = int(np.floor(tmax/dt))

# Mapping - Jacobian
J  = le/2 # Jacobian
Ji = 1/J  # Inverse Jacobian

# 1st derivative of Lagrange polynomials
l1d = lagrange1st(N)


I am working on the file "09_discontinuos_galerkin/dg_homo_1d.ipynb".
The first task is:
2. Elemental Mass and Stiffness matrices
The mass and the stiffness matrix are calculated prior time extrapolation, so they are pre-calculated and stored at the beginning of the code.
The integrals defined in the mass and stiffness matrices are computed using a numerical quadrature, in this cases the GLL quadrature that uses the GLL points and their corresponding weights to approximate the integrals. Hence,
\begin{equation}
M_{ij}^k=\int_{-1}^1 \ell_i^k(\xi)  \ell_j^k(\xi) \ J \ d\xi = \sum_{m=1}^{N_p} w_m \ \ell_i^k (x_m)  \ell_j^k(x_m)\ J =\sum_{m=1}^{N_p} w_m \delta_{im}\ \delta_{jm} \ J= \begin{cases} w_i \ J \ \ \text{ if } i=j \ 0 \ \ \ \ \ \ \ \text{   if } i \neq j\end{cases}
\end{equation}
that is a diagonal mass matrix!. Subsequently, the stiffness matrices is given as
\begin{equation}
K_{i,j}= \int_{-1}^1 \ell_i^k(\xi) \cdot \partial x  \ell_j^k(\xi) \ d\xi= \sum{m=1}^{N_p} w_m \ \ell_i^k(x_m)\cdot \partial_x \ell_j^k(x_m)= \sum_{m=1}^{N_p} w_m \delta_{im}\cdot \partial_x\ell_j^k(x_m)= w_i \cdot \partial_x \ell_j^k(x_i)
\end{equation}
The Lagrange polynomials and their properties have been already used, they determine the integration weights $w_i$ that are returned by the python method "gll". Additionally, the fist derivatives of such basis, $\partial_x \ell_j^k(x_i)$, are needed, the python method "Lagrange1st" returns them.
Exercise 1
Now we have all the ingredients to calculate the mass and stiffness matrix, initialize these matrices in the following cell. Compute the inverse mass matrix, keep in mind that it is diagonal.
Can you write Python code for it?


I am working on the file "09_discontinuos_galerkin/dg_homo_1d.ipynb".
The first task is:
4. Discontinuous Galerkin Solution
The principal characteristic of the discontinuous Galerkin Method is the communication between the element neighbors using a flux term, in general it is given
\begin{equation}
\mathbf{Flux} = \int_{\partial D_k} \mathbf{A}\mathbf{Q}\ell_j(\xi)\mathbf{n}d\xi
\end{equation}
this term leads to four flux contributions for left and right sides of the elements
\begin{equation}
\mathbf{Flux} = -\mathbf{A}{k}^{-}\mathbf{Q}{l}^{k}\mathbf{F}^{l} + \mathbf{A}{k}^{+}\mathbf{Q}{r}^{k}\mathbf{F}^{r} - \mathbf{A}{k}^{+}\mathbf{Q}{r}^{k-1}\mathbf{F}^{l} + \mathbf{A}{k}^{-}\mathbf{Q}{l}^{k+1}\mathbf{F}^{r}
\end{equation}
Last but not least, we have to solve our semi-discrete scheme that we derived above using an appropriate time extrapolation, in the code below we implemented two different time extrapolation schemes:
1) Euler scheme
\begin{equation}
\mathbf{Q}^{t+1} \approx \mathbf{Q}^{t} + dt\mathbf{M}^{-1}(\mathbf{A}\mathbf{K}\mathbf{Q} - \mathbf{Flux})
\end{equation}
2) Second-order Runge-Kutta method (also called predictor-corrector scheme)
\begin{eqnarray}
k_1 &=& f(t_i, y_i) \
k_2 &=& f(t_i + dt, y_i + dt k_1) \
& & \
y_{i+1} &=& y_i + \frac{dt}{2} (k_1 + k_2)
\end{eqnarray}
with $f$ that corresponds with $\mathbf{M}^{-1}(\mathbf{A}\mathbf{K}\mathbf{Q} - \mathbf{Flux})$
Exercise 3
In order to validate our numerical solution, we ask to compare with the corresponding analytical solution as we did in the finite volumes implementation. Implement the analytical solution for the 1D elastic wave equation in homogeneous media and plot it together with the discontinuous Galerkin numerical solution.
Can you write Python code for it?

# DG Solution, Time extrapolation
# ---------------------------------------------------------------

# Initalize solution vectors
Q    = np.zeros([ne, N+1, 2])
Qnew = np.zeros([ne, N+1, 2])

k1 = np.zeros([ne, N+1, 2])
k2 = np.zeros([ne, N+1, 2])

Q[:,:,0] = np.exp(-1/sig**2*((x-x0))**2)
Qs = np.zeros(xg.size)  # for plotting
Qv = np.zeros(xg.size)  # for plotting
Qa = np.zeros((2, xg.size))  # for analytical solution

# Initialize animated plot
# ---------------------------------------------------------------
fig = plt.figure(figsize=(10,6))
ax1 = fig.add_subplot(2,1,1)
ax2 = fig.add_subplot(2,1,2)
line1 = ax1.plot(xg, Qs, 'k', xg, Qa[0,:], 'r--', lw=1.5)
line2 = ax2.plot(xg, Qv, 'k', xg, Qa[1,:], 'r--', lw=1.5)
ax1.set_ylabel('Stress')
ax2.set_ylabel('Velocity')
ax2.set_xlabel(' x ')
plt.suptitle('Homogeneous Disc. Galerkin  - %s method'%imethod, size=16)

plt.ion()   # set interective mode
plt.show()

# ---------------------------------------------------------------
# Time extrapolation
# ---------------------------------------------------------------
for it in range(nt):
    if imethod == 'Euler': # Euler
        # Calculate Fluxes
        Flux = flux(Q, N, ne, Ap, Am)
        # Extrapolate each element using flux F
        for i in range(1,ne-1):
            Qnew[i,:,0] = dt * Minv @ (-mu * K @ Q[i,:,1].T - Flux[i,:,0].T) + Q[i,:,0].T
            Qnew[i,:,1] = dt * Minv @ (-1/rho * K @ Q[i,:,0].T - Flux[i,:,1].T) + Q[i,:,1].T

    elif imethod == 'RK':
        # Calculate Fluxes
        Flux = flux(Q, N, ne, Ap, Am)
        # Extrapolate each element using flux F
        for i in range(1,ne-1):
            k1[i,:,0] = Minv @ (-mu * K @ Q[i,:,1].T - Flux[i,:,0].T)
            k1[i,:,1] = Minv @ (-1/rho * K @ Q[i,:,0].T - Flux[i,:,1].T)

        for i in range(1,ne-1):
            Qnew[i,:,0] = dt * Minv @ (-mu * K @ Q[i,:,1].T - Flux[i,:,0].T) + Q[i,:,0].T
            Qnew[i,:,1] = dt * Minv @ (-1/rho * K @ Q[i,:,0].T - Flux[i,:,1].T) +...


I am working on the file "09_discontinuos_galerkin/dg_homo_1d_solution.ipynb".
The first task is:
1. Initialization of setup
Can you write Python code for it?


I am working on the file "09_discontinuos_galerkin/dg_homo_1d_solution.ipynb".
The first task is:
2. Elemental Mass and Stiffness matrices
The mass and the stiffness matrix are calculated prior time extrapolation, so they are pre-calculated and stored at the beginning of the code.
The integrals defined in the mass and stiffness matrices are computed using a numerical quadrature, in this cases the GLL quadrature that uses the GLL points and their corresponding weights to approximate the integrals. Hence,
\begin{equation}
M_{ij}^k=\int_{-1}^1 \ell_i^k(\xi)  \ell_j^k(\xi) \ J \ d\xi = \sum_{m=1}^{N_p} w_m \ \ell_i^k (x_m)  \ell_j^k(x_m)\ J =\sum_{m=1}^{N_p} w_m \delta_{im}\ \delta_{jm} \ J= \begin{cases} w_i \ J \ \ \text{ if } i=j \ 0 \ \ \ \ \ \ \ \text{   if } i \neq j\end{cases}
\end{equation}
that is a diagonal mass matrix!. Subsequently, the stiffness matrices is given as
\begin{equation}
K_{i,j}= \int_{-1}^1 \ell_i^k(\xi) \cdot \partial x  \ell_j^k(\xi) \ d\xi= \sum{m=1}^{N_p} w_m \ \ell_i^k(x_m)\cdot \partial_x \ell_j^k(x_m)= \sum_{m=1}^{N_p} w_m \delta_{im}\cdot \partial_x\ell_j^k(x_m)= w_i \cdot \partial_x \ell_j^k(x_i)
\end{equation}
The Lagrange polynomials and their properties have been already used, they determine the integration weights $w_i$ that are returned by the python method "gll". Additionally, the fist derivatives of such basis, $\partial_x \ell_j^k(x_i)$, are needed, the python method "Lagrange1st" returns them.
Exercise 1
Now we have all the ingredients to calculate the mass and stiffness matrix, initialize these matrices in the following cell. Compute the inverse mass matrix, keep in mind that it is diagonal.
Can you write Python code for it?

# Initialization of system matrices
# -----------------------------------------------------------------
# Elemental Mass matrix
M = np.zeros((N+1, N+1))
for i in range(0, N+1):
    M[i, i] = w[i] * J

# Inverse matrix of M (M is diagonal!)
Minv = np.identity(N+1)
for i in range(0, N+1):
    Minv[i,i] = 1. / M[i,i]

# Elemental Stiffness Matrix
K = np.zeros((N+1, N+1))
for i in range(0, N+1):
    for j in range(0, N+1):
            K[i,j] = w[j] * l1d[i,j] # NxN matrix for every element


I am working on the file "09_discontinuos_galerkin/dg_homo_1d_solution.ipynb".
The first task is:
4. Discontinuous Galerkin Solution
The principal characteristic of the discontinuous Galerkin Method is the communication between the element neighbors using a flux term, in general it is given
\begin{equation}
\mathbf{Flux} = \int_{\partial D_k} \mathbf{A}\mathbf{Q}\ell_j(\xi)\mathbf{n}d\xi
\end{equation}
this term leads to four flux contributions for left and right sides of the elements
\begin{equation}
\mathbf{Flux} = -\mathbf{A}{k}^{-}\mathbf{Q}{l}^{k}\mathbf{F}^{l} + \mathbf{A}{k}^{+}\mathbf{Q}{r}^{k}\mathbf{F}^{r} - \mathbf{A}{k}^{+}\mathbf{Q}{r}^{k-1}\mathbf{F}^{l} + \mathbf{A}{k}^{-}\mathbf{Q}{l}^{k+1}\mathbf{F}^{r}
\end{equation}
Last but not least, we have to solve our semi-discrete scheme that we derived above using an appropriate time extrapolation, in the code below we implemented two different time extrapolation schemes:
1) Euler scheme
\begin{equation}
\mathbf{Q}^{t+1} \approx \mathbf{Q}^{t} + dt\mathbf{M}^{-1}(\mathbf{A}\mathbf{K}\mathbf{Q} - \mathbf{Flux})
\end{equation}
2) Second-order Runge-Kutta method (also called predictor-corrector scheme)
\begin{eqnarray}
k_1 &=& f(t_i, y_i) \
k_2 &=& f(t_i + dt, y_i + dt k_1) \
& & \
y_{i+1} &=& y_i + \frac{dt}{2} (k_1 + k_2)
\end{eqnarray}
with $f$ that corresponds with $\mathbf{M}^{-1}(\mathbf{A}\mathbf{K}\mathbf{Q} - \mathbf{Flux})$
Exercise 3
In order to validate our numerical solution, we ask to compare with the corresponding analytical solution as we did in the finite volumes implementation. Implement the analytical solution for the 1D elastic wave equation in homogeneous media and plot it together with the discontinuous Galerkin numerical solution.
Can you write Python code for it?

# DG Solution, Time extrapolation
# ---------------------------------------------------------------

# Initalize solution vectors
Q    = np.zeros([ne, N+1, 2])
#Qa   = np.zeros([ne, N+1, 2])
Qnew = np.zeros([ne, N+1, 2])

k1 = np.zeros([ne, N+1, 2])
k2 = np.zeros([ne, N+1, 2])

Q[:,:,0] = np.exp(-1/sig**2*((x-x0))**2)
Qs = np.zeros(xg.size)  # for plotting
Qv = np.zeros(xg.size)  # for plotting
Qa = np.zeros((2, xg.size))  # for analytical solution

# Initialize animated plot
# ---------------------------------------------------------------
fig = plt.figure(figsize=(10,6))
ax1 = fig.add_subplot(2,1,1)
ax2 = fig.add_subplot(2,1,2)
line1 = ax1.plot(xg, Qs, 'k', xg, Qa[0,:], 'r--', lw=1.5)
line2 = ax2.plot(xg, Qv, 'k', xg, Qa[1,:], 'r--', lw=1.5)
ax1.set_ylabel('Stress')
ax2.set_ylabel('Velocity')
ax2.set_xlabel(' x ')
plt.suptitle('Homogeneous Disc. Galerkin  - %s method'%imethod, size=16)

plt.ion()   # set interective mode
plt.show()

# ---------------------------------------------------------------
# Time extrapolation
# ---------------------------------------------------------------
for it in range(nt):
    if imethod == 'Euler': # Euler
        # Calculate Fluxes
        Flux = flux(Q, N, ne, Ap, Am)
        # Extrapolate each element using flux F
        for i in range(1,ne-1):
            Qnew[i,:,0] = dt * Minv @ (-mu * K @ Q[i,:,1].T - Flux[i,:,0].T) + Q[i,:,0].T
            Qnew[i,:,1] = dt * Minv @ (-1/rho * K @ Q[i,:,0].T - Flux[i,:,1].T) + Q[i,:,1].T

    elif imethod == 'RK':
        # Calculate Fluxes
        Flux = flux(Q, N, ne, Ap, Am)
        # Extrapolate each element using flux F
        for i in range(1,ne-1):
            k1[i,:,0] = Minv @ (-mu * K @ Q[i,:,1].T - Flux[i,:,0].T)
            k1[i,:,1] = Minv @ (-1/rho * K @ Q[i,:,0].T - Flux[i,:,1].T)

        for i in range(1,ne-1):
            Qnew[i,:,0] = dt * Minv @ (-mu * K @ Q[i,:,1].T - Flux[i,:,0].T) + Q[i,:,0].T
            Qnew[i,:,1] = dt * Minv @ (-1/rho * K @...


I am working on the file "09_dx_calibration.ipynb".
The first task is:
And the complete results visualized.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
plot_data = options_data[options_data.IMP_VOL > 0]
plt.figure(figsize=(10, 6))
for maturity in sorted(set(options_data['MATURITY'])):
    data = plot_data.isin({'MATURITY': [maturity,]})
    data = plot_data[plot_data.MATURITY == maturity]
      # select data for this maturity
    plt.plot(data['STRIKE'], data['IMP_VOL'],
             label=maturity.date(), lw=1.5)
    plt.plot(data['STRIKE'], data['IMP_VOL'], 'r.')
plt.xlabel('strike')
plt.ylabel('implied volatility of volatility')
plt.legend()
plt.show()


I am working on the file "09_dx_calibration.ipynb".
The first task is:
As a helper function we need a function to calculate all relevant third Fridays for all relevant maturity months of the data sets.
Can you write Python code for it?

import datetime as dt
import calendar

def third_friday(date):
    day = 21 - (calendar.weekday(date.year, date.month, 1) + 2) % 7
    return dt.datetime(date.year, date.month, day)

third_fridays = {}
for month in set(vstoxx_futures['EXP_MONTH']):
    third_fridays[month] = third_friday(dt.datetime(2014, month, 1))

third_fridays


I am working on the file "09_dx_calibration.ipynb".
The first task is:
Calibration Functions
Mean-Squared Error Calculation
The calibration of the pricing model is based on the minimization of the mean-squared error (MSE) of the model values vs. the market quotes. The MSE calculation is implemented by the function mean_squared_error which also penalizes economically implausible parameter values.
Can you write Python code for it?

i = 0
def mean_squared_error(p0):
    ''' Returns the mean-squared error given
    the model and market values.

    Parameters
    ===========
    p0 : tuple/list
        tuple of kappa, theta, volatility

    Returns
    =======
    MSE : float
        mean-squared error
    '''
    if p0[0] < 0 or p0[1] < 5. or p0[2] < 0 or p0[2] > 10.:
        return 100
    global i, option_selection, vstoxx_model, option_models, first, last
    pd = dt.datetime.strftime(
            option_selection['DATE'].iloc[0].to_pydatetime(),
            '%d-%b-%Y')
    mat = dt.datetime.strftime(
            option_selection['MATURITY'].iloc[0].to_pydatetime(),
            '%d-%b-%Y')
    model_values = calculate_model_values(p0)
    option_diffs = {}
    for option in model_values:
        option_diffs[option] = (model_values[option]
                              - option_selection['PRICE'].loc[option])
    MSE = np.sum(np.array(list(option_diffs.values())) ** 2) / len(option_diffs)
    if i % 150 == 0:
        # output every 0th and 100th iteration
        if i == 0:
            print('%12s %13s %4s  %6s  %6s  %6s  --> %6s' % \
                 ('pricing_date', 'maturity_date', 'i', 'kappa',
                  'theta', 'vola', 'MSE'))
        print('%12s %13s %4d  %6.3f  %6.3f  %6.3f  --> %6.3f' % \
                (pd, mat, i, p0[0], p0[1], p0[2], MSE))
    i += 1
    return MSE


i = 0
def mean_squared_error(p0):
    ''' Returns the mean-squared error given
    the model and market values.

    Parameters
    ===========
    p0 : tuple/list
        tuple of kappa, theta, volatility

    Returns
    =======
    MSE : float
        mean-squared error
    '''
    if p0[0] < 0 or p0[1] < 5. or p0[2] < 0 or p0[2] > 10.:
        return 100
    global i, option_selection, vstoxx_model, option_models, first, last
    pd = dt.datetime.strftime(
            option_selection['DATE'].iloc[0].to_pydatetime(),
            '%d-%b-%Y')
    mat = dt.datetime.strftime(
            option_selection['MATURITY'].iloc[0].to_pydatetime(),
            '%d-%b-%Y')
    model_values = calculate_model_values(p0)
    option_diffs = {}
    for option in model_values:
        option_diffs[option] = (model_values[option]
                              - option_selection['PRICE'].loc[option])
    MSE = np.sum(np.array(option_diffs.values()) ** 2) / len(option_diffs)
    if i % 150 == 0:
        # output every 0th and 100th iteration
        if i == 0:
            print '%12s %13s %4s  %6s  %6s  %6s --> %6s' % \
                 ('pricing_date', 'maturity_date', 'i', 'kappa',
                  'theta', 'vola', 'MSE')
        print '%12s %13s %4d  %6.3f  %6.3f  %6.3f  --> %6.3f' % \
                (pd, mat, i, p0[0], p0[1], p0[2], MSE)
    i += 1
    return MSE


I am working on the file "09_dx_calibration.ipynb".
The first task is:
Calibration Results
The results are now stored in the pandas DataFrame called parameters. We set a new index and inspect the last results. Throughout the MSE is pretty low indicated a good fit of the model to the market quotes.
Can you write Python code for it?

paramet = parameters.set_index('pricing_date')
paramet.tail()


I am working on the file "09_dx_calibration.ipynb".
The first task is:
Implementing the Calibration Procedure
The function get_parameter_series calibrates the model to the market data for every date contained in the pricing_date_list object for all maturities contained in the maturity_list object.
Can you write Python code for it?

import scipy.optimize as spo
def get_parameter_series(pricing_date_list, maturity_list):
    global i, option_selection, vstoxx_model, option_models, first, last
    # collects optimization results for later use (eg. visualization)
    parameters = pd.DataFrame()
    for maturity in maturity_list:
        first = True
        for pricing_date in pricing_date_list:
            option_selection, forward = \
                    get_option_selection(pricing_date, maturity)
            vstoxx_model, option_models = \
                    get_option_models(pricing_date, maturity, option_selection)
            if first is True:
                # use brute force for the first run
                i = 0
                opt = spo.brute(mean_squared_error,
                    ((0.5, 2.51, 1.),   # range for kappa
                     (10., 20.1, 5.),   # range for theta
                     (0.5, 10.51, 5.0)),  # range for volatility
                     finish=None)
            i = 0
            opt = spo.fmin(mean_squared_error, opt,
                 maxiter=200, maxfun=350, xtol=0.0000001, ftol=0.0000001)
            parameters = parameters.append(
                     pd.DataFrame(
                     {'pricing_date' : pricing_date,
                      'maturity' : maturity,
                      'initial_value' : vstoxx_model.initial_value,
                      'kappa' : opt[0],
                      'theta' : opt[1],
                      'sigma' : opt[2],
                      'MSE' : mean_squared_error(opt)}, index=[0,]),
                     ignore_index=True)
            first = False
            last = opt
    return parameters


I am working on the file "09_dx_calibration.ipynb".
The first task is:
Implied Volatilities from Market Quotes
Often calibration efforts are undertaken to replicate the market implied volatilities or the so-called volatility surface as good as possible. With DX Analytics and the BSM_european_option class, you can efficiently calculate (i.e. numerically estimate) implied volatilities. For the example, we use the VSTOXX futures and call options data from 31. March 2014.
Some definitions, the pre-selection of option data and the pre-definition of the market environment needed.
Can you write Python code for it?

V0 = 17.6639  # VSTOXX level on 31.03.2014
futures_data = vstoxx_futures[vstoxx_futures.DATE == '2014/3/31'].copy()
options_data = vstoxx_options[(vstoxx_options.DATE == '2014/3/31')
                              & (vstoxx_options.TYPE == 'C')].copy()
me = market_environment('me', dt.datetime(2014, 3, 31))
me.add_constant('initial_value', 17.6639)  # index on 31.03.2014
me.add_constant('volatility', 2.0)  # for initialization
me.add_curve('discount_curve', constant_short_rate('r', 0.01))  # assumption
options_data['IMP_VOL'] = 0.0  # initialization new iv column


I am working on the file "09_dx_calibration.ipynb".
The first task is:
Market Modeling
This sub-section now implements the model calibration based on selected options data. In particular, we choose, for a given pricing date, the following options data:

for a single maturity only
call options only
for a certain moneyness of the options

Relevant Market Data
The following following returns the relevant market data per calibration date:
Can you write Python code for it?

tol = 0.2
def get_option_selection(pricing_date, maturity, tol=tol):
    ''' Function selects relevant options data. '''
    forward = vstoxx_futures[(vstoxx_futures.DATE == pricing_date)
                & (vstoxx_futures.MATURITY == maturity)]['PRICE'].values[0]
    option_selection = \
        vstoxx_options[(vstoxx_options.DATE == pricing_date)
                     & (vstoxx_options.MATURITY == maturity)
                     & (vstoxx_options.TYPE == 'C')
                     & (vstoxx_options.STRIKE > (1 - tol) * forward)
                     & (vstoxx_options.STRIKE < (1 + tol) * forward)]
    return option_selection, forward


I am working on the file "09_dx_calibration.ipynb".
The first task is:
Options Modeling
Given the options and their respective quotes to which to calibrate the model, the function get_option_models returns the DX Analytics option models for all relevant options. As risk factor model we use the square_root_diffusion class.
Can you write Python code for it?

def get_option_models(pricing_date, maturity, option_selection):
    ''' Models and returns traded options for given option_selection object. '''
    me_vstoxx = market_environment('me_vstoxx', pricing_date)
    initial_value = vstoxx_index['V2TX'][pricing_date]
    me_vstoxx.add_constant('initial_value', initial_value)
    me_vstoxx.add_constant('final_date', maturity)
    me_vstoxx.add_constant('currency', 'EUR')
    me_vstoxx.add_constant('frequency', 'W')
    me_vstoxx.add_constant('paths', 10000)
    csr = constant_short_rate('csr', 0.01)
      # somewhat arbitrarily chosen here
    me_vstoxx.add_curve('discount_curve', csr)

    # parameters to be calibrated later
    me_vstoxx.add_constant('kappa', 1.0)
    me_vstoxx.add_constant('theta', 1.2 * initial_value)
    me_vstoxx.add_constant('volatility', 1.0)

    vstoxx_model = square_root_diffusion('vstoxx_model', me_vstoxx)
      # square-root diffusion for volatility modeling
      # mean-reverting, positive process

    # option parameters and payoff
    me_vstoxx.add_constant('maturity', maturity)
    payoff_func = 'np.maximum(maturity_value - strike, 0)'

    option_models = {}
    for option in option_selection.index:
        strike = option_selection['STRIKE'].ix[option]
        me_vstoxx.add_constant('strike', strike)
        option_models[option] = \
                            valuation_mcs_european_single(
                                    'eur_call_%d' % strike,
                                    vstoxx_model,
                                    me_vstoxx,
                                    payoff_func)
    return vstoxx_model, option_models


I am working on the file "09_dx_calibration.ipynb".
The first task is:
The Calibration Itself
This completes the set of necessary function to implement such a larger calibration effort. The following code defines the dates for which a calibration shall be conducted and for which maturities the calibration is required.
Can you write Python code for it?

%%time
pricing_date_list = pd.date_range('2014/3/1', '2014/3/31', freq='B')
maturity_list = [third_fridays[7]]
parameters = get_parameter_series(pricing_date_list, maturity_list)


I am working on the file "09_dx_calibration.ipynb".
The first task is:
The VSTOXX futures data (8 futures maturities/quotes per day).
Can you write Python code for it?


I am working on the file "09_dx_calibration.ipynb".
The first task is:
The VSTOXX options data. This data set is quite large due to the large number of European put and call options on the VSTOXX.
Can you write Python code for it?


I am working on the file "09_dx_calibration.ipynb".
The first task is:
The following generates a plot of the calibration results for the last calibration day. The absolute price differences are below 0.10 EUR for all options.
Can you write Python code for it?

index = paramet.index[-1]
opt = np.array(paramet[['kappa', 'theta', 'sigma']].loc[index])
option_selection = get_option_selection(index, maturity_list[0], tol=tol)[0]
model_values = np.sort(np.array(calculate_model_values(opt).values()))[::-1]
import matplotlib.pyplot as plt
%matplotlib inline
fix, (ax1, ax2) = plt.subplots(2, sharex=True, figsize=(10, 8))
strikes = option_selection['STRIKE'].values
ax1.plot(strikes, option_selection['PRICE'], label='market quotes')
ax1.plot(strikes, model_values, 'ro', label='model values')
ax1.set_ylabel('option values')
ax1.grid(True)
ax1.legend(loc=0)
wi = 0.25
ax2.bar(strikes - wi / 2., model_values - option_selection['PRICE'],
        label='market quotes', width=wi)
ax2.grid(True)
ax2.set_ylabel('differences')
ax2.set_xlabel('strikes')


index = paramet.index[-1]
opt = np.array(paramet[['kappa', 'theta', 'sigma']].loc[index])
option_selection = get_option_selection(index, maturity_list[0], tol=tol)[0]
model_values = np.sort(np.array(list(calculate_model_values(opt).values())))[::-1]
import matplotlib.pyplot as plt
%matplotlib inline
fix, (ax1, ax2) = plt.subplots(2, sharex=True, figsize=(10, 8))
strikes = option_selection['STRIKE'].values
ax1.plot(strikes, option_selection['PRICE'], label='market quotes')
ax1.plot(strikes, model_values, 'ro', label='model values')
ax1.set_ylabel('option values')
ax1.grid(True)
ax1.legend(loc=0)
wi = 0.25
ax2.bar(strikes - wi / 2., model_values - option_selection['PRICE'],
        label='market quotes', width=wi)
ax2.grid(True)
ax2.set_ylabel('differences')
ax2.set_xlabel('strikes')


I am working on the file "09_dx_calibration.ipynb".
The first task is:
The following loop now calculates the implied volatilities for all those options whose strike lies within the defined tolerance level.
Can you write Python code for it?

%%time
tol = 0.3  # tolerance level for moneyness
for option in options_data.index:
    # iterating over all option quotes
    forward = futures_data[futures_data['MATURITY'] == \
                options_data.loc[option]['MATURITY']]['PRICE'].values
                # picking the right futures value
    if (forward * (1 - tol) < options_data.loc[option]['STRIKE']
                             < forward * (1 + tol)):
        # only for options with moneyness within tolerance
        call = options_data.loc[option]
        me.add_constant('strike', call['STRIKE'])
        me.add_constant('maturity', call['MATURITY'])
        call_option = BSM_european_option('call', me)
        options_data.loc[option, 'IMP_VOL'] = \
            call_option.imp_vol(call['PRICE'], 'call', volatility_est=0.6)


I am working on the file "09_dx_calibration.ipynb".
The first task is:
The function calculate_model_values estimates and returns model value estimates for all relevant options given a parameter set for the square_root_diffusion risk factor model.
Can you write Python code for it?

def calculate_model_values(p0):
    ''' Returns all relevant option values.

    Parameters
    ===========
    p0 : tuple/list
        tuple of kappa, theta, volatility

    Returns
    =======
    model_values : dict
        dictionary with model values
    '''
    kappa, theta, volatility = p0
    vstoxx_model.update(kappa=kappa,
                        theta=theta,
                        volatility=volatility)
    model_values = {}
    for option in option_models:
       model_values[option] = \
         option_models[option].present_value(fixed_seed=True)
    return model_values


I am working on the file "09_dx_calibration.ipynb".
The first task is:
This is also illustrated by the visualization of the time series data for the calibrated/optimal parameter values. The MSE is below 0.01 throughout.
Can you write Python code for it?

%matplotlib inline
paramet[['kappa', 'theta', 'sigma', 'MSE']].plot(subplots=True, color='b', figsize=(10, 12))
plt.tight_layout()


I am working on the file "09_dx_calibration.ipynb".
The first task is:
VSTOXX Futures & Options Data
We start by loading VSTOXX data from a pandas HDFStore into DataFrame objects (source: Eurex, cf. http://www.eurexchange.com/advanced-services/).
Can you write Python code for it?

h5 = pd.HDFStore('./data/vstoxx_march_2014.h5', 'r')
vstoxx_index = h5['vstoxx_index']
vstoxx_futures = h5['vstoxx_futures']
vstoxx_options = h5['vstoxx_options']
h5.close()


I am working on the file "09_dx_calibration.ipynb".
The first task is:
VSTOXX index for the first quarter of 2014.
Can you write Python code for it?

%matplotlib inline
vstoxx_index['V2TX'].plot(figsize=(10, 6))


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
1.- ¿Porqué utilizar pandas?
Razón oficial:
Porque en numpy no es posible mezclar tipos de datos, lo cual complica cargar, usar, limpiar y guardar datos mixtos.
Razón personal:
Porque habían cosas que en R eran más fáciles pero no pythonísticas. La librería pandas es un excelente compromiso.
Can you write Python code for it?

import numpy as np
df = np.loadtxt("data/data.csv", delimiter=";", dtype=str)
print( df )

import pandas as pd
df = pd.read_csv("data/data.csv", sep=";")
print( df )
#df

inch2m = 0.0254
feet2m = 0.3048
df.diametro = df.diametro * inch2m
df.altura = df.altura * feet2m
df.volumen = df.volumen * (feet2m**3)
df.tipo_de_arbol = "Cherry Tree"
df

print( df.columns )

print( df.index )

print( df["altura"]*2 )

print( df["diametro"]**2 * df["altura"] / df.volumen )


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
2. Lo básico de pandas

Pandas imita los dataframes de R, pero en python. Todo lo que no tiene sentido es porque se parece demasiado a R.
Pandas permite tener datos como en tablas de excel: datos en una columna pueden ser mixtos.
La idea central es que la indexación es "a medida": las columnas y las filas (index) pueden ser enteros o floats, pero también pueden ser strings. Depende de lo que tenga sentido.
Los elementos básicos de pandas son:
Series: Conjunto de valores con indexación variable.
DataFrames: Conjunto de Series.

2.1 Series
Una serie es un conveniente conjunto de datos, como una columna de datos de excel, pero con indexación más genérica.
python
pd.Series(self, data=None, index=None, dtype=None, name=None, copy=False, fastpath=False)
Can you write Python code for it?

import pandas as pd
s1 = pd.Series([False, 1, 2., "3", 4 + 0j])
print( s1 )

# Casting a otros tipos
print( list(s1) )
print( set(s1) )
print( np.array(s1) )

# Ejemplo de operatoria
s0 = pd.Series(range(6), index=range(6))
s1 = pd.Series([1,2,3], index=[1,2,3])
s2 = pd.Series([4,5,6], index=[4,5,6])
s3 = pd.Series([10,10,10], index=[1,4,6])

print( s0 )

print( s0 + s1 )

print( s0 + s1 + s2 )

print( s0.add(s1, fill_value=0) )


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
2.2 DataFrames
Un Dataframe es una colección de Series con una indexación común. Como una planilla de excel.
python
pd.DataFrame(self, data=None, index=None,
             columns=None, dtype=None, copy=False)
Can you write Python code for it?


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
3.1 Obteniendo datos

Archivo csv
Archivo json
Archivo de excel: convertir a csv cuidadosamente (elegir un separador apropiado, no comentar strings).
Can you write Python code for it?

# csv
df = pd.read_csv("data/data.csv", sep=";")
df

df = pd.read_json("data/data.json")
df


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
4.- Inspeccionando datos

Accesando las columnas
shape
head, tail, describe
histogram
pd.scatter_matrix
count_values
Can you write Python code for it?

df = pd.read_csv("data/data.csv", sep=";")
df.columns

df['altura']

df.shape

df.head()

df.tail()

df.describe()

df.describe(include="all")

from matplotlib import pyplot as plt
df.hist(figsize=(10,10), layout=(3,1))
#df.hist(figsize=(8,8), layout=(3,1), by="tipo_de_arbol")
plt.show()

from matplotlib import pyplot as plt
pd.scatter_matrix(df, figsize=(10,10), range_padding=0.2)
plt.show()

df.tipo_de_arbol.value_counts()


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
5.- Manipulando DataFrames

Agregando columnas
Borrando columnas
Agregando filas
Borrando filas
Mask
Grouping
Imputación de datos
Apply
Merge (a la SQL)
Accesamiento

5.1 Agregando columnas
Can you write Python code for it?

df = pd.read_csv("data/data.csv", sep=";")
df["radio"] = .5 * df.diametro
df

df.area = np.pi * df.radio **2
df.columns


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
5.10 Apply
Can you write Python code for it?

df = pd.read_csv("data/data.csv", sep=";")
df1 = df.diametro.apply(lambda x: x*2)
df1

# Aplicación incorrecta
df2 = df["tipo_de_arbol"].apply(str.upper) # Error
df2

# Aplicación correcta
df2 = df["tipo_de_arbol"].apply(lambda s: str(s).upper() )
df2

# Error (o no?)
df3 = df.apply(lambda x: x*2)
df3


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
5.11 Merge
Can you write Python code for it?

df1 = pd.read_csv("data/data.csv", sep=";")
df1

df2 = pd.DataFrame(data={"tipo_de_arbol":["Cherry Tree", "Apple Tree", "Pear Tree"],
                         "fruto":["guinda","manzana", "pera"],
                         "precio_pesos_por_kg":[500, 2000, np.nan]})
df2

df3 = df1.merge(df2, how="left", on="tipo_de_arbol")
df3

df3 = df1.merge(df2, how="right", on="tipo_de_arbol")
df3

df3 = df1.merge(df2, how="inner", on="tipo_de_arbol")
df3

df3 = df1.merge(df2, how="outer", on="tipo_de_arbol")
df3


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
5.3 Borrando columnas
Can you write Python code for it?

df = pd.read_csv("data/data.csv", sep=";")
print( df.columns )

df = df[["tipo_de_arbol","volumen", "diametro"]]
df

df = df.drop("tipo_de_arbol", axis=1)
df

df.drop("diametro", axis=1, inplace=True)
df


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
5.5 Renombrando filas (índices)
Can you write Python code for it?

df = pd.read_csv("data/data.csv", sep=";")
print df.index

df.index = df.index + 10
print df.index

df.index = ["i_%d"%idx for idx in df.index]
print df.index


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
5.8.- Grouping
Can you write Python code for it?

df = pd.read_csv("data/data.csv", sep=";")
df.columns

g = df.groupby("tipo_de_arbol")
print( g )

print( g.count() )

print( g.sum() ) # .mean(), .std()

# Ejemplo real
df[["tipo_de_arbol","diametro", "altura"]].groupby("tipo_de_arbol").mean()


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
Aprender haciendo
Consideraremos el siguiente archivo data.csv que contiene datos incompletos:
Can you write Python code for it?


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
Guardando datos

csv
json
excel

Lo más importante es tener cuidado de cómo se guardan los nombres de las columnas (header), y el indice (index).
Depende de la utilización, pero mi recomendación es guardar el header explícitamente y guardar el index como una columna.
Can you write Python code for it?

# guardar un csv
df = pd.read_csv("data/data.csv", sep=";")
df = df[df.tipo_de_arbol=="Cherry Tree"]
df.to_csv("data/output.csv", sep="|", index=True) # header=True by default
df

# Leer el csv anterior
df2 = pd.read_csv("data/output.csv", sep="|", index_col=0) # get index from first column
df2

%%bash
cat data/output.csv

# guardar un json
df = pd.read_csv("data/data.csv", sep=";")
df = df[df.tipo_de_arbol=="Cherry Tree"]
df.to_json("data/output.json")
df

# Leyendo el json anterior
df2 = pd.read_json("data/output.json")
df2

%%bash
cat data/output.json


I am working on the file "09_libreria_pandas/pandas.ipynb".
The first task is:
Observación
```python
seleccionar la columna col
regresa una serie
df[col]
seleccionar las columnas col1, col2, ..., coln
regresa dataframe
df[[col1,col2,.., coln]]
selecciona solo el indice inicio
regresa un dataframe
df[inicio:(inicio+1)]
selecciona los indices en notacion
regresa un dataframe
df[inicio:fin:salto]
seleccion mixta
regresa un dataframe
df.loc[inicio:fin:salto, col1:col2]
```
5.7 Masking
Can you write Python code for it?

df = pd.read_csv("data/data.csv", sep=";")
vol_mean = df.volumen.mean()
vol_std = df.volumen.std()

mask_1 = df.altura < 80
mask_2 =  df.volumen <= vol_mean + vol_std
df1 = df[ mask_1 & mask_2 ]
df1

# Si se hace dinamicamente, utilizar suficientes parentesis
#df2 = df[ ((vol_mean - vol_std) <= df.volumen) & (df.volumen <= (vol_mean + vol_std) ) ]
df2 = df[ (df.volumen >=(vol_mean - vol_std)) & (df.volumen <= (vol_mean + vol_std) ) ]
df2

# A veces para simplificar numpy ayuda
mask_1 = df.volumen >= (vol_mean - vol_std)
mask_2 = df.volumen <= (vol_mean + vol_std)
mask = np.logical_and(mask_1, mask_2)
df3 = df[np.logical_not(mask)]
df3


I am working on the file "09_matrix_factorization/src/py_part_9_iris_pca.ipynb".
The first task is:
Create symmetrical covariance matrix for PCA
Covariance $C_{i,j}$ measures the amount one feature $x_i$ changes with another feature $x_j$ for all the features $j$ in the data set $X$.
\begin{equation}
C_{i, j} = \frac{1}{N} x_i x_j, \text{  }  x_i, x_j \in X_j
\end{equation}
Can you write Python code for it?

covariance_matrix = np.cov(iris, rowvar=False)
print('Covariance Matrix:\n', covariance_matrix)


I am working on the file "09_matrix_factorization/src/py_part_9_iris_pca.ipynb".
The first task is:
Eigen decomposition (a very important type of matrix factorization in machine learning)
Eigen decomposition of a covariance or correlation matrix is known as principal components analysis (PCA). Eigen decomposition involves calculating two matrices $\mathbf{Q}$ and $\mathbf{\Lambda}$, such that the covariance or correlation matrix $\mathbf{C} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^{-1}$, where $\mathbf{Q}$ is a p x p matrix of eigenvectors and $\mathbf{\Lambda}$ is a diagonal, p x p matrix of eigenvalues. Eigenvectors are orthogonal vectors in the directions of the highest variance in the data matrix. Eigenvalues determine the length of the eigenvectors and eigenvectors are ranked by the magnitude of their corresponsing eigenvalue. The eigenvalue with the largest magnitude corresponds to the eigenvector which spans the direction of the highest variance in the original data set and so on.
Eigen decomposition
\begin{equation}
\mathbf{C} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^{-1}
\end{equation}
\begin{equation}
\mathbf{C}\mathbf{Q} = \mathbf{Q}\mathbf{\Lambda}\mathbf{Q}^{-1}\mathbf{Q}
\end{equation}
\begin{equation}
\mathbf{C}\mathbf{Q} = \mathbf{Q}\mathbf{\Lambda}
\end{equation}
The above equation can be decomposed in sets of simultaneous equations. For any eigenvector, $\mathbf{q}_j$:
\begin{equation}
\mathbf{C}\mathbf{q}_j = \mathbf{q}_j\lambda_j
\end{equation}
\begin{equation}
\mathbf{C}\mathbf{q}_j = \lambda_j\mathbf{q}_j
\end{equation}
\begin{equation}
\mathbf{C}\mathbf{q}_j - \lambda_j\mathbf{q}_j = 0
\end{equation}
\begin{equation}
(\mathbf{C} - \lambda_j\mathbf{I})\mathbf{q}_j = 0
\end{equation}
Because $\mathbf{q}$ comes from the non-singular matrix of eigenvectors, $(\mathbf{C} - \lambda_j\mathbf{I})$, and thus $det(\mathbf{C} - \lambda\mathbf{I})$, must equal 0. Which implies a polynomial equation in which roots $\lambda_{j}$ can be determined using:
\begin{equation}
\prod_{j} (\mathbf{c}{j,j} - \lambda{j}) = 0, \text{for } j \leq p
\end{equation}
Once all $\lambda_{j}$, and hence...
Can you write Python code for it?

eigen_values, eigen_vectors = np.linalg.eig(covariance_matrix)
print('Eigen Values:\n', eigen_values)
print()
print('Eigen Vectors:\n', eigen_vectors)


I am working on the file "09_matrix_factorization/src/py_part_9_iris_pca.ipynb".
The first task is:
Extract three features and plot
Can you write Python code for it?

three_PCs = iris.dot(eigen_vectors[:, :3])

fig = plt.figure(figsize=(7,  7))
ax = plt.axes(projection='3d')

setosa = ax.scatter(three_PCs [0:50, 0], three_PCs [0:50, 1], three_PCs [0:50, 2], alpha=0.5, color='blue')
versicolor = ax.scatter(three_PCs [50:100, 0], three_PCs [50:100, 1], three_PCs [50:100, 2], alpha=0.5, color='red')
virginica = ax.scatter(three_PCs [100:150, 0], three_PCs [100:150, 1], three_PCs [100:150, 2], alpha=0.5, color='green')

plt.title('Three Dimensional Feature Extraction from Iris Data')
ax.set_xlabel('First Principal Component')
ax.set_ylabel('Second Principal Component')
ax.set_zlabel('Third Principal Component')

plt.legend([setosa, versicolor, virginica], ['Setosa', 'Versicolor', 'virginica'], bbox_to_anchor=(1.05, 0.5), loc=3,
                   borderaxespad=0.)

_ = plt.show()


I am working on the file "09_matrix_factorization/src/py_part_9_kaggle_GLRM_example.ipynb".
The first task is:
Embed categorical vars using GLRM
Training data
Can you write Python code for it?


I am working on the file "09_matrix_factorization/src/py_part_9_kaggle_GLRM_example.ipynb".
The first task is:
Embed with GLRM
Can you write Python code for it?

def glrm_cat_embed(frame):

    """ Helper function for embedding caetgorical variables using GLRM.

    :param frame: H2OFrame to be embedded.
    :return: H2OFrame of embedded categorical features.

    """

    # initialize GLRM
    cat_embed_glrm = H2OGeneralizedLowRankEstimator(
        k=50,
        transform='STANDARDIZE',
        loss='Quadratic',
        regularization_x='Quadratic',
        regularization_y='L1',
        gamma_x=0.25,
        gamma_y=0.5)

    # train GLRM
    cat_embed_glrm.train(training_frame=frame, x=cats)

    # plot iteration history to ensure convergence
    cat_embed_glrm.score_history().plot(x='iterations', y='objective', title='GLRM Score History')

    # extracted embedded features
    cat_embed = h2o.get_frame(cat_embed_glrm._model_json['output']['representation_name'])

    return cat_embed


I am working on the file "09_matrix_factorization/src/py_part_9_kaggle_GLRM_example.ipynb".
The first task is:
Helper Functions
Determine data types
Can you write Python code for it?

def get_type_lists(frame, rejects=['Id', 'SalePrice']):

    """Creates lists of numeric and categorical variables.

    :param frame: The frame from which to determine types.
    :param rejects: Variable names not to be included in returned lists.
    :return: Tuple of lists for numeric and categorical variables in the frame.

    """

    nums, cats = [], []
    for key, val in frame.types.items():
        if key not in rejects:
            if val == 'enum':
                cats.append(key)
            else:
                nums.append(key)

    print('Numeric =', nums)
    print()
    print('Categorical =', cats)

    return nums, cats


I am working on the file "09_matrix_factorization/src/py_part_9_kaggle_GLRM_example.ipynb".
The first task is:
Import data
Can you write Python code for it?

train = h2o.import_file('../../03_regression/data/train.csv')
test = h2o.import_file('../../03_regression/data/test.csv')

# bug fix - from Keston
dummy_col = np.random.rand(test.shape[0])
test = test.cbind(h2o.H2OFrame(dummy_col))
cols = test.columns
cols[-1] = 'SalePrice'
test.columns = cols
print(train.shape)
print(test.shape)

original_nums, cats = get_type_lists(train)


I am working on the file "09_matrix_factorization/src/py_part_9_kaggle_GLRM_example.ipynb".
The first task is:
Merge imputed and embedded frames
Can you write Python code for it?

imputed_embedded_train = train[['Id', 'SalePrice']].cbind(train_num_impute).cbind(train_cat_embed)
imputed_embedded_valid = valid[['Id', 'SalePrice']].cbind(valid_num_impute).cbind(valid_cat_embed)
imputed_embedded_test = test[['Id', 'SalePrice']].cbind(test_num_impute).cbind(test_cat_embed)


I am working on the file "09_matrix_factorization/src/py_part_9_kaggle_GLRM_example.ipynb".
The first task is:
Redefine numerics and explore
Can you write Python code for it?

imputed_embedded_nums, cats = get_type_lists(imputed_embedded_train)

print('Imputed and encoded numeric training data:')
imputed_embedded_train.describe()
print('--------------------------------------------------------------------------------')
print('Imputed and encoded numeric validation data:')
imputed_embedded_valid.describe()
print('--------------------------------------------------------------------------------')
print('Imputed and encoded numeric test data:')
imputed_embedded_test.describe()


I am working on the file "09_matrix_factorization/src/py_part_9_kaggle_GLRM_example.ipynb".
The first task is:
Split into to train and validation (before doing data prep!!!)
Can you write Python code for it?

train, valid = train.split_frame([0.7], seed=12345)
print(train.shape)
print(valid.shape)


I am working on the file "09_matrix_factorization/src/py_part_9_kaggle_GLRM_example.ipynb".
The first task is:
Test data
Can you write Python code for it?


I am working on the file "09_matrix_factorization/src/py_part_9_kaggle_GLRM_example.ipynb".
The first task is:
Train model on imputed, embedded features
Can you write Python code for it?

h2o.show_progress() # turn on progress bars

# Check log transform - looks good
%matplotlib inline
imputed_embedded_train['SalePrice'].log().as_data_frame().hist()

# Execute log transform
imputed_embedded_train['SalePrice'] = imputed_embedded_train['SalePrice'].log()
imputed_embedded_valid['SalePrice'] = imputed_embedded_valid['SalePrice'].log()
print(imputed_embedded_train[0:3, 'SalePrice'])


I am working on the file "09_matrix_factorization/src/py_part_9_kaggle_GLRM_example.ipynb".
The first task is:
Validation data
Can you write Python code for it?


valid_num_impute = glrm_num_impute('validation', valid)


I am working on the file "09_organizing_code.ipynb".
The first task is:
Check that training works well:
Can you write Python code for it?

n_epochs = 20
batch_size = 50

with tf.Session(graph=graph) as sess:
    init.run()
    for epoch in range(n_epochs):
        for iteration in range(mnist.train.num_examples // batch_size):
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
        acc_train = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
        acc_val = accuracy.eval(feed_dict={X: mnist.validation.images, y: mnist.validation.labels})
        print(epoch, "Train accuracy:", acc_train, "Validation accuracy:", acc_val)

    save_path = saver.save(sess, "./my_mnist_model")


I am working on the file "09_organizing_code.ipynb".
The first task is:
How would we implement variable sharing in neural_net_layer()?
Can you write Python code for it?

def neural_net_layer(inputs, n_neurons, activation=None, name=None, reuse=None, seed=None):
    with tf.variable_scope(name, default_name="layer", reuse=reuse):
        n_inputs = int(inputs.get_shape()[1])
        rnd_init = lambda shape, dtype, partition_info: tf.random_uniform(shape, -1.0, 1.0, dtype=dtype, seed=seed)
        b = tf.get_variable("biases", shape=[n_neurons], initializer=rnd_init)
        W = tf.get_variable("weights", shape=[n_inputs, n_neurons], initializer=rnd_init)
        logits = tf.matmul(inputs, W) + b
        if activation:
            return activation(logits)
        else:
            return logits

graph = tf.Graph()
with graph.as_default():
    with tf.variable_scope("foo"):
        a = tf.constant(1., name="a")
        with tf.name_scope("bar"):
            b = tf.constant(2., name="b")
            with tf.name_scope("baz"):
                c = tf.get_variable("c", shape=[], initializer=tf.constant_initializer(2))
                s = tf.add_n([a,b,c], name="s")

a.name

b.name

c.name

s.name


I am working on the file "09_organizing_code.ipynb".
The first task is:
Let's simplify our code by using neural_net_layer():
Can you write Python code for it?

n_inputs = 28 * 28
n_hidden1 = 100
n_outputs = 10

graph = tf.Graph()
with graph.as_default():
    with tf.name_scope("inputs"):
        X = tf.placeholder(tf.float32, shape=[None, n_inputs], name="X")
        y = tf.placeholder(tf.int32, shape=[None], name="y")

#########################################################################
# This section is simplified (the rest is unchanged)
#
    with tf.name_scope("hidden1"):
        hidden1 = neural_net_layer(X, n_hidden1, activation=tf.nn.relu) # <= CHANGED

    with tf.name_scope("output"):
        logits = neural_net_layer(hidden1, n_outputs)                   # <= CHANGED
        Y_proba = tf.nn.softmax(logits, name="Y_proba")
#
#
#########################################################################

    with tf.name_scope("train"):
        xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=y)
        loss = tf.reduce_mean(xentropy)
        optimizer = tf.train.AdamOptimizer()
        training_op = optimizer.minimize(loss)

    with tf.name_scope("eval"):
        correct = tf.nn.in_top_k(logits, y, 1)
        accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))

    with tf.name_scope("init_and_save"):
        init = tf.global_variables_initializer()
        saver = tf.train.Saver()

[var.op.name for var in graph.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)]


I am working on the file "09_organizing_code.ipynb".
The first task is:
Now let's use tf.layers.dense() instead:
Can you write Python code for it?

n_inputs = 28 * 28
n_hidden1 = 100
n_outputs = 10

graph = tf.Graph()
with graph.as_default():
    with tf.name_scope("inputs"):
        X = tf.placeholder(tf.float32, shape=[None, n_inputs], name="X")
        y = tf.placeholder(tf.int32, shape=[None], name="y")

    with tf.name_scope("hidden1"):
        hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.relu, name="hidden1") # <= CHANGED

    with tf.name_scope("output"):
        logits = tf.layers.dense(hidden1, n_outputs, name="output")                    # <= CHANGED
        Y_proba = tf.nn.softmax(logits)

    with tf.name_scope("train"):
        xentropy = tf.nn.sparse_softmax_cross_entropy_with_logits(logits=logits, labels=y)
        loss = tf.reduce_mean(xentropy)
        optimizer = tf.train.AdamOptimizer()
        training_op = optimizer.minimize(loss)

    with tf.name_scope("eval"):
        correct = tf.nn.in_top_k(logits, y, 1)
        accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))

    with tf.name_scope("init_and_save"):
        init = tf.global_variables_initializer()
        saver = tf.train.Saver()

[var.op.name for var in graph.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)]


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
$(3 + 4ε)\times(5 + 7ε) = 3 \times 5 + 3 \times 7ε + 4ε \times 5 + 4ε \times 7ε = 15 + 21ε + 20ε + 28ε^2 = 15 + 41ε + 28 \times 0 = 15 + 41ε$
Can you write Python code for it?

DualNumber(3, 4) * DualNumber(5, 7)

x.value = DualNumber(3.0)
y.value = DualNumber(4.0)

f.evaluate()

x.value = DualNumber(3.0, 1.0)  # 3 + ε
y.value = DualNumber(4.0)       # 4

df_dx = f.evaluate().eps

x.value = DualNumber(3.0)       # 3
y.value = DualNumber(4.0, 1.0)  # 4 + ε

df_dy = f.evaluate().eps

df_dx

df_dy


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
$3 + (3 + 4 \epsilon) = 6 + 4\epsilon$
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
And this subdirectory contains one log file (called a "TF events" file) for the graph:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
And we are ready to train the model and use it for predictions!
There's really nothing special about this code, it's virtually the same as the one we used earlier for Linear Regression:
Can you write Python code for it?

n_epochs = 1000
batch_size = 50
n_batches = int(np.ceil(m / batch_size))

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(n_epochs):
        for batch_index in range(n_batches):
            X_batch, y_batch = random_batch(X_train, y_train, batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
        loss_val = loss.eval({X: X_test, y: y_test})
        if epoch % 100 == 0:
            print("Epoch:", epoch, "\tLoss:", loss_val)

    y_proba_val = y_proba.eval(feed_dict={X: X_test, y: y_test})


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
As we saw in chapter 4, the log loss is a good cost function to use for Logistic Regression:
$J(\boldsymbol{\theta}) = -\dfrac{1}{m} \sum\limits_{i=1}^{m}{\left[ y^{(i)} \log\left(\hat{p}^{(i)}\right) + (1 - y^{(i)}) \log\left(1 - \hat{p}^{(i)}\right)\right]}$
One option is to implement it ourselves:
Can you write Python code for it?

epsilon = 1e-7  # to avoid an overflow when computing the log
loss = -tf.reduce_mean(y * tf.log(y_proba + epsilon) + (1 - y) * tf.log(1 - y_proba + epsilon))


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
As we saw in chapter 4, the log loss is a good cost function to use for Logistic Regression:
$J(\mathbf{\theta}) = -\dfrac{1}{m} \sum\limits_{i=1}^{m}{\left[ y^{(i)} log\left(\hat{p}^{(i)}\right) + (1 - y^{(i)}) log\left(1 - \hat{p}^{(i)}\right)\right]}$
One option is to implement it ourselves:
Can you write Python code for it?


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
At last we can train the model! We will start by checking whether a previous training session was interrupted, and if so we will load the checkpoint and continue training from the epoch number we saved. In this example we just save the epoch number to a separate file, but in chapter 11 we will see how to store the training step directly as part of the model, using a non-trainable variable called global_step that we pass to the optimizer's minimize() method.
You can try interrupting training to verify that it does indeed restore the last checkpoint when you start it again.
Can you write Python code for it?

n_epochs = 10001
batch_size = 50
n_batches = int(np.ceil(m / batch_size))

checkpoint_path = "/tmp/my_logreg_model.ckpt"
checkpoint_epoch_path = checkpoint_path + ".epoch"
final_model_path = "./my_logreg_model"

with tf.Session() as sess:
    if os.path.isfile(checkpoint_epoch_path):
        # if the checkpoint file exists, restore the model and load the epoch number
        with open(checkpoint_epoch_path, "rb") as f:
            start_epoch = int(f.read())
        print("Training was interrupted. Continuing at epoch", start_epoch)
        saver.restore(sess, checkpoint_path)
    else:
        start_epoch = 0
        sess.run(init)

    for epoch in range(start_epoch, n_epochs):
        for batch_index in range(n_batches):
            X_batch, y_batch = random_batch(X_train_enhanced, y_train, batch_size)
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
        loss_val, summary_str = sess.run([loss, loss_summary], feed_dict={X: X_test_enhanced, y: y_test})
        file_writer.add_summary(summary_str, epoch)
        if epoch % 500 == 0:
            print("Epoch:", epoch, "\tLoss:", loss_val)
            saver.save(sess, checkpoint_path)
            with open(checkpoint_epoch_path, "wb") as f:
                f.write(b"%d" % (epoch + 1))

    saver.save(sess, final_model_path)
    y_proba_val = y_proba.eval(feed_dict={X: X_test_enhanced, y: y_test})
    os.remove(checkpoint_epoch_path)

file_writer.close()


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Autodiff
Note: the autodiff content was moved to the extra_autodiff.ipynb notebook.
Exercise solutions
1. to 11.
See appendix A.
12. Logistic Regression with Mini-Batch Gradient Descent using TensorFlow
First, let's create the moons dataset using Scikit-Learn's make_moons() function:
Can you write Python code for it?

from sklearn.datasets import make_moons

m = 1000
X_moons, y_moons = make_moons(m, noise=0.1, random_state=42)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Autodiff – Reverse mode
Can you write Python code for it?

class Const(object):
    def __init__(self, value):
        self.value = value
    def evaluate(self):
        return self.value
    def backpropagate(self, gradient):
        pass
    def __str__(self):
        return str(self.value)

class Var(object):
    def __init__(self, init_value, name):
        self.value = init_value
        self.name = name
        self.gradient = 0
    def evaluate(self):
        return self.value
    def backpropagate(self, gradient):
        self.gradient += gradient
    def __str__(self):
        return self.name

class BinaryOperator(object):
    def __init__(self, a, b):
        self.a = a
        self.b = b

class Add(BinaryOperator):
    def evaluate(self):
        self.value = self.a.evaluate() + self.b.evaluate()
        return self.value
    def backpropagate(self, gradient):
        self.a.backpropagate(gradient)
        self.b.backpropagate(gradient)
    def __str__(self):
        return "{} + {}".format(self.a, self.b)

class Mul(BinaryOperator):
    def evaluate(self):
        self.value = self.a.evaluate() * self.b.evaluate()
        return self.value
    def backpropagate(self, gradient):
        self.a.backpropagate(gradient * self.b.value)
        self.b.backpropagate(gradient * self.a.value)
    def __str__(self):
        return "({}) * ({})".format(self.a, self.b)

x = Var(3, name="x")
y = Var(4, name="y")
f = Add(Mul(Mul(x, x), y), Add(y, Const(2))) # f(x,y) = x²y + y + 2

result = f.evaluate()
f.backpropagate(1.0)

print("f(x,y) =", f)
print("f(3,4) =", result)
print("df_dx =", x.gradient)
print("df_dy =", y.gradient)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Automatic differentiation (autodiff) – forward mode
Can you write Python code for it?

class DualNumber(object):
    def __init__(self, value=0.0, eps=0.0):
        self.value = value
        self.eps = eps
    def __add__(self, b):
        return DualNumber(self.value + self.to_dual(b).value,
                          self.eps + self.to_dual(b).eps)
    def __radd__(self, a):
        return self.to_dual(a).__add__(self)
    def __mul__(self, b):
        return DualNumber(self.value * self.to_dual(b).value,
                          self.eps * self.to_dual(b).value + self.value * self.to_dual(b).eps)
    def __rmul__(self, a):
        return self.to_dual(a).__mul__(self)
    def __str__(self):
        if self.eps:
            return "{:.1f} + {:.1f}ε".format(self.value, self.eps)
        else:
            return "{:.1f}".format(self.value)
    def __repr__(self):
        return str(self)
    @classmethod
    def to_dual(cls, n):
        if hasattr(n, "value"):
            return n
        else:
            return cls(n)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
But we might as well use TensorFlow's tf.losses.log_loss() function:
Can you write Python code for it?

loss = tf.losses.log_loss(y, y_proba)  # uses epsilon = 1e-7 by default


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Compare with Scikit-Learn
Can you write Python code for it?

from sklearn.linear_model import LinearRegression
lin_reg = LinearRegression()
lin_reg.fit(housing.data, housing.target.reshape(-1, 1))

print(np.r_[lin_reg.intercept_.reshape(-1, 1), lin_reg.coef_.T])


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Compare with pure NumPy
Can you write Python code for it?

X = housing_data_plus_bias
y = housing.target.reshape(-1, 1)
theta_numpy = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)

print(theta_numpy)


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Creating and running a graph
Can you write Python code for it?

import tensorflow as tf

reset_graph()

x = tf.Variable(3, name="x")
y = tf.Variable(4, name="y")
f = x*x*y + y + 2

f

sess = tf.Session()
sess.run(x.initializer)
sess.run(y.initializer)
result = sess.run(f)
print(result)

sess.close()

with tf.Session() as sess:
    x.initializer.run()
    y.initializer.run()
    result = f.eval()

result

init = tf.global_variables_initializer()

with tf.Session() as sess:
    init.run()
    result = f.eval()

result

init = tf.global_variables_initializer()

sess = tf.InteractiveSession()
init.run()
result = f.eval()
print(result)

sess.close()

result


I am working on the file "09_up_and_running_with_tensorflow.ipynb".
The first task is:
Depending on the use case, you may want to choose a different threshold than 0.5: make it higher if you want high precision (but lower recall), and make it lower if you want high recall (but lower precision). See chapter 3 for more details.
Let's compute the model's precision and recall:
Can you write Python code for it?

from sklearn.metrics import precision_score, recall_score

precision_score(y_test, y_pred)

recall_score(y_test, y_pred)