I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Alternatively, Matplotlib understands the MATLAB syntax, e.g., for the above command (does not work with 'inline' enabled):
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Alternatively, you can also save figures in PNG bitmap and PDF vector formats. (Note that some export formats may not be supported on your platform.)
Can you write Python code for it?

fig.savefig('foo.png', dpi=600, format='png',orientation='landscape')
fig.savefig('foo.pdf', dpi=600, format='pdf',orientation='landscape')


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Analogously, you can use semilogy() and loglog() for plots with log y-axis and loglog plots.
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
As you will have seen, we retrieved handles to the individual pie slices. Let's do something with them:
Can you write Python code for it?

plt.figure(figsize=(5,5))
handles=plt.pie([1,2,3,4], explode=[0.2,0,0,0], shadow=True, labels=['A','B','C','D'])
plt.setp(handles[0] [0], color='y')
plt.setp(handles[1] [0], text='Blubber');


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Available settings for a plot can be found this way (does not work with 'inline' enabled):
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Certainly, you can do plots with logarithmic scale:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Check whether they are correlated using a scatter plot
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Compare the two figures, spot the similarities and differences.
Matplotlib sports an add-on module for 3D graphics as detailed below. First, we need to import this module.
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Especially for figures with multiple subpanels it may be advisable to increase the figure size somewhat. Do this by using function arguments in the figure() call:
Can you write Python code for it?

# switch back to inline mode
%matplotlib inline

fig2=plt.figure(figsize=(10,10))
plt.subplot(2,1,1)
plt.plot(xv,yv,'b-')
plt.subplot(2,1,2)
plt.plot(yv,xv,'ro');


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Exercise 2
generate two 1D-arrays $A$ and $B$ of size $N$ containing Gaussian random numbers
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Figures can be saved in a number of output formats such as Postscript:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
For horizontal bar charts, you would use barh().
Where there are bars, there should be pies:
Can you write Python code for it?

plt.figure(figsize=(5,5))
handles=plt.pie([1,2,3,4], explode=[0.2,0,0,0], shadow=True, labels=['A','B','C','D']);
handles


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
In addition to the above 3D scatter plot, other plot types are supported, such as 3D surface plots:
Can you write Python code for it?

plt.close('all'); fig=plt.figure()
ax=Axes3D(fig)
xv=np.linspace(-10,10,100); yv=np.linspace(-10,10,100)
cx,cy=np.meshgrid(xv,yv)
cz=0.5*cx+np.exp(-cy**2)
tilt=ax.plot_surface(cx,cy,cz,linewidth=0, cmap=plt.cm.jet);
plt.show()


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Let's add grid lines:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Let's do a figure with several subpanels:
Can you write Python code for it?

fig = plt.figure()
plt.subplot(2,1,1)
plt.plot(xv,yv,'b-')
plt.subplot(2,1,2)
plt.plot(yv,xv,'ro')
plt.show()

plt.close(fig)


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Let's pimp the plot a little:
Can you write Python code for it?

mybar=plt.bar(xv, yv, width=1, yerr=0.5);
plt.xticks(xv, ['A','B','C','D'])
plt.setp(mybar, facecolor='r', edgecolor='w');


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Let's switch to the Qt backend for 3D plotting
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Let's switch to the Qt backend for plotting
Can you write Python code for it?

%matplotlib
plt.ioff() # deactivates 'interactive' mode

fig = plt.figure()
ti=plt.title('Very important data')
xl=plt.xlabel('time'); yl=plt.ylabel('value')
plt.setp(xl,fontweight='bold');
plt.show()


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Or in MATLAB style, close all open figures:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Polar plots are also nicely illustrated on the very same homepage:
Can you write Python code for it?

ax=plt.axes([0.025,0.025,0.95,0.95],polar=True)
N=20; theta=np.arange(0.0,2*np.pi,2*np.pi/N)
radii=10*np.random.rand(N)
width=np.pi/4*np.random.rand(N)
bars=plt.bar(theta,radii,width=width,bottom=0.0)
for r,bar in zip(radii,bars):
    bar.set_facecolor( plt.cm.jet(r/10.))
    bar.set_alpha(0.5)


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Some more commands for plot formatting:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Style sheets
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
Then, we can play around.
Can you write Python code for it?

plt.close('all'); fig=plt.figure()
ax=Axes3D(fig)
plt.show()


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
You can close an open figure using its handle:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2016.ipynb".
The first task is:
to pick one of them, type e.g.
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
A similar yet distinct representation is provided by pcolormesh().
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Above commands enable pylab environment => direct access to numpy, scipy and matplotlib. The option 'inline' results in plot outputs to be directly embedded in the Notebook. If this causes problems, remove the option 'inline'.
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Alternatively, you can also save figures in PNG bitmap and PDF vector formats. (Note that some export formats may not be supported on your platform.)
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Analogously, you can use semilogy() and loglog() for plots with log y-axis and loglog plots.
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Anybody bar charts?
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Available settings for a plot can be found this way (does not work with 'inline' enabled):
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
By using Numpy arrays, Matplotlib can conveniently be used as a function ploting program:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
For horizontal bar charts, you would use barh().
Where there are bars, there should be pies:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
In addition to the above 3D scatter plot, other plot types are supported, such as 3D surface plots:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Let's add grid lines:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Let's display the same data in contour representation:
Can you write Python code for it?

plt.close('all'); fig=plt.figure()
ax=Axes3D(fig)
ax.contour(cx,cy,cz,cstride=2,rstride=2, cmap=plt.cm.jet)
plt.show()


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Let's do a figure with several subpanels:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Let's pimp the plot a little:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Let's switch to the Qt backend for 3D plotting
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Let's switch to the Qt backend for plotting
Can you write Python code for it?

%matplotlib
plt.ioff() # deactivates 'interactive' mode

fig = plt.figure()
ti=plt.title('Very important data')
xl=plt.xlabel('time'); yl=plt.ylabel('value')
plt.setp(xl,fontweight='bold');
plt.plot(xv,yv,'k--')
plt.show()


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Matplotlib also offers quiver() plots which are illustrated in the following example (taken from http://www.loria.fr/~rougier/teaching/matplotlib/):
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Polar plots are also nicely illustrated on the very same homepage:
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Style sheets
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Then, we can play around.
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Try moving and rotating the (so far empty) plot in three dimensions. After the first excitement has settled, let's populate the plot with some data:
Can you write Python code for it?

plt.close('all'); fig=plt.figure()
ax=Axes3D(fig)
import random as rn
xv=[]; yv=[]; zv=[]
for c in range(100):
   xv.append(rn.random()); yv.append(rn.random())
   zv.append(rn.random())
ax.scatter(xv,yv,zv)
plt.show()


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
Try some other colormaps such as cm.bone, cm.spring or cm.cool (once more, these are the MATLAB color schemes).
We do one more example that shows the use of 3D surface and 3D contour plots (as opposed to the 2D contour plots above).
Can you write Python code for it?

plt.close('all'); fig=plt.figure()
ax=Axes3D(fig)
xv=np.linspace(-10,10,100); yv=np.linspace(-10,10,100)
cx,cy=np.meshgrid(xv,yv)
cz=0*cx

def gauss2D(x0,y0,sigx=1,sigy=1,height=1):
   z=height*np.exp(-((cx-x0)**2/2/sigx**2)-((cy-y0)**2/2/sigy**2))
   return z

cz=cz+gauss2D(-2,3)
cz=cz+gauss2D(2,4,2,3)
ax.plot_surface(cx,cy,cz,linewidth=0,cstride=2, rstride=2,cmap=plt.cm.jet)
plt.show()


I am working on the file "05_dataio_matplotlib/Matplotlib_2017.ipynb".
The first task is:
to pick one of them, type e.g.
Can you write Python code for it?


I am working on the file "05_dataio_matplotlib/dataio_matplotlib_exercise.ipynb".
The first task is:
Exercise 2: Plot a histogram
take the nucleotide composition of the gene above and plot a histogram of the A, T, G and C frequency. Have your histogram labeled nicely and give it a title. Please, choose yourself if you would like to display horizontal or vertical bars. Advanced options include change of color for individual bars, width of the bars and alignment of labels and bars.
Can you write Python code for it?


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Adding access to a dataset programmatically
Can you write Python code for it?

dsinfo = bq.get_dataset("ch05")
entry = bigquery.AccessEntry(
    role="READER",
    entity_type="userByEmail",
    entity_id="vlakshmanan@google.com",
)
if entry not in dsinfo.access_entries:
  entries = list(dsinfo.access_entries)
  entries.append(entry)
  dsinfo.access_entries = entries
  dsinfo = bq.update_dataset(dsinfo, ["access_entries"])  # API request
else:
  print('{} already has access'.format(entry.entity_id))
print(dsinfo.access_entries)


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Another way is to create a dataset reference
Can you write Python code for it?

from google.cloud.bigquery.dataset import DatasetReference
dsinfo = bq.get_dataset('bigquery-public-data.london_bicycles')
print('{} created on {} in {}'.format(dsinfo.dataset_id, dsinfo.created, dsinfo.location))
for access in dsinfo.access_entries:
    if access.role == 'READER':
        print(access)


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Authenticate and build stubs
Can you write Python code for it?

PROJECT='cloud-training-demos'  # CHANGE THIS
from google.cloud import bigquery
bq = bigquery.Client(project=PROJECT)


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
By default, this appends rows:
Can you write Python code for it?

print('Num rows = ', bq.get_table(table_id).num_rows)
job = bq.load_table_from_dataframe(df, table_id)
job.result() # blocks and waits
print("Loaded {} rows into {}".format(job.output_rows, table_id))
print('Num rows = ', bq.get_table(table_id).num_rows)


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Copying a table
Can you write Python code for it?

# copy london stations table to our dataset
source_tbl = 'bigquery-public-data.london_bicycles.cycle_stations'
dest_tbl = '{}.ch05eu.cycle_stations_copy'.format(PROJECT)
job = bq.copy_table(source_tbl, dest_tbl, location='EU')
job.result() # blocks and waits
dest_table = bq.get_table(dest_tbl)
print(dest_table.num_rows)


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Creating a dataset
Can you write Python code for it?

dataset_id = "{}.ch05".format(PROJECT)
ds = bq.create_dataset(dataset_id, exists_ok=True)
print('{} created on {} in {}'.format(ds.dataset_id, ds.created, ds.location))


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Creating a table
Can you write Python code for it?

table_id = '{}.ch05.temp_table'.format(PROJECT)
table = bq.create_table(table_id, exists_ok=True)
print('{} created on {}'.format(table.table_id, table.created))


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Creating an empty table with schema
Can you write Python code for it?

schema = [
  bigquery.SchemaField("chapter", "INTEGER", mode="REQUIRED"),
  bigquery.SchemaField("title", "STRING", mode="REQUIRED"),
]

table_id = '{}.ch05.temp_table2'.format(PROJECT)
table = bigquery.Table(table_id, schema)
table = bq.create_table(table, exists_ok=True)
print('{} created on {}'.format(table.table_id, table.created))
print(table.schema)

# remove the two temporary tables
bq.delete_table('ch05.temp_table', not_found_ok=True)
bq.delete_table('ch05.temp_table2', not_found_ok=True)


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Dataset manipulation
Get info about a dataset
Can you write Python code for it?

# information about the ch04 dataset in our project
dataset_id = "{}.ch04".format(PROJECT)
dsinfo = bq.get_dataset(dataset_id)
print(dsinfo.dataset_id)
print(dsinfo.created)


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Deleting a dataset
Can you write Python code for it?


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Deleting a table
Can you write Python code for it?


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Dry run
Can you write Python code for it?

config = bigquery.QueryJobConfig()
config.dry_run = True
job = bq.query(query, location='EU', job_config=config)
print("This query will process {} bytes.".format(job.total_bytes_processed))


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Exporting from a table to Cloud Storage
Can you write Python code for it?

BUCKET=PROJECT + '-eu-temp'
!gsutil mb -l EU gs://$BUCKET

source_tbl = 'bigquery-public-data.london_bicycles.cycle_stations'
dest_uri = 'gs://{}/tmp/exported/cycle_stations'.format(BUCKET)
config = bigquery.job.ExtractJobConfig(
  destination_format=bigquery.job.DestinationFormat.NEWLINE_DELIMITED_JSON)
job = bq.extract_table(source_tbl, dest_uri, location='EU', job_config=config)
job.result() # blocks and waits

!gsutil cat $dest_uri | head -5

!gsutil rm -rf gs://$BUCKET
!gsutil rb -f gs://$BUCKET


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Get info about a dataset in some other project
Can you write Python code for it?

dsinfo = bq.get_dataset('bigquery-public-data.london_bicycles')
print('{} created on {}'.format(dsinfo.dataset_id, dsinfo.created))


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Query and get result
Can you write Python code for it?

query = """
SELECT
  start_station_name
  , AVG(duration) as duration
  , COUNT(duration) as num_trips
FROM `bigquery-public-data`.london_bicycles.cycle_hire
GROUP BY start_station_name
ORDER BY num_trips DESC
LIMIT 10
"""
print(query)


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Query result to Pandas dataframe
Can you write Python code for it?

query = """
SELECT
  start_station_name
  , AVG(duration) as duration
  , COUNT(duration) as num_trips
FROM `bigquery-public-data`.london_bicycles.cycle_hire
GROUP BY start_station_name
"""
df = bq.query(query, location='EU').to_dataframe()
print(df.describe())


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Table manipulation
List tables in dataset
Can you write Python code for it?

# list tables in dataset
tables = bq.list_tables("bigquery-public-data.london_bicycles")
for table in tables:
    print(table.table_id)


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Update table schema
Can you write Python code for it?

schema = [
  bigquery.SchemaField("chapter", "INTEGER", mode="REQUIRED"),
  bigquery.SchemaField("title", "STRING", mode="REQUIRED"),
]
table_id = '{}.ch05.temp_table'.format(PROJECT)
table = bq.get_table(table_id)
print(table.etag)
table.schema = schema
table = bq.update_table(table, ["schema"])
print(table.schema)
print(table.etag)


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Updating a dataset
Can you write Python code for it?

dsinfo = bq.get_dataset("ch05")
print(dsinfo.description)
dsinfo.description = "Chapter 5 of BigQuery: The Definitive Guide"
dsinfo = bq.update_dataset(dsinfo, ['description'])
print(dsinfo.description)


I am working on the file "05_devel/bigquery_cloud_client.ipynb".
The first task is:
Using the write disposition allows you to truncate the table
Can you write Python code for it?

from google.cloud.bigquery.job import LoadJobConfig, WriteDisposition, CreateDisposition

print('Num rows = ', bq.get_table(table_id).num_rows)
load_config = LoadJobConfig(
  create_disposition=CreateDisposition.CREATE_IF_NEEDED,
  write_disposition=WriteDisposition.WRITE_TRUNCATE)
job = bq.load_table_from_dataframe(df, table_id, job_config=load_config)
job.result() # blocks and waits
print("Loaded {} rows into {}".format(job.output_rows, table_id))
print('Num rows = ', bq.get_table(table_id).num_rows)


I am working on the file "05_devel/google_api_client.ipynb".
The first task is:
Asynchronous query and paging through results
Can you write Python code for it?

# send a query request that will not terminate within the timeout specified and will require paging
request={
  "useLegacySql": False,
  "timeoutMs": 0,
  "useQueryCache": False,
  "query": "SELECT start_station_name , AVG(duration) as duration , COUNT(duration) as num_trips FROM `bigquery-public-data`.london_bicycles.cycle_hire GROUP BY start_station_name ORDER BY num_trips DESC LIMIT 5"
}
response = service.jobs().query(projectId=PROJECT, body=request).execute()
print(response)

jobId = response['jobReference']['jobId']
print(jobId)

# get query results
while (not response['jobComplete']):
  response = service.jobs().getQueryResults(projectId=PROJECT,
                                            jobId=jobId,
                                            maxResults=2,
                                            timeoutMs=5).execute()

while (True):
  # print responses
  for row in response['rows']:
    print(row['f'][0]['v']) # station name
  print('--' * 5)
  # page through responses
  if 'pageToken' in response:
    pageToken = response['pageToken']
    # get next page
    response = service.jobs().getQueryResults(projectId=PROJECT,
                                              jobId=jobId,
                                              maxResults=2,
                                              pageToken=pageToken,
                                              timeoutMs=5).execute()
  else:
    break


I am working on the file "05_devel/google_api_client.ipynb".
The first task is:
Get info about a dataset
Can you write Python code for it?

# information about the ch04 dataset
dsinfo = service.datasets().get(datasetId="ch04", projectId=PROJECT).execute()
for info in dsinfo.items():
  print(info)


I am working on the file "05_devel/magics.ipynb".
The first task is:
Into a dataframe
Can you write Python code for it?

%%bigquery df --project $PROJECT
SELECT
  start_station_name
  , AVG(duration) as duration
  , COUNT(duration) as num_trips
FROM `bigquery-public-data`.london_bicycles.cycle_hire
GROUP BY start_station_name
ORDER BY num_trips DESC

df.describe()

df.plot.scatter('duration', 'num_trips');


I am working on the file "05_devel/pandas.ipynb".
The first task is:
Indeed, we see that higher fraction_bad are associated with lower num_trips.  We need to use the fraction_bad carefully.
Banding
Let's pick the 5 worst of the really busy stations, 5 of the next most busy, etc.
We can do this by creating
4 different bands from the quantile of the station by num_trips, and within each band, finding the 5 stations that are most bad.
Can you write Python code for it?

stations_to_examine = []

import pandas as pd
for band in range(1,5):
  min_trips = badtrips['num_trips'].quantile(0.2*(band))
  max_trips = badtrips['num_trips'].quantile(0.2*(band+1))
  query = 'num_trips >= {} and num_trips < {}'.format(min_trips, max_trips)
  print(query) # band
  stations = badtrips.query(query)
  stations = stations.sort_values(by=['fraction_bad'], ascending=False)[:5]
  print(stations) # 5 worst
  stations_to_examine.append(stations)
  print()


I am working on the file "05_devel/pandas.ipynb".
The first task is:
Looks like the fraction_bad ranges from 0 to 0.4, but is not clear how relevant this ratio is because the stations also vary quite dramatically. For example, the number of trips ranges from 11 to 95740.  A 0.4 fraction_bad is not terrible if num_trips=11.
Let's look at a scatter plot to see if there is any clear trend here.
Can you write Python code for it?


I am working on the file "05_devel/pandas.ipynb".
The first task is:
Problem
We believe that if someone rents a bicycle for less than 10 minutes and returns the bicycle to the same station that they rented it at, it is likely that the bicycle has a problem. We'll call this a "bad" trip.
We want to send a crew out to examine a few of the stations that had lots of bad trips to see if there are any systemic problems.
Find stations with problem bikes
As a first step, let's find which stations had the most "bad" trips in 2015
Can you write Python code for it?

%%bigquery badtrips --project $PROJECT
SELECT *, bad_trips / num_trips AS fraction_bad FROM (
    SELECT
      start_station_name
      , SUM(IF(duration < 600 AND start_station_name = end_station_name, 1, 0)) AS bad_trips
      , COUNT(start_station_name) as num_trips
    FROM `bigquery-public-data`.london_bicycles.cycle_hire
    WHERE EXTRACT(YEAR FROM start_date) = 2015
    GROUP BY start_station_name
    HAVING num_trips > 10
)
ORDER BY fraction_bad DESC


I am working on the file "05_devel/pandas.ipynb".
The first task is:
Statistics
Let's look at the statistics of the dataset to learn what kind of threshold of the fraction_bad is appropriate
Can you write Python code for it?


I am working on the file "05_devel/pandas.ipynb".
The first task is:
Write the stations to a new table in BigQuery
Can you write Python code for it?

!pip install pyarrow

from google.cloud import bigquery
bq = bigquery.Client(project=PROJECT)

table_id = '{}.ch05eu.bad_bikes'.format(PROJECT)
job = bq.load_table_from_dataframe(stations_to_examine, table_id)
job.result() # blocks and waits
print("Loaded {} rows into {}".format(job.output_rows, table_id))


I am working on the file "05_devel/statfit.ipynb".
The first task is:
After doing the pip install (above), please restart your kernel.
Fit rides at a single station to a Gamma distribution
Can you write Python code for it?

# CHANGE THIS to try this notebook out
PROJECT='cloud-training-demos'
# London Bicycles is in EU, so process it in EU and make sure bucket is in EU
BUCKET='cloud-training-demos-eu'
REGION='europe-west1'


I am working on the file "05_devel/statfit.ipynb".
The first task is:
Fit to a Gamma distribution
Can you write Python code for it?

from scipy import stats
ag,bg,cg = stats.gamma.fit(df['duration'])
print(ag, bg, cg)


I am working on the file "05_devel/statfit.ipynb".
The first task is:
Let's plot the distribution of these rides
Can you write Python code for it?

ax = df.plot.hist(range=[0, 3000], bins=20);


I am working on the file "05_devel/statfit.ipynb".
The first task is:
The Beam/Dataflow pipeline to do this at scale
Can you write Python code for it?

%%writefile requirements.txt
numpy
scipy

# output dataset
!bq --location=EU mk --dataset ch05eu

import apache_beam as beam
import logging
import datetime, os

def compute_fit(row):
  from scipy import stats
  import numpy as np
  durations = row['duration_array']
  ag, bg, cg = stats.gamma.fit(durations)
  if np.isfinite(ag) and np.isfinite(bg) and np.isfinite(cg):
      result = {}
      result['station_id'] = str(row['start_station_id'])
      result['ag'] = ag
      result['bg'] = bg
      result['cg'] = cg
      yield result

def run_job(in_test_mode=True):
  import shutil, os, subprocess
  job_name = 'computestats' + '-' + datetime.datetime.now().strftime('%y%m%d-%H%M%S')

  if in_test_mode:
      print('Launching local job ... hang on')
      OUTPUT_DIR = './station_stats'
      shutil.rmtree(OUTPUT_DIR, ignore_errors=True)
      os.makedirs(OUTPUT_DIR)
  else:
      print('Launching Dataflow job {} ... hang on'.format(job_name))
      OUTPUT_DIR = 'gs://{0}/station_stats'.format(BUCKET)
      try:
        subprocess.check_call('gsutil -m rm -r {}'.format(OUTPUT_DIR).split())
      except:
        pass

  options = {
      'staging_location': os.path.join(OUTPUT_DIR, 'tmp', 'staging'),
      'temp_location': os.path.join(OUTPUT_DIR, 'tmp'),
      'job_name': job_name,
      'project': PROJECT,
      'region': REGION,
      'teardown_policy': 'TEARDOWN_ALWAYS',
      'no_save_main_session': True,
      'requirements_file': 'requirements.txt'
  }
  opts = beam.pipeline.PipelineOptions(flags = [], **options)
  if in_test_mode:
      RUNNER = 'DirectRunner'
  else:
      RUNNER = 'DataflowRunner'

  if in_test_mode:
    query = """
      SELECT start_station_id, ARRAY_AGG(duration) AS duration_array
      FROM `bigquery-public-data.london_bicycles.cycle_hire`
      WHERE start_station_id BETWEEN 700 AND 710
      GROUP BY start_station_id
      """
  else:
     query = """
      SELECT start_station_id, ARRAY_AGG(duration) AS duration_array
      FROM...


I am working on the file "05_devel/statfit.ipynb".
The first task is:
Verify output table
Can you write Python code for it?

%%bigquery stations
SELECT * from ch05eu.station_stats

stats = stations.describe()
stats

stations.plot.hist('ag', range=[stats['ag']['25%'], stats['ag']['75%']]);

stations.plot.hist('bg', range=[stats['bg']['25%'], stats['bg']['75%']]);

stations.plot.hist('cg', range=[stats['cg']['25%'], stats['cg']['75%']]);

ax = stations.plot.scatter('ag', 'cg');
ax.set_xlim(stats['ag']['25%'], stats['ag']['75%'])
ax.set_ylim(stats['cg']['25%'], stats['cg']['75%'])


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
All is together to instantiate the derivatives_portfolio class.
Can you write Python code for it?

port = derivatives_portfolio(name='portfolio',
                             positions=positions,
                             val_env=val_env,
                             risk_factors=risk_factors,
                             correlations=correlations)


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
Assumptions for the geometric_brownian_motion object.
Can you write Python code for it?

# geometric Brownian motion
me_gbm.add_constant('initial_value', 36.)
me_gbm.add_constant('volatility', 0.2)
me_gbm.add_constant('currency', 'EUR')
me_gbm.add_constant('model', 'gbm')


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
Assumptions for the jump_diffusion object.
Can you write Python code for it?

# jump diffusion
me_jd.add_constant('initial_value', 36.)
me_jd.add_constant('volatility', 0.2)
me_jd.add_constant('lambda', 0.5)
    # probability for jump p.a.
me_jd.add_constant('mu', -0.75)
    # expected jump size [%]
me_jd.add_constant('delta', 0.1)
    # volatility of jump
me_jd.add_constant('currency', 'EUR')
me_jd.add_constant('model', 'jd')


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
Assumptions for the stochastic_volatility object.
Can you write Python code for it?

# stochastic volatility model
me_sv.add_constant('initial_value', 36.)
me_sv.add_constant('volatility', 0.2)
me_sv.add_constant('vol_vol', 0.1)
me_sv.add_constant('kappa', 2.5)
me_sv.add_constant('theta', 0.4)
me_sv.add_constant('rho', -0.5)
me_sv.add_constant('currency', 'EUR')
me_sv.add_constant('model', 'sv')


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
Derivatives
In this sub-section, we model the single derivatives instruments.
American Put Option
The first derivative instrument is an American put option.
Can you write Python code for it?

gbm = geometric_brownian_motion('gbm_obj', me_gbm)

me_put = market_environment('put', dt.datetime(2015, 1, 1))
me_put.add_constant('maturity', dt.datetime(2015, 12, 31))
me_put.add_constant('strike', 40.)
me_put.add_constant('currency', 'EUR')
me_put.add_environment(val_env)

am_put = valuation_mcs_american_single('am_put', mar_env=me_put, underlying=gbm,
                       payoff_func='np.maximum(strike - instrument_values, 0)')

am_put.present_value(fixed_seed=True, bf=5)


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
European Maximum Call on 2 Assets
The second derivative instrument is a European maximum call option on two risk factors.
Can you write Python code for it?

jd = jump_diffusion('jd_obj', me_jd)

me_max_call = market_environment('put', dt.datetime(2015, 1, 1))
me_max_call.add_constant('maturity', dt.datetime(2015, 9, 15))
me_max_call.add_constant('currency', 'EUR')
me_max_call.add_environment(val_env)

payoff_call = "np.maximum(np.maximum(maturity_value['gbm'], maturity_value['jd']) - 34., 0)"

assets = {'gbm' : me_gbm, 'jd' : me_jd}
asset_corr = [correlations[0]]

asset_corr

max_call = valuation_mcs_european_multi('max_call', me_max_call, assets, asset_corr,
        payoff_func=payoff_call)

max_call.present_value(fixed_seed=False)

max_call.delta('jd')

max_call.delta('gbm')


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
Finally, a graphical look at two selected, simulated paths of the stochastic volatility risk factor and the jump diffusion risk factor, respectively.
Can you write Python code for it?

path_no = 1
paths1 = port.underlying_objects['sv'].get_instrument_values()[:, path_no]
paths2 = port.underlying_objects['jd'].get_instrument_values()[:, path_no]

paths1

paths2


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
Finally, the market model with the risk factors and the correlations between them.
Can you write Python code for it?

risk_factors = {'gbm' : me_gbm, 'jd' : me_jd, 'sv' : me_sv}
correlations = [['gbm', 'jd', 0.66], ['jd', 'sv', -0.75]]


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
Finally, the unifying valuation assumption for the valuation environment.
Can you write Python code for it?

# valuation environment
val_env = market_environment('val_env', dt.datetime(2015, 1, 1))
val_env.add_constant('paths', 10000)
val_env.add_constant('frequency', 'W')
val_env.add_curve('discount_curve', r)
val_env.add_constant('starting_date', dt.datetime(2015, 1, 1))
val_env.add_constant('final_date', dt.datetime(2015, 12, 31))


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
Let us have a look at the major portfolio statistics.
Can you write Python code for it?

%time stats = port.get_statistics()
stats

stats['pos_value'].sum()


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
Portfolio
To compose a derivatives portfolio, derivatives_position objects are needed.
Can you write Python code for it?

am_put_pos = derivatives_position(
                    name='am_put_pos',
                    quantity=2,
                    underlyings=['gbm'],
                    mar_env=me_put,
                    otype='American single',
                    payoff_func='np.maximum(instrument_values - 36., 0)')

max_call_pos = derivatives_position(
                    'max_call_pos', 3, ['gbm', 'jd'],
                    me_max_call, 'European multi',
                    payoff_call)

min_put_pos = derivatives_position(
                    'min_put_pos', 5, ['sv', 'jd'],
                    me_min_put, 'American multi',
                    payoff_put)


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
Risk Factors
This sub-section models the single risk factors. We start with definition of the risk-neutral discounting object.
Can you write Python code for it?

# constant short rate
r = constant_short_rate('r', 0.02)


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
The resulting plot illustrates the strong negative correlation.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline
plt.figure(figsize=(10, 6))
plt.plot(port.time_grid, paths1, 'r', label='sv')
plt.plot(port.time_grid, paths2, 'b', label='jd')
plt.gcf().autofmt_xdate()
plt.legend(loc=0); plt.grid(True)
# negatively correlated underlyings


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
These are added to the single market_environment objects of the risk factors.
Can you write Python code for it?

# add valuation environment to market environments
me_gbm.add_environment(val_env)
me_jd.add_environment(val_env)
me_sv.add_environment(val_env)


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
These objects are to be collected in dictionary objects.
Can you write Python code for it?

positions = {'am_put_pos' : am_put_pos, 'max_call_pos' : max_call_pos,
             'min_put_pos' : min_put_pos}


I am working on the file "05_dx_portfolio_multi_risk.ipynb".
The first task is:
Three risk factors ares modeled:

geometric Brownian motion
jump diffusion
stochastic volatility process
Can you write Python code for it?

# market environments
me_gbm = market_environment('gbm', dt.datetime(2015, 1, 1))
me_jd = market_environment('jd', dt.datetime(2015, 1, 1))
me_sv = market_environment('sv', dt.datetime(2015, 1, 1))


I am working on the file "05_hypothesis_testing.ipynb".
The first task is:
Comparison of means
for this purpose a new function is defined - draw permutation replicates, that uses permutation samples function inside:
def draw_perm_reps(data_1, data_2, func, size=1):
    """Generate multiple permutation replicates."""

    # Initialize array of replicates: perm_replicates
    perm_replicates = np.empty(size)

    for i in range(size):
        # Generate permutation sample
        perm_sample_1, perm_sample_2 = permutation_sample(data_1, data_2)

        # Compute the test statistic
        perm_replicates[i] = func(perm_sample_1, perm_sample_2)

    return perm_replicates

For this example difference of means is looked at
Can you write Python code for it?

# definition of the difference of means
def diff_of_means(data_1, data_2):
    """Difference in means of two arrays."""

    # The difference of means of data_1, data_2: diff
    diff = np.mean(data_1) - np.mean(data_2)

    return diff

# empirical difference of means
empirical_diff_means = diff_of_means(sepalLengthSetosa, sepalLengthVirginica)
empirical_diff_means

# permutation replicates
perm_replicates = draw_perm_reps(sepalLengthSetosa, sepalLengthVirginica, diff_of_means, size=10000)

# computing p-value
p = np.sum(perm_replicates <= empirical_diff_means) / len(perm_replicates)
p


I am working on the file "05_hypothesis_testing.ipynb".
The first task is:
Samples permutation
Special function is defined to mix the data (that should be very similar):
def permutation_sample(data1, data2):
    """Generate a permutation sample from two data sets."""

    # Concatenate the data sets: data
    data = np.concatenate((data1, data2))

    # Permute the concatenated array: permuted_data
    permuted_data = np.random.permutation(data)

    # Split the permuted array into two: perm_sample_1, perm_sample_2
    perm_sample_1 = permuted_data[:len(data1)]
    perm_sample_2 = permuted_data[len(data1):]

    return perm_sample_1, perm_sample_2
Can you write Python code for it?

(np.mean(sepalLengthSetosa), np.mean(sepalLengthVirginica))

for i in range(50):
    # Generate permutation samples
    perm_sample_1, perm_sample_2 = permutation_sample(sepalLengthSetosa, sepalLengthVirginica)

    # Compute ECDFs
    x_1, y_1 = ecdf(perm_sample_1)
    x_2, y_2 = ecdf(perm_sample_2)

    # Plot ECDFs of permutation sample
    _ = plt.plot(x_1, y_1, marker='.', linestyle='none',
                 color='red', alpha=0.02)
    _ = plt.plot(x_2, y_2, marker='.', linestyle='none',
                 color='blue', alpha=0.02)

# Create and plot ECDFs from original data
x_1, y_1 = ecdf(sepalLengthSetosa)
x_2, y_2 = ecdf(sepalLengthVirginica)
_ = plt.plot(x_1, y_1, marker='.', linestyle='none', color='red', label='Sepal Length - setosa')
_ = plt.plot(x_2, y_2, marker='.', linestyle='none', color='blue', label='Sepal Length - virginica')

# Label axes, set margin, and show plot
plt.margins(0.02)
_ = plt.xlabel('sepal lengths and widths [cm]')
_ = plt.ylabel('ECDF')
plt.legend();


I am working on the file "05_memmapping.ipynb".
The first task is:
Before allocating more data let us define a couple of utility functions from the previous exercise (and more) to monitor what is used by which engine and what is still free on the cluster as a whole:

Let's allocate a 80MB memmap array:
Can you write Python code for it?

np.memmap('bigger_test.mmap', shape=10 * int(1e6), dtype=np.float64, mode='w+')


I am working on the file "05_memmapping.ipynb".
The first task is:
Let's feed the array with random values, one column at a time because our system's memory is limited!
Can you write Python code for it?

for i in range(ncols):
    f[:, i] = np.random.rand(nrows)


I am working on the file "05_memmapping.ipynb".
The first task is:
Let's trigger an actual load of the data from the drive into the in-memory disk cache of the OS, this can take some time depending on the speed of the hard drive (on the order of 100MB/s to 300MB/s hence 3s to 8s for this dataset):
Can you write Python code for it?


I am working on the file "05_memmapping.ipynb".
The first task is:
Memmap arrays generally behave very much like regular in-memory numpy arrays:
Can you write Python code for it?

print(mm_r.sum())
print("sum={0}, mean={1}, std={2}".format(mm_r.sum(),
                                          np.mean(mm_r), np.std(mm_r)))


I am working on the file "05_memmapping.ipynb".
The first task is:
No significant memory was used in this operation as we just asked the OS to allocate the buffer on the hard drive and just maitain a virtual memory area as a cheap reference to this buffer.
Let's open new references to the same buffer from all the engines at once:
Can you write Python code for it?

%time big_mmap = np.memmap('bigger_test.mmap', dtype=np.float64, mode='r+')

big_mmap


I am working on the file "05_memmapping.ipynb".
The first task is:
Now back into memory
Can you write Python code for it?


I am working on the file "05_memmapping.ipynb".
The first task is:
Now, we flush memory changes to disk by deleting the object:
Can you write Python code for it?


I am working on the file "05_memmapping.ipynb".
The first task is:
Reading a memory-mapped array from disk involves the same memmap() function. The data type and the shape need to be specified again, as this information is not stored in the file:
Can you write Python code for it?

f = np.memmap('memmapped.dat', dtype=np.float32,
                  shape=(nrows, ncols))

np.array_equal(f[:, -1], x)

del f


I am working on the file "05_memmapping.ipynb".
The first task is:
Save the last column of the Array
Can you write Python code for it?


I am working on the file "05_memmapping.ipynb".
The first task is:
This binary file can then be mapped as a new numpy array by all the engines having access to the same filesystem.
The mode='r+' opens this shared memory area in read write mode:
Can you write Python code for it?

mm_r = np.memmap('small_test.mmap', dtype=np.float32, mode='r+')
print(mm_r)

mm_w[0] = 42
print(mm_w)

print(mm_r)


I am working on the file "05_model_evaluation.ipynb".
The first task is:
Can we locate an even better value for K?
Can you write Python code for it?

# try K=1 through K=25 and record testing accuracy
k_range = list(range(1, 26))
scores = []
for k in k_range:
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train, y_train)
    y_pred = knn.predict(X_test)
    scores.append(metrics.accuracy_score(y_test, y_pred))

# import Matplotlib (scientific plotting library)
import matplotlib.pyplot as plt

# allow plots to appear within the notebook
%matplotlib inline

# plot the relationship between K and testing accuracy
plt.plot(k_range, scores)
plt.xlabel('Value of K for KNN')
plt.ylabel('Testing Accuracy')


I am working on the file "05_model_evaluation.ipynb".
The first task is:
KNN (K=1)
Can you write Python code for it?

knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X, y)
y_pred = knn.predict(X)
print(metrics.accuracy_score(y, y_pred))


I am working on the file "05_model_evaluation.ipynb".
The first task is:
Logistic regression
Can you write Python code for it?

# import the class
from sklearn.linear_model import LogisticRegression

# instantiate the model (using the default parameters)
logreg = LogisticRegression()

# fit the model with data
logreg.fit(X, y)

# predict the response values for the observations in X
logreg.predict(X)

# store the predicted response values
y_pred = logreg.predict(X)

# check how many predictions were generated
len(y_pred)


I am working on the file "05_model_evaluation.ipynb".
The first task is:
Problems with training and testing on the same data

Goal is to estimate likely performance of a model on out-of-sample data
But, maximizing training accuracy rewards overly complex models that won't necessarily generalize
Unnecessarily complex models overfit the training data

Image Credit: Overfitting by Chabacano. Licensed under GFDL via Wikimedia Commons.
Evaluation procedure #2: Train/test split

Split the dataset into two pieces: a training set and a testing set.
Train the model on the training set.
Test the model on the testing set, and evaluate how well we did.
Can you write Python code for it?

# print the shapes of X and y
print(X.shape)
print(y.shape)

# STEP 1: split X and y into training and testing sets
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=4)


I am working on the file "05_model_evaluation.ipynb".
The first task is:
Repeat for KNN with K=5:
Can you write Python code for it?

knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train, y_train)
y_pred = knn.predict(X_test)
print(metrics.accuracy_score(y_test, y_pred))


I am working on the file "05_model_evaluation.ipynb".
The first task is:
What did this accomplish?

Model can be trained and tested on different data
Response values are known for the testing set, and thus predictions can be evaluated
Testing accuracy is a better estimate than training accuracy of out-of-sample performance
Can you write Python code for it?

# print the shapes of the new X objects
print(X_train.shape)
print(X_test.shape)

# print the shapes of the new y objects
print(y_train.shape)
print(y_test.shape)

# STEP 2: train the model on the training set
logreg = LogisticRegression()
logreg.fit(X_train, y_train)

# STEP 3: make predictions on the testing set
y_pred = logreg.predict(X_test)

# compare actual response values (y_test) with predicted response values (y_pred)
print(metrics.accuracy_score(y_test, y_pred))


I am working on the file "05_neural_networks/src/py_part_5_MNIST_DNN.ipynb".
The first task is:
Import data
Can you write Python code for it?

# load clean data
path = '../data/'

# define input variable measurement levels
# strings automatically parsed as enums (nominal)
# numbers automatically parsed as numeric
col_types = {'label': 'enum'}

train = h2o.import_file(path=path + 'train.csv', col_types=col_types) # multi-threaded import
test = h2o.import_file(path=path + 'test.csv')

train.describe()

# split into 40% training, 30% validation, and 30% test
train, valid = train.split_frame([0.8])

# assign target and inputs
y = 'label'
X = [name for name in train.columns if name != y]
print(y)
print(X)

# set target to factor - for multinomial classification
train[y] = train[y].asfactor()
valid[y] = valid[y].asfactor()


I am working on the file "05_neural_networks/src/py_part_5_MNIST_autoencoder.ipynb".
The first task is:
Extract and display deep features
Can you write Python code for it?

deep_features = sdae.deepfeatures(train, 2)
deep_features_pandas = deep_features.as_data_frame()
deep_features_pandas['label'] = train['label'].as_data_frame()
deep_features_pandas.head()


I am working on the file "05_neural_networks/src/py_part_5_MNIST_autoencoder.ipynb".
The first task is:
Import training data and assign all pixel columns to be inputs
Can you write Python code for it?

train = h2o.import_file('../data/train.csv')
X = [name for name in train.columns if name != 'label']


I am working on the file "05_neural_networks/src/py_part_5_MNIST_autoencoder.ipynb".
The first task is:
Show most anomolous 1
Can you write Python code for it?

ones = train[train['label'] == 1].as_data_frame().as_matrix()
pixels = ones[top, 1:]
pixels = np.array(pixels, dtype='uint8')
pixels = pixels.reshape((28, 28))

plt.imshow(pixels, cmap='gray')
plt.show()


I am working on the file "05_neural_networks/src/py_part_5_MNIST_autoencoder.ipynb".
The first task is:
Train five level stacked denoising autoencoder model
Can you write Python code for it?

sdae = H2ODeepLearningEstimator(
    epochs=3,
    hidden=[250, 50, 2, 50, 250],                          # 5 layers, 2 in the middle for viz
    activation='rectifier',
    l2=0.2,                                                # L2 for numeric stability and can help with plotting rectifier activations
    adaptive_rate=True,
    sparse=True,                                           # handles data w/ many zeros more efficiently
    seed=88888,                                            # enables exact reproducibility
    reproducible=True,                                     # slow
    ignore_const_cols=True,
    autoencoder=True)

sdae.train(x=X, training_frame=train)


I am working on the file "05_neural_networks/src/py_part_5_MNIST_autoencoder.ipynb".
The first task is:
Use reconstruction error to find anomolous 7's
Can you write Python code for it?

# h2o anomoly function calculates row-wise reconstrunction MSE
reconstruction_mse = sdae.anomaly(train[train['label'] == 7])

# use pandas to sort reconstrunction MSE
pandas_reconstruction_mse = reconstruction_mse.as_data_frame().sort_values(by='Reconstruction.MSE', ascending=False)
print(pandas_reconstruction_mse.head())
top = pandas_reconstruction_mse.idxmax().values[0]


I am working on the file "05_neural_networks/src/py_part_5_basic_mlp_example.ipynb".
The first task is:
Analyze results
Can you write Python code for it?

y_yhat_frame = pd.DataFrame(columns = ['y', 'yhat'])
y_yhat_frame['y'] = y.reshape(-1)
y_yhat_frame['yhat'] = output_layer.reshape(-1)
y_yhat_frame

%matplotlib inline
_ = iter_frame.plot(kind='line', x='Iteration', y='Error', title='Iteration Chart')


I am working on the file "05_neural_networks/src/py_part_5_basic_mlp_example.ipynb".
The first task is:
Fetch simple Iris dataset
Can you write Python code for it?

# load and preprocess Iris data set
# easy binomial classification task: seperate Setosa irises from Versicolor irises

# fetch data from UCI repository
url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data'
connection = urllib2.urlopen(url)
raw = connection.read()

# initialize empty X and y arrays
X = np.zeros((100, 4))
y = np.zeros((100, 1))

# load iris data into X and y arrays
row_idx = 0
for line in str(raw)[2:-5].split('\\n'):
    line = line.replace('Iris-setosa', '1').replace('Iris-versicolor', '0')
    line = line.split(',')
    # remove Virginica irises from data set
    if line[-1] != 'Iris-virginica':
        line = np.asarray(line)
        X[row_idx, :] = line[:-1]
        y[row_idx, :] = line[-1]
        row_idx += 1

print('Data inputs:\n', X)
print('\n')
print('Data target:\n', y)


I am working on the file "05_neural_networks/src/py_part_5_basic_mlp_example.ipynb".
The first task is:
Set simple hyperparameters
Can you write Python code for it?

LEARN_RATE = 0.005
ITERATIONS = 600
HIDDEN_UNITS = 30


I am working on the file "05_neural_networks/src/py_part_5_basic_mlp_example.ipynb".
The first task is:
Training routine
Can you write Python code for it?

# very simple MLP routine
# with logistic activation for hidden and output layer

# set random seed
# always do this when working with random numbers
np.random.seed(12345)

# randomly initialize our weights with mean 0
hidden_weights = np.random.random((4, HIDDEN_UNITS)) - 0.5 # 4 X HIDDEN_UNITS weights in hidden layer
output_weights = np.random.random((HIDDEN_UNITS, 1)) - 0.5 # HIDDEN_UNITS X 1 weights in output layer

print('MLP architecture is: 4 input units -> %d hidden units -> 1 output units.' % HIDDEN_UNITS)
print()
print('There are %d hidden weights to optimize.' % (4 * HIDDEN_UNITS))
print('Initial hidden weights:\n', hidden_weights)
print()
print('There are %d output weights to optimize.' % HIDDEN_UNITS)
print('Initial output weights:\n', output_weights)

# initialize empty pandas DataFrame to hold iteration scores
iter_frame = pd.DataFrame(columns=['Iteration', 'Error'])

# activation function
def logistic_activation_function(weights_times_inputs):

    return 1 / (1 + np.exp(-weights_times_inputs))

# trainign loop
for iteration in range(0, ITERATIONS):

    ### feed-forward phase ##########
    # run data through input, hidden, and output layers

    input_layer = X
    hidden_layer = logistic_activation_function(np.dot(input_layer, hidden_weights))
    output_layer = logistic_activation_function(np.dot(hidden_layer, output_weights))

    if iteration == 0:
        print('\nInitial yhat:\n', output_layer)
        print()
        print('Training ...')

    ### evaluate error function ##########
    output_logloss_error = -y * np.log(output_layer) + (1 - y)*np.log(1 - output_layer)
    if ((iteration + 1) % 100) == 0:
        print('Iteration %4i, Error: %5.2f' % (iteration + 1, np.sum(output_logloss_error)))

    # record iteration and error
    iter_frame = iter_frame.append({'Iteration': iteration,
                                    'Error': np.sum(output_logloss_error)},
                                   ignore_index=True)

    ### back-propogation...


I am working on the file "05_particle_in_a_box.ipynb".
The first task is:
Exercise 5: Energy of linear combinations
The expectation value of an operator $O$ on the wavefunction $\Psi$ is defined as:
\begin{equation}
\bar{O} = \frac{\langle\Psi|O|\Psi\rangle}{\langle\Psi|\Psi\rangle} = \langle\Psi|O|\Psi\rangle
\end{equation}
where the last equality holds only if $\Psi$ is normalized.
The energy is the expectation value of the Hamiltonian operator:
\begin{equation}
\bar{H} = E = \langle\Psi|H|\Psi\rangle
\end{equation}
Given $\Psi$ a linear combination of particle in a box eigenfunctions, calculate its respective energy.
Hint: remember that $H\psi_i(x) = E_i\psi_i(x)$ and that the $\psi_i(x)$ are normalizd. You can also assume $\Psi$ to be already normalized.
Write a Python function to calculate the energy of such linear combinations. The function should accept the coefficients $c_i$ of the linear combination, the mass $M$ of the particle and the length $L$ of the box as arguments.
```Python
def energy(coeffs, M, L):
    """ Return energy of a linear combination of 1D particle in a box eigenfunctions.
coeffs -- the coefficients of the linear combination
M -- the mass of the particle
L -- the size of the box
"""

```
Can you write Python code for it?

def energy(coeffs, M, L):
    """ Return energy of a linear combination of 1D particle in a box eigenfunctions.

    coeffs -- the coefficients of the linear combination
    M -- the mass of the particle
    L -- the size of the box
    """
    raise NotImplementedError('You have to write this function')

coeffs = np.array([0.5, 0.5])
energy(coeffs, 1.0, 10.0)


I am working on the file "05_pseudospectral/cheby_derivative.ipynb".
The first task is:
Exercise 1
Define a python function call "get_cheby_matrix(nx)" that initializes the Chebyshev derivative matrix $D_{ij}$
Can you write Python code for it?


I am working on the file "05_pseudospectral/cheby_derivative.ipynb".
The first task is:
Exercise 2
Calculate the numerical derivative by applying the differentiation matrix $D_{ij}$. Define an arbitrary function (e.g. a Gaussian) and initialize its analytical derivative on the Chebyshev collocation points. Calculate the numerical derivative and the difference to the analytical solution. Vary the wavenumber content of the analytical function. Does it make a difference? Why is the numerical result not entirely exact?
Can you write Python code for it?


I am working on the file "05_pseudospectral/cheby_derivative.ipynb".
The first task is:
Exercise 3
Now that the numerical derivative is available, we can visually inspect our results. Make a plot of both, the analytical and numerical derivatives together with the difference error.
Can you write Python code for it?


I am working on the file "05_pseudospectral/cheby_derivative_solution.ipynb".
The first task is:
Exercise 1
Define a python function call "get_cheby_matrix(nx)" that initializes the Chebyshev derivative matrix $D_{ij}$
Can you write Python code for it?

# Function for setting up the Chebyshev derivative matrix
def get_cheby_matrix(nx):
    cx = np.zeros(nx+1)
    x = np.zeros(nx+1)
    for ix in range(0,nx+1):
        x[ix] = np.cos(np.pi * ix / nx)

    cx[0] = 2.
    cx[nx] = 2.
    cx[1:nx] = 1.

    D = np.zeros((nx+1,nx+1))
    for i in range(0, nx+1):
        for j in range(0, nx+1):
            if i==j and i!=0 and i!=nx:
                D[i,i]=-x[i]/(2.0*(1.0-x[i]*x[i]))
            else:
                D[i,j]=(cx[i]*(-1)**(i+j))/(cx[j]*(x[i]-x[j]))

    D[0,0] = (2.*nx**2+1.)/6.
    D[nx,nx] = -D[0,0]
    return D


I am working on the file "05_pseudospectral/cheby_derivative_solution.ipynb".
The first task is:
Exercise 2
Calculate the numerical derivative by applying the differentiation matrix $D_{ij}$. Define an arbitrary function (e.g. a Gaussian) and initialize its analytical derivative on the Chebyshev collocation points. Calculate the numerical derivative and the difference to the analytical solution. Vary the wavenumber content of the analytical function. Does it make a difference? Why is the numerical result not entirely exact?
Can you write Python code for it?

# Initialize arbitrary test function on Chebyshev collocation points
nx = 200     # Number of grid points
x = np.zeros(nx+1)
for ix in range(0,nx+1):
    x[ix] = np.cos(ix * np.pi / nx)
dxmin = min(abs(np.diff(x)))
dxmax = max(abs(np.diff(x)))

# Function example: Gaussian
# Width of Gaussian
s = .2
# Gaussian function (modify!)
f = np.exp(-1/s**2 * x**2)

# Initialize differentiation matrix
D = get_cheby_matrix(nx)

# Analytical derivative
df_ana = -2/s**2 * x * np.exp(-1/s**2 * x**2)

# Calculate numerical derivative using differentiation matrix
df_num = D @ f

# To make the error visible, it is multiply by 10^12
df_err = 1e12*(df_ana - df_num)

# Calculate error between analytical and numerical solution
err = np.sum((df_num - df_ana)**2) / np.sum(df_ana**2) * 100
print('Error: %s' %err)


I am working on the file "05_pseudospectral/cheby_elastic_1d.ipynb".
The first task is:
1. Chebyshev derivative method
Exercise
Define a python function call "get_cheby_matrix(nx)" that initializes the Chebyshev derivative matrix $D_{ij}$, call this function and display the Chebyshev derivative matrix.
Can you write Python code for it?


I am working on the file "05_pseudospectral/cheby_elastic_1d.ipynb".
The first task is:
2. Initialization of setup
Can you write Python code for it?

# Basic parameters
# ---------------------------------------------------------------
#nt = 5000        # number of time steps
tmax  = 0.0006
eps   = 1.4       # stability limit
isx   = 100
lw    = 0.7
ft    = 10
f0    = 100000    # dominant frequency
iplot = 20        # Snapshot frequency

# material parameters
rho = 2500.
c   = 3000.
mu  = rho*c**2

# space domain
nx = 100     # number of grid points in x 199
xs = np.floor(nx/2)      # source location
xr = np.floor(nx*0.8)
x  = np.zeros(nx+1)

# initialization of pressure fields
p = np.zeros(nx+1)
pnew = np.zeros(nx+1)
pold = np.zeros(nx+1)
d2p  = np.zeros(nx+1)

for ix in range(0,nx+1):
    x[ix] = np.cos(ix * np.pi / nx)
dxmin = min(abs(np.diff(x)))
dxmax = max(abs(np.diff(x)))

dt = eps*dxmin/c    # calculate time step from stability criterion
nt = int(round(tmax/dt))


I am working on the file "05_pseudospectral/cheby_elastic_1d.ipynb".
The first task is:
3. Source Initialization
Can you write Python code for it?


I am working on the file "05_pseudospectral/cheby_elastic_1d.ipynb".
The first task is:
4. Time Extrapolation
Now we time extrapolate using the previously defined get_cheby_matrix(nx) method to call the differentiation matrix. The discrete values of the numerical simulation are indicated by dots in the animation, they represent the Chebyshev collocation points. Observe how the wavefield near the domain center is less dense than towards the boundaries.
Can you write Python code for it?

# Initialize animated plot
# ---------------------------------------------------------------
plt.figure(figsize=(10,6))
line = plt.plot(x, p, 'k.', lw=2)
plt.title('Chebyshev Method - 1D Elastic wave', size=16)
plt.xlabel(' x(m)', size=14)
plt.ylabel(' Amplitude ', size=14)

plt.ion() # set interective mode
plt.show()
# ---------------------------------------------------------------
# Time extrapolation
# ---------------------------------------------------------------
# Differentiation matrix
D = get_cheby_matrix(nx)
for it in range(nt):
    # Space derivatives
    dp = np.dot(D, p.T)
    dp = mu/rho * dp
    dp = D @ dp

    # Time extrapolation
    pnew = 2*p - pold + np.transpose(dp) * dt**2

    # Source injection
    pnew = pnew + sg*src[it]*dt**2/rho

    # Remapping
    pold, p = p, pnew
    p[0] = 0; p[nx] = 0 # set boundaries pressure free

    # --------------------------------------
    # Animation plot. Display solution
    if not it % iplot:
        for l in line:
            l.remove()
            del l

        # --------------------------------------
        # Display lines
        line = plt.plot(x, p, 'k.', lw=1.5)
        plt.gcf().canvas.draw()


I am working on the file "05_pseudospectral/cheby_elastic_1d_solution.ipynb".
The first task is:
1. Chebyshev derivative method
Exercise
Define a python function call "get_cheby_matrix(nx)" that initializes the Chebyshev derivative matrix $D_{ij}$, call this function and display the Chebyshev derivative matrix.
Can you write Python code for it?


I am working on the file "05_pseudospectral/cheby_elastic_1d_solution.ipynb".
The first task is:
3. Source Initialization
Can you write Python code for it?


I am working on the file "05_pseudospectral/cheby_elastic_1d_solution.ipynb".
The first task is:
4. Time Extrapolation
Now we time extrapolate using the previously defined get_cheby_matrix(nx) method to call the differentiation matrix. The discrete values of the numerical simulation are indicated by dots in the animation, they represent the Chebyshev collocation points. Observe how the wavefield near the domain center is less dense than towards the boundaries.
Can you write Python code for it?


I am working on the file "05_pseudospectral/fourier_acoustic_1d.ipynb".
The first task is:
1. Fourier derivative method
The second spatial derivative is computed by multiplying the spatial Fourier transform of the pressure field $P(k,t)$ with $ (ik)^2 $
\begin{equation}
\partial_x^2 p(x,t) = \mathscr{F}^{-1}[(ik)^{2}P(k,t)] = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} (ik)^{2} P(k,t) e^{ikx} dk
\end{equation}
where $k$ is the wavenumber and $IFT$ the Inverse Fourier Transform. A function to perform this task is implemented in the next cell.
Can you write Python code for it?

def fourier_derivative_2nd(f, dx):
    # Length of vector f
    nx = np.size(f)
    # Initialize k vector up to Nyquist wavenumber
    kmax = np.pi / dx
    dk = kmax / (nx / 2)
    k = np.arange(float(nx))
    k[: int(nx/2)] = k[: int(nx/2)] * dk
    k[int(nx/2) :] = k[: int(nx/2)] - kmax

    # Fourier derivative
    ff = np.fft.fft(f)
    ff = (1j*k)**2 * ff
    df_num = np.real(np.fft.ifft(ff))
    return df_num


I am working on the file "05_pseudospectral/fourier_acoustic_1d.ipynb".
The first task is:
2. Initialization of setup
Can you write Python code for it?

# Basic parameters
# ---------------------------------------------------------------
nt    = 3500    # number of time steps
c     = 343.    # acoustic velocity [m/s]
eps   = 0.2     # stability limit
isnap = 50      # snapshot frequency
isx   = 1250    # source location
f0    = 60.     # Frequency [Hz](div by 5)
nx    = 2024    # number of grid points in x

# pressure fields Initialization
p = np.zeros(nx) ; pnew = p ; pold = p ; d2p = p; dp = p
ap = np.zeros(nx); apnew = ap; apold = ap; ad2p = ap; adp = ap
sp = np.zeros(nx); spnew = sp; spold = sp; sd2p = sp; sdp = p

dx = 1250./(nx-1)  # calculate space increment
x  = np.arange(0, nx)*dx   # initialize space coordinates
dt = eps*dx/c; # calculate time step from stability criterion


I am working on the file "05_pseudospectral/fourier_acoustic_1d.ipynb".
The first task is:
4. Time Extrapolation
The final solution for our 1D acoustic wave problem after introducing a finite differences time extrapolation schem can be written as
\begin{equation}
p_{j}^{n+1} = dt^2c_{j}^{2} \partial_{x}^{2}p_{j}^{n} + dt^2s_{j}^{n} + 2p_{j}^{n} - p_{j}^{n-1}
\end{equation}
where the space derivative is computed with the Fourier method. In order to compare the above numerical solution, we implement a 3-point finite difference operator, as well as a 5-point finite difference operator to compute spatial derivatives. They are given as:
1.) 3-point finite difference operator
\begin{equation}
\partial_x^2 p(x,t) = \frac{p(x+\mathrm{d}x,t) - 2 p(x,t) + p(x-\mathrm{d}x,t)}{\mathrm{d}x^2}
\end{equation}
2.) 5-point finite difference operator
\begin{equation}
\partial_x^2 p(x,t) = \frac{-p(x+2\mathrm{d}x,t) + 16p(x+\mathrm{d}x,t) - 30p(x,t) + 16p(x-\mathrm{d}x,t) - p(x-2\mathrm{d}x,t)}{12\mathrm{d}x^2}
\end{equation}
Numerical dispersion
One of the most prominent characteristic of the Fourier method is the low numerical dispersion in comparison with the finite difference method. The animation displayed below compare the effects of numerical dispersion on the solution of the 1D acoustic equation using our three different approaches.
Can you write Python code for it?


I am working on the file "05_pseudospectral/fourier_acoustic_2d.ipynb".
The first task is:
2. Initialization of setup
Can you write Python code for it?

# Basic parameters
# ---------------------------------------------------------------
nt    = 600     # number of time steps
nx    = 256     # number of grid points in x
nz    = nx      # number of grid points in z
c     = 343.    # acoustic velocity
eps   = .2      # stability limit
isnap = 600     # snapshot frequency
isx   = int(nx/2)   # source location
isz   = int(nz/2)
f0    = 200.    # Frequency (div by 5)
xmax  = 200
iplot = 20

# initialization of pressure fields
ap = np.zeros((nx, nz), dtype=float)
apnew = np.zeros((nx, nz), dtype=float)
apold = np.zeros((nx, nz), dtype=float)
ad2px = np.zeros((nx, nz), dtype=float)
ad2pz = np.zeros((nx, nz), dtype=float)

sp = np.zeros((nx, nz), dtype= float)
spnew = np.zeros((nx, nz), dtype=float)
spold = np.zeros((nx, nz), dtype=float)
sd2px = np.zeros((nx, nz), dtype=float)
sd2pz = np.zeros((nx, nz), dtype=float);

sp_sec = -np.abs(sp[1:int(nx/2), 1:int(nz/2)])
ap_sec = -np.abs(ap[int(nx/2):nx, 1:int(nz/2)].T)

dx = xmax/(nx-1)  # calculate space increment
x  = np.arange(0, nx)*dx # initialize space coordinates
z  = np.arange(0, nx)*dx # initialize space coordinates
dt = eps*dx/c # calculate tim step from stability criterion


I am working on the file "05_pseudospectral/fourier_acoustic_2d.ipynb".
The first task is:
3. Source Initialization
Can you write Python code for it?


I am working on the file "05_pseudospectral/fourier_acoustic_2d.ipynb".
The first task is:
4. Time Extrapolation
The final solution for our 2D acoustic wave problem after taking into account the finite differences time extrapolation can be written as
\begin{equation}
p_{j,k}^{n+1} = dt^2c_{j,k}^{2} (\partial_{x}^{2}p + \partial_{z}^{2}p){j,k} + dt^2s{j,k}^{n} + 2p_{j,k}^{n} - p_{j,k}^{n-1}
\end{equation}
In order to compare the above numerical solution, we implement a 5-point finite difference operator to compute spatial derivatives
\begin{equation}
\partial_x^2 p(x,t) = \frac{-p(x+2\mathrm{d}x,t) + 16p(x+\mathrm{d}x,t) - 30p(x,t) + 16p(x-\mathrm{d}x,t) - p(x-2\mathrm{d}x,t)}{12\mathrm{d}x^2}
\end{equation}
temporal derivative is done with a 3-point finite difference operator.
Numerical dispersion and anysotropy
One of the most significant characteristics of the fourier method is the low numerical dispersion in comparison with the finite difference method. The snapshots displayed below for both solutions allow us to brifly comment two significant observations:
1) There is strong anisotropic dispersion behaviour visible for the finite-difference solution. The most accurate direction occur at $\theta = \pi/4$
2) The Fourier solution do not exhibit significant dispersion, but the most importantly, it does not seem to be directionally dependent. In other words the error is isotropic.
Can you write Python code for it?

# Initialize animated plot
# ---------------------------------------------------------------
fig = plt.figure(figsize=(12,6))
ax1 = fig.add_subplot(1,2,1)
ax2 = fig.add_subplot(1,2,2)
line1 = ax1.imshow(sp_sec, interpolation="bicubic", cmap=plt.cm.RdBu)
line2 = ax2.imshow(ap_sec, interpolation="bicubic", cmap=plt.cm.RdBu)
ax1.set_title('Fourier Method', size=14)
ax2.set_title('Finite-Difference Method', size=14)

plt.ion()   # set interective mode
plt.show()

# ---------------------------------------------------------------
# Time extrapolation
# ---------------------------------------------------------------
for it in range(nt):
    # ----------------------------------------
    # Fourier Pseudospectral Method
    # ----------------------------------------
    # 2nd space derivative
    for j in np.arange(nz):
        sd2px[:,j] = fourier_derivative_2nd(sp[:,j].T, dx)
    for i in np.arange(nx):
        sd2pz[i,:] = fourier_derivative_2nd(sp[i,:], dx)

    # Time Extrapolation
    spnew = 2*sp - spold + c**2 * dt**2 * (sd2px + sd2pz)
    spnew = spnew + sg*src[it]*dt**2 # Add sources
    spold, sp = sp, spnew # Time levels

    # ----------------------------------------
    # Finite Differences Method 5pt
    # ----------------------------------------
    for i in range(2, nz-2):
        ad2px[i,:] = (-1./12*ap[i+2,:] + 4./3*ap[i+1,:] - 5./2*ap[i,:] \
                      + 4./3*ap[i-1,:] - 1./12*ap[i-2,:])/dx**2 # Space derivative
    for i in range(2, nx-2):
        ad2pz[:,i] = (-1./12*ap[:,i+2] + 4./3*ap[:,i+1] - 5./2*ap[:,i] \
                      + 4./3*ap[:,i-1] - 1./12*ap[:,i-2])/dx**2 # Space derivative

    apnew = 2*ap - apold + dt**2 * c**2 * (ad2px + ad2pz) # Time Extrapolation
    apnew = apnew + sg*src[it]*dt**2 # Add source
    apold, ap = ap, apnew # Time levels

    # Select Sections for plotting
    sp_sec = -np.abs(sp[1:int(nx/2), 1:int(nz/2)])
    ap_sec = -np.abs(ap[int(nx/2):nx, 1:int(nz/2)].T)

    #...


I am working on the file "05_pseudospectral/ps_derivative.ipynb".
The first task is:
Exercise 1
Define a python function call "fourier_derivative(f, dx)" that compute the first derivative of a function $f$ using the Fourier transform properties.
Can you write Python code for it?


I am working on the file "05_pseudospectral/ps_derivative.ipynb".
The first task is:
Exercise 2
Calculate the numerical derivative based on the Fourier transform to show that the derivative is exact. Define an arbitrary function (e.g. a Gaussian) and initialize its analytical derivative on the same spatial grid. Calculate the numerical derivative and the difference to the analytical solution. Vary the wavenumber content of the analytical function. Does it make a difference? Why is the numerical result not entirely exact?
Can you write Python code for it?


I am working on the file "05_pseudospectral/ps_derivative.ipynb".
The first task is:
Exercise 3
Now that the numerical derivative is available, we can visually inspect our results. Make a plot of both, the analytical and numerical derivatives together with the difference error.
Can you write Python code for it?


I am working on the file "05_pseudospectral/ps_derivative_solution.ipynb".
The first task is:
Exercise 3
Now that the numerical derivative is available, we can visually inspect our results. Make a plot of both, the analytical and numerical derivatives together with the difference error.
Can you write Python code for it?

# Plot analytical and numerical derivatives
# ---------------------------------------------------------------

plt.subplot(2,1,1)
plt.plot(x, f, "g", lw = 1.5, label='Gaussian')
plt.legend(loc='upper right', shadow=True)
plt.xlabel('$x$')
plt.ylabel('$f(x)$')
plt.axis([2*np.pi/nx, 2*np.pi, 0, 1])

plt.subplot(2,1,2)
plt.plot(x, df_ana, "b", lw = 1.5, label='Analytical')
plt.plot(x, df_num, 'k--', lw = 1.5, label='Numerical')
plt.plot(x, df_err, "r", lw = 1.5, label='Difference')
plt.legend(loc='upper right', shadow=True)
plt.xlabel('$x$')
plt.ylabel('$\partial_x f(x)$')
plt.axis([2*np.pi/nx, 2*np.pi, -2, 2])

plt.show()
#plt.savefig('Fig_5.9.png')


I am working on the file "05_python_errores/errores.ipynb".
The first task is:
2.2 Debug: Utilización de print
Utilizar print es la técnica más sencilla y habitual, apropiada si los errores son sencillos.
PRO:
* Fácil y rápido de implementar.
* Permite inspeccionar valores de variable a lo largo de todo un programa
CONTRA:
* Requiere escribir expresiones más complicadas para estudiar más de una variable simultáneamente.
* Impresión no ayuda para estudiar datos multidimensionales (arreglos, matrices, diccionarios grandes).
* Eliminacion de múltiples print puede ser compleja en programa grande.
* Inapropiado si la ejecución del programa tarde demasiado (por ejemplo si tiene que leer un archivo de disco), pues habitualmente se van insertando prints a lo largo de varias ejecuciones "persiguiendo" el valor de una variable.
Consejo
Si se desea inspeccionar la variable mi_variable_con_error, utilice
print("!!!" + str(mi_variable_con_error))

o bien
print(mi_variable_con_error) #!!!

De esa forma será más facil ver en el output donde está la variable impresa, y luego de solucionar el bug, será también más fácil eliminar las expresiones print que han sido insertadas para debugear (no se confundirá con los print que sí son necesarios y naturales al programa).
Desafío 2
Detecte porqué el programa se comporta de manera inadecuada, utilizando print donde parezca adecuado.
No elimine los print que usted haya introducido, sólo coméntelos con #.
Arregle el desperfecto e indique con un comentario en el código donde estaba el error.
Can you write Python code for it?

def fibonacci(n):
    """
    Debe regresar la lista con los primeros n numeros de fibonacci.
    Para n<1, regresar [].
    Para n=1, regresar [1].
    Para n=2, regresar [1,1].
    Para n=3, regresar [1,1,2].
    Para n=4, regresar [1,1,2,3].
    Y sucesivamente
    """
    a = 1
    b = 1
    fib = [a,b]
    count = 2
    if n<1:
        return []
    if n=1:
        return [1]
    while count <= n:
        aux = a
        a = b
        b = aux + b
        count += 1
        fib.append(aux)
    return fib

print "fibonacci(-1):", fibonacci(-1) # Deberia ser []
print "fibonacci(0):", fibonacci(0) # Deberia ser []
print "fibonacci(1):", fibonacci(1) # Deberia ser [1]
print "fibonacci(2):", fibonacci(2) # Deberia ser [1,1]
print "fibonacci(3):", fibonacci(3) # Deberia ser [1,1,2]
print "fibonacci(5):", fibonacci(5) # Deberia ser [1,1,2,3,5]
print "fibonacci(10):", fibonacci(10) # Deberia ser ...

"""
ERRORES DETECTADOS:
1)
2)
3)
"""


I am working on the file "05_python_errores/errores.ipynb".
The first task is:
Contenido

Introducción
Técnicas de debugging.

Sobre el Notebook
Existen 4 desafíos:
* En todos los casos, documenten lo encontrado. Escriban como un #comentario o """comentario""" los errores que vayan detectando.
* En el desafío 1: Ejecute la celda, lea el output y arregle el código. Comente los 5 errores en la misma celda.
* En el desafío 2: Ejecute la celda y encuentre los errores utilizando print. Comente los 3 errores en la misma celda.
* En el desafío 3: Ejecute el archivo ./mat281_code/desafio_3.py, y encuentre los 3 errores utilizando pdb.set_trace()
* En el desafío 4: Ejecute el archivo ./mat281_code/desafio_4.py, y encuentre los 3 errores utilizando IPython.embed()
1. Introducción
Debugging: Eliminación de errores de un programa computacional.
* Fácilmente 40-60% del tiempo utilizado en la creación de un programa.
* Ningún programa está excento de bugs/errores.
* Imposible garantizar utilización 100% segura por parte del usuario.
* Programas computaciones tienen inconsistencias/errores de implementación.
* ¡Hardware también puede tener errores!
1. Introducción
¿Porqué se le llama bugs?
Existen registros en la correspondencia de Thomas Edisson, en 1848, hablaba de bugs para referirse a errores en sus inventos. El término se utilizaba ocasionalmente en el dominio informático. En 1947, el ordenador Mark II presentaba un error. Al buscar el origen del error, los técnicos encontraron una polilla, que se había introducido en la máquina.
<img src="images/bug.jpg" alt="" width="600px" align="middle"/>
Toda la historia en el siguiente enlace a wikipedia (ingles).
2. Técnicas para Debug

Leer output entregado por python para posibles errores
Utilizando print
Utilizando pdb: python debugger
Lanzamiento condicional de Ipython embed

2.1 Debug: Leer output de errores
Cuando el programa no funciona y entrega un error normalmente es fácil solucionarlo.
El mensaje de error entregará: la línea donde se detecta el error y el tipo de error.
PROS:
* Explicativo
* Fácil de detectar y reparar
CONTRA:
* No todos los...
Can you write Python code for it?

import numpy as np
def promedio_positivo(a):
    pos_mean = a[a>0].mean()
    return pos_mean

N = 100
x = np.linspace(-1,1,N)
y = 0.5 - x**2   # No cambiar esta linea
print(promedio_positivo(y))

# Error 1:
# Error 2:
# Error 3:
# Error 4:
# Error 5:


I am working on the file "05_resilience/batch_serving.ipynb".
The first task is:
Load model into BigQuery for batch serving
Can you write Python code for it?

%%bigquery
CREATE OR REPLACE MODEL mlpatterns.imdb_sentiment
OPTIONS(model_type='tensorflow', model_path='gs://ai-analytics-solutions-kfpdemo/mlpatterns/batchserving/*')

%%bigquery
SELECT * FROM ML.PREDICT(MODEL mlpatterns.imdb_sentiment,
  (SELECT 'This was very well done.' AS reviews)
)


I am working on the file "05_resilience/batch_serving.ipynb".
The first task is:
Now, do it at scale, on consumer complaints about financial products and services:
Can you write Python code for it?

%%bigquery preds
SELECT * FROM ML.PREDICT(MODEL mlpatterns.imdb_sentiment,
  (SELECT consumer_complaint_narrative AS reviews
   FROM `bigquery-public-data`.cfpb_complaints.complaint_database
   WHERE consumer_complaint_narrative IS NOT NULL
   )
)

preds[:3]

# what's does a "positive" complaint look like?
preds.sort_values(by='positive_review_probability', ascending=False).iloc[1]['reviews']

# what's does a "typical" complaint look like?
preds.sort_values(by='positive_review_probability', ascending=False).iloc[len(preds)//2]['reviews']


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
....and examine the model's serving default signature. As expected the model takes as input a text string (e.g. an article title) and retrns a 3-dimensional vector of floats (i.e. the softmax output layer).
Can you write Python code for it?

!saved_model_cli show \
 --tag_set serve \
 --signature_def serving_default \
 --dir ./txtcls_swivel/


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
...and create a train/test split.
Can you write Python code for it?

N_TRAIN = int(len(titles_df) * 0.80)

titles_train, sources_train = (
    titles_df.title[:N_TRAIN], titles_df.source[:N_TRAIN])

titles_valid, sources_valid = (
    titles_df.title[N_TRAIN:], titles_df.source[N_TRAIN:])

X_train, Y_train = titles_train.values, encode_labels(sources_train)
X_valid, Y_valid = titles_valid.values, encode_labels(sources_valid)

X_train[:3]


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
Can also examine a confusion matrix:
Can you write Python code for it?

cm = confusion_matrix(groundtruth, prediction, labels=sources)

ax= plt.subplot()
sns.heatmap(cm, annot=True, ax = ax, cmap="Blues")

# labels, title and ticks
ax.set_xlabel('Predicted labels')
ax.set_ylabel('True labels')
ax.set_title('Confusion Matrix')
ax.xaxis.set_ticklabels(sources)
ax.yaxis.set_ticklabels(sources)
plt.savefig("./txtcls_cm.png")


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
Compute evaluation metrics
With the raw prediction input, the model output and the groundtruth in one place, we can evaluation how our model performs. And how the model performs across various aspects (e.g. over time, different model versions, different labels, etc)
Can you write Python code for it?

import seaborn as sns
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
from sklearn.metrics import precision_recall_fscore_support as score
from sklearn.metrics import classification_report


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
Deploy the model for online serving
Once the model is trained and the assets saved, deploying the model to GCP is straightforward. After some time you should be able to see your deployed model and its version on the model page of GCP console.
Can you write Python code for it?

%%bash
MODEL_NAME="txtcls"
MODEL_VERSION="swivel"
MODEL_LOCATION="./txtcls_swivel/"

gcloud ai-platform models create ${MODEL_NAME}

gcloud ai-platform versions create ${MODEL_VERSION} \
--model ${MODEL_NAME} \
--origin ${MODEL_LOCATION} \
--staging-bucket gs://${BUCKET} \
--runtime-version=2.1


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
Examine eval metrics by model version or timestamp
By specifying the same evaluation table, two different model versions can be evaluated. Also, since the timestamp is captured, it is straightforward to evaluation model performance over time.
Can you write Python code for it?

now = pd.Timestamp.now(tz='UTC')
one_week_ago = now - pd.DateOffset(weeks=1)
one_month_ago = now - pd.DateOffset(months=1)

df_prev_week = df_results[df_results.time > one_week_ago]
df_prev_month = df_results[df_results.time > one_month_ago]

df_prev_month


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
Now when we call predictions using the updated serving input function, the model will return the predicted article source as a readable string, and the model's confidence for that prediction.
Can you write Python code for it?

title1 = "House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force"
title2 = "YouTube introduces Video Chapters to make it easier to navigate longer videos"
title3 = "As facebook turns 10  zuckerberg wants to change how tech industry works"

restored = tf.keras.models.load_model('./txtcls_swivel')
infer = restored.signatures['serving_default']
outputs = infer(text=tf.constant([title1, title2, title3]))

print(outputs['source'].numpy())
print(outputs['confidence'].numpy())


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
Now, we'll re-save the new Swivel model that has this updated model signature by referencing the source_name function for the model's serving_default.
Can you write Python code for it?

shutil.rmtree('./txtcls_swivel', ignore_errors=True)
txtcls_model.save('./txtcls_swivel', signatures={'serving_default': source_name})


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
Provide the ground truth for the raw prediction input
Notice the groundtruth is missing. We'll update the evaluation table to contain the ground truth.
Can you write Python code for it?

%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET
    groundtruth = '{"predictions": [{"source": "techcrunch"}]}'
WHERE
    raw_data = '{"instances": [{"text": "YouTube introduces Video Chapters to make it easier to navigate longer videos"}]}';

%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET
    groundtruth = '{"predictions": [{"source": "nytimes"}]}'
WHERE
    raw_data = '{"instances": [{"text": "A Filmmaker Put Away for Tax Fraud Takes Us Inside a British Prison"}]}';

%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET
    groundtruth = '{"predictions": [{"source": "github"}]}'
WHERE
    raw_data = '{"instances": [{"text": "A native Mac app wrapper for WhatsApp Web"}]}';

%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET
    groundtruth = '{"predictions": [{"source": "nytimes"}]}'
WHERE
    raw_data = '{"instances": [{"text": "Astronauts Dock With Space Station After Historic SpaceX Launch"}]}';

%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET
    groundtruth = '{"predictions": [{"source": "nytimes"}]}'
WHERE
    raw_data = '{"instances": [{"text": "House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force"}]}';

%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET
    groundtruth = '{"predictions": [{"source": "github"}]}'
WHERE
    raw_data = '{"instances": [{"text": "Scrollability"}]}';

%%bigquery --project $PROJECT
UPDATE `txtcls_eval.swivel`
SET
    groundtruth = '{"predictions": [{"source": "techcrunch"}]}'
WHERE
    raw_data = '{"instances": [{"text": "iOS 14 lets deaf users set alerts for important sounds, among other clever accessibility perks"}]}';


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
Set up the Evaluation job on CAIP
Now that the model is deployed, go to Cloud AI Platform to see the model version you've deployed and set up an evaluation job by clicking on the button called "Create Evaluation Job". You will be asked to provide some relevant information:
 - Job description: txtcls_swivel_eval
 - Model objective: text classification
 - Classification type: single-label classification
 - Prediction label file path for the annotation specification set: When you create an evaluation job on CAIP, you must specify a CSV file that defines your annotation specification set. This file must have one row for every possible label your model outputs during prediction. Each row should be a comma-separated pair containing the label and a description of the label: label-name,description
 - Daily sample percentage: We'll set this to 100% so that all online predicitons are captured for evaluation.
 - BigQuery table to house online prediction requests: We'll use the BQ dataset and table txtcls_eval.swivel. If you enter a BigQuery table that doesn’t exist, one with that name will be created with the correct schema.
 - Prediction input
     - Data key: this is The key for the raw prediction data. From examining our deployed model signature, the input data key is text.
     - Data reference key: this is for image models, so we can ignore
 - Prediction output
     - Prediction labels key: This is the prediction key which contains the predicted label (i.e. the article source). For our model, the label key is source.
     - Prediction score key: This is the prediction key which contains the predicted scores (i.e. the model confidence). For our model, the score key is confidence.
 - Ground-truth method: Check the box that indicates we will provide our own labels, and not use a Human data labeling service.
Once the evaluation job is set up, the table will be made in BigQuery to capture the online prediction requests.
Can you write Python code for it?

%load_ext google.cloud.bigquery

%%bigquery --project $PROJECT
SELECT * FROM `txtcls_eval.swivel`


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
Summarizing the results from our model:
| title  | groundtruth  | predicted
|---|---|---|
| YouTube introduces Video Chapters to make it easier to navigate longer videos  |  techcrunch | techcrunch |
| A Filmmaker Put Away for Tax Fraud Takes Us Inside a British Prison | nytimes | techcrunch |
| A native Mac app wrapper for WhatsApp Web | github | techcrunch |
| Astronauts Dock With Space Station After Historic SpaceX Launch | nytimes | techcrunch |
| House Passes Sweeping Policing Bill Targeting Racial Bias and Use of Force | nytimes | nytimes |
| Scrollability | github | techcrunch |
| iOS 14 lets deaf users set alerts for important sounds, among other clever accessibility perks | techcrunch | nytimes |
Can you write Python code for it?

%%bigquery --project $PROJECT
SELECT * FROM `txtcls_eval.swivel`


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
The build_model function is written so that the TF Hub module can easily be exchanged with another module.
Can you write Python code for it?

def build_model(hub_module, model_name):
    inputs = Input(shape=[], dtype=tf.string, name="text")
    module = hub_module(inputs)
    h1 = Dense(16, activation='relu', name="h1")(module)
    outputs = Dense(N_CLASSES, activation='softmax', name='outputs')(h1)
    model = Model(inputs=inputs, outputs=[outputs], name=model_name)

    model.compile(
        optimizer='adam',
        loss='categorical_crossentropy',
        metrics=['accuracy']
    )
    return model

def train_and_evaluate(train_data, val_data, model, batch_size=5000):
    tf.random.set_seed(33)
    X_train, Y_train = train_data

    history = model.fit(
        X_train, Y_train,
        epochs=100,
        batch_size=batch_size,
        validation_data=val_data,
        callbacks=[EarlyStopping()],
    )
    return history

txtcls_model = build_model(swivel_module, model_name='txtcls_swivel')

txtcls_model.summary()


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
To simplify the returned predictions, we'll modify the model signature so that the model outputs the predicted article source (either nytimes, techcrunch, or github) rather than the final softmax layer. We'll also return the 'confidence' of the model's prediction. This will be the softmax value corresonding to the predicted article source.
Can you write Python code for it?

@tf.function(input_signature=[tf.TensorSpec([None], dtype=tf.string)])
def source_name(text):
    labels = tf.constant(['github', 'nytimes', 'techcrunch'], dtype=tf.string)
    probs = txtcls_model(text, training=False)
    indices = tf.argmax(probs, axis=1)
    pred_source = tf.gather(params=labels, indices=indices)
    pred_confidence = tf.reduce_max(probs, axis=1)

    return {'source': pred_source,
            'confidence': pred_confidence}


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
Train and deploy the model
For this notebook, we'll build a text classification model using the Hacker News dataset. Each training example consists of an article title and the article source. The model will be trained to classify a given article title as belonging to either nytimes, github or techcrunch.
Load the data
Can you write Python code for it?

DATASET_NAME = "titles_full.csv"
COLUMNS = ['title', 'source']

titles_df = pd.read_csv(DATASET_NAME, header=None, names=COLUMNS)
titles_df.head()


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
Train and evaluation the model
With the model defined and data set up, next we'll train and evaluate the model.
Can you write Python code for it?

# set up train and validation data
train_data = (X_train, Y_train)
val_data = (X_valid, Y_valid)


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
We can confirm that the ground truch has been properly added to the table.
Can you write Python code for it?


I am working on the file "05_resilience/continuous_eval.ipynb".
The first task is:
We one-hot encode the label...
Can you write Python code for it?

CLASSES = {
    'github': 0,
    'nytimes': 1,
    'techcrunch': 2
}
N_CLASSES = len(CLASSES)

def encode_labels(sources):
    classes = [CLASSES[source] for source in sources]
    one_hots = to_categorical(classes, num_classes=N_CLASSES)
    return one_hots

encode_labels(titles_df.source[:4])


I am working on the file "05_resilience/gpu_serving.ipynb".
The first task is:
Delete the web service
The REST endpoint scales down to one node when it doesn't encounter any traffic.
To avoid paying for that node, let's delete the model.
Can you write Python code for it?

%%bash

MODEL_NAME=sunrise
MODEL_VERSION=v1

# delete the model version if it already exists
modelver=$(gcloud ai-platform versions list --model "$MODEL_NAME" | grep -w "$MODEL_VERSION")
echo $modelver
if [ "$modelver" ]; then
   echo "Deleting version $MODEL_VERSION"
   yes | gcloud ai-platform versions delete ${MODEL_VERSION} --model ${MODEL_NAME}
   sleep 10
fi


I am working on the file "05_resilience/gpu_serving.ipynb".
The first task is:
Keras Model
Create a no-op Keras Model and export it with the above serving function.
Can you write Python code for it?

import os, datetime, shutil

@tf.function(input_signature=[
    tf.TensorSpec([None], dtype=tf.float32),
    tf.TensorSpec([None], dtype=tf.float32),
    tf.TensorSpec([None], dtype=tf.float32),
    tf.TensorSpec([None], dtype=tf.float32),
])
def calc_on_gpu(lat, lng, dayno, utc_offset):
    return calc_sunrise_sunset(lat, lng, dayno, utc_offset)

model = tf.keras.Sequential([
    tf.keras.layers.Dense(1, input_shape=[1])
  ])

shutil.rmtree('export', ignore_errors=True)
export_path = os.path.join('export', 'sunrise_{}'.format(datetime.datetime.now().strftime("%Y%m%d_%H%M%S")))
model.save(export_path, signatures={'serving_default': calc_on_gpu})

!saved_model_cli show --dir {export_path} --tag_set serve --signature_def serving_default

restored = tf.keras.models.load_model(export_path)
infer = restored.signatures['serving_default']
# note input name
outputs = infer(lat=tf.constant(39.833), lng=tf.constant(-98.583), dayno=tf.constant(15.0), utc_offset=tf.constant(-6.0))
print(outputs)


I am working on the file "05_resilience/nlp_api.ipynb".
The first task is:
Changing input to BigQuery and running on Cloud
Use DataflowRunner
Can you write Python code for it?

%%bigquery
SELECT text FROM `bigquery-public-data.hacker_news.comments`
WHERE author = 'AF' LIMIT 10

%%writefile nlp_pipeline.py

PROJECT='ai-analytics-solutions'
BUCKET='ai-analytics-solutions-kfpdemo'
REGION='us-central1'

from datetime import datetime
import apache_beam as beam

def parse_nlp_result(response):
    return [
        # response, # entire string
        response.sentences[0].text.content,
        response.language,
        response.document_sentiment.score
    ]

def run():
    from apache_beam.ml.gcp import naturallanguageml as nlp

    features = nlp.types.AnnotateTextRequest.Features(
        extract_entities=True,
        extract_document_sentiment=True,
        extract_syntax=False
    )
    options = beam.options.pipeline_options.PipelineOptions()
    google_cloud_options = options.view_as(beam.options.pipeline_options.GoogleCloudOptions)
    google_cloud_options.project = PROJECT
    google_cloud_options.region = REGION
    google_cloud_options.job_name = 'nlpapi-{}'.format(datetime.now().strftime("%Y%m%d-%H%M%S"))
    google_cloud_options.staging_location = 'gs://{}/staging'.format(BUCKET)
    google_cloud_options.temp_location = 'gs://{}/temp'.format(BUCKET)
    options.view_as(beam.options.pipeline_options.StandardOptions).runner = 'DataflowRunner' # 'DirectRunner'

    p = beam.Pipeline(options=options)
    (p
     | 'bigquery' >> beam.io.Read(beam.io.BigQuerySource(
         query="SELECT text FROM `bigquery-public-data.hacker_news.comments` WHERE author = 'AF' AND LENGTH(text) > 10",
         use_standard_sql=True))
      | 'txt'      >> beam.Map(lambda x : x['text'])
      | 'doc'      >> beam.Map(lambda x : nlp.Document(x, type='PLAIN_TEXT'))
    #  | 'todict'   >> beam.Map(lambda x : nlp.Document.to_dict(x))
      | 'nlp'      >> nlp.AnnotateText(features, timeout=10)
      | 'parse'    >> beam.Map(parse_nlp_result)
      | 'gcs'      >> beam.io.WriteToText('gs://{}/output.txt'.format(BUCKET), num_shards=1)
    )
    result = p.run()
    result.wait_until_finish()

if...


I am working on the file "05_resilience/serving_function.ipynb".
The first task is:
Better is to export the model and then deploy that ...
Can you write Python code for it?

%%bash
BUCKET=ai-analytics-solutions-kfpdemo
bq extract -m --destination_format=ML_TF_SAVED_MODEL mlpatterns.neutral_3classes  gs://${BUCKET}/export/baby_health

%%bash

TFVERSION=1.15
REGION=us-central1
BUCKET=ai-analytics-solutions-kfpdemo
MODEL_LOCATION=gs://${BUCKET}/export/baby_health
MODEL_NAME=babyhealth
MODEL_VERSION=v1

# create the model if it doesn't already exist
modelname=$(gcloud ai-platform models list | grep -w "$MODEL_NAME")
echo $modelname
if [ -z "$modelname" ]; then
   echo "Creating model $MODEL_NAME"
   gcloud ai-platform models create ${MODEL_NAME} --regions $REGION
else
   echo "Model $MODEL_NAME already exists"
fi

# delete the model version if it already exists
modelver=$(gcloud ai-platform versions list --model "$MODEL_NAME" | grep -w "$MODEL_VERSION")
echo $modelver
if [ "$modelver" ]; then
   echo "Deleting version $MODEL_VERSION"
   yes | gcloud ai-platform versions delete ${MODEL_VERSION} --model ${MODEL_NAME}
   sleep 10
fi

echo "Creating version $MODEL_VERSION from $MODEL_LOCATION"
gcloud ai-platform versions create ${MODEL_VERSION} \
       --model ${MODEL_NAME} --origin ${MODEL_LOCATION} --staging-bucket gs://${BUCKET} \
       --runtime-version $TFVERSION

%%writefile input.json
{"plurality": 2, "mother_age": 32, "gestation_weeks": 41, "ever_born": 1}

!gcloud ai-platform predict --model babyhealth --json-instances input.json --version v1


I am working on the file "05_resilience/serving_function.ipynb".
The first task is:
Export the model for serving
model.save() writes out a "serve" tag_set
Can you write Python code for it?

import os, datetime, shutil
shutil.rmtree('export/default', ignore_errors=True)
export_path = os.path.join('export', 'default', 'sentiment_{}'.format(datetime.datetime.now().strftime("%Y%m%d_%H%M%S")))
model.save(export_path)

!find export/default