I am working on the file "02_WP/General/PrintHead_Calculations.ipynb".
The first task is:
Printing Dimensions
For calculating the Printing dimensions in x and y axis. The following values are needed.
* section_witdh = Number of bytes of a section. Can be found in CoE Memory Organization 8040:1
* section_depth = Number of bytes of a section. Can be found in CoE Memory Organization 8040:2
* Number of sections = Number of bytes of a section. Can be found in CoE Memory Organization 8040:3
* Number_of_channels = Equals number of colors per board
* Resolution of the Printhead
$print_{width} = section_{width} * 2\frac{pixel}{byte} * \frac{0.0254\frac{m}{inch}}{ph_{dpi}} = [m]$
$print_{length} = section_{depth} * section_{numbers} * 2\frac{pixel}{byte} * \frac{0.0254\frac{m}{inch}}{ph_{dpi}} = [m]$
Can you write Python code for it?

import math
def printing_dimensions(section_width, section_depth, section_numbers, number_of_channels, ph_dpi):
  """
  printing_dimensions calculates the printing width and length of a printhead with the available memory
  :param section_width      : (bytes^=pixel/channel) Can be found in CoE Memory Organization 8040:1
  :param section_depth      : (bytes => 1 byte=2 px) Can be found in CoE Memory Organization 8040:2
  :param section_numbers    : Can be found in CoE Memory Organization 8040:3
  :param number_of_channels : Equals number of colors per board
  :param ph_dpi             : Dpi of the printheads
  :return print_width       : height (cross print axis) in mm of the printable width per printhead
  :return print_length      : length (print axis) in mm of the printable job
  """
  # Constants
  inch2m     = 0.0254 # m/inch
  pixel_byte = 2      # pixel/byte

  # Calc Pixel Pitch
  pixel_pitch = inch2m/ph_dpi # in m

  # Calc Print width = Y Axis = Crossprint Axis
  section_width_pixel = section_width # this is already in bytes (1 byte = 1 px)
  print_width = section_width_pixel*pixel_pitch

  # Calc Print length = X Axis = Print Axis
  section_depth_pixel = section_depth * pixel_byte
  print_length = section_depth_pixel*section_numbers*pixel_pitch

  # Calc memory size
  memory_size_byte  = section_width*section_depth*section_numbers*number_of_channels
  memory_size_pixel = memory_size_byte * pixel_byte

  # Print
  print("Memory Size              : {} MBytes = {} MPixels".format((memory_size_byte/1000/1000), (memory_size_pixel/1000/1000)))
  print("PixelPitch @ {:3}dpi     : {} um".format(ph_dpi,pixel_pitch*1000*1000))
  print("Image width              : {} mm".format(print_width*1000))
  print("Print length             : {} m".format(print_length))

  # m_number_of_sections=MEMORY_SIZE/(Memory_Section::get_width()*Memory_Section::get_depth()*anheads);
  # m_memory_depth=m_number_of_sections*Memory_Section::get_depth();

  return print_width, print_length

def...


I am working on the file "02_WP/General/PrintHead_Calculations.ipynb".
The first task is:
RDA Multihead
Can you write Python code for it?

section_width      = 1536  # bytes ^= pixel per channel
section_depth      = 8192  # bytes => 1bytes = 2 pixel (by 4bpp)
section_numbers    = 21    # Nbr per Channel (Color)
number_of_channels = 4     # Number of printheads per board
ph_dpi             = 360   # dpi
subrate_length     = (115.5822, 80) # (x, y) in mm
print_width, print_length = printing_dimensions(section_width, section_depth, section_numbers, number_of_channels, ph_dpi)
nbr_of_multipages = biggest_multipage_job(print_length,subrate_length[0] )


I am working on the file "02_WP/General/PrintHead_Calculations.ipynb".
The first task is:
Resolution
Can you write Python code for it?

pixel_pitch = 0.1692 # mm
inch2cm = 2.54 # cm/inch
resolution = (inch2cm/pixel_pitch)*10
print("Resolution: {:4} dpi   Pixel Pitch: {} um".format(resolution, pixel_pitch))


I am working on the file "02_WP/General/VarData_SpeedCalculation_Comparison.ipynb".
The first task is:
Test Example
Can you write Python code for it?

test_case         = 1 # Unique test number (UTN)
test_title        = "Digiround Max Speed Test"
test_description  = "Digiround Spectra PC \n CPU: i5-4570 \n RAM: 8GB 12800MHz \n SSD: 120GB \n 1 Gbps Ethernet Port for Calmar data"
#test_time         = "2017-11-29 14:02:00" # if not existing uses current time
test_time         = None
cups_speed        = 250.0            # cups/min
substrate_size_mm = [280.0, 140.0]   # [mm]
substrate_size_px = None             # [px]
dpi_image         = [360.0, 360.0]   # [dpi] dpi_x and dpy_y

varData_speedCalc_comparison(test_case, test_title, test_description, test_time, cups_speed, substrate_size_mm, substrate_size_px, dpi_image)


I am working on the file "02_WP/General/VarData_SpeedCalculation_Comparison.ipynb".
The first task is:
Your Test Here
Can you write Python code for it?

test_case         = 1 # Unique test number (UTN)
test_title        = "Digiround Max Speed Test"
test_description  = "Digiround Spectra PC \n CPU: i5-4570 \n RAM: 8GB 12800MHz \n SSD: 120GB \n 1 Gbps Ethernet Port for Calmar data"
#test_time         = "2017-11-29 14:02:00" # if not existing uses current time
test_time         = None
cups_speed        = 120.0            # cups/min
substrate_size_mm = [280.0, 140.0]   # [mm]
substrate_size_px = None             # [px]
dpi_image         = [360.0, 360.0]   # [dpi] dpi_x and dpy_y

varData_speedCalc_comparison(test_case, test_title, test_description, test_time, cups_speed, substrate_size_mm, substrate_size_px, dpi_image)

test_case         = 1 # Unique test number (UTN)
test_title        = "WK308 120 cups/min + Ergosoft running"
test_description  = "Spectra PC \n CPU: i5-4570 \n RAM: 24GB 12800MHz \n SSD: 223GB \n 1 Gbps Ethernet Port for Calmar data"
#test_time         = "2017-11-29 14:02:00" # if not existing uses current time
test_time         = None
cups_speed        = 240.0            # cups/min
substrate_size_mm = None   # [mm]
substrate_size_px = [3971, 2014]             # [px]
dpi_image         = [360.0, 360.0]   # [dpi] dpi_x and dpy_y

varData_speedCalc_comparison(test_case, test_title, test_description, test_time, cups_speed, substrate_size_mm, substrate_size_px, dpi_image)


I am working on the file "02_WP/General/Variable_Printhead_Resolution.ipynb".
The first task is:
KY KJ4B 40KHz
Can you write Python code for it?


I am working on the file "02_WP/General/Variable_Printhead_Resolution.ipynb".
The first task is:
Kyocera KJ4B 30kHz
Can you write Python code for it?


I am working on the file "02_WP/General/Variable_Printhead_Resolution.ipynb".
The first task is:
Kyocera KJ4B 40kHz
Can you write Python code for it?


I am working on the file "02_WP/General/Variable_Printhead_Resolution.ipynb".
The first task is:
Pixel Pos Values
Can you write Python code for it?

# vhdl function ported to python
def unsigned_num_bits(num):
  _nbits = 1
  _n = num
  while(_n > 1):
    _nbits = _nbits + 1
    _n     = _n / 2
  return _nbits

def calcPosXVal(maxVal_x, maxMult, xpos_BitNb):
  for i in range(int(round(maxMult))):
    val = maxVal_x * i

    print("MaxVal: {:4}  Multiplication: {:2} Result: {:5} NumberofBits(needed/available): ({:2}/{:2})".format(maxVal_x, i, val, unsigned_num_bits(val), xpos_BitNb))


I am working on the file "02_WP/General/Variable_Printhead_Resolution.ipynb".
The first task is:
Ricoh GEN5
Can you write Python code for it?


I am working on the file "02_WP/VHDL/HW Encoder and PLC Encoder.ipynb".
The first task is:
HW Encoder divider calculation
Can you write Python code for it?

# Settings
encoder_resolution = 6    # external encoder resolution [um p. pulse]
input_resolution   = 32   # resolution wanted for interpolator [sub pixel]
print_resolution   = 600  # print resolution [dpi]
maxResBitNb        = 19   # FPGA limitation for HW Encoder resolution

# Calculations & print
px_dist = (25.4 * 1000 / print_resolution)     # pixel distance [um]
pulse_p_pixel = px_dist / encoder_resolution   # pulse generated between pixel
#print "pixel distance [um]:      {}".format(px_dist)

print("Hardware Encoder Settings")
print("--------------------------")
print("encoder resolution[um]:       {}".format(encoder_resolution))
print("pixel resolution:             {}".format(print_resolution))
print("required pulse per pixel:     {}".format(input_resolution))
print("")
print("pulse per pixel:              {} ( @{:4}dpi )".format(pulse_p_pixel, print_resolution))

HW_Enc_resolution_range = 2**16                                   # full divider range of fpga

HW_Enc_reso_div       = pulse_p_pixel / input_resolution          # divider needed (exact)
HW_Enc_reso_setting_f = HW_Enc_resolution_range / HW_Enc_reso_div # setting for hw encoder resolution (floating number)
HW_Enc_reso_setting   = int(HW_Enc_reso_setting_f)                # setting for hw encoder resolution (divider rounded)

# if (pulse_p_pixel < input_resolution):
# hint = "!!! Encoder resolution is too small !!!"
if unsigned_num_bits(int(HW_Enc_reso_setting)) > maxResBitNb:
  hint = "!!! Hardware Encoder divider has too many bits (max {}bit)".format(maxResBitNb)
else:
  hint = "Encoder resolution is OK"

print("HWEnc resolution range:       {} ( {}bit )".format(HW_Enc_resolution_range, unsigned_num_bits(int(HW_Enc_resolution_range)-1)))
print("Resolution divider:           {} ( e:{},r:{},s:{} )".format(HW_Enc_reso_div, encoder_resolution, print_resolution, input_resolution))
print("HWEnc divider:                {} ( rounded: {} )".format(HW_Enc_reso_setting_f, HW_Enc_reso_setting))
print("")
print("Register...


I am working on the file "02_WP/VHDL/HW Encoder and PLC Encoder.ipynb".
The first task is:
Resolution Gain Calculations
$Resolution_{Gain} = 2^{16}\frac{Encoder_{Resolution}}{Input_{Resolution}}$
Can you write Python code for it?

import numpy as np
encoder_resolution = 2 # um
input_resolution   = sub_pixel_pitch

gain = np.empty(shape=[len(resolutions)], dtype=np.float64)
for i in range(len(resolutions)):
  gain[i] = 2**16 * (encoder_resolution/input_resolution[i])

for i in range(len(resolutions)):
  print("Resolution: {:4} dpi   Resolution Gain: {:>12.5f} or 0x{:08X} needs {} Bits".format(resolutions[i],gain[i],int(gain[i]),unsigned_num_bits(int(gain[i]))))


I am working on the file "02_WP/VHDL/Steppermotordriver_L6208PD.ipynb".
The first task is:
Max Acceleration calculations
$g_MAX_ACCELERATION = \frac{speed_{max}clk_{freq}}{(speed_{resolution}-1)acceleration_speed} = \frac{[\frac{rev}{s}][Hz]}{([values]-1)[\frac{rev}{s^{2}}]}$
$g_MAX_DECCELERATION = \frac{speed_{max}clk_{freq}}{(speed_{resolution}-1)decceleration_speed} = \frac{[\frac{rev}{s}][Hz]}{([values]-1)[\frac{rev}{s^{2}}]}$
Can you write Python code for it?

speed_resolution     = 2**8   # different speed values
clk_freq             = 100e6  # Hz
speed_max            = 60*1/60 # rev/min * min/s = rev/s

max_acceleration_time = 2.0 # seconds from 0 to max speed
max_acceleration_rev  = speed_max/max_acceleration_time # rev/s^2

max_decceleration_time = 1.0 # seconds from max to 0 speed
max_decceleration_rev  = speed_max/max_decceleration_time # rev/s^2

g_max_acceleration  = (speed_max*clk_freq)/((speed_resolution-1)*max_acceleration_rev)
g_max_decceleration = (speed_max*clk_freq)/((speed_resolution-1)*max_decceleration_rev)

print("g_MAX_ACCELERATION  = {} needs {} Bits".format(int(g_max_acceleration),unsigned_num_bits(int(g_max_acceleration))))
print("g_MAX_DECCELERATION = {} needs {} Bits".format(int(g_max_decceleration),unsigned_num_bits(int(g_max_decceleration))))


I am working on the file "02_WP/VHDL/Steppermotordriver_L6208PD.ipynb".
The first task is:
Max Frequency calulation
$f_{max} = speed * steps = \frac{1}{s}*1 = \frac{1}{s} $
Can you write Python code for it?

speed_max =  60# rev/min
step_angle = 1.8 # °

steps_per_rev = 360/step_angle
speed_max_sec = speed_max/60 # rev/sec

f_max = speed_max_sec * steps_per_rev
print("Max Frequency of Steppermotor is {} Hz".format(f_max))


I am working on the file "02_WP/VHDL/Steppermotordriver_L6208PD.ipynb".
The first task is:
Max Step Frequency
$g_STEP_FREQ = \frac{f_{clk}}{f_step_driver_{max}}$
For $f_step_driver_{max}$ see datasheet motor driver (L6208 = $100kHz$)
Can you write Python code for it?

f_clk      = 100e6 # Hz
f_step_max = 100e3 # Hz

g_step_freq = f_clk/f_step_max
print("Number of steps for max step frequency: {} needs {} Bits".format(int(g_step_freq), unsigned_num_bits(g_step_freq)))


I am working on the file "02_WP/VHDL/Steppermotordriver_L6208PD.ipynb".
The first task is:
Speed intended calculations
$ speed_{indended} = \sqrt[2]{264g_MAX_ACCELERATION|position_{difference}|} $
or
$ speed_{indended} = 41\log({g_MAX_ACCELERATION*|position_{difference}|}) $
Can you write Python code for it?

import math
speed_resolution      = 2**8 # different speed values
speed_max             = 120*1/60 # rev/min * min/s = rev/s
max_acceleration_time = 2.0 # seconds from 0 to max speed
max_acceleration_rev  = speed_max/max_acceleration_time # rev/s^2

def calc_speed_intended(max_acceleration_rev, position_difference):
#  return round(math.sqrt(2*64*max_acceleration_rev*position_difference))
  return round(41*math.log(max_acceleration_rev*position_difference+1))

for position_difference in [0,1,2,4,8,16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536]:
  speed_intended = calc_speed_intended(max_acceleration_rev, position_difference)
  if speed_intended > speed_resolution-1:
    speed_intended = speed_resolution-1
  print("speed_intended: {:3} @ position_difference: {:5}".format(int(speed_intended),position_difference))

# Draw Plot
import numpy as np
import pylab as pl
pl.clf()

nbrOfPoints = 600
position_difference = np.linspace(0,nbrOfPoints,nbrOfPoints)

speed_intended = np.empty(shape=[len(position_difference)], dtype=np.float64)
for i in range(len(position_difference)):
  speed_intended[i] = calc_speed_intended(max_acceleration_rev, position_difference[i])
  if speed_intended[i] > speed_resolution-1:
    speed_intended[i] = speed_resolution-1
# Plot graph
pl.plot(position_difference,speed_intended, label="Acceleration")

speed_intended = np.empty(shape=[len(position_difference)], dtype=np.float64)
for i in range(len(position_difference)):
  speed_intended[i] = 255-calc_speed_intended(max_acceleration_rev, position_difference[i])
  if speed_intended[i] <= 0:
    speed_intended[i] = 0
# Plot graph
pl.plot(position_difference,speed_intended, label="Decceleration")

# Place legend, Axis and Title
pl.legend(loc='best')
pl.xlabel("PositionDifference [Steps]")
pl.ylabel("Speed [0-255]")
pl.title("Acceleration & Deccleration")


I am working on the file "02_WP/VHDL/VectorAccessors.ipynb".
The first task is:
KonicaMinolta KM1024i
jetmapping.vhd lvec_mask_address
Can you write Python code for it?

import numpy as np
channels     = 2
elements     = 512
mask_address = np.empty(shape=[channels, elements],  dtype=np.int32)

for i in range(elements):
  mask_address[0,i] = (511-i) * 2 + 0
  mask_address[1,i] =      i  * 2 + 1

print("  i | High  Low \n----|-----------")
for i in range(elements):
  print("{:3} | {:4} {:4}".format(i,mask_address[0,i],mask_address[1,i]))


I am working on the file "02_WP/VHDL/VectorAccessors.ipynb".
The first task is:
Kyocera KJ4B_1200_64k CPLD
Printhead data
Can you write Python code for it?

import numpy as np
channels  = 10
data_size = 3
bounds    = np.empty(shape=[channels,2],  dtype=np.int32)

for i in range(1,channels+1):
  bounds[i-1][0] = data_size * i - 1   # High
  bounds[i-1][1] = data_size * (i - 1) # Low

print("  i | High  Low \n----|-----------")
for i in range(1,channels+1):
  print("{:3} | {:4} {:4}".format(i,bounds[i-1,0],bounds[i-1,1]))


I am working on the file "02_WP/VHDL/VectorAccessors.ipynb".
The first task is:
Ricoh GEN5
jetmapping.vhd lvec_mask_address
Can you write Python code for it?

import numpy as np
channels     = 4
elements     = 320
mask_address = np.empty(shape=[channels, elements],  dtype=np.int32)

for i in range(elements):
  mask_address[0,i] = (319-i) * 4 + 0
  mask_address[1,i] =      i  * 4 + 2
  mask_address[2,i] = (319-i) * 4 + 1
  mask_address[3,i] =      i  * 4 + 3

print("  i |    A    B    C    D \n----|---------------------")
for i in range(elements):
  print("{:3} | {:4} {:4} {:4} {:4}".format(i,mask_address[0,i],mask_address[1,i],mask_address[2,i],mask_address[3,i]))


I am working on the file "02_analytical_data_prep/src/housing.ipynb".
The first task is:
Feature interaction
Feature interaction is multiplying two variables together (example columns $x$ and $y$ -> $xy$)
Can you write Python code for it?

#Here is how to do polynomical expansion
train_corr = DataPreperation.get_top_correlations(original_train,numerics)

# https://plot.ly/python/figure-factory/table/
import plotly.figure_factory as ff

corr_df = pd.DataFrame(columns=['columns', 'correlation', 'correlation_abs'])
for idx, d in enumerate(train_corr):
    corr_df.loc[idx] = [d['columns'],d['correlation'],d['correlation_abs']]

table = ff.create_table(corr_df)
iplot(table, filename='pandas_table')

for idx, row in corr_df.iterrows():
    if(corr_df.loc[idx]['correlation_abs'] >0.5 and corr_df.loc[idx]['correlation_abs'] != 1): #Set a cutoff only combine values greater then .7
        original_train = DataPreperation.feature_combiner(original_train,columns=corr_df.loc[idx]['columns'])
        original_test = DataPreperation.feature_combiner(original_test,columns=corr_df.loc[idx]['columns'])
#show the results
table = ff.create_table(original_train.select('GarageArea','GarageCars','GarageArea|GarageCars').toPandas().sample(10))
table.layout.width=1000
iplot(table, filename='pandas_table')


I am working on the file "02_analytical_data_prep/src/housing.ipynb".
The first task is:
Label Encoding
When you have an algorithm like an SVM or decision tree that doesn't always numerical values as greater then one another. Or you have an ordinal variable label encoding is a good choice.
(example XGBoost requires you to do this)
convert not likely, likely, very likey into lets say 1,2,3
Note: this must be done before you split the data unlike other data prep techniques
Can you write Python code for it?

print('Column before encoding...')
print(original_train.select('RoofStyle').rdd.flatMap(list).collect()[0:49])
print()
original_train = DataPreperation.label_encoder(original_train,['RoofStyle'])
print()
numerics, categoricals = DataPreperation.get_type_lists(frame=original_train,rejects=[ID_VAR,Y],frame_type='spark')
print()
print('Column after encoding...')
print(original_train.select('RoofStyle_encoded').rdd.flatMap(list).collect()[0:49])


I am working on the file "02_analytical_data_prep/src/housing.ipynb".
The first task is:
New Chart
After winsorizing the new chart moved all the values > 3200 are now = 3200
Can you write Python code for it?

import plotly.graph_objs as go
from plotly.offline import download_plotlyjs, init_notebook_mode, plot, iplot
init_notebook_mode(connected=True)

original_train.select('TotalBsmtSF',Y).toPandas().head()
trace = go.Scatter(
    x = original_train.select('TotalBsmtSF').rdd.flatMap(list).collect(),
    y = original_train.select(Y).rdd.flatMap(list).collect(),
    mode = 'markers'
)
data = [trace]

# Plot and embed in ipython notebook!
iplot(data)#, filename='basic-scatter')


I am working on the file "02_analytical_data_prep/src/housing.ipynb".
The first task is:
Perturbed Rate-by-Level with Shrunken Averages
This algorithm is good for hanlding any kind of categorical column when the algoithm needs a continuous column. For this slgorithm you MUST split the data BEFORE putting it in other wise you will have feature leakage and will overfit very very very very badly. You also want to perturb the data in insert random noise to further prevent overfitting.
Formula:
$$(1 − λ) * levelmean + λ * overallmean*purtubedamount$$
Can you write Python code for it?

(train,valid) = original_train.randomSplit([0.7,0.3], seed=123)

print("Encoding numberic variables...")
for i, var in enumerate(['MSZoning']):
    total = len(categoricals)

    print('Encoding: ' + var + ' (' + str(i+1) + '/' + str(total) + ') ...')
    train,valid, original_test = DataPreperation.shrunken_averages_encoder(train, valid_frame = valid,test_frame=original_test,\
                                                     x=var, y=Y, lambda_=0.15, perturb_range=0.05,threshold=150,\
                                                     test=False, frame_type='spark',test_does_have_y=False,id_col=ID_VAR)
table = ff.create_table(train.select('MSZoning','MSZoning_Tencode').toPandas().sample(15))
iplot(table, filename='pandas_table')


I am working on the file "02_analytical_data_prep/src/housing.ipynb".
The first task is:
Polynomial Expansion
Polynomial expansion is taking a variable and adding polynomial terms such as $x^2$$ $$x ^3$ etc. This can be very helpful especially in regression based models.
Can you write Python code for it?

original_train = DataPreperation.polynomial_expansion(original_train,['1stFlrSF'],degree=3)
original_test = DataPreperation.polynomial_expansion(original_test,['1stFlrSF'],degree=3)

#show the results
print(original_train.select(ID_VAR,'1stFlrSF','1stFlrSF_^2','1stFlrSF_^3').toPandas().sample(2))
table = ff.create_table(original_train.select(ID_VAR,'1stFlrSF','1stFlrSF_^2','1stFlrSF_^3').toPandas().sample(10))
table.layout.width=1000
iplot(table, filename='pandas_table')


I am working on the file "02_analytical_data_prep/src/py_part_2_discretization.ipynb".
The first task is:
Create sample data set
Can you write Python code for it?

scratch_df = pd.DataFrame({'x1': pd.Series(np.random.randn(20))})

scratch_df


I am working on the file "02_analytical_data_prep/src/py_part_2_discretization.ipynb".
The first task is:
Discretize
Can you write Python code for it?

scratch_df['x1_discrete'] = pd.DataFrame(pd.cut(scratch_df['x1'], 5))
scratch_df


I am working on the file "02_analytical_data_prep/src/py_part_2_encoding.ipynb".
The first task is:
Create a sample data set
Can you write Python code for it?


I am working on the file "02_analytical_data_prep/src/py_part_2_feature_extraction.ipynb".
The first task is:
Compress x1 and x2 into a single principal component
Can you write Python code for it?

pca = PCA(n_components=1)

pca.fit(scratch_df)


I am working on the file "02_analytical_data_prep/src/py_part_2_feature_extraction.ipynb".
The first task is:
Perform basic feature extraction
Create a sample data set
Can you write Python code for it?


I am working on the file "02_analytical_data_prep/src/py_part_2_feature_extraction.ipynb".
The first task is:
Principal components are the projection of the data onto these eigenvectors. Principal components are usually centered around zero and each principal component is uncorrelated with all the others, i.e. principal components are orthogonal to one-another. Becuase prinicipal components represent the highest variance dimensions in the data and are not correlated with one another, they do an excellent job summarizing a data set with only a few dimensions (e.g. columns) and PCA is probably the most popular feature extraction technique.
Can you write Python code for it?

scratch_df['Centered_PC1'] = pca.transform(scratch_df[['x1', 'x2']])
scratch_df['Non_centered_PC1'] = pca.transform(scratch_df[['x1', 'x2']] + pca.mean_)
scratch_df['PC1_x1_back_projection'] = pd.Series(np.arange(1,8,2)) * pca.components_[0][0]
scratch_df['PC1_x2_back_projection'] = pd.Series(np.arange(1,8,2)) * pca.components_[0][1]
scratch_df

x = plt.scatter(scratch_df.x1, scratch_df.x2, color='b')
pc, = plt.plot(scratch_df.PC1_x1_back_projection, scratch_df.PC1_x2_back_projection, color='r')
plt.legend([x, pc], ['Observed data (x)', 'First principal component projection'], loc=4)
plt.xlabel('x1')
plt.ylabel('x2')
plt.show()


I am working on the file "02_analytical_data_prep/src/py_part_2_feature_selection.ipynb".
The first task is:
Calculate Pearson correlation for numeric variables
pandas.DataFrame.corr() function shows that x1 is much more correlated with y than x2.
Can you write Python code for it?


I am working on the file "02_analytical_data_prep/src/py_part_2_feature_selection.ipynb".
The first task is:
Perform simple feature selection
Create a sample data set
Can you write Python code for it?


I am working on the file "02_analytical_data_prep/src/py_part_2_impute.ipynb".
The first task is:
Create sample data set
Can you write Python code for it?

scratch_df = pd.DataFrame({'x1': [0, 1, 2, 3, np.nan, 5, 6, 7, np.nan, 8, 9]})

scratch_df


I am working on the file "02_analytical_data_prep/src/py_part_2_impute.ipynb".
The first task is:
Impute
Can you write Python code for it?

scratch_df['x1_impute'] = scratch_df.fillna(scratch_df.mean())
scratch_df


I am working on the file "02_analytical_data_prep/src/py_part_2_over_sample.ipynb".
The first task is:
If the event in a classification problem or the value in a prediction problem is imbalanced (usually toward zero) this can lead to biased models, single class predictions for classification models, and biased predictions for prediction models. The simplest approach for an imbalanced target is to oversample the data set.
Can you write Python code for it?

# fit random oversampling function
# cannot pass single array for X, must use numpy.reshape(-1, 1)
ros = RandomOverSampler()
over_sample_x, over_sample_y = ros.fit_sample(scratch_df.x.get_values().reshape(-1, 1),
                                              scratch_df.y)

# create Pandas dataframe from oversampling results
over_sample_df = pd.DataFrame({'over_sample_x': over_sample_x.reshape(16,),
                               'over_sample_y': over_sample_y})
over_sample_df


I am working on the file "02_analytical_data_prep/src/py_part_2_over_sample.ipynb".
The first task is:
Proportional oversampling
Create a sample data set
Can you write Python code for it?

scratch_df = pd.DataFrame({'x': pd.Series(np.arange(0, 10)),
                           'y': [0, 1, 0, 0, 0, 0, 1, 0, 0, 0]})
scratch_df


I am working on the file "02_analytical_data_prep/src/py_part_2_standardize.ipynb".
The first task is:
Create sample data set
Can you write Python code for it?

# create a data frame containing variables of disparate scale
scratch_df = pd.DataFrame({'x1': pd.Series(np.random.choice(1000, 20)),
                           'x2': pd.Series(np.random.choice(20, 20))})

scratch_df


I am working on the file "02_analytical_data_prep/src/py_part_2_standardize.ipynb".
The first task is:
Standardize
Can you write Python code for it?

# create a deep copy
# so this cell can be run many times w/o error
scratch_df1 = scratch_df.copy()

# loop through columns
# create new column
# apply z-score formula to new column
for col_name in scratch_df.columns:
    new_col_name = col_name + '_std'
    scratch_df1[new_col_name] = (scratch_df[col_name] - scratch_df[col_name].mean())/scratch_df[col_name].std()

# new variables are on the same scale
scratch_df1


I am working on the file "02_analytical_data_prep/src/py_part_2_target_encode_categorical.ipynb".
The first task is:
Encode categorical variables using a rate-by-level approach
Can you write Python code for it?

# make a new deep copy of scratch_df
# so you can run this cell many times w/o errors
scratch_df1 = scratch_df.copy()

# loop through columns to create new encoded columns
for col_name in scratch_df.columns[:-1]:
    new_col_name = col_name + '_encode'
    # create a dictionary of original categorical value:event rate for that value
    row_val_dict = {}
    for level in scratch_df[col_name].unique():
        row_val_dict[level] = scratch_df[scratch_df[col_name] == level]['y'].mean()
    # apply the transform from the dictionary on all rows in the column
    scratch_df1[new_col_name] = scratch_df[col_name].apply(lambda i: row_val_dict[i])

scratch_df1


I am working on the file "02_analytical_data_prep/src/py_part_2_target_encode_numeric.ipynb".
The first task is:
Create a sample data set
Can you write Python code for it?


I am working on the file "02_analytical_data_prep/src/py_part_2_winsorize.ipynb".
The first task is:
Create sample data set
Can you write Python code for it?

scratch_df = pd.DataFrame({'x1': pd.Series(np.random.choice(1000, 20))})

scratch_df


I am working on the file "02_analytical_data_prep/src/py_part_2_winsorize.ipynb".
The first task is:
Winsorize
Can you write Python code for it?

scratch_df['x1_winsor'] = winsorize(scratch_df['x1'], limits=[0.1, 0.1])
scratch_df


I am working on the file "02_basics_variables_ex2.ipynb".
The first task is:
2.2)
Can you write Python code for it?

with tf.Session(graph=graph):
    init.run()
    for iteration in range(20):
        add.eval()
        divide.eval()
    result = x.eval()

print(result)


I am working on the file "02_basics_variables_ex2.ipynb".
The first task is:
2.3)
Can you write Python code for it?

with tf.Session(graph=graph) as sess:
    init.run()
    for iteration in range(20):
        sess.run([add, divide])
    result = x.eval()

result


I am working on the file "02_basics_variables_ex2.ipynb".
The first task is:
2.4)
Can you write Python code for it?

graph = tf.Graph()
with graph.as_default():
    x = tf.Variable(0.0)
    y = tf.Variable(1.0)
    add = tf.assign(x, x + y)
    with tf.control_dependencies([add]):
        divide = tf.assign(y, y / 2)
    init = tf.global_variables_initializer()

with tf.Session(graph=graph) as sess:
    init.run()
    for iteration in range(30):
        sess.run([add, divide])
    result = x.eval()

result


I am working on the file "02_basics_variables_ex2.ipynb".
The first task is:
Variable State
Can you write Python code for it?

>>> session1 = tf.Session(graph=graph)
>>> session2 = tf.Session(graph=graph)
>>> x.initializer.run(session=session1)
>>> x.initializer.run(session=session2)

>>> increment_op.eval(session=session1)

>>> x.eval(session=session1)

>>> x.eval(session=session2)

>>> session1.close()
>>> session2.close()


I am working on the file "02_basics_variables_ex2.ipynb".
The first task is:
Variables Initializer
Can you write Python code for it?

>>> graph = tf.Graph()
>>> with graph.as_default():
...     x = tf.Variable(100)
...     c = tf.constant(5)
...     increment_op = tf.assign(x, x + c)
...     init = tf.global_variables_initializer()
...

>>> with tf.Session(graph=graph) as sess:
...     init.run()
...     print(x.eval())    # 100
...     for iteration in range(10):
...         increment_op.eval()
...     print(x.eval())    # 150
...


I am working on the file "02_building.ipynb".
The first task is:
Assignments
Your name: ...

Construct two functions (or_query and and_query) that will each take as input a single string, consisting of one or more words, and return a list of matching documents. or_query will return documents that contain at least one of the words in the query, while and_query requires all query terms to be present in the documents.

Note that you can use the tokenize and preprocess functions we defined above to tokenize and preprocess your query. You can also exploit the fact that the posting lists are sets, which means you can easily perform set operations such as union, difference and intersect on them.
Can you write Python code for it?


I am working on the file "02_building.ipynb".
The first task is:
Assignments
Your name: ...
Task 1
Construct a function called and_query that takes as input a single string, consisting of one or more words, and returns a list of matching documents. and_query, as its name suggests, should require that all query terms are present in the documents of the result list. Demonstrate the working of your function with an example (choose one that leads to fewer than 100 hits to not overblow this notebook file).
(You can use the tokenize and preprocess functions we defined above to tokenize and preprocess your query. You can also exploit the fact that the posting lists are sets, which means you can easily perform set operations such as union, difference and intersect on them.)
Can you write Python code for it?


I am working on the file "02_building.ipynb".
The first task is:
Creating our first index
We will now create an inverted index based on the words in the abstracts of the papers in our dataset.
We will implement our inverted index as a Python dictionary with terms as keys and posting lists as values. For the posting lists, instead of using Python lists and then implementing the different operations on them ourselves, we will use Python sets and use the predefined set operations to process these posting "lists". This will also ensure that each document is added at most once per term. The use of Python sets is not the most efficient solution but will work for our purposes. (As an optional additional exercise, you can try to implement the posting lists as Python lists for this and the following mini-assignments.)
Not every paper in our dataset has an abstract; we will only index papers for which an abstract is available.
Can you write Python code for it?

from collections import defaultdict

inverted_index = defaultdict(set)

# This may take a while:
for (id, abstract) in Abstracts.items():
    for term in preprocess(tokenize(abstract)):
        inverted_index[term].add(id)


I am working on the file "02_building.ipynb".
The first task is:
Let's have a look at how the data looks for our example paper:
Can you write Python code for it?


I am working on the file "02_building.ipynb".
The first task is:
Let's have a look at what the data looks like for our example paper:
Can you write Python code for it?


I am working on the file "02_building.ipynb".
The first task is:
Let's see what's in the index for the example term 'network':
Can you write Python code for it?


I am working on the file "02_building.ipynb".
The first task is:
Some utility functions
We'll define some utility functions that allow us to tokenize a string into terms, perform linguistic preprocessing on a list of terms, as well as a function to display information about a paper in a nice way. Note that these tokenization and preprocessing functions are rather naive - you may have to make them smarter in a later assignment.
Can you write Python code for it?

def tokenize(text):
    """
    Function that tokenizes a string in a rather naive way. Can be extended later.
    """
    return text.split(' ')

def preprocess(tokens):
    """
    Perform linguistic preprocessing on a list of tokens. Can be extended later.
    """
    result = []
    for token in tokens:
        result.append(token.lower())
    return result

print(preprocess(tokenize("Lorem ipsum dolor sit AMET")))

from IPython.display import display, HTML
import re

def display_summary( id, extra_text='' ):
    """
    Function for printing a paper's summary through IPython's Rich Display System.
    Trims long titles or author lists, and links to the paper's  DOI (when available).
    """
    s = Summaries[ id ]

    title = ( s.title if s.title[-1]!='.' else s.title[:-1] )
    title = title[:150].rstrip() + ('' if len(title)<=150 else '...')
    if s.doi!='':
        title = '%s' % (s.doi, title)

    authors = ', '.join( s.authors[:5] ) + ('' if len(s.authors)<=5 else ', ...')

    lines = [
        title,
        authors,
        str(s.year),
        'id: %d%s' % (id, extra_text)
        ]

    display( HTML( '
%s
' % '
'.join(lines) ) )

def display_abstract( id, highlights=[]):
    """
    Function for displaying an abstract. Includes optional (naive) highlighting
    """
    a = Abstracts[ id ]
    for h in highlights:
        a = re.sub(r'\b(%s)\b'%h,'\\1',a, flags=re.IGNORECASE)
    display( HTML( '
%s</blockquote' % a ) )

display_summary(26488732)
display_abstract(26488732, ['sequencing'])


I am working on the file "02_building.ipynb".
The first task is:
Task 2
Construct a second function called or_query that works in the same way as and_query you just implemented, but returns documents that contain at least one of the words in the query. Demonstrate the working of this second function also with an example (again, choose one that leads to fewer than 100 hits).
Can you write Python code for it?


I am working on the file "02_building.ipynb".
The first task is:
We can now use this inverted index to answer simple one-word queries, for example to show all papers that contain the word 'amsterdam':
Can you write Python code for it?

query_word = 'amsterdam'
for i in inverted_index[query_word]:
    display_summary(i)


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
3D Placeholder
Can you write Python code for it?

image = tf.placeholder("uint8", [None, None, 3])

reverse = tf.reverse(image, [True, False]) # [True, False, False]

with tf.Session() as session:
    result = session.run(reverse, feed_dict={image: raw_image_data})
    print(result.shape)

plt.imshow(result)
plt.show()


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
Feed Dictionaries

Variable Scope

Complicated tensorFlow models can have 100's of variables

tf.variable_scope() provides simple name-spacing to avoid clashes
tf.get_variable() creates/accesses variables from withing a variable scope

Variable scope is a simple type of namespacing that adds prefixes to variable names within scope
Can you write Python code for it?

# with tf.variable_scope("foo"):
#    with tf.variable_scope("bar"):
#        v = tf.get_variable("v", [1])
# assert v.name == "foo/bar/v:0"


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
Fetching Variable State
Can you write Python code for it?

input1 = tf.constant(3.0)

input2 = tf.constant(2.0)

input3 = tf.constant(5.0)

intermed = tf.add(input2, input3)

mul = tf.multiply(input1, intermed)

with tf.Session() as sess:
    result = sess.run([mul, intermed])
    print(result)

# calling sess.run(var) on a tf.Session() object
# retrives its value.

# if you want to retrieve multiple variables
# simultaneously can do
# like sess.run([var1, var2])


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
Get Variable

Behaviour of get_variable() depends on
reuse is set to false
create and return new variable

reuse is set to true
search for existing variable with given name
raise ValueError if none found
Can you write Python code for it?

#
# with tf.variable_scope("foo"):
#     v = tf.get_variable("v", [1])
# assert v.name == "foo/v:0"
#

#
# with tf.variable_scope("foo"):
#     v = tf.get_variable("v", [1])
# with tf.variable_scope("foo", reuse=True):
#    v1 = tf.get_variable("v", [1])
# assert v1 = v
#


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
NOTE-1

tf.InteractiveSession()
syntactic sugar for keeping a default session
open in jupyter or ipython

NOTE-2

session.run(c)
is an example of a tensorFlow Fetch
coming up soon

Computation Graph

Idea Repeated to stress..
Tensorflow program structured as
A Graph construction phase
A Graph execution phase
This uses a session to execute operations in the graph

All computations add nodes to global default graph

Variables

All tensors we have used previously have been constant tensors
None of the tensors we have used untill now were variables
Lets define our first ever tensor variable
Can you write Python code for it?

w1 = tf.ones((2,2))
w1

w2 = tf.Variable(tf.zeros((2,2)), name='weights')
w2

with tf.Session() as sess:
    # this is a tensor flow constant
    print(sess.run(w1))
    # this is a tensor flow variable
    # please note the init call
    sess.run(tf.global_variables_initializer())
    print(sess.run(w2))


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
Numpy to TensorFlow Dictionary

Explicit Evaluation
Can you write Python code for it?

# TensorFlow computations define a computation graph
# This means no numerical value until evaluated explicitly

a = np.zeros((2,2))

a

ta = tf.zeros((2,2))

print(a)

print(ta)

print(ta.eval())


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
Placeholders

Getting the data with tf.convert_to_tensor() is cool, but as you
  see it does not scale.
Use Dummy nodes that provide entry points for data to the computational graph
This takes us to tf.placeholder variables

Now we need a mapping from tf.placeholder variables or their names
  to data like numpy arrays, lists, etc..
This takes us to feed_dict
This is a python dictionary
Can you write Python code for it?

# define placeholder objects for data entry
input1 = tf.placeholder(tf.float32)
input2 = tf.placeholder(tf.float32)
output = tf.multiply(input1, input2)

with tf.Session() as sess:
    print(sess.run([output], feed_dict={input1:[7.], input2:[2.] }))

# fetch value of input from computation graph
# feed data into the compuation graph


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
Simple TensorFlow Scripts
TensorFlow Constants
Can you write Python code for it?

# this is our first tensorflow line of code
# 1. its a tensorflow constant !
# welcome to tensorflow land
x = tf.constant(35, name='x')


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
TensorFlow Variables
Stepping into TensorFlow-land from Numpy-land
Can you write Python code for it?

# create a
# 1, tenforflow constant (last time)
# 2. tensorflow variable (now)
x = tf.Variable(raw_image_data, name='x')

# tf.initialize_all_variables() was deprecated recently
model = tf.global_variables_initializer()

with tf.Session() as session:
    # perform a basic operation
    transpose_op = tf.transpose(x, perm=[1, 0, 2])
    session.run(model)
    result = session.run(transpose_op)

# he may not like it, but here is the transpose
plt.imshow(result)


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
TensorFlow variables must be initialized before they have values!
Please contrast with constant tensors.
Can you write Python code for it?

W = tf.Variable(tf.zeros((2,2)), name="weights")
# variable objects can be initialized from constants

R = tf.Variable(tf.random_normal((2,2)), name="random_weights")
# variable objects can be initialized from random values

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(W))
    print(sess.run(R))


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
Thinking in TensorFlow-Land
Can you write Python code for it?

import tensorflow as tf

tf.InteractiveSession()

a = tf.zeros((2,2))

a

b = tf.ones((2,2))

b

tf.reduce_sum(b, reduction_indices=1).eval()

a.get_shape()

tf.reshape(a, (1,4)).eval()


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
Updating Variable State
Can you write Python code for it?

state = tf.Variable(0, name="counter")

new_value = tf.add(state, tf.constant(1))
# think of this as doing
# new_value = state + 1

update = tf.assign(state, new_value)
# think of this as state = new_value

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(state))
    for _ in range(3):
        sess.run(update)
        print(sess.run(state))

#
# state = 0
#
# print(state)
# for _ in range(3)
#     state = state + 1
#     print(state)
#


I am working on the file "02_code/Warmup To GANs.ipynb".
The first task is:
Variable scope control variable reuse
Can you write Python code for it?

# with tf.variable_scope("foo"):
#    v = tf.get_variable("v", [1])
#    tf.get_variable_scope().reuse_variables()
#    v1 = tf.get_variable("v", [1])
# assert v1 == v


I am working on the file "02_code/gan.ipynb".
The first task is:
Let $p^*(\mathbf{x})$ represent the true data distribution.
The optimization problem used in GANs is
\begin{equation}
\min_\theta \max_\phi~
\mathbb{E}_{p^(\mathbf{x})} [ \log D(\mathbf{x}; \phi) ]
+ \mathbb{E}_{p(\mathbf{x}; \theta)} [ \log (1 - D(\mathbf{x}; \phi)) ].
\end{equation*}
This optimization problem is bilevel: it requires a minima solution
with respect to generative parameters and a maxima solution with
respect to discriminative parameters.
In practice, the algorithm proceeds by iterating gradient updates on
each. An
additional heuristic also modifies the objective function for the
generative model in order to avoid saturation of gradients
(I. J. Goodfellow, 2014).
Many sources of intuition exist behind GAN-style training. One, which
is the original motivation, is based on idea that the two neural
networks are playing a game. The discriminator tries to best
distinguish samples away from the generator. The generator tries
to produce samples that are indistinguishable by the discriminator.
The goal of training is to reach a Nash equilibrium.
Another source is the idea of casting unsupervised learning as
supervised learning
(M. U. Gutmann, Dutta, Kaski, & Corander, 2014; M. Gutmann & Hyvärinen, 2010).
This allows one to leverage the power of classification—a problem that
in recent years is (relatively speaking) very easy.
A third comes from classical statistics, where the discriminator is
interpreted as a proxy of the density ratio between the true data
distribution and the model
 (Mohamed & Lakshminarayanan, 2016; Sugiyama, Suzuki, & Kanamori, 2012). By augmenting an
original problem that may require the model's density with a
discriminator (such as maximum likelihood), one can recover the
original problem when the discriminator is optimal. Furthermore, this
approximation is very fast, and it justifies GANs from the perspective
of approximate inference.
In Edward, the GAN algorithm (GANInference) simply takes the
implicit density model on x as input, binded to its
realizations x_ph. In addition, a...
Can you write Python code for it?

inference = ed.GANInference(
    data={x: x_ph}, discriminator=discriminative_network)


I am working on the file "02_code/gan.ipynb".
The first task is:
Model
GANs posit generative models using an implicit mechanism. Given some
random noise, the data is assumed to be generated by a deterministic
function of that noise.
Formally, the generative process is
\begin{align}
\mathbf{\epsilon} &\sim p(\mathbf{\epsilon}), \
\mathbf{x} &= G(\mathbf{\epsilon}; \theta),
\end{align}
where $G(\cdot; \theta)$ is a neural network that takes the samples
$\mathbf{\epsilon}$ as input. The distribution
$p(\mathbf{\epsilon})$ is interpreted as random noise injected to
produce stochasticity in a physical system; it is typically a fixed
uniform or normal distribution with some latent dimensionality.
In Edward, we build the model as follows, using TensorFlow Slim to
specify the neural network. It defines a 3-layer fully connected neural
network and outputs a vector of length data_width $\times$ data_height with values in
$[0,1]$.
Can you write Python code for it?

def generative_network(eps):
  h1 = slim.fully_connected(eps, 128, activation_fn=tf.nn.relu)
  h2 = slim.fully_connected(h1, 128, activation_fn=tf.nn.relu)
  x = slim.fully_connected(h2, data_dim, activation_fn=tf.sigmoid)
  return x

with tf.variable_scope("Gen"):
  eps = Uniform(tf.zeros([M, d]) - 1.0, tf.ones([M, d]))
  x = generative_network(eps)


I am working on the file "02_code/gan.ipynb".
The first task is:
We now form the main loop which trains the GAN. At each iteration, it
takes a minibatch and updates the parameters according to the
algorithm. At every 1000 iterations, it will print progress and also
saves a figure of generated samples from the model.
Can you write Python code for it?

sess = ed.get_session()
tf.global_variables_initializer().run()

idx = np.random.randint(M, size=16)
i = 0
for t in range(inference.n_iter):
  if t % inference.n_print == 0:
    samples = sess.run(x)
    samples = samples[idx, ]

    fig = plot(samples)
    plt.savefig(os.path.join(IMG_DIR, '{}.png').format(
        str(i).zfill(3)), bbox_inches='tight')
    plt.close(fig)
    i += 1

#   x_batch, _ = mnist.train.next_batch(M)
  x_batch = X[np.random.choice(np.arange(X.shape[0]), size=M)]
  info_dict = inference.update(feed_dict={x_ph: x_batch})
  inference.print_progress(info_dict)


I am working on the file "02_code/tensorflow perceptron.ipynb".
The first task is:
Perceptron (No cheating, for real)

Attempting to write our own update/optimize at each step
Can you write Python code for it?

margin = 0.1

x = tf.placeholder("float", [None, n_input])
y = tf.placeholder("float", [None, n_classes])

w = tf.Variable(tf.random_normal([n_input, n_classes]))

init = tf.global_variables_initializer()

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(training_epochs):
        avg_cost = 0.
        total_batch = int(train_x.shape[0]/batch_size)

        for i in range(total_batch):
            start = i * batch_size
            end = start + batch_size
            batch_x = train_x[start:end]
            batch_y = train_y[start:end]

            loss = tf.reduce_mean(tf.maximum(0., -y * tf.matmul(x, w), b))
            is_mistake = tf.less_equal(y * tf.matmul(x, w), margin)
            eta = (margin - (y[is_mistake] * tf.matmul(x[is_mistake], w))) / (tf.matmul(x[is_mistake], tf.transpose(x[is_mistake])) + 1)
            update = tf.assign(w, eta * tf.matmul(x[is_mistake], y[is_mistake]))

            _, c = sess.run([update, loss])

            avg_cost += c / total_batch

        if epoch % display_step == 0:
            print("Epoch:", "%04d" % (epoch+1), "cost=", "{:.9f}".format(avg_cost))

    print("Finished")

    correct_prediction = tf.equal(tf.argmax(perceptron, 1), tf.argmax(y, 1))

    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
    print("Accuracy:", accuracy.eval({x: test_x, y: test_y}))


I am working on the file "02_code/tensorflow perceptron.ipynb".
The first task is:
Perceptron (No cheating, mostly)
Perceptron loss
$$
\text{loss} := \min(0, -y * (\mathbf{X}w + b))
$$

Using our own perceptron loss
Using the gradient descent optimizer (partial cheating, but useful in practice)
Can you write Python code for it?

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
import numpy as np

classification_x, classification_y = make_classification(1000, n_features=100, n_informative=30)
classification_y = np.expand_dims(classification_y, axis=1)

print(classification_x.shape, classification_y.shape)

train_x, test_x, train_y, test_y = train_test_split(classification_x, classification_y, test_size=0.33)

n_input = classification_x.shape[1]
n_classes = classification_y.shape[1]

x = tf.placeholder("float", [None, n_input])
y = tf.placeholder("float", [None, n_classes])

w = tf.Variable(tf.random_normal([n_input, n_classes]))
b = tf.Variable(tf.random_normal([n_classes]))

perceptron = tf.add(tf.matmul(x, w), b)

perceptron_loss = tf.reduce_mean(tf.maximum(0., -y * tf.add(tf.matmul(x, w), b)))

optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(perceptron_loss)

init = tf.global_variables_initializer()

training_epochs = 100
learning_rate = 0.001
display_step = 10

with tf.Session() as sess:
    sess.run(init)

    for epoch in range(training_epochs):
        avg_cost = 0.
        total_batch = int(train_x.shape[0]/batch_size)

        for i in range(total_batch):
            start = i * batch_size
            end = start + batch_size
            batch_x = train_x[start:end]
            batch_y = train_y[start:end]

            _, c = sess.run([optimizer, perceptron_loss], feed_dict={x: batch_x, y: batch_y})

            avg_cost += c / total_batch

        if epoch % display_step == 0:
            print("Epoch:", "%04d" % (epoch+1), "cost=", "{:.9f}".format(avg_cost))

    print("Finished")

    correct_prediction = tf.equal(tf.argmax(perceptron, 1), tf.argmax(y, 1))

    accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))
    print("Accuracy:", accuracy.eval({x: test_x, y: test_y}))


I am working on the file "02_data_representation/embeddings.ipynb".
The first task is:
Build DNN model using TF-Hub Embedding layer
Next, we can add this TF-Hub module to our DNN model.
Can you write Python code for it?

def build_hub_model():
    model = models.Sequential([
        KerasLayer(handle=NNLM,
                   output_shape=[50],
                   input_shape=[],
                   dtype=tf.string,
                   trainable=True),
        layers.Dense(N_CLASSES, activation='softmax')
    ])

    model.compile(
        optimizer='adam',
        loss='categorical_crossentropy',
        metrics=['accuracy']
    )
    return model

%%time

tf.random.set_seed(33)

MODEL_DIR = os.path.join(LOGDIR, 'hub')
shutil.rmtree(MODEL_DIR, ignore_errors=True)

BATCH_SIZE = 300
EPOCHS = 100
EMBED_DIM = 10
PATIENCE = 3

hub_model = build_hub_model()

hub_history = hub_model.fit(
    X_train, Y_train,
    epochs=EPOCHS,
    batch_size=BATCH_SIZE,
    validation_data=(X_valid, Y_valid),
    callbacks=[callbacks.EarlyStopping(patience=PATIENCE),
               callbacks.TensorBoard(MODEL_DIR)],
)

pd.DataFrame(hub_history.history)[['loss', 'val_loss']].plot()
pd.DataFrame(hub_history.history)[['accuracy', 'val_accuracy']].plot()

hub_model.summary()


I am working on the file "02_data_representation/embeddings.ipynb".
The first task is:
Build a DNN model
The build_dnn_model function below returns a compiled Keras model that implements a simple embedding layer transforming the word integers into dense vectors, followed by a Dense softmax layer that returns the probabilities for each class.
Note that we need to put a custom Keras Lambda layer in between the Embedding layer and the Dense softmax layer to do an average of the word vectors returned by the embedding layer. This is the average that's fed to the dense softmax layer. By doing so, we create a model that is simple but that loses information about the word order, creating a model that sees sentences as "bag-of-words".
Can you write Python code for it?

def build_dnn_model(embed_dim):

    model = models.Sequential([
        layers.Embedding(VOCAB_SIZE + 1,
                         embed_dim,
                         input_shape=[MAX_LEN]),
        layers.Lambda(lambda x: tf.reduce_mean(x, axis=1)),
        layers.Dense(N_CLASSES, activation='softmax')
    ])

    model.compile(
        optimizer='adam',
        loss='categorical_crossentropy',
        metrics=['accuracy']
    )
    return model

Y_train.shape

%%time

tf.random.set_seed(33)

MODEL_DIR = os.path.join(LOGDIR, 'dnn')
shutil.rmtree(MODEL_DIR, ignore_errors=True)

BATCH_SIZE = 300
EPOCHS = 100
EMBED_DIM = 10
PATIENCE = 0

dnn_model = build_dnn_model(embed_dim=EMBED_DIM)

dnn_history = dnn_model.fit(
    X_train, Y_train,
    epochs=EPOCHS,
    batch_size=BATCH_SIZE,
    validation_data=(X_valid, Y_valid),
    callbacks=[callbacks.EarlyStopping(patience=PATIENCE),
               callbacks.TensorBoard(MODEL_DIR)],
)

pd.DataFrame(dnn_history.history)[['loss', 'val_loss']].plot()
pd.DataFrame(dnn_history.history)[['accuracy', 'val_accuracy']].plot()

dnn_model.summary()


I am working on the file "02_data_representation/embeddings.ipynb".
The first task is:
Convert the plurality to a numeric index.
Can you write Python code for it?

plurality_class = [CLASSES[plurality] for plurality in df.plurality]

print(df.plurality[:5])
print(plurality_class[:5])


I am working on the file "02_data_representation/embeddings.ipynb".
The first task is:
Create an embedding layer. Supply arguments input_dim and output_dim
 - input_dim indicates the size of the vocabulary. For plurality this is 6.
 - ouptut_dim indicates the dimension of the dense embedding.
Can you write Python code for it?

EMBED_DIM = 2

embedding_layer = layers.Embedding(input_dim=N_CLASSES,
                                   output_dim=EMBED_DIM)
embeds = embedding_layer(tf.constant(plurality_class))


I am working on the file "02_data_representation/embeddings.ipynb".
The first task is:
Embedding layer for categorical data
Can you write Python code for it?

!head ./data/babyweight_sample.csv

df = pd.read_csv("./data/babyweight_sample.csv")
df.plurality.head(5)

df.plurality.unique()

CLASSES = {
    'Single(1)': 0,
    'Multiple(2+)': 1,
    'Twins(2)': 2,
    'Triplets(3)': 3,
    'Quadruplets(4)': 4,
    'Quintuplets(5)': 5
}
N_CLASSES = len(CLASSES)


I am working on the file "02_data_representation/embeddings.ipynb".
The first task is:
First, we'll find how many words we have in our dataset (VOCAB_SIZE), how many titles we have (DATASET_SIZE), and what the maximum length of the titles we have (MAX_LEN) is. Keras offers the Tokenizer class in its keras.preprocessing.text module to help us with this.
Can you write Python code for it?

tokenizer = Tokenizer()
tokenizer.fit_on_texts(titles_df.title)
integerized_titles = tokenizer.texts_to_sequences(titles_df.title)


I am working on the file "02_data_representation/embeddings.ipynb".
The first task is:
The variable 'integerized_titles' contains the integer representation of each article title in out dataset.
Can you write Python code for it?


I am working on the file "02_data_representation/embeddings.ipynb".
The first task is:
The variable embeds contains the two-dimensional for each plurality class.
Can you write Python code for it?


I am working on the file "02_data_representation/embeddings.ipynb".
The first task is:
Then, prepare the data for the model.
Can you write Python code for it?

X_train, Y_train = create_sequences(titles_train), encode_labels(sources_train)
X_valid, Y_valid = create_sequences(titles_valid), encode_labels(sources_valid)

X_train[:3], Y_train[:3]


I am working on the file "02_data_representation/embeddings.ipynb".
The first task is:
Transfer Learning with Pre-trained Embedding
We can also use a word embedding from a pre-trained modle using a Neural Probabilistic Language Model. TF-Hub has a 50-dimensional one called
nnlm-en-dim50-with-normalization, which also normalizes the vectors produced.
Once loaded from its url, the TF-hub module can be used as a normal Keras layer in a sequential or functional model. Since we have enough data to fine-tune the parameters of the pre-trained embedding itself, we will set trainable=True in the KerasLayer that loads the pre-trained embedding:
Can you write Python code for it?

NNLM = "https://tfhub.dev/google/nnlm-en-dim50/2"

nnlm_module = KerasLayer(
    handle=NNLM,
    output_shape=[50],
    input_shape=[],
    dtype=tf.string,
    trainable=True)


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
Create datasets for training and evaluation
Can you write Python code for it?

%%bigquery
CREATE OR REPLACE TABLE
    babyweight.babyweight_data AS
SELECT
    weight_pounds,
    CAST(is_male AS STRING) AS is_male,
    mother_age,
    CASE
        WHEN plurality = 1 THEN "Single(1)"
        WHEN plurality = 2 THEN "Twins(2)"
        WHEN plurality = 3 THEN "Triplets(3)"
        WHEN plurality = 4 THEN "Quadruplets(4)"
        WHEN plurality = 5 THEN "Quintuplets(5)"
    END AS plurality,
    gestation_weeks,
    CAST(mother_race AS STRING) AS mother_race,
    FARM_FINGERPRINT(
        CONCAT(
            CAST(year AS STRING),
            CAST(month AS STRING)
        )
    ) AS hashmonth
FROM
    publicdata.samples.natality
WHERE
    year > 2000
    AND weight_pounds > 0
    AND mother_age > 0
    AND plurality > 0
    AND gestation_weeks > 0


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
Create feature columns for inputs
Next, define the feature columns. mother_age and gestation_weeks should be numeric. The others, is_male, plurality and mother_race, should be categorical. Remember, only dense feature columns can be inputs to a DNN.
The last feature column created in the create_feature_columns function is a feature cross with is_male and plurality. To implement a feature cross in Tensorflow we use tf.feature_column.crossed_column which takes two arguments: a list of the feature keys to be crossed and the hash bucket size. Crossed features will be hashed according to hash_bucket_size so it should be large enough to accommodate all possible crossed categories. Since the feature is_male can take 3 values (True, False or Unknown) and the feature plurality can take 6 values (Single(1), Twins(2), Triplets(3), Quadruplets(4), Quintuplets(5), Multiple(2+)), we'll set hash_bucket_size=18.
Finally, to use crossed column in DNN model, you need to wrap it either in an indicator_column or an embedding_column. In the code below, we use an embedding column and take the embedding dimension to be 2.
To create a crossed column with features of numeric type, you can use categorical_column, or bucketized_column before passing to a crossed_column.
Can you write Python code for it?

def categorical_fc(name, values):
    cat_column = fc.categorical_column_with_vocabulary_list(
            key=name, vocabulary_list=values)

    return fc.indicator_column(categorical_column=cat_column)

def create_feature_columns():
    feature_columns = {
        colname : fc.numeric_column(key=colname)
           for colname in ["mother_age", "gestation_weeks"]
    }

    feature_columns["is_male"] = categorical_fc(
        "is_male", ["True", "False", "Unknown"])
    feature_columns["plurality"] = categorical_fc(
        "plurality", ["Single(1)", "Twins(2)", "Triplets(3)",
                      "Quadruplets(4)", "Quintuplets(5)", "Multiple(2+)"])
    feature_columns["mother_race"] = fc.indicator_column(
        fc.categorical_column_with_hash_bucket(
            "mother_race", hash_bucket_size=17, dtype=tf.dtypes.string))

    feature_columns["gender_x_plurality"] = fc.embedding_column(
        fc.crossed_column(["is_male", "plurality"], hash_bucket_size=18),
        dimension=2)

    return feature_columns


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
Create model in BigQuery
Can you write Python code for it?

%%bigquery
CREATE OR REPLACE MODEL `babyweight.natality_model`
OPTIONS
  (MODEL_TYPE="DNN_REGRESSOR",
    HIDDEN_UNITS=[64, 32],
    BATCH_SIZE=32,
    INPUT_LABEL_COLS=["weight_pounds"],
    DATA_SPLIT_METHOD="NO_SPLIT") AS
SELECT
  weight_pounds,
  is_male,
  plurality,
  gestation_weeks,
  mother_age,
  CAST(mother_race AS string) AS mother_race
FROM
  babyweight.babyweight_data_train


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
Creating a Feature Cross with BQML
Next, we'll create a feature cross of the features is_male and mother_race. To create a feature cross we apply ML.FEATURE_CROSS to a STRUCT of the features is_male and mother_race cast as a string.
The STRUCT clause creates an ordered pair of the two features. The TRANSFORM clause is used for engineering features of our model. This allows us to specify all preprocessing during model creation and apply those preprocessing steps during prediction and evaluation. The rest of the features within the TRANSFORM clause remain unchanged.
Can you write Python code for it?

%%bigquery
CREATE OR REPLACE MODEL `babyweight.natality_model_feat_eng`
TRANSFORM(weight_pounds,
    is_male,
    plurality,
    gestation_weeks,
    mother_age,
    CAST(mother_race AS string) AS mother_race,
    ML.FEATURE_CROSS(
            STRUCT(
                is_male,
                plurality)
        ) AS gender_X_plurality)
OPTIONS
  (MODEL_TYPE='linear_reg',
   INPUT_LABEL_COLS=['weight_pounds'],
   DATA_SPLIT_METHOD="NO_SPLIT") AS
SELECT
  *
FROM
    babyweight.babyweight_data_train


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
Define a DNN model
Next we define our model. This is regression so make sure the output layer activation is correct and that the shape is right. We'll create deep neural network model, similar to what we use in BigQuery.
Can you write Python code for it?

def get_model_outputs(inputs):
    # Create two hidden layers of [64, 32] just in like the BQML DNN
    h1 = layers.Dense(64, activation="relu", name="h1")(inputs)
    h2 = layers.Dense(32, activation="relu", name="h2")(h1)

    # Final output is a linear activation because this is regression
    output = layers.Dense(units=1, activation="linear", name="weight")(h2)

    return output

def rmse(y_true, y_pred):
    return tf.sqrt(tf.reduce_mean((y_pred - y_true) ** 2))


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
Feature Crosses in BigQuery
We'll first explore how to create a feature cross in BigQuery. The cell below will create a dataset called babyweight in your GCP project, if it does not already exist. This dataset will will house our tables and models.
Can you write Python code for it?

bq = bigquery.Client()
dataset = bigquery.Dataset(bq.dataset("babyweight"))

try:
    bq.create_dataset(dataset)
    print("Dataset created.")
except:
    print("Dataset already exists.")


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
Finally, we will build the model using tf.keras.models.Model giving our inputs and outputs and then compile our model with an optimizer, a loss function, and evaluation metrics.
Can you write Python code for it?

def build_dnn_model():
    """Builds simple DNN using Keras Functional API.

    Returns:
        `tf.keras.models.Model` object.
    """
    # Create input layer
    inputs = create_input_layers()

    # Create feature columns
    feature_columns = create_feature_columns()

    # The constructor for DenseFeatures takes a list of numeric columns
    # The Functional API in Keras requires: LayerConstructor()(inputs)
    dnn_inputs = layers.DenseFeatures(
        feature_columns=feature_columns.values())(inputs)

    # Get output of model given inputs
    output = get_model_outputs(dnn_inputs)

    # Build model and compile it all together
    model = tf.keras.models.Model(inputs=inputs, outputs=output)
    model.compile(optimizer="adam", loss="mse", metrics=[rmse, "mse"])

    return model

print("Here is our DNN architecture so far:\n")
model = build_dnn_model()
print(model.summary())

tf.keras.utils.plot_model(
    model=model, to_file="dnn_model.png", show_shapes=False, rankdir="LR")


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
Make a dataset of features and label.
Can you write Python code for it?

def features_and_labels(row_data):
    """Splits features and labels from feature dictionary.
    Args:
        row_data: Dictionary of CSV column names and tensor values.
    Returns:
        Dictionary of feature tensors and label tensor.
    """
    label = row_data.pop(LABEL_COLUMN)

    return row_data, label

def load_dataset(pattern, batch_size=1, mode=tf.estimator.ModeKeys.EVAL):
    """Loads dataset using the tf.data API from CSV files.
    Args:
        pattern: str, file pattern to glob into list of files.
        batch_size: int, the number of examples per batch.
        mode: tf.estimator.ModeKeys to determine if training or evaluating.
    Returns:
        `Dataset` object.
    """
    # Make a CSV dataset
    dataset = tf.data.experimental.make_csv_dataset(
        file_pattern=pattern,
        batch_size=batch_size,
        column_names=CSV_COLUMNS,
        column_defaults=DEFAULTS)

    # Map dataset to features and label
    dataset = dataset.map(map_func=features_and_labels)  # features, label

    # Shuffle and repeat for training
    if mode == tf.estimator.ModeKeys.TRAIN:
        dataset = dataset.shuffle(buffer_size=1000).repeat()

    # Take advantage of multi-threading; 1=AUTOTUNE
    dataset = dataset.prefetch(buffer_size=1)

    return dataset


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
Need for regularization
Let's use a high-cardinality feature cross to illustrate the point. In this model, we are predicting taxifare in New York city using a feature cross of lat and lon
Can you write Python code for it?

!bq show mlpatterns || bq mk mlpatterns

%%bigquery
CREATE OR REPLACE TABLE mlpatterns.taxi_data AS

SELECT
  (tolls_amount + fare_amount) AS fare_amount,
  pickup_datetime,
  pickup_longitude AS pickuplon,
  pickup_latitude AS pickuplat,
  dropoff_longitude AS dropofflon,
  dropoff_latitude AS dropofflat,
  passenger_count*1.0 AS passengers
FROM `nyc-tlc.yellow.trips`
# The full dataset has 1+ Billion rows, let's take only 1 out of 1,000 (or 1 Million total)
WHERE ABS(MOD(FARM_FINGERPRINT(CAST(pickup_datetime AS STRING)), 1000)) = 1
AND
  trip_distance > 0
  AND fare_amount >= 2.5
  AND pickup_longitude > -78
  AND pickup_longitude < -70
  AND dropoff_longitude > -78
  AND dropoff_longitude < -70
  AND pickup_latitude > 37
  AND pickup_latitude < 45
  AND dropoff_latitude > 37
  AND dropoff_latitude < 45
  AND passenger_count > 0

%%bigquery
CREATE OR REPLACE MODEL mlpatterns.taxi_noreg
TRANSFORM(
  fare_amount
  , ML.FEATURE_CROSS(STRUCT(CAST(EXTRACT(DAYOFWEEK FROM pickup_datetime) AS STRING) AS dayofweek,
                            CAST(EXTRACT(HOUR FROM pickup_datetime) AS STRING) AS hourofday), 2) AS day_hr
  , CONCAT(
     ML.BUCKETIZE(pickuplon, GENERATE_ARRAY(-78, -70, 0.01)),
     ML.BUCKETIZE(pickuplat, GENERATE_ARRAY(37, 45, 0.01)),
     ML.BUCKETIZE(dropofflon, GENERATE_ARRAY(-78, -70, 0.01)),
     ML.BUCKETIZE(dropofflat, GENERATE_ARRAY(37, 45, 0.01))
  ) AS pickup_and_dropoff
)
OPTIONS(input_label_cols=['fare_amount'], model_type='linear_reg')
AS

SELECT * FROM mlpatterns.taxi_data

%%bigquery
SELECT SQRT(mean_squared_error) AS rmse FROM ML.EVALUATE(MODEL mlpatterns.taxi_noreg)

%%bigquery
CREATE OR REPLACE MODEL mlpatterns.taxi_l2reg
TRANSFORM(
  fare_amount
 , ML.FEATURE_CROSS(STRUCT(CAST(EXTRACT(DAYOFWEEK FROM pickup_datetime) AS STRING) AS dayofweek,
                            CAST(EXTRACT(HOUR FROM pickup_datetime) AS STRING) AS hourofday), 2) AS day_hr
  , CONCAT(
     ML.BUCKETIZE(pickuplon, GENERATE_ARRAY(-78, -70, 0.01)),
     ML.BUCKETIZE(pickuplat, GENERATE_ARRAY(37, 45, 0.01)),
  ...


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
Next, we'll create tables in BigQuery that we'll use for training and evaluation.
Can you write Python code for it?

%%bigquery
CREATE OR REPLACE TABLE
    babyweight.babyweight_data_train AS
SELECT
    weight_pounds,
    is_male,
    mother_age,
    plurality,
    gestation_weeks,
    mother_race
FROM
    babyweight.babyweight_data
WHERE
    ABS(MOD(hashmonth, 4)) < 3

%%bigquery
CREATE OR REPLACE TABLE
    babyweight.babyweight_data_eval AS
SELECT
    weight_pounds,
    is_male,
    mother_age,
    plurality,
    gestation_weeks,
    mother_race
FROM
    babyweight.babyweight_data
WHERE
    ABS(MOD(hashmonth, 4)) = 3


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
Train and evaluate our model
We've built our Keras model using our inputs from our CSV files and the architecture we designed. Let's now run our model by training our model parameters and periodically running an evaluation to track how well we are doing on outside data as training goes on. We'll need to load both our train and eval datasets and send those to our model through the fit method. Make sure you have the right pattern, batch size, and mode when loading the data.
Can you write Python code for it?

%%time

tf.random.set_seed(33)

TRAIN_BATCH_SIZE = 32
NUM_TRAIN_EXAMPLES = 1000 * 5  # training dataset repeats, it'll wrap around
NUM_EVALS = 5  # how many times to evaluate
# Enough to get a reasonable sample, but not so much that it slows down
NUM_EVAL_EXAMPLES = 1000

trainds = load_dataset(
    pattern="./data/babyweight_train*",
    batch_size=TRAIN_BATCH_SIZE,
    mode=tf.estimator.ModeKeys.TRAIN)

evalds = load_dataset(
    pattern="./data/babyweight_eval*",
    batch_size=1000,
    mode=tf.estimator.ModeKeys.EVAL).take(count=NUM_EVAL_EXAMPLES // 1000)

steps_per_epoch = NUM_TRAIN_EXAMPLES // (TRAIN_BATCH_SIZE * NUM_EVALS)

logdir = os.path.join(
    "logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
tensorboard_callback = tf.keras.callbacks.TensorBoard(
    log_dir=logdir, histogram_freq=1)

history = model.fit(
    trainds,
    validation_data=evalds,
    epochs=NUM_EVALS,
    steps_per_epoch=steps_per_epoch,
    callbacks=[tensorboard_callback])


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
We can use ML.EVALUATE to determine the root mean square error of our model on the evaluation set.
Can you write Python code for it?

query = """
SELECT
  *, SQRT(mean_squared_error) AS rmse
FROM
  ML.EVALUATE(MODEL `babyweight.natality_model`,
    (
    SELECT
      weight_pounds,
      is_male,
      plurality,
      gestation_weeks,
      mother_age,
      CAST(mother_race AS STRING) AS mother_race
    FROM
      babyweight.babyweight_data_eval ))
"""

df = bq.query(query).to_dataframe()
df.head()


I am working on the file "02_data_representation/feature_cross.ipynb".
The first task is:
We'll need to get the data read in by our input function to our model function, but just how do we go about connecting the dots? We can use Keras input layers (tf.Keras.layers.Input).
Can you write Python code for it?

def create_input_layers():
    """Creates dictionary of input layers for each feature.

    Returns:
        Dictionary of `tf.Keras.layers.Input` layers for each feature.
    """
    inputs = {
        colname: tf.keras.layers.Input(
            name=colname, shape=(), dtype="float32")
        for colname in ["mother_age", "gestation_weeks"]}

    inputs.update({
        colname: tf.keras.layers.Input(
            name=colname, shape=(), dtype="string")
        for colname in ["is_male", "plurality", "mother_race"]})

    return inputs


I am working on the file "02_data_representation/hashed_feature.ipynb".
The first task is:
Airports that share hash buckets
Which airports have the same hashbucket as Chicago, assuming 100 hash buckets?
Can you write Python code for it?

%%bigquery
CREATE TEMPORARY FUNCTION hashed(airport STRING, numbuckets INT64) AS (
   ABS(MOD(FARM_FINGERPRINT(airport), numbuckets))
);

WITH airports AS (
SELECT
   departure_airport, COUNT(1) AS num_flights
FROM `bigquery-samples.airline_ontime_data.flights`
GROUP BY departure_airport
)

SELECT
   departure_airport, num_flights
FROM airports
WHERE hashed(departure_airport, 100) = hashed('ORD', 100)


I am working on the file "02_data_representation/hashed_feature.ipynb".
The first task is:
Some airports had very few flights
Can you write Python code for it?

%%bigquery
SELECT
   departure_airport, COUNT(1) AS num_flights
FROM `bigquery-samples.airline_ontime_data.flights`
GROUP BY departure_airport
ORDER BY num_flights ASC
LIMIT 10


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Combining text and tabular inputs
To demonstrate this we'll use a toy example dataset of restaurant reviews, defined below. For demo purposes, we've only defined the training dataset here.
Can you write Python code for it?

reviews_data = {
    "review_text": ["The food was great, but it took forever to get seated.", "The tacos were life changing.", "This food made me question the presence of my taste buds."],
    "meal_type": ["lunch", "dinner", "dinner"],
    "meal_total": [50, 75, 60],
    "rating": [4, 5, 1]
}


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Extracting tabular features from text
We'll create a new dataset of Stack Overflow questions, this time adding some tabular features extracted from the text and changing the prediction task to whether a question is answered.
Step 1: Get the data in BigQuery. Remember to replace your-project below with the name of our GCP project.
Can you write Python code for it?

%%bigquery df_tabular --project your-project
SELECT
  title,
  answer_count,
  LENGTH(title) AS title_len,
  ARRAY_LENGTH(SPLIT(title, " ")) AS word_count,
  ENDS_WITH(title, "?") AS ends_with_q_mark,
  REPLACE(tags, "|", ",") as tags,
  IF
    (answer_count > 0,
      1,
      0) AS is_answered
FROM
  `bigquery-public-data.stackoverflow.posts_questions`
WHERE
  REGEXP_CONTAINS( tags, r"(?:keras|matplotlib|pandas)")
LIMIT 1000

df_tabular.head()


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Mixed image representations
Note: there is no dataset used here, but the example below provides a demo of how you could create a model that merges both pixel value and tiled representations.
Step 1: Define the pixel value and tiled representation layers
Can you write Python code for it?

# Define image input layer (same shape for both pixel and tiled representation)
image_input = Input(shape=(28,28,1))

# Define pixel representation
pixel_layer = Flatten()(image_input)

# Define tiled representation
tiled_layer = Conv2D(filters=16, kernel_size=3, activation='relu')(image_input)
tiled_layer = MaxPooling2D()(tiled_layer)
tiled_layer = tf.keras.layers.Flatten()(tiled_layer)


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Step 1: Concatenate all tabular features
Can you write Python code for it?

tabular_image_features = np.concatenate((
    np.expand_dims(tabular_image_metadata['time'], axis=1),
    np.expand_dims(tabular_image_metadata['visibility'], axis=1),
    np.array(tabular_image_metadata['inclement_weather']),
    np.array(tabular_image_metadata['location'])
), axis=1)

# Preview the data
tabular_image_features


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Step 1: Get the data from BigQuery and save it to a Pandas DataFrame.
Be sure to replace your-project below with the name of your Google Cloud project.
Can you write Python code for it?

%%bigquery df --project your-project
SELECT
  title,
  answer_count,
  REPLACE(tags, "|", ",") as tags
FROM
  `bigquery-public-data.stackoverflow.posts_questions`
WHERE
  REGEXP_CONTAINS( tags, r"(?:keras|matplotlib|pandas)")
LIMIT 1000

# Preview the dataset, note that we'll use some of the
df.head()


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Step 1: process review_text so it can be fed in as an embedding
Can you write Python code for it?

vocab_size = 50
tokenize = keras.preprocessing.text.Tokenizer(num_words=vocab_size)
tokenize.fit_on_texts(reviews_data['review_text'])

reviews_train = tokenize.texts_to_sequences(reviews_data['review_text'])
max_sequence_len = 20
reviews_train = keras.preprocessing.sequence.pad_sequences(reviews_train, maxlen=max_sequence_len, padding='post')

print(reviews_train)


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Step 2: Bucket the data and create a new input array with both the numeric and bucketed (boolean) rating
Can you write Python code for it?

rating_processed = []

for i in reviews_data['rating']:
  rating_processed.append([i, good_or_bad(i)])

print(rating_processed)


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Step 2: Define the vocab size and max sequence length, and create an instance of the Tokenizer class, fitting it on our Stack Overflow data.
Can you write Python code for it?

stacko_vocab_size = 200
stacko_sequence_len = 40

# Create a tokenizer for this data
stacko_tokenize = keras.preprocessing.text.Tokenizer(num_words=stacko_vocab_size)
stacko_tokenize.fit_on_texts(df['title'].values)

# Preview the first 20 words in the Tokenizer's vocabulary
list(stacko_tokenize.word_index.keys())[:20]


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Step 2: Extract the tabular features
Can you write Python code for it?

stacko_tabular_features = df_tabular[['title_len', 'word_count', 'ends_with_q_mark']]
stacko_tabular_features['ends_with_q_mark'] = stacko_tabular_features['ends_with_q_mark'].astype(int)

stacko_tabular_features.head()


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Step 2: convert meal_type to one-hot
Can you write Python code for it?

possible_meal_vocab = ['breakfast', 'lunch', 'dinner']
one_hot_meals = []

for i in reviews_data['meal_type']:
  one_hot_arr = [0] * len(possible_meal_vocab)
  one_index = possible_meal_vocab.index(i)
  one_hot_arr[one_index] = 1
  one_hot_meals.append(one_hot_arr)


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Step 3: Merge the tabular layer with the tiled layer we defined above
Can you write Python code for it?

mixed_image_layers = keras.layers.concatenate([image_tabular_layer, tiled_layer])

merged_image_dense = Dense(16, activation='relu')(mixed_image_layers)
merged_image_output = Dense(1)(merged_image_dense)

mixed_image_tabular_model = Model(inputs=[image_tabular_input, tiled_input], outputs=merged_image_output)

mixed_image_tabular_model.summary()


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Step 4: Create the Bag of Words representation
Can you write Python code for it?

questions_train_matrix = stacko_tokenize.texts_to_matrix(df['title'].values)
print(questions_train_matrix[0])


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Step 5: concatenate the layers into a model
Can you write Python code for it?

merged_input = keras.layers.concatenate([embedding_layer, tabular_layer])
merged_dense = Dense(16)(merged_input)
output = Dense(1)(merged_dense)

model = Model(inputs=[embedding_input, tabular_input], outputs=output)

# Preview the model architecture
model.summary()


I am working on the file "02_data_representation/mixed_representation.ipynb".
The first task is:
Tabular data multiple ways
We'll build on the review dataset example in the previous section to show a quick demo of how you might represent rating as a feature in two different ways.
Step 1: write a function to bucket ratings using a threshold
Can you write Python code for it?

def good_or_bad(rating):
  if rating > 3:
    return 1
  else:
    return 0


I am working on the file "02_data_representation/simple_data_representation.ipynb".
The first task is:
Let's look at the data after applying different forms of scaling.
Can you write Python code for it?

base_sql = """
CREATE TEMPORARY FUNCTION CLIP_LESS(x FLOAT64, a FLOAT64) AS (
  IF (x < a, a, x)
);
CREATE TEMPORARY FUNCTION CLIP_GT(x FLOAT64, b FLOAT64) AS (
  IF (x > b, b, x)
);
CREATE TEMPORARY FUNCTION CLIP(x FLOAT64, a FLOAT64, b FLOAT64) AS (
  CLIP_GT(CLIP_LESS(x, a), b)
);

WITH stats AS (
    SELECT
      MIN(mother_age) AS min_age,
      MAX(mother_age) AS max_age,
      AVG(mother_age) AS avg_age,
      STDDEV(mother_age) AS stddev_age,
      APPROX_QUANTILES(mother_age, 100)[OFFSET(1)] AS percentile_1,
      APPROX_QUANTILES(mother_age, 100)[OFFSET(99)] AS percentile_99
    FROM
      publicdata.samples.natality
    WHERE
      year > 2000
),

scaling AS (
    SELECT
      mother_age,
      weight_pounds,
      SAFE_DIVIDE(2*mother_age - max_age - min_age, max_age - min_age) AS minmax_scaled,
      CLIP( (mother_age - 30)/15, -1, 1 ) AS clipped,
      SAFE_DIVIDE(mother_age - avg_age, stddev_age) AS zscore,
      CLIP(mother_age, percentile_1, percentile_99) AS winsorized_1_99,
      SAFE_DIVIDE(2*CLIP(mother_age, percentile_1, percentile_99) - percentile_1 - percentile_99, percentile_99 - percentile_1) AS winsorized_scaled
    FROM
      publicdata.samples.natality, stats
)
"""

def scaled_stats(age_col):
    sql = base_sql + """
SELECT
   {0},
   AVG(weight_pounds) AS avg_wt,
   COUNT(1) AS num_babies
FROM
   scaling
GROUP BY {0}
ORDER BY {0}
    """.format(age_col)
    from google.cloud import bigquery
    return bigquery.Client().query(sql).to_dataframe()

import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = [15, 15]
plt.rcParams.update({'font.size': 15})
fig, axs = plt.subplots(3, 2);
scaled_stats('mother_age').plot(x='mother_age', y='num_babies', ax=axs[0, 0]);
scaled_stats('minmax_scaled').plot(x='minmax_scaled', y='num_babies', ax=axs[0, 1]);
scaled_stats('clipped').plot(x='clipped', y='num_babies', ax=axs[1, 0]);
scaled_stats('zscore').plot(x='zscore', y='num_babies', ax=axs[1, 1], xlim=[-2, 2]);
scaled_stats('winsorized_1_99').plot(x='winsorized_1_99', y='num_babies',...


I am working on the file "02_data_representation/simple_data_representation.ipynb".
The first task is:
Looking at the distribution (histogram) of the raw mother's age makes the weird behavior at the edges clear. We don't have enough data for mothers in their low-teens and in their fifties. In statistical terms, these are outliers.
Can you write Python code for it?


I am working on the file "02_data_representation/simple_data_representation.ipynb".
The first task is:
Skewed data
For an example of highly skewed data, assume that we are building a model to predict the likely sales of a non-fiction book. One of the inputs to the model is the popularity of the Wikipedia page corresponding to the topic. The number of views of pages in Wikipedia is highly skewed.
Can you write Python code for it?

%%bigquery df
WITH bypage AS (
SELECT
  title,
  SUM(views) AS num_views
FROM `bigquery-samples.wikipedia_benchmark.Wiki1M`
WHERE language = 'en'
GROUP BY title
HAVING num_views > 10 # non-niche
ORDER by num_views desc
),

percentile AS (
SELECT
  APPROX_QUANTILES(num_views, 100) AS bins
FROM
  bypage
)

SELECT
  title,
  num_views,
  (ROUND(POW(LOG(num_views), 0.25), 1) - 1.3) AS fourthroot_log_views,
  CAST(REPLACE(ML.BUCKETIZE(num_views, bins), 'bin_', '') AS int64) AS bin,
FROM
  percentile, bypage

from scipy import stats
data, est_lambda = stats.boxcox(df['num_views'])
df['boxcox'] = data

df

import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = [15, 10]
plt.rcParams.update({'font.size': 15})
fig, axs = plt.subplots(1, 4);

for axno, name in enumerate('num_views,fourthroot_log_views,bin,boxcox'.split(',')):
   df.hist(histtype='bar', bins=20, column=name, ax=axs[axno]);
fig.savefig('skew_log.png')
plt.close(fig)


I am working on the file "02_data_representation/text_embeddings.ipynb".
The first task is:
Create lookup table
Let's create a lookup table of embeddings. We'll use the comments field of a storm reports table from NOAA.
This is an example of the Feature Store design pattern.
Can you write Python code for it?

%%bigquery
CREATE OR REPLACE TABLE advdata.comments_embedding AS
SELECT
  output_0 as comments_embedding,
  comments
FROM ML.PREDICT(MODEL advdata.swivel_text_embed,(
  SELECT comments, LOWER(comments) AS sentences
  FROM `bigquery-public-data.noaa_preliminary_severe_storms.wind_reports`
))


I am working on the file "02_data_representation/text_embeddings.ipynb".
The first task is:
From the BigQuery web console, click on "schema" tab for the newly loaded model. We see that the input is called sentences and the output is called output_0:
<img src="swivel_schema.png" />
Can you write Python code for it?

%%bigquery
SELECT output_0 FROM
ML.PREDICT(MODEL advdata.swivel_text_embed,(
SELECT "Long years ago, we made a tryst with destiny; and now the time comes when we shall redeem our pledge, not wholly or in full measure, but very substantially." AS sentences))


I am working on the file "02_data_representation/text_embeddings.ipynb".
The first task is:
Let's load the model into a BigQuery dataset named advdata (create it if necessary)
Can you write Python code for it?

%%bigquery
CREATE OR REPLACE MODEL advdata.swivel_text_embed
OPTIONS(model_type='tensorflow', model_path='gs://ai-analytics-solutions-kfpdemo/swivel/*')


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
Both these have weather in approximately the same places. On Jan 1, things are more widespread than in July.
The date we searched for (May) is somewhere in between the Jan and July weather scenarios ...
To make the search work better, we should use smaller tiles (not the whole country, but perhaps 500kmx500km tiles). Then, we'll get mesoscale phenomena.
Embeddings for interpolation ...
Can we use the embeddings for interpolating between images?
Recall that the sqdist between subsequent hours is about 0.5
What happens if we use the image at t-1 and t+1 to get t=0?
What's the error?
Can you write Python code for it?

%%bigquery
WITH refl1 AS (
SELECT ref1_value, idx
FROM `ai-analytics-solutions.advdata.wxembed`,
     UNNEST(ref) AS ref1_value WITH OFFSET AS idx
WHERE time = '2019-09-20 05:00:00 UTC'
),

refl2 AS (
SELECT ref2_value, idx
FROM `ai-analytics-solutions.advdata.wxembed`,
     UNNEST(ref) AS ref2_value WITH OFFSET AS idx
WHERE time = '2019-09-20 06:00:00 UTC'
),

refl3 AS (
SELECT ref3_value, idx
FROM `ai-analytics-solutions.advdata.wxembed`,
     UNNEST(ref) AS ref3_value WITH OFFSET AS idx
WHERE time = '2019-09-20 07:00:00 UTC'
)

-- SELECT idx, ref1_value, ref2_value, ref3_value, ABS( ref2_value - (ref1_value + ref3_value)/2 ) AS diff
SELECT SUM( (ref2_value - (ref1_value + ref3_value)/2) * (ref2_value - (ref1_value + ref3_value)/2) ) AS sqdist
FROM refl1
JOIN refl2 USING (idx)
JOIN refl3 USING (idx)
-- ORDER BY idx ASC


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
Clustering the embeddings
If the differences between images are meaningful, then it makes sense that we could cluster the images using just the embeddings.
Let's do K-Means clustering into 5 categories and visualize the five centroids.
Can you write Python code for it?

# Unfortunately, BigQueryML does not accept arrays as input
# so we convert it into a struct. Generate the boilerplate code ...
def create_array_to_struct(N):
    sql = ("""
CREATE TEMPORARY FUNCTION arr_to_input(arr ARRAY<FLOAT64>)
RETURNS
STRUCT<"""
    + ', '.join(["u{} FLOAT64".format(idx+1) for idx in range(N)]) + ">\n"
    + "AS (STRUCT(\n"
    + ', '.join(["arr[OFFSET({})]".format(idx) for idx in range(N)])
    + "\n));"
    )
    return sql

print(create_array_to_struct(50))

%%bigquery

-- Unfortunately, BigQueryML does not accept arrays as input, so we convert it into a struct

CREATE TEMPORARY FUNCTION arr_to_input(arr ARRAY<FLOAT64>)
RETURNS
STRUCT<u1 FLOAT64, u2 FLOAT64, u3 FLOAT64, u4 FLOAT64, u5 FLOAT64,
u6 FLOAT64, u7 FLOAT64, u8 FLOAT64, u9 FLOAT64, u10 FLOAT64,
u11 FLOAT64, u12 FLOAT64, u13 FLOAT64, u14 FLOAT64, u15 FLOAT64,
u16 FLOAT64, u17 FLOAT64, u18 FLOAT64, u19 FLOAT64, u20 FLOAT64,

u21 FLOAT64, u22 FLOAT64, u23 FLOAT64, u24 FLOAT64, u25 FLOAT64,
u26 FLOAT64, u27 FLOAT64, u28 FLOAT64, u29 FLOAT64, u30 FLOAT64,

u31 FLOAT64, u32 FLOAT64, u33 FLOAT64, u34 FLOAT64, u35 FLOAT64,
u36 FLOAT64, u37 FLOAT64, u38 FLOAT64, u39 FLOAT64, u40 FLOAT64,

u41 FLOAT64, u42 FLOAT64, u43 FLOAT64, u44 FLOAT64, u45 FLOAT64,
u46 FLOAT64, u47 FLOAT64, u48 FLOAT64, u49 FLOAT64, u50 FLOAT64>

AS (
STRUCT(
    arr[OFFSET(0)], arr[OFFSET(1)], arr[OFFSET(2)], arr[OFFSET(3)], arr[OFFSET(4)]
    , arr[OFFSET(5)], arr[OFFSET(6)], arr[OFFSET(7)], arr[OFFSET(8)], arr[OFFSET(9)]
    , arr[OFFSET(10)], arr[OFFSET(11)], arr[OFFSET(12)], arr[OFFSET(13)], arr[OFFSET(14)]
    , arr[OFFSET(15)], arr[OFFSET(16)], arr[OFFSET(17)], arr[OFFSET(18)], arr[OFFSET(19)]
    , arr[OFFSET(20)], arr[OFFSET(21)], arr[OFFSET(22)], arr[OFFSET(23)], arr[OFFSET(24)]
    , arr[OFFSET(25)], arr[OFFSET(26)], arr[OFFSET(27)], arr[OFFSET(28)], arr[OFFSET(29)]
    , arr[OFFSET(30)], arr[OFFSET(31)], arr[OFFSET(32)], arr[OFFSET(33)], arr[OFFSET(34)]
    , arr[OFFSET(35)], arr[OFFSET(36)], arr[OFFSET(37)], arr[OFFSET(38)], arr[OFFSET(39)]
 ...


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
Create autoencoder in Keras
Can you write Python code for it?

## A model without the intermediate Dense layer, so that end result is effectively tiled
## We use more filters to represent the tiles

import tensorflow as tf
def create_model(nlayers=4, poolsize=4, numfilters=5, num_dense=0):
    input_img = tf.keras.Input(shape=(1059, 1799, 1), name='refc_input')

    x = tf.keras.layers.Cropping2D(cropping=((17, 18),(4, 3)), name='cropped')(input_img)
    last_pool_layer = None
    for layerno in range(nlayers):
        x = tf.keras.layers.Conv2D(2**(layerno + numfilters), poolsize, activation='relu', padding='same', name='encoder_conv_{}'.format(layerno))(x)
        last_pool_layer = tf.keras.layers.MaxPooling2D(poolsize, padding='same', name='encoder_pool_{}'.format(layerno))
        x = last_pool_layer(x)
    output_shape = last_pool_layer.output_shape[1:]

    if num_dense == 0:
        # flatten to create the embedding
        x = tf.keras.layers.Flatten(name='refc_embedding')(x)
        embed_size = output_shape[0] * output_shape[1] * output_shape[2]
        if embed_size > 1024:
            print("Embedding size={} is too large".format(embed_size))
            return None, embed_size
    else:
        # flatten, send through dense layer to create the embedding
        x = tf.keras.layers.Flatten(name='encoder_flatten')(x)
        x = tf.keras.layers.Dense(num_dense, name='refc_embedding')(x)
        x = tf.keras.layers.Dense(output_shape[0] * output_shape[1] * output_shape[2], name='decoder_dense')(x)
        embed_size = num_dense

    x = tf.keras.layers.Reshape(output_shape, name='decoder_reshape')(x)
    for layerno in range(nlayers):
        x = tf.keras.layers.Conv2D(2**(nlayers-layerno-1 + numfilters), poolsize, activation='relu', padding='same', name='decoder_conv_{}'.format(layerno))(x)
        x = tf.keras.layers.UpSampling2D(poolsize, name='decoder_upsamp_{}'.format(layerno))(x)
    before_padding_layer = tf.keras.layers.Conv2D(1, 3, activation='relu', padding='same', name='before_padding')
    x = before_padding_layer(x)
    htdiff = 1059 -...


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
Let's decode the first row. First, we create a decoder
Can you write Python code for it?

import tensorflow as tf
def create_decoder(model_dir):
    model = tf.keras.models.load_model(model_dir)
    decoder_input = tf.keras.Input([50], name='embed_input')
    embed_seen = False
    x = decoder_input
    for layer in model.layers:
        if embed_seen:
            x = layer(x)
        elif layer.name == 'refc_embedding':
            embed_seen = True
    decoder = tf.keras.Model(decoder_input, x, name='decoder')
    print(decoder.summary())
    return decoder

decoder = create_decoder('gs://ai-analytics-solutions-kfpdemo/wxsearch/trained/savedmodel')


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
Read the written TF Records
Can you write Python code for it?

# try reading what was written out
import tensorflow as tf

def parse_tfrecord(example_data):
    parsed = tf.io.parse_single_example(example_data, {
        'size': tf.io.VarLenFeature(tf.int64),
        'ref': tf.io.VarLenFeature(tf.float32),
        'time': tf.io.FixedLenFeature([], tf.string),
        'valid_time': tf.io.FixedLenFeature([], tf.string)
     })
    parsed['size'] = tf.sparse.to_dense(parsed['size'])
    parsed['ref'] = tf.reshape(tf.sparse.to_dense(parsed['ref']), (1059, 1799))/60. # 0 to 1
    return parsed

def read_dataset(pattern):
    filenames = tf.io.gfile.glob(pattern)
    ds = tf.data.TFRecordDataset(filenames, compression_type=None, buffer_size=None, num_parallel_reads=None)
    return ds.prefetch(tf.data.experimental.AUTOTUNE).map(parse_tfrecord)

ds = read_dataset('gs://{}/wxsearch/data/2019/tfrecord-00000-*'.format(BUCKET))
for refc in ds.take(1):
    print(repr(refc))


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
Reading HRRR data and converting to TensorFlow Records
HRRR data comes in a Grib2 files on Cloud Storage.
Can you write Python code for it?

!gsutil ls -l gs://high-resolution-rapid-refresh/hrrr.20200811/conus/hrrr.*.wrfsfcf00*

FILENAME="gs://high-resolution-rapid-refresh/hrrr.20200811/conus/hrrr.t18z.wrfsfcf06.grib2"   # derecho in the Midwest
!gsutil ls -l {FILENAME}

import xarray as xr
import tensorflow as tf
import tempfile
import cfgrib

with tempfile.TemporaryDirectory() as tmpdirname:
    TMPFILE="{}/read_grib".format(tmpdirname)
    tf.io.gfile.copy(FILENAME, TMPFILE, overwrite=True)
    ds = cfgrib.open_datasets(TMPFILE)
    print(ds)


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
Really, Jan. 2 in 2019 had similar weather to Sep 20? Let's see ...
Can you write Python code for it?

import pandas as pd
dt = pd.Timestamp(df['time'].values[0])
# gs://high-resolution-rapid-refresh/hrrr.20200811/conus/hrrr.t04z.wrfsfcf00.grib2
FILENAME = '{}/hrrr.{:4}{:02}{:02}/conus/hrrr.t{:02}z.wrfsfcf00.grib2'.format(
                'gs://high-resolution-rapid-refresh',
                dt.year, dt.month, dt.day, dt.hour)
print(FILENAME)

import xarray as xr
import tensorflow as tf
import tempfile
import cfgrib
import numpy as np

refc = 0
with tempfile.TemporaryDirectory() as tmpdirname:
    TMPFILE="{}/read_grib".format(tmpdirname)
    tf.io.gfile.copy(FILENAME, TMPFILE, overwrite=True)
    #ds = xr.open_dataset(TMPFILE, engine='cfgrib', backend_kwargs={'filter_by_keys': {'typeOfLevel': 'surface', 'stepType': 'instant'}})
    #ds.data_vars['prate'].plot()  # crain, prate
    ds = xr.open_dataset(TMPFILE, engine='cfgrib', backend_kwargs={'filter_by_keys': {'typeOfLevel': 'unknown', 'stepType': 'instant'}})
    #ds = xr.open_dataset(TMPFILE, engine='cfgrib', backend_kwargs={'filter_by_keys': {'typeOfLevel': 'atmosphere', 'stepType': 'instant'}})
    refc = ds.data_vars['refc']
    refc.plot()


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
Run at scale
Can you write Python code for it?

%run -m wxsearch.hrrr_to_tfrecord -- --startdate 20190101 --enddate 20200101  --outdir gs://{BUCKET}/wxsearch/data/2019 --project {PROJECT}


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
Searching for similar images
Suppose we want to find an image similar to the above image.
Can you write Python code for it?

%%bigquery
WITH ref1 AS (
SELECT time AS ref1_time, ref1_value, ref1_offset
FROM `ai-analytics-solutions.advdata.wxembed`,
     UNNEST(ref) AS ref1_value WITH OFFSET AS ref1_offset
WHERE time = '2019-09-20 05:00:00 UTC'
)

SELECT
  time,
  SUM( (ref1_value - ref[OFFSET(ref1_offset)]) * (ref1_value - ref[OFFSET(ref1_offset)]) ) AS sqdist
FROM ref1, `ai-analytics-solutions.advdata.wxembed`
GROUP BY 1
ORDER By sqdist ASC
LIMIT 5


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
Then, we invoke decoder.predict() to reconstruct the image from the 50 numbers in the embedding.
Can you write Python code for it?

import tensorflow as tf
import numpy as np
embed = tf.reshape( tf.convert_to_tensor(df['ref'].values[0], dtype=tf.float32), [-1, 50])
outimg = decoder.predict(embed).squeeze() * 60
print(len(df['ref'].values[0]))
print(np.max(outimg))

import matplotlib.pyplot as plt
plt.imshow(outimg, origin='lower');