I am working on the file "08/Homework_8_Emelike.ipynb".
The first task is:
2) Where are most weapons obtained?
Can you write Python code for it?

df['Gun Show'] = df['Where obtained'].str.contains('[Ss]how', na=False)
df['Online'] = df['Where obtained'].str.contains('[Oo]nline') | df['Where obtained'].str.contains('[Ii]nternet')
df['Family/Friends'] = df['Where obtained'].str.contains('[Gg]randfather')| df['Where obtained'].str.contains('[Mm]other') | df['Where obtained'].str.contains('[Ff]ather') | df['Where obtained'].str.contains('[Ff]riend') | df['Where obtained'].str.contains('[Ii]ndividual')
df['Store/Retailer'] = df['Where obtained'].str.contains('Trading')| df['Where obtained'].str.contains('[Ss]ports') | df['Where obtained'].str.contains('Big')| df['Where obtained'].str.contains('[Ss]portsman') | df['Where obtained'].str.contains("[Ss]portsman's") | df['Where obtained'].str.contains('[Ff]irearms') | df['Where obtained'].str.contains('Gander') | df['Where obtained'].str.contains('Galore')| df['Where obtained'].str.contains('[Dd]ealer') | df['Where obtained'].str.contains('[Ss]upply') | df['Where obtained'].str.contains('Fin') | df['Where obtained'].str.contains('[Ss]tore') | df['Where obtained'].str.contains('[Ss]tores') | df['Where obtained'].str.contains('[Cc]enter') |df['Where obtained'].str.contains('[Pp]awn') | df['Where obtained'].str.contains('[Rr]etailer') | df['Where obtained'].str.contains('[Rr]etailers') | df['Where obtained'].str.contains('[Ff]lea') | df['Where obtained'].str.contains('[Ss]uppliers') | df['Where obtained'].str.contains('[Rr]ange') | df['Where obtained'].str.contains("Frank's") | df['Where obtained'].str.contains("Frank's") | df['Where obtained'].str.contains("[Ss]upplies") | df['Where obtained'].str.contains("Frank's") | df['Where obtained'].str.contains('[Ss]ales') | df['Where obtained'].str.contains('[Ww]arehouse') | df['Where obtained'].str.contains('Bullseye' )| df['Where obtained'].str.contains('Outdoorsman')
df['Stolen'] = df['Where obtained'].str.contains('[Ss]tolen') | df['Where obtained'].str.contains('[Bb]urglary')
df['Unknown'] = df['Where obtained'].str.contains('Unknown') | df['Where...


I am working on the file "08/Homework_8_Emelike.ipynb".
The first task is:
3) Where state has the most mass shootings?
Can you write Python code for it?

# create empty list
states = []

# split the string values in location such that city, state become tuples
# iterate over list of tuples, appending only the state to the above list
for item in df['Location'].str.split(','):
    states.append(item[1])

# create series by setting series equal to the list
df['State'] = pd.Series(states)

# value counts on the series
df['State'].value_counts().head(5)


I am working on the file "08/Homework_8_Emelike.ipynb".
The first task is:
5) Which mass shootings had the most victims?
Can you write Python code for it?


I am working on the file "08/Homework_8_Emelike.ipynb".
The first task is:
6) What mass shooting was most fatal?
Can you write Python code for it?

df['Fatalities'].idxmax() # returns index of maximum of values in a series

df['Case'].iloc[1]


I am working on the file "08/Homework_8_Emelike.ipynb".
The first task is:
7) What kind of weapons were used in the most fatal shootings? / Do shooters use a glock more often than not?
Can you write Python code for it?

df['Weapon details'].iloc[1] #case = Orlando nightclub

df['Glock'] = df['Weapon details'].str.contains('[Gg]locks') | df['Weapon details'].str.contains('[Gg]lock')
df['Glock'].value_counts().plot(kind='bar', title = 'Shooter used a Glock?')


I am working on the file "08/Homework_8_Emelike.ipynb".
The first task is:
8) What kind of weapons were used in the shootings that had the most victims (wounded and fatal)?
Can you write Python code for it?


I am working on the file "08/Homework_8_Emelike.ipynb".
The first task is:
9) How many shooters showed prior signs of mental illness?
Can you write Python code for it?

df['Prior signs of possible mental illness'].value_counts()
df['Prior signs of possible mental illness'] = df['Prior signs of possible mental illness'].str.replace('Unclear', 'Unknown')
df['Prior signs of possible mental illness']= df['Prior signs of possible mental illness'].str.replace('unknown', 'Unknown')
df['Prior signs of possible mental illness']= df['Prior signs of possible mental illness'].str.replace('(pending)', 'Unknown')
df['Prior signs of possible mental illness']= df['Prior signs of possible mental illness'].str.replace('(Unknown) ', 'Unknown')
df['Prior signs of possible mental illness']= df['Prior signs of possible mental illness'].str.replace('(Unknown)', 'Unknown')
df['Prior signs of possible mental illness']= df['Prior signs of possible mental illness'].str.strip()
df['Prior signs of possible mental illness'].value_counts().plot(kind='bar', title = 'Prior Signs of Mental Illness?')


I am working on the file "08/Homework_8_Paolo_Rivas.ipynb".
The first task is:
0. How is this data distributed?
The presence of extrame values is a common pattern in this dataframe. We need to be carefull while using cetered measures.
Can you write Python code for it?


I am working on the file "08/Homework_8_Paolo_Rivas.ipynb".
The first task is:
10. Same for african americans?
Same distribution with the exception of an increase in deaths produced by diabetes type 2 and HIV
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(9, 7))
only_blacks = df[df['Ethnicity'] == 'NON-HISPANIC BLACK']
only_blacks.groupby('Cause of Death')['Count'].sum().sort_values(ascending=True).plot.barh(color='black')


I am working on the file "08/Homework_8_Paolo_Rivas.ipynb".
The first task is:
11. What about spanish and asians?
Similar results for spanish population
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(9, 7))
only_hispanic = df[df['Ethnicity'] == 'HISPANIC']
only_hispanic.groupby('Cause of Death')['Count'].sum().sort_values(ascending=True).plot.barh(color='brown')


I am working on the file "08/Homework_8_Paolo_Rivas.ipynb".
The first task is:
4. Which one are the most common deseases for men and woman? Is there a difference?
Mayor causes of death like heart deseases or Cancer seems to behave similar in the top 3 for men and women.
Can you write Python code for it?

df.groupby(['Cause of Death', 'Sex'])['Count'].sum().sort_values(ascending=False).head(6)


I am working on the file "08/Homework_8_Paolo_Rivas.ipynb".
The first task is:
5. What about in other deseases? Are there any sex differences?
Crebrovascular, diabetes and cronic lower respiratory deseases are more comen in women than men. On the other hand, Diabetes, HIV and deaths by accidents (note related to drugs) are most common in men.
Can you write Python code for it?

df.groupby(['Cause of Death', 'Sex'])['Count'].sum().sort_values(ascending=False)


I am working on the file "08/Homework_8_Paolo_Rivas.ipynb".
The first task is:
8. Which ethnicity is overall more vulnerable to deseases in NYC?
According to the number of cases reported, non-hispanic white newyorkers have the highest mortality rate.
Can you write Python code for it?

df.groupby('Ethnicity')['Count'].sum().sort_values(ascending=False)

df.groupby('Ethnicity')['Count'].sum().plot.barh(color=['Black', 'Black', 'Black', 'darkred'])


I am working on the file "08/Homework_8_Paolo_Rivas.ipynb".
The first task is:
Asians are by far the most different population in relations to causes of death. It has the highest incidents of death by Cancer
Can you write Python code for it?

only_asians = df[df['Ethnicity'] == 'ASIAN & PACIFIC ISLANDER']

fig, ax = plt.subplots(figsize=(9, 7))
only_asians.groupby('Cause of Death')['Count'].sum().sort_values(ascending=True).plot.barh(color='green')


I am working on the file "08/Swiss_Accident_Data/Unfall_Daten_11_15.ipynb".
The first task is:
3. Compare the top ten various vehicle categories involved in accidents. Total and by year.
Can you write Python code for it?

df_fahrzeuge['Fahrzeugart UAP'].value_counts().head(5)
#money_players = nba[nba['2013 $'] != 'n/a']


I am working on the file "08/Swiss_Accident_Data/Unfall_Daten_11_15.ipynb".
The first task is:
7. Combine as much information about breatheliser record as possible.
MERGING: Look at this again!!!*
Can you write Python code for it?

df_fahrzeuge["Atemtest"].describe()

df_fahrzeuge["Atemtest"].value_counts()

Alcohol_record = df_fahrzeuge[df_fahrzeuge['Atemtest'] == 9.0]

Alcohol_record[['BUM Probe angeordnet UAP', 'Blutalkoholtest', 'Datum', 'Vertrautheit Strecke UAP', 'Kennzeichen Fahrzeug Kanton']]

df_merged = df_personen.merge(df_fahrzeuge, how = 'left', left_on = 'Unfall-UID', right_on ='Unfall-UID')

df = df_merged.merge(df_unfaelle, how = 'left', left_on = 'Unfall-UID', right_on ='Unfall-UID')

df_fahrzeuge.info()

df_unfaelle.info()

df_personen.info()


I am working on the file "08/Swiss_Accident_Data/Unfall_Daten_11_15.ipynb".
The first task is:
Importing files on accidents, .csv Files, organised by year
Can you write Python code for it?

unfaelle2011 = pd.read_csv("Export_1_Unfallblatt_2011.csv", delimiter = ";", encoding = "latin-1")
unfaelle2012 = pd.read_csv("Export_1_Unfallblatt_2012.csv", delimiter = ";", encoding = "latin-1")
unfaelle2013 = pd.read_csv("Export_1_Unfallblatt_2013.csv", delimiter = ";", encoding = "latin-1")
unfaelle2014 = pd.read_csv("Export_1_Unfallblatt_2014.csv", delimiter = ";", encoding = "latin-1")
unfaelle2015 = pd.read_csv("Export_1_Unfallblatt_2015.csv", delimiter = "\t", encoding = "UTF-8")


I am working on the file "08/Swiss_Accident_Data/Unfall_Daten_11_15.ipynb".
The first task is:
Importing files on people involved in accidents, .csv Files, organised by year
Can you write Python code for it?

personen2011 = pd.read_csv("Export_3_Personenblatt_2011.csv", delimiter = ";", encoding = "latin-1")
personen2012 = pd.read_csv("Export_3_Personenblatt_2012.csv", delimiter = ";", encoding = "latin-1")
personen2013 = pd.read_csv("Export_3_Personenblatt_2013.csv", delimiter = ";", encoding = "latin-1")
personen2014 = pd.read_csv("Export_3_Personenblatt_2014.csv", delimiter = ";", encoding = "latin-1")
personen2015 = pd.read_csv("Export_3_Personenblatt_2015.csv", delimiter = "\t", encoding = "UTF-8")


I am working on the file "08/Swiss_Accident_Data/Unfall_Daten_11_15.ipynb".
The first task is:
The "|" sign ist causing a lot of problems as it doesn't allow me to change the names of the numbers in one go. And I haven't figures out how to do it in two goes.
Can you write Python code for it?

#df_fahrzeuge['Fahrzeugart UAP'].astype(str).str.replace('|', "").value_counts(ascending=False)
#vehicles_without_pipes.astype(str).str.replace('710', "Cars").value_counts(ascending=False).head(10)
test = df_fahrzeuge['Fahrzeugart UAP'].astype(str).str.replace('|', "").value_counts(ascending=False)
pd.DataFrame(test).head(10)
#test['Fahrzeugart UAP']
#test['Fahrzeugart UAP'].astype(str).str.replace('712', "Vans")
#test.columns()

#.str.replace('712', "Vans").str.replace('733', "n/a").str.replace('730', "Bicycles").str.replace('725', "Motorbikes, above 25kw").str.replace('718', "Trucks, above 7,4t").str.replace('722', "Mopeds").str.replace('724', "Motorbike, to 25kw").str.replace('720', "Semi trailer, above 7,5t").


I am working on the file "08/Swiss_Accident_Data/Unfall_Daten_11_15.ipynb".
The first task is:
The most deadly accident was a bus accident involving mostly school kids from Belgien. And this one more recently.
5. How many accidents occured on roads without ligthing? (Maybe even work out an accident hotspot?
Can you write Python code for it?

Strassenbeleuchtung UAP 640 keine 641 ausser Betrieb

df_unfaelle[(df_unfaelle['Lichtverhältnis UAP'] == 622) & (df_unfaelle['Strassenbeleuchtung UAP'] == 640)]

#df[(df['animal'] == 'cat') & (df['inches'] > 12)]

#df[df['feet'] > 6.5]

df_unfaelle[(df_unfaelle['Lichtverhältnis UAP'] == 622) & (df_unfaelle['Strassenbeleuchtung UAP'] == 641)]


I am working on the file "08/Swiss_Accident_Data/Unfall_Daten_11_15.ipynb".
The first task is:
Transforming the data into times is proving a problem. They need to be transformed, so I can make a histogram of the times. With time as X-axis and count of accidents corresponding to the time on Y-xis. This post on timestamps in Pandas may be of some help.
Can you write Python code for it?

#df_unfaelle['Unfallzeit'].value_counts(ascending=False).hist(kind='scatter', x=)
#df_unfaelle['Unfallzeit'].value_counts().plot(kind='barh', x=['Unfallzeit'])
#df_unfaelle['Unfallzeit_Double'] = df_unfaelle['Unfallzeit']
df_unfaelle['Unfallzeit'].value_counts().head(4)


I am working on the file "08/機械学習.ipynb".
The first task is:
71. ストップワード
英語のストップワードのリスト（ストップリスト）を適当に作成せよ．さらに，引数に与えられた単語（文字列）がストップリストに含まれている場合は真，それ以外は偽を返す関数を実装せよ．さらに，その関数に対するテストを記述せよ．
Can you write Python code for it?

from nltk.corpus import stopwords
stopwords_list = [s for s in stopwords.words('english')]
print(stopwords_list)


I am working on the file "08/機械学習.ipynb".
The first task is:
No.73
72で抽出した素性を用いて，ロジスティック回帰モデルを学習せよ．
Can you write Python code for it?

# import passages to construct logistic regression and learn the model.
from sklearn.linear_model import LogisticRegression
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS

tfv = TfidfVectorizer(encoding='utf-8', lowercase=True,
                      stop_words=ENGLISH_STOP_WORDS,
                      #token_pattern='(?u)\b\w\w+\b',
                      ngram_range=(1, 2))

with open('sentiment.txt') as f:
    features = [(s[:2], s[3:]) for s in f]

# make label list
label = [i[0] for i in features]

# make sentence list that is removed English Stop Words
sentence = []
for i in features:
    temp = i[1].split(' ')
    temp2 = [i+' ' for i in temp]
    s = "".join(temp2)
    sentence.append(s)

tfv_vector = tfv.fit_transform("".join(sentence).split(' '))


I am working on the file "08/機械学習.ipynb".
The first task is:
TfidfVectorizer.fit()の引数は単語"リスト"
Can you write Python code for it?


I am working on the file "0809下午python第三课课件.ipynb".
The first task is:
mysql数据类型    http://www.runoob.com/mysql/mysql-data-types.html
插入数据
Can you write Python code for it?

insert into `class`(`id`, `name`)
values(1, '高一三班');


I am working on the file "0809下午python第三课课件.ipynb".
The first task is:
修改数据
Can you write Python code for it?

update `class` set `name` = '高一五班'
where `name` = '高一三班';


I am working on the file "0809下午python第三课课件.ipynb".
The first task is:
删除操作
Can you write Python code for it?

sql = "delete from `class` where `name`='高一五班'"
cursor = db.cursor()
cursor.execute(sql)
db.commit()


I am working on the file "0809下午python第三课课件.ipynb".
The first task is:
捕捉异常
Can you write Python code for it?

a = 10
b = a + 'hello'

try:
    a = 10
    b = a + 'hello'
except TypeError as e:
    print(e)


I am working on the file "0809下午python第三课课件.ipynb".
The first task is:
插入操作
Can you write Python code for it?

sql = "insert into `class`(`name`) values('高一五班');"
cursor = db.cursor()
cursor.execute(sql)
cursor.execute(sql)
db.commit()


I am working on the file "0809下午python第三课课件.ipynb".
The first task is:
查询
Can you write Python code for it?

sql = "select * from student where id <= 20 limit 4"
cursor.execute(sql)
results = cursor.fetchall()
for row in results:
    print(row)


I am working on the file "0809下午python第三课课件.ipynb".
The first task is:
游标
Can you write Python code for it?


I am working on the file "08_1D_visco_elastic_SH_modelling/1_Intro_viscoelasticity.ipynb".
The first task is:
"Damping" of elastic waves
Even without the incorporation of viscoelastic effects, seismic waves can be damped.
Due to geometrical spreading the seismic energy is distributed over the surface of the wavefront, as we can see in the acoustic modelling result below:
Can you write Python code for it?

%matplotlib notebook
time, seis_hom = FD_2D_acoustic_JIT('hom')


I am working on the file "08_1D_visco_elastic_SH_modelling/2_viscoelastic_stress_strain.ipynb".
The first task is:
In the viscoelastic case stress and strain are out of phase. As we will see later, the phase shift $\phi$ can be related to the quality factor $Q$ by
\begin{equation}
\phi = \arctan{\frac{1}{Q}} \notag
\end{equation}
which leads to the viscoelastic stress-strain relation:
Can you write Python code for it?

# Stress-stain relation for linear viscoelastic medium
# ----------------------------------------------------
t = np.arange(0.0,10.0,0.1)  # time (s)

sigma0 = 1.0             # maximum stress (Pa)
epsilon0 = 1.0           # maximum strain ()
omega = 1.0              # Circular frequency (rad/s)

# define phase shift of strain
Q = 10                     # quality factor ()
phi = np.arctan(1/Q)       # phase shift (rad)

# Calculate temporal changes of sigma and epsilon
# -----------------------------------------------
sigma = sigma0 * np.cos(omega*t)
epsilon = epsilon0 * np.cos(omega*t - phi)

# Define figure size
rcParams['figure.figsize'] = 7, 5

# plot stress-strain relation
plt.plot(epsilon, sigma, 'r-',lw=3,label="linear viscoelastic medium")
plt.title('Stress-Strain relation')
plt.xlabel('Strain []')
plt.ylabel('Stress [Pa]')
plt.legend()
plt.grid()
plt.show()


I am working on the file "08_1D_visco_elastic_SH_modelling/2_viscoelastic_stress_strain.ipynb".
The first task is:
The energy loss $\Delta E$ for example via heat, is defined by the area within the hysteresis loop, so we can calculate $\Delta E$ by:
\begin{equation}
\Delta E = \oint \sigma(\epsilon') d\epsilon' \notag
\end{equation}
Furthermore, we can distinguish two special cases of linear viscoelasticity:

The case above is irreversible (inelastic)
A reversible (anelastic) medium can recover its original state after removing the load

Viscoelastic models
So far the details of the relaxation function are not defined. Therefore, the next step is to find a relaxation function with a frequency-independent $Q(\omega)$-value. Similar to electrical networks, we can construct viscoelastic models composed of two basic elements.
Basic elements
These basic elements are

The Hooke element (spring), representing the linear elastic medium

\begin{align}
\sigma_{Hooke} &= \mu \epsilon \notag\
&\text{or} \notag\
\epsilon_{Hooke} &= \frac{\sigma}{\mu}\notag\
\end{align}

The Newton element (dashpot), representing the viscous damping part with the stress-strainrate relation:

\begin{align}
\sigma_{Newton} &= \eta \dot{\epsilon} \notag\
&\text{or} \notag\
\dot{\epsilon}_{Newton} &= \frac{\sigma}{\eta}\notag\
\end{align}
where $\eta$ denotes the viscosity of the medium.
Maxwell-model
To realize different viscoelastic media we can combine the basic elements in different networks. A serial connection of the Hooke with the Newton element yields the Maxwell-model:

In a serial connection, the effective deformation can be calculated by the sum of the basic element deformations
\begin{equation}
\epsilon = \epsilon_1 + \epsilon_2 \notag
\end{equation}
or by taking the time derivative
\begin{equation}
\dot{\epsilon} = \dot{\epsilon}_1 + \dot{\epsilon}_2 \notag
\end{equation}
For the Maxwell-model we get the following ordinary differential equation for $\epsilon$ and $\sigma$
\begin{equation}
\dot{\epsilon}{M} = \dot{\epsilon}{Newton} +...
Can you write Python code for it?

# 1/Q(omega) for the Maxwell-model
# --------------------------------
f = np.arange(5.0,100.0,0.5)  # frequency (Hz)
omega = 2 * np.pi * f

mu = 4e8             # shear modulus (Pa) for soil from Dokter et al. (2017)
eta = 1.0            # viscosity (Pa s)

# Define complex shear modulus for Maxwell-model
# ----------------------------------------------
muM = (1/mu) + (1/(1j*omega*eta))

# Calculate 1/Q(omega)
# --------------------
Qinv = np.abs(np.imag(muM)/np.real(muM))

# Define figure size
rcParams['figure.figsize'] = 7, 5

# plot stress-strain relation
plt.plot(f, Qinv, 'r-',lw=3,label="Maxwell-model")
plt.title(r'$\frac{1}{Q}(\omega)$ for the Maxwell-model')
plt.xlabel('Frequency f [Hz]')
plt.ylabel(r'$\frac{1}{Q}$ []')
plt.grid()
plt.show()


I am working on the file "08_1D_visco_elastic_SH_modelling/2_viscoelastic_stress_strain.ipynb".
The first task is:
This result looks promising. While $1/Q$ is not frequency independent, we see for the Maxwell SLS model a peak at a given frequency and decreases at lower and higher frequencies. For completeness, I note that we could also build a Kelvin-Voigt SLS model by connecting a Hooke element with a Kelvin-Voigt element in parallel. However, in the following we will further focus on the potential of the Maxwell SLS model.
Generalized Maxwell-model
Based on the promising results of the Maxwell SLS model, we add multiple Maxwell models in parallel, which yields the Generalized Maxwell model or Generalized Maxwell body (GMB), also known as Maxwell-Wiechert model. By the superposition of multiple Maxwell models with different elastic modules $\mu_l$ and viscosities $\eta_l$, we can achieve a constant Q-value over a given frequency range.
<img src="images/GMB.png" width="70%">
Because we assemble the Maxwell SLS model with additional L Maxwell bodies in parallel, we have to add the stresses in frequency domain:
\begin{equation}
\tilde{\sigma}{GMB} = \tilde{\sigma}{SLSM} + \sum_{l=2}^{L} \tilde{\sigma}_{Maxwell, l} \notag
\end{equation}
Inserting the stresses
\begin{align}
\tilde{\sigma}{SLSM} &= \biggl(\mu_0 + \frac{i \mu_1 \omega \eta_1}{i \omega \eta_1 + \mu_1}\biggr) \tilde{\epsilon} \notag \
\tilde{\sigma}{Maxwell, l} &= \frac{i \mu_l \omega \eta_l}{i \omega \eta_l + \mu_l} \tilde{\epsilon}\notag \
\end{align}
yields the frequency-domain stress-strain relation for the GMB
\begin{equation}
\tilde{\sigma}{GMB} = \biggl(\mu_0 + \frac{i \mu_1 \omega \eta_1}{i \omega \eta_1 + \mu_1} + \sum{l=2}^{L} \frac{i \mu_l \omega \eta_l}{i \omega \eta_l + \mu_l}\biggr) \tilde{\epsilon} \notag
\end{equation}
We can move the second term into the sum over the L Maxwell-models:
\begin{equation}
\tilde{\sigma}{GMB} = \biggl(\mu_0 + \sum{l=1}^{L} \frac{i \mu_l \omega \eta_l}{i \omega \eta_l + \mu_l}\biggr) \tilde{\epsilon} \notag
\end{equation}
Introducing the relaxation frequencies
\begin{equation}
\omega_l :=\frac{\mu_1}{\eta_l}...
Can you write Python code for it?

# Q(omega) for the GMB with 4 Maxwell bodies
# --------------------------------------------
f = np.arange(5.0,100.0,0.5)  # frequency (Hz)
nf = len(f)
omega = 2 * np.pi * f

# relaxation frequencies
L = 4   # number of Maxwell bodies
fl = np.linspace(5.0,100.0,L)
omegal = 2. * np.pi * fl

# Define relaxed shear modulus and defect for GMB
mu0 = 4e8          # shear modulus (Pa) for soil from Dokter et al. (2017)
dmu = 1.4e8        # shear moduli of Maxwell bodies (Pa)

# Simply set a_l = 1.0 / L
a_l = 1.0 / L

# Define complex shear modulus for GMB model
# ------------------------------------------
muM = np.zeros(nf,dtype='complex128')
for l in range(0,L):
    muM += (1j * a_l * omega)/((1j * omega) + omegal[l])
muGMB = mu0 + dmu * muM

# Calculate Q(omega)
# --------------------
Q = np.real(muGMB)/np.imag(muGMB)

# Define figure size
rcParams['figure.figsize'] = 7, 5

# plot stress-strain relation
plt.plot(f, Q, 'r-',lw=3,label="General Maxwell model")
plt.title(r'$Q(\omega)$ for General Maxwell model')
plt.xlabel('Frequency f [Hz]')
plt.ylabel(r'$Q$ []')
plt.ylim(0,20)
plt.grid()
plt.show()


I am working on the file "08_1D_visco_elastic_SH_modelling/2_viscoelastic_stress_strain.ipynb".
The first task is:
Unfortunately, the Kelvin-Voigt model is also not suitable to model viscoelastic wave propagation, because the damping increases linear with frequency.
Standard Linear Solid (SLS) - Maxwell
Another idea to construct a viscoelastic model is to combine the Maxwell element with a Hooke element in parallel. This is denoted as Standard Linear Solid (SLS) in Maxwell representation
<img src="images/SLS.png" width="70%">
Because we have to combine elements in parallel, the stresses of components have to be added:
\begin{equation}
\sigma_{SLSM} = \sigma_{Hooke} + \sigma_{Maxwell} \notag
\end{equation}
For simplicity we will do this directly in the frequency domain:
\begin{equation}
\tilde{\sigma}{SLSM} = \tilde{\sigma}{Hooke} + \tilde{\sigma}_{Maxwell} \notag
\end{equation}
Inserting the stresses
\begin{align}
\tilde{\sigma}{Hooke} &= \mu_0 \tilde{\epsilon}\notag \
\tilde{\sigma}{Maxwell} &= \frac{i \mu_1 \omega \eta}{i \omega \eta + \mu_1} \tilde{\epsilon}\notag \
\end{align}
yields the frequency-domain stress-strain relation for the SLS in Maxwell representation:
\begin{equation}
\tilde{\sigma}_{SLSM} = \biggl(\mu_0 + \frac{i \mu_1 \omega \eta}{i \omega \eta + \mu_1}\biggr) \tilde{\epsilon} \notag
\end{equation}
The complex SLS shear modulus $\tilde{\mu}_{SLSM}$ can be rewritten as
\begin{equation}
\tilde{\mu}_{SLSM} = \mu_0 + \frac{i \mu_1 \omega}{i \omega + \frac{\mu_1}{\eta}} \notag
\end{equation}
Notice that the quantity $\frac{\mu_1}{\eta}$ has the dimension $\left[\frac{Pa}{Pa\; s}\right] = \left[\frac{1}{s}\right]$ of a frequency. A more detailed analysis reveals that this is the relaxation frequency
\begin{equation}
\omega_1 :=\frac{\mu_1}{\eta} \notag
\end{equation}
Therefore, we get:
\begin{equation}
\tilde{\mu}_{SLSM} = \mu_0 + \frac{i \mu_1 \omega}{i \omega + \omega_1} \notag
\end{equation}
How does the $1/Q(\omega)$ frequency spectrum look like?
Can you write Python code for it?

# 1/Q(omega) for the Maxwell SLS
# ------------------------------
f = np.arange(5.0,100.0,0.5)  # frequency (Hz)
omega = 2 * np.pi * f

mu0 = 4e8             # shear modulus (Pa) for soil from Dokter et al. (2017)
mu1 = 1               # shear modulus (Pa)

f1 = 50.0
omega1 = 2 * np.pi * f1

# Define complex shear modulus for Kelvin-Voigt model
# ---------------------------------------------------
muSLSM = mu0 + (1j * mu1 * omega)/((1j * omega) + omega1)

# Calculate 1/Q(omega)
# --------------------
Qinv = np.imag(muSLSM)/np.real(muSLSM)

# Define figure size
rcParams['figure.figsize'] = 7, 5

# plot stress-strain relation
plt.plot(f, Qinv, 'r-',lw=3,label="Maxwell SLS model")
plt.title(r'$\frac{1}{Q}(\omega)$ for Maxwell SLS model')
plt.xlabel('Frequency f [Hz]')
plt.ylabel(r'$\frac{1}{Q}$ []')
plt.grid()
plt.show()


I am working on the file "08_1D_visco_elastic_SH_modelling/4_1D_visc_SH_FD_modelling.ipynb".
The first task is:
... and a function for the shear stress update $\sigma_{yx}$ for the viscoelastic case ...
Can you write Python code for it?

# Shear stress syx updates (viscoelastic)
# ---------------------------------------
@jit(nopython=True) # use JIT for C-performance
def update_stress_visc(vy, syx, xi, Yl, wl, L, dx, dt, nx, mux):

    for i in range(1, nx - 1):

        # Calculate spatial derivatives
        vy_x = (vy[i + 1] - vy[i]) / dx

        # Calculate sum over memory variables
        xi_sum = 0.0
        for l in range(0, L):
            xi_sum += Yl[l] * xi[i,l]

        # Update shear stress
        # Note that the factor 0.5 in front of the memory variables sum
        # is due to the arithmetic averaging of the xi variables at
        # the staggered time steps
        syx[i] = syx[i] + dt * mux[i] * (vy_x - 0.5 * xi_sum)

        # Update memory variables
        xi_sum = 0.0
        for l in range(0, L):
            xi[i,l] += dt * wl[l] * (vy_x - xi[i,l])

            # After xi update calculate new xi_sum ...
            xi_sum += Yl[l] * xi[i,l]

        # ... and finish stress update
        syx[i] = syx[i] - dt * mux[i] * xi_sum

    return syx, xi


I am working on the file "08_1D_visco_elastic_SH_modelling/4_1D_visc_SH_FD_modelling.ipynb".
The first task is:
... and harmonically averaging the (unrelaxed) shear modulus ...
Can you write Python code for it?

# Harmonic averages of shear modulus
# ----------------------------------
@jit(nopython=True) # use JIT for C-performance
def shear_avg(mu, nx, mux):

    for i in range(1, nx - 1):

        # Calculate harmonic averages of shear moduli
        mux[i] = 2 / (1 / mu[i + 1] + 1 / mu[i])

    return mux


I am working on the file "08_1D_visco_elastic_SH_modelling/4_1D_visc_SH_FD_modelling.ipynb".
The first task is:
... run the elastic FD code and compare the result with the analytical solution:
Can you write Python code for it?

%%time
# FD modelling of homogeneous elastic medium
# ------------------------------------------
dx   = 1.0   # grid point distance in x-direction (m)
dt = 0.001   # time step (s)

FD_1D_visc_SH_JIT(dt,dx,f0,xsrc,Yl,wl,L,'elast')


I am working on the file "08_1D_visco_elastic_SH_modelling/4_1D_visc_SH_FD_modelling.ipynb".
The first task is:
... update the shear stress component $\sigma_{yx}$ for the elastic medium ...
Can you write Python code for it?

# Shear stress syx updates (elastic)
# ----------------------------------
@jit(nopython=True) # use JIT for C-performance
def update_stress(vy, syx, dx, dt, nx, mux):

    for i in range(1, nx - 1):

        # Calculate spatial derivatives
        vy_x = (vy[i + 1] - vy[i]) / dx

        # Update shear stress
        syx[i] = syx[i] + dt * mux[i] * vy_x

    return syx


I am working on the file "08_1D_visco_elastic_SH_modelling/4_1D_visc_SH_FD_modelling.ipynb".
The first task is:
As usual, we define the modelling parameters ...
Can you write Python code for it?

# Definition of modelling parameters
# ----------------------------------
xmax = 500.0 # maximum spatial extension of the 1D model in x-direction (m)
tmax = 0.502   # maximum recording time of the seismogram (s)

vs0  = 580.   # S-wave speed in medium (m/s)
rho0 = 1000.  # Density in medium (kg/m^3)

# acquisition geometry
xr = 330.0 # x-receiver position (m)

xsrc = 250.0 # x-source position (m)

f0   = 40. # dominant frequency of the source (Hz)
t0   = 4. / f0 # source time shift (s)


I am working on the file "08_1D_visco_elastic_SH_modelling/4_1D_visc_SH_FD_modelling.ipynb".
The first task is:
Finally, we can assemble the main FD code ...
Can you write Python code for it?

# 2D SH viscoelastic wave propagation (Finite Difference Solution)
# ----------------------------------------------------------------
def FD_1D_visc_SH_JIT(dt,dx,f0,xsrc,Yl,wl,L,mode):

    nx = (int)(xmax/dx) # number of grid points in x-direction
    print('nx = ',nx)

    nt = (int)(tmax/dt) # maximum number of time steps
    print('nt = ',nt)

    ir = (int)(xr/dx)      # receiver location in grid in x-direction
    isrc = (int)(xsrc/dx)  # source location in grid in x-direction

    # Source time function (Gaussian)
    # -------------------------------
    src  = np.zeros(nt + 1)
    time = np.linspace(0 * dt, nt * dt, nt)

    # 1st derivative of a Gaussian
    src  = -2. * (time - t0) * (f0 ** 2) * (np.exp(- (f0 ** 2) * (time - t0) ** 2))

    # Analytical solution
    # -------------------
    G        = time * 0.
    vy_analy = time * 0.

    # Initialize coordinates
    # ----------------------
    x    = np.arange(nx)
    x    = x * dx       # coordinates in x-direction (m)

    # calculate source-receiver distance
    r = np.sqrt((x[ir] - x[isrc])**2)

    for it in range(nt): # Calculate Green's function (Heaviside function)
        if (time[it] - r / vs0) >= 0:
            G[it] = 1. / (2 * rho0 * vs0)
    Gc   = np.convolve(G, src * dt)
    Gc   = Gc[0:nt]

    # compute vy_analy from uy_analy
    for i in range(1, nt - 1):
        vy_analy[i] = (Gc[i+1] - Gc[i-1]) / (2.0 * dt)

    # Initialize empty wavefield arrays
    # ---------------------------------
    vy    = np.zeros(nx) # particle velocity vy
    syx   = np.zeros(nx) # shear stress syx

    # Initialize model (assume homogeneous model)
    # -------------------------------------------
    vs    = np.zeros(nx)
    vs    = vs + vs0      # initialize wave velocity in model

    rho   = np.zeros(nx)
    rho   = rho + rho0    # initialize wave velocity in model

    # calculate shear modulus
    # -----------------------
    mu    = np.zeros(nx)
   ...


I am working on the file "08_1D_visco_elastic_SH_modelling/4_1D_visc_SH_FD_modelling.ipynb".
The first task is:
Finally, we run the viscoelastic modelling run and compare it with the elastic analytical solution:
Can you write Python code for it?

%%time
# FD modelling of homogeneous viscoelastic medium
# -----------------------------------------------
dx   = 1.0   # grid point distance in x-direction (m)
dt = 0.001   # time step (s)

FD_1D_visc_SH_JIT(dt,dx,f0,xsrc,Yl,wl,L,'visc')


I am working on the file "08_1D_visco_elastic_SH_modelling/4_1D_visc_SH_FD_modelling.ipynb".
The first task is:
Next, we define the objective function to optimize the $Y_l$ values ...
Can you write Python code for it?

# Objective function to optimize Yl values
# ----------------------------------------
def obj_Yl(Yl):

    # Calculate Qs model based on GMB
    # -------------------------------
    Qinv_GMB = np.zeros(nfreq)
    Qinv_const = (1/Qopt) * np.ones(nfreq)

    for l in range (0,L):
        Qinv_GMB += Yl[l] * (wl[l] * w) / (w**2+wl[l]**2)

    # Calculate objective function
    obj_Qinv = np.sum((Qinv_GMB - Qinv_const)**2)

    return obj_Qinv


I am working on the file "08_1D_visco_elastic_SH_modelling/4_1D_visc_SH_FD_modelling.ipynb".
The first task is:
Optimizing Yl coefficients
To achieve a constant Q-spectrum, we have to optimize the anelastic coefficients $Y_l$. This can be achieved by minimizing the objective function:
\begin{equation}
\chi(Y_l) = \int_{\omega_{min}}^{\omega_{max}} \biggl(Q^{-1}(Y_l,\omega) - Q^{-1}_{opt}\biggr)^2 d\omega\notag
\end{equation}
where $Q^{-1}_{opt}$ denotes the target constant inverse Q-value within the frequency range of the source wavelet and
\begin{equation}
Q^{-1}(Y_l,\omega) = \sum_{l=1}^L Y_l \frac{\omega_l\omega}{\omega^2+\omega_l^2}\notag
\end{equation}
is an approximation of the GMB Q-spectrum for $Q>>1$ according to (Blanch et al. 1995, Bohlen 2002, Yang et al. 2016). The objective function can be minimized using local or global optimization methods. In this case, we will use a global optimization Differential Evolution (DE) algorithm available from SciPy Optimization library.
In the following example, we want to solve the viscoelastic SH-problem by the (2,2)-FD algorithm defined above for a homogeneous model using 4 Maxwell bodies and target for a constant Q=20 value in a frequency band between 5 and 100 Hz:
Can you write Python code for it?

# Define GMB models
# -----------------
Qopt = 20     # target Q-value
L = 4         # number of Maxwell bodies
nfreq = 50    # number of frequencies to estimate Q-model

fmin = 5.     # minimum frequency
fmax = 100.0  # maximum frequency

f = np.linspace(fmin,fmax,num=nfreq)  # calculate frequencies to estimate Q-model
w = 2 * np.pi * f                     # circular frequencies to estimate Q-model

fl = np.linspace(fmin,fmax,num=L)  # calculate relaxation frequencies
wl = 2 * np.pi * fl                # circular relaxation frequencies


I am working on the file "08_1D_visco_elastic_SH_modelling/4_1D_visc_SH_FD_modelling.ipynb".
The first task is:
To optimize the $Y_l$ values, we define their bounds necessary for the DE-algorithm, minimize the objective function, store the resulting $Y_l$ values and plot the optimized Q-spectrum
Can you write Python code for it?

# Optimize dimensionless, anelastic coefficients Yl
# -------------------------------------------------

# Define bound constraints for DE algorithm
bounds = [(0.0, 1), (0.0, 1), (0.0, 1), (0.0, 1)]

# Optimize Q-model by Differential Evolution
DE_result = differential_evolution(obj_Yl, bounds)
print('  Final obj_Qinv = ', DE_result.fun)

# Calculate optimum Q model
f, Qinv_GMB = calc_Q(DE_result.x)

# Store and display optimized Yl, wl values
Yl = np.zeros(L)
Yl = DE_result.x
print('Yl = ', Yl)
print('wl = ', wl)

# Calculate Q(omega)
Q = 1 / Qinv_GMB

# Define figure size
rcParams['figure.figsize'] = 7, 5

# plot stress-strain relation
plt.plot(f, Q, 'r-',lw=3,label="Generalized Maxwell model")
plt.title(r'$Q(\omega)$ for Generalized Maxwell model')
plt.xlabel('Frequency f [Hz]')
plt.ylabel(r'$Q$ []')
plt.ylim(0,2*Qopt)
plt.grid()
plt.show()


I am working on the file "08_Flowers_retraining.ipynb".
The first task is:
2. 사전 설정

model

여기서 약간 특별한 처리를 해주어야 합니다.
1. resnet152는 pretrained된 것은 분류개수가 1000개이라서, flower폴더에 있는 5개로 분류개수를 가지도록 재구성합니다.
2. 마지막 parameter만를 update할 수 있도록 나머지 layer는 requires_grad를 False로 합니다.

loss
opimizer

그리고 최적화에는 재구성한 마지막 layer만을 update하도록 설정합니다.
Can you write Python code for it?

model = torchvision.models.resnet152(pretrained = True)

### don't update model parameters
for param in model.parameters() :
    param.requires_grad = False
#modify last fully connected layter
model.fc = nn.Linear(model.fc.in_features, cls_num)

fc_parameters = [
    {'params': model.fc.parameters()},
]
optimizer = torch.optim.Adam(fc_parameters, lr=1e-4, weight_decay=1e-4)
loss_fn = nn.CrossEntropyLoss()

if is_cuda : model.cuda(), loss_fn.cuda()


I am working on the file "08_Flowers_retraining.ipynb".
The first task is:
3. Trainning loop
* (입력 생성)
* model 생성
* loss 생성
* zeroGrad
* backpropagation
* optimizer step (update model parameter)

```
Can you write Python code for it?

# trainning
model.train()
train_loss = []
train_accu = []
i = 0
for epoch in range(35):
    for image, target in train_loader:
        image, target = Variable(image.float()), Variable(target) # 입력image Target 설정
        if is_cuda :  image, target = image.cuda(), target.cuda()
        output = model(image) # model 생성
        loss = loss_fn(output, target) #loss 생성
        optimizer.zero_grad() # zero_grad
        loss.backward() # calc backward grad
        optimizer.step() # update parameter

        pred = output.data.max(1)[1]
        accuracy = pred.eq(target.data).sum()/batch_size

        train_loss.append(loss.data[0])
        train_accu.append(accuracy)

        if i % 300 == 0:
            print(i, loss.data[0])
        i += 1

plt.plot(train_accu)

plt.plot(train_loss)


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Caution the * will just multiply the matricies on an element-by-element basis:
Can you write Python code for it?


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Colormap reference: http://matplotlib.org/examples/color/colormaps_reference.html
Can you write Python code for it?

fig, ax = plt.subplots(1,5,sharey=True)

fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(test_image, cmap=plt.cm.viridis)
ax[0].set_xlabel('viridis')

ax[1].imshow(test_image, cmap=plt.cm.hot)
ax[1].set_xlabel('hot')

ax[2].imshow(test_image, cmap=plt.cm.magma)
ax[2].set_xlabel('magma')

ax[3].imshow(test_image, cmap=plt.cm.spectral)
ax[3].set_xlabel('spectral')

ax[4].imshow(test_image, cmap=plt.cm.gray)
ax[4].set_xlabel('gray')


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Creating images from math
Can you write Python code for it?

X = np.linspace(-5, 5, 500)
Y = np.linspace(-5, 5, 500)

X, Y = np.meshgrid(X, Y)     # turns two 1-d arrays (X, Y) into one 2-d grid

Z = np.sqrt(X**2+Y**2)+np.sin(X**2+Y**2)

Z.min(), Z.max(), Z.mean()


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
FITS Images - An astronomical example

World Coordinate System wcs
Can you write Python code for it?

from astropy.wcs import WCS

from matplotlib.colors import LogNorm

my_image_file = "./MyData/m51.fits"

image_data = fits.getdata(my_image_file)
image_header = fits.getheader(my_image_file)

fig, ax = plt.subplots(1,3)

fig.set_size_inches(12,4)

fig.tight_layout()

ax[0].set_title("Upside down!")
ax[1].set_title("Right side up!")
ax[2].set_title("LOG image intensity")

ax[0].imshow(image_data, cmap=plt.cm.gray)
ax[1].imshow(image_data, origin='lower', cmap=plt.cm.gray)
ax[2].imshow(image_data, origin='lower', cmap=plt.cm.gray, norm=LogNorm());

image_header

my_wcs = WCS(image_header)

my_wcs

fig = plt.figure()
ax = fig.add_subplot(111, projection=my_wcs)

fig.set_size_inches(6,6)

fig.tight_layout()

ax.grid(color='r', ls='--')
ax.set_xlabel('Right Ascension (J2000)')
ax.set_ylabel('Declination (J2000)')

ax.imshow(image_data, origin='lower', cmap=plt.cm.gray);

fig = plt.figure()
ax = fig.add_subplot(111, projection=my_wcs)

fig.set_size_inches(6,6)

fig.tight_layout()

ax.set_xlabel('Right Ascension (J2000)')
ax.set_ylabel('Declination (J2000)')

ax.grid(color='r', ls='--')

overlay = ax.get_coords_overlay('galactic')

overlay.grid(color='y', ls='dotted')

overlay[0].set_axislabel('Galactic Longitude')
overlay[1].set_axislabel('Galactic Latitude')

ax.imshow(image_data, origin='lower', cmap=plt.cm.gray);


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Fancy Image Display
Can you write Python code for it?

from matplotlib.colors import LightSource

ls = LightSource(azdeg=0,altdeg=40)
shadedfig = ls.shade(Z,plt.cm.copper)

fig, ax = plt.subplots(1,3)

fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(shadedfig)

contlevels = [1,2,Z.mean()]

ax[1].axis('equal')
ax[1].contour(Z,contlevels)

ax[2].imshow(shadedfig)
ax[2].contour(Z,contlevels);


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Fill the array with the filtered images
Can you write Python code for it?

rgb[:,:,0] = redfilter
rgb[:,:,1] = greenfilter
rgb[:,:,2] = bluefilter

fig, ax = plt.subplots(1,4)
fig.set_size_inches(14,3)

fig.tight_layout()

ax[0].set_title("Red Filter")
ax[1].set_title("Green Filter")
ax[2].set_title("Blue Filter")
ax[3].set_title("All Filters Stacked")

ax[0].imshow(redfilter,cmap=plt.cm.gray)
ax[1].imshow(greenfilter,cmap=plt.cm.gray)
ax[2].imshow(bluefilter,cmap=plt.cm.gray)
ax[3].imshow(rgb,cmap=plt.cm.gray);

print("The image rgb has a shape [height,width] of {0}".format(rgb.shape))
print("The image rgb is made up of data of type {0}".format(rgb.dtype))
print("The image rgb has a maximum value of {0}".format(rgb.max()))
print("The image rgb has a minimum value of {0}".format(rgb.min()))

rgb[:,:,0] = redfilter * 1.5

plt.imshow(rgb)


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Images are just 2-d arrays - imshow will display 2-d arrays as images
Can you write Python code for it?

print(A)

plt.imshow(A, interpolation='nearest', cmap=plt.cm.Blues);


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Images are just arrays that can be sliced.

For common image formats the origin is the upper left hand corner
Can you write Python code for it?

fig, ax = plt.subplots(1,4)
fig.set_size_inches(12,6)

fig.tight_layout()

# You can show just slices of the image - Rememeber: The origin is the upper left corner

ax[0].imshow(my_doctor, cmap=plt.cm.gray)
ax[0].set_xlabel('Original')

ax[1].imshow(my_doctor[0:300,0:100], cmap=plt.cm.gray)
ax[1].set_xlabel('[0:300,0:100]')                 # 300 rows, 100 columns

ax[2].imshow(my_doctor[:,0:100], cmap=plt.cm.gray)       # ":" = whole range
ax[2].set_xlabel('[:,0:100]')                     # all rows, 100 columns

ax[3].imshow(my_doctor[:,::-1], cmap=plt.cm.gray);
ax[3].set_xlabel('[:,::-1]')                      # reverse the columns

fig, ax = plt.subplots(1,2)
fig.set_size_inches(12,6)

fig.tight_layout()

CutLine = 300

ax[0].imshow(my_doctor, cmap=plt.cm.gray)
ax[0].hlines(CutLine, 0, 194, color='b', linewidth=3)

ax[1].plot(my_doctor[CutLine,:], color='b', linewidth=3)
ax[1].set_xlabel("X Value")
ax[1].set_ylabel("Pixel Value")


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Let us work with the matrix:
$
\left[
\begin{array}{cc}
1 & 2 \
1 & 1
\end{array}
\right]
$
Can you write Python code for it?

my_matrix = np.array([[1,2],[1,1]])

print(my_matrix.shape)

print(my_matrix)

my_matrix_transposed = np.transpose(my_matrix)

print(my_matrix_transposed)

my_matrix_inverse = linalg.inv(my_matrix)

print(my_matrix_inverse)


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Pseudocolor - All color astronomy images are fake.
Color images are composed of three 2-d images: <img src="images/Layers.png" width="150">
JPG images are 3-d, even grayscale images
Can you write Python code for it?

redfilter = plt.imread("./MyData/sphereR.jpg")

redfilter.shape,redfilter.dtype


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Reading in images (imread) - Common Formats
Can you write Python code for it?

my_doctor = plt.imread('./MyData/doctor5.png')

print("The image my_doctor has a shape [height,width] of {0}".format(my_doctor.shape))
print("The image my_doctor is made up of data of type {0}".format(my_doctor.dtype))
print("The image my_doctor has a maximum value of {0}".format(my_doctor.max()))
print("The image my_doctor has a minimum value of {0}".format(my_doctor.min()))

plt.imshow(my_doctor,cmap=plt.cm.gray);


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Show the image represenation of I with a colorbar
Can you write Python code for it?


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Solving system of linear equations
$$
\begin{array}{c}
x + 2y = 4 \
x + y = 3 \
\end{array}
\hspace{2cm}
\left[
\begin{array}{cc}
1 & 2 \
1 & 1 \
\end{array}
\right]
\left[
\begin{array}{c}
x\
y
\end{array}
\right]
=
\left[
\begin{array}{c}
4\
3\
\end{array}
\right]
\hspace{2cm}
{\bf A}x = {\bf b}
\hspace{2cm}
\left[
\begin{array}{c}
x\
y
\end{array}
\right]
=
\left[
\begin{array}{cc}
1 & 2 \
1 & 1 \
\end{array}
\right]^{-1}
\left[
\begin{array}{c}
4\
3\
\end{array}
\right]
=
\left[
\begin{array}{c}
2\
1
\end{array}
\right]
$$
Can you write Python code for it?

A = np.array([[1,2],[1,1]])

print(A)

b = np.array([[4],[3]])

print(b)

# Solve by inverting A and then mulitply by b

linalg.inv(A).dot(b)

# Cleaner looking

linalg.solve(A,b)


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
Stellar spectral classes
Can you write Python code for it?

import glob

star_list = glob.glob('./MyData/Star_*.fits')

star_list

fig, ax = plt.subplots(1,1)
fig.set_size_inches(9,5)

fig.tight_layout()

# Full spectra

ax.set_ylabel("Flux")
ax.set_xlabel("Wavelength [angstroms]")
ax.set_ylim(0.0, 1.05)

for file in star_list:

    spectra = fits.getdata(file)

    spectra_normalized = spectra / spectra.max()

    ax.plot(Wavelength, spectra_normalized, label=file)

ax.legend(loc=0,shadow=True);


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
System of 3 equations
$$
\begin{array}{c}
x + 3y + 5z = 10 \
2x + 5y + z = 8 \
2x + 3y + 8z = 3 \
\end{array}
\hspace{3cm}
\left[
\begin{array}{ccc}
1 & 3 & 5 \
2 & 5 & 1 \
2 & 3 & 8
\end{array}
\right]
\left[
\begin{array}{c}
x\
y\
z
\end{array}
\right]
=
\left[
\begin{array}{c}
10\
8\
3
\end{array}
\right]
$$
Can you write Python code for it?


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
The two images above may look the same but they are not! Subtracting the two images reveals the truth.
Can you write Python code for it?

fig, ax = plt.subplots(1,3)
fig.set_size_inches(12,6)

fig.tight_layout()

ax[0].imshow(image_data, cmap=plt.cm.gray)
ax[1].imshow(another_image_data, cmap=plt.cm.gray);

real_image = image_data - another_image_data                 # Subtract the images pixel by pixel

ax[2].imshow(real_image, cmap=plt.cm.gray);


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
We just want to read in one of the three channels
Can you write Python code for it?

redfilter = plt.imread("./MyData/sphereR.jpg")[:,:,0]

redfilter.shape,redfilter.dtype

plt.imshow(redfilter,cmap=plt.cm.gray);

greenfilter = plt.imread("./MyData/sphereG.jpg")[:,:,0]
bluefilter = plt.imread("./MyData/sphereB.jpg")[:,:,0]

fig, ax = plt.subplots(1,3)
fig.set_size_inches(12,3)

fig.tight_layout()

ax[0].set_title("Red Filter")
ax[1].set_title("Green Filter")
ax[2].set_title("Blue Filter")

ax[0].imshow(redfilter,cmap=plt.cm.gray)
ax[1].imshow(greenfilter,cmap=plt.cm.gray)
ax[2].imshow(bluefilter,cmap=plt.cm.gray);


I am working on the file "08_Images_In_Python.ipynb".
The first task is:
numpy matrix multiply uses the dot() function:
Can you write Python code for it?


I am working on the file "08_Python_LaTeX.ipynb".
The first task is:
$$ \int z^{2}\, dz $$
Astropy can output $\LaTeX$ tables
Can you write Python code for it?

from astropy.io import ascii
from astropy.table import QTable

T = QTable.read('Zodiac.csv', format='ascii.csv')

T[0:3]

ascii.write(T, format='latex')


I am working on the file "08_Python_LaTeX.ipynb".
The first task is:
Python uses the $\LaTeX$ language to typeset equations.
Use a single set of $ to make your $\LaTeX$ inline and a double set $$ to center
This code will produce the output:
$$ \int \cos(x)\ dx = \sin(x) $$
Use can use $\LaTeX$ in plots:
Can you write Python code for it?

plt.style.use('ggplot')

x = np.linspace(0,2*np.pi,100)
y = np.sin(5*x) * np.exp(-x)

plt.plot(x,y)
plt.title("The function $y\ =\ \sin(5x)\ e^{-x}$")
plt.xlabel("This is in units of 2$\pi$")

plt.text(2.0, 0.4, '$\Delta t = \gamma\, \Delta t$', color='green', fontsize=36)


I am working on the file "08_Python_LaTeX.ipynb".
The first task is:
Some websites to open up for class:

Special Relativity

ShareLatex

ShareLatex Docs and Help

Latex Symbols

Latex draw symbols

The SAO/NASA Astrophysics Data System

Latex wikibook

Assignment for Week 8
Can you write Python code for it?

t = np.linspace(0,12*np.pi,2000)

fig,ax = plt.subplots(1,1)                    # One window
fig.set_size_inches(11,8.5)                   # (width,height) - letter paper landscape

fig.tight_layout()                            # Make better use of space on plot


I am working on the file "08_Text_Analysis/01 - Text Classification - SMS Ham vs. Spam - Data Preparation.ipynb".
The first task is:
Create Training and Validation Datasets
Can you write Python code for it?

exclude = ['\t', '"']
def clean_text(text):
    for c in exclude:
        text=text.replace(c,'')
    return text.lower().strip()

sms_processed = list(map(lambda text: clean_text(text),
                         dataset['sms'].values))

dataset['sms'] = sms_processed

splitter =  model_selection.StratifiedShuffleSplit(n_splits=1,
                                                   test_size=0.25,
                                                   random_state=19850610)

splits = list(splitter.split(X=dataset['sms'], y=dataset['class']))
train_index = splits[0][0]
valid_index = splits[0][1]

train_df = dataset.loc[train_index,:]
print(len(train_df))

valid_df = dataset.loc[valid_index,:]
print(len(valid_df))

print("Training Set")
training_value_counts = train_df['class'].value_counts()
print(training_value_counts)
print("ham %: {}".format(round(training_value_counts[0]/len(train_df)*100,2)))
print("ham %: {}".format(round(training_value_counts[1]/len(train_df)*100,2)))
print("")
print("Validation Set")
validation_value_counts = valid_df['class'].value_counts()
print(validation_value_counts)
print("ham %: {}".format(round(validation_value_counts[0]/len(valid_df)*100,2)))
print("ham %: {}".format(round(validation_value_counts[1]/len(valid_df)*100,2)))


I am working on the file "08_Text_Analysis/01 - Text Classification - SMS Ham vs. Spam - Data Preparation.ipynb".
The first task is:
Save Training and Validation Datasets
Can you write Python code for it?

train_df.to_csv("data/sms-spam/train-data.tsv", header=False, index=False, sep='\t')
valid_df.to_csv("data/sms-spam/valid-data.tsv", header=False, index=False, sep='\t')

pd.read_csv("data/sms-spam/train-data.tsv", sep='\t', names=['class','sms']).tail()

pd.read_csv("data/sms-spam/valid-data.tsv", sep='\t', names=['class','sms']).tail()


I am working on the file "08_Text_Analysis/02 - Text Classification - SMS Ham vs. Spam - Document Embedding.ipynb".
The first task is:
3. Define Model Function
Can you write Python code for it?

def process_text(text_feature):

    # Load vocabolary lookup table to map word => word_id
    vocab_table = tf.contrib.lookup.index_table_from_file(vocabulary_file=VOCAB_LIST_FILE,
                                                          num_oov_buckets=1, default_value=-1)
    # Get text feature
    smss = text_feature
    # Split text to words -> this will produce sparse tensor with variable-lengthes (word count) entries
    words = tf.string_split(smss)
    # Convert sparse tensor to dense tensor by padding each entry to match the longest in the batch
    dense_words = tf.sparse_tensor_to_dense(words, default_value=PAD_WORD)
    # Convert word to word_ids via the vocab lookup table
    word_ids = vocab_table.lookup(dense_words)
    # Create a word_ids padding
    padding = tf.constant([[0,0],[0,MAX_DOCUMENT_LENGTH]])
    # Pad all the word_ids entries to the maximum document length
    word_ids_padded = tf.pad(word_ids, padding)
    word_id_vector = tf.slice(word_ids_padded, [0,0], [-1, MAX_DOCUMENT_LENGTH])

    # Return the final word_id_vector
    return word_id_vector

def model_fn(features, labels, mode, params):

    hidden_units = params.hidden_units
    output_layer_size = len(TARGET_LABELS)
    embedding_size = params.embedding_size

    # word_id_vector
    word_id_vector = process_text(features[TEXT_FEATURE_NAME])
    # print("word_id_vector: {}".format(word_id_vector)) # (?, MAX_DOCUMENT_LENGTH)

    # layer to take each word_id and convert it into vector (embeddings)
    word_embeddings = tf.contrib.layers.embed_sequence(word_id_vector, vocab_size=N_WORDS,
                                                 embed_dim=embedding_size)
    # print("word_embeddings: {}".format(word_embeddings)) # (?, MAX_DOCUMENT_LENGTH, embbeding_size)

    # Get the average of the word embeddings in each entry to get a single embedding vector for each doc
    #doc_embeddings = tf.reduce_mean(word_embeddings, axis=1)
    #print("doc_embeddings: {}".format(doc_embeddings)) # (?,...


I am working on the file "08_Text_Analysis/02 - Text Classification - SMS Ham vs. Spam - Document Embedding.ipynb".
The first task is:
4. Run Experiment
a. Set HParam and RunConfig
Can you write Python code for it?

TRAIN_SIZE = 4179
NUM_EPOCHS = 10
BATCH_SIZE = 250
EVAL_AFTER_SEC = 60
TOTAL_STEPS = int((TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS)

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    embedding_size = 3,
    hidden_units=None, #[8],
    max_steps = TOTAL_STEPS,
    learning_rate = 0.01
)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.estimator.RunConfig(
    log_step_count_steps=5000,
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", TOTAL_STEPS)
print("That is 1 evaluation step after each",EVAL_AFTER_SEC,"training seconds")


I am working on the file "08_Text_Analysis/02 - Text Classification - SMS Ham vs. Spam - Document Embedding.ipynb".
The first task is:
6. Predict Using Serving Function
Can you write Python code for it?

import os

export_dir = model_dir +"/export/predict/"

saved_model_dir = export_dir + "/" + os.listdir(path=export_dir)[-1]

print(saved_model_dir)
print("")

predictor_fn = tf.contrib.predictor.from_saved_model(
    export_dir = saved_model_dir,
    signature_def_key="prediction"
)

output = predictor_fn(
    {
        'sms':[
            'ok, I will be with you in 5 min. see you then',
            'win 1000 cash free of charge promo hot deal sexy',
            'hot girls sexy tonight call girls waiting call chat'
        ]

    }
)
print(output)


I am working on the file "08_Text_Analysis/02 - Text Classification - SMS Ham vs. Spam - Document Embedding.ipynb".
The first task is:
b. Data pipeline input function
Can you write Python code for it?

def parse_label_column(label_string_tensor):
    table = tf.contrib.lookup.index_table_from_tensor(tf.constant(TARGET_LABELS))
    return table.lookup(label_string_tensor)

def input_fn(files_name_pattern, mode=tf.estimator.ModeKeys.EVAL,
                 skip_header_lines=0,
                 num_epochs=1,
                 batch_size=200):

    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False

    num_threads = multiprocessing.cpu_count() if MULTI_THREADING else 1

    buffer_size = 2 * batch_size + 1

    print("")
    print("* data input_fn:")
    print("================")
    print("Input file(s): {}".format(files_name_pattern))
    print("Batch size: {}".format(batch_size))
    print("Epoch Count: {}".format(num_epochs))
    print("Mode: {}".format(mode))
    print("Thread Count: {}".format(num_threads))
    print("Shuffle: {}".format(shuffle))
    print("================")
    print("")

    file_names = tf.matching_files(files_name_pattern)
    dataset = data.TextLineDataset(filenames=file_names)

    dataset = dataset.skip(skip_header_lines)

    if shuffle:
        dataset = dataset.shuffle(buffer_size)

    dataset = dataset.map(lambda tsv_row: parse_tsv_row(tsv_row),
                          num_parallel_calls=num_threads)

    dataset = dataset.batch(batch_size)
    dataset = dataset.repeat(num_epochs)
    dataset = dataset.prefetch(buffer_size)

    iterator = dataset.make_one_shot_iterator()

    features, target = iterator.get_next()
    return features, parse_label_column(target)


I am working on the file "08_Text_Analysis/02 - Text Classification - SMS Ham vs. Spam - Document Embedding.ipynb".
The first task is:
b. Define serving function
Can you write Python code for it?

def serving_input_fn():

    receiver_tensor = {
      'sms': tf.placeholder(tf.string, [None]),
    }

    features = {
      key: tensor
      for key, tensor in receiver_tensor.items()
    }

    return tf.estimator.export.ServingInputReceiver(
        features, receiver_tensor)


I am working on the file "08_Text_Analysis/02 - Text Classification - SMS Ham vs. Spam - Document Embedding.ipynb".
The first task is:
c. Define TrainSpec and EvaluSpec
Can you write Python code for it?

train_spec = tf.estimator.TrainSpec(
    input_fn = lambda: input_fn(
        TRAIN_DATA_FILES_PATTERN,
        mode = tf.estimator.ModeKeys.TRAIN,
        num_epochs=hparams.num_epochs,
        batch_size=hparams.batch_size
    ),
    max_steps=hparams.max_steps,
    hooks=None
)

eval_spec = tf.estimator.EvalSpec(
    input_fn = lambda: input_fn(
        VALID_DATA_FILES_PATTERN,
        mode=tf.estimator.ModeKeys.EVAL,
        batch_size=hparams.batch_size
    ),
    exporters=[tf.estimator.LatestExporter(
        name="predict", # the name of the folder in which the model will be exported to under export
        serving_input_receiver_fn=serving_input_fn,
        exports_to_keep=1,
        as_text=True)],
    steps=None,
    throttle_secs = EVAL_AFTER_SEC
)


I am working on the file "08_Text_Analysis/02 - Text Classification - SMS Ham vs. Spam - Document Embedding.ipynb".
The first task is:
d. Run Experiment via train_and_evaluate
Can you write Python code for it?

if not RESUME_TRAINING:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...")

tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow()
print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................")

estimator = create_estimator(run_config, hparams)

tf.estimator.train_and_evaluate(
    estimator=estimator,
    train_spec=train_spec,
    eval_spec=eval_spec
)

time_end = datetime.utcnow()
print(".......................................")
print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "08_Text_Analysis/03 - Text Classification - SMS Ham vs. Spam - Word Embeddings + CNN.ipynb".
The first task is:
3. Define Model Function
Can you write Python code for it?

def process_text(text_feature):

    # Load vocabolary lookup table to map word => word_id
    vocab_table = tf.contrib.lookup.index_table_from_file(vocabulary_file=VOCAB_LIST_FILE,
                                                          num_oov_buckets=1, default_value=-1)
    # Get text feature
    smss = text_feature
    # Split text to words -> this will produce sparse tensor with variable-lengthes (word count) entries
    words = tf.string_split(smss)
    # Convert sparse tensor to dense tensor by padding each entry to match the longest in the batch
    dense_words = tf.sparse_tensor_to_dense(words, default_value=PAD_WORD)
    # Convert word to word_ids via the vocab lookup table
    word_ids = vocab_table.lookup(dense_words)
    # Create a word_ids padding
    padding = tf.constant([[0,0],[0,MAX_DOCUMENT_LENGTH]])
    # Pad all the word_ids entries to the maximum document length
    word_ids_padded = tf.pad(word_ids, padding)
    word_id_vector = tf.slice(word_ids_padded, [0,0], [-1, MAX_DOCUMENT_LENGTH])

    # Return the final word_id_vector
    return word_id_vector

def model_fn(features, labels, mode, params):

    hidden_units = params.hidden_units
    output_layer_size = len(TARGET_LABELS)
    embedding_size = params.embedding_size
    window_size = params.window_size
    stride = int(window_size/2)
    filters = params.filters

    # word_id_vector
    word_id_vector = process_text(features[TEXT_FEATURE_NAME])
    # print("word_id_vector: {}".format(word_id_vector)) # (?, MAX_DOCUMENT_LENGTH)

    # layer to take each word_id and convert it into vector (embeddings)
    word_embeddings = tf.contrib.layers.embed_sequence(word_id_vector, vocab_size=N_WORDS,
                                                 embed_dim=embedding_size)
    #print("word_embeddings: {}".format(word_embeddings)) # (?, MAX_DOCUMENT_LENGTH, embbeding_size)

    # convolution
    words_conv = tf.layers.conv1d(word_embeddings, filters=filters, kernel_size=window_size,
                          ...


I am working on the file "08_Text_Analysis/03 - Text Classification - SMS Ham vs. Spam - Word Embeddings + CNN.ipynb".
The first task is:
4. Run Experiment
a. Set HParam and RunConfig
Can you write Python code for it?

TRAIN_SIZE = 4179
NUM_EPOCHS = 10
BATCH_SIZE = 250
EVAL_AFTER_SEC = 60
TOTAL_STEPS = int((TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS)

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    embedding_size = 3,
    window_size = 3,
    filters = 2,
    hidden_units=None, #[8],
    max_steps = TOTAL_STEPS,
    learning_rate = 0.01
)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.estimator.RunConfig(
    log_step_count_steps=5000,
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", TOTAL_STEPS)
print("That is 1 evaluation step after each",EVAL_AFTER_SEC,"training seconds")


I am working on the file "08_Text_Analysis/03 - Text Classification - SMS Ham vs. Spam - Word Embeddings + CNN.ipynb".
The first task is:
6. Predict Using Serving Function
Can you write Python code for it?


I am working on the file "08_Text_Analysis/03 - Text Classification - SMS Ham vs. Spam - Word Embeddings + CNN.ipynb".
The first task is:
b. Data pipeline input function
Can you write Python code for it?


I am working on the file "08_Text_Analysis/03 - Text Classification - SMS Ham vs. Spam - Word Embeddings + CNN.ipynb".
The first task is:
d. Run Experiment via train_and_evaluate
Can you write Python code for it?


I am working on the file "08_Text_Analysis/04 - Text Classification - SMS Ham vs. Spam - Word Embeddings + LSTM.ipynb".
The first task is:
1. Define Dataset Metadata
Can you write Python code for it?

MAX_DOCUMENT_LENGTH = 50

PAD_WORD = '#=KS=#'

HEADER = ['class', 'sms']
HEADER_DEFAULTS = [['NA'], ['NA']]

TEXT_FEATURE_NAME = 'sms'

TARGET_NAME = 'class'

WEIGHT_COLUNM_NAME = 'weight'

TARGET_LABELS = ['spam', 'ham']

with open(N_WORDS_FILE) as file:
    N_WORDS = int(file.read())+2
print(N_WORDS)


I am working on the file "08_Text_Analysis/04 - Text Classification - SMS Ham vs. Spam - Word Embeddings + LSTM.ipynb".
The first task is:
3. Define Model Function
Can you write Python code for it?

def process_text(text_feature):

    # Load vocabolary lookup table to map word => word_id
    vocab_table = tf.contrib.lookup.index_table_from_file(vocabulary_file=VOCAB_LIST_FILE,
                                                          num_oov_buckets=1, default_value=-1)
    # Get text feature
    smss = text_feature
    # Split text to words -> this will produce sparse tensor with variable-lengthes (word count) entries
    words = tf.string_split(smss)
    # Convert sparse tensor to dense tensor by padding each entry to match the longest in the batch
    dense_words = tf.sparse_tensor_to_dense(words, default_value=PAD_WORD)
    # Convert word to word_ids via the vocab lookup table
    word_ids = vocab_table.lookup(dense_words)
    # Create a word_ids padding
    padding = tf.constant([[0,0],[0,MAX_DOCUMENT_LENGTH]])
    # Pad all the word_ids entries to the maximum document length
    word_ids_padded = tf.pad(word_ids, padding)
    word_id_vector = tf.slice(word_ids_padded, [0,0], [-1, MAX_DOCUMENT_LENGTH])

    # Return the final word_id_vector
    return word_id_vector

def model_fn(features, labels, mode, params):

    hidden_units = params.hidden_units
    output_layer_size = len(TARGET_LABELS)
    embedding_size = params.embedding_size
    forget_bias = params.forget_bias
    keep_prob = params.keep_prob

    # word_id_vector
    word_id_vector = process_text(features[TEXT_FEATURE_NAME])
    # print("word_id_vector: {}".format(word_id_vector)) # (?, MAX_DOCUMENT_LENGTH)

    # layer to take each word_id and convert it into vector (embeddings)
    word_embeddings = tf.contrib.layers.embed_sequence(word_id_vector, vocab_size=N_WORDS,
                                                 embed_dim=embedding_size)
    #print("word_embeddings: {}".format(word_embeddings)) # (?, MAX_DOCUMENT_LENGTH, embbeding_size)

    # configure the RNN
    rnn_layers = [tf.nn.rnn_cell.LSTMCell(
        num_units=size,
        forget_bias=params.forget_bias,
        activation=tf.nn.tanh) for...


I am working on the file "08_Text_Analysis/04 - Text Classification - SMS Ham vs. Spam - Word Embeddings + LSTM.ipynb".
The first task is:
4. Run Experiment
a. Set HParam and RunConfig
Can you write Python code for it?

TRAIN_SIZE = 4179
NUM_EPOCHS = 100
BATCH_SIZE = 250
EVAL_AFTER_SEC = 60
TOTAL_STEPS = int((TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS)

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    embedding_size = 5,
    forget_bias=1.0,
    keep_prob = 0.8,
    hidden_units=[24, 16],
    max_steps = TOTAL_STEPS,
    learning_rate = 0.01
)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.estimator.RunConfig(
    log_step_count_steps=5000,
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", TOTAL_STEPS)
print("That is 1 evaluation step after each",EVAL_AFTER_SEC,"training seconds")


I am working on the file "08_Text_Analysis/04 - Text Classification - SMS Ham vs. Spam - Word Embeddings + LSTM.ipynb".
The first task is:
5. Evaluate the Model
Can you write Python code for it?


I am working on the file "08_Text_Analysis/04 - Text Classification - SMS Ham vs. Spam - Word Embeddings + LSTM.ipynb".
The first task is:
b. Data pipeline input function
Can you write Python code for it?


I am working on the file "08_Text_Analysis/04 - Text Classification - SMS Ham vs. Spam - Word Embeddings + LSTM.ipynb".
The first task is:
b. Define serving function
Can you write Python code for it?


I am working on the file "08_Text_Analysis/04 - Text Classification - SMS Ham vs. Spam - Word Embeddings + LSTM.ipynb".
The first task is:
d. Run Experiment via train_and_evaluate
Can you write Python code for it?


I am working on the file "08_Text_Analysis/05 - Text Classification - Hacker News - End-to-End + TF-Hub Sentence Embedding.ipynb".
The first task is:
2. Create feature columns
Can you write Python code for it?

import tensorflow_hub as hub
print hub.__version__

def create_feature_columns(hparams):

    title_embeding_column = hub.text_embedding_column(
        "title", "https://tfhub.dev/google/universal-sentence-encoder/1",
        trainable=hparams.trainable_embedding)

    feature_columns = [title_embeding_column]

    print "feature columns: \n {}".format(feature_columns)
    print ""

    return feature_columns


I am working on the file "08_Text_Analysis/05 - Text Classification - Hacker News - End-to-End + TF-Hub Sentence Embedding.ipynb".
The first task is:
3. Create a model using the  premade DNNClassifer
Can you write Python code for it?

def create_estimator_hub(hparams, run_config):

    feature_columns = create_feature_columns(hparams)

    optimizer = tf.train.AdamOptimizer(learning_rate=hparams.learning_rate)

    estimator = tf.estimator.DNNClassifier(
        feature_columns=feature_columns,
        n_classes =len(TARGET_LABELS),
        label_vocabulary=TARGET_LABELS,
        hidden_units=hparams.hidden_units,
        optimizer=optimizer,
        config=run_config
    )

    return estimator


I am working on the file "08_Text_Analysis/05 - Text Classification - Hacker News - End-to-End + TF-Hub Sentence Embedding.ipynb".
The first task is:
4. Define experiment
a) HParams and RunConfig
Can you write Python code for it?

TRAIN_SIZE = 73124
NUM_EPOCHS = 10
BATCH_SIZE = 1000

TOTAL_STEPS = (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS
EVAL_EVERY_SEC = 60

hparams  = tf.contrib.training.HParams(
    num_epochs=NUM_EPOCHS,
    batch_size=BATCH_SIZE,
    trainable_embedding=False,
    learning_rate=0.01,
    hidden_units=[256, 128],
    max_steps=TOTAL_STEPS
)

MODEL_NAME = 'dnn_estimator_hub'
model_dir = os.path.join(Params.MODELS_DIR, MODEL_NAME)

run_config = tf.estimator.RunConfig(
    tf_random_seed=19830610,
    log_step_count_steps=1000,
    save_checkpoints_secs=EVAL_EVERY_SEC,
    keep_checkpoint_max=1,
    model_dir=model_dir
)

print(hparams)
print("")
print("Model Directory:", run_config.model_dir)
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", TOTAL_STEPS)


I am working on the file "08_Text_Analysis/05 - Text Classification - Hacker News - End-to-End + TF-Hub Sentence Embedding.ipynb".
The first task is:
5. Run Pipeline
Can you write Python code for it?

from datetime import datetime
import shutil

job_name = 'preprocess-hackernews-data' + '-' + datetime.utcnow().strftime('%y%m%d-%H%M%S')

options = {
    'region': Params.REGION,
    'staging_location': os.path.join(Params.TEMP_DIR, 'staging'),
    'temp_location': Params.TEMP_DIR,
    'job_name': job_name,
    'project': Params.GCP_PROJECT_ID
}

opts = beam.pipeline.PipelineOptions(flags=[], **options)
runner = 'DirectRunner' if Params.PLATFORM == 'local' else 'DirectRunner'

if Params.TRANSFORM:

    if Params.PLATFORM == 'local':
        shutil.rmtree(Params.DATA_DIR, ignore_errors=True)

    print 'Launching {} job {} ... hang on'.format(runner, job_name)

    run_pipeline(runner, opts)
    print "Pipline completed."
else:
    print "Transformation skipped!"

%%bash

ls data/news
echo ""
head data/news/train-00000-of-00005.tsv


I am working on the file "08_Text_Analysis/05 - Text Classification - Hacker News - End-to-End + TF-Hub Sentence Embedding.ipynb".
The first task is:
5. Run experiment
Can you write Python code for it?

from datetime import datetime
import shutil

if Params.TRAIN:
    if not Params.RESUME_TRAINING:
        print("Removing previous training artefacts...")
        shutil.rmtree(model_dir, ignore_errors=True)
    else:
        print("Resuming training...")

    tf.logging.set_verbosity(tf.logging.INFO)

    time_start = datetime.utcnow()
    print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
    print(".......................................")

    estimator = create_estimator_hub(hparams, run_config)

    tf.estimator.train_and_evaluate(
        estimator=estimator,
        train_spec=train_spec,
        eval_spec=eval_spec
    )

    time_end = datetime.utcnow()
    print(".......................................")
    print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
    print("")
    time_elapsed = time_end - time_start
    print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))
else:
    print "Training was skipped!"


I am working on the file "08_Text_Analysis/05 - Text Classification - Hacker News - End-to-End + TF-Hub Sentence Embedding.ipynb".
The first task is:
6. Evaluate the model
Can you write Python code for it?


I am working on the file "08_Text_Analysis/05 - Text Classification - Hacker News - End-to-End + TF-Hub Sentence Embedding.ipynb".
The first task is:
7. Use SavedModel for predictions
Can you write Python code for it?

import os

export_dir = model_dir +"/export/estimate/"
saved_model_dir = os.path.join(export_dir, os.listdir(export_dir)[0])

print(saved_model_dir)
print("")

predictor_fn = tf.contrib.predictor.from_saved_model(
    export_dir = saved_model_dir,
    signature_def_key="predict"
)

output = predictor_fn(
    {
        'title':[
            'Microsoft and Google are joining forces for a new AI framework',
            'A new version of Python is mind blowing',
            'EU is investigating new data privacy policies'
        ]

    }
)
print(output)


I am working on the file "08_Text_Analysis/05 - Text Classification - Hacker News - End-to-End + TF-Hub Sentence Embedding.ipynb".
The first task is:
Create a ML Data Files using Dataflow
The data processing pipeline will do the following:
1. Read the data (key, title, source) from BigQuery
2. Process text (if needed) and convert each BQ raw to tsv
3. Save data to tsv files
1. Source Query
Can you write Python code for it?

bq_query = '''
SELECT
    key,
    REGEXP_REPLACE(title, '[^a-zA-Z0-9 $.-]', ' ') AS title,
    source
FROM
(
    SELECT
        ARRAY_REVERSE(SPLIT(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.'))[OFFSET(1)] AS source,
        title,
        ABS(FARM_FINGERPRINT(title)) AS Key
    FROM
      `bigquery-public-data.hacker_news.stories`
    WHERE
      REGEXP_CONTAINS(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.com$')
      AND LENGTH(title) > 10
)
WHERE (source = 'github' OR source = 'nytimes' OR source = 'techcrunch')
'''


I am working on the file "08_Text_Analysis/05 - Text Classification - Hacker News - End-to-End + TF-Hub Sentence Embedding.ipynb".
The first task is:
TF Text Classification Model with TF Hub for Text Encoding
1. Define metadata & input function
Can you write Python code for it?

import tensorflow as tf
from tensorflow import data
print tf.__version__

RAW_HEADER = 'key,title,source'.split(',')
RAW_DEFAULTS = [['NA'],['NA'],['NA']]
TARGET_FEATRUE_NAME = 'source'
TARGET_LABELS = ['github', 'nytimes', 'techcrunch']
TEXT_FEATURE_NAME = 'title'
KEY_COLUMN = 'key'

def parse_tsv(tsv_row):

    columns = tf.decode_csv(tsv_row, record_defaults=RAW_DEFAULTS, field_delim='\t')
    features = dict(zip(RAW_HEADER, columns))

    features.pop(KEY_COLUMN)
    target = features.pop(TARGET_FEATRUE_NAME)

    return features, target

def generate_tsv_input_fn(files_pattern,
                          mode=tf.estimator.ModeKeys.EVAL,
                          num_epochs=1,
                          batch_size=200):

    def _input_fn():

        #file_names = data.Dataset.list_files(files_pattern)
        file_names = tf.matching_files(files_pattern)

        if Params.EAGER:
            print file_names

        dataset = data.TextLineDataset(file_names)

        dataset = dataset.apply(
                tf.contrib.data.shuffle_and_repeat(count=num_epochs,
                                                   buffer_size=batch_size*2)
        )

        dataset = dataset.apply(
                tf.contrib.data.map_and_batch(parse_tsv,
                                              batch_size=batch_size,
                                              num_parallel_batches=2)
        )

        datset = dataset.prefetch(batch_size)

        if Params.EAGER:
            return dataset

        iterator = dataset.make_one_shot_iterator()
        features, target = iterator.get_next()
        return features, target

    return _input_fn


I am working on the file "08_Text_Analysis/05 - Text Classification - Hacker News - End-to-End + TF-Hub Sentence Embedding.ipynb".
The first task is:
b) Serving function
Can you write Python code for it?

def generate_serving_input_fn():

    def _serving_fn():

        receiver_tensor = {
          'title': tf.placeholder(dtype=tf.string, shape=[None])
        }

        return tf.estimator.export.ServingInputReceiver(
            receiver_tensor, receiver_tensor)

    return _serving_fn


I am working on the file "08_Text_Analysis/05 - Text Classification - Hacker News - End-to-End + TF-Hub Sentence Embedding.ipynb".
The first task is:
c) TrainSpec & EvalSpec
Can you write Python code for it?

train_spec = tf.estimator.TrainSpec(
    input_fn = generate_tsv_input_fn(
        Params.RAW_TRAIN_DATA_FILE_PREFEX+"*",
        mode = tf.estimator.ModeKeys.TRAIN,
        num_epochs=hparams.num_epochs,
        batch_size=hparams.batch_size
    ),
    max_steps=hparams.max_steps,
    hooks=None
)

eval_spec = tf.estimator.EvalSpec(
    input_fn = generate_tsv_input_fn(
        Params.RAW_EVAL_DATA_FILE_PREFEX+"*",
        mode=tf.estimator.ModeKeys.EVAL,
        num_epochs=1,
        batch_size=hparams.batch_size
    ),
    exporters=[tf.estimator.LatestExporter(
        name="estimate", # the name of the folder in which the model will be exported to under export
        serving_input_receiver_fn=generate_serving_input_fn(),
        exports_to_keep=1,
        as_text=False)],
    steps=None,
    throttle_secs=EVAL_EVERY_SEC
)


I am working on the file "08_Text_Analysis/06 - Part_1 - Text Classification - Hacker News - Data Preprocessing with TFT.ipynb".
The first task is:
1. Source Query
Can you write Python code for it?

bq_query = '''
SELECT
    key,
    REGEXP_REPLACE(title, '[^a-zA-Z0-9 $.-]', ' ') AS title,
    source
FROM
(
    SELECT
        ARRAY_REVERSE(SPLIT(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.'))[OFFSET(1)] AS source,
        title,
        ABS(FARM_FINGERPRINT(title)) AS Key
    FROM
      `bigquery-public-data.hacker_news.stories`
    WHERE
      REGEXP_CONTAINS(REGEXP_EXTRACT(url, '.*://(.[^/]+)/'), '.com$')
      AND LENGTH(title) > 10
)
WHERE (source = 'github' OR source = 'nytimes' OR source = 'techcrunch')
'''

def get_source_query(step):

    if step == 'train':
        source_query = 'SELECT * FROM ({}) WHERE MOD(key,100) <= 75'.format(bq_query)
    else:
        source_query = 'SELECT * FROM ({}) WHERE MOD(key,100) > 75'.format(bq_query)

    return source_query


I am working on the file "08_Text_Analysis/06 - Part_1 - Text Classification - Hacker News - Data Preprocessing with TFT.ipynb".
The first task is:
3. Preprocessing functions
Can you write Python code for it?

def get_features(bq_row):

    CSV_HEADER = 'key,title,source'.split(',')

    input_features = {}

    for feature_name in CSV_HEADER:
        input_features[feature_name] = str(bq_row[feature_name]).lower()

    return input_features

def preprocessing_fn(input_features):

    text = input_features[TEXT_FEATURE_NAME]

    text_tokens = tf.string_split(text, DELIMITERS)
    text_tokens_indcies = tft.string_to_int(text_tokens, top_k=VOCAB_SIZE)
    bag_of_words_indices, text_weight = tft.tfidf(text_tokens_indcies, VOCAB_SIZE + 1)

    output_features = {}
    output_features[TEXT_FEATURE_NAME] = input_features[TEXT_FEATURE_NAME]
    output_features['bow'] = bag_of_words_indices
    output_features['weight'] = text_weight
    output_features[TARGET_FEATURE_NAME] = input_features[TARGET_FEATURE_NAME]

    return output_features


I am working on the file "08_Text_Analysis/06 - Part_1 - Text Classification - Hacker News - Data Preprocessing with TFT.ipynb".
The first task is:
4. Beam Pipeline
Can you write Python code for it?

import apache_beam as beam

def run_pipeline(runner, opts):

    print("Sink train data files: {}".format(Params.TRANSFORMED_TRAIN_DATA_FILE_PREFIX))
    print("Sink data files: {}".format(Params.TRANSFORMED_EVAL_DATA_FILE_PREFIX))
    print("Temporary directory: {}".format(Params.TEMP_DIR))
    print("")

    with beam.Pipeline(runner, options=opts) as pipeline:
        with impl.Context(Params.TEMP_DIR):

            ###### analyze & transform train #########################################################
            if(runner=='DirectRunner'):
                print("")
                print("Transform training data....")
                print("")

            step = 'train'
            source_query = get_source_query(step)

            # Read raw train data from BQ and cleanup
            raw_train_data = (
              pipeline
              | '{} - Read Data from BigQuery'.format(step) >> beam.io.Read(beam.io.BigQuerySource(query=source_query, use_standard_sql=True))
              | '{} - Extract Features'.format(step) >> beam.Map(get_features)
            )

            # create a train dataset from the data and schema
            raw_train_dataset = (raw_train_data, raw_metadata)

            # analyze and transform raw_train_dataset to produced transformed_train_dataset and transform_fn
            transformed_train_dataset, transform_fn = (
                raw_train_dataset
                | '{} - Analyze & Transform'.format(step) >> impl.AnalyzeAndTransformDataset(preprocessing_fn)
            )

            # get data and schema separately from the transformed_train_dataset
            transformed_train_data, transformed_metadata = transformed_train_dataset

            # write transformed train data to sink
            _ = (
                transformed_train_data
                | '{} - Write Transformed Data as tfrecords'.format(step) >> beam.io.tfrecordio.WriteToTFRecord(
                   ...


I am working on the file "08_Text_Analysis/06 - Part_1 - Text Classification - Hacker News - Data Preprocessing with TFT.ipynb".
The first task is:
Importing libraries
Can you write Python code for it?

import apache_beam as beam

import tensorflow as tf
import tensorflow_transform as tft
import tensorflow_transform.coders as tft_coders

from tensorflow.contrib.learn.python.learn.utils import input_fn_utils

from tensorflow_transform.beam import impl
from tensorflow_transform.beam.tft_beam_io import transform_fn_io
from tensorflow_transform.tf_metadata import metadata_io
from tensorflow_transform.tf_metadata import dataset_schema
from tensorflow_transform.tf_metadata import dataset_metadata
from tensorflow_transform.saved import saved_transform_io


I am working on the file "08_Text_Analysis/06 - Part_1 - Text Classification - Hacker News - Data Preprocessing with TFT.ipynb".
The first task is:
Text Classification using TensorFlow and Google Cloud - Part 1
This bigquery-public-data:hacker_news contains all stories and comments from Hacker News from its launch in 2006.  Each story contains a story id, url, the title of the story, tthe author that made the post, when it was written, and the number of points the story received.
The objective is, given the title of the story, we want to build an ML model that can predict the source of this story.
Data preparation with tf.Transform and DataFlow
This notebook illustrates how to build a Beam pipeline using tf.transform to prepare ML 'train' and 'eval' datasets.
The pipeline includes the following steps:
1. Read data from BigQuery
2. Extract and clean features from BQ rows
3. Use tf.transfrom to process the text and produce the following features for each entry
 * title: Raw text - string
 * bow: Bag of word indecies - sparse vector of integers
 * weight: TF.IDF values - sparse vector of floats
 * source: target feature - string
4. Save the data as .tfrecord files
Setting Global Parameters
Can you write Python code for it?

import os

class Params:
    pass

# Set to run on GCP
Params.GCP_PROJECT_ID = 'ksalama-gcp-playground'
Params.REGION = 'europe-west1'
Params.BUCKET = 'ksalama-gcs-cloudml'

Params.PLATFORM = 'local' # local | GCP

Params.DATA_DIR = 'data/news'  if Params.PLATFORM == 'local' else 'gs://{}/data/news'.format(Params.BUCKET)

Params.TRANSFORMED_DATA_DIR = os.path.join(Params.DATA_DIR, 'transformed')
Params.TRANSFORMED_TRAIN_DATA_FILE_PREFIX = os.path.join(Params.TRANSFORMED_DATA_DIR, 'train')
Params.TRANSFORMED_EVAL_DATA_FILE_PREFIX = os.path.join(Params.TRANSFORMED_DATA_DIR, 'eval')

Params.TEMP_DIR = os.path.join(Params.DATA_DIR, 'tmp')

Params.MODELS_DIR = 'models/news' if Params.PLATFORM == 'local' else 'gs://{}/models/news'.format(Params.BUCKET)

Params.TRANSFORM_ARTEFACTS_DIR = os.path.join(Params.MODELS_DIR,'transform')

Params.TRANSFORM = True


I am working on the file "08_Text_Analysis/06 - Part_2 - Text Classification - Hacker News - DNNClassifier with TF-Hub Sentence Embedding.ipynb".
The first task is:
1. Define Metadata
Can you write Python code for it?

RAW_HEADER = 'key,title,source'.split(',')
RAW_DEFAULTS = [['NA'],['NA'],['NA']]
TARGET_FEATURE_NAME = 'source'
TARGET_LABELS = ['github', 'nytimes', 'techcrunch']
TEXT_FEATURE_NAME = 'title'
KEY_COLUMN = 'key'

VOCAB_SIZE = 20000
TRAIN_SIZE = 73124
EVAL_SIZE = 23079

DELIMITERS = '.,!?() '

raw_metadata = dataset_metadata.DatasetMetadata(dataset_schema.Schema({
    KEY_COLUMN: dataset_schema.ColumnSchema(
        tf.string, [], dataset_schema.FixedColumnRepresentation()),
    TEXT_FEATURE_NAME: dataset_schema.ColumnSchema(
        tf.string, [], dataset_schema.FixedColumnRepresentation()),
    TARGET_FEATURE_NAME: dataset_schema.ColumnSchema(
        tf.string, [], dataset_schema.FixedColumnRepresentation()),
}))

transformed_metadata = metadata_io.read_metadata(
    os.path.join(Params.TRANSFORM_ARTEFACTS_DIR,"transformed_metadata"))

raw_feature_spec = raw_metadata.schema.as_feature_spec()
transformed_feature_spec = transformed_metadata.schema.as_feature_spec()

print transformed_feature_spec


I am working on the file "08_Text_Analysis/06 - Part_2 - Text Classification - Hacker News - DNNClassifier with TF-Hub Sentence Embedding.ipynb".
The first task is:
2. Define Input Function
Can you write Python code for it?

def parse_tf_example(tf_example):

    parsed_features = tf.parse_single_example(serialized=tf_example, features=transformed_feature_spec)
    target = parsed_features.pop(TARGET_FEATURE_NAME)

    return parsed_features, target

def generate_tfrecords_input_fn(files_pattern,
                          mode=tf.estimator.ModeKeys.EVAL,
                          num_epochs=1,
                          batch_size=200):

    def _input_fn():

        file_names = data.Dataset.list_files(files_pattern)

        if Params.EAGER:
            print file_names

        dataset = data.TFRecordDataset(file_names )

        dataset = dataset.apply(
                tf.contrib.data.shuffle_and_repeat(count=num_epochs,
                                                   buffer_size=batch_size*2)
        )

        dataset = dataset.apply(
                tf.contrib.data.map_and_batch(parse_tf_example,
                                              batch_size=batch_size,
                                              num_parallel_batches=2)
        )

        datset = dataset.prefetch(batch_size)

        if Params.EAGER:
            return dataset

        iterator = dataset.make_one_shot_iterator()
        features, target = iterator.get_next()
        return features, target

    return _input_fn


I am working on the file "08_Text_Analysis/06 - Part_2 - Text Classification - Hacker News - DNNClassifier with TF-Hub Sentence Embedding.ipynb".
The first task is:
5. Setup Experiment
5.1 HParams and RunConfig
Can you write Python code for it?

NUM_EPOCHS = 10
BATCH_SIZE = 1000

TOTAL_STEPS = (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS
EVAL_EVERY_SEC = 60

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    trainable_embedding = False,
    learning_rate = 0.01,
    hidden_units=[128, 64],
    max_steps = TOTAL_STEPS,

)

MODEL_NAME = 'dnn_estimator_hub'
model_dir = os.path.join(Params.MODELS_DIR, MODEL_NAME)

run_config = tf.estimator.RunConfig(
    tf_random_seed=19830610,
    log_step_count_steps=1000,
    save_checkpoints_secs=EVAL_EVERY_SEC,
    keep_checkpoint_max=1,
    model_dir=model_dir
)

print(hparams)
print("")
print("Model Directory:", run_config.model_dir)
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", TOTAL_STEPS)


I am working on the file "08_Text_Analysis/06 - Part_2 - Text Classification - Hacker News - DNNClassifier with TF-Hub Sentence Embedding.ipynb".
The first task is:
5.2 Serving function
Can you write Python code for it?


I am working on the file "08_Text_Analysis/06 - Part_2 - Text Classification - Hacker News - DNNClassifier with TF-Hub Sentence Embedding.ipynb".
The first task is:
6. Run experiment
Can you write Python code for it?

from datetime import datetime
import shutil

if Params.TRAIN:
    if not Params.RESUME_TRAINING:
        print("Removing previous training artefacts...")
        shutil.rmtree(model_dir, ignore_errors=True)
    else:
        print("Resuming training...")

    tf.logging.set_verbosity(tf.logging.INFO)

    time_start = datetime.utcnow()
    print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
    print(".......................................")

    estimator = create_estimator(hparams, run_config)

    tf.estimator.train_and_evaluate(
        estimator=estimator,
        train_spec=train_spec,
        eval_spec=eval_spec
    )

    time_end = datetime.utcnow()
    print(".......................................")
    print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
    print("")
    time_elapsed = time_end - time_start
    print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))
else:
    print "Training was skipped!"


I am working on the file "08_Text_Analysis/06 - Part_2 - Text Classification - Hacker News - DNNClassifier with TF-Hub Sentence Embedding.ipynb".
The first task is:
7. Evaluate the model
Can you write Python code for it?


I am working on the file "08_Text_Analysis/06 - Part_2 - Text Classification - Hacker News - DNNClassifier with TF-Hub Sentence Embedding.ipynb".
The first task is:
8. Use Saved Model for Predictions
Can you write Python code for it?


I am working on the file "08_Text_Analysis/06 - Part_3 - Text Classification - Hacker News - Custom Estimator Word Embedding.ipynb".
The first task is:
2. Define Input Function
Can you write Python code for it?


I am working on the file "08_Text_Analysis/06 - Part_3 - Text Classification - Hacker News - Custom Estimator Word Embedding.ipynb".
The first task is:
3. Create Custom Estimator using Model Function
3.1 Define model_fn
Can you write Python code for it?

def _bow_to_vector(sparse_bow_indecies):

    # Convert sparse tensor to dense tensor by padding each entry to match the longest in the batch
    bow_indecies = tf.sparse_tensor_to_dense(sparse_bow_indecies, default_value=PAD_VALUE)

    # Create a word_ids padding
    padding = tf.constant([[0,0],[0, MAX_WORDS_PER_TITLE]])

    # Pad all the word_ids entries to the maximum document length
    bow_indecies_padded = tf.pad(bow_indecies, padding)

    bow_vector = tf.slice(bow_indecies_padded, [0,0], [-1, MAX_WORDS_PER_TITLE])

    # Return the final word_id_vector
    return bow_vector

def _shallow_layers(features, params):

        # word_id_vector
        bow_vector = _bow_to_vector(features['bow'])

        # layer to take each word_id and convert it into vector (embeddings)
        word_embeddings = tf.contrib.layers.embed_sequence(bow_vector,
                                                           vocab_size=VOCAB_SIZE+2,
                                                           embed_dim=params.embedding_size)

        ### CNN Model ############################################################################
        if params.model_type == 'CNN':

            words_conv = tf.layers.conv1d(word_embeddings,
                                          filters=params.filters,
                                          kernel_size=params.window_size,
                                          strides=int(window_size/2),
                                          padding='SAME', activation=tf.nn.relu)

            words_conv_shape = words_conv.get_shape()
            dim = words_conv_shape[1] * words_conv_shape[2]
            shallow_layer = tf.reshape(words_conv,[-1, dim])

        ### LSTM Model ############################################################################
        elif params.model_type == 'LSTM':

            rnn_layers = [tf.nn.rnn_cell.LSTMCell(
                num_units=size,
               ...


I am working on the file "08_Text_Analysis/06 - Part_3 - Text Classification - Hacker News - Custom Estimator Word Embedding.ipynb".
The first task is:
4. Setup Experiment
4.1 HParams and RunConfig
Can you write Python code for it?

NUM_EPOCHS = 10
BATCH_SIZE = 1000

TOTAL_STEPS = (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS
EVAL_EVERY_SEC = 60

hparams  = tf.contrib.training.HParams(
    num_epochs=NUM_EPOCHS,
    batch_size=BATCH_SIZE,
    embedding_size = 50, # word embedding vector size
    learning_rate=0.01,
    hidden_units=[64, 32],
    max_steps=TOTAL_STEPS,
    model_type='MAXMIN_EMBEDDING', # CNN | LSTM | MAXMIN_EMBEDDING

    #CNN Params
    window_size = 3,
    filters = 2,

    #LSTM Params
    forget_bias=1.0,
    keep_prob = 0.8,

)

MODEL_NAME = 'dnn_estimator_custom'
model_dir = os.path.join(Params.MODELS_DIR, MODEL_NAME)

run_config = tf.estimator.RunConfig(
    tf_random_seed=19830610,
    log_step_count_steps=1000,
    save_checkpoints_secs=EVAL_EVERY_SEC,
    keep_checkpoint_max=1,
    model_dir=model_dir
)

print(hparams)
print("")
print("Model Directory:", run_config.model_dir)
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", TOTAL_STEPS)


I am working on the file "08_Text_Analysis/06 - Part_3 - Text Classification - Hacker News - Custom Estimator Word Embedding.ipynb".
The first task is:
4.2 Serving function
Can you write Python code for it?

def generate_serving_input_fn():

    def _serving_fn():

        receiver_tensor = {
          'title': tf.placeholder(dtype=tf.string, shape=[None])
        }

        _, transformed_features = (
            saved_transform_io.partially_apply_saved_transform(
                os.path.join(Params.TRANSFORM_ARTEFACTS_DIR, transform_fn_io.TRANSFORM_FN_DIR),
            receiver_tensor)
        )

        return tf.estimator.export.ServingInputReceiver(
            transformed_features, receiver_tensor)

    return _serving_fn


I am working on the file "08_Text_Analysis/06 - Part_3 - Text Classification - Hacker News - Custom Estimator Word Embedding.ipynb".
The first task is:
4.3 TrainSpec & EvalSpec
Can you write Python code for it?

train_spec = tf.estimator.TrainSpec(
    input_fn = generate_tfrecords_input_fn(
        Params.TRANSFORMED_TRAIN_DATA_FILE_PREFIX+"*",
        mode = tf.estimator.ModeKeys.TRAIN,
        num_epochs=hparams.num_epochs,
        batch_size=hparams.batch_size
    ),
    max_steps=hparams.max_steps,
    hooks=None
)

eval_spec = tf.estimator.EvalSpec(
    input_fn = generate_tfrecords_input_fn(
        Params.TRANSFORMED_EVAL_DATA_FILE_PREFIX+"*",
        mode=tf.estimator.ModeKeys.EVAL,
        num_epochs=1,
        batch_size=hparams.batch_size
    ),
    exporters=[tf.estimator.LatestExporter(
        name="estimate", # the name of the folder in which the model will be exported to under export
        serving_input_receiver_fn=generate_serving_input_fn(),
        exports_to_keep=1,
        as_text=False)],
    steps=None,
    throttle_secs=EVAL_EVERY_SEC
)


I am working on the file "08_Text_Analysis/06 - Part_3 - Text Classification - Hacker News - Custom Estimator Word Embedding.ipynb".
The first task is:
5. Run experiment
Can you write Python code for it?


I am working on the file "08_Text_Analysis/06 - Part_3 - Text Classification - Hacker News - Custom Estimator Word Embedding.ipynb".
The first task is:
7. Evaluate the model
Can you write Python code for it?


I am working on the file "08_Text_Analysis/06 - Part_3 - Text Classification - Hacker News - Custom Estimator Word Embedding.ipynb".
The first task is:
8. Use Saved Model for Predictions
Can you write Python code for it?

import os

export_dir = model_dir +"/export/estimate/"
saved_model_dir = os.path.join(export_dir, os.listdir(export_dir)[0])

print(saved_model_dir)
print("")

predictor_fn = tf.contrib.predictor.from_saved_model(
    export_dir = saved_model_dir,
    signature_def_key="predict"
)

output = predictor_fn(
    {
        'title':[
            'Microsoft and Google are joining forces for a new AI framework',
            'A new version of Python is mind blowing',
            'EU is investigating new data privacy policies'
        ]

    }
)
print(output)


I am working on the file "08_Text_Analysis/06 - Part_3 - Text Classification - Hacker News - Custom Estimator Word Embedding.ipynb".
The first task is:
Importing libraries
Can you write Python code for it?

import tensorflow as tf
from tensorflow import data

from tensorflow.contrib.learn.python.learn.utils import input_fn_utils
from tensorflow_transform.beam.tft_beam_io import transform_fn_io
from tensorflow_transform.tf_metadata import metadata_io
from tensorflow_transform.tf_metadata import dataset_schema
from tensorflow_transform.tf_metadata import dataset_metadata
from tensorflow_transform.saved import saved_transform_io

print tf.__version__


I am working on the file "08_Text_Analysis/06 - Part_4 - Text Classification - Hacker News - DNNClassifier with TF.IDF.ipynb".
The first task is:
1. Define Metadata
Can you write Python code for it?


I am working on the file "08_Text_Analysis/06 - Part_4 - Text Classification - Hacker News - DNNClassifier with TF.IDF.ipynb".
The first task is:
2. Define Input Function
Can you write Python code for it?


I am working on the file "08_Text_Analysis/06 - Part_4 - Text Classification - Hacker News - DNNClassifier with TF.IDF.ipynb".
The first task is:
4. Create a model using a premade DNNClassifer
Can you write Python code for it?

def create_estimator(hparams, run_config):

    feature_columns = create_feature_columns()

    optimizer = tf.train.AdamOptimizer(learning_rate=hparams.learning_rate)

    estimator = tf.estimator.DNNClassifier(
        feature_columns=feature_columns,
        n_classes =len(TARGET_LABELS),
        label_vocabulary=TARGET_LABELS,
        hidden_units=hparams.hidden_units,
        optimizer=optimizer,
        config=run_config
    )

    return estimator