I am working on the file "15. 선형 회귀 분석/01. 회귀 분석용 가상 데이터 생성 방법.ipynb".
The first task is:
위 선형 모형은 다음과 같다.
$$
y = 100 + 79.1725 x
$$
noise 인수를 증가시키면 $\text{Var}[e]$가 증가하고 bias 인수를 증가시키면 y 절편이 증가한다.
Can you write Python code for it?

X, y, c = make_regression(n_samples=50, n_features=1, bias=100, noise=10, coef=True, random_state=0)
plt.scatter(X, y, s=100)
plt.show()


I am working on the file "15. 선형 회귀 분석/01. 회귀 분석용 가상 데이터 생성 방법.ipynb".
The first task is:
이번에는 n_features 즉, 독립 변수가 2개인 표본 데이터를 생성하여 스캐터 플롯을 그리면 다음과 같다. 종속 변수 값은 점의 명암으로 표시하였다.
Can you write Python code for it?

X, y, c = make_regression(n_samples=300, n_features=2, noise=10, coef=True, random_state=0)

plt.scatter(X[:,0], X[:,1], c=y, s=100)
plt.xlabel("x1")
plt.ylabel("x2")
plt.axis("equal")
plt.show()


I am working on the file "15. 선형 회귀 분석/02. 선형 회귀 분석의 기초.ipynb".
The first task is:
Boston Housing Price
Can you write Python code for it?

from sklearn.datasets import load_boston
boston = load_boston()
dfX_boston = pd.DataFrame(boston.data, columns=boston.feature_names)
dfy_boston = pd.DataFrame(boston.target, columns=["MEDV"])
df_boston0 = pd.concat([dfX_boston, dfy_boston], axis=1)
df_boston0.tail()

model_boston = LinearRegression().fit(boston.data, boston.target)
print(model_boston.coef_)
print(model_boston.intercept_)

predictions = model_boston.predict(boston.data)
plt.scatter(boston.target, predictions)
plt.xlabel("target")
plt.ylabel("prediction")
plt.show()

mean_abs_error = (np.abs(((boston.target - predictions)/boston.target)*100)).mean()
print("MAE: %.2f%%" % (mean_abs_error))

sk.metrics.median_absolute_error(boston.target, predictions)

sk.metrics.mean_squared_error(boston.target, predictions)


I am working on the file "15. 선형 회귀 분석/02. 선형 회귀 분석의 기초.ipynb".
The first task is:
OLS (Ordinary Least Squares)
OLS는 가장 기본적인 결정론적 회귀 방법으로 Residual Sum of Squares(RSS)를 최소화하는 가중치 벡터 값을 미분을 통해 구한다.

Residual 잔차
$$ e_i = {y}_i - x_i^T w $$

Stacking (Vector Form)
$$ e = {y} - Xw $$

Residual Sum of Squares (RSS)
$$\begin{eqnarray}
\text{RSS}
&=& \sum (y_i - \hat{y}_i)^2 \
&=& \sum e_i^2 = e^Te \
&=& (y - Xw)^T(y - Xw) \
&=& y^Ty - 2y^T X w + w^TX^TXw
\end{eqnarray}$$

Minimize using Gradient
$$ \dfrac{\partial \text{RSS}}{\partial w} = -2 X^T y + 2 X^TX w = 0 $$

$$ X^TX w = X^T y $$
$$  w = (X^TX)^{-1} X^T y $$
여기에서 그레디언트를 나타내는 다음 식을 Normal equation 이라고 한다.
$$ X^T y - X^TX w = 0 $$
Normal equation 에서 잔차에 대한 다음 특성을 알 수 있다.
$$ X^T (y - X w ) = X^T e = 0 $$
Can you write Python code for it?

y = y.reshape(len(y), 1)
w = np.dot(np.dot(np.linalg.inv(np.dot(X.T, X)), X.T), y)
print("bias:", bias)
print("coef:", coef)
print("w:\n", w)

w = np.linalg.lstsq(X, y)[0]
w

xx = np.linspace(np.min(X0) - 1, np.max(X0) + 1, 1000)
XX = np.vstack([np.ones(xx.shape[0]), xx.T]).T
yy = np.dot(XX, w)
plt.scatter(X0, y)
plt.plot(xx, yy, 'r-')
plt.show()


I am working on the file "15. 선형 회귀 분석/02. 선형 회귀 분석의 기초.ipynb".
The first task is:
RegressionResults 클래스는 분석 결과를 다양한 속성에 저장해주므로 추후 사용자가 선택하여 활용할 수 있다.
Can you write Python code for it?


I am working on the file "15. 선형 회귀 분석/02. 선형 회귀 분석의 기초.ipynb".
The first task is:
statsmodel는 다양한 회귀 분석 결과 플롯도 제공한다.

plot_fit(results, exog_idx)   Plot fit against one regressor.
abline_plot([intercept, ...]) Plots a line given an intercept and slope.
influence_plot(results[, ...])    Plot of influence in regression.
plot_leverage_resid2(results) Plots leverage statistics vs.
plot_partregress(endog, ...)  Plot partial regression for a single regressor.
plot_ccpr(results, exog_idx)  Plot CCPR against one regressor.
plot_regress_exog(results, ...)   Plot regression results against one regressor.
Can you write Python code for it?


I am working on the file "15. 선형 회귀 분석/04. 확률론적 선형 회귀 모형.ipynb".
The first task is:
가중치 중 상수항의 경우 평균은 -1.6이지만 표분 편차가 $\sqrt{4.81}=2.19$이므로 0일 가능성을 배제할 수 없다.
이 결과를 statsmodels 의 회귀 분석 보고서와 비교하자.
Can you write Python code for it?


I am working on the file "15. 선형 회귀 분석/04. 확률론적 선형 회귀 모형.ipynb".
The first task is:
다음으로 이 데이터에서 중복을 허락하여 N개의 데이터를 선택한 후 다시 회귀 분석을 한다. 이론적으로 $2^{100}$개의 경우가 있지만 1,000번만 반복해 본다.
Can you write Python code for it?

N = 1000
params_c = np.zeros(N)
params_x1 = np.zeros(N)
for i in range(N):
    idx = np.random.choice(len(dfy), len(dfy), replace=True)
    dfX2 = dfX.ix[idx, :]
    dfy2 = dfy.ix[idx]
    r = sm.OLS(dfy2, dfX2).fit()
    params_c[i] = r.params.const
    params_x1[i] = r.params.X1


I am working on the file "15. 선형 회귀 분석/04. 확률론적 선형 회귀 모형.ipynb".
The first task is:
보고서의 std err 항목을 보면 표준 편차의 경우 2.163 이고 마지막의 신뢰 구간(confidence interval)이 -5.920 ~ 2.663 임을 보이고 있다. 부트스트래핑으로 얻은 결과와 유사하다. 이 결과는 다음에 설명할 확률론적 가정에 의해 계산된 값이다.
확률론적 선형 회귀 모형
확률론적 선형 회귀 모형에서는 $y$가 확률 변수로부터 생성된 표본이라고 가정하며 다음과 같은 조건을 만족한다.

선형 정규 분포 가정
종속 변수 $y$는 기댓값 $w^Tx$, 분산 $\sigma^2$ 를 가지는 정규 분포 확률 변수이다.
   $$ p(y \mid x, \theta) = \mathcal{N}(y \mid w^Tx, \sigma^2 ) $$

따라서 오차(disturbance)  $ \epsilon = y-w^Tx $ 도 정규 분포 확률 변수이다.
   $$ p(\epsilon \mid \theta) = \mathcal{N}(0, \sigma^2 ) $$

외생성(Exogeneity) 가정

오차 $\epsilon$와 독립 변수 $x$는 서로 독립이다. 즉, $ \text{E}[\epsilon] = 0$ 은 $x$의 값에 상관없이 유지된다.
   $$ \text{E}[\epsilon \mid x] = 0$$

조건부 독립 가정

$i$번째 샘플의 오차 $\epsilon_i$와 $j$번째 샘플의 오차 $\epsilon_j$는 $x$에 대해 서로 조건부 독립이다.
   $$ \text{Cov}[\epsilon_i, \epsilon_j \mid x] =  0$$

샘플의 수가 복수이면 샘플 오차 벡터 $\epsilon$에 대해 다음과 같이 쓸 수 있다. 아래 식에서의 $\epsilon$은 샘플 수와 같은 길이의 벡터이다.
   $$  \text{Cov}[ \epsilon \epsilon^T \mid x ] = \text{Cov}[ \epsilon \epsilon^T ] = \sigma^2 I $$
MLE를 사용한 선형 회귀 분석
앞의 확률론적 선형 회귀 모형과 MLE(Maximum Likelihood Estimation)을 사용하여 가중치 벡터 $w$의 값을 구해보자.
Likelihood는 다음과 같다.
$$
\begin{eqnarray}
p(y_{1:N} \,\big|\, x_{1:N}, \theta)
&=& \prod_{i=1}^N N(y_i \,\big|\, w^T x_i , \sigma^2) \
&=& \prod_{i=1}^N \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left{-\frac{(y_i-w^T x_i)^2}{2\sigma^2} \right}  \
\end{eqnarray}
$$
계산을 용이하기 위해 Log를 취하면 다음과 같다.
$$
\begin{eqnarray}
\text{LL}
&=& \log p(y_{1:N} \,\big|\, x_{1:N}, \theta) \
&=& \log \prod_{i=1}^N \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left{-\frac{(y_i-w^T x_i)^2}{2\sigma^2} \right}  \
&=& -\dfrac{1}{2\sigma^2} \sum_{i=1}^N (y_i-w^T x_i)^2 + \dfrac{1}{2} \sum_{i=1}^N  \log{2\pi}{\sigma^2}  \
\end{eqnarray}
$$
이를 행렬로 표시하면 다음과 같다.
$$
\text{LL}  =  -C_1 (y - Xw)^T(y-Xw) + C_0 = -C_1(w^TX^TXw -2 y^TXw + y^Ty) + C_0
$$
$$
C_1 =  -\dfrac{1}{2\sigma^2}
$$
$$
C_0 =  \dfrac{1}{2} \sum_{i=1}^N  \log{2\pi}{\sigma^2}
$$
이를 최적화하면 OLS와 동일한 결과를 얻을 수 있다.
$$
\dfrac{\partial}{\partial w} \text{LL} \propto - 2X^TX \hat{w} + 2X^Ty = 0
$$
$$
\hat{w} = (X^TX)^{-1}X^T y
$$
잔차의 분포
위의 확률론적 선형 회귀 모형에...
Can you write Python code for it?

sp.stats.probplot(result.resid, plot=plt)
plt.show()

plt.plot(X0, result.resid, 'o')
plt.axhline(y=0, c='k')
plt.xlabel("X1")
plt.ylabel("Residual")
plt.show()

import statsmodels.stats.api as sms
test = sms.omni_normtest(result.resid)
for x in zip(['Chi^2', 'P-value'], test):
    print("%-12s: %6.3f" % x)

import statsmodels.stats.api as sms
test = sms.jarque_bera(result.resid)
for x in zip(['Jarque-Bera', 'P-value', 'Skew', 'Kurtosis'], test):
    print("%-12s: %6.3f" % x)


I am working on the file "15. 선형 회귀 분석/04. 확률론적 선형 회귀 모형.ipynb".
The first task is:
전체 가중치 집합을 히스토그램으로 나타내면 다음과 같다.
Can you write Python code for it?

ax1 = plt.subplot(121)
sns.distplot(params_c, ax=ax1)
plt.axvline(params_c.mean(), c='r')
plt.title("const parameter")
ax2 = plt.subplot(122)
sns.distplot(params_x1, ax=ax2)
plt.axvline(params_x1.mean(), c='r')
plt.title("x1 parameter")
plt.show()


I am working on the file "15. 선형 회귀 분석/07. 분산 분석 기반의 카테고리 분석.ipynb".
The first task is:
더미 변수와 모형 비교
더미 변수를 사용하면 사실상 회귀 모형 복수개를 동시에 사용하는 것과 실질적으로 동일하다.
더미 변수의 예 1
$$ Y = \alpha_{1} + \alpha_{2} D_2 + \alpha_{3} D_3  $$

$D_2 = 0, D_3 = 0$ 이면 $Y = \alpha_{1} $
$D_2 = 1, D_3 = 0$ 이면 $Y = \alpha_{1} + \alpha_{2} $
$D_2 = 0, D_3 = 1$ 이면 $Y = \alpha_{1} + \alpha_{3} $

더미 변수의 예 2
$$ Y = \alpha_{1} + \alpha_{2} D_2 + \alpha_{3} D_3  + \alpha_{4} X $$

$D_2 = 0, D_3 = 0$ 이면 $Y = \alpha_{1}  + \alpha_{4} X $
$D_2 = 1, D_3 = 0$ 이면 $Y = \alpha_{1} + \alpha_{2}  + \alpha_{4} X $
$D_2 = 0, D_3 = 1$ 이면 $Y = \alpha_{1} + \alpha_{3}  + \alpha_{4} X $

더미 변수의 예 3
$$ Y = \alpha_{1} + \alpha_{2} D_2 + \alpha_{3} D_3  + \alpha_{4} X + \alpha_{5} D_4 X + \alpha_{6} D_5 X  $$

$D_2 = 0, D_3 = 0, D_4 = 0, D_5 = 0$ 이면 $Y = \alpha_{1} + \alpha_{4} X $
$D_2 = 1, D_3 = 0, D_4 = 1, D_5 = 0$ 이면 $Y = \alpha_{1} + \alpha_{2}  + (\alpha_{4} + \alpha_{5}) X $
$D_2 = 0, D_3 = 1, D_4 = 0, D_5 = 1$ 이면 $Y = \alpha_{1} + \alpha_{3}  + (\alpha_{4} + \alpha_{6}) X $

더미 변수의 예 4: Boston Dataset
Can you write Python code for it?

from sklearn.datasets import load_boston
boston = load_boston()
dfX0_boston = pd.DataFrame(boston.data, columns=boston.feature_names)
dfy_boston = pd.DataFrame(boston.target, columns=["MEDV"])

import statsmodels.api as sm
dfX_boston = sm.add_constant(dfX0_boston)

df_boston = pd.concat([dfX_boston, dfy_boston], axis=1)
df_boston.tail()

dfX_boston.CHAS.plot()
dfX_boston.CHAS.unique()

model = sm.OLS(dfy_boston, dfX_boston)
result = model.fit()
print(result.summary())

params1 = result.params.drop("CHAS")
params1

params2 = params1.copy()
params2["const"] += result.params["CHAS"]
params2

df_boston.boxplot("MEDV", "CHAS")
plt.show()

sns.stripplot(x="CHAS", y="MEDV", data=df_boston, jitter=True, alpha=.3)
sns.pointplot(x="CHAS", y="MEDV", data=df_boston, dodge=True, color='r')
plt.show()


I am working on the file "15. 선형 회귀 분석/07. 분산 분석 기반의 카테고리 분석.ipynb".
The first task is:
분산 분석을 이용한 모형 비교
$K$개의 복수의 카테고리 값을 가지는 더미 변수의 영향을 보기 위해서는 F-검정을 통해 복수 개의 모형을 비교하는 분산 분석을 사용할 수 있다.
이 경우에는 분산 분석에 사용되는 각 분산의 의미가 다음과 같다.

ESS: 각 그룹 평균의 분산 (Between-Group Variance)
  $$ BSS = \sum_{k=1}^K (\bar{x} - \bar{x}_k)^2 $$

RSS: 각 그룹 내의 오차의 분산의 합 (Within-Group Variance)
  $$ WSS = \sum_{k=1}^K \sum_{i} (x_{i} - \bar{x}_k)^2 $$

TSS : 전체 오차의 분산
  $$ TSS = \sum_{i} (x_{i} - \bar{x})^2 $$

| | source  | degree of freedom | mean square | F statstics |
|-|-|-|-|-|
| Between | $$\text{BSS}$$ | $$K-1$$ | $$\dfrac{\text{ESS}}{K-1}$$   | $$F$$ |
| Within   | $$\text{WSS}$$ | $$N-K$$ | $$\dfrac{\text{RSS}}{N-K}$$ |
| Total      | $$\text{TSS}$$ | $$N-1$$ | $$\dfrac{\text{TSS}}{N-1}$$ |
| $R^2$      | $$\text{BSS} / \text{TSS}$$ |
이 때 F-검정의 귀무가설은 $\text{BSS}=0$ 즉 $\text{WSS}=\text{TSS}$ 이다. 즉, 그룹간 차이가 없는 경우이다.
Can you write Python code for it?

import statsmodels.api as sm
model = sm.OLS.from_formula("MEDV ~ C(CHAS)", data=df_boston)
result = model.fit()
table = sm.stats.anova_lm(result)
table

model1 = sm.OLS.from_formula("MEDV ~ CRIM + ZN +INDUS + NOX + RM + AGE + DIS + RAD + TAX + PTRATIO + B + LSTAT", data=df_boston)
model2 = sm.OLS.from_formula("MEDV ~ CRIM + ZN +INDUS + NOX + RM + AGE + DIS + RAD + TAX + PTRATIO + B + LSTAT + C(CHAS)", data=df_boston)
result1 = model1.fit()
result2 = model2.fit()
table = sm.stats.anova_lm(result1, result2)
table


I am working on the file "15. 선형 회귀 분석/08. patsy 패키지 소개.ipynb".
The first task is:
R-style formula
| 기호 | 설명 |
|-|-|
|+| 설명 변수 추가 |
|-| 설명 변수 제거 |
|1, 0| intercept. (제거시 사용) |
|:| interaction (곱) |
|*| a*b = a + b + a:b |
|/| a/b = a + a:b |
|~| 종속 - 독립 관계 |
Can you write Python code for it?

dmatrix("x1 - 1")

dmatrix("x1 + 0")

dmatrix("x1 + x2")

dmatrix("x1 + x2 - 1")

df = pd.DataFrame(np.array([x1, x2]).T, columns=["x1", "x2"])
df

dmatrix("x1 + x2 - 1", data=df)


I am working on the file "15. 선형 회귀 분석/08. patsy 패키지 소개.ipynb".
The first task is:
다항 선형 회귀
Can you write Python code for it?


I am working on the file "15. 선형 회귀 분석/08. patsy 패키지 소개.ipynb".
The first task is:
변수 보호 I()

다른 formula 기호로부터 보호
Can you write Python code for it?

dmatrix("x1 + x2", data=df)

dmatrix("I(x1 + x2)", data=df)


I am working on the file "15. 선형 회귀 분석/08. patsy 패키지 소개.ipynb".
The first task is:
카테고리 변수
Can you write Python code for it?


I am working on the file "1504_Seismic_petrophysics_1/Seismic_petrophysics_1.ipynb".
The first task is:
As you can see, I have all the required logs for a quantitative analysis: the elastic logs (compressional and shear velocity Vp and Vs, density RHO) as well as the main petrophysical logs (water saturation Sw, shale volume VSH and porosity PHI).
Logs stored into a Pandas DataFrame are easily accessed using the syntax DataFrame.log, e.g. L.VP is the P-wave velocity log and L.PHI the porosity log.
To select more than one log, use the syntax DataFrame.[['log1','log2']]; an alternative is DataFrame.ix[':',['log1','log2']] which means selecting all the rows (indicated by :) of log1 and log2. This second syntax allows to all kind of crazy selections and filtering like:

DataFrame.ix[(DataFrame.Z>=top) & (DataFrame.Z<=bottom)]: to make a selection on the basis of the Z log (e.g., a depth curve), i.e. to perform a selection on the depth window top-bottom;
DataFrame.ix[DataFrame.class==4,['log1','log2']]: to get log1 and log2 when class is 4 (assuming that class is a categorical variable, for example a litho-facies log like the LFC log I have created earlier);
DataFrame[pd.notnull(DataFrame.log0)].ix[:,'log1','log2']: to get log1 and log2 only when log0 is not empty.

If I want to remove a specific log (e.g., that RHO_OLD) I use the drop method with the optional axis=1 (required otherwise Pandas will search for a row named TVD instead of a column; ys, Pandas allows):
Can you write Python code for it?


I am working on the file "1504_Seismic_petrophysics_1/Seismic_petrophysics_1.ipynb".
The first task is:
But what if I need to rename multiple logs? Simply assign a new list of names to the DataFrame columns; for example, if I have a 3 logs called log1, log2, log3, I can rename them all in one go using this syntax:
DataFrame.columns=['newlog1','newlog2','newlog3']

Some other useful functions of Pandas are DataFrame.describe(), DataFrame.log..
What's great about Pandas however is that it simply is the best way I have ever seen to manage large and complex datasets;  the basic Dataframe is a compact, portable and powerful data structure that allows you to quickly inspect your data, either using standard Python functions (like min(), .max(), .mean()):
Can you write Python code for it?

print("min", logs.DEPTH.min())
print("max", logs.DEPTH.max())
print("avg", logs.VP.mean())


I am working on the file "1504_Seismic_petrophysics_1/Seismic_petrophysics_1.ipynb".
The first task is:
It is very easy to do plots of all types with pandas, and to show that here's a one-liner that plots $V_\mathrm{P}$ histograms for each class (remember that at this point we only have classes 1, 2 and 4, i.e. brine sand, oil sand and shales):
Can you write Python code for it?

logs.IP.hist(bins=50, color='black', by=logs.LFC, figsize=(15,2), layout=(1,3), lw=0)
#plt.savefig("histograms.png", dpi=250)


I am working on the file "1504_Seismic_petrophysics_1/Seismic_petrophysics_1.ipynb".
The first task is:
Let's check if I have really deleted RHO_OLD:
Can you write Python code for it?


I am working on the file "1504_Seismic_petrophysics_1/Seismic_petrophysics_1.ipynb".
The first task is:
Let's now display a summary view of all the logs:
Can you write Python code for it?

ztop=2100; zbot=2400
ll=logs.ix[(logs.DEPTH>=ztop) & (logs.DEPTH<=zbot)]

cluster=np.repeat(np.expand_dims(ll['LFC'].values,1), 100, 1)

f, ax = plt.subplots(nrows=1, ncols=4, figsize=(8, 12))
ax[0].plot(ll.VSH, ll.DEPTH, '-g', label='Vsh')
ax[0].plot(ll.SW, ll.DEPTH, '-b', label='Sw')
ax[0].plot(ll.PHI, ll.DEPTH, '-k', label='phi')
ax[1].plot(ll.IP, ll.DEPTH, '-', color='0.5')
ax[2].plot(ll.VPVS, ll.DEPTH, '-', color='0.5')
im=ax[3].imshow(cluster, interpolation='none', aspect='auto',cmap=cmap_facies,vmin=0,vmax=4)

cbar=plt.colorbar(im, ax=ax[3])
# cbar.set_label('0=undef,1=brine,2=oil,3=gas,4=shale')
# cbar.set_ticks(range(0,4+1)); cbar.set_ticklabels(range(0,4+1))
cbar.set_label((12*' ').join(['undef', 'brine', 'oil', 'gas', 'shale']))
cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

for i in range(len(ax)-1):
    ax[i].set_ylim(ztop,zbot)
    ax[i].invert_yaxis()
    ax[i].grid()
    ax[i].locator_params(axis='x', nbins=4)
ax[0].legend(fontsize='small', loc='lower right')
ax[0].set_xlabel("Vcl/phi/Sw"),    ax[0].set_xlim(-.1,1.1)
ax[1].set_xlabel("Ip [m/s*g/cc]"), ax[1].set_xlim(3000,9000)
ax[2].set_xlabel("Vp/Vs"),         ax[2].set_xlim(1.5,3)
ax[3].set_xlabel('LFC')
ax[1].set_yticklabels([]); ax[2].set_yticklabels([]); ax[3].set_yticklabels([]); ax[3].set_xticklabels([])

#plt.savefig("log.png", dpi=250)


I am working on the file "1504_Seismic_petrophysics_1/Seismic_petrophysics_1.ipynb".
The first task is:
Notice that there is no gas_sand log (occurrence of gas sands) yet, as the insitu log did not record any gas sand. We will deal with that later.
I use the above flag logs to create the LFC log and store it into the logs DataFrame:
Can you write Python code for it?

temp_lfc = np.zeros(np.shape(logs.VSH))
temp_lfc[brine_sand.values] = 1    # LFC will be 1 when ssb (brine sand flag) is True
temp_lfc[oil_sand.values] = 2      # LFC will be 2 when sso (oil sand flag) is True
temp_lfc[shale.values] = 4         # LFC will be 4 when sh (shale flag) is True
logs['LFC'] = temp_lfc             # Copy the temporary log temp_lfc into the DataFrame with name `LFC`

logs.to_csv('qsiwell2_lfc.csv',index=False) # save the data for use in Part 2


I am working on the file "1504_Seismic_petrophysics_1/Seismic_petrophysics_1.ipynb".
The first task is:
Now let's see the same data in crossplot domain that should be familiar to geophysicists, i.e. acoustic impedance (Ip) versus Vp/Vs ratio:
Can you write Python code for it?

plt.figure(figsize=(8,6))

plt.scatter(logs.IP, logs.VPVS,20, c=logs.LFC,
            marker='o', edgecolors='none',
            alpha=0.7, cmap=cmap_facies,
            vmin=0, vmax=4)
plt.xlim(3000, 9000); plt.ylim(1.5, 3);
plt.xlabel('Ip'); plt.ylabel('Vp/Vs')
plt.grid();

cbar = plt.colorbar()
cbar.set_label((15*' ').join(['undef', 'brine', 'oil', 'gas', 'shale']))
cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

#plt.savefig("crossplot.png", dpi=250)


I am working on the file "1504_Seismic_petrophysics_1/Seismic_petrophysics_1.ipynb".
The first task is:
Summary plots
First I need a custom colormap for my classes, i.e. a discrete colormap with following classes-colors association:

LFC=0: undef, GRAY
LFC=1: brine sand, BLUE
LFC=2: oil sand, GREEN
LFC=3: gas sand, RED
LFC=4: shale, BROWN

This is the way to define this colormap:
Can you write Python code for it?

import matplotlib.colors as colors
#      0=undef   1=bri  2=oil   3=gas 4=shale
ccc = ['#B3B3B3','blue','green','red','#996633',]
cmap_facies = colors.ListedColormap(ccc[0:len(ccc)], 'indexed')


I am working on the file "1504_Seismic_petrophysics_1/Seismic_petrophysics_1.ipynb".
The first task is:
To rename a log use the .rename method:
Can you write Python code for it?


I am working on the file "1504_Seismic_petrophysics_1/Seismic_petrophysics_1.ipynb".
The first task is:
and the following prints out how many samples each facies is made of:
Can you write Python code for it?

string = "brine sst={0}, oil sst={1}, shale={2}"
data = (np.count_nonzero(brine_sand),
        np.count_nonzero(oil_sand),
        np.count_nonzero(shale))
print(string.format(*data))


I am working on the file "150528 How Much of Earth is Raining at Any One Time.ipynb".
The first task is:
Calculate the ratio of the number of grid boxes with rain (above 0.01 mm / h threshold) to the number of valid grid boxes (mainly between ±60° latitudes).
Can you write Python code for it?

print(np.ma.sum(Pimerg > 0.01) / np.ma.count(Pimerg))


I am working on the file "150528 How Much of Earth is Raining at Any One Time.ipynb".
The first task is:
Read the IMERG files for one day.
Can you write Python code for it?

inpath = '%s%4d/%02d/%02d/' % (imergpath, year, 4, 1)
files = sorted(glob('%s3B-HHR*' % inpath))
nt = len(files)

with h5py.File(files[0]) as f:
    lats = f['Grid/lat'][:]
    lons = f['Grid/lon'][:]
    fillvalue = f['Grid/precipitationCal'].attrs['_FillValue']

nlon, nlat = len(lons), len(lats)

Pimerg = np.ma.masked_all([nt, nlon, nlat], dtype = np.float32)

inpath = '%s%4d/%02d/%02d/' % (imergpath, year, month, day)
files = sorted(glob('%s3B-HHR*' % inpath))

for tt in range(nt):
    with h5py.File(files[tt]) as f:
        Pimerg[tt] = np.ma.masked_less(f['Grid/precipitationCal'][:], -100.)


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Clustering
Clustering Prozess
Clustering ist Non Supervised, es werden keine Trainingslabels verwendet

Entfernung und Clusterkriterium
Die Entfernung von Dokumenten, z.B. im Vectorraummodell, ist häufig Grundlage von Clusterzuordnungen

K Means - Clustering
K Means ist ein randomisierter, Iterativer Prozess mit vorheriger Auswahl der Custerzahl
1. Wähle X Punkte als erste Clustermittelpunkte aus
2. Weise alle Punkte dem nächsten Clustermittelpunkt zu
3. Aktualisiere den Clustermittelpunkt (daher K-Means)
4. Wiederhole 2 - 4 bis die Clusterzuordnungen stabil sind
K Means - Beispiel

Programm zur Ausführung des Clusterings
Can you write Python code for it?

import sklearn.cluster
import numpy as np
from collections import Counter

def get_kmeans(num_clusters,term_matrix):
    kmeans = sklearn.cluster.MiniBatchKMeans(num_clusters)
    kmeans.fit(term_matrix)
    return kmeans

kmeans = get_kmeans(5,term_mat)
labels=kmeans.labels_
for v in Counter(labels).items():
    print "Cluster %s has %4i Elements " % v


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Darstellung der häufigsten Termine je Cluster
Can you write Python code for it?

import pandas as pd

def top_terms(num_clusters,km, vectorizer):
    order_centroids = km.cluster_centers_.argsort()[:, ::-1]
    terms = vectorizer.get_feature_names()
    acum = {}
    for i in range(num_clusters):
        for i2 in order_centroids[i, :20]:
            acum.setdefault("Cluster %s Term" % i,[]).append(terms[i2])
    return pd.DataFrame(acum)

top_terms = top_terms(5,kmeans,idf_vec)
top_terms


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Der Entscheidungsbaum

Can you write Python code for it?


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Ein Beispieldokument
Can you write Python code for it?

filename = './webkb/course/cornell/http:^^cs.cornell.edu^Info^Courses^Current^CS415^CS414.html'
from IPython.display import IFrame
IFrame(filename, width=700, height=500)


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Ende - Abschluss und Ausblick
Wichtige Punkte

Classification und Clustering für Texte sind verbreite Verfahren
In den Grundzügen einfach zu verstehen und einzusetzen
Eine gute Umsetzung ist nicht einfach
Es gibt eine Vielzahl von Wahlmöglichkeiten bei den Verfahren
Das Verständnis der Daten ist wichtig
Die Vorbereitung von Features können leicht 80% der Arbeit sein

Mehr zum Entdecken

Can you write Python code for it?

from IPython.display import display, HTML
s = """

<style>

.rendered_html {
    font-family: "proxima-nova", helvetica;
    font-size: 120%;
    line-height: 1.3;
}

.rendered_html h1 {
    margin: 0.25em 0em 0.5em;
    color: #015C9C;
    text-align: center;
    line-height: 1.2;
    page-break-before: always;
}

.rendered_html h2 {
    margin: 1.1em 0em 0.5em;
    color: #26465D;
    line-height: 1.2;
}

.rendered_html h3 {
    margin: 1.1em 0em 0.5em;
    color: #002845;
    line-height: 1.2;
}

.rendered_html li {
    line-height: 1.2;
}

.prompt {
    font-size: 110%;
}

.CodeMirror-lines {
    font-size: 110%;
}

.output_area {
    font-size: 110%;
}

h1.bigtitle {
    margin: 4cm 1cm 4cm 1cm;
    font-size: 300%;
}

h3.point {
    font-size: 200%;
    text-align: center;
    margin: 2em 0em 2em 0em;
    #26465D
}

.sbpic {
    margin: 10px 10px 10px 10px;
}

a.anchor-link {
    display: none;
}

h1.title {
    font-size: 250%;
}

.CodeMirror-code { width:100% !important; }

.container { width:100% !important; }

</style>
"""

display(HTML(s))

from IPython.html.services.config import ConfigManager
from IPython.utils.path import locate_profile
cm = ConfigManager(profile_dir=locate_profile(get_ipython().profile))
cm.update('livereveal', {
              'width': 1024,
              'height': 768,
})


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Entfernen der HTML Tags
Can you write Python code for it?

from bs4 import BeautifulSoup
txt_2 = BeautifulSoup(txt_1).get_text()
txt_2


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Exploration der Daten
Can you write Python code for it?

from collections import Counter
counter = Counter(tokens_3)
counter.most_common(10)


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Feature Extraction für Texte

Texte müssen für die hier besprochenen Verfahren in eine passende Form gebraucht werden
Dies umfasst technische und inhaltliche Vorbereitung
Entfernen von HTML Tags
Aufteilung in Terme oder N-Grams
Löschen von nicht relevanten oder störenden Inhalten
Verkleinerung des Vokabulars
Umwandeln in eine Dokument-Term Matrix
Festlegung der Termgewichte in der Matrix durch entsprechende Vor-Verarbeitung

Die ersten 1000 Zeichen des ersten Dokuments
Can you write Python code for it?

txt_1 = open(filename).read()
txt_1[:1000]


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Hierarchisches Clustering
Hierarchisches Clustering - Theorie

Auch hier ist wieder die Entfernung von Punkte zueinander entscheidend.
Es werden solange die zwei jeweils nächsten Punkte verschmolzen, bis es nur noch einen Punkt gibt.
Der Vorteil des Verfahrens ist ein schöne Darstellung der Nähe der jeweiligen Cluster

Hierarchisches Clustering - Beispiel

Hierarchisches Clustering - In der Praxis
Can you write Python code for it?

import matplotlib.pyplot as plt
from sklearn.metrics.pairwise import cosine_similarity
from scipy.cluster.hierarchy import ward, dendrogram
from sklearn.decomposition import TruncatedSVD
def draw_hier_tree_2(term_matrix,dims = 50, docs = 500):
    plt.figure(figsize=(12,10))
    tsvd = TruncatedSVD(n_components=dims)
    red_dim = tsvd.fit_transform(term_matrix[:docs])
    dist = 1 - cosine_similarity(red_dim)
    dend = dendrogram(ward(dist))
    plt.tight_layout()
    plt.show()

draw_hier_tree_2(term_mat)


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Machine Learning
Modelle und Einsatz von Modellen 1

Ein Modell ist eine Vereinfachung zu einem bestimmten Zweck
Ein Beispiel sind Karten in unterschiedlichen Maßstäben und mit unterschiedlichen Darstellungen
Jedes Modell ist ein Kompromiss
Qualität der Vorhersagen
Komplexität und Verständlichkeit
Laufzeitverhalten
Aufwand der Implementierung
Folgen von Fehlern

Modelle und Einsatz von Modellen 2

Modelle werden vorbereitet, trainiert und evaluiert
Dies geschieht in einem iterativen Prozess
Alle Schritte sind wichtig

Data Science Prozess. Angepasst aus aus Data Science for Business, O'Reilly Media, 2013

Modelle und Einsatz von Modellen 3

Problem verstehen Der wichtigste Punkt, und organisatorisch der schwierigste
Daten verstehen Was für Überraschungen gibt es in den Daten?
Daten vorbereiten In der Praxis häufig am Aufwendigsten
Modell erstellen
Auswahl der Verfahrens
Parametrisierung des Modells
Ausführungen des Trainings

Modell erproben Wie gut ist das Modell auf Trainingsdaten und auf Testdaten?
Modell einsetzen Dies kann mit einer technischen Reimplementierung einhergehen.

Features und Werte

Feature sind jene Eigenschaften von Entitäten/Documenten, mit denen Modelle arbeiten
Features entstehen erst durch Extraktion und Transformation
Intern findet eine Umwandlung in Zahlen statt, dies können wir hier ignorieren
Die Anzahl der Feature ist die Dimensionalität der Daten
Text ist hochdimensional

Besondere Eigenschaften von Texten
Beispiel für den Vortrag

Der webkb Datenbestand besteht aus Universitätswebseiten vom Ende der 90er Jahre
Ursprüngliche Anwendung war die Klassifizierung nach der Art der Seite
Person, Lehrveranstaltung, etc...

Die Daten liegen als HTML vor, insgesamt ~8000 Dokumente
Can you write Python code for it?


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Naive Bayes
Naive Bayes - Die Theorie
Die Formel sieht komplex aus:
$$ P(c|d)\propto P(c)\prod { 1\le k\le { n }{ d } }^{  }{ P({ t }_{ k }|c) } $$
Der Wesentliche Gedanke ist, dass die Klassenzugehörig von zwei Aspekten abhängt:
- Der Wahrscheinlichkeit der Klasse
- Die multiplizierte Wahrscheinlichkeit für jeden Term, dass dieser in der Klasse erscheint.
Naive Bayes - Praxis und Evaluation
Can you write Python code for it?

from sklearn.naive_bayes import MultinomialNB
from sklearn.metrics import confusion_matrix
from sklearn.cross_validation import train_test_split
splitted = train_test_split(term_mat,target_classes,test_size=0.25, random_state=42)
train_dtm, test_dtm, train_target, test_target = splitted
bayes_cls = MultinomialNB()
bayes_cls = bayes_cls.fit(train_dtm,train_target)
bayes_pred_cls = bayes_cls.predict(test_dtm)
cm = confusion_matrix(test_target, bayes_pred_cls)
classes = bayes_cls.classes_
def plot_confusion_matrix(cm, classes, cmap=plt.cm.Blues):
    plt.figure(figsize=(12,10))
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)
    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

plot_confusion_matrix(cm,classes)


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Umwandlung in Tokens
Can you write Python code for it?

import nltk
tokens_1 = nltk.word_tokenize(txt_2)
print tokens_1
print len(tokens_1)


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Validierung anhand der Wahrscheinlichkeit - ROC Kurven
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
import sklearn.naive_bayes
from sklearn import svm, datasets
from sklearn.metrics import roc_curve, auc
from sklearn.cross_validation import train_test_split
from sklearn.preprocessing import label_binarize
from sklearn.multiclass import OneVsRestClassifier

import warnings
warnings.filterwarnings('ignore')

def draw_roc(term_mat,target_classes):
    classes = np.array(['project', 'course', 'other', 'student', 'faculty', 'department', 'staff'])
    target_classes_bin = label_binarize(target_classes, classes=classes)
    splitted = train_test_split(term_mat,target_classes_bin,test_size=0.25, random_state=42)
    train_dtm, test_dtm, train_target, test_target = splitted
    classifier = OneVsRestClassifier(sklearn.naive_bayes.MultinomialNB())
    y_score = classifier.fit(train_dtm, train_target).predict_proba(test_dtm)

    plt.figure()
    plt.figure(figsize=(12,10))
    for i in range(len(classes)):
        fpr, tpr, _ = roc_curve(test_target[:, i], y_score[:, i])
        roc_auc = auc(fpr, tpr)
        plt.plot(fpr, tpr, label='%s, area:%.2f' % (classes[i],roc_auc))

    plt.plot([0, 1], [0, 1], 'k--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate')
    plt.ylabel('True Positive Rate')
    plt.legend(loc="lower right")
    plt.show()

draw_roc(term_mat,target_classes)


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Visualisierung von Clustern - Grundlagen

Die Elemente des Clusters sind Punkte in einem Raum
Die Lage dieser Punkte lässt sich in 3 Dimension gut darstellen
Wenn es mehr Dimensionen / Features gibt, müssen wir entweder kombinieren oder auswählen
Hier sind es gut 28000 Feautures

Visualisierung von Clustern - In der Praxis
Can you write Python code for it?

from sklearn.decomposition import TruncatedSVD
from itertools import cycle
import matplotlib.pyplot as plt
def draw_cluster(num_clusters,term_mat,labels):
    colors = 'rgbcmykw'
    tsvd = TruncatedSVD()
    tsvd.fit(term_mat)
    two_dim = tsvd.transform(term_mat)
    plt.figure(figsize=(12,10))
    for i in range(num_clusters):
        cluster_points = two_dim[np.where(labels == i)[0]]
        c = colors[min(i,len(colors)-1)]
        l = "Cluster %i" % i
        plt.scatter(x = cluster_points[:,1], y = cluster_points[:,0], c=c, alpha=0.5,label=l)
    plt.legend(loc=4)
    plt.show()

draw_cluster(5, term_mat, kmeans.labels_)


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Vorbereitung der Daten

Für den weiteren Vortrag gibt es einige  Standardwerte
Dies sind nicht unbedingt gute Werte, mehr Prüfung wäre notwendig

Es findet keine Filterung der Termstruktur statt
Die Artefakte sind im Weiteren zu sehen
Can you write Python code for it?

idf_vec,term_mat = prepare(filenames,max_df= 0.5, min_df = 4)
idf_vec, term_mat


I am working on the file "150627_hu_slides.ipynb".
The first task is:
Vorbereitung der weiteren Schritte
Die Klassen sind Teil des Pfads innerhalb des Datenbestands
Can you write Python code for it?

import glob
filenames = glob.glob("./webkb/*/*/http*")
print len(filenames)

print filenames[0]
print filenames[0].split("/")[2]
target_classes = [f.split("/")[2] for f in filenames]
print target_classes[:5]


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
...and put it all together using the frm function defined above:
Can you write Python code for it?

vpb, vsb, rhob, kb = frm(logs.VP, logs.VS, logs.RHO, rho_fl, k_fl, rho_b, k_b, k0, logs.PHI)
vpo, vso, rhoo, ko = frm(logs.VP, logs.VS, logs.RHO, rho_fl, k_fl, rho_o, k_o, k0, logs.PHI)
vpg, vsg, rhog, kg = frm(logs.VP, logs.VS, logs.RHO, rho_fl, k_fl, rho_g, k_g, k0, logs.PHI)


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
And this is the same summary plot that I have used above, updated to show the fluid changes in the elastic logs Ip and Vp/Vs. It is also zoomed into the reservoir between 2150 and 2200 m, and the LFC log is the original one, i.e. it reflects the insitu case.
Can you write Python code for it?

import matplotlib.colors as colors
#      0=undef   1=bri  2=oil   3=gas 4=shale
ccc = ['#B3B3B3','blue','green','red','#996633',]
cmap_facies = colors.ListedColormap(ccc[0:len(ccc)], 'indexed')

ztop = 2150; zbot = 2200
ll = logs.ix[(logs.DEPTH>=ztop) & (logs.DEPTH<=zbot)]

cluster=np.repeat(np.expand_dims(ll['LFC'].values,1),100,1)

f, ax = plt.subplots(nrows=1, ncols=4, figsize=(8, 12))
ax[0].plot(ll.VSH, ll.DEPTH, '-g', label='Vsh')
ax[0].plot(ll.SW, ll.DEPTH, '-b', label='Sw')
ax[0].plot(ll.PHI, ll.DEPTH, '-k', label='phi')
ax[1].plot(ll.IP_FRMG, ll.DEPTH, '-r')
ax[1].plot(ll.IP_FRMB, ll.DEPTH, '-b')
ax[1].plot(ll.IP, ll.DEPTH, '-', color='0.5')
ax[2].plot(ll.VPVS_FRMG, ll.DEPTH, '-r')
ax[2].plot(ll.VPVS_FRMB, ll.DEPTH, '-b')
ax[2].plot(ll.VPVS, ll.DEPTH, '-', color='0.5')
im=ax[3].imshow(cluster, interpolation='none', aspect='auto',cmap=cmap_facies,vmin=0,vmax=4)

cbar=plt.colorbar(im, ax=ax[3])
# cbar.set_label('0=undef,1=brine,2=oil,3=gas,4=shale')
# cbar.set_ticks(range(0,4+1));
cbar.set_label((12*' ').join(['undef', 'brine', 'oil', 'gas', 'shale']))
cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

for i in ax[:-1]:
    i.set_ylim(ztop,zbot)
    i.invert_yaxis()
    i.grid()
    i.locator_params(axis='x', nbins=4)
ax[0].legend(fontsize='small', loc='lower right')
ax[0].set_xlabel("Vcl/phi/Sw"),    ax[0].set_xlim(-.1,1.1)
ax[1].set_xlabel("Ip [m/s*g/cc]"), ax[1].set_xlim(3000,9000)
ax[2].set_xlabel("Vp/Vs"),         ax[2].set_xlim(1.5,3)
ax[3].set_xlabel('LFC')
ax[1].set_yticklabels([]); ax[2].set_yticklabels([]); ax[3].set_yticklabels([]); ax[3].set_xticklabels([]);


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
Finally, I will add three more LFC logs that will be companions to the new fluid-replaced logs.
The LFC log for brine-replaced logs will be always 1 whenever there's sand, because fluid replacement will have acted on all sand points and replaced whatever fluid we had originally with brine. Same thing for LFC_O and LFC_G (LFC for oil and gas-replaced logs): they will always be equal to 2 (oil) or 3 (gas) for all the sand samples. That translates into Python like:
Can you write Python code for it?

temp_lfc_b = np.zeros(np.shape(logs.VSH))
temp_lfc_b[brine_sand.values | oil_sand.values] = 1  # LFC is 1 when either brine_sand (brine sand flag) or oil_sand (oil) is True
temp_lfc_b[shale.values] = 4                # LFC 4=shale
logs['LFC_B'] = temp_lfc_b

temp_lfc_o = np.zeros(np.shape(logs.VSH))
temp_lfc_o[brine_sand.values | oil_sand.values] = 2  # LFC is now 2 when there's sand (brine_sand or oil_sand is True)
temp_lfc_o[shale.values] = 4                # LFC 4=shale
logs['LFC_O'] = temp_lfc_o

temp_lfc_g = np.zeros(np.shape(logs.VSH))
temp_lfc_g[brine_sand.values | oil_sand.values] = 3  # LFC 3=gas sand
temp_lfc_g[shale.values] = 4                # LFC 4=shale
logs['LFC_G'] = temp_lfc_g


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
I can also interrogate stat to know for example the average Ip for the litho-fluid class 2 (brine sands):
Can you write Python code for it?


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
I can use the same function to also compute the fluid bulk modulus log which is usually done via Reuss average (the lower bound k_l in the vrh function above):
First I will load the Well 2 logs with the litho-fluid class log created in Part 1, then define the various mineral and fluid elastic constants:
Can you write Python code for it?

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

logs = pd.read_csv('qsiwell2_lfc.csv')

rho_qz=2.65;  k_qz=37;  mu_qz=44    # mineral properties, quartz (i.e., sands)
rho_sh=2.81;  k_sh=15;  mu_sh=5     # mineral properties, clay (i.e., shales)
rho_b=1.09;   k_b=2.8               # fluid properties, brine
rho_o=0.78;   k_o=0.94              # fluid properties, oil
rho_g=0.25;   k_g=0.06              # fluid properties, gas


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
Let's have a look at the results in the Ip versus Vp/Vs crossplot domain; I will now plot 4 different plots to compare the initial situation to the results of the 4 fluid replacements:
Can you write Python code for it?

f, ax = plt.subplots(nrows=1, ncols=4, sharey=True, sharex=True, figsize=(16, 4))

ax[0].scatter(logs.IP,logs.VPVS,20,logs.LFC,marker='o',edgecolors='none',alpha=0.5,cmap=cmap_facies,vmin=0,vmax=4)
ax[1].scatter(logs.IP_FRMB,logs.VPVS_FRMB,20,logs.LFC_B,marker='o',edgecolors='none',alpha=0.5,cmap=cmap_facies,vmin=0,vmax=4)
ax[2].scatter(logs.IP_FRMO,logs.VPVS_FRMO,20,logs.LFC_O,marker='o',edgecolors='none',alpha=0.5,cmap=cmap_facies,vmin=0,vmax=4)
ax[3].scatter(logs.IP_FRMG,logs.VPVS_FRMG,20,logs.LFC_G,marker='o',edgecolors='none',alpha=0.5,cmap=cmap_facies,vmin=0,vmax=4)
ax[0].set_xlim(3000,9000); ax[0].set_ylim(1.5,3);
ax[0].set_title('original data');
ax[1].set_title('FRM to brine');
ax[2].set_title('FRM to oil');
ax[3].set_title('FRM to gas');
for i in ax: i.grid()


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
Obviously I need to remember that the first property is Ip, so that's why I am querying the column mean0 (mean1 holds the average values for the second property, in this case Vp/Vs).
If I were working with 3 properties, e.g. Ip, Vp/Vs and density, then the average density value for a hypothetical class 5 would be: stat.ix[stat.LFC==5,'mean2']; remember that Python works with zero-based lists and vectors so the first one has always an index of 0.
To display graphically the same information I use Pandas' scatter_matrix:
Can you write Python code for it?

i = 2
pd.scatter_matrix(ww[ww.LFC==i].drop('LFC',1),
                  color='black',
                  diagonal='kde',
                  alpha=0.1,
                  density_kwds={'color':'#000000','lw':2})
plt.suptitle('LFC=%d' % i)


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
So it's like an empty box, made of four rows (because we have 4 classes: shale, brine, oil and gas sands), and for each row we have space to store the mean values of property n.1 (Ip) and n.2 (Vp/Vs), plus their covariances and the number of samples for each class (that will inform me on the robustness of the analysis, i.e. if I have too few samples then I need to consider how my statistical analysis will not be reliable).
The following snippet shows how to populate stat and get a few plots to control if everything makes sense; also note the use of .flatten() at line 5 that linearize the covariance matrix as discussed above, so that I can save it in a series of contiguous cells along a row of stat:
Can you write Python code for it?

for i in range(1, 1+nlfc):
    temp = ww[ww.LFC==i].drop('LFC',1)
    stat.ix[(stat.LFC==i),'SAMPLES'] = temp.count()[0]
    stat.ix[stat.LFC==i,means[0]:means[-1]] = np.mean(temp.values,0)
    stat.ix[stat.LFC==i,covs[0]:covs[-1]] = np.cov(temp,rowvar=0).flatten()
    print (temp.describe().ix['mean':'std'])
    print ("LFC=%d, number of samples=%d" % (i, temp.count()[0]))


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
The syntax I use to do this is:
logs['VP_FRMB'][brine_sand|oil_sand]=vpb[brine_sand|oil_sand]

Which means, copy the values from the output of fluid replacement (vpb, vsb, rhob, etc.) only where there's sand (vpb[brine_sand|oil_sand]), i.e. only when either the flag logs brine_sand or oil_sand are True).
I also compute the additional elastic logs (acoustic and shear impedances IP, Is, and Vp/Vs ratio, VPVS) in their fluid-replaced version.
Can you write Python code for it?

logs['VP_FRMB'] = logs.VP
logs['VS_FRMB'] = logs.VS
logs['RHO_FRMB'] = logs.RHO
logs['VP_FRMB'][brine_sand|oil_sand] = vpb[brine_sand|oil_sand]
logs['VS_FRMB'][brine_sand|oil_sand] = vsb[brine_sand|oil_sand]
logs['RHO_FRMB'][brine_sand|oil_sand] = rhob[brine_sand|oil_sand]
logs['IP_FRMB'] = logs.VP_FRMB*logs.RHO_FRMB
logs['IS_FRMB'] = logs.VS_FRMB*logs.RHO_FRMB
logs['VPVS_FRMB'] = logs.VP_FRMB/logs.VS_FRMB

logs['VP_FRMO'] = logs.VP
logs['VS_FRMO'] = logs.VS
logs['RHO_FRMO'] = logs.RHO
logs['VP_FRMO'][brine_sand|oil_sand] = vpo[brine_sand|oil_sand]
logs['VS_FRMO'][brine_sand|oil_sand] = vso[brine_sand|oil_sand]
logs['RHO_FRMO'][brine_sand|oil_sand] = rhoo[brine_sand|oil_sand]
logs['IP_FRMO'] = logs.VP_FRMO*logs.RHO_FRMO
logs['IS_FRMO'] = logs.VS_FRMO*logs.RHO_FRMO
logs['VPVS_FRMO'] = logs.VP_FRMO/logs.VS_FRMO

logs['VP_FRMG'] = logs.VP
logs['VS_FRMG'] = logs.VS
logs['RHO_FRMG'] = logs.RHO
logs['VP_FRMG'][brine_sand|oil_sand] = vpg[brine_sand|oil_sand]
logs['VS_FRMG'][brine_sand|oil_sand] = vsg[brine_sand|oil_sand]
logs['RHO_FRMG'][brine_sand|oil_sand] = rhog[brine_sand|oil_sand]
logs['IP_FRMG'] = logs.VP_FRMG*logs.RHO_FRMG
logs['IS_FRMG'] = logs.VS_FRMG*logs.RHO_FRMG
logs['VPVS_FRMG'] = logs.VP_FRMG/logs.VS_FRMG


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
Then I  calculate the original (insitu) fluid density rho_fl and bulk modulus k_fl, and the average mineral bulk modulus k0:
Can you write Python code for it?

# mineral mixture bulk and shear moduli, k0 and mu0
shale = logs.VSH.values
sand = 1 - shale - logs.PHI.values
shaleN = shale / (shale+sand)  # normalized shale and sand volumes
sandN  = sand / (shale+sand)
k_u, k_l, mu_u, mu_l, k0, mu0 = vrh([shaleN, sandN], [k_sh, k_qz], [mu_sh, mu_qz])

# fluid mixture bulk modulus, using the same vrh function but capturing the Reuss average (second output)
water = logs.SW.values
hc = 1 - logs.SW.values
tmp, k_fl, tmp, tmp, tmp, tmp = vrh([water, hc], [k_b, k_o], [0, 0])

# fluid mixture density
rho_fl = water*rho_b + hc*rho_o


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
This is how the stat DataFrame looks like now:
Can you write Python code for it?


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
With the code above I simply build the headers for a Pandas DataFrame to store mean and covariances for each class.
I will now create a DataFrame which is dynamically dimensioned and made of nfacies rows, i.e. one row per each facies, and 1+n+m+1 columns, where n is the number of mean columns and m is the length of the linearized covariance matrix; for our sample case where we have only two properties this means:

1 column to store the facies number;
2 columns to store mean values for each of the four logs;
3 columns to store variance and covariance values;
1 column to store the number of samples belonging to each facies as a way to control the robustness of our statistical analysis (i.e., undersampled classes could be taken out of the study).
Can you write Python code for it?

stat = pd.DataFrame(data=None,
                    columns=['LFC']+means+covs+['SAMPLES'],
                    index=np.arange(nlfc))
stat['LFC'] = range(1, nlfc+1)


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
creation of synthetic datasets
I can now use all this information to create a brand new synthetic dataset that will replicate the average behaviour of the reservoir complex and at the same time overcome typical problems when using real data like undersampling of a certain class, presence of outliers, spurious occurrence of anomalies.
To create the synthetic datasets I use a Monte Carlo simulation relying on multivariate normal distribution to draw samples that are random but correlated in the elastic domain of choice (Ip and Vp/Vs).
Can you write Python code for it?

NN = 300

mc = pd.DataFrame(data=None,
                  columns=lognames0,
                  index=np.arange(nlfc*NN),
                  dtype='float')

for i in range(1, nlfc+1):
    mc.loc[NN*i-NN:NN*i-1, 'LFC'] = i

from numpy.random import multivariate_normal

for i in range(1, nlfc+1):
    mean = stat.loc[i-1,
                    means[0]:means[-1]].values
    sigma = np.reshape(stat.loc[i-1,
                                covs[0]:covs[-1]].values,
                                (nlogs,nlogs))
    m = multivariate_normal(mean,sigma,NN)
    mc.ix[mc.LFC==i,1:] = m


I am working on the file "1506_Seismic_petrophysics_2/Seismic_petrophysics_2.ipynb".
The first task is:
statistical analysis
generalities
After FRM I have created an augmented dataset. What I need to do now is to do a further abstraction, i.e. moving away from the intricacies and local irregularities of the real data with the final goal of creating a fully synthetic dataset representing an idealized version of a reservoir complex.
To do that I will do a statistical analysis to describe tendency, dispersion and correlation between certain elastic properties for each litho-fluid class.
Central tendency is simply described by calculating the mean values of some desired elastic property for all the existing classes; dispersion and correlation are summarised with the covariance matrix, which can be written like this (for two generic variables X and Y):
[ var_X     cov_XY ]
[ cov_XY    var_Y  ]

if I had three variables instead:
[ var_X     cov_XY    cov_XZ ]
[ cov_XY    var_Y     cov_YZ ]
[ cov_XZ    cov_YZ    var_Z  ]

Where var_X is the variance of property X, i.e. a measure of dispersion about the mean, while the covariance cov_XY is a measure of similarity between two properties X and Y. A detailed description of the covariance matrix can be found at this wikipedia entry.
Python allows me to easily perform these calculations, but what I need is a way to to store this matrix in a way that is easily accessible (which will be another Pandas DataFrame). To do this I first linearize the matrices removing the duplicates, to get something like this (example for the 2-variable scenario):
var_X  cov_XY  cov_XY var_Y

implementation in python
For the rest of this exercise I will work with the two variables used so far, i.e. Ip and Vp/Vs.
First I need to prepare a few things to make the procedure easily extendable to other situations (e.g., using more than two variables):

collect all the insitu and fluid-replaced logs together to create a megalog;
create a Pandas DataFrame to hold statistical information for all the litho-fluid classes.

Step 1 works like this:
Can you write Python code for it?

lognames0 = ['LFC','IP','VPVS']
lognames1 = ['LFC_B','IP_FRMB', 'VPVS_FRMB']
lognames2 = ['LFC_O','IP_FRMO', 'VPVS_FRMO']
lognames3 = ['LFC_G','IP_FRMG', 'VPVS_FRMG']
ww0 = logs[pd.notnull(logs.LFC)].ix[:,lognames0];
ww1 = logs[pd.notnull(logs.LFC)].ix[:,lognames1];  ww1.columns=[lognames0]
ww2 = logs[pd.notnull(logs.LFC)].ix[:,lognames2];  ww2.columns=[lognames0]
ww3 = logs[pd.notnull(logs.LFC)].ix[:,lognames3];  ww3.columns=[lognames0]
ww = pd.concat([ww0, ww1, ww2, ww3])

import itertools
list(itertools.product(['a', 'b'], ['a', 'b']))


I am working on the file "15_Transpression.ipynb".
The first task is:
Here we will examine strain evolution during transpression deformation. Transpression (Sanderson and Marchini, 1984) is considered as a wrench or transcurrent shear accompanied by horizontal shortening across, and vertical lengthening along, the shear plane.
<img width="65%" src="images/trasnpression.png">
Velocity gradient associated with transpressional deformation is defined as: $ \mathbf{L} = \begin{bmatrix} 0 & \dot{\gamma} & 0 \ 0 & -\dot{\varepsilon} & 0 \ 0 & 0 & \dot{\varepsilon} \end{bmatrix} $, where $\dot{\gamma}$ and $\dot{\varepsilon}$ are components of bulk strain rate in direction of convergence.
At first, we will define function to calculate symmetry and intensity of deformation from defoirmation gradient.
Can you write Python code for it?

def KDparams(F):
    u, s, v = svd(F)
    Rxy = s[0]/s[1]
    Ryz = s[1]/s[2]
    K = (Rxy-1)/(Ryz-1)
    D = sqrt((Rxy-1)**2 + (Ryz-1)**2)
    return K, D


I am working on the file "15_Transpression.ipynb".
The first task is:
We define 2D arrays of angles and times to be examined...
Can you write Python code for it?

times = linspace(0.00000001,10,20)
alphas = linspace(0,90,20)
time, alpha = meshgrid(times, alphas)
K = zeros_like(alpha)
D = zeros_like(alpha)


I am working on the file "15_Transpression.ipynb".
The first task is:
and l;oop over to calculate symmetry and intensity for each combination
Can you write Python code for it?

for (r,c) in np.ndindex(alpha.shape):
    a = deg2rad(alpha[r,c])
    t = time[r,c]*1e6*yearsec
    edot = sr*sin(a)
    gdot = sr*cos(a)
    L = array([[0, gdot, 0], [0, -edot, 0],[0, 0, edot]])
    F = la.expm(L*t)
    K[r,c], D[r,c] = KDparams(F)


I am working on the file "15_autoencoders.ipynb".
The first task is:
A couple utility functions to plot grayscale 28x28 image:
Can you write Python code for it?

def plot_image(image, shape=[28, 28]):
    plt.imshow(image.reshape(shape), cmap="Greys", interpolation="nearest")
    plt.axis("off")

def plot_multiple_images(images, n_rows, n_cols, pad=2):
    images = images - images.min()  # make the minimum == 0, so the padding looks white
    w,h = images.shape[1:]
    image = np.zeros(((w+pad)*n_rows+pad, (h+pad)*n_cols+pad))
    for y in range(n_rows):
        for x in range(n_cols):
            image[(y*(h+pad)+pad):(y*(h+pad)+pad+h),(x*(w+pad)+pad):(x*(w+pad)+pad+w)] = images[y*n_cols+x]
    plt.imshow(image, cmap="Greys", interpolation="nearest")
    plt.axis("off")


I am working on the file "15_autoencoders.ipynb".
The first task is:
Cache the frozen layer outputs
Can you write Python code for it?

training_ops = [phase1_training_op, phase2_training_op]
reconstruction_losses = [phase1_reconstruction_loss, phase2_reconstruction_loss]
n_epochs = [4, 4]
batch_sizes = [150, 150]

with tf.Session() as sess:
    init.run()
    for phase in range(2):
        print("Training phase #{}".format(phase + 1))
        if phase == 1:
            hidden1_cache = hidden1.eval(feed_dict={X: mnist.train.images})
        for epoch in range(n_epochs[phase]):
            n_batches = mnist.train.num_examples // batch_sizes[phase]
            for iteration in range(n_batches):
                print("\r{}%".format(100 * iteration // n_batches), end="")
                sys.stdout.flush()
                if phase == 1:
                    indices = rnd.permutation(mnist.train.num_examples)
                    hidden1_batch = hidden1_cache[indices[:batch_sizes[phase]]]
                    feed_dict = {hidden1: hidden1_batch}
                    sess.run(training_ops[phase], feed_dict=feed_dict)
                else:
                    X_batch, y_batch = mnist.train.next_batch(batch_sizes[phase])
                    feed_dict = {X: X_batch}
                    sess.run(training_ops[phase], feed_dict=feed_dict)
            loss_train = reconstruction_losses[phase].eval(feed_dict=feed_dict)
            print("\r{}".format(epoch), "Train MSE:", loss_train)
            saver.save(sess, "./my_model_cache_frozen.ckpt")
    loss_test = reconstruction_loss.eval(feed_dict={X: mnist.test.images})
    print("Test MSE:", loss_test)


I am working on the file "15_autoencoders.ipynb".
The first task is:
Decode:
Can you write Python code for it?

with tf.Session() as sess:
    saver.restore(sess, "./my_model_variational.ckpt")
    outputs_val = outputs.eval(feed_dict={codings: codings_val})


I am working on the file "15_autoencoders.ipynb".
The first task is:
Encode & Decode
Encode:
Can you write Python code for it?

n_digits = 3
X_test, y_test = mnist.test.next_batch(batch_size)
codings = hidden3

with tf.Session() as sess:
    saver.restore(sess, "./my_model_variational.ckpt")
    codings_val = codings.eval(feed_dict={X: X_test})


I am working on the file "15_autoencoders.ipynb".
The first task is:
Finally, we can create a Stacked Autoencoder by simply reusing the weights and biases from the Autoencoders we just trained:
Can you write Python code for it?

reset_graph()

n_inputs = 28*28

X = tf.placeholder(tf.float32, shape=[None, n_inputs])
hidden1 = tf.nn.elu(tf.matmul(X, W1) + b1)
hidden2 = tf.nn.elu(tf.matmul(hidden1, W2) + b2)
hidden3 = tf.nn.elu(tf.matmul(hidden2, W3) + b3)
outputs = tf.matmul(hidden3, W4) + b4

show_reconstructed_digits(X, outputs)


I am working on the file "15_autoencoders.ipynb".
The first task is:
Generate digits
Let's train the model and generate a few random digits:
Can you write Python code for it?

import numpy as np

n_digits = 60
n_epochs = 50
batch_size = 150

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="") # not shown in the book
            sys.stdout.flush()                                          # not shown
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch})
        loss_val, reconstruction_loss_val, latent_loss_val = sess.run([loss, reconstruction_loss, latent_loss], feed_dict={X: X_batch}) # not shown
        print("\r{}".format(epoch), "Train total loss:", loss_val, "\tReconstruction loss:", reconstruction_loss_val, "\tLatent loss:", latent_loss_val)  # not shown
        saver.save(sess, "./my_model_variational.ckpt")  # not shown

    codings_rnd = np.random.normal(size=[n_digits, n_hidden3])
    outputs_val = outputs.eval(feed_dict={hidden3: codings_rnd})

plt.figure(figsize=(8,50)) # not shown in the book
for iteration in range(n_digits):
    plt.subplot(n_digits, 10, iteration + 1)
    plot_image(outputs_val[iteration])

n_rows = 6
n_cols = 10
plot_multiple_images(outputs_val.reshape(-1, 28, 28), n_rows, n_cols)
save_fig("generated_digits_plot")
plt.show()


I am working on the file "15_autoencoders.ipynb".
The first task is:
Let's plot the reconstructions:
Can you write Python code for it?

fig = plt.figure(figsize=(8, 2.5 * n_digits))
for iteration in range(n_digits):
    plt.subplot(n_digits, 2, 1 + 2 * iteration)
    plot_image(X_test[iteration])
    plt.subplot(n_digits, 2, 2 + 2 * iteration)
    plot_image(outputs_val[iteration])


I am working on the file "15_autoencoders.ipynb".
The first task is:
Normalize the data:
Can you write Python code for it?

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train = scaler.fit_transform(data[:100])
X_test = scaler.transform(data[100:])


I am working on the file "15_autoencoders.ipynb".
The first task is:
Note that the latent loss is computed differently in this second variant:
Can you write Python code for it?

latent_loss = 0.5 * tf.reduce_sum(
    tf.exp(hidden3_gamma) + tf.square(hidden3_mean) - 1 - hidden3_gamma)


I am working on the file "15_autoencoders.ipynb".
The first task is:
Now let's train it! Note that we don't feed target values (y_batch is not used). This is unsupervised training.
Can you write Python code for it?

n_epochs = 5
batch_size = 150

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="") # not shown in the book
            sys.stdout.flush()                                          # not shown
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch})
        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})   # not shown
        print("\r{}".format(epoch), "Train MSE:", loss_train)           # not shown
        saver.save(sess, "./my_model_all_layers.ckpt")                  # not shown


I am working on the file "15_autoencoders.ipynb".
The first task is:
Now let's train two Autoencoders. The first one is trained on the training data, and the second is trained on the previous Autoencoder's hidden layer output:
Can you write Python code for it?

hidden_output, W1, b1, W4, b4 = train_autoencoder(mnist.train.images, n_neurons=300, n_epochs=4, batch_size=150,
                                                  output_activation=None)
_, W2, b2, W3, b3 = train_autoencoder(hidden_output, n_neurons=150, n_epochs=4, batch_size=150)


I am working on the file "15_autoencoders.ipynb".
The first task is:
Now reusing the first two layers of the autoencoder we pretrained:
Can you write Python code for it?

n_epochs = 4
batch_size = 150
n_labeled_instances = 20000

#training_op = optimizer.minimize(loss, var_list=[weights3, biases3])  # Freeze layers 1 and 2 (optional)

with tf.Session() as sess:
    init.run()
    pretrain_saver.restore(sess, "./my_model_cache_frozen.ckpt")
    for epoch in range(n_epochs):
        n_batches = n_labeled_instances // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="")
            sys.stdout.flush()
            indices = rnd.permutation(n_labeled_instances)[:batch_size]
            X_batch, y_batch = mnist.train.images[indices], mnist.train.labels[indices]
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
        accuracy_val = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
        print("\r{}".format(epoch), "Train accuracy:", accuracy_val, end="\t")
        saver.save(sess, "./my_model_supervised_pretrained.ckpt")
        accuracy_val = accuracy.eval(feed_dict={X: mnist.test.images, y: mnist.test.labels})
        print("Test accuracy:", accuracy_val)


I am working on the file "15_autoencoders.ipynb".
The first task is:
PCA with a linear Autoencoder
Build 3D dataset:
Can you write Python code for it?

import numpy.random as rnd

rnd.seed(4)
m = 200
w1, w2 = 0.1, 0.3
noise = 0.1

angles = rnd.rand(m) * 3 * np.pi / 2 - 0.5
data = np.empty((m, 3))
data[:, 0] = np.cos(angles) + np.sin(angles)/2 + noise * rnd.randn(m) / 2
data[:, 1] = np.sin(angles) * 0.7 + noise * rnd.randn(m) / 2
data[:, 2] = data[:, 0] * w1 + data[:, 1] * w2 + noise * rnd.randn(m)


rnd.seed(4)
m = 200
w1, w2 = 0.1, 0.3
noise = 0.1

angles = rnd.rand(m) * 3 * np.pi / 2 - 0.5
data = np.empty((m, 3))
data[:, 0] = np.cos(angles) + np.sin(angles)/2 + noise * rnd.randn(m) / 2
data[:, 1] = np.sin(angles) * 0.7 + noise * rnd.randn(m) / 2
data[:, 2] = data[:, 0] * w1 + data[:, 1] * w2 + noise * rnd.randn(m)


I am working on the file "15_autoencoders.ipynb".
The first task is:
Regular training (without pretraining):
Can you write Python code for it?

n_epochs = 4
batch_size = 150
n_labeled_instances = 20000

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = n_labeled_instances // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="")
            sys.stdout.flush()
            indices = rnd.permutation(n_labeled_instances)[:batch_size]
            X_batch, y_batch = mnist.train.images[indices], mnist.train.labels[indices]
            sess.run(training_op, feed_dict={X: X_batch, y: y_batch})
        accuracy_val = accuracy.eval(feed_dict={X: X_batch, y: y_batch})
        print("\r{}".format(epoch), "Train accuracy:", accuracy_val, end=" ")
        saver.save(sess, "./my_model_supervised.ckpt")
        accuracy_val = accuracy.eval(feed_dict={X: mnist.test.images, y: mnist.test.labels})
        print("Test accuracy:", accuracy_val)


I am working on the file "15_autoencoders.ipynb".
The first task is:
Sparse Autoencoder
Can you write Python code for it?

p = 0.1
q = np.linspace(0.001, 0.999, 500)
kl_div = p * np.log(p / q) + (1 - p) * np.log((1 - p) / (1 - q))
mse = (p - q)**2
plt.plot([p, p], [0, 0.3], "k:")
plt.text(0.05, 0.32, "Target\nsparsity", fontsize=14)
plt.plot(q, kl_div, "b-", label="KL divergence")
plt.plot(q, mse, "r--", label="MSE")
plt.legend(loc="upper left")
plt.xlabel("Actual sparsity")
plt.ylabel("Cost", rotation=0)
plt.axis([0, 1, 0, 0.95])
save_fig("sparsity_loss_plot")

reset_graph()

n_inputs = 28 * 28
n_hidden1 = 1000  # sparse codings
n_outputs = n_inputs

def kl_divergence(p, q):
    # Kullback Leibler divergence
    return p * tf.log(p / q) + (1 - p) * tf.log((1 - p) / (1 - q))

learning_rate = 0.01
sparsity_target = 0.1
sparsity_weight = 0.2

X = tf.placeholder(tf.float32, shape=[None, n_inputs])            # not shown in the book

hidden1 = tf.layers.dense(X, n_hidden1, activation=tf.nn.sigmoid) # not shown
outputs = tf.layers.dense(hidden1, n_outputs)                     # not shown

hidden1_mean = tf.reduce_mean(hidden1, axis=0) # batch mean
sparsity_loss = tf.reduce_sum(kl_divergence(sparsity_target, hidden1_mean))
reconstruction_loss = tf.reduce_mean(tf.square(outputs - X)) # MSE
loss = reconstruction_loss + sparsity_weight * sparsity_loss

optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(loss)

init = tf.global_variables_initializer()
saver = tf.train.Saver()

n_epochs = 100
batch_size = 1000

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="")
            sys.stdout.flush()
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch})
        reconstruction_loss_val, sparsity_loss_val, loss_val = sess.run([reconstruction_loss, sparsity_loss, loss], feed_dict={X: X_batch})
        print("\r{}".format(epoch), "Train MSE:",...


I am working on the file "15_autoencoders.ipynb".
The first task is:
Stacked Autoencoders
Let's use MNIST:
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets("/tmp/data/")


I am working on the file "15_autoencoders.ipynb".
The first task is:
Stacked denoising Autoencoder
Note: the book uses tf.contrib.layers.dropout() rather than tf.layers.dropout() (which did not exist when this chapter was written). It is now preferable to use tf.layers.dropout(), because anything in the contrib module may change or be deleted without notice. The tf.layers.dropout() function is almost identical to the tf.contrib.layers.dropout() function, except for a few minor differences. Most importantly:
* you must specify the dropout rate (rate) rather than the keep probability (keep_prob), where rate is simply equal to 1 - keep_prob,
* the is_training parameter is renamed to training.
Using Gaussian noise:
Can you write Python code for it?

reset_graph()

n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 150  # codings
n_hidden3 = n_hidden1
n_outputs = n_inputs

learning_rate = 0.01

noise_level = 1.0

X = tf.placeholder(tf.float32, shape=[None, n_inputs])
X_noisy = X + noise_level * tf.random_normal(tf.shape(X))

hidden1 = tf.layers.dense(X_noisy, n_hidden1, activation=tf.nn.relu,
                          name="hidden1")
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, # not shown in the book
                          name="hidden2")                            # not shown
hidden3 = tf.layers.dense(hidden2, n_hidden3, activation=tf.nn.relu, # not shown
                          name="hidden3")                            # not shown
outputs = tf.layers.dense(hidden3, n_outputs, name="outputs")        # not shown

reconstruction_loss = tf.reduce_mean(tf.square(outputs - X)) # MSE

optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(reconstruction_loss)

init = tf.global_variables_initializer()
saver = tf.train.Saver()

n_epochs = 10
batch_size = 150

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="")
            sys.stdout.flush()
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch})
        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})
        print("\r{}".format(epoch), "Train MSE:", loss_train)
        saver.save(sess, "./my_model_stacked_denoising_gaussian.ckpt")


I am working on the file "15_autoencoders.ipynb".
The first task is:
This function loads the model, evaluates it on the test set (it measures the reconstruction error), then it displays the original image and its reconstruction:
Can you write Python code for it?

def show_reconstructed_digits(X, outputs, model_path = None, n_test_digits = 2):
    with tf.Session() as sess:
        if model_path:
            saver.restore(sess, model_path)
        X_test = mnist.test.images[:n_test_digits]
        outputs_val = outputs.eval(feed_dict={X: X_test})

    fig = plt.figure(figsize=(8, 3 * n_test_digits))
    for digit_index in range(n_test_digits):
        plt.subplot(n_test_digits, 2, digit_index * 2 + 1)
        plot_image(X_test[digit_index])
        plt.subplot(n_test_digits, 2, digit_index * 2 + 2)
        plot_image(outputs_val[digit_index])

show_reconstructed_digits(X, outputs, "./my_model_all_layers.ckpt")
save_fig("reconstruction_plot")


I am working on the file "15_autoencoders.ipynb".
The first task is:
To speed up training, you can normalize the inputs between 0 and 1, and use the cross entropy instead of the MSE for the cost function:
Can you write Python code for it?

logits = tf.layers.dense(hidden1, n_outputs)
outputs = tf.nn.sigmoid(logits)

xentropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=X, logits=logits)
reconstruction_loss = tf.reduce_mean(xentropy)


I am working on the file "15_autoencoders.ipynb".
The first task is:
Train all layers at once
Let's build a stacked Autoencoder with 3 hidden layers and 1 output layer (ie. 2 stacked Autoencoders). We will use ELU activation, He initialization and L2 regularization.
Note: since the tf.layers.dense() function is incompatible with tf.contrib.layers.arg_scope() (which is used in the book), we now use python's functools.partial() function instead. It makes it easy to create a my_dense_layer() function that just calls tf.layers.dense() with the desired parameters automatically set (unless they are overridden when calling my_dense_layer()).
Can you write Python code for it?

reset_graph()

from functools import partial

n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 150  # codings
n_hidden3 = n_hidden1
n_outputs = n_inputs

learning_rate = 0.01
l2_reg = 0.0001

X = tf.placeholder(tf.float32, shape=[None, n_inputs])

he_init = tf.contrib.layers.variance_scaling_initializer() # He initialization
#Equivalent to:
#he_init = lambda shape, dtype=tf.float32: tf.truncated_normal(shape, 0., stddev=np.sqrt(2/shape[0]))
l2_regularizer = tf.contrib.layers.l2_regularizer(l2_reg)
my_dense_layer = partial(tf.layers.dense,
                         activation=tf.nn.elu,
                         kernel_initializer=he_init,
                         kernel_regularizer=l2_regularizer)

hidden1 = my_dense_layer(X, n_hidden1)
hidden2 = my_dense_layer(hidden1, n_hidden2)
hidden3 = my_dense_layer(hidden2, n_hidden3)
outputs = my_dense_layer(hidden3, n_outputs, activation=None)

reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))

reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
loss = tf.add_n([reconstruction_loss] + reg_losses)

optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(loss)

init = tf.global_variables_initializer()
saver = tf.train.Saver() # not shown in the book


I am working on the file "15_autoencoders.ipynb".
The first task is:
Training one Autoencoder at a time in a single graph
Another approach is to use a single graph. To do this, we create the graph for the full Stacked Autoencoder, but then we also add operations to train each Autoencoder independently: phase 1 trains the bottom and top layer (ie. the first Autoencoder) and phase 2 trains the two middle layers (ie. the second Autoencoder).
Can you write Python code for it?

reset_graph()

n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 150  # codings
n_hidden3 = n_hidden1
n_outputs = n_inputs

learning_rate = 0.01
l2_reg = 0.0001

activation = tf.nn.elu
regularizer = tf.contrib.layers.l2_regularizer(l2_reg)
initializer = tf.contrib.layers.variance_scaling_initializer()

X = tf.placeholder(tf.float32, shape=[None, n_inputs])

weights1_init = initializer([n_inputs, n_hidden1])
weights2_init = initializer([n_hidden1, n_hidden2])
weights3_init = initializer([n_hidden2, n_hidden3])
weights4_init = initializer([n_hidden3, n_outputs])

weights1 = tf.Variable(weights1_init, dtype=tf.float32, name="weights1")
weights2 = tf.Variable(weights2_init, dtype=tf.float32, name="weights2")
weights3 = tf.Variable(weights3_init, dtype=tf.float32, name="weights3")
weights4 = tf.Variable(weights4_init, dtype=tf.float32, name="weights4")

biases1 = tf.Variable(tf.zeros(n_hidden1), name="biases1")
biases2 = tf.Variable(tf.zeros(n_hidden2), name="biases2")
biases3 = tf.Variable(tf.zeros(n_hidden3), name="biases3")
biases4 = tf.Variable(tf.zeros(n_outputs), name="biases4")

hidden1 = activation(tf.matmul(X, weights1) + biases1)
hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)
hidden3 = activation(tf.matmul(hidden2, weights3) + biases3)
outputs = tf.matmul(hidden3, weights4) + biases4

reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))

optimizer = tf.train.AdamOptimizer(learning_rate)

with tf.name_scope("phase1"):
    phase1_outputs = tf.matmul(hidden1, weights4) + biases4  # bypass hidden2 and hidden3
    phase1_reconstruction_loss = tf.reduce_mean(tf.square(phase1_outputs - X))
    phase1_reg_loss = regularizer(weights1) + regularizer(weights4)
    phase1_loss = phase1_reconstruction_loss + phase1_reg_loss
    phase1_training_op = optimizer.minimize(phase1_loss)

with tf.name_scope("phase2"):
    phase2_reconstruction_loss = tf.reduce_mean(tf.square(hidden3 - hidden1))
    phase2_reg_loss = regularizer(weights2) + regularizer(weights3)
    phase2_loss = phase2_reconstruction_loss...


I am working on the file "15_autoencoders.ipynb".
The first task is:
Training one Autoencoder at a time in multiple graphs
There are many ways to train one Autoencoder at a time. The first approach is to train each Autoencoder using a different graph, then we create the Stacked Autoencoder by simply initializing it with the weights and biases copied from these Autoencoders.
Let's create a function that will train one autoencoder and return the transformed training set (i.e., the output of the hidden layer) and the model parameters.
Can you write Python code for it?

reset_graph()

from functools import partial

def train_autoencoder(X_train, n_neurons, n_epochs, batch_size,
                      learning_rate = 0.01, l2_reg = 0.0005, seed=42,
                      hidden_activation=tf.nn.elu,
                      output_activation=tf.nn.elu):
    graph = tf.Graph()
    with graph.as_default():
        tf.set_random_seed(seed)

        n_inputs = X_train.shape[1]

        X = tf.placeholder(tf.float32, shape=[None, n_inputs])

        my_dense_layer = partial(
            tf.layers.dense,
            kernel_initializer=tf.contrib.layers.variance_scaling_initializer(),
            kernel_regularizer=tf.contrib.layers.l2_regularizer(l2_reg))

        hidden = my_dense_layer(X, n_neurons, activation=hidden_activation, name="hidden")
        outputs = my_dense_layer(hidden, n_inputs, activation=output_activation, name="outputs")

        reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))

        reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
        loss = tf.add_n([reconstruction_loss] + reg_losses)

        optimizer = tf.train.AdamOptimizer(learning_rate)
        training_op = optimizer.minimize(loss)

        init = tf.global_variables_initializer()

    with tf.Session(graph=graph) as sess:
        init.run()
        for epoch in range(n_epochs):
            n_batches = len(X_train) // batch_size
            for iteration in range(n_batches):
                print("\r{}%".format(100 * iteration // n_batches), end="")
                sys.stdout.flush()
                indices = rnd.permutation(len(X_train))[:batch_size]
                X_batch = X_train[indices]
                sess.run(training_op, feed_dict={X: X_batch})
            loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})
            print("\r{}".format(epoch), "Train MSE:", loss_train)
        params = dict([(var.name, var.eval()) for var in tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)])
        hidden_val = hidden.eval(feed_dict={X: X_train})
       ...


I am working on the file "15_autoencoders.ipynb".
The first task is:
Training one Autoencoder at a time in multiple graphs
There are many ways to train one Autoencoder at a time. The first approach it to train each Autoencoder using a different graph, then we create the Stacked Autoencoder by simply initializing it with the weights and biases copied from these Autoencoders.
Let's create a function that will train one autoencoder and return the transformed training set (i.e., the output of the hidden layer) and the model parameters.
Can you write Python code for it?

reset_graph()

from functools import partial

def train_autoencoder(X_train, n_neurons, n_epochs, batch_size,
                      learning_rate = 0.01, l2_reg = 0.0005,
                      activation=tf.nn.elu, seed=42):
    graph = tf.Graph()
    with graph.as_default():
        tf.set_random_seed(seed)

        n_inputs = X_train.shape[1]

        X = tf.placeholder(tf.float32, shape=[None, n_inputs])

        my_dense_layer = partial(
            tf.layers.dense,
            activation=activation,
            kernel_initializer=tf.contrib.layers.variance_scaling_initializer(),
            kernel_regularizer=tf.contrib.layers.l2_regularizer(l2_reg))

        hidden = my_dense_layer(X, n_neurons, name="hidden")
        outputs = my_dense_layer(hidden, n_inputs, activation=None, name="outputs")

        reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))

        reg_losses = tf.get_collection(tf.GraphKeys.REGULARIZATION_LOSSES)
        loss = tf.add_n([reconstruction_loss] + reg_losses)

        optimizer = tf.train.AdamOptimizer(learning_rate)
        training_op = optimizer.minimize(loss)

        init = tf.global_variables_initializer()

    with tf.Session(graph=graph) as sess:
        init.run()
        for epoch in range(n_epochs):
            n_batches = len(X_train) // batch_size
            for iteration in range(n_batches):
                print("\r{}%".format(100 * iteration // n_batches), end="")
                sys.stdout.flush()
                indices = rnd.permutation(len(X_train))[:batch_size]
                X_batch = X_train[indices]
                sess.run(training_op, feed_dict={X: X_batch})
            loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})
            print("\r{}".format(epoch), "Train MSE:", loss_train)
        params = dict([(var.name, var.eval()) for var in tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)])
        hidden_val = hidden.eval(feed_dict={X: X_train})
        return hidden_val, params["hidden/kernel:0"],...


I am working on the file "15_autoencoders.ipynb".
The first task is:
Tying weights
It is common to tie the weights of the encoder and the decoder (weights_decoder = tf.transpose(weights_encoder)). Unfortunately this makes it impossible (or very tricky) to use the tf.layers.dense() function, so we need to build the Autoencoder manually:
Can you write Python code for it?

reset_graph()

n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 150  # codings
n_hidden3 = n_hidden1
n_outputs = n_inputs

learning_rate = 0.01
l2_reg = 0.0005

activation = tf.nn.elu
regularizer = tf.contrib.layers.l2_regularizer(l2_reg)
initializer = tf.contrib.layers.variance_scaling_initializer()

X = tf.placeholder(tf.float32, shape=[None, n_inputs])

weights1_init = initializer([n_inputs, n_hidden1])
weights2_init = initializer([n_hidden1, n_hidden2])

weights1 = tf.Variable(weights1_init, dtype=tf.float32, name="weights1")
weights2 = tf.Variable(weights2_init, dtype=tf.float32, name="weights2")
weights3 = tf.transpose(weights2, name="weights3")  # tied weights
weights4 = tf.transpose(weights1, name="weights4")  # tied weights

biases1 = tf.Variable(tf.zeros(n_hidden1), name="biases1")
biases2 = tf.Variable(tf.zeros(n_hidden2), name="biases2")
biases3 = tf.Variable(tf.zeros(n_hidden3), name="biases3")
biases4 = tf.Variable(tf.zeros(n_outputs), name="biases4")

hidden1 = activation(tf.matmul(X, weights1) + biases1)
hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)
hidden3 = activation(tf.matmul(hidden2, weights3) + biases3)
outputs = tf.matmul(hidden3, weights4) + biases4

reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))
reg_loss = regularizer(weights1) + regularizer(weights2)
loss = reconstruction_loss + reg_loss

optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(loss)

init = tf.global_variables_initializer()

saver = tf.train.Saver()

n_epochs = 5
batch_size = 150

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="")
            sys.stdout.flush()
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch})
        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})
       ...


I am working on the file "15_autoencoders.ipynb".
The first task is:
Unsupervised pretraining
Let's create a small neural network for MNIST classification:
Can you write Python code for it?

reset_graph()

n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 150
n_outputs = 10

learning_rate = 0.01
l2_reg = 0.0005

activation = tf.nn.elu
regularizer = tf.contrib.layers.l2_regularizer(l2_reg)
initializer = tf.contrib.layers.variance_scaling_initializer()

X = tf.placeholder(tf.float32, shape=[None, n_inputs])
y = tf.placeholder(tf.int32, shape=[None])

weights1_init = initializer([n_inputs, n_hidden1])
weights2_init = initializer([n_hidden1, n_hidden2])
weights3_init = initializer([n_hidden2, n_hidden3])

weights1 = tf.Variable(weights1_init, dtype=tf.float32, name="weights1")
weights2 = tf.Variable(weights2_init, dtype=tf.float32, name="weights2")
weights3 = tf.Variable(weights3_init, dtype=tf.float32, name="weights3")

biases1 = tf.Variable(tf.zeros(n_hidden1), name="biases1")
biases2 = tf.Variable(tf.zeros(n_hidden2), name="biases2")
biases3 = tf.Variable(tf.zeros(n_hidden3), name="biases3")

hidden1 = activation(tf.matmul(X, weights1) + biases1)
hidden2 = activation(tf.matmul(hidden1, weights2) + biases2)
logits = tf.matmul(hidden2, weights3) + biases3

cross_entropy = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=logits)
reg_loss = regularizer(weights1) + regularizer(weights2) + regularizer(weights3)
loss = cross_entropy + reg_loss
optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(loss)

correct = tf.nn.in_top_k(logits, y, 1)
accuracy = tf.reduce_mean(tf.cast(correct, tf.float32))

init = tf.global_variables_initializer()
pretrain_saver = tf.train.Saver([weights1, weights2, biases1, biases2])
saver = tf.train.Saver()


I am working on the file "15_autoencoders.ipynb".
The first task is:
Using dropout:
Can you write Python code for it?

reset_graph()

n_inputs = 28 * 28
n_hidden1 = 300
n_hidden2 = 150  # codings
n_hidden3 = n_hidden1
n_outputs = n_inputs

learning_rate = 0.01

dropout_rate = 0.3

training = tf.placeholder_with_default(False, shape=(), name='training')

X = tf.placeholder(tf.float32, shape=[None, n_inputs])
X_drop = tf.layers.dropout(X, dropout_rate, training=training)

hidden1 = tf.layers.dense(X_drop, n_hidden1, activation=tf.nn.relu,
                          name="hidden1")
hidden2 = tf.layers.dense(hidden1, n_hidden2, activation=tf.nn.relu, # not shown in the book
                          name="hidden2")                            # not shown
hidden3 = tf.layers.dense(hidden2, n_hidden3, activation=tf.nn.relu, # not shown
                          name="hidden3")                            # not shown
outputs = tf.layers.dense(hidden3, n_outputs, name="outputs")        # not shown

reconstruction_loss = tf.reduce_mean(tf.square(outputs - X)) # MSE

optimizer = tf.train.AdamOptimizer(learning_rate)
training_op = optimizer.minimize(reconstruction_loss)

init = tf.global_variables_initializer()
saver = tf.train.Saver()

n_epochs = 10
batch_size = 150

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="")
            sys.stdout.flush()
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch, training: True})
        loss_train = reconstruction_loss.eval(feed_dict={X: X_batch})
        print("\r{}".format(epoch), "Train MSE:", loss_train)
        saver.save(sess, "./my_model_stacked_denoising_dropout.ckpt")

show_reconstructed_digits(X, outputs, "./my_model_stacked_denoising_dropout.ckpt")


I am working on the file "15_autoencoders.ipynb".
The first task is:
Variational Autoencoder
Can you write Python code for it?

reset_graph()

from functools import partial

n_inputs = 28 * 28
n_hidden1 = 500
n_hidden2 = 500
n_hidden3 = 20  # codings
n_hidden4 = n_hidden2
n_hidden5 = n_hidden1
n_outputs = n_inputs
learning_rate = 0.001

initializer = tf.contrib.layers.variance_scaling_initializer()

my_dense_layer = partial(
    tf.layers.dense,
    activation=tf.nn.elu,
    kernel_initializer=initializer)

X = tf.placeholder(tf.float32, [None, n_inputs])
hidden1 = my_dense_layer(X, n_hidden1)
hidden2 = my_dense_layer(hidden1, n_hidden2)
hidden3_mean = my_dense_layer(hidden2, n_hidden3, activation=None)
hidden3_sigma = my_dense_layer(hidden2, n_hidden3, activation=None)
noise = tf.random_normal(tf.shape(hidden3_sigma), dtype=tf.float32)
hidden3 = hidden3_mean + hidden3_sigma * noise
hidden4 = my_dense_layer(hidden3, n_hidden4)
hidden5 = my_dense_layer(hidden4, n_hidden5)
logits = my_dense_layer(hidden5, n_outputs, activation=None)
outputs = tf.sigmoid(logits)

xentropy = tf.nn.sigmoid_cross_entropy_with_logits(labels=X, logits=logits)
reconstruction_loss = tf.reduce_sum(xentropy)

eps = 1e-10 # smoothing term to avoid computing log(0) which is NaN
latent_loss = 0.5 * tf.reduce_sum(
    tf.square(hidden3_sigma) + tf.square(hidden3_mean)
    - 1 - tf.log(eps + tf.square(hidden3_sigma)))

loss = reconstruction_loss + latent_loss

optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
training_op = optimizer.minimize(loss)

init = tf.global_variables_initializer()
saver = tf.train.Saver()

n_epochs = 50
batch_size = 150

with tf.Session() as sess:
    init.run()
    for epoch in range(n_epochs):
        n_batches = mnist.train.num_examples // batch_size
        for iteration in range(n_batches):
            print("\r{}%".format(100 * iteration // n_batches), end="")
            sys.stdout.flush()
            X_batch, y_batch = mnist.train.next_batch(batch_size)
            sess.run(training_op, feed_dict={X: X_batch})
        loss_val, reconstruction_loss_val, latent_loss_val = sess.run([loss, reconstruction_loss, latent_loss],...


I am working on the file "15_autoencoders.ipynb".
The first task is:
Visualizing the Reconstructions
Can you write Python code for it?

n_test_digits = 2
X_test = mnist.test.images[:n_test_digits]

with tf.Session() as sess:
    saver.restore(sess, "./my_model_one_at_a_time.ckpt") # not shown in the book
    outputs_val = outputs.eval(feed_dict={X: X_test})

def plot_image(image, shape=[28, 28]):
    plt.imshow(image.reshape(shape), cmap="Greys", interpolation="nearest")
    plt.axis("off")

for digit_index in range(n_test_digits):
    plt.subplot(n_test_digits, 2, digit_index * 2 + 1)
    plot_image(X_test[digit_index])
    plt.subplot(n_test_digits, 2, digit_index * 2 + 2)
    plot_image(outputs_val[digit_index])


I am working on the file "15_autoencoders.ipynb".
The first task is:
Visualizing the extracted features
Can you write Python code for it?

with tf.Session() as sess:
    saver.restore(sess, "./my_model_one_at_a_time.ckpt") # not shown in the book
    weights1_val = weights1.eval()

for i in range(5):
    plt.subplot(1, 5, i + 1)
    plot_image(weights1_val.T[i])

save_fig("extracted_features_plot") # not shown
plt.show()                          # not shown


I am working on the file "15_dataframe_analysis/attic/lowfreq_bike_stations.ipynb".
The first task is:
25 meters is approximately 0.000225˚ (not accounting for latitude)
latitude will give a factor of  cos(41˚N)=0.76 at 41˚N. So we are really testing
if the point lies within an ellipse with a radius of 25.6 meters in latitude
and 25.6*0.75=19.2 meters in longitude. Should be good enough.
Can you write Python code for it?

0.000225*111194.9 # 111.1949 km per degree latitude

query_results = bt.query_radius(trip_points.ix[:, 2:], 0.000225)


I am working on the file "15_dataframe_analysis/attic/lowfreq_bike_stations.ipynb".
The first task is:
Do a spatial nearest neighbor search
Construct a BallTree using the actual correct coordinate data from the JSON feed. Use that to lookup the locations in trip_points.
Can you write Python code for it?

bt = sklearn.neighbors.BallTree(df_cur[['latitude', 'longitude']].values, metric='euclidean')

np.cos(np.deg2rad(40.7))


I am working on the file "15_dataframe_analysis/attic/lowfreq_bike_stations.ipynb".
The first task is:
Merge spatial query results with trip_points
Historical trip data with can be compared to the current station list.
Can you write Python code for it?

# nan if no match, else row_id in df_cur
trip_points['match_id'] = np.array([np.float64(x)
                                    if x.shape[0] > 0 else np.nan
                                    for x in query_results.T])

trip_points.head()

df_cur['id'] = df_cur['id'].astype(np.float64)

lookup_table = trip_points.merge(df_cur, left_on='match_id', right_index=True, how='left')
lookup_table


I am working on the file "15_dataframe_analysis/attic/lowfreq_bike_stations.ipynb".
The first task is:
Read current data state from web
This will be the authoritative list of bike stations and their coordinates
Can you write Python code for it?

# Uncomment these two lines to fetch latest file and rename it

# !wget https://feeds.citibikenyc.com/stations/stations.json
# !mv stations.json stations.`date +"%Y.%m.%d.%H.%M"`.json

df_cur = pd.DataFrame(
    json.loads(
        open('stations.2017.04.20.09.43.json', 'r').read()
    )['stationBeanList'])
df_cur.head()

df_cur = df_cur[['id', 'stationName', 'latitude', 'longitude']].sort_values('id')
df_cur.head()

# We see this has less than half the rows of trip_points
print(df_cur.shape)


I am working on the file "15_dataframe_analysis/attic/lowfreq_bike_stations.ipynb".
The first task is:
Read historical bike data, print head and tail
Can you write Python code for it?

df = dd.read_parquet('/bigdata/citibike.parquet')

df.head()

df.tail()

# Count rows
df.start_station_id.compute().shape

# Select start and end columns as separate dataframes,
# apply range checks on lats and lons to eliminate corrupted rows

df_s = df[['start_station_id', 'start_station_name',
           'start_station_latitude', 'start_station_longitude']]
df_e = df[['end_station_id', 'end_station_name',
           'end_station_latitude', 'end_station_longitude']]

df_s = df_s[(df_s.start_station_latitude > 40.) & (df_s.start_station_latitude < 41.)]
df_s = df_s[(df_s.start_station_longitude + 74.0 > -0.25) & (df_s.start_station_longitude + 74.0 < 0.25)]

df_e = df_e[(df_e.end_station_latitude > 40.) & (df_e.end_station_latitude < 41.)]
df_e = df_e[(df_e.end_station_longitude + 74.0 > -0.25) & (df_e.end_station_longitude + 74.0 < 0.25)]

df_s.head()


I am working on the file "15_dataframe_analysis/attic/lowfreq_bike_stations.ipynb".
The first task is:
Set up a local cluster of Dask Distributed
Can you write Python code for it?


I am working on the file "15_dataframe_analysis/attic/lowfreq_bike_stations.ipynb".
The first task is:
Take the start locations, end locations, calculate mean latitude and longitude, convert to pandas, rename columns and concatenate. Sort by id. This will contain duplicates due to numerical precision and due to actual duplication due to some other reason.
Can you write Python code for it?

a1 = df_s.groupby(['start_station_id', 'start_station_name']).mean().compute()
a2 = df_e.groupby(['end_station_id', 'end_station_name']).mean().compute()

a1.index = a1.index.rename(['id', 'name'])
a1.columns = ['lat', 'lon']
a2.index = a2.index.rename(['id', 'name'])
a2.columns = ['lat', 'lon']

trip_points = a1.append(a2).drop_duplicates().reset_index()
trip_points = trip_points.sort_values('id').reset_index(drop=True)

del a1, a2
trip_points.head()

trip_points.shape


I am working on the file "15_dataframe_analysis/attic/lowfreq_bike_stations.ipynb".
The first task is:
The above table shows rows with NaN. These stations have no current counterpart.
Can you write Python code for it?

defunct_stations = lookup_table[np.isnan(lookup_table.match_id)]
defunct_stations

defunct_stations = defunct_stations[['id_x', 'name']].drop_duplicates()
defunct_stations

defunct_trips_start = df.merge(defunct_stations, left_on=['start_station_id', 'start_station_name'],
         right_on=['id_x', 'name'])
defunct_trips_end = df.merge(defunct_stations, left_on=['end_station_id', 'end_station_name'],
         right_on=['id_x', 'name'])
defunct_trips_all = (defunct_trips_start.append(defunct_trips_end).compute().drop_duplicates())
defunct_trips_all


I am working on the file "15_dataframe_analysis/attic/lowfreq_bike_stations.ipynb".
The first task is:
The value of each row indicates the row in df_cur that matches each row in trip_points
A zero for the first and second rows indicates that those rows in trip points match with
the zeroth row in df_cur.
Can you write Python code for it?

results_list = [[int(y) for y in x.tolist()] for x in query_results]

print(len(results_list))
results_list


I am working on the file "15_dataframe_analysis/attic/lowfreq_bike_stations.ipynb".
The first task is:
^^
Over five million rows that start or end at defunct stations!
We can't drop that many given the total is 36 million!
Can you write Python code for it?

defunct_start_trips_count = defunct_trips_all[['start_station_id', 'start_station_name', 'trip_duration']].groupby(['start_station_id', 'start_station_name']).count().sort_values('trip_duration',
                                                                                ascending=False)
bad_ids = defunct_start_trips_count[defunct_start_trips_count.trip_duration < 50].reset_index()[['start_station_id']]
defunct_start_trips_count[defunct_start_trips_count.trip_duration < 50]

defunct_end_trips_count = defunct_trips_all[[
    'end_station_id', 'end_station_name', 'trip_duration']].groupby(
    ['end_station_id', 'end_station_name']).count().sort_values('trip_duration', ascending=False)
bad_ids_2 = (
    defunct_end_trips_count[defunct_end_trips_count.trip_duration < 50].reset_index()
)[['end_station_id']]
defunct_end_trips_count[defunct_end_trips_count.trip_duration < 50]

bad_ids.columns =['id']
bad_ids_2.columns = ['id']
bad_ids = bad_ids.append(bad_ids_2).drop_duplicates().sort_values('id').reset_index(drop=True)
from IPython.display import display, HTML
display(HTML(bad_ids.T.to_html()))

bad_ids.id.T.values

df2 = df.merge(bad_ids, left_on='start_station_id', right_on='id', how='left').merge(
    bad_ids, left_on='end_station_id', right_on='id', how='left'
)

df3 = df2[df2.id_x.isnull() & df2.id_y.isnull()]

df3.start_station_id.compute().shape

df.start_station_id.compute().shape


I am working on the file "15_gan_mnist/Intro_to_GANs_Exercises.ipynb".
The first task is:
Below I'm showing the generated images as the network was training, every 10 epochs. With bonus optical illusion!
Can you write Python code for it?

rows, cols = 10, 6
fig, axes = plt.subplots(figsize=(7,12), nrows=rows, ncols=cols, sharex=True, sharey=True)
print(np.array(samples).shape)

for sample, ax_row in zip(samples[::int(len(samples)/rows)], axes):
    for img, ax in zip(sample[0][::int(len(sample[0])/cols)], ax_row):
        ax.imshow(img.reshape((28,28)), cmap='Greys_r')
        ax.xaxis.set_visible(False)
        ax.yaxis.set_visible(False)


I am working on the file "15_gan_mnist/Intro_to_GANs_Exercises.ipynb".
The first task is:
Build network
Now we're building the network from the functions defined above.
First is to get our inputs, input_real, input_z from model_inputs using the sizes of the input and z.
Then, we'll create the generator, generator(input_z, input_size). This builds the generator with the appropriate input and output sizes.
Then the discriminators. We'll build two of them, one for real data and one for fake data. Since we want the weights to be the same for both real and fake data, we need to reuse the variables. For the fake data, we're getting it from the generator as g_model. So the real data discriminator is discriminator(input_real) while the fake discriminator is discriminator(g_model, reuse=True).

Exercise: Build the network from the functions you defined earlier.
Can you write Python code for it?

tf.reset_default_graph()
# Create our input placeholders
input_real, input_z = model_inputs(input_size, z_size)

# Generator network here
g_model, g_logits = generator(input_z, input_size, g_hidden_size, False, alpha)
# g_model is the generator output

# Disriminator network here
d_model_real, d_logits_real = discriminator(input_real, d_hidden_size, False, alpha)
d_model_fake, d_logits_fake = discriminator(g_model, d_hidden_size, True, alpha)


I am working on the file "15_gan_mnist/Intro_to_GANs_Exercises.ipynb".
The first task is:
Discriminator
The discriminator network is almost exactly the same as the generator network, except that we're using a sigmoid output layer.

Exercise: Implement the discriminator network in the function below. Same as above, you'll need to return both the logits and the sigmoid output. Make sure to wrap your code in a variable scope, with 'discriminator' as the scope name, and pass the reuse keyword argument from the function arguments to tf.variable_scope.
Can you write Python code for it?

def discriminator(x, n_units=128, reuse=False, alpha=0.01):
    ''' Build the discriminator network.

        Arguments
        ---------
        x : Input tensor for the discriminator
        n_units: Number of units in hidden layer
        reuse : Reuse the variables with tf.variable_scope
        alpha : leak parameter for leaky ReLU

        Returns
        -------
        out, logits: generated output after and before activation
    '''
    with tf.variable_scope('discriminator', reuse=reuse):
        # Hidden layer
        #h1 = tf.contrib.layers.fully_connected(x, n_units, activation_fn=None)
        # shorter
        h1 = tf.layers.dense(x, n_units)
        # Leaky ReLU
        h1 = tf.maximum(alpha * h1, h1)

        #logits = tf.contrib.layers.fully_connected(h1, 1, activation_fn=None)
        # shape
        logits = tf.layers.dense(h1, 1)
        out = tf.sigmoid(logits)

        return out, logits


I am working on the file "15_gan_mnist/Intro_to_GANs_Exercises.ipynb".
The first task is:
Discriminator and Generator Losses
For the discriminator, the total loss is the sum of the losses for real and fake images, d_loss = d_loss_real + d_loss_fake. The losses will by sigmoid cross-entropies, which we can get with tf.nn.sigmoid_cross_entropy_with_logits. We'll also wrap that in tf.reduce_mean to get the mean for all the images in the batch. So the losses will look something like
python
tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=labels))
For the real image labels, we want them to be all ones, since these are all real images. To help the discriminator generalize better, the labels are reduced a bit from 1.0 to 0.9, for example,  using the parameter smooth. This is known as label smoothing, typically used with classifiers to improve performance. In TensorFlow, it looks something like labels = tf.ones_like(tensor) * (1 - smooth)
The discriminator loss for the fake data is similar. The fake logits are used with labels of all zeros. We want the discriminator to output 1 for real images and 0 for fake images, so we need to set up the losses to reflect that.
Finally, the generator losses are using the labels that are all ones. The generator is trying to fool the discriminator, so it wants to discriminator to output ones for fake images.
Can you write Python code for it?

# Calculate losses
d_loss_real = tf.reduce_mean(
    tf.nn.sigmoid_cross_entropy_with_logits(
        labels=tf.ones_like(d_logits_real) * (1-smooth),
        logits=d_logits_real))

d_loss_fake = tf.reduce_mean(
    tf.nn.sigmoid_cross_entropy_with_logits(
        labels=tf.zeros_like(d_logits_fake),
        logits=d_logits_fake))

d_loss = d_loss_real + d_loss_fake

g_loss = tf.reduce_mean(
    tf.nn.sigmoid_cross_entropy_with_logits(
        labels=tf.ones_like(d_logits_fake),
        logits=d_logits_fake))