I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/2_Optimizing_fdac2d_code.ipynb".
The first task is:
Finally, we produce some nice bar charts to compare the performance of the different codes developed in this Jupyter notebook:
Can you write Python code for it?

# define codes
codes = ('Python', 'Python + JIT', 'Python + NumPy', 'Python + NumPy + JIT', 'C++')
y_pos = np.arange(len(codes))

# runtime
performance = [t_vanilla_python,t_JIT_python,t_numpy_python,t_numpy_python_JIT,t_cxx]

# speed-up with respect to the non-optimized code
speedup = [t_vanilla_python/t_vanilla_python,
           t_vanilla_python/t_JIT_python,
           t_vanilla_python/t_numpy_python,
           t_vanilla_python/t_numpy_python_JIT,
           t_vanilla_python/t_cxx]

# Define figure size
rcParams['figure.figsize'] = 12, 8

# Plot runtimes of 2D acoustic FD codes
ax1 = plt.subplot(211)

plt.bar(y_pos, performance, align='center', alpha=0.5)
plt.xticks(y_pos, codes)
plt.ylabel('Runtime (s)')
plt.title('Performance comparison of 2D acoustic FD modelling codes')

# make tick labels invisible
plt.setp(ax1.get_xticklabels(), visible=False)

# Plot speedup of 2D acoustic FD codes
ax2 = plt.subplot(212, sharex=ax1)

plt.bar(y_pos, speedup, align='center', alpha=0.5,color='r')
plt.xticks(y_pos, codes)
plt.ylabel('Speedup ()')
plt.tight_layout()
plt.show()


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/2_Optimizing_fdac2d_code.ipynb".
The first task is:
Is this the best result we can achieve or are further code improvements possible?
Using domain decomposition with the Message-Passing Interface MPI to distribute the workload over multiple CPU cores, combined with a partioning of the tasks in each domain using Multithreading can significantly improve the code performance. One key is the manual optimization of CPU and GPU kernels, especially regarding memory access times or communication between MPI processes. As an example I plotted the runtime and speedup for the same homogeneous acoustic problem from this Jupyter notebook using the 2D acoustic modelling code DENISE Black-Edition which only relies on MPI:
Can you write Python code for it?

# Define figure size
rcParams['figure.figsize'] = 12, 6

# number of cores and runtime
cores = np.array([1, 2, 4, 8, 16, 25])
t_denise = np.array([0.926, 0.482, 0.234, 0.123, 0.067, 0.055])

# speed-up with respect to the runtime of the 1st core
# and linear speedup
speedup_denise = t_denise[0] / t_denise
linear_speedup = cores

# plot runtime
ax2 = plt.subplot(121)
plt.plot(cores, t_denise, 'rs-',lw=3,label="Runtime")
plt.title('Runtime DENISE Black-Edition')
plt.xlabel('Number of cores')
plt.ylabel('Runtime (s)')
plt.legend()
plt.grid()

# plot speedup
ax2 = plt.subplot(122)
plt.plot(cores, speedup_denise, 'bs-',lw=3,label="Speedup DENISE")
plt.plot(cores, linear_speedup, 'k-',lw=3,label="Linear speedup")
plt.title('Speedup DENISE Black-Edition')
plt.xlabel('Number of cores')
plt.ylabel('Speedup')
plt.legend()
plt.grid()

plt.show()


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/2_Optimizing_fdac2d_code.ipynb".
The first task is:
Performance optimization of the 2D acoustic finite difference modelling code
During the last class, it took us only 15 minutes to develop a 2D acoustic FD code based on the 1D code. However, with a runtime of roughly 3 minutes, the performance of this "vanilla" Python implementation was quite underwhelming. Therefore, the aim of this lesson is to optimize the performance of this code.
Let's start with a slightly modified version of the original code. Basically, I moved the computation of the analytical solution outside the main code, the discretization parameters $nx,\; nz,\; nt,\; dx,\; dz,\; dt$ are also fixed in order to minimize the input to the FD modelling function.
Can you write Python code for it?

# Import Libraries
# ----------------
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
from pylab import rcParams

# Ignore Warning Messages
# -----------------------
import warnings
warnings.filterwarnings("ignore")

# Definition of modelling parameters
# ----------------------------------
xmax = 500.0 # maximum spatial extension of the 1D model in x-direction (m)
zmax = xmax  # maximum spatial extension of the 1D model in z-direction(m)
dx   = 1.0   # grid point distance in x-direction
dz   = dx    # grid point distance in z-direction

tmax = 0.502   # maximum recording time of the seismogram (s)
dt   = 0.0010  # time step

vp0  = 580.   # P-wave speed in medium (m/s)

# acquisition geometry
xr = 330.0 # x-receiver position (m)
zr = xr    # z-receiver position (m)

xsrc = 250.0 # x-source position (m)
zsrc = 250.0 # z-source position (m)

f0   = 40. # dominant frequency of the source (Hz)
t0   = 4. / f0 # source time shift (s)

# define model discretization
# ---------------------------

nx = (int)(xmax/dx) # number of grid points in x-direction
print('nx = ',nx)

nz = (int)(zmax/dz) # number of grid points in x-direction
print('nz = ',nz)

nt = (int)(tmax/dt) # maximum number of time steps
print('nt = ',nt)

ir = (int)(xr/dx)      # receiver location in grid in x-direction
jr = (int)(zr/dz)      # receiver location in grid in z-direction

isrc = (int)(xsrc/dx)  # source location in grid in x-direction
jsrc = (int)(zsrc/dz)  # source location in grid in x-direction

# Source time function (Gaussian)
# -------------------------------
src  = np.zeros(nt + 1)
time = np.linspace(0 * dt, nt * dt, nt)

# 1st derivative of a Gaussian
src  = -2. * (time - t0) * (f0 ** 2) * (np.exp(- (f0 ** 2) * (time - t0) ** 2))

# Analytical solution
# -------------------
G    = time * 0.

# Initialize coordinates
# ----------------------
x    = np.arange(nx)
x    = x * dx       # coordinates in x-direction (m)

z    = np.arange(nz)
z    = z * dz       # coordinates in z-direction (m)

#...


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/2_Optimizing_fdac2d_code.ipynb".
The first task is:
So JIT could also improve the performance of the code using NumPy array operations, but the performance of the compiled code with the nested FOR loops has a slight edge in terms of performance.
Comparison with a C++ implementation
How does the performance of the JIT-codes compare to a C++ bully code? I invested 1 hour to write this C++ code, which is similar to the 2D acoustic FD Python code.
In order to use similar matrix data structures in C++ as in Python, I use the Eigen library:
www.eigen.tuxfamily.org/
which also allows auto-vectorization of matrix-matrix products. To compile the source code, you need a C++ compiler, e.g. g++ and the Eigen library which can either be compiled from source or installed using the package manager of your Linux distribution.
I also recommend to use the moderate optimization option -O2 and Advanced Vector Extensions (AVX) -mavx during code compilation for a significant performance increase of the code. Let's compile and run the code:
Can you write Python code for it?

# Compile and run C++-version

# load seismogram
time_Cpp, seis_FD_Cpp = np.loadtxt('seis.dat', delimiter='\t', skiprows=0, unpack=True)

t_cxx =  # runtime of C++ code (s)


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/3_fdac2d_num_stability_anisotropy.ipynb".
The first task is:
Numerical Grid Dispersion
While the FD solution above is stable, it is subject to some numerical dispersion when compared with the analytical solution. The grid point distance $dx = 10\; m$, P-wave velocity $vp = 3000\; m/s$ and a maximum frequency $f_{max} \approx 2*f_0 = 40\; Hz$ leads to ...
Can you write Python code for it?

fmax = 2 * f0
N_lam = vp0 / (dx * fmax)
print("N_lam = ",N_lam)


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/3_fdac2d_num_stability_anisotropy.ipynb".
The first task is:
Numerical stability, dispersion and anisotropy of the 2D acoustic finite difference modelling code
Similar to the 1D acoustic FD modelling code, we have to investigate the stability and dispersion of the 2D numerical scheme. Additionally, the numerical dispersion shows in the 2D case an anisotropic behaviour.
Let's begin with the CFL-stability criterion ...
CFL-stability criterion for the 2D acoustic FD modelling code
As for the 1D code, the maximum size of the timestep $dt$ is limited by the Courant-Friedrichs-Lewy (CFL) criterion:
\begin{equation}
dt \le \frac{dx}{\zeta v_{max}}, \nonumber
\end{equation}
where $dx$ denotes the spatial grid point distance and $v_{max}$ the maximum P-wave velocity. The factor $\zeta$ depends on the used FD-operators, dimension and numerical scheme.
As for the 1D case, we estimate the factor $\zeta$ by the von Neumann analysis, starting with the finite difference approximation of the 2D acoustic wave equation
\begin{equation}
 \frac{p_{j,l}^{n+1} - 2 p_{j,l}^n + p_{j,l}^{n-1}}{\mathrm{d}t^2} \ = \ vp_{j,l}^2\biggl(\frac{p_{j,l+1}^{n} - 2 p_{j,l}^n + p_{j,l-1}^{n}}{\mathrm{d}x^2} + \frac{p_{j+1,l}^{n} - 2 p_{j,l}^n + p_{j-1,l}^{n}}{\mathrm{d}z^2}\biggr),
\end{equation}
and assuming harmonic plane wave solutions for the pressure wavefield of the form:
\begin{equation}
p = exp(i(k_x x + k_z z -\omega t)),\nonumber
\end{equation}
with $i^2=-1$, the wavenumbers $(k_x, k_z)$ in x-/z-direction, respectively, and the circular frequency $\omega$. Using a regular grid with
$dx = dz = dh,$
discrete spatial coordinates
$x_j = j dh,$
$z_l = l dh,$
and times
$t_n = n dt.$
we can calculate discrete plane wave solutions at the discrete locations and times in eq. (1):
\begin{align}
p_{j,l}^{n+1} &= exp(-i\omega dt)\; p_{j,l}^{n},\
p_{j,l}^{n-1} &= exp(i\omega dt)\; p_{j,l}^{n},\
p_{j+1,l}^{n} &= exp(ik_x dh)\; p_{j,l}^{n},\
p_{j-1,l}^{n} &= exp(-ik_x dh)\; p_{j,l}^{n},\
p_{j,l+1}^{n} &= exp(ik_z dh)\; p_{j,l}^{n},\
p_{j,l-1}^{n} &= exp(-ik_z dh)\; p_{j,l}^{n},\
\end{align}
Inserting eqs....
Can you write Python code for it?

# Import Libraries
# ----------------
import numpy as np
from numba import jit
import matplotlib
import matplotlib.pyplot as plt
from pylab import rcParams

# Ignore Warning Messages
# -----------------------
import warnings
warnings.filterwarnings("ignore")

# Definition of modelling parameters
# ----------------------------------
xmax = 5000.0 # maximum spatial extension of the 1D model in x-direction (m)
zmax = xmax   # maximum spatial extension of the 1D model in z-direction (m)
dx   = 10.0   # grid point distance in x-direction (m)
dz   = dx     # grid point distance in z-direction (m)

tmax = 0.8    # maximum recording time of the seismogram (s)
dt   = 0.0010 # time step

vp0  = 3000.  # P-wave speed in medium (m/s)

# acquisition geometry
xr = 2000.0   # x-receiver position (m)
zr = xr       # z-receiver position (m)

xsrc = 2500.0 # x-source position (m)
zsrc = xsrc   # z-source position (m)

f0   = 20. # dominant frequency of the source (Hz)
t0   = 4. / f0 # source time shift (s)

# FD_2D_acoustic code with JIT optimization
# -----------------------------------------
@jit(nopython=True) # use Just-In-Time (JIT) Compilation for C-performance
def FD_2D_acoustic_JIT(dt,dx,dz,f0):

    # define model discretization
    # ---------------------------

    nx = (int)(xmax/dx) # number of grid points in x-direction
    print('nx = ',nx)

    nz = (int)(zmax/dz) # number of grid points in x-direction
    print('nz = ',nz)

    nt = (int)(tmax/dt) # maximum number of time steps
    print('nt = ',nt)

    ir = (int)(xr/dx)      # receiver location in grid in x-direction
    jr = (int)(zr/dz)      # receiver location in grid in z-direction

    isrc = (int)(xsrc/dx)  # source location in grid in x-direction
    jsrc = (int)(zsrc/dz)  # source location in grid in x-direction

    # Source time function (Gaussian)
    # -------------------------------
    src  = np.zeros(nt + 1)
    time = np.linspace(0 * dt, nt * dt, nt)

    # 1st derivative of a Gaussian
    src  = -2. * (time - t0)...


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/4_fdac2d_absorbing_boundary.ipynb".
The first task is:
After defining the modelling parameters, we can run the modified FD code ...
Can you write Python code for it?

%matplotlib notebook
dx   = 5.0   # grid point distance in x-direction (m)
dz   = dx     # grid point distance in z-direction (m)
f0   = 100.0  # centre frequency of the source wavelet (Hz)

# calculate dt according to the CFL-criterion
dt = dx / (np.sqrt(2.0) * vp0)

FD_2D_acoustic_JIT(dt,dx,dz,f0)


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/4_fdac2d_absorbing_boundary.ipynb".
The first task is:
In order to modularize the code, we move the 2nd partial derivatives of the wave equation into a function update_d2px_d2pz, so the application of the JIT decorator can be restriced to this function:
Can you write Python code for it?

@jit(nopython=True) # use JIT for C-performance
def update_d2px_d2pz(p, dx, dz, nx, nz, d2px, d2pz):

    for i in range(1, nx - 1):
        for j in range(1, nz - 1):

            d2px[i,j] = (p[i + 1,j] - 2 * p[i,j] + p[i - 1,j]) / dx**2
            d2pz[i,j] = (p[i,j + 1] - 2 * p[i,j] + p[i,j - 1]) / dz**2

    return d2px, d2pz


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/4_fdac2d_absorbing_boundary.ipynb".
The first task is:
In the FD modelling code FD_2D_acoustic_JIT, a more flexible model definition is introduced by the function model. The block Initalize animation of pressure wavefield before the time loop displays the velocity model and initial pressure wavefield. During the time-loop, the pressure wavefield is updated with
image.set_data(p.T)
fig.canvas.draw()
at the every isnap timestep:
Can you write Python code for it?

# FD_2D_acoustic code with JIT optimization
# -----------------------------------------
def FD_2D_acoustic_JIT(dt,dx,dz,f0):

    # define model discretization
    # ---------------------------

    nx = (int)(xmax/dx) # number of grid points in x-direction
    print('nx = ',nx)

    nz = (int)(zmax/dz) # number of grid points in x-direction
    print('nz = ',nz)

    nt = (int)(tmax/dt) # maximum number of time steps
    print('nt = ',nt)

    isrc = (int)(xsrc/dx)  # source location in grid in x-direction
    jsrc = (int)(zsrc/dz)  # source location in grid in x-direction

    # Source time function (Gaussian)
    # -------------------------------
    src  = np.zeros(nt + 1)
    time = np.linspace(0 * dt, nt * dt, nt)

    # 1st derivative of Gaussian
    src  = -2. * (time - t0) * (f0 ** 2) * (np.exp(- (f0 ** 2) * (time - t0) ** 2))

    # define clip value: 0.1 * absolute maximum value of source wavelet
    clip = 0.1 * max([np.abs(src.min()), np.abs(src.max())]) / (dx*dz) * dt**2

    # Define model
    # ------------
    vp  = np.zeros((nx,nz))
    vp  = model(nx,nz,vp,dx,dz)
    vp2 = vp**2

    # Initialize empty pressure arrays
    # --------------------------------
    p    = np.zeros((nx,nz)) # p at time n (now)
    pold = np.zeros((nx,nz)) # p at time n-1 (past)
    pnew = np.zeros((nx,nz)) # p at time n+1 (present)
    d2px = np.zeros((nx,nz)) # 2nd spatial x-derivative of p
    d2pz = np.zeros((nx,nz)) # 2nd spatial z-derivative of p

    # Initalize animation of pressure wavefield
    # -----------------------------------------
    fig = plt.figure(figsize=(7,3.5))  # define figure size
    plt.tight_layout()
    extent = [0.0,xmax,zmax,0.0]     # define model extension

    # Plot pressure wavefield movie
    ax1 = plt.subplot(121)
    image = plt.imshow(p.T, animated=True, cmap="RdBu", extent=extent,
                          interpolation='nearest', vmin=-clip, vmax=clip)
    plt.title('Pressure wavefield')
   ...


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/4_fdac2d_absorbing_boundary.ipynb".
The first task is:
Simple absorbing boundary for 2D acoustic FD modelling
Realistic FD modelling results for surface seismic acquisition geometries require a further modification of the 2D acoustic FD code. Except for the free surface boundary condition on top of the model, we want to suppress the artifical reflections from the other boundaries.
Such absorbing boundaries can be implemented by different approaches. A comprehensive overview is compiled in
Gao et al. 2015, Comparison of artificial absorbing boundaries for acoustic wave equation modelling
Before implementing the absorbing boundary frame, we modify some parts of the optimized 2D acoustic FD code:
Can you write Python code for it?

# Import Libraries
# ----------------
import numpy as np
from numba import jit
import matplotlib
import matplotlib.pyplot as plt
from pylab import rcParams

# Ignore Warning Messages
# -----------------------
import warnings
warnings.filterwarnings("ignore")

from mpl_toolkits.axes_grid1 import make_axes_locatable

# Definition of initial modelling parameters
# ------------------------------------------
xmax = 2000.0 # maximum spatial extension of the 1D model in x-direction (m)
zmax = xmax   # maximum spatial extension of the 1D model in z-direction (m)
dx   = 10.0   # grid point distance in x-direction (m)
dz   = dx     # grid point distance in z-direction (m)

tmax = 0.75    # maximum recording time of the seismogram (s)
dt   = 0.0010 # time step

vp0  = 3000.  # P-wave speed in medium (m/s)

# acquisition geometry
xsrc = 1000.0 # x-source position (m)
zsrc = xsrc   # z-source position (m)

f0   = 100.0 # dominant frequency of the source (Hz)
t0   = 0.1   # source time shift (s)

isnap = 2  # snapshot interval (timesteps)


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/4_fdac2d_absorbing_boundary.ipynb".
The first task is:
The FD code itself requires only some small modifications, we have to add the absorb function to define the amount of damping in the boundary frame and apply the damping function to the pressure wavefields pnew and p
Can you write Python code for it?

# FD_2D_acoustic code with JIT optimization
# -----------------------------------------
def FD_2D_acoustic_JIT_absorb(dt,dx,dz,f0):

    # define model discretization
    # ---------------------------

    nx = (int)(xmax/dx) # number of grid points in x-direction
    print('nx = ',nx)

    nz = (int)(zmax/dz) # number of grid points in x-direction
    print('nz = ',nz)

    nt = (int)(tmax/dt) # maximum number of time steps
    print('nt = ',nt)

    isrc = (int)(xsrc/dx)  # source location in grid in x-direction
    jsrc = (int)(zsrc/dz)  # source location in grid in x-direction

    # Source time function (Gaussian)
    # -------------------------------
    src  = np.zeros(nt + 1)
    time = np.linspace(0 * dt, nt * dt, nt)

    # 1st derivative of Gaussian
    src  = -2. * (time - t0) * (f0 ** 2) * (np.exp(- (f0 ** 2) * (time - t0) ** 2))

    # define clip value: 0.1 * absolute maximum value of source wavelet
    clip = 0.1 * max([np.abs(src.min()), np.abs(src.max())]) / (dx*dz) * dt**2

    # Define absorbing boundary frame
    # -------------------------------

    # Define model
    # ------------
    vp  = np.zeros((nx,nz))
    vp  = model(nx,nz,vp,dx,dz)
    vp2 = vp**2

    # Initialize empty pressure arrays
    # --------------------------------
    p    = np.zeros((nx,nz)) # p at time n (now)
    pold = np.zeros((nx,nz)) # p at time n-1 (past)
    pnew = np.zeros((nx,nz)) # p at time n+1 (present)
    d2px = np.zeros((nx,nz)) # 2nd spatial x-derivative of p
    d2pz = np.zeros((nx,nz)) # 2nd spatial z-derivative of p

    # Initalize animation of pressure wavefield
    # -----------------------------------------
    fig = plt.figure(figsize=(7,3.5))  # define figure size
    plt.tight_layout()
    extent = [0.0,xmax,zmax,0.0]     # define model extension

    # Plot pressure wavefield movie
    ax1 = plt.subplot(121)
    image = plt.imshow(p.T, animated=True, cmap="RdBu", extent=extent,
                         ...


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/5_fdac2d_heterogeneous.ipynb".
The first task is:
2D acoustic FD modelling for heterogeneous media
So far, we only compared 2D acoustic FD modelling results for homogeneous acoustic media with analytical solutions. Next, we want to model some more interesting, heterogeneous problems.
Can you write Python code for it?

# Import Libraries
# ----------------
import numpy as np
from numba import jit
import matplotlib
import matplotlib.pyplot as plt
from pylab import rcParams

# Ignore Warning Messages
# -----------------------
import warnings
warnings.filterwarnings("ignore")

from mpl_toolkits.axes_grid1 import make_axes_locatable

# Definition of initial modelling parameters
# ------------------------------------------
xmax = 2000.0 # maximum spatial extension of the 1D model in x-direction (m)
zmax = xmax   # maximum spatial extension of the 1D model in z-direction (m)
dx   = 10.0   # grid point distance in x-direction (m)
dz   = dx     # grid point distance in z-direction (m)

tmax = 0.75    # maximum recording time of the seismogram (s)
dt   = 0.0010 # time step

vp0  = 3000.  # P-wave speed in medium (m/s)

# acquisition geometry
xsrc = 1000.0 # x-source position (m)
zsrc = xsrc   # z-source position (m)

f0   = 100.0 # dominant frequency of the source (Hz)
t0   = 0.1   # source time shift (s)

isnap = 2  # snapshot interval (timesteps)

@jit(nopython=True) # use JIT for C-performance
def update_d2px_d2pz(p, dx, dz, nx, nz, d2px, d2pz):

    for i in range(1, nx - 1):
        for j in range(1, nz - 1):

            d2px[i,j] = (p[i + 1,j] - 2 * p[i,j] + p[i - 1,j]) / dx**2
            d2pz[i,j] = (p[i,j + 1] - 2 * p[i,j] + p[i,j - 1]) / dz**2

    return d2px, d2pz

# Define simple absorbing boundary frame based on wavefield damping
# according to Cerjan et al., 1985, Geophysics, 50, 705-708
def absorb(nx,nz):

    FW = 60     # thickness of absorbing frame (gridpoints)
    a = 0.0053

    coeff = np.zeros(FW)

    # define coefficients in absorbing frame
    for i in range(FW):
        coeff[i] = np.exp(-(a**2 * (FW-i)**2))

    # initialize array of absorbing coefficients
    absorb_coeff = np.ones((nx,nz))

    # compute coefficients for left grid boundaries (x-direction)
    zb=0
    for i in range(FW):
        ze = nz - i - 1
        for j in...


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/5_fdac2d_heterogeneous.ipynb".
The first task is:
Problem 1: Homogeneous Model
As a reference, we start with a problem, which should be quite familiar to you - the homogeneous model. I modified the FD code to define models in a separate Python function:
Can you write Python code for it?

# Homogeneous model
def model(nx,nz,vp,dx,dz):

    return vp


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/5_fdac2d_heterogeneous.ipynb".
The first task is:
Problem 2: Random Medium
Next, we add some random perturbations to the homogeneous Vp-model:
Can you write Python code for it?

# Random medium model
def model(nx,nz,vp,dx,dz):

    return vp

xsrc = 1000.0 # x-source position (m)
zsrc = xsrc    # z-source position (m)

dx   = 5.0   # grid point distance in x-direction (m)
dz   = dx     # grid point distance in z-direction (m)
f0   = 100.0  # centre frequency of the source wavelet (Hz)
vpmax = vp0

# calculate dt according to the CFL-criterion
dt = dx / (np.sqrt(2.0) * vpmax)

vp_rand = FD_2D_acoustic_JIT(dt,dx,dz,f0,xsrc,zsrc)


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/5_fdac2d_heterogeneous.ipynb".
The first task is:
Problem 3: Fault Zone
In this problem, we model acoustic wave propagation in a vertical fault zone
Can you write Python code for it?

# Vertical fault zone model
def model(nx,nz,vp,dx,dz):

    return vp

xsrc = 1000.0 # x-source position (m)
zsrc = 1000.0 # z-source position (m)

dx   = 5.0   # grid point distance in x-direction (m)
dz   = dx     # grid point distance in z-direction (m)
f0   = 100.0  # centre frequency of the source wavelet (Hz)
vpmax = 4200.0
# vpmax = np.max(vp0)

# calculate dt according to the CFL-criterion
dt = dx / (np.sqrt(2.0) * vpmax)

vp_fault = FD_2D_acoustic_JIT(dt,dx,dz,f0,xsrc,zsrc)


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/5_fdac2d_heterogeneous.ipynb".
The first task is:
Time to define the modelling parameters and run the new FD code ...
Can you write Python code for it?

%matplotlib notebook
xsrc = 1000.0 # x-source position (m)
zsrc = xsrc   # z-source position (m)

dx   = 5.0   # grid point distance in x-direction (m)
dz   = dx     # grid point distance in z-direction (m)
f0   = 100.0  # centre frequency of the source wavelet (Hz)

# calculate dt according to the CFL-criterion
dt = dx / (np.sqrt(2.0) * vp0)

vp_hom = FD_2D_acoustic_JIT(dt,dx,dz,f0,xsrc,zsrc)


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/7_fdac2d_sensitivity_kernels.ipynb".
The first task is:
... define a JIT-ed function for the spatial FD approximation ...
Can you write Python code for it?

@jit(nopython=True) # use JIT for C-performance
def update_d2px_d2pz_3pt(p, dx, dz, nx, nz, d2px, d2pz):

    for i in range(1, nx - 1):
        for j in range(1, nz - 1):

            d2px[i,j] = (p[i + 1,j] - 2 * p[i,j] + p[i - 1,j]) / dx**2
            d2pz[i,j] = (p[i,j + 1] - 2 * p[i,j] + p[i,j - 1]) / dz**2

    return d2px, d2pz


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/7_fdac2d_sensitivity_kernels.ipynb".
The first task is:
... initialize the absorbing boundary frame ...
Can you write Python code for it?

# Define simple absorbing boundary frame based on wavefield damping
# according to Cerjan et al., 1985, Geophysics, 50, 705-708
def absorb(nx,nz):

    FW = 60     # thickness of absorbing frame (gridpoints)
    a = 0.0053

    coeff = np.zeros(FW)

    # define coefficients in absorbing frame
    for i in range(FW):
        coeff[i] = np.exp(-(a**2 * (FW-i)**2))

    # initialize array of absorbing coefficients
    absorb_coeff = np.ones((nx,nz))

    # compute coefficients for left grid boundaries (x-direction)
    zb=0
    for i in range(FW):
        ze = nz - i - 1
        for j in range(zb,ze):
            absorb_coeff[i,j] = coeff[i]

    # compute coefficients for right grid boundaries (x-direction)
    zb=0
    for i in range(FW):
        ii = nx - i - 1
        ze = nz - i - 1
        for j in range(zb,ze):
            absorb_coeff[ii,j] = coeff[i]

    # compute coefficients for bottom grid boundaries (z-direction)
    xb=0
    for j in range(FW):
        jj = nz - j - 1
        xb = j
        xe = nx - j
        for i in range(xb,xe):
            absorb_coeff[i,jj] = coeff[j]

    return absorb_coeff


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/7_fdac2d_sensitivity_kernels.ipynb".
The first task is:
Computation of Sensitivity Kernels by 2D acoustic FD modelling
Beside the modelling of seismic surveys, our 2D acoustic FD code can be used as the core of a seismic full waveform inversion (FWI) approach. A very efficient implementation is possible in the frequency domain.
The aim of acoustic frequency domain FWI is to minimize the data residuals $\mathbf{\delta \tilde{P} = \tilde{P}^{mod} - \tilde{P}^{obs}}$ between the modelled frequency domain data $\mathbf{\tilde{P}^{mod}}$ and field data $\mathbf{\tilde{P}^{obs}}$ to deduce high resolution models of the P-wave velocity distribution in the subsurface. To solve this non-linear inversion problem, an objective function E, as a measure of the data misfit, has to be defined. The classical choice is the L2-norm of the data residuals
\begin{equation}
E  = ||\mathbf{\delta \tilde{P}}||2 = \frac{1}{2}\mathbf{\delta \tilde{P}}^\dagger \mathbf{\delta \tilde{P}} = \frac{1}{2} \sum{k=1}^{n_\omega} \sum_{i=1}^{ns} \sum_{j=1}^{nr} \delta \tilde{P}^*(\mathbf{x_s}_i, \mathbf{x_r}_j, \omega_k) \delta \tilde{P}(\mathbf{x_s}_i, \mathbf{x_r}_j, \omega_k) \notag
\end{equation}
where ns and nr are the number of shots and receivers, $n_\omega$ the number of discrete frequencies, $\dagger$ the complex transpose, $$ the complex conjugate, $\mathbf{x_s},\; \mathbf{x_r}$  the source and receiver positions, respectively. The objective function can be minimized by iteratively updating the P-wave velocity $\mathbf{Vp}$ at iteration step n, starting with an initial background model $\mathbf{Vp_0}$, along a search direction using the Newton* method:
\begin{equation}
\mathbf{Vp}{n+1} = \mathbf{Vp}{n} - \mu_n \mathbf{H}_n^{-1} \left(\mathbf{\frac{\partial E}{\partial Vp}}\right)_n, \notag
\end{equation}
where $\mu$ denotes the step length, $\mathbf{\frac{\partial E}{\partial Vp}}$ the gradient and ${\mathbf H}$ the second derivative (Hessian) of the objective function with respect to $\mathbf{Vp}$. The step length $\mu_{n}$ can be estimated by an inexact parabolic line search.
The...
Can you write Python code for it?

# Import Libraries
# ----------------
import numpy as np
from numba import jit
import matplotlib
import matplotlib.pyplot as plt
from pylab import rcParams

# Ignore Warning Messages
# -----------------------
import warnings
warnings.filterwarnings("ignore")

from mpl_toolkits.axes_grid1 import make_axes_locatable


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/7_fdac2d_sensitivity_kernels.ipynb".
The first task is:
In the 2D FD acoustic modelling code, we implement the DFT of the time-domain wavefields, by initializing the real and imaginary parts of the pressure wavefields, calculate the trigonometric factors for the DFT within the time-loop, apply the DFT to the pressure wavefield p for the discrete frequency freq and finally return the real and imaginary parts of the frequency domain wavefields ...
Can you write Python code for it?

# FD_2D_acoustic code with JIT optimization
# -----------------------------------------
def FD_2D_acoustic_JIT(vp,dt,dx,dz,f0,xsrc,zsrc,op,freq):

    # calculate number of time steps nt
    # ---------------------------------
    nt = (int)(tmax/dt)

    # locate source on Cartesian FD grid
    # ----------------------------------
    isrc = (int)(xsrc/dx)  # source location in grid in x-direction
    jsrc = (int)(zsrc/dz)  # source location in grid in x-direction

    # Source time function (Gaussian)
    # -------------------------------
    src  = np.zeros(nt + 1)
    time = np.linspace(0 * dt, nt * dt, nt)

    # 1st derivative of Gaussian
    src  = -2. * (time - t0) * (f0 ** 2) * (np.exp(- (f0 ** 2) * (time - t0) ** 2))

    # define clip value: 0.1 * absolute maximum value of source wavelet
    clip = 0.1 * max([np.abs(src.min()), np.abs(src.max())]) / (dx*dz) * dt**2

    # Define absorbing boundary frame
    # -------------------------------
    absorb_coeff = absorb(nx,nz)

    # Define squared vp-model
    # -----------------------
    vp2 = vp**2

    # Initialize empty pressure arrays
    # --------------------------------
    p    = np.zeros((nx,nz)) # p at time n (now)
    pold = np.zeros((nx,nz)) # p at time n-1 (past)
    pnew = np.zeros((nx,nz)) # p at time n+1 (present)
    d2px = np.zeros((nx,nz)) # 2nd spatial x-derivative of p
    d2pz = np.zeros((nx,nz)) # 2nd spatial z-derivative of p

    # INITIALIZE ARRAYS FOR REAL AND IMAGINARY PARTS OF MONOCHROMATIC WAVEFIELDS HERE!
    # --------------------------------------------------------------------------------
      # real part of the monochromatic wavefield
      # imaginary part of the monochromatic wavefield

    # Initalize animation of pressure wavefield
    # -----------------------------------------
    fig = plt.figure(figsize=(7,3))  # define figure size
    extent = [0.0,xmax,zmax,0.0]     # define model extension

    # Plot Vp-model
    image =...


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/7_fdac2d_sensitivity_kernels.ipynb".
The first task is:
Modelling monochromatic frequency domain wavefields for a homogeneous acoustic medium
Now, everything is assembled to compute frequency domain wavefields. We only have to define the discrete frequency freq for which the monochromatic wavefields should be calculated. Let's start with a homogeneous model:
Can you write Python code for it?

# Run 2D acoustic FD modelling with 3-point spatial operater
# ----------------------------------------------------------
%matplotlib notebook
op = 3  # define spatial FD operator (3-point)

# define homogeneous model with vp = 2500 m/s

# Define time step
dt =  dx / (np.sqrt(2) * np.max(vp_hom))# time step (s)

p_hom_re, p_hom_im = FD_2D_acoustic_JIT(vp_hom,dt,dx,dz,f0,xsrc,zsrc,op,freq)


I am working on the file "05_2D_acoustic_FD_modelling/lecture_notebooks/7_fdac2d_sensitivity_kernels.ipynb".
The first task is:
Sensitivity Kernels
The monochromatic frequency domain wavefields are the key to calculate sensitivty kernels, which will be computed in the next exercise ...
Exercise
Compute 5 Hz frequency domain sensitivity kernels
\begin{equation}
K(x,z,\omega) = {\cal{Re}}\biggl{\frac{2 \omega^2}{Vp^3} \frac{\tilde{P}(x,z,\omega,\mathbf{x_{s}}) \tilde{G}(x,z,\omega,\mathbf{x_{r}})}{max(\tilde{P}(x,z,\omega,\mathbf{x_{s}}))}\biggr} \notag
\end{equation}
for the homogenous and FATT Marmousi-2 model.

The frequency domain forward wavefields $\tilde{P}(x,z,\omega,\mathbf{x_{s}})$ for a source at $x_{s} = 2000.0\; m$ and  $z_{s} = 40.0\; m$ where already computated in the previous sections of the notebook (p_hom_re, p_hom_im,p_fatt_re, p_fatt_im). You only have to compute the receiver Green's functions $\tilde{G}(x,z,\omega,\mathbf{x_{r}})$ by placing a source at the receiver position $x_{r} = 8000.0\; m$ and  $z_{r} = 40.0\; m$
Compute the sensitivity kernels $K(x,z,\omega)$. Hint: In Python complex numbers are defined as real_part + 1j*imag_part. This can also be applied to NumPy arrays.
Plot, describe and interpret the sensitivity kernels for the homogeneous and Marmousi-2 FATT model. Where would you expect model updates in a subsequent FWI?
Can you write Python code for it?

# COMPUTE RECEIVER GREEN'S FUNCTION FOR HOMOGENEOUS MODEL HERE!
# -------------------------------------------------------------
%matplotlib notebook
op = 3  # define spatial FD operator (3-point)

# Define time step
dt =  dx / (np.sqrt(2) * np.max(vp_hom))# time step (s)

g_hom_re, g_hom_im = FD_2D_acoustic_JIT(vp_hom,dt,dx,dz,f0,xsrc,zsrc,op,freq)

%matplotlib notebook

# COMPUTE SENSITVITY KERNEL FOR HOMOGENEOUS MODEL HERE!

clip = 4e-18
extent = [0.0,xmax/1000,zmax/1000,0.0] # define model extension

fig = plt.figure(figsize=(7,3))  # define figure size

# Plot Vp-model
image = plt.imshow((vp_hom.T)/1000, cmap=plt.cm.gray, interpolation='nearest',
                    extent=extent)

# Plot Sensitivity Kernel
image1 = plt.imshow(K_hom.T, cmap="RdBu", alpha=.75, extent=extent,
                      interpolation='nearest', vmin=-clip, vmax=clip)
plt.title('Sensitivity Kernel (homogeneous model)')
plt.xlabel('x [km]')
plt.ylabel('z [km]')

plt.show()

# COMPUTE RECEIVER GREEN'S FUNCTION FOR MARMOUSI-2 FATT MODEL HERE!
# -----------------------------------------------------------------
%matplotlib notebook
op = 3  # define spatial FD operator (3-point)

# Define time step
dt =  dx / (np.sqrt(2) * np.max(vp_fatt))# time step (s)

g_fatt_re, g_fatt_im = FD_2D_acoustic_JIT(vp_fatt,dt,dx,dz,f0,xsrc,zsrc,op,freq)

%matplotlib notebook

# COMPUTE SENSITVITY KERNEL FOR Marmousi-2 FATT MODEL HERE!

clip = 8e-17
extent = [0.0,xmax/1000,zmax/1000,0.0] # define model extension

fig = plt.figure(figsize=(7,3))  # define figure size

# Plot Vp-model
image = plt.imshow((vp_fatt.T)/1000, cmap=plt.cm.gray, interpolation='nearest',
                    extent=extent)

# Plot Sensitivity Kernel
image1 = plt.imshow(K_hom.T, cmap="RdBu", alpha=.5, extent=extent,
                      interpolation='nearest', vmin=-clip, vmax=clip)
plt.title('Sensitivity Kernel (Marmousi-2 FATT model)')
plt.xlabel('x [km]')
plt.ylabel('z [km]')

plt.show()


I am working on the file "05_Autoencoding/02.0 - Dimensionality Reduction - Autoencoding + Custom Estimator.ipynb".
The first task is:
1. Define Dataset Metadata
Can you write Python code for it?

FEATURE_COUNT = 64

HEADER = ['key']
HEADER_DEFAULTS = [[0]]
UNUSED_FEATURE_NAMES = ['key']
CLASS_FEATURE_NAME = 'CLASS'
FEATURE_NAMES = []

for i in range(FEATURE_COUNT):
    HEADER += ['x_{}'.format(str(i+1))]
    FEATURE_NAMES += ['x_{}'.format(str(i+1))]
    HEADER_DEFAULTS += [[0.0]]

HEADER += [CLASS_FEATURE_NAME]
HEADER_DEFAULTS += [['NA']]

print("Header: {}".format(HEADER))
print("Features: {}".format(FEATURE_NAMES))
print("Class Feature: {}".format(CLASS_FEATURE_NAME))
print("Unused Features: {}".format(UNUSED_FEATURE_NAMES))


I am working on the file "05_Autoencoding/02.0 - Dimensionality Reduction - Autoencoding + Custom Estimator.ipynb".
The first task is:
2. Define CSV Data Input Function
Can you write Python code for it?

def parse_csv_row(csv_row):

    columns = tf.decode_csv(csv_row, record_defaults=HEADER_DEFAULTS)
    features = dict(zip(HEADER, columns))

    for column in UNUSED_FEATURE_NAMES:
        features.pop(column)

    target = features.pop(CLASS_FEATURE_NAME)

    return features, target

def csv_input_fn(files_name_pattern, mode=tf.estimator.ModeKeys.EVAL,
                 skip_header_lines=0,
                 num_epochs=None,
                 batch_size=200):

    shuffle = True if mode == tf.estimator.ModeKeys.TRAIN else False

    print("")
    print("* data input_fn:")
    print("================")
    print("Input file(s): {}".format(files_name_pattern))
    print("Batch size: {}".format(batch_size))
    print("Epoch Count: {}".format(num_epochs))
    print("Mode: {}".format(mode))
    print("Shuffle: {}".format(shuffle))
    print("================")
    print("")

    file_names = tf.matching_files(files_name_pattern)

    dataset = data.TextLineDataset(filenames=file_names)
    dataset = dataset.skip(skip_header_lines)

    if shuffle:
        dataset = dataset.shuffle(buffer_size=2 * batch_size + 1)

    num_threads = multiprocessing.cpu_count() if MULTI_THREADING else 1

    dataset = dataset.batch(batch_size)
    dataset = dataset.map(lambda csv_row: parse_csv_row(csv_row), num_parallel_calls=num_threads)

    dataset = dataset.repeat(num_epochs)
    iterator = dataset.make_one_shot_iterator()

    features, target = iterator.get_next()

    return features, target

features, target = csv_input_fn(files_name_pattern="")
print("Feature read from CSV: {}".format(list(features.keys())))
print("Target read from CSV: {}".format(target))


I am working on the file "05_Autoencoding/02.0 - Dimensionality Reduction - Autoencoding + Custom Estimator.ipynb".
The first task is:
3. Define Feature Columns
Can you write Python code for it?

def get_feature_columns():

    feature_columns = {feature_name: tf.feature_column.numeric_column(feature_name)
                       for feature_name in FEATURE_NAMES}

    return feature_columns


I am working on the file "05_Autoencoding/02.0 - Dimensionality Reduction - Autoencoding + Custom Estimator.ipynb".
The first task is:
6. Use the trained model to encode data (prediction)
Can you write Python code for it?

import itertools

DATA_SIZE = 2000

input_fn = lambda: csv_input_fn(
    TRAIN_DATA_FILES_PATTERN,
    mode=tf.contrib.learn.ModeKeys.INFER,
    num_epochs=1,
    batch_size=500
)

estimator = create_estimator(run_config, hparams)

predictions = estimator.predict(input_fn=input_fn)
predictions = itertools.islice(predictions, DATA_SIZE)
predictions = list(map(lambda item: list(item["encoding"]), predictions))

print(predictions[:5])


I am working on the file "05_Autoencoding/02.0 - Dimensionality Reduction - Autoencoding + Custom Estimator.ipynb".
The first task is:
TF Custom Estimator to Build a NN Autoencoder for Feature Extraction
Can you write Python code for it?

MODEL_NAME = 'auto-encoder-01'

TRAIN_DATA_FILES_PATTERN = 'data/data-*.csv'

RESUME_TRAINING = False

MULTI_THREADING = True


I am working on the file "05_Autoencoding/02.0 - Dimensionality Reduction - Autoencoding + Custom Estimator.ipynb".
The first task is:
b. Set the parameters
Can you write Python code for it?

TRAIN_SIZE = 2000
NUM_EPOCHS = 1000
BATCH_SIZE = 100
NUM_EVAL = 10
CHECKPOINT_STEPS = int((TRAIN_SIZE/BATCH_SIZE) * (NUM_EPOCHS/NUM_EVAL))

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    encoder_hidden_units=[30,3],
    learning_rate = 0.01,
    l2_reg = 0.0001,
    noise_level = 0.0,
    dropout_rate = 0.1)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.contrib.learn.RunConfig(
    save_checkpoints_steps=CHECKPOINT_STEPS,
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS)
print("Required Evaluation Steps:", NUM_EVAL)
print("That is 1 evaluation step after each",NUM_EPOCHS/NUM_EVAL," epochs")
print("Save Checkpoint After",CHECKPOINT_STEPS,"steps")


I am working on the file "05_Autoencoding/02.0 - Dimensionality Reduction - Autoencoding + Custom Estimator.ipynb".
The first task is:
c. Run the experiment using learn_runner
Can you write Python code for it?

if not RESUME_TRAINING:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...")

tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow()
print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................")

learn_runner.run(
    experiment_fn=generate_experiment_fn(
    ),
    run_config=run_config,
    schedule="train_and_evaluate", #'train_and_evaluate'
    hparams=hparams
)

time_end = datetime.utcnow()
print(".......................................")
print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "05_Autoencoding/03.0 - Dimensionality Reduction - Autoencoding + Normalizer + XEntropy Loss.ipynb".
The first task is:
1. Define Dataset Metadata
Can you write Python code for it?


I am working on the file "05_Autoencoding/03.0 - Dimensionality Reduction - Autoencoding + Normalizer + XEntropy Loss.ipynb".
The first task is:
3. Define Feature Columns
a. Load normalizarion params
Can you write Python code for it?

df_params = pd.read_csv("data/params.csv", header=0, index_col=0)
len(df_params)
df_params['feature_name'] = FEATURE_NAMES
df_params.head()


I am working on the file "05_Autoencoding/03.0 - Dimensionality Reduction - Autoencoding + Normalizer + XEntropy Loss.ipynb".
The first task is:
4. Define Autoencoder Model Function
Can you write Python code for it?

def autoencoder_model_fn(features, labels, mode, params):

    feature_columns = list(get_feature_columns().values())

    input_layer_size = len(feature_columns)

    encoder_hidden_units = params.encoder_hidden_units

    # decoder units are the reverse of the encoder units, without the middle layer (redundant)
    decoder_hidden_units = encoder_hidden_units.copy()
    decoder_hidden_units.reverse()
    decoder_hidden_units.pop(0)

    output_layer_size = len(FEATURE_NAMES)

    he_initialiser = tf.contrib.layers.variance_scaling_initializer()
    l2_regulariser = tf.contrib.layers.l2_regularizer(scale=params.l2_reg)

    print("[{}]->{}-{}->[{}]".format(len(feature_columns)
                                     ,encoder_hidden_units
                                     ,decoder_hidden_units,
                                     output_layer_size))

    is_training = (mode == tf.estimator.ModeKeys.TRAIN)

    # input layer
    input_layer = tf.feature_column.input_layer(features=features,
                                                feature_columns=feature_columns)

    # Adding Gaussian Noise to input layer
    noisy_input_layer = input_layer + (params.noise_level * tf.random_normal(tf.shape(input_layer)))

    # Dropout layer
    dropout_layer = tf.layers.dropout(inputs=noisy_input_layer,
                                     rate=params.dropout_rate,
                                     training=is_training)

#     # Dropout layer without Gaussian Nosing
#     dropout_layer = tf.layers.dropout(inputs=input_layer,
#                                       rate=params.dropout_rate,
#                                       training=is_training)

    # Encoder layers stack
    encoding_hidden_layers = tf.contrib.layers.stack(inputs= dropout_layer,
                                                     layer= tf.contrib.layers.fully_connected,
                                                     stack_args=encoder_hidden_units,
                                    ...


I am working on the file "05_Autoencoding/03.0 - Dimensionality Reduction - Autoencoding + Normalizer + XEntropy Loss.ipynb".
The first task is:
5. Run Experiment using Estimator Train_And_Evaluate
a. Set the parameters
Can you write Python code for it?

TRAIN_SIZE = 2000
NUM_EPOCHS = 1000
BATCH_SIZE = 100
NUM_EVAL = 10

TOTAL_STEPS = (TRAIN_SIZE/BATCH_SIZE)*NUM_EPOCHS
CHECKPOINT_STEPS = int((TRAIN_SIZE/BATCH_SIZE) * (NUM_EPOCHS/NUM_EVAL))

hparams  = tf.contrib.training.HParams(
    num_epochs = NUM_EPOCHS,
    batch_size = BATCH_SIZE,
    encoder_hidden_units=[30,3],
    learning_rate = 0.01,
    l2_reg = 0.0001,
    noise_level = 0.0,
    max_steps = TOTAL_STEPS,
    dropout_rate = 0.05
)

model_dir = 'trained_models/{}'.format(MODEL_NAME)

run_config = tf.contrib.learn.RunConfig(
    save_checkpoints_steps=CHECKPOINT_STEPS,
    tf_random_seed=19830610,
    model_dir=model_dir
)

print(hparams)
print("Model Directory:", run_config.model_dir)
print("")
print("Dataset Size:", TRAIN_SIZE)
print("Batch Size:", BATCH_SIZE)
print("Steps per Epoch:",TRAIN_SIZE/BATCH_SIZE)
print("Total Steps:", TOTAL_STEPS)
print("Required Evaluation Steps:", NUM_EVAL)
print("That is 1 evaluation step after each",NUM_EPOCHS/NUM_EVAL," epochs")
print("Save Checkpoint After",CHECKPOINT_STEPS,"steps")


I am working on the file "05_Autoencoding/03.0 - Dimensionality Reduction - Autoencoding + Normalizer + XEntropy Loss.ipynb".
The first task is:
b. Create normalized feature columns
Can you write Python code for it?

def standard_scaler(x, mean, stdv):
    return (x-mean)/stdv

def maxmin_scaler(x, max_value, min_value):
    return (x-min_value)/(max_value-min_value)

def get_feature_columns():

    feature_columns = {}

#     feature_columns = {feature_name: tf.feature_column.numeric_column(feature_name)
#                        for feature_name in FEATURE_NAMES}

    for feature_name in FEATURE_NAMES:

        feature_max = df_params[df_params.feature_name == feature_name]['max'].values[0]
        feature_min = df_params[df_params.feature_name == feature_name]['min'].values[0]
        normalizer_fn = lambda x: maxmin_scaler(x, feature_max, feature_min)

        feature_columns[feature_name] = tf.feature_column.numeric_column(feature_name,
                                                                         normalizer_fn=normalizer_fn
                                                                        )

    return feature_columns

print(get_feature_columns())


I am working on the file "05_Autoencoding/03.0 - Dimensionality Reduction - Autoencoding + Normalizer + XEntropy Loss.ipynb".
The first task is:
b. Define TrainSpec and EvaluSpec
Can you write Python code for it?

train_spec = tf.estimator.TrainSpec(
    input_fn = lambda: csv_input_fn(
        TRAIN_DATA_FILES_PATTERN,
        mode = tf.contrib.learn.ModeKeys.TRAIN,
        num_epochs=hparams.num_epochs,
        batch_size=hparams.batch_size
    ),
    max_steps=hparams.max_steps,
    hooks=None
)

eval_spec = tf.estimator.EvalSpec(
    input_fn = lambda: csv_input_fn(
        TRAIN_DATA_FILES_PATTERN,
        mode=tf.contrib.learn.ModeKeys.EVAL,
        num_epochs=1,
        batch_size=hparams.batch_size
    ),
#     exporters=[tf.estimator.LatestExporter(
#         name="encode",  # the name of the folder in which the model will be exported to under export
#         serving_input_receiver_fn=csv_serving_input_fn,
#         exports_to_keep=1,
#         as_text=True)],
    steps=None,
    hooks=None
)


I am working on the file "05_Autoencoding/03.0 - Dimensionality Reduction - Autoencoding + Normalizer + XEntropy Loss.ipynb".
The first task is:
d. Run Experiment via train_and_evaluate
Can you write Python code for it?

if not RESUME_TRAINING:
    print("Removing previous artifacts...")
    shutil.rmtree(model_dir, ignore_errors=True)
else:
    print("Resuming training...")

tf.logging.set_verbosity(tf.logging.INFO)

time_start = datetime.utcnow()
print("Experiment started at {}".format(time_start.strftime("%H:%M:%S")))
print(".......................................")

estimator = create_estimator(run_config, hparams)

tf.estimator.train_and_evaluate(
    estimator=estimator,
    train_spec=train_spec,
    eval_spec=eval_spec
)

time_end = datetime.utcnow()
print(".......................................")
print("Experiment finished at {}".format(time_end.strftime("%H:%M:%S")))
print("")
time_elapsed = time_end - time_start
print("Experiment elapsed time: {} seconds".format(time_elapsed.total_seconds()))


I am working on the file "05_BeyondArcPy/notebooks/ACS5.ipynb".
The first task is:
2012 Survey of Business Owners: Company Summary
https://www.census.gov/data/developers/data-sets/business-owners.html

Variables: api.census.gov/data/2012/sbo
Can you write Python code for it?

#Get list of variables
dfVars = pd.read_json('http://api.census.gov/data/2012/sbo')
dfVars.dataset.values[0]['description']

endpointURL = 'api.census.gov/data/2012/sbo'


I am working on the file "05_BeyondArcPy/notebooks/ACS5.ipynb".
The first task is:
American Community Survey 5-Year Data (2009-2015)
https://www.census.gov/data/developers/data-sets/acs-5year.html<br>
https://api.census.gov/data/2015/acs5/examples.html
Can you write Python code for it?

#Endpoint and payload
baseURL = 'https://api.census.gov/data/2015/acs5'
payload = {'get':'NAME,B01001_001E',
           'for':'county:*',
           'in':'state:37',
           'key':key
          }

response = requests.get(baseURL,payload)

t = response.text
c = response.content
print(type(t), type(c))

print t


I am working on the file "05_CorrectiveSolutions-6nodes-2loops.ipynb".
The first task is:
Compute $\tilde{K}$ and $\tilde{M}$
Can you write Python code for it?

S1.compute_KM(A,h,t)

S1.Ktilde

S1.Mtilde.subs(datav)


I am working on the file "05_CorrectiveSolutions-6nodes-2loops.ipynb".
The first task is:
Compute H matrix
Can you write Python code for it?


I am working on the file "05_CorrectiveSolutions-6nodes-2loops.ipynb".
The first task is:
Compute eigenvalues and eigenvectors: results are in the form:
- eigenvalue
- multiplicity
- eigenvector
Can you write Python code for it?

sol_data = (S1.Ktilde.inv()*(S1.Mtilde.subs(datav))).eigenvects()
sol_data


I am working on the file "05_CorrectiveSolutions-6nodes-2loops.ipynb".
The first task is:
Compute numerical value of $\lambda$
Can you write Python code for it?

λ = [sympy.N(sympy.sqrt(E*A*h/(G*t)*βi).subs(datav)) for βi in β2]
λ


I am working on the file "05_CorrectiveSolutions-6nodes-2loops.ipynb".
The first task is:
Define section and perform first calculations
Can you write Python code for it?

S1 = Section(stringers, panels)

S1.cycles


I am working on the file "05_CorrectiveSolutions-6nodes-2loops.ipynb".
The first task is:
Define sympy parameters used for geometric description of sections
Can you write Python code for it?

A, A0, t, t0, a, b, h, L, E, G = sympy.symbols('A A_0 t t_0 a b h L E G', positive=True)


I am working on the file "05_CorrectiveSolutions-6nodes-2loops.ipynb".
The first task is:
Expression of inertial properties in principal reference frame
Can you write Python code for it?

sympy.simplify(S1.Ixx), sympy.simplify(S1.Iyy), sympy.simplify(S1.Ixy), sympy.simplify(S1.θ)

S1.symmetry


I am working on the file "05_CorrectiveSolutions-6nodes-2loops.ipynb".
The first task is:
Extract and normalize eigenvectors
Can you write Python code for it?

X = [sol[2][0]/sol[2][0].norm() for sol in sol_data]
X


I am working on the file "05_CorrectiveSolutions-6nodes-2loops.ipynb".
The first task is:
Extract eigenvalues
Can you write Python code for it?


I am working on the file "05_CorrectiveSolutions-6nodes-2loops.ipynb".
The first task is:
Initialization of sympy symbolic tool and pint for dimension analysis (not really implemented rn as not directly compatible with sympy)
Can you write Python code for it?

ureg = UnitRegistry()
sympy.init_printing()


I am working on the file "05_CorrectiveSolutions-6nodes-2loops.ipynb".
The first task is:
Plot of S1 section in inertial reference Frame
Section is plotted wrt center of gravity and rotated (if necessary) so that x and y are principal axes.
Center of Gravity and Shear Center are drawn
Can you write Python code for it?

positions={ii: [float(S1.g.node[ii]['pos'][i].subs(datav)) for i in range(2)] for ii in S1.g.nodes() }

x_ct, y_ct = S1.ct.subs(datav)

plt.figure(figsize=(12,8),dpi=300)
nx.draw(S1.g,with_labels=True, pos=positions)
plt.plot([0],[0],'o',ms=12,label='CG')
plt.plot([x_ct],[y_ct],'^',ms=12, label='SC')
#plt.text(0,0, 'CG', fontsize=24)
#plt.text(x_ct,y_ct, 'SC', fontsize=24)
plt.legend(loc='lower right', shadow=True)
plt.axis('equal')
plt.title("Section in pricipal reference Frame",fontsize=16);


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Best neural network
Now we find the single neural network that performed best on the test-set.
First list the classification accuracies on the test-set for all the neural networks in the ensemble.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Boolean array whether each image in the test-set was correctly classified by the best single network and incorrectly classified by the ensemble.
Can you write Python code for it?

best_net_better = np.logical_and(best_net_correct,
                                 ensemble_incorrect)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Boolean array whether each image in the test-set was correctly classified by the ensemble and incorrectly classified by the best neural network.
Can you write Python code for it?

ensemble_better = np.logical_and(best_net_incorrect,
                                 ensemble_correct)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Boolean array whether each image is incorrectly classified.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Boolean array whether each of the images in the test-set was correctly classified by the ensemble of neural networks.
Can you write Python code for it?

ensemble_correct = (ensemble_cls_pred == data.test.cls)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Boolean array whether the best neural network classified each image in the test-set correctly.
Can you write Python code for it?

best_net_correct = (best_net_cls_pred == data.test.cls)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Calculate a boolean array whether the images in the test-set are classified correctly.
Can you write Python code for it?

def test_correct():
    return correct_prediction(images = data.test.images,
                              labels = data.test.labels,
                              cls_true = data.test.cls)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Calculate a boolean array whether the images in the validation-set are classified correctly.
Can you write Python code for it?

def validation_correct():
    return correct_prediction(images = data.validation.images,
                              labels = data.validation.labels,
                              cls_true = data.validation.cls)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Calculate the classification accuracy on the original validation-set.
Can you write Python code for it?

def validation_accuracy():
    # Get the array of booleans whether the classifications are correct
    # for the validation-set.
    correct = validation_correct()

    # Calculate the classification accuracy and return it.
    return classification_accuracy(correct)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Calculate the classification accuracy on the test-set.
Can you write Python code for it?

def test_accuracy():
    # Get the array of booleans whether the classifications are correct
    # for the test-set.
    correct = test_correct()

    # Calculate the classification accuracy and return it.
    return classification_accuracy(correct)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Check that the shape of the combined arrays is correct.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Class numbers
The class-labels are One-Hot encoded, which means that each label is a vector with 10 elements, all of which are zero except for one element. The index of this one element is the class-number, that is, the digit shown in the associated image. We also need the class-numbers as integers for the test- and validation-sets, so we calculate them now.
Can you write Python code for it?

data.test.cls = np.argmax(data.test.labels, axis=1)
data.validation.cls = np.argmax(data.validation.labels, axis=1)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Comparison of ensemble vs. the best single network
The number of images in the test-set that were correctly classified by the ensemble.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Create ensemble of neural networks
Number of neural networks in the ensemble.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Create the directory if it does not exist.
Can you write Python code for it?

if not os.path.exists(save_dir):
    os.makedirs(save_dir)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Create the ensemble of neural networks. All networks use the same TensorFlow graph that was defined above. For each neural network the TensorFlow weights and variables are initialized to random values and then optimized. The variables are then saved to disk so they can be reloaded later.
You may want to skip this computation if you just want to re-run the Notebook with different analysis of the results.
Can you write Python code for it?

if True:
    # For each of the neural networks.
    for i in range(num_networks):
        print("Neural network: {0}".format(i))

        # Create a random training-set. Ignore the validation-set.
        x_train, y_train, _, _ = random_training_set()

        # Initialize the variables of the TensorFlow graph.
        session.run(tf.global_variables_initializer())

        # Optimize the variables using this training-set.
        optimize(num_iterations=num_iterations,
                 x_train=x_train,
                 y_train=y_train)

        # Save the optimized variables to disk.
        saver.save(sess=session, save_path=get_save_path(i))

        # Print newline.
        print()


if True:
    # For each of the neural networks.
    for i in range(num_networks):
        print("Neural network: {0}".format(i))

        # Create a random training-set. Ignore the validation-set.
        x_train, y_train, _, _ = random_training_set()

        # Initialize the variables of the TensorFlow graph.
        session.run(tf.initialize_all_variables())

        # Optimize the variables using this training-set.
        optimize(num_iterations=num_iterations,
                 x_train=x_train,
                 y_train=y_train)

        # Save the optimized variables to disk.
        saver.save(sess=session, save_path=get_save_path(i))

        # Print newline.
        print()


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Data Dimensions
The data dimensions are used in several places in the source-code below. They are defined once so we can use these variables instead of numbers throughout the source-code below.
Can you write Python code for it?

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
img_shape = (img_size, img_size)

# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = 1

# Number of classes, one class for each of 10 digits.
num_classes = 10


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Define the size of the training-set used for each neural network. You can try and change this.
Can you write Python code for it?

train_size = int(0.8 * combined_size)
train_size


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Ensemble predictions
There are different ways to calculate the predicted labels for the ensemble. One way is to calculate the predicted class-number for each neural network, and then select the class-number with most votes. But this requires a large number of neural networks relative to the number of classes.
The method used here is instead to take the average of the predicted labels for all the networks in the ensemble. This is simple to calculate and does not require a large number of networks in the ensemble.
Can you write Python code for it?

ensemble_pred_labels = np.mean(pred_labels, axis=0)
ensemble_pred_labels.shape


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Examples: Ensemble is better than the best network
Plot examples of images that were correctly classified by the ensemble and incorrectly classified by the best single network.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Function for printing the predicted labels for the best single network.
Can you write Python code for it?

def print_labels_best_net(idx, **kwargs):
    print_labels(labels=best_net_pred_labels, idx=idx, **kwargs)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Function for printing the predicted labels for the ensemble of neural networks.
Can you write Python code for it?

def print_labels_ensemble(idx, **kwargs):
    print_labels(labels=ensemble_pred_labels, idx=idx, **kwargs)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Function for printing the predicted labels of all the neural networks in the ensemble. This only prints the labels for the first image.
Can you write Python code for it?

def print_labels_all_nets(idx):
    for i in range(num_networks):
        print_labels(labels=pred_labels[i, :, :], idx=idx, num=1)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Function for printing the predicted labels.
Can you write Python code for it?

def print_labels(labels, idx, num=1):
    # Select the relevant labels based on idx.
    labels = labels[idx, :]

    # Select the first num labels.
    labels = labels[0:num, :]

    # Round numbers to 2 decimal points so they are easier to read.
    labels_rounded = np.round(labels, 2)

    # Print the rounded labels.
    print(labels_rounded)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Function for selecting a random training-batch of the given size.
Can you write Python code for it?

def random_batch(x_train, y_train):
    # Total number of images in the training-set.
    num_images = len(x_train)

    # Create a random index into the training-set.
    idx = np.random.choice(num_images,
                           size=train_batch_size,
                           replace=False)

    # Use the random index to select random images and labels.
    x_batch = x_train[idx, :]  # Images.
    y_batch = y_train[idx, :]  # Labels.

    # Return the batch.
    return x_batch, y_batch


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Helper-function for creating random training-sets
We will train 5 neural networks on different training-sets that are selected at random. First we combine the original training- and validation-sets into one big set. This is done for both the images and the labels.
Can you write Python code for it?

combined_images = np.concatenate([data.train.images, data.validation.images], axis=0)
combined_labels = np.concatenate([data.train.labels, data.validation.labels], axis=0)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Helper-function for splitting the combined data-set into a random training- and validation-set.
Can you write Python code for it?

def random_training_set():
    # Create a randomized index into the full / combined training-set.
    idx = np.random.permutation(combined_size)

    # Split the random index into training- and validation-sets.
    idx_train = idx[0:train_size]
    idx_validation = idx[train_size:]

    # Select the images and labels for the new training-set.
    x_train = combined_images[idx_train, :]
    y_train = combined_labels[idx_train, :]

    # Select the images and labels for the new validation-set.
    x_validation = combined_images[idx_validation, :]
    y_validation = combined_labels[idx_validation, :]

    # Return the new training- and validation-sets.
    return x_train, y_train, x_validation, y_validation


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Helper-function to create a random training batch.
There are thousands of images in the training-set. It takes a long time to calculate the gradient of the model using all these images. We therefore only use a small batch of images in each iteration of the optimizer.
If your computer crashes or becomes very slow because you run out of RAM, then you may try and lower this number, but you may then need to perform more optimization iterations.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Helper-function to perform optimization iterations
Function for performing a number of optimization iterations so as to gradually improve the variables of the network layers. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples.  The progress is printed every 100 iterations.
Can you write Python code for it?

def optimize(num_iterations, x_train, y_train):
    # Start-time used for printing time-usage below.
    start_time = time.time()

    for i in range(num_iterations):

        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = random_batch(x_train, y_train)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)

        # Print status every 100 iterations and after last iteration.
        if i % 100 == 0:

            # Calculate the accuracy on the training-batch.
            acc = session.run(accuracy, feed_dict=feed_dict_train)

            # Status-message for printing.
            msg = "Optimization Iteration: {0:>6}, Training Batch Accuracy: {1:>6.1%}"

            # Print it.
            print(msg.format(i + 1, acc))

    # Ending time.
    end_time = time.time()

    # Difference between start and end-times.
    time_dif = end_time - start_time

    # Print the time-usage.
    print("Time usage: " + str(timedelta(seconds=int(round(time_dif)))))


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Helper-functions for calculating and predicting classifications
This function calculates the predicted labels of images, that is, for each image it calculates a vector of length 10 indicating which of the 10 classes the image is.
The calculation is done in batches because it might use too much RAM otherwise. If your computer crashes then you can try and lower the batch-size.
Can you write Python code for it?

# Split the data-set in batches of this size to limit RAM usage.
batch_size = 256

def predict_labels(images):
    # Number of images.
    num_images = len(images)

    # Allocate an array for the predicted labels which
    # will be calculated in batches and filled into this array.
    pred_labels = np.zeros(shape=(num_images, num_classes),
                           dtype=np.float)

    # Now calculate the predicted labels for the batches.
    # We will just iterate through all the batches.
    # There might be a more clever and Pythonic way of doing this.

    # The starting index for the next batch is denoted i.
    i = 0

    while i < num_images:
        # The ending index for the next batch is denoted j.
        j = min(i + batch_size, num_images)

        # Create a feed-dict with the images between index i and j.
        feed_dict = {x: images[i:j, :]}

        # Calculate the predicted labels using TensorFlow.
        pred_labels[i:j] = session.run(y_pred, feed_dict=feed_dict)

        # Set the start-index for the next batch to the
        # end-index of the current batch.
        i = j

    return pred_labels


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Helper-functions for calculating the classification accuracy
This function calculates the classification accuracy given a boolean array whether each image was correctly classified. E.g. classification_accuracy([True, True, False, False, False]) = 2/5 = 0.4
Can you write Python code for it?

def classification_accuracy(correct):
    # When averaging a boolean array, False means 0 and True means 1.
    # So we are calculating: number of True / len(correct) which is
    # the same as the classification accuracy.
    return correct.mean()


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Helper-functions for plotting and printing comparisons
Function for plotting images from the test-set and their true and predicted class-numbers.
Can you write Python code for it?

def plot_images_comparison(idx):
    plot_images(images=data.test.images[idx, :],
                cls_true=data.test.cls[idx],
                ensemble_cls_pred=ensemble_cls_pred[idx],
                best_cls_pred=best_net_cls_pred[idx])


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Imports
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
from sklearn.metrics import confusion_matrix
import time
from datetime import timedelta
import math
import os

# Use PrettyTensor to simplify Neural Network construction.
import prettytensor as pt


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Initialize variables
The variables for weights and biases must be initialized before we start optimizing them. We make a simple wrapper-function for this, because we will call it several times below.
Can you write Python code for it?

def init_variables():
    session.run(tf.initialize_all_variables())


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Negate the boolean array so we can use it to lookup incorrectly classified images.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Neural Network
This section implements the Convolutional Neural Network using Pretty Tensor, which is much simpler than a direct implementation in TensorFlow, see Tutorial #03.
The basic idea is to wrap the input tensor x_image in a Pretty Tensor object which has helper-functions for adding new computational layers so as to create an entire neural network. Pretty Tensor takes care of the variable allocation, etc.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Next we have the placeholder variable for the true labels associated with the images that were input in the placeholder variable x. The shape of this placeholder variable is [None, num_classes] which means it may hold an arbitrary number of labels and each label is a vector of length num_classes which is 10 in this case.
Can you write Python code for it?

y_true = tf.placeholder(tf.float32, shape=[None, 10], name='y_true')


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Now that we have wrapped the input image in a Pretty Tensor object, we can add the convolutional and fully-connected layers in just a few lines of source-code.
Note that pt.defaults_scope(activation_fn=tf.nn.relu) makes activation_fn=tf.nn.relu an argument for each of the layers constructed inside the with-block, so that Rectified Linear Units (ReLU) are used for each of these layers. The defaults_scope makes it easy to change arguments for all of the layers.
Can you write Python code for it?

with pt.defaults_scope(activation_fn=tf.nn.relu):
    y_pred, loss = x_pretty.\
        conv2d(kernel=5, depth=16, name='layer_conv1').\
        max_pool(kernel=2, stride=2).\
        conv2d(kernel=5, depth=36, name='layer_conv2').\
        max_pool(kernel=2, stride=2).\
        flatten().\
        fully_connected(size=128, name='layer_fc1').\
        softmax_classifier(class_count=10, labels=y_true)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Number of images in the test-set where the best single network was better than the ensemble.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Number of images in the test-set where the ensemble was better than the best single network:
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Number of optimization iterations for each neural network.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Performance Measures
We need a few more performance measures to display the progress to the user.
First we calculate the predicted class number from the output of the neural network y_pred, which is a vector with 10 elements. The class number is the index of the largest element.
Can you write Python code for it?

y_pred_cls = tf.argmax(y_pred, dimension=1)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Predicted labels of the best neural network.
Can you write Python code for it?

best_net_pred_labels = pred_labels[best_net, :, :]


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
PrettyTensor version:
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Saver
In order to save the variables of the neural network, we now create a Saver-object which is used for storing and retrieving all the variables of the TensorFlow graph. Nothing is actually saved at this point, which will be done further below.
Note that if you have more than 100 neural networks in the ensemble then you must increase max_to_keep accordingly.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Size of the combined data-set.
Can you write Python code for it?

combined_size = len(combined_images)
combined_size


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Summarize the classification accuracies on the test-set for the neural networks in the ensemble.
Can you write Python code for it?

print("Mean test-set accuracy: {0:.4f}".format(np.mean(test_accuracies)))
print("Min test-set accuracy:  {0:.4f}".format(np.min(test_accuracies)))
print("Max test-set accuracy:  {0:.4f}".format(np.max(test_accuracies)))


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
TensorFlow Graph
The entire purpose of TensorFlow is to have a so-called computational graph that can be executed much more efficiently than if the same calculations were to be performed directly in Python. TensorFlow can be more efficient than NumPy because TensorFlow knows the entire computation graph that must be executed, while NumPy only knows the computation of a single mathematical operation at a time.
TensorFlow can also automatically calculate the gradients that are needed to optimize the variables of the graph so as to make the model perform better. This is because the graph is a combination of simple mathematical expressions so the gradient of the entire graph can be calculated using the chain-rule for derivatives.
TensorFlow can also take advantage of multi-core CPUs as well as GPUs - and Google has even built special chips just for TensorFlow which are called TPUs (Tensor Processing Units) and are even faster than GPUs.
A TensorFlow graph consists of the following parts which will be detailed below:

Placeholder variables used for inputting data to the graph.
Variables that are going to be optimized so as to make the convolutional network perform better.
The mathematical formulas for the neural network.
A loss measure that can be used to guide the optimization of the variables.
An optimization method which updates the variables.

In addition, the TensorFlow graph may also contain various debugging statements e.g. for logging data to be displayed using TensorBoard, which is not covered in this tutorial.
Placeholder variables
Placeholder variables serve as the input to the TensorFlow computational graph that we may change each time we execute the graph. We call this feeding the placeholder variables and it is demonstrated further below.
First we define the placeholder variable for the input images. This allows us to change the images that are input to the TensorFlow graph. This is a so-called tensor, which just means that it is a multi-dimensional array. The data-type is set to float32 and the...
Can you write Python code for it?

x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
TensorFlow Run
Create TensorFlow session
Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The MNIST data-set has now been loaded and consists of 70,000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets, but we will make random training-sets further below.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validation-set:\t{}".format(len(data.validation.labels)))


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The best network's predicted labels for the first of these images:
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The best neural network's classification accuracy on the test-set.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The best single network's predicted labels for the first of these images:
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The classification accuracy is calculated by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then taking the average of these numbers.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The convolutional layers expect x to be encoded as a 4-dim tensor so we have to reshape it so its shape is instead [num_images, img_height, img_width, num_channels]. Note that img_height == img_width == img_size and num_images can be inferred automatically by using -1 for the size of the first dimension. So the reshape operation is:
Can you write Python code for it?

x_image = tf.reshape(x, [-1, img_size, img_size, num_channels])


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The ensemble's predicted class number is then the index of the highest number in the label, which is calculated using argmax as usual.
Can you write Python code for it?

ensemble_cls_pred = np.argmax(ensemble_pred_labels, axis=1)
ensemble_cls_pred.shape


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The ensemble's predicted labels for the first of these images (top left image):
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The index of the neural network with the highest classification accuracy.
Can you write Python code for it?

best_net = np.argmax(test_accuracies)
best_net


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The number of images in the test-set that were correctly classified by the best neural network.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The predicted class-number.
Can you write Python code for it?

best_net_cls_pred = np.argmax(best_net_pred_labels, axis=1)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The predicted labels of all the networks in the ensemble, for the first of these images:
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
The predicted labels of the ensemble is a 3-dim array, the first dim is the network-number, the second dim is the image-number, the third dim is the classification vector.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
Then we create a vector of booleans telling us whether the predicted class equals the true class of each image.
Can you write Python code for it?

correct_prediction = tf.equal(y_pred_cls, y_true_cls)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
This function returns the save-path for the data-file with the given network number.
Can you write Python code for it?

def get_save_path(net_number):
    return save_dir + 'network' + str(net_number)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
This is the directory used for saving and retrieving the data.
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
This was developed using Python 3.5.2 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
We could also have a placeholder variable for the class-number, but we will instead calculate it using argmax. Note that this is a TensorFlow operator so nothing is calculated at this point.
Can you write Python code for it?

y_true_cls = tf.argmax(y_true, dimension=1)


I am working on the file "05_Ensemble_Learning.ipynb".
The first task is:
We do not use a validation-set during training, but this would be the size.
Can you write Python code for it?

validation_size = combined_size - train_size
validation_size


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
10000次优化迭代后的结果
Can you write Python code for it?

optimize(num_iterations=9000) # We performed 1000 iterations above.

print_validation_accuracy(show_example_errors=True, show_confusion_matrix=True)


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
1000次优化迭代后的结果
Can you write Python code for it?

optimize(num_iterations=900)  # We performed 100 iterations above.

print_validation_accuracy(show_example_errors=True)


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
100次优化迭代后的结果
After 100 optimization iterations, the model should have significantly improved its classification accuracy.
Can you write Python code for it?

optimize(num_iterations=99)  # We already performed 1 iteration above.

print_validation_accuracy(show_example_errors=True)


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
Check that the output of the fully-connected layer is a tensor with shape (?, 128) where the ? means there is an arbitrary number of images and fc_size == 128.
Can you write Python code for it?


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
It can be difficult to understand and keep track of how these filters are applied because of the high dimensionality.
Applying these convolutional filters to the images that were ouput from the first conv-layer gives the following images.
Note that these are down-sampled yet again to half the resolution of the images from the first conv-layer.
Can you write Python code for it?


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
Plot another example image from the test-set.
Can you write Python code for it?

image2 = test_images[13]
plot_image(image2)


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
TensorFlow 图模型
主要包含以下部分：
* Placeholder variables used for inputting data to the graph.
* Variables that are going to be optimized so as to make the convolutional network perform better.
* The mathematical formulas for the convolutional network.
* A cost measure that can be used to guide the optimization of the variables.
* An optimization method which updates the variables.
协助创建新参数的函数
Functions for creating new TensorFlow variables in the given shape and initializing them with random values. Note that the initialization is not actually done at this point, it is merely being defined in the TensorFlow graph.
Can you write Python code for it?

def new_weights(shape):
    return tf.Variable(tf.truncated_normal(shape, stddev=0.05))

def new_biases(length):
    return tf.Variable(tf.constant(0.05, shape=[length]))


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
TensorFlow图模型的编译与运行
创建 TensorFlow session
Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph.
Can you write Python code for it?


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
The class-number is the index of the largest element.
Can you write Python code for it?

y_pred_cls = tf.argmax(y_pred, dimension=1)


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
The convolutional layers expect x to be encoded as a 4-dim tensor so we have to reshape it so its shape is instead [num_images, img_height, img_width, num_channels]. Note that img_height == img_width == img_size and num_images can be inferred automatically by using -1 for the size of the first dimension. So the reshape operation is:
Can you write Python code for it?


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
The following images are the results of applying the convolutional filters to the second image.
Can you write Python code for it?


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
There are 16 input channels to the second convolutional layer, so we can make another 15 plots of filter-weights like this. We just make one more with the filter-weights for the second channel.
Can you write Python code for it?


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
This calculates the classification accuracy by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then calculating the average of these numbers.
Can you write Python code for it?


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
We could also have a placeholder variable for the class-number, but we will instead calculate it using argmax. Note that this is a TensorFlow operator so nothing is calculated at this point.
Can you write Python code for it?


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
We have now calculated the cross-entropy for each of the image classifications so we have a measure of how well the model performs on each image individually. But in order to use the cross-entropy to guide the optimization of the model's variables we need a single scalar value, so we simply take the average of the cross-entropy for all the image classifications.
Can you write Python code for it?


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
优化方法
Now that we have a cost measure that must be minimized, we can then create an optimizer. In this case it is the AdamOptimizer which is an advanced form of Gradient Descent.
Note that optimization is not performed at this point. In fact, nothing is calculated at all, we just add the optimizer-object to the TensorFlow graph for later execution.
Can you write Python code for it?


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
全连接层 2
Add another fully-connected layer that outputs vectors of length num_classes for determining which of the classes the input image belongs to. Note that ReLU is not used in this layer.
Can you write Python code for it?

layer_fc2 = new_fc_layer(input=layer_fc1,
                         num_inputs=fc_size,
                         num_outputs=num_classes,
                         use_relu=False)

layer_fc2


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
关闭 TensorFlow Session
We are now done using TensorFlow, so we close the session to release its resources.
Can you write Python code for it?


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
协助优化迭代的函数
It takes a long time to calculate the gradient of the model using the entirety of a large dataset
. We therefore only use a small batch of images in each iteration of the optimizer.
If your computer crashes or becomes very slow because you run out of RAM, then you may try and lower this number, but you may then need to perform more optimization iterations.
Can you write Python code for it?

train_batch_size = batch_size

def print_progress(epoch, feed_dict_train, feed_dict_validate, val_loss):
    # Calculate the accuracy on the training-set.
    acc = session.run(accuracy, feed_dict=feed_dict_train)
    val_acc = session.run(accuracy, feed_dict=feed_dict_validate)
    msg = "Epoch {0} --- Training Accuracy: {1:>6.1%}, Validation Accuracy: {2:>6.1%}, Validation Loss: {3:.3f}"
    print(msg.format(epoch + 1, acc, val_acc, val_loss))


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
协助展示实验结果与模型性能的函数
Function for printing the classification accuracy on the test-set.
It takes a while to compute the classification for all the images in the test-set, that's why the results are re-used by calling the above functions directly from this function, so the classifications don't have to be recalculated by each function.
Note that this function can use a lot of computer memory, which is why the test-set is split into smaller batches. If you have little RAM in your computer and it crashes, then you can try and lower the batch-size.
Can you write Python code for it?

def print_validation_accuracy(show_example_errors=False,
                        show_confusion_matrix=False):

    # Number of images in the test-set.
    num_test = len(data.valid.images)

    # Allocate an array for the predicted classes which
    # will be calculated in batches and filled into this array.
    cls_pred = np.zeros(shape=num_test, dtype=np.int)

    # Now calculate the predicted classes for the batches.
    # We will just iterate through all the batches.
    # There might be a more clever and Pythonic way of doing this.

    # The starting index for the next batch is denoted i.
    i = 0

    while i < num_test:
        # The ending index for the next batch is denoted j.
        j = min(i + batch_size, num_test)

        # Get the images from the test-set between index i and j.
        images = data.valid.images[i:j, :].reshape(batch_size, img_size_flat)

        # Get the associated labels.
        labels = data.valid.labels[i:j, :]

        # Create a feed-dict with these images and labels.
        feed_dict = {x: images,
                     y_true: labels}

        # Calculate the predicted class using TensorFlow.
        cls_pred[i:j] = session.run(y_pred_cls, feed_dict=feed_dict)

        # Set the start-index for the next batch to the
        # end-index of the current batch.
        i = j

    cls_true = np.array(data.valid.cls)
    cls_pred = np.array([classes[x] for x in cls_pred])

    # Create a boolean array whether each image is correctly classified.
    correct = (cls_true == cls_pred)

    # Calculate the number of correctly classified images.
    # When summing a boolean array, False means 0 and True means 1.
    correct_sum = correct.sum()

    # Classification accuracy is the number of correctly classified
    # images divided by the total number of images in the test-set.
    acc = float(correct_sum) / num_test

    # Print the accuracy.
    msg = "Accuracy on Test-Set: {0:.1%} ({1} / {2})"
    print(msg.format(acc, correct_sum, num_test))

    # Plot some...


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
协助绘制卷积层输出的函数
Can you write Python code for it?

def plot_conv_layer(layer, image):
    # Assume layer is a TensorFlow op that outputs a 4-dim tensor
    # which is the output of a convolutional layer,
    # e.g. layer_conv1 or layer_conv2.

    image = image.reshape(img_size_flat)

    # Create a feed-dict containing just one image.
    # Note that we don't need to feed y_true because it is
    # not used in this calculation.
    feed_dict = {x: [image]}

    # Calculate and retrieve the output values of the layer
    # when inputting that image.
    values = session.run(layer, feed_dict=feed_dict)

    # Number of filters used in the conv. layer.
    num_filters = values.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot the output images of all the filters.
    for i, ax in enumerate(axes.flat):
        # Only plot the images for valid filters.
        if i<num_filters:
            # Get the output image of using the i'th filter.
            # See new_conv_layer() for details on the format
            # of this 4-dim tensor.
            img = values[0, :, :, i]

            # Plot image.
            ax.imshow(img, interpolation='nearest', cmap='binary')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
协助绘制混淆矩阵的函数
Can you write Python code for it?

def plot_confusion_matrix(cls_pred):
    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Get the true classifications for the test-set.
    cls_true = data.valid.cls

    # Get the confusion matrix using sklearn.
    cm = confusion_matrix(y_true=cls_true,
                          y_pred=cls_pred)

    # Print the confusion matrix as text.
    print(cm)

    # Plot the confusion matrix as an image.
    plt.matshow(cm)

    # Make various adjustments to the plot.
    plt.colorbar()
    tick_marks = np.arange(num_classes)
    plt.xticks(tick_marks, range(num_classes))
    plt.yticks(tick_marks, range(num_classes))
    plt.xlabel('Predicted')
    plt.ylabel('True')

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
卷积层 1
Create the first convolutional layer. It takes x_image as input and creates num_filters1 different filters, each having width and height equal to filter_size1. Finally we wish to down-sample the image so it is half the size by using 2x2 max-pooling.
Can you write Python code for it?

layer_conv1, weights_conv1 = \
    new_conv_layer(input=x_image,
                   num_input_channels=num_channels,
                   filter_size=filter_size1,
                   num_filters=num_filters1,
                   use_pooling=True)

layer_conv1


I am working on the file "05_Image_recognition_and_classification/cnn.ipynb".
The first task is:
卷积层 1
Now plot the filter-weights for the first convolutional layer.
Note that positive weights are red and negative weights are blue.
Can you write Python code for it?