I am working on the file "BinaryDetection.ipynb".
The first task is:
Next, I show the probability of detection for circular orbits. The plots on the left and on the right show the same data, but one has a linear and one has logarithmic axis. Both of them are a little coarse, because the grid is not very dense. That can easily be changed if we want to make nicer looking plots.
On the right plot, there are islands where the chance to detect a binary is essentially 0. This happens when the sampling frequency is (or is close to) an integer multiple of the orbital period. Again, this will look better (and more regular) when I use a denser grid.
Those thoughts aside, we see that we expect to detect almost every solar-mass or heavier companion out to 20 AU, and have a chance of 50% out to about 40 AU.
For all plots I use 2 km/s as minimum $\Delta v$ that would be detected in the observations.
Can you write Python code for it?

prop_e0 = np.zeros_like(Pgrid)
for x, y in np.ndindex(Pgrid.shape):
    prop_e0[x,y] = prob_to_detect(2.*u.km/u.s, times, a1grid[x,y], Pgrid[x,y], 0)

def plot_m_x(array):
    fig = plt.figure(figsize=(10,5))
    ax1 = fig.add_subplot(121)
    ax2 = fig.add_subplot(122)
    for ax in [ax1, ax2]:
        cs = ax.contour(agrid, m2grid, array)
        ax.clabel(cs)
        ax.set_xlabel('Semi-major axis [AU]')
        ax.set_ylabel('mass of secondary [$M_{\odot}$]')
        ax.set_title('Probability of detection for $e$ = 0')
    ax2.set_title('Same data, logarithmic axes')
    ax2.set_xscale('log')
    ax2.set_yscale('log')
    return fig

def plot_q_p(array):
    fig = plt.figure(figsize=(10,5))
    ax1 = fig.add_subplot(121)
    ax2 = fig.add_subplot(122)
    for ax in [ax1, ax2]:
        cs = ax.contour(Pgrid, m2grid/m1, array)
        ax.clabel(cs)
        ax.set_xlabel('Period [yr]')
        ax.set_ylabel('mass ratio [$M_{\odot}$]')
        ax.set_title('Probability of detection for $e$ = 0')
    ax2.set_title('Same data, logarithmic axes')
    ax2.set_xscale('log')
    ax2.set_yscale('log')
    return fig

fig = plot_m_x(prop_e0)

# Same plot as above, but with different axis

fig = plot_q_p(prop_e0)


I am working on the file "BinaryDetection.ipynb".
The first task is:
Now, we repeat the excersice for $e=0.5$. As you can see the probability contours move inwards a little, but overall that is not a dramatic change.
Can you write Python code for it?

times = np.arange(0.,10.,1.)
prop_e05 = np.zeros_like(Pgrid)
for x, y in np.ndindex(Pgrid.shape):
    prop_e05[x,y] = prob_to_detect(2.*u.km/u.s, times, a1grid[x,y], Pgrid[x,y], 0.5)

cs = plt.contour(agrid, m2grid, prop_e05)
plt.clabel(cs)
plt.xlabel('Semi-major axis [AU]')
plt.ylabel('mass of secondary [$M_{\odot}$]')
plt.title('Probability of detection for $e$ = 0.5')


I am working on the file "BinaryDetection.ipynb".
The first task is:
Set up the grid of parameters. I chose the secondary mass $m_2$ and the total semi-major axis $a$ as parameters. From this I calculate the semi-major axis of the primary and the period assuming a constant primary mass of $m_1 = 6 M_{\odot}$.
Can you write Python code for it?

# Observation cadence. Here: 10 years, once per year, regular intervals.
times = np.arange(0.,10.,1.)

# Grid for semi-major axis between the two stars (in AU)
a = np.logspace(0.1,2)
# Masses (in M_sun)
m1 = 5   # primary
m2 = np.array([0.02, 0.05, 0.08, 0.1,0.5, 0.8,1,2,3,4,5])  # secondary
M = m1 + m2  # total mass of system
agrid, m2grid = np.meshgrid(a, m2)
# semi-major axis of primary
a1grid = agrid * m2grid / (m1 + m2grid)
# Period
Pgrid = (m1 + m2grid) * np.sqrt(a1grid**3 / m2grid**3)

fig = plt.figure(figsize=(10,5))
ax1 = fig.add_subplot(121)
ax2 = fig.add_subplot(122)

cs = ax1.contour(Pgrid, [0.1,0.2,1,5,25,50,100,300])
ax1.clabel(cs)
ax1.set_title('Period grid [yr]')
cs = ax2.contour(m2grid/m1)
ax2.clabel(cs)
ax2.set_title('mass ratio grid')


I am working on the file "BinaryDetection.ipynb".
The first task is:
The observed magnitude of the RV variation depends crucially on the angle between the line-of-sight (LOS) and the orbit, particularly the inclination and the longitude of periastron. They way I handle that here is that for each orbit I calculate the observed RV values for a large number of sightlines. I then check for which fraction of those sightlines the RV change is large enough to be detected.
To obtain a fair estimate of the probability of detection I need an isotropic distribution of sightlines. There are schemes in the literature to tesselate a sphere in such a way that each tesselate has the same area ("geodesic grid") and then I could use the mid-point of this tesselate for my sightlines. However, all those schemes are a pain to implement (or if implemented already, a pain to use) due to all the sin and cos involved. It's not impossible, but for now I just generate a large number of random LOS.
Can you write Python code for it?

def generate_LOSs(n):
    '''Generate a random distribution of Line of sights

    Generate a random distribution of sightlines in cartesian coordinates,
    with an isotropic distribution (no clustering around the poles).
    Analytically, a truely isotropic distribution could be generated using
    geodesic grids, but this is much faster to program and (for large n) just as good.

    Parameters
    ----------
    n : integer
        number of sightlines to be generated

    Returns
    -------
    los : np.array of dim [3,m] with m < n
        x,y,z cartesian coordinates for each sightline. Only a subset of all generated
        sightlines is returend, selected in such a way that they are (within the limits
        of their randomness) isotropically distributed over the sphere.
    '''
    # Make points in box with x,y,z in [-1,1].
    los = np.random.random((3, n)) * 2 - 1.
    # Throw out all points beyond a unit sphere.
    # Yes, this is a wasteful implementation, but it's fast enough that I don't care.
    r = np.sqrt((los * los).sum(axis=0))
    ind = ((r <= 1) & (r >= 1e-6))  # Throw out inner values where round-off errors matter.
    return los[:, ind] / r[ind]

def get_LOSs(n = 1000):
    '''Generate a random distribution of Line of sights

    Generate a random distribution of sightlines in cartesian coordinates,
    with an isotropic distribution (no clustering around the poles).
    Analytically, a truely isotropic distribution could be generated using
    geodesic grids, but this is much faster to program and (for large n) just as good.

    Parameters
    ----------
    n : integer
        number of sightlines to be generated

    Returns
    -------
    los : np.array of dim [3,m] with m < n
        x,y,z cartesian coordinates for each sightline.
    '''
    while True:
        los = generate_LOSs(4 * n) # 2 should be big enough that this usually succeeds.
        if los.shape[1] >= n:
            break
    return los[:, :n]

def calc_many_vobs(times, a, p, e, n_tau=50, n_los=1000,...


I am working on the file "BinaryDetection.ipynb".
The first task is:
We just do this for eccentricity 0 and 0.5. Since there is little prior information and (see above) it also does not matter much in that range, we will just show that the contours are very similar and no more details are required.
Can you write Python code for it?

allsims00 = {}
for star, group in izip(actual_obs_years.groups.keys, actual_obs_years.groups):
    print datetime.datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S'), 'Working on: ', star[0]
    prop = np.zeros_like(Pgrid)
    for x, y in np.ndindex(Pgrid.shape):
        prop[x,y] = prob_to_detect(2.*u.km/u.s, np.array(group['Year']), a1grid[x,y], Pgrid[x,y], 0.0)
    allsims00[star[0]] = prop

allsims05 = {}
for star, group in izip(actual_obs_years.groups.keys, actual_obs_years.groups):
    print datetime.datetime.fromtimestamp(time.time()).strftime('%Y-%m-%d %H:%M:%S'), 'Working on: ', star[0]
    prop = np.zeros_like(Pgrid)
    for x, y in np.ndindex(Pgrid.shape):
        prop[x, y] = prob_to_detect(2. * u.km/u.s, np.array(group['Year']), a1grid[x, y], Pgrid[x, y], 0.5)
    allsims05[star[0]] = prop

allnames = set(allsims00.keys()) - set(['CK  Cam'])
all00 = np.dstack([allsims00[n] for n in allnames]).mean(axis=2)
all05 = np.dstack([allsims05[n] for n in allnames]).mean(axis=2)

fig = plot_q_p(allsims00[list(allnames)[1]])

fig = plot_q_p(all00)


I am working on the file "Bio311-ODE-modeling-network-motifs.ipynb".
The first task is:
<h2> <font color='firebrick'> Question Set 1 (3 pts) </font> </h2>

What happens to the shape of the Hill function curve when you vary $n$ between 1 and 8?
What happens to the curve when you vary $\beta$ between 2 and 20?
What happens to the curve when you vary $K$ between 2 and 20?

Modeling transcriptional repression
If rather than stimulating the production of $Y$, $X$ "represses" $Y$, we can write the corresponding Hill function as:
$$
f(X) = \frac{\beta}{1 + (X/K)^n}
$$
Remember that both of these Hill functions (activating and repressing)  describe the production of $Y$ as a function of the levels of $X$, not the temporal dynamics of $Y$ which we'll look at after developing a few more ideas.
Can you write Python code for it?

## Python implementation of repressive Hill function

def hill_repressing(X, B, K, n):
    return B/(1.0 + (X/K)**n)

## generate a plot using the hill_repressing function defined above

# CHANGE THESE VALUES TO EXPLORE THE EFFECT OF THESE PARAMTERS
# (see questions below)
n = 1
B = 5
K = 10

# generate a range of x values, representing a range of concentrations of our
# transcription factor X
x = linspace(0,30,200)  # generate 200 evenly spaced points between 0 and 30

# calculating corresponding rates of production of Y
y = hill_repressing(x, B, K, n)

# plot the hill fxn with the user set parameters
plot(x, y, label='B = {}, K = {}, n={}\n(user specified)'.format(B, K, n))

# plot the hill fxn with a set of reference parameters to facilitate comparison
plot(x, hill_repressing(x, 5, 10, 1), label='B = 5, K = 10, n=1\n(reference)')

xlabel('Concentration of X')
ylabel('Rate of production of Y')
legend(loc='best')

pass # suppress further output


I am working on the file "Bio311-ODE-modeling-network-motifs.ipynb".
The first task is:
<h2> <font color='firebrick'> Question Set 4 (3 pts) </font> </h2>

The concentration of $Y$ eventually reaches a steady state, $Y_{st}$.  How does $Y_{st}$ relate to $\beta$ and $\alpha$?

The response time of a dynamical system, $T_{1/2}$is defined as the time it takes for it to go half-way between it's initial and final value.

a) How does the response time change as you vary $\beta$?
b) How does the response time change as you vary $\alpha$?

Estimate the response time as you vary the parameter  $\alpha$ and see if you can create a plot (in Python, or R, or Excel) showing the relationship between $\alpha$ and response time, for $0.01 \leq \alpha \leq 0.1$.

Toggling the activator X
In the proceeding example the activator $X$ was on at the beginning of the simulation and just stayed on.  Let's see what happens when $X$ has pulsatile dynamics.  This would be akin to  toggling $X$ on then off, and asking what happens to $Y$.
Can you write Python code for it?

B = 5
K = 10
a = 0.05

# setup pulse of X
# off (0) for first 50 steps, on for next 100 steps, off again for last 100 steps
X = [0]*50 + [3*K]*100 + [0]*100

Y = [0] # initial value of Y

nsteps = 250
for i in range(1, nsteps):
    xnow = X[i]
    yprevious = Y[-1]
    deltay = dYdt(B, K, a, xnow, yprevious)
    ynew = yprevious + deltay
    Y.append(ynew)

plot(X, color='red', linestyle='dashed', label="X")
plot(Y, color='blue', label="Y")
ylim(0, max(max(X)*1.1, (B/a)*1.1))
xlabel('Time units')
ylabel('Concentration')
legend(loc="best")
pass


I am working on the file "Bio311-ODE-modeling-network-motifs.ipynb".
The first task is:

 Question Set 5 (2 pts)

How do the dynamics of $Y$ and $Z$ differ in the simulation above?

Try varying the length of the first (short) pulse.  How does changing the length of the pulse affect the dynamics of $Y$ and $Z$?

Performance of the Coherent FFL under noisy inputs
Let's further explore the behavior of the coherent FFL defined given noisy inputs.  As before we're going to define an input signal, $X$, that has a short and long pulse, but now we're going to pollute $X$ with random noise.
Can you write Python code for it?

nsteps = 150

p1start = 10
p1duration = 5

p2start = 50
p2duration = 50

short_pulse = pulse(p1start, p1start + p1duration, nsteps)  # short pulse
long_pulse = pulse(p2start, p2start + p2duration, nsteps)  # long pulse
X = short_pulse + long_pulse

# change this `scale` argument to increase/decrease noise
noise = np.random.normal(loc=0, scale=0.2, size=nsteps) # mean=0, sd=0.2

X = X + noise

# setup parameters for Y and Z
Y = [0]
betay, alphay = 0.2, 0.1
Kxy = 0.5

Z = [0]
betaz, alphaz = 0.2, 0.1
Kxz = 0.5
Kyz = 1

for i in range(nsteps):
    xnow = X[i]
    ynow, znow = Y[-1], Z[-1]
    ynew = ynow + dYdt(betay, Kxy, alphay, xnow, ynow)
    znew = znow + dZdt(betaz, Kxz, Kyz, alphaz, xnow, ynow, znow)
    Y.append(ynew)
    Z.append(znew)

# draw each trace as a subfigure
# subfigures stacked in a vertical grid

subplot2grid((3,1),(0,0))
plot(X, 'k', label='X', linewidth=1)
legend()

subplot2grid((3,1),(1,0))
plot(Y, 'b', label='Y', linewidth=2)
legend()

subplot2grid((3,1),(2,0))
plot(Z, 'r', label='Z', linewidth=2)

vlines(p1start, min(Z),max(Z)*1.1,color='black',linestyle='dashed')
annotate("pulse 1 on", xy=(p1start,1),xytext=(40,20),
         textcoords='offset points',
         horizontalalignment="center",
         verticalalignment="bottom",
         arrowprops=dict(arrowstyle="->",color='black',
                         connectionstyle='arc3,rad=0.5',
                         linewidth=1))
vlines(p2start, min(Z),max(Z)*1.1,color='black',linestyle='dashed')
annotate("pulse 2 on", xy=(p2start,1),xytext=(-40,0),
         textcoords='offset points',
         horizontalalignment="center",
         verticalalignment="bottom",
         arrowprops=dict(arrowstyle="->",color='black',
                         connectionstyle='arc3,rad=0.5',
                         linewidth=1))
legend()
pass


I am working on the file "Bio311-ODE-modeling-network-motifs.ipynb".
The first task is:
<h2> <font color='firebrick'>Question Set 6 (2 pts) </font> </h2>

In the code cell above, try changing the duration of the first pulse and the scale of the noise (see comments in code) to get a sense of how good a filter the FFL is.  Is there a bias to the filtering with respect to Z turning on versus Z turning off?
Dynamics of Y and Z in the Coherent FFL
As before we can solve for Y as a function of time and calculate what its steady state value will be:
$$
Y(t) = Y_{st}(1-e^{-\alpha_{y}t})
$$
and
$$
Y_{st}=\frac{\beta_y}{\alpha_y}
$$
How about $Z$?
Since $Z$ is governed by an AND function it needs both $X$ and $Y$ to be above their respective thresholds, $K_{xz}$ and $K_{yz}$. For the sake of simplicity let's assume that both $Y$ and $Z$ have the same threshold with respect to $X$, i.e. $K_{xy} = K_{xz}$. This allows us just to consider how long it takes for $Y$ to reach the threshold value $K_{yz}$. Given this we can calculate the delay before $Z$ turns on, $T_{\mathrm{on}}$ as follows.
$$
Y(T_{\mathrm{on}}) = Y_{st}(1-e^{-\alpha_y T_{\mathrm{on}}}) = K_{yz}
$$
and solving for $T_{\mathrm{on}}$ we find:
$$
T_{\mathrm{on}} = \frac{1}{\alpha_y} \log\left[\frac{1}{(1-K_{yz}/Y_{st})}\right]
$$
Thus we see that the delay before $Z$ turns on is a function of the degradation rate of $Y$ and the ratio between $Y_{st}$ and $K_{yz}$.
Exploring the Parameter space of $Z$'s turn-on time
From the above formula, we see that there are two parameters that affect the turn-on time of $Z$ -- $\alpha_y$ (the scaling factor for the decay rate of $Y$)  and the compound parameter $K_{yz}/Y_{st}$ (the threshold concentration where $Y$ activate $Z$ relative to the steady state of $Y$).  To explore the two-dimensional parameter space of $Z's$ $T_on$ we can create a contour plot.
Can you write Python code for it?

def Ton(alpha, KYratio):
    return (1.0/alpha) * log(1.0/(1.0-KYratio))

## Create a contour plot for a range of alpha and Kyz/Yst
x = alpha = linspace(0.01, 0.2, 100)
y = KYratio = linspace(0.01, 0.99, 100)
X,Y = meshgrid(x, y)

Z = Ton(X,Y)
levels = MaxNLocator(nbins=20).tick_values(Z.min(), Z.max())

im = contourf(X,Y,Z, cmap=cm.afmhot_r, levels=levels)
contour(X, Y, Z, levels,
                  colors=('k',),
                  linewidths=(0.5,))
colorbar(im)
xlabel('alpha')
ylabel("Kyz/Yst")
pass


I am working on the file "Bio311-ODE-modeling-network-motifs.ipynb".
The first task is:
Define Python functions for dY/dt and dZ/dt
Recall from above that
\begin{eqnarray}
\frac{dY}{dt} & = & \beta_y\ \Theta(X > K_{xy}) - \alpha_{y}Y \ \
\frac{dZ}{dt} & = & \beta_z\ \Theta(X > K_{xz})\Theta(Y > K_{yz}) - \alpha_{z}Z
\end{eqnarray}
Can you write Python code for it?

def dYdt(B, K, a, X, Y):
    if X > K:
        theta = 1
    else:
        theta = 0
    return B * theta - a * Y

def dZdt(B, Kx, Ky, a, X, Y, Z):
    theta = 0
    if (X > Kx) and (Y > Ky):
        theta = 1
    return B * theta - a * Z

## Plot X, Y, and Z on the same time scale

nsteps = 150
short_pulse = pulse(20, 23, nsteps)  # 5 sec pulse
long_pulse = pulse(50, 100, nsteps)  # 50 sec pulse
X = short_pulse + long_pulse

# setup parameters for Y and Z
Y = [0]
betay, alphay = 0.2, 0.1
Kxy = 0.5

Z = [0]
betaz, alphaz = 0.2, 0.1
Kxz = 0.5
Kyz = 1

for i in range(nsteps):
    xnow = X[i]
    ynow, znow = Y[-1], Z[-1]

    ynew = ynow + dYdt(betay, Kxy, alphay, xnow, ynow)
    znew = znow + dZdt(betaz, Kxz, Kyz, alphaz, xnow, ynow, znow)

    Y.append(ynew)
    Z.append(znew)

plot(X, 'k--', label='X', linewidth=1.5)
plot(Y, 'b', label='Y')
plot(Z, 'r', label='Z')
ylim(-0.1, 2.5)
xlabel("Time")
ylabel("Concentration")
legend()
pass


I am working on the file "Bio311-ODE-modeling-network-motifs.ipynb".
The first task is:
Visualizing the activating Hill function
Now we'll setup a plot to visualize what this function looks like.
Can you write Python code for it?

## generate a plot using the hill_activating function defined above

# setup paramters for our simulation
# CHANGE THESE VALUES TO EXPLORE THE EFFECT OF THESE PARAMTERS
# (see questions below)
n = 1
B = 5
K = 10

# generate a range of x values, representing a range of concentrations of our
# transcription factor X
x = linspace(0,30,200)  # generate 200 evenly spaced points between 0 and 30

# calculating corresponding rates of production of Y
y = hill_activating(x, B, K, n)

# plot the hill fxn with the user set parameters
plot(x, y, label='B = {}, K = {}, n={}\n(user specified)'.format(B, K, n))

# plot the hill fxn with a set of reference parameters to facilitate comparison
plot(x, hill_activating(x, 5, 10, 1), label='B = 5, K = 10, n=1\n(reference)', alpha=0.75)

xlabel('Concentration of X')
ylabel('Rate of production of Y')
legend(loc='best')

pass # suppress further output


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
$\beta$ e $\Delta z$
Vamos agora estimar as mudanças evolutivas em cada ramo da filogenia, e, usando as matrizes ancestrais, calcular os gradientes de seleção estimados.
Os $\Delta z$ são apenas as diferenças nas médias de um nó com o seu acestral. Os $\beta$ são estimados com a equação de Lande:
$
\beta = G^{-1}\Delta z
$
Can you write Python code for it?

delta_z = {}
beta = {}
for n in tree.nodes()[1:]: #começamos do 1 para pular a raiz, que não tem ancestral
    parent = get_node_name(n.parent_node)
    branch = get_node_name(n) + '_' + parent
    delta_z[branch] = node_means[get_node_name(n)] - node_means[parent]
    beta[branch] = np.linalg.solve(node_matrices[parent], delta_z[branch])

delta_z

beta


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Analise de variância
Calculando as matrizes dentro de grupos, entre grupos e total
Can you write Python code for it?


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Calculando matrizes de covariância
Vamos calcular as matrizes de covariância e correlação por espécie, além das médias, e agrupá-las num dicionário.
Can you write Python code for it?

cov_matrices = dados_brutos.groupby('ESPECIE').apply(lambda x: x.cov())
cor_matrices = dados_brutos.groupby('ESPECIE').apply(lambda x: x.corr())
especies_labels = list(pd.unique(dados_brutos['ESPECIE']))

cov_matrices

cor_matrices


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Comparando as matrizes
Can you write Python code for it?


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
E um dicionário de médias.
Can you write Python code for it?


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Filogenia
Vamos incluir uma filogenia e calcular estados ancestrais
Can you write Python code for it?

tree = dp.Tree.get_from_string("(E, ((C, B)4,(A,D)3)2)1;", "newick")
tree.print_plot(display_width = 50, show_internal_node_labels = True, leaf_spacing_factor = 4)


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Podemos também acessar as matrizes pelo nome da espécie:
Can you write Python code for it?


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Podemos também usar a função groupby para aplicar funções em sub conjuntos dos dados, agrupando pela coluna ESPECIE
Can you write Python code for it?


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Podemos usar a função describe para ter um resumo global, olhando para médias, desvio padrão e quartis.
Can you write Python code for it?


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Podemos utilizar o pandas para formatar esses resultados em tabelas
Can you write Python code for it?

df_betas = pd.DataFrame.from_dict(beta, orient='index')
df_betas.columns = traits
df_betas

df_dz = pd.DataFrame.from_dict(delta_z, orient='index')
df_dz.columns = traits
df_dz

traits_c = list(traits)
traits_c.append('otu')
df_matrices = pd.DataFrame(columns=traits_c)
for node in node_matrices:
    df = pd.DataFrame(node_matrices[node], columns=traits, index = traits)
    df['otu'] = node
    df_matrices = df_matrices.append(df)
df_matrices


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Primeiro vamos ler os dados usando o pandas.
A função read_csv cria um data.frame com os dados.
Can you write Python code for it?

import numpy as np
import pandas as pd
import dendropy  as dp
import matplotlib as mpl

dados_brutos = pd.read_csv("./dados.csv")
num_traits = 4
traits = dados_brutos.columns[:num_traits]
dados_brutos.head(10)


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Temos tb uma lista de todos os nós:
Can you write Python code for it?


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Um dicionário de tamanhos amostrais
Can you write Python code for it?


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Vamos calcular as médias, desvios padrão e coeficiantes de variação por caractere por espécie.
Can you write Python code for it?

medias = dados_brutos.groupby('ESPECIE').mean()
medias

std = dados_brutos.groupby('ESPECIE').std()
std

cv = std/medias
cv


I am working on the file "Biologia Evolutiva.ipynb".
The first task is:
Visualizando os dados
Vamos usar o pacote matplotlib para fazer gráficos diagnósticos dos dados.
Can you write Python code for it?

# histogramas

# scatter plots

# boxplots


I am working on the file "BiologicalPatternFormation/Biological Pattern Formation_instructor.ipynb".
The first task is:
The simulation parameters. We'll use the following condition to ensure that we're stable (see Lorena Barba)
$$ dt \le \frac{dx^2}{2} $$
Can you write Python code for it?

size = 20 #size of 2D grid
dx = 2/size # space step
dt = .9 * dx**2/2 # see stability condition above
T = 40000*dt # might want to specify time directly
n = int(T/dt) # number of steps

U = np.random.rand(size, size)
V = np.random.rand(size, size)

def laplacian(Z):
    Ztop = Z[0:-2,1:-1]
    Zleft = Z[1:-1,0:-2]
    Zbottom = Z[2:,1:-1]
    Zright = Z[1:-1,2:]
    Zcenter = Z[1:-1,1:-1]
    return (Ztop + Zleft + Zbottom + Zright) / 4*dx**2

plt.imshow(U, cmap=plt.cm.copper, interpolation='none',extent=[-1,1,-1,1]);

# We simulate the PDE with the finite difference method.
for i in range(n):
    # We compute the Laplacian of u and v.
    deltaU = laplacian(U)
    deltaV = laplacian(V)
    # We take the values of u and v inside the grid.
    Uc = U[1:-1,1:-1]
    Vc = V[1:-1,1:-1]
    # We update the variables.
    U[1:-1,1:-1], V[1:-1,1:-1] = \
        Uc + dt * Du * deltaU, \
        Vc + dt * Dv * deltaV
    # Neumann conditions: derivatives at the edges
    # are null.
    for Z in (U, V):
        Z[0,:] = Z[1,:]
        Z[-1,:] = Z[-2,:]
        Z[:,0] = Z[:,1]
        Z[:,-1] = Z[:,-2]
plt.imshow(U, cmap=plt.cm.copper,  interpolation='none', extent=[-1,1,-1,1]);

plt.imshow(U, cmap=plt.cm.copper,  interpolation='none', extent=[-1,1,-1,1]);

from IPython.display import display, clear_output
from scipy.special import jn
import time
x = np.linspace(0,5)
f, ax = plt.subplots()
ax.set_title("Bessel functions")

for n in range(1,10):
    time.sleep(1)
    ax.plot(x, jn(x,n))
    clear_output(wait=True)
    display(f)

# close the figure at the end, so we don't get a duplicate
# of the last plot
plt.close()

# We simulate the PDE with the finite difference method.
U = np.random.rand(size, size)
V = np.random.rand(size, size)
f,ax = plt.subplots()
for i in range(10):
    # We compute the Laplacian of u and v.
    deltaU = laplacian(U)
    deltaV = laplacian(V)
    # We take the values of u and v inside the grid.
    Uc = U[1:-1,1:-1]
    Vc = V[1:-1,1:-1]
    # We update the variables.
    U[1:-1,1:-1],...


I am working on the file "BiologicalPatternFormation/WorkingReactionDiffusion.ipynb".
The first task is:
Now, we define a function that computes the discrete Laplace operator of a 2D variable on the grid, using a five-point stencil finite difference method. This operator is defined by:

$$\Delta u(x,y) \simeq \frac{u(x+h,y)+u(x-h,y)+u(x,y+h)+u(x,y-h)-4u(x,y)}{dx^2}$$
We can compute the values of this operator on the grid using vectorized matrix operations. Because of side effects on the edges of the matrix, we need to remove the borders of the grid in the computation.
Can you write Python code for it?

def laplacian(Z):
    Ztop = Z[0:-2,1:-1]
    Zleft = Z[1:-1,0:-2]
    Zbottom = Z[2:,1:-1]
    Zright = Z[1:-1,2:]
    Zcenter = Z[1:-1,1:-1]
    return (Ztop + Zleft + Zbottom + Zright - 4 * Zcenter) / dx**2


I am working on the file "BiologicalPatternFormation/WorkingReactionDiffusion.ipynb".
The first task is:
We initialize the variables $u$ and $v$. The matrices $U$ and $V$ contain the values of these variables on the vertices of the 2D grid. These variables are initialized with a uniform noise between $0$ and $1$.
Can you write Python code for it?

U = np.random.rand(size, size)
V = np.random.rand(size, size)


I am working on the file "BiologicalPatternFormation/WorkingReactionDiffusion.ipynb".
The first task is:
We will simulate the following system of partial differential equations on the domain $E=[-1,1]^2$:

\begin{align}
\frac{\partial u}{\partial t} &= a \Delta u + u - u^3 - v + k\
\tau\frac{\partial v}{\partial t} &= b \Delta v + u - v\
\end{align}
The variable $u$ represents the concentration of a substance favoring skin pigmentation, whereas $v$ represents another substance that reacts with the first and impedes pigmentation.
At initialization time, we assume that $u$ and $v$ contain independent random numbers on every grid point. Besides, we take Neumann boundary conditions: we require the spatial derivatives of the variables with respect to the normal vectors to be null on the boundaries of the domain $E$.
Let's define the four parameters of the model.
Can you write Python code for it?


I am working on the file "Biomass/.ipynb_checkpoints/ReadFromDBv2-Copy1-checkpoint.ipynb".
The first task is:
Afterwards, use pandas to import the data from Excel files or any other text file format. Make sure that the data in good shape before trying to push it into the server. In this example I use previous knowledge of the structure of the tabs in the excel file to recursively upload each tab and match the name of the table with the tab name.
If you are using csv files just change the commands to pd.read_csv() in this link you can find the documentation.
Before doing this I already checked that the data is properly organized, crate new cells to explore the data beforehand if needed
excel_file = 'substations_table.xlsx'
tab_name = 'sheet1'
schema_for_upload = 'geographic_data'
pd_data.to_sql(name, engine_db, schema=schema_for_upload, if_exists='replace',chunksize=100)
Can you write Python code for it?

#excel_file = 'substations_table.xlsx'
#tab_name = 'sheet1'
csv_name = ['LEMMA_ADS_AllSpp_2016_Turbo_01252016.csv']
schema_for_upload = 'lemma2016'
for name in csv_name:
    pd_data = pd.read_csv(name, encoding='UTF-8')
    pd_data.to_sql(name, engine_db, schema=schema_for_upload, if_exists='replace',chunksize=1000)


I am working on the file "Biomass/.ipynb_checkpoints/ReadFromDBv2-Copy1-checkpoint.ipynb".
The first task is:
The function created the geom column, the next step is to define a function to create the Primary-Key in the db. Remember that the index from the data frame is included as an index in the db, sometimes an index is not really neded and might need to be dropped.
Can you write Python code for it?

def create_pk(table,schema,column,engine):
    k = engine.connect()
    query = ('set search_path = "'+ schema +'"'+ ', public;')
    print query
    k.execute(query)
    query = ('alter table ' + table + ' ADD CONSTRAINT '+ table +'_pk PRIMARY KEY (' + column + ')')
    print query
    k.execute(query)
    k = engine.dispose()
    return 'Primary key created with column' + column

col = ''
create_pk(table,schema,col,engine_db)


I am working on the file "Biomass/.ipynb_checkpoints/ReadFromDBv2-Copy1-checkpoint.ipynb".
The first task is:
The reason why we use postgis is to improve geospatial queries and provide a better data structure for geospatial operations. Many of the ST_ functions have improved performance when a geospatial index is created. The process implemented here comes from this workshop. This re-creates the process using python functions so that it can be easily replicated for many tables.
The query to create a geospatial index is as follows:
SQL
set search_path = SCHEMA, public;
CREATE INDEX vTABLE_gix ON vTABLE USING GIST (geom);
This assumes that the column name with the geometry is named geom. If the process follows from the previous code, it will work ok.
The following step is to run a VACUUM, creating an index is not enough to allow PostgreSQL to use it effectively. VACUUMing must be performed when ever a new index is created or after a large number of UPDATEs, INSERTs or DELETEs are issued against a table.
SQL
VACUUM ANALYZE vTABLE;
The final step corresponds to CLUSTERING, this process re-orders the table according to the geospatial index we created. This ensures that records with similar attributes have a high likelihood of being found in the same page, reducing the number of pages that must be read into memory for some types of queries. When a query to find nearest neighbors or within a certain are is needed, geometries that are near each other in space are near each other on disk. The query to perform this clustering is as follows:
CLUSTER vTABLE USING vTABLE_gix;
ANALYZE vTABLE;
Can you write Python code for it?

def create_gidx(table,schema,engine,column='geom'):
    k = engine.connect()
    query = ('set search_path = "'+ schema +'"'+ ', public;')
    k.execute(query)
    print query
    query = ('CREATE INDEX ' + table + '_gix ON ' + table + ' USING GIST (' + column + ');')
    k.execute(query)
    print query
    query = ('VACUUM ' + table + ';')
    k.execute(query)
    print query
    query = ('CLUSTER ' + table + ' USING ' + table + '_gix;')
    k.execute(query)
    print query
    query = ('ANALYZE ' + table + ';')
    k.execute(query)
    print query
    k = engine.dispose()
    return k

create_gidx(table,schema,engine_db)


I am working on the file "Biomass/.ipynb_checkpoints/ReadFromDBv2-checkpoint.ipynb".
The first task is:
After importing the required packages, first create the engine to connect to the DB. The approach I generally use is to create a string based on the username and password. The code is a function, you just need to fill in with the username, password and the dbname.
It allows you to create different engines to connect to serveral dbs.
Can you write Python code for it?

def connection(user,passwd,dbname, echo_i=False):
    str1 = ('postgresql+pg8000://' + user +':' + passw + '@switch-db2.erg.berkeley.edu:5432/'
            + dbname + '?ssl=true&sslfactory=org.postgresql.ssl.NonValidatingFactory')
    engine = create_engine(str1,echo=echo_i,isolation_level='AUTOCOMMIT')
    return engine

user = 'jdlara'
passw = 'Amadeus-2010'
dbname = 'apl_cec'
engine_db= connection(user,passw,dbname)


I am working on the file "Biomass/.ipynb_checkpoints/ReadFromDBv2-checkpoint.ipynb".
The first task is:
Afterwards, use pandas to import the data from Excel files or any other text file format. Make sure that the data in good shape before trying to push it into the server. In this example I use previous knowledge of the structure of the tabs in the excel file to recursively upload each tab and match the name of the table with the tab name.
If you are using csv files just change the commands to pd.read_csv() in this link you can find the documentation.
Before doing this I already checked that the data is properly organized, crate new cells to explore the data beforehand if needed
Can you write Python code for it?

excel_file = 'substations_table.xlsx'
tab_name = 'sheet1'
schema_for_upload = 'geographic_data'
pd_data.to_sql(name, engine_db, schema=schema_for_upload, if_exists='replace',chunksize=100)

excel_file = 'substations_table.xlsx'
tab_name = 'sheet1'
#csv_name = ''
schema_for_upload = 'geographic_data'
for name in tab_name:
    pd_data = pd.read_excel(excel_file, encoding='UTF-8')
    pd_data.to_sql(name, engine_db, schema=schema_for_upload, if_exists='replace',chunksize=100)


I am working on the file "Biomass/.ipynb_checkpoints/ReadFromDBv2-checkpoint.ipynb".
The first task is:
Once the data is updated, it is possible to run the SQL commands to properly create geom columns in the tables, this can be done as follows. The ojective is to run an SQL querie like this:
PGSQL
set search_path = SCHEMA, public;
alter table vTABLE drop column if exists geom;
SELECT AddGeometryColumn ('SCHEMA','vTABLE','geom',4326,'POINT',2);
UPDATE TABLE set geom = ST_SetSRID(st_makepoint(vTABLE.lon, vTABLE.lat), 4326)::geometry;
where SCHEMA and vTABLE are the variable portions. Also note, that this query assumes that your columns with latitude and longitude are named lat and lon respectively; moreover, it also assumes that the coordinates are in the 4326 projection.
The following function runs the query for you, considering again that the data is clean and nice.
Can you write Python code for it?

def create_geom(table,schema,engine, projection=4326):
    k = engine.connect()
    query = ('set search_path = "'+ schema +'"'+ ', public;')
    print query
    k.execute(query)
    query = ('alter table ' + table + ' drop column if exists geom;')
    print query
    k.execute(query)
    query = 'SELECT AddGeometryColumn (\''+ schema + '\',\''+ table + '\',\'geom\''+',4326,\'POINT\',2);'
    print query
    k.execute(query)
    query = ('UPDATE ' + table + ' set geom = ST_SetSRID(st_makepoint(' + table + '.lon, ' +
             table + '.lat),' + str(projection) + ')::geometry;')
    k.execute(query)
    print query
    k = engine.dispose()
    return 'geom column added with SRID ' + str(projection)

table = 'substation_table'
schema = 'geographic_data'
create_geom(table,schema,engine_db)


I am working on the file "Biomass/.ipynb_checkpoints/ReadFromDBv2-checkpoint.ipynb".
The first task is:
The reason why we use postgis is to improve geospatial queries and provide a better data structure for geospatial operations. Many of the ST_ functions have improved performance when a geospatial index is created. The process implemented here comes from this workshop. This re-creates the process using python functions so that it can be easily replicated for many tables.
The query to create a geospatial index is as follows:
SQL
set search_path = SCHEMA, public;
CREATE INDEX vTABLE_gix ON vTABLE USING GIST (geom);
This assumes that the column name with the geometry is named geom. If the process follows from the previous code, it will work ok.
The following step is to run a VACUUM, creating an index is not enough to allow PostgreSQL to use it effectively. VACUUMing must be performed when ever a new index is created or after a large number of UPDATEs, INSERTs or DELETEs are issued against a table.
SQL
VACUUM ANALYZE vTABLE;
The final step corresponds to CLUSTERING, this process re-orders the table according to the geospatial index we created. This ensures that records with similar attributes have a high likelihood of being found in the same page, reducing the number of pages that must be read into memory for some types of queries. When a query to find nearest neighbors or within a certain are is needed, geometries that are near each other in space are near each other on disk. The query to perform this clustering is as follows:
CLUSTER vTABLE USING vTABLE_gix;
ANALYZE vTABLE;
Can you write Python code for it?


I am working on the file "Biomass/ReadFromDBv2-Copy1.ipynb".
The first task is:
After importing the required packages, first create the engine to connect to the DB. The approach I generally use is to create a string based on the username and password. The code is a function, you just need to fill in with the username, password and the dbname.
It allows you to create different engines to connect to serveral dbs.
Can you write Python code for it?


I am working on the file "Biomass/ReadFromDBv2-Copy1.ipynb".
The first task is:
Afterwards, use pandas to import the data from Excel files or any other text file format. Make sure that the data in good shape before trying to push it into the server. In this example I use previous knowledge of the structure of the tabs in the excel file to recursively upload each tab and match the name of the table with the tab name.
If you are using csv files just change the commands to pd.read_csv() in this link you can find the documentation.
Before doing this I already checked that the data is properly organized, crate new cells to explore the data beforehand if needed
excel_file = 'substations_table.xlsx'
tab_name = 'sheet1'
schema_for_upload = 'geographic_data'
pd_data.to_sql(name, engine_db, schema=schema_for_upload, if_exists='replace',chunksize=100)
Can you write Python code for it?

#excel_file = 'substations_table.xlsx'
#tab_name = 'sheet1'
csv_name = ['Results_2016_wholestate_noBA.csv']
schema_for_upload = 'lemma2016'
for name in csv_name:
    pd_data = pd.read_csv(name, encoding='UTF-8')
    pd_data.to_sql(name, engine_db, schema=schema_for_upload, if_exists='replace',chunksize=1000)


I am working on the file "Biomass/ReadFromDBv2-Copy1.ipynb".
The first task is:
Once the data is updated, it is possible to run the SQL commands to properly create geom columns in the tables, this can be done as follows. The ojective is to run an SQL querie like this:
PGSQL
set search_path = SCHEMA, public;
alter table vTABLE drop column if exists geom;
SELECT AddGeometryColumn ('SCHEMA','vTABLE','geom',4326,'POINT',2);
UPDATE TABLE set geom = ST_SetSRID(st_makepoint(vTABLE.lon, vTABLE.lat), 4326)::geometry;
where SCHEMA and vTABLE are the variable portions. Also note, that this query assumes that your columns with latitude and longitude are named lat and lon respectively; moreover, it also assumes that the coordinates are in the 4326 projection.
The following function runs the query for you, considering again that the data is clean and nice.
Can you write Python code for it?

def create_geom(table,schema,engine, projection=5070):
    k = engine.connect()
    query = ('set search_path = "'+ schema +'"'+ ', public;')
    print query
    k.execute(query)
    query = ('alter table ' + table + ' drop column if exists geom;')
    print query
    k.execute(query)
    query = 'SELECT AddGeometryColumn (\''+ schema + '\',\''+ table + '\',\'geom\''+',5070,\'POINT\',2);'
    print query
    k.execute(query)
    query = ('UPDATE ' + table + ' set geom = ST_SetSRID(st_makepoint(' + table + '.x, ' +
             table + '.y),' + str(projection) + ')::geometry;')
    k.execute(query)
    print query
    k = engine.dispose()
    return 'geom column added with SRID ' + str(projection)

table = 'results_approach2'
schema = 'lemma2016'
create_geom(table,schema,engine_db)


I am working on the file "Biomass/ReadFromDBv2-Copy1.ipynb".
The first task is:
The reason why we use postgis is to improve geospatial queries and provide a better data structure for geospatial operations. Many of the ST_ functions have improved performance when a geospatial index is created. The process implemented here comes from this workshop. This re-creates the process using python functions so that it can be easily replicated for many tables.
The query to create a geospatial index is as follows:
SQL
set search_path = SCHEMA, public;
CREATE INDEX vTABLE_gix ON vTABLE USING GIST (geom);
This assumes that the column name with the geometry is named geom. If the process follows from the previous code, it will work ok.
The following step is to run a VACUUM, creating an index is not enough to allow PostgreSQL to use it effectively. VACUUMing must be performed when ever a new index is created or after a large number of UPDATEs, INSERTs or DELETEs are issued against a table.
SQL
VACUUM ANALYZE vTABLE;
The final step corresponds to CLUSTERING, this process re-orders the table according to the geospatial index we created. This ensures that records with similar attributes have a high likelihood of being found in the same page, reducing the number of pages that must be read into memory for some types of queries. When a query to find nearest neighbors or within a certain are is needed, geometries that are near each other in space are near each other on disk. The query to perform this clustering is as follows:
CLUSTER vTABLE USING vTABLE_gix;
ANALYZE vTABLE;
Can you write Python code for it?


I am working on the file "Biomass/ReadFromDBv2.ipynb".
The first task is:
After importing the required packages, first create the engine to connect to the DB. The approach I generally use is to create a string based on the username and password. The code is a function, you just need to fill in with the username, password and the dbname.
It allows you to create different engines to connect to serveral dbs.
Can you write Python code for it?


I am working on the file "Biomass/ReadFromDBv2.ipynb".
The first task is:
Afterwards, use pandas to import the data from Excel files or any other text file format. Make sure that the data in good shape before trying to push it into the server. In this example I use previous knowledge of the structure of the tabs in the excel file to recursively upload each tab and match the name of the table with the tab name.
If you are using csv files just change the commands to pd.read_csv() in this link you can find the documentation.
Before doing this I already checked that the data is properly organized, crate new cells to explore the data beforehand if needed
excel_file = 'substations_table.xlsx'
tab_name = 'sheet1'
schema_for_upload = 'geographic_data'
pd_data.to_sql(name, engine_db, schema=schema_for_upload, if_exists='replace',chunksize=100)
Can you write Python code for it?


I am working on the file "Biomass/ReadFromDBv2.ipynb".
The first task is:
Once the data is updated, it is possible to run the SQL commands to properly create geom columns in the tables, this can be done as follows. The ojective is to run an SQL querie like this:
PGSQL
set search_path = SCHEMA, public;
alter table vTABLE drop column if exists geom;
SELECT AddGeometryColumn ('SCHEMA','vTABLE','geom',4326,'POINT',2);
UPDATE TABLE set geom = ST_SetSRID(st_makepoint(vTABLE.lon, vTABLE.lat), 4326)::geometry;
where SCHEMA and vTABLE are the variable portions. Also note, that this query assumes that your columns with latitude and longitude are named lat and lon respectively; moreover, it also assumes that the coordinates are in the 4326 projection.
The following function runs the query for you, considering again that the data is clean and nice.
Can you write Python code for it?

def create_geom(table,schema,engine, projection=5070):
    k = engine.connect()
    query = ('set search_path = "'+ schema +'"'+ ', public;')
    print query
    k.execute(query)
    query = ('alter table ' + table + ' drop column if exists geom;')
    print query
    k.execute(query)
    query = 'SELECT AddGeometryColumn (\''+ schema + '\',\''+ table + '\',\'geom\''+',5070,\'POINT\',2);'
    print query
    k.execute(query)
    query = ('UPDATE ' + table + ' set geom = ST_SetSRID(st_makepoint(' + table + '.x, ' +
             table + '.y),' + str(projection) + ')::geometry;')
    k.execute(query)
    print query
    k = engine.dispose()
    return 'geom column added with SRID ' + str(projection)

table = 'results_approach1'
schema = 'lemma2016'
create_geom(table,schema,engine_db)


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
1b. Obtaining a list of entries in a database with kegg_list()
The kegg_list() function retrieves a handle, which provides a plaintext tab-separated key:value list of entry identifiers and definitions in the given database or set of database entries.
The kegg_list() function allows the organism to be specified for a database, with an optional second argument.
Can you write Python code for it?

# List all pathways in the pathway database
head(kegg_list('pathway').read())

# Only list pathways present in E. coli K-12 MG1655
head(kegg_list('pathway', 'eco').read())


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
1c. Finding a specific entry with kegg_find()
The kegg_find(database, query) function finds entries with matching query keywords or other query data in a given database. Data is returned as tab-separated key:value plain text.
Can you write Python code for it?

# Find shiga toxin genes
head(kegg_find('genes', 'shiga+toxin').read())

# Find shiga toxin genes only in Escherichia coli O111 H-11128 (EHEC)
print(kegg_find('eoi', 'shiga+toxin').read())


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
1d. Retrieving specific entries with kegg_get()
The kegg_get() function lets us retrieve data from KEGG in a range of different formats.
Database entries can be retrieved directly as plain text; sequences may be returned as database entries, amino acid or nucleotide sequences in FASTA format; compounds as database entries, or images (.gif); pathways as database entries, KGML, or images (.png).
Compounds can be retrieved as their database entry in plain text, or as a .gif image.
Can you write Python code for it?

# Compound as database entry
head(kegg_get("cpd:C01290").read())

# Compound as image
Image(kegg_get("cpd:C01290", "image").read())


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
2b. Reproducing KEGG renderings from KGML
KEGG provides descriptions of their pathway maps in an exchange format called KGML (spec). This doesn't quite match the maps that can be downloaded through the public interface. If you're a subscriber to KEGG, then you have access to an alternative set of KGML+ files. The Biopython interface uses the public API, so we only work with KGML files.
To obtain a pathway's KGML file from KEGG, we can use the kegg_get() function. In the cell below, we store the downloaded data for two maps in the variables ko_map and eco_map.
Can you write Python code for it?

# Get KGML for fatty-acid biosynthesis
ko_map = (kegg_get("ko00061", "kgml").read())    # KO version (KEGG orthologues)
eco_map = (kegg_get("eco00061", "kgml").read())  # E. coli version


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
2c. Customised renderings from KGML - Part 1
In the Bio.Graphics model of rendering KGML, we are superimposing the elements described in a KGML file over an associated background image. As a result, only the elements that are described in the KGML file (or that we introduce ourselves...) can be modified.
The current interface to identifying and retrieving pathway elements is a little obscure as of Biopython 1.65 - if you want to retrieve a particular element by ID or annotation -  but it is possible to modify the graphical elements that are rendered, to produce customised maps from KGML data.
The modifications we can make are:

shape (graphics.type)
location ($x,y$ co-ordinates: graphics.x, graphics.y)
height (graphics.height)
width (graphics.width)
foreground colour (graphics.fgcolor)
background colour (graphics.bgcolor)
label (graphics.name)

but this still gives us a lot of flexibility in terms of visual annotation from a dataset. We can also choose to display, or not, elements of interest, and transparency effects are available.
A motivating example
As a motivating example, we're going to use the fatty-acid biosynthesis pathway (map00061) from earlier, and make the following modifications:

colour the ortholog elements according to a random choice of colours
change the size of each compound element to reflect its molecular mass

We saw the default rendering of the ko00061 pathway earlier:
Can you write Python code for it?

pathway = KGML_parser.read(kegg_get("ko00061", "kgml"))
canvas = KGMLCanvas(pathway, import_imagemap=True)
canvas.draw(os.path.join(outdir, "fab_map_with_image.pdf"))
PDF(os.path.join(outdir, "fab_map_with_image.pdf"))


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
2c.1 Changing ortholog background colours
We can get access to all the pathway ortholog elements via the pathway.orthologs attribute. This will list all ortholog elements in the pathway as Entry objects.
Can you write Python code for it?


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
2c.2 Changing compound sizes
We can get access to the pathway compound elements via pathway.compounds, just as for orthologs, and to change sizes, we can loop over them, modifying both width and height.
As with the colour changes, the sizes used are arbitrary, but this kind of modification (maybe incorporating colour changes, too) could be used to represent biochemical measurements such as metabolite concentration.
Can you write Python code for it?

# Change the sizes of compound elements
for size, element in zip(range(8, 8+len(pathway.compounds)), pathway.compounds):
    for graphic in element.graphics:
        graphic.width = size
        graphic.height = size

canvas = KGMLCanvas(pathway, import_imagemap=True)
canvas.draw(os.path.join(outdir, "fab_map_new_sizes.pdf"))
PDF(os.path.join(outdir, "fab_map_new_sizes.pdf"))


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
But kegg_list() and other functions will take a Python list of identifiers, and combine them automatically:
Can you write Python code for it?

# Compound entry C01290 and glycan entry G00092
print(kegg_list(['C01290', 'G00092']).read())


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
But there's something wrong with this picture. It shows only the elements defined in the KGML file. All the beautiful KEGG layout, including the lines indicating the flow of the biochemistry, is missing. That's because the KGML file doesn't contain any information about it, directly.
However, each Pathway object created from KGML file has an image attribute with the URL of its .png format representation. To use this as a background, all we need to do to render this is specify import_imagemap=True when creating the canvas, or set canvas.import_imagemap = True once the canvas is created. (It is also possible to specify an image file on disk as the background.)
Can you write Python code for it?

canvas.import_imagemap = True
canvas.draw(os.path.join(outdir, "fab_map_with_image.pdf"))
PDF(os.path.join(outdir, "fab_map_with_image.pdf"))


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
Each Entry object has a graphics attribute that stores information about how the item is rendered, in a list of Graphics objects - this reflects the data structure in the KGML representation, which may have more than one graphical entity per entry in the pathway.
Can you write Python code for it?


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
Each Graphics object has attributes that may be modified directly:
name         Label for the graphics object
x            X-axis position of the object (int)
y            Y-axis position of the object (int)
coords       polyline co-ordinates, list of (int, int) tuples
type         object shape
width        object width (int)
height       object height (int)
fgcolor      object foreground color (hex RGB)
bgcolor      object background color (hex RGB)

There are also two read-only properties that are provided for convenience
bounds       returns the graphical bounds of the object
centre       returns the centre of the object as an (x, y) tuple
Can you write Python code for it?

element = pathway.orthologs[0].graphics[0]
attrs = ["name", "x", "y", "coords", "type", "width", "height",
         "fgcolor", "bgcolor", "bounds", "centre"]
for a in attrs:
    print("element.{0: <10}:\t{1}".format(a, getattr(element, a)))


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
For this example we're defining arbitrary colours, but some obvious uses might include the use of colour to represent some biologically-significant value, like transcript abundance, $\frac{dN}{dS}$, or flux through that step.
Can you write Python code for it?

# Define arbitrary colours
colorspiral = ColorSpiral()
colorlist = colorspiral.get_colors(len(pathway.orthologs))

# Change the colours of ortholog elements
for color, element in zip(colorlist, pathway.orthologs):
    for graphic in element.graphics:
        graphic.bgcolor = rgb_to_hex(color)

canvas = KGMLCanvas(pathway, import_imagemap=True)
canvas.draw(os.path.join(outdir, "fab_map_new_colours.pdf"))
PDF(os.path.join(outdir, "fab_map_new_colours.pdf"))


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
KEGG also provides renderings that are specific to an organism, indicating which elements of the pathway are present in their database for that organism.
For central metabolism, compounds and reactions are only coloured if they are present in the specified organism. Otherwise, they are rendered in grey.
Can you write Python code for it?

# Render E.coli K-12 MG1655 central metabolism
Image(kegg_get("eco01100", "image").read())


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
Output files will be stored in the subdirectory kgml_output, which may need to be created:
Can you write Python code for it?

# Create subdirectory to store output
outdir = "kgml_output"
try:
    os.makedirs(outdir, exist_ok=True)
except TypeError:
    if not os.path.isdir(outdir):
        os.mkdir(outdir)


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
Pathways can be returned as database entries in plain text, or in the KGML format, or as .png images. These last two formats are especially useful for generating custom pathway renderings with Bio.Graphics.KGML.
Can you write Python code for it?

# Pathway as database entry
head(kegg_get("hsa05130").read())

# Pathway as image (png)
Image(kegg_get("hsa05130", "image").read())

# Pathway as KGML
head(kegg_get("hsa05130", "kgml").read())


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
The KEGG API also allows the + operator to be used to combine list identifiers.
Can you write Python code for it?

# Compound entry C01290 and glycan entry G00092
print(kegg_list('C01290+G00092').read())


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
This could be encapsulated in a useful helper function, as in the cell below:
Can you write Python code for it?

def draw_kegg_map(map_id, outdir):
    """Render a local PDF of a KEGG map with the passed map ID."""
    # Get the background image first
    pathway = KGML_parser.read(kegg_get(map_id, "kgml"))
    canvas = KGMLCanvas(pathway, import_imagemap=True)
    img_filename = "%s.pdf" % map_id
    canvas.draw(os.path.join(outdir, img_filename))


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
Three types of reference pathway can also be rendered: ko, ec, and rn. These are shown with blue boxes.
Can you write Python code for it?

# Render reference fatty-acid biosynthesis
#Image(kegg_get("ko00061", "image").read())
#Image(kegg_get("ec00061", "image").read())
Image(kegg_get("rn00061", "image").read())


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
To change the ortholog colours, it's possible to loop over the ortholog elements in the pathway, and modify their graphics.bgcolor attributes.
As KGML expects hex colours in its specification, we also provide hex colours here, using the helper function below:
Can you write Python code for it?

# Helper function to convert colour as RGB tuple to hex string
def rgb_to_hex(rgb):
    rgb = tuple([int(255*val) for val in rgb])
    return '#' + ''.join([hex(val)[2:] for val in rgb]).upper()


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
To render this data as an image, we need to set up a KGMLCanvas object, from Bio.Graphics.KGML_vis, to draw on. We do this by instantiating a KGMLCanvas object with our pathway data.
Can you write Python code for it?


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
To visualise the PDF output generated by this library inline, and to provide a head-like peek at text output, the functions below are defined:
Can you write Python code for it?

# A bit of code that will help us display the PDF output
def PDF(filename):
    return HTML('<iframe src=%s width=700 height=350></iframe>' % filename)

# A bit of helper code to shorten long text
def head(text, lines=10):
    """Print the first lines lines of the passed text."""
    print('\n'.join(text.split('\n')[:lines] + ['[...]']))


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
Typically, orthologs and compounds are the only items that can be manipulated on a KEGG pathway map. KGML does not describe the complete map, with its connectors and labels, as provided in the downloaded image.
The base maps (such as, in this case, map00061) do not contain ortholog or compound features, and consequently there is no associated KGML file. Attempting to retrieve KGML data for these maps produces an HTTP 404: not found error.
Can you write Python code for it?

# Trying to retrieve base map KGML generates an HTTP 404: not found error
#base_map = (kegg_get("map00061", "kgml").read())


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
Unlike the custom renderings that can be obtained from KEGG directly, with this module we now have full control over line width, element size, element colour, and labelling options.
To avoid noise in the image, we can set import_imagemap=False when making the canvas or drawing to a file for these maps.
First, let's set each reaction line to a random value between 1 and 10. This could be adapted to represent carbon flux measurements, for example.
Can you write Python code for it?

# Use the bacterial diverse environments map
pathway = KGML_parser.read(kegg_get("ko01120", "kgml"))

# Change the widths of reaction entries elements
for element in pathway.orthologs:
    for graphic in element.graphics:
        graphic.width = random.randrange(1, 10, 1)

canvas = KGMLCanvas(pathway, import_imagemap=False)
canvas.draw(os.path.join(outdir, "bacteria_mod_widths.pdf"))
PDF(os.path.join(outdir, "bacteria_mod_widths.pdf"))


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
We can modify pathway colours in the same way as before, except that now we have to modify the foreground, not the background colour, through the attribute fgcolor. The foreground colour should be specified as a hex string.
Can you write Python code for it?

# Define arbitrary colours
colorspiral = ColorSpiral()
colorlist = colorspiral.get_colors(len(pathway.orthologs))

# Change the colours of ortholog elements
for color, element in zip(colorlist, pathway.orthologs):
    for graphic in element.graphics:
        graphic.fgcolor = rgb_to_hex(color)

canvas = KGMLCanvas(pathway, import_imagemap=False)
canvas.draw(os.path.join(outdir, "bacteria_mod_colour.pdf"))
PDF(os.path.join(outdir, "bacteria_mod_colour.pdf"))


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
We're going to be using some other Biopython modules, and working with images in the notebook, so we also prepare for this, with the following imports.
Can you write Python code for it?

from Bio import SeqIO
from Bio.KEGG.REST import *
from Bio.KEGG.KGML import KGML_parser
from Bio.Graphics.KGML_vis import KGMLCanvas
from Bio.Graphics.ColorSpiral import ColorSpiral

from IPython.display import Image, HTML

import os
import random


I am working on the file "Biopython_KGML_intro.ipynb".
The first task is:
With other pathways, the reactions and/or genes are coloured green when they are present in the organism, and in white otherwise.
Can you write Python code for it?

# Render E.coli K-12 MG1655 fatty-acid biosynthesis
Image(kegg_get("eco00061", "image").read())


I am working on the file "Biopython_NCBI_Entrez_downloads.ipynb".
The first task is:
By running the cell below, you can see that each sequence in the Ralstonia solanacearum assembly has been downloaded into a SeqRecord, and that it contains useful metadata, describing the sequence assembly and properties of the annotation.
Can you write Python code for it?

# Show the contents of each downloaded `SeqRecord`.
for record in records:
    print(record, "\n")


I am working on the file "Biopython_NCBI_Entrez_downloads.ipynb".
The first task is:
Fetching sequence records from NCBI
Now we have accession UIDs for the nucleotide sequences of the assembly, you will use Entrez.efetch as before to fetch  each sequence record from NCBI.
We need to tell NCBI which database we want to use (in this case, nucleotide), and the identifiers for the records (the values in nuc_uids). To get all the data at the same time, we can join the accession ids into a single string, with commas to separate the individual UIDs.
We will also tell NCBI two further pieces of information:

The format we want the data returned in. We will ask for GenBank format (gbwithparts) to obtain the genome sequence and feature annotations.
How we want the data returned. We will ask for plain text (text).
Can you write Python code for it?

# The lines below retrieve (fetch) the GenBank records for
# each database entry specified in `nuc_uids`, in plain text
# format. These are parsed with Biopython's SeqIO module into
# SeqRecords, which structure the data into a usable format.
# The SeqRecords are placed in the variable 'records'.
records = []
for nuc_uid in nuc_uids:
    handle = Entrez.efetch(db="nucleotide", rettype="gbwithparts", retmode="text",
                          id=nuc_uid)
    records.append(SeqIO.read(handle, 'genbank'))


I am working on the file "Biopython_NCBI_Entrez_downloads.ipynb".
The first task is:
For now, we are interested in the list of database identifiers, in record['IdList']. We will use these to get information from the assembly database.
We will look at a single record first, and then consider how to get all the Ralstonia genomes at the same time.
4. Downloading a single genome from NCBI
In this section, you will use one of the database identifiers returned from your search at NCBI to identify and download the GenBank records corresponding to a single assembly of Ralstonia solanacearum.
To do this, we will select a single accession from the list in record["IdList"], using the code in the cell below.
<div class="alert alert-danger" role="alert">
Although this is a single assembly, with a single accession ID, we shall see that we need to download more than one sequence to cover the complete genome.
</div>
Can you write Python code for it?

# The line below takes the first value in the list of
# database accessions record["IdList"], and places it in
# the variable 'accession'
accession = record["IdList"][0]

# Show the contents of the variable 'accession'
print(accession)


I am working on the file "Biopython_NCBI_Entrez_downloads.ipynb".
The first task is:
If you inspect the newly-created ralstonia.gbk file, you should see that it contains complete GenBank records, describing this genome.
GenBank files are detailed and large, and sometimes we only want to consider the genome sequence itself, not its annotation. The FASTA sequence can be written out on its own by specifyinf the "fasta" format to SeqIO.write() instead. This time, we write the output to data/ralstonia.fasta.
Can you write Python code for it?

# The line below writes the sequence data in 'seqdata' to
# the local file "data/ralstonia.fasta", in FASTA format.
SeqIO.write(records, os.path.join(outdir, "ralstonia.fasta"), "fasta")


I am working on the file "Biopython_NCBI_Entrez_downloads.ipynb".
The first task is:
The links variable may contain links to more than one version of the genome (NCBI keep third-party managed genome data in GenBank/INSDC records, and NCBI-'owned' data in RefSeq records).
The function below extracts only the INSDC information from the Elink() query. It is not important that you understand the code.
Can you write Python code for it?

# The code below provides a function that extracts nucleotide
# database accessions for INSDC data from the result of an
# Entrez.elink() query.
def extract_insdc(links):
    """Returns the link UIDs for RefSeq entries, from the
    passed Elink search results"""
    # Work only with INSDC accession UIDs
    linkset = [ls for ls in links[0]['LinkSetDb'] if
              ls['LinkName'] == 'assembly_nuccore_insdc']
    if 0 == len(linkset):  # There are no INSDC UIDs
        raise ValueError("Elink() output has no assembly_nuccore_insdc data")
    # Make a list of the INSDC UIDs
    uids = [i['Id'] for i in linkset[0]['Link']]
    return uids


I am working on the file "Biopython_NCBI_Entrez_downloads.ipynb".
The first task is:
The variable record contains a list of the available databases at NCBI, which you can see by executing the cell below:
Can you write Python code for it?


I am working on the file "Biopython_NCBI_Entrez_downloads.ipynb".
The first task is:
Writing sequence data with Biopython
The SeqIO module can be used to write sequence data out to a file on your local hard drive. You will do this in the cells below, using the SeqIO.write() function.

The SeqRecords you downloaded contain sequence and feature annotation data, and can be written in any of several file formats. Some of these formats preserve annotation information, and some do not.

Firstly, in the cell below, you will write GenBank format files that preserve both sequence and annotation data. For the SeqIO.write() function, we need to specify the list of SeqRecords (records), the output filename to which they will be written, and the format we wish to write (in this case "genbank").
Can you write Python code for it?

# The line below writes the sequence data in 'seqdata' to
# the local file "data/ralstonia.gbk", in GenBank format.
# The function returns the number of sequences that were written to file
SeqIO.write(records, os.path.join(outdir, "ralstonia.gbk"), "genbank")


I am working on the file "Biopython_NCBI_Entrez_downloads.ipynb".
The first task is:
You may recognise some of the database names, such as pubmed, nuccore, assembly, sra, and taxonomy.
Entrez allows you to query these databases using Entrez.esearch() in much the same way that you just obtained the list of databases with Entrez.einfo().
3. Using Bio.Entrez to find genome assemblies at NCBI
In the cells below, you will use Bio.Entrez to identify assemblies for the bacterial plant pathogen Ralstonia solanacearum. As our interest is genome data, we will query against the assembly database at NCBI. This database contains entries for all genome assemblies, whether complete or draft.
We are interested in Ralstonia solanacearum, so will search against the assembly database with the text "Ralstonia solanacearum" as a query. The function that allows us to do this is Entrez.esearch(). By default, searches are limited to 20 results (as on the NCBI webpage), but we can change this.
Can you write Python code for it?

# The line below carries out a search of the `assembly` database at NCBI,
# using the phrase `Ralstonia solanacearum` as the search query,
# and asks NCBI to return up to the first 100 results
handle = Entrez.esearch(db="assembly", term="Ralstonia solanacearum", retmax=100)

# This line converts the returned information from NCBI into a form we
# can use, as before.
record = Entrez.read(handle)


I am working on the file "Biopython_get_GenBank_genomes.ipynb".
The first task is:
But what are our records? We can see their GenBank identifiers by looking at record["IdList"]:
Can you write Python code for it?


I am working on the file "Biopython_get_GenBank_genomes.ipynb".
The first task is:
Now we can recover links to the nucleotide database for each of these UIDs. There may be several such links, but as we are looking for the full assembly, we care only about the assembly_nuccore_insdc sequences, which are the contigs.
We collect these into a dictionary of contig UIDs, keyed by assembly UID, called sequid_links:
Can you write Python code for it?

sequid_links = {}
for uid in asm_links:
    links = Entrez.read(Entrez.elink(dbfrom="assembly", db="nucleotide", retmode="gb", from_uid=uid))
    contigs = [l for l in links[0]['LinkSetDb'] if l['LinkName'] == 'assembly_nuccore_insdc'][0]
    sequid_links[uid] = [e['Id'] for e in contigs['Link']]
expected_contigs = {}
print("There are {0} genomes identified for {1}:".format(len(sequid_links), genus))
for k, v in sorted(sequid_links.items()):
    print("Assembly UID {0}: {1} contigs".format(k, len(v)))
    expected_contigs[k] = len(v)


I am working on the file "Biopython_get_GenBank_genomes.ipynb".
The first task is:
Once we have these nucleotide database identifiers, we can grab all the sequences and write them out as multi-FASTA files, with Entrez.efetch(). The assembly records themselves though, have to be obtained with Entrez.esummary(), and then a byzantine set of keywords navigated, to get the information we're interested in.
We use the assembly UID without version number as the filename, and write a labels.txt file suitable for use with pyani.
Can you write Python code for it?

# Make sure there's a relevant output directory
if not os.path.exists(genus):
    os.mkdir(genus)
if not os.path.exists("failures"):
    os.mkdir("failures")

# Write output
with open(os.path.join(genus, 'labels.txt'), 'w') as lfh:
    with open(os.path.join(genus, 'classes.txt'), 'w') as cfh:
        for asm_uid, contigs in sorted(sequid_links.items()):
            # Get assembly record information
            asm_record = Entrez.read(Entrez.esummary(db='assembly', id=asm_uid, rettype='text'))
            asm_organism = asm_record['DocumentSummarySet']['DocumentSummary'][0]['SpeciesName']
            try:
                asm_strain = asm_record['DocumentSummarySet']['DocumentSummary'][0]['Biosource']['InfraspeciesList'][0]['Sub_value']
            except:
                asm_strain = ""
            gname = asm_record['DocumentSummarySet']['DocumentSummary'][0]['AssemblyAccession'].split('.')[0]
            filestem = os.path.join(genus, gname)
            # Write a labels.txt and a classes.txt file suitable for pyani
            glab, species = asm_organism.split(' ', 1)
            glab = glab[0]
            labelstr = "{0}\t{1}. {2} {3}".format(gname, glab, species, asm_strain)
            print >> lfh, labelstr
            print >> cfh, "{0}\t{1}".format(gname, asm_organism)
            print(labelstr)
            # Get FASTA records for each of the contigs (we could do this with GenBank instead,
            # but sometimes these are not formatted correctly with sequences)
            query_uids = ','.join(contigs)
            tries, success = 0, False
            while success == False and tries < 20:
                # Also check for total sequence length errors?
                try:
                    print("UID:{0} download attempt {1}".format(asm_uid, tries + 1))
                    records = list(SeqIO.parse(Entrez.efetch(db='nucleotide', id=query_uids,
                                                             rettype="fasta", retmode='text'),
                                              ...


I am working on the file "Biopython_get_GenBank_genomes.ipynb".
The first task is:
We can get the number of returned records by looking at record["Count"]:
Can you write Python code for it?


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
A prediction is an array of 10 numbers. These describe the "confidence" of the model that the image corresponds to each of the 10 different articles of clothing. We can see which label has the highest confidence value:
Can you write Python code for it?


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
Build the model
Building the neural network requires configuring the layers of the model, then compiling the model.
Setup the layers
The basic building block of a neural network is the layer. Layers extract representations from the data fed into them. And, hopefully, these representations are more meaningful for the problem at hand.
Most of deep learning consists of chaining together simple layers. Most layers, like tf.keras.layers.Dense, have parameters that are learned during training.
Can you write Python code for it?

model = keras.Sequential([
    keras.layers.Flatten(input_shape=(28, 28)),
    keras.layers.Dense(128, activation=tf.nn.relu),
    keras.layers.Dense(10, activation=tf.nn.softmax)
])


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
Each label is an integer between 0 and 9:
Can you write Python code for it?


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
Explore the data
Let's explore the format of the dataset before training the model. The following shows there are 60,000 images in the training set, with each image represented as 28 x 28 pixels:
Can you write Python code for it?


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
Finally, use the trained model to make a prediction about a single image.
Can you write Python code for it?

# Grab an image from the test dataset
img = test_images[0]

print(img.shape)


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
Let's plot several images with their predictions. Correct prediction labels are blue and incorrect prediction labels are red. The number gives the percent (out of 100) for the predicted label. Note that it can be wrong even when very confident.
Can you write Python code for it?

# Plot the first X test images, their predicted label, and the true label
# Color correct predictions in blue, incorrect predictions in red
num_rows = 5
num_cols = 3
num_images = num_rows*num_cols
plt.figure(figsize=(2*2*num_cols, 2*num_rows))
for i in range(num_images):
  plt.subplot(num_rows, 2*num_cols, 2*i+1)
  plot_image(i, predictions, test_labels, test_images)
  plt.subplot(num_rows, 2*num_cols, 2*i+2)
  plot_value_array(i, predictions, test_labels)


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
Likewise, there are 60,000 labels in the training set:
Can you write Python code for it?


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
Loading the dataset returns four NumPy arrays:

The train_images and train_labels arrays are the training set—the data the model uses to learn.
The model is tested against the test set, the test_images, and test_labels arrays.

The images are 28x28 NumPy arrays, with pixel values ranging between 0 and 255. The labels are an array of integers, ranging from 0 to 9. These correspond to the class of clothing the image represents:

    Label
    Class

    0
    T-shirt/top

    1
    Trouser

    2
    Pullover

    3
    Dress

    4
    Coat

    5
    Sandal

    6
    Shirt

    7
    Sneaker

    8
    Bag

    9
    Ankle boot

Each image is mapped to a single label. Since the class names are not included with the dataset, store them here to use later when plotting the images:
Can you write Python code for it?

class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
Now predict the image:
Can you write Python code for it?

predictions_single = model.predict(img)

print(predictions_single)

plot_value_array(0, predictions_single, test_labels)
_ = plt.xticks(range(10), class_names, rotation=45)


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
Preprocess the data
The data must be preprocessed before training the network. If you inspect the first image in the training set, you will see that the pixel values fall in the range of 0 to 255:
Can you write Python code for it?

plt.figure()
plt.imshow(train_images[0])
plt.colorbar()
plt.grid(False)


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
So the model is most confident that this image is an ankle boot, or class_names[9]. And we can check the test label to see this is correct:
Can you write Python code for it?


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
The first layer in this network, tf.keras.layers.Flatten, transforms the format of the images from a 2d-array (of 28 by 28 pixels), to a 1d-array of 28 * 28 = 784 pixels. Think of this layer as unstacking rows of pixels in the image and lining them up. This layer has no parameters to learn; it only reformats the data.
After the pixels are flattened, the network consists of a sequence of two tf.keras.layers.Dense layers. These are densely-connected, or fully-connected, neural layers. The first Dense layer has 128 nodes (or neurons). The second (and last) layer is a 10-node softmax layer—this returns an array of 10 probability scores that sum to 1. Each node contains a score that indicates the probability that the current image belongs to one of the 10 classes.
Compile the model
Before the model is ready for training, it needs a few more settings. These are added during the model's compile step:

Loss function —This measures how accurate the model is during training. We want to minimize this function to "steer" the model in the right direction.
Optimizer —This is how the model is updated based on the data it sees and its loss function.
Metrics —Used to monitor the training and testing steps. The following example uses accuracy, the fraction of the images that are correctly classified.
Can you write Python code for it?

model.compile(optimizer=tf.train.AdamOptimizer(),
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
Train the model
Training the neural network model requires the following steps:

Feed the training data to the model—in this example, the train_images and train_labels arrays.
The model learns to associate images and labels.
We ask the model to make predictions about a test set—in this example, the test_images array. We verify that the predictions match the labels from the test_labels array.

To start training,  call the model.fit method—the model is "fit" to the training data:
Can you write Python code for it?


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
We can graph this to look at the full set of 10 channels
Can you write Python code for it?

def plot_image(i, predictions_array, true_label, img):
  predictions_array, true_label, img = predictions_array[i], true_label[i], img[i]
  plt.grid(False)
  plt.xticks([])
  plt.yticks([])

  plt.imshow(img, cmap=plt.cm.binary)

  predicted_label = np.argmax(predictions_array)
  if predicted_label == true_label:
    color = 'blue'
  else:
    color = 'red'

  plt.xlabel("{} {:2.0f}% ({})".format(class_names[predicted_label],
                                100*np.max(predictions_array),
                                class_names[true_label]),
                                color=color)

def plot_value_array(i, predictions_array, true_label):
  predictions_array, true_label = predictions_array[i], true_label[i]
  plt.grid(False)
  plt.xticks([])
  plt.yticks([])
  thisplot = plt.bar(range(10), predictions_array, color="#777777")
  plt.ylim([0, 1])
  predicted_label = np.argmax(predictions_array)

  thisplot[predicted_label].set_color('red')
  thisplot[true_label].set_color('blue')


I am working on the file "Bipropagation_fashionMNIST.ipynb".
The first task is:
tf.keras models are optimized to make predictions on a batch, or collection, of examples at once. So even though we're using a single image, we need to add it to a list:
Can you write Python code for it?

# Add the image to a batch where it's the only member.
img = (np.expand_dims(img,0))

print(img.shape)


I am working on the file "Birthday-problem-ro.ipynb".
The first task is:
Acum repetam experimentul aleator de $N$ ori cu acelasi nr de participanti si calculam nr de coincidente in fiecare
incercare:
Can you write Python code for it?

N=1000
nrP=30
Lcoinc=[bdayExper(nrP=30) for k in range(N)]# lista numarului de coincidente in fiecare incercare din cele N

#E mai simplu sa numaram cate incercari exista fara nicio coincidenta:
NrCoinc0=Lcoinc.count(0)# numarul de incercari cu nicio coincidenta

print  'Probabilitate experimentala a cel putin unei coincidente intr-un grup de ', nrP, 'persoane este:',\
       1.0*(N-NrCoinc0)/N, '\n', 'Probabilitatea teoretica:', 1-prob_theor(nrP)

print 'Lista numarului de coincidente in fiecare incercare din cele N', Lcoinc

from IPython.core.display import HTML
def  css_styling():
    styles = open("./custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Birthday-problem-ro.ipynb".
The first task is:
Afisam probabilitatile calculate:
Can you write Python code for it?

import numpy as np
print np.array(probsEn).round(3)


I am working on the file "Birthday-problem-ro.ipynb".
The first task is:
Calculam acum probabilitatile de coincidenta a cel putin 2 zile de nastere intr-un grup de $n$ persoane cu
$2\leq n\leq 60$ si le reprezentam grafic:
Can you write Python code for it?

nrPartic=60
probsEn=[1- prob_theor(n) for n in range(2,nrPartic+1)]
n=[ k for k in range(2, nrPartic+1)]
fig = plt.figure()
ax = fig.add_subplot(111)
ax.stem(n, probsEn,  'b', bottom=0)
ax.set_xlabel('n')
ax.set_ylabel(r'$P(E_n)$')
ax.set_title('Probabilitatile $P(E_n)$, $n=2,\ldots, 60$')


I am working on the file "Birthday-problem-ro.ipynb".
The first task is:
Pentru a incepe numararea de la 1, nu de la 0, setam  startul  astfel:
Can you write Python code for it?


I am working on the file "Birthweight Regression.ipynb".
The first task is:
Build a model by fitting a line to data in a Table
Selecting a column of a Table yields a numpy array, allowing the numpy numerical
tools to be used for fitting curves to the data.  For documentation on the polyfit
function, try help(np.polyfit).  It returns the polynomial coefficients, highest power first, which is the slope for a line.
A linear model is only meaningful in the range around the normal gestation period,
and indeed the intercept is not physcially meaningful
Can you write Python code for it?

# what is the coeeficients of the line fitted to these data?

np.polyfit(nosmoke['Gest'],nosmoke['Wgt'],1)

np.polyfit(smoke['Gest'],smoke['Wgt'],1)


I am working on the file "Birthweight Regression.ipynb".
The first task is:
Capturing statistical computations and general tools
Rather than compute the null hypothesis for this particular table, we can build a very general tool, as a function, that will do it in general.
Then we can use it to build a sample distribution under the null hypothesis
Can you write Python code for it?

def null_diff_at(tbl, x, y, w):
    A, B = tbl.split(tbl.num_rows//2)
    return make_lm(A[x], A[y])(w) - make_lm(B[x], B[y])(w)

null_diff_at(births, 'Gest', 'Wgt', 40)

null = Table().with_column('Diff', [null_diff_at(births, 'Gest', 'Wgt', 40) for i in range(1000)])
null.hist()


I am working on the file "Birthweight Regression.ipynb".
The first task is:
Determining if the conclusion is sound
At this point, we might ask how accurately these linear models fit the data.  The 'residual' in the fit would give us some idea of this.  But the error in summarizing a collection of empirical data with an analytical model is only a part of the question.
The deeper point is that this data is not "truth", it is merely a sample of a population, and a tiny one at that.  We should be asking, is an inference drawn from this data valid for the population that the data is intended to represent.  Of course, the population is not directly observable, only the sample of it.  How can we use the sample we have to get some idea of how representative it is of the larger population.  That is what bootstrap seeks to accomplish.
Tables provide a method sample for just this purpose.  Here we return to looking at
the coefficients, rather than build a function for the model.
Can you write Python code for it?

# Construct a new model by forming a new sample from our existing one and fiting a line to that
# Here we create quite a general function, which takes a table and column names over which
# the model is to be formed.
def rboot(table, x, y):
    sample = table.sample(table.num_rows, with_replacement=True)
    return np.polyfit(sample[x],sample[y],1)

# Try it out for non-smokers.  Note that every time this cell is evaluated (ctrl-enter)
# the result is a little different, since a new sample is drawn.
rboot(nosmoke, 'Gest', 'Wgt')

# And for smokers
rboot(smoke,'Gest','Wgt')


I am working on the file "Birthweight Regression.ipynb".
The first task is:
Drawing a conclusion
Using the models to remove the contribution due to gestation time, we can attempt to draw a conclusion about the effect of smoking at typical gestation age.
Can you write Python code for it?

# based on this data set, fitting the data to models of weight as a function of gestation
# We might conclude that at 40 weeks the effect of smoking on birthweigth in grams is
smoke_diff = nosmoker_weight(40) - smoker_weight(40)
smoke_diff

# Or in relative terms
"{0:.1f}%".format(100*(nosmoker_weight(40)-smoker_weight(40))/nosmoker_weight(40))


I am working on the file "Birthweight Regression.ipynb".
The first task is:
Estimation of birthweights at 40 weeks
Selecting a column of a Table yields a numpy array.  Arithmetic operators work
elementwise on the entire array.
Can you write Python code for it?

smokesamples['slope']*40+smokesamples['intercept']

# So now we have an estimate of the distribution of birthweights at week 40 for
# something closer to the populations that these small samples represent.
weights_40 = Table().with_columns([
        ('nosmoke', nosamples['slope']*40+nosamples['intercept']),
        ('smoke', smokesamples['slope']*40+smokesamples['intercept'])])
weights_40

weights_40['Smoke Wgt Loss'] = weights_40['nosmoke'] - weights_40['smoke']

# what do we expect the distribution of birthweight reduction due to smoking to look like
# for the population represented by the original sample?
weights_40.select('Smoke Wgt Loss').hist(bins=30,normed=True)

smoke_diff

def firstQtile(x) : return np.percentile(x,25)
def thirdQtile(x) : return np.percentile(x,25)
summary_ops = (min, firstQtile, np.median, np.mean, thirdQtile, max)

summary = weights_40.stats(summary_ops)
summary

summary['diff']=summary['nosmoke']-summary['smoke']

# the bottom line
summary


I am working on the file "Birthweight Regression.ipynb".
The first task is:
Fitting a line to the data on a scatter plot
Here we drop the smoke column and look at the birthweight for the whole population.
Can you write Python code for it?

# As there is a trend among birthweight and gestation, we can show a fit line to try to
# capture it
births.drop('Smoke').scatter('Gest', fit_line=True)


I am working on the file "Birthweight Regression.ipynb".
The first task is:
Looking at the raw data
Table.scatter produces a scatter plot of columns versus a specific column.  Here we look at how birthweight varies with gestation. And we note whether the mother smoked.
Can you write Python code for it?

# First let's just look at what is here.  This needs to be a scatter, rather
# than a plot because there is no simple ordering - just relationships between
# birthweight and gestation time along with whether the mother smokes.
births.scatter('Gest')

births.num_rows

# How many samples in each category
births.where('Smoke').num_rows


I am working on the file "Birthweight Regression.ipynb".
The first task is:
Partitioning data
The question is whether smoking causes the trend to be substantially different.
Split the data into two tables using the Smoke column.  Then we can find the trends for each.
Can you write Python code for it?

nosmoke=births.where('Smoke',0).drop('Smoke')
smoke=births.where('Smoke',1).drop('Smoke')

# we can attempt to find the trend for each
nosmoke.scatter('Gest',fit_line=True)

smoke.scatter('Gest',fit_line=True, marker='x')


I am working on the file "Birthweight Regression.ipynb".
The first task is:
Use the models to build a Table and visualize the effect of the categorical parameter - smoking
Can you write Python code for it?

# Create a table with a column containing the independent variable
estimated_birthweight = Table().with_column('week', np.arange(32,44))
# Add columns of dependent variables
estimated_birthweight['nosmoke'] = estimated_birthweight.apply(nosmoker_weight,'week')
estimated_birthweight['smoke'] = estimated_birthweight.apply(smoker_weight,'week')
estimated_birthweight

# plot it to visualize
estimated_birthweight.plot('week',overlay=True)


I am working on the file "Birthweight Regression.ipynb".
The first task is:
We see that both the constant and the weight increase per week is lower for the smokers.
Higher order functions as models
At this point, we could do mx+b all over the place, or we could utilize higher order functions to capture the concept of a model.  Here is an example, building such a model directly from the data.  It takes the set of x and y values and returns a function
that evaluates the model built from that data at a particular x.
Can you write Python code for it?

# Build a linear model from data and return it is a function
def make_lm(x, y):
    m,b = np.polyfit(x, y, 1)
    def lm(a):
        return m*a+b
    return lm

# Create model of non-smokers that returns estimated weight a function of weeks gestation
nosmoker_weight = make_lm(nosmoke['Gest'], nosmoke['Wgt'])

# Evaluate it at normal gestations
nosmoker_weight(40)

smoker_weight = make_lm(smoke['Gest'], smoke['Wgt'])

smoker_weight(40)


I am working on the file "Birthweight Regression.ipynb".
The first task is:
What is the probablility that we got a 260g difference in birthweight at 40 weeks as an artifact of the sample?
Zero
Can you write Python code for it?


I am working on the file "BirthweightRegression.ipynb".
The first task is:
And we repeate this for the other category.
Can you write Python code for it?

smokesamples = Table(['slope','intercept']).with_rows([rboot(smoke,'Gest','Wgt') for i in range(num_samples)])
smokesamples.hist(bins=50,normed=True, overlay=False)


I am working on the file "BirthweightRegression.ipynb".
The first task is:
Bootstrap
Using this model builder as a function, draw many samples and form a model for each.
Then we can look at the distribution of the model parameters over lots of models.
This illustrates the construction of tables by rows.  The Table constructor accepts the
column names and with_rows fills them in row by row.
hist forms and shows a histogram of the result.
Can you write Python code for it?

# Bootstrap a distribution of models by drawing many random samples, with replacement, from our samples
num_samples = 1000
nosamples = Table(['slope','intercept']).with_rows([rboot(nosmoke,'Gest','Wgt') for i in range(num_samples)])
nosamples.hist(bins=50,normed=True, overlay=False)


I am working on the file "BirthweightRegression.ipynb".
The first task is:
Capturing statistical computations and general tools
Rather than compute the null hypothesis for this particular table, we can build a very general tool, as a function, that will do it in general.
Then we can use it to build a sample distribution under the null hypothesis
Can you write Python code for it?


I am working on the file "BirthweightRegression.ipynb".
The first task is:
Drawing a conclusion
Using the models to remove the contribution due to gestation time, we can attempt to draw a conclusion about the effect of smoking at typical gestation age.
Can you write Python code for it?


I am working on the file "BirthweightRegression.ipynb".
The first task is:
Empirical p-values
A more formal approach would be to take as the null hypothesis that smoking does not affect the birthweight.  Repeatedly split the data into random halves and model the birthweight difference.  What is the chance that the difference we see in summary table is an artifact?
Can you write Python code for it?

# As an example, split the original data into two random halves
A, B = births.split(births.num_rows//2)

A

B

make_lm(A['Gest'], A['Wgt'])(40)

make_lm(B['Gest'], B['Wgt'])(40)


I am working on the file "BirthweightRegression.ipynb".
The first task is:
Estimation of birthweights at 40 weeks
Selecting a column of a Table yields a numpy array.  Arithmetic operators work
elementwise on the entire array.
Can you write Python code for it?


I am working on the file "BirthweightRegression.ipynb".
The first task is:
Fitting a line to the data on a scatter plot
Here we drop the smoke column and look at the birthweight for the whole population.
Can you write Python code for it?


I am working on the file "BirthweightRegression.ipynb".
The first task is:
Looking at the raw data
Table.scatter produces a scatter plot of columns versus a specific column.  Here we look at how birthweight varies with gestation. And we note whether the mother smoked.
Can you write Python code for it?


I am working on the file "BirthweightRegression.ipynb".
The first task is:
Partitioning data
The question is whether smoking causes the trend to be substantially different.
Split the data into two tables using the Smoke column.  Then we can find the trends for each.
Can you write Python code for it?


I am working on the file "BirthweightRegression.ipynb".
The first task is:
Read data from a local or remote csv file to create a Table
Can you write Python code for it?

# Source https://onlinecourses.science.psu.edu/stat501/node/380
births=Table.read_table("data/birthsmokers.csv")
births


I am working on the file "BirthweightRegression.ipynb".
The first task is:
Use the models to build a Table and visualize the effect of the categorical parameter - smoking
Can you write Python code for it?


I am working on the file "BirthweightRegression.ipynb".
The first task is:
We see that both the constant and the weight increase per week is lower for the smokers.
Higher order functions as models
At this point, we could do mx+b all over the place, or we could utilize higher order functions to capture the concept of a model.  Here is an example, building such a model directly from the data.  It takes the set of x and y values and returns a function
that evaluates the model built from that data at a particular x.
Can you write Python code for it?


I am working on the file "BirthweightRegression.ipynb".
The first task is:
What is the probablility that we got a 260g difference in birthweight at 40 weeks as an artifact of the sample?
Zero
Can you write Python code for it?


I am working on the file "BitCoinInContextDE.ipynb".
The first task is:
Für Deutschland ergibt sich damit ein
Can you write Python code for it?


I am working on the file "BitCoinInContextDE.ipynb".
The first task is:
Wieviele Länder mit ähnlicher Staatsverschuldung könnte man damit abbilden? Und was ist mit dem BIP der Welt?
Can you write Python code for it?

bip_wo_units= bip_wo*100
print("Abbildbare Staatsverschuldungen: %.2f" %(btcunits/verschuldungunits))
print("Anteil Welt BIP: %.2f%%" %(100*btcunits/bip_wo_units))


I am working on the file "BitCoinInContextDE.ipynb".
The first task is:
Würde man diesen Markt auflösen, wieviel würde dann jeder Einwohner Deutschlands bekommen? Wie verhält sich das mit BIP und Staatsverschuldung?
Can you write Python code for it?

print("USD pro Einwohner dank BitCoins: %.2f" %(marktkap/einwohner))
print("Anteil am BIP: %.4f%%" %(marktkap/bip_de))
print("Anteil an der Staatsverschuldung: %.4f%%" %(marktkap/verschuldung))


I am working on the file "BitCoinInContextDE.ipynb".
The first task is:
nett
Mir war vorher nicht klar, in welchen Dimensionen von Geld hier gedacht werden muss. Die Maximale Anzahl von BitCoins ist 21,0 Millionen und jedes BitCoin lässt sich in 100 Millionen Satoshi (sat) aufteilen. Wir haben also
Can you write Python code for it?

btcunits=21*mil*100*mil # Unteilbare Einheiten


I am working on the file "BitCoinInContextDE.ipynb".
The first task is:
sanity check
https://de.wikipedia.org/wiki/Liste_europ%C3%A4ischer_L%C3%A4nder_nach_Staatsschuldenquote spricht von 68,2 mit der
Anmerkung dies sei eine Schätzung. Das ist ok für mich ich will ja nur die Größenordnung von BitCoins wissen.
Marktkapitalisierung von BitCoin:
Auf https://blockchainbdgpzk.onion/ (resp https://blockchain.info) bekommt man die Marktkapetalisierung angezeigt.
Stand: Dec 25 2017
Can you write Python code for it?


I am working on the file "Bitcoin Anomaly Analysis.ipynb".
The first task is:
Anomaly Scores
Can you write Python code for it?

import seaborn as sns
import matplotlib.pyplot as plt

plt.figure(figsize=(12, 8))
sns.distplot(scores, kde=False)
line = plt.vlines(clf.threshold_, 0, 30000, colors='r', linestyles='dotted')
line.set_label('Threshold = -0.0948')
plt.legend(loc='upper left', fontsize='medium')
plt.title('Anomaly Scores returned by Isolation Forest', fontsize=16);