I am working on the file "Notebooks/EURPLN_exchange_rate_analysis.ipynb".
The first task is:
Plot exchange rates along with different SMA
Can you write Python code for it?

plt.figure(figsize=(20,7))

plt.plot(recent_exchange_rates)
# sma-10
plt.plot(df_sma[10][-last_days_to_observe:].values)
# sma-20
plt.plot(df_sma[20][-last_days_to_observe:].values)
# sma-50
plt.plot(df_sma[50][-last_days_to_observe:].values)

plt.legend(['exchange rates', 'SMA-10', 'SMA-20', 'SMA-50'], loc='upper left')


I am working on the file "Notebooks/EURPLN_exchange_rate_analysis.ipynb".
The first task is:
SMA - simple moving average
Can you write Python code for it?

# an example on how to calculate SMA for a collection of data
v = {'score': [1, 2, 2, 3, 3, 4]}
data_frame = pd.DataFrame(v)
pd.rolling_mean(data_frame, 2)


I am working on the file "Notebooks/EURPLN_exchange_rate_analysis.ipynb".
The first task is:
Show first 3 rows from file
Can you write Python code for it?


I am working on the file "Notebooks/EURPLN_exchange_rate_analysis.ipynb".
The first task is:
What's wrong with intersection between 40-50??
Can you write Python code for it?

x1 = 40
x2 = 50

test_sma_1 = df_sma[3][-200+x1:-200+x2].values
test_sma_2 = df_sma[10][-200+x1:-200+x2].values

intpoints_v1 = np.argwhere(np.isclose(test_sma_1, test_sma_2, atol=0.01)).reshape(-1)
intpoints_v2 = np.argwhere(np.isclose(test_sma_1, test_sma_2, atol=0.02)).reshape(-1)
intpoints_v3 = np.argwhere(np.isclose(test_sma_1, test_sma_2, atol=0.03)).reshape(-1)

f, (ax1, ax2, ax3) = plt.subplots(3, sharex=True, sharey=True)
ax1.set_title("atol=0.01")
ax1.plot(test_sma_1)
ax1.plot(test_sma_2)
ax1.plot(intpoints_v1, test_sma_1[intpoints_v1], 'bo')
ax1.plot(intpoints_v1, test_sma_2[intpoints_v1], 'go')
ax1.plot(6, test_sma_1[6], 'ro')
ax1.plot(7, test_sma_1[7], 'ro')
ax1.plot(6, test_sma_2[6], 'ro')
ax1.plot(7, test_sma_2[7], 'ro')

ax2.set_title("atol=0.02")
ax2.plot(test_sma_1)
ax2.plot(test_sma_2)
ax2.plot(intpoints_v2, test_sma_1[intpoints_v2], 'bo')
ax2.plot(intpoints_v2, test_sma_2[intpoints_v2], 'go')

ax3.set_title("atol=0.03")
ax3.plot(test_sma_1)
ax3.plot(test_sma_2)
ax3.plot(intpoints_v3, test_sma_1[intpoints_v3], 'bo')
ax3.plot(intpoints_v3, test_sma_2[intpoints_v3], 'go')


I am working on the file "Notebooks/Ejemplo-Difusion-Calor-1D.ipynb".
The first task is:
Análogamente, para la segunda solución, creamos el archivo difusion-calor-1D-escalon.gif:
Can you write Python code for it?

x = linspace(-5,5,1000)

fig, ax = plt.subplots()
ax.set_ylim(0,1.2)
ax.grid()
ax.set_xlim(-5,5)
ax.set_xlabel('$x$',fontsize=15)
ax.set_ylabel("$\psi(x,t)$",fontsize=15)
ax.legend()

line = ax.plot(x, Psi2(x,0), lw=2)[0]

def animate(i):
    t = 0.2*i
    ax.set_title('Difusión del calor $1D$: Escalón, $t=%.1f$' %t)
    line.set_ydata(Psi2(x,t))

anim = FuncAnimation(fig, animate, interval=100, frames=200) #100 msec entre frames

anim.save('difusion-calor-1D-Escalon.gif', writer='imagemagick')


I am working on the file "Notebooks/Ejemplo-Difusion-Calor-1D.ipynb".
The first task is:
Con esto, la solución es de la forma (consideramos $a=1$, $T_0=1$ y $\alpha=0.01$)
Can you write Python code for it?

a = 1
alpha = 0.01
def Psi2(x,t):
    return (erf((x+a)/(sqrt(4*alpha*t))) + erf((a-x)/(sqrt(4*alpha*t))))/2.


I am working on the file "Notebooks/Ejemplo-Difusion-Calor-1D.ipynb".
The first task is:
Definimos una función para evaluar nuestra solución
Can you write Python code for it?

def Psi1(x,t,alpha):
    return (1/(2*sqrt(pi*alpha*t)))*exp(-x**2/(4.*alpha*t))


I am working on the file "Notebooks/Ejemplo-Difusion-Calor-1D.ipynb".
The first task is:
Graficamos ahora en forma interactiva
Can you write Python code for it?


I am working on the file "Notebooks/Ejemplo-Difusion-Calor-1D.ipynb".
The first task is:
Otra forma interesante de representar la solución es por medio de un gráfico de colores:
Can you write Python code for it?

fig = plt.figure(figsize=(10,0.2))
ax = plt.subplot()
ax.set_xlim(-5,5)
ax.set_xlabel(r'$x$',fontsize=15)
ax.set_yticklabels([])

x = linspace(-5,5,1000)
y = linspace(0,0.2,2)
X,Y = meshgrid(x,y)
Z = Psi2(X,0)

cax = ax.pcolormesh(X, Y, Z, cmap='plasma')

def animate(i):
    t = 0.05*i
    ax.set_title('Difusión del calor $1D$: Escalón, $t=%.1f$' %t)
    cax.set_array(Psi2(X,t).flatten())

anim = FuncAnimation(fig, animate, interval=50, frames=100) #100 msec entre frames=10fps

anim.save('difusion-calor-1D-escalon-colores.gif', writer='imagemagick')


I am working on the file "Notebooks/Ejemplo-Difusion-Calor-1D.ipynb".
The first task is:
Y otra función que la grafica, para valor de $t$ y $\alpha$ dados:
Can you write Python code for it?

x = linspace(-50,50,1000)
def g1(t,alpha):
    plt.plot(x,Psi1(x,t,alpha), label='$\psi(x,%d)$' %t)
    plt.xlim(-50,50)
    plt.ylim(0,Psi1(0,0.1,alpha))
    plt.grid(True)
    plt.xlabel('$x$',fontsize=15)
    plt.ylabel('$\psi(x,t)$',fontsize=15)
    plt.title('Difusión del calor $1D$, $\phi(x)=\delta(x)$, $t=%.1f$' %t)
    plt.legend()


I am working on the file "Notebooks/Ejemplo-MSV-Difusion-Calor-2D-rectangulo.ipynb".
The first task is:
Definimos una función para evaluar nuestra solución
Can you write Python code for it?

def Psi(x,y,t):
    S = 0
    for n in range(1,nmax+1):
        for m in range(1,mmax+1):
            S += A(n,m)*sin(n*pi*x/a)*sin(m*pi*y/b)*exp(-pi**2*alpha*((n/a)**2+(m/b)**2)*t)
    return S

def A(n,m):
    return 4*T0*sin(n*pi*x0/a)*sin(m*pi*y0/b)


I am working on the file "Notebooks/Ejemplo-MSV-Difusion-Calor-2D-rectangulo.ipynb".
The first task is:
Exportando animaciones a archivos .gif
Es posible crear una animación de la solución usando la función FuncAnimation. El código siguiente genera el archivo difusion-calor-1D-Delta.gif con la animación de la primera solución:
Can you write Python code for it?

from matplotlib.animation import FuncAnimation

fig = figure()
ax = subplot()
ax.set_xlabel(r'$x$',fontsize=15)
ax.set_ylabel(r'$y$',fontsize=15)

cax = ax.imshow(Psi(X,Y,0.001), origin='lower', extent=[0, a, 0, b], vmax=100, vmin=0)
ax.set_aspect('equal')

def animate(i):
    t = 0.001*i
    ax.set_title(r'$\Psi(x,y,t), t=%.3f$' %t)
    cax.set_array(Psi(X,Y,t))

anim = FuncAnimation(fig, animate, interval=100, frames=30) #100 msec entre frames=10fps

anim.save('difusion-calor-2D-rectangulo.gif', writer='imagemagick')


I am working on the file "Notebooks/Ejemplo-MSV-Difusion-Calor-2D-rectangulo.ipynb".
The first task is:
Finalmente, una animación de la representación 3D de la misma solución.
Can you write Python code for it?

fig = figure()
ax = fig.gca(projection='3d')

# animation function
def animate(i):
    t = 0.001+0.001*i
    animate.p.remove()
    animate.p = ax.plot_surface(X, Y, Psi(X,Y,t), cmap=cm.jet, rstride=3, cstride=2, linewidth=0.1, vmax=100, vmin=0)
    return animate.p,

animate.p = ax.plot_surface(X, Y, Psi(X,Y,0.001), cmap=cm.jet, rstride=3, cstride=2, linewidth=0.1, vmax=100, vmin=0)
ax.set_zlim(0,100)
ax.set_xlabel('$x$')
ax.set_ylabel('$y$')
#ax.set_aspect('equal', 'datalim')
#ax.set_axis_off()
ax.set_title('$\psi(x,y,t)$',fontsize=15)
anim = FuncAnimation(fig, animate, interval=100, frames=30, repeat=True, blit=True)
anim.save('difusion-calor-2D-rectangulo-02.gif', writer='imagemagick')


I am working on the file "Notebooks/Ejemplo-MSV-Difusion-Calor-2D-rectangulo.ipynb".
The first task is:
Por ejemplo, podemos graficar la solución para $t=0.01$
Can you write Python code for it?


I am working on the file "Notebooks/Ejemplo-MSV-Difusion-Calor-2D-rectangulo.ipynb".
The first task is:
Y otra función que la grafica:
Can you write Python code for it?

def g(t):
    imshow(Psi(X,Y,t), origin='lower', extent=[0, a, 0, b], vmax=100)
    xlabel(r'$x$',fontsize=15)
    ylabel(r'$y$',fontsize=15)
    title(r'$\Psi(x,y,t), t=%.3f$' %t)
    colorbar()
    axes().set_aspect('equal')

a = 1.
b = 2.
alpha = 1.
T0 = 1.
nmax = 30
mmax = 30
x0 = 0.6*a
y0 = 0.7*b

x=linspace(0,a,200)
y=linspace(0,b,200)
X,Y = meshgrid(x,y)


I am working on the file "Notebooks/Ejemplo-MSV-Ecuacion-Onda-1D.ipynb".
The first task is:
Ahora definimos la suma parcial $S(n_{\rm max},x,t)$ dada pr
$$
S(n_{\rm max},x,t)=\sum_{n =1}^{n_{\rm max}}B_{n}\sin\left(\frac{n \pi v}{L}t\right)\sin(\frac{n\pi x}{L}).
$$
Can you write Python code for it?

def S(nmax,x,t):
    S = 0
    for n in range(1,nmax+1):
        S = S + B(n)*sin(((n*pi*v)/L)*t)*sin(((n*pi)/L)*x)
    return S


I am working on the file "Notebooks/Ejemplo-MSV-Ecuacion-Onda-1D.ipynb".
The first task is:
Finalmente, podemos crear una animación:
Can you write Python code for it?

from matplotlib.animation import FuncAnimation

fig, ax = subplots()

ax.set_ylim(-1,1)
ax.grid()
ax.set_xlim(0,L)
ax.set_xlabel('$x$',fontsize=15)
ax.set_ylabel('$\Psi(x,t)$',fontsize=15)
#ax.yticks([])
#ax.xticks([0,L,xi,xi-d/2.,xi+d/2.],['$0$','$L$',r'$\xi$',r'$\xi-d/2$',r'$\xi+d/2$'])
ax.legend()
ax.set_title('Cuerda con extremos fijos, $t=0$')

line = ax.plot(x,S(nmax,x,0))[0]

def animate(i):
    t = 0.05*i
    ax.set_title(ur'Cuerda con extremos fijos, $t=%.1f$' %t)
    line.set_ydata(S(nmax,x,t))

anim = FuncAnimation(fig, animate, interval=100, frames=40) #100 msec entre frames

anim.save('fig-cuerda-1D-01.gif', writer='imagemagick')


I am working on the file "Notebooks/Ejemplo-MSV-Ecuacion-Onda-1D.ipynb".
The first task is:
Y preparamos el gráfico correspondiente a cada valor de $t$:
Can you write Python code for it?


I am working on the file "Notebooks/Ejemplo-MSV-Laplace-polares-numerico.ipynb".
The first task is:
A continuación, un gráfico en colores en el dominio.
Can you write Python code for it?

fig = figure()
pcolormesh(X, Y, Z, shading='gouraud')
xlabel('$x$',fontsize=15)
ylabel('$y$',fontsize=15)
title('$\psi(x,y)$',fontsize=15)
colorbar()
axes().set_aspect('equal', 'datalim')


I am working on the file "Notebooks/Ejemplo-MSV-Laplace-polares-numerico.ipynb".
The first task is:
Ahora evaluamos algunos parámetros que usaremos para confeccionar el gráfico:
Can you write Python code for it?

n_ang = 120 #número de divisiones de la coordenada angular
n_radio = 50 #número de divisiones de la coordenada radial
radio = linspace(0.001, R, n_radio) #se define coordenada radial entre 0 y R
ang = linspace(0, 2*pi, n_ang) #se define coordenada angular entre 0 y 2 veces pi

r,p = meshgrid(radio,ang)

X = r*cos(p) #Define dominio en x
Y = r*sin(p) #Define dominio en y
n_max = 20 #número máximo de terminos de la serie truncada
Z = S(n_max, p, r) #valor del potencial


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Epiciclos.ipynb".
The first task is:
Calcula los coeficientes de Fourier
Can you write Python code for it?

tau = max(t)-min(t)
order = 10
coef = coef_list(t, x, y, order)
for id,n in enumerate(range(-order,order+1)):
    print('n = ',str(n),': ',coef[id])


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Epiciclos.ipynb".
The first task is:
Crea animación
Can you write Python code for it?

anim = visualize(x_Sn, y_Sn, tau, coef, order, space, [xmin, xmax, ymin, ymax])
#Writer = animation.writers['html']
Writer = animation.writers['ffmpeg']
writer = Writer(fps=60)
anim.save('ani.mp4',writer=writer, dpi=150)


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Epiciclos.ipynb".
The first task is:
Ejemplo 1: Un cuadrado
Can you write Python code for it?


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Epiciclos.ipynb".
The first task is:
Ejemplo 2: Elipse
Can you write Python code for it?


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Epiciclos.ipynb".
The first task is:
Ejemplo 3: Corazón
$$
r(\varphi) = \frac{\sin(\varphi)\sqrt{|\cos(\varphi)|}}{\sin(\varphi)+\frac{7}{5}}-2\sin(\varphi)+2
$$
Can you write Python code for it?

#t = np.linspace(0,2*np.pi,1000)
#r = np.sin(t)*np.sqrt(np.abs(np.cos(t)))/(np.sin(t)+7/5.)-2*np.sin(t)+2

#x = r*np.cos(t)
#y = r*np.sin(t)

fig, ax = plt.subplots()
ax.set_aspect('equal')
ax.plot(x,y, 'k-')
ax.plot(0,0,'x')
plt.show()


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Epiciclos.ipynb".
The first task is:
Evalúa serie Truncada
Can you write Python code for it?

space = np.linspace(min(t), max(t), 300)
x_Sn, y_Sn = Sn(space, coef, order)


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Epiciclos.ipynb".
The first task is:
Gráfico
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot(x_Sn, y_Sn, 'r--')
ax.plot(x, y, 'k-')
ax.set_aspect('equal')
xmin, xmax = plt.xlim()
ymin, ymax = plt.ylim()
plt.show()


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Funcion-Signo.ipynb".
The first task is:
Ahora desplegamos el gráfico en forma interactiva usando la función interact de ipywidgets:
Can you write Python code for it?

from ipywidgets import interact

interact(g,n=(0,50))


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Funcion-Signo.ipynb".
The first task is:
Con Sympy
Podemos además analizar este ejemplo usando el paquete de matemática simbólica Sympy
Can you write Python code for it?


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Funcion-Signo.ipynb".
The first task is:
Con esto podemos graficar la serie truncada hasta un orden dado, por ejemplo, $n=10$:
Can you write Python code for it?

plt.plot(x,Sn(50,x))
plt.grid()
plt.xlabel(r'$\theta$',fontsize=15)
plt.ylabel(r'$S_n(\theta)$',fontsize=15)


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Funcion-Signo.ipynb".
The first task is:
Finalmente, podemos calcular los coeficientes $b_n$ a partir de su definición
Can you write Python code for it?

bn = (1/sp.pi)*sp.integrate(f1*sp.sin(n*x),(x,-sp.pi,sp.pi))
bn

sp.simplify(bn)

Sm = sp.Sum(bn*sp.sin(n*x),(n,0,sp.oo))
Sm

def g3(m=0):
    sp.plot(sp.Sum(bn*sp.sin(n*x),(n,0,m)).doit(), (x,-2*sp.pi,2*sp.pi))
# Sum().doit() es equivalente a summation()

interact(g3,m=(1,50))


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Funcion-Signo.ipynb".
The first task is:
Para graficar las curvas en el intervalo $[-2\pi,2\pi]$, creamos el arreglo de valores correspondiente:
Can you write Python code for it?


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Funcion-Signo.ipynb".
The first task is:
También podemos explorar en forma interactiva cómo cambia la serie truncada al variar el número de términos, es decir, variando $n$. Para esto, definimos una función que crea el gráfico para un valor de $n$ dado:
Can you write Python code for it?

def g(n=0):
    plt.plot(x,Sn(n,x))
    plt.xlim(-np.pi,np.pi)
    plt.grid()
    plt.xlabel(r'$\theta$',fontsize=15)
    plt.ylabel(r'$S_n(\theta)$',fontsize=15)


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Funcion-Signo.ipynb".
The first task is:
Una forma rápida de intruducir la función que estamos estudiando es usando la función Heaviside que está predefinida en Sympy
Can you write Python code for it?

f1 = 2*(sp.Heaviside(x)- sp.Rational(1,2))


I am working on the file "Notebooks/Ejemplo-Serie-Fourier-Funcion-Signo.ipynb".
The first task is:
Y graficar en forma interactiva la serie truncada usando la función plot de Sympy
Can you write Python code for it?

def g2(m=0):
    sp.plot(sf.truncate(n=m), (x,-2*sp.pi,2*sp.pi))

interact(g2,m=(1,50))


I am working on the file "Notebooks/Ejemplos-Transformada-Fourier-Sympy.ipynb".
The first task is:
Delta de Dirac
Can you write Python code for it?

f4 = DiracDelta(x-xi)
f4

fourier_transform(f4,x,k/(2*pi))


I am working on the file "Notebooks/Ejemplos-Transformada-Fourier-Sympy.ipynb".
The first task is:
Esta forma debería ser equivalente a la calculada en clases. Podemos simplificarla un poco:
Can you write Python code for it?


I am working on the file "Notebooks/Ejemplos-Transformada-Fourier-Sympy.ipynb".
The first task is:
La implementación de la función fourier_transform requiere ingresar como argumento la función a transformar, la variable respecto a la cual se calcula la transformada, y el valor de la frecuencia correspondiente. Esto significa que si queremos transformar la función $f(t)$ debemos ingresar $(f,t,\omega/2\pi)$ como argumento:
Can you write Python code for it?


I am working on the file "Notebooks/Ejemplos-Transformada-Fourier-Sympy.ipynb".
The first task is:
Sinusoidal amorgtiuada con función gaussiana
Can you write Python code for it?

f2 = exp(-a*x**2)*cos(k0*x)
f2

fourier_transform(f2,x,k/(2*pi))


I am working on the file "Notebooks/Ejemplos-Transformada-Fourier.ipynb".
The first task is:
Definimos 1000 puntos en el intervalo $[-\pi,\pi]$:
Can you write Python code for it?

x = np.linspace(-10,10,1000)
k = np.linspace(-10,10,1000)

def p(a=1):
    plt.figure(figsize=(12,5))
    plt.subplot(1,2,1)
    #fig,ej=subplots(1,2,figsize=(14,5))
    plt.plot(x,pulso(x,a), lw = 2)
    plt.xlim(-10,10)
    plt.ylim(-.1,1.1)
    plt.grid(True)
    plt.xlabel(r'$x$',fontsize=15)
    plt.ylabel(r'$f(x)$',fontsize=15)
    plt.subplot(1,2,2)
    plt.plot(k,2*(np.sin(k*a)/k), lw = 2)
    plt.xlim(-10,10)
    plt.grid(True)
    plt.xlabel('$k$',fontsize=15)
    plt.ylabel('$\\tilde{f}(k)$',fontsize=15)

#p(5)
#plt.savefig('fig-transformada-Fourier-pulso-cuadrado.pdf')

interact(p, a=(1,10))


I am working on the file "Notebooks/Ejemplos-Transformada-Fourier.ipynb".
The first task is:
Pulso cuadrado:
Sea
$$
f(x):=\left{
\begin{matrix}
1, &\rm{si}\quad |x|<a \\
0, &\rm{si}\quad|x|>a \
\end{matrix}\right. ,
$$
con transformada de Fourier
$$
{\cal F}f=2\frac{\sin(k a)}{k}.
$$
Can you write Python code for it?

def p(x,a):
    if abs(x)<a:
        return 1.
    else:
        return 0.
pulso = np.vectorize(p) #vectorizando la función pulso


I am working on the file "Notebooks/Ensemble-Prediction-Techniques.ipynb".
The first task is:
The Power of Ensembles
<img src='../Images/power-of-ensemble-predictors.png' width='400' />
Can you write Python code for it?

# 10 classifiers, data has 1000 rows, each classifier's accuracy varies from
# 0 to 1
clf_accs = np.arange(0,1.02,0.02)
%time ensemble_accs = [ensemble_accuracy(100, x, truth) for x in clf_accs]
#ensemble_accs


I am working on the file "Notebooks/Ensemble-Prediction-Techniques.ipynb".
The first task is:
The accuracy of a majority-vote system from an ensemble of weak classifiers is a big improvement on the accuracy of any individual classifier in the ensemble.
We can see how an ensemble of classifiers does under the majority-vote scheme as the predictive accuracy of the individual classifiers in the ensemble vary from 0 (always wrong) to 1 (always right).
Can you write Python code for it?

fig = plt.figure(1, figsize=(12, 8))

# Create an axes instance
ax = fig.add_subplot(111)
plt.title('Ensemble Prediction Accuracy (100 Individual Models/Classifiers)')
plt.xlabel('Individual Classifier Prediction Accuracy')
plt.ylabel('Ensemble Prediction Accuracy')
ax.set_xticks(np.arange(0, 1.1, 0.05))
ax.set_yticks(np.arange(0, 1.1, 0.05))
plt.grid()

# Create the plot
plt.plot(clf_accs, ensemble_accs, marker='o');


I am working on the file "Notebooks/Ensemble-Prediction-Techniques.ipynb".
The first task is:
The majority value jumps around even when the number of coins tossed is high.
However, the majority value jumps around less as the number of coins tossed grows larger and larger.
Let's get a better view of this...
Can you write Python code for it?

fig = plt.figure(1, figsize=(12, 8))

# Create an axes instance
ax = fig.add_subplot(111)
plt.title('Ensemble Majority Value as Number of Classifiers Change')
plt.xlabel('Number of Classifiers')
plt.ylabel('Ensemble Majority Value')
#ax.set_xticks(np.arange(0, 1.1, 0.05))
#ax.set_yticks(np.arange(0, 1.1, 0.05))
plt.ylim([0,1.2])
plt.grid()

# Create the plot
plt.plot(range(1, num_clfs+1), majority_vals);

fig = plt.figure(1, figsize=(12, 8))

# Create an axes instance
ax = fig.add_subplot(111)
plt.title('Ratio of Heads as Number of Classifiers in the Ensemble Increases')
plt.xlabel('Number of Classifiers')
plt.ylabel('Ratio of Heads')
#ax.set_xticks(np.arange(0, 1.1, 0.05))
ax.set_yticks(np.arange(0, 1.1, 0.1))
plt.ylim([0,1.2])
#plt.xlim([0,100])
plt.grid()

# Create the plot
plt.plot(range(1, num_clfs+1), outcome_ratios);


I am working on the file "Notebooks/Examen_Centrale/MetropolisMCMCTimeDivergence_questions.ipynb".
The first task is:
Commentez brièvement les trois graphes ci-dessus.
Let's compare the divergence time inferred according to the posterior to that obtained according to the prior only
Pourquoi est-il important de comparer la distribution a posteriori de la distribution a priori?
Can you write Python code for it?

axes = plt.gca()
axes.set_xlim([mini,maxi])

sns.kdeplot(np.array(time_params), lw=5, color="green", shade=True, legend=True, label="Posterior")
sns.kdeplot(np.array(time_params_prior), lw=5, color="blue", shade=True, legend=True, label="Prior")
plt.legend()
plt.xlabel("Divergence time")
plt.ylabel("Density/Frequency")


I am working on the file "Notebooks/Examen_Centrale/MetropolisMCMCTimeDivergence_questions.ipynb".
The first task is:
Analysis of the run to sample from the posterior
Posterior probability across iterations of the MCMC run
Can you write Python code for it?

plt.plot(posteriors[0::5])
plt.xlabel("Iteration")
plt.ylabel("Posterior probability")


I am working on the file "Notebooks/Examen_Centrale/MetropolisMCMCTimeDivergence_questions.ipynb".
The first task is:
Let's compute a 95% credibility interval on the divergence time
Can you write Python code for it?

def HDI_from_MCMC(posterior_samples, credible_mass):
    # Computes highest density interval from a sample of representative values,
    # estimated as the shortest credible interval
    # Takes Arguments posterior_samples (samples from posterior) and credible mass (normally .95)
    sorted_points = sorted(posterior_samples)
    ciIdxInc = np.ceil(credible_mass * len(sorted_points)).astype('int')
    nCIs = len(sorted_points) - ciIdxInc
    ciWidth = [0]*nCIs
    for i in range(0, nCIs):
        ciWidth[i] = sorted_points[i + ciIdxInc] - sorted_points[i]
    HDImin = sorted_points[ciWidth.index(min(ciWidth))]
    HDImax = sorted_points[ciWidth.index(min(ciWidth))+ciIdxInc]
    return(HDImin, HDImax)

print("95% credibility interval from the posterior: ")
print(HDI_from_MCMC(time_params, 0.95))
print("Compared to the interval from the prior: ")
print(HDI_from_MCMC(time_params_prior, 0.95))


I am working on the file "Notebooks/Examen_Centrale/MetropolisMCMCTimeDivergence_questions.ipynb".
The first task is:
Rate parameter value across iterations of the MCMC run
Can you write Python code for it?

plt.plot(rate_params[0::5])
plt.xlabel("Iteration")
plt.ylabel("Rate parameter value")


I am working on the file "Notebooks/Examen_Centrale/MetropolisMCMCTimeDivergence_questions.ipynb".
The first task is:
Running the Metropolis MCMC on the data
First, we run the MCMC with the data, then we run the MCMC without considering the data, so that we can compare the two results.
Expliquez pourquoi il est important de comparer les résultats obtenus en utilisant les données aux résultats obtenus en se basant seulement sur les distributions a priori.
Can you write Python code for it?

# Running with the data: inference of the posterior
rate_params, time_params, posteriors = MetropolisMCMC(1347,15988-1347, 50000, False)

# Running under the prior only, without data
rate_params_prior, time_params_prior, posteriors_prior = MetropolisMCMC(1347, 15988-1347, 20000, True)


I am working on the file "Notebooks/Examen_Centrale/MetropolisMCMCTimeDivergence_questions.ipynb".
The first task is:
Time parameter value across iterations of the MCMC run
Can you write Python code for it?

plt.plot(time_params[0::5])
plt.xlabel("Iteration")
plt.ylabel("Time parameter value")


I am working on the file "Notebooks/Experiments/17_01_09_batching.ipynb".
The first task is:
Loading
Can you write Python code for it?

dir_list = os.listdir(path=this_dir)

if "df.p" in dir_list:
    _pickle_path = os.path.join(this_dir, "df.p")
    _old_df = pd.read_pickle(_pickle_path)

_old_df["hash"]

_old_df.dtypes

pmtx = gk.generator.gendy1.gen_params(rows=20)
df = gk.generator.gendy1.format_params(pmtx)
df.sort_values(["hash"])

for i, row in df.iterrows():

    session = nonrealtimetools.Session()

    builder = gk.generator.gendy1.make_builder(row)

    out = gk.generator.gendy1.build_out(builder)

    synthdef = builder.build()

    with session.at(0):
        synth_a = session.add_synth(duration=10, synthdef=synthdef)

    gk.util.render_session(session, this_dir, row["hash"])

dt = datetime.now().strftime("%Y_%m_%d") #_%H-%M-%S")
identifier = '{0}-len{1}'.format(dt, str(df.shape[0]))

df.to_pickle("{0}/df-{1}.p".format(this_dir, dt))

df.to_pickle("{0}/df.p".format(this_dir, dt))


I am working on the file "Notebooks/Experiments/17_01_10_regress.ipynb".
The first task is:
Feature extraction timing
~$0.88$ seconds per example/.aiff.
1m 28s for 100
Can you write Python code for it?

%timeit

for i, row in df.iterrows():

    y, sr = librosa.load(os.path.join(this_dir, "aif_files", row["hash"] + ".aiff"))

    _y_normed = librosa.util.normalize(y)
    _mfcc = librosa.feature.mfcc(y=_y_normed, sr=sr, n_mfcc=13)
    _cent = np.mean(librosa.feature.spectral_centroid(y=_y_normed, sr=sr))

    _mfcc_mean = gk.feature_extraction.get_stats(_mfcc)["mean"]

    X_row = np.append(_mfcc_mean, _cent)

    if i==0:
        X_mtx = X_row
    else:
        X_mtx = np.vstack((X_mtx, X_row))


I am working on the file "Notebooks/Experiments/17_01_10_regress.ipynb".
The first task is:
Loading utility
Add: join old df to new df
Can you write Python code for it?

dir_list = os.listdir(path=this_dir)
_pickle_path = os.path.join(this_dir, "df.p")

if "df.p" in dir_list:
    #_pickle_path = os.path.join(this_dir, "df.p")
    _old_df = pd.read_pickle(_pickle_path)
    _pickle_dir = make_out_dir(this_dir, "pickle_files")
    dt_identifier = datetime.now().strftime("df-%Y_%m_%d_%H%M%S.p")
    _old_pickle_path = os.path.join(_pickle_dir, dt_identifier)
    pd.to_pickle(_old_pickle_path)

pmtx = gk.generator.gendy1.gen_params(dists=(0., 0.), rows=100)

df = gk.generator.gendy1.format_params(pmtx)

df

pmtx


I am working on the file "Notebooks/Experiments/17_01_10_regress.ipynb".
The first task is:
Preprocessing
Can you write Python code for it?

# Scale data
standard_scaler = sk.preprocessing.StandardScaler()
X_scaled = standard_scaler.fit_transform(X_mtx)
#Xte_s = standard_scaler.transform(X_test)

robust_scaler = sk.preprocessing.RobustScaler()
X_rscaled = robust_scaler.fit_transform(X_mtx)
#Xte_r = robust_scaler.transform(X_test)

X_scaled.mean(axis=0)

X_scaled.mean(axis=0).mean()

X_scaled.std(axis=0)

X_train, X_test, y_train, y_test = sk.model_selection.train_test_split(
    X_scaled, pmtx, test_size=0.4, random_state=1)

# Create linear regression objectc
regr = linear_model.LinearRegression()

# Train the model using the training sets
regr.fit(X_train, y_train)

# The coefficients
print('Coefficients: \n', regr.coef_)
# The mean squared error
print("Mean squared error: %.2f"
      % np.mean((regr.predict(X_test) - y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % regr.score(X_test, y_test))

y_test[0]

X_test[0]

regr.predict(X_test[0])

y_test[0]


I am working on the file "Notebooks/Experiments/17_01_10_regress_w_gendy_dists.ipynb".
The first task is:
Feature extraction timing
~$0.88$ seconds per example/.aiff.
1m 28s for 100
Can you write Python code for it?


I am working on the file "Notebooks/Experiments/17_01_10_regress_w_gendy_dists.ipynb".
The first task is:
Generation/rendering timing
~$0.189$ seconds per example/.aiff.
18.9s for 100
Can you write Python code for it?

%time

for i, row in df.iterrows():

    session = nonrealtimetools.Session()

    builder = gk.generator.gendy1.make_builder(row)

    out = gk.generator.gendy1.build_out(builder)

    synthdef = builder.build()

    with session.at(0):
        synth_a = session.add_synth(duration=10, synthdef=synthdef)

    gk.util.render_session(session, this_dir, row["hash"])


I am working on the file "Notebooks/Experiments/17_01_10_regress_w_gendy_dists.ipynb".
The first task is:
Thought: For feature extraction, it would probably be faster to extract all time domain vectors $y$ into a NumPy array and perform the necessary LibROSA operations across the rows of the vector, possibly leveraging under-the-hood efficiencies.
"1min 43s per loop" below
Can you write Python code for it?

for i, row in df.iterrows():

    session = nonrealtimetools.Session()

    builder = gk.generator.gendy1.make_builder(row)

    out = gk.generator.gendy1.build_out(builder)

    synthdef = builder.build()

    with session.at(0):
        synth_a = session.add_synth(duration=10, synthdef=synthdef)

    gk.util.render_session(session, this_dir, row["hash"])

    y, sr = librosa.load(os.path.join(this_dir, "aif_files", row["hash"] + ".aiff"))

    _y_normed = librosa.util.normalize(y)
    _mfcc = librosa.feature.mfcc(y=_y_normed, sr=sr, n_mfcc=13)
    _cent = np.mean(librosa.feature.spectral_centroid(y=_y_normed, sr=sr))

    _mfcc_mean = gk.feature_extraction.get_stats(_mfcc)["mean"]

    X_row = np.append(_mfcc_mean, _cent)

    if i==0:
        X_mtx = X_row
    else:
        X_mtx = np.vstack((X_mtx, X_row))

X_mtx.shape

def col_rename_4_mfcc(c):
    if (c < 13):
        return "mfcc_mean_{}".format(c)
    else:
        return "spectral_centroid"

pd.DataFrame(X_mtx).rename_axis(lambda c: col_rename_4_mfcc(c), axis=1)

from sklearn import linear_model
from sklearn import model_selection
from sklearn import preprocessing

import sklearn as sk

import matplotlib.pyplot as plt
%matplotlib inline

pmtx.shape

X_mtx.shape

X_mtx[0]

X_train, X_test, y_train, y_test = sk.model_selection.train_test_split(
    X_mtx, pmtx, test_size=0.4, random_state=1)

# Create linear regression objectc
regr = linear_model.LinearRegression()

# Train the model using the training sets
regr.fit(X_train, y_train)

# The coefficients
print('Coefficients: \n', regr.coef_)
# The mean squared error
print("Mean squared error: %.2f"
      % np.mean((regr.predict(X_test) - y_test) ** 2))
# Explained variance score: 1 is perfect prediction
print('Variance score: %.2f' % regr.score(X_test, y_test))


I am working on the file "Notebooks/Exploratory/DX_screen_TCGA_and_GEO.ipynb".
The first task is:
I am folling up on Fatty Acid Metabolism as opposed to biological oxidations, because it has a larger effect size, although the smaller gene-set size gives it a less extreme p-value.
Can you write Python code for it?

def fig_1f(ax):
    v = pd.concat([dx_rna.frac,
                   dx_rna.frac.ix[ti(gs2['REACTOME_CELL_CYCLE_MITOTIC']>0)],
                   dx_rna.frac.ix[ti(gs2['KEGG_FATTY_ACID_METABOLISM']>0)]])

    v1 = pd.concat([pd.Series('All Genes', dx_rna.frac.index),
                    pd.Series('Cell Cycle\nMitotic',
                              ti(gs2['REACTOME_CELL_CYCLE_MITOTIC']>0)),
                    pd.Series('Fatty Acid\nMetabolism',
                              ti(gs2['KEGG_FATTY_ACID_METABOLISM']>0))])
    v1.name = ''
    v.name = 'Fraction Overexpressed'
    violin_plot_pandas(v1, v, ann=None, ax=ax)
    prettify_ax(ax)
    return ax

#Do not import
fig, ax = subplots(1,1, figsize=(5,3))
fig_1f(ax);


I am working on the file "Notebooks/Exploratory/DX_screen_TCGA_and_GEO.ipynb".
The first task is:
Overexpressed pathways
Can you write Python code for it?


I am working on the file "Notebooks/Exploratory/DX_screen_TCGA_and_GEO.ipynb".
The first task is:
Pathway and Gene Annotation Analysis
Can you write Python code for it?

gs2 = gene_sets.ix[dx_rna.index].fillna(0)

rr = screen_feature(dx_rna.frac, rev_kruskal, gs2.T,
                    align=False)
fp = (1.*gene_sets.T * dx_rna.frac).T.dropna().replace(0, np.nan).mean().order()
fp.name = 'mean frac'


I am working on the file "Notebooks/Exploratory/DX_screen_TCGA_and_GEO.ipynb".
The first task is:
Read in matched Gene expression data.
Can you write Python code for it?

matched_rna = pd.read_hdf('/data_ssd/RNASeq_2014_07_15.h5', 'matched_tn')
rna_microarray = pd.read_hdf('/data_ssd/GEO_microarray_dx.h5', 'data')
matched_rna = rna_microarray.join(matched_rna)


I am working on the file "Notebooks/Exploratory/DX_screen_TCGA_and_GEO.ipynb".
The first task is:
Underexpressed pathways
Can you write Python code for it?


I am working on the file "Notebooks/Exploratory/sigmoid_updown.ipynb".
The first task is:
SGEF is overexpressed in prostate cancer and contributes to prostate cancer progression.
Can you write Python code for it?


I am working on the file "Notebooks/Figures/Purgatory/DX_screen_figs.ipynb".
The first task is:
I am folling up on Fatty Acid Metabolism as opposed to biological oxidations, because it has a larger effect size, although the smaller gene-set size gives it a less extreme p-value.
Can you write Python code for it?

def fig_1f(ax):
    v = pd.concat([dx_rna.frac,
                   dx_rna.frac.ix[ti(gs2['REACTOME_CELL_CYCLE_MITOTIC']>0)],
                   dx_rna.frac.ix[ti(gs2['KEGG_FATTY_ACID_METABOLISM']>0)]])

    v1 = pd.concat([pd.Series('All Genes', dx_rna.frac.index),
                    pd.Series('Cell Cycle\nMitotic',
                              ti(gs2['REACTOME_CELL_CYCLE_MITOTIC']>0)),
                    pd.Series('Fatty Acid\nMetabolism',
                              ti(gs2['KEGG_FATTY_ACID_METABOLISM']>0))])
    v1.name = ''
    v.name = 'Fraction Overexpressed'
    violin_plot_pandas(v1, v, ann=None, ax=ax)
    prettify_ax(ax)
    return ax

#Do not import
fig, ax = subplots(1,1, figsize=(4,3))
fig_1f(ax)
fig.tight_layout()
fig.savefig('/cellar/users/agross/figures/gsea.pdf')


I am working on the file "Notebooks/Figures/Purgatory/DX_screen_figs.ipynb".
The first task is:
Overexpressed pathways
Can you write Python code for it?


I am working on the file "Notebooks/Figures/Purgatory/DX_screen_figs.ipynb".
The first task is:
Pathway and Gene Annotation Analysis
Can you write Python code for it?


I am working on the file "Notebooks/Figures/Purgatory/DX_screen_figs.ipynb".
The first task is:
Read in matched Gene and miRNA expression data.
Can you write Python code for it?

matched_rna = pd.read_hdf('/data_ssd/RNASeq_2014_07_15.h5', 'matched_tn')
matched_mir = pd.read_hdf('/data_ssd/miRNASeq_2014_07_15.h5', 'matched_tn')


I am working on the file "Notebooks/Figures/Purgatory/DX_screen_figs.ipynb".
The first task is:
Run a simple screen for DX probes

Here we take the matched data and run a basic screen
fc = 1 means that we have no foldchange buffer for a gene to be considered over or underexpressed in a patient
If there are ties or missing data, I omit these from the test.  This can cause underpowered tests which have extreme test statistics but weak p-values.  For this reason I filter all gene/probes/markers with a sample size of less than 300 patients.
Can you write Python code for it?

dx_rna = binomial_test_screen(matched_rna, fc=1.)
dx_rna = dx_rna[dx_rna.num_dx > 300]

dx_rna.ix[['ADH1A','ADH1B','ADH1C']]

dx_rna.shape

dx_rna.p.rank().ix[['ADH1A','ADH1B','ADH1C']]

dx_rna.sort('p').head(10)

plt.rcParams['savefig.dpi'] = 150

fig, ax = subplots(figsize=(7.75,2.))
paired_bp_tn_split(matched_rna.ix['ADH1B'], codes, data_type='mRNA',
                   ax=ax)
fig.tight_layout()
fig.savefig('/cellar/users/agross/figures/ADH1B.pdf')

dx_mir = binomial_test_screen(matched_mir, fc=1.)
dx_mir = dx_mir[dx_mir.num_dx > 300]

dx_mir.sort('p').head()

paired_bp_tn_split(matched_mir.ix['hsa-mir-139'], codes, data_type='')

fig, ax = subplots(figsize=(6.5,2.))
paired_bp_tn_split(matched_mir.ix['hsa-mir-21'], codes, data_type='',
                   ax=ax)
fig.tight_layout()
fig.savefig('/cellar/users/agross/figures/mir21.pdf')

dx_meth = binomial_test_screen(matched_meth, fc=1.)
dx_meth = dx_meth[dx_meth.num_dx > 300]

dx_meth.sort('p').head()

paired_bp_tn_split(matched_meth.ix['cg10216717'], codes, data_type='Beta')


I am working on the file "Notebooks/Figures/Purgatory/DX_screen_figs.ipynb".
The first task is:
Underexpressed pathways
Can you write Python code for it?


I am working on the file "Notebooks/Figures/Purgatory/DX_screen_figs.ipynb".
The first task is:
matched_meth is our matched methylation data.
Can you write Python code for it?

store = '/data_ssd/TCGA_methylation_2014_04_16.h5'
matched_meth = pd.read_hdf(store, 'matched_tn')
matched_meth = matched_meth.groupby(axis=1, level=[0,1]).first()

matched_meth.T.head(10).T.head()


I am working on the file "Notebooks/ForAssessment/Exercise2.ipynb".
The first task is:
Hint
You will need to have the two datasets at the same resolution and valid data shape. The data are stored in an image so re-interpolation can be done with the image resizing fanciness available in ... (guess where !)
Can you write Python code for it?

from scipy.misc import imresize

etopoH = gdal.Open("../../Data/Resources/ETOPO1_Ice_c_geotiff.tif")
etopoH_img = etopoH.ReadAsArray()[::4,::4]
del(etopoH)

ages = np.load("../../Data/Resources/global_age_data.3.6.z.npz")["ageData"]

etopoH_1 = imresize(etopoH_img, ages.shape, interp='bilinear', mode="F")
etopoH_1[ np.isnan(ages) ] = np.nan

print etopoH_img.shape
print etopoH_1.shape
print ages.shape


I am working on the file "Notebooks/ForAssessment/Exercise2.ipynb".
The first task is:
Smoothing
Now try interpolating the bathymetry to a smoothed / downsampled version of the ages.
```python
ages_reduced = ages[::8,::8] # This is arbitrary, you should try some different ones
OR
ages_reduced2 = imresize(ages, (226,451), interp='bilinear', mode="F")
OR
ages_reduced2 = imresize(ages, (226,451), interp='bicubic', mode="F")
OR
ages_reduced2 = imresize(ages, (226,451), interp='lanczos', mode="F")
```
Comment on which of these is the more effective.
Can you write Python code for it?

ages_reduced  = ages[::8,::8] # This is arbitrary, you should try some different ones
plt.imshow(ages_reduced)
plt.show()

ages_reduced2 = imresize(ages, (226,451), interp='bicubic', mode="F")
plt.imshow(ages_reduced2)
plt.show()


I am working on the file "Notebooks/ForAssessment/Exercise2.ipynb".
The first task is:
Task
Fit $ \textrm{depth} = A + B\sqrt{\textrm{age}} $ to this data and create a plot
You should be able to use
python
from scipy.optimize import curve_fit
help(curve_fit)
Can you write Python code for it?


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_corpus_statistics.ipynb".
The first task is:
1. Introduction
We are interested in observing the word frequency across e-mail replies of the Full Disclosure (FD) mailing list, which is a "public, vendor-neutral forum for detailed discussion of vulnerabilities and exploitation techniques, as well as tools, papers, news, and events of interest to the community."
The analysis and the plots shared in the notebook involves analyzing the corpus of full disclosure mailing list which has been extracted using crawlers and derive insights from this data.
In this notebook, 2012 will be used for single-year analysis.
2. Total Word Count per Document Table
We create a dataframe from the input file where every row is a document from the corpus, the total word count and the number of unique words for every document. A snippet of the dataframe is shared below for better understanding.
Can you write Python code for it?

#Input year for which the word count histogram is being plotted
year='2012'
#Directory with the files
path = 'data/input/bodymessage_corpus/'+year

file_name_list=[]
file_wc_list=[]
file_uq_wc_list=[]
file_wc_df = pd.DataFrame(columns = ["file_name","word_count","unique_word_count"])

#function that returns the word count for all the documents in a list
def doc_wordcnt(path_file_name, file_name):
    #use open() for opening file.
    with open(path_file_name) as f:
        #create a list of all words fetched from the file using a list comprehension
        words = [word for line in f for word in line.split()]
        unique_words = set(words)

        #append the file name and the word count to a list
        file_name_list.append(file_name)
        file_wc_list.append(len(words))
        file_uq_wc_list.append(len(unique_words))

#loop within the directory to get all the file names
for filename in os.listdir(path):
    #get the word count for every file
    doc_wordcnt(path+'/'+filename, filename)

#populate the column of the dataframe using the values populated in the list
file_wc_df["file_name"] = file_name_list
file_wc_df["word_count"] = file_wc_list
file_wc_df["unique_word_count"] = file_uq_wc_list

file_wc_df.head()

#Sort in ascending order of word count to split and plot multiple histograms
file_wc_asc_df = file_wc_df.sort_values("word_count", ascending=False);
file_uq_wc_asc_df = file_wc_df.sort_values("unique_word_count", ascending=False);


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_corpus_statistics.ipynb".
The first task is:
3. Word Count per Document Histogram
The grid displayed below consists of histograms that show the number of words per Document for '2012'.
To achieve this, the dataframe is split into five subsets and the histogram is plotted for each of the subset to get a better picture of the number of words.
Can you write Python code for it?

#Create a list that consists of plots to be plotted in a grid
plot_lst=[]

#Function to plot multiple histograms
def plot_main_hist(plot_index, plot_df, plot_year):
    #plot the histogram based on values in the dataframe
    range_max=plot_df["word_count"].iloc[0]
    range_min=plot_df["word_count"].iloc[-1]
    p = Histogram(plot_df,bins=20, values='word_count',title="Plot "+str(plot_index)+". "+plot_year+" Histogram; Range: "+str(range_min)+" to "+str(range_max))

    p.xaxis.axis_label = 'Word Count per Document'
    p.xaxis.axis_label_text_font_size = '10pt'
    p.xaxis.major_label_text_font_size = '10pt'

    p.yaxis.axis_label = 'Frequency'
    p.yaxis.axis_label_text_font_size = '10pt'
    p.yaxis.major_label_text_font_size = '10pt'
    show(p)

#Function to plot multiple histograms
def plot_hist(plot_index, plot_df, plot_year):
    #plot the histogram based on values in the dataframe
    range_max=plot_df["word_count"].iloc[0]
    range_min=plot_df["word_count"].iloc[-1]
    p = Histogram(plot_df,bins=20, values='word_count',title="Plot "+str(plot_index)+". "+plot_year+" Histogram; Range: "+str(range_min)+" to "+str(range_max))

    p.xaxis.axis_label = 'Word Count per Document'
    p.xaxis.axis_label_text_font_size = '10pt'
    p.xaxis.major_label_text_font_size = '10pt'

    p.yaxis.axis_label = 'Frequency'
    p.yaxis.axis_label_text_font_size = '10pt'
    p.yaxis.major_label_text_font_size = '10pt'
    plot_lst.append(p)

#Create dataframes and split the sorted dataframe into 5 dataframes
file_wc_asc_df1 = pd.DataFrame(data=None, columns=file_wc_asc_df.columns)
file_wc_asc_df2 = pd.DataFrame(data=None, columns=file_wc_asc_df.columns)
file_wc_asc_df3 = pd.DataFrame(data=None, columns=file_wc_asc_df.columns)
file_wc_asc_df4 = pd.DataFrame(data=None, columns=file_wc_asc_df.columns)
file_wc_asc_df5 = pd.DataFrame(data=None, columns=file_wc_asc_df.columns)

file_wc_asc_df1=file_wc_asc_df.loc[(file_wc_asc_df['word_count'] >= 0) & (file_wc_asc_df['word_count'] <=...


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_corpus_statistics.ipynb".
The first task is:
Vocabulary per Document Histogram
The grid displayed below consists of histograms that show the number of unique words per Document for '2012'.
As shown below, there is only a single document in the entire dataset that has a count of unique words greater than 2000. Majority of the documents in the dataset have a unique word count between 100 and 200.
Can you write Python code for it?

#Create a list that consists of plots to be plotted in a grid
plot_lst=[]

#Function to histograms
def plot_hist(plot_index, plot_df, plot_year):
    #plot the histogram based on values in the dataframe
    range_max=plot_df["unique_word_count"].iloc[0]
    range_min=plot_df["unique_word_count"].iloc[-1]
    p = Histogram(plot_df,bins=20, values='unique_word_count',title="Plot "+str(plot_index)+". "+plot_year+" Histogram; Range: "+str(range_min)+" to "+str(range_max))

    p.xaxis.axis_label = 'Unique Word Count per Document'
    p.xaxis.axis_label_text_font_size = '10pt'
    p.xaxis.major_label_text_font_size = '10pt'

    p.yaxis.axis_label = 'Frequency'
    p.yaxis.axis_label_text_font_size = '10pt'
    p.yaxis.major_label_text_font_size = '10pt'
    show(p)

#Create dataframes
file_uq_wc_asc_df2 = pd.DataFrame(data=None, columns=file_uq_wc_asc_df.columns)
file_uq_wc_asc_df2=file_uq_wc_asc_df.loc[(file_uq_wc_asc_df['unique_word_count'] >= 700) & (file_uq_wc_asc_df['unique_word_count'] <= 4000)]

#Get the year from the filename for the first record
year=file_uq_wc_asc_df.file_name[0][:4]
#call the histogram function for each dataframe
plot_hist(1, file_uq_wc_asc_df, year)

year=file_uq_wc_asc_df.file_name[0][:4]
plot_hist(2.1, file_uq_wc_asc_df2, year)


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_introduction.ipynb".
The first task is:
5.2.1 Community and centrality analysis
Examining the results in igraph can reveal some useful information. Beginning with the clusters:
Can you write Python code for it?

# First, we will define a function to summarize the clustering attempt.
def summarizeClusters(com, n=5):
    print(len(com), 'clusters.')
    print('maximum size:', len(max(com, key=len)))
    print('minimum size:', len(min(com, key=len)))

    print('\nSummary of first', n, 'clusters:')
    for i in range(n):
        etc = ''
        if len(com[i]) > 5:
            etc += '(and ' + str(len(com[i]) - 5) + ' more)'
        print('[{}] has size of {}. Vertices: {} {}'.format(i, len(com[i]), com[i][0:5], etc))

summarizeClusters(com, n=10)


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_introduction.ipynb".
The first task is:
A simple function to list total numbers of nodes and authors will be useful as we attempt to shrink the analyzed graph:
Can you write Python code for it?

def nodeCount(g):
    print('Total vertices for', g['name'], ':\n    ', len(g.vs), 'including', len(g.vs(nodetype_eq='author')), 'authors')

nodeCount(ml)


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_introduction.ipynb".
The first task is:
Igraph allows a suggested number of communities generated, but on the entire network, the applied settings fail (the program cannot divide the network into 3 clusters, in this case).
It will be useful to define a method of saving cluster information in vertex attributes.
Can you write Python code for it?

def saveClusterInfo(g, com, attr):
    '''add to graph 'g' a cluster number from community 'com' as 'attr' attribute'''
    for idx, c in enumerate(com):
        for _ in c:
            g.vs[_][attr] = idx

# Apply above function to our overall graph
saveClusterInfo(ml, com, 'filtered_clustering')


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_introduction.ipynb".
The first task is:
Igraph's clustering algorithms are far more effective on the blob than they were on the whole graph. Here, we can specify the number of clusters we are seeking. Experimenting with results, though, demonstrated that there was little significant difference among different numbers of communities; in each case the next steps are the same.
5.3.2 The "blob" colored by subcommunity
8-cluster visualization | 15-cluster visualization |
:-------------------------:|:-------------------------:|
[]img/blob8clusters.png?raw=true) |
The blob displays distinctive subcommunities. A few of these new clusters still include fireworks (green in these visualizations), but others (colored purple here) are strongly suggestive of discussion communities in which multiple authors interact with each others' messages.
Examining these subcommunities in turn provides an opportunity to discover real communities and begin to develop a corpus from the Full Disclosure discussion.
5.4 Saving the blob subcommunities
Using 8 clusters as a baseline for analysis, the relevant subgraphs can include all useful attributes and be exported in turn.
Can you write Python code for it?

# now let's try each cluster separately

subs = {}  # build all the clusters into a dictionary
for idx, c in enumerate(com):
    subs[idx] = blob.induced_subgraph(com[idx])
    subs[idx]['name'] = year + ' blob subgraph ' + str(idx) + ' of ' + str(len(com))
    # rerun the centrality, clustering, pagerank analyses
    subcom = subs[idx].community_leading_eigenvector(weights='weight', clusters=5)
    saveClusterInfo(subs[idx], subcom, 'local_clustering')
    bc, pr = saveCentrality(subs[idx], 'local')
    filename = 'output/' + subs[idx]['name'].replace(' ', '_') + '_out.graphml'
    # subs[idx].save(filename, format='graphml')


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_introduction.ipynb".
The first task is:
The above output immediately demonstrates that one cluster is by far the largest in the network -- consisting here of 5594 nodes (out of 7179 in the total network). Most of the other clusters are tiny in comparison.
The initial impressions from betweenness measures are also interesting.
Can you write Python code for it?

# sort the betweenness and then list nodes in order of centrality
bc.sort(reverse=True)
max_bc = max(bc)
bc_normalized = [x / max_bc for x in bc]

for idx,val in enumerate(bc_normalized[0:15]):
    print('Node', idx, ':', '{:.0%}'.format(bc_normalized[idx]))


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_introduction.ipynb".
The first task is:
The above reduction in network size is helpful, but the removal of firework hubs is likely to leave a number of isolated documents in the graph. Removing all nodes with degree 0 can complete the firework cleanup.
Can you write Python code for it?

print('Filtering out documents or authors with degree 0...')
for node in ml.vs:
    if ml.degree(node) == 0:
        ml.delete_vertices(node)
ml['name'] = ml['name'] + ' (isolated nodes removed)'
nodeCount(ml)


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_introduction.ipynb".
The first task is:
The numbers after firework-filtering are still large. In order to focus more narrowly on relevant portions of this network, some deeper social network analysis techniques are required.
5 Social network analysis
The network structure varies considerably across the years. This provides an opportunity to partition a network for a given year into several clusters and investigate if the visualized structure has any association to the textual content of a given e-mail thread discussion.
If such association exists, then we can leverage the social network structure in order to simplify the identification of emerging concepts through text alone. For instance, identifying a group of individuals who prefer certain subjects, or are spammers or trolls, may become a trivial pre-processing stage before textual content is analyzed for emerging concepts.
We begin by considering 2 methods to more precisely define how to partition a network at any given year:

Community Detection
Betweenness Centrality

5.1 Community Detection
In real-world social networks, there is a tendency towards clustering of nodes with strong ties: if I have a close friend who has another close friend, I am likely to make a connection with that person. There are mathematical methods for identifying clusters based on the existing edges of their neighbors in comparison to the possible connections among them.
Both Gephi and igraph have clustering functions, but igraph’s are more robust and allow us to easily filter out subgraphs.
Igraph's community measures do not support directed graphs, as seen in the warning message below, but the directionality can be removed. It is worth noting that directionality is implied in the structure of the graph (authors write or respond to documents).
Can you write Python code for it?

com = ml.community_leading_eigenvector(weights='weight', clusters=3)

ml.to_undirected(combine_edges='max') # eliminate the directionality

# Attempt to identify communities

com = ml.community_leading_eigenvector(weights='weight', clusters=3)
print('clustering attempt, leading eigenvector:')
summary(com)


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_introduction.ipynb".
The first task is:
The summary command providing the output seen above is explained in the igraph documentation[10]. In this example, the four-character code "D-W-" indicates that the graph is directed and weighted. The graph for 2008 has 9793 vertices (nodes) and 12073 edges.
The list of attributes in the summary ("name", "color", etc.) are those for the graph (g), vertices (v), or edges (e).
In the remainder of this notebook, 2008 will be used for single-year analysis.
3 Preliminary overview
3.1 Visual assessment via Gephi-generated images
Importing the entire graphs for each year into Gephi allowed for early visual inspection and exploration of the FD lists.
The following table includes full-network Gephi graphics for each year of the email list's existence; blue nodes represent authors and red nodes represent documents. Click through for full-size renderings.
|||
:-------------------------:|:-------------------------:|:-------------------------:|:-------------------------:|
  |   |   |
  |   |   |
  |   |   |
  |   |
3.2 The expansion and contraction of Full Disclosure
It is clear from a quick look at the graphs that the overall pattern of the Full Disclosure network has shifted over time. In the early years of the list, there were tightly clustered conversations, particularly from about 2003-2006. As the years went by, the level of discussion seems to have dropped. Some of this pattern can be inferred simply from the file sizes of the edge and node lists, or by counting the nodes.
The visual impression is further confirmed by a documented change in the FD network in March, 2014, when one of the original managers of the list decided to halt the list entirely[11] but another team soon revived it[12].
Following the 2014 format change, the number of edges drops significantly, and the list displays a different network structure from the earlier years. Currently, the list remains active but few members respond to published posts.
3.3 "Firework" patterns
Visual analysis of the graphs...
Can you write Python code for it?

print('List of authors and their connected (document) vertices, among first 45 in total list,\nlimited to those with neighbors with max in-degree < 2')
for node in ml.vs(nodetype_eq='author')[0:45]:
    indegrees = [ml.degree(_, mode='in') for _ in ml.neighbors(node)]
    if indegrees:  # a few authors have 0 neighbors...
        if max(indegrees) < 2:
            print()
            print('-' * 55)
            print(node['label'], '\n')
            print('  In-degree\t Label')
            print('  ---------\t -----')
            for _ in ml.neighbors(node):
                print(' ', ml.degree(_, mode='in'), '\t\t', ml.vs[_]['label'][0:60] )
            print('  MAX:', max(indegrees))
            print('  AVERAGE:', sum(indegrees) / max(len(indegrees), 1))


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_introduction.ipynb".
The first task is:
We can observe that the betweenness centrality quickly drops, which may indicate nodes that often discuss different subjects for contributing e-mail discussions for different "community of threats".
At this point it is helpful to look at some visualizations to assess the 2008 graph in its current state.
5.2.2 Visualizing the current graph
Can you write Python code for it?

# export the ml graph
filename = 'data/output/out_' + year + '_filtered.graphml'
ml.save(filename, format='graphml') # export graph


I am working on the file "Notebooks/Full_Disclosure/full_disclosure_introduction.ipynb".
The first task is:
When the resulting graph is opened in Gephi, and colored by cluster, it is apparent that one large "blob" still comprises the bulk of the network.

There are still too many communities for easy analysis. The next objective is to isolate the central blob.
5.3 Breaking down the network into subgraphs
5.3.1 The blob
In the case of year 2008, the blob is easily identifiable, taking up majority of the graph. In other years -- particularly later years -- this will not be the case. More sophisticated selection criteria will have to be developed based on the initial clustering results.
Can you write Python code for it?

# The blob has been manually identified as cluster 0

blob = ml.induced_subgraph(com[0])
blob['name'] = year + ' subgraph for cluster 0 (blob)'
summary(blob)

# Now working with the blob, we can re-attempt clustering and bc!

com = blob.community_leading_eigenvector(weights='weight', clusters=8)
print('clustering attempt, leading eigenvector:')
summary(com)
saveClusterInfo(blob, com, 'blob_clustering')

bc, pr = saveCentrality(blob, 'blob')


I am working on the file "Notebooks/GABA_Receptors.ipynb".
The first task is:
GABRA2 and GABRA4 seem to be downregulated most of the time.
Can you write Python code for it?


I am working on the file "Notebooks/GABA_Receptors.ipynb".
The first task is:
GABRD is the top hit for tumor-associated, proliferation independent genes.
Can you write Python code for it?


I am working on the file "Notebooks/GABA_Receptors.ipynb".
The first task is:
I'm storing intermediate results here because they can take a bit to compute. If anything gets changed, you are going to want to restore the results by setting the RERUN flag.
Can you write Python code for it?

RERUN = False
try:
    assert RERUN == False
    corr_profile_t = pd.read_hdf(RNA_SUBREAD_STORE, 'dfs_correlations')
    corr_profile_m = pd.read_hdf(RNA_SUBREAD_STORE, 'dx_correlations')
    corr_change = corr_profile_m - corr_profile_t
except:
    corr_profile_t = df_s.T.corr()
    corr_profile_t.to_hdf(RNA_SUBREAD_STORE, 'dfs_correlations')

    corr_profile_m = dx.T.corr()
    corr_profile_m.to_hdf(RNA_SUBREAD_STORE, 'dx_correlations')

rr = screen_feature(corr_change.ix['GABRD'], rev_kruskal, gene_sets.T, align=False)
rr.head()

%%prun
tstats = {}
for gs in rr[rr.q < 10e-10].index:
    mm = corr_change.groupby(gene_sets[gs]).mean()
    vv = corr_change.groupby(gene_sets[gs]).var()
    tt = ((mm.ix[1] - mm.ix[0]) /
          np.sqrt(vv.div(gene_sets[gs].value_counts(), axis=0)).sum())
    tt = tt.dropna().order()
    tstats[gs] = tt
tstats = pd.DataFrame(tstats)

tstats.rank(ascending=False).ix['GABRD'].order().head(10)

tstats['BIOCARTA_NKT_PATHWAY'].order().tail()

corr_change.ix['GABRD'].ix[ti(gene_sets['BIOCARTA_NKT_PATHWAY'] > 0)].order()

pd.crosstab(gene_sets['BIOCARTA_NKT_PATHWAY'],
            gene_sets['KEGG_HEMATOPOIETIC_CELL_LINEAGE'])

violin_plot_pandas(gene_sets['KEGG_HEMATOPOIETIC_CELL_LINEAGE'],
                   corr_change.ix['GABRD'].dropna())

violin_plot_pandas(gene_sets['BIOCARTA_NKT_PATHWAY'], corr_change.ix['GABRD'].dropna())

gs = 'BIOCARTA_NKT_PATHWAY'
fig, ax = subplots()
series_scatter(corr_profile_m['GABRD'], corr_profile_t['GABRD'], s=10, ax=ax, ann=None,
               alpha=.1)
series_scatter(corr_profile_m['GABRD'], corr_profile_t['GABRD'].ix[ti(gene_sets[gs]>0)],
               s=10, ax=ax, color='red', alpha=1, ann=None)

cc = combine(gene_sets['KEGG_HEMATOPOIETIC_CELL_LINEAGE'],
            gene_sets['BIOCARTA_NKT_PATHWAY'])
violin_plot_pandas(cc, corr_change.ix['GABRD'].dropna())


I am working on the file "Notebooks/GABA_Receptors.ipynb".
The first task is:
Looking for gene sets concordinant with GABRD differential expression.
Can you write Python code for it?

rr = df_s.corrwith(df_s.ix['GABRD'], 1).dropna()
enr = screen_feature(rr.order(), rev_kruskal, gene_sets.T, align=False)
enr.head(4)

violin_plot_pandas(gene_sets['NABA_CORE_MATRISOME'], rr.replace(1, np.nan))


I am working on the file "Notebooks/GABA_Receptors_GTEX.ipynb".
The first task is:
GABRD is highly expressed in many areas of the brain as well as in the testis. Interestingly it is the highest expressed subunit in the testis.
Can you write Python code for it?

mm =  gtex.ix[gabr].dropna().T.groupby(meta['SMTSD']).median()
mm = mm.clip_lower(-10)
mm.rank(1, ascending=False)['GABRD'].order().head(10)


I am working on the file "Notebooks/GSEA_fraction_upregulated_expression.ipynb".
The first task is:
First I do a greedy filter based on p-values to find non-overlapping gene sets that are significantly associated with the tumor signal. For details see the filter_pathway_hits funciton.
Can you write Python code for it?

ff_u = filter_pathway_hits(rr.ix[ti(fp>.5)].p.order(), gs2)
ff_p = filter_pathway_hits(rr.ix[ti(fp<.5)].p.order(), gs2)
ff = ff_u.append(ff_p)

selected = rr.ix[ff[ff < .00001].index].join(fp)
selected.sort('p')


I am working on the file "Notebooks/GSEA_fraction_upregulated_expression.ipynb".
The first task is:
Here I'm running GSEA on the fraction upregulated signal across genes.
Can you write Python code for it?

gs2 = gene_sets.ix[dx_rna.index].fillna(0)
rr = screen_feature(dx_rna.frac, rev_kruskal, gs2.T,
                    align=False)
fp = (1.*gene_sets.T * dx_rna.frac).T.dropna().replace(0, np.nan).mean().order()
fp.name = 'mean frac'


I am working on the file "Notebooks/GSEA_fraction_upregulated_expression.ipynb".
The first task is:
I can also change the greedy filter to look for gene-sets with large effect sizes as opposed to p-values. This is going to give us smaller, but more specific gene-sets.
Can you write Python code for it?

f2 = fp.ix[ti(rr.q < .00001)]
ff_u = filter_pathway_hits(fp.ix[ti(f2>.5)].order()[::-1], gs2)
ff_p = filter_pathway_hits(fp.ix[ti(f2<.5)].order(), gs2)
ff = ff_u.append(ff_p)

selected = rr.ix[ff.index].join(f2)
selected.ix[(f2 - .5).abs().order().index[::-1]].dropna()


I am working on the file "Notebooks/GSEA_fraction_upregulated_expression.ipynb".
The first task is:
The cell-cycle is a large pathway with lots of subsets in the mSigDB database. Here I'm looking for significant subsets within this pathway.
Can you write Python code for it?

d = pd.DataFrame({g: gs2['REACTOME_CELL_CYCLE'] for g in gs2.columns})
a,b = odds_ratio_df(d.T>0, gs2.T>0)

dd = rr.ix[ti((a > 100) & (rr.q < 10e-15))].join(fp).sort(fp.name, ascending=False)
filter_pathway_hits(dd, gs2)


I am working on the file "Notebooks/GSEA_fraction_upregulated_expression.ipynb".
The first task is:
These two gene sets are completely non-overlapping subsets of the cell-cycle.
Can you write Python code for it?

m_g1 = 'REACTOME_M_G1_TRANSITION'
cepna = 'REACTOME_DEPOSITION_OF_NEW_CENPA_CONTAINING_NUCLEOSOMES_AT_THE_CENTROMERE'
combine(gs2[m_g1]>0, gs2[cepna]>0).value_counts()

fig, ax = subplots()
v = pd.concat([dx_rna.frac,
               dx_rna.frac.ix[ti(gs2['REACTOME_CELL_CYCLE']>0)],
               dx_rna.frac.ix[ti(gs2[m_g1]>0)],
               dx_rna.frac.ix[ti(gs2[cepna]>0)]
               ]).dropna()
v1 = pd.concat([pd.Series('All Genes', dx_rna.frac.index),
                pd.Series('Cell Cycle', ti(gs2['REACTOME_CELL_CYCLE']>0)),
                pd.Series('M/G1\nTransition', ti(gs2[m_g1]>0)),
                pd.Series('CEPNA\nDeposition', ti(gs2[cepna]>0))
                ])
v1.name = ''
v.name = 'Fraction Overexpressed'
o = ['All Genes','Cell Cycle','CEPNA\nDeposition',
     'M/G1\nTransition']
violin_plot_pandas(v1, v, order=o, ann=None, ax=ax)
prettify_ax(ax)
ax.spines['bottom'].set_visible(False)
ax.axhline(.5, color='grey', lw=2, ls='--');


I am working on the file "Notebooks/Global_Parameters.ipynb".
The first task is:
Used by unify_methylation_probes notebook to store methylation with probe set matched across all tissues.
Can you write Python code for it?

METH_STORE_MATCHED = HDFS_DIRECTORY + '/TCGA_methylation_2015_04_02_matched.h5'


I am working on the file "Notebooks/Global_Parameters.ipynb".
The first task is:
Used to store GEO matched tumor/normal microarray data.
Can you write Python code for it?

MICROARRAY_PATH = '/cellar/users/agross/Data/DiffExp/Data/'

MICROARRAY_STORE = HDFS_DIRECTORY + '/GEO_microarray_dx.h5'


I am working on the file "Notebooks/Global_Parameters.ipynb".
The first task is:
Where I download the data from the RSubread reprocessing. These are available at GEO under the the accession GSE62944.
Can you write Python code for it?


I am working on the file "Notebooks/HGT_flagging.ipynb".
The first task is:
Calculate average number of lossTaxa for each gene and add this as a new column to lossTaxa
Can you write Python code for it?

dbsTrimmed = 0
avgs = pd.Series()
for index,row in lossTaxa.iterrows():
    if index in outlierD:
        dbsTrimmed += len(outlierD[index])
        dbs = [i for i in lossTaxa.columns if i not in outlierD[index]]
    else:
        dbs = lossTaxa.columns
    avgs[index] = row[dbs].mean() # only include algorithm that have not been trimmed

dbsTrimmed

lossTaxa["Avg"] = avgs
lossTaxa.head()


I am working on the file "Notebooks/HNSCC_MATH.ipynb".
The first task is:
Combination Analysis in HPV-

TP53-3p (Our finding from this paper).
I use a different working set with some old patients filtered
Can you write Python code for it?

survival_and_stats(combine(del_3p<0, mut.features.ix['TP53']>0).ix[keepers_o].dropna(),
                   clinical.survival.survival_5y)


I am working on the file "Notebooks/HNSCC_MATH.ipynb".
The first task is:
I'm going to tweak their threshold as my calculation is a bit miscalibrated.
Can you write Python code for it?


I am working on the file "Notebooks/HNSCC_MATH.ipynb".
The first task is:
MATH in the context of TP53-3p
Can you write Python code for it?

draw_survival_curves(math_t=='MATH High', surv, combo)


I am working on the file "Notebooks/HNSCC_MATH.ipynb".
The first task is:
Survival Analysis
Curves look a little off, maybe I'm using more recent data?
Can you write Python code for it?


I am working on the file "Notebooks/HNSCC_MATH.ipynb".
The first task is:
TP53-3p combination in in the context of MATH
Can you write Python code for it?


I am working on the file "Notebooks/HNSCC_MATH.ipynb".
The first task is:
This is a bit closer but its hard to tell where the missing samples are.
Can you write Python code for it?

(math > 31.5).value_counts()

math_t = (math > 31.5).map({True:'MATH High', False:'MATH Low'})


I am working on the file "Notebooks/HNSCC_MATH.ipynb".
The first task is:
This is with the old patients back in.
Can you write Python code for it?

survival_and_stats(combine(del_3p<0, mut.features.ix['TP53']>0).ix[ti(hpv==False)].dropna(),
                   clinical.survival.survival_5y)


I am working on the file "Notebooks/Hazard_Curves.ipynb".
The first task is:
An example with many events
The next example shows a hazard curve computed from 10 events.  All events have the same probability, $p_k = 0.002$ for $k=0,~1,~2,~\ldots,~9$, but the values $h_k$ vary between $0.5m$ and $2.5m$.  They are chosen as random numbers from a uniform distribution over this interval.
This might be viewed as a hazard curve arising from a single event with probability $\hat p \approx 0.02$ but that has a range of different flooding levels (e.g. tsunamis from a single fault zone what has an earthquake every 50 years on average but for which the magnitude can vary considerably).
Can you write Python code for it?

from numpy import random
random.seed(12345)  # seed the random number generator

event_list = []  # initialize to empty list
n_events = 10
random_h = 0.5 + 2*rand(n_events)  # rand(n) returns array of n random number in interval [0,1]
print("Flooding depths h_k = ",random_h)

for k in range(n_events):
    pk = 0.002
    hk = random_h[k]
    Ek = [pk,hk]
    event_list.append(Ek)   # add this event to the list
h_ev,p_ev = HC.hazard_curve(event_list, 3)
fig = HC.plot_hazard_curve(h_ev,p_ev)


I am working on the file "Notebooks/Hazard_Curves.ipynb".
The first task is:
Combining two events
The plot above is the hazard curve if we assume only one possible event might happen.  What if there are two possible earthquakes that could happen, each of which would cause flooding to different maximum depths, call them $h_1$ and $h_2$.  The events might also have different annual probabilities of occurring, call these $p_1$ and $p_2$.  We will assume that they are independent, meaning that if one happens or not has no effect on whether the other occurs.  This is a good assumption if one event is a potential earthquake on the Cascadia subduction zone and the other is a potential earthquake on a different subduction zone.  It is perhaps less justified if the two events are two different possible earthquakes in the same place.  The occurrance of one Magnitude 9 earthquake on CSZ might eliminate the possibility of another M9 event, or it might increase the occurance of an M8 event as an aftershock.  But we will not consider this more complicated case and assume that they are independent events.
The plots below show the new hazard curve if we assume the first event $E_1$ is as above, and we add a new event $E_2$ with $p_2 = 0.003$ and $h_2 = 1.0 m$.
Can you write Python code for it?

E2 = [0.003, 1.0]      # the second event
event_list = [E1, E2]  # assumes E1 already defined in previous cell
h_ev,p_ev = HC.hazard_curve(event_list, 3)
fig = HC.plot_hazard_curve(h_ev,p_ev)


I am working on the file "Notebooks/Hazard_Curves.ipynb".
The first task is:
Hazard curve for a single event
First we consider the simplest case where only one possible event might occur,  say with annual probability $p_1 = 0.001$ (i.e., with a 1000-year return time).  Recall that we are considering some particular fixed location in the community of interest, and suppose our tsunami simulation shows that the maximum depth of inundation at this point will be $h_1 = 1.5 m$ if this event occurs (measured in meters of water depth).  Then we can plot the annual probability $P(h)$ that flooding will exceed the value $h$ as a function of $h$ (the exceedance value).  This is a simple piecewise constant function since $P(h) = p_1$ for any $h < h_1$ and $P(h) = 0$ for any $h > h_1$.  This plot is shown below.
Note that since we are dealing with very small probabilities and it is important to distinguish between values like 0.01 (100-year flood) and 0.001 (1000-year flood), it is best to plot this using a logarithmic scale in $p$.  We do this in the plots on the left below.  Note that for all $h>h_1 = 1.5$ in the plot below, the function value $P(h) = 0$ does not show up since $\log(0) = -\infty$.
The plots on the right show the same data plotted with a linear scale in p, since this is somewhat easier to read and understand for these simple examples.
Can you write Python code for it?

p1 = 0.001
h1 = 1.5
E1 = [p1,h1]
event_list = [E1]  # only one event in this example
h_ev,p_ev = HC.hazard_curve(event_list, 3)
fig = HC.plot_hazard_curve(h_ev,p_ev)


I am working on the file "Notebooks/Hazard_Curves.ipynb".
The first task is:
Note:

On the linear scale plot, each event has nearly the same jump in $p$.  Not quite because of the discussion below about combining multiple events, but nearly so.  Note the jump around $h = 1.8$ appears twice as big because there happen to be two events with $h_k$ very close together.
On the log scale the jumps look different because of the rescaling.
The value $P(0) = 0.019821$ (the constant value up to $h = \min_k(h_k)$) is not quite equal to $10p_k = 0.02$, again because of the rule discussed above for combining probabilities.  Instead it is equal to $1 - (1-0.002)^{10}$.
If we had chosen $p_k = 1 - (1-0.02)^{1/10} \approx 0.00201823$ then we would get $P(0) = 0.02$, but when dealing with such small probabilities this makes little difference in practice.

Quiz
From the plot above, estimate the annual probability of exceeding $1.5m$ of flooding.
Can you write Python code for it?


I am working on the file "Notebooks/Hazard_Maps.ipynb".
The first task is:
Hazard Maps for fixed probability
It is possible to read such a hazard curve in two different ways, either specifying an exceedance value and reading off the corresponding probability, or specifying a probability and reading of the corresponding exceedance value.  The latter approach would allow one to estimate the "100-year flood", for example, by setting $p = 0.01$ and determining what depth of flooding can be expected with this annual probability.
By fixing a probability such as 0.01 and determining the corresponding depth at each location, it is then possible to produce a map of the "100-year flood".  Several such maps are illustrated below.  Slide the bar all the way to the right to see the map for $p=0.01$.  As you now slide the bar to the left, going to lower probabilities, you will see a larger inundation region and deeper flooding at each location.
Note that the black contours show the topography, the colors show the depth in meters.
Can you write Python code for it?

def show_zeta(k):
    display(Image(HM.zeta_contours_plots[k],width=500))

interact(show_zeta,k=(0,len(HM.zeta_contours_plots)-1))


I am working on the file "Notebooks/Hazard_Maps.ipynb".
The first task is:
Hazard maps for fixed depth
Alternatively, by fixing an exceedance value such as 1 meter of flooding, and then determining the annual probability of exceeding this depth from the hazard curves at all locations, one can construct a color map of probabilities associated with that depth.  This is illustrated below for several different depths.  Slide the bar all the way to the left to see probabilities for zeta = 0 m, which gives the probability of any flooding at all.  As you slide the bar to the right, you are looking at probabilities associated with greater depths.  White areas represent regions where there is zero probability of exceeding the specified value, and colors from blue to red indicate increasing annual probabilities.
Can you write Python code for it?

def show_p(k):
    display(Image(HM.p_contours_plots[k],width=500))

interact(show_p,k=(0,len(HM.p_contours_plots)-1))


I am working on the file "Notebooks/Hazard_Maps.ipynb".
The first task is:
Use the slider bar above to sweep over the events.  The text that appears to the lower left of the figure indicates what hypothetical event has been modeled.  These came from a probabilistic study involving a set of 35 possible sources, of which only a few have been selected to work with in this workshop.  The naming convention is the name of a subduction zone, followed by an event number, followed by a realization number, but for our purposes we will think of these as independent events.   The subduction zones represented are:
- AASZ: Alaska-Aleutian
- CSZ: Cascadia
- KmSZ: Kamchatka
- KrSZ: Kurils
- SChSZ: Southern Chile
- TOH: Japan / Tohoku
About these results
The inundation patterns shown above were computed as part of a project to develop new techniques for probabilistic tsunami hazard assessment (PTHA), and are reprinted from the report

Probabilistic Tsunami Hazard Assessment (PTHA) for Crescent City, CA., by Frank I. Gonzalez, Randall J. LeVeque, Loyce M. Adams, Chris Goldfinger, George R. Priest, and Kelin Wang, http://hdl.handle.net/1773/25916

They were computed by running the GeoClaw tsunami model with the given earthquake source and then monitoring the maximum inundation depth observed at each point on shore over the course of the entire simulation.  (Actually on a fine grid of points covering the region shown above, with horizontal resolution of 1/3 arcsecond, about 10 meters in latitude and 7 meters in longitude).
The hazard curves and maps shown below also came from that study.  The goal of the study was to explore techniques and the specific values obtained should not be viewed as definitive.  Work on improving these results continues.
Hazard curves
The idea in a probabilistic analysis is to combine information from many different tsunami simulations into a map that indicates the relative probability of flooding at different points.  By assuming a finite number of possible events and assigning an annual probability of occurance to each, it is possible to combine the results.  The first step...
Can you write Python code for it?

def show_hc(k):
    display(Image(HM.hc_plots[k],width=500))

interact(show_hc,k=(0,len(HM.hc_plots)-1))


I am working on the file "Notebooks/Hemispherical Shell.ipynb".
The first task is:
COM of solid hemisphere
Can you write Python code for it?


I am working on the file "Notebooks/Hemispherical Shell.ipynb".
The first task is:
Center of Mass of Body
Can you write Python code for it?

rGO = simplify(integrate(integrate(integrate(rdmO*r**2*sin(ph),(r,R1,R2)),(th,0,2*pi)),(ph,0,pi/2))/V);
mat2vec(rGO,'b')


I am working on the file "Notebooks/Hemispherical Shell.ipynb".
The first task is:
MOI for solid hemisphere:
Can you write Python code for it?


I am working on the file "Notebooks/Hemispherical Shell.ipynb".
The first task is:
Moment of Inertia of Body about $O$
Can you write Python code for it?

Io = m/V*integrate(integrate(integrate(tmp*r**2*sin(ph),(r,R1,R2)),(th,0,2*pi)),(ph,0,pi/2)); Io


I am working on the file "Notebooks/Hemispherical Shell.ipynb".
The first task is:
Parallel Axis Theorem to calculated MOI about $G$
Can you write Python code for it?

Ig = simplify(Io  - m*((rGO.T*rGO)[0]*eye(3)-rGO*rGO.T)); Ig


I am working on the file "Notebooks/Hemispherical Shell.ipynb".
The first task is:
Volume of Body
Can you write Python code for it?

V = integrate(integrate(integrate(r**2*sin(ph),(r,R1,R2)),(th,0,2*pi)),(ph,0,pi/2)); V


I am working on the file "Notebooks/HistFactory_Examples/One-Bin.ipynb".
The first task is:
As this is a counting experiment, should have Poisson uncertainties ($\sqrt{n_{\text{events}}}$) on all histograms, so let's check:
Can you write Python code for it?

hist_list = [signal_hist, background_hist, data_hist]

for hist in hist_list:
    print("{0}: {1} events and uncertainty of {2} with sqrt({1}) being {3}".format(
        hist.GetName().rsplit('_')[0],
        hist.GetBinContent(1),
        hist.GetBinError(1),
        ROOT.TMath.Sqrt(hist.GetBinContent(1))))


I am working on the file "Notebooks/HistFactory_Examples/One-Bin.ipynb".
The first task is:
It seems that the uncertainties on the model data (which as I understand it is stored as "obsData" in the model) have blown up here after just creating the model. So I must be doing something wrong in my make_model() function.
Can you write Python code for it?

x = ROOT.RooRealVar('x','x', range_low, range_high)
fr = x.frame()
f = ws.var("obs_x_SR").frame()
data = ws.data('obsData')
#data.plotOn(fr) #plotOn() is complaining that it can not find a working method :?
print(data) # check to make sure that it is not a null pointer
#h_data = super(ROOT.RooDataSet,data).createHistogram("data_hist", x, ROOT.RooFit.Binning(n_bins, range_low, range_high))
h_data = super(ROOT.RooDataSet,data).createHistogram("data_hist", ws.var("obs_x_SR"), ROOT.RooFit.Binning(n_bins, range_low, range_high))
h_data.Sumw2(False)
h_data.Sumw2()
h_data.SetMinimum(0)
h_data.SetMaximum(h_data.GetBinContent(1) * 1.1)

c = ROOT.TCanvas()
c.cd()
h_data.Draw("E")
c.Draw()

print("{0}: {1} events and uncertainty of {2} with sqrt({1}) being {3}".format(
        h_data.GetName().rsplit('_')[0],
        h_data.GetBinContent(1),
        h_data.GetBinError(1),
        ROOT.TMath.Sqrt(h_data.GetBinContent(1))))

# draw the histogram before the model was made
c = ROOT.TCanvas()
c.cd()
data_hist.Draw("E")
data_hist.SetMinimum(0)
data_hist.SetMaximum(data_hist.GetBinContent(1) * 1.1)
c.Draw()

model_config = ws.obj("ModelConfig")

hist_factory_nav = ROOT.RooStats.HistFactory.HistFactoryNavigation(model_config)
#print(hist_factory_nav)

# draw the histogram stack
stack = hist_factory_nav.GetChannelStack("SR")
#channel_data = hist_factory_nav.GetDataHist(ws.data("obsData"),"SR","channel_data")
channel_data = h_data
sig_hist = hist_factory_nav.GetSampleHist("SR", "signal", "sig_hist")
bkg_hist = hist_factory_nav.GetSampleHist("SR", "background", "bkg_hist")

c1 = ROOT.TCanvas()
stack.Draw()
channel_data.Draw("SAME")
leg = ROOT.TLegend(0.65, 0.65, 0.9, 0.9)
for hist in stack.GetHists():
    samplename = hist.GetName().rsplit('_')[0]
    leg.AddEntry(hist, samplename,"f")
leg.AddEntry(channel_data, "channel data","lep")
leg.Draw()
c1.Draw()

# draw the histogram stored as "obsData" in the model
c2 = ROOT.TCanvas()
channel_data.Draw()
channel_data.SetMinimum(0)
channel_data.SetMaximum(channel_data.GetBinContent(1) *...


I am working on the file "Notebooks/HistFactory_Examples/One-Bin.ipynb".
The first task is:
Now make the samples and add them and the channel data histogram to the channel
Can you write Python code for it?

samples = []
samples.append(hfbench.make_sample("signal", signal_hist, 0.01, 0.01))
samples.append(hfbench.make_sample("background", background_hist, 0.01, 0.01))

channels = []
SR = hfbench.make_channel(channel_name="SR", channel_data=data_hist, channel_samples=samples)
channels.append(SR)


I am working on the file "Notebooks/HistFactory_Examples/One-Bin.ipynb".
The first task is:
Overview of Project
Goal: The goal of the project is to be able to benchmark the speed/performance of the current implementation of HistFactory to perform a maximum likelihood fit for a given binned model generated by the user. The user can do this by using the project Python library (which will essentially just be a collection of helper functions).
The binned model is supposed to be parameterized by:
* number of events
* number of bins
* number of channels
* number of signal components (samples)
* number of background components (samples)
* number of parameters of interest
* number of nuisance parameters
If this is all being made "on the fly" then all components are necessary. However, if the channels are being loaded in from ROOT files, then it would seem that the number of events and number of bins could be inferred(?).
As nuisance parameters are any model parameter which is not a parameter of interest, then if a parameter is not explicitly set as a POI (Measurement.SetPOI("parameter name")) then it is treated as a nuisance parameter.
Building a Simple Model
Let's build the simplest model: A counting experiment. One bin with a flat background sample and signal model.
Functions
Can you write Python code for it?

help(hfbench.sample_func_histogram)

help(hfbench.make_sample)

help(hfbench.make_channel)

help(hfbench.make_model)

help(hfbench.benchmark_fit)

help(hfbench.benchmark_fit_with_workspace)


I am working on the file "Notebooks/HistFactory_Examples/Signal-Only.ipynb".
The first task is:
Define the parameter of interest (POI) and then make the workspace and model
Can you write Python code for it?

POI = ["SignalStrength"]

ws, model = hfbench.make_model(n_events, n_bins, channels, POI)

ws.Print()

model.graphVizTree("model.dot")

%%bash
dot -Tpdf -o Images/model.pdf model.dot

#hfbench.benchmark_fit(ws.var("obs_x_SR"), model, n_events, n_events, verbose=True)
hfbench.benchmark_fit_with_workspace(ws, model, n_events, n_events, verbose=True);

# check information on the fit
hfbench.benchmark_fit_with_workspace(ws, model, n_events, 1, verbose=True);

model.Print("v")

mc = ws.obj("ModelConfig")
mc.LoadSnapshot()

pl = ROOT.RooStats.ProfileLikelihoodCalculator(ws.data("obsData"), mc)
ROOT.SetOwnership(pl, False)

##ws.data("obsData").Print("v")
##mc.Print("v")

model.Print("v")

x = ws.var("obs_x_SR")
#pl.SetConfidenceLevel(0.99)
pl.SetConfidenceLevel(0.683)

interval = pl.GetInterval()

firstPOI = mc.GetParametersOfInterest().first()
lowerLimit = interval.LowerLimit(firstPOI)
upperLimit = interval.UpperLimit(firstPOI)

print("68% interval on {0} is [{1},{2}]\n".format(firstPOI.GetName(),lowerLimit,upperLimit))

plot = ROOT.RooStats.LikelihoodIntervalPlot(interval)
plot.SetRange(-3,3)
plot.SetNPoints(50)
plot.SetMaximum(0.1)
c = ROOT.TCanvas()
#c.SetLogy(True)
plot.Draw()
c.Draw()
c.SaveAs("nll.pdf")
c.IsA().Destructor(c) # As currently not exiting properly

c = ROOT.TCanvas()
f = ws.var("obs_x_SR").frame()
#print(ws.var("obs_x_SR"))
ws.data("obsData").plotOn(f)
f.Draw()
c.Draw()
print(ws.data("obsData"))

#h_test = super(ROOT.RooDataSet,ws.data("obsData")).createHistogram("h_test", ws.var("obs_x_SR"), ROOT.RooFit.Binning(n_bins, range_low, range_high))
h_test = hfbench.roodataset_to_hist("h_test", ws.var("obs_x_SR"), ws.data("obsData"))

c_test = ROOT.TCanvas()
h_test.Draw()
c_test.Draw()


I am working on the file "Notebooks/HistFactory_Examples/Signal-Only.ipynb".
The first task is:
Functions
Can you write Python code for it?


I am working on the file "Notebooks/HistFactory_Examples/Signal-Only.ipynb".
The first task is:
Now make the samples and add them and the channel data histogram to the channel
Can you write Python code for it?

samples = []
samples.append(hfbench.make_sample("signal", signal_hist, 0.01, 0.01))

channels = []
SR = hfbench.make_channel(channel_name="SR", channel_data=data_hist, channel_samples=samples)
channels.append(SR)


I am working on the file "Notebooks/HistFactory_Examples/Signal-Only.ipynb".
The first task is:
Plot the channel (signal sample and data) together
Can you write Python code for it?

model_config = ws.obj("ModelConfig")

hist_factory_nav = ROOT.RooStats.HistFactory.HistFactoryNavigation(model_config)
ROOT.SetOwnership(hist_factory_nav, False)

# draw the histogram stack
stack = hist_factory_nav.GetChannelStack("SR")
channel_data = hist_factory_nav.GetDataHist(ws.data("obsData"),"SR","channel_data")
channel_data.Sumw2(False)
channel_data.Sumw2()
sig_hist = hist_factory_nav.GetSampleHist("SR", "signal", "sig_hist")

fit_frame = ws.var("obs_x_SR").frame()
model.plotOn(fit_frame, ROOT.RooFit.Normalization(n_events), ROOT.RooFit.LineStyle(1)) #kDashed = 2

c1 = ROOT.TCanvas()
stack.Draw()
channel_data.Draw("SAME")
fit_frame.Draw("SAME")
leg = ROOT.TLegend(0.65, 0.65, 0.9, 0.9)
for hist in stack.GetHists():
    samplename = hist.GetName().rsplit('_')[0]
    leg.AddEntry(hist, samplename,"f")
leg.AddEntry(channel_data, "channel data","lep")
#leg.AddEntry(fit_frame, "fit to data","lep")
leg.AddEntry("", "fit to data","lep")
leg.Draw()
c1.Draw()

# To avoid a segfault at kerenel exit
canvas_list = [c, c_test, c1]
for canvas in canvas_list:
    canvas.IsA().Destructor(canvas)


I am working on the file "Notebooks/HowTo.ipynb".
The first task is:
Computing requested parameters
We assume that the provided graphs are simple and thus multi edges in the files are ignored.
Since one of the graphs has 1000 vertices and 3989 edges, computation of diameter will take some time.
Can you write Python code for it?

# compute parameters
for i, G in enumerate(graphs):
    temp_results = [G.number_of_vertices(), G.number_of_edges(),
                    G.density(), G.diameter(),
                    G.global_clustering_coefficient()
                   ]
    results.append(temp_results)


I am working on the file "Notebooks/HowTo.ipynb".
The first task is:
We will check if the library is imported correctly by computing a few parameters of empty graph.
Can you write Python code for it?

emptyG = graph.Graph()
diam = emptyG.diameter()
gcc = emptyG.global_clustering_coefficient()
print("The diameter of empty graph: %d" % diam)
print("The global cluster coeff of empty graph is: %f" % gcc)


I am working on the file "Notebooks/Importing Notebooks.ipynb".
The first task is:
After this point, my notebooks should be importable.
Let's look at what we have in the CWD:
Can you write Python code for it?


I am working on the file "Notebooks/Importing Notebooks.ipynb".
The first task is:
Hooray again!
Even the function that contains IPython syntax works:
Can you write Python code for it?


I am working on the file "Notebooks/Importing Notebooks.ipynb".
The first task is:
Note that the __init__.py is necessary for nb to be considered a package,
just like usual.
Can you write Python code for it?

show_notebook(os.path.join("nbpackage", "nbs", "other.ipynb"))

from nbpackage.nbs import other
other.bar(5)


I am working on the file "Notebooks/Importing Notebooks.ipynb".
The first task is:
Notebook Loader
Here we have our Notebook Loader.
It's actually quite simple - once we figure out the filename of the module,
all it does is:

load the notebook document into memory
create an empty Module
execute every cell in the Module namespace

Since IPython cells can have extended syntax,
the IPython transform is applied to turn each of these cells into their pure-Python counterparts before executing them.
If all of your notebook cells are pure-Python,
this step is unnecessary.
Can you write Python code for it?

class NotebookLoader(object):
    """Module Loader for IPython Notebooks"""
    def __init__(self, path=None):
        self.shell = InteractiveShell.instance()
        self.path = path

    def load_module(self, fullname):
        """import a notebook as a module"""
        path = find_notebook(fullname, self.path)

        print ("importing IPython notebook from %s" % path)

        # load the notebook object
        with io.open(path, 'r', encoding='utf-8') as f:
            nb = current.read(f, 'json')

        # create the module and add it to sys.modules
        # if name in sys.modules:
        #    return sys.modules[name]
        mod = types.ModuleType(fullname)
        mod.__file__ = path
        mod.__loader__ = self
        mod.__dict__['get_ipython'] = get_ipython
        sys.modules[fullname] = mod

        # extra work to ensure that magics that would affect the user_ns
        # actually affect the notebook module's ns
        save_user_ns = self.shell.user_ns
        self.shell.user_ns = mod.__dict__

        try:
          for cell in nb.worksheets[0].cells:
            if cell.cell_type == 'code' and cell.language == 'python':
                # transform the input to executable Python
                code = self.shell.input_transformer_manager.transform_cell(cell.input)
                # run the code in themodule
                exec(code, mod.__dict__)
        finally:
            self.shell.user_ns = save_user_ns
        return mod


I am working on the file "Notebooks/Importing Notebooks.ipynb".
The first task is:
Notebooks in packages
We also have a notebook inside the nb package,
so let's make sure that works as well.
Can you write Python code for it?


I am working on the file "Notebooks/Importing Notebooks.ipynb".
The first task is:
Register the hook
Now we register the NotebookFinder with sys.meta_path
Can you write Python code for it?


I am working on the file "Notebooks/Importing Notebooks.ipynb".
The first task is:
So I should be able to import nbimp.mynotebook.
Aside: displaying notebooks
Here is some simple code to display the contents of a notebook
with syntax highlighting, etc.
Can you write Python code for it?

from pygments import highlight
from pygments.lexers import PythonLexer
from pygments.formatters import HtmlFormatter

from IPython.display import display, HTML

formatter = HtmlFormatter()
lexer = PythonLexer()

# publish the CSS for pygments highlighting
display(HTML("""

""" % formatter.get_style_defs()
))

def show_notebook(fname):
    """display a short summary of the cells of a notebook"""
    with io.open(fname, 'r', encoding='utf-8') as f:
        nb = current.read(f, 'json')
    html = []
    for cell in nb.worksheets[0].cells:
        html.append("
%s cell
" % cell.cell_type)
        if cell.cell_type == 'code':
            html.append(highlight(cell.input, lexer, formatter))
        else:
            html.append("
%s
" % cell.source)
    display(HTML('\n'.join(html)))

show_notebook(os.path.join("nbpackage", "mynotebook.ipynb"))


I am working on the file "Notebooks/Importing Notebooks.ipynb".
The first task is:
So my notebook has a heading cell and some code cells,
one of which contains some IPython syntax.
Let's see what happens when we import it
Can you write Python code for it?


I am working on the file "Notebooks/Importing Notebooks.ipynb".
The first task is:
So now we have importable notebooks, from both the local directory and inside packages.
I can even put a notebook inside IPython, to further demonstrate that this is working properly:
Can you write Python code for it?

import shutil
from IPython.utils.path import get_ipython_package_dir

utils = os.path.join(get_ipython_package_dir(), 'utils')
shutil.copy(os.path.join("nbpackage", "mynotebook.ipynb"),
            os.path.join(utils, "inside_ipython.ipynb")
)


I am working on the file "Notebooks/Importing Notebooks.ipynb".
The first task is:
The Module Finder
The finder is a simple object that tells you whether a name can be imported,
and returns the appropriate loader.
All this one does is check, when you do:
python
import mynotebook
it checks whether mynotebook.ipynb exists.
If a notebook is found, then it returns a NotebookLoader.
Any extra logic is just for resolving paths within packages.
Can you write Python code for it?

class NotebookFinder(object):
    """Module finder that locates IPython Notebooks"""
    def __init__(self):
        self.loaders = {}

    def find_module(self, fullname, path=None):
        nb_path = find_notebook(fullname, path)
        if not nb_path:
            return

        key = path
        if path:
            # lists aren't hashable
            key = os.path.sep.join(path)

        if key not in self.loaders:
            self.loaders[key] = NotebookLoader(path)
        return self.loaders[key]


I am working on the file "Notebooks/Importing Notebooks.ipynb".
The first task is:
and import the notebook from IPython.utils
Can you write Python code for it?

from IPython.utils import inside_ipython
inside_ipython.whatsmyname()


I am working on the file "Notebooks/Imports.ipynb".
The first task is:
External Package Imports
Can you write Python code for it?

import os as os
import pickle as pickle
import pandas as pd


I am working on the file "Notebooks/Imports.ipynb".
The first task is:
Initialize function for calling model-based gene set enrichment
Can you write Python code for it?

from rpy2 import robjects
from rpy2.robjects import pandas2ri
pandas2ri.activate()

mgsa = robjects.packages.importr('mgsa')

gs_r = robjects.ListVector({i: robjects.StrVector(list(ti(g>0))) for i,g in
                             gene_sets.iteritems()})
def run_mgsa(vec):
    v = robjects.r.c(*ti(vec))
    r = mgsa.mgsa(v, gs_r)
    res = pandas2ri.ri2pandas(mgsa.setsResults(r))
    return res


I am working on the file "Notebooks/Imports.ipynb".
The first task is:
Pandas display parameters
Can you write Python code for it?

pd.set_option('precision', 3)
pd.set_option('display.width', 300)
plt.rcParams['font.size'] = 12


I am working on the file "Notebooks/Imports.ipynb".
The first task is:
Read in All of the Expression Data
This reads in data that was pre-processed in the ./Preprocessing/init_RNA notebook.
Can you write Python code for it?

codes = pd.read_hdf(RNA_SUBREAD_STORE, 'codes')
matched_tn = pd.read_hdf(RNA_SUBREAD_STORE, 'matched_tn')
rna_df = pd.read_hdf(RNA_SUBREAD_STORE, 'all_rna')

data_portal = pd.read_hdf(RNA_STORE, 'matched_tn')
genes = data_portal.index.intersection(matched_tn.index)
pts = data_portal.columns.intersection(matched_tn.columns)
rna_df = rna_df.ix[genes]
matched_tn = matched_tn.ix[genes, pts]


I am working on the file "Notebooks/Imports.ipynb".
The first task is:
Tweaking color scheme
Can you write Python code for it?

'''Color schemes for paper taken from http://colorbrewer2.org/'''
colors = plt.rcParams['axes.color_cycle']
colors_st = ['#CA0020', '#F4A582', '#92C5DE', '#0571B0']
colors_th = ['#E66101', '#FDB863', '#B2ABD2', '#5E3C99']


I am working on the file "Notebooks/Imports.ipynb".
The first task is:
filter_pathway_hits
Can you write Python code for it?

def filter_pathway_hits(hits, gs, cutoff=.00001):
    '''
    Takes a vector of p-values and a DataFrame of binary defined gene-sets.
    Uses the ordering defined by hits to do a greedy filtering on the gene sets.
    '''
    l = [hits.index[0]]
    for gg in hits.index:
        flag = 0
        for g2 in l:
            if gg in l:
                flag = 1
                break
            elif (chi2_cont_test(gs[gg], gs[g2])['p'] < cutoff):
                flag = 1
                break
        if flag == 0:
            l.append(gg)
    hits_filtered = hits.ix[l]
    return hits_filtered


I am working on the file "Notebooks/Intro/One Dimensional Data Worksheet-Python Answers.ipynb".
The first task is:
Exercise 1
Create a Series object with 100 random integers, then filter out odd integers and reindex the Series.  Hint:  you can use python np.random.random_integers(1, 100, 100) to create the random numbers.  Print out the first 20 numbers.
Can you write Python code for it?

#First initialize the series by calling the pd.Series() function
randomNumbers = pd.Series( np.random.randint(1, 100, 100) )
#Display the first 5 random numbers
print( randomNumbers.head() )

#Next filter out the odd numbers by using the mod operator and reset the index
evenRandomNumbers = randomNumbers[ randomNumbers % 2 == 0].reset_index( drop=True )

#Display the first 5

evenRandomNumbers.head()


I am working on the file "Notebooks/Intro/One Dimensional Data Worksheet-Python Answers.ipynb".
The first task is:
Exercise 2
You will be given a list containing 10 strings. Create a new Series called validPhoneNumbers that only contains data in the format (XXX)XXX-XXXX.  Don't forget to reindex the series after you've filtered it.
Can you write Python code for it?


I am working on the file "Notebooks/Intro/One Dimensional Data Worksheet-Python Answers.ipynb".
The first task is:
Exercise 3
The code below contains a lambda function which converts a temperature from Farenheit to Celsius.  You are given a Series called temperatures in Farhenheit.  Using the .apply() function, convert the data into degrees Celsius.
Can you write Python code for it?

#This function converts a number from Farenheit to Celsius
toCelsius = lambda x: (float(5)/9)*(x-32)

#Creates a series with numbers that represent temperatures in Farenheit
tempsInFarenheit = pd.Series( [92,33,-5,17,122,87 ])

#Your code here...
tempsinCelsius = tempsInFarenheit.apply( toCelsius )
print( tempsinCelsius)


I am working on the file "Notebooks/Intro/One Dimensional Data Worksheet-Python Answers.ipynb".
The first task is:
Exercise 4
You are given a list of numbers called numList.  Without using a loop, write a script to count occurances of each value in the list.
Can you write Python code for it?