I am working on the file "Mayreck_style_transfer.ipynb".
The first task is:
Since this is a float image, define a function to keep the pixel values between 0 and 1:
Can you write Python code for it?

def clip_0_1(image):
  return tf.clip_by_value(image, clip_value_min=0.0, clip_value_max=1.0)


I am working on the file "Mayreck_style_transfer.ipynb".
The first task is:
The regularization loss associated with this is the sum of the squares of the values:
Can you write Python code for it?

def total_variation_loss(image):
  x_deltas, y_deltas = high_pass_x_y(image)
  return tf.reduce_sum(tf.abs(x_deltas)) + tf.reduce_sum(tf.abs(y_deltas))

total_variation_loss(image).numpy()


I am working on the file "Mayreck_style_transfer.ipynb".
The first task is:
Total variation loss
One downside to this basic implementation is that it produces a lot of high frequency artifacts. Decrease these using an explicit regularization term on the high frequency components of the image. In style transfer, this is often called the total variation loss:
Can you write Python code for it?

def high_pass_x_y(image):
  x_var = image[:,:,1:,:] - image[:,:,:-1,:]
  y_var = image[:,1:,:,:] - image[:,:-1,:,:]

  return x_var, y_var

x_deltas, y_deltas = high_pass_x_y(content_image)

plt.figure(figsize=(14,10))
plt.subplot(2,2,1)
imshow(clip_0_1(2*y_deltas+0.5), "Horizontal Deltas: Original")

plt.subplot(2,2,2)
imshow(clip_0_1(2*x_deltas+0.5), "Vertical Deltas: Original")

x_deltas, y_deltas = high_pass_x_y(image)

plt.subplot(2,2,3)
imshow(clip_0_1(2*y_deltas+0.5), "Horizontal Deltas: Styled")

plt.subplot(2,2,4)
imshow(clip_0_1(2*x_deltas+0.5), "Vertical Deltas: Styled")


I am working on the file "Mayreck_style_transfer.ipynb".
The first task is:
Use tf.GradientTape to update the image.
Can you write Python code for it?

@tf.function()
def train_step(image):
  with tf.GradientTape() as tape:
    outputs = extractor(image)
    loss = style_content_loss(outputs)

  grad = tape.gradient(loss, image)
  opt.apply_gradients([(grad, image)])
  image.assign(clip_0_1(image))


I am working on the file "Mayreck_style_transfer.ipynb".
The first task is:
When called on an image, this model returns the gram matrix (style) of the style_layers and content of the content_layers:
Can you write Python code for it?

extractor = StyleContentModel(style_layers, content_layers)

results = extractor(tf.constant(content_image))

style_results = results['style']

print('Styles:')
for name, output in sorted(results['style'].items()):
  print("  ", name)
  print("    shape: ", output.numpy().shape)
  print("    min: ", output.numpy().min())
  print("    max: ", output.numpy().max())
  print("    mean: ", output.numpy().mean())
  print()

print("Contents:")
for name, output in sorted(results['content'].items()):
  print("  ", name)
  print("    shape: ", output.numpy().shape)
  print("    min: ", output.numpy().min())
  print("    max: ", output.numpy().max())
  print("    mean: ", output.numpy().mean())


I am working on the file "McQ_workshop.ipynb".
The first task is:
An example from SDSS:
Can you write Python code for it?

# Read in the three images downloaded from here:
# g: http://dr13.sdss.org/sas/dr13/eboss/photoObj/frames/301/1737/5/frame-g-001737-5-0039.fits.bz2
# r: http://dr13.sdss.org/sas/dr13/eboss/photoObj/frames/301/1737/5/frame-r-001737-5-0039.fits.bz2
# i: http://dr13.sdss.org/sas/dr13/eboss/photoObj/frames/301/1737/5/frame-i-001737-5-0039.fits.bz2
g = fits.open('data/frame-g-001737-5-0039.fits.bz2')[0]
r = fits.open('data/frame-r-001737-5-0039.fits.bz2')[0]
i = fits.open('data/frame-i-001737-5-0039.fits.bz2')[0]

# remap r and i onto g
r_new, r_mask = reproject_interp(r, g.header)
i_new, i_mask = reproject_interp(i, g.header)

# zero out the unmapped values
i_new[np.logical_not(i_mask)] = 0
r_new[np.logical_not(r_mask)] = 0

# red=i, green=r, blue=g
# make a file with the default scaling
rgb_default = make_lupton_rgb(i_new, r_new, g.data, filename="ngc6976-default.jpeg")
# this scaling is very similar to the one used in Lupton et al. (2004)
rgb = make_lupton_rgb(i_new, r_new, g.data, Q=10, stretch=0.5, filename="ngc6976.jpeg")

plt.imshow(rgb)


I am working on the file "McQ_workshop.ipynb".
The first task is:
If you want to know more about Jupyter:
https://github.com/fjaviersanchez/JupyterTutorial/blob/master/TutorialJupyter.ipynb
Aperture photometry
Astronomers use the magnitude scale to characterize the bright of an object. With the magnitude scale you quantify the brightness of an object by comparing it with other objects. Astronomers have agreed to use "Vega" as the zero magnitude point (like the freezing point for water is the zero-point for the Celsius temperature scale). The magnitude scale goes "backwards" in the sense that brighter objects have smaller magnitude. For example the Sun has magnitude -27, the full Moon -13, and Venus -5.
How can we measure magnitudes from an image?
A first approach is to use an object which magnitude we know, called "standard" and refer the rest of the objects in an image to it.
But what do you use to count the total brightness of an object?

Use the brightest pixel?
Add the brightness in a certain radius?
Count only the pixels which belong to each object?
Can you write Python code for it?

positions = [(550., 600.), (450., 500.)] #Change it and include the position of an object in your image
apertures = CircularAperture(positions, r=20.)
phot_table = aperture_photometry(new_sci_image_g, apertures)

print phot_table


I am working on the file "McQ_workshop.ipynb".
The first task is:
Let's create a better image!
Can you write Python code for it?

new_sci_image_g = reduce_image(sci_im_g,drk_im,flat_image,bias_image, filter_dark=False)

plt.imshow(new_sci_image_g, cmap='gray', vmax=4000, vmin=50, norm=matplotlib.colors.LogNorm())
plt.colorbar()


I am working on the file "McQ_workshop.ipynb".
The first task is:
Let's take a look at the original images that you took
Can you write Python code for it?

science_image_path_g = 'data/seo_m66_g-band_180s_apagul_1.fits' #Type the path to your image
sci_g = fits.open(science_image_path_g)
sci_im_g = fits.open(science_image_path_g)[0].data
plt.imshow(sci_im_g,cmap='gray', vmax=1800, norm=matplotlib.colors.LogNorm())
plt.colorbar()


I am working on the file "McQ_workshop.ipynb".
The first task is:
This image is not science-ready yet...
Dark image: If you take a shot with the shutter closed (i.e., no light/photons incoming in the camera) you still have a non-zero image.
Can you write Python code for it?

dark_image_path='data/dark.fits' #Type the path to your dark image
drk_im = fits.open(dark_image_path)[0].data
plt.imshow(drk_im,cmap='gray', vmax=2000)
plt.colorbar()

bias_image_path = 'data/bias.fits' #Type the path to your bias image
bias_image = fits.open(bias_image_path)[0].data
plt.imshow(bias_image, cmap='gray')
plt.colorbar()

plt.hist(drk_im.flatten());
plt.yscale('log')
plt.xlabel('Output counts')
plt.ylabel('Number of pixels')


I am working on the file "McQ_workshop.ipynb".
The first task is:
Why is this?
Another interesting feature of CCD cameras is that the chips do not respond equally to the same light intensity. For example if you illuminate the camera with uniform light (this is called flat image).
Can you write Python code for it?

flat_image_path = 'data/FLAT_g-band_2016-10-06_bin1_id5908.fits' #Type the path to your flat image here
flat_image = fits.open(flat_image_path)[0].data
#You can try cmap='hot' or cmap='jet' to see how it changes
plt.imshow(flat_image, cmap='gray')
plt.colorbar()

plt.hist(flat_image.flatten())

def reduce_image(sci_im,drk_im,flat_im, bias_im, filter_dark=True):
    from scipy.stats import mode
    dkr_im = drk_im - bias_im
    #First part: We take "zero" the image
    #The next part is optional and averages the dark image in a 10 pixel radius
    #to get rid of salt/pepper noise
    if(filter_dark):
        selem = disk(10) #We are going to perform averages in 10 pixel radius disks
        selem2 = disk(4)
        drk_im = rank.mean(drk_im, selem=selem) #We perform an average to remove salt-pepper noise
        flat_im = rank.mean(flat_im, selem=selem2)
    #Second part: Make every part have the same sensitivity
    #flat_im = (flat_im - drk_im)/mode(flat_im-drk_im,axis=None)[0] #most common pixel value will equal 1
    flat_im = (flat_im - drk_im)/np.median(flat_im-drk_im)
    #Lower than 1 where the CCD is less sensitive and more than 1 where it's more sensitive
    sci_im = (sci_im -drk_im)/flat_im
    #Error image
    return sci_im


I am working on the file "MeanVarianceCorrelation.ipynb".
The first task is:
Bayesian approach to statistics

Assume there is only one dataset $X$ -- namely only the one that we have observed
Postulate a prior for the parameter $p(\Sigma)$
Compute the posterior $p(\Sigma|X)$
Can you write Python code for it?

EPOCH = 20

fig = plt.figure(figsize=(6,6))
ax = fig.gca()
Lim = 6
ax.set_ylim([-Lim,Lim])
ax.set_xlim([-Lim,Lim])

x = np.random.randn(N)
y = a*x + np.sqrt(R)*np.random.randn(N)
X = np.c_[x,y].T
cov2_est = X.dot(X.T)/N
W = np.linalg.cholesky(cov2_est)

plt.plot(x,y,'.')

for i in range(EPOCH):
    U = W.dot(np.random.randn(2,N))
    S = U.dot(U.T)/N
    dx,dy = pnorm_ball_points(3*np.linalg.cholesky(S))
    ln = plt.Line2D(dx,dy, color='k')
    ax.add_line(ln)

dx,dy = pnorm_ball_points(3*np.linalg.cholesky(Cov))
ln = plt.Line2D(dx,dy, color='r', linewidth=3)
ax.add_line(ln)

plt.show()

from notes_utilities import mat2latex

print(mat2latex(np.mat([[1,0],[0,1]])))


I am working on the file "MeanVarianceCorrelation.ipynb".
The first task is:
Every green ellipse corresponds to an estimated covariance $\Sigma^{(i)}$ from each new dataset $X^{(i)}$ sampled from the data distribution. The picture suggests that the true covariance could be somehow obtained as the average ellipse.
An estimator is called unbiased, if the true parameter is exactly the expected value of the estimator. Otherwise, the estimator is called biased.
The variance of the estimator is the amount of fluctuation around the mean. Ideally, we wish it to be small, in fact zero. However, obtaining a zero variance turns out to be impossible when the bias is zero. The variance is always greater or equal to a positive quantity called the Cramer-Rao bound.
Bootstrap
In practice, we have only a single dataset, so we need to approximate the data distribution $p(X)$. The effect of sampling new datasets can be done by sampling data points with replacement. This procedure is known as the bootstrap.
Below, we use a dataset of $M+N$
Can you write Python code for it?

EPOCH = 100
M = N

x = np.random.randn(N+M)
y = a*x + np.sqrt(R)*np.random.randn(N+M)

fig = plt.figure(figsize=(6,6))
ax = fig.gca()
Lim = 6
ax.set_ylim([-Lim,Lim])
ax.set_xlim([-Lim,Lim])

for i in range(EPOCH):
    idx = np.random.permutation(N+M)
    X = np.c_[x[idx[0:N]],y[idx[0:N]]].T
    cov2_est = X.dot(X.T)/N
    dx,dy = pnorm_ball_points(3*np.linalg.cholesky(cov2_est))
    ln = plt.Line2D(dx,dy, color='g')
    ax.add_line(ln)

dx,dy = pnorm_ball_points(3*np.linalg.cholesky(Cov))
ln = plt.Line2D(dx,dy, color='r', linewidth=3)
ax.add_line(ln)

plt.show()


I am working on the file "MeanVarianceCorrelation.ipynb".
The first task is:
Example
Generate a data set as follows
\begin{eqnarray}
x & \sim & {\mathcal N}(x; 0, 1) \
y|x & \sim & {\mathcal N}(a x, R)
\end{eqnarray}
How is this model related to the PCA?
Can you write Python code for it?

%matplotlib inline

from IPython.display import display, Math, Latex
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np

from notes_utilities import pnorm_ball_points
from notes_utilities import mat2latex

import pandas as pd
#import seaborn as sns
#sns.set(color_codes=True)

# Number of points
N = 10

# Parameters
a = -0.8
R = 0.1

x = np.random.randn(N)
y = a*x + np.sqrt(R)*np.random.randn(N)
sns.jointplot(x=x, y=y);


I am working on the file "MeanVarianceCorrelation.ipynb".
The first task is:
Frequentist approach to statistics

Assume there is a true parameter that we don't know. For example the covariance $\Sigma$
Construct an estimator (=a function that spits out a parameter value given data)
$$
\bar{\Sigma} = X^\top X/N
$$
(Conceptually) sample new random dataset from the same distribution for $i=1\dots K$
$$
X^{(i)} \sim p(X)
$$
Study the distribution of the estimator -- the output of the estimator is random as input data is random
$$
\bar{\Sigma}^{(i)} = {X^{(i)}}^\top X^{(i)}/N
$$
Can you write Python code for it?

EPOCH = 20

fig = plt.figure(figsize=(6,6))
ax = fig.gca()
Lim = 6
ax.set_ylim([-Lim,Lim])
ax.set_xlim([-Lim,Lim])

for i in range(EPOCH):
    x = np.random.randn(N)
    y = a*x + np.sqrt(R)*np.random.randn(N)
    X = np.c_[x,y].T
    cov2_est = X.dot(X.T)/N
    dx,dy = pnorm_ball_points(3*np.linalg.cholesky(cov2_est))
    ln = plt.Line2D(dx,dy, color='g')
    ax.add_line(ln)

dx,dy = pnorm_ball_points(3*np.linalg.cholesky(Cov))
ln = plt.Line2D(dx,dy, color='r', linewidth=3)
ax.add_line(ln)

plt.show()


I am working on the file "MeanVarianceCorrelation.ipynb".
The first task is:
Generative Model Example
\begin{eqnarray}
w & \sim & \mathcal{U}(0,1) \
u & = & 2 w - 1 \
x|u & \sim & \mathcal{N}\left(x; u^2, r \right)
\end{eqnarray}
Can you write Python code for it?

%matplotlib inline

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np

N = 100
r = 0.01

u = 2*np.random.randn(1,N)-1
x = u**2 + np.sqrt(r)*np.random.randn(1,N)

plt.figure(figsize=(6,6))
plt.plot(u,x,'k.')
plt.xlabel('u')
plt.ylabel('x')
plt.show()


I am working on the file "MeanVarianceCorrelation.ipynb".
The first task is:
Generative Model Example
\begin{eqnarray}
w & \sim & \mathcal{U}(0,1) \
u & = & \cos(2\pi w) \
e & \sim & \mathcal{N}\left(0, (\sigma u)^2 \left(\begin{array}{cc}  1 & 0\  0 & 1\\end{array}\right) \right) \
x & \sim & \left(\begin{array}{c} \theta_1 \ \theta_2 \end{array} \right)u + e
\end{eqnarray}
Can you write Python code for it?

%matplotlib inline

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np

N = 100
sigma = 0.8
theta = np.mat([3,-1]).T

u = np.cos(2*np.pi*np.random.rand(1,N))
X = theta*u
X = X + sigma*u*np.random.randn(X.shape[0],X.shape[1])

plt.figure(figsize=(6,6))
plt.plot(X[0,:],X[1,:],'k.')

plt.show()

import seaborn as sns
import pandas as pd

sns.set(color_codes=True)
plt.figure(figsize=(5,5))
df = pd.DataFrame(X.T, columns=['x','y'])
sns.jointplot(x="x", y="y", data=df);
plt.show()


I am working on the file "MeanVarianceCorrelation.ipynb".
The first task is:
Generative Model Example
\begin{eqnarray}
w & \sim & \mathcal{U}(w; 0,2\pi) \
\epsilon & \sim & \mathcal{N}(\epsilon; 0, I) \
u & = & \left(\begin{array}{c} \mu_1 \  \mu_2 \end{array}\right) + \left(\begin{array}{cc} s_1 & 0 \ 0& s_2 \end{array}\right) \left(\begin{array}{c} \cos(w) \  \sin(w) \end{array}\right) +  \left(\begin{array}{cc} \sigma_1 & 0 \ 0& \sigma_2 \end{array}\right)  \left(\begin{array}{c} \epsilon_1 \  \epsilon_2 \end{array}\right)
\end{eqnarray}
Can you write Python code for it?

%matplotlib inline

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np

N = 100
sigma_1 = 0.1
sigma_2 = 0.0
mu_1 = 5
mu_2 = 5
s_1 = 1
s_2 = 3

w = 2*np.pi*np.random.rand(1,N)

u1 = mu_1 + s_1*np.cos(w) + sigma_1*np.random.randn(1,N)
u2 = mu_2 + s_2*np.sin(w) + sigma_2*np.random.randn(1,N)

plt.figure(figsize=(6,6))
plt.plot(u1, u2,'k.')
plt.axis('equal')
plt.show()

for i in range(N):
    print('%3.3f %3.3f' % (u1[0,i],u2[0,i] ))


I am working on the file "MeanVarianceCorrelation.ipynb".
The first task is:
We can work out the joint distribution as:
\begin{eqnarray}
\left(\begin{array}{c}  x \  y \end{array}\right) \sim
\mathcal{N}\left( \left(\begin{array}{c}  0 \ 0 \end{array}\right) ,
\left(\begin{array}{cc}  1 & a\  a & a^2 + R \end{array}\right)
\right)
\end{eqnarray}
Can you write Python code for it?

%matplotlib inline

from IPython.display import display, Math, Latex
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np

from notes_utilities import pnorm_ball_points
from notes_utilities import mat2latex

import pandas as pd
#import seaborn as sns
#sns.set(color_codes=True)

# Number of points
N = 10

# Parameters
a = -0.8
R = 0.1

# Theoretical Covariance
Cov = np.mat([[1,a],[a, a**2+R]])

x = np.random.randn(N)
y = a*x + np.sqrt(R)*np.random.randn(N)

np.set_printoptions(precision=4)

X = np.c_[x,y].T
N = X.shape[1]

print('True Covariance')
display(Math(r'\mu='+mat2latex(np.mat('[0;0]'))))
display(Math(r'\Sigma='+mat2latex(Cov)))

print('The ML Estimates from Data')
mean_est = np.mean(X,axis=1,keepdims=True)
cov_est = np.cov(X,bias=True)
display(Math(r'\bar{m}='+mat2latex(mean_est)))
display(Math(r'\bar{S}='+mat2latex(cov_est)))

print('The estimate when we assume that we know the true mean')
cov2_est = X.dot(X.T)/N
display(Math(r'\bar{\Sigma}='+mat2latex(cov2_est)))

plt.figure(figsize=(8,8))
plt.plot(x, y, '.')
ax = plt.gca()
ax.axis('equal')
ax.set_xlabel('x')
ax.set_ylabel('y')

# True mean and Covariance
dx,dy = pnorm_ball_points(3*np.linalg.cholesky(Cov))
ln = plt.Line2D(dx,dy, color='r')
ln.set_label('True')
ax.add_line(ln)
ln = plt.Line2D([0],[0], color='r', marker='o')
ax.add_line(ln)

dx,dy = pnorm_ball_points(3*np.linalg.cholesky(Cov), mu=mean_est)
ln = plt.Line2D(dx,dy, color='b')
ln.set_label('ML Estimate')
ax.add_line(ln)
ln = plt.Line2D(mean_est[0],mean_est[1], color='b', marker='o')
ax.add_line(ln)

# Estimate conditioned on knowing the true mean
dx,dy = pnorm_ball_points(3*np.linalg.cholesky(cov2_est))
ln = plt.Line2D(dx,dy, color='g')
ln.set_label('Conditioned on true mean')
ax.add_line(ln)
ln = plt.Line2D([0],[0], color='g', marker='o')
ax.add_line(ln)

Lim = 6
ax.set_ylim([-Lim,Lim])
ax.set_xlim([-Lim,Lim])

ax.legend()
plt.title('Covariance Matrix Estimates')
plt.show()


I am working on the file "MeasureCompanionRVs.ipynb".
The first task is:
Fix the dates to line up with the RV output file
Can you write Python code for it?

import pandas as pd
rv1_data = pd.read_fwf('../Planet-Finder/data/psi1draa_140p_28_37_ASW.dat', header=None)
t1 = rv1_data[0].values
rv1 = rv1_data[2].values / 1000. # Convert from m/s to km/s
rv1_err = rv1_data[3].values / 1000.

new_rv2 = np.empty_like(t1)
new_rv2_err = np.empty_like(t1)
for i, t1_i in enumerate(t1):
    #idx = np.searchsorted(t1, t2)
    idx = np.argmin(np.abs(date-t1_i))
    if abs(t1_i - date[idx]) < 0.001:
        print i, t1_i, date[idx], t1_i-date[idx]
        new_rv2[i] = rv2[idx]
        new_rv2_err[i] = rv2_err[idx]
    else:
        print i, t1_i, date[idx], np.nan
        new_rv2[i] = np.nan
        new_rv2_err[i] = np.nan

np.savetxt('../Planet-Finder/data/rv_data.txt', (t1, rv1, rv1_err, new_rv2, new_rv2_err))


I am working on the file "MeasureRV.ipynb".
The first task is:
Fringing. Check the FFT
Can you write Python code for it?

fft = np.fft.rfft(denoised.y)
freq = np.fft.rfftfreq(denoised.size(), d=denoised.x[1] - denoised.x[0])

fig, ax = plt.subplots(1, 1, figsize=(15,6))
ax.semilogy(freq, fft.imag**2 + fft.real**2)
ax.set_xlim((0, 5))

good = freq > 0.5
pars = np.polyfit(freq[good], np.log10(fft[good]), 4)
fit = np.poly1d(pars)
fft_fit = 10**fit(freq)
plt.plot(freq, fft_fit.imag**2 + fft_fit.real**2, 'r-')


I am working on the file "MeasureRV.ipynb".
The first task is:
Is the wavelength solution bad?
I fit each order separately when I do the telluric correction? What if I applied a global solution to the wavelength from the telluric lines?
Can you write Python code for it?

def fit_2dspec(xl, yl, zl, x_degree=4, y_degree=3,
               x_domain=None, y_domain=None):
    from astropy.modeling import fitting
    # Fit the data using astropy.modeling
    if x_domain is None:
        x_domain = [min(xl), max(xl)]
    # more room for y_domain??
    if y_domain is None:
        #y_domain = [orders[0]-2, orders[-1]+2]
        y_domain = [min(yl), max(yl)]
    from astropy.modeling.polynomial import Chebyshev2D
    p_init = Chebyshev2D(x_degree=x_degree, y_degree=y_degree,
                         x_domain=x_domain, y_domain=y_domain)
    f = fitting.LinearLSQFitter()

    p = f(p_init, xl, yl, zl)

    for i in [0]:
        dd = p(xl, yl) - zl
        m = np.abs(dd) < 3.*dd.std()
        p = f(p, xl[m], yl[m], zl[m])

    return p, m

from FittingUtilities import FindLines
def fit_wavelength(orders, ordernums, first_order=None, last_order=None, x_degree=4, y_degree=3):
    """ Fit the wavelength in a whole chip, and return the 2D polynomial callable
    """
    pixel_list = []
    ordernum_list = []
    wave_list = []
    if first_order is None:
        first_order = 0
    if last_order is None:
        last_order = len(orders) - 1

    for order, ordernum in zip(orders[first_order:last_order+1], ordernums[first_order:last_order+1]):
        lines = FindLines(order)
        pixel_list.extend(lines)
        ordernum_list.extend(np.ones_like(lines)*ordernum)
        wave_list.extend(order.x[lines])

    pixel_list = np.array(pixel_list)
    ordernum_list = np.array(ordernum_list)
    wave_list = np.array(wave_list)
    p, m = fit_2dspec(pixel_list, ordernum_list, wave_list*ordernum_list,
                      x_degree=x_degree, y_degree=y_degree)

    return p

import DataStructures
fname = '../20141015/Corrected_HIP_109056-0.fits'
hdulist = fits.open(fname)
model_orders = [DataStructures.xypoint(x=hdu.data['wavelength'], y=hdu.data['model']) for hdu in hdulist[1:]]
data_orders = [DataStructures.xypoint(x=hdu.data['wavelength'], y=hdu.data['flux'],
            ...


I am working on the file "MeasureRV.ipynb".
The first task is:
There is definitely something there visible in the FFTs. I will fit the fft, and replace all values with frequencies between 1 and 2 cycles/nm to the fitted value.
Can you write Python code for it?

good = (freq > 1) & (freq < 2)
fft[good] = fft_fit[good]
spec = np.fft.irfft(fft)
defringed = total.copy()
defringed.y = spec

fig, ax = plt.subplots(1, 1, figsize=(13,10))
ax.plot(total.x, total.y, 'k-', alpha=0.5)
#ax.plot(denoised.x, denoised.y, 'r-', alpha=0.6)
ax.plot(defringed.x, defringed.y, 'g-', alpha=0.6)
ax.set_xlim((1635, 1645))
ax.set_ylim((0.85, 0.99))


I am working on the file "MeasureRV.ipynb".
The first task is:
Try cross-correlating the whole spectrum at once.
Can you write Python code for it?

import StellarModel
reload(StellarModel)
HDF5_FILENAME = '/media/ExtraSpace/Kurucz_FullGrid/IGRINS_grid_air.hdf5'
model_list = StellarModel.GetModelList(metal=0.0, logg=4.0, temperature=9250,
                                       type='hdf5', hdf5_file=HDF5_FILENAME)
modeldict, _ = StellarModel.MakeModelDicts(model_list, type='hdf5', hdf5_file=HDF5_FILENAME,
                                        vsini_values=150, vac2air=True, logspace=True)
model = modeldict[9250][4.0][0.0][0.0][150]

import Correlate
ccf = Correlate.GetCCF([defringed], model, vsini=150, resolution=45000, addmode='ml')

corr = ccf['CCF']
plt.plot(corr.x, corr.y)
Correlate.get_rv(corr.x, corr.y, Npix=defringed.size())

good = (corr.y > 0.9*corr.y.max())
plt.plot(corr.x[good], corr.y[good])
pars = np.polyfit(corr.x[good], corr.y[good], 2)
plt.plot(corr.x[good], np.poly1d(pars)(corr.x[good]), 'r--')
centroid = pars[1] / (2.0*pars[0])
print(centroid)


I am working on the file "MeasureRV.ipynb".
The first task is:
Try something else
IGRINS data only has the broad Hydrogen lines to go off of, and has lots of systematics/noise. So, let's try to just measure the effective wavelength of each line and get a mean velocity from the ~3-4 lines.
Can you write Python code for it?

fname = '../20141015/HIP_109056_telluric_corrected_renormalized.fits'
orders = HelperFunctions.ReadExtensionFits(fname)
total = HelperFunctions.CombineXYpoints(orders)
denoised = HelperFunctions.Denoise(total.copy())

fig, ax = plt.subplots(1, 1, figsize=(13,6))
ax.plot(total.x, total.y, 'k-', alpha=0.5)
ax.plot(denoised.x, denoised.y, 'r-', alpha=0.6)


I am working on the file "Measurements_and_their_errors/Measurements_and_their_errors.ipynb".
The first task is:
Contents
1.Base Units
2.Derived Units
- List of ones with special names
3.Prefixes
4.Physical Measurements
- Errors
- Measurements

Base Units

these are globally agreed systems of units
they can also be called SI base units (Système International d'Unités)
there are 7
lower-case symbols unless named after person
form set of mutually independent dimensions

Ampere (A)

electrical current

Candela (cd)

luminous intensity

Kelvin (K)

thermodynamic temperature

Kilogram (kg)

mass

Metre (m)

length

Mole (mol)

amount of substance

Second (s)

time

Derived Units

These are units are derived from base units
either dimensionless, expressed as product of one or more of the base units, possibly scaled by appropriate power of exponentiation
22 units with special names
many units don't (eg. $m^2$)
for Equivalents and SI equivalents, go here

List of ones with special names
hertz (Hz)

frequency

radian (rad)

angle

steradian (sr)

solid angle

newton (N)

force, weight

pascal (Pa)

pressure, stress

joule (J)

energy, work, heat

watt (W)

power, radiant flux

coulomb (C)

electric charge or quantity of electricity

volt (V)

voltage, electrical potential difference, electromotive force

farad (F)

electrical capacitance

ohm ($\Omega$)

electrical resistance, impedance, reactance

siemens (S)

electrical conductance

weber (Wb)

magnetic flux

tesla (T)

magnetic field strength, magnetic flux density

henry (H)

electrical inductance

degree celsius ($^{\circ} C$)

temperature relative to 273.15 K

lumen (lm)

luminous flux

lux (lx)

illuminance

becquerel (Bq)

radioactivity (decays per unit time)

gray (Gy)

absorbed dose (of ionizing radiation)

sievert (Sv)

equivalent dose (of ionizing radiation)

katal (kat)

catalytic activity

Prefixes
| Symbol | Prefix | Value      |
|--------|--------|------------|
| E      | exa    | $10^{18}$  |
| P      | peta   | $10^{15}$  |
| T      |...
Can you write Python code for it?

x = np.linspace(0, 1, 4)
y = np.exp(x)

yerr = 0.1 + 0.08*np.sqrt(x)
xerr = 0.1 + yerr

fig, ax = plt.subplots()
ax.errorbar(x, y, xerr=xerr, yerr=yerr, color='red', linestyle='-', ecolor='b')

ax.set_title("error bar example")
ax.grid(True)

plt.show()


I am working on the file "Measurements_and_their_errors/Measurements_and_their_errors.ipynb".
The first task is:
Gradient Uncertainty
On a graph, we:
- draw the main line
- draw the error bars
- draw a line which fit in the error bars area of maximum gradient
- draw a line which fit in the error bars area of minimum gradient
- compare the gradients to the optimal line
Intercept Uncertainty
The same as gradient uncertainty but instead comparing the y intercept
Can you write Python code for it?

x = np.linspace(1, 4, 3)

x2 = np.linspace(0, 5, 5)

def f(x):
    return 2*x

y = f(x)

yerr = 0.9
xerr = 0.1

maxg = f(0.7*x)+1.5
ming = f(1.3*x)-1.5

maxi = f(0.7*x2)+1.5
mini = f(1.3*x2)-1.5

fig, ax = plt.subplots(1, 2, figsize=(15,7))

ax[0].errorbar(x, y, xerr=xerr, yerr=yerr, color='red', linestyle='-', ecolor='b', label='optimal line')

ax[0].plot([x[0]-xerr, x[1], x[-1]+xerr], maxg, label='min gradient')
ax[0].plot([x[0]+xerr, x[1], x[-1]-xerr], ming, label='max gradient')

ax[0].set_title("error bar gradient example")

ax[1].errorbar(x, y, xerr=xerr, yerr=yerr, color='red', linestyle='-', ecolor='b', label='optimal line')

ax[1].plot(x2, maxi, label='max intercept')
ax[1].plot(x2, mini, label='min intercept')

ax[1].set_title("error bar intercept example")

for axes in ax:
    axes.grid(True)
    axes.legend()

plt.show()


I am working on the file "MeasuringPROVProvenanceWebofData.ipynb".
The first task is:
Additionally, all code is available online
Can you write Python code for it?

import requests
nsr = requests.get("http://index.lodlaundromat.org/ns2d/", params={"uri":"http://www.w3.org/ns/prov#"})
total_prov_docs = nsr.json()["totalResults"]

nsr = requests.get("http://index.lodlaundromat.org/ns2d/", params={"uri":"http://www.w3.org/ns/prov#","limit":total_prov_docs} )

import io
from rdflib.namespace import RDFS, RDF
from rdflib.namespace import Namespace
from rdflib import Graph
from rdflib import URIRef
PROV = Namespace('http://www.w3.org/ns/prov#')

entitySubclasses = []
activitySubclasses = []
agentSubclasses = []
totalNumberOfEntities = 0
totalNumberOfActivities = 0
totalNumberOfAgents = 0
numWasDerivedFrom = 0
numUsed = 0
numWGB = 0
numWAW = 0
numWasAttributedTo = 0

for doc in nsr.json()["results"]:
    #print(doc)
    headers = {'Accept': 'text/turtle'}
    x = requests.get("http://ldf.lodlaundromat.org/" + doc, headers=headers)
    txt_res = x.text
    tmpGraph = Graph()
    tmpGraph.parse(io.StringIO(txt_res), format="turtle")
    #print(doc + " " + str(len(tmpGraph)))
    for entityClass in tmpGraph.subjects(RDFS.subClassOf, PROV.Entity):
        #print(entityClass)
        entitySubclasses.append(entityClass)
    for entity in tmpGraph.subjects(RDF.type, PROV.Entity):
        totalNumberOfEntities = totalNumberOfEntities + 1

    for activityClass in tmpGraph.subjects(RDFS.subClassOf, PROV.Activity):
        #print(activityClass)
        activitySubclasses.append(activityClass)

    for activity in tmpGraph.subjects(RDF.type, PROV.Activity):
        totalNumberOfActivities = totalNumberOfActivities + 1

    for agentClass in tmpGraph.subjects(RDFS.subClassOf, PROV.Agent):
        #print(agentClass)
        agentSubclasses.append(agentClass)

    for agent in tmpGraph.subjects(RDF.type, PROV.Agent):
        totalNumberOfAgents = totalNumberOfAgents + 1

    ##look at relations

    for s,p,o in tmpGraph.triples( (None, PROV.wasDerivedFrom, None )):
        numWasDerivedFrom = numWasDerivedFrom + 1

    for s,p,o in tmpGraph.triples( (None,...


I am working on the file "MeasuringPROVProvenanceWebofData.ipynb".
The first task is:
We also note that PROV has been extended by 8 other ontologies as calculated by manual inspection of the extensions of the various core classes as listed in the appendix.
Conclusion
This initial analysis shows some uptake within the Semantic Web community. However, while PROV is widely referenced within the community's literature, it appears, that direct usage of the standard could be improved (at least within the dataset represented by the LOD Laudromat). It should be noted that our analysis is preliminary and there is a much  room for further work. In particular, we aim to look at the indirect usage of PROV through usage by ontologies that extend it (e.g. The Provenance Vocabulary) or that map to it such as Dublin Core or PAV. Understanding such indirect usage will help us better understand the true state of provenance interoperability within Linked Data. Likewise, it would be interesting to perform network analysis to understand the role that PROV plays within the Linked Data network.
References

Beek, W. & Rietveld, L & Bazoobandi, H.R. & Wielemaker, J. & Schlobach, S.: LOD Laundromat: A Uniform Way of Publishing Other People's Dirty Data. Proceedings of the International Semantic Web Conference (2014).
Rietveld, L. & Beek, W. & Schlobach, S.: LOD Lab: Experiments at LOD Scale. Proceedings of the International Semantic Web Conference (2015).

Appendix: Classes that subclass a PROV core class
Can you write Python code for it?

print("Subclasses of Entity")
for i in entitySubclasses:
    print(i)
print("Subclasses of Activity")
for i in activitySubclasses:
    print(i)
print("Subclasses of Agent")
for i in agentSubclasses:
    print(i)


I am working on the file "MedianFilter/.ipynb_checkpoints/Basic Test Error of the Median filter with different window length-checkpoint.ipynb".
The first task is:
I plot the error of the filtered wave. I use the absulte values of the difference between sine wave and median filtered wave and calculate the mean, to get the error. I use a wave number of 5 and different window lengths
Functions
Can you write Python code for it?

def ErrorPlot( waveNumber,windowLength ):
        data = np.fromfunction( lambda x: np.sin((x-windowLength / 2)/128 * 2 * np.pi * waveNumber), (128 + windowLength / 2, ) )    #creating an array with a sine wave
        datafiltered = medianFilter(data, windowLength)  #calculate the filtered wave with the medianFiltered function
        data = data[ windowLength / 2 : - windowLength ] # slice the data array to synchronize both waves
        datafiltered = datafiltered[ : len(data) ]       # cut the filtered wave to the same length as the data wave
        error = ErrorRate(data,datafiltered,windowLength,waveNumber) #calculate the error with the ErrorRate function
        plt.axis([0, y + 1, 0, 1.2])
        plt.xlabel('Window Length', fontsize = 20)
        plt.ylabel('Error rate', fontsize = 20)
        plt.scatter(*error)

def ErrorRate(data,datafiltered,windowLength, waveNumber):
    errorrate = data-datafiltered  #calculate the difference between the sine wave and the filtered wave
    error = [] #creating a list and save the error rate with the matching wavenumber in it
    errorrate = np.abs(errorrate)
    error.append([windowLength ,np.mean(errorrate)])# fill the list with the errorrate and corresponding window length
    error = zip(*error) #zip the error ([1,1],[2,2],[3,3]) = ([1,2,3],[1,2,3])
    return error

def medianFilter( data, windowLength ):
    if (windowLength < len(data)and data.ndim == 1):
        tempret = np.zeros(len(data)-windowLength+1)  # creating an array where the filtered values will be saved in
        if windowLength % 2 ==0:                      # check if the window length is odd or even because with even window length we get an unsynchrone filtered wave
            for c in range(0, len(tempret)):
                tempret[c] = np.median( data[ c : c + windowLength +1 ] ) # write the values of the median filtered wave in tempret, calculate the median of all values in the window
            return tempret
        else:
            for c in range(0, len(tempret)):
           ...


I am working on the file "MedianFilter/.ipynb_checkpoints/Summary of the error rate with different wave numbers-checkpoint.ipynb".
The first task is:
Here you can see, that the error rate at wave number 16 and 32 is lower then expected.
Can you write Python code for it?

fig = plt.figure(1, figsize=(15,3))
medianSinPlot(15,5)
plt.title('Wave number 15')
fig = plt.figure(2,figsize=(15,3))
medianSinPlot(16,5)
plt.title('Wave number 16')
fig = plt.figure(3,figsize=(15,3))
medianSinPlot(17,5)
plt.title('Wave number 17')

fig = plt.figure(1, figsize=(15,3))
medianSinPlot(31,5)
plt.title('Wave number 31')
fig = plt.figure(2,figsize=(15,3))
medianSinPlot(32,5)
plt.title('Wave number 32')
fig = plt.figure(3,figsize=(15,3))
medianSinPlot(33,5)
plt.title('Wave number 33')


I am working on the file "MedianFilter/.ipynb_checkpoints/Summary of the error rate with different wave numbers-checkpoint.ipynb".
The first task is:
Plots
Can you write Python code for it?

fig = plt.figure()
for y in range (0,40,2):
    ErrorPlot(y,5)


I am working on the file "MedianFilter/.ipynb_checkpoints/Summary sine with more samples(1024)-checkpoint.ipynb".
The first task is:
Testing with more samples ( now 1024, before 128)
Functions
Can you write Python code for it?

def ErrorPlot( waveNumber,windowLength ):
        data = np.fromfunction( lambda x: np.sin((x-windowLength / 2)/1024 * 2 * np.pi * waveNumber), (1024 + windowLength / 2, ) )    #creating an array with a sine wave
        datafiltered = medianFilter(data, windowLength)  #calculate the filtered wave with the medianFiltered function
        data = data[ windowLength / 2 : - windowLength ] # slice the data array to synchronize both waves
        datafiltered = datafiltered[ : len(data) ]       # cut the filtered wave to the same length as the data wave
        error = ErrorRate(data,datafiltered,windowLength,waveNumber) #calculate the error with the ErrorRate function
        plt.axis([0, y + 1, 0, 1.2])
        plt.xlabel('Wave number', fontsize = 20)
        plt.ylabel('Error rate', fontsize = 20)
        plt.scatter(*error)

def ErrorRate(data,datafiltered,windowLength, waveNumber):
    errorrate = data-datafiltered  #calculate the difference between the sine wave and the filtered wave
    error = [] #creating a list and save the error rate with the matching wavenumber in it
    errorrate = np.abs(errorrate)
    error.append([waveNumber ,np.mean(errorrate)])# fill the list with the errorrate and corresponding wave number
    error = zip(*error) #zip the error ([1,1],[2,2],[3,3]) = ([1,2,3],[1,2,3])
    return error

def medianFilter( data, windowLength ):
    if (windowLength < len(data)and data.ndim == 1):
        tempret = np.zeros(len(data)-windowLength+1)  # creating an array where the filtered values will be saved in
        if windowLength % 2 ==0:                      # check if the window length is odd or even because with even window length we get an unsynchrone filtered wave
            for c in range(0, len(tempret)):
                tempret[c] = np.median( data[ c : c + windowLength +1 ] ) # write the values of the median filtered wave in tempret, calculate the median of all values in the window
            return tempret
        else:
            for c in range(0, len(tempret)):
               ...


I am working on the file "MedianFilter/.ipynb_checkpoints/Summary sine with more samples(1024)-checkpoint.ipynb".
The first task is:
With more samples the error rate at wave number 16 and 32 is no longer lower then expected.
Can you write Python code for it?

fig = plt.figure(1, figsize=(15,3))
medianSinPlot(15,128)
plt.title('Wave number 15')
fig = plt.figure(2,figsize=(15,3))
medianSinPlot(16,128)
plt.title('Wave number 16')
fig = plt.figure(3,figsize=(15,3))
medianSinPlot(17,128)
plt.title('Wave number 17')

fig = plt.figure(1, figsize=(15,3))
medianSinPlot(31,128)
plt.title('Wave number 31')
fig = plt.figure(2,figsize=(15,3))
medianSinPlot(32,128)
plt.title('Wave number 32')
fig = plt.figure(3,figsize=(15,3))
medianSinPlot(33,128)
plt.title('Wave number 33')


I am working on the file "MedianFilter/Python/01. Basic Tests Median Filter/basic median filter with window length 16.ipynb".
The first task is:
Testing the median filter with a fixed window length of 16.
Can you write Python code for it?

median = plt.figure(figsize=(30,20))
for x in range(1, 5):
    for y in range(1, 6):
        plt.subplot(5, 5, x + (y-1)*4)
        wavenum = (x-1) + (y-1)*4
        functions.medianSinPlot(wavenum, 15)
        plt.suptitle('Median filtered Sine Waves with window length 15', fontsize = 60)
        plt.xlabel(("Wave number = "+str((x-1) + (y-1)*4)), fontsize=18)


I am working on the file "MedianFilter/Python/04. Summaries/.ipynb_checkpoints/Summary sine with more samples(1024)-checkpoint.ipynb".
The first task is:
Testing with more samples ( now 1024, before 128)
Plots
Can you write Python code for it?

fig = plt.figure()
for i in range (0, 40):
    functions.ErrorPlotWave1024(i, 128)


I am working on the file "MedianFilter/Python/04. Summaries/.ipynb_checkpoints/Summary sine with more samples(1024)-checkpoint.ipynb".
The first task is:
With more samples the error rate at wave number 16 and 32 is no longer lower then expected.
Can you write Python code for it?

fig = plt.figure(1, figsize=(15, 3))
functions.medianSinPlot1024(15, 128)
plt.title('Wave number 15')
fig = plt.figure(2, figsize=(15, 3))
functions.medianSinPlot1024(16, 128)
plt.title('Wave number 16')
fig = plt.figure(3, figsize=(15, 3))
functions.medianSinPlot1024(17, 128)
plt.title('Wave number 17')

fig = plt.figure(1, figsize=(15, 3))
functions.medianSinPlot1024(31, 128)
plt.title('Wave number 31')
fig = plt.figure(2, figsize=(15, 3))
functions.medianSinPlot1024(32, 128)
plt.title('Wave number 32')
fig = plt.figure(3, figsize=(15, 3))
functions.medianSinPlot1024(33, 128)
plt.title('Wave number 33')


I am working on the file "MedianFilter/Python/04. Summaries/Summary of the error rate of the median with different window lengths.ipynb".
The first task is:
As you can see, the error is higher when the window is bigger.
The error rate is the mean of the absolute of the red line. The red line is the difference between the sine wave and the median filtered wave. With a bigger window the difference between median filtered wave and sine wave gets bigger. Thus we can say that the error depends on the size of the window.
Can you write Python code for it?

plt.figure(figsize=(30,20))
for x in range(1, 5):
    for y in range(1, 6):
        # Creates different subplots in one figure, with x and y.
        # the window length is calculate.
        plt.subplot(5, 5, x + (y-1)*4)
        windowLength = ((x-1)*2 + (y-1)*8)*8 +1
        functions.medianSinPlot(16, windowLength, 16*128,128*5+1,-128*5-1)
        plt.suptitle('Median filtered sine waves', fontsize = 48)
        plt.xlabel(("Window length (/2 pi) = " + str(windowLength/128.)), fontsize=18)


I am working on the file "MedianFilter/Python/04. Summaries/Summary of the error rate of the median with different window lengths.ipynb".
The first task is:
At the beginning you can see the error rises with a higher wave number. But at the end there are some unlikely curves. This is because of the low samples(128).
Visualization of the Error rate with different wave numbers and different window lengths(1024 samples)
Can you write Python code for it?

fig = plt.figure(figsize=(30,20))
for x in range(1, 5):
    for y in range(1, 6):
        for z in range(0,160,3):
            if z%2 ==1:
                plt.subplot(5, 5, x + (y-1)*4)
                wavenum = (x-1) + (y-1)*4
                functions.ErrorPlotWindow( wavenum, z, 1024 )
                plt.suptitle(' Error of the Median filter with different wave numbers and window lengths ',
                             fontsize = 40)
                plt.xlabel(("Wave number = " + str((x-1) + (y-1)*4)), fontsize = 18)
plt.savefig('SumErrorWL')


I am working on the file "MedianFilter/Python/04. Summaries/Summary of the error rate of the median with different window lengths.ipynb".
The first task is:
Visualization of the Error rate with different wave numbers and different window lengths
Can you write Python code for it?

fig = plt.figure(figsize=(30,20))
for x in range(1, 5):
    for y in range(1, 6):
        for z in range(0,40):
            if z%2 ==1:
                plt.subplot(5, 5, x + (y-1)*4)
                wavenum = (x-1) + (y-1)*4
                functions.ErrorPlotWindow(wavenum, z)
                plt.suptitle(' Error of the Median filter with different wave numbers and window lengths',
                             fontsize = 40)
                plt.xlabel(("Wave number = " + str((x-1) + (y-1)*4)), fontsize = 18)


I am working on the file "MedianFilter/Python/04. Summaries/Summary sine with more samples(1024).ipynb".
The first task is:
Testing with more samples ( now 1024, before 128)
Plots
Can you write Python code for it?

fig = plt.figure()
for i in range (0, 40):
    functions.ErrorPlotWave(i, 127,1024)


I am working on the file "MedianFilter/Python/04. Summaries/Summary sine with more samples(1024).ipynb".
The first task is:
With more samples the error rate at wave number 16 and 32 is no longer lower then expected.
Can you write Python code for it?

fig = plt.figure(1, figsize=(15, 3))
functions.medianSinPlot(15, 127)
plt.title('Wave number 15')
fig = plt.figure(2, figsize=(15, 3))
functions.medianSinPlot(16, 127)
plt.title('Wave number 16')
fig = plt.figure(3, figsize=(15, 3))
functions.medianSinPlot(17, 127)
plt.title('Wave number 17')

fig = plt.figure(1, figsize=(15, 3))
functions.medianSinPlot(31, 127,1024)
plt.title('Wave number 31')
fig = plt.figure(2, figsize=(15, 3))
functions.medianSinPlot(32, 127,1024)
plt.title('Wave number 32')
fig = plt.figure(3, figsize=(15, 3))
functions.medianSinPlot(33, 127,1024)
plt.title('Wave number 33')


I am working on the file "MedianFilter/Python/05. Smooth the Signal/Noise removal with moving averege and gaussian filter.ipynb".
The first task is:
Smooth the signal with a moving averege filter.
Can you write Python code for it?

# Moving averege filter with a length of 30, 5 times calculated.
for i in range (0,5):
    signal = np.convolve(signal, np.ones(30)/30, mode = 'same')

plt.plot(signal)
plt.title("Result of the moving averege filter")


I am working on the file "MedianFilter/Python/06. Behavior of the median filter with white noise/.ipynb_checkpoints/Behavior of the median filter with a noised sine wave-checkpoint.ipynb".
The first task is:
Figure 1. Behavior of the median filter with given window length and different S/N ratio.
Can you write Python code for it?

# Calculate and save all values.
# Because the for loop doesn't count from 1 to 10 for example,
# we need a counter to iterate through the array.
# The counter is assigne to -1, so we can iterate from 0 to len(values)
count = -1
count2 = -1
values = np.zeros((len(sn), len(wl)))
for w in  wl[:11]:
    count = count + 1
    for x in sn:
        count2 = count2 + 1
        for i in range (len(diff_noise)):
            # Create different noises, with x we change the signal to noise
            # ratio from 10 to 1.
            diff_noise[i, :] = np.random.normal(0, 0.706341266/np.sqrt(x), len(data))
            # Add noise to each sine wave, to create a realisitc signal.
            noised_sines[i, :] = data + diff_noise[i, :]
            # Filter the all noised sine waves.
            medfilter[i, :] = medfilt(noised_sines[i, :], w)
            # Subtract the filtered wave from the noised sine waves.
            filtered_sines[i, :] = noised_sines[i, :] - medfilter[i, :]
            # Calculate the root mean square (RMS) of each sine wave
            behav[i] = np.sqrt(np.mean(np.square(filtered_sines[i, :])))
        # Calculate the mean of the bahvior, so we can see how
        # the signal to noise ratio effects the median filter
        # with different window lengths.
        mean = np.mean(behav)
        # Save the result in the 'values' array
        values[count2:count2+1:,count] = mean
    # Set coun2 back to -1, so we can iterate again from 0 to len(values).
    # Otherwise the counter would get higher and is out of range.
    count2 = - 1
# Save the array, because the calculation take some time.
# Load the array with "values = np.loadtxt('values.txt')".
np.savetxt("values.txt", values)

values = np.loadtxt("values.txt")

viridis_data = np.loadtxt('viridis_data.txt')
plasma_data = np.loadtxt('plasma_data.txt')
# viris_data and plasma_data taken from
# https://github.com/BIDS/colormap/blob/master/colormaps.py

fig = plt.figure(figsize=(20, 7))
for p in range(0,11):
    ax = plt.subplot(2, 5,  p)
   ...


I am working on the file "MedianFilter/Python/06. Behavior of the median filter with white noise/.ipynb_checkpoints/Behavior of the median filter with a noised sine wave-checkpoint.ipynb".
The first task is:
Figure 1.1 Behavior of the median filter with given window length and different S/N ratio.
Can you write Python code for it?

# Alternative we subtract the filtered wave from the original sine wave,
# not from the noised sine wave.
count = -1
count2 = -1
values = np.zeros((len(sn), len(wl)))
for w in  wl[:11]:
    count = count + 1
    for x in sn:
        count2 = count2 + 1
        for i in range (len(diff_noise)):
            diff_noise[i, :] = np.random.normal(0, 0.706341266/np.sqrt(x), len(data))
            noised_sines[i, :] = data + diff_noise[i, :]
            medfilter[i, :] = medfilt(noised_sines[i, :], w)
            filtered_sines[i, :] = data - medfilter[i, :]
            behav[i] = np.sqrt(np.mean(np.square(filtered_sines[i, :])))
        mean = np.mean(behav)
        values[count2:count2+1:,count] = mean
    count2 = - 1
np.savetxt("valuesA.txt", values)

valuesA = np.loadtxt("valuesA.txt")

fig = plt.figure()
values3 = np.zeros((len(sn),len(wl)))
for p in range(6):
    ax = plt.subplot()
    values3[::,p] = valuesA[::,p]/0.7069341
    plt.axis([0, 11, 0, 2])
    plt.ylabel('Normalized RMS', size = 14)
    plt.xlabel('S/N Ratio', size = 14)
    plt.hlines(1,1,101, color = 'b', linestyle = '--')
    plt.plot(sn,values3[::,p], color=plasma_data[(p*40),:])
plt.savefig('Behavior with given SN ratio and different wl3A.png',dpi=300)


I am working on the file "MedianFilter/Python/06. Behavior of the median filter with white noise/.ipynb_checkpoints/Behavior of the median filter with a noised sine wave-checkpoint.ipynb".
The first task is:
Figure 2.1: Behavior of the median filter with given window length and different S/N ratio
Can you write Python code for it?

# Alternative
values = np.zeros((len(wl), len(sn)))
count = -1
count2 = -1
for x in sn:
    count = count + 1
    for w in wl:
        count2 = count2 + 1
        for i in range (len(diff_noise)):
            diff_noise[i, :] = np.random.normal(0, 0.706341266/np.sqrt(x), len(data))
            noised_sines[i, :] = data + diff_noise[i, :]
            medfilter[i, :] = medfilt(noised_sines[i, :], w)
            filtered_sines[i, :] = data - medfilter[i, :]
            behav[i] = np.sqrt(np.mean(np.square(filtered_sines[i, :])))
        mean = np.mean(behav)
        values[count2:count2+1:,-count] = mean
    count2 = -1
np.savetxt("values2A.txt", values)

values2A = np.loadtxt("values2A.txt")

fig = plt.figure()
values4 = np.zeros((len(wl), len(sn)))
for i in range (11):
    # Normalize the RMS with the RMS of a normal sine wave
    values4[::,i] = values2A[::,i]/0.7069341
    ax = plt.subplot()
    plt.axis([0, 450, 0, 2])
    # Set xticks at each 64th point
    xticks = np.arange(0, max(wl) + 1, 64)
    ax.set_xticks(xticks)
    # x_labe = pi at each 64th point
    x_label = [r"${%s\pi}$" % (v) for v in range(len(xticks))]
    ax.set_xticklabels(x_label)
    plt.ylabel('Normalized RMS', size = 14)
    plt.xlabel('Window length', size = 14)
    plt.plot(wl,values4[::,i], color=viridis_data[(i*25)-25,:])
    plt.hlines(1,1,max(wl), color = 'b', linestyle = '--')
plt.savefig('Behavior with given wl and different SN ratio2A.png',dpi=300)


I am working on the file "MedianFilter/Python/06. Behavior of the median filter with white noise/.ipynb_checkpoints/Behavior of the median filter with a noised sine wave-checkpoint.ipynb".
The first task is:
Figure 2: Behavior of the median filter with given window length and different S/N ratio
Can you write Python code for it?

values = np.zeros((len(wl), len(sn)))
count = -1
count2 = -1
for x in sn:
    count = count + 1
    for w in wl:
        count2 = count2 + 1
        for i in range (len(diff_noise)):
            diff_noise[i, :] = np.random.normal(0, 0.706341266/np.sqrt(x), len(data))
            noised_sines[i, :] = data + diff_noise[i, :]
            medfilter[i, :] = medfilt(noised_sines[i, :], w)
            filtered_sines[i, :] = noised_sines[i, :] - medfilter[i, :]
            behav[i] = np.sqrt(np.mean(np.square(filtered_sines[i, :])))
        mean = np.mean(behav)
        values[count2:count2+1:,-count] = mean
    count2 = -1
np.savetxt("values2.txt", values)

values2 = np.loadtxt("values2.txt")

fig = plt.figure(figsize=(30,7))
for p in range(11):
    ax = plt.subplot(2,5,p)
    plt.axis([0, 450, 0, 1.5])
    xticks = np.arange(0, max(wl), 64)
    ax.set_xticks(xticks)
    x_label = [r"${%s\pi}$" % (v) for v in range(len(xticks))]
    ax.set_xticklabels(x_label)
    plt.plot(wl,values2[::,p], color=viridis_data[(p*25),:])
plt.savefig('Behavior with given wl and different SN ratio.png',dpi=300)

fig = plt.figure()
values4 = np.zeros((len(wl), len(sn)))
for p in range (10):
    # Normalize the RMS with the RMS of a normal sine wave
    values4[::,p] = values2[::,p]/0.7069341
    ax = plt.subplot()
    plt.axis([0, 450, 0, 2])
    # Set xticks at each 64th point
    xticks = np.arange(0, max(wl) + 1, 64)
    ax.set_xticks(xticks)
    # x_label = pi at each 64th point
    x_label = [r"${%s\pi}$" % (v) for v in range(len(xticks))]
    ax.set_xticklabels(x_label)
    plt.ylabel('Normalized RMS', size = 14)
    plt.xlabel('Window length', size = 14)
    plt.plot(wl,values4[::,p], color=viridis_data[(p*25)-25,:])
    plt.hlines(1,1,max(wl), color = 'b', linestyle = '--')
plt.savefig('Behavior with given wl and different SN ratio3.png',dpi=300)


I am working on the file "MedianFilter/Python/06. Behavior of the median filter with white noise/Behavior of the median filter with a noised sine wave.ipynb".
The first task is:
1. Create all needed arrays and data.
Can you write Python code for it?

# Sine wave, 16 wave numbers, 16*128 samples.
x = np.linspace(0, 2, 16*128)
data = np.sin(16*np.pi*x)
# Different noises with different standard deviations (spread or "width")
# will be saved in, so we can generate different signal to noise ratios
diff_noise = np.zeros((140,len(data)))
# Noised sine waves.
noised_sines = np.zeros((140,len(data)))
# Median filtered wave.
medfilter = np.zeros((140,len(data)))
# Filtered sine waves (noised_sines - medfilter)
filtered_sines = np.zeros((140,len(data)))
# Behavior of the median filter. Save the max values of the filtered waves in it.
behav = np.zeros(140)
# Lists with used window lengths and Signal to noise ratios
wl = [17,33,65,97, 129, 161, 193, 225, 257, 289, 321, 353, 385, 417, 449]
sn = [1, 1.5, 2, 3, 4, 5, 6, 7, 8, 9, 10]


I am working on the file "MedianFilter/Python/06. Behavior of the median filter with white noise/Behavior of the median filter with a noised sine wave.ipynb".
The first task is:
Figure 1. Behavior of the median filter with given window length and different S/N ratio.
Can you write Python code for it?

# Calculate and save all values.
# Because the for loop doesn't count from 1 to 10 for example,
# we need a counter to iterate through the array.
# The counter is assigne to -1, so we can iterate from 0 to len(values)
count = -1
count2 = -1
values = np.zeros((len(sn), len(wl)))
for w in  wl[:11]:
    count = count + 1
    for x in sn:
        count2 = count2 + 1
        for i in range (len(diff_noise)):
            # Create different noises, with x we change the signal to noise
            # ratio from 10 to 1.
            diff_noise[i, :] = np.random.normal(0, 0.706341266/np.sqrt(x), len(data))
            # Add noise to each sine wave, to create a realisitc signal.
            noised_sines[i, :] = data + diff_noise[i, :]
            # Filter the all noised sine waves.
            medfilter[i, :] = medfilt(noised_sines[i, :], w)
            # Subtract the filtered wave from the noised sine waves.
            filtered_sines[i, :] = noised_sines[i, :] - medfilter[i, :]
            # Calculate the root mean square (RMS) of each sine wave
            behav[i] = np.sqrt(np.mean(np.square(filtered_sines[i, :])))
        # Calculate the mean of the bahvior, so we can see how
        # the signal to noise ratio effects the median filter
        # with different window lengths.
        mean = np.mean(behav)
        # Save the result in the 'values' array
        values[count2:count2+1:,count] = mean
    # Set coun2 back to -1, so we can iterate again from 0 to len(values).
    # Otherwise the counter would get higher and is out of range.
    count2 = - 1
# Save the array, because the calculation take some time.
# Load the array with "values = np.loadtxt('values.txt')".
np.savetxt("values.txt", values)

plt.plot(noised_sines[1,:])

values = np.loadtxt("values.txt")

plt.plot(values)

viridis_data = np.loadtxt('viridis_data.txt')
plasma_data = np.loadtxt('plasma_data.txt')
# viris_data and plasma_data taken from
# https://github.com/BIDS/colormap/blob/master/colormaps.py

fig = plt.figure(figsize=(20, 7))
for p in...


I am working on the file "MedianFilter/Python/06. Behavior of the median filter with white noise/Behavior of the median filter with a noised sine wave.ipynb".
The first task is:
Figure 1.1 Behavior of the median filter with given window length and different S/N ratio.
Can you write Python code for it?


I am working on the file "MedianFilter/Python/06. Behavior of the median filter with white noise/Behavior of the median filter with a noised sine wave.ipynb".
The first task is:
Figure 2.1: Behavior of the median filter with given window length and different S/N ratio
Can you write Python code for it?

# Alternative
values = np.zeros((len(wl), len(sn)))
count = -1
count2 = -1
for x in sn:
    count = count + 1
    for w in wl:
        count2 = count2 + 1
        for i in range (len(diff_noise)):
            diff_noise[i, :] = np.random.normal(0, 0.706341266/np.sqrt(x), len(data))
            noised_sines[i, :] = data + diff_noise[i, :]
            medfilter[i, :] = medfilt(noised_sines[i, :], w)
            filtered_sines[i, :] = data - medfilter[i, :]
            behav[i] = np.sqrt(np.mean(np.square(filtered_sines[i, :])))
        mean = np.mean(behav)
        values[count2:count2+1:,-count] = mean
    count2 = -1
np.savetxt("values2A.txt", values)

values2A = np.loadtxt("values2A.txt")

fig = plt.figure()
values4 = np.zeros((len(wl), len(sn)))
for i in range (11):
    # Normalize the RMS with the RMS of a normal sine wave
    values4[::,i] = values2A[::,i]/0.7069341
    ax = plt.subplot()
    plt.axis([0, 450, 0, 2])
    # Set xticks at each 64th point
    xticks = np.arange(0, max(wl) + 1, 64)
    ax.set_xticks(xticks)
    # x_label = pi at each 64th point
    x_label = [r"${%s\pi}$" % (v) for v in range(len(xticks))]
    ax.set_xticklabels(x_label)
    plt.ylabel('Normalized RMS', size = 14)
    plt.xlabel('Window length', size = 14)
    plt.plot(wl,values4[::,i], color=viridis_data[(i*25)-25,:])
    plt.hlines(1,1,max(wl), color = 'b', linestyle = '--')
plt.savefig('Behavior with given wl and different SN ratio2A.png',dpi=300)


I am working on the file "MedianFilter/Python/06. Behavior of the median filter with white noise/Behavior of the median filter with a noised sine wave.ipynb".
The first task is:
Figure 2: Behavior of the median filter with given window length and different S/N ratio
Can you write Python code for it?


I am working on the file "MedianFilter/Python/07. Real Data/Comparison between DTW and fast DTW.ipynb".
The first task is:
Comparison between fastDTW and normal DTW
Can you write Python code for it?

label = np.zeros(len(thresholds_norm[0,:]))
def fastDtw(thresholds_norm, template1, label):
    count = 0
    # Go through all detected windows and compare them with the template
    for k in range(len(thresholds_norm[0,:])):
        dist = fastdtw.fastdtw(template1,thresholds_norm[:,k])
        dist = dist[0]
        # If the distance between both templates is smaller then 1.6, it's a match
        if dist < 1.6:
            label[k] = 1
        else:
            pass
    return label

label1 = np.zeros(len(thresholds_norm[0,:]))
def normalDtw(thresholds_norm, template1, label1):
    count = 0
    for k in range(len(thresholds_norm[0,:])):
        temp = thresholds_norm[:,k]
        o = template1.reshape(-1,1)
        dist, cost, acc, path = dtw.dtw(o, temp, dist=lambda o, temp: np.linalg.norm(o - temp, ord=1))

        if dist < 0.03:
            label1[k] = 1
        else:
            pass

import time
start_time = time.time()
fastDtw(thresholds_norm, template1, label)
print("--- %s seconds ---" % (time.time() - start_time))

import time
start_time = time.time()
normalDtw(thresholds_norm, template1, label1)
print("--- %s seconds ---" % (time.time() - start_time))


I am working on the file "MedianFilter/Python/07. Real Data/Comparison between DTW and fast DTW.ipynb".
The first task is:
Fast DTW is about 3 times faster then the normal DTW. Using an interpolation with 10 times more samples would make the algorithm 16 times slower then normal.
Can you write Python code for it?

for i in range(len(label1)):
    if label1[i] == 1:
        plt.plot(thresholds_array[:,i], color = 'black')

for i in range(len(label)):
    if label[i] == 1:
        plt.plot(thresholds_array[:,i], color = 'black')


I am working on the file "MedianFilter/Python/07. Real Data/Test with Threshold.ipynb".
The first task is:
Calculate the threshold for each filtered data
Can you write Python code for it?

threshold1 = 4*np.median(abs(new_data1)/0.6745)
threshold2 = 4*np.median(abs(new_data2)/0.6745)
threshold3 = 4*np.median(abs(new_data3)/0.6745)
threshold4 = 4*np.median(abs(new_data4)/0.6745)
threshold5 = 4*np.median(abs(new_data5)/0.6745)
threshold6 = 4*np.median(abs(csc)/0.6745)


I am working on the file "MedianFilter/Python/07. Real Data/Test with Threshold.ipynb".
The first task is:
Filter data with different window lenghts.
Can you write Python code for it?

filtered1 = medfilt(csc,25)
new_data1 = csc-filtered1

filtered2 = medfilt(csc,35)
new_data2 = csc-filtered2

filtered3 = medfilt(csc,45)
new_data3 = csc-filtered3

filtered4 = medfilt(csc,55)
new_data4 = csc-filtered4

filtered5 = medfilt(csc,95)
new_data5 = csc-filtered5


I am working on the file "MedianFilter/Python/07. Real Data/Test with Threshold.ipynb".
The first task is:
Plot Each data and the threshold
Can you write Python code for it?

plt.figure(figsize=(30,7))
plt.axis([0, 63000, -0.00003, 0.00016])
plt.plot(new_data1, color = 'r')
plt.hlines(threshold1, 0, len(new_data1), color = 'black')
plt.savefig('Threshold.png', dpi = 400)

plt.figure(figsize=(30,7))
plt.axis([0, 63000, -0.00003, 0.00016])
plt.plot(new_data2, color = 'r')
plt.hlines(threshold2, 0, len(new_data1), color = 'black')
plt.savefig('ThresholdWL35.png', dpi = 400)

plt.figure(figsize=(30,7))
plt.axis([0, 63000, -0.00003, 0.00016])
plt.plot(new_data3, color = 'r')
plt.hlines(threshold3, 0, len(new_data1), color = 'black')
plt.savefig('ThresholdWL45.png', dpi = 400)

plt.figure(figsize=(30,7))
plt.axis([0, 63000, -0.00003, 0.00016])
plt.plot(new_data4, color = 'r')
plt.hlines(threshold4, 0, len(new_data1), color = 'black')
plt.savefig('ThresholdWL55.png', dpi = 400)

plt.figure(figsize=(30,7))
plt.axis([0, 63000, -0.00003, 0.00016])
plt.plot(new_data5, color = 'r')
plt.hlines(threshold5, 0, len(new_data1), color = 'black')
plt.savefig('ThresholdWL95.png', dpi = 400)


I am working on the file "MedianFilter/Python/08. Tests/Test Threshold.ipynb".
The first task is:
Load, save and filter all the data
Can you write Python code for it?

# Session folder with all needed neuralynx files
sessionfolder = 'C:\\Users\\Dominik\\Documents\\GitRep\\kt-2015-DSPHandsOn\\MedianFilter\\Python\\08. Tests'

NIO = NeuralynxIO(sessiondir = sessionfolder, cachedir = sessionfolder)
block = NIO.read_block()
seg = block.segments[0]

analogsignals = {}
# Save all recorded datas in a analogsignals dictionary.
for i in range(len(seg.analogsignalarrays)):
    analogsignals["analogsignal{0}".format(i)] = seg.analogsignalarrays[i]

csc = {}
count = -1
# Extract the magnitude of each data.
for i in analogsignals:
    csc["csc{0}".format(i[-1])] = analogsignals[i].magnitude

# Check if the Signal is already inverted, if not, invert it
for i in analogsignals:
    temp = analogsignals[i]
    if temp.annotations['InputInverted'] == 'False':
        csc["csc{0}".format(i[-1])] = csc["csc{0}".format(i[-1])] * (-1)

filtered = {}
# Filter each data with a median filter with window length 25(seems to be the best)
for i in csc:
    temp = csc[i]
    filtered["filtered{0}".format(i[-1])] = medfilt(temp, 25)

new_data = {}
for i in filtered:
    temp = filtered[i]
    new_data["new_data{0}".format(i[-1])] = csc["csc{0}".format(i[-1])] - temp

threshold = {}
# Calculate the Threshold
for i in new_data:
    temp = new_data[i]
    threshold["threshold{0}".format(i[-1])] = 5*np.median(abs(temp)/0.6745)


I am working on the file "MedianFilter/Python/08. Tests/Test Threshold.ipynb".
The first task is:
Plot data and the thresholds
Can you write Python code for it?

plt.figure(figsize=(30,7))
plt.plot(new_data["new_data0"])
plt.hlines(threshold["threshold0"], 0, len(new_data["new_data0"]), color = 'red')
plt.figure(figsize=(30,7))
plt.plot(new_data["new_data1"])
plt.hlines(threshold["threshold1"], 0, len(new_data["new_data0"]), color = 'red')
plt.figure(figsize=(30,7))
plt.plot(new_data["new_data2"])
plt.hlines(threshold["threshold2"], 0, len(new_data["new_data0"]), color = 'red')
plt.figure(figsize=(30,7))
plt.plot(new_data["new_data3"])
plt.hlines(threshold["threshold3"], 0, len(new_data["new_data0"]), color = 'red')

def thresHold(new_data, threshold):
    count  = -1
    count2 = 0
    timer = 0
    positionlist = []
    # Dictionary with all thresholded shapes
    thresholds = {}

    for v in new_data:
        temp = new_data[v]

        for element in temp:
            # Increment the counter (counter = position in the array)
            count += 1
            # Check if the threshold is reached, and check if the window is inside the array
            if element >= threshold["threshold{0}".format(v[-1])] and (count - 10) >= 0 and (count + 14) <= len(temp):
                # Save a window with 24 samples
                tempdata = [w for w in temp[count -10 : count + 14]]
                # Save the position of the window in the original array
                tempsavedcounter = np.linspace(count-10, count + 14, 25)
                # Calculate the maximum value of the window
                maximum = max(tempdata)
                # Get the position of the maximum value inside the window
                maxposition = [p for p, q in enumerate(tempdata) if q == maximum]
                # Store the position (Note that the position here is the position in the original data)
                position = tempsavedcounter[maxposition[0]]
                tempdata = [w for w in temp[position -10 : position + 14]]

                # Search in the other recorded data at the same position for spikes, check if the maximum
                # in any other...


I am working on the file "MedianMeanTimeit.ipynb".
The first task is:
Build a little dictionary with the median and mean setup and statement.
Can you write Python code for it?

timeTestSetup = {'median': timeit.Timer(setup=setup,stmt=median_statement),
'mean': timeit.Timer(setup=setup, stmt=mean_statement)}


I am working on the file "MedianMeanTimeit.ipynb".
The first task is:
Run them both and report the results.
Can you write Python code for it?

for (k,v) in timeTestSetup.items():
    print("Method:\t{}: Time:\t{}".format(k,v.timeit()))


I am working on the file "MedianMeanTimeit.ipynb".
The first task is:
calculate the median of the column 'value' time how long it takes.
Can you write Python code for it?


I am working on the file "MedianMeanTimeit.ipynb".
The first task is:
run timeit with a timer containing the setup and the median statement.  The median statement will be called 100000 times.
Can you write Python code for it?

help('timeit')

timeit.Timer(setup=setup, stmt=median_statement).timeit(number=100000)


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Another view on the distribution of the prediction scores of our classifier
Divide the test set into 2 sets - True (blue) / False(red) - and plot them in a scatter plot
Can you write Python code for it?

pospts = [v[1] for i,v in enumerate(zip(Y_test, diff_log_proba)) if v[0] == 1]
negpts = [v[1] for i,v in enumerate(zip(Y_test, diff_log_proba)) if v[0] == 0]

import random

plt.scatter(
    pospts,
    np.random.uniform(0.9, 1.1, len(pospts)),
    color='blue')

plt.scatter(
    negpts,
    np.random.uniform(0.9, 1.1, len(negpts)),
    color='red')

plt.xlim(-500, 450)

plt.ylim(0.8, 1.2)

plt.axvline(0, color='r')

values = np.array(diff_log_proba[iwrong_predictions])

plt.axvspan(
    np.mean(values)-2*np.std(values),
    np.mean(values)+2*np.std(values),
    facecolor='b', alpha=0.1)

plt.axvline(np.mean(values), linewidth=1);


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Assessing Classifier Performance
Confusion matrix and derivative estimators
Can you write Python code for it?

from ipy_table import *

confusion_matrix_binary = [
    ['', 'Predicted 0', 'Predicted 1'],
    ['Actual 0', 'True Negative', 'False Positive'],
    ['Actual 1', 'False Negative', 'True Positive']
]

make_table(confusion_matrix_binary)
apply_theme('basic_both')


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
At this point of the workflow, there are many feature selection and engineering techniques we could apply. Some generic, some based on natural language processing and some unique to the medical domain. However, to keep it simple for now, let's move on to the classifier:
Naive Bayes classifier

Naive conditional independence assumption: The counts of individual features (or n-grams) are independent
Bayes' rule is used to calculate the conditional probabilities for each class/feature pair:
    $P(Y|W)=\frac{P(W|Y) \cdot P(Y)}{P(W)}$

$\implies P(Y|w_{1}, w_{2}, ..., w_{N})=\frac{1}{P(W)} \cdot P(Y) \cdot \prod_{k=1}^{N}  P(w_{k}|Y)$
Since the probability of all the features is not dependent on the probability of Y and all we care about is finding the Y that is most likely, we can drop $\frac{1}{P(W)}$ and stay with:
$P(Y|w_{1}, w_{2}, ..., w_{N})=P(Y) \cdot \prod_{k=1}^{N}  P(w_{k}|Y)$
We estimate the probabilities from the training set using multinomial distribution with a smoothing factor alpha:
$P(w_{i}) = \frac{count_{i}+alpha}{overallcount_{i}+alpha \cdot N}$
<img  width="575" src="words_proba.png"/>
Fit (aka train) a classifier with some ad-hoc parameters on the test data
Can you write Python code for it?

from sklearn.naive_bayes import MultinomialNB

clf_nb = MultinomialNB(alpha=0.1)

clf_nb.fit(wcounts, Y_train)


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Comparing actual values to the predictions of the baseline classifier
Can you write Python code for it?

dictY = {'Actual' : Ytrue, 'Predicted': Ypred}

dfY = pd.DataFrame.from_dict(dictY)

dfY.sample(20)


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Creating binary classification dataset
 As we see, each document is assigned one class or more.  In this exercise I would like to implement a simple binary classification workflow, so I define 2 binary classes:
* Positive / True - Documents that belong to class C14 - Cardiovascular Diseases
* Negative / False - Documents that do not belong to C14
The following code converts that data to a binary classification dataset:
Can you write Python code for it?

import re

def ConvertCategoryColToBinVal(df, poscat):
    df['CATEGORY'] = df['CATEGORY'].apply(lambda x: bool(re.search(poscat, x, re.IGNORECASE) and True))

ConvertCategoryColToBinVal(df_train, 'C14')

ConvertCategoryColToBinVal(df_test, 'C14')


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Data distribution
Can you write Python code for it?

plt.axis('equal')
plt.pie(
    df_train.CATEGORY.value_counts().tolist(),
    labels=['False', 'True'],
    autopct='%1.1f%%',
    colors=("#E13F29", "#D69A80"));


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Data exploration and preparation
Read the data from the sqlite databases into Pandas dataframes
Can you write Python code for it?

con = sqlite.connect('training.sqlite')

df_train = pd.read_sql_query("SELECT * from Documents", con)

con.close()

con = sqlite.connect('test.sqlite')

df_test = pd.read_sql_query("SELECT * from Documents", con)

con.close()


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Distribution of the predictions on the test set, as function of the difference between the True/False log probabilities
Can you write Python code for it?

# Plot  histogram.
plt.hist(diff_log_proba, range=[-500, 500], bins=30, normed=True, alpha=0.5)
plt.axvline(0, color='r')


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Even though the accuracy is OK, the recall is misearable. Let's build a real classifier
Building a ML classifier
Vectorizing textual data using Bag-of-Words (or Bag-of-Ngrams)
Our vector space is a dictionary of all the N-grams in our set of documents.
Defining tokenizer to extract better features from the text
Can you write Python code for it?

from nltk.stem.porter import *
from nltk.tokenize import word_tokenize
import string

stemmer = PorterStemmer()
def stem_tokens(tokens, stemmer):
    stemmed = []
    for item in tokens:
        stemmed.append(stemmer.stem(item))
    return stemmed

def tokenize(text):
    text = "".join([ch for ch in text if ch not in string.punctuation])
    tokens = word_tokenize(text)
    tokens = [item for item in tokens if item.isalpha()]
    stems = stem_tokens(tokens, stemmer)
    return stems


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Few words about IPython Notebook
Some of the main features of the IPython Notebook app include:

In-browser editing for code, with automatic syntax highlighting, indentation, and tab completion.
The ability to execute code from the browser, with the results of computations attached to the code which generated them.
Displaying the result of computation using rich media representations, such as HTML, LaTeX, PNG, SVG, etc, up to publication-quality figures content.
In-browser editing for rich text using the Markdown markup language, which can provide commentary for the code.
The ability to easily include mathematical notation within markdown cells using LaTeX.

Personal advantages for my projects:

Great support for  Interpretable Data Science  - This greatly contributes to the process of harnessing the medical expertise of the users to train the algorithms better.

Architecture of IPython notebook
<img src="images/ipython_architecture.png">
The data
The Ohsumed test collection (available at ftp://medir.ohsu.edu/pub/ohsumed) is a subset of the MEDLINE database, which is a bibliographic database of important, peer-reviewed medical literature maintained by the National Library of Medicine. The initial subset I consider in the project is the collection consisting of the first 20,000 documents from the 50,216 medical abstracts of the year 1991. The classification scheme consists of the 23 Medical Subject Headings (MeSH) categories of cardiovascular diseases group:
| Category      | Description   |
| ------------- | ------------- |
| C01 | Bacterial Infections and Mycoses |
| C02 | Virus Diseases |
| C03 | Parasitic Diseases |
| C04 | Neoplasms |
| C05 | Musculoskeletal Diseases |
| C06 | Digestive System Diseases |
| C07 | Stomatognathic Diseases |
| C08 | Respiratory Tract Diseases |
| C09 | Otorhinolaryngologic Diseases |
| C10 | Nervous System Diseases |
| C11 | Eye Diseases |
| C12 | Urologic and Male Genital Diseases |
| C13 | Female Genital Diseases and Pregnancy Complications |
| C14 | Cardiovascular...
Can you write Python code for it?

import sqlite3 as sqlite

def get_all_tables(c):
    """
    Helper function - Gets a list of all the tables in the database.
    """
    all_tables = []
    c.execute('SELECT name FROM SQLITE_MASTER WHERE type = "table"')
    for tbl in c:
        all_tables.append(tbl[0])
    return all_tables

def drop_tables(c, tables):
    """
    Helper function - Checks that the specified tables exist, and for those that do, drop them.
    """
    all_tables = get_all_tables(c)
    for t in tables:
        if t in all_tables:
            c.execute('DROP TABLE %s' % t)

def create_documents_table(c):
    """
    Helper function - This function uses SQL to create the Documents table
    """
    drop_tables(c, [ 'Documents' ])
    c.execute("""CREATE TABLE Documents (
            DOCID TEXT PRIMARY KEY,
            NOTE_TEXT TEXT,
            CATEGORY TEXT
        )""")

def add_document(c, docid, text, category):
    """
    Helper function - Adds one document to sql Documents database
    """
    c.execute('insert or replace into Documents values ( ?, ?, ? )', (docid, text, category))

def ohsumed2sqlite(src_root_dir, dest_sqlite):
    start_time = time.time()
    print 'Converting ohsumed directory structure {0} to sqlite database'.format(src_root_dir)

    conn_out = sqlite.connect(dest_sqlite)
    c_out = conn_out.cursor()

    fout = open(dest_sqlite, 'w')
    fout.close()

    create_documents_table(c_out)
    # Process the ohsumed directory
    dict = {}
    for root, dirs, files in os.walk(src_root_dir):
        for f in files:
            category = os.path.basename(root) # directory name is the category
            with open (os.path.join(root, f), "r") as cur_file:
                data=cur_file.read()
                if f in dict:
                    category = dict[f] + ',' + category
                dict[f] = category
                add_document(c_out, f, data, category)
    conn_out.commit()
    c_out.close()
    print("--- ohsumed2sqlite %s seconds ---" % (time.time() - start_time))

# Convert the...


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
How many records do we have in the test / train datasets ?
Can you write Python code for it?

print 'Training data has: ', len(df_train.index), ' documents'

print 'Test data has: ', len(df_test.index), ' documents'


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Let's take another look at the data:
Can you write Python code for it?


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Most Predictive Features
A cool feature of the Naive Bayes classifier is that it can list for us most relevant features for each class. These are the features that are most relevant to the positive documents. Some of them are trivial English words, that will also appear in the list of features relevant to the Negative documents. However - we also see some domain specific features, such as arteri and  coronari :
Can you write Python code for it?

pf = [(clf_nb.feature_log_prob_[1, i], feats[i]) for i in range(len(feats))]
pf.sort(reverse=True)
for p in pf[:25]:
    print 'Positive word %.2f: %s' % (p[0], p[1])


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Now we build a full pipeline and use the GridSearchCV sklearn utility to fit the best paramaters for the pipeline. Normally I use longer lists of parameters, but that takes hours to run.
NOTE - we will use GridSearchCV default 3-fold cross-validation. Other cross-validation schemes can be defined if needed.
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.grid_search import GridSearchCV
from sklearn.pipeline import Pipeline

pipeline = Pipeline([
    ('vect', CountVectorizer(tokenizer=tokenize)),
    ('clf', MultinomialNB())
])

# Define some possible parameters for feature extraction and for the classifier
parameters = {
    'vect__max_features': (20000, 30000),
    'vect__ngram_range': ((1, 1), (1, 2)),  # unigrams or bigrams
    'clf__alpha': (0.05, 0.1, 0.2)
}

# find the best parameters for both the feature extraction and the
# classifier, based on the scorer function we defined
grid_search = GridSearchCV(pipeline, parameters, verbose=1, scoring=recall_scorer)

grid_search.fit(df_train['NOTE_TEXT'].tolist(), df_train.CATEGORY.tolist())

print("Best parameters set found on development set:")
print
print(grid_search.best_params_)


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Optimizing the classification pipeline
Scorer function
When optimizing parameters for classification pipeline we can write or define our own scoring function. The optimizing routine GridSearchCV which runs a brute force parameter search, will select the parameters that got the highest score.
What do we want to optimize exactly?
When classifing medical documents, we sometimes want to maximize sensitivity (recall) while keeping specificity and accuracy in check. In other words - the sensitivity is our target optimization parameter. To maximize sensitivity, I'll define a scorer function that returns the recall score.
Can you write Python code for it?

from sklearn.metrics import recall_score, make_scorer

# Define scorer function that returns the recall score
recall_scorer = make_scorer(recall_score)


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Output a random sample of 20 records
Can you write Python code for it?

pd.options.display.max_colwidth = 200

df_train.sample(20)


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Probabilities of a particular classification
Bayesian models like the Naive Bayes classifier have the nice property that they compute probabilities of a particular classification -- the predict_proba and predict_log_proba methods of MultinomialNB compute these probabilities.
Can you write Python code for it?

proba = clf_nb.predict_proba(counts_test)
log_proba = clf_nb.predict_log_proba(counts_test)

proba[:10]


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Some code to prettify the printout of the confusion matrix in the notebook:
Can you write Python code for it?

def render_confusion_matrix(ytrue, ypred):
    return pd.crosstab(pd.Series(ytrue), pd.Series(ypred), rownames=['Actual'], colnames=['Predicted'], margins=True)

render_confusion_matrix(Ytrue, Ypred)

from sklearn.metrics import classification_report, accuracy_score

print
print 'Accuracy: ', accuracy_score(Ytrue, Ypred)
print
print classification_report(Ytrue, Ypred)


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Test the classifier with the test data
Can you write Python code for it?

# Read the data from pandas dataframe to an array
X_test = df_test['NOTE_TEXT'].tolist()
Y_test = df_test.CATEGORY.tolist()

# Convert the text to arrays of numbers
counts_test = vectorizer.transform(X_test)

counts_test


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
The accuracy of a dumb classifier that classifies all the documents as False is >80%
Baseline classifier
We define a trivial baseline classifier that simply look for the string 'Cardio' in the text
Can you write Python code for it?

import re

def baseline_cpr_classifier(txt):
    if re.search('Cardio', txt, re.IGNORECASE):
        return True
    else:
        return False


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Try the optimized classifier on the test set and output the performance parameters
Can you write Python code for it?

# Use the optimal classifier to make predictions on the test set

opt_predictions = grid_search.predict(df_test['NOTE_TEXT'].tolist())

render_confusion_matrix(df_test.CATEGORY.tolist(), opt_predictions)

print
print 'Accuracy: ', accuracy_score(df_test.CATEGORY.tolist(), opt_predictions)
print
print classification_report(df_test.CATEGORY.tolist(), opt_predictions)


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
Vectorize the test data
Can you write Python code for it?

X_train = df_train['NOTE_TEXT'].tolist()
Y_train = df_train.CATEGORY.tolist()

vectorizer = CountVectorizer(tokenizer=tokenize, ngram_range=(1,2))

wcounts = vectorizer.fit_transform(X_train)


I am working on the file "Medical_Text_Classification.ipynb".
The first task is:
We have 7643 documents in our test set, each has 355860 features - exactly the same features we set while processing the training set of course.
Can you write Python code for it?

# Predict the values of the test set
predictions = clf_nb.predict(counts_test)

# Look at the first 20 predictions
predictions[:20]


I am working on the file "Meeting 5/APS Captcha.ipynb".
The first task is:
Preprocessing
Can you write Python code for it?

import os
from skimage import io
from skimage.color import rgb2gray
from skimage import transform
from math import ceil

IMGSIZE = (100, 100)

def load_images(folder, scalefactor=(2, 2), labeldict=None):
    images = []
    labels = []
    files = os.listdir(folder)

    for file in (fname for fname in files if fname.endswith('.png')):

        img = io.imread(folder + file).astype(float)
        img = rgb2gray(img)
        # Crop since some of the real world pictures are other shape
        img = img[:IMGSIZE[0], :IMGSIZE[1]]
        # Possibly downscale to speed up processing
        img = transform.downscale_local_mean(img, scalefactor)
        # normalize image range
        img -= np.min(img)
        img /= np.max(img)
        images.append(img)

        if labeldict is not None:
            # lookup label for real world data in dict generated from labels.txt
            key, _ = os.path.splitext(file)
            labels.append(labeldict[key])
        else:
            # infere label from filename
            if file.find("einstein") > -1 or file.find("curie") > -1:
                labels.append(1)
            else:
                labels.append(0)

    return np.asarray(images)[:, None], np.asarray(labels)

x_train, y_train = load_images('data/aps/train/')
# Artifically pad Einstein's and Curie't to have balanced training set
# ok, since we use data augmentation later anyway
sel = y_train == 1
repeats = len(sel) // sum(sel) - 1
x_train = np.concatenate((x_train[~sel], np.repeat(x_train[sel], repeats, axis=0)),
                         axis=0)
y_train = np.concatenate((y_train[~sel], np.repeat(y_train[sel], repeats, axis=0)),
                         axis=0)

x_test, y_test = load_images('data/aps/test/')

rw_labels = {str(key): 0 if label == 0 else 1
             for key, label in np.loadtxt('data/aps/real_world/labels.txt', dtype=int)}
x_rw, y_rw = load_images('data/aps/real_world/', labeldict=rw_labels)

from mpl_toolkits.axes_grid import ImageGrid
from math...


I am working on the file "Meeting 5/APS Captcha.ipynb".
The first task is:
Training LeNet
First, we will train a simple CNN with a single hidden fully connected layer as a classifier.
Can you write Python code for it?

from keras.layers import Conv2D, Dense, Flatten, MaxPooling2D
from keras.models import Sequential
from keras.backend import image_data_format

def generate(figsize, nr_classes, cunits=[20, 50], fcunits=[500]):
    model = Sequential()
    cunits = list(cunits)
    input_shape = figsize + (1,) if image_data_format == 'channels_last' \
        else (1,) + figsize

    model.add(Conv2D(cunits[0], (5, 5), padding='same',
                     activation='relu', input_shape=input_shape))
    model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Convolutional layers
    for nr_units in cunits[1:]:
        model.add(Conv2D(nr_units, (5, 5), padding='same',
                         activation='relu'))
        model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

    # Fully connected layers
    model.add(Flatten())
    for nr_units in fcunits:
        model.add(Dense(nr_units, activation='relu'))

    # Output layer
    activation = 'softmax' if nr_classes > 1 else 'sigmoid'
    model.add(Dense(nr_classes, activation=activation))

    return model

from keras.optimizers import Adam
from keras.models import load_model

try:
    model = load_model('aps_lenet.h5')
    print("Model succesfully loaded...")
except OSError:
    print("Saved model not found, traing...")
    model = generate(figsize=x_train.shape[-2:], nr_classes=1,
                     cunits=[24, 48], fcunits=[100])
    optimizer = Adam()
    model.compile(loss='binary_crossentropy', optimizer=optimizer,
                  metrics=['accuracy'])

    model.fit_generator(imggen.flow(x_train, y_train, batch_size=len(x_train)),
                        validation_data=imggen.flow(x_test, y_test),
                        steps_per_epoch=100,  epochs=5,
                        verbose=1, validation_steps=256)
    model.save('aps_lenet.h5')

from sklearn.metrics import confusion_matrix

def plot_cm(cm, classes, normalize=False,
            title='Confusion matrix', cmap=pl.cm.viridis):
    """
    This function prints and plots the confusion...


I am working on the file "Meeting 5/aps_capcha.ipynb".
The first task is:
Image before/after compression
Can you write Python code for it?

i=11
plt.subplot(1,2,1)
plt.imshow(training_set[i],cmap='gray')
plt.subplot(1,2,2)
plt.imshow(X_train[i][0],cmap='gray')


I am working on the file "Meeting 5/aps_capcha.ipynb".
The first task is:
Lenet neural network
Can you write Python code for it?

# import the necessary packages
from keras.models import Sequential
from keras.layers.convolutional import Convolution2D
from keras.layers.convolutional import MaxPooling2D
from keras.layers.core import Activation
from keras.layers.core import Flatten
from keras.layers.core import Dense
from keras.optimizers import SGD

# this code comes from http://www.pyimagesearch.com/2016/08/01/lenet-convolutional-neural-network-in-python/

class LeNet:
	@staticmethod
	def build(width, height, depth, classes, weightsPath=None):
		# initialize the model
		model = Sequential()

		# first set of CONV => RELU => POOL
		model.add(Convolution2D(20, 5, 5, border_mode="same",input_shape=(depth, height, width)))
		model.add(Activation("relu"))
		model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

		# second set of CONV => RELU => POOL
		model.add(Convolution2D(50, 5, 5, border_mode="same"))
		model.add(Activation("relu"))
		model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))

		# set of FC => RELU layers
		model.add(Flatten())
		model.add(Dense(500))
		model.add(Activation("relu"))

		# softmax classifier
		model.add(Dense(classes))
		model.add(Activation("softmax"))

		# return the constructed network architecture
		return model


I am working on the file "Meeting 5/aps_capcha.ipynb".
The first task is:
We build the neural network and fit it on the training set
Can you write Python code for it?

model = LeNet.build(width=32, height=32, depth=1, classes=2)
opt = SGD(lr=0.01)#Sochastic gradient descent with learning rate 0.01
model.compile(loss="categorical_crossentropy", optimizer=opt,metrics=["accuracy"])
model.fit(X_train, Y_train, batch_size=10, nb_epoch=300,verbose=1)

y_pred = model.predict_classes(X_test)
print(y_pred)
print(test_labels)


I am working on the file "Meeting 5/aps_capcha.ipynb".
The first task is:
We now compare with the real world images (with the deshear method)
Can you write Python code for it?

real_world_set=[]
for i in np.arange(1,73):
    filename=path+'images/real_world/'+str(i)+'.png'
    real_world_set.append(im.deshear(filename))
fake_label=np.ones(len(real_world_set),dtype='int32')
X_real,Y_real=prep_datas(real_world_set,fake_label)
y_pred = model.predict_classes(X_real)


I am working on the file "Meeting 5/aps_capcha.ipynb".
The first task is:
with the labels of Peter
Can you write Python code for it?

f=open(path+'images/real_world/labels.txt',"r")
lines=f.readlines()
result=[]
for x in lines:
    result.append((x.split('	')[1]).replace('\n',''))
f.close()

result=np.array([int(x) for x in result])
result[result>1]=1
plt.plot(y_pred,'o')
plt.plot(2*result,'o')
plt.ylim(-0.5,2.5);


I am working on the file "Meeting 6/aps_with_classifiers.ipynb".
The first task is:
Now we load the real set of images and test it. This part of the program has been taken from Alexandre's program from last week. First we load the 'real world' images
Can you write Python code for it?

real_world_set=[]
for i in np.arange(1,73):
    filename=path+'images/real_world/'+str(i)+'.png'
    real_world_set.append(im.deshear(filename))
fake_label=np.ones(len(real_world_set),dtype='int32')
X_real,Y_real=prep_datas(real_world_set,fake_label)


I am working on the file "Meeting 6/aps_with_classifiers.ipynb".
The first task is:
Then we load the training and the test set:
Can you write Python code for it?

training_set, training_labels = im.load_images(path_train)
X_train, Y_train = prep_datas(training_set,training_labels)

test_set, test_labels = im.load_images(path_test)
X_test,Y_test=prep_datas(test_set,test_labels)


I am working on the file "Meeting 6/aps_with_classifiers.ipynb".
The first task is:
We define the classifiers Random Forest Classifier and Support Machine Classifier and we train them throught the fit function. Taking a linear kernel for SVC gives the best results for this classifier.
Can you write Python code for it?

classifierForest = RandomForestClassifier(n_estimators=1000)
classifierSVC=svm.SVC(kernel='linear')

classifierForest.fit(X_train, Y_train)
classifierSVC.fit(X_train,Y_train)


I am working on the file "Meeting 7/qutipHHL.ipynb".
The first task is:
$\left|finalstate\right\rangle$ is essentially a pure state (it should be if all the process was perfect), so now we just isolate that main part, that is our $\left|x\right\rangle$ (after a transformation if our original A was not Hermitian) in the basis that diagonalizes $A$.
Can you write Python code for it?

fsevls, fsevcs = finalstate.eigenstates()
x = math.sqrt(fsevls.max()) * fsevcs[fsevls.argmax()]
x


I am working on the file "Meeting 7/qutipHHL.ipynb".
The first task is:
STEP 2-1: Making true the assumption of perfect phase estimation
Can you write Python code for it?

# w = (ψfourier[:T] / b[0]).argmax()
# prj = qt.ket2dm(qt.basis(T, w))

# ψfourier = qt.tensor(qt.identity(b.shape[0]), prj) * ψfourier


I am working on the file "Meeting 7/qutipHHL.ipynb".
The first task is:
STEP 2: QUANTUM SIMULATION AND QUANTUM PHASE ESTIMATION
[HHL: page 2, column 2, bottom]
Can you write Python code for it?

T = prec
t0 = κ / ϵ    # It should be O(κ/ϵ), whatever that means
ψ0 = qt.Qobj([[math.sqrt(2 / T) * math.sin(π * (τ + 0.5) / T)]
                for τ in range(T)])

# Order is b, τ, and then ancilla
evo = qt.tensor(qt.identity(A.shape[0]), qt.ket2dm(qt.basis(T, 0)))

for τ in range(1, T):
    evo += qt.tensor((1j * A * τ * t0 / T).expm(), qt.ket2dm(qt.basis(T, τ)))

ψev = evo * qt.tensor(b, ψ0)

ftrans = qt.tensor(qt.identity(b.shape[0]), qft(T))

ψfourier = ftrans * ψev    # This is Eq. 3 in HHL


I am working on the file "Meeting 7/qutipHHL.ipynb".
The first task is:
STEP 3-1: Conditional Rotation of Ancilla
Can you write Python code for it?

total_state = qt.tensor(ψfourier, qt.basis(2, 0))   # Add ancilla for swapping

C = 1 / κ    # Constant, should be O(1/κ)

# Do conditional rotation only on τ and ancilla
rotation = qt.tensor(qt.ket2dm(qt.basis(T, 0)), qt.identity(2))

for τ in range(1, T):
    rotation += qt.tensor(qt.ket2dm(qt.basis(T, τ)), rot(τ, t0, C))

final_state = qt.tensor(qt.identity(b.shape[0]), rotation) * total_state


I am working on the file "Meeting 7/qutipHHL.ipynb".
The first task is:
STEP 3-2: Post-selection on $\left|1\right\rangle$ on the ancilla register
We perform the post-selection by projecting onto the desired ancilla state and later tracing out the ancilla and the $\left|\psi_0\right\rangle$ registers.
Can you write Python code for it?

projector = qt.tensor(qt.identity(b.shape[0]), qt.identity(T), qt.ket2dm(qt.basis(2, 1)))
postsel = projector * final_state
prob1 = qt.expect(projector, final_state)

# Trace out ancilla and τ registers, leaving only the b register
finalstate = qt.ket2dm(postsel).ptrace([0]) / prob1
finalstate.eigenenergies()


I am working on the file "Meetup Stats.ipynb".
The first task is:
Part 4: Membership join history
Note
If getting members from an endpoint returns 0, despite the member count in the group data being a positive number, then the group is set to private & accessible only to members (you can join that group to be able to have access that data, but I did not; I already have too much email).
Note
There's a "pseudo" race condition where the group data member # may be one number, but you actually receive a different number (+/- ~3), it's (probably) due to people leaving or joining the group between the group API call and the members API call.
API endpoint docs:

Members
Can you write Python code for it?

MEETUP_MEMBER_URL = "https://api.meetup.com/2/members"
PARAMS = {
    "signed": True,
    "key": MEETUP_API_KEY,
}

def get_members(group):
    PARAMS["group_id"] = group.get("id")
    members_count = group.get("members")
    print(u"MEMBERS: Getting {0} members for group {1}".format(members_count, group.get("name")))
    pages = members_count / 200
    remainder = members_count % 200
    if remainder > 0:
        pages += 1

    members = []
    for i in xrange(pages):
        print("MEMBERS: Iteration {0} out of {1}".format(i+1, pages+1))
        PARAMS["offset"] = i
        resp = requests.get(MEETUP_MEMBER_URL, PARAMS)
        if resp.ok:
            results = resp.json().get("results")
            members.extend(results)
        time.sleep(1)
    print("MEMBERS: Got {0} members".format(len(members)))
    return members

def get_members_collection(pylady, groups):
    pylady_members = get_members(pylady)
    pug_members = defaultdict(list)
    for g in groups:
        pg_mbrs = get_members(g)
        pug_members[g.get("name")].append(pg_mbrs)
    return pylady_members, pug_members

# NOTE: this takes *FOREVER*.
start = time.time()
for i, item in enumerate(collect.items()):
    print("COLLECTING: {0} out of {1}".format(i+1, len(collect)+1))
    pylady = [p for p in pyladies if p.get("name") == item[0]][0]
    pylady_members, pug_members = get_members_collection(pylady, item[1])

    print("COLLECTING: Saving all the data!")
    pylady_name = pylady.get("name")
    outdir = pylady_dir(pylady_name)
    outdir = os.path.join(data_dir, outdir)
    outfile = os.path.join(outdir, "pyladies_members.json")
    save_output(pylady_members, outfile)
    outfile = os.path.join(outdir, "pug_members.json")
    save_output(pug_members, outfile)
end = time.time()
delta_s = end - start
delta_m = delta_s / 60
print("**DONE**")
print("Completed in {:.0f} minutes".format(delta_m))


I am working on the file "MeetupPrez.ipynb".
The first task is:

First look at the competition data

            Training data set

            train.csv

            - 3.5 years of sales data

            - number of customers

            - promotion activities

            - store opening hours

            - state / school holiday details

            Store details

            stores.csv

             - store type code

             - assortment type code

             - long term promotions

             - nearest competition

             - locality unknown

            Test data set

            test.csv

            - same as train.csv

            - no sales and customer data

Load the training, store and test data using pandas
Can you write Python code for it?

types = {
    'CompetitionOpenSinceYear': np.dtype(int),
    'CompetitionOpenSinceMonth': np.dtype(int),
    'StateHoliday': np.dtype(str),
    'Promo2SinceWeek': np.dtype(int),
    'SchoolHoliday': np.dtype(int),
    'PromoInterval': np.dtype(str)
}
train = pd.read_csv("../train.csv", parse_dates=[2], dtype=types)
test  = pd.read_csv("../test.csv",  parse_dates=[3], dtype=types)
store = pd.read_csv("../store.csv")

train['Day']   = train.Date.dt.day
train['Month'] = train.Date.dt.month
train['Year']  = train.Date.dt.year

train.sort(['Store','Year','Month','Day'], inplace = True)

train.head(5)

test.head(5)