I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
I'm going to set bounds on the available parameters space, i.e. our prior volume, using the ranges taken from Table 4 of https://arxiv.org/pdf/1706.05459.pdf
Can you write Python code for it?

import scipy.optimize as op

# extract our initial guess at parameters
# from the celerite kernel and put it in a
# vector:
p0 = gp.get_parameter_vector()

# set prior ranges
# Note that these are in *logarithmic* space
bnds = ((-10.,0.),(-5.,5.),(-5.,-1.5),(-3.,5.))

# run optimization:
results = op.minimize(nll, p0, method='L-BFGS-B', jac=grad_nll, bounds=bnds, args=(value, gp))

# print the value of the optimised parameters:
print np.exp(results.x)
print("Final log-likelihood: {0}".format(-results.fun))


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
Import some libraries:
Can you write Python code for it?


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
In the paper there are two suggested kernels for modelling the covariance of the Kepler data (Eqs. 55 & 56). In the paper the authors fit Eq 56 - here we are going to fit Eq. 56.
$$
k(\tau) = \frac{B}{1+C}\exp^{-\tau/L} \left[ \cos{\left( \frac{2\pi\tau}{P} \right)} + (1+C) \right]
$$
This is the same as the CustomTerm described in the celerite documentation here: http://celerite.readthedocs.io/en/stable/python/kernel/
There is one small difference though - the exponent is expressed differently. This doesn't mean we need to change anything... except for our prior bounds because we're going to apply those as logarithmic bounds so we will need to put a minus sign in front of them since $\log(1/x) = -\log(x)$.
Can you write Python code for it?

import autograd.numpy as np

class CustomTerm(terms.Term):
    parameter_names = ("log_a", "log_b", "log_c", "log_P")

    def get_real_coefficients(self, params):
        log_a, log_b, log_c, log_P = params
        b = np.exp(log_b)
        return (
            np.exp(log_a) * (1.0 + b) / (2.0 + b), np.exp(log_c),
        )

    def get_complex_coefficients(self, params):
        log_a, log_b, log_c, log_P = params
        b = np.exp(log_b)
        return (
            np.exp(log_a) / (2.0 + b), 0.0,
            np.exp(log_c), 2*np.pi*np.exp(-log_P),
        )


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
Make a plot like the one in Figure 7:
Can you write Python code for it?

pl.subplot(111)
pl.scatter(time,value,s=0.2)
pl.axis([0.,60.,-1.,1.])
pl.ylabel("Relative flux [ppt]")
pl.xlabel("Time [days]")
pl.show()


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
Now let's run the production MCMC:
Can you write Python code for it?

# take the highest likelihood point from the burn-in as a
# starting point and now begin your production run:
print("Running production")
p = p0[np.argmax(lnp)]
p0 = [p + 1e-5 * np.random.randn(ndim) for i in xrange(nwalkers)]
p0, lnp, _ = sampler.run_mcmc(p0, 5000)

print "Finished"

import acor

# calculate the convergence time of our
# MCMC chains:
samples = sampler.flatchain
s2 = np.ndarray.transpose(samples)
tau, mean, sigma = acor.acor(s2)
print "Convergence time from acor: ", tau

# get rid of the samples that were taken
# before convergence:
delta = int(20*tau)
samples = sampler.flatchain[delta:,:]

#samples[:, 2] = np.exp(samples[:, 2])
b_mcmc, c_mcmc, l_mcmc, p_mcmc = map(lambda v: (v[1], v[2]-v[1], v[1]-v[0]),
                             zip(*np.percentile(samples, [16, 50, 84],
                                                axis=0)))

# specify prediction points:
t = np.arange(np.min(time),np.max(time),0.1)

# extract the expectation values:
ep = np.array([b_mcmc[0], c_mcmc[0], l_mcmc[0], p_mcmc[0]])
print "Expectation Parameters:", ep

lnB = ep[0]
lnC = ep[1]
lnL = -1.*ep[2]
lnP = np.log(ep[3])

p = np.array([lnB,lnC,lnL,lnP])
gp.set_parameter_vector(p)

# calculate expectation and variance at each point:
mu, cov = gp.predict(value, t)

# final log(likelihood):
finl_logL = gp.log_likelihood(y_train)

print "Initial logL: ", init_logL
print "Final logL:", finl_logL

ax = pl.subplot(111)
pl.plot(t,mu)
ax.fill_between(t,mu-std,mu+std,facecolor='lightblue', lw=0, interpolate=True)
pl.scatter(time,value,s=2)
pl.axis([0.,60.,-1.,1.])
pl.ylabel("Relative flux [ppt]")
pl.xlabel("Time [days]")
pl.show()

import corner

# Find the maximum likelihood values:
ml = p0[np.argmax(lnp)]
print "Maximum likelihood parameters: ",ml

lnB = ml[0]
lnC = ml[1]
lnL = -1.*ml[2]
lnP = np.log(ml[3])

p = np.array([lnB,lnC,lnL,lnP])
gp.set_parameter_vector(p)
ml_logL = gp.log_likelihood(y_train)
print "ML logL:", ml_logL

# Plot it.
figure = corner.corner(samples, labels=[r"$lnB$", r"$lnC$", r"$lnL$",...


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
Read the Kepler data from the file:
Can you write Python code for it?

time=[];value=[]
while True:
    line = datafile.readline()
    if not line: break

    items=line.split()
    if (items[0][0]!='|'):
        time.append(float(items[1]))
        value.append(float(items[2]))

time=np.array(time)
value=np.array(value)

print "There are ",len(time)," data points"


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
Specify the datafile containing Kepler data for the object KIC 1430163:
Can you write Python code for it?

filename="KIC1430163.tbl"
datafile = open(filename,'r')


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
The paper says:
As with the earlier examples, we start by estimating the MAP parameters using L-BFGS-B
So let's do that. We'll use the scipy optimiser, which requires us to define a log(likelihood) function and a function for the gradient of the log(likelihood):
Can you write Python code for it?

def nll(p, y, gp):

    # Update the kernel parameters:
    gp.set_parameter_vector(p)

    #  Compute the loglikelihood:
    ll = gp.log_likelihood(y)

    # The scipy optimizer doesn’t play well with infinities:
    return -ll if np.isfinite(ll) else 1e25

def grad_nll(p, y, gp):

    # Update the kernel parameters:
    gp.set_parameter_vector(p)

    #  Compute the gradient of the loglikelihood:
    gll = gp.grad_log_likelihood(y)[1]

    return -gll


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
The paper says:
We run 500 steps of burn-in, followed by 5000 steps of MCMC using emcee.
First let's run the burn-in:
Can you write Python code for it?

# run a few samples as a burn-in:
print("Running burn-in")
p0, lnp, _ = sampler.run_mcmc(p0, 500)
sampler.reset()

print "Finished"


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
The paper says:
We set the mean function to zero
and we can see from Fig 7 that the data have also been normalised to have a maximum value of one.
So, let's also do that:
Can you write Python code for it?

mean = np.mean(value)
value-=mean

norm = np.max(value)
value/=norm


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
We also need to specify our parameter priors. Here we'll just use uniform logarithmic priors. The ranges are the same as specified in Table 3 of https://arxiv.org/pdf/1703.09710.pdf.
<img src="table3.png">
Can you write Python code for it?

# set the logprior
def lnprior(p):

    # These ranges are taken from Table 3
    # of https://arxiv.org/pdf/1703.09710.pdf

    lnB = p[0]
    lnC = p[1]
    lnL = p[2]  # our prior is defined on ln(L), not ln(c)
    lnP = np.log(p[3])

    # really crappy prior:
    if (-10.<lnB<0.) and (-15.<lnC<5.) and (0.<lnL<5.0) and (-3.<lnP<5.):
        return 0.0

    return -np.inf
    #return gp.log_prior()


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
We can then use these inputs to initiate our sampler:
Can you write Python code for it?

# initalise the sampler:
sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, args=data)


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
We need to pick some first guess parameters. Because we're lazy we'll just start by setting them all to 1:
Can you write Python code for it?

log_a = 0.0;log_b = 0.0; log_c = 0.0; log_P = 0.0
kernel = CustomTerm(log_a, log_b, log_c, log_P)

gp = celerite.GP(kernel, mean=0.0)

#yerr = 1e-6*np.ones(time.shape)
yerr = np.zeros(time.shape)

#gp.compute(time,yerr)
gp.compute(time)

print("Initial log-likelihood: {0}".format(gp.log_likelihood(value)))

t = np.arange(np.min(time),np.max(time),0.1)

# calculate expectation and variance at each point:
mu, cov = gp.predict(value, t)
std = np.sqrt(np.diag(cov))

ax = pl.subplot(111)
pl.plot(t,mu)
ax.fill_between(t,mu-std,mu+std,facecolor='lightblue', lw=0, interpolate=True)
pl.scatter(time,value,s=2)
pl.axis([0.,60.,-1.,1.])
pl.ylabel("Relative flux [ppt]")
pl.xlabel("Time [days]")
pl.show()


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
We then need to combine our log likelihood and our log prior into an (unnormalised) log posterior:
Can you write Python code for it?

# set the logposterior:
def lnprob(p, x, y):

    lp = lnprior(p)

    return lp + lnlike(p, x, y) if np.isfinite(lp) else -np.inf


I am working on the file "CDT-KickOff/TUTORIAL/KeplerLightCurveCelerite.ipynb".
The first task is:
ok, now we have our probability stuff set up we can run the MCMC. We'll start by explicitly specifying our Kepler data as our training data:
Can you write Python code for it?


I am working on the file "CE5/CE-5 frame analysis ATA 2021-01-23.ipynb".
The first task is:
AOS frames
AOS frames come from spacecraft 91 and virtual channels 1 and 2. Other combinations are most likely to corruted frames despite the fact that the Reed-Solomon decoder was successful.
Can you write Python code for it?

aos = [CE5_AOSFrame.parse(f) for f in frames]

collections.Counter([a.primary_header.transfer_frame_version_number for a in aos])

collections.Counter([a.primary_header.spacecraft_id for a in aos
                    if a.primary_header.transfer_frame_version_number == 1])

collections.Counter([a.primary_header.virtual_channel_id for a in aos
                    if a.primary_header.transfer_frame_version_number == 1
                    and a.primary_header.spacecraft_id == 108])


I am working on the file "CE5/CE-5 frame analysis ATA 2021-01-23.ipynb".
The first task is:
Here we look at some Chang'e 5 low data rate telemetry received with Allen Telescope array on 2021-01-23, during its transfer to the Sun-Earth L1 point. The recorded data corresponds to the frequency 8471.2 MHz, which corresponds to the lander.
The frames are CCSDS concatenated frames with a frames size of 220 bytes.
Can you write Python code for it?

def load_frames(path):
    frame_size = 220
    frames = np.fromfile(path, dtype = 'uint8')
    frames = frames[:frames.size//frame_size*frame_size].reshape((-1, frame_size))
    return frames

frames = load_frames('ATA_2021-01-23/ce5_frames_1.u8')


I am working on the file "CE5/CE-5 frame analysis ATA 2021-01-23.ipynb".
The first task is:
There are space packets in may APIDs. The contents of each APID are shown belown in plot form, but it's not easy to guess what any of the values mean.
Can you write Python code for it?

vc1_apids = collections.Counter([p.APID for p in vc1_sp_headers])
vc1_apids

vc1_by_apid = {apid : [p for h,p in zip(vc1_sp_headers, vc1_packets)
                           if h.APID == apid] for apid in vc1_apids}

plot_apids(vc1_by_apid, 108, 1)


I am working on the file "CE5/CE-5 frame analysis ATA 2021-01-23.ipynb".
The first task is:
Virtual channel 1
The vast majority of frames belong to virtual channel 1, which seems to send real-time telemetry.
Can you write Python code for it?

[a.primary_header for a in aos if a.primary_header.virtual_channel_id == 1][:10]

vc1 = [a for a in aos if a.primary_header.virtual_channel_id == 1]
fc = np.array([a.primary_header.virtual_channel_frame_count for a in vc1])
[a.insert_zone for a in aos[:10]]

t_vc1 = np.datetime64('2012-08-01') + np.timedelta64(1, 's') * np.array([a.insert_zone.timestamp for a in vc1])

plt.figure(figsize = (10,6), facecolor = 'w')
plt.plot(t_vc1, fc, '.')
plt.title("Chang'e 5 virtual channel 1 timestamps")
plt.xlabel('AOS frame timestamp')
plt.ylabel('AOS virtual channel frame counter');

plt.figure(figsize = (10,6), facecolor = 'w')
plt.plot(t_vc1[1:], np.diff(fc)-1, '.')
plt.title("Chang'e 5 spacecraft 91 virtual channel 1 frame loss")
plt.xlabel('AOS frame timestamp')
plt.ylabel('Frame loss')
plt.ylim((-1,50));


I am working on the file "CE5/CE-5 frame analysis ATA 2021-01-23.ipynb".
The first task is:
We need to sort the data, since the different files we've loaded up are not in chronological order.
Can you write Python code for it?

vc1_packets = list(ccsds.extract_space_packets(vc1, 108, 1, get_timestamps = True))

vc1_sp_headers = [ccsds.SpacePacketPrimaryHeader.parse(p[0]) for p in vc1_packets]


I am working on the file "CE5/CE-5 frame analysis ATA 2021-09-18.ipynb".
The first task is:
AOS frames
AOS frames come from spacecraft 108 and virtual channels 1 and 2. Other combinations are most likely to corruted frames despite the fact that the Reed-Solomon decoder was successful.
Can you write Python code for it?


I am working on the file "CE5/CE-5 frame analysis ATA 2021-09-18.ipynb".
The first task is:
The frames are CCSDS concatenated frames with a frames size of 220 bytes.
Can you write Python code for it?

def load_frames(path):
    frame_size = 220
    frames = np.fromfile(path, dtype = 'uint8')
    frames = frames[:frames.size//frame_size*frame_size].reshape((-1, frame_size))
    return frames

frames = load_frames('ATA_2021-09-18/ce5_frames.u8')


I am working on the file "CE5/CE-5 frame analysis ATA 2021-09-18.ipynb".
The first task is:
There are space packets in may APIDs. The contents of each APID are shown belown in plot form, but it's not easy to guess what any of the values mean.
Can you write Python code for it?


I am working on the file "CE5/CE-5 frame analysis ATA 2021-09-18.ipynb".
The first task is:
Virtual channel 1
The vast majority of frames belong to virtual channel 1, which seems to send real-time telemetry.
Can you write Python code for it?

[a.primary_header for a in aos if a.primary_header.virtual_channel_id == 1][:10]

vc1 = [a for a in aos if a.primary_header.virtual_channel_id == 1]
fc = np.array([a.primary_header.virtual_channel_frame_count for a in vc1])
[a.insert_zone for a in aos[:10]]

t_vc1 = np.datetime64('2012-08-01') + np.timedelta64(1, 's') * np.array([a.insert_zone.timestamp for a in vc1])

t_vc1[0]

plt.figure(figsize = (10,6), facecolor = 'w')
plt.plot(t_vc1, fc, '.')
plt.title("Chang'e 5 virtual channel 1 timestamps")
plt.xlabel('AOS frame timestamp')
plt.ylabel('AOS virtual channel frame counter');

plt.figure(figsize = (10,6), facecolor = 'w')
plt.plot(t_vc1[1:], np.diff(fc)-1, '.')
plt.title("Chang'e 5 spacecraft 91 virtual channel 1 frame loss")
plt.xlabel('AOS frame timestamp')
plt.ylabel('Frame loss');


I am working on the file "CE5/CE-5 frame analysis ATA 2021-09-18.ipynb".
The first task is:
We need to sort the data, since the different files we've loaded up are not in chronological order.
Can you write Python code for it?


I am working on the file "CE5/CE-5 frame analysis Bochum.ipynb".
The first task is:
AOS frames
AOS frames come from spacecraft 91 and virtual channels 1.
Can you write Python code for it?

aos = [CE5_AOSFrame.parse(f) for f in frames]

collections.Counter([a.primary_header.transfer_frame_version_number for a in aos])

collections.Counter([a.primary_header.spacecraft_id for a in aos
                    if a.primary_header.transfer_frame_version_number == 1])

collections.Counter([a.primary_header.virtual_channel_id for a in aos
                    if a.primary_header.transfer_frame_version_number == 1])

[a.primary_header for a in aos if a.primary_header.virtual_channel_id == 1][:10]

fc = np.array([a.primary_header.virtual_channel_frame_count for a in aos])
t = get_timestamps([a.insert_zone.timestamp for a in aos])

plt.figure(figsize = (10,6), facecolor = 'w')
plt.plot(t, fc, '.')
plt.title("Chang'e 5 spacecraft 91 virtual channel 1 timestamps")
plt.xlabel('AOS frame timestamp')
plt.ylabel('AOS virtual channel frame counter');

plt.figure(figsize = (10,6), facecolor = 'w')
plt.plot(t[1:], np.diff(fc)-1)
plt.title("Chang'e 5 spacecraft 91 virtual channel 1 frame loss")
plt.xlabel('AOS frame timestamp')
plt.ylabel('Frame loss');

vc1_packets = list(ccsds.extract_space_packets(aos, 91, 1, get_timestamps = True))

vc1_sp_headers = [ccsds.SpacePacketPrimaryHeader.parse(p[0]) for p in vc1_packets]


I am working on the file "CE5/CE-5 frame analysis Bochum.ipynb".
The first task is:
In this notebook we analyzed the frames decoded off the low datarate (PCM/PM/PSK 2048baud with 65536Hz subcarrier) signal of the Chang'e 5 lander+ascender combo at 8463.7 MHz transmitted from the lunar surface on 2020-12-02.
Can you write Python code for it?

def load_frames(path):
    frame_size = 220
    frames = np.fromfile(path, dtype = 'uint8')
    return frames[:frames.size//frame_size*frame_size].reshape((-1, frame_size))

frames = load_frames('Bochum_2020-12-02_8463/SDRSharp_20201202_060001Z_8463752190Hz_IQ_2M5_dec8_CE5_frames.u8')


I am working on the file "CE5/CE-5 high-speed frame.ipynb".
The first task is:
The data field of the Space Packets contains two back to back AOS packets of 252 bytes.
The AOS frames have the replay flag set and belong to spacecraft 197 virtual channel 52. The insert zone of these frames contains 3 unknown bytes followed by a 32 bit timestamp in the usual format.
Can you write Python code for it?

vc38_replay_aos = vc38_packet_data.reshape((vc38_packet_data.shape[0]*2, -1))
vc38_replay_aos_frames = [CE5_AOSReplayFrame.parse(f) for f in vc38_replay_aos]

[a.primary_header for a in vc38_replay_aos_frames[:10]]

{a.primary_header.virtual_channel_id for a in vc38_replay_aos_frames}

{a.primary_header.spacecraft_id for a in vc38_replay_aos_frames}

vc38_replay_timestamps = np.timedelta64(1,'s')*np.array([a.insert_zone.timestamp for a in vc38_replay_aos_frames])\
    + np.datetime64('2012-08-01')

fc = [a.primary_header.virtual_channel_frame_count for a in vc38_replay_aos_frames]

plt.figure(figsize = (10,6), facecolor = 'w')
plt.plot(vc38_replay_timestamps, fc, '.')
plt.title("Chang'e 5 spacecraft 197 virtual channel 52 timestamps (replay)")
plt.xlabel('AOS frame timestamp')
plt.ylabel('AOS virtual channel frame counter');


I am working on the file "CE5/CE-5 high-speed frame.ipynb".
The first task is:
The timestamps of these packets seem to span too much time for this short recording. I think this has something to do with the underlying replay data.
Can you write Python code for it?

fc = np.array([a.primary_header.virtual_channel_frame_count for a in vc38])
vc38_timestamps = np.timedelta64(1,'s')*np.array([a.insert_zone.timestamp for a in vc38])\
    + np.datetime64('2012-08-01')
good = vc38_timestamps <= np.datetime64('2020-12-01')

plt.figure(figsize = (10,6), facecolor = 'w')
plt.plot(vc38_timestamps[good], fc[good], '.')
plt.title("Chang'e 5 spacecraft 26 virtual channel 38 timestamps")
plt.xlabel('AOS frame timestamp')
plt.ylabel('AOS virtual channel frame counter');

frames_per_second = (fc[-1]-fc[0])/(vc38_timestamps[-1] - vc38_timestamps[0]).astype('float')
frames_per_second

frames_vc38 = frames[[a.primary_header.virtual_channel_id == 38
                      and a.primary_header.transfer_frame_version_number == 1
                      and a.primary_header.spacecraft_id == 26
                      for a in aos]]

plt.figure(figsize = (15,15), facecolor = 'w')
plt.imshow(frames_vc38[:200], aspect = 1)
plt.title("Chang'e 5 spacecraft 26 virtual channel 38 AOS frames");


I am working on the file "CE5/CE-5 high-speed frame.ipynb".
The first task is:
These AOS frames have a CRC-16. All of the frames are OK.
Can you write Python code for it?

crc_ok = np.array([crc16_ccitt_false(f) for f in vc38_replay_aos]) == 0

np.all(crc_ok)


I am working on the file "CE5/CE-5 high-speed frame.ipynb".
The first task is:
Virtual channel 38
The contents of virtual channel 38 are rather complicated due to the large amount of layer present. Here we peel the layers one by one. The first layer are AOS frames with an insert zone just like in virtual channel 63.
The data field of the AOS frames contains a single Space Packet.
Can you write Python code for it?

vc38 = [a for a in aos if a.primary_header.virtual_channel_id == 38]
[a.primary_header for a in vc38]


I am working on the file "CE5/CE-5 high-speed frame.ipynb".
The first task is:
Virtual channel 63
Virtual channel 63 contains padding. Besides the AOS header, there is a 6 byte insert zone which contains 0xf0f0 and a timestamp (in the same format as the low rate telemetry). The remaining data is filled with 0xaa.
Can you write Python code for it?

vc63 = [a for a in aos if a.primary_header.virtual_channel_id == 63
       and a.primary_header.transfer_frame_version_number == 1
       and a.primary_header.spacecraft_id == 26]
[a.primary_header for a in vc63[:10]]

frames_vc63 = frames[[a.primary_header.virtual_channel_id == 63
                      and a.primary_header.transfer_frame_version_number == 1
                      and a.primary_header.spacecraft_id == 26
                      for a in aos]]

np.unique(frames_vc63[:,12:])

hex(170)

{hex(a.insert_zone.unknown) for a in vc63}

hex(240)

vc63_timestamps = np.timedelta64(1,'s')*np.array([a.insert_zone.timestamp for a in vc63])\
    + np.datetime64('2012-08-01')

fc = [a.primary_header.virtual_channel_frame_count for a in vc63]

plt.figure(figsize = (10,6), facecolor = 'w')
plt.plot(vc63_timestamps, fc, '.')
plt.title("Chang'e 5 spacecraft 26 virtual channel 63 timestamps")
plt.xlabel('AOS frame timestamp')
plt.ylabel('AOS virtual channel frame counter');

plt.figure(figsize = (10,6), facecolor = 'w')
plt.plot(vc63_timestamps[1:], np.diff(fc)-1)
plt.ylim((-1,20))
plt.title("Chang'e 5 spacecraft 26 virtual channel 63 frame loss")
plt.xlabel('AOS frame timestamp')
plt.ylabel('Lost frames');

frames_per_second = (fc[-1]-fc[0])/(vc63_timestamps[-1] - vc63_timestamps[0]).astype('float')
frames_per_second


I am working on the file "CERN.ipynb".
The first task is:
Capture waveforms and plot it
Can you write Python code for it?

%matplotlib notebook

fig,ax = plt.subplots(1,1)

# looking for oscilloscope
devices =  getDeviceList()
# initiate oscilloscope
osc = UsbTmcDriver(devices[0])

df = pd.Series(0) # Create a new dataframe

for n in range(10):
    # waiting for SINGLE trigger
    osc.write(':SINGLE')
    osc.write(':TRIG:STAT?')
    while (osc.read(4) != 'STOP'):
        osc.write(':TRIG:STAT?')

    #osc.write(':STOP')
    osc.write(':WAV:SOUR CHAN1')
    osc.write(':WAV:MODE NORM')
    osc.write(':WAV:FORM BYTE')
    osc.write(':WAV:DATA?')

    #wave = map(float, osc.read(100).split(','))
    wave = bytearray(osc.read(1400))

    fig.clf()
    plt.ylim(0, 255)
    plt.plot(wave[10:])
    fig.canvas.draw()
    time.sleep(0.1)

runs = 10

osc.write(':WAV:XINC?')
xinc = float(osc.read(100))
print 'XINC:', xinc
osc.write(':WAV:YINC?')
yinc = float(osc.read(100))
print 'YINC:', yinc

osc.write(':WAV:SOUR CHAN1')
osc.write(':WAV:MODE NORM')
osc.write(':WAV:FORM BYTE')
osc.write(':WAV:POIN 700')

with h5py.File('data.h5', 'w') as hf:
    hf.create_dataset('XINC', data=(xinc))
    hf.create_dataset('YINC', data=(yinc))
    #hf.create_dataset('YINC', bytearray(yinc))
    for n in range(runs):
        #osc.write(':RUN')
        # waiting for SINGLE trigger
        osc.write(':SING')
        osc.write(':TRIG:STAT?')
        while (osc.read(4) != 'STOP'):
            osc.write(':TRIG:STAT?')
        time.sleep(0.2)

        #osc.write(':STOP')
        osc.write(':WAV:DATA?')
        wave1 = bytearray(osc.read(500))
        wave2 = bytearray(osc.read(500))
        wave = np.concatenate((wave1[11:],wave2[:(700-489)]))
        #osc.write(':WAV:END')
        hf.create_dataset(str(n), data=wave)

%matplotlib inline

with h5py.File('data.h5','r') as hf:
    #print('List of arrays in this file: \n', hf.keys())
    print 'XINC:', np.array(hf.get('XINC')), '   YINC:', np.array(hf.get('YINC'))
    for n in range(10):
        data = hf.get(str(n))
        np_data = np.array(data)
        plt.plot(np_data)
        #print np_data
        maximum =...


I am working on the file "CERN.ipynb".
The first task is:
Read number of waveforms and write it to a file
Can you write Python code for it?

runs = 10

%matplotlib notebook
fig,ax = plt.subplots(1,1)

osc.write(':WAV:XINC?')
xinc = float(osc.read(100))
print 'XINC:', xinc
osc.write(':WAV:YINC?')
yinc = float(osc.read(100))
print 'YINC:', yinc

osc.write(':WAV:SOUR CHAN1')
osc.write(':WAV:MODE NORM')
osc.write(':WAV:FORM BYTE')
osc.write(':WAV:POIN 700')

with h5py.File('data.h5', 'w') as hf:
    hf.create_dataset('XINC', data=(xinc))
    hf.create_dataset('YINC', data=(yinc))
    #hf.create_dataset('YINC', bytearray(yinc))
    for n in range(runs):
        #osc.write(':RUN')
        # waiting for SINGLE trigger
        osc.write(':SING')
        osc.write(':TRIG:STAT?')
        while (osc.read(4) != 'STOP'):
            osc.write(':TRIG:STAT?')
        time.sleep(0.1)

        #osc.write(':STOP')
        osc.write(':WAV:DATA?')
        wave1 = bytearray(osc.read(500))
        wave2 = bytearray(osc.read(500))
        wave = np.concatenate((wave1[11:],wave2[:(700-489)]))
        #osc.write(':WAV:RES')
        hf.create_dataset(str(n), data=wave)
        fig.clf()
        plt.ylim(0, 255)
        plt.plot(wave)
        fig.canvas.draw()


I am working on the file "CERN.ipynb".
The first task is:
Read repeatedly records from oscilloscope
This should be run after the initialization step.
Timeout at the end should be enlarged if not all 508 frames are transferred.
Can you write Python code for it?

for f in glob.iglob("./*.h5"): # delete all .h5 files
    #print 'Deleting', f
    os.remove(f)

filename = 1
while True:
    osc.write(':WAV:SOUR CHAN1')
    osc.write(':WAV:MODE NORM')
    osc.write(':WAV:FORM BYTE')
    osc.write(':WAV:POIN 1400')

    osc.write(':WAV:XINC?')
    xinc = float(osc.read(100))
    print 'XINC:', xinc,
    osc.write(':WAV:YINC?')
    yinc = float(osc.read(100))
    print 'YINC:', yinc,

    #osc.write(':FUNC:WREC:FEND?') # get number of last frame
    #end = int(osc.read(100))
    #while end<508:
    #    time.sleep(10)
    #    osc.write(':FUNC:WREC:FEND?') # get number of last frame
    #    end = int(osc.read(100))
    #    print 'Waiting', end

    osc.write(':FUNC:WREP:FEND?') # get number of last frame
    frames = int(osc.read(100))
    print 'FRAMES:', frames, 'SUBRUN', filename

    with h5py.File('data'+str(filename)+'.h5', 'w') as hf:
        hf.create_dataset('FRAMES', data=(frames)) # write number of frames
        hf.create_dataset('XINC', data=(xinc)) # write axis parameters
        hf.create_dataset('YINC', data=(yinc))
        osc.write(':FUNC:WREP:FCUR 1') # skip to n-th frame
        time.sleep(0.5)
        for n in range(1,frames+1):
            osc.write(':FUNC:WREP:FCUR ' + str(n)) # skip to n-th frame
            time.sleep(0.1)

            osc.write(':WAV:DATA?') # read data
            #time.sleep(0.4)
            wave1 = bytearray(osc.read(500))
            wave2 = bytearray(osc.read(500))
            wave3 = bytearray(osc.read(500))
            #wave4 = bytearray(osc.read(500))
            #wave = np.concatenate((wave1[11:],wave2[:(500-489)],wave3[:(700-489)]))
            wave = np.concatenate((wave1[11:],wave2,wave3[:-1]))
            hf.create_dataset(str(n), data=wave)
    filename = filename + 1
    osc.write(':FUNC:WREC:OPER REC') # start recording
    #time.sleep(1800) # delay for capturing
    input("Press Enter to continue...")


I am working on the file "CERN.ipynb".
The first task is:
Reading buffer with stored records
Can you write Python code for it?

osc.write(':WAV:SOUR CHAN1')
osc.write(':WAV:MODE NORM')
osc.write(':WAV:FORM BYTE')
osc.write(':WAV:POIN 1400')

with h5py.File('data.h5', 'w') as hf:
    osc.write(':WAV:XINC?')
    xinc = float(osc.read(100))
    print 'XINC:', xinc
    osc.write(':WAV:YINC?')
    yinc = float(osc.read(100))
    print 'YINC:', yinc
    osc.write(':FUNC:WREP:FEND?') # get number of last frame
    frames = int(osc.read(100))
    print 'FRAMES:', frames
    hf.create_dataset('FRAMES', data=(frames)) # write number of frames
    hf.create_dataset('XINC', data=(xinc)) # write axis parameters
    hf.create_dataset('YINC', data=(yinc))
    osc.write(':FUNC:WREP:FCUR 1') # skip to n-th frame
    time.sleep(0.5)
    for n in range(1,frames+1):
        osc.write(':FUNC:WREP:FCUR ' + str(n)) # skip to n-th frame
        time.sleep(0.1)

        osc.write(':WAV:DATA?') # read data
        #time.sleep(0.4)
        wave1 = bytearray(osc.read(500))
        wave2 = bytearray(osc.read(500))
        wave3 = bytearray(osc.read(500))
        #wave4 = bytearray(osc.read(500))
        #wave = np.concatenate((wave1[11:],wave2[:(500-489)],wave3[:(700-489)]))
        wave = np.concatenate((wave1[11:],wave2,wave3[:-1]))
        hf.create_dataset(str(n), data=wave)

%matplotlib inline

with h5py.File('data1.h5','r') as hf:
    #print('List of arrays in this file: \n', hf.keys())
    print 'XINC:', np.array(hf.get('XINC')), '   YINC:', np.array(hf.get('YINC')), '   FRAMES:', np.array(hf.get('FRAMES'))
    frames = np.array(hf.get('FRAMES'))
    for n in range(1,frames+1):
        data = hf.get(str(n))
        np_data = np.array(data)
        plt.plot(np_data)
        #print np_data
        maximum = 0
        minimum = 255
        suma = 0
        for i in range(700):
            if np_data[i] > maximum:
                maximum = np_data[i]
            if np_data[i] < minimum:
                minimum = np_data[i]
            suma = suma + np_data[i]
        if n < 10:
            print n, minimum, maximum, suma


I am working on the file "CEsprey - RandomForest/Facies_Feature_Engineering_And_ML.ipynb".
The first task is:
2.0 - Feature Generation
Can you write Python code for it?

## Create a difference vector for each feature e.g. x1-x2, x1-x3... x2-x3...

# order features in depth.

feature_vectors = training_data.drop(['Formation', 'Well Name','Facies'], axis=1)
feature_vectors = feature_vectors[['Depth','GR', 'ILD_log10', 'DeltaPHI', 'PHIND', 'PE']]

def difference_vector(feature_vectors):
    length = len(feature_vectors['Depth'])
    df_temp = np.zeros((25, length))

    for i in range(0,int(len(feature_vectors['Depth']))):

        vector_i = feature_vectors.iloc[i,:]
        vector_i = vector_i[['GR', 'ILD_log10', 'DeltaPHI', 'PHIND', 'PE']]
        for j, value_j in enumerate(vector_i):
            for k, value_k in enumerate(vector_i):
                differ_j_k = value_j - value_k
                df_temp[5*j+k, i] = np.abs(differ_j_k)

    return df_temp

def diff_vec2frame(feature_vectors, df_temp):

    heads = feature_vectors.columns[1::]
    for i in range(0,5):
        string_i = heads[i]
        for j in range(0,5):
            string_j = heads[j]
            col_head = 'diff'+string_i+string_j

            df = pd.Series(df_temp[5*i+j, :])
            feature_vectors[col_head] = df
    return feature_vectors

df_diff = difference_vector(feature_vectors)
feature_vectors = diff_vec2frame(feature_vectors, df_diff)

# Drop duplicated columns and column of zeros
feature_vectors = feature_vectors.T.drop_duplicates().T
feature_vectors.drop('diffGRGR', axis = 1, inplace = True)

# Add Facies column back into features vector

feature_vectors['Facies'] = training_data['Facies']

# # group by facies, take statistics of each facies e.g. mean, std. Take sample difference of each row with

def facies_stats(feature_vectors):
    facies_labels = np.sort(feature_vectors['Facies'].unique())
    frame_mean = pd.DataFrame()
    frame_std = pd.DataFrame()
    for i, value in enumerate(facies_labels):
        facies_subframe = feature_vectors[feature_vectors['Facies']==value]
  ...


I am working on the file "CEsprey - RandomForest/Facies_Feature_Engineering_And_ML.ipynb".
The first task is:
3.0 - Generate plots of each feature
Can you write Python code for it?

# A = feature_vectors.sort_values(by='Facies')
# A.reset_index(drop=True).plot(subplots=True, style='b', figsize = [12, 400])


I am working on the file "CEvNS-examples.ipynb".
The first task is:
Now we calculate (and plot) a whole bunch of recoil rates.
Choose which rates to plot by specifying SM, magnetic, scalar, vector in interactions:
Can you write Python code for it?

interactions = ["SM", "scalar"]

#Specify a nucleus - Germanium
A_Ge = 73
Z_Ge = 32

weight = 1.0

#Calculate differential rate
E_R = np.logspace(-3,2,500)

#Vectorize the differentialRate functions
diffRate_CEvNS = np.vectorize(CEvNS.differentialRate_CEvNS) #Vector exchange
diffRate_mag = np.vectorize(CEvNS.differentialRate_magnetic)   #Magnetic dipole
diffRate_scalar = np.vectorize(CEvNS.differentialRate_scalar)   #Scalar exchange

#Do some plotting
pl.figure(figsize=(8,5))

pl.fill_between(np.linspace(0.05,1), 1e-2, 1e3, alpha=0.2, facecolor='green')

if ("SM" in interactions):
    pl.loglog(E_R, diffRate_CEvNS(E_R, A_Ge, Z_Ge),'k',label=r"Standard Model CE$\nu$NS")

if ("magnetic" in interactions):
    pl.loglog(E_R, diffRate_mag(E_R, A_Ge, Z_Ge, 2.2e-12), label=r"$\mu_\nu = 2.2\times10^{-12}\,\mu_B$")
    pl.loglog(E_R, diffRate_mag(E_R, A_Ge, Z_Ge, 2.9e-11), label=r"$\mu_\nu = 2.9\times10^{-11}\,\mu_B$")
    pl.loglog(E_R, diffRate_mag(E_R, A_Ge, Z_Ge, 2.0e-10), label=r"$\mu_\nu = 3.0\times10^{-10}\, \mu_B$")

if ("scalar" in interactions):
    pl.loglog(E_R, diffRate_scalar(E_R, A_Ge, Z_Ge, (1.0e-6)**2, 1e-3),label=r"$m_\phi = 1$ keV; $g_\phi =  10^{-6}$")
    pl.loglog(E_R, diffRate_scalar(E_R, A_Ge, Z_Ge, (1.0e-6)**2, 1e0),label=r"$m_\phi = 1$ MeV; $g_\phi =  10^{-6}$")
    pl.loglog(E_R, diffRate_scalar(E_R, A_Ge, Z_Ge, (5.0e-2)**2, 1e5), label=r"$m_\phi = 100$ GeV; $g_\phi = 5\times10^{-2}$")

if ("Zprime" in interactions or "vector" in interactions):
    pl.loglog(E_R, diffRate_CEvNS(E_R, A_Ge, Z_Ge, (4e-6)**2, 1e-2), 'm-',label=r"+ $m_{Z'} = 10$ keV; $g_{Z'} = 4 \times 10^{-6}$")
    pl.loglog(E_R, diffRate_CEvNS(E_R, A_Ge, Z_Ge, (1e-5)**2, 1), 'r-',label=r"+ $m_{Z'} = 1$ MeV; $g_{Z'} = 10^{-5}$")
    pl.loglog(E_R, diffRate_CEvNS(E_R, A_Ge, Z_Ge, (1e-1)**2, 1e5), 'b-',label=r"+ $m_{Z'} = 100$ GeV; $g_{Z'} = 10^{-1}$")

pl.ylim(1e-2, 1e3)
pl.xlim(1e-3, 1e1)

leg = pl.legend(fontsize=12.0, loc="best", fancybox=True)
leg.get_frame().set_alpha(0.75)

pl.title(r"CE$\nu$NS on a Germanium target...


I am working on the file "CFL/DCT-Domain Subsampling.ipynb".
The first task is:
Back to the Pixel Domain
To make sure we don't have any errors, let's perform the inverse dct on each 8x8 block.
Can you write Python code for it?

from scipy.fftpack import idct

block_size = 8
Y_idct = np.zeros((y_height, y_width))

for y in range(0,y_height - block_size, block_size):
    yRange = np.arange(y,y+block_size)
    for x in range(0, y_width - block_size, block_size):
        xRange = np.arange(x,x+block_size)
        Y_idct[np.ix_(yRange,xRange)] = idct(idct(Y_dct[np.ix_(yRange,xRange)].T, norm='ortho').T, norm='ortho')

plt.imshow(Y_idct, cmap = plt.get_cmap('gray'), vmin = 0, vmax = 255, aspect='equal', interpolation='nearest');


I am working on the file "CFL/DCT-Domain Subsampling.ipynb".
The first task is:
Comparing with spatial domain subsampling
We can compare the results with a pixel domain subsampling. Looking at the owl, we notice that the DCT domain subsampling is blurrier. This is not necessarily a bad thing, looking back at the Cb and Cr planes in the first image, these planes don't have the sharp details of the luma plane, so a blurred prediction is ideal
Can you write Python code for it?

plt.figure(figsize=(15,10))
plt.subplot(1,2,1)
plt.title('Pixel Domain Subsampling (no filtering)')
plt.imshow(Y[::2, ::2], cmap = plt.get_cmap('gray'), vmin = 0, vmax = 255, aspect='equal', interpolation='nearest');
plt.subplot(1,2,2)
plt.title('DCT Domain Subsampling')
plt.imshow(Y_sub_scaled, cmap = plt.get_cmap('gray'), vmin = 0, vmax = 255, aspect='equal', interpolation='nearest');


I am working on the file "CFL/DCT-Domain Subsampling.ipynb".
The first task is:
The DCT Domain
Let's  convert the luma plane of our image to the DCT domain. To do so, we will use 8x8 blocks and the normalized DCT-II:
DCT-II
$$X_k = \sum_{n=0}^{N-1} x_n \cos \left( \frac{\pi}{N} \left( n + \frac{1}{2} \right)k \right) \quad k = 0,\ldots,N-1$$
Normalization
$$X_0 = X_0 \times \frac{1}{\sqrt{2}}$$
$$X_k = X_k \times \sqrt{\frac{2}{N}} \quad k=0,\ldots,N-1$$
Can you write Python code for it?

from scipy.fftpack import dct

block_size = 8
Y_dct = np.zeros((y_height, y_width))

for y in range(0,y_height - (block_size-1), block_size):
    yRange = np.arange(y,y+block_size)
    for x in range(0, y_width - (block_size-1), block_size):
        xRange = np.arange(x,x+block_size)
        Y_dct[np.ix_(yRange,xRange)] = dct(dct(Y[np.ix_(yRange,xRange)].T, norm='ortho').T, norm='ortho')

plt.imshow(Y_dct);


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report.ipynb".
The first task is:
First lets set the output path for where we want our plots to be saved and the database path and see what tables it contains
Can you write Python code for it?

database_path = '../csvdb'
output_path = '.'


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report.ipynb".
The first task is:
Great thats all the filtering stats done by now you should have a good idea about:
* the number of reads present in your origional bam file
* the number of reads left after filtering
* the proportion of reads filtered by each filter
* whether the proportion of reads filtered at each stage looks quite consistent across samples
Calculate the nonredundant fraction (NRF)
NRF = Number of distinct uniquely mapping reads (after removing duplicates)/Total number of reads
for ChIP-Seq
- NRF < 0.5 = concerning
- 0.8 > NRF > 0.5 = Acceptable
- 0.9 > NRF > 0.8 = compliant
- NRF > 0.9 = Ideal
for ATAC-Seq
- NRF < 0.7 = Concerning
- 0.9 > NRF > 0.7 = Acceptable
- NRF > 0.9 = Ideal
Can you write Python code for it?

filtering_stats['NRF'] = filtering_stats.duplicates/filtering_stats.pre_filtering
filtering_stats


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report.ipynb".
The first task is:
Now lets get the number of reads that are filtered out at each stage of the filtering by subtracting the number of reads at the filtering stage of interest from the number of reads in the stage prior to the filter of interst being applied
Can you write Python code for it?

#Get number of reads removed by each stage of filtering
order_of_filters = get_filter_order(filtering_stats)

df_reads_removed = pd.DataFrame(index=filtering_stats.index)
for loc in range(len(order_of_filters)):
    filt = order_of_filters[loc]
    if filt == 'pre_filtering':
        df_reads_removed['total_reads'] = filtering_stats['pre_filtering']
    else:
        previous_filter_step = order_of_filters[loc-1]
        #print("calcultation number removed by %s filtering step by doing number of reads in %s - number of reads in %s column \n" % (filt, previous_filter_step, filt))
        df_reads_removed['removed_by_%s_filter' % filt] = filtering_stats[previous_filter_step] - filtering_stats[filt]

print('\n\nTable shown as million reads removed by each filter:')
display(df_reads_removed.T.divide(1000000))
#plot how the reads have been filtered

ax = df_reads_removed.divide(1000000).plot(rot=90)
ax.set_xlabel('filters')
ax.set_ylabel('million reads removed (not pairs)')
ax.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
ax.set_title('number of reads removed at each stage of filtering')

ax = df_reads_removed.T.divide(1000000).plot(rot=90)
ax.set_xlabel('filters')
ax.set_ylabel('million reads removed (not pairs)')
ax.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
ax.set_title('number of reads removed at each stage of filtering')

ax = df_reads_removed.T.divide(1000000).drop('total_reads').plot(rot=90,kind='bar')
ax.set_xlabel('filters')
ax.set_ylabel('million reads removed (not pairs)')
ax.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
ax.set_title('number of reads removed at each stage of filtering')


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report.ipynb".
The first task is:
Now lets look at the number of reads filtered at each step side by side - this uses R for plotting
Can you write Python code for it?

filtered_df = filtering_stats.copy()
filtered_df = filtered_df.divide(1000000)
filtered_df['Name'] = filtered_df.index

%%R -i filtered_df -w 600 -h 600 -u px
library("reshape2")
filtered_df$Name <- factor(filtered_df$Name)

df.m = melt(filtered_df)
cbPalette <- c("#999999", "#E69F00", "#56B4E9", "#009E73", "#F0E442", "#0072B2", "#D55E00", "#CC79A7")
## pink #CC79A7 # orange #D55E00 # 0072B2 blue # yellow #F0E442 # green #009E73 # light blue

g = ggplot(data=df.m, aes(factor(Name), y=value,fill=variable)) + labs(title="Number of individual reads remaining after each filtering step") + geom_bar(stat="identity",position="dodge", width=0.7)  +  scale_fill_manual(values=cbPalette) + theme_bw()
g + scale_y_continuous(name="million reads remaining \n (individual reads not pairs)") + theme(plot.title=element_text(size=16, hjust=0.5, face='bold'), legend.position="top",axis.text=element_text(size=15,face='bold'),axis.text.x=element_text(size=15,face='bold',angle=90),axis.title.x=element_blank(),axis.title.y=element_text(size=10,face='bold'))


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report.ipynb".
The first task is:
Now lets plot the number of reads reamining at each filtering step side by side
Can you write Python code for it?

df_reads_removed_mills = df_reads_removed.divide(1000000)
df_reads_removed_mills['Name'] = df_reads_removed_mills.index

%%R -i df_reads_removed_mills -w 900 -h 800 -u px
library("reshape2")
df_reads_removed_mills$Name <- factor(df_reads_removed_mills$Name)

df.m = melt(df_reads_removed_mills)
cbPalette <- c("#999999", "#E69F00", "#56B4E9", "#009E73", "#F0E442", "#0072B2", "#D55E00", "#CC79A7")
## pink #CC79A7 # orange #D55E00 # 0072B2 blue # yellow #F0E442 # green #009E73 # light blue

g = ggplot(data=df.m, aes(factor(Name), y=value,fill=variable)) + labs(title="Number of reads remaining after each filtering step") + geom_bar(stat="identity",position="dodge", width=0.7)  +  scale_fill_manual(values=cbPalette) + theme_bw()
g + scale_y_continuous(name="Number of reads filtered at each step") + theme(plot.title=element_text(size=16, hjust=0.5, face='bold'), legend.position='top',axis.text=element_text(size=15,face='bold'),axis.text.x=element_text(size=15,face='bold',angle=90),axis.title.x=element_blank(),axis.title.y=element_text(size=10,face='bold'))


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report.ipynb".
The first task is:
Number of reads per samples
Firstly lets look at the size of our bam files pre and post filtering - hopefully the post-filtering bams will be smaller showing that some filtering has taken place
Can you write Python code for it?

#get table of bam file size
filtering_stats = getTableFromDB('select * from post_filtering_read_counts;',database_path)
filtering_stats.index = filtering_stats['Input_Bam']
filtering_stats.drop('Input_Bam',1,inplace=True)

#sort dataframe by values in rows to get order filters were applied
#this is based on the number of reads in each row
new_cols = filtering_stats.columns[filtering_stats.ix[filtering_stats.last_valid_index()].argsort()]
filtering_stats = filtering_stats[new_cols[::-1]]

#get number of reads in the bams before and after filtering - smallest_col = last filtering step applied
smallest_col  = filtering_stats.idxmin(axis=1)[1]

#plot bar graph of pre vs post filtering sizes
ax = filtering_stats[['pre_filtering',smallest_col]].divide(1000000).plot.bar()
ax.set_ylabel('Million Reads (not pairs)')
ax.legend(['pre_filtering','post_filtering'], loc=2,bbox_to_anchor=(1.05, 1),borderaxespad=0. )
ax.set_title('number of reads (not pairs) pre and post filtering')


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report.ipynb".
The first task is:
This code adds a button to see/hide code in html
Can you write Python code for it?

HTML('''<script>
code_show=true;
function code_toggle() {
 if (code_show){
 $('div.input').hide();
 } else {
 $('div.input').show();
 }
 code_show = !code_show
}
$( document ).ready(code_toggle);
</script>
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>''')


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report.ipynb".
The first task is:
This is where and when the notebook was run
Can you write Python code for it?


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report.ipynb".
The first task is:
This should give you a good idea of:
1) whether filtering has been applied - if post and pre filtering differ it has!
2) whether the proportion of filtering corresponds to the initial size of library
3) whether the proportion of filtering is consistent across samples
4) final bam size that is being taken forward to peakcalling - the requirements will differ for different technologies

Encode ATAC-Seq Guidelines: Each replicate should have 25 million non duplicate, non-mitochondrial aligned reads for single-end or 50 million paired-end reads (i.e. there should be 25 million fragments regardless of sequencing type)  - Jan 2017
Can you write Python code for it?

#get the order of filters applied
def get_filter_order(dataframe):
    '''function to print out the order of filters in dataframe'''
    print('order of filters applied to bam file:')
    for x in list(dataframe):
        if x != 'pre_filtering':
            print ('\t%s' % x)
    return list(dataframe)

filter_order = get_filter_order(filtering_stats)

print('Table of number of reads remaining at each state of filtering ')
display(filtering_stats.T)


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_insert_sizes.ipynb".
The first task is:
First lets set the output path for where we want our plots to be saved and the database path and see what tables it contains
Can you write Python code for it?

database_path = '../csvdb'
output_path = '.'
#database_path= "/ifs/projects/charlotteg/pipeline_peakcalling/csvdb"


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_insert_sizes.ipynb".
The first task is:
Insert Size Summary
1) lets getthe insert_sizes table from database
Firsly lets look at the summary statistics that us the mean fragment size, sequencing type and mean read length. This table is produced using macs2 for PE data, or bamtools for SE data
If IDR has been run the insert_size table will contain entries for the pooled and pseudo replicates too - we don't really want this as it will duplicate the data from the origional samples so we subset this out
Can you write Python code for it?

insert_df = getTableFromDB('select * from insert_sizes;',database_path)
insert_df = insert_df[insert_df["filename"].str.contains('pseudo')==False].copy()
insert_df = insert_df[insert_df["filename"].str.contains('pooled')==False].copy()

def add_expt_to_insertdf(dataframe):
    ''' splits track name for example HsTh1-RATotal-R1.star into expt
    featues, expt, sample_treatment and replicate and adds these as
    collumns to the dataframe'''
    expt = []
    treatment = []
    replicate = []
    for value in dataframe.filename:
        x = value.split('/')[-1]
        x = x.split('_insert')[0]
        # split into design features
        y = x.split('-')
        expt.append(y[-3])
        treatment.append(y[-2])
        replicate.append(y[-1])

    if len(expt) == len(treatment) and len(expt)== len(replicate):
        print ('all values in list correctly')
    else:
        print ('error in loading values into lists')

    #add collums to dataframe
    dataframe['expt_name'] = expt
    dataframe['sample_treatment'] = treatment
    dataframe['replicate'] = replicate

    return dataframe

insert_df = add_expt_to_insertdf(insert_df)
insert_df


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_insert_sizes.ipynb".
The first task is:
Now lets zoom in on the interesting region of the plot (the default in the code looks at fragment lengths from 0 to 800bp - you can change this below by setting the tuple in the ax.set_xlim() function
Can you write Python code for it?

ax = frag_length_df.divide(1000).plot(figsize=(9,9))
ax.set_ylabel('Number of fragments\n(thousands)')
ax.legend(loc=2,bbox_to_anchor=(1.05, 1),borderaxespad=0. )
ax.set_title('fragment length distribution')
ax.set_xlabel('fragment length (bp)')
ax.set_xlim((0,800))


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_insert_sizes.ipynb".
The first task is:
The code below provides functions for accessing the project database and extract a table names so you can see what tables have been loaded into the database and are available for plotting. It also has a function for geting table from the database and indexing the table with the track name
Can you write Python code for it?

def getTableNamesFromDB(database_path):
    # Create a SQL connection to our SQLite database
    con = sqlite3.connect(database_path)
    cur = con.cursor()
    # the result of a "cursor.execute" can be iterated over by row
    cur.execute("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name;")
    available_tables = (cur.fetchall())
    #Be sure to close the connection.
    con.close()
    return available_tables

db_tables = getTableNamesFromDB(database_path)
print('Tables contained by the database:')
for x in db_tables:
    print('\t\t%s' % x[0])

#This function retrieves a table from sql database and indexes it with track name
def getTableFromDB(statement,database_path):
    '''gets table from sql database depending on statement
    and set track as index if contains track in column names'''
    conn = sqlite3.connect(database_path)
    df = pd.read_sql_query(statement,conn)
    if 'track' in df.columns:
        df.index = df['track']
    return df


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_insert_sizes.ipynb".
The first task is:
This code adds a button to see/hide code in html
Can you write Python code for it?


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_insert_sizes.ipynb".
The first task is:
This is where we are and when the notebook was run
Can you write Python code for it?


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_insert_sizes.ipynb".
The first task is:
it is a bit trickly to see differences between samples of different library sizes so lets look and see if the reads for each fragment length is similar
Can you write Python code for it?

percent_frag_length_df = pd.DataFrame(index=frag_length_df.index)

for column in frag_length_df:
    total_frags = frag_length_df[column].sum()
    percent_frag_length_df[column] = frag_length_df[column].divide(total_frags)*100

ax = percent_frag_length_df.plot(figsize=(9,9))
ax.set_ylabel('Percentage of fragments')
ax.legend(loc=2,bbox_to_anchor=(1.05, 1),borderaxespad=0. )
ax.set_title('percentage fragment length distribution')
ax.set_xlabel('fragment length (bp)')
ax.set_xlim((0,800))


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_insert_sizes.ipynb".
The first task is:
lets graph the fragment length mean and tag size grouped by sample so we can see if they are much different
Can you write Python code for it?

ax = insert_df.boxplot(column='fragmentsize_mean', by='sample_treatment')
ax.set_title('for mean fragment size',size=10)
ax.set_ylabel('mean fragment length')
ax.set_xlabel('sample treatment')

ax = insert_df.boxplot(column='tagsize', by='sample_treatment')
ax.set_title('for tag size',size=10)
ax.set_ylabel('tag size')
ax.set_xlabel('sample treatment')
ax.set_ylim(((insert_df.tagsize.min()-2),(insert_df.tagsize.max()+2)))


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_reads_per_chr.ipynb".
The first task is:
Contigs that have been filtered should clearly show up with 0 reads across the row
Can you write Python code for it?

 def makeReadsPerChrPlot(df,path):
    '''takes table from database of chromosome lengths and makes individual plot for
    each sample of how many reads map to each chromosome'''
    to_keep = []
    for item in df.columns:
        if 'chr' in item:
            to_keep.append(item)

    df = df[to_keep]
    df = df.divide(1000000)

    #where plot will be sent to
    file_path = "/".join([path,'mapped_reads_per_chromosome_plot.pdf'])
    print ('figure_saved_to %s' % file_path)

    ax = df.T.plot(figsize=(11,5),
                   xticks = range(len(to_keep)),
                  title = 'Million reads mapped to each chromosome',
                  ylim=(0,10))

    #set labels for plots
    ax.set_xlabel("Contig")
    ax.set_ylabel("million reads")
    fig = matplotlib.figure.Figure()
    ax.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
    matplotlib.pyplot.savefig(file_path, bbox_inches='tight')
    matplotlib.pyplot.show()

makeReadsPerChrPlot(reads_per_chr_df.T,output_path)

def makePercentReadsPerChrPlot(df,path):
    '''takes the idxstats_reads_per_chromosome table from database and calculates percentage
    of reads mapping to each chromosome and plots this for each chromosome and returns
    percentage table'''
    c = df.copy()
    for item in c.columns:
        if 'chr' not in item and item != 'total_reads':
            c = c.drop(item,1)

    y = c.div(c.total_reads, axis ='index')*100
    y = y.drop('total_reads',1)
    file_path = "/".join([path,'percentage_mapped_reads_per_chromosome_plot.pdf'])

    print ('figure_saved_to %s' % file_path)
    ax = y.T.plot(figsize=(10,5),
                 xticks = range(len(y.columns)),
                 title = 'Percentage of total input reads that map to each contig',
                 ylim=(0,100))

    ax.set_xlabel("Contig")
    ax.set_ylabel("percentage_reads")
    ax.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
    fig = matplotlib.figure.Figure()
    matplotlib.pyplot.savefig(file_path,...


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_reads_per_chr.ipynb".
The first task is:
Here are some functions we need
Can you write Python code for it?

###These are functions used to manipulate the table so order of chromsomes is consistent with numbers
def StrIsInt(string):
    '''function that takes string and tests if it can be represented as an int
    e.g. returns true for "3", but False for "Chr3" '''
    try:
        int(string)
        return True
    except ValueError:
        return False

def orderListOfChr(unordered_chr_list):
    '''take a list of chromosomes  and return them in
    order of chromosome number not string order
    e.g. input = ["chr1",'chr11","chr2","chrM"]
    output = ["chr1",'chr2","chr11","chrM"]'''

    #make a empty list same length as chromosomes
    chr_id = [None]* len(unordered_chr_list)

    for value in unordered_chr_list:
        x = value.split("chr")[-1]

        # check if chr name is int or str
        if StrIsInt(x):
            chr_id[int(x)-1] = value
        else:
            chr_id.append(value)

    #remove none values from list
    ordered_chr_list = [x for x in chr_id if x is not None]
    return ordered_chr_list

def reorderDFbyChrOrder(df):
    '''Takes a dataframe indexed on chr name
    and returns dataframe so that index is sorted based on the
    chromosome number
    e.g.dataframe with index chr1,chr11,chr12,chr2,chrM
    will be returned with rows in the order "chr1, chr2, chr11, chr12, chrM" '''

    list_of_reordered_chr = orderListOfChr(df.index)
    return df.reindex(list_of_reordered_chr)

# this subsets dataframe so only includes columns containing chr
def getChrNames(df):
    '''takes dataframe with chromocome names in columns and returns a list of the chromosomes present'''
    to_keep = []
    for item in df.columns:
        if 'chr' in item:
            to_keep.append(item)
    return to_keep


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_reads_per_chr.ipynb".
The first task is:
Reads per Chromsome
1) lets get IDXstats tabel from database lets look at total number of maapped reads per chromosome for each sample
Can you write Python code for it?

idxstats_df = getTableFromDB('select * from idxstats_reads_per_chromosome;',database_path)
idxstats_df.index = idxstats_df['region']
reads_per_chr_df = reorderDFbyChrOrder(idxstats_df.drop('region', 1))
print ('this table shows million reads per chromosome')
reads_per_chr_df.divide(1000000)


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_reads_per_chr.ipynb".
The first task is:
The code below provides functions for accessing the project database and extract a table names so you can see what tables have been loaded into the database and are available for plotting. It also has a function for geting table from the database and indexing the table with the track name
Can you write Python code for it?


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_filtering_Report_reads_per_chr.ipynb".
The first task is:
This code adds a button to see/hide code in html
Can you write Python code for it?


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_peakstats.ipynb".
The first task is:
Check the files are matched up correctly - if they are not there is a bug in the peakcalling section of the pipeline

now lets look at the insert sizes that are callculated by macs2 (for PE samples) or bamtools (SE reads)
Can you write Python code for it?

insert_df = getTableFromDB('select * from insert_sizes;',database_path)


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_peakstats.ipynb".
The first task is:
Design of Experiment
Firstly lets check out the experimental design - this is specified in the design_file.tsv that is used to run the pipeline
1) lets get the table from database
Can you write Python code for it?

design_df= getTableFromDB('select * from design;',database_path)
design_df


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_peakstats.ipynb".
The first task is:
First lets set the output path for where we want our plots to be saved and the database path and see what tables it contains
Can you write Python code for it?


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_peakstats.ipynb".
The first task is:
Is there any correlation between the number of peaks and the number of fragments? lets plot this. Can you see any saturation where an increase in fragment number does not result in any further gains in peak number?
Can you write Python code for it?

ax =peakcalling_summary_df[['number_of_peaks','fragment_treatment_total']].divide(1000).plot.scatter(x='fragment_treatment_total',
                                                                                y='number_of_peaks')
ax.set_xlabel('number of PE fragments')
ax.set_title('correlation of number of fragments \n& number of peaks')
#ax.set_ylim((50000,160000))
#ax.set_xlim((20000000,70000000))


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_peakstats.ipynb".
The first task is:
Lets also have a quick check of the number of reads  & number of fragments in our samples
Can you write Python code for it?

peakcalling_frags_df = getTableFromDB('select * from post_filtering_check;',database_path)
peakcalling_frags_df = peakcalling_frags_df[['Input_Filename','total_reads']].copy()
peakcalling_frags_df['total_fragments'] = peakcalling_frags_df['total_reads'].divide(2)
peakcalling_frags_df


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_peakstats.ipynb".
The first task is:
Now lets double check what files peakcalling was performed for and whether they were paired with an input file. Input file is used in peakcalling to control for background noise. If the bamControl collumn has 'None' in it then a input control was not used for peakcalling
Lets also double check this in the 'peakcalling_bams_and_inputs' table that is used to generate the peakcalling statement:
Can you write Python code for it?

peakcalling_design_df= getTableFromDB('select * from peakcalling_bams_and_inputs;',database_path)
print ('''peakcalling_bams_and_inputs table used to generate the peakcalling statement:
           ChIPBams = the file you want to call peaks in e.g. ChIP or ATAC-Seq sample.
           InputBam = the sample used as the control in peakcalling. In ChIP-Seq this would be your input control\n''')
peakcalling_design_df


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_peakstats.ipynb".
The first task is:
Plot bar graph of  a number of peaks
Can you write Python code for it?


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_peakstats.ipynb".
The first task is:
The code below provides functions for accessing the project database and extract a table names so you can see what tables have been loaded into the database and are available for plotting. It also has a function for geting table from the database and indexing the table with the track name
Can you write Python code for it?


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_peakstats.ipynb".
The first task is:
This code allows you to see/hide the code in the html verision
Can you write Python code for it?

from IPython.display import HTML

HTML('''<script>
code_show=true;
function code_toggle() {
 if (code_show){
 $('div.input').hide();
 } else {
 $('div.input').show();
 }
 code_show = !code_show
}
$( document ).ready(code_toggle);
</script>
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>''')


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_peakstats.ipynb".
The first task is:
This is where we are and when the notebook was run
Can you write Python code for it?


I am working on the file "CGATPipelines/pipeline_docs/pipeline_peakcalling/notebooks/template_peakcalling_peakstats.ipynb".
The first task is:
below code provides a look at published datasets you can look at if you want to
Can you write Python code for it?

#greenleaf_data = pd.read_csv('/Users/charlotteg/Documents/7_BassonProj/Mar17/allelic-atac-seq.csv')
#greenleaf_data.drop([0],inplace=True)
#greenleaf_data['total usable reads'] = greenleaf_data['total usable reads'] / 2
#ax = greenleaf_data.plot.scatter(x='total usable reads', y='# of allelic informative(AI) peaks (>=10 reads)')
#ax.set_ylim((50000,160000))
#ax.set_xlim((20000000,70000000))
#greenleaf_data

#factor between number of reads and number of peaks


I am working on the file "CH5-Getting Started With Pandas/CH5 - Getting Started With Pandas.ipynb".
The first task is:
A critical Series feature for many applications is that it automatically aligns differently- indexed data in arithmetic operations:
Can you write Python code for it?


I am working on the file "CH5-Getting Started With Pandas/CH5 - Getting Started With Pandas.ipynb".
The first task is:
Series
Can you write Python code for it?

obj = Series([4,7,-5,3])
obj

obj.values

obj.index

obj2 = Series([4,6,8,9], index = ['a','d','t','y'])

obj2

obj2.index

obj2['y']

obj2['y']=3

obj2['y']

obj2[obj2 >= 6] # note that not y =3

obj2*2


I am working on the file "CH5-Getting Started With Pandas/CH5 - Getting Started With Pandas.ipynb".
The first task is:
Should you have data contained in a Python dict, you can create a Series from it by passing the dict
Can you write Python code for it?

sdata = {'Ohio':30000, 'Texas': 71000, 'Oregon': 16000, 'Utah': 5000}

obj3 = Series(sdata) # treats the first colomn as the index

obj3

obj3.index


I am working on the file "CH5-Getting Started With Pandas/CH5 - Getting Started With Pandas.ipynb".
The first task is:
Use a function in pandas
Can you write Python code for it?

states = ['California', 'Ohio', 'Oregon', 'Texas']

obj4 = Series(sdata, index = states)

obj4

pd.isnull(obj4)


I am working on the file "CH5-Getting Started With Pandas/CH5 - Getting Started With Pandas.ipynb".
The first task is:
You can also do this:
Can you write Python code for it?


I am working on the file "CLASSIC.ipynb".
The first task is:
Generate a master model from all folds
Can you write Python code for it?

svc = SVC(kernel='linear').fit(np.append(X[0], X[1], axis=0), target);

print(svc.coef_) # CADD / RVIS / PROVEAN
print(svc.intercept_)


I am working on the file "CLASSIC.ipynb".
The first task is:
Generate k folds with sklearn. This is done for both classes to ensure a balanced input. Training / test groups from each fold are paired (non-pathogenic / pathogenic) to be passed to a parallel pool for cross-validation.
The ROC AUC is calculated using a linear SVC for combination of CADD, RVIS, and PROVEAN. This is compared to the CADD-only AUC for the same test-set.
Can you write Python code for it?

# select a subset of features to test independent contributions; 0=CADD, 1=RVIS, 2=PROVEAN
featureCols = [0,1,2]

# folds is a (2,1) list of 1 of the folds generated by cross_validation.KFolds on both X[0] and X[1]
# these are paired by class to maintain a balance from each
# foldType corresponds to 0=train and 1=test
def getFoldFeatures(folds, foldType):
    features = X[0][folds[0][foldType]];
    features = np.append(features, X[1][folds[1][foldType]], axis=0)
    return features;

# as with getFoldFeatures, but returns a vector similar to targets, but shorter
def getFoldTargets(folds, foldType):
    return [0]*len(folds[0][foldType]) + [1]*len(folds[1][foldType]);

# function to be mapped over all paired folds
def trainAndValidate(folds):
    X_train = getFoldFeatures(folds, 0);
    y_train = getFoldTargets(folds, 0);
    assert(len((X_train)==len(y_train)))

    X_test = getFoldFeatures(folds, 1);
    y_test = getFoldTargets(folds, 1);
    assert(len((X_test)==len(y_test)))

    svc = SVC(kernel='linear').fit(X_train[:,featureCols], y_train);
    decision = svc.decision_function(X_test[:,featureCols])

    # X_test[:,0] corresponds to CADD scores alone
    return ({'y': y_test, 'decision': list(decision)}, roc_auc_score(y_test, decision), roc_auc_score(y_test, X_test[:,0]))

if __name__ == '__main__':
    p = Pool(n_cores)

    kf = list(); # folds for each class in X; shape=(2,1)
    for i in range(2):
        kf.append(list(cross_validation.KFold(n=len(X[i]), n_folds=k, shuffle=True, random_state=314159))); # constant random_state allows for reproduction of results; 314159 (pi * 10^5) is a nothing-up-my-sleeve number https://en.wikipedia.org/wiki/Nothing_up_my_sleeve_number

    pairedFolds = list(); # list of tuples containing fold for each class
    for f in range(k):
        pairedFolds.append((kf[0][f], kf[1][f]));

    results = p.map(trainAndValidate, pairedFolds);
    pairedAUC = [];
    for r in results:
        pairedAUC.append([r[1], r[2]])

    pairedAUC =...


I am working on the file "CLASSIC.ipynb".
The first task is:
Import the tab-separated values, and partition them according to the class.
0 = non-pathogenic (ESP)
1 = pathogenic (ClinVar)
Can you write Python code for it?

X = [[], []]

src = open("./all-scores.tsv")
for line in csv.reader(src, dialect="excel-tab"):
    y = int(line[0]);
    map(float, line);
    X[y].append(line[1:]);

for i in range(2):
    X[i] = np.array(X[i], dtype='float')

target = np.array([0]*len(X[0]) + [1]*len(X[1])) # vector of 0s then 1s


I am working on the file "CLASSIC.ipynb".
The first task is:
No need to edit below here  Just run each cell
Can you write Python code for it?

import csv, math, numpy as np, matplotlib.pyplot as plt

from sklearn.svm import SVC
from sklearn.metrics import roc_curve, roc_auc_score, auc, matthews_corrcoef
from sklearn import cross_validation
from multiprocessing import Pool
from scipy import stats


I am working on the file "CLASSIC.ipynb".
The first task is:
Output some meta data for inclusion in the manuscript
Can you write Python code for it?

print('n = %u; %0.2f%% of all CADD-tested sites' % (len(target), len(target)/16334*100))
print('neutral n=%u' % len(X[0]))
print('pathogenic n=%u' % len(X[1]))
print(max(len(X[0]), len(X[1]))/len(target))

import sys, sklearn
print(sys.version)
print(sklearn.__version__)


I am working on the file "CLV.ipynb".
The first task is:
A sample script for calculate CLV of Lorenz 63 model is placed at examples/clv.rs
Can you write Python code for it?

! cargo run --release --example clv > clv.csv


I am working on the file "CLV.ipynb".
The first task is:
Tangency of CLVs
Can you write Python code for it?

df = np.arccos(pd.read_csv("clv.csv"))

for col in df.columns:
    plt.figure()
    plt.title(col)
    df[col].hist(bins=100)
    plt.xlim(0, np.pi)
    plt.yscale("log")


I am working on the file "CMA-ES/分段函数平滑拟合/分段平滑拟合.ipynb".
The first task is:

Can you write Python code for it?

draw('fitting of interval  sin curve with curve')


draw('fitting of interval  quadratic curve with line')


I am working on the file "CMA-ES/分段函数平滑拟合/分段平滑拟合.ipynb".
The first task is:
4种策略下最优评价函数值
interval         | continuity1  |continuity2    |continuity3
 ---------------- |---------------|--------------|--------------
 0.0166098265438 |-0.0418342446941|-0.0174967646088|-0.0757380809252

正弦函数分段二次平滑
构造函数
构造正弦函数$y=10sin0.6x , x\in [-10,10]$，分七段用二次曲线进行拟合
构造数据集
Can you write Python code for it?

#构造数据集
def linefunc3(x):
    return 10*math.sin(0.7*x)+rd.uniform(-1.5,1.5)*rd.gauss(0,5)
X=np.linspace(-7,7,301)
Y=map(linefunc3,X)
DIM=14
RANGE=max(Y)-min(Y)
ALPHA=0.01*RANGE
BETA=PI/18.0
scatter(X,Y)


I am working on the file "CMA-ES/分段函数平滑拟合/分段平滑拟合.ipynb".
The first task is:
4种策略下最优评价函数值
interval         | continuity1  |continuity2    |continuity3
 ---------------- |---------------|--------------|--------------
 0.0171697128166  |-0.0416912639382|0.0146455354793|-0.0480936701006
设置BETA=PI/4.0（45度）

二次曲线分段二次平滑
利用二次曲线$y=ax^2+bx+c$进行分段拟合
Can you write Python code for it?

def realLineFunc(param):
    def f(x):
        if x<-5:
            return np.poly1d([param[0],param[1],param[2]])(x)
        elif x<-3:
            return np.poly1d([param[3],param[4],param[5]])(x)
        elif x<-1:
            return np.poly1d([param[6],param[7],param[8]])(x)
        elif x<1:
            return np.poly1d([param[9],param[10],param[11]])(x)
        elif x<3:
            return np.poly1d([param[12],param[13],param[14]])(x)
        elif x<5:
            return np.poly1d([param[15],param[16],param[17]])(x)
        else:
            return np.poly1d([param[18],param[19],param[20]])(x)
    return f
def mse(x,y,param):
    s=0
    for j in range(0,21,3):
        for i in range(int(j/3)*10,int(j/3+1)*10,1):
            s+=(y[i]-fittingFunc(param[j:j+3])(x[i]))**2
    return math.sqrt(s/200.0)
 #间断点
def evalfunc1(param):
    s=mse(X,Y,param)/RANGE
    a=0
    for j in range(3,21,3):
        a+=E(abs(fittingFunc(param[j-3:j])(X[(j/3)*10])-fittingFunc(param[j:j+3])(X[(j/3)*10]))/RANGE-ALPHA)-1
    return s+a
#间断点一阶导数
def evalfunc2(param):
    s=mse(X,Y,param)/RANGE
    b=0
    for j in range(3,21,3):
        b+=(E((abs(math.atan(2*param[j-3]+param[j-2])-math.atan(2*param[j])+param(j+1))-BETA))-1)/(10*math.e)
    return s+b
#考虑两种
def evalfunc3(param):
    s=mse(X,Y,param)/RANGE
    a=0
    for j in range(3,21,3):
        a+=E(abs(fittingFunc(param[j-3:j])(X[(j/3)*10])-fittingFunc(param[j:j+3])(X[(j/3)*10]))/RANGE-ALPHA)-1
    b=0
    for j in range(3,21,3):
        b+=(E((abs(math.atan(2*param[j-3]+param[j-2])-math.atan(2*param[j])+param(j+1))-BETA))-1)/(10*math.e)
    return s+a+b
#不考虑其他因素
def evalfunc(param):
    s=mse(X,Y,param)/RANGE
    return s

def draw(title):
    global X

    res=cmaUser(evalfunc)
    res1=cmaUser(evalfunc1)
    res2=cmaUser(evalfunc2)
    res3=cmaUser(evalfunc3)

    plt.figure(1)
    plt.plot(X,Y,'b.',alpha=0.6,label="measure point")
    X1=np.insert(X,0,[-5.01,-3.01,-1.01,0.99,2.99,4.99])
    X1.sort()

   ...


I am working on the file "CMA-ES/分段函数平滑拟合/分段平滑拟合.ipynb".
The first task is:
4种策略下最优评价函数值
interval         | continuity1  |continuity2    |continuity3
 ---------------- |---------------|--------------|--------------
 0.0309909258183  |0.0237448247489|0.0339220595649|0.0256780571605

最优拟合与真实曲线对比图

二次曲线分段线性平滑
构造分段函数
构造函数$f(x)=100-x^2$,在定义域$x\in [-7,7]$上均匀划分7段进行线性拟合。
构造测试数据集
Can you write Python code for it?

#构造数据集
E=math.exp
DIM=14
PI=math.pi
def linefunc2(x):
    return 100-x**2++rd.uniform(-3,3)*rd.gauss(0,4)
X=np.linspace(-7,7,301)
Y=map(linefunc2,X)
RANGE=max(Y)-min(Y)
ALPHA=0.01
BETA=PI/4.0
scatter(X,Y)


I am working on the file "CMA-ES/分段函数平滑拟合/分段平滑拟合.ipynb".
The first task is:
实验
划分七段用CMA线性拟合14个参数
Can you write Python code for it?

def realLineFunc(param):
    def f(x):
        if x<-5:
            return np.poly1d([param[0],param[1]])(x)
        elif x<-3:
            return np.poly1d([param[2],param[3]])(x)
        elif x<-1:
            return np.poly1d([param[4],param[5]])(x)
        elif x<1:
            return np.poly1d([param[6],param[7]])(x)
        elif x<3:
            return np.poly1d([param[8],param[9]])(x)
        elif x<5:
            return np.poly1d([param[10],param[11]])(x)
        else:
            return np.poly1d([param[12],param[13]])(x)
    return f
def mse(x,y,param):
    s=0
    for j in range(0,14,2):
        for i in range(int(j/2)*10,int(j/2+1)*10,1):
            s+=(y[i]-fittingFunc(param[j:j+2])(x[i]))**2
    return math.sqrt(s/200.0)
 #间断点
def evalfunc1(param):
    s=mse(X,Y,param)/RANGE
    a=0
    for j in range(2,14,2):
        a+=E(abs(fittingFunc(param[j-2:j])(X[(j/2)*10])-fittingFunc(param[j:j+2])(X[(j/2)*10]))/RANGE-ALPHA)-1
    return s+a
#间断点一阶导数
def evalfunc2(param):
    s=mse(X,Y,param)/RANGE
    b=0
    for j in range(2,14,2):
        b+=(E((abs(math.atan(param[j-2])-math.atan(param[j]))-BETA))-1)/(100*math.e)
    return s+b
#考虑两种
def evalfunc3(param):
    s=mse(X,Y,param)/RANGE
    a=0
    for j in range(2,14,2):
        a+=E(abs(fittingFunc(param[j-2:j])(X[(j/2)*10])-fittingFunc(param[j:j+2])(X[(j/2)*10]))/RANGE-ALPHA)-1
    b=0
    for j in range(2,14,2):
        b+=(E((abs(math.atan(param[j-2])-math.atan(param[j]))-BETA)/(100*math.e))-1)
    return s+a+b
#不考虑其他因素
def evalfunc(param):
    s=mse(X,Y,param)/RANGE
    return s

def draw(title):
    global X
    res=cmaUser(evalfunc)
    res1=cmaUser(evalfunc1)
    res2=cmaUser(evalfunc2)
    res3=cmaUser(evalfunc3)

    plt.figure(1)
    plt.plot(X,Y,'b.',alpha=0.6,label="measure point")
    X1=np.insert(X,0,[-5.01,-3.01,-1.01,0.99,2.99,4.99])
    X1.sort()
    plt.plot(X1,map(realLineFunc(res),X1),c="red",lw=2,ls="-",alpha=0.7,label="interval")
    plt.plot(X1,map(realLineFunc(res1),X1),c="blue",lw=2,ls="-",alpha=0.7,label="continuity1")
   ...


I am working on the file "CMA-ES/分段数对效果的影响/分段数对拟合效果的影响.ipynb".
The first task is:
cpu计算耗时与分段数关系图表
Can you write Python code for it?

b=[]
for i in range(5,35,5):
    temp=[]
    for j in range(3):
        temp.append(X[i][j]['time'])
    b.append(temp)
bs=np.array(b).T
ind=range(5,35,5)
d={'M1':pd.Series(bs[0],index=ind),
   'M2':pd.Series(bs[1],index=ind),
   'M3':pd.Series(bs[2],index=ind)}
df = pd.DataFrame(d)
df.columns.name='function'
df.index.name='partition'
df

df.plot(kind='bar',fontsize=20)
leg = plt.gca().get_legend()
ltext  = leg.get_texts()
plt.setp(ltext, fontsize='20')
plt.title("CPU time ",fontsize=16)
df.plot()
leg = plt.gca().get_legend()
ltext  = leg.get_texts()
plt.setp(ltext, fontsize='20')
plt.title("CPU time",fontsize=16)


I am working on the file "CMEMS_Folium_Visalization.ipynb".
The first task is:
Could do intersection of the list, but for that we need to specify the provider name as specified in the index file.
netCDF file name conventions
The data specifications are coded within the netCDF file name following the conventions:

File naming convention in the latest directory:

RR_LATEST_XX_YY_CODE_YYYYMMDD.nc

RR: region bigram

LATEST: fixed name

XX: TS (timeserie) or PR (profile)

YY: data type

CODE: platform code

YYYYMMDD: year month day of observations

.nc: NetCDF file name suffix

    Example: GL_LATEST_PR_GL_58970_20151112.nc

Data types

BA: data from Bathy messages on GTS

CT: CTD profiles

DB: Drifting buoys

FB: FerryBox

GL: Gliders

MO: Fixed buoys or mooring time series

PF: Profiling floats vertical profiles

RE: Recopesca

RF: River flows

TE: data from TESAC messages on GTS

TS: Thermosalinographs

XB: XBT or XCTD profiles

Region bigram

GL: Global

AR: Arctic

BO: Baltic

NO: North West Shelf

IR: IBI (Iberia-Biscay-Ireland)

MO: Mediterranean

BS: Black Sea

We convert these information to a python dictionary:
Can you write Python code for it?

regions_lut = dict()
regions_lut['GL'] = 'Global'
regions_lut['AR'] = 'Arctic'
regions_lut['BO'] = 'Baltic'
regions_lut['NO'] = 'North West Shelf'
regions_lut['IR'] = 'IBI (Iberia-Biscay-Ireland)'
regions_lut['MO'] = 'Mediterranean'
regions_lut['BS'] = 'Black Sea'
data_types_lut = dict()
data_types_lut['BA'] = 'data from Bathy messages on GTS'
data_types_lut['CT'] = 'CTD profiles'
data_types_lut['DB'] = 'Drifting buoys'
data_types_lut['FB'] = 'FerryBox'
data_types_lut['GL'] = 'Gliders'
data_types_lut['MO'] = 'Fixed buoys or mooring time series'
data_types_lut['PF'] = 'Profiling floats vertical profiles'
data_types_lut['RE'] = 'Recopesca'
data_types_lut['RF'] = 'River flows'
data_types_lut['TE'] = 'data from TESAC messages on GTS'
data_types_lut['TS'] = 'Thermosalinographs'
data_types_lut['XB'] = 'XBT or XCTD profiles'
data_specs_lut = dict()
data_specs_lut['TS'] = 'Timeseries'
data_specs_lut['PR'] = 'Profile'


I am working on the file "CMEMS_Folium_Visalization.ipynb".
The first task is:
Here we could also add something for the time or space domain.

Easy but not time to do it now.
Load and prepare data
Since the index_latest.txt is a formatted file, we use the numpy function genfromtxt to extract the data from the document.
Can you write Python code for it?

indexfile = "./index_latest.txt"
dataindex = np.genfromtxt(indexfile, skip_header=6, unpack=True, delimiter=',', dtype=None, \
              names=['catalog_id', 'file_name', 'geospatial_lat_min', 'geospatial_lat_max',
                     'geospatial_lon_min', 'geospatial_lon_max',
                     'time_coverage_start', 'time_coverage_end',
                     'provider', 'date_update', 'data_mode', 'parameters'])


I am working on the file "CMEMS_Folium_Visalization.ipynb".
The first task is:
Selection criteria
Provider
The list dataprovider contains the name of the providers we want to keep for the plot.
Can you write Python code for it?

dataproviderlist = ['IEO', 'INSTITUTO ESPANOL DE OCEANOGRAFIA', 'SOCIB']


I am working on the file "CMI-Meteorology.ipynb".
The first task is:
Advance the model by another time step:
Can you write Python code for it?


I am working on the file "CMI-Meteorology.ipynb".
The first task is:
And on line 764 of met_base.py it's applied the input value of P:
Can you write Python code for it?


I am working on the file "CMI-Meteorology.ipynb".
The first task is:
Import the Meteorology component and create an instance:
Can you write Python code for it?

from cmt.components import Meteorology

m = Meteorology()


I am working on the file "CMI-Meteorology.ipynb".
The first task is:
Locate the cfg file:
Can you write Python code for it?


I am working on the file "CMI-Meteorology.ipynb".
The first task is:
Maybe this will change after the first update?
Show the model start, current, and stop times:
Can you write Python code for it?

print m.get_start_time(), m.get_current_time(), m.get_end_time()


I am working on the file "CMI-Meteorology.ipynb".
The first task is:
Note that I set P (the precipitation rate) to 20.0 mm/hr in the cfg file. Why does Meteorology now report a different value?
Aha! On line 552 of met_base.py, the constant mmph_to_mps is defined:
Can you write Python code for it?


I am working on the file "CMSIS/DSP/PythonWrapper/examples/cmsisdsp_tests.ipynb".
The first task is:
Creating the signal
Conversion functions to use CMSIS-DSP FFTs with complex numbers
CMSIS-DSP FFTs are processing  array of complex numbers which are represented in memory asan array of floats. There is no specific data types for complex numbers.
The Python array is containing complex numbers. They need to be replaced by a sequence of real numbers.
The two functions below are doing those conversions.
Can you write Python code for it?

# Array of complex numbers as an array of real numbers
def imToReal1D(a):
    ar=np.zeros(np.array(a.shape) * 2)
    ar[0::2]=a.real
    ar[1::2]=a.imag
    return(ar)

# Array of real numbers as an array of complex numbers
def realToIm1D(ar):
    return(ar[0::2] + 1j * ar[1::2])

nb = 512
signal = None


I am working on the file "CMSIS/DSP/PythonWrapper/examples/cmsisdsp_tests.ipynb".
The first task is:
The arm_cfft_instance_q15 is created and initialized
Can you write Python code for it?

# Initialize the Q15 CFFT
cfftq15 = dsp.arm_cfft_instance_q15()
status = dsp.arm_cfft_init_q15(cfftq15,nb)
print(status)

# Compute the Q15 CFFT and convert back to float and complex array
resultR = dsp.arm_cfft_q15(cfftq15,signalRQ15,0,1)
resultR = f.Q15toF32(resultR)
resultI = realToIm1D(resultR)*nb

mag = 20 * np.log10(np.abs(resultI))
plt.plot(mag[1:nb//2])
plt.show()


I am working on the file "CMSIS/DSP/PythonWrapper/examples/cmsisdsp_tests.ipynb".
The first task is:
The log magnitude of the FFT is computed and siplayed.
Can you write Python code for it?

# Re-evaluate this each time you change the signal
signalR = imToReal1D(signal)
resultR = dsp.arm_cfft_f32(cfftf32,signalR,0,1)
resultI = realToIm1D(resultR)
mag=20 * np.log10(np.abs(resultI))

plt.plot(mag[1:nb//2])
plt.show()


I am working on the file "CMSIS/DSP/PythonWrapper/examples/cmsisdsp_tests.ipynb".
The first task is:
Using the F32 CMSIS-DSP FFT
The arm_cfft_instance_f32 is created and initialized.
Can you write Python code for it?

# CMSIS-DSP FFT F32 initialization
cfftf32=dsp.arm_cfft_instance_f32()
status=dsp.arm_cfft_init_f32(cfftf32,nb)
print(status)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/cmsisdsp_tests.ipynb".
The first task is:
Using the Q15 CMSIS-DSP FFT
The signal must be converted to Q15 each time it is changed with the slider above.
Can you write Python code for it?

# Convert the signal to Q15 and viewed as a real array
signalR = imToReal1D(signal)
signalRQ15 = f.toQ15(signalR)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/cmsisdsp_tests.ipynb".
The first task is:
You can play with the slider to change the frequency of the signal.
Don't forget to reconvert the signal to a Q15 format if you want to test the Q15 FFT.
Can you write Python code for it?

@interact(f=FloatSlider(100,min=10,max=150,step=20,continuous_update=False))
def gen_signal(f=100):
    global signal
    global nb
    signal = np.sin(2 * np.pi * np.arange(nb)*f / nb) + 0.1*np.random.randn(nb)
    plt.plot(signal)
    plt.show()


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
And finally we can check the CMSIS-DSP behavior of the test patterns:
Can you write Python code for it?

y_pred_ref = [dsp_predict(dsp_feature(x.signal)) for x in test_patterns]
labels=["Unknown"] + to_keep
ConfusionMatrixDisplay.from_predictions(y_test, y_pred_ref,display_labels=labels)

np.count_nonzero(np.equal(y_test,y_pred_ref))/len(y_test)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
And we can reload the saved model:
Can you write Python code for it?

with open("logistic.pickle","rb") as f:
    clfb=pickle.load(f)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Below code is extracting the training and test patterns.
This will be used later to generate the array used by scikit learn to train the model.
Can you write Python code for it?

print(len(patterns))
patterns=np.array(patterns)

nb_patterns = len(patterns)
nb_train= int(np.floor(0.8 * nb_patterns))
nb_tests=nb_patterns-nb_train

train_patterns = patterns[:nb_train]
test_patterns = patterns[-nb_tests:]


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Below function is :
* Defining the compute graph by connecting all the nodes
* Generating a Python implementation of the compute graph and its scheduling
* Generating a C++ implementation of the compute graph and its scheduling
* Generating a graphviz description of the graph
The feature length is hardcoded. So if the sliding window parameters are changed, you'll need to change the values for the FEATURE_LENGTH and FEATURE_OVERLAP.
Can you write Python code for it?

def gen_sched(python_code=True):
    src=Source("src",AUDIO_INTERRUPT_LENGTH)
    # For Python code, the input is a numpy array which is passed
    # as argument of the node
    if python_code:
        src.addVariableArg("input_array")
    sink=Sink("sink",1)

    feature=Feature("feature",winLength)
    feature.addVariableArg("window")

    sliding_audio=SlidingBuffer("audioWin",q15Type,winLength,winLength-audio_input_length)

    FEATURE_LENGTH=98 # for one second
    FEATURE_OVERLAP = 49 # We slide feature by 0.5 second
    sliding_feature=SlidingBuffer("featureWin",q15Type,FEATURE_LENGTH,FEATURE_OVERLAP)
    kws=KWS("kws",FEATURE_LENGTH)
    # Parameters of the ML model used by the node.
    kws.addVariableArg("coef_q15")
    kws.addVariableArg("coef_shift")
    kws.addVariableArg("intercept_q15")
    kws.addVariableArg("intercept_shift")

    fir=FIR("fir",FEATURE_LENGTH,FEATURE_LENGTH)

    # Description of the compute graph
    g = Graph()

    g.connect(src.o, sliding_audio.i)
    g.connect(sliding_audio.o, feature.i)
    g.connect(feature.o, sliding_feature.i)
    g.connect(sliding_feature.o, fir.i)
    g.connect(fir.o, kws.i)
    g.connect(kws.o, sink.i)

    # For Python we run for only around 13 seconds of input signal.
    # Without this, it would run forever.
    conf=Configuration()
    if python_code:
        conf.debugLimit=13

    # We compute the scheduling
    sched = g.computeSchedule(conf)

    print("Schedule length = %d" % sched.scheduleLength)
    print("Memory usage %d bytes" % sched.memory)

    # We generate the scheduling code for a Python and C++ implementations
    if python_code:
        conf.pyOptionalArgs="input_array,window,coef_q15,coef_shift,intercept_q15,intercept_shift"
        sched.pythoncode(".",config=conf)
        with open("test.dot","w") as f:
             sched.graphviz(f)
    else:
        conf.cOptionalArgs="""const q15_t *window,
        const q15_t *coef_q15,
        const int coef_shift,
        const q15_t intercept_q15,
        const int...


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Display of the feature to compare with the spectrogram.
Can you write Python code for it?


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Generation of the coef code:
Can you write Python code for it?


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Let's check that the feature is giving the same result as the reference implemenattion using linear algebra.
Can you write Python code for it?


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Let's check the feature on the data to compare with the F32 version and check it is working:
Can you write Python code for it?


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Let's plot the signal to check we have the right one:
Can you write Python code for it?


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Next line is generating the C++ schedule that we will need for the Arduino implementation : kws/scheduler.cpp
Can you write Python code for it?


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Now we can check the Q31 implementation on the test patterns:
Can you write Python code for it?

y_pred_ref = [dsp_predict_q31(dsp_feature_q31(fix.toQ31(x.signal))) for x in test_patterns]
labels=["Unknown"] + to_keep
ConfusionMatrixDisplay.from_predictions(y_test, y_pred_ref,display_labels=labels)

np.count_nonzero(np.equal(y_test,y_pred_ref))/len(y_test)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Now we can implement the zcr and feature in Q31.
Can you write Python code for it?

def dsp_zcr_q31(w):
    m = dsp.arm_mean_q31(w)
    # Negate can saturate so we use CMSIS-DSP function which is working on array (and we have a scalar)
    m = dsp.arm_negate_q31(np.array([m]))[0]
    w = dsp.arm_offset_q31(w,m)

    f=w[:-1]
    g=w[1:]
    k=np.count_nonzero(np.logical_and(np.logical_or(np.logical_and(f>0,g<0), np.logical_and(f<0,g>0)),g>f))

    # k < len(f) so shift should be 0 except when k == len(f)
    # When k==len(f) normally quotient is 0x40000000 and shift 1 and we convert
    # this to 0x7FFFFFF and shift 0
    status,quotient,shift_val=dsp.arm_divide_q31(k,len(f))
    if shift_val==1:
        return(dsp.arm_shift_q31(np.array([quotient]),shift)[0])
    else:
        return(quotient)

firq31 = dsp.arm_fir_instance_q31()

def dsp_feature_q31(data):
    samplerate=16000
    input_len = 16000

    waveform = data[:input_len]

    zero_padding = np.zeros(
      16000 - waveform.shape[0],
      dtype=np.int32)

    signal = np.hstack([waveform, zero_padding])

    winDuration=25e-3
    audioOffsetDuration=10e-3
    winLength=int(np.floor(samplerate*winDuration))
    audioOffset=int(np.floor(samplerate*audioOffsetDuration))
    overlap=winLength-audioOffset

    window=fix.toQ31(hann(winLength,sym=False))
    reta=[dsp_zcr_q31(dsp.arm_mult_q31(x,window)) for x in sliding_window_view(signal,winLength)[::audioOffset,:]]

    # Reset state and filter
    blockSize=98
    numTaps=10
    stateLength = numTaps + blockSize - 1
    dsp.arm_fir_init_q31(firq31,10,fix.toQ31(np.ones(10)/10.0),np.zeros(stateLength,dtype=np.int32))
    reta=dsp.arm_fir_q31(firq31,reta)
    return(np.array(reta))


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Now we can run our static schedule on this file.
The reload function are needed when debugging (or implementing) the appnodes.py. Without this, the package would not be reloaded in the notebook.
This code needs some variables evaluated in the Q15 estimator code above.
Can you write Python code for it?

import sched as s
from importlib import reload
import appnodes

appnodes= reload(appnodes)

s = reload(s)

dataQ15=fix.toQ15(data)
windowQ15=fix.toQ15(hann(winLength,sym=False))

nb,error = s.scheduler(dataQ15,windowQ15,coef_q15,coef_shift,intercept_q15,intercept_shift)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Now we'd like to test the Q15 classifier and the static schedule on a real patterns.
We are using the Yes/No pattern from our VHT-SystemModeling example.
Below code is loading the pattern into a NumPy array.
Can you write Python code for it?

from urllib.request import urlopen
import io
import soundfile as sf

test_pattern_url="https://github.com/ARM-software/VHT-SystemModeling/blob/main/EchoCanceller/sounds/yesno.wav?raw=true"
f = urlopen(test_pattern_url)
filedata = f.read()
data, samplerate = sf.read(io.BytesIO(filedata))
if len(data.shape)>1:
    data=data[:,0]


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Once the model is loaded, we extract the values and convert them to Q15:
Can you write Python code for it?

scaled_coef=clfb.best_estimator_.coef_
coef_shift=0
while np.max(np.abs(scaled_coef)) > 1:
    scaled_coef = scaled_coef / 2.0
    coef_shift = coef_shift + 1

coef_q15=fix.toQ15(scaled_coef)

scaled_intercept = clfb.best_estimator_.intercept_
intercept_shift = 0
while np.abs(scaled_intercept) > 1:
    scaled_intercept = scaled_intercept / 2.0
    intercept_shift = intercept_shift + 1

intercept_q15=fix.toQ15(scaled_intercept)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Patterns for training
Now we generate the arrays needed to train and test the model.
We have an array of feature : X_array.
An array of label ID : y
and similar arrays for the tests.
Can you write Python code for it?

X=np.array([x.feature for x in train_patterns])
X.shape

y=np.array([x.label for x in train_patterns])
y.shape

y_test = [x.label for x in test_patterns]
X_test = [x.feature for x in test_patterns]


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Reference implementation with Matrix
This is the reference implementation which will be used to build the CMSIS-DSP implementation. We are no more using the scikit-learn predict function instead we are using an implementation of predict using linear algebra. It should give the same results.
Can you write Python code for it?

def predict(feat):
    coef=clfb.best_estimator_.coef_
    intercept=clfb.best_estimator_.intercept_

    res=np.dot(coef,feat) + intercept

    if res<0:
        return(-1)
    else:
        return(0)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Testing
Below code is connecting to the Arduino board an displaying the output in a cell.
If you say Yes loudly enough (so not too far from the board) and if you don't have too much background noise in your room, then it should work.
You need to install the pyserial python package
Can you write Python code for it?

import serial
import ipywidgets as widgets
import time
import threading

STOPSERIAL=False
def stop_action(btn):
    global STOPSERIAL
    STOPSERIAL=True

out = widgets.Output(layout={'border': '1px solid black','height':'40px'})
button = widgets.Button(
    description='Stop',
    disabled=False,
    button_style='', # 'success', 'info', 'warning', 'danger' or ''
    tooltip='Click me'
)
button.on_click(stop_action)

out.clear_output()
display(widgets.VBox([out,button]))

STOPSERIAL = False

def get_serial():
    try:
        with serial.Serial('COM6', 115200, timeout=1) as ser:
            ser.reset_input_buffer()
            global STOPSERIAL
            while not STOPSERIAL:
                data=ser.readline()
                if (len(data)>0):
                    with out:
                        out.clear_output()
                        res=data.decode('ascii').rstrip()
                        if res=="Yes":
                            display(HTML("
YES
"))
                        else:
                            print(res)
            with out:
                out.clear_output()
                print("Communication closed")
    except Exception as inst:
        with out:
            out.clear_output()
            print(inst)

t = threading.Thread(target=get_serial)
t.start()


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
Testing on a signal
The following code is displaying a pattern as example.
Can you write Python code for it?

nbpat=50
data = patterns[nbpat].signal
samplerate=16000
plt.plot(data)
plt.show()

audio=Audio(data=data,rate=samplerate,autoplay=False)
audio


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
The Q31 feature is very similar to the F32 one so now we can implement the predict:
Can you write Python code for it?

def dsp_predict_q31(feat):

    res=dsp.arm_dot_prod_q31(coef_q31,feat)

    # Before adding the res and the intercept we need to ensure they are in the same Qx.x format
    # The scaling applied to the coefs and to the intercept is different so we need to scale
    # the intercept to take this into account
    scaled=dsp.arm_shift_q31(np.array([intercept_q31]),intercept_shift-coef_shift)[0]
    # Because dot prod output is in Q16.48
    # and ret is on 64 bits
    scaled = np.int64(scaled) << 17

    res = res + scaled

    if res<0:
        return(-1)
    else:
        return(0)


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
The code is working. We are getting more printed Yes than Yes in the pattern because we are sliding by 0.5 second between each recognition and the same word can be recognized several time.
Now we are ready to implement the same on an Arduino.
First we need to generate the parameters of the model. If you have saved the model, you can reload it with the code below:
Can you write Python code for it?


I am working on the file "CMSIS/DSP/PythonWrapper/examples/kws_example/kws.ipynb".
The first task is:
The confusion matrix is generated from the test patterns to check the behavior of the classifier:
Can you write Python code for it?

y_pred = clfb.predict(X_test)
labels=["Unknown"] + to_keep
ConfusionMatrixDisplay.from_predictions(y_test, y_pred,display_labels=labels)