I am working on the file "LHCb_Open_Data_Project.ipynb".
The first task is:
Hints
Histogram plotting - You can use the hist() function. The parameters bins(n) and range(x,y) allow youto plot n bins over the range x to y.
Vector Magnitude The square magnitude of a magnitude of a vector is given by the sum of the square of its of its components in the x,y and z directions: $p^2 = p_x^2+p_y^2+p_z^2$, where $p$ is the magnitude of the momentum, and $p_x,p_y,p_z$ are the components of the momentum in the X,Y, and Z directions.
Energy and mass
Einstein's theory of special relativity relates Energy, mass and momentum. We have measured the momentum of the kaon candidates in the detector, and have just plotted one of the components of the momentum of the kaon, and the magnitude of the momentum. The invariant mass of the kaon is well known and you can look this up. We wish to determine the energy of the kaons.
Here is a brief guide to the energy-momentum relation of special relativity. Further information can be found on wikipedia pages on Invariant Mass  and the Energy-momentum relation.
Now, calculate the energy of the first kaon candidate using:
<center> $E^2 = p^2 + m^2$ </center>
Can you write Python code for it?

# calculate the energy of the first kaon

# plot a histogram of this variable


I am working on the file "LHCb_Open_Data_Project.ipynb".
The first task is:
In order to calculate the Asymmetry, you can make use of the formula:
(note you may need to run this box in order to see the image)
<img src="Images/AsymmetryEq.png" width="200" />
Can you write Python code for it?

# calculate the value of the asymmetry, by using the formula above, and then print it


I am working on the file "LHCb_Open_Data_Project.ipynb".
The first task is:
Make histograms of the probability of a final state particle being a kaon or a pion.
These will help guide you on suitable probability values at which to cut.
You can also consider more sophisticated options like 2-D plots of kaon and pion probabilities or different values of the cuts for the different final state particles.
Can you write Python code for it?

# plot the probability that a final state particle is a kaon

# plot the probability that a final state particle is a pion


I am working on the file "LHCb_Open_Data_Project.ipynb".
The first task is:
Now calculate the invariant mass of the B meson for the real data and plot a histogram of this.
Compare it with the one you drew for the simulation data.
Can you explain the differences you observe?
Can you write Python code for it?

# draw a histogram for the B meson mass in the real data


I am working on the file "LHCb_Open_Data_Project.ipynb".
The first task is:
Now count the numbers of events of each of the two types (N+ and N-). Also calculate the difference between these two numbers.
Can you write Python code for it?

# make variables for the numbers of positive and negative B mesons


I am working on the file "LHCb_Open_Data_Project.ipynb".
The first task is:
Observing CP violation
From your studies of the asymmetry plot, and the plot of its significance, you will be able to identify regions in the Dalitz plots that show indications of sizeable and significant CP Violation. You may find you have several consecutive bins with significant positive, or negative, asymmetries. You may wish to try alternative binnings of the Dalitz plots to best isolate the regions in which the significant asymmetries occur.
You can select events that are in these regions of the Dalitz plots where you observe signs of CP Violation. You can then plot a simple 1D histogram of the invariant mass distribution of the B+ and the B- events, just as you did at the start of the project, but only for events that lie in the region of the Dalitz plot that you are interested in. Make the plots of the B+ and the B- events with the same scale, or superimpose the two plots, so that you can observe if the particle and anti-particle decay processes are occurring at the same rate.
Can you write Python code for it?

# Make a plot showing the invariant mass of the B+ meson particles
# using events from a region of the Dalitz plot showing sizeable CP asymmetries

# Make a plot showing the invariant mass of the B- meson particles using events from the same region


I am working on the file "LHCb_Open_Data_Project.ipynb".
The first task is:
Observing a large asymmetry in some regions of the plot does not necessarily mean you have observed CP violation. If there are very few events in that region of the plot the uncertainty on that large asymmetry may be large. Hence, the value may still be compatible with zero.
You can calculate the statistical uncertainty on the asymmetry, for each bin of the plot, using the same formulas as you used in the global asymmetry section. You can then make a plot showing the uncertainty on the asymmetry.
Dividing the plot showing the asymmetry by the plot showing the statistical uncertainty you can then obtain the significance of the asymmetry in each bin. You can then plot the significance of the asymmetry to see if there is any evidence for CP violation.
Can you write Python code for it?

# Make a plot showing the uncertainty on the asymmetry

# Make a plot showing the statistical significance of the asymmetry


I am working on the file "LHCb_Open_Data_Project.ipynb".
The first task is:
Searching for global matter anti-matter differences
In this section you will start to study matter antimatter differences (CP Violation). Here 'global' means that you are looking for differences across all ranges of energy and momentum (the kinematics) of the kaons into which the charge B mesons have decayed. Later we look at 'local' differences in different regions of the kinematics.
Aims:

Calculate the global CP asymmetry
Work out the statistical uncertainty
Determine if there is evidence for CP violation in this decay

In order to quantify the matter antimatter asymmetry in this process we wish to compare the B+ and the B- particles. The B- is the anti-particle of the B+.
How can you distinguish between events that contain B+ and B- particles using H1_Charge, H2_Charge and H3_Charge?
Can you write Python code for it?

# make a variable for the charge of the B mesons


I am working on the file "LHCb_Open_Data_Project.ipynb".
The first task is:
This next line of code just loads the real data into a new DataFrame, this may take a few minutes.
It also applies the preselection that you have created if you called it preselection.
Can you write Python code for it?

real_data = read_root(['Data/B2HHH_MagnetDown.root', 'Data/B2HHH_MagnetUp.root'], where=preselection)


I am working on the file "LHCb_Open_Data_Project.ipynb".
The first task is:
Two body resonances
You can now use your Dalitz plot to identify the intermediate resonances that you see in your plots. The resonances will have shown up as bands of higher density of points on the plots. You can use the particle data group tables of mesons to identify which particles these correspond to. The tables give the masses and widths of the particles and their decay modes. You are looking for mesons with the masses corresponding to where you see the bands and that decay into K+K-.
Congratulations! You have succesfully made a Dalitz plot and used it to observe the presence of intermediate particles in the decay of your charged B meson into three charged kaons.
Searching for local matter anti-matter differences
Aims:

Observe matter antimatter differences (CP violation) in regions of the Dalitz plots of the B+ and B- mesons.
For the data in these regions produce plots to best display the CP violation.

In a section above you searched for global CP violation. You probably did not find a result with very high significance.
CP violation may arise from interference between decays through different resonances, and hence the magnitude and sign of the CP violation may vary across the Dalitz plot. We can apply the same equation as in the global CP violation study

but apply this only to events in particular regions of the Dalitz plot.
Removing charm resonances
The analysis performed here is to study the CP violation in the charmless B meson decays to kaons. "charmless" means that the decay does not proceed through a charm quark. However, the most frequent decay of the B mesons occur through the b quark decaying into a c quark. The majority of these events can be removed by rejecting the events that are proceeding through a D0 meson (which contains the charm quark).
In the section above you plotted a histogram of the invariant mass of the intermediate resonances and will have observed the D0...
Can you write Python code for it?

# make a Dalitz plot for the B+ events

# make a Dalitz plot for the B- events

# Make a plot showing the asymmetry between these two Daltz plots
# i.e. calculate the asymmetry between each bin of the B+ and B- Dalitz plots and show the result in another 2D plot


I am working on the file "LHCb_Open_Data_Project.ipynb".
The first task is:
You should have a graph that sharply peaks at the mass of the B+ meson. The mass of the B+ and B- meson are the same. Check that the peak of your graph is at the known mass of the B meson. Congratulations!
Recall that you have made this plot for simulated data. How might you expect the plots for real data to look different ? In the next section you will start to work with the real LHC data.
Hint
B Meson Energy - From energy conservation, the energy of the B meson will be the sum of the energies of the three kaons: $E_B=E_{K1}+E_{K2}+E_{K3}$, where $E_B$ is the energy of the B meson, $E_{K1}, E_{K2}, E_{K3}$ are the energies of each of the kaons.
B meson momentum - From momentum conservation, the X component of the momentum of the B meson will be the sum of the X momentum components of the three Kaons : $px_B=px_{K1}+px_{K2}+px_{K3}$, where $px$ is the X direction component of the momentum of the B meson, $px_{K1},px_{K2},px_{K3}$ are the X direction components of the momenta of the three kaons. You can then do the same with the Y and Z components. Having obtained the X,Y, and z components of the B momentum you can find the magnitude of the momentum of the B meson.
 B meson invariant mass* - Rearrange the equation $E^2=p^2+m^2$ to find $m^2$. Using the values of the magnitude of the momentum of the B meson and the B meson Energy, find the mass of the B meson.
Histogram plotting - Take care that the range of your mass plot is set suitably that you can see the mass peak. Once you have found the peak you can set the range appropriately. You do not have to start your graph at a mass of 0.
Units - The data you are provided has energies in 'MeV' (106 electron volts). The mass of the B meson is often quoted in 'GeV/c2' (109 electron volts).
Working with real data and applying cuts
Aims:

Filter out data that is not from the B+ → K+K+K− channel, or the antiparticle equivalent B- →...
Can you write Python code for it?

# Make your preselection here, this line applies no preselection
preselection = ""


I am working on the file "LMAdetection_efficiency.ipynb".
The first task is:
Converting and checking station locations
Can you write Python code for it?

center = (np.mean(aves[:,1]), np.mean(aves[:,2]), np.mean(aves[:,0]))
geo  = GeographicSystem()
tanp = TangentPlaneCartesianSystem(center[0], center[1], center[2])
mapp = MapProjection
projl = MapProjection(projection='laea', lat_0=center[0], lon_0=center[1])

alt, lat, lon  = aves[:,:3].T

plt.scatter(lon, lat, c=aves[:,3])
plt.colorbar(label='Station Threshold (dBm)')
plt.show()


I am working on the file "LMAdetection_efficiency.ipynb".
The first task is:
Detection efficiency plots
Can you write Python code for it?

domain = 197.5*1000
maps = Basemap(projection='laea', resolution='i',
               lat_0=center[0], lon_0=center[1], width=domain*2, height=domain*2)
x, y = maps(lonp, latp)

# Source detection efficiency
s = plt.pcolormesh(x,y,sde,cmap = 'magma')
plt.colorbar(label='Source Detection Efficiency')
s.set_clim(vmin=0,vmax=100)

# Draw flash detection efficiency contours
CS = plt.contour(x,y,fde_a, colors='k',levels=(20,40,60,70,80,85,90,95,99))
plt.clabel(CS, inline=1, fontsize=10,fmt='%3.0f')

# Overlay station locations
xs, ys = maps(lon,lat)
plt.scatter(xs,ys, color='k',s=5)

maps.drawstates()
maps.drawcoastlines()
# maps.drawcounties()
plt.show()


I am working on the file "LMAdetection_efficiency.ipynb".
The first task is:
Inclusive function for detection calculations

Make sure the input array of station information matches the given dimensions
Will check for solution in the line of sight for each station within 300 km of the network at the chosen altitude (default = 7 km) and grid spacing (default = 5 km)
Minimum number of stations required to participate in solutions can be set (default = 6)
Can you write Python code for it?

latp, lonp, sde, fde_a, minp = pf.quick_method(
                 # input array must be in N x (lat, lon, alt, threshold)
                 np.array([aves[:,1],aves[:,2],aves[:,0],aves[:,3]]).transpose(),
                 sq, fde,
                 xint=5000, # Grid spacing
                 altitude=7000, # Altitude of grid MSL
                 station_requirement=6, # Minimum number of stations required to trigger
                )


I am working on the file "LMAdetection_efficiency.ipynb".
The first task is:
Station coordinates from csv file
Input network title and csv file here
Can you write Python code for it?

Network = 'grid_LMA' # name of network in the csv file

# network csv file with one or multiple networks
stations = pd.read_csv('network.csv')
aves = np.array(stations.set_index('network').loc[Network])[:,:-1].astype('float')


I am working on the file "LMAsimulation_ellipse.ipynb".
The first task is:
Detection efficiency
Can you write Python code for it?

# Currently hard-coded to calculate over a 300 x 300 km grid around the network
latp, lonp, sde, fde_a, minp = pf.quick_method(
                 # input array must be in N x (lat, lon, alt, threshold)
                 np.array([aves[:,1],aves[:,2],aves[:,0],aves[:,3]]).transpose(),
                 sq, fde,
                 xint=5000, # Grid spacing
                 altitude=7000, # Altitude of grid MSL
                 station_requirement=6, # Minimum number of stations required to trigger
                 mindist = 300000 # Grid ends 300 km from the most distant station in each direction
                )


I am working on the file "LMAsimulation_ellipse.ipynb".
The first task is:
General calculations at grid points
Set number of iterations and solution requirements here (minimum number of contributing stations, maximum reduced chi squared value)
This fuction will return the dimensions of the covariance ellipses for solutions at each point at 'ntsd' standard deviations in the 'evalues' array (width (m), height (m), angle) and the standard deviation of altitude solution in the 'svalues' array (m)
If a source is not sampled by enough stations for a solution a RuntimeWarning will be generated, but this will not negatively impact the following calculations
Can you write Python code for it?

iterations=500

evalues = np.zeros((np.shape(points2)[0],3))
svalues = np.zeros((np.shape(points2)[0],1))

# # for r,theta,z errors and standard deviations and overall detection efficiency
for i in range(len(x)):
    evalues[i],svalues[i]= se.black_boxtesting(points2[i,0], points2[i,1], points2[i,2], iterations,
             stations_local,ordered_threshs,stations_ecef,center_ecef,
             tanp_all,
             c0,dt_rms,tanp,projl,
             chi2_filter=5.,min_stations=6,ntsd=3
             )


I am working on the file "LMAsimulation_ellipse.ipynb".
The first task is:
Setting up and checking station locations
Can you write Python code for it?

center = (np.mean(aves[:,1]), np.mean(aves[:,2]), np.mean(aves[:,0]))
geo  = GeographicSystem()
tanp = TangentPlaneCartesianSystem(center[0], center[1], center[2])
mapp = MapProjection
projl = MapProjection(projection='laea', lat_0=center[0], lon_0=center[1])

alt, lat, lon  = aves[:,:3].T
stations_ecef  = np.array(geo.toECEF(lon, lat, alt)).T
stations_local = tanp.toLocal(stations_ecef.T).T

center_ecef = np.array(geo.toECEF(center[1],center[0],center[2]))
ordered_threshs = aves[:,-1]

plt.scatter(stations_local[:,0]/1000., stations_local[:,1]/1000., c=aves[:,3])
plt.colorbar()

circle=plt.Circle((0,0),30,color='k',fill=False)
# plt.xlim(-80,80)
# plt.ylim(-80,80)
# fig = plt.gcf()
# fig.gca().add_artist(circle)
plt.show()


I am working on the file "LMAsimulation_ellipse.ipynb".
The first task is:
Setting up grid
Input desired grid boundaries and interval here in meters from the center of the network (no point located over the center!)
Can you write Python code for it?

xmin, xmax, xint = -300001, 299999, 20000
ymin, ymax, yint = -300001, 299999, 20000
# alts = np.arange(500,20500,500.)
alts = np.array([7000])

initial_points = np.array(np.meshgrid(np.arange(xmin,xmax+xint,xint),
                                      np.arange(ymin,ymax+yint,yint), alts))

x,y,z=initial_points.reshape((3,int(np.size(initial_points)/3)))
points2 = tanp.toLocal(np.array(projl.toECEF(x,y,z))).T

tanp_all = []
for i in range(len(aves[:,0])):
    tanp_all = tanp_all + [TangentPlaneCartesianSystem(aves[i,1],aves[i,2],aves[i,0])]


I am working on the file "LMAsimulation_ellipse.ipynb".
The first task is:
Station coordinates and thresholds from a set of log files
Specify:

start time
end time
the directory holding the log files
any stations you wish to exclude from the analysis
Can you write Python code for it?

# import os

# # start_time = datetime.datetime(2014,5,26,2) #25 set
# # end_time   = datetime.datetime(2014,5,26,3,50)
# useddir = '/Users/Vanna/Documents/logs/'
# exclude = np.array(['W','A',])

# days = np.array([start_time+datetime.timedelta(days=i) for i in range((end_time-start_time).days+1)])
# days_string = np.array([i.strftime("%y%m%d") for i in days])

# logs = pd.DataFrame()
# dir = os.listdir(useddir)
# for file in dir:
#     if np.any(file[2:] == days_string) & np.all(exclude!=file[1]):
#         print file
#         logs = logs.combine_first(read_logs.parsing(useddir+file,T_set='True'))

# aves = logs[start_time:end_time].mean()
# aves = np.array(aves).reshape(4,len(aves)/4).T


I am working on the file "LMAsimulation_full.ipynb".
The first task is:
Detection efficiency
Can you write Python code for it?

# iterations=100 # Set if reading in from a file

xs  = 1000./np.arange(10,1000,1.) # Theoretical source detection efficiency that corresponds with fde
fde = 100-np.load('fde.csv',fix_imports=True, encoding='latin1') # Corresponding flash DE
sde = 100-np.mean(misses[0,:,:,:], axis=2)*100./iterations # Calculated source detection efficiency
fde_a = np.empty_like(sde)

selects = sde == 100. # Put into the next lowest or equivalent flash DE from given source DE
fde_a[selects] = 100.
for i in range(len(xs)-1):
    selects = (sde >= xs[1+i]) & (sde < xs[i])
    fde_a[selects] = fde[i]

# Find center of 95% SOURCE detection efficiency
goods = (100-np.mean(misses[0,:,:,:], axis=2)*100./iterations)>95.
de_centery = np.mean(initial_points[1,:,:,0][goods])
de_centerx = np.mean(initial_points[0,:,:,0][goods])
print ("DE center location in km: ", de_centerx/1000., ", ", de_centery/1000.)

domain = 197.5*1000 # Relates back to domain size (xmax - xint/2) to shift map
maps = Basemap(projection='laea', lat_0=center[0], lon_0=center[1], width=domain*2, height=domain*2)
s = plt.pcolormesh(np.arange(xmin-xint/2.,xmax+3*xint/2.,xint)+domain,
                   np.arange(ymin-yint/2.,ymax+3*yint/2.,yint)+domain,
                   100-np.mean(misses[0,:,:,:], axis=2)*100./iterations,
                  cmap = 'jet_r')
s.set_clim(vmin=0,vmax=100)
plt.colorbar(label='Source Detection Efficiency')
CS = plt.contour(np.arange(xmin,xmax+xint,xint)+domain,
                   np.arange(ymin,ymax+yint,yint)+domain,
                   fde_a, colors='k',levels=(20,40,60,70,80,85,90,95,99))
plt.clabel(CS, inline=1, fontsize=10,fmt='%3.0f')
plt.scatter(stations_local[:,0]+domain, stations_local[:,1]+domain, color='k')
# plt.scatter(np.array([de_centerx+domain]), np.array([de_centery+domain]), color='r')

maps.drawstates()
plt.tight_layout()
plt.show()


I am working on the file "LMAsimulation_full.ipynb".
The first task is:
Flash Distortion
Can you write Python code for it?

# # Full Levels
# sigmar = np.mean(stds[0,:,:,9:30],axis=2)
# sigmaa = np.mean(stds[1,:,:,9:30],axis=2)
# sde = 100-np.mean(misses[0,:,:,9:30], axis=2)*100./iterations # Calculated source detection efficiency

# One Level
sigmar = (stds[0,:,:,0])
sigmaa = (stds[1,:,:,0])
sde    = (1-misses[0,:,:,0]/iterations)*100

xf = np.arange(xmin,xmax+xint,xint)/1000.
yf = np.arange(ymin,ymax+yint,yint)/1000.
xy = np.meshgrid(xf,yf)
ranges = (xy[0]**2+xy[1]**2)**0.5*1000.

fl_areas,fl_numbers = np.load('typical_flashes.csv',fix_imports=True, encoding='latin1')

areas = np.empty_like(sde)
numbers = np.empty_like(sde)

selects = sde == 100. # Put into the next lowest or equivalent flash DE from given source DE
areas[selects] = fl_areas[0]
numbers[selects] = fl_numbers[0]
for i in range(len(xs)-1):
    selects = (sde >= xs[1+i]) & (sde < xs[i])
    areas[selects] = fl_areas[i]
    numbers[selects] = fl_numbers[i]

domain = (xmax-xint/2.)
maps = Basemap(projection='laea',lat_0=center[0],lon_0=center[1],width=domain*2,height=domain*2)
s = plt.pcolormesh(np.arange(xmin-xint/2.,xmax+3*xint/2.,xint)+domain,
                   np.arange(ymin-yint/2.,ymax+3*yint/2.,yint)+domain,
                   (areas), cmap = 'viridis_r')
s.set_clim(vmin=18,vmax=50)
plt.colorbar()
plt.scatter(stations_local[:,0]+domain, stations_local[:,1]+domain, color='k', s=0.5)
maps.drawstates()
circle=plt.Circle((domain,domain),100000,color='k',fill=False)
fig = plt.gcf()
fig.gca().add_artist(circle)
circle=plt.Circle((domain,domain),200000,color='k',fill=False)
fig.gca().add_artist(circle)
plt.title('Typical Flash area')

plt.tight_layout()
plt.show()

major_axis_est = norm.ppf(0.975**(1./(numbers*sde/100.)))*sigmar*2+2000*(areas/np.pi)**0.5

domain = (xmax-xint/2.)
maps = Basemap(projection='laea',lat_0=center[0],lon_0=center[1],width=domain*2,height=domain*2)
s = plt.pcolormesh(np.arange(xmin-xint/2.,xmax+3*xint/2.,xint)+domain,
                   np.arange(ymin-yint/2.,ymax+3*yint/2.,yint)+domain,
                  ...


I am working on the file "LMAsimulation_full.ipynb".
The first task is:
Mean Altitude Error
Can you write Python code for it?

sf.mapped_plot(np.mean(means[2,:,:,:],axis=2)/1000.,-1,1,'PuOr',stations_local,xmin,xmax,xint,ymin,ymax,yint,center)
plt.title('Average Error')
plt.show()

sf.mapped_plot(np.mean(stds[2,:,:,:],axis=2)/1000.,0,7,'rainbow',stations_local,xmin,xmax,xint,ymin,ymax,yint,center)
plt.title('Standard Deviation')
plt.show()


I am working on the file "LMAsimulation_full.ipynb".
The first task is:
Reading/writing the arrays from/to a file
Can you write Python code for it?

case_name = 'WTLMA_ave_6_1000'

# misses.dump('cases/%s/error_misses.csv' %(case_name))
# stds.dump(  'cases/%s/error_stds.csv'   %(case_name))
# means.dump( 'cases/%s/error_means.csv'  %(case_name))

stds   = np.load('cases/%s/error_stds.csv'   %(case_name),fix_imports=True, encoding='latin1')
means  = np.load('cases/%s/error_means.csv'  %(case_name),fix_imports=True, encoding='latin1')
misses = np.load('cases/%s/error_misses.csv' %(case_name),fix_imports=True, encoding='latin1')


I am working on the file "LMAsimulation_full.ipynb".
The first task is:
Setting up and checking station locations
Can you write Python code for it?


I am working on the file "LMAsimulation_full.ipynb".
The first task is:
Setting up grid
Input desired grid boundaries and interval here in meters from the center of the network (no point located over the center!)
Can you write Python code for it?

xmin, xmax, xint = -200001, 199999, 5000
ymin, ymax, yint = -200001, 199999, 5000
# alts = np.arange(500,20500,500.)
alts = np.array([7000])

initial_points = np.array(np.meshgrid(np.arange(xmin,xmax+xint,xint),
                                      np.arange(ymin,ymax+yint,yint), alts))

x,y,z=initial_points.reshape((3,int(np.size(initial_points)/3)))
points2 = tanp.toLocal(np.array(projl.toECEF(x,y,z))).T

means = np.empty(np.shape(points2))
stds  = np.empty(np.shape(points2))
misses= np.empty(np.shape(points2))

tanp_all = []
for i in range(len(aves[:,0])):
    tanp_all = tanp_all + [TangentPlaneCartesianSystem(aves[i,1],aves[i,2],aves[i,0])]


I am working on the file "LMAsimulation_full.ipynb".
The first task is:
Station coordinates and thresholds from a set of log files
Specify:

start time
end time
the directory holding the log files
any stations you wish to exclude from the analysis
Can you write Python code for it?


I am working on the file "LMBTR.ipynb".
The first task is:
Plotting
We will now plot all the tensors, in the same plot for K2, and K3.
Can you write Python code for it?

# Plot K2
x2 = mbtr.get_k2_axis() # this is the x axis of the histogram

# create some dictionaries to make atom Z <-- type index --> type name
imap = mbtr.index_to_atomic_number
smap = {}
for index, number in imap.items():
    smap[index] = ase.data.chemical_symbols[number]

# make the plots
for i in range(1, mbtr.n_elements): # avoid showing type 0 = X (the central atom)

    # this is the slice of the flattened MBTR tensor that contains the histogram
    # for X-type_i - X is the central atom of the LMBTR expansion
    slc = mbtr.get_location(('X',smap[i]))

    # this is the slice
    y2 = desc[0][slc]

    mpl.plot(x2, y2, label="{}".format(smap[i]))

mpl.ylabel("$\phi$ (arbitrary units)", size=14)
mpl.xlabel("Distance (angstrom)", size=14)
mpl.title("Distance distribution", size=20)
mpl.legend()
mpl.show()

# Plot K3
x3 = mbtr.get_k3_axis()

for i in range(1, mbtr.n_elements):
    for j in range(1, mbtr.n_elements):
        if j <= i:
            slc = mbtr.get_location(('X',smap[i],smap[j]))
            mpl.plot(x3, desc[0][slc], label="{}, {}".format(smap[i], smap[j]))

mpl.xlim(left=-2)
mpl.ylabel("$\phi$ (arbitrary units)", size=14)
mpl.xlabel("cos(angle)", size=14)
mpl.title("Angle distribution", size=20)
mpl.legend(loc=3)
mpl.show()


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 2.ipynb".
The first task is:
PuLP supports open source linear programming solvers such as CBC and GLPK, as well as commercial solvers such as Gurobi and IBM's CPLEX.
The default solver is CBC, which comes packaged with PuLP upon installation.
For most applications, the open source CBC from COIN-OR will be enough for most simple linear programming optimisation algorithms.
Can you write Python code for it?


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 2.ipynb".
The first task is:
The objective function and constraints are added using the += operator to our model.
The objective function is added first, then the individual constraints.
Can you write Python code for it?

# Objective function
my_lp_problem += 4 * x + 3 * y, "Z"

# Constraints
my_lp_problem += 2 * y <= 25 - x
my_lp_problem += 4 * y >= 2 * x - 8
my_lp_problem += y <= 2 * x - 5


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 2.ipynb".
The first task is:
We have also checked the status of the solver, there are 5 status codes:
* Not Solved: Status prior to solving the problem.
* Optimal: An optimal solution has been found.
* Infeasible: There are no feasible solutions (e.g. if you set the constraints x <= 1 and x >=2).
* Unbounded: The constraints are not bounded, maximising the solution will tend towards infinity (e.g. if the only constraint was x >= 3).
* Undefined: The optimal solution may exist but may not have been found.
We can now view our maximal variable values and the maximum value of Z.
We can use the varValue method to retrieve the values of our variables x and y, and the pulp.value function to view the maximum value of the objective function.
Can you write Python code for it?

for variable in my_lp_problem.variables():
    print "{} = {}".format(variable.name, variable.varValue)

print pulp.value(my_lp_problem.objective)


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 2.ipynb".
The first task is:
We have now constructed our problem and can have a look at it.
Can you write Python code for it?


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 4.ipynb".
The first task is:
Here we have 6 decision variables, we could name them individually but this wouldn't scale up if we had hundreds/thousands of variables (you don't want to be entering all of these by hand multiple times).
We'll create a couple of lists from which we can create tuple indices.
Can you write Python code for it?

# Construct our decision variable lists
sausage_types = ['economy', 'premium']
ingredients = ['pork', 'wheat', 'starch']


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 4.ipynb".
The first task is:
Now we add our constraints, bear in mind again here how the use of list comprehensions allows for scaling up to many ingredients or sausage types
Can you write Python code for it?

# Constraints
# 350 economy and 500 premium sausages at 0.05 kg
model += pulp.lpSum([ing_weight['economy', j] for j in ingredients]) == 350 * 0.05
model += pulp.lpSum([ing_weight['premium', j] for j in ingredients]) == 500 * 0.05

# Economy has >= 40% pork, premium >= 60% pork
model += ing_weight['economy', 'pork'] >= (
    0.4 * pulp.lpSum([ing_weight['economy', j] for j in ingredients]))

model += ing_weight['premium', 'pork'] >= (
    0.6 * pulp.lpSum([ing_weight['premium', j] for j in ingredients]))

# Sausages must be <= 25% starch
model += ing_weight['economy', 'starch'] <= (
    0.25 * pulp.lpSum([ing_weight['economy', j] for j in ingredients]))

model += ing_weight['premium', 'starch'] <= (
    0.25 * pulp.lpSum([ing_weight['premium', j] for j in ingredients]))

# We have at most 30 kg of pork, 20 kg of wheat and 17 kg of starch available
model += pulp.lpSum([ing_weight[i, 'pork'] for i in sausage_types]) <= 30
model += pulp.lpSum([ing_weight[i, 'wheat'] for i in sausage_types]) <= 20
model += pulp.lpSum([ing_weight[i, 'starch'] for i in sausage_types]) <= 17

# We have at least 23 kg of pork to use up
model += pulp.lpSum([ing_weight[i, 'pork'] for i in sausage_types]) >= 23

# Solve our problem
model.solve()
pulp.LpStatus[model.status]

for var in ing_weight:
    var_value = ing_weight[var].varValue
    print("The weight of {0} in {1} sausages is {2} kg".format(var[1], var[0], var_value))

total_cost = pulp.value(model.objective)

print("The total cost is €{} for 350 economy sausages and 500 premium sausages".format(round(total_cost, 2)))


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
An issue we run into here is that in linear programming we can't use conditional constraints.
For example we can't add to our model that if the factory is off factory status must be 0, and if it is on factory status must be 1. Before we've solved our model though, we don't know if the factory will be on or off in a given month.
In this case construct constraints that have minimum and maximum capacities that are constant variables, which we multiply by the factory status.
Now, either factory status is 0 and:
* min_production >= 0
* max_production <= 0
Or factory status is 1 and:
* min_production >= min_capacity
* max_production <= max_capacity
(In some cases we can use linear constraints to model conditional statements, we'll explore this in part 6)
Can you write Python code for it?

# Production in any month must be between minimum and maximum capacity, or zero.
for month, factory in factories.index:
    min_production = factories.loc[(month, factory), 'Min_Capacity']
    max_production = factories.loc[(month, factory), 'Max_Capacity']
    model += production[(month, factory)] >= min_production * factory_status[month, factory]
    model += production[(month, factory)] <= max_production * factory_status[month, factory]
    #model += production[(month, factory)] >= min_production
    #model += production[(month, factory)] <= max_production

# Factory B is off in May
model += factory_status[5, 'B'] == 0
model += production[5, 'B'] == 0


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
As we have fixed costs and variable costs, we'll need to model both production and the status of the factory i.e. whether it is on or off.
Production is modelled as an integer variable.
We have a value for production for each month for each factory, this is given by the tuples of our multi-index pandas DataFrame index.
Can you write Python code for it?

production = pulp.LpVariable.dicts("production",
                                     ((month, factory) for month, factory in factories.index),
                                     lowBound=0,
                                     cat='Integer')

production


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
Factory status is modelled as a binary variable. It will have a value of 1 if the factory is on and a value of 0 when the factory is off.
Binary variables are the same as integer variables but constrained to be >= 0 and <=1
Again this has a value for each month for each factory, again given by the index of our DataFrame
Can you write Python code for it?

factory_status = pulp.LpVariable.dicts("factory_status",
                                     ((month, factory) for month, factory in factories.index),
                                     cat='Binary')


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
In our objective function we include our 2 costs:
* Our variable costs is the product of the variable costs per unit and production
* Our fixed costs is the factory status - 1 (on) or 0 (off) - multiplied by the fixed cost of production
Can you write Python code for it?

model += pulp.lpSum(
    [production[month, factory] * factories.loc[(month, factory), 'Variable_Costs'] for month, factory in factories.index]
    + [factory_status[month, factory] * factories.loc[(month, factory), 'Fixed_Costs'] for month, factory in factories.index]
)


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
Let's take a look at the optimal production schedule output for each month from each factory. For ease of viewing we'll output the data to a pandas DataFrame.
Can you write Python code for it?

x=production[1, 'A']
type(x)

output = []
for month, factory in production:
    var_output = {
        'Month': month,
        'Factory': factory,
        'Production': production[(month, factory)].varValue,
        'Factory Status': factory_status[(month, factory)].varValue
    }
    output.append(var_output)
output_df = pd.DataFrame.from_records(output).sort_values(['Month', 'Factory'])
output_df.set_index(['Month', 'Factory'], inplace=True)
output_df


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
Notice above that the factory status is 0 when not producing and 1 when it is producing
Can you write Python code for it?

# Print our objective function value (Total Costs)
print(pulp.value(model.objective))


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
We build up our constraints
Can you write Python code for it?

# Production in any month must be equal to demand
months = demand.index
for month in months:
    model += production[(month, 'A')] + production[(month, 'B')] == demand.loc[month, 'Demand']


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
We then solve the model
Can you write Python code for it?


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 6.ipynb".
The first task is:
But now we want to add in our constraints for switching on.
A factory switches on if:
* It is off in the previous month (m-1)
* AND it on in the current month (m).
As we don't know if the factory is on before month 0, we'll assume that the factory has switched on if it is on in month 1.
Can you write Python code for it?

for month, factory in factories.index:
    # In month 1, if the factory ison, we assume it turned on
    if month == 1:
        model += switch_on[month, factory] == factory_status[month, factory]

    # In other months, if the factory is on in the current month AND off in the previous month, switch on = 1
    else:
        model += switch_on[month, factory] >= factory_status[month, factory] - factory_status[month-1, factory]
        model += switch_on[month, factory] <= 1 - factory_status[month-1, factory]
        model += switch_on[month, factory] <= factory_status[month, factory]


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 6.ipynb".
The first task is:
Logical 'OR' operator
This is all well and good for the 'and' logical operator. What about the 'or' logical operator.
If we would like the following:
python
if x1 == 1 or x2 == 1:
    y1 == 1
else:
    y1 == 0
We can use the following linear constraints:
python
y1 <= x1 + x2
y1 * 2 >= x1 + x2
So that:
* if x1 is 1 and x2 is 1:
  * y1 ≤ 2
  * 2y1 ≥ 2
  * y1 must equal 1
* if x1 is 1 and x2 is 0:
  * y1 ≤ 1
  * 2y1 ≥ 1
  * y1 must equal 1
* if x1 is 0 and x2 is 1:
  * y1 ≤ 1
  * 2y1 ≥ 1
  * y1 must equal 1
* if x1 is 0 and x2 is 0:
  * y1 ≤ 0
  * 2y1 ≥ 0
  * y1 must equal 0
Again, we'll consider the alternative option:
python
if x1 == 0 or x2 == 0:
    y1 == 1
else:
    y1 == 0
We can use the following linear constraints:
python
y1 * 2 <= 2 - (x1 + x2)
y1 >= 1 - (x1 + x2)
An Example - Scheduling Example Extended
In our last example, we explored the scheduling of 2 factories.
Both factories had 2 costs:
* Fixed Costs - Costs incurred while the factory is running
* Variable Costs - Cost per unit of production
We're going to introduce a third cost - Start up cost.
This will be a cost incurred by turning on the machines at one of the factories.
In this example, our start-up costs will be:
* Factory A - €20,000
* Factory B - €400,000
Let's start by reminding ourselves of the input data.
Can you write Python code for it?

import pandas as pd
import pulp

factories = pd.DataFrame.from_csv('csv/factory_variables.csv', index_col=['Month', 'Factory'])
factories

demand = pd.DataFrame.from_csv('csv/monthly_demand.csv', index_col=['Month'])
demand


I am working on the file "LP/Introduction-to-linear-programming/Introduction to Linear Programming with Python - Part 6.ipynb".
The first task is:
We'll then solve our model
Can you write Python code for it?

model.solve()
pulp.LpStatus[model.status]

output = []
for month, factory in production:
    var_output = {
        'Month': month,
        'Factory': factory,
        'Production': production[(month, factory)].varValue,
        'Factory Status': factory_status[(month, factory)].varValue,
        'Switch On': switch_on[(month, factory)].varValue
    }
    output.append(var_output)
output_df = pd.DataFrame.from_records(output).sort_values(['Month', 'Factory'])
output_df.set_index(['Month', 'Factory'], inplace=True)
output_df


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 2.ipynb".
The first task is:
We have also checked the status of the solver, there are 5 status codes:
* Not Solved: Status prior to solving the problem.
* Optimal: An optimal solution has been found.
* Infeasible: There are no feasible solutions (e.g. if you set the constraints x <= 1 and x >=2).
* Unbounded: The constraints are not bounded, maximising the solution will tend towards infinity (e.g. if the only constraint was x >= 3).
* Undefined: The optimal solution may exist but may not have been found.
We can now view our maximal variable values and the maximum value of Z.
We can use the varValue method to retrieve the values of our variables x and y, and the pulp.value function to view the maximum value of the objective function.
Can you write Python code for it?


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 2.ipynb".
The first task is:
We then model our decision variables using the LpVariable class. In our example, x had a lower bound of 0 and y had a lower bound of 2.
Upper bounds can be assigned using the upBound parameter.
Can you write Python code for it?

x = pulp.LpVariable('x', lowBound=0, cat='Continuous')
y = pulp.LpVariable('y', lowBound=2, cat='Continuous')


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 4.ipynb".
The first task is:
Each of these decision variables will have similar characteristics (lower bound of 0, continuous variables). Therefore we can use PuLP's LpVariable object's dict functionality, we can provide our tuple indices.
These tuples will be keys for the ing_weight dict of decision variables
Can you write Python code for it?

ing_weight = pulp.LpVariable.dicts("weight kg",
                                     ((i, j) for i in sausage_types for j in ingredients),
                                     lowBound=0,
                                     cat='Continuous')


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 4.ipynb".
The first task is:
Now we add our constraints, bear in mind again here how the use of list comprehensions allows for scaling up to many ingredients or sausage types
Can you write Python code for it?

# Constraints
# 350 economy and 500 premium sausages at 0.05 kg
model += pulp.lpSum([ing_weight['economy', j] for j in ingredients]) == 350 * 0.05
model += pulp.lpSum([ing_weight['premium', j] for j in ingredients]) == 500 * 0.05

# Economy has >= 40% pork, premium >= 60% pork
model += ing_weight['economy', 'pork'] >= (
    0.4 * pulp.lpSum([ing_weight['economy', j] for j in ingredients]))

model += ing_weight['premium', 'pork'] >= (
    0.6 * pulp.lpSum([ing_weight['premium', j] for j in ingredients]))

# Sausages must be <= 25% starch
model += ing_weight['economy', 'starch'] <= (
    0.25 * pulp.lpSum([ing_weight['economy', j] for j in ingredients]))

model += ing_weight['premium', 'starch'] <= (
    0.25 * pulp.lpSum([ing_weight['premium', j] for j in ingredients]))

# We have at most 30 kg of pork, 20 kg of wheat and 17 kg of starch available
model += pulp.lpSum([ing_weight[i, 'pork'] for i in sausage_types]) <= 30
model += pulp.lpSum([ing_weight[i, 'wheat'] for i in sausage_types]) <= 20
model += pulp.lpSum([ing_weight[i, 'starch'] for i in sausage_types]) <= 17

# We have at least 23 kg of pork to use up
model += pulp.lpSum([ing_weight[i, 'pork'] for i in sausage_types]) >= 23

# Solve our problem
model.solve()
pulp.LpStatus[model.status]

for var in ing_weight:
    var_value = ing_weight[var].varValue
    print "The weight of {0} in {1} sausages is {2} kg".format(var[1], var[0], var_value)

total_cost = pulp.value(model.objective)

print "The total cost is €{} for 350 economy sausages and 500 premium sausages".format(round(total_cost, 2))


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 4.ipynb".
The first task is:
PuLP provides an lpSum vector calculation for the sum of a list of linear expressions.
Whilst we only have 6 decision variables, I will demonstrate how the problem would be constructed in a way that could be scaled up to many variables using list comprehensions.
Can you write Python code for it?

# Objective Function
model += (
    pulp.lpSum([
        4.32 * ing_weight[(i, 'pork')]
        + 2.46 * ing_weight[(i, 'wheat')]
        + 1.86 * ing_weight[(i, 'starch')]
        for i in sausage_types])
)


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
An issue we run into here is that in linear programming we can't use conditional constraints.
For example we can't add to our model that if the factory is off factory status must be 0, and if it is on factory status must be 1. Before we've solved our model though, we don't know if the factory will be on or off in a given month.
In this case construct constraints that have minimum and maximum capacities that are constant variables, which we multiply by the factory status.
Now, either factory status is 0 and:
* $ \text{min_production} \geq 0$
* $ \text{max_production} \leq 0$
Or factory status is 1 and:
* $ \text{min_production} \leq \text{min_capacity}$
* $ \text{max_production} \leq \text{max_capacity}$
(In some cases we can use linear constraints to model conditional statements, we'll explore this in part 6)
Can you write Python code for it?

# Production in any month must be between minimum and maximum capacity, or zero.
for month, factory in factories.index:
    min_production = factories.loc[(month, factory), 'Min_Capacity']
    max_production = factories.loc[(month, factory), 'Max_Capacity']
    model += production[(month, factory)] >= min_production * factory_status[month, factory]
    model += production[(month, factory)] <= max_production * factory_status[month, factory]

# Factory B is off in May
model += factory_status[5, 'B'] == 0
model += production[5, 'B'] == 0


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
As we have fixed costs and variable costs, we'll need to model both production and the status of the factory i.e. whether it is on or off.
Production is modelled as an integer variable.
We have a value for production for each month for each factory, this is given by the tuples of our multi-index pandas DataFrame index.
Can you write Python code for it?

production = pulp.LpVariable.dicts("production",
                                     ((month, factory) for month, factory in factories.index),
                                     lowBound=0,
                                     cat='Integer')


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
Factory status is modelled as a binary variable. It will have a value of 1 if the factory is on and a value of 0 when the factory is off.
Binary variables are the same as integer variables but constrained to be >= 0 and <=1
Again this has a value for each month for each factory, again given by the index of our DataFrame
Can you write Python code for it?


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
Let's take a look at the optimal production schedule output for each month from each factory. For ease of viewing we'll output the data to a pandas DataFrame.
Can you write Python code for it?

output = []
for month, factory in production:
    var_output = {
        'Month': month,
        'Factory': factory,
        'Production': production[(month, factory)].varValue,
        'Factory Status': factory_status[(month, factory)].varValue
    }
    output.append(var_output)
output_df = pd.DataFrame.from_records(output).sort_values(['Month', 'Factory'])
output_df.set_index(['Month', 'Factory'], inplace=True)
output_df


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
Notice above that the factory status is 0 when not producing and 1 when it is producing
Can you write Python code for it?

# Print our objective function value (Total Costs)
print pulp.value(model.objective)


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
We build up our constraints
Can you write Python code for it?


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
We instantiate our model and use LpMinimize as the aim is to minimise costs.
Can you write Python code for it?

model = pulp.LpProblem("Cost minimising scheduling problem", pulp.LpMinimize)


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
We then solve the model
Can you write Python code for it?


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 5.ipynb".
The first task is:
We'll also import our demand data
Can you write Python code for it?

demand = pd.DataFrame.from_csv('csv/monthly_demand.csv', index_col=['Month'])

demand


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 6.ipynb".
The first task is:
We instantiate our model and define our objective function, including start up costs
Can you write Python code for it?

# Instantiate the model
model = pulp.LpProblem("Cost minimising scheduling problem", pulp.LpMinimize)

# Select index on factory A or B
factory_A_index = [tpl for tpl in factories.index if tpl[1] == 'A']
factory_B_index = [tpl for tpl in factories.index if tpl[1] == 'B']

# Define objective function
model += pulp.lpSum(
    [production[m, f] * factories.loc[(m, f), 'Variable_Costs'] for m, f in factories.index]
    + [factory_status[m, f] * factories.loc[(m, f), 'Fixed_Costs'] for m, f in factories.index]
    + [switch_on[m, f] * 20000 for m, f in factory_A_index]
    + [switch_on[m, f] * 400000 for m, f in factory_B_index]
)


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 6.ipynb".
The first task is:
We'll begin by defining our decision variables, we have an additional binary variable for switching on the factory.
Can you write Python code for it?

# Production
production = pulp.LpVariable.dicts("production",
                                     ((month, factory) for month, factory in factories.index),
                                     lowBound=0,
                                     cat='Integer')

# Factory Status, On or Off
factory_status = pulp.LpVariable.dicts("factory_status",
                                     ((month, factory) for month, factory in factories.index),
                                     cat='Binary')

# Factory switch on or off
switch_on = pulp.LpVariable.dicts("switch_on",
                                    ((month, factory) for month, factory in factories.index),
                                    cat='Binary')


I am working on the file "LP/Introduction-to-linear-programming/LaTeX_formatted_ipynb_files/Introduction to Linear Programming with Python - Part 6.ipynb".
The first task is:
We'll then solve our model
Can you write Python code for it?


I am working on the file "LPI Notebook.ipynb".
The first task is:
Alright so lets load up our actual data. First input the filepath for the formated and complete CSV file after filepath = ' '
You should be inputing something like:
filepath = '/Users/Jimmy/Desktop/LPI results.csv'
**Make sure you type your filepath in like I did above, computers are very unsympatheic listeners so leaving out the:
" ' / .csv" bits will result in a scary error popup
If using a CSV file make sure the filepath ends with .csv if it's excel .xls, when in doubt look at the file name on your computer to be certain
you'll see down bellow a bit of code that looks like this:
lpidf = pd.read_csv(filepath, index_col=0)
If you are loading up a CSV file leave this alone
If you are loading up a xls file or xlsx **use the bellow code instead
lpidf = pd.read_excel(filepath, sheetname=0, index_col=0)
Just copy which ever code you are supposed to use and then paste it over the code you need to replace
Can you write Python code for it?

filepath = '../LPI Project/Data Sheets/Master LPI sheet.csv'

import pandas as pd
import numpy as np

try:
    lpidf = pd.read_csv(filepath, index_col=0)
    print "All good... hopefully"
except IOError:
    print "double check file path \n\nand no you did not load the data correctly"


I am working on the file "LPI Notebook.ipynb".
The first task is:
Gender
Lets start out by getting an idea of the distribution of
how many boys and girls we have in our sample run the next cell to
get a breakdown of the proportion of boys and girls
Can you write Python code for it?

print "***Ratio of Male and Female*** \n \n" , lpidf['Gender'].value_counts(normalize=True)


I am working on the file "LPI Notebook.ipynb".
The first task is:
Ok lets move on to visualizing the LPI Ratings, would you kindly run the next cell
Can you write Python code for it?

sns.set_context("talk")
sns.countplot(x='Rating', hue='Gender',
              order=['N/a', 'Novice', 'Intermediate', 'Advanced'], data=lpidf, palette="Set3")
sns.despine(offset=2, trim=False)
plt.title('LPI Rating by Gender')

sns.countplot(x='Rating', hue='Internet Access',
              order=['N/a', 'Novice', 'Intermediate', 'Advanced'], data=lpidf, palette="Set2")
sns.despine(offset=2, trim=False)
plt.title('LPI Rating by Internet Access')

sns.countplot(x='Rating', hue='Taking Private Courses',
              order=['N/a', 'Novice', 'Intermediate', 'Advanced'], data=lpidf, palette="Set3")
sns.despine(offset=2, trim=False)
plt.title('LPI Rating and Enrollment in Private Engligh courses')

sns.countplot(x='Rating', hue='school_type',
              order=['N/a', 'Novice', 'Intermediate', 'Advanced'], data=lpidf, palette="GnBu_d")
sns.despine(offset=2, trim=False)
plt.title('LPI Rating and Type of School attended')

sns.countplot(x='Rating', hue='English Speaking family member',
              order=['N/a', 'Novice', 'Intermediate', 'Advanced'], data=lpidf, palette="BrBG")
sns.despine(offset=2, trim=False)
plt.title('LPI Rating and English Speaking Family Members')


I am working on the file "LPI Notebook.ipynb".
The first task is:
Ok so now lets go ahead and start digging into what factors could be influencing student scores on the LPI
Would you kindly run the next cell, you'll see graph where the large dots represent the average age of a student
with a particular LPI score. The lines extending from the dot represent the range of student ages for a particular LPI score.
Can you write Python code for it?

from numpy import median
sns.pointplot(x='Rating', y='Age', dodge=True, hue='Gender',
              join=False, data=lpidf, pallette='BrBG',
             order=['N/a', 'Novice', 'Intermediate', 'Advanced'])
sns.despine(offset=2, trim=True)
plt.title('LPI by age and Gender')


I am working on the file "LPI Notebook.ipynb".
The first task is:
Would you kindly run the next cell to get a visualizations of our students' ages
Can you write Python code for it?

%matplotlib inline
import seaborn as sns
import ggplot
from ggplot import aes
import matplotlib as mpl
import matplotlib.mlab as mlab
import matplotlib.pyplot as plt
sns.set(color_codes=True)
sns.set_style("ticks")
sns.set_context("talk")

sns.distplot(lpidf['Age'], kde=True, rug=True, bins=[9,10, 11, 12, 13, 14, 15, 16, 17, 18, 19 ,20])
plt.title("Student Ages")
sns.despine(offset=10, trim=False)


I am working on the file "LR_Example.ipynb".
The first task is:
Just right
This is an example of a good, overall, fit to the data. We can see that the function is smooth and does not oscillate wildly between the datapoints.
Can you write Python code for it?

graph_just_right = interactive(graph_polyfit, degree=3, size=10)
graph_just_right


I am working on the file "LR_Example.ipynb".
The first task is:
Overfit
Just as we show in the example above, we can also generate another example of what would be considered an overfitting of our $sin(2\pi x)$ function.
As you can see, the 9th degree polynomial oscillates wildly between the different datapoints in an effort to find a best fit that ends up being too "close" to the data that the model sees.
Can you write Python code for it?

graph_overfit = interactive(graph_polyfit, degree=9, size=10)
graph_overfit


I am working on the file "LR_Example.ipynb".
The first task is:
You can see above, just be sampling 90 more data points from our mock function, the 9th degree polynomial is already starting to smooth out a bunch. If you grab the size slider and move it to the right, generating more sample data, you'll see that the recomputed polynomial that is modeling the data smooths even more. Giving us this regularization affect.
Regularization - when it's hard to get more data
If you can't get your hands on more data from the system that has produced the data, then you may have to move into the realm of adding a penalty to your cost, or error, function $E(w)$.
A common penalization to add to our error function is the sum of the squares of all parameters, or coefficients, of the model.
$$ E(w) = \frac{1}{2}{y(x_n, w) - t_n}^2 + \frac{\lambda}{2}\|w\|^2$$
where $\|w\|^2 = w^Tw = w_0^2 + w_1^2 + w_3^2 ... w_M^2$. The coefficient $\lambda$ is what governs the relative importants of the regularization effect on the model.
Can you write Python code for it?

# generate some random data and target values
rr_X, rr_y = example_data_generating_dist(100)

rr_y

clf = Ridge(alpha=1.0, solver='lsqr')
clf.fit(rr_X[:,np.newaxis], rr_y)

plt.scatter(rr_X, rr_y)

colors = ['teal', 'yellowgreen', 'gold']
lw = 2
for count, degree in enumerate([2]):
    model = Ridge()
    model.fit(rr_X[:,np.newaxis], rr_y)
    y_plot = model.predict(rr_X[:,np.newaxis])
    plt.plot(rr_X, y_plot, color=colors[count], linewidth=lw,
             label="degree %d" % degree)


I am working on the file "LSST/SuperNovaLightCurves/Fit_Errors.ipynb".
The first task is:
Heres another example where the flux predictions are the same for both the matern and the real kernel
Can you write Python code for it?

#Create new supernovae object
SN = Supernovae('../../../OSC_data/JSON_data/sne-2010-2014-master/LSQ12ca.json')
#Load in lightcurve data
SN.load_LightCurves()

print(SN.Lightcurves['B_2'].Gaussian_process(Real_Kernel))
print(SN.Lightcurves['B_2'].Gaussian_process(Matern_Kernel))


I am working on the file "LSST/SuperNovaLightCurves/Fit_Errors.ipynb".
The first task is:
One last Example
Can you write Python code for it?

#Create new supernovae object
SN = Supernovae('../../../OSC_data/JSON_data/sne-2010-2014-master/iPTF13bjx.json')
#Load in lightcurve data
SN.load_LightCurves()

SN.Lightcurves['R_1'].Gaussian_process(Real_Kernel)
SN.Lightcurves['R_1'].Gaussian_process(Matern_Kernel)


I am working on the file "LSST/SuperNovaLightCurves/Fit_Errors.ipynb".
The first task is:
The arrays printed out are the individual flux predictions. Some are being calculated correctly, but some turn out as NaN, causing the resulting Rchi2 value to be NaN.
Can you write Python code for it?


I am working on the file "LSST/VariableStarClassification/First_Sources.ipynb".
The first task is:
Again, this oid doesn't have any apparent variability.
Source 2
Can you write Python code for it?

plt.errorbar(times_2, obs_mag_2, yerr = obs_mag_error_2, fmt = 'bo', markersize = 3)

plt.xlabel('MJD')
plt.ylabel('Observed mag')
plt.title('Source 2 Lightcurve "All Oids"')


I am working on the file "LSST/VariableStarClassification/First_Sources.ipynb".
The first task is:
Decomposed Oids
Can you write Python code for it?

oid_21 = np.where(source_2[3][:] == 226831060005494)
plt.errorbar(times_2[oid_21], obs_mag_2[oid_21], yerr = obs_mag_error_2[oid_21], fmt = 'bo', markersize = 3)

plt.xlabel('MJD')
plt.ylabel('Observed mag')
plt.title('Source 2 Lightcurve "Oid 226831060005494"')


I am working on the file "LSST/VariableStarClassification/First_Sources.ipynb".
The first task is:
Folded Lightcurves
For oids 226832060006908 and 26832000005734
Can you write Python code for it?

primary_period_1 = 0.191486 #taken from the NASA Exoplanet Archieve Periodogram Service
phase_21 = (times_2 % primary_period_1) / primary_period_1

plt.errorbar(phase_21[oid_23], obs_mag_2[oid_23], yerr = obs_mag_error_2[oid_23], fmt = 'bo', markersize = 3)

plt.xlabel('Phase')
plt.ylabel('Observed mag')
plt.title('Source 2 Periodic Lightcurve For Oid 226832060006908')


I am working on the file "LSST/VariableStarClassification/First_Sources.ipynb".
The first task is:
Picking out the relevant data into their own arrays to work with.
Can you write Python code for it?

times_1 = source_1[0][:] #date expressed in juilian days
obs_mag_1 = source_1[1][:] #observed magnitude, auto corrected? correlated?
obs_mag_error_1 = source_1[2][:] #error on the observed magnitude

times_2 = source_2[0][:]
obs_mag_2 = source_2[1][:]
obs_mag_error_2 = source_2[2][:]


I am working on the file "LSST/VariableStarClassification/First_Sources.ipynb".
The first task is:
Source 1
As each data file had multiple oid's present, I plotted both the raw file and also the individual sources on their own.
Can you write Python code for it?

plt.errorbar(times_1, obs_mag_1, yerr = obs_mag_error_1, fmt = 'ro', markersize = 3)

plt.xlabel('MJD')
plt.ylabel('Observed Magnitude')
plt.title('Source 1 Lightcurve "All Oids"')


I am working on the file "LSST/VariableStarClassification/First_Sources.ipynb".
The first task is:
There maybe some periodic variability here. A fit of a cosine might be able to reproduce this data. However, it apprears to be scattered fairly randomly.
Can you write Python code for it?

primary_period_2 = 2.440220
phase_22 = (times_2 % primary_period_2) / primary_period_2

plt.errorbar(phase_22[oid_23], obs_mag_2[oid_23], yerr = obs_mag_error_2[oid_23], fmt = 'bo', markersize = 3)

plt.xlabel('Phase')
plt.ylabel('Observed mag')
plt.title('Source 2 Periodic Lightcurve For Oid 26832000005734')


I am working on the file "LSST/VariableStarClassification/First_Sources.ipynb".
The first task is:
This is just a single point so it is likely to be some sort of outlier or misattributed source.
Can you write Python code for it?

oid_22 = np.where(source_2[3][:] == 226832060006908)
plt.errorbar(times_2[oid_22], obs_mag_2[oid_22], yerr = obs_mag_error_2[oid_22], fmt = 'bo', markersize = 3)

plt.xlabel('MJD')
plt.ylabel('Observed mag')
plt.title('Source 2 Lightcurve "Oid 226832060006908"')

oid_23 = np.where(source_2[3][:] == 26832000005734)
plt.errorbar(times_2[oid_23], obs_mag_2[oid_23], yerr = obs_mag_error_2[oid_23], fmt = 'bo', markersize = 3)

plt.xlabel('MJD')
plt.ylabel('Observed mag')
plt.title('Source 2 Lightcurve "Oid 26832000005734"')


I am working on the file "LSST/VariableStarClassification/First_Sources.ipynb".
The first task is:
This oid doesnt seem to have an variability. And, given the plot above, it would seem that these are in fact distinct sources.
Can you write Python code for it?

oid_12 = np.where(source_1[3][:] == 33262000001431)
plt.errorbar(times_1[oid_12], obs_mag_1[oid_12], yerr = obs_mag_error_1[oid_12], fmt = 'ro', markersize = 3)

plt.xlabel('MJD')
plt.ylabel('Observed Magnitude')
plt.title('Source 1 Lightcurve "Oid 33262000001431')


I am working on the file "LSST/VariableStarClassification/scripts/ptf_query/byOid/url_by_oid.ipynb".
The first task is:
If reading in for the first times then use this cell. This will loop over each oid within the objects queried above and excute queries for their source lightcurves. Additionally, reads that returned data file into a dict to reference later.
Can you write Python code for it?

#curves = {}

for i in range(0,18849):
    loc = 'curves_oid{:_>17}.tbl'.format(objects['oid'][i])
    cmd = 'curl -F catalog=ptf_lightcurves -F spatial=None -F constraints=' + '"(oid={:})"'.format(objects['oid'][i]) + ' -F outfmt=1 -F selcols=oid,obsmjd,mag_autocorr,magerr_auto,fid,ra,dec "http://irsa.ipac.caltech.edu/cgi-bin/Gator/nph-query?" -o curves_oid{:_>17}.tbl'.format(objects['oid'][i])
    subprocess.call(cmd, shell = True)
    print(i)
    #curves = tbl.read(loc, format = 'ipac')
    #saveFeat(curves, 'feats2', cur, conn)

    #%run '/home/nke2/NUREU17/LSST/VariableStarClassification/saveFeat.py' (curves, 'feats2', cur, conn)
    #curves[i] = tbl.read('curves_oid{:_>17}.tbl'.format(objects['oid'][i]), format = 'ipac')


I am working on the file "LSST/VariableStarClassification/scripts/ptf_query/byOid/url_by_oid.ipynb".
The first task is:
Im not sure why subprocess.call doesnt seem to work for this specific case. However, the urllib work below does the job.
Can you write Python code for it?

%%timeit
data = urllib.parse.urlencode(values)
data = data.encode('utf-8')
req = urllib.request.Request(url, data)
resp = urllib.request.urlopen(req)
respdata = resp.read()
saveFile = open('objects.tbl', 'wb')
saveFile.write(respdata)
saveFile.close()


I am working on the file "LSST/VariableStarClassification/scripts/ptf_query/byOid/url_by_oid.ipynb".
The first task is:
Read that data file in
Can you write Python code for it?

objects = tbl.read('/home/nick/Desktop/NUREU17/LSST/VariableStarClassification/scripts/ptf_query/objects.tbl', format = 'ipac')

conn = sqlite3.connect('/home/nick/Desktop/NUREU17/LSST/VariableStarClassification/features.db')
cur = conn.cursor()

def saveFeat (lc, tName, cur, conn): #pass in lightcurve table and cursor
    feats_to_use = [
                'amplitude',
                'flux_percentile_ratio_mid20',
                'flux_percentile_ratio_mid35',
                'flux_percentile_ratio_mid50',
                'flux_percentile_ratio_mid65',
                'flux_percentile_ratio_mid80',
                'max_slope',
                'maximum',
                'median',
                'median_absolute_deviation',
                'minimum',
                'percent_amplitude',
                'percent_beyond_1_std',
                'percent_close_to_median',
                'percent_difference_flux_percentile',
                'period_fast',
                'qso_log_chi2_qsonu',
                'qso_log_chi2nuNULL_chi2nu',
                'skew',
                'std',
                'stetson_j',
                'stetson_k',
                'weighted_average',
                'fold2P_slope_10percentile',
                'fold2P_slope_90percentile',
                'freq1_amplitude1',
                'freq1_amplitude2',
                'freq1_amplitude3',
                'freq1_amplitude4',
                'freq1_freq',
                'freq1_lambda',
                'freq1_rel_phase2',
                'freq1_rel_phase3',
                'freq1_rel_phase4',
                'freq1_signif',
                'freq2_amplitude1',
                'freq2_amplitude2',
                'freq2_amplitude3',
                'freq2_amplitude4',
                'freq2_freq',
                'freq2_rel_phase2',
                'freq2_rel_phase3',
                'freq2_rel_phase4',
                'freq3_amplitude1',
                'freq3_amplitude2',
                'freq3_amplitude3',
               ...


I am working on the file "LSST/VariableStarClassification/scripts/ptf_query/byOid/url_by_oid.ipynb".
The first task is:
Same as above. However, if the sources have already been queried just reads in the files to save some time.
Can you write Python code for it?

#curves = {}
cur.execute("""select oid from feats2""")
where = cur.fetchall()

for i in range(0, 18849):
    if (objects['oid'][i] not in where):
        try:
            curves = tbl.read('curves_oid{:_>17}.tbl'.format(objects['oid'][i]), format = 'ipac')
            print(objects['oid'][i], i)
            saveFeat(curves, 'feats2', cur, conn)
        except:
            cmd = 'curl -F catalog=ptf_lightcurves -F spatial=None -F constraints=' + '"(oid={:})"'.format(objects['oid'][i]) + ' -F outfmt=1 -F selcols=oid,obsmjd,mag_autocorr,magerr_auto,fid,ra,dec "http://irsa.ipac.caltech.edu/cgi-bin/Gator/nph-query?" -o curves_oid{:_>17}.tbl'.format(objects['oid'][i])
            subprocess.call(cmd, shell = True)
            curves = tbl.read('curves_oid{:_>17}.tbl'.format(objects['oid'][i]), format = 'ipac')
            print('queried, ', objects['oid'][i], i)
            saveFeat(curves, 'feats2', cur, conn)

130122070001802

np.where(objects['oid']==35782020008798)

cur.execute("""select * from feats2 where oid=35782020008798""")
cur.fetchall()


I am working on the file "LSTM.ipynb".
The first task is:
Additionnal Parameters:
Here are some core parameter definitions for the training.
For example, the whole neural network's structure could be summarised by enumerating those parameters and the fact that two LSTM are used one on top of another (stacked) output-to-input as hidden layers through time steps.
Can you write Python code for it?

# Input Data

training_data_count = len(X_train)  # 7352 training series (with 50% overlap between each serie)
test_data_count = len(X_test)  # 2947 testing series
n_steps = len(X_train[0])  # 128 timesteps per series
n_input = len(X_train[0][0])  # 9 input parameters per timestep

# LSTM Neural Network's internal structure

n_hidden = 32 # Hidden layer num of features
n_classes = 6 # Total classes (should go up, or should go down)

# Training

learning_rate = 0.0025
lambda_loss_amount = 0.0015
training_iters = training_data_count * 300  # Loop 300 times on the dataset
batch_size = 1500
display_iter = 30000  # To show test set accuracy during training

# Some debugging info

print("Some useful info to get an insight on dataset's shape and normalisation:")
print("(X shape, y shape, every X's mean, every X's standard deviation)")
print(X_test.shape, y_test.shape, np.mean(X_test), np.std(X_test))
print("The dataset is therefore properly normalised, as expected, but not yet one-hot encoded.")


I am working on the file "LSTM.ipynb".
The first task is:
Additionnal Parameters:
Here are some core parameter definitions for the training.
The whole neural network's structure could be summarised by enumerating those parameters and the fact an LSTM is used.
Can you write Python code for it?

# Input Data

training_data_count = len(X_train)  # 7352 training series (with 50% overlap between each serie)
test_data_count = len(X_test)  # 2947 testing series
n_steps = len(X_train[0])  # 128 timesteps per series
n_input = len(X_train[0][0])  # 9 input parameters per timestep

# LSTM Neural Network's internal structure

n_hidden = 32 # Hidden layer num of features
n_classes = 6 # Total classes (should go up, or should go down)

# Training

learning_rate = 0.0025
lambda_loss_amount = 0.0015
training_iters = training_data_count * 300  # Loop 300 times on the dataset
batch_size = 1500
display_iter = 30000  # To show test set accuracy during training

# Some debugging info

print "Some useful info to get an insight on dataset's shape and normalisation:"
print "(X shape, y shape, every X's mean, every X's standard deviation)"
print (X_test.shape, y_test.shape, np.mean(X_test), np.std(X_test))
print "The dataset is therefore properly normalised, as expected, but not yet one-hot encoded."


I am working on the file "LSTM.ipynb".
The first task is:
And finally, the multi-class confusion matrix and metrics!
Can you write Python code for it?

# Results

predictions = one_hot_predictions.argmax(1)

print("Testing Accuracy: {}%".format(100*accuracy))

print("")
print("Precision: {}%".format(100*metrics.precision_score(y_test, predictions, average="weighted")))
print("Recall: {}%".format(100*metrics.recall_score(y_test, predictions, average="weighted")))
print("f1_score: {}%".format(100*metrics.f1_score(y_test, predictions, average="weighted")))

print("")
print("Confusion Matrix:")
confusion_matrix = metrics.confusion_matrix(y_test, predictions)
print(confusion_matrix)
normalised_confusion_matrix = np.array(confusion_matrix, dtype=np.float32)/np.sum(confusion_matrix)*100

print("")
print("Confusion matrix (normalised to % of total test data):")
print(normalised_confusion_matrix)
print("Note: training and testing data is not equally distributed amongst classes, ")
print("so it is normal that more than a 6th of the data is correctly classifier in the last category.")

# Plot Results:
width = 12
height = 12
plt.figure(figsize=(width, height))
plt.imshow(
    normalised_confusion_matrix,
    interpolation='nearest',
    cmap=plt.cm.rainbow
)
plt.title("Confusion matrix \n(normalised to % of total test data)")
plt.colorbar()
tick_marks = np.arange(n_classes)
plt.xticks(tick_marks, LABELS, rotation=90)
plt.yticks(tick_marks, LABELS)
plt.tight_layout()
plt.ylabel('True label')
plt.xlabel('Predicted label')
plt.show()

sess.close()


I am working on the file "LSTM.ipynb".
The first task is:
Conclusion
Outstandingly, the final accuracy is of 91%! And it can peak to values such as 93.25%, at some moments of luck during the training, depending on how the neural network's weights got initialized at the start of the training, randomly.
This means that the neural networks is almost always able to correctly identify the movement type! Remember, the phone is attached on the waist and each series to classify has just a 128 sample window of two internal sensors (a.k.a. 2.56 seconds at 50 FPS), so it amazes me how those predictions are extremely accurate given this small window of context and raw data. I've validated and re-validated that there is no important bug, and the community used and tried this code a lot. (Note: be sure to report something in the issue tab if you find bugs, otherwise Quora, StackOverflow, and other StackExchange sites are the places for asking questions.)
I specially did not expect such good results for guessing between the labels "SITTING" and "STANDING". Those are seemingly almost the same thing from the point of view of a device placed at waist level according to how the dataset was originally gathered. Thought, it is still possible to see a little cluster on the matrix between those classes, which drifts away just a bit from the identity. This is great.
It is also possible to see that there was a slight difficulty in doing the difference between "WALKING", "WALKING_UPSTAIRS" and "WALKING_DOWNSTAIRS". Obviously, those activities are quite similar in terms of movements.
I also tried my code without the gyroscope, using only the 3D accelerometer's 6 features (and not changing the training hyperparameters), and got an accuracy of 87%. In general, gyroscopes consumes more power than accelerometers, so it is preferable to turn them off.
Improvements
In another open-source repository of mine, the accuracy is pushed up to nearly 94% using a special deep LSTM architecture which combines the concepts of bidirectional RNNs, residual connections, and stacked cells. This architecture is...
Can you write Python code for it?

# Let's convert this notebook to a README automatically for the GitHub project's title page:
!jupyter nbconvert --to markdown LSTM.ipynb
!mv LSTM.md README.md


I am working on the file "LSTM.ipynb".
The first task is:
Hooray, now train the neural network:
Can you write Python code for it?

# To keep track of training's performance
test_losses = []
test_accuracies = []
train_losses = []
train_accuracies = []

# Launch the graph
sess = tf.InteractiveSession(config=tf.ConfigProto(log_device_placement=True))
init = tf.initialize_all_variables()
sess.run(init)

# Perform Training steps with "batch_size" amount of example data at each loop
step = 1
while step * batch_size <= training_iters:
    batch_xs =         extract_batch_size(X_train, step, batch_size)
    batch_ys = one_hot(extract_batch_size(y_train, step, batch_size))

    # Fit training using batch data
    _, loss, acc = sess.run(
        [optimizer, cost, accuracy],
        feed_dict={
            x: batch_xs,
            y: batch_ys
        }
    )
    train_losses.append(loss)
    train_accuracies.append(acc)

    # Evaluate network only at some steps for faster training:
    if (step*batch_size % display_iter == 0) or (step == 1) or (step * batch_size > training_iters):

        # To not spam console, show training accuracy/loss in this "if"
        print "Training iter #" + str(step*batch_size) + \
              ":   Batch Loss = " + "{:.6f}".format(loss) + \
              ", Accuracy = {}".format(acc)

        # Evaluation on the test set (no learning made here - just evaluation for diagnosis)
        loss, acc = sess.run(
            [cost, accuracy],
            feed_dict={
                x: X_test,
                y: one_hot(y_test)
            }
        )
        test_losses.append(loss)
        test_accuracies.append(acc)
        print "PERFORMANCE ON TEST SET: " + \
              "Batch Loss = {}".format(loss) + \
              ", Accuracy = {}".format(acc)

    step += 1

print "Optimization Finished!"

# Accuracy for test data

one_hot_predictions, accuracy, final_loss = sess.run(
    [pred, accuracy, cost],
    feed_dict={
        x: X_test,
        y: one_hot(y_test)
    }
)

test_losses.append(final_loss)
test_accuracies.append(accuracy)

print "FINAL RESULT: " + \
      "Batch Loss =...


# To keep track of training's performance
test_losses = []
test_accuracies = []
train_losses = []
train_accuracies = []

# Launch the graph
sess = tf.InteractiveSession(config=tf.ConfigProto(log_device_placement=True))
init = tf.global_variables_initializer()
sess.run(init)

# Perform Training steps with "batch_size" amount of example data at each loop
step = 1
while step * batch_size <= training_iters:
    batch_xs =         extract_batch_size(X_train, step, batch_size)
    batch_ys = one_hot(extract_batch_size(y_train, step, batch_size))

    # Fit training using batch data
    _, loss, acc = sess.run(
        [optimizer, cost, accuracy],
        feed_dict={
            x: batch_xs,
            y: batch_ys
        }
    )
    train_losses.append(loss)
    train_accuracies.append(acc)

    # Evaluate network only at some steps for faster training:
    if (step*batch_size % display_iter == 0) or (step == 1) or (step * batch_size > training_iters):

        # To not spam console, show training accuracy/loss in this "if"
        print("Training iter #" + str(step*batch_size) + \
              ":   Batch Loss = " + "{:.6f}".format(loss) + \
              ", Accuracy = {}".format(acc))

        # Evaluation on the test set (no learning made here - just evaluation for diagnosis)
        loss, acc = sess.run(
            [cost, accuracy],
            feed_dict={
                x: X_test,
                y: one_hot(y_test)
            }
        )
        test_losses.append(loss)
        test_accuracies.append(acc)
        print("PERFORMANCE ON TEST SET: " + \
              "Batch Loss = {}".format(loss) + \
              ", Accuracy = {}".format(acc))

    step += 1

print("Optimization Finished!")

# Accuracy for test data

one_hot_predictions, accuracy, final_loss = sess.run(
    [pred, accuracy, cost],
    feed_dict={
        x: X_test,
        y: one_hot(y_test)
    }
)

test_losses.append(final_loss)
test_accuracies.append(accuracy)

print("FINAL RESULT: " + \
      "Batch Loss =...


I am working on the file "LSTM.ipynb".
The first task is:
Let's start by downloading the data:
Can you write Python code for it?

# Note: Linux bash commands start with a "!" inside those "ipython notebook" cells

DATA_PATH = "data/"

!pwd && ls
os.chdir(DATA_PATH)
!pwd && ls

!python download_dataset.py

!pwd && ls
os.chdir("..")
!pwd && ls

DATASET_PATH = DATA_PATH + "UCI HAR Dataset/"
print("\n" + "Dataset is now located at: " + DATASET_PATH)


I am working on the file "LSTM.ipynb".
The first task is:
Looking good ...
Lets check with a larger generated sample.
Can you write Python code for it?

generate_chars = 1000
sentence = data[seq_start_index: seq_start_index + seq_len] #sentence = sequence
print('----- Generating with seed: "' + sentence + '"')

for i in range(generate_chars):
    x_pred = np.zeros((1, seq_len, vocab_size)) #single sequence will be passed to trained RNN
    for t, char in enumerate(sentence): #turn sequence to one-hot
        x_pred[0, t, char_to_ix[char]] = 1. #sequence is from data = characters not numbers

    preds = model.predict(x_pred, verbose=0)[0] #get next_char prediction = vector of probabilities
    #next_index = sample(preds, diversity) #TODO implement sample function supporting different diversities
    next_char = vector_to_char(preds, ix_to_char)

    sentence = sentence[1:] + next_char

    sys.stdout.write(next_char)
    sys.stdout.flush()


I am working on the file "LSTM.ipynb".
The first task is:
Preparing dataset:
Can you write Python code for it?

TRAIN = "train/"
TEST = "test/"

# Load "X" (the neural network's training and testing inputs)

def load_X(X_signals_paths):
    X_signals = []

    for signal_type_path in X_signals_paths:
        file = open(signal_type_path, 'rb')
        # Read dataset from disk, dealing with text files' syntax
        X_signals.append(
            [np.array(serie, dtype=np.float32) for serie in [
                row.replace('  ', ' ').strip().split(' ') for row in file
            ]]
        )
        file.close()

    return np.transpose(np.array(X_signals), (1, 2, 0))

X_train_signals_paths = [
    DATASET_PATH + TRAIN + "Inertial Signals/" + signal + "train.txt" for signal in INPUT_SIGNAL_TYPES
]
X_test_signals_paths = [
    DATASET_PATH + TEST + "Inertial Signals/" + signal + "test.txt" for signal in INPUT_SIGNAL_TYPES
]

X_train = load_X(X_train_signals_paths)
X_test = load_X(X_test_signals_paths)

# Load "y" (the neural network's training and testing outputs)

def load_y(y_path):
    file = open(y_path, 'rb')
    # Read dataset from disk, dealing with text file's syntax
    y_ = np.array(
        [elem for elem in [
            row.replace('  ', ' ').strip().split(' ') for row in file
        ]],
        dtype=np.int32
    )
    file.close()

    # Substract 1 to each output class for friendly 0-based indexing
    return y_ - 1

y_train_path = DATASET_PATH + TRAIN + "y_train.txt"
y_test_path = DATASET_PATH + TEST + "y_test.txt"

y_train = load_y(y_train_path)
y_test = load_y(y_test_path)


TRAIN = "train/"
TEST = "test/"

# Load "X" (the neural network's training and testing inputs)

def load_X(X_signals_paths):
    X_signals = []

    for signal_type_path in X_signals_paths:
        file = open(signal_type_path, 'r')
        # Read dataset from disk, dealing with text files' syntax
        X_signals.append(
            [np.array(serie, dtype=np.float32) for serie in [
                row.replace('  ', ' ').strip().split(' ') for row in file
            ]]
        )
        file.close()

    return np.transpose(np.array(X_signals), (1, 2, 0))

X_train_signals_paths = [
    DATASET_PATH + TRAIN + "Inertial Signals/" + signal + "train.txt" for signal in INPUT_SIGNAL_TYPES
]
X_test_signals_paths = [
    DATASET_PATH + TEST + "Inertial Signals/" + signal + "test.txt" for signal in INPUT_SIGNAL_TYPES
]

X_train = load_X(X_train_signals_paths)
X_test = load_X(X_test_signals_paths)

# Load "y" (the neural network's training and testing outputs)

def load_y(y_path):
    file = open(y_path, 'r')
    # Read dataset from disk, dealing with text file's syntax
    y_ = np.array(
        [elem for elem in [
            row.replace('  ', ' ').strip().split(' ') for row in file
        ]],
        dtype=np.int32
    )
    file.close()

    # Substract 1 to each output class for friendly 0-based indexing
    return y_ - 1

y_train_path = DATASET_PATH + TRAIN + "y_train.txt"
y_test_path = DATASET_PATH + TEST + "y_test.txt"

y_train = load_y(y_train_path)
y_test = load_y(y_test_path)


I am working on the file "LSTM.ipynb".
The first task is:
Repetitions gone?
Looks like the network has got rid of the repetitions but they are actually just hiding on a larger scale,
let's continue training...
Can you write Python code for it?

model_backup = model
epochs = 20
batch_size = 128
generate_chars = 250 #5 times as more chars generated after each epoch
train_model(epochs, batch_size, generate_chars, model, data, char_to_ix, ix_to_char, seq_len, vocab_size)


I am working on the file "LSTM.ipynb".
The first task is:
Training is good, but having visual insight is even better:
Okay, let's plot this simply in the notebook for now.
Can you write Python code for it?

# (Inline plots: )
%matplotlib inline

font = {
    'family' : 'Bitstream Vera Sans',
    'weight' : 'bold',
    'size'   : 18
}
matplotlib.rc('font', **font)

width = 12
height = 12
plt.figure(figsize=(width, height))

indep_train_axis = np.array(range(batch_size, (len(train_losses)+1)*batch_size, batch_size))
plt.plot(indep_train_axis, np.array(train_losses),     "b--", label="Train losses")
plt.plot(indep_train_axis, np.array(train_accuracies), "g--", label="Train accuracies")

indep_test_axis = np.array(range(batch_size, len(test_losses)*display_iter, display_iter)[:-1] + [training_iters])
plt.plot(indep_test_axis, np.array(test_losses),     "b-", label="Test losses")
plt.plot(indep_test_axis, np.array(test_accuracies), "g-", label="Test accuracies")

plt.title("Training session's progress over iterations")
plt.legend(loc='upper right', shadow=True)
plt.ylabel('Training Progress (Loss or Accuracy values)')
plt.xlabel('Training iteration')

plt.show()


I am working on the file "LSTM.ipynb".
The first task is:
Utility functions for training:
Can you write Python code for it?

def LSTM_RNN(_X, _weights, _biases):
    # Function returns a tensorflow LSTM (RNN) artificial neural network from given parameters.
    # Moreover, two LSTM cells are stacked which adds deepness to the neural network.
    # Note, some code of this notebook is inspired from an slightly different
    # RNN architecture used on another dataset, some of the credits goes to
    # "aymericdamien" under the MIT license.

    # (NOTE: This step could be greatly optimised by shaping the dataset once
    # input shape: (batch_size, n_steps, n_input)
    _X = tf.transpose(_X, [1, 0, 2])  # permute n_steps and batch_size
    # Reshape to prepare input to hidden activation
    _X = tf.reshape(_X, [-1, n_input])
    # new shape: (n_steps*batch_size, n_input)

    # ReLU activation, thanks to Yu Zhao for adding this improvement here:
    _X = tf.nn.relu(tf.matmul(_X, _weights['hidden']) + _biases['hidden'])
    # Split data because rnn cell needs a list of inputs for the RNN inner loop
    _X = tf.split(_X, n_steps, 0)
    # new shape: n_steps * (batch_size, n_hidden)

    # Define two stacked LSTM cells (two recurrent layers deep) with tensorflow
    lstm_cell_1 = tf.contrib.rnn.BasicLSTMCell(n_hidden, forget_bias=1.0, state_is_tuple=True)
    lstm_cell_2 = tf.contrib.rnn.BasicLSTMCell(n_hidden, forget_bias=1.0, state_is_tuple=True)
    lstm_cells = tf.contrib.rnn.MultiRNNCell([lstm_cell_1, lstm_cell_2], state_is_tuple=True)
    # Get LSTM cell output
    outputs, states = tf.contrib.rnn.static_rnn(lstm_cells, _X, dtype=tf.float32)

    # Get last time step's output feature for a "many-to-one" style classifier,
    # as in the image describing RNNs at the top of this page
    lstm_last_output = outputs[-1]

    # Linear activation
    return tf.matmul(lstm_last_output, _weights['out']) + _biases['out']

def extract_batch_size(_train, step, batch_size):
    # Function to fetch a "batch_size" amount of data from "(X|y)_train" data.

    shape = list(_train.shape)
    shape[0] = batch_size
    batch_s =...


I am working on the file "LSTM/LSTM.ipynb".
The first task is:
Dataset Preparation
Can you write Python code for it?

#Function to get on hot
def get_on_hot(number):
    on_hot=[0]*10
    on_hot[number]=1
    return on_hot

#Using Sklearn MNIST dataset.
digits = datasets.load_digits()
X=digits.images
Y_=digits.target

Y=map(get_on_hot,Y_)

#Getting Train and test Dataset
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.22, random_state=42)

#Cuttting for simple iteration
X_train=X_train[:1400]
y_train=y_train[:1400]

sess=tf.InteractiveSession()
sess.run(tf.initialize_all_variables())

#Iterations to do trainning
for epoch in range(120):

    start=0
    end=100
    for i in range(14):

        X=X_train[start:end]
        Y=y_train[start:end]
        start=end
        end=start+100
        sess.run(train_step,feed_dict={rnn._inputs:X, y:Y})

    Loss=str(sess.run(cross_entropy,feed_dict={rnn._inputs:X, y:Y}))
    Train_accuracy=str(sess.run(accuracy,feed_dict={rnn._inputs:X_train[:500], y:y_train[:500]}))
    Test_accuracy=str(sess.run(accuracy,feed_dict={rnn._inputs:X_test, y:y_test}))

    pl.plot([epoch],Loss,'b.',)
    pl.plot([epoch],Train_accuracy,'r*',)
    pl.plot([epoch],Test_accuracy,'g+')
    display.clear_output(wait=True)
    display.display(pl.gcf())

    sys.stdout.flush()
    print("\rIteration: %s Loss: %s Train Accuracy: %s Test Accuracy: %s"%(epoch,Loss,Train_accuracy,Test_accuracy)),
    sys.stdout.flush()


I am working on the file "LSTM/LSTM.ipynb".
The first task is:
LSTM class and functions
Can you write Python code for it?

class LSTM_cell(object):

    """
    LSTM cell object which takes 3 arguments for initialization.
    input_size = Input Vector size
    hidden_layer_size = Hidden layer size
    target_size = Output vector size

    """

    def __init__(self, input_size, hidden_layer_size, target_size):

        # Initialization of given values
        self.input_size = input_size
        self.hidden_layer_size = hidden_layer_size
        self.target_size = target_size

        # Weights and Bias for input and hidden tensor
        self.Wi = tf.Variable(tf.zeros(
            [self.input_size, self.hidden_layer_size]))
        self.Ui = tf.Variable(tf.zeros(
            [self.hidden_layer_size, self.hidden_layer_size]))
        self.bi = tf.Variable(tf.zeros([self.hidden_layer_size]))

        self.Wf = tf.Variable(tf.zeros(
            [self.input_size, self.hidden_layer_size]))
        self.Uf = tf.Variable(tf.zeros(
            [self.hidden_layer_size, self.hidden_layer_size]))
        self.bf = tf.Variable(tf.zeros([self.hidden_layer_size]))

        self.Wog = tf.Variable(tf.zeros(
            [self.input_size, self.hidden_layer_size]))
        self.Uog = tf.Variable(tf.zeros(
            [self.hidden_layer_size, self.hidden_layer_size]))
        self.bog = tf.Variable(tf.zeros([self.hidden_layer_size]))

        self.Wc = tf.Variable(tf.zeros(
            [self.input_size, self.hidden_layer_size]))
        self.Uc = tf.Variable(tf.zeros(
            [self.hidden_layer_size, self.hidden_layer_size]))
        self.bc = tf.Variable(tf.zeros([self.hidden_layer_size]))

        # Weights for output layers
        self.Wo = tf.Variable(tf.truncated_normal(
            [self.hidden_layer_size, self.target_size],mean=0,stddev=.01))
        self.bo = tf.Variable(tf.truncated_normal([self.target_size],mean=0,stddev=.01))

        # Placeholder for input vector with shape[batch, seq, embeddings]
        self._inputs =...


I am working on the file "LSTM/LSTM.ipynb".
The first task is:
Models
Can you write Python code for it?

#Initializing rnn object
rnn=LSTM_cell( input_size, hidden_layer_size, target_size)

#Getting all outputs from rnn
outputs = rnn.get_outputs()

#Getting final output through indexing after reversing
#last_output = tf.reverse(outputs,[True,False,False])[0,:,:]
last_output = outputs[-1]

#As rnn model output the final layer through Relu activation softmax is used for final output.
output=tf.nn.softmax(last_output)

#Computing the Cross Entropy loss
cross_entropy = -tf.reduce_sum(y * tf.log(output))

# Trainning with Adadelta Optimizer
train_step = tf.train.AdamOptimizer().minimize(cross_entropy)

#Calculatio of correct prediction and accuracy
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(output,1))
accuracy = (tf.reduce_mean(tf.cast(correct_prediction, tf.float32)))*100


I am working on the file "LSTM/LSTM.ipynb".
The first task is:
Placeholder and initializers
Can you write Python code for it?

hidden_layer_size = 30
input_size = 8
target_size = 10

y = tf.placeholder(tf.float32, shape=[None, target_size],name='inputs')


I am working on the file "LSTM_Timeseries_predict.ipynb".
The first task is:
CNTK Time series prediction with LSTM
This demo demonstrates how to use CNTK to predict future values in a time series using a Recurrent Neural Network (RNN). It is based on a LSTM tutorial that comes with the CNTK distribution.
RNNs are particularly well suited to learn sequence data.
For details on RNNs, see this excellent post.

Goal
We will download stock prices for a chosen symbol, then train a recurrent neural net to predict the closing price on the following  day from $N$ previous days' closing prices.
Our RNN will We will use Long-Short Term Memory LSTM units in the hidden layer. An LSTM network is well-suited to learn from experience to classify, process or predict time series when there are time lags of unknown size between important events.

Organization
The example has the following sections:
- Download and prepare data
- LSTM network modeling
- Model training and evaluation
Can you write Python code for it?

# Standard packages
import math
from matplotlib import pyplot as plt
import numpy as np
import os
import pandas as pd
import time

# Helpers for reading stock prices
import pandas_datareader.data as pdr
import datetime as dt

# Images
from IPython.display import Image

# CNTK packages
import cntk as C
import cntk.axis
from cntk.layers import Input, Dense, Dropout, Recurrence

%matplotlib inline


I am working on the file "LSTM_Timeseries_predict.ipynb".
The first task is:
Execute
Download data, generate the RNN training and evaluation data and visualize
Can you write Python code for it?

symbol = 'MSFT'
start = dt.datetime(2010, 1, 1)
end = dt.datetime(2017, 3, 1)
window = 30

#raw_data = download_data1(symbol=symbol, start=start, end=end)
#rd100 = raw_data['Close']/10.0
raw_data = read_data('milk')
X, Y = generate_RNN_data(raw_data, window)

f, a = plt.subplots(3, 1, figsize=(12, 8))
for j, ds in enumerate(['train', 'val', 'test']):
    a[j].plot(Y[ds], label=ds + ' raw')
[i.legend() for i in a];


I am working on the file "LSTM_Timeseries_predict.ipynb".
The first task is:
Let's look at how the loss function decreases over time to see if the model is converging
Can you write Python code for it?

plt.plot(loss_summary[:,0], loss_summary[:,1], label='training loss');


I am working on the file "LSTM_Timeseries_predict.ipynb".
The first task is:
Network modeling
We setup our network with $N$ LSTM cells, each receiving the single value of our sequence as input at every time step. The $N$ outputs from the LSTM layer are the input into a dense layer that produces a single output. So, we have 1 input, $N$ hidden LSTM nodes and again a single output node.

To train, CNTK will unroll the network over time steps and backpropagate the error over time.

Between LSTM and dense layer we insert a special dropout operation that randomly ignores 20% of the values coming the LSTM during training to prevent overfitting. When using the model to make predictions all values will be retained.
We are only interested in predicting one step ahead when we get to the end of each training sequence, so we use another operator to identify the last item in the sequence before connecting the output layer.
Using CNTK we can easily express our model:
Can you write Python code for it?

def create_model(I, H, O):
    """Create the model for time series prediction"""
    with C.layers.default_options(initial_state = 0.1):
        x = C.layers.Input(I)
        m = C.layers.Recurrence(C.layers.LSTM(H))(x)
        m = C.ops.sequence.last(m)
        m = C.layers.Dropout(0.2)(m)
        m = cntk.layers.Dense(O)(m)

    # Also create a layer to represent the target. It has the same number of units as the output
    # and has to share the same dynamic axes
    y = C.layers.Input(1, dynamic_axes=m.dynamic_axes, name="y")

    return (m, x, y)


I am working on the file "LSTM_Timeseries_predict.ipynb".
The first task is:
Normally we would validate the training on the data that we set aside for validation but since the input data is small we can run validattion on all parts of the dataset.
Can you write Python code for it?

# validate
def get_mse(X,Y,labeltxt):
    result = 0.0
    for x1, y1 in next_batch(X, Y, labeltxt, BATCH_SIZE):
        eval_error = trainer.test_minibatch({x:x1, y:y1})
        result += eval_error
    return result/len(X[labeltxt])

# Print the train and validation errors
for labeltxt in ["train", "val", 'test']:
    print("mse for {}: {:.6f}".format(labeltxt, get_mse(X, Y, labeltxt)))


I am working on the file "LSTM_Timeseries_predict.ipynb".
The first task is:
Quick check on the dimensions of the data + make sure we don't have any NaNs
Can you write Python code for it?

print([(a, X[a].shape) for a in X.keys()])
print([(a, Y[a].shape) for a in Y.keys()])
print([(a, np.isnan(X[a]).any()) for a in X.keys()])
print([(a, np.isnan(Y[a]).any()) for a in Y.keys()])


I am working on the file "LSTM_Timeseries_predict.ipynb".
The first task is:
Select the notebook runtime environment devices / settings
Set the device. If you have both CPU and GPU on your machine, you can optionally switch the devices.
Can you write Python code for it?

# If you have a GPU, uncomment the GPU line below
#C.device.set_default_device(C.device.gpu(0))
C.device.set_default_device(C.device.cpu())


I am working on the file "LSTM_Timeseries_predict.ipynb".
The first task is:
Setup everything else we need for training the model: define user specified training parameters, define inputs, outputs, model and the optimizer.
Can you write Python code for it?

# create the model with 1 input (x), 10 LSTM units, and 1 output unit (y)
(z, x, y) = create_model(1, 10, 1)

# Construct the trainer
BATCH_SIZE = 2
trainer = create_trainer(z, y, learning_rate=0.0002, batch_size=BATCH_SIZE)