I am working on the file "Notebooks/SphericalMeshing/SphericalTriangulations/Ex5-Smoothing.ipynb".
The first task is:
Smoothing operations
The sTriangulation.smoothing method directly wraps the SSRFPack smoother that smooths a surface f described
by values on the mesh vertices to find a new surface f' (also described on the mesh vertices) by choosing nodal function values and gradients to minimize the linearized curvature of F subject to a bound on the deviation from the data values.
```python
help(mesh.smoothing)
```
    smoothing(self, f, w, sm, smtol, gstol)
method of stripy.spherical_meshes.icosahedral_mesh instance
Smooths a surface f by choosing nodal function values and gradients to
minimize the linearized curvature of F subject to a bound on the
deviation from the data values. This is more appropriate than interpolation
when significant errors are present in the data.

Parameters
----------
 f : array of floats, shape (n,)
    field to apply smoothing on
 w : array of floats, shape (n,)
    weights associated with data value in f
    w[i] = 1/sigma_f^2 is a good rule of thumb.
 sm : float
    positive parameter specifying an upper bound on Q2(f).
    generally n-sqrt(2n) <= sm <= n+sqrt(2n)
 smtol : float
    specifies relative error in satisfying the constraint
    sm(1-smtol) <= Q2 <= sm(1+smtol) between 0 and 1.
 gstol : float
    tolerance for convergence.
    gstol = 0.05*mean(sigma_f)^2 is a good rule of thumb.

Returns
-------
 f_smooth : array of floats, shape (n,)
    smoothed version of f
 (dfdx, dfdy, dfdz) : tuple of floats, tuple of 3 shape (n,) arrays
    first derivatives of f_smooth in the x, y, and z directions
Can you write Python code for it?

stripy_smoothed, dds  = mesh.smoothing(analytic_sol_n, np.ones_like(analytic_sol_n), 10.0, 0.1, 0.01)
stripy_smoothed2, dds = mesh.smoothing(analytic_sol_n, np.ones_like(analytic_sol_n), 1.0, 0.1, 0.01)
stripy_smoothed3, dds = mesh.smoothing(analytic_sol_n, np.ones_like(analytic_sol_n), 50.0, 0.1, 0.01)

delta_n  = analytic_sol_n - stripy_smoothed
delta_ns = analytic_sol   - stripy_smoothed

delta_n2  = analytic_sol_n - stripy_smoothed2
delta_ns2 = analytic_sol   - stripy_smoothed2

delta_n3  = analytic_sol_n - stripy_smoothed3
delta_ns3 = analytic_sol   - stripy_smoothed3


I am working on the file "Notebooks/SphericalMeshing/SphericalTriangulations/Ex6-Scattered-Data.ipynb".
The first task is:
Count earthquakes per triangle
This is a numpy wrapper around the STRIPACK routine which operates by retriangulation and is therefore not particularly fast.
Can you write Python code for it?

triangles = mesh.containing_triangle(lons, lats)
tris, counts = np.unique(triangles, return_counts=True)

tris.shape

## map to nodes so we can plot this

hit_count = np.zeros_like(mesh.lons)

for i in range(0, tris.shape[0]):
    hit_count[mesh.simplices[tris[i]]] += counts[i]

hit_count /= 3.0

print(hit_count.mean())

fig = plt.figure(figsize=(10, 10), facecolor="none")
ax  = plt.subplot(111, projection=ccrs.Mollweide())
ax.coastlines(color="lightgrey", )
ax.set_global()

lons0 = np.degrees(mesh.lons)
lats0 = np.degrees(mesh.lats)

ax.scatter(lons0, lats0,
            marker="o", s=30.0, transform=ccrs.Geodetic(), c=hit_count, cmap=plt.cm.Reds, vmin=0.333, vmax=20.0, alpha=0.25)

pass


I am working on the file "Notebooks/SphericalMeshing/SphericalTriangulations/Ex6-Scattered-Data.ipynb".
The first task is:
Count earthquakes per vertex
The sTriangulation.nearest_vertices method uses a k-d tree to find the nearest vertices to a set of longitude / latitude points. It returns the great circle distance. This requires the k-d tree to have been built when the mesh was initialised (tree=True)
Can you write Python code for it?

distances, vertices = mesh.nearest_vertices(lons, lats, k=1)
nodes, ncounts = np.unique(vertices, return_counts=True)

hit_countn = np.zeros_like(mesh.lons)
hit_countn[nodes] = ncounts

fig = plt.figure(figsize=(10, 10), facecolor="none")
ax  = plt.subplot(111, projection=ccrs.Mollweide())
ax.coastlines(color="lightgrey", )
ax.set_global()

lons0 = np.degrees(mesh.lons)
lats0 = np.degrees(mesh.lats)

ax.scatter(lons0, lats0,
            marker="o", s=30.0, transform=ccrs.Geodetic(), c=hit_countn, cmap=plt.cm.Reds, vmin=0.333, vmax=10.0, alpha=0.25)

pass


I am working on the file "Notebooks/SphericalMeshing/SphericalTriangulations/Ex6-Scattered-Data.ipynb".
The first task is:
Mapping data other than frequency to the regular mesh
Here we show how to map point data to the regular mesh - produce a representation of the depth of the events instead of just their frequency. When plotting, we need to distinguish between zero information and zero (shallow) depth. This is done by using the weight function to determine the opacity of the symbol or field that we plot. This has the effect of washing out the regions with few, large events compared to those with many small ones (which in this case means washed out regions where earthquakes are deep).
Can you write Python code for it?

depth_idr = np.zeros_like(mesh.lons)

## numpy shouldn't try to vectorise this reduction operation

for i in range(0,distances.shape[0]):
    depth_idr[vertices[i,:]] += depths[i] * np.exp( -distances[i,:] / 0.02 )

depth_idr[hit_countidr != 0.0] /=  hit_countidr[hit_countidr != 0.0]

fig = plt.figure(figsize=(10, 10), facecolor="none")
ax  = plt.subplot(111, projection=ccrs.Mollweide())
ax.coastlines(color="lightgrey", )
ax.set_global()

lons0 = np.degrees(mesh.lons)
lats0 = np.degrees(mesh.lats)

ax.scatter(lons0, lats0,
            marker="o", transform=ccrs.Geodetic(), c=depth_idr, s=hit_countidr,
            cmap=plt.cm.RdBu, vmin=0.0, vmax=500.0, alpha=0.25)

pass


I am working on the file "Notebooks/SphericalMeshing/SphericalTriangulations/Ex6-Scattered-Data.ipynb".
The first task is:
Point data with uneven spatial distribution
Define a relatively smooth function that we can interpolate from the coarse mesh to the fine mesh and analyse. As it is a familiar pattern, we use the seismic event catalogue for M5.5+ (dawn of time to 2017-12-31) from IRIS
Can you write Python code for it?

import numpy as np

# Note - these data have some places where depth is unknown (appears as NaN in the depth )
# The IRIS data has  lat, lon, depth, mag  ... date/time in col 2, 3, 4, 10 (starting from zero)

eqs = np.genfromtxt("../Data/EQ-M5.5-IRIS-ALL.txt", usecols=(2,3,4,10), delimiter='|', comments="#")

lons = np.radians(eqs[:,1])
lats = np.radians(eqs[:,0])
depths = eqs[:,2]
depths[np.isnan(depths)] = -1.0

%matplotlib inline

import gdal
import cartopy
import cartopy.crs as ccrs
import matplotlib.pyplot as plt

fig = plt.figure(figsize=(10, 5), facecolor="none")
ax  = plt.subplot(111, projection=ccrs.Mollweide())
ax.coastlines(color="#777777" )
ax.set_global()

lons0 = np.degrees(lons)
lats0 = np.degrees(lats)

ax.scatter(lons0, lats0,
            marker="o", s=10.0, alpha=0.5,
            transform=ccrs.Geodetic(), c=depths, cmap=plt.cm.RdBu)

pass


I am working on the file "Notebooks/SphericalMeshing/SphericalTriangulations/Ex6-Scattered-Data.ipynb".
The first task is:
Visualisation
Can you write Python code for it?

import lavavu

depth_range = depths.max()

lv = lavavu.Viewer(border=False, background="#FFFFFF", resolution=[666,666], near=-10.0)

tris = lv.triangles("triangles",  wireframe=False, colour="#77ff88", opacity=1.0)
tris.vertices(mesh.points)
tris.indices(mesh.simplices)

tris.values(hit_count, label="hit_count")
tris.values(hit_countn, label="hit_countn")
tris.values(hit_countid, label="hit_countid")
tris.values(hit_countidr, label="hit_countidr")
tris.colourmap("#FFFFFF:0.0 (1.0)#FF0000:0.2 (50.0)#550044:0.5")

depth = lv.triangles("depth_field",  wireframe=False, colour="#FFFFFF", opacity=0.999)
depth.vertices(mesh.points*(6370-depth_range*0.99) / 6370)
depth.indices(mesh.simplices)
depth.values(depth_idr, label="depths")
depth.values(hit_countidr/hit_countidr.max(), label="weights")
depth.colourmap("#550000 #0099FF #000055")

depth["opacitymap"] = "#000000:0.0 (0.1)#000000:0.9 #000000:0.9"
depth["opacityby"] = "weights"
depth["colourby"]  = "depths"

bg = lv.triangles("background",  wireframe=False, colour="#FFFFFF", opacity=1.0)
bg.vertices(mesh.points*(6370-depth_range) / 6370)
bg.indices(mesh.simplices)

ll = np.array(stripy.spherical.lonlat2xyz(lons, lats)).T

nodes = lv.points("events", pointsize=3.0, pointtype="shiny", colour="#448080", opacity=0.75)
nodes.vertices(ll * (6370.0 - depths.reshape(-1,1)) / 6370.0 )
nodes.values(depths, label="depths")
nodes.colourmap("#550000 #0099FF #000055")

# View from the pacific hemisphere
lv.translation(0.0, 0.0, -3.0)
lv.rotation(0.0,90.0, 90.0)

lv.control.Panel()
lv.control.Range('specular', range=(0,1), step=0.1, value=0.4)
lv.control.Checkbox(property='axis')
lv.control.ObjectList()
tris.control.List(["hit_count", "hit_countn", "hit_countid", "hit_countidr"], property="colourby", value="hit_count", command="redraw")
lv.control.show()


I am working on the file "Notebooks/SphericalMeshing/SphericalTriangulations/Ex7-Refinement-of-Triangulations.ipynb".
The first task is:
Refinement strategies
Five refinement strategies:

Bisect all segments connected to a given node
Refine all triangles connected to a given node by adding a point at the centroid or bisecting all edges
Refune a given triangle by adding a point at the centroid or bisecting all edges

These are provided as follows:
Can you write Python code for it?

mlons, mlats = ico3.midpoint_refine_triangulation_by_vertices(vertices=[1,2,3,4,5,6,7,8,9,10])
ico3mv = stripy.sTriangulation(mlons, mlats)

mlons, mlats = ico3.edge_refine_triangulation_by_vertices(vertices=[1,2,3,4,5,6,7,8,9,10])
ico3ev = stripy.sTriangulation(mlons, mlats)

mlons, mlats = ico3.centroid_refine_triangulation_by_vertices(vertices=[1,2,3,4,5,6,7,8,9,10])
ico3cv = stripy.sTriangulation(mlons, mlats)

mlons, mlats = ico3.edge_refine_triangulation_by_triangles(triangles=[1,2,3,4,5,6,7,8,9,10])
ico3et = stripy.sTriangulation(mlons, mlats)

mlons, mlats = ico3.centroid_refine_triangulation_by_triangles(triangles=[1,2,3,4,5,6,7,8,9,10])
ico3ct = stripy.sTriangulation(mlons, mlats)

print ico3mv.npoints, ico3mv.simplices.shape[0]
print ico3ev.npoints, ico3ev.simplices.shape[0]
print ico3cv.npoints, ico3cv.simplices.shape[0]
print ico3et.npoints, ico3et.simplices.shape[0]
print ico3ct.npoints, ico3ct.simplices.shape[0]


I am working on the file "Notebooks/SphericalMeshing/SphericalTriangulations/Ex7-Refinement-of-Triangulations.ipynb".
The first task is:
Targetted refinement
Here we refine a triangulation to a specific criterion - resolving two points in distinct triangles or with distinct nearest neighbour vertices.
Can you write Python code for it?

points = np.array([[ 0.03, 0.035], [0.05, 0.055]]).T

triangulations = [ico1]
nearest, distances = triangulations[-1].nearest_vertex(points[:,0], points[:,1])

max_depth = 15

while nearest[0] == nearest[1] and max_depth > 0:

    lons, lats = triangulations[-1].centroid_refine_triangulation_by_vertices(vertices=nearest[0])
    new_triangulation = stripy.sTriangulation(lons, lats)
    nearest, distances = new_triangulation.nearest_vertex(points[:,0], points[:,1])
    triangulations.append(new_triangulation)

    max_depth -= 1

print "refinement_steps =", len(triangulations)

centroid_triangulations = triangulations[:]

triangulations = [ico1]
nearest, distances = triangulations[-1].nearest_vertex(points[:,0], points[:,1])

max_depth = 15

while nearest[0] == nearest[1] and max_depth > 0:

    lons, lats = triangulations[-1].edge_refine_triangulation_by_vertices(vertices=nearest[0])
    new_triangulation = stripy.sTriangulation(lons, lats)
    nearest, distances = new_triangulation.nearest_vertex(points[:,0], points[:,1])
    triangulations.append(new_triangulation)

    max_depth -= 1

print "refinement_steps =", len(triangulations)

edge_triangulations = triangulations[:]

triangulations = [ico1]

in_triangle = triangulations[-1].containing_triangle(points[:,0], points[:,1])

max_depth = 100

while in_triangle[0] == in_triangle[1] and max_depth > 0:

    lons, lats = triangulations[-1].edge_refine_triangulation_by_triangles(in_triangle[0])
    new_triangulation = stripy.sTriangulation(lons, lats)
    in_triangle = new_triangulation.containing_triangle(points[:,0], points[:,1])
    triangulations.append(new_triangulation)

    print in_triangle

    if in_triangle.shape[0] == 0:
        break

    max_depth -= 1

print "refinement_steps =", len(triangulations)

edge_t_triangulations = triangulations[:]

triangulations = [ico1]

in_triangle = triangulations[-1].containing_triangle(points[:,0], points[:,1])

max_depth = 100

while in_triangle[0] == in_triangle[1] and max_depth > 0:

   ...


I am working on the file "Notebooks/SphericalMeshing/SphericalTriangulations/Ex7-Refinement-of-Triangulations.ipynb".
The first task is:
Uniform meshes by refinement
The refinement_level parameter of the stripy meshes makes repeated loops determining the bisection points of all the existing edges in the triangulation and then creating a new triangulation that includes these points and the original ones. These refinement operations can also be used for non-uniform refinement.
Can you write Python code for it?

ico0 = stripy.spherical_meshes.icosahedral_mesh(refinement_levels=0)
ico1 = stripy.spherical_meshes.icosahedral_mesh(refinement_levels=1)
ico2 = stripy.spherical_meshes.icosahedral_mesh(refinement_levels=2)
ico3 = stripy.spherical_meshes.icosahedral_mesh(refinement_levels=3)
ico4 = stripy.spherical_meshes.icosahedral_mesh(refinement_levels=4)
ico5 = stripy.spherical_meshes.icosahedral_mesh(refinement_levels=5)
ico6 = stripy.spherical_meshes.icosahedral_mesh(refinement_levels=6)
ico7 = stripy.spherical_meshes.icosahedral_mesh(refinement_levels=7)

print("Size of mesh - 1  {}".format(ico1.points.shape[0]))
print("Size of mesh - 2  {}".format(ico2.points.shape[0]))
print("Size of mesh - 3  {}".format(ico3.points.shape[0]))
print("Size of mesh - 4  {}".format(ico4.points.shape[0]))
print("Size of mesh - 5  {}".format(ico5.points.shape[0]))
print("Size of mesh - 6  {}".format(ico6.points.shape[0]))
print("Size of mesh - 7  {}".format(ico7.points.shape[0]))


I am working on the file "Notebooks/SphericalMeshing/SphericalTriangulations/Ex7-Refinement-of-Triangulations.ipynb".
The first task is:
Visualisation of refinement strategies
Can you write Python code for it?

import matplotlib.pyplot as plt

%matplotlib inline

import gdal

import cartopy
import cartopy.crs as ccrs

def mesh_fig(mesh, meshR, name):

    fig = plt.figure(figsize=(10, 10), facecolor="none")
    ax  = plt.subplot(111, projection=ccrs.Orthographic(central_longitude=0.0, central_latitude=0.0, globe=None))
    ax.coastlines(color="lightgrey")
    ax.set_global()

    generator = mesh
    refined   = meshR

    lons0 = np.degrees(generator.lons)
    lats0 = np.degrees(generator.lats)

    lonsR = np.degrees(refined.lons)
    latsR = np.degrees(refined.lats)

    lst = refined.lst
    lptr = refined.lptr

    ax.scatter(lons0, lats0, color="Red",
                marker="o", s=150.0, transform=ccrs.Geodetic())

    ax.scatter(lonsR, latsR, color="DarkBlue",
                marker="o", s=50.0, transform=ccrs.Geodetic())

    segs = refined.identify_segments()

    for s1, s2 in segs:
        ax.plot( [lonsR[s1], lonsR[s2]],
                 [latsR[s1], latsR[s2]],
                 linewidth=0.5, color="black", transform=ccrs.Geodetic())

    fig.savefig(name, dpi=250, transparent=True)

    return

mesh_fig(ico3,        ico3mv,     "EdgeByVertex1to10" )
mesh_fig(ico3,        ico3ev,     "EdgeByVertexT1to10" )
mesh_fig(ico3,        ico3cv,     "CentroidByVertexT1to10" )
mesh_fig(ico3,        ico3et,     "EdgeByTriangle1to10" )
mesh_fig(ico3,        ico3ct,     "CentroidByTriangle1to10" )


I am working on the file "Notebooks/SphericalMeshing/SphericalTriangulations/Ex7-Refinement-of-Triangulations.ipynb".
The first task is:
Visualisation of targetted refinement
Can you write Python code for it?

import lavavu

## The four different triangulation strategies

t1 = edge_triangulations[-1]
t2 = edge_t_triangulations[-1]
t3 = centroid_triangulations[-1]
t4 = centroid_t_triangulations[-1]

## Fire up the viewer

lv = lavavu.Viewer(border=False, background="#FFFFFF", resolution=[1000,600], near=-10.0)

## Add the nodes to mark the original triangulation

nodes = lv.points("nodes", pointsize=10.0, pointtype="shiny", colour="#448080", opacity=0.75)
nodes.vertices(ico1.points*1.01)
nodes2 = lv.points("SplitPoints", pointsize=2.0, pointtype="shiny", colour="#FF3300", opacity=1.0)
nodes2.vertices(np.array(stripy.spherical.lonlat2xyz(points[:,0], points[:,1])).T * 1.01)

##

tris1w = lv.triangles("t1w",  wireframe=True, colour="#444444", opacity=0.8)
tris1w.vertices(t1.points)
tris1w.indices(t1.simplices)

tris1s = lv.triangles("t1s",  wireframe=False, colour="#77ff88", opacity=0.8)
tris1s.vertices(t1.points*0.999)
tris1s.indices(t1.simplices)

tris2w = lv.triangles("t2w",  wireframe=True, colour="#444444", opacity=0.8)
tris2w.vertices(t2.points)
tris2w.indices(t2.simplices)

tris2s = lv.triangles("t2s",  wireframe=False, colour="#77ff88", opacity=0.8)
tris2s.vertices(t2.points*0.999)
tris2s.indices(t2.simplices)

tris3w = lv.triangles("t3w",  wireframe=True, colour="#444444", opacity=0.8)
tris3w.vertices(t3.points)
tris3w.indices(t3.simplices)

tris3s = lv.triangles("t3s",  wireframe=False, colour="#77ff88", opacity=0.8)
tris3s.vertices(t3.points*0.999)
tris3s.indices(t3.simplices)

tris4w = lv.triangles("t4w",  wireframe=True, colour="#444444", opacity=0.8)
tris4w.vertices(t4.points)
tris4w.indices(t4.simplices)

tris4s = lv.triangles("t4s",  wireframe=False, colour="#77ff88", opacity=0.8)
tris4s.vertices(t4.points*0.999)
tris4s.indices(t4.simplices)

lv.hide("t1s")
lv.hide("t1w")
lv.hide("t2s")
lv.hide("t2w")
lv.hide("t4s")
lv.hide("t4w")

lv.translation(0.0, 0.0, -3.748)
lv.rotation(37.5, -90.0, -37.5)

lv.control.Panel()
lv.control.Button(command="hide triangles; show t1s; show t1w; redraw",...


I am working on the file "Notebooks/Spinning Symmetric Rigid Body.ipynb".
The first task is:
$\left[ \mathbb I_G \right]_\mathcal B$:
Can you write Python code for it?


I am working on the file "Notebooks/Spinning Symmetric Rigid Body.ipynb".
The first task is:
$\left[{}^\mathcal{I} \mathbf h_G\right]_\mathcal{B}$:
Can you write Python code for it?


I am working on the file "Notebooks/Spinning Symmetric Rigid Body.ipynb".
The first task is:
$\left[{}^\mathcal{I}\boldsymbol{\omega}^\mathcal{B}\right]_\mathcal{B}$:
Can you write Python code for it?

iWb_B = bCa*Matrix([0,0,psid])+ Matrix([thd,0,0]); iWb_B


I am working on the file "Notebooks/Spinning Symmetric Rigid Body.ipynb".
The first task is:
$\vphantom{\frac{\mathrm{d}}{\mathrm{d}t}}^\mathcal{I}\frac{\mathrm{d}}{\mathrm{d}t}  {}^\mathcal{I} \mathbf h_G = \vphantom{\frac{\mathrm{d}}{\mathrm{d}t}}^\mathcal{B}\frac{\mathrm{d}}{\mathrm{d}t}  {}^\mathcal{I} \mathbf h_G + {}^\mathcal{I}\boldsymbol{\omega}^\mathcal{B} \times  \mathbf h_G$.
$\left[\vphantom{\frac{\mathrm{d}}{\mathrm{d}t}}^\mathcal{I}\frac{\mathrm{d}}{\mathrm{d}t}  {}^\mathcal{I} \mathbf h_G\right]_\mathcal{B}$:
Can you write Python code for it?

dhG_B = difftotalmat(hG_B,t,diffmap) + skew(iWb_B)*hG_B; dhG_B


I am working on the file "Notebooks/Spinning Symmetric Rigid Body.ipynb".
The first task is:
${}^\mathcal{I}\boldsymbol{\omega}^\mathcal{C} = {}^\mathcal{I}\boldsymbol{\omega}^\mathcal{B} + {}^\mathcal{B}\boldsymbol{\omega}^\mathcal{C}$.
$\left[{}^\mathcal{I}\boldsymbol{\omega}^\mathcal{C}\right]_\mathcal{B}$:
Can you write Python code for it?

iWc_B = iWb_B +Matrix([0,Omega,0]); iWc_B


I am working on the file "Notebooks/Spinning Symmetric Rigid Body.ipynb".
The first task is:
Assume an external torque generating moment about $G$ of $\mathbf M_G = -M_1\mathbf b_1$:
Can you write Python code for it?


I am working on the file "Notebooks/Spinning Symmetric Rigid Body.ipynb".
The first task is:
Spinning Symmetric Rigid Body setup:  The body's orientation in inertial frame $\mathcal I$ is described by a 3-1-2 $(\psi,\theta,\phi)$ rotation:  the body is rotated by angle $\psi$ about $\mathbf e_3$ (creating intermediate frame $\mathcal A$), by an angle $\theta$ about $\mathbf a_1$ (creating intermediate frame $\mathcal B$), and finally spinning about $\mathbf b_2 \equiv \mathbf c_2$ at a rate $\Omega \equiv \dot\phi$, creating body-fixed frame $\mathcal C$. Note that while the body fixed frame is $\mathcal C$, all computations here are performed in $\mathcal B$ frame components.
${}^\mathcal{B}C^\mathcal{A}$:
Can you write Python code for it?


I am working on the file "Notebooks/State figures.ipynb".
The first task is:
Largest and smallest changes
Can you write Python code for it?

barbell_index['change'] = barbell_index[2017] - barbell_index[2001]
barbell_index['% change'] = barbell_index['change'] / barbell_index[2001]

max_change_state = barbell_index.sort_values('change').index[0]
max_change_value = barbell_index.sort_values('change')['change'].values[0]
start = barbell_index.loc[max_change_state, 2001]
end = barbell_index.loc[max_change_state, 2017]
print('The largest absolute reduction is {:0.1f} g/kWh in {}, from {:.1f} to {:.1f}'
      .format(max_change_value, max_change_state, start, end))

min_change_state = barbell_index.sort_values('change').index[-1]
min_change_value = barbell_index.sort_values('change')['change'].values[-1]
start = barbell_index.loc[min_change_state, 2001]
end = barbell_index.loc[min_change_state, 2017]
print('The smallest absolute reduction is {:0.1f} g/kWh in {}, from {:.1f} to {:.1f}'
      .format(min_change_value, min_change_state, start, end))

max_relchange_state = barbell_index.sort_values('% change').index[0]
max_relchange_value = barbell_index.sort_values('% change')['% change'].values[0]
start = barbell_index.loc[max_relchange_state, 2001]
end = barbell_index.loc[max_relchange_state, 2017]
print('The largest relative reduction is {:0.1%} g/kWh in {}, from {:.1f} to {:.1f}'
      .format(max_relchange_value, max_relchange_state, start, end))

min_relchange_state = barbell_index.sort_values('% change').index[-1]
min_relchange_value = barbell_index.sort_values('% change')['% change'].values[-1]
start = barbell_index.loc[min_relchange_state, 2001]
end = barbell_index.loc[min_relchange_state, 2017]
print('The smallest relative reduction is {:0.1%} g/kWh in {}, from {:.1f} to {:.1f}'
      .format(min_relchange_value, min_relchange_state, start, end))


I am working on the file "Notebooks/State figures.ipynb".
The first task is:
State CO₂ intensity changes
Can you write Python code for it?

barbell_index = annual_index.pivot_table(values='index (g/kwh)',
                                         index='state', columns='year')
barbell_index.sort_values(by=[2017], inplace=True)
barbell_index.head()

len([x for x in rps_start.values() if x <=2008])

rps_start

states_index = list(barbell_index.index)

dumbell_plot(barbell_index, [2001, 2008, 2017], states_index,
             legend_loc=[500, 300, 100], rps_legend=True,
             rps_start=rps_start, palette='colorblind')

plt.vlines(439, -1, 50, colors=['0.5'], zorder=1, #linestyles='dashed',
           linewidth=1)

plt.ylim(-1, 53)
# plt.text(x=200, y=40, s='2017\nNational Average\n(439 g CO$_2$ $\mathregular{kWh^{-1}}$)',
#          ha='center', va='center', size=11)
plt.text(x=680, y=3, s='2017\nNational Average\n(439 g CO$_2$ $\mathregular{kWh^{-1}}$)',
         ha='center', va='center', size=11)
sns.despine(left=True)
plt.xlabel('g CO$_2$ $\mathregular{kWh^{-1}}$')

# label as part of a larger figure
plt.text(-0.02, 0.89, s='a)', ha='right', size=11, transform=ax.transAxes)

path = join(cwd, '..', 'Figures',
            'State CO2 intensity {}.pdf'.format(file_date))
plt.savefig(path, bbox_inches='tight')

barbell_index.loc['USA', 2001] = 630
barbell_index.loc['USA', 2008] = 580
barbell_index.loc['USA', 2017] = 439
barbell_index.sort_values(by=[2017], inplace=True)

states_index = list(barbell_index.index)

dumbell_plot(barbell_index, [2001, 2008, 2017], states_index,
             legend_loc=[500, 300, 100],
             rps_start=rps_start, palette='colorblind')

plt.vlines(439, -1, 50, colors=['0.5'], zorder=1, #linestyles='dashed',
           linewidth=1)

plt.ylim(-1, 53)
plt.text(x=200, y=40, s='2017\nNational Average\n(439 g CO$_2$ $\mathregular{kWh^{-1}}$)',
         ha='center', size=11)
sns.despine(left=True)
plt.xlabel('g CO$_2$ $\mathregular{kWh^{-1}}$')
path = join(cwd, '..', 'Figures',
            'State CO2 intensity {}.pdf'.format('David Hawkins'))
plt.savefig(path, bbox_inches='tight')


I am working on the file "Notebooks/State figures.ipynb".
The first task is:
State electricity imports/exports
Data is from EIA State Energy Data System. Most recent update was March 9, 2018. Code are described in EIA documentation. Negative values indicate flows out of the state.
Can you write Python code for it?

path = os.path.join(base_path, 'Data storage', 'State energy flows',
                    'use_all_phy_update.csv')
flows = pd.read_csv(path)

# ELISP is electricity flows
flows = flows.loc[flows['MSN'] == 'ELISP']

# Drop all years before 2001
flows.drop([str(x) for x in range(1960, 2001)], axis=1, inplace=True)
flows.drop(['Data_Status', 'MSN'], axis=1, inplace=True)

# Electricity flows are given in million kWh (see documentation)
flows.loc[:, '2001':] *= 1000

flows.columns = flows.columns.str.lower()

flows = flows.melt(id_vars='state', var_name='year',
                   value_name='MWh flow')
flows['year'] = flows['year'].astype(int)

flows.tail()

annual_index_pop_flows = pd.merge(annual_index_pop, flows, on=['state', 'year'])
annual_index_pop_flows['MWh flow/pop'] = (annual_index_pop_flows['MWh flow']
                                          / annual_index_pop_flows['population'])
annual_index_pop_flows['share flow'] = (annual_index_pop_flows['MWh/pop']
                                          / annual_index_pop_flows['MWh flow/pop'])

annual_index_pop_flows.tail()

barbell_flows = annual_index_pop_flows.pivot_table(values='MWh flow/pop',
                                              index='state', columns='year')

barbell_flows.head()

barbell_flows = annual_index_pop_flows.pivot_table(values='MWh flow/pop',
                                              index='state', columns='year')
barbell_flows.sort_values(by=2015, ascending=False, inplace=True)
# barbell_flows.drop('VT', inplace=True)

states_flows = list(barbell_flows.index)

dumbell_plot(barbell_flows, [2001, 2008, 2015], states_flows, offset_divider=8,
             rps_start=rps_start, legend=True, palette='colorblind',
             legend_loc=[-20, -40, -60])

plt.ylim(-1, 53)
sns.despine(left=True)
plt.xlabel('MWh flow/Capita')
plt.vlines(0, -1, 53, colors=['0.5'], linewidth=1)
# plt.xticks()
# ax = plt.gca()
# ax.set_xscale("log", nonposx='clip')
path = join(fig_export_path, 'SI', 'State MWh flow per capita.pdf')
# plt.savefig(path,...


I am working on the file "Notebooks/Subtraction/etmy_l1.ipynb".
The first task is:
The subtraction works pretty well. I adjusted the gain until the calibration lines were as low as I could get them (because they're the strongest feature and are well above the noise). The subtraction only seems to be good to about 1 part in 10. Probably the noisemon has some extra phase or the zeros/poles are not exact. This should be good enough for our purposes.
Now we go through a few loud glitches and do this subtraction. In each case, the loud glitch in DARM, which sent a large feedback to the L1 coil, is easily visible in the noisemon. But after subtraction, there is almost nothing left in the noisemon. This means that the voltage sent to the coil (which the noisemon witnesses) is exactly what was requested of the DAC. There's no glitch coming from the coil electronics.
Note: I've also checked this for the other three quadrants. The same gain work for all of them.
Can you write Python code for it?

def plot_glitch(glitch, quadrant='LR'):
    nmon, proj, drive = project_out_drive(glitch, quadrant=quadrant)
    p1 = nmon.highpass(8).crop(glitch-1, glitch+1).plot(label='Original')
    p1.gca().plot((nmon-proj).highpass(8).crop(glitch-1, glitch+1),label='Drive removed')
    p1.gca().legend()

plot_glitch(1183061141.71)

plot_glitch(1183077855.74)

plot_glitch(1183749148.39)

plot_glitch(1184624917.96)

plot_glitch(1184972637.76)


I am working on the file "Notebooks/Subtraction/etmy_l1.ipynb".
The first task is:
There is a document in the DCC which give the ZPK filter for the noisemon circuit: LIGO-T1100378
There are four zeros at 0, four poles at 5 Hz, and two poles at 4000 Hz.
I just eyeballed the gain by matching the ASDs of the filtered drive signal and the noisemon.
Passing the drive signal (e.g. ETMY_L1_MASTER_LR) through this filter, you get the predicted noisemon signal. The noisemon at th L1 stage is mostly limited by noise, but you can see things like the calibration lines stick up.
Subtracting the filtered drive signal from the noisemon gives the voltage going to the coil which was not 'requested' by the drive/feedback.
Can you write Python code for it?

def project_out_drive(glitch_time, dur=64, level='L1', quadrant='LR', gain=3.5e10):
    pad = 8
    gt = int(round(glitch_time))
    st=gt-dur/2
    et=gt+dur
    drive = TimeSeries.fetch('L1:SUS-ETMY_%s_MASTER_OUT_%s_DQ' % (level, quadrant),
                                st-pad, et)
    nmon = TimeSeries.fetch('L1:SUS-ETMY_%s_NOISEMON_%s_OUT_DQ' % (level, quadrant),
                                st, et)
    proj = drive.zpk([0,0,0,0], [5,5,5,5,4e3,4e3], gain).crop(st,et)
    return nmon, proj, drive.crop(st, et)

glitch = 1183077855.74

nmon, proj, drive = project_out_drive(1183077600, dur=300, quadrant='UR')

p1 = nmon.asd(16, 8).plot(label='Noisemon')
p1.gca().plot(proj.asd(16, 8),label='Projection')
p1.gca().plot((nmon-proj).asd(16, 8),label='Subtraction')
p1.gca().legend(loc='lower left')
p1.gca().set_ylim(1e-2,10)
p1.gca().set_xlim(10,30)


I am working on the file "Notebooks/Supervised/Supervised Example.ipynb".
The first task is:
And again for the SVM classifier.
Can you write Python code for it?

# Compute confusion matrix
svm_cnf_matrix = confusion_matrix( test['isMalicious'], test['svm-predictions'])
np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(svm_cnf_matrix, classes=['Not Malicious', 'Malicious'],
                      title='SVM Confusion matrix, without normalization')

# Plot normalized confusion matrix
plt.figure()
plot_confusion_matrix(svm_cnf_matrix, classes=['Not Malicious', 'Malicious'], normalize=True,
                      title='SVM Normalized confusion matrix')

plt.show()


I am working on the file "Notebooks/Supervised/Supervised Example.ipynb".
The first task is:
Confusion Matrix
These are a little confusing (yuk yuk), but are a very valuable tool in evaluating your models.  Scikit-learn has a function to generate a confusion matrix as shown below.
Can you write Python code for it?


I am working on the file "Notebooks/Supervised/Supervised Example.ipynb".
The first task is:
Perform the Train/Test Split
For this, let’s create a rather small training data se as it will reduce the time to train up a model.
Feel free to try a 15%, 20% or even a 30% portion for the training data (lower percentages for slower machines).
In this example, we will split 30% for train and 70% for test.
Normally you would want most of the data in the training data, but more training data can considerably extend the time neede to train up a model.
We're also going to need a list of column names for the feature columns as well as the target column.
Can you write Python code for it?

train, test = train_test_split(df, test_size = 0.7)
features = ['length', 'dicts', 'entropy','numbers', 'ngram']
target = 'isMalicious'


I am working on the file "Notebooks/Supervised/Supervised Example.ipynb".
The first task is:
The code below generates a nicer presentation of the confusion matrix for the random forest classifer.
From: http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html#sphx-glr-auto-examples-model-selection-plot-confusion-matrix-py
Can you write Python code for it?

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

# Compute confusion matrix
cnf_matrix = confusion_matrix( test['isMalicious'], test['predictions'])
np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=['Not Malicious', 'Malicious'],
                      title='RF Confusion matrix, without normalization')

# Plot normalized confusion matrix
plt.figure()
plot_confusion_matrix(cnf_matrix, classes=['Not Malicious', 'Malicious'], normalize=True,
                      title='RF Normalized confusion matrix')

plt.show()


I am working on the file "Notebooks/Supervised/Supervised Example.ipynb".
The first task is:
You can also visualize this with the following code from: #From: http://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html
Can you write Python code for it?

std = np.std([random_forest_clf.feature_importances_ for tree in random_forest_clf.estimators_],
             axis=0)
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")

for f in range(test[features].shape[1]):
    print("%d. feature %d (%f)" % (f + 1, indices[f], importances[indices[f]]))

# Plot the feature importances of the forest
plt.figure()
plt.title("Feature importances")
plt.bar(range(test[features].shape[1]), importances[indices],
       color="r", yerr=std[indices], align="center")
plt.xticks(range(test[features].shape[1]), indices)
plt.xlim([-1, test[features].shape[1]])
plt.show()


I am working on the file "Notebooks/Supervised/Supervised Example.ipynb".
The first task is:
You can calculate the accuracy with the metrics.accuracy() method, and finally, there is the metrics.classification-report() which will calculate all the metrics except accuracy at once.
Can you write Python code for it?

pscore = metrics.accuracy_score(test['isMalicious'], test['predictions'])
pscore_train = metrics.accuracy_score(train['isMalicious'], train['predictions'])

print( metrics.classification_report(test['isMalicious'], test['predictions'], target_names=['Malicious', 'Not Malicious'] ) )

svm_pscore = metrics.accuracy_score(test['isMalicious'], test['svm-predictions'])
svm_pscore_train = metrics.accuracy_score(train['isMalicious'], train['svm-predictions'])
print( metrics.classification_report(test['isMalicious'], test['svm-predictions'], target_names=['Malicious', 'Not Malicious'] ) )

print( svm_pscore, svm_pscore_train)

print( pscore, pscore_train)


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
5. Visualize the Categorical Features
Can you write Python code for it?

# Remind ourselves of the categorical features in the dataset
[feature_list[feature_index] for feature_index in categorical]


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
8. Find and Handle Missing Values
Can you write Python code for it?

# For each column in the dataset, add up the rows in which the column data is missing
# You can do this across the entire dataset for a quick look at what's missing
df_churn.isnull().sum()


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
A Simple Display of Slices of the Dataset
Can you write Python code for it?

# Display the elements
HBox([VBox([dataFrame, allFeatures, button]), VBox([startRow, endRow])])


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
Box Plots
Can you write Python code for it?

fig, (ax1,ax2,ax3) = plt.subplots(nrows=1, ncols=3, figsize=(12,4))
sns.boxplot(x=df_churn['tenure'], ax = ax1, palette='Set1')
#ax1.set_title("Tenure")
sns.boxplot(x=df_churn['MonthlyCharges'], ax = ax2, palette='Set2')
#ax2.set_title("Monthly Charge")
# There are some monthly charges missing -- supress for now because we haven't yet handled missing values
#sns.boxplot(x=df_churn['TotalCharges'], ax = ax3, palette='Set3')
#ax3.set_title("Total Charge")
plt.tight_layout()


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
Display Categorical Features
Can you write Python code for it?

# Show a default plot
plotFeatureCount(w_features.value, w_radio.value)
# Show the widgets
HBox([w_features, w_radio])


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
Do senior citizens pay more per month?
Can you write Python code for it?

# Pivots
# Do monthly charges depend on gender? They don't seem to; but they do seem to depend on whether
# or not the person is a senior citizen -- 0 = Not a senior citizen, 1 = senior citizen
fig, ax = plt.subplots(figsize=(10,7))
sns.boxplot(x='gender', y='MonthlyCharges', hue="SeniorCitizen", data=df_churn, palette='Set3')
plt.legend(loc='upper right')


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
Does tenure in addition to gender affect churn?
Can you write Python code for it?

# Facet Plots
g = sns.FacetGrid(df_churn, row='Churn', col='gender', margin_titles=True)
g.map(sns.distplot, 'tenure')


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
EXERCISE 1
How balanced are the categorical features? Do you anticipate any problems using this dataset to predict if a customer will switch (be subject to churn) or not?
6. Visualize Relationships Between Categorical Features
Does gender make a difference for churn?
Can you write Python code for it?

# One way to visualize the effect of gender on churn -- i.e. interdependence between variables
from plotnine import *
(ggplot(df_churn, aes(x='Churn', fill='gender')) + geom_bar(position='fill'))


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
Histogram
Can you write Python code for it?

# Set up the plot
def plotFeatureHist(feature_name):
    fig, ax = plt.subplots(figsize=(12,7))
    sns.distplot(df_churn[feature_name], kde=False)
    return plt.show()

# How are the monthly charges distributed?
plotFeatureHist('MonthlyCharges')


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
How are the numerical attributes correlated?
Can you write Python code for it?

# Calculate the correlation table
# Not sure why SeniorCitizen appears but TotalCharges doesn't appear.
corr = df_churn.corr()
corr

# Correlation Density Plot
feature_display_names = ['Tenure', 'Monthly Charges', 'Total Charges']
cm = df_churn.corr()
sns.set(font_scale=1)
# NOTE: fmt directive controls number of decimal points displayed
hm = sns.heatmap(cm,
                cbar=True,
                annot=True,
                square=False,
                fmt='.2f',
                annot_kws={'size':14},
                yticklabels=feature_display_names,
                xticklabels=feature_display_names)
plt.title('Correlation of Numerical Features')


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
It turns out that TotalCharges are empty when tenure = 0. These rows do have a monthly charge. We'll just make the total charge equal the monthly charge in these cases.
Can you write Python code for it?

df_churn.iloc[488]['MonthlyCharges']

# Get the monthly charges for these rows
monthly_charges = [df_churn.iloc[loc]['MonthlyCharges'] for loc in empty[2][1]]
monthly_charges
#df_churn.set_value(488, 'TotalCharges', 52.555)
#df_churn.iloc[488]

# For all customers whose tenure is 0 months, set the TotalCharges equal to the MonthlyCharges
# This is the new dataset
[df_churn.set_value(loc, 'TotalCharges', df_churn.iloc[loc]['MonthlyCharges']) for loc in empty[2][1]]
df_churn.shape

# Let's check if the values of TotalCharges are as they should be.
df_churn.iloc[empty[2][1]][['MonthlyCharges','TotalCharges']]


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
Notice that the SeniorCitizen attribute or feature is respresented numerically as a 1 or 0 -- but these numbers actually represent "Yes" or "No". In othere words, SeniorCitizen is a categorical attribute.
3. Visualize the Numerical Features
SIDEBAR
Let's build ourselves a handy way to look at any set of attributes we choose. We can use this to isolate and explore various groups of attributes.
Can you write Python code for it?

# View a selection of rows and columns in the df_churn dataframe
df_map = {'telco churn data': df_churn}
# We're calling our dataset 'telco churn data'
def table_view(data_frame_name, feature_list, start_row=3, end_row=5):
    '''
    Displays selected columns and rows of a data frame.
    '''
    # Verify the inputs are sane
    # get the size of the dataframe
    num_rows, num_cols = df_map[data_frame_name].shape
    if (start_row < 0) | (start_row > num_rows) :
        return print("Please use a valid Start Row number. It can be any number from 0 to {}".format(num_rows))
    if (end_row < 0) | (end_row > num_rows) | (end_row < start_row + 1):
        return print("Please use a valid End Row number. \
                     It can be any number from 0 to {} \
                     and must be greater than your start row number".format(num_rows))

    view = df_map[data_frame_name][feature_list].iloc[start_row:end_row]

    return view

# Using the table_view function defined above
# The name of our dataset (as defined above) is 'telco churn data'
# You can select any and any number of attributes in selected_columns
selected_columns = ['tenure', 'SeniorCitizen', 'MonthlyCharges', 'TotalCharges']
table_view('telco churn data', selected_columns, 10, 14)

# We can  explore views in an interactive way
from __future__ import print_function
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
from ipywidgets import Button, HBox, VBox
from IPython.display import display
from IPython.display import clear_output

# Layout the interactive view widgets

# Data Frame Chooser Dropdown
dataFrame = widgets.Select(
             options=['telco churn data'],
             value='telco churn data',
             description='Data Source:',
             disabled=False
            )

# Start Row Text Field
startRow = widgets.IntText(
            value=7,
            description='Start Row:',
            disabled=False
           )

# End Row Text Field
endRow = widgets.IntText(
 ...


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
Summary Statistics
Can you write Python code for it?

# Here are summary statistics - rough format but still useful
#df_churn['TotalCharges'].astype(float)
df_churn[['tenure', 'MonthlyCharges', 'TotalCharges']].describe(include='all')


I am working on the file "Notebooks/Systematic-Approach-to-Visualizing-Data.ipynb".
The first task is:
The zeros mean there are no missing data values. This is very nice, but unusual in most datasets, so we've lucked out. When there are missing values it takes effort and judgement to decide how to handle them. Sometimes it's not clear how to handle missing data even though there are a number of standard techniques to choose from.
One thing to watch out for is a value that seems to be missing, except that it really is an empty string like '' or a string with some spaces such as '   '. These usually trip up the plotting functions and that's one (stressful) way to identify them.
Can you write Python code for it?

# For each of the features, find rows where they might be empty -- we'll have to handle these appropriately
def isEmpty(feature):
    empty_rows = []
    for i in range(len(df_churn)):
        if isinstance(df_churn[feature][i], str):
            empty_rows.append(i)

    return empty_rows

# Which of the numerical features have no numerical values?
empty = [[feature,isEmpty(feature)] for feature in numeric_features]
empty

# Let's have a look at these rows where one or more numberical features are empty.
df_churn.iloc[empty[2][1]]


I am working on the file "Notebooks/Testing_Training_Quadgram.ipynb".
The first task is:
<u> For Computing Interpolated Probability</u>
Can you write Python code for it?

#returns: float
#arg: float,float,float,float,list,list,dict,dict,dict,dict
#for calculating the interpolated probablity
def interpolatedProbability(quad_token,token_len, vocab_dict, bi_dict, tri_dict, quad_dict,
                            l1 = 0.25, l2 = 0.25, l3 = 0.25 , l4 = 0.25):

    sen = ' '.join(quad_token)
    prob =(
              l1*(quad_dict[sen] / tri_dict[' '.join(quad_token[0:3])])
            + l2*(tri_dict[' '.join(quad_token[1:4])] / bi_dict[' '.join(quad_token[1:3])])
            + l3*(bi_dict[' '.join(quad_token[2:4])] / vocab_dict[quad_token[2]])
            + l4*(vocab_dict[quad_token[3]] / token_len)
          )
    return prob


I am working on the file "Notebooks/Testing_Training_Quadgram.ipynb".
The first task is:
<u>Do preprocessing</u>:
Remove the punctuations and lowercase the tokens
Can you write Python code for it?

#returns: string
#arg: string
#remove punctuations and make the string lowercase
def removePunctuations(sen):

      #split the string into word tokens
    temp_l = sen.split()
    i = 0

    #changes the word to lowercase and removes punctuations from it
    for word in temp_l :
        for l in word :
            if l in string.punctuation:
                word = word.replace(l," ")
        temp_l[i] = word.lower()
        i=i+1

    #spliting is being don here beacause in sentences line here---so after punctuation removal it should
    #become "here so"
    content = " ".join(temp_l)

    return content


I am working on the file "Notebooks/Testing_Training_Quadgram.ipynb".
The first task is:
<u>Driver Function for Testing the Language Model</u>
Can you write Python code for it?

#return: void
#arg:string,string,dict,dict,dict,dict,dict
#Used for testing the Language Model
def trainCorpus(train_file,test_file,bi_dict,tri_dict,quad_dict,vocab_dict,prob_dict):

    test_result = ''
    score = 0
    #load the training corpus for the dataset
    token_len = loadCorpus('training_corpus.txt',bi_dict,tri_dict,quad_dict,vocab_dict)
    print("---Processing Time for Corpus Loading: %s seconds ---" % (time.time() - start_time))

    start_time1 = time.time()

    #creates a dictionary of probable words
    createProbableWordDict(bi_dict,tri_dict,quad_dict,prob_dict,vocab_dict,token_len)
    #sort the dictionary of probable words
    sortProbWordDict(prob_dict)
    gc.collect()
    print("---Processing Time for Creating Probable Word Dict: %s seconds ---" % (time.time() - start_time1))

    test_data = ''
    #Now load the test corpus
    with open('testing_corpus.txt','r') as file :
        test_data = file.read()

    #remove punctuations from the test data
    test_data = removePunctuations(test_data)
    test_token = test_data.split()

    #split the test data into 4 words list
    test_token = test_data.split()
    test_quadgrams = list(ngrams(test_token,4))

    #print(len(test_token))
    start_time1 = time.time()
    score = computeTestScore(test_quadgrams,tri_dict,quad_dict,vocab_dict,prob_dict)
    print('Score:',score)
    print("---Processing Time for computing score: %s seconds ---" % (time.time() - start_time1))

    start_time2 = time.time()
    perplexity = computePerplexity(test_token,token_len,tri_dict,quad_dict,vocab_dict,prob_dict)
    print('Perplexity:',perplexity)
    print("---Processing Time for computing Perplexity: %s seconds ---" % (time.time() - start_time2))

    test_result += 'TEST RESULTS\nScore: '+str(score) + '\nPerplexity: '+str(perplexity)
    with open('test_results.txt','w') as file:
      	file.write(test_result)


I am working on the file "Notebooks/Testing_Training_Quadgram.ipynb".
The first task is:
<u>For Taking input from the User</u>
Can you write Python code for it?

#returns: string
#arg: void
#for taking input from user
def takeInput():
    cond = False
    #take input
    while(cond == False):
        sen = input('Enter the string\n')
        sen = removePunctuations(sen)
        temp = sen.split()
        if len(temp) < 3:
            print("Please enter atleast 3 words !")
        else:
            cond = True
            temp = temp[-3:]
    sen = " ".join(temp)
    return sen


I am working on the file "Notebooks/Testing_Training_Quadgram.ipynb".
The first task is:
<u>main function</u>
Can you write Python code for it?

"""
def main():

    #variable declaration
    tri_dict = defaultdict(int)            #for keeping count of sentences of three words
    quad_dict = defaultdict(int)           #for keeping count of sentences of three words
    vocab_dict = defaultdict(int)          #for storing the different words with their frequencies
    prob_dict = OrderedDict()              #for storing the probabilities of probable words for a sentence
    bi_dict = defaultdict(int)

    #load the corpus for the dataset
    loadCorpus('corpusfile.txt',tri_dict,quad_dict,vocab_dict)
    print("---Preprocessing Time for Corpus loading: %s seconds ---" % (time.time() - start_time))

    start_time1 = time.time()

    #creates a dictionary of probable words
    createProbableWordDict(tri_dict,quad_dict,prob_dict)
    #sort the dictionary of probable words
    sortProbWordDict(prob_dict)

    gc.collect()
    print("---Preprocessing Time for Creating Probable Word Dict: %s seconds ---" % (time.time() - start_time1))

    input_sen = takeInput()

    start_time2 = time.time()
    prediction = doPrediction(input_sen,prob_dict)
    print('Word Prediction:',prediction)
    print("---Time for Prediction Operation: %s seconds ---" % (time.time() - start_time2))
""""

"""
if __name__ == '__main__':
    main()
"""


I am working on the file "Notebooks/Testing_Training_Quadgram.ipynb".
The first task is:
Create a Hash Table for Probable words for Trigram sentences
Can you write Python code for it?

#returns: void
#arg: dict,dict,dict,dict,dict,int
#creates dict for storing probable words with their probabilities for a trigram sentence
def createProbableWordDict(bi_dict,tri_dict,quad_dict,prob_dict,vocab_dict,token_len):
    for quad_sen in quad_dict:
        prob = 0.0
        quad_token = quad_sen.split()
        tri_sen = ' '.join(quad_token[:3])
        tri_count = tri_dict[tri_sen]

        if tri_count != 0:
            prob = interpolatedProbability(quad_token,token_len, vocab_dict, bi_dict, tri_dict, quad_dict,
                            l1 = 0.25, l2 = 0.25, l3 = 0.25 , l4 = 0.25)

        if tri_sen not in prob_dict:
            prob_dict[tri_sen] = []
            prob_dict[tri_sen].append([prob,quad_token[-1]])
        else:
            prob_dict[tri_sen].append([prob,quad_token[-1]])

    prob = None
    tri_count = None
    quad_token = None
    tri_sen = None


I am working on the file "Notebooks/Testing_Training_Quadgram.ipynb".
The first task is:
Sort the probable words
Can you write Python code for it?

#returns: void
#arg: dict
#for sorting the probable word acc. to their probabilities
def sortProbWordDict(prob_dict):
    for key in prob_dict:
        if len(prob_dict[key])>1:
            sorted(prob_dict[key],reverse = True)


I am working on the file "Notebooks/Testing_Training_Quadgram.ipynb".
The first task is:
Tokenize and load the corpus data
Can you write Python code for it?

#returns : void
#arg: string,dict,dict,dict,dict
#loads the corpus for the dataset and makes the frequency count of quadgram and trigram strings
def loadCorpus(file_path,bi_dict,tri_dict,quad_dict,vocab_dict):

    w1 = ''    #for storing the 3rd last word to be used for next token set
    w2 = ''    #for storing the 2nd last word to be used for next token set
    w3 = ''    #for storing the last word to be used for next token set
    token = []
    word_len = 0

    #open the corpus file and read it line by line
    with open(file_path,'r') as file:
        for line in file:

            #split the line into tokens
            token = line.split()

            i = 0
            #for each word in the token list ,remove pucntuations and change to lowercase
            for word in token :
                for l in word :
                    if l in string.punctuation:
                        word = word.replace(l," ")
                token[i] = word.lower()
                i += 1

            #make the token list into a string
            content = " ".join(token)
            token = content.split()
            word_len = word_len + len(token)

            if not token:
                continue

            #add the last word from previous line
            if w3!= '':
                token.insert(0,w3)

            temp0 = list(ngrams(token,2))

            #since we are reading line by line some combinations of word might get missed for pairing
            #for trigram
            #first add the previous words
            if w2!= '':
                token.insert(0,w2)

            #tokens for trigrams
            temp1 = list(ngrams(token,3))

            #insert the 3rd last word from previous line for quadgram pairing
            if w1!= '':
                token.insert(0,w1)

            #add new unique words to the vocaulary set if available
            for word in token:
                if word not in vocab_dict:
                    vocab_dict[word] = 1
                else:
            ...


I am working on the file "Notebooks/Tilting_Estimates_of_ATT.ipynb".
The first task is:
Computing the AST estimates of the ATT using the PSID controls
This next snippet of code computes the ATT estimate using the PSID controls. This shows how use the ipt.att() function. The vector $ t(W) $ is specified to include earnings in both 1974 and 1975, as well as their squares and cross product. This choice ensures exact balancing of the first two moments of the pre-treatment earnings distributions across treated and controls. I additionally include in $t(W)$ age, education and the black and hispanic dummies. I rescale some of these variables to make computation easier (which is good to do in practice). The constructed auxiliary tilt satisfies the exact balancing conditioning to (at least) four decimal places. I also set $r(W) = t(W)$
Can you write Python code for it?

# Treatment indicator
D = nsw_obs['treat']

# Propensity score model
h_W = nsw_obs[['constant','black','hispanic','age','married','nodegree',\
               're74','re75','re74_X_re75','ue74','ue75','ue74_X_ue75']]

# Balancing moments
t_W = h_W

# Outcome
Y = nsw_obs['re78']

# Compute AST estimate of ATT
[gamma_ast, vcov_gamma_ast, pscore_tests, tilts, exitflag] = \
                                            ipt.att(D, Y, h_W, t_W, study_tilt = False, rlgrz = 1/2, \
                                                    s_wgt=None, nocons=True, c_id=None, silent=False)

# Compute log of auxiliary/control sample normalized weights to construct histogram below
pi_a = tilts[:,2]               # Auxiliary tilt is the last columnn in np.array tilts
b = np.where(1-D)[0]            # find indices of control units
Na = len(pi_a[b])
log_wgts = np.log(Na*pi_a[b])   # log of normalized weights to undo right skewness for plotting purposes


I am working on the file "Notebooks/Tilting_Estimates_of_ATT.ipynb".
The first task is:
Experimental benchmark estimate of the ATT
A simple comparision of post-treatment earnings across NSW treated and control units provides an unbiased estimate of the ATT. Such an estimate is computed below. Treated individuals earned $1,794 more in 1978 than non-treated individuals.
Can you write Python code for it?


I am working on the file "Notebooks/Tilting_Estimates_of_ATT.ipynb".
The first task is:
Hide Runtime warning messages to clean up output.
Can you write Python code for it?

import warnings

def fxn():
    warnings.warn("runtime", RuntimeWarning)

with warnings.catch_warnings():
    warnings.simplefilter("ignore")
    fxn()


I am working on the file "Notebooks/Tilting_Estimates_of_ATT.ipynb".
The first task is:
Now load additional libraries that will be used below.
Can you write Python code for it?

# Direct Python to plot all figures inline (i.e., not in a separate window)
%matplotlib inline

# Load additional libraries
import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import statsmodels.api as sm


I am working on the file "Notebooks/Tilting_Estimates_of_ATT.ipynb".
The first task is:
Observe that the test of $H_{0} : \lambda_{a} = 0$ is decisively rejected. This should be unsurprising. The covariate distribution in the PSID is so different from that observed across the NSW treated units that it would be extraordinary if the p-score took a parsimonious form (recall from Graham, Pinto and Egel (2016, JBES), and work on discriminant analysis more generally, that the propensity score is related to the ratio of the covariate densities in the treated and control samples).

The ATT estimate is close to its experimental benchmark, with a somewhat larger standard error. Despite the PSID including 2,490 controls the standard error goes up relative to the experimental benchmark (which is based on a much smaller number of controls). This reflects poor overlap.
Experimental benchmark estimate of the ATT with covariate adjustment by AST
Next I compute the AST estimate of the ATT using the NSW experimental controls and the same specification for $t(W)$ given above. However, I set $r(W)$ to include a constant only (corresponding to the fact that we know a priori that the propensity score is constant due the RCT structure of the data). Note in this case that both the study and auxiliary tilts need to be computed.
Can you write Python code for it?

# Make some adjustments to variable definitions in experimental dataframe
nsw['constant'] = 1                # Add constant to observational dataframe
nsw['age']      = nsw['age']/10    # Rescale age to be in decades

nsw['re74']        = nsw['re74']/1000          # Recale earnings to be in thousands
nsw['re75']        = nsw['re75']/1000          # Recale earnings to be in thousands
nsw['re74_X_re75'] = nsw['re74']*nsw['re75']   # Prior earnings interaction
nsw['re74_sq']     = nsw['re74']**2            # Squares of prior earnings
nsw['re75_sq']     = nsw['re75']**2            # Squares of prior earnings

nsw['ue74']        = 1*(nsw['re74']==0)        # Dummy for unemployed in 1974
nsw['ue75']        = 1*(nsw['re75']==0)        # Dummy for unemployed in 1975
nsw['ue74_X_ue75'] = nsw['ue74']*nsw['ue75']   # Prior unemployment interaction

# Treatment indicator
D = nsw['treat']

# Balancing moments
t_W = nsw[['constant','black','hispanic','age','married','nodegree',\
           're74','re75','re74_X_re75','ue74','ue75','ue74_X_ue75']]

# Propensity score variables
h_W = nsw[['constant']]

# Outcome
Y = nsw['re78']

# Compute AST estimate of ATT
[gamma_ast, vcov_gamma_ast, pscore_tests, tilts, exitflag] = \
                                            ipt.att(D, Y, h_W, t_W, study_tilt = True, rlgrz = 1/2, \
                                                    s_wgt=None, nocons=True, c_id=None, silent=False)

# Compute log of auxiliary/control sample normalized weights to construct histogram below

pi_a_nsw = tilts[:,2]                  # Auxiliary tilt is the last columnn in np.array tilts
b = np.where(1-D)[0]                   # find indices of control units
Na = len(pi_a_nsw[b])
log_wgts_nsw = np.log(Na*pi_a_nsw[b])  # log of normalized weights to undo right skewness for plotting purposes


I am working on the file "Notebooks/Time_functions.ipynb".
The first task is:
Binarization
Can you write Python code for it?

lam_factor = 3
area_factor_large = 0.001
area_factor_verylarge = 0.1
lam = 50
connectivity = 4
weights=(0.33,0.33,0.33)
grayscale = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

%%timeit
#Creation of the binarizer
binarizer = sr.DatadrivenBinarizer(area_factor_large=area_factor_large, area_factor_verylarge=area_factor_verylarge,
                                           lam=lam, weights=weights, connectivity=connectivity)

binarizer = sr.DatadrivenBinarizer(area_factor_large=area_factor_large, area_factor_verylarge=area_factor_verylarge,
                                           lam=lam, weights=weights, connectivity=connectivity)

%%timeit
#The binarization
binarized = binarizer.binarize(grayscale, visualize=False)

cProfile.run('binarizer.binarize(grayscale, visualize=False)')


I am working on the file "Notebooks/Time_functions.ipynb".
The first task is:
MSER detection
Can you write Python code for it?

mser = cv2.MSER_create()

%%timeit
regions = mser.detectRegions(img, None)

cProfile.run('mser.detectRegions(img, None)')


I am working on the file "Notebooks/Torque Free 3-1-3 Body Dynamics.ipynb".
The first task is:
$\tilde{\omega} = {}^\mathcal{B}C^{\mathcal{I}}  {}^\mathcal{B}{\dot{C}}^{\mathcal{I}}$
Can you write Python code for it?

omega_tilde = bCi*bCi_dot.T; omega_tilde


I am working on the file "Notebooks/Torque Free 3-1-3 Body Dynamics.ipynb".
The first task is:
Generate MATLAB Code
Can you write Python code for it?

out = codegen(("eom1",sol_psidd[0]), 'Octave', argument_sequence=[th,thd,psi,psid,ph,phd,I1,I2,I3]);out

codegen(("eom1",sol_thdd[0]), 'Octave', argument_sequence=[th,thd,psi,psid,ph,phd,I1,I2,I3])

codegen(("eom1",sol_phdd[0]), 'Octave', argument_sequence=[th,thd,psi,psid,ph,phd,I1,I2,I3,psidd])

codegen(("eom1",[s0[psid],s0[thd],s0[phd]]), 'Octave', argument_sequence=[w1,w2,w3,th,thd,psi,psid,ph,phd,I1,I2,I3,psidd])

codegen(("eom1",bCi), 'Octave', argument_sequence=[th,thd,psi,psid,ph,phd,I1,I2,I3,psidd])

codegen(("eom1",omega), 'Octave', argument_sequence=[w1,w2,w3,th,thd,psi,psid,ph,phd,I1,I2,I3,psidd])


I am working on the file "Notebooks/Torque Free 3-1-3 Body Dynamics.ipynb".
The first task is:
Set up rotation matrices representing a 3-1-3 $(\psi,\theta,\phi)$ Euler angle set.
Can you write Python code for it?

aCi = rotMat(3,psi)
cCa = rotMat(1,th)
bCc = rotMat(3,ph)
aCi,cCa,bCc

bCi = bCc*cCa*aCi; bCi #3-1-3 rotation

bCi_dot = difftotalmat(bCi,t,{th:thd,psi:psid,ph:phd});
bCi_dot


I am working on the file "Notebooks/Tutorial.ipynb".
The first task is:
If you need access to the dispersion axis or flux you can do it by the flux and xaxis attributes.
Can you write Python code for it?

wavelength = spec_uncalibrated.xaxis
I = spec_uncalibrated.flux


I am working on the file "Notebooks/Tutorial.ipynb".
The first task is:
Now the telluric model wavelength spans a much longer wavelength than we have observed.
It is a good idea to shorten it to the values we have to save memory.
Can you write Python code for it?

# Shorten telluric spectra to just around our observation (1 nm either side)
print("Before ", len(telluric))
telluric.wav_select(min(observation.xaxis)-1, max(observation.xaxis)+1)
print("After  ", len(telluric))

# Plot the two Spectrum to see what we have
plt.figure()
plt.plot(telluric.xaxis, telluric.flux, label="Telluric")
plt.plot(observation.xaxis, observation.flux, "r--", label="Observation")
plt.xlabel("Wavelength (nm)")
plt.ylabel("Flux/Absorption")
#plt.xlim([np.min(observation.xaxis), np.max(observation.xaxis)])   # Veiw this detector only
plt.legend(loc=0)
plt.show()


I am working on the file "Notebooks/Tutorial.ipynb".
The first task is:
Now we want to correct for the atmospheric absorption be dividing the observation by the telluric model.
Can you write Python code for it?

# Warning! When I attempted a division before applying the telluric.wav_select() above I obtained a MemoryError in the interpolation
# This needs to be looked into (it takes a while to run atm)

Corrected = observation / scaled_telluric

# Plot Result
plt.plot(observation.xaxis, observation.flux, label="Observation")
plt.plot(Corrected.xaxis, Corrected.flux, "r--", label="Corrected")
plt.xlabel("Wavelength (nm)")
plt.ylabel("Flux/Absorption")
plt.xlim([np.min(observation.xaxis), np.max(observation.xaxis)])   # Veiw this detector only
plt.legend(loc=0)
plt.show()


I am working on the file "Notebooks/Tutorial.ipynb".
The first task is:
Practical Example - Telluric Correction
We want to divide an observation by a model of the telluric absorbtion to correct for Earths absoption
Can you write Python code for it?

# A function to read header and data from tapas ipac file.
def load_telluric(filename):
    """ Returns telluric data and header
    if just want the data then call as load_telluric()[0]
    or data, __ = load_telluric()

    likewise just the header as hdr = load_telluric()[1]"""
    ext = filename.split(".")[-1]
    if ext == "ipac":
        tell_hdr = fits.Header()
        with open(filename) as f:
            col1 = []
            col2 = []
            for line in f:
                #firstchar = line[0]
                #print("first char =", firstchar)
                if line.startswith("\\"):
                    # Get the Tapas Header
                    line = line[1:] # remove the leading \
                    line = line.strip()
                    items = line.split("=")

                    tell_hdr[items[0]] = items[1] # Add to header

                elif line.startswith("|"):
                    # Obtian wavelength scale from piped lines
                    if "in air" in line:
                        tell_hdr["WAVSCALE"] = "air"
                    elif "nm|" in line:
                        tell_hdr["WAVSCALE"] = "vacuum"
                    # Need extra condition to deal with wavenumber
                else:
                    line = line.strip()
                    val1, val2 = line.split()
                    col1.append(float(val1))
                    col2.append(float(val2))

    elif ext == "fits":
        i_tell = fits.getdata(filename, 1)
        tell_hdr = fits.getheader(filename, 1)
        # TODO ... Need to get wavelenght scale (air/wavelenght) from fits file somehow...
        col1 = i_tell["wavelength"]
        col2 = i_tell["transmittance"]

    else:
        print(" Could not load file", filename," with extention", ext)
        return None

        # put in ascending order
    if col1[-1]-col1[0] < 0:  # wl is backwards
            col1 = col1[::-1]
            col2 = col2[::-1]
    tell_data = np.array([col1, col2], dtype="float64")

    return...


I am working on the file "Notebooks/Tutorial.ipynb".
The first task is:
Spectrum has a wave_select metod which allows you to select a section of you spectra between an lower and upper dispersion coordinate(pixel/wavelenght).
Can you write Python code for it?

#spec_calibrated.wav_select(lower, upper)
spec_calibrated.wav_select(2030, 2070)


I am working on the file "Notebooks/Tutorial.ipynb".
The first task is:
You create a spectrum object by passing in the flux and the dispersion values. As positional arguments flux is first. If your dispersion axis has been wavelenght calibrated (its a wavelength rather than pixel positions) then you need to also pass in calibrated=True keyword.
Can you write Python code for it?

x = np.arange(2000, 2050)
y = np.random.rand(len(x))
spec_uncalibrated = Spectrum(y, x)
spec_calibrated = Spectrum(flux=y, xaxis=x, calibrated=True)


I am working on the file "Notebooks/Unsaturated_Zone_Validation.ipynb".
The first task is:
2. EMPTYING AND FILLING THE UNSATURATED ZONE
In this example we investigate the filling and emptying of the unsaturated zone. The unsaturated zone has certain physical boundaries, as it cannot have negative values for $S_r$ not values larger than $S_{rmax}$.
$P = 0.002$ m/d for:  $0 < t < 183$
$P = 0.00$ m/d for:  $183 < t $
We model the situation where the unsaturated zone is full and where it is empty, to validate that the the amount of water cannot excess the soil capacity $S_{cap}$ and cannot go below zero. We therefore first consider half a year with very wet conditions, followed by half a year with no rain.
Can you write Python code for it?

n = 365
t = np.arange(0,n,1)
P = np.zeros(n)
P[:int(n/2.0)] = 2.0 / 1000 # All forcings must be provided in meters
E = 0.001 * np.ones(n)

# Provide the model parameters for the soil module
S_rmax = 0.01 # is equal to ±100 milimeters
K_p = 0.001 # is equal to 0.01 m/d
Gamma = 1.0
Imax = 0.0015

R, S = Unsat_Zone.perc(t, P, E, S_rmax, K_p, Gamma, Imax, dt, solver=0)
R1, S1 = Unsat_Zone.perc(t, P, E, S_rmax, K_p, Gamma, Imax, dt, solver=1)

plt.figure(figsize=(15,4))
plt.subplot(121)
plt.plot(t,S)
plt.plot(t,S1)
plt.ylabel('S [meter]')
plt.xlabel('Time [days]')
plt.axhline(S_rmax, color='r')
plt.legend(['explicit', 'implicit', r'$S_rmax$'], loc='best')
plt.title('Soil State over Time')
plt.ylim(0, S_rmax+0.001)

plt.subplot(122)
plt.plot(R)
plt.plot(R1)
plt.title('recharge over time')
plt.ylabel('Recharge [m/d]')
plt.xlabel('Time [days]')
plt.legend(['explicit', 'implicit', r'$S_rmax$'], loc='best')
plt.ylim(0, max(R)+0.0001)


I am working on the file "Notebooks/Unsupervised/K-Means Clustering Example.ipynb".
The first task is:
K-Means Clustering Example
In this example notebook, you will see how to implement K-Means Clustering in Python using Scikit-Learn and Pandas.
Adapted from https://pythonprogramming.net/flat-clustering-machine-learning-python-scikit-learn/
Step 1:  Get Data:
The first step is to prepare or generate the data.  In this dataset, the observations only have two features, but K-Means can be used with any number of features.  Since this is an unsupervised example, it is not necessary to have a "target" column.
Can you write Python code for it?


I am working on the file "Notebooks/Unsupervised/K-Means Clustering Example.ipynb".
The first task is:
Step 2:  Build the Model:
Much like the supervised models, you first create the model then call the .fit() method using your data source.  The model is now populated with both your centroids and labels.  These can be accessed via the .cluster_centers_ and labels_ properties respectively.
You can view the complete documentation here: http://scikit-learn.org/stable/modules/generated/sklearn.cluster.KMeans.html
K-Means also has a .predict() method which can be used to predict the label for an observation.
Can you write Python code for it?

kmeans = KMeans(n_clusters=2).fit(data)

centroids = kmeans.cluster_centers_
labels = kmeans.labels_

print(centroids)
print(labels)


I am working on the file "Notebooks/Unsupervised/K-Means Clustering Example.ipynb".
The first task is:
Visualizing the Clusters
The code below visualizes the clusters.
Can you write Python code for it?

data['labels'] = labels

#plt.plot(data, colors[data['labels'], markersize = 10)

group1 = data[data['labels']==1].plot( kind='scatter', x='x', y='y', color='DarkGreen', label="Group 1" )
group2 = data[data['labels']==0].plot( kind='scatter', x='x', y='y', color='Brown', ax=group1, label="Group 2" )
group1.legend(loc='upper center', bbox_to_anchor=(0.5, 1.05),
          ncol=3, fancybox=True, shadow=True)
plt.scatter(centroids[:, 0],centroids[:, 1], marker = "x", s=150, linewidths = 5, zorder = 10)

plt.show()


I am working on the file "Notebooks/Unused/CicrularCrop.ipynb".
The first task is:
And radius and the center of the image. If we have at least upper or lower side of the disk non-cropped, use it to determine the vertical center. Otherwise use Pithagoras theorem :-)
Can you write Python code for it?

radius = (x_max - x_min)/2
center_x = x_min + radius
center_y = y_min + radius # the default case (if y_min != 0)
if y_min == 0: # the upper side is cropped
    if height - y_max > 0: # lower border is not 0
        center_y = y_max - radius
    else:
        upper_line_width = np.sum(s[0,:] > 100) # threshold for single line
        center_y = math.sqrt( radius**2 - (upper_line_width/2)**2)
radius1 = radius - 200

mask = np.zeros(img.shape[0:2])
rr, cc = circle(center_y, center_x, radius1, img.shape)
mask[rr, cc] = 1
img[:,:,0] *= mask
img[:,:,1] *= mask
img[:,:,2] *= mask

x_borders = (center_x - radius1, img.shape[1] - center_x - radius1)
y_borders = (max(center_y - radius1,0), max(img.shape[0] - center_y - radius1, 0))

img2 = util.crop(img, (y_borders, x_borders,  (0,0)))
maskT = util.crop(mask, (y_borders, x_borders))

border_pixels = np.sum(1 - maskT)

plt.imshow(img2)


I am working on the file "Notebooks/Unused/CicrularCrop.ipynb".
The first task is:
Cropped Image Case
now let's experiment with cropped image
Can you write Python code for it?

baseFolder = '/Users/boris/Dropbox/Kaggle/Retina/train/sample'
imgFile = '263_right.jpeg'
filename = os.path.join(baseFolder, imgFile)
img = skimage.io.imread(filename)
plt.imshow(img)

s = np.sum(img, axis=2)
cols = np.sum(s, axis=0) > threshold
rows = np.sum(s, axis=1) > threshold


I am working on the file "Notebooks/Unused/CicrularCrop.ipynb".
The first task is:
Non-Cropped image case
Load some image that is not cropped.
Can you write Python code for it?

baseFolder = '/Users/boris/Dropbox/Kaggle/Retina/train/sample'
imgFile = '78_left.jpeg'
filename = os.path.join(baseFolder, imgFile)
img = io.imread(filename)
plt.imshow(img)


I am working on the file "Notebooks/Unused/CicrularCrop.ipynb".
The first task is:
The simplest way to detect edges for cropping of a circular image with dark background is to sum up along different axes. Let's see how it works. First we sum up all the color channels, then compute horizontal and vertical borders.
Can you write Python code for it?

threshold = 20000

s = np.sum(img, axis=2)
cols = np.sum(s, axis=0) > threshold
rows = np.sum(s, axis=1) > threshold


I am working on the file "Notebooks/Unused/CicrularCrop.ipynb".
The first task is:
This is simple case, not trimmed image. Let's determine the radius and center of it. We reduce the radius in orger to get rid of the edge distortions
Can you write Python code for it?

radius = (x_max - x_min)/2
center_x = x_min + radius
center_y = y_min + radius
radius1 = radius - 100


I am working on the file "Notebooks/Unused/CicrularCrop.ipynb".
The first task is:
now compute borders of the image
Can you write Python code for it?

height = rows.shape[0]
width = cols.shape[0]

x_min = np.argmax(cols[0:width])
x_max = width/2 + np.argmin(cols[width/2:width-1])
y_min = np.argmax(rows[0:height/2])
y_max = height/2 + np.argmin(cols[height/2:height-1])


I am working on the file "Notebooks/Unused/Resizing.ipynb".
The first task is:
LANCZOS anti-aliasing method is recommended for downsampling by PIL tutorial, but is much slower:
Can you write Python code for it?

%timeit im.resize((hsize, vsize), Image.LANCZOS)


I am working on the file "Notebooks/Unused/Resizing.ipynb".
The first task is:
Let us define functions that do the above in one go for all files:
Can you write Python code for it?

%%px --local
def resize_method(filename, method):
    filepath = os.path.join(src_dir, filename)
    im = Image.open(filepath)
    resized_im = im.resize((hsize, vsize), method)
    resized_filepath = os.path.join(trg_dir, prefix + filename)
    resized_im.save(resized_filepath, "JPEG", quality = 100)

def resize_NEAREST(filename):
    resize_method(filename, Image.NEAREST)

def resize_LANCZOS(filename):
    resize_method(filename, Image.LANCZOS)


I am working on the file "Notebooks/Unused/Resizing.ipynb".
The first task is:
Let us set up imports and parameters in one place for the cluster and the local engine:
Can you write Python code for it?

%%px --local
import os
import os.path
from PIL import Image

src_dir = "/kaggle/retina/sample" # source directory of images to resize
trg_dir = "/kaggle/retina/resized" # target directory of the resized images
prefix = "resized_" # string to prepend to the resized file name
hsize = 256 # horizontal size of the resized image
vsize = 256 # vertical size of the resized image
all_files = filter(lambda x: x.endswith(".jpeg"), os.listdir(src_dir))


I am working on the file "Notebooks/Unused/Resizing.ipynb".
The first task is:
Load an image:
Can you write Python code for it?

filename = all_files[0]
filepath = os.path.join(src_dir, filename)
%timeit -n1 -r1 Image.open(filepath)
im = Image.open(filepath)


I am working on the file "Notebooks/Unused/Resizing.ipynb".
The first task is:
Save the resized image.
Parameter value quality > 95 is not recommended due to excessive file size with minimal benefits, but we do not care.
More info on file formats can be found here: http://pillow.readthedocs.org/handbook/image-file-formats.html
Can you write Python code for it?

if not os.path.exists(trg_dir):
    os.makedirs(trg_dir)

resized_filepath = os.path.join(trg_dir, prefix + filename)
%timeit -n1 -r1 resized_im.save(resized_filepath, "JPEG", quality = 100)


I am working on the file "Notebooks/Unused/Resizing.ipynb".
The first task is:
Since the processing is dominated by the CPU-bound resizing we can benefit from parallelization:
Can you write Python code for it?

v = rc[:]
%timeit -n1 -r1 v.map_sync(resize_NEAREST, all_files)
%timeit -n1 -r1 v.map_sync(resize_LANCZOS, all_files)


I am working on the file "Notebooks/Using nbconvert as a Library.ipynb".
The first task is:
Example
The following demonstration was requested in an IPython GitHub issue, the ability to exclude a cell by index.
Inject cells is similar, and won't be covered here.  If you want to inject static content at the beginning/end of a notebook, use a custom template.
Can you write Python code for it?

from traitlets import Integer

class PelicanSubCell(Preprocessor):
    """A Pelican specific preprocessor to remove some of the cells of a notebook"""

    # I could also read the cells from nbc.metadata.pelican is someone wrote a JS extension
    # But I'll stay with configurable value.
    start = Integer(0, config=True, help="first cell of notebook to be converted")
    end   = Integer(-1, config=True, help="last cell of notebook to be converted")

    def preprocess(self, nb, resources):

        #nbc = deepcopy(nb)
        nbc = nb
        # don't print in real preprocessor !!!
        print("I'll keep only cells from ", self.start, "to ", self.end, "\n\n")
        nbc.cells = nb.cells[self.start:self.end]
        return nbc, resources

# I create this on the fly, but this could be loaded from a DB, and config object support merging...
c =  Config()
c.PelicanSubCell.enabled = True
c.PelicanSubCell.start = 4
c.PelicanSubCell.end = 6


I am working on the file "Notebooks/Using nbconvert as a Library.ipynb".
The first task is:
If you do not have requests, install it by running pip install requests (or if you don't have pip installed, you can find it on PYPI).
The response is a JSON string which represents an IPython notebook. Next, read the response using nbformat.
Doing this will guarantee that the notebook structure is valid. Note that the in-memory format and on disk format are slightly different. In particual, on disk, multiline strings might be splitted into a list of strings.
Can you write Python code for it?

from IPython import nbformat
jake_notebook = nbformat.reads(response.text, as_version=4)
jake_notebook.cells[0]


I am working on the file "Notebooks/Using nbconvert as a Library.ipynb".
The first task is:
If you understand HTML, you'll notice that some common tags are ommited, like the body tag.  Those tags are included in the default HtmlExporter, which is what would have been constructed if no Config object was passed into it.
Extracting Figures
When exporting you may want to extract the base64 encoded figures as files, this is by default what the RstExporter does (as seen below).
Can you write Python code for it?

from IPython.nbconvert import RSTExporter

rst_exporter = RSTExporter()

(body,resources) = rst_exporter.from_notebook_node(jake_notebook)

print(body[:970]+'...')
print('[.....]')
print(body[800:1200]+'...')


I am working on the file "Notebooks/Using nbconvert as a Library.ipynb".
The first task is:
There are 5 extracted binary figures, all pngs, but they could have been svgs which then wouldn't appear in the binary sub dict.  Keep in mind that objects with multiple reprs will have every repr stored in the notebook avaliable for conversion.
Hence if the object provides _repr_javascript_, _repr_latex_, and _repr_png_, you will be able to determine, at conversion time, which representaition is most appropriate. You could even show all of the representaitions of an object in a single export, it's up to you. Doing so would require a little more involvement on your part and a custom Jinja template.
Back to the task of extracting an image, the Image display object can be used to display one of the images (as seen below).
Can you write Python code for it?

from IPython.display import Image
Image(data=resources['outputs']['output_3_0.png'],format='png')


I am working on the file "Notebooks/Using nbconvert as a Library.ipynb".
The first task is:
This image is being rendered without reading or writing to the disk.
Extracting figures with HTML Exporter ?
Use case:

I write an awesome blog using IPython notebooks converted to HTML, and I want the images to be cached.  Having one html file with all of the images base64 encoded inside it is nice when sharing with a coworker, but for a website, not so much.  I need an HTML exporter, and I want it to extract the figures!

Some theory
The process of converting a notebook to a another format with happens in a few steps:

Retrieve the notebook and it's accompanying resource (you are responsible for this).
Feed them into the exporter, which:
Sequentially feeds them into an array of Preprocessors.  Preprocessors only act on the structure of the notebook, and have unrestricted access to it.
Feeds the notebook into the Jinja templating engine.
The template is configured (you can change which one is used).
Templates make use of configurable macros called filters.

The exporter returns the converted notebook and other relevant resources as a tuple.
You write the data to the disk, or elsewhere (you are responsible for this too).

You can use Preprocessors to accomplish the task at hand.  IPython has preprocessors built in which you can use.  One of them, the ExtractOutputPreprocessor is responsible for crawling the notebook, finding all of the figures, and putting them into the resources directory, as well as choosing the key (i.e. filename_xx_y.extension) that can replace the figure inside the template.
The ExtractOutputPreprocessor is special because it's available in all of the Exporters, and is just disabled in some by default.
Can you write Python code for it?

# 3rd one should be <ExtractOutputPreprocessor>
html_exporter._preprocessors


I am working on the file "Notebooks/VAE.ipynb".
The first task is:
Generate new data
Generate samples from the approximate posterior distribution over the latent variables p(z|x)
Can you write Python code for it?

Zdemo = np.random.normal(size=[128, latentD], loc=0.0, scale=1.).astype(np.float32)

gen_sample = demo_sess.run(Xgen_op, feed_dict={Zph: Zdemo , is_training:False})
vis_square(gen_sample[:121], [11, 11], save_path=work_dir + 'sample.jpg')
Image(filename=work_dir + 'sample.jpg')


I am working on the file "Notebooks/VAE.ipynb".
The first task is:
Training VAE
You can either get the fully trained models from the google drive or train your own models using the VAE.py script.
Evolution of the latent variable during the training phase
Evolution of approximate posterior Q(z|X) and likelihood P(X|z) during training with 2D latent space.
Can you write Python code for it?


I am working on the file "Notebooks/VAE.ipynb".
The first task is:
Visualization of the 2D latent variable corresponding to a Convolutional Variational Autoencoder during training on MNIST dataset (handwritten digits). The image to the left is the mean of the approximate posterior Q(z|X) and each color represents a class of digits within the dataset. The image to the left shows samples from the decoder (likelihood) P(X|z). The title above shows the iteration number and total loss [Reconstruction + KL] of the model at the point that images below were produced from the model under training. One can observe that by the time the generated outputs (left image) get better, the points on the latent space (posterior) also get into better seperated clusters. Also note that points get closer to each other because the KL part of the total loss is imposing a zero mean gaussian distribution on the latent variable, which is realized on the latent variable as the trainig proceeds.
Experiments
Create demo networks and restore weights
Can you write Python code for it?

iter_num = 37752
best_model = work_dir + "Model_Iter_%.3d.ckpt"%iter_num
best_img = work_dir + 'Rec_Iter_%d.jpg'%iter_num
Image(filename=best_img)

latentD = 2
batch_size = 128

tf.reset_default_graph()
demo_sess = tf.InteractiveSession()

is_training = tf.placeholder(tf.bool, [], 'is_training')

Zph = tf.placeholder(tf.float32, [None, latentD])
Xph = tf.placeholder(tf.float32, [None, 28, 28, 1])

z_mu_op, z_log_sigma_sq_op = create_encoder(Xph, is_training, latentD, reuse=False, networktype=networktype + '_Enc')
Z_op = tf.add(z_mu_op, tf.multiply(tf.sqrt(tf.exp(z_log_sigma_sq_op)), Zph))
Xrec_op = create_decoder(Z_op, is_training, latentD, reuse=False, networktype=networktype + '_Dec')
Xgen_op = create_decoder(Zph, is_training, latentD, reuse=True, networktype=networktype + '_Dec')

tf.global_variables_initializer().run()

Enc_varlist = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope=networktype + '_Enc')
Dec_varlist = tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, scope=networktype + '_Dec')
print('List of vars in Encoder:%d -- %s\n' % (len(Enc_varlist), '; '.join([var.name.split('/')[1] for var in Enc_varlist])))
print('List of vars in Decoder:%d -- %s\n' % (len(Dec_varlist), '; '.join([var.name.split('/')[1] for var in Dec_varlist])))

saver = tf.train.Saver(var_list=Enc_varlist+Dec_varlist)
saver.restore(demo_sess, best_model)

#Get uniform samples over the labels
spl = 800  # sample_per_label
data = input_data.read_data_sets(data_dir, one_hot=False, reshape=False)
Xdemo, Xdemo_labels = get_demo_data(data, spl)

decoded_data = demo_sess.run(z_mu_op, feed_dict={Xph:Xdemo, is_training:False})
plot_latent_variable(decoded_data, Xdemo_labels)


I am working on the file "Notebooks/Viromics/Analysis_of_first_libraries_Clone_reference.ipynb".
The first task is:
Control, replicate E
Can you write Python code for it?

tables_E = [DD6_E, DD9_E]
increasing_E = get_increasing_variants(tables_E, clone)
print("There are "+str(len(increasing_E))+" positions that rise in frequency.")
print("There are too many of them, we choose not to print them.")


I am working on the file "Notebooks/Viromics/Analysis_of_first_libraries_Clone_reference.ipynb".
The first task is:
Functions to plot interesting positions and gene boundaries
Can you write Python code for it?

# Interesting positions
positions=[316,1670,1785,2340,5935,7172,8449,9165]
def plot_positions():
    for x in positions:
        plt.axvline(x=x, linewidth=1, linestyle=':')

def plot_genes():
    for i in range(len(begins)):
        plt.plot([begins[i], begins[i]], [0.99,1.0], linewidth=2, linestyle='-', color="black")
        if i%2==0:
            plt.text (begins[i] + ((ends[i] - begins[i])/10), 1.005, (names[i].replace('"', ''))[0:3], size='xx-small')
        else:
            plt.text (begins[i] + ((ends[i] - begins[i])/10), 1.015, (names[i].replace('"', ''))[0:3], size='xx-small')
    plt.plot([ends[-1], ends[-1]], [0.99,1.0], linewidth=2, linestyle='-', color="black")


I am working on the file "Notebooks/Viromics/Analysis_of_first_libraries_Clone_reference.ipynb".
The first task is:
Obtaining the clone sequence
Can you write Python code for it?

file = "cloneSequence/SP6-ZIKV_seq_only.txt"
clone = ""
with open (file) as f:
    for l in f:
        if ">" in l:
            pass
        else:
            clone +=l.strip()


I am working on the file "Notebooks/Viromics/Analysis_of_first_libraries_Clone_reference.ipynb".
The first task is:
Obtaining the sequence annotation
Can you write Python code for it?

begins=[]
ends=[]
names =[]
with open ("sequence.gb") as f:
    in_pep = False
    for l in f:
        if "mat_peptide" in l:
            begins.append(int(l.split()[1].split("..")[0]))
            ends.append(int(l.split()[1].split("..")[1]))
            in_pep = True
        elif in_pep :
            names.append(l.split("=")[1])
            in_pep = False

print(begins)
print(ends)
print(names)


I am working on the file "Notebooks/Viromics/Analysis_of_first_libraries_Clone_reference.ipynb".
The first task is:
Positions that increase in frequency
Control, replicate A
Can you write Python code for it?

tables_A = [DD3_A, DD6_A, DD9_A, DD12_A, DD24_A, DD51_A]
increasing_A = get_increasing_variants(tables_A, clone)
print("There are "+str(len(increasing_A))+" positions that rise in frequency.")
print("Those are:")
print_variants(increasing_A)


I am working on the file "Notebooks/Viromics/Analysis_of_first_libraries_Clone_reference.ipynb".
The first task is:
Reading all data
Can you write Python code for it?

# CirSeq initial sample
cirseq = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1CirseqD3_1_sequence.txt.assembled.fastq_mapped_AA.csv", na_values=" -nan")

# Control runs, replicate A
DD3_A = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD3A_1_sequence.txt.assembled.fastq_mapped_AA.csv", na_values=" -nan")
DD6_A = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD6A_1_sequence.txt.assembled.fastq_mapped_AA.csv", na_values=" -nan")
DD9_A = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD9A_1_sequence.txt.assembled.fastq_mapped_AA.csv", na_values=" -nan")
DD12_A = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD12A_1_sequence.txt.assembled.fastq_mapped_AA.csv", na_values=" -nan")
DD24_A = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD24A_1_sequence.txt.assembled.fastq_mapped_AA.csv", na_values=" -nan")
DD51_A = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD51A_1_sequence.txt.assembled.fastq_mapped_AA.csv", na_values=" -nan")
DD51_A_no_reamp = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD51Anoreamplification_1_sequence.txt.assembled.fastq_mapped_AA.csv", na_values=" -nan")

# Control runs, replicate D
DD3_D = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD3D_1_sequence.txt.assembled.fastq_mapped_AA.csv")
DD6_D = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD6D_1_sequence.txt.assembled.fastq_mapped_AA.csv")
DD9_D = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD9D_1_sequence.txt.assembled.fastq_mapped_AA.csv")
DD12_D = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD12D_1_sequence.txt.assembled.fastq_mapped_AA.csv")
DD24_D = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD24D_1_sequence.txt.assembled.fastq_mapped_AA.csv")

# Control runs, replicate E
DD6_E = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD6E_1_sequence.txt.assembled.fastq_mapped_AA.csv")
DD9_E = pd.read_csv ("HV5GLBCXY_ZIKV_17s006139-1-1_DREUX_lane1DD9E_1_sequence.txt.assembled.fastq_mapped_AA.csv")

# TLR3 activation runs,...


I am working on the file "Notebooks/Viromics/Analysis_of_first_libraries_Clone_reference.ipynb".
The first task is:
TLR3 treatment
Can you write Python code for it?

tables_TA = [TD9_A, TD12_A, TD24_A, TD51_A]
increasing_TA = get_increasing_variants(tables_TA, clone)
print("There are "+str(len(increasing_TA))+" positions that rise in frequency.")
print("Those are:")
print_variants(increasing_TA)


I am working on the file "Notebooks/Viromics/analysis_CirSeqAndCloneSamples.ipynb".
The first task is:
Analysis of coverage
Can you write Python code for it?

def plotCoverage(tables, names):
    variable = 'Coverage'
    sample = list()
    posList = list()
    variableList = list()
    for i in range(len(names)):
        sample = sample + len(tables[i][variable]) * [names[i]]
        posList.append(tables[i]['Position'])
        variableList.append(tables[i][variable])
    positions = pd.concat(posList)
    variableValues = pd.concat(variableList)
    overlay_table_concat = pd.DataFrame ({'Position':positions, variable:variableValues, 'sample':sample})
    sns.lmplot( x="Position", y=variable, data=overlay_table_concat, fit_reg=False, hue='sample', legend=False, size=7, aspect=2, lowess=True,scatter_kws={"s": 20})
    plt.legend(loc='lower right')
    plot_positions()
    plot_genes()

plotCoverage(tables_clone, all_table_names)


I am working on the file "Notebooks/Viromics/analysis_CirSeqAndCloneSamples.ipynb".
The first task is:
Analysis of positions 1670 and 2193
Can you write Python code for it?

def test_positions (tables, positions):
    num_tables = len(tables)
    first = tables[0]
    last = tables[num_tables-1]
    major = ""
    minor = ""
    major_frequencies = array('d',[0.0]*num_tables)
    for i in positions:
        minor_frequencies = array('d',[0.0]*num_tables)
        major = first["Major_variant"][i]
        minor = tables[-1]["Second_variant"][i]
        if minor == major:
            minor = tables[-1]["Major_variant"][i]
        counter = 0
        print(str(i)+" : ")
        for j in range(num_tables):
            minor_frequencies[j] = get_variant_frequency(minor, tables[j], i)
            print("\texp "+str(j) +", variant "+minor + " : "+str(minor_frequencies[j]))
            if minor_frequencies[j] > 0.1:
                counter += 1
        print("Counter of times > 0.1: "+ str(counter))
    return

def test_positions_1670_2193 (tables):
    test_positions(tables, [1670,2193])
    return

def test1670_2193 (tables):
    num_tables = len(tables)
    first = tables[0]
    last = tables[num_tables-1]
    major = ""
    minor = ""
    major_frequencies = array('d',[0.0]*num_tables)
    for i in [1670,2193]:
        minor_frequencies = array('d',[0.0]*num_tables)
        major = first["Major_variant"][i]
        minor = tables[-1]["Second_variant"][i]
        if minor == major:
            minor = tables[-1]["Major_variant"][i]
        counter = 0
        print(str(i)+" : ")
        for j in range(num_tables):
            minor_frequencies[j] = get_variant_frequency(minor, tables[j], i)
            print("\texp "+str(j) +", variant "+minor + " : "+str(minor_frequencies[j]))
            if minor_frequencies[j] > 0.1:
                counter += 1
        print("Counter of times > 0.1: "+ str(counter))
    return

test_positions_1670_2193 (tables_clone)


I am working on the file "Notebooks/Viromics/analysis_CirSeqAndCloneSamples.ipynb".
The first task is:
Diversity plot
Can you write Python code for it?

f, ax = plt.subplots(figsize=(15, 7))
lm=sns.kdeplot(np.log(tables_clone[0]["1_major_variant_frequency"]), label="cirseq")
lm.set_ylabel('Density')
lm.set_xlabel('Logarithm of the diversity per position')

lm=sns.kdeplot(np.log(tables_clone[1]["1_major_variant_frequency"]), label="clone_12")
lm=sns.kdeplot(np.log(tables_clone[2]["1_major_variant_frequency"]), label="clone_15")
lm=sns.kdeplot(np.log(tables_clone[3]["1_major_variant_frequency"]), label="clone_19")

axes = lm.axes
axes.set_xlim(-8,-4)
axes.set_ylim(0,1.2)


I am working on the file "Notebooks/Viromics/analysis_CirSeqAndCloneSamples.ipynb".
The first task is:
Other selection method
We want to select all the positions in which the minor variant reaches >0.1 in at least two time points, in two different replicates or experiments.
To do so, we get the variant at time point 0, then we look at the following time points and see if one site has some minor variant >0.1 in 2 time points. We collect all the sites with this characteristic for this experiment.
We do this for all experiments, then we take all sites that appear at least twice.
Can you write Python code for it?

def getInterestingVariants (tables):
    num_tables = len(tables)
    first = tables[0]
    last = tables[num_tables-1]
    major = ""
    minor = ""
    major_frequencies = array('d',[0.0]*num_tables)
    interestingVariants = list()
    for i in first["Position"]:
        minor_frequencies = array('d',[0.0]*num_tables)
        major = first["Major_variant"][i]
        minor = tables[-1]["Second_variant"][i]
        if minor == major:
            minor = tables[-1]["Major_variant"][i]
        counter = 0
        for j in range(num_tables):
            minor_frequencies[j] = get_variant_frequency(minor, tables[j], i)
            if minor_frequencies[j] > 0.1:
                counter += 1
        if counter >=2:
            interestingVariants.append(i)
    return interestingVariants

interesting_clone = getInterestingVariants (tables_clone)

interesting_clone

for_output = [322,
 418,
 1043,
 1312,
 1474,
 1903,
 2340,
 2455,
 2783,
 2852,
 2903,
 3565,
 3592,
 4192,
 4201,
 4552,
 5125,
 5155,
 5230,
 5314,
 5675,
 6334,
 6628,
 6670,
 6991,
 7093,
 7228,
 7441,
 7636,
 8093,
 8410,
 8776,
 9025,
 9241,
 9295,
 9556,
 9736,
 9952,
 10801,
 10802,
 10803,
 10804,
 10805]
write_increasing_positions(for_output, tables_clone, all_table_names, "positions_morethan01_clone_samples.csv")


I am working on the file "Notebooks/Viromics/analysis_CirSeqAndCloneSamples.ipynb".
The first task is:
Plotting only positions that consistently rise in frequency
Can you write Python code for it?

def plot_increasing_positions(increasing_pos, last_time_point) :
    sns.set_palette("hls")
    sns.set_context("poster")
    increasing_pos_keys = increasing_pos.keys()
    Is_increasing = []
    for i in last_time_point['Position']:
        if i in increasing_pos_keys:
            Is_increasing.append("Increasing")
        else:
            Is_increasing.append("Not increasing")

    to_plot = pd.DataFrame ({'Position':last_time_point['Position'], 'Major_variant_frequency_quality_corrected':last_time_point ['Major_variant_frequency_quality_corrected'],'Is_increasing':Is_increasing, 'is_synonymous':last_time_point ['is_synonymous']})
    ax=sns.lmplot( x="Position", y="Major_variant_frequency_quality_corrected", data=to_plot, fit_reg=False, hue='Is_increasing', row='is_synonymous', legend=False, size=7, aspect=2)
    ax.set(xlabel='Position along the genome', ylabel='Major variant frequency, last time point')
    plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)
    plot_positions()

plot_increasing_positions(increasing, clone_19)