I am working on the file "Plotly-Mesh3d.ipynb".
The first task is:
We notice that the Delaunay triangulation has as boundary a convex set (it is a triangulation of the convex hull
                                                                        of the given point set).
Each $\alpha$-complex is obtained from the Delaunay triangulation, removing  the triangles whose circumcircle has radius greater or equal to alpha.
In the last subplot the triangles of the  $0.115$-complex are filled in with light blue. The filled in region  is the underlying space of the $0.115$-complex.
The $0.115$-alpha shape of the given point set can be considered either the filled in region or its boundary.
This example illustrates that the underlying space of an  $\alpha$-complex in neither convex nor necessarily connected. It can consist in many connected components (in our illustration above, $|\mathcal{C}_{0.115}|$ has three components).
In a family of alpha shapes, the parameter $\alpha$ controls the level of detail of the associated alpha shape.  If $\alpha$ decreases to zero, the corresponding alpha shape degenerates to the point set, $S$, while  if it tends to infinity the alpha shape tends to the convex hull of the set $S$.
Plotly Mesh3d
In order to generate the alpha shape of a given set of 3D points corresponding to a parameter $\alpha$,
the Delaunay triagulation or the convex hull we define an instance of the  go.Mesh3d class. The real value
of the key alphahull points out the mesh type to be generated:
alphahull=$1/\alpha$    generates the $\alpha$-shape, -1 corresponds to the Delaunay
triangulation and 0, to the convex hull of the point set.
The other parameters in the definition of a Mesh3d are given here.
Mesh3d generates and displays an $\alpha$-shape as the boundary of the $\alpha$-complex.
An intuitive idea on the topological structure modification, as $\alpha=1/$alphahull varies can be gained from the following three different alpha shapes of the same point set:
Can you write Python code for it?

IFrame('https://plot.ly/~empet/13481/',  width=900, height=950)


I am working on the file "Plotly-Mesh3d.ipynb.ipynb".
The first task is:
Alpha shape of a finite set of points
The notion of Alpha Shape was introduced by Edelsbrunner with the aim to give  a mathematical description of the  shape of  a point set.
In this notebook we give a constructive definition of this geometric structure. A more detailed approach of 3D alpha shapes can be found in the original paper.
An intuitive description of the alpha shape was given by Edelsbrunner and his coauthor
in  a preprint of the last paper mentioned above:
A huge mass of ice-cream fills a region in the 3D space,
and the point set $S$ consists in hard chocolate pieces spread in the ice-cream mass.
Using a sphere-formed ice-cream spoon we carve out the ice-cream such that to avoid  bumping into
chocolate pieces.  At the end of this operation the region  containing the ciocolate pieces
and the remaining ice cream is bounded by caps, arcs and points of chocolate. Straightening
all round faces to triangles and line segments we get the intuitive image of the
alpha shape of the point set $S$.
Now we give the steps of the computational alpha shape construction.
Let  $S$ be a finite set of points from $\mathbb{R}^d$, in general position,  $\mathcal{D}$ its Delaunay triangulation
and $\alpha$ a positive number.
Select the d-simplexes of $\mathcal{D}$ (i.e.  triangles in the case d=2, respectively  tetrahedra
for d=3) whose circumsphere has the radius less than $\alpha$. These simplexes and their faces form
a simplicial subcomplex of the Delaunay triangulation, $\mathcal{D}$.
It is denoted $\mathcal{C}_\alpha$, and called  $\alpha$-complex.
The $\alpha$-shape of the set $S$ is defined by its authors, either as the underlying space of the $\alpha$-complex,
i.e. the union of all its simplexes or as the boundary of the $\alpha$-complex.
The boundary of the $\alpha$-complex is the subcomplex consisting in all k-simplexes,  $0\leq k<d$, that are faces of a single $d$-simplex (these are called external faces).
In the ice-cream example the alpha shape was defined as the boundary of the alpha-complex.
The...
Can you write Python code for it?

HTML('<iframe src=https://plot.ly/~empet/13479/ width=825 height=950></iframe>')


I am working on the file "Plotly-Mesh3d.ipynb.ipynb".
The first task is:
Filter the Delaunay triangulation to get the $\alpha$-complex:
Can you write Python code for it?

def get_alpha_complex(alpha, points, simplexes):
    #alpha is the parameter for the alpha shape
    #points are given data points
    #simplexes is the  list of indices in the array of points
    #that define 2-simplexes in the Delaunay triangulation

    return filter(lambda simplex: circumcircle(points,simplex)[1]<alpha, simplexes)

pts=np.loadtxt('data-ex-2d.txt')
tri = Delaunay(pts)

colors=['#C0223B', '#404ca0', 'rgba(173,216,230, 0.5)']# colors for vertices, edges and 2-simplexes


I am working on the file "Plotly-Mesh3d.ipynb.ipynb".
The first task is:
Get data for Plotly plot of a subcomplex of the Delaunay triangulation:
Can you write Python code for it?

def Plotly_data(points, complex_s):
    #points are the given data points,
    #complex_s is the list of indices in the array of points defining 2-simplexes(triangles)
    #in the simplicial complex to be plotted
    X=[]
    Y=[]
    for s in complex_s:
        X+=[points[s[k]][0] for k in [0,1,2,0]]+[None]
        Y+=[points[s[k]][1] for k in [0,1,2,0]]+[None]
    return X,Y

def make_trace(x, y,  point_color=colors[0], line_color=colors[1]):# define the trace
                                                                   #for an alpha complex
    return Scatter(mode='markers+lines', #set vertices and
                                         #edges of the alpha-complex
                   name='',
                   x=x,
                   y=y,
                   marker=Marker(size=6.5, color=point_color),
                   line=Line(width=1.25, color=line_color),

                  )

def make_XAxis(axis_style):
    return XAxis(axis_style)

def make_YAxis(axis_style):
    return YAxis(axis_style)

figure = tls.make_subplots(rows=1, cols=2,
                           subplot_titles=('Delaunay triangulation',  'Alpha shape, alpha=0.15'),
                           horizontal_spacing=0.1,
                          )

pl_width=800
pl_height=460
title = 'Delaunay triangulation and Alpha Complex/Shape for a Set of 2D Points'

figure['layout'].update(title=title,
                        font= Font(family="Open Sans, sans-serif"),
                        showlegend=False,
                        hovermode='closest',
                        autosize=False,
                        width=pl_width,
                        height=pl_height,
                        margin=Margin(
                            l=65,
                            r=65,
                            b=85,
                            t=120
                            ),
                       shapes=[]
                       )

axis_style...


I am working on the file "Plotly-Mesh3d.ipynb.ipynb".
The first task is:
If $T$ is the set of points defining a $k$-simplex, then any proper subset of $T$ defines an $\ell$-simplex, with $\ell<k$.
These $\ell$-simplexes (or $\ell$-simplices) are called faces.
A 2-simplex has three $1$-simplexes, and three 0-simplexes as faces, whereas a tetrahedron has as faces three  2-simplexes, six 1-simplexes and four zero simplexes.
k-simplexes are building blocks for different structures in Computational Geometry, mainly for creating meshes from point clouds.
Let $S$ be a finite set in $\mathbb{R}^d$, $d=2,3$ (i.e. a set of 2D or 3D points). A collection  $\mathcal{K}$ of k-simplexes, $0\leq k\leq d$,  having as vertices  the  points of $S$,
 is a simplicial complex if its simplexes have the following properties:
  1. If $\sigma$ is a simplex in $\mathcal{K}$, then all its faces are also simplexes in $\mathcal{K}$;
  2. If $\sigma, \tau$ are two simplexes in $\mathcal{K}$, then their intersection is either empty or a face in both simplexes.
The next figure illustrates a simplicial complex(left), and a collection of $k$-simplexes (right), $0\leq k\leq 2$
that do not form a simplicial complex because the condition 2 in the definition above is violated.
Can you write Python code for it?

HTML('<iframe src=https://plot.ly/~empet/13503/ width=600 height=475></iframe>')


I am working on the file "Plotly-Mesh3d.ipynb.ipynb".
The first task is:
Load data:
Can you write Python code for it?

pts=np.loadtxt('data-file.txt')
x,y,z=zip(*pts)

import plotly.plotly as py
from plotly.graph_objs import *
from plotly import tools as tls


I am working on the file "Plotly-Mesh3d.ipynb.ipynb".
The first task is:
We notice in the subplots above that as alphahull increases, i.e. $\alpha$ decreases, some parts of the alpha shape shrink and
develop enclosed void regions. The last plotted alpha shape points out a polytope that contains faces of tetrahedra,
and patches of triangles.
In some cases as $\alpha$ varies it is also possible to develop  components that are strings of edges and even isolated points.
Such experimental results suggested the use of alpha shapes in modeling molecular structure.
A search on WEB gives many results related to applications of alpha shapes in structural molecular biology.
Here
is an alpha shape illustrating a molecular-like structure associated to a point set of 5000 points.
Generating an alpha shape with Mesh3d
Can you write Python code for it?


I am working on the file "Plotly-Mesh3d.ipynb.ipynb".
The first task is:
We notice that the Delaunay triangulation has as boundary a convex set (it is a triangulation of the convex hull
                                                                        of the given point set).
Each $\alpha$-complex is obtained from the Delaunay triangulation, removing  the triangles whose circumcircle has radius greater or equal to alpha.
In the last subplot the triangles of the  $0.115$-complex are filled in with light blue. The filled in region  is the underlying space of the $0.115$-complex.
The $0.115$-alpha shape of the given point set can be considered either the filled in region or its boundary.
This example illustrates that the underlying space of an  $\alpha$-complex in neither convex nor necessarily connected. It can consist in many connected components (in our illustration above, $|\mathcal{C}_{0.115}|$ has three components).
In a family of alpha shapes, the parameter $\alpha$ controls the level of detail of the associated alpha shape.  If $\alpha$ decreases to zero, the corresponding alpha shape degenerates to the point set, $S$, while  if it tends to infinity the alpha shape tends to the convex hull of the set $S$.
Plotly Mesh3d
In order to generate the alpha shape of a given set of 3D points corresponding to a parameter $\alpha$,
the Delaunay triagulation or the convex hull we define a Mesh3d object or a dict. The real value
of the key alphahull points out the mesh type to be generated:
alphahull=$1/\alpha$    generates the $\alpha$-shape, -1 corresponds to the Delaunay
triangulation and 0, to the convex hull of the point set.
The other keys in the definition of a Mesh3d are given here.
Mesh3d generates and displays an $\alpha$-shape as the boundary of the $\alpha$-complex.
An intuitive idea on the topological structure modification, as $\alpha=1/$alphahull varies can be gained from the following three different alpha shapes of the same point set:
Can you write Python code for it?

HTML('<iframe src=https://plot.ly/~empet/13481/ width=900 height=950></iframe>')


I am working on the file "Plotly-Slice-in-volumetric-data.ipynb".
The first task is:
Define a function that returns a slice as a Plotly Surface:
Can you write Python code for it?

def get_the_slice(x,y,z, surfacecolor):
    return go.Surface(x=x,
                      y=y,
                      z=z,
                      surfacecolor=surfacecolor,
                      coloraxis='coloraxis')

def get_lims_colors(surfacecolor):# color limits for a slice
    return np.min(surfacecolor), np.max(surfacecolor)


I am working on the file "Plotly-Slice-in-volumetric-data.ipynb".
The first task is:
Let us plot the slices z=0 and y=-0.5 in the volume defined by:
Can you write Python code for it?

scalar_f = lambda x,y,z: x*np.exp(-x**2-y**2-z**2)

x = np.linspace(-2,2, 50)
y = np.linspace(-2,2, 50)
x, y = np.meshgrid(x,y)
z = np.zeros(x.shape)
surfcolor_z = scalar_f(x,y,z)
sminz, smaxz = get_lims_colors(surfcolor_z)

slice_z = get_the_slice(x, y, z, surfcolor_z)

x = np.linspace(-2,2, 50)
z = np.linspace(-2,2, 50)
x, z = np.meshgrid(x,y)
y = -0.5 * np.ones(x.shape)
surfcolor_y = scalar_f(x,y,z)
sminy, smaxy = get_lims_colors(surfcolor_y)
vmin = min([sminz, sminy])
vmax = max([smaxz, smaxy])
slice_y = get_the_slice(x, y, z, surfcolor_y)


I am working on the file "Plotly-Slice-in-volumetric-data.ipynb".
The first task is:
Oblique slice in volumetric data
As an example we plot comparatively two slices: a slice through $z=0$ and an oblique planar slice, that is defined by rotating the plane z=0 by $\alpha=\pi/4$, about Oy.
Rotating the plane $z=c$ about Oy (from Oz towards Ox) with $\alpha$ radians we get the plane of equation
$z=c/\cos(\alpha)-x\tan(\alpha)$
Can you write Python code for it?

alpha = np.pi/4
x = np.linspace(-2, 2, 50)
y = np.linspace(-2, 2, 50)
x, y = np.meshgrid(x,y)
z = -x * np.tan(alpha)

surfcolor_obl = scalar_f(x,y,z)

smino, smaxo = get_lims_colors(surfcolor_obl)
vmin = min([sminz, smino])
vmax = max([smaxz, smaxo])

slice_obl = get_the_slice(x,y,z, surfcolor_obl)

fig2 = go.Figure(data=[slice_z, slice_obl], layout=fig1.layout)
fig2.update_layout( coloraxis=colorax(vmin, vmax))

#fig2.show()

IFrame('https://chart-studio.plotly.com/~empet/13864', width=700, height=700)

from IPython.core.display import HTML
def  css_styling():
    styles = open("./custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Plotly-cube.ipynb".
The first task is:
Crop the image:
Can you write Python code for it?

my_img=img[10:-10, :, :]
my_img.shape

plt.imshow(my_img)


I am working on the file "Plotly-cube.ipynb".
The first task is:
Define the array  "equations" of cube faces.
The    upper face has the equation zM=L-1, the lower one, zm=0, and similarly for x=constant faces and y=constant faces:
Can you write Python code for it?


I am working on the file "Plotly-cube.ipynb".
The first task is:
In order to define a cube face as a Plotly Surface, it is referenced to a positively oriented cartesian system of coordinates, (X,Y), associated to the induced planar coordinate system of that face (when looking at it  from the outside) from the 3d system of coordinates of the cube.
The image represented by pl_img is then  fitted to this system of coordinates,  eventually by  flipping its rows or columns.
The Surface instances, representing the cube faces, are defined as follows:
Can you write Python code for it?

trace_zm=make_cube_face(x=X, y=Y, z=zm,  is_scl_reversed=True, surfacecolor=pl_img)
trace_zM=make_cube_face(x=X, y=Y, z=zM,  is_scl_reversed=True, surfacecolor=np.flipud(pl_img))
trace_xm=make_cube_face(x=zm, y=Y, z=X, surfacecolor=np.flipud(pl_img))
trace_xM=make_cube_face(x=zM, y=Y, z=X, surfacecolor=pl_img)
trace_ym=make_cube_face(x=Y, y=zm, z=X, surfacecolor=pl_img)
trace_yM=make_cube_face(x=Y, y=zM, z=X, surfacecolor=np.fliplr(pl_img))


I am working on the file "Plotly-cube.ipynb".
The first task is:
Set the plot layout:
Can you write Python code for it?

noaxis=dict(
            showbackground=False,
            showgrid=False,
            showline=False,
            showticklabels=False,
            ticks='',
            title='',
            zeroline=False)

min_val=-0.01
max_val=L-1+0.01

layout = Layout(
         title="",
         width=500,
         height=500,
         scene=Scene(xaxis=XAxis(noaxis, range=[min_val, max_val]),
                     yaxis=YAxis(noaxis, range=[min_val, max_val]),
                     zaxis=ZAxis(noaxis, range=[min_val, max_val]),
                     aspectratio=dict(x=1,
                                      y=1,
                                      z=1
                                     ),
                     camera=dict(eye=dict(x=-1.25, y=-1.25, z=1.25)),
                    ),

        paper_bgcolor='rgb(240,240,240)',
        hovermode='closest',
        margin=dict(t=50)
        )

fig=Figure(data=Data([trace_zm, trace_zM, trace_xm, trace_xM, trace_ym, trace_yM]), layout=layout)
py.sign_in('empet', 'api_key')
py.iplot(fig, filename='Plotly-cube')

from IPython.core.display import HTML
def  css_styling():
    styles = open("./custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Plotly-cube.ipynb".
The first task is:
Since our image contains only two colors (white and blue) we select from my_img the array corresponding to the blue chanel:
Can you write Python code for it?

pl_img=my_img[:,:, 2] #
L, C=pl_img.shape
assert L==C

plotly_blue='rgb(68, 122, 219)'# the  blue color in Plotly logo

import plotly.plotly as py
from plotly.graph_objs import *


I am working on the file "Plotly-cube.ipynb".
The first task is:
The next function returns  a Surface:
Can you write Python code for it?

def make_cube_face(x,y,z, colorscale=pl_scl, is_scl_reversed=False,
                   surfacecolor=pl_img, text='Plotly cube'):
    return Surface(x=x, y=y, z=z,
                   colorscale=colorscale,
                   reversescale=is_scl_reversed,
                   showscale=False,
                   surfacecolor=surfacecolor,
                   text=text,
                   hoverinfo='text'
                  )


I am working on the file "Plotly-interactive-visualization-of-complex-valued-functions.ipynb".
The first task is:
Define a function that associates to each point $z$ in a meshgrid, the strings containing the values to be displayed when hovering the mouse over that point:
Can you write Python code for it?

def text_to_display(x, y, argum, modulus):
    m, n = argum.shape
    return [['z='+'{:.2f}'.format(x[j])+'+' + '{:.2f}'.format(y[i])+' j'+'
arg(f(z))='+'{:.2f}'.format(argum[i][j])+\
       '
log(modulus(f(z)))='+'{:.2f}'.format(modulus[i][j])  for j in range(n)] for i in range(m)]


I am working on the file "Plotly-interactive-visualization-of-complex-valued-functions.ipynb".
The first task is:
Finally, the function plotly_plot calls all above defined functions in order to generate the phase plot
of a given complex function:
Can you write Python code for it?

def plotly_plot(f, re=(-1,1), im=(-1,1), N=50, nr=10, title='', width=500, height=500, **kwargs):
    x, y, argument, log_modulus=evaluate_function(f, re=re, im=im, N=N)

    levels = get_levels(log_modulus, nr=nr)
    plt.figure(figsize=(0.05,0.05))
    plt.axis('off')
    cp = plt.contour(x,y, log_modulus, levels=levels)
    cl = plotly_contour_lines(cp)
    text = text_to_display(x,y, argument, log_modulus.data)
    tickvals = [-np.pi, -2*np.pi/3, -np.pi/3, 0, np.pi/3, 2*np.pi/3, np.pi]

    #define the above values as strings with pi-unicode
    ticktext=['-\u03c0', '-2\u03c0/3', '-\u03c0/3', '0', '\u03c0/3', '2\u03c0/3', '\u03c0']

    dom = go.Heatmap(x=x, y=y, z=argument,  colorscale=pl_hsv,
                    text=text, hoverinfo='text',
                    colorbar=dict(thickness=20, tickvals=tickvals,
                    ticktext=ticktext,
                    title='arg(f(z))'))
    if len(cl) == 2  and isinstance(cl[0], go.Scatter):
        data = [dom, cl[0], cl[1]]
    else:
        data = [dom, cl]

    layout = set_plot_layout(title=title)
    fig=go.Figure(data=data, layout =layout)
    return fig


I am working on the file "Plotly-interactive-visualization-of-complex-valued-functions.ipynb".
The first task is:
Set the layout of the plot:
Can you write Python code for it?

def  set_plot_layout(title, width=500, height=500):

    return go.Layout(title_text=title,
                     title_x=0.5,
                     width=width,
                     height=height,
                     showlegend=False,
                     xaxis_title='Re(z)',
                     yaxis_title='Im(z)')


I am working on the file "Plotly-interactive-visualization-of-complex-valued-functions.ipynb".
The first task is:
We extract the contour lines from an attribute of the matplotlib.contour.QuadContourSet object returned by the matplotlib.pyplot.contour.
The function defined in the next cell retrieves the points on the contour lines segments, and defines the corresponding Plotly traces:
Can you write Python code for it?

def plotly_contour_lines(contour_data):
    #contour_data is a matplotlib.contour.QuadContourSet object returned by plt.contour
    contours=contour_data.allsegs  #
    if len(contours)==0:
        raise ValueError('Something wrong hapend in computing contour lines')
    #contours is a list of lists; if contour is a list in contours, its elements are arrays.
    #Each array defines a segment of contour line at some level
    xl = []# list of x coordinates of points on a contour line
    yl = []#         y
    #lists of coordinates for contour lines consisting in one point:
    xp = []#
    yp = []

    for k,contour in enumerate(contours):
        L = len(contour)
        if L!= 0: # sometimes the list of points at the level np.max(modulus) is empty
            for ar in contour:
                if ar.shape[0] == 1:
                    xp += [ar[0,0], None]
                    yp += [ar[0,1], None]
                else:
                    xl += ar[:,0].tolist()
                    yl += ar[:,1].tolist()
                    xl.append(None)
                    yl.append(None)

    lines = go.Scatter(x=xl,
                       y=yl,
                       mode='lines',
                       name='modulus',
                       line=dict(width=1,
                            color='#a5bab7',
                            shape='spline',
                            smoothing=1),
                       hoverinfo='skip'
                 )
    if len(xp) == 0:
        return lines
    else:
        points = go.Scatter(x=xp, y=yp, mode='markers',
                            marker=dict(size=4, color='#a5bab7'),
                            hoverinfo='none')
        return lines, points


I am working on the file "Plotnine/Plotnine.ipynb".
The first task is:
Bar Chart
Let's plot the weight of each brand
Can you write Python code for it?

(gg.ggplot(mtcars, gg.aes(x='brand', y='wt'))
    + gg.geom_bar(stat='identity'))


I am working on the file "Plotnine/Plotnine.ipynb".
The first task is:
Box Plot
Let's build a boxplot, and see the distribution of horspower on wether a car is automatic or manual
Can you write Python code for it?

(gg.ggplot(mtcars, gg.aes(y='hp', x='factor(gear)'))
 + gg.geom_boxplot())


I am working on the file "Plotnine/Plotnine.ipynb".
The first task is:
Plotnine
plotnine is an implementation of a grammar of graphics in Python, it is based on ggplot2. The grammar allows users to compose plots by explicitly mapping data to the visual objects that make up the plot. - Plotnine Official Website

Plotting with a grammar is powerful, it makes custom (and otherwise complex) plots are easy to think about and then create, while the simple plots remain simple.

Let's take a create a quick example using the mtcars dataset.
Can you write Python code for it?

DATASET_URL = 'https://gist.githubusercontent.com/ZeccaLehn/4e06d2575eb9589dbe8c365d61cb056c/raw/64f1660f38ef523b2a1a13be77b002b98665cdfe/mtcars.csv'
mtcars = pd.read_csv(DATASET_URL).rename(columns={'Unnamed: 0': 'brand'})

mtcars.head()


I am working on the file "Plotnine/Plotnine.ipynb".
The first task is:
Scatter Plot
Can you write Python code for it?

(gg.ggplot(mtcars, gg.aes('disp', 'mpg', color='factor(gear)'))
 + gg.geom_point()
 + gg.geom_smooth(method="lm")
 + gg.facet_wrap("~gear"))


I am working on the file "Plotnine/Plotnine.ipynb".
The first task is:
The brands are all quished together, let's try making the bar graph horizontal
Can you write Python code for it?

(gg.ggplot(mtcars, gg.aes(x='brand', y='wt'))
    + gg.geom_bar(stat='identity')
    + gg.coord_flip())

(gg.ggplot(mtcars, gg.aes(x='gear', y='hp', fill="factor(brand)"))
    + gg.geom_bar(stat='identity', color="black")
    + gg.coord_flip()
    + gg.theme_xkcd())

plt.barh(mtcars['brand'], mtcars['wt'])
plt.xlabel('Brand')
plt.ylabel('wt')
plt.title('Brand by weight')
plt.show()


I am working on the file "Plotnine/Plotnine.ipynb".
The first task is:
The matplotlib equivalent
Can you write Python code for it?

plt.bar(mtcars['brand'], mtcars['wt'])
plt.xlabel('Brand')
plt.ylabel('wt')
plt.title('Brand by weight')
plt.show()


I am working on the file "Plots related to the drift barrier and mutation rate landscape catastrophe-Revisions.ipynb".
The first task is:
I need to remove the runs where the distribution of mutations got screwed up because the whole population had a mutation rate of 1.
Can you write Python code for it?

for mud in mu_dists[200]:
    print(mud[0])

for mud in mu_dists[400]:
    print(mud[0])

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
bar_plot_mudist(ax, *mean_std_mu_dists([mu_dists[200][i] for i in [0,2,5,6,7,8,9,10,11,13,14,16,17]]))
ax.set_xlabel('Mode of the mutation rate, $\mu_{mode}$', fontsize=28)
ax.set_ylabel('Probability', fontsize=28)
plt.savefig('db_mu_mode_distK200.pdf')

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
bar_plot_mudist(ax, *mean_std_mu_dists(mu_dists[1600]))
ax.set_xlabel('Mode of the mutation rate, $\mu_{mode}$', fontsize=28)
ax.set_ylabel('Probability', fontsize=28)
plt.savefig('db_mu_mode_distK1600.pdf')

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
bar_plot_mudist(ax, *mean_std_mu_dists(mu_dists[25600]))
ax.set_xlabel('Mode of the mutation rate, $\mu_{mode}$', fontsize=28)
ax.set_ylabel('Probability', fontsize=28)
plt.savefig('db_mu_mode_distK25600.pdf')


I am working on the file "Plots related to the drift barrier and mutation rate landscape catastrophe-Revisions.ipynb".
The first task is:
The mutation rate landscape catastrophe
Can you write Python code for it?

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
plt.plot(f_maxes[200][0][:5*10**4],marker='')
ax.set_xlabel('generation', fontsize=28)
ax.set_ylabel('maximum fitness, $f_{max}$', fontsize=28);
plt.savefig('fitness_catastrophe.pdf')

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
plot_mu_trajectory(ax, mu_modes[200][0][:5*10**4])
ax.set_xlabel('generation', fontsize=28);
ax.set_ylabel('mode of the mutation rate, $\mu_{mode}$', fontsize=28);
plt.savefig('mutation_rate_during_catastrophe.pdf')

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
plt.plot(f_maxes[400][0][:5*10**4],marker='')
ax.set_xlabel('generation', fontsize=28)
ax.set_ylabel('maximum fitness, $f_{max}$', fontsize=28);
plt.savefig('fitness_not_catastrophe.pdf')

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
plot_mu_trajectory(ax, mu_modes[400][0][:5*10**4])
ax.set_xlabel('generation', fontsize=28);
ax.set_ylabel('mode of the mutation rate, $\mu_{mode}$', fontsize=28);
plt.savefig('mutation_rate_not_catastrophe.pdf')


I am working on the file "Plots related to the traveling wave regime-Revisions.ipynb".
The first task is:
Given a delta_f of .03 and a K of 1 million, we expect it would take about 450 generations for a new mutation increasing fitness by one step to fix after it invades the population
Can you write Python code for it?

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111, projection='3d')
logNbarplot(ax, basepop(15000))
plt.savefig('traveling_wave_full_distribution.pdf')

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
logNvsfplot(ax, basepop(15000))
plt.savefig('traveling_wave_f_distribution.pdf')

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
logNvsmuplot(ax, basepop(15000))
plt.savefig('traveling_wave_mu_distribution.pdf')

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
plt.plot(basepop.mode_fitness[:10000], marker='');
ax.set_xlabel('generation', fontsize=32);
ax.set_ylabel('mode of population fitness, $f_{mode}$', fontsize=28);
plt.savefig('traveling_wave_f_evolution.pdf')

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
plt.semilogy(basepop.mode_mutation_rate[:], marker='');
ax.minorticks_off()
ax.get_yaxis().set_major_formatter(matplotlib.ticker.ScalarFormatter())
ax.set_yticks(np.unique(basepop.mode_mutation_rate[:]))
ax.set_xlabel('generation', fontsize=32);
ax.set_ylabel('mode of population\nmutation rate, $\mu_{mode}$', fontsize=28);
plt.savefig('traveling_wave_mu_evolution.pdf')


I am working on the file "Plots related to the traveling wave regime-Revisions.ipynb".
The first task is:
Varying the probability of beneficial mutations
Can you write Python code for it?

fbs = [pop.fraction_beneficial for pop in fbvary_results]
dfdts = [mean_df_dt(pop) for pop in fbvary_results]
fbs, mean_dfdts, std_dfdts = mean_and_std_by(fbs, dfdts)

fig = plt.figure(figsize=(10,10))
plt.errorbar(fbs, mean_dfdts, 2*std_dfdts, linestyle='');
plt.xscale('log')
plt.yscale('log')
plt.xlabel('probability of a beneficial mutation $P_{b}$', fontsize=28);
plt.ylabel(r'rate of fitness increase $\frac{df_{mean}}{dt}$', fontsize=28);
plt.savefig('df_dt_varying_P_b.pdf')

fbs = [pop.fraction_beneficial for pop in fbvary_results]
mu_avs = [np.mean(pop.mean_mutation_rate[:]) for pop in fbvary_results]
fbs, mean_mu_avs, std_mu_avs = mean_and_std_by(fbs, mu_avs)

fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
ax.errorbar(fbs, mean_mu_avs, 2*std_mu_avs, linestyle='')
ax.set_xscale('log')
ax.set_yscale('log')
ax.minorticks_off()
ax.get_yaxis().set_major_formatter(matplotlib.ticker.ScalarFormatter())
ax.set_yticks([.00125, .0025, .005, .01, .02, .04])
ax.set_xlabel('probability of a beneficial mutation $P_{b}$', fontsize=28);
ax.set_ylabel('mean mutation rate, $\mu$,\n(over population and time)', fontsize=28);
plt.savefig('mean_mu_varying_P_b.pdf')


I am working on the file "Plots.ipynb".
The first task is:
Build the data used for the figure
Can you write Python code for it?

df['paidoff'] = (df.stat== 1)

bystate = pd.DataFrame()
bystate['mean_rate'] = df.groupby([u'addr_state']).mean().paidoff
bystate['default_rate'] = 1- df.groupby([u'addr_state']).mean().paidoff

bystate['count'] = df.groupby([u'addr_state']).count().paidoff
bystate['fico_range_high'] = df.groupby([u'addr_state']).mean().fico_range_high
bystate['fico_range_high'] = df.groupby([u'addr_state']).mean().fico_range_high

bystate.to_csv('bystate.csv')

df[df.addr_state=='IA']

from bokeh.plotting import figure, show, output_file


I am working on the file "Plots.ipynb".
The first task is:
Loan by credit score
Can you write Python code for it?

byfico = pd.DataFrame()
byfico['mean_rate'] = df.groupby([u'fico_range_high']).mean().paidoff.iloc[2:]
byfico['default_rate'] = 1- df.groupby([u'fico_range_high']).mean().paidoff[2:]
byfico['count'] = df.groupby([u'fico_range_high']).count().paidoff[2:]

# TOOLS = "hover,save"
# xfico = [(str(int(byfico.index[i]-4)) + ' - ' +str(int(byfico.index[i]))) for i in xrange(len(byfico.index))]

# p = figure(background_fill="#EFE8E2",
#            x_axis_label='Pay off rate (%)', y_axis_label='FICO score',
#            x_range = [0, 100] ,
#            y_range = xfico[::-1],
#            title="Loan Outcome by FICO score",
#            tools = TOOLS,
#            plot_width=800,
#            plot_height=600)

# source1 = ColumnDataSource(
#     data=dict(fico_score=xfico, dft_rate=byfico['default_rate'].iloc[::-1]*100,payoff_rate=byfico['mean_rate'].iloc[::-1]*100)
# )
# source2 = ColumnDataSource(
#     data=dict(fico_score=xfico, dft_rate=byfico['default_rate'].iloc[::-1]*100,payoff_rate=byfico['mean_rate'].iloc[::-1]*100)
# )

# p.rect(byfico['mean_rate'].iloc[::-1]*100/2,
#        xfico[::-1],byfico['mean_rate'].iloc[::-1]*100 ,0.8,
#        fill_color="#08c994", source = source1)

# p.rect(byfico['default_rate'].iloc[::-1]*100/2 + byfico['mean_rate'].iloc[::-1]*100,
#        xfico[::-1], byfico['default_rate'].iloc[::-1]*100, 0.8,
#        fill_color="#ff5a00", source = source2)

# hover = p.select(dict(type=HoverTool))
# hover.tooltips = OrderedDict([
#     ('FICO score', "$y"),
#     ('Payoff rate (%)', '@payoff_rate'),
#     ('Default rate (%)', '@dft_rate'),

# ])

# show(p)

df.columns

df['dti_bin'] = np.floor(df.dti/5)*5

(pd.unique(df.emp_length))

# N = 100
# x = np.random.random(size=N) * 100
# y = np.random.random(size=N) * 100
# radii = np.random.random(size=N) * 1.5
# colors = ["#%02x%02x%02x" % (r, g, 150) for r, g in zip(np.floor(50+2*x), np.floor(30+2*y))]
# colors


I am working on the file "Plots.ipynb".
The first task is:
Payoff rate by home ownership and emp_length
Can you write Python code for it?

# data =  df.groupby(['home_ownership','emp_length']).mean().paidoff

# from collections import OrderedDict

# import numpy as np

# from bokeh.plotting import ColumnDataSource, figure, show, output_file
# from bokeh.models import HoverTool

# # Read in the data with pandas. Convert the year column to string
# home_ownership = ['RENT', 'OWN', 'MORTGAGE']
# emp_lenght = [ 'n/a', '< 1 year', '1 year', '3 years', '2 years', '4 years', '5 years',
#         '6 years', '7 years', '8 years', '9 years','10+ years']

# # colors = [
# #     "#08C994", "#26BB81", "#45AD6F", "#649F5C", "#83914A",
# #     "#A28337", "#C17525", "#E06712", "#FF5A00"
# # ]
# colors = [
#     "#75968f", "#a5bab7", "#c9d9d3", "#e2e2e2", "#dfccce",
#     "#ddb7b1", "#cc7878", "#933b41", "#550b1d"
# ]
# a=sorted(data.values)[::7]

# home = []
# emp = []
# color = []
# rate = []
# for y in emp_lenght:
#     for m in home_ownership:
#         home.append(m)
#         emp.append(y)
#         rate_by_home_emp = data[m][y]
#         rate.append(rate_by_home_emp*100)
#         for i in xrange(1,9):
#             if rate_by_home_emp > a[i-1] and rate_by_home_emp<= a[i]:
#                 ci = 9-i
# #         ci = int((rate_by_home_emp - min(data))/(max(data)-min(data))*8.9)

#         color.append(colors[ci])

# output_notebook()

# TOOLS = "hover,save"

# p = figure(
#     x_axis_label='Employment length (%)', y_axis_label='House Ownership',
#     y_range=home_ownership, x_range=emp_lenght ,
#     x_axis_location="above", plot_width=800, plot_height=400,
#     toolbar_location="left", tools=TOOLS)

# source = ColumnDataSource(
#     data=dict(home=home, emp=emp,color=color, rate=rate)
# )
# p.rect("emp", "home", 1, 1, source=source,  color="color",line_color=None)

# p.grid.grid_line_color = None
# p.axis.axis_line_color = None
# p.axis.major_tick_line_color = None
# p.axis.major_label_text_font_size = "12pt"
# p.axis.major_label_standoff = 0
# p.xaxis.major_label_orientation = np.pi/3

# hover = p.select(dict(type=HoverTool))
# hover.tooltips...


I am working on the file "Plots_technical_background.ipynb".
The first task is:
Support vector machine
This is matlab code on how the image demonstrating linear vs RBF kernel was made (svm_kernel.png)
Can you write Python code for it?

'''
clear
clc
load('d2.mat');

hold on
gscatter(X(:,1),X(:,2),Y,'rb','x+',6);

SVMstruct = svmtrain(X,Y,'boxconstraint',1,'autoscale',false,'kernel_function','RBF');

% Make a grid of values to classify the entire space
x1_axis = linspace(min(X(:,1)), max(X(:,1)), 1000)';
x2_axis = linspace(min(X(:,2)), max(X(:,2)), 1000)';

[x1_space, x2_space] = meshgrid(x1_axis, x2_axis);

for i = 1:size(x1_space, 2)
   point_in_space = [x1_space(:, i), x2_space(:, i)];
   class(:, i) = svmclassify(SVMstruct, point_in_space);
end

% Plot the SVM boundary
hold on
contour(x1_space, x2_space, class, [0 0], 'k');
legend('-1','1','RBF boundary');
xlabel('x1');
ylabel('x2');
title('Data points and decision boundary');
hold off;
'''

import pickle
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.ensemble import RandomForestClassifier

def get2Grams(payload_obj):
    '''Divides a string into 2-grams

    Example: input - payload: "<script>"
             output- ["<s","sc","cr","ri","ip","pt","t>"]
    '''
    payload = str(payload_obj)
    ngrams = []
    for i in range(0,len(payload)-2):
        ngrams.append(payload[i:i+2])
    return ngrams

classifier = pickle.load( open("data/tfidf_2grams_randomforest.p", "rb"))

def injection_test(inputs):
    variables = inputs.split('&')
    values = [ variable.split('=')[1] for variable in variables]
    print(values)
    return 'MALICIOUS' if classifier.predict(values).sum() > 0 else 'NOT_MALICIOUS'

injection_test('var1=<rip cookie')

def get1Grams(payload_obj):
    '''Divides a string into 1-grams

    Example: input - payload: "<script>"
             output- ["<","s","c","r","i","p","t",">"]
    '''
    payload = str(payload_obj)
    ngrams = []
    for i in range(0,len(payload)-1):
        ngrams.append(payload[i:i+1])
    return ngrams

def get2Grams(payload_obj):
    '''Divides a string into 2-grams

    Example: input - payload: "<script>"
             output- ["<s","sc","cr","ri","ip","pt","t>"]
    '''
    payload = str(payload_obj)
 ...


I am working on the file "Plotting - seaborn.ipynb".
The first task is:
Controlling figure aesthetics (aka make your plots look awesome)
Seaborn provides a number of different aesthetics that can be activated with the set function. These are defined by three things:

The context affects things like label and line scaling for different output media. The contexts are "notebook" (default), "paper", "talk", and "poster". This makes it easy to reuse a figure from a paper for a poster with decent font scaling if you properly set your figsize to what the final output size should be. (Figsize is given in inches and the journals usually tell you what size your figures are allowed to be.) If you save your figures as PDF and include them in a LaTeX document you won't need to adjust the scaling anymore if you set it correctly with figsize.
The style affects how the coordinate axes look. Available styles are "darkgrid" (default), "whitegrid", "dark", "white", "ticks".
The palette affects the colors of your plot. There are a number of ways to generate different palettes in seaborn and I suggest checking the documentation because the best palette depends on the type of plot and data you plot.

Let us try out some of the combinations:
Can you write Python code for it?

sns.set(context='paper')
sinplot()

sns.set(style='whitegrid')
sinplot()

sns.set(style='ticks')
sinplot()

sns.set(palette='colorblind')
sinplot()


I am working on the file "Plotting - seaborn.ipynb".
The first task is:
Let us first define a function plotting some sine waves so that we have something to plot.
Can you write Python code for it?

def sinplot(flip=1):
    x = np.linspace(0, 14, 100)
    for i in range(1, 7):
        plt.plot(x, np.sin(x + i * .5) * (7 - i) * flip)


I am working on the file "Plotting - seaborn.ipynb".
The first task is:
More options for fitting polynomials, robust regression (against outliers), logistic regression, and more.
And easy to extent to more complex plots:
Can you write Python code for it?

sns.lmplot('total_bill', 'tip', hue='smoker', col='time', row='sex', data=tips, markers=['o', 'x'], palette='Set1')


I am working on the file "Plotting - seaborn.ipynb".
The first task is:
Plotting categorical data
Can you write Python code for it?

sns.stripplot('day', 'total_bill', data=tips)

sns.stripplot('day', 'total_bill', data=tips, jitter=True)

sns.swarmplot('day', 'total_bill', data=tips)

sns.swarmplot('day', 'total_bill', hue='time', data=tips)

sns.boxplot('day', 'total_bill', hue='sex', data=tips)

sns.violinplot('day', 'total_bill', hue='sex', data=tips)

sns.violinplot('day', 'total_bill', hue='sex', data=tips, split=True, inner='stick')

sns.barplot('day', 'total_bill', hue='sex', data=tips)

sns.pointplot('day', 'total_bill', hue='time', data=tips, dodge=True)


I am working on the file "Plotting - seaborn.ipynb".
The first task is:
Some options to make such a plot of discrete values nicer:
Can you write Python code for it?

sns.lmplot('size', 'tip', data=tips, x_jitter=.05)

sns.lmplot('size', 'tip', data=tips, x_estimator=np.mean)


I am working on the file "Plotting - seaborn.ipynb".
The first task is:
Some plots look even better when offsetting the axes.
Can you write Python code for it?

sns.set(style='ticks')
sinplot()
sns.despine(offset=10)

sns.set()  # Restore Seaborn defaults


I am working on the file "Plotting - seaborn.ipynb".
The first task is:
Statistical plotting functions
Seaborn not only improves the matplotlib styling, it also provides a number of (mainly statistical) plotting functions that make it easy to do a defined set of plots that are usually much more complicated in matplotlib. The following will just be a spotlight of what is offered by Seaborn. Refer to the documentation for the full glory.
The Seaborn plotting functions make have use of Pandas data frames and will use meta data from those frames to automatically label axes.
Visualizing distributions
Can you write Python code for it?

x = np.random.normal(size=100)

sns.distplot(x)

sns.distplot(x, hist=False, rug=True)

mean, cov = [0, 1], [(1, .5), (.5, 1)]
data = np.random.multivariate_normal(mean, cov, 200)
df = pd.DataFrame(data, columns=['x', 'y'])

sns.jointplot('x', 'y', data=df)

sns.jointplot('x', 'y', kind='hex', data=df)

sns.jointplot('x', 'y', kind='kde', data=df)

iris = sns.load_dataset('iris')
sns.pairplot(iris)


I am working on the file "Plotting - seaborn.ipynb".
The first task is:
Visualizing linear relationships
Mainly for exploratory analysis, to get actual quantitative measures use statsmodels.
Can you write Python code for it?

tips = sns.load_dataset('tips')
sns.lmplot('total_bill', 'tip', data=tips)

sns.lmplot('size', 'tip', data=tips)


I am working on the file "Plotting - seaborn.ipynb".
The first task is:
With pure matplotlib the plot looks somewhat "blunt":
Can you write Python code for it?


I am working on the file "Plotting Routines.ipynb".
The first task is:
Interactive Plotting
And it is possible to do interactive plotting where you can scan througha simulation - the function takes in a list of filepaths/names and generates a plot which cycles through them. Due to the interactive nature this cannot be saved as a png file, but should aid with visualization.
Can you write Python code for it?

import int_plot as ip
filelist = ['./example/noise_10kSI_MASP.h5',
             './example/noise_10kSI_MASP.h5',
             './example/noise_10kSI_MASP.h5']

t = ip.interactive_plot(filelist,'x','y')

show(t)

files_2 = ['./example/beam50k_A2S.h5',
           './example/beam50k_A2S.h5',
           './example/beam50k_A2S.h5']
y = ip.interactive_plot(files_2,'z_pos','std_y')
show(y)


I am working on the file "Plotting Routines.ipynb".
The first task is:
This generates a filename.html file that can be opened in any browser. For large datasets automatic binning will be implemented so that the raw data scatter plots do not fill up memory or make the file unusable. To create custom plots one can call custom_plot. This will overwrite the files generated above, so it is reccomended that file_name be filled in. Alternatively, don't use the same object to plot defaults and custom plots.
Can you write Python code for it?

FEL_2 = pt.ProcessedData(filepath,num_slices=100,undulator_period=0.00275,k_fact=2.7)
plots_2 = pt.Bokeh_Plotting(FEL_2)
a = plots_2.custom_plot(x_axis='x', y_axis='y', key='pop', plotter='circle', color='pink',
                  file_name='transverse', text_color='black', legend='thing',
                      title='Transverse position', save= True)


I am working on the file "Plotting_data.ipynb".
The first task is:
Question 1.1 complete!
This is the line chart plotting each day's temperature.
Can you write Python code for it?

weather['date'] = pd.to_datetime(weather['date'])

weather.head()

type(weather.date[0])

weather[['temp', 'humidity']].groupby(weather.date.dt.month).mean()

weather[['temp', 'humidity']].groupby(weather.date.dt.month).mean().plot(kind='bar')


I am working on the file "Plotting_data.ipynb".
The first task is:
Question 1.2 complete!
Bar chart showing temperature and humidity by month
Question 2
Use a scatterplot to show how the daily rental volume varies with temperature. Use a different series (with different colors) for each season.
Can you write Python code for it?

weather.head()

newseasons = {'Summer': 'Spring', 'Spring': 'Winter', 'Fall': 'Summer', 'Winter': 'Fall'}
weather['season_desc'] = weather['season_desc'].map(newseasons)

winter_slice = weather[weather.season_desc == 'Winter'][['temp', 'total_riders']]

weather[weather.season_desc == 'Winter'][['temp', 'total_riders']].plot()

weather[weather.season_desc == 'Spring'][['temp', 'total_riders']].plot()

spring_slice = weather[weather.season_desc == 'Spring'][['temp', 'total_riders']]
summer_slice = weather[weather.season_desc == 'Summer'][['temp', 'total_riders']]
fall_slice = weather[weather.season_desc == 'Fall'][['temp', 'total_riders']]

ax = winter_slice.plot(kind='scatter', x='total_riders', y='temp', color='b', alpha=0.25)
ax = spring_slice.plot(kind='scatter', x='total_riders', y='temp', color='g', alpha=0.25, ax=ax)
ax = summer_slice.plot(kind='scatter', x='total_riders', y='temp', color='r', alpha=0.25, ax=ax)
ax = fall_slice.plot(kind='scatter', x='total_riders', y='temp', color='y', alpha=0.25, ax=ax)
plt.show()


I am working on the file "Plotting_data.ipynb".
The first task is:
Question 2 Complete!
Here's the plot of all four seasons.
Question 3
Create another scatterplot to show how daily rental volume varies with windspeed.  As above, use a different series for each season.
Can you write Python code for it?

winter_slice = weather[weather.season_desc == 'Winter'][['windspeed', 'total_riders']]
spring_slice = weather[weather.season_desc == 'Spring'][['windspeed', 'total_riders']]
summer_slice = weather[weather.season_desc == 'Summer'][['windspeed', 'total_riders']]
fall_slice = weather[weather.season_desc == 'Fall'][['windspeed', 'total_riders']]

ax = winter_slice.plot(kind='scatter', x='total_riders', y='windspeed', color='b', alpha=0.25)
ax = spring_slice.plot(kind='scatter', x='total_riders', y='windspeed', color='g', alpha=0.25, ax=ax)
ax = summer_slice.plot(kind='scatter', x='total_riders', y='windspeed', color='r', alpha=0.25, ax=ax)
ax = fall_slice.plot(kind='scatter', x='total_riders', y='windspeed', color='y', alpha=0.25, ax=ax)
plt.show()


I am working on the file "Plurals (population learnability).ipynb".
The first task is:
A prediction is that the typology of number systems should roughly correspond to how learnable each type of number system is
Can you write Python code for it?

for n in ('sg / pl', 'sg / du / pl', 'sg / du / tr / pl', 'du / non-du', 'sg / du / tr / qu / pl'):
    plt.plot(range(1,len(P[n])+1), P[n], '-', linewidth=1.5, label=n)

plt.suptitle('Proportion of 100 learners who label all 15 items correctly')
plt.xlabel('Trials')
plt.legend(loc=(-0.55,0.5))


I am working on the file "Plurals (population learnability).ipynb".
The first task is:
Dual / non-dual distinction
Can you write Python code for it?

data['Outcomes'] = 'notdual'
data['Outcomes'][2] = 'dual'
data

P['du / non-du'] = [population_accuracy(M=i) for i in xrange(1,MAX_TRIALS)]

plt.plot(range(1,len(P['du / non-du'])+1), P['du / non-du'], '-', linewidth=2)
plt.title('Dual / non-dual distinction')
plt.xlabel('Trial Number')
plt.suptitle('Proportion of 100 learners who label all 15 items correctly')


I am working on the file "Plurals (population learnability).ipynb".
The first task is:
For a population of 100 learners trying to acquire the number system, what proportion are able to successfully label all 15 items given M trials?
Can you write Python code for it?

np.mean([accuracy(data, M=10) == 1 for i in xrange(100)])

def population_accuracy(M=10, pop=100):
    return np.mean([accuracy(data, M=M) == 1 for i in xrange(pop)])

MAX_TRIALS = 500
P = {}

P['sg / pl'] = [population_accuracy(M=i) for i in xrange(1,MAX_TRIALS)]

import matplotlib.pyplot as plt

plt.plot(range(1,len(P['sg / pl'])+1), P['sg / pl'], '-', linewidth=2)
plt.title('Singular / plural distinction')
plt.xlabel('Trial Number')
plt.suptitle('Proportion of 100 learners who label all 15 items correctly')


I am working on the file "Plurals (population learnability).ipynb".
The first task is:
How often are they correct (using relative item frequencies)?
Can you write Python code for it?

float(sum(data['Frequency'] * (A.idxmax(1) == data['Outcomes']))) / float(sum(data['Frequency']))

def accuracy(data, M):
    W = ndl.rw(data, M=M)
    A = activation(W)
    return np.mean(A.idxmax(1) == data['Outcomes'])

accuracy(data, 10)


I am working on the file "Plurals (population learnability).ipynb".
The first task is:
Singular, dual, plural
Can you write Python code for it?

data['Outcomes'] = 'plural'
data['Outcomes'][1] = 'singular'
data['Outcomes'][2] = 'dual'

P['sg / du / pl'] = [population_accuracy(M=i) for i in xrange(1,MAX_TRIALS)]

plt.plot(range(1,len(P['sg / du / pl'])+1), P['sg / du / pl'], '-', linewidth=2)
plt.title('Singular / dual / plural distinction')
plt.xlabel('Trial Number')
plt.suptitle('Proportion of 100 learners who label all 15 items correctly')


I am working on the file "Plurals (population learnability).ipynb".
The first task is:
Singular, dual, trial, plural
Can you write Python code for it?

data['Outcomes'] = 'plural'
data['Outcomes'][1] = 'singular'
data['Outcomes'][2] = 'dual'
data['Outcomes'][3] = 'trial'

P['sg / du / tr / pl'] = [population_accuracy(M=i) for i in xrange(1,MAX_TRIALS)]

plt.plot(range(1,len(P['sg / du / tr / pl'])+1), P['sg / du / tr / pl'], '-', linewidth=2)
plt.title('Singular / dual / trial / plural distinction')
plt.xlabel('Trial Number')
plt.suptitle('Proportion of 100 learners who label all 15 items correctly')


I am working on the file "Plurals (population learnability).ipynb".
The first task is:
With these associations, how many of the 15 items will the learner correctly label?
Can you write Python code for it?

pd.DataFrame([data['Outcomes'], A.idxmax(1), A.idxmax(1) == data['Outcomes']], index = ['Truth', 'Prediction', 'Accurate?']).T

np.mean(A.idxmax(1) == data['Outcomes'])


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
As we have used precedence declarations to resolve all shift/reduce conflicts, the action table contains no conflict.
Can you write Python code for it?


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
Below, we define the tokens for operator symbols consisting of more than one character.
Can you write Python code for it?


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
Given a token, the function find_colum returns the column where token starts.
This is possible, because token.lexer.lexdata stores the string that is given to the scanner and token.lexpos is the number of characters that precede token.
Can you write Python code for it?

def find_column(token):
    program    = token.lexer.lexdata
    line_start = program.rfind('\n', 0, token.lexpos) + 1
    return (token.lexpos - line_start) + 1


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
In the grammar rule
$$ \texttt{stmnt} \rightarrow \texttt{'if'}\;\texttt{'('}\; \texttt{bool_expr}\; \texttt{')'}\; \texttt{stmnt}$$
the rightmost token is ')'.  However, this token does not have a precedence.  Therefore, the grammar rule
does not have a precedence either.  Hence, we manually assign the precedence of the token IF to this rule via
the keyword %prec.  This way, the shift/reduce conflict resulting from the dangling-else ambiguity is resolved.
Can you write Python code for it?

def p_stmnt_if(p):
    "stmnt : IF '(' bool_expr ')' stmnt %prec IF"
    p[0] = ('if', p[3], p[5])

def p_stmnt_if_else(p):
    "stmnt : IF '(' bool_expr ')' stmnt ELSE stmnt"
    p[0] = ('if-else', p[3], p[5], p[7])

def p_stmnt_while(p):
    "stmnt : WHILE '(' bool_expr ')' stmnt"
    p[0] = ('while', p[3], p[5])

def p_stmnt_block(p):
    "stmnt : '{' stmnt_list '}'"
    p[0] = p[2]

def p_stmnt_assign(p):
    "stmnt : ID '=' expr ';'"
    p[0] = ('=', p[1], p[3])

def p_stmnt_return(p):
    "stmnt : RETURN expr ';'"
    p[0] = ('return', p[2])

def p_stmnt_expr(p):
    "stmnt : expr ';'"
    p[0] = p[1]

def p_bool_expr_or(p):
    "bool_expr : bool_expr OR bool_expr"
    p[0] = ('||', p[1], p[3])

def p_bool_expr_and(p):
    "bool_expr : bool_expr AND bool_expr"
    p[0] = ('&&', p[1], p[3])

def p_bool_expr_neg(p):
    "bool_expr : '!' bool_expr"
    p[0] = ('!', p[2])

def p_bool_expr_paren(p):
    "bool_expr : '(' bool_expr ')'"
    p[0] = p[2]

def p_bool_expr_eq(p):
    "bool_expr : expr EQ expr"
    p[0] = ('==', p[1], p[3])

def p_bool_expr_ne(p):
    "bool_expr : expr NE expr"
    p[0] = ('!=', p[1], p[3])

def p_bool_expr_le(p):
    "bool_expr : expr LE expr"
    p[0] = ('<=', p[1], p[3])

def p_bool_expr_ge(p):
    "bool_expr : expr GE expr"
    p[0] = ('>=', p[1], p[3])

def p_bool_expr_lt(p):
    "bool_expr : expr '<' expr"
    p[0] = ('<', p[1], p[3])

def p_bool_expr_gt(p):
    "bool_expr : expr '>' expr"
    p[0] = ('>', p[1], p[3])

def p_expr_plus(p):
    "expr : expr '+' expr"
    p[0] = ('+', p[1], p[3])

def p_expr_minus(p):
    "expr : expr '-' expr"
    p[0] = ('-', p[1], p[3])

def p_expr_times(p):
    "expr : expr '*' expr"
    p[0] = ('*', p[1], p[3])

def p_expr_divide(p):
    "expr : expr '/' expr"
    p[0] = ('/', p[1], p[3])

def p_expr_modulo(p):
    "expr : expr '%' expr"
    p[0] = ('%', p[1], p[3])

def p_expr_group(p):
    "expr : '(' expr ')'"
    p[0] = p[2]

def p_expr_number(p):
    "expr : NUMBER"
   ...


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
Our version of C allows both single line comments and multi line comments.
- The regular expression //[^\n]* recognizes single line comments.
  A single line comment starts with // and extends to the end of the line.
- The regular expression \/\*([^*]|\*+[^*/])*\*+\ recognizes multi line comments.
  Multi line comments start with the string /* and end with the string */.
  Between these strings, the string */ must not occur.
  * \/\* matches the opening /*.
  * [^*]|\*+[^*/] matches any character that is different from the character * as
    well as non-empty sequences of *s that are not followed by a /.
  * \*+/ matches any non-empty sequence of *s that is followed by a /.
  The expression needs to be this complicated in order to match multi line comments of the
  following form:
  /*** abc *** xyz ***/
Can you write Python code for it?


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
Setting the optional argument write_tables to False is required to prevent an obscure bug where the parser generator tries to read an empty parse table.
Can you write Python code for it?

parser = yacc.yacc(write_tables=False, debug=True)


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
Specification of the Parser
We will use the following grammar to specify the language that our compiler can translate:
```
program
    : function
    | function program
function
    : INT ID '(' param_list ')' '{' decl_list stmnt_list '}'
param_list
    : / epsilon /
    | INT ID
    | INT ID ',' ne_param_list
ne_param_list
    : INT ID
    | INT ID ',' ne_param_list
decl_list
    : / epsilon /
    | INT ID ';' decl_list
stmnt_list
    : / epsilon /
    | stmnt stmnt_list
stmnt
    : IF '(' bool_expr ')' stmnt
    | IF '(' bool_expr ')' stmnt ELSE stmnt
    | WHILE '(' bool_expr ')' stmnt
    | '{' stmnt_list '}'
    | ID '=' expr ';'
    | RETURN expr ';'
    | expr ';'
bool_expr
    : bool_expr '||' bool_expr
    | bool_expr '&&' bool_expr
    | '!' bool_expr
    | '(' bool_expr ')'
    | expr '==' expr
    | expr '!=' expr
    | expr '<=' expr
    | expr '>=' expr
    | expr '<'  expr
    | expr '>'  expr
expr: expr '+'  expr
    | expr '-' expr
    | expr '*' expr
    | expr '/' expr
    | expr '%' expr
    | '(' expr ')'
    | NUMBER
    | ID
    | ID '(' expr_list ')'
expr_list
    : / epsilon /
    | expr
    | expr ',' ne_expr_list
ne_expr_list
    : expr
    | expr ',' ne_expr_list
```
We will use precedence declarations to resolve the ambiguity that is inherent in this grammar.
Can you write Python code for it?


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
Syntactically, newline characters are ignored. However, we still need to keep track of them in order to know which line we are in.  This information is needed later for error messages.
Can you write Python code for it?

def t_newline(t):
    r'\n'
    t.lexer.lineno += 1
    return


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
The following is a test of the function compile_expr.
Can you write Python code for it?


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
The function parse takes a file_name as ist sole argument.  The file is read and parsed.
The resulting parse tree is visualized using graphviz.  It is important to reset the
attribute lineno of the scanner, for otherwise error messages will not have the correct line numbers.
Can you write Python code for it?

def parse(file_name):
    lexer.lineno = 1
    with open(file_name, 'r') as handle:
        program = handle.read()
    ast = yacc.parse(program)
    print(ast)
    return tuple2dot(ast)

!cat -n Examples/MySum.c

parse('Examples/MySum.c')


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
The function t_error is called for any token t that can not be scanned by the lexer.  In this case, t.value[0] is the first character that can not be recognized by the scanner.
Can you write Python code for it?

def t_error(t):
    column = find_column(t)
    print(f"Illegal character '{t.value[0]}' in line {t.lineno}, column {column}.")
    t.lexer.skip(1)


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
The method compile_bool(expr, st, class_name) takes three arguments:
- expr is an abstract syntax tree that represents a Boolean expression.
  This abstract syntax tree is in turn represented as a nested tuple.
- st is short for symbol table.  This is a dictionary that maps variable
  names to natural numbers.  Given a variable x, the number st[x] specifies
  the location where the variable x is stored on the stack with respect to the
  local stack frame.
- class_name is the name of the class that is to be generated.
The function returns a pair of the form (cmds, size).
- cmds is a list of assembler commands,
- size is the maximum size of the stack that is needed.
Can you write Python code for it?

def compile_bool(expr, st, class_name):
    if expr[0] in ['==', '!=', '<=', '>=', '<', '>']:
        OpToCmd = { '==': 'if_icmpeq',
                    '!=': 'if_icmpne',
                    '<=': 'if_icmple',
                    '>=': 'if_icmpge',
                    '<' : 'if_icmplt',
                    '>' : 'if_icmpgt'
                  }
        op, lhs, rhs = expr
        L1, sz1    = compile_expr(lhs, st, class_name)
        L2, sz2    = compile_expr(rhs, st, class_name)
        true_label = new_label()
        next_label = new_label()
        CmdLst     = L1 + L2
        cmd        = OpToCmd[op]
        CmdLst    += [indent(cmd + ' ' + true_label)]
        CmdLst    += [indent('bipush 0')]
        CmdLst    += [indent('goto ' + next_label)]
        CmdLst    += [' ' * 4 + true_label + ':']
        CmdLst    += [indent('bipush 1')]
        CmdLst    += [' ' * 4 + next_label + ':']
        return CmdLst, max(sz1, 1 + sz2)
    elif expr[0] in ['&&', '||']:
        op, lhs, rhs = expr
        OpToCmd      = { '&&': 'iand', '||': 'ior' }
        L1, sz1      = compile_bool(lhs, st, class_name)
        L2, sz2      = compile_bool(rhs, st, class_name)
        cmd          = OpToCmd[op]
        CmdLst       = L1 + L2 + [indent(cmd)]
        return CmdLst, max(sz1, 1 + sz2)
    elif expr[0] == '!':
        _, arg = expr
        L, sz  = compile_expr(arg, st, class_name)
        CmdLst = [indent('bipush 1')] + L + [indent('isub')]
        return CmdLst, sz + 1
    else:
        assert False, f'Error in compile_bool({expr}, {st}, {class_name})'


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
The method compile_expr(expr, st, class_name) takes three arguments:
- expr is an abstract syntax tree that represents an expression.
  This abstract syntax tree is in turn represented as a nested tuple.
- st is short for symbol table.  This is a dictionary that maps variable
  names to natural numbers.  Given a variable x, the number st[x] specifies
  the location where the variable x is stored on the stack with respect to the
  local stack frame.
- class_name is the name of the class that is to be generated.
The function returns a pair of the form (cmds, size).
- cmds is a list of assembler commands,
- size is the maximum size of the stack that is needed.
Can you write Python code for it?

def compile_expr(expr, st, class_name):
    if isinstance(expr, str):
        Cmd = indent(f'iload {st[expr]}')
        return [Cmd], 1
    elif expr[0] == 'Number':
        _, n = expr
        Cmd  = indent(f'ldc {n}')
        return [Cmd], 1
    elif expr[0] in  ['+', '-', '*', '/', '%']:
        op, lhs, rhs = expr
        L1, sz1 = compile_expr(lhs, st, class_name)
        L2, sz2 = compile_expr(rhs, st, class_name)
        OpToCmd = { '+': 'iadd', '-': 'isub', '*': 'imul', '/': 'idiv', '%': 'irem' }
        Cmd     = indent(OpToCmd[op])
        return L1 + L2 + [Cmd], max(sz1, 1 + sz2)
    elif expr[0] == 'call' and expr[1] == 'println':
        _, _, *args = expr
        CmdLst    = [indent('getstatic java/lang/System/out Ljava/io/PrintStream;')]
        stck_size = 0
        cnt       = 0
        for arg in args:
            L, sz_arg = compile_expr(arg, st, class_name)
            stck_size = max(stck_size, cnt + 1 + sz_arg)
            CmdLst   += L
            cnt      += 1
        CmdLst += [indent(f'invokevirtual java/io/PrintStream/println({"I"*cnt})V')]
        return CmdLst, stck_size
    elif expr[0] == 'call' and expr[1] != 'println':
        _, f, *args = expr
        CmdLst    = []
        stck_size = 0
        cnt       = 0
        for arg in args:
            L, sz_arg = compile_expr(arg, st, class_name)
            stck_size = max(stck_size, cnt + sz_arg)
            CmdLst   += L
            cnt      += 1
        CmdLst += [indent(f'invokestatic {class_name}/{f}({"I"*cnt})I')]
        return CmdLst, max(stck_size, 1)
    else:
        assert False, f'Error in compile_expr({expr}, {st}, {class_name})'


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
The method compile_stmnt(stmnt, st, class_name) takes three arguments:
- stmnt is an abstract syntax tree that represents a statement.
  This abstract syntax tree is in turn represented as a nested tuple.
- st is short for symbol table.  This is a dictionary that maps variable
  names to natural numbers.  Given a variable x, the number st[x] specifies
  the location where the variable x is stored on the stack with respect to the
  local stack frame.
- class_name is the name of the class that is to be generated.
The function returns a pair of the form (cmds, size).
- cmds is a list of assembler commands,
- size is the maximum size of the stack that is needed.
Can you write Python code for it?

def compile_stmnt(stmnt, st, class_name):
    if stmnt[0] == '=':
        _, var, expr = stmnt
        CmdLst, sz = compile_expr(expr, st, class_name)
        CmdLst    += [indent(f'istore {st[var]}')]
        return CmdLst, sz
    elif stmnt[0] == 'if':
        _, expr, sub_stmnt = stmnt
        L1, sz1    = compile_bool(expr, st, class_name)
        L2, sz2    = compile_stmnt(sub_stmnt, st, class_name)
        else_label = new_label()
        lbl_stmnt  = ' ' * 4 + else_label + ':'
        CmdLst = L1 + [indent(f'ifeq {else_label}')] + L2 + [lbl_stmnt]
        return CmdLst, max(sz1, sz2)
    elif stmnt[0] == 'if-else':
        _, expr, then_stmnt, else_stmnt = stmnt
        L1, sz1    = compile_bool(expr, st, class_name)
        L2, sz2    = compile_stmnt(then_stmnt, st, class_name)
        L3, sz3    = compile_stmnt(else_stmnt, st, class_name)
        else_label = new_label()
        next_label = new_label()
        if_stmnt   = indent(f'ifeq {else_label}')
        else_stmnt = ' ' * 4 + else_label + ':'
        next_stmnt = ' ' * 4 + next_label + ':'
        goto_stmnt = indent(f'goto {next_label}')
        CmdLst = L1 + [if_stmnt] + L2 + [goto_stmnt, else_stmnt] + L3 + [next_stmnt]
        return CmdLst, max(sz1, sz2, sz3)
    elif stmnt[0] == 'while':
        _, expr, body_stmnt = stmnt
        L1, sz1    = compile_bool(expr, st, class_name)
        L2, sz2    = compile_stmnt(body_stmnt, st, class_name)
        loop_label = new_label()
        next_label = new_label()
        if_stmnt   = indent(f'ifeq {next_label}')
        loop_stmnt = ' ' * 4 + loop_label + ':'
        next_stmnt = ' ' * 4 + next_label + ':'
        goto_stmnt = indent(f'goto {loop_label}')
        CmdLst = [loop_stmnt] + L1 + [if_stmnt] + L2 + [goto_stmnt, next_stmnt]
        return CmdLst, max(sz1, sz2)
    elif stmnt[0] == 'return':
        _, expr = stmnt
        CmdLst, sz = compile_expr(expr, st, class_name)
        CmdLst    += [indent('ireturn')]
        return CmdLst, sz
    elif stmnt[0] == '.':
        _,...


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
The next assignment is necessary to make the lexer think that the code given above is part of some file.
Can you write Python code for it?

__file__ = 'main'

lexer = lex.lex()

def test_scanner(file_name):
    with open(file_name, 'r') as handle:
        program = handle.read()
    print(program)
    lexer.input(program)
    lexer.lineno = 1
    return [t for t in lexer]

for t in test_scanner('Examples/MySum.c'):
    print(t)


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
The start variable of our grammar is program.
Can you write Python code for it?

start = 'program'

precedence = (
    ('nonassoc', 'IF'),
    ('nonassoc', 'ELSE'),
    ('left', 'OR'),
    ('left', 'AND'),
    ('right', '!'),
    ('nonassoc', 'EQ', 'NE', 'LE', 'GE', '<', '>'),
    ('left', '+', '-'),
    ('left', '*', '/', '%')
)

def p_program_one(p):
    "program : function"
    p[0] = ('program', p[1])

def p_program_more(p):
    "program : function program"
    p[0] = ('program', p[1]) + p[2][1:]

def p_function(p):
    "function : INT ID '(' param_list ')' '{' decl_list stmnt_list '}'"
    p[0] = ('fct', p[2], p[4], p[7], p[8])

def p_param_list_empty(p):
    "param_list :"
    p[0] = ('.', )

def p_param_list_one(p):
    "param_list : INT ID"
    p[0] = ('.', p[2])

def p_param_list_more(p):
    "param_list : INT ID ',' ne_param_list"
    p[0] = ('.', p[2]) + p[4][1:]

def p_ne_param_list_one(p):
    "ne_param_list : INT ID"
    p[0] = ('.', p[2])

def p_ne_param_list_more(p):
    "ne_param_list : INT ID ',' ne_param_list"
    p[0] = ('.', p[2]) + p[4][1:]

def p_decl_list_one(p):
    "decl_list :"
    p[0] = ('.',)

def p_decl_list_more(p):
    "decl_list : INT ID ';' decl_list"
    p[0] = ('.', p[2]) + p[4][1:]

def p_stmnt_list_one(p):
    "stmnt_list :"
    p[0] = ('.',)

def p_stmnt_list_more(p):
    "stmnt_list : stmnt stmnt_list"
    p[0] = ('.', p[1]) + p[2][1:]


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
The token Number specifies a natural number.
Can you write Python code for it?


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
The variable label_counter is a global counter that is used to create unique label names.
Every call of new_label creates a new, unique label.
Can you write Python code for it?

label_counter = 0

def new_label():
    global label_counter
    label_counter += 1
    return 'l' + str(label_counter)


I am working on the file "Ply/Compiler-Old.ipynb".
The first task is:
When an identifier is read, we first have to check whether the identifier is one of our keywords.  If so, we assign the corresponding token type that is stored in the dictionary Keywords.  Otherwise, the token type is set to ID.
Can you write Python code for it?

def t_ID(t):
    r'[a-zA-Z][a-zA-Z0-9_]*'
    t.type = Keywords.get(t.value, 'ID')
    return t


I am working on the file "Ply/Compiler.ipynb".
The first task is:
A Simple Compiler for a Fragment of C
This file shows how a simple compiler for a fragment of the programming language C can be implemented using Ply.
Specification of the Scanner
Can you write Python code for it?

import ply.lex as lex

tokens = [ 'NUMBER', 'ID', 'EQ', 'NE', 'LE', 'GE', 'AND', 'OR',
           'INT', 'IF', 'ELSE', 'WHILE', 'RETURN'
         ]


I am working on the file "Ply/Compiler.ipynb".
The first task is:
A Simple Compiler for a Fragment of C
This file shows how a simple compiler for a fragment of the programming language C can be implemented using Ply.
Specification of the Scanner
The scanner that we implement here is similar to the scanner that we had used for our symbolic calculator.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
As we have used precedence declarations to resolve all shift/reduce conflicts, the action table contains no conflict.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
Below is a test for the function compile_bool.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
Finally, we run the generated byte code.
Can you write Python code for it?

!java Primes

!del *.jas *.class

!rm *.jas *.class

!dir

!ls -al


I am working on the file "Ply/Compiler.ipynb".
The first task is:
Given a token, the function find_colum returns the column where token starts.
This is possible, because token.lexer.lexdata stores the string that is given to the scanner and token.lexpos is the number of characters that precede token.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
Given a token, the function find_colum returns the column where token starts.  This is possible, because every token contains a reference to the current lexer as token.lexer and this lexer in turn stores the string that is given to it via the reference lexer.lexdata.  Furthermore, token.lexpos is the number of characters that precede token.
Can you write Python code for it?

def find_column(token):
    program    = token.lexer.lexdata  # the complete string presented to the scanner
    line_start = program.rfind('\n', 0, token.lexpos) + 1
    return (token.lexpos - line_start) + 1


I am working on the file "Ply/Compiler.ipynb".
The first task is:
In the grammar rule
$$ \texttt{stmnt} \rightarrow \texttt{'if'}\;\texttt{'('}\; \texttt{bool_expr}\; \texttt{')'}\; \texttt{stmnt}$$
the rightmost token is ')'.  However, this token does not have a precedence.  Therefore, the grammar rule
does not have a precedence either.  Hence, we manually assign the precedence of the token IF to this rule via
the keyword %prec.  This way, the shift/reduce conflict resulting from the dangling-else ambiguity is resolved.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
Next, we generate Java byte code using
jasmin.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
Operators consisting of a single character do not need an associated token type.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
Our version of C allows both single line comments and multi line comments.
- The regular expression //[^\n]* recognizes single line comments.
  A single line comment starts with // and extends to the end of the line.
- The regular expression \/\*([^*]|\*+[^*/])*\*+\ recognizes multi line comments.
  Multi line comments start with the string /* and end with the string */.
  Between these strings, the string */ must not occur.
  * \/\* matches the opening /*.
  * [^*]|\*+[^*/] matches any character that is different from the character * as
    well as non-empty sequences of *s that are not followed by a /.
  * \*+/ matches any non-empty sequence of *s that is followed by a /.
  The expression needs to be this complicated in order to match multi line comments of the
  following form:
  /*** abc *** xyz ***/
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
Our version of C allows both single-line comments and multi-line comments.
- The regular expression //[^\n]* recognizes single-line comments.
  A single line comment starts with // and extends to the end of the line.
- The regular expression \/\*([^*]|\*+[^*/])*\*+\/ recognizes multi-line comments.
  Multi-line comments start with the string /* and end with the string */.
  Between these strings, the string */ must not occur.
  * \/\* matches the opening /*.
  * [^*]|\*+[^*/] matches any character that is different from the character * as
    well as non-empty sequences of *s that are not followed by a /.
  * \*+/ matches any non-empty sequence of *s that is followed by a /.
  The expression needs to be this complicated in order to match multi-line comments of the
  following form:
  /*** abc *** xyz ***/
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
Setting the optional argument write_tables to False is required to prevent an obscure bug where the parser generator tries to read an empty parse table.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
Specification of the Parser
We will use the following grammar to specify the language that our compiler can translate:
```
program
    : function
    | function program
function
    : INT ID '(' param_list ')' '{' decl_list stmnt_list '}'
param_list
    : / epsilon /
    | INT ID
    | INT ID ',' ne_param_list
ne_param_list
    : INT ID
    | INT ID ',' ne_param_list
decl_list
    : / epsilon /
    | INT ID ';' decl_list
stmnt_list
    : / epsilon /
    | stmnt stmnt_list
stmnt
    : IF '(' bool_expr ')' stmnt
    | IF '(' bool_expr ')' stmnt ELSE stmnt
    | WHILE '(' bool_expr ')' stmnt
    | '{' stmnt_list '}'
    | ID '=' expr ';'
    | RETURN expr ';'
    | expr ';'
bool_expr
    : bool_expr '||' bool_expr
    | bool_expr '&&' bool_expr
    | '!' bool_expr
    | '(' bool_expr ')'
    | expr '==' expr
    | expr '!=' expr
    | expr '<=' expr
    | expr '>=' expr
    | expr '<'  expr
    | expr '>'  expr
expr: expr '+'  expr
    | expr '-' expr
    | expr '*' expr
    | expr '/' expr
    | expr '%' expr
    | '(' expr ')'
    | NUMBER
    | ID
    | ID '(' expr_list ')'
expr_list
    : / epsilon /
    | expr
    | expr ',' ne_expr_list
ne_expr_list
    : expr
    | expr ',' ne_expr_list
```
We will use precedence declarations to resolve the ambiguity that is inherent in this grammar.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
Syntactically, newline characters are ignored. However, we still need to keep track of them in order to know the current line number, which is used for error messages.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
Syntactically, newline characters are ignored. However, we still need to keep track of them in order to know which line we are in.  This information is needed later for error messages.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The file Primes.c is a simple C program that computes the prime numbers in a naive way.
Can you write Python code for it?

!type Primes.c

!cat Primes.c

compile_program('Primes.c')

!dir

!ls -l

!type Primes.jas

!cat Primes.jas


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The following is a test of the function compile_expr.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The function indent is used to indent the generated assembler commands by preceding them with 8 space characters.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The function parse takes a file_name as ist sole argument.  The file is read and parsed.
The resulting parse tree is visualized using graphviz.  It is important to reset the
attribute lineno of the scanner, for otherwise error messages will not have the correct line numbers.
Can you write Python code for it?


def parse(file_name):
    lexer.lineno = 1
    with open(file_name, 'r') as handle:
        program = handle.read()
    ast = yacc.parse(program)
    print(ast)
    return tuple2dot(ast)

!type Examples\MySum.c

!cat -n Examples/MySum.c

parse('Examples/MySum.c')


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The function t_error is called for any token t that can not be scanned by the lexer.  In this case, t.value[0] is the first character that can not be recognized by the scanner.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The input to compile_program(file_name) is the name of a C file.  This file assumed to have the ending .c.
This file is compiled and the resulting assembler file is written into a file with the same name but the ending .jas.
Can you write Python code for it?

def compile_program(file_name):
    directory = os.path.dirname(file_name)
    base      = os.path.basename(file_name)
    base      = base[:-2]
    outfile   = os.path.join(directory, base + '.jas')
    with open(file_name, 'r') as handle:
        program = handle.read()
    lexer.lineno = 1
    ast = yacc.parse(program)
    _, *fct_lst = ast
    CmdLst = []
    for fct in fct_lst:
        L, _ = compile_fct(fct, base)
        CmdLst += L + ['\n']
    with open(outfile, 'w') as handle:
        handle.write('.class public ' + base + '\n');
        handle.write('.super java/lang/Object\n\n');
        handle.write('.method public <init>()V\n');
        handle.write('    aload 0\n');
        handle.write('    invokenonvirtual java/lang/Object/<init>()V\n');
        handle.write('    return\n');
        handle.write('.end method\n\n');
        for cmd in CmdLst:
            handle.write(cmd + '\n')

%cd Examples


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The keywords 'int', 'if', 'else', 'while', 'return' have to be dealt with separately as they are syntactical identical to identifiers.  The dictionary Keywords shown below maps every keyword to its token type.
Can you write Python code for it?

Keywords = { 'int'   : 'INT',
             'if'    : 'IF',
             'else'  : 'ELSE',
             'while' : 'WHILE',
             'return': 'RETURN'
           }


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The method compile_bool(expr, st, class_name) takes three arguments:
- expr is an abstract syntax tree that represents a Boolean expression.
  This abstract syntax tree is in turn represented as a nested tuple.
- st is short for symbol table.  This is a dictionary that maps variable
  names to natural numbers.  Given a variable x, the number st[x] specifies
  the location where the variable x is stored on the stack with respect to the
  local stack frame.
- class_name is the name of the class that is to be generated.
The function returns a pair of the form (cmds, size).
- cmds is a list of assembler commands,
- size is the maximum size of the stack that is needed.
Can you write Python code for it?

def compile_bool(expr, st, class_name):
    match expr:
        case ('==' | '!=' | '<=' | '>=' | '<' | '>') as op, lhs, rhs:
            OpToCmd = { '==': 'if_icmpeq',
                        '!=': 'if_icmpne',
                        '<=': 'if_icmple',
                        '>=': 'if_icmpge',
                        '<' : 'if_icmplt',
                        '>' : 'if_icmpgt'
                      }
            L1, sz1    = compile_expr(lhs, st, class_name)
            L2, sz2    = compile_expr(rhs, st, class_name)
            true_label = new_label()
            next_label = new_label()
            CmdLst     = L1 + L2
            cmd        = OpToCmd[op]
            CmdLst    += [indent(cmd + ' ' + true_label)]
            CmdLst    += [indent('bipush 0')]
            CmdLst    += [indent('goto ' + next_label)]
            CmdLst    += [' ' * 4 + true_label + ':']
            CmdLst    += [indent('bipush 1')]
            CmdLst    += [' ' * 4 + next_label + ':']
            return CmdLst, max(sz1, 1 + sz2)
        case ('&&' | '||') as op, lhs, rhs:
            OpToCmd      = { '&&': 'iand', '||': 'ior' }
            L1, sz1      = compile_bool(lhs, st, class_name)
            L2, sz2      = compile_bool(rhs, st, class_name)
            cmd          = OpToCmd[op]
            CmdLst       = L1 + L2 + [indent(cmd)]
            return CmdLst, max(sz1, 1 + sz2)
        case '!', arg:
            L, sz  = compile_expr(arg, st, class_name)
            CmdLst = [indent('bipush 1')] + L + [indent('isub')]
            return CmdLst, sz + 1
        case _:
            assert False, f'Error in compile_bool({expr}, {st}, {class_name})'


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The next assignment is necessary to make the lexer think that the code given above is part of some file.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The notebook AST-2-Dot.ipynb provides the function tuple2dot.  This function can be used to visualize the abstract syntax tree that is generated by the function yacc.parse.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The start variable of our grammar is program.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The token Number specifies a natural number.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
The variable label_counter is a global counter that is used to create unique label names.
Every call of new_label creates a new, unique label.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
When an identifier is read, we first have to check whether the identifier is one of our keywords.  If so, we assign the corresponding token type that is stored in the dictionary Keywords.  Otherwise, the token type is set to ID.
Can you write Python code for it?


I am working on the file "Ply/Compiler.ipynb".
The first task is:
White space, i.e. space characters, tabulators, and carriage returns are ignored.
Can you write Python code for it?


I am working on the file "Ply/Conflicts-Resolved.ipynb".
The first task is:
As there are no warnings all conflicts have been resolved using the precedence declarations.
Let's look at the action table that is generated.
Can you write Python code for it?

!type parser.out

!cat parser.out

%run ../ANTLR4-Python/AST-2-Dot.ipynb


I am working on the file "Ply/Conflicts-Resolved.ipynb".
The first task is:
Let's look at the action table that is generated.  Conflicts are always resolved in favour of shifting.
Can you write Python code for it?


I am working on the file "Ply/Conflicts-Resolved.ipynb".
The first task is:
Resolving Conflicts Using Precedence Declarations
This file shows how shift/reduce and reduce/reduce conflicts can be resolved using operator precedence declarations.
The following grammar is ambiguous because it does not specify the precedence of the arithmetical operators:
expr : expr '+' expr
         | expr '-' expr
         | expr '*' expr
         | expr '/' expr
         | expr '^' expr
         | '(' expr ')'
         | NUMBER
         ;
We will see how the use of precedence declarations can be used to resolve shift/reduce-conflicts.
Specification of the Scanner
We implement a minimal scanner for arithmetic expressions.
Can you write Python code for it?

import ply.lex as lex

tokens = [ 'NUMBER' ]

def t_NUMBER(t):
    r'0|[1-9][0-9]*'
    t.value = int(t.value)
    return t

literals = ['+', '-', '*', '/', '^', '(', ')']

t_ignore  = ' \t'

def t_newline(t):
    r'\n+'
    t.lexer.lineno += t.value.count('\n')

def t_error(t):
    print(f"Illegal character '{t.value[0]}'")
    t.lexer.skip(1)

__file__ = 'main'

lexer = lex.lex()


I am working on the file "Ply/Conflicts-Resolved.ipynb".
The first task is:
Setting the optional argument write_tables to False <B style="color:red">is required</B> to prevent an obscure bug where the parser generator tries  to read an empty parse table.
Can you write Python code for it?

parser = yacc.yacc(write_tables=False, debug=True)


I am working on the file "Ply/Conflicts-Resolved.ipynb".
The first task is:
Specification of the Parser
Can you write Python code for it?


I am working on the file "Ply/Conflicts-Resolved.ipynb".
The first task is:
The following operator precedence declarations declare that the operators '+'and '-' have a lower precedence than the operators '*'and '/'.  The operator '^' has the highest precedence.  Furthermore, the declarations specify that the operators  '+', '-', '*', and '/' are left associative, while the operator '^' is declared as right associative using the keyword right.
Operators can also be defined as being non-associative using the keyword nonassoc.
Can you write Python code for it?

precedence = (
    ('left', '+', '-') ,  # precedence 1
    ('left', '*', '/'),   # precedence 2
    ('right', '^')        # precedence 3
)

def p_expr_plus(p):
    "expr : expr '+' expr"
    p[0] = ('+', p[1], p[3])

def p_expr_minus(p):
    "expr : expr '-' expr"
    p[0] = ('-', p[1], p[3])

def p_expr_mult(p):
    "expr : expr '*' expr"
    p[0] = ('*', p[1], p[3])

def p_expr_div(p):
    "expr : expr '/' expr"
    p[0] = ('/', p[1], p[3])

def p_expr_power(p):
    "expr : expr '^' expr"
    p[0] = ('^', p[1], p[3])

def p_expr_paren(p):
    "expr : '(' expr ')'"
    p[0] = p[2]

def p_expr_NUMBER(p):
    "expr : NUMBER"
    p[0] = p[1]

def p_error(p):
    if p:
        print(f"Syntax error at character number {p.lexer.lexpos} at token '{p.value}' in line {p.lexer.lineno}.")
    else:
        print('Syntax error at end of input.')


I am working on the file "Ply/Conflicts-Resolved.ipynb".
The first task is:
The following operator precedence declarations declare that the operators +and - have a lower precedence than the operators *and /.  Furthermore, they specify that all these operators are left associative.  Operators can also be declared as right associative using the keyword right or as non-associative using the keyword nonassoc.
Can you write Python code for it?


I am working on the file "Ply/Conflicts-Resolved.ipynb".
The first task is:
The function test(s) takes a string s as its argument an tries to parse this string.  If all goes well, an abstract syntax tree is returned.
If the string can't be parsed,  an error message is printed by the parser.
Can you write Python code for it?

def test(s):
    t = yacc.parse(s)
    d = tuple2dot(t)
    display(d)
    return t

test('2^3*4+5')

test('1+2*3^4')

test('1 + 2 * -3^4')


I am working on the file "Ply/Conflicts-Resolved.ipynb".
The first task is:
The start variable of our grammar is expr, but we don't have to specify that.  The default
start variable is the first variable that is defined.
Can you write Python code for it?


I am working on the file "Ply/Conflicts-Resolved.ipynb".
The first task is:
We define p_error to prevent a warning.
Can you write Python code for it?


I am working on the file "Ply/Conflicts.ipynb".
The first task is:
Dealing with Conflicts
This file shows how Ply deals with shift/reduce and reduce/reduce conflicts.
The following grammar is ambiguous because it does not specify the precedence of the arithmetical operators:
expr : expr '+' expr
         | expr '-' expr
         | expr '*' expr
         | expr '/' expr
         | '(' expr ')'
         | NUMBER
         ;
Specification of the Scanner
We implement a minimal scanner for arithmetic expressions.
Can you write Python code for it?


I am working on the file "Ply/Conflicts.ipynb".
The first task is:
Let's look at the action table that is generated.  Conflicts are always resolved in favour of shifting.
Can you write Python code for it?


I am working on the file "Ply/Conflicts.ipynb".
The first task is:
Setting the optional argument write_tables to False <B style="color:red">is required</B> to prevent an obscure bug where the parser generator tries  to read an empty parse table.
Can you write Python code for it?


I am working on the file "Ply/Conflicts.ipynb".
The first task is:
Specification of the Parser
Can you write Python code for it?


I am working on the file "Ply/Conflicts.ipynb".
The first task is:
The function test(s) takes a string s as its argument an tries to parse this string.  If all goes well, an abstract syntax tree is returned.
If the string can't be parsed,  an error message is printed by the parser.
Can you write Python code for it?

def test(s):
    t = yacc.parse(s)
    d = tuple2dot(t)
    display(d)
    return t


I am working on the file "Ply/Conflicts.ipynb".
The first task is:
The start variable of our grammar is expr, but we don't have to specify that.  The default
start variable is the first variable that is defined.
Can you write Python code for it?


I am working on the file "Ply/Conflicts.ipynb".
The first task is:
The start variable of our grammar is expr, but we dont't have to specify that.  The default
start variable is the first vvariable that is defined.
Can you write Python code for it?


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:

We have now defined all of the tokens.
Note that the scanner tries the regular expressions in the same order that we
have used to define these tokens.

Ignoring Characters
The string t_ignore specifies those characters that should be ignored.  Note that this string is not interpreted as a regular expression.  It is just a string of single characters.  These characters are allowed to occur as part of other tokens, but when they occur on their own and would otherwise generate a scanning error, they are silently discarded instead of triggering an error.
In this example we ignore hyphens, blanks, and tabs.
Can you write Python code for it?


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
Auxiliary Functions
The function mark(lexer) takes the object lexer as its argument.  This object is supposed to be a scanner object that is generated by the expression lex.lex().  This object provides two member variables:
- lexer.sum_points is the number of points achieved by the student whose mark is to be computed.
- lexer.max_points is the number of points that need to be achieved in order to get the best mark
  of $1.0$.
It is assumed that the relation between the mark of an exam and the number of points achieved in this exam is mostly linear and that a student who has achieved $50\%$ of lexer.max_points points will get the mark $4.0$, while a student who has achieved  $100\%$ of lexer.max_points points will be marked as $1.0$.
However, the worst mark is $5.0$.  Therefore, if the mark would fall below that line, the min function below assures that it is lifted up to $5.0$.
Can you write Python code for it?

def mark(lexer):
    return round(min(5.0, 7 - 6 * lexer.sum_points / lexer.max_points), 2)


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
Evaluating an Exam Using Ply
This notebook shows how we can use the package ply
to implement a scanner.  Our goal is to implement a program that can be used to evaluate the results of an exam.  Assume the result of an exam is stored in the string data that is defined below:
Can you write Python code for it?

data = '''Class: Algorithms and Complexity
          Group: TIT09AID
          MaxPoints = 60

          Exercise:      1. 2. 3. 4. 5. 6.
          Jim Smith:     9 12 10  6  6  0
          John Slow:     4  4  2  0  -  -
          Susi Sorglos:  9 12 12  9  9  6
       '''


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
Finally, we can run the scanner.
Can you write Python code for it?


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
In order to scan the data that we provided in the last line, the function scan iterates
over all tokens generated by our scanner.
Can you write Python code for it?

def scan(lexer):
    for t in lexer:
        pass


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
Lets test this function.
Can you write Python code for it?

for points in range(0, 60+1, 5):
    print(f'mark(60, {points}) = {mark(60, points)}')


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
Next, we feed an input string into the generated scanner.
Can you write Python code for it?


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
The Token IGNORE
The token IGNORE matches a line that contains only whitespace.  In order to keep track of line numbers we have to increment lexer.lineno.  However, we do not return a token at the end of the function.  Hence, if the input contains an empty line, this line is silently discarded.
Can you write Python code for it?

def t_IGNORE(t):
    r'^[ \t]*\n'
    t.lexer.lineno += 1


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
The Token LINEBREAK
The token LINEBREAK matches a single newline character \n.  If a student name is
currently defined, then we output the result for this student.  Note that we set lexer.name back to the empty string once we have processed the student.
This allows for empty lines between different students.
Can you write Python code for it?

def t_LINEBREAK(t):
    r'\n'
    t.lexer.lineno += 1
    if t.lexer.name != '':
        n = t.lexer.name
        m = t.lexer.max_points
        p = t.lexer.sum_points
        print(f'{n} has {p} points and achieved the mark {mark(m, p)}.')
        t.lexer.name = ''


def t_LINEBREAK(t):
    r'\n'
    t.lexer.lineno += 1
    if t.lexer.name != '':
        n = t.lexer.name
        p = t.lexer.sum_points
        l = t.lexer
        print(f'{n} has {p} points and achieved the mark {mark(l)}.')
        t.lexer.name = ''


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
The Token MAXDEF
The token MAXDEF matches a substring of the form MaxPoints = 60.  Note that the regular expression defining the semantics of this token uses the expression \s* to match the white space before and after the character =.  This is necessary because ply.lex uses verbose regular expressions that can contain whitespace for formatting.  Hence a blank character "" inside a regular expression is silently discarded.
After defining the regular expression, the function t_MAXDEF has some action code that is used to extract the maximal number of points from the token value and store this number in the variable t.lexer.max_points.
t.value is the string that is matched by the regular expression.
We extract the maximum number of points using conventional Python regular expressions.  Furthermore, we initialize the student name,
which is stored in t.lexer.name,  to the empty string.
Can you write Python code for it?

def t_MAXDEF(t):
    r'MaxPoints\s*=\s*[1-9][0-9]*'
    t.lexer.max_points = int(re.findall(r'[0-9]+', t.value)[0])
    t.lexer.name       = ''


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
The Token MAXDEF
The token MAXDEF matches a substring of the form MaxPoints = 60.  Note that the regular expression defining the semantics of this token uses the expression \s* to match the white space before and after the character =.  This is necessary because ply.lex uses verbose regular expressions that can contain whitespace for formatting.  Hence a blank character "" inside a regular expression is silently discarded.
After defining the regular expression, the function t_MAXDEF has some action code that is used to extract the maximal number of points from the token value and store this number in the variable t.lexer.name.
t.value is the string that is matched by the regular expression.
We extract the maximum number of points using conventional Python regular expressions.  Furthermore, we initialize the student name to the empty string.
Can you write Python code for it?


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
The Token NUMBER
The token NUMBER matches a natural number.  We have to convert the value, which is initially a string of digits, into an integer.  Furthermore, this value is then added to the number of points the current student has achieved in previous exercises.
Can you write Python code for it?

def t_NUMBER(t):
    r'0|[1-9][0-9]*'
    t.value = int(t.value)
    t.lexer.sum_points += t.value


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
The Token Name
The token NAME matches the name of a student followed by a colon.  In general, a student name can be any sequence of letters that contain optional hyphens and blanks.  Note that it is not necessary to use \s inside of a character range.
The action code has to reset the variable sum_points that is stored in lexer.sum_pointsto 0.
Can you write Python code for it?

def t_NAME(t):
    r'[a-zA-Z- ]+:'
    t.lexer.name = t.value[:-1] # cut of colon
    t.lexer.sum_points = 0


I am working on the file "Ply/Exam-Evaluation.ipynb".
The first task is:
Token Declarations
We begin by declaring the list of tokens.  Note that the variable tokens is a keyword of ply to define the names of the token classes.  In this case, we have declared six different tokens.
The definitions of these tokens are given later.
- HEADER will match the first two lines of the string data as well as the fifth line that begins with
  the string Exercise:.
- MAXDEF is a token that will match the line MaxPoints = 60.
- NAME is a token that will match the name of a student.
- NUMBER is a token that will match a natural number.
- IGNORE is a token that will match an empty line.  For example, the fourth line in data is empty.
- LINEBREAK is a token that will match the newline character \n at the end of a line.
Can you write Python code for it?

tokens = [ 'HEADER',
           'MAXDEF',
           'NAME',
           'NUMBER',
           'IGNORE',
           'LINEBREAK'
         ]

print(data)


tokens = [ 'HEADER',
           'MAXDEF',
           'NAME',
           'NUMBER',
           'IGNORE',
           'LINEBREAK'
         ]