I am working on the file "ABoVE/Vegetaion_File_from_GEE.ipynb".
The first task is:
LULC Classes

Water
Evergreen_Needleleaf_Forest
Evergreen_Broadleaf_Forest
Deciduous_Needleleaf_Forest
Deciduous_Broadleaf_Forest
Mixed_Forest
Closed_Shrublands
Open_Shrublands
Woody_Savannas
Savannas
Grasslands
Permanent_Wetlands
Croplands
Urban_and_Built-up
Cropland_Natural_Vegetation_Mosaic
Snow_and_Ice
Barren_Or_Sparsely_Vegetated
Unclassified
No_Data

Get the LULC classes for years 2001-2012
Can you write Python code for it?

#lat, lon = x, y

def make_LULC(lat, lon):
    rectangle = ee.Geometry.Rectangle([lon-0.125, lat-0.125, lon+0.125, lat+0.125]) #ll_lon, ll_lat, lon2, lat2
    lists = []
    for year in range(2001, 2013, 1):
        image = ee.Image("Oxford/MAP/IGBP_Fractional_Landcover_5km_Annual/"+str(year))#.filterDate(startTime, endTime)
        # Change the scale parameter below to see the effects
        meanDictionary = image.reduceRegion(reducer = ee.Reducer.mean(), geometry=rectangle,  scale = 300, maxPixels = 1e10)
        m = meanDictionary.getInfo()
        lists.append(m)
        sorted_m = sorted(m.items(), key=operator.itemgetter(1))
    #print(meanDictionary.getInfo())
    #print(sorted_m)
    # Plot LULC classes over time
    R = lists
    S = {k:[ R[j][k] for j in range(len(R)) ] for k in R[0].keys()}
    SigClass = {}
    for s in S:
        if sum(S[s]) > 10 and s != "Overall_Class":
            #print s
            SigClass[s.replace("_", " ")] = S[s]
    #print SigClass
    plt.figure(dpi=resolution)
    for name, values in SigClass.items():
        plt.plot(range(2001,2013), values, linewidth=3.0, label=name)
    plt.xlabel("Years")
    plt.ylabel("% of LULC class in grid cell "); plt.ylim(0,100)
    plt.title("LULC classes at {}, {}".format(lat,lon))
    leg = plt.legend(bbox_to_anchor=(0.1,0.35), fontsize=12, loc=(0,0))
    leg.get_frame().set_alpha(0.3)
    lulc_png = "LULC_{}_{}.png".format(lon,lat)
    plt.savefig(lulc_png)
    plt.show()
    plt.clf()
    return lulc_png
make_LULC(lat, lon)


I am working on the file "ABoVE/Vegetaion_File_from_GEE.ipynb".
The first task is:
Leaf Area Index (2003-2016)
Leaf area index
Definition: one-sided green leaf area per unit ground area in broadleaf canopies; one-half the total needle surface area per unit ground area in coniferous canopies (dimensionless). Scale=0.1.
The MCD15A3H version 6 MODIS Level 4, Combined Fraction of Photosynthetically Active Radiation (FPAR), and Leaf Area Index (LAI) product is a 4-day composite data set with 500 meter pixel size.  The algorithm chooses the 'best' pixel available from all the acquisitions of both MODIS sensors located on NASA's Terra and Aqua satellites from within the 4-day period.
Earth Engine dataset
MCD15A3H.006 Terra+Aqua Leaf Area Index/FPAR 4-Day L4 Global 500 m
Can you write Python code for it?

def make_LAI(lat, lon):
    """
    Given a point, makes a plot of LAI for 1/4th degree box around the point.
    """
    rectangle = ee.Geometry.Rectangle([lon-0.125, lat-0.125, lon+0.125, lat+0.125])
    lais = []
    for year in range(2003,2017):
        lai = []
        for month in range(1,13):
            startDate = str(year)+'-' + str(month)+'-01'
            if month == 2:
                endDate = str(year)+'-' + str(month)+'-28'
            else:
                endDate = str(year)+'-' + str(month)+'-30'
            imageCollection = ee.ImageCollection("MODIS/006/MCD15A3H").filterDate(startDate, endDate)
            #ee.mapclient.centerMap(-93.7848, 30.3252, 11)
            image = ee.Image(imageCollection.mean())
            data = image.reduceRegion(reducer = ee.Reducer.mean(), geometry=rectangle,  scale = 300, maxPixels = 1e10)
            lai.append(data.getInfo()["Lai"])
        lais.append(lai)
    plt.figure(dpi=resolution)
    plt.title("Leaf Area Index for {}, {}, (2003-2016)".format(lat,lon))
    for i, year in enumerate(range(2003,2017)):
        plt.scatter(np.arange(1,13,1), lais[i],label=str(year))

    maxValue = max([max(x) for x in lais])
    avg = [float(sum(col))/len(col) for col in zip(*lais)]
    plt.plot(np.arange(1,13,1), avg, linewidth=3.0, label="Mean")
    plt.ylabel("Leaf Area Index"); plt.ylim(0,maxValue)
    plt.xlabel("Months")
    leg = plt.legend(loc=1, fontsize=7)
    #leg = plt.legend(loc=2,fancybox=True, fontsize=14)
    leg.get_frame().set_alpha(0.3)
    png = 'LAI_{}_{}.png'.format(lon,lat)
    plt.savefig(png)
    plt.show()
    plt.clf()
    return png
make_LAI(lat, lon)

import xarray
import numpy as np
data = xarray.open_dataset("/mnt/nfs/home/solomon/Data/CFSR_SK/data_201001.nc")
lons = [x-360 for x in list(np.array(data.lon))]
lats = list(np.array(data.lat))
lats, lons

llats = []; llons = []
for lat in set(lats):
    for lon in set(lons):
        llats.append(lat)
        llons.append(lon)

len(llats), len(llons)
llats, llons

path =...


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
A peek at the topmost data items.
Can you write Python code for it?


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
All the pre-1990 publications are:
Can you write Python code for it?

legacybib[legacybib.year <= 1990][["year", "title"]].sort_values("year")


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
And the keyword lists for the above
Can you write Python code for it?

[keywordlist for keywordlist in legacybib[legacybib.keywords.map(lambda kws: len(kws)) > 15].keywords]


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
Communities
Community detection with the Louvain algorithm, explained in Blondel, Guillaume, Lambiotte, Lefebvre: Fast unfolding of communities in large networks (2008). For weighted networks, modularity of a partition is $Q = \frac{1}{2m}\sum_{i, j} \Big[A_{ij} - \frac{k_i k_j}{2m}\Big] \delta(c_i, c_j)$, where $A_{ij}$ is the weight matrix, $c_i$ the community of $i$, and $\delta$-function is 1 if $u = v$, and 0 otherwise (erm what?) and $m = \frac{1}{2}\sum_{i,j}A_{ij}$.
Can you write Python code for it?

def plotCommunities(graph):
    """Plot community information from a graph.

    Basically just copied from http://perso.crans.org/aynaud/communities/index.html
    at this point, while in development
    """
    # zoom in on something, for dev. purposes
    graph = graph.subgraph(graph.neighbors('legacy software'))
    # graph = [c for c in nx.connected_component_subgraphs(graph)][0]
    graph = max(nx.connected_component_subgraphs(graph), key=len) # I love you Python
    partition = community.best_partition(graph)
    size = float(len(set(partition.values())))
    pos = nx.spring_layout(graph)
    count = 0
    for com in set(partition.values()):
        count = count + 1
        list_nodes = [nodes for nodes in partition.keys() if partition[nodes] == com]
        plt.axis('off')
        nx.draw_networkx_nodes(graph, pos, list_nodes, node_size = 40, node_color = str(count/size), alpha=0.4)
        nx.draw_networkx_labels(graph, pos, font_size = 9)

    nx.draw_networkx_edges(graph, pos, alpha=0.1)
    plt.show()

plotCommunities(keywordg)


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
Complementary data
To contextualize the legacy search results, get the number of total publications in ACM per year.
These were semimanually extracted from the ACM DL search results listing DOM, with the following Javascript
javascript
acmYearly = {};
theChartData.labels.forEach(function(y) {acmYearly[y] = theChartData.datasets[0].data[theChartData.labels.indexOf(y)]});
console.log(acmYearly);
Can you write Python code for it?


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
Data exploration
Fraction of published items per year which ACM identifies as relevant for legacy search.
Histogram of publication years
Can you write Python code for it?

legacybib.year.hist(bins=legacybib.year.max() - legacybib.year.min(), figsize=(10,2))


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
Data loading, sanitization and massage
Search "legacy" at ACM Digital Library. Just a simple search, to which the web interface gives 1541 results in mid November 2016. The number of items in the library is ~460000.
A CSV is downloaded from ACM DL, in default sorting order of the library's own idea of relevance... whatever that means for them. BibTeX is also available.
Can you write Python code for it?


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
Data overview
Let's check how many percent all the 1541 search results given by the website search list. Would be great if this was 100%.
Can you write Python code for it?


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
Did something happen around 1990s, as the fraction of publications related to legacy started increasing? Let's plot a global linear regression model, as well as separate linear regression models before and after 1990.
Can you write Python code for it?

pre1990range = np.arange(legacybib.year.min(), 1991)
post1990range = np.arange(1990, legacybib.year.max())

# Linear regression models
# note the use of np.polyfit
propLm = np.polyfit(pd.Series(legacybib.groupby(legacybib.year).year.count() / acmPerYear).dropna().index, pd.Series(legacybib.groupby(legacybib.year).year.count() / acmPerYear).dropna(), 1)
pre1990 = np.polyfit(pd.Series(legacybib.groupby(legacybib.year).year.count() / acmPerYear)[pre1990range].dropna().index, pd.Series(legacybib.groupby(legacybib.year).year.count() / acmPerYear)[pre1990range].dropna(), 1)
post1990 = np.polyfit(pd.Series(legacybib.groupby(legacybib.year).year.count() / acmPerYear)[post1990range].dropna().index, pd.Series(legacybib.groupby(legacybib.year).year.count() / acmPerYear)[post1990range].dropna(), 1)

# Plot the fractions of legacy vs. all publications, the models, and a legend
plt.plot(pd.Series(legacybib.groupby(legacybib.year).year.count() / acmPerYear), 'o')
plt.plot(np.arange(legacybib.year.min(), legacybib.year.max()), np.poly1d(propLm)(np.arange(legacybib.year.min(), legacybib.year.max())), label="global lm")
plt.plot(pre1990range, np.poly1d(pre1990)(pre1990range), linestyle="dashed", label="pre 1990 lm")
plt.plot(post1990range, np.poly1d(post1990)(post1990range), linestyle="dashed", label="post 1990 lm")
plt.title("Fraction of legacy related publications against ACM")
plt.legend(loc="best")


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
Does the id field uniquely identify items on the search list? If so, using it as index could be a good idea.
Can you write Python code for it?

assert 0, sum(legacybib.id.duplicated())

legacybib[legacybib.id.duplicated(keep=False)].head(2 * 2)


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
How many keywords do each item have?
Can you write Python code for it?

legacybib.keywords.map(lambda kws: len(kws)).describe()

plt.title("Histogram of numbers of keywords per item")
plt.hist(legacybib.keywords.map(lambda kws: len(kws)), bins=max(legacybib.keywords.map(lambda kws: len(kws))) - 1)


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
Let's plot the top hub out.
Can you write Python code for it?

def plotNeighborhood(graph, ego, color = "green", includeEgo = False):
    from math import sqrt
    """
    Plot neighbourhood of keyword in graph, after possibly removing the ego.

    graph : networkx.Graph-like graph
        The graph to get the neighbourhood from
    ego : node in graph
        The node whose neighbourhood to plot
    color : string
        Name of the color to use for plotting
    includeEgo : bool
        Include the ego node

    The function defaults to removing the ego node, because by definition
    it is connected to each of the nodes in the subgraph. With the ego
    removed, the result basically tells how the neighbours are connected
    with one another.
    """
    plt.rcParams["figure.figsize"] = (10, 10)
    subgraph = nx.Graph()
    if includeEgo:
        subgraph = graph.subgraph(graph.neighbors(ego) + [ego])
    else:
        subgraph = graph.subgraph(graph.neighbors(ego))
    plt.title("Neighbourhood of " + ego + " (" + str(len(subgraph)) + ")")
    plt.axis('off')
    pos = nx.spring_layout(subgraph, k = 1/sqrt(len(subgraph) * 2))
    nx.draw_networkx(subgraph,
                     pos = pos,
                     font_size = 9,
                     node_color = color,
                     alpha = 0.8,
                     edge_color = "light" + color)
    plt.show()

plotNeighborhood(keywordg, "legacy systems")

plotNeighborhood(keywordg, "legacy software")


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
Massage the keywords to be lists. Note that str.split(',') returns a [''], therefore the little if filter in there.
Can you write Python code for it?

legacybib.keywords.fillna('', inplace=True)
legacybib.keywords = legacybib.keywords.map(lambda l: [k.lower().strip() for k in l.split(',') if k])


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
Similar overall shape, which isn't a surprise. Overlay, with arbitrary scaling of 300.
Can you write Python code for it?

#plt.hist(legacybib.year.dropna(), label="Year histogram")
plt.plot(legacybib.year.groupby(legacybib.year).count(), label='legacy publication')
plt.plot(acmPerYear * 0.003, label="total publications * 0.003")
plt.legend()
plt.legend(loc='best')


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
So there is one dominant component, and 150 small ones. It's best to explore them interactively with Gephi.
Can you write Python code for it?


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
Statistical validation of the above would be good, of course, to check against randomness.
A histogram of keywords
The keywords are interesting. All keywords in this dataset are already related to legacy one way or the other, since the data under inspection here is a subset of the total ACM Digital Library.
Keywords of course live a life of their own, and I guess there increase in number forever.
Which keywords are popular?
Can you write Python code for it?

# this could be a pandas.Series instead of dict
keywordhist = {}
for kws in legacybib.keywords:
    for k in kws:
        if k in keywordhist:
            keywordhist[k] = keywordhist[k] + 1
        else:
            keywordhist[k] = 1


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
The available data columns are
Can you write Python code for it?


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
What about the ACM Digital Library total, what does it's profile look like over time?
Can you write Python code for it?


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
With the above peek at the ID field, how many unique items did we receive in the download?
Can you write Python code for it?


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
and further those that occur in 3-10 items
Can you write Python code for it?

[(k, keywordhist[k]) for k in sorted(keywordhist, key=keywordhist.get, reverse=True) if keywordhist[k] < 10 and keywordhist[k] >= 3]


I am working on the file "ACM DL bibliometry of legacy.ipynb".
The first task is:
and only on one item
Can you write Python code for it?

len([k for k in keywordhist if keywordhist[k] == 1])


I am working on the file "ACSF-Dimer.ipynb".
The first task is:
The fitting may not be trivial since our database is small... the next instruction can be executed multiple times let the NN train more and hopefully improve.
Can you write Python code for it?

# change some training parameters
nn.set_params(solver='lbfgs', alpha=0.001, tol=1.0e-10, learning_rate='constant', learning_rate_init=0.01)
# do some training steps
nn.fit(acsf, energy);

# evaluate the training error
energyML = nn.predict(acsf)

print ("Mean Abs Error (training)  : ", (numpy.abs(energyML-energy)).mean())

# energy curve
plt.plot(dists, energy,'-.' )
plt.plot(dists, energyML,'o' )
plt.xlabel('Pair distance')
plt.ylabel('Energy')
plt.show()

# regression plot
plt.plot(energy,energyML,'o')
plt.plot([-2.3,-1.7],[-2.3,-1.7]) # perfect fit line
plt.xlabel('correct energy')
plt.ylabel('NN energy')
plt.show()


I am working on the file "ACSF-Dimer.ipynb".
The first task is:
Training
We create a neural network and train it on the ACSF database we just constructed.
Can you write Python code for it?

# setup the neural network
# the network uses tanh function on all hidden neurons

nn = MLPRegressor(hidden_layer_sizes=(5,), activation='tanh')


I am working on the file "ACSF.ipynb".
The first task is:
Computing the ACSFs - two-body terms
The simplest ACSF quantifies the amount of atoms within cutoff:<br>
$$G^0_i = \sum_j f_c\left(r_{ij}\right)$$
with
$$f_c(r) = \frac{1}{2} \left[\cos\left(\pi \frac{r}{r_c}\right) + 1 \right]$$
Can you write Python code for it?

# cutoff distance for all ACSFs
Rcut = 5.0

def fcut(r):
    return (math.cos(math.pi * r/Rcut)+1) * 0.5

# compute the ACSF G0 (sum of cutoff)
G0 = numpy.zeros(3)
for i in range(3): # compute one ACSF for each atom i
    for j in range(3): # loop over all OTHER atoms
        if i == j: continue
        r = numpy.linalg.norm(atoms[i]-atoms[j]) # compute distance i-j
        if r > Rcut: continue # skip if too far
        G0[i] += fcut(r)

print(G0)


I am working on the file "AD.ipynb".
The first task is:
First, let's find the derivatives of the first output (fzero) with respect to x using finite differencing.  We already know how to do this.
Can you write Python code for it?

# ------ finite difference --------
output, a, ap = function(x, params)  # we are ignoring the other outputs although we could easily get there derivatives as well

g_fd = np.zeros(n)  # initialize gradient vector for finite difference

for i in range(n):  # iterate across all vars

    # step size
    step = 1e-6*x[i]

    # take a step
    xplus = np.copy(x)
    xplus[i] += step

    output_plus, a, ap = function(xplus, params)

    g_fd[i] = (output_plus - output) / step


I am working on the file "AD.ipynb".
The first task is:
Let's see how we did.  We will compare our errors relative to complex step.  First finite differencing:
Can you write Python code for it?

from __future__ import print_function

print('error_fd =', (g_fd - g_cs)/g_cs)


I am working on the file "AD.ipynb".
The first task is:
Now let's compute exact derivatives using automatic differentiation (AD).  We will use the algopy module in Python.  This is an operator overloading method.  If you are using Matlab there are not any AD methods built in, but you can find some 3rd party tools.
To use algopy properly we are need to import overloaded versions of the functions we are using (sin, cos, etc.).  These overloaded versions will be able to keep track of partial derivatives through a chain rule.
Can you write Python code for it?

# You can ignore this.  I'm just ignoring a printed FutureWarning about a change affecting something internal to algopy
import warnings
warnings.filterwarnings("ignore", category=FutureWarning)

from algopy import UTPM  # just the name of the algorithm (stands for univariate Taylor propagation of matrices)
from algopy import sin, cos, exp, sqrt  # overloaded versions of functions we use
from algopy import arccos as acos  # need to rename b.c. using the math version (acos) whereas numpy uses arccos

# create an algopy version of x
x_algopy = UTPM.init_jacobian(x)

# create an algopy version of outputs
output, a, ap = function(x_algopy, params)

# extract the gradients
g_ad_oo = UTPM.extract_jacobian(output)  # could call again for the other outputs


I am working on the file "AD.ipynb".
The first task is:
Now we setup some values for the inputs and parameters and put them into a variable array and an parameter array.  Also we will save n which is the number of variables in x.
Can you write Python code for it?

# setup inputs
r = 0.5
chord = 0.1
Rhub = 0.1
Rtip = 1.0
phi = 0.2
cl = 0.3
cd = 0.002
B = 3
Vx = 1.0
Vy = 5.0
useCd = True
hubLoss = True
tipLoss = True
wakerotation = True

x = np.array([r, chord, Rhub, Rtip, phi, cl, cd, Vx, Vy])
params = np.array([B, useCd, hubLoss, tipLoss, wakerotation])

n = len(x)


I am working on the file "AD.ipynb".
The first task is:
The errors are pretty small, except for the third entry is really bad.  If we use a different step size for that one entry, we can do a little better.  It turns out the function is very insensitive to Rhub at this point and so getting an accurate gradient with FD is difficult.
Let's now look at the two AD methods: one with operator overloading and one with source code transformation
Can you write Python code for it?

print('error_ad_oo =', (g_ad_oo - g_cs)/g_cs)
print('error_ad_sc =', (g_ad_sc - g_cs)/g_cs)


I am working on the file "AFAString.ipynb".
The first task is:
3.2. Partition of Unity
The Atomic Function pulses superposition set at points -2, -1, 0, +1, +2... can exactly represent a Unity (number 1):
1 = ... up(x-3) + up(x-2) + up(x-1) + up(x-0) + up(x+1) + up(x+2) + up(x+3) + ...
1 = ... up(x-3) + up(x-2) + up(x-1) + up(x-0) + up(x+1) + up(x+2) + up(x+3) + ...
Can you write Python code for it?

x = np.arange(-2.0, 2.0, 0.01)
pl.plot(x, up(x, 1, -1), '--', linewidth=1, label='Atomic Function at x=-1')
pl.plot(x, up(x, 1, +0), '--', linewidth=1, label='Atomic Function at x=0')
pl.plot(x, up(x, 1, -1), '--', linewidth=1, label='Atomic Function at x=-1')
pl.plot(x, Sum(up(x, 1, -1), Sum(up(x), up(x, 1, 1))), linewidth=2, label='Atomic Function Compounding')
pl.title('Atomic Function Compounding represent 1')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AFAString.ipynb".
The first task is:
3.3. Atomic Function (AF) is a 'finite', 'compactly supported', or 'solitary' function
Like a Spline, Atomic Function (AF) 'compactly supported' not equal to zero only on section |x|<=1
Can you write Python code for it?

x = np.arange(-5.0, 5.0, 0.01)
pl.plot(x, up(x), label='Atomic Function', linewidth=2)
#pl.plot(x, dydx, '--', label='Atomic Function Derivative', linewidth=1, color="Green")

pl.title('Atomic Function is compactly supported')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AFAString.ipynb".
The first task is:
3.4 Atomic Function is a non-analytical function (can not be represented by Taylor's series), but with known Fourier Transformation allowing to exactly calculate AF in certain points, with tabular representation provided in script above.
4. Properties of Atomic String Function
4.1. AString is not only Integral but also Composing Branch of Atomic Function
AString'(x) = AString(2x+1) - AString(2x-1) = up(x)
Astring is a swing-like function - Integral of Atomic Function (AF) which can be expressed via AF itself:
AString(x) = Integral(0,x)(Up(x)) = Up(x/2 - 1/2) - 1/2
AString(x) = Integral(0,x)(Up(x)) = Up(x/2 - 1/2) - 1/2
4.2. Atomic Function is a 'solitonic atom' composed from two opposite AStrings
The concept of 'Solitonic Atoms' (bions) composed from opposite kinks is known in soliton theory [3,5].
up(x) = AString(2x + 1) - AString(2x - 1)
Can you write Python code for it?

#########   Presentation of Atomic Function via Atomic Strings ##########
x = np.arange(-2.0, 2.0, 0.01)

pl.plot(x, AString(x, 1, 0, 1, 0),  '--', linewidth=1,  label='AString(x)')
pl.plot(x, AString(x, 0.5, -0.5, +1, 0), '--', linewidth=2, label='+AString(2x+1)')
pl.plot(x, AString(x, 0.5, +0.5, -1, 0), '--', linewidth=2, label='-AString(2x-1)')
#pl.plot(x, up(x, 1.0, 0, 1, 0),     '--', linewidth=1, label='Atomic Function')
AS2 = Sum(AString(x, 0.5, -0.5, +1, 0), AString(x, 0.5, +0.5, -1, 0))
pl.plot(x, AS2, linewidth=3, label='Up(x) via Strings')
pl.title('Atomic Function as a Combination of AStrings')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AFAString.ipynb".
The first task is:
4.3. AStrings and Atomic Solitons
Solitonic mathematical properties of AString and Atomic Functions have been explored in author's paper [3] (Eremenko, S.Yu. Atomic solitons as a new class of solitons; 2018; https://www.researchgate.net/publication/329465767). They both satisfy differential equations with shifted arguments which introduce special kind of nonlinearity typical for all mathematical solitons.
AString belong to the class of Solitonic Kinks similar to sine-Gordon, Frenkel-Kontorova, tanh and others. Unlike other kinks, AStrings are truly solitary (compactly-supported) and also have a unique property of composing of both straight-line and solitonic atoms on lattice resembling particle-like properties of solitons.
Atomic Function up(x) is not actually a mathematical soliton, but a complex object composed from summation of two opposite AString kinks, and in solitonic terminology, is called 'solitonic atoms' (like bions).
4.4. All derivatives of AString can be represented via AString itself
AString'(x) = AString(2x + 1) - AString(2x - 1)
It means AString is a smooth (infinitely divisible) function, with fractalic properties.
4.5. AString and Fabius Function
Fabius Function https://en.wikipedia.org/wiki/Fabius_function, with unique property f'(x) = 2f(2x), published in 1966 but was probably known since 1935, is shifted and stretched AString function. Fabius function is not directly an integral of atomic function up(x).
Fabius(x) = AString(2x - 1) + 0.5
Can you write Python code for it?

x = np.arange(-2, 2.0, 0.01)
pl.title('AString and Fabius Functions')
pl.plot(x, AString(x, 0.5, 0.5, 1, 0.5), label='Fabius Function')
pl.plot(x, AString(x, 1, 0, 1, 0), label='AString Function')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AFAString.ipynb".
The first task is:
4.6. Partition of Line from Atomic String functions
Combination/summation of Atomic Strings can exactly represent a straight line:
x = ...Astring(x-2) + Astring(x-1) + AString(x) + Astring(x+1) + Astring(x+2)...
x = ...Astring(x-2) + Astring(x-1) + AString(x) + Astring(x+1) + Astring(x+2)...
Partition based on AString function with width 1 and height 1
Can you write Python code for it?

x = np.arange(-3, 3, 0.01)

pl.plot(x, AString(x, 1, -1.0, 1, 0),   '--',  linewidth=1, label='AString 1')
pl.plot(x, AString(x, 1, +0.0, 1, 0),   '--',  linewidth=1, label='AString 2')
pl.plot(x, AString(x, 1, +1.0, 1, 0),   '--',  linewidth=1, label='AString 3')

AS2 = Sum(AString(x, 1, -1.0, 1, 0), AString(x, 1, +0.0, 1, 0))
AS3 = Sum(AS2, AString(x, 1, +1.0, 1, 0))
pl.plot(x, AS3,  label='AStrings Sum', linewidth=2)

pl.title('Atomic Strings compose Line')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AFAString.ipynb".
The first task is:
2. Atomic String Function (AString) is an Integral and Composing Branch of Atomic Function up(x) (introduced in 2017 by S. Yu. Eremenko)
AString function is solitary kink function which simultaneously is integral and composing branch of atomic function up(x)
AString'(x) = AString(2x+1) - AString(2x-1) = up(x)
Can you write Python code for it?

############### Atomic String #############
def AString1(x: float) -> float:
    res = 1 * (upulse(x/2.0 - 0.5) - 0.5)
    return res

############### Atomic String Pulse with width, shift and scale #############
def AStringPulse(t: float, a = 1., b = 0., c = 1., d = 0.) -> float:
    x = (t - b)/a
    if (x < -1):
        res = -0.5
    elif (x > 1):
        res = 0.5
    else:
        res = AString1(x)
    res = d + res * c
    return res

###### Atomic String Applied to list with width, shift and scale #############
def AString(x: list, a = 1., b = 0., c = 1., d = 0.) -> list:
    res = []
    for i in range(len(x)):
        res.append(AStringPulse(x[i], a, b, c, d))
        #res[i] = AStringPulse(x[i], a, b, c)
    return res

###### Summation of two lists #############
def Sum(x1: list, x2: list) -> list:
    res = []
    for i in range(len(x1)):
       res.append(x1[i] + x2[i])
    return res

x = np.arange(-2.0, 2.0, 0.01)
pl.title('Atomic String Function')
pl.plot(x, AString(x, 1.0, 0, 1, 0), label='Atomic String')
pl.grid(True)
pl.show()


I am working on the file "AFAString.ipynb".
The first task is:
3. Properties of Atomic Function Up(x)
3.1. Atomic Function Derivative expressed via Atomic Function itself
Atomic Function Derivative can be exressed via Atomic Function itself - up'(x)= 2up(2x+1)-2up(2x-1) meaning the shape of pulses for derivative function can be represented by shifted and stratched Atomic Function itself - remarkable property
up'(x)= 2up(2x+1)-2up(2x-1)
Atomic Function and its Derivative plotted together
Can you write Python code for it?

x = np.arange(-2.0, 2.0, 0.01)
pl.plot(x, up(x), label='Atomic Function', linewidth=2)
pl.plot(x, dydx, '--', label='Atomic Function Derivative', linewidth=1, color="Green")

pl.title('Atomic Function and Its Derivative')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AFAString.ipynb".
The first task is:
6. 'Soliton Nature' book
6.1. AStrings and Atomic functions are also described in the book 'Soliton Nature'
Soliton Nature book is easy-to-read, pictorial, interactive book which uses beautiful photography, video channel, and computer scripts in R and Python to demonstrate existing and explore new solitons – the magnificent and versatile energy concentration phenomenon of nature.  New class of atomic solitons can be used to describe Higgs boson (‘the god particle’) fields, spacetime quanta and other fundamental building blocks of nature.
Can you write Python code for it?

#pl.rcParams["figure.figsize"] = 16,12
book = pl.imread('BookSpread_small.png')
pl.imshow(book)


I am working on the file "AFAString.ipynb".
The first task is:
Atomic String, Atomic Function (AF) and AF Derivative plotted together
Can you write Python code for it?

x = np.arange(-2.0, 2.0, 0.01)

#This Calculates Derivative
dx = x[1] - x[0]
dydx = np.gradient(up(x), dx)

pl.plot(x, up(x), label='Atomic Function')
pl.plot(x, AString(x, 1.0, 0, 1, 0), linewidth=2, label='Atomic String Function')
pl.plot(x, dydx, '--', label='A-Function Derivative')

pl.title('Atomic and AString Functions')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AFIntegrals.ipynb".
The first task is:
2. Atomic String Function (AString) is an Integral and Composing Branch of Atomic Function up(x) (introduced in 2017 by S. Yu. Eremenko)
AString function is solitary kink function which simultaneously is integral and composing branch of atomic function up(x)
AString'(x) = AString(2x+1) - AString(2x-1) = up(x)
Can you write Python code for it?


I am working on the file "AFIntegrals.ipynb".
The first task is:
3. Atomic Function Integrals
3.1. Integral of basic Atomic Function up(x)
Can you write Python code for it?

from scipy.integrate import simps

x = np.arange(-1.0, 1.0, 0.01)
I1 = simps(up(x), x)
print(I1)


I am working on the file "AFIntegrals.ipynb".
The first task is:
6. 'Soliton Nature' book by S.Eremenko
6.1. AStrings and Atomic functions are described in the book 'Soliton Nature'
Soliton Nature book is easy-to-read, pictorial, interactive book which uses beautiful photography, video channel, and computer scripts in R and Python to demonstrate existing and explore new solitons – the magnificent and versatile energy concentration phenomenon of nature.  New class of atomic solitons can be used to describe Higgs boson (‘the god particle’) fields, spacetime quanta and other fundamental building blocks of nature.
Can you write Python code for it?


I am working on the file "AFIntegrals.ipynb".
The first task is:
Atomic String, Atomic Function (AF) and AF Derivative plotted together
Can you write Python code for it?


I am working on the file "AFIntegrals.ipynb".
The first task is:
Integral(Sum(up(x),n),-1,1) = n - Integral of Sum of n AF pulses equal to n. Important for quantisation
4. Integrals of Atomic String Function
4.1. Integral of basic Atomic String Function AString(x)
Can you write Python code for it?

x = np.arange(-1.0, 1.0, 0.001)
I1 = simps(AString(x), x)
print(I1)

x = np.arange(0, 1.0, 0.001)
I1 = simps(AString(x), x)
print(I1)

x = np.arange(-1, 0.0, 0.001)
I1 = simps(AString(x), x)
print(I1)


I am working on the file "AFIntegrals.ipynb".
The first task is:
Partition based on AString with certain width and height depending on a size of 'quanta'
Can you write Python code for it?

x = np.arange(-40.0, 40.0, 0.01)

width  = 10.0
height = 10.0

#pl.plot(x, ABline (x, 1, 0),            label='ABLine 1*x')
pl.plot(x, AString(x, width, -3*width/2, height, -3*width/2), '--',  linewidth=1, label='AString 1')
pl.plot(x, AString(x, width, -1*width/2, height, -1*width/2), '--',  linewidth=1, label='AString 2')
pl.plot(x, AString(x, width, +1*width/2, height, +1*width/2), '--',  linewidth=1, label='AString 3')
pl.plot(x, AString(x, width, +3*width/2, height, +3*width/2), '--',  linewidth=1, label='AString 4')

AS2 = Sum(AString(x, width, -3*width/2, height, -3*width/2), AString(x, width, -1*width/2, height, -1*width/2))
AS3 = Sum(AS2, AString(x, width,+1*width/2, height, +1*width/2))
AS4 = Sum(AS3, AString(x, width,+3*width/2, height, +3*width/2))
pl.plot(x, AS4,  label='AStrings Joins', linewidth=2)

pl.title('Atomic Strings Combinations')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AFIntegrals.ipynb".
The first task is:
Summary - Integral(AString(x),-1,1) = 0; Integral(AString(x),0,1) ~ 0.36;
4.2. Integrals of stretched Atomic String function c*AString(x/a)
Can you write Python code for it?

a = 0.5; c = 1
x = np.arange(0, a, 0.001)

I2 = simps(AString(x, a, 0., c), x)
print(I2)

a = 2; c = 5
x = np.arange(0, a, 0.001)

I3 = simps(AString(x, a, 0., c), x)
print(I3)


I am working on the file "AFIntegrals.ipynb".
The first task is:
Summary - Integral(cup(x/a),-1,1) = ca - Width times Height
3.3. Integral of the chain of Atomic Function pulses
The Atomic Function pulses superposition set at points -2, -1, 0, +1, +2... can exactly represent a Unity (number 1), so-called 'Partition of Unity'
1 = ... up(x-3) + up(x-2) + up(x-1) + up(x-0) + up(x+1) + up(x+2) + up(x+3) + ...
Can you write Python code for it?

x = np.arange(-2.0, 2.0, 0.01)
pl.plot(x, up(x, 1, -1), '--', linewidth=1, label='Atomic Function at x=-1')
pl.plot(x, up(x, 1, +0), '--', linewidth=1, label='Atomic Function at x=0')
pl.plot(x, up(x, 1, -1), '--', linewidth=1, label='Atomic Function at x=-1')
pl.plot(x, Sum(up(x, 1, -1), Sum(up(x), up(x, 1, 1))), linewidth=2, label='Atomic Function Compounding')
pl.title('Partition of Unity - AF Compounding represent 1')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()

Continuum = Sum(up(x, 1, -1), Sum(up(x), up(x, 1, 1))) # Summation of three atomic functions

I4 = simps(Continuum, x)
print(I4)


I am working on the file "AFIntegrals.ipynb".
The first task is:
Summary - Integral(up(x),-1,1) = 1, as expected
3.2. Integrals of stretched Atomic Function c*up(x/a)
Can you write Python code for it?


I am working on the file "AI/DL/week1_神经网络doc.ipynb".
The first task is:
### 前向和后向传播
正向传播：
得到X
计算 A = \sigma(w^T X + b) = (a^{(0)}, a^{(1)}, ..., a^{(m-1)}, a^{(m)})

计算损失函数：J = -\frac{1}{m}\sum_{i=1}^{m}y^{(i)}\log(a^{(i)})+(1-y^{(i)})\log(1-a^{(i)})
Can you write Python code for it?

# GRADED FUNCTION: propagate
def propagate(w, b, X, Y):
    """
    Implement the cost function and its gradient for the propagation explained above

    Arguments:
    w -- weights, a numpy array of size (num_px * num_px * 3, 1)
    b -- bias, a scalar
    X -- data of size (num_px * num_px * 3, number of examples)
    Y -- true "label" vector (containing 0 if non-cat, 1 if cat) of size (1, number of examples)

    Return:
    cost -- negative log-likelihood cost for logistic regression
    dw -- gradient of the loss with respect to w, thus same shape as w
    db -- gradient of the loss with respect to b, thus same shape as b

    Tips:
    - Write your code step by step for the propagation. np.log(), np.dot()
    """
    m = X.shape[1]
    # FORWARD PROPAGATION (FROM X TO COST)
    ### START CODE HERE ### (≈ 2 lines of code)
    A = sigmoid(np.dot(w.T, X) + b)            # compute activation
    cost = -1 / m * np.sum(Y * np.log(A) + (1 - Y) * np.log(1 - A),keepdims=True)         # compute cost
    ### END CODE HERE ###

    # BACKWARD PROPAGATION (TO FIND GRAD)
    ### START CODE HERE ### (≈ 2 lines of code)
    dw = (1 / m) * np.dot(X, (A - Y).T)
    db = (1 / m) * np.sum(A - Y)
    ### END CODE HERE ###
    assert(dw.shape == w.shape)
    assert(db.dtype == float)
    cost = np.squeeze(cost)
    assert(cost.shape == ())

    grads = {"dw": dw,
             "db": db}

    return grads, cost


I am working on the file "AI/DL/week1_神经网络doc.ipynb".
The first task is:
训练准确100％。 这是一个很好的情况：你的模型正在运行，并且具有足够的容量来适合训练数据。 测试误差为68％。 考虑到我们使用的数据集很小，并且逻辑回归是线性分类器，对于这个简单的模型来说，这实际上还不错。 但请放心，下周你将建立一个更好的分类器！
此外，你会看到该模型明显适合训练数据。 在本专业的稍后部分，你将学习如何减少过度拟合，例如通过使用正则化。 使用下面的代码（并更改index变量），你可以查看测试集图片上的预测。
Can you write Python code for it?

# Example of a picture that was wrongly classified.
index = 2
plt.imshow(test_set_x[:,index].reshape((num_px, num_px, 3)))
print ("y = " + str(test_set_y[0,index]) + ", you predicted that it is a \"" + classes[int(d["Y_prediction_test"][0,index])].decode("utf-8") +  "\" picture.")

#  学习率 为了使梯度下降起作用，你必须明智地选择学习率。
learning_rates = [0.01, 0.001, 0.0001]
models = {}
for i in learning_rates:
    print ("learning rate is: " + str(i))
    models[str(i)] = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 2000, learning_rate = i, print_cost = False)
    print ('\n' + "-------------------------------------------------------" + '\n')

for i in learning_rates:
    plt.plot(np.squeeze(models[str(i)]["costs"]), label= str(models[str(i)]["learning_rate"]))

plt.ylabel('cost')
plt.xlabel('iterations')

legend = plt.legend(loc='upper center', shadow=True)
frame = legend.get_frame()
frame.set_facecolor('0.90')
plt.show()

# 使用自己的图像进行测试
## START CODE HERE ## (PUT YOUR IMAGE NAME)
#my_image = "cat_in_iran.jpg"   # change this to the name of your image file
## END CODE HERE ##

# We preprocess the image to fit your algorithm.
from skimage.transform import resize
fname = '../data/deeplearningai17761/cat_in_iran.jpg'
image = np.array(plt.imread(fname))
image.shape
# my_image = scipy.misc.imresize(image, size=(num_px,num_px)).reshape((1, num_px*num_px*3)).T
my_image=resize(image, output_shape=(num_px,num_px)).reshape((1, num_px*num_px*3)).T
my_predicted_image = predict(d["w"], d["b"], my_image)

plt.imshow(image)
print("y = " + str(np.squeeze(my_predicted_image)) + ", your algorithm predicts a \"" + classes[int(np.squeeze(my_predicted_image)),].decode("utf-8") +  "\" picture.")


I am working on the file "AI/DL/week1_神经网络doc.ipynb".
The first task is:
预处理数据集的常见步骤是：
找出数据的尺寸和维度（m_train，m_test，num_px等）
重塑数据集，以使每个示例都是大小为（num_px \ num_px \ 3，1）的向量
“标准化”数据
Can you write Python code for it?

# 标准化，像素值红、绿、蓝色通道（RGB），因此像素值实际上是一个从0到255的三个数字的向量
train_set_x = train_set_x_flatten/255.
test_set_x = test_set_x_flatten/255.


I am working on the file "AI/ML/week2逻辑回归代码实现.ipynb".
The first task is:
${\theta_j}:={\theta_j}-\alpha \frac{1}{m}\sum\limits_{i=1}^{m}{({h_\theta}({{x}^{(i)}})-{{y}^{(i)}}){{x}_{j}}^{(i)}}$来同时更新所有$\theta $的值。
Can you write Python code for it?

# 梯度下降函数，和第一种自定义学习率和循环次数
def gradientSelf(theta,X,y,alpha,num):
    X = np.matrix(X)
    y = np.matrix(y)
    tc = int(theta.ravel().shape[1])
    for i in range(num):
        dist = sigmoid(X*theta.T) - y
        for j in range(tc):
            term = np.multiply(dist,X[:,j])
            theta[0,j] = theta[0,j] - (alpha/len(X)) * np.sum(term)
    return theta

## 单纯梯度下降函数(无学习率，也无需更新theta,相当于循环一次返回)，为第三方算法调用
def gradient(theta,X,y):
    theta = np.matrix(theta)
    X = np.matrix(X)
    y = np.matrix(y)
    tc = int(theta.ravel().shape[1])
    grad = np.zeros(tc)
    dist = sigmoid(X*theta.T) - y
    for j in range(tc):
        term = np.multiply(dist,X[:,j])
        grad[j] = np.sum(term)/ len(X)
    return grad

# 执行测试
# 加一列常数列
data.insert(0, 'Ones', 1)

# 初始化X，y，θ
cols = data.shape[1]
X = data.iloc[:,0:cols-1]
y = data.iloc[:,cols-1:cols]
# 转换X，y的类型
X = np.array(X.values)
y = np.array(y.values)

theta = np.matrix(np.zeros(3))
selfpredict=gradientSelf(theta,X,y,0.05,100000)
print(selfpredict)

# 作图查看 两个变量时，分割线 sigma(z)=1/2 => -z=0 => x*θ.T = 0  => x2 = （-θ0 - θ1x1)/θ2
# thetaR = np.array([-130.02753949,    0.767743  ,    1.38019249])
thetaR = np.array([-133.85163041   , 1.1383581 ,    1.09589532])

predict = sigmoid(X*thetaR.T)
fig, ax = plt.subplots(figsize=(10,6))
plotting_x1 = np.linspace(20,100, 100)
plotting_h1 = ( - thetaR[0] - thetaR[1] * plotting_x1) / thetaR[2]

ax.plot(plotting_x1, plotting_h1, 'y', label='Prediction')
ax.scatter(positive['Exam1'], positive['Exam2'], s=50, c='b', marker='o', label='通过')
ax.scatter(negative['Exam1'], negative['Exam2'], s=50, c='r', marker='x', label='不通过')
ax.legend()
ax.set_xlabel('Exam 1 Score')
ax.set_ylabel('Exam 2 Score')
plt.show()
plt.show()

#
import scipy.optimize as opt
theta = theta = np.matrix(np.zeros(3))
result = opt.fmin_tnc(func=cost, x0=theta, fprime=gradient, args=(X, y))
result
# 带入上面的图


I am working on the file "AI/ML/week2逻辑回归代码实现.ipynb".
The first task is:
正则化逻辑回归
Can you write Python code for it?

data_init = pd.read_csv('../data/weekRegData.txt', header=None, names=['x1', 'x2', 'Accepted'])
data_init.head()

positive2 = data_init[data_init['Accepted'].isin([1])]
negative2 = data_init[data_init['Accepted'].isin([0])]

fig, ax = plt.subplots(figsize=(12,8))
ax.scatter(positive2['x1'], positive2['x2'], s=50, c='b', marker='o', label='Accepted')
ax.scatter(negative2['x1'], negative2['x2'], s=50, c='r', marker='x', label='Rejected')
ax.legend()
ax.set_xlabel('x1')
ax.set_ylabel('x2')
plt.show()


I am working on the file "AI/ML/week3多类逻辑回归和前馈神经网络.ipynb".
The first task is:
一对多分类器
已经定义了代价函数和梯度函数，现在是构建分类器的时候了。 对于这个任务，我们有10个可能的类，并且由于逻辑回归只能一次在2个类之间进行分类，我们需要多类分类的策略。 在本练习中，我们的任务是实现一对一全分类方法，其中具有k个不同类的标签就有k个分类器，每个分类器在“类别 i”和“不是 i”之间决定。 我们将把分类器训练包含在一个函数中，该函数计算10个分类器中的每个分类器的最终权重，并将权重返回为k*(n + 1)数组，其中n是参数数量
Can you write Python code for it?

from scipy.optimize import minimize
#当我我们需要预测多个结果（y=1 y=2..y=10）循环找到1，2...到10的训练集，分别训练，分别得到我们需要的特征值
def one_vs_all(X, y, num_labels, learning_rate):
    rows = X.shape[0]
    params = X.shape[1]
    # 当我我们需要预测多个结果（y=1 y=2..y=10）
    all_theta = np.zeros((num_labels, params + 1))
    # insert a column of ones at the beginning for the intercept term
    X = np.insert(X, 0, values=np.ones(rows), axis=1)
    # 循环找到1，2...到10的训练集，分别训练，分别得到我们需要的特征值
    for i in range(1, num_labels + 1):
        theta = np.zeros(params + 1)
        # 分别求对应结果的训练集
        y_i = np.array([1 if value == i else 0 for value in y])
        y_i = np.reshape(y_i, (rows, 1))
        # minimize the objective function
        fmin = minimize(fun=costReg, x0=theta, args=(X, y_i, learning_rate), method='TNC', jac=gradientReg)
        all_theta[i-1,:] = fmin.x
    return all_theta


I am working on the file "AI/ML/week3多类逻辑回归和前馈神经网络.ipynb".
The first task is:
一对多预测
我们现在准备好最后一步 - 使用训练完毕的分类器预测每个图像的标签。 对于这一步，我们将计算每个类的类概率，对于每个训练样本（使用当然的向量化代码），并将输出类标签为具有最高概率的类。
Can you write Python code for it?

def predict_all(X, all_theta):
    rows = X.shape[0]
    params = X.shape[1]
    num_labels = all_theta.shape[0]
    X = np.insert(X, 0, values=np.ones(rows), axis=1)
    # z转换成矩阵
    X = np.matrix(X)
    all_theta = np.matrix(all_theta)#（10，401）
    # 计算sigmoid值 shape=（5000，10）
    h = sigmoid(X * all_theta.T)
    # 找到每行计算的最大值，就是当前一行的训练集计算预测最大的y值
    h_argmax = np.argmax(h, axis=1)
    # 由于我们插入了新值θ0，后续计算这里要去掉第一行
    h_argmax = h_argmax + 1
    return h_argmax

# 现在我们可以使用predict_all函数为每个实例生成类预测，看看我们的分类器是如何工作的。
y_pred = predict_all(data['X'], all_theta)
# 打印预测分析
# sklearn中的classification_report函数用于显示主要分类指标的文本报告．在报告中显示每个类的精确度，召回率，F1值等信息。
print(classification_report(data['y'], y_pred))
# support列为每个标签的出现次数 precision recall f1-score三列分别为各个类别的精确度/召回率及 F1值．


I am working on the file "AI/ML/week3多类逻辑回归和前馈神经网络.ipynb".
The first task is:
数据可视化
Can you write Python code for it?

# 随机展示20个数据
sample_idx = np.random.choice(np.arange(data['X'].shape[0]), 25)
sample_images = data['X'][sample_idx, :]
sample_images

#展示二进制图
fig, ax_array = plt.subplots(nrows=5, ncols=5, sharey=True, sharex=True, figsize=(8, 8))
for r in range(5):
    for c in range(5):
        ax_array[r, c].matshow(np.array(sample_images[5 * r + c].reshape((20, 20))).T,cmap=matplotlib.cm.binary)
        plt.xticks(np.array([]))
        plt.yticks(np.array([]))


I am working on the file "AI/ML/week4反向传播实现.ipynb".
The first task is:
初始话参数
到目前为止我们都是初始所有参数为0，这样的初始方法对于逻辑回归来说是可行的，但是对于神经网络来说是不可行的。如果我们令所有的初始参数都为0，这将意味着我们第二层的所有激活单元都会有相同的值。同理，如果我们初始所有的参数都为一个非0的数，结果也是一样的。
我们通常初始参数为正负ε之间的随机值，假设我们要随机初始一个尺寸为10×11的参数矩阵，代码如下：
Theta1 = rand(10, 11) (2eps) – eps
Can you write Python code for it?

# 初始化设置
input_size = 400 #输入单元数量
hidden_size = 25 # y隐藏单元数量
num_labels = 10 # 输出单元数
epsilon = 0.001
theta01=np.random.rand(hidden_size,input_size+1) * 2*epsilon - epsilon# +1是添加偏置单元
theta02 =np.random.rand(num_labels,hidden_size+1)* 2*epsilon - epsilon
theta01.shape,theta02.shape


I am working on the file "AI/ML/week4反向传播实现.ipynb".
The first task is:
反向传播
反向传播的步骤是，给定训练集，先计算正向传播，再对于层的每个节点，计算误差项，这个数据衡量这个节点对最后输出的误差“贡献”了多少。 对于每个输出节点，我们可以直接计算输出值与目标值的差值，定义为δ。对于每个隐藏节点，需要基于现有权重及（l+1）层的误差，计算
 步骤：

随机初始化权重theta
实现前向传递对任何xi 都能取得h（xi）
实现Jθ
Can you write Python code for it?

# 分别得出
def forward_propagateNEW(X,thetalist):
    m= X.shape[0]
    a = np.insert(X,0, values=np.ones(m), axis=1)
    alist=[a]
    zlist=[]
    for i in range(len(thetalist)):
        theta= thetalist[i]
        z = a * theta
        # a= np.insert(sigmoid(z),0, values=np.ones(m), axis=1)
        a=sigmoid(z)
        if(i<len(thetalist)-1):
            a= np.insert(a,0, values=np.ones(m), axis=1)
        zlist.append(z)
        alist.append(a)
    return zlist,alist

# Δ 用delta1 和delta2 替代
def backpropRegSelf(input_size, hidden_size, num_labels, X, y, learning_rate,L=3):  # 随机化后的  这里为3层
    m = X.shape[0]
    X = np.matrix(X)
    y = np.matrix(y)
    #初始化参数
    theta1 = (np.random.random((input_size+1,hidden_size))- 0.5)* 0.24
    theta2 = (np.random.random((hidden_size+1,num_labels))- 0.5)* 0.24
    encoder = OneHotEncoder(sparse=False)
    y_onehot  = encoder.fit_transform(y) # 格式化y
    # 前向计算 每层值
    theta = [theta1, theta2]
    zlist,alist = forward_propagateNEW(X, theta)# 返回 a1 z2 a2 。。。
    # 初始化Deta
    Delta=[]
    for th in theta:
        Delta.append(np.zeros(th.shape))
    for i in range(m):
        # 以计算a z
        for l in range(L,1,-1): # 3,2 表示层数，最后一层已经算出来，单独列放
            #最后一层
            if l==L:
                delta=alist[-1][i,:]-y_onehot[i,:] # 最后一层得δ
                Delta[l-2] = Delta[l-2] +   alist[l-2][i,:].T  * delta
            else:
                zl = zlist[l-2][i,:]
                zl = np.insert(zl, 0, values=np.ones(1))  # (1, 26) 怎加偏执项
                # d2t = np.multiply((theta2.T * d3t.T).T, sigmoid_gradient(z2t))  # (1, 26)
                # delta1 = delta1 + (d2t[:,1:]).T * a1t
                delta = np.multiply(delta*theta[l-1].T, sigmoid_gradient(zl))  #
                 # 因为数组从零开始，且 Delta 为 1 2 层开始 delta 从2 层开始   # (25, 401)# (10, 26)
                Delta[l-2] = Delta[l-2] +   alist[l-2][i,:].T  * delta[:,1:]
    # add the gradient regularization term
    gradAll = None
    for j in range(len(Delta)):
        Delta[j][:,1:] = Delta[j][:,1:]/m +...


I am working on the file "AI/ML/week4反向传播实现.ipynb".
The first task is:
模型展示
按照默认 我们设计一个输入层，一个隐藏层，一个输出层
前向传播和代价函数
在逻辑回归中，我们只有一个输出变量，又称标量（scalar），也只有一个因变量$y$，但是在神经网络中，我们可以有很多输出变量，我们的$h_\theta(x)$是一个维度为$K$的向量，并且我们训练集中的因变量也是同样维度的一个向量，因此我们的代价函数会比逻辑回归更加复杂一些，为：$\newcommand{\subk}[1]{ #1_k }$ $$h_\theta\left(x\right)\in \mathbb{R}^{K}$$ $${\left({h_\theta}\left(x\right)\right)}_{i}={i}^{th} \text{output}$$
$J(\Theta) = -\frac{1}{m} \left[ \sum\limits_{i=1}^{m} \sum\limits_{k=1}^{k} {y_k}^{(i)} \log \subk{(h_\Theta(x^{(i)}))} + \left( 1 - y_k^{(i)} \right) \log \left( 1- \subk{\left( h_\Theta \left( x^{(i)} \right) \right)} \right) \right] + \frac{\lambda}{2m} \sum\limits_{l=1}^{L-1} \sum\limits_{i=1}^{s_l} \sum\limits_{j=1}^{s_{l+1}} \left( \Theta_{ji}^{(l)} \right)^2$
Can you write Python code for it?

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

#2st 上面传播规律,定义第一层，并计算第二层（隐藏层）的值,并添加额外值
def forward_propagate(X,theta1,theta2):
    m= X.shape[0]
    a1 = np.insert(X,0, values=np.ones(m), axis=1)
    Z2 = a1*theta1.T
    a2=  np.insert(sigmoid(Z2),0, values=np.ones(m), axis=1)
    Z3= a2*theta2.T
    h= sigmoid(Z3)
    return a1,Z2,a2,Z3,h

# 代价函数（不带规则化项（也叫权重衰减项） Y=R(5000*10) ,这里直接使用二维矩阵，代替循环累加
def cost(X,Y,theta1,theta2):
    m = X.shape[0]
    X = np.matrix(X)
    Y = np.matrix(Y)
    h=forward_propagate(X,theta1,theta2)
    # multiply 矩阵size相同对应相乘
    first = np.multiply(Y,np.log(h))
    second = np.multiply((1-Y),np.log((1-h)))
    J= np.sum(first+second)
    J = (-1/m)*J
    return J

# 对y标签进行编码 一开始我们得到的y是维500*1 的向量，但我们要把他编码成的矩阵。 比如说，原始y0=2，那么转化后的Y对应行就是[0,1,0...0]，原始转化后的Y对应行就是[0,0...0,1]
# Scikitlearn有一个内置的编码函数，我们可以使用这个。
encoder = OneHotEncoder(sparse=False)
y_onehot = encoder.fit_transform(y)
y_onehot.shape

y[0], y_onehot[0,:] # y0是数字0

# 初始化设置
input_size = 400
num_labels = 10

cost(X, y_onehot,theta1, theta2)

# 加入正则项
def cost_reg(X,Y,theta1,theta2,learning_rate):
    m = X.shape[0]
    X = np.matrix(X)
    Y = np.matrix(Y)
    _,_,_,_,h=forward_propagate(X,theta1,theta2)
    first = np.multiply(Y,np.log(h))
    second = np.multiply((1-Y),np.log((1-h)))
    J= np.sum(first+second)
    # 计算正则时，第一项时不参与计算
    J = (-1/m)*J + (float(learning_rate) / (2 * m))*(np.sum(np.power(theta1[:,1:],2))+np.sum(np.power(theta2[:,1:],2)))
    return J

# theta1.shape,theta2.shape
cost_reg(X, y_onehot,theta1, theta2,1)


I am working on the file "AI/ML/week5_code.ipynb".
The first task is:
拟合线性回归
Can you write Python code for it?

# 调用工具库找到theta最优解
theta = np.ones(X.shape[1])
final_theta = opt.minimize(fun=costReg, x0=theta, args=(X, y, 0), method='TNC', jac=gradientReg, options={'disp': True}).x
final_theta

# 作图拟合
predict  = X*(np.matrix(final_theta).T)
fig, ax = plt.subplots(figsize=(12,8))
plt.scatter(X[:,1], y, c='r', label="Training data")
plt.plot(X[:, 1], predict, c='b', label="Prediction")
ax.set_xlabel('water_level')
ax.set_ylabel('flow')
ax.legend()
plt.show()


I am working on the file "AI/ML/week5_code.ipynb".
The first task is:
方差和偏差
机器学习中的一个重要概念是偏差-方差权衡。偏差较大的模型会欠拟合，而方差较大的模型会过拟合
学习曲线

使用训练集的子集来拟合应模型
在计算训练代价和验证集代价时，没有用正则化
记住使用相同的训练集子集来计算训练代价

$J_{train}(\theta) = \frac{1}{m}\sum_\limits{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)})^2$
Can you write Python code for it?

def linear_regression(X,y,l=1):
    """linear regression
        args:
            X: feature matrix, (m, n+1) # with incercept x0=1
            y: target vector, (m, )
            l: lambda constant for regularization
        return: trained parameters
    """
    # init theta
    theta = np.ones(X.shape[1])
    res=opt.minimize(fun=costReg,
                        x0=theta,
                        args=(X, y, l),
                        method='TNC',
                        jac=gradientReg,
                        options={'disp': True})
    return res

#初始化学习曲线tc 和cv得cost值
training_cost, cv_cost = [], []
m=X.shape[0]
for i in range(m):
    res = linear_regression(X[:i+1,:],y[:i+1])
    training_cost.append(costReg(res.x, X[:i+1, :], y[:i+1], 0))
    cv_cost.append(costReg(res.x,Xval,yval,0))
# 作图展示训练得学习曲线和交叉验证得曲线对比
fig, ax = plt.subplots(figsize=(12,8))
plt.plot(np.arange(1,m+1), training_cost, c='b', label="train learnning plot")
plt.plot(np.arange(1,m+1), cv_cost, c='r', label="cross test learnning plot")
ax.set_xlabel('train sets')
ax.set_ylabel('cost')
ax.legend()
plt.show()


I am working on the file "AI/ML/week5_code.ipynb".
The first task is:
正则化线性回归代价函数
$J\left( \theta \right)=\frac{1}{2m}\sum\limits_{i=1}^{m}{[({{({h_\theta}({{x}^{(i)}})-{{y}^{(i)}})}^{2}}+\lambda \sum\limits_{j=1}^{n}{\theta _{j}^{2}})]}$
Can you write Python code for it?

#为输入值添加偏置X0
X, Xval, Xtest = [np.insert(x.reshape(x.shape[0], 1), 0, np.ones(x.shape[0]), axis=1) for x in (X, Xval, Xtest)]
print(X.shape,Xval.shape,Xtest.shape)

def cost(theta,X,y):
    theta = np.matrix(theta)
    X=np.matrix(X)
    y=np.matrix(y)
    m=X.shape[0]
    inner = np.power((X*theta.T - y.T),2)
    return np.sum(inner)/(2*m)

def costReg(theta,X,y,reg=1):
    m=X.shape[0]
    return  cost(theta,X,y)+ np.sum(np.power(theta[1:],2))/(2*m)

theta = np.ones(X.shape[1])
costReg(theta, X, y, 1),cost(theta, X, y)

# 正则化线性回归的梯度

def gradient(theta,X,y):
    theta = np.matrix(theta)
    X=np.matrix(X)
    y=np.matrix(y)
    m=X.shape[0]
    dis = (X * theta.T) - y.T
    term= dis.T * X
    return term/m

def gradientReg(theta, X, y, reg = 1):
    m=X.shape[0]
    regularized_term = theta.copy()  # same shape as theta
    regularized_term[0] = 0  # don't regularize intercept theta
    regularized_term = (reg/m)*regularized_term
    return gradient(theta,X,y) + regularized_term

gradientReg(theta, X, y, 1),gradient(theta,X,y)


I am working on the file "AI/ML/week5_code.ipynb".
The first task is:
由上图可以看的λ大致为0.05时，交叉验证和训练得cost值都达到最小
Can you write Python code for it?

# 用测试集合和我们预测集合比较，这里我们去λ=0.1带入

# 预测值与实际值比较
# from sklearn.metrics import classification_report#这个包是评价报告
# l=0.1
# predict = Xtest_poly*np.matrix(linear_regression(X_poly, y, l).x).T
# predict = np.ravel(predict)
# use test data to compute the cost
for l in l_candidate:
    theta = linear_regression(X_poly, y, l).x
    print('test cost(l={}) = {}'.format(l, cost(theta, Xtest_poly, ytest)))


I am working on the file "AI/ML/专利.ipynb".
The first task is:
r = \frac{\sum (x - m_x) (y - m_y)}
             {\sqrt{\sum (x - m_x)^2 \sum (y - m_y)^2}}
Can you write Python code for it?

from scipy.stats import pearsonr
def pearsonrs(x,y):
    pears = []
    # 循环列处理
    for arr in x.T:
        pears.append(pearsonr(arr,Y)[0])
    return np.array(pears)

pears = np.abs(pearsonrs(X1,Y))
index = np.argsort(-pears)
pears,index

# 选取相关系数大于0.1的，即前9个特征
X2 = X1[:,index[0:9]]
X2.shape

# 1，分别将数据分为训练集和测试集,随机分配
X2Y = np.insert(X2,0,Y,axis=1)
X1Y= np.insert(X1,0,Y,axis=1)
# 过滤后
np.array(np.random.shuffle(X2Y))

trp = X2Y[:140,:]
tep =X2Y[140:,:]
# 原始
np.random.shuffle(X1Y)
trraw= X1Y[:140,:]
teraw= X1Y[140:,:]
trp.shape,tep.shape,trraw.shape,teraw.shape


I am working on the file "AI/ML/专利.ipynb".
The first task is:
皮尔逊相关系数定义为两个变量之间的协方差和标准差的商：

总体相关系数，常用希腊小写字母ρ
作为代表符号。估算样本的协方差和标准差，可得到皮尔逊相关系数，常用英文小写字母 r

皮尔逊相关系数有一个重要的数学特性是，因两个变量的位置和尺度的变化并不会引起该系数的改变，即它该变化的不变量(由符号确定)。
svd
Can you write Python code for it?

求相关系数矩阵 rij,当0<i,j<n 其中n为原数据集类别数量

求得相关矩阵后分别计算每个特征值对整体影响带入函数计算整理权重值w

得到相关权重W矩阵后，适当去除特征值相关权重较小的特征集合

剩下的数据集分离要预测的结果集，将剩下特征集做PCA（主要成分分析），得到训练需要的维度K

利用PCA降维后车辆数据后通过SVR回归训练

训练后测试集验证是否满足


I am working on the file "AI_Academy/trend_micro_basic_data_intro.ipynb".
The first task is:
其它會用到的工具
Can you write Python code for it?

#AUC計算範例
import numpy as np
from sklearn import metrics
y = np.array([1, 1, 2, 2])
pred = np.array([0.5, 1, 0.9, 1])
fpr, tpr, thresholds = metrics.roc_curve(y, pred, pos_label=2)
metrics.auc(fpr, tpr)

#timestamp轉換
import datetime
print(
    datetime.datetime.fromtimestamp(
        int("1488326402")
    ).strftime('%Y-%m-%d %H:%M:%S'))


I am working on the file "AI_Academy/trend_micro_basic_data_intro.ipynb".
The first task is:
聚合函數
count（個數）, sum（加總）, mean（平均）, median（中位數）, std（標準差）, var（變異數）, first（第一個非NA）, last（最後一個非NA）
Can you write Python code for it?


I am working on the file "AI_Experiments/LSTM_changing_batch_size.ipynb".
The first task is:
3 Solutions

If we just want to predict 1 step, there are 3 solutions:
Solution 1 - batch size = 1
Both training and testing are using batch_size=1.
This can have the effect of faster learning, but also adds instability to the learning process as the weights widely vary with each batch.

Solution 2 - batch size = n
Make all the predictions at once in a batch.
But later you need to use all predictions made at once, or only keep the first prediction and discard the rest.

Solution 3 - different batch size for training & testing
The better solution
majorly is to copy the weights from the fit network and to create a new network with the pre-trained weights.
Can you write Python code for it?

# Solution 3
## Model 1 - 3 batches for training

### configure network
n_batch = 3
n_epoch = 1000
n_neurons = 10

### design network
model = Sequential()
model.add(LSTM(n_neurons, batch_input_shape=(n_batch, X.shape[1], X.shape[2]), stateful=True))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
### fit network
for i in range(n_epoch):
    model.fit(X, y, epochs=1, batch_size=n_batch, verbose=1, shuffle=False)
    model.reset_states() # manually set internal state update after epoch, otherwise state will reset after each batch

## Model 2 - 1 batch for prediction

### re-define the batch size
n_batch = 1

### re-define model
new_model = Sequential()
new_model.add(LSTM(n_neurons, batch_input_shape=(n_batch, X.shape[1], X.shape[2]), stateful=True))
new_model.add(Dense(1))

### copy weights
old_weights = model.get_weights()
new_model.set_weights(old_weights)

### compile model
new_model.compile(loss='mean_squared_error', optimizer='adam')

# Use original data to check the prediction (NOT suggested)
for i in range(len(X)):
    testX, testy = X[i], y[i]
    testX = testX.reshape(1, 1, 1)
    yhat = new_model.predict(testX, batch_size=n_batch)
    print('>Expected=%.1f, Predicted=%.1f' % (testy, yhat))

# Try a new set of data but has the same values as original data
length = 5
sequence = [i/10.0 for i in range(length)]
print sequence

## create X/y pairs
df = pd.DataFrame(sequence)
df = pd.concat([df, df.shift(1)], axis=1)  # add second column which is the first column shift 1 period
df.dropna(inplace=True)
df

values = df.values
X, y = values[:, 0], values[:, 1]
X = X.reshape(len(X), 1, 1)  # LSTM needs (number of records, timesteps, number of features)
print(X.shape, y.shape)

# predict
for i in range(len(X)):
    testX, testy = X[i], y[i]
    testX = testX.reshape(1, 1, 1)
    yhat = new_model.predict(testX, batch_size=n_batch)
    print('>Expected=%.1f, Predicted=%.1f' % (testy, yhat))

# Try a new set of data but has the different values as original data
length = 20
sequence =...


I am working on the file "AI_Experiments/after_2020/test_local_GPU.ipynb".
The first task is:
CPU

Note: the warning below doesn't mean you are using GPU. In fact it's using CPU here since I'm tracking the Windows Task Manager's "Performance".
To track whether there is really GPU in use, need to make sure tf.config.experimental.list_physical_devices('GPU') has at least 1 GPU available.
Can you write Python code for it?

# Stacked LSTMs
device_name = '/cpu:0'
with tf.device(device_name):
    model_name = 'stacking_lstm_model_tanh_cpu'
    model = stack_models(X_train, device_name)
    history = fit_diy_model(model, X_train, y_train, X_val, y_val, model_name)
    evaluate_diy_model(model_name, history, X_val, df_val)


I am working on the file "AI_Experiments/after_2020/test_local_GPU.ipynb".
The first task is:
GPU

The GPU utilization for this example is very small (less than 10% utilization) even after I have increased the batch size from 16 to 3000
People suggest to increase the batch size when GPU utilization is low....
<b>GPU does't always work better than CPU.</b> As you can see from the the results below, CPU is much faster than GPU.
Because the overhead of invoking GPU kernels, and copying data to and from GPU, is very high. For operations on models with very little parameters it is not worth of using GPU
I was tracking Windows "Task Manage", when it's using GPU, the utilization of CPU is higher than GPU utilization, might be caused by GPU related operations. (When it's using CPU, the utilization of CPU is much higher, 0% utilization of GPU)
I have also tried CuDNNLSTM to replace lstm for GPU, since it's faster, but it's still underdeveopped
"dropout" and "activation" are not supported in CuDNNLSTM for now....😔😔
Somehow it's compalining the data didn't have initial state. I'm wondering whether it's because my current CUDA server is the latest (released on Dec. 2020) but the matched cuDNN hasn't come out... (released on Nov. 2020)
Can you write Python code for it?

# Stacked LSTMs
device_name = '/gpu:0'
with tf.device(device_name):
    model_name = 'stacking_lstm_model_tanh_gpu'
    model = stack_models(X_train, device_name)
    history = fit_diy_model(model, X_train, y_train, X_val, y_val, model_name)
    evaluate_diy_model(model_name, history, X_val, df_val)


I am working on the file "AI_Experiments/after_2020/test_local_GPU.ipynb".
The first task is:
GPU vs CPU
Data Preprocessing

Data is downloaded from https://archive.ics.uci.edu/ml/datasets/Beijing+PM2.5+Data
Can you write Python code for it?

df = pd.read_csv('../time_series/data/pm25.csv')

print(df.shape)
df.head()

df.dropna(subset=['pm2.5'], axis=0, inplace=True)
df.reset_index(drop=True, inplace=True)

df['datetime'] = df[['year', 'month', 'day', 'hour']].apply(
    lambda row: datetime.datetime(year=row['year'],
                                  month=row['month'], day=row['day'],hour=row['hour']), axis=1)
df.sort_values('datetime', ascending=True, inplace=True)

scaler = MinMaxScaler(feature_range=(0, 1))
df['scaled_pm2.5'] = scaler.fit_transform(np.array(df['pm2.5']).reshape(-1, 1))

split_date = datetime.datetime(year=2014, month=1, day=1, hour=0)
df_train = df.loc[df['datetime']<split_date]
df_val = df.loc[df['datetime']>=split_date]
df_val.reset_index(drop=True, inplace=True)
print('Shape of train:', df_train.shape)
print('Shape of test:', df_val.shape)

def makeXy(ts, nb_timesteps):
    """
    Input:
           ts: original time series
           nb_timesteps: number of time steps in the regressors
    Output:
           X: 2-D array of regressors
           y: 1-D array of target
    """
    X = []
    y = []
    for i in range(nb_timesteps, ts.shape[0]):
        X.append(list(ts.loc[i-nb_timesteps:i-1]))
        y.append(ts.loc[i])

    X, y = np.array(X), np.array(y)
    return X, y

X_train, y_train = makeXy(df_train['scaled_pm2.5'], 7)
X_val, y_val = makeXy(df_val['scaled_pm2.5'], 7)

print('Shape of train arrays:', X_train.shape, y_train.shape)
print('Shape of validation arrays:', X_val.shape, y_val.shape)

X_train = X_train.reshape((X_train.shape[0], X_train.shape[1], 1))
X_val = X_val.reshape((X_val.shape[0], X_val.shape[1], 1))
print('Shape of arrays after reshaping:', X_train.shape, X_val.shape)


I am working on the file "AI_ML/TimeSeriesWorkShop.ipynb".
The first task is:
A simple example to discuss the cons of Regression model in predicting Time Series
Can you write Python code for it?


I am working on the file "AI_ML/TimeSeriesWorkShop.ipynb".
The first task is:
AD Fuller Test for Stationarity
if p-value <0.05, the series stationary
else it is not stationary
Can you write Python code for it?

pd.options.display.float_format = '{:,.5f}'.format
print('Statitionarity test for original series')
print('_________________________________________')
test_stationarity(values)
print('')
print('')
print('Statitionarity test for difference series')
print('_________________________________________')
test_stationarity(value_d1)

# pip install scipy==1.2 --upgrade
import statsmodels.api as sm
from statsmodels.tsa.stattools import acf, pacf
fig = plt.figure(figsize=(12,8))
ax1 = fig.add_subplot(211)
fig = sm.graphics.tsa.plot_acf(value_d1, lags=40, ax=ax1)
ax2 = fig.add_subplot(212)
fig = sm.graphics.tsa.plot_pacf(value_d1, lags=40, ax=ax2)


I am working on the file "AI_ML/TimeSeriesWorkShop.ipynb".
The first task is:
ARIMA
ARIMA stands for Auto-Regressive Integrated Moving Average. There are three integers (p, d, q) that are used to parametrize ARIMA models. Because of that, a nonseasonal ARIMA model is denoted with ARIMA(p, d, q):
p  is the number of autoregressive terms (AR part). It allows to incorporate the effect of past values into our model. Intuitively, this would be similar to stating that it is likely to be warm tomorrow if it has been warm the past 3 days.
d is the number of nonseasonal differences needed for stationarity. Intuitively, this would be similar to stating that it is likely to be same temperature tomorrow if the difference in temperature in the last three days has been very small.
q is the number of lagged forecast errors in the prediction equation (MA part). This allows us to set the error of our model as a linear combination of the error values observed at previous time points in the past.
When dealing with seasonal effects, as in our example, seasonal ARIMA is used, which is denoted as ARIMA(p,d,q)(P,D,Q)s. Here, (p, d, q) are the nonseasonal parameters described above, (P, D, Q) follow the same definition but are applied to the seasonal component of the time series. The term s is the periodicity of the time series.
Can you write Python code for it?

cutoff=30
model = ARIMA(pd.Series(values)[:cutoff], order=(1, 2, 0))
# model = ARIMA(pd.Series(value_d1)[:cutoff], order=(1, 1, 0))
results_ARIMA = model.fit()
print(results_ARIMA.summary())

results_ARIMA.plot_predict(dynamic=False)
plt.show()

# plt.plot(value_d1, label='original')
# arima_pred_recent = results_ARIMA.predict(pd.Series(value_d1).index[10], pd.Series(value_d1).index[-1])
# plt.plot(arima_pred_recent, label='Prediction')

# https://machinelearningmastery.com/make-sample-forecasts-arima-python/
arima_pred_recent = results_ARIMA.predict(pd.Series(values).index[cutoff], pd.Series(values).index[-1])
plt.figure(figsize=(20,10))
plt.plot(values[:cutoff+25], label='Original', color = "grey")
prediction=[]
for i in range(cutoff):
  prediction.append(0)
prediction1=prediction[:]
prediction1[i]=values[cutoff+1]

for i in range(cutoff,len(values)):
  prediction1.append(prediction1[i-1]+arima_pred_recent[i])

plt.plot(prediction1[:cutoff+25], label='Prediction', color = "red", linestyle='dashed')
plt.legend(loc=1)

df=pd.DataFrame(Air,columns=['Month','Traffic'])
plt.plot(df['Traffic'], label='Original')

import statsmodels.api as sm
from statsmodels.tsa.stattools import acf, pacf
fig = plt.figure(figsize=(12,8))
ax1 = fig.add_subplot(211)
fig = sm.graphics.tsa.plot_acf(df['Traffic'].tolist(), lags=10, ax=ax1)
ax2 = fig.add_subplot(212)
fig = sm.graphics.tsa.plot_pacf(df['Traffic'].tolist(), lags=10, ax=ax2)

from statsmodels.tsa.arima_model import ARIMA
cutoff=47+12
model = ARIMA(df['Traffic'][:cutoff], order=(12, 1, 0))
# model = ARIMA(pd.Series(value_d1), order=(1, 1, 0))
results_ARIMA = model.fit()
print(results_ARIMA.summary())

# https://machinelearningmastery.com/make-sample-forecasts-arima-python/

arima_pred_recent = results_ARIMA.predict(df['Traffic'].index[cutoff], df['Traffic'].index[-1])
plt.figure(figsize=(20,5))
plt.plot(df['Traffic'], label='Original')
prediction=[]
for i in range(cutoff):
  prediction.append([df['Traffic'][i]])
#   print(prediction[i-1])
for i in...


I am working on the file "AI_ML/TimeSeriesWorkShop.ipynb".
The first task is:
From difference chart the following notions can be arrived

the absolute difference is constant during most part
the direction of difference oscillate

Zt is a stationary series that demonstrates 25 is the difference between points
Henceforth Predicted Yt = Yt-1 +/- 25
Henceforth, let us predict by adding and subtracting the difference alternative to the initial points and predict next points
Can you write Python code for it?

predicted=[]
for t in time:
    idx=t-1
    if t<=5:
        predicted.append(values[idx])
    else:
        value=values[idx-1]-25 if values[idx-1]-values[idx-2]>0 else values[idx-1]+25
        predicted.append(value)

plt.figure(figsize=(20,5))
plotrange=[1,50]
plt.plot(time[plotrange[0]:plotrange[1]], values[plotrange[0]:plotrange[1]], color = "grey", marker = "o")
plt.plot(time[plotrange[0]:plotrange[1]], predicted[plotrange[0]:plotrange[1]], color = "red", marker = "v", linestyle='dashed')
plt.show()

math.sqrt(sum([(predicted[i]-values[i])**2 for i in range(len(values))])/len(values))


I am working on the file "AI_ML/TimeSeriesWorkShop.ipynb".
The first task is:
Notice the difference plot.  Is that stationary?
Can we predict the next value in time series with the help of difference value?
From difference chart the following notions can be arrived

the absolute difference is constant during most part
the direction of difference oscillate

Let us predict the timeseries with regression function and compare later with Arima models
Can you write Python code for it?


I am working on the file "AI_ML/TimeSeriesWorkShop.ipynb".
The first task is:
Regression Function
Can you write Python code for it?

def estimate_coef(x, y):
    # number of observations/points
    n = np.size(x)

    # mean of x and y vector
    m_x, m_y = np.mean(x), np.mean(y)

    # calculating cross-deviation and deviation about x
    SS_xy = np.sum(y*x) - n*m_y*m_x
    SS_xx = np.sum(x*x) - n*m_x*m_x

    # calculating regression coefficients
    b_1 = SS_xy / SS_xx
    b_0 = m_y - b_1*m_x

    return(b_0, b_1)

def plot_regression_line(x, y, b):

    fig= plt.figure(figsize=(20,5))

    # plotting the actual points as scatter plot
    plt.plot(x, y, color = "m", marker = "o")

    # predicted response vector
    y_pred = b[0] + b[1]*x
    plt.plot(x, y_pred, color = "r", marker = "o")
#     # plotting the regression line
#     plt.plot(x, y_pred, color = "g")

    # putting labels
    plt.xlabel('Time')
    plt.ylabel('Value')
    plt.xticks(np.arange(min(x), max(x)+1, 1.0))

    # function to show plot
    plt.show()

def main(time,values):
    # estimating coefficients
    b = estimate_coef(time, values)
    print("Estimated coefficients: c = {:+.2f}  m = {:+.2f}".format(b[0], b[1]))
    print("\n")
    print("Estimate for next period ={:+.2f}".format(b[1]*20 + b[0]))
    print("\n")
    y_pred = b[0] + b[1]*time
    print("RMSE ={:+.10f}".format(math.sqrt(sum(((b[0] + b[1]*time)-values)**2)/(len(time)))))
    print("\n\n")

    # plotting regression line
    plot_regression_line(time, values, b)


I am working on the file "AI_ML/TimeSeriesWorkShop.ipynb".
The first task is:
The RMSE improved drastically.  There is still scope to improve.  Let us try to compensate previous errors in the current prediction
Can you write Python code for it?

predicted=[]
for t in time:
    idx=t-1
    if t<=5:
        predicted.append(values[idx])
        error=0
    else:
        value=values[idx-1]-25 if values[idx-1]-values[idx-2]>0 else values[idx-1]+25
        predicted.append(value+0.5*error)
#         print(values[idx-1]-values[idx-2],values[idx],value+error,error)
        error=values[idx]-value

plt.figure(figsize=(20,5))
plotrange=[1,50]
plt.plot(time[plotrange[0]:plotrange[1]], values[plotrange[0]:plotrange[1]], color = "grey", marker = "o")
plt.plot(time[plotrange[0]:plotrange[1]], predicted[plotrange[0]:plotrange[1]], color = "red", marker = "v", linestyle='dashed')
plt.show()

math.sqrt(sum([(predicted[i]-values[i])**2 for i in range(len(values))])/len(values))


I am working on the file "AI_ML/TimeSeriesWorkShop.ipynb".
The first task is:
The point is becoming clearer that the If we predict Stationary Series zt, the result could be more encouraging
In reality, the stationary distribution may not oscillate as we got in our cooked up example.
Let us see how we can ensure the series is stationary
Can you write Python code for it?

def test_stationarity(ts):
    stats = ['Test Statistic','p-value','Lags','Observations']
    df_test = adfuller(ts, autolag='AIC')
    df_results = pd.Series(df_test[0:4], index=stats)
    for key,value in df_test[4].items():
        df_results['Critical Value (%s)'%key] = value
    print(df_results)


I am working on the file "AI_ML/TimeSeriesWorkShop.ipynb".
The first task is:
When dealing with seasonal effects, as in our example, seasonal ARIMA is used, which is denoted as ARIMA(p,d,q)(P,D,Q)s. Here, (p, d, q) are the nonseasonal parameters described above, (P,D, Q) follow the same definition but are applied to the seasonal component of the time series. The term s is the periodicity of the time series.
Can you write Python code for it?

train_data = df[:70]
test_data = df[70:]

# https://github.com/gmonaci/ARIMA/blob/master/time-series-analysis-ARIMA.ipynb
model = sm.tsa.statespace.SARIMAX(train_data['Traffic'],order=(2, 1, 0),seasonal_order=(1, 1, 0, 12),enforce_stationarity=False,enforce_invertibility=False)
results=model.fit()
ax = df.plot(figsize=(20, 5))
pred0=results.get_prediction(start=60, dynamic=False)
pred0.predicted_mean.plot(ax=ax)
pred1 = results.get_forecast(70)
pred1.predicted_mean.plot(ax=ax)
plt.show()

decomposition = seasonal_decompose(df['Traffic'], freq = 12)

trend = decomposition.trend
seasonal = decomposition.seasonal
residual = decomposition.resid

plt.subplot(411)
plt.plot(df['Traffic'], label='Original')
plt.legend(loc='best')
plt.subplot(412)
plt.plot(trend, label='Trend')
plt.legend(loc='best')
plt.subplot(413)
plt.plot(seasonal,label='Seasonality')
plt.legend(loc='best')
plt.subplot(414)
plt.plot(residual,...


I am working on the file "AMF.ipynb".
The first task is:
As shown from the above print, AMF results in almost twice as higher deviation than the native median filter technique.
Can you write Python code for it?

### Thereofore, the built-in technique is nowhere as good as compared to the Adaptive Median Filter technique.


I am working on the file "AMF.ipynb".
The first task is:
Original Image (converted to grayscale)
Can you write Python code for it?


I am working on the file "AMF.ipynb".
The first task is:
Output with Python's native Median Filter function
Can you write Python code for it?

native_output = image_org.filter(ImageFilter.MedianFilter(size = 3))
native_output

deviation_native = np.sqrt(np.sum(np.square(grayscale_image-np.array(rgb2gray(np.array(native_output))))))

deviation_original = np.sum(np.square(grayscale_image-np.array(output)))

print("Deviation from the original salt and pepper images:")
print("Deviation via Median Filter (built-in): ", deviation_native)
print("Deviation via Adaptive Median Filter: ", deviation_original)
print(f"Percent difference b/w deviations: {100*(deviation_original - deviation_native)/deviation_original}%")


I am working on the file "ANTLR4-Python/04-Calc/Parser.ipynb".
The first task is:
Now we can parse a file of simple statements.  The function parser_file takes the string data as its argument and
executes the statements in this string by calling the method visit of the visitor we have defined.
Can you write Python code for it?

def parse_data(data):
    input_stream = antlr4.InputStream(data)
    lexer        = ExprLexer(input_stream)
    token_stream = antlr4.CommonTokenStream(lexer)
    parser       = ExprParser(token_stream)
    tree         = parser.prog()
    visitor      = MyVisitor()
    visitor.visit(tree)

parse_data(data)

!rm Expr.interp Expr.tokens ExprLexer.* ExprListener.py ExprParser.py ExprVisitor.py


I am working on the file "ANTLR4-Python/04-Calc/Parser.ipynb".
The first task is:
Our grammar is stored in the file Expr.g4.  Let's have a look:
Can you write Python code for it?


I am working on the file "ANTLR4-Python/04-Calc/Parser.ipynb".
The first task is:
The files ExprLexer.py and ExprParser.py contain the generated scanner and parser, respectively.
Additionally we also have the file ExprVisitor which contains a generic visitor for our grammar.
We have to import these files.
Can you write Python code for it?

from ExprLexer   import ExprLexer
from ExprParser  import ExprParser
from ExprVisitor import ExprVisitor


I am working on the file "ANTLR4-Python/AST-2-Dot.ipynb".
The first task is:
The function create_nodes takes three arguments:
- dot is an object of class graphviz.digraph,
- t is an abstract syntax tree represented as a nested tuple.
- Nodes_2_Namesis a dictionary mapping nodes in t to unique names
   that can be used as node names in graphviz.
The function creates the nodes in t and connects them via directed edges so that t is represented as a tree.
Can you write Python code for it?

def create_nodes(dot, a, t, Nodes_2_Names):
    root = Nodes_2_Names[a]
    if isinstance(t, str) or isinstance(t, int):
        dot.node(root, label=str(t))
        return
    dot.node(root, label=t[0])
    j = 1
    for c in t[1:]:
        child = Nodes_2_Names[a + (j,)]
        dot.edge(root, child)
        create_nodes(dot, a + (j,), c, Nodes_2_Names)
        j += 1


I am working on the file "ANTLR4-Python/Calculator/Calculator.ipynb".
The first task is:
First, we have to generate both the scanner and the parser.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/Calculator/Calculator.ipynb".
The first task is:
Let us parse and evaluate the input that we read from a prompt.
Can you write Python code for it?

def main():
    parser        = CalculatorParser(None) # generate parser without lexer
    parser.Values = {}
    line          = input('> ')
    while line != '':
        input_stream  = antlr4.InputStream(line)
        lexer         = CalculatorLexer(input_stream)
        token_stream  = antlr4.CommonTokenStream(lexer)
        parser.setInputStream(token_stream)
        parser.start()
        line = input('Input an arithmetic expression: ')
    return parser.Values

main()


def main():
    parser        = CalculatorParser(None) # generate parser without lexer
    parser.Values = {}
    line          = input('> ')
    while line != '':
        input_stream  = antlr4.InputStream(line)
        lexer         = CalculatorLexer(input_stream)
        token_stream  = antlr4.CommonTokenStream(lexer)
        parser.setInputStream(token_stream)
        parser.start()
        line = input('> ')
    return parser.Values

main()

!rm *.py *.tokens *.interp
!rm -r __pycache__/

!ls -l


I am working on the file "ANTLR4-Python/Calculator/Calculator.ipynb".
The first task is:
The files CalculatorLexer.py and CalculatorParser.py contain the generated scanner and parser, respectively.  We have to import these files.  Furthermore, the runtime of
Antlr
needs to be imported.
Can you write Python code for it?

from CalculatorLexer  import CalculatorLexer
from CalculatorParser import CalculatorParser
import antlr4


I am working on the file "ANTLR4-Python/Calculator/Calculator.ipynb".
The first task is:
The grammar shown above has no semantic actions (with the exception of the skip action).
We extend this grammar now with semantic actions so that we can actually compute something.
This grammar is stored in the file Calculator.g4.  It describes a language for a
symbolic calculator: This calculator is able to evaluate arithmetic expressions and, furthermore,
where we can store the results of our computations in variables.
Can you write Python code for it?

!cat -n Calculator.g4

!type Calculator.g4


I am working on the file "ANTLR4-Python/Calculator/Calculator.ipynb".
The first task is:
We can use the system command ls to see which files have been generated by Antlr.
If you are using a windows system you have to use the command dir instead.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/Differentiator/Differentiator.ipynb".
The first task is:
The files CalculatorLexer.py and CalculatorParser.py contain the generated scanner and parser, respectively.  We have to import these files.  Furthermore, the runtime of
Antlr
needs to be imported.
Can you write Python code for it?

from DifferentiatorLexer  import DifferentiatorLexer
from DifferentiatorParser import DifferentiatorParser
import antlr4

%run ../AST-2-Dot.ipynb


I am working on the file "ANTLR4-Python/Differentiator/Differentiator.ipynb".
The first task is:
The function diff takes the parse tree e of an arithmetic expression and differentiate this expressions e with respect to the variable x.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/Differentiator/Differentiator.ipynb".
The first task is:
The function main prompts for an expression that is then parsed and differentiated with respect to the variable x.
Can you write Python code for it?

def main():
    line = input('> ')
    while line != '':
        input_stream  = antlr4.InputStream(line)
        lexer         = DifferentiatorLexer(input_stream)
        token_stream  = antlr4.CommonTokenStream(lexer)
        parser        = DifferentiatorParser(token_stream)
        context       = parser.expr()
        term          = context.result
        display(tuple2dot(term))
        derivative    = diff(term)
        print(toString(derivative))
        line = input('> ')


I am working on the file "ANTLR4-Python/Differentiator/Differentiator.ipynb".
The first task is:
The function toString takes an arithmetical expression that is represented as a nested tuple and converts it into a string.
Can you write Python code for it?

def toString(e):
    if isinstance(e, int):
        return str(e)
    if e[0] == '+':
        f, g = e[1:]
        return toString(f) + ' + ' + toString(g)
    if e[0] == '-':
        f, g = e[1:]
        return toString(f) + ' - (' + toString(g) + ')'
    if e[0] == '*':
        f, g = e[1:]
        return parenString(f) + ' * ' + parenString(g)
    if e[0] == '/':
        f, g = e[1:]
        return parenString(f) + ' / (' + toString(g) + ')'
    if e[0] == 'ln':
        return 'ln(' + toString(e[1]) + ')'
    if e[0] == 'exp':
        return 'exp(' + toString(e[1]) + ')'
    return str(e)


I am working on the file "ANTLR4-Python/Differentiator/Differentiator.ipynb".
The first task is:
We start by generating both scanner and parser.
Can you write Python code for it?


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
Earley's Algorithm
Given a context-free grammar $G = \langle V, \Sigma, R, S \rangle$ and a string $s = x_1x_2 \cdots x_n \in \Sigma^$ of length $n$,
an Earley item* is a pair of the form
$$\langle A \rightarrow \alpha \bullet \beta, k \rangle$$
such that
- $(A \rightarrow \alpha \beta) \in R\quad$  and
- $k \in {0,1,\cdots,n}$.
The class EarleyItem represents a single Earley item.
- mVariable is the variable $A$,
- mAlpha is $\alpha$,
- mBeta is $\beta$, and
- mIndex is $k$.
Since we later have to store objects of class EarleyItem in sets, we have to implement the functions
- __eq__,
- __ne__,
- __hash__.
It is easiest to implement __hash__ by first converting the object into a string.  Hence we also
implement the function __repr__, that converts an EarleyItem into a string.
Can you write Python code for it?

class EarleyItem():
    def __init__(self, variable, alpha, beta, index):
        self.mVariable = variable
        self.mAlpha    = alpha
        self.mBeta     = beta
        self.mIndex    = index

    def __eq__(self, other):
        return isinstance(other, EarleyItem)     and \
               self.mVariable == other.mVariable and \
               self.mAlpha    == other.mAlpha    and \
               self.mBeta     == other.mBeta     and \
               self.mIndex    == other.mIndex

    def __ne__(self, other):
        return not self.__eq__(other)

    def __hash__(self):
        return hash(self.__repr__())

    def __repr__(self):
        alphaStr = ' '.join(self.mAlpha)
        betaStr  = ' '.join(self.mBeta)
        return f'<{self.mVariable} → {alphaStr} • {betaStr}, {self.mIndex}>'


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
Given an Earley item self, the function isComplete checks, whether the Earley item self has the form
$$\langle A \rightarrow \alpha \bullet, k \rangle,$$
i.e. whether the $\bullet$ is at the end of the grammar rule.
Can you write Python code for it?

def isComplete(self):
    return self.mBeta == ()

EarleyItem.isComplete = isComplete
del isComplete


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
Given an Earley item, this function returns the name of the variable following the dot.  If there is no variable following the dot, the function returns None.  The function can distinguish variables from token names because variable names consist only of lower case letters.
Can you write Python code for it?

def nextVar(self):
    if len(self.mBeta) > 0:
        var = self.mBeta[0]
        if var[0] != "'" and var.islower():
            return var
    return None

EarleyItem.nextVar = nextVar
del nextVar


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The class EarleyParser implements the parsing algorithm of Jay Earley.
The class maintains the following member variables:
- mGrammar is the grammar that is used to parse the given token string.
- mString is the list of tokens and literals that has to be parsed.
As a hack, the first element of this list in None.
   Therefore, mString[i] is the $i^\textrm{th}$ token.
- mStateList is a list of sets of Earley items.  If $n$ is the length of the given token string
  (excluding the first element None), then $Q_i = \texttt{mStateList}[i]$.
  The idea is that the set $Q_i$ is the set of those Earley items that the parser could be in
  when it has read the tokens mString[1], $\cdots$,  mString[n].  $Q_0$ is initialized as follows:
  $$ Q_0 = \bigl{\langle\hat{S} \rightarrow \bullet S, 0\rangle\bigr}. $$
The Earley items are interpreted as follows: If we have
$$ \langle C \rightarrow \alpha \bullet \beta, k\rangle \in Q_i, $$
then we know the following:
- After having read the tokens mString[:k+1] the parser tries to parse the variable $C$
  in the token string mString[k+1:].
- After having read the token string mString[k+1:i+1] the parser has already recognized $\alpha$
  and now needs to recognize $\beta$ in the token string mString[i+1:] in order to parse the variable $C$.
Can you write Python code for it?

class EarleyParser():
    def __init__(self, grammar, TokenList):
        self.mGrammar   = grammar
        self.mString    = [None] + TokenList  # hack so mString[1] is the first token
        self.mStateList = [set() for i in range(len(TokenList)+1)]
        print('Grammar:\n')
        print(self.mGrammar)
        print(f'Input: {self.mString}\n')
        self.mStateList[0] = { self.mGrammar.startItem() }


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The class EarleyParser implements the parsing algorithm of Jay Earley.
The class maintains the following member variables:
- mGrammar is the grammar that is used to parse the given token string.
- mString is the list of tokens and literals that has to be parsed.
As a hack, the first element of this list in None.
   Therefore, mString[i] is the ith token.
- mStateList is a list of sets of Earley items.  If $n$ is the length of the given token string
  (excluding the first element None), then $Q_i = \texttt{mStateList}[i]$.
  The idea is that the set $Q_i$ is the set of those Earley items that the parser could be in
  when it has read the tokens mString[1], $\cdots$,  mString[n].  $Q_0$ is initialized as follows:
  $$ Q_0 = \bigl{\langle\hat{S} \rightarrow \bullet S, 0\rangle\bigr}. $$
The Earley items are interpreted as follows: If we have
$$ \langle C \rightarrow \alpha \bullet \beta, k\rangle \in Q_i, $$
then we know the following:
- After having read the tokens mString[:k+1] the parser tries to parse the variable $C$
  in the token string mString[k+1:].
- After having read the token string mString[k+1:i+1] the parser has already recognized $\alpha$
  and now needs to recognize $\beta$ in the token string mString[i+1:] in order to parse the variable $C$.
Can you write Python code for it?

class EarleyParser():
    def __init__(self, grammar, TokenList):
        self.mGrammar   = grammar
        self.mString    = [None] + TokenList  # dirty hack so mString[1] is first token
        self.mStateList = [set() for i in range(len(TokenList)+1)]
        print('Grammar:\n')
        print(self.mGrammar)
        print(f'Input: {self.mString}\n')
        self.mStateList[0] = { self.mGrammar.startItem() }


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The class Grammar represents a context free grammar.  It stores a list of the rules of the grammar.
Each grammar rule of the form
$$ a \rightarrow \beta $$
is stored as the tuple $(a,) + \beta$.  The start symbol is assumed to be the variable on the left hand side of
the first rule. To distinguish syntactical variables form tokens, variables contain only lower case letters,
while tokens either contain only upper case letters or they start and end with a single quote character "'".
Can you write Python code for it?

class Grammar():
    def __init__(self, Rules):
        self.mRules = Rules


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The command below cleans the directory.
Can you write Python code for it?

!del GrammarLexer.* GrammarParser.* Grammar.tokens GrammarListener.py Grammar.interp
!rmdir /S /Q __pycache__

!dir /B

!rm GrammarLexer.* GrammarParser.* Grammar.tokens GrammarListener.py Grammar.interp
!rm -r __pycache__

!ls


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The command below cleans the directory.  If you are running windows, you have to replace rmwith del.
Can you write Python code for it?

!rm GrammarLexer.* GrammarParser.* Grammar.tokens GrammarListener.py Grammar.interp
!rm -r __pycache__

!ls


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The function finishItem returns the Earley item
$$ \langle\hat{S} \rightarrow S \bullet, 0\rangle $$
where $S$ is the start variable of the given grammar and $\hat{S}$ is a new variable.
Can you write Python code for it?

def finishItem(self):
    return EarleyItem('Start', (self.startVar(),), (), 0)

Grammar.finishItem = finishItem
del finishItem


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The function moveDot(self) moves the $\bullet$ in the Earley item self, where self has the form
$$\langle A \rightarrow \alpha \bullet \beta, k \rangle$$
over the next variable, token, or literal in $\beta$.  It assumes that $\beta$ is not empty.
Can you write Python code for it?

def moveDot(self):
    return EarleyItem(self.mVariable,
                      self.mAlpha + (self.mBeta[0],),
                      self.mBeta[1:],
                      self.mIndex)

EarleyItem.moveDot = moveDot
del moveDot


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The function parse_grammar takes a filename as its argument and returns the grammar that is stored in the given file.  The grammar is represented as list of rules.  Each rule is represented as a tuple.  The example below will clarify this structure.
Can you write Python code for it?

def parse_grammar(filename):
    input_stream  = antlr4.FileStream(filename)
    lexer         = GrammarLexer(input_stream)
    token_stream  = antlr4.CommonTokenStream(lexer)
    parser        = GrammarParser(token_stream)
    grammar       = parser.start()
    return grammar.g

parse_grammar('simple.g')


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The function sameVar(self, C) checks, whether the item following the dot is the same as the variable
given as argument, i.e. sameVar(self, C) returns True if self is an Earley item of the form
$$\langle A \rightarrow \alpha \bullet C\beta, k \rangle.$$
Can you write Python code for it?

def sameVar(self, C):
    return len(self.mBeta) > 0 and self.mBeta[0] == C

EarleyItem.sameVar = sameVar
del sameVar


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The function scan(self, t) checks, whether the item following the dot matches the token t,
i.e. scan(self, t) returns True if self is an Earley item of the form
$$\langle A \rightarrow \alpha \bullet t\beta, k \rangle.$$
The argument $t$ can either be the name of a token or a literal.
Can you write Python code for it?

def scan(self, t):
    if len(self.mBeta) > 0:
        return self.mBeta[0] == t or self.mBeta[0] == "'" + t + "'"
    return False

EarleyItem.scan = scan
del scan


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The function self.scan(i) applies the scanning operation to the state $Q_i$.
If $\langle A \rightarrow \beta \bullet a \gamma, k\rangle \in Q_i$ and $a$ is a token,
then the parser tries to recognize the right hand side of the grammar rule
$$ A \rightarrow \beta a \gamma$$
and after having read mString[k+1:i+1] it has already recognized  $\beta$.
If we now have mString[i+1] == a, then the parser still has to recognize $\gamma$ in mString[i+2:].
Therefore, the Earley object $\langle A \rightarrow \beta a \bullet \gamma, k\rangle$ is added to
the set $Q_{i+1}$:
$$\langle A \rightarrow \beta \bullet a \gamma, k\rangle \in Q_i \wedge x_{i+1} = a
       \;\rightarrow\;
       Q_{i+1} := Q_{i+1} \cup \bigl{ \langle A \rightarrow \beta a \bullet \gamma, k\rangle \bigr}
$$
Can you write Python code for it?

def scan(self, i):
    Qi = self.mStateList[i]
    n  = len(self.mString) - 1 # remember mStateList[0] == None
    if i + 1 <= n:
        a = self.mString[i+1]
        for item in Qi:
            if item.scan(a):
                self.mStateList[i+1].add(item.moveDot())
                print('scanning:')
                print(f'{item.moveDot()} added to Q{i+1}')

EarleyParser.scan = scan
del scan

import re


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The function startItem returns the Earley item
$$ \langle\hat{S} \rightarrow \bullet S, 0\rangle $$
where $S$ is the start variable of the given grammar and $\hat{S}$ is a new variable.
Can you write Python code for it?

def startItem(self):
    return EarleyItem('Start', (), (self.startVar(),), 0)

Grammar.startItem = startItem
del startItem


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The function startVar returns the start variable of the grammar.  It is assumed that
the first rule grammar starts with the start variable of the grammar.
Can you write Python code for it?

def startVar(self):
    return self.mRules[0][0]

Grammar.startVar = startVar
del startVar


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The function test takes two arguments.
- file is the name of a file containing a grammar,
- word is a string that should be parsed.
word is first tokenized.  Then the resulting token list is parsed using Earley's algorithm.
Can you write Python code for it?

def test(file, word):
    Rules     = parse_grammar(file)
    grammar   = Grammar(Rules)
    TokenList = tokenize(word)
    ep        = EarleyParser(grammar, TokenList)
    ep.parse()

test('simple.g', '1 + 2 * 3')


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The function toString creates a readable presentation of the grammar rules.
Can you write Python code for it?

def toString(self):
    result = ''
    for head, *body in self.mRules:
        result += f'{head}: {body};\n'
    return result

Grammar.__str__ = toString
del toString


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The function tokenize transforms the string s into a list of tokens. See below for an example.
Can you write Python code for it?

def tokenize(s):
    '''Transform the string s into a list of tokens.  The string s
       is supposed to represent an arithmetic expression.
    '''
    lexSpec = r'''([ \t]+)        |  # blanks and tabs
                  ([1-9][0-9]*|0) |  # number
                  ([()])          |  # parentheses
                  ([-+*/])        |  # arithmetical operators
                  (.)                # unrecognized character
               '''
    tokenList = re.findall(lexSpec, s, re.VERBOSE)
    result    = []
    for ws, number, parenthesis, operator, error in tokenList:
        if ws:        # skip blanks and tabs
            continue
        elif number:
            result += [ 'NUMBER' ]
        elif parenthesis:
            result += [ parenthesis ]
        elif operator:
            result += [ operator ]
        else:
            result += [ f'ERROR({error})']
    return result

tokenize('1 + 2 * 3')


I am working on the file "ANTLR4-Python/Earley-Parser/Earley-Parser.ipynb".
The first task is:
The method complete(self, i) applies the completion operation to the state $Q_i$:
If we have
- $\langle C \rightarrow \gamma \bullet, j\rangle \in Q_i$ and
- $\langle A \rightarrow \beta \bullet C \delta, k\rangle \in Q_j$,
then the parser tried to parse the variable $C$ after having read mString[:j+1]
and we know that
$$ C \Rightarrow^ \texttt{mString[j+1:i+1]}, $$
i.e. the parser has recognized $C$ after having read mString[j+1:i+1].
Therefore the parser should proceed to recognize $\delta$ in state $Q_i$.
Therefore we add the Earley item* $\langle A \rightarrow \beta C \bullet \delta,k\rangle$ to the set $Q_i$:
$$\langle C \rightarrow \gamma \bullet, j\rangle \in Q_i \wedge
  \langle A \rightarrow \beta \bullet C \delta, k\rangle \in Q_j \;\rightarrow\;
          Q_i := Q_i \cup \bigl{ \langle A \rightarrow \beta C \bullet \delta, k\rangle \bigr}
$$
Can you write Python code for it?

def complete(self, i):
    change = False
    added  = True
    Qi     = self.mStateList[i]
    while added:
        added = False
        newQi = set()
        for item in Qi:
            if item.isComplete():
                C  = item.mVariable
                j  = item.mIndex
                Qj = self.mStateList[j]
                for newItem in Qj:
                    if newItem.sameVar(C):
                        moved = newItem.moveDot()
                        newQi.add(moved)
        if not (newQi <= Qi):
            change = True
            added  = True
            print("completion:")
            for newItem in newQi:
                if newItem not in Qi:
                    print(f'{newItem} added to Q{i}')
            self.mStateList[i] |= newQi
            Qi = self.mStateList[i]
    return change

EarleyParser.complete = complete
del complete