I am working on the file "Photon_Diffusion/Photon_Diffusion.ipynb".
The first task is:

Calculation of the phosphor layer thickness of lanex regular given its areal density:

Can you write Python code for it?

Dcell = 55*10**-6
DPET = 175*10**-6
Dcell2 = 13*10**-6
rhocell = 1.44*10**3
rhoPET = 1.38*10**3
rhophos = 4.48*10**3
sigma = 70*10**-2

Dphos = (sigma - Dcell*rhocell - DPET*rhoPET - Dcell2*rhocell)/rhophos
print(Dphos*10**6)


I am working on the file "Photon_Diffusion/Photon_Diffusion.ipynb".
The first task is:

Define functions for photon density and photon current density

These functions were derived from Fick's laws of diffusion:
$$ \frac{\partial n(z,t)}{\partial t} = D \frac{\partial^2 n(z,t)}{\partial z^2}, \: \: \phi (z,t) = -D \frac{\partial n(z,t)}{\partial z}$$
where n(z,t) is the photon density, D is the diffusion constant $D=\lambda_s c /6$ with $\lambda_s$ corresponding to the mean photon scattering length, and $\phi (z,t)$ is the photon current density. The fluorescence of the lanex phosphor was modeled as the instantaneous generation of light within a infinitely thin segment within a rectangular slab. The slab thickness $L$ is taken to be small compared to the other dimensions so that the problem can be treated one-dimensionally. The initial condition is taken to be
$$ n(z,0) = \frac{N_0}{A} \delta (z) \: \: \mathrm{with} \: \: n(z,t)=0 \: \: \mathrm{for} \: t<0 $$
where $N_0$ is the number of photons generated and $A$ is the cross-sectional area of the slab. In other words, at $t=0$, $N_0$ photons are generated and modeled as a dirac delta function at $z=0$. Fick's laws are then solved with absorbing boundary conditions at the edges of the lanex:
$$ n(d,t)=0, \: n(-l,t)=0$$
where $z=d$ is the location of the CCD and $z=-l$ is the location of the top edge of the phosphor. This yields
$$ n(z,t) = \frac{N_0}{2 A \sqrt{\pi D t}} \sum\limits_{m=-\infty}^\infty \left[ e^{-(z-2mL)^2/4Dt} - e^{-(z+2mL-2d)^2/4Dt} \right] $$
The photon current density then follows by differentiation with respect to $z$.

Can you write Python code for it?

N0 = 10**6 #Number of photons emitted at t=0
lambdas = 2.85*10**-6 #Diffusion length in m
D = lambdas*c/6 #Diffusion constant
A = 100*10**-6*100*10**-6 #Area of segment in m^2
L = 81*10**-6 #Depth of lanex in m
l = 10.0*10**-6 #Distance from top lanex edge to segment in m
d = L-l #Distance from bottom lanex edge to segment

def n(z,t):
    '''Returns the photon density at position z and time t'''
    n0 = N0/(2*A*sqrt(pi*D*t))
    Sum = 0
    maxm = 10
    for m in range(-maxm,maxm+1):
        Sum += exp(-(z-2*m*(l+d))**2/(4*D*t))-exp(-(z+2*m*(l+d)-2*d)**2/(4*D*t))
    return n0*Sum

def particlecurrent(t):
    '''Returns the particle current (photons per second per meter^2) at the boundary z=d at time t'''
    Sum = 0
    maxm = 10
    for m in range(-maxm,maxm+1):
        am = d-2*m*L
        Sum += am*exp(-am**2/(4*D*t))
    return N0/(A*sqrt(4*pi*D*t**3))*Sum


I am working on the file "Photon_Diffusion/Photon_Diffusion.ipynb".
The first task is:

Integrate photon current density

The photon current density at $z=d$ is then integrated over large times and multiplied by the area $A$ to determine the total number of photons absorbed by the CCD. This is done numerically and analytically with the function defined below. A plot of the fraction of photons absorbed is plotted as a function of time to ensure that the integral converges.

Can you write Python code for it?

Nabs = A*quad(particlecurrent,0,400*10**-12)[0] #Total number of photons absorbed at the boundary z=d
print(Nabs/N0)

def F(t,maxm,distance):
    Sum1 = 0
    Sum2 = 0
    for m in range(-maxm,1):
        am = distance-2*m*L
        Sum1 += 1 - erf(am/sqrt(4*D*t))
    for m in range(1,maxm+1):
        am = distance-2*m*L
        Sum2 += 1 + erf(am/sqrt(4*D*t))
    return (Sum1 - Sum2)

FractionAbsArray = []
FractionAbsArrayAnalytic = []
tarray = []
for t in linspace(10**-12,50*10**-12,10000):
    tarray.append(t*10**12)
    #FractionAbsArray.append(A*quad(particlecurrent,0,t)[0]/N0)
    FractionAbsArrayAnalytic.append(F(t,100,d))

#Adjust figure size
plt.subplots(figsize=(12,6))

plt.plot(tarray,FractionAbsArrayAnalytic,linewidth=2)
plt.xlim(np.min(tarray),np.max(tarray))
plt.ylim(0,1.0)
plt.xlim(0,50)
plt.xlabel('time (ps)')
plt.ylabel('Fraction Absorbed at $z=d$')
#plt.semilogy()
plt.legend(loc=4)


I am working on the file "Photon_Diffusion/Photon_Diffusion.ipynb".
The first task is:

Plot photon current density

Photon current density is then calculated at $z=d$ and plotted below as a function of time.

Can you write Python code for it?

particlecurrentarray = []
tarray = []
for t in linspace(10**-15,50*10**-12,1000):
    tarray.append(t*10**12)
    particlecurrentarray.append(particlecurrent(t))

#Update the matplotlib configuration parameters
mpl.rcParams.update({'font.size': 18, 'font.family': 'serif'})

#Adjust figure size
plt.subplots(figsize=(12,6))

plt.plot(tarray,particlecurrentarray,linewidth=2)
plt.xlim(np.min(tarray),np.max(tarray))
plt.ylim(0)
plt.xlabel('time (ps)')
plt.ylabel('Photon Current at $z=d$ $(s^{-1} \cdot m^{-2})$')
#plt.semilogy()
plt.legend(loc=4)


I am working on the file "Photon_Diffusion/Photon_Diffusion.ipynb".
The first task is:

Plot photon density

The function $n(z,t)$ is calculated from 1 fs to 10 ps and plotted below. The boundary conditions are visibly satisfied and the function approaches a dirac delta function for short times. It then spreads out with the total number of photons decreasing as they're absorbed at the boundaries.

Can you write Python code for it?

narray = []
zarray = np.linspace(-l,d,1000)
time = [1,10,10**2,10**3,10**4]
time = np.multiply(time,10**-15) #convert to s

for i in range(len(time)):
    narray.append([])
    for z in zarray:
        narray[i].append(n(z,time[i])*10**-6)

zarray = np.multiply(zarray,10**6)

#Update the matplotlib configuration parameters
mpl.rcParams.update({'font.size': 18, 'font.family': 'serif'})

#Adjust figure size
plt.subplots(figsize=(12,6))

color = ['r','g','b','c','m','y','k']
legend = []
for i in range(5):
    legend.append(str(int(time[i]*10**15))+' fs')
    plt.plot(zarray,narray[i],color=color[i],linewidth=2,label=legend[i])
plt.xlim(np.min(zarray),np.max(zarray))
plt.ylim(1.0*10**6,np.max(narray[0]))
plt.xlabel('Position (um)')
plt.ylabel('Photon Density (m^-3)')
#plt.semilogy()
plt.legend(loc=1)


I am working on the file "Physique.ipynb".
The first task is:
Find a Fundamental Constant you are interested in using the usual panda modules
Can you write Python code for it?

g_0pd = FPC[FPC["Quantity"].str.contains("gravity") ]
# standard acceleration of gravity as a panda DataFrame
print(g_0pd)

g_0 = g_0pd["Value"].values[0]
print(type(g_0))
print(g_0)

# access the values you're interested in
print(g_0pd.Quantity)
print(g_0pd.Value.get_values()[0])
print(g_0pd.Unit.get_values()[0])

# you can also grab just the 1 entry from this DataFrame using the .loc module
FPC[FPC["Quantity"].str.contains("Boltzmann")].loc[49,:]

g_0pd.loc[303,:]


I am working on the file "Physique.ipynb".
The first task is:
From the list of columns, search for the quantity you desired by trying out different search terms: e.g. I'm reading Huzel and Huang's Modern Engineering for Design of Liquid-Propellant Rocket Engines and I want to know how to convert from
* lb (pound or pound-force) for thrust into force in Newton (N)
* psia (pounds per square inch absolute) for (chamber) pressure into pressure in Pascal (Pa)
We can try to look up the U.S. or Imperial units from the Toconvertfrom column.
Can you write Python code for it?


I am working on the file "Physique.ipynb".
The first task is:
Look at what you want and see the index; it happens to be 340 in this example.
Can you write Python code for it?

lbf2N = Conversions.loc[340,:];
print(lbf2N)


I am working on the file "Physique.ipynb".
The first task is:
NIST Fundamental Constants
Can you write Python code for it?

from Physique import FundamentalPhysicalConstants as FPC

print(FPC.columns)

print(FPC)


I am working on the file "Physique.ipynb".
The first task is:
So for a chamber pressure of 3028 psia for the SSME,
Can you write Python code for it?

psi2Pa = Conversions.loc[372,:]
print(3028 * psi2Pa.Multiplyby, psi2Pa.to)


I am working on the file "Physique.ipynb".
The first task is:
So for example, the reusable SSME delivers a vacuum thrust of 470000 lb or
Can you write Python code for it?

print(470000 * lbf2N.Multiplyby, lbf2N.to)


I am working on the file "Physique.ipynb".
The first task is:
To obtain the conversion for pressure in psia, which we search for with "psi"
Can you write Python code for it?


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
Also, get the conversion for atmospheres (atm):
Can you write Python code for it?

convDF[convDF['Toconvertfrom'].str.match("atm")]

atm2Pa = convDF.loc[15,:]

print 3028*psi2Pa.Multiplyby/atm2Pa.Multiplyby, atm2Pa.Toconvertfrom


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
And so to access this, to use in Python, do the following, using .read_pickle of pandas:
Can you write Python code for it?


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
Find a Fundamental Constant you are interested in using the usual panda modules
Can you write Python code for it?

g_0pd = FundConst[ FundConst["Quantity"].str.contains("gravity") ]
# standard acceleration of gravity as a panda DataFrame
g_0pd

# access the values you're interested in
print g_0pd.Quantity
print g_0pd.Value.get_values()[0]
print g_0pd.Unit.get_values()[0]

# you can also grab just the 1 entry from this DataFrame using the .loc module
FundConst[FundConst["Quantity"].str.contains("Boltzmann")].loc[49,:]

g_0pd.loc[303,:]


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
From the list of columns, search for the quantity you desired by trying out different search terms: e.g. I'm reading Huzel and Huang's Modern Engineering for Design of Liquid-Propellant Rocket Engines and I want to know how to convert from
* lb (pound or pound-force) for thrust into force in Newton (N)
* psia (pounds per square inch absolute) for (chamber) pressure into pressure in Pascal (Pa)
We can try to look up the U.S. or Imperial units from the Toconvertfrom column.
Can you write Python code for it?


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
Get the current directory
Can you write Python code for it?


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
I'll add back the units as part of the data (I don't know a sane and civilized way of attaching to each of the column names in pandas, a pandas DataFrame, the units, as extra information)
Can you write Python code for it?

data = [hdrs[1],] + data

import pandas as pd
data = pd.DataFrame( data )
data.columns = hdrs[0]

data


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
Let's just focus on the Physical Data subpage for today.  This is the way to find a specific tag (in this case img) with a specific attribute (in this case alt="PHYSICAL DATA"), and then the parent module gets its parent.  Then the href index in the square brackets [] gets the web address we desire.
Can you write Python code for it?

jpl_ssd_BS.soup.find('img',{"alt":"PHYSICAL DATA"}).parent['href']

JPL_SSD_PHYS_DATA_URL = JPL_SSD_URL + jpl_ssd_BS.soup.find('img',{"alt":"PHYSICAL DATA"}).parent['href'][1:]
JPL_SSD_PHYS_DATA_URL

jpl_ssd_phys_data_BS = Physique.scrape_BS.scraped_BS(JPL_SSD_PHYS_DATA_URL)


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
NIST Official Conversions (to metric)
This is the pandas DataFrame containing all the NIST Official Conversions to SI.
Can you write Python code for it?


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
Or we can look up the SI unit we want to convert to.
Can you write Python code for it?


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
Programming note: __init__.py in the main directory uses os.path.dirname(__file__) with __file__ (literally that, it's not a placeholder name) being the string with the absolute pathname of the "file from which the module was loaded, if it was loaded from a file" (cf. stackoverflow  Python file attribute absolute or relative?), i.e. "When a module is loaded in Python, __file__ is set to its name. You can then use that with other functions to find the directory that the file is located in." (cf. stackoverflow  what does the file wildcard mean/do?)
NIST Fundamental Constants
Can you write Python code for it?

from Physique import FundConst
print Physique.FundConst.columns
Physique.FundConst


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
So for a chamber pressure of 3028 psia for the SSME,
Can you write Python code for it?

psi2Pa = convDF.loc[372,:]

print 3028*psi2Pa.Multiplyby, psi2Pa.to


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
So for example, the reusable SSME delivers a vacuum thrust of 470000 lb or
Can you write Python code for it?


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
Take a look at it with the usual BeautifulSoup modules (i.e. functions).
Now, as the Udacity Data Wrangling instructor said, Shannon Bradshaw, taught, we're going to need to use the Inspect Element (Firefox), or Develop -> Web Inspector (Mac OS X Safari) functions on your web browswer to see what the relevant html codes are.
Now in this particular case (webpage formats are all different; assume the worst), there are no distinguishing classes for the tables (they're just nested tables on tables).  cf. stackoverflow.com BeautifulSoup scraping nested tables  I'm using the solution from this stackoverflow answer.
Can you write Python code for it?

# for table in jpl_ssd_BS.soup.find_all("table"):
#    for subtable in table.find_all("table"):
#        print subtable.find("table") # uncomment this out and run it to see the whole darn thing


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
Then append the directory containing the Physique package/library (it's just a folder) with sys.path.append; the absolute path for where I placed it just happened to be "/Users/ernestyeung/Documents/TeslaModelSP85D": substitute that for the absolute path you find (look at your Finder or File organizing program)
Can you write Python code for it?

currentdir # I'm on a different computer now

sys.path.append('/home/topolo/PropD/Propulsion/')

import Physique


I am working on the file "Physique/Physique_jupyter.ipynb".
The first task is:
Time to scrape the actual html code for the table we desire: jpl_ssd_planet_phys_parTBL.  Take a look at the function make_conv_lst in scrape_BS.py and take a look at that first for loop.  That's the procedure we'll take (and I confirmed that this is in practice what's done on stackoverflow).  But wait: the data values are themselves tables.  So again, there is no rhyme or reason for the logic or rationale for the html tables for data, in general, for any websites.  So I'll get the headers first (which makes sense) and do an ugly hack for the data values (also notice the recursive=False option).
Can you write Python code for it?

data = []
for row in jpl_ssd_planet_phys_parTBL.find_all('tr', recursive=False):
    cols = row.find_all('td', recursive=False)
    cols = [ele.text if ele.text != u'\xa0' else u'' for ele in cols]
    data.append(cols)
hdrs = data[:2]    # get the headers first

jpl_ssd_planet_phys_parTBL.find_all('tr')[2].find_all('td')[18].text
data = [[row[0].replace(u'\xa0',''),]+row[1:] for row in data[2:]]  # remove the space, \xa0 from each of the planet's names
data = [[row[0],]+[ele.replace('\n','') for ele in row[1:]] for row in data] # remove the '\n' strings
data = [[row[0],]+[ele.split(u'\xb1')[0] for ele in row[1:]] for row in data] # let's just get the values
data = [[row[0],]+[ele.split(u'\xa0')[0] for ele in row[1:]] for row in data] # let's just get the values


from fractions import Fraction
from itertools import count, islice
from decimal import Decimal, localcontext

def convert(f):
    """get a Decimal from a Fraction (and multiply by 4)"""
    return (Decimal(f.numerator) / Decimal(f.denominator)) * Decimal(4)

def pi_series():
    "...converges very slowly"
    denoms = count(1,2)  # odd numbers from 1
    total = Fraction(1,next(denoms)) # 1/1
    while True:
        yield total
        total -= Fraction(1, next(denoms))  # - 1/3
        total += Fraction(1, next(denoms))  # + 1/5 and so on

def nth(iterable, n, default=None):
    "Returns the nth item or a default value"
    return next(islice(iterable, n, None), default)

with localcontext() as ctx:  # <-- context manager object
    ctx.prec = 3000
    pi = pi_series()
    print("{0}".format(convert(nth(pi, 1000)))[:10])


I am working on the file "Pi Day Fun.ipynb".
The first task is:
The Youtube above describes how to use successive primes in successive terms to build a running product that converges to 2/pi.
Can you write Python code for it?

def Primes():
    """generate successive prime numbers (trial by division)"""
    candidate = 1
    _primes_so_far = [2]  # first prime, only even prime
    yield _primes_so_far[-1]
    while True:
        candidate += 2           # check odds only from now on
        for prev in _primes_so_far:
            if prev**2 > candidate:
                yield candidate
                _primes_so_far.append(candidate)
                break
            if not divmod(candidate, prev)[1]: # no remainder!
                break                          # done looping

p = Primes()
print([next(p) for _ in range(100)])  # next 30 primes please!

def convert(f):
    """get a Decimal from a Fraction (and multiply by 4)"""
    return (Decimal(f.denominator) / Decimal(f.numerator))

def Pi():
    primes = Primes()
    result = Fraction(1,1)
    while True:
        p = next(primes)
        if divmod(p, 4)[1] == 1:
            term = (1 + Fraction(1,p))
        else:
            term = (1 - Fraction(1,p))
        result *= term
        yield result

with localcontext() as ctx:  # <-- context manager object
    ctx.prec = 300  # feel free to boost
    pi = Pi()
    print("{0}".format(convert(nth(pi, 333)))[:10])
    # print("{0}".format(convert(nth(pi, 3000)))[:20])


I am working on the file "Pi sampling.ipynb".
The first task is:
Using Plotly
Can you write Python code for it?

from plotly import tools
from plotly import plotly as py
from plotly.graph_objs import Scatter, Line,\
    Marker, Font, Scattergl, Annotation, Annotations

figure = tools.make_subplots(1, 1)
stream_ids = tools.get_credentials_file()['stream_ids']
print stream_ids

pi_inner = Scatter(
    x=[],
    y=[],
    stream=dict(
        token=stream_ids[0],
        maxpoints=1e6,
    ),
    mode='markers',
    marker=Marker(
        #color='rgb({},{},{})'.format(*np.multiply(b,255).astype(int)),
        symbol='square',
        size=4,
    )
)
pi_square = Scatter(
    x=[],
    y=[],
    stream=dict(
        token=stream_ids[1],
        maxpoints=1e6,
    ),
    mode='markers',
    marker=Marker(
        #color='rgb({},{},{})'.format(*np.multiply(r,255).astype(int)),
        symbol='square',
        size=4,
    )
)

pi_estimate = Scatter(
    x=[],
    y=[],
    stream=dict(
        token=stream_ids[2],
        maxpoints=400,
    ),
    mode='lines+markers',
    marker=Marker(
        #color=r,
        symbol='square',
        size=3,
        line=Line(
            width=.5,
            color='white'
        ),
    ),
    line=Line(width=.1),
)

#figure.append_trace(pi_inner, 1, 1)
#figure.append_trace(pi_square, 1, 1)
figure.append_trace(pi_estimate, 1, 1)

figure.layout.shapes = [
        # Line Horizontal
        {
            'type': 'line',
            'x0': 0,
            'y0': np.pi,
            'x1': 1,
            'y1': np.pi,
            'yref': 'y1',
            'xref':'paper',
            'line': {
                'color': 'rgb(255, 20, 96)',
                'width': 1.5,
                'dash': 'dash',
            },
        },]

py.plot(figure, filename='pi_markerssampling')

# reset the plot:
res = 500
iter_samples=1e4
nums = np.array([0., 0.])
i = 0

try:
    streams = [py.Stream(token) for token in stream_ids[:3]]
    #for s in streams:
    #    s.open()
    streams[2].open()
    count = 0
    while count<200:
        samples, w = next_sample(nums, iter_samples)
        inner = samples[w]
    ...


I am working on the file "Pi/String formatting for printing.ipynb".
The first task is:
Can also repeat an index eg. {0} {0} {0} - and use K2VP as below - cannot use numerical values (expressions and maloc rules).
Can you write Python code for it?

print('The {q} {g} {f}'.format(f='fox', q='quick', g='golden'))


I am working on the file "Pi/String formatting for printing.ipynb".
The first task is:
Formatted String literals AKA 'FString'.
Can you write Python code for it?

name = 'Rusha'

print(f'Hi, he said his name was {name} because he drinks too much caffeine and rushes around.')

age = 3000
real_name = 'Russell'
hobbits = 'AI, Art and Coding among others.'

print(f'HI, he said his name was {name} because he drinks a lot of caffeine and rushes about. I later found out after getting to know him, his real name is {real_name} and he is {age} years old and has a keen interest in {hobbits}')

pi = u"\U0001D70B"

print(f'He also said he likes the idea of {pi}.')


I am working on the file "Pi/String formatting for printing.ipynb".
The first task is:
Repeat can also be used with keys: {f} {f} {f}
Float formatting follows "{value:width.precision f}"
Can you write Python code for it?

result = 12987/8921

result

print('The result was {}'.format(result))

print('The result was {r:1.6f}'.format(r=result))


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Comme avec R, il est possible de calculer un indicateur d'importance des variables pour aider à une forme d'interprétation. Celui-ci dépend de la position de la variable dans l'arbre et correspond donc au mean decrease in Gini index de R plutôt qu'au mean descrease in accuracy. La forêt doit être réestimée car GridSearch ne connaît pas le paramètre d'importance.
Can you write Python code for it?

rf= RandomForestClassifier(n_estimators=100,max_features=2)
rfFit=rf.fit(Xr_train, Yb_train)
# Importance décroissante des variables
importances = rfFit.feature_importances_
indices = np.argsort(importances)[::-1]
for f in range(Xr_train.shape[1]):
    print(dfC.columns[indices[f]], importances[indices[f]])

# Graphe des importances
plt.figure()
plt.title("Importances des variables")
plt.bar(range(Xr_train.shape[1]), importances[indices])
plt.xticks(range(Xr_train.shape[1]), indices)
plt.xlim([-1, Xr_train.shape[1]])
plt.show()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Comparer l'erreur out-of-bag ci-dessus avec celle sur l'échantillon test.
Can you write Python code for it?

# erreur de prévision sur le test
1-rfFit.score(Xr_test,Yb_test)


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Définition des paramètres dont le nombre de neurones et alpha qui règle la régularisation par défaut 10-5. Le nombre de neurones est optimisé mais ce peut être alpha avec un nombre grand de neurones. Le nombre max d'itérations par défaut (200) semble insuffisant. Il est fixé à 500.
Can you write Python code for it?

param_grid=[{"hidden_layer_sizes":list([(5,),(6,),(7,),(8,)])}]
nnet= GridSearchCV(MLPClassifier(max_iter=500),param_grid,cv=10,n_jobs=-1)
nnetOpt=nnet.fit(Xr_train, Yb_train)
# paramètre optimal
print("Meilleur score = %f, Meilleur paramètre = %s" % (1. - nnetOpt.best_score_,nnetOpt.best_params_))

# Estimation de l'erreur de prévision sur le test
1-nnetOpt.score(Xr_test,Yb_test)

# prévision de l'échantillon test
y_chap = nnetOpt.predict(Xr_test)
# matrice de confusion
table=pd.crosstab(y_chap,Yb_test)
print(table)


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Exercice Comme précédemment, remplacer ensuite la fonction SVC par celle SVR de régression. Optimiser le paramètre, calculer la prévision, les résidus; le MSE.
Synthèse: comparaison des méthodes
Courbes ROC
Dans toute méthode, la prévision de dépassement ou non est associée au choix d'un seuil qui est par défaut 0.5. L'optimisaiton de ce seuil dépend des coûts respectifs associés aux faux positifs et aux faux négatifs qui ne sont pas nécessairement égaux. La courbe ROC permet de représenter l'influence de ce seuil sur les taux de faux positifs et vrais positifs.
Can you write Python code for it?

from sklearn.metrics import roc_curve
listMethod=[["RF",rfOpt],["NN",nnetOpt],["Tree",treeOpt],["K-nn",knnOpt],["Logit",logitOpt]]

for method in enumerate(listMethod):
    probas_ = method[1][1].fit(Xr_train, Yb_train).predict_proba(Xr_test)
    fpr, tpr, thresholds = roc_curve(Yb_test, probas_[:,1])
    plt.plot(fpr, tpr, lw=1,label="%s"%method[1][0])
plt.xlabel('Taux de faux positifs')
plt.ylabel('Taux de vrais positifs')
plt.legend(loc="best")
plt.show()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Exercice Compléter les résultats en utilisant la fonction KNeighborsRegressor pour modéliser la concentration; optimiser $k$, calculer la prévision de l'échantillon test, tracer le graphe des résidus, calculer le MSE sur l'échantillon test.
Arbre binaire de décision
Les arbres binaires de décision : discrimination ou régression, sont bien implémentés dans scikit-learn mais avec une insuffisance pour leur élagage. Ce n'est pas une pénalisation de la complexité, et donc précisément le nombre de feuilles qui est optimisé, mais la profondeur globale de l'arbre au risque d'élaguer, à une profondeur donnée, des feuilles importantes ou de conserver des feuilles ambigües.
Comme précédemment, la validation croisée permet d'optimiser le paramètre sur une grille.
Can you write Python code for it?

from sklearn.tree import DecisionTreeClassifier
# Optimisation de la profondeur de l'arbre
param=[{"max_depth":list(range(2,10))}]
tree= GridSearchCV(DecisionTreeClassifier(),param,cv=10,n_jobs=-1)
treeOpt=tree.fit(Xr_train, Yb_train)
# paramètre optimal
print("Meilleur score = %f, Meilleur paramètre = %s" % (1. - treeOpt.best_score_,treeOpt.best_params_))

# Estimation de l'erreur de prévision
1-treeOpt.score(Xr_test,Yb_test)

# prévision de l'échantillon test
y_chap = treeOpt.predict(Xr_test)
# matrice de confusion
table=pd.crosstab(y_chap,Yb_test)
print(table)


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Exercice Tester différentes valeurs de min_samples_split de celle trouvée optimale. Conclusion sur la sensibilité de l'optimisation de ce paramètre ?
Can you write Python code for it?

# prévision
y_chap = rfFit.predict(Xr_test)
# matrice de confusion
table=pd.crosstab(y_chap,Yb_test)
print(table)


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Exercice Traiter ainsi toutes les variables. Ceci suggère des transformations pour une meilleure utilisation des modèles linéaires.
Can you write Python code for it?

from math import sqrt, log
ozone["SRMH2O"]=ozone["RMH2O"].map(lambda x: sqrt(x))
ozone["LNO2"]=ozone["NO2"].map(lambda x: log(x))
ozone["LNO"]=ozone["NO"].map(lambda x: log(x))
del ozone["RMH2O"]
del ozone["NO2"]
del ozone["NO"]


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Exploration
Même si les données ne présentent pas de défauts particuliers, une étude exploratoire préliminaire est indispensable afin de s'assurer le leur bonne cohérence, proposer d'éventuelles transformations et analyser les structures de corrélations ou plus généralement de liaisons entre les variables, de groupes des individus ou observations.
Unidimensionnelle
Can you write Python code for it?

%matplotlib inline

import matplotlib.pyplot as plt
ozone["O3obs"].hist()
plt.show()

ozone["MOCAGE"].hist()
plt.show()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Extractions des échantillons d'apprentissage  et test pour les deux types de modèles. Comme le générateur est initalisé de façon identique, ce sont les mêmes échantillons dans les deux cas.
Can you write Python code for it?


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Isolation forest
Q Comment se mesure l"atypicité" d'une observation dans le cas d'isolation forest?
Can you write Python code for it?

from sklearn.ensemble import IsolationForest
clf = IsolationForest(max_samples=1000, contamination=0.05,behaviour="new")
scoreIF=clf.fit(ozoneR)
scoreAtypIF=clf.decision_function(ozoneR)
plt.boxplot(scoreAtypIF)
plt.show()

atypIFInd = clf.predict(ozoneR)
## Repésentation des atypiques
plt.figure(figsize=(5,5))
for i, j, nom in zip(C[:,0], C[:,1], atypIFInd):
    color = "red" if nom!=1  else "blue"
    plt.plot(i, j, "o",color=color)
plt.axis((-4,6,-4,6))
plt.show()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Le paramètre de pénalisation lasso est optimisé par validation croisée.
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV
# grille de valeurs du paramètre alpha à optimiser
param=[{"alpha":[0.05,0.1,0.2,0.3,0.4,0.5,1]}]
regLasso = GridSearchCV(linear_model.Lasso(), param,cv=5,n_jobs=-1)
regLassOpt=regLasso.fit(Xr_train, Yr_train)
# paramètre optimal
regLassOpt.best_params_["alpha"]
print("Meilleur R2 = %f, Meilleur paramètre = %s" % (regLassOpt.best_score_,regLassOpt.best_params_))


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Modèles linéaires
Les fonctions de modéles linéaires et linéaires généralisées sont limitées dans Scikit-learn et sans sorties numériques (tests) détaillées qui sont à rechercher dans une autre librairie (StatsModels). Dans les deux cas, les stratégies classiques (forward, backward, stepwise, Furnival et Wilson) de sélection de variables par optimisation d'un critère (Cp, AIC, BIC) ne semblent pas disponibles, même si AIC et BIC sont présents dans scikit-learn, et le type DataFrame (package pandas) n'est pas reconnu.
La façon efficace de procéder est donc d'introduire une pénalisation Lasso pour opérer une sélection de variables ou plutôt la sélection de variables quantitatives et d'indicatrices des modalités de celles qualitatives mais sans analyse fine des interactions comme cela est possible avec R.
Q Quel autre type de pénalisation est aussi utilisée en régression?
Q Quelle la méthode qui combine les deux?
A titre de comparaison, on trace la prévision de la concentration de l'échantillon test par la seule valeur du modèle Mocage ainsi que les résidus à ce modèle fonction de la valeur prédite (Mocage)
Can you write Python code for it?

plt.plot(X_train["MOCAGE"],Yr_train,"o")
plt.xlabel("Mocage")
plt.ylabel("O3 observee")
plt.show()

from sklearn.metrics import r2_score
print("R2=",r2_score(Yr_train,X_train["MOCAGE"]))

plt.plot(X_test["MOCAGE"],Yr_test,"o")
plt.xlabel("Mocage")
plt.ylabel("O3 observee")
plt.show()

plt.plot(X_test["MOCAGE"],X_test["MOCAGE"]-Yr_test,"o")
plt.xlabel("Mocage")
plt.ylabel("Residus")
plt.show()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Modélisations
La recherche  d'une meilleure méthode de prévision suit généralement le protocole suivant dont la première étape est déja réalisée.

Etape descriptive préliminaire uni et multidimensionnelle visant à repérer les incohérences, les variables non significatives ou de distribution exotique, les individus non concernés ou atypiques... et à étudier les structures des données. Ce peut être aussi la longue étape de construction de variables, attributs ou features spécifiques des données.
Procéder à un tirage aléatoire d'un échantillon test qui ne sera utilisé que lors de la dernière étape de comparaison des méthodes.
La partie restante est l'échantillon d'apprentissage pour l'estimation des paramètres des modèles.
Pour chacune des méthodes, optimiser la complexité des modèles en minimisant une estimation "sans biais" de l'erreur de prévision, par exemple par validation croisée.
Variables et interactions à prendre en compte dans la régression linéaire ou logistique;
variables et méthode pour l'analyse discriminante;
nombre de feuilles dans l'arbre de régression ou de classification;
architecture (nombre de neurones, pénalisation) du perceptron;
algorithme d'agrégation,
noyau et pénalisation des SVMs.

Comparaison des qualités de prévision sur la base du taux de mal classés pour le seul échantillon test qui est resté à l'écart de tout effort ou "acharnement" pour l'optimisation des modèles.

Remarques
* En cas d'échantillon relativement "petit" il est recommandé d'itérer la procédure de découpage apprentissage / test (validation croisée Monte Carlo), afin de réduire la variance (moyenne) des estimations des erreurs de prévision.
* Attention: ne pas "tricher" en modifiant le modèle obtenu lors de l'étape précédente afin d'améliorer le résultat sur l'échantillon test !
* Le critère utilisé dépend du problème : erreur quadratique, taux de mauvais classement, AUC (aire sous la courbe ROC), indice de Pierce, log loss function...
* L'étape "choix" de la meilleure méthode peut être remplacée par une...
Can you write Python code for it?

ozone.head()

# Variables explicatives
ozoneDum=pd.get_dummies(ozone[["JOUR","STATION"]])
del ozoneDum["JOUR_0"]
ozoneQuant=ozone[["MOCAGE","TEMPE","VentMOD","VentANG","SRMH2O","LNO2","LNO"]]
dfC=pd.concat([ozoneDum,ozoneQuant],axis=1)
dfC.head()

# variable à expliquer binaire
Yb=ozone["DepSeuil"].map(lambda x: int(x))
# variable à expliquer réelle
Yr=ozone["O3obs"]

Yr.hist()
plt.show()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Prévision par forêt aléatoire
Prévision du dépassement d'ozone sans données manquantes et avec données manquantes imputées. Comparaison des erreurs de prévision sur l'échantillon test. Les valeurs par défaut des paramètres sont conservées.
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier
# prévision sans trous
forest = RandomForestClassifier(n_estimators=500)
# apprentissage
rfFit = forest.fit(X_train,Yb_train)
# erreur de prévision
# erreur de prévision sur le test
1-rfFit.score(X_test,Yb_test)

# prévision avec trous imputés
forest = RandomForestClassifier(n_estimators=500)
# apprentissage
rfFit = forest.fit(XnaImp_train,Yb_train)
# erreur de prévision
# erreur de prévision sur le test
1-rfFit.score(XnaImp_test,Yb_test)


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Q Commenter ces résultats: quel choix de la dimension?
Q Présence de valeurs atypiques.
Can you write Python code for it?

## Repésentation des individus
plt.figure(figsize=(5,5))
for i, j, nom in zip(C[:,0], C[:,1], ozone["DepSeuil"]):
    color = "red" if nom  else "blue"
    plt.plot(i, j, "o",color=color)
plt.axis((-4,6,-4,6))
plt.show()

## coordonnées et représentation des variables
coord1=pca.components_[0]*np.sqrt(pca.explained_variance_[0])
coord2=pca.components_[1]*np.sqrt(pca.explained_variance_[1])
fig = plt.figure(figsize=(5,5))
ax = fig.add_subplot(1, 1, 1)
for i, j, nom in zip(coord1,coord2, ozone[["MOCAGE","TEMPE","VentMOD",
                                           "VentANG","SRMH2O","LNO2","LNO"]].columns):
    plt.text(i, j, nom)
    plt.arrow(0,0,i,j,color='black')
plt.axis((-1.2,1.2,-1.2,1.2))
# cercle
c=plt.Circle((0,0), radius=1, color='gray', fill=False)
ax.add_patch(c)
plt.show()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Q Commenter la qualité de ces résidus.
Can you write Python code for it?

# Erreur quadratique moyenne
from sklearn.metrics import mean_squared_error
print("MSE=",mean_squared_error(X_test["MOCAGE"],Yr_test))

# Le coefficient de détermination
# peut être négatif en prévision avec un mauvais modèle,
# est nul si la prévision est constante égale à la moyennne
from sklearn.metrics import r2_score
print("R2=",r2_score(Yr_test,X_test["MOCAGE"]))


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Q Commenter la structure de corrélation des variables.
Q L'objectif est de définir une surface séparant les deux classes. Une discriminaiton linéaire (hyperplan) semble-t-elle possible?
Ce n'est pas utile ici mais une classification non supervisée est facile à obtenir à titre illustratif, par exemple en 4 classes, par l'algorithme k-means:
Can you write Python code for it?

from sklearn.cluster  import  KMeans
from  sklearn.metrics  import confusion_matrix
clust=KMeans(n_clusters=4)
clust.fit(X)
classe=clust.labels_
print(classe)

## Repésentation des individus dans les coordonnées de l'acp.
plt.figure(figsize=(10,8))
plt.scatter(C[:,0], C[:,1], c=classe)
plt.show()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Q Commenter les relations entre les variables prises 2 à 2.
Analyse en composantes principales
Can you write Python code for it?

from sklearn.decomposition import PCA
from sklearn.preprocessing import scale
# réduction des variables
#X=scale(ozone[["O3obs","MOCAGE","TEMPE","VentMOD","VentANG","SRMH2O","LNO2","LNO"]])
X=scale(ozone[["MOCAGE","TEMPE","VentMOD","VentANG","SRMH2O","LNO2","LNO"]])


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Q Comparer ces résidus avec ceux précédents (mocage) et noter l'amélioration.
Q Commenter la forme du nuage et donc la validité du modèle.
L'interprétation nécessite de connaître les valeurs des coefficients du modèle alors que l'objet regLassOpt issu de GridSearchCV ne retient pas les paramètres estimés. Il faut donc le ré-estimer avec la valeur optimale du paramètre de pénalisation si l'on souhaite afficher ces coefficients.
Can you write Python code for it?

# Coefficients
regLasso=linear_model.Lasso(alpha=regLassOpt.best_params_['alpha'])
model_lasso=regLasso.fit(Xr_train,Yr_train)
model_lasso.coef_

coef = pd.Series(model_lasso.coef_, index = X_train.columns)
print("Lasso conserve " + str(sum(coef != 0)) +
      " variables et en supprime " +  str(sum(coef == 0)))

imp_coef = coef.sort_values()
plt.rcParams['figure.figsize'] = (8.0, 10.0)
imp_coef.plot(kind = "barh")
plt.title(u"Coefficients du modèle lasso")


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Q Interpréter l'effet des variables retenues.
Can you write Python code for it?

from sklearn.metrics import roc_curve
probas_ = LogisticRegression(penalty="l1", solver="liblinear",
                    C=logitOpt.best_params_['C']).fit(X_train, Yb_train).predict_proba(X_test)
fpr, tpr, thresholds = roc_curve(Yb_test, probas_[:,1])
plt.plot(fpr, tpr, lw=1)
plt.xlabel('Taux de faux positifs')
plt.ylabel('Taux de vrais positifs')
plt.show()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Q Que dire de la qualité de prévision avec 10% de trous
Exercice Faire varier le taux de trous et étudier la dégradation de la prévision.
Exercice Comparer avec une approche directe de la prévision avec XGBoost sans imputation préalable.
Épisode 5 bis
Remarque Il est possible d'exécuter directement l'épisode 5 sans passer par toutes les étapes de classification supervisée. Il suffit d'exécuter jusqu'à la section 4.1 de l'épidode 1, phase exploratoire et préparation des échantillons, afin de construire les données utilisées dans les sections 6 et 7 d'imputation des données manquantes et de détection d'atypiques.
Détection d'observations atypiques
Le calepin R d'analyse de ces mêmes données propose une comparaison assez détaillée des scores de détection des anomalies. Comme dans R, Scikit-learn propose des fonctions en Pyhton de détection d'atypiques multidimensionnels. Les principales sont LOF et Isolation Forest dont les résultats sont comparés ci-dessous.
Local Outlier Factor
Les données sont restreintes aux seules variables quantitatives explicatives.
Q Quel est le rôle du paramètre k ci-dessous?
Can you write Python code for it?

ozone.head()

ozoneR=ozone[["MOCAGE","TEMPE","VentMOD","VentANG","SRMH2O","LNO2","LNO"]]
ozoneR.head()

from sklearn.neighbors import LocalOutlierFactor
clf = LocalOutlierFactor(n_neighbors=20, contamination=0.05 ) # choix de n_n par défaut
scoreLOF=clf.fit_predict(ozoneR)
scoreAtyp=-clf._decision_function(ozoneR)# opposé du LOF
plt.boxplot(scoreAtyp)
plt.show()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Q Quel taux d'atypiques par défaut?
Can you write Python code for it?

atypSVMInd = clf.predict(ozoneR)

## Repésentation des atypiques
plt.figure(figsize=(5,5))
for i, j, nom in zip(C[:,0], C[:,1], atypSVMInd):
    color = "red" if nom!=1  else "blue"
    plt.plot(i, j, "o",color=color)
plt.axis((-4,6,-4,6))
plt.show()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Q Vérifier que c'est bien la même valeur optimale que celle précédemment trouvée.
Tracés des chemins de régularisation.
Can you write Python code for it?

from itertools import cycle

from sklearn.linear_model import lasso_path
alphas_lasso, coefs_lasso, _ = lasso_path(Xr_train,Yr_train, alphas=np.array(range(1,50,1))/20.,)

plt.figure()
ax = plt.gca()

styles = cycle(['-', '--', '-.', ':'])

neg_log_alphas_lasso = -np.log10(alphas_lasso)
for coef_l, s in zip(coefs_lasso, styles):
    l1 = plt.plot(neg_log_alphas_lasso, coef_l, linestyle=s,c='b')
plt.xlabel('-Log(alpha)')
plt.ylabel('Coefficients')
plt.show()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Q22 Le critère d'AUC (aire sous la courbe) permet-il d'ordonner les courbes et donc les méthodes?
C'est à un taux de faux positif admissible et donc à valeur de seuil fixé qu'il faut choisir la méthode d'apprentissage à privilégier.
Itération sur plusieurs échantillons de test (validation croisée Monte Carlo)
L'échantillon test est de taille modeste et donc l'estimation de l'erreur de prévision peut présenter une variance importante. Celle-ci est réduite en opérant une forme de validation croisée (Monte Carlo) en tirant plusieurs couples d'échantillon apprentissage et test pour itérer les traitements précédents. Les données sont normalisées pour toutes les méthodes car les autres que SVM et NN ne sont pas affectées.
Les fonctionnalités de scikit-learn se prètent bien à l'automatisation de ces traitements enchaînant extraction d'échantillons, estimation de plusieurs modèles, optimisation de leurs paramètres et estimation de l'erreur de prévision sur le test.
Le code est compact et d'exécution efficace car bien parallélisé par les fonctions utilisées.
Can you write Python code for it?

from sklearn.utils import check_random_state
import time
check_random_state(13)
tps0=time.perf_counter()
# définition des estimateurs
logit= LogisticRegression(penalty="l1",solver="liblinear")
knn  = KNeighborsClassifier()
tree = DecisionTreeClassifier()
nnet = MLPClassifier(max_iter=600)
rf   = RandomForestClassifier(n_estimators=100)
svm  = SVC()
# Nombre d'itérations
B=3 # pour exécuter après le test, mettre plutôt B=30
# définition des grilles de paramètres
listMethGrid=[[svm,{"C":[0.4,0.5,0.6,0.8,1,1.4]}],
    [rf,{"max_features":list(range(2,10,2))}],
    [nnet,{"hidden_layer_sizes":list([(5,),(6,),(7,),(8,)])}],
    [tree,{"max_depth":list(range(2,10))}],
    [knn,{"n_neighbors":list(range(1,15))}],
    [logit,{"C":[0.5,1,5,10,12,15,30]}]]
# Initialisation à 0 des erreurs pour chaque méthode (colonne) et chaque itération (ligne)
arrayErreur=np.empty((B,6))
for i in range(B):   # itérations sur B échantillons test
    # extraction apprentissage et test
    X_train,X_test,Yb_train,Yb_test=train_test_split(dfC,Yb,test_size=200)
    scaler = StandardScaler()
    scaler.fit(X_train)
    X_train = scaler.transform(X_train)
    # Meme transformation sur le test
    X_test = scaler.transform(X_test)
    # optimisation de chaque méthode et calcul de l'erreur sur le test
    for j,(method, grid_list) in enumerate(listMethGrid):
        methodGrid=GridSearchCV(method,grid_list,cv=10,n_jobs=-1,
                               iid="TRUE").fit(X_train, Yb_train)
        methodOpt = methodGrid.best_estimator_
        methFit=methodOpt.fit(X_train, Yb_train)
        arrayErreur[i,j]=1-methFit.score(X_test,Yb_test)
tps1=time.perf_counter()
print("Temps execution en mn :",(tps1 - tps0))
dataframeErreur=pd.DataFrame(arrayErreur,columns=["SVM","RF","NN","Tree","Knn","Logit"])

# Distribution des erreurs de prévisions
# Les SVM présentant des erreurs atypiques sont laissés de côté.
dataframeErreur[["SVM","RF","NN","Tree","Knn","Logit"]].boxplot(return_type='dict')
plt.show()

# Moyennes
dataframeErreur.mean()


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Séparation des échantillons
Des cas sont consiédérés: les données sans données manquantes et les données après imputation des données manquantes. Les mêmes échantillons sont considérés en utilisant la même initialisation du générateur.
Can you write Python code for it?


I am working on the file "Pic-ozone/Apprent-Python-Ozone.ipynb".
The first task is:
Tous les résultats numétriques classiques sont fournis par l'implémentation de scikit-learn mais des  efforts sont à produire pour construire les graphiques usuels généralement automatiquement produits par des librairies dédiées comme FactoMineR de R.
Les commandes suivantes permettent de réaliser une analyse en composantes principales sur les seules variables quantitatives. Par ailleurs la variable à modéliser  (O3obs, concentration observée) n'est pas utilisée.
Can you write Python code for it?

pca = PCA()
## Estimation, calcul des composantes principales
C = pca.fit(X).transform(X)
## Décroissance de la variance expliquée
plt.plot(pca.explained_variance_ratio_)
plt.show()

## distribution des composantes principales
plt.boxplot(C[:,0:20])
plt.show()


I am working on the file "Pima Project/.ipynb_checkpoints/Pima Project 2.0-checkpoint.ipynb".
The first task is:
تجربة استخدام تقييس و تدريج الخواص لتحسين اداء النموذج
Can you write Python code for it?

columns = ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age']
labels = df['Outcome'].values
features = df[list(columns)].values

X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.30)

clf = RandomForestClassifier(n_estimators=1)
clf = clf.fit(X_train, y_train)

accuracy = clf.score(X_train, y_train)
print ' اداء النموذج في عينة التدريب بدقة ', accuracy*100

accuracy = clf.score(X_test, y_test)
print ' اداء النموذج في عينة الفحص بدقة ', accuracy*100

ypredict = clf.predict(X_train)
print '\n Training classification report\n', classification_report(y_train, ypredict)
print "\n Confusion matrix of training \n", confusion_matrix(y_train, ypredict)

ypredict = clf.predict(X_test)
print '\n Training classification report\n', classification_report(y_test, ypredict)
print "\n Confusion matrix of training \n", confusion_matrix(y_test, ypredict)


I am working on the file "Pima Project/.ipynb_checkpoints/Pima Project 2.0-checkpoint.ipynb".
The first task is:
سيبورن مكتبة جميلة للرسوميات سهلة في الكتابة لكن مفيدة جداً في المعلومات التي ممكن ان نقراءها عبر الهيستوقرام
فائدها ممكن ان تكون في
1- تلخيص توزيع البينات في رسوميات
2- فهم او الإطلاع على القيم الفريدة
3- تحمل الرسوميات معنى اعمق من الكلمات
Can you write Python code for it?

sb.countplot(x='Outcome',data=df, palette='hls')

sb.countplot(x='Pregnancies',data=df, palette='hls')

sb.countplot(x='Glucose',data=df, palette='hls')

sb.heatmap(df.corr())

sb.pairplot(df, hue="Outcome")

from scipy.stats import kendalltau
sb.jointplot(df['Pregnancies'], df['Glucose'], kind="hex", stat_func=kendalltau, color="#4CB391")

import matplotlib.pyplot as plt
g = sb.FacetGrid(df, row="Pregnancies", col="Outcome", margin_titles=True)
bins = np.linspace(0, 50, 13)
g.map(plt.hist, "BMI", color="steelblue", bins=bins, lw=0)

sb.pairplot(df, vars=["Pregnancies", "BMI"])


I am working on the file "Pima Project/.ipynb_checkpoints/Pima Project 2.0-checkpoint.ipynb".
The first task is:
هذه الدالة تعطينا توصيف كامل للبيانات و تكشف لنا في ما إذا كانت هناك قيم مفقودة
Can you write Python code for it?


I am working on the file "Pima Project/Pima Project 2.0.ipynb".
The first task is:
تجربة استخدام تقييس و تدريج الخواص لتحسين اداء النموذج
Can you write Python code for it?

columns = ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age']
labels = df['Outcome'].values
features = df[list(columns)].values

X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.30)

clf = RandomForestClassifier(n_estimators=1)
clf = clf.fit(X_train, y_train)

accuracy = clf.score(X_train, y_train)
print ' اداء النموذج في عينة التدريب بدقة ', accuracy*100

accuracy = clf.score(X_test, y_test)
print ' اداء النموذج في عينة الفحص بدقة ', accuracy*100

ypredict = clf.predict(X_train)
print '\n Training classification report\n', classification_report(y_train, ypredict)
print "\n Confusion matrix of training \n", confusion_matrix(y_train, ypredict)

ypredict = clf.predict(X_test)
print '\n Testing classification report\n', classification_report(y_test, ypredict)
print "\n Confusion matrix of Testing \n", confusion_matrix(y_test, ypredict)


I am working on the file "Pima Project/Pima Project 2.0.ipynb".
The first task is:
تجربة تحسين اداء النموذج بطريقة
min-max scaler
Can you write Python code for it?

columns = ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age']
labels = df['Outcome'].values
features = df[list(columns)].values

X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.30)

scaler = preprocessing.MinMaxScaler()

scaler.fit(X_train)
X_train = scaler.transform(X_train)
# apply same transformation to test data
X_test = scaler.transform(X_test)

clf = RandomForestClassifier(n_estimators=1)
clf = clf.fit(X_train, y_train)

accuracy = clf.score(X_train, y_train)
print ' اداء النموذج في عينة التدريب بدقة ', accuracy*100

accuracy = clf.score(X_test, y_test)
print ' اداء النموذج في عينة الفحص بدقة ', accuracy*100

ypredict = clf.predict(X_train)
print '\n Training classification report\n', classification_report(y_train, ypredict)
print "\n Confusion matrix of training \n", confusion_matrix(y_train, ypredict)

ypredict = clf.predict(X_test)
print '\n Testing classification report\n', classification_report(y_test, ypredict)
print "\n Confusion matrix of Testing \n", confusion_matrix(y_test, ypredict)

columns = ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age']
labels = df['Outcome'].values
features = df[list(columns)].values

X_train, X_test, y_train, y_test = train_test_split(features, labels, test_size=0.30)

clf = RandomForestClassifier(n_estimators=5)
clf = clf.fit(X_train, y_train)

accuracy = clf.score(X_train, y_train)
print ' اداء النموذج في عينة التدريب بدقة ', accuracy*100

accuracy = clf.score(X_test, y_test)
print ' اداء النموذج في عينة الفحص بدقة ', accuracy*100

ypredict = clf.predict(X_train)
print '\n Training classification report\n', classification_report(y_train, ypredict)
print "\n Confusion matrix of training \n", confusion_matrix(y_train, ypredict)

ypredict = clf.predict(X_test)
print '\n Testing classification report\n', classification_report(y_test, ypredict)
print "\n Confusion matrix of Testing \n", confusion_matrix(y_test, ypredict)


I am working on the file "Pima Project/Pima Project 2.0.ipynb".
The first task is:
سيبورن مكتبة جميلة للرسوميات سهلة في الكتابة لكن مفيدة جداً في المعلومات التي ممكن ان نقراءها عبر الهيستوقرام
فائدها ممكن ان تكون في
1- تلخيص توزيع البينات في رسوميات
2- فهم او الإطلاع على القيم الفريدة
3- تحمل الرسوميات معنى اعمق من الكلمات
Can you write Python code for it?


I am working on the file "Pima Project/Pima Project.ipynb".
The first task is:
confusion matrix and classification report for the testing sample
Can you write Python code for it?

from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

ypredict = clf.predict(X_train)
print '\nTraining classification report\n', classification_report(y_train, ypredict)

print "\n Confusion matrix of training \n", confusion_matrix(y_train, ypredict)

ypredict = clf.predict(X_test)
print '\nTraining classification report\n', classification_report(y_test, ypredict)
print "\n Confusion matrix of training \n", confusion_matrix(y_test, ypredict)


I am working on the file "Pipeline construction 1.ipynb".
The first task is:
Explore Data
The below table shows outcomes split by data labeled delinquent or nondelinquent. The number of times delinquents are past due is an order of magnitude higher than nondelinquents, but the high variance on the measure and low median suggest that is driven by people with particularly high numbers of past due bills. Nondelinquents are older and have higher income on average.
Can you write Python code for it?

summary = x.summary_by_outcome(data, "serious_dlqin2yrs")

summary.ix[:,[0,1,2,4,6,7,8,10]]

null_income = u.check_nulls(data, "monthly_income")
null_sum = x.summary_by_outcome(null_income, "serious_dlqin2yrs")

not_null_income = u.get_notnulls(data, "monthly_income")
not_null_sum = x.summary_by_outcome(not_null_income, "serious_dlqin2yrs")
ratio = round(null_sum.ix[:,[0,1,2,5,6,7,8,11]]/not_null_sum.ix[:,[0,1,2,5,6,7,8,11]],2)
ratio.drop("monthly_income")


I am working on the file "Pipeline construction 1.ipynb".
The first task is:
The analysis above compares people who have a reported monthly income and those with null monthly incomes. The ratio of missing income data over debt ratio with income data is very large. Those without income data have 62 to 77 times the debt ratio of those with reported data. In general, we would hope these ratios were close to 1. The count data show for each person with missing data there are 4 to 5 people with reported income. In other words, those with missing data make up slightly under 20 percent of our data. I will have to be careful when developing models to account for this discrepancy. For example, in a large project I would compare models with imputed values for the missing income data with dropping the data and would consider capping debt ratios. The graphs below show that the data with nulls has more extreme debt ratios (compare the bottom corners of the following two pair plots). The solution is I will implement will be to group debt ratios.
Can you write Python code for it?

sns.set(style="white")
sns.pairplot(data, vars=["debt_ratio","revolving_utilization_of_unsecured_lines","number_real_estate_loans_or_lines"],\
             hue="serious_dlqin2yrs", size=3, plot_kws={'alpha':0.2})
pl.suptitle("Paired distribution of indictors by delinquincy status (keeping NA's)")

sns.pairplot(u.get_notnulls(data,"monthly_income"), vars=["monthly_income","debt_ratio","revolving_utilization_of_unsecured_lines","number_real_estate_loans_or_lines"],\
             hue="serious_dlqin2yrs", size=3, plot_kws={'alpha':0.2})
pl.suptitle("Paired distribution of indictors by delinquincy status (removing NA's)")


I am working on the file "Pipeline.ipynb".
The first task is:
Raw heatmap of function (mito. membrane potential ΔΨ).
Can you write Python code for it?

Image(filename=op.join(datadir, 'DY_raw' + '.png'))


I am working on the file "Pipeline.ipynb".
The first task is:
The network has can now be viewed as a weighted graph with branchpoints (magenta dots) and end points (cyan dots).
Can you write Python code for it?

figone = mlab.figure(figure=key,
                     size=(1200, 800),
                     bgcolor=(.086, .086, .086))
vtkobj, vtktube = vf.cellplot(figone,
                              vtkNorm[key],
                              rad=.06)
vtktube.actor.mapper.scalar_visibility = False
_, _, nxgrph = mn.makegraph(vf.callreader(vtkNorm[key]), key)
vf.labelbpoints(nxgrph, bsize=.10, esize=0.08)
mlab.savefig(op.join(datadir, 'bare2' + '.png'))
mlab.close(figone)
Image(filename=op.join(datadir, 'bare2' + '.png'))


I am working on the file "Pipeline.ipynb".
The first task is:
The pipeline starts with taking the skeletonized data output from MitoGraph v2.0 along with the raw 3D volume data in both channels (GFP and RFP) along with the surface file
Can you write Python code for it?

try:
    with open(op.join(datadir, 'fileMetas.pkl'), 'rb') as inpt:
        filemetas = pickle.load(inpt)
except IOError:
    print "Error: Make sure you have file metadatas in working directory"

try:
    vtkSkel = wr.swalk(datadir, '*skeleton.vtk', stop=-13)
    vtkVolRfp = wr.swalk(datadir, '*RF*resampled.vtk', stop=-14)
    vtkVolGfp = wr.swalk(datadir, '*GF*resampled.vtk', stop=-14)
    vtkSurf =  wr.swalk(datadir, '*urface*.vtk', stop=-12)
except Exception:
    print "Error: check your filepaths"
    sys.exit()

key = '052315_009_RFPstack_012'


I am working on the file "Pipeline.ipynb".
The first task is:
The skeleton has no scalar values yet
Can you write Python code for it?

figone = mlab.figure(figure=key,
                     size=(1200, 800),
                     bgcolor=(.086, .086, .086))
vtkobj, vtktube = vf.cellplot(figone,
                              vtkSkel[key],
                              scalartype='Width',
                              rad=.06)
vtktube.actor.mapper.scalar_visibility = False
mlab.savefig(op.join(datadir, 'bare' + '.png'))
mlab.close(figone)
Image(filename=op.join(datadir, 'bare' + '.png'))


I am working on the file "Pipeline.ipynb".
The first task is:
We calculate scalar values using a point cloud averaging along every pixel in the skeleton. A graph network is also calculated from by examining the end point of each line segment in the skeleton for coincident points.
Can you write Python code for it?

data = pf.add_scalars(vtkSkel[key],
                      vtkVolRfp[key],
                      vtkVolGfp[key.replace('RFP', 'GFP')])
filename = op.join(datadir, 'Norm_%s_skeleton.vtk' % key)
nm, rw, rb, gb, wq = pf.normSkel(data, filemetas['YPD_'+key])
calc = {'DY_minmax': nm,
        'DY_raw': rw,
        'bkstRFP': rb,
        'bkstGFP': gb,
        'WidthEq': wq}
pf.writevtk(data, filename, **calc)
vtkNorm = wr.swalk(op.join(datadir, ),
                   'N*Skeleton.vtk', start=5, stop=-13)
# render the various skeleton heatmaps offscreen as Mayavi doesnt allow easy inline plotting in Ipython
for t in calc:
    figone = mlab.figure(figure=key,
                         size=(1200, 800),
                         bgcolor=(.086, .086, .086))
    _, vtktube = vf.cellplot(figone,
                                  vtkNorm[key],
                                  scalartype=t,
                                  rad=.08)
    mlab.savefig(op.join(datadir, t + '.png'))
    mlab.close(figone)


I am working on the file "Pipeline.ipynb".
The first task is:
Working directory for input and output data
Can you write Python code for it?

datadir = op.join(os.getcwd(),'data', 'pipeline')


I am working on the file "Pivot-demo-2.ipynb".
The first task is:
Create a simple dataset to demonstrate a basic pivot:
Can you write Python code for it?

%%sql
DROP TABLE IF EXISTS pivset CASCADE;
CREATE TABLE pivset(
                  id INTEGER,
                  piv INTEGER,
                  val FLOAT8
                );
INSERT INTO pivset VALUES
    (0, 10, 1),
    (0, 10, 2),
    (0, 20, 3),
    (1, 20, 4),
    (1, 30, 5),
    (1, 30, 6),
    (1, 10, 7),
    (NULL, 10, 8),
    (1, NULL, 9),
    (1, 10, NULL);

SELECT * FROM pivset ORDER BY id;

%%sql
DROP TABLE IF EXISTS pivout;
SELECT madlib.pivot('pivset',     -- source data
                    'pivout',     -- output data
                    'id',         -- index (rows in the output table)
                    'piv',        -- pivot columns
                    'val');       -- values to be summarized in the output table
SELECT * FROM pivout ORDER BY id;


I am working on the file "Pivot-demo-2.ipynb".
The first task is:
Fill null results with a value of interest:
Can you write Python code for it?

%%sql
DROP TABLE IF EXISTS pivout;
SELECT madlib.pivot('pivset_ext', 'pivout', 'id', 'piv', 'val', 'sum', '111');
SELECT * FROM pivout ORDER BY id;


I am working on the file "Pivot-demo-2.ipynb".
The first task is:
Keep null values in the pivot column:
Can you write Python code for it?

%%sql
DROP TABLE IF EXISTS pivout;
SELECT madlib.pivot('pivset_ext', 'pivout', 'id', 'piv', 'val', 'sum', True);
SELECT * FROM pivout ORDER BY id;


I am working on the file "Pivot-demo-2.ipynb".
The first task is:
Let's use a different aggregate function on the view we just created:
Can you write Python code for it?

%%sql
DROP TABLE IF EXISTS pivout;
SELECT madlib.pivot('pivset_ext', 'pivout', 'id', 'piv', 'val', 'sum');
SELECT * FROM pivout ORDER BY id;


I am working on the file "Pivot-demo-2.ipynb".
The first task is:
Now let's add some more columns to our data set and create a view:
Can you write Python code for it?

%%sql
DROP VIEW IF EXISTS pivset_ext;
CREATE VIEW pivset_ext AS
    SELECT *,
    COALESCE(id + (val / 3)::integer, 0) AS id2,
    COALESCE(100*(val / 3)::integer, 0) AS piv2,
    COALESCE(val + 10, 0) AS val2
   FROM pivset;
SELECT id,id2,piv,piv2,val,val2 FROM pivset_ext
ORDER BY id,id2,piv,piv2,val,val2;


I am working on the file "Pivot-demo-2.ipynb".
The first task is:
Use different aggregate functions for different value columns:
Can you write Python code for it?

%%sql
DROP TABLE IF EXISTS pivout;
SELECT madlib.pivot('pivset_ext', 'pivout', 'id', 'piv', 'val, val2',
    'val=avg, val2=sum');
SELECT * FROM pivout ORDER BY id;


I am working on the file "Pivot-demo-2.ipynb".
The first task is:
Use multiple index columns:
Can you write Python code for it?

%%sql
DROP TABLE IF EXISTS pivout;
SELECT madlib.pivot('pivset_ext', 'pivout', 'id,id2', 'piv', 'val');
SELECT * FROM pivout ORDER BY id,id2;


I am working on the file "Pivot-demo-2.ipynb".
The first task is:
Use multiple pivot columns:
Can you write Python code for it?

%%sql
DROP TABLE IF EXISTS pivout;
SELECT madlib.pivot('pivset_ext', 'pivout', 'id', 'piv, piv2', 'val');
SELECT * FROM pivout ORDER BY id;


I am working on the file "Pixar Movies/Pixar Movies.ipynb".
The first task is:
Data cleaning
Because Domestic % and International % columns data end with %, and its data type are objects, it is necessary to transfer its data type to float.
Can you write Python code for it?

pixar_movies['Domestic %'] = pixar_movies['Domestic %'].str.rstrip('%').astype('float')

pixar_movies['International %'] = pixar_movies['International %'].str.rstrip('%').astype('float')


I am working on the file "Pixar Movies/Pixar Movies.ipynb".
The first task is:
How has the ratio of where the revenue comes from changed since the first movie? Now that Pixar is more well known internationally, is more revenue being made internationally for newer movies?
Can you write Python code for it?

revenue_proportions = filtered_pixar[['Domestic %', 'International %']]

revenue_proportions.plot(kind='bar', stacked=True, figsize=(12,6))
#sns.plt.show()
plt.show()


I am working on the file "Pixar Movies/Pixar Movies.ipynb".
The first task is:
Is there any correlation between the number of Oscars a movie was nominated for and the number it actually won
Can you write Python code for it?

filtered_pixar[['Oscars Nominated', 'Oscars Won']].plot(kind='bar', figsize=(12,6))
plt.show()


I am working on the file "Pixel Regression - Step by Step.ipynb".
The first task is:
Our objective is to train a deep MLP that is able to reconstruct the image:
Can you write Python code for it?

import keras
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Dropout
from keras.optimizers import Adam, RMSprop, Nadam

# Model architecture
model = Sequential()

model.add(Dense(500, input_dim=2, init='uniform'))
model.add(Activation('relu'))

model.add(Dense(500, init='uniform'))
model.add(Activation('relu'))

model.add(Dense(500, init='uniform'))
model.add(Activation('relu'))

model.add(Dense(500, init='uniform'))
model.add(Activation('relu'))

model.add(Dense(500, init='uniform'))
model.add(Activation('relu'))

model.add(Dense(3, init='uniform'))
model.add(Activation('linear'))

model.summary()

# Compile model
model.compile(loss='mean_squared_error',
              optimizer=Nadam(),
              metrics=['accuracy'])
# Why use NAdam Optimizer?
# Much like Adam is essentially RMSprop with momentum, Nadam is Adam RMSprop with Nesterov momentum.

# use this cell to find the best model architecture
history = model.fit(X_train, Y_train, nb_epoch=1000, shuffle=True, verbose=1, batch_size=500)
Y = model.predict(X_train, batch_size=10000)
k = 0
im_out = im[:]
for i in range(im.shape[0]):
    for j in range(im.shape[1]):
        im_out[i,j]= Y[k]
        k += 1

print "Mona Lisa by DL"
plt.imshow(im_out)
plt.show()

plt.imshow(im_out)
plt.show()

# summarize history for accuracy
plt.plot(history.history['acc'], 'b')
plt.title('Model Accuracy')
plt.xlabel('epoch')
plt.show()

# summarize history for loss
plt.plot(history.history['loss'], 'r')
plt.title('Model Loss')
plt.xlabel('epoch')
plt.show()


I am working on the file "PixieApp+for+Outlier+Detection.ipynb".
The first task is:
2. 4 Format variables
Some numeric variables, like earnings and energy cost, are saved as string and need to be converted to numeric.
Can you write Python code for it?

#Remove schools with missing energy cost (4 schools)
schools_df['no_energy_cost'] = schools_df.apply(lambda x: '-' in x[' BPS_Total_Energy_Cost '], axis = 1)
schools_df = schools_df[~schools_df['no_energy_cost']]

#Restore numeric fields stored as strings to their former glory
schools_df[' BPS_Total_Energy_Cost '] = schools_df[' BPS_Total_Energy_Cost '].apply(lambda x: float(sub(r'[^\d.]', '', x)))
schools_df['TOTAL EARNINGS'] = schools_df['TOTAL EARNINGS'].apply(lambda x: float(sub(r'[^\d.]', '', x)))
schools_df['DOE_Total'] = schools_df['DOE_Total'].apply(lambda x: int(x))

#Drop intermediate linking and boolean fields
schools_df = schools_df[['SMMA_Identifier', 'BPS_School_Name', 'BPS_Address', 'SMMA_latitude', 'SMMA_longitude', 'BPS_Grades_Offered', 'SMMA_Site_SF', 'DOE_Total', ' BPS_Total_Energy_Cost ', 'BPS_Principal', 'TOTAL EARNINGS', 'POSTAL', 'TITLE']]

#Summary statistics:
schools_df.describe(include = 'all')


I am working on the file "PixieApp+for+Outlier+Detection.ipynb".
The first task is:
2. Create the Boston Public Schools data set
2.1 Import the BuildBPS data set

2.2 Link BuildBPS data set to the Employee Earnings data set

2.3 Merge the employee information with the school data set

2.4 Format variables
2.1 Import the BuildBPS data set
The backbone of the Boston Public Schools (BPS) data set, BuildBPS, includes:

Latitude and longitude for schools in BPS
Energy costs
Facility information
Enrollment information
Administration information

Principal name is separated into first and last name to link to the Employee Earnings data set.
Can you write Python code for it?

#Import the data and transform to Pandas. Pixiedust works with Spark dataframes, but unfortunately, scikitlearn will not.
schools = pixiedust.sampleData("https://data.boston.gov/dataset/b2c5a9d3-609d-49ec-906c-b0e850a8d62a/resource/33c5f44a-3c67-4390-a1d5-1bf018e4728c/download/buildbps.csv")
schools = schools.toPandas()

#Pull a subset of variables of interest
schools = schools[['SMMA_Identifier', 'BPS_School_Name', 'BPS_Address', 'SMMA_latitude', 'SMMA_longitude', 'BPS_Grades_Offered', 'SMMA_Site_SF', 'DOE_Total', ' BPS_Total_Energy_Cost ', 'BPS_Principal']]

#Separate the principals' first and last names to aid linking
def findFirstName(name):
    if isinstance(name, basestring):
        if 'Dr' in name:
            name = name.split('. ')[-1]
        firstname = name.split(' ')[0]
    else:
        firstname = name
    return firstname

schools['Principal_Firstname'] = schools['BPS_Principal'].apply(lambda x : findFirstName(x))

def findLastName(name):
#takes the last word before an optional comma, which should be the last name in this dataset
    if isinstance(name, basestring):
        name = name.split(',')[0]
        if isinstance(name, basestring):
            lastname = name.split(' ')[-1]
            if lastname == 'Jr.':
                lastname = name.split(' ')[-2]
        else:
            lastname = name
    else:
        lastname = name
    return lastname

schools['Principal_Lastname'] = schools['BPS_Principal'].apply(lambda x : findLastName(x))

#Print the first 5 rows to get a look at the data
schools.head()


I am working on the file "PixieApp+for+Outlier+Detection.ipynb".
The first task is:
2.2 Link BuildBPS to the Employee Earnings data set
To make it more interesting, link the BuildBPS data set to get principal earnings and zip code from the Employee Earnings data set. Because principal name is the only available linking variable, this gets a bit messy.

This data set includes:
- Employee names
- Departments
- Titles
- Earnings
- Zip code
Can you write Python code for it?

#Import the data and transform to Pandas
employees = pixiedust.sampleData("https://data.boston.gov/dataset/418983dc-7cae-42bb-88e4-d56f5adcf869/resource/2ff6343f-850d-46e7-98d1-aca79b619fd6/download/employee-earnings-report-2015.csv")
employees = employees.toPandas()

#Keep variables of interest
employees = employees[['NAME', 'DEPARTMENT_NAME', 'TITLE', 'TOTAL EARNINGS', 'POSTAL']]

#Keep employees with title containing "headmaster" or "principal"
employees = employees[employees['TITLE'].str.contains('headmaster|principal', case = False)]

#Separate first and last names for merging with the key dataset
employees['Principal_Lastname'] = employees['NAME'].str.split(',', 1, expand = True)[0]
employees['Principal_Firstname'] = employees['NAME'].str.split(',', 1, expand = True)[1]

#Print the first 5 rows to get a look at the data
employees.head()


I am working on the file "PixieApp+for+Outlier+Detection.ipynb".
The first task is:
2.3 Merge the employee information with the school data set
Next, merge the employee information into the school data set. First names differ slightly and nicknames are sometimes used (for example, "Rob" vs. "Robert"), so we match on last name and first three letters of first name.
To make this more accurate in the future, consider using a fuzzy matching algorithm. 40 schools (about 30% of the data set) are lost in the matching, either due to missing principal name in the BuildBPS data set (11) or inadequate matching (29).
Can you write Python code for it?

#Merge on last name
schools_df = pd.merge(schools, employees, on = 'Principal_Lastname')

#Keep those matches that match in the first 3 letters of first name
schools_df['Firstname_match'] = schools_df['Principal_Firstname_x'].apply(lambda x : x[0:3].lower())
schools_df['Principal_Firstname_y'] = schools_df['Principal_Firstname_y'].apply(lambda x: x.lower())
schools_df['correct_name'] = schools_df.apply(lambda x: x['Firstname_match'] in x['Principal_Firstname_y'], axis = 1)
schools_df = schools_df[schools_df['correct_name']]

schools_df.describe(include = 'all')


I am working on the file "PixieApp+for+Outlier+Detection.ipynb".
The first task is:
4. Run the PixieApp
Can you write Python code for it?


I am working on the file "PixieApp+for+Outlier+Detection.ipynb".
The first task is:
Continuing with square feet as one-dimension (to give a sense of where our last data points fell), look at what differentiated the outliers with normal square footage. A few schools with very high enrollment (1500+) were labeled as outliers.
Can you write Python code for it?


I am working on the file "PixieApp+for+Outlier+Detection.ipynb".
The first task is:
The first screen of the PixieApp looks like this:

Click Isolation Forest to display the following screen:

The map shows a map of all the BPS schools, with outliers in dark blue:

Following the same path for One Class SVM, it yields almost twice as many outliers in this data set. The map looks like this:

5. Explore the data in two-dimensions
Now that the PixieApp has identified outliers, you can look at the data in one or two dimensions using PixieDust.
Use PixieDust to display schools_df and navigate to scatterplot. This example uses "bokeh" as the renderer, and  the key and value fields are switched out while using is_outlier_iforest as color. PixieDust makes it very easy to explore scatterplots with different variables quickly.
Isolation Forest
First, look at Total Energy Cost against Square Feet. It's clear that IForest labeled two schools with very large square footage as outliers.
Can you write Python code for it?


I am working on the file "Planck-vs-WMAP.ipynb".
The first task is:
Make the GTC!
Produce the plot and save it as Planck-vs-WMAP.pdf.
Can you write Python code for it?

GTC = pygtc.plotGTC(chains=[Planckplot,WMAPplot],
                    weights=[Planckall[:,0],
                    WMAPall[:,0]],
                    paramNames=params,
                    chainLabels=chainLabels,
                    colorsOrder=('greens','blues'),
                    figureSize='APJ_page',
                    plotName='Planck-vs-WMAP.pdf')


I am working on the file "Planck-vs-WMAP.ipynb".
The first task is:
Read in and format the data
Can you write Python code for it?

WMAP, Planck = [],[]
for i in range(1,5):
    WMAP.append(np.loadtxt('./base/WMAP/base_WMAP_'+str(i)+'.txt'))
    Planck.append(np.loadtxt('./base/plikHM_TT_lowTEB/base_plikHM_TT_lowTEB_'+str(i)+'.txt'))

# Copy all four chains into a single array
WMAPall = np.concatenate((WMAP[0],WMAP[1],WMAP[2],WMAP[3]))
Planckall = np.concatenate((Planck[0],Planck[1],Planck[2],Planck[3]))


I am working on the file "Planck-vs-WMAP.ipynb".
The first task is:
Select the parameters and make labels
In the chain directories, there are .paramnames files that allow you
to find the parameters you are interested in.
Can you write Python code for it?

WMAPplot = WMAPall[:,[2,3,4,5,6,7,9,15]]
Planckplot = Planckall[:,[2,3,4,5,6,7,23,29]]

# Labels, pyGTC supports Tex enclosed in $..$
params = ('$\Omega_\mathrm{b}h^2$',
          '$\Omega_\mathrm{c}h^2$',
          '$100\\theta_\mathrm{MC}$',
          '$\\tau$',
          '$\ln(10^{10}A_s)$',
          '$n_s$','$H_0$',
          '$\\sigma_8$')

chainLabels = ('$Planck$ (TT+lowTEB)','WMAP')


I am working on the file "Play with the FORTH kernel on jupyter notebook.ipynb".
The first task is:
5. Define commands + , .s , and s".
These examples demonstrate project-k built-in methods push(), pop(), and nexttoken().
Can you write Python code for it?

vm.dictate("code + push(pop(1)+pop()) end-code");


I am working on the file "Play with the FORTH kernel on jupyter notebook.ipynb".
The first task is:
7. Get help of project-k global functions
Project-k built-in functions are explained with comments in the projectk.py source code. View them by using the python help() function.
Can you write Python code for it?

help(vm.tos)
help(vm.pop)
help(vm.push)
help(vm.nexttoken)


I am working on the file "Play with the FORTH kernel on jupyter notebook.ipynb".
The first task is:
8. Self-reference of a code word
Can you write Python code for it?

vm.dictate("code see-locals print(locals()) end-code see-locals");


I am working on the file "Play with the FORTH kernel on jupyter notebook.ipynb".
The first task is:
9. Let's see the globals
You will want to know about the globals when at the view point in a code word definition. Many of them have been seen above yet this is from a different view point.
Can you write Python code for it?

vm.dictate("code see-globals print(globals().keys()) end-code see-globals");


I am working on the file "Play with the FORTH kernel on jupyter notebook.ipynb".
The first task is:
A code word is defined between code and end-code. The first token after the leading code, which is 'hi' in this example, is the name of the new FORTH word. All the rests down to the ending end-code are python statements. Note after end-code above we execute hi the new word immediately and it works!
4. Define the 'words' command
'words' is a basic FORTH word that lists all words. This example shows you how to define it.
Can you write Python code for it?

vm.dictate("code words print([w.name for w in words['forth'][1:]]) end-code words");


I am working on the file "Play with the FORTH kernel on jupyter notebook.ipynb".
The first task is:
FORTH is a programming language of postfix-expression. "123 456 +" means: "Please push 123 into the data stack, please push 456 too. Now please get the top two cells out of the data stack, add them and push the result back to the data stack". The result is 579 left in the data stack.
The common FORTH word to view its data stack is .s, we can have it by this definition:
Can you write Python code for it?

vm.dictate("code .s print(stack) end-code");
vm.execute('.s');


I am working on the file "Play with the FORTH kernel on jupyter notebook.ipynb".
The first task is:
For example, this jupyter notebook itself is the main program we are running and through vm.push() we can pass this information into the FORTH vm and get it back by vm.pop():
Can you write Python code for it?

vm.push(__name__).pop()
vm.push(__IPYTHON__).pop()  # see another property from the main program


I am working on the file "Play with the FORTH kernel on jupyter notebook.ipynb".
The first task is:
The FORTH term TIB (terminal input buffer) is simply the string given from vm.dictate('I am the TIB'). nexttoken() is a project-k built-in function that gets a quote out of the TIB from the current position to the given delimiter which is " in the example above. So FORTH strings can now be expressed by s" this is a string" and that will be pushed to the FORTH data stack. Let's use it:
Can you write Python code for it?

# Put a string onto the TOS (Top of the data stack)
vm.dictate('s" The wise build bridges, " .s');

vm.dictate('s" while the foolish build barriers." .s');


I am working on the file "Play with the FORTH kernel on jupyter notebook.ipynb".
The first task is:
The _me object points to the new word itself. For example, this word prints its own name:
Can you write Python code for it?

vm.dictate("code who-am-I? print('My name is: ' + _me.name) end-code").execute('who-am-I?');


I am working on the file "Play with the FORTH kernel on jupyter notebook.ipynb".
The first task is:
The vm knows how to do the '+' now, let's try:
Can you write Python code for it?

vm.stack=[]  # clear the data stack
vm.dictate("123 456 +").stack


I am working on the file "Play with the FORTH kernel on jupyter notebook.ipynb".
The first task is:
and according to the + word we defined above, it can concatenate two strings too. So let's execute + and check the result on the data stack:
Can you write Python code for it?


I am working on the file "Play with the FORTH kernel on jupyter notebook.ipynb".
The first task is:
vm.dictate() method is the way project-k VM receives your commands (in a multiple lines string). It actually is also the way we feed it an entire Forth source code file. "123" means: "please push this number into the FORTH data stack". vm.stack is the Forth VM data stack which was empty at first and now has one item, 123, that we've just put in it. Some methods as vm.dictate() returns the vm object itself so we can cascade multiple function calls in one line. Therefore the above statement vm.dictate("123").stack is cascated from the two lines:
vm.dictate("123")
vm.stack

3. Use code and end-code to define a high-level FORTH word
In FORTH we call an identifier a word that is a command or a variable name. Now we can start to define a new word:
Can you write Python code for it?

vm.dictate("code hi print('Hello World!!') end-code hi");


I am working on the file "Playground.ipynb".
The first task is:
Operations
Dot product
Can you write Python code for it?

v1 = Vector(np.array([1,3]))
v2 = Vector(np.array([2,1]))
v1.dot(v2)


I am working on the file "Playground.ipynb".
The first task is:
Other
Changing basis
Can you write Python code for it?

fig, ax = plt.subplots()
v1 = Vector(np.array([1, 1]), base=np.array([5, 2]))
v2 = Vector(np.array([1, 1]), base=np.array([-2, 3]))
v1.plot(ax)
v2.plot(ax)


I am working on the file "Playground.ipynb".
The first task is:
Plotting multiple vectors in 2D
Can you write Python code for it?

fig, ax = plt.subplots()
v1 = Vector(np.array([1,2]))
v2 = Vector(np.array([5,-1]))
v1.plot(ax)
v2.plot(ax)


I am working on the file "Playground.ipynb".
The first task is:
Roadmap

<s>Addition and subtraction</s>
<s>Scaling (multiplication)</s>
<s>Visualizing in 2D</s>
<s>Visualizing in 3D</s>
Visualization legends
Visualize dot products as projections
Compute determinant
Cross products
Can you write Python code for it?

class Vector():
    """The base class for all vector operations"""

    def __init__(self, arr, base=np.array([1, 1])):
        self._arr = arr
        self.base = base

    def dot(self, other):
        return np.dot(self._arr, other._arr)

    def cross(self, other):
        return Vector(np.cross(self._arr, other._arr))

    def plot(self, ax=None):
        dims = len(self._arr)
        if dims > 3:
            raise Exception('Cannot plot over 3 dimensions')
        if not ax:
            fig = plt.figure()
            proj = '3d' if dims == 3 else None
            ax = fig.add_subplot(111, projection=proj)
        if dims == 1:
            self._plot1d(ax)
        elif dims == 2:
            self._plot2d(ax)
        elif dims == 3:
            self._plot3d(ax)

    def _plot2d(self, ax):
        x, y = self._arr * self.base
        ax.plot([0, x], [0, y])
        min_, max_ = min(x, y), max(x, y)
        ax.set_xlim([min(0, min_), max_])
        ax.set_ylim([min(0, min_), max_])

    def _plot2d_quiver(self, ax):
        """Work in progress."""
        x, y = self._arr
        ax.quiver(0, 0, x, y, angles='xy', scale_units='xy', scale=1)
        xmin = 0 if x >= 0 else x
        xmax = 0 if x <= 0 else x
        ymin = 0 if y >= 0 else y
        ymax = 0 if y <= 0 else y
        ax.set_xlim([xmin, xmax])
        ax.set_ylim([ymin, ymax])
        return ax

    def _plot3d(self, ax):
        x, y, z = self._arr
        ax.plot([0, x], [0, y], [0, z])

    def __add__(self, other):
        return Vector(self._arr + other._arr)

    def __sub__(self, other):
        return Vector(self._arr - other._arr)

    def __mul__(self, scalar):
        return self._arr * scalar

    def __eq__(self, other):
        return np.all(self._arr == other._arr)

    def _repr_png_(self):
        return display_png(self.plot())

    def __repr__(self):
        return 'vector({})'.format([x for x in self._arr])


I am working on the file "Playground/.ipynb_checkpoints/Bayesian regression demo-checkpoint.ipynb".
The first task is:
General guides to Bayesian regression
http://twiecki.github.io/blog/2015/11/10/mcmc-sampling/
PyMC
https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers
http://conference.scipy.org/scipy2014/schedule/presentation/1662/
Can you write Python code for it?

# Showing plots inline, rather than in a new window
%matplotlib inline

# Modules
from pymc3 import  *

import numpy as np
from ggplot import *

# Generating data
size = 200
true_intercept = 1
true_slope = 2

x = np.linspace(0, 1, size)
# y = a + b*x
true_regression_line = true_intercept + true_slope * x
# add noise
y = true_regression_line + np.random.normal(scale=.5, size=size)

# Plotting data
sim_data = pd.DataFrame({"x" : x, "y" : y})

sim_plot = ggplot(sim_data, aes(x="x", y="y")) + geom_point() +\
    geom_abline(intercept=true_intercept, slope=true_slope)

print(sim_plot)


I am working on the file "Playground/Bayesian regression demo.ipynb".
The first task is:
General guides to Bayesian regression
http://twiecki.github.io/blog/2015/11/10/mcmc-sampling/
PyMC
https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers
http://conference.scipy.org/scipy2014/schedule/presentation/1662/
Can you write Python code for it?

# Showing plots inline, rather than in a new window
%matplotlib inline

# Modules
from pymc3 import  *

import numpy as np
from ggplot import *

# Generating data
size = 200
true_intercept = 1
true_slope = 2

x = np.linspace(0, 1, size)
# y = a + b*x
true_regression_line = true_intercept + true_slope * x
# add noise
y = true_regression_line + np.random.normal(scale=.5, size=size)

# Plotting data
sim_data = pd.DataFrame({"x" : x, "y" : y})

sim_plot = ggplot(sim_data, aes(x="x", y="y")) + geom_point() +\
    geom_abline(intercept=true_intercept, slope=true_slope)

print(sim_plot)

with Model() as model:
    # specify glm and pass in data. The resulting linear model, its likelihood and
    # and all its parameters are automatically added to our model.
    glm.glm('y ~ x', data)
    step = NUTS() # Instantiate MCMC sampling algorithm
    trace = sample(2000, step, progressbar=False) # draw 2000 posterior samples using NUTS sampling

plt.figure(figsize=(7, 7))
traceplot(trace)
plt.tight_layout();

plt.figure(figsize=(7, 7))
plt.plot(x, y, 'x', label='data')
glm.plot_posterior_predictive(trace, samples=100,
                              label='posterior predictive regression lines')
plt.plot(x, true_regression_line, label='true regression line', lw=3., c='y')

plt.title('Posterior predictive regression lines')
plt.legend(loc=0)
plt.xlabel('x')
plt.ylabel('y');


I am working on the file "Playing With yt.ipynb".
The first task is:
Configuration for this figure.
Can you write Python code for it?

config = Foo()
config.name     = "HighAspect/HA_visc/HA_visc"
#config.arch_end = "alcf#dtn_mira/projects/alpha-nek"
config.arch_end = "maxhutch#alpha-admin/pub/"
config.frame = 1
config.lower = .25
config.upper = .75


I am working on the file "PlotCorr/PlotCorr.ipynb".
The first task is:
Download the master data files, if we don't already have them. Currently, they are stored in Tom's public Dropbox folder.
Can you write Python code for it?

targdir = 'data_files'
if not os.path.isdir(targdir):
    os.mkdir(targdir)

filenames = ('sdss_master.csv','snls_master.csv')

remotedir = 'https://dl.dropboxusercontent.com/u/5900205/empiriciSN_data/'

for filename in filenames:
    path = os.path.join(targdir, filename)
    url = os.path.join(remotedir, filename)

    #Always download the files, in case something has been updated.
    urllib.urlretrieve(url, path)


I am working on the file "PlotCorr/PlotCorr.ipynb".
The first task is:
Finally, plot the local color at the location of the SN. Here I used the surface brightness in each band and calculated a 2 arcsec aperture flux, which was then converted into a magnitude and used for color calculations. The aperture can be changed easily, so this can be redone if needed.
Like above, SNe with Exponential and deVaucouleurs hosts are plotted separately. There are a few cases where the surface brightness calculated to zero, or close to it, causing some issues with the magnitude calculation. These have not been plotted.
Can you write Python code for it?


I am working on the file "PlotCorr/PlotCorr.ipynb".
The first task is:
In all plots, SDSS SNe are displayed in blue and SNLS SNe are displayed in red. I am only including those SNLS SNe which had hosts that were also detected and modeled in SDSS, to keep the magnitudes and modeling consistent. For each SN, I show the SALT2 x0, x1, and c values, as well as the log of the separation from the host nucleus in units of the r-band effective radius.
First, plot SN properties vs. Host colors. All host mags have been K-corrected using kcorrect.
Can you write Python code for it?


I am working on the file "PlotCorr/PlotCorr.ipynb".
The first task is:
Read in the lines from each file.
Can you write Python code for it?

infile = open('data_files/sdss_master.csv','r')
sdss_lines = infile.readlines()
infile.close()

infile = open('data_files/snls_master.csv','r')
snls_lines = infile.readlines()
infile.close()


I am working on the file "Plot_TimeSeries1.ipynb".
The first task is:
4.1 Read variables and units
Can you write Python code for it?

with netCDF4.Dataset(datafile, 'r') as ds:
    time_values = ds.variables['TIME'][:]
    temperature_values = ds.variables['TEMP'][:]
    temperatureQC = ds.variables['TEMP_QC'][:]
    time_units = ds.variables['TIME'].units
    temperature_units = ds.variables['TEMP'].units
    time2 = netCDF4.num2date(time_values, time_units)


I am working on the file "Plot_TimeSeries1.ipynb".
The first task is:
4.2 Basic plot
We create the most simple plot, without any additional option.
Can you write Python code for it?

fig = plt.figure()
plt.plot(time2, temperature_values)
plt.ylabel(temperature_units)


I am working on the file "Plot_TimeSeries1.ipynb".
The first task is:
4.4 Final version
We want to add a title containing the coordinates of the station. Longitude and latitude are both stored as vectors, but we will only keep the mean position to be included in the title.
LaTeX syntax can be used, as in this example, with the degree symbol.
Can you write Python code for it?

with netCDF4.Dataset(datafile, 'r') as ds:
    lon = ds.variables['LONGITUDE'][:]
    lat = ds.variables['LATITUDE'][:]
figure_title = r'Temperature evolution at \n%s$^\circ$E, %s$^\circ$N' % (lon.mean(), lat.mean())
print figure_title


I am working on the file "Plot_TimeSeries1.ipynb".
The first task is:
Main problems:
* The figure is not large enough.
* The labels are too small.
4.3 Improved plot
With some commands the previous plot can be improved:
* The figure size is increased
* The font size is set to 20 (pts)
* The year labels are rotated 45º
Can you write Python code for it?

mpl.rcParams.update({'font.size': 20})

fig = plt.figure(figsize=(15, 8))
ax = fig.add_subplot(111)
plt.plot(time2, temperature_values, linewidth=0.5)
plt.ylabel(temperature_units)
plt.xlabel('Year')
fig.autofmt_xdate()
plt.grid()


I am working on the file "Plot_TimeSeries1.ipynb".
The first task is:
The units for the temperature are also changed:
Can you write Python code for it?

temperature_units2 = '($^{\circ}$C)'

fig = plt.figure(figsize=(15, 8))
ax = fig.add_subplot(111)
ax.xaxis.set_major_locator(dates.YearLocator(base=2))
ax.xaxis.set_minor_locator(dates.YearLocator())
plt.plot(time2, temperature_values, linewidth=0.5)
plt.ylabel(temperature_units2, rotation=0., horizontalalignment='right')
plt.title(figure_title)
plt.xlabel('Year')
fig.autofmt_xdate()
plt.grid()


I am working on the file "Plotly-Mesh3d.ipynb".
The first task is:
Compute the circumcenter and circumradius of a triangle (see their definitions here):
Can you write Python code for it?

def circumcircle(points,simplex):
    A = [points[simplex[k]] for k in range(3)]
    M = [[1.0]*4]
    M += [[sq_norm(A[k]), A[k][0], A[k][1], 1.0 ] for k in range(3)]
    M = np.asarray(M, dtype=np.float32)
    S = np.array([0.5*np.linalg.det(M[1:, [0,2,3]]), -0.5*np.linalg.det(M[1:, [0,1,3]])])
    a = np.linalg.det(M[1:, 1:])
    b = np.linalg.det(M[1:, [0,1,2]])
    return S/a,  np.sqrt(b/a + sq_norm(S)/a**2) #center=S/a, radius=np.sqrt(b/a+sq_norm(S)/a**2)


I am working on the file "Plotly-Mesh3d.ipynb".
The first task is:
Define two traces: one for plotting the point set and another for the alpha shape:
Can you write Python code for it?

points = go.Scatter3d(mode='markers',
                      name='',
                      x =x,
                      y= y,
                      z= z,
                      marker=dict(size=2, color='#458B00'))

simplexes = go.Mesh3d(alphahull =10.0,
                      name = '',
                      x =x,
                      y= y,
                      z= z,
                      color='#90EE90',
                      opacity=0.15)

axis = dict(showbackground=True,
            backgroundcolor="rgb(245, 245, 245)",
            gridcolor="rgb(255, 255, 255)",
            gridwidth=2,
            zerolinecolor="rgb(255, 255, 255)",
            tickfont=dict(size=11),
            titlefont =dict(size=12))

x_style = dict(axis, range=[-2.85, 4.25], tickvals=np.linspace(-2.85, 4.25, 5)[1:].round(1))
y_style = dict(axis, range=[-2.65, 1.32], tickvals=np.linspace(-2.65, 1.32, 4)[1:].round(1))
z_style = dict(axis, range=[-3.67,1.4], tickvals=np.linspace(-3.67, 1.4, 5).round(1))

layout = go.Layout(title='Alpha shape of a set of 3D points. Alpha=0.1',
                   width=500,
                   height=500,
                   scene=dict(xaxis=x_style,
                              yaxis=y_style,
                              zaxis=z_style))

fig = go.FigureWidget(data=[points, simplexes], layout=layout)
#fig

fig = go.FigureWidget(data=[points, simplexes], layout=layout)
#py.plot(fig, filename='3D-AlphaS-ex')

IFrame('https://plot.ly/~empet/13499/',  width=550, height=550)


I am working on the file "Plotly-Mesh3d.ipynb".
The first task is:
Filter out the Delaunay triangulation to get the $\alpha$-complex:
Can you write Python code for it?

def get_alpha_complex(alpha, points, simplexes):
    #alpha is the parameter for the alpha shape
    #points are given data points
    #simplexes is the  list of indices in the array of points
    #that define 2-simplexes in the Delaunay triangulation

    return filter(lambda simplex: circumcircle(points,simplex)[1] < alpha, simplexes)

pts = np.loadtxt('Data/data-ex-2d.txt')
tri = Delaunay(pts)

colors = ['#C0223B', '#404ca0', 'rgba(173,216,230, 0.5)']# colors for vertices, edges and 2-simplexes


I am working on the file "Plotly-Mesh3d.ipynb".
The first task is:
Get data for the Plotly plot of a subcomplex of the Delaunay triangulation:
Can you write Python code for it?

def Plotly_data(points, complex_s):
    #points are the given data points,
    #complex_s is the list of indices in the array of points defining 2-simplexes(triangles)
    #in the simplicial complex to be plotted
    X = []
    Y = []
    for s in complex_s:
        X += [points[s[k]][0] for k in [0,1,2,0]] + [None]
        Y += [points[s[k]][1] for k in [0,1,2,0]] + [None]
    return X, Y

def make_trace(x, y,  point_color=colors[0], line_color=colors[1]):# define the trace
                                                                   #for an alpha complex
    return go.Scatter(mode='markers+lines', #vertices and
                                            #edges of the alpha-complex
                      name='',
                      x=x,
                      y=y,
                      marker=dict(size=6.5, color=point_color),
                      line=dict(width=1.25, color=line_color))

figure = tls.make_subplots(rows=1, cols=2,
                           subplot_titles=('Delaunay triangulation',  'Alpha shape, alpha=0.15'),
                           horizontal_spacing=0.1,
                          )

title = 'Delaunay triangulation and Alpha Complex/Shape for a Set of 2D Points'

figure.layout.update(title=title,
                     font=dict(family="Open Sans, sans-serif"),
                     showlegend=False,
                     hovermode='closest',
                     autosize=False,
                     width=800,
                     height=460,
                     margin=dict(l=65,
                                 r=65,
                                 b=85,
                                 t=120));

axis_style = dict(showline=True,
                  mirror=True,
                  zeroline=False,
                  showgrid=False,
                  showticklabels=True,
                  range=[-0.1,1.1],
                  tickvals=[0, 0.2, 0.4, 0.6, 0.8, 1.0],
                  ticklen=5
                 )

for s in...


I am working on the file "Plotly-Mesh3d.ipynb".
The first task is:
If $T$ is the set of points defining a $k$-simplex, then any proper subset of $T$ defines an $\ell$-simplex, $\ell<k$.
These $\ell$-simplexes (or $\ell$-simplices) are called faces.
A 2-simplex has three $1$-simplexes, and three 0-simplexes as faces, whereas a tetrahedron has as faces three  2-simplexes, six 1-simplexes and four zero simplexes.
k-simplexes are building blocks for different structures in Computational Geometry, mainly for creating meshes from point clouds.
Let $S$ be a finite set in $\mathbb{R}^d$, $d=2,3$ (i.e. a set of 2D or 3D points). A collection  $\mathcal{K}$ of k-simplexes, $0\leq k\leq d$,  having as vertices  the  points of $S$,
 is a simplicial complex if its simplexes have the following properties:
  1. If $\sigma$ is a simplex in $\mathcal{K}$, then all its faces are also simplexes in $\mathcal{K}$;
  2. If $\sigma, \tau$ are two simplexes in $\mathcal{K}$, then their intersection is either empty or a face in both simplexes.
The next figure illustrates a simplicial complex(left), and a collection of $k$-simplexes (right), $0\leq k\leq 2$
that do not form a simplicial complex because the condition 2 in the definition above is violated.
Can you write Python code for it?

IFrame('https://plot.ly/~empet/13503/',  width=600, height=475)


I am working on the file "Plotly-Mesh3d.ipynb".
The first task is:
We notice in the subplots above that as alphahull increases, i.e. $\alpha$ decreases, some parts of the alpha shape shrink and
develop enclosed void regions. The last plotted alpha shape points out a polytope that contains faces of tetrahedra,
and patches of triangles.
In some cases as $\alpha$ varies it is also possible to develop  components that are strings of edges and even isolated points.
Such experimental results suggested the use of alpha shapes in modeling molecular structure.
A search on WEB gives many results related to applications of alpha shapes in structural molecular biology.
Here
is an alpha shape illustrating a molecular-like structure associated to a point set of 5000 points.
Generating an alpha shape with Mesh3d
Can you write Python code for it?

import numpy as np
import plotly.plotly as py
import plotly.graph_objs as go
from plotly import tools as tls