I am working on the file "FP-Bezier-Bspline.ipynb".
The first task is:
3. The function redefineBezier can  also be invoked to compute the left and right subpolygons resulted from a subdivision of a  Bézier curve,
at a point corresponding to the paramater $s$. Namely the left subpolygon is returned by
redefineBezier(b,  0, s), whereas the right subpolygon by redefineBezier(b,  s, 1)
B-spline curves
We give a procedural definition of a B-spline curve of degree $k$, not an analytical one.
The following data:

an interval $[a,b]$;

an integer $k\geq 1$;

a sequence of knots:
$$u_0 = u_1 = \cdots = u_k< u_{k+1} \leq \cdots \leq u_{m−k−1} < u_{m−k} = \cdots = u_{m−1} = u_m$$

with  $u_0 = a, u_m = b$,    $m−k > k$, and each knot $u_{k+1}, \ldots, u_{m-k-1}$ of  multiplicity at most $k$;
- the control points ${\bf d}0, {\bf d}_1, \ldots, {\bf d}{m-k-1}$, called de Boor points;
define a curve $s:[a,b]\to\mathbb{R}^2$, such that for each $t$ in an interval $[u_J, u_{J+1}]$, with
$u_J<u_{J+1}$, $s(t)$ is computed in the last step of the Cox-de Boor recursive formula [Gallier]:
 $$\begin{array}{lll}
{\bf d}^0_i&=&{\bf d}i,\quad i=\overline{J-k,J}\
 {\bf d}_i^r(t)&=&(1-\omega_i^r(t))\,{\bf d}{i-1}^{r-1}(t)+\omega_i^r(t)\,{\bf d}{i}^{r-1}(t),\,\,
r=\overline{1, k}, i=\overline{J-k, J}\end{array}$$
with $\omega_i^r(t)=\displaystyle\frac{t-u{i}}{u_{i+k-r+1}-u_{i}}$
The curve defined in this way is called B-spline curve. It is a piecewise polynomial curve of degree
 at most $k$, i.e. it is a polynomial curve  on each nondegenerate interval $[u_J, u_{J+1}]$, and at each knot $u_J$ of multiplicity $1\leq p\leq k$, it is of class $C^{k-p}$.
The points computed in the steps $r=\overline{1,k}$ of the recursion can be written in a lower triangular matrix:
$$
\begin{array}{ccccc}
{\bf d}^0_{0}&             &   &   &  \
{\bf d}^0_{1} & {\bf d}{1}^1&   &   &   \
\vdots    & \vdots     &   &   &   \
{\bf d}^0{k-1}     & {\bf d}{k-1}^1    &\ldots & {\bf...
Can you write Python code for it?

from functools import partial

def Omegas(u, k, t):
    # compute  a list of lists
    #an inner list contains the values $\omega_j^r(t)$ values from a step r

    #if (len(u)!=2*k+1):
        #raise InvalidInputError('the list u must have length 2k+1')
    return map(lambda r: map(partial(lambda r, j: (t-u[j])/(u[j+k-r+1]-u[j]), r), \
                         xrange(r,k+1)), xrange(1, k+1))


I am working on the file "FP-Bezier-Bspline.ipynb".
The first task is:
After these experiments our conclusion is that you can insert functional programming code in your imperative  Python code, but trying to write pure functions could be  difficult, sometimes impossible.
Writing FP style Python code is challenging, but the  code is not so easy  to read, unlike the imperative Python code which is characterized as a readable one.
The next step in experimenting FP will be to use a  Python implementation of  monads (see for example  Pymonad or fn.py.
In this IPython Notebook we avoided to raise errors and commented the lines handling errors.
The  algorithms implemented in this IPython Notebook are presented in detail in:
 1. G Farin,  Curves and Surfaces for Computer Aided Geometric Design: A Practical Guide,   Morgan Kaufmann, 2002.
 2. J. Gallier, Curves and Surfaces in Geometric Modeling: Theory and Algorithms,  Morgan Kaufmann, 1999. Free electronic version can be downloaded here.
2/10/2015
Can you write Python code for it?

from IPython.core.display import HTML
def  css_styling():
    styles = open("./custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "FP-Bezier-Bspline.ipynb".
The first task is:
Define a function to plot the subpolygons:
Can you write Python code for it?

def plot_polygon(pol, ch):# plot a  control polygon computed  by an algorithm from a Bezier curve

    plt.plot(zip(*pol)[0], zip(*pol)[1], linestyle='-',  marker='o', color=ch)


I am working on the file "FP-Bezier-Bspline.ipynb".
The first task is:
For a FP implementation of de Casteljau algorithm we use standard Python.
First we define functions that return an affine/ convex combination of two numbers, two 2D points, respectively of each pair of  consecutive points in a list of 2D control points:
Can you write Python code for it?

cvx=lambda x, y, t: (1-t) * x + t * y # affine/convex combination of two numbers x, y

cvxP=lambda (P, Q, t): (cvx(P[0], Q[0], t), cvx(P[1], Q[1], t))# affine/cvx comb of two points P,Q

def cvxCtrlP(ctrl, t):# affine/cvx combination of each two consecutive points in a list ctrl
                      # with the same coefficient t
    return map(cvxP, zip(ctrl[:-1], ctrl[1:], [t]*(len(ctrl)-1)))


I am working on the file "FP-Bezier-Bspline.ipynb".
The first task is:
Noticing that in each step $r>1$, the coefficients $\omega^r_i$ are computed with the same formula as for
$r=1$, i.e.  $\omega_i^r=\displaystyle\frac{t-u_i}{u_i+k-u_i}$ but for the  knots in the list
u[r-1: -r+1] and $k=k-r+1$ we
define the function omega below, instead of Omegas:
Can you write Python code for it?

def omega(u, k, t):
    # defines the list of coefficients for the convex combinations performed in a step of de Boor algo

    #if (len(u)!=2*k+1 or t<u[k] or t>u[k+1]):
        #raise InvalidInputError('the list u  has not the length 2k+1 or t isn't within right interval')

    return map(lambda j: (t-u[j]) / (u[j+k]-u[j]),  xrange(1,k+1))


I am working on the file "FP-Bezier-Bspline.ipynb".
The first task is:
Now let us test this function:
Can you write Python code for it?

Bez=BuildB()
Bez.B_ax()

br=redefineBezier(Bez.ctrl,  0.3, 0.67)
plot_polygon(br, 'g')


I am working on the file "FP-Bezier-Bspline.ipynb".
The first task is:
This is a typical imperative programming code: assignment statements,  for looping.
Each call of this function copies the control points into a numpy array a.
The convex combinations a[i,:] =(1-t) *a[i,:]+t*a[i-1,:], $i=\overline{1,n-r}$, that must be calculated in each step $r$
are vectorized, computing convex combinations of points from two slices in  the numpy.array  a.
A discrete version of a Bezier curve, of nr points is computed by this function:
Can you write Python code for it?

def BezierCv(b, nr=200):# compute nr points on the Bezier curve of control points in list bom
    t=np.linspace(0, 1, nr)
    return [deCasteljauImp(b, t[k]) for k in range(nr)]


I am working on the file "FP-Bezier-Bspline.ipynb".
The first task is:
To get the left subpolygon we exploit the invariance of a Bézier curve to reversing its control points.
The Bézier curve defined by the control points
${\bf b}n,{\bf b}{n-1}, \ldots, {\bf b}0$, coincides with that defined by
${\bf b}_0,{\bf b}{1}, \ldots, {\bf b}_n$.
If $p$ is the Bernstein parameterization of the curve defined by
${\bf b}0,{\bf b}{1}, \ldots, {\bf b}n$
 and $\tilde{p}$ of that  defined by the reversed control polygon, ${\bf b}_n,{\bf b}{n-1}, \ldots, {\bf b}_0$, then $p(t)=\tilde{p}(1-t)$ [Farin].
This means that the left subpolygon of the subdivision of the former curve at  $s$,
    is the right subpolygon resulted by dividing the latter curve  at $1-s$.
Now  we can define the function  that returns the left and right subpolygon of a Bézier curve subdivision:
Can you write Python code for it?

def subdivision(b, s):
    #returns the left and right subpolygon, resulted dividing the curve of control points b, at s
        #if(s<=0 or s>=1):
            #raise InvalidInputError('The subdivision parameter must be in the interval (0,1)')

        return (get_right_subpol( b[::-1], 1-s, right=[]), get_right_subpol( b, s, right=[]))


I am working on the file "FP-Bezier-Bspline.ipynb".
The first task is:
We also need a new function that calculates the convex combinations of each pair of points in a list,
with distinct coefficients given in a list alpha:
Can you write Python code for it?

def cvxList(d, alpha):
    #len(d)=len(alpha)+1
    return  map(cvxP, zip(d[:-1], d[1:], alpha))


I am working on the file "FP-Bezier-Bspline.ipynb".
The first task is:
We build the control polygon and draw the corresponding closed B-spline curve:
Can you write Python code for it?


I am working on the file "FP-Bezier-Bspline.ipynb".
The first task is:
We define two functions that implement this recursive scheme. In both cases
we consider  2D points as tuples of float numbers and control polygons  as lists of tuples.
First an imperative programming implementation of the de Casteljau algorithm:
Can you write Python code for it?

import numpy as np

class InvalidInputError(Exception):
    pass

def deCasteljauImp(b,t):
    N=len(b)
    if(N<2):
        raise InvalidInputError("The  control polygon must have at least two points")
    a=np.copy(b) #shallow copy of the list of control points and its conversion to a numpy.array
    for r in range(1,N):
        a[:N-r,:]=(1-t)*a[:N-r,:]+t*a[1:N-r+1,:]# convex combinations in step r
    return a[0,:]


I am working on the file "FP-Bezier-Bspline.ipynb".
The first task is:
map(lambda s: deCasteljauF(b,s),  map(lambda j: j*1.0/(nr-1), xrange(nr)) means that the function
     deCasteljauF with fixed list of control points, b, and the variable parameter s is mapped to the list
     of  parameters $t_j$ defined above. Instead of defining the list of parameters through comprehension [j*1.0/(nr-1) for j in xrange(nr)]
     we created it calling the  higher order function map:
map(lambda j: j*1.0/(nr-1), xrange(nr))
Obvioulsly the FP versions of de Casteljau algorithm   and BezCurve are more compact
than the corresponding imperative versions, but they are not quite readable at the first sight.
To choose interactively the control points of a Bézier curve (and later of a B-spline curve) and  to plot the curve, we set now  the matplotlib nbagg backend, and  define the class BuildB, below:
Can you write Python code for it?

%matplotlib notebook

import matplotlib.pyplot as plt

def Curve_plot(ctrl, func):
    # plot the control polygon and the corresponding curve discretized by the function func
    xc, yc = zip(*func(ctrl))
    xd, yd = zip(*ctrl)
    plt.plot(xd,yd , 'bo-', xc,yc, 'r')

class BuildB(object): #Build a Bezier/B-spline Curve

    def __init__(self, xlims=[0,1], ylims=[0,1], func=BezCurve):
        self.ctrl=[]     # list of control points
        self.xlims=xlims # limits for x coordinate of control points
        self.ylims=ylims # limits for y coordinate
        self.func=func   # func - function that discretizes a curve defined by the control polygon ctrl

    def callback(self, event):  #select control points with  left  mouse button click
        if event.button==1 and event.inaxes:
            x,y = event.xdata,event.ydata
            self.ctrl.append((x,y))
            plt.plot(x, y,  'bo')

        elif event.button==3:  #press right  button to plot the curve
            Curve_plot(self.ctrl, self.func)
            plt.draw()

        else: pass

    def B_ax(self):#define axes lims

        fig = plt.figure(figsize=(6, 5))
        ax = fig.add_subplot(111)
        ax.set_xlim(self.xlims)
        ax.set_ylim(self.ylims)
        ax.grid('on')
        ax.set_autoscale_on(False)
        fig.canvas.mpl_connect('button_press_event', self.callback)


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
2) Photograph the front of the assembly.  Use a service like Flickr to upload your photo and paste the URL below, in place of the placeholder image.
You will need to double-click the active area to the left of the image to see the embedded link, then delete it and replace it with your new link.  For Flickr, use the link given at "Share Photo" then "Embed".  Hit shift-enter to run the cell and see the image.

3) Photograph the back of the assembly and upload it also. Paste the URL below, in place of the placeholder image.

Standard Inspections

Weigh assembly, note any non-flight configurations
Can you write Python code for it?

test_check["Assembly Weight"] = None
test_check["Non-Flight Configurations"] = None


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
2) Take a set of housekeeping data.
Can you write Python code for it?

from tessfpe.data.housekeeping_channels import housekeeping_channels
housekeeping_channels

# TODO compare measured to reference values


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
3) Capture FLIR images; check for hot spots. Use a service like Flickr to upload your photo and paste the URL below, in place of the placeholder image.
You will need to double-click the active area to the left of the image to see the embedded link, then delete it and replace it with your new link.  For Flickr, use the link given at "Share Photo" then "Embed".  Hit shift-enter to run the cell and see the image.

Testing basic FPGA functions and Loading Default Parameters
Start the Observatory Simulator and Load the FPE FPGA
Remember that whenever you power-cycle the Observatory Simulator, you should set preload=True below.
When you are running this notebook and it has not been power cycled, you should set preload=False.
Make sure that FPE has both power and data cables connected, and make sure that the FPE is connected to the "FPE 1" socket on the observatory simulator. Execute the cell below to program the FPGA:
Can you write Python code for it?

from tessfpe.dhu.fpe import FPE
from tessfpe.dhu.unit_tests import check_house_keeping_voltages
fpe1 = FPE(1, debug=False, preload=True, FPE_Wrapper_version='6.1.1')
print fpe1.version
if check_house_keeping_voltages(fpe1):
    print "Wrapper load complete. Interface voltages OK."


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
6) Capture an image and display it. Note that with only the Interface Board connected, the captured image should be comprised entirely of pixels with a value of -1.
Can you write Python code for it?


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
7) Issue the stop frames command.
Can you write Python code for it?


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
7) With power OFF, measure the resistance to ground of all pins on the JS stack connector.
Set up the dictionary to hold the results:
Can you write Python code for it?

from collections import defaultdict
test_check["JS stack connector resistances"] = defaultdict(dict)


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
<pre>
- FPE Test Procedures
    - FPE Bring-up Procedure (Check boxes for board type? Flight or not?)
        - Verify work area is ESD safe
        - Setup per diagram, take photos
        - Note test equipment model #'s and serial #'s
        - Standard inspections for all 3 PCB types; capture images
            - Weigh assembly, note non-flight configurations
            - Visual inspection under stereo microscope
                - Workmanship and mechanical damage
                - DNP parts not installed
                - No missing components
                Verify req'd jumpers installed
                Component orientation (chips, polarized caps, diodes, etc)
                Verify chips are correct parts (& date codes if specified in design)
                Verify connector savers installed if req'd

        Power OFF resistance measurements (compare to reference values)
            Power lines
            Stack connector
            CCD connectors (video only) Maybe delete this? Discuss.
            Temp connector (video only)
        Power ON voltage measurements (compare to reference values/images)
            DHU supply voltages before connection to setup
            DHU supply voltages and currents with setup connected
            Capture FLIR images; check for hot spots
            Program FPGA
            Start frames
                Take raw image as verification that FPGA was programmed OK
            Stop frames
                Take HK data as further verification
                    ref values for 3 cases: Interface only, Interface + Driver, full stack)
            Measure voltages on all open connector pins (= PCB safe-to-mate)
                Interface: stack connector only
                Driver: stack connector only
                Video:
                    Stack connector
                    CCD connectors (discuss need for this)
                    Temp connector
        *Bring up complete*
    FPE Functional Test and calibration
        (Assume for now we test full stack, not...
Can you write Python code for it?

# TODO: Recursively search test_check to make sure that no value is `None`

def url_exists(site, path):
    # TODO: Fix me so that site is broken up from path
    import httplib
    conn = httplib.HTTPConnection(site)
    conn.request('HEAD', path)
    response = conn.getresponse()
    conn.close()
    return response.status == 200

assert type(test_check["NAME"]) is str, "Name should be entered as a string"
assert type(test_check["EMAIL"]) is str, "Email should be entered as a string"
# TODO: check that email is a valid email with a regex
# assert type(test_check["EMAIL"]) is str, "Email should be <blah>@<blah>.(com|edu|org|net|gov)"

assert type(test_check["Part Number"]) is str
assert type(test_check["Serial Number"]) is str

# TODO: check that "DATE" is valid time stamp of the form "MM/DD/YY"
# TODO: check that "ESD_SAFE" is the string 'ESD Safe'.

# assert type(test_check["EMAIL"]) is str, "Email should be <blah>@<blah>.(com|edu|org|net|gov)"

assert test_check["FRONT_ASSEMBLY_PHOTO"] not in placeholder_photos, "Front assembly photo should not be a stock photo"
#assert url_exists(test_check["FRONT_ASSEMBLY_PHOTO"]), "URL for front assembly photo should exist"
assert test_check["BACK_ASSEMBLY_PHOTO"] not in placeholder_photos, "Back assembly photo should not be a stock photo"
#assert url_exists(test_check["BACK_ASSEMBLY_PHOTO"]), "URL for back assembly photo should exist

assert 'Multimeter' in test_check["Equipment"], "'Multimeter' should be in test_check['Equipment']"
assert 'Oscilloscope' in test_check["Equipment"], "'Oscilloscope' should be in test_check['Equipment']"
assert 'DHU Emulator' in test_check["Equipment"], "'DHU Emulator' should be in test_check['Equipment']"
assert len(test_check["Equipment"]) == 3, "test_check['Equipment'] should not contain superfluous information"
assert type(test_check["Equipment"]["Multimeter"]["Model Number"]) is str, 'Multimeter model number should be a string'
assert type(test_check["Equipment"]["Multimeter"]["Serial Number"]) is str, 'Multimeter serial number...


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Executing the next cell will compare the recorded values above to the expected values, within a margin of +/- 10%.  The output should not produce any errors.
Can you write Python code for it?

assert test_check["JS stack connector resistances"][1]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][2]["SDO-A-2"] <= 1.98
assert test_check["JS stack connector resistances"][3]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][4]["SDO-B-1"] <= 1.98
assert test_check["JS stack connector resistances"][5]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][6]["SDO-C-1"] <= 1.98
assert test_check["JS stack connector resistances"][7]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][8]["SDO-D-1"] <= 1.98
assert test_check["JS stack connector resistances"][9]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][10]["SCK"] <= 1.98
assert test_check["JS stack connector resistances"][11]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][12]["CNV"] <= 1.98
assert test_check["JS stack connector resistances"][13]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][14]["INT"] <= 1.98
assert test_check["JS stack connector resistances"][15]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][16]["DEINT"] <= 1.98
assert test_check["JS stack connector resistances"][17]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][18]["CLAMP"] <= 1.98
assert test_check["JS stack connector resistances"][19]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][20]["CWCLK"] <= 1.98
assert test_check["JS stack connector resistances"][21]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][22]["DD"] <= 1.98
assert test_check["JS stack connector resistances"][23]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][24]["DCK"] <= 1.98
assert test_check["JS stack connector resistances"][25]["GND"] <= 0.2
assert 1.62 <= test_check["JS stack connector resistances"][26]["SP1OR"] <= 1.98
assert test_check["JS stack connector resistances"][27]["GND"] <= 0.2
assert 1.62 <=...


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Now enter the measurement data (replace the word "None"). Note that expected units are given:
Can you write Python code for it?

test_check["JS stack connector resistances"][1]["GND"] = None # Ohms
test_check["JS stack connector resistances"][2]["SDO-A-1"] = None # MOhms
test_check["JS stack connector resistances"][3]["GND"] = None # Ohms
test_check["JS stack connector resistances"][4]["SDO-B-1"] = None # MOhms
test_check["JS stack connector resistances"][5]["GND"] = None # Ohms
test_check["JS stack connector resistances"][6]["SDO-C-1"] = None # MOhms
test_check["JS stack connector resistances"][7]["GND"] = None # Ohms
test_check["JS stack connector resistances"][8]["SDO-D-1"] = None # MOhms
test_check["JS stack connector resistances"][9]["GND"] = None # Ohms
test_check["JS stack connector resistances"][10]["SCK"] = None # MOhms
test_check["JS stack connector resistances"][11]["GND"] = None # Ohms
test_check["JS stack connector resistances"][12]["CNV"] = None # MOhms
test_check["JS stack connector resistances"][13]["GND"] = None # Ohms
test_check["JS stack connector resistances"][14]["INT"] = None # MOhms
test_check["JS stack connector resistances"][15]["GND"] = None # Ohms
test_check["JS stack connector resistances"][16]["DEINT"] = None # MOhms
test_check["JS stack connector resistances"][17]["GND"] = None # Ohms
test_check["JS stack connector resistances"][18]["CLAMP"] = None # MOhms
test_check["JS stack connector resistances"][19]["GND"] = None # Ohms
test_check["JS stack connector resistances"][20]["CWCLK"] = None # MOhms
test_check["JS stack connector resistances"][21]["GND"] = None # Ohms
test_check["JS stack connector resistances"][22]["DD"] = None # MOhms
test_check["JS stack connector resistances"][23]["GND"] = None # Ohms
test_check["JS stack connector resistances"][24]["DCK"] = None # MOhms
test_check["JS stack connector resistances"][25]["GND"] = None # Ohms
test_check["JS stack connector resistances"][26]["SP1OR"] = None # MOhms
test_check["JS stack connector resistances"][27]["GND"] = None # Ohms
test_check["JS stack connector resistances"][28]["SP2OR"] = None # MOhms
test_check["JS stack connector...


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Now see if the values are within tolerance:
Can you write Python code for it?

assert 4.75 <= test_check["Power On Measurements"]["+5V"] <= 5.25
assert 14.25 <= test_check["Power On Measurements"]["+15V"] <= 15.75
assert -12.6 <= test_check["Power On Measurements"]["-12V"] <= -11.4


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Power OFF Resistance Measurements
6) With the power off and no external connections to the PCB, measure the resistance on all power lines.  All measurements should be referenced to circuit ground.
Can you write Python code for it?

test_check["Power Off Measurements"] = {}

test_check["Power Off Measurements"]["GND"] = None
test_check["Power Off Measurements"]["+5V"] = None
test_check["Power Off Measurements"]["+15V"] = None
test_check["Power Off Measurements"]["-12V"] = None
test_check["Power Off Measurements"]["+24"] = None
test_check["Power Off Measurements"]["-50"] = None
test_check["Power Off Measurements"]["+1.8F"] = None
test_check["Power Off Measurements"]["+1F"] = None
test_check["Power Off Measurements"]["+2.5"] = None
test_check["Power Off Measurements"]["+3.3B"] = None
test_check["Power Off Measurements"]["+3.3Dac"] = None
test_check["Power Off Measurements"]["+3.3F"] = None

#TODO: Add check of power off resistance measurements.


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Power On Voltage Measurements
1) Verify that voltages on power connector from DHU Emulator are correct before connecting to test assembly. The relevant FPE 1 & FPE 2 connector pins are wired as follows:
pin 1 & 5: GND
pin 2 & 7: +15V
pin 3 & 8: +5V
pin 4 & 9: -12V
(pin 6 is NC in 6.2 Interface design, but DHU FPE1 connector pin 6 is wired to FPE2 connector pin 6.)
Set up the dictionary to hold the results:
Can you write Python code for it?


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Prepare for Inspections
1) Verify work area is ESD safe.
Can you write Python code for it?

test_check["ESD_SAFE"] =  None # TODO: If the area is ESD safe enter 'ESD Safe'


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Record the voltage and current measurements:
Can you write Python code for it?

test_check["Power On Measurements Connected"]["+5V"] = None # Volts
test_check["Power On Measurements Connected"]["+5V Current"] = None # milliamps
test_check["Power On Measurements Connected"]["+15V"] = None # Volts
test_check["Power On Measurements Connected"]["+15V Current"] = None # milliamps
test_check["Power On Measurements Connected"]["-12V"] = None # Volts
test_check["Power On Measurements Connected"]["12V Current"] = None # milliamps

assert 4.75 <= test_check["Power On Measurements"]["+5V"] <= 5.25
assert 187 <= test_check["Power On Measurements Connected"]["+5V Current"] <= 207
assert 14.25 <= test_check["Power On Measurements"]["+15V"] <= 15.75
assert 12.8 <= test_check["Power On Measurements Connected"]["+15V Current"] <= 14.2
assert -12.6 <= test_check["Power On Measurements"]["-12V"] <= -11.4
assert 24.1 <= test_check["Power On Measurements Connected"]["12V Current"] <= 26.7


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Record the voltage measurements:
Can you write Python code for it?

test_check["Power On Measurements"]["+5V"] = None # Volts
test_check["Power On Measurements"]["+15V"] =  None # Volts
test_check["Power On Measurements"]["-12V"] =  None # Volts


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Set the operating parameters to their default values:
Can you write Python code for it?

from tessfpe.data.operating_parameters import operating_parameters
def set_fpe_defaults(fpe):
    "Set the FPE to the default operating parameters and return a list of the default values"
    defaults = {}
    for k in range(len(fpe.ops.address)):
        if fpe.ops.address[k] is None:
            continue
        fpe.ops.address[k].value = fpe.ops.address[k].default
        defaults[fpe.ops.address[k].name] = fpe.ops.address[k].default
    return defaults

# Print out the default values
set_fpe_defaults(fpe1)


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Summary
Below is a summary of test results and notes:
Can you write Python code for it?

# TODO: pretty print summary of data, consider incorporating
# - Visual inspection results
# - Operator Notes


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Test Equipment
Note test equipment model #'s and serial #'s
Can you write Python code for it?

test_check["Equipment"] = {
    "Multimeter": { "Model Number": None, # TODO: Enter a string here for the model number
                    "Serial Number": None # TODO: Enter a string here for the serial number
                  },
    "Oscilloscope": { "Model Number": None, # TODO: Enter a string here for the model number
                      "Serial Number": None # TODO: Enter a string here for the serial number
                    },
    "DHU Emulator": { "Model Number": None, # TODO: Enter a string here for the model number
                      "Serial Number": None # TODO: Enter a string here for the serial number
                    }
}


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Test Start Date
Can you write Python code for it?

test_check["DATE"] = None # TODO: Fill in today's date here as a string in the form "MM/DD/YY"


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Verify the image capturing function:
5) Issue the start frames command.
Can you write Python code for it?

# TODO Insert start_frames command

fpe1.cmd_stop_frames()


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
Visual Inspection under Stereo Microscope
Can you write Python code for it?


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
５.b DNP parts not installed
Can you write Python code for it?

test_check["Visual Inspection"]["DNP parts not installed"] = None # TODO: If no parts are missing write 'None missing'


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
５.c No missing components
Can you write Python code for it?

test_check["Visual Inspection"]["No Missing Components"] = None # TODO: Set to True if no components are missing


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
５.e Component orientation (chips, polarized caps, diodes, etc)
TODO: Make a labeled picture of component orientation
Can you write Python code for it?

test_check["Visual Inspection"]["Components are oriented correctly"] = None # TODO: Set to True if all components are oriented correctly


I am working on the file "FPE Interface Bring-Up Procedure.ipynb".
The first task is:
５.f Verify chips are correct parts (& date codes if specified in design)
TODO: Make a labeled picture with part names
Can you write Python code for it?

test_check["Visual Inspection"]["Chips are correct parts"] = None # TODO: Set to True if all chips are correct parts


I am working on the file "FPV_ANN/notebooks/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
Inference test
prepare frontend for plotting
Can you write Python code for it?

#@title import plotly
import plotly.plotly as py
import numpy as np
from plotly.offline import init_notebook_mode, iplot
# from plotly.graph_objs import Contours, Histogram2dContour, Marker, Scatter
import plotly.graph_objs as go

def configure_plotly_browser_state():
  import IPython
  display(IPython.core.display.HTML('''
        <script src="/static/components/requirejs/require.js"></script>
        <script>
          requirejs.config({
            paths: {
              base: '/static/base',
              plotly: 'https://cdn.plot.ly/plotly-1.5.1.min.js?noext',
            },
          });
        </script>
        '''))


I am working on the file "FPV_ANN/notebooks/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
Training loss plot
Can you write Python code for it?

fig = plt.figure()
plt.semilogy(history.history['loss'])
if vsplit:
    plt.semilogy(history.history['val_loss'])
plt.title(loss_type)
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper right')
plt.show()


I am working on the file "FPV_ANN/notebooks/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
data_reader
The read_h5_data function read the table from the hdf5 file.
In the FGM case we chose not to scale the input features, since they all falls between 0 and 1. There are a great variety in the output features. In the reaction region close to stoichiometry the gradient in the output properties are great. A good example is the source term for progress variable, which rises from 0 to 1e5. So the output features are first transformed to logrithmic scale and then rearranged between  0 and 1. The outputs are normalised by its variance. This way the output value will be large where the gradient is great. So during training more focus would be put. The same 'focus design' has been put on the loss function selection as well. mse is selected over mae for that the squared error put more weights on the data samples that shows great changes.
Can you write Python code for it?

import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler, StandardScaler

class data_scaler(object):
    def __init__(self):
        self.norm = None
        self.norm_1 = None
        self.std = None
        self.case = None
        self.scale = 1
        self.bias = 1e-20
#         self.bias = 1

        self.switcher = {
            'min_std': 'min_std',
            'std2': 'std2',
            'std_min':'std_min',
            'min': 'min',
            'no':'no',
            'log': 'log',
            'log_min':'log_min',
            'log2': 'log2',
            'tan': 'tan'
        }

    def fit_transform(self, input_data, case):
        self.case = case
        if self.switcher.get(self.case) == 'min_std':
            self.norm = MinMaxScaler()
            self.std = StandardScaler()
            out = self.norm.fit_transform(input_data)
            out = self.std.fit_transform(out)

        if self.switcher.get(self.case) == 'std2':
            self.std = StandardScaler()
            out = self.std.fit_transform(input_data)

        if self.switcher.get(self.case) == 'std_min':
            self.norm = MinMaxScaler()
            self.std = StandardScaler()
            out = self.std.fit_transform(input_data)
            out = self.norm.fit_transform(out)

        if self.switcher.get(self.case) == 'min':
            self.norm = MinMaxScaler()
            out = self.norm.fit_transform(input_data)

        if self.switcher.get(self.case) == 'no':
            self.norm = MinMaxScaler()
            self.std = StandardScaler()
            out = input_data

        if self.switcher.get(self.case) == 'log':
            out = - np.log(np.asarray(input_data / self.scale) + self.bias)
            self.std = StandardScaler()
            out = self.std.fit_transform(out)

        if self.switcher.get(self.case) == 'log_min':
            out = - np.log(np.asarray(input_data / self.scale) + self.bias)
            self.norm = MinMaxScaler()
            out = self.norm.fit_transform(out)

   ...


I am working on the file "FPV_ANN/notebooks/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
interactive plot
Can you write Python code for it?

#@title Default title text
# species = np.random.choice(labels)
species = 'T' #@param {type:"string"}
z_level =  0 #@param {type:"integer"}

# configure_plotly_browser_state()
# init_notebook_mode(connected=False)

from sklearn.metrics import r2_score

df_t=df_test[df_test['zeta']==zeta_level[z_level]].sample(frac=1)
# df_p=df_pred.loc[df_pred['zeta']==zeta_level[1]].sample(frac=0.1)
df_p=df_pred.loc[df_t.index]
error=df_p[species]-df_t[species]
r2=round(r2_score(df_p[species],df_t[species]),4)

print(species,'r2:',r2,'max:',df_t[species].max())

fig_db = {
    'data': [
        {'name':'test data from table',
         'x': df_t['f'],
         'y': df_t['pv'],
         'z': df_t[species],
         'type':'scatter3d',
        'mode': 'markers',
          'marker':{
              'size':1
          }
        },
        {'name':'prediction from neural networks',
         'x': df_p['f'],
         'y': df_p['pv'],
         'z': df_p[species],
         'type':'scatter3d',
        'mode': 'markers',
          'marker':{
              'size':1
          },
        },
        {'name':'error in difference',
         'x': df_p['f'],
         'y': df_p['pv'],
         'z': error,
         'type':'scatter3d',
         'mode': 'markers',
          'marker':{
              'size':1
          },
         }
    ],
    'layout': {
        'scene':{
            'xaxis': {'title':'mixture fraction'},
            'yaxis': {'title':'progress variable'},
            'zaxis': {'title': species+'_r2:'+str(r2)}
                 }
    }
}
# iplot(fig_db, filename='multiple-scatter')
iplot(fig_db)

model.save('trained_fgm_nn.h5')

model.save('trained_fgm_nn.h5')
%run -i k2tf.py --input_model='trained_fgm_nn.h5' --output_model='exported/fgm.pb'


I am working on the file "FPV_ANN/notebooks/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
model training
gpu training
Can you write Python code for it?

import keras.backend as K
from keras.callbacks import LearningRateScheduler
import math

def cubic_loss(y_true, y_pred):
    return K.mean(K.square(y_true - y_pred)*K.abs(y_true - y_pred), axis=-1)

def coeff_r2(y_true, y_pred):
    from keras import backend as K
    SS_res =  K.sum(K.square( y_true-y_pred ))
    SS_tot = K.sum(K.square( y_true - K.mean(y_true) ) )
    return ( 1 - SS_res/(SS_tot + K.epsilon()) )

def step_decay(epoch):
   initial_lrate = 0.001
   drop = 0.5
   epochs_drop = 200.0
   lrate = initial_lrate * math.pow(drop,math.floor((1+epoch)/epochs_drop))
   return lrate

lrate = LearningRateScheduler(step_decay)

from keras import optimizers

batch_size = 1024*32
epochs = 60
vsplit = 0.1

loss_type='mse'

adam_op = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999,epsilon=1e-8, decay=0.0, amsgrad=True)

model.compile(loss=loss_type, optimizer=adam_op, metrics=[coeff_r2])
# model.compile(loss=cubic_loss, optimizer=adam_op, metrics=['accuracy'])

# checkpoint (save the best model based validate loss)
!mkdir ./tmp
filepath = "./tmp/weights.best.cntk.hdf5"

checkpoint = ModelCheckpoint(filepath,
                             monitor='val_loss',
                             verbose=1,
                             save_best_only=True,
                             mode='min',
                             period=20)

# callbacks_list = [checkpoint]
callbacks_list = [lrate]

# fit the model
history = model.fit(
    x_train, y_train,
    epochs=epochs,
    batch_size=batch_size,
    validation_split=vsplit,
    verbose=2,
#     callbacks=callbacks_list,
    shuffle=True)

model.save('trained_fgm_nn.h5')


I am working on the file "FPV_ANN/notebooks/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
prepare data for plotting
GPU data prepare
Can you write Python code for it?

from sklearn.metrics import r2_score
# model.load_weights("./tmp/weights.best.cntk.hdf5")

x_test_df = pd.DataFrame(in_scaler.inverse_transform(x_test),columns=input_features)
y_test_df = pd.DataFrame(out_scaler.inverse_transform(y_test),columns=labels)

predict_val = model.predict(x_test,batch_size=1024*8)
predict_df = pd.DataFrame(out_scaler.inverse_transform(predict_val), columns=labels)

test_data=pd.concat([x_test_df,y_test_df],axis=1)
pred_data=pd.concat([x_test_df,predict_df],axis=1)

!rm sim_check.h5
test_data.to_hdf('sim_check.h5',key='test')
pred_data.to_hdf('sim_check.h5',key='pred')

df_test=pd.read_hdf('sim_check.h5',key='test')
df_pred=pd.read_hdf('sim_check.h5',key='pred')

zeta_level=list(set(df_test['zeta']))
zeta_level.sort()

res_sum=pd.DataFrame()
r2s=[]
r2s_i=[]

names=[]
maxs_0=[]
maxs_9=[]

for r2,name in zip(r2_score(df_test,df_pred,multioutput='raw_values'),df_test.columns):
  names.append(name)
  r2s.append(r2)
  maxs_0.append(df_test[df_test['zeta']==zeta_level[0]][name].max())
  maxs_9.append(df_test[df_test['zeta']==zeta_level[8]][name].max())
  for i in zeta_level:
    r2s_i.append(r2_score(df_pred[df_pred['zeta']==i][name],
                          df_test[df_test['zeta']==i][name]))

res_sum['name']=names
# res_sum['max_0']=maxs_0
# res_sum['max_9']=maxs_9
res_sum['z_scale']=[m_9/(m_0+1e-20) for m_9,m_0 in zip(maxs_9,maxs_0)]
# res_sum['r2']=r2s

tmp=np.asarray(r2s_i).reshape(-1,10)
for idx,z in enumerate(zeta_level):
  res_sum['r2s_'+str(z)]=tmp[:,idx]

res_sum[3:]

no_drop=res_sum[3:]
no_drop


I am working on the file "FPV_ANN/notebooks/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
save student network weights
Can you write Python code for it?

import h5py
!rm student_model_weights.h5
student_model.save('student_model_weights.h5')
f = h5py.File('student_model_weights.h5','r')
dset=f['model_weights']
list(dset)

l1_w=dset['dense_5']['dense_5']['kernel:0'][:]
l1_b=dset['dense_5']['dense_5']['bias:0'][:]
l1_c=np.vstack([l1_w,l1_b])
l1_c=pd.Series(list(l1_c)).to_json()

l2_w=dset['dense_6']['dense_6']['kernel:0'][:]
l2_b=dset['dense_6']['dense_6']['bias:0'][:]
l2_c=np.vstack([l2_w,l2_b])
l2_c=pd.Series(list(l2_c)).to_json()

l3_w=dset['output_1']['output_1_2']['kernel:0'][:]
l3_b=dset['output_1']['output_1_2']['bias:0'][:]
l3_c=np.vstack([l3_w,l3_b])
l3_c=pd.Series(list(l3_c)).to_json()

!rm data.json
print("{",file=open('data.json','w'))
print('"l1":',l1_c,file=open('data.json','a'))
print(',"l2":',l2_c,file=open('data.json','a'))
print(',"output":',l3_c,file=open('data.json','a'))
print("}",file=open('data.json','a'))

test_id=888
print(x_test[test_id])
print(student_model.predict(x_test[test_id].reshape(-1,3)))
print(y_test[test_id])

l1_b

np.vstack([l1,l1_b])

student_model.predict(np.asarray([0.5,0.1,0.1]).reshape(-1,3))

student_model.save_weights('student_weights.h5')


I am working on the file "FPV_ANN/notebooks/fgm_nn_inhouse.ipynb".
The first task is:
Stutdent networ
The student network is trained on the synsetic data generated from the full teacher network. It is mean to simplify the final model used in production.
Can you write Python code for it?

from keras.models import Model
from keras.layers import Dense, Input
from keras.callbacks import ModelCheckpoint

n_neuron = 50
# %%
print('set up student network')
# ANN parameters
dim_input = x_train.shape[1]
dim_label = y_train.shape[1]

batch_norm = False

# This returns a tensor
inputs = Input(shape=(dim_input,),name='input_1')

# a layer instance is callable on a tensor, and returns a tensor
x = Dense(n_neuron, activation='relu',name='l1')(inputs)
x = Dense(n_neuron, activation='relu',name='l2')(x)
x = Dropout(0.1)(x)
predictions = Dense(dim_label, activation='linear', name='output_1')(x)

student_model = Model(inputs=inputs, outputs=predictions)
student_model.summary()

batch_size = 1024*32
epochs = 60
vsplit = 0.1

loss_type='mse'

adam_op = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999,epsilon=1e-8, decay=0.0, amsgrad=True)

student_model.compile(loss=loss_type, optimizer=adam_op, metrics=[coeff_r2])
# model.compile(loss=cubic_loss, optimizer=adam_op, metrics=['accuracy'])

# checkpoint (save the best model based validate loss)
!mkdir ./tmp
filepath = "./tmp/student_weights.best.cntk.hdf5"

checkpoint = ModelCheckpoint(filepath,
                             monitor='val_loss',
                             verbose=1,
                             save_best_only=True,
                             mode='min',
                             period=20)

# callbacks_list = [checkpoint]
callbacks_list = [lrate]

x_train_teacher = x_train
y_train_teacher = model.predict(x_train, batch_size=1024*8)
# fit the model
history = student_model.fit(
    x_train_teacher, y_train_teacher,
    epochs=epochs,
    batch_size=batch_size,
    validation_split=vsplit,
    verbose=2,
#     callbacks=callbacks_list,
    shuffle=True)

from sklearn.metrics import r2_score

x_test_df = pd.DataFrame(in_scaler.inverse_transform(x_test),columns=input_features)

predict_val = student_model.predict(x_test,batch_size=1024*8)
predict_df = pd.DataFrame(out_scaler.inverse_transform(predict_val),...


I am working on the file "FPV_ANN/notebooks/fgm_nn_inhouse.ipynb".
The first task is:
data_reader
The read_h5_data function read the table from the hdf5 file.
In the FGM case we chose not to scale the input features, since they all falls between 0 and 1. There are a great variety in the output features. In the reaction region close to stoichiometry the gradient in the output properties are great. A good example is the source term for progress variable, which rises from 0 to 1e5. So the output features are first transformed to logrithmic scale and then rearranged between  0 and 1. The outputs are normalised by its variance. This way the output value will be large where the gradient is great. So during training more focus would be put. The same 'focus design' has been put on the loss function selection as well. mse is selected over mae for that the squared error put more weights on the data samples that shows great changes.
Can you write Python code for it?

import numpy as np
import pandas as pd
from sklearn.preprocessing import MinMaxScaler, StandardScaler

class data_scaler(object):
    def __init__(self):
        self.norm = None
        self.norm_1 = None
        self.std = None
        self.bc = None
        self.case = None
        self.scale = 1
        self.bias = 1e-20
#         self.bias = 1

        self.switcher = {
            'min_std': 'min_std',
            'std2': 'std2',
            'std_min':'std_min',
            'min': 'min',
            'no':'no',
            'log': 'log',
            'log_min':'log_min',
            'log_std':'log_std',
            'log2': 'log2',
            'sqrt_std': 'sqrt_std',
            'cbrt_std': 'cbrt_std',
            'nrt_std':'nrt_std',
            'bc':'bc',
            'tan': 'tan'
        }

    def fit_transform(self, input_data, case):
        self.case = case
        if self.switcher.get(self.case) == 'min_std':
            self.norm = MinMaxScaler()
            self.std = StandardScaler()
            out = self.norm.fit_transform(input_data)
            out = self.std.fit_transform(out)

        if self.switcher.get(self.case) == 'std2':
            self.std = StandardScaler()
            out = self.std.fit_transform(input_data)

        if self.switcher.get(self.case) == 'std_min':
            self.norm = MinMaxScaler()
            self.std = StandardScaler()
            out = self.std.fit_transform(input_data)
            out = self.norm.fit_transform(out)

        if self.switcher.get(self.case) == 'min':
            self.norm = MinMaxScaler()
            out = self.norm.fit_transform(input_data)

        if self.switcher.get(self.case) == 'no':
            self.norm = MinMaxScaler()
            self.std = StandardScaler()
            out = input_data

        if self.switcher.get(self.case) == 'log_min':
            out = - np.log(np.asarray(input_data / self.scale) + self.bias)
            self.norm = MinMaxScaler()
            out = self.norm.fit_transform(out)

        if...


I am working on the file "FPV_ANN/notebooks/fgm_nn_inhouse.ipynb".
The first task is:
prepare data for plotting
GPU data prepare
Can you write Python code for it?

from sklearn.metrics import r2_score
# model.load_weights("./tmp/weights.best.cntk.hdf5")

x_test_df = pd.DataFrame(in_scaler.inverse_transform(x_test),columns=input_features)
y_test_df = pd.DataFrame(out_scaler.inverse_transform(y_test),columns=labels)

predict_val = model.predict(x_test,batch_size=1024*8)
predict_df = pd.DataFrame(out_scaler.inverse_transform(predict_val), columns=labels)

test_data=pd.concat([x_test_df,y_test_df],axis=1)
pred_data=pd.concat([x_test_df,predict_df],axis=1)

!rm sim_check.h5
test_data.to_hdf('sim_check.h5',key='test')
pred_data.to_hdf('sim_check.h5',key='pred')

df_test=pd.read_hdf('sim_check.h5',key='test')
df_pred=pd.read_hdf('sim_check.h5',key='pred')

zeta_level=list(set(df_test['zeta']))
zeta_level.sort()

res_sum=pd.DataFrame()
r2s=[]
r2s_i=[]

names=[]
maxs_0=[]
maxs_9=[]

for r2,name in zip(r2_score(df_test,df_pred,multioutput='raw_values'),df_test.columns):
  names.append(name)
  r2s.append(r2)
  maxs_0.append(df_test[df_test['zeta']==zeta_level[0]][name].max())
  maxs_9.append(df_test[df_test['zeta']==zeta_level[8]][name].max())
  for i in zeta_level:
    r2s_i.append(r2_score(df_pred[df_pred['zeta']==i][name],
                          df_test[df_test['zeta']==i][name]))

res_sum['name']=names
# res_sum['max_0']=maxs_0
# res_sum['max_9']=maxs_9
res_sum['z_scale']=[m_9/(m_0+1e-20) for m_9,m_0 in zip(maxs_9,maxs_0)]
# res_sum['r2']=r2s

tmp=np.asarray(r2s_i).reshape(-1,10)
for idx,z in enumerate(zeta_level):
  res_sum['r2s_'+str(z)]=tmp[:,idx]

res_sum[3:]

scaler_std2=res_sum[3:]
scaler_std2

scaler_cbrt_std=res_sum[3:]
scaler_cbrt_std

scaler_tb=pd.concat([scaler_std2,scaler_cbrt_std],axis=0)
scaler_tb=scaler_tb.drop(['z_scale','name'],axis=1)
scaler_tb

with open('tb.txt','wb') as f:
    scaler_tb.to_latex('tb.txt')


I am working on the file "FPV_ANN_pureResNet/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
Function libaries
ResBlock
res_block is the backbone of the resnet structure. The resblock has multi branch, bottle neck layer and skip connection build in.  This modularized design has made create deep neural network easy.
Can you write Python code for it?

import tensorflow as tf
import keras
from keras.layers import Dense, Activation, Input, BatchNormalization, Dropout, concatenate
from keras import layers

def res_branch(bi, conv_name_base, bn_name_base, scale, input_tensor, n_neuron, stage, block, bn=False):
    x_1 = Dense(scale * n_neuron, name=conv_name_base + '2a_'+str(bi))(input_tensor)
    if bn:
        x_1 = BatchNormalization(axis=-1, name=bn_name_base + '2a_'+str(bi))(x_1)
    x_1 = Activation('relu')(x_1)
#     x_1 = Dropout(0.)(x_1)
    return x_1

def res_block(input_tensor,scale, n_neuron, stage, block, bn=False,branches=0):
    conv_name_base = 'res' + str(stage) + block + '_branch'
    bn_name_base = 'bn' + str(stage) + block + '_branch'

#     scale = 2
    x = Dense(scale * n_neuron, name=conv_name_base + '2a')(input_tensor)
    if bn:
        x = BatchNormalization(axis=-1, name=bn_name_base + '2a')(x)
    x = Activation('relu')(x)
    dp1=0.0
    if dp1 >0:
        x = Droout(0.)(x)

    branch_list=[x]
    for i in range(branches-1):
        branch_list.append(res_branch(i,conv_name_base, bn_name_base, scale,input_tensor,n_neuron,stage,block,bn))
    if branches-1 > 0:
        x = Dense(n_neuron, name=conv_name_base + '2b')(concatenate(branch_list,axis=-1))
#         x = Dense(n_neuron, name=conv_name_base + '2b')(layers.add(branch_list))
    else:
        x = Dense(n_neuron, name=conv_name_base + '2b')(x)

    if bn:
        x = BatchNormalization(axis=-1, name=bn_name_base + '2b')(x)
    x = layers.add([x, input_tensor])
    x = Activation('relu')(x)
    if dp1 >0:
        x = Droout(0.)(x)

    return x


I am working on the file "FPV_ANN_pureResNet/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
GPU data prepare
Can you write Python code for it?

model.load_weights("./tmp/weights.best.cntk.hdf5")

x_test_df = pd.DataFrame(in_scaler.inverse_transform(x_test),columns=input_features)
y_test_df = pd.DataFrame(out_scaler.inverse_transform(y_test),columns=labels)

predict_val = model.predict(x_test,batch_size=1024*8)
predict_df = pd.DataFrame(out_scaler.inverse_transform(predict_val), columns=labels)

test_data=pd.concat([x_test_df,y_test_df],axis=1)
pred_data=pd.concat([x_test_df,predict_df],axis=1)

!rm sim_check.h5
test_data.to_hdf('sim_check.h5',key='test')
pred_data.to_hdf('sim_check.h5',key='pred')

df_test=pd.read_hdf('sim_check.h5',key='test')
df_pred=pd.read_hdf('sim_check.h5',key='pred')

zeta_level=list(set(df_test['zeta']))
zeta_level.sort()


I am working on the file "FPV_ANN_pureResNet/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
Inference test
prepare frontend for plotting
Can you write Python code for it?


I am working on the file "FPV_ANN_pureResNet/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
model
load data
Can you write Python code for it?

%matplotlib inline
import os
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# define the labels
col_labels=['C2H3', 'C2H6', 'CH2', 'H2CN', 'C2H4', 'H2O2', 'C2H', 'CN',
       'heatRelease', 'NCO', 'NNH', 'N2', 'AR', 'psi', 'CO', 'CH4', 'HNCO',
       'CH2OH', 'HCCO', 'CH2CO', 'CH', 'mu', 'C2H2', 'C2H5', 'H2', 'T', 'PVs',
       'O', 'O2', 'N2O', 'C', 'C3H7', 'CH2(S)', 'NH3', 'HO2', 'NO', 'HCO',
       'NO2', 'OH', 'HCNO', 'CH3CHO', 'CH3', 'NH', 'alpha', 'CH3O', 'CO2',
       'CH3OH', 'CH2CHO', 'CH2O', 'C3H8', 'HNO', 'NH2', 'HCN', 'H', 'N', 'H2O',
       'HCCOH', 'HCNN']

# labels = ['T','PVs']
labels = ['T','CH4','O2','CO2','CO','H2O','H2','OH','psi']
# labels = ['CH2OH','HNCO','CH3OH', 'CH2CHO', 'CH2O', 'C3H8', 'HNO', 'NH2', 'HCN']

# labels = np.random.choice(col_labels,9,replace=False).tolist()
labels.append('PVs')

# labels = col_labels

print(labels)

input_features=['f','pv','zeta']

# read in the data
x_input, y_label, df, in_scaler, out_scaler = read_h5_data('./data/tables_of_fgm.h5',input_features=input_features, labels = labels)


I am working on the file "FPV_ANN_pureResNet/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
model training
gpu training
Can you write Python code for it?

import keras.backend as K

def cubic_loss(y_true, y_pred):
    return K.mean(K.square(y_true - y_pred)*K.abs(y_true - y_pred), axis=-1)

def coeff_r2(y_true, y_pred):
    from keras import backend as K
    SS_res =  K.sum(K.square( y_true-y_pred ))
    SS_tot = K.sum(K.square( y_true - K.mean(y_true) ) )
    return ( 1 - SS_res/(SS_tot + K.epsilon()) )

from keras import optimizers

batch_size = 1024*32
epochs = 200
vsplit = 0.1

loss_type='mse'

adam_op = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999,epsilon=1e-8, decay=0.0, amsgrad=True)

model.compile(loss=loss_type, optimizer=adam_op, metrics=[coeff_r2])
# model.compile(loss=cubic_loss, optimizer=adam_op, metrics=['accuracy'])

# checkpoint (save the best model based validate loss)
!mkdir ./tmp
filepath = "./tmp/weights.best.cntk.hdf5"

checkpoint = ModelCheckpoint(filepath,
                             monitor='val_loss',
                             verbose=1,
                             save_best_only=True,
                             mode='min',
                             period=20)

callbacks_list = [checkpoint]

# fit the model
history = model.fit(
    x_train, y_train,
    epochs=epochs,
    batch_size=batch_size,
    validation_split=vsplit,
    verbose=2,
    callbacks=callbacks_list,
    shuffle=True)

model.save('trained_fgm_nn.h5')


I am working on the file "FPV_ANN_pureResNet/.ipynb_checkpoints/fgm_nn_inhouse-checkpoint.ipynb".
The first task is:
prepare data for plotting
TPU data prepare
Can you write Python code for it?

cpu_model = tpu_model.sync_to_cpu()
predict_val=cpu_model.predict(X_test)

X_test_df = pd.DataFrame(in_scaler.inverse_transform(X_test),columns=input_features)
y_test_df = pd.DataFrame(out_scaler.inverse_transform(y_test),columns=labels)

predict_val = model.predict(X_test)
predict_df = pd.DataFrame(out_scaler.inverse_transform(predict_val), columns=labels)

test_data=pd.concat([X_test_df,y_test_df],axis=1)
pred_data=pd.concat([X_test_df,predict_df],axis=1)

test_data.to_hdf('sim_check.H5',key='test')
pred_data.to_hdf('sim_check.H5',key='pred')

df_test=pd.read_hdf('sim_check.H5',key='test')
df_pred=pd.read_hdf('sim_check.H5',key='pred')
zeta_level=list(set(df_test['zeta']))
zeta_level.sort()


I am working on the file "FPV_ANN_pureResNet/fgm_nn_inhouse.ipynb".
The first task is:
Function libaries
ResBlock
res_block is the backbone of the resnet structure. The resblock has multi branch, bottle neck layer and skip connection build in.  This modularized design has made create deep neural network easy.
Can you write Python code for it?


I am working on the file "FPV_ANN_pureResNet/fgm_nn_inhouse.ipynb".
The first task is:
GPU data prepare
Can you write Python code for it?


I am working on the file "FPV_ANN_pureResNet/fgm_nn_inhouse.ipynb".
The first task is:
TPU training
Can you write Python code for it?

import os

batch_size = 1024*128
epochs = 100
vsplit = 0.2

tpu_model = tf.contrib.tpu.keras_to_tpu_model(
    model,
    strategy=tf.contrib.tpu.TPUDistributionStrategy(
        tf.contrib.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR'])
    )
)

tpu_model.compile(
    optimizer=tf.train.AdamOptimizer(learning_rate=1e-4),
    loss=tf.keras.losses.mae,
    metrics=['accuracy']
)

tpu_model.fit(
    X_train,y_train,
    epochs=epochs,
    batch_size=batch_size,
    validation_split=vsplit
)


I am working on the file "FPV_ANN_pureResNet/fgm_nn_inhouse.ipynb".
The first task is:
build neural network model
Can you write Python code for it?

from sklearn.model_selection import train_test_split

import tensorflow as tf
from keras.models import Model
from keras.layers import Dense, Input
from keras.callbacks import ModelCheckpoint

# split into train and test data
x_train, x_test, y_train, y_test = train_test_split(x_input,y_label, test_size=0.01)

n_neuron = 10
scale=3
branches=3
# %%
print('set up ANN')
# ANN parameters
dim_input = x_train.shape[1]
dim_label = y_train.shape[1]

batch_norm = False

# This returns a tensor
inputs = Input(shape=(dim_input,),name='input_1')

# a layer instance is callable on a tensor, and returns a tensor
x = Dense(n_neuron, activation='relu')(inputs)

# less then 2 res_block, there will be variance
x = res_block(x, scale, n_neuron, stage=1, block='a', bn=batch_norm,branches=branches)
x = res_block(x, scale, n_neuron, stage=1, block='b', bn=batch_norm,branches=branches)
# x = res_block(x, scale, n_neuron, stage=1, block='c', bn=batch_norm,branches=branches)

x_1 = Dense(100, activation='relu')(x)
x_1 = Dense(100, activation='relu')(x)
predictions = Dense(dim_label, activation='linear', name='output_1')(x)

model = Model(inputs=inputs, outputs=predictions)
model.summary()
x_1 = Dense(100, activation='relu')(x)


I am working on the file "FPV_ANN_pureResNet/fgm_nn_inhouse.ipynb".
The first task is:
interactive plot
Can you write Python code for it?

#@title Default title text
# species = 'PVs' #@param {type:"string"}
species = np.random.choice(labels)
# configure_plotly_browser_state()
# init_notebook_mode(connected=False)

from sklearn.metrics import r2_score

df_t=df_test.loc[df_test['zeta']==zeta_level[0]].sample(frac=0.5)
# df_p=df_pred.loc[df_pred['zeta']==zeta_level[1]].sample(frac=0.1)
df_p=df_pred.loc[df_t.index]
error=df_p[species]-df_t[species]
r2=round(r2_score(df_p[species],df_t[species]),4)
fig_db = {
    'data': [
        {'name':'test data from table',
         'x': df_t['f'],
         'y': df_t['pv'],
         'z': df_t[species],
         'type':'scatter3d',
        'mode': 'markers',
          'marker':{
              'size':1
          }
        },
        {'name':'prediction from neural networks',
         'x': df_p['f'],
         'y': df_p['pv'],
         'z': df_p[species],
         'type':'scatter3d',
        'mode': 'markers',
          'marker':{
              'size':1
          },
        },
        {'name':'error in difference',
         'x': df_p['f'],
         'y': df_p['pv'],
         'z': error,
         'type':'scatter3d',
         'mode': 'markers',
          'marker':{
              'size':1
          },
         }
    ],
    'layout': {
        'scene':{
            'xaxis': {'title':'mixture fraction'},
            'yaxis': {'title':'progress variable'},
            'zaxis': {'title': species+'_r2:'+str(r2)}
                 }
    }
}
# iplot(fig_db, filename='multiple-scatter')
iplot(fig_db)
print(species,r2)

df_p['HNO']

%run -i k2tf.py --input_model='trained_fgm_nn.h5' --output_model='exported/fgm.pb'


I am working on the file "FPV_ANN_pureResNet/fgm_nn_inhouse.ipynb".
The first task is:
model training
gpu training
Can you write Python code for it?

import keras.backend as K

def cubic_loss(y_true, y_pred):
    return K.mean(K.square(y_true - y_pred)*K.abs(y_true - y_pred), axis=-1)

def coeff_r2(y_true, y_pred):
    from keras import backend as K
    SS_res =  K.sum(K.square( y_true-y_pred ))
    SS_tot = K.sum(K.square( y_true - K.mean(y_true) ) )
    return ( 1 - SS_res/(SS_tot + K.epsilon()) )

from keras import optimizers

batch_size = 1024*32
epochs = 5000
vsplit = 0.1

loss_type='mse'

adam_op = optimizers.Adam(lr=0.001, beta_1=0.9, beta_2=0.999,epsilon=1e-8, decay=0.0, amsgrad=True)

model.compile(loss=loss_type, optimizer=adam_op, metrics=[coeff_r2])
# model.compile(loss=cubic_loss, optimizer=adam_op, metrics=['accuracy'])

# checkpoint (save the best model based validate loss)
!mkdir ./tmp
filepath = "./tmp/weights.best.cntk.hdf5"

checkpoint = ModelCheckpoint(filepath,
                             monitor='val_loss',
                             verbose=1,
                             save_best_only=True,
                             mode='min',
                             period=20)

callbacks_list = [checkpoint]

# fit the model
history = model.fit(
    x_train, y_train,
    epochs=epochs,
    batch_size=batch_size,
    validation_split=vsplit,
    verbose=2,
#     callbacks=callbacks_list,
    shuffle=True)

model.save('trained_fgm_nn.h5')


I am working on the file "FPV_ANN_pureResNet/fgm_nn_inhouse.ipynb".
The first task is:
prepare data for plotting
TPU data prepare
Can you write Python code for it?


I am working on the file "FRF_plots.ipynb".
The first task is:
Back to top
Frequency response function
A frequency response function is a complex valued function of frequency. Let us see how it looks when we plot the real and imaginary parts in separate:
Can you write Python code for it?

w, H = system.freqresp()
fig, ax = plt.subplots(2, 1)
fig.suptitle('Real and imaginary plots')
# Real part plot
ax[0].plot(w, H.real, label='FRF')
ax[0].axvline(wn[0], color='k', label='First mode', linestyle='--')
ax[0].axvline(wn[2], color='k', label='Second mode', linestyle='--')
ax[0].set_ylabel('Real [-]')
ax[0].grid(True)
ax[0].legend()
# Imaginary part plot
ax[1].plot(w, H.imag, label='FRF')
ax[1].axvline(wn[0], color='k', label='First mode', linestyle='--')
ax[1].axvline(wn[2], color='k', label='Second mode', linestyle='--')
ax[1].set_ylabel('Imaginary [-]')
ax[1].set_xlabel('Frequency [rad/s]')
ax[1].grid(True)
ax[1].legend()
plt.show()


I am working on the file "FRF_plots.ipynb".
The first task is:
Back to top
Nichols plot
A Nichols plot combines the Bode plot in a single plot of magnitude versus phase:
Can you write Python code for it?

plt.figure()
plt.title('Nichols plot')
plt.plot(phase*np.pi/180., mag)
plt.xlabel('Phase [rad/s]')
plt.ylabel('Magnitude [dB]')
plt.grid(True)
plt.show()


I am working on the file "FRF_plots.ipynb".
The first task is:
Back to top
Nyquist plot
A Nyquist plot represents the real and imaginary parts of the complex FRF in a single plot:
Can you write Python code for it?

plt.figure()
plt.title('Nyquist plot')
plt.plot(H.real, H.imag, 'b')
plt.plot(H.real, -H.imag, 'r')
plt.xlabel('Real [-]')
plt.ylabel('Imaginary[-]')
plt.grid(True)
plt.axis('equal')
plt.show()


I am working on the file "FRF_plots.ipynb".
The first task is:
To check the results presented ahead we will need the angular frequencies and damping coefficients of this system. The eigenanalysis of the system matrix yields them after some computations:
Can you write Python code for it?

w1, v1 = LA.eig(A)
ix = np.argsort(np.absolute(w1)) # order of ascending eigenvalues
w1 = w1[ix] # sorted eigenvalues
v1 = v1[:,ix] # sorted eigenvectors
zw = -w1.real # damping coefficient time angular frequency
wD = w1.imag # damped angular frequency
zn = 1./np.sqrt(1.+(wD/-zw)**2) # the minus sign is formally correct!
wn = zw/zn # undamped angular frequency
print('Angular frequency: {}'.format(wn[[0,2]]))
print('Damping coefficient: {}'.format(zn[[0,2]]))


I am working on the file "FRF_plots.ipynb".
The first task is:
We will also need some specific modules and a litle "IPython magic" to show the plots:
Can you write Python code for it?

from numpy import linalg as LA
from scipy import signal
import matplotlib.pyplot as plt

%matplotlib inline


I am working on the file "FSC.ipynb".
The first task is:

ASA ↑ AHT ↑

As the amount of time it takes to answer a typical call once it has been routed to the FSC increases, the amount of time it takes an Agent to deal with all aspects of a call including talk time plus ACW increases.
A bivariate distribution of ASA/AHT variables along with the univariate (or marginal) distribution of each on separate axes shows a Pearson correlation coefficient of 0.77 and a p-value of 1.8e-09. The p-value roughly indicates the probability of an uncorrelated system producing datasets that have a Pearson correlation at least as extreme as the one computed from these datasets. The p-values are not entirely reliable but are probably reasonable for datasets larger than 500 or so. Positive correlations imply that as AHT increases, so does ASA.
An altogether different approach is to fit a nonparametric regression using a lowess smoother. This approach has the fewest assumptions, although it is computationally intensive and so currently confidence intervals are not computed at all.
Can you write Python code for it?

sns.lmplot(x='AHT', y='ASA', data=data, robust=True)

sns.lmplot(x='AHT', y='ASA', data=data, lowess=True)

sns.jointplot(x='AHT', y='ASA', data=data, kind="reg", robust=True);


I am working on the file "FTAF/example/03-FTAF-Octane.ipynb".
The first task is:
$P-\alpha$:
Can you write Python code for it?

plt.plot(SOL.α, SOL.P)
plt.plot(SOL.α, SOL.P, 'b.')
plt.grid(dashes = (5, 3))


I am working on the file "FTAF/example/03-FTAF-Octane.ipynb".
The first task is:
$P-v$, linear and loglog:
Can you write Python code for it?

plt.plot(SOL.v, SOL.P)
plt.plot(SOL.v, SOL.P, 'b.')
plt.grid(dashes = (5, 3))

plt.loglog(SOL.v, SOL.P)
plt.loglog(SOL.v, SOL.P, 'b.')
plt.grid(dashes = (5, 3))


I am working on the file "FTAF/example/03-FTAF-Octane.ipynb".
The first task is:
$Q_r-\alpha$: heat release due to reaction
Can you write Python code for it?

plt.plot(SOL.α[:-1], SOL.Qr)
plt.plot(SOL.α[:-1], SOL.Qr, 'b.')
plt.grid(dashes = (5, 3))


I am working on the file "FTAF/example/03-FTAF-Octane.ipynb".
The first task is:
$n-\alpha$: polytropic exponent
Can you write Python code for it?

plt.plot(SOL.α[:-1], SOL.n)
plt.plot(SOL.α[:-1], SOL.n, 'b.')
plt.grid(dashes = (5, 3))


I am working on the file "FTAF/example/03-FTAF-Octane.ipynb".
The first task is:
2. Reactive Mixture
Next, the reactive mixture is setup as follows:
Can you write Python code for it?

# Initialize admission conditions, as well as operating and mixture parameters:
P0, T0, V0, φ, ψ, ζ = 100.00, 300, CR.V0(), 0.5, 3.76, 0

# Initialize FUEL proportions with a thermo-chemical mixture (100% octane)
# FUEL proportions dont't need to add to 1 kmol. Any non-zero amount is OK.
fuelProp = FTAF.tc.tcTypes.tcMixture({
    'H2'   : 0.00,  #   0% Hydrogen
    'C8H18': 5.00,  # 100% Octane - nevermind the 5 kmol amount
})

# Create the reactive mixture with FTAF.tc.tcAF.freshAFMix:
RM = FTAF.tc.tcAF.freshAFMix(P0, T0, V0, φ, ψ, fuelProp, ζ)

# The component amounts are automatically adjusted.
# Even the combustion products appear (but count as zero kmol each)
print(RM)

# The combustion reaction is also setup:
print(RM.asReaction())

# OPTIONALLY, the combustion time can be set:
# If this is set, the solver can calculate the δ parameter.
# If a value of δ is passed to the solver, it will override this.
RM.setReactionTime(2200e-6)  # In seconds
print(RM.Δt)


I am working on the file "FTAF/example/03-FTAF-Octane.ipynb".
The first task is:
3. Cumulative Reaction Function
The cumulative reaction function (of cranckshaft angle) is defined as a function as below:
Can you write Python code for it?

# This function has to have the three parameters in this order: α, θ, and δ.
def ga(α, θ, δ):
    return (1 - numpy.cos((α - θ) * numpy.pi / δ)) / 2


I am working on the file "FTAF/example/03-FTAF-Octane.ipynb".
The first task is:
5. Solver solution
Once everything is setup, iteratively solving is simple:
Can you write Python code for it?

# Solve iteratively
for i in range(len(SOL.α)-1):
    # Prints the iteration number as the solution proceeds
    print('{:3d}, '.format(i), end = '', flush = True)
    if i % 10 == 9:
        print('')
    # SOL.iterate() makes one iteration
    SOL.iterate()

# Closes the solution --- isochoric heat release
# ... and calculates the thermal efficiency
SOL.close()


I am working on the file "FTAF/example/03-FTAF-Octane.ipynb".
The first task is:
Case setup and solution
1. Engine
To setup a case, first, instantiate the engine and operate as follows:
Can you write Python code for it?

# Reciprocating engine instantiation
ENG = FTAF.eng.recipr.engine({
    'Vd' : 2000e-6,             # 2000 cc
    'z'  : 4,                   # 4 cyl.
    'rDS': 1,                   # D/S = 1 (square)
    'rLR': 3.6,                 # L/R
    'rV' : 8,                   # 8:1 compression
    'th' : -30,                 # ignition angle, deg
    'N'  : 2400,                # rotation, RPM
})

# Engine calculation and OPTIONAL output
ENG.calc()
print(ENG.Ɛ)


I am working on the file "FTAF/example/03-FTAF-Octane.ipynb".
The first task is:
From the engine, obtain a crankRod object as follows:
Can you write Python code for it?

# Get the crankRod object from the solved engine
CR = ENG.CR()

# At this point, mechanism data can be OPTIONALLY plotted:
α = numpy.linspace(-numpy.pi, numpy.pi, 180)
V = CR.V(α)
# System volume (m³) as function of crankshaft angle (rad):
plt.plot(α, V, 'b-')


I am working on the file "FTAF/example/03-FTAF-Octane.ipynb".
The first task is:
Solution Plots
Once a case is solved, plots can be easily generated:
Can you write Python code for it?

matplotlib.rcParams['figure.figsize'] = (12, 4)


I am working on the file "FTE_Narrative.ipynb".
The first task is:
A Computational Narrative about FTE and Graduates
This computational document assumes the basic operations on individual tables
and illustrates the process of refining raw data into a form that can be worked
with more sensibly and combines data from multiple tables.
Can you write Python code for it?

#raw_fte = Table.read_table("./data/deptFTE.csv")
raw_fte = Table.read_table("https://deculler.github.io/DataScienceTableDemos/data/deptFTE.csv")
raw_fte

fte = raw_fte.where(raw_fte['FTE'] > 0)
fte

raw_degrees = Table().read_table("./data/DegreeCounts-1983-2016.csv")  # Raw degree data from CalAnswers
raw_degrees


I am working on the file "FTE_Narrative.ipynb".
The first task is:
Six thousand rows!  Now that's hard to deal with mousing around in a spreadsheet.  But it is also hard to deal with the categories.  So let's clean it up.
Can you write Python code for it?

degree_rows = raw_degrees.where(raw_degrees['Academic Yr'] != '2015-16')
degrees = degree_rows.drop('All Major Acad Hier Level Nm')
degrees.relabel('Academic Yr', 'Year')
degrees.relabel('Academic Department Short Nm', 'Dept')
degrees.relabel('Reporting Clg School Short Nm', 'College')
degrees.relabel('Academic Division Short Nm','Div')
degrees.relabel('Student Headcount', 'Grads')
degrees.relabel('Prorated Student Major Cnt Sum', 'Prorated')
degrees['Year'] = degrees.apply(lambda x:int(x[0:4])+1, 'Year')
degrees.show()

degrees.num_rows

Totals = degrees.select(['Year', 'DegLvlDesc', 'Grads']).groups(['Year', 'DegLvlDesc'], collect=sum)
Totals

# Transpose this data into columns of years, per degree type, summing the number of grads.
Grads_by_year = Totals.pivot('DegLvlDesc', 'Year', 'Grads sum', collect=sum)
Grads_by_year

# What does this look like?
Grads_by_year.plot('Year')

# Let's look at the last year of data
degrees.where('Year', 2015)

graduates = degrees.where('Year', 2015).select(['DegLvlDesc', 'College', 'Dept', 'Grads']).pivot('DegLvlDesc', ['College', 'Dept'], 'Grads', collect=sum)
graduates


I am working on the file "FTICR_1.ipynb".
The first task is:
3 remarks
 - imaginary discarded (eventually silently)
 - phase
 - symetry
1 syntax remark
Can you write Python code for it?

from numpy.fft import fft, rfft
sp = rfft(fid)
print sp[0]
len(sp)

plt.plot(abs(sp))     # absolute value is modulus

# let's zoom in
plt.plot(abs(sp[715000:725000]))

# without the abs() we see the phase rotating
plt.plot(sp[720000:722000])
plt.figure()
# we rotate by 90° the phase
plt.plot(1j*sp[720000:722000])


I am working on the file "FTICR_1.ipynb".
The first task is:
Doing the same thing with SPIKE
SPIKE is a complete software suite we're building to do this, and much more
It is distributed here : https://bitbucket.org/delsuc/spike
However, the program is not fully operational yet, stay tuned !
Can you write Python code for it?

import sys
sys.path.append('/Users/mad/NPKV2/')  # adapt this to your own set-up
import spike
from spike.File import Solarix
%matplotlib inline

File = 'FTICR-Files/ESI_pos_Ubiquitin_000006.d'
dd = Solarix.Import_1D(File)

dd.unit = "sec"
dd.display()

dd.hamming().zf(2).rfft().modulus()
dd.unit = "m/z"
dd.display(zoom=(500,2000))

# wrong calibration !
dd.axis1.calibA = float(dd.params["ML1"])
dd.axis1.calibB = float(dd.params["ML2"])
dd.display(zoom=(500,2000))

dd.pp(threshold=1E7)
print ("detected %d peaks"%len(dd.peaks))

dd.display(zoom=(856.5, 858.5))
dd.display_peaks(zoom=(856.5, 858.5))

p = dd.peaks[603]

p.report()

p.label = "Peak 103"
p.report()  # in index
p.report(f=dd.axis1.itomz)  # provides in m/z
# string is :
# id, label, m/z, intensity, width (0 if not determined)

dd.centroid()
for p in dd.peaks:
    p.label = "%.5f"%(dd.axis1.itomz(p.pos))

# this switches to a different display mode
dd.display(zoom=(856.5, 858.5))
dd.display_peaks(zoom=(856.5, 858.5), peak_label=True)


I am working on the file "FTICR_1.ipynb".
The first task is:
Fourier processing
small complication
There are 2 fft packages available
 - numpy.fft
   - simple, basic, efficient
 - scipy.fftpack
   - slightly faster, more complex
we are going to use numpy.fft
Can you write Python code for it?

sp = np.fft.fft(fid)

print sp[0]
len(sp)

# default size of the figures
mpl.rcParams['figure.figsize'] = [10.0,8.0]
# this helps matplotlib to draw the huge vectors we're using (4-8Mpoints)
mpl.rcParams['agg.path.chunksize'] = 100000

plt.plot(sp)


I am working on the file "FTICR_1.ipynb".
The first task is:
Notice how lineshape is improved, and the isotopic pattern intensities restored
We can superimpose both processing, zooming on a line you can see that FWHM is sligthly worse when apodized.
Theory is 1.5 for hamming window, but does not really show here.
Can you write Python code for it?


I am working on the file "FTICR_1.ipynb".
The first task is:
locate the data-set which should be in the repository
Can you write Python code for it?


I am working on the file "FTICR_1.ipynb".
The first task is:
m/z Calibration
In FT-ICR, the frequency $f$ depends on the magnetic field $B_o$ the charge $z$ and the mass $m$ :
$$ f = \frac{B_o z}{m} \quad \Rightarrow \quad  m/z = \frac{B_o}{f}$$
We are going to use a sligthly extended equation which takes cares of experimental imperfections:
$$ m/z = \frac{M_1}{M_2+f}$$
where $M_1$ and $M_2$ are constants determined precisely by a calibration procedure
parameter files
FT-ICR parameters are stored in the .method files
Can you write Python code for it?


I am working on the file "FTICR_1.ipynb".
The first task is:
packing it up
Now we can simply build a function doing all this at once :
Can you write Python code for it?

import glob
def read_fticr(folder):
    """
    load and process the data Solarix Apex FTICR file found in folder
    uses the calibration from the parameter file

    eg:
    spectrum,axis = read_fticr('FTICR/Files/bruker ubiquitin file/ESI_pos_Ubiquitin_000006.d')
    """
    # find and load parameters
    L = glob.glob(op.join(folder,"*","apexAcquisition.method"))
    if len(L)>1:
        raise Exception( "You have more than 1 apexAcquisition.method file in the %s folder, using the first one"%folder )
    elif len(L) == 0:
        raise Exception( "You don't have any apexAcquisition.method file in the  %s folder, please double check the path"%folder )
    param = read_param(L[0])

    # load data
    n = int(param['TD'])
    fidzf = np.zeros(2*n)
    with  open(  op.join(BASE, 'fid'), 'r') as F:  # 'with' a better way of reading a file
        tbuf = F.read(4*int(param['TD']))
        fidzf[:n] = np.array(array.array('i',tbuf)) # [:] is less memory intensive

    # process
    fidzf[:n] *= np.hamming(n)
    spectrum = abs( rfft( fidzf ) )

    # calibrate
    sw = float(param['SW_h_Broadband'])
    ml1 = float(param['ML1'])
    ml2 = float(param['ML2'])
    faxis = np.linspace(0, sw, len(spectrum))    # the freq axis from 0 to sw
    mzaxis = ml1/(ml2+faxis)    # and the mzaxis
    return spectrum, mzaxis

spectrum,axis = read_fticr('FTICR-Files/ESI_pos_Ubiquitin_000006.d')

# %matplotlib
plt.plot(axis, spectrum)
plt.xlabel("$m/z$")
plt.xlim(200,2500)

plt.plot(axis, spectrum)
plt.xlabel("$m/z$")
plt.xlim(650,1300)
plt.title("Zooming in")

plt.plot(axis, spectrum)
plt.xlabel("$m/z$")
plt.xlim(855,865)
plt.title("Zooming further")


I am working on the file "FTICR_1.ipynb".
The first task is:
so now we can read it
Can you write Python code for it?

BASE = 'FTICR-Files/ESI_pos_Ubiquitin_000006.d'

F = open(  op.join(BASE, 'fid'), 'rb')
tbuf = F.read()
fid = np.array(array.array('i',tbuf))
F.close()

len(fid)

fid[1000:1010]

plt.plot(fid)


I am working on the file "FTICR_1.ipynb".
The first task is:
there is also blackman hanning  bartlett...
Can you write Python code for it?

np.blackman.func_name

for apod in (np.blackman, np.hamming, np.hanning, np.bartlett):
    plt.plot(apod(100),label=apod.func_name)
plt.legend()

sp2 = rfft( np.hamming(len(fid))*fid )

plt.plot(abs(sp2))
plt.figure()
plt.plot(abs(sp2[720000:722000]))


I am working on the file "FWD_17_intro_to_pandas.ipynb".
The first task is:
Hmm.. DFs look similar to SQL Tables, don't they?

Similarity: The arrangement of data to in tabular format
Similarity: Ability to perform JOIN operations
Disimilarity: Pandas is not a language
Disimilarity: You don't use Pandas as a datastore/backend
Disimilarity: No need define schema in Pandas

Advantage of Pandas over SQL for data analysis

Performs automatic data alignment.
Performs faster subsetting.
Real Power Load data from various backend sources, making backend aganostic analysis.
Real Power Store the result in any and as many datasources as needed.

Basic Stats on Loaded Data
Getting to know how the data looks
Can you write Python code for it?

df.head()

df.tail()

df['deaths'].count()

df['deaths'].min()

df['deaths'].max()

df['deaths'].mean()

df['deaths'].describe()

df['deaths'].plot(kind='box')


I am working on the file "FWD_17_intro_to_pandas.ipynb".
The first task is:
Merging Data Frames
How to merge?

concat()
merge()

append()

Vertically merge

Horizontal merge
Inner JOIN merge
Outer JOIN merge
Add new Keys while merging

Real Power: Merge from different datasources!
Inline Visualization
Visulaize as you analyse.
Can you write Python code for it?


I am working on the file "FWD_17_intro_to_pandas.ipynb".
The first task is:
Subsetting DataSets

Why do we need subsets?

We want to screen out anomalous, partial datapoints.
We want to divide our huge data into small chunks and apply different analysis on each set.
Divide data into train set and test set.

Ways of subsetting in Pandas:

By Column names
By row labels
By row-column index
Combination of both labels and columns
Can you write Python code for it?

num = range(1,6)
mul2 = [x*2 for x in num]
mul3 = [x*3 for x in num]
mul4 = [x*4 for x in num]
mul5 = [x*35 for x in num]
data = [num, mul2, mul3, mul4, mul5]
df1 = pd.DataFrame(data, index=['v', 'w', 'x', 'y', 'z'], columns=['A', 'B','C','D', 'E'])

df1

#### Only Column
df1[['A']]

#### Only Row
df1.loc[['v']]

df1.loc[['v','w'],['A','B']] # rows and columns

df1.iloc[0:2, 0:2] #Using default index numbers


I am working on the file "FWHR_calculator.ipynb".
The first task is:
Difference eyelid vs. eyebrow
Can you write Python code for it?

obama_url = 'http://mienshiang.com/wp-content/uploads/images-29.jpg'
get_fwhr(obama_url, url=True, top = 'eyelid')

get_fwhr(obama_url, url=True, top = 'eyebrow')


I am working on the file "FWHR_calculator.ipynb".
The first task is:
Examples

Basic example
Can you write Python code for it?


I am working on the file "FWHR_calculator.ipynb".
The first task is:
FWHR only
Can you write Python code for it?

get_fwhr(obama_url, url=True, top = 'eyebrow', show=False)


I am working on the file "FWHR_calculator.ipynb".
The first task is:
Final function that combines the previous functions
This function combines all the previous logic into one function.
Arguments:
image_path $\rightarrow$ path or URL to image
url $\rightarrow$ set to True if image_path is a url
show $\rightarrow$ set to False if you only want it to return the FWHR
method $\rightarrow$ determines which eye to use for the top point: left, right, or average
top $\rightarrow$ determines whether to use the eyebrow as top point or eyelid as top point
Can you write Python code for it?

def get_fwhr(image_path, url=False, show=True, method='average', top='eyelid'):
    image = load_image(image_path, url)
    landmarks = face_recognition.api._raw_face_landmarks(image)
    landmarks_as_tuples = [(p.x, p.y) for p in landmarks[0].parts()]

    if good_picture_check(landmarks_as_tuples):
        corners = get_face_points(landmarks_as_tuples, method=method, top = top)
        fwh_ratio = FWHR_calc(corners)

        if show:
            print('The Facial-Width-Height ratio is: {}'.format(fwh_ratio))
            show_box(image, corners)
        else:
            return fwh_ratio
    else:
        if show:
            print("Picture is not suitable to calculate fwhr.")
            imshow(image)
        else:
            return None


I am working on the file "FWHR_calculator.ipynb".
The first task is:
Function to calculate the FWHR
This function calculates the FWHR based on the corners: Width / Height
Can you write Python code for it?

def FWHR_calc(corners):
    width = corners['top_right'][0] - corners['top_left'][0]
    height = corners['bottom_left'][1] - corners['top_left'][1]
    return float(width) / float(height)


I am working on the file "FWHR_calculator.ipynb".
The first task is:
Function to check suitability of picture
This function checks whether a picture contains a person that is looking straight at the camera.
Can you write Python code for it?

def good_picture_check(p, debug=False):
    ## To scale for picture size
    width_im = (p[16][0] - p[0][0]) / 100

    ## Difference in height between eyes
    eye_y_l = (p[37][1] + p[41][1]) / 2.0
    eye_y_r = (p[44][1] + p[46][1]) / 2.0
    eye_dif = (eye_y_r - eye_y_l) / width_im

    ## Difference top / bottom point nose
    nose_dif = (p[30][0] - p[27][0]) / width_im

    ## Space between face-edge to eye, left vs. right
    left_space = p[36][0] - p[0][0]
    right_space = p[16][0] - p[45][0]
    space_ratio = left_space / right_space

    if debug:
        print(eye_dif, nose_dif, space_ratio)

    ## These rules are not perfect, determined by trying a bunch of "bad" pictures
    if eye_dif > 5 or nose_dif > 3.5 or space_ratio > 3:
        return False
    else:
        return True


I am working on the file "FWHR_calculator.ipynb".
The first task is:
Function to get the contour face points
This function calculates the coordinates for the corners of the "FWHR" box.
Note 1: it is possible to calculate the top line based on either the bottom of the eyebrows (top = "eyebrow") or the eyelids (top = "eyelid").
Note 2: to counter-act small amounts of rotation it will by default take the average between the height of the two top points.
Can you write Python code for it?

def get_face_points(points, method='average', top='eyebrow'):
    width_left, width_right = points[0], points[16]

    if top == 'eyebrow':
        top_left = points[18]
        top_right = points[25]

    elif top == 'eyelid':
        top_left = points[37]
        top_right = points[43]

    else:
        raise ValueError('Invalid top point, use either "eyebrow" or "eyelid"')

    bottom_left, bottom_right = points[50], points[52]

    if method == 'left':
        coords = (width_left[0], width_right[0], top_left[1], bottom_left[1])

    elif method == 'right':
        coords = (width_left[0], width_right[0], top_right[1], bottom_right[1])

    else:
        top_average = int((top_left[1] + top_right[1]) / 2)
        bottom_average = int((bottom_left[1] + bottom_right[1]) / 2)
        coords = (width_left[0], width_right[0], top_average, bottom_average)

    ## Move the line just a little above the top of the eye to the eyelid
    if top == 'eyelid':
        coords = (coords[0], coords[1], coords[2] - 4, coords[3])

    return {'top_left' : (coords[0], coords[2]),
            'bottom_left' : (coords[0], coords[3]),
            'top_right' : (coords[1], coords[2]),
            'bottom_right' : (coords[1], coords[3])
           }


I am working on the file "FWHR_calculator.ipynb".
The first task is:
Function to show the FWHR box on the image
This function shows the FWHR box on the original image.
Can you write Python code for it?

def show_box(image, corners):
    pil_image = Image.fromarray(image)
    w, h = pil_image.size

    ## Automatically determine width of the line depending on size of picture
    line_width = math.ceil(h / 100)

    d = ImageDraw.Draw(pil_image)
    d.line([corners['bottom_left'], corners['top_left']], width = line_width)
    d.line([corners['bottom_left'], corners['bottom_right']], width = line_width)
    d.line([corners['top_left'], corners['top_right']], width = line_width)
    d.line([corners['top_right'], corners['bottom_right']], width = line_width)

    imshow(pil_image)


I am working on the file "FWHR_calculator.ipynb".
The first task is:
Looking for examples? Click here: Examples
Preamble
Imports
Can you write Python code for it?

%pylab inline

import math

from matplotlib.pyplot import imshow
from PIL import Image, ImageDraw
import face_recognition

import urllib.request


I am working on the file "Face-Verification-Siamese-Network.ipynb".
The first task is:
Dataset is augmented by making stochastic affine transformations. These include shifting, both horizontally as well as vertically, horizontal flipping, and rotating the images.
Can you write Python code for it?

data_gen_args = dict(
	rotation_range=10,
	width_shift_range=0.05,
	height_shift_range=0.05,
	horizontal_flip=True)

image_datagen = ImageDataGenerator(**data_gen_args)

maxDraws = 6 # Number of transformations per image
augmentedImagesByClass = [[] for k in range(40)]
for nClass in uniqueClasses:
	numDraws = 0
	emptArr = []
	for _x, _y in image_datagen.flow(allImagesByClass[nClass], [nClass]*numSamplesPerClass):
		emptArr += _x
		numDraws += 1
		if numDraws == 1:
			augmentedImagesByClass[nClass] = _x
		elif numDraws <= maxDraws:
			augmentedImagesByClass[nClass] = np.concatenate((augmentedImagesByClass[nClass], _x))
		if numDraws >= maxDraws:
			break
augmentedImagesByClass = np.array(augmentedImagesByClass)


I am working on the file "Face-Verification-Siamese-Network.ipynb".
The first task is:
Due to the limited number of classes in the training data, a leave one out cross validation is performed. This way each class appears in the test set once.
Can you write Python code for it?

numFolds = len(uniqueClasses)
# Lists to store the results of the cross validation folds
valScoresList = [[] for k in range(numFolds)]
test1ScoresList = [[] for k in range(numFolds)]
test2ScoresList = [[] for k in range(numFolds)]

# Hyperparmaters used for the network
lambdaConv = 1e-4
lambdaDense = 1e-4
normalStddev = 1e-2
biasMean = 0.5
numEpochs = 10
miniBatchSize = 128

# Cross validation loop
for testClass in range(numFolds):
	trainClasses = uniqueClasses[:testClass]+uniqueClasses[testClass+1:]
	randClass = random.randrange(1, numTrainClasses) # Pick a random class for validation set
	valClass = trainClasses.pop(randClass)

	# Creating training set
	# Training data has equal distribution of same and different class images
	trainPairs = []
	trainLabels = []
	for nClass in trainClasses:
		for nSample1, nSample2 in combinations(range(numSamplesPerClass*6),2):
			trainPairs += [[augmentedImagesByClass[nClass, nSample1], augmentedImagesByClass[nClass, nSample2]]]
			inc = random.randrange(1, numTrainClasses)
			idx = (nClass + inc) % numTrainClasses
			nClass2 = trainClasses[idx]
			trainPairs += [[augmentedImagesByClass[nClass, nSample1], augmentedImagesByClass[nClass2, nSample2]]]
			trainLabels += [1, 0]
	trainPairs = np.array(trainPairs)
	trainLabels = np.array(trainLabels)

	# Creating validation set
	# Validation data has equal distribution of same and different class images
	valPairs = []
	valLabels = []
	for nSample1, nSample2 in combinations(range(numSamplesPerClass),2):
		valPairs += [[allImagesByClass[valClass, nSample1], allImagesByClass[valClass, nSample2]]]
		inc = random.randrange(1, numTrainClasses)
		nClass2 = trainClasses[inc % numTrainClasses]
		valPairs += [[allImagesByClass[valClass, nSample1], allImagesByClass[nClass2, nSample2]]]
		valLabels += [1, 0]
	valPairs = np.array(valPairs)
	valLabels = np.array(valLabels)

	# Creating test set 1
	# Constructed by drawing a pair of images from held out test class
	testPairs = []
	testLabels = []
	for nSample1, nSample2 in...


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
comments
Can you write Python code for it?

# 讀入comment csv
c_path = path = 'comment/'+page_id+'_comment.csv'
c_df = pd.read_csv(c_path)

c_df.head()

c_df = c_df[c_df['comment_message'].notnull()].reindex()

sorted_comment_message = jieba_extract(list(c_df['comment_message']))
print (sorted_comment_message[:10])
tpath = '/home/wy/font/NotoSansCJKtc-Black.otf'
wordcloud = WordCloud(max_font_size=120, relative_scaling=.1, width=900, height=600, font_path=tpath).fit_words(sorted_comment_message)
plt.figure()
plt.imshow(wordcloud)
plt.axis("off")
plt.show()

c_df = c_df[c_df['comment_author'].notnull()].reindex()

def word_count(data_list):
    d = {}
    for one in data_list:
        if one not in d:
            d[one] = 1
        else:
            d[one] += 1
    return d

d = word_count(list(c_df['comment_author']))
comment_authors = [(k, d[k]) for k in sorted(d, key=d.get, reverse=True)]
print (comment_authors[:10])
tpath = '/home/wy/font/NotoSansCJKtc-Black.otf'
wordcloud = WordCloud(max_font_size=120, relative_scaling=.1, width=900, height=600, font_path=tpath).fit_words(comment_authors)
plt.figure()
plt.imshow(wordcloud)
plt.axis("off")
plt.show()


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
share和comment相關性較高,留言和分享的人數相關性比留言和reaction的人數相關性高
Can you write Python code for it?

df_tmp = df[['num_reactions', 'num_comments', 'num_shares']]
colormap = plt.cm.viridis
plt.title('Pearson Correlation of Features', y=1.05, size=15)
sns.heatmap(df_tmp.astype(float).corr(),linewidths=0.1,vmax=1.0, square=True, cmap=colormap, linecolor='white', annot=True)


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
先介紹一下jieba的使用
一般使用jieba分詞呼叫的api jieba.cut 而本文則是使用 jieba.analyse.extract_tags(基於 TF-IDF算法關鍵詞抽取)
原因在於透過TF-IDF評估單詞對於文件的集合或詞庫中一份文件的重要程度,就可以過濾掉不重要的字,看以下範例 :
Can you write Python code for it?

list(df['status_message'])[99]

for one in jieba.cut(list(df['status_message'])[99]):
    print (one)

jieba.analyse.extract_tags(list(df['status_message'])[99], topK=120)


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
出現文字雲！ 文字雲其實還有蠻多可以調整的,甚至可以把圖片填滿字,請參考  wordcloud
會出現http com www 是因為每篇發文都會有link連結,所以就把link連結解析出來了
Can you write Python code for it?

tpath = '/home/wy/font/NotoSansCJKtc-Black.otf'
wordcloud = WordCloud(max_font_size=120, relative_scaling=.1, width=900, height=600, font_path=tpath).fit_words(sorted_word_count)
plt.imshow(wordcloud)
plt.axis("off")
plt.show()


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
分析post中的文字
需要用到jieba!
jieba是一個強大的中文斷詞程式,可以幫你去分析你要分析的文本內容
wordcloud 則是畫文字雲的套件
Can you write Python code for it?

import jieba
import jieba.analyse
import operator
from wordcloud import WordCloud

# 安裝jieba套件的時候,就有繁體詞庫
jieba.set_dictionary('/home/wy/anaconda3/envs/python3/lib/python3.6/site-packages/jieba/extra_dict/dict.txt.big')


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
各個不同發文類型的reaction長條圖
Can you write Python code for it?

g = sns.FacetGrid(df, col="status_type")
g.map(plt.hist, "num_reactions")


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
單日第幾時發文散佈圖
單一個點代表該post是什麼種類和得到多少reaction
Can you write Python code for it?

sns.stripplot(x="hour", y="num_reactions", data=df, jitter=True)


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
因此我們可以把每一篇發文的內容透過jieba的關鍵字抽取,抽取出重要的字
然後統計這些字出現的頻率,並使用WordCloud畫成文字雲
Can you write Python code for it?

def jieba_extract(message_list):
    word_count = {}
    for message in message_list:
        # 在抽取關鍵字時,可能會發生錯誤,先把錯誤的message收集起來,看看是怎麼一回事
        seg_list = jieba.analyse.extract_tags(message, topK=120)
        for seg in seg_list:
            if not seg in word_count:
                word_count[seg] = 1
            else:
                word_count[seg] += 1

    sorted_word_count = sorted(word_count.items(), key=operator.itemgetter(1))
    sorted_word_count.reverse()
    return sorted_word_count
sorted_word_count = jieba_extract(list(df['status_message']))


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
我們先用pandas中的describe (Generate various summary statistics),去統計該欄位的平均數,中位數,標準差,四分位數...
Can you write Python code for it?

df_num_reactions = df['num_reactions'].describe().to_frame(name='reactions')
df_num_reactions

df_num_comments = df['num_comments'].describe().to_frame(name='comments')
df_num_comments

df_num_shares = df['num_shares'].describe().to_frame(name='shares')
df_num_shares


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
把這些過濾掉,並且重新做reindex,原因是因為內建過濾的時候,它的index是不會改變的
Can you write Python code for it?

df = df[(df['num_reactions']!=0) & (df['status_message'].notnull())].reindex()


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
按讚,留言,分享隨時間變化趨勢圖
x軸是時間,y軸是reactions,comments,wows,shares的數目,可以看出隨時間變化的按讚,留言,分享的趨勢
2016有數次綠色高於其他顏色,也就是說留言數大於反應和分享,推測可能是留言送禮的post吧
Can you write Python code for it?

df.plot(x='datetime', y=['num_reactions', 'num_comments', 'num_shares'],
       figsize=(12,8))


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
星期幾發文數目統計長條圖
看起來星期日較少發文
Can you write Python code for it?

sns.barplot(x='index', y='weekday', data = df_weekday)

def hour(d):
    list_key = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23]
    list_value = []
    for one in list_key:
        if one in d.keys():
            list_value.append(d[one])
        else:
            list_value.append(0)
    df = pd.DataFrame(index = list_key, data = {'hour': list_value}).reset_index()
    return df

df_hour = hour(dict(df['hour'].value_counts()))
df_hour


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
發文種類散佈圖
單一個點代表該發文是什麼種類和得到多少reaction,photo發文的reaction在0-50000最為密集
Can you write Python code for it?

sns.stripplot(x="status_type", y="num_reactions", data=df, jitter=True)


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
發文種類數目統計長條圖
最喜歡的發文形式是圖片,再來是分享連結和影片
Can you write Python code for it?

sns.barplot(x='index', y='status_type', data = df_status_type.reset_index())


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
發文頻率統計
統計每個post的發布時間間隔,就可以了解發文的頻率
雖然平均的發文頻率是12小時,但是看四分位數,前四分之一是半小時,前四分之三是五小時,
表示發文是非常頻繁的,一天好幾po文,至於平均數和標準差會這麼大,推測是初期經營臉書的發文頻率過低的影響
被前期的outlier影響
Can you write Python code for it?

import datetime
delta_datetime = df['datetime'].shift(1) - df['datetime']
delta_datetime_df = pd.Series(delta_datetime).describe().apply(str)
delta_datetime_df = delta_datetime_df.to_frame(name='frequent of posts')
delta_datetime_df


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
看一下最常出現在post的字前十名是啥
Can you write Python code for it?


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
處理前總共5234筆
Can you write Python code for it?


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
處理後剩下5061筆,總共過濾掉了173筆
Can you write Python code for it?


I am working on the file "Facebook粉絲頁分析三部曲-分析和輸出報表篇.ipynb".
The first task is:
輸出excel報表
當我們在上面做了許多分析，那能不能把這些分析都輸成excel報表！？
用python輸出excel就靠它了 xlsxwriter ！
Can you write Python code for it?


I am working on the file "Facebook粉絲頁分析三部曲-爬取篇(comments).ipynb".
The first task is:
url = base + node + fields + parameters
base : 可以設定Facebook Graph API的版本,這邊設定v2.6
node : 分析哪個粉絲頁的post 由page_id去設定
fields : 你要取得資料的種類
parameters : 權限設定和每次取多少筆(num_statuses)
Can you write Python code for it?

def getFacebookCommentFeedData(status_id, access_token, num_comments):

    base = "https://graph.facebook.com/v2.6"
    node = "/%s/comments" % status_id
    fields = "?fields=id,message,like_count,created_time,comments,from,attachment"
    parameters = "&order=chronological&limit=%s&access_token=%s" % \
            (num_comments, access_token)
    url = base + node + fields + parameters

    # 取得data
    data = request_until_succeed(url)
    if data is None:
        return None
    else:
        return data.json()

def processFacebookComment(comment, status_id, parent_id = ''):

    # 確認資料欄位是否有值,並做處理
    comment_id = comment['id']
    comment_author = comment['from']['name']
    if 'message' not in comment:
        comment_message = ''
    else:
        comment_message = comment['message']
    if 'like_count' not in comment:
        comment_likes = 0
    else:
        comment_likes = comment['like_count']

    if 'attachment' in comment:
        attach_tag = "[[%s]]" % comment['attachment']['type'].upper()
        if comment_message is '':
            comment_message = attach_tag
        else:
            comment_message = (comment_message+ " " +attach_tag)

    comment_published = datetime.datetime.strptime(comment['created_time'],'%Y-%m-%dT%H:%M:%S+0000')
     # 根據所在時區 TW +8
    comment_published = comment_published + datetime.timedelta(hours=8)
    comment_published = comment_published.strftime('%Y-%m-%d %H:%M:%S')

    # 回傳tuple形式的資料
    return (comment_id, status_id, parent_id, comment_message, comment_author,
            comment_published, comment_likes)


I am working on the file "Facebook粉絲頁分析三部曲-爬取篇(comments).ipynb".
The first task is:
總共跑完有690628筆,106Mb,要花十幾個小時
all_statuses[0] 為 column name
all_statuses[1:] 為處理後結構化的資料
Can you write Python code for it?

all_comments = scrapeFacebookPageFeedComments(page_id, access_token, post_path)
df = pd.DataFrame(all_comments[1:], columns=all_comments[0])

path = 'comment/'+page_id+'_comment.csv'
df.to_csv(path,index=False,encoding='utf8')