I am working on the file "RollingDice.ipynb".
The first task is:
Computing on distributions
While visualization is useful for humans in the data exploration process, everything you see you should be able to compute upon.  The analog of Table.hist that yields a table, rather than a chart is table.bin.  It returns a new table with a row for each bin.
Here we also illustrate doing some computing on the distribution table:
* A column of a table is accessed using the standard python get syntax: <object> [ <key> ]. This actually yields an object that is a numpy array, but part of the beauty of tables is you don't have to worry about what that is. The beauty of numpy arrays is that you can work with them pretty much like values, i.e., you can scale them by a constant, add them together and things like that.
* A column is inserted in the table using the standard python set syntax for objects <object> [ <key> ] = <value>. Note that this modifies the table, adding a column if it does not exist ro updating it if it does.  The transformations on tables are functional, they produce new tables.  Set treats a table like an object and modifies it.
Can you write Python code for it?

roll_dist = rolls.bin(normed=True,bins=bins).take(range(6))
roll_dist

roll_dist['roll density']

roll_dist['Variation'] = (roll_dist['roll density'] - 1/6)/(1/6)
roll_dist

# What is the average value of a roll?
sum(roll_dist['bin']*roll_dist['roll density'])

np.mean(rolls['roll'])


I am working on the file "RollingDice.ipynb".
The first task is:
Interactive visualization
The central concept of computational thinking - abstraction. Here it is illustrated again by wrapping up the process of rolling many die and visualizing the resulting distribution into a function.
Once we have it as a function, we can illustrate the central concept of inferential thinking - the law of large numbers - through interactive visualization.  When a dies is rolled only a few times, the resulting distribution may be very uneven.  But when it is rolled many many times, it is extremely rare for the result to be uneven.
Can you write Python code for it?

def show_die_dist(n):
    """Roll a die n times and show the distribution of sides that appear."""
    roll(n).hist(bins=np.arange(1,8))

# We can now use the ipywidget we had included at the beginning.
interact(show_die_dist, n=(10, 1000, 10))


I am working on the file "RollingDice.ipynb".
The first task is:
Likelihood
If we really roll the dice several times in life, what might we expect the overall outcome to be like?
We can extend our computational approach further by simulating the rolling of several die many many times.
Can you write Python code for it?

num_die = 10

num_rolls = 100

# Remember - referencing a column gives an array
roll(num_die)['side']

# Simulate rolling num_die dice num_rolls times and build a table of the result
rolls = Table(["die_"+str(i) for i in range(num_die)]).with_rows([roll(num_die)['side'] for i in range(num_rolls)])
rolls

# If we think of each row as a life experience, what is the life like?
label = "{}_dice".format(num_die)
sum_rolls = Table().with_column(label, [np.sum(roll(num_die)['side']) for i in range(num_rolls)])
sum_rolls.hist(range=[10,6*num_die], normed=False)
sum_rolls.stats()

# Or as a distribution
sum_rolls.hist(range=[10,6*num_die],normed=True)

# Or normalize by the number of die ...
#
Table().with_column(label, [np.sum(roll(num_die)['side'])/num_die for i in range(num_rolls)]).hist(normed=False)


I am working on the file "Rootfinding.ipynb".
The first task is:
Automatically choosing a suitable $\epsilon$
Can you write Python code for it?

def diff_wp(f, x, eps=1e-8):
    """Numerical derivative with Walker and Pernice (1998) choice of step"""
    h = eps * (1 + abs(x))
    return (f(x+h) - f(x)) / h

x = 1
[(eps, diff_wp(numpy.log, x, eps) - 1/x) for eps in [1e-14, 1e-12, 1e-10, 1e-8, 1e-6, 1e-4, 1e-2]]

x = 1e-4
[(eps, diff_wp(numpy.log, x, eps) - 1/x) for eps in [1e-14, 1e-12, 1e-10, 1e-8, 1e-6, 1e-4, 1e-2]]


I am working on the file "Rootfinding.ipynb".
The first task is:
Did we solve all of these equations?
How can the iteration break down?
Does choosing a different initial guess lead to different solutions?
How is this convergence test different from the one we used for bisection?
Is the convergence rate similar for all test equations?

Convergence of Newton-type algorithms
We would like to know sharp conditions on when Newton-type algorithms converge, and if so, how fast.  This theory will build on that for a general Fixed Point Iteration $x_{i+1} = g(x_i)$ where $g$ is a continuously differentiable function.  Suppose that there exists a fixed point $r = g(r)$.  By the mean value theorem, we have that
$$ x_{i+1} - r = g(x_i) - g(r) = g'(c_i) (x_i - r) $$
for some $c$ with $|c - r| < |x_i - r|$.
In other words, $|e_{i+1}| = |g'(c_i)| |e_i|$, which converges to zero if $|g'(c_i)| < 1$.
If $|g'(r)| < 1$ then for any $\epsilon > 0$ there is a neighborhood of $r$ such that $|g'(c)| < |g'(r)| + \epsilon$ for all $c$ in that neighborhood.
Consequently, we have:
Theorem (Sauer 1.6): Linear Convergence of Fixed Point Iteration
If $g$ is continuously differentiable, $r = g(r)$, and $|g'(r)| < 1$ then the fixed point iteration $x_{i+1} = g(x_i)$ is locally linearly convergent with rate $|g'(r)|$.
Observations

A rootfinding problem $f(x) = 0$ can be converted to a fixed point problem $x = x - f(x) =: g(x)$ but there is no guarantee that $g'(r) = 1 - f'(r)$ will have magnitude less than 1.
Problem-specific algebraic manipulation can be used to make $|g'(r)|$ small.
$x = x - h(x)f(x)$ is also a valid formulation for any $h(x)$ bounded away from $0$.
Can we choose $h(x)$ such that $ 1 - h'(x)f(x) - h(x)f'(x) = 0$ when $f(x) = 0$?

In other words,
$$ x_{i+1} = x_i - ??? . $$

It turns out that Newton's method has locally quadratic convergence to simple roots, $\lim_{i \to \infty} |e_{i+1}|/|e_i^2| < \infty$.
"The number of correct digits doubles each iteration."
Now that we know how to make a good guess accurate, the effort lies in getting a good guess.

Culture: fast inverse square...
Can you write Python code for it?

def fquartic(x):
    return (x - 0.9)**4, 4*(x - 0.9)**3

newton(fquartic, 0)


I am working on the file "Rootfinding.ipynb".
The first task is:
In class 2016-09-14
We are looking for a function for which the linear fit (newton solver) is more reliable than a quadratic fit (cubic solver).
Can you write Python code for it?

x = numpy.linspace(-5,5)
pyplot.figure()
f = numpy.sin(x) + 1.1*x
pyplot.plot(x, f)


I am working on the file "Rootfinding.ipynb".
The first task is:
Notice that we need to define hasroot above.

Let's try running it:
Can you write Python code for it?

bisect(tests[0], 0, 2)

numpy.sqrt(2) - bisect(tests[0], 0, 2)


I am working on the file "Rootfinding.ipynb".
The first task is:
This algorithm is imperfect, leaving some scaling responsibility to the user.
It is the default in PETSc's "matrix-free" Newton-type solvers.
Tinkering in class (2016-09-06)
Can you write Python code for it?

numpy.log(-1)

x = numpy.sqrt(-1)
1/numpy.inf

x = numpy.linspace(0,3,100)
pyplot.plot(x, numpy.sqrt(x + 1e-1))
pyplot.show()

numpy.tan(1e100)

numpy.tan(1e100*(1 + 2*eps_machine))

1e100 - 1e100*(1 + 2*eps_machine)


I am working on the file "Rootfinding.ipynb".
The first task is:
We get about 5 digits of accuracy.  Why?  How fast did we get there?
Can you write Python code for it?


I am working on the file "Rootfinding.ipynb".
The first task is:
We only get three digits correct despite a very small residual (and it takes many iterations to get there).
Difficulty computing zeros of polynomials can also arise when all the roots are simple.  For example, the Wilkinson polynomial
$$ \prod_{i=1}^{20} (x - i) = \sum_{i=0}^{20} a_i x^i $$
has roots at each of the positive integers up to 20, but the roots are extremely sensitive to perturbations of the coefficients $a_i$, as shown in this figure from Trefethen and Bau (1999).

Numerical difficulties in which "correct" algorithms produce unreliable solutions almost always stem from lack of stability and/or ill conditioning.
Absolute condition number
Consider a function $f: X \to Y$ and define the absolute condition number
$$ \hat\kappa = \lim_{\delta \to 0} \max_{|\delta x| < \delta} \frac{|f(x + \delta x) - f(x)|}{|\delta x|} = \max_{\delta x} \frac{|\delta f|}{|\delta x|}. $$
If $f$ is differentiable, then $\hat\kappa = |f'(x)|$.
Floating point arithmetic
Floating point arithmetic $x \circledast y := \text{float}(x * y)$ is exact within a relative accuracy $\epsilon_{\text{machine}}$.  Formally,
$$ x \circledast y = (x * y) (1 + \epsilon) $$
for some $|\epsilon| \le \epsilon_{\text{machine}}$.
Can you write Python code for it?


I am working on the file "Rootpy Ratio Plots.ipynb".
The first task is:
<b>REMEMBER:</b>
If you don't know the name of the tree or branches.. or even structures you can use
list_trees(), list_structures(), list_branches()
for example....
Can you write Python code for it?

branches = rtnp.list_branches("vbf125spin2pmin.root")

branches


I am working on the file "Rootpy Ratio Plots.ipynb".
The first task is:
Anyways let's fill the histogram..
<b> Remember </b> Mll is in [MeV] so simply divide each index by 1000. to get GeV's
<b> WEIGHTS: </b>
This is Monte Carlo sample. So the events are weighted! The weights are stored in the 'EventWeight' branch.. So simply use it as weights, like this:
Can you write Python code for it?

rtnp.fill_hist(hist_mll, data['Mll']/1000., weights=data['EventWeight'])

hist_mll


I am working on the file "Rootpy Ratio Plots.ipynb".
The first task is:
Let's make a fake ratio plot
Can you write Python code for it?

ratio = hist_mll.Clone()

ratio.Divide(hist_mll)

pad_hist = Pad(0,0.3,1,1); set_pad_style(pad_hist); pad_hist.Draw()

pad_ratio = Pad(0,0,1,0.3); set_subpad_style(pad_ratio); pad_ratio.Draw()

pad_hist.cd()

hist_mll.Draw()

pad_ratio.cd()

ratio.draw('E4')

canvas

canvas


I am working on the file "Rootpy Ratio Plots.ipynb".
The first task is:
Let's make it simple for newbs... now get the tree...
Can you write Python code for it?


I am working on the file "Rootpy Ratio Plots.ipynb".
The first task is:
Let's use the default 'ATLAS' style of roottpy (I dont like it hmm, but to bored to make it myself)
Can you write Python code for it?

from rootpy.plotting.style import get_style, set_style

style = get_hgs3_style()#set_style(#'ATLAS')

#style.SetLabelOffset(0.4,'y')

set_style(style)


I am working on the file "Rootpy Ratio Plots.ipynb".
The first task is:
Open my file with root_open (rootpy.io)
Can you write Python code for it?


I am working on the file "Rosenzweig-MacArthur.ipynb".
The first task is:
Condicion inicial dentro del ciclo límite
Can you write Python code for it?

t = np.arange(pasos)

plt.xlabel("tiempo")
plt.ylabel("$\dot{x},\dot{y}$")
fig = plt.plot(t, x_dentro, color="darkmagenta")
fig = plt.plot(t, y_dentro, color="darkorchid")


I am working on the file "Rosenzweig-MacArthur.ipynb".
The first task is:
Condicion inicial fuera del ciclo límite
Can you write Python code for it?

t = np.arange(pasos)

plt.xlabel("tiempo")
plt.ylabel("$\dot{x},\dot{y}$")
fig = plt.plot(t, x_fuera, color="lawngreen")
fig = plt.plot(t, y_fuera, color="lime")


I am working on the file "Rosenzweig-MacArthur.ipynb".
The first task is:
Equilibrios
Can you write Python code for it?

# ¿en qué lugares las tasas de cambio son iguales a cero?
f0 = Eq(dx, 0)
g0 = Eq(dy, 0)
equilibrios = solve((f0, g0), x, y)

# dos equilibrios dependen de los valores de e, m y k
eq3x = lambdify((e,m), equilibrios[2][0])
eq3y = lambdify((e,m,k), equilibrios[2][1])

equilibrios

def presas(m, k):
    """
    devuelve una función de (x, y)
    con los argumentos (m, k) interpolados
    """
    def f(x, y):
        return x*(1-x/k)-((m*x*y)/(1+x))
    return f

def depredadores(e, m):
    """
    devuelve una función de (x, y)
    con los argumentos (e, m) interpolados
    """
    def g(x, y):
        return (-e*y)+((m*x*y)/(1+x))
    return g

m = 3
k = 3
e = 1
f = presas(m, k)
g = depredadores(e, m)

x, y = np.meshgrid(np.linspace(0, 3.5, 22),
                   np.linspace(-0.5, 1, 22))

u = f(x, y)
v = g(x, y)

# campo de vectores
plt.quiver(x, y, u, v)

# equilibrios
plt.plot(0, 0, 'red', marker='o', markersize=10)
plt.plot(k, 0, 'red', marker='o', markersize=10)
plt.plot(eq3x(e,m), eq3y(e,m,k), 'red', marker='o', markersize=10)

plt.xlabel("$\dot{x}$")
plt.ylabel("$\dot{y}$")

plt.show()


I am working on the file "Rosenzweig-MacArthur.ipynb".
The first task is:
Interacción de ambas poblaciones
La sección 2 del mismo artículo, Hal L. Smith discute una forma más conveniente del mismo modelo en la que los parámetros de tiempo $a$ y $b$ se reemplazan por el argumento adimensional $m$.
$$\begin{cases}\dot{x}=x\left(1-\frac{x}{k}\right)-\frac{mxy}{1+x}\\dot{y}=-ey+\frac{mxy}{1+x}\end{cases}$$
Can you write Python code for it?


I am working on the file "Rosenzweig-MacArthur.ipynb".
The first task is:
Tasa de cambio de la población de presas
La tasa de cambio de $ x $ (nacimientos y muertes de presas) podría modelarse como crecimiento Maltusiano $f(x)=\frac{dx}{dt}=rx\left(1-\frac{x}{K}\right)$, donde la tasa de reproducción es proporcional tanto al tamaño de la población actual como a la cantidad de recursos disponible ($ r,K>0 $).
Can you write Python code for it?

malthusian = r*x*(1-x/K)
malthusian

F = lambdify((x, r, K), malthusian)
pop = np.zeros(100)

# población inicial
pop[0] = 1
reproduccion = 0.1
capacidad = 200

# sumamos población en cada t
for t in range(99):
    pop[t+1] = pop[t] + F(pop[t],
                          reproduccion,
                          capacidad)

plt.xlabel("tiempo")
plt.ylabel("población de presas sin depredación")
fig = plt.plot(pop)


I am working on the file "RotazioniSubreflettore.ipynb".
The first task is:
Presupponiamo ora di inquadrare un ricevitore con un angolo di $\theta = 4^{\circ}$ e di voler effettuare un focheggiamento muovendo il subreflettore verso il ricevitore di una quantità che facciamo variare tra 0 e 18cm, calcolati come $ 3\lambda  @ 5 GHz$ . Anche in questo caso possiamo controllare il variare delle correzioni applicate in Z e X al variare della distanza del subreflettore dal fuoco.
Can you write Python code for it?

xaxis = linspace(0, 180, 1000)
z_at_fixed_t = array([N(z.subs(t, radians(4)).subs(d, _d)) for _d in xaxis])
x_at_fixed_t = array([N(x.subs(t, radians(4)).subs(d, _d)) for _d in xaxis])

pylab.title("Z correction at theta = 4 deg.")
pylab.xlabel("D mm")
pylab.ylabel("Z mm")
pylab.grid(True)
pylab.plot(xaxis, z_at_fixed_t, 'b-')

pylab.title("X correction at theta = 4 deg.")
pylab.xlabel("D mm")
pylab.ylabel("X mm")
pylab.grid(True)
pylab.plot(xaxis, x_at_fixed_t, 'g-')


I am working on the file "RotazioniSubreflettore.ipynb".
The first task is:
Sapendo che $ D = 310.7mm $ calcolato geometricamente per il subreflettore possiamo calcolare il variare di X e Z al variare dell'angolo in vertex
Can you write Python code for it?

xaxis = linspace(0, radians(8), 1000)
xaxis_labels = linspace(0, 8, 1000)
#z_at_fixed_d = 2 * 310.7 * np.sin(xaxis/2)**2
#x_at_fixed_d = 310.7 * np.sin(xaxis)
z_at_fixed_d = array([N(z.subs(d, 310.7).subs(t, _theta)) for _theta in xaxis])
x_at_fixed_d = array([N(x.subs(d, 310.7).subs(t, _theta)) for _theta in xaxis])


I am working on the file "RotazioniSubreflettore.ipynb".
The first task is:
Sia dato il sistema in figura in cui:
* $AB$ è metà di un lato del triangolo subreflettore (vista B in Fig. 3.1.2 pag. 90 "Matematica di sistema" 4 di 4)
* $|AB| = 2068mm / 2 $
* $\varphi$ è l'angolo di cui vgliamo ruotare attorno ad un asse il subreflettore
* $BC$ rappresenta il posizionamento del subreflettore una volta compiuta la rotazione
NOTA: Per semplicità consideriamo il caso di rotazione sull'asse X applicando un angolo $\theta_y$ che si riflette sul solo movimento di due attuatori. In questo caso sto solo cercando di quantificare un possibile errore.
Ragionando sui triangoli e sfruttando il fatto che $|AB| = |BC|$ possiamo desumere che:
 * $\omega = \Omega$ perchè alterni-interni
$$ \Omega = \omega = \frac{\pi - \varphi}{2} = \frac{\pi}{2} - \frac{\varphi}{2} $$
$$ \sin{\Omega} = \sin{\omega} = \cos{\frac{\varphi}{2}} $$
$$ \cos{\Omega} = \cos{\omega} = \sin{\frac{\varphi}{2}} $$
Ragionando a questo punto sui lati otteniamo che:
$$ |AC| = 2|AB|\sin(\frac{\varphi}{2}) $$
$$ |AE| = |AC| * \sin{\omega} = 2|AB|\sin{\frac{\varphi}{2}} * \cos{\frac{\varphi}{2}} = 2|AB|\frac{\sin{\varphi}}{2} = |AB|\sin{\varphi}$$
$$ |EC| = |AC| * \cos{\omega} = 2|AB|\sin{\frac{\varphi}{2}} * \sin{\frac{\varphi}{2}}  = 2|AB|(\sin{\frac{\varphi}{2}})^2 = |AB|(1 - \cos{\varphi}) $$
$$ |ED| = |EC| * \tan{\varphi} = |AB| \tan{\varphi} - |AB|\cos{\varphi}*\frac{\sin{\varphi}}{\cos{\varphi}} = |AB| * (\tan{\varphi} - \sin{\varphi})$$
$$ |AD| = |AB| * \tan{\varphi} $$
Que e ci interessa è che quindi quando noi vogliamo comandare un tilt di $\varphi$ e diamo come posizione comandata $|AD|$ in realtà dovremmo dare un valore di $|AC|$ , per cui:
$$ |AD| - |AC| = |AB| * \tan{\varphi} - 2|AB|\sin{\frac{\varphi}{2}} = |AB|(\tan{\varphi} - 2\sin{\frac{\varphi}{2}}) $$
Fissanso quindi come da ipotesi $|AB| = 2098/2mm$ calcoliamo l'errore al variare di $\varphi$:
Can you write Python code for it?

xaxis = linspace(0, radians(8), 1000)
xaxis_labels = linspace(0, 8, 1000)
error = (2098/2) * (np.tan(xaxis) - 2*np.sin(xaxis/2))

pylab.title("Correzione attuatore vs. angolo di tilt")
pylab.xlabel("Phi deg.")
pylab.ylabel("Error mm")
pylab.grid(True)
pylab.plot(xaxis_labels, error, 'b-',
           xaxis_labels[500], error[500], 'bo',
           markersize = 12)


I am working on the file "RotazioniSubreflettore.ipynb".
The first task is:
Tilt del subreflettore
Can you write Python code for it?


I am working on the file "Route_option/.ipynb_checkpoints/Route Prediction-checkpoint.ipynb".
The first task is:
1 signifies that it is a metro
Let's visualize the data based on the rating of the drivers
Can you write Python code for it?

group_by_age = pd.cut(route_df["rating"], np.arange(0, 6, 1))
rating_grouping = route_df.groupby(group_by_age).mean()
rating_grouping['mapUsed'].plot.bar()


I am working on the file "Route_option/.ipynb_checkpoints/Route Prediction-checkpoint.ipynb".
The first task is:
47% of the drivers are following the map.
Let's see the groupings by the country
Can you write Python code for it?


I am working on the file "Route_option/.ipynb_checkpoints/Route Prediction-checkpoint.ipynb".
The first task is:
Let's look at the data
Can you write Python code for it?


I am working on the file "Route_option/.ipynb_checkpoints/Route Prediction-checkpoint.ipynb".
The first task is:
Most of the drivers are between the range of 1-2
Let's check for missing values by doing a count on each of the columns
Can you write Python code for it?


I am working on the file "Route_option/.ipynb_checkpoints/Route Prediction-checkpoint.ipynb".
The first task is:
What we are basically doing here is processing the data to produce numeric labels for the string data
Let's look at the data again
Can you write Python code for it?

processed_df = preprocess_route_df(route_df)
processed_df


I am working on the file "Route_option/.ipynb_checkpoints/Route Prediction-checkpoint.ipynb".
The first task is:
decision tree
Can you write Python code for it?

clf_dt = tree.DecisionTreeClassifier(max_depth=10)

clf_dt.fit (X_train, y_train)
clf_dt.score (X_test, y_test)

shuffle_validator = cross_validation.ShuffleSplit(len(X), n_iter=20, test_size=0.2, random_state=0)
def test_classifier(clf):
    scores = cross_validation.cross_val_score(clf, X, y, cv=shuffle_validator)
    print("Accuracy: %0.4f (+/- %0.2f)" % (scores.mean(), scores.std()))

test_classifier(clf_dt)

clf_rf = ske.RandomForestClassifier(n_estimators=50)
test_classifier(clf_rf)

clf_gb = ske.GradientBoostingClassifier(n_estimators=50)
test_classifier(clf_gb)

eclf = ske.VotingClassifier([('dt', clf_dt), ('rf', clf_rf), ('gb', clf_gb)])
test_classifier(eclf)


I am working on the file "Route_option/.ipynb_checkpoints/Route Prediction-checkpoint.ipynb".
The first task is:
neural Network
Can you write Python code for it?

#tf_clf_dnn = skflow.TensorFlowDNNClassifier(hidden_units=[20, 40, 20], n_classes=2, batch_size=256, steps=1000, learning_rate=0.05)
feature_columns = [tf.contrib.layers.real_valued_column("")]
tf_clf_dnn = skflow.DNNClassifier(feature_columns=feature_columns, hidden_units=[20, 40, 20], n_classes=2, model_dir="/tmp")
#tf_clf_dnn.evaluate(batch_size=256, steps=1000)
tf_clf_dnn.fit(X_train, y_train, steps=1000)
accuracy_score = tf_clf_dnn.evaluate(X_test, y_test,steps=1000)["accuracy"]
print("\nTest Accuracy: {0:f}\n".format(accuracy_score))


I am working on the file "Route_option/Route_option.ipynb".
The first task is:
1 signifies that it is a metro
Let's visualize the data based on the rating of the drivers
Can you write Python code for it?


I am working on the file "Route_option/Route_option.ipynb".
The first task is:
47% of the drivers are following the map.
Let's see the groupings by the country
Can you write Python code for it?


I am working on the file "Route_option/Route_option.ipynb".
The first task is:
Let's look at what percentage of the drivers are using the map?
Can you write Python code for it?


I am working on the file "Route_option/Route_option.ipynb".
The first task is:
There are no missing values. However if there are missing values we can deal with them in the following way:

If the column (col1) from which the values are missing is an important factor, we need to drop the rows containing those values.
python
 route_df["col1"] = route_df["col1"].fillna("NA") #first fill the columns with value NA
 route_df = route_df.dropna() #drop the rows with missing values

If the column(col2,col3) is not an important factor, we can drop the column
python
route_df = route_df.drop(['col2','col3'], axis=1) #drop the columns

Now for the preprocessing
Can you write Python code for it?

def preprocess_route_df(df):
    processed_df = df.copy()
    le = preprocessing.LabelEncoder()
    processed_df.country = le.fit_transform(processed_df.country)
    processed_df.oldRoute = le.fit_transform(processed_df.oldRoute)
    processed_df.newRoute = le.fit_transform(processed_df.newRoute)
    processed_df = processed_df.drop(['name','uid'],axis=1)
    return processed_df


I am working on the file "Route_option/Route_option.ipynb".
The first task is:
decision tree
Can you write Python code for it?


I am working on the file "Route_option/Route_option.ipynb".
The first task is:
neural Network
Can you write Python code for it?


I am working on the file "Rule 18 Domain.ipynb".
The first task is:
Look at same stuff at depth 4 light cones
Can you write Python code for it?

sixth_test = epsilon_field(domain.get_spacetime())
sixth_test.estimate_states(4,4,1,alpha = 0)
sixth_test.filter_data()

print sixth_test.number_of_states()

print np.size(np.unique(sixth_test.PLCs().all_labels()))

for state in sixth_test.causal_states():
    print state.index(), state.plc_configs(), '\n'

for state in sixth_test.causal_states():
    print state.index(), state.morph_support_configs(), np.size(state.morph_support_configs()), '\n'

diagram(domain.get_spacetime(), t_min = 400, t_max = 440, x_min = 400, x_max = 440)


I am working on the file "Rule 18 Domain.ipynb".
The first task is:
Theoretically, the domain is just a checkerboard tiling of fixed 0 and wildcard. So there should be just two states, and depth 2 light cones should be sufficient.
Let's try depth 2, with default $\alpha$
Can you write Python code for it?

first_test = epsilon_field(domain.get_spacetime())
first_test.estimate_states(2,2,1)
first_test.filter_data()

print first_test.number_of_states()

for state in first_test.causal_states():
    print state.plc_configs()

second_test = epsilon_field(domain.get_spacetime())
second_test.estimate_states(2,2,1,alpha= 0)
second_test.filter_data()

print second_test.number_of_states()

for state in second_test.causal_states():
    print state.plc_configs()

big_domain = ECA(18, domain_18(2000))
big_domain.evolve(2000)

third_test = epsilon_field(big_domain.get_spacetime())
third_test.estimate_states(2,2,1,alpha=0)
third_test.filter_data()

print third_test.number_of_states()

for state in third_test.causal_states():
    print state.plc_configs()

print 2**5

print np.size(np.unique(third_test.PLCs().all_labels()))

print np.size(np.unique(third_test.FLCs().all_labels()))

print third_test.causal_states()[0].plc_shape(0)

print third_test.causal_states()[1].plc_shape(0), '\n'
print third_test.causal_states()[1].plc_shape(1)

print third_test.causal_states()[2].plc_shape(0)

diagram(domain.get_spacetime(), t_min = 400, t_max = 440, x_min = 400, x_max = 440)

print np.size([0,1]*10)

checkerish = ECA(18, ([0,0,1]+[0,1,0])*200)
checkerish.evolve(1200)

checkerish.diagram()

fourth_test = epsilon_field(checkerish.get_spacetime())
fourth_test.estimate_states(3,3,1)
fourth_test.filter_data()

print fourth_test.number_of_states()

for state in fourth_test.causal_states():
    print state.plc_configs()


I am working on the file "Run_TCAV.ipynb".
The first task is:
Download example models and images
Open a terminal and run the following commands:
```
cd tcav/tcav_examples/image_models/imagenet
python download_and_make_datasets.py --source_dir=YOUR_PATH --number_of_images_per_folder=50 --number_of_random_folders=3
```
This script will download the following content into separate folders into a directory you specify with the --source_dir= argument:
Images
*  ImageNet images for the target Zebra class
*  Broden dataset images for three concepts (e.g. striped, dotted, zigzagged)
*  Random ImageNet class images used by TCAV for hypothesis testing of important concepts
Models
*  Inception 5h model
*  Mobilenet V2 model
Import extensions and libraries
Can you write Python code for it?

%load_ext autoreload
%autoreload 2

import tcav.activation_generator as act_gen
import tcav.cav as cav
import tcav.model  as model
import tcav.tcav as tcav
import tcav.utils as utils
import tcav.utils_plot as utils_plot # utils_plot requires matplotlib
import os
import tensorflow as tf


I am working on the file "Run_TCAV.ipynb".
The first task is:
Step 2: Write your model wrapper
The next step is to tell TCAV how to communicate with your model. See model.GoogleNetWrapper_public for details.
You can define a subclass of ModelWrapper abstract class to do this. Let me walk you thru what each function does (tho they are pretty self-explanatory).  This wrapper includes a lot of the functions that you already have, for example, get_prediction.
2.1: Tensors from the graph: bottleneck tensors and ends
First, store your bottleneck tensors in self.bottlenecks_tensors as a dictionary. You only need bottlenecks that you are interested in running TCAV with. Similarly, fill in self.ends dictionary with input, logit and prediction tensors.
2.2: Define loss
Get your loss tensor, and assigned it to self.loss. This is what TCAV uses to take directional derivatives.
While doing so, you would also want to set
python
self.y_input
this simply is a tensorflow place holder for the target index in the logit layer (e.g., 0 index for a dog, 1 for a cat).
For multi-class classification, typically something like this works:
python
self.y_input = tf.placeholder(tf.int64, shape=[None])
For example, for a multiclass classifier, something like below would work.
```python
    # Construct gradient ops.
    with g.as_default():
      self.y_input = tf.placeholder(tf.int64, shape=[None])
  self.pred = tf.expand_dims(self.ends['prediction'][0], 0)

  self.loss = tf.reduce_mean(
      tf.nn.softmax_cross_entropy_with_logits(
          labels=tf.one_hot(self.y_input, len(self.labels)),
          logits=self.pred))
self._make_gradient_tensors()

```
2.3: Call _make_gradient_tensors in init() of your wrapper
python
_make_gradient_tensors()
does what you expect - given the loss and bottleneck tensors defined above, it adds gradient tensors.
2.4: Fill in labels, image shapes and a model name.
Get the mapping from labels (strings) to indice in the logit layer (int) in a dictionary format.
python
def id_to_label(self, idx)
def label_to_id(self, label)
Set your input image shape at ...
Can you write Python code for it?

# Create TensorFlow session.
sess = utils.create_session()

# GRAPH_PATH is where the trained model is stored.
GRAPH_PATH = source_dir + "/inception5h/tensorflow_inception_graph.pb"
# LABEL_PATH is where the labels are stored. Each line contains one class, and they are ordered with respect to their index in
# the logit layer. (yes, id_to_label function in the model wrapper reads from this file.)
# For example, imagenet_comp_graph_label_strings.txt looks like:
# dummy
# kit fox
# English setter
# Siberian husky ...

LABEL_PATH = source_dir + "/inception5h/imagenet_comp_graph_label_strings.txt"

mymodel = model.GoogleNetWrapper_public(sess,
                                        GRAPH_PATH,
                                        LABEL_PATH)


I am working on the file "Run_TCAV.ipynb".
The first task is:
Step 3: Implement a class that returns activations (maybe with caching!)
Lastly, you will implement a class of the ActivationGenerationInterface which TCAV uses to load example data for a given concept or target, call into your model wrapper and return activations. I pulled out this logic outside of mymodel because this step often takes the longest. By making it modular, you can cache your activations and/or parallelize your computations, as I have done in ActivationGeneratorBase.process_and_load_activations in activation_generator.py.
The process_and_load_activations method of the activation generator must return a dictionary of activations that has concept or target name as  a first key, and the bottleneck name as a second key. So something like:
python
{concept1: {bottleneck1: [[0.2, 0.1, ....]]},
concept2: {bottleneck1: [[0.1, 0.02, ....]]},
target1: {bottleneck1: [[0.02, 0.99, ....]]}
Can you write Python code for it?

act_generator = act_gen.ImageActivationGenerator(mymodel, source_dir, activation_dir, max_examples=100)


I am working on the file "Run_TCAV.ipynb".
The first task is:
Step 4: Run TCAV and visualize concept importance
You are now ready to run TCAV! Let's do it.
num_random_exp: number of experiments to confirm meaningful concept direction. TCAV will search for this many folders named random500_0, random500_1, etc. You can alternatively set the random_concepts keyword to be a list of folders of random concepts. Run at least 10-20 for meaningful tests.
random_counterpart: as well as the above, you can optionally supply a single folder with random images as the "positive set" for statistical testing. Reduces computation time at the cost of less reliable random TCAV scores.
Can you write Python code for it?

import absl
absl.logging.set_verbosity(0)
num_random_exp=3
## only running num_random_exp = 10 to save some time. The paper number are reported for 500 random runs.
mytcav = tcav.TCAV(sess,
                   target,
                   concepts,
                   bottlenecks,
                   act_generator,
                   alphas,
                   cav_dir=cav_dir,
                   num_random_exp=num_random_exp)#10)
print ('This may take a while... Go get coffee!')
results = mytcav.run(run_parallel=False)
print ('done!')

utils_plot.plot_results(results, num_random_exp=num_random_exp)


I am working on the file "Run_TCAV.ipynb".
The first task is:
TCAV step-by-step
You will walk through the following steps below:

Store example images in each folder (you have this if you ran the above)
images for each concept
images for the class/labels of interest
random images that will be negative examples when learning CAVs (images that probably don't belong to any concepts)
Write a model wrapper (below uses example from tcav/model.py)
an instance of  ModelWrapper abstract class (in model.py). This tells TCAV class (tcav.py) how to communicate with your model (e.g., getting internal tensors)
Retrieve model activations (below uses example from tcav/activation_generator.py)
an instance of ActivationGeneratorInterface that tells TCAV class how to load example data and how to get activations from the model
Run TCAV and visualize scores for important concepts.

Step 1: Store concept and target class images to local folders
... and tell TCAV where they are.
source_dir: where images of concepts, target class and random images (negative samples when learning CAVs) live. Each should be a sub-folder within this directory.
Note that random image directories can be in any name. In this example, we are using random500_0, random500_1,.. for an arbitrary reason.
You need roughly 50-200 images per concept and target class (10-20 pictures also tend to work, but 200 is pretty safe).
cav_dir: directory to store CAVs (None if you don't want to store)
target, concept: names of the target class (that you want to investigate) and concepts (strings) - these are folder names in source_dir
bottlenecks: list of bottleneck names (intermediate layers in your model) that you want to use for TCAV. These names are defined in the model wrapper below.
Can you write Python code for it?

print ('REMEMBER TO UPDATE YOUR_PATH (where images, models are)!')

# This is the name of your model wrapper (InceptionV3 and GoogleNet are provided in model.py)
model_to_run = 'GoogleNet'
user = 'beenkim'
# the name of the parent directory that results are stored (only if you want to cache)
project_name = 'tcav_class_test'
working_dir = "/tmp/" + user + '/' + project_name
# where activations are stored (only if your act_gen_wrapper does so)
activation_dir =  working_dir+ '/activations/'
# where CAVs are stored.
# You can say None if you don't wish to store any.
cav_dir = working_dir + '/cavs/'
# where the images live.

# TODO: replace 'YOUR_PATH' with path to downloaded models and images.
source_dir = ''
bottlenecks = [ 'mixed4c']  # @param

utils.make_dir_if_not_exists(activation_dir)
utils.make_dir_if_not_exists(working_dir)
utils.make_dir_if_not_exists(cav_dir)

# this is a regularizer penalty parameter for linear classifier to get CAVs.
alphas = [0.1]

target = 'zebra'
concepts = ["dotted","striped","zigzagged"]


I am working on the file "Run_TCAV_on_colab.ipynb".
The first task is:
Running TCAV
This notebook walks you through things you need to run TCAV.
Before running this notebook, run the following to download all the data.
```
cd tcav/tcav_examples/image_models/imagenet
python download_and_make_datasets.py --source_dir=YOUR_PATH --number_of_images_per_folder=50 --number_of_random_folders=3
```
In high level, you need:

example images in each folder (you have this if you ran the above)
images for each concept
images for the class/labels of interest
random images that will be negative examples when learning CAVs (images that probably don't belong to any concepts)
model wrapper (below uses example from tcav/model.py)
an instance of  ModelWrapper abstract class (in model.py). This tells TCAV class (tcav.py) how to communicate with your model (e.g., getting internal tensors)
act_generator (below uses example from tcav/activation_generator.py)
an instance of ActivationGeneratorInterface that tells TCAV class how to load example data and how to get activations from the model

Requirements
pip install the tcav and tensorflow packages (or tensorflow-gpu if using GPU)
Can you write Python code for it?


I am working on the file "Run_TCAV_on_colab.ipynb".
The first task is:
Step 2. Write your model wrapper
Next step is to tell TCAV how to communicate with your model. See model.GoogleNetWrapper_public for details.
You can define a subclass of ModelWrapper abstract class to do this. Let me walk you thru what each function does (tho they are pretty self-explanatory).  This wrapper includes a lot of the functions that you already have, for example, get_prediction.
1. Tensors from the graph: bottleneck tensors and ends
First, store your bottleneck tensors in self.bottlenecks_tensors as a dictionary. You only need bottlenecks that you are interested in running TCAV with. Similarly, fill in self.ends dictionary with input, logit and prediction tensors.
2. Define loss
Get your loss tensor, and assigned it to self.loss. This is what TCAV uses to take directional derivatives.
While doing so, you would also want to set
python
self.y_input
this simply is a tensorflow place holder for the target index in the logit layer (e.g., 0 index for a dog, 1 for a cat).
For multi-class classification, typically something like this works:
python
self.y_input = tf.placeholder(tf.int64, shape=[None])
For example, for a multiclass classifier, something like below would work.
```python
    # Construct gradient ops.
    with g.as_default():
      self.y_input = tf.placeholder(tf.int64, shape=[None])
  self.pred = tf.expand_dims(self.ends['prediction'][0], 0)

  self.loss = tf.reduce_mean(
      tf.nn.softmax_cross_entropy_with_logits(
          labels=tf.one_hot(self.y_input, len(self.labels)),
          logits=self.pred))
self._make_gradient_tensors()

```
3. Call _make_gradient_tensors in init() of your wrapper
python
_make_gradient_tensors()
does what you expect - given the loss and bottleneck tensors defined above, it adds gradient tensors.
4. Fill in labels, image shapes and a model name.
Get the mapping from labels (strings) to indice in the logit layer (int) in a dictionary format.
python
def id_to_label(self, idx)
def label_to_id(self, label)
Set your input image shape at  self.image_shape
Set...
Can you write Python code for it?

%cp -av '/content/tcav/tcav/tcav_examples/image_models/imagenet/YOUR_FOLDER/mobilenet_v2_1.0_224' '/content/tcav/tcav/mobilenet_v2_1.0_224'
%rm '/content/tcav/tcav/tcav_examples/image_models/imagenet/YOUR_FOLDER/mobilenet_v2_1.0_224'

%cp -av '/content/tcav/tcav/tcav_examples/image_models/imagenet/YOUR_FOLDER/inception5h' '/content/tcav/tcav/inception5h'
%rm '/content/tcav/tcav/tcav_examples/image_models/imagenet/YOUR_FOLDER/inception5h'

sess = utils.create_session()

# GRAPH_PATH is where the trained model is stored.
GRAPH_PATH = "/content/tcav/tcav/inception5h/tensorflow_inception_graph.pb"
# LABEL_PATH is where the labels are stored. Each line contains one class, and they are ordered with respect to their index in
# the logit layer. (yes, id_to_label function in the model wrapper reads from this file.)
# For example, imagenet_comp_graph_label_strings.txt looks like:
# dummy
# kit fox
# English setter
# Siberian husky ...

LABEL_PATH = "/content/tcav/tcav/inception5h/imagenet_comp_graph_label_strings.txt"

mymodel = model.GoogleNetWrapper_public(sess,
                                        GRAPH_PATH,
                                        LABEL_PATH)


I am working on the file "Run_TCAV_on_colab.ipynb".
The first task is:
You are ready to run TCAV!
Let's do it.
num_random_exp: number of experiments to confirm meaningful concept direction. TCAV will search for this many folders named random500_0, random500_1, etc. You can alternatively set the random_concepts keyword to be a list of folders of random concepts. Run at least 10-20 for meaningful tests.
random_counterpart: as well as the above, you can optionally supply a single folder with random images as the "positive set" for statistical testing. Reduces computation time at the cost of less reliable random TCAV scores.
Can you write Python code for it?

import absl
absl.logging.set_verbosity(0)
num_random_exp=10
## only running num_random_exp = 10 to save some time. The paper number are reported for 500 random runs.
mytcav = tcav.TCAV(sess,
                   target,
                   concepts,
                   bottlenecks,
                   act_generator,
                   alphas,
                   cav_dir=cav_dir,
                   num_random_exp=num_random_exp)#10)
print ('This may take a while... Go get coffee!')
results = mytcav.run(run_parallel=False)
print ('done!')

utils_plot.plot_results(results, num_random_exp=num_random_exp)


I am working on the file "Runge-Kutta_methods_for_ODE_integration_in_Python.ipynb".
The first task is:
Both versions compute the same thing.
Can you write Python code for it?

t2 = np.linspace(0, 10, 1001)
sol2 = rungekutta4(pend, y0, t2, args=(b, c))
sol2_jit = rungekutta4_jit(pend, y0, t2, args=(b, c))
np.linalg.norm(sol2 - sol2_jit)


I am working on the file "Runge-Kutta_methods_for_ODE_integration_in_Python.ipynb".
The first task is:
Comparisons on another integration problem
Consider the following ODE on $t\in[0, 1]$:
$$
\begin{cases}
 y'''(t) = 12 y(t)^{4/5} + \cos(y'(t))^3 - \sin(y''(t)) \
 y(0) = 0, y'(0) = 1, y''(0) = 0.1
\end{cases}
$$
It can be written in a vectorial form like the first one:
Can you write Python code for it?

def f(y, t):
    return np.array([y[1], y[2], 12 * y[0] ** (4/5.) + np.cos(y[1])**3 - np.sin(y[2])])

def test_2(n=101):
    t = np.linspace(0, 1, n)
    y0 = np.array([0, 1, 0.1])
    for method, m in zip(methods, markers):
        sol = method(f, y0, t)
        plt.plot(t, sol[:, 0], label=method.__name__, marker=m)
    plt.legend(loc='best')
    plt.title("Comparison of different ODE integration methods for $n={}$ points".format(n))
    plt.xlabel("$t = [0, 1]$")
    plt.grid()
    plt.show()

test_2(10)

test_2(50)


I am working on the file "Runge-Kutta_methods_for_ODE_integration_in_Python.ipynb".
The first task is:
For our simple ODE example, this method is already quite efficient.
Can you write Python code for it?

t4 = np.linspace(0, 10, 21)
sol4 = rungekutta2(pend, y0, t4, args=(b, c))

t = np.linspace(0, 10, 101)
sol = rungekutta2(pend, y0, t, args=(b, c))

t2 = np.linspace(0, 10, 1001)
sol2 = rungekutta2(pend, y0, t2, args=(b, c))

t3 = np.linspace(0, 10, 10001)
sol3 = rungekutta2(pend, y0, t3, args=(b, c))

plt.plot(t4, sol4[:, 0], label='with 11 points')
plt.plot(t, sol[:, 0], label='with 101 points')
plt.plot(t2, sol2[:, 0], label='with 1001 points')
plt.plot(t3, sol3[:, 0], label='with 10001 points')
plt.legend(loc='best')
plt.xlabel('t')
plt.grid()
plt.show()


I am working on the file "Runge-Kutta_methods_for_ODE_integration_in_Python.ipynb".
The first task is:
For our simple ODE example, this method is even more efficient.
Can you write Python code for it?

t4 = np.linspace(0, 10, 21)
sol4 = rungekutta4(pend, y0, t4, args=(b, c))

t = np.linspace(0, 10, 101)
sol = rungekutta4(pend, y0, t, args=(b, c))

t2 = np.linspace(0, 10, 1001)
sol2 = rungekutta4(pend, y0, t2, args=(b, c))

plt.plot(t4, sol4[:, 0], label='with 21 points')
plt.plot(t, sol[:, 0], label='with 101 points')
plt.plot(t2, sol2[:, 0], label='with 1001 points')
plt.legend(loc='best')
plt.xlabel('t')
plt.grid()
plt.show()


I am working on the file "Runge-Kutta_methods_for_ODE_integration_in_Python.ipynb".
The first task is:
I will use as a first example the one included in the scipy documentation for this odeint function.
$$\theta''(t) + b \theta'(t) + c \sin(\theta(t)) = 0.$$
If $\omega(t) = \theta'(t)$, this gives
$$ \begin{cases}
\theta'(t) = \omega(t) \
\omega'(t) = -b \omega(t) - c \sin(\theta(t))
\end{cases} $$
Vectorially, if $y(t) = [\theta(t), \omega(t)]$, then the equation is $y' = f(t, y)$ where $f(t, y) = [y_2(t), -b y_2(t) - c \sin(y_1(t))]$.
Can you write Python code for it?

def pend(y, t, b, c):
    return np.array([y[1], -b*y[1] - c*np.sin(y[0])])


I am working on the file "Runge-Kutta_methods_for_ODE_integration_in_Python.ipynb".
The first task is:
It is used like this, and our implementations will follow this signature.
Can you write Python code for it?

sol = odeint(pend, y0, t, args=(b, c))

plt.plot(t, sol[:, 0], 'b', label=r'$\theta(t)$')
plt.plot(t, sol[:, 1], 'g', label=r'$\omega(t)$')
plt.legend(loc='best')
plt.xlabel('t')
plt.grid()
plt.show()


I am working on the file "Runge-Kutta_methods_for_ODE_integration_in_Python.ipynb".
The first task is:
Our hand-written Runge-Kutta method of order 4 seems to be as efficient as the odeint method from scipy... and that's because odeint basically uses a Runge-Kutta method of order 4 (with smart variants).
Small benchmark
We can also compare their speed:
Can you write Python code for it?

methods = [odeint, rungekutta1, rungekutta2, rungekutta4, rungekutta4_jit]

y0 = np.array([10, -3, 1, 1])
for n in [20, 100, 1000]:
    print("\n")
    t = np.linspace(0, 3, n)
    for method in methods:
        print("Time of solving this ODE for {} points with {} method...".format(n, method.__name__))
        %timeit sol = method(f, y0, t)


I am working on the file "Runge-Kutta_methods_for_ODE_integration_in_Python.ipynb".
The first task is:
Runge-Kutta method of order 1, or the Euler method
The approximation is computed using this update:
$$y_{n+1} = y_n + (t_{n+1} - t_n) f(y_n, t_n).$$
The math behind this formula are the following: if $g$ is a solution to the ODE, and so far the approximation is correct, $y_n \simeq g(t_n)$, then a small step $h = t_{n+1} - t_n$ satisfy $g(t_n + h) \simeq g(t_n) + h g'(t_n) \simeq y_n + h f(g(t_n), t_n) + \simeq y_n + h f(y_n, t_n)$.
Can you write Python code for it?

def rungekutta1(f, y0, t, args=()):
    n = len(t)
    y = np.zeros((n, len(y0)))
    y[0] = y0
    for i in range(n - 1):
        y[i+1] = y[i] + (t[i+1] - t[i]) * f(y[i], t[i], *args)
    return y

sol = rungekutta1(pend, y0, t, args=(b, c))

plt.plot(t, sol[:, 0], 'b', label=r'$\theta(t)$')
plt.plot(t, sol[:, 1], 'g', label=r'$\omega(t)$')
plt.legend(loc='best')
plt.xlabel('t')
plt.grid()
plt.show()


I am working on the file "Runge-Kutta_methods_for_ODE_integration_in_Python.ipynb".
The first task is:
Runge-Kutta method of order 2
The order 2 Runge-Method uses this update:
$$ y_{n+1} = y_n + h f(t + \frac{h}{2}, y_n + \frac{h}{2} f(t, y_n)),$$
if $h = t_{n+1} - t_n$.
Can you write Python code for it?

def rungekutta2(f, y0, t, args=()):
    n = len(t)
    y = np.zeros((n, len(y0)))
    y[0] = y0
    for i in range(n - 1):
        h = t[i+1] - t[i]
        y[i+1] = y[i] + h * f(y[i] + f(y[i], t[i], *args) * h / 2., t[i] + h / 2., *args)
    return y


I am working on the file "Runge-Kutta_methods_for_ODE_integration_in_Python.ipynb".
The first task is:
Runge-Kutta method of order 4, "RK4"
The order 4 Runge-Method uses this update:
$$ y_{n+1} = y_n + \frac{h}{6} (k_1 + 2 k_2 + 2 k_3 + k_4),$$
if $h = t_{n+1} - t_n$, and
$$\begin{cases}
k_1 &= f(y_n, t_n), \
k_2 &= f(y_n + \frac{h}{2} k_1, t_n + \frac{h}{2}), \
k_3 &= f(y_n + \frac{h}{2} k_2, t_n + \frac{h}{2}), \
k_4 &= f(y_n + h k_3, t_n + h).
\end{cases}$$
Can you write Python code for it?


I am working on the file "Runge-Kutta_methods_for_ODE_integration_in_Python.ipynb".
The first task is:
The odeint function will be used to solve this ODE on the interval $t \in [0, 10]$, with $101$ points.
Can you write Python code for it?


I am working on the file "Runge-Kutta_methods_for_ODE_integration_in_Python.ipynb".
The first task is:
With the same number of points, the Euler method (i.e. the Runge-Kutta method of order 1) is less precise than the reference odeint method. With more points, it can give a satisfactory approximation of the solution:
Can you write Python code for it?

t2 = np.linspace(0, 10, 1001)
sol2 = rungekutta1(pend, y0, t2, args=(b, c))

t3 = np.linspace(0, 10, 10001)
sol3 = rungekutta1(pend, y0, t3, args=(b, c))

plt.plot(t, sol[:, 0], label=r'$\theta(t)$ with 101 points')
plt.plot(t2, sol2[:, 0], label=r'$\theta(t)$ with 1001 points')
plt.plot(t3, sol3[:, 0], label=r'$\theta(t)$ with 10001 points')
plt.legend(loc='best')
plt.xlabel('t')
plt.grid()
plt.show()


I am working on the file "RunningFeatures/Gc.ipynb".
The first task is:
Collection Thresholds and Generations
Can you write Python code for it?


I am working on the file "RushHourPy/analysis_prototypes/solution_path_nodes_prototype.ipynb".
The first task is:
Develop data structures to support drawing the connected component.
Can you write Python code for it?

def partition_color(depth):
    return depth
    #if depth == 0:
    #    return 'r'
    #else:
    #    return depth

    #if depth >0  and depth < 9:
    #    return 'g'
    #if depth == 9:
    #    return 'b'

def nodes_colors(dist_partition): #dist_partition is dict k:v, K is partition depth (0,...,n); v is list of game numbers
    partition_list = [[k,elt] for k in dist_partition.keys() for elt in dist_partition[k]]

    nodes = [x[1] for x in partition_list]
    colors = [partition_color(x[0]) for x in partition_list]

    return [nodes,colors]

[graph_nodes,graph_colors] = nodes_colors(dist_partition)
node_labels = {node:str(depth) for (depth,nodes) in dist_partition.items() for node in nodes}
shell_list = [list(dist_partition[k]) for k in dist_partition.keys() ]
#nodes,node_labels

#nx.draw(G)
%pylab qt
G=nx.Graph()
G.add_nodes_from(graph_nodes)
G.add_edges_from([ [x[0],x[1]] for x in component_edges])
nx.draw_shell(G,nlist=shell_list,scale=23,nodelist=graph_nodes, node_color=graph_colors, with_labels=True,labels=node_labels,font_size=24,node_size=300)


I am working on the file "RushHourPy/analysis_prototypes/solution_path_nodes_prototype.ipynb".
The first task is:
Saving Updates to the DB
Can you write Python code for it?

comb_class_id = 36
update_sql = "update game_state set optimal_transition_game_number = ?, solution_distance = ? where comb_class_id = ? and game_number = ?"
update_data = [ (v['optimal_nbr_game_number'], v['depth'],comb_class_id, k) for k,v in black_states.items() ]
c.executemany(update_sql,update_data)
con.commit()

%run compute_edges_connected_components

set_solution_data_for_comb_class(36)
#res = solution_data_for_connected_component(20)
#res

#conn = db.connect('C:/Users/cliff/workspace/rushHour/Data Model and SQL/Schema Definition and Bulk Load/rush_hour.db')
#cursor = conn.cursor()

sql = 'select distinct connected_component_id from game_state where comb_class_id = %d' %(comb_class_id)
c.execute(sql)
component_ids = c.fetchall()
component_ids


I am working on the file "RushHourPy/analysis_prototypes/solution_path_nodes_prototype.ipynb".
The first task is:
The algorithm is to follow breadth-first search but from a starting set rather than a single node.
Can you write Python code for it?

"""
#gray_states = collections.OrderedDict()

# test code in preparation for work in next cell
adj_list = {x[1]:[] for x in component_states} # component_state = ()
for [x,y,z]in component_edges: # [t1,t2,...tn] ti = (pre_tran_state_game_number, post_tran_state_game_number, comb_class_id):
    adj_list[x].append(y)
    adj_list[y].append(x)

# component_states: (comb class id, game number, is Goal State?)
# states as an array of dictionaries: [d1,d2,d3,...dn] di = {game_number:{'depth':,'optimal_nbr_game_number}}

white_states = { x[1]:{'depth':None, 'optimal_nbr_game_number':None} for x in component_states if x[2] !=1}
gray_states =  collections.OrderedDict( { x[1]:{'depth':0, 'optimal_nbr_game_number':None} for x in component_states if x[2] == 1 })
black_states = {}

len(gray_states), len(white_states),set([x for k in adj_list.keys() for x in adj_list[k] if x not in white_states])
len(set([k for k in white_states.keys()]))
#cur_state_key,cur_state_values = gray_states.popitem(last=False)
#nbr = (adj_list[cur_state_key])[0]
#cur_state_key,nbr, [x for x in component_states if x[1] == nbr],nbr in gray_states,nbr in white_states, gray_states
"""

adj_list = {x[1]:[] for x in component_states} # component_state = ()
for [x,y,z]in component_edges: # [t1,t2,...tn] ti = (pre_tran_state_game_number, post_tran_state_game_number, comb_class_id):
    adj_list[x].append(y)
    adj_list[y].append(x)

# component_states: (comb class id, game number, is Goal State?)
# states as an array of dictionaries: [d1,d2,d3,...dn] di = {game_number:{'depth':,'optimal_nbr_game_number}}

white_states = { x[1]:{'depth':None, 'optimal_nbr_game_number':None} for x in component_states if x[2] !=1}
gray_states =  collections.OrderedDict( { x[1]:{'depth':0, 'optimal_nbr_game_number':None} for x in component_states if x[2] == 1 })
black_states = {}

while gray_states:
    cur_state_key,cur_state_values = gray_states.popitem(last=False)
    for nbr in adj_list[cur_state_key]:
        if nbr in white_states:
           ...


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
A continuación se resume la característica del dataset:

Tensor 3D
Tipos de datos: enteros de 8-bits.

Visualicemos una imágen:
Can you write Python code for it?

digit = train_images[9]
import matplotlib.pyplot as plt
plt.imshow(digit, cmap=plt.cm.binary)
plt.show()


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Ahora dividamos el conjunto de datos de entrenamiento en un conjunto de validación y un conjunto de entrenamiento (más pequeño). También escalamos las intensidades de píxeles en el rango 0-1 y las convertimos en flotantes al dividiéndolas por 255.
Can you write Python code for it?

X_valid, X_train = X_train_full[:5000] / 255., X_train_full[5000:] / 255.
y_valid, y_train = y_train_full[:5000], y_train_full[5000:]
X_test = X_test / 255


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Como puede ver, para cada caso, el modelo estima una probabilidad por clase, desde la clase 0 hasta la clase 9. Si solo le importa la clase con la probabilidad estimada más alta (incluso si esa probabilidad es bastante baja), puede usar el método pre dict_classes() en su lugar
Can you write Python code for it?

y_pred = model.predict_classes(X_new)
y_pred

np.array(class_names)[y_pred]

y_new = y_test[:3]
y_new

plt.figure(figsize=(7.2, 2.4))
for index, image in enumerate(X_new):
    plt.subplot(1, 3, index + 1)
    plt.imshow(image, cmap="binary", interpolation="nearest")
    plt.axis('off')
    plt.title(class_names[y_test[index]], fontsize=12)


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Compilando el modelo
Después de crear un modelo, se debe llamar el método compile() para especificar la función de pérdida y el optimizador a utilizar. Opcionalmente, también puede especificar una lista de métricas  para calcular durante el entrenamiento y la evaluación:
Can you write Python code for it?

model.compile(loss='sparse_categorical_crossentropy',
              optimizer='sgd',
              metrics=["accuracy"])


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Dentro de las clases podemos definir funciones, conocidas como métodos
Can you write Python code for it?

class Persona():
  nombre = ''
  edad = ''

  def print_nombre(self):
    print('nombre:',self.nombre)

  def print_edad(self):
    print('edad:',self.edad)

Person1 = Persona()
Person1.nombre = 'carolina'
Person1.edad = 20

Person1.print_nombre()
Person1.print_edad()


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Ejemplo: clasificador de imágenes usando Keras
Para ilustrar la manera en que Keras es usado en el aprendizaje profundo, realizaremos un ejemplo de clasificación usando conjunto de datos Fashion MNIST el cual consta de 70,000 imágenes (las imágenes representan artículos de moda) en escala de grises de 28 × 28 píxeles cada una y con 10 clases.
Uso de Keras para cargar el conjunto de datos
Comencemos cargando el conjunto de datos Fashion MNIST. Keras tiene una serie de funciones para cargar conjuntos de datos populares en keras.datasets. El conjunto de datos ya está dividido entre un conjunto de entrenamiento y un conjunto de prueba, pero puede ser útil dividir aún más el conjunto de entrenamiento para tener un conjunto de validación:
Can you write Python code for it?

from keras.datasets import fashion_mnist

fashion_mnist = keras.datasets.fashion_mnist
(X_train_full, y_train_full), (X_test, y_test) = fashion_mnist.load_data()


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
El conjunto de entrenamiento contiene 60,000 imágenes en escala de grises, cada una de 28x28 píxeles:
Can you write Python code for it?


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Entrenando y evaluando el modelo
Ahora el modelo está listo para ser entrenado. Para esto simplemente necesitamos llamar a su método fit(). Le pasamos las características de entrada (X_train) y las clases de destino (y_train), así como la cantidad de épocas para entrenar (o de lo contrario, sería predeterminado a solo 1). También pasamos un conjunto de validación (esto es opcional): Keras medirá la pérdida y las métricas adicionales en este conjunto al final de cada época, lo cual es muy útil para ver qué tan bien se desempeña realmente el modelo.
Can you write Python code for it?

history =  model.fit(X_train, y_train, epochs=20, validation_data=(X_valid, y_valid))


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Haga click aquí si tiene problemas con la solución.
<!----:
# solución:
my_slice_corner = train_images[:, 14:, 14:]

digit = my_slice_corner[3]
import matplotlib.pyplot as plt
plt.imshow(digit, cmap=plt.cm.binary)
plt.show()

----->

Recordemos que también es posible usar índices negativos. Los índices negativos indican una posición relativa al final del eje en cuestión. Recorte, por ejemplo, las imagenes anteriores para que queden centradas en matríces de 14x14.
Haga click aquí si tiene problemas con la solución:
<!-----
my_crop_slices_centered = train_images[:, 7:-7, 7:-7]

digit = my_crop_slices_centered[0]
import matplotlib.pyplot as plt
plt.imshow(digit, cmap=plt.cm.binary)
plt.show()
----->

Noción de batches
En general, el primer eje (eje 0, ya que la indexación comienza en 0) de todos los  tensores de datos que encuentre en DLserá el eje de muestras (a veces denominado dimensión de muestras). En el ejemplo MNIST, las muestras son imágenes de dígitos.
Además, los modelos de DL no procesan un conjunto de datos completo a la vez; en su lugar, dividen los datos en pequeños lotes. Por ejemplo:
Can you write Python code for it?

batch1 = train_images[:128]
batch2 = train_images[128:256]


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Haga click en esta celda si tiene problemas con la solución.

Clases/Objetos
Python es un lenguaje orientado a objetos. Casi todo en Python representa un objeto, el cual esta constituidos por sus atributos y métodos.
Una clase es un "ente creador", o un prototipo para la creación de objetos.
Las clases proporcionan un medio de agrupar datos y "funcionalidad". La creación de una nueva clase crea un nuevo tipo de objeto, lo que permite crear nuevas instancias de ese tipo. Cada instancia de la clase puede tener sus propios atributos. Las instancias de una clase también pueden tener métodos (definidos por su clase) para modificar su estado.

class Name():

sentencias

Los atributos son como propiedades que queremos añadir a la clase
Can you write Python code for it?


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
La intensidad de cada píxel se representa como un byte (enteros entre 0 a 255):
Can you write Python code for it?


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
La operación de creación de instancias crea un objeto vacío. A muchas clases les gusta crear objetos con instancias personalizadas a un estado inicial específico. Por lo tanto, una clase puede definir un método especial llamado init  (), como este:
Can you write Python code for it?

class Person:
  def __init__(self,n,e):
    self.nombre = n
    self.edad = e
  def __str__(self):
    return self.nombre

Person3 = Person('mario','55')
print('nombre:',Person3.nombre)
print('edad:',Person3.edad)
print(Person3)


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Las etiquetas van de 0 a 9, por lo que hay 10 tipos diferentes de ropa.
Can you write Python code for it?


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Manipulación de tensores en Numpy:
En el ejeumplo anterior se selecionó un digito específico a junto al primer eje usando la sintaxis train_images [i]. La selección de elementos específicos en un tensor se denomina corte de "tensor slicing". Veamos las operaciones de "tensor slicing" que puede hacerse con los arreglos de numpy.
Para selecionar los digitos de 10 a 100 (si incluir el 100) hacemos:
Can you write Python code for it?

my_slice = train_images[10:100]
print(my_slice.shape)


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Uso de Keras para crean un modelo de aprendizaje profundo
Can you write Python code for it?

keras.backend.clear_session()

model = keras.models.Sequential();
model.add(keras.layers.Flatten(input_shape=[28, 28]));
model.add(keras.layers.Dense(300, activation="relu"));
model.add(keras.layers.Dense(100, activation="relu"));
model.add(keras.layers.Dense(10, activation="softmax"));


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Veamos este código línea por línea:
* La primera línea crea un modelo secuencial. Este es el tipo más simple de modelo para redes neuronales que están compuestas de una sola pila de capas, conectadas secuencialmente. Esto se llama la API secuencial.

Luego , construimos la primera capa y la agregamos al modelo. Es una capa Flatten cuyo papel es simplemente convertir cada imagen de entrada en un arregle 1D:

Luego, agregamos una capa oculta densa con 300 neuronas la cual utilizará la función de activación ReLU.

Luego, agregamos una segunda capa oculta densa con 100 neuronas, que también utiliza la función de activación ReLU.

Finalmente, agregamos una capa de salida densa con 10 neuronas (una por clase), usando la función de activación softmax (porque las clases son exclusivas).

El método summary() muestra un resumen de todas las capas del modelo.
Can you write Python code for it?


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Veamos un ejemplo concreto de una base de datos MNIST de keras:
Can you write Python code for it?

from keras.datasets import mnist
(train_images, train_labels), (test_images, test_labels) = mnist.load_data()

print(train_images.ndim)

print(train_images.shape)

print(train_images.dtype)


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
Vectores (Tensores 1D)
A un array de número se le llama vector, o tensor 1D. Decimo entonces que un tensor 1D tiene exactamente 1 eje.
En numpy lo definimos como:
Can you write Python code for it?


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
los nombres para cada una de las clases son:
Can you write Python code for it?

class_names = ["T-shirt/top", "Trouser", "Pullover", "Dress", "Coat",
               "Sandal", "Shirt", "Sneaker", "Bag", "Ankle boot"]


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
visualicemos ahora algunas de las imágenes de los datos de entrenamiento con su respectiva etiqueta
Can you write Python code for it?

fig , ax =  plt.subplots(3,10, figsize=(15,5))
for i , ax in enumerate(ax.flat):
  ax.imshow(X_train[i], cmap='binary')
  ax.set_axis_off()
  ax.set_title(class_names[y_train[i]])


I am working on the file "S12_Keras_intro_y_analisis_tensorial.ipynb".
The first task is:
¡Y eso es todo! La red neuronal está entrenada. En cada época durante el entrenamiento, Keras muestra el número de instancias procesadas hasta el momento (junto con una barra de progreso), el tiempo medio de entrenamiento, la pérdida (loss) y la precisión (accuracy), ambas calculadas en el conjunto de entrenamiento y el conjunto de validación.
El método fit() devuelve un objeto "History" que contiene los parámetros de entrenamiento (history.params), la lista de épocas por las que pasó (history.epoch) y, lo más importante, un diccionario (history.history) que contiene la pérdida y las métricas adicionales que midió al final de cada época en el conjunto de entrenamiento y en el conjunto de validación (si corresponde).
usando la librería Pandas y matplotlib podemos visualizar las curvas de aprendizaje
Can you write Python code for it?

import pandas as pd

df=pd.DataFrame(history.history)
df.head()

df.plot(figsize=(8, 5))


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
Default settings
Can you write Python code for it?

# Digitizer sample size
dt = 2

# Waveform time labels
spe_ts = np.linspace(0, 639*2, 640) - 340 * 2
# Valid time (because the waveform does not range the full time span)
valid_t_range = (-100, 300)
t_mask = (valid_t_range[0] <= spe_ts) & (spe_ts < valid_t_range[1])
spe_ts = spe_ts[t_mask]
spe_t_edges = np.concatenate([[spe_ts[0] - dt/2], spe_ts + dt/2])

default_params = dict(
    t1 = 3.1,    # Singlet lifetime, Nest 2014 p2
    t3 = 24,     # Triplet lifetime, Nest 2014 p2
    fs = 0.2,    # Singlet fraction
    tts = 2.,     # Transit time spread.
    s1_min=50,
    s1_max=100,
    dset='er',
    pulse_model=1, # This is the CHANNEL that is used...
    n_photons = int(2e5),
    t_min = -15.,
    t_max = 125.,
    s1_sample = 'data', # 'uniform'
    error_offset  = 0. ,
    error_pct = 0.

)

def get_params(params):
    '''
    Returns full set of parameters, setting the values given in `params` and setting the values in
    `default_params` if not set explicity.
    '''
    for k, v in default_params.items(): # key, value
        params.setdefault(k, v)
    if params['tts'] < 0:
        params['tts'] = 1e-6
    return params


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
Fit four parameters
ER
Can you write Python code for it?

from scipy import optimize
optresult = optimize.minimize(
    lambda x: gof(fs=x[0], t1=x[1], t3=x[2], tts=x[3], s1_min=30, s1_max = 100, dst='er'),
    [0.2, 3.1, 24, 3],
    bounds=[[.01, 1], [.1, 5], [20, 30], [.1, 5]],
    options=dict(maxfev=1000),
    method='Powell',
)

# fit = optresult.x
# ydata, _ = real_s1_wv()
# minus, base, plus = s1_models_error(fs=fit[0], t1=fit[1], t3=fit[2], tts=fit[3])
# comparison_plot(ydata, minus, base, plus)

fit = optresult.x
print(fit)
ydata, _ = real_s1_wv()
minus, base, plus = s1_models_error(fs=fit[0], t1=fit[1], t3=fit[2], tts=fit[3], s1_min=30, s1_max = 100)
comparison_plot(ydata, minus, base, plus)
plt.show()
comparison_plot_2(ydata, minus, base, plus)
for _l in (-20, 100):
    plt.axvline(_l)
plt.xlim(-50, 200)
plt.show()


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
Fit singlet fraction and TTS
Can you write Python code for it?

from scipy import optimize
optresult = optimize.minimize(
    lambda x: gof(fs=x[0], tts=x[1], s1_min=30, s1_max = 100, error_pct = 1e-2, error_offset = 1e-4, mode='chi2_ndf'),
    [0.2, 2],
    bounds=[[.01, 1], [20, 30], [.1, 5]],
    options=dict(maxfev=1000),
    method='Powell',
)
print('Done')

optresult

fit = optresult.x
print(fit)
ydata, _ = real_s1_wv()
minus, base, plus = s1_models_error(fs=fit[0], tts=fit[1], s1_min = 30, s1_max = 100,
                                    error_pct = 1e-2, error_offset = 1e-4)
comparison_plot(ydata, minus, base, plus, error_pct = 1e-2, error_offset = 1e-4)
plt.show()
comparison_plot_2(ydata, minus, base, plus, error_pct = 1e-2, error_offset = 1e-4)
plt.show()


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
Fitting
Residuals function
Can you write Python code for it?

def gof(verbose=True, mode = 'chi2_ndf', **params):
    '''
    Get the mean residuals for given model parameters.

    '''
    params = get_params(params)

    # Do not allow unphysical values
    if params['t1'] < 0 or params['t3'] < 0 or not (0 <= params['fs'] <= 1):
        result = float('inf')
    else:
        ydata, _ = real_s1_wv(**params)
        # By default, the errors are set to: [0,1] for pulse model, 1.0 for tts
        minus, base, plus = s1_models_error(**params)
        res = residuals(ydata, minus, base, plus, **params)
        assert len(res) == len(spe_ts)
        res = res[(spe_ts >= params['t_min']) & (spe_ts < params['t_max'])]
        if mode == 'mean':
            result = np.abs(res).mean()
        elif mode == 'median':
            result = np.median(np.abs(res))
        elif mode == 'chi2':
            result = np.sum(res**2)
        elif mode == 'chi2_ndf':
            result = 1/len(res) *np.sum(res**2)
        elif mode == 'res':
            result = res
        else:
            raise ValueError('Mode unknown, fot this: %s' % mode)
    if verbose and (mode != 'res'):
        print('gof={gof}, fs={fs}, t1={t1}, t3={t3}, tts={tts}'.format(gof=result, **params))
    return result

from copy import deepcopy
def gof_simultaneous(fs_er, fs_nr, verbose=True, mode='mean', **params):
    params = get_params(params)
    params_er = deepcopy(params)
    params_nr = deepcopy(params)
    params_er['dset'] = 'er'
    params_nr['dset'] = 'nr'
    params_er['fs'] = fs_er
    params_nr['fs'] = fs_nr
    gof_er = gof(verbose=False, mode=mode, **params_er)
    gof_nr = gof(verbose=False, mode=mode, **params_nr)
    if verbose:
        print('gof_er={gof_er}, gof_nr={gof_nr}, fs_er={fs_er}, fs_nr={fs_nr} t1={t1}, t3={t3}, tts={tts}'.format(
            gof_er=gof_er, gof_nr=gof_nr, fs_er = params_er['fs'], fs_nr = params_nr['fs'], **params))
    return gof_er + gof_nr

gof_simultaneous(fs_er = 0.2, fs_nr = 0.16, mode='chi2', error_offset = 2e-4)


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
GOF uncertainty
Need higher stats?
Fit three parameters
Can you write Python code for it?

from scipy import optimize
optresult = optimize.minimize(
    lambda x: gof(fs=x[0], t3=x[1], tts=x[2], s1_min = 30, s1_max = 100,
                                    error_pct = 0.5e-2, error_offset = 1e-5),
    [0.2, 24, 3],
    bounds=[[.01, 1], [20, 30], [.1, 5]],
    options=dict(maxfev=1000),
    method='Powell',
)

fit = optresult.x
print(fit)
ydata, _ = real_s1_wv()
minus, base, plus = s1_models_error(fs=fit[0], t3=fit[1], tts=fit[2], error_pct = 1e-2, error_offset = 1e-4)
comparison_plot(ydata, minus, base, plus, error_pct = 0.5e-2, error_offset = 1e-5)
plt.show()
comparison_plot_2(ydata, minus, base, plus, error_pct = 0.5e-2, error_offset = 1e-5)
plt.show()

def gof_v_parameter(parameter, variation_range, num, **params):
    params_to_try = np.linspace(*variation_range, num=num)
    gofs = []
    for param_value in params_to_try:
        params[parameter] = param_value
        gofs.append(gof(**params))
    return params_to_try, np.array(gofs)

def gof_v_2_paramters(parameter1, parameter2, variation_range1, variation_range2, num1, num2, **params):
    import time
    start = time.time()
    params_to_try1 = np.linspace(*variation_range1, num=num1)
    params_to_try2 = np.linspace(*variation_range2, num=num2)
    gvd = []
    for par1 in params_to_try1:
        for par2 in params_to_try2:
            params[parameter1] = par1
            params[parameter2] = par2
            gof_value = gof(**params)
            gvd.append([par1, par2, gof_value])
    stop = time.time()
    print('Computation took %d seconds (%.1f s/it)' % ((stop - start), (stop - start) / len(gvd)))
    return np.array(gvd)

nx = 20
ny = 20
ding = gof_v_2_paramters('fs', 't3', (0.16, 0.24), (23., 27.), nx, ny, tts=fit[2],
                         error_pct = 1e-2, error_offset = 1e-4, verbose=False)

plt.scatter(ding[:,0], ding[:,1], c=ding[:, 2])
plt.colorbar()

x = np.reshape(ding[:, 0], (nx, ny))
y = np.reshape(ding[:, 1], (nx, ny))
z = np.reshape(ding[:, 2], (nx, ny))

plt.pcolormesh(x, y, z/...


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
Gain variation
Can you write Python code for it?

gain_params = []
for ch, fn in enumerate(['170323_103732', '170323_104831']):
    with open('../pulse_shape_single_pe/%s_ch%d_function.pickle' % (fn, ch) , 'rb') as infile:
        _norm, _popt, _perr = pickle.load(infile)
        gain_params.append(np.concatenate([np.array([_norm]), _popt, _perr]))
gain_params = np.array(gain_params)

import scipy

def area_sample(n_values, gain_params, **params):
    params = get_params(params)
    channel = params['pulse_model']
    norm, mu, sigma, _, _ = gain_params[channel]
    lower, upper = (0., 3.)
    X = stats.truncnorm((lower - mu) / sigma, (upper - mu) / sigma, loc=mu, scale=sigma)
    return X.rvs(n_values)

def gaus_trunc(x, mu, sigma):
    return (x > 0) * np.exp( - (x - mu)**2 / (2 * sigma**2))

nbins = 600
ran = (-0.5, 3.5)

for channel in (0, 1):
    plt.hist(area_sample(200000, gain_params, pulse_model = channel), bins=nbins, histtype='step', normed=True, range=ran)
    x_plot = np.linspace(*ran, num=nbins)
    y_plot =  gaus_trunc(x_plot,gain_params[channel][1], gain_params[channel][2])
    norm = 1 / (np.sum(y_plot) * (ran[1] - ran[0])) * nbins
    plt.plot(x_plot, norm * y_plot)
    plt.title('Channel %d' % channel)
    plt.show()


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
Here is what we get out.
wv_matrix is a matrix containing the y-coordinates of the waveforms. The columns are the individual waveforms, to get the first waveform, go wv_matrix[:, 0]. time_matrix is the same thing except for it contains the times. t_shift_vector contains the shift of the waveform in ns (based on pulse times).
Statistical errors
Here we simulate statistical errors by simulating n_data_s1s and then performing bootstrap trials. The conclusion:....
Can you write Python code for it?

def s1_models_resample(*args, n_data_s1s=1000, bootstrap_trials=10, **kwargs):
    """Return bootstrap_trials waveform templates from sampling n_data_s1s s1s"""
    wv_matrix, time_matrix, _ = simulate_s1_pulse(*args, **kwargs)
    n_s1s = wv_matrix.shape[1]

    waveform_templates = np.zeros((len(spe_ts), bootstrap_trials))

    for i in range(bootstrap_trials):
        new_indices = np.random.randint(n_s1s, size=n_data_s1s)

        waveform_templates[:, i] = average_pulse(time_matrix[:, new_indices],
                                                 wv_matrix[:, new_indices])

    return waveform_templates

def sigmas_plot(x, q, color='b', **kwargs):
    for n_sigma, alpha in [(1,0.5), (2, 0.1)]:
        plt.fill_between(x,
                         np.percentile(q, 100 * stats.norm.cdf(-n_sigma), axis=1),
                         np.percentile(q, 100 * stats.norm.cdf(n_sigma), axis=1),
                         alpha=alpha, linewidth=0, color=color, step='mid')
    plt.plot(x,
             np.percentile(q, 50, axis=1),
             color=color, linestyle='-', alpha=0.5, linewidth=1, **kwargs)

waveform_templates = s1_models_resample(n_data_s1s=100, s1_min=50, s1_max=60, bootstrap_trials=100)
sigmas_plot(spe_ts, waveform_templates)


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
Model-data comparison
Plotting
Can you write Python code for it?

def residuals(ydata, minus, base, plus, **params):
    params = get_params(params)
    # CHANGED BY ERIK check for zero
    sigma = get_sigma(minus, base, plus, **params)
    if 0. in sigma:
        zero_positions = np.where(sigma == 0)
        print('Warning: found zero in error array at positions: ', zero_positions)
        print('Replacing with infinite error instead...')
        for pos in zero_positions:
            sigma[pos] = np.inf
    return (ydata - base) / sigma

def get_sigma(minus, base, plus, **params):
    params = get_params(params)
    sigma = np.abs(plus - minus)/2 + params['error_offset'] + params['error_pct'] * np.abs(base)
    return sigma

def comparison_plot(ydata, minus, base, plus, **params):
    params = get_params(params)
    sigmas = get_sigma(minus, base, plus, **params)

    # large subplot
    ax2 = plt.subplot2grid((3,1), (2,0))
    ax1 = plt.subplot2grid((3,1), (0,0), rowspan=2, sharex=ax2)

    #f, (ax1, ax2) = plt.subplots(2, sharex=True)
    plt.sca(ax1)
    # plt.fill_between(spe_ts, minus, plus, alpha=0.5, linewidth=0, step='mid')
    plt.fill_between(spe_ts, base - sigmas, base + sigmas,
                     alpha=0.5, linewidth=0, step='mid')
    plt.plot(spe_ts, base, linestyle='steps-mid', label='Model')
    plt.plot(spe_ts, ydata, marker='.', linestyle='', markersize=3, c='k', label='Observed')

    plt.grid(alpha=0.1, linestyle='-', which='both')
    plt.setp(ax1.get_xticklabels(), visible=False)
    plt.ylabel("Fraction of amplitude")
    plt.axhline(0, c='k', alpha=0.5)
    leg = plt.legend(loc='upper right', numpoints=1)
    leg.get_frame().set_linewidth(0.0)
    leg.get_frame().set_alpha(0.5)
    plt.ylim(0, None)

    #ax1.set_xticklabels([])

    # Add residuals
    plt.sca(ax2)
    plt.subplot2grid((3,1), (2,0), sharex=ax1)
    plt.xlim(params['t_min'], params['t_max'])

    res = residuals(ydata, minus, base, plus)

    plt.plot(spe_ts, res,
             linestyle='', marker='x', c='k', markersize=3)
    plt.ylim(-3, 3)
    plt.grid(which='both',...


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
Real data waveforms
Here we read the S1 data for three (highfield) datasets: NR, ER and BG_NR. We store it in the form of a dict (keys: er, nr, bg_nr). Each dict item is an array containing the waveforms (per row).
Can you write Python code for it?

xams_data = dict()
xams_data['nr'], xams_data['er'], xams_data['bg_nr'] = pickle.load(open('highfield_dataframes.pickle', 'rb'))

xams_s1s = dict()
# Get pulse waveforms to matrix rather than object column
for k, d in xams_data.items():
    xams_s1s[k] = np.array([x for x in d['s1_pulse']])
    del d['s1_pulse']


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
S1 model
Simulation
Can you write Python code for it?

import numba

# def split_s1_groups(x, n_x, s1_min, s1_max):
#     """Splits x into groups with uniform(s1_min, s1_max) elements, then return matrix of histograms per group.
#     Returns: integer array (n_x, n_groups)
#     n_x: number of possible values in x. Assumed to be from 0 ... n_x - 1
#     s1_min: minimum S1 number of hits
#     s1_max: maximum S1 number of hits
#     """
#     # We want to exhaust the indices x. Simulate a generous amount of S1 sizes
#     n_s1_est = int(1.5 * 2 * len(x) / (s1_min + s1_max))
#     if
#     hits_per_s1 = np.random.randint(s1_min, s1_max, size=n_s1_est)

#     result = np.zeros((n_x, n_s1_est), dtype=np.int)
#     s1_i = _split_s1_groups(x, hits_per_s1, result)
#     return result[:,:s1_i - 1]

# @numba.jit(nopython=True)
# def _split_s1_groups(x, hits_per_s1, result):
#     s1_i = 0
#     for i in x:
#         if hits_per_s1[s1_i] == 0:
#             s1_i += 1
#             continue
#         result[i, s1_i] += 1
#         hits_per_s1[s1_i] -= 1
#     return s1_i

def split_s1_groups(x, n_x, areas, **params):
    """Splits x into groups with uniform (s1_min, s1_max) elements, then return matrix of histograms per group.
    Returns: integer array (n_x, n_groups)
    n_x: number of possible values in x. Assumed to be from 0 ... n_x - 1
    s1_min: minimum S1 number of hits
    s1_max: maximum S1 number of hits
    """
    params = get_params(params)
    # We want to exhaust the indices x. Simulate a generous amount of S1 sizes
    n_s1_est = int(1.5 * 2 * len(x) / (params['s1_min'] + params['s1_max']))

    if params['s1_sample'] == 'data' and 'xams_data' not in globals():
        print('Warning: data-derived s1 area distribution not possible, reverting to uniform...')
        params['s1_sample'] = 'uniform'
    if params['s1_sample'] == 'uniform':
        pe_per_s1 = (params['s1_max'] - params['s1_min']) * np.random.random(size=n_s1_est) + params['s1_min']
    elif params['s1_sample'] == 'data':
        # Take S1 from the data sample
        s1s_data =...


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
Statistical errors are negligible if you have more than a few hundred waveforms.
Systematic errors
Can you write Python code for it?

import itertools

def s1_models_error(*args, shifts=None, **kwargs):
    '''
    Compute the error on the S1 waveform given errors on specific parameters.
    This will compute the S1 model for parameter +error, +0, and -error.
    All combinations of paramters are tried.
    `shifts` is a dict containting the allowed shift (+/-) for each model parameter.
    `*args` and `**kwargs` will be passed to `s1_average_pulse_model` to compute the base model.
    This function can also be used for getting the difference in pulse model for channel 0 and 1.
    '''
    if shifts is None:
        # Default uncertainty: in pulse model and in TTS
        shifts = dict(tts=0.5, pulse_model=[0,1])

    base_model = s1_average_pulse_model(*args, **kwargs)

    # Allow specifying a single +- amplitude of variation
    for p, shift_values in shifts.items():
        if isinstance(shift_values, (float, int)):
            shifts[p] = kwargs.get(p, default_params[p]) + np.array([-1, 0, 1]) * shift_values

    shift_pars = sorted(shifts.keys())
    shift_values = [shifts[k] for k in shift_pars]
    # shift_value_combs is a list of paramters that will be tried to compute the average pulse.
    # Contains all combintations: (+, 0, -) for all the parameters. ((3n)^2 for n number of parameters.)
    shift_value_combs = list(itertools.product(*shift_values))

    alt_models = []
    for vs in shift_value_combs:
        kw = dict()
        kw.update(kwargs)
        for i, p in enumerate(shift_pars):
            kw[p] = vs[i]
        alt_models.append(s1_average_pulse_model(*args, **kw))

    alt_models = np.vstack(alt_models)
    # Hmmm. this seems like an upper estimate of the error, no?
    # ask jelle
    minus = np.min(alt_models, axis=0)
    plus = np.max(alt_models, axis=0)
    return minus, base_model, plus

#     return [s1_average_pulse_model(*args, **kwargs)
#              for q in [-tts_sigma, 0, tts_sigma]]

minus, base, plus = s1_models_error()
plt.fill_between(spe_ts, minus, plus,...


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
The fit is pushing the singlet livetime to very low values... There is some degeneracy here, and also some mis-modeling, it seems. The sample at 0 is always under-estimated. Why? Maybe because the tts is actually quite low but modeled here as large. The effects may not be symmetric: there are many things causing a delay, but not a negative delay.
NR
Can you write Python code for it?

from scipy import optimize
optresult = optimize.minimize(
    lambda x: gof(fs=x[0], t1=x[1], t3=x[2], tts=x[3], s1_min=30, s1_max = 100, dst='nr'),
    [0.2, 3.1, 24, 3],
    bounds=[[.01, 1], [.1, 5], [20, 30], [.1, 5]],
    options=dict(maxfev=1000),
    method='Powell',
)

fit = optresult.x
print(fit)
ydata, _ = real_s1_wv()
minus, base, plus = s1_models_error(fs=fit[0], t1=fit[1], t3=fit[2], tts=fit[3], s1_min=30, s1_max = 100, dset='nr')
comparison_plot(ydata, minus, base, plus)
plt.show()
comparison_plot_2(ydata, minus, base, plus)
for _l in (-20, 100):
    plt.axvline(_l)
plt.xlim(-50, 200)
plt.show()


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
Wait, what? The residuals spread get larger with increasing stats? That does not sound right.
Can you write Python code for it?

for i in range(10):
    plt.plot(gof(mode='res', error_offset = 0.))

for i in range(10):
    plt.plot((gof(mode='res', error_offset = 0., error_pct = 0.1))**2)

def sigma_from_params(**params):
    params = get_params(params)
    # ydata, _ = real_s1_wv(**params)
    minus, base, plus = s1_models_error(**params)
    sigma = get_sigma(minus, base, plus, **params)
    sigma = sigma[(spe_ts >= params['t_min']) & (spe_ts < params['t_max'])]

    return sigma

plt.plot(1/sigma_from_params(error_pct = 5e-2, error_ofset = 1e-3))
plt.ylim(0)

iterations = 250
n_photons_scan = [int(1e4), int(3e4), int(7e4), int(2e5)]

const_gofs = []
for n_photons in n_photons_scan:
    print(n_photons)
    const_gofs.append([gof(verbose = False, mode='chi2', n_photons = n_photons,
                           error_pct = 1e-2, error_ofset = 1e-4) for _ in range(iterations)])

for gofs, n_photons, c in zip(const_gofs, n_photons_scan, ['blue', 'orange', 'green', 'red', 'black']):
    plt.hist(gofs / np.average(gofs), label="%d" % n_photons, histtype='step', range=(0, 2), bins=200, color = c)
    plt.axvline(color = c)
plt.legend()
plt.show()

ydata, n_data_s1s = real_s1_wv()
minus, base, plus = s1_models_error()
# res = residuals(ydata, minus, base, plus)

comparison_plot_2(ydata, minus, base, plus, error_pct = 1e-2, error_offset = 1e-4, t_max= 125)
# plt.ylim(0, 2)


I am working on the file "S1_psd_mc_Erik.ipynb".
The first task is:
What do we need the cumulative fraction for? Well, we input this into the custom_pmt_pulse_current in pax.simulation. Here is a quick check that all is well. There is just a little shift, but the alignment is quite arbitrary anyway.
Can you write Python code for it?

# custom_pmt_pulse_current(pmt_pulse, offset, dt, samples_before, samples_after)
from pax.simulation import custom_pmt_pulse_current
for ch, c in zip([0, 1], ['blue', 'red']):
    plt.plot(custom_pmt_pulse_current(spe_pulses_cum[ch], 0.1, 2, 10, 100), color=c)
    plt.plot(spe_ts * 0.5 + 10 - 0.5, spe_ys[ch] * 0.5, color=c, ls='--')
plt.xlim(-10, 60)
plt.xlabel('Time sample number')
plt.ylabel('Relative amplitude')
plt.show()


I am working on the file "SANER2016/notebooks/CRAN Distribution of GitHub Packages.ipynb".
The first task is:
Let's prepare the data for a survival analysis.
Can you write Python code for it?

import lifelines

survival = packages.copy()

# Remove packages that were first on CRAN
survival = survival.fillna(pandas.datetime(2050,1,1)).query('github < cran').replace(pandas.datetime(2050,1,1), pandas.np.nan)

# Observed packages
survival['observed'] = survival['elapsed'].apply(lambda x: not pandas.np.isnan(x))

# Censored packages (NaN) are set to now
survival['elapsed'] = survival.apply(lambda r: r['elapsed'] if not pandas.np.isnan(r['elapsed'])
               else (pandas.datetime.now() - r['github']) / pandas.np.timedelta64(1, 'D'), axis=1)

print len(packages), len(survival), len(survival[survival['observed']])

kmf = lifelines.KaplanMeierFitter()

plot_groups = [
    [
        {'label': 'All', 'df': survival.query('observed or not observed')},
        {'label': 'Without outliers', 'df': survival.query('observed == False or elapsed < 1200')},
    ],[
        {'label': 'Version >= 1', 'df': survival[survival['Version'].str[0] >= '1']},
        {'label': 'Version < 1', 'df': survival[survival['Version'].str[0] < '1']},
    ],[
        {'label': 'With CRAN dependencies', 'df': survival.query('cran_deps == True')},
        {'label': 'Without CRAN dependencies', 'df': survival.query('cran_deps == False')},
    ],
]

for i, group in enumerate(plot_groups):
    ax = plt.subplot(len(plot_groups), 1, i+1)
    for cond in group:
        print '{}: {} items, {} observed'.format(cond['label'], len(cond['df']), len(cond['df'].query('observed == True')))
        kmf.fit(cond['df']['elapsed'], event_observed=cond['df']['observed'], label=cond['label'])
        ax = kmf.plot(ax=ax)
    ax.set_title('Kaplan Meier Estimation, duration needed for a GitHub package to appear on CRAN')
    ax.figure.set_size_inches(12, 8)
    ax.set_xlabel('Duration (in days)')


I am working on the file "SANER2016/notebooks/GitHub - Data Extraction.ipynb".
The first task is:
We will make use of the following commands:
 - git clone <url> <path> where <url> is the url of the repository and <path> is the location to store the repository.
 - git log --follow --format="%H/%ci" <path> where <path> will be DESCRIPTION. The output of this command is a list of <commit> / <date> for this file.
 - git show <commit>:<path> where <commit> is the considered commit, and <path> will be DESCRIPTION. This command outputs the content of the file at the given commit.
Can you write Python code for it?

github = pandas.DataFrame.from_csv('../data/RPackage-Repositories-150101-150601.csv')
repositories = github[['owner.login', 'name']].rename(columns={'owner.login': 'owner', 'name': 'repositories'})

FILENAME = '../data/github-raw-150601.csv'

# Root of the directory where the repositories were collected
GIT_DIR = '/data/github/'


I am working on the file "SANER2016/notebooks/GitHub - Data Extraction.ipynb".
The first task is:
We will retrieve a lot of data, we can benefit from IPython's parallel computation tool.
To use this notebook, you need either to configure your IPController or to start a cluster of IPython nodes, using ipcluster start -n 4 for example. See https://ipython.org/ipython-doc/dev/parallel/parallel_process.html for more information.
It seems that most recent versions of IPython Notebook can directly start cluster from the web interface, under the Cluster tab.
Can you write Python code for it?

from IPython import parallel
clients = parallel.Client()
clients.block = False # asynchronous computations
print 'Clients:', str(clients.ids)

def get_data_from((owner, repository)):
    # Move to target directory
    try:
        os.chdir(os.path.join(GIT_DIR, owner, repository))
    except OSError as e:
        # Should happen when directory does not exist
        return []

    data_list = []

    # Get commits for DESCRIPTION
    try:
        commits = subprocess.check_output(['git', 'log', '--format=%H/%ci', '--', 'DESCRIPTION'])
    except subprocess.CalledProcessError as e:
        # Should not happen!?
        raise Exception(owner + ' ' + repository + '/ log : ' + e.output)

    for commit in [x for x in commits.split('\n') if len(x.strip())!=0]:
        commit_sha, date = map(lambda x: x.strip(), commit.split('/'))

        # Get file content
        try:
            content = subprocess.check_output(['git', 'show', '{id}:{path}'.format(id=commit_sha, path='DESCRIPTION')])
        except subprocess.CalledProcessError as e:
            # Could happen when DESCRIPTION was added in this commit. Silently ignore
            continue

        try:
            metadata = deb822.Deb822(content.split('\n'))
        except Exception as e:
            # I don't known which are the exceptions that Deb822 may throw!
            continue # Go further

        data = {}

        for md in ['Package', 'Version', 'License', 'Imports', 'Suggests', 'Depends', 'Author', 'Authors', 'Maintainer']:
            data[md] = metadata.get(md, '')

        data['CommitDate'] = date
        data['Owner'] = owner
        data['Repository'] = repository
        data_list.append(data)

    # Return to root directory
    os.chdir(GIT_DIR)
    return data_list

data = []

clients[:].execute('import subprocess, os')
clients[:].execute('from debian import deb822')
clients[:]['GIT_DIR'] = GIT_DIR

balanced = clients.load_balanced_view()

items = [(owner, repo) for idx,...


I am working on the file "SAT Scores/SAT Project.ipynb".
The first task is:
12. Print the min and max of each column
Can you write Python code for it?

print 'The minimum Rate is' + ' ' + str(min(scores.Rate))
print 'The maximum Rate is' + ' ' + str(max(scores.Rate))
print 'The minimum Verbal score is' + ' ' + str(min(scores.Verbal))
print 'The maximum Verbal score is' + ' ' + str(max(scores.Verbal))
print 'The minimum Math score is' + ' ' + str(min(scores.Math))
print 'The maximum Math score is' + ' ' + str(max(scores.Math))


I am working on the file "SAT Scores/SAT Project.ipynb".
The first task is:
15. Plot the Math distribution
Can you write Python code for it?

scores.Math.plot(kind='hist', bins=6, title='Mean SAT Math Scores for 2001')
plt.xlabel('Math Score')
plt.ylabel('Frequency')


I am working on the file "SAT Scores/SAT Project.ipynb".
The first task is:
17. What is the typical assumption for data distribution?
Typical assumption for data distribution is a normal distribution
18. Does that distribution hold true for our data?
It does not hold true for the above 3 histograms representing Rate, Verbal and Math. Rate is high on both ends of the graph. Math peaks at the low 500s. While Verbal peaks around 510 and 570.
19. Plot some scatterplots. BONUS: Use a PyPlot figure to present multiple plots at once.
Can you write Python code for it?

import matplotlib.pyplot as plt
scores.plot(kind='scatter', x='Verbal', y='Math', alpha=0.5)

scores.plot(kind='scatter', x='Verbal', y='Rate', alpha=0.5)

scores.plot(kind='scatter', x='Math', y='Rate', alpha=0.5)


I am working on the file "SAT Scores/SAT Project.ipynb".
The first task is:
20. Are there any interesting relationships to note?
Both the Verbal and Math mean scores were postively correlated. As one increases so did the other. In the scatterplot for Verbal and Rate chart, the higher scores were in the lowest rate of student participation. The scores were in closer proximity when compared to the scores displayed in the top left of the graph. As rates of participation increased the scores became more spreadout and lower. This held true (for the most part-excluding a couple or outliers) for the scatterplot for Math and Rate. This implies that higher rate of participation does not guarentee higher scores. The more student that take the SATs the more likely it is that the mean scores will decrease.
21. Create box plots for each variable.
Can you write Python code for it?

scores.Rate.plot(kind='box')

scores.Verbal.plot(kind='box')

scores.Math.plot(kind='box')


I am working on the file "SAT Scores/SAT Project.ipynb".
The first task is:
Creating a list of State names extracted from the data.
Can you write Python code for it?

list(scores.State)

### Checking that types are accurate.
scores.dtypes


I am working on the file "SAT Scores/SAT Project.ipynb".
The first task is:
The data lists the rate of participation and mean verbal/math scores of students who took the SAT in 2001 broken down by states.
Initial Analysis:
From my initial observations I look at row 51 and assume that the 45 in the rate column (and 506-Verbal and 514-Math) is the mean of the data in rows 0-50. Iowa and North Dakota, which are in the bottom 5 for rate of participation (at 47th and 48th, respectively), have some of the highest Math and Verbal mean scores. So, rate of participation is not indicative of strong scores. Also, I want to ask the College Board if they want to rank it by participation rate, which is the current method. Or would they prefer ranking the states by highest mean Verbal and Math scores for their presentation this year?
Can you write Python code for it?

scores.describe()

#Sorting Verbal scores in ascending order
scores.sort('Verbal')


I am working on the file "SAT Scores/SAT Project.ipynb".
The first task is:
Visualizing the data
Can you write Python code for it?

scores.Rate.plot(kind='hist', bins=5, title='Histogram of Rates of Participation')

import matplotlib.pyplot as plt
% matplotlib inline
scores.Rate.order().values


I am working on the file "SA_and_PSO.ipynb".
The first task is:
100回の試行を100回繰り返したが収束は続いている。
Can you write Python code for it?

for i in range(100):
    pso.simulate(100)
    print pso.GBest


I am working on the file "SA_and_PSO.ipynb".
The first task is:
PSOでxの初期値を1のまわりでのみでとる。
Can you write Python code for it?

pso = PSO(limit_x=[0.9, 1.1])
for i in range(10):
    pso.simulate(1000)
    print pso.GBest

pso.GBest_x


I am working on the file "SA_and_PSO.ipynb".
The first task is:
PSOを粒子数を100にして100回繰り返す。
Can you write Python code for it?


I am working on the file "SA_and_PSO.ipynb".
The first task is:
SAを1000回繰り返す。下はrosenbrock関数が返す値の推移である。
Can you write Python code for it?

sa = SA()
sa.simulate(1000)

plt.plot(sa.log)


I am working on the file "SA_and_PSO.ipynb".
The first task is:
[test1]収束の推移を視覚化する。以下はそのための関数と、その図。
Can you write Python code for it?

def test1(n):
    pso = PSO()
    pso.set_particles()
    p = []
    for i in range(n):
        best, best_x = pso.one_simulate(1)
        p.append(best)
    plt.plot(p)


I am working on the file "SA_and_PSO.ipynb".
The first task is:
[test2]粒子の数を200に増やしてtest1を行う。以下はそのための関数と、その図。
Can you write Python code for it?

def test2(n):
    pso = PSO(pN=200)
    pso.set_particles()
    p = []
    for i in range(n):
        best, best_x = pso.one_simulate(1)
        p.append(best)
    plt.plot(p)

for i in range(10):
    test2(30)


I am working on the file "SA_and_PSO.ipynb".
The first task is:
[test3]粒子の数と収束の関係を確認する。以下はそのための関数と、その図。図の見出しは粒子の数。
Can you write Python code for it?

def test3(n, maxpN):
    pso = PSO(pN=maxpN)
    pso.set_particles()
    p = []
    for i in range(n):
        best, best_x = pso.one_simulate(1)
        p.append(best)
    plt.plot(p, label=maxpN)
    plt.legend()

for i in range(1, 11):
    test3(50, i*10)

for i in range(5, 11):
    test3(50, i*10)

for i in range(7, 11):
    test3(50, i*10)

for i in range(11, 20):
    test3(50, i*10)


I am working on the file "SA_and_PSO.ipynb".
The first task is:
③SAとPSOを比較する。
1 SAの収束を確認する。rosenbrock関数が返す数がある値になって、xの更新が行われなくなったら試行をとめるテスト。
Can you write Python code for it?

def test_sa():
    sa = SA()
    a = sa.simulate(10)
    b = sa.simulate(10)
    c = sa.simulate(10)
    p = []
    while c != b and c != a:
        a = b
        b = c
        c = sa.simulate(10)
        p.append(c)
    plt.plot(p[5:])
    print p[-1]

for i in range(10):
    test_sa()


I am working on the file "SA_and_PSO.ipynb".
The first task is:
となった。すべての値が0.555の近くにいる。これより、おそらくすべての値が等しい状態をできるだけ崩さす大域的最適解に向かっていると考えられる。
すべてのxが1の値をとるとき、rosenbrock関数は0となることを発見している。これが唯一の解かどうかは議論しないが、最初に述べたようにすべてのxが同じ値をとることが局所最適解であり、あるxが他のxと違う値をとると局所最適解ではなくなってしまう。そのためすべての値が1以外の値をとる局所最適解に陥ってしまい、抜け出せなくなると考えられる。SAはその局所最適解にはまっている。PSOは局所最適解に陥りながらも、少しずつすべての値が1になるようにxの値を更新しているように見える。この試行を繰り返せば大域的最適解であると予想されるすべての値が1で、rosenbrock関数が0を返す状態に収束すると考えれらるが、上記で示したように、確かに収束しているものの、収束速度が遅いため最後まで収束させることは省略する。ただし、参考のために下にxの初期値を1のまわりでのみでとった場合の試行を乗せた。以上をもってレポートを締めくくる。
[おまけ:すべてのxが1の値をとるとき、rosenbrock関数は0となることの確認]
Can you write Python code for it?

c = [1.0 for i in range(50)]
rosenbrock(c)