I am working on the file "Daily/20150912_mag_profiles.ipynb".
The first task is:
Regions where $B/\rho$ has a negative slope are unstable to interchange modes. This is a significant problem for the Guassian radial field profile, which has should be unstable to interchange modes between 75% of the star, by radius. By constrast, however, the dipole magnetic field profile is only unstable over about 35% of the stellar interior, when the peak magnetic field is placed at $R = 0.15R_{\star}$.
For both the Gaussian and the dipole radial profiles, this can be mitigated by shifting the peak field strength to $R = 0.50R_{\star}$, shown by the light and dark blue lines. We adopted this peak magnetic field location in Feiden & Chaboyer (2014, A&A, 571, A70) and found that we were able to generate necessary structrual changes to the models to reproduce the properties of CM Dra. Peak magnetic field strengths, in these instances, were around 40 kG and 113 kG for the dipole and Gaussian profiles, repsectively. Although this was more of an aside in the text, we have shown that field configurations with sub-MG magnetic fields and seeming stability to interchange modes can induce required radius changes.
Estimate growth rate of instability, $\tau \propto H_p/v_a$.
Can you write Python code for it?

Hp = 1.0e10  # rough approximation for deep interior
va_d = 10**d_mag[:,1]/np.sqrt(4.0*np.pi*10**d_grad[:,1])
va_g = 10**g_mag[:,1]/np.sqrt(4.0*np.pi*10**g_grad[:,1])


I am working on the file "Daily/20151123_agb_inner_boundary.ipynb".
The first task is:
In the case where the radiative opacity is close to $10^{-2}\ {\rm cm}^2\, {\rm g}^{-1}$, the radiative gradient may exceed the adiabatic temperature gradient, suggesting that layers may be unstable to convective instabilities. It is therefore quite possible that thermodynamic conditions developing at the inner boundary of the RHD atmosphere models are ammenable to convective energy transport.
Since the inner boundary ignores convection and ionization, a temperature purturbation may lead to a growing instability in the temperature as radiation is becomes less able to transport the flux (at a given temperature) and ionization stages of hydrogen do not exist to alter the specific heat when the temperature begins to rapidly increase toward $8\,000 - 10\,000\ {\rm K}$.
A temperature instability is all inevitable once conditions become ammenable to convection (at least within the framework of adiabaic convection). Radiation will be unable to efficiently transport flux and the neglect of physics related to convective flux transport will force the local temperature gradient (and thus the deep interior temperature) to force the flux through the gas using only radiation.
We may be able to estimate what temperature is required, assuming the pressure is defined by the condition for hydrostatic equilibrium. The approximate conditions required for radiation to carry all of the flux through an ideal gas (neglecting ionization) is
\begin{equation}
    \nabla_{\rm rad} = \nabla_{\rm ad} = \frac{2}{5}.
\end{equation}
Therefore, the local temperature can be approximated at the inner boundary
\begin{equation}
    T_{\rm gas} = \left(\frac{15}{32\pi acG} \kappa P_{\rm gas} \frac{L_r}{M_r}\right )^{1/4}.
\end{equation}
For conditions at the inner boundary, the temperature must exceed
Can you write Python code for it?


I am working on the file "Daily/20151123_agb_inner_boundary.ipynb".
The first task is:
which produces a density
Can you write Python code for it?


I am working on the file "Daily/20160326_SMC_PreMS.ipynb".
The first task is:
And compute equipartition magnetic field strengths along each isochrone.
Can you write Python code for it?

Beq_01 = computeBeq(iso_01)
Beq_05 = computeBeq(iso_05)
Beq_10 = computeBeq(iso_10)
Beq_15 = computeBeq(iso_15)


I am working on the file "Daily/20160326_SMC_PreMS.ipynb".
The first task is:
Atmosphere tables are structured so that $T_{\rm eff}$ values are listed in the first column, but $\log g$ is not tabulated; it must be prescribed.
Can you write Python code for it?


I am working on the file "Daily/20160326_SMC_PreMS.ipynb".
The first task is:
Next, we'll load the atmospheric boundary condition table used to compute the isochrones. The tables are compiled using PHOENIX AMES-COND model atmospheres with a solar composition from Grevesse & Sauval (1998). While interior structure models rely on prescribing the gas pressure and temperature at $\tau_{\rm ross}=10$, we need to choose the table for $\tau_{\rm ross}=1$.
Can you write Python code for it?


I am working on the file "Daily/20160326_SMC_PreMS.ipynb".
The first task is:
Now to compare how equipartition magnetic field strengths vary with age.
Can you write Python code for it?


I am working on the file "Daily/20160326_SMC_PreMS.ipynb".
The first task is:
The rest of the table is structured as a set of values (P, T) at each $\log g$. It'll be easier to create new arrays for each of these.
Can you write Python code for it?

temps = np.empty((len(teffs), len(loggs)))
press = np.empty((len(teffs), len(loggs)))
for i, teff in enumerate(atm[:, 1:]):
    for j, prop in enumerate(teff):
        if j%2 == 0:
            press[i, j/2] = prop
        else:
            temps[i, j/2] = prop


I am working on the file "Daily/20160712_Seasonal_Variation_Halflife.ipynb".
The first task is:
Alternatively, we can base the estimate on the relative strength, which relieves uncertainties due to GMsun and MEarth.
Can you write Python code for it?

g_ratio = np.power(d_aphelion/d_perihelion, 2)


I am working on the file "Daily/20160712_Seasonal_Variation_Halflife.ipynb".
The first task is:
Assuming the maximum contamination fraction for Potassium-40 of 20 ppb, there are roughly
Can you write Python code for it?

N_40K = 20.0*N_NaI/1.0e9
print N_40K, "Potassium-40 atoms in the sample."


I am working on the file "Daily/20160712_Seasonal_Variation_Halflife.ipynb".
The first task is:
Basic properties concerning Earth's orbit about the Sun and some of the physics parameters involved in the problem.
Can you write Python code for it?


I am working on the file "Daily/20160712_Seasonal_Variation_Halflife.ipynb".
The first task is:
Now our task is to estimate the approximate increase or decrease in Potassium-40 decays based on variations in the Sun-Earth gravitational potential.
During a single half-life for Potassium-40, approximately $1\times10^{19}$ atoms decay. Given a mean half-life of $1.251\times10^{9}$ years, there will be approximately
Can you write Python code for it?

decay_rate_average = 1.0e19/(1.251e9*60.0*60.0*24.0*365.256366)
print decay_rate_average, "decays per second on average."


I am working on the file "Daily/20160712_Seasonal_Variation_Halflife.ipynb".
The first task is:
Now we must convert the relative difference in gravitational force into a difference in time dilation for various processes. Since we're dealing with the Sun, we may adopt the Schwarzschild metric, which suggests
\begin{equation}
    t_0 = t_f \left( 1 - \frac{r_s}{r} \right)^{1/2}.
\end{equation}
The Schwarzschild radius for the Sun is
Can you write Python code for it?


I am working on the file "Daily/20160712_Seasonal_Variation_Halflife.ipynb".
The first task is:
The decay rate at perihelion and aphelion will be slightly different due to the gravitational potential of the Earth.
Can you write Python code for it?

decay_rate_aphelion = 1.0e19/(1.251e9*60.0*60.0*24.0*365.256366*t0_ratio_aphelion)
print decay_rate_aphelion, "decays per second at aphelion."

decay_rate_perihelion = 1.0e19/(1.251e9*60.0*60.0*24.0*365.256366*t0_ratio_perihelion)
print decay_rate_perihelion, "decays per second at perihelion."


I am working on the file "Daily/20160712_Seasonal_Variation_Halflife.ipynb".
The first task is:
We see that the strength of gravitational force between the Earth and Sun is about 7% larger at perihelion as compared to aphelion.
Can you write Python code for it?


I am working on the file "Daily/20160712_Seasonal_Variation_Halflife.ipynb".
The first task is:
which is small, but perhaps non-neligible when it comes to seasonal variations in the decay of Potassium-40, for example. In the case of the DAMA experiment, they have 250 kg of sodium iodide crystals with a small 20 ppb contamination of Potassium-40. Potassium-40 is radioactive with a long half-life of $1.251\times10^9$ years. There are two primary decay channels:
\begin{equation}
    ^{40}\textrm{K}\ \rightarrow\ ^{40}\textrm{Ca} + \beta^{-} + \bar{\nu}
\end{equation}
and
\begin{equation}
    ^{40}\textrm{K} + \beta^{-}\ \rightarrow\ ^{40}\textrm{Ar} + \gamma + \nu
\end{equation}
which occur about 90% and 10% of the time, respectively.
The molar mass of sodium iodide is 149.89 g mol$^{-1}$, meaning in 250 kg there are
Can you write Python code for it?

moles_NaI = 250.0e3/149.89
print moles_NaI, " moles of NaI,"


I am working on the file "Daily/20160712_Seasonal_Variation_Halflife.ipynb".
The first task is:
which translates into
Can you write Python code for it?

N_NaI = moles_NaI*6.022140857e23
print N_NaI, "NaI molecules."


I am working on the file "Daily/20160910_low_mass_polytrope_lum.ipynb".
The first task is:
Current seed values
Scripts used to generate a new polytrope for DMESTAR models rely on a piece-wise function to generate an appropriate combination of $T_{\rm eff}$ and luminosity for a model based on the requested stellar mass and solar composition. That piece-wise function is
\begin{align}
    \log(T) & = 3.64 & M \ge 3.9 \
    \log(L) & = 0.2\cdot (M - 5.0) + 2.6 & \
     & \
    \log(T) & = -0.028\cdot M + 3.875 & 3.9 > M \ge 3.0 \
    \log(L) & =  0.55 \cdot M + 0.1   & \
     & \
    \log(T) & =  0.039\cdot M + 3.5765 & 3.0 > M \ge 1.5 \
    \log(L) & =  1.7  & \
     & \
    \log(T) & =  0.039\cdot M + 3.5765 & 1.5 > M \ge 0.23 \
    \log(L) & =  0.85\cdot M + 0.4  & \
     & \
    \log(T) & =  0.614\cdot M + 3.3863 & 0.23 > M \
    \log(L) & = -0.16877\cdot M - 0.117637  & \
\end{align}
While models with masses below $0.23 M$ are found to converge, the greatest issues occur right in the vicinity of the final piecewise condition. We can view this graphically,
Can you write Python code for it?

fig, ax = plt.subplots(2, 1, figsize=(8, 8))

masses = np.arange(0.08, 5.0, 0.02)

# compute and plot temperature relationship
p1 = [3.64 for m in masses if m >= 3.9]
p2 = [-0.028*m + 3.875 for m in masses if 3.9 > m >= 3.0]
p3 = [0.039*m + 3.5765 for m in masses if 3.0 > m >= 0.23]
p4 = [0.614*m + 3.3863 for m in masses if m < 0.23]
tr = p4 + p3 + p2 + p1

ax[0].set_xlabel("initial mass [Msun]")
ax[0].set_ylabel("log(T / K)")

ax[0].plot(masses, tr, '-', c='#dc143c', lw=3)

# plot luminosity relationship
# compute and plot temperature relationship
p1 = [0.2*(m - 5.0) + 2.6 for m in masses if m >= 3.9]
p2 = [0.55*m + 0.1 for m in masses if 3.9 > m >= 3.0]
p3 = [1.7 for m in masses if 3.0 > m >= 1.5]
p4 = [0.85*m + 0.4 for m in masses if 1.5 > m >= 0.23]
p5 = [-0.16877*m - 0.117637 for m in masses if m < 0.23]
lr = p5 + p4 + p3 + p2 + p1

ax[1].set_xlabel("initial mass [Msun]")
ax[1].set_ylabel("log(L / Lsun)")

ax[1].plot(masses, lr, '-', c='#dc143c', lw=3)


I am working on the file "Daily/20160910_low_mass_polytrope_lum.ipynb".
The first task is:
Implementation
These new fits better represent the relaxed models, but will they work when implemented as seed values?
Can you write Python code for it?

print "log(T) and log(L) Coefficients for the lowest mass objects: \n", tpoly1, poly1
print "\n\nlog(T) and log(L) Coefficients for low mass objects: \n", tpoly2, poly2


I am working on the file "Daily/20160910_low_mass_polytrope_lum.ipynb".
The first task is:
Luminosity
Above 1.5 Msun, there appear to be very little deviations of the true model sequence from the initial seed model sequence. We can thus leave this parameteriztion alone. Below 1.5 Msun, we can alter the shape of the relationship down to 0.23 Msun. In addition, we can prescribe a new shape to the relationship for objects with masses below 0.23 Msun.
Can you write Python code for it?


I am working on the file "Daily/20160910_low_mass_polytrope_lum.ipynb".
The first task is:
Plotting these two relations, we can compare against the function used to generate the polytrope seed model.
Can you write Python code for it?

fig, ax = plt.subplots(2, 1, figsize=(8, 8))

masses = np.arange(0.08, 5.0, 0.02)

ax[0].set_xlabel("initial mass [Msun]")
ax[0].set_ylabel("log(T / K)")

ax[0].semilogx(model_props[:,0], model_props[:,1], '-', c='#008b8b', lw=3)
ax[0].semilogx(masses, tr, '-', c='#dc143c', lw=3)

ax[1].set_xlabel("initial mass [Msun]")
ax[1].set_ylabel("log(L / Lsun)")

ax[1].semilogx(model_props[:,0], model_props[:,2], '-', c='#008b8b', lw=3)
ax[1].semilogx(masses, lr, '-', c='#dc143c', lw=3)


I am working on the file "Daily/20160910_low_mass_polytrope_lum.ipynb".
The first task is:
Relaxed model values
We can compare the relationship(s) quoted above with model values for temperature and luminosity after the model has relaxed to a stable configuration. This takes only a couple time steps to achieve, so we will look at the model relationship during the third time step for all models with masses between 0.08 and 5.0 Msun. Models are taken from a recent study where we used the most up-to-date version of the Dartmouth models for young stars (Feiden 2016).
Can you write Python code for it?

model_directory = "../../papers/MagneticUpperSco/models/trk/std/"

# get all file names
from os import listdir
all_fnames = listdir(model_directory)

# sort out only those file names that end in .trk
fnames = [f for f in all_fnames if f[-4:] == ".trk"]

# sort numerically
fnames = sorted(fnames)


I am working on the file "Daily/20160910_low_mass_polytrope_lum.ipynb".
The first task is:
To select which model time step is most representative of a relaxed model, we can step through the first 50 iterations to find if there are any noticable jumps in model properties.
Can you write Python code for it?

fig, ax = plt.subplots(2, 1, figsize=(8, 8))

model_props = np.empty((len(fnames), 3))
for j in range(0, 50):
    for i, f in enumerate(fnames):
        model_props[i, 0] = float(f[1:5])/1000.0

        try:
            trk = np.genfromtxt(model_directory + f, usecols=(0, 1, 2, 3))
        except ValueError:
            model_props[i, 1] = 0.0    # temperature
            model_props[i, 2] = 0.0    # luminosity
            continue

        model_props[i, 1] = trk[j, 1]  # temperature
        model_props[i, 2] = trk[j, 3]  # luminosity

    ax[0].semilogx(model_props[:,0], model_props[:,1], '-', c='#008b8b', lw=3)
    ax[1].semilogx(model_props[:,0], model_props[:,2], '-', c='#008b8b', lw=3)


I am working on the file "Daily/20160910_low_mass_polytrope_lum.ipynb".
The first task is:
We can now iterate through these filenames and save the third timestep to an array.
Can you write Python code for it?

model_props = np.empty((len(fnames), 3))
for i, f in enumerate(fnames):
    model_props[i, 0] = float(f[1:5])/1000.0

    try:
        trk = np.genfromtxt(model_directory + f, usecols=(0, 1, 2, 3))
    except ValueError:
        model_props[i, 1] = 0.0    # temperature
        model_props[i, 2] = 0.0    # luminosity
        continue

    model_props[i, 1] = trk[1, 1]  # temperature
    model_props[i, 2] = trk[1, 3]  # luminosity


I am working on the file "Damped_Oscillator_SlimplecticGGLvsRK.ipynb".
The first task is:
Comparison plots
Plot the oscillator amplitude $q(t)$ for the 2nd and 4th order slimplectic and RK integrators along with the analytical solution.
Can you write Python code for it?

# Analytical solution
Omega = np.sqrt(k/m - ll**2/4.)
phi0 = - np.arctan(-ll/(2.*Omega))

def q(time):
    """Analytical solution for simple damped harmonic oscillator amplitude with q0=1, v0=0"""
    return np.exp(-ll*time/2.)*np.cos(Omega*time + phi0)

def v(time):
    """Analytical solution for simple damped harmonic oscillator velocity with q0=1, v0=0"""
    return np.exp(-ll*time/2.)*( -ll/2.*np.cos(Omega*time+phi0) - Omega*np.sin(Omega*time+phi0) )

fig1 = plt.figure(figsize=(12,5), dpi=800)
fig1.subplots_adjust(wspace=0.05)

# Early times...
ax1a = fig1.add_subplot(1,4,1)
ax1a.set_ylim(-2, 2)
ax1a.set_xlim(0,24)
ax1a.plot(t, q_slim2[0], 'r-', linewidth=2.0, rasterized=True)
ax1a.plot(t, q_slim4[0], color='orange', linewidth=2.0, rasterized=True)
ax1a.plot(t, q_rk2[0], 'g--', linewidth=2.0, rasterized=True)
ax1a.plot(t, q_rk4[0], 'b--', linewidth=2.0, rasterized=True)
ax1a.plot(t, q(t), 'k:', linewidth=2.0, rasterized=True)

# Middle times...
ax1b = fig1.add_subplot(1,4,(2,3))
plt.setp(ax1b.get_yticklabels(), visible=False)
ax1b.set_ylim(-2, 2)
ax1b.set_xlim(24.9,9975.9)
ax1b.plot(t, q_slim2[0], linewidth=2.0, linestyle='-', color='red', rasterized=True)
ax1b.plot(t, q_slim4[0], linewidth=2.0, linestyle='-', color='orange', rasterized=True)
ax1b.plot(t, q_rk2[0], linewidth=2.0, linestyle='--', color='g', rasterized=True)
ax1b.plot(t, q_rk4[0], linewidth=2.0, linestyle='--', color='blue', rasterized=True)

# Late times...
ax1c = fig1.add_subplot(1,4,4)
plt.setp(ax1c.get_yticklabels(), visible=False)
ax1c.set_ylim(-2, 2)
ax1c.set_xlim(9976,10000)
ax1c.plot(t, q_slim2[0], linewidth=2.0, linestyle='-', color='red', rasterized=True)
ax1c.plot(t, q_slim4[0], linewidth=2.0, linestyle='-', color='orange', rasterized=True)
ax1c.plot(t, q_rk2[0], linewidth=2.0, linestyle='--', color='g', rasterized=True)
ax1c.plot(t, q_rk4[0], linewidth=2.0, linestyle='--', color='blue', rasterized=True)
ax1c.plot(t, q(t), 'k:', linewidth=2.0, rasterized=True)

ax1a.tick_params(axis='both', which='major',...


I am working on the file "Damped_Oscillator_SlimplecticGGLvsRK.ipynb".
The first task is:
Let's see how the oscillator's energy changes with time according to the different orders and integration schemes. The energy is
$$E = \frac{1}{2} m \dot{q}^2 + \frac{1}{2} k q^2$$
and changes in time according to
$$\frac{ dE}{dt} = - \lambda \dot{q}^2.$$
To quantify the errors incurred by discretization and subsequent numerical integration, we define the fractional or relative energy difference as $\delta E / E = ( E_X - E )/ E$ where $E_X$ is the energy as measured by integrator $X$ with $X \in { {\rm Slim2,~Slim4,~RK2,~RK4} }$.
Can you write Python code for it?

# Energy function
def Energy(q, v):
    return 0.5*m*v**2 + 0.5*k*q**2

# Energies from the analytic solution and from different integrators
E = Energy(q(t), v(t))

E_slim2 = Energy(q_slim2[0], pi_slim2[0]/m)
E_slim4 = Energy(q_slim4[0], pi_slim4[0]/m)

E_rk2 = Energy(q_rk2[0], v_rk2[0])
E_rk4 = Energy(q_rk4[0], v_rk4[0])

fig2 = plt.figure(figsize=(12,5), dpi=500)

ax2 = fig2.add_subplot(1,1,1)
ax2.set_ylim(1e-10, 1e1)
ax2.set_xlim(0.1, 10000)

ax2.loglog(t, np.abs(E_slim2/E-1.), 'r-', linewidth=2.0, rasterized=True)
ax2.loglog(t, np.abs(E_slim4/E-1.), color='orange', linestyle='-', linewidth=2.0, rasterized=True)
ax2.loglog(t, np.abs(E_rk2/E-1.), 'g--', linewidth=2.0, rasterized=True)
ax2.loglog(t, np.abs(E_rk4/E-1.), 'b--', linewidth=2.0, rasterized=True)

ax2.set_xlabel('Time, $t$ [$(m/k)^{1/2}$]', fontsize=18)
ax2.set_ylabel('Fractional energy error, $\delta E/E$', fontsize=18)

ax2.text(0.12, 0.75, r'$\Delta t = 0.1 (m/k)^{1/2}$', fontsize=18, color='k')
ax2.text(50, 3e-4, r'$2^{nd}$ order Slimplectic', fontsize=15, color='k')
ax2.text(50, 6e-8, r'$4^{th}$ order Slimplectic', fontsize=15, color='black')
ax2.text(2, 3e-6, r'$4^{th}$ order RK', fontsize=15, color='b', rotation=11)
ax2.text(50, 1.5e-1, r'$2^{nd}$ order RK', fontsize=15, color='g', rotation=11)

ax2.tick_params(axis='both', which='major', labelsize=16)

#fig2.savefig(plot_path + "DampSimpleELog.pdf", transparent=True,bbox_inches='tight', dpi=300)


I am working on the file "Damping_radius_relation.ipynb".
The first task is:
Also relation 3 (different derivation by Millen et al.)
James Millen derives the following form of the damping due to impinging particles:
$$ \Gamma^{imp} = \dfrac{4\pi}{3}\dfrac{mNr^2 \bar{v}{T{imp}}}{M} $$
https://arxiv.org/abs/1309.3990 -
https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.114.123602
However in their earlier paper http://iopscience.iop.org/article/10.1088/1367-2630/15/1/015001/meta they get double this, which is what Gieseler gets in his thermal non-linearities paper.
Where:
- $m$ is the molecular mass of the gas
- $N$ is the particle density of the gas
- $r$ is the radius of the nanoparticle
- $M$ is the mass of the nanoparticle
- $\bar{v}{T{imp}}$ is the mean thermal velocity $\sqrt{\dfrac{8 k_B T^{imp}}{\pi m}}$
Using the ideal gas equation $P = R\rho T$ and $N= \dfrac{\rho}{m}$ with $R=\dfrac{k_B}{m}$ we get $N = \dfrac{P}{k_BT}$
Can you write Python code for it?

def Gamma_Millen_imp(radius, Pressure_mbar):
    Pressure = 100*Pressure_mbar
    ave_velocity = (8*k_B*T0/(pi*m_gas))**0.5
    mass = rho_SiO2*4/3*pi*radius**3
    N = Pressure/(k_B*T0)
    Gamma0 = 4*pi*m_gas*N*radius**2*ave_velocity/(3*mass)
    return Gamma0

Gamma_Millen_imp(R, 3)


I am working on the file "Damping_radius_relation.ipynb".
The first task is:
Alternativity one can use:
$$ s = \dfrac{\eta_{air}}{P_{gas}} \sqrt{\dfrac{\pi k_B T_0}{2m}} $$
this produces the same result as the previous form
https://en.wikipedia.org/wiki/Mean_free_path
Where
- $\eta_{air}$ is the viscosity of air
- $m$ is the molecualar mass of air
- $T_0$ is the temperature of the gas
- $P_{gas}$ is the pressure of the gas
molecular mass of air is $28.97 g/mol$ and the number of molecules in a mole is Avogadro's Number $6.0221409e^{23}$ therefore we get the molecular mass of air to be $4.81e^{-26} Kg$
Can you write Python code for it?

m_gas = 4.81e-26

def mfp_2(P_gas):
    mfp_val = eta_air/P_gas * (pi*k_B*T0/(2*m_gas))**0.5
    return mfp_val

s = mfp(300) # 3mbar = 300 Pascals
print(s)

s2 = mfp_2(300) # 3mbar = 300 Pascals
print(s2)

def Gamma_env(radius, Pressure_mbar):
    mass = rho_SiO2 * 4/3*pi*radius**3
    Pressure_pascals = 100*Pressure_mbar
    s = mfp(Pressure_pascals)
    K_n = s/radius
    c_K = 0.31*K_n/(0.785 + 1.152*K_n + K_n**2)
    Gamma_0 = 6*pi*eta_air*radius/mass * 0.619/(0.619 + K_n) * (1+c_K)
    return Gamma_0

Gamma_env(R, 3)


I am working on the file "Damping_radius_relation.ipynb".
The first task is:
Plot of all 3 relations and measured data
Can you write Python code for it?

P_exp = np.load("Pressure_mbar.npy")
Gamma_exp = np.load("Gamma_radians.npy")

P_G_Dict = dict(zip(P_exp, Gamma_exp))

r = np.linspace(5e-9, 1000e-9, 1000)
P = 3.6 # mbar
alpha=0.5

plt.figure(figsize=[10, 10])
plt.loglog(r, Gamma_env_simple(r, P), 'k', label="Rashid/Gieseler Full form", alpha=alpha)
#plt.semilogy(r, Gamma_env_simple(r, P), 'grey', label="Rashid/Gieseler simplfied form", alpha=alpha)
plt.loglog(r, Gamma_alternative(r, P), label="Gieseler Thermal Non-linearities form", alpha=alpha)
plt.loglog(r, Gamma_chang(r, P), label="Chang form", alpha=alpha)
plt.loglog(r, Gamma_Millen_imp(r, P), label="Millen (imp) form", alpha=alpha)
plt.xlabel("radius (nm)")
plt.ylabel("Γ (radians/s)")
plt.legend(loc='best')
plt.show()

r = 50e-9
P = np.linspace(1e-2, 1000, 1000)

plt.figure(figsize=[10, 10])
plt.loglog(P, Gamma_env_simple(r, P), 'k', label="Rashid/Gieseler Full form", alpha=alpha)
#plt.loglog(P, Gamma_env_simple(r, P), 'grey', label="Rashid/Gieseler simplfied form", alpha=alpha)
plt.loglog(P, Gamma_alternative(r, P), label="Gieseler Thermal Non-linearities form", alpha=alpha)
plt.loglog(P, Gamma_chang(r, P), label="Chang form", alpha=alpha)
plt.loglog(P, Gamma_Millen_imp(r, P), label="Millen (imp) form", alpha=alpha)
plt.loglog(P_exp, Gamma_exp, label="Experiment", alpha=alpha)
plt.xlabel("P (mbar)")
plt.ylabel("Γ (radians/s)")
plt.legend(loc='best')
plt.show()


I am working on the file "Damping_radius_relation.ipynb".
The first task is:
Relation 2
In Gieseler's Thermal Nonlinearities paper he has the following equation for $\Gamma_0$
$$ \Gamma_0 = \dfrac{64a^2}{3m\bar{v}}P $$
https://www.nature.com/nphys/journal/v9/n12/full/nphys2798.html
This appears to be incorrect as it is exactly double that which you get with Chang's formula and James Millen's formula
Where:
- $a$ is the radius of the particle
- $m$ is the mass of the particle
- $\bar{v}$ is the average verlocity of the gas particles
Where we can use the following formula for $\bar{v}$
$$ \bar{v} = \sqrt{\dfrac{8k_B T_0}{\pi \mu}} $$
Where:
- $T_0$ is the temperature of the gas
- $\mu$ is the mass of the air molecules
Can you write Python code for it?

def Gamma_alternative(radius, Pressure_mbar):
    Pressure = 100*Pressure_mbar
    ave_velocity = (8*k_B*T0/(pi*m_gas))**0.5
    mass= rho_SiO2*4/3*pi*radius**3
    Gamma0 = 64*radius**2*Pressure/(3*mass*ave_velocity)
    return Gamma0

Gamma_alternative(R, 3)


I am working on the file "Damping_radius_relation.ipynb".
The first task is:
Relation 3+ (more damping due to considering emerging particles)
James Millen derives the following form of the damping due to emerging particles:
$\Gamma^{em} = \dfrac{mNr^2\pi^{\frac{3}{2}}}{3\sqrt{h'}M}$
https://arxiv.org/abs/1309.3990 -
https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.114.123602
Where:
- $m$ is the molecular mass of the gas
- $N$ is the particle density of the gas
- $r$ is the radius of the nanoparticle
- $h'$ is $\dfrac{m}{2k_B T_0}$ where $T_0$ is the temperature of the gas
- $M$ is the mass of the nanoparticle
Using the ideal gas equation $P = R\rho T$ and $N= \dfrac{\rho}{m}$ with $R=\dfrac{k_B}{m}$ we get $N = \dfrac{P}{k_BT}$
He also says that this leads to $\Gamma^{em} = \dfrac{\pi}{8}\dfrac{T^{em}}{T^{imp}}$
From this you get the total effective damping rate is
$$ \Gamma_0 = \Gamma^{em} + \Gamma^{imp} = \dfrac{\pi}{8}\sqrt{\dfrac{T^{em}}{T^{imp}}}\Gamma^{imp} + \Gamma^{imp} $$
Therefore damping rate is higher if you consider this
Can you write Python code for it?

def Gamma_Millen_em(radius, Pressure_mbar, T_em):
    Pressure = 100*Pressure_mbar
    h_prime = m_gas/(k_B*T_em)
    mass = rho_SiO2*4/3*pi*radius**3
    N = Pressure/(k_B*T_em)
    Gamma0 = (m_gas*N*radius**2*pi**(3/2))/(3*np.sqrt(h_prime)*mass)
    return Gamma0

def calc_surface_temp_Millen(T_em, T_imp=300):
    accomodation_coef = 0.777 # accomodation coefficient of silica (from Nanoscale temp measurement paper)
    T_surf = T_imp + (T_em + T_imp)/accomodation_coef
    return T_surf


I am working on the file "Damping_radius_relation.ipynb".
The first task is:
This agrees exactly with Chang's result
Can you write Python code for it?


I am working on the file "Daqss API How-to.ipynb".
The first task is:
Exporting
Now we are going to export this data to a csv for whatever reason!
Can you write Python code for it?


I am working on the file "Daqss API How-to.ipynb".
The first task is:
Set up the class instance of Daqss
Can you write Python code for it?

# Set up an instance of the class Daqss with keyword arguments key and password
api = Daqss(key, pswd)

# Ping to see if you have a connection
api.ping()


I am working on the file "Daqss API How-to.ipynb".
The first task is:
Your first API call
To begin, let's try grabbing all of the nodes from the database. Instead of worrying about the API endpoint, we can just call api.get_nodes() and you will get what you desire!
Everything is in json format, so you will need to learn how to deal with that..don't worry, it's easy!
Can you write Python code for it?

status, req = api.get_nodes()

print ("The Status Code: {0}".format(status))
print ("The response: \n")
req['nodes']


I am working on the file "Das Code Check.ipynb".
The first task is:
Checking rough meaning of each topic-mean
Using Gensim Word2Vec model's Most_Similar() method.  This gives us a super quick and dirty idea of the peak of the distribution lays in the coordinate space.   While this may not be informative of the covariance and what words it might produce, its fast + easy
More data munging just to look at it
Can you write Python code for it?

for k in range(1, 51): # choose on number of topics.. only doing +1 since I was lazy and made it start at one, and not zero.
    mean = das_data[k]['mean'].T
    print wvmodel.most_similar(positive=[mean])

for k in range(1, 51):
    das_data[k]['mean'] = np.array(das_data[k]['mean'])
    das_data[k]['cov'] = np.array(das_data[k]['cov'])


I am working on the file "Das Code Check.ipynb".
The first task is:
Less words now!
Can you write Python code for it?

print len(list(vocab))
print len(words)
vocab = list(vocab)


I am working on the file "Das Code Check.ipynb".
The first task is:
Removing very common words
Can you write Python code for it?

temp_corp = []
common_words = zip(*c.most_common(n=50))[0]
for doc in corpus:
    doc = doc.split()
    doc = filter(lambda x: x not in common_words, doc)
    doc = " ".join(doc)
    temp_corp.append(doc)
corpus = temp_corp


I am working on the file "Das Code Check.ipynb".
The first task is:
Sanity checking output - list (corpus) of lists(docs) of ints(words)
Can you write Python code for it?


I am working on the file "Das Code Check.ipynb".
The first task is:
Shaping data for the java model
It requires strange files where words are indexed, and we provide the word-vectors corropsonding to the word indicies.
Can you write Python code for it?

wordids = {word: i for i, word in enumerate(vocab)}

vecs = {i: wvmodel[word] for word, i in wordids.iteritems()}

index_corpus = []
for doc in corpus:
    docids = []
    for word in doc.split():
        docids.append(wordids[word])
    index_corpus.append(docids)


I am working on the file "Das Code Check.ipynb".
The first task is:
Using 50-dim word vectors trained on wikipedia
Can you write Python code for it?

wvmodel = Word2Vec.load_word2vec_format(
    "/Users/michael/Documents/Gaussian_LDA-master/data/glove.wiki/glove.6B.50d.txt", binary=False)
print "word-vector dimension: {}".format(wvmodel.vector_size)


I am working on the file "Dask Intro.ipynb".
The first task is:
Delayed functions
Have a look at the cell below - it does various basic arithemitc opertions.
Can you write Python code for it?

def inc(x):
    return x + 1

def double(x):
    return x * 2

def add(x, y):
    return x + y

def process_list(xs):
    output = []
    for x in data:
        a = inc(x)
        b = double(x)
        c = add(a, b)
        output.append(c)
    return output

def mysum(xs):
    total = 0
    for x in xs:
        total += x

    return total

double(2)

inc(double(2))

data = [1, 2, 3, 4, 5]

result = mysum(process_list(data))

print(result)


I am working on the file "Dask Intro.ipynb".
The first task is:
The cell above was just run on the same computer that this Notebook is running on. Let's submit it to the compute cluster using our Executor e.
You should open this window and click on Status where you can watch all the jobs (yours and others) being submitted to the comput node.
Can you write Python code for it?


I am working on the file "Dask Intro.ipynb".
The first task is:
The first thing to do is grab the Dask executor - the thing we'll use for submitting jobs to our cluster.
Can you write Python code for it?

e = Executor('dask.informaticslab.co.uk:8786')
print(e)


I am working on the file "Dask Intro.ipynb".
The first task is:
You should see now how many nodes are available in the executor. Ask Niall to show you how we scale this up and down using AWS.
Execute a function on the nodes
We're going to write a simple function which we can exectue on the nodes. The code in the cell below just gets the IP of the computer its run on
Can you write Python code for it?

import socket
getIP = lambda: "Hello world from IP " + socket.gethostbyname(socket.gethostname())

getIP()


I am working on the file "Data 608/Homework 4/Homeword 4 Notebook.ipynb".
The first task is:
2. Which sites have been tested most regularly?
Can you write Python code for it?

testRegularly = data.groupby(['Site'])['EnteroCount'].count()
testRegularly = testRegularly.to_frame().reset_index()
testRegularly.columns = ['Site','Number of Tests']
testRegularly = testRegularly.sort_values(['Number of Tests'],ascending=False).head(10).reset_index(drop=True)
testRegularly


I am working on the file "Data 608/Homework 4/Homeword 4 Notebook.ipynb".
The first task is:
Create lists & graphs of the best and worst places to swim in the dataset
Can you write Python code for it?

# water is unacceptable if a single sample is greater than 110 Enterococcus/100 mL
# water is unacceptable if five(5) or more samples weighted average is more than 30 Enterococcus/100 mL

data['Year'] = data['Date'].apply(lambda x: x.year)
placesToSwim = data.groupby(['Site','Year'])['EnteroCount'].sum()
placesToSwim = placesToSwim.to_frame().reset_index()
placesToSwim['RecordCount'] = data.groupby(['Site','Year'])['EnteroCount'].transform('count')
placesToSwim['yearSum'] = placesToSwim.groupby(['Year'])['RecordCount'].transform('sum')
placesToSwim['WeightedMean'] = (placesToSwim['EnteroCount'] * placesToSwim['RecordCount'])/ placesToSwim['yearSum']

# add unaccecptable/ acceptable check
placesToSwim['Classification'] = np.where(((placesToSwim.WeightedMean > 110) & (placesToSwim.RecordCount < 5 )) |
                                          ((placesToSwim.WeightedMean > 30) & (placesToSwim.RecordCount > 4 ))
                                          ,'unacceptable', 'acceptable')

placesToSwim.head()

#Worse Places to Swim

def plot_it(Year,Filter,Sorting):
    placesToSwimData = placesToSwim[(placesToSwim['Year']==int(Year)) & (placesToSwim['Classification']==Filter)]
    if Filter == 'acceptable':
        placesToSwimData = placesToSwimData.head(10)
    else:
        placesToSwimData = placesToSwimData.tail(10)

    if Sorting == 'ascending':
        placesToSwimData = placesToSwimData.sort_values(['WeightedMean'],ascending=[True])
    else:
        placesToSwimData = placesToSwimData.sort_values(['WeightedMean'],ascending=[False])

    fig = {
        'data': [
            {
                'x': placesToSwimData['Site'],
                'y': placesToSwimData['WeightedMean'],
                'name': Year,
                'type': 'bar'
            }
        ],

    'layout': Layout(xaxis=XAxis(title='Site'), yaxis=YAxis(title='Weighted Mean'))
    }
    py.iplot(fig)

interact(plot_it, Year=('2006', '2007','2008', '2009', '2010', '2011', '2012', '2013'),
         Filter...


I am working on the file "Data 608/Homework 4/Homeword 4 Notebook.ipynb".
The first task is:
Scatterplot of the data
Can you write Python code for it?

trace = Scatter(
    x = relationship['FourDayRainTotal'],
    y = relationship['EnteroCount'],
    mode = 'markers',
    name='Scatter Plot'
)

plotData = [trace]

layout = Layout(
    title='Scatter Plot',
    xaxis=dict(
        title='Four Day Rain Total',
        titlefont=dict(
            family='Courier New, monospace',
            size=18,
            color='#7f7f7f'
        )
    ),
    yaxis=dict(
        title='Entero Count',
        titlefont=dict(
            family='Courier New, monospace',
            size=18,
            color='#7f7f7f'
        )
    )
)
fig = Figure(data=plotData, layout=layout)
py.iplot(fig, filename='basic-scatter')


I am working on the file "Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data.ipynb".
The first task is:
Beers in Pubs
Germans do love their beers and the dataset shows that certain pubs, restaurants or bars are sponsored by certain beer brands (often advertised on the pubs entrance). We can analyze the prevalence of beer brands by grouping and sorting by occurence of the attribute brewery for all the amenities classified as respective establishment. Most popular are Radeberger, a very popular local beer, Feldschlösschen, a swiss beer and Dresdner Felsenkeller, a very local and niche-sort-of beer.
Can you write Python code for it?

from Project.notebook_stub import project_coll
import pprint

# Query used - see function: Project.audit_stats_map.stats_beers(...):
pipeline = [
        {"$match": {"amenity": {"$in":["pub","bar","restaurant"]}}},
        {"$group": {"_id": "$brewery", "count": {"$sum": 1}}},
        {"$sort": {"count": -1}}
    ]
l = list(project_coll.aggregate(pipeline))
pprint.pprint(l)


I am working on the file "Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data.ipynb".
The first task is:
Cleaning Phone Numbers
Try it out: Use python project.py -C to clean in debug mode. See Sample Output in file Project/output_project.py_-C.txt. The script also writes a CSV file to Project/data/clean_phones.csv, which is also beautified into a Excel File.
Cleaning the phone numbers involves:
* unifying the different phone attributes (phone, address.phone and mobile_phone) - this is already taken care by extracting the phone numbers during the audit stage
* if possible, canonicalizing the phone number notations by parsing them using a regular expression:
python
phone_regex = re.compile(ur'^(\(?([\+|\*]|00) *(?P<country>[1-9][0-9]*)\)?)?' +  # country code
                         ur'[ \/\-\.]*\(?0?\)?[ \/\-\.]*' +  # separator
                         ur'(\(0?(?P<area1>[1-9][0-9 ]*)\)|0?(?P<area2>[1-9][0-9]*))?' +  # area code
                         ur'[ \/\-\.]*' +  # separator
                         ur'(?P<number>([0-9]+ *[\/\-.]? *)*)$', # number
                         re.UNICODE)
The regular expression is resilient to various separators ("/", "-", " ", "(0)") and bracket notation of phone numbers. It is not resilient for some unicode characters or written lists of phone numbers which are designed to be interpreted by humans (using separators like ",", "/-" or "oder" lit. or). During the cleaning stage, an output is written which phone numbers could not be parsed. This contains only a tiny fraction of phone numbers (9 or 0.5%) which would be easily cleanable by hand.
The following objects couldn't be parsed:
                                                 normalized
55f57294b1c8a72c34523897           +49 35207 81429 or 81469
55f57299b1c8a72c345272cd  +49 351 8386837, +49 176 67032256
55f572c2b1c8a72c34546689                      0351 4810426
55f572c3b1c8a72c34546829         +49 351 8902284 or 2525375
55f572fdb1c8a72c34574963   +49 351 4706625, +49 351 0350602
55f573bdb1c8a72c3460bdb3                +49 351 87?44?44?00
55f573bdb1c8a72c3460c066         0162 2648953, 0162...
Can you write Python code for it?

from Project.notebook_stub import project_coll

# Query used - see function: Project.audit_quality_map.audit_streets(...):
expectedStreetPattern = \
    u"^.*(?<![Ss]tra\u00dfe)(?<![Ww]eg)(?<![Aa]llee)(?<![Rr]ing)(?<![Bb]erg)" + \
    u"(?<![Pp]ark)(?<![Hh]\u00f6he)(?<![Pp]latz)(?<![Bb]r\u00fccke)(?<![Gg]rund)$"
l = list(project_coll.distinct("name", {
                    "type": "way",
                    "name": {"$regex": expectedStreetPattern}
                }))
# Output too long... See the file Project/output_project.py_-q.txt


I am working on the file "Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data.ipynb".
The first task is:
Cuisines in Restaurants
We can list the types of cuisines in restaurants (elements with attribute amenity matching restaurant) and sort them in decending order. We can notice certain inconsistencies or overlaps in the classifications of this data: e.g., a kebab cuisine may very well be also classified as an arab cuisine or may, in fact a sub- or super-classification of this cuisine. One could, e.g., eliminate or cluster together especially occurences of cuisines which are less common, but Without having a formal taxonomy of all cuisines, I decided that is probably best to leave the data as-is in order to not sacrifice preciseness for consistency.
Can you write Python code for it?

from Project.notebook_stub import project_coll
import pprint

# Query used - see function: Project.audit_stats_map.stats_cuisines(...):
pipeline = [
        {"$match": {"amenity": "restaurant"}},
        {"$group": {"_id": "$cuisine", "count": {"$sum": 1}}},
        {"$sort": {"count": -1}}
    ]
l = list(project_coll.aggregate(pipeline))
pprint.pprint(l[1:10]+['...'])


I am working on the file "Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data.ipynb".
The first task is:
Popular Leisure Activities
The attribute leisure shows the types of leisure activities one can do in Dresden and inspired me to invesigate more on popular sports in the city (leisure=sports_center or leisure=stadium).
Can you write Python code for it?

from Project.notebook_stub import project_coll
import pprint

# Query used - see function: Project.audit_stats_map.stats_amenities(...):
pipeline = [
        {"$match": {"leisure": {"$exists": True}}},
        {"$group": {"_id": "$leisure", "count": {"$sum": 1}}},
        {"$sort": {"count": -1}}
    ]
l = list(project_coll.aggregate(pipeline))
pprint.pprint(l[1:10]+['...'])


I am working on the file "Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data.ipynb".
The first task is:
Popular Sports
To investigate, which sports are popular, we can group and sort by the (occurence of the) sport attribute for all elements classified as sports_centre or stadium in their leisure attribute. Unsurprisingly for a german city, we notice that 9pin (bowling) and soccer are the most popular sports, followed by climbing, an activity very much enjoyed by people in Dresden, presumably because of the close-by sand-stone mountains of the national park Sächsische Schweiz.
Can you write Python code for it?

from Project.notebook_stub import project_coll
import pprint

# Query used - see function: Project.audit_stats_map.stats_sports(...):
pipeline = [
        {"$match": {"leisure": {"$in": ["sports_centre","stadium"]}}},
        {"$group": {"_id": "$sport", "count": {"$sum": 1}}},
        {"$sort": {"count": -1}}
    ]
l = list(project_coll.aggregate(pipeline))
pprint.pprint(l[1:5]+['...'])


I am working on the file "Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data.ipynb".
The first task is:
Problems Encountered in the Map / Data Quality
Try it out: Use python project.py -q to obtain the data from this chapter. See Sample Output in file Project/output_project.py_-q.txt. The script also writes a CSV file to Project/data/audit_buildings.csv, which is also beautified into a Excel File.
Leading Zeros
As already discussed, during the parsing stage, we are using an optimistic approach of parsing any numerical value as integer or float, if it is parsable as such. However, we noticed that we should not do this, if leading zeros are present as those hold semantics for phone numbers and zip codes. Otherwise, this cleaning approach gives us a much smaller representation of the data in MongoDB and in-memory.
Normalizing / Cleaning Cuisines
As hinted in section Cuisines in Restaurant, classification of cuisines is inconsistent. There are two problems with this value:

There are multiple values separated by ';' which makes the parameter hard to parse. We overcome this by creating a parameter cuisineTag which stores the cuisine classifications as an array:

python
  db.eval('''db.osmnodes.find({
                        "cuisine": {"$exists": true},
                        "amenity": "restaurant"
                    }).snapshot().forEach(function(val, idx) {
                        val.cuisineTags = val.cuisine.split(';');
                        db.osmnodes.save(val)
                    })
             ''')

Some values are inconsistently used; therefore, we unify them with a mapping table and a subsequent MongoDB update:

```python
  cuisines_synonyms = {
      'german': ['regional', 'schnitzel', 'buschenschank'],
      'portuguese': ['Portugiesisches_Restaurant_&_Weinbar'],
      'italian': ['pizza', 'pasta'],
      'mediterranean': ['fish', 'seafood'],
      'japanese': ['sushi'],
      'turkish': ['kebab'],
      'american': ['steak_house']
    }
  # not mapped:
  # greek, asian, chinese, indian, international, vietnamese, thai, spanish, arabic
  # sudanese, russian, korean, hungarian, syrian, vegan, soup,...
Can you write Python code for it?

from Project.notebook_stub import project_coll

# Query used - see function: Project.audit_quality_map.audit_phone_numbers(...):
pipeline = [
        {"$match": {"$or": [
                {"phone": {"$exists": True}},
                {"mobile_phone": {"$exists": True}},
                {"address.phone": {"$exists": True}}
            ]}},
        {"$project": {
                "_id": 1,
                "phone": {"$ifNull": ["$phone", {"$ifNull": ["$mobile_phone", "$address.phone"]}]}
            }}
    ]
l = project_coll.aggregate(pipeline)

# Output too long... See the file Project/output_project.py_-q.txt


I am working on the file "Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data.ipynb".
The first task is:
The resulting objects are then iterated through and the best and worst fitting nearby street name are identified each using the Levenshtein distance. For each object, a row is created in a DataFrame which is subsequently exported to a csv file Project/data/audit_buildings.csv that was manually beautified into an Excel File.

As can be seen, street names of nearby objects mostly match those of the building itself (Levenshtein distance is zero). If they deviate greatly, they are totally different street names in the same area and not just "typos" or non-conforming abbreviations.
Auditing Zip Codes (Spoiler Alert: No Cleaning Necessary)
Try it out: Use python project.py -Z which runs the auditing script for zipcodes. See Sample Output in file Project/output_project.py_-Z.txt. To be able to run this script correctly, the zipcode data from Geonames.org needs to be downloaded  and installed first using the -z option.
This part of the auditing process makes use of an additional at Geonames.org to resolve and audit the zip codes in the data set. During the "installation process" (option -z) the zipcode data (provided as a tab-separated file) is downloaded and, line-by-line, stored to a (separate) MongoDB collection. However, we are only interested "zipcode" (2) and  "place" (3)
During the auditing stage (option -Z) we first get a list of all used zipcode using the following query:
python
pipeline = [
            { "$match": {"address.postcode": {"$exists": 1}} },
            { "$group": {"_id": "$address.postcode", "count": {"$sum": 1}} },
            { "$sort": {"count": 1} }
        ]
The zipcodes are then all looked up in the zipcode collection using the $in-operator. The data obtained is joined back into the original result.
python
zipcodeObjects = zipcodeColl.find( {"zipcode": {"$in": [z["_id"] for z in zipcodeList]}} )
The following output shows that there the lesser used zipcodes are from the Dresden metropolitan area, not Dresden itself:
Can you write Python code for it?

from Project.audit_zipcode_map import audit_zipcode_map
from Project.notebook_stub import project_server, project_port
import pprint

zipcodeJoined = audit_zipcode_map(project_server, project_port, quiet=True)
pprint.pprint(zipcodeJoined[1:10]+['...'])


I am working on the file "Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data.ipynb".
The first task is:
Where to Dance in Dresden
I am a passionate social dancer, so a list of dance schools in Dresden should not be abscent from this investigation. We can quickly grab all elements which have the leisure attribute set to dancing.
Can you write Python code for it?

from Project.notebook_stub import project_coll
import pprint

# Query used - see function: Project.audit_stats_map.stats_dances(...):
l = list(project_coll.distinct("name", {"leisure": "dance"}))
pprint.pprint(l[1:10]+['...'])


I am working on the file "Data Analysis learning/Data_Analysis_2(Numpy Pandas).ipynb".
The first task is:
<h5> All element less than 50 ??
Can you write Python code for it?


I am working on the file "Data Analysis learning/Data_Analysis_2(Numpy Pandas).ipynb".
The first task is:
<h6>See All basic results using describe() function
Can you write Python code for it?


I am working on the file "Data Analysis learning/Data_Analysis_2(Numpy Pandas).ipynb".
The first task is:
Array are similar to python list , but it all element must be of same data type, and it faster than list
Can you write Python code for it?


I am working on the file "Data Analysis learning/Data_Analysis_2(Numpy Pandas).ipynb".
The first task is:
In-place operation in numpay (Diff between += and +)
Can you write Python code for it?

a=np.array([5,6,15])
b=a
a += 2
print b
print "this happen becouse a and b both point to same array and += is In-place operation so it maintain that"

a=np.array([5,6,15])
b=a
a = a + 2
print b


I am working on the file "Data Analysis learning/Data_Analysis_2(Numpy Pandas).ipynb".
The first task is:
Lets see some of functionality
Can you write Python code for it?

print "Mean :",num.mean()
print "sum :",num.sum()
print "max :",num.max()
print "std :",num.std()

#slicing
num[:5]

#find index of any element let say max
print "index of max :",num.argmax()

print "data Type of array :",num.dtype


I am working on the file "Data Analysis learning/Data_Analysis_2(Numpy Pandas).ipynb".
The first task is:
Vector Operation
Can you write Python code for it?

a=np.array([5,6,15])
b=np.array([5,4,-5])

# Addition
print "{} + {} = {}".format(a,b,a+b)

print "{} * {} = {}".format(a,b,a*b)

print "{} / {} = {}".format(a,b,a/b)

# If size mismatch then error occure
b=np.array([5,4,-5,5])
print "{} + {} = {}".format(a,b,a+b)


I am working on the file "Data Analysis learning/Data_Analysis_2(Numpy Pandas).ipynb".
The first task is:
find all elemnt greter than 100 from num
Can you write Python code for it?


I am working on the file "Data Analysis learning/Data_Analysis_2(Numpy Pandas).ipynb".
The first task is:
vector [+-*/] Scaler
Can you write Python code for it?


I am working on the file "Data Analysis learning/Data_Analysis_3(pandas Basics).ipynb".
The first task is:
s.apply(function_name) used to apply some operation on each element.

Example:

adding 5 to each element , we can do this by simply series+5 becouse it is a vector, But lets do using this new techniqe s.apply(function)

Can you write Python code for it?

print res

print res+5

def add_5(x):
    return x+5

print res.apply(add_5)


I am working on the file "Data Analysis learning/Data_Analysis_3(pandas Basics).ipynb".
The first task is:
<h3> Pandas Index </h3>
<p>
Hybrid of list and python Dictionary. It map key value pair.
</p>
Can you write Python code for it?

sal=pd.Series([40,12,43,56],
             index=['Ram',
                  'Syam',
                  "Rahul",
                  "Ganesh"])
print sal

print sal[0]


I am working on the file "Data Analysis learning/Data_Analysis_3(pandas Basics).ipynb".
The first task is:
<h3> Plotting </h3>
<p>
automaticaly plot index vs data plot
</p>
Can you write Python code for it?


I am working on the file "Data Analysis learning/Data_Analysis_3(pandas Basics).ipynb".
The first task is:

Pandas series

pandas series is similar to numpy array, But it suppport lots of extra functionality like  Pandaseries.describe()

Basic acces is samilar to numpy arrary, it support access by index( s[5] ) or slicing ( s[5:10] ).

It also support vectorise operation and looping like numpy array.

Implemented in C so it works very fast.

Benfits of Pandas series
Can you write Python code for it?


I am working on the file "Data Analysis learning/Data_Analysis_3(pandas Basics).ipynb".
The first task is:

lookUp by index

Can you write Python code for it?


I am working on the file "Data Analysis learning/Data_Analysis_3(pandas Basics).ipynb".
The first task is:

argmax() function return index of max value element

Can you write Python code for it?

print sal.argmax()

print sal.loc["Ganesh"]
print sal.max()


I am working on the file "Data Analysis learning/Data_Analysis_3(pandas Basics).ipynb".
The first task is:

Using sal[position] is not prefered instead prefer to use sal.iloc[position]
becouse Index has different meaning in series so it avoid confusion

Can you write Python code for it?


I am working on the file "Data Analysis learning/Data_Analysis_3(pandas Basics).ipynb".
The first task is:
Adding series with Differen index
Can you write Python code for it?


I am working on the file "Data Analysis of NBA Players Challenge.ipynb".
The first task is:
Find the Average of Each Statistic
Find the average value for each statistic. The columns have names like PER (player efficiency rating) and GP (Games Played) that represent the season statistics for each player. For more on the various statistics look here.
Can you write Python code for it?


I am working on the file "Data Analysis of NBA Players Challenge.ipynb".
The first task is:
If we want to get summary statistics about the fit, like r-squared value, we can use the score method of the Sckit-Learn model. However, if we want more advanced regression statistics we’ll need to do a bit more. The statsmodels package enables many statistical methods to be used in Python and is a good tool to know.
Fit a Random Forest Model
Our linear regression worked well in the single variable case, but we suspect there may be nonlinearities in the data. Thus, we want to fit a random forest model.
Can you write Python code for it?

# Compute random forest from the predictors "AGE", "MPG", "TO", "HT", "WT", "REBR" to the target, "AST"


I am working on the file "Data Analysis of NBA Players Challenge.ipynb".
The first task is:
In Python, matplotlib is the primary plotting package, and seaborn is a widely used layer over matplotlib. You could have also used the pandas scatter_matrix for a similar result.
Make Clusters of the Players
One good way to explore this kind of data is to generate cluster plots. These will show which players are most similar. Use Scikit-Learn to cluster the data.
Can you write Python code for it?

# Use a clustering model like K-means to cluster the players


I am working on the file "Data Analysis of NBA Players Challenge.ipynb".
The first task is:
In Python, the recent version of pandas came with a sample method that returns a certain proportion of rows randomly sampled from a source dataframe – this makes the code much more concise. We could also use Scikit-Learns KFolds and train_test_splits for different types of shuffle and splits in the data set. In both cases, we set a random seed to make the results reproducible.
Univariate Linear Regression
Let’s say we want to predict number of assists per player from the turnovers made per player.
Can you write Python code for it?

# Compute the univariate regression of TO to AST


I am working on the file "Data Analysis of NBA Players Challenge.ipynb".
The first task is:
Look at the First Row of Data
What does the first row look like?
Can you write Python code for it?


I am working on the file "Data Analysis of NBA Players Challenge.ipynb".
The first task is:
Remember to import the pandas library to get access to Dataframes. Dataframes are two-dimensional arrays (matrices) where each column can be of a different datatype.
Find the number of Players
How many players are in the dataset?
Can you write Python code for it?

# Print the number of rows and columns in the dataframe


I am working on the file "Data Analysis of NBA Players Challenge.ipynb".
The first task is:
We can use the main Python machine learning package, scikit-learn, to fit a k-means clustering model and get our cluster labels. In order to cluster properly, we remove any non-numeric columns, or columns with missing values (NA, Nan, etc) with the get_numeric_data and dropna methods.
Plot Players by Cluster
We can now plot out the players by cluster to discover patterns. One way to do this is to first use PCA to make our data 2-dimensional, then plot it, and shade each point according to cluster association.
Can you write Python code for it?

# Use PCA to plot the clusters in 2 dimensions


I am working on the file "Data Analysis of NBA Players Challenge.ipynb".
The first task is:
With Python, we used the PCA class in the scikit-learn library. We used matplotlib to create the plot.
Split into Training and Testing Sets
If we want to do supervised machine learning, it’s a good idea to split the data into training and testing sets so we don’t overfit.
Can you write Python code for it?

# Create train (80%) and test (20%) splits


I am working on the file "Data Analysis with Pandas - Intermediate/Challenge - Summarizing Data.ipynb".
The first task is:
3: Summarizing Major Categories
In both of these datasets, majors are grouped into categories. There are multiple rows with a common value for Major_category but different values for Major. We would like to know the total number of people in each Major_category for both datasets.
Can you write Python code for it?

all_ages_major_categories = {}
recent_grads_major_categories = {}

def calculate_major_cat_totals(df):
  counts_dictionary = {}
  for cat in df["Major_category"].value_counts().index:
    counts_dictionary[cat] = df["Total"][df["Major_category"] == cat].sum()
  return counts_dictionary

all_ages_major_categories = calculate_major_cat_totals(all_ages)
recent_grads_major_categories = calculate_major_cat_totals(recent_grads)

print all_ages_major_categories
print recent_grads_major_categories


I am working on the file "Data Analysis with Pandas - Intermediate/Challenge - Summarizing Data.ipynb".
The first task is:
4: Low Wage Jobs Rates
The press likes to talk a lot about how many college grads are unable to get higher wage, skilled jobs and end up working lower wage, unskilled jobs instead. As a data person, it is your job to be skeptical of any broad claims and analyze relevant data to obtain a more nuanced view. Let's run some basic calculations to explore that idea further.
Can you write Python code for it?

low_wage_percent = recent_grads["Low_wage_jobs"].astype(float).sum() / recent_grads["Total"].sum()

print low_wage_percent


I am working on the file "Data Analysis.ipynb".
The first task is:
Below, we will print out each tracker as well as the number of pings it gave to any base station, and the time at which the pings occurred.
Can you write Python code for it?

print(list(tracker_dfs.keys())[0])
tracker_dfs[list(tracker_dfs.keys())[0]]

print(list(tracker_dfs.keys())[1])
tracker_dfs[list(tracker_dfs.keys())[1]]

print(list(tracker_dfs.keys())[2])
tracker_dfs[list(tracker_dfs.keys())[2]]

print(list(tracker_dfs.keys())[3])
tracker_dfs[list(tracker_dfs.keys())[3]]


I am working on the file "Data Analysis.ipynb".
The first task is:
Perform the filtering for each tracker. Legitimate pings are the ones that occurred between the start and end times of each day, inclusive.
Can you write Python code for it?

tracker_dfs = dict()

for i, (tracker_id, dt) in enumerate(tracker_ids_time.items()):
    # tracker_id = some hexadecimal number.
    # dt = the datetime object indicating the time at which the seal was finished sampling
    tracker_dfs[tracker_id] = master_data[(master_data.tracker_id == tracker_id) & (master_data.date >= dt)]
    trkr = tracker_dfs[tracker_id]
    trkr = trkr[((trkr['date'] >= day1_start) & (trkr['date'] <= day1_end)) | \
                ((trkr['date'] >= day2_start) & (trkr['date'] <= day2_end)) | \
                ((trkr['date'] >= day3_start) & (trkr['date'] <= day3_end))]
    tracker_dfs[tracker_id] = trkr


I am working on the file "Data Analysis.ipynb".
The first task is:
Se guardan los datos en un archivo para poder ser reconstruidos facilmente
Can you write Python code for it?


I am working on the file "Data Analysis.ipynb".
The first task is:
Se lee la base de datos sobre el archivo de excel, se guarda en un 'dataframe' (objeto sobre el cual se hacen operaciones)
Can you write Python code for it?

data_set = pd.read_excel('/home/sergio/Workspace/enc/__BD Prejuicios Luz (5).xlsx', sheetname='BD')

data_set.columns.tolist()

data_set.to_pickle('/home/sergio/Workspace/enc/full_data_enc.pkl')

data = pd.read_pickle('/home/sergio/Workspace/enc/full_data_enc.pkl')

data.columns.tolist()


I am working on the file "Data Analysis.ipynb".
The first task is:
Se limpian los datos, se seleccionan las columnas que dan informacion acerca de prejuicios
contra jóvenes y contra adultos. Se normalizan y escalan los datos de una escala de 1 a 5 a
una escala de 0 a 1 para simplificar los cálculos y poder obtener un score para cada encuestado
Can you write Python code for it?

n_data[cols_jovenes].describe()

n_data[cols_adultos].describe()

n_data['score_vs_adultos'] = n_data[cols_adultos].sum(axis=1)/14
n_data['score_vs_jovenes'] = n_data[cols_jovenes].sum(axis=1)/14


I am working on the file "Data Analysis.ipynb".
The first task is:
Se observa que algunas preguntas no tienen respuestas validas, esto puede ser
por error humano de transcripcion de datos.
Can you write Python code for it?

plot_1 = sns.distplot(data_useful.Edad.dropna(), bins=10)
plot_1.set(xlabel='Edad', ylabel='Kernel Density')
plt.show()

data_useful.Edad.plot(kind='hist', bins=10, title='Histograma edad')
plt.show

data_useful.Orientacion_Religiosa.plot(kind='hist', title='Histograma Religion', color='m')

data_useful.Escolaridad.plot(kind='hist', bins=23, title='Escolaridad', color='r')

data_useful.Sexo.plot(kind='hist', bins=3, title='Sexo', color='b')

columnas = ['Emocion1', 'Emocion2', 'Emocion3', 'Emocion4', 'Emocion8', 'Emocion9',
                   'Emocion14', 'Emocion15', 'Emocion17', 'Emocion21', 'Emocion26', 'Emocion28',
                   'Emocion34', 'Emocion36', 'Cognicion6', 'Cognicion7', 'Cognicion9', 'Cognicion10',
                   'Cognicion11', 'Cognicion15', 'Cognicion21', 'Cognicion30', 'Cognicion41', 'Cognicion47',
                   'Cognicion50', 'Cognicion54', 'Cognicion58', 'Cognicion65', 'Cognicion74', 'Cognicion75',
                   'Cognicion78', 'Conducta1', 'Conducta2', 'Conducta6', 'Conducta13', 'Conducta19', 'Conducta21',
                   'Conducta23', 'Conducta25', 'Conducta31', 'Conducta32', 'Conducta36', 'Conducta41', 'Conducta46',
                   'Conducta52', 'Creencia2', 'Creencia12', 'Creencia27', 'Creencia37', 'Creencia39', 'Creencia49']

data_clean = data_useful[data_useful[columnas].isin([1,2,3,4,5])]

data_clean[['Sexo', 'Edad', 'Lugar_Origen', 'Estado_civil',
                   'Escolaridad', 'Orientacion_Religiosa', 'Practicante', 'Orientacion_Sexual',
                   'Pareja']] = data_useful[['Sexo', 'Edad', 'Lugar_Origen', 'Estado_civil',
                   'Escolaridad', 'Orientacion_Religiosa', 'Practicante', 'Orientacion_Sexual',
                   'Pareja']]

data_clean.head()

data_useful.head()

n_data = data_clean.dropna(subset=[columnas]).copy()
n_data[['Emocion1', 'Emocion2', 'Emocion3', 'Emocion4', 'Emocion8', 'Emocion9',
                   'Emocion14', 'Emocion15', 'Emocion17', 'Emocion21', 'Emocion26', 'Emocion28',
                   'Emocion34',...


I am working on the file "Data Analysis.ipynb".
The first task is:
Se seleccionan las columnas que nos sirven para contestar la pregunta:
¿Qué grupo de edad sufre de más prejucios sexuales ? Adultos mayores o Jóvenes
Can you write Python code for it?

data_useful = data[['Sexo', 'Edad', 'Lugar_Origen', 'Estado_civil',
                   'Escolaridad', 'Orientacion_Religiosa', 'Practicante', 'Orientacion_Sexual',
                   'Pareja', 'Emocion1', 'Emocion2', 'Emocion3', 'Emocion4', 'Emocion8', 'Emocion9',
                   'Emocion14', 'Emocion15', 'Emocion17', 'Emocion21', 'Emocion26', 'Emocion28',
                   'Emocion34', 'Emocion36', 'Cognicion6', 'Cognicion7', 'Cognicion9', 'Cognicion10',
                   'Cognicion11', 'Cognicion15', 'Cognicion21', 'Cognicion30', 'Cognicion41', 'Cognicion47',
                   'Cognicion50', 'Cognicion54', 'Cognicion58', 'Cognicion65', 'Cognicion74', 'Cognicion75',
                   'Cognicion78', 'Conducta1', 'Conducta2', 'Conducta6', 'Conducta13', 'Conducta19', 'Conducta21',
                   'Conducta23', 'Conducta25', 'Conducta31', 'Conducta32', 'Conducta36', 'Conducta41', 'Conducta46',
                   'Conducta52', 'Creencia2', 'Creencia12', 'Creencia27', 'Creencia37', 'Creencia39', 'Creencia49']]

data_useful.columns.tolist()

data_useful.to_pickle('/home/sergio/Workspace/enc/data_usefull_enc.pkl')

data_useful = pd.read_pickle('/home/sergio/Workspace/enc/data_usefull_enc.pkl')


I am working on the file "Data Analysis.ipynb".
The first task is:
The above seal is the only one that stuck around, within some proximity, to a base station (BT5). From a technical standpoint, this is one example of a successful deployment of the tracker, in that we are able to detect a ping.
Can you write Python code for it?

print(list(tracker_dfs.keys())[4])
tracker_dfs[list(tracker_dfs.keys())[4]]

print(list(tracker_dfs.keys())[5])
tracker_dfs[list(tracker_dfs.keys())[5]]

print(list(tracker_dfs.keys())[6])
tracker_dfs[list(tracker_dfs.keys())[6]]

print(list(tracker_dfs.keys())[7])
tracker_dfs[list(tracker_dfs.keys())[7]]


I am working on the file "Data Analysis.ipynb".
The first task is:
Time filters for base stations
On each of the listed days, the Tiki base stations were set up at:

26 Jan: 12:37 pm
27 Jan: 9:42 am
28 Jan: 8:25 am
Can you write Python code for it?


I am working on the file "Data Analyst Nanodegree Project 1.ipynb".
The first task is:
At a glance, the difference distribution looks roughly normally-distributed (a manually fitted normal distribution is plotted for comparison), justifying the use of a paired t-test.
Now, perform the statistical test and report your results. What is your confidence level and your critical statistic value? Do you reject the null hypothesis or fail to reject it? Come to a conclusion in terms of the experiment task. Did the results match up with your expectations?
Can you write Python code for it?


I am working on the file "Data Analyst Nanodegree Project 1.ipynb".
The first task is:
Notice that not a single participant performed better on the incongruent task than on the congruent task (all differences are above zero).
Can you write Python code for it?

plt.plot(x,norm.pdf(x, 7.3, 5), 'g-', label="N(7.3, 5)")
sns.kdeplot(df['Incongruent']-df['Congruent'], label="KDE")
plt.title('Kernel Density Estimate of the Distribution\nof Differences Between the Two Conditions');
x = np.linspace(-10, 30, 1000)
plt.legend();


I am working on the file "Data Analyst Nanodegree Project 1.ipynb".
The first task is:
What is our independent variable? What is our dependent variable?
Independent variable: the text color condition (congruent or incongruent).
Dependent variable: the reaction time (the time it takes to name the ink color).
What is an appropriate set of hypotheses for this task? What kind of statistical test do you expect to perform? Justify your choices.
Null hypothesis: the reaction time for the congruent and incongruent conditions is the same.
$$\mu(\text{congruent}) = \mu(\text{incongruent})$$
Alternative hypothesis: the reaction time for the two conditons are different.
$$\mu(\text{congruent})  \neq \mu(\text{incongruent})$$
It's hard to imagine the reverse being true; that the reaction time would be faster for the incongruent condition. On the other hand, it is not a priori completely implausible that the  reaction times would be the same.
We will use the paired t-test to test whether the mean reaction times under the two conditions are equal or not. The t-test allows finding out whether the means of two sets of data are different.
We believe the use of a paired t-test is appropriate since the following conditions are met (see [1], Analysis checklist: Paired t test):
1. Are the differences distributed according to a Gaussian distribution?
As will be shown graphically in a later section, the differences are distributed close to normally.
2. Was the pairing effective?
The pairing was part of the design of the experiment; each pair represents trials from a single participant. This is an effective pairing method.
3. Are the pairs independent?
Each pair corresponds to trials by a different participant; they are independent.
4. Are you comparing exactly two groups?
Yes.
5. If you chose a one-tail P value, did you predict correctly?
We did not choose a one-tail P value (although we could have).
6. Do you care about differences or ratios?
We expect the control values and differences to be of the same order of magnitude, so differences should be fine.
Report some descriptive statistics regarding this dataset....
Can you write Python code for it?

IQR_congruent = df['Congruent'].quantile(0.75) - df['Congruent'].quantile(0.25)
IQR_incongruent = df['Incongruent'].quantile(0.75) - df['Incongruent'].quantile(0.25)
print(df['Congruent'].median(), df['Incongruent'].median(), IQR_congruent, IQR_incongruent)


I am working on the file "Data Analytics/.ipynb_checkpoints/Fitting curve-checkpoint.ipynb".
The first task is:
now let fit different consumer groups
Can you write Python code for it?

#b = self.base_demand
#m = self.max_demand
#y = b + m * (.5 * (1 + np.cos((x/6)*np.pi)))
#b = 603
#m = 3615

N = 84 # number of data points
t = np.linspace(0, 83, N)
#data = b + m*(.5 * (1 + np.cos((t/6)*np.pi))) + 100*np.random.randn(N) # create artificial data with noise
data = np.array(df['Via regionale netten'].values, dtype=np.float64)

guess_mean = np.mean(data)
guess_std = 2695.9075546 #2*np.std(data)/(2**0.5)
guess_phase = 0

# we'll use this to plot our first estimate. This might already be good enough for you
data_first_guess = guess_mean + guess_std*(.5 * (1 + np.cos((t/6)*np.pi + guess_phase)))

# Define the function to optimize, in this case, we want to minimize the difference
# between the actual data and our "guessed" parameters
optimize_func = lambda x: x[0]*(.5 * (1 + np.cos((t/6)*np.pi+x[1]))) + x[2] - data
est_std, est_phase, est_mean = leastsq(optimize_func, [guess_std, guess_phase, guess_mean])[0]

# recreate the fitted curve using the optimized parameters
data_fit = est_mean + est_std*(.5 * (1 + np.cos((t/6)*np.pi + est_phase)))

plt.plot(data, '.')
plt.plot(data_fit, label='after fitting')
plt.plot(data_first_guess, label='first guess')
plt.legend()
plt.show()
print('Via regionale netten')
print('max_demand: %s' %(est_std))
print('phase_shift: %s' %(est_phase))
print('base_demand: %s' %(est_mean))

#data = b + m*(.5 * (1 + np.cos((t/6)*np.pi))) + 100*np.random.randn(N) # create artificial data with noise
data = np.array(df['Elektriciteitscentrales'].values, dtype=np.float64)

guess_mean = np.mean(data)
guess_std = 3*np.std(data)/(2**0.5)
guess_phase = 0

# we'll use this to plot our first estimate. This might already be good enough for you
data_first_guess = guess_mean + guess_std*(.5 * (1 + np.cos((t/6)*np.pi + guess_phase)))

# Define the function to optimize, in this case, we want to minimize the difference
# between the actual data and our "guessed" parameters
optimize_func = lambda x: x[0]*(.5 * (1 + np.cos((t/6)*np.pi+x[1]))) + x[2] - data
est_std, est_phase, est_mean =...


I am working on the file "Data Analytics/.ipynb_checkpoints/Fitting curve-checkpoint.ipynb".
The first task is:
price forming
In order to estimate willingness to sell en willingness to buy we look at historical data over the past view years. We look at the DayAhead market at the TTF. Altough this data does not reflect real consumption necessarily
Can you write Python code for it?

inputexcel = 'TTFDA.xlsx'
outputexcel = 'pythonoutput.xlsx'

price = pd.read_excel(inputexcel, sheetname='Sheet1', index_col=0)
quantity = pd.read_excel(inputexcel, sheetname='Sheet2', index_col=0)

price.index = pd.to_datetime(price.index, format="%d-%m-%y")
quantity.index = pd.to_datetime(quantity.index, format="%d-%m-%y")

pq = pd.concat([price, quantity], axis=1, join_axes=[price.index])
pqna = pq.dropna()

year = np.arange(2008,2017,1)

coefficientyear = []

for i in year:
    x= pqna['Volume'].sort_index().ix["%s"%i]
    y= pqna['Last'].sort_index().ix["%s"%i]
    #plot the trendline
    plt.plot(x,y,'o')
    # calc the trendline
    z = np.polyfit(x, y, 1)
    p = np.poly1d(z)
    plt.plot(x,p(x),"r--", label="%s"%i)
    plt.xlabel("Volume")
    plt.ylabel("Price Euro per MWH")
    plt.title('%s: y=%.10fx+(%.10f)'%(i,z[0],z[1]))
    # plt.savefig('%s.png' %i)
    plt.show()
    # the line equation:
    print("y=%.10fx+(%.10f)"%(z[0],z[1]))
    # save the variables in a list
    coefficientyear.append([i, z[0], z[1]])

len(year)


I am working on the file "Data Analytics/Fitting curve.ipynb".
The first task is:
import data for our model
This is data imported from statline CBS webportal.
Can you write Python code for it?

importfile = 'CBS Statline Gas Usage.xlsx'
df = pd.read_excel(importfile, sheetname='Month', skiprows=1)
df.drop(['Onderwerpen_1', 'Onderwerpen_2', 'Perioden'], axis=1, inplace=True)

#df

# transpose
df = df.transpose()

# provide headers
new_header = df.iloc[0]
df = df[1:]
df.rename(columns = new_header, inplace=True)

#df.drop(['nan'], axis=0, inplace=True)
df

x = range(len(df.index))
df['Via regionale netten'].plot(figsize=(18,5))
plt.xticks(x, df.index, rotation='vertical')
plt.show()


I am working on the file "Data Analytics/Fitting curve.ipynb".
The first task is:
price forming
In order to estimate willingness to sell en willingness to buy we look at historical data over the past view years. We look at the DayAhead market at the TTF. Altough this data does not reflect real consumption necessarily
Can you write Python code for it?


I am working on the file "Data Collection.ipynb".
The first task is:
As as test, we will use AstroPy to get a nice image for the project page. We will use SIMBAD to find a random quasar.
Can you write Python code for it?

# Limit the number of results we get from our query.
Simbad.ROW_LIMIT = 1000

result = Simbad.query_criteria('region(box,180d +30d, 8d +8d)', otype='QSO')

result

# Choose a random quasar.
qnumber = np.random.randint(0, 1000)
# Get that quasar's coordinates, and format them for SkyCoordinate
resultRA = result[qnumber]['RA'].split()
RA = '%sh%sm%ss' % (resultRA[0], resultRA[1], resultRA[2])
resultDEC = result[qnumber]['DEC'].split()
DEC = '%sd%sm%ss' % (resultDEC[0], resultDEC[1], resultDEC[2])
# Convert to a SkyCoordinate
QCoord = SkyCoord(RA, DEC, frame='icrs')
QCoord


I am working on the file "Data Collection.ipynb".
The first task is:
Gathering Quasar Candidates
We will now use SIMBAD to identify quasar candidates for analysis with our trained model.
Can you write Python code for it?

# As with the Non-quasar data, we will stay in the 8h to 16h +0d to +60 footprint
# region of SDSS. Note that there are some regions in SDSS DR 12 and DR 13 outside
# of this range, but this range covers a majority of the footprint.
# Due to timeout issues from SIMBAD, we will use smaller regions
# of width 10d by +6d to gather the candidates.
QuasarCandidates = pd.DataFrame()
for i in range(12):  # Separate longitude into 12 segments of length 10d
    for j in range(10):  # Separate latitude into 10 segments of length 6d
        try:
            # For otype Q? are quasar candidates.
            result = Simbad.query_criteria(
                'region(box,' + str(125 + 10 * i) + 'd +'
                + str(3 + 6 * j) + 'd' + ', 5d +3d)', 'otype = Q?')
            QuasarCandidates = pd.concat(
                [QuasarCandidates, result.to_pandas()], axis=0)
            if (i * 10 + j) % 20 == 0:
                print('At attempt %s' % (i * 10 + j))
        except:
            print('Attempt Failed at i=%s and j=%s.' % (i, j))

QuasarCandidates.head()

QuasarCandidates.tail()

# Checking for duplication
QuasarCandidates[QuasarCandidates.duplicated()]

# Reindexing
QuasarCandidates.reset_index(inplace=True)
QuasarCandidates.drop('index', axis=1, inplace=True)

QuasarCandidates.tail()  # 5418 rows

# Saving a copy of the data to accompany the images.
QuasarCandidates.to_csv('QuasarCandidatesData.csv', index=False)

# Data manipulation to get Sky Coordinates for each entry.
# Note that SIMBAD gives the values separated as hours, minutes, seconds for RA and degrees, minutes, seconds for Dec
QuasarCandidateLocs = pd.concat([QuasarCandidates['RA'].apply(RAtoICRS),
                                 QuasarCandidates['DEC'].apply(DECtoICRS)], axis=1)
QuasarCandidateLocs['Coords'] = QuasarCandidateLocs[['RA', 'DEC']].apply(
    lambda x: SkyCoord(x[0], x[1], frame='icrs'), axis=1)

QuasarCandidateLocs.head()

# We will now download these images from SDSS DR12
for i in range(5418):
   ...


I am working on the file "Data Conversion.ipynb".
The first task is:
Given a hex/octal/binary number, how do you get the ascii/utf-8 string back

convert everything back to an int using int() and specifying the base
convert ints back into ordinal using chr()
Can you write Python code for it?


I am working on the file "Data Conversion.ipynb".
The first task is:
Lets convert a string of characters into its unicode value for each character using format and join method
Can you write Python code for it?


I am working on the file "Data Conversion.ipynb".
The first task is:
What about a text string into hex using format and join
converting each character into its unicode ordinal value then format that result into a hex while joining each character into a string thats space seperated each value
Can you write Python code for it?

' '.join([format(ord(x), 'x') for x in s1])


I am working on the file "Data Conversion.ipynb".
The first task is:
What about a text string into octal
Can you write Python code for it?

' '.join([format(ord(x), 'o') for x in s1])


I am working on the file "Data Conversion.ipynb".
The first task is:
chr()
returns a unicode string of one character with ordinal i
Can you write Python code for it?


I am working on the file "Data Conversion.ipynb".
The first task is:
hex()
Takes an integer and returns hexadecimal representation
Can you write Python code for it?


I am working on the file "Data Conversion.ipynb".
The first task is:
int
takes a number or string representing a number and a base and converts into an integer
When specifying a base, the first argument must be a string
With int you specify the base of the string, not the base your trying to convert to converts everything to ints
Can you write Python code for it?

int('0xDEADBEEF', 16)

int('DEADBEEF', 16)

int('DEADBEEF')

int('0oDEADBEEF', 16)


I am working on the file "Data Exploration/Challenge - Cleaning Data.ipynb".
The first task is:
4: Filter Out The Bad Years
Since the data was collected from a community site, where most of the contributions came from individual users, there's room for errors to surface in the dataset. If you plot a histogram of the values in the Year column, which describes the year each Avenger was introduced, you'll immediately notice some oddities. There are quite a few Avengers who look like they were introduced in 1900, which we know is a little fishy. The Avengers weren't introduced in the comic series until the 1960's!
This is obviously a mistake in the data and you should remove all Avengers before 1960 from the DataFrame.
Can you write Python code for it?

true_avengers = avengers[avengers['Year'] >= 1960]

print('All: ' + str(len(avengers.index)))
print('After 1960: ' + str(len(true_avengers.index)))


I am working on the file "Data Exploration/Challenge - Cleaning Data.ipynb".
The first task is:
5: Consolidating Deaths
We are interested in the number of total deaths each character experienced and we'd like a field containing that distilled information. Right now, there are 5 fields (Death1 to Death5) that each contain a binary value representing if a superhero experienced that death or not. For example, a superhero can experience Death1, then Death2, etc. until they were no longer brought back to life by the writers.
We'd like to coalesce that information into just one field so we can do numerical analysis more easily.
Can you write Python code for it?

columns = ['Death1', 'Death2', 'Death3', 'Death4', 'Death5']

def death_count(row):
  death = 0
  for column in columns:
    if row[column] == 'YES':
      death += 1
  return death
true_avengers['Deaths'] = true_avengers[columns].apply(death_count, axis=1)
true_avengers['Deaths'].head()
# true_avengers[columns].head()


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
2°) Exponential Form (different parameters for scaling)
Can you write Python code for it?

params_exp = (15, 0.05)
p = (p_x0, p_x1, p_x2)
exp_forward, exp_myopic = contraction_mapping(S=70, p=p, MF=exp_cost, params=params_exp, beta = 0.75)
pchoice = exp_forward.T[0]
plt.plot(exp_forward.T[0])
plt.plot(exp_myopic.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking", "Myopic"])
plt.xlim(20, 65)
plt.title("Action probabilities of the Agent (Exp. Spec.)");

n_bus = 1000
bus_array = np.hstack((np.linspace(1, n_bus, n_bus).reshape(-1, 1), np.zeros((n_bus, 3))))
for i in range(n_periods):
    bus_array = transition(bus_array, pchoice)
exp_dataframe = pd.DataFrame(bus_array, columns=["Identifier", "Choice", "Mileage", "State"])
choice_freq = exp_dataframe.loc[:, ["Choice", "State"]].groupby(["State"]).mean().sort_values("Choice").reset_index()
plt.plot(choice_freq["State"], 1-choice_freq["Choice"])
plt.plot(pchoice, "--")
plt.xlabel("Observed state of the bus")
plt.ylabel("Probability of maintenance")
plt.legend(["Observed behavior", "True choice probabilities"])
plt.title("Behavior of forward-looking agent (Exp. Spec.)")
plt.xlim(20, 65)
plt.ylim(-0.05, 1.05);
exp_dataframe.loc[n_bus+1:,["Choice", "State"]].to_csv("Datasets\Exp_Dataset.csv", index=False)


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
2°) Fitting the other cost data
In this section, we fit the data generated by the quadratic, exponential, and logarithmic cost function and recover the parameters $\text{RC}$ and $\theta_{11}$ and $\theta_{12}$. We do not illustrate the impact of misspecification for those cost functions.
A. Fitting the true quadratic cost function
We start by initializing a DynamicLogit instance, and fit the likelihood function to the data.
Recall that in the quadratic model:
$$\text{RC} = 20$$
$$\theta_{11} = 0.5$$
$$\theta_{12} = 0.01$$
Can you write Python code for it?

quad_to_quad = DynamicLogit(quad_dataframe, "Choice", "State", p, quad_cost, npars=3)
quad_to_quad.fit_likelihood(x0=[20, 1, 1])


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
2°) Styling of the notebook
Can you write Python code for it?

%matplotlib inline
figsize(12, 8)
pd.set_option("display.precision", 3)
np.set_printoptions(precision=4)
np.set_printoptions(suppress=True)
s = json.load( open("./styles/bmh_matplotlibrc.json") )
mpl.rcParams.update(s)

def css_styling():
    styles = open("./styles/custom.css", "r").read()
    return HTML(styles)
css_styling()


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
3°) The bus replacement dataset.
A. Initialization
First, we decide on a number of bus, that we initialize in state 0.
We treat the dataset as an N_Bus x 4 array, containing:
* The identifier of the buses
* The choice made for the buses at this time
* The mileage
* The states
Can you write Python code for it?

n_bus = 1000
bus_array = np.hstack((np.linspace(1, n_bus, n_bus).reshape(-1, 1), np.zeros((n_bus, 3))))


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
B. Definition of the cost function
Can you write Python code for it?

def myopic_costs(S, MF, params, p):
    """
    This function computes the myopic expected cost associated with each decision for each state,
    and returns an array of state/decision costs.

    Takes:
        * An integer S, describing the possible states of the bus
        * A maintenance cost function MF, which takes a vector of parameters and a `state' argument
        * A vector params, to be supplied to the maintenance cost function MF. The first element of
          the vector is the replacement cost rc.
        * A (3x1) vector p describing the state transitions probabilities

    Returns:
        * A (Nx2) array containing the maintenance and replacement costs for the N possible states of the bus
    """
    rc = params[0]
    maint_cost = [MF(s, params[1:]) for s in range(0, S)]
    repl_cost = [rc for state in range(0, S)]
    return np.vstack((maint_cost, repl_cost)).T


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
B. Fitting the exponential cost function
Recall that in the exponential model:
$$\text{RC} = 15$$
$$\theta_{11} = 0.05$$
Can you write Python code for it?

exp_to_exp = DynamicLogit(exp_dataframe, "Choice", "State", p, exp_cost, npars=2)
exp_to_exp.fit_likelihood()


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
C. Definition of the choice probabilities, as a function of an array of costs
Can you write Python code for it?

def choice_prob(cost_array):
    """
    Returns the probability of each choice, conditional on an array of state/decision costs.
    """
    S = cost_array.shape[0]
    cost = cost_array - cost_array.min(1).reshape(S, -1)
    util = np.exp(-cost)
    pchoice = util/(np.sum(util, 1).reshape(S, -1))
    return pchoice


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
C. Fitting an incorrect exponential cost model to the data
We initialize a third DynamicLogit instance, this time with our exponential cost specification
Can you write Python code for it?

exp_to_lin = DynamicLogit(lin_dataframe, "Choice", "State", p, exp_cost, npars=2)
exp_to_lin.fit_likelihood(bounds=[(0, None), (0.1, None)])


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
C. Fitting the logarithmic cost function
Recall that in the logarithmic model:
$$\text{RC} = 4$$
$$\theta_{11} = 1$$
$$\theta_{12} = 10$$
Can you write Python code for it?

log_to_log = DynamicLogit(log_dataframe, "Choice", "State", p, log_cost, npars=3)
log_to_log.fit_likelihood()


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
C. Generation of the full dataset
The complete dataset is generated by making the agent take a serie of decisions. Here, we assume that the agent takes 1000 consecutive decisions. The total number of data points thus generated will be equal to 100,000 (1000 buses and 100 time periods).
Can you write Python code for it?

n_periods = 100

for i in range(n_periods):
    bus_array = transition(bus_array, pchoice)


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
C. Plot of the data
Can you write Python code for it?

choice_freq = lin_dataframe.loc[:, ["Choice", "State"]].groupby(["State"]).mean().sort_values("Choice").reset_index()
plt.plot(choice_freq["State"], 1-choice_freq["Choice"])
plt.plot(pchoice, "--")
plt.xlabel("Observed state of the bus")
plt.ylabel("Probability of maintenance")
plt.legend(["Observed behavior", "True choice probabilities"])
plt.title("Behavior of forward-looking agent (Lin. Spec.)")
plt.xlim(5, 25)
plt.ylim(-0.05, 1.05);


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
Computing the state transitions probabilities:
Can you write Python code for it?

p_x0 = mileage_dist.cdf(5000)
p_x1 = mileage_dist.cdf(10000) - p_x0
p_x2 = 1 - p_x1 - p_x0
p = (p_x0, p_x1, p_x2)


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
D. Fitting an incorrect log cost function to the data.
We initialize a fourth DynamicLogit instance, this time with our log cost specification
Can you write Python code for it?

log_to_lin = DynamicLogit(lin_dataframe, "Choice", "State", p, log_cost, npars=3)
log_to_lin.fit_likelihood(x0=[10, 10, 10])


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
Finally, the array is stored into a Dataframe, to make plot and data exports more convenient.
Can you write Python code for it?

lin_dataframe = pd.DataFrame(bus_array, columns=["Identifier", "Choice", "Mileage", "State"])


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
II. Notations and definitions
1°) Number of states and state transitions
In this paper, the agent observes each bus at a given time $t$, and makes a decision regarding the replacement of the engine or its maintenance as a function of the observed state of the engine (i.e. its mileage).
As in the original paper, we discretize the number of states into intervals of length 5000, and define a transition density which governs the evolution of mileage from one time period to another:
$$ p(x_{t+1} | x_{t} , i_{t} , \theta_{3})  = \left{
  \begin{array}{l l}
    g(x_{t+1} - x_{t}, \theta_{3})   & \quad \text{if } i_t = 0\
    g(x_{t+1} - 0, \theta_{3}) & \quad \text{if } i_t = 1
  \end{array} \right.$$
Here, the function g is defined by a multinomial distribution on the set {0, 1, 2}. In other words, the probability that a bus' mileage will increase by a value x between now and the next maintenance decision is:
* $\theta_{31}$ if $x \in [0, 5000[$
* $\theta_{32}$ if $x \in [5000, 10,000[$
* $\theta_{33} = 1 - (\theta_{31} + \theta_{32})$ if $x \in [10,000, +\infty[$
2°) Decision criteria
The utility function of the decision maker for a single time period is:
$$ u(x_{t}, i , \theta_{1})  = \left{
  \begin{array}{l l}
    \qquad \enspace -c(x_t, \theta_1) + \epsilon_t(0)& \quad \text{if } i = 0\
    -RC -c(0, \theta_1) + \epsilon_t(1) & \quad \text{if } i = 1
  \end{array} \right. \quad \text{(Errors are I.I.D. standard Gumbel)}$$
The agent will chose the decision which maximizes its utility. If the agent is forward looking (i.e. if $\beta \neq 0$), he does not only maximize his utility in the present time period, but also his future utility discounted by $\beta$.
Since $\beta$ cannot be estimated without knowing the utility function, we will later set $\beta$ to its true value.
3°) Cost function
A functional form has to be imposed on the cost function $c(x_t)$.
In this recreation, we assume that the replacement cost is a constant $\text{RC}$, and allow the maintenance cost $\text{MC}(s, \theta_1)$ to be a linear,...
Can you write Python code for it?

lower, upper = 0, 15000
mu, sigma = 6000, 4000
mileage_dist = stats.truncnorm((lower - mu) / sigma, (upper - mu) / sigma, loc=mu, scale=sigma)


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
IV. Generation of simulated data (using a linear cost specification)
1°) True coefficients
Here, we use the same coefficients as the one described earlier:
* Cost and discount coefficients:
    * $RC = 20$
    * $\theta_{11} = 0.5$
    * $\theta_{12} = 0.01$
    * $\beta = 0.75$
* State transition probabilities: $(\text{Mileage}{t+1} - \text{Mileage}{t}) \sim \mathcal{N}_{Trunc}(6000, 4000)$, such that:
    * $P(x_t = 0) =  0.36$
    * $P(x_t = 1) =  0.48$
    * $P(x_t = 2) =  0.16$
Finally, we assume that there are 70 discrete states for the bus' mileage (i.e. no bus has a mileage greater than 345,000 in the dataset).
2°) Choice probabilities, as a function of the bus' state:
Using the Contraction Mapping algorithm implemented earlier, we can obtain the probabibility of maintenance as a function of the bus' state for a forward-looking agent and a myopic agent.
Can you write Python code for it?

params_lin = (rc, theta1_1)
p = (p_x0, p_x1, p_x2)
lin_forward, lin_myopic = contraction_mapping(S=70, p=p, MF=lin_cost, params=params_lin, beta = 0.75)
pchoice = lin_forward.T[0]


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
In this case, both the myopic and forward looking choice probabilities are very different from the true ones. This shows the importance of using a correctly specified functional form for the model!
Can you write Python code for it?

plt.plot(lin_forward.T[0])
plt.plot(lin_myopic.T[0])
plt.plot(log_forward_app.T[0])
plt.plot(log_myopic_app.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking (Lin)", "Myopic (Lin)", "Forward-Looking (Log)", "Myopic (Log)"])
plt.xlim(0, 70)
plt.title("Action probabilities of the Agent (Log. Cost Specification)");


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
Plotting the state transitions probabilities:
Can you write Python code for it?

x = np.linspace(0,150001,150001)
y = mileage_dist.pdf(x)
plt.plot(x, y)
plt.title("State transitions probabilities", fontsize=25)
plt.xlabel("Mileage")
plt.ylabel("Probabilies")
plt.text(750, 0.00001, "$P(x_t = 0) = {:.2f}$".format(p_x0), fontsize=20, weight="bold")
plt.text(5750, 0.00001, "$P(x_t = 1) = {:.2f}$".format(p_x1), fontsize=20, weight="bold")
plt.text(10500, 0.00001, "$P(x_t = 2) =  {:.2f}$".format(p_x2), fontsize=20, weight="bold")
plt.fill_between(x, y, where=x>=0, facecolor='#F6E3CE', interpolate=True)
plt.fill_between(x, y, where=x>=5000, facecolor='#CEF6CE', interpolate=True)
plt.fill_between(x, y, where=x>=10000, facecolor='#CECEF6', interpolate=True)
plt.grid(False)
plt.xlim(0, 15000);


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
The myopic choice probabilities look different, but the forward-looking ones fit the data quite well.
Can you write Python code for it?

exp_forward_app, exp_myopic_app = contraction_mapping(S=70, p=p,
                                                      MF=exp_cost, params=exp_to_lin.get_parameters(),
                                                      beta = 0.75)

plt.plot(lin_forward.T[0])
plt.plot(lin_myopic.T[0])
plt.plot(exp_forward_app.T[0])
plt.plot(exp_myopic_app.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking (Lin)", "Myopic (Lin)", "Forward-Looking (Exp)", "Myopic (Exp)"])
plt.xlim(0, 70)
plt.title("Action probabilities of the Agent (Exp. Cost Specification)");


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
The results:
Can you write Python code for it?

quad_to_quad.print_parameters()

quad_forward_app, quad_myopic_app = contraction_mapping(S=70, p=p, MF=quad_cost,
                                                      params=quad_to_quad.get_parameters(), beta = 0.75)

plt.plot(quad_forward.T[0])
plt.plot(quad_myopic.T[0])
plt.plot(quad_forward_app.T[0])
plt.plot(quad_myopic_app.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking (Quad)", "Myopic (Quad)", "Forward-Looking (Quad, Est.)", "Myopic (Quad, Est.)"])
plt.xlim(5, 40)
plt.title("Action probabilities of the Agent (Quad. Cost Specification)");


log_to_lin.print_parameters()

log_forward_app, log_myopic_app = contraction_mapping(S=70, p=p, MF=log_cost,
                                                      params=log_to_lin.get_parameters(), beta = 0.75)


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
V. Generation of simulated data (using other cost specifications)
1°) Quadratic form
Can you write Python code for it?

params_quad = (rc, theta1_1, theta1_2)
p = (p_x0, p_x1, p_x2)
quad_forward, quad_myopic = contraction_mapping(S=70, p=p, MF=quad_cost, params=params_quad, beta = 0.75)
pchoice = quad_forward.T[0]
plt.plot(quad_forward.T[0])
plt.plot(quad_myopic.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking", "Myopic"])
plt.xlim(5, 35)
plt.title("Action probabilities of the Agent (Quad. Spec.)");

n_bus = 1000
bus_array = np.hstack((np.linspace(1, n_bus, n_bus).reshape(-1, 1), np.zeros((n_bus, 3))))
for i in range(n_periods):
    bus_array = transition(bus_array, pchoice)
quad_dataframe = pd.DataFrame(bus_array, columns=["Identifier", "Choice", "Mileage", "State"])
choice_freq = quad_dataframe.loc[:, ["Choice", "State"]].groupby(["State"]).mean().sort_values("Choice").reset_index()
plt.plot(choice_freq["State"], 1-choice_freq["Choice"])
plt.plot(pchoice, "--")
plt.xlabel("Observed state of the bus")
plt.ylabel("Probability of maintenance")
plt.legend(["Observed behavior", "True choice probabilities"])
plt.title("Behavior of forward-looking agent (Quad. Spec.)")
plt.xlim(5, 20)
plt.ylim(-0.05, 1.05);
quad_dataframe.loc[n_bus+1:,["Choice", "State"]].to_csv("Datasets\Quad_Dataset.csv", index=False)


I am working on the file "Data Generation and Likelihood Fit.ipynb".
The first task is:
VI. Estimation
1°) Dynamic Utility Logit Model
In this section, we implement a Dynamic Utility Logit model, which will be our workbench for the estimation of the data. We wrap in this class the functions defined earlier (to avoid global references) and refactor some portions of the code (to make the model more flexible). The maintenance cost function is kept outside of the scope of the class, and must be supplied at the initialization of the object as an argument.
Can you write Python code for it?

class DynamicLogit(object):
    def __init__(self, data, Y, X, p, MF, npars):
        """
        A statistics workbench used to evaluate the cost parameters underlying
        a bus replacement pattern by a forward-looking agent.

        Takes:
            * Data: a Pandas dataframe, which contains:
                -Y: the name of the column containing the dummy exogenous
                    variable (here, the choice)
                -X: the name of the column containing the endogenous variable
                    (here, the state of the bus)

            * p: The state-transition vector of endogenous variable.
                    For instance, p = [0, 0.6, 0.4] means that the bus will
                    transition to the next mileage state with probability 0.6,
                    and to the second next mileage state with probability 0.4.

            * MF: A function passed as an argument, which is the functional
                  form for the maintenance cost. This function must accept as
                  a first argument a state s, and as a second argument a vector
                  of parameters.

            * npars: The number of parameters to evalutate (i.e. the number of
                     parameters of the maintenance cost function, plus 1 for
                     the replacement cost)
        """

        self.endog = data.loc[:, Y].values
        self.exog = data.loc[:, X].values

        self.N = self.endog.shape[0]
        self.S = int(self.exog.max()*2) # Assumes that the true maximum number
                                         # states is twice the maximum observed
                                         # state.

        # Check that p is a correct vector of probabilities (i.e. sums to 1)
        p = np.array(p)
        if p.sum() == 1:
            self.p = p
        else:
            raise ValueError(("The probability of state transitions should add"
                              " up to 1!"))

  ...