I am working on the file "DNN_Tutorial.ipynb".
The first task is:
Try training a linear classifier on the Even-Odd labels: train_eo!
Using a Deep Nets library
If you're just starting off with deep nets and want to quickly try them on a dataset it is probably easiest to start with an existing library rather than writing your own. There are now a bunch of different libraries written for Python. We'll be using Keras which is designed to be easy to use. In matlab, there is the Neural Network Toolbox.
Keras documentation can be found here:
http://keras.io/
We'll do the next most complicated network comparer to linear regression: a two layer network!
Can you write Python code for it?

# Import things from Keras Library
from keras.models import Sequential
from keras.layers import Dense, Dropout, Activation, Flatten
from keras.layers.convolutional import Convolution2D
from keras.regularizers import l2
from keras.optimizers import SGD, Adam, RMSprop
from keras.utils import np_utils


I am working on the file "DNN_Tutorial.ipynb".
The first task is:
Visualizing the filters
Linear classifier
Can you write Python code for it?

W = linear_classifier.coef_
ims = tri(W, (im_size, im_size), (1, 10), (1,1))
plt.imshow(ims, cmap='gray', interpolation='nearest')
plt.axis('off')


I am working on the file "DOC/Capabilities/AddingDataToStellab.ipynb".
The first task is:
The goal is to add your data to STELLAB to produce plots such as the plot below:
Can you write Python code for it?

s1=st.stellab()
xaxis='[Fe/H]'
yaxis='[O/Fe]'
s1.plot_spectro(fig=1,xaxis=xaxis,galaxy='carina')
plt.xlim(-4.5,1),plt.ylim(-1.5,1.5)


I am working on the file "DOC/Capabilities/Delayed_extra_sources.ipynb".
The first task is:
2.2 Two deyaled extra sources
Can you write Python code for it?

# Create the DTD and yields information for the second extra source
# =================================================================

# Event rate [yr^-1] as a function of time [yr].
t2 = [1.4e9, 1.6e9, 1.8e9]
R2 = [4.0, 1.0, 4.0]

# Build the input DTD array
dtd2 = []
for i in range(0,len(t2)):
    dtd2.append([t2[i], R2[i]])

# Add the DTD array in the delayed_extra_dtd array.
delayed_extra_dtd = [[dtd],[dtd2]]

# Define the total number of event per unit of Msun formed.  This will normalize the DTD.
delayed_extra_dtd_norm = [[1.0e-2],[5.0e-3]]

# Define the yields path for the extra source
delayed_extra_yields = ['yield_tables/r_process_arnould_2007.txt','yield_tables/r_process_arnould_2007.txt']

# Define the total mass ejected by an extra source
delayed_extra_yields_norm = [[1.0e-3],[2.0e-3]]

# Run SYGMA, one SSP with a total mass of 1 Msun at Z = 0.02
mgal = 1.0
s2 = sygma.sygma(iniZ=0.02, delayed_extra_dtd=delayed_extra_dtd, delayed_extra_dtd_norm=delayed_extra_dtd_norm,\
        delayed_extra_yields=delayed_extra_yields, delayed_extra_yields_norm=delayed_extra_yields_norm, mgal=mgal,\
        dt=1e8, special_timesteps=-1, tend=1.1*t[-1])

%matplotlib nbagg
plt.plot(s2.history.age[1:], np.array(s2.delayed_extra_numbers[0])/s2.history.timesteps)
plt.plot(s2.history.age[1:], np.array(s2.delayed_extra_numbers[1])/s2.history.timesteps)
plt.xlim(0, 1.1*t[-1])
plt.xlabel('Time [yr]', fontsize=12)
plt.ylabel('Rate [event yr$^{-1}$]', fontsize=12)


I am working on the file "DOC/Capabilities/Delayed_extra_sources.ipynb".
The first task is:
Let's plot the DTD
Can you write Python code for it?

%matplotlib nbagg
plt.plot(s1.history.age[1:], np.array(s1.delayed_extra_numbers[0])/s1.history.timesteps)
plt.xlim(0, 1.1*t[-1])
plt.xlabel('Time [yr]', fontsize=12)
plt.ylabel('Rate [event yr$^{-1}$]', fontsize=12)


I am working on the file "DOC/Capabilities/Delayed_extra_sources.ipynb".
The first task is:
Let's test the total number of events and the total mass ejected by those events.
Can you write Python code for it?

# Predicted number of events
N_pred = delayed_extra_dtd_norm[0][0] * mgal

# Predicted mass ejected by events
M_ej_pred = N_pred * delayed_extra_yields_norm[0][0]

# Calculated number of events
N_sim = sum(s1.delayed_extra_numbers[0])

# Calculated mass ejected by events
M_ej_sim = sum(s1.ymgal_delayed_extra[0][-1])

# Print the test
print ('The following numbers should be 1.0')
print ('   Number of events (predicted/calculated):', N_pred / N_sim)
print ('   Mass ejected by events (predicted/calculated):', M_ej_pred / M_ej_sim)


I am working on the file "DOC/Capabilities/Delayed_extra_sources_Z_dependent.ipynb".
The first task is:
Defining Z-dependent DTD for a single enrichment source
The following is an example with 3 metallicites. The GCE code reads these metallicities directly from the yields table, they are not defined as input parameter. Using the z_dependent_DTD.txt yields table, the metallicities are 0.02, 0.01, and 0.001.
Can you write Python code for it?

# DTD for Z = 0.02
# ================
t_0_02 = [0.0, 1.0e8, 1.0e9]
R_0_02 = [1.0, 3.0, 2.0]
dtd_0_02 = []
for i in range(0,len(t_0_02)):
    dtd_0_02.append([t_0_02[i], R_0_02[i]])

# DTD for Z = 0.01
# ================
t_0_01 = [0.0, 1.0e8, 1.0e9, 1.5e9]
R_0_01 = [3.0, 1.0, 0.5, 0.01]
dtd_0_01 = []
for i in range(0,len(t_0_01)):
    dtd_0_01.append([t_0_01[i], R_0_01[i]])

# DTD for Z = 0.001
# =================
t_0_001 = [0.0, 3.0e7, 6.0e7, 9.0e7, 3.0e8]
R_0_001 = [1.0, 1.0, 1.0, 1.0, 1.0]
dtd_0_001 = []
for i in range(0,len(t_0_001)):
    dtd_0_001.append([t_0_001[i], R_0_001[i]])

# Keep track of the highest time
t_max = max([t_0_02[-1], t_0_01[-1], t_0_001[-1]])

# Generate the NuPyCEE input parameters
# =====================================

# Add the DTD array in the delayed_extra_dtd array.
delayed_extra_dtd = [[dtd_0_02, dtd_0_01, dtd_0_001]]
# delayed_extra_dtd[ source index ][ Z index ]

# Define the total number of event per unit of Msun formed.  This will normalize the DTD.
delayed_extra_dtd_norm = [[1.0e-1, 1.0e-2, 1.0e-3]]
# delayed_extra_dtd_norm[ source index ][ Z index ]

# Define the yields path for the extra source
delayed_extra_yields = ['yield_tables/z_dependent_DTD.txt']
# delayed_extra_yields[ source index ] --> Z-dependence is in the yields table

# Define the total mass ejected by an extra source
# Here we are going to assume that there is no multiplication factor applied to the yields
delayed_extra_yields_norm = [[1.0, 1.0, 1.0]]
# delayed_extra_yields_norm[ source index ][ Z index ]

# NOTE: You can define if you want to interpolate DTDs and yields in the log-log space
# log(quantity) = A*log(Z) + B
# It is worth to try both options when you do GCE to see the impact !!
delayed_extra_log = False
delayed_extra_yields_log_int = False

# Define the arguments to be thrown in the NuPyCEE codes
kwargs = {"delayed_extra_dtd":delayed_extra_dtd, "delayed_extra_dtd_norm":delayed_extra_dtd_norm,\
          "delayed_extra_yields":delayed_extra_yields,...


I am working on the file "DOC/Capabilities/Delayed_extra_sources_Z_dependent.ipynb".
The first task is:
Quantitative test with SYGMA
Can you write Python code for it?

# Set all simple stellar population to a total mass of 1 Msun
mgal = 1.0

# Run SYGMA, one SSP with a total mass of 1 Msun at Z = 0.02
s_0_02 = sygma.sygma(iniZ=0.02, mgal=mgal, special_timesteps=120, tend=1.1*t_0_02[-1], **kwargs)

# Run SYGMA, one SSP with a total mass of 1 Msun at Z = 0.01
s_0_01 = sygma.sygma(iniZ=0.01, mgal=mgal, special_timesteps=120, tend=1.1*t_0_01[-1], **kwargs)

# Run SYGMA, one SSP with a total mass of 1 Msun at Z = 0.001
s_0_001 = sygma.sygma(iniZ=0.001, mgal=mgal, special_timesteps=120, tend=1.1*t_0_001[-1], **kwargs)

# Function to test SYGMA for the ejected mass and number of event for a given metallicity
def test_sygma_quantity(i_Z, s, i_source=0):

    # Predicted number of events
    N_pred = delayed_extra_dtd_norm[i_source][i_Z] * mgal

    # Predicted mass ejected by events
    yields_from_table = sum(s.ytables_delayed_extra[i_source].get(Z=s.iniZ,quantity="Yields"))
    M_ej_pred = N_pred * yields_from_table * delayed_extra_yields_norm[i_source][i_Z]

    # Calculated number of events, from SYGMA
    N_sim = np.sum(s.delayed_extra_numbers[i_source])

    # Calculated mass ejected by events, from SYGMA
    M_ej_sim = np.sum(s.ymgal_delayed_extra[i_source][-1])

    # Print the test
    print ('The following numbers should be 1.0')
    print ('   Number of events (predicted/calculated):', N_pred / N_sim)
    print ('   Mass ejected by events (predicted/calculated):', M_ej_pred / M_ej_sim)

# Print the test results for Z = 0.02
print("SYGMA (Z = 0.02)")
print("================")
test_sygma_quantity(0, s_0_02)
print()

# Print the test results for Z = 0.01
print("SYGMA (Z = 0.01)")
print("================")
test_sygma_quantity(1, s_0_01)
print()

# Print the test results for Z = 0.001
print("SYGMA (Z = 0.001)")
print("================")
test_sygma_quantity(2, s_0_001)
print()


I am working on the file "DOC/Capabilities/Delayed_extra_sources_Z_dependent.ipynb".
The first task is:
Test with the GCE OMEGA code
Can you write Python code for it?

# Run OMEGA with the Z-dependent extra-delayed source
o = omega.omega(special_timesteps=120, mgal=1e10, **kwargs)

# Plot the evolution of metallicity
%matplotlib nbagg
plt.plot(o.history.age, o.history.metallicity, color="k")
plt.plot([0,13e9], [0.02,0.02],   ls="--", color="C0", label="Z=0.02")
plt.plot([0,13e9], [0.01,0.01],   ls="--", color="C1", label="Z=0.01")
plt.plot([0,13e9], [0.001,0.001], ls="--", color="C2", label="Z=0.001")

# Visual
plt.ylabel("Metallicity (Z)")
plt.xlabel("Time [yr]")
plt.yscale("log")
plt.title("OMEGA", fontsize=12)
plt.ylim(1e-4,1e-1)
plt.legend(fontsize=12)
plt.tight_layout()

# Plot the evolution of W-180 in the interstellar medium
%matplotlib nbagg
o.plot_mass(specie="W-180", markevery=10000)
plt.xlabel("Time [yr]")
plt.ylabel("Interstellar medium abundance")
plt.title("OMEGA", fontsize=12)
plt.xscale("linear")


I am working on the file "DOC/Capabilities/Fast_chem_evol.ipynb".
The first task is:
Comparison Between the Original and Fast Versions
Can you write Python code for it?

%matplotlib nbagg
o_ori.plot_spectro(color='b',  label='Original')
o_fast.plot_spectro(color='g', shape='--', label='Fast')
plt.xscale('log')

%matplotlib nbagg
o_ori.plot_mass( specie='Mg', color='b', markevery=100000, label='Original')
o_fast.plot_mass(specie='Mg', color='g',  markevery=100,   shape='--', label='Fast')
plt.ylabel('Mass of Mg [Msun]')

%matplotlib nbagg
o_ori.plot_spectro( xaxis='[O/H]', yaxis='[Ca/O]', color='b', \
                    markevery=100000, label='Original')
o_fast.plot_spectro(xaxis='[O/H]', yaxis='[Ca/O]', color='g', \
                    shape='--', markevery=300, label='Fast')
plt.ylim(-2.0,1.0)


I am working on the file "DOC/Capabilities/Fitting_curves_to_STELLAB_data.ipynb".
The first task is:
Plot Results
Can you write Python code for it?

# Define the line colours and styles
lines = ['k-','m-','g-','b-','r-','c-','y-','k--','m--','g--','b--','r--','c--']

# Plot the observed data
%matplotlib inline
pl.plot(x,y,'mo',label='data')

# Plot the regression curves for each polynomial order
# Symbols and labels are automatically set
x_fit = np.arange(min(x)-0.01, max(x)+0.01, 0.001)
for i in range(pol_order):
    coeffs = np.polyfit(x, y, i+1)
    p = np.poly1d(coeffs)
    pl.plot(x_fit,p(x_fit),lines[i],label='polyfit ord '+ str(i+1)+', R$^2$ adj='+'%2.3f' %results_adj[i])

# Set labels and digits size
pl.ylabel(y_label, fontsize=15.)
pl.xlabel(x_label, fontsize=15.)
matplotlib.rcParams.update({'font.size': 14.0})

# Legend outside the plot box
pl.legend(numpoints=1, bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0., fontsize=14)
pl.show()

# Create the polynomial order array
order = np.arange(1,pol_order+1,1)

# Plot R-squared and its maximal value as a function of polynomial order
pl.plot(order, results, 'k-o', label='R$^2$')
pl.plot(order[np.argmax(results)], results[np.argmax(results)],'k-o',markersize=15.)

# Plot R-squared adjusted and its maximal value as a function of polynomial order
#pl.plot(order,results_adj,'r-s',label='R$^2$ adj')
#pl.plot(order[np.argmax(results_adj)],results_adj[np.argmax(results_adj)],'r-s',markersize=15.)

# Set the X-axis range and xticks (only integers)
pl.xlim(min(order)-0.5, max(order)+0.5)
pl.xticks(np.arange(min(order), max(order)+1, 1.0))

# Set labels and digits size
pl.ylabel('R$^2$', fontsize=15.)
pl.xlabel('Order polynomial', fontsize=15.)
matplotlib.rcParams.update({'font.size': 14.0})

# Legend
pl.legend(numpoints=1, loc='upper left', fontsize=14)
pl.show()


I am working on the file "DOC/Capabilities/Including_radioactive_isotopes.ipynb".
The first task is:
1. Input Parameters
The inputs that need to be provided to activate the radioactive option are:

the list of selected radioactive isotopes,
the radioactive yield tables.

The list of isotopes is declared in the yield_tables/decay_info.txt file and can be modified prior any simulation.  The radioactive yields are found (or need to be added) in the yield_tables/ folder.  Each stable yield table can have their associated radioactive yield table:

Massive and AGB stars
Stable isotopes: table
Radioactive isotopes: table_radio
Type Ia supernovae
Stable isotopes: sn1a_table
Radioactive isotopes: sn1a_table_radio
Neutron star mergers
Stable isotopes: nsmerger_table
Radioactive isotopes: nsmerger_table_radio
Etc..

Each enrichment source can be activated independently by providing its input radioactive yield table.  The radioactive yield table file format needs to be identical to its stable counterpart.
Warning: Radioactive isotopes will decay into stable isotopes.  When using radioactive yields, please make sure that the stable yields do not include the decayed isotopes already.
2. Single Decay Channel (Default Option)
If the radioactive isotopes you selected have only one decay channel, you can use the default decay option, which uses the following exponential law,
$N_r(t)=N_r(t_0)\,\mathrm{exp}\left[\frac{-(t-t_0)}{\tau}\right],$
$\tau=\frac{T_{1/2}}{\mathrm{ln}(2)},$
where $t_0$ is the reference time where the number of radioactive isotopes was equal to $N_0$. $T_{1/2}$ is the half-life of the isotope, which needs to be specified in yield_tables/decay_info.txt.  The decayed product will be added to the corresponding stable isotope, as defined in yield_tables/decay_info.txt.
Example with Al-26
Below, a SYGMA simulation is ran with no star formation to better isolate the decay process.  Here we choose Al-26 as an example, which decays into Mg-26.
Can you write Python code for it?

# Number of timesteps in the simulaton.
# See https://github.com/NuGrid/NuPyCEE/blob/master/DOC/Capabilities/Timesteps_size_management.ipynb
special_timesteps = -1
nb_dt = 100
tend = 2.0e6
dt = tend / float(nb_dt)

# No star formation.
no_sf = True

# Dummy neutron star merger yields to activate the radioactive option.
nsmerger_table_radio = 'yield_tables/extra_table_radio_dummy.txt'

# Add 1 Msun of radioactive Al-26 in the gas.
# The indexes of this array reflect the order seen in the yield_tables/decay_file.txt file
# Index 0, 1, 2 --> Al-26, K-40, U-238
ism_ini_radio = [1.0, 0.0, 0.0]


I am working on the file "DOC/Capabilities/Including_radioactive_isotopes.ipynb".
The first task is:
3. Multiple Decay Channels
If the radioactive isotopes you selected have more than one decay channel, you need to use the provided decay module.  This option can be activated by adding use_decay_module=True in the list of parameters when creating an instance of SYGMA and OMEGA.  When using the decay module, the yield_tables/decay_file.txt file still needs to be provided as an input to define which radioactive isotopes are selected for the calculation.
Example with K-40
Below we still run a SYGMA simulation with no star formation to better isolate the decay process.  A fraction of K-40 decays into Ca-40, and another fraction decays into Ar-40.
Run SYGMA
Can you write Python code for it?

# Add 1 Msun of radioactive K-40 in the gas.
# The indexes of this array reflect the order seen in the yield_tables/decay_file.txt file
# Index 0, 1, 2 --> Al-26, K-40, U-238
ism_ini_radio = [0.0, 1.0, 0.0]

# Number of timesteps in the simulaton.
# See https://github.com/NuGrid/NuPyCEE/blob/master/DOC/Capabilities/Timesteps_size_management.ipynb
special_timesteps = -1
nb_dt = 100
tend = 5.0e9
dt = tend / float(nb_dt)

# Run SYGMA (or in this case, the decay process)
# with the decay module
s = sygma.sygma(iniZ=0.0, sfr=sfr, starbursts=starbursts, ism_ini_radio=ism_ini_radio,\
                special_timesteps=special_timesteps, tend=tend, dt=dt,\
                decay_file='yield_tables/decay_file.txt', nsmerger_table_radio=nsmerger_table_radio,\
                use_decay_module=True, radio_refinement=1)

# Get the K-40 (radioactive) and Ca-40 and Ar-40 (stable) indexes in the gas arrays
i_K_40 = s.radio_iso.index('K-40')
i_Ca_40 = s.history.isotopes.index('Ca-40')
i_Ar_40 = s.history.isotopes.index('Ar-40')

# Extract the evolution of these isotopes as a function of time
K_40  = np.zeros(s.nb_timesteps+1)
Ca_40 = np.zeros(s.nb_timesteps+1)
Ar_40 = np.zeros(s.nb_timesteps+1)
for i_t in range(s.nb_timesteps+1):
    K_40[i_t]  = s.ymgal_radio[i_t][i_K_40]
    Ca_40[i_t] = s.ymgal[i_t][i_Ca_40]
    Ar_40[i_t] = s.ymgal[i_t][i_Ar_40]

# Plot the evolution of Al-26 and Mg-26
%matplotlib nbagg
plt.figure(figsize=(8,4.5))
plt.plot( np.array(s.history.age)/1e6, K_40, '--b', label='K-40' )
plt.plot( np.array(s.history.age)/1e6, Ca_40, '-r', label='Ca-40' )
plt.plot( np.array(s.history.age)/1e6, Ar_40, '-g', label='Ar-40' )

# Labels and fontsizes
plt.xlabel('Time [Myr]', fontsize=16)
plt.ylabel('Mass of isotope [M$_\odot$]', fontsize=16)
plt.legend(fontsize=14, loc='center left', bbox_to_anchor=(1, 0.5))
plt.subplots_adjust(top=0.96)
plt.subplots_adjust(bottom=0.15)
plt.subplots_adjust(right=0.75)
matplotlib.rcParams.update({'font.size': 14.0})


I am working on the file "DOC/Capabilities/Including_radioactive_isotopes.ipynb".
The first task is:
Example with U-238
Can you write Python code for it?

# Add 1 Msun of radioactive U-238 in the gas.
# The indexes of this array reflect the order seen in the yield_tables/decay_file.txt file
# Index 0, 1, 2 --> Al-26, K-40, U-238
ism_ini_radio = [0.0, 0.0, 1.0]

# Number of timesteps in the simulaton.
# See https://github.com/NuGrid/NuPyCEE/blob/master/DOC/Capabilities/Timesteps_size_management.ipynb
special_timesteps = -1
nb_dt = 100
tend = 5.0e9
dt = tend / float(nb_dt)

# Run SYGMA (or in this case, the decay process)
# with the decay module
s = sygma.sygma(iniZ=0.0, sfr=sfr, starbursts=starbursts, ism_ini_radio=ism_ini_radio,\
                special_timesteps=special_timesteps, tend=tend, dt=dt,\
                decay_file='yield_tables/decay_file.txt', nsmerger_table_radio=nsmerger_table_radio,\
                use_decay_module=True, radio_refinement=1)


I am working on the file "DOC/Capabilities/Including_radioactive_isotopes.ipynb".
The first task is:
Run SYGMA
Can you write Python code for it?

# Run SYGMA (or in this case, the decay process)
s = sygma.sygma(iniZ=0.02, no_sf=no_sf, ism_ini_radio=ism_ini_radio,\
                special_timesteps=special_timesteps, tend=tend, dt=dt,\
                decay_file='yield_tables/decay_file.txt', nsmerger_table_radio=nsmerger_table_radio)

# Get the Al-26 (radioactive) and Mg-26 (stable) indexes in the gas arrays
i_Al_26 = s.radio_iso.index('Al-26')
i_Mg_26 = s.history.isotopes.index('Mg-26')

# Extract the evolution of these isotopes as a function of time
Al_26 = np.zeros(s.nb_timesteps+1)
Mg_26 = np.zeros(s.nb_timesteps+1)
for i_t in range(s.nb_timesteps+1):
    Al_26[i_t] = s.ymgal_radio[i_t][i_Al_26]
    Mg_26[i_t] = s.ymgal[i_t][i_Mg_26]


I am working on the file "DOC/Capabilities/Modifying_yields.ipynb".
The first task is:
Modifying the isotopic yields of several stellar models
Can you write Python code for it?

# Select an isotope
iso = "Si-28"

# Define the list of multiplication factor for each stellar model
factor_list = [2, 4, 8, 16]
M_list = [12.0, 15.0, 20.0, 25.0]
Z = 0.02

# Fill the yield_modifier array
yield_modifier = []
for M, factor in zip(M_list, factor_list):
    yield_modifier.append([iso, M, Z, "multiply", factor])

# Run SYGMA with yields modification
s4 = sygma.sygma(iniZ=Z, yield_modifier=yield_modifier)

# Plot the yields as a function of stellar mass (IMF weighted yields for a 1Msun population)
%matplotlib nbagg
s4.plot_mass_range_contributions(specie=iso, color="C1", label="Multiplied")
s1.plot_mass_range_contributions(specie=iso, color="C2", label="Original")
plt.title(iso,fontsize=12)


I am working on the file "DOC/Capabilities/Netyields.ipynb".
The first task is:
Setup with total yields as input but net yields are calculated in the code and then applied
Can you write Python code for it?

# Initial metallicity 0
#includes pop3_table='yield_tables/popIII_heger10.txt',
iniZ = 0.0

#turn on net yield capability
yield_interp='wiersma'

#yield input not net yields
netyields_on=False

#should not matter
wiersmamod=False

Z_trans=-1

# Original yields
o_NG_net = o.omega(galaxy='milky_way', table=table, \
         special_timesteps=60, exp_ml=1.0, mass_frac_SSP=0.35, nb_1a_per_m=1.5e-3, DM_evolution=True, sfe=0.04,\
           t_sf_z_dep=0.3, mass_loading=1.02, iniZ=iniZ,yield_interp=yield_interp,netyields_on=netyields_on,\
             Z_trans=0.0)


I am working on the file "DOC/GettingStarted/SYGMA_Userguide.ipynb".
The first task is:
1.) Runnig the simulation
In the following the evolution of a single stellar population (SSP, one starburst) with an initial metallicty of $Z=0.0001$ is calculated. The SSP has a total mass of $10^{11}M_{\odot}$.  When reaching $1.3*10^{10}$ years the code stops. Start the simulation by creating an instance s1 of the class $sygma$:
Can you write Python code for it?


I am working on the file "DOC/GettingStarted/SYGMA_Userguide.ipynb".
The first task is:
First the massive stars eject their matter due to their short lifetime. Later AGB stars and SNIa start to contribute.
After about $3*10^8$ years AGB stars have ejected the most matter.
Analysis of the composition
One can also analyze the contribution of specific isotopes and elements.
Let us look a the element carbon.
Can you write Python code for it?


I am working on the file "DOC/GettingStarted/SYGMA_Userguide.ipynb".
The first task is:
For each run you could add here plots introduced in 2) by using the instances s2, s3 and s4.
5) Modifying the yield input
All yield input is provided as tables which can be easily modified. Or you can provide your own yield tables.
As default SYGMA uses NuGrid's yield tables for AGB and massive stars (Pignatari et al. 2016). Other defaults include the
choice of SNIa yields from Thielemann 1986
and PopIII stars from Heger 2010.
Using your own yield is not possible within the WENDI web interface.
To use own yields for AGB stars and massive stars it is necessary to specify the input variable table = 'yourtablename'. The table needs to have the NuGrid yield table format which is found as the default yield table file (specified with $table='yield_tables/isotope_yield_table.txt'$). All NuPyCEE yield table files are available in the $yield_tables$ directory.
Note that yield tables and initial abundance files must include the same isotopes. Hence it is necessary to specify a initial abundance file via the input variable $iniabu_table$ which must be placed in the $iniabu$ directory
within $yield_tables$.
In the following we show a example with artificial yields with only the species H-1.
Can you write Python code for it?

s_other_yields=s.sygma(iniZ=0.0001,tend=1.0e10,mgal=1.3e10,
             table='yield_tables/agb_and_massive_stars_h1.txt',
             iniabu_table='yield_tables/iniabu/iniab_h1.ppn')

s_other_yields.plot_totmasses(fig=6,source='all')


I am working on the file "DOC/GettingStarted/SYGMA_Userguide.ipynb".
The first task is:
The amount of H, He and C is normalized to the total amount of the gas mass specified initially ($10^{11}M_{\odot}$ in our case.)
4) Demonstration of basic capabilities
In the following we show basic capabilities of SYGMA which include changing the IMF, adaptiong the IMF range and changing the delay-time distribution of SNIa.
Can you write Python code for it?

#default type 'salpeter', change to chabrier IMF
s2=s.sygma(iniZ=0.0001,tend=1.0e10,mgal=1.3e10,imf_type='chabrier')

#default from 1 to 30Msun, changed to 2 to 15Msun
s3=s.sygma(iniZ=0.0001,tend=1.0e10,mgal=1.3e10,imf_bdys=[2,15])

#turning SNIa  off
s4=s.sygma(iniZ=0.0001,tend=1.0e10,mgal=1.3e10,sn1a_on=False)


I am working on the file "DOC/Teaching/.ipynb_checkpoints/Section2.1-checkpoint.ipynb".
The first task is:
Comparison of chemical evolution prediction with observation
Can you write Python code for it?

# Choose abundance ratios
%matplotlib nbagg
xaxis = '[Fe/H]'
yaxis = '[C/Fe]'

# Plot observational data points (Stellab)
stellab.plot_spectro(xaxis=xaxis, yaxis=yaxis,norm='Grevesse_Noels_1993',galaxy='milky_way',show_err=False)

# Extract the numerical predictions (OMEGA)
xy_f = o_mw.plot_spectro(fig=3,xaxis=xaxis,yaxis=yaxis,return_x_y=True)

# Overplot the numerical predictions (they are normalized according to Grevesse & Noels 1993)
plt.plot(xy_f[0],xy_f[1],linewidth=4,color='w')
plt.plot(xy_f[0],xy_f[1],linewidth=2,color='k',label='OMEGA')

# Update the existing legend
plt.legend(loc='center left', bbox_to_anchor=(1.01, 0.5), markerscale=0.8, fontsize=13)

# Choose X and Y limits
plt.xlim(-4.5,0.5)
plt.ylim(-1.4,1.6)


I am working on the file "DOC/Teaching/.ipynb_checkpoints/Section2.1-checkpoint.ipynb".
The first task is:
Now lets focus on C. What is the evolution of the total mass of C?
Can you write Python code for it?

# Plot the ejected mass of a certain element
elem='C'
s0p0001.plot_mass(fig=4,specie=elem,marker='D',color='b',label='Z=0.0001')
s0p006.plot_mass(fig=4,specie=elem,label='Z=0.006')


I am working on the file "DOC/Teaching/.ipynb_checkpoints/Section2.1-checkpoint.ipynb".
The first task is:
Simulation of the Milky Way
Can you write Python code for it?

# OMEGA parameters for MW
mass_loading = 1     # How much mass is ejected from the galaxy per stellar mass formed
nb_1a_per_m = 3.0e-3  # Number of SNe Ia per stellar mass formed
sfe = 0.005           # Star formation efficiency, which sets the mass of gas
table = 'yield_tables/isotope_yield_table_MESA_only_ye.txt' # Yields for AGB and massive stars
#milky_way

o_mw = omega.omega(galaxy='milky_way',Z_trans=-1, table=table,sfe=sfe, DM_evolution=True,\
                  mass_loading=mass_loading, nb_1a_per_m=nb_1a_per_m, special_timesteps=60)


I am working on the file "DOC/Teaching/.ipynb_checkpoints/Section2.1-checkpoint.ipynb".
The first task is:
Tracing back to simple stellar populations.
Can you write Python code for it?


I am working on the file "DOC/Teaching/.ipynb_checkpoints/Section2.1-checkpoint.ipynb".
The first task is:
Which stars contribute the most to C?
Can you write Python code for it?


I am working on the file "DOC/Teaching/ExtraSources.ipynb".
The first task is:
Default setup
Can you write Python code for it?

o0=o.omega(iniZ=iniZ,galaxy='milky_way',Z_trans=Z_trans, table=table,sfe=sfe, DM_evolution=DM_evolution,\
                  mass_loading=mass_loading, nb_1a_per_m=nb_1a_per_m, special_timesteps=special_timesteps,
                  imf_yields_range=imf_yields_range,
                SF_law=SF_law)


I am working on the file "DOC/Teaching/ExtraSources.ipynb".
The first task is:
SYGMA
Can you write Python code for it?

s0 = s.sygma(iniZ=0.0001,extra_source_on=False) #default False

s0p1 = s.sygma(iniZ=0.0001,extra_source_on=True,
             extra_source_table=extra_source_table,extra_source_mass_range=extra_source_mass_range,
           f_extra_source=f_extra_source, extra_source_exclude_Z=extra_source_exclude_Z)


I am working on the file "DOC/Teaching/ExtraSources.ipynb".
The first task is:
Setup
Can you write Python code for it?

# OMEGA parameters for MW
mass_loading = 0.0
nb_1a_per_m = 3.0e-3
sfe=0.04
SF_law=True
DM_evolution=False
imf_yields_range=[1.0,30.0]
special_timesteps=30
Z_trans=0.0
iniZ=0.0001


I am working on the file "DOC/Teaching/ExtraSources.ipynb".
The first task is:
Setup with different extra sources
Here we use yields in two (extra source) yield tables which we apply in the mass range from 8Msun to 12Msun and from 12Msun to 30Msun respectively. We apply a factor of 0.5 to the extra yields of the first yield table and 1. to the second yield table.
Can you write Python code for it?

extra_source_table=['yield_tables/r_process_arnould_2007.txt',
                    'yield_tables/r_process_arnould_2007.txt']
#Apply yields only in specific mass ranges;
extra_source_mass_range = [[8,12],[12,30]]
#percentage of stars to which the yields are added. First entry for first yield table etc.
f_extra_source = [0.5,1.]
#metallicity to exclude (in this case none)
extra_source_exclude_Z = [[], []]

#you can look at the yields directly with the y1 and y2 parameter below.
y1=ry.read_yields_Z("./NuPyCEE/"+extra_source_table[0])
y2=ry.read_yields_Z("./NuPyCEE/"+extra_source_table[1])


I am working on the file "DOC/Teaching/The inclusion of Faint Supernovae in the predictions of stellar yield relationships.ipynb".
The first task is:
Plotting the stellar abundance ratios
We can now plot the stellar abundance ratios. The element(s) of interest can be changed by altering the xaxis/yaxis identities below. In this case, O/Fe is plotted against Fe/H.
The abundance ratios are logarithmic as shown below.
$$[A/B]=\log(n_A/n_B)-\log(n_A/n_B)\odot.$$
where A and B are the relevant elements. This equation links $n_A/n_B$ (the abundance ratio for a star) and $(n_A/n_B)\odot$ (the solar normalisation). This ensures the resulting ratios are notated relative to the Suns own abundance ratios. If [A/B]= 0 this indicates that the star shares the same element abundance ratio for A and B. A star therefore has an abundance ratio $10^{A/B}$ times that of the Sun.The solar normalisation can be extracted from a reference paper, using the norm parameter, which in this case is 'Grevesse_Noels_1993'.
Can you write Python code for it?

#Obtain a list of normalisation reference papers
stellab.list_solar_norm()


I am working on the file "DOC/Teaching/The inclusion of Faint Supernovae in the predictions of stellar yield relationships.ipynb".
The first task is:
The Initial Mass Function
The Initial Mass Function describes the initial mass distribution of stars in a stellar population. Applying limits to this function allows us to control the mass range of the data it couples with. In this case it limits the range of the extra yields data. We have chosen the mass range 15-40 (Solar Mass), the mass range of the 1997D and 1999br faint supernovae Nomoto et al. 2013.  Every star within this mass range is assumed to release the same ejecta as the 20MSun model. In reality, stars with different masses will create ejecta of different masses and composition. The 20 Msun star model is however an average representative of the 15-40MSun mass range.
Z (iniZ) refers to the metallicity of the stellar population (one starburst).
Can you write Python code for it?

# Run a SYGMA instance to access the Initial Mass Function (IMF)
s_imf = sygma.sygma(iniZ=0.02, mgal=1.0)

# Define the mass range in which the extra yields will be applied
m_low = 15.0
m_up = 40.0

# Calculate the number of stars in that mass range per units of Msun formed
A = IMF = 1.0 / s_imf._imf(s_imf.imf_bdys[0], s_imf.imf_bdys[1], 2)
nb_extra_star_per_m = A * s_imf._imf(m_low, m_up, 1)
print (nb_extra_star_per_m)


I am working on the file "DOC/Teaching/The inclusion of Faint Supernovae in the predictions of stellar yield relationships.ipynb".
The first task is:
The abundance ratios that are plotted can be changinging by swapping the element symbols in the x/y axis brackets. A list of included data is incorporated in obs, which STELLAB will plot. References can be removed from this list to reduce data dispersion and clouding. Error bars are included for detailed and accurate analysis of the ratios. These can be turned on by including 'show_err=True, show_mean_err=True' within the plot_spectro function above. However the inclusion of error may lead to OMEGA being plotted underneath STELLAB,so it is important to use an overplotting procedure as shown above. OMEGA plots in white underneath the coloured lines in this instance. This makes it easier to differentiate between OMEGA plots and STELLAB data. The colours are changed by changing the 'color' parameter. See Dale et.al for colour examples.
Not all STELLAB data currently contain error bars, as many of these papers do not include them in yield tables, and they instead state an average error. These error bars can be plotted separately on the graph. The error markers respond to the STELLAB data which match their shape and colour.
Can you write Python code for it?

#Define your X and Y axis
xaxis = '[Fe/H]'
yaxis = '[O/Fe]'
# Plot observational data
obs = [
'stellab_data/milky_way_data/Venn_et_al_2004_stellab',
'stellab_data/milky_way_data/Akerman_et_al_2004_stellab',
'stellab_data/milky_way_data/Andrievsky_et_al_2007_stellab',
'stellab_data/milky_way_data/Andrievsky_et_al_2008_stellab',
'stellab_data/milky_way_data/Andrievsky_et_al_2010_stellab',
'stellab_data/milky_way_data/Bensby_et_al_2005_stellab',
'stellab_data/milky_way_data/Bihain_et_al_2004_stellab',
'stellab_data/milky_way_data/Bonifacio_et_al_2009_stellab',
'stellab_data/milky_way_data/Caffau_et_al_2005_stellab',
'stellab_data/milky_way_data/Gratton_et_al_2003_stellab',
'stellab_data/milky_way_data/Israelian_et_al_2004_stellab',
'stellab_data/milky_way_data/Nissen_et_al_2007_stellab',
'stellab_data/milky_way_data/Reddy_et_al_2003_stellab',
'stellab_data/milky_way_data/Spite_et_al_2005_stellab',
'stellab_data/milky_way_data/Battistini_Bensby_2016_stellab',
'stellab_data/milky_way_data/Nissen_et_al_2014_stellab',
'stellab_data/milky_way_data/Ramirez_et_al_2013_stellab',
'stellab_data/milky_way_data/Bensby_et_al_2014_stellab',
'stellab_data/milky_way_data/Battistini_Bensby_2015_stellab',
'stellab_data/milky_way_data/Adibekyan_et_al_2012_stellab',
'stellab_data/milky_way_data/Aoki_Honda_2008_stellab',
'stellab_data/milky_way_data/Hansen_et_al_2012_pecu_excluded_stellab',
'stellab_data/milky_way_data/Ishigaki_et_al_2012_2013_stellab',
'stellab_data/milky_way_data/Roederer_et_al_2009_stellab',
'stellab_data/milky_way_data/Roederer_et_al_2014_pecu_excluded_stellab']

stellab.plot_spectro(xaxis=xaxis, yaxis=yaxis, galaxy= 'milky way', norm='Grevesse_Noels_1993', show_err=True, show_mean_err=True, obs=obs, ms=4)

# Extract numerical predictions
xy = o.plot_spectro(xaxis=xaxis, yaxis=yaxis, return_x_y=True)
xy_trans = o_trans.plot_spectro(xaxis=xaxis, yaxis=yaxis, return_x_y=True)
xy_no = o_no.plot_spectro(xaxis=xaxis, yaxis=yaxis, return_x_y=True)
xy_no_trans = o_no_trans.plot_spectro(xaxis=xaxis, yaxis=yaxis,...


I am working on the file "DOC/Teaching/The inclusion of Faint Supernovae in the predictions of stellar yield relationships.ipynb".
The first task is:
The model assumes that all the faint SNs occur between 7.581 x 10^6 and 9.588 x 10^6 years. These are the respective lifetimes of the 20MSun and 25MSun star models at Z = 0.02. The number of faint SN per unit solar mass is constant throughout this time period (~0.00368 / MSun). The t and R arrays define the extra source time-distribution rate for a
simple stellar population. DTD stands for delay-time distribution. The extra pre-supernovae data yields are contained within  'yield_tables/extra_source_20Msun_O.txt'. The yield tables for preSN of different mass ranges are needed for future simulations for more precise analysis of faint SN ejecta.
Can you write Python code for it?

# Create the DTD and yields information for the extra source
# ==========================================================
# Event rate [yr^-1] as a function of time [yr].
# This assumes that all extra yields will be ejected
# between 7.581E+06 and 9.588E+06 years (the lifetimes
# of the 20 and 25 Msun models at Z = 0.02).
t = [7.580E+06, 7.581E+06, 9.588E+06, 9.589E+06]
R = [0.0, 1.0, 1.0, 0.0]

# Build the input DTD array
dtd = []
for i in range(0,len(t)):
    dtd.append([t[i], R[i]])

# Add the DTD array in the delayed_extra_dtd array.
delayed_extra_dtd = [[dtd]]

# Define the total number of event per unit of Msun formed.
delayed_extra_dtd_norm = [[nb_extra_star_per_m]]

# Define the total mass ejected by an extra source
# Here, it would be best to find a correction factor
# to account for the different total mass ejected by
# stars having different masses. For now, each star
# in the mass range eject the same ejecta as the 20Msun
# model.
delayed_extra_yields_norm = [[1.0]]

# Define the yields path for the extra source
extra_yields = ['yield_tables/extra_source_20Msun_O.txt']
delayed_extra_yields = extra_yields


I am working on the file "DOC/Teaching/The inclusion of Faint Supernovae in the predictions of stellar yield relationships.ipynb".
The first task is:
The observational data included by STELLAB can be changed by altering the list of references in obs. It is recommended that this list is kept relatively compact. Adding too much data may dominate the graph, making it difficult to observe important relationships. We have removed APOGEE, as it clouds particular areas of the graph, and does not currently feature notable error data. APOGEE can of course be returned if necessary. A list of references can be obtained using the command below.
Can you write Python code for it?


I am working on the file "DOCS.ipynb".
The first task is:
By default, invalid keys passed to the extract_subdict function are ignored:
Can you write Python code for it?

extract_subdict(person, ['name', 'birth', 'hair_color'])  # 'hair_color' is invalid


I am working on the file "DOCS.ipynb".
The first task is:
However, by passing True to the strict parameter of the function, invalid keys will raise a KeyError exception:
Can you write Python code for it?

extract_subdict(person, ['name', 'birth', 'hair_color'], strict=True)


I am working on the file "DOCS.ipynb".
The first task is:
If we need only the keys 'a' and 'd', we can do this:
Can you write Python code for it?

from subdict import extract_subdict  # The main function of the library

from pprint import pprint  # Just for a nice presentation here

pprint( extract_subdict(d, ['a', 'd']) )


I am working on the file "DOCS.ipynb".
The first task is:
Specifying invalid keys behavior
Let's consider the following dict from now on:
Can you write Python code for it?

person = {
    'name': 'John Frusciante',
    'birth': '1970-03-05',
    'city': {
        'name': 'New York City',
        'state': {'name': 'New York', 'country': 'USA'}
    },
    'albums': [
        {
            'year': 2001,
            'name': 'To Record Only Water For Ten Days',
            'label': {
                'name': 'Warner Bros Records',
                'link': 'https://en.wikipedia.org/wiki/Warner_Bros._Records'
            }
        },
        {
            'year': 2004,
            'name': 'Shadows Collide With People',
            'label': {
                'name': 'Warner Bros Records',
                'link': 'https://en.wikipedia.org/wiki/Warner_Bros._Records'
            }
        },
        {
            'year': 2009,
            'name': 'The Empyrean',
            'label': {
                'name': 'Record Collection',
                'link': 'https://en.wikipedia.org/wiki/Record_Collection'
            }
        }
    ]
}


I am working on the file "DOCS.ipynb".
The first task is:
The dotted-syntax can have any needed level of depth:
Can you write Python code for it?

pprint( extract_subdict(d, ['a', 'd.z.II.1']) )


I am working on the file "DOCS.ipynb".
The first task is:
The result is the following:
Can you write Python code for it?


I am working on the file "DP/Gamblers Problem Solution.ipynb".
The first task is:
Exercise 4.9 (programming)
Implement value iteration for the gambler’s problem and solve it for p_h = 0.25 and p_h = 0.55.
Can you write Python code for it?

def value_iteration_for_gamblers(p_h, theta=0.0001, discount_factor=1.0):
    """
    Args:
        p_h: Probability of the coin coming up heads
    """
    # The reward is zero on all transitions except those on which the gambler reaches his goal,
    # when it is +1.
    rewards = np.zeros(101)
    rewards[100] = 1

    # We introduce two dummy states corresponding to termination with capital of 0 and 100
    V = np.zeros(101)

    def one_step_lookahead(s, V, rewards):
        """
        Helper function to calculate the value for all action in a given state.

        Args:
            s: The gambler’s capital. Integer.
            V: The vector that contains values at each state.
            rewards: The reward vector.

        Returns:
            A vector containing the expected value of each action.
            Its length equals to the number of actions.
        """
        A = np.zeros(101)
        stakes = range(1, min(s, 100-s)+1) # Your minimum bet is 1, maximum bet is min(s, 100-s).
        for a in stakes:
            # rewards[s+a], rewards[s-a] are immediate rewards.
            # V[s+a], V[s-a] are values of the next states.
            # This is the core of the Bellman equation: The expected value of your action is
            # the sum of immediate rewards and the value of the next state.
            A[a] = p_h * (rewards[s+a] + V[s+a]*discount_factor) + (1-p_h) * (rewards[s-a] + V[s-a]*discount_factor)
        return A

    while True:
        # Stopping condition
        delta = 0
        # Update each state...
        for s in range(1, 100):
            # Do a one-step lookahead to find the best action
            A = one_step_lookahead(s, V, rewards)
            # print(s,A,V) # if you want to debug.
            best_action_value = np.max(A)
            # Calculate delta across all states seen so far
            delta = max(delta, np.abs(best_action_value - V[s]))
            # Update the value function. Ref: Sutton book eq. 4.10.
        ...


I am working on the file "DP/Gamblers Problem Solution.ipynb".
The first task is:
Show your results graphically, as in Figure 4.3.
Can you write Python code for it?

# Plotting Final Policy (action stake) vs State (Capital)

# x axis values
x = range(100)
# corresponding y axis values
y = v[:100]

# plotting the points
plt.plot(x, y)

# naming the x axis
plt.xlabel('Capital')
# naming the y axis
plt.ylabel('Value Estimates')

# giving a title to the graph
plt.title('Final Policy (action stake) vs State (Capital)')

# function to show the plot
plt.show()

# Plotting Capital vs Final Policy

# x axis values
x = range(100)
# corresponding y axis values
y = policy

# plotting the bars
plt.bar(x, y, align='center', alpha=0.5)

# naming the x axis
plt.xlabel('Capital')
# naming the y axis
plt.ylabel('Final policy (stake)')

# giving a title to the graph
plt.title('Capital vs Final Policy')

# function to show the plot
plt.show()


I am working on the file "DQN/Model-Train.ipynb".
The first task is:

Can you write Python code for it?

while episode_number < 30000 and agent.steps < 2980000:
    episode_number += 1
    reward = run_episode(agent, render=False)
    reward_sum += reward

    if agent.steps >= last_batch_steps + BATCH_SIZE:
        reward_avg = reward_sum / (episode_number - last_batch_episode)
        last_batch_episode = episode_number
        last_batch_steps = int(agent.steps / BATCH_SIZE) * BATCH_SIZE
        episodes.append((episode_number, agent.steps, reward_avg))

        print('Episode: %d, steps: %d, epsilon: %f, average reward: %f.' \
              % (episode_number, agent.steps, agent.epsilon, reward_avg))

        if reward_avg > reward_best:
            reward_best = reward_avg
            agent.brain.model.save_model('best.mod', False)

        reward_sum = 0

agent.brain.model.save_model('last.mod', False)
print('Done.')

plt.plot([e[1]/1000 for e in episodes], [e[2] for e in episodes])
plt.xlabel('steps x 1000');plt.ylabel('reward')
plt.savefig('learning.png', dpi=300)

slice_plot(n=20000, size=5, alpha=0.5)
plt.savefig('qslice.png', dpi=300)

pca_plot(n=20000, size=5, alpha=0.5)
plt.savefig('pca.png', dpi=300)


I am working on the file "DQN/Model-Train.ipynb".
The first task is:
Discounted Reward
We tune $\gamma$ to look ahead only a short timespan, in which the current action is of significant importance.
Can you write Python code for it?

r = range(0,600)
plt.plot([t/60.0 for t in r], [GAMMA ** x for x in r], 'r')
plt.xlabel('time [s]');plt.ylabel('discount')
GAMMA


I am working on the file "DQN/Model-Train.ipynb".
The first task is:
Exploration - exploitation trade-off
Note initiall $\epsilon$ is set to 1 which implies we are enitrely exploraing but as steps increase we reduce exploration and start leveraging the learnt space to collect rewards (a.k.a exploitation) as well.
Can you write Python code for it?

def epsilon(steps):
    return MIN_EPSILON + (MAX_EPSILON - MIN_EPSILON) * np.exp(-LAMBDA * steps)

r = range(0,EXPLORATION_STOP,int(EXPLORATION_STOP/100))
plt.plot(r, [min(epsilon(x),1) for x in r], 'r')
#plt.plot(r, [min(epsilon(x),1)**EPSILON_TRAIN_FACTOR for x in r], 'b')
plt.xlabel('step');plt.ylabel('$\epsilon$')


I am working on the file "DQN/Model-Train.ipynb".
The first task is:
Training
Can you write Python code for it?

def run_episode(agent, render=False):
    s = env.reset()
    R = 0
    while True:
        if render: env.render()

        a = agent.act(s.astype(np.float32))
        s_, r, done, info = env.step(a)

        if done:
            s_ = None

        agent.observe((s, a, r, s_))
        s = s_
        R += r

        if done:
            agent.endEpisode()
            return R

from DriveItGym import DriveItEnv
from agent import Agent

BATCH_SIZE = 20000

env = DriveItEnv(time_limit=10.0, throttle_limit=1.0)
stateCnt  = env.observation_space.shape[0]
actionCnt = env.action_space.n
agent = Agent(stateCnt, actionCnt)

episode_number = 0
last_batch_episode = 0
last_batch_steps = 0
episodes = []
reward_sum = 0
reward_best = 18.0

(stateCnt, actionCnt)

while episode_number < 50000 and agent.steps < 1980000:
    episode_number += 1
    reward = run_episode(agent, render=False)
    reward_sum += reward

    if agent.steps >= last_batch_steps + BATCH_SIZE:
        reward_avg = reward_sum / (episode_number - last_batch_episode)
        last_batch_episode = episode_number
        last_batch_steps = int(agent.steps / BATCH_SIZE) * BATCH_SIZE
        episodes.append((episode_number, agent.steps, reward_avg))

        print('Episode: %d, steps: %d, epsilon: %f, average reward: %f.' \
              % (episode_number, agent.steps, agent.epsilon, reward_avg))

        if reward_avg > reward_best:
            reward_best = reward_avg
            agent.brain.model.save_model('best.mod')

        reward_sum = 0

agent.brain.model.save_model('last.mod')
print('Done.')

plt.plot([e[1]/1000 for e in episodes], [e[2] for e in episodes])
plt.xlabel('steps x 1000');plt.ylabel('reward')


I am working on the file "DQN/Model-Train.ipynb".
The first task is:
Visualization
We use PCA decomposition on memory samples to visualize the $Q(s,a)$ values across the state space:
Can you write Python code for it?

def pca_plot(n=10000, alpha=1.0, size=5):
    _, samples = agent.memory.sample(n)
    states = np.array([ o[0] for o in samples ], dtype=np.float32)
    qsa = agent.brain.predict(states)[0]

    from sklearn import decomposition
    pca = decomposition.PCA(n_components=2)
    pca.fit(states)
    X = pca.transform(states)

    fig, axes = plt.subplots(nrows=2, ncols=2, sharex=True, sharey=True)
    ax = axes[0,0]; plt.sca(ax);ax.set_title('Action')
    plt.scatter(X[:, 0], X[:, 1], c=np.argmax(qsa, 1), alpha=alpha, s=size, cmap="rainbow")
    ax = axes[0,1]; plt.sca(ax);ax.set_title('Q(s,no-change)')
    plt.scatter(X[:, 0], X[:, 1], c=qsa[:,0], alpha=alpha, s=size, cmap="rainbow")
    ax = axes[1,0]; plt.sca(ax);ax.set_title('Q(s,left)')
    plt.scatter(X[:, 0], X[:, 1], c=qsa[:,1], alpha=alpha, s=size, cmap="rainbow")
    ax = axes[1,1]; plt.sca(ax);ax.set_title('Q(s,right)')
    plt.scatter(X[:, 0], X[:, 1], c=qsa[:,2], alpha=alpha, s=size, cmap="rainbow")


I am working on the file "DR12Q/DR12Q_correl_V01_LCDMr2.ipynb".
The first task is:
BallTree.two_point_correlation works almost 10 times faster! with leaf_size=5 Going with it to the random catalog
Can you write Python code for it?

dataR=ascii.read("./output/rand200kDR12Q.dat")

dataR

len(dataR)

len(dat)

rdr12f = open("./output/DR12Qsrarf.dat",'w')
rdr12f.write("z\t ra\t dec\t s\t rar\t decr \n")

for i in range(0,len(dataR)):
    rdr12f.write("%f\t " %dataR['z'][i])
    rdr12f.write("%f\t %f\t " %(dataR['ra'][i],dataR['dec'][i]))
    rdr12f.write("%f\t " %DC_LCDM(dataR['z'][i]))
    rdr12f.write("%f\t %f\n " %((dataR['ra'][i]*pi)/180.0,(dataR['dec'][i]*pi)/180.0))
rdr12f.close()

datR=ascii.read("./output/DR12Qsrarf.dat")

datR

rs=np.array(datR['s'])
rrar=np.array(datR['rar'])
rdecr=np.array(datR['decr'])

datR=np.array([rs,rrar,rdecr])

datR

datR.reshape(3,len(dataR))

datR=datR.transpose()

datR

# Saving the objects:
with open('./output/rDR12Qsrarf.pkl', 'w') as f:  # Python 3: open(..., 'wb')
    pickle.dump(datR, f)

# Getting back the objects:
with open('./output/rDR12Qsrarf.pkl') as f:  # Python 3: open(..., 'rb')
    datR = pickle.load(f)
datR

%%time
BT_R2 = BallTree(datR,metric='pyfunc',func=LCDMmetricsq,leaf_size=5)

with open('./output/BTRDR12QLCDM.pkl', 'w') as f:
    pickle.dump(BT_R2,f)

with open('./output/BTRDR12QLCDM.pkl') as f:
    BTR = pickle.load(f)
BTR

%%time
start_time=time.time()
counts_RR=BTR.two_point_correlation(datR,bins)
print counts_RR
end_time=time.time()
tottime=end_time-start_time
print "Total run time:"
print tottime

with open('./output/BTRDR12QcRRLCDM.pkl', 'w') as f:
    pickle.dump(counts_RR,f)

with open('./output/BTRDR12QcRRLCDM.pkl') as f:
    counts_RR = pickle.load(f)

counts_RR

counts_RR

RR=np.diff(counts_RR)

RR

plt.plot(bins[1:len(bins)],RR,'bo-')

RR_zero = (RR == 0)
RR[RR_zero] = 1

%%time
start_time=time.time()
counts_DR=BTR.two_point_correlation(dat,bins)
print counts_DR
end_time=time.time()
tottime=end_time-start_time
print "Total run time:"
print tottime

with open('./output/BTRDR12QcDRLCDM.pkl', 'w') as f:
    pickle.dump(counts_DR,f)

with open('./output/BTRDR12QcDRLCDM.pkl') as f:
    counts_DR = pickle.load(f)

counts_DR

DR=np.diff(counts_DR)

corrells=(4.0 *...


I am working on the file "DR12Q/DR12Q_correl_V01_LCDMr2.ipynb".
The first task is:
Read the data file (taken from http://cosmo.nyu.edu/~eak306/SDSS-LRG.html ) converted to ascii with comoving distance etc. in V01 reading from pkl files for faster read
Can you write Python code for it?

# Saving the objects:
with open('datDR12Q.pkl', 'w') as f:  # Python 3: open(..., 'wb')
    pickle.dump(dat, f)

# Getting back the objects:
with open('datDR12Q.pkl') as f:  # Python 3: open(..., 'rb')
    dat = pickle.load(f)
dat

Ez = lambda x: 1/m.sqrt(0.3*(1+x)**3+0.7)

np.vectorize(Ez)
#Calculate comoving distance of a data point using the Redshift - This definition is based on the cosmology model we take. Here the distance for E-dS universe is considered. Also note that c/H0 ratio is cancelled in the equations and hence not taken.

def DC_LCDM(z):
  return integrate.quad(Ez, 0, z)[0]
DC_LCDM=np.vectorize(DC_LCDM)

dr12f = open("./output/DR12Qsrarf.dat",'w')
dr12f.write("z\t ra\t dec\t s\t rar\t decr \n")

for i in range(0,len(z)):
    dr12f.write("%f\t " %z[i])
    dr12f.write("%f\t %f\t " %(ra[i],dec[i]))
    dr12f.write("%f\t " %DC_LCDM(z[i]))
    dr12f.write("%f\t %f\n " %(ra[i]*pi/180.0,dec[i]*pi/180.0))
dr12f.close()

dr12Qdat=ascii.read("./output/DR12Qsrarf.dat")

s=dr12Qdat['s']
rar=dr12Qdat['ra']
decr=dr12Qdat['dec']

dat=np.array([s,rar,decr])

dat

dat=dat.transpose()

dat


I am working on the file "DRB_vizer_json_services.ipynb".
The first task is:
Create dtutc column with parsed datetime. Also, it's safer to rename the "value" column to something unlikely to conflict with pandas method names.
Can you write Python code for it?

data_df['dtutc'] = data_df.time.map(lambda es: epochsec_to_dt(es))
data_df.set_index('dtutc', inplace=True, verify_integrity=True)
data_df.index.name = 'dtutc'
data_df = data_df.rename(columns={'value':var_id})

data_df.info()

data_df.head()

data_df.describe()

var_info = get_measurement_byvarid(metaresult, var_id)
title = "%s (%s) at %s" % (var_info['name'], var_id, metaresult['name'])
data_df[var_id].plot(title=title, figsize=[11,5])
plt.ylabel(var_info['units']);


I am working on the file "DRB_vizer_json_services.ipynb".
The first task is:
Data request
Can you write Python code for it?

data_r = requests.get(vz_gai_url, params={'asset_type':'siso', 'opt':'data', 'units_mode': 'v1',
                                          'asset_id':siso_id, 'var_id':'all'})
data = data_r.json()

data['success'], len(data['result']), data['result'][0].keys(), len(data['result'][0]['data'])

# Mapping of var_id string to 'result' list element index
var_ids = {e['var_id']:i for i,e in enumerate(data['result'])}
var_ids

var_id = 'H1_Discharge'

get_measurement_byvarid(metaresult, var_id)

data['result'][var_ids[var_id]]['data'][-10:]

# Pull out data time series for one variable based on var_id
# returns a list of dicts
data_lst = data['result'][var_ids[var_id]]['data']

data_df = pd.DataFrame.from_records(data_lst)

data_df.head()


I am working on the file "DRB_vizer_json_services.ipynb".
The first task is:
Meta info (metadata) requests
siso = "stationary in-situ observations" (eg, a river gage, weather stationg, moored buoy, etc)
Can you write Python code for it?

meta_r = requests.get(vz_gai_url, params={'asset_type':'siso', 'opt':'meta'})
meta = meta_r.json()

meta.keys(), meta['success']

type(meta['result']), len(meta['result'])

# siso_id is the unique identifier (string type) for the station
siso_id_lst = [e['siso_id'] for e in meta['result']]

# Examine the response for the first station (index 0) in the returned list
meta['result'][0]['siso_id']

meta['result'][0]


I am working on the file "DRB_vizer_json_services.ipynb".
The first task is:
Request and examine one station
Can you write Python code for it?

# USGS Schuylkill River at Philadelphia
siso_id = 'USGS_01474500'


I am working on the file "DRB_vizer_json_services.ipynb".
The first task is:
Service end point
For vizer instance that currently serves only the CRB and outlying areas, but will be expanded to the DRB to support the William Penn project needs.
Can you write Python code for it?


I am working on the file "DRB_vizer_json_services.ipynb".
The first task is:
Summaries (station counts) by platform_type and provider
Can you write Python code for it?


I am working on the file "DRILL+Mo%27+blobs%2C+mo%27+problems.ipynb".
The first task is:
Affinity Propagation
Can you write Python code for it?

# Declare the model and fit it in one statement.
# Note that you can provide arguments to the model, but we didn't.
af = AffinityPropagation().fit(X_train)
print('Done')

# Pull the number of clusters and cluster assignments for each data point.
cluster_centers_indices = af.cluster_centers_indices_
n_clusters_ = len(cluster_centers_indices)
labels = af.labels_

print('Estimated number of clusters: {}'.format(n_clusters_))

# Plot the clusters
plt.figure(1)
plt.clf()

# Cycle through each cluster and graph them with a center point for the
# exemplar and lines from the exemplar to each data point in the cluster.
colors = cycle('bgrcmykbgrcmykbgrcmykbgrcmyk')
for k, col in zip(range(n_clusters_), colors):
    class_members = labels == k
    cluster_center = X_train[cluster_centers_indices[k]]
    plt.plot(X_train[class_members, 0], X_train[class_members, 1], col + '.')
    plt.plot(cluster_center[0],
             cluster_center[1],
             'o',
             markerfacecolor=col,
             markeredgecolor='k')
    for x in X_train[class_members]:
        plt.plot([cluster_center[0], x[0]], [cluster_center[1], x[1]], col)

plt.title('Estimated number of clusters: {}'.format(n_clusters_))
plt.show()


I am working on the file "DRILL+Mo%27+blobs%2C+mo%27+problems.ipynb".
The first task is:
Blobs 1
Can you write Python code for it?

# Create blobs
# The coordinates of the centers of our blobs.
centers = [[0, 0], [-10, -10], [10, 10], [5,-5], [-5,5]]

# Make 10,000 rows worth of data with two features representing three
# clusters, each having a standard deviation of 1.
X, y = make_blobs(
    n_samples=10000,
    centers=centers,
    cluster_std=1,
    n_features=2,
    random_state=42)

plt.scatter(X[:, 0], X[:, 1], c=y)
plt.show()

#Divide into training and test sets.
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.9,
    random_state=42)


I am working on the file "DRILL+Mo%27+blobs%2C+mo%27+problems.ipynb".
The first task is:
K-Means
Can you write Python code for it?

# Normalize the data.
#X_norm = normalize(X_train)

# Reduce it to two components.
X_pca = PCA(2).fit_transform(X_train)

# Calculate predicted values.
y_pred = KMeans(n_clusters=3, random_state=42).fit_predict(X_pca)

# Plot the solution.
plt.scatter(X_pca[:, 0], X_pca[:, 1], c=y_pred)
plt.show()

# Check the solution against the data.
print('Comparing k-means clusters against the data:')
print(pd.crosstab(y_pred, y_train))


I am working on the file "DRILL+Mo%27+blobs%2C+mo%27+problems.ipynb".
The first task is:
Mean Shift Clustering
Can you write Python code for it?

# Here we set the bandwidth. This function automatically derives a bandwidth
# number based on an inspection of the distances among points in the data.
bandwidth = estimate_bandwidth(X_train, quantile=0.2, n_samples=500)

# Declare and fit the model.
ms = MeanShift(bandwidth=bandwidth, bin_seeding=True)
ms.fit(X_train)

# Extract cluster assignments for each data point.
labels = ms.labels_

# Coordinates of the cluster centers.
cluster_centers = ms.cluster_centers_

# Count our clusters.
n_clusters_ = len(np.unique(labels))

print("Number of estimated clusters: {}".format(n_clusters_))

plt.scatter(X_train[:, 0], X_train[:, 1], c=labels)
plt.show()

print('Comparing the assigned categories to the ones in the data:')
print(pd.crosstab(y_train,labels))


I am working on the file "DRILL+Mo%27+blobs%2C+mo%27+problems.ipynb".
The first task is:
Spectral Clustering
Can you write Python code for it?

# We know we're looking for three clusters.
n_clusters=3

# Declare and fit the model.
sc = SpectralClustering(n_clusters=n_clusters)
sc.fit(X_train)

#Predicted clusters.
predict=sc.fit_predict(X_train)

#Graph results.
plt.scatter(X_train[:, 0], X_train[:, 1], c=predict)
plt.show()

print('Comparing the assigned categories to the ones in the data:')
print(pd.crosstab(y_train,predict))


# We know we're looking for three clusters.
n_clusters=4

# Declare and fit the model.
sc = SpectralClustering(n_clusters=n_clusters)
sc.fit(X_train)

#Predicted clusters.
predict=sc.fit_predict(X_train)

#Graph results.
plt.scatter(X_train[:, 0], X_train[:, 1], c=predict)
plt.show()

print('Comparing the assigned categories to the ones in the data:')
print(pd.crosstab(y_train,predict))


I am working on the file "DS-ML_Nov_train_test.ipynb".
The first task is:
ARCH
    https://www.quantopian.com/posts/some-code-from-ernie-chans-new-book-implemented-in-python
    http://auquan.com/cointegration-stationarity/
    https://www.analyticsvidhya.com/blog/2016/02/time-series-forecasting-codes-python/
    http://machinelearningmastery.com/time-series-forecast-case-study-python-monthly-armed-robberies-boston/
    https://www.quantstart.com/articles/Basics-of-Statistical-Mean-Reversion-Testing-Part-II
    https://datascience.ibm.com/exchange/public/entry/view/815137c868b916821dec777bdc23013c
    http://machinelearningmastery.com/time-series-data-stationary-python/

Classification
Can you write Python code for it?

X=df[['mid','vwap','spread','v','return','sigma','high','low',]]
y=df['mom']

len(df)

from sklearn import linear_model
regr = linear_model.LinearRegression()
#regr.fit(X.tail(20),y.tail(20))
#predict=regr.predict(X.tail(5))
regr.fit(X.dropna(),y.dropna())
predict=regr.predict(X)
#X=X.dropna()
#y=y.dropna()
#y[y == inf] = 0
dt=df[['mid']]
dt['predict']=predict
#dt['predict']=dt.mid+dt.mid*dt.predict
dt['predict']=dt.predict*df.v+df.mid.shift(12)
classify_df=df
classify_df['pREG']=dt.predict

from sklearn.svm import SVR
# Fit regression model
svr_rbf = SVR(kernel='rbf', C=1e3, gamma=0.9) #kernel='linear' #kernel='poly'
predict_svr = svr_rbf.fit(X, y).predict(X)
dt1=df[['mid']]
dt1['predict']=predict_svr
dt1['predict']=dt1.predict*df.v+df.mid.shift(12)
classify_df['pSVR']=dt1.predict

def classification(df):
    mid1=(df.high+df.low)/2
    #flagUD=np.where(np.logical_and(df.mid>df.pREG,df.mid>df.pSVR),1,np.where(np.logical_and(df.mid<df.pREG,df.mid<df.pSVR),-1,0))
    #df['UD']= np.where(np.logical_and(df.mid>mid1,flagUD==1),1,np.where(np.logical_and(df.mid<mid1,flagUD==-1),-1,0))
    flagUD=np.where(np.logical_and(df.mid>df.pREG,df.mid>df.pSVR),1,np.where(np.logical_and(df.mid<df.pREG,df.mid<df.pSVR),-1,0))
    UD= np.where(np.logical_and(df.mid>mid1,flagUD==1),1,np.where(np.logical_and(df.mid<mid1,flagUD==-1),-1,0))
    df['U']= np.where(UD==1,1,0)
    df['D']= np.where(UD==-1,-1,0)
    df['UD']=df.U+df.D
    return df

data_class=classification(classify_df)
data_class=data_class.dropna()
df=df.dropna()
# both df and data_class have U,D,UD

data_class.head()

df.head()

# split into train and test sets
train_size = int(len(data_class) * 0.80)
test_size = len(data_class) - train_size
train= data_class[0:train_size]
test= data_class[train_size:len(data_class)]
print(len(train),...


I am working on the file "DS-ML_Nov_train_test.ipynb".
The first task is:
Ada Boosting binary Classification
Can you write Python code for it?

from sklearn.ensemble import AdaBoostClassifier

clf = AdaBoostClassifier()
clf.fit(train_X,train_U)
accuracy = clf.score(test_X,test_U)
print(accuracy)


I am working on the file "DS-ML_Nov_train_test.ipynb".
The first task is:
Dataset
Can you write Python code for it?

filename = '/home/octo/Dropbox'+ '/SPY7Dec.csv'

data=get_csv_pd(filename)

data=preprocessing(data)

df=data.dropna()
df=df[['mid','vwap','spread','v','mom','return','sigma','high','low',]]

# split into train and test sets
train_size = int(len(df) * 0.80)
test_size = len(df) - train_size
train= df[0:train_size]
test= df[train_size:len(df)]
print(len(train), len(test))

train_X=train[['mid','vwap','spread','v','return','sigma','high','low',]]
train_y=train['mom']
test_X=test[['mid','vwap','spread','v','return','sigma','high','low',]]
test_y=test['mom']

train_X.head()

test_y.head()


I am working on the file "DS-ML_Nov_train_test.ipynb".
The first task is:
Decision Trees
Can you write Python code for it?

from sklearn import metrics
from sklearn.tree import DecisionTreeClassifier

# fit a CART model to the data
model = DecisionTreeClassifier()
model.fit(train_X,train_U)
print(model)

# make predictions
expected =test_U
predicted = model.predict(test_X)

# summarize the fit of the model
print(metrics.classification_report(expected, predicted))
print(metrics.confusion_matrix(expected, predicted))

accuracy =model.score(test_X,test_U)
accuracy

# fit a CART model to the data
model = DecisionTreeClassifier()
model.fit(train_X,train_D)
print(model)

# make predictions
expected =test_D
predicted = model.predict(test_X)

# summarize the fit of the model
print(metrics.classification_report(expected, predicted))
print(metrics.confusion_matrix(expected, predicted))

accuracy =model.score(test_X,test_D)
accuracy


I am working on the file "DS-ML_Nov_train_test.ipynb".
The first task is:
Gradient Tree Boosting binary Classification
Can you write Python code for it?

from sklearn.ensemble import GradientBoostingClassifier

clf = GradientBoostingClassifier(n_estimators=100)
clf.fit(train_X,train_U)
accuracy = clf.score(test_X,test_U)
print(accuracy)


I am working on the file "DS-ML_Nov_train_test.ipynb".
The first task is:
KNN
Can you write Python code for it?

from sklearn import neighbors

clf = neighbors.KNeighborsClassifier()
clf.fit(train_X,train_U)

accuracy = clf.score(test_X,test_U)
print(accuracy)


I am working on the file "DS-ML_Nov_train_test.ipynb".
The first task is:
Quadratic Discriminant Analysis binary Classification
Can you write Python code for it?

from sklearn.qda import QDA

clf = QDA()
clf.fit(train_X,train_U)
accuracy = clf.score(test_X,test_U)
print(accuracy)


I am working on the file "DS-ML_Nov_train_test.ipynb".
The first task is:
Random Forest
The random forest algorithm can find nonlinearities in data that a linear regression wouldn’t be able to pick up on.
Can you write Python code for it?

# Import the random forest model.
from sklearn.ensemble import RandomForestRegressor

# Initialize the model with some parameters.
model = RandomForestRegressor(n_estimators=100, min_samples_leaf=10, random_state=1)
# Fit the model to the data.
model.fit(train_X,train_U)
print(model)

# Make predictions.
expected=test_U
predicted = model.predict(test_X)

accuracy =model.score(test_X,test_U)
accuracy

from sklearn.ensemble import RandomForestClassifier

clf = RandomForestClassifier(n_estimators=100, n_jobs=-1)
clf.fit(train_X,train_U)
accuracy = clf.score(test_X,test_U)
print(accuracy)


I am working on the file "DS-ML_Nov_train_test.ipynb".
The first task is:
Regression
Can you write Python code for it?

from sklearn import linear_model
regr = linear_model.LinearRegression()
#regr.fit(X.tail(20),y.tail(20))
#predict=regr.predict(X.tail(5))
regr.fit(train_X,train_y)
predict=regr.predict(test_X)
#X=X.dropna()
#y=y.dropna()
#y[y == inf] = 0
dt=test[['mid']]
dt['predict']=predict
#dt['predict']=dt.mid+dt.mid*dt.predict
dt['predict']=dt.predict*test.v+test.mid.shift(12)
pdf=test
pdf['pREG']=dt.predict

pdf.tail()

from sklearn.svm import SVR
# Fit regression model
svr_rbf = SVR(kernel='rbf', C=1e3, gamma=0.9) #kernel='linear' #kernel='poly'
predict_svr = svr_rbf.fit(train_X,train_y).predict(test_X)
dt1=test[['mid']]
dt1['predict']=predict_svr
dt1['predict']=dt1.predict*test.v+test.mid.shift(12)
pdf['pSVR']=dt1.predict

pdf.dropna().head()

pdf[['mid','high','low','pREG','pSVR']].tail(300).plot(figsize=(15,9))
#df[['Volume']].tail(5000).plot(figsize=(15,9))
#data[['AvgVolume']].tail(5000).plot(figsize=(15,9))
plt.show()

# look at the results
plt.scatter(pdf['mid'],test_y, c='k', label='data')
plt.hold('on')
plt.plot(pdf['pREG'],test_y, c='g', label='pREG')
#plt.plot(pdf['pSVR'], y, c='g', label='pSVR')
plt.xlabel('data')
plt.ylabel('target')
plt.title('Support Vector Regression')
plt.legend()
plt.show()

# look at the results
plt.scatter(pdf['mid'],pdf['pREG'], c='k', label='pSVR')
plt.hold('on')
plt.plot(pdf['mid'],pdf['pSVR'], c='g', label='pREG')
plt.plot(pdf['mid'], pdf['high'], c='g', label='high')
plt.xlabel('data')
plt.ylabel('target')
plt.title('Support Vector Regression')
plt.legend()
plt.show()


I am working on the file "DS-ML_Nov_train_test.ipynb".
The first task is:
graphical
Can you write Python code for it?

# plotting
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()
%matplotlib inline
#plt.rcParams['figure.figsize'] = 8,6

test.boxplot(column='v')

test.boxplot(by='v')
plt.ylim(245,248)

test.boxplot(by='UD')

#some descriptive statistics
test.describe()

test['v'].plot(kind='hist', grid=True, title='velocity')

test['UD'].plot(kind='hist', grid=True, title='up-down')

test['v'].plot(kind='line', grid=True, title='velocity')

test['UD'].plot(kind='line', grid=True, title='up-down')

# Find 7, 30, 120 day moving averages (very broadly, rolling week, month and quarter)
spy_12 = test.rolling(window=12).mean()
spy_60 = test.rolling(window=60).mean()
spy_360 = test.rolling(window=360).mean()

fig = plt.figure()
fig.autofmt_xdate()
ax = fig.add_subplot(1,1,1)
ax.plot(test.index,test, label='SPY')
ax.plot(spy_12.index, spy_12, label='1 min rolling')
ax.plot(spy_60.index, spy_60, label='5 min rolling')
ax.plot(spy_360.index,spy_360, label='30 min rolling')
ax.grid()
ax.legend(loc=2)
ax.set_xlabel('Date')
plt.title('SPY Closes & Rolling Averages')
plt.show()

#frequency
round(test['mom']).value_counts()

round(test['vwap'],1).hist(bins=50)

test.boxplot(column='mid')

#df for datascience
#signal=df.DataFrame(data=df.mid)
signal=df
#df['time']=df.index.strftime('%H:%M:%S')
time=signal.index.strftime('%H:%M:%S')

P=(signal.high+signal.low+signal.mid)/3
signal['UT']=(P+signal.high.rolling(60).max()-signal.low.rolling(60).max())
signal['DT']=(P-signal.high.rolling(60).min()+signal.low.rolling(60).min())
signal['BS']=np.where(signal.mid<=df.DT,"B",np.where(signal.mid>=df.UT,"S","H"))
signal=signal.dropna()

signal.head()

df[['UT','DT','mid','high','low','pREG','pSVR']].tail(100).plot(figsize=(16, 10))
plt.show()

signal.boxplot(column='mid',by ='BS')

temp1 = round(signal['UD']).value_counts(ascending=True)
temp2 = signal.pivot_table(values='UD',index=['BS'],aggfunc=lambda x: x.map({'B':1,'S':-1,'H':0}).mean())

print ('Frequency Table for spread:')
print (temp1)
print ('\nProbility')
print...


I am working on the file "DS502-1704/MXNet-week1-part1/gluon-MNIST/Deep Learning Fundamentals-Gluon.ipynb".
The first task is:
Brodcasting
Can you write Python code for it?

x = nd.ones(shape=(3,3))
print('x = ', x)
y = nd.arange(3)
print('y = ', y)
print('x + y = ', x + y)


I am working on the file "DS502-1704/MXNet-week1-part1/gluon-MNIST/Deep Learning Fundamentals-Gluon.ipynb".
The first task is:
Scala, Vector, Matrices, Tensors
Can you write Python code for it?

# scalars
x = nd.array([3.0])
y = nd.array([2.0])
print 'x + y = ', x + y
print 'x * y = ', x * y
print 'x / y = ', x / y
print 'x ** y = ', nd.power(x,y)

# convert it to python scala
x.asscalar()

# Vector
u = nd.arange(4)
print('u = ', u)
print u[3]
print len(u)
print u.shape

a = 2
x = nd.array([1,2,3])
y = nd.array([10,20,30])
print(a * x)
print(a * x + y)

# Matrices
x = nd.arange(20)
A = x.reshape((5, 4))
print A
print 'A[2, 3] = ', A[2, 3]

print('row 2', A[2, :])
print('column 3', A[:, 3])
print A.T

# Tensor
X = nd.arange(24).reshape((2, 3, 4))
print 'X.shape =', X.shape
print 'X =', X

u = nd.array([1, 2, 4, 8])
v = nd.ones_like(u) * 2
print 'v =', v
print 'u + v', u + v
print 'u - v', u - v
print 'u * v', u * v
print 'u / v', u / v
print nd.sum(u)
print nd.mean(u)
print nd.sum(u) / u.size
print nd.dot(u, v)
print nd.sum(u * v)

# Matrices multiple Vector
print nd.dot(A, u)
print nd.dot(A, A.T)

print nd.norm(u)
print nd.sqrt(nd.sum(u**2))
print nd.sum(nd.abs(u))


I am working on the file "DS502-1704/MXNet-week1-part1/gluon-MNIST/Deep Learning Fundamentals-Gluon.ipynb".
The first task is:
nd array <-> Numpy
Can you write Python code for it?

a = x.asnumpy()
print "The type of a is {}".format(type(a))
y = nd.array(a)
print "The type of a is {}".format(type(y))


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_MNIST/Nearest_neighbor_MNIST.ipynb".
The first task is:
2. Visualizing the data
Each data point is stored as 784-dimensional vector. To visualize a data point, we first reshape it to a 28x28 image.
Can you write Python code for it?

## Define a function that displays a digit given its vector representation
def show_digit(x):
    plt.axis('off')
    plt.imshow(x.reshape((28,28)), cmap=plt.cm.gray)
    plt.show()
    return

## Define a function that takes an index into a particular data set ("train" or "test") and displays that image.
def vis_image(index, dataset="train"):
    if(dataset=="train"):
        show_digit(train_data[index,])
        label = train_labels[index]
    else:
        show_digit(test_data[index,])
        label = test_labels[index]
    print("Label " + str(label))
    return

## View the first data point in the training set
vis_image(0, "train")

## Now view the first data point in the test set
vis_image(0, "test")


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_MNIST/Nearest_neighbor_MNIST.ipynb".
The first task is:
3. Squared Euclidean distance
To compute nearest neighbors in our data set, we need to first be able to compute distances between data points. A natural distance function is Euclidean distance: for two vectors $x, y \in \mathbb{R}^d$, their Euclidean distance is defined as
$$\|x - y\| = \sqrt{\sum_{i=1}^d (x_i - y_i)^2}.$$
Often we omit the square root, and simply compute squared Euclidean distance:
$$\|x - y\|^2 = \sum_{i=1}^d (x_i - y_i)^2.$$
For the purposes of nearest neighbor computations, the two are equivalent: for three vectors $x, y, z \in \mathbb{R}^d$, we have $\|x - y\| \leq \|x - z\|$ if and only if $\|x - y\|^2 \leq \|x - z\|^2$.
Now we just need to be able to compute squared Euclidean distance. The following function does so.
Can you write Python code for it?

## Computes squared Euclidean distance between two vectors.
def squared_dist(x,y):
    return np.sum(np.square(x-y))

## Compute distance between a seven and a one in our training set.
print("Distance from 7 to 1: ", squared_dist(train_data[4,],train_data[5,]))

## Compute distance between a seven and a two in our training set.
print("Distance from 7 to 2: ", squared_dist(train_data[4,],train_data[1,]))

## Compute distance between two seven's in our training set.
print("Distance from 7 to 7: ", squared_dist(train_data[4,],train_data[7,]))


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_MNIST/Nearest_neighbor_MNIST.ipynb".
The first task is:
5. For you to try
The above two examples show the results of the NN classifier on test points number 0 and 39.
Now try test point number 100.
* What is the index of its nearest neighbor in the training set? Record the answer: you will enter it as part of this week's assignment.
* Display both the test point and its nearest neighbor.
* What label is predicted? Is this the correct label?
Can you write Python code for it?

print("NN classification: ", NN_classifier(test_data[100,]))
print("True label: ", test_labels[100])
print("The test image:")
vis_image(100, "test")
print("The corresponding nearest neighbor image:")
vis_image(find_NN(test_data[100,]), "train")

find_NN(test_data[100,])


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_MNIST/Nearest_neighbor_MNIST.ipynb".
The first task is:
6. Processing the full test set
Now let's apply our nearest neighbor classifier over the full data set.
Note that to classify each test point, our code takes a full pass over each of the 7500 training examples. Thus we should not expect testing to be very fast. The following code takes about 100-150 seconds on 2.6 GHz Intel Core i5.
Can you write Python code for it?

## Predict on each test data point (and time it!)
t_before = time.time()
test_predictions = [NN_classifier(test_data[i,]) for i in range(len(test_labels))]
t_after = time.time()

## Compute the error
err_positions = np.not_equal(test_predictions, test_labels)
error = float(np.sum(err_positions))/len(test_labels)

print("Error of nearest neighbor classifier: ", error)
print("Classification time (seconds): ", t_after - t_before)


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_spine/.ipynb_checkpoints/Nearest_neighbor_spine-checkpoint.ipynb".
The first task is:
3. Nearest neighbor classification with L1 distance
We now compute nearest neighbors using the L1 distance (sometimes called Manhattan Distance).
For you to do: Write a function, NN_L1, which again takes as input the arrays trainx, trainy, and testx, and predicts labels for the test points using 1-nearest neighbor classification. For NN_L1, the L1 distance metric should be used. As before, the predicted labels should be returned in a numpy array with one entry per test point.
Notice that NN_L1 and NN_L2 may well produce different predictions on the test set.
 Code
```python
test function
testy_L2 = NN_L2(trainx, trainy, testx)
testy_L1 = NN_L1(trainx, trainy, testx)
print( type( testy_L1) )
print( len(testy_L1) )
print( testy_L1[40:50] )
print( all(testy_L1 == testy_L2) )
```
 Output
<class 'numpy.ndarray'>
62
[ 2.  2.  0.  0.  0.  0.  0.  0.  0.  0.]
False
Can you write Python code for it?

# Modify this Cell

def NN_L1(trainx, trainy, testx):
    # inputs: trainx, trainy, testx <-- as defined above
    # output: an np.array of the predicted values for testy

    ### BEGIN SOLUTION
    ### END SOLUTION


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_spine/.ipynb_checkpoints/Nearest_neighbor_spine-checkpoint.ipynb".
The first task is:
4. Test errors and the confusion matrix
Let's see if the L1 and L2 distance functions yield different error rates for nearest neighbor classification of the test data.
Can you write Python code for it?

def error_rate(testy, testy_fit):
    return float(sum(testy!=testy_fit))/len(testy)

print("Error rate of NN_L1: ", error_rate(testy,testy_L1) )
print("Error rate of NN_L2: ", error_rate(testy,testy_L2) )


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_spine/.ipynb_checkpoints/Nearest_neighbor_spine-checkpoint.ipynb".
The first task is:
After you are done, run the cell below to check your function. If an error is triggered, you should go back and revise your function.
Can you write Python code for it?


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_spine/.ipynb_checkpoints/Nearest_neighbor_spine-checkpoint.ipynb".
The first task is:
We now load the dataset. We divide the data into a training set of 248 patients and a separate test set of 62 patients. The following arrays are created:

trainx : The training data's features, one point per row.
trainy : The training data's labels.
testx : The test data's features, one point per row.
testy : The test data's labels.

We will use the training set (trainx and trainy), with nearest neighbor classification, to predict labels for the test data (testx). We will then compare these predictions with the correct labels, testy.
Notice that we code the three labels as 0. = ’NO’, 1. = ’DH’, 2. = ’SL’.
Can you write Python code for it?

# Load data set and code labels as 0 = ’NO’, 1 = ’DH’, 2 = ’SL’
labels = [b'NO', b'DH', b'SL']
data = np.loadtxt('column_3C.dat', converters={6: lambda s: labels.index(s)} )

# Separate features from labels
x = data[:,0:6]
y = data[:,6]

# Divide into training and test set
training_indices = list(range(0,20)) + list(range(40,188)) + list(range(230,310))
test_indices = list(range(20,40)) + list(range(188,230))

trainx = x[training_indices,:]
trainy = y[training_indices]
testx = x[test_indices,:]
testy = y[test_indices]


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_spine/.ipynb_checkpoints/Nearest_neighbor_spine-checkpoint.ipynb".
The first task is:
We will now look a bit more deeply into the specific types of errors made by nearest neighbor classification, by constructing the confusion matrix.
Since there are three labels, the confusion matrix is a 3x3 matrix whose rows correspond to the true label and whose columns correspond to the predicted label. For example, the entry at row DH, column SL, contains the number of test points whose correct label was DH but which were classified as SL.

Write a function, confusion, which takes as input the true labels for the test set (that is, testy) as well as the predicted labels and returns the confusion matrix. The confusion matrix should be a np.array of shape (3,3) .
 Code
python
L2_neo = confusion(testy, testy_L2)
print( type(L2_neo) )
print( L2_neo.shape )
print( L2_neo )
 Output
<class 'numpy.ndarray'>
(3, 3)
[[ 0.  9.  2.]
 [ 0.  0.  0.]
 [ 3.  0.  0.]]
Can you write Python code for it?

# Modify this cell

def confusion(testy,testy_fit):
    # inputs: the correct labels, the fitted NN labels
    # output: a 3x3 np.array representing the confusion matrix as above

    ### BEGIN SOLUTION
    ### END SOLUTION


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_spine/Nearest_neighbor_spine.ipynb".
The first task is:
2. Nearest neighbor classification with L2 distance
In this exercise we will build a nearest neighbor classifier based on L2 (Euclidean) distance.
For you to do: Write a function, NN_L2, which takes as input the training data (trainx and trainy) and the test points (testx) and predicts labels for these test points using 1-NN classification. These labels should be returned in a numpy array with one entry per test point. For NN_L2, the L2 norm should be used as the distance metric.
 Code
```python
test function
testy_L2 = NN_L2(trainx, trainy, testx)
print( type( testy_L2) )
print( len(testy_L2) )
print( testy_L2[40:50] )
```
 Output
<class 'numpy.ndarray'>
62
[ 2.  2.  1.  0.  0.  2.  0.  0.  0.  0.]
Can you write Python code for it?

# Modify this Cell

def NN_L2(trainx, trainy, testx):
    # inputs: trainx, trainy, testx <-- as defined above
    # output: an np.array of the predicted values for testy

    ### BEGIN SOLUTION
    ### END SOLUTION


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_spine/Nearest_neighbor_spine.ipynb".
The first task is:
3. Nearest neighbor classification with L1 distance
We now compute nearest neighbors using the L1 distance (sometimes called Manhattan Distance).
For you to do: Write a function, NN_L1, which again takes as input the arrays trainx, trainy, and testx, and predicts labels for the test points using 1-nearest neighbor classification. For NN_L1, the L1 distance metric should be used. As before, the predicted labels should be returned in a numpy array with one entry per test point.
Notice that NN_L1 and NN_L2 may well produce different predictions on the test set.
 Code
```python
test function
testy_L2 = NN_L2(trainx, trainy, testx)
testy_L1 = NN_L1(trainx, trainy, testx)
print( type( testy_L1) )
print( len(testy_L1) )
print( testy_L1[40:50] )
print( all(testy_L1 == testy_L2) )
```
 Output
<class 'numpy.ndarray'>
62
[ 2.  2.  0.  0.  0.  0.  0.  0.  0.  0.]
False
Can you write Python code for it?


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_spine/Nearest_neighbor_spine.ipynb".
The first task is:
4. Test errors and the confusion matrix
Let's see if the L1 and L2 distance functions yield different error rates for nearest neighbor classification of the test data.
Can you write Python code for it?


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_spine/Nearest_neighbor_spine.ipynb".
The first task is:
After you are done, run the cell below to check your function. If an error is triggered, you should go back and revise your function.
Can you write Python code for it?


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_spine/Nearest_neighbor_spine.ipynb".
The first task is:
Now check your code by running the following cell.
Can you write Python code for it?


I am working on the file "DSE220x-MLFundamentals/Week-1/NN_spine/Nearest_neighbor_spine.ipynb".
The first task is:
We now load the dataset. We divide the data into a training set of 248 patients and a separate test set of 62 patients. The following arrays are created:

trainx : The training data's features, one point per row.
trainy : The training data's labels.
testx : The test data's features, one point per row.
testy : The test data's labels.

We will use the training set (trainx and trainy), with nearest neighbor classification, to predict labels for the test data (testx). We will then compare these predictions with the correct labels, testy.
Notice that we code the three labels as 0. = ’NO’, 1. = ’DH’, 2. = ’SL’.
Can you write Python code for it?


I am working on the file "DSMCER_Hw/dsmcer-hw-2-danielfather7/HW2 Tai-Yu Pan.ipynb".
The first task is:
1-1 Information

Five Terms:

PCE: Power conversion efficiency, means how much sunlight can be converted to electricity.
VOC: Voltage Open Circuit. The output Voltage of a photovoltaic(PV) under no load.
JSC: Short circuit current. The current through the solar cell when the voltage across the solar cell is zero.
E_Homo: Highest occupied molecular orbital.
E_Lumo: Lowest unoccupied molecular orbital.
(HOMO–LUMO gap is the energy difference between the HOMO and LUMO.)

Changes I made:

Change figure size to clearly display all the labels and titles.
Change the subplot() to make a 2x2 figure.
Use sign "," to display each point.
Give all figures labels and titles.
Chnage the x limit and y limit to clearly show the points at 0.
Add grid lines.

1-2 Contour plot
Can you write Python code for it?

#Read the file first.
data2 = pd.read_csv('ALA2fes.dat', delim_whitespace=True, comment='#', names=['phi','psi','file.free','der_phi','der_psi'])

#Take a look at the data.
data2.head()

#We should know how many columns there are before doing contour plot.
data2.shape

#Because it has 2500 columns, shape the data into 50x50 matrix.
N = 50
M = 50

X = np.reshape(data2.psi,[N,M])
Y = np.reshape(data2.phi,[N,M])
Z = np.reshape(data2['file.free']-data2['file.free'].min(),[N,M])

#I change unit from kJ/mol to kT, and it is appromately to time 2.5, so I pick this as spacer.
#Levels should contain all the FES, so I pick lines=42.
spacer = 2.5
lines = 42

levels = np.linspace(0,lines*spacer,num=(lines+1),endpoint=True)

fig2 = plt.figure(figsize=(5,5))
axes = fig.add_subplot(111)
plt.contour(X,Y,Z,levels)

#Give plot title and labels.
plt.title('$\Phi$ vs $\Psi$ on free energy surface')
plt.xlabel('$\Psi$')
plt.ylabel('$\Phi$')
plt.colorbar().ax.set_ylabel('FES (kT)')


I am working on the file "DSMCER_Hw/dsmcer-hw-2-danielfather7/HW2 Tai-Yu Pan.ipynb".
The first task is:
DNC: Begin Part 1
Part 1: Data Visualization
1-1: Plotting x-y data

Use the HCEPDB file to create a single 4x4 composite plot (not 4 separate figures).  The plots should contain the following data

Upper-left: PCE vs VOC
UR: PCE vs JCS
LL: E_HOMO vs VOC
LR: E_LUMO vs PCE

You should make the plots the highest quality possible and, in your judgement, ready for inclusion in a formal report or publication.

In the cell after you are finished making the plot add a markdown cell and add the following information

There are five terms above from the HCEPDB that relate to photovoltaic materials - define them as they pertain to molecules that could be used for energy conversion applications

Briefly explain the changes you made from the default plot and why you made them

1-2: Contour plotting

Use the ALA2fes.dat file to create a contour plot of the alanine dipeptide $\Phi$ vs $\Psi$ free-energy surface. Guidelines and information:
The energy scale in the data input file is on kJ/mol and the free-energy surface (FES) was collected at a temperature of 300K:
You should create a contour plot that draws contour lines spaced every kT in energy and stops drawing contours once all of the features can be clearly seen.
This is a slightly different visualization than what we drew in class which used shaded coloring to draw the contours

Annotate the cell so I can follow all the steps you are doing. The final energy plot need not be in kJ/mol (you can convert it to other energy or use units of kT if you prefer.

1-1 Plot
Can you write Python code for it?

data = pd.read_csv('HCEPD_100K.csv')

data.head()

#create a single 4x4 composite plot
#ref: https://plot.ly/matplotlib/subplots/
fig = plt.figure()
fig.set_figheight(10)
fig.set_figwidth(10)

ax1 = fig.add_subplot(221)
ax1.plot(data['voc'],data['pce'],',')
ax1.set_xlabel('VOC')
ax1.set_ylabel('PCE')
ax1.set_title('PCE vs VOC')
ax1.set_ylim([-0.5,12])
ax1.grid()

ax2 = fig.add_subplot(222)
ax2.plot(data['jsc'],data['pce'],',')
ax2.set_xlabel('JSC')
ax2.set_ylabel('PCE')
ax2.set_title('PCE vs JSC')
ax2.set_ylim([-0.5,12])
ax2.grid()

ax3 = fig.add_subplot(223)
ax3.plot(data['voc'],data['e_homo_alpha'],',')
ax3.set_xlabel('VOC')
ax3.set_ylabel('$E_{HOMO}$')
ax3.set_title('$E_{HOMO}$ vs VOC')
ax3.set_xlim([-0.1,1.8])
ax3.grid()

ax4 = fig.add_subplot(224)
ax4.plot(data['pce'],data['e_lumo_alpha'],',')
ax4.set_xlabel('VOC')
ax4.set_ylabel('$E_{LUMO}$')
ax4.set_title('$E_{LUMO}$ vs VOC')
ax4.set_xlim([-0.5,12])
ax4.grid()


I am working on the file "DSMCER_Hw/dsmcer-hw-3-statistics-danielfather7/HW3-Tai-Yu Pan.ipynb".
The first task is:
Also, I can plot without showing any outliers, shown below.
Can you write Python code for it?

plt.figure(figsize=(10,5))
plt.subplot(121)
plt.boxplot(data.Btu,showfliers = False)
plt.xticks([1], ['Chilled water usuage'])
plt.ylabel('Btu')
plt.grid()
plt.subplot(122)
plt.boxplot(data.kWh,showfliers = False)
plt.xticks([1],['Electricity usuage'])
plt.ylabel('kWh')
plt.ylim([-2,42])
plt.grid()

plt.suptitle('Chilled Water and Electricity Usage for Gerberding Hall (without any outliers)')


I am working on the file "DSMCER_Hw/dsmcer-hw-3-statistics-danielfather7/HW3-Tai-Yu Pan.ipynb".
The first task is:
As shown in above, the line(green one) in the box is the median(Q2, 50th percentile, the value separating the higher half from the lower half) of data. The lower boundary of the box is first quartile(Q1, 25th percentile), which is defined as the middle number between the smallest number and the median of the data. The upper boundary of the box is third quartile(Q3, 75th percentile), which is the middle value between the median and the highest value of the data. The interquartile range(IQR) is Q3 - Q1. The upper line outside the box is Q3 + 1.5 x IQR, and the lower line is Q1 - 1.5 x IQR. Data can be viewed as outliers outside these two lines. There is also a definition that the range [Q3+1.5 x IQR, Q3+3 x IQR] and [Q1-3 x IQR, Q1-1.5 x IQR] are called mild outiliers, and outside the 3 x IQR are called extreme outliers. I calculate these values below.
Can you write Python code for it?

IQR = data.quantile(q = 0.75) - data.quantile(q = 0.25)
upermild = data.quantile(q = 0.75) + 1.5*IQR
uperextre = data.quantile(q = 0.75) + 3*IQR
lowermild = data.quantile(q = 0.25) - 1.5*IQR
lowerextre = data.quantile(q = 0.25) - 3*IQR
print(lowerextre)
print(lowermild)
print(upermild)
print(uperextre)


I am working on the file "DSMCER_Hw/dsmcer-hw-3-statistics-danielfather7/HW3-Tai-Yu Pan.ipynb".
The first task is:
Comment:

Chilled water usuage is closer to normal distribution than electricity usuage is. And we can see that the CDF from the normal distribution is a good approxmation to the measured CDF, so I think it has a good ability to describe chilled water usuage. In contrast, in electricity usuage, both plots show that PDF and CDF from normal distribution don't fit the measured data very well.

DNC: Begin Part 3
Part 3: Hypothesis testing
The file energyuse.csv contains energy use data for 6 UW students. The data include electricity for lighting, all other electricity use and total electricity use. The final entry in the data file is the national average of the same values.
Please do the following
1) Formulate a statistical hypothesis to test about the three data sets and clearly state it
2) Perform a test a significance level of P=0.05 (make sure to clearly comment your work so I can follow what you are doing)
3) Clearly state the meaning of the results in plain language
Can you write Python code for it?

eusedata = pd.read_csv('energyuse.csv')
eusedata


I am working on the file "DSMCER_Hw/dsmcer-hw-3-statistics-danielfather7/HW3-Tai-Yu Pan.ipynb".
The first task is:
DNC: Begin Part 2
Part 2: Distributions

Your goal is to prepare a side-by-side plot describing the distribution of data from part 1 related to Gerberding Hall.

1) To complete the plot you should assume the data are normally distributed and determine the mean and standard deviation of the chilled water data series.
2) The left panel of the plot should be a normalized histogram of the chilled water data with an overlay of the PDF of the normal distribution based on the $\bar x$ and $s$ values from the data series.  The right panel of the plot should be contain two lines corresponding to the measured CDF and CDF from the normal distribution estimated from the data.
3) Repeat (2) for the electricity water
4) Comment in a Markdown cell on the ability of a normal distribution to describe this data
Can you write Python code for it?

from scipy.stats import norm
#left
x=np.arange(lowerextre[0],uperextre[0])
y = norm.pdf(x,data['Btu'].mean(),data['Btu'].std())
plt.figure(figsize=(10,5))
plt.subplot(121)
plt.plot(x, y, label = 'PDF of the normal distribution', linewidth = 1.5,color = 'k')
plt.hist(data['Btu'], bins = 20, normed = True, label = 'normalized histogram', color = '#668cff',edgecolor='k', lw=1.5)
plt.ylim([0,0.000027])
plt.grid()
plt.legend()
plt.xlabel('Chilled water usuage (Btu)')
plt.ylabel('Frequency')

#right
y = norm.cdf(x,data['Btu'].mean(),data['Btu'].std())
counts, bin_edges = np.histogram(data['Btu'], bins= 20, density=True)
cdf = np.cumsum(counts * np.diff(bin_edges))

plt.subplot(122)
plt.plot(x, y, label = 'CDF from the normal distribution',color = 'k')
plt.plot(bin_edges[1:], cdf, label = 'measured CDF',color = '#668cff')
plt.grid()
plt.legend(loc='upper right')
plt.xlabel('Chilled water usuage (Btu)')
plt.ylim([0,1.2])
plt.ylabel('Cumulative frequency')

plt.suptitle('Chilled Water Usage for Gerberding Hall')

#left
x=np.arange(lowerextre[1],uperextre[1])
y = norm.pdf(x,data['kWh'].mean(),data['kWh'].std())
plt.figure(figsize=(10,5))
plt.subplot(121)
plt.plot(x, y, label = 'PDF of the normal distribution', linewidth = 1.5,color = 'k')
plt.hist(data['kWh'], bins = 20, normed = True, label = 'normalized histogram', color = '#668cff',edgecolor='k', lw=1.5)
plt.ylim([0,0.14])
plt.grid()
plt.legend()
plt.xlabel('Electricity usuage (kWh)')
plt.ylabel('Frequency')
#right
y = norm.cdf(x,data['kWh'].mean(),data['kWh'].std())
counts, bin_edges = np.histogram(data['kWh'], bins= 20, density=True)
cdf = np.cumsum(counts * np.diff(bin_edges))

plt.subplot(122)
plt.plot(x, y, label = 'CDF from the normal distribution',color = 'k')
plt.plot(bin_edges[1:], cdf, label = 'measured CDF',color = '#668cff')
plt.grid()
plt.legend(loc='upper right')
plt.xlabel('Electricity usuage (kWh)')
plt.ylim([0,1.2])
plt.ylabel('Cumulative frequency')

plt.suptitle('Electricity Usage for Gerberding Hall')


I am working on the file "DSMCER_Hw/dsmcer-hw-3-statistics-danielfather7/HW3-Tai-Yu Pan.ipynb".
The first task is:
Explanation

In statistics, the p-value represents the probablity of extreme value by assuming H0 is true. When p-value is smaller enough(under desired significance level, $\alpha$), extreme value has very low probablity, but it still exists. Thus, we have confidence to reject the assumption H0 is true. That is, we can state that Ha is true. In contrast, when p-value is larger than $\alpha$, we can only state that we don't have enough evidence to reject H0.
In this case, p-value is 0.0192, which is smaller than $\alpha$ = 0.05, so we reject H0 and state that Ha is true. That is, the average total usuage for 6 UW students is not equal to the national Avg = 3.040 in this case.
I try to plot the p-value concept(based on my understanding) below. We can see that the average total usuage for 6 UW students falls into the rejection area, so we reject H0.
Can you write Python code for it?

x=np.arange(2.0,4.0,0.01)
#Normal distribution based on mu and sem
y = norm.pdf(x,3.040,sem)
plt.figure(figsize=(10,10))
plt.subplot(111)
plt.plot(x, y, label = 'Normal distribution by sem', linewidth = 1.5,color = 'k')
plt.xlabel('Total energy usuage')
plt.ylabel('Frequency')
plt.ylim([-0.15,2.25])
plt.xlim([1.8,4.2])

#tag mu
x = np.full(2,3.040)
plt.plot(x,[-0.15,2.0],'--',label = '$\mu$ = 3.040')

#tag mean
x = np.full(2,mean)
plt.plot(x,[-0.15, p],'--',label = 'mean = 2.363, p=0.199', color = 'g')
y = np.full(2,p)
plt.plot([1.8,mean],y,'--',color = 'g')

#tag alpha
[L,U]=stats.t.interval(.95,N-1,loc=3.040, scale=sem)
x = np.full(2,L)
plt.plot(x,[-0.15, 0.05],label = 'Rejection area based on alpha = 0.05', color = 'g')
y = np.full(2,0.05)
plt.plot([1.8,L],y,color = 'g')

plt.grid()
plt.legend(loc='upper right')


I am working on the file "DSMCER_Hw/dsmcer-hw-3-statistics-danielfather7/HW3-Tai-Yu Pan.ipynb".
The first task is:
Part 1-2
Can you write Python code for it?

data = pd.read_csv('GerberdingElectricityChilledWater.csv')
print(data.head())

plt.figure(figsize=(10,5))
plt.subplot(121)
data.boxplot(['Btu'])
plt.xticks([1], ['Chilled water usuage'])
plt.ylabel('Btu')
plt.subplot(122)
data.boxplot(['kWh'])
plt.xticks([1],['Electricity usuage'])
plt.ylabel('kWh')

plt.suptitle('Chilled Water and Electricity Usage for Gerberding Hall (with outliers)')

print(data.describe())


I am working on the file "DSMCER_Hw/dsmcer-hw-3-statistics-danielfather7/HW3-Tai-Yu Pan.ipynb".
The first task is:
The data fall in extreme outliers are most probably wrong data (or mostly not common one). Furthermore, data in this case can't be negative. Thus, I rearrange data by eliminating those values.
Can you write Python code for it?

for index, row in data.iterrows():
    if row['Btu'] > uperextre[0]:
        data.drop(index,inplace = True)
    elif row['kWh'] > uperextre[1]:
        data.drop(index,inplace = True)
    elif row['kWh'] < 0:
        data.drop(index,inplace = True)

#plot again
plt.figure(figsize=(10,5))
plt.subplot(121)
data.boxplot(['Btu'])
plt.xticks([1], ['Chilled water usuage'])
plt.ylabel('Btu')
plt.subplot(122)
data.boxplot(['kWh'])
plt.xticks([1],['Electricity usuage'])
plt.ylabel('kWh')

plt.suptitle('Chilled Water and Electricity Usage for Gerberding Hall (without extreme outliers)')


I am working on the file "DSMCER_Hw/dsmcer-hw4-danielfather7/HW4 skel.ipynb".
The first task is:
F-statistic comes from a hypothesis test, setting the null hypothesis that all $\beta_i = 0$ except for $\beta_0$. When there is no relationship between the response and predictors, we would expect the F-statistic to take on a value close to 1. On the other hand, if Ha (at least one $\beta_i$ is non-zero) is true, we expect F to be greater than 1. In this case, F is larger than a hundred thousand, suggesting that at least one of the predictors(mass, voc, $E_{LUMO}$) must be related to PCE.
Can you write Python code for it?

#beta 1
RSS0 = 0
for i in range(n):
    RSS0 += (harvard.pce[i] - regr2.intercept_ - regr2.coef_[1]*harvard.voc[i]
             - regr2.coef_[2]*harvard.e_lumo_alpha[i])**2
F = (RSS0-RSS)/(RSS/(n-p-1))
p = stats.t.sf(np.abs(F),n-2)*2
print("p value of beta1:",p,"(note: 0.0 means < 1e-6)")

#beta 2
RSS0 = 0
for i in range(n):
    RSS += (harvard.pce[i] - regr2.intercept_ - regr2.coef_[0]*harvard.mass[i]
             - regr2.coef_[2]*harvard.e_lumo_alpha[i])**2
F = (RSS0-RSS)/(RSS/(n-p-1))
p = stats.t.sf(np.abs(F),n-2)*2
print("p value of beta2:",p,"(note: 0.0 means < 1e-6)")

#beta 3
RSS0 = 0
for i in range(n):
    RSS += (harvard.pce[i] - regr2.intercept_ - regr2.coef_[0]*harvard.mass[i]
            - regr2.coef_[1]*harvard.voc[i])**2
F = (RSS0-RSS)/(RSS/(n-p-1))
p = stats.t.sf(np.abs(F),n-2)*2
print("p value of beta3:",p,"(note: 0.0 means < 1e-6)")