I am working on the file "Euler_approximate.ipynb".
The first task is:
Single-shock solution
Next we demonstrate the exactness property of the Roe solver by applying it to a case where the left and right states are connected by a single shock wave.
Can you write Python code for it?

M = 2.  # Mach number of the shock wave
gamma = 1.4
mu = 2*(M**2-1)/(M*(gamma+1.))
right = State(Density = 1.,
              Velocity = 0.,
              Pressure = 1.)
c_r = np.sqrt(gamma*right.Pressure/right.Density)

rho_l = right.Density * M/(M-mu)
p_l = right.Pressure * ((2*M**2-1)*gamma+1)/(gamma+1)
u_l = mu*c_r

left = State(Density = rho_l,
             Velocity = u_l,
            Pressure = p_l)

states = compare_solutions(left, right, solvers=['Exact','Roe'])

euler.phase_plane_plot(left, right, approx_states=states['roe'])


I am working on the file "EuroScipy 2015 demo.ipynb".
The first task is:
A signal can have multiple upstream signals.
Can you write Python code for it?

@react.connect('first_name', 'last_name')
def greet(first, last):
    print('hello %s %s!' % (first, last))


I am working on the file "EuroScipy 2015 demo.ipynb".
The first task is:
flexx.pyscript
Can you write Python code for it?

from flexx.pyscript import py2js

print(py2js('square = lambda x: x**2'))

def foo(n):
    res = []
    for i in range(n):
        res.append(i**2)
    return res
print(py2js(foo))

def foo(n):
    return [i**2 for i in range(n)]
print(py2js(foo))


I am working on the file "EuroScipy 2015 demo.ipynb".
The first task is:
flexx.react
Reactive programming uses signals to communicate between different components of an app, and provides easy ways to react to changes in the values of these signals.
The API for flexx.react consists of a few decorators to turn functions into signals. One signal is the input signal.
Can you write Python code for it?

from flexx import react

@react.input
def name(n='john doe'):
    if not isinstance(n, str):
        raise ValueError('Name must be a string')
    return n.capitalize()

name

@react.connect('name')
def greet(n):
    print('hello %s' % n)

name("almar klein")


I am working on the file "EuroVision-Circular-Graph.ipynb".
The first task is:
Define the list of edge weights:
Can you write Python code for it?

weights = list(map(int, G.es["weight"]))


I am working on the file "EuroVision-Circular-Graph.ipynb".
The first task is:
Get the edge list as a list of tuples, having as elements the end nodes indices:
Can you write Python code for it?

E = [e.tuple for e in G.es]# list of edges as tuple of node indices


I am working on the file "EuroVision-Circular-Graph.ipynb".
The first task is:
Get the node positions, assigned by the circular layout:
Can you write Python code for it?

pos = np.array(G.layout('circular'))
L = len(pos)


I am working on the file "EuroVision-Circular-Graph.ipynb".
The first task is:
In the sequel we define a few functions that lead to the edge definition as a Bézier curve:
dist(A,B) computes the distance between two 2D points, A, B:
Can you write Python code for it?

def p_dist(A, B):
    return np.linalg.norm(np.asarray(A)-np.asarray(B))


I am working on the file "EuroVision-Circular-Graph.ipynb".
The first task is:
The graph edges are colored with  colors depending on the distance between end nodes:
Can you write Python code for it?


I am working on the file "EuroVision-Circular-Graph.ipynb".
The first task is:
The list of parameters for interior control points:
Can you write Python code for it?


I am working on the file "Europe-Happiness.ipynb".
The first task is:
Define the hover text:
Can you write Python code for it?

text = [f'{country[k]}
Score: {round(score[k],3)}
World rank: {world_rk[k]}'  for  k in range(len(score))]


I am working on the file "Europe-Happiness.ipynb".
The first task is:
The bars are inserted along the unit circle, starting with the angular position $\pi/2$, in the anti-clockwise direction:
Can you write Python code for it?

theta = [np.pi/2+(2*k+1)*np.pi/72  for k in range(N)] # angular position of base bar centers

xi = [np.cos(theta[k]) for k in range(N)]# starting bar position
yi = [np.sin(theta[k]) for k in range(N)]

xf = [(bar_height[k]+1)*np.cos(theta[k]) for k in range(N)]#end bar position
yf = [(bar_height[k]+1)*np.sin(theta[k]) for k in range(N)]

xm = [(xi[k]+xf[k])*0.5   for k in range(N)]#mid bar position for inserting hovering text
ym = [(yi[k]+yf[k])*0.5   for k in range(N)]

xpos_t = [(bar_height[k]+1.32)*np.cos(theta[k]) for k in range(N)]#text position
ypos_t = [(bar_height[k]+1.32)*np.sin(theta[k]) for k in range(N)]


I am working on the file "Europe-Happiness.ipynb".
The first task is:
The decreased level of happiness is illustrated through  the Viridis colormap:
Can you write Python code for it?

import matplotlib.cm as cm
cmap = cm.viridis

vmin = score[-1]
vmax = score[0]
bar_colors = [map_z2color(score[k], cmap, score[-1], score[0]) for k in range(N)]


I am working on the file "Europe-Happiness.ipynb".
The first task is:
We visualize  countries happiness through a circular barchart. The height of each bar is proportional to the corresponding happiness score:
Can you write Python code for it?

bar_height = [score[k]/4.75 for k in range(N)]


I am working on the file "EvalReformatting-htmlpdf.ipynb".
The first task is:
But what does it look like?
We know the output name of the html file, so let's just display it directly.
Can you write Python code for it?

from IPython.display import display, HTML
import os

filename,ext = os.path.splitext(xl_filename)
assert ext == '.xlsx'
html_filename = filename + '.html'
wc_filename = filename + '-wordcloud.png'
html = open(html_filename).read()


I am working on the file "EvalReformatting-htmlpdf.ipynb".
The first task is:
ERRORS!!
At the moment, the below generates a ton of PangoFontDescription and PangoLayoutIter errors. These are annoying, but don't affect the output. Proceed with confidence.
Can you write Python code for it?

xl_filename = 'data/MGLFall2015/Analytical-Physics-I--(Fall-2015-16).xlsx'
stopwords = wordcloud.STOPWORDS.union(['class','course'])
makepdf.generatepdf(xl_filename,stopwords=stopwords)


I am working on the file "Evaluating Parameter Interdependence.ipynb".
The first task is:
Plot selected data:
Can you write Python code for it?

def get_range_square(X,Y):
    return [min(X + Y)-1, max(X + Y)+1]

# Plot the set vs. measured values of selected channels:
for nom in sorted(data.keys()):
    print nom
    for base_value in sorted(data[nom].keys()):
        print base_value
        X = data[nom][base_value]["X"]
        Y = data[nom][base_value]["Y"]
        ran = get_range_square(X,Y)
        pylab.ylim(ran)
        pylab.xlim(ran)
        pylab.grid(True)
        plt.axes().set_aspect(1)
        plt.title("{derived_param} with base {base}".format(
                derived_param=nom,
                base=base_value
            ))
        plt.scatter(X,Y,color='red')
        plt.plot(X,Y,color='blue')
        plt.show()


I am working on the file "Evaluating Parameter Interdependence.ipynb".
The first task is:
Set the operating parameters to the default values:
Can you write Python code for it?

def set_fpe_defaults(fpe):
    "Set the FPE to the default operating parameters, and outputs a table of the default values"
    defaults = {}
    for k in range(len(fpe.ops.address)):
        if fpe.ops.address[k] is None:
            continue
        fpe.ops.address[k].value = fpe.ops.address[k].default
        defaults[fpe.ops.address[k].name] = fpe.ops.address[k].default
    return defaults


I am working on the file "Evaluating Parameter Interdependence.ipynb".
The first task is:
Set up to plot:
Can you write Python code for it?

%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import numpy as np
import matplotlib.pyplot as plt
import pylab


I am working on the file "Evaluation Metrics.ipynb".
The first task is:
How many POIs are in the test set for your POI identifier?

(Note that we said test set! We are not looking for the number of POIs in the whole dataset.)
Can you write Python code for it?

numPoiInTestSet = len([p for p in y_test if p == 1.0])
print numPoiInTestSet


I am working on the file "Evaluation Metrics.ipynb".
The first task is:
If your identifier predicted 0. (not POI) for everyone in the test set, what would its accuracy be?
Can you write Python code for it?

from __future__ import division

1.0 - numPoiInTestSet/29


I am working on the file "Evaluation Metrics.ipynb".
The first task is:
Now you’ll add in training and testing, so that you get a trustworthy accuracy number. Use the train_test_split validation available in sklearn.cross_validation; hold out 30% of the data for testing and set the random_state parameter to 42 (random_state controls which points go into the training set and which are used for testing; setting it to 42 means we know exactly which events are in which set, and can check the results you get).
- What’s your updated accuracy?
Can you write Python code for it?

from sklearn import cross_validation

X_train, X_test, y_train, y_test = cross_validation.train_test_split(features, labels, test_size=0.30, random_state=42)

print len(X_train), len(y_train)
print len(X_test), len(y_test)

clf = tree.DecisionTreeClassifier()
t0 = time()
clf.fit(X_train, y_train)
print("done in %0.3fs" % (time() - t0))

pred = clf.predict(X_test)
print "Classifier with accurancy %.2f%%" % (submitAcc(X_test, y_test))


I am working on the file "EvaluationMachineLearning.ipynb".
The first task is:
Accuracy and Error Rate
The next cell illustrates how accuracy changes with respect to different distributions between two classes if the model always predict that an element belongs to class A.
$$
Accuracy=\frac{|tp+tn|}{|tp|+|tn|+|fp|+|fn|}\equiv\frac{|\mbox{correct predictions}|}{|\mbox{predictions}|}
$$
Can you write Python code for it?

# arrays for plotting
x=[] # samples in A
y=[] # samples in B
accuracies=[] # calculated accuracies for each distribution

# distributions between class A and B, first entry means 90% in A, 10% in B
distributions=[[90,10],[55,45],[70,30],[50,50],[20,80]]

for distribution in distributions:
    x.append(distribution[0])
    y.append(distribution[1])

    samplesA=np.ones((1,distribution[0])) # membership of class A is encoded as 1
    samplesB=np.zeros((1,distribution[1])) # membership of class B is encoded as 0
    # combine both arrays
    reality=np.concatenate((samplesA,samplesB),axis=None)
    # as said above, our model always associates the elements with class A (encoded by 1)
    prediction=np.ones((1,100))

    tpCount=0

    # count the true positives
    for (i,val) in enumerate(prediction[0]):
        if not reality[i]==val:
            pass
        else:
            tpCount+=1

    # calculate the accuracy and add the to the accuracies array for later visualization
    acc=float(tpCount+tnCount)/100.0
    accuracies.append(acc*1000) # the multiplication by 1000 is done for visualization purposes only
    print("Accuracy: %.2f"%(acc))

# plot the results as a bubble chart
plt.xlim(0,100)
plt.ylim(0,100)
plt.xlabel("Samples in A")
plt.ylabel("Samples in B")
plt.title("Accuracy of a Always-A Predictor")
plt.scatter(x, y, s=accuracies*100000,alpha=0.5)
#plt.savefig("test.png",format="png")
plt.show()


I am working on the file "EvaluationMachineLearning.ipynb".
The first task is:
Logarithmic Loss
The
$Logarithmic ~Loss=\frac{-1}{N}\sum_{i=1}^N\sum_{j=1}^M y_{ij}\log(p_{ij}) \rightarrow [0,\infty)$ penalizes wrong predicitions. For the sake of simplicity, we simply use the function provided by sklearn, a machine-learning toolkit for Python.
The manual will give you more details.
Can you write Python code for it?

from sklearn.metrics import log_loss

# the correct cluster for each sample, i.e., sample 1 is in class 0
y_true = [0, 0, 1, 1,2]
# the predictions: 1st sample is 90% predicted to be in class 0
y_pred = [[.9, .1,.0], [.8, .2,.0], [.3, .7,.0], [.01, .99,.0],[.0,.0,1.0]]
print(log_loss(y_true, y_pred))

# perfect prediction
y_perfect = [[1.0, .0,.0], [1.0, .0,.0], [.0, 1.0,.0], [0, 1.0,.0],[.0,.0,1.0]]
print(log_loss(y_true, y_perfect))

x=[]
y=[]
# the for loop modifies the first prediction of an element belonging to class 0 from 0 to 1
# in other words, from a wrong to a correct prediction
for i in range(1,11):
    r2=y_perfect
    r2[0][0]=float(i/10)
    x.append(r2[0][0])
    y.append(log_loss(y_true,r2))

# plot the result
plt.xlabel("Predicted Probability")
plt.ylabel("Logarithmic Loss")
plt.title("Does an object of class X belong do class X?")
plt.plot(x,y,'o-')
#plt.savefig("test.pdf",format="pdf")


I am working on the file "EvaluationMachineLearning.ipynb".
The first task is:
Next, we define out machine learning model with different layers. Roughly speaking, the function baseline_model() defines how the neural network looks like. For more details, see the documentation.
Can you write Python code for it?

# define baseline model
def baseline_model():
	# create model
	model = Sequential()
	model.add(Dense(num_pixels, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))
	model.add(Dense(num_classes, kernel_initializer='normal', activation='softmax'))
	# Compile model, use logarithmic loss for evaluation
	model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
	return model

# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

# flatten 28*28 images from the MNIST data set to a 784 vector for each image
num_pixels = X_train.shape[1] * X_train.shape[2]
X_train = X_train.reshape(X_train.shape[0], num_pixels).astype('float32')
X_test = X_test.reshape(X_test.shape[0], num_pixels).astype('float32')

# normalize inputs from 0-255 to 0-1
X_train = X_train / 255
X_test = X_test / 255

# one hot encode outputs
y_train = np_utils.to_categorical(y_train)
y_test = np_utils.to_categorical(y_test)
num_classes = y_test.shape[1]

# build the model
model = baseline_model()
# fit the model, i.e., start the actual learning
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=200, verbose=2)
# Final evaluation of the model
scores = model.evaluate(X_test, y_test, verbose=0)
# print the error rate of the algorithm
print("Baseline Error: %.2f%%" % (100-scores[1]*100))


I am working on the file "EvaluationMachineLearning.ipynb".
The first task is:
Next, we will illustrate our results.
Can you write Python code for it?

print(errorPerStep)
x=[]
y=[]
for e in errorPerStep:
    x.append(e)
    y.append(errorPerStep[e])

plt.xlabel("Training Samples")
plt.ylabel("Baseline Error (%)")
plt.plot(x,y,'o-')
plt.savefig("test.pdf",format="pdf")


I am working on the file "EvaluationMachineLearning.ipynb".
The first task is:
Overfitting
In the next cell, we will use very few training data up to the same amount of training data used before to illustrate the overfitting phenomenon.
ATTENTION! This will take some time.
Can you write Python code for it?

# define baseline model
def baseline_model():
	# create model
	model = Sequential()
	model.add(Dense(num_pixels, input_dim=num_pixels, kernel_initializer='normal', activation='relu'))
	model.add(Dense(num_classes, kernel_initializer='normal', activation='softmax'))
	# Compile model
	model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
	return model

# the steps indicate the size of the training sample
steps=[18,100,1000,5000,10000,20000,30000,40000,50000]
# this dict (basically a hashmap) holds the error rate for each iteration
errorPerStep=dict()

# fix random seed for reproducibility
seed = 7
np.random.seed(seed)

for step in steps:
    # load data
    (X_train, y_train), (X_test, y_test) = mnist.load_data()

    # limit the training data size to the current step, the : means "from 0 to step"
    X_train=X_train[0:step]
    y_train=y_train[0:step]

    # flatten 28*28 images to a 784 vector for each image
    num_pixels = X_train.shape[1] * X_train.shape[2]
    X_train = X_train.reshape(X_train.shape[0], num_pixels).astype('float32')
    X_test = X_test.reshape(X_test.shape[0], num_pixels).astype('float32')

    # normalize inputs from 0-255 to 0-1
    X_train = X_train / 255
    X_test = X_test / 255

    # one hot encode outputs
    y_train = np_utils.to_categorical(y_train)
    y_test = np_utils.to_categorical(y_test)
    num_classes = y_test.shape[1]

    # build the model
    model = baseline_model()
    # Fit the model
    model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=10, batch_size=200, verbose=2)
    # Final evaluation of the model
    scores = model.evaluate(X_test, y_test, verbose=0)
    print("Baseline Error: %.2f%%" % (100-scores[1]*100))
    errorPerStep[step]=(100-scores[1]*100)


I am working on the file "EvaluationMachineLearning.ipynb".
The first task is:
Setting Up the Experiment
In this example, we will rely on the NIST MNIST data set, a data set for the recognition of hand-written digits. MNIST is a data set that has been used by the NIST such as the discussed TREC campaign.
The following script will display some sample digits to give an example of the contents of the data set.
Can you write Python code for it?

# load (download if needed) the MNIST dataset of handwritten numbers
# we will get a training and test set consisting of bitmaps
# in the X_* arrays and the associated labels in the y_* arrays
(X_train, y_train), (X_test, y_test) = mnist.load_data()

# plot 4 images as gray scale images using subplots without axis labels
plt.subplot(221)
plt.axis('off')
# -1 inverts the image because of aesthetical reasons
plt.imshow(X_train[0]*-1, cmap=plt.get_cmap('gray'))
plt.subplot(222)
plt.axis('off')
plt.imshow(X_train[1]*-1, cmap=plt.get_cmap('gray'))
plt.subplot(223)
plt.axis('off')
plt.imshow(X_train[2]*-1, cmap=plt.get_cmap('gray'))
plt.subplot(224)
plt.axis('off')
plt.imshow(X_train[3]*-1, cmap=plt.get_cmap('gray'))

# show the plot
#plt.savefig("test.pdf",format="pdf")
plt.show()


I am working on the file "EvaluationMachineLearning.ipynb".
The first task is:
The graph indicates clearly that the baseline error decreases with the increase of training data. In other words, the overfitting effect is limited in relation to the amount of data the learning algorithm has seen.
To end the example, we will check how well the model can predict new input.
Can you write Python code for it?

(X_train, y_train), (X_test, y_test) = mnist.load_data()

# choose a random sample as our test image
test_im = X_train[25]
# display the image
plt.imshow(test_im.reshape(28,28)*-1, cmap=plt.get_cmap('gray'), interpolation='none')
plt.axis('off')

num_pixels = X_train.shape[1] * X_train.shape[2]
# as we are dealing with only one image, we have to restrict the array to a 1D * 784
test_im = test_im.reshape(1, num_pixels).astype('float32')

# let the model predict the image
r=model.predict(test_im)

itemindex = np.where(r[0]==1)
print("The model predicts: %i for the following image:"%itemindex[0])


I am working on the file "Evaluation_Notebooks/X1_default_setup.ipynb".
The first task is:
Curiously, the total output of agriculture is higher, if cities whithout agriculure die.
For the other outputs, leaving settlements without agriculture alive increases the output.
Apparently, climate variability has big impact on Income from agriculture, little impact on income from ecosystems and no impact on income from trade.
All of this leaves me wondering
* 1) if trade links are removed propperly when cities population shrinks.
It looks as if trade links only vanish if settlements die. Check this!
* 2) What impact the decay of the ecosystem and climate variability actually has on the income from ecosystem services. It seems like the impact ist negligible and in the original model, the impact came primarily from loss of settlements. More details on this in experiment 4
* 3) If this behavior depends on the weights of income from agriculture and ecosystem services relative to the income from trade. More on this in Experiment 2
Can you write Python code for it?

key=['total_trade_links']
level=['observables']
unstack = ['kill_cropless']

plotlines(data, level, key, unstack)


I am working on the file "Evaluation_Notebooks/X1_default_setup.ipynb".
The first task is:
To compare the two conditions better, lets have a look at the different source of income separately:
Can you write Python code for it?

key=['total_income_agriculture']
level=['observables']
unstack = ['kill_cropless']

plotlines(data, level, key, unstack)

key=['total_income_ecosystem']
level=['observables']
unstack = ['kill_cropless']

plotlines(data, level, key, unstack)

key=['total_income_trade']
level=['observables']
unstack = ['kill_cropless']

plotlines(data, level, key, unstack)


I am working on the file "Evaluation_Notebooks/X8_long_term_dynamics.ipynb".
The first task is:
The plots above show the long run data (aggregated from 50 runs) for total population and total income from trade
We see:

an overshoot and collapse behavior with fluctuation in the long run for possible income from trade,
some sort of saturation for intermediate income from taded and
an even quicker saturation for high possible income from trade.

To get some more information, a closer look at individual trajectories is required.
Can you write Python code for it?

location = '/home/jakob/Project_MayaSim/Python/output_data/X8_longterm_dynamics/results/all_trajectories'
all_data = pd.read_pickle(location)

print(all_data.columns)

pylab.rcParams['figure.figsize'] = (6, 12)
all_data.columns = range(0,2)
all_data.loc[6000, 1]

runs  = range(0, 2)
r_trades = all_data.index.values
fig = plt.figure()

annotations=['A)', 'B)', 'C)', 'D)']
for i, r_trade in enumerate(r_trades[[0, 1, 3]]):

    ax = fig.add_subplot(4, 1, i + 1)
    observable = 'total_population'

    for run in [0,1]:
        all_data.loc[r_trade, run][observable].plot(legend=(True if i == len(r_trades) - 1 else False))
    ax.set_xlim([0,2100])
    ax.annotate(annotations[i], xy=(0,0), xycoords='data', xytext=(0.92, 0.1), textcoords='axes fraction', fontsize=18)
    ax.set_ylabel('total population')
    if i == len(r_trades)-1:
        ax.set_xlabel('time in years')
        leg = ax.get_legend()
        for j, text in enumerate(leg.get_texts()):
            text.set_text('run {}'.format(j+1))

plt.savefig('longterm_population_development.pdf')


I am working on the file "Evaluation_Notebooks/X8_long_term_dynamics.ipynb".
The first task is:
The plots above show the total population of single two single runs for different possible trade income.
We see:

cyclical rise and fall for low possible income from trade,
stable society with high trade income with higher possible income from trade

This is in accordance with findings in archeological reserach that state that there have been cycles of growth and collapse in the Maya civilisation from 2000 BC to ~ 0 BC and that only after that in the socalled Classical Period a signifficant overshoot and collapse without recovery was observed.
In our model, this can be reproduced via an increase in trade efficiency, that leads to qualitatively different model behavior.
Can you write Python code for it?

pylab.rcParams['figure.figsize'] = (6, 12)
all_data.columns = range(0,2)
all_data.loc[6000, 1]

runs  = range(0, 2)
r_trades = all_data.index.values
fig = plt.figure()

annotations=['A)', 'B)', 'C)']
colors = ['#336600', '#66FF33', '#FF9900', 'black']
for i, r_trade in enumerate(r_trades[[0,1,3]]):

    ax = fig.add_subplot(3, 1, i + 1)
    observables = ['forest_state_3_cells', 'forest_state_2_cells', 'forest_state_1_cells', 'total_agriculture_cells']
    forest_data = all_data.loc[r_trade, run][observables]
    forest_data['forest_state_1_cells'] = forest_data['forest_state_1_cells'].sub(forest_data['total_agriculture_cells'])
    forest_data.columns = ['climax forest', 'secondary regrowth', 'cleared land', 'agriculture cells']
    run = 1
    ln1 = forest_data.plot.area(stacked=True,
                                legend=(True if i == len(r_trades) - 2 else False),
                                ax=ax,
                                color=colors)
    ax2 = ax.twinx()
    ln2 = all_data.loc[r_trade, run]['total_population'].plot(legend=(True if i == len(r_trades) - 2 else False),
                                                              ax=ax2,
                                                              color='k')
    #print(all_data.loc[r_trade, run][observables])
    ax.set_xlim([0,1000])
    ax.annotate(annotations[i], xy=(0,0), xycoords='data', xytext=(0.9, 0.9), textcoords='axes fraction', fontsize=18)
    ax2.set_ylabel('total population')
    ax.set_ylabel('forest state')
    ax.set_ylim([0, 101000])
    if i == len(r_trades)-2:
        ax.set_xlabel('time in years')
        lgd2 = ax.legend(loc=1, bbox_to_anchor=[1, .4])
        lgd1 = ax2.legend(loc=1, bbox_to_anchor=[1., .55])
#fig.tight_layout()
fig.savefig('longterm_population_development.pdf', transparent=True, dpi=200)


I am working on the file "EvapEnergyBalance-Meteorology-SnowDegreeDay.ipynb".
The first task is:
Initialize the components with cfg files that, for simplicity, use the same time step and run duration:
Can you write Python code for it?

%cd input

evp.initialize('evap_energy_balance-1.cfg')
met.initialize('meteorology-2.cfg')
sno.initialize('snow_degree_day-1.cfg')


I am working on the file "EvapEnergyBalance-Meteorology-SnowDegreeDay.ipynb".
The first task is:
Plot snow depth versus time.
Can you write Python code for it?

%matplotlib inline
from matplotlib import pyplot as plt

fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(12, 5))

snow_depth_plot = axes[0].plot(time[1:], snow_depth[1:], 'b')
axes[0].set_title('Snow depth versus time')
axes[0].set_xlabel('Time [s]')
axes[0].set_ylabel('Snow depth [m]')

evap_flux_plot = axes[1].plot(time[1:], evap_flux[1:], 'r')
axes[1].set_title('Evaporative flux versus time')
axes[1].set_xlabel('Time [s]')
axes[1].set_ylabel('Evaporative flux [m s-1]')


I am working on the file "EvapEnergyBalance-Meteorology-SnowDegreeDay.ipynb".
The first task is:
Store initial values of time, snow depth, and air temperature:
Can you write Python code for it?

time = [met.get_current_time()]
snow_depth = [sno.get_value('snowpack__depth').max()]
air_temp = [met.get_value('atmosphere_bottom_air__temperature').max()]
evap_flux = [evp.get_value('land_surface_water__evaporation_volume_flux').max()]


I am working on the file "EvapEnergyBalance-Meteorology-SnowEnergyBalance.ipynb".
The first task is:
Initialize the components with cfg files that, for simplicity, use the same time step and run duration:
Can you write Python code for it?

%cd input

evp.initialize('evap_energy_balance-1.cfg')
met.initialize('meteorology-2.cfg')
sno.initialize('snow_energy_balance-1.cfg')


I am working on the file "EvapEnergyBalance-Meteorology-SnowEnergyBalance.ipynb".
The first task is:
Plot snow depth versus time.
Can you write Python code for it?


I am working on the file "EvapEnergyBalance-Meteorology-SnowEnergyBalance.ipynb".
The first task is:
Run the coupled models to completion. In each time step, perform the following actions:

Get variables from Meteorology; set into SnowEnergyBalance
Advance SnowEnergyBalance
Get variables from SnowEnergyBalance; set into Meteorology
Advance Meteorology
Get variables from Meteorology and SnowEnergyBalance; set into EvapEnergyBalance
Advance EvapEnergyBalance
Can you write Python code for it?

count = 1
while evp.get_current_time() < evp.get_end_time():
    rho_air = met.get_value('atmosphere_bottom_air__mass-per-volume_density')
    Cp_air = met.get_value('atmosphere_bottom_air__mass-specific_isobaric_heat_capacity')
    T_air = met.get_value('atmosphere_bottom_air__temperature')
    P_snow = met.get_value('atmosphere_water__snowfall_leq-volume_flux')
    T_surf = met.get_value('land_surface__temperature')
    Q_sum = met.get_value('land_surface_net-total-energy__energy_flux')
    rho_H2O = met.get_value('water-liquid__mass-per-volume_density')
    sno.set_value('atmosphere_bottom_air__mass-per-volume_density', rho_air)
    sno.set_value('atmosphere_bottom_air__mass-specific_isobaric_heat_capacity', Cp_air)
    sno.set_value('atmosphere_bottom_air__temperature', T_air)
    sno.set_value('atmosphere_water__snowfall_leq-volume_flux', P_snow)
    sno.set_value('land_surface__temperature', T_surf)
    sno.set_value('land_surface_net-total-energy__energy_flux', Q_sum)
    sno.set_value('water-liquid__mass-per-volume_density', rho_H2O)

    sno.update(sno.get_time_step()*count)

    rho_snow = sno.get_value('snowpack__z_mean_of_mass-per-volume_density')
    h_snow = sno.get_value('snowpack__depth')
    h_swe = sno.get_value('snowpack__liquid-equivalent_depth')
    SM = sno.get_value('snowpack__melt_volume_flux')
    met.set_value('snowpack__z_mean_of_mass-per-volume_density', rho_snow)
    met.set_value('snowpack__depth', h_snow)
    met.set_value('snowpack__liquid-equivalent_depth', h_swe)
    met.set_value('snowpack__melt_volume_flux', SM)

    met.update(met.get_time_step()*count)

    T_air = met.get_value('atmosphere_bottom_air__temperature')
    Qe = met.get_value('atmosphere_bottom_air_land_net-latent-heat__energy_flux')
    Q_sum = met.get_value('land_surface_net-total-energy__energy_flux')
    T_surf = met.get_value('land_surface__temperature')
    h_snow = sno.get_value('snowpack__depth')
    evp.set_value('atmosphere_bottom_air__temperature', T_air)
   ...


I am working on the file "EventDec/event_dec/notebook/0_Cleaning.ipynb".
The first task is:
Check DF and Output
Checking
Can you write Python code for it?


I am working on the file "EventDec/event_dec/notebook/0_Cleaning.ipynb".
The first task is:
Cleaning
Can you write Python code for it?

raw_data.date = pd.to_datetime(raw_data.date, format="%d.%m.%Y")
raw_data.start_time = pd.to_datetime(raw_data.start_time, format="%H:%M").dt.time
raw_data.duration = pd.to_datetime(raw_data.duration, format="%H:%M").dt.time
raw_data["weekday"] = raw_data.date.dt.weekday_name
raw_data.ticket_prize = raw_data.ticket_prize.replace("[\€,]", "", regex=True).astype(int)


I am working on the file "EventDec/event_dec/notebook/0_Cleaning.ipynb".
The first task is:
Enhancement
Buzzwords
Can you write Python code for it?

# Generate Buzzword List
buzzword_list = []

def parse_buzzword_file(file_path, buzzword_list):
    """
        parses textfile and appends word to buzzword list
    """
    with open(file_path, "r") as f:
        content = f.readlines()
        for buzzword in content:
            buzzword_list.append(buzzword.strip("\n").lower())
    return buzzword_list

buzzwords_wiki = parse_buzzword_file("../data/buzzwords_wiki.txt", buzzword_list)
buzzwords_de = parse_buzzword_file("../data/buzzwords_de.txt", buzzword_list)
buzzwords_personal = parse_buzzword_file("../data/buzzwords_personal.txt", buzzword_list)

print("Buzzword Count:", len(buzzword_list))

# Pickling Buzzword List for App
file_path = "../data/buzzword_list.pkl"
with open(file_path, "wb") as f:
    pickle.dump(buzzword_list, f)

# Create binary colunms (buzzwordy_title, buzzwordy_organizer)
buzzword_mask_title = raw_data.title.str.contains("|".join(buzzword_list), case=False)
buzzword_mask_organizer =  raw_data.organizer.str.contains("|".join(buzzword_list), case=False)

raw_data["buzzwordy_title"] = np.where(buzzword_mask_title, 1, 0)
raw_data["buzzwordy_organizer"] = np.where(buzzword_mask_organizer, 1, 0)


I am working on the file "EventDec/event_dec/notebook/0_Cleaning.ipynb".
The first task is:
Input
Privacy restriction:
Original (personal) dataset not in Repo. Provide self collected features and ratings about attended events in "../data/events.csv" (schema info in REFS.md)
Can you write Python code for it?

data_f = "../data/events.csv"
raw_data = pd.read_csv(data_f)

print(raw_data.shape)
print(raw_data.dtypes)
raw_data.head()


I am working on the file "EventDec/event_dec/notebook/0_Cleaning.ipynb".
The first task is:
Output
Can you write Python code for it?

# Pickling DF
file_path = "../data/events_df.pkl"
raw_data.to_pickle(file_path)


I am working on the file "EventDec/event_dec/notebook/1_Exploration.ipynb".
The first task is:
DFs for Modeling
Can you write Python code for it?

# Minimal Features Model
model01_cols = [u"main_topic", u"buzzwordy_title", u"buzzwordy_organizer", u"days", u"weekday", u"city",
                u"country", u"distance", u"ticket_prize", u"rating"]
df_model01 = df_cleaned[model01_cols]

df_model01.head()


I am working on the file "EventDec/event_dec/notebook/1_Exploration.ipynb".
The first task is:
Dummie Encoding
Can you write Python code for it?

df_model01 = pd.get_dummies(df_model01, prefix=["main_topic", "weekday", "city", "country"])


I am working on the file "EventDec/event_dec/notebook/1_Exploration.ipynb".
The first task is:
Exploration
Can you write Python code for it?

print("Stats (continuous Vars):")
print(df.describe())
print("")
print("NaN values count:")
print(df.isnull().sum())

for col in df:
    print(df[col].value_counts())
    print("")

df.groupby(df.main_topic).mean()[["distance", "rating"]]

df.groupby(df.city).mean()[["distance", "rating"]]


I am working on the file "EventDec/event_dec/notebook/1_Exploration.ipynb".
The first task is:
Input
Privacy restriction:
Original (personal) cleaned DF not in Repo. Go through nb "0_Cleaning" with self provided data to reproduce pickled DF of attended events ("events_df.pkl").
For further steps: Repo contains pickled DF for modeling (nb "3_Modeling"), in which private informations are elimated.
Can you write Python code for it?

file_path = "../data/events_df.pkl"
df = pd.read_pickle(file_path)

print(df.shape)
print(df.dtypes)
df.head()


I am working on the file "EventDec/event_dec/notebook/1_Exploration.ipynb".
The first task is:
Preparation for Modeling
Missing Values
Can you write Python code for it?

df_cleaned = df.fillna("missing") # Nan in String val Cols

print(df_cleaned.isnull().sum())


I am working on the file "EventDec/event_dec/notebook/2_Modeling.ipynb".
The first task is:
=> Generalization acceptable, but not good (as expected from a Decision Tree)
Persist Model
Can you write Python code for it?

from sklearn.externals import joblib

file_path = "../data/model_trained.pkl"

joblib.dump(model, file_path)


I am working on the file "EventDec/event_dec/notebook/2_Modeling.ipynb".
The first task is:
=> Indeed only slightly better than random guess, ~ 0.05 percentage points (AUC)
Can you write Python code for it?

from sklearn.model_selection import cross_val_score

scores_benchmark = cross_val_score(model, X, y, cv=5) # 5 folds

print("Cross-Val Scores (Accuracy): {}".format(scores_benchmark))
print("Cross-Val Mean (Accuracy): {}".format(scores_benchmark.mean()))


I am working on the file "EventDec/event_dec/notebook/2_Modeling.ipynb".
The first task is:
=> Not so good in predicting class 1 ("worth attending), better in predicting class 0. Weighted average of precision and recall 0.62 (F1 Score)
ROC Curve
Can you write Python code for it?

from sklearn.metrics import roc_curve, roc_auc_score

fpr, tpr, thresholds = roc_curve(y_test, y_pred)

plt.plot(fpr, tpr, label="ROC")
plt.plot([0, 1], c="r", label="Chance level")
plt.title("ROC Curve")
plt.xlabel("False Positive Rate")
plt.ylabel("Recall")
plt.legend(loc=4)

plt.savefig("plots/ROC_Curve_Model.png", dpi=180)
plt.show()

print("AUC: {:.3f}".format(roc_auc_score(y_test, y_pred)))


I am working on the file "EventDec/event_dec/notebook/2_Modeling.ipynb".
The first task is:
=> Really bad perfomance
Model
Decision Tree CV
<= Ability to visualize, hasn't performed much worse than other applied models
Parameter Tuning
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV

parameter_grid = {"criterion": ["gini", "entropy"],
                  "max_depth": [None, 1, 2, 3, 4, 5, 6],
                  "min_samples_leaf": list(range(1, 14)),
                  "max_leaf_nodes": list(range(3, 25))}

grid_search = GridSearchCV(DecisionTreeClassifier(presort=True), parameter_grid, cv=5) # 5 fold cross-val
grid_search.fit(X_train, y_train)

print("Score (Accuracy): {:.3f}".format(grid_search.score(X_test, y_test)))
print("Best Estimator: {}".format(grid_search.best_estimator_))
print("Best Parameters: {}".format(grid_search.best_params_))


I am working on the file "EventDec/event_dec/notebook/2_Modeling.ipynb".
The first task is:
Conversion for Scikit-learn
Feature selection based on expert knowledge. Model-based hardly interpretable, at least confirmed "binned_distance" as relevant feature.
Can you write Python code for it?

feature_names = ["buzzwordy_title", "main_topic_Daten", "binned_distance"]
X = df[feature_names].values
y = df.rating.map(lambda x: 1 if x>5 else 0).values # binary target: >5 (better as all the same) was worth attending

print("X:", X.shape, "y:", y.shape)


I am working on the file "EventDec/event_dec/notebook/2_Modeling.ipynb".
The first task is:
Model Evaluation
Evaluation Scores
Can you write Python code for it?

from sklearn.metrics import classification_report

y_pred = model.predict(X_test)
print(classification_report(y_test, y_pred))


I am working on the file "EventDec/event_dec/notebook/2_Modeling.ipynb".
The first task is:
Modeling
Model
Can you write Python code for it?

from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeClassifier

linreg = LinearRegression() # Benchmark model
dec_tree = DecisionTreeClassifier() # Actual model


I am working on the file "EventDec/event_dec/notebook/2_Modeling.ipynb".
The first task is:
Train-Test Split
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=23,
                                                    test_size=0.5) # 50% split, small dataset size


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
... and their difference
Can you write Python code for it?

plt.plot(lc.time, lc.counts - lc_new.counts, 'g-', drawstyle="steps-mid")
plt.xlabel('Times')
plt.ylabel('Counts')


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
As can be seen from the figure above, the recovered light curve is aligned with the original light curve.
Simulating Energies
In order to simulate photon energies, a spectral distribution needs to be passed.
The spectrum input is a two-dimensional array, with the energies in keV in the first dimension and the number of counts in the second. The count array will be normalized before the simulation: the raw counts do not matter, but only the ratio of the counts in each bin to the total.
Again, the energies are simulated using an inverse CDF method.
Can you write Python code for it?

spectrum = [[1, 2, 3, 4, 5, 6],[1000, 2040, 1000, 3000, 4020, 2070]]

ev = EventList(time=np.sort(np.random.uniform(0, 1000, 12)))
ev.simulate_energies(spectrum)

ev.energy


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
Create event list object by passing arrival times as argument.
Can you write Python code for it?

ev = EventList(times, mjdref=mjdref)
ev.time


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
Creating a light curve from an EventList object
After simulating event list, the original light curve can be recovered. Let's demonstrate by creating a light curve.
Can you write Python code for it?

dt = 1.
times = np.arange(50)
counts = np.floor(np.random.rand(50)*50000)
lc = Lightcurve(times, counts, skip_checks=True, dt=1.)


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
Import some relevant stingray classes.
Can you write Python code for it?

from stingray import EventList, Lightcurve


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
It is always recommended to specify the good time intervals (GTIs) of the event list, as the time intervals where the instrument was fully operational. If not specified, GTIs are defined automatically as the first and the last event time.
Can you write Python code for it?

gti = [[0, 4]]
ev = EventList(times, gti=gti, energy=energy, mjdref=mjdref)


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
Joining EventLists
Two event lists can also be joined together. If the GTI do not overlap, the event times and GTIs are appended. Otherwise, the GTIs are crossed (i.e., only the overlapping parts are saved) and the events merged together.
Can you write Python code for it?


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
Loading and writing EventList objects
We made it possible to save and load data in a number of different formats.
The general syntax is
```
ev = EventList.read(filename, format)
ev.write(filename, format)
```
There are three main blocks of formats that might be useful:

(read-only) HEASoft-compatible formats -> read event data from HEASOFT-supported missions

pickle: reading and saving EventLists from/to Python pickle objects

Any format compatible with astropy.table.Table objects.

Loading an EventList from an X-ray observation in HEASoft-compatible format
Loading event data from HEASoft-supported missions in FITS format is easy. It's sufficient to use the read method with hea or, equivalently, ogip, as format.
Beware: please use hea or ogip, not fits! It would make the roundrip to Astropy tables more complicated, as Astropy supports a generic FITS writer which is not necessarily compatible with HEASoft.
Can you write Python code for it?

ev = EventList.read('events.fits', 'ogip')


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
Of course, these objects can be converted back to event lists. The user should be careful in defining the proper column names and metadata so that the final object is a valid EventList
Can you write Python code for it?

table_ev = EventList.from_astropy_table(table)
table_ts = EventList.from_astropy_timeseries(timeseries)

table_ev.time, table_ts.time


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
One can add all sorts of data to the EventList object, it is very flexible. In general, we suggest to stick with easily interpretable attributes, like energy or pi.
ev.energy = [0., 3., 4., 20.]
is the same as
Can you write Python code for it?

energy = [0., 3., 4., 20.]
ev = EventList(times, energy=energy, mjdref=mjdref)


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
Recover original light curve curve using to_lc() method. Here, dt defines time resolution, tstart the starting time, and tseg the total time duration.
Can you write Python code for it?


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
Roundtrip to Astropy-compatible formats
EventList has the following methods that allow an easy roundtrip to Astropy objects: to_astropy_table, to_astropy_timeseries, from_astropy_table, from_astropy_timeseries
This allows a better interoperability with the Astropy ecosystem.
In this roundtrip, a Table or Timeseries object is created, having as columns time and all other attributes of the same size (e.g. pi, energy), and the rest of the attributes (e.g. gti, mjdref) in the table's metadata.
Can you write Python code for it?


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
Roundtrip to Astropy-compatible formats
If the read and write methods receive a format which is not hea, ogip, or pickle, the event list is transformed into an Astropy Table object with the methods described above, and the readers and writers from the Table class are used instead. This allows to extend the save/load operations to a large number of formats, including hdf5 and enhanced CSV (ascii.ecsv).
Note that columns coming from the EVENTS (or equivalent) fits extension, those having the same length as time, when converting to astropy tables they become columns of the table. All the others, including gti, are treated as metadata.
Care should be used in selecting formats that preserve metadata. For example, simple CSV format loses all metadata, including MJDREF, GTIs etc.
Can you write Python code for it?

ev.write("events.hdf5", "hdf5")
ev3 = EventList.read("events.hdf5", "hdf5")
ev3.time[:10]

# Try the round trip again to verify that everything works

ev.write("events.ecsv", "ascii.ecsv")
ev4 = EventList.read("events.ecsv", "ascii.ecsv")
!cat events.ecsv
ev4.time[:10]


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
Roundtrip to pickle objects
It is possible to save and load eventlist objects using pickle.
Can you write Python code for it?

ev.write("events.p", "pickle")
ev2 = EventList.read("events.p", "pickle")

np.allclose(ev2.time, ev.time), np.allclose(ev2.gti, ev.gti)


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
Simulating EventList from Lightcurve
An arguably better way is having proper random events, reproducing the initial light curve within the errors. Stingray does this by using the inverse CDF method, using the light curve as a binned probability distribution.
Please note that in this case we will have to create the EventList object before (in technical terms, simulate_times is not a static method.). See simulation tutorial for more details.
Can you write Python code for it?

ev = EventList()
ev.simulate_times(lc)
ev.time


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
Transforming a Lightcurve into an EventList.
Event lists can be obtained from light curves, where the standard followed is as follows: as many events are created as the counts in the lightcurve at the time specified by time bins.
To demonstrate this, let us define a light curve.
Can you write Python code for it?

times = np.arange(3)
counts = np.floor(np.random.rand(3)*5)
lc = Lightcurve(times, counts, skip_checks=True, dt=1.)

lc.time, lc.counts


I am working on the file "EventList/EventList Tutorial.ipynb".
The first task is:
When converting to Timeseries, times are transformed into astropy.time.TimeDelta objects.
Can you write Python code for it?

timeseries = ev.to_astropy_timeseries()
timeseries

table.meta


I am working on the file "EverythingData.ipynb".
The first task is:
Data Wrangling with Python and Pandas (tutorial)
Introduction: http://pandas.pydata.org/pandas-docs/stable/10min.html
Tutorial on data wrangling:
https://github.com/jvns/pandas-cookbook
Can you write Python code for it?

# Run some exploration on tutorial
%matplotlib inline
import pandas as pd
import matplotlib
matplotlib.style.use('ggplot')
#montreal weather
weather_url = "https://raw.githubusercontent.com/jvns/pandas-cookbook/master/data/weather_2012.csv"

weather_2012_final = pd.read_csv(weather_url, parse_dates='Date/Time', index_col='Date/Time')
weather_2012_final['Temp (C)'].plot(figsize=(15, 6))
#weather_2012_final.to_hdf()

print weather_2012_final[weather_2012_final['Weather'] == 'Cloudy']['Temp (C)'].median()
print weather_2012_final[weather_2012_final['Weather'] == 'Snow']['Temp (C)'].median()
weather_2012_final.to_hdf('ciao.h5', compression='blocs')


I am working on the file "EverythingData.ipynb".
The first task is:
One benchmark a day
Goldmine: https://github.com/rasbt/One-Python-benchmark-per-day/tree/master/
Try:

6 different ways for counting elements using a dictionary

Python vs Cython vs Numba

Memory, cores, I/O

Latency: Register, Cache, RAM, Disk (SSD/HDD), network
Out of core vs distributed
Embarrassingly parallel problems (shell/python parallel)
Can you write Python code for it?

from IPython.display import Image
Image(url='http://i.imgur.com/k0t1e.png')


I am working on the file "EverythingData.ipynb".
The first task is:
Scale:

Parallel : doc and example
Multiprocess : http://sebastianraschka.com/Articles/2014_multiprocessing_intro.html
Big Data (Spark and BDAS) https://spark.apache.org/examples.html
Can you write Python code for it?

!pip install ipyparallel

# Example: Run some parallel code
from ipyparallel import Client
client = Client(profile='mycluster')
%px print("Hello from the cluster engines!")


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
As noted above, the boundaries between the facies classes are not all sharp, and some of them blend into one another.  The error within these 'adjacent facies' can also be calculated.  We define an array to represent the facies adjacent to each other.  For facies label i, adjacent_facies[i] is an array of the adjacent facies labels.
Can you write Python code for it?

adjacent_facies = np.array([[1], [0,2], [1], [4], [3,5], [4,6,7], [5,7], [5,6,8], [6,7]])

def accuracy_adjacent(conf, adjacent_facies):
    nb_classes = conf.shape[0]
    total_correct = 0.
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
        for j in adjacent_facies[i]:
            total_correct += conf[i][j]
    return total_correct / sum(sum(conf))

print('Facies classification accuracy = %f' % accuracy(conf))
print('Adjacent facies classification accuracy = %f' % accuracy_adjacent(conf, adjacent_facies))

display_adj_cm(conf, facies_labels, adjacent_facies,
           display_metrics=True, hide_zeros=True)


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
Considering adjacent facies, the F1 scores for all facies types are above 0.9, except when classifying SiSh or marine siltstone and shale.  The classifier often misclassifies this facies (recall of 0.87), most often as wackestone.
Now we can train the classifier using the entire data set
Can you write Python code for it?

mv.fit(X,y)
print('Predicted F1-Score: {}'.format(mv.score(X,y)))


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
Feature-imputation
Can you write Python code for it?

from sklearn.preprocessing import Imputer
def feat_imput(training_data, drop_columns):
    """adopted RandomForest feature regressor by ar4"""
    #imp = Imputer(missing_values='NaN', strategy='mean', axis=0)
    #X = training_data.drop(drop_columns, axis=1).values
    #X_imputed = imp.fit_transform(X)
    training_data_imputed = training_data.copy()
    #for ift,ft in enumerate(training_data.columns.drop(drop_columns)):
    #    training_data_imputed[ft] = X_imputed[:,ift]

    reg = RandomForestRegressor(max_features='sqrt', n_estimators=50)
    feat_labels =training_data.columns.drop(drop_columns)
    DataImpAll = training_data[feat_labels].copy()
    DataImp = DataImpAll.dropna(axis = 0, inplace=False)
    Ximp=DataImp.loc[:, DataImp.columns != 'PE']
    Yimp=DataImp.loc[:, 'PE']
    Y_PE = DataImpAll.loc[:, 'PE']
    reg.fit(Ximp, Yimp)
    Y_PE[DataImpAll.PE.isnull()] = reg.predict(DataImpAll.loc[DataImpAll.PE.isnull(),:].drop('PE',axis=1,inplace=False))
    training_data_imputed['PE'] = Y_PE
    #X[np.array(DataImpAll.PE.isnull()),4] = reg.predict(DataImpAll.loc[DataImpAll.PE.isnull(),:].drop('PE',axis=1,inplace=False))

    return training_data_imputed

drop_columns= ['Formation', 'Well Name', 'Depth','Facies']
depth = training_data['Depth']
well = data['Well Name'].values
training_data = feat_imput(training_data,drop_columns)


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
Finally we can write out a csv file with the well data along with the facies classification results.
Can you write Python code for it?


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
Finally we predict facies labels for the unknown data, and store the results in a Facies column of the test_data dataframe.
Can you write Python code for it?

#predict facies of unclassified data
y_unknown = mv.predict(X_unknown)
well_data['Facies'] = y_unknown
well_data

well_data['Well Name'].unique()


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
Looks like not all features are nearly important, e.g. may consider dropping those with relative importance below 2.3%
Can you write Python code for it?

if sklearn_version < '0.18':
    X_selected = forest.transform(X, threshold=0.023)
else:
    from sklearn.feature_selection import SelectFromModel
    sfm = SelectFromModel(forest, threshold=0.023, prefit=True)
    X_selected = sfm.transform(X)

X_selected = X

from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
        X_selected,y, test_size=0.1, random_state=42)

#tuned forest
forest =  RandomForestClassifier(
    n_estimators = 303,
    min_samples_leaf = 1,
    oob_score = True,
    random_state=50, n_jobs=-1)
forest.fit(X_train, y_train)
y_pred = forest.predict(X_test)
print ("F1-score: %.4g" % f1_score(y_test,y_pred, average='micro'))

#tuned xgb
xg = XGBClassifier(learning_rate =0.005,
 n_estimators=1888,
 max_depth=10,
 min_child_weight=1,
 gamma=0.2,
 subsample=0.9,
 colsample_bytree=0.7,
 reg_alpha=0,
 nthread=-1,
 objective='multi:softprob',
 scale_pos_weight=1,
 seed=43)

xg.fit(X_train, y_train,eval_metric=pre_scorer)
y_pred = xg.predict(X_test)
print ("F1-score: %.4g" % f1_score(y_test,y_pred, average='micro'))

print('Predicted F1-Score: {}'.format(xg.score(X_test,y_test)))

#tuned SVM
pipe_svm = Pipeline([('scl',StandardScaler()),('clf',svm.SVC(kernel = 'rbf', random_state=0, gamma=0.1, C=100))])


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
Next: make labels and features vectors
Can you write Python code for it?


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
Split the training data into training and test sets. Let's use 10% of the data for the test set.
Can you write Python code for it?

from sklearn.cross_validation import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
        X,y, test_size=0.1, random_state=42)

# the scores
from sklearn.model_selection import cross_val_score
from sklearn.metrics import precision_score, recall_score, f1_score, make_scorer
pre_scorer= make_scorer(score_func=f1_score, greater_is_better=True,average = 'micro')
#the scorer is f1_scrore with micro averaging, i.e.
from sklearn.cross_validation import StratifiedKFold
kfold = StratifiedKFold(y_train, n_folds=10, shuffle=True, random_state=7)
# stratified kfold cross-validation keeps the classes balanced in each fold.

from sklearn.model_selection import GridSearchCV


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
The combined majority vote classifier is constructed out of 3-individual classifiers
Can you write Python code for it?

from sklearn.ensemble import VotingClassifier
mv = VotingClassifier(estimators=[('forest',forest),('XGBoost',xg),('svn',pipe_svm)])

clf_labels = [ 'Random Forest' ,'XGBoost','SVN', 'Majority-Vote']

print('10-fold cross validation:\n')
for clf, label in zip([forest,xg,pipe_svm,mv], clf_labels):
    scores = cross_val_score(estimator=clf,
                             X=X_train,
                             y=y_train,
                             cv=kfold,
                             scoring=pre_scorer)
    print("F1-score: %0.2f (+/- %0.2f) [%s]"
          % (scores.mean(), scores.std(), label))

mv.fit(X_train,y_train)
print('Predicted F1-Score: {}'.format(mv.score(X_test,y_test)))

y_pred=mv.predict(X_test)
print('Predicted F1-Score: {}'.format(f1_score(y_true=y_test,y_pred=y_pred, average='micro')))


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
The combined majority vote classifier is constructed out of 3-individual classifiers
from sklearn.ensemble import VotingClassifier
mv = VotingClassifier(estimators=[('forest',forest),('XGBoost',xg)])
mv.fit(X,y)
print ("F1-score: %.4g" % my_scorer(mv,X,y))
Applying the classification model to new data
Now that we have a trained facies classification model we can use it to identify facies in wells that do not have core data.  In this case, we will apply the classifier to two wells, but we could use it on any number of wells for which we have the same set of well logs for input.
This dataset is similar to the training data except it does not have facies labels.  It is loaded into a dataframe called test_data.
Can you write Python code for it?

well_data = pd.read_csv('../validation_data_nofacies.csv')
well_data['Well Name'] = well_data['Well Name'].astype('category')
well_data.columns[4:]
depth = well_data['Depth']
well_data= add_del(well_data,[fn for fn in well_data.columns[3:] if fn != 'NM_M'], depth)
well_data = add_del(well_data,[fn for fn in well_data.columns[3:] if fn != 'NM_M'], depth)
well_data.columns[3:]
feat_labels =well_data.columns[3:]
X_unknown = well_data.drop(['Formation', 'Well Name', 'Depth'], axis=1).values

del_added = well_data.columns[0:]

feat_labels


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
The entries along the diagonal are the facies that have been correctly classified.  Below we define two functions that will give an overall value for how the algorithm is performing.  The accuracy is defined as the number of correct classifications divided by the total number of classifications.
Can you write Python code for it?

def accuracy(conf):
    total_correct = 0.
    nb_classes = conf.shape[0]
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
    acc = total_correct/sum(sum(conf))
    return acc


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
This data is from the Council Grove gas reservoir in Southwest Kansas.  The Panoma Council Grove Field is predominantly a carbonate gas reservoir encompassing 2700 square miles in Southwestern Kansas.  This dataset is from nine wells (with 4149 examples), consisting of a set of seven predictor variables and a rock facies (class) for each example vector and validation (test) data (830 examples from two wells) having the same seven predictor variables in the feature vector.  Facies are based on examination of cores from nine wells taken vertically at half-foot intervals. Predictor variables include five from wireline log measurements and two geologic constraining variables that are derived from geologic knowledge. These are essentially continuous variables sampled at a half-foot sample rate.
The seven predictor variables are:
* Five wire line log curves include gamma ray (GR), resistivity logging (ILD_log10),
photoelectric effect (PE), neutron-density porosity difference and average neutron-density porosity (DeltaPHI and PHIND). Note, some wells do not have PE.
* Two geologic constraining variables: nonmarine-marine indicator (NM_M) and relative position (RELPOS)
The nine discrete facies (classes of rocks) are:
1. Nonmarine sandstone
2. Nonmarine coarse siltstone
3. Nonmarine fine siltstone
4. Marine siltstone and shale
5. Mudstone (limestone)
6. Wackestone (limestone)
7. Dolomite
8. Packstone-grainstone (limestone)
9. Phylloid-algal bafflestone (limestone)
These facies aren't discrete, and gradually blend into one another. Some have neighboring facies that are rather close.  Mislabeling within these neighboring facies can be expected to occur.  The following table lists the facies, their abbreviated labels and their approximate neighbors.
Facies |Label| Adjacent Facies
:---: | :---: |:--:
1 |SS| 2
2 |CSiS| 1,3
3 |FSiS| 2
4 |SiSh| 5
5 |MS| 4,6
6 |WS| 5,7
7 |D| 6,8
8 |PS| 6,7,9
9 |BS| 7,8
Let's clean up this dataset.  The 'Well Name' and 'Formation' columns can be turned into a categorical data type.
Can you write Python code for it?

training_data['Well Name'] = training_data['Well Name'].astype('category')
training_data['Formation'] = training_data['Formation'].astype('category')
training_data['Well Name'].unique()

training_data.columns[4:]

training_data.describe()


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
This is a quick view of the statistical distribution of the input variables.  Looking at the count values, there are 3232 feature vectors in the training set.
Can you write Python code for it?

# 1=sandstone  2=c_siltstone   3=f_siltstone
# 4=marine_silt_shale 5=mudstone 6=wackestone 7=dolomite
# 8=packstone 9=bafflestone
facies_colors = ['#F4D03F', '#F5B041','#DC7633','#6E2C00',
       '#1B4F72','#2E86C1', '#AED6F1', '#A569BD', '#196F3D']

facies_labels = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS',
                 'WS', 'D','PS', 'BS']
#facies_color_map is a dictionary that maps facies labels
#to their respective colors
facies_color_map = {}
for ind, label in enumerate(facies_labels):
    facies_color_map[label] = facies_colors[ind]

def label_facies(row, labels):
    return labels[ row['Facies'] -1]

def make_facies_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=6, figsize=(8, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im=ax[5].imshow(cluster, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[5])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS',
                                'SiSh', ' MS ', ' WS ', ' D  ',
                                ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-1):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
    ax[0].set_xlim(logs.GR.min(),logs.GR.max())
   ...


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
We can use the well log plot to view the classification results along with the well logs.
Can you write Python code for it?

make_facies_log_plot(
    well_data[well_data['Well Name'] == 'STUART'],
    facies_colors=facies_colors)

make_facies_log_plot(
    well_data[well_data['Well Name'] == 'CRAWFORD'],
    facies_colors=facies_colors)


I am working on the file "EvgenyS/Facies_classification_ES.ipynb".
The first task is:
import and initilize a few classifires that we play with
Can you write Python code for it?

y = training_data['Facies'].values -1
X = training_data.drop(['Formation', 'Well Name', 'Depth','Facies'], axis=1).values
feat_labels =training_data.columns[4:]
label_encoded_y  = np.unique(y)


I am working on the file "EvoML - Example Usage.ipynb".
The first task is:
2. Subspacing - sampling in the domain of features - evolving and mutating columns
Can you write Python code for it?

from evoml.subspacing import FeatureStackerFEGT, FeatureStackerFEMPO

print(FeatureStackerFEGT.__doc__)

clf = FeatureStackerFEGT(ngen=30)

clf.fit(X, y)

clf.score(X, y)

## Get the Hall of Fame individual
hof = clf.segment[0]

sampled_datasets = [eg.get_data() for eg in hof]

[data.columns.tolist() for data in sampled_datasets]

## Original X columns
X.columns


from evoml.subspacing import FeatureStackerFEGT, FeatureStackerFEMPO

print(FeatureStackerFEGT.__doc__)

clf = FeatureStackerFEGT(ngen=30)

clf.fit(X, y)

clf.score(X, y)

## Get the Hall of Fame individual
hof = clf.segment[0]

sampled_datasets = [eg.get_data() for eg in hof]

[data.columns.tolist() for data in sampled_datasets]

## Original X columns
X.columns

# The exploration of the dataset by benchmark algorithms
clf = DecisionTreeClassifier(random_state=34092)
clf.fit(X_train, y_train)
pred_DTC = clf.predict(X_test)
print('Base DecisionTreeClassifier accuracy: {}'.format(clf.score(X_test, y_test)))

clf = RandomForestClassifier(random_state=34092)
clf.fit(X_train_tot, y_train)
pred_RFC = clf.predict(X_test)
print('Base RandomForestClassifier accuracy: {}'.format(clf.score(X_test, y_test)))

clf = GradientBoostingClassifier(random_state=34092)
clf.fit(X_train, y_train)
pred_GBC = clf.predict(X_test)
print('Base GradientBoostingClassifier accuracy: {}'.format(clf.score(X_test, y_test)))

print('')


I am working on the file "EvoML - Example Usage.ipynb".
The first task is:
Aim
Motive of the notebook is to give a brief overview as to how to use the evolutionary sampling powered ensemble models as part of the EvoML research project.
Will make the notebook more verbose if time permits. Priority will be to showcase the flexible API of the new estimators which encourage research and tinkering.
Contents

Subsampling
Subspacing

1. Subsampling - Sampling in the example space - rows will be mutated and evolved.
Can you write Python code for it?

from evoml.subsampling import BasicSegmenter_FEMPO, BasicSegmenter_FEGT, BasicSegmenter_FEMPT

df = pd.read_csv('datasets/ozone.csv')

df.head(2)

X, y = df.iloc[:,:-1], df['output']

print(BasicSegmenter_FEGT.__doc__)

from sklearn.tree import DecisionTreeRegressor
clf_dt = DecisionTreeRegressor(max_depth=3)
clf = BasicSegmenter_FEGT(base_estimator=clf_dt, statistics=True)

clf.fit(X, y)

clf.score(X, y)

EGs = clf.segments_

len(EGs)

sampled_datasets = [eg.get_data() for eg in EGs]

[sd.shape for sd in sampled_datasets]


I am working on the file "EvolAlgoCalibration.ipynb".
The first task is:
1 Selection:
A Select fittest members of the population
For that we already sorted the list of individuals. So it is easy to select the fittest individuals
Can you write Python code for it?

percentage_parameters_to_mutate = 0.1
percentage_individuals_to_mutate = 0.5
retain=0.3
random_select=0.1

retain_lenght = int(len(next_population) * retain)
parents = next_population[:retain_lenght]


I am working on the file "EvolAlgoCalibration.ipynb".
The first task is:
2 Mutation: vary random parameters of random individuals
Can you write Python code for it?

amount_of_individuals_to_mutate = int(percentage_individuals_to_mutate * len(parents))
# determine if a mutation will take place in this parent
for parent in random.sample(parents, amount_of_params_to_mutate):
    # determine how many parameters should be mutated
    amount_of_params_to_mutate = int(percentage_parameter_to_mutate * len(parents[0].parameters))
    # sample the indexes of the parameters to mutate
    print('I will mutate ', amount_of_params_to_mutate, ' parameters from', parent)
    indexes_of_mutable_params = random.sample(range(len(parents[0].parameters)), amount_of_params_to_mutate)
    for idx in indexes_of_mutable_params:
        # identify the range for this parameter to mutate
        min_value, max_value = problem['bounds'][idx][0], problem['bounds'][idx][1]
        print('I mutate ', problem['names'][idx], ' which has min, max ', min_value, max_value, 'and current val= ', parent.parameters[idx])
        if type(min_value) == float:
            parent.parameters[idx] = random.uniform(min_value, max_value)
        else:
            parent.parameters[idx] = random.randint(min_value, max_value)
        print('new variable value is ', parent.parameters[idx])


I am working on the file "EvolAlgoCalibration.ipynb".
The first task is:
3 Breeding: fill up the rest of the population with combinations of the most fittest individuals
Can you write Python code for it?

# keep in mind if it is a float or an integer
# let parents breed to create children
parents_lenght = len(parents)
desired_lenght = len(next_population) - parents_lenght
children = []
while len(children) < desired_lenght:
    male = random.randint(0, parents_lenght - 1)
    female = random.randint(0, parents_lenght - 1)
    print('parents are ', male, female)
    if male != female:
        male = parents[male]
        female = parents[female]
        half = int(len(male.parameters) / 2)
        # here I should create a new child
        child_parameters = male.parameters[:half] + female.parameters[half:]
        print('male params are ', male.parameters)
        print('female params are ', female.parameters)
        print('child params are', child_parameters)
        child = Individual(child_parameters, [], np.inf)
        children.append(child)
parents.extend(children)


I am working on the file "EvolAlgoCalibration.ipynb".
The first task is:
Calibrate the zero-intelligence model using an evolutionary algorithm
Can you write Python code for it?

SIMTIME = 200
NRUNS = 5
backward_simulated_time = 400
initial_total_money = 26000
init_profit = 1000
init_discount_rate = 0.17


I am working on the file "EvolAlgoCalibration.ipynb".
The first task is:
Data used for calibration
Can you write Python code for it?

start_date = '2010-01-01'
end_date = '2016-12-31'

spy = data.DataReader("SPY",
                       start=start_date,
                       end=end_date,
                       data_source='google')['Close']
spy_returns = spy.pct_change()[1:]

spy_volume = data.DataReader("SPY",
                       start=start_date,
                       end=end_date,
                       data_source='google')['Volume']

# calculate stylized facts for SP500
spy_autocorrelation = autocorrelation_returns(spy_returns, 25)
spy_kurtosis = kurtosis(spy_returns)
spy_autocorrelation_abs = autocorrelation_abs_returns(spy_returns, 25)
spy_hurst = hurst(spy, lag1=2 , lag2=20)
spy_cor_volu_vola = correlation_volume_volatility(spy_volume, spy_returns, window=10)
stylized_facts_spy = [spy_autocorrelation, spy_kurtosis, spy_autocorrelation_abs, spy_hurst, spy_cor_volu_vola]
print('spy_autocorrelation ',spy_autocorrelation)
print('spy_kurtosis ', spy_kurtosis)
print('spy_autocorrelation_abs ', spy_autocorrelation_abs)
print('spy_hurst ', spy_hurst)
print('spy_cor_volu_vola ',spy_cor_volu_vola)


I am working on the file "EvolAlgoCalibration.ipynb".
The first task is:
Function to evolve the population
Can you write Python code for it?

def evolve_population(population, fittest_to_retain, random_to_retain, parents_to_mutate, parameters_to_mutate):
    """
    Evolves a population. First, the fittest members of the population plus some random individuals become parents.
    Then, some random mutations take place in the parents. Finally, the parents breed to create children.
    :param population: population individuals sorted by cost (cheapest left) which contain parameter values
    :param fittest_to_retain: percentage of fittest individuals which should be maintained as parents
    :param random_to_retain: percentage of other random individuals which should be maintained as parents
    :param individuals_to_mutate: percentage of parents in which mutations will take place
    :param parameters_to_mutate: percentage of parameters in chosen individuals which will mutate
    :return:
    """
    # 1 retain parents
    retain_lenght = int(len(population) * fittest_to_retain)
    parents = population[:retain_lenght]

    # 2 retain random individuals
    amount_random_indiv = int(len(population) * random_to_retain)
    parents.extend(random.sample(population[retain_lenght:], amount_random_indiv))

    # 3 mutate random parameters of random individuals
    amount_of_individuals_to_mutate = int(parents_to_mutate * len(parents))
    amount_of_params_to_mutate = int(parameters_to_mutate * len(parents[0].parameters))
    for parent in random.sample(parents, amount_of_params_to_mutate):
        indexes_of_mutable_params = random.sample(range(len(parent.parameters)), amount_of_params_to_mutate)
        for idx in indexes_of_mutable_params:
            min_value, max_value = problem['bounds'][idx][0], problem['bounds'][idx][1]
            if type(min_value) == float:
                parent.parameters[idx] = random.uniform(min_value, max_value)
            else:
                parent.parameters[idx] = random.randint(min_value, max_value)

    # 4 parents breed to create a new population
    parents_lenght = len(parents)
   ...


I am working on the file "EvolAlgoCalibration.ipynb".
The first task is:
Parameter space
We define the parameter bounds as follows.
| Parameter | Values (start, stop, step) |
| -------------| ------------|
| share_chartists       | 0 - 1, 0.1      |
| share_mean_reversion       | 0 - 1, 0.1   |
| order_expiration_time       | 1000 - 10000, 1000      |
| agent_order_price_variability       | 1 - 10, 1      |
| agent_order_variability       | 0.1 - 5       |
| agent_ma_short       | 5 - 100, 5      |
| agent_ma_long       | 50 - 400, 50      |
| agents_hold_thresholds       | 0.0005        |
| Agent_volume_risk_aversion       | 0.1 - 1, 0.1      |
| Agent_propensity_to_switch       | 0.1 - 2.2, 0.1      |
| profit_announcement_working_days       | 5 - 50, 5       |
| price_to_earnings_spread       | 5 - 50, 5       |
| price_to_earnings_heterogeneity       | 5 - 50, 5       |
Can you write Python code for it?

parameter_space = {'share_chartists':[0.0, 1.0], 'share_mean_reversion':[0.0, 1.0], 'order_expiration_time':[1000, 10000],
                   'agent_order_price_variability':[1, 10], 'agent_order_variability':[0.1, 5.0],
                   'agent_ma_short':[5, 100], 'agent_ma_long':[50, 400], 'agents_hold_thresholds':[0.00005,0.01],
                   'agent_volume_risk_aversion':[0.1, 1.0], 'agent_propensity_to_switch':[0.1, 2.2],
                   'profit_announcement_working_days':[5, 50], 'price_to_earnings_base':[10,20],
                   'price_to_earnings_heterogeneity':[1.1,2.5], 'price_to_earnings_gap':[4,20],
                   'longMA_heterogeneity':[1.1,1.8], 'shortMA_heterogeneity':[1.1,1.8], 'shortMA_memory_divider':[1, 10]}


I am working on the file "EvolAlgoCalibration.ipynb".
The first task is:
Problem
We try to match average simulation stylized facts as closely as possible to observed stylized facts.
For that, we use an evolutionary algorithm to minimize a cost function.
Create population of individuals
In our algorithm, an individual is a set a of parameters and its average associated values for stylized facts over several simulation runs.
Can you write Python code for it?

class Individual:
    """The order class can represent both bid or ask type orders"""
    def __init__(self, parameters, stylized_facts, cost):
        self.parameters = parameters
        self.stylized_facts = stylized_facts
        self.cost = cost

    def __lt__(self, other):
        """Allows comparison to other individuals based on its cost (negative fitness)"""
        return self.cost < other.cost

# create initial population
population = []
for parameters in latin_hyper_cube:
    # add an individual to the population
    population.append(Individual(parameters, [], np.inf))
# create populations_over_time
populations_over_time = [population]


I am working on the file "EvolAlgoCalibration.ipynb".
The first task is:
Simulate evolution
Can you write Python code for it?


I am working on the file "EvolAlgoCalibration.ipynb".
The first task is:
Simulate the evolutionary model

simulated_population = simulate_population(population, kwargs)
evolved_population, generation_fitness = evolve_population(population, kwargs)
fitness.append(generation_fitness)

for i in range(iterations):
    pop = evolve(pop, target)
    fitness_history.append(average_fitness(pop, target))
Can you write Python code for it?

iterations = 2
av_pop_fitness = []
all_populations = [population]

for i in range(iterations):
    simulated_population, fitness = simulate_population(all_populations[i], number_of_runs=3, simulation_time=10, number_of_agents=200)
    av_pop_fitness.append(fitness)
    all_populations.append(evolve_population(simulated_population, fittest_to_retain=0.2, random_to_retain=0.1,
                                             parents_to_mutate=0.5, parameters_to_mutate=0.1))


I am working on the file "EvolAlgoCalibration.ipynb".
The first task is:
Then, we determine the amount of starting points we want for the genetic algorithm and sample the parameter space using a Latin hypercube sample.
Can you write Python code for it?

population_size = 8

problem = {
  'num_vars': 17,
  'names': ['share_chartists', 'share_mean_reversion', 'order_expiration_time', 'agent_order_price_variability',
            'agent_order_variability', 'agent_ma_short', 'agent_ma_long', 'agents_hold_thresholds',
           'agent_volume_risk_aversion', 'agent_propensity_to_switch', 'profit_announcement_working_days',
           'price_to_earnings_base', 'price_to_earnings_heterogeneity', 'price_to_earnings_gap',
           'longMA_heterogeneity', 'shortMA_heterogeneity', 'shortMA_memory_divider'],
  'bounds': [[0.0, 1.0], [0.0, 1.0], [1000, 10000], [1, 10],
             [0.1, 5.0], [5, 100], [50, 400], [0.00005,0.01],
             [0.1, 1], [0.1, 2.2], [5, 50],
             [10,20], [1.1,2.5], [4,20],
             [1.1,1.8], [1.1,1.8], [1, 10]]
}

latin_hyper_cube = latin.sample(problem=problem, N=population_size)
latin_hyper_cube = latin_hyper_cube.tolist()

# transform some of the parameters to integer
for idx, parameters in enumerate(latin_hyper_cube):
    latin_hyper_cube[idx][2] = int(latin_hyper_cube[idx][2])
    latin_hyper_cube[idx][3] = int(latin_hyper_cube[idx][3])
    latin_hyper_cube[idx][4] = int(latin_hyper_cube[idx][4])
    latin_hyper_cube[idx][5] = int(latin_hyper_cube[idx][5])
    latin_hyper_cube[idx][6] = int(latin_hyper_cube[idx][6])
    latin_hyper_cube[idx][10] = int(latin_hyper_cube[idx][10])
    latin_hyper_cube[idx][11] = int(latin_hyper_cube[idx][11])
    latin_hyper_cube[idx][13] = int(latin_hyper_cube[idx][13])
    latin_hyper_cube[idx][16] = int(latin_hyper_cube[idx][16])


I am working on the file "Evolution+of+GDP+and+Household+Electrcity+Consumption.ipynb".
The first task is:
A closer look at each of the variables confirms what was anticipated by the Skewness and Kurtosis coefficients. When each of the variables is compared against a normal distribution, none of them follow this kind of distribution.
Can you write Python code for it?

# Making two variables.
norm = np.random.normal(0, 1, 2404)
norm.sort()
a = result.sort_values('Household Consumption (GWh)',ascending=True)
b = result.sort_values('GDP Current Prices (Million USD)',ascending=True)
c = result.sort_values('Population (Thousands)',ascending=True)
d = result.sort_values('Standard GDP Current Prices (USD)',ascending=True)
e = result.sort_values('Standard Household Consumption (kWh)',ascending=True)

#Checking that variables are not normally distributed
plt.figure(figsize=(20, 5))

plt.subplot(1, 5, 1)
plt.plot(norm,a['Household Consumption (GWh)'], "o")
plt.title('Household Consumption (GWh')

plt.subplot(1, 5, 2)
plt.plot(norm,b['GDP Current Prices (Million USD)'], "o")
plt.title('GDP Current Prices (Million USD)')

plt.subplot(1, 5, 3)
plt.plot(norm,c['Population (Thousands)'], "o")
plt.title('Population (Thousands)')

plt.subplot(1, 5, 4)
plt.plot(norm,d['Standard GDP Current Prices (USD)'], "o")
plt.title('Standard GDP Current Prices (USD p. Capita)')

plt.subplot(1, 5, 5)
plt.plot(norm,e['Standard Household Consumption (kWh)'], "o")
plt.title('Household Consumption (kWh) p. Capita')

plt.tight_layout()
plt.show()


I am working on the file "Evolution+of+GDP+and+Household+Electrcity+Consumption.ipynb".
The first task is:
After checking that all countries considered by the UN are captured in the eternal list of countries used to group them by continent the final data set is created. The following table shows the first five rows of the final data set that will be used for the purpose of this report:
Can you write Python code for it?


I am working on the file "Evolution+of+GDP+and+Household+Electrcity+Consumption.ipynb".
The first task is:
As it can be seen from the graphs above, each continent follows a different pattern in terms of standard electricity consumption and standard GDP against population. Hence detailed analysis show how each continent behaves in a different way although initially, when considering global data, the relationship between both variables was high and positive.
Question 2: How has the economic welfare of countries evolved from the year 2000?
From the graphs below it can be seen that standard electricity consumption and standard GDP is higher in Europe than in the rest of the continents. In all cases both variables experience an increase during the period and except for Europe the increase seems to be smoother in the rest of the continents.
From an electricity consumption perspective, the graph depicts how the economic downturns of 2008 affected the welfare of the population. It also shows how up until that year, when natural gas was considered an alternative cheaper solution for household heating systems, the population was relying on its use. The rest of the continents were more conservative in the use of electricity for household needs. It can be seen how there is a strong decrease in electricity consumption in 2002 and 2004 both in Asia and North America. The first change in slope can be seen in Asia in 2002 (due to the economic downturn that the country experienced that year). This decrease impacts North America in 2004 as part of its knock on effect. On the flipside, Europe slightly decreases its rate of increase but still increases the electricity consumption up until 2008. The economic Africa´s growth, in the lower-end is mainly due to the penetration of new electricity infrastructure during this period.
From a GDP perspective, the graph shows the deep impact that the 2008 economic downturn had in Europe. Although hiking up until then, the loss by 2014 was considerable in relative terms. For the rest of the continents, the 2002 economic downturn can be observed. It is worth mentioning the economic acceleration...
Can you write Python code for it?

#Plot time series showing the evolution of the standardised variables

C = result1.index.levels[0]
colors = itertools.cycle(["r", "b", "g", 'purple','yellow','brown'])

plt.figure(figsize=(20, 5))

plt.subplot(1, 2, 1)
for cont in C :
    graph = result1.loc[cont]
    plt.plot(graph.index, graph['Standard Household Consumption (kWh)'], color = next(colors), linestyle = "-", label = cont)
plt.ylabel('kWh')
plt.xlabel('Year')
plt.legend(bbox_to_anchor=(0., 1.02, 1., .102), loc=3, ncol=3, mode="expand", borderaxespad=0.)

plt.subplot(1, 2, 2)
for cont in C :
    graph = result1.loc[cont]
    plt.plot(graph.index, graph['Standard GDP Current Prices (USD)'], color = next(colors), linestyle = "-", label = cont)
plt.ylabel('USD')
plt.xlabel('Year')
plt.legend(bbox_to_anchor=(0., 1.02, 1., .102), loc=3, ncol=3, mode="expand", borderaxespad=0.)

plt.tight_layout()
plt.show()


I am working on the file "Evolution+of+GDP+and+Household+Electrcity+Consumption.ipynb".
The first task is:
Capston Project: Analytic Report & Research Proposal
Evolution of GPD and Household Electricy Consumption 2000-2014
Can you write Python code for it?

import numpy as np
import pandas as pd
import scipy
import matplotlib.pyplot as plt
from scipy.stats import ttest_ind
from scipy import stats
import itertools
%matplotlib inline

w = pd.read_csv('https://raw.githubusercontent.com/borja876/Thinkful-DataScience-Borja/master/Electricity%20Consumption.csv')
x = pd.read_csv('https://raw.githubusercontent.com/borja876/Thinkful-DataScience-Borja/master/GDP%20current%20prices.csv')
y = pd.read_csv('https://raw.githubusercontent.com/borja876/Thinkful-DataScience-Borja/master/Population.csv')

df = pd.DataFrame(w)
df1 = pd.DataFrame(x)
df2 = pd.DataFrame(y)


I am working on the file "Evolution+of+GDP+and+Household+Electrcity+Consumption.ipynb".
The first task is:
Data sets description
The purpose of this report is to analyze the evolution of the household electricity market and GDP per continent in the period 2000-2014. This will show how different economic cycles have affected both variables and the potential correlation between them. Furthermore, in a per continent basis, the both variables will be scrutinized to depict the similarities and differences between continents that will show the relationship between them. Three data sets have been imported from United Nations (from now onwards UN) database. The data sets contain global information for the time span 2000-2014 regarding:

Electricity consumption for the household market (GhW). In this dataset, the UN has estimated values for some countries (ex-colonies of the UK) based on their April-March consumption. This estimation has been done up until 2004 yearly electricity consumption was standardized following the natural year and not the fiscal year. Electricity consumption in Iraq in 2014 has a null value, as it was not reported due to the war that started in 2015.

GDP per country in current USD. From all the data sets available, measuring GDP the one measuring it in current USD has been chosen to avoid the impact of the different base years across countries when calculating the base of the deflation year and to avoid the use of different exchange rates across countries during the time span under analysis.

Population. In this case, the population has been converted into (Thousands) as a unit to make the standardized GDP and electricity consumption significant. This variable shows the net population at the end of the year considering births, deaths and declared migration.

The three of them are significant because, although different data sets containing this information exist, only this institution (UN) gathers them. They are consistent in terms of the methods used to gather this information and credible due to the institution that is providing them. The three variables can be compared against each other...
Can you write Python code for it?

#Cleanse Data Frame Electricity Consumption:
#Delete rows without countries, rename columns and take out those that are not used
#Zeros are identified to see their potential impact on the data set and further results
dfa= df[:3206]
dfb= dfa.rename(columns={'Country or Area': 'Country', 'Quantity': 'Household Consumption (GWh)'})
dfc = dfb.drop(["Quantity Footnotes", "Unit", "Commodity - Transaction"], axis=1)
dfc['Year'] = dfc['Year'].astype(int)
#dfc.loc[dfc['Household Consumption (GWh)'] == 0]

#Cleanse Data Frame GDP Current Prices (USD):
#Cleanse Data Frame Electricity Consumption:
#Delete rows without countries, rename columns and take out those that are not used
#Zeros are identified to see their potential impact on the data set and further results
dfd= df1[:3322]
dfe= dfd.rename(columns={'Country or Area': 'Country', 'Value': 'GDP Current Prices (USD)'})
dfg = dfe.drop('Value Footnotes', axis=1)
dfg['Year'] = dfg['Year'].astype(int)
#dfg.loc[dfg['GDP Current Prices (USD)'] == 0]

#Cleanse Data Frame Population:
#Take out rows without countries
#Rename columns
#Clean columns taking out those that are not used
dfh= df2[:3522]
dfi= dfh.rename(columns={'Country or Area': 'Country', 'Value': 'Population'})
dfj = dfi.drop('Value Footnotes', axis=1)
dfj['Year'] = dfj['Year'].astype(int)
#dfj.loc[dfj['Population'] == 0]

#Merge data into a single dataset
#Cleanse new dataset
result = dfc.merge(dfg, left_on=["Country","Year"], right_on=["Country","Year"], how='outer')
result = result.merge(dfj, left_on=["Country","Year"], right_on=["Country","Year"], how='outer')
result = result.dropna()

#Rescale & rename variables in the new data set to GDP Current Prices (Million USD) & Population (Thousands)
result['GDP Current Prices (Million USD)']=result['GDP Current Prices (USD)']/1000000
result['Population (Thousands)']=result['Population']/1000

#Drop redundant columns after merging the data sets
result = result.drop('GDP Current Prices (USD)',1)
result = result.drop('Population',1)
result= result.rename(columns={'GDP...


I am working on the file "Evolution+of+GDP+and+Household+Electrcity+Consumption.ipynb".
The first task is:
Electricity consumption per household decreases in all continents between 2002 ad 2004. This is due to two main reasons. The increase of electricity prices correlated to and oil prices basket at that time and the economic slowdown that took place in that period. Although it shows an increase in all six continents, the behavior during the period under analysis is significantly different:
In Africa, electricity consumption decreases in 2004 and steadily increases during the period. After 2004 every 2 years Africa experiences a contraction in consumption followed by an increase that compensates and surpasses in absolute terms the contraction. Compared to the rest of the continents it seems too have the steepest slope only comparable to South America. This can be mainly because of the penetration of electricity infrastructure in the continent and its low starting point in terms of standard electricity consumption.
In Australia there is a decrease in consumption in 2002 and after a continuous increase up unitl 2008, it stabilizes around the value achieve in 2014. This is mainly due to the environmental and demand management regulation introduced in the country in 2006 added to the economic downturn that affected Australia in 2008.
In South America, there is a steady increase in electricity consumption due to the emigration after the recession at the beginning of the century.
In North America and Europe, electricity consumption growth is similar to Europe. In both there is a decrease in 2002-2004 due to the economic downturn in the case of Europe and to the introduction of natural gas and regulation giving positive incentives to the use of it in heating systems. Additionally, in North America the deregulation of household production makes the impact in this country higher than in Europe. In both, the main responsible for the 2008 decrease in electricity consumption is the real estate crisis that affected Europe more than America but it had its consequences in both.
In Asia, electricity consumption has followed an...
Can you write Python code for it?

#Plot six graphs showing the evolution of standard electricity consumption per continent
C = result1.index.levels[0]
colors = itertools.cycle(["r", "b", "g", 'purple','yellow','brown'])

plt.figure(figsize=(26, 10))

plt.subplot(2, 6, 1)

graph = result1.loc['Africa']
plt.plot(graph.index, graph['Standard Household Consumption (kWh)'], color = next(colors), linestyle = "-")
plt.ylabel('kWh')
plt.xlabel('Year')
plt.title('Africa: Household Consumption')
plt.xlim([2000, 2014])

plt.subplot(2, 6, 2)

graph = result1.loc['Oceania']
plt.plot(graph.index, graph['Standard Household Consumption (kWh)'], color = next(colors), linestyle = "-")
plt.ylabel('kWh')
plt.xlabel('Year')
plt.title('Oceania: Household Consumption')
plt.xlim([2000, 2014])
plt.ylim([180, 240])

plt.subplot(2, 6, 3)

graph = result1.loc['South America']
plt.plot(graph.index, graph['Standard Household Consumption (kWh)'], color = next(colors), linestyle = "-")
plt.ylabel('x10 kWh')
plt.xlabel('Year')
plt.title('South America: Household Consumption')
plt.xlim([2000, 2014])

plt.subplot(2, 6, 4)

graph = result1.loc['Europe']
plt.plot(graph.index, graph['Standard Household Consumption (kWh)'], color = next(colors), linestyle = "-")
plt.ylabel('kWh')
plt.xlabel('Year')
plt.title('Europe: Household Consumption')
plt.xlim([2000, 2014])

plt.subplot(2, 6, 5)
graph = result1.loc['Asia']
plt.plot(graph.index, graph['Standard Household Consumption (kWh)'], color = next(colors), linestyle = "-")
plt.ylabel('kWh')
plt.xlabel('Year')
plt.title('Asia: Household Consumption')
plt.xlim([2000, 2014])

plt.subplot(2, 6, 6)
graph = result1.loc['North America']
plt.plot(graph.index, graph['Standard Household Consumption (kWh)'], color = next(colors), linestyle = "-")
plt.ylabel('kWh')
plt.xlabel('Year')
plt.title('North America: Household Consumption')
plt.xlim([2000, 2014])

plt.tight_layout()
plt.show()


I am working on the file "Evolution+of+GDP+and+Household+Electrcity+Consumption.ipynb".
The first task is:
Question 1: Is there a relationship between GDP and Household
consumption? Is it the same across continents?
When comparing the data sets without grouping the data into continents, the correlation matrix below shows that there is no correlation between years GDP and Electricity consumption, hence there is no inflation effect on the variables. There is a strnger correlation between the standard GDP and the years due to the impact of population.
The table below highlights the positive and strong relationship between electricity consumption and GDP. Furthermore, it evidences the relationship between our standardized variables: both vary in the same direction but at a lower rate due to the impact in the variable “Population” used to standardize them. It also evidences the positive relationship between population and electricity consumption. As expected, it shows that only half of the increase is due to an increase in population leaving the other half unexplained.
Can you write Python code for it?

#Calculate the correlation matrix
result.corr()


I am working on the file "Evolution+of+GDP+and+Household+Electrcity+Consumption.ipynb".
The first task is:
Question 3: What was the behavior of each continent during this period?
All six continents have increased their wealth during the period 2000-2014. Europe and America have more than doubled their GDP per capita while the rest of the continents have multiplied by 3 except for Asia that has multiplied its GDP by 7.  All continents show a flat or declining GDP during the period 2002-2004, being the case of Asia the most significant.
All except for Europe and South America have experienced a slight decrease in its GDP due to the economic crisis if compared to Europe. It can be seen that Europe was one of the continents that was experiencing higher growth but also higher decline due to the crisis. In the case of America, periods of growth and slowdown have been alternating every two years.
Europe has been the continent that has experienced the highest economic downturn during the period. Although it was one of the highest growing ones up until 2008, the economic downturn has significantly reduced the GDP per capita up until more than USD10,000.
Can you write Python code for it?

#Plot 6 graphs showing the evolution of standard GDP per continent
C = result1.index.levels[0]
colors = itertools.cycle(["r", "b", "g", 'purple','yellow','brown'])

plt.figure(figsize=(26, 10))

plt.subplot(2, 6, 1)

graph = result1.loc['Africa']
plt.plot(graph.index, graph['Standard GDP Current Prices (USD)'], color = next(colors), linestyle = "-")
plt.ylabel('USD')
plt.xlabel('Year')
plt.title('Africa: Standard GDP Current Prices')
plt.xlim([2000, 2014])

plt.subplot(2, 6, 2)

graph = result1.loc['Oceania']
plt.plot(graph.index, graph['Standard GDP Current Prices (USD)'], color = next(colors), linestyle = "-")
plt.ylabel('USD')
plt.xlabel('Year')
plt.title('Oceania: Standard GDP Current Prices')
plt.xlim([2000, 2014])

plt.subplot(2, 6, 3)

graph = result1.loc['South America']
plt.plot(graph.index, graph['Standard GDP Current Prices (USD)'], color = next(colors), linestyle = "-")
plt.ylabel('USD')
plt.xlabel('Year')
plt.title('South America: Standard GDP Current Prices')
plt.xlim([2000, 2014])

plt.subplot(2, 6, 4)

graph = result1.loc['Europe']
plt.plot(graph.index, graph['Standard GDP Current Prices (USD)'], color = next(colors), linestyle = "-")
plt.ylabel('USD')
plt.xlabel('Year')
plt.title('Europe: Standard GDP Current Prices')
plt.xlim([2000, 2014])

plt.subplot(2, 6, 5)
graph = result1.loc['Asia']
plt.plot(graph.index, graph['Standard GDP Current Prices (USD)'], color = next(colors), linestyle = "-")
plt.ylabel('USD')
plt.xlabel('Year')
plt.title('Asia: Standard GDP Current Prices')
plt.xlim([2000, 2014])

plt.subplot(2, 6, 6)
graph = result1.loc['North America']
plt.plot(graph.index, graph['Standard GDP Current Prices (USD)'], color = next(colors), linestyle = "-")
plt.ylabel('USD')
plt.xlabel('Year')
plt.title('North America: Standard GDP Current Prices')
plt.xlim([2000, 2014])
plt.tight_layout()
plt.show()


I am working on the file "Evolution+of+GDP+and+Household+Electrcity+Consumption.ipynb".
The first task is:
The following box plot shows the summary statistics describing the values per continent of each of the variables. It can be appreciated that variance is lower in the case of the standard household consumption compared to the standard GDP. The min value in terms of standard GDP is much lower thant the percentil 25 while in the case of the electricity consumption values are more compact.
Can you write Python code for it?

#Box-plot
result1=result.groupby(['Continent','Year']).median()
result1[['Standard Household Consumption (kWh)','Standard GDP Current Prices (USD)']].boxplot(figsize=(26, 10))


I am working on the file "Evolutionary Multi-Objective Optimization.ipynb".
The first task is:
A compact NSGA-II implementation
Storing all the required information in the toolbox and using DEAP's algorithms.eaMuPlusLambda function allows us to create a very compact -albeit not a 100% exact copy of the original- implementation of NSGA-II.
Can you write Python code for it?

def nsga_ii(toolbox, stats=None, verbose=False):
    pop = toolbox.population(n=toolbox.pop_size)
    pop = toolbox.select(pop, len(pop))
    return algorithms.eaMuPlusLambda(pop, toolbox, mu=toolbox.pop_size,
                                     lambda_=toolbox.pop_size,
                                     cxpb=1-toolbox.mut_prob,
                                     mutpb=toolbox.mut_prob,
                                     stats=stats,
                                     ngen=toolbox.max_gen,
                                     verbose=verbose)


I am working on the file "Evolutionary Multi-Objective Optimization.ipynb".
The first task is:
A side-effect of using process-based parallelization
Process-based parallelization based on multiprocessing requires that the parameters passed to map() be pickleable.

The direct consequence is that lambda functions can not be directly used.
This is will certainly ruin the party to all lambda fans out there! -me included.
Hence we need to write some wrapper functions instead.
But, that wrapper function can take care of filtering out dominated individuals in the results.
Can you write Python code for it?

def run_algo_wrapper(toolboox):
    result,a = nsga_ii(toolbox)
    pareto_sets = tools.emo.sortLogNondominated(result, len(result))
    return pareto_sets[0]


I am working on the file "Evolutionary Multi-Objective Optimization.ipynb".
The first task is:
All set! Run the experiments...
Can you write Python code for it?

%%time
from multiprocessing import Pool
display(progress_bar)

results = {}
pool = Pool()
for toolbox in toolboxes:
    results[toolbox.experiment_name] = pool.map(run_algo_wrapper, [toolbox] * number_of_runs)
    progress_bar.value +=number_of_runs
    progress_bar.description = "Finished %03d of %03d:" % (progress_bar.value, progress_bar.max)