I am working on the file "Analysis_Variance.ipynb".
The first task is:
The p-value is much smaller than 5%. Therefore the null hypothesis is rejected. We can now look at which group are really different from the other.
Can you write Python code for it?

import statsmodels.stats.multicomp as multi

multi1 = multi.MultiComparison(subnesarc['nb_joints_day'], subnesarc['ethnicity'])
result1 = multi1.tukeyhsd()
result1.summary()


I am working on the file "Analysis_overview.ipynb".
The first task is:
Calculate ISPC over time
Can you write Python code for it?

ispc_vol = np.empty(tfr_vol.shape[-1])
ispc_invol = np.empty(tfr_invol.shape[-1])

chan_A, chan_B = 37, 47  # change for different channels
freq_idx = 14  # change for another frequciency

for i in range(len(ispc_vol)):
    ispc_vol[i] = np.abs(np.mean(np.exp(
        1j*(np.angle(tfr_vol[:, chan_A, freq_idx, i]) -
            np.angle(tfr_vol[:, chan_B, freq_idx, i])))))

for i in range(len(ispc_invol)):
    ispc_invol[i] = np.abs(np.mean(np.exp(
        1j*(np.angle(tfr_invol[:, chan_A, freq_idx, i]) -
            np.angle(tfr_invol[:, chan_B, freq_idx, i])))))

plt.figure()
plt.plot(times, ispc_vol, 'b', label="Voluntary")
plt.plot(times, ispc_invol, 'r', label="Involuntary")
plt.legend()
plt.title("ISPC over time between %s and %s for freq: %s:%s" % (epochs.ch_names[chan_A],
                                                  epochs.ch_names[chan_B],
                                                  frequencies[2], frequencies[7]))
plt.show()


I am working on the file "Analysis_overview.ipynb".
The first task is:
Calculate spectral and temporal resolution of the Morlet wavelets
From http://www.fieldtriptoolbox.org/tutorial/timefrequencyanalysis (Time-frequency analysis IV -- Morlet Wavelets). We get the spectral bandwidth is:
$$\frac{\text{Frequency}}{\text{number of cycles}}\times2 = \text{Spectral resolution Hz}$$
Can you write Python code for it?

frequencies = np.arange(6., 31., 1.)

spectral_resolution_4 = calc_spatial_resolution(frequencies, n_cycles=4)
spectral_resolution_5 = calc_spatial_resolution(frequencies, n_cycles=5)

plt.figure()
plt.plot(frequencies, spectral_resolution_4, label="4 cycles")
plt.plot(frequencies, spectral_resolution_5, label="5 cycles")
plt.ylabel("Spectral bandwidth (Hz)")
plt.xlabel("Frequencies")
plt.legend()
plt.show()

plt.figure()
plt.plot(frequencies, spectral_resolution_4 - spectral_resolution_5)
plt.ylabel("Spectral bandwidth Hz ")
plt.xlabel("Frequencies")
plt.title("Difference: 4 cycles - 5 cycles")
plt.show()


I am working on the file "Analysis_overview.ipynb".
The first task is:
Connectivity: ISPC
Calculate ISPC over trials
Can you write Python code for it?

chan_A, chan_B = 37, 47  # change for different channels
freq_idx = 14  # change for another frequciency

ispc_vol = ISPC_over_trials(tfr_vol, freqs=[2, 7], channels=[chan_A, chan_B], faverage=True)
ispc_vol = ISPC_over_trials(tfr_invol, freqs=[2, 7], channels=[chan_A, chan_B], faverage=True)

plt.figure()
plt.plot(times, ispc_vol, 'b', label="Voluntary")
plt.plot(times, ispc_vol, 'r', label="Involuntary")
plt.legend()
plt.title("ISPC over time between %s and %s for freq: %s:%s" % (epochs.ch_names[chan_A],
                                                  epochs.ch_names[chan_B],
                                                  frequencies[2], frequencies[7]))
plt.show()


I am working on the file "Analysis_overview.ipynb".
The first task is:
Epocing
Can you write Python code for it?

epoch_data("P2", data_folder)

epochs = mne.read_epochs(data_folder + "P2_ds_bp_ica-epo.fif")


I am working on the file "Analysis_overview.ipynb".
The first task is:
Filter data

The filter_raw function apply a bandpass from 1Hz to 98Hz
Can you write Python code for it?


I am working on the file "Analysis_overview.ipynb".
The first task is:
From http://www.fieldtriptoolbox.org/tutorial/timefrequencyanalysis (Time-frequency analysis IV -- Morlet Wavelets). We get the wavelet duration is:
$$\Bigg(\frac{\frac{\text{Number of cycles}}{\text{Frequency}}}{\pi}\Bigg)\times 1000 = \text{Wavelet duration (in miliseconds)}$$
Can you write Python code for it?

Wavelet_duration_4 = calc_wavelet_duration(frequencies, n_cycles=4)
Wavelet_duration_5 = calc_wavelet_duration(frequencies, n_cycles=5)

plt.figure()
plt.plot(frequencies, Wavelet_duration_4, label="4 cycles")
plt.plot(frequencies, Wavelet_duration_5, label="5 cycles")
plt.ylabel("Wavelet duration (in ms)")
plt.xlabel("Frequencies")
plt.legend()
plt.show()

plt.figure()
plt.plot(frequencies, Wavelet_duration_5 - Wavelet_duration_4, label="difference")
plt.ylabel("Wavelet duration (in ms)")
plt.xlabel("Frequencies")
plt.title("Difference: 5 cycles - 4 cycles")
plt.legend()
plt.show()


I am working on the file "Analysis_overview.ipynb".
The first task is:
ICA
Compute ICA and automactically remove the component with the highest correlation with the EOG channel (EXGF4)
Can you write Python code for it?


I am working on the file "Analysis_overview.ipynb".
The first task is:
Intertrial coherence
Can you write Python code for it?

itc_vol.plot_topo(cmap="viridis", vmax=1, vmin=0)

itc_invol.plot_topo(cmap="viridis", vmax=1, vmin=0)


I am working on the file "Analysis_overview.ipynb".
The first task is:
Make single trial Morlet transform of epochs
The "frequencies", "n_cycles", and "times" are not used but for convinience when plotting etc.
Can you write Python code for it?

frequencies = np.arange(6., 30., 1.)
n_cycles = 5.
times = epochs.times

tfr_vol = single_trial_tf(epochs["voluntary"], n_cycles=4)
tfr_invol = single_trial_tf(epochs["involuntary"], n_cycles=4)

# convert to numpy array
tfr_vol = np.asarray(tfr_vol)
tfr_invol = np.asarray(tfr_invol)


I am working on the file "Analysis_overview.ipynb".
The first task is:
Plot results
Average power
Can you write Python code for it?

tfr_vol.plot_topo(baseline=(None, -1.8), mode="zscore", cmap="viridis", vmax=30, vmin=-30)

tfr_invol.plot_topo(baseline=(None, -1.8), mode="zscore", cmap="viridis", vmax=30, vmin=-30)


I am working on the file "Analysis_overview.ipynb".
The first task is:
Time-frequency analysis
Can you write Python code for it?

# load epochs
epochs = mne.read_epochs(data_folder + "P2_ds_bp_ica-epo.fif")

tfr_vol, itc_vol = morlet_analysis(epochs["voluntary"], n_cycles=4)
tfr_invol, itc_invol = morlet_analysis(epochs["involuntary"], n_cycles=4)

print("vol max: %s \ninvol max: %s") % (tfr_vol.data.max(), tfr_invol.data.max())
print("vol min: %s \ninvol min: %s") % (tfr_vol.data.min(), tfr_invol.data.min())


I am working on the file "Analytic FEM.ipynb".
The first task is:
Example: $2 \times 1$ cantilever beam
Can you write Python code for it?

coords = Matrix([
        [0, 0],
        [1, 0],
        [2, 0],
        [0, 1],
        [1, 1],
        [2, 1]])

elems = [[0, 1, 4, 3],
        [1, 2, 5, 4]]

mat_props = [[1, S(3)/10, 1]]*4

stiff, mass = assembler(coords, elems, mat_props, const="plane_stress")

load = zeros(12, 1)
load[5] = -S(1)/2
load[11] = -S(1)/2
load

stiff2 = stiff.copy()

stiff2[0, :] = eye(12)[0, :]
stiff2[:, 0] = eye(12)[:, 0]
stiff2[1, :] = eye(12)[1, :]
stiff2[:, 1] = eye(12)[:, 1]
stiff2[6, :] = eye(12)[6, :]
stiff2[:, 6] = eye(12)[:, 6]
stiff2[7, :] = eye(12)[7, :]
stiff2[:, 7] = eye(12)[:, 7]

sol = linsolve((stiff2, load))
sol

from IPython.core.display import HTML
def css_styling():
    styles = open('./styles/custom_barba.css', 'r').read()
    return HTML(styles)
css_styling()


I am working on the file "Analytic FEM.ipynb".
The first task is:
Example: $2 \times 2$ mesh
Can you write Python code for it?


I am working on the file "Analytic FEM.ipynb".
The first task is:
Example: Single element
Can you write Python code for it?

coords = Matrix([
    [-1, -1],
    [1, -1],
    [1, 1],
    [-1, 1]])

elems = [[0, 1, 2, 3]]

mat_props = [[S(8)/3, S(1)/3, 1]]

stiff, mass = assembler(coords, elems, mat_props, const="plane_strain")

stiff

mass


I am working on the file "Analytic FEM.ipynb".
The first task is:
Interpolation functions and matrices
Can you write Python code for it?


I am working on the file "Analytic FEM.ipynb".
The first task is:
The interpolation matrix is a matrix with the interpolation
functions reordered for each degree of freedom.
Can you write Python code for it?

H = zeros(2, 8)
for cont in range(4):
    H[0, 2*cont] = N[cont]
    H[1, 2*cont + 1] = N[cont]

display(H.T)


I am working on the file "Analytic FEM.ipynb".
The first task is:
The local derivatives matrix is formed with the derivatives of the interpolation functions
Can you write Python code for it?

dHdr = zeros(2, 4)
for cont in range(4):
    dHdr[0, cont] = diff(N[cont], r)
    dHdr[1, cont] = diff(N[cont], s)

display(dHdr)


I am working on the file "Analytic FEM.ipynb".
The first task is:
With these elements we can form the local stiffness and mass matrices.
Can you write Python code for it?

def local_mass(H, coord_el, rho):
    det = jaco(dHdr, coord_el).det()
    integrand = rho * det * expand(H.T * H)
    mass_mat = zeros(8, 8)
    for row in range(8):
        for col in range(row, 8):
            mass_mat[row, col] = gauss_int2d(integrand[row, col], r, s)
            mass_mat[col, row] = mass_mat[row, col]

    return mass_mat

def local_stiff(dHdr, coord_el, C):
    det = jaco(dHdr, coord_el).det()
    B = B_mat(dHdr, coord_el)
    integrand = det * expand(B.T * C * B)
    stiff_mat = zeros(8, 8)
    for row in range(8):
        for col in range(row, 8):
            stiff_mat[row, col] = gauss_int2d(integrand[row, col], r, s)
            stiff_mat[col, row] = stiff_mat[row, col]

    return stiff_mat


I am working on the file "Analytical Helmholtz.ipynb".
The first task is:
Class Definition
Here, we build a class called AnalyticalHelmholtz, which inherits from object.
The __init__ method is called automatically when we instantiate the class (i.e., make a new object based on AnalyticalHelmholtz). The __init__ method expects one or two inputs: 1) a required input (dictionary) called sc, which contains the configuration for the system, and 2) an optional input called hf, which defaults to the hankel function from scipy.special.
When running __init__, Python first stores some properties from the sc parameter, and determines whether it is going to run in 2D mode or 3D mode. At the end of this method, a member object _hf is created, which stores a reference to the chosen Hankel function (from hf).
When an object of class AnalyticalHelmholtz is called like a function, Python executes the __call__ method. This uses whichever Green's function was set up during __init__, and returns a wavefield. The distance from the source is calculated for each point in the grid (from the $x$ and $z$ locations passed into the function call), and the response is returned as a numpy array (based on the configuration in the sc object from when the AnalyticalHelmholtz class was instantiated).
NB: k is a property, which behaves like a variable bound to the instance, but actually runs some code inside the k method when its value is asked for. This way, it can be calculated and stored based on the omega and c properties.
Can you write Python code for it?

class AnalyticalHelmholtz(object):
    '''
    Implements the analytical Green's functions for the 2D and 3D
    Helmholtz equation(s).
    '''

    def __init__(self, sc, hf = hankel1):

        self.dimension = sc.get('dimension', 2)
        self.omega = sc['omega']
        self.c = sc['c']

        if self.dimension == 3:
            self.Green = self.Green3D
        else:
            self.Green = self.Green2D

        self._x, self._z = np.mgrid[
            sc['xorig']:sc['xorig']+sc['dx']*sc['nx']:sc['dx'],
            sc['zorig']:sc['zorig']+sc['dz']*sc['nz']:sc['dz']
        ]

        self._hf = hf

    def Green2D(self, x):

        return -0.5j * self._hf(0, self.k*x)

    def Green3D(self, x):

        return (1./(4*np.pi*x)) * np.exp(1j*self.k*x)

    def __call__(self, x, z):

        return np.nan_to_num(self.Green(np.sqrt((x - self._x)**2 + (z - self._z)**2)))

    @property
    def k(self):
        if getattr(self, '_k', None) is None:
            self._k = self.omega / self.c
        return self._k


I am working on the file "Analytical Helmholtz.ipynb".
The first task is:
Imports
Here, we import the necessary libraries to do the computations: numpy, which is the main scientific & numerical computing library in Python, and the hankel1 and hankel2 functions from scipy.special.
Can you write Python code for it?

import numpy as np
from scipy.special import hankel1, hankel2


I am working on the file "Analytical Helmholtz.ipynb".
The first task is:
Plotting
We create a dict called plotopts, which contains a number of default settings for the imshow function. We then use the matplotlib object-oriented interface to build up a figure. We have 4 subplots, each of which includes a call to plt.imshow to plot the appropriate wavefield, and a call to plt.title to label it.
NB: The call to fig.tight_layout rearranges the plots so that their axis labels are not overlapping.
Can you write Python code for it?

plotopts = {
    'cmap':     cm.bwr,
    'vmin':     -1.,
    'vmax':     1.,
    'extent':   [
        systemConfig['xorig'],
        systemConfig['xorig'] + systemConfig['dx'] * systemConfig['nx'],
        systemConfig['zorig'] + systemConfig['dz'] * systemConfig['nz'],
        systemConfig['zorig'],
    ],
}

fig = plt.figure()

ax = fig.add_subplot(1,4,1, aspect=1)
plt.imshow(u1.real.T, **plotopts)
plt.title('$\mathscr{R}\{ H_0^{(1)} \}$')

ax = fig.add_subplot(1,4,2, aspect=1)
plt.imshow(u2.real.T, **plotopts)
plt.title('$\mathscr{R}\{ H_0^{(2)} \}$')

ax = fig.add_subplot(1,4,3, aspect=1)
plt.imshow(u1.imag.T, **plotopts)
plt.title('$\mathscr{I}\{ H_0^{(1)} \}$')

ax = fig.add_subplot(1,4,4, aspect=1)
plt.imshow(u2.imag.T, **plotopts)
plt.title('$\mathscr{I}\{ H_0^{(2)} \}$')

fig.tight_layout()


I am working on the file "Analytics2.ipynb".
The first task is:
Data Processing

We need to make a column for average stats for each 'mon
We need to label each 'mon by its generation

(We should figure out a way to ignore non-stat changed formes i.e. Arceus, as he may be upsetting the Gen IV data)
Can you write Python code for it?

mons["AVERAGE_STAT"] = mons["STAT_TOTAL"]/6

gens = pd.Series([0 for i in range(len(mons.index))], index=mons.index)
for ID, mon in mons.iterrows():
    if 0<mon.DEXID<=151:
        gens[ID] = 1
    elif 151<mon.DEXID<=251:
        gens[ID] = 2
    elif 251<mon.DEXID<=386:
        gens[ID] = 3
    elif 386<mon.DEXID<=493:
        gens[ID] = 4
    elif 493<mon.DEXID<=649:
        gens[ID] = 5
    elif 649<mon.DEXID<=721:
        gens[ID] = 6
    elif 721<mon.DEXID<=805:
        gens[ID] = 7
    else:
        gens[ID] = 0

mons["GEN"] = gens

mons.to_csv("./data/pokemon_preUSUM_data.csv")

gen = {}
for i in range(1,8):
    gen[i] = mons[mons.GEN == i]

plt.figure(100)
colors = sns.color_palette("colorblind", 7)

for i in range(1,8):
    sns.distplot( mons[mons["GEN"] == i]["STAT_TOTAL"], hist=False,kde=True, color=colors[i-1], label=f"Gen {i}")
plt.legend()
plt.show()


I am working on the file "Analytics2.ipynb".
The first task is:
K-Means Clustering
Can you write Python code for it?

from sklearn import cluster

k_means = cluster.KMeans(n_clusters = 6)

k_means.fit(X)

mons["KMEANS_LABEL"] = pd.Series(k_means.labels_)

plotData = mons[["GEN", "STAT_TOTAL", "KMEANS_LABEL"]]

colors = sns.color_palette("colorblind", 7)

for i in range(1,8):
    sns.distplot( plotData[plotData["GEN"] == i]["STAT_TOTAL"], color=colors[i-1])

plt.figure(925)
sns.boxplot(x="KMEANS_LABEL", y="STAT_TOTAL", data=plotData)
plt.show()

plt.figure(9050624)
sns.pairplot(plotData, kind="scatter", hue="GEN", palette=colors)
plt.show()

plotData.to_csv("./data/kmeans.csv")


I am working on the file "Analytics2.ipynb".
The first task is:
Machine Learning and Clustering
Can you write Python code for it?

np.random.seed(525_600)

stats_gens = mons[['HP', 'ATTACK', 'DEFENSE',
       'SPECIAL_ATTACK', 'SPECIAL_DEFENSE', 'SPEED', 'GEN']]

X = np.c_[stats_gens]


I am working on the file "Analytics2.ipynb".
The first task is:
PCA
Can you write Python code for it?

pca = decomposition.PCA()
pca.fit(X)

pca.explained_variance_

pca.n_components = 3

X_reduced = pca.fit_transform(X)

X_reduced.shape

pca.get_params()


I am working on the file "Analytics2.ipynb".
The first task is:
Some Stats
Can you write Python code for it?

stat_averages_by_gen = {i:gen[i].AVERAGE_STAT for i in range(1,8)}

testable_data = list(stat_averages_by_gen.values())
data = [list(gen) for gen in testable_data]
data = np.array(data)

averages = {i: stat_averages_by_gen[i].mean() for i in range(1,8)}
averages

stats.kruskal(*data)

recarray = mons.to_records()

test = comp.pairwise_tukeyhsd(recarray["AVERAGE_STAT"], recarray["GEN"])

test.summary()


I am working on the file "Analyze precipitation data.ipynb".
The first task is:
After defining the compute_slope() method, use the pandas apply() method to compute the slope for each row in the DataFrame and add the computed slope value to a new column called TREND:
Can you write Python code for it?

precipitation_df["TREND"] = precipitation_sorted_df[years].apply(compute_slope, axis=1)
precipitation_df.head()


I am working on the file "Analyze precipitation data.ipynb".
The first task is:
Each row in the table lists:

The country or region where the measurements were taken
The annual precipitation for 1990, and for the years 1995 to 2009

By using the DataFrame API, you can list all countries or areas for which precipitation measurements exist. Run the following cell to list these countries:
Can you write Python code for it?


I am working on the file "Analyze precipitation data.ipynb".
The first task is:
Finally, compare the annual precipitation for the top five countries. China has the highest annual precipitation, followed by Colombia. The line graph also shows that some values are missing, which makes comparing measurements difficult.
To show the total precipitation for the top five countries in relation to the total precipitation for all countries over the entire time period, use a pie chart. Run the next code cell to create a pie chart by using the matplotlib library:
Can you write Python code for it?

precipitation_sums = top5_sums
other_sums = precipitation_sorted_df["SUM"][5:].sum()
precipitation_sums["Other"] = other_sums

plt.axis('equal')
plt.title("Annual precipitation percentage",y=1.08)
plt.pie(
    precipitation_sums,
    labels=precipitation_sums.index,
    colors=['blue', 'green', 'red', 'turquoise', 'magenta','yellow'],
    autopct="%1.2f%%",
    radius=1.25);


I am working on the file "Analyze precipitation data.ipynb".
The first task is:
For Chile, the line plot shows a positive trend in annual precipitation, despite the fact that during the last couple of years the annual precipitation decreased.
The next cells show you how to determine the precipitation trends for 91 countries.
Run the next code cell to define the compute_slope() method to compute the slope of a fitted line  between given points:
Can you write Python code for it?

def compute_slope(row):
    """ This method computes the slope of a fittet line for given data points.
    For arguments containing less than two valid data points it returns a NaN."""
    slope = np.nan
    years = row[row>0.0].index
    if(len(years)>=2):
        fit = np.polyfit(years.astype('int'), row[years].values,1)
        slope = fit[0]
    return slope


I am working on the file "Analyze precipitation data.ipynb".
The first task is:
Having the names of the countries or areas as a column comes in handy, but it would be more useful to have the names as an index to the rows. To manipulate the DataFrame in this way, run the next cell:
Can you write Python code for it?

precipitation_df = precipitation_df.set_index(precipitation_df["Country or Area"])
precipitation_df.drop(['Country or Area'], axis=1, inplace=True)
precipitation_df.head()


I am working on the file "Analyze precipitation data.ipynb".
The first task is:
Next, sort the DataFrame by total precipitation and print the top five countries or areas with the highest total precipitation:
Can you write Python code for it?

precipitation_sorted_df = precipitation_df.sort_values(by="SUM",ascending=False)
top5_sums = pd.Series(precipitation_sorted_df["SUM"].head(5))
top5_sums


I am working on the file "Analyze precipitation data.ipynb".
The first task is:
Now, plot the top five countries next to each another in a line graph for a better comparison of the recordings:
Can you write Python code for it?

top5_bars = precipitation_sorted_df[years][0:5].transpose()
ax = top5_bars.plot(figsize=(10,8), marker='o', linestyle='-', title="Top 5 Countries with highest Precipitation")
ax.set_xlabel("Years")
ax.set_ylabel("Precipitation (million cubic meters)");


I am working on the file "Analyze precipitation data.ipynb".
The first task is:
Plot the countries or areas with the highest total precipitation
To get a list of the countries with the highest total precipitation, you have to compute the sum of all annual precipitation. You can do this by using the pandas sum() method. This method adds an  additional column called SUM, which contains the sums of the annual precipitation values to the precipitation_df DataFrame.
Can you write Python code for it?


I am working on the file "Analyze_USGS_data.ipynb".
The first task is:
Load USGS data into a dataframe
Can you write Python code for it?

# Use HydroCloud.org to find a stream gauge to investigate.
# Click on the red points to find the site number.
from IPython.display import HTML
HTML('<iframe src=http://hydrocloud.org/ width=700 height=400></iframe>')

# Create a Pandas dataframe using the USGS daily discharge for Herring Run.
herring = hp.get_usgs('01585200', 'dv', '2011-01-01', '2016-01-01')

# List the first few values from the top of the dataframe.
herring.head()

# Calculate some basic statistics for the dataframe.
herring.describe()

# For more advanced analysis, use the HydroAnalysis class.
my_analysis = hp.HydroAnalysis(herring)

# Plot discharge on a logarithmic scale for the Y axis.
my_analysis.plot(figsize=(16,6), logy=True)

## Finding Help

# Use help() to learn more about a particular function.
help(hp.get_usgs)


I am working on the file "Analyzes/Volatile movements/01 Volatile movements in python and pandas 1.ipynb".
The first task is:
Identifikace volatilní úsečky
Volatilní svíce identifikuji pomocí funkcionality rolling a funkce apply. Funkcionalita rolling umožnuje rozdělit pandas DataFrame na jednotlivé menší "okna", které bude předávat postupně funkci apply v parametru. Tzn. že v následujícím kódu se provede výpočet funkce is_bigger pro každý řádek dat uložených v spy_data. Do parametru rows se bude postupně vkládat výřez dat, obsahující 4 řádky (aktuální počátaný řádek + 3 řádky předchozí). Jako výsledek funkce is_bigger bude hodnota, zda je aktuálně počítaný řádek volatilnější, než 4 předchozí.
Can you write Python code for it?

def is_bigger(rows):
    result = rows[-1] > rows[:-1].max() # rows[-1] - poslední hodnota je větší než maximum z předchozích
    return result

spy_data['VolBar'] = spy_data['Abs(C-O)'].rolling(4).apply(is_bigger,raw=True)
spy_data.tail(10)


I am working on the file "Analyzes/Volatile movements/01 Volatile movements in python and pandas 1.ipynb".
The first task is:
Které svíce jsou volatilnější, než 4 předchozí, si zobrazím pomocí jednoduché selekce, kde ve sloupečku VolBar == 1.
Can you write Python code for it?


I am working on the file "Analyzes/Volatile movements/01 Volatile movements in python and pandas 1.ipynb".
The first task is:
Nyní znám přesnou změnu ceny každý den. Abych mohl porovnávat velikosti, aniž by mi záleželo na tom, zda se daný den cena propadla, nebo stoupla, aplikuji absolutní hodnotu.
Can you write Python code for it?

spy_data['Abs(C-O)'] = spy_data['C-O'].abs()
spy_data.tail()


I am working on the file "Analyzes/Volatile movements/02 Volatile movements in python and pandas 2.ipynb".
The first task is:
Analýza rozložení výnosů po volatilní svíčce
Can you write Python code for it?

BINS = 50 # pouze pomocná proměnná, která určuje, kolik maximálně má být zobrazeno sloupečků v histogramu


I am working on the file "Analyzes/Volatile movements/02 Volatile movements in python and pandas 2.ipynb".
The first task is:
Nyní přichází na řadu princip analýzy, na který chci tímhle článkem upozornit.
Analýza vlivu určité podmínky na budoucí výsledek
Jako podmínku beru volatilní úsečku, tu mám označenou ve sloupci VolBar a změnu ceny v procentech mám ve sloupečku pct_change. Nyní můžu posunout data ve sloupečku pct_change o jeden řádek výše a získám tak procentuální změnu Close ceny (výsledek) v následující svíčce v řádku s označeným VolBarem (podmínka).
Can you write Python code for it?

spy_data['pct_change-1'] = spy_data['pct_change'].shift(-1)
spy_data[spy_data['VolBar']].head()


I am working on the file "Analyzes/Volatile movements/02 Volatile movements in python and pandas 2.ipynb".
The first task is:
Pandas.DataFrame má jednoduchou funkci, která se nazývá pct_change. Ta mi zjistí procentuální změnu hodnoty aktuálního řádku od přechozího. V mém případě to v jednoduchosti znamená, že zjistím, jak se dnešní Close procentuálně změnilo od včerejška. Výsledek vložím do nového sloupečku s názvem pct_change.
Pozn.: Uvažuji obchodování na Close cenách jednotlivých svíček.
Can you write Python code for it?

spy_data['pct_change'] = spy_data.Close.pct_change()
spy_data.head()


I am working on the file "Analyzes/Volatile movements/02 Volatile movements in python and pandas 2.ipynb".
The first task is:
Proměnná up_volbars nyní obsahuje pouze řádky stoupajících volatilních úseček => jejich uzavírací cena Close je výšší než otevírací cena Open. Pro klesající volatilní úsečky v proměnné down_volbars platí opačná podmínka.
Pravděpodobnost návratu po reverzní svíčce
Příprava filtrů definujících reverzní pohyb po volatilní úsečce. Kladný pct_change-1 znamená, že je následující svíčka stoupající a záporný klesající.
Can you write Python code for it?

down_revers_filter = up_volbars['pct_change-1']<0
up_revers_filter = down_volbars['pct_change-1']>0


I am working on the file "Analyzes/Volatile movements/02 Volatile movements in python and pandas 2.ipynb".
The first task is:
Rozložení velikosti výnosů up reverzu po down volatilním pohybu
Podobný rozbor pro reverzní pohyb po volatilní klesající úsečce:
Can you write Python code for it?


I am working on the file "Analyzes/Volatile movements/02 Volatile movements in python and pandas 2.ipynb".
The first task is:
Stažení vzorových dat
Nejprve si připravím vzorová data. Například využiji trh SPY, což je ETF kopírující hlavní americký akciový index S&P 500.
Can you write Python code for it?

start = datetime.datetime(2005, 1, 1)
end = datetime.datetime(2018, 12, 31)

spy_data = web.DataReader('SPY', 'yahoo', start, end)
spy_data = spy_data.drop(['Volume', 'Adj Close'], axis=1) # sloupce 'Volume' a 'Adj Close' nebudu potřebovat
spy_data.tail()


I am working on the file "Analyzes/Volatile movements/02 Volatile movements in python and pandas 2.ipynb".
The first task is:
V předchozím článku jsem definoval, jak najdu volatilní svíčku. Připomínám, že se jedná o svíčku, která je větší než 4 předchozí svíčky.
Abych tento kód nemusel psát pořád dokola, vytvořil jsem si osobní balíček, obsahující mé používané analýzy. Zde je jednoznačná výhoda pythonu a jiných programovacích jazyků. Funkce volatile_bars přebírá jako první parametr tabulku dat typu pandas.DataFrame, která musí obsahovat sloupečky Open, High, Low, Close. Jako jednotlivé řádky bere data svíček trhů, takže tam klidně můžu mít data jiného timeframe.
Funkce volatile_bars vytvoří nový sloupeček s názvem VolBar, ve kterém označí volatilní svíčky hodnotou True a všechny ostatní svíčky hodnotou False.
Can you write Python code for it?

# vhat je můj osobní balíček analytických funkcí, které nechci psát pokaždé znovu a znovu
from vhat.analyse.volbars import volatile_bars

volatile_bars(spy_data, N=4, drop_calculations=False)
spy_data.tail()


I am working on the file "Analyzes/Volatile movements/02 Volatile movements in python and pandas 2.ipynb".
The first task is:
Volatilní down svíčka
Can you write Python code for it?

perc = down_volbars[up_revers_filter].shape[0] / down_volbars.shape[0]
print('Probability of the following up bar after the volatile short bar:', f'{perc*100:.2f}%')


I am working on the file "Analyzes/Volatile movements/02 Volatile movements in python and pandas 2.ipynb".
The first task is:
Volatilní up svíčka
Výpočet pravděpodobnosti, že po volatilní úsečce bude následovat klesající úsečka:
Can you write Python code for it?

perc = up_volbars[down_revers_filter].shape[0] / up_volbars.shape[0]
print('Probability of the following short bar after the volatile up bar:', f'{perc*100:.2f}%')


I am working on the file "Analyzes/Volatile movements/02 Volatile movements in python and pandas 2.ipynb".
The first task is:
Více se mi ale líbí možnosti knihovny seaborn, která využívá pandas.DataFrame a pandas.Series jako vstupní data a pro statistické účely je dokáže pěkně zobrazit.
Funkce Pandas.DataFrame.describe() vypočítá pro dané data základní statistické údaje jako jsou průměr, směrodatná odchylka, atd.
Can you write Python code for it?

import seaborn as sns
sns.distplot(up_volbars.loc[down_revers_filter,'pct_change-1'], bins=BINS)
up_volbars.loc[down_revers_filter,'pct_change-1'].describe()


I am working on the file "Analyzes/Volatile movements/02 Volatile movements in python and pandas 2.ipynb".
The first task is:
Výsledek
Z výše popsané analýzy jsem zjistil následující charakteristiku pro volatilní svíčky v trhu SPY v časovém období od 1.1.2005 do 31.12.2018:

Po volatilní stoupající svíčce následovala reverzní svíce (klesající) v 44.83% případů.
Po volatilní klesající svíčce následovala reverzní svíce (stoupající) v 57.89% případů.

Už tenhle první výsledek nám ukazuje, že:

Má větší smysl hledat v trhu SPY reverz po klesající volatilní úsečce směrem nahoru.

Samozřejmě je to dáno charakterem trhu SPY. Na grafu SPY je možné vidět na první pohled, že má spíše tendenci stoupat:
Can you write Python code for it?


I am working on the file "Analyzing product sentiment.ipynb".
The first task is:
Applying the learned model to understand sentiment for Giraffe
Can you write Python code for it?

giraffe_reviews['predicted_sentiment'] = sentiment_model.predict(giraffe_reviews, output_type='probability')

giraffe_reviews.head()


I am working on the file "Analyzing product sentiment.ipynb".
The first task is:
Build the word count vector for each review
Can you write Python code for it?

products['word_count'] = graphlab.text_analytics.count_words(products['review'])

products.head()

graphlab.canvas.set_target('ipynb')

products['name'].show()


I am working on the file "Analyzing product sentiment.ipynb".
The first task is:
Define what's a positive and a negative sentiment
We will ignore all reviews with rating = 3, since they tend to have a neutral sentiment.  Reviews with a rating of 4 or higher will be considered positive, while the ones with rating of 2 or lower will have a negative sentiment.
Can you write Python code for it?

#ignore all 3* reviews
products = products[products['rating'] != 3]

#positive sentiment = 4* or 5* reviews
products['sentiment'] = products['rating'] >=4

products.head()


I am working on the file "Analyzing product sentiment.ipynb".
The first task is:
Evaluate the sentiment model
Can you write Python code for it?

sentiment_model.evaluate(test_data, metric='roc_curve')

sentiment_model.show(view='Evaluation')


I am working on the file "Analyzing product sentiment.ipynb".
The first task is:
Examining the reviews for most-sold product:  'Vulli Sophie the Giraffe Teether'
Can you write Python code for it?

giraffe_reviews = products[products['name'] == 'Vulli Sophie the Giraffe Teether']

len(giraffe_reviews)

giraffe_reviews['rating'].show(view='Categorical')


I am working on the file "Analyzing product sentiment.ipynb".
The first task is:
Let's explore this data together
Data includes the product name, the review text and the rating of the review.
Can you write Python code for it?


I am working on the file "Analyzing product sentiment.ipynb".
The first task is:
Let's train the sentiment classifier
Can you write Python code for it?

train_data,test_data = products.random_split(.8, seed=0)

sentiment_model = graphlab.logistic_classifier.create(train_data,
                                                     target='sentiment',
                                                     features=['word_count'],
                                                     validation_set=test_data)


I am working on the file "Analyzing product sentiment.ipynb".
The first task is:
Most positive reviews for the giraffe
Can you write Python code for it?


I am working on the file "Analyzing product sentiment.ipynb".
The first task is:
Read some product review data
Loading reviews for a set of baby products.
Can you write Python code for it?


I am working on the file "Analyzing product sentiment.ipynb".
The first task is:
Show most negative reviews for giraffe
Can you write Python code for it?

giraffe_reviews[-1]['review']

giraffe_reviews[-2]['review']

selected_words = ['awesome', 'great', 'fantastic', 'amazing', 'love', 'horrible', 'bad', 'terrible', 'awful', 'wow', 'hate']

def awesome_count(word_count):
    return word_count.get('awesome', 0)

products['awesome'] = products['word_count'].apply(awesome_count)

def get_count(word_count, word):
    return word_count.get(word, 0)

products['awesome'].head()

products['awesome'].sum()

for word in selected_words:
    products[word] = products['word_count'].apply(lambda word_count: get_count(word_count, word))

products.head()

len(selected_words)

for word in selected_words:
    print word, products[word].sum()

train_data, test_data = products.random_split(.8, seed=0)

simple_model = graphlab.logistic_classifier.create(train_data,
                                                     target='sentiment',
                                                     features=selected_words,
                                                     validation_set=test_data)

simple_model['coefficients'].sort('value').print_rows(15)

simple_model.evaluate(test_data)

sentiment_model.evaluate(test_data)

diaper_champ_reviews = products[products['name']=='Baby Trend Diaper Champ']

diaper_champ_reviews['predicted_sentiment'] = sentiment_model.predict(diaper_champ_reviews, output_type='probability')

diaper_champ_reviews = diaper_champ_reviews.sort('predicted_sentiment', ascending=False)

simple_model.predict(diaper_champ_reviews[0:1], output_type='probability')

diaper_champ_reviews[0]

test_data['sentiment'].show(view='Categorical')

products[['word_count']].stack('word_count',new_column_name=['word', 'count'])


I am working on the file "Analyzing product sentiment.ipynb".
The first task is:
Sort the reviews based on the predicted sentiment and explore
Can you write Python code for it?

giraffe_reviews = giraffe_reviews.sort('predicted_sentiment', ascending=False)

giraffe_reviews.head()


I am working on the file "Analyzing_Subway_Data_NDFDSI.ipynb".
The first task is:
*Exercise 2.4
Now, calculate the mean for 'meantempi' for the days that are Saturdays or Sundays (weekend):
Can you write Python code for it?

def avg_weekend_temperature(filename):

    mean_temp_weekends =

    return mean_temp_weekends


I am working on the file "Analyzing_Subway_Data_NDFDSI.ipynb".
The first task is:
*Exercise 2.5
Calculate the mean of the minimum temperature 'mintempi' for the days when the minimum temperature was greater that 55 degrees:
Can you write Python code for it?

def avg_min_temperature(filename):

    avg_min_temp_rainy =

    return avg_min_temp_rainy


I am working on the file "Analyzing_Subway_Data_NDFDSI.ipynb".
The first task is:
*Exercise 2.6
Before you make any analysis, it might be useful to look at the data we want to analyse. More specifically, we will evaluate the entries by hour in our data from the NYC Subway to determine the data distribution. This data is stored in the column ['ENTRIESn_hourly'].

Draw two histogramns in the same axis, to show the entries when it's raining vs when it's not.
Below, you will find an example of how to draw histogramns with Pandas and Matplotlib:

python
Turnstile_weather ['column_to_graph']. Hist ()
Can you write Python code for it?

import numpy as np
import pandas
import matplotlib.pyplot as plt

def entries_histogram(turnstile_weather):

    plt.figure()
    turnstile_weather['...'] # your code here to plot a historgram for hourly entries when it is raining
    turnstile_weather['...'] # your code here to plot a histogram for hourly entries when it is not raining
    return plt


I am working on the file "Analyzing_Subway_Data_NDFDSI.ipynb".
The first task is:
*Exercise 2.7
The data you just plotted is in what kind of distribution? Is there a difference in distribution between rainy and non-rainy days?
 Answer : Replace this text with your answer!
*Exercise 2.8
Build a function that returns:

The mean of entries when it's raining
The mean of entries when it's not raining
Can you write Python code for it?

import numpy as np

import pandas

def means(turnstile_weather):

    ### YOUR CODE HERE ###

    return with_rain_mean, without_rain_mean, p # leave this line for the grader


I am working on the file "Analyzing_Subway_Data_NDFDSI.ipynb".
The first task is:
Exercise 1.2
Write down a function that takes the list of all names of the files you downloaded in Exercise 1.1 and compile them into one single file. There must be only one header line in the output file.
For example, if file_1 has:
line 1...
line 2...
and the other file, file_2, has:
line 3...
line 4...
line 5...
We must combine file_1 and file_2 into one master file, as follows:
'C/A, UNIT, SCP, DATEn, TIMEn, DESCn, ENTRIESn, EXITSn'
line 1...
line 2...
line 3...
line 4...
line 5...
Can you write Python code for it?

def create_master_turnstile_file(filenames, output_file):
    with open(output_file, 'w') as master_file:
        master_file.write('C/A,UNIT,SCP,STATION, LINENAME, DIVISION, DATEn,TIMEn,DESCn,ENTRIESn,EXITSn\n')
        for filename in filenames:
            # your code here


I am working on the file "Analyzing_Subway_Data_NDFDSI.ipynb".
The first task is:
Exercise 1.5
Do the same thing you did in the previous exercise, but taking into account the exits, column EXITSn.
For this, you need to create a column called EXITSn_hourly and insert the difference between the column EXITSn in the current line vs he previous line. If there is any NaN, fill it out/replace by 0.
Can you write Python code for it?

import pandas

def get_hourly_exits(df):

    #your code here
    return df


I am working on the file "Analyzing_Subway_Data_NDFDSI.ipynb".
The first task is:
Exercise 1.6
Given an entry variable that represents time, in the format:
     "00:00:00" (hour: minutes: seconds)

Write a function to extract the hour part from the time in the entry variable
And return it as an integer. For example:

         1) if hour is 00, your code must return 0
         2) if hour is 01, your code must return 1
         3) if hour is 21, your code must return 21

Please return te hour as an integer.
Can you write Python code for it?

def time_to_hour(time):

    hour = # your code here
    return hour


I am working on the file "Analyzing_Subway_Data_NDFDSI.ipynb".
The first task is:
Exercise 2.2
Now, create a function that calculates the number of rainy days. For this, return the count of the number of days where the column "rain" is equal to 1.
Tip: You might think that interpreting numbers as integers or floats might not
     work at first. To handle this issue, it might be useful to convert
     these numbers into integers. You can do this by writting cast (column as integer).
     So, for example, if we want to launch the column maxtempi as an integer, we have to
     write something like cast (maxtempi as integer) = 76, instead of just
     where maxtempi = 76.
Can you write Python code for it?

def num_rainy_days(df):

    #your code here
    return


I am working on the file "Anillos de Newton/Anillos de Newton.ipynb".
The first task is:
El patrón de interferencias que aparece al reflejarse la luz entre dos superficies transparentes, una curva y otra plana, es conocido como Anillos de Newton.
El patrón consiste en un conjunto de anillos centrados en el punto de contacto de las dos superficies. Se trata de franjas de luz de igual espesor, es decir, cada anillo representa la región en la cual el espesor que hay entre las dos superficies es el mismo. El espesor de la lámina que queda encerrada entre las dos superficies aumenta más rápidamente al alejarnos del centro (punto de contacto). Por ello los anillos más alejados del centro están más juntos.
Esta técnica se emplea para evaluar la calidad de superficies ópticas, veáse como ejemplo el siguiente artículo sobre caracterización de lentes oftálmicas.
Can you write Python code for it?

from IPython.display import Image
Image(filename="PaperAnillosNewton.JPG")


I am working on the file "Anillos de Newton/Anillos de Newton.ipynb".
The first task is:
Teoría
Para estudiar los anillos de Newton analizamos las ondas reflejadas en la lámina de espesor variable (generalmente de aire) que se forma al depositar una lente plano-convexa sobre una superficie plana, tal y como se muestra en la siguiente figura. El radio de curvatura de la lente es $R$. Según nos alejamos del centro (punto de contacto) el espesor de la capa de aire aumenta. Nos quedamos a una distancia $x$ del centro. A esa distancia el espesor de la capa de aire es $e$. Si iluminamos el sistema desde arriba con una haz colimado de longitud de onda $\lambda$, la onda reflejada en la cara inferior de la lente plano-convexa y la onda reflejada en la cara superior de la superficie plana interfieren. Y lo hacen como si se tratara de una lámina delgada de espesor $e$. Considerando incidencias muy cercanas a la normal, la diferencia de camino entre esas dos ondas será:
$$ \Delta = 2 e$$
y por lo tanto el desfase
$$ \delta = \frac{2 \pi}{\lambda} 2 e + \pi$$
donde hemos incluido un desfase de $\pi$ debido a las reflexiones, ya que una onda se refleja en la interfase vidrio-aire y la otra lo contrario (aire-vidrio).
El desfase entre las dos ondas depende del espesor de la capa de aire. Este espesor va aumentando al alejarnos del centro con lo cual la condición de interferencia irá igualmente variando, pasando por máximos ($\delta=2m\pi$) y mínimos $\delta=(2m+1)\pi$ de luz.
Justo en el centro o punto de contacto el espesor es nulo. Así que el desfase será $\pi$, el correspondiente a las reflexiones, y tendremos que las ondas interfieren destructivamente. Por eso en el punto de contacto se observará un mínimo de luz.
Can you write Python code for it?

from IPython.display import Image
Image(filename="esquemaAnillosNewton1.jpg")


I am working on the file "Anillos de Newton/Anillos de Newton.ipynb".
The first task is:
Vamos a calcular a que distancias del centro tenemos mínimos de luz, o lo que es lo mismo las posiciones de los anillos oscuros.
Para ello vamos a expresar el espesor de la capa de aire $e$ en función de la distancia al centro $x$ y del radio de la lente $R$.
Nos fijamos en la figura anterior en el triángulo rectángulo marcado con un patrón de rayas diagonales. La relación entre los lados del triángulo es:
$$ R^2 = x^2 + (R-e)^2 $$
Estamos considerando que la capa de aire es delgada y por tanto el espesor será mucho menor que el valor del radio de la lente $R$. Así podemos escribir $(R-e)^2 = R^2 + e^2 - 2 e R \simeq R^2 - 2 e R $, es decir, despreciar $e^2$. Por tanto nos queda
$$ R^2 = x^2 + R^2 - 2 e R $$
y despejando el espesor de la expresión anterior obtenemos
$$ e =  \frac{x^2}{2R} $$
es decir, el espesor varía de forma parabólica con la distancia al centro. Además la variación del espesor es más apreciable cuando el radio de la lente es más pequeño. Es decir, el radio de la lente va a influir en como varía el espesor de la lámina de aire y por tanto en la distribución de luz resultante.
Si empleamos el espesor anterior en el desfase entre las dos ondas obtenemos finalmente
$$ \delta = \frac{2 \pi}{\lambda} 2 \frac{x^2}{2R}  + \pi $$
Distribución de los anillos oscuros
Ahora ya estamos en disposición de calcular las distancias al centro $x$ para obtener mínimos de luz $(\delta=(2m+1)\pi)$
$$ \frac{2 \pi}{\lambda} \frac{x^2}{R}  + \pi  = 2m\pi + \pi$$
Despejamos $x$ y obtenemos lo que llamamos los radios de los anillos oscuros
$$ x_m^2 = m \lambda R $$
Vemos que el cuadrado de los radios de los anillos oscuros es proporcional al radio de la lente. La distancia o separación entre anillos oscuros consecutivos no es fija, y por eso no se habla de la interfranja. En particular la distancia entre el anillo de orden $m$ y el $m+1$ es
$$ x_{m+1} - x_m = \sqrt{\lambda R} \left( \sqrt{m+1} - \sqrt{m} \right)$$
Con esta expresión podemos comprobar fácilmente que la distancia entre los anillos oscuros cerca del...
Can you write Python code for it?

from IPython.display import Image
Image(filename="esquemaAnillosNewton2.jpg")


I am working on the file "Animating a family-of-complex-functions.ipynb".
The first task is:
Define a family of complex functions, depending on the parameter t. To animate the motion of function zeros and poles, as t varies,
we use functools.partial to get the
function corresponding to a particular parameter:
Can you write Python code for it?

from functools import partial
def func(t, z):
    return z**4+np.exp(2*t*1j)/z**2*np.exp(1j*t)
f0 =  partial(func, 0)

x, y, z =  evaluate_function(f0, re=(-1.5, 1.5), im=(-1.5,1.5), N=50)
w = f0(z)
argument = np.angle(w)

fig = go.Figure(go.Heatmap(x=x, y=y, z=argument,  colorscale=pl_hsv,
                    colorbar=dict(thickness=20, tickvals=tickvals,
                    ticktext=ticktext,
                    title='arg(f(z))')))

frames = []
t = np.linspace(0, 3, 45)  #6, 85
for s  in t:
    g = partial(func, s)
    w = g(z)
    argument = np.angle(w)
    frames.append(go.Frame(data=[go.Heatmap(z=argument)]))
fig.update(frames=frames);

fig.update_layout(width=500, height=475,
                  updatemenus=[dict(type='buttons',
                                y=1,
                                x=1.45,
                                active=0,
                                buttons=[dict(label='Play',
                                              method='animate',
                                              args=[None,
                                                    dict(frame=dict(duration=10,
                                                                    redraw=True),
                                                         transition=dict(duration=0),
                                                         fromcurrent=True,
                                                         mode='immediate')])])]);


I am working on the file "Animating a family-of-complex-functions.ipynb".
The first task is:
For a particular parameter,  the corresponding function of the given family is represented as heatmap of its argument, which illustrates
its zeros and poles position.
Define tickvals and ticktext for the heatmap colorbar
Can you write Python code for it?


I am working on the file "Animating-the-Dragon-curve-construction.ipynb".
The first task is:
The frame 0 displays the initial vertical segment, as the dragon cuve defined in step 0 of the iterative
process of construction.
Can you write Python code for it?

alpha = pi/10 # The rotation of 90 degrees is defined as 5 successive rotations of 18 degrees=pi10 radians
n_rot90 = 13 # we have 13 steps
frames = []

for k in range(n_rot90):
    #Record the last point on the dragon, defined in the previous step
    x0, y0 = X[-1], Y[-1]
    x = X-x0  #Translation with origin at (x0, y0)  to be the center of rotation
    y = Y-y0
    for j in range(5):
        X, Y = rotate_dragon(x, y, alpha=(j+1)*alpha)
        X = np.concatenate((x[:-1], X[::-1]), axis=None) #concatenate to the (k-1)^th step dragon  its rotated version
        Y = np.concatenate((y[:-1], Y[::-1]), axis=None)
        X = X+x0
        Y = Y+y0
        frames.append(go.Frame(data=[go.Scatter(x=X,y=Y)],
                               traces=[0]))


I am working on the file "AnimationEmbedding.ipynb".
The first task is:
Example of Embedding an Animation
The result looks something like this -- we'll use a basic animation example
taken from my earlier
Matplotlib Animation Tutorial post:
Can you write Python code for it?

from matplotlib import animation

# First set up the figure, the axis, and the plot element we want to animate
fig = plt.figure()
ax = plt.axes(xlim=(0, 2), ylim=(-2, 2))
line, = ax.plot([], [], lw=2)

# initialization function: plot the background of each frame
def init():
    line.set_data([], [])
    return line,

# animation function.  This is called sequentially
def animate(i):
    x = np.linspace(0, 2, 1000)
    y = np.sin(2 * np.pi * (x - 0.01 * i))
    line.set_data(x, y)
    return line,

# call the animator.  blit=True means only re-draw the parts that have changed.
anim = animation.FuncAnimation(fig, animate, init_func=init,
                               frames=100, interval=20, blit=True)

# call our new function to display the animation
display_animation(anim)


I am working on the file "AnimationEmbedding.ipynb".
The first task is:
Now we'll create a function that will save an animation and embed it in
an html string.  Note that this will require ffmpeg or mencoder to be
installed on your system.  For reasons entirely beyond my limited understanding
of video encoding details, this also requires using the libx264 encoding
for the resulting mp4 to be properly embedded into HTML5.
Can you write Python code for it?

from tempfile import NamedTemporaryFile

VIDEO_TAG = """<video controls>
 <source src="data:video/x-m4v;base64,{0}" type="video/mp4">
 Your browser does not support the video tag.
</video>"""

def anim_to_html(anim):
    if not hasattr(anim, '_encoded_video'):
        with NamedTemporaryFile(suffix='.mp4') as f:
            anim.save(f.name, fps=20, extra_args=['-vcodec', 'libx264'])
            video = open(f.name, "rb").read()
        anim._encoded_video = video.encode("base64")

    return VIDEO_TAG.format(anim._encoded_video)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
AdaBoost classifier
Can you write Python code for it?

from sklearn.ensemble import AdaBoostClassifier

clf_ab = AdaBoostClassifier(n_estimators=100)
clf_ab.fit(X_train, y_train)
predicted_labels = clf_ab.predict(X_test)

conf = confusion_matrix(y_test, predicted_labels)
display_cm(conf, facies_labels, display_metrics=True, hide_zeros=True)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
As noted above, the boundaries between the facies classes are not all sharp, and some of them blend into one another.  The error within these 'adjacent facies' can also be calculated.  We define an array to represent the facies adjacent to each other.  For facies label i, adjacent_facies[i] is an array of the adjacent facies labels.
Can you write Python code for it?

adjacent_facies = np.array([[1], [0,2], [1], [4], [3,5], [4,6,7], [5,7], [5,6,8], [6,7]])

def accuracy_adjacent(conf, adjacent_facies):
    nb_classes = conf.shape[0]
    total_correct = 0.
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
        for j in adjacent_facies[i]:
            total_correct += conf[i][j]
    return total_correct / sum(sum(conf))

print('Facies classification accuracy = %f' % accuracy(conf))
print('Adjacent facies classification accuracy = %f' % accuracy_adjacent(conf, adjacent_facies))


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Average test F1 score with leave one well out
Can you write Python code for it?

from sklearn.model_selection import LeaveOneGroupOut
from sklearn.metrics import f1_score, classification_report

f1_eclf = []
logo = LeaveOneGroupOut()

for train, test in logo.split(scaled_features, correct_facies_labels, groups=well_names):
    eclf.fit(scaled_features[train], correct_facies_labels[train])
    pred = eclf.predict(scaled_features[test])
    sc = f1_score(correct_facies_labels[test], pred, labels=np.arange(10), average='micro')

    well_name = well_names[test[0]]
    print("{}  {:.3f}".format(well_name, sc))
    f1_eclf.append(sc)

#     conf = confusion_matrix(correct_facies_labels[test], pred)
#     display_cm(conf, facies_labels, display_metrics=True, hide_zeros=True)
#     print("")

print("Average leave-one-well-out F1 Score: %6f" % (sum(f1_eclf)/(1.0*(len(f1_eclf)))))


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
C = 1000 and gamma = 1 seem to give the best F1 score. Let's try using these against the test dataset
Can you write Python code for it?

clf_svm = SVC(C=1000, gamma=1)
clf_svm.fit(X_train, y_train)
predicted_labels = clf_svm.predict(X_test)

conf = confusion_matrix(y_test, predicted_labels)
display_cm(conf, facies_labels, display_metrics=True, hide_zeros=True)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Conditioning the data set
Now we extract just the feature variables we need to perform the classification.  The predictor variables are the five wireline values and two geologic constraining variables. We also get a vector of the facies labels that correspond to each feature vector.
Can you write Python code for it?

correct_facies_labels = training_data['Facies'].values
well_names = training_data['Well Name']

feature_vectors = training_data.drop(['Formation', 'Well Name', 'Depth','Facies','FaciesLabels'], axis=1)
feature_vectors.describe()


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Ensemble classifier
The best classifiers so far are SVM, DecisionTree, KNN, RandomForest and DNN. Lets put all 5 together and take a majority vote to decide the final classification. Due to the way TensorFlow estimators work, they are currently incompatible with the VotingClassifier in scikit-learn, so we'll do the majority vote using the modal value calculated in a pandas Dataframe.
Update In practice, using XGBoost instead of the DNN gave slightly better F1 scores.
Can you write Python code for it?

from sklearn.ensemble import VotingClassifier

eclf = VotingClassifier(estimators=[('SVM', clf_svm),
                                    ('DecisionTree', clf_dt),
                                    ('KNN', clf_knn),
                                    ('RandomForest', clf_rf),
                                    ('XGBoost', clf_xgb)
                                   ],
                        voting='hard')

eclf.fit(X_train, y_train)
predicted_labels = eclf.predict(X_test)

conf = confusion_matrix(y_test, predicted_labels)
display_cm(conf, facies_labels, display_metrics=True, hide_zeros=True)

display_adj_cm(conf, facies_labels, adjacent_facies, display_metrics=True, hide_zeros=True)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Issue I'm not sure why the leave one well out scores are so much worse than the randomised train/test split (most likely variation between wells due to a combination of geology and well log quality). Need more work in this area
Prediction
Let's use the full dataset to train the models and make the prediction
Can you write Python code for it?

#Load testing data and standardise
test_data = pd.read_csv('../validation_data_nofacies.csv')
test_features = test_data.drop(['Well Name', 'Depth', "Formation"], axis=1)
scaled_test_features = scaler.transform(test_features)

eclf.fit(scaled_features, correct_facies_labels)
predicted_test_labels = eclf.predict(scaled_test_features)

# Save predicted labels
test_data['Facies'] = predicted_test_labels
test_data.to_csv('Anjum48_Prediction_Submission.csv')

# predicted_test_labels = fit_and_predict(scaled_features, correct_facies_labels,
#                                         scaled_test_features, classifiers)

# # Save predicted labels
# test_data['Facies'] = predicted_test_labels.mode(axis=1)[0]
# test_data.to_csv('Anjum48_Prediction_Submission.csv')

# Plot predicted labels
make_facies_log_plot(
    test_data[test_data['Well Name'] == 'STUART'],
    facies_colors=facies_colors)

make_facies_log_plot(
    test_data[test_data['Well Name'] == 'CRAWFORD'],
    facies_colors=facies_colors)
mpl.rcParams.update(inline_rc)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Let's check if the facies distributions still look right
Can you write Python code for it?

upsampled_data.describe()

facies_counts = upsampled_data['Facies'].value_counts().sort_index()
facies_counts.index = facies_labels

facies_counts.plot(kind='bar',color=facies_colors, title='Distribution of Training Data by Facies')
facies_counts


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Let's plot the GridSearchCV results in a heatmap
Can you write Python code for it?

cv_results = {"C": clf.cv_results_["param_C"], "gamma": clf.cv_results_["param_gamma"],
              "Score": clf.cv_results_['mean_test_score']}
cv_results = pd.DataFrame(cv_results)
cv_results = cv_results[cv_results.columns].astype(float)
cv_results = cv_results.pivot("C", "gamma", "Score")
plt.figure(figsize=(10, 8))
sns.heatmap(cv_results, annot=True, square=True, cmap="YlGnBu", fmt='.3g')
plt.title('F1 Score');


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Let's take a look at the data from individual wells in a more familiar log plot form. We will create plots for the five well log variables, as well as a log for facies labels. The plots are based on the those described in Alessandro Amato del Monte's excellent tutorial.
Can you write Python code for it?

def make_facies_log_plot(logs, facies_colors):
    #make sure logs are sorted by depth
    logs = logs.sort_values(by='Depth')
    cmap_facies = colors.ListedColormap(
            facies_colors[0:len(facies_colors)], 'indexed')

    ztop=logs.Depth.min(); zbot=logs.Depth.max()

    cluster=np.repeat(np.expand_dims(logs['Facies'].values,1), 100, 1)

    f, ax = plt.subplots(nrows=1, ncols=6, figsize=(8, 12))
    ax[0].plot(logs.GR, logs.Depth, '-g')
    ax[1].plot(logs.ILD_log10, logs.Depth, '-')
    ax[2].plot(logs.DeltaPHI, logs.Depth, '-', color='0.5')
    ax[3].plot(logs.PHIND, logs.Depth, '-', color='r')
    ax[4].plot(logs.PE, logs.Depth, '-', color='black')
    im=ax[5].imshow(cluster, interpolation='none', aspect='auto',
                    cmap=cmap_facies,vmin=1,vmax=9)

    divider = make_axes_locatable(ax[5])
    cax = divider.append_axes("right", size="20%", pad=0.05)
    cbar=plt.colorbar(im, cax=cax)
    cbar.set_label((17*' ').join([' SS ', 'CSiS', 'FSiS', 'SiSh', ' MS ', ' WS ', ' D  ', ' PS ', ' BS ']))
    cbar.set_ticks(range(0,1)); cbar.set_ticklabels('')

    for i in range(len(ax)-1):
        ax[i].set_ylim(ztop,zbot)
        ax[i].invert_yaxis()
        ax[i].grid()
        ax[i].locator_params(axis='x', nbins=3)

    ax[0].set_xlabel("GR")
    ax[0].set_xlim(logs.GR.min(),logs.GR.max())
    ax[1].set_xlabel("ILD_log10")
    ax[1].set_xlim(logs.ILD_log10.min(),logs.ILD_log10.max())
    ax[2].set_xlabel("DeltaPHI")
    ax[2].set_xlim(logs.DeltaPHI.min(),logs.DeltaPHI.max())
    ax[3].set_xlabel("PHIND")
    ax[3].set_xlim(logs.PHIND.min(),logs.PHIND.max())
    ax[4].set_xlabel("PE")
    ax[4].set_xlim(logs.PE.min(),logs.PE.max())
    ax[5].set_xlabel('Facies')

    ax[1].set_yticklabels([]); ax[2].set_yticklabels([]); ax[3].set_yticklabels([])
    ax[4].set_yticklabels([]); ax[5].set_yticklabels([])
    ax[5].set_xticklabels([])
    f.suptitle('Well: %s'%logs.iloc[0]['Well Name'], fontsize=14,y=0.94)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Naive Bayes classifier
Can you write Python code for it?

from sklearn.naive_bayes import GaussianNB

clf_nb = GaussianNB()
clf_nb.fit(X_train, y_train)
predicted_labels = clf_nb.predict(X_test)

conf = confusion_matrix(y_test, predicted_labels)
display_cm(conf, facies_labels, display_metrics=True, hide_zeros=True)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Nearest Neighbours classifier
This classifer has two parameters that can be tuned: the number of neighbours to use, and the method of weighting the neighbours (uniform or distance based). We'll use GridSearchCV again to find the optimal set.
Can you write Python code for it?

from sklearn import neighbors

parameters = {'n_neighbors': np.arange(1, 25), 'weights': ['uniform', 'distance']}
knn = neighbors.KNeighborsClassifier()
clf = GridSearchCV(knn, parameters, n_jobs=-1, verbose=3, scoring="f1_micro")
clf.fit(X_train, y_train)

pd.DataFrame(clf.cv_results_).sort_values(by="rank_test_score").head(10)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Now we can train the classifier using the training set we created above.
Now that the model has been trained on our data, we can use it to predict the facies of the feature vectors in the test set.  Because we know the true facies labels of the vectors in the test set, we can use the results to evaluate the accuracy of the classifier.
We need some metrics to evaluate how good our classifier is doing.  A confusion matrix is a table that can be used to describe the performance of a classification model.  Scikit-learn allows us to easily create a confusion matrix by supplying the actual and predicted facies labels.
The confusion matrix is simply a 2D array.  The entries of confusion matrix C[i][j] are equal to the number of observations predicted to have facies j, but are known to have facies i.
To simplify reading the confusion matrix, a function has been written to display the matrix along with facies labels and various error metrics.  See the file classification_utilities.py in this repo for the display_cm() function.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix, f1_score, accuracy_score
from classification_utilities import display_cm, display_adj_cm

conf = confusion_matrix(y_test, predicted_labels)
display_cm(conf, facies_labels, display_metrics=True, hide_zeros=True)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Placing the log plotting code in a function will make it easy to plot the logs from multiples wells, and can be reused later to view the results when we apply the facies classification model to other wells.  The function was written to take a list of colors and facies labels as parameters.
We then show log plots for wells SHRIMPLIN.
Can you write Python code for it?

make_facies_log_plot(training_data[training_data['Well Name'] == 'SHRIMPLIN'], facies_colors)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Random Forest classifier
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

clf_rf = RandomForestClassifier(n_estimators=10, max_depth=None, min_samples_split=2, random_state=0)
clf_rf.fit(X_train, y_train)
predicted_labels = clf_rf.predict(X_test)

conf = confusion_matrix(y_test, predicted_labels)
display_cm(conf, facies_labels, display_metrics=True, hide_zeros=True)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
SVM parameter selection
The classifier so far has been built with the default parameters.  However, we may be able to get improved classification results with optimal parameter choices.
We will consider two parameters.  The parameter C is a regularization factor, and tells the classifier how much we want to avoid misclassifying training examples.  A large value of C will try to correctly classify more examples from the training set, but if C is too large  it may 'overfit' the data and fail to generalize when classifying new data. If C is too small then the model will not be good at fitting outliers and will have a large error on the training set.
The SVM learning algorithm uses a kernel function to compute the distance between feature vectors.  Many kernel functions exist, but in this case we are using the radial basis function rbf kernel (the default).  The gamma parameter describes the size of the radial basis functions, which is how far away two vectors in the feature space need to be to be considered close.
We will train a series of classifiers with different values for C and gamma.  Two nested loops are used to train a classifier for every possible combination of values in the ranges specified.  The classification accuracy is recorded for each combination of parameter values.  The results are shown in a series of plots, so the parameter values that give the best classification accuracy on the test set can be selected.
This process is also known as 'cross validation'.  Often a separate 'cross validation' dataset will be created in addition to the training and test sets to do model selection.  For this tutorial we will just use the train set to choose model parameters within GridSearchCV.
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV

parameters = {'C': [.01, 1, 5, 10, 20, 50, 100, 1000, 5000, 10000],
              'gamma': [0.0001, 0.001, 0.01, 0.1, 1, 10],
              'kernel': ['rbf']}  # This could be extended to the linear kernel but it takes a long time

svr = SVC()
clf = GridSearchCV(svr, parameters, n_jobs=-1, verbose=3, scoring="f1_micro")
clf.fit(X_train, y_train)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Scikit also includes a handy function to randomly split the training data into training and test sets.  The test set contains a small subset of feature vectors that are not used to train the network.  Because we know the true facies labels for these examples, we can compare the results of the classifier to the actual facies and determine the accuracy of the model.  Let's use 20% of the data for the test set.
Can you write Python code for it?

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(scaled_features, correct_facies_labels, test_size=0.1, random_state=48)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
TensorFlow DNN
Can you write Python code for it?

import tensorflow as tf
from tensorflow.contrib import learn
from tensorflow.contrib import layers

feature_columns = learn.infer_real_valued_columns_from_input(X_train)
clf_dnn = learn.DNNClassifier(feature_columns=feature_columns, hidden_units=[200, 400, 200], n_classes=10)

clf_dnn.fit(X_train, y_train, steps=5000)
predicted_labels = list(clf_dnn.predict(X_test, as_iterable=True))

conf = confusion_matrix(y_test, predicted_labels)
display_cm(conf, facies_labels, display_metrics=True, hide_zeros=True)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
The rows of the confusion matrix correspond to the actual facies labels.  The columns correspond to the labels assigned by the classifier.  For example, consider the first row. For the feature vectors in the test set that actually have label SS, 23 were correctly indentified as SS, 21 were classified as CSiS and 2 were classified as FSiS.
The entries along the diagonal are the facies that have been correctly classified.  Below we define two functions that will give an overall value for how the algorithm is performing.  The accuracy is defined as the number of correct classifications divided by the total number of classifications.
Can you write Python code for it?

def accuracy(conf):
    total_correct = 0.
    nb_classes = conf.shape[0]
    for i in np.arange(0,nb_classes):
        total_correct += conf[i][i]
    acc = total_correct/sum(sum(conf))
    return acc


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
This data is from the Council Grove gas reservoir in Southwest Kansas.  The Panoma Council Grove Field is predominantly a carbonate gas reservoir encompassing 2700 square miles in Southwestern Kansas.  This dataset is from nine wells (with 4149 examples), consisting of a set of seven predictor variables and a rock facies (class) for each example vector and validation (test) data (830 examples from two wells) having the same seven predictor variables in the feature vector.  Facies are based on examination of cores from nine wells taken vertically at half-foot intervals. Predictor variables include five from wireline log measurements and two geologic constraining variables that are derived from geologic knowledge. These are essentially continuous variables sampled at a half-foot sample rate.
The seven predictor variables are:
* Five wire line log curves include gamma ray (GR), resistivity logging (ILD_log10),
photoelectric effect (PE), neutron-density porosity difference and average neutron-density porosity (DeltaPHI and PHIND). Note, some wells do not have PE.
* Two geologic constraining variables: nonmarine-marine indicator (NM_M) and relative position (RELPOS)
The nine discrete facies (classes of rocks) are:
1. Nonmarine sandstone
2. Nonmarine coarse siltstone
3. Nonmarine fine siltstone
4. Marine siltstone and shale
5. Mudstone (limestone)
6. Wackestone (limestone)
7. Dolomite
8. Packstone-grainstone (limestone)
9. Phylloid-algal bafflestone (limestone)
These facies aren't discrete, and gradually blend into one another. Some have neighboring facies that are rather close.  Mislabeling within these neighboring facies can be expected to occur.  The following table lists the facies, their abbreviated labels and their approximate neighbors.
Facies |Label| Adjacent Facies
:---: | :---: |:--:
1 |SS| 2
2 |CSiS| 1,3
3 |FSiS| 2
4 |SiSh| 5
5 |MS| 4,6
6 |WS| 5,7
7 |D| 6,8
8 |PS| 6,7,9
9 |BS| 7,8
Let's clean up this dataset.  The 'Well Name' and 'Formation' columns can be turned into a categorical data type.
Can you write Python code for it?

training_data['Well Name'] = training_data['Well Name'].astype('category')
training_data['Formation'] = training_data['Formation'].astype('category')
training_data['Well Name'].unique()

# Drop the rows with missing PEF values
training_data.dropna(inplace=True)

training_data.describe()


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
This is a quick view of the statistical distribution of the input variables. Looking at the count values, there are 3232 feature vectors in the training set.
These are the names of the 10 training wells in the Council Grove reservoir. Data has been recruited into pseudo-well 'Recruit F9' to better represent facies 9, the Phylloid-algal bafflestone.
Before we plot the well data, let's define a color map so the facies are represented by consistent color in all the plots in this tutorial. We also create the abbreviated facies labels, and add those to the facies_vectors dataframe.
Can you write Python code for it?

# 1=sandstone  2=c_siltstone   3=f_siltstone
# 4=marine_silt_shale 5=mudstone 6=wackestone 7=dolomite
# 8=packstone 9=bafflestone
facies_colors = ['#F4D03F', '#F5B041','#DC7633','#6E2C00', '#1B4F72','#2E86C1', '#AED6F1', '#A569BD', '#196F3D']
facies_labels = ['SS', 'CSiS', 'FSiS', 'SiSh', 'MS', 'WS', 'D','PS', 'BS']
#facies_color_map is a dictionary that maps facies labels
#to their respective colors
facies_color_map = {}
for ind, label in enumerate(facies_labels):
    facies_color_map[label] = facies_colors[ind]

def label_facies(row, labels):
    return labels[ row['Facies'] -1]

training_data.loc[:,'FaciesLabels'] = training_data.apply(lambda row: label_facies(row, facies_labels), axis=1)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
This shows the distribution of examples by facies for the examples in the training set.  Dolomite (facies 7) has the fewest with 81 examples.  Depending on the performance of the classifier we are going to train, we may consider getting more examples of these facies.
Crossplots are a familiar tool in the geosciences to visualize how two properties vary with rock type.  This dataset contains 5 log variables, and scatter matrix can help to quickly visualize the variation between the all the variables in the dataset.  We can employ the very useful Seaborn library to quickly create a nice looking scatter matrix. Each pane in the plot shows the relationship between two of the variables on the x and y axis, with each point is colored according to its facies.  The same colormap is used to represent the 9 facies.
Can you write Python code for it?

#save plot display settings to change back to when done plotting with seaborn
inline_rc = dict(mpl.rcParams)

import seaborn as sns
sns.set()
sns.pairplot(training_data.drop(['Well Name','Facies','Formation','Depth','NM_M','RELPOS'],axis=1),
             hue='FaciesLabels', palette=facies_color_map,
             hue_order=list(reversed(facies_labels)))

#switch back to default matplotlib plot style
mpl.rcParams.update(inline_rc)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Training the SVM classifier
Now we use the cleaned and conditioned training set to create a facies classifier.  As mentioned above, we will use a type of machine learning model known as a support vector machine.  The SVM is a map of the feature vectors as points in a multi dimensional space, mapped so that examples from different facies are divided by a clear gap that is as wide as possible.
The SVM implementation in scikit-learn takes a number of important parameters.  First we create a classifier using the default settings.
Can you write Python code for it?

from sklearn.svm import SVC

clf = SVC()
clf.fit(X_train, y_train)
predicted_labels = clf.predict(X_test)


I am working on the file "Anjum48/Anjum48 Facies Classification.ipynb".
The first task is:
Use a homemade majority voting to use with TensorFlow
Can you write Python code for it?

# classifiers = {
#     "SVM": SVC(C=1000, gamma=1),
#     "DecisionTree": tree.DecisionTreeClassifier(max_depth=32),
#     "KNN": neighbors.KNeighborsClassifier(1, weights="distance"),
# #     "DNN": learn.DNNClassifier(feature_columns=feature_columns, hidden_units=[200, 400, 200], n_classes=10),
#     "RandomForest": RandomForestClassifier(n_estimators=10, max_depth=None, min_samples_split=2, random_state=0),
#     "XGBoost": xgb.XGBClassifier()
#               }

# def fit_and_predict(X_train, y_train, X_test, classifiers):
#     predicted_values = {}
#     for key, classifier in classifiers.items():
#         if key == "DNN":
#             classifier.fit(X_train, y_train, steps=5000)
#             list(classifier.predict(X_test, as_iterable=True))
#         else:
#             classifier.fit(X_train, y_train)
#             predicted_values[key] = classifier.predict(X_test)
#     return pd.DataFrame(predicted_values)

# predicted_values = fit_and_predict(X_train, y_train, X_test, classifiers)

# majority_vote = predicted_values.mode(axis=1)[0]

# conf = confusion_matrix(y_test, majority_vote.fillna(2))
# display_cm(conf, facies_labels, display_metrics=True, hide_zeros=True)

# display_adj_cm(conf, facies_labels, adjacent_facies, display_metrics=True, hide_zeros=True)


I am working on the file "AnnGroups.ipynb".
The first task is:
Load some data
Can you write Python code for it?

from datasets import get_pbc

d = get_pbc(prints=True, norm_in=True, norm_out=False)

durcol = d.columns[0]
eventcol = d.columns[1]

if np.any(d[durcol] < 0):
    raise ValueError("Negative times encountered")

# Sort the data before training - handled by ensemble
#d.sort(d.columns[0], inplace=True)

# Example: d.iloc[:, :2] for times, events
d


I am working on the file "AnnGroups.ipynb".
The first task is:
Train the ANNs
And print groupings on training data.
Can you write Python code for it?

e.fit(d, durcol, eventcol)

# grouplabels = e.predict_classes
grouplabels, mems = e.label_data(d)
for l, m in mems.items():
    print("Group", l, "has", len(m), "members")


I am working on the file "AnnVariables.ipynb".
The first task is:
Define what the best is
The best parameter is the one that results in a longer median survival time (networks are configured to find low-risk groups).
Can you write Python code for it?

def get_winning_value(values, repeat_results, current_val):
    winner = 0, 0
    for i, x in enumerate(values):
        mres = np.median(np.array(repeat_results)[:, i, :])
        # For stability
        if mres > winner[0] or (x == current_val and mres >= winner[0]):
            winner = mres, x
    return winner[1]


I am working on the file "AnnVariables.ipynb".
The first task is:
Load data
To load a different data, use the relevant get-method as shown by the commented line.
Can you write Python code for it?

from datasets import get_nwtco, get_colon, get_lung, get_pbc, get_flchain

#d = get_colon(prints=True, norm_in=True, norm_out=False, training=True)
d = get_nwtco(prints=True, norm_in=True, norm_out=False, training=True)
d = d.astype(float)

durcol = d.columns[0]
eventcol = d.columns[1]

if np.any(d[durcol] < 0):
    raise ValueError("Negative times encountered")

print("End time:", d.iloc[:, 0].max())
#d


I am working on the file "AnnVariables.ipynb".
The first task is:
Plot group
Lower is better.
Can you write Python code for it?

def plot_score(repeat_results, models):
    boxes = []
    labels = []
    var_label = None
    # Makes no sense for low here for many datasets...
    for i, m in enumerate(models):
        labels.append(str(m.var_value))
        var_label = m.var_label
        vals = []
        for result in repeat_results:
            vals.extend(result[i])
        boxes.append(vals)

    plt.figure()
    plt.boxplot(boxes, labels=labels, vert=False, colors=plt.colors[:len(models)])
    plt.ylabel(var_label)
    plt.title("Cross-validation: n={} k={}".format(n, k))
    plt.xlabel("Median survival time (max={:.0f})".format(d[durcol].max()))
    #plt.gca().set_xscale('log')

plot_score(repeat_results, models)


I am working on the file "Annealed importance sampling.ipynb".
The first task is:
2. Define annealing distributions
Here we'll be annealing between two unnormalized Gaussian distributions with geometric mean intermediates.
They have the same variances, so they should have the same normalizing constants.
Can you write Python code for it?

num_intermediates = 25
betas = np.linspace(0,1,num_intermediates+2)
dim=1

def initial_density(x):
    return np.exp(-((x)**2).sum()/2)

def draw_from_initial():
    return npr.randn(dim)

def target_density(x):
    return np.exp(-((x-4)**2).sum()/2)

class GeometricMean():
    def __init__(self,initial,target,beta):
        self.initial = initial
        self.target = target
        self.beta = beta

    def __call__(self,x):
        f1_x = self.initial(x)
        fT_x = self.target(x)
        return f1_x**(1-self.beta) * fT_x**self.beta

annealing_distributions = [GeometricMean(initial_density,target_density,beta) for beta in betas]


I am working on the file "Annealed importance sampling.ipynb".
The first task is:
3. Define transition kernel
Here we'll just do a metropolized random walk with spherical gaussian proposals.
Can you write Python code for it?

def gaussian_random_walk(x,
                      target_f,
                      n_steps=10,
                      scale=0.5):

    x_old = x
    f_old = target_f(x_old)
    dim=len(x)

    for i in range(n_steps):

        proposal = x_old + npr.randn(dim)*scale
        f_prop = target_f(proposal)

        if (f_prop / f_old) > npr.rand():
            x_old = proposal
            f_old = f_prop

    return x_old

transition_kernels = [gaussian_random_walk]*len(annealing_distributions)


I am working on the file "Annealed importance sampling.ipynb".
The first task is:
4.1. Plot results
It should converge to one, since the initial and target distributions have the same normalizing constant.
Can you write Python code for it?

plt.plot((np.cumsum(weights)/np.arange(1,len(weights)+1)))
plt.hlines(1.0,0,len(weights))
plt.xlabel('# samples')
plt.ylabel(r'Estimated $\mathcal{Z}_T / \mathcal{Z}_1$')
plt.title(r'Estimated $\mathcal{Z}_T / \mathcal{Z}_1$')

ratios_ = ratios

mean=ratios_.mean(0)[1:]
err = ratios_.std(0)[1:]

plt.plot(mean);
plt.fill_between(range(len(mean)),mean-err,mean+err,alpha=0.4);
plt.xlabel(r'Annealing distribution index ($t$)')
plt.ylabel(r'$f_{t+1}(\mathbf{x}_{t})/f_{t}(\mathbf{x}_{t})$')
plt.title(r'Weight updates $f_{t+1}(\mathbf{x}_{t})/f_{t}(\mathbf{x}_{t})$')

end_samples = np.array([x_[-1] for x_ in xs])
plt.hist(end_samples,bins=50,normed=True);
plt.plot(x,[initial_density(x_)/2 for x_ in x])
plt.plot(x,[target_density(x_)/2 for x_ in x])
plt.title(r"$x_T$ samples")
plt.xlabel(r'$x$')
plt.ylabel(r'$p_T(x)$')


I am working on the file "Annealed importance sampling.ipynb".
The first task is:
5. A more interesting example
Let's sample a biomolecule's configuration space in this way, and maybe estimate its partition function.
For now, let's do alanine dipeptide.
5.1-5.2. Define annealing distributions and transition kernels
Can you write Python code for it?

from simtk.openmm.app import *
from simtk.openmm import *
from simtk.unit import *
from openmmtools.integrators import MetropolisMonteCarloIntegrator,HMCIntegrator

# all I want is the alanine dipeptide topology
from msmbuilder.example_datasets import AlanineDipeptide
ala = AlanineDipeptide().get().trajectories

top_md = ala[0][0].topology
topology = top_md.to_openmm()
n_atoms = top_md.n_atoms
dim = n_atoms*3

# create an openmm system
forcefield = ForceField('amber99sb.xml','amber10_obc.xml')
system = forcefield.createSystem(topology,
    nonbondedCutoff=1*nanometer, constraints=HBonds)
integrator = HMCIntegrator(300*kelvin)
simulation = Simulation(topology, system, integrator)

# create a thin wrapper class
class PeptideSystem():
    def __init__(self):
        self.simulation = simulation
        self.positions = self.simulation.context.getState(getPositions=True).getPositions().value_in_unit(nanometer)
        self.n_atoms = len(self.positions)

    def evaluate_potential_flat(self,position_vec):
        positions = position_vec.reshape(self.n_atoms,3)
        self.simulation.context.setPositions(positions)
        return self.simulation.context.getState(getEnergy=True).getPotentialEnergy()

    def propagate(self,position_vec,n_steps=1000,temp=300):
        integrator = HMCIntegrator(temp*kelvin)
        self.simulation = Simulation(topology,system,integrator)
        positions = position_vec.reshape(self.n_atoms,3)
        self.simulation.context.setPositions(positions)
        simulation.step(n_steps)

        return np.array(self.simulation.context.getState(getPositions=True).getPositions().value_in_unit(nanometer)).flatten()

    def probability_at_temp(self,position_vec,temp=300.0):
        return np.exp(-self.evaluate_potential_flat(position_vec).value_in_unit(kilojoule/mole)/temp)

peptide = PeptideSystem()
temperatures = np.logspace(3,0,1000)*300

# for some reason I can't create a bunch of parametrized anonymous functions in a list comprehension?
# i.e. [lambda...


I am working on the file "Annealed importance sampling.ipynb".
The first task is:
5.3. Run AIS
From $\mathcal{N}(\mathbf{0},\mathbf{I})$ to $\exp[-U(\x)/k_BT]$ in only a gazillion annealing distributions!
Can you write Python code for it?

%%time
xs, weights, ratios = annealed_importance_sampling(draw_from_initial,
                                 transition_kernels,
                                 annealing_distributions,
                                 n_samples=1)

weights,np.log(weights)

# expected number of hours to collect 1000 samples:
(1000*36/60)/60

%%time
xs, weights, ratios = annealed_importance_sampling(draw_from_initial,
                                 transition_kernels,
                                 annealing_distributions,
                                 n_samples=1000)

best_traj = weights.argmax()

coords = [xs[best_traj][i].reshape(n_atoms,3) for i in range(len(xs[0]))]

import mdtraj as md
annealing_traj = md.Trajectory(coords,top_md)
annealing_traj.save_pdb('annealing_traj.pdb')

plt.plot(np.log((np.cumsum(weights)/np.arange(1,len(weights)+1))))
plt.xlabel('# samples')
plt.ylabel(r'Estimated $\log ( \mathcal{Z}_T / \mathcal{Z}_1 )$')
plt.title(r'Estimated $\log ( \mathcal{Z}_T / \mathcal{Z}_1 )$')

plt.plot(np.log(weights))

ratios.mean(0)[:10]

ratios_ = ratios

mean=ratios_.mean(0)[1:]
err = ratios_.std(0)[1:]

plt.plot(mean);
plt.fill_between(range(len(mean)),mean-err,mean+err,alpha=0.4);
plt.xlabel(r'Annealing distribution index ($t$)')
plt.ylabel(r'$f_t/f_{t-1}$')
plt.title('Weight updates')
plt.savefig('weight_updates.jpg',dpi=300)
plt.close()

from IPython.display import Image
Image('weight_updates.jpg',retina=True)

# numerical underflow isn't as big a concern as I thought
np.exp(sum(np.log(ratios[0]))),weights[0]

np.savez('AIS_results_alanine_dipeptide.npz',ratios)

# what if, instead of running 30 steps of rw metropolis between each of a 1000 annealing distributions, we instead
# run 1 step of rw metropolis between each of 30,000 annealing distributions?


I am working on the file "Annotating YouTubes.ipynb".
The first task is:
Of course the SciPy docs comes with it's own documentation on how the eigen-stuff is found.
Are the above solutions and Khan's really the same?
We may show that the 2nd and 3rd solutions obey the rule:
{a * [1/2, 0, 1] + b * [1/2, 1, 0], a,b both floats}
per Khan's algebraic solution.
To show this, divide through by x in [x,y,z] to get [1.0, 3.2500000543426637, -1.2500000181142212] i.e. ratios [4.0, 13.0, -5.0]. So a=-5, b=13 in Khan's equation of the eigenspace (back to top video).  Likewise [1, 1, 1] (same ratios as Eigenvector 2) is obtained with a = b = 1.
Now say you want to prove that the original matrix, applied to any of the above eigenvectors, simply scales each one by some linear amount (the definition of an eigenvector):
Can you write Python code for it?

eigen1 = v[:,0].reshape(3, 1)
print("Scaling E1", (M_A * eigen1)/eigen1,  sep="\n")  # show the scale factor
eigen2 = v[:,1].reshape(3, 1)
print("Scaling E2", (M_A * eigen2)/eigen2,  sep="\n")  # show the scale factor
eigen3 = v[:,2].reshape(3, 1)
print("Scaling E3", (M_A * eigen3)/eigen3,  sep="\n")  # show the scale factor


I am working on the file "AnomaliesTwitterText/anomalies_in_tweets.ipynb".
The first task is:
A text polarity depends on what words appear in that text, discarding any grammar or word order but keeping multiplicity.
1) All the above text processing for features ended up with the same entries in our dataset
2) Instead of having them defined by a whole text, they are now defined by a series of counts of the most frequent words in our whole corpus.
3) These vectors are used as features to train a classifier.
Training the model
Can you write Python code for it?

from sklearn.svm import SVC
clf = SVC()
clf.fit(X=X_train,y=y_train)

wclf = SVC(class_weight={0: 20})
wclf.fit(X=X_train,y=y_train)

y_pred = clf.predict(X_test)
y_pred_weighted = wclf.predict(X_test)

from sklearn.metrics import classification_report
print ("Basic SVM metrics")
print(classification_report(y_test, y_pred))
print ("Weighted SVM metrics")
print(classification_report(y_test, y_pred_weighted))

from sklearn.metrics import confusion_matrix
print ("Basic SVM Confusion Matrix")
print (confusion_matrix(y_test, y_pred))
print ("Weighted SVM Confusion Matrix")
print (confusion_matrix(y_test, y_pred_weighted))

tn, fp, fn, tp = confusion_matrix(y_test, y_pred_weighted).ravel()
(tn, fp, fn, tp)


I am working on the file "AnomaliesTwitterText/anomalies_in_tweets.ipynb".
The first task is:
Data Preparation
Data prepration with the available data. I made the combination such that the classes are highly imbalanced making it apt for anomaly detection problem
Can you write Python code for it?

data_positive = data.loc[data["Polarity"]==1]
data_negative = data.loc[data["Polarity"]==0]
anomaly_data = pd.concat([data_negative.sample(n=10),data_positive,data_negative.sample(n=10)])
anomaly_data.Polarity.value_counts()

#Number of words per sentence
print ("No of words for sentence in train data",np.mean([len(s.split(" ")) for s in anomaly_data.TextFeed]))


I am working on the file "AnomaliesTwitterText/anomalies_in_tweets.ipynb".
The first task is:
Data pre-processing - text analytics to create a corpus
1) Converting text to matrix of token counts [Bag of words]
      Stemming -  lowercasing, removing stop-words, removing punctuation and reducing words to its lexical roots
2) Stemmer, tokenizer(removes non-letters) are created by ourselves.These are passed as parameters to CountVectorizer of sklearn.
3) Extracting important words and using them as input to the classifier

Feature Engineering
Can you write Python code for it?

import re
from sklearn.feature_extraction.text import CountVectorizer
nltk.download()
from nltk.stem.porter import PorterStemmer

''' this code is taken from
http://www.cs.duke.edu/courses/spring14/compsci290/assignments/lab02.html
'''
# a stemmer widely used
stemmer = PorterStemmer()

def stem_tokens(tokens, stemmer):
    stemmed = []
    for item in tokens:
        stemmed.append(stemmer.stem(item))
    return stemmed

def tokenize(text):
    # remove non letters
    text = re.sub("[^a-zA-Z]", " ", text)
    # tokenize
    tokens = nltk.word_tokenize(text)
    # stem
    stems = stem_tokens(tokens, stemmer)
    return stems


I am working on the file "AnomaliesTwitterText/anomalies_in_tweets.ipynb".
The first task is:
Fit_Transform:
    1) Fits the model and learns the vocabulary
    2) transoforms the data into feature vectors
Can you write Python code for it?

#using only the "Text Feed" column to build the features
features = vector_data.fit_transform(anomaly_data.TextFeed.tolist())
#converting the data into the array
features = features.toarray()
features.shape

#printing the words in the vocabulary
vocab = vector_data.get_feature_names()
print (vocab)

# Sum up the counts of each vocabulary word
dist = np.sum(corpus_data_features_nd, axis=0)

# For each, print the vocabulary word and the number of times it
# appears in the data set
a = zip(vocab,dist)
print (list(a))


I am working on the file "AnomaliesTwitterText/anomalies_in_tweets.ipynb".
The first task is:
The below implementation produces a sparse representation of the counts using scipy.sparse.csr_matrix.
Note: I am not using frequencies(TfidTransformer, apt for longer documents) because the text size is small and can be dealt with occurences(CountVectorizer).
Can you write Python code for it?

#Max_Features selected as 80 - can be changed for the better trade-off
vector_data = CountVectorizer(
    analyzer = 'word',
    tokenizer = tokenize,
    lowercase = True,
    stop_words = 'english',
    max_features = 90
)


I am working on the file "AnomaliesTwitterText/anomalies_in_tweets.ipynb".
The first task is:
Train-Test Split
Can you write Python code for it?

from sklearn.cross_validation import train_test_split
#80:20 ratio
X_train, X_test, y_train, y_test  = train_test_split(
        features,
        anomaly_data.Polarity,
        train_size=0.80,
        random_state=1234)

print ("Training data - positive and negative values")
print (pd.value_counts(pd.Series(y_train)))
print ("Testing data - positive and negative values")
print (pd.value_counts(pd.Series(y_test)))


I am working on the file "AntSearch for CHPswarm.ipynb".
The first task is:
Ant Colony Optimization (ACO)
For a general overview, and as a general reference for this section, see Dorigo and Stülze, 2004.
The Ant Colony Optimization algorithm mimmicks the food gathering behavior of an ant colony - a prominent example of swarm intelligence. Indeed, individually, ants are neither very smart, nor do they have a capacity to communicate complex information such as the shortest path to a food resource. Still, collectively, the logistic operations of an ant hill are highly structured and very efficient (in terms of distance travelled).
In the common model thinking, ants achieve this through swarm intelligence: initially, individual ants wander the surroundings of their colony more or less aimlessly (they forage). As soon as an ant encounters food, it gathers as much as it can carry and attempts to find its way back to the colony. In the following, we are going to assume that the ant remembers its path, and that it follows it back without deviation. On its way back, the ant secretes a special pheromon, leaving a chemical trail to the food resource. The pheromon entices another foraging ants to follow the same path - although they may deviate from it. Probabilistically, the liklihood of a foraging ant heading a certain direction increases with the pheromone concentration in that particular direction; but the pheromon evaporates over time. Given a large number of ants and enough time, this mechanism stabilizes along the most effective paths.
The above describes the classical ACO algorithm used to find the shortest path in a weighted graph.
Here, we try to essentially solve the Knapsack problem though, so the behavior of the "ants" has to be somewhat modified.
From a problem-oriented perspective, the ants become swarm configurators: starting from an initial "swarm of one", each ant sequentially includes one more buildings from the list, until no other building could be added without violating the biogas constraint (total swarm consumption exceeding $E_{tot}$). The result is a viable candidate...
Can you write Python code for it?

class Ant(object):

    def __init__(self, trail, initial_position=None):
        self.path = []
        if initial_position is not None:
            self.path.append(int(initial_position))
        self.trail = trail

    def worth(self):
        return CHPswarm_cost_balance(self.path)

    def chose_next_site(self):

        # viable next candidates
        E_fl_swarm = CHPswarm_fuel_demand(self.path)
        candidates = np.flatnonzero(E_fl_swarm + E_fl < E_tot)

        # eliminate already selected paths
        candidates = np.setdiff1d(candidates, self.path, assume_unique=True)

        # got a complete swarm if no more sites can be added
        if candidates.size==0:
            return

        # heuristic value --> just revenues:
        #   since operation times are identical for all, the largest plant always gets to produce most
        #   so there is no need to account for efficiency, e.g. by taking the fuel costs into consideration
        revenues = CHPswarm_production_revenues(candidates)

        # transition probabilities
        p = revenues * self.trail[candidates]
        p /= np.sum(p)

        return np.random.choice(candidates, size=1, p=p)

    def forage(self):
        next_site = self.chose_next_site()

        # nowhere to go = found a solution, give control back to ant system
        if next_site is None:
            raise FoundViableSolution(self)

        # continue foraging
        self.path.append(int(next_site))

class FoundViableSolution(Exception):
    def __init__(self, ant):
        self.path = frozenset(ant.path)
        self.score = ant.worth()


I am working on the file "AntSearch for CHPswarm.ipynb".
The first task is:
Ant Colony Optimization for Energy Systems Research
This notebook explores using Ant Colony Optimization to solve the combinatorial optimization problem of optimally configuring a branch of an energy system.
NOTICE
The implementation in this document is flawed. First, my 'Ant Colony Optimization' optimization does in no way correspond to the state of the art. This combines badly with an error in the heuristic I used: since I did not include investment costs in the model, and since heat is significantly more valuable than electricity, large CHPPs are actually not economically beneficial (as the heuristic presumes). There is thus a much more profitable swarm that can be found using general purpose solvers (as we discovered when using Particle Swarm Optimization in Pol Duhr's BT).
To address this, towards the end an IP approach using Gurobi was added as an illustration. The performance of branch and bound seems much better than that of ant-searc - of course one would have to redo the whole calculation and adjust the implementation of the 'Ant' class to get a fair comparison.
I may adjust all of this at a later time - for now, this current version has to do, even though it is somewhat messy.
Problem setting
This notebook illustrates the procedure on a simplified version of the decentralized energy provision problem the Energy Systems Group solves in the CHPswarm project: a list of buildings and their annual thermal energy demand (for space heating and warm water supply) is given. Each building is a potential installation site for a CHP plant - per our modeling paradigm, plants are designed so as to be completely (at all times and without the need for auxiliary heaters) satisfy the local heat demand. However, the number of production sites is limited: to ensure sustainability, individual plants may only burn biogas; the supply of which is finite.
Now the challenge is to build a swarm, i.e. select a subset of production sites that, collectively, optimally utilize the biogas potential at minimal costs. These...
Can you write Python code for it?

a, b, P_el = sympy.symbols("a b P_el ")
eta_el = a * sympy.log(P_el) + b

t_op, E_th, eta_th = sympy.symbols("t_op E_th eta_th")
P_th = P_el / eta_el * (1 - eta_el) * eta_th

sols = sympy.solve(sympy.Eq(P_th, E_th/t_op), sympy.log(P_el))
assert len(sols) == 1
sols[0]


I am working on the file "AntSearch for CHPswarm.ipynb".
The first task is:
If we try to build a swarm manually, starting with the biggest machines:
The fuel utilization seems to be very good:
Can you write Python code for it?

print "fuel utilization: {:.2f} of {:.2f} kWh".format(CHPswarm_fuel_demand(list(swarm)) / 3.6e6, E_tot/3.6e6)