I am working on the file "02_numpy_md/recall_last_session.ipynb".
The first task is:
As you can see there is a difference between the numerical and analytical solution.
If you decrease the time step by a factor of 10 to:
python
time_step = 0.004
and rerun the numerical solution and then analytical solution you will see that for the given height the curves look the same
but we still have a error between them which increases with increasing time / height
Numerical solution second order
A more accurate solution can be provided by using a numerical integrator of higher order.
Taylor series
A Taylor series is a representation of a function as an infinite sum of terms that are calculated from the values of the function's derivatives at a single point.
<div style="background:lightblue">
Note: see math script of Luik for more
</div>

One can use Taylor series to numericaly approximate functions by using a finite number of terms.
Here, we give as an example a 1-d Taylor series which was terminated after the third order term:
$
\begin{align}
x(t + \Delta t) &= x(t) + \dot{x}(t) \Delta t + \frac{1}{2} \ddot{x}(t) (\Delta t)^2 + \frac{1}{3!} \dddot{x}(t) (\Delta t)^3 + \mathcal{O}(\Delta t ^ 4) \
\end{align}
$
If we now take the second order form and use it for our example we get:
$
\begin{align}
x(t + \Delta t) &= x(t) + \dot{x}(t) \Delta t + \frac{1}{2} \ddot{x}(t) (\Delta t)^2 + \mathcal{O}(\Delta t ^ 3) \
x(t + \Delta t) &= x(t) + v(t) \Delta t + \frac{F}{2 m} \Delta t^2 \
\end{align}
$
Since we have a constant force in both direction we get for the x-direction:
$
\begin{align}
x(t + \Delta t) &= x(t) + v_x(t) \Delta t + \frac{F_x}{2 m} \Delta t^2 \
x(t + \Delta t) &= x(t) + v_x(t) \Delta t &\textrm{with: }F_x=0 \
\end{align}
$
and in y-direction:
$
\begin{align}
y(t + \Delta t) &= y(t) + v_y(t) \Delta t + \frac{F_y}{2 m} \Delta t^2 \
y(t + \Delta t) &= y(t) + v_y(t) \Delta t - \frac{g}{2} \Delta t^2
&\textrm{with: }F_y=-g m \
\end{align}
$
This equations are implemented in the code below.
Which will now update the positions with a second order integrator
python
self.position[0] +=...
Can you write Python code for it?

class Ball:
    def __init__(self, start_position, start_velocity, radius):
        self.position = start_position
        self.velocity = start_velocity
        self.radius = radius

    def move(self, time_step):
        self.position[0] += self.velocity[0] * time_step
        self.position[1] += self.velocity[1] * time_step - 9.81 / 2.0 * time_step**2

        self.velocity[0] += 0
        self.velocity[1] -= 9.81 * time_step
        if self.position[1] <= 0.0:
            self.velocity[1] *= -1

tennis_ball = Ball([0, 1], [0.2, 0], 0.033)
time_step = 0.04
number_of_steps = 1500

trajectory_numerical_2nd = []  # this is an empty list
trajectory_time = [] # also a empty list to store the time
for step in range(number_of_steps):
    trajectory_numerical_2nd.append(tennis_ball.position[:])  # append adds something at the end of a list
    trajectory_time.append(step*time_step)
    if tennis_ball.position[1] <= 0.0: break # stop the calculation here
    tennis_ball.move(time_step)

# don't worry about the code below too much it just crates an animation
# of your calculated positions.
from animation import animate
%matplotlib notebook

#ani = animate(trajectory_numerical, time_step*100/0.04)

trajectory_analytical = [] # an empty list to store the analytic results
start_position=[0, 1]
start_velocity=[0.2, 0]
for point_in_time in trajectory_time:
    position_x = start_position[0] + start_velocity[0] * point_in_time
    position_y = start_position[1] - 0.5 * 9.81 * point_in_time**2
    trajectory_analytical.append([position_x,position_y])

error = []
error_x_sq=error_y_sq=0
for i in range(len(trajectory_analytical)):
    diff_x=trajectory_analytical[i][0]-trajectory_numerical_2nd[i][0]
    diff_y=trajectory_analytical[i][1]-trajectory_numerical_2nd[i][1]
    error_x_sq+=diff_x**2
    error_y_sq+=diff_y**2
    error.append([error_x_sq,error_y_sq])

from animation import plot
%matplotlib inline

plot(trajectory_time, trajectory_numerical_2nd,trajectory_analytical,error)


I am working on the file "02_numpy_md/recall_last_session_solution.ipynb".
The first task is:
Analytical solution
Initialisation

We have first to define the starting position and velocity of our ball as in the numerical example.

Analytical solution

Then we can iterate over the saved points in time and calculate the analytical solution for this point and add it to the trajectory.

 Here you have to fill in the equations.

Error calculation

To be able to better compare the numeric and analytic solution we calculate the squared error between them and then plot it.

Can you write Python code for it?

# Initialisation
start_position=[0, 1] # start position of the ball (as in the numeric example)
start_velocity=[0.2, 0] # start velocity of the ball (as in the numeric example)

# Analytical solution
trajectory_analytical = [] # an empty list to store the analytic results
for point_in_time in trajectory_time: # iterate of the timepoints
    position_x = start_position[0] + start_velocity[0] * point_in_time # fill in here
    position_y = start_position[1] - 0.5 * 9.81 * point_in_time**2 # fill in here
    trajectory_analytical.append([position_x,position_y]) # update trajectory

# Error calculation
error = []
error_x_sq=error_y_sq=0
for i in range(len(trajectory_analytical)):
    diff_x=trajectory_analytical[i][0]-trajectory_numerical[i][0]
    diff_y=trajectory_analytical[i][1]-trajectory_numerical[i][1]
    error_x_sq+=diff_x**2
    error_y_sq+=diff_y**2
    error.append([error_x_sq,error_y_sq])

# Plotting
from animation import plot
%matplotlib inline

plot(trajectory_time, trajectory_numerical,trajectory_analytical,error)


I am working on the file "02_numpy_md/recall_last_session_solution.ipynb".
The first task is:
As you can see there is a difference between the numerical and analytical solution.
If you decrease the time step by a factor of 10 to:
python
time_step = 0.004
and rerun the numerical solution and then analytical solution you will see that for the given height the curves look the same
but we still have a error between them which increases with increasing time / height
Numerical solution second order
A more accurate solution can be provided by using a numerical integrator of higher order.
Taylor series
A Taylor series is a representation of a function as an infinite sum of terms that are calculated from the values of the function's derivatives at a single point.
<div style="background:lightblue">
Note: see math script of Luik for more
</div>

One can use Taylor series to numericaly approximate functions by using a finite number of terms.
Here, we give as an example a 1-d Taylor series which was terminated after the third order term:
$
\begin{align}
x(t + \Delta t) &= x(t) + \dot{x}(t) \Delta t + \frac{1}{2} \ddot{x}(t) (\Delta t)^2 + \frac{1}{3!} \dddot{x}(t) (\Delta t)^3 + \mathcal{O}(\Delta t ^ 4) \
\end{align}
$
If we now take the second order form and use it for our example we get:
$
\begin{align}
x(t + \Delta t) &= x(t)
+\dot{x}(t) \Delta t + \frac{1}{2} \ddot{x}(t) (\Delta t)^2 + \mathcal{O}(\Delta t ^ 3) \
x(t + \Delta t) &= x(t) + v(t) \Delta t + \frac{F}{2 m} \Delta t^2 \
\end{align}
$
Since we have a constant force in both direction we get for the x-direction:
$
\begin{align}
x(t + \Delta t) &= x(t) + v_x(t) \Delta t + \frac{F_x}{2 m} \Delta t^2 \
x(t + \Delta t) &= x(t) + v_x(t) \Delta t &\textrm{with: }F_x=0 \
\end{align}
$
and in y-direction:
$
\begin{align}
y(t + \Delta t) &= y(t) + v_y(t) \Delta t + \frac{F_y}{2 m} \Delta t^2 \
y(t + \Delta t) &= y(t) + v_y(t) \Delta t - \frac{g}{2} \Delta t^2
&\textrm{with: }F_y=-g m \
\end{align}
$
This equations are implemented in the code below.
Which will now update the positions with a second order integrator
python
self.position[0] +=...
Can you write Python code for it?


I am working on the file "02_pilot/Radical_Pilot.ipynb".
The first task is:
/bin/sleep assigns a random sleep time. We plot the distribution of runtimes of the above 20 ComputeUnits using Seaborn. See distplot documentation.
Can you write Python code for it?


I am working on the file "02_pilot/Radical_Pilot.ipynb".
The first task is:
2. Local Pilot Example
This example shows how to execute a task using a Pilot-Job running on the local machine. In this case, the Pilot-Job is started using ssh on the edge node machine of the Hadoop cluster (which runs Jupyterhub - the iPython notebook server).
2.1 Create a new Session and Pilot-Manager.
Can you write Python code for it?

session = rp.Session()
pmgr = rp.PilotManager(session=session)
umgr = rp.UnitManager (session=session,
                       scheduler=rp.SCHED_ROUND_ROBIN)
print "Session id: %s Pilot Manager: %s" % (session.uid, str(pmgr.as_dict()))

print_details(umgr.as_dict())


I am working on the file "02_pilot/Radical_Pilot.ipynb".
The first task is:
2.2 Submit Pilot and add to Unit Manager
Can you write Python code for it?

pdesc = rp.ComputePilotDescription()
pdesc.resource = "local.localhost_anaconda"  # NOTE: This is a "label", not a hostname
pdesc.runtime  = 10 # minutes
pdesc.cores    = 2
pdesc.cleanup  = False
pilot = pmgr.submit_pilots(pdesc)
umgr.add_pilots(pilot)

print_details(pilot.as_dict())


I am working on the file "02_pilot/Radical_Pilot.ipynb".
The first task is:
2.4 Exercise
Write a task (i.e., a ComputeUnit) that prints out the hostname of the machine!
Answer: In the example above, in cudesc.executable replace /bin/echo with hostname.
2.5 Performance Analysis
In the examples below we will show how RADICAL-Pilot can be used for interactive analytics. We will plot and analyze the execution times of a set of ComputeUnits.
Can you write Python code for it?

def get_runtime(compute_unit):
    details=compute_unit.as_dict()
    execution_details=details['execution_details']
    state_details=execution_details["statehistory"]
    results = {}
    for i in state_details:
        results[i["state"]]=i["timestamp"]
    start = results["Scheduling"]
    end = results["Done"]
    runtime = end-start
    return runtime

import random
cudesc_list = []
for i in range(20):
    cudesc = rp.ComputeUnitDescription()
    cudesc.executable  = "/bin/sleep"
    cudesc.environment = {'CU_NO': i}
    cudesc.arguments   = ['%d'%(random.randrange(10))]
    cudesc.cores       = 1
    cudesc_list.append(cudesc)

cu_set = umgr.submit_units(cudesc_list)

states = umgr.wait_units()

runtimes=[]
for compute_unit in cu_set:
    str(compute_unit)
    runtimes.append(get_runtime(compute_unit))


I am working on the file "02_pilot/Radical_Pilot.ipynb".
The first task is:
2.6 Close and Delete Session
Can you write Python code for it?


I am working on the file "02_pilot/Radical_Pilot.ipynb".
The first task is:
3. YARN Pilot Example
Having submitted multiple jobs using RADICAL-Pilot locally, in this section we will examine how to submit multiple tasks to a YARN cluster using RADICAL-Pilot. Although not of primary importance for this tutorial, it is worth noting that the YARN cluster is a remote resource compared to the submission host (edge node).

3.1 Create a new Session and Pilot-Manager.
Can you write Python code for it?

import getpass
yarn_session = rp.Session()
c = rp.Context('ssh')
c.user_id = getpass.getuser()
yarn_session.add_context(c)
pmgr = rp.PilotManager(session=yarn_session)
umgr = rp.UnitManager(session=yarn_session,
                      scheduler=rp.SCHED_ROUND_ROBIN)
print "Session id: %s Pilot Manager: %s" % (yarn_session.uid, str(pmgr.as_dict()))


I am working on the file "02_pilot/Radical_Pilot.ipynb".
The first task is:
3.3 Submit Compute Units
Create a description of the compute unit, which specifies the details on the executed task.
Can you write Python code for it?

cudesc = rp.ComputeUnitDescription()
cudesc.environment = {'CU_NO': "1"}
cudesc.executable  = "/bin/echo"
cudesc.arguments   = ['I am CU number $CU_NO']
cudesc.cores       = 1
print_details(cudesc.as_dict())


I am working on the file "02_pilot/Radical_Pilot.ipynb".
The first task is:
And some more details...
Can you write Python code for it?


I am working on the file "02_pilot/Radical_Pilot.ipynb".
The first task is:
Parse the output of the CU
Can you write Python code for it?


I am working on the file "02_pilot/Radical_Pilot.ipynb".
The first task is:
Submit the previously created ComputeUnit descriptions to the PilotManager. This will trigger the selected scheduler (in this case the round-robin scheduler) to start assigning ComputeUnits to the ComputePilots.
Can you write Python code for it?

print "Submit Compute Units to Unit Manager ..."
cu_set = umgr.submit_units([cudesc])
print "Waiting for CUs to complete ..."
umgr.wait_units()
print "All CUs completed successfully!"
cu_results = cu_set[0]
details=cu_results.as_dict()


I am working on the file "02_pilot/Radical_Pilot.ipynb".
The first task is:
Submit the previously created ComputeUnit descriptions to the PilotManager. This will trigger the selected scheduler to start assigning ComputeUnits to the ComputePilots.
Can you write Python code for it?

print "Submit Compute Units to Unit Manager ..."
cu_set = umgr.submit_units([cudesc])
print "Waiting for CUs to complete ..."
umgr.wait_units()
print "All CUs completed successfully!"
cu_results = cu_set[0]
details=cu_results.as_dict()

print_details(details)

print_details(details["execution_details"])

print cu_results.stdout.strip()


I am working on the file "02_pilot/Radical_Pilot.ipynb".
The first task is:
The next command will provide the state of the Pilot and other pilot details.
Can you write Python code for it?


I am working on the file "02_pilot/Radical_Pilot_YARN_Stampede.ipynb".
The first task is:
4. Create Pilot and Unit Managers
Can you write Python code for it?

print "Initializing Pilot Manager ..."
pmgr = rp.PilotManager(session=session)
print "Initializing Unit Manager ..."
umgr = rp.UnitManager (session=session,
                           scheduler=rp.SCHED_DIRECT_SUBMISSION)


I am working on the file "02_pilot/Radical_Pilot_YARN_Stampede.ipynb".
The first task is:
5. Submit the pilot to the Pilot and Unit Managers
Can you write Python code for it?

pdesc = rp.ComputePilotDescription ()
pdesc.resource = "yarn.stampede"  # NOTE: This is a "label", not a hostname
pdesc.runtime  = 60 # minutes
pdesc.cores    = 16
pdesc.cleanup  = False
pdesc.project  = '' #Include the Allocation here
pdesc.queue    = 'development' #You can select a different queue if you want.
# submit the pilot.
print "Submitting Compute Pilot to Pilot Manager ..."
pilot = pmgr.submit_pilots(pdesc)
print "Registering Compute Pilot with Unit Manager ..."
umgr.add_pilots(pilot)


I am working on the file "02_pilot/Radical_Pilot_YARN_Stampede.ipynb".
The first task is:
6. Submit Compute Units
First create the description of the compute units which define the task to be executed
Can you write Python code for it?

NUMBER_JOBS  = 16
cudesc_list = []
for i in range(NUMBER_JOBS):
    cudesc = rp.ComputeUnitDescription()
    cudesc.environment = {'CU_NO': i}
    cudesc.executable  = "/bin/echo"
    cudesc.arguments   = ['I am CU number $CU_NO']
    cudesc.cores       = 1
    cudesc_list.append(cudesc)


I am working on the file "02_pilot/Radical_Pilot_YARN_Stampede.ipynb".
The first task is:
7. Always clean up the session
Can you write Python code for it?


I am working on the file "02_pilot/Radical_Pilot_YARN_Stampede.ipynb".
The first task is:
Submit the created Compute Units to the Unit Manager.
Can you write Python code for it?

print "Submit Compute Units to Unit Manager ..."
cu_set = umgr.submit_units (cudesc_list)
print "Waiting for CUs to complete ..."
umgr.wait_units()
print "All CUs completed successfully!"


I am working on the file "02_preprocess/05_preprocess_expectation_df(mean, std, skewness).ipynb".
The first task is:
$ x \cdot p(x)$
Can you write Python code for it?

col

# X*p(X)
Xprob_df = prob_df.mul(col)
Xprob_df.head()


I am working on the file "02_preprocess/05_preprocess_expectation_df(mean, std, skewness).ipynb".
The first task is:
$p(X)$
Can you write Python code for it?

1312 / 58122

# p(X)
prob_df = df.ix[:,:'5'].apply(lambda x: x / df['sum'])
prob_df.head()


I am working on the file "02_preprocess/05_preprocess_expectation_df(mean, std, skewness).ipynb".
The first task is:
expect_df
Can you write Python code for it?

expect_df = pd.DataFrame({'mean':mean,
             'std':std,})
expect_df['skew'] = skew
expect_df.head()

expect_df.to_csv('../resource/preprocess_expectation_df.csv', index=False)


I am working on the file "02_preprocess/05_preprocess_expectation_df(mean, std, skewness).ipynb".
The first task is:
mean : $\mu = E[X] = \sum x \cdot p(x)$
Can you write Python code for it?

mean = Xprob_df.sum(axis=1)
mean.head()

prob_df['mean'] = Xprob_df.sum(axis=1)
prob_df.head()


I am working on the file "02_preprocess/05_preprocess_expectation_df(mean, std, skewness).ipynb".
The first task is:
skewness : $\frac{\mu_3}{\sigma^3} = \operatorname{E}\left[\left(\frac{X-\mu}{\sigma}\right)^3 \right]$
Can you write Python code for it?

# (X - u)^3
sub3_df = sub_df.applymap(lambda x: x*x*x)
sub3_df.head()

# theta^3
std3 = std.map(lambda x: x**3)
std3.head()

#
skew_df = sub3_df.apply(lambda x: x / std3)
skew_df.head()

skew = skew_df.mul(prob_df.ix[:,:'5']).sum(axis=1)
skew.head()


I am working on the file "02_preprocess/05_preprocess_expectation_df(mean, std, skewness).ipynb".
The first task is:
별점분포의 X_count
Can you write Python code for it?


I am working on the file "02_preprocessing/.ipynb_checkpoints/exploration05-price_history_clustering-checkpoint.ipynb".
The first task is:
6 htan 0.23
7 htan 0.24
Can you write Python code for it?

n_clusters = 6
print n_clusters
silhouette, inertia = calc_n_clusters(n_clusters = n_clusters)
print silhouette, inertia
silhouette_scores[n_clusters] = silhouette

n_clusters = 7
print n_clusters
silhouette, inertia = calc_n_clusters(n_clusters = n_clusters)
print silhouette, inertia
silhouette_scores[n_clusters] = silhouette

n_clusters = 8
print n_clusters
silhouette, inertia = calc_n_clusters(n_clusters = n_clusters)
print silhouette, inertia
silhouette_scores[n_clusters] = silhouette


I am working on the file "02_preprocessing/.ipynb_checkpoints/exploration05-price_history_clustering-checkpoint.ipynb".
The first task is:
Dynamic Time Wrapping
Dynamic time warping finds the optimal non-linear alignment between two time series.
K means clustering
https://stackoverflow.com/questions/5529625/is-it-possible-to-specify-your-own-distance-function-using-scikit-learn-k-means
Can you write Python code for it?

mms = MinMaxScaler()

seqs_norm = mms.fit_transform(seqs_fixed_width.T).T
seqs_norm.shape

#set random centroids or pseudorandom
#calculate the distance of all points with centroid 1, then the same for centroid 2
#then find a new centroid according to the points you had considered
#then repeat

#kmeans = KMeans(n_clusters=n_clusters, random_state=random_state)

# centroids = random.sample(seqs, 3)
# len(centroids), len(centroids[0]), len(centroids[1]), len(centroids[2])

# aa , _ = fastdtw(seqs[0], seqs[2], dist=euclidean)
# aa

def k_means_clust(data, n_clusters, tol=0.1, n_max_iter=100): #tol=0.0001
    centroids = random.sample(data, n_clusters)
    prev_centroids = centroids[:]
    best_max_diff = float('inf')

    data_len = len(data)

    labels = np.empty(data_len)

    counter = 0
    while counter < n_max_iter:
        counter += 1
        print counter

        #we only care about the last inertia but it is minimum cost to calc it on every iter
        cur_inertia = 0

        assignments = {}  #dictionary cluster -> data points

        for data_ind, xx in enumerate(data):
            min_dist = float('inf')
            closest_clust_ind = None

            for c_ind, cc in enumerate(centroids):
                cur_dist, _ = fastdtw(xx, cc, dist=euclidean)
                if cur_dist < min_dist: #this means that current centroid is better
                    closest_clust_ind = c_ind
                    min_dist = cur_dist

            labels[data_ind] = closest_clust_ind

            cur_inertia += min_dist

            if closest_clust_ind in assignments:
                assignments[closest_clust_ind].append(data_ind)
            else:
                assignments[closest_clust_ind] = [data_ind]

        #recalculate centroids of clusters, here we do it in a euclidean way
        #(we do not have reverse dtw)
        for key in assignments:
            cur_assignment_list = assignments[key]
        ...


I am working on the file "02_preprocessing/.ipynb_checkpoints/exploration05-price_history_clustering-checkpoint.ipynb".
The first task is:
Some processing
Can you write Python code for it?

csv_in = "../price_history_03_seq_start_suddens_trimmed.csv"

df_fixed_width = PreprocessPriceHistory.keepOnlyPriceHistoriesWithMostFrequentLength(csv_in=csv_in)

csv_path = "../price_history_03a_fixed_width.csv"
#df_fixed_width.to_csv(csv_path, encoding='utf-8', quoting=csv.QUOTE_ALL)


I am working on the file "02_preprocessing/.ipynb_checkpoints/exploration05-price_history_clustering-checkpoint.ipynb".
The first task is:
http://nbviewer.jupyter.org/github/alexminnaar/time-series-classification-and-clustering/blob/master/Time%20Series%20Classification%20and%20Clustering.ipynb
Can you write Python code for it?

from __future__ import division
import numpy as np
import pandas as pd
import sys
import math
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
import re
import os
import csv
from helpers.outliers import MyOutliers
from skroutz_mobile import SkroutzMobile
from sklearn.ensemble import IsolationForest
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, r2_score
from skroutz_mobile import SkroutzMobile
from sklearn.model_selection import StratifiedShuffleSplit
from helpers.my_train_test_split import MySplitTrainTest
from sklearn.preprocessing import StandardScaler
from preprocess_price_history import PreprocessPriceHistory
from price_history import PriceHistory
from dfa import dfa
import scipy.signal as ss
import nolds
from scipy.spatial.distance import euclidean
from fastdtw import fastdtw
from sklearn.cluster import KMeans
from sklearn.preprocessing import MinMaxScaler
import random
from sklearn.metrics import silhouette_score
from os.path import isfile
from preprocess_price_history import PreprocessPriceHistory

random_state = np.random.RandomState(seed=16011984)
%matplotlib inline


I am working on the file "02_preprocessing/.ipynb_checkpoints/exploration05-price_history_clustering-checkpoint.ipynb".
The first task is:
testing
Can you write Python code for it?

km = KMeans(init='random', #init='k-means++',
            n_clusters=3, n_init=10)
km

km.fit(df_fixed_width.values)

km.inertia_

km.labels_


I am working on the file "02_preprocessing/.ipynb_checkpoints/exploration09-price_history_gaussian_process_regressor_clustered_data-checkpoint.ipynb".
The first task is:
Gaussian Process Regressor
Can you write Python code for it?

XX = arr[:, :MobAttrsPriceHistoryMerger.PRICE_IND]
XX.shape

yy = arr[:, MobAttrsPriceHistoryMerger.PRICE_IND]
yy.shape

%%time
gp = GaussianProcessRegressor()
gp.fit(XX, yy)


I am working on the file "02_preprocessing/.ipynb_checkpoints/exploration09-price_history_gaussian_process_regressor_clustered_data-checkpoint.ipynb".
The first task is:
Reconstruct time series
we want to get the price values that correspond to a particular sku
but have we reserved this info ?? no..
Can you write Python code for it?

# with open('cur_gp.pickle', 'w') as fp:  # Python 3: open(..., 'wb')
#     pickle.dump(gp, fp)

cur_sku_id = list(cluster_sku_ids)[0]
cur_sku_id

vals = dataframe.loc[cur_sku_id].values

xx = vals[:, :-1]
xx.shape

tars = vals[:, -1]
tars.shape

preds = gp.predict(xx)
preds.shape

plt.figure()
plt.plot(tars, 'r-', label='targets')
plt.plot(preds, 'b.', label='predictions')
plt.legend()
plt.show()


I am working on the file "02_preprocessing/.ipynb_checkpoints/exploration09-price_history_gaussian_process_regressor_clustered_data-checkpoint.ipynb".
The first task is:
Train - Test
Can you write Python code for it?

ph = PriceHistory(csv_in)

seqs = ph.extractAllSequences()

selseq = [seq for seq in seqs if seq.name == cur_sku_id]
assert len(selseq) == 1
selseq = selseq[0]

selseq.index


I am working on the file "02_preprocessing/.ipynb_checkpoints/exploration09-price_history_gaussian_process_regressor_clustered_data-checkpoint.ipynb".
The first task is:
http://nbviewer.jupyter.org/github/alexminnaar/time-series-classification-and-clustering/blob/master/Time%20Series%20Classification%20and%20Clustering.ipynb
Can you write Python code for it?

from __future__ import division
import numpy as np
import pandas as pd
import sys
import math
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
import re
import os
import csv
from helpers.outliers import MyOutliers
from skroutz_mobile import SkroutzMobile
from sklearn.ensemble import IsolationForest
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, r2_score
from skroutz_mobile import SkroutzMobile
from sklearn.model_selection import StratifiedShuffleSplit
from helpers.my_train_test_split import MySplitTrainTest
from sklearn.preprocessing import StandardScaler
from preprocess_price_history import PreprocessPriceHistory
from price_history import PriceHistory
from dfa import dfa
import scipy.signal as ss
from scipy.spatial.distance import euclidean
from fastdtw import fastdtw
from sklearn.cluster import KMeans
from sklearn.preprocessing import MinMaxScaler
import random
from sklearn.metrics import silhouette_score
from os.path import isfile
from preprocess_price_history import PreprocessPriceHistory
from os.path import isfile
from sklearn.gaussian_process import GaussianProcessRegressor
from mobattrs_price_history_merger import MobAttrsPriceHistoryMerger
#from george import kernels
#import george
from sklearn.manifold import TSNE
import matplotlib  as mpl
import pickle
import dill

random_state = np.random.RandomState(seed=16011984)
%matplotlib inline

mpl.rc('figure', figsize=(17,7)) #setting the default value of figsize for our plots
#https://matplotlib.org/users/customizing.html

data_path = '../../../../Dropbox/data'
mobattrs_ph_path = data_path + '/mobattrs_price_history'
mobattrs_ph_norm_path = mobattrs_ph_path + '/mobattrs_ph_norm.npy'
sku_ids_groups_path = data_path...


I am working on the file "02_preprocessing/.ipynb_checkpoints/exploration09-price_history_gaussian_process_regressor_clustered_data-checkpoint.ipynb".
The first task is:
merging
Can you write Python code for it?

obj = MobAttrsPriceHistoryMerger(mobs_norm_path=mobs_norm_path, price_history_csv=csv_in)

%%time
dataframe = obj.get_table(df = df_cluster.drop(labels=SkroutzMobile.PRICE_COLS, axis=1), normalize_dates=True,
                         normalize_price=True)

dataframe

arr = dataframe.values

arr.shape

#mobattrs_ph_raw_path = mobattrs_ph_path + '/mobattrs_ph_raw.npy'
#np.save(mobattrs_ph_raw_path, arr)
#assert isfile(mobattrs_ph_raw_path)

np.all(np.logical_not(np.isnan(arr.flatten())))

# we are not saving
# np.save(mobattrs_ph_norm_path, arr_norm)
# assert isfile(mobattrs_ph_norm_path)


I am working on the file "02_preprocessing/exploration00_features_vs_price_for_regression_potential.ipynb".
The first task is:
Analysis
Can you write Python code for it?

df.columns

cols = list(set(df.columns).difference(SkroutzMobile.BINARY_COLS + SkroutzMobile.PRICE_COLS))
len(cols)

cols

ppc = 0

#fig = plt.figure(figsize=(15,7))
sns.pairplot(data=df, x_vars=[cols[ppc*3], cols[ppc*3+1], cols[ppc*3+2]],
             y_vars='price_min', size=4)
plt.show()
ppc += 1

#fig = plt.figure(figsize=(15,7))
sns.pairplot(data=df, x_vars=[cols[ppc*3], cols[ppc*3+1], cols[ppc*3+2]],
             y_vars='price_min', size=4)
plt.show()
ppc += 1

#fig = plt.figure(figsize=(15,7))
sns.pairplot(data=df, x_vars=[cols[ppc*3], cols[ppc*3+1], cols[ppc*3+2]],
             y_vars='price_min', size=4)
plt.show()
ppc += 1

#fig = plt.figure(figsize=(15,7))
sns.pairplot(data=df, x_vars=[cols[ppc*3], cols[ppc*3+1], cols[ppc*3+2]],
             y_vars='price_min', size=4)
plt.show()
ppc += 1

#fig = plt.figure(figsize=(15,7))
sns.pairplot(data=df, x_vars=[cols[ppc*3], cols[ppc*3+1], cols[ppc*3+2]],
             y_vars='price_min', size=4)
plt.show()
ppc += 1

#fig = plt.figure(figsize=(15,7))
sns.pairplot(data=df, x_vars=[cols[ppc*3], cols[ppc*3+1], cols[ppc*3+2]],
             y_vars='price_min', size=4)
plt.show()
ppc += 1

#fig = plt.figure(figsize=(15,7))
sns.pairplot(data=df, x_vars=[cols[ppc*3], cols[ppc*3+1], cols[ppc*3+2]],
             y_vars='price_min', size=4)
plt.show()
ppc += 1

#fig = plt.figure(figsize=(15,7))
sns.pairplot(data=df, x_vars=[cols[ppc*3], cols[ppc*3+1], cols[ppc*3+2]],
             y_vars='price_min', size=4)
plt.show()
ppc += 1

#fig = plt.figure(figsize=(15,7))
sns.pairplot(data=df, x_vars=[cols[ppc*3], cols[ppc*3+1], cols[ppc*3+2]],
             y_vars='price_min', size=4)
plt.show()
ppc += 1


I am working on the file "02_preprocessing/exploration00_features_vs_price_for_regression_potential.ipynb".
The first task is:
PAIRPLOTS CONCLUSIONS
From the above it does not seem that a single attribute has the ability to predict the price better than others, perhaps cpu power can be consider such and it must a combination of them
Standard Scale non binary columns
Can you write Python code for it?

df_norm = df.copy(deep=True)

ss = StandardScaler()

df_norm[cols] = ss.fit_transform(df[cols])

#df_norm.to_csv('../data/mobiles/mobiles_no_outliers_norm.csv', encoding='utf-8', quoting=csv.QUOTE_ALL)


I am working on the file "02_preprocessing/exploration00_features_vs_price_for_regression_potential.ipynb".
The first task is:
Standard Scale non binary columns for the mobiles with outliers
Can you write Python code for it?

orig_df = pd.read_csv('../mobiles_01_preprocessed.csv', index_col=0, encoding='utf-8', quoting=csv.QUOTE_ALL)

df = orig_df.drop(
    labels=SkroutzMobile.TEMP_DROP_COLS, axis=1
)

df_norm_with_outliers = df.copy(deep=True)

ss = StandardScaler()

df_norm_with_outliers[df.columns] = ss.fit_transform(df[df.columns])

df_norm_with_outliers.to_csv('../data/mobiles/mobiles_norm.csv', encoding='utf-8', quoting=csv.QUOTE_ALL)


I am working on the file "02_preprocessing/exploration01-linear_regression.ipynb".
The first task is:
Kind of gaussian distribution of the error but the test dataset is rather small
Which coefficient is stronger ?
Can you write Python code for it?

coefdic = dict(zip(XX.columns, np.absolute(model.coef_) ))

from collections import OrderedDict

#weights_sorted = sorted(coefdic, key=coefdic.get)[::-1]
weights_sorted = OrderedDict(sorted(coefdic.items(), key=lambda x: x[1]))

weights_sorted.keys()[::-1]

plt.rc('ytick', labelsize=20)

def weight_plot(weights_dic, step=1):
    fig = plt.figure(figsize=(14, 17))
    xvalues = np.arange(0, len(weights_dic)) + 1
    #plt.bar(left = weights_dic.keys(), height=weights_dic.keys()) #, align='center')
    #plt.xticks(xvalues[::step], weights_dic.keys()[::step])
    sns.barplot(weights_dic.values(), weights_dic.keys())
    plt.show()

weight_plot(weights_dic=weights_sorted, step=15)

strong_weights = OrderedDict(weights_sorted.items()[::-1][:40])

weight_plot(weights_dic=strong_weights)


I am working on the file "02_preprocessing/exploration01-linear_regression.ipynb".
The first task is:
Linear Regression with L2 regularization (Ridge Regression)
Can you write Python code for it?

def getModel(alpha):
    return Ridge(alpha=alpha, fit_intercept=True, normalize=False, copy_X=True, random_state=random_state)

model = getModel(alpha=0.01)

cvs = cross_val_score(estimator=model, X=XX, y=yy, cv=10)
cvs

cv_score = np.mean(cvs)
cv_score

def gpOptimization(n_jobs=n_jobs, cv=10, verbose=True):

    def objective(params): # Here we define the metric we want to minimise
        (alpha) = params
        if verbose:
            print "current L2: {}".format(alpha)

        start = time()

        model = getModel(alpha=alpha)

        cv_results = cross_val_score(estimator=model, X=XX, y=yy, cv=cv)

        cv_score = np.mean(cv_results)

        if verbose:
            print "current CV score: {}".format(cv_score)
            print "time elapsed: {}".format(time() - start)
            print

        # We want to maximise accuracy score, i.e. minimise minus average accuracy
        return -cv_score

    alpha_space = Real(1e-4, 1e1, prior='log-uniform')  #uniform or log-uniform

    return gp_minimize(
        func=objective, # function that we wish to minimise
        dimensions=[alpha_space], #the search space for the hyper-parameters
        #x0=x0, #inital values for the hyper-parameters
        n_calls=30, #number of times the function will be evaluated
        random_state=random_state, #random seed
        n_random_starts=5,  #before we start modelling the optimised function with a GP Regression
                            #model, we want to try a few random choices for the hyper-parameters.
        kappa=1.9, #trade-off between exploration vs. exploitation.
        n_jobs=n_jobs
    )

res_gp = gpOptimization()

fig = plt.figure(figsize=(12,6))
plot_convergence(res_gp)
plt.grid()
plt.show()

plt.figure(figsize=(12,6))
plt.plot(res_gp.func_vals)
plt.scatter(range(len(res_gp.func_vals)), res_gp.func_vals)
plt.ylabel(r'$f(x)$')
plt.xlabel('Number of calls $n$')
plt.xlim([0, len(res_gp.func_vals)])
plt.show()

best_alpha = res_gp.x[0]
best_alpha

model =...


I am working on the file "02_preprocessing/exploration01-linear_regression.ipynb".
The first task is:
We see that by numbers we have a good coefficient correlation between the true values and the predicted ones
Can you write Python code for it?

def fit_scatter(y_true, y_pred, x_label = 'Measured', y_label = 'Predicted'):
    assert y_true.shape == y_pred.shape
    fig, ax = plt.subplots()
    ax.scatter(y_true, y_pred)
    ax.plot([y_true.min(), y_true.max()], [y_true.min(), y_true.max()], 'k--', lw=4)
    ax.set_xlabel(x_label)
    ax.set_ylabel(y_label)
    return fig, ax

preds_test = model.predict(XX_test)

fig = plt.figure()

#plt.figure(figsize=(17,7))
fig, ax = fit_scatter(y_true=yy_test, y_pred=preds_test,
            x_label='Measured Price', y_label='Predicted Price')
#ax.set_figsize((17,7))
fig.set_figheight(12)
fig.set_figwidth(12)
plt.show()

#fig, ax = plt.subplots(figsize=(12,6))
#ax.set_xlabel('price error')
plt.figure(figsize=(12,6))
sns.distplot( yy_test - preds_test, rug=True, axlabel='Price Error')
plt.show()


I am working on the file "02_preprocessing/exploration03-price_history_standardization.ipynb".
The first task is:
Trust only up to three standard deviations.
Which is expected, ~75 euros difference from the original price is the maximum of what
normally see as a customer
Can you write Python code for it?

stds_threshold = std*3
stds_threshold

min(df_norm_prices.iloc[0])


I am working on the file "02_preprocessing/exploration03-price_history_standardization.ipynb".
The first task is:
and indeed those changing more than 75 euros / day are outliers
TODO maybe later we can filter those when choosing windows directly, because now we neglect entire time series
Can you write Python code for it?

df_seq_start_trimmed = df_with_seq_start.iloc[keep_inds]
df_seq_start_trimmed.shape

flatvals2 = df_norm_prices.iloc[keep_inds].values.flatten()
len(flatvals2)

plt.figure(figsize=(15,6))
sns.distplot(flatvals2)
plt.show()

csv_path = "../price_history_03_seq_start_suddens_trimmed.csv"

df_seq_start_trimmed.to_csv(csv_path, encoding='utf-8', quoting=csv.QUOTE_ALL)


I am working on the file "02_preprocessing/exploration04-price_history_dfa.ipynb".
The first task is:
Conclusion
the estimate alpha for the Hurst parameter (alpha < 1: stationary process similar to fractional Gaussian noise with H = alpha, alpha > 1: non-stationary process similar to fractional Brownian motion with H = alpha - 1)
So most price histories are identified as we would expect, as non-stationary processes
Can you write Python code for it?


I am working on the file "02_preprocessing/exploration04-price_history_dfa.ipynb".
The first task is:
https://cschoel.github.io/nolds/nolds.html#detrended-fluctuation-analysis
Can you write Python code for it?

# -*- coding: UTF-8 -*-
from __future__ import division
import numpy as np
import pandas as pd
import sys
import math
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
import re
import os
import csv
from helpers.outliers import MyOutliers
from skroutz_mobile import SkroutzMobile
from sklearn.ensemble import IsolationForest
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, r2_score
from skroutz_mobile import SkroutzMobile
from sklearn.model_selection import StratifiedShuffleSplit
from helpers.my_train_test_split import MySplitTrainTest
from sklearn.preprocessing import StandardScaler
from preprocess_price_history import PreprocessPriceHistory
from price_history import PriceHistory
from dfa import dfa
import scipy.signal as ss
import nolds
%matplotlib inline

random_state = np.random.RandomState(seed=16011984)

csv_in = "../price_history_02_with_seq_start.csv"

orig_df = pd.read_csv(csv_in, index_col=0, encoding='utf-8', quoting=csv.QUOTE_ALL)
orig_df.shape

df = orig_df.drop(labels=PriceHistory.SPECIAL_COLS, axis=1)
df.shape

CSV_FILEPATH = "../price_history_02_with_seq_start.csv"

#xx = df.iloc[0, ]
ph = PriceHistory(CSV_FILEPATH)

tt = ph.extractSequenceByLocation(iloc=0)
tt.shape

tt[-1]

alpha = nolds.dfa(tt)
alpha

seqs = [ph.extractSequenceByLocation(iloc=ii) for ii in xrange(len(ph.df))]
len(seqs)

len(seqs[0])

alphas = []
for seq in seqs:
    try:
        alpha = nolds.dfa(seq.values)
        if not np.isnan(alpha):
            alphas.append(alpha)
    except AssertionError, ee:
        pass

#alphas = [seq for seq in seqs if len(seq) > 1 and not np.all(seq[0] == seq)]
len(alphas)

plt.figure(figsize=(17,8))
sns.distplot(alphas, rug=True,
            ...


I am working on the file "02_preprocessing/exploration05-price_history_clustering.ipynb".
The first task is:
6 htan 0.23
7 htan 0.24
Can you write Python code for it?

n_clusters = 6
print n_clusters
silhouette, inertia = calc_n_clusters(n_clusters = n_clusters)
print silhouette, inertia
silhouette_scores[n_clusters] = silhouette

n_clusters = 7
print n_clusters
silhouette, inertia = calc_n_clusters(n_clusters = n_clusters)
print silhouette, inertia
silhouette_scores[n_clusters] = silhouette

n_clusters = 8
print n_clusters
silhouette, inertia = calc_n_clusters(n_clusters = n_clusters)
print silhouette, inertia
silhouette_scores[n_clusters] = silhouette


I am working on the file "02_preprocessing/exploration05-price_history_clustering.ipynb".
The first task is:
Dynamic Time Wrapping
Dynamic time warping finds the optimal non-linear alignment between two time series.
K means clustering
https://stackoverflow.com/questions/5529625/is-it-possible-to-specify-your-own-distance-function-using-scikit-learn-k-means
Can you write Python code for it?

mms = MinMaxScaler()

seqs_norm = mms.fit_transform(seqs_fixed_width.T).T
seqs_norm.shape

#set random centroids or pseudorandom
#calculate the distance of all points with centroid 1, then the same for centroid 2
#then find a new centroid according to the points you had considered
#then repeat

#kmeans = KMeans(n_clusters=n_clusters, random_state=random_state)

# centroids = random.sample(seqs, 3)
# len(centroids), len(centroids[0]), len(centroids[1]), len(centroids[2])

# aa , _ = fastdtw(seqs[0], seqs[2], dist=euclidean)
# aa

def k_means_clust(data, n_clusters, tol=0.1, n_max_iter=100): #tol=0.0001
    centroids = random.sample(data, n_clusters)
    prev_centroids = centroids[:]
    best_max_diff = float('inf')

    data_len = len(data)

    labels = np.empty(data_len)

    counter = 0
    while counter < n_max_iter:
        counter += 1
        print counter

        #we only care about the last inertia but it is minimum cost to calc it on every iter
        cur_inertia = 0

        assignments = {}  #dictionary cluster -> data points

        for data_ind, xx in enumerate(data):
            min_dist = float('inf')
            closest_clust_ind = None

            for c_ind, cc in enumerate(centroids):
                cur_dist, _ = fastdtw(xx, cc, dist=euclidean)
                if cur_dist < min_dist: #this means that current centroid is better
                    closest_clust_ind = c_ind
                    min_dist = cur_dist

            labels[data_ind] = closest_clust_ind

            cur_inertia += min_dist

            if closest_clust_ind in assignments:
                assignments[closest_clust_ind].append(data_ind)
            else:
                assignments[closest_clust_ind] = [data_ind]

        #recalculate centroids of clusters, here we do it in a euclidean way
        #(we do not have reverse dtw)
        for key in assignments:
            cur_assignment_list = assignments[key]
        ...


I am working on the file "02_preprocessing/exploration05-price_history_clustering.ipynb".
The first task is:
Some processing
Can you write Python code for it?

csv_in = "../price_history_03_seq_start_suddens_trimmed.csv"

df_fixed_width = PreprocessPriceHistory.keepOnlyPriceHistoriesWithMostFrequentLength(csv_in=csv_in)

csv_path = "../price_history_03a_fixed_width.csv"
#df_fixed_width.to_csv(csv_path, encoding='utf-8', quoting=csv.QUOTE_ALL)


I am working on the file "02_preprocessing/exploration05-price_history_clustering.ipynb".
The first task is:
Time Series Classification under More Realistic Assumptions
Project URL: http://sites.google.com/site/sdm13realistic/
Paper URL: http://www.cs.ucr.edu/~eamonn/SDM_RealisticTSClassifcation_cameraReady.pdf
How to choose the best k in k-means clustering?
Use the Silhouette score:
http://scikit-learn.org/stable/modules/generated/sklearn.metrics.silhouette_score.html
Plotting Silhouette Scores
Can you write Python code for it?


I am working on the file "02_preprocessing/exploration05-price_history_clustering.ipynb".
The first task is:
http://nbviewer.jupyter.org/github/alexminnaar/time-series-classification-and-clustering/blob/master/Time%20Series%20Classification%20and%20Clustering.ipynb
Can you write Python code for it?

from __future__ import division
import numpy as np
import pandas as pd
import sys
import math
from sklearn.preprocessing import LabelEncoder, OneHotEncoder
import re
import os
import csv
from helpers.outliers import MyOutliers
from skroutz_mobile import SkroutzMobile
from sklearn.ensemble import IsolationForest
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeClassifier, export_graphviz
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, confusion_matrix, r2_score
from skroutz_mobile import SkroutzMobile
from sklearn.model_selection import StratifiedShuffleSplit
from helpers.my_train_test_split import MySplitTrainTest
from sklearn.preprocessing import StandardScaler
from preprocess_price_history import PreprocessPriceHistory
from price_history import PriceHistory
from dfa import dfa
import scipy.signal as ss
import nolds
from scipy.spatial.distance import euclidean
from fastdtw import fastdtw
from sklearn.cluster import KMeans
from sklearn.preprocessing import MinMaxScaler
import random
from sklearn.metrics import silhouette_score
from os.path import isfile
from preprocess_price_history import PreprocessPriceHistory

random_state = np.random.RandomState(seed=16011984)
%matplotlib inline


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
Conclusion
is that the issue is with the large number of rows / instances that bring the gaussian process regressor to memory error even if we use only the single first principal component. There is nothing more of a dimensionality reduction we could do
Let's see how much instances GP regressor can handle without hitting out of memory error
Can you write Python code for it?

from time import sleep

count_prev = None
count = len(XX)

while count_prev != count:
    count_prev = count
    try:
        gp = GaussianProcessRegressor(copy_X_train=False)
        gp.fit(XX[:count], yy[:count])
        count = int(count * 1.5)
    except MemoryError:
        print "memory error"
        count //= 2

    print count
    sleep(1)


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
Dimensionality reduction
Can you write Python code for it?

from sklearn.decomposition import PCA

pca = PCA()

%%time
pca.fit(XX)

plt.figure(1, figsize=(12, 5))
plt.clf()
plt.axes([.2, .2, .7, .7])
plt.plot(pca.explained_variance_ratio_, linewidth=2)
plt.axis('tight')
plt.xlabel('n_components')
plt.ylabel('explained_variance_')
plt.show()

explainedVar = 0
explainedVars = []
for ii in range(len(pca.explained_variance_ratio_)):
    explainedVar += pca.explained_variance_ratio_[ii]
    explainedVars.append(explainedVar)

plt.figure(1, figsize=(16, 5))
plt.clf()
plt.axes([.2, .2, .7, .7])
plt.plot(explainedVars, linewidth=2)
plt.axis('tight')
plt.xlabel('n_components')
plt.ylabel('cumulative explained variance')
plt.show()

explainedVariancePercentage = 0
pointWherVarExplainedOver95percent = -1
ii = 0
while explainedVariancePercentage < 0.95:
    explainedVariancePercentage += pca.explained_variance_ratio_[ii]
    pointWherVarExplainedOver95percent += 1
    ii += 1
pointWherVarExplainedOver95percent

pca_reduced = PCA(n_components=pointWherVarExplainedOver95percent)

%%time
XX_reduced = pca_reduced.fit_transform(XX)

XX_reduced.shape

# gp = GaussianProcessRegressor(copy_X_train=False)
# gp.fit(XX_reduced, yy)

sum(explainedVars[:2])

pca_2d = PCA(n_components=2, random_state=random_state)
pca_2d

%time
XX_pca2d = pca_2d.fit_transform(XX)
XX_pca2d.shape

plt.scatter(XX_pca2d[:, 0], XX_pca2d[:, 1], marker ='.')
plt.show()

#memory error
# gp = GaussianProcessRegressor(copy_X_train=False)
# gp.fit(XX_pca2d, yy)

XX_1 = PCA(n_components=1, random_state=random_state).fit_transform(XX)
XX_1.shape

#memory error
# gp = GaussianProcessRegressor(copy_X_train=False)
# gp.fit(XX_1, yy)


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
Gaussian Process George Library
Can you write Python code for it?

kernel = np.var(yy) * kernels.ExpSquaredKernel(0.5)
kernel

gp = george.GP(kernel)
gp

gp.compute(xx, yerr=yerr)

xx_pred = np.linspace(0, 15, 500)
xx_pred.shape

pred, pred_var = gp.predict(yy, xx_pred, return_var=True)

plt.figure(figsize=(17,7))
plt.fill_between(xx_pred, pred - np.sqrt(pred_var), pred + np.sqrt(pred_var),
                color="k", alpha=0.2)
plt.plot(xx_pred, pred, "k", lw=1.5, alpha=0.5)
plt.errorbar(xx, yy, yerr=yerr, fmt=".k", capsize=0)
plt.plot(xx_pred, np.sin(xx_pred), "--g")
plt.xlim(0, 15)
plt.ylim(-1.45, 1.45)
plt.xlabel("x")
plt.ylabel("y");


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
Gaussian Process Regressor
Can you write Python code for it?

arr_norm = np.load(mobattrs_ph_norm_path)
arr_norm.shape

XX = arr_norm[:, :MobAttrsPriceHistoryMerger.PRICE_IND]
XX.shape

yy = arr_norm[:, MobAttrsPriceHistoryMerger.PRICE_IND]
yy.shape

# gp = GaussianProcessRegressor(copy_X_train=False)
# gp.fit(XX, yy)


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
Isomap Dimensionality Reduction
Can you write Python code for it?

from sklearn.manifold import Isomap

isomap = Isomap(n_components=2, n_neighbors=2)

#Memory Error
# %%time
# XX_isomap = isomap.fit_transform(XX)

%%time
XX_isomap = isomap.fit_transform(XX_reduced)

XX_isomap.shape

plt.scatter(XX[:, 0], XX[:, 1])
plt.show()


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
Kernel PCA
Can you write Python code for it?

from sklearn.decomposition import KernelPCA

rbf_pca = KernelPCA(n_components=2, random_state=random_state, kernel='rbf')
rbf_pca

%%time
XX_rbf = rbf_pca.fit_transform(XX)


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
Loading data
Can you write Python code for it?

mobiles_path = data_path + '/mobiles'

mobs_norm_path = mobiles_path + '/mobiles_norm.csv'
assert isfile(mobs_norm_path)

df = pd.read_csv(mobs_norm_path, index_col=0, encoding='utf-8', quoting=csv.QUOTE_ALL)
df.shape

seqs = ph.extractAllSequences()
len(seqs)


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
MDS dimensionality reduction
Can you write Python code for it?

from sklearn.manifold import MDS

mds = MDS(n_components=2, max_iter=10, n_init=1, random_state=random_state)

#memory error
# %%time
# XX_mds = mds.fit_transform(XX)

#memory error
# %%time
# XX_mds = mds.fit_transform(XX_reduced)


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
Quick Gaussian Process Regressor test
Can you write Python code for it?

gp = GaussianProcessRegressor()

xx = 10 * np.sort(np.random.rand(15))
xx.shape

yerr = 0.2 * np.ones_like(xx)
yerr

yy = np.sin(xx) + yerr * np.random.randn(len(xx))
yy.shape

plt.errorbar(xx, yy, yerr=yerr, fmt='.k', capsize=0)
plt.show()


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
Some processing
Can you write Python code for it?

csv_in = "../price_history_03_seq_start_suddens_trimmed.csv"

#csv_out = "../price_history_for_sfa.csv"
#df_fixed_width.to_csv(csv_path, encoding='utf-8', quoting=csv.QUOTE_ALL)

ph = PriceHistory(csv_in)

seq = ph.extractSequenceByLocation(0)
print type(seq)
seq.shape, seq.name

#seq
#plt.figure(figsize=(16,6))
seq.plot()
plt.show()

plt.figure(figsize=(16,6))
seq.hist()
plt.show()


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
merging
Can you write Python code for it?

obj = MobAttrsPriceHistoryMerger(mobs_norm_path=mobs_norm_path, price_history_csv=csv_in)

%%time
arr = obj.get_table()

arr.shape

mobattrs_ph_raw_path = mobattrs_ph_path + '/mobattrs_ph_raw.npy'
mobattrs_ph_raw_path

np.save(mobattrs_ph_raw_path, arr)
assert isfile(mobattrs_ph_raw_path)


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
normalizing price history info
actually normalizing the columns related to dates and the targets, the price itself
Can you write Python code for it?

obj.get_mob_attrs_indices()

obj.get_date_indices()

arr_norm = obj.normalize_date_columns(arr)

for cur_date_ind in obj.get_date_indices():
    cur_col = arr_norm[:, cur_date_ind]
    cur_mean = np.mean(cur_col)
    cur_std = np.std(cur_col)
    assert np.allclose(arr_norm[:, cur_date_ind], (cur_col - cur_mean) / cur_std)

#Let's also normalize the price, the targets

price_col = arr_norm[:, MobAttrsPriceHistoryMerger.PRICE_IND]
price_mean = np.mean(price_col)
price_std = np.std(price_col)
arr_norm[:, MobAttrsPriceHistoryMerger.PRICE_IND] = (price_col - price_mean) / price_std

np.save(mobattrs_ph_norm_path, arr_norm)
assert isfile(mobattrs_ph_norm_path)

arr_norm.shape


I am working on the file "02_preprocessing/exploration08-price_history_gaussian_process_regressor_memory_errors.ipynb".
The first task is:
t-SNE dimensionality reduction
let's bring on the big guns
Can you write Python code for it?

tsne = TSNE(n_components=2, perplexity=2)

#memory error
# XX_tsne2 = tsne.fit_transform(XX)
# XX_tsne2.shape

#memory error
# %%time
# XX_tsne2 = tsne.fit_transform(XX_reduced)


I am working on the file "02_preprocessing/exploration11-price_history_gaussian_process_regressor_clustered_cross_valid.ipynb".
The first task is:
Cluster 1
Can you write Python code for it?

%%time
cur_gp_opt = GaussianProcessPricePredictorGpOpt(chosen_cluster=1,
                                               bayes_opt_dir=bayes_opt_dir,
                                               cv_score_dict_npy_filename=cv_score_dict_npy_filename,
                                               pairs_ts_npy_filename=pairs_ts_npy_filename,
                                               res_gp_filename=res_gp_filename,
                                               npz_sku_ids_group_kmeans=npz_sku_ids_group_kmeans,
                                               mobs_norm_path=mobs_norm_path,
                                               price_history_csv=price_history_csv,
                                               input_min_len=input_min_len,
                                               target_len=target_len,
                                               random_state=random_state,
                                               verbose = True,
                                               n_restarts_optimizer=2)

opt_res = cur_gp_opt.run_opt(n_random_starts=7, n_calls=15)

plot_res_gp(opt_res)

opt_res.best_params


I am working on the file "02_preprocessing/exploration11-price_history_gaussian_process_regressor_clustered_cross_valid.ipynb".
The first task is:
Cluster 4
Can you write Python code for it?

%%time
cur_gp_opt = GaussianProcessPricePredictorGpOpt(chosen_cluster=4,
                                               bayes_opt_dir=bayes_opt_dir,
                                               cv_score_dict_npy_filename=cv_score_dict_npy_filename,
                                               pairs_ts_npy_filename=pairs_ts_npy_filename,
                                               res_gp_filename=res_gp_filename,
                                               npz_sku_ids_group_kmeans=npz_sku_ids_group_kmeans,
                                               mobs_norm_path=mobs_norm_path,
                                               price_history_csv=price_history_csv,
                                               input_min_len=input_min_len,
                                               target_len=target_len,
                                               random_state=random_state,
                                               verbose = True,
                                               n_restarts_optimizer=4)

opt_res = cur_gp_opt.run_opt(n_random_starts=5, n_calls=20)

plot_res_gp(opt_res)

opt_res.best_params


I am working on the file "02_preprocessing/exploration11-price_history_gaussian_process_regressor_clustered_cross_valid.ipynb".
The first task is:
Cluster 6
Can you write Python code for it?

%%time
cur_gp_opt = GaussianProcessPricePredictorGpOpt(chosen_cluster=6,
                                               bayes_opt_dir=bayes_opt_dir,
                                               cv_score_dict_npy_filename=cv_score_dict_npy_filename,
                                               pairs_ts_npy_filename=pairs_ts_npy_filename,
                                               res_gp_filename=res_gp_filename,
                                               npz_sku_ids_group_kmeans=npz_sku_ids_group_kmeans,
                                               mobs_norm_path=mobs_norm_path,
                                               price_history_csv=price_history_csv,
                                               input_min_len=input_min_len,
                                               target_len=target_len,
                                               random_state=random_state,
                                               verbose = False,
                                               n_restarts_optimizer=5)

opt_res = cur_gp_opt.run_opt(n_random_starts=3, n_calls=10)

plot_res_gp(opt_res)

opt_res.best_params


I am working on the file "02_preprocessing/exploration11-price_history_gaussian_process_regressor_clustered_cross_valid.ipynb".
The first task is:
Cluster 7
Can you write Python code for it?

%%time
cur_gp_opt = GaussianProcessPricePredictorGpOpt(chosen_cluster=7,
                                               bayes_opt_dir=bayes_opt_dir,
                                               cv_score_dict_npy_filename=cv_score_dict_npy_filename,
                                               pairs_ts_npy_filename=pairs_ts_npy_filename,
                                               res_gp_filename=res_gp_filename,
                                               npz_sku_ids_group_kmeans=npz_sku_ids_group_kmeans,
                                               mobs_norm_path=mobs_norm_path,
                                               price_history_csv=price_history_csv,
                                               input_min_len=input_min_len,
                                               target_len=target_len,
                                               random_state=random_state,
                                               verbose = True,
                                               n_restarts_optimizer=2)

opt_res = cur_gp_opt.run_opt(n_random_starts=6, n_calls=13)

plot_res_gp(opt_res)

opt_res.best_params


I am working on the file "02_preprocessing/exploration11-price_history_gaussian_process_regressor_clustered_cross_valid.ipynb".
The first task is:
train test split
Can you write Python code for it?

sku_id_groups = np.load(npz_sku_ids_group_kmeans)
for key, val in sku_id_groups.iteritems():
    print key, ",", val.shape

# gp_predictor = GaussianProcessPricePredictorForCluster(npz_sku_ids_group_kmeans=npz_sku_ids_group_kmeans,
#                                                       mobs_norm_path=mobs_norm_path,
#                                                       price_history_csv=price_history_csv,
#                                                       input_min_len=input_min_len,
#                                                       target_len=target_len)

%%time
#gp_predictor.prepare(chosen_cluster=9)

%%time
#dtw_mean = gp_predictor.train_validate()

#dtw_mean

# Do not run this again unless you have enough space in the disk and lots of memory
# with open('cur_gp.pickle', 'w') as fp:  # Python 3: open(..., 'wb')
#     pickle.dump(gp, fp)


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
2. Operaciones de busqueda en un string
Realizar operaciones de búsqueda de un caracter o subcadena dentro de una cadena, pudiendo contabilizar cuantas veces se repite este carácter o determinar la posición en la cual se encuentra la subcadena dentro de la cadena es posible mediante los siguientes comandos.
```Python
```
Can you write Python code for it?

cadena_3 = "buscando a wally".title()

cadena_3.find("Wally")


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
3. Operaciones lógicas de formato
Veremos que podemos realizar sentencias lógicas en relación a características de un string, esto puede ser de mucha ayuda a la hora de definir ciertos formatos para variables que el programador o usuario deben manipular.
Comenzamos con sentencias que refieren al formato de un string, por ejemplo saber si comienza o termina con determinados caracteres, utilizando los comandos startswith() y endswith() respectivamente, a lo que Python respondera con verdadero o falso.
Can you write Python code for it?

cadena_5 = "construyendo un detector de mentiras"
print cadena_5.startswith("construyendo un")

print cadena_5.endswith("detector")


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
4. Sentencias de condicionamiento
Por último en este tutorial introduciremos una sentencia de condicionamiento que nos permitirá ampliar el espectro de aplicación de operaciones lógicas a partir de elementos básicos.
Can you write Python code for it?

contrasena_1 = "abcd1234"

if len(contrasena_1)<10:
    print "la contrasena contiene menos de 10 caracteres"
else:
    print "contrasena válida"

contrasena_2 = "abcd1234ab"
respuesta = contrasena_2.islower()

if len(contrasena_2)<10:
    print "la contrasena contiene menos de 10 caracteres"
elif respuesta == True:
    print "la contrasena debe contener al menos 1 caracter mayúscula"
else:
    print "contrasena válida"


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
5. Ejercicio de práctica
Para aplicar lo aprendido en este tutorial, se deja el ejercicio de crear una pequeña rutina que verifique si los datos referente a las variables nombre_usuario y contrasena cumplen con las siguientes condiciones:

El nombre de usuario debe encontrarse en formato de título
El nombre de usuario sólo debe contener caracteres alfabéticos
Para la contraseña se debe especificar que si tiene menor a 5 caracteres no es válida
Si la contraseña tiene menos de 8 caracteres es riesgosa y mayor o igual a 10 es segura
La contraseña debe ser de tipo alfanumérica y no debe tener espacios en blanco
Por último la contrasena debe contener mayúsculas y minúsculas
Can you write Python code for it?


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
Además podemos verificar si el string contiene caracteres sólo numéricos, alfabéticos o alfanuméricos, usando los comandos isdigit(), isalpha() y isalnum() respectivamente.
Can you write Python code for it?

cadena_6 = "ano"
print cadena_6.isdigit()

cadena_7 = "1987"
print cadena_7.isalpha()

cadena_8 = cadena_6 + cadena_7
print cadena_8
print cadena_8.isalnum()


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
Centrar o alinear un texto a la izquierda o derecha de la pantalla son otras opciones de formato posibles de realizar con comandos de python definidos para un string.
Can you write Python code for it?

cadena_2 = "lo maravilloso de programar"

print cadena_2.center(50,"=")


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
EL comando find nos arroja un número que nos indica la posición que antecede al primer caracter de Wally, es decir encontramos a Wally luego del carácter que se ubica luego de la posición 11 dentro de la cadena 3. Otra opción es realizar una búsqueda de un subtexto en un tramo definido de la variable, mencionando además que cuando no se encuentra el carácter en el tramo definido, por defecto nos arroja el valor -1.
Can you write Python code for it?


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
Introducción a STRING
En Python podemos definir distintos tipos de variables, en este caso un objeto definido como un string tiene una serie de cualidades que nos permiten realizar ciertas operaciones que definiremos en este tutorial.
En particular el nombre string refiere a una secuencia de caracteres que puede ser utilizada como variable para representar otro objeto de distinta naturaleza como un número, también puede construirse un algoritmo que nos permita dar acceso a algún archivo o directorio a partir de una contraseña, entre otras posibilidades. A continuación presentamos una serie de operaciones comunes que nos permitirán tener un manejo básico de un string.
Objetivos

Operaciones básicas para crear un formato
Operaciones de búsqueda
Operaciones lógicas de formato
Sentencias de condicionamiento
Ejercicio de práctica

1. Operaciones básicas para asignar un formato
En ocasiones se vuelve útil poder controlar a partir de ciertos comandos el formato de un string, pudiendo cambiar letras mayúsculas y minúsculas, centrar o mover la variable a un extremo de la pantalla y/o darle un formato en específico a nuestra secuencia.
Por ejemplo un string que definiremos como variable podemos hacer que represente un pequeño texto, una vez definido nuestro string le daremos un formato de título mediante el siguiente comando.
Can you write Python code for it?

cadena_1 = "aprender haciendo"

print cadena_1.title()


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
La última característica de formato aplicada desde una operación lógica será si la cadena contiene sólo espacios en blanco, para lo que utilizaremos el comando isspace() como sigue a continuación.
Can you write Python code for it?

nombre_completo = "isaac asimov".title()
print nombre_completo.isspace()


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
Luego podemos intentar con otras características de formato, como por ejemplo saber si la cadena contiene caracteres sólo mayúsculas, sólo minúsculas o inclusive algún formato especifico como de título, usando los comandos isupper(), islower() y por último istitle()
Can you write Python code for it?

nombre = "isaac"
print nombre.isupper()

apellido = "asimov".upper()
print apellido.islower()
respuesta = apellido.islower()
print respuesta

nombre_completo = nombre.title() + apellido.title()

print nombre_completo


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
Podemos redefinir una variable con cierto formato y luego convinar con otra opción de formato como sigue.
Can you write Python code for it?

cadena_1 = "aprender haciendo".title()

print cadena_1.rjust(40, " ")


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
Podemos también contabilizar cuantas veces se encuentra un caracter o subcadena dentro de una cadena con el comando count. Y de igual modo que antes la contabilización es posible de realizar en un tramo definido del texto.
Can you write Python code for it?

cadena_4 = "que fácil de encontrar"

cadena_4.count("e")

cadena_4.count("e",0,13)


I am working on the file "02_python_strings/strings.ipynb".
The first task is:
Se puede observar que el comando anterior ajusta el texto a la derecha, además de que se puede introducir cualquier símbolo incluso espacio, su homólogo a la izquierda es el siguiente.
Can you write Python code for it?


I am working on the file "02_representacion_vectorial_de_texto.ipynb".
The first task is:
TFIDF
buen dia microtc
Can you write Python code for it?

from microtc.textmodel import TextModel
model = TextModel(docs, token_list=[-1])
print(model[docs[0]])


I am working on the file "02_representacion_vectorial_de_texto.ipynb".
The first task is:
diac, dup y punc
 Autoridades de la Ciudad de México aclaran que el equipo del cineasta mexicano no fue asaltado, pero sí una riña ahhh.
Can you write Python code for it?

diac = norm_chars(text, del_diac=True, del_dup=False, del_punc=False).replace('~', ' ')
Markdown("## diac\n" + diac)

dup = norm_chars(text, del_diac=False, del_dup=True, del_punc=False).replace('~', ' ')
Markdown("## dup\n" + dup)

punc = norm_chars(text, del_diac=False, del_dup=False, del_punc=True).replace('~', ' ')
Markdown("## punc\n" + punc)

from microtc.emoticons import EmoticonClassifier
from microtc.params import OPTION_GROUP, OPTION_DELETE
text = "Hoy es un día feliz :) :) o no :( "


I am working on the file "02_representacion_vectorial_de_texto.ipynb".
The first task is:
lc
@mgraffg pon atención para sacar 10 en http://github.com/INGEOTEC
Can you write Python code for it?


I am working on the file "02_representacion_vectorial_de_texto.ipynb".
The first task is:
num
@mgraffg pon atención para sacar 10 en http://github.com/INGEOTEC
Can you write Python code for it?

delete = re.sub(r"\d+\.?\d+", "", text)
group = re.sub(r"\d+\.?\d+", "_num", text)
Markdown("## delete\n%s\n## group\n%s" % (delete, group))


I am working on the file "02_representacion_vectorial_de_texto.ipynb".
The first task is:
q-grams
que buena esta la platica
Can you write Python code for it?

model = TextModel([], token_list=[3])
words = model.tokenize(text)
model = TextModel([], token_list=[4])
biw = model.tokenize(text)
Markdown("## 3\n %s\n## 4\n%s" % (", ".join(words), ", ".join(biw)))


I am working on the file "02_representacion_vectorial_de_texto.ipynb".
The first task is:
skip-grams
que buena esta la platica
Can you write Python code for it?

model = TextModel([], token_list=[(2, 1)])
words = model.tokenize(text)
model = TextModel([], token_list=[(2, 2)])
biw = model.tokenize(text)
Markdown("## (2, 1)\n %s\n## (2, 2)\n%s" % (", ".join(words), ", ".join(biw)))


I am working on the file "02_representacion_vectorial_de_texto.ipynb".
The first task is:
url
@mgraffg pon atención para sacar 10 en http://github.com/INGEOTEC
Can you write Python code for it?

delete = re.sub(r"https?://\S+", "", text)
group = re.sub(r"https?://\S+", "_url", text)
Markdown("## delete\n%s\n## group\n%s" % (delete, group))


I am working on the file "02_representacion_vectorial_de_texto.ipynb".
The first task is:
usr
@mgraffg pon atención para sacar 10 en http://github.com/INGEOTEC
Can you write Python code for it?

delete = re.sub(r"@\S+", "", text)
group = re.sub(r"@\S+", "_usr", text)
Markdown("## delete\n%s\n## group\n%s" % (delete, group))


I am working on the file "02_representacion_vectorial_de_texto.ipynb".
The first task is:
¿Por qué es robusto a errores?
Considere los siguientes textos $T=I_like_vanilla$, $T' = I_lik3_vanila$
Para fijar ideas pongamos que se usar el coeficiente de Jaccard como medida de similitud, i.e.
$$\frac{|{{I, like, vanilla}} \cap {{I, lik3, vanila}}|}{|{{I, like, vanilla}} \cup {{I, lik3, vanila}}|} = 0.2$$
$$Q^T_3 = { I_l, _li, lik, ike, ke_, e_v, _va, van, ani, nil, ill, lla }$$
$$Q^{T'}_3 = { I_l, _li, lik, ik3, k3_, 3_v, _va, van, ani, nil, ila }$$
Bajo la misma medida
$$\frac{|Q^T_3 \cap Q^{T'}_3|}{|Q^T_3 \cup Q^{T'}_3|} = 0.448.$$
Se puede ver que estos conjuntos son más similares que los tokenizados por palabra
La idea es que un algoritmo de aprendizaje tenga un poco más de soporte para determinar que $T$ es parecido a $T'$
Pesado de texto
|   nombre  | valores             |        descripción                   |
|-----------|---------------------|--------------------------------------|
|token_min_filter | ${0.01, 0.03, 0.1, 0.30, -1, -5, -10}$ | Filtro de frequencias bajas |
|token_max_filter | ${0.9, 99, 1.0}$ | Filtro de frequencias altas |
|   tfidf    | yes, no | Determina si se debe realizar un pesado TFIDF de terminos |
Sobre el pesado
El pesado de tokens esta fijo a TFIDF. Su nombre viene de la formulación $tf \times idf$
$tf$ es term frequency; es una medida de importancia local del término $t$ en el documento $d$, de manera normalizada esta definida como:
    $$tf(t,d) = \frac{freq(t, d)}{\max_{w \in d}{freq(w, d)}}$$
entre más veces aparece en el documento $d$, $t$ es más importante
$idf$ quiere decir inverse document frequency; es una medida global a la colección $D$, esta definida como:
$$ idf(t,d) = log{\frac{|D|}{1+|{d \in D: t \in d}|}} $$
entre más veces aparece $t$ en la colección, el término es más común y menos discriminante; por lo tanto, menos importante
Can you write Python code for it?

docs = ["buen dia microtc", "excelente dia", "buenas tardes",
	"las vacas me deprimen", "odio los lunes", "odio el trafico",
	"la computadora", "la mesa", "la ventana"]
l = ["* " + x for x in docs]
Markdown("# Corpus\n" + "\n".join(l))


I am working on the file "02_selenium/00_selenium_crawling_render_image.ipynb".
The first task is:
使用 Selenium 做動態爬蟲
Can you write Python code for it?

driver = webdriver.Chrome()
results = os.path.abspath('../results')
if not os.path.exists(results):
    os.makedirs(results)

try:
    # webdriver setting
    driver.get(url)
    driver.maximize_window()
    driver.implicitly_wait(10)

    # xpath
    imgs = driver.find_elements(By.XPATH, '/html/body/div/div/div/a/img')
    imgs = [i.get_attribute('src') for i in imgs]
    imgs = list(set(imgs))
    print(imgs)

    # download
    for img in imgs:
        headers = {'User-Agent': fu.random}
        img_resp = requests.get(img, stream=True, headers=headers)

        filename = os.path.basename(img)
        print('catch - {}'.format(filename))
        filename = os.path.join(results, filename)

        with open(filename, 'wb') as f:
            for chunk in img_resp.iter_content(2048):
                f.write(chunk)
            print('save - {}'.format(filename))

except Exception as e:
    print(e)
finally:
    driver.quit()


I am working on the file "02_selenium/00_selenium_crawling_render_image.ipynb".
The first task is:
使用 requests 做靜態爬蟲
Can you write Python code for it?

resp = requests.get(url)
soup = BeautifulSoup(resp.text, 'lxml')
imgs = soup.find_all('img', class_=re.compile('.*img-change'))
imgs = [i['src'] for i in imgs]
imgs = list(set(imgs))
pprint(imgs)


I am working on the file "02_starting_notebooks.ipynb".
The first task is:
2. Create a self-signed certificate
comet$ openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
3. Create a password hash
In ipython execute
Can you write Python code for it?

from IPython.lib import passwd

passwd('test password')


I am working on the file "02_starting_notebooks.ipynb".
The first task is:
Not recommended: Use interactive ipython and then just passwd()
4. Edit the configuration for your profile
vim .ipython/profile_nbserver/ipython_notebook_config.py
Can you write Python code for it?

!ssh thauser@gordon.sdsc.xsede.org 'head -n 12 .ipython/profile_nbserver/ipython_notebook_config.py'


I am working on the file "02_starting_notebooks.ipynb".
The first task is:
Running the secured remote notebook
ipython notebook --profile=nbserver
connect to
https://comet-ln3.sdsc.edu:11111
you need to enter your password
Running the notebook in an interactive job on Comet

Start your interactive job
comet$ srun -u -t 10 /bin/bash -i

Start your notebook
comet-04-45$ hostname
    comet-04-45.sdsc.edu
comet-04-45$ ipython notebook --profile=nbserver

Connect to it

https://comet-04-45.sdsc.edu:11111
Port forwarding using ssh
On some machines compute nodes maybe not accessible from the outside
ssh port forwarding
On your local machine:
```
ssh -L localport:nodename:remote_port -f -N comet.sdsc.edu
$ ssh -L 8088:comet-04-45:9088 -f -N comet comet.sdsc.edu
```
open your localbrowser at https://localhost:8088
Using SAGA to run a job on a remote node of a cluster
A light-weight access layer for distributed computing infrastructure
http://saga-project.github.io/saga-python/
Install on your local machine
$ pip install saga-python
Create a saga script, e.g
Can you write Python code for it?

import sys
import time
import saga

# Adapted from the saga example
# Your ssh identity on the remote machine.
ctx = saga.Context("ssh")
ctx.user_id = 'thauser'

session = saga.Session()
session.add_context(ctx)

# Create a job service object that represent a remote pbs cluster.
js = saga.job.Service("slurm+ssh://comet.sdsc.edu", session=session)

# Set the parameters for this example
local_port=9988
remote_port=11111
username='thauser'
hostname='comet.sdsc.edu'

# Next, we describe the job we want to run. A complete set of job
# description attributes can be found in the API documentation.
jd = saga.job.Description()
jd.wall_time_limit   = 10 # minutes
jd.executable        = "ipython notebook --profile=nbserver"
jd.queue             = "compute"
jd.working_directory = "A"
jd.output            = "ipythonjob.out"
jd.error             = "ipythonjob.err"

touchjob = js.create_job(jd)
# Check our job's id and state
print "Job ID    : %s" % (touchjob.id)
print "Job State : %s" % (touchjob.state)

# Now we can start our job.
print "\n...starting job...\n"
touchjob.run()

print "Job ID    : %s" % (touchjob.id)
print "Job State : %s" % (touchjob.state)

# List all jobs that are known by the adaptor.
# This should show our job as well.
print "\nListing active jobs: "
for job in js.list():
    print " * %s" % job

# Now we disconnect and reconnect to our job by using the get_job()
# method and our job's id. While this doesn't make a lot of sense
# here,  disconnect / reconnect can become very important for
# long-running job.

touchjob_clone = js.get_job(touchjob.id)

print touchjob_clone.state

while touchjob_clone.state == 'Pending':
    print "...Waiting for Job to start...."
    time.sleep(30)

nodename = touchjob_clone.execution_hosts[0]
nodename = nodename[:-2]

touchjob_clone.wait()

print "Job State   : %s" % (touchjob_clone.state)
print "Exitcode    : %s" % (touchjob_clone.exit_code)
js.close()

touchjob.id

js.get_job(touchjob.id)

touchjob_clone = js.get_job(touchjob.id)

print...


I am working on the file "02_tf_basics.ipynb".
The first task is:
Most of the times, you can use TensorFlow types and NumPy types interchangeably

Note 1 : There is a catch here for string data types. For numeric and boolean types, TensorFlow and NumPy dtypes match down the line. However, tf.string does not have an exact match in NumPy due to the way NumPy handles strings. TensorFlow can still import string arrays from NumPy perfectly fine -- just don’t specify a dtype in NumPy!

Note 2 : Both TensorFlow and NumPy are n-d array libraries. NumPy supports ndarray, but doesn’t offer methods to create tensor functions and automatically compute derivatives, nor GPU support. So TensorFlow still wins!

Note 3: Using Python types to specify TensorFlow objects is quick and easy, and it is useful for prototyping ideas. However, there is an important pitfall in doing it this way.  Python types lack  the ability to explicitly state the data type, but TensorFlow’s data types are more specific. For example, all integers are the same type, but TensorFlow has 8-bit, 16-bit, 32-bit, and 64-bit integers available.

Therefore, if you use a Python type, TensorFlow has to infer which data type you mean. It’s possible to convert the data into the appropriate type when you pass it into TensorFlow, but certain data types still may be difficult to declare correctly, such as complex numbers. Because of this, it is common to create hand-defined Tensor objects as NumPy arrays.

So, always use TensorFlow types whenever possible, because both TensorFlow and NumPy can evolve to a point that such compatibility no longer exists.

more on constants, variables
To declare a variable, you create an instance of the class tf.Variable. Note that it’s tf.constant but
tf.Variable and not tf.variable because tf.constant is an op, while tf.Variable is a class.
The main difference between a constant and a variable:
* A constant is constant. A variable can be assigned to, its value can be changed.
* A constant's value is stored in the graph and its value is replicated wherever the graph is
loaded. A variable is...
Can you write Python code for it?

# We can create a tf.Variable w as 784 x 10 tensor filled with zeros

w = tf.Variable(tf.zeros([784,10]))
print("shape of w", w.shape)


I am working on the file "02_tf_basics.ipynb".
The first task is:
most of the operations are same as numpy
Can you write Python code for it?

zer = tf.zeros([2,3], tf.int32)
print(sess.run(zer))
print("shape", zer.shape)

# we can get tensors of zeros / ones like another tensor
input_tensor = [[0,], [1], [2], [3]]
zer_like = tf.zeros_like(input_tensor, dtype=None, name=None, optimize=True)
print("zero_like", sess.run(zer_like))
# same goes for  tf.ones_like()

# we can create a tensor filled with user given values using tf.fill(dims,valu,name=None)
print(sess.run(tf.fill([3,4], 8.0)))

# You can create sequences of evenly spaced values begining at start ending at stop
# tf.linspace(start, stop, num, name=None)
linsp = tf.linspace(10.0, 13.0, 4, name='linsp')
print("linspace    ", sess.run(linsp))

# tf.range(start, limit=None, delta=1, dtype=None, name='range')
# stop number is no included
print("range     ", sess.run(tf.range(3, 18, 3)))
print("-----------")
print("NOTE: important thing to note here is tensorflow sequences are not iterable(like in numpy)")
#for _ in np.linspace(0,10,4) # works fine
#for _ in tf.linspace(0,10,4) # TypeError("'Tensor' object is not iterable.")

#for _ in range(4) # OK
#for _ in tf.range(4) # TypeError("'Tensor' object is not iterable.")


I am working on the file "02_tf_basics.ipynb".
The first task is:
tf.Variable holds several ops:
x = tf.Variable(...) ==>
x.initializer # init
x.value() # read op
x.assign(...) # write op
x.assign_add(...)
.....and more
Can you write Python code for it?

# we need to initialize the variables before using them

init = tf.global_variables_initializer()
with tf.Session() as sess:
   sess.run(init)

# Note that you use sess.run() to run the initializer, not fetching any value.

# To initialize only a subset of variables, you use tf.variables_initializer() with a list of variables you want to initialize:
a = tf.Variable(1, dtype=tf.int32)
b = tf.Variable(0.5, dtype=tf.float32)
c = tf.Variable(10.3, dtype=tf.float64)

init = tf.variables_initializer([a,b], name="init")

with tf.Session() as sess:
    sess.run(init)
    print("a ",sess.run(a))
    print("b ",sess.run(b))
    # print("c ",sess.run(c)) # throws error "Attempting to use uninitialized value Variable"

# You can also initialize each variable separately using  tf.Variable.initializer

W = tf.Variable(tf.zeros ([ 784 , 10 ])) #create variable W as 784 x 10 tensor, filled with zeros
with tf.Session() as sess:
    sess.run(W.initializer)
    print("shape of w  ", w.shape)

# to evaluate a variable we use .eval() method
x = tf.Variable(tf.truncated_normal([4,2]))

with tf.Session() as sess:
    sess.run(x.initializer)
    print("not evaluating correctly ", x) # wont print the value of x, rather it will print Tensor("Variable/read:0",shape =(700,10),dtype=float32)
    print("evaluating correctly ",x.eval()) # will print the value of variable x

# assign values to variables-->the incorrect way:

w = tf.Variable(10)
w.assign(100)

with tf.Session() as sess:
    sess.run(w.initializer)
    print("should print new value 100, but printed old value ",w.eval()) #prints 10 not 100 why!!!!!

    # w.assign(100) doesn't assign the value 100 to w, but instead it creates an assign op to do that.
    # we have to run this assign op in session

    # assign values to variables-->the correct way:

    w = tf.Variable(10)
    assign_op = w.assign(100)
    sess.run(assign_op)
    #dont need a sess.run(w.initilizer) here, assign() will do that for us.
    #infact, initilizer op is the assign op which assigns the...


I am working on the file "02a_fwi.ipynb".
The first task is:
Acquisition geometry
In this tutorial, we will use the easiest case for inversion, namely a transmission experiment. The sources are located on one side of the model and the receivers on the other side. This allow to record most of the information necessary for inversion, as reflections usually lead to poor inversion results.
Can you write Python code for it?

#NBVAL_IGNORE_OUTPUT
# Define acquisition geometry: source
from examples.seismic import RickerSource, Receiver

# Define time discretization according to grid spacing
t0 = 0.
tn = 1000.  # Simulation lasts 1 second (1000 ms)
dt = model.critical_dt  # Time step from model grid spacing
nt = int(1 + (tn-t0) / dt)  # Discrete time axis length
time = np.linspace(t0, tn, nt)  # Discrete modelling time

f0 = 0.010  # Source peak frequency is 10Hz (0.010 kHz)
src = RickerSource(name='src', grid=model.grid, f0=f0, time=np.linspace(t0, tn, nt))
src.coordinates.data[0, :] = np.array(model.domain_size) * .5
src.coordinates.data[0, 0] = 20.  # 20m from the left end

# We can plot the time signature to see the wavelet
src.show()

#NBVAL_IGNORE_OUTPUT
# Define acquisition geometry: receivers

# Initialize receivers for synthetic data
rec = Receiver(name='rec', grid=model.grid, npoint=nreceivers, ntime=nt)
rec.coordinates.data[:, 1] = np.linspace(0, model.domain_size[0], num=nreceivers)
rec.coordinates.data[:, 0] = 980. # 20m from the right end

# Plot acquisition geometry
plot_velocity(model, source=src.coordinates.data,
              receiver=rec.coordinates.data[::4, :])


I am working on the file "02a_fwi.ipynb".
The first task is:
Computational considerations
As we will see in this tutorial, FWI is again very computationally demanding, even more so than RTM. To keep this tutorial as light-wight as possible we therefore again use a very small demonstration model. We also define here a few parameters for the final example runs that can be changed to modify the overall runtime of the tutorial.
Can you write Python code for it?

nshots = 9  # Number of shots to create gradient from
nreceivers = 101  # Number of receiver locations per shot
fwi_iterations = 8  # Number of outer FWI iterations


I am working on the file "02a_fwi.ipynb".
The first task is:
True and smooth velocity models
As before, we will again use a very simple model domain, consisting of a circle within a 2D domain. We will again use the "true" model to generate our synthetic shot data and use a "smooth" model as our initial guess. In this case the smooth model is very smooth indeed - it is simply a constant background velocity without any features.
Can you write Python code for it?

#NBVAL_IGNORE_OUTPUT
from examples.seismic import demo_model, plot_velocity, plot_perturbation

# Define true and initial model
shape = (101, 101)  # Number of grid point (nx, nz)
spacing = (10., 10.)  # Grid spacing in m. The domain size is now 1km by 1km
origin = (0., 0.)  # Need origin to define relative source and receiver locations

model = demo_model('circle-isotropic', vp=3.0, vp_background=2.5,
                    origin=origin, shape=shape, spacing=spacing, nbpml=40)

model0 = demo_model('circle-isotropic', vp=2.5, vp_background=2.5,
                     origin=origin, shape=shape, spacing=spacing, nbpml=40)

plot_velocity(model)
plot_velocity(model0)
plot_perturbation(model0, model)


I am working on the file "02a_zaporedja.ipynb".
The first task is:
>> naprej: vrste
Can you write Python code for it?

import disqus
%reload_ext disqus
%disqus matpy


I am working on the file "02a_zaporedja.ipynb".
The first task is:
Primer
Zaporedje za računanje $\sqrt{2}$ je dano z rekurzivno formulo
$$a_{n+1}=\frac{1}{2}\left(a_n+\frac{2}{a_n}\right).$$
Izračunaj $\sqrt{2}$ na 12 decimalk natančno.
Can you write Python code for it?

koren2 = gen_rekurzijo(2,lambda x:(x+2/x)/2)
koren2(1)
koren2(3)

n = 0
while abs(koren2(n)**2-2)>1e-12:
    n += 1
print("n: %d, koren: %1.12f" % (n,koren2(n)))


I am working on the file "02a_zaporedja.ipynb".
The first task is:
Prvih nekaj členov nam razkrije, da je zaporedje lihih členov $a_{2n+1}$ padajoče, zaporedje sodih členov $a_{2n}$ pa naraščajoče. Da bi to preverili/dokazali, zapišimo rekurzivno enačbo, ki ji zadoščata podzaporedji $a_{2n+1}$ in $a_{2n}$.
Rekurzivna enačba
$$a_{n+1}=f(a_n)=1+\frac{1}{a_n}$$
je določena s funkcijo $f(x)=1+\frac{1}{x}$. Za zaporedji sodih in lihih členov $a_{2n}$ in $a_{2n+1}$ dobimo rekurzivni formuli, če dvakrat uporabimo funkcijo $f$
$$a_{2(n+1)}=f(a_{2n+1})=f(f(a_{2n}))$$
in
$$a_{2(n+1)+1} = f(f(a_{2n+1}))$$
Can you write Python code for it?

import sympy as sym
sym.init_printing()
# zaporedje je podano s funkcijo f(x) = 1+1/x
f = lambda x: 1+1/x
n = sym.Symbol('n')
a_n = sym.Symbol('a_n')
f(f(a_n))

sym.simplify(f(f(a_n)))


I am working on the file "02a_zaporedja.ipynb".
The first task is:
Rekurzivna zaporedja
Zaporedje lahko podamo tudi z rekurzivno formulo
$$a_{n+1}=f(a_n)$$
in prvim členom
$a_0$.
Primer
Zaporedje je dano z rekurzivno formulo
$$a_{n+1}=1+\frac{1}{a_n}$$
in začetnim členom $a_0=\frac{1}{2}$.

Nariši nekaj členov zaporedja
Pokaži, da sta zaporedji $a_{2n}$ in $a_{2n+1}$ monotoni in omejeni.
Poišči limito.
Can you write Python code for it?

# narišemo nekaj prvih členov
n0 = 20
a = n0*[0]
a[0] = 1/2
for n in range(n0-1):
    a[n+1] = 1+1./a[n]
print("Prvih %d členov zaporedja\n" % n0)
print((n0*"%0.8f\n")%tuple(a))

from matplotlib import pyplot as plt
%matplotlib inline
plt.plot(range(n0),a,'o')
plt.title("Členi zaporedja $a_n$")
plt.show()


I am working on the file "02a_zaporedja.ipynb".
The first task is:
Rešitvi sta nam že poznani iz predhodne analize. Limita je očitno enaka prvi rešitvi $$\frac{1}{2}+\frac{\sqrt{5}}{2},$$
saj je $a_n>0$.
Can you write Python code for it?

resitve[0].evalf() # numerični približek za limito se ujema s prej izračunanimi členi zaporedja


I am working on the file "02a_zaporedja.ipynb".
The first task is:
Zaporedje sodih členov $a_{2n}$ bo padajoče, če so členi
$$a_{2n}>\frac{1}{2}+\frac{\sqrt{5}}{2}$$
in naraščajoče, če so členi
$$a_{2n}<\frac{1}{2}+\frac{\sqrt{5}}{2}.$$
Podobno velja za zaporedje lihih členov $a_{2n+1}$.
Can you write Python code for it?

neenacba = ff(a_2n)<1/sym.Integer(2)+sym.sqrt(sym.Integer(5))/2
neenacba

sym.solve_univariate_inequality(neenacba,a_2n)


I am working on the file "02a_zaporedja.ipynb".
The first task is:
Zaporedji lihih členov in sodih členov zadoščata rekurzivni formuli
$$a_{n+2}=\frac{2 a_{n} + 1}{a_{n} + 1}.$$
Zaporedje $a_{2n}$ bo naraščalo, če bo veljalo
$$a_{2n+2} > a_{2n}$$
oziroma
$$\frac{2 a_{2n} + 1}{a_{2n} + 1} > a_{2n}.$$
Can you write Python code for it?

ff = lambda x: f(f(x))
a_2n = sym.Symbol('a_2n')
sym.simplify(ff(a_2n) > a_2n)

sym.solve_univariate_inequality(ff(a_2n)>a_2n,a_2n)


I am working on the file "02b_scipy_optimize.ipynb".
The first task is:
FWI with L-BFGS-B
Equipped with a function to calculate the functional and gradient, we are finally ready to call scipy.optimize.minimize.
Can you write Python code for it?

#NBVAL_SKIP

# Change to the WARNING log level to reduce log output
# as compared to the default DEBUG
from devito import configuration
configuration['log_level'] = 'WARNING'

# Set up a dictionary of inversion parameters.
param = {'t0': 0.,
         'tn': 1000.,              # Simulation lasts 1 second (1000 ms)
         'f0': 0.010,              # Source peak frequency is 10Hz (0.010 kHz)
         'nshots': 9}              # Number of shots to create gradient from

# Define bounding box constraints on the solution.
def apply_box_constraint(m):
    # Maximum possible 'realistic' velocity is 3.5 km/sec
    # Minimum possible 'realistic' velocity is 2 km/sec
    return np.clip(m, 1/3.5**2, 1/2**2)

# Many optimization methods in scipy.optimize.minimize accept a callback
# function that can operate on the solution after every iteration. Here
# we use this to apply box constraints and to monitor the true relative
# solution error.
relative_error = []
def fwi_callbacks(x):
    # Apply boundary constraint
    x.data[:] = apply_box_constraint(x)

    # Calculate true relative error
    true_x = get_true_model().m.data.flatten()
    relative_error.append(np.linalg.norm((x-true_x)/true_x))

# Initialize solution
model0 = get_initial_model()

# Finally, calling the minimizing function. We are limiting the maximum number
# of iterations here to 10 so that it runs quickly for the purpose of the
# tutorial.
from scipy import optimize
result = optimize.minimize(fwi_gradient, model0.m.data.flatten().astype(np.float64),
                           args=(param, ), method='L-BFGS-B', jac=True,
                           callback=fwi_callbacks,
                           options={'maxiter':10, 'disp':True})

# Print out results of optimizer.
print(result)

#NBVAL_SKIP

# Show what the update does to the model
from examples.seismic import plot_image, plot_velocity

model0.m.data[:] = result.x.astype(np.float32).reshape(model0.m.data.shape)
model0.vp = np.sqrt(1. / model0.m.data[40:-40,...


I am working on the file "02b_vrste.ipynb".
The first task is:
Metoda args() vrne n-terico(tuple) posameznih ulomkov. Če namesto $n$ vstavimo prvih nekaj števil, lahko vidimo, kateri členi se pokrajšajo.
Can you write Python code for it?

args = ul.args
s = []
for i in range(5):
    s=s + [arg.subs(n,i+1) for arg in args]
s = s + [arg.subs(n,n-1) for arg in args] + list(args)
s


I am working on the file "02b_vrste.ipynb".
The first task is:
Postopek „na roke“
Poglejmo si, kako bi vsoto izračunali „na roke“. Člen vrste
$$\frac{1}{n(n+2)}$$
razstavimo na parcialne ulomke in upamo, da se nam vmesni členi pokrajšajo.
Can you write Python code for it?


I am working on the file "02b_vrste.ipynb".
The first task is:
Vidimo, da se vmesni členi pokrajšajo, razen 1. in 3. ter zadnjega in predpredzadnjega. Delna vsota je torej enaka
Can you write Python code for it?

Sn = s[0]+s[2]+s[-3]+s[-1]
Sn

Sn.together().factor()

limit(Sn,n,oo) #vsota vrste je limita delnih vsot


I am working on the file "02b_vrste.ipynb".
The first task is:
Vsoto vrste in delne vsote lahko izračunamo s funkcijo sympy.Sum. Za zgornjo mejo neskončno $\infty$ uporabimo znak sympy.oo.
Can you write Python code for it?

Sum(a(n),(n,1,oo)).doit() # vsota vrste

k = Symbol('k')
Sum(a(k),(k,1,n)).doit() # Funkcija sympy.Sum izračuna tudi delne vsote


I am working on the file "02c_dask.ipynb".
The first task is:
Dask specifics
Previously we defined a function to calculate the individual contribution to the functional and gradient for each shot, which was then used in a loop over all shots. However, when using distributed frameworks such as Dask we instead think in terms of creating a worklist which gets mapped onto the worker pool. The sum reduction is also performed in parallel. For now however we assume that the scipy.optimize.minimize itself is running on the master process; this is a reasonable simplification because the computational cost of calculating (f, g) far exceeds the other compute costs.
Because we want to be able to use standard reduction operators such as sum on (f, g) we first define it as a type so that we can define the __add__ (and __rand__ method).
Can you write Python code for it?

# Define a type to store the functional and gradient.
class fg_pair:
    def __init__(self, f, g):
        self.f = f
        self.g = g

    def __add__(self, other):
        f = self.f + other.f
        g = self.g + other.g

        return fg_pair(f, g)

    def __radd__(self, other):
        if other == 0:
            return self
        else:
            return self.__add__(other)


I am working on the file "02c_dask.ipynb".
The first task is:
Define the global functional-gradient operator. This does the following:
* Creates the Dask cluster
* Maps the worklist (shots) to the workers so that the invidual contributions to (f, g) are computed.
* Sum individual contributions to (f, g) and returns the result.
Can you write Python code for it?

import numpy as np
from distributed import Client

# Dumps the model to disk; workers will pick this up when they need it.
def dump_model(param, model):
    np.save(param['model'], model.astype(np.float32))

def fwi_gradient(model, param):
    # Dump a copy of the current model for the workers
    # to pick up when they are ready.
    param['model'] = "model_0.npy"
    dump_model(param, model)

    # Define work list
    work = [dict(param) for i in range(param['nshots'])]
    for i in  range(param['nshots']):
        work[i]['shot_id'] = i

    # Start dask cluster
    client = Client()

    # Distribute worklist to workers.
    fgi = client.map(fwi_gradient_i, work)

    # Perform reduction.
    fg = client.submit(sum, fgi).result()

    # Shutdown cluster
    client.close()

    # L-BFGS in scipy expects a flat array in 64-bit floats.
    return fg.f, fg.g.flatten().astype(np.float64)


I am working on the file "02c_dask.ipynb".
The first task is:
We now apply our FWI function and have a look at the result.
Can you write Python code for it?

#NBVAL_SKIP

# Change to the WARNING log level to reduce log output
# as compared to the default DEBUG
from devito import configuration
configuration['log_level'] = 'WARNING'

# Set up inversion parameters. This time we are using
# more shots because we are running in parallel.
param = {'t0': 0.,
         'tn': 1000.,           # Simulation lasts 1 second (1000 ms)
         'f0': 0.010,           # Source peak frequency is 10Hz (0.010 kHz)
         'nshots': 16}          # Number of shots to create gradient from

model0 = get_initial_model()

# Baby steps
result = fwi(model0, param)

# Print out results of optimizer.
print(result)

#NBVAL_SKIP

# Show what the update does to the model
from examples.seismic import plot_image, plot_velocity

model0.m.data[:] = result.x.astype(np.float32).reshape(model0.m.data.shape)
model0.vp = np.sqrt(1. / model0.m.data[40:-40, 40:-40])
plot_velocity(model0)

#NBVAL_SKIP

# Plot percentage error
plot_image(100*np.abs(model0.vp-get_true_model().vp.data)/get_true_model().vp.data, vmax=15, cmap="hot")

#NBVAL_SKIP
import matplotlib.pyplot as plt

# Plot objective function decrease
plt.figure()
plt.loglog(relative_error)
plt.xlabel('Iteration number')
plt.ylabel('True relative error')
plt.title('Convergence')
plt.show()


I am working on the file "03 - Cross-validation.ipynb".
The first task is:
Cross-Validation
Can you write Python code for it?

from sklearn.datasets import load_iris

iris = load_iris()
X = iris.data
y = iris.target

from sklearn.cross_validation import cross_val_score
from sklearn.svm import LinearSVC

cross_val_score(LinearSVC(), X, y, cv=5)

cross_val_score(LinearSVC(), X, y, cv=5, scoring="f1_macro")


I am working on the file "03 - Cross-validation.ipynb".
The first task is:
Exercises
Use  KFold cross validation and StratifiedKFold cross validation (3 or 5 folds) for LinearSVC on the iris dataset.
Why are the results so different? How could you get more similar results?
Can you write Python code for it?


I am working on the file "03 - Cross-validation.ipynb".
The first task is:
Let's go to a binary task for a moment
Can you write Python code for it?

y % 2

cross_val_score(LinearSVC(), X, y % 2)

cross_val_score(LinearSVC(), X, y % 2, scoring="average_precision")

cross_val_score(LinearSVC(), X, y % 2, scoring="roc_auc")

from sklearn.metrics.scorer import SCORERS
print(SCORERS.keys())