I am working on the file "BString-Copy1.ipynb".
The first task is:
3) Atomic Function (AF) is a 'finite' function (like spline) not equal to zero only on section |x|<=1
4) Atomic Function is a non-analytical function (can not be represented by Taylor's series), but with known Fourier Transformation allowing to exactly calculate AF in certain points, with tabular representation provided in script above.
Atomic String is a generalisation of an Atomic Function
1) Astring is an swing-like function - Integral of Atomic Function (AF) which can be expressed via AF itself:
AString(x) = Integral(0,x)(Up(x)) = Up(x/2 - 1/2) - 1/2
2) Atomic Function can be represented via simple superposition of Atomic Strings:
up(x) = AString(2x + 1) - AString(2x - 1)
Can you write Python code for it?

#########   Presentation of Atomic Function via Atomic Strings ##########
x = np.arange(-2.0, 2.0, 0.01)

pl.plot(x, AString(x, 1, 0, 1, 0),  '--', linewidth=1,  label='AString(x)')
pl.plot(x, AString(x, 0.5, -0.5, +1, 0), '--', linewidth=2, label='+AString(2x+1)')
pl.plot(x, AString(x, 0.5, +0.5, -1, 0), '--', linewidth=2, label='-AString(2x-1)')
#pl.plot(x, up(x, 1.0, 0, 1, 0),     '--', linewidth=1, label='Atomic Function')
AS2 = Sum(AString(x, 0.5, -0.5, +1, 0), AString(x, 0.5, +0.5, -1, 0))
pl.plot(x, AS2, linewidth=3, label='Up(x) via Strings')
pl.title('Atomic Function as a Combination of AStrings')
pl.legend(loc='center left', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "BString-Copy1.ipynb".
The first task is:
Atomic String, Atomic Function and Atomic Function Derivative plotted together
Can you write Python code for it?

pl.plot(x, String(AString1, x, 1.0, 0, 1, 0), label='Atomic String')
pl.plot(x, String(BString1, x, 1.0, 0, 1, 0), label='B String')

pl.title('Atomic and B String')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()

#This Calculates Derivative
dx = x[1] - x[0]
dydx = np.gradient(up(up1, x), dx)
pl.plot(x, up(up1, x), label='Atomic Function')
pl.plot(x, String(AString1, x, 1.0, 0, 1, 0), label='Atomic String')
pl.plot(x, dydx, label='A-Function Derivative')

pl.title('Atomic Function and String')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()

# BUP Function = BString(2x+1) - BString(2x-1)
def bup1(x : float) -> float:
    #res = String(BString1, x, 0.5, -0.5, 1, 0) - String(BString1, x, 0.5, +0.5, 1, 0)
    res = BString1(2.*x+1) - BString1(2.*x-1)
    return res

bup1(0.5)

#This Calculates Derivative
dx = x[1] - x[0]
dydx = np.gradient(up(bup1, x), dx)
pl.plot(x, up(bup1, x), label='BUP Atomic Function')
pl.plot(x, String(BString1, x, 1.0, 0, 1, 0), label='BUP Atomic String')
pl.plot(x, dydx, label='BUP-Function Derivative')

pl.title('BUP Atomic Function and String')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "BString-Copy1.ipynb".
The first task is:
Properties of atomic function Up(x)
1) Remarkably, Atomic Function Derivative can be exressed via Atomic Function itself - up'(x)= 2up(2x+1)-2up(2x-1) meaning the shape of pulses for derivative function can be represented by shifted and stratched Atomic Function itself - remarkable property
2) The Atomic Function pulses superposition set at points -2, -1, 0, +1, +2... can exactly represent a Unity (number 1):
1 = ... up(x-3) + up(x-2) + up(x-1) + up(x-0) + up(x+1) + up(x+2) + up(x+3) + ...
Can you write Python code for it?

pl.plot(x, up(up1, x, 1, -1), linewidth=1, label='Atomic Function at x=-1')
pl.plot(x, up(up1, x, 1, +0), linewidth=1, label='Atomic Function at x=0')
pl.plot(x, up(up1, x, 1, -1), linewidth=1, label='Atomic Function at x=-1')
pl.plot(x, Sum(up(up1, x, 1, -1), Sum(up(up1, x), up(up1, x, 1, 1))), linewidth=2, label='Atomic Function Compounding')
pl.title('Atomic Function Compounding represent 1')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "BString-Copy1.ipynb".
The first task is:
Representing of flat Spacetime Fabric by joining of Atomic Strings Quanta (Metriants)
Can you write Python code for it?

x = np.arange(-30.0, 30.0, 0.01)
#pl.plot(x, ABline (x, 1, 0),            label='ABLine 1*x')
pl.plot(x, AString(x, 10.0,-15, 10, -15),   '--',  linewidth=1, label='AString Quantum 1')
pl.plot(x, AString(x, 10.0, -5, 10, -5),    '--',  linewidth=1, label='AString Quantum 2')
pl.plot(x, AString(x, 10.0, +5, 10, +5),    '--',  linewidth=1, label='AString Quantum 3')
pl.plot(x, AString(x, 10.0,+15, 10, +15),   '--',  linewidth=1, label='AString Quantum 4')

AS2 = Sum(AString(x, 10.0, -15, 10, -15), AString(x, 10., -5, 10, -5))
AS3 = Sum(AS2, AString(x, 10, +5, 10, +5))
AS4 = Sum(AS3, AString(x, 10,+15, 10, +15))
pl.plot(x, AS4,  label='Spacetime Dimension', linewidth=2)

pl.title('Representing Spacetime by joining of Atomic Strings')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "BString-Copy1.ipynb".
The first task is:
Schematic model of Gravitation explaining General Relativity effects where spacetime Shape, Density and Curvature are deeply related being expressed via the same AString or Atomic Function
Can you write Python code for it?

x = np.arange(-50.0, 50.0, 0.1)
dx = x[1] - x[0]

CS6 = Sum(up(x, 5, -30, 5, 5), up(x, 15, 0, 15, 5))
CS6 = Sum(CS6, up(x, 10, +30, 10, 5))
pl.plot(x, CS6,  label='Spacetime Density')

IntC6 = np.cumsum(CS6)*dx/50
pl.plot(x, IntC6, label='Spacetime Shape')

DerC6 = np.gradient(CS6, dx)
pl.plot(x, DerC6, label='Spacetime Curvature')

LightTrajectory = -10 -IntC6/5
pl.plot(x, LightTrajectory, label='Light Trajectory')

pl.title('Fabric of Curved Spacetime')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "BString.ipynb".
The first task is:
Apart from standard Python code, this script and material is the intellectual property of Professor Sergei Yu. Eremenko (https://au.linkedin.com/in/sergei-eremenko-3862079). You may not reproduce, edit, translate, distribute, publish or host this document in any way without the permission of Professor Eremenko.
Integration of BStrings
Can you write Python code for it?

def integrate(a, b, N = 100):
    t = np.linspace(a + (b-a)/(2*N), b- (b-a)/(2*N), N) # Central Points of subintervals
    # fx = f(x)
    fx = []
    for i in range(len(t)):
        # fx.append(pulse(up1, t[i]))
        fx.append(pulse(bup1, t[i]))
        #print (i, t[i], fx[i])
    area = np.sum(fx) * (b-a)/N
    return area

integrate(-1, 1)


I am working on the file "BString.ipynb".
The first task is:
Properties of BUP atomic function BUp(x)
Can you write Python code for it?

pl.plot(x, up(bup1, x, 1, -1), linewidth=1, label='B-Atomic Function at x=-1')
pl.plot(x, up(bup1, x, 1, +0), linewidth=1, label='B-Atomic Function at x=0')
pl.plot(x, up(bup1, x, 1, -1), linewidth=1, label='b-Atomic Function at x=-1')
pl.plot(x, Sum(up(bup1, x, 1, -1), Sum(up(bup1, x), up(bup1, x, 1, 1))), linewidth=2, label='Atomic Function Compounding')
pl.title('Atomic Function Compounding represent 1')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()

### Integration of BUP
CS6 = Sum(up(bup1, x, 1, -1), Sum(up(bup1, x), up(bup1, x, 1, 1)))
pl.plot(x, CS6,  label='BUP Spacetime Density')
pl.legend(loc='best', numpoints=1)
pl.grid(True)

IntC6 = np.cumsum(CS6)*dx/50
pl.plot(x, IntC6, label='Spacetime Shape')
pl.legend(loc='best', numpoints=1)
pl.grid(True)

#pl.plot(x, ABline (x, 1, 0),            label='ABLine 1*x')
pl.plot(x, String(BString1, x, 1.0, -1.5, 1, -1.5), '--',  linewidth=1, label='BString 1')
pl.plot(x, String(BString1, x, 1.0, -0.5, 1, -0.5), '--',  linewidth=1, label='BString 2')
pl.plot(x, String(BString1, x, 1.0, +0.5, 1, +0.5), '--',  linewidth=1, label='BString 3')
pl.plot(x, String(BString1, x, 1.0, +1.5, 1, +1.5), '--',  linewidth=1, label='BString 4')

AS2 = Sum(String(BString1, x, 1.0, -1.5, 1,  -1.5), String(BString1, x, 1.0, -0.5, 1.0, -0.5))
AS3 = Sum(AS2, String(BString1, x, 1, 0.5, 1, +0.5))
AS4 = Sum(AS3, String(BString1, x, 1,+1.5, 1, +1.5))
pl.plot(x, AS4, label='BStrings Joins', linewidth=2)

pl.title('BUP Atomic Strings Combinations')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "BString.ipynb".
The first task is:
Schematic model of Gravitation explaining General Relativity effects where spacetime Shape, Density and Curvature are deeply related being expressed via the same AString or Atomic Function
Can you write Python code for it?

x = np.arange(-50.0, 50.0, 0.1)
dx = x[1] - x[0]

CS6 = Sum(up(up1, x, 5, -30, 5, 5), up(up1, x, 15, 0, 15, 5))
CS6 = Sum(CS6, up(up1, x, 10, +30, 10, 5))
pl.plot(x, CS6,  label='Spacetime Density')

IntC6 = np.cumsum(CS6)*dx/50
pl.plot(x, IntC6, label='Spacetime Shape')

DerC6 = np.gradient(CS6, dx)
pl.plot(x, DerC6, label='Spacetime Curvature')

LightTrajectory = -10 -IntC6/5
pl.plot(x, LightTrajectory, label='Light Trajectory')

pl.title('Fabric of Curved Spacetime')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()

x = np.arange(-50.0, 50.0, 0.1)
dx = x[1] - x[0]

CS6 = Sum(up(bup1, x, 5, -30, 5, 5), up(bup1, x, 15, 0, 15, 5))
CS6 = Sum(CS6, up(bup1, x, 10, +30, 10, 5))
pl.plot(x, CS6,  label='Spacetime Density')

IntC6 = np.cumsum(CS6)*dx/50
pl.plot(x, IntC6, label='Spacetime Shape')

DerC6 = np.gradient(CS6, dx)
pl.plot(x, DerC6, label='Spacetime Curvature')

LightTrajectory = -10 -IntC6/5
pl.plot(x, LightTrajectory, label='Light Trajectory')

pl.title('Fabric of Curved Spacetime')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "BTC trade strategy utilities demo.ipynb".
The first task is:
Customize and test a trading strategy
You can run a custom made bot trading strategy and evaluate its performance.
inside trade_strategies.py you can find a template for trade strategy creation that will fit with the rest of the code.
Can you write Python code for it?

from trade_strategies import hindsight_trade_strategy
# This is dummy strategy, it gives ideal buy/sell decision based on future behavior,can only work on old data that was collected
# (or if you know how to see into the future)
df = hindsight_trade_strategy(df, start=1, sensitivity=0.7)

from plotting import plot_trade_strategy
plot_trade_strategy(df)


I am working on the file "BTC trade strategy utilities demo.ipynb".
The first task is:
Load the data
You can find historical bitcoin data in here: https://www.kaggle.com/mczielinski/bitcoin-historical-data/data
please note that currently the code was tested only on Bitstamp data.
for the this demo we will use a small sample of data.
Can you write Python code for it?

from data_handling import get_OHLC_data_from_file
directory_path = 'data'
file_name = 'OHLC_sample.csv'

df = get_OHLC_data_from_file(directory_path, file_name)

df.head(5)


I am working on the file "BackProp-example.ipynb".
The first task is:
Main code
Can you write Python code for it?

# specify the model. We use numpy.matrix as numpy 1d array cannot do transpose()
A0 = np.matrix([0.35, 0.9])
W1 = np.matrix([[0.1, 0.4], [0.8, 0.6]])
W2 = np.array([[0.3], [0.9]])
t = np.array([0.5])

# forward pass
Z1 = np.dot(A0, W1)
A1 = acti(Z1)
Z2 = np.dot(A1, W2)
A2 = acti(Z2)
y = A2

# print
p('Z1, A1 = ', Z1, A1)
p('Z2, A2 = ', Z2, A2)
p('loss=', loss(y, t))


I am working on the file "BackProp-example.ipynb".
The first task is:
PyTorch implementation
Now we use PyTorch's autograd to verify our computation above.
Can you write Python code for it?

tT = torch.tensor(t, requires_grad=False) # ground truth
tx = torch.tensor(A0, requires_grad=False) # input x
tW1= torch.tensor(W1, requires_grad=True)
tW2 = torch.tensor(W2, requires_grad=True)

igma = torch.sigmoid
# our network (and its output Y for input X)
Z1 = tx.mm(tW1)
A1 = sigma(Z1)
Z2 = A1.mm(tW2)
A2 = sigma(Z2)
Y = A2
loss2 = (Y-tT).pow(2).sum()/2.0 # just sum all
print(loss2.data)

loss2.backward()
print(tW1.grad)
print(tW2.grad)


I am working on the file "BackProp-simple.ipynb".
The first task is:
Note that if we set requires_grad=True for a Tensor, then any PyTorch operations on that Tensor will cause a computational graph to be constructed, allowing us to later perform backpropagation. If x is a Tensor with requires_grad=True, then after backpropagation x.grad will be another Tensor holding the gradient of x with respect to some scalar value.
Can you write Python code for it?

# our network (and its output Y for input X)
Z = X.mm(W)
Y = Z

#loss1 = torch.nn.MSELoss(size_average=False)(Y, T) # non-averaged version
loss1 = torch.nn.MSELoss()(Y, T) # averaged version
loss2 = (Y-T).pow(2).sum()/2.0 # just sum all

# the two loss values should differ by a factor of n.
print(loss1)
print(loss2)

loss2.backward()
#print(Z.grad)
print(W.grad)
print(X.grad)


I am working on the file "BackProp-simple.ipynb".
The first task is:
Now compare with the manually computed gradient:
$$ \frac{\partial l}{\partial W} =  \frac{\partial l}{\partial Y} \frac{\partial Y}{\partial Z} \frac{\partial Y}{\partial W} = X^{\top} (Y \circ (1-Y) \circ (Y-T)) $$
where $\circ$ is the Hadamard product (https://en.wikipedia.org/wiki/Hadamard_product_(matrices))
Can you write Python code for it?

gradient_w = torch.t(X).mm(Y * (1-Y) * (Y-T))
gradient_w


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Compare loss and accuracy of keras and numpy
Can you write Python code for it?


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Dataset
The dataset we used is composed of 8x8 images pixel of hand written digits available within sklearn library.

sklearn.datasets.load_digits
Can you write Python code for it?

from sklearn.datasets import load_digits
digits = load_digits()
N = reduce(lambda x,y: x*y,digits.images[0].shape)
print("Image dimension : N=%d"%N)
K = len(set(digits.target))
print("Number of classes : K=%d"%K)


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Display the sigmoid function and its derivative
Can you write Python code for it?

fig = plt.figure(figsize=(12,6))
ax = fig.add_subplot(1,1,1)
x = np.linspace(-5, 5, 100)
ax.plot(x, sigmoid(x), label='sigmoid')
ax.plot(x, dsigmoid(x), label='dsigmoid')
ax.legend(loc='best');


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Evaluate the model without training
Can you write Python code for it?

# Init the model
lr = LogisticRegression(N, K)

print("Evaluation of the untrained model:")
train_loss = lr.loss(X_train, y_train)
train_acc = lr.accuracy(X_train, y_train)
test_acc = lr.accuracy(X_test, y_test)

print("train loss: %0.4f, train acc: %0.3f, test acc: %0.3f"
      % (train_loss, train_acc, test_acc))

lr.W.shape


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Evaluate the randomly initialized model on the first example:
Can you write Python code for it?

def plot_prediction(model, sample_idx=0, classes=range(10)):
    fig, (ax0, ax1) = plt.subplots(nrows=1, ncols=2, figsize=(10, 4))

    ax0.imshow(scaler.inverse_transform(X_test[sample_idx]).reshape(8, 8), cmap=plt.cm.gray_r,
               interpolation='nearest')
    ax0.set_title("True image label: %d" % y_test[sample_idx]);
    ax0.grid(False)
    ax0.axis('off')

    ax1.bar(classes, one_hot(len(classes), y_test[sample_idx]), label='true')
    ax1.bar(classes, model.forward(X_test[sample_idx]), label='prediction', color="red")
    ax1.set_xticks(classes)
    prediction = model.predict(X_test[sample_idx])
    ax1.set_title('Output probabilities (prediction: %d)'
                  % prediction)
    ax1.set_xlabel('Digit class')
    ax1.legend()

plot_prediction(lr, sample_idx=0)


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Implement a function that produces the same results that the plot_prediction function but with keras model output
Can you write Python code for it?


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Loss evolution per epoch
Can you write Python code for it?

fig = plt.figure(figsize=(20,5))
ax = fig.add_subplot(1,2,1)
ax.plot(losses,label='train')
ax.plot(losses_test,label='test')
ax.set_xlabel("Epochs")
ax.set_ylabel("Loss")
ax.legend(loc='best');
ax.set_title("Training loss");
ax = fig.add_subplot(1,2,2)
ax.plot(accuracies, label='train')
ax.plot(accuracies_test, label='test')
ax.set_ylabel("accuracy")
ax.set_xlabel("Epochs")
ax.legend(loc='best');
ax.set_title("Accuracy");

plot_prediction(model, sample_idx=4)


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Loss function
We consider the loss function associated to the cross-entropy. Minimizing this loss function corresponds to minimization of the negative log likelihood  (which is equivalent to the  maximization of the log likelihood).
According to course's notations, we have
$$ \ell(f(x),y) = -\log (f(x))y = -  \sum{k=1}^K  \mathbb{1}_{y=k} \log (f(x))_k
$$
where $(f(x))_k =  \mathbb{P}(Y=k~/~x)$, the predicted probability for the class $k$, when the input equals $x$.
Exercice:
Write a function that computes the mean negative likelihood (empirical loss) of a group of observations Y_true and Fx, where Y_true and Fx are respectively the one-hot encoded representation of the observed labels and the predictions for the associated inputs $x$ i.e. :

Y_true is the one-hot encoded representation of $y$
Fx is the output of a softmax function.
Can you write Python code for it?

# Write here the negative_log_likelihood function
EPSILON = 1e-8
def NegLogLike(Y_true, Fx):
    ###
    return nll_mean

# %load solutions/negative_log_likelihood_function.py


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Make sure that your function works for a  2D array :
Can you write Python code for it?


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Make sure that your function works for a 1D array :
Can you write Python code for it?


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Make sure that your implementation can compute the  empirical loss for several observations.
Can you write Python code for it?


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Make sure that your implementation can compute the loss function  for a single observation.
Can you write Python code for it?

# Simple case
ohy_true = [1, 0, 0]
fx = [.99, 0.01, 0]
nll1 = NegLogLike(ohy_true, fx)
print("A small value for the loss function :")
print("Expected value : 0.01005032575249135")
print("Computed : " + str(nll1) )

# Case with bad prediction
ohy_true = [1, 0, 0]
fx = [0.01, .99, 0]
nll2 = NegLogLike(ohy_true, fx)
print("Higher value for the loss function):")
print("Expected value : 4.605169185988592")
print("Computed : " + str(nll2) )


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Make sure the solution works on 2D array :
Can you write Python code for it?


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Preprocessing

Normalization
Train / test split
Can you write Python code for it?

from sklearn import preprocessing
from sklearn.model_selection import train_test_split

data = np.asarray(digits.data, dtype='float32')
target = np.asarray(digits.target, dtype='int32')

X_train, X_test, y_train, y_test = train_test_split(
    data, target, test_size=0.15, random_state=37)

# mean = 0 ; standard deviation = 1.0
scaler = preprocessing.StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

print("Data dimension and type")
print("X_train : " + str(X_train.shape) + ", " +str(X_train.dtype))
print("y_train : " + str(y_train.shape) + ", " +str(y_train.dtype))
print("X_test : " + str(X_test.shape) + ", " +str(X_test.dtype))
print("y_test : " + str(y_test.shape) + ", " +str(y_test.dtype))


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Sigmoid Function

Implement the sigmoid and its element-wise derivative dsigmoid functions:

$$
sigmoid(x) = \frac{1}{1 + e^{-x}}
$$
$$
dsigmoid(x) = sigmoid(x) \cdot (1 - sigmoid(x))
$$
Can you write Python code for it?

def sigmoid(X):
    ###
    return sigX

def dsigmoid(X):
    ###
    return dsig

# %load solutions/sigmoid.py


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Tensorflow/keras
Implement the same multi layer perceptron using Keras
Can you write Python code for it?

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.optimizers import SGD
from tensorflow.keras.utils import to_categorical

n_features = 64
n_classes = 10
n_hidden = 10

# %load solutions/mlp_keras.py


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
The softmax function
$$
softmax(\mathbf{x}) = \frac{1}{\sum_{i=1}^{n}{e^{x_i}}}
\cdot
\begin{bmatrix}
  e^{x_1}\\
  e^{x_2}\\
  \vdots\\
  e^{x_n}
\end{bmatrix}
$$
Exercise : Implement the softmax function.
Ensure that your function works for several vectors at a time.
Hint : use the axis and keepdims argument of the numpy function np.sum.
Can you write Python code for it?

# keepdims option
x = np.array([[1,2,3],
              [4,5,6]])

print("Sum all elements of array :")
sx = np.sum(x)
print(sx)

print("Sum all elements over axis (dimension) :" )
sx = np.sum(x, axis=-1)
print(str(sx), str(", Dimension :") ,str(sx.shape))

print("Sum all elements over axis and with keepdims (dimension) :" )
sx = np.sum(x, axis=-1,  keepdims=True)
print(str(sx), str(", Dimension :") ,str(sx.shape))

# Write here the softmax function
def softmax(x):
    ###
    return softmaxX

# %load solutions/softmax.py


I am working on the file "BackPropagation/backpropagation.ipynb".
The first task is:
Train the model for one epoch
Can you write Python code for it?

learning_rate = 0.01

for i, (x, y) in enumerate(zip(X_train, y_train)):
    lr.train(x, y, learning_rate)
    if i % 100 == 0:
        train_loss = lr.loss(X_train, y_train)
        train_acc = lr.accuracy(X_train, y_train)
        test_acc = lr.accuracy(X_test, y_test)
        print("Update #%d, train loss: %0.4f, train acc: %0.3f, test acc: %0.3f"
              % (i, train_loss, train_acc, test_acc))


I am working on the file "Backpropagation.ipynb".
The first task is:
1. Introdução
A melhor maneira de pensar em redes neurais é como circuitos de valores reais. Mas, ao invés de valores booleanos, valores reais e, ao invés de portas lógicas como and ou or, portas binárias (dois operandos) como $$ (multiplicação), + (adição), max, exp, etc. Além disso, também teremos gradientes* fluindo pelo circuito, mas na direção oposta.
<img src='images/porta_multiplicacao.png' width="250">
De forma matemática, a gente pode considerar que essa porta implementa a seguinte função:
$$f(x,y)=x*y$$
O Objetivo
Vamos imaginar que temos o seguinte problema:
1. Nós vamos providenciar a um circuito valores específicos como entrada (x=-2, y=3)
2. O circuito vai calcular o valor de saída (-6)
3. A questão é: Quanto mudar a entrada para levemente aumentar a saída?
No nosso caso, em que direção devemos mudar x,y para conseguir um número maior que -6? Note que, pro nosso exemplo, se x = -1.99 e y = 2.99, x$$y = -5.95 que é maior que -6. -5.95 é melhor (maior) que 6*, e obtivemos uma melhora de 0.05.
Estratégia 1: Busca Aleatória
Ok. Isso não é trivial? A gente pode simplesmente gerar valores aleatórios, calcular a saída e guardar o melhor resultado.
Can you write Python code for it?

x, y = -2, 3
melhor_saida = forwardMultiplyGate(x,y)
melhor_x, melhor_y = 0, 0

for k in range(0,100):
    x_try = 5*np.random.random() - 5
    y_try = 5*np.random.random() - 5
    out = forwardMultiplyGate(x_try, y_try)

    if out > melhor_saida:
        melhor_saida = out
        melhor_x, melhor_y = x_try, y_try

print(melhor_x, melhor_y, forwardMultiplyGate(melhor_x, melhor_y))


I am working on the file "Backpropagation.ipynb".
The first task is:
<img src="images/circuito_2_back.png">
É isso! Vamos agora fazer nossas entradas responderem ao gradiente. Lembrando que queremos um valor maior que -12.
Vamos agora analisar os resultados separadamente. Analisando primeiramente $q$ e $z$, vemos que o circuito quer que $z$ aumente (der_f_rel_z = +3) e o valor de $q$ diminua (der_f_rel_q = -4) com uma força maior (4 contra 3).
Em relação a porta de soma, como vimos, o padrão é que aumentando as entradas a saída também aumenta. Porém, o circuito quer que $q$ diminua (der_f_rel_q = -4). Esse é o ponto crucial: em vez de aplicarmos uma força de +1 as entradas da porta de soma como normalmente faríamos (derivada local), o circuito quer que os gradientes em $x$ e $y$ se tornem 1x-4=-4. Isso faz sentido: o circuito quer $x$ e $y$ pequeno para que $q$ seja pequeno também, o que vai aumentar $f$.

Se isso fez sentido, você entendeu backpropagation.

Recapitulando:
- Vimos que, para uma simples porta (or simples expressão), podemos derivar o gradiente analítico usando cálculo simples. Nós interpretamos o gradiente como uma força que puxa as entradas na direção necessária para fazer a saída aumentar.
- No caso de múltiplas portas, cada porta é tratada individualmente até que o circuito seja tratado como um todo. A única diferença é que agora o circuito diz como a saída de outras portas devem se comportar (como da porta de adição), que é o gradiente final do circuito em relação a saída da porta. É como o circuito pedindo aquela porta maior ou menor valor de saída, e com alguma força. A porta simplesmente pega essa força e multiplica em relação a todas as forças calculadas para suas entradas anteriores (regra da cadeia) - repare como a força de q (-4) é multiplicada as forças de x e y. Isso pode ter dois efeitos desejados:
    - Se a porta tem uma força positiva de saída, essa força também é multiplicada nas suas entradas, escalonada pelo valor da força das entradas.
    - Se a porta tem uma força negativa de saída, isso significa que o circuito quer que a saída decresça,...
Can you write Python code for it?

x,y,z = -2,5,-4
h = 0.0001

#insira seu código aqui


I am working on the file "Backpropagation.ipynb".
The first task is:
Como a gente pode ver, a derivada em relação a $x$ é igual a $+3$. O sinal positivo indica que alterando o valor de $x$ pelo passo $h$, a saída se torna maior. O valor $3$ pode ser considerado como o valor da força que puxa $x$. O inverso acontece com $y$.

A derivada em relação a alguma entrada pode ser calculada ajustando levemente aquela entrada e observando a mudança no valor da saída

A derivada é calculada sobre cada entrada, enquanto o gradiente representa todas as derivadas sobre as entradas concatenadas em um vetor.
Como a gente pode perceber $-5.87 > -6$. Apenas 3 avaliações foram necessárias para aumentar o valor da saída (ao invés de centenas) e conseguimos um melhor resultado.
Passo maior nem sempre é melhor: É importante destacar que qualquer valor de passo maior que 0.01 ia sempre funcionar melhor (por exemplo, passo = 1 gera a saída = 1). No entanto, à medida que os circuitos vão ficando mais complexos (como em redes neurais completas), a função vai ser tornando mais caótica e complexa. O gradiente garante que se você tem um passo muito pequeno (o ideal seria infinitesimal), então você definitivamente aumenta a saída seguindo aquela direção. O passo que estamos utilizando (0.01) ainda é muito grande, mas como nosso circuito é simples, podemos esperar pelo melhor resultado. Lembre-se da analogia do escalador cego.
Estratégia 4: Gradiente Analítico
A estratégia que utilizamos até agora de ajustar levemente a entrada e ver o que acontece com a saída pode não ser muito cômoda na prática quando temos milhares de entradas para ajustar. Então, a gente precisa de algo melhor.
Felizmente, existe uma estratégia mais fácil e muito mais rápida para calcular o gradiente: podemos usar cálculo para derivar diretamente a nossa função. Chamamos isso de gradiente analítico e dessa forma não precisamos ajustar levemente nada.

O gradiente analítico evita o leve ajustamento das entradas. O circuito pode ser derivado usando cálculo.

É muito fácil calcular derivadas parciais para funções simples como $x*y$. Se...
Can you write Python code for it?

x, y = -2, 3
out = forwardMultiplyGate(x,y)

# insira seu código aqui!


I am working on the file "Backpropagation.ipynb".
The first task is:
Estratégia 3: Gradiente Numérico
Imagine agora que a gente pega as entradas de um circuito e puxa-as para uma direção positiva. Essa força puxando $x$ e $y$ vai nos dizer como $x$ e $y$ devem mudar para aumentar a saída. Não entendeu? Vamos explicar:
Se olharmos para as entradas, a gente pode intuitivamente ver que a força em $x$ deveria sempre ser positiva, porque tornando $x$ um pouquinho maior de $x=-2$ para $x=-1$ aumenta a saída do circuito para $-3$, o que é bem maior que $-6$. Por outro lado, se a força em $y$ for negativa, tornando-o menor, como de $y=3$ para $y=2$, também aumenta a saída: $-2\times2 = -4$, de novo maior que $-6$.
E como calcular essa força? Usando derivadas.

A derivada pode ser pensada como a força que a gente aplica em cada entrada para aumentar a saída

<img src='images/derivada.gif'>
E como exatamente a gente vai fazer isso? Em vez de olhar para o valor de saída, como fizemos anteriormente, nós vamos iterar sobre as cada entrada individualmente, aumentando-as bem devagar e vendo o que acontece com a saída. A quantidade que a saída muda é a resposta da derivada.
Vamos para definição matemática. A derivada em relação a $x$ pode ser definida como:
$$\frac{\partial f(x,y)}{\partial x} = \frac{f(x+h,y) - f(x,y)}{h}$$
Onde $h$ é pequeno. Nós vamos, então, calcular a saída inicial $f(x,y)$ e aumentar $x$ por um valor pequeno $h$ e calcular a nova saída $f(x+h,y)$. Então, nós subtraimos esse valores para ver a diferença e dividimos por $f(x+h,y)$ para normalizar essa mudança pelo valor (arbitrário) que nós usamos.
Em termos de código, teremos:
Can you write Python code for it?

x, y = -2, 3
out = forwardMultiplyGate(x,y)
h = 0.0001

# derivada em relação a x

# derivada em relação a y


I am working on the file "Backpropagation.ipynb".
The first task is:
Layer 1 is very similar to Layer 2, but with an addition partial derivative term.
$$ \frac{\partial C}{\partial \mathbf{W}^{(1)}} =
   \frac{\partial C}{\partial \mathbf{a}^{(3)}}
   \left(
   \frac{\partial \mathbf{a}^{(3)}}{\partial \mathbf{a}^{(2)}}
   \frac{\partial \mathbf{a}^{(2)}}{\partial \mathbf{a}^{(1)}}
   \right)
   \frac{\partial \mathbf{a}^{(1)}}{\partial \mathbf{z}^{(1)}}
   \frac{\partial \mathbf{z}^{(1)}}{\partial \mathbf{W}^{(1)}}
   ,$$
$$ \frac{\partial C}{\partial \mathbf{b}^{(1)}} =
   \frac{\partial C}{\partial \mathbf{a}^{(3)}}
   \left(
   \frac{\partial \mathbf{a}^{(3)}}{\partial \mathbf{a}^{(2)}}
   \frac{\partial \mathbf{a}^{(2)}}{\partial \mathbf{a}^{(1)}}
   \right)
   \frac{\partial \mathbf{a}^{(1)}}{\partial \mathbf{z}^{(1)}}
   \frac{\partial \mathbf{z}^{(1)}}{\partial \mathbf{b}^{(1)}}
   .$$
You should be able to adapt lines from the previous cells to complete both the weight and bias Jacobian.
Can you write Python code for it?

# GRADED FUNCTION

# Fill in all incomplete lines.
# ===YOU SHOULD EDIT THIS FUNCTION===
def J_W1 (x, y) :
    a0, z1, a1, z2, a2, z3, a3 = network_function(x)
    J = 2 * (a3 - y)
    J = J * d_sigma(z3)
    J = (J.T @ W3).T
    J = J * d_sigma(z2)
    J = (J.T @ W2).T
    J = J * d_sigma(z1)
    J = J @ a0.T / x.size
    return J

# Fill in all incomplete lines.
# ===YOU SHOULD EDIT THIS FUNCTION===
def J_b1 (x, y) :
    a0, z1, a1, z2, a2, z3, a3 = network_function(x)
    J = 2 * (a3 - y)
    J = J * d_sigma(z3)
    J = (J.T @ W3).T
    J = J * d_sigma(z2)
    J = (J.T @ W2).T
    J = J * d_sigma(z1)
    J = np.sum(J, axis=1, keepdims=True) / x.size
    return J


I am working on the file "Backpropagation.ipynb".
The first task is:
Neurônio Sigmóide
Qualquer função diferenciável pode atuar como uma porta, como também podemos agrupar múltiplas portas para formar uma simples porta, ou decompor um função em múltiplas portas quando for conveniente. Para exemplificar, vamos utilizar a função de ativação sigmoid com entradas x e pesos w:
$$f(w,x) = \frac{1}{1+e^{-(w_0x_0 + w_1x_1 + w_2)}}$$
Como dito, a função acima nada mais é que a função sigmoid $\sigma(x)$. Sabendo, então, que a derivada da função sigmoid é:
$$\sigma(x)=\frac{1}{1+e^{-x}}=(1-\sigma(x))\sigma(x)$$
Vamos calcular a gradiente em relação as entradas:
Can you write Python code for it?

w0, w1, w2 = 2, -3, -3
x0, x1 = -1, -2

# forward pass

# backward pass

# Nova saida


I am working on the file "Backpropagation.ipynb".
The first task is:
Next, if you've implemented the assignment correctly, the following code will iterate through a steepest descent algorithm using the Jacobians you have calculated.
The function will plot the training data (in green), and your neural network solutions in pink for each iteration, and orange for the last output.
It takes about 50,000 iterations to train this network.
We can split this up though - 10,000 iterations should take about a minute to run.
Run the line below as many times as you like.
Can you write Python code for it?

plot_training(x, y, iterations=50000, aggression=7, noise=1)


I am working on the file "Backpropagation.ipynb".
The first task is:
Ok, foi bem melhor. Mas, e se tivermos milhões de entradas? É claro que essa estratégia não funcionará. Vamos tentar algo mais aprimorado.
Estratégia 2: Busca Aleatória Local
Can you write Python code for it?

x, y = -2, 3
passo = 0.01
melhor_saida = forwardMultiplyGate(x,y)
melhor_x, melhor_y = 0, 0

for k in range(0,100):
    x_try = x + passo * (2*np.random.random() - 1)
    y_try = y + passo * (2*np.random.random() - 1)
    out = forwardMultiplyGate(x_try, y_try)

    if out > melhor_saida:
        melhor_saida = out
        melhor_x, melhor_y = x_try, y_try

print(melhor_x, melhor_y, forwardMultiplyGate(melhor_x, melhor_y))


I am working on the file "Backpropagation.ipynb".
The first task is:
Vamos supor agora que não sabemos a derivada da função $\sigma(x)$ muito menos de $f(w,x)$. O que podemos fazer?.
Decompor essa função em circuito com múltiplas portas! Dessa forma:

Calculando a saída para cada porta, temos:

Onde sabemos as seguintes derivadas:
$$f(x) = \frac{1}{x} \rightarrow \frac{df}{dx} = -1/x^2
\\
f_c(x) = c + x \rightarrow \frac{df}{dx} = 1
\\
f(x) = e^x \rightarrow \frac{df}{dx} = e^x
\\
f_a(x) = ax \rightarrow \frac{df}{dx} = a$$
Onde as funções $f_c(x)$ e $f_a(x)$ transladam a entrada por uma constante $c$ e escala por uma contante $a$, respectivamente. Na verdade, são apenas casos especias de adição e multiplicação, mas que foram introduzidos como portas unárias.
Como podemos calcular a derivada em relação as entradas agora? Usando Backpropagation!!
2. Backpropagation
Se tornando um Ninja em Backpropagation!
Antes de resolver o circuito acima, vamos praticar um pouco de backpropagation com alguns exemplos. Vamos esquecer funções por enquanto e trabalhar só com 4 variáveis: $a$, $b$, $c$, e $x$. Vamos também nos referir as seus gradientes como $da$, $db$, $dc$, e $dx$. Além disso, vamos assumir que $dx$ é dado (ou é +1 como nos casos acima). Nosso primeiro exemplo é a porta $*$, que já conhecemos:
$$x  = a * b$$
$$da = b * dx$$
$$db = a * dx$$
Se você reparar bem, vai perceber que a porta $$ atua como um switcher* durante a backpropagation, ou seja, o gradiente de cada entrada é o valor da outra multiplicado pelo gradiente da anterior (regra da cadeia). Por outro lado, vamos analisar a porta +:
$$x  = a + b$$
$$da = 1.0 * dx$$
$$db = 1.0 * dx$$
Nesse caso, 1.0 é o gradiente local e a multiplicação é a nossa regra da cadeia. E se fosse a adição de 3 números?:
$$q = a + b$$
$$x = q + c$$
$$dc = 1.0 * dx$$
$$dq = 1.0 * dx$$
$$da = 1.0 * dq$$
$$db = 1.0 * dq$$
Você percebe o que está acontecendo? Se você olhar nos diagramas dos circuitos que já resolvemos, vai perceber que a porta +...
Can you write Python code for it?

w0, w1, w2 = 2, -3, -3
x0, x1 = -1, -2

# forward pass

# backward pass


I am working on the file "Backtests/Notebooks/MULTI-STRATEGY - EQUAL WEIGHTING (FUNDS).ipynb".
The first task is:
Let's look at some more recent performance
Can you write Python code for it?

import datetime as dt
p_value[p_value.index > dt.datetime(2010,1,1)].plot(figsize=(15,10), grid=True)


I am working on the file "Backtests/Notebooks/MULTI-STRATEGY - EQUAL WEIGHTING (FUNDS).ipynb".
The first task is:
Portfolio Moving Average (PMA), Relative Strength (RS) and Dual Momentum (DM) Backtests
Can you write Python code for it?

import os
os.chdir('C:\\users\\scuba\\pycharmprojects\\simplebacktester')
os.getcwd()

import pandas as pd
import ffn

%matplotlib inline

from backtest_helpers.compute_weights_RS_DM import compute_weights_RS_DM
from backtest_helpers.compute_weights_PMA import compute_weights_PMA
from backtest_helpers.monthly_return_table import monthly_return_table
from backtest_helpers.endpoints import endpoints
from backtest_helpers.backtest import backtest

strategies = {
    'RS0001': { 'symbols': ['VCVSX','VWEHX','VFIIX','FGOVX','VWAHX'], 'prices': 'yahoo',
               'rs_lookback': 1, 'risk_lookback': 1, 'n_top': 2, 'frequency': 'm',
              'cash_proxy': 'CASHX', 'risk_free': 0},
    'RS0002': {'symbols': ['MMHYX','FAGIX','VFIIX'],  'prices': 'yahoo',
               'rs_lookback': 3, 'risk_lookback': 2, 'n_top': 1, 'frequency': 'm',
              'cash_proxy': 'CASHX', 'risk_free': 0},
    'RS0003': {'symbols': ['MMHYX','FAGIX','VFIIX'], 'prices': 'yahoo',
               'rs_lookback': 1, 'risk_lookback': 1, 'n_top': 1, 'frequency': 'q',
              'cash_proxy': 'CASHX', 'risk_free': 0},
    'DM0001': {'symbols': ['VCVSX','VWINX','VWEHX','VGHCX','VUSTX','VFIIX','VWAHX','FGOVX','FFXSX'],
                'prices': 'yahoo',
               'rs_lookback': 1, 'risk_lookback': 1, 'n_top': 3, 'frequency': 'm',
              'cash_proxy': 'CASHX', 'risk_free': 'FFXSX'},
    'DM0002': {'symbols': ['VCVSX','VUSTX','VWEHX','VFIIX','VGHCX','FRESX'], 'prices': 'yahoo',
               'rs_lookback': 1, 'risk_lookback': 1, 'n_top': 5, 'frequency': 'm',
              'cash_proxy': 'VFIIX', 'risk_free': 'FFXSX'},
    'PMA001': {'symbols': ['VCVSX', 'VFIIX'],  'prices': 'yahoo',
               'risk_lookback': 3, 'frequency': 'm', 'allocations': [0.6, 0.4],
              'cash_proxy': 'VUSTX'},
    'PMA002': {'symbols': ['VCVSX', 'VWINX', 'VWEHX'], 'prices': 'yahoo',
               'risk_lookback': 3, 'frequency': 'm', 'allocations': [0.6, 0.2, 0.2],
              'cash_proxy': 'VUSTX'},
    'PMA003':...


I am working on the file "Backtests/Notebooks/MULTI-STRATEGY - EQUAL WEIGHTING (FUNDS).ipynb".
The first task is:
The CAGR has dropped slightly but the Sharpe Ratio has increased significantly and the Max Drawdown has been reduced
Can you write Python code for it?

# algo stats
ffn.calc_perf_stats(p_value).display()

frame = df['Annual Returns'].to_frame()
frame['positive'] = df['Annual Returns'] >= 0
frame['Annual Returns'].plot(figsize=(15,10),kind='bar',color=frame.positive.map({True: 'g', False: 'r'}), grid=True)

p_value[p_value.index > dt.datetime(2010,1,1)].plot(figsize=(15,10), grid=True)

p_holdings.loc[rebalance_dates].round(0)[:5]


I am working on the file "Backup (not final delivery)/Homework 1 - with output.ipynb".
The first task is:
Creating and filling the DataFrame
In order to iterate only once over the data folder, we will attach the metadata to each excel spreadsheet right after creating a DataFrame with it. This will allow the code to be shorter and clearer, but also to iterate only once on every line and therefore be more efficient.
Can you write Python code for it?

#Creating an empty DataFrame to store our data and initializing a counter.
Combined_data = pd.DataFrame()
K = 0
while (K < int(len(allFiles))):

    #Creating a DataFrame and filling it with the excel's data
    df = pd.read_excel(allFiles[K], header=None)

    #Getting the metadata of the corresponding spreadsheet
    df['BARCODE'] = Sheet10_Meta.at[int(K), 'BARCODE']
    df['GROUP'] = Sheet10_Meta.at[int(K), 'GROUP']
    df['SAMPLE'] = Sheet10_Meta.at[int(K),'SAMPLE']

    #Append the recently created DataFrame to our combined one
    Combined_data = Combined_data.append(df)

    K = K + 1

#Renaming the columns with meaningfull names
Combined_data.columns = ['Name', 'Value','BARCODE','GROUP','SAMPLE']
Combined_data.head()


I am working on the file "Backup (not final delivery)/Homework 1 - with output.ipynb".
The first task is:
Question 3.4
For each travel class, calculate the proportion of the passengers that survived. Present your results in pie charts.
assumptions:
Can you write Python code for it?

# function that returns the number of people that survived and died given a specific travelclass
def survivedPerClass(pclass):
    survived = len(titanic.survived[titanic.survived == 1][titanic.pclass == pclass])
    died = len(titanic.survived[titanic.survived == 0][titanic.pclass == pclass])
    return [survived, died]

# Fixing the layout horizontal
the_grid = plt.GridSpec(1, 3)
labels = ["Survived", "Died"]

# Each iteration plots a pie chart
for p in titanic.pclass.unique():
    sizes = survivedPerClass(p)
    plt.subplot(the_grid[0, p-1], aspect=1 )
    plt.pie(sizes, labels=labels, autopct='%1.1f%%')

plt.show()


I am working on the file "Backup (not final delivery)/Homework 1 - with output.ipynb".
The first task is:
Question 3.6
"Create 2 equally populated age categories and calculate survival proportions by age category, travel class and sex. Present your results in a DataFrame with unique index."
assumptions:
1. By "proportions" we assume it is a likelyhood-percentage of surviving
2. To create 2 equally populated age categories; we will find the median and round up from the median to nearest whole year difference before splitting.
Can you write Python code for it?

#drop NaN rows
age_without_nan = titanic.age.dropna()

#categorizing
age_categories = pd.qcut(age_without_nan, 2, labels=["Younger", "Older"])

#Numbers to explain difference
median = int(np.float64(age_without_nan.median()))
amount = int(age_without_nan[median])
print("The Median age is {median} years old".format(median = median))
print("and there are {amount} passengers that are {median} year old \n".format(amount=amount, median=median))

print(age_categories.groupby(age_categories).count())
print("\nAs you can see the pd.qcut does not cut into entirely equal sized bins, because the age is of a discreet nature")

# imported for the sake of surpressing some warnings
import warnings
warnings.filterwarnings('ignore')

# extract relevant attributes
csas = titanic[['pclass', 'sex', 'age', 'survived']]
csas.dropna(subset=['age'], inplace=True)

# Defining the categories
csas['age_group'] = csas.age > csas.age.median()
csas['age_group'] = csas['age_group'].map(lambda age_category: 'older' if age_category else "younger")

# Converting to int to make it able to aggregate and give percentage
csas.survived = csas.survived.astype(int)

g_categories = csas.groupby(['pclass', 'age_group', 'sex'])
result = pd.DataFrame(g_categories.survived.mean()).rename(columns={'survived': 'survived proportion'})

# reset current index and spesify the unique index
result.reset_index(inplace=True)
unique_index = result.pclass.astype(str) + ': ' + result.age_group.astype(str) + ' ' + result.sex.astype(str)

# Finalize the unique index dataframe
result_w_unique = result[['survived proportion']]
result_w_unique.set_index(unique_index, inplace=True)
print(result_w_unique)


I am working on the file "Backup (not final delivery)/Homework 1 - with output.ipynb".
The first task is:
Task 1. Compiling Ebola Data
The DATA_FOLDER/ebola folder contains summarized reports of Ebola cases from three countries (Guinea, Liberia and Sierra Leone) during the recent outbreak of the disease in West Africa. For each country, there are daily reports that contain various information about the outbreak in several cities in each country.
Use pandas to import these data files into a single Dataframe.
Using this DataFrame, calculate for each country, the daily average per month of new cases and deaths.
Make sure you handle all the different expressions for new cases and deaths that are used in the reports.
Can you write Python code for it?

'''
Functions needed to solve task 1
'''

#function to import excel file into a dataframe
def importdata(path,date):
    allpathFiles = glob.glob(DATA_FOLDER+path+'/*.csv')
    list_data = []
    for file in allpathFiles:
        excel = pd.read_csv(file,parse_dates=[date])
        list_data.append(excel)
    return pd.concat(list_data)

#function to add the month on a new column of a DataFrame
def add_month(df):
    copy_df = df.copy()
    months = [calendar.month_name[x.month] for x in copy_df.Date]
    copy_df['Month'] = months
    return copy_df

#founction which loc only the column within a country and a specified month
#return a dataframe
def chooseCountry_month(dataframe,country,descr,month):
    df = dataframe.loc[(dataframe['Country']==country) & (dataframe['Description']==descr)]
    #df = add_month(df)
    df_month = df.loc[(df['Month']==month)]
    return df_month

# Create a dataframe with the number of death, the new cases and the daily infos for a country and a specified month
def getmonthresults(dataframe,country,month):
    if country =='Liberia':
        descr_kill ='Total death/s in confirmed cases'
        descr_cases ='Total confirmed cases'
    if country =='Guinea':
        descr_kill ='Total deaths of confirmed'
        descr_cases ='Total cases of confirmed'
    if country == 'Sierra Leone':
        descr_kill ='death_confirmed'
        descr_cases ='cum_confirmed'

    df_kill = chooseCountry_month(dataframe,country,descr_kill,month)
    df_cases = chooseCountry_month(dataframe,country,descr_cases,month)

    #calculate the number of new cases and of new deaths for the all month
    res_kill = int(df_kill.iloc[len(df_kill)-1].Totals)-int(df_kill.iloc[0].Totals)
    res_cases = int(df_cases.iloc[len(df_cases)-1].Totals)-int(df_cases.iloc[0].Totals)
    #calculate the number of days counted which is last day of register - first day of register
    nb_day = df_kill.iloc[len(df_kill)-1].Date.day-df_kill.iloc[0].Date.day

    # Sometimes the values in the dataframe are...


I am working on the file "Backup (not final delivery)/Homework 1 - with output.ipynb".
The first task is:
Task 2. RNA Sequences
In the DATA_FOLDER/microbiome subdirectory, there are 9 spreadsheets of microbiome data that was acquired from high-throughput RNA sequencing procedures, along with a 10th file that describes the content of each.
Use pandas to import the first 9 spreadsheets into a single DataFrame.
Then, add the metadata information from the 10th spreadsheet as columns in the combined DataFrame.
Make sure that the final DataFrame has a unique index and all the NaN values have been replaced by the tag unknown.
Can you write Python code for it?

Sheet10_Meta = pd.read_excel(DATA_FOLDER +'microbiome/metadata.xls')
allFiles = glob.glob(DATA_FOLDER + 'microbiome' + "/MID*.xls")
allFiles


I am working on the file "Backup (not final delivery)/Homework 1.ipynb".
The first task is:
3. Cleaning and reindexing

At first we get rid of the NaN value, we must replace them by "unknown". In order to have a more meaningful and single index, we will reset it to be the name of the RNA sequence.
Can you write Python code for it?

#Replacing the NaN values with unknwown
Combined_data = Combined_data.fillna('unknown')

#Reseting the index
Combined_data = Combined_data.set_index('Name')

#Showing the result
Combined_data


I am working on the file "Backup (not final delivery)/Homework 1.ipynb".
The first task is:
Question 3.2
"Plot histograms for the travel class, embarkation port, sex and age attributes. For the latter one, use discrete decade intervals. "
assumptions:
Can you write Python code for it?

#Plotting the ratio different classes(1st, 2nd and 3rd class) the passengers have
pc = titanic.pclass.value_counts().sort_index().plot(kind='bar')
pc.set_title('Travel classes')
pc.set_ylabel('Number of passengers')
pc.set_xlabel('Travel class')
pc.set_xticklabels(('1st class', '2nd class', '3rd class'))
plt.show(pc)

#Plotting the amount of people that embarked from different cities(C=Cherbourg, Q=Queenstown, S=Southampton)
em = titanic.embarked.value_counts().sort_index().plot(kind='bar')
em.set_title('Ports of embarkation')
em.set_ylabel('Number of passengers')
em.set_xlabel('Port of embarkation')
em.set_xticklabels(('Cherbourg', 'Queenstown', 'Southampton'))
plt.show(em)

#Plotting what sex the passengers are
sex = titanic.sex.value_counts().plot(kind='bar')
sex.set_title('Gender of the passengers')
sex.set_ylabel('Number of Passengers')
sex.set_xlabel('Gender')
sex.set_xticklabels(('Female', 'Male'))
plt.show(sex)

#Plotting agegroup of passengers
bins = [0,10,20,30,40,50,60,70,80]
age_grouped = pd.DataFrame(pd.cut(titanic.age, bins))
ag = age_grouped.age.value_counts().sort_index().plot.bar()
ag.set_title('Age of Passengers ')
ag.set_ylabel('Number of passengers')
ag.set_xlabel('Age groups')
plt.show(ag)


I am working on the file "Backup (not final delivery)/Homework 1.ipynb".
The first task is:
Question 3.3
Calculate the proportion of passengers by cabin floor. Present your results in a pie chart.
assumptions:
- Because we are tasked with categorizing persons by the floor of their cabin it was problematic that you had cabin input: "F E57" and "F G63". There were only 7 of these instances with conflicting cabinfloors. We also presumed that the was a floor "T". Even though there was only one instance, so it might have been a typo.
- We assume that you don't want to include people without cabinfloor
Can you write Python code for it?

'''
Parsing the cabinfloor, into floors A, B, C, D, E, F, G, T and display in a pie chart

'''
#Dropping NaN (People without cabin)
cabin_floors = titanic.cabin.dropna()

# removes digits and spaces
cabin_floors = cabin_floors.str.replace(r'[\d ]+', '')
# removes duplicate letters and leave unique (CC -> C) (FG -> G)
cabin_floors = cabin_floors.str.replace(r'(.)(?=.*\1)', '')
# removes ambigous data from the dataset (FE -> NaN)(FG -> NaN)
cabin_floors = cabin_floors.str.replace(r'([A-Z]{1})\w+', 'NaN' )

# Recategorizing (Since we altered the entries, we messed with the categories)
cabin_floors = cabin_floors.astype('category')
# Removing NaN (uin this case ambigous data)
cabin_floors = cabin_floors.cat.remove_categories('NaN')
cabin_floors = cabin_floors.dropna()

# Preparing data for plt.pie
numberOfCabinPlaces = cabin_floors.count()
grouped = cabin_floors.groupby(cabin_floors).count()
sizes = np.array(grouped)
labels = np.array(grouped.index)

# Plotting the pie chart
plt.pie(sizes, labels=labels, autopct='%1.1f%%', pctdistance=0.75, labeldistance=1.1)
print("There are {cabin} passengers that have cabins and {nocabin} passengers without a cabin"
      .format(cabin=numberOfCabinPlaces, nocabin=(len(titanic) - numberOfCabinPlaces)))


I am working on the file "Backup (not final delivery)/Homework 1.ipynb".
The first task is:
Question 3.4
For each travel class, calculate the proportion of the passengers that survived. Present your results in pie charts.
assumptions:
Can you write Python code for it?


I am working on the file "Backup (not final delivery)/Homework 1.ipynb".
The first task is:
Task 1. Compiling Ebola Data
The DATA_FOLDER/ebola folder contains summarized reports of Ebola cases from three countries (Guinea, Liberia and Sierra Leone) during the recent outbreak of the disease in West Africa. For each country, there are daily reports that contain various information about the outbreak in several cities in each country.
Use pandas to import these data files into a single Dataframe.
Using this DataFrame, calculate for each country, the daily average per month of new cases and deaths.
Make sure you handle all the different expressions for new cases and deaths that are used in the reports.
Can you write Python code for it?


I am working on the file "Backup (not final delivery)/Homework 1.ipynb".
The first task is:
Task 2. RNA Sequences
In the DATA_FOLDER/microbiome subdirectory, there are 9 spreadsheets of microbiome data that was acquired from high-throughput RNA sequencing procedures, along with a 10th file that describes the content of each.
Use pandas to import the first 9 spreadsheets into a single DataFrame.
Then, add the metadata information from the 10th spreadsheet as columns in the combined DataFrame.
Make sure that the final DataFrame has a unique index and all the NaN values have been replaced by the tag unknown.
Can you write Python code for it?


I am working on the file "Backup (not final delivery)/Homework 1.ipynb".
The first task is:
Task 3. Class War in Titanic
Use pandas to import the data file Data/titanic.xls. It contains data on all the passengers that travelled on the Titanic.
For each of the following questions state clearly your assumptions and discuss your findings:

Describe the type and the value range of each attribute. Indicate and transform the attributes that can be Categorical.
Plot histograms for the travel class, embarkation port, sex and age attributes. For the latter one, use discrete decade intervals.
Calculate the proportion of passengers by cabin floor. Present your results in a pie chart.
For each travel class, calculate the proportion of the passengers that survived. Present your results in pie charts.
Calculate the proportion of the passengers that survived by travel class and sex. Present your results in a single histogram.
Create 2 equally populated age categories and calculate survival proportions by age category, travel class and sex. Present your results in a DataFrame with unique index.

Question 3.1
Describe the type and the value range of each attribute. Indicate and transform the attributes that can be Categorical.
Assumptions:
    - "For each exercise, please provide both a written explanation of the steps you will apply to manipulate the data, and the corresponding code." We assume that "written explanation can come in the form of commented code as well as text"
    - We assume that we must not describe the value range of attributes that contain string as we dont feel the length of strings or ASCI-values don't give any insight
Can you write Python code for it?

'''
Here is a sample of the information in the titanic dataframe
'''

# Importing titanic.xls info with Pandas
titanic = pd.read_excel('Data/titanic.xls')

# printing only the 30 first and last rows of information
print(titanic.head)

'''
To describe the INTENDED values and types of the data we will show you the titanic.html file that was provided to us
Notice:
    - 'age' is of type double, so someone can be 17.5 years old, mostly used with babies that are 0.x years old
    - 'cabin' is stored as integer, but it har characters and letters
    - By this model, embarked is stored as an integer, witch has to be interpreted as the 3 different embarkation ports
    - It says that 'boat' is stored as a integer even though it has spaces and letters, it should be stored as string

PS: it might be that the information stored as integer is supposed to be categorical data,
        ...because they have a "small" amount of valid options
'''

# Display html info in Jupyter Notebook
from IPython.core.display import display, HTML
htmlFile = 'Data/titanic.html'
display(HTML(htmlFile))

'''
The default types of the data after import:
Notice:
    - the strings and characters are imported as objects
    - 'survived' is imported as int instead of double (which is in our opinion better since it's only 0 and 1
    - 'sex' is imported as object not integer because it is a string
'''

titanic.dtypes

'''
Below you can see the value range of the different numerical values.

name, sex, ticket, cabin, embarked, boat and home.dest is not included because they can't be quantified numerically.
'''

titanic.describe()

'''
Additional information that is important to remember when manipulation the data
is if/where there are NaN values in the dataset
'''

# This displays the number of NaN there is in different attributes
print(pd.isnull(titanic).sum())

'''
Some of this data is missing while some is meant to describe 'No' or something of meaning.
Example:
    Cabin has 1014 NaN in its column, it might be that every passenger had...


I am working on the file "Backup/Matching Market v8.ipynb".
The first task is:
Construct the market
For the market two classes are made. The market itself, which controls the buyers and the sellers, and the book. The market has a book where the results of the clearing procedure are stored.
Can you write Python code for it?

# the book is an object of the market used for the clearing procedure
class Book():
    def __init__(self):
        self.ledger = pd.DataFrame(columns = ("role","name","price","cleared"))

    def set_asks(self,seller_list):
        # ask each seller their name
        # ask each seller their willingness
        # for each willingness append the data frame
        for seller in seller_list:
            seller_name = seller.get_name()
            seller_price = seller.get_asks()
            for price in seller_price:
                self.ledger=self.ledger.append({"role":"seller","name":seller_name,"price":price,"cleared":"in process"},
                                               ignore_index=True)

    def set_bids(self,buyer_list):
        # ask each seller their name
        # ask each seller their willingness
        # for each willingness append the data frame
        for buyer in buyer_list:
            buyer_name = buyer.get_name()
            buyer_price = buyer.get_bids()
            for price in buyer_price:
                self.ledger=self.ledger.append({"role":"buyer","name":buyer_name,"price":price,"cleared":"in process"},
                                               ignore_index=True)

    def update_ledger(self,ledger):
        self.ledger = ledger

    def get_ledger(self):
        return self.ledger

    def clean_ledger(self):
        self.ledger = pd.DataFrame(columns = ("role","name","price","cleared"))

class Market():
    def __init__(self):
        self.count = 0
        self.last_price = ''
        self.book = Book()
        self.b = []
        self.s = []
        self.buyer_list = []
        self.seller_list = []
        self.buyer_dict = {}
        self.seller_dict = {}
        self.ledger = ''

    def update_seller(self):
        for i in self.seller_dict:
            self.seller_dict[i].step += 1
            self.seller_dict[i].set_quantity()

    def update_buyer(self):
        for i in self.buyer_dict:
            self.buyer_dict[i].step += 1
           ...


I am working on the file "Backup/Matching Market v8.ipynb".
The first task is:
Example Market
In the following code example we use the buyer and supplier objects to create a market. At the market a single price is announced which causes as many units of goods to be swapped as possible. The buyers and sellers stop trading when it is no longer in their own interest to continue.
Can you write Python code for it?

# Show some real consumption data, for more data see folder data analytics
#read montly consumption data of 2010 into a dataframe
df = pd.read_csv('2010cbstestrun.csv', header=0, index_col=0)
df = df.transpose()

#plot the 2010 monthly consumption data
df.plot();
df

# make initialization dictionary
init_buyer = {'elec':{'b':400, 'm' : 673}, 'indu':{'b':400, 'm':1171}, 'home':{'b': 603, 'm': 3615}}
init_seller = {'netherlands' : (2000, 0, 10), 'Russia' : (2000, 0, 10)}

# make a history book to record every timestep
hist_book = []

# set the starting time
period_null= datetime.date(2010,1,1)


I am working on the file "Backup/Matching Market v8.ipynb".
The first task is:
Operations Research Formulation
The market can also be formulated as a very simple linear program or linear complementarity problem. It is clearer and easier to implement this market clearing mechanism with agents. One merit of the agent-based approach is that we don't need linear or linearizable supply and demand function.
The auctioneer is effectively following a very simple linear program subject to constraints on units sold. The auctioneer is, in the primal model, maximizing the consumer utility received by customers, with respect to the price being paid, subject to a fixed supply curve. On the dual side the auctioneer is minimizing the cost of production for the supplier, with respect to quantity sold, subject to a fixed demand curve. It is the presumed neutrality of the auctioneer which justifies the honest statement of supply and demand.
An alternative formulation is a linear complementarity problem. Here the presence of an optimal space of trades ensures that there is a Pareto optimal front of possible trades. The perfect opposition of interests in dividing the consumer and producer surplus means that this is a zero sum game. Furthermore the solution to this zero-sum game maximizes societal welfare and is therefore the Hicks optimal solution.
Next Steps
A possible addition of this model would be to have a weekly varying demand of customers, for instance caused by the use of natural gas as a heating agent. This would require the bids and asks to be time varying, and for the market to be run over successive time periods. A second addition would be to create transport costs, or enable intermediate goods to be produced. This would need a more elaborate market operator.  Another possible addition would be to add a profit maximizing broker. This may require adding belief, fictitious play, or message passing.
The object-orientation of the models will probably need to be further rationalized. Right now the market requires very particular ordering of calls to function correctly.
Can you write Python code for it?

# timeit

stopit = time.time()
dtstopit = datetime.datetime.now()

print('it took us %s seconds to get to this conclusion' % (stopit-startit))
print('in another notation (h:m:s) %s'% (dtstopit - dtstartit))

# print the run results
price = df_hb['price']
fig = price.plot()
plt.ylabel('€ / unit')
plt.show()

quantity = df_hb['quantity']
fig = quantity.plot()
plt.ylabel('quantity')
plt.show()


I am working on the file "Backup/Matching Market v8.ipynb".
The first task is:
Time of last run
Time and date of the last run of this notebook file
Can you write Python code for it?

# print the time of last run
print('last run of this notebook:')
time.strftime("%a, %d %b %Y %H:%M:%S", time.localtime())

#df_hb
df_res = pd.DataFrame(obser1.reserve, columns=['time', *[i for i in init_seller]])
df_res = df_res.set_index('time')
df_res.plot();


I am working on the file "Backup/Matching Market v8.ipynb".
The first task is:
classes buyers and sellers
Below we are constructing the buyers and sellers in classes.
Can you write Python code for it?

# measure how long it takes to run the script
startit = time.time()
dtstartit = datetime.datetime.now()

class Seller():
    def __init__(self, name):
        self.name = name
        self.wta = []
        self.step = 0
        self.prod = 2000
        self.lb_price = 10
        self.ub_price = 20
        self.reserve = 500000
        #multiple market idea, also ga away from market
        self.subscr_market = {}

    # the supplier has n quantities that they can sell
    # they may be willing to sell this quantity anywhere from a lower price of l
    # to a higher price of u
    def set_quantity(self):
        n = self.prod
        l = self.lb_price
        u = self.ub_price
        wta = []
        for i in range(n):
            p = rnd.uniform(l, u)
            wta.append(p)
        self.wta = wta

    def get_name(self):
        return self.name

    def get_asks(self):
        return self.wta

    def clear_wta(self):
        self.wta = []

    def extract(self, cur_extraction):
        if self.reserve > 0:
            self.reserve = self.reserve - cur_extraction
        else:
            self.prod = 0

class Buyer():
    def __init__(self, name):
        self.name = name
        self.wtp = []
        self.step = 0
        self.base_demand = 0
        self.max_demand = 0
        self.lb_price = 10
        self.ub_price = 20

    # the supplier has n quantities that they can buy
    # they may be willing to sell this quantity anywhere from a lower price of l
    # to a higher price of u
    def set_quantity(self):
        n = int(self.consumption(self.step))
        l = self.lb_price
        u = self.ub_price
        wtp = []
        for i in range(n):
            p = rnd.uniform(l, u)
            wtp.append(p)
        self.wtp = wtp

    # gets a little to obvious
    def get_name(self):
        return self.name

    # return list of willingness to pay
    def get_bids(self):
        return self.wtp

    # is this neccesary?
    def clear_wtp(self):
        self.wtp = []

   ...


I am working on the file "Backup/Matching Market v8.ipynb".
The first task is:
run the model
To run the model we create the observer. The observer creates all the other objects and runs the model.
Can you write Python code for it?

# create observer and run the model
# first data about buyers then sellers and then model ticks
years = 5
timestep = 12

obser1 = Observer(init_buyer, init_seller, years*timestep)
obser1.run_it()
#get the info from the observer
hist_book = obser1.hist_book

# recording the total run
def write_to_csv(hist_book):
    f = open('hist_book.csv', 'a')
    for item in hist_book:
        f.write('%s,%s\n' % (item[0], item[1]))
    f.close()

#write_to_csv(hist_book)

# make a dataframe of clearing prices
df_hb = pd.DataFrame(hist_book)
df_hb = df_hb.set_index(0)
df_hb.index.name = 'month'
df_hb.rename(columns={1: 'price', 2: 'quantity'}, inplace=True)


I am working on the file "Backup_code_not_used.ipynb".
The first task is:
Check number of 1's in the test on the LB using logloss value for a fixed prediction probability
Can you write Python code for it?

import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
from sklearn.metrics import log_loss

## we use the fact that we know the logloss value of 0.37, user your value for a new try
myp = 1
mylogloss = 28
p = [myp] * 10000
l = np.array([log_loss([1] * r + [0] * (10000 - r), p) for r in range(1, 10000)])
x = np.arange(0.01, 100, 0.01)

plt.plot(x, l, '_')
plt.title('Log Loss vs. Pct. Positve with Constant Prediction of {}'.format(myp))
plt.xlabel('% Positve in LB')
plt.ylabel('Log Loss for Constant Prediction {}'.format(myp))
plt.grid()
plt.show()

df_check = pd.Series(data=np.abs(l - mylogloss), index=x)
print('percentage of 1 in prediction is: {}'.format(x[int(df_check.argmin())]))
df_check.plot();
plt.show()


I am working on the file "Backup_code_not_used.ipynb".
The first task is:
Proprietà transitiva: q è duplicata di B e anche di C allora B e C sono tra loro duplicati
Can you write Python code for it?

df = train_df[train_df.is_duplicate == 1]

qid1_qid2_dup = (df[['qid1',
                     'qid2']].groupby('qid1')['qid2'].apply(list)).to_dict()
qid2_qid1_dup = (df[['qid1',
                     'qid2']].groupby('qid2')['qid1'].apply(list)).to_dict()

def get_value_dict(d, v):
    r = []
    try:
        r = d[v]
    except KeyError:
        pass
    return r

def get_qid_grouped(n_g):
    qid_g = get_value_dict(qid1_qid2_dup, n_g).copy()
    qid_g.extend(get_value_dict(qid2_qid1_dup, n_g))
    while True:
        temp = qid_g.copy()
        for l in temp:
            qid_g.extend(get_value_dict(qid1_qid2_dup, l))
            qid_g.extend(get_value_dict(qid2_qid1_dup, l))
        qid_g = list(set(qid_g))
        if len(temp) == len(qid_g): break
    return qid_g

run_on_me = set(list(df.qid1) + list(df.qid2))
group_dict = {}
while len(run_on_me) != 0:
    i = next(iter(run_on_me))
    group_dict[i] = get_qid_grouped(i)
    run_on_me = run_on_me - set(group_dict[i])

# q1 and q2 are never the same question
print(len(df[df.qid1 == df.qid2]))

present_cases1 = set(
    df[['qid1', 'qid2']].apply(axis=1, func=lambda x: (x['qid1'], x['qid2'])))
present_cases2 = set(
    df[['qid1', 'qid2']].apply(axis=1, func=lambda x: (x['qid2'], x['qid1'])))

# check if there are duplicate q1-q2 pairs (q2-q1) -----> It tooked a long time, no duplicate pairs found
# in the original dataframe
# [j for j in present_cases if (j[1], j[0]) in present_cases] # ---> RESULT: []

all_combo = []
for ng in group_dict:
    for i, l in enumerate(group_dict[ng]):
        for j in group_dict[ng][i + 1:]:
            all_combo.append((l, j))
new_combo = list((set(all_combo) - set(present_cases1)) - set(present_cases2))

print(
    len(all_combo),
    len(new_combo),
    len(present_cases1), len(present_cases2), len(new_combo) / len(all_combo))

dq = (df[['qid1', question1]].set_index('qid1',
                                          drop=True)).to_dict()[question1]
dq.update((df[['qid2', question2]].set_index(
    'qid2',...


I am working on the file "Backup_code_not_used.ipynb".
The first task is:
To speed up most_similar we can use a restriction in the vocab, restriction in the top most similarities don't help
Find the top-N most similar words. Positive words contribute positively towards the
similarity, negative words negatively.
This method computes cosine similarity between a simple mean of the projection
weight vectors of the given words and the vectors for each word in the model.
The method corresponds to the word-analogy and distance scripts in the original
word2vec implementation.
If topn is False, most_similar returns the vector of similarity scores.
restrict_vocab is an optional integer which limits the range of vectors which
are searched for most-similar values. For example, restrict_vocab=10000 would
only check the first 10000 word vectors in the vocabulary order. (This may be
meaningful if you've sorted the vocabulary by descending frequency.)
Can you write Python code for it?


I am working on the file "Backup_code_not_used.ipynb".
The first task is:
Tuning w2v
** The hyper-parameter choice is crucial for performance (both speed and accuracy):
- architecture: skip-gram (slower, better for infrequent words) vs CBOW (fast)the training
- algorithm: hierarchical softmax (better for infrequent words) vs negative sampling (better for frequent words, better with low dimensional vectors)
- sub-sampling of frequent words: can improve both accuracy and speed for large data sets (useful values are in range 1e-3 to 1e-5)
- context (window) size: for skip-gram usually around 10, for CBOW around 5
- alpha is the initial learning rate (will linearly drop to min_alpha as training progresses).
- min_count = ignore all words with total frequency lower than this.
- negative = if > 0, negative sampling will be used, the int for negative specifies how many “noise words” should be drawn (usually between 5-20). Default is 5. If set to 0, no negative samping is used.
- cbow_mean = if 0, use the sum of the context word vectors. If 1 (default), use the mean. Only applies when cbow is used.
- iter = number of iterations (epochs) over the corpus. Default is 5.
- batch_words = target size (in words) for batches of examples passed to worker threads (and thus cython routines). Default is 10000.
-
The model can be stored/loaded via its save() and load() methods, or stored/loaded in a format compatible with the original word2vec implementation via wv.save_word2vec_format() and KeyedVectors.load_word2vec_format().
If you’re finished training a model (=no more updates, only querying), then switch to the gensim.models.KeyedVectors instance in wv
word_vectors = model.wv
 del model
to trim unneeded model memory
Can you write Python code for it?

def get_train(m, win, s, sort, m_c, h):
    print('New model:', m, win, s, sort, m_c, h)
    model = word2vec.Word2Vec(
        corpus,
        size=s,
        window=win,
        min_count=m_c,
        workers=4,
        sorted_vocab=sort,
        hs=h,
        iter=10,
        sg=m,
        negative=10)

    df = train_df[[q1, q2, 'is_duplicate']].copy()
    tempvar = df.apply(
        axis=1, func=lambda x: getback_function(pca_vars(x[q1], x[q2], model)))
    tempvar.columns = ['diff_eigenv_pca', 'cos_pca', 'diff_ratio_ecc_pca']
    #  + ['first_pca_'] + str(i) for i in range(0, model.vector_size + 1)
    df = pd.concat([df, tempvar], axis=1)
    df.fillna(0, inplace=True)

    tempvar = df.apply(
        axis=1,
        func=
        lambda x: getback_function(get_W2V_variables(x[q1], x[q2], model)))
    tempvar.columns = [
        'norm_mean_wv', 'norm_sum_wv', 'cos_mean_wv', 'cos_sum_wv'
    ]
    #  + ['vect_diff_'] + str(i) for i in range(0, model.vector_size + 1)
    df = pd.concat([df, tempvar], axis=1)
    df.replace(np.inf, np.nan, inplace=True)
    df.fillna(0, inplace=True)

    eventuali = [
        'question1_clear_1', 'question2_clear_1', 'question1_clear_1_clear_2',
        'question1_clear_1_clear_2', 'question2_clear_1_clear_2',
        'question1_final', 'question2_final'
    ]
    features = list(
        set(df.columns) -
        set(['id', 'qid1', 'qid2', 'question1', 'question2', 'is_duplicate'] +
            eventuali))

    x_train = df[features + ['is_duplicate']]

    model_xgb, roc_auc_test, roc_auc_train, var_imp = train_xgboost(
        x_train, features, 'is_duplicate', params, other_par)

#150 the best
for s in [120, 135, 150, 165, 180]:
    get_train(0, 5, s, 1, 5, 0)

for h in [0, 1]:
    get_train(0, 5, 150, 1, 5, h)

for sort in [-1, 1]:
    get_train(0, 5, 150, sort, 5, 0)

for m in [0, 1]:
    win = 5 if m == 0 else 10
    get_train(m, win, 150, 1, 5, 0)


I am working on the file "Backup_code_not_used.ipynb".
The first task is:
creazione tagger_counter
Can you write Python code for it?

tagger = pd.read_csv('/home/ale/random_program/Quora_double_question/risultato_tagger.csv', encoding='latin1')

def recover_tagg_list(tag_string):
    l = [
        re.split(r'[\'\"],\s[\'\"]', t[2:-1])
        for t in tag_string[1:-2].split('], ')
    ]
    return l

def tagged_ppf_verb(x):
    app = recover_tagg_list(x)
    return ([i[1] for i in app if len(i) > 1])

tagger['new'] = tagger.apply(
    axis=1, func=lambda x: Counter(tagged_ppf_verb(x['tagger'])))
dizionario = tagger.set_index('quest_final')['new'].to_dict()

import pickle

def save_obj(obj, name):
    with open('./' + name + '.pkl', 'wb') as f:
        pickle.dump(obj, f, pickle.HIGHEST_PROTOCOL)

def load_obj(name):
    with open('./' + name + '.pkl', 'rb') as f:
        return pickle.load(f)

save_obj(dizionario, 'tag_counter')

dizionario_max = pd.read_csv(
    '/home/ale/random_program/Quora_double_question/tagged_list_counted_solved.csv',
    sep=";",
    encoding='latin1')
dizionario_max = dizionario_max.set_index('name')['tag_recod'].to_dict()
save_obj(dizionario_max, 'tag_max_freq')

alltag = from_tag_tocluster(tagger[0:100]['tagger'].values)
app = pd.DataFrame(alltag.groupby(['name', 'tag']).size()).reset_index()
app.columns =['name','tag','count']
app.to_csv('.\\taggestlist_count.csv',sep=";")

def recodtag(x):
    if x[0] == 'N':
        return('NOUN')
    elif x[0] == 'V':
        return('VERB')
    elif x[0] == 'J':
        return('JJ')
    else:
        return(x)

app['tag_recod'] = app.apply(axis=1, func = lambda x: recodtag(x.tag))
app2 = pd.DataFrame(app.groupby(['name','tag_recod'])['count'].sum()).reset_index()
app2.columns =['name','tag_recod','count']
app3 = pd.DataFrame(app2.groupby(['name'])['count'].max()).reset_index()
app3 = pd.merge(app3, app2, on =['name','count'])
app3.to_csv('.\\tagged_list_counted_solved.csv',sep=";")
app3


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Extending bag-of-words with TF-IDF weights
We could extend the bag-of-words representation with tf-idf to reflect how important a word to a document in a corpus
tdf-idf can be applied with class TfidfVectorizer in sklearn
Can you write Python code for it?

vectorizer= TfidfVectorizer(stop_words='english')
test_corpus=train_data.review[0:5]
test_corpus= corpus_preprocessing(test_corpus)
test_corpus=vectorizer.fit_transform(test_corpus)
print (test_corpus.todense())


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Fitting the training data
Can you write Python code for it?

#features from train set
train_features=train_data.review

#pro-processing train features
train_features=corpus_preprocessing(train_features)
vectorizer= TfidfVectorizer(stop_words='english')
train_features=vectorizer.fit_transform(train_features)
tsvd=TruncatedSVD(100)
tsvd.fit(train_features)
train_features=tsvd.transform(train_features)

#target from train set
train_target=train_data.sentiment

#fitting the model
model.fit(train_features,train_target)


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Import dataset
Can you write Python code for it?

train_data=pd.read_csv("../../../data-project1/labeledTrainData.tsv", header=0, delimiter="\t", quoting=3)
train_data.head()

train_data.tail()


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Let's try to generate score over a range of alpha
Can you write Python code for it?

alphas=np.logspace(-5,0,6)
print alphas

def evaluate_alpha(train_features,train_target,test_features,test_target,model,parameter_value, parameter_name):
    scores=[]
    for test_alpha in params:
        model. set_params(**{parameter_name:test_alpha})
        model.fit(train_features,train_target)
        prediction=model.predict(test_features)
        score=roc_auc_score(test_target, prediction)
        scores.append((test_alpha,score))

model=MultinomialNB()
alpha_score=evaluate_alpha(split_train_features,split_train_target,split_test_features,split_test_target,model,alphas,'alpha')


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Notice that 'sentiment' is binary
Can you write Python code for it?


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Now that we already have general idea of Data Set. We next clean, transform data to create useful features for machine learning

First Attempt Summary

Feature 'review'
Processing raw text
Transforming feature 'review': bag-of-words model
Extending bag-of-words with TF-IDF weights
Dimensionality reduction

Training Naive Bayes
Predicting with Naive Bayes
Preparing for kaggle submission
Performance Evaluation
Splitting train data set
Evaluating performance using splitted data set
Plotting ROC curve

Hyperparameters
Other improvements

Feature 'review'
Processing raw text
We will start wrting function for analyzing and cleaning the deature 'review', using first review as a point of illustration
Can you write Python code for it?


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Now we start stemming and lemmatizing the text, but it is generally better to first create the pos tagger as we only want to lemmatize verb and noum
Can you write Python code for it?


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Observing that the word 'going' has been stemmed with porter but not with lancaster, I'll choose porter for this task.
let's lemmatizing
Can you write Python code for it?

tagged_words_after_stem=nltk.pos_tag(stemmed_text_with_potter)
wnl = nltk.WordNetLemmatizer()
def lemmatize_with_WordNet(token,tag):
    if tag[0].lower in ['v','n']:
        return wnl.lemmatize(token)
    return token
stemmed_and_lemmatized_text=[lemmatize_with_WordNet(token,tag) for token,tag in tagged_words_after_stem]
stemmed_and_lemmatized_text[0:10]


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Performance Evaluation
A variety of metrics exist to evaluate the performance for binary classifiers, i.e accuracy, precision, recall, F1 measure, ROC AUC score. We shall use ROC AUC score for this task as specified by competition site.
Splitting train data set
We first splitting the train data set for cross validation, let's choose 80% for split_train set and 20% for split test_set
Can you write Python code for it?

# Split 80-20 train vs test data
split_train_features, split_test_features, split_train_target, split_test_target = train_test_split(train_features,
                                                                                train_target,
                                                                                test_size = 0.20,
                                                                                random_state = 0)


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Plotting ROC curve
ROC curves plot the classifier's recall against its fall-out.
Can you write Python code for it?

false_positive_rates ,recall,thresholds=roc_curve(split_test_target,split_prediction)
plt.title('Receiver Operating Charisteristic')
plt.plot(false_positive_rates,recall,'r', label='AUC = %0.2f' %score)
plt.legend(loc = 'lower right')
plt.ylable('Recall')
plt.xlable('False positive rate')
plt.show()


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Predicting with Naive Bayes
Can you write Python code for it?

#reading test data
test_data=train_data=pd.read_csv("../../../data-project1/testData.tsv", header=0,delimiter="\t", quoting=3)

#features from test data
test_features=test_data.review

#pre-processing test features
test_features=corpus_preprocessing(test_features)
test_features=vectorizer.transform(test_features)
test_features=tsvd.transform(test_features)

#predicting the sentiment for test set
prediction=model.predict(test_features)


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Preparing for kaggle submission
Can you write Python code for it?

#writing out submission file
pd.DataFrame( data={"id":test_data["id"], "sentiment":prediction} ).to_csv("../../../data-project1/first_attempt.csv", index=False, quoting=3 )


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
The source code of the first attempt can be found here and evaluation script here
Hyperparameters
Class MultinomialNb has a parameter value alpha (default=1.0)  We could try to run on another value of alpha to see how the score would change.
Can you write Python code for it?

model=MultinomialNB(alpha=0.1)
model.fit(split_train_features,split_train_target)
split_prediction=model.predict(split_test_features)
score=roc_auc_score(split_test_target, split_predict)
print (score(split_test_target, split_predict))


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Training Naive Bayes
Sklearn provides several kinds of Naives classifiers: GaussianNB, MultinomialNB and BernoulliNB. We will choose MultinomialNB for this task
Can you write Python code for it?


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Transforming feature 'review': bag-of-words model
Let's transform feature 'review' into numerical representation to feed into machine learning. The common representation of text is the bag-of-words model
in Sklearn, we can use class CountVectorize to transform the data. We shall also use stop-words to reduce the dimension of feature space. Let's now first 5 data from train Dataset to be test_corpus
Can you write Python code for it?


I am working on the file "Bag of Words Meets Bags of Popcorn/notebook/bag of words meet bag of porncorn.ipynb".
The first task is:
Type 'object' is a string for pandas. We shall later convert to number representation,maybe using typical bag-of-words or word2vec
Starting getting basic information of data:
Can you write Python code for it?


I am working on the file "Bar Graph for Multiple Genes.ipynb".
The first task is:
In this graph, along with upcoming graphs, I chose a gene that the cerebellum had a significantly high expression for. This gene, number 75 in the array, proved my previous prediction wrong since the cerebellum, the coronary artery, and the aorta all showed significantly higher gene expressions for this gene.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
import csv
aortaData = []
aortaDataNumbers = []
cerebellumData = []
cerebellumDataNumbers= []
arteryData = []
arteryDataNumbers = []
with open ("genomicdata.csv") as csvfile:
    readCSV = csv.reader(csvfile, delimiter= '\t') #gives access to the CSV file

    for col in readCSV:
        if col[7] == '':
            aortaData.append('0')
        else:
            aortaData.append(col[7])
        if col[13] == '': #cerebellum
            cerebellumData.append('0')
        else:
            cerebellumData.append(col[13])
        if col[15] == '': #coronary artery
            arteryData.append('0')
        else:
            arteryData.append(col[15])
    aortaDataNumbers = list(map(float, aortaData[722:727]))
    cerebellumDataNumbers = list(map(float, cerebellumData[722:727]))
    arteryDataNumbers = list(map(float, arteryData[722:727]))

ind = np.arange(len(aortaDataNumbers))  # the x locations for the groups
width = 0.35       # the width of the bars

fig, ax = plt.subplots()
rects1 = ax.bar(ind, aortaDataNumbers, width, color='r') #creates rectangles for aorta
rects2 = ax.bar(ind+width, cerebellumDataNumbers, width, color='g') #creates rectangles for cerebellum
rects3 = ax.bar(ind+width*2, arteryDataNumbers, width, color='b') #creates rectangles for coronary artery

ax.set_ylabel('Expression Vector') #Y axis label
ax.set_title('Gene Expressed') #X axis label
ax.set_xticks(ind + width) #the distance between each bar

def autolabel(rects): #creates a different label for each bar to show the height
    for rect in rects:
        height = rect.get_height() #height of each bar
        ax.text(rect.get_x() + rect.get_width()/2., 1.05*height,
                '%d' % int(height),
                ha='center', va='bottom') #gives the value

plt.show()


I am working on the file "BaseClass/Porazdelitve.ipynb".
The first task is:
Graf da osnovno idejo o tem, kaj uporabiti
Can you write Python code for it?

ratingsNum=list()
for number in np.arange(1,10):
    ratingsNum.append(len(data[data[:,2]==number,2]))

plt.figure()
plt.bar(np.arange(1,10),ratingsNum, 0.8, color="blue")
plt.show()


I am working on the file "BaseClass/Porazdelitve.ipynb".
The first task is:
Ker imamo vnaprej dolečen interval, ki ne ustreza Gaussu najbolje, sem se odločil uporabiti beta porazdelitev
Can you write Python code for it?

from scipy.stats import beta

a=8
b=2
n=1000

sample=beta.rvs(a, b, size=n)

xr = np.linspace(0, 1, 100)# interval X
P  = [beta.pdf(x, a, b) for x in xr]  # porazdelitvena funkcija

# Histogram - porazdelitev naključlnih VZORCEV x glede na P(x)

plt.figure(figsize=(10, 4))
plt.subplot(1, 2, 1)
plt.title("Vzorec")
plt.hist(sample, color="red")
plt.xlabel("X")
plt.ylabel("Število primerov")

# Graf porazdelitvene funkcije
plt.subplot(1, 2, 2)
plt.title("Graf porazdelitve")
plt.plot(xr, P, color="red")   # nariši P(x)
plt.ylabel("P(x)")
plt.xlabel("X")
plt.show()


I am working on the file "BaseLine_CountVectorizer.ipynb".
The first task is:
Выходная переменная
Can you write Python code for it?


I am working on the file "BaseLine_CountVectorizer.ipynb".
The first task is:
Посмотрим самые частые и самые редкие категории.
Can you write Python code for it?


I am working on the file "BaseLine_CountVectorizer.ipynb".
The first task is:
Эксперимент с CountVectorizer и RF
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier

max_features = range(100, 1000, 100)

for mf in max_features:
    cv = CountVectorizer(max_features=mf)
    cv_train_data = cv.fit_transform(train_data['NAME'])
    rf = RandomForestClassifier(n_estimators=200, random_state=11)
    scores = cross_val_score(rf, cv_train_data, train_data['GROUP_ID'], cv=2, scoring='accuracy')

    print 'Max features: ', mf
    print 'Score: ', scores.mean()

# пробное решение
cv = CountVectorizer(max_features=1200)
cv_train_data = cv.fit_transform(train_data['NAME'])
cv_test_data = cv.transform(test_data['NAME'])

rf = RandomForestClassifier(n_estimators=400, random_state=11)
rf.fit(cv_train_data, train_data['GROUP_ID'])

predict_rf = rf.predict(cv_test_data)
test_data['GROUP_ID'] = predict_rf

test_data.head()

test_data[['id', 'GROUP_ID']].to_csv('res_3105.csv', sep=',', header=True, index=False)


I am working on the file "BaseLine_TFIDF.ipynb".
The first task is:
7 = сигареты
6 = продукты питания
32 = позиция по свободной цене

Предобработка данных
!!По-хорошему:
1. К нижнему регистру
2. Вычистить стоп-слова и знаки препинания
3. Лемматизация
4. CountVectorizer, TFIDF, Word2Vec
Can you write Python code for it?

# удаление спецсимволов
import re

def remove_special_character(somestring):
    pattern = re.compile(u'[^a-zA-Zа-яА-Я0-9_]+')
    tokens = pattern.sub(' ', somestring).strip()
    return tokens

train_data['NAME'] = train_data['NAME'].str.decode('utf-8')
train_data['NAME'] = train_data['NAME'].apply(remove_special_character)

test_data['NAME'] = test_data['NAME'].str.decode('utf-8')
test_data['NAME'] = test_data['NAME'].apply(remove_special_character)

# лемматизация
from pymystem3 import Mystem

m = Mystem()
def lemmatize(text):
    lemmas = m.lemmatize(text)
    return (''.join(lemmas)).strip()

train_data['NAME'] = train_data['NAME'].apply(lemmatize)
test_data['NAME'] = test_data['NAME'].apply(lemmatize)

test_data.head(10)

train_data.to_csv('./data/lem_train_data.csv', header=True, index=False, encoding='utf-8', sep=';')
test_data.to_csv('./data/lem_test_data.csv', header=True, index=False, encoding='utf-8', sep=';')


I am working on the file "BaseLine_TFIDF.ipynb".
The first task is:
NaiveBayes
Can you write Python code for it?

from sklearn.naive_bayes import MultinomialNB

mnb = MultinomialNB()
scores = cross_val_score(mnb, cv_train_data, train_data['GROUP_ID'], cv=3, scoring='accuracy')

print scores


I am working on the file "BaseLine_TFIDF.ipynb".
The first task is:
Выходная переменная
Can you write Python code for it?


I am working on the file "BaseLine_TFIDF.ipynb".
The first task is:
Загрузка данных
Can you write Python code for it?

train_data = pd.read_csv('./data/evo_train.csv', sep=',', header=0)
test_data = pd.read_csv('./data/evo_test.csv', sep=',', header=0)

print train_data.shape
print test_data.shape

train_data.head()


I am working on the file "BaseLine_TFIDF.ipynb".
The first task is:
Посмотрим самые частые и самые редкие категории.
Can you write Python code for it?


I am working on the file "Baseline 2.ipynb".
The first task is:
Epsilon-Support Vector Regression (SVR)
RBF Kernel
Can you write Python code for it?

# 5 Fold CV, to calculate avg RMSE
clf = SVR(C=500000.0, epsilon=0.1, kernel='rbf', gamma=0.0008)
scores = cross_val_score(clf, X_, y.values.ravel(), cv=5, scoring='neg_mean_squared_error')
for i in range(0,5):
    scores[i] = sqrt(-1*scores[i])

print(scores)
avg_rmse = scores.mean()
print("\n\nAvg RMSE is ",scores.mean())

# Just the 4 original features (no soil data)
X_old = X[["X1_zscore", "X2_zscore", "X3_zscore", "X4_zscore"]]

# 5 Fold CV, to calculate avg RMSE
clf = SVR(C=1000.0, epsilon=0.1, kernel='rbf', gamma=0.027)
scores = cross_val_score(clf, X_old, y.values.ravel(), cv=5, scoring='neg_mean_squared_error')
for i in range(0,5):
    scores[i] = sqrt(-1*scores[i])

print(scores)
avg_rmse = scores.mean()
print("\n\nAvg RMSE is ",scores.mean())


I am working on the file "Baseline 2.ipynb".
The first task is:
First checking the avg RMSE for Linear Regression
Can you write Python code for it?

clf = LinearRegression()
scores = cross_val_score(clf, X_, y, cv=5, scoring='neg_mean_squared_error')
for i in range(0,5):
    scores[i] = sqrt(-1*scores[i])

print(scores)
avg_rmse = scores.mean()
print("\n\nAvg RMSE is ",scores.mean())


I am working on the file "Baseline 2.ipynb".
The first task is:
Polynomial Kernel also does better than Linear Regression
Degree 4 Polynomial
Can you write Python code for it?

# 5 Fold CV, to calculate avg RMSE
clf = SVR(kernel='poly', gamma='auto', degree=4, coef0=2)
scores = cross_val_score(clf, X_old, y.values.ravel(), cv=5, scoring='neg_mean_squared_error')
for i in range(0,5):
    scores[i] = sqrt(-1*scores[i])

print(scores)
avg_rmse = scores.mean()
print("\n\nAvg RMSE is ",scores.mean())


I am working on the file "Baseline DQN Gym_Trading Tutorial.ipynb".
The first task is:
Create a new OpenAI Gym environment with the customised Trading environment
.initialise_simulator() must be invoked after env.make('trading-v0') . Within this function, provide these arguments:

csv_name: Address of the data

trade_period: (int), Max of duration of each trades. Default: 1000

train_split: (float), Percentage of data set for training. Default: 0.7
Can you write Python code for it?

env = gym.make('trading-v0')
env.initialise_simulator(csv, trade_period=50, train_split=0.7)


I am working on the file "Baseline DQN Gym_Trading Tutorial.ipynb".
The first task is:
Trading Framework
This framework is developed based on Peter Henry https://github.com/Henry-bee/gym_trading/ which in turn on developed of Tito Ingargiola's https://github.com/hackthemarket/gym-trading.
First, define the address for the CSV data
Can you write Python code for it?


I am working on the file "Baseline.ipynb".
The first task is:
Adding a new feature
Quantity of Phosphorous in soil (low, medium, high)
This feaure is missing for some states in India, so little less number of samples
Can you write Python code for it?

# loading the dataset
ricep = pd.read_csv("/Users/macbook/Documents/BTP/Notebook/rice with soil(P).csv")

# Removing the rows with missing value of phosphorus
ricep = ricep[np.isfinite(ricep["phosphorus"])]

# Adding collumns for the other 4 features
a = np.empty((ricep.shape[0],1))*np.NAN
ricep = ricep.assign(X1 = a)
ricep = ricep.assign(X2 = a)
ricep = ricep.assign(X3 = a)
ricep = ricep.assign(X4 = a)
ricep.head()

# Constructing features X1 and X2
l = ricep.shape[0]
for row in range(0,l):
    if row-1<0 or ricep.iloc[row,1] != ricep.iloc[row-1,1]:
        continue
    else:
        ricep.iloc[row,8] = ricep.iloc[row-1,6]
        if row-2<0 or ricep.iloc[row,1] != ricep.iloc[row-2,1]:
            continue
        else:
            ricep.iloc[row,9] = ricep.iloc[row-2,6]

# Making the strings in the dataset uniform, with other datasets
ricep['ind_district'] = ricep['ind_district'].apply(func)
ricep['Season'] = ricep['Season'].apply(func)

# Constructing features X3 and X4
l = ricep.shape[0]

for row in range(0,l):

    dt = ricep.iloc[row,1]
    yr = ricep.iloc[row,2]

    if yr<=2002:

        # rainfall for the same year
        r = rain1.loc[(rain1.ind_district == dt) & (rain1.Year == yr)]
        if r.shape[0] == 1:
            ricep.iloc[row,10] = r.iloc[0,3]

        # rainfall for the previous year
        r = rain1.loc[(rain1.ind_district == dt) & (rain1.Year == yr-1)]
        if r.shape[0] == 1:
            ricep.iloc[row,11] = r.iloc[0,3]

    if yr>2004:

        # rainfall for the same year
        r = rain2.loc[(rain2.ind_district == dt) & (rain2.Year == yr)]
        if r.shape[0] == 1:
            ricep.iloc[row,10] = r.iloc[0,3]

        # rainfall for the previous year
        r = rain2.loc[(rain2.ind_district == dt) & (rain2.Year == yr-1)]
        if r.shape[0] == 1:
            ricep.iloc[row,11] = r.iloc[0,3]

# Removing rows with any missing values
ricep = ricep[np.isfinite(ricep["Production"])]
ricep =...


I am working on the file "Baseline.ipynb".
The first task is:
Avg RMSE
Can you write Python code for it?

clf = linear_model.Ridge(alpha = 7.1)
scores = cross_val_score(clf, X_old, y, cv=5, scoring='neg_mean_squared_error')

for i in range(0,5):
    scores[i] = sqrt(-1*scores[i])

print(scores)
avg_rmse_ridge = scores.mean()
print("\n\nAvg RMSE is ",scores.mean())


clf = LinearRegression()
scores = cross_val_score(clf, X_no_rain, y, cv=5, scoring='neg_mean_squared_error')

for i in range(0,5):
    scores[i] = sqrt(-1*scores[i])

print(scores)
avg_rmse_no_rain = scores.mean()
print("\n\nAvg RMSE is ",scores.mean())


clf = linear_model.Lasso(alpha = 0.01)
scores = cross_val_score(clf, X_old, y, cv=5, scoring='neg_mean_squared_error')

for i in range(0,5):
    scores[i] = sqrt(-1*scores[i])

print(scores)
avg_rmse_las = scores.mean()
print("\n\nAvg RMSE is ",scores.mean())


I am working on the file "Baseline.ipynb".
The first task is:
Avg RMSE with Linear Regression : 51874.9
Avg RMSE with Ridge Regression : 51840.9
So Ridge is slightly better than Linear
Lasso Regression
Can you write Python code for it?

from sklearn import linear_model

reg = linear_model.LassoCV(alphas=[0.01,0.02,0.03,0.04,0.05,0.06,0.07,0.08,0.09,0.1])
reg.fit(X_old, y)
reg.alpha_

X_train, X_test, y_train, y_test = train_test_split(X_old, y, test_size=0.2, random_state=1)

las = linear_model.Lasso(alpha = 0.01)
las.fit (X_train, y_train)
print(las.coef_)

y_predict = las.predict(X_test)
rmse = sqrt(mean_squared_error(y_predict, y_test))
print(rmse)


I am working on the file "Baseline.ipynb".
The first task is:
Avg RMSE with original 4 features  :  51874.9
Avg RMSE with 5 features(soil too) : 52272.6
So there was NO improvement in the RMSE, due to the new feature.
Now lets try by removing the rainfall features
Can you write Python code for it?

X_no_rain = X[["X1_zscore", "X2_zscore"]]

X_train, X_test, y_train, y_test = train_test_split(X_no_rain, y, test_size=0.2, random_state=1)

alg = LinearRegression()
alg.fit(X_train, y_train)

coef = alg.coef_
intercept = alg.intercept_

coef = coef.round(decimals=2)
pp = pprint.PrettyPrinter()
pp.pprint(coef)
pp.pprint(intercept)

y_predict = alg.predict(X_test)
yp = y_predict
yt = y_test.as_matrix()
rmse = sqrt(mean_squared_error(y_predict, y_test))
print(rmse)


I am working on the file "Baseline.ipynb".
The first task is:
Avg RMSE with the original 4 features : 51874.9
Avg RMSE without the 2  rain features : 51921.9
Ridge Regression
Can you write Python code for it?

from sklearn import linear_model

reg = linear_model.RidgeCV(alphas=[1,2,3,4,5,6,7,7.1,7.2,7.3,8,9,10])
reg.fit(X_old, y)
reg.alpha_

X_train, X_test, y_train, y_test = train_test_split(X_old, y, test_size=0.2, random_state=1)

reg = linear_model.Ridge(alpha = 7.1)
reg.fit (X_train, y_train)
print(reg.coef_)

y_predict = reg.predict(X_test)
rmse = sqrt(mean_squared_error(y_predict, y_test))
print(rmse)