def create_master_turnstile_file(filenames, output_file):
    '''
    Write a function that
    - takes the files in the list filenames, which all have the
    columns 'C/A, UNIT, SCP, DATEn, TIMEn, DESCn, ENTRIESn, EXITSn', and
    - consolidates them into one file located at output_file.
    - There should be ONE row with the column headers, located at the top of the file.
    - The input files do not have column header rows of their own.

    For example, if file_1 has:
    line 1 ...
    line 2 ...

    and another file, file_2 has:
    line 3 ...
    line 4 ...
    line 5 ...

    We need to combine file_1 and file_2 into a master_file like below:
     'C/A, UNIT, SCP, DATEn, TIMEn, DESCn, ENTRIESn, EXITSn'
    line 1 ...
    line 2 ...
    line 3 ...
    line 4 ...
    line 5 ...
    '''
    with open(output_file, 'w') as master_file:
        master_file.write('C/A,UNIT,SCP,DATEn,TIMEn,DESCn,ENTRIESn,EXITSn\n')
        for filename in filenames:
            with open(filename, 'r') as content:
                # Write everything read from `content` ( which is rows) to file `master_file`
                master_file.write(content.read())

return None


I am working on the file "1-uIDS-quiz/ps3-AnalyzingSubwayData.ipynb".
The first task is:
Quiz: 8 - More Linear Regression (Optional)
Before choosing to use Linear Regression, follow a checklist like this:
http://www.graphpad.com/guides/prism/6/curve-fitting/index.htm?reg_analysischeck_linearreg.htm
Analysis checklist: Linear regression

[ ] Can the relationship between X and Y be graphed as a straight line?
    In many experiments the relationship between X and Y is curved, making linear regression inappropriate. It rarely helps to transform the data to force the relationship to be linear. Better, use nonlinear curve fitting.

[ ] Is the scatter of data around the line Gaussian (at least approximately)?
    Linear regression analysis assumes that the scatter of data around the best-fit line is Gaussian.

[ ] Is the variability the same everywhere?
    Linear regression assumes that scatter of points around the best-fit line has the same standard deviation all along the curve. The assumption is violated if the points with high or low X values tend to be further from the best-fit line. The assumption that the standard deviation is the same everywhere is termed homoscedasticity. (If the scatter goes up as Y goes up, you need to perform a weighted regression. Prism can't do this via the linear regression analysis. Instead, use nonlinear regression but choose to fit to a straight-line model.

[ ] Do you know the X values precisely?
    The linear regression model assumes that X values are exactly correct, and that experimental error or biological variability only affects the Y values. This is rarely the case, but it is sufficient to assume that any imprecision in measuring X is very small compared to the variability in Y.

[ ] Are the data points independent?
    Whether one point is above or below the line is a matter of chance, and does not influence whether another point is above or below the line.

[ ] Are the X and Y values intertwined?
    If the value of X is used to calculate Y (or the value of Y is used to calculate X) then linear regression calculations are invalid. One example is a...
Can you write Python code for it?

# -*- coding: utf-8 -*-

import numpy as np
import pandas
import scipy
import statsmodels.api as sm

"""

"""

def predictions(weather_turnstile):
    #
    # Your implementation goes here. Feel free to write additional
    # helper functions
    #
    return prediction


I am working on the file "1-uIDS-quiz/ps3-AnalyzingSubwayData.ipynb".
The first task is:
Result:
(1105.4463767458733, 1090.278780151855, 1924409167.0, 0.024940392294493356)
Quiz: 4 - Ridership On Rainy Vs. Nonrainy Days
Q: Is the distribution of the number of entries statistically different between rainy and non rainy days?
A: Yes
For more intuitions about this test:
https://www.graphpad.com/guides/prism/7/statistics/how_the_mann-whitney_test_works.htm?toc=0&printWindow
Quiz: 5 - Linear Regression
Task description
In this question, you need to:
- 1) Implement the compute_cost() and gradient_descent() procedures
- 2) Select features (in the predictions procedure) and make predictions.
Note:
Plotting your cost history will help you convince yourself that gradient descent has converged to the minimum cost, but it's more of a learning tool than data to include in a report.
Can you write Python code for it?

import csv
import numpy as np
import pandas as pd
from ggplot import *

"""
In this question, you need to:
1) implement the compute_cost() and gradient_descent() procedures
2) Select features (in the predictions procedure) and make predictions.

"""

def normalize_features(df):
    """
    Normalize the features in the data set.
    """
    mu = df.mean()
    sigma = df.std()

    if (sigma == 0).any():
        raise Exception("One or more features had the same value for all samples, and thus could " + \
                         "not be normalized. Please do not include features with only a single value " + \
                         "in your model.")
    df_normalized = (df - df.mean()) / df.std()

    return df_normalized, mu, sigma

def compute_cost(features, values, theta):
    """
    Compute the cost function given a set of features / values,
    and the values for our thetas.

    This can be the same code as the compute_cost function in the lesson #3 exercises,
    but feel free to implement your own.
    """

    # your code here
    m = len(values)
    sum_of_square_errors = np.square(np.dot(features, theta) - values)
    cost = sum_of_square_errors / (2*m)

    return cost

def gradient_descent(features, values, theta, alpha, num_iterations):
    """
    Perform gradient descent given a data set with an arbitrary number of features.

    This can be the same gradient descent code as in the lesson #3 exercises,
    but feel free to implement your own.
    """

    m = len(values)
    cost_history = []

    for i in range(num_iterations):
        # Append new computed cost of given list of thets to cost_history list
        cost_history.append(compute_cost(features, values, theta))
        # Compute
        diff = np.dot(features.transpose(), values - np.dot(features, theta))
        theta += (alpha/len(values))*diff

    return theta, pandas.Series(cost_history)

def predictions(dataframe):
    '''
    The NYC turnstile data is stored in a pandas dataframe called...


I am working on the file "1. ANN/1.1 Introduction - Deep Learning and ANN.ipynb".
The first task is:
Exercise:
Create Neural networks with 10 hidden nodes on the above code.
What's the impact on accuracy?
Can you write Python code for it?

# Put your code here
#(or load the solution if you wanna cheat :-)

# %load ../solutions/sol_111.py


I am working on the file "1. ANN/1.1 Introduction - Deep Learning and ANN.ipynb".
The first task is:
Our neural networks class
When we first create a neural networks architecture, we need to know the number of inputs, number of hidden layers and number of outputs.
The weights have to be randomly initialized.
```python
class MLP:
    def init(self, ni, nh, no):
        # number of input, hidden, and output nodes
        self.ni = ni + 1 # +1 for bias node
        self.nh = nh
        self.no = no
    # activations for nodes
    self.ai = [1.0]*self.ni
    self.ah = [1.0]*self.nh
    self.ao = [1.0]*self.no

    # create weights
    self.wi = makeMatrix(self.ni, self.nh)
    self.wo = makeMatrix(self.nh, self.no)

    # set them to random vaules
    self.wi = rand(-0.2, 0.2, size=self.wi.shape)
    self.wo = rand(-2.0, 2.0, size=self.wo.shape)

    # last change in weights for momentum
    self.ci = makeMatrix(self.ni, self.nh)
    self.co = makeMatrix(self.nh, self.no)

```
Activation Function
```python
def activate(self, inputs):
if len(inputs) != self.ni-1:
    print(inputs)
    raise ValueError('wrong number of inputs')

# input activations
for i in range(self.ni-1):
    self.ai[i] = inputs[i]

# hidden activations
for j in range(self.nh):
    sum_h = 0.0
    for i in range(self.ni):
        sum_h += self.ai[i] * self.wi[i][j]
    self.ah[j] = sigmoid(sum_h)

# output activations
for k in range(self.no):
    sum_o = 0.0
    for j in range(self.nh):
        sum_o += self.ah[j] * self.wo[j][k]
    self.ao[k] = sigmoid(sum_o)

return self.ao[:]

```
BackPropagation
```python
def backPropagate(self, targets, N, M):
if len(targets) != self.no:
    print(targets)
    raise ValueError('wrong number of target values')

# calculate error terms for output
output_deltas = np.zeros(self.no)
for k in range(self.no):
    error = targets[k]-self.ao[k]
    output_deltas[k] = dsigmoid(self.ao[k]) * error

# calculate error terms for hidden
hidden_deltas = np.zeros(self.nh)
for j in range(self.nh):
    error = 0.0
    for k in range(self.no):
        error += output_deltas[k]*self.wo[j][k]
    hidden_deltas[j] =...
Can you write Python code for it?

# Putting all together

class MLP:
    def __init__(self, ni, nh, no):
        # number of input, hidden, and output nodes
        self.ni = ni + 1 # +1 for bias node
        self.nh = nh
        self.no = no

        # activations for nodes
        self.ai = [1.0]*self.ni
        self.ah = [1.0]*self.nh
        self.ao = [1.0]*self.no

        # create weights
        self.wi = makeMatrix(self.ni, self.nh)
        self.wo = makeMatrix(self.nh, self.no)

        # set them to random vaules
        for i in range(self.ni):
            for j in range(self.nh):
                self.wi[i][j] = rand(-0.2, 0.2)
        for j in range(self.nh):
            for k in range(self.no):
                self.wo[j][k] = rand(-2.0, 2.0)

        # last change in weights for momentum
        self.ci = makeMatrix(self.ni, self.nh)
        self.co = makeMatrix(self.nh, self.no)

    def backPropagate(self, targets, N, M):

        if len(targets) != self.no:
            print(targets)
            raise ValueError('wrong number of target values')

        # calculate error terms for output
        output_deltas = np.zeros(self.no)
        for k in range(self.no):
            error = targets[k]-self.ao[k]
            output_deltas[k] = dsigmoid(self.ao[k]) * error

        # calculate error terms for hidden
        hidden_deltas = np.zeros(self.nh)
        for j in range(self.nh):
            error = 0.0
            for k in range(self.no):
                error += output_deltas[k]*self.wo[j][k]
            hidden_deltas[j] = dsigmoid(self.ah[j]) * error

        # update output weights
        for j in range(self.nh):
            for k in range(self.no):
                change = output_deltas[k] * self.ah[j]
                self.wo[j][k] += N*change + M*self.co[j][k]
                self.co[j][k] = change

        # update input weights
        for i in range(self.ni):
            for j in range(self.nh):
                change = hidden_deltas[j]*self.ai[i]
                self.wi[i][j] +=...


I am working on the file "1. ANN/1.1 Introduction - Deep Learning and ANN.ipynb".
The first task is:
Predicting on training dataset and measuring in-sample accuracy
Can you write Python code for it?

%timeit -n 1 -r 1 ann.test(X)

prediction = pd.DataFrame(data=np.array([y, np.ravel(ann.predict)]).T,
                          columns=["actual", "prediction"])
prediction.head()

np.min(prediction.prediction)


I am working on the file "1. ANN/1.1 Introduction - Deep Learning and ANN.ipynb".
The first task is:
Running the model on our dataset
Can you write Python code for it?

# create a network with two inputs, one hidden, and one output nodes
ann = MLP(2, 1, 1)

%timeit -n 1 -r 1 ann.train(zip(X,y), iterations=2)


I am working on the file "1. ANN/1.1 Introduction - Deep Learning and ANN.ipynb".
The first task is:
Start Building our MLP building blocks
Note: This process will eventually result in our own Neural Networks class
A look at the details
<img src="../imgs/mlp_details.png" width="65%" />
Can you write Python code for it?

import random
random.seed(123)

# calculate a random number where:  a <= rand < b
def rand(a, b):
    return (b-a)*random.random() + a


I am working on the file "1. ANN/1.1.1 Perceptron and Adaline.ipynb".
The first task is:

A function for plotting decision regions
Can you write Python code for it?

from matplotlib.colors import ListedColormap

def plot_decision_regions(X, y, classifier, resolution=0.02):

    # setup marker generator and color map
    markers = ('s', 'x', 'o', '^', 'v')
    colors = ('red', 'blue', 'lightgreen', 'gray', 'cyan')
    cmap = ListedColormap(colors[:len(np.unique(y))])

    # plot the decision surface
    x1_min, x1_max = X[:, 0].min() - 1, X[:, 0].max() + 1
    x2_min, x2_max = X[:, 1].min() - 1, X[:, 1].max() + 1
    xx1, xx2 = np.meshgrid(np.arange(x1_min, x1_max, resolution),
                         np.arange(x2_min, x2_max, resolution))
    Z = classifier.predict(np.array([xx1.ravel(), xx2.ravel()]).T)
    Z = Z.reshape(xx1.shape)
    plt.contourf(xx1, xx2, Z, alpha=0.4, cmap=cmap)
    plt.xlim(xx1.min(), xx1.max())
    plt.ylim(xx2.min(), xx2.max())

    # plot class samples
    for idx, cl in enumerate(np.unique(y)):
        plt.scatter(x=X[y == cl, 0], y=X[y == cl, 1],
                    alpha=0.8, c=cmap(idx),
                    marker=markers[idx], label=cl)

plot_decision_regions(X, y, classifier=ppn)
plt.xlabel('sepal length [cm]')
plt.ylabel('petal length [cm]')
plt.legend(loc='upper left')

plt.tight_layout()
plt.show()


I am working on the file "1. ANN/1.1.1 Perceptron and Adaline.ipynb".
The first task is:

Adaptive linear neurons and the convergence of learning
[back to top]
Implementing an adaptive linear neuron in Python
Can you write Python code for it?

class AdalineGD(object):
    """ADAptive LInear NEuron classifier.

    Parameters
    ------------
    eta : float
        Learning rate (between 0.0 and 1.0)
    n_iter : int
        Passes over the training dataset.

    Attributes
    -----------
    w_ : 1d-array
        Weights after fitting.
    errors_ : list
        Number of misclassifications in every epoch.

    """
    def __init__(self, eta=0.01, n_iter=50):
        self.eta = eta
        self.n_iter = n_iter

    def fit(self, X, y):
        """ Fit training data.

        Parameters
        ----------
        X : {array-like}, shape = [n_samples, n_features]
            Training vectors, where n_samples is the number of samples and
            n_features is the number of features.
        y : array-like, shape = [n_samples]
            Target values.

        Returns
        -------
        self : object

        """
        self.w_ = np.zeros(1 + X.shape[1])
        self.cost_ = []

        for i in range(self.n_iter):
            output = self.net_input(X)
            errors = (y - output)
            self.w_[1:] += self.eta * X.T.dot(errors)
            self.w_[0] += self.eta * errors.sum()
            cost = (errors**2).sum() / 2.0
            self.cost_.append(cost)
        return self

    def net_input(self, X):
        """Calculate net input"""
        return np.dot(X, self.w_[1:]) + self.w_[0]

    def activation(self, X):
        """Compute linear activation"""
        return self.net_input(X)

    def predict(self, X):
        """Return class label after unit step"""
        return np.where(self.activation(X) >= 0.0, 1, -1)

fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(8, 4))

ada1 = AdalineGD(n_iter=10, eta=0.01).fit(X, y)
ax[0].plot(range(1, len(ada1.cost_) + 1), np.log10(ada1.cost_), marker='o')
ax[0].set_xlabel('Epochs')
ax[0].set_ylabel('log(Sum-squared-error)')
ax[0].set_title('Adaline - Learning rate 0.01')

ada2 = AdalineGD(n_iter=10, eta=0.0001).fit(X, y)
ax[1].plot(range(1, len(ada2.cost_) + 1), ada2.cost_,...


I am working on the file "1. ANN/1.1.1 Perceptron and Adaline.ipynb".
The first task is:

Large scale machine learning and stochastic gradient descent
[back to top]
Can you write Python code for it?

from numpy.random import seed

class AdalineSGD(object):
    """ADAptive LInear NEuron classifier.

    Parameters
    ------------
    eta : float
        Learning rate (between 0.0 and 1.0)
    n_iter : int
        Passes over the training dataset.

    Attributes
    -----------
    w_ : 1d-array
        Weights after fitting.
    errors_ : list
        Number of misclassifications in every epoch.
    shuffle : bool (default: True)
        Shuffles training data every epoch if True to prevent cycles.
    random_state : int (default: None)
        Set random state for shuffling and initializing the weights.

    """
    def __init__(self, eta=0.01, n_iter=10, shuffle=True, random_state=None):
        self.eta = eta
        self.n_iter = n_iter
        self.w_initialized = False
        self.shuffle = shuffle
        if random_state:
            seed(random_state)

    def fit(self, X, y):
        """ Fit training data.

        Parameters
        ----------
        X : {array-like}, shape = [n_samples, n_features]
            Training vectors, where n_samples is the number of samples and
            n_features is the number of features.
        y : array-like, shape = [n_samples]
            Target values.

        Returns
        -------
        self : object

        """
        self._initialize_weights(X.shape[1])
        self.cost_ = []
        for i in range(self.n_iter):
            if self.shuffle:
                X, y = self._shuffle(X, y)
            cost = []
            for xi, target in zip(X, y):
                cost.append(self._update_weights(xi, target))
            avg_cost = sum(cost)/len(y)
            self.cost_.append(avg_cost)
        return self

    def partial_fit(self, X, y):
        """Fit training data without reinitializing the weights"""
        if not self.w_initialized:
            self._initialize_weights(X.shape[1])
        if y.ravel().shape[0] > 1:
            for xi, target in zip(X, y):
                self._update_weights(xi, target)
        else:
     ...


I am working on the file "1. ANN/1.1.1 Perceptron and Adaline.ipynb".
The first task is:

Standardizing features and re-training adaline
Can you write Python code for it?

# standardize features
X_std = np.copy(X)
X_std[:,0] = (X[:,0] - X[:,0].mean()) / X[:,0].std()
X_std[:,1] = (X[:,1] - X[:,1].mean()) / X[:,1].std()

ada = AdalineGD(n_iter=15, eta=0.01)
ada.fit(X_std, y)

plot_decision_regions(X_std, y, classifier=ada)
plt.title('Adaline - Gradient Descent')
plt.xlabel('sepal length [standardized]')
plt.ylabel('petal length [standardized]')
plt.legend(loc='upper left')
plt.tight_layout()
plt.show()

plt.plot(range(1, len(ada.cost_) + 1), ada.cost_, marker='o')
plt.xlabel('Epochs')
plt.ylabel('Sum-squared-error')

plt.tight_layout()
plt.show()


I am working on the file "1. ANN/1.1.1 Perceptron and Adaline.ipynb".
The first task is:

Training a perceptron model on the Iris dataset
[back to top]
Reading-in the Iris data
Can you write Python code for it?

import numpy as np
import pandas as pd
from sklearn.datasets import load_iris

iris = load_iris()
X = iris.data
y = iris.target
data = np.hstack((X, y[:, np.newaxis]))

labels = iris.target_names
features = iris.feature_names

df = pd.DataFrame(data, columns=iris.feature_names+['label'])
df.label = df.label.map({k:v for k,v in enumerate(labels)})
df.tail()


I am working on the file "1. Analysis.ipynb".
The first task is:
a. histogram
plot the histgram of one feature, to see what distribution the feature is.
Can you write Python code for it?

import matplotlib.pyplot as plt
%matplotlib inline

feature_table=[1,10,100,300]
for ind,fea in enumerate(feature_table):
    f= lld.feature[:,fea]

    plt.subplot(2,2,ind+1)
    plt.hist(f)
    #plt.title("Histogram of feature "+str(ind))
plt.axis('tight')


I am working on the file "1. Basic Concepts I.ipynb".
The first task is:
An important feature of gradient descent is that there should be a visible improvement over time: In this example, we simply plotted the squared distance from the local minima calculated by gradient descent and the true local minimum,  cost, against the iteration during which it was calculated. As we can see, the distance gets smaller over time, but barely changes in later iterations.
Can you write Python code for it?

TOOLS = [WheelZoomTool(), ResetTool(), PanTool()]

x, y = (zip(*enumerate(cost)))
s1 = bp.figure(width=W,
               height=H,
               title='Squared distance to true local minimum',
#                title_text_font_size='14pt',
               tools=TOOLS,
               x_axis_label = 'Iteration',
               y_axis_label = 'Distance'
)
s1.line(x, y, color="navy", alpha=0.5, line_width=3)
s1.title_text_font_size = '16pt'
s1.yaxis.axis_label_text_font_size = "14pt"
s1.xaxis.axis_label_text_font_size = "14pt"

bp.show(s1)


I am working on the file "1. Basic Concepts I.ipynb".
The first task is:
Complete the following code and look at the plot of the first gradient descent updates. Explore the behavior of the proposed learning rates.
Can you write Python code for it?

def gradient(w, x, y):
    return 2 * x * (nn(x, w) - y)

def step(w_k, x, y, learning_rate):
    return learning_rate * gradient(w_k, x, y).sum()

w = 0.01

# define a learning_rate
learning_rate = 0.1

nb_of_iterations = 20
w_cost = [(w, cost(nn(x, w), y))]
for i in range(nb_of_iterations):
    # Here your code
    w_cost.append((w, cost(nn(x, w), y)))

for i in range(0, len(w_cost)):
    print('w({}): {:.4f} \t cost: {:.4f}'.format(i, w_cost[i][0], w_cost[i][1]))

# Plotting the first gradient descent updates
plt.plot(ws, cost_ws, 'r-')  # Plot the error curve
# Plot the updates
for i in range(1, len(w_cost)-2):
    w1, c1 = w_cost[i-1]
    w2, c2 = w_cost[i]
    plt.plot(w1, c1, 'bo')
    plt.plot([w1, w2],[c1, c2], 'b-')
    plt.text(w1, c1+0.5, '$w({})$'.format(i))
# Plot the last weight, axis, and show figure
w1, c1 = w_cost[len(w_cost)-3]
plt.plot(w1, c1, 'bo')
plt.text(w1, c1+0.5, '$w({})$'.format(nb_of_iterations))
plt.xlabel('$w$', fontsize=15)
plt.ylabel('$\\xi$', fontsize=15)
plt.title('Gradient descent updates plotted on cost function')
plt.grid()
plt.gcf().set_size_inches((10,6))
plt.show()

w = 0
nb_of_iterations = 10
for i in range(nb_of_iterations):
    dw = step(w, x, y, learning_rate)
    w = w - dw

plt.plot(x, y, 'o', label='t')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)')
plt.plot([0, 1], [0*w, 1*w], 'r-', label='fitted line')
plt.xlabel('input x')
plt.ylabel('target t')
plt.ylim([0,2])
plt.title('input vs. target')
plt.grid()
plt.legend(loc=2)
plt.gcf().set_size_inches((10,6))
plt.show()


I am working on the file "1. Basic Concepts I.ipynb".
The first task is:
It can be shown that the “centered difference formula" is better when computing numerical derivatives:
$$ \lim_{h \rightarrow 0} \frac{f(x + h) - f(x - h)}{2h} $$
The error in the "finite difference" approximation can be derived from Taylor's theorem and, assuming that $f$ is differentiable, is $O(h)$. In the case of “centered difference" the error is $O(h^2)$.
The derivative tells how to chage $x$ in order to make a small improvement in $f$.
Then, we can follow these steps to decrease the value of the function:

Start from a random $x$ value.
Compute the derivative $f'(x) = \lim_{h \rightarrow 0} \frac{f(x + h) - f(x - h)}{2h}$.
Walk a small step in the opposite direction of the derivative, because we know that $f(x - h \mbox{ sign}(f'(x))$ is less than $f(x)$ for  small enough $h$.

The search for the minima ends when the derivative is zero because we have no more information about which direction to move. $x$ is a critical o stationary point if $f'(x)=0$.

A minimum (maximum) is a critical point where $f(x)$ is lower (higher) than at all neighboring points.
There is a third class of critical points: saddle points.

If $f$ is a convex function, this should be the minimum (maximum) of our functions. In other cases it could be a local minimum (maximum) or a saddle point.
Can you write Python code for it?

W = 400
H = 250
bp.output_notebook()

x = np.linspace(-15,15,100)
y = x**2

TOOLS = [WheelZoomTool(), ResetTool(), PanTool()]

s1 = bp.figure(width=W, plot_height=H,
               title='Local minimum of function',
               tools=TOOLS)
s1.line(x, y, color="navy", alpha=0.5, line_width=3)
s1.circle(0, 0, size =10, color="orange")
s1.title_text_font_size = '12pt'
s1.yaxis.axis_label_text_font_size = "14pt"
s1.xaxis.axis_label_text_font_size = "14pt"

bp.show(s1)

x = np.linspace(-15,15,100)
y = -x**2

TOOLS = [WheelZoomTool(), ResetTool(), PanTool()]

s1 = bp.figure(width=W, plot_height=H,
               title='Local maximum of function',
               tools=TOOLS)
s1.line(x, y, color="navy", alpha=0.5, line_width=3)
s1.circle(0, 0, size =10, color="orange")
s1.title_text_font_size = '12pt'
s1.yaxis.axis_label_text_font_size = "14pt"
s1.xaxis.axis_label_text_font_size = "14pt"

bp.show(s1)

x = np.linspace(-15,15,100)
y = x**3

TOOLS = [WheelZoomTool(), ResetTool(), PanTool()]

s1 = bp.figure(width=W, plot_height=H,
               title='Saddle point of function',
               tools=TOOLS)
s1.line(x, y, color="navy", alpha=0.5, line_width=3)
s1.circle(0, 0, size =10, color="orange")
s1.title_text_font_size = '12pt'
s1.yaxis.axis_label_text_font_size = "14pt"
s1.xaxis.axis_label_text_font_size = "14pt"

bp.show(s1)


I am working on the file "1. Basic Concepts I.ipynb".
The first task is:
Let's start by choosing a random vector and then walking a step in the opposite direction of the gradient vector. We will stop when the difference between the new solution and the old solution is less than a tolerance value.
Can you write Python code for it?

# choosing a random vector

import random
import numpy as np

x = [random.randint(-10,10) for i in range(3)]
x

def step(x,grad,alpha):
    return [x_i - alpha * grad_i for x_i, grad_i in zip(x,grad)]

tol = 1e-15
alpha = 0.01
while True:
    grad = gradient_centered(x,f)
    next_x = step(x,grad,alpha)
    if euc_dist(next_x,x) < tol:
        break
    x = next_x
print [round(i,10) for i in x]


I am working on the file "1. Basic Concepts I.ipynb".
The first task is:
Second approach
To find the local minimum using gradient descend: you start at a random point, and move into the direction of steepest descent relative to the derivative:

Start from a random $x$ value.
Compute the derivative $f'(x)$ analitically.
Walk a small step in the opposite direction of the derivative.

In this example, let's suppose we start at $x=15$. The derivative at this point is $2×15−6=24$.
Because we're using gradient descent, we need to subtract the gradient from our $x$-coordinate: $f(x - f'(x))$. However, notice that $15−24$ gives us $−9$, clearly overshooting over target of $3$.
Can you write Python code for it?

x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5
start = 15

TOOLS = [WheelZoomTool(), ResetTool(), PanTool()]

s1 = bp.figure(width=W, plot_height=H,
               tools=TOOLS)
s1.line(x, y, color="navy", alpha=0.5, line_width=3)
s1.circle(start, start**2 - 6*start + 5, size =10, color="orange")

d = 2 * start - 6
end = start - d

s1.circle(end, end**2 - 6*end + 5, size =10, color="red")
s1.title_text_font_size = '12pt'
s1.yaxis.axis_label_text_font_size = "14pt"
s1.xaxis.axis_label_text_font_size = "14pt"

bp.show(s1)


I am working on the file "1. Basic Concepts I.ipynb".
The first task is:
Stochastic Gradient Descend
The last function evals the whole dataset $(\mathbf{x}_i,y_i)$ at every step.
If the dataset is large, this strategy is too costly. In this case we will use a strategy called SGD (Stochastic Gradient Descend).
When learning from data, the cost function is additive: it is computed by adding sample reconstruction errors.
Then, we can compute the estimate the gradient (and move towards the minimum) by using only one data sample (or a small data sample).
Thus, we will find the minimum by iterating this gradient estimation over the dataset.
A full iteration over the dataset is called epoch. During an epoch, data must be used in a random order.
If we apply this method we have some theoretical guarantees to find the minimum.
Can you write Python code for it?

import numpy as np
x = range(10)
y = [2*i for i in x]
data = zip(x,y)

def in_random_order(data):
    import random
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

for (x_i,y_i) in in_random_order(data):
    print x_i,y_i

def gradient_f_SGD(x,y,w):
    import numpy as np
    return 2*w*(np.array(x)**2) - 2*np.array(x)*np.array(y)

def SGD(target_f, gradient_f, x, y, alpha_0=0.01):
    import numpy as np
    import random
    data = zip(x,y)
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    iteration_no_increase = 0
    while iteration_no_increase < 100:
        val = sum(target_f(x_i, y_i, w) for x_i,y_i in data)
        if val < min_val:
            min_w, min_val = w, val
            iteration_no_increase = 0
            alpha = alpha_0
        else:
            iteration_no_increase += 1
            alpha *= 0.9
        for x_i, y_i in in_random_order(data):
            gradient_i = gradient_f(x_i, y_i, w)
            w = np.array(w) - (alpha *  np.array(gradient_i))
    return min_w

print "w:", SGD(target_f, gradient_f_SGD, x, y, 0.01)


I am working on the file "1. Basic Concepts I.ipynb".
The first task is:
The limit as $h$ approaches zero, if it exists, should represent the slope of the tangent line to $(x, f(x))$.
For values that are not zero it is only an approximation.
Can you write Python code for it?


I am working on the file "1. Basic Concepts I.ipynb".
The first task is:
There are two problems with numerical derivatives:
+ It is approximate.
+ It is very slow to evaluate (two function evaluations: $f(x + h) , f(x - h)$ ).
Our knowledge from Calculus could help!
We know that we can get an analytical expression of the derivative for some functions.
For example, let's suppose we have a simple quadratic function, $f(x)=x^2−6x+5$, and we want to find the minimum of this function.
First approach
We can solve this analytically using Calculus, by finding the derivate $f'(x)  =  2x-6$ and setting it to zero:
\begin{equation}
\begin{split}
2x-6  & = & 0 \
2x  & = & 6 \
x & = & 3 \
\end{split}
\end{equation}
Can you write Python code for it?

x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5

TOOLS = [WheelZoomTool(), ResetTool(), PanTool()]

s1 = bp.figure(width=W, plot_height=H,
               tools=TOOLS)
s1.line(x, y, color="navy", alpha=0.5, line_width=3)
s1.circle(3, 3**2 - 6*3 + 5, size =10, color="orange")
s1.title_text_font_size = '12pt'
s1.yaxis.axis_label_text_font_size = "14pt"
s1.xaxis.axis_label_text_font_size = "14pt"

bp.show(s1)


I am working on the file "1. Basic Concepts I.ipynb".
The first task is:
To fix this, we multiply the gradient by a step size. This step size (often called alpha) has to be chosen carefully, as a value too small will result in a long computation time, while a value too large will not give you the right result (by overshooting) or even fail to converge.
In this example, we'll set the step size to 0.01, which means we'll subtract $24×0.01$ from $15$, which is $14.76$.
This is now our new temporary local minimum: We continue this method until we either don't see a change after we subtracted the derivative step size, or until we've completed a pre-set number of iterations.
Can you write Python code for it?

old_min = 0
temp_min = 15
step_size = 0.01
precision = 0.0001

def f_derivative(x):
    import math
    return 2*x -6

mins = []
cost = []

while abs(temp_min - old_min) > precision:
    old_min = temp_min
    gradient = f_derivative(old_min)
    move = gradient * step_size
    temp_min = old_min - move
    cost.append((3-temp_min)**2)
    mins.append(temp_min)

# rounding the result to 2 digits because of the step size
print "Local minimum occurs at {:3.2f}.".format(round(temp_min,2))


I am working on the file "1. Basic Concepts.ipynb".
The first task is:
Alpha
The step size, alpha, is a slippy concept: if it is too small we will slowly converge to the solution, if it is too large we can diverge from the solution.
There are several policies to follow when selecting the step size:

Constant size steps. In this case, the size step determines the precision of the solution.
Decreasing step sizes.
At each step, select the optimal step.

The last policy is good, but too expensive. In this case we would consider a fixed set of values:
Can you write Python code for it?


I am working on the file "1. Basic Concepts.ipynb".
The first task is:
An important feature of gradient descent is that there should be a visible improvement over time: In this example, we simply plotted the squared distance from the local minima calculated by gradient descent and the true local minimum,  cost, against the iteration during which it was calculated. As we can see, the distance gets smaller over time, but barely changes in later iterations.
Can you write Python code for it?

x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5

x, y = (zip(*enumerate(cost)))

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-', alpha=0.7)
plt.ylim([-10,150])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show

x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.plot(mins,cost,'o', alpha=0.3)
ax.text(start,
        start**2 - 6*start + 25,
        'Start',
        ha='center',
        color=sns.xkcd_rgb['blue'],
       )
ax.text(mins[-1],
        cost[-1]+20,
        'End (%s steps)' % len(mins),
        ha='center',
        color=sns.xkcd_rgb['blue'],
       )
plt.show


I am working on the file "1. Basic Concepts.ipynb".
The first task is:
Basic Concepts
What is "learning from data"?

In general Learning from Data is a scientific discipline that is concerned with the design and development of algorithms that allow computers to infer (from data) a  model that allows compact representation (unsupervised learning) and/or good generalization (supervised learning).

This is an important technology because it enables computational systems to adaptively improve their performance with experience accumulated from the observed data.
Most of these algorithms are based on the iterative solution of a mathematical problem that involves data and model. If there was an analytical solution to the problem, this should be the adopted one, but this is not the case for most of the cases.
So, the most common strategy for learning from data is based on solving a system of equations as a way to find a series of parameters of the model that minimizes a mathematical problem. This is called optimization.
The most important technique for solving optimization problems is gradient descend.
Preliminary: Nelder-Mead method for function minimization.
The most simple thing we can try to minimize a function $f(x)$ would be to sample two points relatively near each other, and just repeatedly take a step down away from the largest value. This simple algorithm has a severe limitation: it can't get closer to the true minima than the step size.
The Nelder-Mead method dynamically adjusts the step size based off the loss of the new point. If the new point is better than any previously seen value, it expands the step size to accelerate towards the bottom. Likewise if the new point is worse it contracts the step size to converge around the minima. The usual settings are to half the step size when contracting and double the step size when expanding.
This method can be easily extended into higher dimensional examples, all that's required is taking one more point than there are dimensions. Then, the simplest approach is to replace the worst point with a point reflected through the...
Can you write Python code for it?

# numerical derivative at a point x

def f(x):
    return x**2

def fin_dif(x,
            f,
            h = 0.00001):
    '''
    This method returns the derivative of f at x
    by using the finite difference method
    '''
    return (f(x+h) - f(x))/h

x = 2.0
print "{:2.4f}".format(fin_dif(x,f))


I am working on the file "1. Basic Concepts.ipynb".
The first task is:
Complete the following code in order to:
+ Compute the value of $w$ by using a estimator based on minimizing the squared error.
+ Get from SGD function a vector, target_value, representing the value of the target function at each iteration.
Can you write Python code for it?

# Write your target function as f_target 1/n Sum (y - wx)**2
def target_f(x,y,w):
    # your code here

# Write your gradient function
def gradient_f(x,y,w):
    # your code here

def in_random_order(data):
    '''
    Random data generator
    '''
    import random
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

# Modify the SGD function to return a 'target_value' vector
def SGD(target_f,
        gradient_f,
        x,
        y,
        toler = 1e-6,
        epochs=100,
        alpha_0=0.01):

    # Insert your code among the following lines

    data = zip(x,y)
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    iteration_no_increase = 0
    epoch = 0
    while epoch < epochs and iteration_no_increase < 100:
        val = target_f(x, y, w)
        if min_val - val > toler:
            min_w, min_val = w, val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            iteration_no_increase += 1
            alpha *= 0.95
        for x_i, y_i in in_random_order(data):
            gradient_i = gradient_f(x_i, y_i, w)
            w = w - (alpha *  gradient_i)
        epoch += 1
    return min_w

# Print the value of the solution

w, target_value = SGD(target_f, gradient_f, x, y)
print 'w: {:.6f}'.format(w)

# Visualize the solution regression line

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x, y, 'o', label='t')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)', alpha=0.5)
plt.plot([0, 1], [0*w, 1*w], 'r-', label='fitted line', alpha=0.5, linestyle='--')
plt.xlabel('input x')
plt.ylabel('target t')
plt.title('input vs. target')
plt.ylim([0,2])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show

# Visualize the evolution of the target function value during iterations.

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(np.arange(target_value.size), target_value, 'o', alpha =...


I am working on the file "1. Basic Concepts.ipynb".
The first task is:
Exercise: Stochastic Gradient Descent and Linear Regression
The linear regression model assumes a linear relationship between data:
$$ y_i = w_1 x_i + w_0 $$
Let's generate a more realistic dataset (with noise), where $w_1 = 2$ and $w_0 = 0$.
Can you write Python code for it?

%reset
import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import random

%matplotlib inline

# x: input data
# y: noisy output data

x = np.random.uniform(0,1,20)

# f = 2x + 0
def f(x): return 2*x + 0

noise_variance =0.1
noise = np.random.randn(x.shape[0])*noise_variance
y = f(x) + noise

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.xlabel('$x$', fontsize=15)
plt.ylabel('$f(x)$', fontsize=15)
plt.plot(x, y, 'o', label='y')
plt.plot([0, 1], [f(0), f(1)], 'b-', label='f(x)')
plt.ylim([0,2])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
plt.show


I am working on the file "1. Basic Concepts.ipynb".
The first task is:
Let's start by choosing a random vector and then walking a step in the opposite direction of the gradient vector. We will stop when the difference between the new solution and the old solution is less than a tolerance value.
Can you write Python code for it?

# choosing a random vector

import random
import numpy as np

x = [random.randint(-10,10) for i in range(3)]
x

def step(x,
         grad,
         alpha):
    '''
    This function makes a step in the opposite direction of the gradient vector
    in order to compute a new value for the target function.
    '''
    return [x_i - alpha * grad_i for x_i, grad_i in zip(x,grad)]

tol = 1e-15
alpha = 0.01
while True:
    grad = gradient_centered(x,f)
    next_x = step(x,grad,alpha)
    if euc_dist(next_x,x) < tol:
        break
    x = next_x
print [round(i,10) for i in x]


I am working on the file "1. Basic Concepts.ipynb".
The first task is:
Loss Funtions
Loss functions $L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i \ell(y_i, f(\mathbf{x_i}))$ represent the price paid for inaccuracy of predictions in classification/regression problems.
In classification this function is often the zero-one loss, that is, $ \ell(y_i, f(\mathbf{x_i}))$ is zero when $y_i = f(\mathbf{x}_i)$ and one otherwise.
This function is discontinuous with flat regions and is thus extremely hard to optimize using gradient-based methods. For this reason it is usual to consider a proxy to the loss called a surrogate loss function. For computational reasons this is usually convex function. Here we have some examples:
Square / Euclidean Loss
In regression problems, the most common loss function is the square loss function:
$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i (y_i - f(\mathbf{x}_i))^2  $$
The square loss function can be re-written and utilized for classification:
$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i (1 - y_i f(\mathbf{x}_i))^2  $$
Hinge / Margin Loss (i.e. Suport Vector Machines)
The hinge loss function is defined as:
$$ L(y, f(\mathbf{x})) = \frac{1}{n} \sum_i \mbox{max}(0, 1 - y_i f(\mathbf{x}_i))  $$
The hinge loss provides a relatively tight, convex upper bound on the 0–1 Loss.
<img src="images/loss_functions.png">
Logistic Loss (Logistic Regression)
This function displays a similar convergence rate to the hinge loss function, and since it is continuous, simple gradient descent methods can be utilized.
$$ L(y, f(\mathbf{x})) = \frac{1}{n} log(1 + exp(-y_i f(\mathbf{x}_i))) $$
Sigmoid Cross-Entropy Loss (Softmax classifier)
Cross-Entropy is a loss function that is very used for training multiclass problems. We'll focus on models that assume that classes are mutually exclusive.
In this case, our labels have this form $\mathbf{y}_i =(1.0,0.0,0.0)$. If our model predicts a different distribution, say  $ f(\mathbf{x}_i)=(0.4,0.1,0.5)$, then we'd like to nudge the parameters so that $f(\mathbf{x}_i)$ gets closer to $\mathbf{y}_i$.
C.Shannon showed that if you want to send a...
Can you write Python code for it?

%reset

import warnings
warnings.filterwarnings('ignore')

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.datasets.samples_generator import make_regression
from scipy import stats
import random

%matplotlib inline

# the function that I'm going to plot
def f(x,y):
    return x**2 + 5*y**2

x = np.arange(-3.0,3.0,0.1)
y = np.arange(-3.0,3.0,0.1)
X,Y = np.meshgrid(x, y, indexing='ij') # grid of point
Z = f(X, Y) # evaluation of the function on the grid

plt.pcolor(X, Y, Z, cmap=plt.cm.gist_earth)
plt.axis([x.min(), x.max(), y.min(), y.max()])
plt.gca().set_aspect('equal', adjustable='box')
plt.gcf().set_size_inches((6,6))
plt.show()

def target_f(x):
    return x[0]**2.0 + 5*x[1]**2.0

def part_f(x,
           f,
           i,
           h=1e-6):
    w1 = [x_j + (h if j==i else 0) for j, x_j in enumerate(x)]
    w2 = [x_j - (h if j==i else 0) for j, x_j in enumerate(x)]
    return (f(w1) - f(w2))/(2*h)

def gradient_f(x,
               f,
               h=1e-6):
    return np.array([round(part_f(x,f,i,h), 10) for i,_ in enumerate(x)])

def SGD(target_f,
        gradient_f,
        x,
        alpha_0=0.01,
        toler = 0.000001):
    alpha = alpha_0
    min_val = float('inf')
    steps = 0
    iteration_no_increase = 0
    trace = []
    while iteration_no_increase < 100:
        val = target_f(x)
        if min_val - val > toler:
            min_val = val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            alpha *= 0.95
            iteration_no_increase += 1
        trace.append(x)
        gradient_i = gradient_f(x, target_f)
        x = x - (alpha *  gradient_i)
        steps += 1
    return x, val, steps, trace

x = np.array([2,-2])
x, val, steps, trace = SGD(target_f, gradient_f, x)
print x
print 'Val: {:.6f}, steps: {:.0f}'.format(val, steps)

def SGD_M(target_f,
        gradient_f,
        x,
        alpha_0=0.01,
        toler = 0.000001,
        m = 0.9):

    alpha = alpha_0
    min_val = float('inf')
   ...


I am working on the file "1. Basic Concepts.ipynb".
The first task is:
Second approach
To find the local minimum using gradient descend: you start at a random point, and move into the direction of steepest descent relative to the derivative:

Start from a random $x$ value.
Compute the derivative $f'(x)$ analitically.
Walk a small step in the opposite direction of the derivative.

In this example, let's suppose we start at $x=15$. The derivative at this point is $2×15−6=24$.
Because we're using gradient descent, we need to subtract the gradient from our $x$-coordinate: $f(x - f'(x))$. However, notice that $15−24$ gives us $−9$, clearly overshooting over target of $3$.
Can you write Python code for it?

x = np.linspace(-10,20,100)
y = x**2 - 6*x + 5
start = 15

fig, ax = plt.subplots(1, 1)
fig.set_facecolor('#EAEAF2')
plt.plot(x,y, 'r-')
plt.plot([start],[start**2 - 6*start + 5],'o')
ax.text(start,
        start**2 - 6*start + 35,
        'Start',
        ha='center',
        color=sns.xkcd_rgb['blue'],
       )

d = 2 * start - 6
end = start - d

plt.plot([end],[end**2 - 6*end + 5],'o')
plt.ylim([-10,250])
plt.gcf().set_size_inches((10,3))
plt.grid(True)
ax.text(end,
        start**2 - 6*start + 35,
        'End',
        ha='center',
        color=sns.xkcd_rgb['green'],
       )
plt.show


I am working on the file "1. Basic Concepts.ipynb".
The first task is:
Stochastic Gradient Descend
The last function evals the whole dataset $(\mathbf{x}_i,y_i)$ at every step.
If the dataset is large, this strategy is too costly. In this case we will use a strategy called SGD (Stochastic Gradient Descend).
When learning from data, the cost function is additive: it is computed by adding sample reconstruction errors.
Then, we can compute the estimate the gradient (and move towards the minimum) by using only one data sample (or a small data sample).
Thus, we will find the minimum by iterating this gradient estimation over the dataset.
A full iteration over the dataset is called epoch. During an epoch, data must be used in a random order.
If we apply this method we have some theoretical guarantees to find a good minimum:
+ SGD essentially uses the inaccurate gradient per iteration. Since there is no free food, what is the cost by using approximate gradient? The answer is that the convergence rate is slower than the gradient descent algorithm.
+ The convergence of SGD has been analyzed using the theories of convex minimization and of stochastic approximation: it converges almost surely to a global minimum when the objective function is convex or pseudoconvex, and otherwise converges almost surely to a local minimum.
Can you write Python code for it?

import numpy as np
x = np.arange(10)
y = np.array([2*i for i in x])
data = zip(x,y)

for (x_i,y_i) in data:
    print '{:3d} {:3d}'.format(x_i,y_i)
print

def in_random_order(data):
    '''
    Random data generator
    '''
    import random
    indexes = [i for i,_ in enumerate(data)]
    random.shuffle(indexes)
    for i in indexes:
        yield data[i]

for (x_i,y_i) in in_random_order(data):
    print '{:3d} {:3d}'.format(x_i,y_i)

import numpy as np
import random

def SGD(target_f,
        gradient_f,
        x,
        y,
        toler = 1e-6,
        epochs=100,
        alpha_0=0.01):
    '''
    Stochastic gradient descend with automatic step adaptation (by
    reducing the step to its 95% when there are iterations with no increase)
    '''
    data = zip(x,y)
    w = random.random()
    alpha = alpha_0
    min_w, min_val = float('inf'), float('inf')
    epoch = 0
    iteration_no_increase = 0
    while epoch < epochs and iteration_no_increase < 100:
        val = target_f(x, y, w)
        if min_val - val > toler:
            min_w, min_val = w, val
            alpha = alpha_0
            iteration_no_increase = 0
        else:
            iteration_no_increase += 1
            alpha *= 0.95
        for x_i, y_i in in_random_order(data):
            gradient_i = gradient_f(x_i, y_i, w)
            w = w - (alpha *  gradient_i)
        epoch += 1
    return min_w

print 'w: {:.6f}'.format(SGD(target_f, gradient_f, x, y))


I am working on the file "1. Basic Concepts.ipynb".
The first task is:
The function we have evaluated, $f({\mathbf x}) = x_1^2+x_2^2+x_3^2$, is $3$ at $(1,1,1)$ and the gradient vector at this point is $(2,2,2)$.
Then, we can follow this steps to maximize (or minimize) the function:

Start from a random $\mathbf{x}$ vector.
Compute the gradient vector.
Walk a small step in the opposite direction of the gradient vector.

It is important to be aware that this gradient computation is very expensive: if $\mathbf{x}$ has dimension $n$, we have to evaluate $f$ at $2*n$ points.

How to use the gradient.
$f(x) = \sum_i x_i^2$, takes its mimimum value when all $x$ are 0.
Let's check it for $n=3$:
Can you write Python code for it?

def euc_dist(v1,v2):
    import numpy as np
    import math
    v = np.array(v1)-np.array(v2)
    return math.sqrt(sum(v_i ** 2 for v_i in v))


I am working on the file "1. Basic Concepts.ipynb".
The first task is:
The limit as $h$ approaches zero, if it exists, should represent the slope of the tangent line to $(x, f(x))$.
For values that are not zero it is only an approximation.
Can you write Python code for it?


I am working on the file "1. Basic Concepts.ipynb".
The first task is:
To fix this, we multiply the gradient by a step size. This step size (often called alpha) has to be chosen carefully, as a value too small will result in a long computation time, while a value too large will not give you the right result (by overshooting) or even fail to converge.
In this example, we'll set the step size to 0.01, which means we'll subtract $24×0.01$ from $15$, which is $14.76$.
This is now our new temporary local minimum: We continue this method until we either don't see a change after we subtracted the derivative step size, or until we've completed a pre-set number of iterations.
Can you write Python code for it?

old_min = 0
temp_min = 15
step_size = 0.01
precision = 0.0001

def f(x):
    return x**2 - 6*x + 5

def f_derivative(x):
    import math
    return 2*x -6

mins = []
cost = []

while abs(temp_min - old_min) > precision:
    old_min = temp_min
    gradient = f_derivative(old_min)
    move = gradient * step_size
    temp_min = old_min - move
    cost.append((3-temp_min)**2)
    mins.append(temp_min)

# rounding the result to 2 digits because of the step size
print "Local minimum occurs at {:3.6f}.".format(round(temp_min,2))


I am working on the file "1. Computer Vision and Deep Learning/L1 TensorFlow Lab/lab.ipynb".
The first task is:
<img src="image/learn_rate_tune.png" style="height: 60%;width: 60%">
Problem 3
Below are 3 parameter configurations for training the neural network. In each configuration, one of the parameters has multiple options. For each configuration, choose the option that gives the best acccuracy.
Parameter configurations:
Configuration 1
* Epochs: 1
* Batch Size:
  * 2000
  * 1000
  * 500
  * 300
  * 50
* Learning Rate: 0.01
Configuration 2
* Epochs: 1
* Batch Size: 100
* Learning Rate:
  * 0.8
  * 0.5
  * 0.1
  * 0.05
  * 0.01
Configuration 3
* Epochs:
  * 1
  * 2
  * 3
  * 4
  * 5
* Batch Size: 100
* Learning Rate: 0.2
The code will print out a Loss and Accuracy graph, so you can see how well the neural network performed.
If you're having trouble solving problem 3, you can view the solution here.
Can you write Python code for it?

# Find the best parameters for each configuration
# Configuration 1
# epochs = 1
# batch_size = 50
# learning_rate = 0.01

# Configuration 2
# epochs = 1
# batch_size = 100
# learning_rate = 0.1

# Configuration 3
epochs = 5
batch_size = 100
learning_rate = 0.2

### DON'T MODIFY ANYTHING BELOW ###
# Gradient Descent
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

# The accuracy measured against the validation set
validation_accuracy = 0.0

# Measurements use for graphing loss and accuracy
log_batch_step = 50
batches = []
loss_batch = []
train_acc_batch = []
valid_acc_batch = []

with tf.Session() as session:
    session.run(init)
    batch_count = int(math.ceil(len(train_features)/batch_size))

    for epoch_i in range(epochs):

        # Progress bar
        batches_pbar = tqdm(range(batch_count), desc='Epoch {:>2}/{}'.format(epoch_i+1, epochs), unit='batches')

        # The training cycle
        for batch_i in batches_pbar:
            # Get a batch of training features and labels
            batch_start = batch_i*batch_size
            batch_features = train_features[batch_start:batch_start + batch_size]
            batch_labels = train_labels[batch_start:batch_start + batch_size]

            # Run optimizer and get loss
            _, l = session.run(
                [optimizer, loss],
                feed_dict={features: batch_features, labels: batch_labels})

            # Log every 50 batches
            if not batch_i % log_batch_step:
                # Calculate Training and Validation accuracy
                training_accuracy = session.run(accuracy, feed_dict=train_feed_dict)
                validation_accuracy = session.run(accuracy, feed_dict=valid_feed_dict)

                # Log batches
                previous_batch = batches[-1] if batches else 0
                batches.append(log_batch_step + previous_batch)
                loss_batch.append(l)
                train_acc_batch.append(training_accuracy)
               ...


I am working on the file "1. Computer Vision and Deep Learning/L1 TensorFlow Lab/lab.ipynb".
The first task is:
Checkpoint
All your progress is now saved to the pickle file.  If you need to leave and comeback to this lab, you no longer have to start from the beginning.  Just run the code block below and it will load all the data and modules required to proceed.
Can you write Python code for it?

%matplotlib inline

# Load the modules
import pickle
import math

import numpy as np
import tensorflow as tf
from tqdm import tqdm
import matplotlib.pyplot as plt

# Reload the data
pickle_file = 'notMNIST.pickle'
with open(pickle_file, 'rb') as f:
  pickle_data = pickle.load(f)
  train_features = pickle_data['train_dataset']
  train_labels = pickle_data['train_labels']
  valid_features = pickle_data['valid_dataset']
  valid_labels = pickle_data['valid_labels']
  test_features = pickle_data['test_dataset']
  test_labels = pickle_data['test_labels']
  del pickle_data  # Free up memory

print('Data and modules loaded.')


I am working on the file "1. Computer Vision and Deep Learning/L1 TensorFlow Lab/lab.ipynb".
The first task is:
The notMNIST dataset is too large for many computers to handle.  It contains 500,000 images for just training.  You'll be using a subset of this data, 15,000 images for each label (A-J).
Can you write Python code for it?

def download(url, file):
    """
    Download file from <url>
    :param url: URL to file
    :param file: Local file path
    """
    if not os.path.isfile(file):
        print('Downloading ' + file + '...')
        urlretrieve(url, file)
        print('Download Finished')

# Download the training and test dataset.
download('https://s3.amazonaws.com/udacity-sdc/notMNIST_train.zip', 'notMNIST_train.zip')
download('https://s3.amazonaws.com/udacity-sdc/notMNIST_test.zip', 'notMNIST_test.zip')

# Make sure the files aren't corrupted
assert hashlib.md5(open('notMNIST_train.zip', 'rb').read()).hexdigest() == 'c8673b3f28f489e9cdf3a3d74e2ac8fa',\
        'notMNIST_train.zip file is corrupted.  Remove the file and try again.'
assert hashlib.md5(open('notMNIST_test.zip', 'rb').read()).hexdigest() == '5d3c7e653e63471c88df796156a9dfa9',\
        'notMNIST_test.zip file is corrupted.  Remove the file and try again.'

# Wait until you see that all files have been downloaded.
print('All files downloaded.')

def uncompress_features_labels(file):
    """
    Uncompress features and labels from a zip file
    :param file: The zip file to extract the data from
    """
    features = []
    labels = []

    with ZipFile(file) as zipf:
        # Progress Bar
        filenames_pbar = tqdm(zipf.namelist(), unit='files')

        # Get features and labels from all files
        for filename in filenames_pbar:
            # Check if the file is a directory
            if not filename.endswith('/'):
                with zipf.open(filename) as image_file:
                    image = Image.open(image_file)
                    image.load()
                    # Load image data as 1 dimensional array
                    # We're using float32 to save on memory space
                    feature = np.array(image, dtype=np.float32).flatten()

                # Get the the letter from the filename.  This is the letter of the image.
                label = os.path.split(filename)[1][0]

                features.append(feature)
        ...


I am working on the file "1. Computer Vision and Deep Learning/L2 LeNet Lab/LeNet-Lab.ipynb".
The first task is:
Evaluate the Model
Once you are completely satisfied with your model, evaluate the performance of the model on the test set.
Be sure to only do this once!
If you were to measure the performance of your trained model on the test set, then improve your model, and then measure the performance of your model on the test set again, that would invalidate your test results. You wouldn't get a true measure of how well your model would perform against real data.
You do not need to modify this section.
Can you write Python code for it?

with tf.Session() as sess:
    saver.restore(sess, tf.train.latest_checkpoint('.'))

    test_accuracy = evaluate(X_test, y_test)
    print("Test Accuracy = {:.3f}".format(test_accuracy))


I am working on the file "1. Computer Vision and Deep Learning/L2 LeNet Lab/LeNet-Lab.ipynb".
The first task is:
Preprocess Data
Shuffle the training data.
You do not need to modify this section.
Can you write Python code for it?

from sklearn.utils import shuffle

X_train, y_train = shuffle(X_train, y_train)


I am working on the file "1. Computer Vision and Deep Learning/L2 LeNet Lab/LeNet-Lab.ipynb".
The first task is:
TODO: Implement LeNet-5
Implement the LeNet-5 neural network architecture.
This is the only cell you need to edit.
Input
The LeNet architecture accepts a 32x32xC image as input, where C is the number of color channels. Since MNIST images are grayscale, C is 1 in this case.
Architecture
Layer 1: Convolutional. The output shape should be 28x28x6.
Activation. Your choice of activation function.
Pooling. The output shape should be 14x14x6.
Layer 2: Convolutional. The output shape should be 10x10x16.
Activation. Your choice of activation function.
Pooling. The output shape should be 5x5x16.
Flatten. Flatten the output shape of the final pooling layer such that it's 1D instead of 3D. The easiest way to do is by using tf.contrib.layers.flatten, which is already imported for you.
Layer 3: Fully Connected. This should have 120 outputs.
Activation. Your choice of activation function.
Layer 4: Fully Connected. This should have 84 outputs.
Activation. Your choice of activation function.
Layer 5: Fully Connected (Logits). This should have 10 outputs.
Output
Return the result of the 2nd fully connected layer.
Can you write Python code for it?

from tensorflow.contrib.layers import flatten

def LeNet(x):
    # Arguments used for tf.truncated_normal, randomly defines variables for the weights and biases for each layer
    mu = 0
    sigma = 0.1

    # Layer 1: Convolutional. Input = 32x32x1. Output = 28x28x6.
    layer_1_filter_shape = [5,5,1,6]
    layer_1_weights = tf.Variable(tf.truncated_normal(shape = layer_1_filter_shape, mean = mu, stddev = sigma))
    layer_1_bias = tf.Variable(tf.zeros(6))
    layer_1_strides = [1, 1, 1, 1]
    layer_1_padding = 'VALID'
    layer_1 = tf.nn.conv2d(x, layer_1_weights, layer_1_strides, layer_1_padding) + layer_1_bias

    # Activation.
    layer_1 = tf.nn.relu(layer_1)

    # Pooling. Input = 28x28x6. Output = 14x14x6.
    p1_filter_shape = [1, 2, 2, 1]
    p1_strides = [1, 2, 2, 1]
    p1_padding = 'VALID'
    layer_1 = tf.nn.max_pool(layer_1, p1_filter_shape, p1_strides, p1_padding)

    # Layer 2: Convolutional. Output = 10x10x16.
    layer_2_filter_shape = [5,5,6,16]
    layer_2_weights = tf.Variable(tf.truncated_normal(shape = layer_2_filter_shape , mean = mu, stddev = sigma))
    layer_2_bias = tf.Variable(tf.zeros(16))
    layer_2_strides = [1, 1, 1, 1]
    layer_2_padding = 'VALID'
    layer_2 = tf.nn.conv2d(layer_1, layer_2_weights, layer_2_strides, layer_2_padding) + layer_2_bias

    # Activation.
    layer_2 = tf.nn.relu(layer_2)

    # Pooling. Input = 10x10x16. Output = 5x5x16.
    p2_filter_shape = [1, 2, 2, 1]
    p2_strides = [1, 2, 2, 1]
    p2_padding = 'VALID'
    layer_2 = tf.nn.max_pool(layer_2, p2_filter_shape, p2_strides, p2_padding)

    # Flatten. Input = 5x5x16. Output = 400.
    layer_2 = flatten(layer_2)

    # Layer 3: Fully Connected. Input = 400. Output = 120.
    layer_3_filter_shape = [400,120]
    layer_3_weights = tf.Variable(tf.truncated_normal(shape = layer_3_filter_shape, mean = mu, stddev = sigma))
    layer_3_bias = tf.Variable(tf.zeros(120))
    layer_3 = tf.matmul(layer_2, layer_3_weights) + layer_3_bias

    # Activation.
   ...


I am working on the file "1. Computer Vision and Deep Learning/L2 LeNet Lab/LeNet-Lab.ipynb".
The first task is:
Train the Model
Run the training data through the training pipeline to train the model.
Before each epoch, shuffle the training set.
After each epoch, measure the loss and accuracy of the validation set.
Save the model after training.
You do not need to modify this section.
Can you write Python code for it?

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    num_examples = len(X_train)

    print("Training...")
    print()
    for i in range(EPOCHS):
        X_train, y_train = shuffle(X_train, y_train)
        for offset in range(0, num_examples, BATCH_SIZE):
            end = offset + BATCH_SIZE
            batch_x, batch_y = X_train[offset:end], y_train[offset:end]
            sess.run(training_operation, feed_dict={x: batch_x, y: batch_y})

        validation_accuracy = evaluate(X_validation, y_validation)
        print("EPOCH {} ...".format(i+1))
        print("Validation Accuracy = {:.3f}".format(validation_accuracy))
        print()

    saver.save(sess, './lenet')
    print("Model saved")


I am working on the file "1. Computer Vision and Deep Learning/L2 LeNet Lab/LeNet-Lab.ipynb".
The first task is:
Training Pipeline
Create a training pipeline that uses the model to classify MNIST data.
You do not need to modify this section.
Can you write Python code for it?

rate = 0.001

logits = LeNet(x)
cross_entropy = tf.nn.softmax_cross_entropy_with_logits(labels=one_hot_y, logits=logits)
loss_operation = tf.reduce_mean(cross_entropy)
optimizer = tf.train.AdamOptimizer(learning_rate = rate)
training_operation = optimizer.minimize(loss_operation)


I am working on the file "1. Computer Vision and Deep Learning/L2 LeNet Lab/LeNet-Lab.ipynb".
The first task is:
Visualize Data
View a sample from the dataset.
You do not need to modify this section.
Can you write Python code for it?

import random
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

index = random.randint(0, len(X_train))
image = X_train[index].squeeze()

plt.figure(figsize=(1,1))
plt.imshow(image, cmap="gray")
print(y_train[index])


I am working on the file "1. Computer Vision and Deep Learning/P1 Finding Lane Lines on the Road/P1.ipynb".
The first task is:
Global Variables
Can you write Python code for it?

# Global Variables
# Region-of-interest ofsets
top_x_offset = 0.45
top_y_offset = 0.60
bottom_x_offset = 0.07

# Stores the left and right lines from an image.
# Notice, need to clear before using it on new set of images (video).
right_lines = []
left_lines = []


I am working on the file "1. Computer Vision and Deep Learning/P1 Finding Lane Lines on the Road/P1.ipynb".
The first task is:
Helper Functions
Can you write Python code for it?

import math

# Global Variables
# Region-of-interest ofsets
top_x_offset = 0.45
top_y_offset = 0.60
bottom_x_offset = 0.07

# Stores the left and right lines from an image.
# Notice, need to clear before using it on new set of images (video).
right_lines = []
left_lines = []

def grayscale(img):
    """Applies the Grayscale transform
    This will return an image with only one color channel
    but NOTE: to see the returned image as grayscale
    (assuming your grayscaled image is called 'gray')
    you should call plt.imshow(gray, cmap='gray')"""
    return cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
    # Or use BGR2GRAY if you read an image with cv2.imread()
    # return cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

def canny(img, low_threshold, high_threshold):
    """Applies the Canny transform"""
    return cv2.Canny(img, low_threshold, high_threshold)

def gaussian_blur(img, kernel_size):
    """Applies a Gaussian Noise kernel"""
    return cv2.GaussianBlur(img, (kernel_size, kernel_size), 0)

def region_of_interest(img, vertices):
    """
    Applies an image mask.

    Only keeps the region of the image defined by the polygon
    formed from `vertices`. The rest of the image is set to black.
    """
    #defining a blank mask to start with
    mask = np.zeros_like(img)

    #defining a 3 channel or 1 channel color to fill the mask with depending on the input image
    if len(img.shape) > 2:
        channel_count = img.shape[2]  # i.e. 3 or 4 depending on your image
        ignore_mask_color = (255,) * channel_count
    else:
        ignore_mask_color = 255

    #filling pixels inside the polygon defined by "vertices" with the fill color
    cv2.fillPoly(mask, vertices, ignore_mask_color)

    #returning the image only where mask pixels are nonzero
    masked_image = cv2.bitwise_and(img, mask)
    return masked_image

def draw_lines(img, lines, color=[255, 0, 0], thickness=2):
    for line in lines:
        for x1,y1,x2,y2 in line:
            cv2.line(img, (x1, y1), (x2, y2),...


I am working on the file "1. Computer Vision and Deep Learning/P1 Finding Lane Lines on the Road/P1.ipynb".
The first task is:
Improve the draw_lines() function
At this point, if you were successful with making the pipeline and tuning parameters, you probably have the Hough line segments drawn onto the road, but what about identifying the full extent of the lane and marking it clearly as in the example video (P1_example.mp4)?  Think about defining a line to run the full length of the visible lane based on the line segments you identified with the Hough Transform. As mentioned previously, try to average and/or extrapolate the line segments you've detected to map out the full extent of the lane lines. You can see an example of the result you're going for in the video "P1_example.mp4".
Go back and modify your draw_lines function accordingly and try re-running your pipeline. The new output should draw a single, solid line over the left lane line and a single, solid line over the right lane line. The lines should start from the bottom of the image and extend out to the top of the region of interest.
Now for the one with the solid yellow lane on the left. This one's more tricky!
Can you write Python code for it?

right_lines.clear()
left_lines.clear()
yellow_output = 'test_videos_output/solidYellowLeft.mp4'
clip2 = VideoFileClip('test_videos/solidYellowLeft.mp4')
yellow_clip = clip2.fl_image(process_image)
%time yellow_clip.write_videofile(yellow_output, audio=False)

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(yellow_output))


I am working on the file "1. Computer Vision and Deep Learning/P1 Finding Lane Lines on the Road/P1.ipynb".
The first task is:
Test on Videos
We can test our solution on two provided videos:
solidWhiteRight.mp4
solidYellowLeft.mp4
Can you write Python code for it?

# Import everything needed to edit/save/watch video clips
from moviepy.editor import VideoFileClip
from IPython.display import HTML
import imageio
imageio.plugins.ffmpeg.download()

right_lines.clear()
left_lines.clear()
white_output = 'test_videos_output/solidWhiteRight.mp4'
clip1 = VideoFileClip("test_videos/solidWhiteRight.mp4")
white_clip = clip1.fl_image(process_image) #NOTE: this function expects color images!!
%time white_clip.write_videofile(white_output, audio=False)

HTML("""
<video width="960" height="540" controls>
  <source src="{0}">
</video>
""".format(white_output))


I am working on the file "1. Computer Vision and Deep Learning/P2 Traffic Sign Classifier/P2.ipynb".
The first task is:
Include an exploratory visualization of the dataset
Visualize the German Traffic Signs Dataset using the pickled file(s). This is open ended, suggestions include: plotting traffic sign images, plotting the count of each sign, etc.
The Matplotlib examples and gallery pages are a great resource for doing visualizations in Python.
NOTE: It's recommended you start with something simple first. If you wish to do more, come back to it after you've completed the rest of the sections. It can be interesting to look at the distribution of classes in the training, validation and test set. Is the distribution the same? Are there more examples of some classes than others?
Can you write Python code for it?

### Data exploration visualization code goes here.
### Feel free to use as many code cells as needed.
import matplotlib.pyplot as plt
# Visualizations will be shown in the notebook.
%matplotlib inline


I am working on the file "1. Computer Vision and Deep Learning/P2 Traffic Sign Classifier/P2.ipynb".
The first task is:
Model Architecture
Can you write Python code for it?

### Define your architecture here.
### Feel free to use as many code cells as needed.


I am working on the file "1. Computer Vision and Deep Learning/P2 Traffic Sign Classifier/P2.ipynb".
The first task is:
Output Top 5 Softmax Probabilities For Each Image Found on the Web
For each of the new images, print out the model's softmax probabilities to show the certainty of the model's predictions (limit the output to the top 5 probabilities for each image). tf.nn.top_k could prove helpful here.
The example below demonstrates how tf.nn.top_k can be used to find the top k predictions for each image.
tf.nn.top_k will return the values and indices (class ids) of the top k predictions. So if k=3, for each sign, it'll return the 3 largest probabilities (out of a possible 43) and the correspoding class ids.
Take this numpy array as an example. The values in the array represent predictions. The array contains softmax probabilities for five candidate images with six possible classes. tf.nn.top_k is used to choose the three classes with the highest probability:
```
(5, 6) array
a = np.array([[ 0.24879643,  0.07032244,  0.12641572,  0.34763842,  0.07893497,
         0.12789202],
       [ 0.28086119,  0.27569815,  0.08594638,  0.0178669 ,  0.18063401,
         0.15899337],
       [ 0.26076848,  0.23664738,  0.08020603,  0.07001922,  0.1134371 ,
         0.23892179],
       [ 0.11943333,  0.29198961,  0.02605103,  0.26234032,  0.1351348 ,
         0.16505091],
       [ 0.09561176,  0.34396535,  0.0643941 ,  0.16240774,  0.24206137,
         0.09155967]])
```
Running it through sess.run(tf.nn.top_k(tf.constant(a), k=3)) produces:
TopKV2(values=array([[ 0.34763842,  0.24879643,  0.12789202],
       [ 0.28086119,  0.27569815,  0.18063401],
       [ 0.26076848,  0.23892179,  0.23664738],
       [ 0.29198961,  0.26234032,  0.16505091],
       [ 0.34396535,  0.24206137,  0.16240774]]), indices=array([[3, 0, 5],
       [0, 1, 4],
       [0, 5, 1],
       [1, 3, 5],
       [1, 4, 3]], dtype=int32))
Looking just at the first row we get [ 0.34763842,  0.24879643,  0.12789202], you can confirm these are the 3 largest probabilities in a. You'll also notice [3, 0, 5] are the corresponding indices.
Can you write Python code for it?

### Print out the top five softmax probabilities for the predictions on the German traffic sign images found on the web.
### Feel free to use as many code cells as needed.


I am working on the file "1. Computer Vision and Deep Learning/P2 Traffic Sign Classifier/P2.ipynb".
The first task is:
Predict the Sign Type for Each Image
Can you write Python code for it?

### Run the predictions here and use the model to output the prediction for each image.
### Make sure to pre-process the images with the same pre-processing pipeline used earlier.
### Feel free to use as many code cells as needed.


I am working on the file "1. Computer Vision and Deep Learning/P2 Traffic Sign Classifier/P2.ipynb".
The first task is:
Project Writeup
Once you have completed the code implementation, document your results in a project writeup using this template as a guide. The writeup can be in a markdown or pdf file.

Note: Once you have completed all of the code implementations and successfully answered each question above, you may finalize your work by exporting the iPython Notebook as an HTML document. You can do this by using the menu above and navigating to  \n",
    "File -> Download as -> HTML (.html). Include the finished document along with this notebook as your submission.

Step 4 (Optional): Visualize the Neural Network's State with Test Images
This Section is not required to complete but acts as an additional excersise for understaning the output of a neural network's weights. While neural networks can be a great learning device they are often referred to as a black box. We can understand what the weights of a neural network look like better by plotting their feature maps. After successfully training your neural network you can see what it's feature maps look like by plotting the output of the network's weight layers in response to a test stimuli image. From these plotted feature maps, it's possible to see what characteristics of an image the network finds interesting. For a sign, maybe the inner network feature maps react with high activation to the sign's boundary outline or to the contrast in the sign's painted symbol.
Provided for you below is the function code that allows you to get the visualization output of any tensorflow weight layer you want. The inputs to the function should be a stimuli image, one used during training or a new one you provided, and then the tensorflow variable name that represents the layer's state during the training process, for instance if you wanted to see what the LeNet lab's feature maps looked like for it's second convolutional layer you could enter conv2 as the tf_activation variable.
For an example of what feature map outputs look like, check out NVIDIA's results in their paper End-to-End...
Can you write Python code for it?

### Visualize your network's feature maps here.
### Feel free to use as many code cells as needed.

# image_input: the test image being fed into the network to produce the feature maps
# tf_activation: should be a tf variable name used during your training procedure that represents the calculated state of a specific weight layer
# activation_min/max: can be used to view the activation contrast in more detail, by default matplot sets min and max to the actual min and max values of the output
# plt_num: used to plot out multiple different weight feature map sets on the same block, just extend the plt number for each new feature map entry

def outputFeatureMap(image_input, tf_activation, activation_min=-1, activation_max=-1 ,plt_num=1):
    # Here make sure to preprocess your image_input in a way your network expects
    # with size, normalization, ect if needed
    # image_input =
    # Note: x should be the same name as your network's tensorflow data placeholder variable
    # If you get an error tf_activation is not defined it may be having trouble accessing the variable from inside a function
    activation = tf_activation.eval(session=sess,feed_dict={x : image_input})
    featuremaps = activation.shape[3]
    plt.figure(plt_num, figsize=(15,15))
    for featuremap in range(featuremaps):
        plt.subplot(6,8, featuremap+1) # sets the number of feature maps to show on each row and column
        plt.title('FeatureMap ' + str(featuremap)) # displays the feature map number
        if activation_min != -1 & activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin =activation_min, vmax=activation_max, cmap="gray")
        elif activation_max != -1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmax=activation_max, cmap="gray")
        elif activation_min !=-1:
            plt.imshow(activation[0,:,:, featuremap], interpolation="nearest", vmin=activation_min, cmap="gray")
        else:
            plt.imshow(activation[0,:,:,...


I am working on the file "1. Computer Vision and Deep Learning/P2 Traffic Sign Classifier/P2.ipynb".
The first task is:
Step 1: Dataset Summary & Exploration
The pickled data is a dictionary with 4 key/value pairs:

'features' is a 4D array containing raw pixel data of the traffic sign images, (num examples, width, height, channels).
'labels' is a 1D array containing the label/class id of the traffic sign. The file signnames.csv contains id -> name mappings for each id.
'sizes' is a list containing tuples, (width, height) representing the original width and height the image.
'coords' is a list containing tuples, (x1, y1, x2, y2) representing coordinates of a bounding box around the sign in the image. THESE COORDINATES ASSUME THE ORIGINAL IMAGE. THE PICKLED DATA CONTAINS RESIZED VERSIONS (32 by 32) OF THESE IMAGES

Complete the basic data summary below. Use python, numpy and/or pandas methods to calculate the data summary rather than hard coding the results. For example, the pandas shape method might be useful for calculating some of the summary results.
Provide a Basic Summary of the Data Set Using Python, Numpy and/or Pandas
Can you write Python code for it?

### Replace each question mark with the appropriate value.
### Use python, pandas or numpy methods rather than hard coding the results

# TODO: Number of training examples
n_train = ?

# TODO: Number of validation examples
n_validation = ?

# TODO: Number of testing examples.
n_test = ?

# TODO: What's the shape of an traffic sign image?
image_shape = ?

# TODO: How many unique classes/labels there are in the dataset.
n_classes = ?

print("Number of training examples =", n_train)
print("Number of testing examples =", n_test)
print("Image data shape =", image_shape)
print("Number of classes =", n_classes)


I am working on the file "1. Computer Vision and Deep Learning/P2 Traffic Sign Classifier/P2.ipynb".
The first task is:
Step 2: Design and Test a Model Architecture
Design and implement a deep learning model that learns to recognize traffic signs. Train and test your model on the German Traffic Sign Dataset.
The LeNet-5 implementation shown in the classroom at the end of the CNN lesson is a solid starting point. You'll have to change the number of classes and possibly the preprocessing, but aside from that it's plug and play!
With the LeNet-5 solution from the lecture, you should expect a validation set accuracy of about 0.89. To meet specifications, the validation set accuracy will need to be at least 0.93. It is possible to get an even higher accuracy, but 0.93 is the minimum for a successful project submission.
There are various aspects to consider when thinking about this problem:

Neural network architecture (is the network over or underfitting?)
Play around preprocessing techniques (normalization, rgb to grayscale, etc)
Number of examples per label (some have more than others).
Generate fake data.

Here is an example of a published baseline model on this problem. It's not required to be familiar with the approach used in the paper but, it's good practice to try to read papers like these.
Pre-process the Data Set (normalization, grayscale, etc.)
Minimally, the image data should be normalized so that the data has mean zero and equal variance. For image data, (pixel - 128)/ 128 is a quick way to approximately normalize the data and can be used in this project.
Other pre-processing steps are optional. You can try different techniques to see if it improves performance.
Use the code cell (or multiple code cells, if necessary) to implement the first step of your project.
Can you write Python code for it?

### Preprocess the data here. It is required to normalize the data. Other preprocessing steps could include
### converting to grayscale, etc.
### Feel free to use as many code cells as needed.


I am working on the file "1. Computer Vision and Deep Learning/P2 Traffic Sign Classifier/P2.ipynb".
The first task is:
Step 3: Test a Model on New Images
To give yourself more insight into how your model is working, download at least five pictures of German traffic signs from the web and use your model to predict the traffic sign type.
You may find signnames.csv useful as it contains mappings from the class id (integer) to the actual sign name.
Load and Output the Images
Can you write Python code for it?

### Load the images and plot them here.
### Feel free to use as many code cells as needed.


I am working on the file "1. Computer Vision and Deep Learning/P2 Traffic Sign Classifier/P2.ipynb".
The first task is:
Train, Validate and Test the Model
A validation set can be used to assess how well the model is performing. A low accuracy on the training and validation
sets imply underfitting. A high accuracy on the training set but low accuracy on the validation set implies overfitting.
Can you write Python code for it?

### Train your model here.
### Calculate and report the accuracy on the training and validation set.
### Once a final model architecture is selected,
### the accuracy on the test set should be calculated and reported as well.
### Feel free to use as many code cells as needed.


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
4. Optimum models
A. Classification Report
Can you write Python code for it?

from sklearn.metrics import classification_report
from sklearn.metrics import auc

# optimum models
best_knn = KNeighborsClassifier(n_neighbors=5)
best_dt = DecisionTreeClassifier(max_depth= None, min_samples_leaf= 10, min_samples_split= 1, max_features= None)
best_rf = RandomForestClassifier(n_estimators=42, bootstrap=True, min_samples_leaf= 3, criterion ='gini')

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2)

best_knn.fit(x_train, y_train);
best_rf.fit(x_train, y_train);
best_dt.fit(x_train, y_train);

print('Success : True'),
print('Fail : False')
print('')
print("[KNN Classification_report]"),
print(classification_report(y_test, best_knn.predict(x_test))),
print("===================================================="),
print("[Random Forest Classification_report]"),
print(classification_report(y_test, best_rf.predict(x_test))),
print("===================================================="),
print("[Decision Tree Classification_report]"),
print(classification_report(y_test, best_dt.predict(x_test)))


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
5. Result
Can you write Python code for it?

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2)

# plot
plt.figure(figsize=(18,4))
plt.plot(y, 'bs-', markersize=10, alpha=0.6, label="Actual result")
plt.plot(best_knn.predict(x), 'ro-', markersize=10, alpha=0.6, label="Predicted result")
plt.legend()
plt.xlim(45, 145);
plt.ylim(-0.1, 1.1);
print('Acrual result vs predicted result')


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
A. Decision Tree

x_features = 'target_money', 'start_month', 'grammar_level', 'funding_duration'
Can you write Python code for it?

#Decision Tree parameters
param_grid = {"max_depth": [3, 5, None],
              "max_features": ['auto','sqrt', None],
              "min_samples_split": np.arange(1, 5),
              "min_samples_leaf": [1, 3, 5, 10]}

# run grid search
grid_search = GridSearchCV(dt, param_grid=param_grid)
grid_search.fit(x_4, y)

report(grid_search.grid_scores_)


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
A. Percentage distribution
Can you write Python code for it?

# percentage distribution
#log scailing
plt.figure(figsize=(8, 6))
sns.kdeplot(cf_df['percentage'].apply(lambda x: np.log(x)));
plt.legend(fontsize = 15);
plt.xticks(fontsize=15);
plt.xlabel('Log Percentage', fontsize=15);
plt.ylabel('Distribution', fontsize=15);


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
A. SVC
Can you write Python code for it?


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
As features reduce, score increases

B. GaussianNB
Can you write Python code for it?


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
B. Category
Can you write Python code for it?

check_number('category')

# percentage boxplot by categories
figure = plt.figure(figsize=(15,8))
sns.boxplot(x = cf_df['category'], y = cf_df['percentage']);
plt.xticks(rotation = 'vertical');
plt.ylim(-30, 500);
plt.xticks(fontsize=15);
plt.xlabel('category', fontsize=15);
plt.ylabel('percentage', fontsize=15);

# success rate by categories
print('Success rate'),
print('=================='),
for i in cf_df['category'].unique():
    success_percentage = (len(cf_df.loc[cf_df['category'] == i][cf_df['end_with_success'] == True]) / \
    len(cf_df.loc[cf_df['category'] == i]))*100
    print("{category} :".format(category = i), round(success_percentage, 2),'%')
print('=================='),
print('Dance has the highest rate')

# category percentage distribution
figure = plt.figure(figsize=(10,7))
for i in cf_df['category'].unique():
    sns.kdeplot(cf_df.loc[cf_df['category'] == '{i}'.format(i=i)]['percentage'], label = '{i}'.format(i=i))
    plt.xlim(-100, 400);
plt.xticks(fontsize=15);
plt.yticks(fontsize=15);
plt.xlabel('Percentage', fontsize=15)
plt.ylabel('Distribution', fontsize=15)
plt.legend(fontsize = 15);

# distribution test between whole distribution and each category distribution
# K-S : Kolmogorov Smirnov test
for i in cf_df['category'].unique()[:-1]:
    all_data = cf_df['funding_rate']
    category_data = cf_df.loc[cf_df['category'] == i]['funding_rate']
    # output values (p-value < 0.05)
    if round(sp.stats.ks_2samp(all_data, category_data)[1], 4) < 0.05:
        print('[all_sample vs {category_i}]'.format(category_i = i)),
        print('  K-S statistic :', round(sp.stats.ks_2samp(all_data, category_data)[0], 4))
        print('  p-value :', round(sp.stats.ks_2samp(all_data, category_data)[1], 4))


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
B. SelectKBest

Univariate feature selection
Univariate feature selection works by selecting the best features based on univariate statistical tests

Removes all but the K-highest scoring features
Scoring Function
f_classif : Compute the ANOVA F-value for the provided sample.
Can you write Python code for it?

x_kbest = SelectKBest(f_classif, k=4).fit_transform(x, y)
x_kbest


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
C. Decision Tree
Can you write Python code for it?


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
D. Pluctuation as a parameter change

Random Forest, KNN
KNN parameter = n_neighbors
Random Forest parameter = n_estimators
Can you write Python code for it?

# Random Forest, KNN
x_model = []
rf_model = []
knn_model = []
figure = plt.figure(figsize=(10,7))
for i in range(2, 100):
    knn_score = cross_val_score(KNeighborsClassifier(n_neighbors=i), x_4, y, cv=sfkfold).mean()
    rf_score = cross_val_score(RandomForestClassifier(n_estimators=i), x_4, y, cv=sfkfold).mean()
    x_model.append(i)
    rf_model.append(rf_score)
    knn_model.append(knn_score)

plt.plot(x_model, rf_model, 'o--', c='r', label = 'Random Forest')
plt.plot(x_model, knn_model, 'o--', c='g', label = 'KNN')
plt.xlabel('Parameter', fontsize=15)
plt.axhline(y = success_percentage, ls = '--', c='b', label = 'Base success rate')
plt.ylabel('Accuracy', fontsize=15)
plt.legend()
print('Random Forest max_accuracy :', round(max(rf_model)*100, 2), '%'),
print('KNN max_accuracy :', round(max(knn_model)*100, 2), '%')


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
D. Radom Forest
Can you write Python code for it?


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
E. KNN
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

knn = KNeighborsClassifier()

print('7 features :', cross_val_score(knn, x, y).mean()),
print('=======================================================')
print('5 features : ', cross_val_score(knn, x_5, y).mean()),
print('======================================================='),
print('4 features : ', cross_val_score(knn, x_4, y).mean())


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
F. Model comparison
Can you write Python code for it?

#StratifiedKFold : proper method for binary classes
sfkfold = StratifiedKFold(y, n_folds = 10)

"""3 scores """
def scores(model, x_1, x_2, x_3, y, cv):
    scores = [cross_val_score(model, x_1, y, cv=cv).mean(),
              cross_val_score(model, x_2, y, cv=cv).mean(),
              cross_val_score(model, x_3, y, cv=cv).mean()]
    return scores

svc_score = scores(svc_rbf, x_4, x_5, x, y, sfkfold)
dt_score = scores(dt,x_4, x_5, x, y, sfkfold)
rf_score = scores(rf, x_4, x_5, x, y, sfkfold)
knn_score = scores(knn, x_4, x_5, x, y, sfkfold)
gnb_score = scores(gnb, x_4, x_5, x, y, sfkfold)

figure = plt.figure(figsize=(10,7));
x_ticks = [len(x_4.columns), len(x_5.columns), len(x.columns)]
plt.plot(x_ticks, svc_score, 'o--', c='r', label = 'SVC');
plt.plot(x_ticks, dt_score, 'o--', c='y', label = 'Decision Tree');
plt.plot(x_ticks, rf_score, 'o--', c='g', label = 'Random Forest');
plt.plot(x_ticks, gnb_score, 'o--', c='c', label = 'GaussianNB');
plt.plot(x_ticks, knn_score, 'o--', c='m', label = 'KNN');
plt.legend(loc = 'lower right');
plt.xlabel('Parameter', fontsize=15);
plt.axhline(y = success_percentage, ls = '--', c='b');
plt.ylabel('Accuracy', fontsize=15);
plt.ylim(0.55, 0.83);
plt.xlim(3.5, 7.5);


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
G. # Result

Kbest = target_money, funding_duration, start_month, has_video
Correlation = target_money, grammar_level, start_month, has_video, category_label
Feature Importance =  target_money, grammar_level, funding_duration, start_month
Mean decrese accuracy = target_money
Reculsive Feature Estimation =  target_money, grammar_level, funding_duration, start_month, category_label

Rank : target_money > start_month > grammar_level > funding_duration > category_label = has_video

'num_funding_type' may be unimportant

Final Selection
4 features : target_money, start_month, grammar_level, funding_duration
5 features : target_money, start_month, grammar_level, funding_duration, category_label(or has_video)

3. Model Selection (Classification)

SVC
Gaussian Naive Bayes
Decision Tree
Random Forest
Select models show better performance than base success rate (62.34%)
Can you write Python code for it?

from sklearn.svm import SVC
from sklearn.naive_bayes import GaussianNB
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.cross_validation import train_test_split
from sklearn.cross_validation import cross_val_score
from sklearn.cross_validation import StratifiedKFold

# using features
x_4 = pd.DataFrame([cf_df['target_money'], cf_df['start_month'],
                           cf_df['grammar_level'], cf_df['funding_duration']]).T
x_5 = pd.DataFrame([cf_df['target_money'], cf_df['start_month'],
                           cf_df['grammar_level'], cf_df['funding_duration'],
                           cf_df['category_label']]).T

success_percentage = y.value_counts()[1] / len(y)
print('Base success rate:' ,round(success_percentage * 100, 2),'%')


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
It has non normality distribution.
Different Distribution, Same mean.

D. Target money
Can you write Python code for it?

# target_money, current_money
figure = plt.figure(figsize=(10,8))
sns.regplot(x ='target_money', y ='current_money', data = cf_df, fit_reg=False);
plt.xlim(-1000, 10000000);
plt.ylim(-2000, 10000000);
plt.xticks(fontsize=15);
plt.yticks(fontsize=15);
plt.xlabel('target_money', fontsize = 15);
plt.ylabel('current_money', fontsize = 15);

#Pearson Correlation
target_current_corr = sp.stats.pearsonr(cf_df['target_money'].tolist(), cf_df['current_money'].tolist())
print('Pearson correlation :', round(target_current_corr[0], 4))


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
Overall, performance of KNN is the best (Accuracy : 0.78, Recall : 0.79)
We have to intepret the result carefully. Because this report is conducted with few features(4) and samples(2000+), and all samples are gathered for 5 years(2011~). It may have overfitting and time parameters' effect.
But still, KNN performs amazingly, even if KNN is ease of use and optimal value(k) is highly data-dependent

Insight

Grammar level is estimated with project description (In previouse project, it was comments)
Grmmar level, date duration effects in this case
I expect grammar level represens project creator's ability or user's annoyance index
Number of funding type (range of choice) doesn't really matter (some samples only have 2 type, but others have over 5types)
Key figure of success is still quality of a project
To predict perfectly, crowd funding company must make measurable index of quality

Appendix : with grammar vs with no grammar
Can you write Python code for it?

x_no_grammars = []
y_no_grammars = []
x_grammars = []
y_grammars = []

x_no_grammar = pd.DataFrame([cf_df['category_label'], cf_df['target_money'],
                  cf_df['has_video']]).T
x_grammar = pd.DataFrame([cf_df['category_label'], cf_df['target_money'],
                  cf_df['has_video'], cf_df['grammar_level']]).T

figure = plt.figure(figsize=(10, 7))
legend =['with_grammar', 'without_grammar']
for i in range(1, 100):
    rf_grammar = RandomForestClassifier(n_estimators=i)
    grammar_score = cross_val_score(rf_grammar, x_grammar, y, cv=10).mean()
    x_grammars.append(i)
    y_grammars.append(grammar_score)

for i in range(1, 100):
    rf_no_grammar = RandomForestClassifier(n_estimators=i)
    no_grammar_score = cross_val_score(rf_no_grammar, x_no_grammar, y, cv=10).mean()
    x_no_grammars.append(i)
    y_no_grammars.append(no_grammar_score)

plt.plot(x_grammars, y_grammars, 'o--', c='r', label = 'with grammar')
plt.plot(x_no_grammars, y_no_grammars, 'o--', c = 'y', label = 'with no grammar')
plt.legend(legend, loc=5, fontsize=15)
plt.xlabel('n_estimator', fontsize=15)
plt.ylabel('accuracy', fontsize=15)
print('max_accuracy(with_grammar_level) :', round(max(y_grammars)*100, 2), '%'),
print('max_accuracy(no_grammar_level) :', round(max(y_no_grammars)*100, 2), '%')


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
Success Prediction

Predict success or fail with information given before project begins
Features : target_money, grammar_level, has_video, funding_type_count, funding_type_1, funding_type_2, funding_type_3, funding_duration

※ Estimation of grammar_level

number of errors / number of tokens
tested by naver grammar tester (https://github.com/ssut/py-hanspell)
suppposed grammar level represents ability of creators

Outline

Check the insight from previous project (https://github.com/surprisoh/crowdfunding_prediction)
Distribution test
Feature selection
Model Selection (classification)
Grid Search & Scores
Optimum model selection
Result & Insight

1. Distribution Test
Can you write Python code for it?

#load_data
cf_df = pd.read_excel('cf_df.xlsx')

#check feature number
def check_number(feature):
    """"feature : 'str'"""
    count = cf_df[feature].value_counts()
    return print(count)

# success rate
print('overall success'),
print('=================='),
success_percentage = cf_df['end_with_success'].value_counts()[1] / cf_df['end_with_success'].count()
print(round(success_percentage*100, 2),'%' )


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
Target_money and result are correlated

E. Grammar level
Can you write Python code for it?

figure = plt.figure(figsize=(10,8));
sns.regplot(cf_df['grammar_level'], cf_df['current_money'], color='g');
plt.ylim(-1000, 10000000);
plt.xticks(fontsize=15);
plt.yticks(fontsize=15);
plt.xlabel('grammar_level', fontsize = 15);
plt.ylabel('current_money', fontsize = 15);

#Pearson Correlation
grammar_current_corr = sp.stats.pearsonr(cf_df['grammar_level'].tolist(), cf_df['current_money'].tolist())
print('Pearson correlation :', round(grammar_current_corr[0], 4))


I am working on the file "1. Crowd Funding Prediction.ipynb".
The first task is:
no normality distribution
same distribution, same mean (between success sample and fail sample)

F. Num_funding_type
Can you write Python code for it?

figure = plt.figure(figsize=(8,6))
sns.kdeplot(cf_df[cf_df['end_with_success'] == True]['num_funding_type'], label= 'success');
sns.kdeplot(cf_df[cf_df['end_with_success'] == False]['num_funding_type'], label= 'fail',
            c='r', linestyle='--');
plt.xticks(fontsize=10);
plt.yticks(fontsize=10);
plt.xlabel('Num_funding_type');
plt.ylabel('Distribution');
#plt.xlim(-0.1, 0.3);
plt.legend(fontsize = 15);

success_count = cf_df[cf_df['end_with_success'] == True]['num_funding_type']
fail_count = cf_df[cf_df['end_with_success'] == False]['num_funding_type']
print('Shapiro test statistics(success_count) :', round(sp.stats.shapiro(success_count)[0], 4),
'  ','Shapiro test p-value(success_count) :', round(sp.stats.shapiro(success_count)[1], 4)),
print('Shapiro test statistics(fail_count) :', round(sp.stats.shapiro(fail_count)[0], 4),
'  ','Shapiro test p-value(fail_count) :', round(sp.stats.shapiro(fail_count)[1], 4))

print(sp.stats.ks_2samp(success_count, fail_count))
print(sp.stats.mannwhitneyu(success_count, fail_count))