I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Regressão Logística com Regularização
Nesta parte do trabalho, será implementada a Regressão Logística Regularizada
para prever se os microchips de uma usina de fabricação passam na garantia
de qualidade (QA). Durante a QA, cada microchip passa por vários testes para
garantir se está funcionando corretamente. Dessa forma, a Gestão de Produto da
fábrica terá o resultados de teste para alguns microchips em dois testes diferentes.
A partir desses dois testes, será determinado se os microchips deveriam ser
aceitos ou rejeitados. Para auxiliar a tomar a decisão, há um conjunto de dados
com resultados de testes anteriores sobre microchips, a partir do qual é possível construir
um modelo de Regressão Logística.
O arquivo $ex2data2.txt$ contém os dados a serem usados nessa parte do trabalho. A primeira
coluna corresponde aos resultados do primeiro teste, enquanto que a segunda coluna corresponde
aos resultados do segundo teste. A terceira coluna contém os valores da classe (y = 0 significa
rejeitado no teste, e y = 1 significa aceito no teste).
Visualização dos Dados
Para a maioria dos conjuntos de dados do mundo real, não é possível criar um gráfico para
visualizar seus pontos. Mas, para o conjunto de dados fornecido, isso é possível. Implemente
um script em Python que produza um gráfico de dispersão $scatter plot$ dos dados fornecidos.
Can you write Python code for it?

data = np.loadtxt('am-T2-dados/ex2data2.txt', delimiter=',')
pd.DataFrame(data).head()

# Visualização da Distribuição dos Dados conforme Histograma
pd.DataFrame(data).hist()
plt.show()

plt.figure(figsize=(10,5))
pd.DataFrame(data)[0].hist(), pd.DataFrame(data)[1].hist(), pd.DataFrame(data)[2].hist()
plt.show();

pd.DataFrame(data).describe().round(2)


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Regressão Polinomial
O problema com nosso modelo linear é que ele é muito simples para os dados e resultou em subajuste (viés alto). Nesta parte, você irá resolver esse problema adicionando mais características. Para usar a regressão polinomial, defina uma hipótese da seguinte forma:
$$H_{\theta}(x) = \theta_{0} + \theta_{1} * waterLevel + \theta_{2} * waterLevel^{2} + ... + \theta_{p} * waterLevel^{p}$$.
Observe que, ao definir $x_{1} = (WaterLevel)$, $x_{2} = (WaterLevel)^{2}$, ... , $x_{p} = (WaterLevel)^{p}$, obtemos um modelo de regressão linear onde os características são as diferentes pontências do valor original $(WaterLevel)$. Agora, você irá adiciona mais características usando as pontências da característica existente x no conjunto de dados. Sua tarefa nesta parte é implementar código em um arquivo de nome polyFeatures.py. Nesse arquivo, crie um função de mesmo nome que mapeie o conjunto de treinamento original $X$ de tamanho $m$ $x$ $1$ em suas potências mais altas. Especificamente, quando um conjunto de treinamento $X$ de tamanho $m$ $x$ $1$ for passado para essa função, ela deve retornar uma matriz $m$ $x$ $p$ de nome $X_poli$, onde a coluna 1 contém os valores originais de $X$, a coluna $2$ contém os valores de $x^{2}$, a coluna 3 contém os valores de $x^{3}$, e assim por diante. Note que você não tem que considerar a potência de expoentezero nessa função. Após implementar o descrito acima, você terá uma função que mapeia características para uma maior dimensão.
Can you write Python code for it?

def polyFeatures(X, p):
    '''
    Função que mapeie o conjunto de treinamento X de tamanho M x 1 em n-ésima potência.
    '''

    X_poly = X
    if p >= 2:
        for k in range(1,p):
            X_poly = np.column_stack((X_poly, np.power(X,k+1)))

    return X_poly

def normalizaFeatures(X):
    '''
    Função que calcula a normalizada para cada elemento, dado um vetor.
    '''

    mu = np.mean(X, axis=0)
    X_norm = X - mu

    sigma = np.std(X_norm, axis=0)
    X_norm = X_norm/sigma

    return X_norm, mu, sigma


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Regressão Polinomial - aprendizado
Depois de ter completado polyFeatures.py, você deve treinar um modelo de regressão polinomial usando sua função de custo da regressão linear. Tenha em mente que, apesar de termos termos polinomiais no vector de características, ainda estamos resolvendo um problema de otimização de regressão linear. Os termos polinomiais simplesmente se transformaram em características que podemos usar para aplicar regressão linear. Estamos usando a mesma função de custo e gradiente que você implementou para a parte anterior deste trabalho.
Para esta parte do trabalho, você usará um polinômio de grau 8. Se executarmos o treinamento diretamente sobre os dados projetados, não iremos obter um bom resultado, porque as características não irão estar na mesma escala (por exemplo, um exemplo com x = 40 agora terá uma característica $x^{8} = 40^{8} = 6.5 \times 10^{1} 2 $).
Portanto, você vai precisar aplicar a normalização de características. Portanto, antes de aprender os parâmetros $\theta$ para a regressão polinomial, você deve normalizar as características do conjunto de treinamento, e armazenar os parâmetros $\mu$ e $\sigma$. Depois de aprender os parâmetros $\theta$, você deve ver gerar dois gráficos (que devem ser similares aos das $Figuras$ abaixo) gerados com a regressão polinomial com $\lambda$ = 0.

Tarefa de Aprendizado com as Funções $polyFeatures$ e $normalizaFeatures$
Can you write Python code for it?

p = 8
m = X.shape[0]

X_poly = polyFeatures(X, p)
X_poly, mu, sigma = normalizaFeatures(X_poly)
X_poly = np.column_stack((np.ones((m,1)), X_poly))

X_poly_test = polyFeatures(Xtest, p)
X_poly_test = (X_poly_test - mu) / sigma
X_poly_test = np.column_stack((np.ones((X_poly_test.shape[0],1)), X_poly_test)) # Add Ones

X_poly_val = polyFeatures(Xval, p)
X_poly_val = (X_poly_val - mu) / sigma
X_poly_val = np.column_stack((np.ones((X_poly_val.shape[0],1)), X_poly_val)) # Add Ones

print('Normalized Training Example 1 ')
print(X_poly[0, :])


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Testando a Função de Curvas de Aprendizado
Can you write Python code for it?

Erro_treino, Erro_valida = learningCurve(X_treino, y_treino, X_valida, y_valida, 0)
print(Erro_treino)
print(Erro_valida)


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Testando a Função de Mapeamento de Features
Can you write Python code for it?

A = np.array( pd.DataFrame(data)[0] )
B = np.array( pd.DataFrame(data)[1] )

X_feature = np.c_[mapFeature(A, B)]
X_feature.shape


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Visualização dos Dados
Você deve começar por produzir uma visualização do conjunto de dados de treinamento. O gráfico que você deve produzir deve ser similar ao apresentado na Figura abaixo:
Can you write Python code for it?

# Obtendo o conjunto de treinamento para a Visualização Gráfica
y_treino = dataMatlab['y']
X_treino = np.c_[np.ones_like(dataMatlab['X']), dataMatlab['X']]

y_teste = dataMatlab['ytest']
X_teste = np.c_[np.ones_like(dataMatlab['Xtest']), dataMatlab['Xtest']]

y_valida = dataMatlab['yval']
X_valida = np.c_[np.ones_like(dataMatlab['Xval']), dataMatlab['Xval']]

print('##############################################################################')
print('Os Conjuntos de Treinamento e Validação: ')
print('\tX_train:', X_treino.shape)
print('\ty_train:', y_treino.shape)
print('\tX_test:', X_teste.shape)
print('\ty_test:', y_teste.shape)
print('\tXval:', X_valida.shape)
print('\tyval:', y_valida.shape)
print('##############################################################################')

plt.figure(figsize=(10,5))
plt.scatter(X_treino[:, 1], y_treino, c='xkcd:red', marker='x', label='Water flow')
plt.xlabel('Change in water level (x)')
plt.ylabel('Water following out of the dam (y)')
plt.ylim(ymin=-5)
plt.xlim(xmin=-50)
plt.legend(loc=4);


I am working on the file "Relatorio-de-Aprendizado-de-Maquina-Trabalho2A.ipynb".
The first task is:
Viés-Variância
Um conceito importante no Aprendizado de Máquina é o relacionamento entre o viés (bias) e a variância (variance) de um modelo de aprendizado. Modelos com viés elevado não são suficientemente complexos para os dados e tendem a sofrer de subajuste (underfitting), enquanto que modelos com alta variância tendem a sofrer de sobreajuste.
Nesta parte do trabalho, você irá produzir gráficos dos erros de treinamento e teste na forma de curvas de aprendizado para diagnosticar problemas de viés-variância.
Curvas de Aprendizado
Agora você implementará código para gerar as curvas de aprendizado que serão úteis na depuração de algoritmos de aprendizagem. Lembre-se de que uma curva de aprendizagem traça erros de treinamento e de validação cruzada como funções do tamanho do conjunto de treinamento. Crie um arquivo de nome learningCurve.py que deve conter uma função (também chamada $learningCurve$) que retorna um vetor de erros para o conjunto de treinamento e conjunto de validação cruzada.
Para traçar a curva de aprendizado, precisamos de um conjunto de treinamento e validação cruzada erro para diferentes tamanhos de conjuntos de treinamento. Para obter diferentes tamanhos de conjuntos de treinamento, você deve usar subconjuntos diferentes do conjunto de treinamento original X. Especificamente, para um tamanho de conjunto de treinamento de $i$, você deve usar os primeiros exemplos de i (ou seja, X(1 : i; :) e y(1 : i)).
Para cada tamanho de conjunto de treinamento, você encontrar os parâmetros $\theta$. Note que o $lambda$ deve ser passado como um parâmetro para a função $learningCurve$. Depois de aprender os parâmetros $\theta$, você deve calcular o erro nos conjuntos de treinamento e de validação. Lembre-se de que o erro de treinamento para um conjunto de dados é definido como:
Em particular, note que o erro de treinamento não inclui o termo de regularização. Uma maneira de calcular o erro de treinamento é usar a sua função de custo já implementada e definir $\lambda$ = 0 apenas para calcular o erro de...
Can you write Python code for it?

def learningCurve(X_treino, y_treino, X_valida, y_valida, Lambda):
    '''
    Função que retorna um vetor de erros para o conjunto de treinamento e conjunto de validação cruzada.
        Conjunto de treinamento: X_treino, y_treino
        Conjunto de validação: X_valida, y_valida
        Lambda = 0
    '''

    theta0 = np.array([[15],[15]])
    m = y_treino.size
    Erro_treino = np.zeros((m, 1))
    Erro_valida = np.zeros((m, 1))

    for i in np.arange(m):
        TR = trainLinearReg(theta0, X_treino[:i+1], y_treino[:i+1], Lambda)
        Erro_treino[i] = linearRegCostFunction(TR.x, X_treino[:i+1], y_treino[:i+1], Lambda)
        Erro_valida[i] = linearRegCostFunction(TR.x, X_valida, y_valida, Lambda)

    return(Erro_treino, Erro_valida)


I am working on the file "Relatório de Aprendizado de Máquina - Trabalho 02 (LoadTXT).ipynb".
The first task is:
1.2 Mapeamento de características (feature mapping)
Uma maneira de tornar os dados mais apropriados para a classificação é criar
mais características a partir das já existentes. Para isso, você deve criar uma
função mapFeature. Essa função deve ser implementada em um arquivo de
nome mapFeature.py, que irá mapear as características para todos os termos
polinomiais de x1 e x2, até a sexta potência. Como resultado desse mapeamento, nosso
vetor de duas características (os escores nos dois testes de QA) será transformado
em um vetor de 28 dimensões.
Um classificador que usa regressão logística treinado nesse vetor de características
de maior dimensão terá uma fronteira de decisão mais complexa e parecerá não-linear
quando desenhado em um gráfico bidimensional.
Embora o mapeamento de características nos permita construir um classificador mais expressivo,
também é mais suscetível a sobreajuste (overfitting). Desse modo, será implementada a
Regressão Logística Regularizada sobre os dados fornecidos e também verá como a regularização pode
ajudar a combater o problema do sobreajuste.
Can you write Python code for it?

def map_feature(X1, X2):
    '''
    Função que mapeia características p/ os termos polinomiais X1 e X2 até a 6ª potência.
    Retorna um novo conjunto com mais características, através do algoritmo de mapping
    X1, X2, X1 ** 2, X2 ** 2, X1*X2, X1*X2 ** 2, etc...
    Os parâmetros X1, X2 devem ser do mesmo tamanho
    '''
    # Potência padrão para o mapeamento
    potencia = 6

    X1.shape = (X1.size, 1)
    X2.shape = (X2.size, 1)

    features = np.ones(shape=(X1.size, 1))

    for i in range(1, potencia + 1):
        for j in range(i + 1):
            r = (X1 ** (i - j)) * (X2 ** j)
            features = append(features, r, axis=1)

    return features

X = data[:, 0:2]
y = data[:, 2]

pos = where(y == 1)
neg = where(y == 0)
scatter(X[pos, 0], X[pos, 1], marker='o', c='b')
scatter(X[neg, 0], X[neg, 1], marker='x', c='r')
xlabel('Microchip Test 1')
ylabel('Microchip Test 2')
legend(['y = 1', 'y = 0'])

m, n = X.shape
y.shape = (m, 1)

it = map_feature(X[:, 0], X[:, 1])

it.shape

print(pd.DataFrame(it))


I am working on the file "Relatório de Aprendizado de Máquina - Trabalho 02 (LoadTXT).ipynb".
The first task is:
1.3 Função de custo e gradiente
Agora, você deverá implementar o código para calcular a função de custo e
o gradiente para a regressão logística regularizada. Crie um arquivo de nome
costFunctionReg.py que contém uma função de nome costFunctionReg.py
e que computa o custo e o gradiente. Lembre-se de que a função de custo
regularizada na regressão logística é dada por:
$$J_{regularizado} = \small \underbrace{-\frac{1}{m} \sum\limits_{i = 1}^{m} \large{(}\small y^{(i)}\log\left(x^{(i)}\right) + (1-y^{(i)})\log\left(1- x^{(i)}\right) \large{)} }\text{Função de Custo} + \underbrace{\frac{\lambda}{2m} \sum\limits{j = 1}^{n}\ {\theta}{j}^{2} }\text{Fator Regularização} $$
Depois de concluir a implementação da função costFunctionReg, você deve
testar a corretude dela usando o valor inicial de ${\theta}$ (inicializado todo com zeros).
Você deve ver que o custo é de cerca de 0.693.
Porém, usando a função costFunctionReg, você agora deve computar os valores ótimos para ${\theta}$.
Can you write Python code for it?

from numpy import loadtxt, where, zeros, e, array, log, ones, append, linspace
from pylab import scatter, show, legend, xlabel, ylabel, contour, title
from scipy.optimize import fmin_bfgs

def sigmoid(X):
    '''Compute the sigmoid function '''
    den = 1.0 + e ** (-1.0 * X)
    d = 1.0 / den
    return d

def cost_function_reg(theta, X, y, l):
    '''Compute the cost and partial derivatives as grads
    '''

    h = sigmoid(X.dot(theta))

    thetaR = theta[1:, 0]

    J = (1.0 / m) * ((-y.T.dot(log(h))) - ((1 - y.T).dot(log(1.0 - h)))) + (l / (2.0 * m)) * (thetaR.T.dot(thetaR))

    delta = h - y
    sumdelta = delta.T.dot(X[:, 1])
    grad1 = (1.0 / m) * sumdelta

    XR = X[:, 1:X.shape[1]]
    sumdelta = delta.T.dot(XR)

    grad = (1.0 / m) * (sumdelta + l * thetaR)

    out = zeros(shape=(grad.shape[0], grad.shape[1] + 1))

    out[:, 0] = grad1
    out[:, 1:] = grad

    return J.flatten(), out.T.flatten()


I am working on the file "Relatório de Aprendizado de Máquina - Trabalho 02 (LoadTXT).ipynb".
The first task is:
1.3 Visualização de J(ø)
Para melhor entender a função de custo, você irá nessa parte do trabalho plotar o custo sobre
uma grade bidimensional de valores de 0 e de 1. Para isso, você deve usar sua implementação da função computarCusto.
O código que você deve implementar deve gerar um array bidimensional de valores de J(ø). Os valores gerados pelo seu código devem estar na faixa a seguir: -10;0;+10 e -1;+4. Utilize incremento de 0.01 para gerar os valores de 0 e de 1.
A seguir, usando a função matplotlib.pyplot.contour da biblioteca matplotlib, produza um gráfico de curvas de contorno (contour plot). Também
utilizando a biblioteca matplotlib, crie um gráfico da superfície correspondente a J(ø).
Can you write Python code for it?

import pandas as pd
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt

from scipy.optimize import minimize

from sklearn.preprocessing import PolynomialFeatures

pd.set_option('display.notebook_repr_html', False)
pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', 150)
pd.set_option('display.max_seq_items', None)

#%config InlineBackend.figure_formats = {'pdf',}
%matplotlib inline

import seaborn as sns
sns.set_context('notebook')
sns.set_style('white')

def predict(theta, X, threshold=0.5):
    p = sigmoid(X.dot(theta.T)) >= threshold
    return(p.astype('int'))

def loaddata(file, delimeter):
    data = np.loadtxt(file, delimiter=delimeter)
    print('Dimensions: ',data.shape)
    print(data[1:6,:])
    return(data)

def plotData(data, label_x, label_y, label_pos, label_neg, axes=None):
    # Get indexes for class 0 and class 1
    neg = data[:,2] == 0
    pos = data[:,2] == 1

    # If no specific axes object has been passed, get the current axes.
    if axes == None:
        axes = plt.gca()
    axes.scatter(data[pos][:,0], data[pos][:,1], marker='+', c='k', s=60, linewidth=2, label=label_pos)
    axes.scatter(data[neg][:,0], data[neg][:,1], c='y', s=60, label=label_neg)
    axes.set_xlabel(label_x)
    axes.set_ylabel(label_y)
    axes.legend(frameon= True, fancybox = True);

fig, axes = plt.subplots(1,3, sharey = True, figsize=(17,5))

# Decision boundaries
# Lambda = 0 : No regularization --> too flexible, overfitting the training data
# Lambda = 1 : Looks about right
# Lambda = 100 : Too much regularization --> high bias

for i, C in enumerate([0, 1, 100]):
    # Optimize costFunctionReg
    res2 = minimize(costFunctionReg, initial_theta, args=(C, X_feature, y), method=None, jac=gradientReg, options={'maxiter':3000})

    # Accuracy
    accuracy = 100*sum(predict(res2.x, X_feature) == y.ravel())/y.size

    # Scatter plot of X,y
    plotData(data2, 'Microchip Test 1', 'Microchip Test 2', 'y = 1', 'y = 0', axes.flatten()[i])

    #...


I am working on the file "Relatório de Aprendizado de Máquina - Trabalho 02 (LoadTXT).ipynb".
The first task is:
1.3.1 Testando a Função de Custo e o Gradiente
Can you write Python code for it?

from scipy.optimize import fmin_bfgs

m, n = X.shape
y.shape = (m, 1)
it = map_feature(X[:, 0], X[:, 1])

#Initialize theta parameters
initial_theta = zeros(shape=(it.shape[1], 1))

#Set regularization parameter lambda to 1
regularizacao = 1

# Compute and display initial cost and gradient for regularized logistic regression
cost, grad = cost_function_reg(initial_theta, it, y, regularizacao)

#def decorated_cost(theta):
#    return cost_function_reg(theta, it, y, l)

#cost_function_reg(theta, it, y, l)

#print fmin_bfgs(decorated_cost, initial_theta, args=(it, y, l), maxfun=400)

cost, grad


I am working on the file "Relatório de Aprendizado de Máquina - Trabalho 02 (LoadTXT).ipynb".
The first task is:
1.4 Esboço da fronteira de decisão
Nessa parte, você deve esboçar (plotar) a fronteira de decisão que foi aprendida
para separar os exemplos positivos dos negativos. Crie uma arquivo de nome
plotDecisionBoundary.py, para criar esse gráfico que traça o limite da decisão
não-linear. Seu gráfico deve ser semelhante ao apresentado na Figura abaixo.
Can you write Python code for it?

#Plot Boundary
l = 1
m, n = X.shape
y.shape = (m, 1)
it = map_feature(X[:, 0], X[:, 1])
theta = ones(shape=(it.shape[1], 1))

u = linspace(-1, 1.5, 50)
v = linspace(-1, 1.5, 50)
z = zeros(shape=(len(u), len(v)))
for i in range(len(u)):
    for j in range(len(v)):
        z[i, j] = (map_feature(array(u[i]), array(v[j])).dot(array(theta)))

z = z.T
contour(u, v, z)
title('lambda = %f' % l)
xlabel('Microchip Test 1')
ylabel('Microchip Test 2')
legend(['y = 1', 'y = 0', 'Decision boundary'])
show()

#load the dataset
data2 = np.loadtxt('am-T2-dados/ex2data2.txt', delimiter=',')

#y = np.c_[data2[:,2]]
y = data2[:,2]
y.shape = (y.size, 1)
X = data2[:,0:2]

print(y.shape)

#it = map_feature(X[:, 0], X[:, 1])

X_feature = map_feature(X[:, 0], X[:, 1])
X_feature.shape

def costFunctionReg(theta, reg, X, y):
    m = y.size
    h = sigmoid(X.dot(theta))
    J = -1*(1/m)*(np.log(h).T.dot(y)+np.log(1-h).T.dot(1-y)) + (reg/(2*m))*np.sum(np.square(theta[1:]))

    if np.isnan(J[0]):
        return(np.inf)
    return(J[0])

def gradientReg(theta, reg,  X, y):
    m = y.size
    h = sigmoid(X.dot(theta.reshape(-1,1)))

    grad = (1/m)*X.T.dot(h-y) + (reg/m)*np.r_[[[0]],theta[1:].reshape(-1,1)]

    return(grad.flatten())

initial_theta = np.zeros(X_feature.shape[1])
costFunctionReg(initial_theta, 1, X_feature, y)

initial_theta


I am working on the file "Relatório de Aprendizado de Máquina - Trabalho 02 (LoadTXT).ipynb".
The first task is:
2 - L2 Regularization
The standard way to avoid overfitting is called L2 regularization. It consists of appropriately modifying your cost function, from:
$$J = -\frac{1}{m} \sum\limits_{i = 1}^{m} \large{(}\small  y^{(i)}\log\left(a^{L}\right) + (1-y^{(i)})\log\left(1- a^{L}\right) \large{)} \tag{1}$$
To:
$$J_{regularized} = \small \underbrace{-\frac{1}{m} \sum\limits_{i = 1}^{m} \large{(}\small y^{(i)}\log\left(a^{L}\right) + (1-y^{(i)})\log\left(1- a^{L}\right) \large{)} }\text{cross-entropy cost} + \underbrace{\frac{1}{m} \frac{\lambda}{2} \sum\limits_l\sum\limits_k\sum\limits_j W{k,j}^{[l]2} }_\text{L2 regularization cost} \tag{2}$$
Let's modify your cost and observe the consequences.
Exercise: Implement compute_cost_with_regularization() which computes the cost given by formula (2). To calculate $\sum\limits_k\sum\limits_j W_{k,j}^{[l]2}$  , use :
python
np.sum(np.square(Wl))
Note that you have to do this for $W^{[1]}$, $W^{[2]}$ and $W^{[3]}$, then sum the three terms and multiply by $ \frac{1}{m} \frac{\lambda}{2} $.
Can you write Python code for it?

import numpy as np

# função de custo
def loss_function(theta, X, y, Lambda):

    m = y.size
    H = np.subtract(X,y)
    J = 1/(2*m) * np.sum(H**2)
    custo = J + (1/(2*m) * np.multiply(Lambda,np.sum(theta**2)))
    return custo

def prints(self, epoch):
    print("--epoca %s: " % epoch)
    print("loss: ", self.loss[epoch])
    print("theta: ", self.theta_0.reshape(theta[0].shape[0]), self.theta_n.reshape(theta[1].shape[0]))

def gradient_descent(self, epochs, X, Y, learning_rate, Lambda, m, print_results):
    for i in xrange(epochs):
        # calcula H
        H = np.dot(self.theta_n.T, X) + self.theta_0

        # calcula gradientes
        gH = H - Y

        gTheta_n = np.dot(X, gH.T)/m
        gTheta_0 = np.sum(gH)/m

        # calcula função de custo
        loss = self.loss_function(Y, gH, Lambda, m)
        self.loss.append(loss)

        # atualiza pesos
        self.theta_0 -= learning_rate*gTheta_0
        self.theta_n = self.theta_n*(1-(learning_rate*Lambda/m)) - learning_rate*gTheta_n

        if print_results:
            self.prints(i)

    # calcula função de custo final
    # calcula H
    H = np.dot(self.theta_n.T, X) + self.theta_0

    # calcula gradientes
    gH = H - Y
    loss = self.loss_function(Y, gH, Lambda, m)
    self.loss.append(loss)


I am working on the file "Relatório de Aprendizado de Máquina - Trabalho 02 (LoadTXT).ipynb".
The first task is:
2. Regressão Linear com Múltiplas Variáveis
Nessa parte do trabalho, você irá implementar a regressão linear com múltiplas variáveis para predizer
o preço de venda de imóveis. O arquivo ex1data2.txt contém informações acerca de preços de imóveis.
A primeira coluna corersponde ao tamanho do imóvel (em pés quadrados). A segunda coluna corresponde à
quantidade de dormitórios no imóvel em questão. A terceira coluna corresponde ao preço do imóvel.
Visualização dos dados
Can you write Python code for it?

# Carregando o dataset na forma de txt, porque tem melhor desempenho p/ representação gráfica e
# principalmente na vertorização, se comparado ao Dataframe, utilizado na primeira seção deste trabalho.

data = np.loadtxt('am-T1-dados/ex1data2.txt', delimiter=',')

data

# Inicializando visualização gráfica
plano = plt.figure()
plano = plano.add_subplot(111, projection='3d')

# Criação dos pontos ('x' vermelhos) para o gráfico
for c, m in [('red', 'x')]:
    tamanho = data[:, 0]
    quartos = data[:, 1]
    valor   = data[:, 2]
    plano.scatter(tamanho, quartos, valor, c=c, marker=m)

# Plotando o Gráfico
plano.set_xlabel('Tamanho')
plano.set_ylabel('Nº de Quartos')
plano.set_zlabel('Valor do Imóvel')
show()


I am working on the file "Relatório de Aprendizado de Máquina - Trabalho 02 (LoadTXT).ipynb".
The first task is:
3. Regressão Logística
Nessa parte do trabalho, você irá implementar a regressão logística. Em particular,
você irá criar uma classificador para predizer se um estudante será admitido em uma
universidade, com base nos resultados de duas avaliações. Suponha que estão disponíveis
dados históricos acerca de realizações passadas dessas avaliações, e que esses dados
históricos podem ser usados como conjunto de treinamento. Para cada exemplo desse conjunto
de treinamento, temos as notas das duas avaliações e a decisão acerca do candidato
(aprovado ou reprovado).
Sua tarefa é construir um modelo de classificação que provê uma estimativa da probabilidade
de admissão de um candidato, com base na notas que ele obteve nas duas avaliações.
O arquivo ex2data1.txt contém os dados a serem usados nessa parte do trabalho.
3.1 Visualização dos dados
Antes de começar a implementar qualquer algoritmo de aprendizado, é adequado
visualizar os dados, quando possível. Nessa parte do trabalho, você deve
carregar o arquivo com o conjunto de treinamento e plotar (i.e., produzir um
gráfico) os pontos de dados. O resultado dessa tarefa deve ser um gráfico similar
ao apresentado na Figura abaixo.
Can you write Python code for it?

# carregando os dados
data = np.loadtxt('am-T1-dados/ex2data1.txt', delimiter=',', usecols=(0,1,2), unpack=True)

# Transportando matriz
X = np.transpose(np.array(data[:2]))
y = np.transpose(np.array(data[2:]))

# Tamanho do conjunto de treinamento
m = y.size

# Adicionando uma coluna de 1's ao novo X
X = np.insert(X,0,1,axis=1)

# Classificando a amostra em Positiva (data[:, 2]=1) e Negativa(data[:, 2]=0)
X_Admitted  = np.array([X[i] for i in range(X.shape[0]) if y[i] == 1])
X_Nadmitted = np.array([X[i] for i in range(X.shape[0]) if y[i] == 0])

plt.figure(figsize=(16,8))
plt.plot(X_Admitted[:, 1], X_Admitted[:, 2],'k+',label='Admitted')
plt.plot(X_Nadmitted[:, 1], X_Nadmitted[:, 2],'yo',label='Not admitted')
plt.title('Gráfico Decision Boundary p/ admissão de candidato', fontsize=18, fontweight='bold')
plt.xlabel('Exam 1 Score', fontweight='bold')
plt.ylabel('Exam 2 Score', fontweight='bold')
plt.legend()
plt.grid(False)
plt.show()


I am working on the file "Relatório de Aprendizado de Máquina - Trabalho 02 (LoadTXT).ipynb".
The first task is:
3.2 Implementação
3.2.1 Função sigmoide
Como primeiro passo nessa parte, implemente a função em Python que calcula
o valor da função sigmoide. Defina essa função em um arquivo denominado
sigmoide.py, de tal forma que ela possa ser chamada de outras parte do seu
código. Após finalizar sua implementação, você pode verificar sua corretude:

Para a sigmoide(0), o valor retornado deve ser 0.5.
Para valores muito grandes positivos (ou negativos), ela retornará um valor muito próximo de 1 (ou de 0).

O seu código também deve funcionar com vetores (i.e., o seu código deve estar vetorizado). Em particular, se
uma matriz for passada, o seu código deve aplicar a função sigmoide a cada componente.
Can you write Python code for it?

def sigmoid(x):
    '''
    A função sigmoid
    '''
    g = np.array([x]).flatten()
    s = 1 / (1 + np.exp(-g))
    return s

print('\t ###########################  Teste para função sigmoid(0)  ###########################\n')
print('\t O Valor da Sigmoid(0) é', sigmoid(0))
print('\t O Valor da Sigmoid([0,1,2,3000]) é', sigmoid(np.array([0,1,2,3000])))
print('\t ######################################################################################\n')

# Exibindo o gráfico da função Sigmoid
X_teste = np.arange(-6,6,.5)
plt.plot(X_teste, sigmoid(X_teste))
plt.title("Função Sigmoid", fontsize=18, fontweight='bold')
plt.grid(True)
plt.show()


I am working on the file "Remembering1.ipynb".
The first task is:
At this point, we're able to seek a specific row from the Elements table, or request all of them.  In a Flask web application, the controlling argument might come from a GET request, i.e. a URL such as /api/elements?elem=H
Can you write Python code for it?

output = ""
with Connector(elemsDB(DB1)) as dbx:
    output = dbx.seek("C")

print(output)


I am working on the file "Remembering1.ipynb".
The first task is:
Lets check our understanding that the native types -- the ones we count on to build more complex types -- live in builtins:
Can you write Python code for it?

for the_string in ["list", "tuple", "dict", "int", "float"]:
    if the_string in the_builtins:
        print("Yes I am a native type: ", the_string)
        assert type(eval(the_string)) == type # all types in this club
    else:
        print("No, I'm not native: ", the_string)


I am working on the file "Rendimiento Computacional.ipynb".
The first task is:
Se crean las estructuras (data frames) que almacenan los datos de los tiempos de ejecución para ambas grillas
Can you write Python code for it?

header_names = ["threads","T2000Mean","T2000Std","T4000Mean","T4000Std","T8000Mean","T8000Std"]

df_128x128 = pd.read_csv("https://raw.githubusercontent.com/gabriel-astudillo/jupyter/master/tejec-128x128.txt",
                         delim_whitespace = True,
                         header = None,
                         names = header_names
                        )

df_256x256 = pd.read_csv("https://raw.githubusercontent.com/gabriel-astudillo/jupyter/master/tejec-256x256.txt",
                         delim_whitespace = True,
                         header = None,
                         names = header_names
                        )


I am working on the file "Report.ipynb".
The first task is:
And for xyz.com, we see a different pattern. the checkout amounts are lower, and the number of items bought together is much lower. So shipping costs are lower.
We can get the data for 7/2/2017 using gb_frames[2], etc, and for 7/3/2017 using gb_frames[4], etc. They will not be displayed here.
It's a bit difficult to generalize this to other columns without further analysis, and it's tedious to do nlargest for each column. Let's look at the correlation between order amounts for each website. First, an example. Then we will look at each date individually.
Note: The checkout amount is collinear with the item counts.
Can you write Python code for it?

corr = gb_frames[0]["frame"][item_lists].corr()
ax = sns.heatmap(corr, xticklabels=corr.columns.values, yticklabels=corr.columns.values,
                 linewidths=.5, cmap="YlGnBu")
plt.title("{} Item Correlation for {}".format(gb_frames[0]["domain_name"], gb_frames[0]["date"]));


I am working on the file "Report.ipynb".
The first task is:
Before we comment further, here are total sales per file_date. This is the date we got from the file name, assuming that each file contains a single day of data.
Can you write Python code for it?

pd.pivot_table(dfr, values="checkout_amount", index="file_date", columns="domain_name",
               aggfunc=[np.sum], margins=True)


I am working on the file "Report.ipynb".
The first task is:
Completeness check for Seattle
Can you write Python code for it?

seattle_missing_count = {}
for col_name in seattle.columns:
    seattle_missing_count[col_name] = len(seattle[seattle[col_name].isnull()])

seattle_missing_count


I am working on the file "Report.ipynb".
The first task is:
Consistency Check for Seattle
Can you write Python code for it?

seattle['date_time'] = seattle['Occurred Date or Date Range Start']
date_idx = pd.DatetimeIndex(seattle['date_time'])
seattle['incident_date'] = date_idx.date.astype('datetime64')
seattle['incident_hour'] = date_idx.hour
seattle['incident_year'] = date_idx.year
seattle['incident_month'] = date_idx.month
seattle['incident_weekday'] = date_idx.weekday

by_year = seattle.pivot_table('General Offense Number', aggfunc='count',
                            index='incident_year',
                            columns='incident_month')
by_year


I am working on the file "Report.ipynb".
The first task is:
Data Challenge
Files are stored in an S3 bucket. The purpose here is to fully analyze the data and make some predictions.
This workbook was exported to a Python script and the resulting code was checked for PEP8 problems. The problems found were with formatting and order of imports. We can't fix the latter due to the way the script is exported from the notebook.
Any known or potential performance problems are flagged for further work.
Some of the information here is based on posts from Stackoverflow. I haven't kept up with it, so I'll just indicate by flagging with SO.
Prepare an AWS Instance
We're using a tiny instance here since the data and processing requirements are minimal.

Install Anaconda for Python 3.6.
Install the boto3 and seaborn packages.
Install and configure the AWS command-line tools.
Use jupyter notebook --generate-config to generate a config file in ~/.jupyter. An example is enclosed in the GitHub repository.
Run conda install -c conda-forge jupyter_contrib_nbextensions.
Configure Jupyter on the remote side to run without a browser and to require a password. Make sure you put the config file in the proper place.
On the client side, run ssh -N -L 8888:localhost:8888 ubuntu@ec2-34-230-78-129.compute-1.amazonaws.com (the hostname may vary). This sets up the ssh tunnel and maps to port 8888 locally.
ssh into the instance using ssh ubuntu@ec2-34-230-78-129.compute-1.amazonaws.com.
Start jupyter notebook.

Note: You can start the tmux terminal multiplexer in order to use notebooks when you are logged out.
Note: This doesn't start X windows tunnelling, which is a separate configuration and is not presented here.
Get the Data from S3
Instead of processing the data with the boto3.s3 class, we chose to
bash
aws s3 cp --recursive s3://my_bucket_name local_folder
This is not scalable, so we can use code such as
```python
s3 = boto3.resource('s3')
bucket_name = "postie-testing-assets"
test = s3.Bucket(bucket_name)
s3.meta.client.head_bucket(Bucket=bucket_name)
{'ResponseMetadata': {'HTTPHeaders':...
Can you write Python code for it?

def convert_list(query_string):
    """Parse the query string of the url into a dictionary.

    Handle special cases:
    - There is a single query "error=True" which is rewritten to 1 if True, else 0.
    - Parsing the query returns a dictionary of key-value pairs. The value is a list.
      We must get the list value as an int.

    Note: This function may be a bottleneck when processing larger data files.
    """
    def handle_error(z, col):
        """
        Called in the dictionary comprehension below to handle the "error" key.
        """
        if "error" in col:
            return 1 if "True" in z else 0
        return z
    dd = parse_qs(query_string)
    return {k: int(handle_error(dd[k][0], k)) for k in dd}


I am working on the file "Report.ipynb".
The first task is:
Here are a few modeling considerations.

We only have three days to work with. That's not enough.
There are questions to clear up about the data.
How does the app version and placeholder affect the data?

However, we can make some predictions at this point. First, we can make the following prediction by averaging.
Can you write Python code for it?

for col in pt.columns.tolist():
    print("Tomorrow's (7/14/2017) sales for {} is {}".format(col, pt[col].mean()))


I am working on the file "Report.ipynb".
The first task is:
It's very interesting to see that the item correlations are much lower. Originally, we used the same colormap as for the first heatmap, but the lower correlations didn't look as well (the colors wash out). It's difficult to distinguish the colors here. More research is needed to produce meaningful colors in these plots.
At this point, we could write a few loops and use subplots to display more information.
Sales Prediction
Let's take a look at the daily sales again.
Can you write Python code for it?

pt = pd.pivot_table(dfr, values="checkout_amount", index="transaction_date", columns="domain_name",
                    aggfunc=[np.sum], margins=False)
pt.columns = ["example.com", "xyz.com"]  # get rid of multiindex
pt


I am working on the file "Report.ipynb".
The first task is:
Just to make sure there are no inconsistencies in the data, let's check the website_id against the domain name we've extracted.
Can you write Python code for it?

pd.pivot_table(dfr, values="website_id", index="transaction_date", columns="domain_name", aggfunc=[np.max])


I am working on the file "Report.ipynb".
The first task is:
Potential Performance Issues

Import code processing by file: handle url and domain name processing outside the loop.
Wrap import code into a function and read in the data using a list comprehension.
Keeping a list of DataFrames for each file imported is not necessary and should be eliminated in pipeline code.
Extract the individual item price by domain name upon import.

Summaries of the Data
First, let's check out the total sales per transaction_date. This is the date from the timestamp field.  A separate field file_date will also be used to reveal problems with the data.
Note: These pivot tables can obviously be reformatted but we will not bother here. That's something to do when presenting externally.
Can you write Python code for it?

pd.pivot_table(dfr, values="checkout_amount", index="transaction_date", columns="domain_name",
               aggfunc=[np.sum], margins=True)


I am working on the file "Report.ipynb".
The first task is:
Seattle Analysis

In general "Property" related crimes are higher than all others.
The next highest category is "Inchoate"
There does not seem to be any specific trend in the crimes over dates or over the weekdays
When we chart the crimes on the hours of the day, there seems to be a definite trend. Number of incidents peak at midnight and then drop as the day starts. After 5 am there is a sharp increase in incidents and it seems to peak around noon. After this peak there is a slower increase in the incident rates.
Most incidents span between 10am and midnight.

San Francisco Incidents trend by date (By Category)
Can you write Python code for it?

by_date = sanfran.pivot_table('IncidntNum', aggfunc='count',
                            index='incident_date',
                            columns='incident_category' )
ax = by_date.plot()
ax.figure.savefig('figs/sanfran_incidents_by_date.png', bbox_inches='tight')


I am working on the file "Report.ipynb".
The first task is:
Seattle Incidents trend by date (By Category)
Can you write Python code for it?

by_date = seattle.pivot_table('General Offense Number', aggfunc='count',
                            index='incident_date',
                            columns='incident_category' )
ax = by_date.plot()
ax.figure.savefig('figs/seattle_incidents_by_date.png', bbox_inches='tight')


I am working on the file "Report.ipynb".
The first task is:
Seattle Incidents trend by weekday (By Category)
Can you write Python code for it?

by_weekday = seattle.pivot_table('General Offense Number', aggfunc='count',
                            index='incident_weekday',
                            columns='incident_category')

ax = by_weekday.plot()
ax.figure.savefig('figs/incidents_by_weekday.png', bbox_inches='tight')


I am working on the file "Report.ipynb".
The first task is:
There is only one year and data spans for the three months of Jun-Aug. Looks consistent although there is no way to confirm.
Incident Categorization based on this site
http://www.legalmatch.com/law-library/article/what-are-the-different-types-of-crimes.html
I took the 4 prominent categories and remapped them to the best of my knowledge.

Personal Crimes – “Offenses against the Person”: These are crimes that result in physical or mental harm to another person.
Property Crimes – “Offenses against Property”: These are crimes that do not necessarily involve harm to another person. Instead, they involve an interference with another person’s right to use or enjoy their property.
Inchoate Crimes – “Inchoate” translates into “incomplete”, meaning crimes that were begun, but not completed. This requires that a person take a substantial step to complete a crime, as opposed to just “intend” to commit a crime.
Statutory Crimes – A violation of a specific state or federal statute and can involve either property offenses or personal offense.
Other Offences – Anything that could not be mapped to the above 4 categories

See more at: http://www.legalmatch.com/law-library/article/what-are-the-different-types-of-crimes.html#sthash.xHKGRbs4.dpuf
Can you write Python code for it?

map_categories = {
'[INC - CASE DC USE ONLY]': 'OTHER OFFENSES',
'ANIMAL COMPLAINT': 'OTHER OFFENSES',
'ARSON': 'PROPERTY',
'ASSAULT': 'PERSONAL',
'BIAS INCIDENT': 'INCHOATE',
'BIKE THEFT': 'PROPERTY',
'BRIBERY': 'PROPERTY',
'BURGLARY': 'PROPERTY',
'BURGLARY-SECURE PARKING-RES': 'PROPERTY',
'CAR PROWL': 'INCHOATE',
'COUNTERFEIT': 'PROPERTY',
'DISORDERLY CONDUCT': 'INCHOATE',
'DISPUTE': 'INCHOATE',
'DISTURBANCE': 'INCHOATE',
'DRIVING UNDER THE INFLUENCE': 'STATUTORY',
'DRUG/NARCOTIC': 'STATUTORY',
'DRUNKENNESS': 'STATUTORY',
'DUI': 'STATUTORY',
'ELUDING': 'INCHOATE',
'EMBEZZLE': 'PROPERTY',
'EMBEZZLEMENT': 'PROPERTY',
'ESCAPE': 'STATUTORY',
'EXTORTION': 'INCHOATE',
'FALSE REPORT': 'INCHOATE',
'FAMILY OFFENSES': 'INCHOATE',
'FIREWORK': 'PROPERTY',
'FORGERY': 'PROPERTY',
'FORGERY/COUNTERFEITING': 'PROPERTY',
'FRAUD': 'PROPERTY',
'GAMBLING': 'PROPERTY',
'HOMICIDE': 'PERSONAL',
'ILLEGAL DUMPING': 'STATUTORY',
'INJURY': 'PERSONAL',
'KIDNAPPING': 'PERSONAL',
'LARCENY/THEFT': 'PROPERTY',
'LIQUOR LAWS': 'STATUTORY',
'LIQUOR VIOLATION': 'STATUTORY',
'LOITERING': 'INCHOATE',
'LOST PROPERTY': 'PROPERTY',
'MAIL THEFT': 'PROPERTY',
'MISSING PERSON': 'PERSONAL',
'NARCOTICS': 'STATUTORY',
'NON-CRIMINAL': 'NON-CRIMINAL',
'OBSTRUCT': 'PROPERTY',
'OTHER OFFENSES': 'OTHER OFFENSES',
'OTHER PROPERTY': 'PROPERTY',
'PICKPOCKET': 'PROPERTY',
'PORNOGRAPHY': 'INCHOATE',
'PORNOGRAPHY/OBSCENE MAT': 'INCHOATE',
'PROPERTY DAMAGE': 'PROPERTY',
'PROSTITUTION': 'INCHOATE',
'PUBLIC NUISANCE': 'INCHOATE',
'PURSE SNATCH': 'PROPERTY',
'RECKLESS BURNING': 'PROPERTY',
'RECOVERED PROPERTY': 'PROPERTY',
'ROBBERY': 'PROPERTY',
'RUNAWAY': 'INCHOATE',
'SECONDARY CODES': 'STATUTORY',
'SHOPLIFTING': 'PROPERTY',
'STOLEN PROPERTY': 'PROPERTY',
'SUICIDE': 'PERSONAL',
'SUSPICIOUS OCC': 'PROPERTY',
'THEFT OF SERVICES': 'PROPERTY',
'THREATS': 'INCHOATE',
'TRAFFIC': 'STATUTORY',
'TRESPASS': 'INCHOATE',
'VANDALISM': 'PROPERTY',
'VEHICLE THEFT': 'PROPERTY',
'VIOLATION OF COURT ORDER': 'STATUTORY',
'WARRANT ARREST': 'STATUTORY',
'WARRANTS': 'STATUTORY',
'WEAPON':...


I am working on the file "Report.ipynb".
The first task is:
This is a boxplot, which is a good way to display outliers, the median, quartiles, and range. We can see immediately that we have a problem with the wide range of checkout amounts. This is why the interquartile range is so compressed. There are two options.

The large values are not legitimate transactions. In this case, they are true outliers and should be ignored.
The large values are definitely legitimate transactions. This will complicate any predictive model.

We need more information about sales. We need to know if the negative values are recorded wrong.
Extracting Prices from the URL
Let's now use the pricing information extracted from the url. We can query items in the DataFrame to find this information. At this point, we assume that prices are not changed on items during the day. This seems to be the case with this data, but we can't make that assumption about future data.
Note: We are doing the item prices separately since the method used to produce the DataFrame above isn't adaptable. A different method of processing the data which avoids using the convienence of the pd.Series constructor can be used, at the expense of more difficult processing of the parsed url. Also, it's possible that an item only appears for a given domain name on a particular day, meaning we must process all the data first.
First, to get the rows where only one item was purchased, we can use (SO)
python
dfr[dfr[cols].astype(bool).sum(axis=1) == 1]
We then calculate the pricing and reformulate the data into a table we can view. This code is annoying to write, so it's preferable to redo the above loop to get the pricing. However, that may slow down processing.
Can you write Python code for it?

cols = [item for item in item_lists if not "error" in item]
pricing_temp = dfr[dfr[cols].astype(bool).sum(axis=1) == 1].copy()  # to avoid setting values on a view
pricing_temp.drop_duplicates(subset=cols + ["domain_name", "transaction_date"], inplace=True)
price_cols = []
for col in cols:
    price_cols.append(np.abs(pricing_temp["checkout_amount"]/pricing_temp[col]))
pricing = pd.concat(price_cols, axis=1)
pricing.columns = cols
price_cols = [col + "_price" for col in cols]
px = pricing_temp.join(pricing, rsuffix="_price")[price_cols + ["transaction_date", "domain_name"]]
px = px.replace([np.inf, -np.inf], np.nan).fillna(value=0)
pd.pivot_table(px, values=price_cols, index="transaction_date", columns="domain_name", aggfunc=np.max).transpose()


I am working on the file "Report.ipynb".
The first task is:
We can also draw a regression plot and make predictions using the regression line.
First, there are a few details with the index. Placing the labels is also annoying. The alternatives are to use the Pandas plotting capabilities. In fact, there is no really good solution to plotting a time series and associated regression line without creating the regression line values in the DataFrame. This latter idea is what I usually do, however, the below charts were produced in a different way.
Note that both show a clear linear trend. However, there are problems - we just don't have enough data. regplot is a good tool for linear regression plots, but it does have its deficiencies, such as not providing the slope and intercept of the regression line. Additionally, the error region is not meaningful with this small amount of data.
Can you write Python code for it?

pt = pd.pivot_table(dfr, values="checkout_amount", index="transaction_date", columns="domain_name",
               aggfunc=[np.sum], margins=False)
pt.columns = ['example.com', 'xyz.com']
pt.index = pd.DatetimeIndex(pt.index)
idx = pd.date_range(pt.index.min(), pt.index.max())
pt = pt.reindex(index=idx)
pt.insert(pt.shape[1],
          'row_count',
          pt.index.value_counts().sort_index().cumsum())
slope, intercept, r_value, p_value, std_err = linregress(x=pt["row_count"].values,
                                                         y=pt["example.com"].values)
pt["example.com regression line"] = intercept + slope * pt["row_count"]
ax = pt[["example.com", "example.com regression line"]].plot()


I am working on the file "Report.ipynb".
The first task is:
Графики по данным, полученным из main_tooled.cpp
Can you write Python code for it?


I am working on the file "Report.ipynb".
The first task is:
Графики по данным, полученным из valgrind
Can you write Python code for it?


I am working on the file "Report3.ipynb".
The first task is:
Below are some examples of foods within the 6 groups made by the K-Means algorithm. We present to you, for fun, a small list of foods in each category, revealing some interesting wonders, as well as potentially some flaw in our design choices.
Can you write Python code for it?

def Examples(df, label_int, si = [0,5]):
    """ Creates a 3D log log plot on the requested subplot.
    Arguments:
    pred_cat = predicted dataframe for a category
    actual_cat = dataframe of the real category
    ax = plt axis instance
    actual_label = string with label for the actual category
    colors = list with two entries of strings for color names
    """
    TP = df[(df.Label == label_int)&(df.correct_guess==True)]
    FP = df[(df.guess == label_int)&(df.correct_guess==False)]
    print("Guessed Similar:")
    print(TP["Food Name"][si[0]:si[1]])
    print("\nSurprising:")
    print(FP["Food Name"][si[0]:si[1]])


I am working on the file "Report3.ipynb".
The first task is:
Below is the 3D scatter plot showing all the clusters from the K-Means algorithm.
Can you write Python code for it?

ax = plt.subplot(projection='3d')

ax.scatter3D(np.log(X.Protein),np.log(X.Carbs), np.log(X.Fat), c = y_kmeans)
ax.view_init(elev=10, azim=45)

ax.set_xlabel('Protein (log g)')
ax.set_ylabel('Carbohydrate (log g)')
ax.set_zlabel('Fat (log g)')


I am working on the file "Report3.ipynb".
The first task is:
The proposed Cereal Group, followed by the proposed Fruits group.
Can you write Python code for it?

print('High Carb, Low Sugar Group')
Examples(all_these, 2)
print('\nHigh Carb, High Sugar Group')
Examples(all_these, 3)


I am working on the file "Report3.ipynb".
The first task is:
The proposed Fat group, followed by the proposed Dairy group.
Can you write Python code for it?

print('High Fat Group')
Examples(all_these, 4)
print('\nMid Everything Group')
Examples(all_these, 5)


I am working on the file "Report3.ipynb".
The first task is:
The proposed Meat group, followed by the proposed Vegetable group.
Can you write Python code for it?

print('High Protein Group')
Examples(all_these, 0)
print('\nLow Everything Group')
Examples(all_these, 1)


I am working on the file "Report3.ipynb".
The first task is:
This one is the same as above, except it shows a comparison between the two for the fruit group.
Can you write Python code for it?

ax = plt.subplot(projection='3d')
ThreeDPlot(labeled3, fruits, ax, 'Fruits', ['firebrick','purple'])


I am working on the file "Report3.ipynb".
The first task is:
To gain a better understanding of what's happening, we decided to create a confusion matrix to view all the different individual accuracies between clusters. Below shows all of the clusters and where the foods ended up being grouped. Note that a perfect match would mean that the main diagonal of the matrix would be entirely black.
Can you write Python code for it?

# Look at confusion matrix for some idea of accuracy. Meats has the highest rate of matching.

labels = ["meats", "vegetables", "cereal", "fruit", "oils", "dairy"]
predlabels = ["high protein", "all low", "high carb, low sugar", "high carb, high sugar", "high fat", "all medium"]
mat = confusion_matrix(all_these.Label, predicted_labels)
sns.heatmap(mat.T, square=True, xticklabels=labels, yticklabels=predlabels, annot=True, fmt="d", linewidth=.5)
plt.xlabel('Food Pyramid label')
plt.ylabel('K-Means label')
plt.title("Matrix Comparison of K-Means vs. Food Pyramid")


I am working on the file "Report3.ipynb".
The first task is:
To get a sense of what we're dealing with, we directly plotted all foods on a log-log scale using a scatter plot, with the Carbs and Proteins as the axes and Fat as the radius of the dots. At first glance, it appears that the foods aren't particularly differentiated by nutrient content.
Can you write Python code for it?

x_vals = 'Protein'
y_vals = 'Carbohydrate'
z_vals = 'Fat'

food_group_dict = {'A':['Cereals','peru'], 'B':['Dairy','beige'], 'C':['Egg','paleturquoise'],
                  'D':['Vegetable','darkolivegreen'], 'F':['Fruit','firebrick'], 'G':['Nuts','saddlebrown'],
                  'J':['Fish','slategray'],'M':['Meat','indianred'], 'O':['Fat','khaki']}

ax = plt.subplot(111)
for key,val in food_group_dict.items():
    df = tester[tester.Group.str.startswith(key, na=False)]
    ax.scatter(df[x_vals],df[y_vals],df[z_vals],color=val[1],label = val[0])
plt.xscale('log')
plt.yscale('log')
ax.set_xlabel(x_vals+' (g)')
ax.set_ylabel(y_vals+' (g)')
ax.legend()


I am working on the file "Report3.ipynb".
The first task is:
We created a function that could take in a category and plot a 3D scatter plot of the data, thus giving a modular function to use going into the rest of the code.
Can you write Python code for it?

def ThreeDPlot(pred_cat, actual_cat, ax, actual_label, colors = ['firebrick', 'peru']):
    """ Creates a 3D log log plot on the requested subplot.
    Arguments:
    pred_cat = predicted dataframe for a category
    actual_cat = dataframe of the real category
    ax = plt axis instance
    actual_label = string with label for the actual category
    colors = list with two entries of strings for color names
    """
    ax.scatter3D(np.log(pred_cat.Protein),np.log(pred_cat.Carbs), np.log(pred_cat.Fat), c = colors[0], label = 'Predicted Group')
    ax.scatter3D(np.log(actual_cat.Protein),np.log(actual_cat.Carbohydrate), np.log(actual_cat.Fat), c = colors[1], label = actual_label, alpha= .5)
    ax.view_init(elev=10, azim=45)

    ax.set_xlabel('Protein (log g)')
    ax.set_ylabel('Carbohydrate (log g)')
    ax.set_zlabel('Fat (log g)')
    plt.legend()


I am working on the file "Report3.ipynb".
The first task is:
We now begin creating labels for the different food groups. Here we add a new column to hold the category given to the food by the Food Pyramid. We then create a new dataframe with only these categories for use in the rest of the code. The specific labels given are based on later knowledge of which K-Means groups matched the best with the Food Pyramid groups based on nutrient content.
Can you write Python code for it?

cereals = tester[tester.Group.str.startswith('A', na=False)]
cereals['Label'] = cereals.Protein*0+2

fruits = tester[tester.Group.str.startswith('F', na=False)]
fruits['Label'] = fruits.Protein*0+3

veggies = tester[tester.Group.str.startswith('D', na=False)]
veggies['Label'] = veggies.Protein*0+1

dairy = tester[tester.Group.str.startswith('B', na=False)]
dairy['Label'] = dairy.Protein*0+5

oils = tester[tester.Group.str.startswith('O', na=False)]
oils['Label'] = oils.Protein*0+4

m1 = tester[tester.Group.str.startswith('J', na=False)]
m2 = tester[tester.Group.str.startswith('M', na=False)]
meats = pd.concat([m1,m2])
meats['Label'] = meats.Protein*0

all_these = pd.concat([cereals, fruits, veggies, dairy, oils, meats])


I am working on the file "Report3.ipynb".
The first task is:
We now separate out different categories for analysis.
Can you write Python code for it?

# Create a way to select the categories

predicted_labels = pd.DataFrame(y_kmeans, index=X.index).astype(float)
X['predictions'] = predicted_labels

# Separate out the categories for individual analysis

labeled0 = X[X.predictions == 0]
labeled1 = X[X.predictions == 1]
labeled2 = X[X.predictions == 2]
labeled3 = X[X.predictions == 3]
labeled4 = X[X.predictions == 4]
labeled5 = X[X.predictions == 5]


I am working on the file "Report3.ipynb".
The first task is:
We the make another column that holds the correct guess for each food, in other words whether that food based on its K-Means group was placed in the same group as the one given to it from the Food Pyramid.
Can you write Python code for it?

all_these['guess'] = predicted_labels[0]
all_these['correct_guess'] = np.where((all_these.Label == all_these.guess), True, False)


I am working on the file "Report3.ipynb".
The first task is:
We then create a function to read in our dataset and clean it, pruning specifically the columns that we care about.
Can you write Python code for it?

def ReadProximates():
    """ Reads the correct sheet from the Excel Spreadsheet downloaded from the databank.
    Cleans the macronutrient data and replaces non-numerical entries with 0.

    Returns: cleaned DataFrame
    """
    df = pd.read_excel('dietary.xls', sheetname='Proximates')
    column_list = ['Water (g)', 'Protein (g)', 'Fat (g)', 'Carbohydrate (g)', 'Total sugars (g)']

    df['Protein'] = pd.to_numeric(df['Protein (g)'], errors='coerce')
    df['Fat'] = pd.to_numeric(df['Fat (g)'], errors='coerce')
    df['Carbohydrate'] = pd.to_numeric(df['Carbohydrate (g)'], errors='coerce')
    df['Sugars'] = pd.to_numeric(df['Total sugars (g)'], errors='coerce')

    df['Protein'].replace([np.nan], 0, inplace=True)
    df['Fat'].replace([np.nan], 0, inplace=True)
    df['Carbohydrate'].replace([np.nan], 0, inplace=True)
    df['Sugars'].replace([np.nan], 0, inplace=True)

    return df


I am working on the file "Report3.ipynb".
The first task is:
We then create our cleaned dataset.
Can you write Python code for it?


I am working on the file "Report3.ipynb".
The first task is:
We then generate an accuracy score using sklearn, which evaluates whether a particular food in K-Means was categorized in the same group as in the Food Pyramid across all foods. The result, 57% is barely above half.
Can you write Python code for it?


I am working on the file "Repos for an Organization.ipynb".
The first task is:
Open Issue Counts
Can you write Python code for it?

print('{0:30} {1:20}'.format("Repository name", "Open Issues"))
print('{0:30} {1:20}'.format("---------------", "-----------"))
for repo in repos:
    #print('%s: %d' % (repo.name, repo.open_issues_count))
    print('{0:30} {1:3d}'.format(repo.name, repo.open_issues_count))


I am working on the file "Repos for an Organization.ipynb".
The first task is:
Repo Creation Date and description
Can you write Python code for it?

import datetime
print('{0:30} {1:25} {2:30}'.format("Repository name", "created", "Description"))
print('{0:30} {1:25} {2:30}'.format("---------------", "-------", "-----------"))
for repo in repos:
    #print('%s: %d' % (repo.name, repo.open_issues_count))
    print('{0:30} {1:25} {2:30}'.format(repo.name, str(repo.created_at), repo.description))

dir(repo)


I am working on the file "Repos for an Organization.ipynb".
The first task is:
Repos
Can you write Python code for it?

# Get all repos of the ORGANIZATION
repos = [r.refresh() for r in gh.repositories_by(ORGANIZATION)]


I am working on the file "Repos_Data/legacy/plotter_notebook.ipynb".
The first task is:
Filter - spline fitting and velocity evaluation
Can you write Python code for it?

def filtering_data(data,meta,volumes,angles,location):
    fig1 = plt.figure(facecolor="white")
    fig2 = plt.figure(facecolor="white")
    #plt.style.use('classic')
    #font = {'family' : 'Times New Roman',
    #        'weight' : 'bold',
    #        'size'   : 16}
    #plt.rc('font',**font)
    v_norm = {}
    x_max = {}

    for key in sorted(meta, key = lambda x: meta[x]['ILback']):
        if(meta[key]['Volume'] in volumes and meta[key]['Angle'] in angles):  # checks for called volumes and angles
            dat = data[key][location][1:]
            t = data[key]['time'][1:]

            slice_num = 5
            dat_sliced = dat[::slice_num]
            t_sliced = t[::slice_num]
            x_max[key] = max(dat_sliced)

            nn = int(((np.ceil(np.size(dat_sliced)/2))*2-1)-10)
            nn2 = nn-10
            nn3 = nn

            poly_fit = 5
            x1 = signal.savgol_filter(dat_sliced,nn,poly_fit)
            x2 = signal.savgol_filter(x1,nn,poly_fit)
            x3 = signal.savgol_filter(x2,nn3,poly_fit)
            data_filt = signal.savgol_filter(x3,slice_num+2,poly_fit)
            #print('nn1='+str(nn)+' nn2='+str(nn2)+' nn3='+str(nn3))

            ax = fig1.add_subplot(1,1,1)
            # some plot formatting stuff, setting marker colors/types for various volumes
            if(meta[key]['Volume'] == 0.5):
                color = 'k'
                style = None
            elif(meta[key]['Volume'] == 1.0):
                color = 'c'
                style = None
            elif(meta[key]['Volume'] == 2.0):
                color = 'b'
                style = None
            elif(meta[key]['Volume'] == 5.0):
                color = 'r'
                style = None
            elif(meta[key]['Volume'] == 10.0):
                color = 'y'
                style = None
            if(meta[key]['Angle'] == 2.0):
                style2 = 'o'
            elif(meta[key]['Angle'] == 3.5):
                style2 = 's'
           ...


I am working on the file "Repos_Data/legacy/plotter_notebook.ipynb".
The first task is:
Linear fit to data
Can you write Python code for it?

xi_vector = np.zeros(n_files-1)
v_vector = np.zeros(n_files-1)
i = int(0)
for key in xi_norm:
    if key != '07261':
        xi_vector[i] = xi_norm[key]
        v_vector[i] = v_norm[key]
        i=i+1
    else:
         print()
fit = np.polyfit(xi_vector,v_vector,1)
fit_fn = np.poly1d(fit)


I am working on the file "Repos_Data/legacy/plotter_notebook.ipynb".
The first task is:
Non-dimenionalized plot: $U_{max}^$ vs $x_i^$
Can you write Python code for it?

from matplotlib.lines import Line2D
v_max = ((4*0.072*9.81)/1000)**(1/4.)*1000
fig1 = plt.figure(facecolor="white")
for key in xi_norm:
    R = ((3000*meta[key]['Volume'])/(4*math.pi))**(1/3)
    sin_angl = math.sin((meta[key]['Angle']/2)*(math.pi/180))
    x_max_theo = (R/sin_angl)

    if x_max[key]>x_max_theo:
        fill = 'none'
        w = 1.5

    elif x_max[key]<=x_max_theo:
        fill = 'full'
        w=0

    if meta[key]['Angle']== 2.0:
        clr = 'c'
    elif meta[key]['Angle']== 3.5:
        clr = 'b'
    elif meta[key]['Angle']== 5.0:
        clr = 'r'

    if meta[key]['Volume']== 0.5:
        style = '^'
    elif meta[key]['Volume']== 1.0:
        style = 's'
    elif meta[key]['Volume']== 2.0:
        style = 'o'
    elif meta[key]['Volume']== 5.0:
        style = 'p'
    elif meta[key]['Volume']== 10.0:
        style = 'D'

    ax = fig1.add_subplot(1,1,1)
    dropno = key+' '
    vol = str(meta[key]['Volume'])+'ml '
    angl = str(meta[key]['Angle'])+'deg '
    IL = str( data[key][location][0])+'mm '
    xii = ' '+str(xi_norm[key])
    legend_label = dropno+xii

    ax.plot(xi_norm[key],v_norm[key]/v_max,\
                marker=style,\
                c = clr,\
                ms=9,\
                fillStyle=fill,\
                markeredgewidth=w,\
                label = legend_label,\
                linewidth = 0
           )
    ax.plot(xi_vector,fit_fn(xi_vector)/v_max,'k')

    legend1 = basic_plot_format(subplot_label = ax, \
                      x_label = '$x_i^*$',  \
                      y_label
                                = '$U_{max,msrd}/U_{limit}$', \
                      major_grid = True,  \
                      minor_grid = True,  \
                      legend = True
                     )
    ax.set_xlim([0,1])
    ax.set_ylim([0,0.75])

fig1.savefig('Figures/dimensionalplot.png',dpi=300,bbox_extra_artists=(legend1,), bbox_inches='tight')
plt.show()

fit/v_max


I am working on the file "Repos_Data/legacy/plotter_notebook.ipynb".
The first task is:
Normalized stuff
Can you write Python code for it?

def normalized_variables(meta,data,location):
    xi_norm = {}
    for key in sorted(data):
        xi = data[key][location][0]/1000 # m
        V = meta[key]['Volume']*(10**(-6)) # m^3
        alpha = (meta[key]['Angle']/2.)*(math.pi/180.) # rad
        xi_norm[key] = xi*((4*math.pi)/(3*V))**(1/3.)*math.sin(alpha)

    return xi_norm


I am working on the file "Repos_Data/legacy/plotter_notebook.ipynb".
The first task is:
Plot data - raw
Can you write Python code for it?

def plot_data(data,meta,volumes,location):
    fig = plt.figure(facecolor="white")
    plt.style.use('classic')
    font = {'family' : 'Times New Roman',
            'weight' : 'bold',
            'size'   : 16}
    plt.rc('font',**font)

    for key in sorted(data):
        drop_num = key
        if(meta[drop_num]['Volume'] in volumes ):
            if(meta[drop_num]['Volume'] == 0.5):
                color = 'r'
            elif(meta[drop_num]['Volume'] == 1.0):
                color = 'k'
            elif(meta[drop_num]['Volume'] == 2.0):
                color = 'b'

            ax = fig.add_subplot(1,1,1)
            ax.plot(data[drop_num]['time'],data[drop_num][location],c=color,marker='.',linestyle='None',\
                    label=str(meta[drop_num]['Volume'])+'ml '+str( meta[drop_num]['ILback'])+'mm '+drop_num)

            ax.spines['right'].set_color('none')
            ax.spines['top'].set_color('none')
            ax.yaxis.set_ticks_position('left')
            ax.xaxis.set_ticks_position('bottom')
            ax.minorticks_on()

            ax.legend(bbox_to_anchor=(1, 1),loc='upper left',frameon=False,fontsize=12,facecolor="white")

            ax.set_xlabel('t (s)',fontsize=18)
            ax.set_ylabel('x (mm)',fontsize=18)

            ax.grid(b=True,which='major')
            ax.grid(b=True,which='minor')
    plt.show()


I am working on the file "Repos_Data/legacy/plotter_notebook.ipynb".
The first task is:
Retrieve drop variables
Can you write Python code for it?

def get_drop_vars(metadata):
    f = open(metadata[0],"r")
    drop_vars = list(csv.reader(f))
    f.close()

    drop_meta = {}
    for i in range(n_files):
        drop_meta[drop_vars[i+1][0]] = {}
        n_meta = np.size(drop_vars[0][1:])
        for n in range(n_meta):
            drop_meta[drop_vars[i+1][0]][drop_vars[0][n+1].split('_')[0]] = float(drop_vars[i+1][n+1])

    meta = drop_meta
    return meta


I am working on the file "Repos_Data/legacy/plotter_notebook.ipynb".
The first task is:
Run stuff
Can you write Python code for it?

location = 'average'
volumes = [0.5,1.0,2.0,5.0,10.0]
angles= [2.0,3.5,5.0]

data = gen_data_repos(wrkdir,csv_files,trackdata,n_files)
meta = get_drop_vars(metadata)

#plot_data(data,meta,volumes,location)
[v_norm,x_max] = filtering_data(data,meta,volumes,angles,location)
xi_norm = normalized_variables(meta,data,location)
plt.show()


I am working on the file "Research/BasicQuantBookTemplate.ipynb".
The first task is:
Historical Data Requests
We can use the QuantConnect API to make Historical Data Requests. The data will be presented as multi-index pandas.DataFrame where the first index is the Symbol.
For more information, please follow the link.
Can you write Python code for it?

# Gets historical data from the subscribed assets, the last 360 datapoints with daily resolution
h1 = qb.History(qb.Securities.Keys, 360, Resolution.Daily)

# Plot closing prices from "SPY"
h1.loc["SPY"]["close"].plot()


I am working on the file "Research/BasicQuantBookTemplate.ipynb".
The first task is:
Indicators
We can easily get the indicator of a given symbol with QuantBook.
For all indicators, please checkout QuantConnect Indicators Reference Table
Can you write Python code for it?

# Example with BB, it is a datapoint indicator
# Define the indicator
bb = BollingerBands(30, 2)

# Gets historical data of indicator
bbdf = qb.Indicator(bb, "SPY", 360, Resolution.Daily)

# drop undesired fields
bbdf = bbdf.drop('standarddeviation', 1)

# Plot
bbdf.plot()


I am working on the file "Research/KitchenSinkQuantBookTemplate.ipynb".
The first task is:
Get Fundamental Data

GetFundamental([symbol], selector, start_date = datetime(1998,1,1), end_date = datetime.now())

We will get a pandas.DataFrame with fundamental data.
Can you write Python code for it?

data = qb.GetFundamental(["AAPL","AIG","BAC","GOOG","IBM"], "ValuationRatios.PERatio")
data


I am working on the file "Research/KitchenSinkQuantBookTemplate.ipynb".
The first task is:
Historical Future Data Requests

Select the future data
Sets the filter, otherwise the default will be used SetFilter(timedelta(0), timedelta(35))
Get the FutureHistory, an object that has information about the historical future data
Can you write Python code for it?

es = qb.AddFuture("ES")
es.SetFilter(timedelta(0), timedelta(180))

future_history = qb.GetFutureHistory(es.Symbol, datetime(2013, 10, 7))
print (future_history.GetExpiryDates())
h7 = future_history.GetAllData()


I am working on the file "Research/KitchenSinkQuantBookTemplate.ipynb".
The first task is:
Historical Options Data Requests

Select the option data
Sets the filter, otherwise the default will be used SetFilter(-1, 1, timedelta(0), timedelta(35))
Get the OptionHistory, an object that has information about the historical options data
Can you write Python code for it?

goog = qb.AddOption("GOOG")
goog.SetFilter(-2, 2, timedelta(0), timedelta(180))

option_history = qb.GetOptionHistory(goog.Symbol, datetime(2015, 12, 24))
print (option_history.GetStrikes())
print (option_history.GetExpiryDates())
h7 = option_history.GetAllData()


I am working on the file "Research/KitchenSinkQuantBookTemplate.ipynb".
The first task is:
Selecting Asset Data
Checkout the QuantConnect docs to learn how to select asset data.
Can you write Python code for it?

spy = qb.AddEquity("SPY")
eur = qb.AddForex("EURUSD")
btc = qb.AddCrypto("BTCUSD")
fxv = qb.AddData[FxcmVolume]("EURUSD_Vol", Resolution.Hour)


I am working on the file "Research/KitchenSinkQuantBookTemplate.ipynb".
The first task is:
Using the Web API
Our script start.py automatically loads an instance of the web API for you to use.**
Look at Lean's Api class for more functions to interact with the cloud
**Note: This will only connect if you have your User ID and Api token in config.json
Can you write Python code for it?

# Show that our api object is connected to the Web Api
print(api.Connected)

# Get our list of projects from the cloud and print their names
projectResponse = api.ListProjects()
for project in projectResponse.Projects:
    print(project.Name)


I am working on the file "Research/NFL_PlayByPlay/Building a Random Forest Classification Model for NFL Decisions.ipynb".
The first task is:
Building a Random Forest Classification Model for NFL Decisions
Game Time Decisions
Its 1st and Goal with less than a minute on the clock in the 4th quarter. Your team is down by 3 points, what do you do? Should you run the ball hoping to break into the endzone? What about passing the ball instead? Or should you go for the field goal just to get the tie. Everyone has an opinion on various NFL scenarios but we will build a model that tells you which decision the pros would make given the same situation.
Data
To build our predictive model we first need some data. The NFL has released play by play information for every game since 2009. We can look at the features of each play to predict what the ultimate play decision was. Given that a play can have many different intended outcomes we will need to build a supervised classification model.
Inputs
From the NFL play-by-play data we can use the following info from each play as inputs to our RF model.
* Quarter
* Time Remaining
* Down
* Yards to 1st Down
* Field Position
* Score
Output
The RF model will be able to return a probability estimation for what the best decision would be for a given set of inputs (NFL situation). The possible play types would be the following:
* Pass
* Run
* Punt
* Field Goal
* QB Kneel
Disclamer!
This model is designed to favor the "average" NFL decision for a given situation. It does not look to see if this decision will result in a positive outcome however since all historical decisions by NFL teams are made under the desire for a positive outcome so the model should reflect that desire as well. Stronger models utilizing more features of the data that indicate play success or failure could be built from the same dataset.
Train/Test Data
For this model we will split our data into train and test data by season. The model will train using data from the 2009-2015 seasons and then will be tested on the 2016 season.
Random Forest Model Tuning
Min_samples_leaf
Lets tune the min_samples_leaf parameter.
We expect the smaller values to take...
Can you write Python code for it?

# Testing "min_samples_leaf"
min_samples_leaf = [1,3,6,9,12,15,18,22,25,50,75,100,125,150,175,200,225,250,275,300]
n_estimators = [30]

min_samples_leaf_scores = []
for n in n_estimators:
    print('-' * 40)
    for l in min_samples_leaf:
        print('--- Testing', '({},{})'.format(n,l))
        start = time.time()
        rfc = nfl_model.build_random_forest_model(
            X_train,
            y_train,
            n_estimators=n,
            max_depth=None,
            min_samples_split=2,
            min_samples_leaf=l,
            max_features='auto',
            bootstrap=True,
            oob_score=True,
            n_jobs=-1,
            random_state=0
        )
        stop = time.time()
        score = rfc.score(X_test, y_test)
        run_time = stop - start
        min_samples_leaf_scores.append([n, l, score, run_time])
        print('     Run Time: ', run_time)
        print('     Score:    ', score)

sns.set_context('talk')
sns.set_style('ticks')

records = [{'n':x[0], 'l':x[1], 'score':x[2]*100, 'time':x[3]} for x in min_samples_leaf_scores]
results_df = pd.DataFrame.from_records(records)
fig,ax = plt.subplots(1,1,figsize=(12,7))
ax.plot(results_df.l, results_df.score)
ax.set_title('Random Forest Parameter Tunning: "Min_sample_leaf"')
ax.set_xlabel('Parameter Value')
ax.set_ylabel('Model Score')
sns.despine()
ax.vlines(x=75,ymin=ax.get_ylim()[0],ymax=ax.get_ylim()[1],colors='g', label='1st - 75')
ax.legend()
plt.show()


I am working on the file "Research/NFL_PlayByPlay/Building a Random Forest Classification Model for NFL Decisions.ipynb".
The first task is:
Given the overall range of model score values to fall within 0.1 the number of estimators is not showing significant change to the model performance. A test was also done at 1000 estimators with a value in line with the above results. We will use a value of 250 n_estimators since the high score at 50 n_estimators could be a result of statistical randomness. The value of 250 strikes a good balance between computation time and statistical rigor.
Can you write Python code for it?

winning_rfc = nfl_model.build_random_forest_model(
                X_train,
                y_train,
                n_estimators=250,
                max_depth=None,
                min_samples_split=2,
                min_samples_leaf=75,
                max_features='auto',
                bootstrap=True,
                oob_score=True,
                n_jobs=-1,
                random_state=0
            )

winning_rfc.score(X_test, y_test)


I am working on the file "Research/NFL_PlayByPlay/Building a Random Forest Classification Model for NFL Decisions.ipynb".
The first task is:
The testing shows the model performs poorly with "min_samples_leaf" values below 20 compared to values above 20.
The highest tested mark occurs at a value of 75 which we will use in the final model.
N_estimators
Next up lets tune the N_estimators parameter.
We expect the smaller values to take less time since we are requiring less trees to be made. The larger we can make this value the better however since more estimators is not going to hurt our model. More models do eventually take too much time/memory to easily compute.
We will build and test the model for the following parameters and then plot each one's score to see it's impact on model performance:
[30,50,75,100,125,150,175,200,250,300]
Can you write Python code for it?

# Testing "n_estimators"
min_samples_leaf = [75]
n_estimators = [30,50,75,100,125,150,175,200,250,300]

n_estimator_scores = []
for n in n_estimators:
    print('-' * 40)
    for l in min_samples_leaf:
        print('--- Testing', '({},{})'.format(n,l))
        start = time.time()
        rfc = nfl_model.build_random_forest_model(
            X_train,
            y_train,
            n_estimators=n,
            max_depth=None,
            min_samples_split=2,
            min_samples_leaf=l,
            max_features='auto',
            bootstrap=True,
            oob_score=True,
            n_jobs=-1,
            random_state=0
        )
        stop = time.time()
        score = rfc.score(X_test, y_test)
        run_time = stop - start
        n_estimator_scores.append([n, l, score, run_time])
        print('     Run Time: ', run_time)
        print('     Score:    ', score)

records = [{'n':x[0], 'l':x[1], 'score':x[2]*100, 'time':x[3]} for x in n_estimator_scores]
results_df = pd.DataFrame.from_records(records)
fig,ax = plt.subplots(1,1,figsize=(12,7))
ax.plot(results_df.n, results_df.score)
ax.set_title('Random Forest Parameter Tunning: "n_estimator"')
ax.set_xlabel('Parameter Value')
ax.set_ylabel('Model Score')
sns.despine()
ax.vlines(x=50,ymin=ax.get_ylim()[0],ymax=ax.get_ylim()[1],colors='green', label = '1st - 50')
ax.vlines(x=250,ymin=ax.get_ylim()[0],ymax=ax.get_ylim()[1],colors='orange', label = '2nd - 250')
ax.legend()
plt.show()


I am working on the file "Research/NFL_PlayByPlay/Building a Random Forest Classification Model for NFL Decisions.ipynb".
The first task is:
Using the model
Model Score - 70.9%
After tuning our model we get a 2016 prediction score of 70.9%. So nearly 3 out of 4 plays were predicted correctly. Based on this it would make good sense to return the probability estimates for each decision when using the model so you can see what the next closest decision was since it is likely that many of these situations have multiple outcomes that different teams tend to use more often than the next. Stronger models using more features can capitalize on this and build a more robust predictor given individual team/player information.
Can you write Python code for it?

qtr = 4
down = 3
ydstogo = 10
TimeUnder = 1
yrdline100 = 40
ScoreDiff = 7

test_case = [[qtr, down, ydstogo, TimeUnder, yrdline100, ScoreDiff]]
classes = winning_rfc.classes_
rfcp = winning_rfc.predict_proba(test_case)[0]*100

rfcp = [str(round(x,2)) for x in rfcp]

print("")
print("Random Forest")
for item in zip(classes,rfcp):
    print(item)

nfl_model.store_model(winning_rfc,'random_forest_classifier', 3)


I am working on the file "ReservoirSampling.ipynb".
The first task is:
在 pypy 下测试，从 $10^8$ 个元素生成 $10^5$ 个元素的样本只需要 1.6 秒，而 Algorithm R 则需要 5 秒。但是，原先的 R 算法对样本大小并不怎么敏感，因为总归要算那么多次，而这个 X 算法所花的时间则会随着样本数增加而有所增加。 上面这个算法来自 Vitter 的一篇著名论文[1]，这篇文章针对 $n$ 比较大的时候进一步改进了计算 $s$ 的过程，利用两个函数对 $F(s)$ 进行夹逼从而避免了对 $s$ 的搜索，由此得到了性能进一步提高的 Z 算法。PostgreSQL 就实现了这个 Z 算法来进行数据抽样，有兴趣的同学可以看看源代码。 前面说的是性能，那如果我们需要不均匀的抽样，有没有办法呢？
加个权吧
前面说的是均匀抽样，要是想加个权怎么办呢？先说加权有什么用呢？比如我们已经统计好了搜索的关键字和词频，那么有了加权就可以直接用这个数据来抽样而无需把关键字重复好多遍了。
我们先来看看这个抽样应该是怎么样的。假设这总共 $n$ 项都摆在你面前了，设第 $i$ 项的权值为 $w_i$，那么我们抽到这一项概率应该是
$$P_i = \frac{w_i}{w_1 + w_2 + \cdots + w_n}$$
我们现在来看看，如果把这 $n$ 项按照第 $n$ 项、第 $n-1$ 项、…… 第 $2$ 项、第 $1$ 项的顺序抽出来，这个概率是多少。因为这 $n$ 项的顺序是可以随便摆的，所以它求出来是有一定普遍意义的。
第一次抽出第 $n$ 项的概率是
$$P_n(1) = \frac{w_n}{w_1 + w_2 + \cdots + w_n}$$
第 $n$ 项已经被抽走了，那第二次抽到第 $n-1$ 项的概率是
$$P_{n-1}(2) = \frac{w_{n-1}}{w_1 + w_2 + \cdots + w_{n-1}}$$
以此类推，按照这个顺序抽出所有元素的概率就是
$$P(S) = \prod_{i=1}^n \frac{w_i}{w_1+w_2+\cdots w_i} $$
这有什么用呢？别着急，先记下这个结果，我们下面来说说算法怎么做。
其实这个算法简单得不能再简单了——还记得第 1 讲里面说的那个给每个数标上一个随机数的方法吗？这次只需要给生成的随机数 $U_i$ 上面加一个方次变成 $U_i^{1/w_i}$ 就好了。记得 $U_i$ 是在 $[0,1]$ 之间的，所以 $w_i$ 越大，这个随机数就放得越大，也就越容易挤进堆里面了。程序如下：
Can you write Python code for it?

def reservoirSampleAES(stream, sample_size):
    result = []
    for index, line in enumerate(stream):
        w = (index + 1)** (1/2)
        key = random.random() ** (1/w)
        if index < sample_size:
            heapq.heappush(result, (key, line))
        elif key > heapq.nsmallest(1, result)[0][0]:
            heapq.heappushpop(result, (key, line))
    return list(zip(*result))[1]


I am working on the file "ReservoirSampling.ipynb".
The first task is:
这个方法也很好，只不过是每次操作要维护这个堆需要 $O(\log k)$ 的时间，稍微慢了那么一点点。但是这个思路还有其他的优势，我们后面再谈。 好了，今天先说到这吧。但是问题还没有完。这两种方法需要产生多少随机数呢？$n -k$ 个。而且每一个元素都要放进来比较。有没有快一点的方法呢？
跳跳跳！—— Algorithm X
前面介绍的方法都很好，但是要计算 $n-k$ 个随机数实在是有点浪费时间…… 有兴趣的同学可以把那个调用函数换成比如 reservoirSample(range(10**8), 10**5)，就知道这东西还是要算上一会儿的了。
再来看看这个过程吧——把前 $k$ 个元素放进去之后，随机决定接下来的元素要不要放进去，但是每次决定时都需要产生一个随机数。要是我们能够确定应该跳过多少个元素，不就可以省掉很多生成随机数的工夫了吗？每一步过程就变成了

确定该跳过多少个元素 $S(k, n)$
跳过 $S(k, n)$ 个元素
从前 $k$ 个元素中随机产生一个要替换的元素，用下一个元素替换

那该跳过的元素数 $S(k,n)$ 要怎么定下来呢？这显然也是一个随机数。我们要计算它的累积分布函数 $\mathbb{P}(S(k,n) \leq s)$，即在第 $n$ 个元素出现时，跳过 $s$ 或 $s$ 以内个元素的概率。这似乎有点不那么直观，但是我们可以反过来计算 $1 - \mathbb{P}(S(k,n) > s)$，要跳过超过 $s$ 个元素，不就是从第 $n + 1$ 个到第 $n+s+1$ 个都不被选中嘛！我们知道第 $n$ 个元素被选中的概率是 $k / n$，所以
$$\begin{aligned} F_S(s) & =\mathbb{P}(S(k,n) \leq s) \ &= 1 - \mathbb{P}(S(k,n) > s) \ &= 1 - \left(1 - \frac{k}{n+1}\right)\left(1 - \frac{k}{n+2}\right)\cdots\left(1 - \frac{k}{n+s+1}\right) \ &= 1 - \frac{(n+1-k)(n+2-k)\cdots(n+s-k+1)}{(n+1)(n+2)\cdots(n+s+1)} \end{aligned} $$
好啦，概率分布知道了，怎么生成这个概率分布呢？这里又要用到一个生成任意概率分布的技巧，就是所谓的反变换法。令 $U$ 是均匀分布在 $[0,1]$ 上的随机变量，如果 $X = F^{-1}(U)$，则
$$F_X(a) = \mathbb{P}(X \leq a) = \mathbb{P}(F^{-1}(U) \leq a) = \mathbb{P}(U \leq F(a)) = F(a)$$
那么在这里，我们只需要生成一个 $[0,1]$ 的随机数 $u$，然后找出让 $F_S(s) \leq u$ 的最小 $s$ 值就好了。这就是所谓的 Algorithm X：
Can you write Python code for it?

import random
import sys
import time

def getS(n, k):
    u = random.random()
    S = 0
    n += 1
    quot = (n - k) / n
    while (quot > u):
        S += 1
        n += 1
        quot *= (n - k) / n
    return S

def reservoirSampleX(stream, sample_size):
    result = []
    s = 0
    for index, line in enumerate(stream):
        if index < sample_size:
            result.append(line)
        else:
            if not s:
                result[int(random.random() * sample_size)] = line
                s = getS(index + 1, sample_size)
            else:
                s -= 1
    return result


I am working on the file "Resources/.ipynb_checkpoints/INTEG 120-checkpoint.ipynb".
The first task is:
Display the Network
Running the next cell
Can you write Python code for it?

url = 'http://localhost:8888/files/network.html'
webbrowser.open(url)


I am working on the file "Resources/.ipynb_checkpoints/INTEG 120-checkpoint.ipynb".
The first task is:
Make Network
Can you write Python code for it?

# This cell creates the network based on
# the variables you provided above.
RC = mk.RecordCollection(inputFile)

if networkType == "CoCitation":
    net = RC.networkCoCitation(nodeType = nodeType, coreOnly=True)
    directed = False
    partition = community.best_partition(net)
elif networkType == "CoAuthor":
    net = RC.networkCoAuthor()
    directed = False
    partition = community.best_partition(net)
elif networkType == "Citation":
    net = RC.networkCitation(nodeType=nodeType, coreOnly=True)
    directed = True

else:
    print("Please ensure networkType has been set to one of the accepted values")

# This code detects centrality measures for your network.
betweenness = nx.betweenness_centrality(net)
# closeness = nx.closeness_centrality(net)   # <- Extra more complicated centrality
# eigenVect = nx.eigenvector_centrality(net) # <--/

for n in net.nodes():
    if not directed:
        betw = round(betweenness[n], 3)
        comm = partition[n]
        net.add_node(n, community=comm, betweenness=betw)
    else:
        betw = round(betweenness[n], 3)
        net.add_node(n, community=comm, betweenness=betw)

# This code writes two .csv files to your computer.
# One is the edgeList and the other is the node Attribute file
mk.writeGraph(net, "myNet")


I am working on the file "Resources/.ipynb_checkpoints/INTEG 120-checkpoint.ipynb".
The first task is:
Multi RPYS
Can you write Python code for it?

years = range(minYear, maxYear+1)
RC = mk.RecordCollection(inputFile)

# ***************************
#  Create the multiRPYS file
# ***************************
dictionary = {'CPY': [],
             "abs-deviation": [],
             "num-cites": [],
             "rank": [],
             "RPY": []}
for i in years:
    try:
        RCyear = RC.yearSplit(i, i)
        if len(RCyear) > 0:
            rpys = RCyear.rpys(minYear=1900, maxYear=maxYear)
            length = len(rpys['year'])
            rpys['CPY'] = [i]*length

            dictionary['CPY'] += rpys['CPY']
            dictionary['abs-deviation'] += rpys['abs-deviation']
            dictionary['num-cites'] += rpys['count']
            dictionary['rank'] += rpys['rank']
            dictionary['RPY'] += rpys['year']
    except:
        pass

df = pandas.DataFrame.from_dict(dictionary)
df.to_csv("multi_rpys.csv")

# ***************************
#  Create the citation file
# ***************************
dictionary = {"author": [],
              "journal": [],
              "num-cites": [],
              "RPY": [],
              "CPY": []}

for i in years:
    try:
        RCyear = RC.yearSplit(i, i)
        if len(RCyear) > 0:
            citations = RCyear.getCitations()
            length = len(citations['year'])
            citations['CPY'] = [i]*length

            dictionary['CPY'] += citations['CPY']
            dictionary['author'] += citations['author']
            dictionary['journal'] += citations['journal']
            dictionary['num-cites'] += citations['num-cites']
            dictionary['RPY'] += citations['year']
    except:
        pass

df = pandas.DataFrame.from_dict(dictionary)

df.to_csv("multi_citation.csv")

%%writefile multiRPYS.html
<!DOCTYPE html>
<head>
    <meta charset="utf-8">
    <title>Title Here</title>
    <link rel="stylesheet" href="http://networkslab.org/mkD3/styles.css">
    <script src="https://d3js.org/d3.v4.js"></script>
    <script src="http://networkslab.org/mkD3/mkd3.js"></script>
</head>
<body>
    <script type =...


I am working on the file "Resources/.ipynb_checkpoints/INTEG 120-checkpoint.ipynb".
The first task is:
Networks
Define Variables
Next, we need to define some variables:
- filepath should be set as the filepath to your isi file.
- networkType should be "CoCitation", "CoAuthor", or "Citation".
- nodeType must be set to one of "full", "original", "author", "journal", or "year".
Can you write Python code for it?

networkType = "CoAuthor"
nodeType = "original"


I am working on the file "Resources/.ipynb_checkpoints/INTEG 120-checkpoint.ipynb".
The first task is:
RPYS Visualization
Can you write Python code for it?

minYear = 1900
maxYear = 2016
rpysType = "StandardBar"


I am working on the file "Resources/.ipynb_checkpoints/INTEG 120-checkpoint.ipynb".
The first task is:
Standard RPYS
Can you write Python code for it?

RC = mk.RecordCollection(inputFile)

rpys = RC.rpys(minYear=1900, maxYear=2016)
df = pandas.DataFrame.from_dict(rpys)
df.to_csv("standard_rpys.csv")

# Creating CitationFile
citations = RC.getCitations()
df = pandas.DataFrame.from_dict(citations)
df.to_csv("standard_citation.csv")

%%writefile standardBar.html
<!DOCTYPE html>
<head>
    <meta charset="utf-8">
    <title>Title Here</title>
    <link rel="stylesheet" href="http://networkslab.org/mkD3/styles.css">
    <script src="https://d3js.org/d3.v4.js"></script>
    <script src="http://networkslab.org/mkD3/mkd3.js"></script>
</head>
<body>
    <script type = "text/javascript">
        mkd3.standardBar("standard_rpys.csv", "standard_citation.csv")
    </script>
</body>

url = 'http://localhost:8888/files/standardBar.html'
webbrowser.open(url)

%%writefile standardLine.html
<!DOCTYPE html>
<head>
    <meta charset="utf-8">
    <title>Title Here</title>
    <link rel="stylesheet" href="http://networkslab.org/mkD3/styles.css">
    <script src="https://d3js.org/d3.v4.js"></script>
    <script src="http://networkslab.org/mkD3/mkd3.js"></script>
</head>
<body>
    <script type = "text/javascript">
        mkd3.standardLine("standard_rpys.csv", "standard_citation.csv")
    </script>
</body>

url = 'http://localhost:8888/files/standardLine.html'
webbrowser.open(url)


I am working on the file "Resources/Jacobs+Python+Tutorial+Draft.ipynb".
The first task is:
Conditionals in Python
Can you write Python code for it?

#Sometimes you want to execute code only in certain circumstances. We saw this on HW1.

#Should be fairly straightforward:
answer = 42

if answer == 42:
    print('This is the answer to the ultimate question')
elif answer < 42:
    print('This is less than the answer to the ultimate question')
else:
    print('This is more than the answer to the ultimate question')
print('This print statement is run no matter what because it is not indented!')

#An if statement is an example of a structure that creates a new block. The block includes all of the code that is
#indented. The indentation (tab character) is imperative. Don't forget it!

#This is normally just good coding style in other languages, but in python it isn't optional

#We can check multiple things at once using boolean operations
rainy = True
day = "Wednesday"

if (rainy == False) and (day != "Tuesday"):
    #&& is boolean and, true only if both are true. False otherwise
    print("The price for golfing is the full $10")
elif (rainy == True) and (day == "Tuesday"):
    print("The price for golfing is reduced to $5!")
elif (rainy == True) or (day == "Tuesday"):
    #|| is boolean inclusive or False only if both are false. True otherwise.
    print("The price for golfing is reduced to $7.50!")

#You can structure these statements more neatly if you "nest" if statements (put an if statement inside an if statement)
#But this is just for edification.


I am working on the file "Resources/Jacobs+Python+Tutorial+Draft.ipynb".
The first task is:
Functions in Python
Can you write Python code for it?

#We can separate off code into functions, that can take input and can give output. They serve as black boxes from the
#perspective of the rest of our code

#use the def keyword, and indent because this creates a new block
def print_me( str ):
   print(str)

   #End with the "return" keyword
   return

#Your functions can return data if you so choose
def my_favorite_song( ):
    ans = "Amsterdam - Imagine Dragons"
    return ans

#call functions by repeating their name, and putting your variable in the parenthesis.
#Your variable need not be named the same thing, but it should be the right type!

text = "I'll take the West train, just by the side of Amsterdam"
print_me(text)
print(my_favorite_song())


I am working on the file "Resources/Jacobs+Python+Tutorial+Draft.ipynb".
The first task is:
Numpy - "The Fundamental Package for Scientific Computing with Python"
Can you write Python code for it?

import numpy as np
#Here, we grab all of the functions and tools from the numpy package and store them in a local variable called np.
#You can call that variable whatever you like, but 'np' is standard.

#numpy has arrays, which function similarly to python lists.
a = np.array([1,2,3])
b = np.array([9,8,7])
#Be careful with syntax. The parentheses and brackets are both required!
print(a)

#Access elements from them just like you would a regular list
print(a[0])

#Element-wise operations are a breeze!
c = a + b
d = a - b
e = a * b
f = a / b

print(c)
print(d)
print(e)
print(f)
#This is different from MATLAB where you add a dot to get element wise operators.

#If you actually do want to do matrix multiplication...
g = np.matmul(np.transpose(a),b)

print(g)

#Now, let's use numpy for something essential for you: Numeric Integration

#Define the function you want to integrate....
def foo(y,t):
    return t
#Note this doesn't use y in the return. That is okay, but we need to include it just to satisfy the function we will use.

#Set your initial or boundary condition
IC = 0

#Give the number of points to evaluate the integration
start_time = 0
end_time = 10
num_times = 101
times = np.linspace(start_time, end_time, num_times)

from scipy.integrate import odeint
integrated_func = odeint(foo,IC,times)

#Can we plot the result? You betcha. Just import a new package
%matplotlib inline
import matplotlib.pyplot as plt
from ipywidgets import interact

plt.plot(times, integrated_func)
#Very similar to MATLAB!


I am working on the file "Resources/Jacobs_Python_Tutorial.ipynb".
The first task is:
Basic Things to do with Variables. Especially Floats.
Can you write Python code for it?

float1 = 5.75
float2 = 2.25
#Addition, subtraction, multiplication, division are as you expect

print(float1 + float2)
print(float1 - float2)
print(float1 * float2)
print(float1 / float2)

#Here's an interesting one that showed up in your first homework. What does this do?
result = 5 % 2

#What are some other math functions you would like to do with floats?


I am working on the file "Resources/Jacobs_Python_Tutorial.ipynb".
The first task is:
Functions in Python
Can you write Python code for it?

#We can separate off code into functions, that can take input and can give output. They serve as black boxes from the
#perspective of the rest of our code

#use the def keyword, and indent because this creates a new block
def print_me( str ):
    print(str)

    #End with the "return" keyword
    return

#What if we want to compute something and return the result? How would you write that return statement?


I am working on the file "Resources/Jacobs_Python_Tutorial.ipynb".
The first task is:
Loops in Python
Can you write Python code for it?

#Repeat code until a conditional statement (or boolean expression, same thing!) ends the loop

#Let's try printing a list
fib = [1, 1, 2, 3, 5, 8]

#Most of your loops can be written in the following way. Before we get to this, we'll go over two less neat ways to write
#the same thing
for e in fib:
    print(e)


I am working on the file "Resources/Jacobs_Python_Tutorial.ipynb".
The first task is:
More Complicated Data Types
Can you write Python code for it?

#Lists
#Defining a list and printing is very simple:
prices = [10, 20, 30, 40, 50]
print(prices)


I am working on the file "Resources/Jacobs_Python_Tutorial.ipynb".
The first task is:
Numpy - "The Fundamental Package for Scientific Computing with Python"
Can you write Python code for it?

import numpy as np
#Here, we grab all of the functions and tools from the numpy package and store them in a local variable called np.
#You can call that variable whatever you like, but 'np' is standard.

#numpy has arrays, which function similarly to python lists.
a = np.array([1,2,3])
b = np.array([9,8,7])
#Be careful with syntax. The parentheses and brackets are both required!
print(a)

#Access elements from them just like you would a regular list
print(a[0])

#Element-wise operations are a breeze!
c = a + b
d = a - b
e = a * b
f = a / b

#What are other things you would like to do with these arrays?

#If we don't get to it, be sure to look at the notes once uploaded for Numeric ODE Integration!


I am working on the file "Resources/Python_Tutorial.ipynb".
The first task is:
Arrays
Can you write Python code for it?

y = np.array([1,2,3,4,5])             #create an array of values
print('y =\t',y)                      #'\t' creates an indent to nicely align answers
print('y[0] =\t',y[0])                           #Python starts counting at 0
print('y[2] =\t',y[2])                           #y[2] gives the third element in y (I don't agree with this syntax but what can ya do?)
print('y*x =\t',y*x)

x = np.array([6,7,8,9])
z = np.concatenate((y,x),axis=None)   #concatenate two arrays
print('[y,x] =\t',z)
print('sum of y elements =\t',np.sum(y))                      #sum the elements of an array
print('sum of z elements =\t',np.sum(z))


I am working on the file "Resources/Python_Tutorial.ipynb".
The first task is:
Integration
This will integrate a function that you provide. There a number of other methods for numerical integration that can be found online.
For our examples we will use:
$I = \int_{0}^{1} ax^2 + b dx$
Can you write Python code for it?

from scipy.integrate import quad

# First define a function that you want to integrate
def integrand(x,a,b):
    return a*x**2 + b

# Set your constants
a = 2
b = 1
I = quad(integrand, 0, 1, args=(a,b))
print(I)
# I has two values, the first value is the estimation of the integration, the second value is the upper bound on the error.
# Notice that the upper bound on the error is extremely small, this is a good estimation.


I am working on the file "Resources/Python_Tutorial.ipynb".
The first task is:
Linear Algebra
Can you write Python code for it?

#Create a matrix
a = np.array([[1,2,3],[4,5,6],[7,8,9]])
print('a =\n',a)

#get eigenvalues and eigvenvectors of a
w,v = linalg.eig(a)
print('eigenvalues =\t',w)
print('eigenvectors =\n',v)

#Matrix multiplication
b = np.array([1,0,0])
print('a*b =\t',a@b.T)     #'@' does matrix multiplication, '.T' transposes a matrix or vector


I am working on the file "Resources/Python_Tutorial.ipynb".
The first task is:
Performing basic math functions
Can you write Python code for it?

x = 2
print('2*x =',2*x)        #multiplcation
print('x^3 =',x**3)       #exponents
print('e^x =',np.exp(x))  #e^x
print('e^x =',np.e**x)    #e^x alternate form
print('Pi =',np.pi)       #Pi


I am working on the file "Resources/Python_Tutorial.ipynb".
The first task is:
Putting it to use: Homework For Fun!
Use 'for' loops to create Taylor expansions of $ln(1+x)$ centered at 0, with an order of 1, 2, 3, and 4. Plot these Taylor expansions along with the original equation on one plot. Label your plots.
As a reminder the formula for a Taylor expansion is:
$f(a) + \sum_{n=1}^{\infty}\frac{f^n(a)}{n!}(x-a)^n$
Since our expansion is centered at zero, $a=0$.
Can you write Python code for it?


I am working on the file "Resources/Python_Tutorial.ipynb".
The first task is:
Something to help with your homework
Assume you have a chemical reaction defined by:
A + 2B -> C
For every mole of A consumed, 2 moles of B are consumed, and 1 mole of C is produced.
If we have the following molar flow rates:
FA0 = 1.5 moles/s = Initial flow of A
FB0 =  = 2.5 moles/s = Initial flow of B
FA = Flow rate of A as a function of reaction advancement
FB = Flow rate of B as a function of reaction advancement
FC = Flow rate of C as a function of reaction advancement
Plot the molar flow rate of each species as a function of reaction advancement.
Can you write Python code for it?

# Set up a vector to store values of advancement
adv = np.arange(0,20,.01)

# Inital Flow Rates
fa0 = 1.5 #moles/s
fb0 = 2.5
fc0 = 0

# Calculate flow rate as a function of advancement
fa = fa0-1*adv
fb = fb0-2*adv
fc = fc0+adv

# Find the maximum value of advancement, value at which one of the reactants hits 0 moles/s
aind = np.where(fa<0)[0][0]
bind = np.where(fb<0)[0][0]
max_adv = min([aind,bind])

# Cut all of the vectors to the maximum value of advancement
adv=adv[:max_adv]
fa=fa[:max_adv]
fb=fb[:max_adv]
fc=fc[:max_adv]

# Plot everything real nice
plt.plot(adv,fa,label='A')
plt.plot(adv,fb,label='B')
plt.plot(adv,fc,label='C')
plt.grid()
plt.xlabel('Reaction Advancement',weight='bold')
plt.ylabel('Molar Flow Rate',weight='bold')
plt.legend()
plt.show()


I am working on the file "Resources/Python_Tutorial.ipynb".
The first task is:
if statements
Can you write Python code for it?

#Order of your if statements matters.
array = np.array([2,4,6,7,11])
for x in array:
    if x<5:
        print('Not a winner')
    elif x<10:
        print(2*x)
    else:
        break


I am working on the file "Resources/Python_Tutorial.ipynb".
The first task is:
while loops
Can you write Python code for it?

#This while loop accomplishes the same thing as the two for loops above
x=0
while x<5:
    print(x*2)
    x+=1


I am working on the file "Restaurant.ipynb".
The first task is:
Now this is a very interesting observation. Record 16 and 85 are identical in predictors(EXCEPT FOR ...) but give very different revenue amounts
The question we face here is, do we want to exclude the outlier or do we want to include it?
Can you write Python code for it?

train_df[ train_df.City == 'İstanbul' ].revenue.plot('box')

train_df_grouped_by_type = train_df.groupby(by='Type')
type_avg = train_df_grouped_by_type.revenue.aggregate(np.average)
type_avg.plot(kind='bar')

trainingdf = pd.read_csv('train.csv')
print trainingdf.columns