I am working on the file "Problema 2/alexandre/Second Project ML.ipynb".
The first task is:
Removing Words / Features
Aqui são removidas as palavras que aparecem menos de 52% como positivo ou como negativo.
Can you write Python code for it?

positiveCount = [0] * len(allWords)
negativeCount = [0] * len(allWords)

sentenceNumber = 0
for sentenceNumber, sentence in enumerate(allSentences) :
    for word in sentence.split(" "):
        wordIndex = allWords.index(word)
        if (Y[sentenceNumber] == 1):
            positiveCount[wordIndex] = positiveCount[wordIndex] + 1
        else:
            negativeCount[wordIndex] = negativeCount[wordIndex] + 1

allCount =  np.array(positiveCount) + np.array(negativeCount)
probPositive = np.divide(positiveCount, allCount)

len(allWords)
allWordsFiltered =  np.array(allWords)

len(allWordsFiltered)
index_to_remove = []
for index, prob in enumerate(probPositive):
    if not((prob <= 0.48) or (prob >= 0.52)):
        index_to_remove.append(index)
        a = 1

allWordsFiltered = np.delete(allWordsFiltered, index_to_remove)


I am working on the file "Problema 2/jeferson/.ipynb_checkpoints/sentiment-analysis-checkpoint.ipynb".
The first task is:
1. Dicionário
Para criar um dicionário, precisamos usar apenas a coluna que contém as sentenças no arquivo e ignorar a coluna da polaridade. Novamente, o nltk facilita esse processo. Obtidas as sentenças, usaremos o nltk para quebrá-las em tokens.
Can you write Python code for it?

import nltk

sentences = big['sentences']
sentences_strings = sentences.str.lower()

sentences_tokens = [] # Has all sentences tokens
for sentence_string in sentences_strings:
	token = nltk.word_tokenize(sentence_string)
	sentences_tokens.extend(token)

dictionary = set()
dictionary.update(sentences_tokens)


I am working on the file "Problema 2/jeferson/.ipynb_checkpoints/sentiment-analysis-checkpoint.ipynb".
The first task is:
1. Tokenization
Para a criação de um dicionário, precisa-se transformar o texto em tokens. Para auxiliar nessa tarefa pode-se utilizar uma biblioteca para processamento de linguagem natural, no exemplo abaixo, usa-se o nltk. O nltk é uma biblioteca Python de código aberto que realiza esta função.
Can you write Python code for it?

import nltk

sentence = 'My test for nltk library!!'
tokens = nltk.word_tokenize(sentence)

print(tokens)


I am working on the file "Problema 2/jeferson/sentiment-analysis.ipynb".
The first task is:
1.  Vetorização
Usa-se a vetorização para tornar possível o trabalho dos classificadores. Na vetorização do texto, cada palavra é tratada como uma característica
Can you write Python code for it?

all_sentences_vetorized = []
for sentence_tokenized in all_sentences_tokenized:
    for token in sentence_tokenized:
        if token in all_tokens_indexed:
            tonken_position = all_tokens_indexed[token]
            sentence_tokenized[tonken_position] += 1
        all_sentences_vetorized.append(sentence_tokenized)

X = numpy.array(all_sentences_vetorized)


I am working on the file "Problema 2/jeferson/sentiment-analysis.ipynb".
The first task is:
1. Dicionário
Para criar um dicionário, precisamos usar apenas a coluna que contém as sentenças no arquivo e ignorar a coluna da polaridade. Novamente, o nltk facilita esse processo. Obtidas as sentenças, usaremos o nltk para quebrá-las em tokens.
Can you write Python code for it?

import nltk

sentences = big['sentences']
all_sentences_strigs = sentences.str.lower()

all_sentences_tokenized = [] #Has
for sentence_string in all_sentences_strigs:
    sentence_tokenized = nltk.word_tokenize(sentence_string)
    all_sentences_tokenized.append(sentence_tokenized)

all_tokens = [] # Has all sentences tokens
for sentence_tokenized in all_sentences_tokenized:
    all_tokens.extend(sentence_tokenized)

dictionary = set()
dictionary.update(all_tokens)


I am working on the file "Problema 2/jeferson/sentiment-analysis.ipynb".
The first task is:
1. Tokenization
Para a criação de um dicionário, precisa-se transformar o texto em tokens. Para auxiliar nessa tarefa pode-se utilizar uma biblioteca para processamento de linguagem natural, no exemplo abaixo, usa-se o nltk. O nltk é uma biblioteca Python de código aberto que realiza esta função.
Can you write Python code for it?


I am working on the file "Problema 4/stars.ipynb".
The first task is:
Aplicando K-means e coletando métricas:
seguin do exemplo disponível em:
http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_digits.html#sphx-glr-auto-examples-cluster-plot-kmeans-digits-py
Can you write Python code for it?

import numpy

number_samples, number_features = data.shape
number_labels = len(numpy.unique(labels))

from time import time
from sklearn import metrics
from sklearn.cluster import KMeans
from sklearn.decomposition import PCA

sample_size = 1900 # Pesquisar melhor valor para esse parâmetro usado na métrica silhouette

def bench_k_means(estimator, name, data):
    initial_time = time()
    estimator.fit(data)
    execution_time = time() - initial_time

    # metrics
    inertia = estimator.inertia_
    homogeneity_score = metrics.homogeneity_score(labels, estimator.labels_)
    completeness_score = metrics.completeness_score(labels, estimator.labels_)
    v_measure_score = metrics.v_measure_score(labels, estimator.labels_)
    adjusted_rand_score = metrics.adjusted_rand_score(labels, estimator.labels_)
    adjusted_mutual_info_score = metrics.adjusted_mutual_info_score(labels,  estimator.labels_)
    silhouette_score = metrics.silhouette_score(data, estimator.labels_, metric='euclidean', sample_size=sample_size)

    #show metrics
    print('%-9s\t%.2fs\t%i\t%.3f\t%.3f\t%.3f\t%.3f\t%.3f\t%.3f'
          % (name, execution_time, inertia, homogeneity_score,completeness_score, v_measure_score,
             adjusted_rand_score, adjusted_mutual_info_score, silhouette_score))

print(90 * '_')
print('init\t\ttime\tinertia\t\thomo\tcompl\tv-meas\tARI\tAMI\tsilhouette')

bench_k_means(KMeans(init='k-means++', n_clusters=number_labels, n_init=10),
              name="k-means++", data=data)

bench_k_means(KMeans(init='random', n_clusters=number_labels, n_init=10),
              name="random", data=data)

# in this case the seeding of the centers is deterministic, hence we run the
# kmeans algorithm only once with n_init=1
pca = PCA(n_components=number_labels).fit(data)
bench_k_means(KMeans(init=pca.components_, n_clusters=number_labels, n_init=1),
              name="PCA-based", data=data)
print(90 * '_')


I am working on the file "Problema 4/stars.ipynb".
The first task is:
Features:

mean_of_the_integrated_profile
standard_deviation_of_the_integrated_profile
excess_kurtosis_of_the_integrated_profile
skewness_of_the_integrated_profile
mean_of_the_DM-SNR_curve
standard_deviation_of_the_DM-SNR_curve
excess_kurtosis_of_the_DM-SNR_curve
skewness_of_the_DM-SNR_curve
class

Labels

Pulsar: 1
Não Pulsar: 0

Missing Data?
A base de dados não possui missig data porque seus campos são preenchidos com numpy.float64 e os rótulos com numpy.int64
Can you write Python code for it?

import numpy

def verify_missing_data(data, features):
    missing_data = []

    for feature in features:
        count = 0
        for x in range(0, len(data)):
            if type(data[feature][x]) is numpy.float64 or type(data[feature][x]) is numpy.int64:
                count = count + 1
        missing_data.append(count)
    print(missing_data)

verify_missing_data(data, features)


I am working on the file "Problemas - Cap1.ipynb".
The first task is:
1.2 Determine a corrente que flui por um elemento se o fluxo de carga for dado por:
(a) q(t) = (3t + 8) mC
(b) q(t) = (8t^2 + 4t - 2) C
(c) q(t) = (3e^-t - 5e^-2t) nC
(d) q(t) = 10sen(120pit) pC
(e) q(t) = 20e^-4tcos(50t) uC
Can you write Python code for it?

print("1.2")
import numpy as np
from sympy import *

t = symbols('t')
q = 3*t + 8
i = diff(q,t)
print("Resposta (a):", i, "mA")
q = 8*t**2 + 4*t - 2
i = diff(q,t)
print("Resposta (b):", i, "A")
q = 3*exp(-t) - 5*exp(-2*t)
i = diff(q,t)
print("Resposta (c):", i, "nA")
q = 10*sin(120*np.pi*t)
i = diff(q,t)
print("Resposta (d):", i, "pA")
q = 20*exp(-4*t)*cos(50*t)
i = diff(q,t)
print("Resposta (e):", i, "uA")


I am working on the file "Problemas - Cap1.ipynb".
The first task is:
1.3 Determine a carga q(t) que flui por um dispositivo se a corrente for:
(a) i(t) = 3 A, q(0) = 1 C
(b) i(t) = (2t + 5) mA, q(0) = 0
(c) i(t) = 20cos(10t + pi/6) uA, q(0) = uC
(d) i(t) = 10e^(-30t)sen(40t) A, q(0) = 0
Can you write Python code for it?

print("1.3")
i = 3
q = integrate(i,t)
q0 = 1
print("Resposta (a):", q + q0, "C")
i = 2*t + 5
q = integrate(i,t)
q0 = 0
print("Resposta (b):", q + q0, "mC")
i = 20*cos(10*t + np.pi/6)
q = integrate(i,t)
q0 = 2
print("Resposta (c):", q + q0, "uC")
i = 10*exp(-30*t)*sin(40*t)
q = integrate(i,t)
q0 = 0
print("Resposta (d):", q + q0, "C")


I am working on the file "Problemas - Cap1.ipynb".
The first task is:
1.6 A carga que entra em determinado elemento é mostrada na Figura 1.23. Determine a corrente em:
(a) t = 1ms
(b) t = 6ms
(c) t = 10ms
Can you write Python code for it?

print("1.6")
from sympy import *
t = symbols('t')
qa = (30/2)*t #intervalo de 0ms a 2ms
i = diff(qa,t)
print("Resposta (a):", i.subs(t,1), "mA")
qb = 30 #intervalo de 2ms a 8ms
i = diff(qb,t)
print("Resposta (b):", i.subs(t,6), "mA")
qc = (-30/4)*t #intervalo de 8ms a 12ms
i = diff(qc,t)
print("Resposta (c):", i.subs(t,10), "mA")


I am working on the file "Problemas - Cap1.ipynb".
The first task is:
1.8 A corrente que flui por um ponto em um dispositivo é mostrada na Figura 1.25. Calcule a carga total através do ponto.
Can you write Python code for it?

print("1.8")
from sympy import *

t = symbols('t')
i1 = 10*t
q1 = integrate(i1,(t,0,1)) #carga do intervalo 0ms a 1ms
i2 = 10
q2 = integrate(i2,(t,1,2)) #carga do intervalo 1ms a 2msb
print("Carga total:", q1+q2, "mC")


I am working on the file "Problemas - Cap1.ipynb".
The first task is:
1.9 A corrente através de um elemento é ilustrada na Figura 1.26. Determine a carga total que passa pelo elemento em:
(a) t = 1s
(b) t = 3s
(c) t = 5s
Can you write Python code for it?

print("1.9")
from sympy import *

t = symbols('t')
i1 = 10
q1 = integrate(i1,(t,0,1))
print("Resposta (a):", q1, "C")
i2 = 10 - 5*t
q2 = integrate(i2,(t,0,1))
i3 = 5
q3 = integrate(i3,(t,0,1))
print("Resposta (b):", q1+q2+q3, "C")
q3 = integrate(i3,(t,0,2))
i4 = 5 - 5*t
q4 = integrate(i4,(t,0,1))
print("Resposta (c):", q1+q2+q3+q4, "C")


I am working on the file "ProbsVVEE/Levitador Magnetico/Problema Levitador Magnético-Python.ipynb".
The first task is:
Control Integral Para Seguimiento Robusto
Para agregar el contro integral podemos usar cualquier observador de estados y aumentar la planta para que este prensente el estado del integrador y darle una posici'on al polo ese.
Can you write Python code for it?

Aaug=np.concatenate((np.concatenate((np.zeros((1,1)),C),axis=1),
                     np.concatenate((np.zeros((3,1)),A),axis=1)),axis=0)
Baug=np.concatenate((np.zeros((1,1)),B),axis=0)
Caug=np.concatenate((np.zeros((1,1)),C),axis=1)
pol4=np.concatenate((np.array([-5.]), pol3),axis=0)
K4=ctrl.acker(Aaug,Baug,pol4)
print K4


I am working on the file "ProbsVVEE/Levitador Magnetico/Problema Levitador Magnético-Python.ipynb".
The first task is:
Definimos los polos del observador el la ubicación mucho más rápido.
Can you write Python code for it?


I am working on the file "ProbsVVEE/Levitador Magnetico/Problema Levitador Magnético-Python.ipynb".
The first task is:
Estas matrices las que usaremos para calcular el estimador. La planta no se tocará para nada ya que no es parte del diseño.
Can you write Python code for it?

Abb=An[1:3,1:3];
Aba=An[1:3,0];
Aab=An[0,1:3];
Aaa=An[0,0];

Ba=Bn[0];
Bb=Bn[1:3];

polRed1=polObs3[1:3]

Lred1=ctrl.acker(Abb.T,Aab.T,polRed1).T;

aux1=np.concatenate((np.concatenate((An,Bn),axis=1),np.concatenate((Cn,Dn),axis=1)),axis=0);
aux2=np.concatenate((np.zeros((3,1)),np.matrix([1])),axis=0)
Nn=np.linalg.inv(aux1)*aux2;
Nxn=Nn[0:3];
Nun=Nn[3];


I am working on the file "ProbsVVEE/Levitador Magnetico/Problema Levitador Magnético-Python.ipynb".
The first task is:
Notar que las ganacias son las mismas que K3, salvo que la tercera ganacia aparece ahora en primer lugar cambiada de signoy dividida por 40 que conicide con el escaleo y reodenamiento de eatados que hicimos para poder usar la teor'ia y hacer el estimador reducido.
Implementación del controlador obtenido por Ecuanciones de estado
Definimos el controlador en forma de ecuaciones de estado
Can you write Python code for it?

aux3=np.concatenate((np.concatenate((A,B),axis=1),np.concatenate((C,D),axis=1)),axis=0);
N= np.linalg.inv(aux3)*aux2;
Nx=N[0:3]
Nu=N[3]
Nvib=Nu+K3*Nx

Acomp=A-B*K3-L3*C;
Bcomp=np.concatenate((B*Nvib,L3),axis=1);
Ccomp=-K3;
Dcomp=np.concatenate((Nvib,np.zeros((1,1))),axis=1);
comp=ctrl.ss(Acomp,Bcomp,Ccomp,Dcomp)
print Nx
print Nu


I am working on the file "ProbsVVEE/Levitador Magnetico/Problema Levitador Magnético-Python.ipynb".
The first task is:
Podemos ver que el sismtea es controlable y observable. Es era de esperar ya que no modelamos ninguna cancelación de polos y ceros en la transferencia a partir de la cual obtuvimos el esapcio de estado.
Can you write Python code for it?

pol1=[-40.,-10-20j,-10+20j]
K1=ctrl.acker(A,B,pol1)
print "Ubicando los polos en", np.linalg.eigvals(A-B*K1),"se tiene una ganancia ", K1
pol2=[-20.,-10-20j,-10+20j]
K2=ctrl.acker(A,B,pol2)
print "Ubicando los polos en", np.linalg.eigvals(A-B*K2),"se tiene una ganancia ", K2


I am working on the file "ProbsVVEE/Levitador Magnetico/Problema Levitador Magnético-Python.ipynb".
The first task is:
Recalculo la ganancia del controlador para el obwervador reducido para que la salida tenga el valor de ganancia requerido (escaleado).
Can you write Python code for it?


I am working on the file "ProbsVVEE/Levitador Magnetico/Problema Levitador Magnético-Python.ipynb".
The first task is:
Vemos que las ganancias son más grandes cuando tratamos de alejar mucho el tercer polo. La ventaja que se tiene es que haciendo esto el sistema se comporta de forma más parecida a un segundo orden puro. El precio que se paga es un mayor esfuerzo de control.
Si hacemos una optimización usando el lugar de las raíces simétrico se puede obtener una ubicación de polo que cumple de forma aproximada el requerimiento de control y bajan bastante el esfuerzo de control.
Can you write Python code for it?

Gtf=ctrl.tf(sys);
Gtfms=ctrl.tf(-40,[-1,20,100,-2000])
print Gtf, Gtfms
rlist,klist=ctrl.rlocus(Gtf*Gtfms,np.logspace(-3,5,1000),Plot=True,PrintGain=False)

rlist,klist=ctrl.rlocus(Gtf*Gtfms,klist=[38700],Plot=False)
pol3 = rlist.reshape(6)[0:3]
print pol3.size
print pol3
K3=ctrl.acker(A,B,pol3)
print np.linalg.eigvals(A-B*K3)


I am working on the file "ProbsVVEE/Levitador Magnetico/Problema Levitador Magnético-Python.ipynb".
The first task is:
Verificamos controlabilidad y observabilidad.
Can you write Python code for it?

MatCtr=np.concatenate((B,A*B,np.linalg.matrix_power(A,2)*B),axis=1)
MatObs=np.concatenate((C,C*A,C*np.linalg.matrix_power(A,2)),axis=0)
print MatCtr
print np.linalg.matrix_rank(MatCtr)
print MatObs
print np.linalg.matrix_rank(MatObs)


I am working on the file "ProbsVVEE/Python Control Notebook/.ipynb_checkpoints/rlocus_test-checkpoint.ipynb".
The first task is:
Here the plot show problems. This is beacuase the axis limits are not similar.
I can resolve his problem redefinig the x limits axis.
Can you write Python code for it?

r5,k5=ctrl.rlocus(sys5, grid = True, xlim=[-5,0.1])

sys6 = ctrl.tf([1],np.poly(np.hstack((np.roots([1,3,10]),[0]))))
r6,k6=ctrl.rlocus(sys6)
plt.show()
r6,k6=ctrl.rlocus(sys6, grid = True)

sys7 = ctrl.tf(np.poly(np.roots([1,2,8])),np.poly(np.hstack((np.roots([1,2,10]),[0]))))
r7,k7=ctrl.rlocus(sys7)
plt.show()
r7,k7=ctrl.rlocus(sys7, grid = True)

sys8 = ctrl.tf([1,0,1],[1,0,4,0])
r8,k8=ctrl.rlocus(sys8)
plt.show()
r8,k8=ctrl.rlocus(sys8, grid = True)

sys9 = ctrl.tf(np.poly(np.roots([1,2,12])),np.poly(np.hstack((np.roots([1,2,10]),[0]))))
r9,k9=ctrl.rlocus(sys9)
plt.show()
r9,k9=ctrl.rlocus(sys9, grid = True)

sys10 = ctrl.tf([1,0,4],[1,0,1,0])
r10,k10=ctrl.rlocus(sys10)
plt.show()
r10,k10=ctrl.rlocus(sys10, grid = True)


I am working on the file "ProbsVVEE/Python Control Notebook/.ipynb_checkpoints/rlocus_test-checkpoint.ipynb".
The first task is:
However, when I plot the grid the figure looks not so good. The problem are the axis limits. If I redefine the y-axis limits the problem is solved.
Can you write Python code for it?

r, k = ctrl.rlocus(sys1, grid=True, ylim=[-10, 10])


I am working on the file "ProbsVVEE/Python Control Notebook/rlocus_test.ipynb".
The first task is:
However, when I plot the grid the figure looks not so good. The problem are the axis limits. If I redefine the y-axis limits the problem is solved.
Can you write Python code for it?


I am working on the file "ProbsVVEE/Python Control Notebook/rlocus_test.ipynb".
The first task is:
Simple example that is not OK
Can you write Python code for it?

sys1 = ctrl.tf([1, 1], [1, 10, 1])
print(sys1)

r, k = ctrl.rlocus(sys1)
plt.show()
r, k = ctrl.rlocus(sys1, grid=True)


I am working on the file "ProbsVVEE/Python Control Notebook/rlocus_test.ipynb".
The first task is:
Simple examples that can cause problems
Can you write Python code for it?

sys11 = ctrl.tf([1, 0, -4], [1, 0, -1])
r11,k11=ctrl.rlocus(sys11)
plt.show()
r11,k11=ctrl.rlocus(sys11, grid = True)

sys12 = ctrl.tf([1, 1], [1, 0])
r12,k12=ctrl.rlocus(sys12)
plt.show()
r12,k12=ctrl.rlocus(sys12, grid = True)


I am working on the file "ProbsVVEE/SegwayMurray/Segway-Python.ipynb".
The first task is:
Create a system that gives output and control action
Can you write Python code for it?

sys2 = ctrl.ss(A-B*K2, B*kr2, np.concatenate((C,-K2),axis=0),
          np.concatenate((np.matrix([0]), kr2), axis=0))

y2, t2 = ctrl.step(sys2,np.linspace(0.,35,1000.))
fig2=create_figure(y2,t2,title="Sistema Lento")


I am working on the file "ProbsVVEE/SegwayMurray/Segway-Python.ipynb".
The first task is:
Diseno con objetivos mas modestos
Can you write Python code for it?

zeta0 = zeta0
omega0l = omega0/3
zeta1 = zeta1
omega1l = omega1/2
Tmax = 40
lam3 = -zeta1*omega1l + omega1l*np.sqrt(1-np.power(zeta1,2))*1j
lam4 = -zeta1*omega1l - omega1l*np.sqrt(1-np.power(zeta1,2))*1j
evdes2 = [-zeta0*omega0l + omega0l*1j, -zeta0*omega0l - omega0l*1j, lam3, lam4]


I am working on the file "ProbsVVEE/SegwayMurray/Segway-Python.ipynb".
The first task is:
Genero funciones que me seran de ayuda para obtener $G(-s)$.
Can you write Python code for it?

def pol_minus_s(pol):
        "Hago los polinomios"
        l=len(pol)
        npol=np.zeros((l))
        for i in xrange(1,l+1):
            if ((l-i)%2==0):
                npol[l-i]=pol[l-i]
            elif ((l-i)%2==1):
                npol[l-i]=-pol[l-i]
        return [npol]

def G_minus_s(G):
    "Función para generar G(-s), y poder hacer el SRL"
    num=pol_minus_s(G.num[0][0])
    den=pol_minus_s(G.den[0][0])
    Gms=ctrl.tf(num[0],den[0])
    return [Gms]


I am working on the file "ProbsVVEE/SegwayMurray/Segway-Python.ipynb".
The first task is:
Genero funciones que usare despues. Primero es la de las figuras
Can you write Python code for it?

def create_figure(y1,t1,title=""):
    "Función para generar las figuras de este trabajo.\n"

    fig1=plt.figure()
    axes1 = fig1.add_axes([0.1, 0.1, 0.8, 0.8])
    axes1.plot(t1,y1[:,0],'b-')
    axes1.set_ylabel('Position (m)',color='b')
    axes1.set_xlabel('Time(s)')
    axes1.grid(which='both')
    axes1.set_title(title)
    for tl in axes1.get_yticklabels():
        tl.set_color('b')

    axes1i=axes1.twinx()
    axes1i.plot(t1,y1[:,1],'r-')
    axes1i.set_ylabel('Input Force (N)',color='r')
    axes1i.grid(which='both')

    for tl in axes1i.get_yticklabels():
        tl.set_color('r')
    return [fig1]


I am working on the file "ProbsVVEE/SegwayMurray/Segway-Python.ipynb".
The first task is:
Plot the reponse for the system
Can you write Python code for it?

y1, t1 = ctrl.step(sys1,np.linspace(0.,35,1000.))
fig1=create_figure(y1,t1,"Sistema rapido")


I am working on the file "ProbsVVEE/SegwayMurray/Segway-Python.ipynb".
The first task is:
Primer diseno con objetivos de control agresivos.
Can you write Python code for it?

zeta0 = 0.5
omega0 = np.sqrt(m*g*l/(J+m*np.power(l,2)))
zeta1 = 0.7
omega1 = 0.5
lam3 = -zeta1*omega1 + omega1*np.sqrt(1-np.power(zeta1,2))*1j;
lam4 = -zeta1*omega1 - omega1*np.sqrt(1-np.power(zeta1,2))*1j;
evdes1 = [-1. + 2j, -1. - 2j, lam3, lam4]


I am working on the file "ProbsVVEE/SegwayMurray/Segway-Python.ipynb".
The first task is:
SRL-LQR design
Can you write Python code for it?

fig4=plt.figure()
axes4 = fig4.add_axes([0.1, 0.1, 0.8, 0.8])
axes4.plot(t1,y1[:,0],'b--',label='Sistema rapido')
axes4.plot(t2,y2[:,0],'b:',label='Sistema lento')
axes4.plot(t3,y3[:,0],'b-',label='Sistema optimo')
plt.legend(loc=4)

axes4.set_ylabel('Position (m)',color='b')
axes4.set_xlabel('Time(s)')
axes4.grid(which='both')
axes4.set_title("Comparacion")
for tl in axes4.get_yticklabels():
    tl.set_color('b')

axes4i=axes4.twinx()
axes4i.plot(t1,y1[:,1],'r--',label='Sistema rapido')
axes4i.plot(t2,y2[:,1],'r:',label='Sistema lento')
axes4i.plot(t3,y3[:,1],'r-',label='Sistema optimo')
axes4i.set_ylabel('Input Force (N)',color='r',)
axes4i.grid(which='both')
plt.legend(loc=5)

for tl in axes4i.get_yticklabels():
    tl.set_color('r')


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Counter object has an update method that allows multiple lists to be counted.
Can you write Python code for it?

text1 = 'This is a text with some words in it'
text2 = 'This is another text with more words that the other one has in it'
tokens1 = text1.lower().split()
tokens2 = text2.lower().split()
print('text1:', tokens1)
print('text2:', tokens2)


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
A simple example of looping of some texts and generating a frequency list using Counter
Can you write Python code for it?

texts = [
    'This is the first text and it has words',
    'This is the second text and it has some more words',
    'Finally this one has the most words of all three examples words and words and words'
]

freq2 = Counter()
for text in texts:
    # turn the text into lower case and split on whitespace
    tokens = text.lower().split()
    freq2.update(tokens)

# show the top 10 most frequent words
print(freq2.most_common(10))


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
AH!  - actually doesn't look like a great candidate for splitting a document in to body and post body sections.
A second look at some sample documents suggests we might be able to use LOAD-DATE instead.
Can you write Python code for it?

for idx, doc in enumerate(docs):
    print('Document', idx, 'has LOAD-DATE?', doc.count('LOAD-DATE:') > 0)


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Back to the LexisNexus task

We are going to use a list of dictionaries approach to store the prebody, body, postbody components for each document
Can you write Python code for it?

doc_dict = {'prebody': pre_body, 'body': body, 'postbody': post_body}

doc_dict['postbody']

rows = []
for idx, doc in enumerate(docs):

    try:
        start_pos = doc.index('LENGTH')
        start_pos = doc.index('\n', start_pos)
        end_pos = doc.index('LOAD-DATE:')
    except:
        print('ERROR with doc', idx)
        continue

    doc_dict = {
        'prebody': doc[:start_pos],
        'body': doc[start_pos: end_pos],
        'postbody': doc[end_pos:]
    }

    rows.append(doc_dict)

print(docs[13])


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Back to the LexisNexus task

We can use the enumerate and for loop idiom to check how good our proposed strings are for splitting the document up into components.
For example, it looks like the string ( END ) could be a good marker of the end of the body. So we can test which documents contain it using the count function and testing whether we get at least one instance:
Can you write Python code for it?

for idx, doc in enumerate(docs):
    print('Document', idx, 'has ( END )?', doc.count('( END )') > 0)


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
But often you want to have both each item in the list and its index without having to do list[idx] to get the item. The enumerate function helps in such cases.
enumerate(list) returns a list of tuples, where each item in the list consists of a pair where the first item is the index and second the item itself.
Can you write Python code for it?

list(enumerate(slist))

result = list(enumerate(slist))
result[0]

for idx, item in enumerate(slist):
    print(idx, item)


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Counting words in the LexisNexus documents

For a single LexisNexus document loaded in from the CSV file (the list of dictionaries), we select the document by index and then the body component:
Can you write Python code for it?

freq_list = Counter(docs2[0]['body'].lower().split())

freq_list.most_common()


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Downloading the text file directly from github using the requests module
Can you write Python code for it?


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Examine the first 20,000 characters to find string patterns that mark divisions between documents
Can you write Python code for it?


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Finally lets make the formatting a bit more pretty by looping over the frequency list and producing a tab separated table:
Can you write Python code for it?

for item in freq2.most_common(7):
    print("{}\t\t{}".format(item[0],item[1]))


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Find the frequency of the word vaping
Can you write Python code for it?


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Python excurcus:  Using enumerate to loop over lists

When you have a list of items and what to process each in turn then using a for loop is a common approach, e.g.
Can you write Python code for it?

alist = [1,2,3,4,5]
slist = ['a','b','c','d']

for item in alist:
    print(item)

for item in slist:
    print('The current item is:',item)


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Python excursus: The Zen of Python

A little bit of poetry from the creators of Python explaining the design and suggestions for truly Pythonic coding!
Can you write Python code for it?


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Simple example of using Counter
Counter works by passing it a list of items, e.g.:
Can you write Python code for it?


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
The get function takes a URL and returns the content at that address.
Can you write Python code for it?

resp = requests.get('https://raw.githubusercontent.com/mbod/intro_python_for_comm/master/data/LexisNexusVapingExample.txt')

text2=resp.text    # assign content of response to a variable

text2[:300]   # show first 300 characters

print(text2[:300])   # use print to see formatting (spacing and newlines etc.)


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
The string of 1000 DOCUMENTS looks like a good candidate for splitting the the text file into the individual documents

Split the text string into n chunks using of 1000 DOCUMENTS:
Can you write Python code for it?

chunks = text.split('of 1000 DOCUMENTS')

len(chunks)    # see how many chunks this produces

docs = chunks[1:]


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Then update it again with the words from text2
Can you write Python code for it?


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
Then we can get the three parts of the document we want
Can you write Python code for it?

pre_body=doc[:start_pos]

body = doc[start_pos:end_pos]

post_body = doc[end_pos:]


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
We can use list interpolation to get a count of the number of documents that contain a feature
Can you write Python code for it?

has_end = sum([doc.count('( END )')>0 for doc in docs])
print(has_end, 'documents with ( END ) out of ', len(docs), 'docs')

has_load_date = sum([doc.count('LOAD-DATE:')>0 for doc in docs])
print(has_load_date, 'documents with LOAD-DATE out of ', len(docs), 'docs')


I am working on the file "Processing a LexisNexus text export into CSV.ipynb".
The first task is:
and it returns a dictionary with a count for the number of times each item occurs:
Can you write Python code for it?


I am working on the file "Proctor_NAO_bandwidth.ipynb".
The first task is:
Calculate correlation and p-values with considering autocorrelation, and the autocorrelations (coef)
Can you write Python code for it?

corr_ratio,pval_ratio,coef = owncorr(bandw_fil,ratio_fil,115) #correlation between smoothed bandwidth and ratio
corr_nao,pval_nao,coef_nao = owncorr(bandwl_fil,wnao_fil,126) #correlation between smoothed bandwidth and winter NAO
corr_n,pval_n,coef_n = owncorr(bandwn,ration,115) #correlation between normalized bandwidth and ratio
corr_naon,pval_naon,coef_naon = owncorr(bandwln,wnao,126) #correlation between normalized bandwidtha and winter NAO


I am working on the file "Proctor_NAO_bandwidth.ipynb".
The first task is:
Define functions for filtering, moving averages, and normalizing data
Can you write Python code for it?

def butter_lowpass(cutoff, fs, order=3):
    nyq = 0.5 * fs
    normal_cutoff = cutoff / nyq
    b, a = butter(order, normal_cutoff, btype='low', analog=False)
    return b, a

def filter(x, cutoff, axis, fs=1.0, order=3):
    b, a = butter_lowpass(cutoff, fs, order=order)
    y = filtfilt(b, a, x, axis=axis)
    return y

def movingaverage(interval, window_size):
    window = np.ones(int(window_size))/float(window_size)
    return np.convolve(interval, window, 'valid')

def owncorr(x,y,n):
    x_ano=np.ma.anomalies(x)
    x_sd=np.sum(x_ano**2,axis=0)
    y_ano=np.ma.anomalies(y)
    y_sd=np.sum(y_ano**2,axis=0)
    nomi = np.dot(x_ano,y_ano)
    corr = nomi/np.sqrt(np.dot(x_sd[None],y_sd[None]))

    # When using AR_est_YW, we should substract mean from
    # time series first
    x_coef, x_sigma = alg.AR_est_YW (x_ano, 1)
    y_coef, y_sigma = alg.AR_est_YW (y_ano, 1)

    if x_coef > 1:
        eps = np.spacing(1.0)
        x_coef = 1.0 - eps**(1/4)
    elif x_coef < 0:
        x_coef = 0.0

    if y_coef > 1:
        eps = np.spacing(1.0)
        y_coef = 1.0 - eps**(1/4)
    elif y_coef < 0:
        y_coef = 0.0

    neff = n*(1-x_coef*y_coef)/(1+x_coef*y_coef)

    if neff <3:
        neff = 3

    coef = []
    coef.append(x_coef)
    coef.append(y_coef)
    tval = corr/np.sqrt(1-corr**2)*np.sqrt(neff-2)
    pval = t.sf(abs(tval),neff-2)*2

    return corr,pval,coef

def gaussianize(X):
    n = X.shape[0]
    #p = X.shape[1]

    Xn = np.empty((n,))
    Xn[:] = np.NAN
    nz = np.logical_not(np.isnan(X))

    index = np.argsort(X[nz])
    rank = np.argsort(index)

    CDF = 1.*(rank+1)/(1.*n) -1./(2*n)
    Xn[nz] = np.sqrt(2)*special.erfinv(2*CDF -1)
    return Xn


I am working on the file "Proctor_NAO_bandwidth.ipynb".
The first task is:
Smoothing data (11-year running average)
Can you write Python code for it?

bandw_fil = movingaverage(bandw, 11)
bandwn_fil = movingaverage(bandwn, 11)
bandwl_fil = movingaverage(bandwl, 11)
rain_fil = movingaverage(rain, 11)
rainn_fil = movingaverage(rainn, 11)
ratio_fil = movingaverage(ratio, 11)
wnao_fil = movingaverage(wnao, 11)


I am working on the file "Production fMRI example/parse_pdxn_fMRI_logs.ipynb".
The first task is:
Select only rows with stimulus words (so, where WORD != NaN)
Can you write Python code for it?

validTrials = df[pd.notnull(df['WORD'])]
validTrials[0:5]


I am working on the file "Production fMRI example/parse_pdxn_fMRI_logs.ipynb".
The first task is:
Write out additional FSL log files that are response-conditioned
This needs to wait until I know what the response keys mean

Red = don't remember
Blue = know
Yellow = recollect
Can you write Python code for it?

for run in fMRIruns:
    for key in Conditions:
        if (run == 'STUDY0' or run == 'STUDY1') and key == 'FOIL':
            pass
        else:
            dfTmp = df[df['CONDITION'].isin([key]) & df['PHASE'].isin([run])][['WORDON','wordDuration']]
            dfTmp['Col3'] = 1
            dfTmp.to_csv(inFile.split('.')[0] + '_fMRIlog_' + run + '_' + Conditions[key] + '.txt', sep='\t', header=False, index=False)


I am working on the file "Profiling_in_a_Jupyter_notebook.ipynb".
The first task is:
After installing line_profiler and enabling the IPython magic, we can use it
Can you write Python code for it?

%lprun -T /tmp/test_lprun.txt -f recip_square -f approx_pi approx_pi()
%cat /tmp/test_lprun.txt


I am working on the file "Profiling_in_a_Jupyter_notebook.ipynb".
The first task is:
And with just a simple change, by typing the intermediate function:
Can you write Python code for it?

%%cython -3
# cython: profile=True

cdef double recip_square3(int i):
    return 1./(i**2)

def approx_pi3(int n=10000000):
    cdef double val = 0.
    cdef int k
    for k in range(1, n + 1):
        val += recip_square3(k)
    return (6 * val)**.5

%timeit approx_pi3()

%prun approx_pi3()


I am working on the file "Profiling_in_a_Jupyter_notebook.ipynb".
The first task is:
Cython version
Can you write Python code for it?

%load_ext cython

%%cython -3
# cython: profile=True

def recip_square2(int i):
    return 1./i**2

def approx_pi2(int n=10000000):
    cdef double val = 0.
    cdef int k
    for k in range(1, n + 1):
        val += recip_square2(k)
    return (6 * val)**.5

%timeit approx_pi()
%timeit approx_pi2()

%prun approx_pi2()


I am working on the file "Profiling_in_a_Jupyter_notebook.ipynb".
The first task is:
First profiling
Can you write Python code for it?

%time approx_pi()

%timeit approx_pi()

# Thanks to https://nbviewer.jupyter.org/gist/minrk/7715212
from __future__ import print_function
from IPython.core import page
def myprint(s):
    try:
        print(s['text/plain'])
    except (KeyError, TypeError):
        print(s)
page.page = myprint

%prun approx_pi()


I am working on the file "Profiling_in_a_Jupyter_notebook.ipynb".
The first task is:
With the memory profiler:
Can you write Python code for it?

%load_ext memory_profiler

%mprun -T /tmp/test_mprun.txt -f recip_square -f approx_pi approx_pi()
%cat /tmp/test_mprun.txt

%memit approx_pi()


I am working on the file "Programacion_imperativa_en_Python.ipynb".
The first task is:
Bucle para sobre listas
Ejercicio 8. Definir, por iteración, la función suma tal que suma(xs) es a suma de los números de la lista xs. Por ejemplo,
~~~

suma([3,2,5])
10
~~~
Can you write Python code for it?

def suma(xs):
  r = 0
  for x in xs:
    r = x + r
  return r

suma([3, 2, 5])


I am working on the file "Programacion_imperativa_en_Python.ipynb".
The first task is:
Condicionales múltiples
Ejercicio 4. Definir la función signo tal que signo(x) es el signo de x. Por ejemplo,
~~~

signo(5)
1
signo(-7)
- 1
signo(0)
0
~~~
Can you write Python code for it?

def signo(x):
  if x > 0:
    return 1
  elif x < 0:
    return -1
  else:
    return 0

signo(5)

signo(-7)

signo(0)


I am working on the file "Programacion_imperativa_en_Python.ipynb".
The first task is:
Definición de funciones
Ejercicio 1. Definir la función suma tal que suma(x, y) es la suma de x e y. . Por ejemplo,
~~~

suma(2,3)
5
~~~
Can you write Python code for it?

def suma(x, y):
  return x+y

suma(2, 3)


I am working on the file "Programacion_imperativa_en_Python.ipynb".
The first task is:
Escritura y lectura
Ejercicio 2. Definir el procedimiento suma que lea dos números y escriba su suma.  Por ejemplo,
~~~

suma()
Escribe el primer número: 2
Escribe el segundo número: 3
La suma es: 5
~~~
Can you write Python code for it?

def suma():
  a = eval(input("Escribe el primer número: "))
  b = eval(input("Escribe el segundo número: "))
  print("La suma es:",a+b)

suma()


I am working on the file "Programacion_imperativa_en_Python.ipynb".
The first task is:
Estructuras iterativas
Bucles mientras
Ejercicio 5. Definir, con un bucle while, la función sumaImpares tal que sumaImpares(n) es la suma de los n primeros números impares. Por ejemplo,
~~~

sumaImpares(3)
9
sumaImpares(4)
16
~~~
Can you write Python code for it?

def sumaImpares(n):
  s, k = 0, 0
  while k < n:
    s = s + 2*k + 1
    k = k + 1
  return s

sumaImpares(3)

sumaImpares(4)


I am working on the file "Programacion_imperativa_en_Python.ipynb".
The first task is:
La estructura condicional
Condicionales simples
Ejercicio 3. Definir, usando condicionales, la función maximo tal que maximo(x,y) es el máximo de x e y. Por ejemplo,
~~~

maximo(2, 5)
5
maximo(2, 1)
2
~~~
Can you write Python code for it?

def maximo(x, y) :
  if x > y:
    return x
  else:
    return y

maximo(2, 5)

maximo (2, 1)


I am working on the file "Programacion_imperativa_en_Python.ipynb".
The first task is:
Variables y asignaciones
Can you write Python code for it?


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
14.
Now, let us repeat this exercise to find the "most negative reviews." Use the prediction probabilities to find the 20 reviews in the test_data with the lowest probability of being classified as a positive review. Repeat the same steps above but make sure you sort in the opposite order.
Quiz Question:
Which of the following products are represented in the 20 most negative reviews?
Can you write Python code for it?

negative_idx = np.argsort(test_scores)[:20]
print negative_idx
print test_scores[negative_idx[0]]
test_data.iloc[negative_idx]


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
18.
Let us inspect the weights (coefficients) of the simple_model. First, build a table to store (word, coefficient) pairs. If you are using SFrame with scikit-learn, you can combine words with coefficients by running
Can you write Python code for it?

simple_model_coef_table = pd.DataFrame({'word':significant_words,
                                         'coefficient':simple_model.coef_.flatten()})
#simple_model_coef_table
simple_model_coef_table.sort_values(['coefficient'], ascending=False)


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
8.
There should be over 100,000 coefficients in this sentiment_model. Recall from the lecture that positive weights w_j correspond to weights that cause positive sentiment, while negative weights correspond to negative sentiment. Calculate the number of positive (>= 0, which is actually nonnegative) coefficients.
Quiz question:
How many weights are >= 0?
Can you write Python code for it?


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Answer
85928
Making predictions with logistic regression
9.
Now that a model is trained, we can make predictions on the test data. In this section, we will explore this in the context of 3 data points in the test data. Take the 11th, 12th, and 13th data points in the test data and save them to sample_test_data. The following cell extracts the three data points from the SFrame test_data and print their content:
Can you write Python code for it?

sample_test_data = test_data.iloc[10:13]
print sample_test_data

sample_test_data.iloc[0]['review']

sample_test_data.iloc[1]['review']

sample_test_matrix = vectorizer.transform(sample_test_data['review_clean'])
scores = sentiment_model.decision_function(sample_test_matrix)
print scores

print sentiment_model.predict(sample_test_matrix)


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Answer:
0.93
Learn another classifier with fewer words
16.
There were a lot of words in the model we trained above. We will now train a simpler logistic regression model using only a subet of words that occur in the reviews. For this assignment, we selected 20 words to work with.
Can you write Python code for it?

significant_words = ['love', 'great', 'easy', 'old', 'little', 'perfect', 'loves',
      'well', 'able', 'car', 'broke', 'less', 'even', 'waste', 'disappointed',
      'work', 'product', 'money', 'would', 'return']

vectorizer_word_subset = CountVectorizer(vocabulary=significant_words) # limit to 20 words
train_matrix_word_subset = vectorizer_word_subset.fit_transform(train_data['review_clean'])
test_matrix_word_subset = vectorizer_word_subset.transform(test_data['review_clean'])


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Answer:
Comparing models
19.
We will now compare the accuracy of the sentiment_model and the simple_model.
First, compute the classification accuracy of the sentiment_model on the train_data.
Now, compute the classification accuracy of the simple_model on the train_data.
Can you write Python code for it?

train_predicted_y = sentiment_model.predict(train_matrix)
correct_num = np.sum(train_predicted_y == train_data['sentiment'])
total_num = len(train_data['sentiment'])
print "correct_num: {}, total_num: {}".format(correct_num, total_num)
train_accuracy = correct_num * 1./ total_num
print "sentiment_model training accuracy: {}".format(train_accuracy)

train_predicted_y = simple_model.predict(train_matrix_word_subset)
correct_num = np.sum(train_predicted_y == train_data['sentiment'])
total_num = len(train_data['sentiment'])
print "correct_num: {}, total_num: {}".format(correct_num, total_num)
train_accuracy = correct_num * 1./ total_num
print "simple_model training accuracy: {}".format(train_accuracy)


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Build the word count vector for each review
6.
We will now compute the word count for each word that appears in the reviews. A vector consisting of word counts is often referred to as bag-of-word features. Since most words occur in only a few reviews, word count vectors are sparse. For this reason, scikit-learn and many other tools use sparse matrices to store a collection of word count vectors. Refer to appropriate manuals to produce sparse word count vectors. General steps for extracting word count vectors are as follows:

Learn a vocabulary (set of all words) from the training data. Only the words that show up in the training data will be considered for feature extraction.
Compute the occurrences of the words in each review and collect them into a row vector.
Build a sparse matrix where each row is the word count vector for the corresponding review. Call this matrix train_matrix.
Using the same mapping between words and columns, convert the test data into a sparse matrix test_matrix.
Can you write Python code for it?

from sklearn.feature_extraction.text import CountVectorizer

vectorizer = CountVectorizer(token_pattern=r'\b\w+\b')
     # Use this token pattern to keep single-letter words
# First, learn vocabulary from the training data and assign columns to words
# Then convert the training data into a sparse matrix
train_matrix = vectorizer.fit_transform(train_data['review_clean'])
# Second, convert the test data into a sparse matrix, using the same word-column mapping
test_matrix = vectorizer.transform(test_data['review_clean'])
#print vectorizer.vocabulary_


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Extract Sentiments
3.
We will ignore all reviews with rating = 3, since they tend to have a neutral sentiment.
Can you write Python code for it?

products = products[products['rating'] != 3]


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Load Amazon dataset
1.
Load the dataset consisting of baby product reviews on Amazon.com. Store the data in a data frame products.
Can you write Python code for it?


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Prediciting Sentiment
11.
These scores can be used to make class predictions as follows:
Using scores, write code to calculate predicted labels for sample_test_data.
Checkpoint: Make sure your class predictions match with the ones obtained from sentiment_model. The logistic regression classifier in scikit-learn comes with the predict function for this purpose.
Probability Predictions
Can you write Python code for it?

print [1./(1+np.exp(-x)) for x in scores]

print sentiment_model.classes_
print sentiment_model.predict_proba(sample_test_matrix)


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Quiz Question:
Consider the coefficients of simple_model. How many of the 20 coefficients (corresponding to the 20 significant_words) are positive for the simple_model?
Can you write Python code for it?


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Quiz Question:
Which model (sentiment_model or simple_model) has higher accuracy on the TEST set?
Answer:
sentiment_model
Baseline: Majority class prediction
21.
It is quite common to use the majority class classifier as the a baseline (or reference) model for comparison with your classifier model. The majority classifier model predicts the majority class for all data points. At the very least, you should healthily beat the majority class classifier, otherwise, the model is (usually) pointless.
Can you write Python code for it?

positive_label = len(test_data[test_data['sentiment']>0])
negative_label = len(test_data[test_data['sentiment']<0])
print "positive_label is {}, negative_label is {}".format(positive_label, negative_label)

baseline_accuracy = positive_label*1./(positive_label+negative_label)
print "baseline_accuracy is {}".format(baseline_accuracy)


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Quiz Question:
Which model (sentiment_model or simple_model) has higher accuracy on the TRAINING set?
Answer:
sentiment_model
20.
Now, we will repeat this exercise on the test_data. Start by computing the classification accuracy of the sentiment_model on the test_data.
Next, compute the classification accuracy of the simple_model on the test_data.
Can you write Python code for it?

test_predicted_y = sentiment_model.predict(test_matrix)
correct_num = np.sum(test_predicted_y == test_data['sentiment'])
total_num = len(test_data['sentiment'])
print "correct_num: {}, total_num: {}".format(correct_num, total_num)
test_accuracy = correct_num * 1./ total_num
print "sentiment_model test accuracy: {}".format(test_accuracy)

test_predicted_y = simple_model.predict(test_matrix_word_subset)
correct_num = np.sum(test_predicted_y == test_data['sentiment'])
total_num = len(test_data['sentiment'])
print "correct_num: {}, total_num: {}".format(correct_num, total_num)
test_accuracy = correct_num * 1./ total_num
print "simple_model test accuracy: {}".format(test_accuracy)


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Quiz question:
Of the three data points in sample_test_data, which one (first, second, or third) has the lowest probability of being classified as a positive review?
Answer:
third
Find the most positive (and negative) review
13.
We now turn to examining the full test dataset, test_data, and use sklearn.linear_model.LogisticRegression to form predictions on all of the test data points.
Using the sentiment_model, find the 20 reviews in the entire test_data with the highest probability of being classified as a positive review. We refer to these as the "most positive reviews."
To calculate these top-20 reviews, use the following steps:

Make probability predictions on test_data using the sentiment_model.
Sort the data according to those predictions and pick the top 20.

Quiz Question:
Which of the following products are represented in the 20 most positive reviews?
Can you write Python code for it?

test_scores = sentiment_model.decision_function(test_matrix)
positive_idx = np.argsort(-test_scores)[:20]
print positive_idx
print test_scores[positive_idx[0]]
test_data.iloc[positive_idx]


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Split into training and test sets
5.
Let's perform a train/test split with 80% of the data in the training set and 20% of the data in the test set. If you are using SFrame, make sure to use seed=1 so that you get the same result as everyone else does. (This way, you will get the right numbers for the quiz.)
Can you write Python code for it?

import json
with open('test_data_idx.json') as test_data_file:
    test_data_idx = json.load(test_data_file)
with open('train_data_idx.json') as train_data_file:
    train_data_idx = json.load(train_data_file)

print train_data_idx[:3]
print test_data_idx[:3]

train_data = products.iloc[train_data_idx]
train_data.head(2)

test_data = products.iloc[test_data_idx]
test_data.head(2)


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Train a logistic regression model on a subset of data
17.
Now build a logistic regression classifier with train_matrix_word_subset as features and sentiment as the target. Call this model simple_model.
Can you write Python code for it?

simple_model = LogisticRegression()
simple_model.fit(train_matrix_word_subset, train_data['sentiment'])


I am working on the file "Programming Assignment 1/Predicting sentiment from product reviews with pandas.ipynb".
The first task is:
Train a sentiment classifier with logistic regression
7.
Learn a logistic regression classifier using the training data. If you are using scikit-learn, you should create an instance of the LogisticRegression class and then call the method fit() to train the classifier. This model should use the sparse word count matrix (train_matrix) as features and the column sentiment of train_data as the target. Use the default values for other parameters. Call this model sentiment_model.
Can you write Python code for it?

from sklearn.linear_model import LogisticRegression
sentiment_model = LogisticRegression()
sentiment_model.fit(train_matrix, train_data['sentiment'])


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
Build the word count vector for each review
Let us explore a specific example of a baby product.
Can you write Python code for it?


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
Data preparation
We will use a dataset consisting of baby product reviews on Amazon.com.
Can you write Python code for it?


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
Extract sentiments
We will ignore all reviews with rating = 3, since they tend to have a neutral sentiment.
Can you write Python code for it?

products = products[products['rating'] != 3]
len(products)


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
For each review, we will use the word_count column and trim out all words that are not in the significant_words list above. We will use the SArray dictionary trim by keys functionality. Note that we are performing this on both the training and test set.
Can you write Python code for it?

train_data['word_count_subset'] = train_data['word_count'].dict_trim_by_keys(significant_words, exclude=False)
test_data['word_count_subset'] = test_data['word_count'].dict_trim_by_keys(significant_words, exclude=False)


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
Let's dig deeper into the first row of the sample_test_data. Here's the full review:
Can you write Python code for it?


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
Let's sort the coefficients (in descending order) by the value to obtain the coefficients with the most positive effect on the sentiment.
Can you write Python code for it?


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
Now, let us see a preview of what the dataset looks like.
Can you write Python code for it?


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
Now, let's compute the classification accuracy of the sentiment_model on the test_data.
Can you write Python code for it?


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
Now, we will assign reviews with a rating of 4 or higher to be positive reviews, while the ones with rating of 2 or lower are negative. For the sentiment column, we use +1 for the positive class label and -1 for the negative class label.
Can you write Python code for it?

products['sentiment'] = products['rating'].apply(lambda rating : +1 if rating > 3 else -1)
products


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
Now, we will perform 2 simple data transformations:

Remove punctuation using Python's built-in string functionality.
Transform the reviews into word-counts.

Aside. In this notebook, we remove all punctuations for the sake of simplicity. A smarter approach to punctuations would preserve phrases such as "I'd", "would've", "hadn't" and so forth. See this page for an example of smart handling of punctuations.
Can you write Python code for it?

def remove_punctuation(text):
    import string
    return text.translate(None, string.punctuation)

review_without_punctuation = products['review'].apply(remove_punctuation)
products['word_count'] = graphlab.text_analytics.count_words(review_without_punctuation)


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
Quiz Question: Consider the coefficients of simple_model. There should be 21 of them, an intercept term + one for each word in significant_words. How many of the 20 coefficients (corresponding to the 20 significant_words and excluding the intercept term) are positive for the simple_model?
Quiz Question: Are the positive words in the simple_model (let us call them positive_significant_words) also positive words in the sentiment_model?
Comparing models
We will now compare the accuracy of the sentiment_model and the simple_model using the get_classification_accuracy method you implemented above.
First, compute the classification accuracy of the sentiment_model on the train_data:
Now, compute the classification accuracy of the simple_model on the train_data:
Quiz Question: Which model (sentiment_model or simple_model) has higher accuracy on the TRAINING set?
Now, we will repeat this exercise on the test_data. Start by computing the classification accuracy of the sentiment_model on the test_data:
Next, we will compute the classification accuracy of the simple_model on the test_data:
Quiz Question: Which model (sentiment_model or simple_model) has higher accuracy on the TEST set?
Baseline: Majority class prediction
It is quite common to use the majority class classifier as the a baseline (or reference) model for comparison with your classifier model. The majority classifier model predicts the majority class for all data points. At the very least, you should healthily beat the majority class classifier, otherwise, the model is (usually) pointless.
What is the majority class in the train_data?
Can you write Python code for it?

num_positive  = (train_data['sentiment'] == +1).sum()
num_negative = (train_data['sentiment'] == -1).sum()
print num_positive
print num_negative


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
Since we are only working with a subset of these words, the column word_count_subset is a subset of the above dictionary. In this example, only 2 significant words are present in this review.
Can you write Python code for it?


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
That review seems pretty positive.
Now, let's see what the next row of the sample_test_data looks like. As we could guess from the sentiment (-1), the review is quite negative.
Can you write Python code for it?


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
The word_count column had been working with before looks like the following:
Can you write Python code for it?


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
There are a total of 121713 coefficients in the model. Recall from the lecture that positive weights $w_j$ correspond to weights that cause positive sentiment, while negative weights correspond to negative sentiment.
Fill in the following block of code to calculate how many weights are positive ( >= 0). (Hint: The 'value' column in SFrame weights must be positive ( >= 0)).
Can you write Python code for it?

num_positive_weights = ...
num_negative_weights = ...

print "Number of positive weights: %s " % num_positive_weights
print "Number of negative weights: %s " % num_negative_weights


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
We can compute the classification accuracy using the get_classification_accuracy function you implemented earlier.
Can you write Python code for it?


I am working on the file "Programming Assignment 1/module-2-linear-classifier-assignment-blank.ipynb".
The first task is:
We will now make a class prediction for the sample_test_data. The sentiment_model should predict +1 if the sentiment is positive and -1 if the sentiment is negative. Recall from the lecture that the score (sometimes called margin) for the logistic regression model  is defined as:
$$
\mbox{score}_i = \mathbf{w}^T h(\mathbf{x}_i)
$$
where $h(\mathbf{x}_i)$ represents the features for example $i$.  We will write some code to obtain the scores using GraphLab Create. For each row, the score (or margin) is a number in the range [-inf, inf].
Can you write Python code for it?

scores = sentiment_model.predict(sample_test_data, output_type='margin')
print scores


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
### 7. compute the number of product reviews that contain the word perfect.
Can you write Python code for it?

products['contains_perfect'] = products['perfect'] >=1
print products['contains_perfect'].sum()


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
1. listing the name of the first 10 products in the dataset.
Can you write Python code for it?


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
12. Write a function compute_log_likelihood
Can you write Python code for it?

def compute_log_likelihood(feature_matrix, sentiment, coefficients):
    indicator = (sentiment==+1)
    scores = np.dot(feature_matrix, coefficients)
    # scores.shape (53072L, 1L)
    # indicator.shape (53072L,)
    lp = np.sum((np.transpose(np.array([indicator]))-1)*scores - np.log(1. + np.exp(-scores)))
    return lp


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
14. run the logistic regression solver

feature_matrix = feature_matrix extracted in #9
sentiment = sentiment extracted in #9
initial_coefficients = a 194-dimensional vector filled with zeros
step_size = 1e-7
max_iter = 301
Can you write Python code for it?

initial_coefficients = np.zeros((194,1))
step_size = 1e-7
max_iter = 301

coefficients = logistic_regression(feature_matrix, sentiment, initial_coefficients, step_size, max_iter)

coefficients = logistic_regression(feature_matrix, sentiment, initial_coefficients, step_size, max_iter)


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
2. Quiz Question:
How many features are there in the feature_matrix?
Can you write Python code for it?


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
4. data transformations:

fill n/a values in the review column with empty strings
Remove punctuation
Compute word counts (only for important_words)
Can you write Python code for it?

products = products.fillna({'review':''})  # fill in N/A's in the review column

def remove_punctuation(text):
    import string
    return text.translate(None, string.punctuation)

products['review_clean'] = products['review'].apply(remove_punctuation)
products.head(3)


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
5. Quiz question :
As each iteration of gradient ascent passes, does the log likelihood increase or decrease?
Answer
increase
15. compute class predictions

First compute the scores using feature_matrix and coefficients using a dot product.
Then apply threshold 0 on the scores to compute the class predictions. Refer to the formula above.
Can you write Python code for it?

"""
feature_matrix: N * D
coefficients: D * 1
predictions: N * 1
"""
predictions = predict_probability(feature_matrix, coefficients)
NumPositive = (predictions > 0.5).sum()
print NumPositive

score = np.dot(feature_matrix, coefficients) # N * 1
print (score > 0).sum()


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
5. compute a count for the number of times the word occurs in the review
Can you write Python code for it?

for word in important_words:
    products[word] = products['review_clean'].apply(lambda s : s.split().count(word))

products.head(1)


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
9. extract two arrays feature_matrix and sentiment
Can you write Python code for it?

feature_matrix, sentiment = get_numpy_data(products, important_words, 'sentiment')


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
Apply text cleaning on the review data
3. load the features
Can you write Python code for it?

import json
with open('important_words.json') as important_words_file:
    important_words = json.load(important_words_file)
print important_words[:3]


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
Load review dataset
Can you write Python code for it?


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
Taking gradient steps
13. Write a function logistic_regression to fit a logistic regression model using gradient ascent.
The function accepts the following parameters:

feature_matrix: 2D array of features
sentiment: 1D array of class labels
initial_coefficients: 1D array containing initial values of coefficients
step_size: a parameter controlling the size of the gradient steps
max_iter: number of iterations to run gradient ascent
The function returns the last set of coefficients after performing gradient ascent.

The function carries out the following steps:

Initialize vector coefficients to initial_coefficients.
Predict the class probability P(yi=+1|xi,w) using your predict_probability function and save it to variable predictions.
Compute indicator value for (yi=+1) by comparing sentiment against +1. Save it to variable indicator.
Compute the errors as difference between indicator and predictions. Save the errors to variable errors.
For each j-th coefficient, compute the per-coefficient derivative by calling feature_derivative with the j-th column of feature_matrix. Then increment the j-th coefficient by (step_size*derivative).
Once in a while, insert code to print out the log likelihood.
Repeat steps 2-6 for max_iter times.
Can you write Python code for it?

# coefficients: D * 1
from math import sqrt
def logistic_regression(feature_matrix, sentiment, initial_coefficients, step_size, max_iter):
    coefficients = np.array(initial_coefficients) # make sure it's a numpy array
    # lplist = []
    for itr in xrange(max_iter):
        # Predict P(y_i = +1|x_1,w) using your predict_probability() function
        # YOUR CODE HERE
        predictions = predict_probability(feature_matrix, coefficients)

        # Compute indicator value for (y_i = +1)
        indicator = (sentiment==+1)

        # Compute the errors as indicator - predictions
        errors = np.transpose(np.array([indicator])) - predictions

        for j in xrange(len(coefficients)): # loop over each coefficient
            # Recall that feature_matrix[:,j] is the feature column associated with coefficients[j]
            # compute the derivative for coefficients[j]. Save it in a variable called derivative
            # YOUR CODE HERE
            derivative = feature_derivative(errors, feature_matrix[:,j])

            # add the step size times the derivative to the current coefficient
            # YOUR CODE HERE
            coefficients[j] += step_size*derivative

        # Checking whether log likelihood is increasing
        if itr <= 15 or (itr <= 100 and itr % 10 == 0) or (itr <= 1000 and itr % 100 == 0) \
        or (itr <= 10000 and itr % 1000 == 0) or itr % 10000 == 0:
            # lplist.append(compute_log_likelihood(feature_matrix, sentiment, coefficients))
            lp = compute_log_likelihood(feature_matrix, sentiment, coefficients)
            print 'iteration %*d: log likelihood of observed labels = %.8f' % \
                (int(np.ceil(np.log10(max_iter))), itr, lp)
    """
    import matplotlib.pyplot as plt
    x= [i for i in range(len(lplist))]
    plt.plot(x,lplist,'ro')
    plt.show()
    """
    return coefficients


I am working on the file "Programming Assignment 2/Implementing logistic regression from scratch.ipynb".
The first task is:
Ten "most positive" words
18. Compute the 10 most positive words
Can you write Python code for it?


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Are you running this notebook on an Amazon EC2 t2.micro instance? (If you are using your own machine, please skip this section)
It has been reported that t2.micro instances do not provide sufficient power to complete the conversion in acceptable amount of time. For interest of time, please refrain from running get_numpy_data function. Instead, download the binary file containing the four NumPy arrays you'll need for the assignment. To load the arrays, run the following commands:
arrays = np.load('module-3-assignment-numpy-arrays.npz')
feature_matrix, sentiment = arrays['feature_matrix'], arrays['sentiment']
Can you write Python code for it?


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Aside. How the link function works with matrix algebra
Since the word counts are stored as columns in feature_matrix, each $i$-th row of the matrix corresponds to the feature vector $h(\mathbf{x}_i)$:
$$
[\text{feature_matrix}] =
\left[
\begin{array}{c}
h(\mathbf{x}_1)^T \
h(\mathbf{x}_2)^T \
\vdots \
h(\mathbf{x}_N)^T
\end{array}
\right] =
\left[
\begin{array}{cccc}
h_0(\mathbf{x}_1) & h_1(\mathbf{x}_1) & \cdots & h_D(\mathbf{x}_1) \
h_0(\mathbf{x}_2) & h_1(\mathbf{x}_2) & \cdots & h_D(\mathbf{x}_2) \
\vdots & \vdots & \ddots & \vdots \
h_0(\mathbf{x}_N) & h_1(\mathbf{x}_N) & \cdots & h_D(\mathbf{x}_N)
\end{array}
\right]
$$
By the rules of matrix multiplication, the score vector containing elements $\mathbf{w}^T h(\mathbf{x}_i)$ is obtained by multiplying feature_matrix and the coefficient vector $\mathbf{w}$.
$$
[\text{score}] =
[\text{feature_matrix}]\mathbf{w} =
\left[
\begin{array}{c}
h(\mathbf{x}_1)^T \
h(\mathbf{x}_2)^T \
\vdots \
h(\mathbf{x}_N)^T
\end{array}
\right]
\mathbf{w}
= \left[
\begin{array}{c}
h(\mathbf{x}_1)^T\mathbf{w} \
h(\mathbf{x}_2)^T\mathbf{w} \
\vdots \
h(\mathbf{x}_N)^T\mathbf{w}
\end{array}
\right]
= \left[
\begin{array}{c}
\mathbf{w}^T h(\mathbf{x}_1) \
\mathbf{w}^T h(\mathbf{x}_2) \
\vdots \
\mathbf{w}^T h(\mathbf{x}_N)
\end{array}
\right]
$$
Checkpoint
Just to make sure you are on the right track, we have provided a few examples. If your predict_probability function is implemented correctly, then the outputs will match:
Can you write Python code for it?

dummy_feature_matrix = np.array([[1.,2.,3.], [1.,-1.,-1]])
dummy_coefficients = np.array([1., 3., -1.])

correct_scores      = np.array( [ 1.*1. + 2.*3. + 3.*(-1.),          1.*1. + (-1.)*3. + (-1.)*(-1.) ] )
correct_predictions = np.array( [ 1./(1+np.exp(-correct_scores[0])), 1./(1+np.exp(-correct_scores[1])) ] )

print 'The following outputs must match '
print '------------------------------------------------'
print 'correct_predictions           =', correct_predictions
print 'output of predict_probability =', predict_probability(dummy_feature_matrix, dummy_coefficients)


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Checkpoint
Just to make sure we are on the same page, run the following code block and check that the outputs match.
Can you write Python code for it?

dummy_feature_matrix = np.array([[1.,2.,3.], [1.,-1.,-1]])
dummy_coefficients = np.array([1., 3., -1.])
dummy_sentiment = np.array([-1, 1])

correct_indicators  = np.array( [ -1==+1,                                       1==+1 ] )
correct_scores      = np.array( [ 1.*1. + 2.*3. + 3.*(-1.),                     1.*1. + (-1.)*3. + (-1.)*(-1.) ] )
correct_first_term  = np.array( [ (correct_indicators[0]-1)*correct_scores[0],  (correct_indicators[1]-1)*correct_scores[1] ] )
correct_second_term = np.array( [ np.log(1. + np.exp(-correct_scores[0])),      np.log(1. + np.exp(-correct_scores[1])) ] )

correct_ll          =      sum( [ correct_first_term[0]-correct_second_term[0], correct_first_term[1]-correct_second_term[1] ] )

print 'The following outputs must match '
print '------------------------------------------------'
print 'correct_log_likelihood           =', correct_ll
print 'output of compute_log_likelihood =', compute_log_likelihood(dummy_feature_matrix, dummy_sentiment, dummy_coefficients)


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
In the main lecture, our focus was on the likelihood.  In the advanced optional video, however, we introduced a transformation of this likelihood---called the log likelihood---that simplifies the derivation of the gradient and is more numerically stable.  Due to its numerical stability, we will use the log likelihood instead of the likelihood to assess the algorithm.
The log likelihood is computed using the following formula (see the advanced optional video if you are curious about the derivation of this equation):
$$\ell\ell(\mathbf{w}) = \sum_{i=1}^N \Big( (\mathbf{1}[y_i = +1] - 1)\mathbf{w}^T h(\mathbf{x}_i) - \ln\left(1 + \exp(-\mathbf{w}^T h(\mathbf{x}_i))\right) \Big) $$
We provide a function to compute the log likelihood for the entire dataset.
Can you write Python code for it?

def compute_log_likelihood(feature_matrix, sentiment, coefficients):
    indicator = (sentiment==+1)
    scores = np.dot(feature_matrix, coefficients)
    logexp = np.log(1. + np.exp(-scores))

    # Simple check to prevent overflow
    mask = np.isinf(logexp)
    logexp[mask] = -scores[mask]

    lp = np.sum((indicator-1)*scores - logexp)
    return lp


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Let us convert the data into NumPy arrays.
Can you write Python code for it?

# Warning: This may take a few minutes...
feature_matrix, sentiment = get_numpy_data(products, important_words, 'sentiment')


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Load review dataset
For this assignment, we will use a subset of the Amazon product review dataset. The subset was chosen to contain similar numbers of positive and negative reviews, as the original dataset consisted primarily of positive reviews.
Can you write Python code for it?


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Now we proceed with Step 2. For each word in important_words, we compute a count for the number of times the word occurs in the review. We will store this count in a separate column (one for each word). The result of this feature processing is a single column for each word in important_words which keeps a count of the number of times the respective word occurs in the review text.
Note: There are several ways of doing this. In this assignment, we use the built-in count function for Python lists. Each review string is first split into individual words and the number of occurances of a given word is counted.
Can you write Python code for it?

for word in important_words:
    products[word] = products['review_clean'].apply(lambda s : s.split().count(word))


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Now, complete the following code block for Step 2 to compute the class predictions using the scores obtained above:
 Quiz Question:  How many reviews were predicted to have positive sentiment?
Measuring accuracy
We will now measure the classification accuracy of the model. Recall from the lecture that the classification accuracy can be computed as follows:
$$
\mbox{accuracy} = \frac{\mbox{# correctly classified data points}}{\mbox{# total data points}}
$$
Complete the following code block to compute the accuracy of the model.
Can you write Python code for it?

num_mistakes = ... # YOUR CODE HERE
accuracy = ... # YOUR CODE HERE
print "-----------------------------------------------------"
print '# Reviews   correctly classified =', len(products) - num_mistakes
print '# Reviews incorrectly classified =', num_mistakes
print '# Reviews total                  =', len(products)
print "-----------------------------------------------------"
print 'Accuracy = %.2f' % accuracy


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Now, we will perform 2 simple data transformations:

Remove punctuation using Python's built-in string functionality.
Compute word counts (only for important_words)

We start with Step 1 which can be done as follows:
Can you write Python code for it?

def remove_punctuation(text):
    import string
    return text.translate(None, string.punctuation)

products['review_clean'] = products['review'].apply(remove_punctuation)


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
One column of this dataset is 'sentiment', corresponding to the class label with +1 indicating a review with positive sentiment and -1 indicating one with negative sentiment.
Can you write Python code for it?


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Quiz Question: As each iteration of gradient ascent passes, does the log likelihood increase or decrease?
Predicting sentiments
Recall from lecture that class predictions for a data point $\mathbf{x}$ can be computed from the coefficients $\mathbf{w}$ using the following formula:
$$
\hat{y}_i =
\left{
\begin{array}{ll}
      +1 & \mathbf{x}_i^T\mathbf{w} > 0 \
      -1 & \mathbf{x}_i^T\mathbf{w} \leq 0 \
\end{array}
\right.
$$
Now, we will write some code to compute class predictions. We will do this in two steps:
* Step 1: First compute the scores using feature_matrix and coefficients using a dot product.
* Step 2: Using the formula above, compute the class predictions from the scores.
Step 1 can be implemented as follows:
Can you write Python code for it?

# Compute the scores as a dot product between feature_matrix and coefficients.
scores = np.dot(feature_matrix, coefficients)


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Quiz Question: How many features are there in the feature_matrix?
 Quiz Question: Assuming that the intercept is present, how does the number of features in feature_matrix relate to the number of features in the logistic regression model?
Now, let us see what the sentiment column looks like:
Can you write Python code for it?


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
Quiz Question: What is the accuracy of the model on predictions made above? (round to 2 digits of accuracy)
Which words contribute most to positive & negative sentiments?
Recall that in Module 2 assignment, we were able to compute the "most positive words". These are words that correspond most strongly with positive reviews. In order to do this, we will first do the following:
* Treat each coefficient as a tuple, i.e. (word, coefficient_value).
* Sort all the (word, coefficient_value) tuples by coefficient_value in descending order.
Can you write Python code for it?

coefficients = list(coefficients[1:]) # exclude intercept
word_coefficient_tuples = [(word, coefficient) for word, coefficient in zip(important_words, coefficients)]
word_coefficient_tuples = sorted(word_coefficient_tuples, key=lambda x:x[1], reverse=True)


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
The SFrame products now contains one column for each of the 193 important_words. As an example, the column perfect contains a count of the number of times the word perfect occurs in each of the reviews.
Can you write Python code for it?


I am working on the file "Programming Assignment 2/module-3-linear-classifier-learning-assignment-blank.ipynb".
The first task is:
We now provide you with a function that extracts columns from an SFrame and converts them into a NumPy array. Two arrays are returned: one representing features and another representing class labels. Note that the feature matrix includes an additional column 'intercept' to take account of the intercept term.
Can you write Python code for it?

def get_numpy_data(data_sframe, features, label):
    data_sframe['intercept'] = 1
    features = ['intercept'] + features
    features_sframe = data_sframe[features]
    feature_matrix = features_sframe.to_numpy()
    label_sarray = data_sframe[label]
    label_array = label_sarray.to_numpy()
    return(feature_matrix, label_array)


I am working on the file "Programming Assignment 3/Logistic Regression with L2 regularization.ipynb".
The first task is:
1. Quiz question:
In the code above, was the intercept term regularized?
1. Answer:
No
10. computing log likelihood  with L2
Can you write Python code for it?

def compute_log_likelihood_with_L2(feature_matrix, sentiment, coefficients, l2_penalty):
    indicator = (sentiment==+1)
    scores = np.dot(feature_matrix, coefficients)
    # scores.shape (53072L, 1L)
    # indicator.shape (53072L,)
    # lp = np.sum((indicator-1)*scores - np.log(1. + np.exp(-scores))) - l2_penalty*np.sum(coefficients[1:]**2)
    lp = np.sum((np.transpose(np.array([indicator]))-1)*scores - np.log(1. + np.exp(-scores))) - l2_penalty*np.sum(coefficients[1:]**2)

    return lp


I am working on the file "Programming Assignment 3/Logistic Regression with L2 regularization.ipynb".
The first task is:
14.  observe the effect of increasing L2 penalty on the 10 words
Can you write Python code for it?

table = pd.DataFrame(data=[coefficients_0_penalty.flatten(), coefficients_4_penalty.flatten(), coefficients_10_penalty.flatten(), coefficients_1e2_penalty.flatten(), coefficients_1e3_penalty.flatten(), coefficients_1e5_penalty.flatten()],
                     index=[0, 4, 10, 100.0, 1000.0, 100000.0],
                     columns=['(intercept)'] + important_words)

table.head(2)

import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = 10, 6

def make_coefficient_plot(table, positive_words, negative_words, l2_penalty_list):
    cmap_positive = plt.get_cmap('Reds')
    cmap_negative = plt.get_cmap('Blues')

    xx = l2_penalty_list
    plt.plot(xx, [0.]*len(xx), '--', lw=1, color='k')

    table_positive_words = table[positive_words]
    table_negative_words = table[negative_words]
    #del table_positive_words['word']
    #del table_negative_words['word']

    for i, value in enumerate(positive_words):
        color = cmap_positive(0.8*((i+1)/(len(positive_words)*1.2)+0.15))
        plt.plot(xx, table_positive_words[value].as_matrix().flatten(),
                 '-', label=positive_words[i], linewidth=4.0, color=color)

    for i, value in enumerate(negative_words):
        color = cmap_negative(0.8*((i+1)/(len(negative_words)*1.2)+0.15))
        plt.plot(xx, table_negative_words[value].as_matrix().flatten(),
                 '-', label=negative_words[i], linewidth=4.0, color=color)

    plt.legend(loc='best', ncol=3, prop={'size':16}, columnspacing=0.5)
    plt.axis([1, 1e5, -1, 2])
    plt.title('Coefficient path')
    plt.xlabel('L2 penalty ($\lambda$)')
    plt.ylabel('Coefficient value')
    plt.xscale('log')
    plt.rcParams.update({'font.size': 18})
    plt.tight_layout()

make_coefficient_plot(table, positive_words, negative_words, l2_penalty_list=[0, 4, 10, 1e2, 1e3, 1e5])


I am working on the file "Programming Assignment 3/Logistic Regression with L2 regularization.ipynb".
The first task is:
2. Quiz question:
Does the term with L2 regularization increase or decrease ℓℓ(w)?
3. Answer:
decrease
11. Write a function logistic_regression_with_L2 to fit a logistic regression model under L2 regularization.
The function accepts the following parameters:

feature_matrix: 2D array of features
sentiment: 1D array of class labels
initial_coefficients: 1D array containing initial values of coefficients
step_size: a parameter controlling the size of the gradient steps
l2_penalty: the L2 penalty constant λ
max_iter: number of iterations to run gradient ascent
The function returns the last set of coefficients after performing gradient ascent.

The function carries out the following steps:

Initialize vector coefficients to initial_coefficients.
Predict the class probability P(yi=+1|xi,w) using your predict_probability function and save it to variable predictions.
Compute indicator value for (yi=+1) by comparing sentiment against +1. Save it to variable indicator.
Compute the errors as difference between indicator and predictions. Save the errors to variable errors.
For each j-th coefficient, compute the per-coefficient derivative by calling feature_derivative_L2 with the j-th column of feature_matrix. Don't forget to supply the L2 penalty. Then increment the j-th coefficient by (step_size*derivative).
Once in a while, insert code to print out the log likelihood.
Repeat steps 2-6 for max_iter times.
Can you write Python code for it?

# coefficients: D * 1
def logistic_regression_with_L2(feature_matrix, sentiment, initial_coefficients, step_size, l2_penalty, max_iter):
    coefficients = np.array(initial_coefficients) # make sure it's a numpy array
    for itr in xrange(max_iter):
        # Predict P(y_i = +1|x_i,w) using your predict_probability() function
        ## YOUR CODE HERE
        predictions = predict_probability(feature_matrix, coefficients)

        # Compute indicator value for (y_i = +1)
        indicator = (sentiment==+1)

        # Compute the errors as indicator - predictions
        errors = np.transpose(np.array([indicator])) - predictions
        for j in xrange(len(coefficients)): # loop over each coefficient
            is_intercept = (j == 0)
            # Recall that feature_matrix[:,j] is the feature column associated with coefficients[j].
            # Compute the derivative for coefficients[j]. Save it in a variable called derivative
            ## YOUR CODE HERE
            derivative = feature_derivative_with_L2(errors, feature_matrix[:,j], coefficients[j], l2_penalty, is_intercept)

            # add the step size times the derivative to the current coefficient
            ## YOUR CODE HERE
            coefficients[j] += step_size*derivative

        # Checking whether log likelihood is increasing
        if itr <= 15 or (itr <= 100 and itr % 10 == 0) or (itr <= 1000 and itr % 100 == 0) \
        or (itr <= 10000 and itr % 1000 == 0) or itr % 10000 == 0:
            lp = compute_log_likelihood_with_L2(feature_matrix, sentiment, coefficients, l2_penalty)
            print 'iteration %*d: log likelihood of observed labels = %.8f' % \
                (int(np.ceil(np.log10(max_iter))), itr, lp)
    return coefficients


I am working on the file "Programming Assignment 3/Logistic Regression with L2 regularization.ipynb".
The first task is:
2. data transformations:

fill n/a values in the review column with empty strings
Remove punctuation
Can you write Python code for it?

products = products.fillna({'review':''})  # fill in N/A's in the review column

def remove_punctuation(text):
    import string
    return text.translate(None, string.punctuation)

products['review_clean'] = products['review'].apply(remove_punctuation)
products.head(3)


I am working on the file "Programming Assignment 3/Logistic Regression with L2 regularization.ipynb".
The first task is:
3. Quiz Question:
(True/False) All coefficients consistently get smaller in size as L2 penalty is increased.
3. Answer:
True
Measuring accuracy
15.  compute the accuracy of the classifier model.
Can you write Python code for it?

"""
feature_matrix: N * D
coefficients: D * 1
predictions: N * 1
"""
training_accuracy = []
for coefficient in [coefficients_0_penalty, coefficients_4_penalty, coefficients_10_penalty, coefficients_1e2_penalty, coefficients_1e3_penalty, coefficients_1e5_penalty]:
    predictions = predict_probability(feature_matrix_train, coefficient)
    correct_num = np.sum((np.transpose(predictions.flatten())> 0.5) == (np.array(sentiment_train)>0))
    total_num = len(sentiment_train)
    #print "correct_num: {}, total_num: {}".format(correct_num, total_num)
    training_accuracy.append(correct_num * 1./ total_num)
    #print accuracy
l2_penalty_list=[x for x in range(6)]
plt.plot(l2_penalty_list, training_accuracy,'ro')
plt.title('training_accuracy')
plt.xlabel('L2 penalty ($\lambda$)')
plt.ylabel('training_accuracy')
plt.show()

[0, 4, 10, 1e2, 1e3, 1e5][training_accuracy.index(max(training_accuracy))]

training_accuracy[training_accuracy.index(max(training_accuracy))]


I am working on the file "Programming Assignment 3/Logistic Regression with L2 regularization.ipynb".
The first task is:
4. Show 'perfect' word counts
Can you write Python code for it?


I am working on the file "Programming Assignment 3/Logistic Regression with L2 regularization.ipynb".
The first task is:
Adding L2 penalty
9. Adding L2 penalty to the derivative

errors: vector whose i-th value contains
feature: vector whose i-th value contains
coefficient: the current value of the j-th coefficient.
l2_penalty: the L2 penalty constant λ
feature_is_constant: a Boolean value indicating whether the j-th feature is constant or not.

The function should do the following:

Take the five parameters as above.
Compute the dot product of errors and feature and save the result to derivative.
If feature_is_constant is False, subtract the L2 penalty term from derivative. Otherwise, do nothing.
Return derivative.
Can you write Python code for it?

def feature_derivative_with_L2(errors, feature, coefficient, l2_penalty, feature_is_constant):

    # Compute the dot product of errors and feature
    ## YOUR CODE HERE
    """
    errors: N * 1
    feature: N * 1
    derivative: 1
    coefficient: 1
    """
    derivative = np.dot(np.transpose(errors), feature)

    # add L2 penalty term for any feature that isn't the intercept.
    if not feature_is_constant:
        ## YOUR CODE HERE
        derivative -= 2 * l2_penalty * coefficient

    return derivative


I am working on the file "Programming Assignment 3/Logistic Regression with L2 regularization.ipynb".
The first task is:
Building on logistic regression with no L2 penalty assignment
7. Compute predictions given by the link function.

Take two parameters: feature_matrix and coefficients.
First compute the dot product of feature_matrix and coefficients.
Then compute the link function P(y=+1|x,w).
Return the predictions given by the link function.
Can you write Python code for it?

'''
feature_matrix: N * D(intercept term included)
coefficients: D * 1
predictions: N * 1
produces probablistic estimate for P(y_i = +1 | x_i, w).
estimate ranges between 0 and 1.
'''

def predict_probability(feature_matrix, coefficients):
    # Take dot product of feature_matrix and coefficients
    # YOUR CODE HERE
    score = np.dot(feature_matrix, coefficients) # N * 1

    # Compute P(y_i = +1 | x_i, w) using the link function
    # YOUR CODE HERE
    predictions = 1.0/(1+np.exp(-score))

    # return predictions
    return predictions


I am working on the file "Programming Assignment 3/Logistic Regression with L2 regularization.ipynb".
The first task is:
Compare coefficients
13. Analysis coefficient without penalty
Can you write Python code for it?

coefficients_0_penalty_without_intercept = list(coefficients_0_penalty[1:]) # exclude intercept
word_coefficient_tuples = [(word, coefficient) for word, coefficient in zip(important_words, coefficients_0_penalty_without_intercept)]
word_coefficient_tuples = sorted(word_coefficient_tuples, key=lambda x:x[1], reverse=True)

positive_words = []
for i in range(5):
    positive_words.append(word_coefficient_tuples[:5][i][0])
positive_words

negative_words = []
for i in range(5):
    negative_words.append(word_coefficient_tuples[-5:][i][0])
negative_words


I am working on the file "Programming Assignment 3/Logistic Regression with L2 regularization.ipynb".
The first task is:
Explore effects of L2 regularization
12.  train models with different L2
Now that we have written up all the pieces needed for an L2 solver with logistic regression, let's explore the benefits of using L2 regularization while analyzing sentiment for product reviews. As iterations pass, the log likelihood should increase.
Let us train models with increasing amounts of regularization, starting with no L2 penalty, which is equivalent to our previous logistic regression implementation. Train 6 models with L2 penalty values 0, 4, 10, 1e2, 1e3, and 1e5. Use the following values for the other parameters:

feature_matrix = feature_matrix_train extracted in #7
sentiment = sentiment_train extracted in #7
initial_coefficients = a 194-dimensional vector filled with zeros
step_size = 5e-6
max_iter = 501

Save the 6 sets of coefficients as coefficients_0_penalty, coefficients_4_penalty, coefficients_10_penalty, coefficients_1e2_penalty, coefficients_1e3_penalty, and coefficients_1e5_penalty respectively.
Can you write Python code for it?

initial_coefficients = np.zeros((194,1))
step_size = 5e-6
max_iter = 501

coefficients_0_penalty = logistic_regression_with_L2(feature_matrix_train , sentiment_train , initial_coefficients, step_size, 0, max_iter)

coefficients_4_penalty = logistic_regression_with_L2(feature_matrix_train , sentiment_train , initial_coefficients, step_size, 4, max_iter)

coefficients_10_penalty = logistic_regression_with_L2(feature_matrix_train , sentiment_train , initial_coefficients, step_size, 10, max_iter)

coefficients_1e2_penalty = logistic_regression_with_L2(feature_matrix_train , sentiment_train , initial_coefficients, step_size, 1e2, max_iter)

coefficients_1e3_penalty = logistic_regression_with_L2(feature_matrix_train , sentiment_train , initial_coefficients, step_size, 1e3, max_iter)

coefficients_1e5_penalty = logistic_regression_with_L2(feature_matrix_train , sentiment_train , initial_coefficients, step_size, 1e5, max_iter)


I am working on the file "Programming Assignment 3/Logistic Regression with L2 regularization.ipynb".
The first task is:
Load and process review dataset
1. Load the dataset into a data frame named products.
Can you write Python code for it?


I am working on the file "Programming Assignment 3/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Are you running this notebook on an Amazon EC2 t2.micro instance? (If you are using your own machine, please skip this section)
It has been reported that t2.micro instances do not provide sufficient power to complete the conversion in acceptable amount of time. For interest of time, please refrain from running get_numpy_data function. Instead, download the binary file containing the four NumPy arrays you'll need for the assignment. To load the arrays, run the following commands:
arrays = np.load('module-4-assignment-numpy-arrays.npz')
feature_matrix_train, sentiment_train = arrays['feature_matrix_train'], arrays['sentiment_train']
feature_matrix_valid, sentiment_valid = arrays['feature_matrix_valid'], arrays['sentiment_valid']
Building on logistic regression with no L2 penalty assignment
Let us now build on Module 3 assignment. Recall from lecture that the link function for logistic regression can be defined as:
$$
P(y_i = +1 | \mathbf{x}_i,\mathbf{w}) = \frac{1}{1 + \exp(-\mathbf{w}^T h(\mathbf{x}_i))},
$$
where the feature vector $h(\mathbf{x}_i)$ is given by the word counts of important_words in the review $\mathbf{x}_i$.
We will use the same code as in this past assignment to make probability predictions since this part is not affected by the L2 penalty.  (Only the way in which the coefficients are learned is affected by the addition of a regularization term.)
Can you write Python code for it?

'''
produces probablistic estimate for P(y_i = +1 | x_i, w).
estimate ranges between 0 and 1.
'''
def predict_probability(feature_matrix, coefficients):
    # Take dot product of feature_matrix and coefficients
    ## YOUR CODE HERE
    ...

    # Compute P(y_i = +1 | x_i, w) using the link function
    ## YOUR CODE HERE
    predictions = ...

    return predictions


I am working on the file "Programming Assignment 3/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Below, we compare the accuracy on the training data and validation data for all the models that were trained in this assignment.  We first calculate the accuracy values and then build a simple report summarizing the performance for the various models.
Can you write Python code for it?

train_accuracy = {}
train_accuracy[0]   = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_0_penalty)
train_accuracy[4]   = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_4_penalty)
train_accuracy[10]  = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_10_penalty)
train_accuracy[1e2] = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_1e2_penalty)
train_accuracy[1e3] = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_1e3_penalty)
train_accuracy[1e5] = get_classification_accuracy(feature_matrix_train, sentiment_train, coefficients_1e5_penalty)

validation_accuracy = {}
validation_accuracy[0]   = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_0_penalty)
validation_accuracy[4]   = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_4_penalty)
validation_accuracy[10]  = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_10_penalty)
validation_accuracy[1e2] = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_1e2_penalty)
validation_accuracy[1e3] = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_1e3_penalty)
validation_accuracy[1e5] = get_classification_accuracy(feature_matrix_valid, sentiment_valid, coefficients_1e5_penalty)

# Build a simple report
for key in sorted(validation_accuracy.keys()):
    print "L2 penalty = %g" % key
    print "train accuracy = %s, validation_accuracy = %s" % (train_accuracy[key], validation_accuracy[key])
    print "--------------------------------------------------------------------------------"

# Optional. Plot accuracy on training and validation sets over choice of L2 penalty.
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = 10, 6

sorted_list = sorted(train_accuracy.items(), key=lambda x:x[0])
plt.plot([p[0] for p in sorted_list], [p[1] for p...


I am working on the file "Programming Assignment 3/module-4-linear-classifier-regularization-assignment-blank.ipynb".
The first task is:
Compare coefficients
We now compare the coefficients for each of the models that were trained above. We will create a table of features and learned coefficients associated with each of the different L2 penalty values.
Below is a simple helper function that will help us create this table.
Can you write Python code for it?

table = graphlab.SFrame({'word': ['(intercept)'] + important_words})
def add_coefficients_to_table(coefficients, column_name):
    table[column_name] = coefficients
    return table