I am working on the file "Atividade02/Lab/Lab3_AnaliseExploratoria.ipynb".
The first task is:
O próximo passo consiste em calcular o total de pares Mes-Ano para ser possível o cálculo da média.
Finalmente, criamos a RDD fractionCrimesDateRDD em que a chave é Mes-Ano e o valor é uma lista da fração de cada tipo de crime ocorridos naquele mês e ano. Para gerar essa lista vamos utilizar o list comprehension do Python de tal forma a calcular a fração para cada crime na variável crimes.
Os dicionários em Python tem um método chamado get() que permite atribuir um valor padrão caso a chave não exista. Ex.: dicionario.get( chave, 0.0) retornará 0.0 caso a chave não exista.
Can you write Python code for it?

# EXERCICIO

totalPerDateRDD = (CrimeHeadlessRDD
                   .<COMPLETAR>
                   .<COMPLETAR>
                  )

fractionCrimesDateRDD = (datesCrimesRDD
                         .<COMPLETAR>
                         .<COMPLETAR>
                         .cache()
                        )

print fractionCrimesDateRDD.take(1)

assert np.abs(fractionCrimesDateRDD.take(1)[0][1][0][1]-0.163950)<1e-6,'valores incorretos'
print 'ok'


I am working on the file "Atividade02/Lab/Lab3_AnaliseExploratoria.ipynb".
The first task is:
Parte 2: Estatísticas Básicas das Variáveis
Nessa parte do notebook vamos aprender a filtrar a base de dados para calcular estatísticas básicas necessárias para a análise exploratória.
 (2a) Contagem de frequência
A contagem de frequência é realizada de forma similar ao exercício de contagem de palavras. Primeiro mapeamos a variável de interesse. Como exemplo vamos gerar uma lista da quantidade total de cada tipo de crime (Category).
Can you write Python code for it?

# EXERCICIO

CatCountRDD = (CrimeHeadlessRDD
               .map(lambda x: (x.Category, 1))
               .reduceByKey(lambda x, y : x+y)
               )
catCount = sorted(CatCountRDD.collect(), key=lambda x: -x[1])
print catCount

assert catCount[0][1]==174900, 'valores incorretos'
print "OK"


I am working on the file "Atividade02/Lab/Lab3_AnaliseExploratoria.ipynb".
The first task is:
Parte 3: Plotagem de Gráficos
Nessa parte do notebook vamos aprender a manipular os dados para gerar listas de valores a serem utilizados na plotagem de gráficos.
Para a plotagem de gráficos vamos utilizar o matplotlib que já vem por padrão na maioria das distribuições do Python (ex.: Anaconda). Outras bibliotecas alternativas interessantes são: Seaborn e Bokeh.
 (3a) Gráfico de Barras
O gráfico de barras é utilizado quando queremos comparar dados entre categorias diferentes de uma variável categórica. Como exemplo, vamos contabilizar o número médio de crimes diários por região.
Vamos primeiro criar a RDD totalDatesRDD que contém a lista de dias únicos, computaremos o total de dias com o comando count() armazenando na variável totalDays. Não se esqueça de converter o valor para float.
Em seguida, crie o RDD avgCrimesRegionRDD que utiliza a RDD RegionCountRDD para calcular a média de crimes por região.
Utilizando o comando zip() do Python é possível descompactar um dicionário em duas variáveis, uma com as chaves e outra com os valores. Utilizaremos essas variáveis para a plotagem do gráfico.
Can you write Python code for it?

#Insert a graphic inline
%matplotlib inline

import matplotlib.pyplot as plt

# Dates,Category,Descript,DayOfWeek,PdDistrict,Resolution,Address,COORD

# Lambda para converter um datetime em `Dia-Mes-Ano`
day2str = lambda x: '{}-{}-{}'.format(x.day,x.month,x.year)

totalDatesRDD = (CrimeHeadlessRDD
                 .map(lambda rec: (day2str(rec.Dates),1))
                 .reduceByKey(lambda x,y: x)
                 )

totalDays = float(totalDatesRDD.count())

avgCrimesRegionRDD = (RegionCountRDD
                      .map(lambda rec: (rec[0],rec[1]/totalDays))
                     )

Xticks,Y = zip(*avgCrimesRegionRDD.collectAsMap().items())
indices = np.arange(len(Xticks))
width = 0.35

fig = plt.figure(figsize=(8,4.2), facecolor='white', edgecolor='white')
plt.bar(indices,Y, width)
plt.grid(b=True, which='major', axis='y')
plt.xticks(indices+width/2., Xticks, rotation=17 )
plt.ylabel('Number of crimes')
plt.xlabel('Region')
pass


I am working on the file "Atividade02/Lab/Lab3_AnaliseExploratoria.ipynb".
The first task is:
Quando temos subcategorias de interesse, podemos plotar através de um gráfico de barras empilhado. Vamos plotar o conteúdo da variável RegionAvg.
Primeiro passo é criar um dicionário Y em que a chave é o dia da semana e o valor é uma np.array contendo a média de cada região para aquele dia.
Em seguida precisamos criar uma matriz Bottom que determina qual é o início de cada uma das barras. O início da barra do dia i deve ser o final da barra do dia i-1.
Com isso calculado podemos gerar um plot por dia com o parâmetro bottom correspondente ao vetor Bottom daquele dia.
Can you write Python code for it?

# Dias da semana como referência
Day = ['Sunday','Monday','Tuesday','Wednesday','Thursday','Friday','Saturday']

# Uma cor para cada dia
Color = ['r','b','g','y','c','k','purple']

# Dicionário (dia, array de médias)
Y = {}
for day in Day:
    Y[day] = np.array([RegionAvg[day][x] for x in Xticks])

# Matriz dias x regiões
Bottom = np.zeros( (len(Day),len(Xticks)) )
for i in range(1,len(Day)):
    Bottom[i,:] = Bottom[i-1,:]+Y[Day[i-1]]

indices = np.arange(len(Xticks))
width = 0.35

fig = plt.figure(figsize=(8,4.2), facecolor='white', edgecolor='white')

# Gera uma lista de plots, um para cada dia
plots = [plt.bar(indices,Y[Day[i]], width, color=Color[i], bottom=Bottom[i]) for i in range(len(Day))]

plt.legend( [p[0] for p in plots], Day,loc='center left', bbox_to_anchor=(1, 0.5) )

plt.grid(b=True, which='major', axis='y')
plt.xticks(indices+width/2., Xticks, rotation=17 )
plt.ylabel('Number of crimes')
plt.xlabel('Region')
pass


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
(1b) Usando LabeledPoint
Na MLlib, bases de dados rotuladas devem ser armazenadas usando o objeto LabeledPoint.  Escreva a função parsePoint que recebe como entrada uma amostra de dados, transforma os dados usandoo comando unicode.split, e retorna um LabeledPoint.
Aplique essa função na variável samplePoints da célula anterior e imprima os atributos e rótulo utilizando os atributos LabeledPoint.features e LabeledPoint.label. Finalmente, calcule o número de atributos nessa base de dados.
Can you write Python code for it?

from pyspark.mllib.regression import LabeledPoint
import numpy as np

# Here is a sample raw data point:
# '2001.0,0.884,0.610,0.600,0.474,0.247,0.357,0.344,0.33,0.600,0.425,0.60,0.419'
# In this raw data point, 2001.0 is the label, and the remaining values are features

# EXERCICIO
def parsePoint(line):
    """Converts a comma separated unicode string into a `LabeledPoint`.

    Args:
        line (unicode): Comma separated unicode string where the first element is the label and the
            remaining elements are features.

    Returns:
        LabeledPoint: The line is converted into a `LabeledPoint`, which consists of a label and
            features.
    """
    Point = line.split(",")
    return LabeledPoint(Point[0], Point[1:])

parsedSamplePoints = map(parsePoint,samplePoints)
firstPointFeatures = parsedSamplePoints[0].features
firstPointLabel = parsedSamplePoints[0].label
print firstPointFeatures, firstPointLabel

d = len(firstPointFeatures)
print d

# TEST Using LabeledPoint (1b)
Test.assertTrue(isinstance(firstPointLabel, float), 'label must be a float')
expectedX0 = [0.8841,0.6105,0.6005,0.4747,0.2472,0.3573,0.3441,0.3396,0.6009,0.4257,0.6049,0.4192]
Test.assertTrue(np.allclose(expectedX0, firstPointFeatures, 1e-4, 1e-4),
                'incorrect features for firstPointFeatures')
Test.assertTrue(np.allclose(2001.0, firstPointLabel), 'incorrect label for firstPointLabel')
Test.assertTrue(d == 12, 'incorrect number of features')


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
(1c) Deslocando os rótulos
Para melhor visualizar as soluções obtidas, calcular o erro de predição e visualizar a relação dos atributos com os rótulos, costuma-se deslocar os rótulos para iniciarem em zero.
Dessa forma vamos verificar qual é a faixa de valores dos rótulos e, em seguida, subtrair os rótulos pelo menor valor encontrado. Em alguns casos também pode ser interessante normalizar tais valores dividindo pelo valor máximo dos rótulos.
Can you write Python code for it?

# EXERCICIO
parsedDataInit = rawData.map(lambda x: parsePoint(x))
onlyLabels = parsedDataInit.map(lambda x: x.label)
minYear = onlyLabels.min()
maxYear = onlyLabels.max()
print maxYear, minYear

# TEST Find the range (1c)
Test.assertEquals(len(parsedDataInit.take(1)[0].features), 12,
                  'unexpected number of features in sample point')
sumFeatTwo = parsedDataInit.map(lambda lp: lp.features[2]).sum()
Test.assertTrue(np.allclose(sumFeatTwo, 3158.96224351), 'parsedDataInit has unexpected values')
yearRange = maxYear - minYear
Test.assertTrue(yearRange == 89, 'incorrect range for minYear to maxYear')

# Debug
parsedDataInit.take(1)

# EXERCICIO
parsedData = parsedDataInit.map(lambda x: LabeledPoint(x.label - minYear, x.features))

# Should be a LabeledPoint
print type(parsedData.take(1)[0])
# View the first point
print '\n{0}'.format(parsedData.take(1))

# TEST Shift labels (1d)
oldSampleFeatures = parsedDataInit.take(1)[0].features
newSampleFeatures = parsedData.take(1)[0].features
Test.assertTrue(np.allclose(oldSampleFeatures, newSampleFeatures),
                'new features do not match old features')
sumFeatTwo = parsedData.map(lambda lp: lp.features[2]).sum()
Test.assertTrue(np.allclose(sumFeatTwo, 3158.96224351), 'parsedData has unexpected values')
minYearNew = parsedData.map(lambda lp: lp.label).min()
maxYearNew = parsedData.map(lambda lp: lp.label).max()
Test.assertTrue(minYearNew == 0, 'incorrect min year in shifted data')
Test.assertTrue(maxYearNew == 89, 'incorrect max year in shifted data')


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
(1d) Conjuntos de treino, validação e teste
Como próximo passo, vamos dividir nossa base de dados em conjunto de treino, validação e teste conforme discutido em sala de aula. Use o método randomSplit method com os pesos (weights) e a semente aleatória (seed) especificados na célula abaixo parar criar a divisão das bases. Em seguida, utilizando o método cache() faça o pré-armazenamento da base processada.
Esse comando faz o processamento da base através das transformações e armazena em um novo RDD que pode ficar armazenado em memória, se couber, ou em um arquivo temporário.
Can you write Python code for it?

# EXERCICIO
weights = [.8, .1, .1]
seed = 42
parsedTrainData, parsedValData, parsedTestData = parsedData.randomSplit(weights, seed)
parsedTrainData.cache()
parsedValData.cache()
parsedTestData.cache()
nTrain = parsedTrainData.count()
nVal = parsedValData.count()
nTest = parsedTestData.count()

print nTrain, nVal, nTest, nTrain + nVal + nTest
print parsedData.count()

# TEST Training, validation, and test sets (1e)
Test.assertEquals(parsedTrainData.getNumPartitions(), numPartitions,
                  'parsedTrainData has wrong number of partitions')
Test.assertEquals(parsedValData.getNumPartitions(), numPartitions,
                  'parsedValData has wrong number of partitions')
Test.assertEquals(parsedTestData.getNumPartitions(), numPartitions,
                  'parsedTestData has wrong number of partitions')
Test.assertEquals(len(parsedTrainData.take(1)[0].features), 12,
                  'parsedTrainData has wrong number of features')
sumFeatTwo = (parsedTrainData
              .map(lambda lp: lp.features[2])
              .sum())
sumFeatThree = (parsedValData
                .map(lambda lp: lp.features[3])
                .reduce(lambda x, y: x + y))
sumFeatFour = (parsedTestData
               .map(lambda lp: lp.features[4])
               .reduce(lambda x, y: x + y))
Test.assertTrue(np.allclose([sumFeatTwo, sumFeatThree, sumFeatFour],
                            2526.87757656, 297.340394298, 184.235876654),
                'parsed Train, Val, Test data has unexpected values')
Test.assertTrue(nTrain + nVal + nTest == 6724, 'unexpected Train, Val, Test data set size')
Test.assertEquals(nTrain, 5371, 'unexpected value for nTrain')
Test.assertEquals(nVal, 682, 'unexpected value for nVal')
Test.assertEquals(nTest, 671, 'unexpected value for nTest')


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
(2c) RMSE do baseline para os conjuntos de treino, validação e teste
Vamos calcular o RMSE para nossa baseline. Primeiro crie uma RDD de (rótulo, predição) para cada conjunto, e então chame a função calcRMSE.
Can you write Python code for it?

#Debug
parsedTrainData.take(1)

# EXERCICIO -> (rótulo, predição)
labelsAndPredsTrain = parsedTrainData.map(lambda x:(x.label, averageTrainYear))
rmseTrainBase = calcRMSE(labelsAndPredsTrain)

labelsAndPredsVal = parsedValData.map(lambda x:(x.label, averageTrainYear))
rmseValBase = calcRMSE(labelsAndPredsVal)

labelsAndPredsTest = parsedTestData.map(lambda x:(x.label, averageTrainYear))
rmseTestBase = calcRMSE(labelsAndPredsTest)

print 'Baseline Train RMSE = {0:.3f}'.format(rmseTrainBase)
print 'Baseline Validation RMSE = {0:.3f}'.format(rmseValBase)
print 'Baseline Test RMSE = {0:.3f}'.format(rmseTestBase)

# TEST Training, validation and test RMSE (2c)
Test.assertTrue(np.allclose([rmseTrainBase, rmseValBase, rmseTestBase],
                            [21.305869, 21.586452, 22.136957]), 'incorrect RMSE value')


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
(3b) Use os pesos para fazer a predição
Agora, implemente a função  getLabeledPredictions que recebe como parâmetro o conjunto de pesos e um LabeledPoint e retorna uma tupla (rótulo, predição). Lembre-se que podemos predizer um rótulo calculando o produto interno dos pesos com os atributos.
Can you write Python code for it?

# EXERCICIO
def getLabeledPrediction(weights, observation):
    """Calculates predictions and returns a (label, prediction) tuple.

    Note:
        The labels should remain unchanged as we'll use this information to calculate prediction
        error later.

    Args:
        weights (np.ndarray): An array with one weight for each features in `trainData`.
        observation (LabeledPoint): A `LabeledPoint` that contain the correct label and the
            features for the data point.

    Returns:
        tuple: A (label, prediction) tuple.
    """
    return ( observation.label, weights.dot(observation.features) )

weights = np.array([1.0, 1.5])
predictionExample = sc.parallelize([LabeledPoint(2, np.array([1.0, .5])),
                                    LabeledPoint(1.5, np.array([.5, .5]))])
labelsAndPredsExample = predictionExample.map(lambda lp: getLabeledPrediction(weights, lp))
print labelsAndPredsExample.collect()

# TEST Use weights to make predictions (3b)
Test.assertEquals(labelsAndPredsExample.collect(), [(2.0, 1.75), (1.5, 1.25)],
                  'incorrect definition for getLabeledPredictions')


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
(4d) Grid search
Já estamos superando o baseline em pelo menos dois anos na média, vamos ver se encontramos um conjunto de parâmetros melhor.  Faça um grid search para encontrar um bom parâmetro de regularização.  Tente valores para regParam dentro do conjunto 1e-10, 1e-5, e 1.
Can you write Python code for it?

# EXERCICIO
bestRMSE = rmseValLR1
bestRegParam = reg
bestModel = firstModel

numIters = 500
alpha = 1.0
miniBatchFrac = 1.0
for reg in [1e-10, 1e-5, 1]:
    model = LinearRegressionWithSGD.train(parsedTrainData, numIters, alpha,
                                          miniBatchFrac, regParam=reg,
                                          regType='l2', intercept=True)
    labelsAndPreds = parsedValData.map(lambda x: (x.label, model.predict(x.features)))
    rmseValGrid = calcRMSE(labelsAndPreds)
    print rmseValGrid

    if rmseValGrid < bestRMSE:
        bestRMSE = rmseValGrid
        bestRegParam = reg
        bestModel = model
rmseValLRGrid = bestRMSE

print ('Validation RMSE:\n\tBaseline = {0:.3f}\n\tLR0 = {1:.3f}\n\tLR1 = {2:.3f}\n' +
       '\tLRGrid = {3:.3f}').format(rmseValBase, rmseValLR0, rmseValLR1, rmseValLRGrid)

# TEST Grid search (4d)
Test.assertTrue(np.allclose(17.017170, rmseValLRGrid), 'incorrect value for rmseValLRGrid')


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
(4e) Grid Search para o valor de alfa e número de iterações
Agora, vamos verificar diferentes valores para alfa e número de iterações para perceber o impacto desses parâmetros em nosso modelo. Especificamente tente os valores  1e-5 e 10 para alpha e os valores 500 e 5 para número de iterações. Avalie todos os modelos no conjunto de valdação.  Reparem que com um valor baixo de alpha, o algoritmo necessita de muito mais iterações para convergir ao ótimo, enquanto um valor muito alto para alpha, pode fazer com que o algoritmo não encontre uma solução.
Can you write Python code for it?

# EXERCICIO
reg = bestRegParam
modelRMSEs = []

for alpha in [1e-5, 10]:
    for numIters in [500, 5]:
        model = LinearRegressionWithSGD.train(parsedTrainData, numIters, alpha,
                                              miniBatchFrac, regParam=reg,
                                              regType='l2', intercept=True)
        labelsAndPreds = parsedValData.map(lambda lp: (lp.label, model.predict(lp.features)))
        rmseVal = calcRMSE(labelsAndPreds)
        print 'alpha = {0:.0e}, numIters = {1}, RMSE = {2:.3f}'.format(alpha, numIters, rmseVal)
        modelRMSEs.append(rmseVal)

# TEST Vary alpha and the number of iterations (4e)
expectedResults = sorted([56.969705, 56.892949, 355124752.221221])
Test.assertTrue(np.allclose(sorted(modelRMSEs)[:3], expectedResults), 'incorrect value for modelRMSEs')


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
(5b) Construindo um novo modelo
Agora construa um novo modelo usando esses novos atributos. Repare que idealmente, com novos atributos, você deve realizar um novo Grid Search para determinar os novos parâmetros ótimos, uma vez que os parâmetros do modelo anterior não necessariamente funcionarão aqui.
Para este exercício, os parâmetros já foram otimizados.
Can you write Python code for it?

# EXERCICIO
numIters = 500
alpha = 1.0
miniBatchFrac = 1.0
reg = 1e-10

modelInteract = LinearRegressionWithSGD.train(trainDataInteract, numIters, alpha,
                                              miniBatchFrac, regParam=reg,
                                              regType='l2', intercept=True)
labelsAndPredsInteract = valDataInteract.<COMPLETAR>
rmseValInteract = calcRMSE(labelsAndPredsInteract)

print ('Validation RMSE:\n\tBaseline = {0:.3f}\n\tLR0 = {1:.3f}\n\tLR1 = {2:.3f}\n\tLRGrid = ' +
       '{3:.3f}\n\tLRInteract = {4:.3f}').format(rmseValBase, rmseValLR0, rmseValLR1,
                                                 rmseValLRGrid, rmseValInteract)

# TEST Build interaction model (5b)
Test.assertTrue(np.allclose(rmseValInteract, 15.6894664683), 'incorrect value for rmseValInteract')


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
(5c) Avaliando o modelo de interação
Finalmente, temos que o melhor modelo para o conjunto de validação foi o modelo de interação. Na prática esse seria o modelo escolhido para aplicar nos modelos não-rotulados. Vamos ver como essa escolha se sairia utilizand a base de teste nesse modelo e no baseline.
Can you write Python code for it?

# EXERCICIO
labelsAndPredsTest = testDataInteract.<COMPLETAR>
rmseTestInteract = calcRMSE(labelsAndPredsTest)

print ('Test RMSE:\n\tBaseline = {0:.3f}\n\tLRInteract = {1:.3f}'
       .format(rmseTestBase, rmseTestInteract))

# TEST Evaluate interaction model on test data (5c)
Test.assertTrue(np.allclose(rmseTestInteract, 16.3272040537),
                'incorrect value for rmseTestInteract')


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
Part 2: Criando o modelo de baseline
(2a) Rótulo médio
O baseline é útil para verificarmos que nosso modelo de regressão está funcionando. Ele deve ser um modelo bem simples que qualquer algoritmo possa fazer melhor.
Um baseline muito utilizado é fazer a mesma predição independente dos dados analisados utilizando o rótulo médio do conjunto de treino. Calcule a média dos rótulos deslocados para a base de treino, utilizaremos esse valor posteriormente para comparar o erro de predição.  Use um método apropriado para essa tarefa, consulte o RDD API.
Can you write Python code for it?

# EXERCICIO
averageTrainYear = (parsedTrainData
                    .map(lambda x: x.label)
                    .mean()
                   )
print averageTrainYear

# TEST Average label (2a)
Test.assertTrue(np.allclose(averageTrainYear, 53.9316700801),
                'incorrect value for averageTrainYear')


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
Part 4: Treino utilizando MLlib e Busca em Grade (Grid Search)
(4a) LinearRegressionWithSGD
Nosso teste inicial já conseguiu obter um desempenho melhor que o baseline, mas vamos ver se conseguimos fazer melhor introduzindo a ordenada de origem da reta além de outros ajustes no algoritmo.  MLlib LinearRegressionWithSGD implementa o mesmo algoritmo da parte (3b), mas de forma mais eficiente para o contexto distribuído e com várias funcionalidades adicionais.
Primeiro utilize a função LinearRegressionWithSGD para treinar um modelo com regularização L2 (Ridge) e com a ordenada de origem. Esse método retorna um LinearRegressionModel.
Em seguida, use os atributos weights e  intercept para imprimir o modelo encontrado.
Can you write Python code for it?

from pyspark.mllib.regression import LinearRegressionWithSGD
# Values to use when training the linear regression model
numIters = 500  # iterations
alpha = 1.0  # step
miniBatchFrac = 1.0  # miniBatchFraction
reg = 1e-1  # regParam
regType = 'l2'  # regType
useIntercept = True  # intercept

# EXERCICIO
firstModel = LinearRegressionWithSGD.train(parsedTrainData, iterations = numIters, step = alpha, miniBatchFraction = 1.0,
                                          regParam=reg,regType=regType, intercept=useIntercept)

# weightsLR1 stores the model weights; interceptLR1 stores the model intercept
weightsLR1 = firstModel.weights
interceptLR1 = firstModel.intercept
print weightsLR1, interceptLR1

# TEST LinearRegressionWithSGD (4a)
expectedIntercept = 13.3335907631
expectedWeights = [16.682292427, 14.7439059559, -0.0935105608897, 6.22080088829, 4.01454261926, -3.30214858535,
                   11.0403027232, 2.67190962854, 7.18925791279, 4.46093254586, 8.14950409475, 2.75135810882]
Test.assertTrue(np.allclose(interceptLR1, expectedIntercept), 'incorrect value for interceptLR1')
Test.assertTrue(np.allclose(weightsLR1, expectedWeights), 'incorrect value for weightsLR1')


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
Parte 3: Treinando e avaliando o modelo de regressão linear
 (3a) Gradiente do erro
Vamos implementar a regressão linear através do gradiente descendente.
Lembrando que para atualizar o peso da regressão linear fazemos: $$ \scriptsize \mathbf{w}_{i+1} = \mathbf{w}_i - \alpha_i \sum_j (\mathbf{w}_i^\top\mathbf{x}_j  - y_j) \mathbf{x}_j \,.$$ onde $ \scriptsize i $ é a iteração do algoritmo, e $ \scriptsize j $ é o objeto sendo observado no momento.
Primeiro, implemente uma função que calcula esse gradiente do erro para certo objeto: $ \scriptsize (\mathbf{w}^\top \mathbf{x} - y) \mathbf{x} \, ,$ e teste a função em dois exemplos. Use o método DenseVector dot para representar a lista de atributos (ele tem funcionalidade parecida com o np.array()).
Can you write Python code for it?

from pyspark.mllib.linalg import DenseVector

# EXERCICIO
def gradientSummand(weights, lp):
    """Calculates the gradient summand for a given weight and `LabeledPoint`.

    Note:
        `DenseVector` behaves similarly to a `numpy.ndarray` and they can be used interchangably
        within this function.  For example, they both implement the `dot` method.

    Args:
        weights (DenseVector): An array of model weights (betas).
        lp (LabeledPoint): The `LabeledPoint` for a single observation.

    Returns:
        DenseVector: An array of values the same length as `weights`.  The gradient summand.
    """
    return (weights.dot(lp.features) - lp.label) * lp.features

exampleW = DenseVector([1, 1, 1])
exampleLP = LabeledPoint(2.0, [3, 1, 4])

summandOne = gradientSummand(exampleW, exampleLP)
print summandOne

exampleW = DenseVector([.24, 1.2, -1.4])
exampleLP = LabeledPoint(3.0, [-1.4, 4.2, 2.1])
summandTwo = gradientSummand(exampleW, exampleLP)
print summandTwo

# TEST Gradient summand (3a)
Test.assertTrue(np.allclose(summandOne, [18., 6., 24.]), 'incorrect value for summandOne')
Test.assertTrue(np.allclose(summandTwo, [1.7304,-5.1912,-2.5956]), 'incorrect value for summandTwo')


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
Parte 5: Adicionando atributos não-lineares
 (5a) Interações par a par
Conforme mencionado em aula, os modelos de regressão linear conseguem capturar as relações lineares entre os atributos e o rótulo. Porém, em muitos casos a relação entre eles é não-linear.
Uma forma de resolver tal problema é criando mais atributos com características não-lineares, como por exemplo a expansão quadrática dos atributos originais. Escreva uma função twoWayInteractions que recebe um  LabeledPoint e gera um novo LabeledPoint que contém os atributos antigos e as interações par a par entre eles. Note que um objeto com 3 atributos terá nove interações ( $ \scriptsize 3^2 $ ) par a par.
Para facilitar, utilize o método itertools.product para gerar tuplas para cada possível interação. Utilize também np.hstack para concatenar dois vetores.
Can you write Python code for it?

# EXERCICIO
import itertools

def twoWayInteractions(lp):
    """Creates a new `LabeledPoint` that includes two-way interactions.

    Note:
        For features [x, y] the two-way interactions would be [x^2, x*y, y*x, y^2] and these
        would be appended to the original [x, y] feature list.

    Args:
        lp (LabeledPoint): The label and features for this observation.

    Returns:
        LabeledPoint: The new `LabeledPoint` should have the same label as `lp`.  Its features
            should include the features from `lp` followed by the two-way interaction features.
    """
    newfeats = <COMPLETAR>
    return LabeledPoint(lp.label, <COMPLETAR>)
    #return lp

print twoWayInteractions(LabeledPoint(0.0, [2, 3]))

# Transform the existing train, validation, and test sets to include two-way interactions.
trainDataInteract = parsedTrainData.map(twoWayInteractions)
valDataInteract = parsedValData.map(twoWayInteractions)
testDataInteract = parsedTestData.map(twoWayInteractions)

# TEST Add two-way interactions (5a)
twoWayExample = twoWayInteractions(LabeledPoint(0.0, [2, 3]))
Test.assertTrue(np.allclose(sorted(twoWayExample.features),
                            sorted([2.0, 3.0, 4.0, 6.0, 6.0, 9.0])),
                'incorrect features generatedBy twoWayInteractions')
twoWayPoint = twoWayInteractions(LabeledPoint(1.0, [1, 2, 3]))
Test.assertTrue(np.allclose(sorted(twoWayPoint.features),
                            sorted([1.0,2.0,3.0,1.0,2.0,3.0,2.0,4.0,6.0,3.0,6.0,9.0])),
                'incorrect features generated by twoWayInteractions')
Test.assertEquals(twoWayPoint.label, 1.0, 'incorrect label generated by twoWayInteractions')
Test.assertTrue(np.allclose(sum(trainDataInteract.take(1)[0].features), 40.821870576035529),
                'incorrect features in trainDataInteract')
Test.assertTrue(np.allclose(sum(valDataInteract.take(1)[0].features), 45.457719932695696),
                'incorrect features in...


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
Visualização 1: Atributos
A próxima célula mostra uma forma de visualizar os atributos através de um mapa de calor. Nesse mapa mostramos os 50 primeiros objetos e seus atributos representados por tons de cinza, sendo o branco representando o valor 0 e o preto representando o valor 1.
Esse tipo de visualização ajuda a perceber a variação dos valores dos atributos.
Can you write Python code for it?

#insert a graphic inline
%matplotlib inline

import matplotlib.pyplot as plt
import matplotlib.cm as cm

sampleMorePoints = rawData.take(50)

parsedSampleMorePoints = map(parsePoint, sampleMorePoints)
dataValues = map(lambda lp: lp.features.toArray(), parsedSampleMorePoints)
#print dataValues

def preparePlot(xticks, yticks, figsize=(10.5, 6), hideLabels=False, gridColor='#999999',
                gridWidth=1.0):
    """Template for generating the plot layout."""
    plt.close()
    fig, ax = plt.subplots(figsize=figsize, facecolor='white', edgecolor='white')
    ax.axes.tick_params(labelcolor='#999999', labelsize='10')
    for axis, ticks in [(ax.get_xaxis(), xticks), (ax.get_yaxis(), yticks)]:
        axis.set_ticks_position('none')
        axis.set_ticks(ticks)
        axis.label.set_color('#999999')
        if hideLabels: axis.set_ticklabels([])
    plt.grid(color=gridColor, linewidth=gridWidth, linestyle='-')
    map(lambda position: ax.spines[position].set_visible(False), ['bottom', 'top', 'left', 'right'])
    return fig, ax

# generate layout and plot
fig, ax = preparePlot(np.arange(.5, 11, 1), np.arange(.5, 49, 1), figsize=(8,7), hideLabels=True,
                      gridColor='#eeeeee', gridWidth=1.1)
image = plt.imshow(dataValues,interpolation='nearest', aspect='auto', cmap=cm.Greys)
for x, y, s in zip(np.arange(-.125, 12, 1), np.repeat(-.75, 12), [str(x) for x in range(12)]):
    plt.text(x, y, s, color='#999999', size='10')
plt.text(4.7, -3, 'Feature', color='#999999', size='11'), ax.set_ylabel('Observation')
pass


I am working on the file "Atividade03/Lab4a_regressao_linear.ipynb".
The first task is:
Visualização 2: Predição vs. real
Vamos visualizar as predições no conjunto de validação. Os gráficos de dispersão abaixo plotam os pontos com a coordenada X sendo o valor predito pelo modelo e a coordenada Y o valor real do rótulo.
O primeiro gráfico mostra a situação ideal, um modelo que acerta todos os rótulos. O segundo gráfico mostra o desempenho do modelo baseline. As cores dos pontos representam o erro quadrático daquela predição, quanto mais próxima do laranja, maior o erro.
Can you write Python code for it?

from matplotlib.colors import ListedColormap, Normalize
from matplotlib.cm import get_cmap
cmap = get_cmap('YlOrRd')
norm = Normalize()

actual = np.asarray(parsedValData
                    .map(lambda lp: lp.label)
                    .collect())
error = np.asarray(parsedValData
                   .map(lambda lp: (lp.label, lp.label))
                   .map(lambda (l, p): squaredError(l, p))
                   .collect())
clrs = cmap(np.asarray(norm(error)))[:,0:3]

fig, ax = preparePlot(np.arange(0, 100, 20), np.arange(0, 100, 20))
plt.scatter(actual, actual, s=14**2, c=clrs, edgecolors='#888888', alpha=0.75, linewidths=0.5)
ax.set_xlabel('Predicted'), ax.set_ylabel('Actual')
pass

predictions = np.asarray(parsedValData
                         .map(lambda lp: averageTrainYear)
                         .collect())
error = np.asarray(parsedValData
                   .map(lambda lp: (lp.label, averageTrainYear))
                   .map(lambda (l, p): squaredError(l, p))
                   .collect())
norm = Normalize()
clrs = cmap(np.asarray(norm(error)))[:,0:3]

fig, ax = preparePlot(np.arange(53.0, 55.0, 0.5), np.arange(0, 100, 20))
ax.set_xlim(53, 55)
plt.scatter(predictions, actual, s=14**2, c=clrs, edgecolors='#888888', alpha=0.75, linewidths=0.3)
ax.set_xlabel('Predicted'), ax.set_ylabel('Actual')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(1b) Vetores Esparsos
Pontos de dados categóricos geralmente apresentam um pequeno conjunto de OHE não-nulos relativo ao total de possíveis atributos. Tirando proveito dessa propriedade podemos representar nossos dados como vetores esparsos, economizando espaço de armazenamento e cálculos computacionais.
No próximo exercício transforme os vetores com nome precedidos de Dense para vetores esparsos. Utilize a classe SparseVector para representá-los e verifique que ambas as representações retornam o mesmo resultado nos cálculos dos produtos interno.
Use SparseVector(tamanho, *args) para criar um novo vetor esparso onde tamanho é o tamanho do vetor e args pode ser um dicionário, uma lista de tuplas (índice, valor) ou duas arrays separadas de índices e valores ordenados por índice.
Can you write Python code for it?

import numpy as np
from pyspark.mllib.linalg import SparseVector

# EXERCICIO
aDense = np.array([0., 3., 0., 4.])
aSparse = SparseVector(4,[(1,3),(3,4)])

bDense = np.array([0., 0., 0., 1.])
bSparse = SparseVector(4,[(3,1)])

w = np.array([0.4, 3.1, -1.4, -.5])
print aDense.dot(w)
print aSparse.dot(w)
print bDense.dot(w)
print bSparse.dot(w)

# TEST Sparse Vectors (1b)
Test.assertTrue(isinstance(aSparse, SparseVector), 'aSparse needs to be an instance of SparseVector')
Test.assertTrue(isinstance(bSparse, SparseVector), 'aSparse needs to be an instance of SparseVector')
Test.assertTrue(aDense.dot(w) == aSparse.dot(w),
                'dot product of aDense and w should equal dot product of aSparse and w')
Test.assertTrue(bDense.dot(w) == bSparse.dot(w),
                'dot product of bDense and w should equal dot product of bSparse and w')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(1e) Aplicar OHE em uma base de dados
Finalmente, use a função da parte (1d) para criar atributos OHE para todos os 3 objetos da base de dados artificial.
Can you write Python code for it?

# EXERCICIO
sampleOHEData = sampleDataRDD.map(lambda x: oneHotEncoding(x,sampleOHEDictManual, numSampleOHEFeats))
print sampleOHEData.collect()

# TEST Apply OHE to a dataset (1e)
sampleOHEDataValues = sampleOHEData.collect()
Test.assertTrue(len(sampleOHEDataValues) == 3, 'sampleOHEData should have three elements')
Test.assertEquals(sampleOHEDataValues[0], SparseVector(7, {2: 1.0, 3: 1.0}),
                  'incorrect OHE for first sample')
Test.assertEquals(sampleOHEDataValues[1], SparseVector(7, {1: 1.0, 4: 1.0, 5: 1.0}),
                  'incorrect OHE for second sample')
Test.assertEquals(sampleOHEDataValues[2], SparseVector(7, {0: 1.0, 3: 1.0, 6: 1.0}),
                  'incorrect OHE for third sample')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(2b) Dicionário OHE de atributos únicos
Agora, vamos criar um RDD de tuplas para cada (IDatributo, categoria) em sampleDistinctFeats. A chave da tupla é a própria tupla original, e o valor será um inteiro variando de 0 até número de tuplas - 1.
Em seguida, converta essa RDD em um dicionário, utilizando o comando collectAsMap.
Use o comando  zipWithIndex seguido de collectAsMap.
Can you write Python code for it?

# EXERCICIO
sampleOHEDict = (sampleDistinctFeats
                           .zipWithIndex()
                           .collectAsMap())
print sampleOHEDict

# TEST OHE Dictionary from distinct features (2b)
Test.assertEquals(sorted(sampleOHEDict.keys()),
                  [(0, 'bear'), (0, 'cat'), (0, 'mouse'), (1, 'black'),
                   (1, 'tabby'), (2, 'mouse'), (2, 'salmon')],
                  'sampleOHEDict has unexpected keys')
Test.assertEquals(sorted(sampleOHEDict.values()), range(7), 'sampleOHEDict has unexpected values')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(2c) Criação automática do dicionário OHE
Agora use os códigos dos exercícios anteriores para criar uma função que retorna um dicionário OHE a partir dos atributos categóricos de uma base de dados.
Can you write Python code for it?

# EXERCICIO
def createOneHotDict(inputData):
    """Creates a one-hot-encoder dictionary based on the input data.

    Args:
        inputData (RDD of lists of (int, str)): An RDD of observations where each observation is
            made up of a list of (featureID, value) tuples.

    Returns:
        dict: A dictionary where the keys are (featureID, value) tuples and map to values that are
            unique integers.
    """
    return (inputData
             .flatMap(lambda x: x)
             .distinct()
             .zipWithIndex()
             .collectAsMap()
             )

sampleOHEDictAuto = createOneHotDict(sampleDataRDD)
print sampleOHEDictAuto

# TEST Automated creation of an OHE dictionary (2c)
Test.assertEquals(sorted(sampleOHEDictAuto.keys()),
                  [(0, 'bear'), (0, 'cat'), (0, 'mouse'), (1, 'black'),
                   (1, 'tabby'), (2, 'mouse'), (2, 'salmon')],
                  'sampleOHEDictAuto has unexpected keys')
Test.assertEquals(sorted(sampleOHEDictAuto.values()), range(7),
                  'sampleOHEDictAuto has unexpected values')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(3a) Carregando e dividindo os dados
Da mesma forma que no notebook anterior, vamos dividir os dados entre treinamento, validação e teste. Use o método randomSplit com os pesos (weights) e semente aleatória (seed) especificados para criar os conjuntos, então faça o cache de cada RDD, pois utilizaremos cada uma delas com frequência durante esse exercício.
Can you write Python code for it?

# EXERCICIO
weights = [.8, .1, .1]
seed = 42
# Use randomSplit with weights and seed
rawTrainData, rawValidationData, rawTestData = rawData.randomSplit(weights, seed)
# Cache the data
rawTrainData.cache()
rawValidationData.cache()
rawTestData.cache()

nTrain = rawTrainData.count()
nVal = rawValidationData.count()
nTest = rawTestData.count()
print nTrain, nVal, nTest, nTrain + nVal + nTest
print rawData.take(1)

# TEST Loading and splitting the data (3a)
Test.assertTrue(all([rawTrainData.is_cached, rawValidationData.is_cached, rawTestData.is_cached]),
                'you must cache the split data')
Test.assertEquals(nTrain, 79911, 'incorrect value for nTrain')
Test.assertEquals(nVal, 10075, 'incorrect value for nVal')
Test.assertEquals(nTest, 10014, 'incorrect value for nTest')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(3b) Extração de atributos
Como próximo passo, crie uma função para ser aplicada em cada objeto do RDD para gerar uma RDD de tuplas (IDatributo, categoria). Ignore o primeiro campo, que é o rótulo e gere uma lista de tuplas para os atributos seguintes. Utilize o comando enumerate para criar essas tuplas.
Can you write Python code for it?

# EXERCICIO
def parsePoint(point):
    """Converts a comma separated string into a list of (featureID, value) tuples.

    Note:
        featureIDs should start at 0 and increase to the number of features - 1.

    Args:
        point (str): A comma separated string where the first value is the label and the rest
            are features.

    Returns:
        list: A list of (featureID, value) tuples.
    """
    seq = []
    for x,y in enumerate(point.split(',')[1:]):
        seq.append((x,y))

    return seq

parsedTrainFeat = rawTrainData.map(parsePoint)

numCategories = (parsedTrainFeat
                 .flatMap(lambda x: x)
                 .distinct()
                 .map(lambda x: (x[0], 1))
                 .reduceByKey(lambda x, y: x + y)
                 .sortByKey()
                 .collect()
                )

print numCategories[2][1]

# TEST Extract features (3b)
Test.assertEquals(numCategories[2][1], 855, 'incorrect implementation of parsePoint')
Test.assertEquals(numCategories[32][1], 4, 'incorrect implementation of parsePoint')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(3c) Crie o dicionário de OHE dessa base de dados
Note que a função parsePoint retorna um objeto em forma de lista (IDatributo, categoria), que é o mesmo formato utilizado pela função createOneHotDict. Utilize o RDD parsedTrainFeat para criar um dicionário OHE.
Can you write Python code for it?

# EXERCICIO
ctrOHEDict = createOneHotDict(parsedTrainFeat)
numCtrOHEFeats = len(ctrOHEDict.keys())
print numCtrOHEFeats
print ctrOHEDict[(0, '')]

# TEST Create an OHE dictionary from the dataset (3c)
Test.assertEquals(numCtrOHEFeats, 233286, 'incorrect number of features in ctrOHEDict')
Test.assertTrue((0, '') in ctrOHEDict, 'incorrect features in ctrOHEDict')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(3d) Aplicando OHE à base de dados
Agora vamos usar o dicionário OHE para criar um RDD de objetos LabeledPoint usando atributos OHE. Complete a função parseOHEPoint. Dica: essa função é uma extensão da função parsePoint criada anteriormente e que usa a função oneHotEncoding.
Can you write Python code for it?

from pyspark.mllib.regression import LabeledPoint

# EXERCICIO
def parseOHEPoint(point, OHEDict, numOHEFeats):
    """Obtain the label and feature vector for this raw observation.

    Note:
        You must use the function `oneHotEncoding` in this implementation or later portions
        of this lab may not function as expected.

    Args:
        point (str): A comma separated string where the first value is the label and the rest
            are features.
        OHEDict (dict of (int, str) to int): Mapping of (featureID, value) to unique integer.
        numOHEFeats (int): The number of unique features in the training dataset.

    Returns:
        LabeledPoint: Contains the label for the observation and the one-hot-encoding of the
            raw features based on the provided OHE dictionary.
    """
    <COMPLETAR>

OHETrainData = rawTrainData.map(lambda point: parseOHEPoint(point, ctrOHEDict, numCtrOHEFeats))
OHETrainData.cache()
print OHETrainData.take(1)

# Check that oneHotEncoding function was used in parseOHEPoint
backupOneHot = oneHotEncoding
oneHotEncoding = None
withOneHot = False
try: parseOHEPoint(rawTrainData.take(1)[0], ctrOHEDict, numCtrOHEFeats)
except TypeError: withOneHot = True
oneHotEncoding = backupOneHot

# TEST Apply OHE to the dataset (3d)
numNZ = sum(parsedTrainFeat.map(lambda x: len(x)).take(5))
numNZAlt = sum(OHETrainData.map(lambda lp: len(lp.features.indices)).take(5))
Test.assertEquals(numNZ, numNZAlt, 'incorrect implementation of parseOHEPoint')
Test.assertTrue(withOneHot, 'oneHotEncoding not present in parseOHEPoint')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(4c)  Baseline log loss
Agora, vamos utilizar a função da Parte (4b) para calcular um baseline da métrica de log-loss na nossa base de treino. Uma forma de calcular um baseline é predizer sempre a média dos rótulos observados. Primeiro calcule a média dos rótulos da base e, em seguida, calcule o log-loss médio para a base de treino.
Can you write Python code for it?

# EXERCICIO
# Note that our dataset has a very high click-through rate by design
# In practice click-through rate can be one to two orders of magnitude lower
classOneFracTrain = OHETrainData.<COMPLETAR>
print classOneFracTrain

logLossTrBase = OHETrainData.<COMPLETAR>
print 'Baseline Train Logloss = {0:.3f}\n'.format(logLossTrBase)

# TEST Baseline log loss (4c)
Test.assertTrue(np.allclose(classOneFracTrain, 0.22717773523), 'incorrect value for classOneFracTrain')
Test.assertTrue(np.allclose(logLossTrBase, 0.535844), 'incorrect value for logLossTrBase')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(4d) Probabilidade da Predição
O modelo gerado na Parte (4a) possui um método chamado predict, porém esse método retorna apenas 0's e 1's. Para calcular a probabilidade de um evento, vamos criar uma função getP que recebe como parâmetro o ponto x, o conjunto de pesos w e o intercept.
Calcule o modelo de regressão linear nesse ponto x e aplique a  função sigmoidal $ \scriptsize \sigma(t) = (1+ e^{-t})^{-1} $ para retornar a probabilidade da predição do objeto x.
Can you write Python code for it?

# EXERCICIO
from math import exp #  exp(-t) = e^-t

def getP(x, w, intercept):
    """Calculate the probability for an observation given a set of weights and intercept.

    Note:
        We'll bound our raw prediction between 20 and -20 for numerical purposes.

    Args:
        x (SparseVector): A vector with values of 1.0 for features that exist in this
            observation and 0.0 otherwise.
        w (DenseVector): A vector of weights (betas) for the model.
        intercept (float): The model's intercept.

    Returns:
        float: A probability between 0 and 1.
    """
    # calculate rawPrediction = w.x + intercept
    rawPrediction = <COMPLETAR>

    # Bound the raw prediction value
    rawPrediction = min(rawPrediction, 20)
    rawPrediction = max(rawPrediction, -20)

    # calculate (1+e^-rawPrediction)^-1
    return <COMPLETAR>

trainingPredictions = OHETrainData.<COMPLETAR>

print trainingPredictions.take(5)

# TEST Predicted probability (4d)
Test.assertTrue(np.allclose(trainingPredictions.sum(), 18135.4834348),
                'incorrect value for trainingPredictions')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(4e) Avalie o modelo
Finalmente, crie uma função evaluateResults que calcula o log-loss médio do modelo em uma base de dados. Em seguida, execute essa função na nossa base de treino.
Can you write Python code for it?

# EXERCICIO
def evaluateResults(model, data):
    """Calculates the log loss for the data given the model.

    Args:
        model (LogisticRegressionModel): A trained logistic regression model.
        data (RDD of LabeledPoint): Labels and features for each observation.

    Returns:
        float: Log loss for the data.
    """
    return (data
            .<COMPLETAR>
            .<COMPLETAR>
            .<COMPLETAR>
            )

logLossTrLR0 = evaluateResults(model0, OHETrainData)
print ('OHE Features Train Logloss:\n\tBaseline = {0:.3f}\n\tLogReg = {1:.3f}'
       .format(logLossTrBase, logLossTrLR0))

# TEST Evaluate the model (4e)
Test.assertTrue(np.allclose(logLossTrLR0, 0.456903), 'incorrect value for logLossTrLR0')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(4f) log-loss da validação
Agora aplique o modelo na nossa base de validação e calcule o log-loss médio, compare com o nosso baseline.
Can you write Python code for it?

# EXERCICIO
logLossValBase = OHEValidationData.<COMPLETAR>

logLossValLR0 = evaluateResults(model0, OHEValidationData)
print ('OHE Features Validation Logloss:\n\tBaseline = {0:.3f}\n\tLogReg = {1:.3f}'
       .format(logLossValBase, logLossValLR0))

# TEST Validation log loss (4f)
Test.assertTrue(np.allclose(logLossValBase, 0.527603), 'incorrect value for logLossValBase')
Test.assertTrue(np.allclose(logLossValLR0, 0.456957), 'incorrect value for logLossValLR0')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(5c) Esparsidade
Uma vez que temos  33 mil hashed features contra 233 mil OHE, devemos esperar que os atributos OHE sejam mais esparsos. Verifique essa hipótese computando a esparsidade média do OHE e do hashed features.
Note que se você tem um SparseVector chamado sparse, chamar len(sparse) retornará o total de atributos, e não o número de valores não nulos. SparseVector tem atributos indices e values que contém informações sobre quais atributos são não nulos.
Can you write Python code for it?

# EXERCICIO
def computeSparsity(data, d, n):
    """Calculates the average sparsity for the features in an RDD of LabeledPoints.

    Args:
        data (RDD of LabeledPoint): The LabeledPoints to use in the sparsity calculation.
        d (int): The total number of features.
        n (int): The number of observations in the RDD.

    Returns:
        float: The average of the ratio of features in a point to total features.
    """
    return (data
            .<COMPLETAR>
            .<COMPLETAR>
           )/(d*n*1.)

averageSparsityHash = computeSparsity(hashTrainData, numBucketsCTR, nTrain)
averageSparsityOHE = computeSparsity(OHETrainData, numCtrOHEFeats, nTrain)

print 'Average OHE Sparsity: {0:.7e}'.format(averageSparsityOHE)
print 'Average Hash Sparsity: {0:.7e}'.format(averageSparsityHash)

# TEST Sparsity (5c)
Test.assertTrue(np.allclose(averageSparsityOHE, 1.6717677e-04),
                'incorrect value for averageSparsityOHE')
Test.assertTrue(np.allclose(averageSparsityHash, 1.1805561e-03),
                'incorrect value for averageSparsityHash')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
(5d) Modelo logístico com hashed features
Agora treine um modelo de regressão logística para os hashed features. Execute um grid search para encontrar parâmetros adequados para essa base, avaliando o log-loss no conjunto de validação. Nota: isso pode demorar alguns minutos para terminar. Use stepSizes de 1 e 10 e regParams  de 1e-6 e 1e-3.
Can you write Python code for it?

numIters = 500
regType = 'l2'
includeIntercept = True

# Initialize variables using values from initial model training
bestModel = None
bestLogLoss = 1e10

# EXERCICIO
stepSizes = [1, 10]
regParams = [1e-6, 1e-3]
for stepSize in stepSizes:
    for regParam in regParams:
        model = (<COMPLETAR>)
        logLossVa = <COMPLETAR>
        print ('\tstepSize = {0:.1f}, regParam = {1:.0e}: logloss = {2:.3f}'
               .format(stepSize, regParam, logLossVa))
        if (logLossVa < bestLogLoss):
            bestModel = model
            bestLogLoss = logLossVa

print ('Hashed Features Validation Logloss:\n\tBaseline = {0:.3f}\n\tLogReg = {1:.3f}'
       .format(logLossValBase, bestLogLoss))

# TEST Logistic model with hashed features (5d)
Test.assertTrue(np.allclose(bestLogLoss, 0.4481683608), 'incorrect value for bestLogLoss')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
Part 4: Predição do CTR e avaliação da perda-log (logloss)
 (4a) Regressão Logística
Um classificador que podemos utilizar nessa base de dados é a regressão logística, que nos dá a probabilidade de um evento de clique em banner ocorrer. Vamos utilizar a função  LogisticRegressionWithSGD para treinar um modelo usando  OHETrainData com a configuração de parâmetros dada.  LogisticRegressionWithSGD retorna um LogisticRegressionModel.
Em seguida, imprima  LogisticRegressionModel.weights e LogisticRegressionModel.intercept para verificar o modelo gerado.
Can you write Python code for it?

from pyspark.mllib.classification import LogisticRegressionWithSGD

# fixed hyperparameters
numIters = 50
stepSize = 10.
regParam = 1e-6
regType = 'l2'
includeIntercept = True

# EXERCICIO
model0 = <COMPLETAR>
sortedWeights = sorted(model0.weights)
print sortedWeights[:5], model0.intercept

# TEST Logistic regression (4a)
Test.assertTrue(np.allclose(model0.intercept,  0.56455084025), 'incorrect value for model0.intercept')
Test.assertTrue(np.allclose(sortedWeights[0:5],
                [-0.45899236853575609, -0.37973707648623956, -0.36996558266753304,
                 -0.36934962879928263, -0.32697945415010637]), 'incorrect value for model0.weights')


I am working on the file "Atividade04/Lab4b_classificacao.ipynb".
The first task is:
Visualização 2: Curva ROC
A curva ROC nos mostra o custo-benefício entre a taxa de falso positivo e a taxa de verdadeiro positivo, conforme diminuimos o limiar de predição. Um modelo aleatório é representado por uma linha pontilhada. Idealmente nosso modelo deve formar uma curva acima dessa linha.
Can you write Python code for it?

labelsAndScores = OHEValidationData.map(lambda lp:
                                            (lp.label, getP(lp.features, model0.weights, model0.intercept)))
labelsAndWeights = labelsAndScores.collect()
labelsAndWeights.sort(key=lambda (k, v): v, reverse=True)
labelsByWeight = np.array([k for (k, v) in labelsAndWeights])

length = labelsByWeight.size
truePositives = labelsByWeight.cumsum()
numPositive = truePositives[-1]
falsePositives = np.arange(1.0, length + 1, 1.) - truePositives

truePositiveRate = truePositives / numPositive
falsePositiveRate = falsePositives / (length - numPositive)

# Generate layout and plot data
fig, ax = preparePlot(np.arange(0., 1.1, 0.1), np.arange(0., 1.1, 0.1))
ax.set_xlim(-.05, 1.05), ax.set_ylim(-.05, 1.05)
ax.set_ylabel('True Positive Rate (Sensitivity)')
ax.set_xlabel('False Positive Rate (1 - Specificity)')
plt.plot(falsePositiveRate, truePositiveRate, color='#8cbfd0', linestyle='-', linewidth=3.)
plt.plot((0., 1.), (0., 1.), linestyle='--', color='#d6ebf2', linewidth=2.)  # Baseline model
pass


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Create a tag_url column that just has the tag's name
Can you write Python code for it?

articles.tag_url = articles.tag_url.apply(get_tag)
articles.head()


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
De-dupe articles but maintain the tagging data using groupby and sum
Can you write Python code for it?

unique_articles = articles.groupby('tagged_url').sum() #made into func

unique_articles = unique_articles.reset_index()

unique_articles = unique_articles.set_index('tagged_url')


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Get dummies for each tag
Can you write Python code for it?

test = pd.get_dummies(articles.tag_url)

test.head()


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Join the dummies back to the main dataframe
Can you write Python code for it?

articles = articles.join(test)

articles.drop(['tag_id','tag_url','tagged_type','tagged_id'],axis=1,inplace=True)

articles.head()


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Let's check average performance when just looking at Simplereach Tag data
Can you write Python code for it?

simplereach = pd.read_csv('simplereach-tags.csv')

simplereach.head(1)

simplereach = simplereach.set_index('Tag')

total_tagged2 = total_tagged

total_tagged2.head(4)

total_tagged2.index = [x.replace('-',' ') for x in total_tagged.index]

simplereach = simplereach.join(total_tagged2)

simplereach['mean-PVs'] = simplereach['Page Views'] // simplereach['num_tagged']
simplereach['mean-shares'] = simplereach['Facebook Shares'] // simplereach['num_tagged']

simplereach = simplereach[['mean-PVs','mean-shares','num_tagged']]

simplereach[simplereach['num_tagged'] > 5].sort_values('mean-PVs',ascending=False)

#regular_features = [x.replace('-',' ') for x in regular_features]
simplereach.ix[regular_features].sort_values('mean-PVs',ascending=False)


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Let's determine how many days post-publication it takes for an article to collect 90% of total pageviews.
Can you write Python code for it?

time_series['days_to_90p']= [(time_series.iloc[x].expanding().sum() > time_series.iloc[x].total*.90).argmax() \
                                 for x in range(len(time_series))]

time_series.reset_index(inplace=True)

time_series.head(1)

time_series['index'] = ['www.atlasobscura.com/articles/' + x for x in time_series['index']]
time_series.set_index('index',inplace=True)
time_series = time_series.join(pageviews.published)
time_series.head(5)

time_series['published'] = pd.to_datetime(time_series.published)

time_series['year_pub'] = pd.DatetimeIndex(time_series['published']).year

time_series.boxplot(column='days_to_90p',by='year_pub')

time_series.year_pub.value_counts(dropna=False)

time_series[['days_to_90p','total','year_pub']].corr()

#I DON'T KNOW WHY THIS WON'T WORK
time_series['30-day-PVs'] = [time_series.fillna(value=0).iloc[x,0:31].sum() for x in range(len(time_series))]

time_series['7-day-PVs'] = [time_series.fillna(value=0).iloc[x,0:8].sum() for x in range(len(time_series))]


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Let's import the time-series I created with a python script that asks GA for the daily time-series of Pageviewsof each article from publication date forward two years.
Can you write Python code for it?

time_series = pd.read_csv('time-series.csv')

type(time_series)

time_series = time_series.drop('Unnamed: 0',axis=1)


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Let's run some regression analysis on our tag_analysis DataFrame
Can you write Python code for it?

from sklearn import linear_model

from sklearn import metrics

tag_analysis.fillna(value=0,inplace=True)

y = tag_analysis.upper_quartile
X = tag_analysis.drop(['pageviews','published','upper_quartile'],axis=1)

from sklearn import cross_validation

kf = cross_validation.KFold(len(tag_analysis),n_folds=5)
scores = []
for train_index, test_index in kf:
    lr = linear_model.LogisticRegression().fit(X.iloc[train_index],y.iloc[train_index])
    scores.append(lr.score(X.iloc[test_index],y.iloc[test_index]))
print "average accuracy for LogisticRegression is", np.mean(scores)
print "average of the set is: ", np.mean(y)

lr_scores = lr.predict_proba(X)[:,1]

print metrics.roc_auc_score(y,lr_scores)

print metrics.roc_auc_score(y,lr_scores)

lr_scores

coefficients = pd.DataFrame(zip(X.columns,lr.coef_[0]),columns=['tags','coefficients'])
probabilities = pd.DataFrame(zip(X.columns,lr_scores),columns=['tags','probabilities'])

probabilities.sort_values('probabilities',ascending=False)

coefficients.sort_values('coefficients',ascending=False)

tag_analysis[tag_analysis['100-wonders'] ==1].describe()

tag_analysis.head()


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Let's try the Logistic Model but with more tags
Can you write Python code for it?

tag_analysis2 = article_set.drop(total_tagged[total_tagged.num_tagged < 15].index,axis=1)

tag_analysis2['ten_thousand'] = [1 if x > 10000 else 0 for x in tag_analysis2.pageviews]

tag_analysis2.fillna(value=0,inplace=True)
y2 = tag_analysis2.ten_thousand
X2 = tag_analysis2.drop(['pageviews','upper_quartile','ten_thousand'],axis=1)

kf2 = cross_validation.KFold(len(tag_analysis2),n_folds=5)
scores2 = []
for train_index, test_index in kf2:
    lr2 = linear_model.LogisticRegression().fit(X2.iloc[train_index],y2.iloc[train_index])
    scores2.append(lr2.score(X2.iloc[test_index],y2.iloc[test_index]))
print "average accuracy for LogisticRegression is", np.mean(scores2)
print "average of the set is: ", np.mean(y2)

print tag_analysis2.shape
print y2.shape
print X2.shape

lr_scores2 = lr2.predict_proba(X2)[:,1]

lr2_probs = pd.DataFrame(zip(X2.columns,lr_scores2),columns=['tags','probabilities'])

lr2_probs.sort_values('probabilities',ascending=False)

metrics.roc_auc_score(y2,lr2.predict_proba(X2)[:,1])

lr2_probs = lr2_probs.set_index('tags')

lr2_probs = lr2_probs.join(total_tagged)

plt.figure(figsize=(10,10))
plt.scatter(lr2_probs.num_tagged,lr2_probs.probabilities)
plt.show()

lr2_probs = lr2_probs.sort_values('probabilities',ascending=False)

lr2_probs = lr2_probs.reset_index()

lr2_probs.to_csv('min15tags_min10000pvs.csv')

lr2_probs.shape

lr2_probs


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Looks like they give similar scores, but the scores are easily manipulated by changing the threshold for the number of articles per tag and by changing the threshold for "success" (currently set at > 10,000 Pageviews).
Can you write Python code for it?

probabilities = probabilities.set_index('tags')

probabilities = probabilities.join(total_tagged)

probabilities.to_csv('tag-probabilities-logisticregression.csv')


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Now I'm going to try this with the time series 30days pvs
Can you write Python code for it?

#doublecheck my work on pageviews vs num-published
pub_volume = time_series[['published','7-day-PVs']]
pub_volume['num_pubbed'] = 1
pub_volume['published'] = pd.to_datetime(pub_volume.published)
pub_volume = pub_volume.set_index('published')

pub_volume.head(10)

num_holder = pub_volume.resample('D').sum().dropna().drop('7-day-PVs',axis=1)
pub_volume = pub_volume.resample('D').sum().dropna().drop('num_pubbed',axis=1)
pub_volume = pub_volume.join(num_holder)
pub_volume['year'] = pub_volume.index.year
pub_volume[pub_volume.index.year >=2015].corr()

pub_volume[pub_volume.index >='2016-01-01'].plot(kind='scatter',x='num_pubbed',y='7-day-PVs',title='7-Day PVs')

import seaborn as sns
ax = sns.regplot(x='num_pubbed',y='7-day-PVs',data=pub_volume)


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Now let's try RandomForest
Can you write Python code for it?

from sklearn.ensemble import RandomForestClassifier

params = {'max_depth': np.arange(20,100,2),
          'min_samples_leaf': np.arange(90,200,2),
          'n_estimators': 20}
gs1 = GridSearchCV(RandomForestClassifier(),param_grid=params, cv=10, scoring='roc_auc',n_jobs=8,verbose=1)
gs1.fit(X,y)
print gs1.best_params_
print gs1.best_score_

rf = RandomForestClassifier(gs1.best_estimator_)
rf.fit(X,y)
probs = rf.predict_proba(X)[:,1]
print rf.score(X,y)
print metrics.roc_auc_score(y,probs)

probs = pd.DataFrame(zip(X.columns,probs),columns=['tags','probabilities'])

probs.sort_values('probabilities',ascending=False)


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Reset index
Can you write Python code for it?

article_set.reset_index()

article_set['upper_quartile'] = [1 if x > 10000 else 0 for x in article_set.pageviews]

article_set.pageviews.plot(kind='hist', bins=100,title='Page View Distribution, All Content')

article_set['published'] = pd.to_datetime(article_set['published'])

article_set

article_set['year'] = pd.DatetimeIndex(article_set['published']).year

ax = article_set.boxplot(column='pageviews',by='year',figsize=(6,6),showfliers=False)
ax.set(title='PV distribution by year',ylabel='pageviews')


I am working on the file "Atlas/Atlas Tag Regression.ipynb".
The first task is:
Using a csv generated by a script I wrote that queries Google Analytics for pageviews per article from publish date to n-days post-publication, import pageview data and join it to the tag/article DataFrame
Can you write Python code for it?

#now we need the pageviews and have to map the URLs to Page Titles
pageviews = pd.read_csv('output_articles_performance.csv',header=None,names=['url','published','pageviews'])
pageviews.head()
#In the future I should import the module and run it here instead of grabbing.

pageviews.url = ['www.atlasobscura.com/articles/' + x for x in pageviews.url]

pageviews.head()

pageviews.describe()


I am working on the file "Atlas/Draft Final Project.ipynb".
The first task is:
But we need to get the tag name out of the url string for the tag
Can you write Python code for it?

articles.tag_url = articles.tag_url.apply(get_tag_name)

articles = get_dummies_and_join(articles,'tag_url')
articles = articles.drop(['tag_id','tag_url','tagged_type','tagged_id'],axis=1)

articles = unique_article_set(articles,'tagged_url')

articles = articles.reset_index().set_index('tagged_url')


I am working on the file "Atlas/Draft Final Project.ipynb".
The first task is:
Import the table of URLs and total pageviews
Can you write Python code for it?

pageviews = pd.read_csv('output_articles_performance.csv',header=None,names=[
        'url','published','pageviews'
    ])

pageviews.url = ['www.atlasobscura.com/articles/' + x for x in pageviews.url]

pageviews.describe()

pageviews.set_index('url',inplace=True)
article_set = articles.join(pageviews)

article_set['ten_thousand'] = target_pageview_cutoff(10000,article_set.pageviews)

article_set['published'] = pd.to_datetime(article_set['published'])
article_set['year'] = get_year(article_set,'published')

article_set.pageviews.plot(kind='density',title='Page View Distribution, All Articles')

ax = article_set.boxplot(column='pageviews',by='year',figsize=(6,6),showfliers=False)
ax.set(title='PV distribution by year of publication, no outliers',ylabel='pageviews')

sns.factorplot(
    x='year',
    y='ten_thousand',
    data = article_set
    )

total_tagged = get_total_tagged(article_set,'num_tagged')

article_set.fillna(value=0,inplace=True)

y = article_set.ten_thousand
X = article_set.drop(['pageviews','published','ten_thousand'],axis=1)

cross_val_score = get_cross_validation_score(X,y,linear_model.LogisticRegression(penalty = 'l1'),
                                            n_folds=5)

lr = linear_model.LogisticRegression(penalty = 'l1').fit(X,y)

lr_scores = lr.predict_proba(X)[:,1]
roc_score = get_roc_scores(y,lr_scores)
print roc_score

single_tag_probabilities = get_probabilities(lr,X)


I am working on the file "Atlas/Draft Final Project.ipynb".
The first task is:
KNN with FB Shares > 500 as target
Can you write Python code for it?

params_social = {'n_neighbors' : [x for x in range(2,100,4)],
          'weights' : ['distance','uniform']}
gs_social = GridSearchCV(estimator = KNeighborsClassifier(),param_grid=params,
                  n_jobs=-1,cv=10,verbose=1)
gs_social.fit(X2,y2)
print gs_social.best_params_
print gs_social.best_score_

knn_social = gs_social.best_estimator_.fit(X2,y2)
knn_probs_social = get_probabilities(knn_social,X2)
knn_cross_val_score_social = get_cross_validation_score(X2,y2,knn_social,5)
knn_scores_social = knn_social.predict_proba(X2)[:,1]
knn_roc_score_social = get_roc_scores(y2,knn_scores_social)


I am working on the file "Atlas/Draft Final Project.ipynb".
The first task is:
Now we will explore how a KNN classifier does with our dataset
Can you write Python code for it?

params = {'n_neighbors' : [x for x in range(2,100,4)],
          'weights' : ['distance','uniform']}
gs = GridSearchCV(estimator = KNeighborsClassifier(),param_grid=params,
                  n_jobs=-1,cv=10,verbose=1)
gs.fit(X,y)
print gs.best_params_
print gs.best_score_

knn = gs.best_estimator_.fit(X,y)
knn_probs = get_probabilities(knn,X)

knn_cross_val_score = get_cross_validation_score(X,y,knn,5)
knn_scores = knn.predict_proba(X)[:,1]
knn_roc_score = get_roc_scores(y,knn_scores)

params_rfc = {'max_depth': np.arange(20,100,5),
             'min_samples_leaf': np.arange(90,200,5),
             'n_estimators': [20],
             'criterion' : ['gini','entropy']
                }
gs1 = GridSearchCV(RandomForestClassifier(),param_grid=params_rfc, cv=10, scoring='roc_auc',n_jobs=-1,verbose=1)
gs1.fit(X,y)
print gs1.best_params_
print gs1.best_score_


I am working on the file "Atlas/Draft Final Project.ipynb".
The first task is:
Now we will explore how a RandomForest classifier does with our dataset
Can you write Python code for it?

rf = gs1.best_estimator_
rf.fit(X,y)

rf_cross_val_score = get_cross_validation_score(X,y,rf,5)
rf_scores = rf.predict_proba(X)[:,1]
rf_roc_score = get_roc_scores(y,rf_scores)

print "Logistic Regression Cross-validation Score: ", cross_val_score
print "K Nearest Neighbors Cross-validation Score: ", knn_cross_val_score
print "RandomForest Cross-validation Score: ", rf_cross_val_score

print "Logistic Regressions ROC AUC score: ", roc_score
print "K Nearest Neighbors ROC AUC score: ", knn_roc_score
print "RandomForest ROC AUC score: ", rf_roc_score


I am working on the file "Atlas/Draft Final Project.ipynb".
The first task is:
Now we will rebuild our model to have it predict if an article will receive over 500 Facebook shares.
Can you write Python code for it?

simplereach = pd.read_csv('~/Downloads/all-content-simplereach.csv')

simplereach.Url = simplereach.Url.apply(get_simplereach_url)
simplereach = simplereach.set_index('Url')
simplereach = simplereach[['Avg Engaged Time','Social Actions','Facebook Shares','FaceBook Referrals']]

article_set2 = article_set.join(simplereach['Facebook Shares'])

article_set2['five_hundred_shares'] = target_pageview_cutoff(500,article_set2['Facebook Shares'])


I am working on the file "Atlas/Draft Final Project.ipynb".
The first task is:
Prediction of a given URL
Can you write Python code for it?

url, taglist = get_article_tags('http://www.atlasobscura.com/articles/the-ao-exit-interview-12-years-in-the-blue-man-group')

transformed_article = transform_article_for_prediction(url,article_set)


I am working on the file "Atlas/Draft Final Project.ipynb".
The first task is:
RandomForest with FB Shares > 500 as target
Can you write Python code for it?

params_rfc = {'max_depth': np.arange(20,100,5),
             'min_samples_leaf': np.arange(90,200,5),
             'n_estimators': [20]}
gs1_social = GridSearchCV(RandomForestClassifier(),param_grid=params_rfc, cv=10, scoring='roc_auc',n_jobs=-1,verbose=1)
gs1_social.fit(X2,y2)

rf_social = gs1_social.best_estimator_
rf_social.fit(X2,y2)
rf_cross_val_score_social = get_cross_validation_score(X2,y2,rf_social,5)
rf_scores_social = rf_social.predict_proba(X)[:,1]
rf_roc_score_social = get_roc_scores(y2,rf_scores_social)

print gs1_social.best_params_
print gs1_social.best_score_

print "Logistic Regression Cross-validation Score: ", cross_val_score_social
print "K Nearest Neighbors Cross-validation Score: ", knn_cross_val_score_social
print "RandomForest Cross-validation Score: ", rf_cross_val_score_social

print "Logistic Regressions ROC AUC score: ", roc_score_social
print "K Nearest Neighbors ROC AUC score: ", knn_roc_score_social
print "RandomForest ROC AUC score: ", rf_roc_score_social

np.mean(y)

simplereach.describe()


I am working on the file "Atlas/Draft Final Project.ipynb".
The first task is:
We only care about the content type "Article"
Can you write Python code for it?

articles = df[df.tagged_type == 'Article']


I am working on the file "Atlas/Erik vs the rest of our content.ipynb".
The first task is:
Above we see that, at first glance, the median Erik article receives about 800 fewer Page Views than the rest of our data set.
Can you write Python code for it?

df_all['Page Views'].describe()

df_erik['Page Views'].describe()


I am working on the file "Atlas/Erik vs the rest of our content.ipynb".
The first task is:
Here we eliminate any article published more recently than last Friday or any article published before Erik joined the company. This is done in order to allow a fair, apples-to-apples comparison. We also remove any article authored by Erik from the "All" dataset.
Can you write Python code for it?

df_all = df_all[(df_all.Published > df_erik.at[6,'Published']) &
                (df_all['Url'].str.contains('/articles/')) & (df_all.Published < df_erik.at[16,'Published']) &
                (df_all.Title[~df_all.Title.isin(df_erik.Title)])]
df_erik = df_erik[(df_erik.Published < df_erik.at[16,'Published'])]

print 'All PVs median'
print df_all['Page Views'].median()
print 'Erik PVs median'
print df_erik['Page Views'].median()


I am working on the file "Atlas/Erik vs the rest of our content.ipynb".
The first task is:
Here we plot the Page Views received per article while eliminating outliers (both high and low).
Can you write Python code for it?

d = {'erik':df_erik['Page Views'],'all':df_all['Page Views']}
df = pd.DataFrame(data=d)
df.plot(kind='box',showfliers=False, title = "Page Views Distribution")


I am working on the file "Atlas/Publishing Volume.ipynb".
The first task is:
Here we plot the total page views, "PVs total", and the average page views based on how many pieces of content were published in a given day.
From this we see increasing returns for publishing more, but sparse data on the high end of the dataset.
Can you write Python code for it?

df_trunc.plot(kind='scatter',x='count',y='Page Views',title='PVs total')
df_trunc.plot(kind='scatter',x='count',y='mean',title='Average PVs')
df_trunc.plot(kind='scatter',x='count',y='Facebook Shares',title='Total Facebook Shares')

df2 = pd.read_csv('All content.csv',index_col='Published',parse_dates=True)

df_articles = df2[(df2['Url'].str.contains('/articles/',na=False))]
df_places = df2[(df2['Url'].str.contains('/places/',na=False))]
df_articles['count']=1
df_places['count']=1

df_articles_resampled = df_articles.resample('D',how='sum')
df_articles_trunc = df_articles_resampled.truncate(before='2015-06-01', after='2016-03-04')
df_articles_trunc = df_articles_trunc.dropna()
df_articles_trunc = df_articles_trunc[['Page Views', 'Social Actions', 'Social Referrals', 'Facebook Shares', 'count']]
df_articles_trunc['mean']=df_articles_trunc['Page Views']//df_articles_trunc['count']


I am working on the file "Atlas/Publishing Volume.ipynb".
The first task is:
It looks like there is actually a weak relationship between median place traffic and overall publishing volume. It seems optimal to publish around 6 places per day.
import statsmodels.formula.api as smf
Can you write Python code for it?

import statsmodels.formula.api as smf
df_trunc

lm = smf.ols(formula="Page Views ~ count", data=df_trunc).fit()
lm.summary()


I am working on the file "Atlas/Publishing Volume.ipynb".
The first task is:
Median Article Traffic
Can you write Python code for it?

df_articles_resampled2 = df_articles.resample('D',how='median')
df_articles_trunc2 = df_articles_resampled2.truncate(before='2015-06-01', after='2016-03-04')
df_articles_trunc2 = df_articles_trunc2.dropna()
df_articles_trunc2 = df_articles_trunc2[['Page Views', 'Social Actions', 'Social Referrals', 'Facebook Shares']]
df_articles_trunc2['count']=df_articles_trunc['count']

df_articles_trunc2.plot(kind='scatter',x='count',y='Page Views',title='Median Articles PVs')


I am working on the file "Atmospheric Data.ipynb".
The first task is:
Now we can make two sets of files.
This first one gives us a list of file paths on the same node as this Notebook i.e. local loading. We're just doing this for one day.
Can you write Python code for it?


I am working on the file "AtomAndElement.ipynb".
The first task is:
Here are some typical operations:
Can you write Python code for it?

print bs.ones(10, int) + bs.arange(10)
print bs.ones(10) / (bs.arange(10) + 1)


I am working on the file "AtomAndElement.ipynb".
The first task is:
In this last example the cardinality of the index is passed on to the Atom being indexed. In this next example the Atom being indexed will have a cardinality to 2 and the index will have a cardinality of 3. The resulting Atom will have cardinality of 4. All of this is happening with out looping through list.
Can you write Python code for it?

z2 = bs.arange(78)
z2.bincounts.append((bs.arange(12) + 1))
print 'z2 =', z2
print 'z2[y] =', z2[y]
print 'z2.cardinality = ', z2.cardinality
print 'y.cardinality = ', y.cardinality
print 'z2[y].cardinality = ', z2[y].cardinality


I am working on the file "AtomAndElement.ipynb".
The first task is:
In this last example we can see that an Atom is made up of:
x.asarray() - the data in the form of a ndarray
x.mask() - the mask to be applied to the data
x.bincounts - bins is a common term used for aggregation. Notice that it is a list. You will see why in the next example.

In this example we will look at how a list of lists of lists is represented by an Atom.
Can you write Python code for it?

y = bs.Atom.fromlist([[[0, 1, 2, None, 4], None, [5, 6]], [[7, 8, 9, 4], [None, 10, 11]]])
print 'y = ', y
print 'y.asarray() = ', y.asarray()
print 'y.mask() = ',    y.mask
print 'y.bincounts = ', y.bincounts
print 'x.cardinality = ', x.cardinality
print 'y.cardinality = ', y.cardinality
print 'x.counts = ', x.counts
print 'y.counts = ', y.counts


I am working on the file "AtomAndElement.ipynb".
The first task is:
Let's operate on these Atoms.
Can you write Python code for it?

print "x * 2 = ", x * 2
print "x + 2 = ", x + 2
print "y - 2 = ", y - 2
print "(y + 1) / 2 = ", (y + 1) / 2
print "x = ", x
print "x.sum() = ", x.sum()
print "x.sum().sum() = ", x.sum().sum()
print "y = ", y
print "y.sum() = ", y.sum()
print "y.sum().sum() = ", y.sum().sum()
print "y.sum().sum().sum() = ", y.sum().sum().sum()
print "y.average() = ", y.average()


I am working on the file "AtomAndElement.ipynb".
The first task is:
Meet the Atom.
First let's create a few and prove that they are Atoms.
Can you write Python code for it?

print bs.zeros(10, int)
print bs.ones(10, int)
print bs.arange(10)
print type(bs.zeros(10, int))


I am working on the file "AtomAndElement.ipynb".
The first task is:
Notice the hierarchy looks kind of like XML or JSON. Check this out:
Can you write Python code for it?

print join.toxml()

print join.toxml(use_attributes=False)

print join.tojson()


I am working on the file "AtomAndElement.ipynb".
The first task is:
Now for group operations. In the Atom indexing example we show how an index imposes it's cardinality on the Atom being indexed. This makes it very easy to build a group operation. The implementation is only a few lines so take a look but the group index is the 2 column sort_index and the bincounts which are calculated by comparing x[:-1] and x[1:]. The index is applied in getattr when Element needs to deligate to source. This is more then you need to know.
Can you write Python code for it?

e3 = e2.group('group_id1,group_id2')
e3.display()
print "e3.__index__ = ", e3.__index__
print "e3.__source__.display():"
e3.__source__.display()


I am working on the file "AtomAndElement.ipynb".
The first task is:
Now for the join:
Can you write Python code for it?

join = e4.inner(groups, group_id='group_id2')
join.display()


I am working on the file "AtomAndElement.ipynb".
The first task is:
Now let's do some caching. For this I am relying on numpy.savez and numpy.savez_compressed. I have yet to find a faster way of storing a billion values. Here is the documentation:
http://docs.scipy.org/doc/numpy/reference/generated/numpy.savez.html
I use it in Atom.load and Atom.persist:
I splay columns into individual npz files:
lbrenner@josie:~/buckysoap$ ls cache/sample
one.npz  range.npz  zero.npz

Each Atom's npz file contains data, mask and neccessary bincounts.
 Filemode      Length  Date         Time      File
- ----------  --------  -----------  --------  --------
  -rw-------       160   4-Jun-2015  12:23:08  data.npy
  -rw-------        90   4-Jun-2015  12:23:08  mask.npy
- ----------  --------  -----------  --------  --------
               250                         2 files
Can you write Python code for it?

def cache_sample():
    def factory():
        print "Running Factory"
        return bs.Element(name='Sample')(
            range = bs.arange(10),
            zero = bs.zeros(10, int),
            one = bs.ones(10, int))
    return bs.Element(
        name='sample',
        cnames='range,zero,one',
        cachedir='cache/sample',
        factory=factory)

#We are just deleting directory for the sake of demonstration
import os
if os.path.exists('cache/sample'):
    print "Deleting cache/sample"
    import shutil
    shutil.rmtree('cache/sample')
cache_sample().display()
cache_sample().display()


I am working on the file "AtomAndElement.ipynb".
The first task is:
Now let's introduce the join which is also implemented using nothing but Numpy. Take a look at the code:
https://github.com/leonhardbrenner/buckysoap/blob/master/src/buckysoap/join.py
The Element uses this packaage method join. Let's start with the setup. I am creating Element(groups) which I will join to Element(e4).
Can you write Python code for it?

groups = (
    bs.Element(name='Groups')
    (group_id = bs.arange(6))
    (name = lambda x: ['group_%d' % y for y in x.group_id[::-1]]))
groups.display()
e4.display()


I am working on the file "AtomAndElement.ipynb".
The first task is:
Now let's look at the expand method. The examples should be clear.
Can you write Python code for it?


I am working on the file "AtomAndElement.ipynb".
The first task is:
The previous Atoms are alot like np.ndarray because Atom just extends np.ndarray. Let's look at what functionality Atom adds to np.ndarray. We are going to pass a list of lists where some of
the lists contain None, some are EMPTY and the root list may contain None.
Can you write Python code for it?

x = bs.Atom.fromlist([[0, 1, 2, None, 4], None, [], [5, 6]])
print 'x = ', x
print 'x.asarray() = ', x.asarray() #This np.ndarray portion of the data
print 'x.mask() = ',    x.mask      #This exists on all Atoms to signify None
print 'x.bincounts = ', x.bincounts #This represents the bincounts on each axis of our Atom


I am working on the file "AtomAndElement.ipynb".
The first task is:
We can stack Atoms together. Note: Atoms must have the same cardinality.
Can you write Python code for it?


I am working on the file "Atomic Pandas.ipynb".
The first task is:
Now is your chance to practice retrieving "slices" of the data using df.loc or df.iloc.
Can you write Python code for it?

newdf.iloc[0:10, 0:3]  # you may need to do some data cleaning as well

newdf.sort_values('Protons')

type(newdf.Protons)

%matplotlib inline
from matplotlib import pyplot as plt
x_axis = newdf.Protons
y_axis = newdf.Mass
plt.scatter(x_axis, y_axis)
plt.title('Protons versus Atomic Mass')
plt.xlabel('Protons')
plt.ylabel('Mass')
plt.show()


I am working on the file "Atomic Pandas.ipynb".
The first task is:
That's right, atoms get heavier the more protons they have.  They have at least as many neutrons as protons, usually, and sometimes more. Remember some elements have isotopes, meaning differing numbers of neutrons for the same atomic number (number of protons, or positive charges in the nucleus).
Can you write Python code for it?

newdf.Initials[1:5]

sorted = newdf.sort_values('Protons')
sorted

sorted[sorted["Protons"] > 22].head()


I am working on the file "AtomicString1.ipynb".
The first task is:
3.2. Partition of Unity
The Atomic Function pulses superposition set at points -2, -1, 0, +1, +2... can exactly represent a Unity (number 1):
1 = ... up(x-3) + up(x-2) + up(x-1) + up(x-0) + up(x+1) + up(x+2) + up(x+3) + ...
1 = ... up(x-3) + up(x-2) + up(x-1) + up(x-0) + up(x+1) + up(x+2) + up(x+3) + ...
Can you write Python code for it?

x = np.arange(-2.0, 2.0, 0.01)
pl.plot(x, up(x, 1, -1), '--', linewidth=1, label='Atomic Function at x=-1')
pl.plot(x, up(x, 1, +0), '--', linewidth=1, label='Atomic Function at x=0')
pl.plot(x, up(x, 1, -1), '--', linewidth=1, label='Atomic Function at x=-1')
pl.plot(x, Sum(up(x, 1, -1), Sum(up(x), up(x, 1, 1))), linewidth=2, label='Atomic Function Compounding')
pl.title('Atomic Function Compounding represent 1')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AtomicString1.ipynb".
The first task is:
3.3. Atomic Function (AF) is a 'finite', 'compactly supported', or 'solitary' function
Like a Spline, Atomic Function (AF) 'compactly supported' not equal to zero only on section |x|<=1
Can you write Python code for it?

x = np.arange(-5.0, 5.0, 0.01)
pl.plot(x, up(x), label='Atomic Function', linewidth=2)
#pl.plot(x, dydx, '--', label='Atomic Function Derivative', linewidth=1, color="Green")

pl.title('Atomic Function is compactly supported')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AtomicString1.ipynb".
The first task is:
4.6. Partition of Line from Atomic String functions
Combination/summation of Atomic Strings can exactly represent a straight line:
x = ...Astring(x-2) + Astring(x-1) + AString(x) + Astring(x+1) + Astring(x+2)...
x = ...Astring(x-2) + Astring(x-1) + AString(x) + Astring(x+1) + Astring(x+2)...
Partition based on AString function with width 1 and height 1
Can you write Python code for it?

x = np.arange(-3, 3, 0.01)

pl.plot(x, AString(x, 1, -1.0, 1, 0),   '--',  linewidth=1, label='AString 1')
pl.plot(x, AString(x, 1, +0.0, 1, 0),   '--',  linewidth=1, label='AString 2')
pl.plot(x, AString(x, 1, +1.0, 1, 0),   '--',  linewidth=1, label='AString 3')

AS2 = Sum(AString(x, 1, -1.0, 1, 0), AString(x, 1, +0.0, 1, 0))
AS3 = Sum(AS2, AString(x, 1, +1.0, 1, 0))
pl.plot(x, AS3,  label='AStrings Sum', linewidth=2)

pl.title('Atomic Strings compose Line')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AtomicString1.ipynb".
The first task is:
5.3. Model of Spacetime curvature and gravity based on AStrings
Schematic model of Gravitation explaining General Relativity effects where spacetime Shape, Density and Curvature are deeply related to each other being expressed via shifts and stretches of the same AString Atomic Function
Can you write Python code for it?

x = np.arange(-50.0, 50.0, 0.1)
dx = x[1] - x[0]

CS6 = Sum(up(x, 5, -30, 5, 5), up(x, 15, 0, 15, 5))
CS6 = Sum(CS6, up(x, 10, +30, 10, 5))
pl.plot(x, CS6,  label='Spacetime Density')

IntC6 = np.cumsum(CS6)*dx/50
pl.plot(x, IntC6, label='Spacetime Shape (Geodesics)')

DerC6 = np.gradient(CS6, dx)
pl.plot(x, DerC6, label='Spacetime Curvature')

LightTrajectory = -10 -IntC6/5
pl.plot(x, LightTrajectory, label='Light Trajectory')

pl.title('Shape of Curved Spacetime model')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AtomicString1.ipynb".
The first task is:
3. Properties of Atomic Function Up(x)
3.1. Atomic Function Derivative expressed via Atomic Function itself
Atomic Function Derivative can be exressed via Atomic Function itself - up'(x)= 2up(2x+1)-2up(2x-1) meaning the shape of pulses for derivative function can be represented by shifted and stratched Atomic Function itself - remarkable property
up'(x)= 2up(2x+1)-2up(2x-1)
Atomic Function and its Derivative plotted together
Can you write Python code for it?

x = np.arange(-2.0, 2.0, 0.01)
pl.plot(x, up(x), label='Atomic Function', linewidth=2)
pl.plot(x, dydx, '--', label='Atomic Function Derivative', linewidth=1, color="Green")

pl.title('Atomic Function and Its Derivative')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "AtomicString1.ipynb".
The first task is:
6. 'Soliton Nature' book
6.1. The theory is also described in a book 'Soliton Nature'
Soliton Nature book is easy-to-read, pictorial, interactive book which uses beautiful photography, video channel, and computer scripts in R and Python to demonstrate existing and explore new solitons – the magnificent and versatile energy concentration phenomenon of nature.  New class of atomic solitons can be used to describe Higgs boson (‘the god particle’) fields, spacetime quanta and other fundamental building blocks of nature.
Can you write Python code for it?

#pl.rcParams["figure.figsize"] = 16,12
book = pl.imread('BookSpread_small.png')
pl.imshow(book)


I am working on the file "AtomicString1.ipynb".
The first task is:
Atomic String, Atomic Function (AF) and AF Derivative plotted together
Can you write Python code for it?

x = np.arange(-2.0, 2.0, 0.01)

#This Calculates Derivative
dx = x[1] - x[0]
dydx = np.gradient(up(x), dx)

pl.plot(x, up(x), label='Atomic Function')
pl.plot(x, AString(x, 1.0, 0, 1, 0), linewidth=2, label='Atomic String Function')
pl.plot(x, dydx, '--', label='A-Function Derivative')

pl.title('Atomic and AString Functions')
pl.legend(loc='best', numpoints=1)
pl.grid(True)
pl.show()


I am working on the file "Atoms R Us.ipynb".
The first task is:

graphic by Kenneth Snelson

Can you write Python code for it?

def print_periodic_table(sortby=1):
    """
    sort all_elements by number of protons, ordered_elements local only
    What about series?
    Sort Order:
    1. protons
    2. symbol
    3. series
    """
    print("Selected:", sortby)

    if sortby == 1:
        ordered_elements = sorted(all_elements.values(), key = lambda k: k.protons)
    elif sortby == 2:
        ordered_elements = sorted(all_elements.values(), key = lambda k: k.symbol)
    elif sortby == 3:
        ordered_elements = sorted(all_elements.values(), key = lambda k: k.series)

    print("PERIODIC TABLE OF THE ELEMENTS")
    print("-" * 70)
    print("Symbol |Long Name             |Protons |Mass   |Series  " )
    print("-" * 70)

    for the_atom in ordered_elements:
        print("{:6} | {:20} | {:6} | {:5.2f} | {:15}".format(the_atom.symbol,
                          the_atom.long_name,
                          the_atom.protons,
                          the_atom.mass,
                          the_atom.series))

print_periodic_table()  # do it for real


I am working on the file "Atoms in Python.ipynb".
The first task is:

graphic by Kenneth Snelson

Can you write Python code for it?


I am working on the file "Aula 1 - Introdução e Conceitos Básicos.ipynb".
The first task is:
Elementos de Circuito
Existem dois tipos de elementos encontrados nos circuitos elétricos: elementos
passivos e elementos ativos. Um elemento ativo é capaz de gerar energia
enquanto um elemento passivo não é. Exemplos de elementos passivos são
resistores, capacitores e indutores; os elementos ativos típicos são geradores,
baterias e amplificadores operacionais.
Uma fonte independente ideal é um elemento ativo que fornece uma tensão
especificada ou corrente que é completamente independente de outros
elementos do circuito.
Uma fonte dependente (ou controlada) ideal é um elemento ativo no
qual a quantidade de energia é controlada por outra tensão ou corrente.
Há quatro
tipos possíveis de fontes dependentes:

Fonte de Tensão Controlada por Tensão (FTCT) - Voltage-Controlled Voltage Source (VCVS)
Fonte de Tensão Controlada por Corrente (FTCC) - Current-Controlled Voltage Source (CCVS)
Fonte de Corrente Controlada por Tensão (FCCT) - Voltage-Controlled Current Source (VCCS)
Fonte de Corrente Controlada por Corrente (FCCC) - Current-Controlled Current Source (CCCS)

As Figuras a seguir apresentam fontes dependentes:

Deve-se observar que uma fonte de tensão ideal (dependente ou independente)
produzirá qualquer corrente necessária para garantir que a tensão entre
seus terminais seja conforme expressa, enquanto uma fonte de corrente ideal
produzirá a tensão necessária para garantir o fluxo de corrente expresso. Portanto,
uma fonte ideal poderia, teoricamente, fornecer uma quantidade de energia
infinita.
Exemplo 1.7
Calcule a potência fornecida ou absorvida em cada elemento na Figura 1.15.
Can you write Python code for it?

print("Exemplo 1.7")
Ip2 = 5 #corrente que passa pelo elemento p2
Ip3 = 6 #corrente que passa pelo elemento p3
Icccs = 0.2*Ip2 #corrente da fonte dependente de corrente controlada por corrente (p4)
v1 = 20 #tensao no elemento p1
p1 = -v1*Ip2 #potencia no elemento p1, corrente a mesma que no elemento p2
v2 = 12 #tensao no elemento p2
p2 = v2*Ip2 #potencia no elemento p2,
v3 = 8 #tensao no elemento p3
p3 = v3*Ip3 #potencia no elemento p3
p4 = -v3*Icccs #potencia no elemento p4, tensao mesma que elemento p3
print("Potencia no elemento p1:", p1, "W")
print("Potencia no elemento p2:", p2, "W")
print("Potencia no elemento p3:", p3, "W")
print("Potencia no elemento p4:", p4, "W")


I am working on the file "Aula 1 - Introdução e Conceitos Básicos.ipynb".
The first task is:
Exemplo 1.2
A carga total entrando em um terminal é dada por q = 5t sen(4 pi t) mC. Calcule a corrente no instante t = 0,5 s
Can you write Python code for it?

print("Exemplo 1.2")

import numpy as np #importa biblioteca numpy
from sympy import * #importa biblioteca sympy
#i = dq/dt

t = symbols('t') #define x como elemento simbolo
q = 5*t*sin(4*np.pi*t) #carga - funcao
i = diff(q, t) #calcula funcao corrente = derivada de q em respeito a t
print("Função corrente, em miliampères, é:", i)
print("Corrente no instante 0.5, em mA, é", i.subs(t,0.5))


I am working on the file "Aula 1 - Introdução e Conceitos Básicos.ipynb".
The first task is:
Exemplo 1.5
Determine a potência fornecida para um elemento no instante t = 3 ms se a corrente que
entra pelo terminal positivo for:
i = 5 cos(60 * pi * t) A
e a tensão for: (a) v = 3i, (b) v = 3di/dt.
Can you write Python code for it?

print("Exemplo 1.5")

import numpy as np
from sympy import *

t = symbols('t')
i = 5*cos(60*np.pi*t)
v1 = 3*i
p1 = i.subs(t,0.003)*v1.subs(t,0.003)
print("Resposta (a):", p1, "W")
v2 = 3*diff(i,t)
p2 = i.subs(t,0.003)*v2.subs(t,0.003)
print("Resposta (b):", p2, "W")


I am working on the file "Aula 1 - Introdução e Conceitos Básicos.ipynb".
The first task is:
Livro Referência utilizado:
ALEXANDER, Chales; SADIKU, Matthew. Fundamentos de Circuitos Elétricos
Introdução
Circuito elétrico é uma interconexão de elementos elétricos
Exemplo de circuito elétrico de um trasmissor de rádio

O Sistema Internacional de Unidades (SI), adotado pela Conferência Geral de Pesos e Medidas em 1960, é conhecido como uma linguagem de medição internacional. Nesse sistema, existem seis unidades principais a partir das quais todas as demais grandezas físicas podem ser derivadas.
Por exemplo, a seguir temos expressões da mesma distância em metros (m):
600.000.000mm     600.000m     600 km
Tabela de multiplicadores de unidades e respectivos símbolos:

Tabela de unidades principais do SI e respectivos símbolos:

Carga e Corrente
Carga é uma propriedade elétrica das partículas atômicas que compõem a matéria, medida em coulombs (C)
Sabemos que a carga e em um elétron é negativa e igual em magnitude a 1,602 * 10^–19 C, enquanto um próton transporta uma carga positiva de mesma magnitude do elétron. A presença de números iguais de prótons e elétrons deixa um átomo com carga neutra.
Em 1 C de carga, existem 1/ (1,602 * 10^–19) = 6,24 * 10^18 elétrons

Corrente elétrica é o fluxo de carga por unidade de tempo, medido em ampères (A)
\begin{align}
i = dq/dt
\end{align}
A carga transferida entre o instante t0 e o instante t é obtida integrando ambos
os lados da Equação
\begin{align}
Q = \int_{t0}^{t} i dt
\end{align}
Se a corrente não muda com o tempo e permanece constante, podemos chamá-la de corrente contínua (CC).
Corrente contínua (CC) é uma corrente que permanece constante ao longo do tempo
Por convenção, o símbolo I é usado para representar uma corrente contínua desse tipo.
Uma corrente que varia com o tempo é conhecida pelo símbolo i. E uma forma comum de corrente é a corrente senoidal ou corrente alternada (CA).
Corrente alternada (CA) é uma corrente que varia com o tempo segundo uma forma de onda senoidal
Gráficos CC (a) e CA (b)

Exemplo 1.1
Qual é a quantidade de carga representada por...
Can you write Python code for it?

print("Exemplo 1.1")
carga_eletron = -1.6*10**(-19) #unidade em Coulombs
numero_eletrons = 4600
carga_total1 = carga_eletron * numero_eletrons
print("Resposta, em Coulombs, é:", carga_total1)


I am working on the file "Aula 1 - Introdução e Conceitos Básicos.ipynb".
The first task is:
Problema Prático 1.1
Calcule a quantidade de carga representada por seis milhões de prótons.
Can you write Python code for it?

print("Problema Prático 1.1")
carga_proton = -carga_eletron
numero_protons = 6000000
carga_total2 = carga_proton * numero_protons
print("Resposta, em Coulombs, é:", carga_total2)


I am working on the file "Aula 1 - Introdução e Conceitos Básicos.ipynb".
The first task is:
Problema Prático 1.2
Se no Exemplo 1.2, q = (10 – 10e–2t) mC, determine a corrente em t = 1,0 s.
Can you write Python code for it?

print("Problema Prático 1.2")

q = 10 - 10*exp(-2*t)
i = diff(q,t)
print("Função corrente, em miliampères, é:", i)
print("Corrente no instante 1.0, em mA, é", i.subs(t,1.0)) #i.subs(t,1.0) substitui o valor t para 1.0


I am working on the file "Aula 1 - Introdução e Conceitos Básicos.ipynb".
The first task is:
Problema Prático 1.6
Um forno elétrico consome 15 A quando conectado a uma linha de 120 V. Quanto tempo
leva para consumir 180 kJ?
Can you write Python code for it?

i = 15
v = 120
q = 180000
p = v*i
print("Potencia:", p, "W")
#p = dq/dt
t = q/p
print("Tempo necessario:", t, "s")


I am working on the file "Aula 10 - Circuitos RL.ipynb".
The first task is:
Exemplo 7.5
No circuito indicado na Figura 7.19, encontre io, vo e i durante todo o tempo, supondo
que a chave fora aberta por um longo período.
Can you write Python code for it?


I am working on the file "Aula 10 - Circuitos RL.ipynb".
The first task is:
Problema Prático 7.4
Para o circuito da Figura 7.18, determine i(t) para t > 0.
Can you write Python code for it?

print("Problema Prático 7.4")

L = 2
Cs = 15
R1 = 24

Req1 = Req(12,8)
i1 = Cs*R1/(R1 + Req1)
I0 = i1*8/(8 + 12)

Rth = Req(12+8,5)
tau = L/Rth

i = I0*exp(-t/tau)

print("Corrente i(t):",i,"A")


I am working on the file "Aula 11 - Funções de Singularidade.ipynb".
The first task is:
Exemplo 7.8
Dado o sinal:
\begin{align}
{\Large g(t) =
\begin{cases}
3, & t < 0
\-2, & 0 < t < 1
\2t - 4, & t >1
\end{cases}}
\end{align}
expresse g(t) em termos de funções degrau e de rampa.
Can you write Python code for it?


I am working on the file "Aula 11 - Funções de Singularidade.ipynb".
The first task is:
Exemplo 7.9
Calcule as seguintes integrais envolvendo a função impulso.
\begin{align}
{\Large \int_{0}^{10} (t^2 + 4t - 2)\delta(t-2)dt}
\ {\Large \int_{-\infty}^{\infty} [\delta(t - 1)e^{-t} cos(t) + \delta(t + 1)e^{-t}sen(t)]dt}
\end{align}
Can you write Python code for it?

print("Exemplo 7.9")

u = (t**2 + 4*t - 2)*DiracDelta(t - 2)
y = integrate(u,(t,0,10))
print("Primeira integral:", y)

u = DiracDelta(t-1)*exp(-t)*cos(t) + DiracDelta(t+1)*exp(-t)*sin(t)
y = integrate(u,(t,-oo,oo))
print("Segunda integral:", y)


I am working on the file "Aula 11 - Funções de Singularidade.ipynb".
The first task is:
Problema Prático 7.6
Expresse o pulso de corrente da Figura 7.33 em termos de degrau unitário. Determine
sua integral e esboce-a.
Can you write Python code for it?

print("Problema Prático 7.6")

u1 = 10*Heaviside(t)
u2 = -20*Heaviside(t-2)

l,m,n = list(), list(), list()

for i in np.linspace(-10,10,20):
    l.append(u1.subs(t,i))
    m.append(u2.subs(t,i))

print(l)
print(m)
n = np.add(l,m)
n = n.reshape((1,20))
n = n[0]
print(n)

o = np.linspace(-10,10,20)
print(o)

plt.step(o,n)
plt.show()

x = np.linspace(0,4,20)

o = list()

for i in range(len(x)):
    if(x[i] < 2):
        o.append(10*x[i])
    elif(x[i] == 2):
        o.append(20)
    else:
        o.append(20 - 10*(x[i] - 2))

plt.plot(x,o)
plt.show()

#Resposta
#10[u(t) – 2u(t – 2) + u(t – 4)] A,
#10[r(t) – 2r(t – 2) + r(t – 4)] A·s


I am working on the file "Aula 11 - Funções de Singularidade.ipynb".
The first task is:
Problema Prático 7.7
Consulte a Figura 7.39. Expresse i(t) em termos de funções de singularidade.
Can you write Python code for it?

print("Problema Prático 7.7")

def ret(t,start=-10,end=10,points=21):
    x = np.linspace(start,end,points)
    l = x > t
    l = l*x
    return l

def heaviside(t, start=-10,end=10,points=21):
    x = np.linspace(start,end,points)
    l = x > t
    l = 1*l
    return l

y = -(2*heaviside(0) - 2*ret(0) + 4*ret(2) - 2*ret(3))
print(y)

x = np.linspace(-10,10,21)
plt.plot(x,y)
plt.show()

#Resposta
#2u(t) - 2r(t) + 4r(t-2) - 2r(t-3) A.


I am working on the file "Aula 11 - Funções de Singularidade.ipynb".
The first task is:
Problema Prático 7.8
Se:
\begin{align}
{\Large h(t) =
\begin{cases}
0, & t < 0
\-4, & 0 < t < 2
\3t - 8, & 2 < t < 6
\0, & t > 6
\end{cases}}
\end{align}
expresse h(t) em termos de funções de singularidade.
Can you write Python code for it?


I am working on the file "Aula 12 - Resposta a Degrau RL e RC.ipynb".
The first task is:
Exemplo 7.11
Na Figura 7.45, a chave foi fechada há um longo tempo e é aberta em t = 0. Determine
i e v durante todo o período.
Can you write Python code for it?