I am working on the file "Spark/Lab4b_classificacao.ipynb".
The first task is:
Visualização 2: Curva ROC
A curva ROC nos mostra o custo-benefício entre a taxa de falso positivo e a taxa de verdadeiro positivo, conforme diminuimos o limiar de predição. Um modelo aleatório é representado por uma linha pontilhada. Idealmente nosso modelo deve formar uma curva acima dessa linha.
Can you write Python code for it?

labelsAndScores = OHEValidationData.map(lambda lp:
                                            (lp.label, getP(lp.features, model0.weights, model0.intercept)))
labelsAndWeights = labelsAndScores.collect()
labelsAndWeights.sort(key=lambda (k, v): v, reverse=True)
labelsByWeight = np.array([k for (k, v) in labelsAndWeights])

length = labelsByWeight.size
truePositives = labelsByWeight.cumsum()
numPositive = truePositives[-1]
falsePositives = np.arange(1.0, length + 1, 1.) - truePositives

truePositiveRate = truePositives / numPositive
falsePositiveRate = falsePositives / (length - numPositive)

# Generate layout and plot data
fig, ax = preparePlot(np.arange(0., 1.1, 0.1), np.arange(0., 1.1, 0.1))
ax.set_xlim(-.05, 1.05), ax.set_ylim(-.05, 1.05)
ax.set_ylabel('True Positive Rate (Sensitivity)')
ax.set_xlabel('False Positive Rate (1 - Specificity)')
plt.plot(falsePositiveRate, truePositiveRate, color='#8cbfd0', linestyle='-', linewidth=3.)
plt.plot((0., 1.), (0., 1.), linestyle='--', color='#d6ebf2', linewidth=2.)  # Baseline model
pass


I am working on the file "Spark/Lab5a_PCA.ipynb".
The first task is:
(1a) Interpretando o PCA
PCA pode ser interpretado como a identificação das direções em que os dados variam mais. No primeiro passo do PCA, precisamos centralizar nossos dadods. O primeiro passo requer que calculemos a média de cada atributo (coluna) da base de dados. Em seguida, para cada observação (linha), modifique os valores dos atributos pela média correspondente. Dessa forma teremos uma base de dados em que cada atributo tem média zero.
Note que correlatedData é uma RDD de NumPy arrays. Isso permite que façamos algumas operações mais facilmente. Por exemplo, se utilizarmos a transformação sum() o PySpark irá fazer a soma dos valores de cada coluna.
Can you write Python code for it?

# EXERCICIO
correlatedData = sc.parallelize(dataCorrelated)

meanCorrelated = correlatedData.<COMPLETAR>
correlatedDataZeroMean = correlatedData.<COMPLETAR>

print meanCorrelated
print correlatedData.take(1)
print correlatedDataZeroMean.take(1)

# TEST Interpreting PCA (1a)
from test_helper import Test
Test.assertTrue(np.allclose(meanCorrelated, [49.95739037, 49.97180477]),
                'incorrect value for meanCorrelated')
Test.assertTrue(np.allclose(correlatedDataZeroMean.take(1)[0], [-0.28561917, 0.10351492]),
                'incorrect value for correlatedDataZeroMean')


I am working on the file "Spark/Lab5a_PCA.ipynb".
The first task is:
(1b) Matriz de Covariância
Vamos calcular a matriz de covariância de nossos dados. Se definirmos $\scriptsize \mathbf{X} \in \mathbb{R}^{n \times d}$ como a matriz de dados centrada em zero, então a matriz de covariância é definida como: $$ \mathbf{C}_{\mathbf X} = \frac{1}{n} \mathbf{X}^\top \mathbf{X} \,.$$  Essa matriz pode ser calculada computando o produto externo de cada linha com ela mesma, em seguida realizando a somatória das matrizes  resultantes e, finalmente, divindo pelo total de objetos na base de dados. Os dados são bidimensionais, então a matriz de covariância é uma matriz 2x2.
Note que np.outer() pode ser utilizado para calcular o produto externo de duas NumPy arrays.
Can you write Python code for it?

# EXERCICIO
# Compute the covariance matrix using outer products and correlatedDataZeroMean
correlatedCov = (correlatedDataZeroMean
                 .<COMPLETAR>
                 .<COMPLETAR>
                 )/correlatedDataZeroMean.count()
print correlatedCov

# TEST Sample covariance matrix (1b)
covResult = [[ 0.99558386,  0.90148989], [0.90148989, 1.08607497]]
Test.assertTrue(np.allclose(covResult, correlatedCov), 'incorrect value for correlatedCov')


I am working on the file "Spark/Lab5a_PCA.ipynb".
The first task is:
(1d) Autovalores e Autovetores
Agora que temos a matriz de covariância, podemos usá-la para encontrar as direções de maior variância nos dados. Os autovalores e autovetores nos trazem tal informação. Os $\scriptsize d $ autovetores da matriz de covariância nos dá as direções de maior variância e são chamadas de componentes principais. Os autovalores associados são as variâncias nessas direções. Particularmente, o atuovetor correspondente ao maior autovalor é a direção de máxima variância. O cálculo dos autovalores e autovetores da matriz de covariância tem uma complexidade aproximademente $O(d^3)$ para uma matriz d x d. Quando nosso $d$ é suficientemente pequeno, podemos computar os autovalores e autovetores localmente.
Use a função eigh da biblioteca numpy.linalg para calcular os autovalores e autovetores. Em seguida, ordene os autovetores baseado nos autovalores, do maior para o menor, gerando uma matriz em que os autovetores são as colunas (e a primeira coluna é o maior autovetor).
Note que a função np.argsort pode ser usada para obter a ordem dos índices dos elementos em ordem crescente ou decrescente.  Finalmente coloque o maior autovetor na variável topComponent.
Can you write Python code for it?

# EXERCICIO
from numpy.linalg import eigh

# Calculate the eigenvalues and eigenvectors from correlatedCovAuto
eigVals, eigVecs = <COMPLETAR>
print 'eigenvalues: {0}'.format(eigVals)
print '\neigenvectors: \n{0}'.format(eigVecs)

# Use np.argsort to find the top eigenvector based on the largest eigenvalue
inds = <COMPLETAR>
topComponent = <COMPLETAR>
print '\ntop principal component: {0}'.format(topComponent)

# TEST Eigendecomposition (1d)
def checkBasis(vectors, correct):
    return np.allclose(vectors, correct) or np.allclose(np.negative(vectors), correct)
Test.assertTrue(checkBasis(topComponent, [0.68915649, 0.72461254]),
                'incorrect value for topComponent')


I am working on the file "Spark/Lab5a_PCA.ipynb".
The first task is:
(1e) PCA scores
Nós calculamos os principais componentes para uma base de dados não-esférica. Agora vamos usar este componente principal para derivar uma representação unidimensional dos dados originais. Para calcular a representação compact, que é chamada PCA scores, calcule o produto interno entre cada objeto na base original e o componente principal.
Can you write Python code for it?

# EXERCICIO
# Use the topComponent and the data from correlatedData to generate PCA scores
correlatedDataScores = correlatedData.<COMPLETAR>
print 'one-dimensional data (first three):\n{0}'.format(np.asarray(correlatedDataScores.take(3)))

# TEST PCA Scores (1e)
firstThree = [70.51682806, 69.30622356, 71.13588168]
Test.assertTrue(checkBasis(correlatedDataScores.take(3), firstThree),
                'incorrect value for correlatedDataScores')


I am working on the file "Spark/Lab5a_PCA.ipynb".
The first task is:
(2b) PCA em dataRandom
Agora, use a função PCA para encontrar os dois componentes principais da base esférica dataRandom.
Can you write Python code for it?

# EXERCICIO
randomData = sc.parallelize(dataRandom)

# Use pca on randomData
topComponentsRandom, randomDataScoresAuto, eigenvaluesRandom = <COMPLETAR>

print 'topComponentsRandom: \n{0}'.format(topComponentsRandom)
print ('\nrandomDataScoresAuto (first three): \n{0}'
       .format('\n'.join(map(str, randomDataScoresAuto.take(3)))))
print '\neigenvaluesRandom: \n{0}'.format(eigenvaluesRandom)

# TEST PCA on `dataRandom` (2b)
Test.assertTrue(checkBasis(topComponentsRandom.T,
                           [[-0.2522559 ,  0.96766056], [-0.96766056,  -0.2522559]]),
                'incorrect value for topComponentsRandom')
firstThreeRandom = [[36.61068572,  35.97314295,  35.59836628],
                    [61.3489929 ,  62.08813671,  60.61390415]]
Test.assertTrue(np.allclose(firstThreeRandom, np.vstack(np.abs(randomDataScoresAuto.take(3))).T),
                'incorrect value for randomDataScoresAuto')
Test.assertTrue(np.allclose(eigenvaluesRandom, [1.4204546, 0.99521397]),
                            'incorrect value for eigenvaluesRandom')


I am working on the file "Spark/Lab5a_PCA.ipynb".
The first task is:
(3c) Determinando o valor de k
Vamos utilizar a função varianceExplained aplicada em millionRDD (dica: deve primeiro realizar uma transformação) para determinar o valor de k que utilizaremos para os testes de Regressão Linear.
Can you write Python code for it?

for k in range(1,10):
    varexp = varianceExplained(millionRDD.map(lambda x: x.features), k)
    print 'Variation explained by {} components is {}'.format(k,varexp)


I am working on the file "Spark/Lab5a_PCA.ipynb".
The first task is:
Part 2: Função PCA para aplicar em um RDD
(2a) Função PCA
Agora temos todos os ingredientes para calcular uma função PCA. Nossa função deve ser genérica para calcular os $k$ componentes principais e PCA scores. Escreva a função genérica pca e teste utilizando correlatedData e $\scriptsize k = 2$. Dica: Use os resultados da Parte (1c), Parte (1d), e Parte (1e).
Nota: Conforme discutido anteriormente, essa implementação é eficiente enquanto $\scriptsize d $ é pequeno, mas algoritmos distribuídos mais eficientes existem para $\scriptsize d $ grande.
Can you write Python code for it?

# EXERCICIO
def pca(data, k=2):
    """Computes the top `k` principal components, corresponding scores, and all eigenvalues.

    Note:
        All eigenvalues should be returned in sorted order (largest to smallest). `eigh` returns
        each eigenvectors as a column.  This function should also return eigenvectors as columns.

    Args:
        data (RDD of np.ndarray): An `RDD` consisting of NumPy arrays.
        k (int): The number of principal components to return.

    Returns:
        tuple of (np.ndarray, RDD of np.ndarray, np.ndarray): A tuple of (eigenvectors, `RDD` of
            scores, eigenvalues).  Eigenvectors is a multi-dimensional array where the number of
            rows equals the length of the arrays in the input `RDD` and the number of columns equals
            `k`.  The `RDD` of scores has the same number of rows as `data` and consists of arrays
            of length `k`.  Eigenvalues is an array of length d (the number of features).
    """
    <COMPLETAR>

    # Return the `k` principal components, `k` scores, and all eigenvalues
    return (vecs, data.map(lambda x: x.dot(vecs)), vals)

# Run pca on correlatedData with k = 2
topComponentsCorrelated, correlatedDataScoresAuto, eigenvaluesCorrelated = pca(correlatedData, 2)

# Note that the 1st principal component is in the first column
print 'topComponentsCorrelated: \n{0}'.format(topComponentsCorrelated)
print ('\ncorrelatedDataScoresAuto (first three): \n{0}'
       .format('\n'.join(map(str, correlatedDataScoresAuto.take(3)))))
print '\neigenvaluesCorrelated: \n{0}'.format(eigenvaluesCorrelated)

# Create a higher dimensional test set
pcaTestData = sc.parallelize([np.arange(x, x + 4) for x in np.arange(0, 20, 4)])
componentsTest, testScores, eigenvaluesTest = pca(pcaTestData, 3)

print '\npcaTestData: \n{0}'.format(np.array(pcaTestData.collect()))
print '\ncomponentsTest: \n{0}'.format(componentsTest)
print ('\ntestScores (first three): \n{0}'
       .format('\n'.join(map(str, testScores.take(3)))))
print '\neigenvaluesTest:...


I am working on the file "Spark/Lab5a_PCA.ipynb".
The first task is:
Part 3:  Aplicação do PCA na base Million Song, com aplicação de Regressão Linear na base projetada
(3a) Carregando os dados do Data Set
Vamos repetir os procedimentos realizados no Lab4a para carregar e fazer o parsing da base de dados. Também vamos reescrever algumas das funções utilitárias daquele lab.
Can you write Python code for it?

baseDir = os.path.join('Data')
inputPath = os.path.join('Aula04', 'millionsong.txt')
fileName = os.path.join(baseDir, inputPath)

numPartitions = 2
rawData = sc.textFile(fileName, numPartitions)

print rawData.first()

from pyspark.mllib.regression import LabeledPoint

def parsePoint(line):
    """Converts a comma separated unicode string into a `LabeledPoint`.

    Args:
        line (unicode): Comma separated unicode string where the first element is the label and the
            remaining elements are features.

    Returns:
        LabeledPoint: The line is converted into a `LabeledPoint`, which consists of a label and
            features.
    """
    Point = map(float,line.split(','))
    return LabeledPoint(Point[0]-1922,Point[1:])

baselineRL = 17.017
baselineInteract = 15.690

millionRDD = rawData.map(parsePoint)
print millionRDD.take(1)

def squaredError(label, prediction):
    """Calculates the the squared error for a single prediction.

    Args:
        label (float): The correct value for this observation.
        prediction (float): The predicted value for this observation.

    Returns:
        float: The difference between the `label` and `prediction` squared.
    """
    return np.square(label-prediction)

def calcRMSE(labelsAndPreds):
    """Calculates the root mean squared error for an `RDD` of (label, prediction) tuples.

    Args:
        labelsAndPred (RDD of (float, float)): An `RDD` consisting of (label, prediction) tuples.

    Returns:
        float: The square root of the mean of the squared errors.
    """
    return np.sqrt(labelsAndPreds.map(lambda rec: squaredError(rec[0],rec[1])).mean())


I am working on the file "Spark/Lab5b_kmeans_quantiza.ipynb".
The first task is:
(1c) Gerando uma RDD de matrizes
Como primeiro passo, precisamos gerar um dicionário em que a chave são as palavras e o valor é o vetor representativo dessa palavra.
Para isso vamos primeiro gerar uma lista uniqueWords contendo as palavras únicas do RDD words, removendo aquelas que aparecem menos do que 5 vezes $^1$. Em seguida, criaremos um dicionário w2v que a chave é um token e o valor é um np.array do vetor transformado daquele token$^2$.
Finalmente, vamos criar uma RDD chamada vectorsRDD em que cada registro é representado por uma matriz onde cada linha representa uma palavra transformada.
1
Na versão 1.3 do PySpark o modelo Word2Vec utiliza apenas os tokens que aparecem mais do que 5 vezes no corpus, na versão 1.4 isso é parametrizado.
2
Na versão 1.4 do PySpark isso pode ser feito utilizando o método `getVectors()
Can you write Python code for it?

# EXERCICIO
uniqueWords = (wordsRDD
               .<COMPLETAR>
               .<COMPLETAR>
               .<COMPLETAR>
               .<COMPLETAR>
               .collect()
               )

print '{} tokens únicos'.format(len(uniqueWords))

w2v = {}
for w in uniqueWords:
    w2v[w] = <COMPLETAR>
w2vb = sc.broadcast(w2v)
print 'Vetor entertaining: {}'.format( w2v[u'entertaining'])

vectorsRDD = (wordsRDD
              .<COMPLETAR>
             )
recs = vectorsRDD.take(2)
firstRec, secondRec = recs[0], recs[1]
print firstRec.shape, secondRec.shape

# TEST Tokenizing the small datasets (1c)
assert len(uniqueWords) == 3332,  'valor incorreto'
print 'ok!'

assert np.mean(np.abs(w2v[u'entertaining']-[-0.24618697, -0.12722649,  0.02719167,  0.01129477, -0.20605306]))<1e-6,'valor incorreto'
print 'ok!'

assert secondRec.shape == (10,5)
print 'ok!'


I am working on the file "Spark/Lab5b_kmeans_quantiza.ipynb".
The first task is:
(4b) Calcule a similaridade do cosseno entre pares de registros
Complete a função cosineSim que receberá um par ( (doc1, vec1), (doc2, vec2) ) para calcular a similaridade do cosseno entre esses dois documentos.

Usando o índice invertido, agrupe a base pela chave (token) e aplique a função genList que deve gerar uma lista de tuplas (token, ((doc1,tfidf),(doc2,tfidf)) de todos os pares de documentos com essa token em comum exceto nos casos doc1==doc2.

Em seguida, gere tuplas do tipo ( (doc1, doc2), tfidf1tfidf2/(norma1norma2) ) a reduza para realizar a somatória desses valores sob a mesma chave.

Dessa forma teremos os registros de pares de documentos que possuem similaridade não nula com sua similaridade calculada.
Can you write Python code for it?

# EXERCICIO
from itertools import product

def calcsim(rec):
    items = list(rec[1])
    return <COMPLETAR>

newRDD = (quantRDD
          .<COMPLETAR>
          .<COMPLETAR>
          .<COMPLETAR>
          .<COMPLETAR>
          .<COMPLETAR>
          .cache()
          )

newcount = newRDD.count()
print newcount

assert newcount==11796442, 'incorrect value'
print 'ok'


I am working on the file "Spark/Lab5b_kmeans_quantiza.ipynb".
The first task is:
(4f) k-NN
Vamos gerar agora a lista dos k documentos mais similares de cada documento.

Gere uma RDD partindo da commonTokens de tal forma a ter ( id1, (id2, sim) )

Agrupe pela chave e transforme a RDD para ( id1, [ (id,sim) ] ) onde a lista deve ter k elementos
Can you write Python code for it?

# EXERCICIO
def genklist(rec,k):
    """ Generate the list of the k most similar documents to the key
    Args:
        record: a pair, (doc, [(doc,sim)])
        k: number of most similar elements
    Returns:
        pair: (doc, [(doc,sim)])
    """
    <COMPLETAR>
    return (key, docs[:k])

def knn(simRDD, k):
    """ Generate the knn RDD for a given RDD.
    Args:
        simRDD: RDD of ( (doc1,doc2), sim)
        k: number of most similar elements
    Returns:
        RDD: RDD of ( doc1, [docs, sims])
    """

    ksimRDD = (simRDD
               .<COMPLETAR>
               .<COMPLETAR>
               .<COMPLETAR>
              )
    return ksimRDD

ksimReviewsRDD = knn(newRDD, 3)
ksimReviewsRDD.take(3)

print dataRDD.filter(lambda x: x[0] in [55300,39009,130973,66284]).collect()


I am working on the file "Spark/Lab5b_kmeans_quantiza.ipynb".
The first task is:
Part 3: Aplicando k-NN
Nessa parte, vamos testar o algoritmo k-NN para verificar se a base quantizada é capaz de capturar a semelhança entre documentos.
(4a) Pré-calcule a norma dos vetores
Da mesma forma que no Lab 4c, calcule a norma dos vetores de quantRDD.
Can you write Python code for it?

dataNorms = quantRDD.map(lambda rec: (rec[0],np.sqrt(rec[1].dot(rec[1]))))
dataNormsBroadcast = sc.broadcast(dataNorms.collectAsMap())


I am working on the file "Spark/Lab5b_kmeans_quantiza.ipynb".
The first task is:
Parte 1: Word2Vec
A técnica word2vec aprende através de uma rede neural semântica uma representação vetorial de cada token em um corpus de tal forma que palavras semanticamente similares sejam similares na representação vetorial.
O PySpark contém uma implementação dessa técnica, para aplicá-la basta passar um RDD em que cada objeto representa um documento e cada documento é representado por uma lista de tokens na ordem em que aparecem originalmente no corpus. Após o processo de treinamento, podemos transformar um token utilizando o método transform para transformar cada token em uma representaçã vetorial.
Nesse ponto, cada objeto de nossa base será representada por uma matriz de tamanho variável.
(1a) Gerando RDD de tokens
Utilize a função de tokenização tokenize do Lab4d para gerar uma RDD wordsRDD contendo listas de tokens da nossa base original.
Can you write Python code for it?

# EXERCICIO
import re

split_regex = r'\W+'

stopfile = os.path.join("Data","Aula04","stopwords.txt")
stopwords = set(sc.textFile(stopfile).collect())

def tokenize(string):
    """ An implementation of input string tokenization that excludes stopwords
    Args:
        string (str): input string
    Returns:
        list: a list of tokens without stopwords
    """
    return <COMPLETAR>

wordsRDD = dataRDD.map(lambda x: tokenize(x[1]))

print wordsRDD.take(1)[0]

# TEST Tokenize a String (1a)
assert wordsRDD.take(1)[0]==[u'quiet', u'introspective', u'entertaining', u'independent', u'worth', u'seeking'], 'lista incorreta!'
print 'ok!'


I am working on the file "Spark/Lab5b_kmeans_quantiza.ipynb".
The first task is:
Parte 2: k-Means para quantizar os atributos
Nesse momento é fácil perceber que não podemos aplicar nossas técnicas de aprendizado supervisionado nessa base de dados:

A regressão logística requer um vetor de tamanho fixo representando cada objeto

O k-NN necessita uma forma clara de comparação entre dois objetos, que métrica de similaridade devemos aplicar?

Para resolver essa situação, vamos executar uma nova transformação em nossa RDD.   Primeiro vamos aproveitar o fato de que dois tokens com significado similar são mapeados em vetores similares, para agrupá-los em um atributo único.
Ao aplicarmos o k-Means nesse conjunto de vetores, podemos criar $k$ pontos representativos e, para cada documento, gerar um histograma de contagem de tokens nos clusters gerados.
(2a) Agrupando os vetores e criando centros representativos
Como primeiro passo vamos gerar um RDD com os valores do dicionário w2v. Em seguida, aplicaremos o algoritmo k-Means com $k = 200$.
Can you write Python code for it?

# EXERCICIO
from  pyspark.mllib.clustering import KMeans

vectors2RDD = sc.parallelize(np.array(w2v.values()),1)
print 'Sample vector: {}'.format(vectors2RDD.take(1))

modelK = KMeans.<COMPLETAR>

clustersRDD = vectors2RDD.<COMPLETAR>
print '10 first clusters allocation: {}'.format(clustersRDD.take(10))

# TEST Amazon record with the most tokens (1d)
assert clustersRDD.take(10)==[134, 126, 209, 221, 401, 485, 197, 269, 296, 265], 'valor incorreto'
print 'ok'


I am working on the file "SparkML - 01 Regression.ipynb".
The first task is:
Convert Spark Dataframe to Pandas Dataframe
Can you write Python code for it?


I am working on the file "SparkML - 01 Regression.ipynb".
The first task is:
Fit Linear Regression Model
Can you write Python code for it?

from pyspark.ml.regression import LinearRegression

lr = LinearRegression()
print(lr.explainParams())

lr.setLabelCol("EP")
lr.setFeaturesCol("features")
model = lr.fit(df_vect)

type(model)


I am working on the file "SparkML - 01 Regression.ipynb".
The first task is:
Load Data
Can you write Python code for it?

df = spark.read.format("csv").option("header","true")\
.option("inferSchema","true").load("/data/Combined_Cycle_Power_Plant.csv")

df.show()

df.cache()


I am working on the file "SparkML - 01 Regression.ipynb".
The first task is:
Load the persisted model from the disk
Can you write Python code for it?

saved_model = PipelineModel.load("/tmp/lr-pipeline")
saved_model.stages[1].coefficients

saved_model.transform(df).show()

df_train, df_test = df.randomSplit(weights=[0.7, 0.3], seed = 200)

pipelineModel = pipeline.fit(df_train)
evaluator.evaluate(pipelineModel.transform(df_test))


I am working on the file "SparkML - 01 Regression.ipynb".
The first task is:
Predict
Can you write Python code for it?

df_pred = model.transform(df_vect)
df_pred.show()


I am working on the file "SparkML - 01 Regression.ipynb".
The first task is:
Verctorize the features
Can you write Python code for it?

from pyspark.ml.feature import *

vectorizer = VectorAssembler()
vectorizer.setInputCols(["AT", "V", "AP", "RH"])
vectorizer.setOutputCol("features")

df_vect = vectorizer.transform(df)
df_vect.show(10, False)

print(vectorizer.explainParams())


I am working on the file "SparkML - 01 Regression.ipynb".
The first task is:
View model summary
Can you write Python code for it?

print("R2:", model.summary.r2)
print("Intercept: ", model.intercept, "Coefficients", model.coefficients)


I am working on the file "SparkML - 02 Credit Default.ipynb".
The first task is:
As I can see, there are number of columns of string type - checking_balance, credit_history etc.
Let me define a function that take a catgorical column and pass it through StringIndexer and OneHotEncoder it gives back a dataframe with same column name as the original categorical column. It reurns a new dataframe that contains categorical column replaced by OneHotEncoded vector.
Find all columns of String datatype
Transform each string column type into OneHotEncoded value and collect distinct values for each categorical column in list as shown below.
Can you write Python code for it?

cols = credit.columns
cols.remove("default")
cols

from pyspark.ml import Model, Estimator

class DFOneHotEncoderModel(Model):

    def get_col_labels(self):

        cols = []
        feature_columns = [c for c in self.columns if not c == self.label_column]

        for col in feature_columns:
            if col in self.categorical_fields:
                string_indexer, _ = self.categorical_fields[col]
                values = string_indexer.labels
                values = values[:-1] if self.drop_last else values
                values = [col + "_" + v for v in values]
                cols.extend(values)
            else:
                cols.append(col)

        return cols

    def transform(self, df, params= None):

        for colname in self.categorical_fields:
            string_indexer, one_hot_encoder = self.categorical_fields[colname]

            df = string_indexer.transform(df)
            df = df.drop(colname)
            df = df.withColumnRenamed(colname + "_idx", colname)

            if one_hot_encoder:
                df = one_hot_encoder.transform(df)
                df = df.drop(colname)
                df = df.withColumnRenamed(colname + "_ohe", colname)

        return df

class DFOneHotEncoder(Estimator):

    def __init__(self, label_column, categorical_fields= None,  one_hot = True, drop_last = True):
        self.categorical_fields = None
        self.one_hot = one_hot
        self.drop_last = drop_last
        self.label_column = label_column

        if not categorical_fields is None:
            self.categorical_fields = dict([(c, None) for c in categorical_fields])

    def fit(self, df):
        cols = df.columns
        if self.categorical_fields is None:
            self.categorical_fields = dict([(col, None) for col, dtype in df.dtypes if dtype == "string"])

        for colname in self.categorical_fields:
            string_indexer =...


I am working on the file "SparkML - 02 Credit Default.ipynb".
The first task is:
Build a RandomForest Classifier
Can you write Python code for it?

forest = RandomForestClassifier(labelCol="default", featuresCol="features", seed = 123)
forest_model = forest.fit(df_train)


I am working on the file "SparkML - 02 Credit Default.ipynb".
The first task is:
Check version of the libraries. For this notebook, I am using Spark 2.2.0
Load Dataset
You can download the dataset from here
Can you write Python code for it?

credit = spark.read.options(header = True, inferSchema = True).csv("/data/credit-default.csv").cache()
print("Total number of records: ", credit.count())
credit.limit(10).toPandas().head()
# Taking 10 samples records from spark dtaframe into a Pandas dataframe to display the values
# I prefer the pandas dataframe display to that by spark dataframe show function.


I am working on the file "SparkML - 02 Credit Default.ipynb".
The first task is:
Create a list of columns except the label column
Use a vector assembler to transform all features into a single feature column
Can you write Python code for it?

df_vect = VectorAssembler(inputCols = cols, outputCol="features").transform(df)
df_vect.select("features", "default").limit(5).toPandas()


I am working on the file "SparkML - 02 Credit Default.ipynb".
The first task is:
Evaluate
Can you write Python code for it?

evaluator = evaluation.MulticlassClassificationEvaluator(labelCol="default",
                                        metricName="accuracy", predictionCol="prediction")
evaluator.evaluate(df_test_pred)

print("Total number of features: ", forest_model.numFeatures, "\nOrder of feature importance: \n")
pd.DataFrame({"importance": forest_model.featureImportances.toArray(),
              "feature": model.get_col_labels()
             }).sort_values("importance", ascending = False)


I am working on the file "SparkML - 02 Credit Default.ipynb".
The first task is:
Let me spot check whether OneHotEncode worked ok.
Can you write Python code for it?

credit.first()

pd.DataFrame({"feature": model.get_col_labels(), "value": df_vect.select("features").first().features})

df_train, df_test = df_vect.randomSplit(weights=[0.7, 0.3], seed=1)
df_train.count(), df_test.count()


I am working on the file "SparkML - 02 Credit Default.ipynb".
The first task is:
View the schema
Can you write Python code for it?


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
Apply LogisticRegression classifier
Can you write Python code for it?

from pyspark.ml.classification import LogisticRegression

lr = LogisticRegression(maxIter=10000, regParam=0.1, elasticNetParam=0.0,
                        featuresCol="features", labelCol="sentiment_idx")


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
Apply StringIndexer to conver the sentiment column from String type to number type - this is prerequisit to apply the LogisticRegression algorithm.
Can you write Python code for it?

from pyspark.ml.feature import StringIndexer

string_indexer = StringIndexer(inputCol="sentiment", outputCol="sentiment_idx")
string_indexer_model = string_indexer.fit(idf_data)
label_encoded = string_indexer_model.transform(idf_data)

label_encoded.select("sentiment", "sentiment_idx").show()


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
Apply the the udf on the imdb dataframe.
Can you write Python code for it?

imdb_clean = imdb.withColumn("content", expr("remove_html_tags(content)")).cache()
imdb_clean.sample(False, 0.001, 1).first().content


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
Density report shows, the count_vectors has very low density which illustrate the benefit of the choice of DenseVector for this column.
Now, calculate tfidf for the document.
Can you write Python code for it?

idf = IDF(inputCol="count_vectors", outputCol="features")
idf_model = idf.fit(count_vectorized)
idf_data = idf_model.transform(count_vectorized)

idf_data.sample(False, 0.001, 1).first().features

idf_data.printSchema()


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
Each line is a self contained json doc. Load the dataset using spark reader specifying the file format as json. As we see above size of the file is 66 MB, we should at least 2 partitons, since I am using dual core system, I will repartition the data to 4. Also will cache the data after repartitioning.
Can you write Python code for it?


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
Find area under the curve. Closer to 1 is better
Can you write Python code for it?

training_summary.areaUnderROC

predictions = lr_model.transform(testing).withColumn("match", expr("prediction == sentiment_idx"))

predictions.select("prediction", "sentiment_idx", "sentiment", "match").sample(False, 0.01).show(10)

predictions.groupBy("sentiment_idx").pivot("prediction").count().show()


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
Find the accuracy of the prediction
Can you write Python code for it?

accuracy = predictions.select(expr("sum(cast(match as int))")).first()[0] / predictions.count()
accuracy


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
From the training summary find out the cost decay of the model.
Can you write Python code for it?

training_summary = lr_model.summary

import pandas as pd
import matplotlib.pyplot as plt

%matplotlib inline

pd.Series(training_summary.objectiveHistory).plot()
plt.xlabel("Iteration")
plt.ylabel("Cost")


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
IMDB comments dataset has been stored in the following location
Can you write Python code for it?


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
Register a UDF function to clean the comment from the html tags. If BeautifulSoup is not installed, you can install it using pip
(shell command)
$ pip install BeautifulSoup4
Can you write Python code for it?

from bs4 import BeautifulSoup
from pyspark.sql.types import *
import re
def remove_html_tags(text):
    text = BeautifulSoup(text, "html5lib").text.lower() #removed html tags
    text = re.sub("[\W]+", " ", text)
    return text

spark.udf.register("remove_html_tags", remove_html_tags, StringType())


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
SparkVector represents a vector of 103999, that means in the dataset (corpus) there are 103999 unique terms. Per document, only a few will be present. Find density of each count_vectors.
Can you write Python code for it?

vocab_len = len(count_vectorizer_model.vocabulary)
spark.udf.register("density", lambda r: r.numNonzeros() / vocab_len, DoubleType())
count_vectorized.select(expr("density(count_vectors) density")).show()


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
Split the data into traininf and testing groups with 70/30 ratio. Cache the dataframe so that training runs faster.
Can you write Python code for it?

training, testing = label_encoded.randomSplit(weights=[0.7, 0.3], seed=1)
training.cache()
testing.cache()


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
Take a look at a few sample data
Can you write Python code for it?


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
Test the remove_html_tags function
Can you write Python code for it?


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
There are 50000 lines in the file. Let's the first line
Can you write Python code for it?


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
Total size of the file is 66MB
Can you write Python code for it?


I am working on the file "SparkML - 04 Text_Analysis.ipynb".
The first task is:
count_vectorized Dataframe contains a column count_vectors that is a SparseVector representing which term appears and how many times. The key is the index of all unique terms. You can find list of terms count_vectorizer_model.vocabulary. See below.
Can you write Python code for it?

print(count_vectorizer_model.vocabulary[:100], "\n\nTotal no of terms", len(count_vectorizer_model.vocabulary))

count_vectorized.show()


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Classification Models for recommendations
Idea is that based on the features of movies and users, can we estimated what will be the rating.
Example of features of the movies
- year of release
- number users who have rated high (4 or 5)
- number of ratings (usually for these of high magnitude, take a log of the count)
- associated genre
- genre properties such as number avg rating by the genre.
Features by users
- how many movies the user has rated high
- standard deviation of his/her rating
Let create a dataset that we can filter by genre.
Can you write Python code for it?

genre_rating = (movies
.withColumn("year", F.regexp_extract("title", r"(\d+)", 1))
.withColumn("genres", F.split("genres", "\|"))
.withColumn("genre", F.explode("genres"))
.join(ratings, on = "movieId")
.withColumn("timestamp", F.expr("from_unixtime(timestamp)"))
)

genre_rating.show()


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Create utility function to load data into a spark dataframe
Can you write Python code for it?

import re
import pandas as pd

import matplotlib.pyplot as plt

%matplotlib inline

base_dir = "/data/movielens/"

def cache_df(df, name):
    df.createOrReplaceTempView(name)
    spark.catalog.cacheTable(name)

def load(file):
    name = re.sub("[^A-Za-z0-9]", "_", file[:-4])
    df = spark.read.option("header", True).option("inferSchema", True).csv(base_dir + file)
    cache_df(df, name)
    df.alias(name)
    return df


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Devide the dataset into training and test
Can you write Python code for it?

df_train, df_test = (ratings.randomSplit([0.7, 0.3], seed = 1))
cache_df(df_train, "df_train")


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Does all the movie id mentioned in rating present in the movies table as well? Expected count from the statement below should be 0, if all values are present.
Can you write Python code for it?

ratings.join(movies, on = ["movieId"], how = "leftanti").show()


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Evaluate the recommendations
One more let's see the recommendations for the user movie combination in the df_test dataset.
Can you write Python code for it?


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Find a few users who have moderate number of ratings.
Can you write Python code for it?

rating_count_by_user.filter("rating_count = 25").show()

reco_for_user(100).show()


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Find number of unique users.
Can you write Python code for it?


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Find root mean squared error based on the df_test
Can you write Python code for it?

(als_model
.transform(df_test)
.filter("not isnan(prediction)")
.selectExpr("sqrt(avg(pow((rating - prediction), 2))) test_rmse")
).show()


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Find the average rating by each user and rating count.
Can you write Python code for it?

avg_rating = ratings.groupBy("userId").agg(F.count("*").alias("count"), F.avg("rating").alias("avg_rating"))
avg_rating.show()


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Find the distribution of rating by plotting histogram.
Can you write Python code for it?


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Find top 10 movies based on the highest avg rating. Each top movies must have at least 100 ratings to make the avg to reliable.
Can you write Python code for it?

top10 = (ratings
 .groupBy("movieId")
 .agg(F.avg("rating").alias("avg_rating"), F.count("*").alias("rating_count"))
 .filter("rating_count > 100")
 .join(movies, on = "movieId")
 .select("movieId", "avg_rating", "title")
 .orderBy(F.desc("avg_rating"))
 .limit(10)
)
top10.show()


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
How many rating levels are there? Find count by each rating level
Can you write Python code for it?


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
How the average rating have changed over years.
Can you write Python code for it?

avg_rating_by_year = (ratings
 .withColumn("year", F.expr("year(from_unixtime(timestamp))"))
 .groupBy("year")
 .agg(F.avg("rating").alias("avg_rating"))
).toPandas()

avg_rating_by_year = avg_rating_by_year.sort_values("year")
avg_rating_by_year.index = avg_rating_by_year.year

avg_rating_by_year.avg_rating.plot()

plt.ylabel("Avg Rating")


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Like item Factors we can see the user factors representing the latent propeties of the user.
Can you write Python code for it?


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Number of unique values of the genres
Can you write Python code for it?

genre_rating.selectExpr("count(distinct genre)").first()

avg_rating_by_genre = genre_rating.groupBy("genre").avg("rating").toPandas()
avg_rating_by_genre.index = avg_rating_by_genre["genre"]
avg_rating_by_genre["avg(rating)"].plot.bar()


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Predict the rating for each movie and user combination in the df_test dataset.
Can you write Python code for it?

df_prediction = als_model.transform(df_test)
df_prediction.show()


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
See movie recommendations for a given user. See the recommendations for one of most active users.
Can you write Python code for it?

def reco_for_user(userId):
    return (
            als_model
             .recommendForUserSubset(df_train.filter(F.col("userId") == userId), 500)
             .withColumn("recommendation", F.explode("recommendations"))
             .withColumn("movieId", F.expr("recommendation.movieId"))
             .withColumn("recommended_rating", F.expr("recommendation.rating"))
             .drop("recommendations")
             .drop("recommendation")
             .join(ratings.filter(F.col("userId") == userId), on = "movieId", how = "leftanti")
             .join(movies, on = "movieId")
             .orderBy(F.desc("recommended_rating"))
            )
reco_for_user(547).show()


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Take a look at movies dataset. It shows by default first 20 records.
Can you write Python code for it?


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
View the iterm factors. These factors are found by ALS model. These represent latent properties of the each movie. Rank or the dimension is 10 as we specied in the ALS model.
Can you write Python code for it?


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
What is distribution of the number of rating by each movie
Can you write Python code for it?


I am working on the file "SparkML - 06 Movie Recommendation.ipynb".
The first task is:
Which movies are more consistent rating from users. We can take standard deviation as measure of consistency.
Can you write Python code for it?

(ratings
 .groupBy("movieId")
 .agg(F.avg("rating")
       , F.stddev("rating").alias("std")
       , F.count("*").alias("count"))
 .filter("not isnan(std)")
 .orderBy(F.desc("std"))
).show()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Average CTR
| Column  |  Description |
|---|---|
|avg_ctr_ad_id, avg_ctr_publisher_id, avg_ctr_advertiser_id, avg_ctr_campain_id, avg_ctr_entity_id_country … | Average CTR (#clicks / #views) given some categorical combinations and CTR confidence (details on Part II post). Eg. P(click  category01, category02).
Can you write Python code for it?

events.cache()
clicks_train.cache()
documents_meta.cache()
promoted_contents.cache()

avg_ctrs_by_ad_id = clicks_train.groupBy("ad_id").agg(F.avg("clicked").alias("avg_ctr_by_ad_id"))
%show avg_ctrs_by_ad_id

avg_ctrs_by_campaign_id = (clicks_train
.join(promoted_contents, on = "ad_id")
.groupBy("campaign_id")
.agg(F.avg("clicked").alias("avg_ctr_by_campaign_id")))

%show avg_ctrs_by_campaign_id

avg_ctrs_by_advertiser_id = (clicks_train
.join(promoted_contents, on = "ad_id")
.groupBy("advertiser_id")
.agg(F.avg("clicked").alias("avg_ctr_by_advertiser_id"))
.cache()
)

%show avg_ctrs_by_advertiser_id

avg_ctrs_by_document_id = (clicks_train
.join(promoted_contents, on = "ad_id")
.groupBy("document_id")
.agg(F.avg("clicked").alias("avg_ctr_by_document_id"))
.cache()
)

%show avg_ctrs_by_document_id

avg_ctrs_by_time = (events
 .join(documents_meta, on = "document_id", how = "left")
 .join(clicks_train, on = "display_id", how = "left")
 .join(promoted_contents, on = "ad_id", how = "left")
 .withColumn("total_clicks_by_ad_id"
    , F.expr("SUM(clicked) OVER (PARTITION BY ad_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND -1 FOLLOWING)"))
 .withColumn("total_events_by_ad_id"
    , F.expr("COUNT(*) OVER (PARTITION BY ad_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND -1 FOLLOWING)"))

 .withColumn("total_clicks_by_advertiser_id"
    , F.expr("SUM(clicked) OVER (PARTITION BY advertiser_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND -1 FOLLOWING)"))
 .withColumn("total_events_by_advertiser_id"
    , F.expr("COUNT(*) OVER (PARTITION BY advertiser_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND -1 FOLLOWING)"))

 .withColumn("total_clicks_by_campaign_id"
    , F.expr("SUM(clicked) OVER (PARTITION BY campaign_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND -1 FOLLOWING)"))
 .withColumn("total_events_by_campaign_id"
    , F.expr("COUNT(*) OVER (PARTITION BY campaign_id ORDER BY timestamp ROWS BETWEEN UNBOUNDED PRECEDING AND -1 FOLLOWING)"))

...


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Average events by user
Can you write Python code for it?


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Clicks
Can you write Python code for it?

clicks_train = load("clicks_train")
clicks_train.show()

clicks_test = load("clicks_test")
clicks_test.show()

clicks_train.count(), clicks_test.count()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Common ad_id in training and test datasets
Can you write Python code for it?

%time clicks_train.select("ad_id").intersect(clicks_test.select("ad_id")).count()

1- 316035/381385


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Consier, the CTR as baseline for click prediction. Using CTR as based, calculate the MAP (mean avg precision)
Can you write Python code for it?

#y_pred = clicks_train.join(ctrs.select("ad_id", "ctr"), on = ["ad_id"], how="left").select("ctr").toPandas()["ctr"]
#y_true = clicks_train.select("clicked").toPandas()["clicked"]

#%time metrics.average_precision_score(y_true, y_pred)

clicks_test_baseline = clicks_test.join(ctrs.select("ad_id", "ctr"), on = ["ad_id"], how="left")
clicks_test_baseline.show()

clicks_test_baseline.groupBy("display_id").count()

clicks_test_baseline.groupBy("display_id").agg(F.sum("ctr").alias("ap")).selectExpr("avg(ap)").show()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Detect and impute null values
Can you write Python code for it?

def show_null_counts(df):
    null_testers = ["sum(if(isnull(%s), 1, 0)) %s" % (f, f) for f in df.columns]
    null_counts = df.selectExpr(*null_testers).toPandas().T
    null_counts.columns = ["Count"]
    null_counts["pct"] = null_counts.Count/df.count()
    null_counts["dtype"] = [t[1] for t in df.dtypes]
    print(null_counts.to_string())

show_null_counts(df.selectExpr(*features))

df_trunc = df.selectExpr(*features)
distinct_counts = df_trunc.selectExpr(*["approx_count_distinct(%s)"
                        % f for f in df.selectExpr(*features).columns]).toPandas()
print(distinct_counts.T.to_string())

fill_na_values = {"platform": "<null>"
    , "source_id": "<null>"
    , "publisher_id": "<null>"
    , "avg_ctr_by_ad_id": 0.0
    , "avg_ctr_by_campaign_id": 0.0
    , "avg_ctr_by_document_id": 0.0
    , "avg_ctr_by_advertiser_id": 0.0
    , "country": "null"
    , "state": "null"
    , "doc_age": -1}

df_null_removed = df.selectExpr(*features).na.fill(fill_na_values)
show_null_counts(df_null_removed)


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Do all the display_id in the clicks_train present in events?
Can you write Python code for it?

%time clicks_train.select("display_id").distinct().join(events, on = ["display_id"], how = "left_anti").count()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Document Attributes
Document Meta Data
Can you write Python code for it?

documents_meta = load("documents_meta")
documents_meta.show(5, False)

documents_meta.count()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Document Categories
Can you write Python code for it?

documents_categories = load("documents_categories").drop_duplicates(["document_id", "category_id"])
documents_categories.printSchema()
documents_categories.show(5, False)

documents_categories.count()

documents_categories.select("category_id").distinct().count()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Document entities
Can you write Python code for it?

documents_entities = load("documents_entities").drop_duplicates(["document_id", "entity_id"])
documents_entities.show(5, False)


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Document topics
Can you write Python code for it?

documents_topics = load("documents_topics").drop_duplicates(["document_id","topic_id"])
documents_topics.show(5, False)

print(sorted([v.topic_id for v in documents_topics.select("topic_id").distinct().collect()]))


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Does ad_id in the clicks dataset have meta data info in promoted_contents?
Can you write Python code for it?

%time (clicks_train.select("ad_id").union(clicks_test.select("ad_id"))\
       .join(promoted_contents, on = ["ad_id"], how = "leftanti").count())


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Does the timestamp exist for each record in events?
Can you write Python code for it?


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Does the timestamp in clicks_train and clicks_test have overlap?
Can you write Python code for it?

%time clicks_train.join(events, on = ["display_id"]).select("timestamp").describe().show()

%time clicks_test.join(events, on = ["display_id"]).select("timestamp").describe().show()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
ERD
https://drive.google.com/open?id=1dHAdBT84rEDf3WiE7FSfFrpnGQVL0wUSsWAGEH7ywZg
Can you write Python code for it?

from datetime import datetime
import matplotlib.pyplot as plt

import pyspark.sql.functions as F
from pyspark.sql.window import Window
import numpy as np
import pandas as pd
from sklearn import metrics

pd.options.display.max_columns = 1000
pd.options.display.max_rows = 10

fast_mode = True

%matplotlib inline

from IPython.core.magic import register_line_magic

@register_line_magic
def show(line, n = 5):
    return eval(line).limit(n).toPandas()

@register_line_magic
def sql(line, n = 10):
    return spark.sql(line)

base_path = "/data/kaggle/outbrain_ctr/parquet/"

def cache_df(df, name, sorage_level = StorageLevel.MEMORY_ONLY):
    df.createOrReplaceTempView(name)
    spark.catalog.cacheTable(name)

def load(name, rebase_timestamp = False, cache = True):
    df = spark.read.load(base_path + name)

    if rebase_timestamp and "timestamp" in df.columns:
        df = df.withColumn("timestamp"
            , F.expr("cast(from_unixtime(cast((timestamp + 1465876799998)/1000 as int)) as timestamp)"))
    if cache:
        cache_df(df, name)

    df.alias(name)
    print("Number of partitions for df %s: %d" % (name, df.rdd.getNumPartitions()))

    return df

!ls -1 /data/kaggle/outbrain_ctr/parquet/


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Events
Can you write Python code for it?

events = load("events")
events.show()

events = events.withColumn("timestamp", F.expr("from_unixtime(cast((timestamp + 1465876799998)/1000 as int))"))
events.show()

events = load("events", rebase_timestamp=True)

events.count()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Events
| Column  |  Description |
|---|---|
|event_local_hour (binned), event_weekend | Event timestamps were in UTC-4, so I processed event geolocation to get timezones and adjust for users' local time. They were binned in periods like morning, afternoon, midday, evening, night. A flag indicating whether it was a weekend was also included. The assumption here is that time influences the kind of content users will read.
|event_country, event_country_state | The field event_geolocation was parsed to extract the user’s country and state in a page visit.
|ad_id, doc_event_id, doc_ad_id, ad_advertiser, … | All of the original categorical fields were One-Hot Encoded to be used by the models, generating about 126,000 features.
Can you write Python code for it?

events.selectExpr("uuid", "geo_location"
                  , "split(geo_location, '>')[0] country"
                  , "split(geo_location, '>')[1] state"
                 ).show()

events.selectExpr("split(geo_location, '>')[0] country").distinct().toPandas()

(events
 .selectExpr("display_id", "timestamp", "hour(timestamp) hour")
 .withColumn("day_session", F.expr("hour % 8"))).show()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Find 99, 95 and 90 percentile values of the view counts of the ads.
Can you write Python code for it?

view_counts = ctrs.select("view_count").toPandas()
np.percentile(view_counts["view_count"], [99, 95, 90])


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Find avg ctr by advertiser.
Can you write Python code for it?

avg_ctr_by_advertiser = promoted_contents.join(ctrs, on = "ad_id").groupBy("advertiser_id")\
.agg(F.avg("ctr").alias("avg_ctr"))

avg_ctr_by_advertiser.show()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Find avg ctr by document.
Can you write Python code for it?

avg_ctr_by_document = promoted_contents.join(ctrs, on = "ad_id").groupBy("document_id")\
.agg(F.avg("ctr").alias("avg_ctr"))

avg_ctr_by_document.show()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Find count of unique entity_ids.
Can you write Python code for it?


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Find stats around the number of entities per document.
Can you write Python code for it?


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Group the data by document_id and pack the other information in a array field.
Can you write Python code for it?

documents_categories_n = documents_categories\
.withColumn("pair", F.struct("id", "confidence_level"))\
.groupBy("document_id")\
.agg(F.collect_list("pair").alias("categories"))

documents_categories_n.printSchema()

documents_categories_n.count()

documents_categories_n.show(3, False)


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
How many document categories are there?
Can you write Python code for it?


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
How many document entities are there?
Can you write Python code for it?

docs.select(F.explode("entities")).select("col.id").distinct().count()

from pyspark.ml.linalg import SparseVector, VectorUDT

def to_vector(values, n):
    if values is not None:
        values = sorted(values, key=lambda v: v.id)
        indices = [v.id for v in values]
        values = [v.confidence_level for v in values]
        return SparseVector(n, indices, values)
    return SparseVector(n, [], [])

spark.udf.register("to_vector", to_vector, VectorUDT())

docs_vectorized = docs\
.withColumn("topics_vector", F.expr("to_vector(topics, 300)"))\
.withColumn("categories_vector", F.expr("to_vector(categories, 97)"))
#.withColumn("entities_vector", F.expr("to_vector(entities, 1326009)"))

docs_vectorized.printSchema()
docs_vectorized.cache().count()

docs_vectorized.select("topics_vector").first()

docs_vectorized.select("categories_vector").count()

from pyspark.ml.feature import IDF, Tokenizer

if "topics_idf" in docs.columns:
    docs = docs.drop("topics_idf")
if "entities_idf" in docs.columns:
    docs = docs.drop("entities_idf")
if "categories_idf" in docs.columns:
    docs = docs.drop("categories_idf")

topics_idf = IDF(inputCol="topics_vector", outputCol="topics_idf")
entities_idf = IDF(inputCol="entities_vector", outputCol="entities_idf")
categories_idf = IDF(inputCol="categories_vector", outputCol="categories_idf")

df1 = docs_vectorized
df2 = topics_idf.fit(df1).transform(df1).cache()
df3 = categories_idf.fit(df2).transform(df2).cache()
#df4 = entities_idf.fit(df3).transform(df3).cache()

docs_idf = df3
docs_idf.printSchema()

docs_idf.select("document_id", "topics_idf", "categories_idf").first()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
How many source_ids are there?
Can you write Python code for it?


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
How many user-document combination does not have any click on ads?
Can you write Python code for it?

%time views_and_event.filter("page_views_count > 0 and events_count = 0").count()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Hypothesis: The page views record come web server logs. Events some user tracking devices such as omniture, google analytics. User may open the page, the after sometime, user may choose to view an ad.
User may view a page more than once. Find how many users have viewed the same page more than once.
Can you write Python code for it?

repeated_page_views = page_views.groupBy(["uuid", "document_id"]).count()\
        .filter("count > 1").orderBy(F.desc("count"))
repeated_page_views.show()

repeated_page_views.count()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Let's take a sample uuid and document_id to see whether the event records have a matching page_views record.
Can you write Python code for it?

%time page_views.filter("uuid = 'a34004004c3e50' and document_id = 140264").show()

%time events.filter("uuid = 'a34004004c3e50' and document_id = 140264").show()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Look at a sample uuid who has repeated visited a page to observe the pattern in the source, location and timestamp
Can you write Python code for it?

if repeated_page_views.count()>0:
    sample_record = repeated_page_views.sample(True, 0.1).take(1)[0]
    page_views.filter(F.col("uuid") == sample_record.uuid)\
        .filter(F.col("document_id") == sample_record.document_id).show()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Machine Learning
Can you write Python code for it?

df = load("merged_enriched", cache = False)
df.count()

features = [
   'platform'
 , 'source_id'
 , 'publisher_id'
 , 'topics_idf'
 , 'categories_idf'
 , 'avg_ctr_by_ad_id'
 , 'avg_ctr_by_campaign_id'
 , 'avg_ctr_by_document_id'
 , 'avg_ctr_by_advertiser_id'
 , "country"
 , "state"
 , "doc_age"
 , "session"
 , "ad_id"
 , "display_id"
 , "is_train"
 , "clicked"
]

df.selectExpr(*features).printSchema()

%show df

df1 = df


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Number of unique users
Can you write Python code for it?


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Page views by platform
Can you write Python code for it?

page_views_by_platform = page_views.groupBy("platform")\
.count().toPandas().set_index("platform").sort_index()
page_views_by_platform

page_views_by_platform.plot.pie(y = "count", labels = ["Desktop", "Mobile", "Tablet"]
                    , figsize = (8, 8), autopct = "%.2f", fontsize = 15)
plt.title("Page views by platform")


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Page views by traffic source
Can you write Python code for it?

page_views_by_traffic_source = page_views.groupBy("traffic_source").count().toPandas()
page_views_by_traffic_source = page_views_by_traffic_source.set_index("traffic_source").sort_index()
page_views_by_traffic_source

page_views_by_traffic_source.plot.pie(y = "count", labels = ["Internal", "Search", "Social"]
                    , figsize = (8, 8), autopct = "%.2f", fontsize = 15)
plt.title("Page views by traffic source")


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Page views table is nearly 100 GB in decompressed csv file. How many records are there?
Can you write Python code for it?


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Promoted_content stores the meta of the ads. Double check the ad_id is unique in this dataset.
Can you write Python code for it?


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
So, all the ad_id in clicks dataset exist in the promoted_contents.
Find average CTR for campaign
Can you write Python code for it?

avg_ctr_by_campaign = promoted_contents.join(ctrs, on = "ad_id").groupBy("campaign_id")\
.agg(F.avg("ctr").alias("avg_ctr"))

avg_ctr_by_campaign.show()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
So, display_id for each record in clicks_train is present in events dataset.
Check the same for clicks_test dataset.
Can you write Python code for it?

%time clicks_test.select("display_id").distinct().join(events, on = ["display_id"], how = "left_anti").count()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Some users are more frequent visitor that the others. Find the number of visitors based on the view counts.
Can you write Python code for it?

page_views_by_user = page_views.groupBy("uuid").count().groupBy("count").count()\
                    .toDF("view_count", "num_users").toPandas().sort_values("view_count")
page_views_by_user

page_views_by_user.iloc[:20, :].plot.bar("view_count", "num_users")


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
To build confidence on the CTR, filter out the ads with fewer than 100 views (approx 99 percentile value)
Can you write Python code for it?

%time ctrs.filter("view_count>100").select("ctr").toPandas()["ctr"].plot.hist(bins = 50, density = True)
plt.xlabel("CTR")
plt.ylabel("Frquency (normalized)")


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Top countries by ad clicks
Can you write Python code for it?

events.selectExpr("split(geo_location, '>')[0] country")\
.groupBy("country").count().orderBy(F.desc("count")).show()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Topic_id seems already indexed. And there 300 topics are there.
Can you write Python code for it?

documents_topics.select("topic_id").distinct().count()

documents_topics_n = documents_topics\
.toDF("document_id", "id", "confidence_level")\
.withColumn("pair", F.struct("id", "confidence_level"))\
.groupBy("document_id")\
.agg(F.collect_list("pair").alias("topics"))

documents_topics_n.printSchema()

documents_topics_n.count()


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Vectorize and caculate weighted IDF scores
How many document topics are there?
Can you write Python code for it?


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
Verify whether the document_id is unique in document_meta.
Can you write Python code for it?


I am working on the file "SparkML - 07 Click Prediction (Outbrain dataset).ipynb".
The first task is:
category_id is not indexed. Using string indexer to index it. One Hot Encoding is another option. Here I am not doing it because the I want to use the confidence_level to weigh in.
Can you write Python code for it?

from pyspark.ml.feature import StringIndexer

#String indexer required string column
documents_categories = documents_categories.withColumn("category_id", F.expr("cast(category_id as string)"))

if "id" in documents_categories.columns:
    documents_categories = documents_categories.drop("id")

category_indexer = StringIndexer(inputCol="category_id", outputCol="id")
documents_categories = category_indexer.fit(documents_categories).transform(documents_categories)

documents_categories.show()


I am working on the file "Spark_Notes/Spark_Histograms/Spark_DataFrame_Frequency_Histograms.ipynb".
The first task is:
Compute the histogram
Can you write Python code for it?

# import the computeHistogram function
# see implementation details at:
# https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_Histograms/python/sparkhistogram/histogram.py

# requires the package sparkhistogram
! pip install sparkhistogram

from sparkhistogram import computeHistogram

# Compute the histogram using the computeHistogram function

histogram = computeHistogram(df, "random_value", -20, 90, 11)

# Alternative syntax: compute the histogram using transform on the DataFrame
# requires Spark 3.3.0 or higher
# histogram = df.transform(computeHistogram, "random_value", -20, 90, 11)

# this triggers the computation as show() is an action
histogram.show()

# Fetch the histogram data into a Pandas DataFrame for visualization
# At this stage data is reduced to a small number of rows (one row per bin)
# so it can be easily handled by the local machine/driver

# toPandas() is an action and triggers the computation
hist_pandasDF = histogram.toPandas()

hist_pandasDF

# Optionally normalize the event count into a frequency
# dividing by the total number of events

hist_pandasDF["frequency"] = hist_pandasDF["count"] / sum(hist_pandasDF["count"])

hist_pandasDF


I am working on the file "Spark_Notes/Spark_Histograms/Spark_DataFrame_Frequency_Histograms.ipynb".
The first task is:
Generate a DataFrame with toy data for demo purposes
Can you write Python code for it?

num_events = 100
scale = 100
seed = 4242

df = spark.sql(f"select random({seed}) * {scale} as random_value from range({num_events})")

df.show(5)


I am working on the file "Spark_Notes/Spark_Histograms/Spark_DataFrame_Frequency_Histograms.ipynb".
The first task is:
Note added
Use this to define the computeHistogram function if you cannot pip install sparkhistogram
Can you write Python code for it?

def computeHistogram(df: "DataFrame", value_col: str, min: float, max: float, bins: int) -> "DataFrame":
    """ This is a dataframe function to compute the count/frequecy histogram of a column

        Parameters
        ----------
        df: the dataframe with the data to compute
        value_col: column name on which to compute the histogram
        min: minimum value in the histogram
        max: maximum value in the histogram
        bins: number of histogram buckets to compute

        Output DataFrame
        ----------------
        bucket: the bucket number, range from 1 to bins (included)
        value: midpoint value of the given bucket
        count: number of values in the bucket
    """
    step = (max - min) / bins
    # this will be used to fill in for missing buckets, i.e. buckets with no corresponding values
    df_buckets = spark.sql(f"select id+1 as bucket from range({bins})")

    histdf = (df
              .selectExpr(f"width_bucket({value_col}, {min}, {max}, {bins}) as bucket")
              .groupBy("bucket")
              .count()
              .join(df_buckets, "bucket", "right_outer") # add missing buckets and remove buckets out of range
              .selectExpr("bucket", f"{min} + (bucket - 1/2) * {step} as value", # use center value of the buckets
                          "nvl(count, 0) as count") # buckets with no values will have a count of 0
              .orderBy("bucket")
             )
    return histdf


I am working on the file "Spark_Notes/Spark_Histograms/Spark_DataFrame_Weighted_Histograms.ipynb".
The first task is:
Histogram plotting
This plots the weighted histogram: wheighted sum of the number of per bin.
Can you write Python code for it?

import matplotlib.pyplot as plt
plt.style.use('seaborn-darkgrid')
plt.rcParams.update({'font.size': 20, 'figure.figsize': [14,10]})

f, ax = plt.subplots()

# histogram data
x = hist_pandasDF["value"]
y = hist_pandasDF["weighted_sum"]

# bar plot
ax.bar(x, y, width = 3.0, color='red')

ax.set_xlabel('Bucket values')
ax.set_ylabel('Sum of the weights (weighted count)')
ax.set_title("Distribution of event weight sums")

# Label for the resonances spectrum peaks
txt_opts = {'horizontalalignment': 'center',
            'verticalalignment': 'center',
            'transform': ax.transAxes}

plt.show()

spark.stop()


I am working on the file "Spark_Notes/Spark_Histograms/Spark_DataFrame_Weighted_Histograms.ipynb".
The first task is:
Note added
Use this to define the computeWeightedHistogram function if you cannot pip install sparkhistogram
Can you write Python code for it?

from pyspark.sql.functions import sum

def computeWeightedHistogram(df: "DataFrame", value_col: str, weight_col: str,
                             min_val: float, max_val: float, bins: int) -> "DataFrame":
    """ This is a dataframe function to compute the weighted histogram of a DataFrame column.
        A weighted histogram is a generalization of a frequency histogram.
        Parameters
        ----------
        df: the dataframe with the data to compute
        value_col: column name on which to compute the histogram
                   the column needs to be of numeric type
        weight_col: numeric-type column with the weights,
                    the bucket value is computed as sum of weights.
                    If all weight are set to 1, you get a frequency histogram
        min_val: minimum value in the histogram
        max_val: maximum value in the histogram
        bins: number of histogram buckets to compute
        Output DataFrame
        ----------------
        bucket: the bucket number, range from 1 to bins (included)
        value: midpoint value of the given bucket
        weighted_sum: weighted sum of the number of values in the bucket
    """
    step = (max_val - min_val) / bins
    # this will be used to fill in for missing buckets, i.e. buckets with no corresponding values
    df_buckets = spark.sql(f"select id+1 as bucket from range({bins})")
    histdf = (df
              .selectExpr(f"width_bucket({value_col}, {min_val}, {max_val}, {bins}) as bucket", f"{weight_col}")
              .groupBy("bucket")
              .agg(sum(f"{weight_col}").alias("weighted_sum"))  # sum the weights on the weight_col
              .join(df_buckets, "bucket", "right_outer") # add missing buckets and remove buckets out of range
              .selectExpr("bucket", f"{min_val} + (bucket - 0.5) * {step} as value",  # use center value of the buckets
                          "nvl(weighted_sum, 0) as weighted_sum") # buckets with no values will have a sum of 0
              .orderBy("bucket")
         ...


I am working on the file "Spark_Notes/Spark_Histograms/Spark_SQL_Frequency_Histograms.ipynb".
The first task is:
Compute the histogram using Spark SQL
Can you write Python code for it?

# This is a Spark SQL stattement to compute the count histogram of a given column

# Input parameters

table_name = "data" # table or temporary view containing the data
value_col = "random_value" # column name on which to compute the histogram
min = -20  # min: minimum value in the histogram
max = 90   # maximum value in the histogram
bins = 11  # number of histogram buckets to compute
step = (max - min) / bins

histogram = spark.sql(f"""
with hist as (
  select
    width_bucket({value_col}, {min}, {max}, {bins}) as bucket,
    count(*) as cnt
  from {table_name}
  group by bucket
  ),
buckets as (
  select id+1 as bucket from range({bins})
)
select
    bucket, {min} + (bucket - 0.5) * {step} as value,
    nvl(cnt, 0) as count
from hist right outer join buckets using(bucket)
order by bucket
""")


I am working on the file "Spark_Notes/Spark_Histograms/Spark_SQL_Frequency_Histograms.ipynb".
The first task is:
Fetch the histogram data it into the driver
The show and toPandas actions trigger the execution and fetches the histogram values
Can you write Python code for it?

# Output DataFrame description
# ----------------------------
# bucket: the bucket number, range from 1 to bins (included)
# value: midpoint value of the given bucket
# count: number of values in the bucket

# this triggers the computation as show() is an action
histogram.show()

# Fetch the histogram data into a Pandas DataFrame for visualization
# At this stage data is reduced to a small number of rows (one row per bin)
# so it can be easily handled by one machine

# toPandas() is an action and triggers the computation
hist_pandasDF = histogram.toPandas()

hist_pandasDF

# Optionally normalize the event count into a frequency
# dividing by the total number of events

hist_pandasDF["frequency"] = hist_pandasDF["count"] / sum(hist_pandasDF["count"])

hist_pandasDF


I am working on the file "Spark_Notes/Spark_Histograms/Spark_SQL_Frequency_Histograms.ipynb".
The first task is:
Histogram plotting
The first plot is a histogram with the event counts (number of events per bin).
The second plot is a histogram of the events frequencies (number of events per bin normalized by the sum of the events).
Can you write Python code for it?

import matplotlib.pyplot as plt
plt.style.use('seaborn-darkgrid')
plt.rcParams.update({'font.size': 20, 'figure.figsize': [14,10]})

f, ax = plt.subplots()

# histogram data
x = hist_pandasDF["value"]
y = hist_pandasDF["count"]

# bar plot
ax.bar(x, y, width = 3.0, color='red')

ax.set_xlabel("Bucket values")
ax.set_ylabel("Event count")
ax.set_title("Distribution of event counts")

# Label for the resonances spectrum peaks
txt_opts = {'horizontalalignment': 'center',
            'verticalalignment': 'center',
            'transform': ax.transAxes}

plt.show()

import matplotlib.pyplot as plt
plt.style.use('seaborn-darkgrid')
plt.rcParams.update({'font.size': 20, 'figure.figsize': [14,10]})

f, ax = plt.subplots()

# histogram data
x = hist_pandasDF["value"]
y = hist_pandasDF["frequency"]

# bar plot
ax.bar(x, y, width = 3.0, color='blue')

ax.set_xlabel("Bucket values")
ax.set_ylabel("Event frequency")
ax.set_title("Distribution of event frequencies")

# Label for the resonances spectrum peaks
txt_opts = {'horizontalalignment': 'center',
            'verticalalignment': 'center',
            'transform': ax.transAxes}

plt.show()

spark.stop()


I am working on the file "Spark_Notes/Spark_SQL_UDF_examples_Mandelbrot/Spark_SQL_UDF_example_notebook_Mandelbrot_Jupyter_Python.ipynb".
The first task is:
Compute and display the Mandelbrot set using Spark SQL using plain old-fashioned ASCII graphics for the output
Can you write Python code for it?

spark.sql("""
with
    x as (select id, -2.0 + 0.027*cast(id as Float) cR from range(0,95)),
    y as (select id, -1.1 + 0.045*cast(id as Float) cI from range(0,50))
select translate(cast(collect_list(substring(' .:::-----++++%%%%@@@@#### ',
       mandelbrot(x.cR, y.cI, 27), 1)) as string), ',', '') as Mandelbrot_Set
from y cross join x
group by y.id
order by y.id desc""").show(200, False)


I am working on the file "Spark_Notes/Spark_SQL_UDF_examples_Mandelbrot/Spark_SQL_UDF_example_notebook_Mandelbrot_Jupyter_Python.ipynb".
The first task is:
Mandelbrot in SQL, displayed with color
This computes and displays the Mandelbrot set on the terminal using color, with a blue palette.
Note: it requires a terminal or notebook with support for ANSI escape codes.
Can you write Python code for it?

result = spark.sql("""
with
    x as (select id, -2.0 + 0.027*cast(id as Float) cR from range(0,95)),
    y as (select id, -1.1 + 0.045*cast(id as Float) cI from range(0,50))
select translate(cast(collect_list(color) as String), ',' , '') as value
from y cross join x cross join values
     (0, concat('\u001B','[48;5;0m ','\u001B','[0m')),  -- Black
     (1, concat('\u001B','[48;5;15m ','\u001B','[0m')), -- White
     (2, concat('\u001B','[48;5;51m ','\u001B','[0m')), -- Light blue
     (3, concat('\u001B','[48;5;45m ','\u001B','[0m')),
     (4, concat('\u001B','[48;5;39m ','\u001B','[0m')),
     (5, concat('\u001B','[48;5;33m ','\u001B','[0m')),
     (6, concat('\u001B','[48;5;27m ','\u001B','[0m')),
     (7, concat('\u001B','[48;5;21m ','\u001B','[0m'))  -- Dark blue
     as palette(id, color)
where  cast(substring('012223333344445555666677770', mandelbrot(x.cR, y.cI, 27), 1) as Int) = palette.id
group by y.id
order by y.id desc""").collect()

# print out the result set of the query line by line
for line in result:
    print(line[0])


I am working on the file "Spark_Notes/Tests_mapInArrow.ipynb".
The first task is:
Test 2: square the array with mapInPandas
Can you write Python code for it?

%%time

# UDF function that squares the input
def UDF_pandas_square(iterator):
    for batch in iterator:
        yield batch*batch

df.mapInPandas(UDF_pandas_square, df.schema).write.format("noop").mode("overwrite").save()


I am working on the file "Spark_Notes/Tests_mapInArrow.ipynb".
The first task is:
mapInArrow tests
Test 3: dummy UDF using mapInArrow
Can you write Python code for it?

%%time

# A dummy UDF that just returns the input data
def UDF_dummy(iterator):
    for batch in iterator:
        yield batch

df.mapInArrow(UDF_dummy, df.schema).write.format("noop").mode("overwrite").save()

### Test 4: dummy UDF using mapInArrow and awkward array

%%time

# this requires pip install awkward
import awkward as ak

# a dummy UDF that convert back and forth to awkward arrays
# it just returns the input data
def UDF_dummy_with_awkward_array(iterator):
    for batch in iterator:
        b = ak.from_arrow(batch)
        yield from ak.to_arrow_table(b).to_batches()

df.mapInArrow(UDF_dummy_with_awkward_array, df.schema).write.format("noop").mode("overwrite").save()


I am working on the file "Spark_Notes/Tests_mapInArrow.ipynb".
The first task is:
mapInPAndas tests
Test 1 - dummy UDF
Can you write Python code for it?

%%time

# A dummy UDF that just returns the input data
def UDF_dummy(iterator):
    for batch in iterator:
        yield batch

df.mapInPandas(UDF_dummy, df.schema).write.format("noop").mode("overwrite").save()


I am working on the file "Spark_Physics/ATLAS_Higgs_opendata/H_ZZ_4l_analysis_basic_experiment_data.ipynb".
The first task is:
Apply basic cuts
More details on the cuts (filters applied to the event data) in the reference ATLAS paper on the discovery of the Higgs boson (mostly Section 4 and 4.1)
Can you write Python code for it?

# Apply filters to the input data
# only events with 4 leptons in the input data

# cut on lepton charge
# paper: "selecting two pairs of isolated leptons, each of which is comprised of two leptons with the same flavour and opposite charge"
df_events = df_events.filter("lep_charge[0] + lep_charge[1] + lep_charge[2] + lep_charge[3] == 0")

# cut on lepton type
# paper: "selecting two pairs of isolated leptons, each of which is comprised of two leptons with the same flavour and opposite charge"
df_events = df_events.filter("lep_type[0] + lep_type[1] + lep_type[2] + lep_type[3] in (44, 48, 52)")

print(f"Number of events after applying cuts: {df_events.count()}")


I am working on the file "Spark_Physics/ATLAS_Higgs_opendata/H_ZZ_4l_analysis_basic_experiment_data.ipynb".
The first task is:
Compute the invariant mass
This computes the 4-vectors sum for the 4-lepton system
using formulas from special relativity.
See also http://edu.itp.phys.ethz.ch/hs10/ppp1/2010_11_02.pdf
and https://en.wikipedia.org/wiki/Invariant_mass
Can you write Python code for it?

# This computes the 4-vectors sum for the 4-lepton system

df_4lep = df_events.selectExpr(
"lep_pt[0] * cos(lep_phi[0]) + lep_pt[1] * cos(lep_phi[1]) + lep_pt[2] * cos(lep_phi[2]) + lep_pt[3] * cos(lep_phi[3]) as Px",
"lep_pt[0] * sin(lep_phi[0]) + lep_pt[1] * sin(lep_phi[1]) + lep_pt[2] * sin(lep_phi[2]) + lep_pt[3] * sin(lep_phi[3]) as Py",
"lep_pt[0] * sinh(lep_eta[0]) + lep_pt[1] * sinh(lep_eta[1]) + lep_pt[2] * sinh(lep_eta[2]) + lep_pt[3] * sinh(lep_eta[3]) as Pz",
"lep_E[0] + lep_E[1] + lep_E[2] + lep_E[3] as E"
)

df_4lep.show(5)

df_4lep_invmass = df_4lep.selectExpr("sqrt(E * E - ( Px * Px + Py * Py + Pz * Pz))/1e3 as invmass_GeV")

df_4lep_invmass.show(5)

# This defines the DataFrame transformation to compute the histogram of invariant mass
# The result is a histogram with (energy) bin values and event counts foreach bin

# Requires sparkhistogram
# See https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_DataFrame_Histograms.md
from sparkhistogram import computeHistogram

# histogram parameters
min_val = 80
max_val = 250
num_bins = (max_val - min_val) / 5.0

# use the helper function computeHistogram in the package sparkhistogram
histogram_data = computeHistogram(df_4lep_invmass, "invmass_GeV", min_val, max_val, num_bins)

# The action toPandas() here triggers the computation.
# Histogram data is fetched into the driver as a Pandas Dataframe.

%time histogram_data_pandas=histogram_data.toPandas()

import numpy as np

# Computes statistical error on the data (histogram)
histogram_data_stat_errors = np.sqrt(histogram_data_pandas)

# This plots the data histogram with error bars

import matplotlib.pyplot as plt

plt.style.use('seaborn-darkgrid')
plt.rcParams.update({'font.size': 20, 'figure.figsize': [14,10]})

f, ax = plt.subplots()

x = histogram_data_pandas["value"]
y = histogram_data_pandas["count"]
err = histogram_data_stat_errors["count"]

# scatter plot
ax.plot(x, y, marker='o', color='red', linewidth=0)
#ax.errorbar(x, y, err, fmt = 'ro')
# histogram with error...


I am working on the file "Spark_Physics/ATLAS_Higgs_opendata/H_ZZ_4l_analysis_basic_experiment_data.ipynb".
The first task is:
Note on sparkhistogram
Use this to define the computeHistogram function if you cannot pip install sparkhistogram
Can you write Python code for it?


I am working on the file "Spark_Physics/ATLAS_Higgs_opendata/H_ZZ_4l_analysis_extra_cuts_montecarlo_data.ipynb".
The first task is:
Compute the invariant mass
This computes the 4-vectors sum for the 4-lepton system
using formulas from special relativity.
See also http://edu.itp.phys.ethz.ch/hs10/ppp1/2010_11_02.pdf
and https://en.wikipedia.org/wiki/Invariant_mass
Can you write Python code for it?

# This computes the 4-vectors sum for the 4-lepton system

df_4lep = df_lep_filtered.selectExpr(
"lep.lep_pt[0] * cos(lep.lep_phi[0]) + lep.lep_pt[1] * cos(lep.lep_phi[1]) + lep.lep_pt[2] * cos(lep.lep_phi[2]) + lep.lep_pt[3] * cos(lep.lep_phi[3]) as Px",
"lep.lep_pt[0] * sin(lep.lep_phi[0]) + lep.lep_pt[1] * sin(lep.lep_phi[1]) + lep.lep_pt[2] * sin(lep.lep_phi[2]) + lep.lep_pt[3] * sin(lep.lep_phi[3]) as Py",
"lep.lep_pt[0] * sinh(lep.lep_eta[0]) + lep.lep_pt[1] * sinh(lep.lep_eta[1]) + lep.lep_pt[2] * sinh(lep.lep_eta[2]) + lep.lep_pt[3] * sinh(lep.lep_eta[3]) as Pz",
"lep.lep_E[0] + lep.lep_E[1] + lep.lep_E[2] + lep.lep_E[3] as E"
)

df_4lep.show(5)

df_4lep_invmass = df_4lep.selectExpr("sqrt(E * E - ( Px * Px + Py * Py + Pz * Pz))/1e3 as invmass_GeV")

df_4lep_invmass.show(5)

# This defines the DataFrame transformation to compute the histogram of invariant mass
# The result is a histogram with (energy) bin values and event counts foreach bin

# Requires sparkhistogram
# See https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_DataFrame_Histograms.md
from sparkhistogram import computeHistogram

# histogram parameters
min_val = 80
max_val = 250
num_bins = (max_val - min_val) / 5.0

# use the helper function computeHistogram in the package sparkhistogram
histogram_data = computeHistogram(df_4lep_invmass, "invmass_GeV", min_val, max_val, num_bins)

# The action toPandas() here triggers the computation.
# Histogram data is fetched into the driver as a Pandas Dataframe.

%time histogram_data_pandas=histogram_data.toPandas()

import numpy as np

# Computes statistical error on the data (histogram)
histogram_data_stat_errors = np.sqrt(histogram_data_pandas)

# This plots the data histogram with error bars

import matplotlib.pyplot as plt

plt.style.use('seaborn-darkgrid')
plt.rcParams.update({'font.size': 20, 'figure.figsize': [14,10]})

f, ax = plt.subplots()

x = histogram_data_pandas["value"]
y = histogram_data_pandas["count"]
err = histogram_data_stat_errors["count"]

# scatter...


I am working on the file "Spark_Physics/CMS_Higgs_opendata/H_ZZ_4l_analysis_basic_monte_carlo_signal.ipynb".
The first task is:
H → ZZ* → 4$\mu$ - cuts and plot, using Monte Carlo signal data
(this is a step of the broader analsys)
Can you write Python code for it?

# Start the Spark Session
# This uses local mode for simplicity
# the use of findspark is optional

import findspark
findspark.init("/home/luca/Spark/spark-3.3.0-bin-hadoop3")

from pyspark.sql import SparkSession
spark = (SparkSession.builder
         .appName("H_ZZ_4Lep")
         .master("local[*]")
         .config("spark.driver.memory", "8g")
         .config("spark.sql.parquet.enableNestedColumnVectorizedReader", "true")
         .getOrCreate()
        )

# Read data with the candidate events
# Only Muon events for this reduced-scope notebook

path = "./"

df_MC_events_signal = spark.read.parquet(path + "SMHiggsToZZTo4L.parquet")

df_MC_events_signal.printSchema()

# Count the number of events before cuts (filter)

print(f"Number of events, MC signal: {df_MC_events_signal.count()}")


I am working on the file "Spark_Physics/CMS_Higgs_opendata/H_ZZ_4l_analysis_basic_monte_carlo_signal.ipynb".
The first task is:
Note on sparkhistogram
Use this to define the computeHistogram function if you cannot pip install sparkhistogram
Can you write Python code for it?


I am working on the file "Spark_Physics/Dimuon_mass_spectrum/Dimuon_mass_spectrum_histogram_Spark_DataFrame_Colab_version.ipynb".
The first task is:
Dimuon mass spectrum calculation with Spark DataFrame API
Can you write Python code for it?

# Start the Spark Session

from pyspark.sql import SparkSession
spark = (SparkSession.builder
         .appName("dimuon mass")
         .master("local[*]")
         .config("spark.driver.memory", "2g")
         .config("spark.sql.orc.enableNestedColumnVectorizedReader", "true")
         .getOrCreate()
        )

# Read data with the muon candidate events
# download data with wget as detailed above
# further details of the available datasets at
# https://github.com/LucaCanali/Miscellaneous/tree/master/Spark_Physics

path = "./"
df_muons = spark.read.orc(path + "Run2012BC_DoubleMuParked_Muons.orc")

df_muons.printSchema()
print(f"Number of events: {df_muons.count()}")

# Apply filters to the input data
# - select only events with 2 muons
# - select only events where the 2 muons have opposite charge

df_muons = df_muons.filter("nMuon == 2").filter("Muon_charge[0] != Muon_charge[1]")

# This computes the 4-vectors sum for the 2 moun system
# using formulas from special relativity, in the limit E >> muons rest mass
# see also http://edu.itp.phys.ethz.ch/hs10/ppp1/2010_11_02.pdf
# and https://en.wikipedia.org/wiki/Invariant_mass

df_with_dimuonmass = df_muons.selectExpr("""
     sqrt(2 * Muon_pt[0] * Muon_pt[1] *
             ( cosh(Muon_eta[0] - Muon_eta[1]) - cos(Muon_phi[0] - Muon_phi[1]) )
         ) as Dimuon_mass""")

# This defines the DataFrame transformation to compute the Dimuon mass spectrum
# The result is a histogram with (energy) bin values and event counts foreach bin

# Requires sparkhistogram
# See https://github.com/LucaCanali/Miscellaneous/blob/master/Spark_Notes/Spark_DataFrame_Histograms.md
from sparkhistogram import computeHistogram

# histogram parameters
min_val = 0.25
max_val = 300
num_bins = 30000

# use the helper function computeHistogram in the package sparkhistogram
histogram_data = computeHistogram(df_with_dimuonmass, "Dimuon_mass", min_val, max_val, num_bins)

# The action toPandas() here triggers the computation.
# Histogram data is fetched into the driver as a Pandas...


I am working on the file "Spark_Physics/Dimuon_mass_spectrum/Dimuon_mass_spectrum_histogram_Spark_DataFrame_Colab_version.ipynb".
The first task is:
Download the data
Can you write Python code for it?

# Download the data (2 GB) if not yet available locally

! wget https://sparkdltrigger.web.cern.ch/sparkdltrigger/Run2012BC_DoubleMuParked_Muons.orc


I am working on the file "Spark_Physics/HEP_benchmark/ADL_HEP_Query_Benchmark_Q1_Q5.ipynb".
The first task is:
Benchmark task: Q1
Plot the $𝐸^{𝑚𝑖𝑠𝑠}_T$ (missing transverse energy) of all events.
Can you write Python code for it?

# Compute the histogram for MET_pt
# The Spark function "width_bucket" is used to generate the histogram bucket number
# a groupBy operation with count is used to fill the histogram
# The result is a histogram with bins value and counts foreach bin (N_events)

min_val = 0
max_val = 100
num_bins = 100
step = (max_val - min_val) / num_bins

histogram_data = (
    df_events
        .selectExpr(f"width_bucket(MET_pt, {min_val}, {max_val}, {num_bins}) as bucket")
        .groupBy("bucket")
        .count()
        .orderBy("bucket")
    )

# convert bucket number to the corresponding value
histogram_data = histogram_data.selectExpr(f"round({min_val} + (bucket - 1/2) * {step},2) as value", "count as N_events")

# The action toPandas() here triggers the computation.
# Histogram data is fetched into the driver as a Pandas Dataframe.

%time histogram_data_pandas=histogram_data.toPandas()

import matplotlib.pyplot as plt
plt.style.use('seaborn-darkgrid')
plt.rcParams.update({'font.size': 20, 'figure.figsize': [14,10]})

# cut the first and last bin
x = histogram_data_pandas.iloc[1:-1]["value"]
y = histogram_data_pandas.iloc[1:-1]["N_events"]

# line plot
f, ax = plt.subplots()
ax.plot(x, y, '-')

ax.set_xlim(min_val, max_val)
ax.set_xlabel('$𝐸^{𝑚𝑖𝑠𝑠}_T$ (GeV)')
ax.set_ylabel('Number of Events')
ax.set_title("Distribution of $𝐸^{𝑚𝑖𝑠𝑠}_T$ ")

plt.show()


I am working on the file "Spark_Physics/HEP_benchmark/ADL_HEP_Query_Benchmark_Q1_Q5.ipynb".
The first task is:
Benchmark task: Q3
Plot the $𝑝_𝑇$ of jets with |𝜂| < 1 (𝜂 is the jet pseudorapidity).
Can you write Python code for it?

# Take Jet arrays for pt and eta and transform them to rows with explode()

df1 = df_events.selectExpr("explode(arrays_zip(Jet_pt, Jet_eta)) as Jet")
df1.printSchema()
df1.show(10, False)

# Apply a filter on Jet_eta

q3 = df1.select("Jet.Jet_pt").filter("abs(Jet.Jet_eta) < 1")
q3.show(10,False)

# Compute the histogram for Jet_pt
# The Spark function "width_bucket" is used to generate the histogram bucket number
# a groupBy operation with count is used to fill the histogram
# The result is a histogram with bins value and counts foreach bin (N_events)

min_val = 15
max_val = 60
num_bins = 100
step = (max_val - min_val) / num_bins

histogram_data = (
    q3
        .selectExpr(f"width_bucket(Jet_pt, {min_val}, {max_val}, {num_bins}) as bucket")
        .groupBy("bucket")
        .count()
        .orderBy("bucket")
    )

# convert bucket number to the corresponding value
histogram_data = histogram_data.selectExpr(f"round({min_val} + (bucket - 1/2) * {step},2) as value", "count as N_events")

# The action toPandas() here triggers the computation.
# Histogram data is fetched into the driver as a Pandas Dataframe.

%time histogram_data_pandas=histogram_data.toPandas()

import matplotlib.pyplot as plt
plt.style.use('seaborn-darkgrid')
plt.rcParams.update({'font.size': 20, 'figure.figsize': [14,10]})

# cut the first and last bin
x = histogram_data_pandas.iloc[1:-1]["value"]
y = histogram_data_pandas.iloc[1:-1]["N_events"]

# line plot
f, ax = plt.subplots()
ax.plot(x, y, '-')

ax.set_xlim(min_val, max_val)
ax.set_xlabel('$p_T$ (GeV)')
ax.set_ylabel('Number of Events')
ax.set_title("Distribution of $p_T$ ")

plt.show()


I am working on the file "Spark_Physics/HEP_benchmark/ADL_HEP_Query_Benchmark_Q1_Q5.ipynb".
The first task is:
Benchmark task: Q4
Plot the $𝐸^{𝑚𝑖𝑠𝑠}_𝑇$ of the events that have at least two jets with
$𝑝_𝑇$ > 40 GeV (gigaelectronvolt).
Can you write Python code for it?

# This will use MET adn Jet_pt
df_events.select("MET_pt","Jet_pt").show(10,False)

# The filter ispushed inside arrays of Jet_pt
# This use Spark's higher order functions for array processing

q4 = df_events.select("MET_pt").where("cardinality(filter(Jet_pt, x -> x > 40)) > 1")
q4.show(5,False)

# compute the histogram for MET_pt

min_val = 0
max_val = 100
num_bins = 100
step = (max_val - min_val) / num_bins

histogram_data = (
    q4
        .selectExpr(f"width_bucket(MET_pt, {min_val}, {max_val}, {num_bins}) as bucket")
        .groupBy("bucket")
        .count()
        .orderBy("bucket")
    )

# convert bucket number to the corresponding value
histogram_data = histogram_data.selectExpr(f"round({min_val} + (bucket - 1/2) * {step},2) as value", "count as N_events")

# The action toPandas() here triggers the computation.
# Histogram data is fetched into the driver as a Pandas Dataframe.

%time histogram_data_pandas=histogram_data.toPandas()

import matplotlib.pyplot as plt
plt.style.use('seaborn-darkgrid')
plt.rcParams.update({'font.size': 20, 'figure.figsize': [14,10]})

# cut the first and last bin
x = histogram_data_pandas.iloc[1:-1]["value"]
y = histogram_data_pandas.iloc[1:-1]["N_events"]

# line plot
f, ax = plt.subplots()
ax.plot(x, y, '-')

ax.set_xlim(min_val, max_val)
ax.set_xlabel('$𝐸^{𝑚𝑖𝑠𝑠}_T$ (GeV)')
ax.set_ylabel('Number of Events')
ax.set_title("Distribution of $𝐸^{𝑚𝑖𝑠𝑠}_T$ ")

plt.show()


I am working on the file "Spark_Physics/HEP_benchmark/ADL_HEP_Query_Benchmark_Q1_Q5.ipynb".
The first task is:
Benchmark task: Q5
Plot the $𝐸^{𝑚𝑖𝑠𝑠}_T$ of events that have an opposite-charge muon
pair with an invariant mass between 60 GeV and 120 GeV.
Can you write Python code for it?

# filter the events
# select only events with 2 muons
# the 2 muons must have opposite charge

df_muons = df_events.filter("nMuon == 2").filter("Muon_charge[0] != Muon_charge[1]")

# Formula for dimuon mass in pt, eta, phi, m coordinates
# see also http://edu.itp.phys.ethz.ch/hs10/ppp1/2010_11_02.pdf
# and https://en.wikipedia.org/wiki/Invariant_mass

df_with_dimuonmass = df_muons.selectExpr("MET_pt","""
     sqrt(2 * Muon_pt[0] * Muon_pt[1] *
           ( cosh(Muon_eta[0] - Muon_eta[1]) - cos(Muon_phi[0] - Muon_phi[1]) )
         ) as Dimuon_mass
         """)

# apply a filter on the dimuon mass

Q5 = df_with_dimuonmass.filter("Dimuon_mass between 60 and 120")

# compute the histogram for MET_pt

min_val = 0
max_val = 100
num_bins = 100
step = (max_val - min_val) / num_bins

histogram_data = (
    Q5
        .selectExpr(f"width_bucket(MET_pt, {min_val}, {max_val}, {num_bins}) as bucket")
        .groupBy("bucket")
        .count()
        .orderBy("bucket")
    )

# convert bucket number to the corresponding dimoun mass value
histogram_data = histogram_data.selectExpr(f"round({min_val} + (bucket - 1/2) * {step},2) as value", "count as N_events")

# The action toPandas() here triggers the computation.
# Histogram data is fetched into the driver as a Pandas Dataframe.

%time histogram_data_pandas=histogram_data.toPandas()

import matplotlib.pyplot as plt
plt.style.use('seaborn-darkgrid')
plt.rcParams.update({'font.size': 20, 'figure.figsize': [14,10]})

# cut the first and last bin
x = histogram_data_pandas.iloc[1:-1]["value"]
y = histogram_data_pandas.iloc[1:-1]["N_events"]

# line plot
f, ax = plt.subplots()
ax.plot(x, y, '-')

ax.set_xlabel('$𝐸^{𝑚𝑖𝑠𝑠}_T$ (GeV)')
ax.set_ylabel('Number of Events')
ax.set_title("Distribution of $𝐸^{𝑚𝑖𝑠𝑠}_T$ ")

plt.show()

spark.stop()


I am working on the file "Spark_Physics/HEP_benchmark/ADL_HEP_Query_Benchmark_Q1_Q5_CERNSWAN_Version.ipynb".
The first task is:
Benchmark task: Q1
Plot the $𝐸^{𝑚𝑖𝑠𝑠}_T$ (missing transverse energy) of all events.
Can you write Python code for it?


I am working on the file "Spark_Physics/HEP_benchmark/ADL_HEP_Query_Benchmark_Q1_Q5_CERNSWAN_Version.ipynb".
The first task is:
Benchmark task: Q4
Plot the $𝐸^{𝑚𝑖𝑠𝑠}_𝑇$ of the events that have at least two jets with
$𝑝_𝑇$ > 40 GeV (gigaelectronvolt).
Can you write Python code for it?


I am working on the file "Spark_Physics/HEP_benchmark/ADL_HEP_Query_Benchmark_Q1_Q5_CERNSWAN_Version.ipynb".
The first task is:
Benchmark task: Q5
Plot the $𝐸^{𝑚𝑖𝑠𝑠}_T$ of events that have an opposite-charge muon
pair with an invariant mass between 60 GeV and 120 GeV.
Can you write Python code for it?


I am working on the file "Spark_Physics/HEP_benchmark/ADL_HEP_Query_Benchmark_Q1_Q5_Colab_Version.ipynb".
The first task is:
Benchmark task: Q1
Plot the $𝐸^{𝑚𝑖𝑠𝑠}_T$ (missing transverse energy) of all events.
Can you write Python code for it?


I am working on the file "Spark_Physics/HEP_benchmark/ADL_HEP_Query_Benchmark_Q1_Q5_Colab_Version.ipynb".
The first task is:
Benchmark task: Q3
Plot the $𝑝_𝑇$ of jets with |𝜂| < 1 (𝜂 is the jet pseudorapidity).
Can you write Python code for it?


I am working on the file "Spark_Physics/HEP_benchmark/ADL_HEP_Query_Benchmark_Q1_Q5_Colab_Version.ipynb".
The first task is:
Benchmark task: Q5
Plot the $𝐸^{𝑚𝑖𝑠𝑠}_T$ of events that have an opposite-charge muon
pair with an invariant mass between 60 GeV and 120 GeV.
Can you write Python code for it?


I am working on the file "Spark_Physics/HEP_benchmark/ADL_HEP_Query_Benchmark_Q1_Q5_Colab_Version.ipynb".
The first task is:
Download data
Can you write Python code for it?

# Download the reduced data set (2 GB)
!wget -r -np -R "index.html*" -e robots=off https://sparkdltrigger.web.cern.ch/sparkdltrigger/Run2012B_SingleMu_sample.orc/

# This downloads the full dataset (16 GB)
#!wget -r -np -R "index.html*" -e robots=off https://sparkdltrigger.web.cern.ch/sparkdltrigger/Run2012B_SingleMu.orc/

# Start the Spark Session

from pyspark.sql import SparkSession
spark = (SparkSession.builder
         .appName("HEP benchmark")
         .master("local[*]")
         .config("spark.driver.memory", "4g")
         .config("spark.sql.orc.enableNestedColumnVectorizedReader", "true")
         .getOrCreate()
        )

# Read data for the benchmark tasks
# download data as detailed at https://github.com/LucaCanali/Miscellaneous/tree/master/Spark_Physics

path = "sparkdltrigger.web.cern.ch/sparkdltrigger/"
input_data = "Run2012B_SingleMu_sample.orc"

# use this if you downloaded the full dataset
# input_data = "Run2012B_SingleMu.orc"

df_events = spark.read.orc(path + input_data)

df_events.printSchema()

print(f"Number of events: {df_events.count()}")