I am working on the file "2019/09-clustering/cl_JoãoCastelo.ipynb".
The first task is:
1.2 Definir os clusters
Na segunda etapa do algoritmo serão definidos o grupo de cada dado, de acordo com os centróides calculados.
1.2.1 Função de distância
Codifique a função de distância euclidiana entre dois pontos (a, b).
Definido pela equação:
$$ dist(a, b) = \sqrt{(a_1-b_1)^{2}+(a_2-b_2)^{2}+ ... + (a_n-b_n)^{2}} $$
$$ dist(a, b) = \sqrt{\sum_{i=1}^{n}(a_i-b_i)^{2}} $$
Can you write Python code for it?

def euclidean_distance(a, b):
    """
    Calcula a distância euclidiana entre os pontos a e b

    Argumentos:
    a -- Um ponto no espaço - [1,n]
    b -- Um ponto no espaço - [1,n]

    Retornos:
    distance -- Distância euclidiana entre os pontos
    """

    #### CODE HERE ####

    quadraticos = 0

    for i in range(0 , len(a)):
      potencia = (a[i] - b[i]) ** 2
      quadraticos = quadraticos + potencia

    ### END OF CODE ###

    return quadraticos ** 0.5


I am working on the file "2019/09-clustering/cl_JoãoCastelo.ipynb".
The first task is:
1.2.2 Calcular o centroide mais próximo
Utilizando a função de distância codificada anteriormente, complete a função abaixo para calcular o centroid mais próximo de um ponto qualquer.

Dica: https://docs.scipy.org/doc/numpy/reference/generated/numpy.argmin.html
Can you write Python code for it?

import math
def nearest_centroid(a, centroids):
    """
    Calcula o índice do centroid mais próximo ao ponto a

    Argumentos:
    a -- Um ponto no espaço - [1,n]
    centroids -- Lista com os centróides - [k,n]

    Retornos:
    nearest_index -- Índice do centróide mais próximo
    """

    #### CODE HERE ####
    index_centroide = 0
    distancia_minima = math.inf

    for i in range(0 , len(centroids)):
      distancia_centroid = euclidean_distance(a,centroids[i])
      if(distancia_centroid <= distancia_minima):
        distancia_minima = distancia_centroid
        index_centroide = i

    ### END OF CODE ###

    return index_centroide


I am working on the file "2019/09-clustering/cl_JoãoCastelo.ipynb".
The first task is:
1.2.3 Calcular centroid mais próximo de cada dado do dataset
Utilizando a função anterior que retorna o índice do centroid mais próximo, calcule o centroid mais próximo de cada dado do dataset.
Can you write Python code for it?

def all_nearest_centroids(dataset, centroids):
    """
    Calcula o índice do centroid mais próximo para cada
    ponto do dataset

    Argumentos:
    dataset -- Conjunto de dados - [m,n]
    centroids -- Lista com os centróides - [k,n]

    Retornos:
    nearest_indexes -- Índices do centróides mais próximos - [m,1]
    """

    #### CODE HERE ####
    nearest_indexes = []
    for ponto in dataset:

      nearest_indexes.append( nearest_centroid(ponto , centroids) )

    ### END OF CODE ###

    return np.array(nearest_indexes)


I am working on the file "2019/09-clustering/cl_JoãoCastelo.ipynb".
The first task is:
1.4 Atualizar os clusters
Nessa etapa, os centróides são recomputados. O novo valor de cada centróide será a media de todos os dados atribuídos ao cluster.
Can you write Python code for it?

def update_centroids(dataset, centroids, nearest_indexes):
    """
    Atualiza os centroids

    Argumentos:
    dataset -- Conjunto de dados - [m,n]
    centroids -- Lista com os centróides - [k,n]
    nearest_indexes -- Índices do centróides mais próximos - [m,1]

    Retornos:
    centroids -- Lista com centróides atualizados - [k,n]
    """

    #### CODE HERE ####

    pontos_relacionados = []
    for i in range(0 , len(centroids)):
      candidatos = []
      for index in range(0,len(nearest_indexes)):
        if(nearest_indexes[index] == i):
          candidatos.append( dataset[index] )
      pontos_relacionados.append(candidatos)

    for i in range(0,len(centroids)):
      if(len(pontos_relacionados[i]) != 0):

        centroids[i] = media(pontos_relacionados[i], len(dataset[0]))

    ### END OF CODE ###

    return centroids


I am working on the file "2019/09-clustering/cl_JoãoCastelo.ipynb".
The first task is:
3. Método do cotovelo
Implemete o método do cotovelo e mostre o melhor K para o conjunto de dados.
Can you write Python code for it?

iner =[]
for i in range(1,20):
  kmeans = KMeans(n_clusters=i)
  kmeans.fit(dataset)
  iner.append(kmeans.inertia_)
print(iner)
iner = np.array(iner)

plt.plot(range(1,20) ,iner , '--')
plt.show()

#com 3 clusters ja temos o cotovelo!


I am working on the file "2019/09-clustering/cl_JoãoCastelo.ipynb".
The first task is:
Teste a função criada
Can you write Python code for it?

# Seleciona um ponto aleatório no dataset
index = np.random.randint(dataset.shape[0])
a = dataset[index,:]

# Usa a função para descobrir o centroid mais próximo
idx_nearest_centroid = nearest_centroid(a, centroids)

# Plota os dados ------------------------------------------------
plt.scatter(dataset[:,0], dataset[:,1], s=10)
# Plota o ponto aleatório escolhido em uma cor diferente
plt.scatter(a[0], a[1], c='magenta', s=30)

# Plota os centroids
plt.scatter(centroids[:,0], centroids[:,1], marker='^', c='red', s=100)
# Plota o centroid mais próximo com uma cor diferente
plt.scatter(centroids[idx_nearest_centroid,0],
            centroids[idx_nearest_centroid,1],
            marker='^', c='springgreen', s=100)

# Cria uma linha do ponto escolhido para o centroid selecionado
plt.plot([a[0], centroids[idx_nearest_centroid,0]],
         [a[1], centroids[idx_nearest_centroid,1]],c='orange')
plt.annotate('CENTROID', (centroids[idx_nearest_centroid,0],
            centroids[idx_nearest_centroid,1],))
plt.show()


I am working on the file "2019/09-clustering/cl_JoãoCastelo.ipynb".
The first task is:
Teste a função criada e visualize os centróides que foram calculados.
Can you write Python code for it?

k = 3
centroids = calculate_initial_centers(dataset, k)

plt.scatter(dataset[:,0], dataset[:,1], s=10)
plt.scatter(centroids[:,0], centroids[:,1], marker='^', c='red',s=100)
plt.show()


I am working on the file "2019/09-clustering/cl_JoãoCastelo.ipynb".
The first task is:
Teste a função criada.
Can you write Python code for it?

a = np.array([1, 5, 9])
b = np.array([3, 7, 8])

if (euclidean_distance(a,b) == 3):
    print("Distância calculada corretamente!")
else:
    print("Função de distância incorreta")


I am working on the file "2019/09-clustering/cl_JoãoCastelo.ipynb".
The first task is:
Verifique o resultado do algoritmo abaixo!
Can you write Python code for it?

kmeans = KMeans(n_clusters=3)
kmeans.fit(dataset)

print("Inércia = ", kmeans.inertia_)

plt.scatter(dataset[:,0], dataset[:,1], c=kmeans.labels_)
plt.scatter(kmeans.cluster_centers_[:,0],
            kmeans.cluster_centers_[:,1], marker='^', c='red', s=100)
plt.show()


I am working on the file "2019/09-clustering/cl_JoãoCastelo.ipynb".
The first task is:
Visualize os clusters formados
Can you write Python code for it?

nearest_indexes = all_nearest_centroids(dataset, centroids)

# Plota os os cluster ------------------------------------------------
plt.scatter(dataset[:,0], dataset[:,1], c=nearest_indexes)

# Plota os centroids
plt.scatter(centroids[:,0], centroids[:,1], marker='^', c='red', s=100)
for index, centroid in enumerate(centroids):
    dataframe = dataset[nearest_indexes == index,:]
    for data in dataframe:
        plt.plot([centroid[0], data[0]], [centroid[1], data[1]],
                 c='lightgray', alpha=0.3)
plt.show()


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
1. Implementar o algoritmo K-means
Nesta etapa você irá implementar as funções que compõe o algoritmo do KMeans uma a uma. É importante entender e ler a documentação de cada função, principalmente as dimensões dos dados esperados na saída.
1.1 Inicializar os centróides
A primeira etapa do algoritmo consiste em inicializar os centróides de maneira aleatória. Essa etapa é uma das mais importantes do algoritmo e uma boa inicialização pode diminuir bastante o tempo de convergência.
Para inicializar os centróides você pode considerar o conhecimento prévio sobre os dados, mesmo sem saber a quantidade de grupos ou sua distribuição.

Dica: https://docs.scipy.org/doc/numpy/reference/generated/numpy.random.uniform.html
Can you write Python code for it?

def calculate_initial_centers(dataset, k):
    """
    Inicializa os centróides iniciais de maneira arbitrária

    Argumentos:
    dataset -- Conjunto de dados - [m,n]
    k -- Número de centróides desejados

    Retornos:
    centroids -- Lista com os centróides calculados - [k,n]
    """

    #### CODE HERE ####

    #Determina o valor máximo e mínimo para cada atributo
    #Cria uma lista de tuplas que contem valores máximos e mínimos de cada atributo
    ranges = [(min([row[i] for row in dataset]),
               max([row[i] for row in dataset]))
               for i in range(len(dataset[0]))]

#     min_x = ranges[0][0]
#     max_x = ranges[0][1]
#     min_y = ranges[1][0]
#     max_y = ranges[1][1]
    centroids=[[np.random.random()*(ranges[i][1] - ranges[i][0])+ranges[i][0]
           for i in range(len(dataset[0]))] for j in range(k)]
    centroids = np.array(centroids)

    ### END OF CODE ###

    return centroids


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
1.2 Definir os clusters
Na segunda etapa do algoritmo serão definidos o grupo de cada dado, de acordo com os centróides calculados.
1.2.1 Função de distância
Codifique a função de distância euclidiana entre dois pontos (a, b).
Definido pela equação:
$$ dist(a, b) = \sqrt{(a_1-b_1)^{2}+(a_2-b_2)^{2}+ ... + (a_n-b_n)^{2}} $$
$$ dist(a, b) = \sqrt{\sum_{i=1}^{n}(a_i-b_i)^{2}} $$
Can you write Python code for it?

def euclidean_distance(a, b):
    """
    Calcula a distância euclidiana entre os pontos a e b

    Argumentos:
    a -- Um ponto no espaço - [1,n]
    b -- Um ponto no espaço - [1,n]

    Retornos:
    distance -- Distância euclidiana entre os pontos
    """

    #### CODE HERE ####
    dist = 0.0
    for i in range(len(a)):
        dist += pow((a[i])-b[i],2)

    distance = math.sqrt(dist)

    ### END OF CODE ###

    return distance


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
1.2.3 Calcular centroid mais próximo de cada dado do dataset
Utilizando a função anterior que retorna o índice do centroid mais próximo, calcule o centroid mais próximo de cada dado do dataset.
Can you write Python code for it?

def all_nearest_centroids(dataset, centroids):
    """
    Calcula o índice do centroid mais próximo para cada
    ponto do dataset

    Argumentos:
    dataset -- Conjunto de dados - [m,n]
    centroids -- Lista com os centróides - [k,n]

    Retornos:
    nearest_indexes -- Índices do centróides mais próximos - [m,1]
    """

    #### CODE HERE ####
    nearest_indexes = []
    for row in dataset:
        nearest_indexes.append(nearest_centroid(row, centroids))

    ### END OF CODE ###

    return nearest_indexes


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
1.3 Métrica de avaliação
Após formar os clusters, como sabemos se o resultado gerado é bom? Para isso, precisamos definir uma métrica de avaliação.
O algoritmo K-means tem como objetivo escolher centróides que minimizem a soma quadrática das distância entre os dados de um cluster e seu centróide. Essa métrica é conhecida como inertia.
$$\sum_{i=0}^{n}\min_{c_j \in C}(||x_i - c_j||^2)$$
A inertia, ou o critério de soma dos quadrados dentro do cluster, pode ser reconhecido como uma medida de o quão internamente coerentes são os clusters, porém ela sofre de alguns inconvenientes:

A inertia pressupõe que os clusters são convexos e isotrópicos, o que nem sempre é o caso. Desta forma, pode não representar bem em aglomerados alongados ou variedades com formas irregulares.
A inertia não é uma métrica normalizada: sabemos apenas que valores mais baixos são melhores e zero é o valor ótimo. Mas em espaços de dimensões muito altas, as distâncias euclidianas tendem a se tornar infladas (este é um exemplo da chamada “maldição da dimensionalidade”). A execução de um algoritmo de redução de dimensionalidade, como o PCA, pode aliviar esse problema e acelerar os cálculos.

Fonte: https://scikit-learn.org/stable/modules/clustering.html
Para podermos avaliar os nosso clusters, codifique a métrica da inertia abaixo, para isso você pode utilizar a função de distância euclidiana construída anteriormente.
$$inertia = \sum_{i=0}^{n}\min_{c_j \in C} (dist(x_i, c_j))^2$$
Can you write Python code for it?

def inertia(dataset, centroids, nearest_indexes):
    """
    Soma das distâncias quadradas das amostras para o
    centro do cluster mais próximo.

    Argumentos:
    dataset -- Conjunto de dados - [m,n]
    centroids -- Lista com os centróides - [k,n]
    nearest_indexes -- Índices do centróides mais próximos - [m,1]

    Retornos:
    inertia -- Soma total do quadrado da distância entre
    os dados de um cluster e seu centróide
    """

    #### CODE HERE ####

    inertia = 0
    for i, centroid in enumerate(centroids):
        dist = 0
        for j,index in enumerate(nearest_indexes):
            if index == i:
                dist += math.pow(euclidean_distance(dataset[j],centroid),2)
        inertia += dist
    ### END OF CODE ###
    return np.array(inertia)


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
1.4 Atualizar os clusters
Nessa etapa, os centróides são recomputados. O novo valor de cada centróide será a media de todos os dados atribuídos ao cluster.
Can you write Python code for it?

def update_centroids(dataset, centroids, nearest_indexes):
    """
    Atualiza os centroids

    Argumentos:
    dataset -- Conjunto de dados - [m,n]
    centroids -- Lista com os centróides - [k,n]
    nearest_indexes -- Índices do centróides mais próximos - [m,1]

    Retornos:
    centroids -- Lista com centróides atualizados - [k,n]
    """

    #### CODE HERE ####

    for i, centroid in enumerate(centroids):
        data_list = []
        data_list_x = data_list_y = []
#         print(centroids[i])
        for j,index in enumerate(nearest_indexes):
            if index == i:
                data_list_x.append(dataset[j][0])
                data_list_y.append(dataset[j][1])
                data_list.append(dataset[j])
#         print("Valores")
#         print(data_list_y)

        centroids[i] = np.mean(data_list,axis=0)
#         print(centroids[i])
#         print("")

    ### END OF CODE ###

    return centroids


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
2. K-means
2.1 Algoritmo completo
Utilizando as funções codificadas anteriormente, complete a classe do algoritmo K-means!
Can you write Python code for it?

class KMeans():

    def __init__(self, n_clusters=8, max_iter=300):
        self.n_clusters = n_clusters
        self.max_iter = max_iter
        self.inertia_ = 0

    def fit(self,X):

        # Inicializa os centróides
        self.cluster_centers_ = calculate_initial_centers(X, self.n_clusters)

        # Computa o cluster de cada amostra
        self.labels_ = all_nearest_centroids(X,  self.cluster_centers_)

        # Calcula a inércia inicial
        old_inertia = inertia(X,  self.cluster_centers_, nearest_indexes)

#         print("Starting Fit...")

        for i in range (self.max_iter):

            #### CODE HERE ####

            self.cluster_centers_ = update_centroids(X, self.cluster_centers_, self.labels_)
            self.labels_ = all_nearest_centroids(X, self.cluster_centers_)
            self.inertia_ = inertia(X,  self.cluster_centers_, self.labels_)
            if(old_inertia <= self.inertia_):
                return self

            ### END OF CODE ###
#         print("Fit Sucess!!!")
        return self

    def predict(self, X):

        return all_nearest_centroids(X, self.cluster_centers_)


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
4. Dataset Real
Exercícios
1 - Aplique o algoritmo do K-means desenvolvido por você no datatse iris [1]. Mostre os resultados obtidos utilizando pelo menos duas métricas de avaliação de clusteres [2].

[1] http://archive.ics.uci.edu/ml/datasets/iris
[2] http://scikit-learn.org/stable/modules/clustering.html#clustering-evaluation

Dica: você pode utilizar as métricas completeness e homogeneity.

2 - Tente melhorar o resultado obtido na questão anterior utilizando uma técnica de mineração de dados. Explique a diferença obtida.

Dica: você pode tentar normalizar os dados [3].
- [3] https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.normalize.html

3 - Qual o número de clusteres (K) você escolheu na questão anterior? Desenvolva o Método do Cotovelo sem usar biblioteca e descubra o valor de K mais adequado. Após descobrir, utilize o valor obtido no algoritmo do K-means.
4 - Utilizando os resultados da questão anterior, refaça o cálculo das métricas e comente os resultados obtidos. Houve uma melhoria? Explique.
Can you write Python code for it?


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
Teste a função criada
Can you write Python code for it?


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
Teste a função criada e visualize os centróides que foram calculados.
Can you write Python code for it?

k = 3
centroids = calculate_initial_centers(dataset, k)
print("Centroids: " + str(centroids))
plt.scatter(dataset[:,0], dataset[:,1], s=10)
plt.scatter(centroids[:,0], centroids[:,1], marker='^', c='red',s=100)
plt.show()


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
Teste a função criada visualizando os cluster formados.
Can you write Python code for it?

nearest_indexes = all_nearest_centroids(dataset, centroids)
print(nearest_indexes)

plt.scatter(dataset[:,0], dataset[:,1], c=nearest_indexes)
plt.scatter(centroids[:,0], centroids[:,1], marker='^', c='red', s=100)
plt.show()


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
Teste a função criada.
Can you write Python code for it?


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
Verifique o resultado do algoritmo abaixo!
Can you write Python code for it?


I am working on the file "2019/09-clustering/cl_Lucas_Simão.ipynb".
The first task is:
Visualize os clusters formados
Can you write Python code for it?


I am working on the file "2019/09-clustering/cl_PedroMotta.ipynb".
The first task is:
1.2 Definir os clusters
Na segunda etapa do algoritmo serão definidos o grupo de cada dado, de acordo com os centróides calculados.
1.2.1 Função de distância
Codifique a função de distância euclidiana entre dois pontos (a, b).
Definido pela equação:
$$ dist(a, b) = \sqrt{(a_1-b_1)^{2}+(a_2-b_2)^{2}+ ... + (a_n-b_n)^{2}} $$
$$ dist(a, b) = \sqrt{\sum_{i=1}^{n}(a_i-b_i)^{2}} $$
Can you write Python code for it?

def euclidean_distance(a, b):
    """
    Calcula a distância euclidiana entre os pontos a e b

    Argumentos:
    a -- Um ponto no espaço - [1,n]
    b -- Um ponto no espaço - [1,n]

    Retornos:
    distance -- Distância euclidiana entre os pontos
    """

    #### CODE HERE ####

    dist = 0

    for ai,bi in zip(a,b):
        dist = dist + (ai - bi)**2
    distance = math.sqrt(dist)
    ### END OF CODE ###

    return distance


I am working on the file "2019/09-clustering/cl_PedroMotta.ipynb".
The first task is:
1.2.3 Calcular centroid mais próximo de cada dado do dataset
Utilizando a função anterior que retorna o índice do centroid mais próximo, calcule o centroid mais próximo de cada dado do dataset.
Can you write Python code for it?

def all_nearest_centroids(dataset, centroids):
    """
    Calcula o índice do centroid mais próximo para cada
    ponto do dataset

    Argumentos:
    dataset -- Conjunto de dados - [m,n]
    centroids -- Lista com os centróides - [k,n]

    Retornos:
    nearest_indexes -- Índices do centróides mais próximos - [m,1]
    """

    #### CODE HERE ####
    nearest_indexes = []
    for data in dataset:
        near = nearest_centroid(data,centroids)
        nearest_indexes.append(near)

    ### END OF CODE ###

    return nearest_indexes


I am working on the file "2019/09-clustering/cl_PedroMotta.ipynb".
The first task is:
1.4 Atualizar os clusters
Nessa etapa, os centróides são recomputados. O novo valor de cada centróide será a media de todos os dados atribuídos ao cluster.
Can you write Python code for it?

def update_centroids(dataset, centroids, nearest_indexes):
    """
    Atualiza os centroids

    Argumentos:
    dataset -- Conjunto de dados - [m,n]
    centroids -- Lista com os centróides - [k,n]
    nearest_indexes -- Índices do centróides mais próximos - [m,1]

    Retornos:
    centroids -- Lista com centróides atualizados - [k,n]
    """

    #### CODE HERE ####

    newCentroids = []
    r,c = np.shape(centroids)
    for i in range(0,r):
        newCentroids.append(0)

    values, repetitions = np.unique(nearest_indexes,return_counts=True)

    for data,index in zip(dataset,nearest_indexes):
        i = 0
        for val in values:
            if val == index:
                indexesRepetitions = repetitions[i]
            i = i+1

        newCentroids[index] = newCentroids[index] + data/indexesRepetitions

    #print('Antigos:',centroids)
    #print('Novos:',np.asarray(newCentroids))
    centroids = np.asarray(newCentroids)
    ### END OF CODE ###

    return centroids


I am working on the file "2019/09-clustering/cl_PedroMotta.ipynb".
The first task is:
2. K-means
2.1 Algoritmo completo
Utilizando as funções codificadas anteriormente, complete a classe do algoritmo K-means!
Can you write Python code for it?

class KMeans():

    def __init__(self, n_clusters=8, max_iter=300):
        self.n_clusters = n_clusters
        self.max_iter = max_iter

    def fit(self,X):
        dataset = np.array(X)
        # Inicializa os centróides
        self.cluster_centers_ = calculate_initial_centers(X, self.n_clusters)

        # Computa o cluster de cada amostra
        self.labels_ = [None]

            # Calcula a inércia inicial
        old_inertia = [None]
        self.inertia_ = [None]

        for index in range(0,self.max_iter):

            nearest_indexes = all_nearest_centroids(dataset, self.cluster_centers_)
            old_inertia = inertia(dataset, self.cluster_centers_, nearest_indexes)
            self.cluster_centers_ = update_centroids(dataset, self.cluster_centers_, nearest_indexes)

            # Plota os os cluster ------------------------------------------------
            centroids = self.cluster_centers_
            plt.scatter(dataset[:,0], dataset[:,1], c=nearest_indexes)

            # Plota os centroids
            plt.scatter(centroids[:,0], centroids[:,1], marker='^', c='red', s=100)
            for index, centroid in enumerate(centroids):
                dataframe = dataset[nearest_indexes == index,:]
                for data in dataframe:
                    plt.plot([centroid[0], data[0]], [centroid[1], data[1]],
                             c='lightgray', alpha=0.3)
            plt.show()

        self.inertia_ = old_inertia
            #### CODE HERE ####
    def calculate_initial_centers(dataset, k):
        """
        Inicializa os centróides iniciais de maneira arbitrária

        Argumentos:
        dataset -- Conjunto de dados - [m,n]
        k -- Número de centróides desejados

        Retornos:
        centroids -- Lista com os centróides calculados - [k,n]
        """

        m,n = np.shape(dataset)
        minimum = np.min(dataset)
        maximum = np.max(dataset)

        centroids =...


I am working on the file "2019/09-clustering/cl_PedroMotta.ipynb".
The first task is:
3. Método do cotovelo
Implemete o método do cotovelo e mostre o melhor K para o conjunto de dados.
Can you write Python code for it?


I am working on the file "2019/09-clustering/cl_PedroMotta.ipynb".
The first task is:
4. Dataset Real
Exercícios
1 - Aplique o algoritmo do K-means desenvolvido por você no datatse iris [1]. Mostre os resultados obtidos utilizando pelo menos duas métricas de avaliação de clusteres [2].

[1] http://archive.ics.uci.edu/ml/datasets/iris
[2] http://scikit-learn.org/stable/modules/clustering.html#clustering-evaluation

Dica: você pode utilizar as métricas completeness e homogeneity.

2 - Tente melhorar o resultado obtido na questão anterior utilizando uma técnica de mineração de dados. Explique a diferença obtida.

Dica: você pode tentar normalizar os dados [3].
- [3] https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.normalize.html

3 - Qual o número de clusteres (K) você escolheu na questão anterior? Desenvolva o Método do Cotovelo sem usar biblioteca e descubra o valor de K mais adequado. Após descobrir, utilize o valor obtido no algoritmo do K-means.
4 - Utilizando os resultados da questão anterior, refaça o cálculo das métricas e comente os resultados obtidos. Houve uma melhoria? Explique.
Can you write Python code for it?


I am working on the file "2019/09-clustering/cl_PedroMotta.ipynb".
The first task is:
Execute a função de atualização e visualize novamente os cluster formados
Can you write Python code for it?

centroids_new = update_centroids(dataset, centroids, nearest_indexes)

plt.scatter(dataset[:,0], dataset[:,1], s=10)
plt.scatter(centroids_new[:,0], centroids_new[:,1], marker='^', c='red',s=100)
plt.show()


I am working on the file "2019/09-clustering/cl_PedroMotta.ipynb".
The first task is:
Teste a função codificada executando o código abaixo.
Can you write Python code for it?

tmp_data = np.array([[1,2,3],[3,4,5],[4,5,6]])
tmp_centroide = np.array([[2,3,4]])

tmp_nearest_indexes = all_nearest_centroids(tmp_data, tmp_centroide)
if np.floor(inertia(tmp_data, tmp_centroide, tmp_nearest_indexes)) == 26:
    print("Inertia calculada corretamente!")
else:
    print("Função de inertia incorreta!")

# Use a função para verificar a inertia dos seus clusters
inertia(dataset, centroids, nearest_indexes)


I am working on the file "2019/09-clustering/cl_PedroMotta.ipynb".
The first task is:
Teste a função criada visualizando os cluster formados.
Can you write Python code for it?

nearest_indexes = all_nearest_centroids(dataset, centroids)
nearest_indexes

plt.scatter(dataset[:,0], dataset[:,1], c=nearest_indexes)
plt.scatter(centroids[:,0], centroids[:,1], marker='^', c='red', s=100)
plt.show()


I am working on the file "2019/09-clustering/cl_PedroMotta.ipynb".
The first task is:
Verifique o resultado do algoritmo abaixo!
Can you write Python code for it?

# load the data with pandas
dataset = pd.read_csv('dataset.csv', header=None)

kmeans = KMeans(n_clusters=3,max_iter=10)
kmeans.fit(dataset)

print("Inércia = ", kmeans.inertia_)


I am working on the file "2019/11-hadoop/WordCountPython_solucao.ipynb".
The first task is:
2. Input split
defina uma função split que recebe os dados e quebre em partes menores para usar como entrada na função mapper.
Exemplo: string.split()
Can you write Python code for it?

def input_split(text):
    text = re.sub(u'[^a-zA-ZáéíóúÁÉÍÓÚâêîôÂÊÎÔãõÃÕçÇ: ]', '', text)
    return pd.DataFrame(text.split(" "))

df_file1 = input_split(text)
df_file2 = input_split(text2)


I am working on the file "2019/11-hadoop/WordCountPython_solucao.ipynb".
The first task is:
4. Combiner
Faz a combinação de "n" dataframes para obter um único dataframe que será usado na função Shufle
Can you write Python code for it?

def combiner(df_list):
    return pd.concat(df_list)

df_combiner = combiner([df_file1, df_file2])

df_combiner.head()


I am working on the file "2019/11-hadoop/WordCountPython_solucao.ipynb".
The first task is:
5. Shuffle
Descreva uma função Shufle para ordenar, filtrar e agrupar dados como pares de chave e valor que serão a entrada na função reducer
Can you write Python code for it?

def shufle(df):
    df.sort_values(by=['key'], inplace=True)
    df = df[df.key != ""]
    return df

df_combiner = shufle(df_combiner)

df_combiner.head()


I am working on the file "2019/11-hadoop/WordCountPython_solucao.ipynb".
The first task is:
6. Reduce
Use uma função que agregue dados pelo valor (v) de acordo com a chave (k) para enviar pares de chave-valor para saída
Can you write Python code for it?

def reducer(df):
    return df_combiner.groupby('key').count()

df_reducer = reducer(df_combiner)
df_reducer.head(10)


I am working on the file "2019/11-hadoop/WordCountPython_solucao.ipynb".
The first task is:
7.Output Format
Forneça os pares de chave-valor finais ordenados por maior valor
Can you write Python code for it?

df_reducer.sort_values(by=['value'], ascending=False, inplace=True)

df_reducer.head(20)


I am working on the file "2019/12-spark/12-spark-intro/Actions.ipynb".
The first task is:
collect()
Retornar todos os elementos do RDD como uma lista (do python, no caso). Isso geralmente é útil após um filtro ou outra operação que retorna um subconjunto suficientemente pequeno dos dados.
Exemplo:
Can you write Python code for it?

data = sc.parallelize(range(1, 1001))
print 'data: ', data.filter(lambda i: i % 10 == 0).collect()


I am working on the file "2019/12-spark/12-spark-intro/Actions.ipynb".
The first task is:
foreach(func)
Executa uma função func para cada elemento do RDD. Isso geralmente é feito para operações sem tanto efeito, como atualizar um acumulador ou interagir com sistemas de armazenamento externo.
Exemplo:
Can you write Python code for it?

data = sc.parallelize(range(1, 1001))

# accumulator
summation_20_mult = sc.accumulator(0)

def conditional_print(i):
    if i % 20 == 0:
        print(i)
        summation_20_mult.add(i)

data.foreach(conditional_print)

print ('Summation of multiple of 20 from 1 to 1000: ', summation_20_mult.value)


I am working on the file "2019/12-spark/12-spark-intro/Actions.ipynb".
The first task is:
saveAsTextFile(path)
Escreve os elementos do RDD como um arquivo de texto (ou conjunto de arquivos de texto) em um determinado diretório no sistema de arquivos local, no HDFS ou qualquer outro sistema de arquivos suportado pelo Hadoop. O Spark chamará toString em cada elemento para convertê-lo em uma linha de texto no arquivo.
Exemplo:
Can you write Python code for it?

data = sc.parallelize(range(1, 1001))
data.saveAsTextFile("range_1_to_100")

# Visualize o diretório deste notebook. Existirá um pasta com o nome range_1_to_100.
# Nela haverá arquivos que do RDD. Cada partição do processamento tem um arquivo correspondente.


I am working on the file "2019/12-spark/12-spark-intro/Actions.ipynb".
The first task is:
take(n)
Retorna uma lista com os primeiros n elementos do RDD.
Exemplo:
Can you write Python code for it?

data = sc.parallelize(range(1, 1001))
print ('data first 10: ', data.take(10))


I am working on the file "2019/12-spark/12-spark-intro/Actions.ipynb".
The first task is:
takeSample(withReplacement, num, [seed])
Retorna uma lista com uma amostra aleatória de num elementos do RDD, com ou sem substituição (withReplacement), opcionalmente pré-especificando uma semente (seed) de gerador de números aleatórios.
Exemplo:
Can you write Python code for it?

data = sc.parallelize(range(1, 11))
# sampling with replacement
print (data.takeSample(True, 4))
# sampling without replacement
print (data.takeSample(False, 4))


I am working on the file "2019/12-spark/12-spark-intro/Transformations.ipynb".
The first task is:
filter(func)
Retorna um novo RDD formado pela seleção daqueles elemento do RDD de origem que, quando passados para função func, retorna true.
Exemplo:
Can you write Python code for it?

data = sc.parallelize(range(1, 11))

res = data.filter(lambda x: x%2 ==1)

print(res.collect())


I am working on the file "2019/12-spark/12-spark-intro/Transformations.ipynb".
The first task is:
flatMap(func)
Semelhante ao map, porém cada item de entrada pode ser mapeado para 0 ou mais itens de saída (assim, func deve retornar uma lista em vez de um único item).
Exemplo:
Can you write Python code for it?

data = sc.parallelize(["Linha 1", "Linha 2"])

def partir(l): return l.split(" ")

print ('map:', data.map(partir).collect())

print ('flatMap:', data.flatMap(partir).collect())


I am working on the file "2019/12-spark/12-spark-intro/Transformations.ipynb".
The first task is:
groupByKey()
Quando chamado em um RDD de pares (K, V), retorna um conjunto de dados de pares (K, Iterable<V>).
Exemplo:
Can you write Python code for it?

data = sc.parallelize([ ('a', 1), ('b', 2), ('c', 3) , ('a', 2), ('b', 5), ('a', 3)])

for pair in data.groupByKey().collect():
    print (pair[0], list(pair[1]))


I am working on the file "2019/12-spark/12-spark-intro/Transformations.ipynb".
The first task is:
sortByKey([asceding])
Quando chamado em um RDD de pares (K, V) em que K é ordenável, retorna um RDD de pares (K, V) ordenados por chaves em ordem ascendente ou descendente, conforme especificado no argumento ascending.
Exemplo:
Can you write Python code for it?

data = sc.parallelize([ ('a', 1), ('b', 2), ('c', 3) , ('a', 2), ('b', 5), ('a', 3)])

print(data.sortByKey(ascending=False).collect())


I am working on the file "2019/12-spark/12-spark-intro/bruno_mourao_spark1.ipynb".
The first task is:
Filtragem de Primos
Dado uma sequência de números de 1 a 1000000, filtre somente os primos dessa sequência.
Can you write Python code for it?

def primes( n ):
# i sera nosso divisor inicial
    i = 1;
# j sera nosso contador de ocorrências
    j = 0;
#Nenhum numero real vai ser divisivel por um numero maior do que sua metade
    n1 = (n/2);

    while (i <= n):

       if (n % i==0):
          i = i+1;
          j = j+1;

       if (i>=n1):
        # damos a i, o valor da variavel entrada, pois o próximo divisor sera o próprio número
          i = n;
          i = i+1;
          j = j+1;

       else:
          i = i+1;
    if(j==2):
        return True
    else:
        return False

data = sc.parallelize(range(1, 1000000))
res = data.filter(primes)
print(res.take(100))


I am working on the file "2019/12-spark/12-spark-intro/bruno_mourao_spark1.ipynb".
The first task is:
Word Count Memória Postumas de Brás Cubas
Memórias Póstumas de Brás Cubas é um romance escrito por Machado de Assis, desenvolvido em princípio como folhetim, de março a dezembro de 1880, na Revista Brasileira, para, no ano seguinte, ser publicado como livro, pela então Tipografia Nacional.
A obra retrata a escravidão, as classes sociais, o cientificismo e o positivismo da época. Dada essas informações, será que conseguimos idenficar essas características pelas palavras mais utilizadas em sua obra?
Utilizando o dataset Machado-de-Assis-Memorias-Postumas.txt, faça um word count e encontre as palavras mais utilizadas por Machado de Assis em sua obra. Não esqueça de utilizar stopwords.pt para remover as stop words!
Can you write Python code for it?

def filterfun(w):
    stopwords = set(open('stopwords.pt').read().split())
    if w[0] in stopwords:
        return False
    else:
        return True

text_file = sc.textFile("Machado-de-Assis-Memorias-Postumas.txt")

counts = text_file.flatMap(lambda line: line.split(" ")) \
             .map(lambda word: (word, 1)) \
             .reduceByKey(lambda a, b: a + b)
counts = counts.filter(filterfun)
counts = counts.takeOrdered(100, key = lambda x: -x[1])

print(counts)


I am working on the file "2019/12-spark/12-spark-intro/r_newton_joaquim.ipynb".
The first task is:
Municípios do Brasil
Dado o dataset mucipios_do_Brasil.csv, faça duas operações com ele:

Monte uma lista dos municípios por estado.
Conte quantos municípios há em cada estado.

Dicas: use as operações groupByKey e reduceByKey, não faça um count na lista da operação 1.
Can you write Python code for it?

file = sc.textFile('municipios_do_Brasil.csv')

file_mapping = file.map(lambda line: line.split(',')).map(lambda word: (word[0], word[1]))

#Contando o numero de municipios por Estado
count_cities = file_mapping.map(lambda word: (word[0], 1)).reduceByKey(lambda x, y: x + y)

print(count_cities.collect())

#Listando os municipios por Estado
list_cities = file_mapping.groupByKey().map(lambda x : (x[0], list(x[1])))

print(list_cities.take(1))


I am working on the file "2019/12-spark/12-spark-intro/r_newton_joaquim.ipynb".
The first task is:
Word Count Memória Postumas de Brás Cubas
Memórias Póstumas de Brás Cubas é um romance escrito por Machado de Assis, desenvolvido em princípio como folhetim, de março a dezembro de 1880, na Revista Brasileira, para, no ano seguinte, ser publicado como livro, pela então Tipografia Nacional.
A obra retrata a escravidão, as classes sociais, o cientificismo e o positivismo da época. Dada essas informações, será que conseguimos idenficar essas características pelas palavras mais utilizadas em sua obra?
Utilizando o dataset Machado-de-Assis-Memorias-Postumas.txt, faça um word count e encontre as palavras mais utilizadas por Machado de Assis em sua obra. Não esqueça de utilizar stopwords.pt para remover as stop words!
Can you write Python code for it?

file2 = sc.textFile('Machado-de-Assis-Memorias-Postumas.txt')

stopwords = sc.textFile('stopwords.pt')
stopwords = stopwords.flatMap(lambda word: word.split())
stopwords = stopwords.collect()

stopwords

def filter_words(word):
    print(word[0])
    if word[0] in stopwords:
        return False
    return True

counts = file2.flatMap(lambda line: line.split(" ")).map(lambda word: word.lower()).map(lambda word: (word, 1)).reduceByKey(lambda a, b: a + b)
counts = counts.filter(filter_words)
counts = counts.takeOrdered(100, key = lambda x: -x[1])

print(counts)


I am working on the file "2019/12-spark/12-spark-intro/spark_OtacilioBezerra.ipynb".
The first task is:
Filtragem de Primos
Dado uma sequência de números de 1 a 1000000, filtre somente os primos dessa sequência.
Can you write Python code for it?

# Função que verifica números primos
def isPrimo(n):
    if(n == 2): return True
    for i in range(2, n):
        if(n % i == 0):
            return False
    return True

# Gera todos os números de 1 até 1000000 e filtra somente os números primos
data = sc.parallelize(range(1,1000000))
res = data.filter(isPrimo)

print(res.collect())


I am working on the file "2019/12-spark/12-spark-intro/spark_OtacilioBezerra.ipynb".
The first task is:
Municípios do Brasil
Dado o dataset mucipios_do_Brasil.csv, faça duas operações com ele:

Monte uma lista dos municípios por estado.
Conte quantos municípios há em cada estado.

Dicas: use as operações groupByKey e reduceByKey, não faça um count na lista da operação 1.
Can you write Python code for it?

# Função para estrutura chave-valor
def chaveUF(l):
    line = l.split(",")
    return (line[0], line[1])

# Carrega o dataset como dicionário {uf:cidade}
lines = sc.textFile("municipios_do_Brasil.csv")
lines = lines.filter( lambda l: l.split(",")[0] != "uf")
dict = lines.map(chaveUF)

for pair in dict.groupByKey().collect():
    print(pair[0], ":", list(pair[1]))
    print()

# Carrega o dataset como um dataframe
df = spark.read.csv("municipios_do_Brasil.csv", header=True)

df.groupBy("uf").count().show()


I am working on the file "2019/12-spark/12-spark-intro/spark_OtacilioBezerra.ipynb".
The first task is:
Word Count Memória Postumas de Brás Cubas
Memórias Póstumas de Brás Cubas é um romance escrito por Machado de Assis, desenvolvido em princípio como folhetim, de março a dezembro de 1880, na Revista Brasileira, para, no ano seguinte, ser publicado como livro, pela então Tipografia Nacional.
A obra retrata a escravidão, as classes sociais, o cientificismo e o positivismo da época. Dada essas informações, será que conseguimos idenficar essas características pelas palavras mais utilizadas em sua obra?
Utilizando o dataset Machado-de-Assis-Memorias-Postumas.txt, faça um word count e encontre as palavras mais utilizadas por Machado de Assis em sua obra. Não esqueça de utilizar stopwords.pt para remover as stop words!
Can you write Python code for it?

stopwords = sc.textFile("stopwords.pt").collect()
livro = sc.textFile("Machado-de-Assis-Memorias-Postumas.txt")

words = livro.flatMap( lambda l: l.split(" ") )
words = words.filter( lambda w: w not in stopwords )
words = words.map( lambda w: (w, 1))

wordCnt = words.reduceByKey( lambda x,y: x+y )

sortedWordCnt = wordCnt.map( lambda w: (w[1], w[0])).sortByKey(False)
for word in sortedWordCnt.take(100):
    print(word)


I am working on the file "2019/12-spark/13-spark-mllib/mllib_OtacilioBezerra.ipynb".
The first task is:
10 Palavras Mais Frequentes
Can you write Python code for it?

tokenizer = Tokenizer(inputCol="text", outputCol="words")
remover = StopWordsRemover(inputCol="words", outputCol="filtered")
count = CountVectorizer(inputCol="filtered", outputCol="features", vocabSize=10)

pipeline = Pipeline(stages=[tokenizer, remover, count])

model = pipeline.fit(data).transform(data)
model.select("features").show(truncate=False)


I am working on the file "2019/12-spark/13-spark-mllib/mllib_OtacilioBezerra.ipynb".
The first task is:
TF-IDF com CountVectorizer
No exemplo TFIDF, atualize a cell 15 para utilizar o Transformer CountVectorizer.
Can you write Python code for it?

countVect = CountVectorizer(inputCol="rawFeatures", outputCol="features")
countVectModel = countVect.fit(featurizedData)
rescaledData = countVectModel.transform(featurizedData)


I am working on the file "2019/12-spark/13-spark-mllib/mllib_OtacilioBezerra.ipynb".
The first task is:
n-Grams
Can you write Python code for it?

tokenizer = Tokenizer(inputCol="text", outputCol="words")
remover = StopWordsRemover(inputCol="words", outputCol="filtered")
ngram = NGram(inputCol="filtered", outputCol="ngrams", n=2)
count = CountVectorizer(inputCol="ngrams", outputCol="features", vocabSize=10)

pipeline = Pipeline(stages=[tokenizer, remover, ngram, count])

model = pipeline.fit(data).transform(data)
model.select("features").show(truncate=False)

ngram = NGram(inputCol="filtered", outputCol="ngrams", n=3)

pipeline = Pipeline(stages=[tokenizer, remover, ngram, count])

model = pipeline.fit(data).transform(data)
model.select("features").show(truncate=False)


I am working on the file "2019/12-spark/14-spark-mllib-classification/LogisticRegressionClassifier.ipynb".
The first task is:
Classification algorithms requires numeric values for labels
Can you write Python code for it?

indexer = StringIndexer(inputCol="label", outputCol="labelIndex")
indexer = indexer.fit(df).transform(df)
indexer.show()

dfLabeled = indexer.rdd.map(mapLibSVM).toDF(["label", "features"])
dfLabeled.show()
train, test = dfLabeled.randomSplit([0.9, 0.1], seed=12345)


I am working on the file "2019/12-spark/14-spark-mllib-classification/LogisticRegressionClassifier.ipynb".
The first task is:
Compute the predictions from the model
Can you write Python code for it?

result = model.transform(test)
predictions = result.select(["prediction", "label"])
predictions.show()

# Instantiate metrics object
metrics = MulticlassMetrics(predictions.rdd)

# Overall statistics
print("Summary Stats")
print("Precision = %s" % metrics.precision())
print("Recall = %s" % metrics.recall())
print("F1 Score = %s" % metrics.fMeasure())
print("Accuracy = %s" % metrics.accuracy)

# Weighted stats
print("Weighted recall = %s" % metrics.weightedRecall)
print("Weighted precision = %s" % metrics.weightedPrecision)
print("Weighted F(1) Score = %s" % metrics.weightedFMeasure())
print("Weighted F(0.5) Score = %s" % metrics.weightedFMeasure(beta=0.5))
print("Weighted false positive rate = %s" % metrics.weightedFalsePositiveRate)


I am working on the file "2019/12-spark/14-spark-mllib-classification/LogisticRegressionClassifier.ipynb".
The first task is:
Fit the pipeline to training documents.
Can you write Python code for it?


I am working on the file "2019/12-spark/14-spark-mllib-classification/LogisticRegressionClassifier.ipynb".
The first task is:
In this case the estimator is simply the linear regression.
A TrainValidationSplit requires an Estimator, a set of Estimator ParamMaps, and an Evaluator.
Can you write Python code for it?

tvs = TrainValidationSplit(estimator=lr,
                           estimatorParamMaps=paramGrid,
                           evaluator=RegressionEvaluator(),
                           # 80% of the data will be used for training, 20% for validation.
                           trainRatio=0.9)


I am working on the file "2019/12-spark/14-spark-mllib-classification/LogisticRegressionClassifier.ipynb".
The first task is:
Utility function to create the appropriate data frame for classification algorithms in MLlib
Can you write Python code for it?

def mapLibSVM(row):
    return (row[5],Vectors.dense(row[:3]))


I am working on the file "2019/12-spark/14-spark-mllib-classification/LogisticRegressionClassifier.ipynb".
The first task is:
We use a ParamGridBuilder to construct a grid of parameters to search over.
TrainValidationSplit will try all combinations of values and determine best model using the evaluator.
Can you write Python code for it?

paramGrid = ParamGridBuilder()\
    .addGrid(lr.regParam, [0.1, 0.001]) \
    .build()


I am working on the file "2019/12-spark/14-spark-mllib-classification/LogisticRegressionClassifier.ipynb".
The first task is:
create the dataframe from a csv
Can you write Python code for it?

df = spark.read \
        .format("csv") \
        .option("header", "true") \
        .option("inferSchema", "true") \
        .load("datasets/iris.data")


I am working on the file "2019/12-spark/14-spark-mllib-classification/mllibClass_OtacilioBezerra.ipynb".
The first task is:
Avaliação do Modelo
Can you write Python code for it?

# Instânciação dos Objetos de Métrics
metrics = MulticlassMetrics(predictions.rdd)

# Estatísticas Gerais para o Método TrainValidationSplit
print("Summary Stats")
print("F1 Score = %s" % metrics.fMeasure())
print("Accuracy = %s" % metrics.accuracy)
print("Weighted recall = %s" % metrics.weightedRecall)
print("Weighted precision = %s" % metrics.weightedPrecision)
print("Weighted F(1) Score = %s" % metrics.weightedFMeasure())
print("Weighted F(0.5) Score = %s" % metrics.weightedFMeasure(beta=0.5))
print("Weighted false positive rate = %s" % metrics.weightedFalsePositiveRate)


I am working on the file "2019/12-spark/14-spark-mllib-classification/mllibClass_OtacilioBezerra.ipynb".
The first task is:
Avaliação dos Modelos
Can you write Python code for it?

# Instânciação dos Objetos de Métrics
metrics_tvs = MulticlassMetrics(preds_tvs.rdd)
metrics_cval = MulticlassMetrics(preds_cval.rdd)

# Estatísticas Gerais para o Método TrainValidationSplit
print("Summary Stats")
print("F1 Score = %s" % metrics_tvs.fMeasure())
print("Accuracy = %s" % metrics_tvs.accuracy)
print("Weighted recall = %s" % metrics_tvs.weightedRecall)
print("Weighted precision = %s" % metrics_tvs.weightedPrecision)
print("Weighted F(1) Score = %s" % metrics_tvs.weightedFMeasure())
print("Weighted F(0.5) Score = %s" % metrics_tvs.weightedFMeasure(beta=0.5))
print("Weighted false positive rate = %s" % metrics_tvs.weightedFalsePositiveRate)

# Estatísticas Gerais para o Método TrainValidationSplit
print("Summary Stats")
print("F1 Score = %s" % metrics_cval.fMeasure())
print("Accuracy = %s" % metrics_cval.accuracy)
print("Weighted recall = %s" % metrics_cval.weightedRecall)
print("Weighted precision = %s" % metrics_cval.weightedPrecision)
print("Weighted F(1) Score = %s" % metrics_cval.weightedFMeasure())
print("Weighted F(0.5) Score = %s" % metrics_cval.weightedFMeasure(beta=0.5))
print("Weighted false positive rate = %s" % metrics_cval.weightedFalsePositiveRate)


I am working on the file "2019/12-spark/14-spark-mllib-classification/mllibClass_OtacilioBezerra.ipynb".
The first task is:
Convertendo a saída de categórica para numérica
Can you write Python code for it?

indexer = StringIndexer(inputCol="label", outputCol="labelIndex")
indexer = indexer.fit(df).transform(df)
indexer.show()

dfLabeled = indexer.rdd.map(mapLibSVM).toDF(["label", "features"])
dfLabeled.show()

train, test = dfLabeled.randomSplit([0.9, 0.1], seed=12345)


I am working on the file "2019/12-spark/14-spark-mllib-classification/mllibClass_OtacilioBezerra.ipynb".
The first task is:
Cross-Validation - TrainValidationSplit e CrossValidator
Can you write Python code for it?

paramGrid = ParamGridBuilder()\
    .addGrid(lr.regParam, [0.1, 0.001]) \
    .build()

tvs = TrainValidationSplit(estimator=lr,
                           estimatorParamMaps=paramGrid,
                           evaluator=MulticlassClassificationEvaluator(),
                           trainRatio=0.8)

cval = CrossValidator(estimator=lr,
                      estimatorParamMaps=paramGrid,
                      evaluator=MulticlassClassificationEvaluator(),
                      numFolds=10)


I am working on the file "2019/12-spark/14-spark-mllib-classification/mllibClass_OtacilioBezerra.ipynb".
The first task is:
Definição do Modelo de Árvores Randômicas
Can you write Python code for it?

rf = RandomForestClassifier(labelCol="label", featuresCol="features")


I am working on the file "2019/12-spark/14-spark-mllib-classification/mllibClass_OtacilioBezerra.ipynb".
The first task is:
Funções
Can you write Python code for it?

def mapLibSVM(row):
    return (row[5],Vectors.dense(row[:3]))

df = spark.read \
        .format("csv") \
        .option("header", "true") \
        .option("inferSchema", "true") \
        .load("datasets/iris.data")


I am working on the file "2019/12-spark/14-spark-mllib-classification/mllibClass_OtacilioBezerra.ipynb".
The first task is:
Treino do Modelo e Predição do Teste
Can you write Python code for it?

results = cval.fit(train).transform(test)

predictions = results.select(["prediction", "label"])


result_tvs = tvs.fit(train).transform(test)
result_cval = cval.fit(train).transform(test)

preds_tvs = result_tvs.select(["prediction", "label"])
preds_cval = result_cval.select(["prediction", "label"])


I am working on the file "2019/Com520B/VisualizationNotebook.ipynb".
The first task is:
Challenge
Last week, we processed the congressional district data to produce a data set with the percent of the working population in each industry compared to the mean and median salary in that district. You opened that data in Excel to produce a plot.
Recreate a plot of percent of the district in engineering and the mean salary for that district. Make sure to label your axes.
(Hint) remember: all of the python you have learned will work in the notebook. You can still use open for instance.
Can you write Python code for it?


I am working on the file "2019/Com520B/VisualizationNotebook.ipynb".
The first task is:
Do it!
Read in the mean salary per congressional district and construct a histogram from that data.
* Investigate the bins argument. What happens if you set this too high? What if you set it too low? Generally, sqrt(n) is a good size for n data points.
Challenge
Get the salary data from here (it's the tsv file).  Import this file into python and build a histogram of salary.

How many bins are a good fit?
There are clearly outliers in this data. Investigate the range argument to hist to help truncate the plot.
What does the normalize argument do?

Hint: here is the documentation
Can you write Python code for it?


I am working on the file "2019/Com520B/VisualizationNotebook.ipynb".
The first task is:
Improving the plot
When I made the plot above, it was a very jumbled line chart, which didn't look very good. Using the tutorial figure out how to plot the points as dots rather than as a line chart.
Can you write Python code for it?


I am working on the file "2019/Com520B/VisualizationNotebook.ipynb".
The first task is:
Try This
Using the documentation for matplotlib.pyplot found here make a plot for the following data set. In this case, you want to plot both x and y values.
Can you write Python code for it?


I am working on the file "2019/Com520B/VisualizationNotebook.ipynb".
The first task is:
Try it!
Set a context like above and rerun any of the plots in your document.
Histograms
Pyplot includes excellent support for histograms. A histogram uses the range of your data on the x-axis. It divides that range in to buckets (usually with equal widths) and counts the number of values that fall into each bucket. Look at the example below:
Can you write Python code for it?


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
... and now, for the apatite images.
Can you write Python code for it?

autootsu_apte = np.loadtxt('auto_count/autootsu_apatite.csv', delimiter=',')
autoyen_apte = np.loadtxt('auto_count/autoyen_apatite.csv', delimiter=',')
autoli_apte = np.loadtxt('auto_count/autoli_apatite.csv', delimiter=',')
autoiso_apte = np.loadtxt('auto_count/autoiso_apatite.csv', delimiter=',')
automlss_apte = np.loadtxt('auto_count/automlss_apatite.csv', delimiter=',')


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
... and now, the apatite samples.
Can you write Python code for it?

regotsu_apte = ds.regions_and_skel(binotsu_apte)
regyen_apte = ds.regions_and_skel(binyen_apte)
regli_apte = ds.regions_and_skel(binli_apte)
regiso_apte = ds.regions_and_skel(biniso_apte)
regmlss_apte = ds.regions_and_skel(binmlss_apte)


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Automatic counting

Here we have functions for automatically classifying and counting tracks.
Can you write Python code for it?

def autocount_tracks():
    # Otsu
    print('* counting tracks in Otsu binary images.')
    ds.count_and_save(regotsu_mica,
                      filename='auto_count/autootsu_mica.csv',
                      show_exectime=False)
    ds.count_and_save(regotsu_apte,
                      filename='auto_count/autootsu_apatite.csv',
                      show_exectime=False)

    # Yen
    print('* counting tracks in Yen binary images.')
    ds.count_and_save(regyen_mica,
                      filename='auto_count/autoyen_mica.csv',
                      show_exectime=False)
    ds.count_and_save(regyen_apte,
                      filename='auto_count/autoyen_apatite.csv',
                      show_exectime=False)

    # Li
    print('* counting tracks in Li binary images.')
    ds.count_and_save(regli_mica,
                      filename='auto_count/autoli_mica.csv',
                      show_exectime=False)
    ds.count_and_save(regli_apte,
                      filename='auto_count/autoli_apatite.csv',
                      show_exectime=False)

    # ISODATA
    print('* counting tracks in ISODATA binary images.')
    ds.count_and_save(regiso_mica,
                      filename='auto_count/autoiso_mica.csv',
                      show_exectime=False)
    ds.count_and_save(regiso_apte,
                      filename='auto_count/autoiso_apatite.csv',
                      show_exectime=False)

    # MLSS
    print('* counting tracks in MLSS binary images.')
    ds.count_and_save(regmlss_mica,
                      filename='auto_count/automlss_mica.csv',
                      show_exectime=False)
    ds.count_and_save(regmlss_apte,
                      filename='auto_count/automlss_apatite.csv',
                      show_exectime=False)

    return None


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Binarizing images.
Can you write Python code for it?

binotsu_mica, binyen_mica, binli_mica, biniso_mica = binarize_imageset(imgset_mica)
binotsu_apte, binyen_apte, binli_apte, biniso_apte = binarize_imageset(imgset_apte)

# reading and processing the binary MLSS images.
binmlss_mica, binmlss_apte = [], []

for fname in fnames_mica:
    mlss_fname = 'auto_count/mlss/' + fname.split('/')[1][:-4] + '.csv'
    binmlss_mica.append(_process_image(np.asarray(pd.read_csv(mlss_fname), dtype='bool'), threshold=None))

for fname in fnames_apte:
    mlss_fname = 'auto_count/mlss/' + fname.split('/')[1][:-4] + '.csv'
    binmlss_apte.append(_process_image(np.asarray(pd.read_csv(mlss_fname), dtype='bool'), threshold=None))


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Calling support functions

This how-to depends on some functions to generate results and to help plotting data. Here we call all these functions.
Can you write Python code for it?


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Defining helping constants

Here we define some helping constants: WEIGHT_FILTER, the weight for the TV Chambolle function, and MIN_SIZE, the minimum area for considering an interest region as a track.
Can you write Python code for it?


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Exemplifying the binarizations used

Here we present a sample binarized using the algorithms proposed. First, we use a muscovite image:
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(18, 9), nrows=1, ncols=3)

SAMPLE = 4

# Otsu
ax[0].imshow(binotsu_mica[SAMPLE], cmap='gray')
ax[0].set_xlabel('Otsu')

# Yen
ax[1].imshow(binyen_mica[SAMPLE], cmap='gray')
ax[1].set_xlabel('Yen')

# Li
ax[2].imshow(binli_mica[SAMPLE], cmap='gray')
ax[2].set_xlabel('Li')

fig, ax = plt.subplots(figsize=(18, 9), nrows=1, ncols=2)

# ISODATA
ax[0].imshow(biniso_mica[SAMPLE], cmap='gray')
ax[0].set_xlabel('ISODATA')

# MLSS
ax[1].imshow(binmlss_mica[SAMPLE], cmap='gray')
ax[1].set_xlabel('MLSS')


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Filtering and binarizing images

Here we binarize all images using different algorithms: Otsu, Yen, Li, ISODATA, triangle, MLSS.

We also perform some cleaning actions:

remove_small_objects(), in its default settings, removes objects with an area smaller than MIN_SIZE pixels.
binary_fill_holes() fills holes contained in objects.
clear_rd_border() removes objects touching the lower and right borders. These objects could not be identified with precision.

After that, we present the results of these actions on the first images from the set.
Can you write Python code for it?

def binarize_imageset(image_set):

    imgbin_otsu, imgbin_yen, imgbin_li, imgbin_iso = [[] for _ in range(4)]

    for image in image_set:
        # Filtering.
        image = denoise_tv_chambolle(image, weight=WEIGHT_FILTER)
        # Binarizing.
        imgbin_otsu.append(_process_image(image, threshold='otsu'))  # Otsu
        imgbin_yen.append(_process_image(image, threshold='yen'))  # Yen
        imgbin_li.append(_process_image(image, threshold='li'))  # Li
        imgbin_iso.append(_process_image(image, threshold='isodata'))  # ISODATA

    return imgbin_otsu, imgbin_yen, imgbin_li, imgbin_iso

def _process_image(image, threshold='isodata', weight=WEIGHT_FILTER, min_size=MIN_SIZE):
    """
    """

    if threshold is not None:
        func_thresh = {'otsu': threshold_otsu,
                       'yen': threshold_yen,
                       'li': threshold_li,
                       'isodata': threshold_isodata,
                       }.get(threshold, threshold_isodata)
        thresh = func_thresh(image)
        image = image < thresh

    img_bin = ds.clear_rd_border(remove_small_objects(
                                     binary_fill_holes(image),
                                         min_size=min_size))

    return img_bin


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
ISODATA
Can you write Python code for it?

for idx, image in enumerate(image_set):
    fig, ax = plt.subplots(figsize=(15, 10))
    ax, count = ds.plot_and_count(imgbin_isodata[idx], intensity_image=image)
    filename = 'res_figures/isodata/image' + str(idx+1) + '_isodata.eps'
    plt.savefig(filename, bbox_inches='tight')
    plt.clf()


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Manual counting

Here we read the manual counting results, obtained by W.M.N.
Can you write Python code for it?

manual_mica = ds.manual_counting(mineral='mica')
manual_apte = ds.manual_counting(mineral='apatite')

def ratio_manauto(manual, auto):
    """
    """

    ratio = np.asarray(auto) / np.asarray(manual)

    print('\n* Manual mean: ', str(manual.mean()), ' +/- ', str(manual.std()),
          '\n* Auto mean: ', str(auto.mean()), ' +/- ', str(auto.std()),
          '\n* Ratio mean: ', str(ratio.mean()), ' +/- ', str(ratio.std()))

    return None

print('* Otsu:')
ratio_manauto(np.asarray(manual_mica), np.asarray(autootsu_mica))
ratio_manauto(np.asarray(manual_apte), np.asarray(autootsu_apte))

np.mean(automlss_mica)/np.mean(automlss_apte)

print('* Yen:')
ratio_manauto(np.asarray(manual_mica), np.asarray(autoyen_mica))
ratio_manauto(np.asarray(manual_apte), np.asarray(autoyen_apte))

print('* Li:')
ratio_manauto(np.asarray(manual_mica), np.asarray(autoli_mica))
ratio_manauto(np.asarray(manual_apte), np.asarray(autoli_apte))

print('* ISODATA:')
ratio_manauto(np.asarray(manual_mica), np.asarray(autoiso_mica))
ratio_manauto(np.asarray(manual_apte), np.asarray(autoiso_apte))

print('* MLSS:')
ratio_manauto(np.asarray(manual_mica), np.asarray(automlss_mica))
ratio_manauto(np.asarray(manual_apte), np.asarray(automlss_apte))

def plot_tracks(image, trk_averages, trk_points):
    '''
    '''

    if trk_averages is None:
        print('No averages (possibly only one track).')
        return None

    image_rgb = gray2rgb(image)

    # preparing the plot window.
    fig, ax = plt.subplots(nrows=1, ncols=1)

    for trk_idx, trk_pt in enumerate(trk_points):

        # calculating route and distances.
        route, _ = route_through_array(~image, trk_pt[0], trk_pt[1])
        distances, _, _ = ds.track_parameters(trk_pt[0], trk_pt[1], route)

        # generating minimal distance line.
        rows, cols = line(trk_pt[0][0], trk_pt[0][1],
                          trk_pt[1][0], trk_pt[1][1])
        image_rgb[rows, cols] = [0, 255, 0]

        # plotting minimal distance and route.
        ax.imshow(image_rgb,...


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Now we import the functions presented in this study. They are inside the file desiqueira2018.py; please ensure that you are executing this how-to in the same folder as these functions, or the following command will return an error.
Can you write Python code for it?


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Now, we show the results on an apatite image.
Can you write Python code for it?

fig, ax = plt.subplots(figsize=(18, 9), nrows=1, ncols=3)

# Otsu
ax[0].imshow(binotsu_apte[SAMPLE], cmap='gray')
ax[0].set_xlabel('Otsu')

# Yen
ax[1].imshow(binyen_apte[SAMPLE], cmap='gray')
ax[1].set_xlabel('Yen')

# Li
ax[2].imshow(binli_apte[SAMPLE], cmap='gray')
ax[2].set_xlabel('Li')

fig, ax = plt.subplots(figsize=(18, 9), nrows=1, ncols=2)

# ISODATA
ax[0].imshow(biniso_apte[SAMPLE], cmap='gray')
ax[0].set_xlabel('ISODATA')

# MLSS
ax[1].imshow(binmlss_apte[SAMPLE], cmap='gray')
ax[1].set_xlabel('MLSS')


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Obtaining the image collection

First, we use ImageCollection() to read all images in the folder orig_figures. After that, we print the number of images in this dataset.
Can you write Python code for it?

fnames_mica = sorted(glob('orig_figures/*mica*.tif'))
imgset_mica = [io.imread(fname, as_gray=True) for fname in fnames_mica]

fnames_apte = sorted(glob('orig_figures/*apatite*.tif'))
imgset_apte = [io.imread(fname, as_gray=True) for fname in fnames_apte]

print('Number of images on dataset: ', len(imgset_mica) + len(imgset_apte))


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Obtaining track stats for each counting.
Can you write Python code for it?

otsu_stats = ds.TrackStats(autootsu_mica, autootsu_apte)
yen_stats = ds.TrackStats(autoyen_mica, autoyen_apte)
li_stats = ds.TrackStats(autoli_mica, autoli_apte)
iso_stats = ds.TrackStats(autoiso_mica, autoiso_apte)
mlss_stats = ds.TrackStats(automlss_mica, automlss_apte)
manual_stats = ds.TrackStats(manual_mica, manual_apte)

otsu_stats.gqr

yen_stats.gqr

iso_stats.gqr

li_stats.gqr

mlss_stats.gqr

manual_stats.gqr

print(otsu_stats.ext_dec.sum(),
      np.asarray(track_stats.n_extdec)/len(track_stats.ext_dec),
      otsu_stats.gqr)


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Some settings

Here we establish a default font size, modify the resulting plots, ask to Matplotlib draw them on this how-to (and not on a separate window), and disable all warnings that could be generated. Since we will work with a large number of images, this is useful to avoid the several warnings that would appear.
Can you write Python code for it?

default_fontsize = 15

plt.rcParams['font.family'] = 'monospace'
plt.rcParams['font.size'] = 15
plt.rcParams['axes.labelsize'] = plt.rcParams['font.size']
plt.rcParams['axes.titlesize'] = 1.5*plt.rcParams['font.size']
plt.rcParams['legend.fontsize'] = 1.5*plt.rcParams['font.size']
plt.rcParams['xtick.labelsize'] = plt.rcParams['font.size']
plt.rcParams['ytick.labelsize'] = plt.rcParams['font.size']

%matplotlib inline
warnings.filterwarnings('ignore')


I am working on the file "2019/skeletracks/afdesiqueira_2018-howto.ipynb".
The first task is:
Yen
Can you write Python code for it?

for idx, image in enumerate(image_set):
    fig, ax = plt.subplots(figsize=(15, 10))
    ax, count = ds.plot_and_count(imgbin_yen[idx], intensity_image=image)
    filename = 'res_figures/yen/image' + str(idx+1) + '_yen.eps'
    plt.savefig(filename, bbox_inches='tight')
    plt.clf()


I am working on the file "20190130-EMSE6992-commandline-demo.ipynb".
The first task is:
Choose multiple fields to extract from all tweets, using jq and the -c flag for compact output.
The following examples uses head to only show a few lines and truncate the output here, but you can also use cat to process the whole file.
Can you write Python code for it?

!head sample.json | jq -c '[.id_str, .text, .user.screen_name, .created_at]'


I am working on the file "20190130-EMSE6992-commandline-demo.ipynb".
The first task is:
Create a CSV with a subset of the fields.
Can you write Python code for it?

!cat sample.json | jq -r '[.id_str, .created_at, .text] | @csv' > tweets.csv

!head -20 tweets.csv


I am working on the file "20190130-EMSE6992-commandline-demo.ipynb".
The first task is:
For filtering and output, it is helpful to transform the date into a more uesful format. Let's look at the original created_at date and transform it into an ISO 8601 date.
Can you write Python code for it?

!head sample.json | jq -c '[.created_at, (.created_at | strptime("%A %B %d %T %z %Y") | todate)]'


I am working on the file "20190130-EMSE6992-commandline-demo.ipynb".
The first task is:
Hashtags are in nested JSON, so let's flattening the hashtags into a semi-colon delimited list.
Can you write Python code for it?

!head -200 sample.json | jq -c 'select(.entities.hashtags | length >= 1) | [([.entities.hashtags[].text] | join(";"))]'


I am working on the file "20190130-EMSE6992-commandline-demo.ipynb".
The first task is:
Note that that newlines in the tweet text break the CSV format, which should only have line breaks at the end of the row. Let's fix problems with the text field in the tweets by replacing the \n newline character with a space.
Can you write Python code for it?

!cat sample.json | jq -r '[.id_str, (.text | gsub("\n";" "))] | @csv' > tweets.csv

!head -20 tweets.csv


I am working on the file "20190130-EMSE6992-commandline-demo.ipynb".
The first task is:
View the first tweet in the file using jq.
Can you write Python code for it?


I am working on the file "20190306_BCS_Scotland_Presentation/SetUpEnvironment.ipynb".
The first task is:
PY27 presentation base
Can you write Python code for it?

# From scratch

# conda install -n py-27-prez [package]

# remove old virtual env

conda remove -n py-27-prez --all

# create new virtual env

conda create -n py-27-prez python=2.7

# activate virtual env

source activate py-27-prez

# install other packages

conda install -c conda-forge jupyter tabulate nbtutor jupyter_contrib_nbextensions rise nbdime nbconvert


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Apply Mask

load and plot mask if exist
otherwise create a mask using Mask pipeline
Reverse the mask in y-direction due to the coordination difference between python and Eiger software
Reverse images in y-direction
Apply the mask

Change the lines below to give mask filename
Can you write Python code for it?

if scat_geometry == 'gi_saxs':
    mask_path = '/XF11ID/analysis/2018_2/masks/'
    mask_name =  'July13_2018_4M.npy'

elif scat_geometry == 'saxs':
    mask_path = '/XF11ID/analysis/2019_1/masks/'
    if md['detector'] =='eiger4m_single_image' or md['detector'] == 'image':
        mask_name = 'Feb6_2019_4M_SAXS.npy'
    elif md['detector'] =='eiger500K_single_image':
        mask_name = 'Feb6_2019_500K_SAXS.npy'

        mask_path = '/XF11ID/analysis/2017_1/masks/'
        mask_name = 'Feb23_500K_SAXS6_mask.npy'

elif scat_geometry == 'gi_waxs':
    mask_path = '/XF11ID/analysis/2018_2/masks/'
    mask_name = 'July20_2018_1M_WAXS.npy'

mask = load_mask(mask_path, mask_name, plot_ =  False, image_name = uidstr + '_mask', reverse= reverse, rot90=rot90  )
mask =  mask * pixel_mask * Chip_Mask
show_img(mask,image_name = uidstr + '_mask', save=True, path=data_dir, aspect=1, center=center[::-1])
mask_load=mask.copy()
imgsa = apply_mask( imgs, mask )


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Attach the PDF report to Olog
Can you write Python code for it?

#%run  /home/yuzhang/chxanalys_link/chxanalys/chx_olog.py

if att_pdf_report:
    os.environ['HTTPS_PROXY'] = 'https://proxy:8888'
    os.environ['no_proxy'] = 'cs.nsls2.local,localhost,127.0.0.1'
    update_olog_uid_with_file( uid[:6], text='Add XPCS Analysis PDF Report',
                              filename=pdf_out_dir + pdf_filename, append_name='_R1' )


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Check one ROI intensity
Can you write Python code for it?

roi_inten = check_ROI_intensity( avg_img, roi_mask, ring_number= 2, uid =uidstr ) #roi starting from 1


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Check several frames average  intensity
Can you write Python code for it?

img_choice_N = 3
img_samp_index = random.sample( range(len(imgs)), img_choice_N)
avg_img =  get_avg_img( imgsa, img_samp_index, plot_ = False, uid =uidstr)
if avg_img.max() == 0:
    print('There are no photons recorded for this uid: %s'%uid)
    print('The data analysis should be terminated! Please try another uid.')

#show_img( imgsa[1000],  vmin=.1, vmax= 1e1, logs=True, aspect=1,
#         image_name= uidstr + '_img_avg',  save=True, path=data_dir,  cmap = cmap_albula )
print(center[::-1])

show_img( imgsa[ 5],  vmin = -1, vmax = 20, logs=False, aspect=1, #save_format='tif',
         image_name= uidstr + '_img_avg',  save=True, path=data_dir, cmap=cmap_albula,center=center[::-1])
# select subregion, hard coded center beam location
#show_img( imgsa[180+40*3/0.05][110:110+840*2, 370:370+840*2],  vmin = 0.01, vmax = 20, logs=False, aspect=1, #save_format='tif',
#         image_name= uidstr + '_img_avg',  save=True, path=data_dir, cmap=cmap_albula,center=[845,839])


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Creat PDF Report
Can you write Python code for it?

pdf_out_dir = os.path.join('/XF11ID/analysis/', CYCLE, username, 'Results/')

pdf_filename = "XPCS_Analysis_Report2_for_uid=%s%s%s.pdf"%(uid,pdf_version,q_mask_name)
if run_xsvs:
    pdf_filename = "XPCS_XSVS_Analysis_Report_for_uid=%s%s%s.pdf"%(uid,pdf_version,q_mask_name)

#%run /home/yuzhang/chxanalys_link/chxanalys/Create_Report.py

data_dir

make_pdf_report( data_dir, uid, pdf_out_dir, pdf_filename, username,
                    run_fit_form,run_one_time, run_two_time, run_four_time, run_xsvs, run_dose,
                report_type= scat_geometry, report_invariant= run_invariant_analysis,
               md = md )


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Creat new mask by masking the bad pixels and get new avg_img
Can you write Python code for it?

imgsum_y = imgsum[ np.array( [i for i in np.arange( len(imgsum)) if i not in bad_frame_list])]
imgsum_x = np.arange( len( imgsum_y))
save_lists(  [imgsum_x, imgsum_y], label=['Frame', 'Total_Intensity'],
           filename=uidstr + '_img_sum_t', path= data_dir  )


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Do a waterfall analysis
Can you write Python code for it?

qth_interest = 2 #the second ring. #qth_interest starting from 1
if scat_geometry =='saxs' or scat_geometry =='gi_waxs':
    if run_waterfall:
        wat = cal_waterfallc( FD, roi_mask, qindex= qth_interest, save =True, path=data_dir, uid=uidstr)
        plot_waterfallc( wat, qth_interest, aspect= None, vmin=1e-1, vmax= wat.max(), uid=uidstr, save =True,
                        path=data_dir, beg= FD.beg, cmap = cmap_vge )

ring_avg = None
if run_t_ROI_Inten:
    times_roi, mean_int_sets = cal_each_ring_mean_intensityc(FD, roi_mask, timeperframe = None, multi_cor=True  )
    plot_each_ring_mean_intensityc( times_roi, mean_int_sets,  uid = uidstr, save=True, path=data_dir )
    roi_avg = np.average( mean_int_sets, axis=0)


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Do historam
Can you write Python code for it?

if run_xsvs:
    if roi_avg is  None:
        times_roi, mean_int_sets = cal_each_ring_mean_intensityc(FD, roi_mask, timeperframe = None,  )
        roi_avg = np.average( mean_int_sets, axis=0)

    t0=time.time()
    spec_bins, spec_his, spec_std, spec_sum  =  xsvsp( FD, np.int_(roi_mask), norm=None,
                max_cts=int(max_cts+2),  bad_images=bad_frame_list, only_two_levels=True )
    spec_kmean =  np.array(  [roi_avg * 2**j for j in  range( spec_his.shape[0] )] )
    run_time(t0)
    spec_pds =  save_bin_his_std( spec_bins, spec_his, spec_std, filename=uid_+'_spec_res.csv', path=data_dir )


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Do historam fit by negtive binominal function with maximum likehood method
Can you write Python code for it?

if run_xsvs:
    ML_val, KL_val,K_ = get_xsvs_fit(  spec_his, spec_sum, spec_kmean,
                        spec_std, max_bins=2, fit_range=[1,60], varyK= False  )
    #print( 'The observed average photon counts are: %s'%np.round(K_mean,4))
    #print( 'The fitted average photon counts are: %s'%np.round(K_,4))
    print( 'The difference sum of average photon counts between fit and data are: %s'%np.round(
            abs(np.sum( spec_kmean[0,:] - K_ )),4))
    print( '#'*30)
    qth=   0
    print( 'The fitted M for Qth= %s are: %s'%(qth, ML_val[qth]) )
    print( K_[qth])
    print( '#'*30)


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Don't Change the lines below here

get metadata
Can you write Python code for it?

md = get_meta_data( uid )
md_blue = md.copy()
#md_blue

#md_blue['detectors'][0]
#if md_blue['OAV_mode'] != 'none':
#    cx , cy = md_blue[md_blue['detectors'][0]+'_beam_center_x'], md_blue[md_blue['detectors'][0]+'_beam_center_x']
#else:
#    cx , cy = md_blue['beam_center_x'], md_blue['beam_center_y']
#print(cx,cy)

detectors = sorted(get_detectors(db[uid]))
print('The detectors are:%s'%detectors)
if len(detectors) >1:
    md['detector'] = detectors[1]
    print( md['detector'])

if md['detector'] =='eiger4m_single_image' or md['detector'] == 'image':
    reverse= True
    rot90= False
elif md['detector'] =='eiger500K_single_image':
    reverse= True
    rot90=True
elif md['detector'] =='eiger1m_single_image':
    reverse= True
    rot90=False
print('Image reverse: %s\nImage rotate 90: %s'%(reverse, rot90))

try:
    cx , cy = md_blue['beam_center_x'], md_blue['beam_center_y']
    print(cx,cy)
except:
    print('Will find cx,cy later.')


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Dynamic Analysis for gi_saxs
Can you write Python code for it?

if scat_geometry =='gi_saxs':
    show_qzr_roi( avg_img, roi_mask, inc_x0, ticks[:4], alpha=0.5, save=True, path=data_dir, uid=uidstr )
    ## Get 1D Curve (Q||-intensity¶)
    qr_1d_pds = cal_1d_qr( avg_img, Qr, Qz, qr_map, qz_map, inc_x0= None, mask=mask, setup_pargs=setup_pargs )
    plot_qr_1d_with_ROI( qr_1d_pds, qr_center=np.unique( np.array(list( qval_dict.values() ) )[:,0] ),
                    loglog=True, save=True, uid=uidstr, path = data_dir)


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Export Results to a HDF5 File
Can you write Python code for it?

md['mask_file']= mask_path + mask_name
md['roi_mask_file']= fp
md['mask'] = mask
#md['NOTEBOOK_FULL_PATH'] =  data_dir + get_current_pipeline_fullpath(NFP).split('/')[-1]
md['good_start'] = good_start
md['bad_frame_list'] = bad_frame_list
md['avg_img'] = avg_img
md['roi_mask'] = roi_mask
md['setup_pargs'] = setup_pargs
if scat_geometry == 'gi_saxs':
    md['Qr'] = Qr
    md['Qz'] = Qz
    md['qval_dict'] = qval_dict
    md['beam_center_x'] =  inc_x0
    md['beam_center_y']=   inc_y0
    md['beam_refl_center_x'] = refl_x0
    md['beam_refl_center_y'] = refl_y0

elif scat_geometry == 'gi_waxs':
    md['beam_center_x'] =  center[1]
    md['beam_center_y']=  center[0]
else:
    md['qr']= qr
    #md['qr_edge'] = qr_edge
    md['qval_dict'] = qval_dict
    md['beam_center_x'] =  center[1]
    md['beam_center_y']=  center[0]

md['beg'] = FD.beg
md['end'] = FD.end
md['t_print0'] = t_print0
md['qth_interest'] = qth_interest
md['metadata_file'] = data_dir + 'uid=%s_md.pkl'%uid

psave_obj(  md, data_dir + 'uid=%s_md.pkl'%uid ) #save the setup parameters
save_dict_csv( md,  data_dir + 'uid=%s_md.csv'%uid, 'w')

Exdt = {}
if scat_geometry == 'gi_saxs':
    for k,v in zip( ['md', 'roi_mask','qval_dict','avg_img','mask','pixel_mask', 'imgsum', 'bad_frame_list', 'qr_1d_pds'],
                [md,    roi_mask, qval_dict, avg_img,mask,pixel_mask, imgsum, bad_frame_list, qr_1d_pds] ):
        Exdt[ k ] = v
elif scat_geometry == 'saxs':
    for k,v in zip( ['md', 'q_saxs', 'iq_saxs','iqst','qt','roi_mask','qval_dict','avg_img','mask','pixel_mask', 'imgsum', 'bad_frame_list'],
                [md, q_saxs, iq_saxs, iqst, qt,roi_mask, qval_dict, avg_img,mask,pixel_mask, imgsum, bad_frame_list] ):
        Exdt[ k ] = v
elif scat_geometry == 'gi_waxs':
    for k,v in zip( ['md', 'roi_mask','qval_dict','avg_img','mask','pixel_mask', 'imgsum', 'bad_frame_list'],
                [md,       roi_mask, qval_dict, avg_img,mask,pixel_mask, imgsum, bad_frame_list] ):
        Exdt[ k ] = v

if...


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Extract the labeled array
Can you write Python code for it?

qind, pixelist = roi.extract_label_indices(roi_mask)
noqs = len(np.unique(qind))
print(noqs)


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
For two-time
Can you write Python code for it?

define_good_series = False
#define_good_series = True
if define_good_series:
    good_start = 5
    FD = Multifile(filename, beg = good_start, end = 1000)
    uid_ = uidstr + '_fra_%s_%s'%(FD.beg, FD.end)
    print( uid_ )

data_pixel = None
if run_two_time:
    data_pixel =   Get_Pixel_Arrayc( FD, pixelist,  norm= norm ).get_data()

import time
t0=time.time()
g12b=None
if run_two_time:
    g12b = auto_two_Arrayc(  data_pixel,  roi_mask, index = None   )
    if run_dose:
        np.save( data_dir + 'uid=%s_g12b'%uid, g12b)
run_time( t0 )

if run_two_time:
    show_C12(g12b, q_ind= 2, qlabel=dict(itertools.islice(qval_dict.items(),g2.shape[1])),N1= FD.beg,logs=False, N2=min( FD.end,10000), vmin= 1.0, vmax=1.18,timeperframe=timeperframe,save=True, path= data_dir, uid = uid_ ,cmap=plt.cm.jet)#cmap=cmap_albula)

multi_tau_steps = True
if run_two_time:
    if lag_steps is None:
        num_bufs=8
        noframes = FD.end - FD.beg
        num_levels = int(np.log( noframes/(num_bufs-1))/np.log(2) +1) +1
        tot_channels, lag_steps, dict_lag = multi_tau_lags(num_levels, num_bufs)
        max_taus= lag_steps.max()

    #max_taus= lag_steps.max()
    max_taus = Nimg
    t0=time.time()
    #tausb = np.arange( g2b.shape[0])[:max_taus] *timeperframe
    if multi_tau_steps:
        lag_steps_ = lag_steps[   lag_steps <= g12b.shape[0] ]
        g2b = get_one_time_from_two_time(g12b)[lag_steps_]
        tausb = lag_steps_ *timeperframe
    else:
        tausb = (np.arange( g12b.shape[0]) *timeperframe)[:-200]
        g2b = (get_one_time_from_two_time(g12b))[:-200]
    run_time(t0)

    g2b_pds = save_g2_general( g2b, taus=tausb, qr= np.array( list( qval_dict.values() ) )[:g2.shape[1],0],
                              qz=None, uid=uid_+'_'+q_mask_name+'_g2b.csv', path= data_dir, return_res=True )

if run_two_time:
    g2b_fit_result, tausb_fit, g2b_fit = get_g2_fit_general( g2b,  tausb,
                function = fit_g2_func,  vlim=[0.95, 1.05], fit_range= None,
         ...


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Get Dynamic Mask (currently designed for 500K)
Can you write Python code for it?

if  md['detector'] =='eiger4m_single_image' or md['detector'] == 'image':
    pass
elif md['detector'] =='eiger500K_single_image':
    #if md['cam_acquire_period'] <= 0.00015:  #will check this logic
    if imgs[0].dtype == 'uint16':
        print('Create dynamic mask for 500K due to 9K data acquistion!!!')
        bdp = find_bad_pixels_FD( bad_frame_list, FD, img_shape = avg_img.shape, threshold=20 )
        mask = mask_copy2.copy()
        mask *=bdp
        mask_copy = mask.copy()
        show_img(  mask, image_name='New Mask_uid=%s'%uid )


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Get bad frame list by a polynominal fit
Can you write Python code for it?

good_end= None # 2000
if good_end is not None:
    FD = Multifile(filename, good_start, min( len(imgs)//bin_frame_number, good_end) )
    uid_ = uidstr + '_fra_%s_%s'%(FD.beg, FD.end)
    print( uid_ )

re_define_good_start =False
if re_define_good_start:
    good_start = 180
    #good_end = 19700
    good_end = len(imgs)
    FD = Multifile(filename, good_start, good_end)
    uid_ = uidstr + '_fra_%s_%s'%(FD.beg, FD.end)
    print( FD.beg, FD.end)

bad_frame_list =  get_bad_frame_list( imgsum, fit='both',  plot=True,polyfit_order = 30,
                        scale= 3.5,  good_start = good_start, good_end=good_end, uid= uidstr, path=data_dir)

print( 'The bad frame list length is: %s'%len(bad_frame_list) )


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Get contrast
Can you write Python code for it?

if run_xsvs:
    contrast_factorL = get_contrast( ML_val)
    spec_km_pds = save_KM(  spec_kmean, KL_val, ML_val, qs=qr, level_time=times_xsvs, uid=uid_, path = data_dir )
    #spec_km_pds


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
GiSAXS Scattering Geometry
Can you write Python code for it?

if scat_geometry =='gi_saxs':
    plot_qzr_map(  qr_map, qz_map, inc_x0, ticks = ticks, data= avg_img, uid= uidstr, path = data_dir   )


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
GiWAXS Scattering Geometry
Can you write Python code for it?

if scat_geometry =='gi_waxs':
    #badpixel = np.where( avg_img[:600,:] >=300 )
    #roi_mask[badpixel] = 0
    show_ROI_on_image( avg_img, roi_mask, label_on = True,  alpha=.5,
                 save=True, path=data_dir, uid=uidstr, vmin=0.1, vmax=5)


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Load Metadata & Image Data
Change this line to give a uid
Can you write Python code for it?

username      =  getpass.getuser()
run_two_time  =   False
run_dose      =   False

uid = '7095d1b4' ##Fri Feb 24 12:38:04 2017 #17400: approx. 9kHz 30k (mbs:.1x.4) energy threshold: +30%C3: SiO2 500nm in water--  7095d1b4-53a4-4104-836e-be43c8d476ad

data_dir0  = create_user_folder(CYCLE, username)
print( data_dir0 )

uid = uid[:8]
print('The current uid for analysis is: %s...'%uid)

#get_last_uids( -1)

sud = get_sid_filenames(db[uid])
for pa in sud[2]:
    if 'master.h5' in pa:
        data_fullpath = pa
print ('scan_id, full-uid, data path are:  %s--%s--%s'%(sud[0], sud[1], data_fullpath ))

#start_time, stop_time = '2017-2-24  12:23:00', '2017-2-24  13:42:00'
#sids, uids, fuids  = find_uids(start_time, stop_time)

data_dir = os.path.join(data_dir0, '%s/'%(sud[1]))
os.makedirs(data_dir, exist_ok=True)
print('Results from this analysis will be stashed in the directory %s' % data_dir)
uidstr = 'uid=%s'%uid


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Load ROI defined by "XPCS_Setup" Pipeline
Define data analysis type
Can you write Python code for it?

if analysis_type_auto:#if True, will take "analysis type" option from data acquisition func series
    try:
        qphi_analysis_ = md['analysis'] #if True, will do q-phi (anisotropic analysis for transmission saxs)
        print(md['analysis'])
        if qphi_analysis_ == 'iso':
            qphi_analysis = False
        elif qphi_analysis_ == '':
            qphi_analysis = False
        else:
            qphi_analysis = True
    except:
        print('There is no analysis in metadata.')

print('Will %s qphis analysis.'%['NOT DO','DO'][qphi_analysis])

if scat_geometry != 'saxs':qphi_analysis = False;scat_geometry_ = scat_geometry
else:scat_geometry_ = ['','ang_'][qphi_analysis]+ scat_geometry
if scat_geometry != 'gi_saxs':run_profile_plot = False
print(scat_geometry_)

scat_geometry


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Load ROI mask depending on data analysis type
Can you write Python code for it?

##For SAXS
roi_path = '/XF11ID/analysis/2019_1/masks/'
roi_date = 'Feb6'

if scat_geometry =='saxs':
    if qphi_analysis ==  False:
        if isotropic_Q_mask == 'normal':
            #print('Here')
            q_mask_name='rings'
            if md['detector'] =='eiger4m_single_image' or md['detector'] == 'image': #for 4M
                fp = roi_path + 'roi_mask_%s_4M_norm.pkl'%roi_date

            elif md['detector'] =='eiger500K_single_image': #for 500K
                fp = roi_path + 'roi_mask_%s_500K_norm.pkl'%roi_date
                fp = '/home/yuzhang/XScattering/CameraTalk/Results/roi_mask_SiO2500nm_500K_SR16Rings.pkl'

        elif isotropic_Q_mask == 'wide':
            q_mask_name='wide_rings'
            if md['detector'] =='eiger4m_single_image' or md['detector'] == 'image': #for 4M
                fp = roi_path + 'roi_mask_%s_4M_wide.pkl'%roi_date
            elif md['detector'] =='eiger500K_single_image': #for 500K
                fp = roi_path + 'roi_mask_%s_500K_wide.pkl'%roi_date

    elif qphi_analysis:
        if phi_Q_mask =='phi_4x_20deg':
            q_mask_name='phi_4x_20deg'
            if md['detector'] =='eiger4m_single_image' or md['detector'] == 'image': #for 4M
                fp = roi_path + 'roi_mask_%s_4M_phi_4x_20deg.pkl'%roi_date
            elif md['detector'] =='eiger500K_single_image': #for 500K
                fp = roi_path + 'roi_mask_%s_500K_phi_4x_20deg.pkl'%roi_date

    #fp = 'XXXXXXX.pkl'
    roi_mask,qval_dict = cpk.load( open(fp, 'rb' )  )  #for load the saved roi data
    #print(fp)

## Gi_SAXS
elif scat_geometry =='gi_saxs':
    # dynamics mask
    fp = '/XF11ID/analysis/2018_2/masks/uid=460a2a3a_roi_mask.pkl'
    roi_mask,qval_dict = cpk.load( open(fp, 'rb' )  )  #for load the saved roi data
    print('The dynamic mask is: %s.'%fp)
    # static mask
    fp =...


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Make a directory for saving results
Can you write Python code for it?

CYCLE= '2019_1'  #change clycle here
path = '/XF11ID/analysis/%s/masks/'%CYCLE


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Number of pixels in each q box
Can you write Python code for it?

nopr = np.bincount(qind, minlength=(noqs+1))[1:]
nopr


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Overwrite Some Metadata if Wrong Input
Define incident beam center (also define reflection beam center for gisaxs)
Can you write Python code for it?

if scat_geometry =='gi_saxs':
    inc_x0 =  md['beam_center_x']
    inc_y0 =  imgs[0].shape[0] - md['beam_center_y']
    refl_x0 =     md['beam_center_x']
    refl_y0 =     1000     #imgs[0].shape[0] -  1758

    print( "inc_x0, inc_y0, ref_x0,ref_y0 are: %s %s %s %s."%(inc_x0, inc_y0, refl_x0, refl_y0) )
else:
    if md['detector'] =='eiger4m_single_image' or md['detector'] == 'image' or md['detector']=='eiger1m_single_image':
        inc_x0 =  imgs[0].shape[0] - md['beam_center_y']
        inc_y0=   md['beam_center_x']
    elif md['detector'] =='eiger500K_single_image':
        inc_y0 =  imgs[0].shape[1] - md['beam_center_y']
        inc_x0 =   imgs[0].shape[0] - md['beam_center_x']

    print(inc_x0, inc_y0)
    ###for this particular uid, manually give x0/y0
    #inc_x0 = 1041
    #inc_y0 = 1085

dpix, lambda_, Ldet,  exposuretime, timeperframe, center = check_lost_metadata(
    md, Nimg, inc_x0 = inc_x0, inc_y0=   inc_y0, pixelsize = 7.5*10*(-5) )
if scat_geometry =='gi_saxs':center=center[::-1]
setup_pargs=dict(uid=uidstr, dpix= dpix, Ldet=Ldet, lambda_= lambda_, exposuretime=exposuretime,
        timeperframe=timeperframe, center=center, path= data_dir)
print_dict( setup_pargs )

setup_pargs


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Plot fit results
Can you write Python code for it?

if run_xsvs:
    qr = [qval_dict[k][0] for k in list(qval_dict.keys()) ]
    plot_xsvs_fit(  spec_his, ML_val, KL_val, K_mean = spec_kmean, spec_std=spec_std,
                  xlim = [0,10], vlim =[.9, 1.1],
        uid=uid_, qth= qth_interest, logy= True, times= times_xsvs, q_ring_center=qr, path=data_dir)

    plot_xsvs_fit(  spec_his, ML_val, KL_val, K_mean = spec_kmean, spec_std = spec_std,
                  xlim = [0,15], vlim =[.9, 1.1],
        uid=uid_, qth= None, logy= True, times= times_xsvs, q_ring_center=qr, path=data_dir )


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Run Dose dependent analysis
Can you write Python code for it?

#run_dose = True

if run_dose:
    get_two_time_mulit_uids( [uid], roi_mask,  norm= norm,  bin_frame_number=1,
                        path= data_dir0, force_generate=False, compress_path =  cmp_path + '/' )

try:
    print( md['transmission'] )
except:
    md['transmission'] =1

exposuretime

if run_dose:
    N = len(imgs)
    print(N)
    #exposure_dose = md['transmission'] * exposuretime* np.int_([     N/16, N/8, N/4 ,N/2, 3*N/4, N*0.99 ])
    exposure_dose = md['transmission'] * exposuretime* np.int_([      N/8, N/4 ,N/2, 3*N/4, N*0.99 ])

    print( exposure_dose )

if run_dose:
    taus_uids, g2_uids = get_series_one_time_mulit_uids( [ uid ],  qval_dict, good_start=good_start,
                    path= data_dir0, exposure_dose = exposure_dose,  num_bufs =8, save_g2= False,
                                                   dead_time = 0, trans = [ md['transmission'] ] )

if run_dose:
    plot_dose_g2( taus_uids, g2_uids, ylim=[1.0, 1.2], vshift= 0.00,
                 qval_dict = qval_dict, fit_res= None,  geometry= scat_geometry_,
                 filename= '%s_dose_analysis'%uid_,
                path= data_dir, function= None,  ylabel='g2_Dose', g2_labels= None, append_name=  '' )

if run_dose:
    qth_interest = 1
    plot_dose_g2( taus_uids, g2_uids, qth_interest= qth_interest, ylim=[0.98, 1.2], vshift= 0.00,
                 qval_dict = qval_dict, fit_res= None,  geometry= scat_geometry_,
                 filename= '%s_dose_analysis'%uidstr,
                path= data_dir, function= None,  ylabel='g2_Dose', g2_labels= None, append_name=  '' )


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Save the OVA image
Can you write Python code for it?

if save_oavs:
    os.environ['HTTPS_PROXY'] = 'https://proxy:8888'
    os.environ['no_proxy'] = 'cs.nsls2.local,localhost,127.0.0.1'
    update_olog_uid_with_file( uid[:6], text='Add OVA images',
                              filename= data_dir + 'uid=%s_OVA_images.png'%uid, append_name='_img' )
       # except:


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Static Analysis
SAXS Scattering Geometry
Can you write Python code for it?

setup_pargs

if scat_geometry =='saxs':
    ## Get circular average| * Do plot and save q~iq
    mask = mask_copy.copy()
    hmask = create_hot_pixel_mask( avg_img, threshold = 1e8, center=center, center_radius= 10)

    qp_saxs, iq_saxs, q_saxs = get_circular_average( avg_img * Chip_Mask , mask * hmask, pargs=setup_pargs  )
    plot_circular_average( qp_saxs, iq_saxs, q_saxs,  pargs=setup_pargs,  show_pixel=True,
                      xlim=[qp_saxs.min(), qp_saxs.max()*1.0], ylim = [iq_saxs.min(), iq_saxs.max()*2] )
    mask =np.array( mask * hmask, dtype=bool)

if scat_geometry =='saxs':
    if run_fit_form:
        form_res = fit_form_factor( q_saxs,iq_saxs,  guess_values={'radius': 2500, 'sigma':0.05,
         'delta_rho':1E-10 },  fit_range=[0.0001, 0.015], fit_variables={'radius': T, 'sigma':T,
         'delta_rho':T},  res_pargs=setup_pargs, xlim=[0.0001, 0.015])

    qr = np.array( [qval_dict[k][0] for k in sorted( qval_dict.keys())] )
    if qphi_analysis ==  False:
        try:
            qr_cal, qr_wid = get_QrQw_From_RoiMask( roi_mask, setup_pargs  )
            print(len(qr))
            if (qr_cal - qr).sum() >=1e-3:
                print( 'The loaded ROI mask might not be applicable to this UID: %s.'%uid)
                print('Please check the loaded roi mask file.')
        except:
            print('Something is wrong with the roi-mask. Please check the loaded roi mask file.')

    show_ROI_on_image( avg_img*roi_mask, roi_mask, center, label_on = False, rwidth = 840, alpha=.9,
                 save=True, path=data_dir, uid=uidstr, vmin= 1e-3,
                 vmax= 1e-1, #np.max(avg_img),
                 aspect=1,
                 show_roi_edge=True,
                 show_ang_cor = True)
    plot_qIq_with_ROI( q_saxs, iq_saxs, np.unique(qr), logs=True, uid=uidstr,
                      xlim=[q_saxs.min(), q_saxs.max()*1.02],#[0.0001,0.08],
                  ylim = [iq_saxs.min(), iq_saxs.max()*1.02],  save=True,...


I am working on the file "2019_1/CameraTalk/XPCS_SiO2_500nm_For_CameraTalk.ipynb".
The first task is:
Static Analysis for gisaxs
Can you write Python code for it?

if scat_geometry =='gi_saxs':
    #roi_masks, qval_dicts = get_gisaxs_roi( Qrs, Qzs, qr_map, qz_map, mask= mask )
    show_qzr_roi( avg_img, roi_masks, inc_x0, ticks[:4], alpha=0.5, save=True, path=data_dir, uid=uidstr )

if  scat_geometry =='gi_saxs':
    Nimg = FD.end - FD.beg
    time_edge = create_time_slice( N= Nimg, slice_num= 3, slice_width= 2, edges = None )
    time_edge =  np.array( time_edge ) + good_start
    print( time_edge )
    qrt_pds = get_t_qrc( FD, time_edge, Qrs, Qzs, qr_map, qz_map, mask=mask, path=data_dir, uid = uidstr )
    plot_qrt_pds( qrt_pds, time_edge, qz_index = 0, uid = uidstr, path =  data_dir )


I am working on the file "2019_1/Template/XPCS_Single_2019_V2.ipynb".
The first task is:
Analysis for mass center of reflective beam center
Can you write Python code for it?

if run_get_mass_center:
    cx, cy = get_mass_center_one_roi(FD, roi_mask, roi_ind=25)

if run_get_mass_center:
    fig,ax=plt.subplots(2)
    plot1D( cx, m='o', c='b',ax=ax[0], legend='mass center-refl_X',
           ylim=[940, 960], ylabel='posX (pixel)')
    plot1D( cy, m='s', c='r',ax=ax[1], legend='mass center-refl_Y',
           ylim=[1540, 1544], xlabel='frames',ylabel='posY (pixel)')


I am working on the file "2019_1/Template/XPCS_Single_2019_V2.ipynb".
The first task is:
Apply Mask

load and plot mask if exist
otherwise create a mask using Mask pipeline
Reverse the mask in y-direction due to the coordination difference between python and Eiger software
Reverse images in y-direction
Apply the mask

Change the lines below to give mask filename
Can you write Python code for it?

if scat_geometry == 'gi_saxs':
    mask_path = '/XF11ID/analysis/2018_2/masks/'
    mask_name =  'July13_2018_4M.npy'

elif scat_geometry == 'saxs':
    mask_path = '/XF11ID/analysis/2019_1/masks/'
    if md['detector'] =='eiger4m_single_image' or md['detector'] == 'image':
        mask_name = 'Feb6_2019_4M_SAXS.npy'
    elif md['detector'] =='eiger500K_single_image':
        mask_name = 'Feb6_2019_500K_SAXS.npy'

elif scat_geometry == 'gi_waxs':
    mask_path = '/XF11ID/analysis/2018_2/masks/'
    mask_name = 'July20_2018_1M_WAXS.npy'

mask = load_mask(mask_path, mask_name, plot_ =  False, image_name = uidstr + '_mask', reverse= reverse, rot90=rot90  )
mask =  mask * pixel_mask * Chip_Mask
show_img(mask,image_name = uidstr + '_mask', save=True, path=data_dir, aspect=1, center=center[::-1])
mask_load=mask.copy()
imgsa = apply_mask( imgs, mask )


I am working on the file "2019_1/Template/XPCS_Single_2019_V2.ipynb".
The first task is:
Attach the PDF report to Olog
Can you write Python code for it?

#%run  /home/yuzhang/chxanalys_link/chxanalys/chx_olog.py

if att_pdf_report:
    os.environ['HTTPS_PROXY'] = 'https://proxy:8888'
    os.environ['no_proxy'] = 'cs.nsls2.local,localhost,127.0.0.1'
    update_olog_uid_with_file( uid[:6], text='Add XPCS Analysis PDF Report',
                              filename=pdf_out_dir + pdf_filename, append_name='_R1' )


I am working on the file "2019_1/Template/XPCS_Single_2019_V2.ipynb".
The first task is:
Check one ROI intensity
Can you write Python code for it?

roi_inten = check_ROI_intensity( avg_img, roi_mask, ring_number= 2, uid =uidstr ) #roi starting from 1