I am working on the file "ElCuadernillo/20160220_TensorFlowRegresionMultiple/tensorflow_regresion_multiple_gradiente_descendente.ipynb".
The first task is:
Problema
Calcular los coeficientes que mejor se ajusten a la muestra sabiendo que es de grado 5
Generamos la matriz de coeficientes de grado 5
Can you write Python code for it?

train_x=gdt.generar_matriz_coeficientes(x,grado) # MatrizA
train_y=np.reshape(y,(y.shape[0],-1)) # VectorColumna


I am working on the file "ElCuadernillo/20160220_TensorFlowRegresionMultiple/tensorflow_regresion_multiple_gradiente_descendente.ipynb".
The first task is:
Resultados:
Can you write Python code for it?


I am working on the file "ElCuadernillo/20160301_TensorFlowGradientDescentWithMomentum/GradientDescentWithMoment.ipynb".
The first task is:
<img src="capturas/gradient_descent.png">
Solución 2: Por medio mini-batch=1000  gradient descent
Can you write Python code for it?

pesos_mgd,ecm_mgd,tiempo_mgd=gdt.gradient_descent_with_momentum(train_x,
                                                             train_y,
                                                             num_mini_batch=10000,
                                                             learning_rate_inicial=learning_rate_inicial,
                                                             momentum=0.0)


I am working on the file "ElCuadernillo/20160301_TensorFlowGradientDescentWithMomentum/GradientDescentWithMoment.ipynb".
The first task is:
<img src="capturas/minibatch_gradient_descent_momentum.png">
Solución 4: Por medio mini-batch=1  Stocastict gradient descent With Moment
Can you write Python code for it?

pesos_sgdm,ecm_sgdm,tiempo_sgdm=gdt.gradient_descent_with_momentum(train_x,
                                                             train_y,
                                                             num_mini_batch=len(train_x),
                                                             learning_rate_inicial=learning_rate_inicial,
                                                             momentum=0.9)


I am working on the file "ElCuadernillo/20160301_TensorFlowGradientDescentWithMomentum/GradientDescentWithMoment.ipynb".
The first task is:
Input
Generamos la muestra de grado 4
Can you write Python code for it?

grado=4
tamano=100000
x,y,coeficentes=gdt.generar_muestra(grado,tamano)
print ("Coeficientes: ",coeficentes)
plt.plot(x,y,'.')


I am working on the file "ElCuadernillo/20160301_TensorFlowGradientDescentWithMomentum/GradientDescentWithMoment.ipynb".
The first task is:
Problema
Calcular los coeficientes que mejor se ajusten a la muestra sabiendo que es de grado 4
Generamos la matriz de coeficientes de grado 4
Can you write Python code for it?

train_x=gdt.generar_matriz_coeficientes(x,grado) # MatrizA
train_y=np.reshape(y,(y.shape[0],-1)) # VectorColumna
learning_rate_inicial=1e-2


I am working on the file "ElCuadernillo/20160301_TensorFlowGradientDescentWithMomentum/GradientDescentWithMoment.ipynb".
The first task is:
Solucion 1: Por medio gradient descent
Can you write Python code for it?

pesos_gd,ecm_gd,tiempo_gd=gdt.gradient_descent_with_momentum(train_x,
                                                             train_y,
                                                             num_mini_batch=1,
                                                             learning_rate_inicial=learning_rate_inicial,
                                                             momentum=0.0)


I am working on the file "ElCuadernillo/20160301_TensorFlowGradientDescentWithMomentum/GradientDescentWithMoment.ipynb".
The first task is:
TensorFlow, Mini-Batch/Stochastic GradientDescent With Moment
Can you write Python code for it?

import numpy as np
import tensorflow as tf
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
%matplotlib inline
import sys
import time

from IPython.display import Image
sys.path.append('/home/pedro/git/ElCuadernillo/ElCuadernillo/20160301_TensorFlowGradientDescentWithMomentum')

import gradient_descent_with_momentum as gdt


I am working on the file "ElCuadernillo/20160301_TensorFlowGradientDescentWithMomentum/GradientDescentWithMomentPlots.ipynb".
The first task is:
<img src="capturas/gradient_descent.png">
Solución 2: Por medio mini-batch=1000  gradient descent
Can you write Python code for it?

pesos_mgd,ecm_mgd,tiempo_mgd=gdt.gradient_descent_with_momentum(train_x,
                                                             train_y,
                                                             num_mini_batch=10000,
                                                             learning_rate_inicial=learning_rate_inicial,
                                                             momentum=0.0,
                                                             activar_sumario=activar_sumario,
                                                             prefijo='mGD_')


I am working on the file "ElCuadernillo/20160301_TensorFlowGradientDescentWithMomentum/GradientDescentWithMomentPlots.ipynb".
The first task is:
TensorFlow, Mini-Batch/Stochastic GradientDescent With Moment
Can you write Python code for it?


I am working on the file "ElCuadernillo/20160725_SistemasDeRecomendacionContentBased/Content-Based paso a paso.ipynb".
The first task is:
Comparativa de las mejoras realizadas

tf: Poderamos y normalizados los tags de cada articulo
tf-idx: Poderamos y normalizados los tags de cada articulo teniendo en cuenta la frecuencia de cada tag en los demas articulos
tf_idx_filtrado: Eliminamos las palabras mas comunes y los numeros a tf-idx
Can you write Python code for it?


I am working on the file "ElCuadernillo/20160725_SistemasDeRecomendacionContentBased/Content-Based paso a paso.ipynb".
The first task is:
Forma rapida
Todos los pasos anteriores se puede simplificar por medio de un metodo de sklearn que los agrutina
Can you write Python code for it?

tfidf_vectorizer = TfidfVectorizer( stop_words=cachedStopWords, token_pattern='(?u)\\b[a-zA-Z]\\w\\w+\\b')
tfidf_vectorizer.fit(datos.text)

CB.mostrar_pesos_tags(tfidf_vectorizer.transform(entrada_elton_john.text),tfidf_vectorizer)


I am working on the file "ElCuadernillo/20160725_SistemasDeRecomendacionContentBased/Content-Based paso a paso.ipynb".
The first task is:
Fuck, los tags mas importante no describen los articulos
- SOLUCION: Normalizar y poderar teniendo en cuenta la frecuencia de cada tag en los demas documenos
Can you write Python code for it?


I am working on the file "ElCuadernillo/20160725_SistemasDeRecomendacionContentBased/Content-Based paso a paso.ipynb".
The first task is:
Normalizacion
Can you write Python code for it?


I am working on the file "ElCuadernillo/20160725_SistemasDeRecomendacionContentBased/Content-Based paso a paso.ipynb".
The first task is:
Prediccion
Una vez que ya hemos extraidos los tags de los productos, buscamos por similitud los mas parecidos
Can you write Python code for it?

vecinos            = NearestNeighbors(n_neighbors=5,metric='cosine',algorithm='brute')
datos_por_tags     = tfidf_vectorizer.transform(datos.text)

vecinos.fit(datos_por_tags)


I am working on the file "ElCuadernillo/20160725_SistemasDeRecomendacionContentBased/Content-Based paso a paso.ipynb".
The first task is:
Recomendacion de productos, Content-Based
A continuación veremos paso a paso como se puede realizar un sistema de recomendacion basado en el contenido en python.
http://www.p.valienteverde.com/sistemas-de-recomendacion-basados-en-el-contenido-content-based/
Basado en el Contenido (ContendBased)
Por medio de la descripción del producto. Relaciones de tags

Vectorizacion del contenido, es decir, generar los tags
Ponderar los tags
Generar el motor de buscada de los productos similares

<img src='imagenes/contect_based.png'>
Can you write Python code for it?

import pandas as pd
import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from sklearn.neighbors import NearestNeighbors

import ContendBased as CB

datos = pd.read_csv("./BD/people_wiki.tar.gz", compression='gzip')
datos.head(3)

entrada_elton_john=datos.query('name == "Elton John"')
entrada_elton_john


I am working on the file "ElCuadernillo/20160725_SistemasDeRecomendacionContentBased/Content-Based paso a paso.ipynb".
The first task is:
Veamos que famosos nos relaciona con Al Pacino...
Can you write Python code for it?

al_pacino_vectorizado  = tfidf_vectorizer.transform(datos.query('name == "Al Pacino"').text)
distancia,indices       = vecinos.kneighbors(al_pacino_vectorizado)
datos.iloc[indices[0],:]


I am working on the file "ElCuadernillo/20160725_SistemasDeRecomendacionContentBased/Content-Based.ipynb".
The first task is:
Ejemplo de busqueda
Vamos a buscar los actores que han podido ganar algun oscar: https://en.wikipedia.org/wiki/Academy_Award_for_Best_Actor
Se puede comprobar que de los 5, 4 de ellos ha ganado por lo menos un oscar !!!!
Can you write Python code for it?


I am working on the file "ElCuadernillo/20160725_SistemasDeRecomendacionContentBased/Content-Based.ipynb".
The first task is:
Entrenamos el modelo
Can you write Python code for it?

content_based = CB.ContentBased()
content_based.fit(datos,columna_descripcion='text')


I am working on the file "ElCuadernillo/20160725_SistemasDeRecomendacionContentBased/Content-Based.ipynb".
The first task is:
Importamos los datos
Can you write Python code for it?

datos = pd.read_csv("BD/people_wiki.csv")
datos.head(3)


I am working on the file "ElCuadernillo/20160725_SistemasDeRecomendacionContentBased/Content-Based.ipynb".
The first task is:
Recomendacion de articulos basados en su contenido
Can you write Python code for it?

import pandas as pd
import sys
sys.path.append('./Scripts')
import ContendBased as CB
# Autorecarga los modulos
%load_ext autoreload
%autoreload 2


I am working on the file "ElDoradoKevin.ipynb".
The first task is:
Clustering
In order to decide which parcels we wanted to place a bid on, we used clustering to first continue the previous data exploration. The elbow method enabled us to retrieve a value of K = 3 to run the K-means algorithm. We decided to build clusters along 2 characteristics features: gold_available and elevation.
Can you write Python code for it?

# choosing K
sample_databis = sample.loc[:,["gold_available","elevation"] ]#, "Northing","gold_available","fixed_cost","variable_cost"]] ,"Northing","gold_available","fixed_cost","variable_cost"
hpc = sample_databis.values
k_range = range(1, 14)
k_means_var = [KMeans(n_clusters=k).fit(hpc) for k in k_range] # fit kmeans model for each n_clusters
centroids = [X.cluster_centers_ for X in k_means_var] # pull out centroids of cluster
k_euclid = [cdist(hpc, cent, 'euclidean') for cent in centroids] # euclidian distance from each point to each centroid
dist = [np.min(ke, axis=1) for ke in k_euclid]

wcss = [sum(d**2) for d in dist] # within-cluster sum of squares
tss = sum(pdist(hpc)**2)/hpc.shape[0] # total sum of squares

bss = tss - wcss # between-cluster sum of squares

# Create a trace
trace = go.Scatter(
    x = k_range,
    y = bss
)

# Edit the layout
layout3 = dict(title = 'Elbow method',
              xaxis = dict(title = 'K'),
              yaxis = dict(title = 'Beetween-cluster sum of squares'),
              )

# Plot and embed in ipython notebook!
fig3 = dict(data=[trace], layout=layout3)
py.iplot(fig3, filename='Elbow method')

K = 3
kmeans = KMeans(n_clusters=K).fit_predict(sample_databis)
# sample_data["cluster_id"] = kmeans
sample["cluster_id"] = kmeans

# statistics summary of the clustered data
sample.loc[:,["cluster_id","elevation","gold_available"]].groupby("cluster_id").describe()

# retrieving the cluster centers coordinates
centers = KMeans(n_clusters=K).fit(sample_databis).cluster_centers_
centers


I am working on the file "ElDoradoKevin.ipynb".
The first task is:
Data Collection
Can you write Python code for it?

sample_data = pd.read_csv("data/sample_data.csv")
costs_data = pd.read_csv("data/costs_data.csv")
auction_parcels = pd.read_csv("data/auction_parcels.csv")
elevation_data = pd.read_csv("data/elevation_data.csv")

sample_data[["gold_available"]].head()


I am working on the file "ElDoradoKevin.ipynb".
The first task is:
Graphical method
Given the provided data and the previous cluster analysis, we first plotted the auctions of the 3 clusters. Then, we foccused on the 3rd cluster category, i.e. with medium elevation values for reduced exploitation costs and high amount of gold (gold_amount > 20000 oz). We plotted the cluster 3 along with the available parcels for auction. That way, we were able to determine the parcels which were the most profitable to invest in.
In fact, our bidding strategy was to invest all our money into the 5 most profitable parcels according to our estimations.
Can you write Python code for it?

fig = plt.gcf()
fig.set_size_inches(18.5, 10.5, forward=True)
plt.axis([0.0,150.0, 0.0,150.0])
for i, col in zip(range(0,K),['green','orange','blue','red','black']):
      #plt.scatter(sample_data[sample_data["cluster_id"]==i].loc[:,"Easting"],sample_data[sample_data["cluster_id"]==i].loc[:,"Northing"],c=col,marker='o')
    plt.scatter(sample[sample["cluster_id"]==i].loc[:,"Easting"],sample[sample["cluster_id"]==i].loc[:,"Northing"],c=col,marker='o')

for i in range(0,K):
     plt.scatter(centers[i][0],centers[i][1],s=300,c='black',marker='*')

# aument costs _data
regr = linear_model.LinearRegression()
X_train  = costs_data.loc[:10, ["gold_amount"]]
X_test =  costs_data.loc[11:, ["gold_amount"]]
for elev in ["low", "med", "high"]:
    y_train  = costs_data.loc[:10,elev]
    regr.fit(X_train,y_train)
    pred = regr.predict(X_test)
    costs_data.loc[11:,elev] = pred


I am working on the file "ElDoradoKevin.ipynb".
The first task is:
One strategy would have to construct the minerals and ores as they are not present in the auction data, by using clustering on those features for example, assigning each parcel to a cluster and completing the auction parcel dataframe with the average value for each feature of each cluster.
For simplicity we chose to only use the rocks categories features. Therefore we trained our classification models by extracting the previous features and normized those continuous values.
Can you write Python code for it?

features = auction_parcels.columns.tolist()
features.append("gold_available")
features.pop(1)
features.pop(1)
features.pop(0)
sample2 = sample_data.merge(elevation, on="parcel_id")
sample_set = sample2.loc[:,features]
sample_set.loc[:,["elevation","Arkonor","Mercoxit","Bistot","Crokite"]] = normalize(sample_set.loc[:,["elevation","Arkonor","Mercoxit","Bistot","Crokite"]], axis =0)

def label_gold_amount(row, threshold):
    if row["gold_available"]  < threshold:
        return 0
    elif row["gold_available"]  >= threshold:
        return 1
sample_set["gold_available"] = sample_set.apply(lambda row: label_gold_amount(row, 10000), axis = 1)

def label_gold_amount(row):
    if row["gold_available"]> 0 and row["gold_available"]  <= 10000:
        return 0
    elif row["gold_available"]> 10000 and row["gold_available"]  <= 20000:
        return 1
    else:
        return 2
sample_set["gold_available"] = sample_set.apply(lambda row: label_gold_amount(row), axis = 1)

#sample_set
X_train, X_test, y_train, y_test = train_test_split(sample_set.iloc[:,0:5],sample_set.iloc[:,5],test_size=0.05)

logreg = LogisticRegression(multi_class='ovr') #
param_range = [0.001, 0.01, 0.1, 1, 10, 100, 1000]
gs = GridSearchCV(estimator=logreg,param_grid= {'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000], 'penalty': ['l1','l2'] },scoring='accuracy',cv=5,n_jobs=-1)

gs.fit(X_train,y_train)

gs.best_score_

gs.best_params_

print('Test accuracy: %.2f %%' % (gs.score(X_test, y_test)*100))

sum(y_train == 2)

# X_test
auction_parcels_processedtmp = auction_parcels
auction_parcels_processedtmp.loc[:,["elevation","Arkonor","Mercoxit","Bistot","Crokite"]] = normalize(auction_parcels.loc[:,["elevation","Arkonor","Mercoxit","Bistot","Crokite"]], axis =0)
auction_parcels_processed = auction_parcels_processedtmp.loc[:,["parcel_id","elevation","Arkonor","Mercoxit","Bistot","Crokite"]]
auction_parcels_processed.head()

gs.predict(auction_parcels_processed.iloc[:,1:])

svc = SVC(random_state=1)
param_range2 = [0.0001, 0.001, 0.01, 0.1, 1.0,...


I am working on the file "ElDoradoKevin.ipynb".
The first task is:
This graphical analysis resulted in the following list of most interesting parcels to invest in:
firstfilter = [10226, 7837, 19114, 20194, 11489,10905,1790,13249,14154,12810,1614,12221]
Let's predict whether a parcel contain a high amount of gold - WIP maybe
Can you write Python code for it?

from sklearn.preprocessing import normalize
from sklearn import metrics
from sklearn.metrics import accuracy_score
from sklearn.cross_validation import cross_val_score
from sklearn.cross_validation import train_test_split
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.feature_extraction.text import TfidfTransformer
from sklearn.naive_bayes import MultinomialNB
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
from sklearn.grid_search import GridSearchCV
from sklearn.svm import SVC
from sklearn.svm import LinearSVC


I am working on the file "ElDoradoKevin.ipynb".
The first task is:
We visualised the frequency of parcels w.r.t. the amount of gold available and the elevation levels in order to get better domain knowledge of the characteristics of gold mine.
Can you write Python code for it?

# gplot = sample.plot(y="gold_available",kind='hist',title="Gold Distribution")
# gplot1 = sample.plot(y="elevation",kind='hist',title="Elevation of gold mines")
# gplot.set_xlabel("Amount of gold")
# gplot1.set_xlabel("Elevation")

data1 = [
    go.Histogram(
        x=sample['gold_available'],
        autobinx=False,
        xbins=dict(
        start=0,
        end=30000,
        size=3000
    )
    )
]

layout1 = go.Layout(
    title='Gold Distribution',
    xaxis=dict(
        title='gold_available'
    ),
    yaxis=dict(
        title='Frequency'
    ),
    #barmode='overlay',
    bargap=0.05,
    #bargroupgap=0.3
)

fig1 = go.Figure(data=data1, layout=layout1)
py.iplot(fig1)

data2 = [
    go.Histogram(
        x=sample['elevation'],
         autobinx=False,
        xbins=dict(
        start=-1600,
        end=1600,
        size=300
    )
    )
]

layout2 = go.Layout(
    title='Elevation of gold mines',
    xaxis=dict(
        title='Elevation'
    ),
    yaxis=dict(
        title='Frequency'
    ),
    #barmode='overlay',
    bargap=0.05,
    #bargroupgap=0.3
)

fig2 = go.Figure(data=data2, layout=layout2)
py.iplot(fig2)


I am working on the file "ElDoradoMihai.ipynb".
The first task is:
- Elevation Data
Can you write Python code for it?

elevation_data = pd.read_csv("data/elevation_data.csv")
elevation_data.head()


I am working on the file "ElDoradoMihai.ipynb".
The first task is:
- Sample Data

Parcels that have already been auctioned and exploited (see the gold_available column)
Why are the Northing and Eastig shifted by .5?
Can you write Python code for it?

sample_data = pd.read_csv("data/sample_data.csv")
sample_data.head()


I am working on the file "ElDoradoMihai.ipynb".
The first task is:
Data Loading
- Costs Data
Can you write Python code for it?

costs_data = pd.read_csv("data/costs_data.csv")
costs_data.head()


I am working on the file "ElDoradoMihai.ipynb".
The first task is:
Radius = 3 seems to fit the bill
Compute the total extraction costs and estimated profit
Can you write Python code for it?

total_costs = gold_estimations.loc[:, ['parcel_id', 'estimated_gold_r3']]

total_costs = total_costs.merge(elevation_data.loc[:, ['parcel_id', 'elevation']], on='parcel_id')

total_costs['total_cost'] = total_costs.apply(lambda row: fixed_cost(row), axis = 1) + total_costs.apply(lambda row: variable_cost(row), axis = 1)

total_costs['estimated_profit'] = total_costs.apply(lambda row: gold_price * row['estimated_gold_r3'] - row['total_cost'], axis = 1)

total_costs.sort_values(by=['estimated_profit'])


I am working on the file "ElDoradoMihai.ipynb".
The first task is:
Remove the total costs from the available budget
Can you write Python code for it?

total_cost = selected_total_costs.total_cost.sum()

bid_money = total_budget - total_cost

bid_money


I am working on the file "ElDoradoMihai.ipynb".
The first task is:
Select top 5 most promising parcels that also match Kevin's predictions
Can you write Python code for it?

# Parcel IDs from Kevin: [7837, 19114, 20194, 11489,10905,1790,13249,14154,12810,11614,12221]

selected_parcels = [19114, 20194, 11489, 11614, 12810]

selected_total_costs = total_costs.loc[total_costs.parcel_id.isin(selected_parcels), :].sort_values(by=['estimated_profit'])

selected_total_costs


I am working on the file "ElDoradoMihai.ipynb".
The first task is:
Variable costs calculator

Use a simple linear regression
Can you write Python code for it?

from sklearn import linear_model

regressions = {}
for elevation in ["low", "med", "high"]:
    regression = linear_model.LinearRegression()
    regression.fit(costs_data.loc[:, ["gold_amount"]], costs_data.loc[:,elevation])
    regressions[elevation] = regression

def variable_cost(row):
    if row["elevation"] < 0:
        return regressions['low'].predict(row["elevation"])[0]
    elif row["elevation"] >= 0 and row["elevation"] <= 700:
        return regressions['med'].predict(row["elevation"])[0]
    else:
        return regressions['high'].predict(row["elevation"])[0]


I am working on the file "ElasticSearch_PySpark_Integration.ipynb".
The first task is:
ES returns key-value RDD where key is ID of the document, and value is content of _source field :  [("_id", {}), ...]
Can you write Python code for it?

print esrdd.getNumPartitions()
esrdd = esrdd.coalesce(4)
print esrdd.getNumPartitions()

esrdd.persist()
print esrdd.is_cached
print esrdd.getStorageLevel()

rdd1 = esrdd.sample(False, 0.01).mapValues( lambda x: {"src": x["netflow"]["ipv4_src_addr"],
                                                                    "dst": x["netflow"]["ipv4_dst_addr"]} )

rdd1.take(4)


I am working on the file "ElasticSearch_PySpark_Integration.ipynb".
The first task is:
Every JSON document  represents a connection between a  source IP and  destination IP.
Can you write Python code for it?


I am working on the file "ElasticSearch_PySpark_Integration.ipynb".
The first task is:
Gather statistics  on our dataset
Can you write Python code for it?

src_stats = rdd_joined.map(lambda x: x[1][0]).stats() # outgoing(src)
print src_stats

dst_stats = rdd_joined.map(lambda x: x[1][1]).stats() # incoming(dst)
print dst_stats

src_stats_bc = sc.broadcast(src_stats)
dst_stats_bc = sc.broadcast(dst_stats)

src_stats_bc.value


I am working on the file "ElasticSearch_PySpark_Integration.ipynb".
The first task is:
Goal:  Identify  3-sigma  deviation connections
Construct  ES Query
Query DSL (Here, we select connections between machines that operate within a restricted environment)
Can you write Python code for it?

def getESQuery():
    query = """{
        "query": {
                "filtered": {
                     "query": {
                         "match_all": {}
                      },
                      "filter": {
                          "bool": {
                             "must": [
                                 {
                                    "range": {
                                        "@timestamp" : {
                                           "gte" : "now-1h",
                                           "lt" : "now"
                                        }
                                    }
                                 }
                              ],
                              "should": [
                                 { "term": { "src_addr_space": "private" } },
                                 { "term": { "dst_addr_space": "private" } }
                              ]
                          }
                    }
                }
           },
        "_source": [ "netflow.ipv4_src_addr", "netflow.ipv4_dst_addr" ]
        }"""
    return query


I am working on the file "ElasticSearch_PySpark_Integration.ipynb".
The first task is:
Identify  IPs that  have  unusual  number of  outgoing or incoming connections
Can you write Python code for it?

rdd_joined.filter(lambda x: x[1][0] > 0).filter( lambda x: (x[1][0] - src_stats_bc.value.mean()) > (3*src_stats_bc.value.stdev()) ).collect()

rdd_joined.filter(lambda x: x[1][1] > 0).filter( lambda x: (x[1][1] - dst_stats_bc.value.mean()) > (3*dst_stats_bc.value.stdev()) ).collect()


I am working on the file "ElasticSearch_PySpark_Integration.ipynb".
The first task is:
List all the  unique ip addresses
Can you write Python code for it?

print rdd1.flatMap( lambda x: [ x[1]["src"], x[1]["dst"] ] ).distinct().count()
print rdd1.flatMap( lambda x: [ x[1]["src"], x[1]["dst"] ] ).distinct().collect()


I am working on the file "ElasticSearch_PySpark_Integration.ipynb".
The first task is:
Replace missing values with zeroes
Can you write Python code for it?

rdd_joined = rdd_joined.mapValues(lambda x: tuple([y if y is not None else 0 for y in x]))
rdd_joined.collect()


I am working on the file "Elecciones 2016.ipynb".
The first task is:
A continuación represento la evolución temporal del voto por cada partido político, incluyendo las incertidumbres estimadas para cada sondeo.
Can you write Python code for it?

labelsPartidos = ['PP', 'PSOE', 'PODEMOS+IU', 'Cs', 'Otros']
f, ax = pl.subplots(nrows=2, ncols=3, figsize=(10,8))
ax = ax.flatten()
for i in range(5):
    ax[i].plot(weekAll, sondeosAll[:,i], '.', color=sns.xkcd_rgb[colors[i]], linewidth=2)
    ax[i].set_ylim([0,0.35])
    ax[i].errorbar(weekAll, sondeosAll[:,i], fmt='none', yerr=sigmaAll, color=sns.xkcd_rgb[colors[i]], linewidth=2, ecolor=sns.xkcd_rgb[colors[i]])
    ax[i].plot(np.arange(max(weekAll))+1, theta[0,:,i], color=sns.xkcd_rgb[colors[i]], linewidth=2)
    ax[i].fill_between(np.arange(max(weekAll))+1, theta[1,:,i], theta[2,:,i], color=sns.xkcd_rgb[colors[i]], alpha=0.3)
    ax[i].set_xlabel('Semana')
    ax[i].set_xlabel('Porcentaje de votos')
    ax[i].set_title(labelsPartidos[i])
pl.tight_layout()
pl.savefig("porPartido.png")


I am working on the file "Elecciones 2016.ipynb".
The first task is:
Análisis de los sondeos electorales 2016
Vamos a plantearnos como ejercicio analizar los sondeos electorales. Para ello cogemos como datos los sondeos recopilados en la página web de la Wikipedia para las elecciones de 2015 y para las de 2016. Consideramos sólo los partidos PP, PSOE, IU, Podemos y Ciudadanos. El resto de partidos se consideran metidos dentro de un mismo saco, que vienen siendo alrededor del 15% en cada sondeo.
Para modelar la evolución temporal de los sondeos para cada partido, usamos un modelado con variables latentes. La idea es que los resultados de los sondeos son sólo una estimación ruidosa y sesgada de la intención de voto real de los votantes. Por tanto, considamos que $\theta_i(t_j)$ es la subyacente intención de voto para el partido $i$ en el tiempo $t_j$. Esta variable se observa de forma ruidosa para cada empresa $k$ que realiza el sondeo, y además incluye un cierto sesgo. Por tanto, nuestro modelo de observación es:
$$
y_{ik}(t_j) \sim N(\theta_i(t_j) + b_{ik}, \sigma_{k}(t_j)^2)
$$
donde $b_{ik}$ es el bias que introduce la empresa de sondeos $k$ sobre el partido $i$ y $\sigma_{ik}^2$ es la varianza del sondeo realizado en el tiempo $t_j$ por la empresa $k$. Para los resultados de las elecciones, elegimos una varianza muy pequeña y no incluimos los sesgos, porque es el único momento en el que tenemos acceso directo a las intenciones subyacentes:
$$
y_{ik}(\mathrm{elecciones}) \sim N(\theta_i(t_j), \sigma_{k}(\mathrm{elecciones})^2).
$$
Es obvio que la suma de porcentajes de cada partido debe sumar 1 en cada sondeo, por lo cual debemos asegurarnos de que
$$
\sum_i \theta_i(t_j) = 1.
$$
Además, es lógico asumir que, dadas suficientes casas de sondeo, los biases sumen 0 para cada partido:
$$
\sum_k b_{ik} = 0.
$$
Por último, asumimos un proceso autoregresivo Markoviano para modelar la variación temporal de la intención de voto real para cada partido. Es decir, en cada sondeo, la intención de voto estará dada por una variable Gaussiana con media igual a la intención de voto anterior,...
Can you write Python code for it?

def stan_cache(model_code, model_name=None, **kwargs):
    """Use just as you would `stan`"""
    code_hash = md5(model_code.encode('ascii')).hexdigest()
    if model_name is None:
        cache_fn = 'cached-model-{}.pkl'.format(code_hash)
    else:
        cache_fn = 'cached-{}-{}.pkl'.format(model_name, code_hash)
    try:
        sm = pickle.load(open(cache_fn, 'rb'))
    except:
        sm = pystan.StanModel(model_code=model_code)
        with open(cache_fn, 'wb') as f:
            pickle.dump(sm, f)
    else:
        print("Using cached StanModel")
    return sm.sampling(**kwargs)

def toenglish(s):
    spanish = ['ene', 'abr', 'ago', 'dic', 'de mayo de']
    english = ['jan', 'apr', 'aug', 'dec', 'may']
    for (j, month) in enumerate(spanish):
        s = s.replace(month, english[j])
    return s

def getPercentage(s):
    if (s[0] not in ['0','1','2','3','4','5','6','7','8','9']):
        return 0
    else:
        if (s.find('%') != -1):
            return float(s.split('%')[0].replace(',','.')) / 100.0
        else:
            return float(s.split('\n')[0].replace(',','.')) / 100.0

def getSigma(s):
    left = s.find('(')
    right = s.find(')')
    if (s[left+1:right] in ['?', '-']):
        return 0.03
    else:
        return 1.0 / np.sqrt(float(s[left+1:right]))

def weeksDifference(d1, d2):
    monday1 = (d1 - datetime.timedelta(days=d1.weekday()))
    monday2 = (d2 - datetime.timedelta(days=d2.weekday()))

    return int((monday2 - monday1).days / 7)


I am working on the file "Elecciones 2016.ipynb".
The first task is:
Estimación de voto actual
A continuación pongo la estimación de voto actual y sus intervalos de confianza.
Can you write Python code for it?

f, ax = pl.subplots()
sns.boxplot(data=thetaChain[:,-1,:], ax=ax)
ax.set_xticks(np.arange(5))
ax.set_xticklabels(labelsPartidos, rotation=90)
boxes = ax.artists
for (i, box) in enumerate(boxes):
    box.set_facecolor(sns.xkcd_rgb[colors[i]])
pl.savefig("estimacionActual.png")


I am working on the file "Elecciones 2016.ipynb".
The first task is:
Leemos el fichero de datos obtenido de la Wikipedia y hacemos el parsing. Escogemos sólo aquellos sondeos en los que hay información para todos los partidos. Podríamos también escoger aquellos parciales porque el modelado Bayesiano los podría aceptar.
Can you write Python code for it?

wb = openpyxl.load_workbook("data/sondeos.xlsx")
ws = wb.active

empresas = ['GAD3', 'Encuestamos', 'GESOP', 'Metroscopia', 'Celeste-Tel','Demoscopia Servicios', 'Simple Lógica', 'CIS',
            'TNS Demoscopia', 'Invymark', 'NC Report', 'El Espanol', 'DYM', 'Sondaxe', 'Sigma Dos', 'Resultados de las elecciones']

empresaSondeoAll = []
sondeosAll = []
dateAll = []
sigmaAll = []

wbnew = openpyxl.Workbook()

# grab the active worksheet
wsnew = wbnew.active
wsnew.append(['Encuestador','Fecha','Muestra','PP','PSOE','PODEMOS','Cs','IU'])

otrosPartidos = ['F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'R']

nSondeos = 302
loopSondeos = 0
for i in range(130):
    empresa = ws['A{0}'.format(i+2)].value
    for (loop, emp) in enumerate(empresas):
        if (empresa.find(emp) != -1):
            empresaSondeo = loop

    if (empresaSondeo == len(empresas)-1):
        sigma = 0.0001
    else:
        sigma = getSigma(empresa)

    PP = getPercentage(ws['C{0}'.format(i+2)].value)
    PSOE = getPercentage(ws['D{0}'.format(i+2)].value)
    IU = getPercentage(ws['E{0}'.format(i+2)].value)
    PODEMOS = getPercentage(ws['P{0}'.format(i+2)].value)
    CS = getPercentage(ws['Q{0}'.format(i+2)].value)

    total = PP + PSOE + IU + PODEMOS + CS
    otros = 1.0 - total

    tmp = ws['B{0}'.format(i+2)].value
    if (isinstance(tmp, datetime.date)):
        date = tmp
    else:
        date = dateutil.parser.parse(toenglish(ws['B{0}'.format(i+2)].value.split('-')[-1].lower()))

    tmp = date.year + (date.month-1.0) / 12.0

    sondeo = [PP, PSOE, IU+PODEMOS, CS, otros]

    if (0 not in sondeo):

        loopSondeos += 1

        sondeosAll.append(sondeo)
        sigmaAll.append(sigma)

        dateAll.append(date)
        empresaSondeoAll.append(empresaSondeo+1)

        print ("{0} - {1} {7} - s={8:4.2f} : PP={2:4.2f} - PSOE={3:4.2f} - UP={4:4.2f} - CS={5:4.2f} - Resto={6:4.2f}".format(i,
            empresas[empresaSondeo], PP*100, PSOE*100, (IU+PODEMOS)*100, CS*100, otros*100, date,...


I am working on the file "Elecciones 2016.ipynb".
The first task is:
Leemos el modelo que hemos definido y hacemos el sampling.
Can you write Python code for it?

date.year-2000

f = open('model.stan', 'r')
model = f.read()
f.close()

out = stan_cache(model, model_name='elecciones', data=dictionary, chains=1)


I am working on the file "Elecciones 2016.ipynb".
The first task is:
Resultados
Extraemos algunas de las cantidades interesantes para hacer algunas representaciones.
Can you write Python code for it?

thetaChain = out.extract('theta')['theta']
theta = np.percentile(thetaChain, [50.0, 50.0-68/2.0, 50.0+68/2], axis=0)
houseChain = out.extract('house')['house']
house = np.percentile(houseChain, [50.0, 50.0-68/2.0, 50.0+68/2], axis=0)
sigmaChain = out.extract('sigma')['sigma']
sigma = np.percentile(sigmaChain, [50.0, 50.0-68/2.0, 50.0+68/2], axis=0)


I am working on the file "Elecciones 2016.ipynb".
The first task is:
Sesgos por casa encuestadora
Represento a continuación las estimaciones de sesgo para cada una de las empresas encuestadoras, lo que muestra algún efecto curioso en Demoscopia Servicios que sería interesante analizar y entender por qué pasa. Nótese que un sesgo positivo implica que esa empresa subestima los votos y hace falta corregirlo por esa cantidad para obtener las observaciones.
Can you write Python code for it?

f, ax = pl.subplots(ncols=3, nrows=2, figsize=(12,10))
ax = ax.flatten()
for i in range(5):
    sns.boxplot(data=houseChain[:,:,i], ax=ax[i])
    ax[i].set_xticks(np.arange(12))
    ax[i].set_xticklabels(labelsEmpresas, rotation=90)
    ax[i].set_title(labelsPartidos[i])
pl.tight_layout()
pl.savefig("bias.png")


I am working on the file "Elecciones 2016.ipynb".
The first task is:
Variabilidad
Por último, aquí muestro la variabilidad intrínseca de la estimación subyacente de voto para cada partido. Es lógico que Podemos y Cs tengan una variabilidad mayor por su largo crecimiento en el último año.
Can you write Python code for it?

f, ax = pl.subplots(figsize=(10,6))
sns.boxplot(data=sigmaChain, ax=ax)
ax.set_xticks(np.arange(5))
ax.set_xticklabels(labelsPartidos, rotation=90)
boxes = ax.artists
for (i, box) in enumerate(boxes):
    box.set_facecolor(sns.xkcd_rgb[colors[i]])
pl.savefig("variabilidad.png", facecolor=f.get_facecolor(), edgecolor='none')


I am working on the file "Election 2016.ipynb".
The first task is:
The results of the k-Means algorithm are plotted below. The code for plotting is taken from the sklearn documentation.
Can you write Python code for it?

h = .02
x_min, x_max = trump_norm[:, 0].min() - 1, trump_norm[:, 0].max() + 0.5
y_min, y_max = trump_norm[:, 1].min() - 1, trump_norm[:, 1].max() + 0.5
xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
# Obtain labels for each point in mesh. Use last trained model.
Z = kmeans.predict(np.c_[xx.ravel(), yy.ravel()])
# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.figure(1)
plt.clf()
plt.imshow(Z, interpolation='nearest',
           extent=(xx.min(), xx.max(), yy.min(), yy.max()),
           cmap=plt.cm.Paired,
           aspect='auto', origin='lower')
plt.plot(trump_norm[:, 0], trump_norm[:, 1], 'k.', markersize=2)
# Plot the centroids as a white X
centroids = kmeans.cluster_centers_
plt.scatter(centroids[:, 0], centroids[:, 1],
            marker='x', s=100, linewidths=3,
            color='w', zorder=10)
plt.title('K-means Clustering on Primary Results')
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)
plt.show()


I am working on the file "Election 2016.ipynb".
The first task is:
With the data normalized the k-Means algorithm can be run on it. In order to find the optimal number of clusters the silhouette score was calculated for different numbers.
Can you write Python code for it?

best_nc = 0
best_ss = 0
for n_clusters in range(2,10):
    clusterer = KMeans(n_clusters=n_clusters, random_state=10)
    cluster_labels = clusterer.fit_predict(trump_norm)
    silhouette_avg = silhouette_score(trump_norm, cluster_labels)
    print("For", n_clusters,"clusters, the average silhouette score is", silhouette_avg)
    if silhouette_avg > best_ss:
        best_nc = n_clusters
        best_ss = silhouette_avg
print("The best number of clusters is",best_nc)


I am working on the file "Election 2016.ipynb".
The first task is:
With the optimal number of clusters, the most accurate model can be created.
Can you write Python code for it?

kmeans = KMeans(n_clusters=best_nc, random_state=10)
kmeans.fit(trump_norm)


I am working on the file "ElectroOptics/.ipynb_checkpoints/CurveFitAttempt-checkpoint.ipynb".
The first task is:
Data
Can you write Python code for it?


I am working on the file "ElectroOptics/.ipynb_checkpoints/CurveFitAttempt-checkpoint.ipynb".
The first task is:
Second, Calculate the Tilt Angle $\psi$
$$ numerator() \:returns: {sin\:{2\theta}\:cos\:{\phi}}\:e^{\frac{-U}{k_bT}}\:sin\:{\theta} $$
Can you write Python code for it?

def numerator(theta,phi,T,p0k,alpha,E):
    boltz = Boltz(theta,phi,T,p0k,alpha,E)
    return np.sin(2*theta)*np.cos(phi)*boltz


I am working on the file "ElectroOptics/.ipynb_checkpoints/CurveFitAttempt-checkpoint.ipynb".
The first task is:
This notebook calculates and plots the theoretical tilt angles. It will also plot the alpha and p0 factors vs temperature that are given in the cell below this.
Material Characteristics
Can you write Python code for it?

thetamin = 25.6*np.pi/180
thetamax = 33.7*np.pi/180
t = 1*10**-6 #Cell Thickness


I am working on the file "ElectroOptics/CurveFitAttempt.ipynb".
The first task is:
$$ tan(2\psi) =  \frac{\int_{\theta_{min}}^{\theta_{max}} \int_0^{2\pi} {sin\:{2\theta}\:cos\:{\phi}}\:e^{\frac{-U}{k_bT}}\:sin\:{\theta}\: d\theta d\phi}{\int_{\theta_{min}}^{\theta_{max}} \int_0^{2\pi} ({{cos}^2{\theta} - {sin}^2{\theta}\:{cos}^2{\phi}})\:e^{\frac{-U}{k_bT}}\:sin\:{\theta}\: d\theta d\phi} $$
Can you write Python code for it?

def compute_psi(E,p0k,alpha):

    def Boltz(theta,phi,T,p0k,alpha,E):
        """Compute the integrand for the Boltzmann factor.
        Returns
        -------
        A function of theta,phi,T,p0k,alpha,E to be used within dblquad
        """
        return np.exp((1/T)*p0k*E*np.sin(theta)*np.cos(phi)*(1+alpha*E*np.cos(phi)))*np.sin(theta)

    def numerator(theta,phi,T,p0k,alpha,E):
        boltz = Boltz(theta,phi,T,p0k,alpha,E)
        return np.sin(2*theta)*np.cos(phi)*boltz

    def denominator(theta,phi,T,p0k,alpha,E):
        boltz = Boltz(theta,phi,T,p0k,alpha,E)
        return ((np.cos(theta)**2) - ((np.sin(theta)**2) * (np.cos(phi)**2)))*boltz
    """Computes the tilt angle(psi) by use of our tan(2psi) equation
    Returns
    -------
    Float:
    The statistical tilt angle with conditions T,p0k,alpha,E
    """

    avg_numerator, avg_numerator_error = dblquad(numerator, 0, 2*np.pi, lambda theta: thetamin, lambda theta: thetamax,args=(T,p0k,alpha,E))

    avg_denominator, avg_denominator_error = dblquad(denominator, 0, 2*np.pi, lambda theta: thetamin, lambda theta: thetamax,args=(T,p0k,alpha,E))

    psi = np.arctan(avg_numerator / (avg_denominator)) * (180 /(2*np.pi)) #Converting to degrees from radians and divide by two

    return psi

compute_psi(tdata[0],p0k_array[0]*1e7,alpha_array[0]*1e10)

PSIdata[0]

guess = [p0k_array[0]*1e7,alpha_array[0]*1e10]

guess

theta_best, theta_cov = opt.curve_fit(compute_psi, Edata, PSIdata,guess,absolute_sigma=True)


I am working on the file "ElectroOptics/CurveFitAttempt.ipynb".
The first task is:
Langevin-Debye Model
$$ U(\theta,\phi) = -\rho_0E\:sin\:\theta\:cos\:\phi\:(1+\alpha E\:cos\:\phi) $$
First, Calculate the Boltzmann Factor and the Partition Function
$$ {Boltz() returns:}\:\: e^{\frac{-U}{k_bT}}\:sin\:{\theta}\ $$
Can you write Python code for it?

def Boltz(theta,phi,T,p0k,alpha,E):
    """Compute the integrand for the Boltzmann factor.
    Returns
    -------
    A function of theta,phi,T,p0k,alpha,E to be used within dblquad
    """
    return np.exp((1/T)*p0k*E*np.sin(theta)*np.cos(phi)*(1+alpha*E*np.cos(phi)))*np.sin(theta)


I am working on the file "ElectroOptics/CurveFitAttempt.ipynb".
The first task is:
This notebook calculates and plots the theoretical tilt angles. It will also plot the alpha and p0 factors vs temperature that are given in the cell below this.
Material Characteristics
Can you write Python code for it?


I am working on the file "ElectroOptics/Main.ipynb".
The first task is:
1. Import Excel Data
Can you write Python code for it?

thetamin = 17.8 * (np.pi/180)
thetamax = 33.4 * (np.pi/180)

df = pd.read_excel('ShenData.xlsx', sheetname=None)

del df['Intro']

for key in df:
    del df[key]['Vp']
    del df[key]['θ1']
    del df[key]['θ2']
    del df[key]['Extinction +']
    del df[key]['Extinction -']
    del df[key]['Retardation']
    del df[key]['Birefringence']

df['40C']

Fields = np.array([df[key]['Field (V/μm)'] for key in df])
Angles = np.array([df[key]['Tilt Angle'] for key in df])
TempsC = np.array([df[key]['Temperature (Celsius)'][0] for key in df])
TempsK = np.array([entry + 273.15 for entry in TempsC])

plt.figure(figsize=(8,6));
colors = np.array(['red','green','blue','orange','brown','pink', 'purple','black','cyan','magenta','yellow','gray'])
for i in range(len(Fields)):
    plt.scatter(Fields[i],Angles[i],c=colors[i],label= str(int(TempsC[i])) + '$^\circ$C',marker = 'o',s=50, edgecolor = colors[i]);
plt.xlabel('Field (V/$\mu$m)',size=15);
plt.ylabel('Tilt Angle',size=15);
plt.title('Tilt Angle for Shen Data')
plt.xlim(0,70);
plt.ylim(0,40);
plt.legend(loc=4);


I am working on the file "ElectroOptics/Main.ipynb".
The first task is:
2. Curve Fitting
Langevin-Debye Model
$$ U(\theta, \phi, E, \rho_0) = -\rho_0E\:sin\:\theta\:cos\:\phi\:(1+\alpha E\:cos\:\phi) $$
$$ tan(2\psi) =  \frac{\langle sin\:2\theta\:cos\:{\phi}\rangle}{\langle{cos}^2{\theta} - {sin}^2{\theta}\:{cos}^2{\phi}\rangle} $$
$$ \langle X \rangle = \int_{\theta_{min}}^{\theta_{max}} \int_0^{2\pi} X(\theta,\phi) f(\theta, \phi) sin(\theta) d\theta d\phi   $$
$$ f(\theta,\phi) = \frac{e^{\frac{-U}{k_b T}}}{\int_{\theta_{min}}^{\theta_{max}} \int_0^{2\pi} e^{\frac{-U}{k_b T}} \:sin\:{\theta}\: d\theta d\phi}  $$
$$ tan(2\psi) = \frac{\int_{\theta_{min}}^{\theta_{max}} \int_0^{2\pi} {sin\:{2\theta}\:cos\:{\phi}}\:e^{\frac{-U}{k_bT}}\:sin\:{\theta}\: d\theta d\phi}{\int_{\theta_{min}}^{\theta_{max}} \int_0^{2\pi} ({{cos}^2{\theta} - {sin}^2{\theta}\:{cos}^2{\phi}})\:e^{\frac{-U}{k_bT}}\:sin\:{\theta}\: d\theta d\phi}  $$
Computing
Can you write Python code for it?

#Test values at 40C
E_array = np.array(Fields[-1])
T = TempsK[-1]
PSI = np.array(Angles[-1])

#Phenomenological points from Shen paper at 40C
p0 = 619 #Debye
alpha = .338 # Micron/Volt

#Convert to SI
p0 = p0 * DebyeToSI
alpha = .338 * 1e-6
E = E_array[1]*1e6

#Combine p0 and kb to a single more reasonable number
p0k = p0/kb

#Scaling try: Puts E, alpha, and p0k all on the same order of magnitude ~200.
E = E*1e-4
alpha = alpha*1e9
p0k = p0k*1e6
E_array = E_array*1e2

def compute_psi(E,p0k,alpha):
    """Computes the tilt angle(psi) via the tan(2psi) equation at one temperature.
    Returns
    -------
    Float:
    The statistical tilt angle with conditions T,p0k,alpha,E"""

    def Boltz(theta,phi,T,p0k,alpha,E):
        """Compute the integrand for the Boltzmann factor.
        Returns
        -------
        A function of theta,phi,T,p0k,alpha,E to be used within dblquad
        """
        return np.exp((1/T)*p0k*E*np.sin(theta)*np.cos(phi)*((1*1e-2)+(alpha*E*np.cos(phi)*1e-7)))*np.sin(theta)
        #Factors: 1e-2 and 1-e7 and to cancel out the rescaling, of E,alpha, and p0k, done in the above cell

    def numerator(theta,phi,T,p0k,alpha,E):
        boltz = Boltz(theta,phi,T,p0k,alpha,E)
        return np.sin(2*theta)*np.cos(phi)*boltz

    def denominator(theta,phi,T,p0k,alpha,E):
        boltz = Boltz(theta,phi,T,p0k,alpha,E)
        return (np.cos(theta)**2 - ((np.sin(theta)*np.cos(phi))**2))*boltz

    avg_numerator, avg_numerator_error = dblquad(numerator, 0, 2*np.pi, lambda theta1: thetamin, lambda theta2: thetamax,args=(T,p0k,alpha,E))

    avg_denominator, avg_denominator_error = dblquad(denominator, 0, 2*np.pi, lambda theta3: thetamin, lambda theta4: thetamax,args=(T,p0k,alpha,E))

    psi = np.arctan(avg_numerator / avg_denominator) * (180 /(2*np.pi)) #Convert to degrees from radians and divide by two
    return psi

compute_psi(E,p0k,alpha)

plt.plot(E_array*1e-2,[compute_psi(entry,p0k,alpha) for entry in E_array],label = "Compute...


I am working on the file "ElectroOptics/Main.ipynb".
The first task is:
Compute_psi is working as intended. When supplied the correct parameters as taken from Shen's paper, Compute_psi outputs the same curve as seen in Shen's paper.
Scipy Curve Fit
Now that compute_psi is functioning, the goal is to reverse engineer the same phenomonological parameters (alpha, p0) that were used above.
Can you write Python code for it?

guess = (p0k+100,alpha+100)

theta_best, theta_cov = opt.curve_fit(compute_psi, E_array, PSI, guess)

E,alpha,p0k

E*1e-4

alpha*1e9

p0k*1e6


I am working on the file "ElectroOptics/Main.ipynb".
The first task is:
Constants
Can you write Python code for it?


I am working on the file "ElectroOptics/MinimizeAttempt.ipynb".
The first task is:
$$ tan(2\psi) =  \frac{\int_{\theta_{min}}^{\theta_{max}} \int_0^{2\pi} {sin\:{2\theta}\:cos\:{\phi}}\:e^{\frac{-U}{k_bT}}\:sin\:{\theta}\: d\theta d\phi}{\int_{\theta_{min}}^{\theta_{max}} \int_0^{2\pi} ({{cos}^2{\theta} - {sin}^2{\theta}\:{cos}^2{\phi}})\:e^{\frac{-U}{k_bT}}\:sin\:{\theta}\: d\theta d\phi} $$
Can you write Python code for it?

def compute_psi(T,p0k,alpha,E,thetamin,thetamax):
    """Computes the tilt angle(psi) by use of our tan(2psi) equation
    Returns
    -------
    Float:
    The statistical tilt angle with conditions T,p0k,alpha,E
    """

    avg_numerator, avg_numerator_error = dblquad(numerator, 0, 2*np.pi, lambda theta: thetamin, lambda theta: thetamax,args=(T,p0k,alpha,E))

    avg_denominator, avg_denominator_error = dblquad(denominator, 0, 2*np.pi, lambda theta: thetamin, lambda theta: thetamax,args=(T,p0k,alpha,E))

    psi = (1/2)*np.arctan(avg_numerator / (avg_denominator)) * (180 /(np.pi)) #Converting to degrees from radians and divide by two

    return psi


I am working on the file "ElectroOptics/MinimizeAttempt.ipynb".
The first task is:
Calculate the Tilt Angle $\psi$
$$ numerator() \:returns: {sin\:{2\theta}\:cos\:{\phi}}\:e^{\frac{-U}{k_bT}}\:sin\:{\theta} $$
Can you write Python code for it?


I am working on the file "ElectroOptics/MinimizeAttempt.ipynb".
The first task is:
Data
Can you write Python code for it?


I am working on the file "ElectroOptics/MinimizeAttempt.ipynb".
The first task is:
It might be better to use the minimization function individually for each temperature range. The minimization function returns a minimization object, which gives extra information about the results. The two important entries are fun and x.
fun is the scalar value of the function that is being minimized. In our case fun is the squared error.
x is the solution array of the form [alpha^10,p0^30]
The reason it might be better to just minimze the squared error function, instead of using the minimize_func that I wrote below is because the minimize function is very picky about the initial guess. Also the minimization function tends to stop when the result of the function is one the order of 10^-3.
Final Result for $\alpha$ and $\rho_0$
Right now everything below this might not work as well as manually guessing and checking. The idea for this section was to automate that process and just return our entire solution arrays at the end of the notebook.
Can you write Python code for it?

def minimize_func(guess,fields,T,thetamin,thetamax,measured_psi,bnds):
    """A utility function that is will help me construct alpha and p0 arrays later.
    Uses the imported minimize function and compute_error to best fit our parameters
    at a temperature.

    Parameters/Conditions
    ----------
    guess:
        The initial guess for minimize().

    Returns
    -------
    Array: [alpha,p0]
    """

    results = minimize(compute_error,guess,args=(fields,T,thetamin,thetamax,measured_psi),method = 'SLSQP',bounds = bnds)
    xres = np.array(dict(results.items())['x'])

    """Minimize returns a special minimization object. That is similar to a dictionary but not quite.
    xres is grabbing just the x result of the minimization object, which is the [alpha,p0] array that
    we care about"""

    alpha_results = xres[0]
    p0_results = xres[1]

    return np.array([alpha_results,p0_results])

guess = (2575,2168)
bnds = ((1000,2600),(200,2400))

results = minimize(compute_error,guess,args=(fields,tempsK[0],thetamin,thetamax,measured_psi1),method = 'TNC',bounds = bnds)
results

res = np.array(dict(results.items())['x'])
alpha = res[0]
p0 = res[1]
alpha = alpha*1e-4
p0 = p0/3.33564
print("alpha micro: " + str(alpha))
print('p0 debye: ' + str(p0))

#Minimization claims that it did not succeed. But the results were pretty good. I think it believes that it did not succeed because I have the squared error scaled up very high.

def solution(initial_guess,fields,tempsK,thetamin,thetamax,AllPsi,initial_bnds):

    """Constructs Alpha and p0 arrays where each entry is the value of alpha,p0 at the corresponding temperature in
    tempsK. Initial guess and initial bounds are changed each iteration of the loop to the previous values of alpha and p0.
    Alpha and p0 decrease so this helps to cut down on the range.

    Parameters/Conditions
    ----------
    initial_guess:
        The initial guess for minimize().
    initial_bnds:
        The initial bounds for...


I am working on the file "ElectroOptics/Plots V6.ipynb".
The first task is:
$$ denominator()\: returns: {({cos}^2{\theta} - {sin}^2{\theta}\:{cos}^2{\phi}})\:e^{\frac{-U}{k_bT}}\:sin\:{\theta} $$
Can you write Python code for it?

def denominator(theta,phi,T,p0k,alpha,E):
    boltz = Boltz(theta,phi,T,p0k,alpha,E)
    return ((np.cos(theta)**2) - ((np.sin(theta)**2) * (np.cos(phi)**2)))*boltz


I am working on the file "ElectroOptics/Plots V6.ipynb".
The first task is:
Construct an Array of all Computed Tilt Angles
Can you write Python code for it?

def psi_all(tempsK,fields,alpha_array,p0k_array,thetamin,thetamax):
    """Constructs an array where each entry in the array is an array of computed Psis at a Temperature.

    Parameters/Conditions
    ----------
    theta:
        Tilt Angle
    phi:
        Azimuthal Angle
    T:
        Temperature
    p0k:
        Dipole moment p0/k. Where k is Boltzmann's constant
    alpha:
        Tilt Susceptibility
    E:
        Applied Electric Field

    Returns
    -------
    Array: M Dimensional Entry with N entries. Where N is the number of Applied Electric Fields.
    M is the number of Temperatures.
    """

    def psi_array(T,p0k,alpha,fields,thetamin,thetamax):
        """Computes thes tilt angles(psi) from tan(2psi) for every electric field applied at a constant Temperature
        Returns
        -------
        Array: 1 Dimensional Entry with N entries. Where N is the number of Applied Electric Fields
        """
        Psi_Array = np.array([compute_psi(T,p0k,alpha,E,thetamin,thetamax) for E in fields])
        return Psi_Array

    return np.array([psi_array(tempsK[i],p0k_array[i],alpha_array[i],fields,thetamin,thetamax) for i in range(len(tempsK))])
AllPsi_Array = psi_all(tempsK,fields,alpha_array,p0k_array,thetamin,thetamax)


I am working on the file "ElectroOptics/Plots V6.ipynb".
The first task is:
Data
Can you write Python code for it?

tempsC = np.array([26, 27, 29, 31, 33, 35, 37])
voltages = np.array([2,3,6,7,9,11,12.5,14,16,18,20,22,23.5,26,27.5,29,31,32.5,34,36])
alpha_micro = np.array([.2575,.2475,.2275,.209,.189,.176,.15])
p0Debye = np.array([650,475,300,225,160,125,100])  #Temperature Increases to the right

#This Block just converts units

fields = np.array([entry/t for entry in voltages])

debye = 3.33564e-30
p0_array = np.array([entry*debye for entry in p0Debye]) #debye units to SI units

k = 1.3806488e-23
p0k_array = np.array([entry/k for entry in p0_array]) #p0k is used because it helps with the integration

KC = 273.15
tempsK = np.array([entry+KC for entry in tempsC]) #Celsius to Kelvin

alpha_array = np.array([entry*1e-6 for entry in alpha_micro])


I am working on the file "ElectroOptics/Plots V6.ipynb".
The first task is:
Langevin-Debye Model
$$ U(\theta,\phi) = -\rho_0E\:sin\:\theta\:cos\:\theta\:(1+\alpha E\:cos\:\phi) $$
First, Calculate the Boltzmann Factor and the Partition Function
$$ {Boltz() returns:}\:\: e^{\frac{-U}{k_bT}}\:sin\:{\theta}\ $$
Can you write Python code for it?


I am working on the file "Electromagnetic-wave-animation.ipynb".
The first task is:
Define a function that returns data to draw  vectors of a vector field with values in the plane yOz or xOy (i.e.  electric or magnetic field vectors, in our case).
Can you write Python code for it?

def rot_matrix(theta):
    return np.array([[np.cos(theta), -np.sin(theta)],
                     [np.sin(theta), np.cos(theta)]])

def get_arrows(start, end, arrow_angle,  plane=2, fract=0.1):
    """
    this i  function defines 3d vectors/quivers
    - start -numpy array of x, y, z-coordinates of the arrow starting points; shape (3, m)
      start[0,:] contains x-coordinates, etc
    - end - numpy array with the same shape as start; contains on rows the x,  y and z-coords
      of ending points of the arrow
    - the arrowhead is an isosceles triangle with the equal sides forming an angle of 2*arrow_angle radians
    - plane=0 or 2 depending on the plane where the vectors are drawn (plane=0 i.e. x=0, plane=2, z=0)
    """
    start = np.asarray(start)
    end = np.asarray(end)
    m = start[0,:].shape[0]
    arr_dir = start-end
    arr_dir_norm = np.linalg.norm(arr_dir, axis=0)
    arr_dir = fract*arr_dir/arr_dir_norm[None,:] # the arrowhead is a fraction fract from the unit vector
    if plane == 2:
        v = np.einsum('ji, im -> jm', rot_matrix(arrow_angle), arr_dir[:plane,:]) # Einstein summation                                                                               # rotation to all vectors at a time
        w = np.einsum('ji, im -> jm', rot_matrix(-arrow_angle), arr_dir[:plane, :])
        v = np.append(v, [[0]*m], axis=0)
        w = np.append(w, [[0]*m], axis=0)
    elif plane == 0:
        v = np.einsum('ji, im -> jm',  rot_matrix(arrow_angle), arr_dir[1:,:])
        w = np.einsum('ji, im -> jm', rot_matrix(-arrow_angle), arr_dir[1:, :])
        v = np.append([[0]*m], v, axis=0)
        w = np.append([[0]*m], w, axis=0)
    else: raise ValueError('the key plane must be 0 or 2')
    p = end+v
    q = end+w

    suppx = np.stack((start[0,:], end[0,:], np.nan*np.ones(m ))) #supp is the line segment as support for arrow
    suppy = np.stack((start[1,:], end[1,:], np.nan*np.ones(m )))
    suppz = np.stack((start[2,:], end[2,:], np.nan*np.ones(m )))
    x =...


I am working on the file "Electromagnetic-wave-animation.ipynb".
The first task is:
Define data for fixed traces in each frame (tr0, tr1, tr2, described below), representing the two orthogonal planes of the electric and magnetic field, and their common line:
Can you write Python code for it?


I am working on the file "Electromagnetic-wave-animation.ipynb".
The first task is:
Define data representing the vectors of the two vector fields:
Can you write Python code for it?

Ze = A*np.sin(Y/lambdA-omega*t[0])
end1 = np.stack((X, Y, Ze))
x1, y1, z1, xx1, yy1, zz1 = get_arrows(start, end1, theta, plane=0)
XXe = A*np.sin(Y/lambdA-omega*t[0])
end2 = np.stack((XXe, Y, ZZe))
x2, y2, z2, xx2, yy2, zz2 = get_arrows(start, end2, theta, plane=2)

tr0 = dict(type='scatter3d',  # a rectangle in xOy
         x=xblue,
         y=yblue,
         z=zblue,
         mode='lines',
         line=dict(width=1.5, color='blue'))
tr1 = dict(type='scatter3d',# a rectangle in yOz
         x=xred,
         y=yred,
         z=zred,
         mode='lines',
         line=dict(width=1.5, color='red'))
tr2 = dict(type='scatter3d',#line of direction Oy
         x=x_Oy,
         y=y_Oy,
         z=z_Oy,
         mode='lines',
         line=dict(width=1.5, color='rgb(140,140,140)'))


I am working on the file "Electromagnetic-wave-animation.ipynb".
The first task is:
Set the wave parameters and the interval of simulation:
Can you write Python code for it?

A = 1 # wave amplitude
lambdA = 0.5 # wavelength
omega = 1 # angular frequency
t = np.arange(0., 10., 0.2)# interval of simulation
Y = np.arange(-b, b, 0.2) # a grid of an interval on Oy, where the vector fields are evaluated
X = np.zeros(Y.shape[0])
ZZe = np.zeros(Y.shape[0])

nr_frames = t.shape[0]
theta = np.pi/13 # the characteristic angle of each arrow
start = np.stack((X, Y, np.zeros(X.shape))) # the numpy array of starting points of the the two classes of vectors
start.shape


I am working on the file "Electron REST API.ipynb".
The first task is:
Creating the electron insert spline model
Once a range of widths, lengths, and factors have been collected (at least 8) for a single applicator, ssd and energy they can be by sending them to http://electronapi.simonbiggs.net/model as follows.
First create a payload to be sent to the server.
Can you write Python code for it?


I am working on the file "Electron REST API.ipynb".
The first task is:
Parameterising an insert shape
To parameterise the insert shape x and y coordinates of the shape need to be sent to the url http://electronapi.simonbiggs.net/parameterise as follows.
First create a payload with x and y coordinates.
Can you write Python code for it?


I am working on the file "Electron REST API.ipynb".
The first task is:
Send the payload to the server api and retrieve the results.
Can you write Python code for it?

response = requests.post(
    "http://electronapi.simonbiggs.net/model",
#     "http://localhost:5000/model",
    data=payload)
result = json.loads(response.text)
model_width = np.array(result['model_width'], dtype=np.float)
model_length = np.array(result['model_length'], dtype=np.float)
model_factor = np.array(result['model_factor'], dtype=np.float)


I am working on the file "Electron REST API.ipynb".
The first task is:
The final response from the server can be plotted as so.
Can you write Python code for it?

plt.plot(insert_x, insert_y)
plt.axis('equal')

plt.plot(circle['x'], circle['y'])
plt.plot(ellipse['x'], ellipse['y'])

plt.title('Insert shape parameterisation')
plt.xlabel('Width (cm)')
plt.ylabel('Length (cm)')

plt.grid(True)


I am working on the file "Electron REST API.ipynb".
The first task is:
The result from the server can be plotted in the following way if you wish.
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(111)

vmin = np.min(np.concatenate([model_factor, factor_data]))
vmax = np.max(np.concatenate([model_factor, factor_data]))
vrange = vmax - vmin

for i in range(len(model_factor)):
    x = model_width[i]
    y = model_length[i]
    z = model_factor[i]
    c = (z - vmin) / vrange

    ax.add_patch(
        patches.Rectangle(
            (x, y),       # (x,y)
            0.1,          # width
            0.1,          # height
            facecolor=cm.viridis(c),
            edgecolor="none"
        )
    )

plt.scatter(
    width_data, length_data, s=100, c=factor_data,
    cmap='viridis', vmin=vmin, vmax=vmax, zorder=2)

ax.set_xlim([
        np.min(model_width) - 0.1,
        np.max(model_width) + 0.2])
ax.set_ylim([
        np.min(model_length) - 0.1,
        np.max(model_length) + 0.2])

plt.colorbar(label='Insert Factor')

plt.title('Insert factor model')
plt.xlabel('Width (cm)')
plt.ylabel('Length (cm)')

plt.grid(True)


I am working on the file "Electron REST API.ipynb".
The first task is:
Then send the payload to the api and retrieve the results. Make sure to loop the request until complete is returned as true.
Can you write Python code for it?

complete = False

while not(complete):
    response = requests.post(
        "http://electronapi.simonbiggs.net/parameterise",
    #     "http://localhost:5000/parameterise",
        data=payload)

    result = json.loads(response.text)

    insert_width = result['width']
    insert_length = result['length']
    circle = result['circle']
    ellipse = result['ellipse']
    complete = result['complete']

    if circle is not(None):
        plt.figure()
        plt.plot(insert_x, insert_y)
        plt.axis('equal')

        plt.plot(circle['x'], circle['y'])
        plt.title('Insert shape parameterisation')
        plt.xlabel('Width (cm)')
        plt.ylabel('Length (cm)')
        plt.grid(True)

        if ellipse is not(None):
            plt.plot(ellipse['x'], ellipse['y'])

        plt.show()

    time.sleep(2)


I am working on the file "Electron Spectrometer Simulation 2D.ipynb".
The first task is:
Load and plot magnetic field
Can you write Python code for it?

#Set tick padding
mpl.rcParams['xtick.major.pad'] = 16
mpl.rcParams['ytick.major.pad'] = 6

especFolderPath = os.curdir + '/Magnetic_Fields'

RadiaFieldPath = especFolderPath + '/3Dmagnetostaticespecdata.csv'
fname = cbook.get_sample_data(RadiaFieldPath, asfileobj=False)

# FakeFieldPath = especFolderPath + '/measuredfakefull_Bx.csv'
# FakeFieldycoordPath = especFolderPath + '/measuredfakefull_ycoord.csv'
# FakeFieldzcoordPath = especFolderPath + '/measuredfakefull_zcoord.csv'

RealFieldPath = especFolderPath + '/march7fields_Bx.csv'
RealFieldycoordPath = especFolderPath + '/march7fields_ycoord.csv'
RealFieldzcoordPath = especFolderPath + '/march7fields_zcoord.csv'

# Bx = np.matrix.transpose(np.loadtxt(open(FakeFieldPath, "rb"), delimiter=","))
# Bx = np.multiply(Bx, 0.1)
# Y = np.matrix.transpose(np.loadtxt(open(FakeFieldycoordPath, "rb"), delimiter=","))
# Z = np.matrix.transpose(np.loadtxt(open(FakeFieldzcoordPath, "rb"), delimiter=","))

Bx = np.loadtxt(open(RealFieldPath, "rb"), delimiter=",")
Bx = np.multiply(Bx, 10**-3)
Y = np.loadtxt(open(RealFieldycoordPath, "rb"), delimiter=",")
Z = np.loadtxt(open(RealFieldzcoordPath, "rb"), delimiter=",")

Ny = len(Bx)
Nz = len(Bx[0])

#Convert 1D radia array to 3D array
# Bx1DArray = np.genfromtxt(fname,delimiter=",",usecols=(5))
# By1DArray = np.genfromtxt(fname,delimiter=",",usecols=(3))
# Bz1DArray = np.genfromtxt(fname,delimiter=",",usecols=(4))
# Nx = 3 #Number of divisions along specified dimension
# Ny = 61
# Nz = 87
# Bx = np.zeros((Nx,Ny,Nz))
# By = np.zeros((Nx,Ny,Nz))
# Bz = np.zeros((Nx,Ny,Nz))
# X = range(Nx)
# Y = range(Ny)
# Z = range(Nz)
# Y, Z = np.meshgrid(Z, Y)
# for i in range(Nx):
#     for j in range(Ny):
#         for k in range(Nz):
#             Bx[i][j][k] = Bx1DArray[k+j*Nz+i*Ny*Nz]
#             By[i][j][k] = By1DArray[k+j*Nz+i*Ny*Nz]
#             Bz[i][j][k] = Bz1DArray[k+j*Nz+i*Ny*Nz]

#Plot
fig =...


I am working on the file "Electron Spectrometer Simulation 2D.ipynb".
The first task is:
Numerical Simulation with uniform magnetic field in 2D (relativistic)
Can you write Python code for it?

#Create Figure
fig = plt.figure(figsize=(16,12))
ax = fig.add_subplot(121)

#Input constants
B0 = 2136.0 #Magnetic field strength in Gauss
B0 = B0/10**4
yM = 0.5 #edge of magnet relative to slit in inches
yf = yM+0.12 #screen position relative to slit in inches
length = 3.0 #magnet and interaction length in inches
width = 0.5 #interaction width in inches
magwidth = 2.0 #magnet width in inches
magsep = 0.125 #magnet separation in inches

#Conversion to base units
yf = yf*.0254
yM = yM*.0254
length = length*0.0254
width = width*0.0254
magwidth = magwidth*0.0254
magsep = magsep*0.0254

#Electron kinetic energy in keV
KE = [round(KEcalcGeneral(9.5*10**-3,yM,yf)/(1000*q),1),round(KEcalcGeneral(length/4,yM,yf)/(1000*q),1),\
      round(KEcalcGeneral(length/3,yM,yf)/(1000*q),1),round(KEcalcGeneral(length/2,yM,yf)/(1000*q),1),\
      round(KEcalcGeneral(length/1.5,yM,yf)/(1000*q),1),round(KEcalcGeneral(length/1.25,yM,yf)/(1000*q),1),\
      round(KEcalcGeneral(length/1.1,yM,yf)/(1000*q),1),round(KEcalcGeneral(length,yM,yf)/(1000*q),1)]

for i in range(len(KE)):

    #Initial conditions
    KE[i] = KE[i]*1000*q #Convert to J
    v = [0,0,c*sqrt(1-(1/(KE[i]/(me*c**2)+1))**2)] #initial velocity
    gamma = 1/sqrt(1-v[2]**2/c**2)
    p = [0,0,gamma*me*v[2]] #initial momentum
    r = [0,0,0] #initial position
    dt = 10**-14 #time step in s

    ypos = []
    zpos = []
    while r[1]<yf:
        #update magnetic field
        if r[1]<=yM:
            B = B0
        else:
            B = 0

        #update velocity (since the magnetic field does no work, |v| and gamma are constant)
        v[1] += q*v[2]*B/(me*gamma)*dt
        v[2] += -q*v[1]*B/(me*gamma)*dt

        #update position:
        r[1] += v[1]*dt
        r[2] += v[2]*dt

        #record position:
        ypos.append(r[1]/.0254)
        zpos.append(r[2]/.0254)

        #prevent infinite loop if electrons never reach screen:
        if r[1]<0:
            break

    if i==0:
        print('KE (keV)','\t','Simulation...


I am working on the file "Electron Spectrometer Simulation 2D.ipynb".
The first task is:
Numerical simulation with measured magnetic field in 2D (relativistic)
Can you write Python code for it?

#Input constants
B0 = 2136.0 #Magnetic field strength in Gauss
r0 = [0,-12.7,-12.7] #initial position in mm
yM = 0.5 #magnet edge position in inches
CCDpos = 3.02 #CCD y-position relative to magnet edge in mm
length = 3.0 #magnet and interaction length in inches
width = 0.5 #interaction width in inches
magwidth = 2.0 #magnet width in inches
magsep = 0.125 #magnet separation in inches

#Conversion to base units
B0 = B0/10**4
yM = yM*.0254
length = length*0.0254
width = width*0.0254
magwidth = magwidth*0.0254
magsep = magsep*0.0254
r0 = np.multiply(r0,10**-3)
CCDpos = CCDpos*10**-3

#Create Figures
# fig1 = plt.figure(figsize=(16,16.4))
# ax1 = fig1.add_subplot(121)
# fig2 = plt.figure(figsize=(8,8))
# ax2 = fig2.add_subplot(111)

#Main Program (older code, should be vectorized)
yf = yM + CCDpos #Screen position
# KE = [225.0, 500.0, 1000.0, 1570.0, round(KEcalcGeneral((1500*8+9500)*10**-6,yM,yf)/(1000*q),1), \
#       round(KEcalcGeneral((2200*8+9500)*10**-6,yM,yf)/(1000*q),1)] #Electron kinetic energy in keV
# KE = [200.0, 500.0, 1000.0, 1700.0]
KE = np.arange(200.0,1710.0,10)
zfinal = []
zfinalTheory = []
Angle = []
AngleSim = []
arclengthArray = [] #arclength of trajectory
for i in range(len(KE)):

    #Initial conditions
    KE[i] = KE[i]*1000*q #Convert to J
    v = [0,0,c*sqrt(1-(1/(KE[i]/(me*c**2)+1))**2)] #initial velocity
    gamma = 1/sqrt(1-v[2]**2/c**2)
    p = [0,0,gamma*me*v[2]] #initial momentum
    r = np.copy(r0) #initial position
    dt = 10**-13 #time step in s

    ypos = []
    zpos = []
    Barray = []
    tarray = []
    t = 0 #initial time
    arclength = 0 #arclength of trajectory
    while r[1]<CCDpos:

        #if the electrons take more than a nanosecond, something wicked occurred
        if t>10*10**-9:
            break

        if r[1]==r0[1]:
            B = Bx[r0[1]*10**3,r0[2]*10**3]
        else:
            B = Binterp2D(r[1],r[2])

        #update velocity (since the magnetic field does no work, |v| and gamma are constant)
        v[1] +=...


I am working on the file "Electron Spectrometer Simulation 2D.ipynb".
The first task is:
Plot kinetic energy vs. position
Can you write Python code for it?

sns.set(context='poster',font_scale=1.5)
sns.set_style("darkgrid")
sns.set_palette(palette='deep')
sns.set_color_codes(palette='deep')

#Set tick padding
mpl.rcParams['xtick.major.pad'] = 16
mpl.rcParams['ytick.major.pad'] = 6

B0 = 1710*10**-4
zfinalPixels = np.multiply(np.subtract(zfinal, 9.5), 1000/8.0)
zfinalTheoryPixels = np.multiply(np.subtract(zfinalTheory, 9.5), 1000/8.0)
fig1, ax1 = plt.subplots(figsize=(10,6))
ax1.plot(zfinalPixels,np.divide(KESim,1000),linewidth=2, color='b', label='Particle Tracking Code')
ax1.plot(zfinalTheoryPixels,KEcalcGeneral(np.multiply(zfinalTheory,10**-3),yM,yf)/(10**6*q)\
         ,linewidth=2, color='r', label='Analytical Expression',linestyle='--')
ax1.set_ylabel('Electron Energy (MeV)')
ax1.set_xlabel('Pixel Number')
ax1.set_xlim(0,3648)
plt.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
plt.legend(loc=2)
plt.title('Energy Calibration')
plt.subplots_adjust(left=0.14,bottom=0.16) #Adjust spacing to prevent clipping of x and y labels
#plt.savefig('Energy_Calibration.svg')
print()

KE = []
zposition = []
for i in linspace(0,66.7,10000):
    zposition.append(i+9.5)
    KE.append(KEcalcGeneral(np.multiply(i+9.5,10**-3),yM,yf)/(1000*q))

# np.savetxt("EvsZ.csv", KE, delimiter=",")
# np.savetxt("Z.csv", zposition, delimiter=",")
# np.savetxt("KE.csv", np.arange(250.0,1580.0,10), delimiter=",")
# np.savetxt("Angle.csv", Angle, delimiter=",")


I am working on the file "ElectronChargePerMass/CurveFit.ipynb".
The first task is:
Fitting function to the data
For physical reasons we expect our data is described by a circle.
The equation of a circle with radius $a$ centered at $(x,y)=(b,c)$ is given by
$$(x-b)^2+(y-c)^2 = a^2$$
Let's rewrite this in terms of $y$,
$$y=-\sqrt{a^2-(x-b)^2}+c$$
We define the function and then want to find the best estimates for $a, b, c$ consistent with our data.
Can you write Python code for it?

def myfun(x,a,b,c):
    ans = -sqrt(a**2-(x-b)**2)+c  # this is y, "the function to be fit"
    return ans


I am working on the file "ElectronChargePerMass/CurveFit.ipynb".
The first task is:
Here are the initial guesses for the parameters $a$, $b$, and $c$ to pass to the fitting function.
Can you write Python code for it?


I am working on the file "ElectronChargePerMass/CurveFit.ipynb".
The first task is:
Now we use the fitted parameters in our function to compare with the data.
Can you write Python code for it?

xlots = linspace(-11,0.5)  # need lots of data points for smooth curve
yfit = myfun(xlots,plsq[0],plsq[1],plsq[2])  # use fit results for a, b, c

errorbar(x,y,ey,linestyle = '',marker = 'o')
xlabel('x (mm)')
ylabel('y (mm)')
plot(xlots,yfit)
title('Least-squares fit to data')
legend(['data','Fit'])
axis([-12,0.5,-0.5,3])
grid(True)


I am working on the file "ElectronChargePerMass/DataAnalysis.ipynb".
The first task is:
Plot Accelerating Voltage vs Square of Current
Can you write Python code for it?

plt.figure(figsize=(10,6));
plt.scatter(currents**2,voltages);
plt.ylabel('Accelerating Voltage (V)');
plt.xlabel('Square of Current ($A^2$)');

def myfun(x,qpm): #Where x is (I*r) and qpm = e/m
    ans = qpm*(128/125)*(mu_o**2)*(N**2)*(1/D**2)*x # this is y, "the function to be fit"
    return ans

p0 = [acceptedvalue]

(currents*a)**2

xlots = np.linspace(.01,.035)  # need lots of data points for smooth curve

plsq, pcov = curve_fit(myfun, (currents*a)**2, voltages, p0)  # curve fit returns p and covariance matrix
# these give the parameters and the uncertainties
epm = plsq[0]
eepm = np.sqrt(pcov[0,0])

yfit = myfun(xlots,epm)  # use fit results for a, b, c

print('epm = %.3f +/- %.3f' % (plsq[0]/1e11, np.sqrt(pcov[0,0])/1e11))

epm/acceptedvalue

plt.figure(figsize=(10,6));
plt.scatter((currents*a)**2,voltages);
plt.ylabel('Accelerating Voltage (V)');
plt.xlabel('(Current*Radius) Squared $A^2$$m^2$');
plt.plot(xlots,yfit);
plt.title('Least-squares fit to data');
plt.legend(['data','Fit'],loc='best');


I am working on the file "ElectronChargePerMass/DataAnalysis.ipynb".
The first task is:
Plot Data
Can you write Python code for it?

plt.figure(figsize=(10,6));
plt.errorbar(xpoints[1]*100,ypoints[1]*100,y_uncertainty[1]*100,linestyle = '',marker = 'o');
plt.xlabel('x (cm)',fontsize=20);
plt.ylabel('y deflection (cm)',fontsize = 20);
plt.xlim(0,12);
plt.xticks(size = 13);
plt.yticks(size = 13);
plt.savefig('Sample')


I am working on the file "ElectronChargePerMass/DataAnalysis.ipynb".
The first task is:
Plot vs Accepted Value
Can you write Python code for it?

plt.scatter(np.arange(0,len(xpoints)),eperm);
xplaces = np.linspace(-.5,5.5,50);
plt.plot(xplaces,np.array([acceptedvalue]*len(xplaces)));
plt.xlim(-.5,4.5);
plt.legend(['Accepted Value','Measured Value'],loc='lower left');
plt.ylabel('Charge to Mass Ratio (C/kg)');
plt.xlabel('Accelerating Voltage')
labels = np.array([(str(entry)+'V') for entry in voltages]);
plt.xticks(np.arange(0,len(xpoints)), labels, rotation='vertical');


I am working on the file "ElectronChargePerMass/DataAnalysis.ipynb".
The first task is:
Uncertainty
Can you write Python code for it?

std = np.std(eperm)
std

u = (np.array([np.sqrt(np.sqrt( (v_uncertainty/voltages[i])**2 + 4*(i_uncertainty/currents[i])**2 + 4*(ea[0]/a[0])**2 +
                                  4*(D_uncertainty/D)))*eperm[i] for i in np.arange(0,len(voltages))]))
avg_u = np.mean(u)

mean_std = avg_u/np.sqrt(len(xpoints))


I am working on the file "Elements.ipynb".
The first task is:
Getting the data
Can you write Python code for it?

# Read in data that will be used for the calculations.
# The data needs to be in the same directory(folder) as the program
# Using pandas read_csv method, we can create a data frame
data = pd.read_csv("./data/elements.csv")

# If you're not using a Binder link, you can get the data with this instead:
#data = pd.read_csv("http://php.scripts.psu.edu/djh300/cmpsc221/pt-data1.csv")"

# displays the first several rows of the data set
data.head()

# the names of all the columns in the dataset
data.columns


I am working on the file "Elements.ipynb".
The first task is:
Looking at some relationships
Can you write Python code for it?

ax = data.plot('Atomic Number', 'Atomic Radius (pm)', title="Atomic Radius vs. Atomic Number", legend=False)
ax.set(xlabel="x label", ylabel="y label")

data.plot('Atomic Number', 'Mass')

data[['Name', 'Year Discovered']].sort_values(by='Year Discovered')


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
Example
Here is a list containing some of the countries in Asia. Notice there are numerous strings that are empty. We'll use the filter function to remove these missing values. We'll simply pass none as the first argument and the second argument is the list of data as before.
Can you write Python code for it?

# Removing missing values from a list

countries_asia = ["Afghanistan","","Bhutan","","China","","Georgia","","","India"]

list(filter(None,countries_asia))


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
Example
The reduce function can determine the maximum of a list containing integers in a single line of code. There does exist a built-in function called max() in Python which is normally used for this purpose as max(list_name).
Can you write Python code for it?

# Determining the maximum number in a given list

from functools import reduce
f = lambda a,b : a if (a>b) else b
reduce(f,[58,69,12,158,698])


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
List Comprehensions vs Filter function
We used the filter function to remove the missing values from a list of Asian countries. Let's use list comprehensions to get the same results.
Can you write Python code for it?

# Removing missing values from a list

countries_asia = ["Afghanistan","","Bhutan","","China","","Georgia","","","India"]
[country for country in countries_asia if country!=""]


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
List Comprehensions vs Reduce function
Similarly, we can determine the maximum of a list containing integers easily using generator comprehension instead of using lambda and reduce. Generator expression are similar to list comprehension but with round brackets instead of the square one.
Can you write Python code for it?

# Determining the maximum number in a given list

numbers = [58,69,12,158,698,956]
max((x) for x in numbers)


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
List Comprehensions: Alternative to map, filter and reduce
List comprehension is a way to define and create lists in Python. in most cases, list comprehensions let us create lists in a single line of code without worrying about initializing lists or setting up loops.
It is also a substitute for the lambda function as well as the functions map(), filter() and reduce(). Some people find it a more pythonic way of writing functions and find it easier to understand.
Syntax
Can you write Python code for it?

# Creating a list of squares of first 10 numbers using loops

squares = []
for x in range(10):
    squares.append(x**2)

squares


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
Now let's see how we can accomplish this task using a single line of code with the map function.
Can you write Python code for it?

# Calculating the volume of given cubes using the Map function

map(volume,edges)


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
Now we would compute the same expression using Lambda functions. To create a lambda expression, we type in the keyword lambda, followed by the inputs. Next, we enter a colon followed by the expression that will be the return value.
Can you write Python code for it?


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
The Filter Function
The filter function constructs an iterator from those elements of iterable for which function returns true. This means filter function is used to select certain pieces of data from a list, tuple, or other collection of data, hence the name.
Syntax
filter(filter(function, iterable)
Usage
Let's see an example where we want to get the list of all numbers that are greater than 5, from a given input list.
Can you write Python code for it?

# Filter out all the numbers greater than 5 from a list

my_list = [1,2,3,4,5,6,7,8,9]
output_list = filter(lambda x : x>5, my_list)

list(output_list)


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
The following diagram shows the intermediate steps of the calculation:

The above program can also be written with an explicit for loop which is more clear.Hence Use functools.reduce if you really need it
Can you write Python code for it?

# Compute the product of a list of integers using a 'For' loop

product = 1
list = [1,2,3,4,5]
for num in list:
    product = product*num

product


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
This filters out all values that are treated as false in a boolean setting.
The Reduce Function
The Reduce function is a bit unusual and in fact, as of Python 3, it is no longer a built-in function. Instead, it has been moved to the functools module.
The 'reduce' function transforms a given list into a single value by applying a function cumulatively to the items of sequence, from left to right,
Syntax
reduce(func, seq)
where reduce continually applies the function func() to the sequence seq and returns a single value.
Usage
Let's illustrate the working of the reduce function with the help of a simple example that computes the product of a list of integers.
Can you write Python code for it?

# Compute the product of a list of integers using 'reduce'function

from functools import reduce
product = reduce((lambda x,y : x*y), [1,2,3,4,5])

product


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
This lambda function will take the input x and return 5x + 2, just like the earlier function f. There is a problem, however.The lambda is not the name of the function. It is a Python keyword that says - what follows is an anonymous function. So how do we use it? One way is to give it a name.
Let us call this lambda expression g. Now, you can use this like any other function.
Can you write Python code for it?


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
Usage
Let's try and replicate the examples used in the above sections with list comprehensions.
List Comprehensions vs Map function
We used map function in conjunction with the lambda function to convert a list of heights from cm to feet. Let's use list comprehensions to achieve the same results.
Can you write Python code for it?

# Convert height from cms to feet using List Comprehension : 1 cm = 0.0328 feet
height_in_cms = [('Tom',183),('Daisy',171),('Margaret',179),('Michael',190),('Nick',165)]

height_in_feet = [(height[0],round(height[1]*0.0328,1)) for height in height_in_cms]

height_in_feet


I am working on the file "Elements_of_Functional_Programming in_Python.ipynb".
The first task is:
What if we need to compute the volumes for many different cubes with different edge lengths?
Can you write Python code for it?


I am working on the file "EllipsesProject/MyEllipsesNotebook.ipynb".
The first task is:
Approach 1: Image Processing method
#### I will try first to see if we could get reasonable results using opencv with traditional image processing methods
I tried playing with contour and fitEllipse, but it did not give good results.
Most of the time it couldn't find the ellipses if they where drawn in the image..
Approach 2: Deep Learning method
I will try and implement a cnn network which outputs True/False if , and the features
Can you write Python code for it?

# build the model
inputs = Input(shape=X_train.shape[1:])
conv1 = Conv2D(32, (3, 3), padding='same', name='conv1')(inputs)
bn1 = BatchNormalization(axis=3,name='bn1')(conv1)
act1 = Activation('relu',name='act1')(bn1)
mp1 = MaxPooling2D(padding='same',name='mp1')(act1)
conv2 = Conv2D(64, (3, 3), padding='same',name='conv2')(mp1)
bn2 = BatchNormalization(axis=3,name='bn2')(conv2)
act2 = Activation('relu',name='act2')(bn2)
conv3 = Conv2D(64, (3, 3), padding='same',name='conv3')(act2)
conv4 = Conv2D(64, (1,1),name='conv4')(mp1)
add1 = Add(name='add1')([conv3, conv4])
bn3 = BatchNormalization(axis=3,name='bn3')(add1)
act3 = Activation('relu',name='act3')(bn3)
conv5 = Conv2D(128, (3, 3), padding='same', strides=(2,2),name='conv5')(act3)
bn4 = BatchNormalization(axis=3,name='bn4')(conv5)
act4 = Activation('relu',name='act4')(bn4)
conv6 = Conv2D(128, (3, 3), padding='same',name='conv6')(act4)
conv7 = Conv2D(128, (1,1), strides=(2,2),name='conv7')(add1)
add2 = Add(name='add2')([conv6, conv7])
bn5 = BatchNormalization(axis=3,name='bn5')(add2)
act5 = Activation('relu',name='act5')(bn5)
flat1 = Flatten(name='flat1')(act5)
dense1 = Dense(512,name='dense1')(flat1)
bn6 = BatchNormalization(name='bn6')(dense1)
act6 = Activation('relu',name='act6')(bn6)
do1 = Dropout(0.2,name='do1')(act6)
dense2 = Dense(1,name='dense2')(do1)
classification = Activation('sigmoid', name = 'class')(dense2)
dense3 = Dense(5,name='dense3')(do1)
features = Activation('relu',name='features')(dense3)

model = Model(inputs=inputs, outputs=[classification,features])

model.summary()
plot_model(model, to_file="model.png")

print("Model structure:")
fig = plt.figure(figsize=(25,30))
model_img = Image.open("model.png")
plt.imshow(model_img)


I am working on the file "EllipsesProject/MyEllipsesNotebook.ipynb".
The first task is:
As we can see, there are more ellipses than non-ellipses (greater by almost 2.4) - in training we will give all non ellipses data more weight
Can you write Python code for it?

class_weights = class_weight.compute_class_weight('balanced', np.unique(Y_train_class), Y_train_class["class"])
class_weights = {cid : weight for cid, weight in enumerate(class_weights)}
print(class_weights)


I am working on the file "EllipsesProject/MyEllipsesNotebook.ipynb".
The first task is:
Data Distribuation
Check if data is balanced
Can you write Python code for it?

print("Train Data:")
print(Counter(Y_train_class['class']))
print("Test Data")
print(Counter(Y_test_class['class']))


I am working on the file "EllipsesProject/MyEllipsesNotebook.ipynb".
The first task is:
Model evaluation
Can you write Python code for it?

# load best model wights
model.load_weights(save_to)
# loss,features_l,ellipses_l,featres_a,ellipses_a
#Y_test_features =Y_test_features.drop(columns=['angle'])
_,_,_,acc_classes,acc_features = model.evaluate(X_test, [Y_test_class,Y_test_features])
print('Test Features Acc:', acc_features)
print('Test Classification Acc:', acc_classes)

# Check errors
features_predicted = model.predict(X_test)[1]
for feat,error in zip(["center_x","center_y","angle","axis_1","axis_2"],
                      (pd.DataFrame(features_predicted) - Y_test_features.values).abs().mean()):
    if feat == "angle":
        continue #we will evaluate error later...
    print(feat,"avg error:",error,"pixels")

# check error percentages for angle
predictions = pd.DataFrame(features_predicted)
ellipses_idx = np.where(Y_test_class['class'] == 1)[0]
angle_idx = Y_test_features.columns.get_loc("angle")
true_angles = Y_test_features.loc[ellipses_idx,"angle"]
predicted_angles = predictions.loc[ellipses_idx,predictions.columns[angle_idx]]
err_percentages = (((true_angles - predicted_angles).abs() +180)%360 - 180)/true_angles
print("Avg angle error:", np.mean(err_percentages[err_percentages != np.inf]))


I am working on the file "EllipsesProject/MyEllipsesNotebook.ipynb".
The first task is:
Model training
Can you write Python code for it?

# hyperparams
epochs = 50
batch_size = 64
optAlgo = Adam()

# compile model
model.compile(optAlgo,loss=['binary_crossentropy','mse'], metrics=['accuracy'],loss_weights=[1.,.01])

# values to track:
save_to = 'best_model.h5'
checkpoint = ModelCheckpoint(filepath=save_to,verbose=0, save_best_only=True)
callbacks = [checkpoint]

# train model
history = model.fit(X_train, [Y_train_class,Y_train_features], shuffle=True, batch_size=batch_size,
                             epochs=epochs, verbose=1, validation_split=0.3, callbacks = callbacks)

# Plot training & validation accuracy values
plt.plot(history.history['class_acc'])
plt.plot(history.history['features_acc'])
plt.plot(history.history['val_class_acc'])
plt.plot(history.history['val_features_acc'])
plt.title('Model accuracy')
plt.ylabel('Accuracy')
plt.xlabel('Epoch')
plt.legend(['class_acc','features_acc','val_class_acc', 'val_features_acc'], loc='lower left')
plt.show()

# Plot training & validation loss values (general loss)
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('Model loss (General Loss)')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['loss','val_loss'], loc='upper left')
plt.show()

# Plot training & validation loss values (classification loss)
plt.plot(history.history['class_loss'])
plt.plot(history.history['val_class_loss'])
plt.title('Model loss (Classification Loss)')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['class_loss','val_class_loss'], loc='upper left')
plt.show()

# Plot training & validation loss values (features loss)
plt.plot(history.history['features_loss'])
plt.plot(history.history['val_features_loss'])
plt.title('Model loss (Features Loss)')
plt.ylabel('Loss')
plt.xlabel('Epoch')
plt.legend(['features_loss','val_features_loss'], loc='upper left')
plt.show()


I am working on the file "EllipsesProject/MyEllipsesNotebook.ipynb".
The first task is:
More preprocessing (data reordering and image normalization)

Normalize the angles - If angles are greater than 180 degrees we can normalize them
Can you write Python code for it?

Y_train_features['angle'] = Y_train_features['angle']%180
Y_test_features['angle'] = Y_test_features['angle']%180


I am working on the file "EllipsesProject/MyEllipsesNotebook.ipynb".
The first task is:
Training images paths
Can you write Python code for it?


I am working on the file "EllipsesProject/MyEllipsesNotebook.ipynb".
The first task is:
Training true features - ellipse parameters
Can you write Python code for it?

Y_train_features.head()

# Load test data
test_data = pd.read_csv("images/test_data.txt",delimiter=", ",header=0, engine='python')
# organazie the data and seperate to different dataframes
seperated_col = test_data[test_data.columns[0]].str.partition(" ")[[0,2]]
X_test_images = seperated_col[[0]].rename(columns = {0:'paths'})
Y_test_class = seperated_col[[2]].rename(columns = {2:'class'})
del test_data[test_data.columns[0]]
Y_test_features = test_data

# show some data samples
fig = plt.figure(figsize=(20,20))
for idx,path in enumerate(X_train_images['paths'].head()):
    img = Image.open(path)
    fig.add_subplot(1,5,idx+1)
    plt.imshow(img)


I am working on the file "EllipsesProject/MyEllipsesNotebook.ipynb".
The first task is:
Training true labels - ellipse or not
Can you write Python code for it?


I am working on the file "EllipsesProject/MyEllipsesNotebook.ipynb".
The first task is:
We would like to constrain the features so that the angle will allways be the ratio between the long axis and the x_axis now, it is just the ratio between the first axis and the x axis (as we can see in the images), so we make sure that the long axis is the first axis (axis_1) and the angle is adjusted accordingly.
Can you write Python code for it?

# For TRAIN
# check where axis_1 is smaller than axis_2
indices = np.where(Y_train_features['axis_1'] < Y_train_features['axis_2'])[0]
print("Before:")
display(Y_train_features.loc[indices].head())
# swap axis_1 with axis_2
tmp = Y_train_features.loc[indices,'axis_1']
Y_train_features.loc[indices,'axis_1'] = Y_train_features.loc[indices,'axis_2']
Y_train_features.loc[indices,'axis_2'] = tmp
# rotate angle by 90
Y_train_features.loc[indices,'angle'] = (Y_train_features.loc[indices,'angle']+90)%180
print("After:")
display(Y_train_features.loc[indices].head())

# For TEST
# check where axis_1 is smaller than axis_2
indices = np.where(Y_test_features['axis_1'] < Y_test_features['axis_2'])[0]
# swap axis_1 with axis_2
tmp = Y_test_features.loc[indices,'axis_1']
Y_test_features.loc[indices,'axis_1'] = Y_test_features.loc[indices,'axis_2']
Y_test_features.loc[indices,'axis_2'] = tmp
# rotate angle by 90
Y_test_features.loc[indices,'angle'] = (Y_test_features.loc[indices,'angle']+90)%180

# load train images to memory
X_train_o = np.array([np.array(Image.open(path)) for path in X_train_images['paths']]).astype('float32')
X_test_o = np.array([np.array(Image.open(path)) for path in X_test_images['paths']]).astype('float32')
print("Train shape:", X_train_o.shape)
print("Test shape:", X_test_o.shape)

# normalize rgb images
X_train = X_train_o / 255
X_test = X_test_o / 255
# y to one-hote encoding
Y_train_class['class'] = (Y_train_class['class'].values == 'True').astype(float)
Y_test_class['class'] = (Y_test_class['class'].values == 'True').astype(float)


I am working on the file "Embeddings + keras.ipynb".
The first task is:
First a look at how dna2vec does it:
Can you write Python code for it?

class Learner:
    def __init__(self, out_fileroot, context_halfsize, gensim_iters, vec_dim):
        self.logger = logbook.Logger(self.__class__.__name__)
        assert(word2vec.FAST_VERSION >= 0)
        self.logger.info('word2vec.FAST_VERSION (should be >= 0): {}'.format(word2vec.FAST_VERSION))
        self.model = None
        self.out_fileroot = out_fileroot
        self.context_halfsize = context_halfsize
        self.gensim_iters = gensim_iters
        self.use_skipgram = 1
        self.vec_dim = vec_dim

        self.logger.info('Context window half size: {}'.format(self.context_halfsize))
        self.logger.info('Use skipgram: {}'.format(self.use_skipgram))
        self.logger.info('gensim_iters: {}'.format(self.gensim_iters))
        self.logger.info('vec_dim: {}'.format(self.vec_dim))

    def train(self, kmer_seq_generator):
        self.model = word2vec.Word2Vec(
            sentences=kmer_seq_generator,
            size=self.vec_dim,
            window=self.context_halfsize,
            min_count=5,
            workers=4,
            sg=self.use_skipgram,
            iter=self.gensim_iters)

        # self.logger.info(model.vocab)

    def write_vec(self):
        out_filename = '{}.w2v'.format(self.out_fileroot)
        self.model.save_word2vec_format(out_filename, binary=False)


I am working on the file "Embeddings + keras.ipynb".
The first task is:
Kind of close? 42 % to 37%....
Can you write Python code for it?

from datasketch import MinHash
import random
nws = []
sims = []
sketches = []
counter = 0
outside_counter = 0

keys = list(word_vectors.vocab.keys())
for i in range(1,1000):
    #print(word_vectors.vocab.keys())
    a = random.choice(keys)
    b = random.choice(keys)
    if a == b: continue
    if len(a) != len(b) != 5: continue
    # get similarity via needleman-wunsch
    new_score = pairwise2.align.globalxx(a, b)[0]
    # ('CCAC--T', '-CA-AAT', 3.0, 0, 7)
    score, length = new_score[2], new_score[-1]
    score += 1

    nws.append(score)

    # get from dna2vec
    similarity = word_vectors.similarity(a, b) # 0.3
    sims.append(similarity)

    # get from minhash
    m1, m2 = MinHash(), MinHash()
    for d in a:
        m1.update(d.encode('utf8'))
    for d in b:
        m2.update(d.encode('utf8'))
    jaccard = m1.jaccard(m2)

    sketches.append(jaccard)

import seaborn as sns
import pandas as pd
df = pd.DataFrame({'Needleman-Wunsch':nws, 'word2vec':sims})
%matplotlib inline
sns.violinplot(x='Needleman-Wunsch', y='word2vec', data=df);

import scipy
print(scipy.stats.pearsonr(nws, sims), scipy.stats.pearsonr(sketches, sims), scipy.stats.pearsonr(nws, sketches))


I am working on the file "Embeddings + keras.ipynb".
The first task is:
That's the weights for Keras!
Can you write Python code for it?

word_vectors.syn0.shape

word_vectors.most_similar(positive=['AAA'], negative=['TTT'])

word_vectors.most_similar(positive=['AAA'])

word_vectors.similarity('AGAAT','AAGTA')


I am working on the file "Embeddings + keras.ipynb".
The first task is:
The above are the weights you can use in embeddings

Now let's look at DNA!
Can you write Python code for it?


from gensim.models import word2vec
word_vectors = word2vec.Word2Vec.load_word2vec_format('./dna2vec-20170621-0833-k3to5-12d-4c-6Mbp-sliding-nqR.w2v')

word_vectors.syn0


I am working on the file "Embeddings + keras.ipynb".
The first task is:
Welp that's not good. I guess the training corpus is too small...
Can you write Python code for it?


I am working on the file "Embeddings + keras.ipynb".
The first task is:
let's put these sentences into a dumb text file for the helper, just as an example
Can you write Python code for it?

text = open('simple_text.txt','w')
for element in all_my_words:
    text.write('%s.\n'%element)

from gensim.models.word2vec import LineSentence
sentences = LineSentence(text.name)
model = Word2Vec(sentences, size=100, window=4, min_count=5)

vocab = list(model.vocab.keys())
vocab[:10]

model['physics']

model.most_similar_cosmul(positive=['nazis', 'japan'])

model.similarity('book', 'novel')


I am working on the file "Embeddings + keras.ipynb".
The first task is:
wat. was expecting something higher. Let's try t-SNE
Can you write Python code for it?

from sklearn.manifold import TSNE
import matplotlib.pyplot as plt

X = word_vectors[word_vectors.vocab]
print(X[0:3,0:3])

tsne = TSNE(n_components=2)
X_tsne = tsne.fit_transform(X)
plt.scatter(X_tsne[:, 0], X_tsne[:, 1])
plt.show()


I am working on the file "Embeddings and Word2Vec/Skip-Gram word2vec.ipynb".
The first task is:
And here I'm creating dictionaries to covert words to integers and backwards, integers to words. The integers are assigned in descending frequency order, so the most frequent word ("the") is given the integer 0 and the next most frequent is 1 and so on. The words are converted to integers and stored in the list int_words.
Can you write Python code for it?

vocab_to_int, int_to_vocab = utils.create_lookup_tables(words)
int_words = [vocab_to_int[word] for word in words]


I am working on the file "Embeddings and Word2Vec/Skip-Gram word2vec.ipynb".
The first task is:
Building the graph
From Chris McCormick's blog, we can see the general structure of our network.

The input words are passed in as one-hot encoded vectors. This will go into a hidden layer of linear units, then into a softmax layer. We'll use the softmax layer to make a prediction like normal.
The idea here is to train the hidden layer weight matrix to find efficient representations for our words. This weight matrix is usually called the embedding matrix or embedding look-up table. We can discard the softmax layer becuase we don't really care about making predictions with this network. We just want the embedding matrix so we can use it in other networks we build from the dataset.
I'm going to have you build the graph in stages now. First off, creating the inputs and labels placeholders like normal.

Exercise: Assign inputs and labels using tf.placeholder. We're going to be passing in integers, so set the data types to tf.int32. The batches we're passing in will have varying sizes, so set the batch sizes to [None]. To make things work later, you'll need to set the second dimension of labels to None or 1.
Can you write Python code for it?

train_graph = tf.Graph()
with train_graph.as_default():
    inputs = tf.placeholder(tf.int32, shape=[None], name="inputs")
    labels = tf.placeholder(tf.int32, shape=[None, None], name="labels")


I am working on the file "Embeddings and Word2Vec/Skip-Gram word2vec.ipynb".
The first task is:
Load the text8 dataset, a file of cleaned up Wikipedia articles from Matt Mahoney. The next cell will download the data set to the data folder. Then you can extract it and delete the archive file to save storage space.
Can you write Python code for it?

from urllib.request import urlretrieve
from os.path import isfile, isdir
from tqdm import tqdm
import zipfile

dataset_folder_path = 'data'
dataset_filename = 'text8.zip'
dataset_name = 'Text8 Dataset'

class DLProgress(tqdm):
    last_block = 0

    def hook(self, block_num=1, block_size=1, total_size=None):
        self.total = total_size
        self.update((block_num - self.last_block) * block_size)
        self.last_block = block_num

if not isfile(dataset_filename):
    with DLProgress(unit='B', unit_scale=True, miniters=1, desc=dataset_name) as pbar:
        urlretrieve(
            'http://mattmahoney.net/dc/text8.zip',
            dataset_filename,
            pbar.hook)

if not isdir(dataset_folder_path):
    with zipfile.ZipFile(dataset_filename) as zip_ref:
        zip_ref.extractall(dataset_folder_path)

with open('data/text8') as f:
    text = f.read()