I am working on the file ".ipynb_checkpoints/CNTK_model_scoring_on_Spark_walkthrough-checkpoint.ipynb".
The first task is:

Score the images using a trained CNTK model
Now that the cluster and sample dataset have been created, we can use PySpark to apply a trained model to the images.

Download the trained CNTK model and mean image to the Spark cluster
We previously trained a twenty-layer ResNet model to classify CIFAR-10 images by following this tutorial from the CNTK git repo. The model expects input images to be preprocessed by subtracting the mean image defined in an OpenCV XML file. The following cell downloads both the trained model and the mean image, and ensures that data from both files can be accessed by worker nodes.
Can you write Python code for it?

urlretrieve(model_uri, local_model_path)
sc.addFile(local_model_path)

urlretrieve(mean_image_uri, local_mean_image_path)
mean_image = xml.etree.ElementTree.parse(local_mean_image_path).getroot()
mean_image = [float(i) for i in mean_image.find('MeanImg').find('data').text.strip().split(' ')]
mean_image = np.array(mean_image).reshape((32, 32, 3)).transpose((2, 0, 1))
mean_image_bc = sc.broadcast(mean_image)


I am working on the file ".ipynb_checkpoints/CNTK_model_scoring_on_Spark_walkthrough-checkpoint.ipynb".
The first task is:
To convince ourselves that the data has been properly loaded, let's visualize a few of these images. For plotting, we will need to transfer them to the local context by way of a Spark dataframe:
Can you write Python code for it?

sample_images = image_rdd.take(5)
image_data = np.array([i[0].reshape((32*32*3)) for i in sample_images]).T
image_labels = [i[2] for i in sample_images]
image_df = pd.DataFrame(image_data, columns=image_labels)
spark.createDataFrame(image_df).coalesce(1).write.mode("overwrite").csv("/tmp/cifar_image", header=True)

import pandas as pd
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from glob import glob

image_df = pd.read_csv(glob('/tmp/cifar_image/*.csv')[0])
plt.figure(figsize=(15,1))
for i, col in enumerate(image_df.columns):
    plt.subplot(1, 5, i+1)
    image = image_df[col].values.reshape((32, 32, 3))
    plt.imshow(image)
    plt.title(col)
    cur_axes = plt.gca()
    cur_axes.axes.get_xaxis().set_visible(False)
    cur_axes.axes.get_yaxis().set_visible(False)


I am working on the file ".ipynb_checkpoints/CNTK_model_scoring_on_Spark_walkthrough-checkpoint.ipynb".
The first task is:
We can construct a confusion matrix to visualize which classification errors are most common:
Can you write Python code for it?

spark.createDataFrame(df).coalesce(1).write.mode("overwrite").csv("/tmp/cifar_scores", header=True)

import pandas as pd
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix
import os
from glob import glob

df = pd.read_csv(glob('/tmp/cifar_scores/*.csv')[0])
print('Constructing a confusion matrix with the first {} samples'.format(len(df.index)))

label_to_name_dict = {0: 'airplane',
                      1: 'automobile',
                      2: 'bird',
                      3: 'cat',
                      4: 'deer',
                      5: 'dog',
                      6: 'frog',
                      7: 'horse',
                      8: 'ship',
                      9: 'truck'}

labels = np.sort(df['true_label'].unique())
named_labels = [label_to_name_dict[i] for i in labels]
cm = confusion_matrix(df['true_label'], df['predicted_label'], labels=labels)

plt.imshow(cm, interpolation='nearest', cmap=plt.cm.Blues)
plt.colorbar()
tick_marks = np.arange(len(labels))
plt.xticks(tick_marks, named_labels, rotation=90)
plt.yticks(tick_marks, named_labels)
plt.xlabel('Predicted label')
plt.ylabel('True Label')
plt.show()


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-checkpoint.ipynb".
The first task is:
For now, we'll play with the cool scientists and use the powerful and fast scikit learn package.
Our Zero-ith tool: cleaning up the text
I've included a little utility function in tm that takes a list of strings and cleans it up a bit
check out the code on your own time later
Can you write Python code for it?

our_texts=tm.data_cleanse(our_texts)

#more necessary when have messy text
#eliminate escaped characters


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-checkpoint.ipynb".
The first task is:
HOMEWORK EXERCISE:
for given document find the most similar and give titles from the csv file
you'll see!
supervised vs. unsupervised learning
slides from class omitted
first example of unsupervised learning
hierarchical clustering
This time we're interested in relations among the words not the texts.
In other words, we're interested in the similarities between one column and another--one term and another term
So we'll work with the transposed matrix--the term-document matrix, rather than the document-term matrix.
For a description of hierarchical clustering, look at the example at https://en.wikipedia.org/wiki/Hierarchical_clustering
Can you write Python code for it?

term_document_matrix=document_term_matrix.T
# .T is the easy transposition method for a
# matrix in python's matrix packages.

# import a bunch of packages we need
import matplotlib.pyplot as plt
from sklearn.metrics.pairwise import cosine_similarity
from scipy.cluster.hierarchy import ward, dendrogram

#distance is 1-similarity, so:

dist=1-cosine_similarity(term_document_matrix)

# ward is an algorithm for hierarchical clustering

linkage_matrix=ward(dist)

#plot dendogram

f=plt.figure(figsize=(9,9))
R=dendrogram(linkage_matrix, orientation="right", labels=vocab)
plt.tight_layout()


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-checkpoint.ipynb".
The first task is:
IMPORTANT: tm is our temporarily helper, not a standard python package!!
download it from my github:
https://github.com/matthewljones/computingincontext
Can you write Python code for it?

#see if package imported correctly
tm.icantbelieve("butter")


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-checkpoint.ipynb".
The first task is:
Let's keep using the remarkable narratives available from Documenting the American South (http://docsouth.unc.edu/docsouthdata/)
Assuming that you are storing your data in a directory in the same place as your iPython notebook.
Put the slave narratives texts within a data directory in the same place as this notebook
Can you write Python code for it?

title_info=pd.read_csv('data/na-slave-narratives/data/toc.csv')
#this is the "metadata" of these files--we'll use today
#why does data appear twice in the filename?

title_info

title_info["Date"].str.replace("\-\?", "5")

title_info["Date"].str.replace("[^0-9]", "") #use regular expressions to clean up

title_info["Date"]=title_info["Date"].str.replace("\-\?", "5")
title_info["Date"]=title_info["Date"].str.replace("[^0-9]", "") # what assumptions have I made about the data?

title_info["Date"]=pd.to_datetime(title_info["Date"], coerce=True)

title_info["Date"]<pd.datetime(1800,1,1)

title_info[title_info["Date"]<pd.datetime(1800,1,1)]


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-checkpoint.ipynb".
The first task is:
back to boolean indexing!
Can you write Python code for it?

#Let's use a brittle thing for reading in a directory of pure txt files.
our_texts=tm.readtextfiles('data/na-slave-narratives/data/texts')
#again, this is not a std python package
#returns a simple list of the document as very long strings

#note if you want the following notebook will work on any directory of text files.


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-checkpoint.ipynb".
The first task is:
back to vectorizer from scikit learn
Can you write Python code for it?

from sklearn.feature_extraction.text import TfidfVectorizer

vectorizer=TfidfVectorizer(min_df=0.5, stop_words='english', use_idf=True)

document_term_matrix=vectorizer.fit_transform(our_texts)

# now let's get our vocabulary--the names corresponding to the rows
vocab=vectorizer.get_feature_names()

len(vocab)

document_term_matrix.shape


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-checkpoint.ipynb".
The first task is:
so  document_term_matrix is a matrix with 294 rows--the documents--and 1650 columns--the vocabulary or terms or features
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-checkpoint.ipynb".
The first task is:
that is a symmetrical matrix relating each of the texts (rows) to another text (row)
Can you write Python code for it?

similarity[100]
#this gives the similarity of row 100 to each of the other rows


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-fixing things-Copy1-checkpoint.ipynb".
The first task is:
For now, we'll play with the cool scientists and use the powerful and fast scikit learn package.
Let's look at Victorian novels for a little while
Can you write Python code for it?

our_texts, names=tm.readtextfiles("data/british-fiction-corpus")

names


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-fixing things-Copy1-checkpoint.ipynb".
The first task is:
It's an 11 by 2 matrix
OR
simply an (x,y) coordinate pair for each of our texts
Can you write Python code for it?

#let's plot it: I've set up a black box
tm.plot_mds(positions,names)

names=[name.replace(".txt", "") for name in names]

tm.plot_mds(positions,names)


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-fixing things-Copy1-checkpoint.ipynb".
The first task is:
Let's keep using the remarkable narratives available from Documenting the American South (http://docsouth.unc.edu/docsouthdata/)
Assuming that you are storing your data in a directory in the same place as your iPython notebook.
Put the slave narratives texts within a data directory in the same place as this notebook
Can you write Python code for it?

title_info["Date"].str.replace("[^0-9]", "") #use regular expressions to clean up

title_info["Date"]=title_info["Date"].str.replace("\-\?", "5")
title_info["Date"]=title_info["Date"].str.replace("[^0-9]", "") # what assumptions have I made about the data?

title_info["Date"]=pd.to_datetime(title_info["Date"], coerce=True)

title_info["Date"]<pd.datetime(1800,1,1)

title_info[title_info["Date"]<pd.datetime(1800,1,1)]


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-fixing things-Copy1-checkpoint.ipynb".
The first task is:
Reading at scale:
Martha Ballard's Diary
http://dohistory.org/diary/index.html
http://www.cameronblevins.org/posts/topic-modeling-martha-ballards-diary/
Richmond Dispatch
https://dsl.richmond.edu/dispatch/pages/home
Source: http://dlxs.richmond.edu/d/ddr/index.html
Can you write Python code for it?

from IPython.display import Image
Image("http://journalofdigitalhumanities.org/wp-content/uploads/2013/02/blei_lda_illustration.png")

import textmining_blackboxes as tm


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-fixing things-Copy1-checkpoint.ipynb".
The first task is:
So which topics most significant for each document?
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-fixing things-Copy1-checkpoint.ipynb".
The first task is:
While this data frame is lovely to look at and useful to think with, it's tough on your computer's memory
Can you write Python code for it?

#easy to program, but let's use a robust version from sklearn!
from sklearn.metrics.pairwise import cosine_similarity

similarity=cosine_similarity(document_term_matrix)

#Note here that the `cosine_similiary` can take
#an entire matrix as its argument

pd.DataFrame(similarity, index=names, columns=names)


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-fixing things-Copy1-checkpoint.ipynb".
The first task is:
back to boolean indexing!
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-fixing things-Copy1-checkpoint.ipynb".
The first task is:
can do lots of things with similarity matrix
you've already seen hierarchical clustering
Multidimension scaling

Technique to visualize distances in high dimensional spaces in ways we can cognize.
Keep distances but reduce dimensionality
Can you write Python code for it?

#here's the blackbox
from sklearn.manifold import MDS
mds = MDS(n_components=2, dissimilarity="precomputed", random_state=1)
positions= mds.fit_transform(1-similarity)

positions.shape


I am working on the file ".ipynb_checkpoints/CiC_lecture_04_text_mining_topics_trends-fixing things-Copy1-checkpoint.ipynb".
The first task is:
that is a symmetrical matrix relating each of the texts (rows) to another text (row)
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
Además, se pueden armar strings multilínea poniendo tres comillas simples o dobles seguidas:
Can you write Python code for it?

cadena_caracteres = """y si quiero
usar un string
que se escriba en varias
líneas?."""
print cadena_caracteres
print type(cadena_caracteres)


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
Como ya dijimos antes, se puede usar el operador in para saber si un elemento se encuentra en una lista:
Can you write Python code for it?

lista_de_numeros = []

if lista_de_numeros:
    print 'la lista tiene elementos'
else:
    print 'la lista no tiene elementos'


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
Cualquier tipo de dato se lo puede evaluar como booleano.
Se toma como falso a:
* None
* False para los bool
* cero para todo tipo de dato numérico: 0, 0L, 0.0, 0j
* vacío para cualquier secuencia o diccionario: '', (), [], {}
Por lo tanto, se puede saber si una lista esta vacía o no con simplemente:
Can you write Python code for it?

if lista_de_numeros:
    print 'La lista no esta vacía'

if False or None or [] or () or {} or 0:
    print 'Alguna de las anteriores no era falsa'
else:
    print 'Todos los valores anteriores son consideradas como Falso'


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
En cambio, en Python lo podemos un poco más compacto y claro:
Can you write Python code for it?

# Más corto y elegante en Python.
if numero1 == numero2:
    print 'Los dos números son iguales'
elif numero1 > numero2:
    print 'numero1 es mayor a numero2'
else:
    print 'numero1 es menor a numero2'


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
En cambio, si alguno de los números es real:
Can you write Python code for it?

divisor = 3.0
resultado = dividendo / divisor
print resultado
print type(resultado)


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
Esto cambia en Python 3, donde la / hace la división real (por más que le pases dos números enteros) y la // hace la división entera.
Complejos
Can you write Python code for it?

complejo = 5 + 3j
print complejo
print type(complejo)
complejo_cuadrado = complejo ** 2
print '(5+3j)*(5+3j) = 5*5 + 5*3j + 3j*5 + 3j*3j = (25-9) + 30j'
print complejo_cuadrado


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
H | o | l | a |   | m | u | n | d | o
-------| ---------- |  ------------
0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
-10 | -9 | -8 | -7 | -6 | -5 | -4 | -3 | -2 | -1
Lo que no se puede hacer es cambiar sólo una letra de un string:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
Incluso, se puede saber fácilmente si un número está dentro de un rango o no.
Can you write Python code for it?

numero = 7
en_rango = 5 < numero < 9
fuera_de_rango = 5 < numero < 6
print 'numero vale {0}'.format(numero)
print '5 < numero < 9: {pertenece}'.format(pertenece=en_rango)
print '5 < numero < 6: %s' % fuera_de_rango


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
Operaciones entre reales y enteros
¿Y si dividimos dos números enteros?, ¿dará un número real?
Can you write Python code for it?

dividendo = 5
divisor = 3
resultado = dividendo / divisor
print resultado
print type(resultado)


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
Reales
Can you write Python code for it?

numero_real = 7.5
print numero_real
print type(numero_real)


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
Si bien Python soporta aritmética de complejos, la verdad es que no es uno de los tipos de datos más usados. Sin embargo, es bueno saber que existe.
Booleanos
Python también soporta el tipo de dato booleano:
Can you write Python code for it?

boolean = True
print boolean
print not boolean
print type(boolean)
print True or False and True


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
Strings
En python los strings se pueden armar tanto con comillas simples (') como dobles ("), lo que no se puede hacer es abrir con unas y cerrar con otras.
Can you write Python code for it?

cadena_caracteres = 'Hola mundo'
print cadena_caracteres
print type(cadena_caracteres)

cadena_caracteres = "Y con doble comilla?, de qué tipo es?"
print cadena_caracteres
print type(cadena_caracteres)


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
También se puede crear un boolean a partir de comparar dos números:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
if-elif-else
Ahora si queremos imprimir si un número es igual, menor o mayor a otro tendríamos que usar if anidados en Pascal o C; y no queda del todo claro:
Can you write Python code for it?

# Como lo tendríamos que hacer en Pascal o C.
if numero1 == numero2:
    print 'Los dos números son iguales'
else:
    if numero1 > numero2:
        print 'numero1 es mayor a numero2'
    else:
        print 'numero1 es menor a numero2'


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
if-else
Can you write Python code for it?

numero1 = 1
numero2 = 2

if numero1 == numero2:
    print 'Los números son iguales'
    print 132
    print 23424
else:
    print 'Los números son distintos'


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
short-if
Otra forma de escribir el if en una sola línea es poner:
Python
variable = valor1 if condicion else valor2
Por ejemplo:
Can you write Python code for it?

num = 5
es_par = True if (num % 2 == 0) else False
print '5 es par?:', es_par

num = 6
es_par = True if (num % 2 == 0) else False

print '6 es par?:', es_par

nulo = None
print nulo
print type(nulo)


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
¿Y si queremos hacer la división entera por más que uno de los números sea real?
Can you write Python code for it?

cociente = dividendo // divisor
print "cociente: ", cociente
print type(cociente)

resto = dividendo % divisor
print "resto: ", resto
print type(resto)


I am working on the file ".ipynb_checkpoints/Clase 01 - Introducción y estructuras selectivas-checkpoint.ipynb".
The first task is:
Índices y Slice en string
Si queremos obtener un caracter del string podemos acceder a él simplemente con poner entre corchetes su posición (comenzando con la 0):
Can you write Python code for it?

cadena_caracteres = 'Hola mundo'
print cadena_caracteres
print 'El septimo caracter de la cadena "{0}" es "{1}"'.format(cadena_caracteres, cadena_caracteres[6])


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
Aunque también podría obtener el primero:
Can you write Python code for it?

lista = range(5)
print 'La lista antes de entrar al while tiene:', lista
while lista:  # Si la lista no esta vacía, sigo sacando elementos
    print lista.pop(0)

print 'La lista después de salir del while tiene:', lista


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
Diccionarios
El equivalente a los registros de Pascal serían los diccionarios, pero éstos también ofrecen mayor flexibilidad:
Can you write Python code for it?

registros_con_campos_variables = {'campo1': 12, 'campo2': 'valor campo2'}
print registros_con_campos_variables
print type(registros_con_campos_variables)
print

print 'Le agrego un campo al diccionario'
registros_con_campos_variables['otro_campo'] = 432
print registros_con_campos_variables
print

print 'Y ahora otro, pero con un int como índice'
registros_con_campos_variables[123] = 'también puede usarse los números como clave'
print registros_con_campos_variables


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
Existe una función llamada range que crea permite crear listas de números:
Can you write Python code for it?

print range.__doc__
print
print 'Ejemplos:'
print '  range(15):', range(15)
print '  range(15)[2:9]:', range(15)[2:9]
print '  range(15)[2:9:3]:', range(15)[2:9:3]
print '  range(2,9):', range(2,9)
print '  range(2,9,3):', range(2,9,3)


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
Incluso, si queremos imprimir los valores de una lista que nosotros armamos, también podemos hacerlo:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
Las listas tienen una función llamada pop que lo que hace es tomar el último elemento de ella y lo elimina:
Can you write Python code for it?

lista = range(5)
print 'La lista antes de entrar al while tiene:', lista
while lista:  # Si la lista no esta vacía, sigo sacando elementos
    print lista.pop()

print 'La lista después de salir del while tiene:', lista


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
O incluso se le puede cambiar el tipo de dato a un campo y agregar uno nuevo:
Can you write Python code for it?

alumno = {
    'nombre': 'Juan',
    'apellido': 'Perez',
    'parcial': 2
}
print 'Alumno:', alumno

alumno['parcial'] = [2, 6]  # Cambio el tipo de dato de int a list
print 'Agrego la nota del recuperatorio:', alumno

alumno['coloquio'] = 8  # Agrego un nuevo campo
print 'Agrego la nota del coloquio:', alumno

del alumno['parcial']  # Elimino el campo nota
print 'Elimino las notas del parcial:', alumno


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
O modificar un elemento de la lista:
Can you write Python code for it?

lista_de_numeros[3] = 152
print lista_de_numeros


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
Para eliminar un elemento sólo tenemos que usar la función del e indicar la posición.
Can you write Python code for it?

lista_de_cosas = [2, 5.5, 'letras', [1, 2, 3], ('tupla', 'de', 'strings')]
print 'Lista de cosas:', lista_de_cosas

del lista_de_cosas[3]
print 'Después de eliminar la posición 3:', lista_de_cosas


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
También se puede usar la función zip para ir tomando los primeros elementos de una lista, después los segundos, y así sucesivamente
Can you write Python code for it?

for par in zip([1, 2, 3], [4, 5, 6]):
    print par


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
También se pueden iterar listas que tengan distintos tipos de elementos, pero hay que tener en cuenta qué se quiere hacer con ellos:
Can you write Python code for it?

lista = [1, 2, "12", "34", [5, 6]]
print 'La lista tiene los elementos:', lista
for elemento in lista:
    print '{0}*2: {1}:'.format(elemento, elemento*2)


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
Tuplas
Las tuplas son listas inmutables, es decir, que no se pueden modificar.
Si no se pueden modificar, ¿para qué existen?. Porque crearlas es mucho más eficiente que crear listas y en muchas ocasiones, como con las constantes, queremos crear variables que no se modifiquen.
Can you write Python code for it?

tupla = (1, 2, 3, 4)  # Se usa paréntesis en lugar de corchetes
print tupla

tupla = tupla[2:4]
print tupl
print type(tupla)


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
Y con las listas también se pueden hacer slices:
Can you write Python code for it?

print 'primer elemento:', lista_de_cosas[0]
ultimo = lista_de_cosas[-1]
print 'último:', ultimo
print 'del_segundo_al_ultimo_sin_incluirlo:', lista_de_cosas[1:4]
print 'del_segundo_al_ultimo_sin_incluirlo:', lista_de_cosas[1:-1]
print 'del_segundo_al_ultimo_incluyendolo:', lista_de_cosas[1:]


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
Y en realidad, se puede iterar sobre cualquier elemento iterable, como por ejemplo los strings:
Can you write Python code for it?

for caracter in "Hola mundo":
    print caracter


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
Y si le queremos modificar la nota a un alumno, sólo tenemos que acceder a ese campo y asignarle un nuevo valor:
Can you write Python code for it?

alumno = {
    'nombre': 'Juan',
    'apellido': 'Perez',
    'nota': 2
}
print alumno

alumno['nota'] = 5
print alumno


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
Y si queremos imprimir cada elemento de la lista junto con su posición podemos usar la función enumerate:
Can you write Python code for it?

lista = range(15, 30, 3)
print lista
for idx, value in enumerate(lista):
    print '%s: %s' % (idx, value)


I am working on the file ".ipynb_checkpoints/Clase 02 - Tipos de datos computestos y ciclos-checkpoint.ipynb".
The first task is:
while
El ciclo while también ejecuta un bloque de código mientras la condición sea verdadera:
Can you write Python code for it?

numero = 5
while numero < 10:
    print numero
    numero += 1


I am working on the file ".ipynb_checkpoints/Clase 03 - Funciones y modulos-checkpoint.ipynb".
The first task is:
Ahora, si la función es un tipo de dato, significa que se lo puedo asignar a una variable...
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Clase 03 - Funciones y modulos-checkpoint.ipynb".
The first task is:
Ahora, si nos pasan 4 parámetros, el intérprete asumirá que el faltante es param5, por lo que dicho parámetro tomará el valor False. Y lo mismo pasa con el resto de los parámetros.
Can you write Python code for it?

imprimir_parametros(1, 2, 3, 4)
imprimir_parametros(1, 2, 3)
imprimir_parametros(1, 2)


I am working on the file ".ipynb_checkpoints/Clase 03 - Funciones y modulos-checkpoint.ipynb".
The first task is:
El resultado de la función no es necesario que lo guarde en una variable, tranquilamente la puedo invocar y perder ese valor.
Can you write Python code for it?

def factorial(n):
    """Calcula el factorial de un número de forma iterativa.
    """
    for i in range(1,n):
        n *= i
    return n

fact_5 = factorial(5)  # calculo el factorial de 5 y lo guardo en fact_5
factorial(10)  # calculo el factorial de 10 y no lo guardo en ninguna variable


I am working on the file ".ipynb_checkpoints/Clase 03 - Funciones y modulos-checkpoint.ipynb".
The first task is:
Para esta función nos pueden pasar 2, 3, 4 o 5 parámetros. Si nos pasan los 5 parámetros, se imprimirán los valores que nos pasen:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Clase 03 - Funciones y modulos-checkpoint.ipynb".
The first task is:
Por lo que no existen los procedimientos, los "procedimientos" en realidad son funciones que devuelven None.
Y una prueba más de esto es el resultado de llamar a la función type y pasarle como parámetro la función sumar y el "procedimiento" imprimir:
Can you write Python code for it?

print type(imprimir)
print type(sumar)
print sumar


I am working on the file ".ipynb_checkpoints/Clase 03 - Funciones y modulos-checkpoint.ipynb".
The first task is:
Uso de módulos externos
Así como en Pascal usando la cláusula Uses podíamos usar código que no pertenecía al archivo que estábamos codificando, en Python podemos hacer lo mismo usando la cláusula import y poniendo a continuación el nombre del módulo.

Por ejemplo, si queremos importar el módulo datetime para trabajar con fechas y horas, tendríamos que hacer:
Python
import datetime
Para usarlo simplemente tenemos que poner el nombre del módulo, un punto y la función que queramos usar.

En este caso, dentro del módulo datetime vamos a usar la función que se encuentra en date y se llama today().
Can you write Python code for it?

import datetime

print datetime.date.today()


I am working on the file ".ipynb_checkpoints/Clase 03 - Funciones y modulos-checkpoint.ipynb".
The first task is:
¿Y qué pasa si quiero pasarle los parámetros 1, 2 y el 5?. <br>
No es problema, para eso tenemos que usar parámetros nombrados:
Can you write Python code for it?

imprimir_parametros(1, 2, param5="Este el parametro5")


I am working on the file ".ipynb_checkpoints/Clase 03 - Funciones y modulos-checkpoint.ipynb".
The first task is:
¿Y qué sucede si no pongo el return en una función?
Can you write Python code for it?

def imprimir(msg):
    print msg

imprimir('Hola mundo')


I am working on the file ".ipynb_checkpoints/Clase 03 - Funciones y modulos-checkpoint.ipynb".
The first task is:
¿Y si le asigno el resultado de este procedimiento a una variable?
Can you write Python code for it?

resultado = imprimir('Hola mundo')
print resultado


I am working on the file ".ipynb_checkpoints/Clase 03 - Funciones y modulos-checkpoint.ipynb".
The first task is:
¿Y si le pasamos un sólo parámetro?.
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Búsquedas en listas
Para saber si un elemento se encuentra en una lista, alcanza con usar el operador in:
Can you write Python code for it?

lista = [11, 4, 6, 1, 3, 5, 7]

if 3 in lista:
    print '3 esta en la lista'
else:
    print '3 no esta en la lista'

if 15 in lista:
    print '15 esta en la lista'
else:
    print '15 no esta en la lista'


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
En cambio, si le pasamos 0 en el lugar de x:
Can you write Python code for it?

resultado = regla_de_tres(0, 13, 100)
print 'Porcentaje de aprobados: %0.2f %%' % resultado


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
En cambio, si lo que queremos es saber es dónde se encuentra el número 3 en la lista es:
Can you write Python code for it?

lista = [11, 4, 6, 1, 3, 5, 7]

pos = lista.index(3)
print 'El 3 se encuentra en la posición', pos

pos = lista.index(15)
print 'El 15 se encuentra en la posición', pos


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Excepciones
Una excepción es la forma que tiene el intérprete de que indicarle al programador y/o usuario que ha ocurrido un error. Si la excepción no es controlada por el desarrollador ésta llega hasta el usuario y termina abruptamente la ejecución del sistema. <br>
Por ejemplo:
Can you write Python code for it?


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Funciones anónimas
Hasta ahora, a todas las funciones que creamos les poníamos un nombre al momento de crearlas, pero cuando tenemos que crear funciones que sólo tienen una línea y no se usen en una gran cantidad de lugares se pueden usar las funciones lambda:
Can you write Python code for it?

help("lambda")

mi_funcion = lambda x, y: x+y

resultado = mi_funcion(1,2)
print resultado


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Incluso, si queremos que los dos errores muestren el mismo mensaje podemos capturar ambas excepciones juntas:
Can you write Python code for it?

def dividir_numeros(x, y):
    try:
        resultado = x/y
        print 'El resultado es: %s' % resultado
    except (ZeroDivisionError, TypeError):
        print 'ERROR: No se puede calcular la división'

dividir_numeros(1, 0)
dividir_numeros(10, 2)
dividir_numeros("10", 2)


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Jerarquía de excepciones
Existe una jerarquía de excepciones, de forma que si se sabe que puede venir un tipo de error, pero no se sabe exactamente qué excepción puede ocurrir siempre se puede poner una excepción de mayor jerarquía:

Por lo que el error de división por cero se puede evitar como:
Can you write Python code for it?

try:
    print 1/0
except ZeroDivisionError:
    print 'Ha ocurrido un error de división por cero'


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Lanzar excepciones
Hasta ahora vimos cómo capturar un error y trabajar con él sin que el programa termine abruptamente, pero en algunos casos somos nosotros mismos quienes van a querer lanzar una excepción. Y para eso, usaremos la palabra reservada raise:
Can you write Python code for it?

def dividir_numeros(x, y):
    if y == 0:
        raise Exception('Error de división por cero')

    resultado = x/y
    print 'El resultado es {0}'.format(resultado)

try:
    dividir_numeros(1, 0)
except ZeroDivisionError as e:
    print 'ERROR: División por cero'
except Exception as e:
    print 'ERROR: ha ocurrido un error del tipo Exception'

print '----------'
dividir_numeros(1, 0)


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Pero en este caso igual muestra 0, por lo que si queremos, podemos poner los try/except incluso más arriba en el stacktrace:
Can you write Python code for it?

def dividir(x, y):
    return x/y

def regla_de_tres(x, y, z):
    return dividir(z*y, x)

try:
    print regla_de_tres(0, 1, 2)
except ZeroDivisionError:
    print 'No se puede calcular la regla de tres porque el divisor es 0'


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Pero no hay que tenerle miedo a las excepciones, sólo hay que tenerlas en cuenta y controlarlas en el caso de que ocurran:
Can you write Python code for it?

dividendo = 1
divisor = 0
print 'Intentare hacer la división de %d/%d' % (dividendo, divisor)
try:
    resultado = dividendo / divisor
    print resultado
except ZeroDivisionError:
    print 'No se puede hacer la división ya que el divisor es 0.'


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Pero supongamos que implementamos la regla de tres de la siguiente forma:
Can you write Python code for it?

def dividir(x, y):
    return x/y

def regla_de_tres(x, y, z):
    return dividir(z*y, x)

# Si de 28 alumnos, aprobaron 15, el porcentaje de aprobados es de...
porcentaje_de_aprobados = regla_de_tres(28, 15, 100)
print 'Porcentaje de aprobados: %0.2f %%' % porcentaje_de_aprobados


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Pero, ¿y cómo hacemos para ordenarla de mayor a menor?. <br>
Simple, interrogamos un poco a la función:
```Python

print sorted.doc
sorted(iterable, cmp=None, key=None, reverse=False) --> new sorted list
``
Entonces, con sólo pasarle el parámetro de *reverse* enTrue` debería alcanzar:
Can you write Python code for it?

lista_de_numeros = [1, 6, 3, 9, 5, 2]
print sorted(lista_de_numeros, reverse=True)


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Si bien siempre se puede poner Exception en lugar del tipo de excepción que se espera, no es una buena práctica de programación ya que se pueden esconder errores indeseados. Por ejemplo, un error de sintaxis.
Además, cuando se lanza una excepción en el bloque try, el intérprete comienza a buscar entre todas cláusulas except una que coincida con el error que se produjo, o que sea de mayor jerarquía. Por lo tanto, es recomendable poner siempre las excepciones más específicas al principio y las más generales al final:
Python
def dividir_numeros(x, y):
    try:
        resultado = x/y
        print 'El resultado es: %s' % resultado
    except TypeError:
        print 'ERROR: Ha ocurrido un error por mezclar tipos de datos'
    except ZeroDivisionError:
        print 'ERROR: Ha ocurrido un error de división por cero'
    except Exception:
        print 'ERROR: Ha ocurrido un error inesperado'
Si el error no es capturado por ninguna clausula se propaga de la misma forma que si no se hubiera puesto nada.
Otras cláusulas para el manejo de excepciones
Además de las cláusulas try y except existen otras relacionadas con las excepciones que nos permiten manejar de mejor manera el flujo del programa:
* else: se usa para definir un bloque de código que se ejecutará sólo si no ocurrió ningún error.
* finally: se usa para definir un bloque de código que se ejecutará siempre, independientemente de si se lanzó una excepción o no.
Can you write Python code for it?

def dividir_numeros(x, y):
    try:
        resultado = x/y
        print 'El resultado es {}'.format(resultado)
    except ZeroDivisionError:
        print 'Error: División por cero'
    else:
        print 'Este mensaje se mostrará sólo si no ocurre ningún error'
    finally:
        print 'Este bloque de código se muestra siempre'

dividir_numeros(1, 0)
print '-------------'
dividir_numeros(10, 2)


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Todos los casos son distintos y no hay UN lugar ideal dónde capturar la excepción; es cuestión del desarrollador decidir dónde conviene ponerlo para cada problema. <br>
Incluso, una única línea puede lanzar distintas excepciones, por lo que capturar un tipo de excepción en particular no me asegura que el programa no pueda lanzar un error en esa línea que supuestamente es segura:
Capturar múltiples excepciones
En algunos casos tenemos en cuenta que el código puede lanzar una excepción como la de ZeroDivisionError, pero eso puede no ser suficiente:
Can you write Python code for it?

def dividir_numeros(x, y):
    try:
        resultado = x/y
        print 'El resultado es: %s' % resultado
    except ZeroDivisionError:
        print 'ERROR: Ha ocurrido un error por mezclar tipos de datos'

dividir_numeros(1, 0)
dividir_numeros(10, 2)
dividir_numeros("10", 2)


I am working on the file ".ipynb_checkpoints/Clase 04 - Excepciones, funciones lambda, búsquedas y ordenamientos-checkpoint.ipynb".
The first task is:
Y también como:
Can you write Python code for it?

try:
    print 1/0
except Exception:
    print 'Ha ocurrido un error inesperado'


I am working on the file ".ipynb_checkpoints/Clase 05 - Manejo de strings y Archivos de texto -checkpoint.ipynb".
The first task is:
Escritura de archivos
Para escribir en un archivo podemos usar las las primitivas write(string) y writelines(lista_strings), que la primera es para escribir una cadena de caracteres y la segunda para escribir una lista de strings, uno a continuación del otro. Es importante destacar que en ningún caso se escribe algún carácter que no figure en los strings, como por ejemplo, caracteres de fin de línea.

El uso de writelines es equivalente a recorrer la lista y hacerle un write a cada elemento.

Pero el costo de escribir algo en el disco es mucho mayor a escribirlo en memoria por lo que, al igual que en C, se usa un buffer, que no es más que una porción de memoria para ir guardando en forma temporal los datos y cuando alcanzan un tamaño considerable se lo manda a escribir al disco. Otra forma de asegurarse que se haga la escritura es usando la primitiva flush, la cual guarda en el disco el contenido del buffer y lo vacía.

Can you write Python code for it?

arch2 = open("ejemplo2.txt", "w")
arch2.write("Es la primer cadena")
arch2.write("Seguida de la segunda con un fin de linea\n")
arch2.writelines(["1. Primero de la lista sin fin de línea. ", "2. Segundo string con fin de línea.\n", "3. Tercero con/\n.\n", "4. y último."])
arch2.flush()
arch2.close()
arch2 = open("ejemplo2.txt", "r+a")
strfile = arch2.read()
print strfile


I am working on the file ".ipynb_checkpoints/Clase 05 - Manejo de strings y Archivos de texto -checkpoint.ipynb".
The first task is:
Otra forma de asegurarse que se escriba lo que hay en el disco es cerrándolo.
Moverse en un archivo
Al igual que en los archivos binarios de Pascal, en Python también podemos saltar a distintas posiciones mediante la primitiva seek(pos) la cual recibe, como mínimo un parámetro que indica la posición a la que nos queremos mover. Opcionalmente puede recibir un segundo parámetro:
* 0: La posición es desde el inicio del archivo y debe ser mayor o igual a 0
* 1: La posición es relativa a la posición actual; puede ser positiva o negativa
* 2: La posición es desde el final del archivo, por lo que debe ser negativa
Can you write Python code for it?

arch = open("ejemplo.txt")
arch.seek(30)        # Voy a la posición número 30 del archivo
print arch.read(7)   # Debería salir 'y 1990s'
arch.seek(-5,1)      # Me muevo 5 posiciones para atrás desde mi posición actual.
print arch.read(7)   # Debería imprimir '1990s b'
arch.seek(-12,2)     # Me muevo a la posición número 12, comenzando a contar desde el final.
print arch.read(10)  # Debería imprimir 'compatible'

arch.close()


I am working on the file ".ipynb_checkpoints/Clase 05 - Manejo de strings y Archivos de texto -checkpoint.ipynb".
The first task is:
Otra primitiva que podemos usar es readline, que al igual que read, también puede recibir un parámetro que indique la cantidad máxima de bytes a leer. Si no se le pasa ningún parámetro, lee toda la línea.
Can you write Python code for it?

arch = open("ejemplo.txt")
linea = arch.readline()  # Notar que también imprime el Enter o \n
print linea
linea = arch.readline(7)  # La segunda línea es 'Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands'
print linea
arch.close()


I am working on the file ".ipynb_checkpoints/Clase 05 - Manejo de strings y Archivos de texto -checkpoint.ipynb".
The first task is:
Pero no es necesario que leamos de a una sola línea, sino que también podemos leer todas las líneas del archivo y guardarlas en una lista haciendo uso de la primitiva readlines.
Can you write Python code for it?

arch = open("ejemplo.txt")
lineas = arch.readlines()
print lineas
arch.close()


I am working on the file ".ipynb_checkpoints/Clase 05 - Manejo de strings y Archivos de texto -checkpoint.ipynb".
The first task is:
Sin embargo, la forma más Pythonic de leer el archivo por líneas es usando la estructura for y quedaría casi como lo diríamos en castellano: "Para cada línea del archivo. <br>
Por ejemplo, si queremos imprimir la cantidad de caracteres de cada línea podríamos hacer:
Can you write Python code for it?

arch = open("ejemplo.txt")
for linea in arch:
    print len(linea)

arch.close()


I am working on the file ".ipynb_checkpoints/Clase 05 - Manejo de strings y Archivos de texto -checkpoint.ipynb".
The first task is:
Y así como podemos movernos en un archivo, también podemos averiguar nuestra posición usando la primitiva tell().
Can you write Python code for it?

arch = open("ejemplo.txt")
arch.seek(30)
print arch.tell()    # Debería imprimir 30
arch.seek(-5,1)      # Retrocedo 5 posiciones
print arch.tell()    # Debería imprimir 25
arch.seek(-12,2)     # Voy a 12 posiciones antes del fin de archivo
print arch.tell()    # Debería imprimir 1132
print arch.read(10)  # Leo 10 caracteres
print arch.tell()    # Debería imprimir 1142


I am working on the file ".ipynb_checkpoints/Clase 05 - Manejo de strings y Archivos de texto -checkpoint.ipynb".
The first task is:
Y también podemos separar y combinar strings:
Can you write Python code for it?

print "Hola mundo".split()
print "Hola mundo".split('o')
print "Hola mundo".split('mu')
print ''.join(['Hola', 'mundo'])
print ' '.join(['Hola', 'mundo'])
var = '#separador#'.join(['Hola', 'mundo'])
var2 = var*2
print var2


I am working on the file ".ipynb_checkpoints/Clase 05 - Manejo de strings y Archivos de texto -checkpoint.ipynb".
The first task is:
¿Y qué pasa si le quiero agregar algunas líneas a este archivo?
Can you write Python code for it?

arch2.write("Esto lo estoy agregando.\n.")
arch2.writelines("Y estas dos líneas también con un \\n al final\n de cada una.\n")
arch2.flush()
arch2 = open("ejemplo2.txt", "r")  # El open hace que me mueva a la primer posición del archivo.
print arch2.read()
arch2.close()


I am working on the file ".ipynb_checkpoints/Clase 06 - Archivos binarios, Apareo de archivos-checkpoint.ipynb".
The first task is:
Para leer de un archivo pickle no puedo usar el método readline que usa la estructura for, por lo que no me queda otra que siempre intentar leer y cuando lance una excepción del tipo EOFError dejar de hacerlo.
Can you write Python code for it?

with open('ejemplo.pkl', 'rb') as archivo:
    seguir_leyendo = True
    while seguir_leyendo:
        try:
            data = pickle.load(archivo)  # Leo del archivo un elemento
        except EOFError:
            seguir_leyendo = False
        else:
            print '### Esta línea no es del archivo ###'
            print data


I am working on the file ".ipynb_checkpoints/Clase 06 - Archivos binarios, Apareo de archivos-checkpoint.ipynb".
The first task is:
Si se fijan, no hay ningún close en esa porción de código, pero sin embargo al salir del bloque que encierra el with, el archivo se encuentra cerrado sin importar si salió exitosamente o con una excepción
Pickles
Los pickles son una forma de guardar estructuras de datos complejas y recuperarlas fácilmente, sin necesidad de convertirlas a texto y luego parsearlas:
Ejemplo 1: Guardar de a un elemento
Can you write Python code for it?

import pickle  # Importo la biblioteca necesaria

# Creo la variable archivo
with open('ejemplo.pkl', 'wb') as archivo:
    pkl = pickle.Pickler(archivo)  # Creo mi punto de acceso a los datos a partir del archivo

    lista1 = [1, 2, 3]
    lista2 = [4, 5]
    diccionario = {'campo1': 1, 'campo2': 'dos'}

    pkl.dump(lista1)         # Guardo la lista1 de [1, 2, 3]
    pkl.dump(None)           # Guardo el valor None
    pkl.dump(lista2)
    pkl.dump('Hola mundo')
    pkl.dump(diccionario)
    pkl.dump(1)


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
Actors
Can you write Python code for it?

nz_actors_count = content_stats.Title_stats(nz_data).top_actors(21)
us_actors_count = content_stats.Title_stats(us_data).top_actors(21)

print '
'
print '
Actor Name	Count of Titles with Actor Name
'

for tup in nz_actors_count[1:]:
    print '
'
    print '        '
    print '            ', tup[0].lstrip()
    print '        	'
    print '    '
    print '              ', tup[1]
    print '        	'
    print '
'
print '
'

print
print '====================='

print '
'
print '
Actor Name	Count of Titles with Actor Name
'
for tup in us_actors_count[1:]:
    print '
'
    print '        '
    print '            ', tup[0].lstrip()
    print '        	'
    print '    '
    print '              ', tup[1]
    print '        	'
    print '
'

print '
'

print '
'
print '
	NZ Service		US Service
'
print '
Rank	Actor Name	Count of Titles with Actor Name	Actor Name	Count of Titles with Actor Name
'

for x in range(1,len(nz_actors_count)):
    print '
'
    print ''
    print str(x)
    print '	'
    print ''
    print nz_actors_count[x][0].lstrip()
    print '	'
    print ''
    print nz_actors_count[x][1]
    print '	'
    print ''
    print us_actors_count[x][0].lstrip()
    print '	'
    print ''
    print us_actors_count[x][1]
    print '	'
    print '
'

print '
'

print '
'
print '
New Zealand Service
'
print '
Movie Name	IMDb Score
'
nzx = []
for tup in top_nz_titles[:21]:
    print '
'
    print '',tup[0],'	','', tup[1],'	'
    print '
'
    nzx.append(tup[0])

print '
'

print
print
print '==============='

print '
'
print '
Movie Name	IMDb Score
'
usx = []
for tup in...


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
But we can look at them side by side.  Lets look at absolute first.
Can you write Python code for it?

release_year_us = content_stats.Title_stats(us_data).year_first_release_count()

us_x_list = []
us_y_list = []

for year, count in release_year_us.iteritems():
    us_x_list.append(year)
    us_y_list.append(count)

release_year_nz = content_stats.Title_stats(nz_data).year_first_release_count()

nz_x_list = []
nz_y_list = []

for year, count in release_year_nz.iteritems():
    nz_x_list.append(year)
    nz_y_list.append(count)

trace1 = (
    Bar( x = us_x_list,
          y = us_y_list,
        name = 'USA',
            marker = Marker(
            color = 'rgba(34, 95, 250, 0.6)')
        )
    )

trace2 = (
    Bar( x = nz_x_list,
          y = nz_y_list,
        name = 'NZ',
            marker = Marker(
            color = 'rgba(255, 144, 33, 0.6)')
        )
    )

layout = Layout(
    title ='Netflix - Year of Titles First Release - US and NZ June 2015',
    yaxis = YAxis(title = 'Count of Titles'),
    xaxis = XAxis(title = 'Year'),
    barmode='group'

    )

data = Data([trace1, trace2])

fig = Figure(data=data, layout=layout)

#py.iplot(fig, filename = 'Netflix-Library-Comparison-Release-US-NZ-June-2015')

HTML('''

    Link to interactive chart and data

''')


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
It appears the NZ library, while smaller has a greater proportion of higher quality content.
New Zealand as Country of Origin
We can see what New Zealand content is represented within our samples. This includes anything that has the identifier New Zealand in 'Country' via IMDb.  This can include co-productions.
Can you write Python code for it?

# test
nz_origin_nz = content_stats.Title_stats(nz_data).nz_origin()
nz_origin_us = content_stats.Title_stats(nz_data).nz_origin()

data = (
    [Bar( x = ['NZ', 'USA'],
          y = [nz_origin_nz, nz_origin_us],
            marker = Marker(
            color = 'rgba(34, 95, 250, 0.6)')
        )]
    )

layout = Layout(
    title ='Netflix - Count of Titles with Country of Origin as New Zealand via IMDb',
    yaxis = YAxis(title = 'Count of Titles'),
    xaxis = XAxis(title = 'Geographic Service'),

    )

fig = Figure(data=data, layout=layout)

# py.iplot(fig, filename = 'Netflix-Library-Comparison-Country-June-2015')

HTML('''

    Link to interactive chart and data

''')


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
It seems NZ might have a smaller library but better quality (based on IMDB rankings).  Though on an absolute value this makes some sense.  We can look at a distribution to get a better understanding.
Can you write Python code for it?

hist_nz_titles = content_stats.Title_stats(nz_data).ratings_distribution()
hist_us_titles = content_stats.Title_stats(us_data).ratings_distribution()

trace1 = Histogram(
    x = hist_nz_titles,
    opacity=0.75,
    name = 'Count of NZ Titles'
)
trace2 = Histogram(
    x = hist_us_titles,
    opacity = 0.75,
    name = 'Count of US Titles'
)
data = Data([trace2, trace1])
layout = Layout(
    barmode ='overlay',
    yaxis = YAxis(title = 'Count of Titles'),
    xaxis = XAxis(title = 'IMdB Score (rounded down)'),
    title = 'Netflix Library Comparison June 2015 Distribution of IMDB scores'
)
fig = Figure(data=data, layout=layout)

# py.iplot(fig, filename = 'Netflix-Library-Comparison-June-2015-Distribution of IMDB scores')

HTML('''

    Link to interactive chart and data

''')

titles_score_count_nz = Counter(hist_nz_titles)
titles_score_count_us = Counter(hist_us_titles)

# make relative
nz_tot = sum(titles_score_count_nz.values())
titles_score_count_nz_relative = { k:round((float(v)/float(nz_tot))*100, 1) for (k,v) in titles_score_count_nz.items()}

us_tot = sum(titles_score_count_us.values())
titles_score_count_us_relative = { k:round((float(v)/float(us_tot))*100,1) for (k,v) in titles_score_count_us.items()}

nz_x_list = []
nz_y_list = []

for score, count in titles_score_count_nz_relative.iteritems():
    nz_x_list.append(score)
    nz_y_list.append(count)

us_x_list = []
us_y_list = []

for score, count in...


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
Neck and neck.  We can see what these titles are, we can get a feel as to how Kiwi they are.
Age of Titles
We can look at age of titles within the catalogues.  There are some interesting caveats to this as age is represented in three ways:
* A single year - this could be a movie that came out in a single year or a series that only ran within a single calendar year
* Across multiple years, for example a series that has run across multiple years (e.g. Friends, 1994–2004)
* Something that is still running (e.g. Orange is the New Black, 2013–)
Initially we could look at year of first release.
Can you write Python code for it?

release_year = content_stats.Title_stats(nz_data).year_first_release_count()

x_list = []
y_list = []

for year, count in release_year.iteritems():
    x_list.append(year)
    y_list.append(count)

data = (
    [Bar( x = x_list,
          y = y_list,
            marker = Marker(
            color = 'rgba(34, 95, 250, 0.6)')
        )]
    )

layout = Layout(
    title ='Netflix - Year of Titles First Release - NZ June 2015',
    yaxis = YAxis(title = 'Count of Titles'),
    xaxis = XAxis(title = 'Year'),

    )

fig = Figure(data=data, layout=layout)

#py.iplot(fig, filename = 'Netflix-Library-Comparison-Release-June-2015')

HTML('''

        Link to interactive chart and data

''')


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
New Zealand may have a smaller catalgoue but it does seem to have a marginally higher average quality than the US.  We can look at what the top movies are (based on IMDb/OMDb ratings) between the two countries.
Top Movies
This give an interesting understanding of the top movies.  At first look we can se the titles are very different.
Can you write Python code for it?

top_nz_titles = content_stats.Title_stats(nz_data).top_movies(25)
top_us_titles = content_stats.Title_stats(us_data).top_movies(25)

print 'Top NZ Titles'
print

# Truncate to 5
for tup in top_nz_titles[:5]:
    print 'Title :', tup[0]
    print 'Rating: ', tup[1]
    print '========='

print
print
print 'Top US Titles'
print

# Truncate to 5
for tup in top_us_titles[:5]:
    print 'Title :', tup[0]
    print 'Rating: ', tup[1]
    print '========='

print
print

print 'Average score of top 25 Titles'
print
print 'NZ top 25 average'

score = 0
count = 0
for tup in top_nz_titles:
    count += 1
    score += tup[1]
print score/count

print

print 'US top 25 average'
score = 0
count = 0
for tup in top_us_titles:
    count += 1
    score += tup[1]
print score/count


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
Quality of Content
Quality by definition is qualitive. Though with enough measure of quality we can hopefully have some quantitive measure of quality, the same way we can hopefully say a five star hotel is normally going to better than a one star hotel.  We know this is not always the case and we do have the Napoleon Dynamite effect where the hate and love for a title can be strong.
To assess quality we looked looked towards IMDB, who make some of their data available via alternative interfaces though not in a structured API.  Luckily OMDB offer a RESTful API that allows querying by title or ID, and returns XML or JSON.
We used this to query the title against OMDB.  Not all returned a useful response, Doctor Who fans will be pleased to know 1995's made for TV movie was not recognised when we queried.
We can look at what we did not get a response for, this is useful, it does not tell us where we got a false response, but we're hoping there were not too many of those.
Can you write Python code for it?

# Count of titles we did not get a response for
nz_no_response = len([k for (k, v) in nz_data.iteritems() if v['Response'] == 'False'])
print 'Count of NZ titles we did not get a reponse for: ', nz_no_response
print 'Percentage of NZ sites: ', round(float(nz_no_response)/float(len(nz_data))*100), '%'

#US count?


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
The Bieber Effect
Across the top 25 both NZ and US are close, with the NZ library taking a slight lead.  How do we compare in terms of the bottom ranked content.
Can you write Python code for it?

bottom_nz_titles = content_stats.Title_stats(nz_data).bottom_movies(25)
bottom_us_titles = content_stats.Title_stats(us_data).bottom_movies(25)

print 'Bottom ranked NZ title', bottom_nz_titles[-1][0], ':' , bottom_nz_titles[-1][1]

poster = Image(nz_data[quote(bottom_nz_titles[-1][0])]['Poster'])
display(poster)

print 'Bottom ranked US title', bottom_us_titles[-1][0], ':' , bottom_us_titles[-1][1]

poster = Image(us_data[quote(bottom_us_titles[-1][0])]['Poster'])
display(poster)

print
print

print 'Average score of bottom 25 Titles'
print
print 'NZ bottom 25 average'

score = 0
count = 0
for tup in bottom_nz_titles:
    count += 1
    score += tup[1]
print score/count

print

print 'US bottom 25 average'
score = 0
count = 0
for tup in bottom_us_titles:
    count += 1
    score += tup[1]
print score/count


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
The smaller New Zealand library makes this a bit tricker to understand which service has newer and older content when put against the larger US library.
Can you write Python code for it?

total_us_titles = sum(us_y_list)
us_y_list = [(float(count)/(float(total_us_titles)) * 100) for count in us_y_list]

total_nz_titles = sum(nz_y_list)
nz_y_list = [(float(count)/(float(total_nz_titles)) * 100) for count in nz_y_list]

trace1 = (
    Bar( x = us_x_list,
          y = us_y_list,
        name = 'USA',
            marker = Marker(
            color = 'rgba(34, 95, 250, 0.6)')
        )
    )

trace2 = (
    Bar( x = nz_x_list,
          y = nz_y_list,
        name = 'NZ',
            marker = Marker(
            color = 'rgba(255, 144, 33, 0.6)')
        )
    )

layout = Layout(
    title ='Netflix - Year of Titles First Release - US and NZ June 2015 - Percentage',
    yaxis = YAxis(title = 'Percentage of Titles'),
    xaxis = XAxis(title = 'Year'),
    barmode='group'

    )

data = Data([trace1, trace2])

fig = Figure(data=data, layout=layout)

py.iplot(fig, filename = 'Relative- Netflix-Library-Comparison-Release-US-NZ-June-2015')


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
Uniqueness of Content
Further up you would notice we imported a python module called 'content_stats'.  This lets us apply some more exploratory stats.
The uniqueness of content between libraries has been questioned by several people.
Can you write Python code for it?

# Number of titles are common
common = len(content_stats.Compare_regions(us_data, nz_data).common_titles())
print 'Titles in common between USA and NZ:', common

# Number of titles unique to us
unique_us = len(content_stats.Compare_regions(us_data, nz_data).unique_to_first())
print 'Titles Unique to the USA :', unique_us

# Number of titles unique to nz
unique_nz = len(content_stats.Compare_regions(nz_data, us_data).unique_to_first())
print 'Titles Unique to the NZ :', unique_nz


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
We can represent this graphically again.
Can you write Python code for it?

trace1 = Bar(
        y=['Service'],
        x=[unique_nz],
        name='Unique to NZ',
        orientation = 'h',

        marker = Marker(
            color = 'rgba(255,127,14,1.0)'        )

        )

trace2 = Bar(
        y=['Service'],
        x=[common],
        name='Common',
        orientation = 'h',
            marker = Marker(
            color = 'rgba(44,160,44,1.0)'        )

        )

trace3 = Bar(
        y = ['Service'],
        x = [unique_us],
        name = 'Unique to USA',
        orientation = 'h',
            marker = Marker(
            color = 'rgba(39,119,180,1.0)'        )

        )

data = Data([trace1, trace2, trace3])
layout = Layout(
    barmode='stack',
    title = 'Content Unique to and Common Between US and NZ Netflix Services - June 2015',
    yaxis = YAxis(title = ''),
    xaxis = XAxis(title = 'Count of Titles')
    )

fig = Figure(data=data, layout=layout)

# used to create
#py.iplot(fig, filename = 'Netflix-Library-Comparison-June-2015-Uniqueness of Content')

# used to embed

HTML('''

    Link to interactive chart and data

''')


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
We could potentially display this visually.
We need to do some trickery as our dictionaries have quoted title names as the keys, so 'The%20Pink%20Panther' instead of 'The Pink Panther', so we need to 'quote' our moviename to get an image.
Top NZ Title Visually
Can you write Python code for it?

#truncate to top 5
for tup in top_nz_titles[:5]:
    print 'Title :', tup[0]
    print 'Title :', tup[1]
    try:
        poster = Image(nz_data[quote(tup[0])]['Poster'])
        display(poster)
    except:
        print '''
        |
        | No Poster
        |
        '''


I am working on the file ".ipynb_checkpoints/Content Analysis-checkpoint.ipynb".
The first task is:
We're pretty happy with these percentages for getting an understanding of the content that is available from a quality perpective.  We might not be able to be absolutely, absolute, but good enough.
We can now look at the difference in quality.
In our assessment of quality we are not looking at Netflix ratings, we are looking at the ratings held in IMDb.  It could be that Netflix have content more suited to its customers and they would rate titles higher.  Those rating IMDb may be skewed in a particular way as they may have more interest in esoteric aspects of movies and content.  Perhaps another useful metric would be Rotten Tomatoes scores, though we don't have complete information on this and we were declied access to the Rotten Tomatoes API.
Average Score
Can you write Python code for it?

# Average IMDB score of NZ geographic content
nz_avg_score = content_stats.Title_stats(nz_data).average_score()

# Average IMDB score of NZ geographic content
us_avg_score = content_stats.Title_stats(us_data).average_score()

print 'NZ Average Score via OMDB: ', round(nz_avg_score,2)
print 'US Average Score via OMDB: ', round(us_avg_score,2)


I am working on the file ".ipynb_checkpoints/Crowdstorming_visualisation-checkpoint.ipynb".
The first task is:
5 Let's look at the implicit and explicit racism scores, and their sample size distributions
Most of the refs come from 4 countries (where the 4 represented leagues are). It's a shame there is no data from Italy but never mind. It's hard to know how reliable the IAT and Exp scores are with respect to the other countries, esp if there is some variation between ref biases within a country. The IAT and Exp scores might not have as much variation as within country variation..
There doesn't seem to be much 'signal' here. Also the sample sizes, though large, are then undermined by the small sample sizes for refs in some countries (data not shown, but see above)
Can you write Python code for it?

# Linked subplots showing these distributions, or a scatter plot with variable dot sizes showing how much overlap there is.
fig, ax = plt.subplots(2,2,figsize=(12, 8))
ax[0,0].hist(dfd.meanIAT.dropna().unique())
ax[0,0].set_title("Mean IAT - all")
ax[0,1].hist(dfd['meanExp'].dropna().unique())
ax[0,1].set_title("Mean Exp - all")

ax[1,0].hist(dfd_good.meanIAT.dropna().unique())
ax[1,0].set_title("Mean IAT - culled")
ax[1,1].hist(dfd_good['meanExp'].dropna().unique())
ax[1,1].set_title("Mean Exp - culled")

sns.jointplot(dfd_good['meanIAT'].dropna().unique(),dfd_good['meanExp'].dropna().unique())


I am working on the file ".ipynb_checkpoints/Crowdstorming_visualisation-checkpoint.ipynb".
The first task is:
A referee-player dyad describes the interactions between a particular ref and one player. This means that each row in the dataset is of a unique player-ref combination, listing all of the games by a given player with a particular referee at any point in his career. Let's look at the first few rows of the dataset as an example:
Can you write Python code for it?

# Display the first 10 rows of the dataset. Only 13 columns for space reasons
df.ix[:10,:13]

# Display the other columns too
df.ix[:10,13:28]


I am working on the file ".ipynb_checkpoints/Crowdstorming_visualisation-checkpoint.ipynb".
The first task is:
By default the data is in a rather counter-intuitive format: referee-player dyads. What is a dyad..?
Can you write Python code for it?

from IPython.display import HTML
HTML('<iframe src=http://en.wikipedia.org/wiki/Dyad_(sociology) width=1000 height=350></iframe>')


I am working on the file ".ipynb_checkpoints/Crowdstorming_visualisation-checkpoint.ipynb".
The first task is:
Let's now look at the countries referees come from. There are a lot of them
Can you write Python code for it?

import mpld3
mpld3.enable_notebook()
# Histogram of country frequency.
fig, ax = plt.subplots(1,1,figsize=(12, 4))
x = dfd.Alpha_3.value_counts()
lines = ax.plot(x,marker='.',ms=20)

y = x.index.tolist()

tooltips = mpld3.plugins.PointLabelTooltip(lines[0], labels=y)
mpld3.plugins.connect(plt.gcf(), tooltips)

ax.set_title('Referee nationality by number of games')
ax.set_xlabel('Country number (ordered by frequency)')
ax.set_ylabel('Frequency of games')
ax.set_xlim([-3,160]) # a hack so we can see the first point most clearly

mpld3.disable_notebook()


I am working on the file ".ipynb_checkpoints/Crowdstorming_visualisation-checkpoint.ipynb".
The first task is:
Now we need some data.
Can you write Python code for it?

# Import original data, then do a bit of data-munging to get it in displayable form.
print "loading data from file"
filename=os.path.join('data','CrowdstormingDataJuly1st.csv')
df = pd.read_csv(filename)


I am working on the file ".ipynb_checkpoints/Crowdstorming_visualisation-checkpoint.ipynb".
The first task is:
Now, that at least makes sense. We see that many England Internationals, and two World Cup winning German Internationals (each with long careers in top-flight football) have played with the same referees many times.
By contrast, how rare are red cards?
Can you write Python code for it?

print "Total interactions =", len(dfd)
clean_interactions = dfd[(dfd['allreds'] == 0)]
print "Number of red cards in the dataset =", len(dfd) - len(clean_interactions)
print "Number of interactions without a red card =", len(clean_interactions)

print "Proportion of interactions that are 'clean' =", len(clean_interactions) / float(len(dfd))


I am working on the file ".ipynb_checkpoints/Crowdstorming_visualisation-checkpoint.ipynb".
The first task is:
Scatter plot of skintone ratings. Jitter added for visualisation
Note: This cell takes quite a long time (minutes) to run
Can you write Python code for it?

c = sns.color_palette()
jitter_x = np.random.normal(0, 0.04, size=len(dfd.rater1))
jitter_y = np.random.normal(0, 0.04, size=len(dfd.rater2))
sns.jointplot(dfd.rater1 + jitter_x, dfd.rater2 + jitter_y, kind='kde')


I am working on the file ".ipynb_checkpoints/Crowdstorming_visualisation-checkpoint.ipynb".
The first task is:
Still 100+ nationalities represented
4 Skintone ratings could be better
We noted that ratings of skintone could be more reliable. The ratings are fairly different at the light end of the spectrum. The two raters disagree on 28742 or 19% of the time, and looking at the histograms of their responses most of these are between the first two categories. These first two categories account for ~ 70% of both rater's classifications, so biases/inconsistencies/uncertainty in this part of the dataset could have a large effect on the rest of the analysis. There could be many reasons for this, but one obvious way of dealing with it would be to use N>2 raters.
Can you write Python code for it?

# Plot of skin tone rating distributions, showing skewed nature of the data w/ histograms
# and degree of disagreement between raters with a scatterplot
rated = ((dfd['rater1']+dfd['rater2'])/2).dropna()

fig, ax = plt.subplots(1,4,figsize=(12, 4))
c = sns.color_palette()
ax[0].hist(rated,bins = 5, range = (0,1),color = c[0])
ax[0].set_title("Mean rating")

ax[1].hist(dfd['rater1'].dropna().tolist(),bins = 5, range = (0,1), color = c[1])
ax[1].set_title('Rater 1')

ax[2].hist(dfd['rater2'].dropna().tolist(),bins = 5, range = (0,1),color = c[2])
ax[2].set_title('Rater 2')

ax[3].hist((dfd['rater1'] - dfd['rater2']).dropna(), bins = 5,range = (-0.5,0.5),color = c[3])
ax[3].set_title('Difference')

fig.tight_layout()

print 'Mean skintone across the population =', np.mean(rated)


I am working on the file ".ipynb_checkpoints/Crowdstorming_visualisation-checkpoint.ipynb".
The first task is:
We already see some strange things in the data. Some refs officiated very few games. The two raters disagree about player skintone quite often. For some players there isn't a photo, so their skin tone couldn't be rated. Most dyads don't feature cards at all. In general, it's difficult to get an intuition for what the population looks like from inspecting small samples of the data. This is particularly difficult in the dyad format, as each player or ref's cards are spread un-evenly across the dataset.
2 Disaggregating the data into single player-ref interactions
Team Sheffield felt a more natural format for the data was to disaggregate it. In other words, to unpack each dyad into singular ref-player interaction. In other words, each time a player and a referee met they would contribute one row to the dataset (and thus that interaction has a maximum of one red card).
If you're interested in how the disaggregation was done, the code is in (our project folder on the OSF](https://osf.io/akqt4/) - the file is disaggregate_v3.py
Can you write Python code for it?

# Load disaggregated data
print "loading disaggregated data"
filename=os.path.join('data','crowdstorm_disaggregated.csv')
dfd = pd.read_csv(filename)


I am working on the file ".ipynb_checkpoints/Crowdstorming_visualisation-checkpoint.ipynb".
The first task is:
We now have a much more 'normal' dataset, where each data point accounts for a single interaction (i.e. a game) .
What is the distribution of individual ref-player dyad game numbers?
Can you write Python code for it?

# Pull out the number of games in each dyad and plot
games = dfd.games

fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(12, 4))

axes[0].hist(games,bins=max(games),histtype = 'stepfilled')
axes[0].set_xlabel('Number of interactions')
axes[0].set_ylabel('Frequency')

axes[1].hist(games,bins=max(games),histtype = 'stepfilled')
axes[1].set_yscale('symlog') # symetric log scale NOTE this breaks the nice plotting tools
axes[1].set_title("Log scaled")
axes[1].set_xlabel('Number of interactions')
axes[1].set_ylabel('Log Frequency')

fig.tight_layout()
# display_d3()

print "highest number of games in a single dyad = " + str(max(games))


I am working on the file ".ipynb_checkpoints/Crowdstorming_visualisation-checkpoint.ipynb".
The first task is:
We see that though most refs are only involved in a small number of dyads, many officiated over thousands. A median of 11 indicates that more than half of the refs officiated less than one game!
Something funny is going on - If a ref officiated a full game in one of our selected premier leagues they would be in at least 22 dyads (2 teams of 11 players each, more if substitutions occur).
Futher analysis (not shown), including a bit of eyeballing, revealled that players' entire career histories are included in the dataset. This means that if a player gets sent off in a Uruguayan league game in 2002, but then transfers to the English premier league by 2012 then this booking ends up in our dataset - as does the referee. This explains the high number of ref nationalities present in the data set
We clean the data by excluding interactions by refs who feature in fewer than 22 dyads
If you aren't in at least 22 dyads you didn't ref a game in one of our four defining leagues
Can you write Python code for it?

goodRefs = allRefs[allRefs>21]

fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(12, 4))
axes.hist(goodRefs.tolist(),numRefs-11)
axes.set_xscale('symlog') # symetric log scale
plt.xlim([1,10000])
axes.set_yscale('symlog')
plt.ylim([0,1000])
axes.set_title("Referee occurance following our cull, log scaled")
axes.set_xlabel('log (number of occurances)')
axes.set_ylabel('log (frequency)')

print "Number of refs featuring in at least 22 dyads =",len(goodRefs)
print "Number of dyads, excluding refs who officiate fewer than 22 games =", sum(goodRefs)


I am working on the file ".ipynb_checkpoints/Crowdstorming_visualisation-checkpoint.ipynb".
The first task is:
implicit (IAT) and explicit (EXP) attitude correlate highly
Can you write Python code for it?

x = dfd_good.nIAT.dropna().unique()
plt.plot(np.sort(x),'.')

plt.xlabel('Country number (ordered by sample size)')
plt.ylabel('Sample size for IAT score')

plt.ylim([0,3000])
plt.title('Most IAT scores are based on samples of <1000')


I am working on the file ".ipynb_checkpoints/Curso Introdução à Ciência da Computação com Python - Parte 2-checkpoint.ipynb".
The first task is:
Este código faz com que primeiramente toda a primeira linha seja preenchida, em seguida a segunda e assim sucessivamente. Se nós quiséssemos que a primeira coluna fosse preenchida e em seguida a segunda coluna e assim por diante, como ficaria o código?
Um exemplo: se o usuário digitasse o seguinte comando “x = cria_matriz(2,3)” e em seguida informasse os seis números para serem armazenados na matriz, na seguinte ordem: 1, 2, 3, 4, 5, 6; o x teria ao final da função a seguinte matriz: [[1, 3, 5], [2, 4, 6]].
Can you write Python code for it?

def cria_matriz(num_linhas, num_colunas):
    matriz = []  #lista vazia
    for i in range(num_linhas):
        linha = []
        for j in range(num_colunas):
            linha.append(0)
        matriz.append(linha)

    for i in range(num_colunas):
        for j in range(num_linhas):
            matriz[j][i] = int(input("Digite o elemento [" + str(j) + "][" + str(i) + "]: "))

    return matriz

x = cria_matriz(2, 3)

x

def tarefa(mat):
    dim = len(mat)
    for i in range(dim):
        print(mat[i][dim-1-i], end="  ")

mat = [[1,2,3],[4,5,6],[7,8,9]]
tarefa(mat)

# Observação: o trecho do print (end = " ") irá mudar a finalização padrão do print
# que é pular para a próxima linha. Com esta mudança, o cursor permanecerá na mesma
# linha aguardando a impressão seguinte.


I am working on the file ".ipynb_checkpoints/Curso Introdução à Ciência da Computação com Python - Parte 2-checkpoint.ipynb".
The first task is:
Exercício 1: Tamanho da matriz
Escreva uma função dimensoes(matriz) que recebe uma matriz como parâmetro e imprime as dimensões da matriz recebida, no formato iXj.
Exemplos:
minha_matriz = [[1], [2], [3]]
dimensoes(minha_matriz)
3X1
minha_matriz = [[1, 2, 3], [4, 5, 6]]
dimensoes(minha_matriz)
2X3
Can you write Python code for it?

def dimensoes(A):

    '''Função que recebe uma matriz como parâmetro e imprime as dimensões da matriz recebida, no formato iXj.

    Obs: i = colunas, j = linhas

    Exemplo:
    >>> minha_matriz = [[1],
                        [2],
                        [3]
                        ]
    >>> dimensoes(minha_matriz)
    >>> 3X1
    '''

    lin = len(A)
    col = len(A[0])

    return print("%dX%d" % (lin, col))

matriz1 = [[1], [2], [3]]
dimensoes(matriz1)

matriz2 = [[1, 2, 3], [4, 5, 6]]
dimensoes(matriz2)


I am working on the file ".ipynb_checkpoints/CythonBootstrap-checkpoint.ipynb".
The first task is:
Introducing runcython !!

Is located on Github
Easy installation == pip install runcython
Russell91 on Github

https://github.com/Russell91/runcython
There is a runcython and makecython function calls . . . . .
Can you write Python code for it?

!makecython ./src/cyFib.pyx

def pyfib(n):
    a = 0
    b = 1
    index = 0
    while b < n:
        a, b = b, a+b
        index += 1
    return b

%timeit pyfib(1000)

import cyFib
%timeit cyFib.cyfib(1000)


I am working on the file ".ipynb_checkpoints/CythonBootstrap-checkpoint.ipynb".
The first task is:
Now let's looking a data that involves arrays, and look at both python and numpy versions as well.
Can you write Python code for it?

%matplotlib inline

import numpy as np

x = np.linspace(0,2*np.pi,2000)
%timeit plot(x,np.sin(x),'r')

## %timeit plot(x,sineWave.sinewave(x),'r') <== Why is this a problem ??

xlim(0,6.28)
title('Sinewave for Array Data')
grid(True)

%%file ./src/myFunc.pyx

import cython
import numpy as np
cimport numpy as np

@cython.boundscheck(False)
@cython.wraparound(False)
def myfunc(np.ndarray[double, ndim=1] A):
    return np.sin(A)

!makecython ./src/myFunc.pyx

%matplotlib inline

from src import myFunc
import cython
import numpy as np

x = np.linspace(0,2*np.pi,2000)
y = myFunc.myfunc(x)

%timeit plot(x,y,'r')

xlim(0,6.28)
title('Sinewave for Array Data with Cython')
grid(True)


I am working on the file ".ipynb_checkpoints/CythonBootstrap-checkpoint.ipynb".
The first task is:
Now let's see the time difference between a cyfib and pyfib ...
Can you write Python code for it?

%%file ./src/cyFib.pyx
def cyfib(int n):
    cdef int a = 0
    cdef int b = 1
    cdef int index = 0
    while b < n:
        a, b = b, a+b
        index += 1
    return b


I am working on the file ".ipynb_checkpoints/CythonBootstrap-checkpoint.ipynb".
The first task is:
Now's lets do something graphically, like plot a trig function. Let's also use a float/double type.
Can you write Python code for it?

%%file ./src/sineWave.pyx
import cython
from libc.math cimport sin

def sinewave(double x):
    """ Calculate a sinewave for specified number of cycles, Ncycles, at a given frequency."""
    return sin(x)

!makecython ./src/sineWave.pyx

from src import sineWave
import math
angle90 = math.pi/2
sineWave.sinewave(angle90)


I am working on the file ".ipynb_checkpoints/DAT208x - Week 1 - Python Basics-checkpoint.ipynb".
The first task is:
When to use python? -- 50xp, Status : Earned

Python is a pretty versatile language. For what applications can you use Python?
Ans: All of the above

Any comments? -- 100xp, Satatus : Earned

We can add comments to python scripts.

Comments are short snippets of plain english, to help you and others understand what the code is about.

To add a comment, use '#'tag, insert it at the front of the text.

Comments have idle state, i.e. they don't affect the code results.

Comments are ignored by the python interpretor.
Can you write Python code for it?

# Just testing division
print(5 / 8)

# Additon works too ( added comment here )
print(7 + 10)


I am working on the file ".ipynb_checkpoints/DFORCE-checkpoint.ipynb".
The first task is:
FORCE does a pretty nice job learning how to be a sin wave. If you rerun this a few times, you'll see the quality of the fits varies. Such if woring with randomness and chaos.
Binary FORCE
Implementation
I first define a function that converts unit rates into a binary output. Then I replace the standard learning FORCE math to use the binary decoded response generated by this function.
Can you write Python code for it?

def decode(x, rho):
    xd = zeros_like(x)

    xd[x > rho] = 1
    xd[x < -rho] = -1

    return xd


I am working on the file ".ipynb_checkpoints/DFORCE-checkpoint.ipynb".
The first task is:
Here's when the binary version looks like. I use random (U) selected thresholds to convert from rates to binary codes. Don't have any idea how this really works, so random seems as good a guess as any.
That said a fixed threshold seems just as good.
I'm curious as to whether different thresholds for different training targets might let me multiplex many decodings onto the same pool?
Can you write Python code for it?

def f1(x,t0):
    return -x + g*dot(J,tanh(x))

N = 500
J = normal(0,sqrt(1/N),(N,N))
x0 = uniform(-0.5,0.5,N)
t = linspace(0,50,500)

rho = uniform(0,0.1,N)  # Rand thresholds!
# rho = 0.5  # fixed threshold!

plt.figure(figsize=(10,5))
for s,g in enumerate(linspace(0.5,1.5,3)):
    plt.subplot(1,3,s+1)
    x = odeint(f1,x0,t)
    xd = decode(x, rho)
    plt.plot(t,xd[:,choice(N,10)])
    plt.title('g = '+str(g),fontweight='bold')
    plt.ylim(-2,2)
plt.show()


I am working on the file ".ipynb_checkpoints/DFORCE-checkpoint.ipynb".
The first task is:
The binary loss function
Now let's learn, with binary codes.
Can you write Python code for it?

target = lambda t0: cos(2 * pi * t0 / 50)  # target pattern

def f3(t0, x):
    return -x + g * dot(J, tanh_x) + dot(w, tanh_x) * u

dt = 1       # time step
tmax = 800   # simulation length
tstop = 500

N = 300
J = normal(0, sqrt(1 / N), (N, N))
x0 = uniform(-0.5, 0.5, N)
t = linspace(0, 50, 500)

rho = 0.1  # Set and rand vec
rho = uniform(0, 0.1, N)

g = 1.5
u = uniform(-1, 1, N)
w = uniform(-1 / sqrt(N), 1 / sqrt(N), N)  # initial weights
P = eye(N)  # Running estimate of the inverse correlation matrix
lr = .4  # learning rate

rho = repeat(0.05, N)

# simulation data: state,
# output, time, weight updates
x, z, t, wu = [x0], [], [0], [0]

# Set up ode solver
solver = ode(f3)
solver.set_initial_value(x0)

# Integrate ode, update weights, repeat
while t[-1] < tmax:
    tanh_x = tanh(x[-1])
    tanh_xd = decode(tanh_x, rho)  # BINARY CODE INTRODUCED HERE!
    z.append(dot(w, tanh_xd))

    error = target(t[-1]) - z[-1]
    q = dot(P, tanh_xd)
    c = lr / (1 + dot(q, tanh_xd))
    P = P - c * outer(q, q)
    w = w + c * error * q

    # Stop training time
    if t[-1] > tstop:
        lr = 0

    wu.append(np.sum(np.abs(c * error * q)))

    solver.integrate(solver.t + dt)
    x.append(solver.y)
    t.append(solver.t)

# last update for readout neuron
z.append(dot(w, tanh_x))

# plot
x = np.array(x)
t = np.array(t)

plt.figure(figsize=(10, 5))
plt.subplot(2, 1, 1)
plt.plot(t, target(t), '-r', lw=2)
plt.plot(t, z, '-b')
plt.legend(('target', 'output'))
plt.ylim([-1.1, 3])
plt.xticks([])
plt.subplot(2, 1, 2)
plt.plot(t, wu, '-k')
plt.yscale('log')
plt.ylabel('$|\Delta w|$', fontsize=20)
plt.xlabel('time', fontweight='bold', fontsize=16)
plt.show()


I am working on the file ".ipynb_checkpoints/Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data-checkpoint.ipynb".
The first task is:
Cleaning Phone Numbers
Try it out: Use python project.py -C to clean in debug mode. See Sample Output in file Project/output_project.py_-C.txt. The script also writes a CSV file to Project/data/clean_phones.csv, which is also beautified into a Excel File.
Cleaning the phone numbers involves:
* unifying the different phone attributes (phone, address.phone and mobile_phone) - this is already taken care by extracting the phone numbers during the audit stage
* if possible, canonicalizing the phone number notations by parsing them using a regular expression:
python
phone_regex = re.compile(ur'^(\(?([\+|\*]|00) *(?P<country>[1-9][0-9]*)\)?)?' +  # country code
                         ur'[ \/\-\.]*\(?0?\)?[ \/\-\.]*' +  # separator
                         ur'(\(0?(?P<area1>[1-9][0-9 ]*)\)|0?(?P<area2>[1-9][0-9]*))?' +  # area code
                         ur'[ \/\-\.]*' +  # separator
                         ur'(?P<number>([0-9]+ *[\/\-.]? *)*)$', # number
                         re.UNICODE)
The regular expression is resilient to various separators ("/", "-", " ", "(0)") and bracket notation of phone numbers. It is not resilient for some unicode characters or written lists of phone numbers which are designed to be interpreted by humans (using separators like ",", "/-" or "oder" lit. or). During the cleaning stage, an output is written which phone numbers could not be parsed. This contains only a tiny fraction of phone numbers (9 or 0.5%) which would be easily cleanable by hand.
The following objects couldn't be parsed:
                                                 normalized
55f57294b1c8a72c34523897           +49 35207 81429 or 81469
55f57299b1c8a72c345272cd  +49 351 8386837, +49 176 67032256
55f572c2b1c8a72c34546689                      0351 4810426
55f572c3b1c8a72c34546829         +49 351 8902284 or 2525375
55f572fdb1c8a72c34574963   +49 351 4706625, +49 351 0350602
55f573bdb1c8a72c3460bdb3                +49 351 87?44?44?00
55f573bdb1c8a72c3460c066         0162 2648953, 0162...
Can you write Python code for it?

from Project.notebook_stub import project_coll

# Query used - see function: Project.audit_quality_map.audit_streets(...):
expectedStreetPattern = \
    u"^.*(?<![Ss]tra\u00dfe)(?<![Ww]eg)(?<![Aa]llee)(?<![Rr]ing)(?<![Bb]erg)" + \
    u"(?<![Pp]ark)(?<![Hh]\u00f6he)(?<![Pp]latz)(?<![Bb]r\u00fccke)(?<![Gg]rund)$"
l = list(project_coll.distinct("name", {
                    "type": "way",
                    "name": {"$regex": expectedStreetPattern}
                }))
# Output too long... See the file Project/output_project.py_-q.txt


I am working on the file ".ipynb_checkpoints/Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data-checkpoint.ipynb".
The first task is:
Cuisines in Restaurants
We can list the types of cuisines in restaurants (elements with attribute amenity matching restaurant) and sort them in decending order. We can notice certain inconsistencies or overlaps in the classifications of this data: e.g., a kebab cuisine may very well be also classified as an arab cuisine or may, in fact a sub- or super-classification of this cuisine. One could, e.g., eliminate or cluster together especially occurences of cuisines which are less common, but Without having a formal taxonomy of all cuisines, I decided that is probably best to leave the data as-is in order to not sacrifice preciseness for consistency.
Can you write Python code for it?

from Project.notebook_stub import project_coll
import pprint

# Query used - see function: Project.audit_stats_map.stats_cuisines(...):
pipeline = [
        {"$match": {"amenity": "restaurant"}},
        {"$group": {"_id": "$cuisine", "count": {"$sum": 1}}},
        {"$sort": {"count": -1}}
    ]
l = list(project_coll.aggregate(pipeline))
pprint.pprint(l[1:10]+['...'])


I am working on the file ".ipynb_checkpoints/Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data-checkpoint.ipynb".
The first task is:
Popular Leisure Activities
The attribute leisure shows the types of leisure activities one can do in Dresden and inspired me to invesigate more on popular sports in the city (leisure=sports_center or leisure=stadium).
Can you write Python code for it?

from Project.notebook_stub import project_coll
import pprint

# Query used - see function: Project.audit_stats_map.stats_amenities(...):
pipeline = [
        {"$match": {"leisure": {"$exists": True}}},
        {"$group": {"_id": "$leisure", "count": {"$sum": 1}}},
        {"$sort": {"count": -1}}
    ]
l = list(project_coll.aggregate(pipeline))
pprint.pprint(l[1:10]+['...'])


I am working on the file ".ipynb_checkpoints/Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data-checkpoint.ipynb".
The first task is:
Problems Encountered in the Map / Data Quality
Try it out: Use python project.py -q to obtain the data from this chapter. See Sample Output in file Project/output_project.py_-q.txt. The script also writes a CSV file to Project/data/audit_buildings.csv, which is also beautified into a Excel File.
Leading Zeros
As already discussed, during the parsing stage, we are using an optimistic approach of parsing any numerical value as integer or float, if it is parsable as such. However, we noticed that we should not do this, if leading zeros are present as those hold semantics for phone numbers and zip codes. Otherwise, this cleaning approach gives us a much smaller representation of the data in MongoDB and in-memory.
Normalizing / Cleaning Cuisines
As hinted in section Cuisines in Restaurant, classification of cuisines is inconsistent. There are two problems with this value:

There are multiple values separated by ';' which makes the parameter hard to parse. We overcome this by creating a parameter cuisineTag which stores the cuisine classifications as an array:

python
  db.eval('''db.osmnodes.find({
                        "cuisine": {"$exists": true},
                        "amenity": "restaurant"
                    }).snapshot().forEach(function(val, idx) {
                        val.cuisineTags = val.cuisine.split(';');
                        db.osmnodes.save(val)
                    })
             ''')

Some values are inconsistently used; therefore, we unify them with a mapping table and a subsequent MongoDB update:

```python
  cuisines_synonyms = {
      'german': ['regional', 'schnitzel', 'buschenschank'],
      'portuguese': ['Portugiesisches_Restaurant_&_Weinbar'],
      'italian': ['pizza', 'pasta'],
      'mediterranean': ['fish', 'seafood'],
      'japanese': ['sushi'],
      'turkish': ['kebab'],
      'american': ['steak_house']
    }
  # not mapped:
  # greek, asian, chinese, indian, international, vietnamese, thai, spanish, arabic
  # sudanese, russian, korean, hungarian, syrian, vegan, soup,...
Can you write Python code for it?

from Project.notebook_stub import project_coll

# Query used - see function: Project.audit_quality_map.audit_phone_numbers(...):
pipeline = [
        {"$match": {"$or": [
                {"phone": {"$exists": True}},
                {"mobile_phone": {"$exists": True}},
                {"address.phone": {"$exists": True}}
            ]}},
        {"$project": {
                "_id": 1,
                "phone": {"$ifNull": ["$phone", {"$ifNull": ["$mobile_phone", "$address.phone"]}]}
            }}
    ]
l = project_coll.aggregate(pipeline)

# Output too long... See the file Project/output_project.py_-q.txt


I am working on the file ".ipynb_checkpoints/Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data-checkpoint.ipynb".
The first task is:
The resulting objects are then iterated through and the best and worst fitting nearby street name are identified each using the Levenshtein distance. For each object, a row is created in a DataFrame which is subsequently exported to a csv file Project/data/audit_buildings.csv that was manually beautified into an Excel File.

As can be seen, street names of nearby objects mostly match those of the building itself (Levenshtein distance is zero). If they deviate greatly, they are totally different street names in the same area and not just "typos" or non-conforming abbreviations.
Auditing Zip Codes (Spoiler Alert: No Cleaning Necessary)
Try it out: Use python project.py -Z which runs the auditing script for zipcodes. See Sample Output in file Project/output_project.py_-Z.txt. To be able to run this script correctly, the zipcode data from Geonames.org needs to be downloaded  and installed first using the -z option (see output in `Project/output_project.py_-Z.txt).
This part of the auditing process makes use of an additional at Geonames.org to resolve and audit the zip codes in the data set. During the "installation process" (option -z) the zipcode data (provided as a tab-separated file) is downloaded and, line-by-line, stored to a (separate) MongoDB collection. However, we are only interested "zipcode" (2) and  "place" (3)
During the auditing stage (option -Z) we first get a list of all used zipcode using the following query:
python
pipeline = [
            { "$match": {"address.postcode": {"$exists": 1}} },
            { "$group": {"_id": "$address.postcode", "count": {"$sum": 1}} },
            { "$sort": {"count": 1} }
        ]
The zipcodes are then all looked up in the zipcode collection using the $in-operator. The data obtained is joined back into the original result.
python
zipcodeObjects = zipcodeColl.find( {"zipcode": {"$in": [z["_id"] for z in zipcodeList]}} )
The following output shows that there the lesser used zipcodes are from the Dresden metropolitan area, not Dresden itself:
Can you write Python code for it?

from Project.audit_zipcode_map import audit_zipcode_map
from Project.notebook_stub import project_server, project_port
import pprint

zipcodeJoined = audit_zipcode_map(project_server, project_port, quiet=True)
pprint.pprint(zipcodeJoined[1:10]+['...'])


I am working on the file ".ipynb_checkpoints/Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data-checkpoint.ipynb".
The first task is:
Types of Amenities
The attribute amenity inspired me to do further research in which kind of buildings / objects / facilities are stored in the Open Street Map data in larger quantities in order to do more detailed research on those objects. Especially Restaurants, Pubs and Churches / Places of Worship were investigated further (as can be seen below).
Can you write Python code for it?

from Project.notebook_stub import project_coll
import pprint

# Query used - see function: Project.audit_stats_map.stats_amenities(...):
pipeline = [
        {"$match": {"amenity": {"$exists": True}}},
        {"$group": {"_id": "$amenity", "count": {"$sum": 1}}},
        {"$sort": {"count": -1}}
    ]
l = list(project_coll.aggregate(pipeline))
pprint.pprint(l[1:10]+['...'])


I am working on the file ".ipynb_checkpoints/Data Analysis Project 3 - Data Wrangle OpenStreetMaps Data-checkpoint.ipynb".
The first task is:
Users Involved
There were about 1634 users involved in creating the data set, the top 10 of all users accounts for 40% of the created data. There is no direct evidence from the user name that any of them are bot-like users. This could be determined by further research. Many users (over 60%) have made less than 10 entries.
Can you write Python code for it?

from Project.notebook_stub import project_coll
import pprint

# Query used - see function: Project.audit_stats_map.stats_users(...):
pipeline = [
        {"$match": {"created.user": {"$exists": True}}},
        {"$group": {"_id": "$created.user", "count": {"$sum": 1}}},
        {"$sort": {"count": -1}}
    ]
l = list(project_coll.aggregate(pipeline))
print str(len(l)) + " users were involved:"
pprint.pprint(l[1:5]+["..."]+l[-5:])


I am working on the file ".ipynb_checkpoints/Data Generation and Likelihood Fit-checkpoint.ipynb".
The first task is:
2°) Exponential Form (other parameters for scaling)
Can you write Python code for it?

params_exp = (15, 0.05)
p = (p_x0, p_x1, p_x2)
exp_forward, exp_myopic = contraction_mapping(S=70, p=p, MF=exp_cost, params=params_exp, beta = 0.75)
pchoice = exp_forward.T[0]
plt.plot(exp_forward.T[0])
plt.plot(exp_myopic.T[0])
plt.ylabel("Probability of maintenance")
plt.xlabel("State of the bus")
plt.legend(["Forward-Looking", "Myopic"])
plt.xlim(20, 65)
plt.title("Action probabilities of the Agents (Exp. Spec.)");

n_bus = 1000
bus_array = np.hstack((np.linspace(1, n_bus, n_bus).reshape(-1, 1), np.zeros((n_bus, 3))))
for i in range(n_periods):
    bus_array = transition(bus_array, pchoice)
exp_dataframe = pd.DataFrame(bus_array, columns=["Identifier", "Choice", "Mileage", "State"])
choice_freq = exp_dataframe.loc[:, ["Choice", "State"]].groupby(["State"]).mean().sort().reset_index()
plt.plot(choice_freq["State"], 1-choice_freq["Choice"])
plt.plot(pchoice, "--")
plt.xlabel("Observed state of the bus")
plt.ylabel("Probability of maintenance")
plt.legend(["Observed behavior", "True choice probabilities"])
plt.title("Behavior of forward-looking agent (Exp. Spec.)")
plt.xlim(20, 65)
plt.ylim(-0.05, 1.05);
exp_dataframe.loc[n_bus+1:,["Choice", "State"]].to_csv("Datasets\Exp_Dataset.csv", index=False)


I am working on the file ".ipynb_checkpoints/Data Generation and Likelihood Fit-checkpoint.ipynb".
The first task is:
2°) Fitting the linear cost data
In this section, we fit the data generated by the linear cost function and recover the parameters $\text{RC}$ and $\theta$. We do it for different characterizations of the cost function, and thereby illustrate the consequences of a misspecification.
Recall that in the linear model:
$$\text{RC} = 20$$
$$\theta_{11} = 0.5$$
A. Fitting the true linear cost function to the data
We start by initializing a DynamicLogit instance, and fit the likelihood function to the data.
Can you write Python code for it?

lin_to_lin = DynamicLogit(lin_dataframe, "Choice", "State", p, lin_cost, npars=2)
lin_to_lin.fit_likelihood()