I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Observemos dos detalles de este ejemplo. El orden en el que aparecen los elementos es el mismo que el orden especificado por el argumento index, comparemos esto con el caso anterior donde creamos una serie a partir de un diccionario, pero sin especificar el índice.
El otro detalle es que hemos pasado un rótulo para un valor que no existe en el diccionario (y hemos omitido dos valores que si existen). Como resultado Pandas no devolvió un error, si no que interpretó que tenemos datos faltantes (missing data). El dato faltante se indica utilizando un tipo especial de float NaN (del inglés Not A Number).
Los índices son una conveniencia para manipular datos haciendo referencia a nombres que nos puede resultar más familiares o convenientes (comparado con recordar la posición de los datos). Además, los índices son usados para alinear datos al operar con más de una serie, por ej podríamos querer obtener el total de bacterias en dos conjuntos de datos.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Otro método que se puede aplicar sobre un objeto groupby es filter. Este método permite filtrar el resultado mediante una función arbitraria. Supongamos que fuese de interés saber cuales pacientes tuvieron un conteo mayor a 120 bacterias para todas las categorías taxonómicas medidas.
Can you write Python code for it?

def filtro(x):
    return np.all(x['Tejido'] > 120)

mb.groupby('Paciente').filter(filtro)


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Pandas es un paquete que Python que provee estructuras de datos rápidas, flexibles y expresivas diseñadas para trabajar con datos rotulados. Dichas estructuras de datos se pueden pensar como arrays de NumPy donde las filas y columnas están rotuladas. O de forma similar como una planilla de cálculo bajo Python.
Así como NumPy es una muy buena herramienta para trabajar con números, vectores, álgebra lineal, etc. Pandas es adecuado para trabajar con:

Datos tabulares y heterogéneos (flotantes, string, enteros, etc)
Series temporales
Los mismos datos que se pueden manipular con arreglos de NumPy!

¿Por qué es importante tener una herramienta como Pandas?
<img src="imagenes/analisis.png"  width=400>
El tiempo consumido (por un humano) en aplicar métodos estadísticos y/o de machine learning es en muchos casos mucho menor que el tiempo requerido para obtener y procesar los datos. Pandas intenta facilitar el procesado de los datos y reducir el tiempo que este consume para que podamos aumentar el tiempo que pasamos pensando en los problemas que queremos resolver. Procesar datos es una tarea que suele involucrar los siguientes pasos

Leer los datos: Los datos pueden estar contenidos en diversos formatos, CSV, HTML, xls, pdf, texto plano, imágenes, hojas de papel, etc
Procesar datos: Los datos rara vez están listos para usar, pueden faltar datos, haber dudas sobre los valores registrados, inconsistencias, etc. Además puede ser necesario generar datos derivados a partir de los datos disponibles, por ej podríamos necesitar la densidad poblacional pero solo tenemos la población y la superficie.
Almacenar datos: Ya sea para pasárselos a otra pieza de software para por ej visualizarlos o hacer un análisis estadístico de los datos, o para su posterior uso por nosotros o terceros.

Pandas introduce fundamentalmente 3 nuevas estructuras de datos:
* Las Series
* Los DataFrame
* Los Index
Empecemos por la primera de estas.
Series
Una Series de Pandas es un conjunto unidimensional de datos (similar a un array) acompañado de un...
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Pandas provee de dos métodos para indexar. El primero de ellos es loc que permite hacer las operaciones de indexado/rebanado usando SIEMPRE el índice explícito.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Para mayor claridad de los resultados podríamos querer ordenar el DataFrame resultante en función de la columna 'Paciente'
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Podemos agregar una Series como una nueva columna en un DataFrame, el resultado dependerá de los índices de ambos objetos.
Can you write Python code for it?

tratamiento = pd.Series([0] * 4 + [1] * 4)
tratamiento

datos['tratamiento'] = tratamiento
datos


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Podemos usar del para eliminar columnas, de la misma forma que podemos hacerlo con diccionarios.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Podemos ver que en promedio el grupo de fallecidos pagó menos que el grupo de sobrevivientes.
Veamos estos mismos datos, pero de forma gráfica.
Can you write Python code for it?

boleto_fallecidos.plot(kind='kde')
boleto_sobrevivientes.plot(kind='kde')

plt.xlim(0);  # valores por debajo de 0 no tienen sentido

sns.violinplot(x="survived", y="fare", data=titanic[titanic['fare'] < 200]);


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Podemos ver que los registros están casi completos excepto para la edad (age), variable para la cual el 20% de los datos son faltantes. Luego volveremos sobre esto, por ahora tratemos de constestar cual fue el porcentaje de sobrevivientes. Dado que los sobrevivientes están indicados con 1.0 y los fallecidos con 0.0, podemos obtener el porcentaje de sobrevivientes haciendo:
Can you write Python code for it?

round(titanic['survived'].mean() * 100, 0)


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Por defecto read_csv usa , como separadores, pero es posible modificar este comportamiento usando el argumento sep. Un caso muy común es el de archivos que tienen una cantidad variable de espacios en blanco. En esos casos podemos usar una expresión regular:
sep='\s+'

'\s+' quiere decir "1 o más espacios en blanco". Otro caso común son archivos separados por tabulaciones (tabs), en ese caso podemos usar \t.
Si queremos omitir datos (por ejemplo datos mal tomados), podemos indicaselo a Pandas usando el argumento skiprows:
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Por defecto read_csv usará la primer linea del archivo como encabezado (header). Este comportamiento lo podemos modificar usando el argumento header.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Por ejemplo podemos indexarlo usando los nombres de las columnas, podríamos querer hacer esto para saber cuantos pacientes tenemos en cada grupo
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Recordemos que son las columnas quienes tiene prioridad en un DataFrame, si quisieramos el ritmo cardíaco de Ana:
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Se puede ver que el conteo (count) para Paciente y Heces no coinciden, que el valor más pequeño para Heces es un número negativo cuando debería ser mayor o igual a cero. Y vemos que no tenemos descripción para Tejido!
¿Se te ocurre por que falta la columna para Tejido?

Podemos indicarle a Pandas que valores debe considerar como datos faltantes. Para eso usamos el argumento na_values.
Can you write Python code for it?

mbm = pd.read_csv('datos/microbioma_faltantes.csv', na_values=['?', -99999])

mbm.describe()


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Se puede ver que tenemos información sobre 1310 personas (filas) que estuvieron a bordo y para cada persona tenemos 14 variables (columnas). En machinelearngua se le suele llamar features a las variables.
¿Qué tipos de variables se corresponden con cada columna?

Recordemos que podemos acceder a los nombres de las variables mediante el atributo columns.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Si tuvieras una Series con índices que fuesen enteros ¿Qué pasaría al indexarla?
una_series[1]

¿Obtendríamos el segundo elemento de la serie o el elemento cuyo índice explícito es 1? ¿Y si el ńumero 1 no estuviera contenido en el índice de la serie?
Más adelante veremos que solución ofrece Pandas para evitar confusiones. Pero mientras quizá podemos pensar una solución y discutirla.

Al igual que con los arreglos de NumPy podemos usar booleanos para indexar una serie. De esta forma podemos contestar la pregunta; cuáles bacterias dieron conteos superiores a 1000 de forma bastante intuitiva (¿Es realmente intuitiva?).
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Slicing
Es posible hacer slicing incluso cuando los índices son strings.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
También podemos hacerlo por atributo.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Una Series tiene siempre dos columnas. La primer columna contiene los índices y la segunda los datos. En el ejemplo anterior, pasamos una lista de datos y omitimos el índice por lo que Pandas creo un índice automáticamente usando una secuencia de enteros, empezando por 0 (como es usual en Python).
La idea que una Series es un array con un índice explícito, no es solo una metáfora. De hecho a partir de una Serie es posible obtener el array de NumPy "contenido" en ella.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Una alternativa es especificar cuales son las columnas que queremos concatenar, para ello debemos pasarle al argumento join_axes una lista de Index.
Can you write Python code for it?

pd.concat([df0, df2], join_axes=[pd.Index(['B', 'C'])])


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Una característica de Python que no vimos en la primer notebook es la posibilidad de escribir funciones anónimas, es decir funciones que no tienen nombre. Esto se conoce como funciones lambda. Siguiendo el ejemplo anterior la sintaxis para una función anónima que estandariza datos es:
lambda x: (x - x.mean()) / x.std()

Por lo que podríamos escribir.
Can you write Python code for it?

mb.groupby('Paciente')[['Tejido', 'Heces']].transform(lambda x: (x - x.mean()) / x.std())


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Una diferencia importante entre este append() y el que vimos para listas de Python es que Pandas no modifica el objeto original (como sucedía con las listas) si no que cada vez que hacemos un append() estamos creando un nuevo objeto con un nuevo índice. Esto hace que este método no sea muy eficiente. Dicho de otras forma mientras las listas están pensadas para hacer uso repetido del método append() los DataFrames no. Por lo tanto en caso de ser necesario hacer varios append() sobre DataFrames es mejor crear una lista de DataFrames y concatenar esa lista una sola vez.
Merge and join
Pandas ofrece operaciones del tipo merge y join que son familiares para quienes hayan trabajado con bases de  datos. merge tiene implementado un subconjunto de lo que se conoce como algebra relacional que son un conjunto de reglas formales para manipular datos relacionales. Este conjunto de reglas funciona como los bloques básicos (o primitivas) con los que se construyen operaciones más complejas sobre los datos. Pandas implementa  varios de estos bloques básicos.
Para mayor información por favor referirse a la documentación de Pandas y/o libros como Python Data Science Handbook y Python for Data Analysis
Indexado jerárquico
Hasta ahora hemos visto como trabajar con datos uni y bi-dimensionales. El indexado jerárquico (o multi-indexado) permite usar más de un índice en un DataFrame, de esta forma podemos trabajar en más de dos dimensiones pero usando un DataFrame.
Can you write Python code for it?

index = pd.MultiIndex.from_product([[2013, 2014], [1, 2]],
names=['año', 'visita'])
columns = pd.MultiIndex.from_product([['Ana', 'Brian', 'Carlos'], ['RC', 'Temp']],
names=['sujeto', 'tipo'])

datos = np.zeros((4, 6))
datos[:, 1::2] += np.random.normal(37, 0.5, (4, 3))
datos[:, ::2] += np.random.normal(65, 1, (4, 3))

datos_médicos = pd.DataFrame(np.round(datos, 1), index=index, columns=columns)
datos_médicos

datos_médicos['Brian']


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Vamos a cerrar esta sección con un ejemplo tomado de Python Data Science Handbook. Este ejemplo combina varias de las operaciones vistas hasta ahora. El objetivo es obtener un listado por década de la cantidad de exoplanetas descubiertos con diversos métodos.
Primero cargamos los datos
Can you write Python code for it?

planets = sns.load_dataset('planets')
planets.head()


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Veamos que además de la variable continua fare tenemos la variable discreta pclass que toma 3 valores de acuerdo a si las personas viajaban en primera (1), segunda (2) o tercera (3) clase. Veamos que nos dicen los datos cuando exploramos la tasa de supervivencia en función de la clase.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
Y también es posible obtener el índice.
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
append
La concatenación directa de de Series y DataFrame es tan común que en vez llamar pd.concat([df0, df1]) es posible obtener el mismo resultado haciendo:
Can you write Python code for it?


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
¿Habrá alguna diferencia para los sobrevivientes en función del costo del boleto que pagaron?
Can you write Python code for it?

boleto_fallecidos = titanic['fare'][titanic['survived'] == 0]
boleto_sobrevivientes = titanic['fare'][titanic['survived'] == 1]

'£{:.0f}, £{:.0f}'.format(boleto_fallecidos.mean(), boleto_sobrevivientes.mean())


I am working on the file "03_Manipulación_de_datos_y_Pandas.ipynb".
The first task is:
¿Qué sucede si intentamos agregar una nueva columna a partir de una lista cuya longitud no coincida con la del DataFrame? ¿Y si en vez de una lista es una Series?

Can you write Python code for it?

datos['mes'] = ['enero'] * len(datos)
datos


I am working on the file "03_Medidas_de_Correlacion.ipynb".
The first task is:
Definición de Spearman.
Se usa como una medida de correlación no paramétrica, en la cual se tiene en cuenta el orden de la serie y no sus datos, por lo que tiende a ser más estable.
$\rho = 1- \frac{6 \sum_{i=1}{N} d_{i}^{2}}{n (n^2-1)}$
Características:

No asume normalidad.
oscila entre -1 y 1, cuando es 0 es que no hay una correlación.
No mide únicamente la relación lineal.
Mide si dos variables $x_i$ y $y_i$ crecen o decresen monoliticamente.

|IQ, $X_i$ | Hours of TV per week, $Y_i$|
|:---:|:---:|
|106|   7|
|86 |0|
|100|   27|
|101|   50|
|99 |28|
|103|   29|
|97 |20|
|113|   12|
|112|   6|
|110|   17|
|IQ, $X_i$| Hours of TV per week, $Y_i$|    rank $x_i$| rank $y_i$| $d_i$|  $d_i^2$|
|:---:|:---:|:---:|:---:|:---:|:---:|
|86|    0|  1|  1   |0| 0|
|97|    20| 2|  6   |−4|    16|
|99 |28 |3  |8| −5  |25|
|100|   27  |4  |7  |−3|    9|
|101|   50  |5  |10|    −5| 25|
|103|   29  |6  |9| −3| 9|
|106|   7   |7  |3| 4|  16|
|110|   17  |8  |5| 3|  9|
|112|   6   |9  |2| 7|  49|
|113|   12| 10  |4| 6|  36|
Can you write Python code for it?

y = np.sin(data)
x = np.sin(data)+np.random.uniform(-0.1,0.1,len(data))
fig = pl.scatter(x,y)
Spearman, PvalueS = st.spearmanr(x,y)
print 'Spearman:' + str(Spearman)
print 'P-valor:' + str(PvalueS)

y = np.sin(data)
x = np.sin(data)+np.random.uniform(-0.1,0.1,len(data))
pos = np.where(y>0.2)[0]
y[pos] = y[pos]-(pos/150.0)**2
fig = pl.scatter(x,y)
Spearman, Pvalue = st.spearmanr(x,y)
print 'Spearman:' + str(Spearman)
print 'P-valor:' + str(Pvalue)


I am working on the file "03_Medidas_de_Correlacion.ipynb".
The first task is:
Histogramas conjuntos
A diferencia del histograma independiente, en este se presenta la probabilidad de ocurrencia de un suceso sólo en el caso en que el otro suceso ocurre de manera simultánea. Cuando este histograma es igual al histograma independiente se dice que las variables son independientes
Can you write Python code for it?

def Hist2D(x,y,bins = 12):
    RangoX = np.linspace(x.min(),x.max(),bins)
    RangoY = np.linspace(y.min(),y.max(),bins)
    h2d = np.zeros(12,12)
    for ic,i in enumerate(x):
        for jc,j in enumerate(y):

x = np.random.normal(0,1,1000)
y = np.random.uniform(0,1,1000)
fig = pl.figure()
a = pl.hist2d(x,y,bins=12)
fig = pl.figure()
a = pl.scatter(x,y)

cont = 0
for i in [1,2,4,12,2]:
    print i,cont
    cont-=10


I am working on the file "03_Medidas_de_Correlacion.ipynb".
The first task is:
Histogramas dentro de histogramas
Can you write Python code for it?

fig = pl.figure(figsize=(16,5))
for cont,i in enumerate([2,10,25,40]):
    ax = fig.add_subplot(1,4,cont+1)
    ax.set_title('Para X>'+str(i),size=16)
    Serie[Serie>i].hist(bins = 15)


I am working on the file "03_Medidas_de_Correlacion.ipynb".
The first task is:
Límite central en la lluvia
Can you write Python code for it?

fig = pl.figure(figsize=(16,5))
for cont,i in enumerate(['D','M','3M','6M']):
    ax = fig.add_subplot(1,4,cont+1)
    ax.set_title('Escala '+i,size = 16)
    Serie.resample(i,how='sum').hist(bins = 10)


I am working on the file "03_Model-Training-CNN.ipynb".
The first task is:
GPU and CPU settings
If GPU is not available, comment out the bottom block.
Can you write Python code for it?

# If GPU is not available:
# GPU_USE = '/cpu:0'
# config = tf.ConfigProto(device_count = {"GPU": 0})

# If GPU is available:
config = tf.ConfigProto()
config.log_device_placement = True
config.allow_soft_placement = True
config.gpu_options.allocator_type = 'BFC'

# Limit the maximum memory used
config.gpu_options.per_process_gpu_memory_fraction = 0.1

# set session config
tf.keras.backend.set_session(tf.Session(config=config))

########## HYPER PARAMETERS
batch_size = 128
epochs = 10
optimizer = tf.keras.optimizers.RMSprop()
########## HYPER PARAMETERS

########## MODEL ARCHITECTURE
model = tf.keras.Sequential()

### OLD ARCHITECTURE START
#model.add(tf.keras.layers.Dense(5, activation='relu', input_shape=(784,)))
### OLD ARCHITECTURE END

### NEW CNN ARCHITECTURE
model.add(tf.keras.layers.Conv2D(5, kernel_size=(3, 3), activation='relu', input_shape=(28,28,1)))
model.add(tf.keras.layers.Flatten())
### NEW CNN ARCHITECTURE

model.add(tf.keras.layers.Dense(num_classes, activation='softmax'))
########## MODEL ARCHITECTURE

# Print summary
model.summary()

# compile model for training
model.compile(loss='categorical_crossentropy',
              optimizer=optimizer,
              metrics=['accuracy'])

history = model.fit(x_train, y_train_one_hot,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(x_test, y_test_one_hot))


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
As we need to compute $x_{sc}$, we have to perform
$$A \cdot q_{ext} = T$$
where:
- A is a matrix with number of nodes + number of loops rows and number of edges +1 columns (it is square)
- q is a column vector of unknowns: #edges fluxes and shear center coordinate
- T is the vector of known terms: $-\frac{T_y}{J_x} \cdot S_{x_i}$ or $-\frac{T_x}{J_y} \cdot S_{y_i}$ for n-1 nodes and the rest are 0
Expression of A
Can you write Python code for it?


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Compute axial loads in Stringers in S3
Set loads on the section:
Example 1: shear in y direction and bending moment in x direction
Can you write Python code for it?

Tx, Ty, Nz, Mx, My, Mz, F, ry, ry, mz = sympy.symbols('T_x T_y N_z M_x M_y M_z F r_y r_x m_z')
S3.set_loads(_Tx=0, _Ty=Ty, _Nz=0, _Mx=Mx, _My=0, _Mz=0)


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Compute axial loads in stringers and shear flows in panels
Can you write Python code for it?


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Define section and perform first calculations
Can you write Python code for it?


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Example 2: twisting moment in z direction
Can you write Python code for it?

S3.set_loads(_Tx=0, _Ty=0, _Nz=0, _Mx=0, _My=0, _Mz=Mz)
S3.compute_stringer_actions()
S3.compute_panel_fluxes();

S3.N

S3.q


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Expression of T
Can you write Python code for it?


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Expression of inertial properties in principal reference frame
Can you write Python code for it?


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Loops detection
Can you write Python code for it?


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Multiconnected Section
Define graph describing the section:
1) stringers are nodes with parameters:
- x coordinate
- y coordinate
- Area
2) panels are oriented edges with parameters:
- thickness
- lenght which is automatically calculated
Can you write Python code for it?


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Plot of S3 section in original reference frame
Can you write Python code for it?

start_pos={ii: [float(S3.g.node[ii]['ip'][i].subs(datav)) for i in range(2)] for ii in S3.g.nodes() }

plt.figure(figsize=(12,8),dpi=300)
nx.draw(S3.g,with_labels=True, arrows= True, pos=start_pos)
plt.arrow(0,0,20,0)
plt.arrow(0,0,0,20)
#plt.text(0,0, 'CG', fontsize=24)
plt.axis('equal')
plt.title("Section in starting reference Frame",fontsize=16);


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Resulting fluxes and coordinate
Can you write Python code for it?


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Set loads on the section:
Example 3: shear in x direction and bending moment in y direction
Can you write Python code for it?

S3.set_loads(_Tx=Tx, _Ty=0, _Nz=0, _Mx=0, _My=My, _Mz=0)
S3.compute_stringer_actions()
S3.compute_panel_fluxes();

S3.N

S3.q


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Shear center expression
Can you write Python code for it?


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
Shear flows
Can you write Python code for it?


I am working on the file "03_Multiconnected_section.ipynb".
The first task is:
We also define numerical values for each symbol in order to plot scaled section and perform calculations
Can you write Python code for it?

values = [(A, 450 * ureg.millimeter**2),(A0, 250  * ureg.millimeter**2),(a, 130 * ureg.millimeter), \
          (b, 300 * ureg.millimeter),(h, 150 * ureg.millimeter),(L, 650 * ureg.millimeter)]
datav = [(v[0],v[1].magnitude) for v in values]


I am working on the file "03_Neural_Network.ipynb".
The first task is:
3. Trainning loop
* (입력 생성)
* model 생성
* loss 생성
* zeroGrad
* backpropagation
* optimizer step (update model parameter)
Can you write Python code for it?

# trainning
for i in range(100) :
    # input variable
    x = Variable(x_data)
    y = Variable(y_data)
    # network model
    y_pred = model(x)
    loss = loss_fn(y_pred, y)
    # zero_grad, backward, step(update parameter) in series
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if i % 10 == 0:
        print(i, loss.data[0])


I am working on the file "03_Neural_Network.ipynb".
The first task is:
4. Predict
Can you write Python code for it?

y_pred = model(x).max(1)[1]
print('prediction :', y_pred.data)
print('true :', y)


I am working on the file "03_Plotting.ipynb".
The first task is:
Box Plots
Can you write Python code for it?

#box plots
cars['mpg'].plot(kind='box')

cars.boxplot('mpg')

#group by gear
cars.boxplot('mpg', by='gear')

# load gapminder again and select 2007
gap = pd.read_csv('data/gapminder-unfiltered.tsv',index_col=0, sep='\t')
gap2007 = gap[gap.year == 2007]
gap2007.columns


I am working on the file "03_Plotting.ipynb".
The first task is:
Scatterplot
Can you write Python code for it?

plt.scatter(x=cars['mpg'],y=cars['wt'])
plt.xlabel('miles per gallon')
plt.ylabel('weight')
plt.title('MPG vs WT')
plt.show()

#integrated in pandas, too
cars.plot(x='mpg',y='wt',kind='scatter')

cars.plot(kind='scatter', x='mpg',y='wt',c='hp',s=cars['cyl']*20,alpha=0.5)

#what if we plot everything?
cars.plot()


I am working on the file "03_Plotting.ipynb".
The first task is:
TASK

create a scatterplot where
* x = lifeExp
* y = gdpPerCap
* color = continent
* size = pop
label the axis appropiately and use a log scale for gdp
Can you write Python code for it?


I am working on the file "03_Plotting.ipynb".
The first task is:
TASK

make the plot from before interactive, such that you can slide the year
Can you write Python code for it?


I am working on the file "03_Plotting.ipynb".
The first task is:
custom build widgets:
http://nbviewer.ipython.org/github/ipython/ipython/blob/3.x/examples/Interactive%20Widgets/Widget%20List.ipynb
Can you write Python code for it?

#hard core

from IPython.html import widgets

[widget for widget in dir(widgets) if not widget.endswith('Widget') and widget[0] == widget[0].upper() and widget[0] != '_']

@interact(bins=widgets.FloatTextWidget(value=5))
def show_distplot(bins):
    cars['mpg'].hist(bins=bins)

text_widget = widgets.Textarea(value='Hello', description='text area')
slider_widget = widgets.BoundedFloatText(5,min=0,max=10, description='slider area')
check_widget = widgets.Checkbox(True,description="CheckboxWidget")
toggle = widgets.RadioButtons(options=['red','green','blue'], description="RadioButtonsWidget")

@interact(text=text_widget, slider=slider_widget,check=check_widget,categories=toggle)
def react(text, slider,check,categories):
    print(text,slider*10,check,categories)

b = widgets.Button(description="Update")
checkbox = widgets.Checkbox(description="CheckboxWidget")

tab1_children = [b,
                 checkbox,
                 widgets.Dropdown(options=['A','B'], description="DropdownWidget"),
                 widgets.RadioButtons(options=['A','B'], description="RadioButtonsWidget"),
                 widgets.Select(options=['A','B'], description="SelectWidget"),
                 widgets.Text(description="TextWidget"),
                 widgets.Textarea(description="TextareaWidget"),
                 widgets.ToggleButton(description="ToggleButtonWidget"),
                 widgets.ToggleButtons(options=["Value 1", "Value2"], description="ToggleButtonsWidget"),
                 ]

tab2_children = [widgets.BoundedFloatText(description="BoundedFloatTextWidget"),
                 widgets.BoundedIntText(description="BoundedIntTextWidget"),
                 widgets.FloatSlider(description="FloatSliderWidget"),
                 widgets.FloatText(description="FloatTextWidget"),
                 widgets.IntSlider(description="IntSliderWidget"),
                 widgets.IntText(description="IntTextWidget"),
                 ]

tab1 = widgets.Box(children=tab1_children)
tab2 = widgets.Box(children=tab2_children)

i =...


I am working on the file "03_Plotting.ipynb".
The first task is:
unbalanced with outliers what about log scale?
Can you write Python code for it?


I am working on the file "03_Plotting_solution.ipynb".
The first task is:
Box Plots
Can you write Python code for it?


I am working on the file "03_Plotting_solution.ipynb".
The first task is:
Histogram
Can you write Python code for it?

cars['mpg'].hist(bins=5)

plt.hist(cars['mpg'],bins=5)
plt.title('miles per gallon')

#seaborn not just a histogram but also an kernel density enstimation and better default settings
sns.distplot(cars['mpg'],bins=5)


I am working on the file "03_Plotting_solution.ipynb".
The first task is:
Interactive plots
simple interaction is possible with IPython by default. That means whenever the user changes some parameter the visualization is recreated on the server side and send to the client.
Can you write Python code for it?

from IPython.html.widgets import interact, interact_manual

@interact(text='Hello', slider=(0,10),check=True,categories=['red','green','blue'])
def react(text, slider,check,categories):
    print(text,slider*10,check,categories)

@interact_manual(text='Hello', slider=(0,10),check=True,categories=['red','green','blue'])
def react(text, slider,check,categories):
    print(text,slider*10,check,categories)

@interact(bins=(5, 25, 5),color=['red','green','orange','blue'])
def show_distplot(bins,color):
    cars['mpg'].hist(bins=bins, color=color)


I am working on the file "03_Plotting_solution.ipynb".
The first task is:
Scatterplot
Can you write Python code for it?


I am working on the file "03_Plotting_solution.ipynb".
The first task is:
TASK

create a scatterplot where
* x = lifeExp
* y = gdpPerCap
* color = continent
* size = pop
label the axis appropiately and use a log scale for gdp
Can you write Python code for it?

#for each group create an own plot an overlay them
pop_max = gap2007['pop'].max()
for (name, group),color in zip(gap2007.groupby('continent'),colors):
    plt.scatter(x=group['lifeExp'],y=group['gdpPercap'],label=name, c=color,s=(group['pop']/pop_max)*400)
plt.yscale('log')
plt.title('Life Expectancy vs GDP')
plt.xlabel('Life Expectancy')
plt.ylabel('GDP Per Cap')
plt.legend()


I am working on the file "03_Plotting_solution.ipynb".
The first task is:
TASK

make the plot from before interactive, such that you can slide the year
Can you write Python code for it?

pop_max = gap['pop'].max()

@interact(year=(gap.year.min(), gap.year.max()))
def plot_gapminder(year):
    gapyear = gap[gap.year == year]
    for (name, group),color in zip(gapyear.groupby('continent'),colors):
        plt.scatter(x=group['lifeExp'],y=group['gdpPercap'],label=name, c=color,s=(group['pop']/pop_max)*400)
    plt.yscale('log')
    plt.title('Life Expectancy vs GDP')
    plt.xlabel('Life Expectancy')
    plt.ylabel('GDP Per Cap')
    plt.xlim(gap.gdpPercap.min(),gap.gdpPercap.max())
    plt.xlim(gap.lifeExp.min(),gap.lifeExp.max())
    plt.legend()


I am working on the file "03_Plotting_solution.ipynb".
The first task is:
custom build widgets:
http://nbviewer.ipython.org/github/ipython/ipython/blob/3.x/examples/Interactive%20Widgets/Widget%20List.ipynb
Can you write Python code for it?


I am working on the file "03_Plotting_solution.ipynb".
The first task is:
unbalanced with outliers what about log scale?
Can you write Python code for it?


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Close TensorFlow Session
We are now done using TensorFlow, so we close the session to release its resources.
Can you write Python code for it?

# This has been commented out in case you want to modify and experiment
# with the Notebook without having to restart it.
# session.close()


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Convolution Layer 1
Now plot the filter-weights for the first convolutional layer.
Note that positive weights are red and negative weights are blue.
Can you write Python code for it?


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Convolution Layer 2
Now plot the filter-weights for the second convolutional layer.
There are 16 output channels from the first conv-layer, which means there are 16 input channels to the second conv-layer. The second conv-layer has a set of filter-weights for each of its input channels. We start by plotting the filter-weigths for the first channel.
Note again that positive weights are red and negative weights are blue.
Can you write Python code for it?


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Data Dimensions
The data dimensions are used in several places in the source-code below. They are defined once so we can use these variables instead of numbers throughout the source-code below.
Can you write Python code for it?

# We know that MNIST images are 28 pixels in each dimension.
img_size = 28

# Images are stored in one-dimensional arrays of this length.
img_size_flat = img_size * img_size

# Tuple with height and width of images used to reshape arrays.
img_shape = (img_size, img_size)

# Number of colour channels for the images: 1 channel for gray-scale.
num_channels = 1

# Number of classes, one class for each of 10 digits.
num_classes = 10


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Function for performing a number of optimization iterations so as to gradually improve the variables of the network layers. In each iteration, a new batch of data is selected from the training-set and then TensorFlow executes the optimizer using those training samples.  The progress is printed every 100 iterations.
Can you write Python code for it?

# Counter for total number of iterations performed so far.
total_iterations = 0

def optimize(num_iterations):
    # Ensure we update the global variable rather than a local copy.
    global total_iterations

    # Start-time used for printing time-usage below.
    start_time = time.time()

    for i in range(total_iterations,
                   total_iterations + num_iterations):

        # Get a batch of training examples.
        # x_batch now holds a batch of images and
        # y_true_batch are the true labels for those images.
        x_batch, y_true_batch = data.train.next_batch(train_batch_size)

        # Put the batch into a dict with the proper names
        # for placeholder variables in the TensorFlow graph.
        feed_dict_train = {x: x_batch,
                           y_true: y_true_batch}

        # Run the optimizer using this batch of training data.
        # TensorFlow assigns the variables in feed_dict_train
        # to the placeholder variables and then runs the optimizer.
        session.run(optimizer, feed_dict=feed_dict_train)

        # Print status every 100 iterations.
        if i % 100 == 0:
            # Calculate the accuracy on the training-set.
            acc = session.run(accuracy, feed_dict=feed_dict_train)

            # Message for printing.
            msg = "Optimization Iteration: {0:>6}, Training Accuracy: {1:>6.1%}"

            # Print it.
            print(msg.format(i + 1, acc))

    # Update the total number of iterations performed.
    total_iterations += num_iterations

    # Ending time.
    end_time = time.time()

    # Difference between start and end-times.
    time_dif = end_time - start_time

    # Print the time-usage.
    print("Time usage: " + str(timedelta(seconds=int(round(time_dif)))))


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Graph Construction
The Convolutional Neural Network will now be constructed using the helper-functions above. Without the helper-functions this would have been very long and confusing
Note that the following code will not actually be executed. It is just shown here for easy comparison to the PrettyTensor code below.
The previous tutorial used constants defined elsewhere so they could be changed easily. For example, instead of having filter_size=5 as an argument to new_conv_layer() we had filter_size=filter_size1 and then defined filter_size1=5 elsewhere. This made it easier to change all the constants.
Can you write Python code for it?

if False:  # Don't execute this! Just show it for easy comparison.
    # First convolutional layer.
    layer_conv1, weights_conv1 = \
        new_conv_layer(input=x_image,
                       num_input_channels=num_channels,
                       filter_size=5,
                       num_filters=16,
                       use_pooling=True)

    # Second convolutional layer.
    layer_conv2, weights_conv2 = \
        new_conv_layer(input=layer_conv1,
                       num_input_channels=16,
                       filter_size=5,
                       num_filters=36,
                       use_pooling=True)

    # Flatten layer.
    layer_flat, num_features = flatten_layer(layer_conv2)

    # First fully-connected layer.
    layer_fc1 = new_fc_layer(input=layer_flat,
                             num_inputs=num_features,
                             num_outputs=128,
                             use_relu=True)

    # Second fully-connected layer.
    layer_fc2 = new_fc_layer(input=layer_fc1,
                             num_inputs=128,
                             num_outputs=num_classes,
                             use_relu=False)

    # Predicted class-label.
    y_pred = tf.nn.softmax(layer_fc2)

    # Cross-entropy for the classification of each image.
    cross_entropy = \
        tf.nn.softmax_cross_entropy_with_logits(logits=layer_fc2,
                                                labels=y_true)

    # Loss aka. cost-measure.
    # This is the scalar value that must be minimized.
    loss = tf.reduce_mean(cross_entropy)


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Helper-function for plotting images
Function used to plot 9 images in a 3x3 grid, and writing the true and predicted classes below each image.
Can you write Python code for it?

def plot_images(images, cls_true, cls_pred=None):
    assert len(images) == len(cls_true) == 9

    # Create figure with 3x3 sub-plots.
    fig, axes = plt.subplots(3, 3)
    fig.subplots_adjust(hspace=0.3, wspace=0.3)

    for i, ax in enumerate(axes.flat):
        # Plot image.
        ax.imshow(images[i].reshape(img_shape), cmap='binary')

        # Show true and predicted classes.
        if cls_pred is None:
            xlabel = "True: {0}".format(cls_true[i])
        else:
            xlabel = "True: {0}, Pred: {1}".format(cls_true[i], cls_pred[i])

        # Show the classes as the label on the x-axis.
        ax.set_xlabel(xlabel)

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Helper-function for showing the performance
Function for printing the classification accuracy on the test-set.
It takes a while to compute the classification for all the images in the test-set, that's why the results are re-used by calling the above functions directly from this function, so the classifications don't have to be recalculated by each function.
Note that this function can use a lot of computer memory, which is why the test-set is split into smaller batches. If you have little RAM in your computer and it crashes, then you can try and lower the batch-size.
Can you write Python code for it?

# Split the test-set into smaller batches of this size.
test_batch_size = 256

def print_test_accuracy(show_example_errors=False,
                        show_confusion_matrix=False):

    # Number of images in the test-set.
    num_test = len(data.test.images)

    # Allocate an array for the predicted classes which
    # will be calculated in batches and filled into this array.
    cls_pred = np.zeros(shape=num_test, dtype=np.int)

    # Now calculate the predicted classes for the batches.
    # We will just iterate through all the batches.
    # There might be a more clever and Pythonic way of doing this.

    # The starting index for the next batch is denoted i.
    i = 0

    while i < num_test:
        # The ending index for the next batch is denoted j.
        j = min(i + test_batch_size, num_test)

        # Get the images from the test-set between index i and j.
        images = data.test.images[i:j, :]

        # Get the associated labels.
        labels = data.test.labels[i:j, :]

        # Create a feed-dict with these images and labels.
        feed_dict = {x: images,
                     y_true: labels}

        # Calculate the predicted class using TensorFlow.
        cls_pred[i:j] = session.run(y_pred_cls, feed_dict=feed_dict)

        # Set the start-index for the next batch to the
        # end-index of the current batch.
        i = j

    # Convenience variable for the true class-numbers of the test-set.
    cls_true = data.test.cls

    # Create a boolean array whether each image is correctly classified.
    correct = (cls_true == cls_pred)

    # Calculate the number of correctly classified images.
    # When summing a boolean array, False means 0 and True means 1.
    correct_sum = correct.sum()

    # Classification accuracy is the number of correctly classified
    # images divided by the total number of images in the test-set.
    acc = float(correct_sum) / num_test

    # Print the accuracy.
    msg = "Accuracy on Test-Set: {0:.1%} ({1} / {2})"
    print(msg.format(acc, correct_sum,...


# Split the test-set into smaller batches of this size.
test_batch_size = 256

def print_test_accuracy(show_example_errors=False,
                        show_confusion_matrix=False):

    # Number of images in the test-set.
    num_test = len(data.test.images)

    # Allocate an array for the predicted classes which
    # will be calculated in batches and filled into this array.
    cls_pred = np.zeros(shape=num_test, dtype=np.int)

    # Now calculate the predicted classes for the batches.
    # We will just iterate through all the batches.
    # There might be a more clever and Pythonic way of doing this.

    # The starting index for the next batch is denoted i.
    i = 0

    while i < num_test:
        # The ending index for the next batch is denoted j.
        j = min(i + test_batch_size, num_test)

        # Get the images from the test-set between index i and j.
        images = data.test.images[i:j, :]

        # Get the associated labels.
        labels = data.test.labels[i:j, :]

        # Create a feed-dict with these images and labels.
        feed_dict = {x: images,
                     y_true: labels,
                     keep_prob:1}

        # Calculate the predicted class using TensorFlow.
        cls_pred[i:j] = session.run(y_pred_cls, feed_dict=feed_dict)

        # Set the start-index for the next batch to the
        # end-index of the current batch.
        i = j

    # Convenience variable for the true class-numbers of the test-set.
    cls_true = data.test.cls

    # Create a boolean array whether each image is correctly classified.
    correct = (cls_true == cls_pred)

    # Calculate the number of correctly classified images.
    # When summing a boolean array, False means 0 and True means 1.
    correct_sum = correct.sum()

    # Classification accuracy is the number of correctly classified
    # images divided by the total number of images in the test-set.
    acc = float(correct_sum) / num_test

    # Print the accuracy.
    msg = "Accuracy on Test-Set: {0:.1%} ({1} / {2})"
   ...


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Helper-function to perform optimization iterations
There are 55,000 images in the training-set. It takes a long time to calculate the gradient of the model using all these images. We therefore only use a small batch of images in each iteration of the optimizer.
If your computer crashes or becomes very slow because you run out of RAM, then you may try and lower this number, but you may then need to perform more optimization iterations.
Can you write Python code for it?


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Helper-function to plot confusion matrix
Can you write Python code for it?

def plot_confusion_matrix(cls_pred):
    # This is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # Get the true classifications for the test-set.
    cls_true = data.test.cls

    # Get the confusion matrix using sklearn.
    cm = confusion_matrix(y_true=cls_true,
                          y_pred=cls_pred)

    # Print the confusion matrix as text.
    print(cm)

    # Plot the confusion matrix as an image.
    plt.matshow(cm)

    # Make various adjustments to the plot.
    plt.colorbar()
    tick_marks = np.arange(num_classes)
    plt.xticks(tick_marks, range(num_classes))
    plt.yticks(tick_marks, range(num_classes))
    plt.xlabel('Predicted')
    plt.ylabel('True')

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Helper-function to plot example errors
Function for plotting examples of images from the test-set that have been mis-classified.
Can you write Python code for it?

def plot_example_errors(cls_pred, correct):
    # This function is called from print_test_accuracy() below.

    # cls_pred is an array of the predicted class-number for
    # all images in the test-set.

    # correct is a boolean array whether the predicted class
    # is equal to the true class for each image in the test-set.

    # Negate the boolean array.
    incorrect = (correct == False)

    # Get the images from the test-set that have been
    # incorrectly classified.
    images = data.test.images[incorrect]

    # Get the predicted classes for those images.
    cls_pred = cls_pred[incorrect]

    # Get the true classes for those images.
    cls_true = data.test.cls[incorrect]

    # Plot the first 9 images.
    plot_images(images=images[0:9],
                cls_true=cls_true[0:9],
                cls_pred=cls_pred[0:9])


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Initialize variables
The variables for weights and biases must be initialized before we start optimizing them.
Can you write Python code for it?


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Load Data
The MNIST data-set is about 12 MB and will be downloaded automatically if it is not located in the given path.
Can you write Python code for it?

from tensorflow.examples.tutorials.mnist import input_data
data = input_data.read_data_sets('data/MNIST/', one_hot=True)


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Next we have the placeholder variable for the true labels associated with the images that were input in the placeholder variable x. The shape of this placeholder variable is [None, num_classes] which means it may hold an arbitrary number of labels and each label is a vector of length num_classes which is 10 in this case.
Can you write Python code for it?

y_true = tf.placeholder(tf.float32, shape=[None, num_classes], name='y_true')


y_true = tf.placeholder(tf.float32, shape=[None, 10], name='y_true')


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Now that we have wrapped the input image in a PrettyTensor object, we can add the convolutional and fully-connected layers in just a few lines of source-code.
Note that pt.defaults_scope(activation_fn=tf.nn.relu) makes activation_fn=tf.nn.relu an argument for each of the layers constructed inside the with-block, so that Rectified Linear Units (ReLU) are used for each of these layers. The defaults_scope makes it easy to change arguments for all of the layers.
Can you write Python code for it?

with pt.defaults_scope(activation_fn=tf.nn.relu):
    y_pred, loss = x_pretty.\
        conv2d(kernel=5, depth=16, name='layer_conv1').\
        max_pool(kernel=2, stride=2).\
        conv2d(kernel=5, depth=36, name='layer_conv2').\
        max_pool(kernel=2, stride=2).\
        flatten().\
        fully_connected(size=128, name='layer_fc1').\
        softmax_classifier(num_classes=num_classes, labels=y_true)


"""
with pt.defaults_scope(activation_fn=tf.nn.relu, l2loss=0.001):
    y_pred, loss = x_pretty.\
        conv2d(kernel=5, depth=16, name='layer_conv1').\
        max_pool(kernel=2, stride=2).\
        conv2d(kernel=5, depth=36, name='layer_conv2').\
        max_pool(kernel=2, stride=2).\
        flatten().\
        fully_connected(size=128, name='layer_fc1').\
        dropout(keep_prob, name='dropout').\
        softmax_classifier(class_count=10, labels=y_true)
"""
with pt.defaults_scope(activation_fn=tf.nn.relu, l2loss=0.001):
    y_pred, loss = x_pretty.\
        conv2d(kernel=5, depth=36, stride=2, name='layer_conv1').\
        flatten().\
        fully_connected(size=128, name='layer_fc1').\
        fully_connected(size=64,  name='layer_fc2').\
        dropout(keep_prob, name='dropout').\
        softmax_classifier(class_count=10, labels=y_true)


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Optimization Method
PrettyTensor gave us the predicted class-label (y_pred) as well as a loss-measure that must be minimized, so as to improve the ability of the Neural Network to classify the input images.
It is unclear from the documentation for PrettyTensor whether the loss-measure is cross-entropy or something else. But we now use the AdamOptimizer to minimize the loss.
Note that optimization is not performed at this point. In fact, nothing is calculated at all, we just add the optimizer-object to the TensorFlow graph for later execution.
Can you write Python code for it?


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Performance Measures
We need a few more performance measures to display the progress to the user.
First we calculate the predicted class number from the output of the Neural Network y_pred, which is a vector with 10 elements. The class number is the index of the largest element.
Can you write Python code for it?

y_pred_cls = tf.argmax(y_pred, dimension=1)


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Performance after 1 optimization iteration
The classification accuracy does not improve much from just 1 optimization iteration, because the learning-rate for the optimizer is set very low.
Can you write Python code for it?


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Performance after 10,000 optimization iterations
After 10,000 optimization iterations, the model has a classification accuracy on the test-set of about 99%.
Can you write Python code for it?

optimize(num_iterations=9000) # We performed 1000 iterations above.

print_test_accuracy(show_example_errors=True,
                    show_confusion_matrix=True)


optimize(num_iterations=900) # We performed 1000 iterations above.

print_test_accuracy(show_example_errors=True,
                    show_confusion_matrix=True)


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Performance after 100 optimization iterations
After 100 optimization iterations, the model has significantly improved its classification accuracy.
Can you write Python code for it?

optimize(num_iterations=99) # We already performed 1 iteration above.

print_test_accuracy(show_example_errors=True)


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Performance after 1000 optimization iterations
After 1000 optimization iterations, the model has greatly increased its accuracy on the test-set to more than 90%.
Can you write Python code for it?

optimize(num_iterations=900) # We performed 100 iterations above.

print_test_accuracy(show_example_errors=True)


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Performance before any optimization
The accuracy on the test-set is very low because the model variables have only been initialized and not optimized at all, so it just classifies the images randomly.
Can you write Python code for it?


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Plot a few images to see if data is correct
Can you write Python code for it?

# Get the first images from the test-set.
images = data.test.images[0:9]

# Get the true classes for those images.
cls_true = data.test.cls[0:9]

# Plot the images and labels using our helper-function above.
plot_images(images=images, cls_true=cls_true)


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
PrettyTensor Implementation
This section shows how to make the exact same implementation of a Convolutional Neural Network using PrettyTensor.
The basic idea is to wrap the input tensor x_image in a PrettyTensor object which has helper-functions for adding new computational layers so as to create an entire Neural Network. This is a bit similar to the helper-functions we implemented above, but it is even simpler because PrettyTensor also keeps track of each layer's input and output dimensionalities, etc.
Can you write Python code for it?


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
TensorFlow Graph
The entire purpose of TensorFlow is to have a so-called computational graph that can be executed much more efficiently than if the same calculations were to be performed directly in Python. TensorFlow can be more efficient than NumPy because TensorFlow knows the entire computation graph that must be executed, while NumPy only knows the computation of a single mathematical operation at a time.
TensorFlow can also automatically calculate the gradients that are needed to optimize the variables of the graph so as to make the model perform better. This is because the graph is a combination of simple mathematical expressions so the gradient of the entire graph can be calculated using the chain-rule for derivatives.
TensorFlow can also take advantage of multi-core CPUs as well as GPUs - and Google has even built special chips just for TensorFlow which are called TPUs (Tensor Processing Units) and are even faster than GPUs.
A TensorFlow graph consists of the following parts which will be detailed below:

Placeholder variables used for inputting data to the graph.
Variables that are going to be optimized so as to make the convolutional network perform better.
The mathematical formulas for the convolutional network.
A cost measure that can be used to guide the optimization of the variables.
An optimization method which updates the variables.

In addition, the TensorFlow graph may also contain various debugging statements e.g. for logging data to be displayed using TensorBoard, which is not covered in this tutorial.
Placeholder variables
Placeholder variables serve as the input to the TensorFlow computational graph that we may change each time we execute the graph. We call this feeding the placeholder variables and it is demonstrated further below.
First we define the placeholder variable for the input images. This allows us to change the images that are input to the TensorFlow graph. This is a so-called tensor, which just means that it is a multi-dimensional array. The data-type is set to float32 and...
Can you write Python code for it?

x = tf.placeholder(tf.float32, shape=[None, img_size_flat], name='x')


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
TensorFlow Implementation
This section shows the original source-code from Tutorial #02 which implements the Convolutional Neural Network directly in TensorFlow. The code is not actually used in this Notebook and is only meant for easy comparison to the PrettyTensor implementation below.
The thing to note here is how many lines of code there are and the low-level details of how TensorFlow stores its data and performs the computation. It is easy to make mistakes even for fairly small Neural Networks.
Helper-functions
In the direct TensorFlow implementation, we first make some helper-functions which will be used several times in the graph construction.
These two functions create new variables in the TensorFlow graph that will be initialized with random values.
Can you write Python code for it?

def new_weights(shape):
    return tf.Variable(tf.truncated_normal(shape, stddev=0.05))

def new_biases(length):
    return tf.Variable(tf.constant(0.05, shape=[length]))


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
TensorFlow Run
Create TensorFlow session
Once the TensorFlow graph has been created, we have to create a TensorFlow session which is used to execute the graph.
Can you write Python code for it?


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
That's it! We have now created the exact same Convolutional Neural Network in a few simple lines of code that required many complex lines of code in the direct TensorFlow implementation.
Using PrettyTensor instead of TensorFlow, we can clearly see the network structure and how the data flows through the network. This allows us to focus on the main ideas of the Neural Network rather than low-level implementation details. It is simple and pretty!
Getting the Weights
Unfortunately, not everything is pretty when using PrettyTensor.
Further below, we want to plot the weights of the convolutional layers. In the TensorFlow implementation we had created the variables ourselves so we could just refer to them directly. But when the network is constructed using PrettyTensor, all the variables of the layers are created indirectly by PrettyTensor. We therefore have to retrieve the variables from TensorFlow.
We used the names layer_conv1 and layer_conv2 for the two convolutional layers. These are also called variable scopes (not to be confused with defaults_scope as described above). PrettyTensor automatically gives names to the variables it creates for each layer, so we can retrieve the weights for a layer using the layer's scope-name and the variable-name.
The implementation is somewhat awkward because we have to use the TensorFlow function get_variable() which was designed for another purpose; either creating a new variable or re-using an existing variable. The easiest thing is to make the following helper-function.
Can you write Python code for it?

def get_weights_variable(layer_name):
    # Retrieve an existing variable named 'weights' in the scope
    # with the given layer_name.
    # This is awkward because the TensorFlow function was
    # really intended for another purpose.

    with tf.variable_scope(layer_name, reuse=True):
        variable = tf.get_variable('weights')
        bias     = tf.get_variable('bias')

    return variable, bias


def get_weights_variable(layer_name):
    # Retrieve an existing variable named 'weights' in the scope
    # with the given layer_name.
    # This is awkward because the TensorFlow function was
    # really intended for another purpose.

    with tf.variable_scope(layer_name, reuse=True):
        variable = tf.get_variable('weights')

    return variable


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
The MNIST data-set has now been loaded and consists of 70,000 images and associated labels (i.e. classifications of the images). The data-set is split into 3 mutually exclusive sub-sets. We will only use the training and test-sets in this tutorial.
Can you write Python code for it?

print("Size of:")
print("- Training-set:\t\t{}".format(len(data.train.labels)))
print("- Test-set:\t\t{}".format(len(data.test.labels)))
print("- Validation-set:\t{}".format(len(data.validation.labels)))


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
The class-labels are One-Hot encoded, which means that each label is a vector with 10 elements, all of which are zero except for one element. The index of this one element is the class-number, that is, the digit shown in the associated image. We also need the class-numbers as integers for the test-set, so we calculate it now.
Can you write Python code for it?

data.test.cls = np.argmax(data.test.labels, axis=1)


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
The classification accuracy is calculated by first type-casting the vector of booleans to floats, so that False becomes 0 and True becomes 1, and then taking the average of these numbers.
Can you write Python code for it?

accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
The convolutional layers expect x to be encoded as a 4-dim tensor so we have to reshape it so its shape is instead [num_images, img_height, img_width, num_channels]. Note that img_height == img_width == img_size and num_images can be inferred automatically by using -1 for the size of the first dimension. So the reshape operation is:
Can you write Python code for it?

x_image = tf.reshape(x, [-1, img_size, img_size, num_channels])


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
The following helper-function creates a fully-connected layer.
Can you write Python code for it?

def new_fc_layer(input,          # The previous layer.
                 num_inputs,     # Num. inputs from prev. layer.
                 num_outputs,    # Num. outputs.
                 use_relu=True): # Use Rectified Linear Unit (ReLU)?

    # Create new weights and biases.
    weights = new_weights(shape=[num_inputs, num_outputs])
    biases = new_biases(length=num_outputs)

    # Calculate the layer as the matrix multiplication of
    # the input and weights, and then add the bias-values.
    layer = tf.matmul(input, weights) + biases

    # Use ReLU?
    if use_relu:
        layer = tf.nn.relu(layer)

    return layer


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
The following helper-function creates a new convolutional network. The input and output are 4-dimensional tensors (aka. 4-rank tensors). Note the low-level details of the TensorFlow API, such as the shape of the weights-variable. It is easy to make a mistake somewhere which may result in strange error-messages that are difficult to debug.
Can you write Python code for it?

def new_conv_layer(input,              # The previous layer.
                   num_input_channels, # Num. channels in prev. layer.
                   filter_size,        # Width and height of filters.
                   num_filters,        # Number of filters.
                   use_pooling=True):  # Use 2x2 max-pooling.

    # Shape of the filter-weights for the convolution.
    # This format is determined by the TensorFlow API.
    shape = [filter_size, filter_size, num_input_channels, num_filters]

    # Create new weights aka. filters with the given shape.
    weights = new_weights(shape=shape)

    # Create new biases, one for each filter.
    biases = new_biases(length=num_filters)

    # Create the TensorFlow operation for convolution.
    # Note the strides are set to 1 in all dimensions.
    # The first and last stride must always be 1,
    # because the first is for the image-number and
    # the last is for the input-channel.
    # But e.g. strides=[1, 2, 2, 1] would mean that the filter
    # is moved 2 pixels across the x- and y-axis of the image.
    # The padding is set to 'SAME' which means the input image
    # is padded with zeroes so the size of the output is the same.
    layer = tf.nn.conv2d(input=input,
                         filter=weights,
                         strides=[1, 1, 1, 1],
                         padding='SAME')

    # Add the biases to the results of the convolution.
    # A bias-value is added to each filter-channel.
    layer += biases

    # Use pooling to down-sample the image resolution?
    if use_pooling:
        # This is 2x2 max-pooling, which means that we
        # consider 2x2 windows and select the largest value
        # in each window. Then we move 2 pixels to the next window.
        layer = tf.nn.max_pool(value=layer,
                               ksize=[1, 2, 2, 1],
                               strides=[1, 2, 2, 1],
                               padding='SAME')

    # Rectified Linear Unit (ReLU).
    # It calculates max(x, 0) for each input...


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
The following helper-function flattens a 4-dim tensor to 2-dim so we can add fully-connected layers after the convolutional layers.
Can you write Python code for it?

def flatten_layer(layer):
    # Get the shape of the input layer.
    layer_shape = layer.get_shape()

    # The shape of the input layer is assumed to be:
    # layer_shape == [num_images, img_height, img_width, num_channels]

    # The number of features is: img_height * img_width * num_channels
    # We can use a function from TensorFlow to calculate this.
    num_features = layer_shape[1:4].num_elements()

    # Reshape the layer to [num_images, num_features].
    # Note that we just set the size of the second dimension
    # to num_features and the size of the first dimension to -1
    # which means the size in that dimension is calculated
    # so the total size of the tensor is unchanged from the reshaping.
    layer_flat = tf.reshape(layer, [-1, num_features])

    # The shape of the flattened layer is now:
    # [num_images, img_height * img_width * num_channels]

    # Return both the flattened layer and the number of features.
    return layer_flat, num_features


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
The input image is processed in the first convolutional layer using the filter-weights. This results in 16 new images, one for each filter in the convolutional layer. The images are also down-sampled so the image resolution is decreased from 28x28 to 14x14.
These 16 smaller images are then processed in the second convolutional layer. We need filter-weights for each of these 16 channels, and we need filter-weights for each output channel of this layer. There are 36 output channels so there are a total of 16 x 36 = 576 filters in the second convolutional layer. The resulting images are down-sampled again to 7x7 pixels.
The output of the second convolutional layer is 36 images of 7x7 pixels each. These are then flattened to a single vector of length 7 x 7 x 36 = 1764, which is used as the input to a fully-connected layer with 128 neurons (or elements). This feeds into another fully-connected layer with 10 neurons, one for each of the classes, which is used to determine the class of the image, that is, which number is depicted in the image.
The convolutional filters are initially chosen at random, so the classification is done randomly. The error between the predicted and true class of the input image is measured as the so-called cross-entropy. The optimizer then automatically propagates this error back through the Convolutional Network using the chain-rule of differentiation and updates the filter-weights so as to improve the classification error. This is done iteratively thousands of times until the classification error is sufficiently low.
These particular filter-weights and intermediate images are the results of one optimization run and may look different if you re-run this Notebook.
Note that the computation in TensorFlow is actually done on a batch of images instead of a single image, which makes the computation more efficient. This means the flowchart actually has one more data-dimension when implemented in TensorFlow.
Imports
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
import tensorflow as tf
import numpy as np
from sklearn.metrics import confusion_matrix
import time
from datetime import timedelta
import math

# We also need PrettyTensor.
import prettytensor as pt


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Then we create a vector of booleans telling us whether the predicted class equals the true class of each image.
Can you write Python code for it?

correct_prediction = tf.equal(y_pred_cls, y_true_cls)


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
There are 16 input channels to the second convolutional layer, so we can make another 15 plots of filter-weights like this. We just make one more with the filter-weights for the second channel.
Can you write Python code for it?


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
This was developed using Python 3.5.2 (Anaconda) and TensorFlow version:
Can you write Python code for it?


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Using this helper-function we can retrieve the variables. These are TensorFlow objects. In order to get the contents of the variables, you must do something like: contents = session.run(weights_conv1) as demonstrated further below.
Can you write Python code for it?

weights_conv1 = get_weights_variable(layer_name='layer_conv1')
weights_conv2 = get_weights_variable(layer_name='layer_conv2')


weights_conv1, biases_conv1 = get_weights_variable(layer_name='layer_conv1')
#weights_conv2 = get_weights_variable(layer_name='layer_conv2')


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
Visualization of Weights and Layers
When the Convolutional Neural Network was implemented directly in TensorFlow, we could easily plot both the convolutional weights and the images that were output from the different layers. When using PrettyTensor instead, we can also retrieve the weights as shown above, but we cannot so easily retrieve the images that are output from the convolutional layers. So in the following we only plot the weights.
Helper-function for plotting convolutional weights
Can you write Python code for it?

def plot_conv_weights(weights, input_channel=0):
    # Assume weights are TensorFlow ops for 4-dim variables
    # e.g. weights_conv1 or weights_conv2.

    # Retrieve the values of the weight-variables from TensorFlow.
    # A feed-dict is not necessary because nothing is calculated.
    w = session.run(weights)

    # Get the lowest and highest values for the weights.
    # This is used to correct the colour intensity across
    # the images so they can be compared with each other.
    w_min = np.min(w)
    w_max = np.max(w)

    # Number of filters used in the conv. layer.
    num_filters = w.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    # Plot all the filter-weights.
    for i, ax in enumerate(axes.flat):
        # Only plot the valid filter-weights.
        if i<num_filters:
            # Get the weights for the i'th filter of the input channel.
            # See new_conv_layer() for details on the format
            # of this 4-dim tensor.
            img = w[:, :, input_channel, i]

            # Plot image.
            ax.imshow(img, vmin=w_min, vmax=w_max,
                      interpolation='nearest', cmap='seismic')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


def plot_conv_weights_bias(weights, bias, input_channel=0):
    # Assume weights are TensorFlow ops for 4-dim variables
    # e.g. weights_conv1 or weights_conv2.

    # Retrieve the values of the weight-variables from TensorFlow.
    # A feed-dict is not necessary because nothing is calculated.
    w = session.run(weights)
    b = session.run(bias)

    # Get the lowest and highest values for the weights.
    # This is used to correct the colour intensity across
    # the images so they can be compared with each other.
    w_min = np.min(w)
    w_max = np.max(w)

    # Number of filters used in the conv. layer.
    num_filters = w.shape[3]

    # Number of grids to plot.
    # Rounded-up, square-root of the number of filters.
    num_grids = math.ceil(math.sqrt(num_filters))

    # Create figure with a grid of sub-plots.
    fig, axes = plt.subplots(num_grids, num_grids)

    print(b)

    # Plot all the filter-weights.
    for i, ax in enumerate(axes.flat):
        # Only plot the valid filter-weights.
        if i<num_filters:
            # Get the weights for the i'th filter of the input channel.
            # See new_conv_layer() for details on the format
            # of this 4-dim tensor.
            img = w[:, :, input_channel, i]

            # Plot image.
            ax.imshow(img, vmin=w_min, vmax=w_max,
                      interpolation='nearest', cmap='seismic')

        # Remove ticks from the plot.
        ax.set_xticks([])
        ax.set_yticks([])

    # Ensure the plot is shown correctly with multiple plots
    # in a single Notebook cell.
    plt.show()


I am working on the file "03_PrettyTensor.ipynb".
The first task is:
We could also have a placeholder variable for the class-number, but we will instead calculate it using argmax. Note that this is a TensorFlow operator so nothing is calculated at this point.
Can you write Python code for it?

y_true_cls = tf.argmax(y_true, dimension=1)


I am working on the file "03_Python/Notebooks/Session2.ipynb".
The first task is:
while loops

while i < 10:
Repeats indented code until the expression is no longer true
Value in the expression must change; possibility for an infinite loop
Can you write Python code for it?

#WhileLoopExample.py

# This example demonstrates how a while loop is used. Here, we
# calculate the area of several circle with a radius 'r'. We loop
# through gradually larger values of r until the area of the circle
# exceeds 1000.

pi = 3.1415
r = 1
area = (r ** 2) * pi

while area < 1000:
    print r, area          # Indentation indicates what's run in the loop
    r = r + 1              # The variable that gets evaluated must change in the
    area = (r ** 2) * pi   # loop otherwise you'll create an infinite loop!

print "The while loop is done"  # Dedented lines run after the loop completes


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Create a variable
Can you write Python code for it?


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Datatypes
In computer programming, a data type is a classification identifying one of various types that data
can have.
The most common data type we will see in this class are:

Integers (int): Integers are the classic cardinal numbers: ... -3, -2, -1, 0, 1, 2, 3, 4, ...

Floating Point (float): Floating Point are numbers with a decimal point: 1.2, 34.98, -67,23354435, ...

Booleans (bool): Booleans types can only have one of two values: True or False. In many languages 0 is considered False, and any other value is considered True.

Strings (str): Strings can be composed of one or more characters: ’a’, ’spam’, ’spam spam eggs and spam’. Usually quotes (’) are used to specify a string. For example ’12’ would refer to the string, not the integer.

Collections of Data Types

Scalar: A single value of any data type.

List: A collection of values. May be mixed data types. (1, 2.34, ’Spam’, True) including lists of lists: (1, (1,2,3), (3,4))

Array: A collection of values. Must be same data type. [1,2,3,4] or [1.2, 4.5, 2.6] or [True, False, False] or [’Spam’, ’Eggs’, ’Spam’]

Matrix: A multi-dimensional array: [[1,2], [3,4]] (an array of arrays).
Can you write Python code for it?

a = 1
b = 2.3
c = True
d = "Spam"

type(a), type(b), type(c), type(d)

a + b, type(a + b)

a + c, type(a + c)    # True = 1

a + d

str(a) + d


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Fancy masking
Can you write Python code for it?

mask3 = np.where(x >= 8)
x[mask3]

# Set all values of x that match mask3 to 0

x[mask3] = 0
x

mask4 = np.where(x != 0)
mask4

#Add 10 to every value of x that matches mask4:

x[mask4] += 100
x


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
For loops are different in python.
You do not need to specify the beginning and end values of the loop
Can you write Python code for it?

for Value in planet_table['Name']:
    print(Value)

for Idx,Val in enumerate(planet_table['Name']):
    print(Idx,Val)

for Idx,Val in enumerate(planet_table['Name']):

    distance = planet_table['a'][Idx]

    S = "The planet " + str(Val) + " is at a distance of " + str(distance) + " AU"

    print(S)


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Functions
In computer science, a function (also called a procedure, method, subroutine, or routine) is a portion
of code within a larger program that performs a specific task and is relatively independent of the
remaining code. The big advantage of a function is that it breaks a program into smaller, easier
to understand pieces. It also makes debugging easier. A function can also be reused in another
program.
The basic idea of a function is that it will take various values, do something with them, and return a result. The variables in a function are local. That means that they do not affect anything outside the function.
Below is an example of a function that finds the perihelion (closest point to the Sun) for an orbit with a semi-major axis of a and an eccntricity of e
$ \mathrm{Perihelion\ Distance\ (a,ecc)} = a\ (1 - ecc)$
In the example the name of the function is find_peri (you can name functions what ever you want). The function find_peri takes two arguments semi_major and eccentricity, and returns the value of the equation to the main program. In the main program a variable named perihelion_distance is assigned the value returned by find_peri. Notice that in the main program the function find_peri is called using the arguments planet_table[a] and planet_table[ecc]. Since the variables in the function are local, you do not have name them semi_major and eccentricity in the main program.
Can you write Python code for it?

def find_peri(semi_major,eccentricity):

    result = semi_major * (1.0 - eccentricity)          # assign the variable result the value of the function
    return result                                       # return the value of the function to the main program

perihelion_distance = find_peri(planet_table['a'],planet_table['ecc'])

perihelion_distance

planet_table['perihelion'] = perihelion_distance

print(planet_table)


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Help about a function:
Can you write Python code for it?


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Here is a link to all Numpy math functions.
Our basic unit will be the NumPy array
Can you write Python code for it?

np.random.seed(42)                 # set the seed - everyone gets the same random numbers
x = np.random.randint(1,10,20)     # 20 random ints between 1 and 10
x


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Indexing
Can you write Python code for it?

x[0]    # The 0th element of the array x

x[-1]    # The last element of the array x


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Loops are slow in Python. Do not use them if you do not have to!
Can you write Python code for it?

np.random.seed(42)
BigZ = np.random.random(10000)    # 10,000 value array
BigZ[:10]

# This is slow!

for Idx,Val in enumerate(BigZ):
    if (Val > 0.5):
        BigZ[Idx] = 0

BigZ[:10]

%%timeit

for Idx,Val in enumerate(BigZ):
    if (Val > 0.5):
        BigZ[Idx] = 0

# Masks are MUCH faster

mask = np.where(BigZ>0.5)
BigZ[mask] = 0

BigZ[:10]

%%timeit -o

mask = np.where(BigZ>0.5)
BigZ[mask] = 0


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Masking - The key to fast programs
Can you write Python code for it?

mask1 = np.where(x>5)
x, mask1

x[mask1], y[mask1]

mask2 = np.where((x>3) & (x<7))
x[mask2]


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
NumPy math works over an entire array:
Can you write Python code for it?

y = x * 2
y

sin(x)     # need to Numpy's math functions

np.sin(x)


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Reading in data - The AstroPy package
Can you write Python code for it?

from astropy.table import QTable

!cat Planets.csv

planet_table = QTable.read('Planets.csv', format='ascii.csv')

planet_table

print(planet_table)


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Renaming columns
Can you write Python code for it?

planet_table.rename_column('col2', 'ecc')
print(planet_table)

planet_table['Name']

planet_table['Name'][0]


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Saving a table
Can you write Python code for it?

planet_table.write('NewPlanets.csv', format='ascii.csv')

!cat NewPlanets.csv


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Slices
x[start:stop:step]

start is the first item that you want [default = first element]
stop  is the first item that you do not want [default = last element]
step  defines size of step and whether you are moving forwards (positive) or backwards (negative) [default = 1]
Can you write Python code for it?

x

x[0:4]           # first 4 items

x[:4]            # same

x[0:4:2]         # first four item, step = 2

x[3::-1]         # first four items backwards, step = -1

x[::-1]          # Reverse the array x

print(x[-5:])    # last 5 elements of the array x


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
Sorting
Can you write Python code for it?


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
There are lots of different methods that can be applied to a NumPy array
Can you write Python code for it?

x.size                   # Number of elements in x

x.mean()                 # Average of the elements in x

x.sum()                  # Total of the elements in x

x[-5:].sum()              # Total of last 5 elements in x

x.cumsum()                # Cumulative sum

x.cumsum()/x.sum()        # Cumulative percentage

x.


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
UNIX commands can be run by placing a ! before the command:
Can you write Python code for it?


I am working on the file "03_Python_Introduction.ipynb".
The first task is:
or even easier:
Can you write Python code for it?


I am working on the file "03_Units_In_Python.ipynb".
The first task is:
An Example: The velocity of an object in circular orbit around the Sun is
$$ v=\sqrt{GM_{\odot}\over d} $$
Can you write Python code for it?

distance = planet_table['a'][0:3]  # Mercury, Venus, Earth

orbit_v = np.sqrt(const.G * const.M_sun / distance)

orbit_v

orbit_v.decompose()

orbit_v.to(u.km/u.s)

orbit_v.to(ringo/u.ms)


I am working on the file "03_Units_In_Python.ipynb".
The first task is:
Dimentionless Units
Can you write Python code for it?

dimless_y = (1 * u.m) / (1 * u.km)

dimless_y

dimless_y.unit

dimless_y.decompose()   # returns the scale of the dimentionless quanity


I am working on the file "03_Units_In_Python.ipynb".
The first task is:
Note: because we imported the units package as u, you cannot use u as a variable name.
You can use the units by themselves:
Can you write Python code for it?

u.m    # The unit of meters

u.s    # The unit of seconds

u.m / u.s    # combine them into a composite unit


I am working on the file "03_Units_In_Python.ipynb".
The first task is:
Notice that the units are a bit strange. We can simplify this using .decompose()
Can you write Python code for it?


I am working on the file "03_Units_In_Python.ipynb".
The first task is:
Or they expect the correct type of unit!
Can you write Python code for it?


I am working on the file "03_Units_In_Python.ipynb".
The first task is:
The units package is much more useful when you combine it with scalars or arrays to create Quantities
Can you write Python code for it?

time_1 = 0.25 * u.s

time_1

position = np.arange(1,6,1) * u.m    # np.arange(x,y,z) - create an array of numbers between x and y in steps z

position

velocity = position / time_1

velocity


I am working on the file "03_Units_In_Python.ipynb".
The first task is:
Unit conversion is really easy!
Can you write Python code for it?

velocity.to(u.cm / u.h)

velocity.to(imperial.mi / u.h)

velocity.si                     # quick conversion to SI units

velocity.cgs                    # quick conversion to CGS units

density = 3000 * (u.kg / u.m**3)     # From last week's homework

density.to(u.kg / u.km**3)


I am working on the file "03_Units_In_Python.ipynb".
The first task is:
Unit conversion works with Time as well
Can you write Python code for it?

time_2 = 123 * u.day + 12 * u.h + 36 * u.min + 12 * u.s

time_2

time_2.to(u.year)

time_2.to(u.s)


I am working on the file "03_Units_In_Python.ipynb".
The first task is:
Using units can save you headaches.

All of the trig functions expect all angles to be in radians.
If you forget this, it can lead to problems that are hard to debug
Can you write Python code for it?

np.sin(90)               # not really what I expected

np.sin(90 * u.deg)       # better


I am working on the file "03_Units_In_Python.ipynb".
The first task is:
You can access the number and unit part of the Quantity separately:
Can you write Python code for it?


I am working on the file "03_Units_In_Python.ipynb".
The first task is:
You can define your own units
Can you write Python code for it?

ringo = u.def_unit('Ringos', 3.712 * imperial.yd)

position.to(ringo)

velocity.to(ringo / u.s)


I am working on the file "03_Visualizar_datos/Visualizacion de datos con Python.ipynb".
The first task is:
3. Gráfico de barras
Otro gráfico ampliamente usado es el diagrama de barras o gráfico de barras. Es una representación gráfica de un conjunto de datos o valores, y está conformado por barras rectangulares de longitudes proporcionales a los valores representados.
A diferencia de un histograma, el gráfico de barras se construye usando variables categóricas o cualitativas.
Digamos que nos interesa separar los vuelos en cortos, intermedios y largos de acuerdo a la distancia recorrida. Para ello, creamos tres categorías y hacemos un gráfico de barras.
Can you write Python code for it?

bins = [0, 2000, 10000, 1000000]
group_names = ['Vuelo corto', 'Vuelo intermedio', 'Vuelo largo']
rutas['Tipo_vuelo'] = pd.cut(rutas['Distancia_km'], bins, labels=group_names)

sns.countplot(x="Tipo_vuelo", order=group_names, data=rutas)
plt.xlabel('Tipo de vuelo')
plt.ylabel('Número de vuelos')
plt.title('Gráfico de barras')
plt.xticks(size = 15, rotation=60)


I am working on the file "03_Visualizar_datos/Visualizacion de datos con Python.ipynb".
The first task is:
4. Gráfico de cajas
Otro gráfico usado es el diagrama de caja o gráfico de caja. Es un gráfico que está basado en cuartiles y mediante el cual se visualiza la distribución de un conjunto de datos. Está compuesto por un rectángulo (caja) y dos brazos (bigotes).
Digamos que nos interesa conocer cómo se distribuyen los vuelos en cada país de Sudamérica. Para ello usamos la variable País de origen como variable categórica y la variable Distancia recorrida para estudiar su distribución.
Can you write Python code for it?

sns.boxplot(x='Pais_origen', y='Distancia_km', order=rutas_sudamerica_pais.index.tolist(), data=rutas_sudamerica)
plt.xlabel('País de origen')
plt.ylabel('Distancia recorrida (km)')
plt.title('Gráfico de cajas')
plt.xticks(size = 15, rotation=60)


I am working on the file "03_Visualizar_datos/Visualizacion de datos con Python.ipynb".
The first task is:
4. Gráfico de puntos
Otro gráfico usado es el diagrama de dispersión o gráfico de puntos. Es un gráfico que utiliza las coordenadas cartesianas para mostrar los valores de dos variables de un conjunto de datos.
Cabe notar que un gráfico de puntos se construye usando variables cuantitativas.
Supongamos que nos interesa saber si los vuelos que parten desde el hemisferio Norte son más largos que los vuelos que parten desde el hemisferio Sur. Para ello usamos la variable Latitud País de origen como variable dependiente y la variable Distancia recorrida como variable independiente.
Can you write Python code for it?

rutas_coordenadas=rutas.merge(aeropuertos[['Id','Latitud','Longitud']], how='inner', left_on='Aeropuerto_origen_Id', right_on='Id')
rutas_coordenadas=rutas_coordenadas.sample(10000)

rutas_coordenadas.plot.scatter(x='Latitud', y='Distancia_km', s=3);
plt.xlim(-90,90)
plt.xlabel('Latitud País de origen')
plt.ylabel('Distancia recorrida (km)')
plt.title('Gráfico de puntos - Vuelos')
plt.xticks(size = 15, rotation=0);


I am working on the file "03_Visualizar_datos/Visualizacion de datos con Python.ipynb".
The first task is:
Digamos que nos interesa conocer el número de vuelos que parten desde algún país en Sudamérica. La mejor manera de visualizar el número de vuelos por país es usando un gráfico de barras.
Can you write Python code for it?

paises_sudamerica=['Brazil','Colombia','Argentina','Peru','Venezuela','Chile','Ecuador','Bolivia','Paraguay','Uruguay','Guyana','Suriname','French Guiana']
rutas_sudamerica=rutas[rutas['Pais_origen'].isin(paises_sudamerica)]
rutas_sudamerica_pais=rutas_sudamerica.groupby('Pais_origen')['Linea_aerea'].agg('count').sort_values(ascending=False)

sns.countplot(x="Pais_origen", order=rutas_sudamerica_pais.index.tolist(), data=rutas_sudamerica)
plt.xlabel('País de origen')
plt.ylabel('Número de vuelos')
plt.title('Gráfico de barras')
plt.xticks(size = 15, rotation=60)


I am working on the file "03_Visualizar_datos/Visualizacion de datos con Python.ipynb".
The first task is:
Gracias por su atención
Preparación de datasets
Can you write Python code for it?

aeropuertos_file='data/airports.dat'
aeropuertos=pd.read_csv(aeropuertos_file, header=None)

aeropuertos.head()

columnas=['Id','Nombre','Ciudad','Pais','IATA','OACI','Latitud','Longitud','Altitud','Zona_horaria','Horario_verano','Zona_horaria_lugar','Tipo','Fuente']
aeropuertos.columns = columnas
aeropuertos = aeropuertos.replace("\\N", value=np.nan)
aeropuertos['Altitud']=np.round(aeropuertos['Altitud']*0.3048,1)
aeropuertos[['Zona_horaria','Altitud']] = aeropuertos[['Zona_horaria','Altitud']].apply(pd.to_numeric)
aeropuertos.head()

aeropuertos.dtypes

aeropuertos.to_csv('data/aeropuertos.csv', index=False)

rutas_file='data/routes.dat'
rutas=pd.read_csv(rutas_file, header=None)

rutas.head()

columnas=['Linea_aerea','Linea_aerea_Id','Aeropuerto_origen','Aeropuerto_origen_Id','Aeropuerto_destino','Aeropuerto_destino_Id','Vuelo_operado','Numero_paradas','Modelo_avion']
rutas.columns = columnas
rutas = rutas[(rutas["Linea_aerea_Id"] != "\\N") & (rutas["Aeropuerto_origen_Id"] != "\\N") & (rutas["Aeropuerto_destino_Id"] != "\\N")]
rutas[['Linea_aerea_Id','Aeropuerto_origen_Id','Aeropuerto_destino_Id']] = rutas[['Linea_aerea_Id','Aeropuerto_origen_Id','Aeropuerto_destino_Id']].apply(pd.to_numeric)
rutas.head()

rutas.dtypes

def pais_aeropuerto(row, name):
    result='Unknown'
    try:
        result = aeropuertos[aeropuertos["Id"] == row[name]].iloc[0]
        result = result['Pais']
    except (ValueError, IndexError):
        pass
    return result

rutas['Pais_origen']=rutas.apply(pais_aeropuerto, name='Aeropuerto_origen_Id', axis=1)
rutas['Pais_destino']=rutas.apply(pais_aeropuerto, name='Aeropuerto_destino_Id', axis=1)

import math

def haversine(lon1, lat1, lon2, lat2):
    # Convert coordinates to floats.
    lon1, lat1, lon2, lat2 = [float(lon1), float(lat1), float(lon2), float(lat2)]
    # Convert to radians from degrees.
    lon1, lat1, lon2, lat2 = map(math.radians, [lon1, lat1, lon2, lat2])
    # Compute distance.
    dlon = lon2 - lon1
    dlat = lat2 - lat1
    a = math.sin(dlat/2)**2 +...


I am working on the file "03_Visualizar_datos/Visualizacion de datos con Python.ipynb".
The first task is:
Gráficos extras
Can you write Python code for it?


I am working on the file "03_Visualizar_datos/Visualizacion de datos con Python.ipynb".
The first task is:
Lectura de datos
Para este tutorial usaremos una base de datos de vuelos áreos publicada por el sitio web Openflights. La base de datos contiene una tabla de los aeropuertos en el mundo (aeropuertos.csv), una tabla de las aerolíneas en el mundo (aerolineas.csv) y una tabla de las rutas de vuelos comerciales (rutas.csv).
Más información en https://github.com/jpatokal/openflights
Can you write Python code for it?

aeropuertos_file='data/aeropuertos.csv'
aerolineas_file='data/aerolineas.csv'
rutas_file='data/rutas.csv'

aeropuertos=pd.read_csv(aeropuertos_file)
aerolineas=pd.read_csv(aerolineas_file)
rutas=pd.read_csv(rutas_file)

aeropuertos.head()

aerolineas.head()

rutas.head()


I am working on the file "03_Visualizar_datos/Visualizacion de datos con Python.ipynb".
The first task is:
Visualización de datos
1. Histograma de frecuencia absoluta
Uno de los gráficos más usados para la visualización de datos es el histograma. Un histograma es una representación gráfica de una variable en forma de barras, donde la superficie de cada barra es proporcional a la frecuencia de los valores representados. Sirven para obtener una primera vista general de la distribución de la muestra de datos respecto a una característica.
Supongamos que nos interesa contar el número de vuelos de acuerdo a la distancia recorrida en kilómetros. Para ello construimos un histograma de la Distancia de los vuelos.
Can you write Python code for it?

n, bins, patches = plt.hist(rutas['Distancia_km'], bins=20)
plt.xlabel('Distancia recorrida (km)')
plt.ylabel('Número de vuelos')
plt.title('Histograma - Frecuencia absoluta')


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
But if you change the value of variable A as it is done blow, variable B changes, too.
Can you write Python code for it?


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
Decorator Syntax
Can you write Python code for it?

@greetings
def shout(sentence):
    print sentence.upper()

shout("Am I polite, or what?!")


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
Flatting Lists
Can you write Python code for it?

import itertools
a_list = [[1,2],[3,4],[5,6]]

a_flatted_list = [x for i in a_list for x in i]
print "a_flatted_list: ", a_flatted_list


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
If we apply this operation on a variable referring to an immutable object - a new object is created, but if we apply this to an object referring to a mutable object like lists, sets and so on we are just changing the object.
Can you write Python code for it?

A = [1,2,3]  # assigning a mutable list object to A
B = A
A += [1]
print "A: ", A
print "B: ", B


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
Inheritance
Classes can inherit traits (and even skills) from other classes:
Can you write Python code for it?

class Dachshund(Dog):
    color = "brownish"
    def __init__(self, name):
        super(Dachshund, self).__init__(name, 20)

d = Dachshund('bello')
d.bark()


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
Inverting dictionaries
Can you write Python code for it?

dictionary = dict(a=1, b=2, c=3)  # normal dict generation
print "dictionary: ", dictionary

# inverting dictionary using zip function
inverted_dict = dict(zip(dictionary.values(), dictionary.keys()))
print "inverted dict: ", inverted_dict


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
Itertools
Sliding window
Can you write Python code for it?

import itertools
def k_mer(word, size):
    z = (itertools.islice(word, i, None) for i in range(size))
    return zip(*z)

string = "HelloWorld"
k_mer(string,3)


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
Mating
Can you write Python code for it?

class Dog(object):
    color = "yellow"
    def __init__(self, name, size):
            self.name = name
            self.size = size

    def __add__(self, dog):
        if not isinstance(dog, Dog):
            raise TypeError
        return Dog(self.name + " " + dog.name, (self.size + dog.size)/2)

    def bark(self):
        print 'whoof, whoof!'

b = Dog("blitzer", 100)
dog = b + p
dog.bark()
print dog.name
print dog.size


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
Permutations
Can you write Python code for it?

import itertools
for p in itertools.permutations([1,2,3]):
    print "permutations: ", p


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
Shadowing
Classes can alter functionality of their parents:
Can you write Python code for it?

class Pekingese(Dog):
    color = "brownish"
    def __init__(self, name):
        super(Pekingese, self).__init__(name, 15)

    def bark(self):
        print 'whif.'

d = Pekingese('tiny')
d.bark()


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
So we end up with two objects and two variables referring to these objects, respectively.
But we have to pay attention when we use operations like +=. The behaviour of this kind of operations depends on the object it is applied on.
Can you write Python code for it?

A = ((1,2,3))  # assigning a immutable tuple object to A
B = A
A += (3,4,5)
print "A: ", A
print "B: ", B


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
Special methods
Can you write Python code for it?

class Pekingese(Dog):
    color = "brownish"
    def __init__(self, name):
        super(Pekingese, self).__init__(name, 15)

    def __len__(self):
        return self.size

    def bark(self):
        print 'whif.'

p = Pekingese("wang")
len(p)


I am working on the file "03_advanced/.ipynb_checkpoints/advanced_python_tricks-checkpoint.ipynb".
The first task is:
The difference in this example is, that we are not incrementing the value of A, but instead we are creating a new object and assigning this new object to our variable A. After this we have two objects the Integer 1 and the Integer 2 and two variables B and A referring to the objects, respectively.
Some operations change the object instead creating a copy of the object assigned to a variable.
Can you write Python code for it?

A = [[2,3,1]]
B = A
A[0].append(4)
print B
A[0].sort()
print B