I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
This makes a lot of sense. The image from the previous/next hour is the most similar.
Then, images from +/- 2 hours ...
What if we want to find the most similar image that is not within +/- 1 day?
Since we have only 1 year of data, we are not going to great analogs, possibly,
but let's see what we get.
Can you write Python code for it?

%%bigquery df
WITH ref1 AS (
SELECT time AS ref1_time, ref1_value, ref1_offset
FROM `ai-analytics-solutions.advdata.wxembed`,
     UNNEST(ref) AS ref1_value WITH OFFSET AS ref1_offset
WHERE time = '2019-09-20 05:00:00 UTC'
)

SELECT
  time,
  SUM( (ref1_value - ref[OFFSET(ref1_offset)]) * (ref1_value - ref[OFFSET(ref1_offset)]) ) AS sqdist
FROM ref1, `ai-analytics-solutions.advdata.wxembed`
WHERE time NOT BETWEEN '2019-09-19' AND '2019-09-21'
GROUP BY 1
ORDER By sqdist ASC
LIMIT 5

df


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
Try decoding
Can you write Python code for it?

%%bigquery df
SELECT *
FROM advdata.wxembed

df.head(n=5)


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
What about July 1, 2019, which is next on the list?
Can you write Python code for it?

import pandas as pd
dt = pd.Timestamp(df['time'].values[2])
# gs://high-resolution-rapid-refresh/hrrr.20200811/conus/hrrr.t04z.wrfsfcf00.grib2
FILENAME = '{}/hrrr.{:4}{:02}{:02}/conus/hrrr.t{:02}z.wrfsfcf00.grib2'.format(
                'gs://high-resolution-rapid-refresh',
                dt.year, dt.month, dt.day, dt.hour)
print(FILENAME)

import xarray as xr
import tensorflow as tf
import tempfile
import cfgrib
import numpy as np

refc = 0
with tempfile.TemporaryDirectory() as tmpdirname:
    TMPFILE="{}/read_grib".format(tmpdirname)
    tf.io.gfile.copy(FILENAME, TMPFILE, overwrite=True)
    #ds = xr.open_dataset(TMPFILE, engine='cfgrib', backend_kwargs={'filter_by_keys': {'typeOfLevel': 'surface', 'stepType': 'instant'}})
    #ds.data_vars['prate'].plot()  # crain, prate
    ds = xr.open_dataset(TMPFILE, engine='cfgrib', backend_kwargs={'filter_by_keys': {'typeOfLevel': 'unknown', 'stepType': 'instant'}})
    #ds = xr.open_dataset(TMPFILE, engine='cfgrib', backend_kwargs={'filter_by_keys': {'typeOfLevel': 'atmosphere', 'stepType': 'instant'}})
    refc = ds.data_vars['refc']
    refc.plot()


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
What does the original look like? Let's pull the original HRRR Grib file from this time stamp
Can you write Python code for it?

import pandas as pd
dt = pd.Timestamp(df['time'].values[0])
# gs://high-resolution-rapid-refresh/hrrr.20200811/conus/hrrr.t04z.wrfsfcf00.grib2
FILENAME = '{}/hrrr.{:4}{:02}{:02}/conus/hrrr.t{:02}z.wrfsfcf00.grib2'.format(
                'gs://high-resolution-rapid-refresh',
                dt.year, dt.month, dt.day, dt.hour)
print(FILENAME)

import xarray as xr
import tensorflow as tf
import tempfile
import cfgrib
import numpy as np

refc = 0
with tempfile.TemporaryDirectory() as tmpdirname:
    TMPFILE="{}/read_grib".format(tmpdirname)
    tf.io.gfile.copy(FILENAME, TMPFILE, overwrite=True)
    #ds = xr.open_dataset(TMPFILE, engine='cfgrib', backend_kwargs={'filter_by_keys': {'typeOfLevel': 'surface', 'stepType': 'instant'}})
    #ds.data_vars['prate'].plot()  # crain, prate
    ds = xr.open_dataset(TMPFILE, engine='cfgrib', backend_kwargs={'filter_by_keys': {'typeOfLevel': 'unknown', 'stepType': 'instant'}})
    #ds = xr.open_dataset(TMPFILE, engine='cfgrib', backend_kwargs={'filter_by_keys': {'typeOfLevel': 'atmosphere', 'stepType': 'instant'}})
    refc = ds.data_vars['refc']
    refc.plot()


I am working on the file "02_data_representation/weather_search/wx_embeddings.ipynb".
The first task is:
Write a Beam pipeline
Can you write Python code for it?

%run -m wxsearch.hrrr_to_tfrecord -- --startdate 20190915 --enddate 20190916  --outdir gs://{BUCKET}/wxsearch/data/2019 --project {PROJECT}
# --outdir tmp


I am working on the file "02_distances.ipynb".
The first task is:
CODING TIME: Generate the 3x3x3 superframe from the unit_frame
Can you write Python code for it?

def skeleton_superframe(frame, a):
    '''
    Generate a 3x3x3 supercell of a given frame.
    '''
    v = [-1, 0, 1]
    n = len(frame)
    unit = frame.loc[:, ['x', 'y', 'z']].values
    coords = np.empty((n * 27, 3))
    h = 0
    for i in v:
        for j in v:
            for k in v:
                #for ...
    return coords

big_frame = skeleton_superframe(unit_frame, a)
big_frame.shape


I am working on the file "02_distances.ipynb".
The first task is:
CODING TIME: Put all atoms back into the unit cell
Can you write Python code for it?

def skeleton_create_unit(df, a):
    '''
    Put all atoms back into the cubic unit cell
    '''
    #...

unit_frame = skeleton_create_unit(frame, a)    # a is defined above
unit_frame


I am working on the file "02_distances.ipynb".
The first task is:
Computing distances between atoms
This is arguably the most difficult part of this tutorial. How do we account for periodicity?
Lets start by considering free boundary conditions first!
Computing all atom to atom distances (per frame) increases factorially,
\begin{equation}
    \frac{nat!}{2!\left(nat - 2\right)!} \left(= \frac{1}{2}\left(nat * \left(nat - 1\right)\right)\right)
\end{equation}
in computations (where nat is the number of atoms). Fortunately for us, computing the distances can be
passed off to scipy's pdist.
CODING TIME: Write a function to compute all of the atom to atom distances in every frame assuming free boundary conditions
Can you write Python code for it?

from scipy.spatial.distance import pdist

def skeleton_free_boundary_distances(frame):    # Note that this is frame, not DataFrame
    '''
    Compute all of the atom to atom distances with free boundary conditions
    '''
    # Compute distances
    xyz = frame.loc[:, ['x', 'y', 'z']]
    distances = pdist(xyz)
    # Compute the symbols
    # ...
    symbols = None
    return pd.DataFrame.from_dict({'distances': distances, 'symbols': symbols})

twobody = xyz.groupby(level='frame').apply(skeleton_free_boundary_distances)
twobody.head()


I am working on the file "02_distances.ipynb".
The first task is:
Distances are no good to us, unless we know where they came from (or at least what two symbols they represent)...
HINT: Check out the "combinations" function in the itertools library (part of the Python standard library)
Can you write Python code for it?

from itertools import combinations

%load -s free_boundary_distances, snippets/distances.py

twobody = xyz.groupby(level='frame').apply(free_boundary_distances)
twobody.head()


I am working on the file "02_distances.ipynb".
The first task is:
Load our previous work
HDF5's are easy to load
Can you write Python code for it?

xyz = pd.read_hdf('xyz.hdf5', 'xyz')
xyz.head()


I am working on the file "02_distances.ipynb".
The first task is:
Now let's convert these Series (a pandas Series is simply a column in a DataFrame)
to symbols using the map function.
Can you write Python code for it?

symbols = unit_frame['symbol'].to_dict()

atom1_symbols = atom1_indices.map(symbols)
atom2_symbols = atom2_indices.map(symbols)

symbols = [''.join((first, atom2_symbols[i])) for i, first in enumerate(atom1_symbols)]


I am working on the file "02_distances.ipynb".
The first task is:
Now let's finish this by generating our (periodic) two body DataFrame for this first frame..
Can you write Python code for it?

frame_twobody = pd.DataFrame.from_dict({'distance': distances.flatten(),
                                        'symbols': symbols})

frame_twobody = frame_twobody.loc[(frame_twobody['distance'] > 0.3) & (frame_twobody['distance'] < 8.3)]
frame_twobody.head()


I am working on the file "02_distances.ipynb".
The first task is:
One solution...or if you want to have some fun...
Can you write Python code for it?

%load -s superframe, snippets/distances.py

big_frame = superframe(unit_frame, a)
big_frame


I am working on the file "02_distances.ipynb".
The first task is:
Periodicity
That was fun but it doesn't do what we need it to! Periodic boundaries can be handled a number of ways, here is one algorithm:

Put all atoms back in unit cell
Generate a 3x3x3 supercell from the unit cell
Compute distances looking only from the central unit cell (the internal cell that is completely surrounded by replicas)

Since this is complicated, we will walk through the pieces of the code individually (applying them to a single frame) before putting it all together.
Can you write Python code for it?


I am working on the file "02_distances.ipynb".
The first task is:
Tests again...
Can you write Python code for it?

twobody.loc[0].head()

first = xyz.loc[0, ['x', 'y', 'z']].values
for i in range(1, 6):
    print(((first[0, :] - first[i, :])**2).sum()**0.5)


I am working on the file "02_distances.ipynb".
The first task is:
The % (modulo) operator is nice for such tasks...
Can you write Python code for it?

%load -s create_unit, snippets/distances.py

unit_frame = create_unit(frame, a)
unit_frame.shape

print(unit_frame is not frame)                          # True if objects are not the same in memory
print(np.all(unit_frame.values == frame.values))        # True if objects' xyz positions are identical


I am working on the file "02_distances.ipynb".
The first task is:
What can we do with twobody data??
Can you write Python code for it?

twobody.loc[((twobody.symbols == 'HO') | (twobody.symbols == 'OH')) & (twobody.distance < 1.2), 'distance'].describe()

twobody.loc[((twobody.symbols == 'HO') | (twobody.symbols == 'OH')) & (twobody.distance < 1.2), 'distance'].median()


I am working on the file "02_distances.ipynb".
The first task is:
What should the function _compute look like/do?
NOT QUITE CODING TIME: Just load the function
Can you write Python code for it?

%load -s _compute, snippets/distances.py

# WARNING: On a fast machine, this operation takes ~10 s
%time twobody = cubic_periodic_distances(xyz, a, nat)
twobody.head()


I am working on the file "02_distribuciones_de_probabilidad.ipynb".
The first task is:
Distribución Gamma
Scipy parametriza a la distribución gamma usando un parámetro $\alpha$ y uno $\theta$, usando estos parámetros la $pdf$ es:
$$
p(x \mid \alpha, \theta) = \frac{1}{\Gamma(\alpha) \theta^\alpha} x^{\alpha \,-\, 1} e^{-\frac{x}{\theta}}
$$
Una parametrización más común en estadística Bayesiana usa los parámetros $\alpha$ y  $\beta$, siendo $\beta = \frac{1}{\theta}$. En este caso la pdf queda como:
$$
p(x \mid \alpha, \beta) = \frac{\beta^{\alpha}x^{\alpha-1}e^{-\beta x}}{\Gamma(\alpha)}
$$
La distribución gamma se reduce a la exponencial cuando $\alpha=1$.
Can you write Python code for it?

distri = stats.gamma(a=3, scale=0.5)  # alfa 3, theta 0.5
x = np.linspace(0, 8, 100)
x_rvs = distri.rvs(500)  # muestrear 500 valores de la distribución
x_pdf = distri.pdf(x)  # la pdf evaluada para todos los x
media, varianza = distri.stats(moments='mv')
plt.plot (x, x_pdf, lw=5,
           label='$\mu$  = {:3.1f}\n$\sigma$ = {:3.1f}'.format(float(media),
                                                               float(varianza)**0.5))
plt.hist(x_rvs, normed=True)

plt.xlabel('x')
plt.ylabel('$p(x)$')
plt.legend(frameon=True);


I am working on the file "02_distribuciones_de_probabilidad.ipynb".
The first task is:
Distribución acumulada
La $pdf$ (o la $pmf$) son formas comunes de representar y trabajar con variables aleatorias, pero no son las únicas formas posibles. Existen otras representaciones equivalentes. Por ejemplo la función de distribución acumulada ($cdf$ en inglés). Al integrar una $pdf$ se obtiene la correspondiente $cdf$, y al derivar la $cdf$ se obtiene la $pdf$.
La integral de la $pdf$ es llamada función de distribución acumulada ($cdf$):
\begin{equation}
cdf(x) = \int_{-\infty}^{x} pdf(x) d(x)
\end{equation}
En algunas situaciones se prefiere hablar de la función de supervivencia:
\begin{equation}
S(x) = 1 - cdf
\end{equation}
A continuación un ejemplo de la $pdf$ y $cdf$ para 4 distribuciones de la familia Gaussiana.
Can you write Python code for it?

_, ax = plt.subplots(2,1, figsize=(6, 8), sharex=True)
x_values = np.linspace(-4, 4, 200)
values = [(0., .2), (0., 1.), (0., 2.), (-2., .5)]
color = ['C0', 'C1', 'C2', 'C3']
for val, c in zip(values, color):
    pdf = stats.norm(*val).pdf(x_values)
    cdf = stats.norm(*val).cdf(x_values)
    ax[0].plot(x_values, pdf, lw=3, color=c,
               label='$\mu$ = {}, $\sigma$ = {}'.format(*val))
    ax[1].plot(x_values, cdf, lw=3, color=c)
ax[0].set_ylabel('$pdf$', fontsize=14, rotation=0, labelpad=20)
ax[0].legend()
ax[1].set_ylabel('$cdf$', fontsize=14, rotation=0, labelpad=20)
ax[1].set_xlabel('$x$');


I am working on the file "02_distribuciones_de_probabilidad.ipynb".
The first task is:
Distribución beta
Es una distribución definida en el intervalo [0, 1]. Se usa para modelar el comportamiento de variables aleatorias limitadas a un intervalo finito. Es útil para modelar proporciones o porcentajes.
$$
p(x \mid \alpha, \beta)= \frac{\Gamma(\alpha+\beta)}{\Gamma(\alpha)\Gamma(\beta)}\, x^{\alpha-1}(1-x)^{\beta-1}
$$
El primer término es simplemente una constante de normalización que asegura que la integral de la $pdf$ de 1. $\Gamma$ es la función gamma. Cuando $\alpha=1$ y $\beta=1$ la distribución beta se reduce a la distribución uniforme.
Si queremos expresar la distribución beta en función de la media y la dispersión alrededor de la media podemos hacerlo de la siguiente forma.
$$\alpha = \mu \kappa$$
$$\beta = (1 − \mu) \kappa$$
Siendo $\mu$ la media y $\kappa$ una parámetro llamado concentración a media que $\kappa$ aumenta la dispersión disminuye. Notese, además que $\kappa = \alpha + \beta$.
Can you write Python code for it?

distri = stats.beta(5, 2)  # alfa=5, beta=2
x = np.linspace(0, 1, 100)
x_rvs = distri.rvs(500)  # muestrear 500 valores de la distribución
x_pdf = distri.pdf(x)  # la pdf evaluada para todos los x
media, varianza = distri.stats(moments='mv')
plt.plot (x, x_pdf, lw=5,
          label='$\mu$  = {:3.1f}\n$\sigma$ = {:3.1f}'.format(float(media),
                                                               float(varianza)**0.5))
plt.hist(x_rvs, density=True)

plt.xlabel('x')
plt.ylabel('$p(x)$')
plt.legend(frameon=True);


I am working on the file "02_distribuciones_de_probabilidad.ipynb".
The first task is:
Distribución binomial
Es la distribución de probabilidad discreta que cuenta el número de éxitos en una secuencia de $n$ ensayos de Bernoulli (experimentos si/no) independientes entre sí, con una probabilidad fija $p$ de ocurrencia del éxito entre los ensayos.
Cuando $n=1$ esta distribución se reduce a la distribución de Bernoulli.
$$p(x \mid n,p) = \frac{n!}{x!(n-x)!}p^x(1-p)^{n-x}$$
El término $p^x(1-p)^{n-x}$ indica la probabilidad de obtener $x$ éxitos en $n$ intentos. Este término solo tiene en cuenta el número total de éxitos obtenidos pero no la secuencia en la que aparecieron. El primer término conocido como coeficiente binomial calcula todas las posibles combinaciones de $n$ en $x$, es decir el número de subconjuntos de $x$ elementos escogidos de un conjunto con $n$ elementos.
Can you write Python code for it?

n = 4  # número de intentos
p = 0.5  # probabilidad de "éxitos"
distri = stats.binom(n, p)
x = np.arange(0, n + 1)
x_pmf = distri.pmf(x)  # la pmf evaluada para todos los x
media, varianza = distri.stats(moments='mv')
plt.vlines(x,  0, x_pmf, colors='C0', lw=5,
           label='$\mu$  = {:3.1f}\n$\sigma^2$ = {:3.1f}'.format(float(media),
                                                                 float(varianza**0.5)))
plt.xlabel('x')
plt.ylabel('$p(x)$')
plt.legend(frameon=True);


I am working on the file "02_distribuciones_de_probabilidad.ipynb".
The first task is:
Distribución de Poisson
Es una distribución de probabilidad discreta que expresa la probabilidad que $x$ eventos sucedan en un intervalo fijo de tiempo (o espacio o volumen) cuando estos eventos suceden con una taza promedio $\mu$ y de forma independiente entre si. Se la utiliza para modelar eventos con probabilidades pequeñas (sucesos raros) como accidentes de tráfico o decaimiento radiactivo.
$$
p(x \mid \mu) = \frac{\mu^{x} e^{-\mu}}{x!}
$$
Tando la media como la varianza de esta distribución están dadas por $\mu$.
A medida que $\mu$ aumenta la distribución de Poisson se aproxima a una distribución Gaussiana (aunque sigue siendo discreta). La distribución de Poisson tiene estrecha relación con otra distribución de probabilidad, la binomial. Una distribución binomial puede ser aproximada con una distribución de Poisson, cuando $n >> p$, es decir, cuando la cantidad de "éxitos" ($p$) es baja respecto de la cantidad de "intentos" (p) entonces $Poisson(np) \approx Binon(n, p)$.  Por esta razón la distribución de Poisson también se conoce como "ley de los pequeños números" o "ley de los eventos raros". Ojo que esto no implica que $\mu$ deba ser pequeña, quien es pequeño/raro es $p$ respecto de $n$.
Can you write Python code for it?

distri = stats.poisson(2.3)  # occurrencia media del evento
x = np.arange(0, 10)
x_pmf = distri.pmf(x)  # la pmf evaluada para todos los x
media, varianza = distri.stats(moments='mv')
plt.vlines(x,  0, x_pmf, colors='C0', lw=5,
           label='$\mu$  = {:3.1f}\n$\sigma^2$ = {:3.1f}'.format(float(media),
                                                                 float(varianza**0.5)))
plt.xlabel('x')
plt.ylabel('$p(x)$')
plt.legend(frameon=True);


I am working on the file "02_distribuciones_de_probabilidad.ipynb".
The first task is:
Distribución exponencial
La distribución exponencial se define solo para $x > 0$. Esta distribución se suele usar para describir el tiempo que transcurre entre dos eventos que ocurren de forma continua e independiente a una taza fija. El número de tales eventos para un tiempo fijo lo da la distribución de Poisson.
$$
p(x \mid \lambda) = \lambda e^{-\lambda x}
$$
La media y la desviación estándar de esta distribución están dadas por $\frac{1}{\lambda}$
Scipy usa una parametrización  diferente donde la escala es igual a $\frac{1}{\lambda}$
Can you write Python code for it?

distri = stats.expon(scale=3)  # escala 3, lambda = 1/3
x = np.linspace(0, 25, 100)
x_rvs = distri.rvs(500)  # muestrear 500 valores de la distribución
x_pdf = distri.pdf(x)  # la pdf evaluada para todos los x
media, varianza = distri.stats(moments='mv')
plt.plot (x, x_pdf, lw=5,
          label='$\mu$  = {:3.1f}\n$\sigma$ = {:3.1f}'.format(float(media),
                                                              float(varianza)**0.5))
plt.hist(x_rvs, density=True)

plt.xlabel('x')
plt.ylabel('$p(x)$')
plt.legend(frameon=True);


I am working on the file "02_distribuciones_de_probabilidad.ipynb".
The first task is:
Distribución t de Student
Históricamente esta distribución surgió para estimar la media de una población normalmente distribuida cuando el tamaño de la muestra es pequeño. En estadística Bayesiana su uso más frecuente es el de generar modelos robustos a datos aberrantes.
$$p(x \mid \nu,\mu,\sigma) = \frac{\Gamma(\frac{\nu + 1}{2})}{\Gamma(\frac{\nu}{2})\sqrt{\pi\nu}\sigma} \left(1+\frac{1}{\nu}\left(\frac{x-\mu}{\sigma}\right)^2\right)^{-\frac{\nu+1}{2}}
$$
donde $\Gamma$ es la función gamma y donde $\nu$ es un parámetro llamado grados de libertad en la mayoría de los textos aunque también se le dice grado de normalidad, ya que a medida que $\nu$ aumenta la distribución se aproxima a una Gaussiana. En el caso extremo de $\lim_{\nu\to\infty}$ la distribución es exactamente igual a una Gaussiana.
En el otro extremo, cuando $\nu=1$, (aunque en realidad $\nu$ puede tomar valores por debajo de 1) estamos frente a una distribución de Cauchy. Es similar a una Gaussiana pero las colas decrecen muy lentamente, eso provoca que en teoría esta distribución no poseen una media o varianza definidas. Es decir, es posible calcular a partir de un conjunto de datos una media, pero si los datos provienen de una distribución de Cauchy, la dispersión alrededor de la media será alta y esta dispersión no disminuirá a medida que aumente el tamaño de la muestra. La razón de este comportamiento extraño es que en distribuciones como la Cauchy están dominadas por lo que sucede en las colas de la distribución, contrario a lo que sucede por ejemplo con la distribución Gaussiana.
Para esta distribución $\sigma$ no es la desviación estándar, que como ya se dijo podría estar indefinida, $\sigma$ es la escala. A medida que $\nu$ aumenta la escala converge a la desviación estándar de una distribución Gaussiana.
Can you write Python code for it?

distri = stats.t(loc=0, scale=2, df=4) # media 0, escala 2, grados de libertad 4
x = np.linspace(-10, 10, 100)
x_rvs = distri.rvs(500)  # muestrear 500 valores de la distribución
x_pdf = distri.pdf(x)  # la pdf evaluada para todos los x
media, varianza = distri.stats(moments='mv')
plt.plot (x, x_pdf, lw=5,
          label='$\mu$  = {:3.1f}\n$\sigma$ = {:3.1f}'.format(float(media),
                                                              float(varianza)**0.5))
plt.hist(x_rvs, density=True)
plt.xlabel('x')
plt.ylabel('$p(x)$')
plt.legend(frameon=True);


I am working on the file "02_distribuciones_de_probabilidad.ipynb".
The first task is:
La siguiente figura tomada del libro Think Stats resume las relaciones entre la $cdf$, $pdf$ y $pmf$.
<img src='imagenes/cmf_pdf_pmf.png' width=600 >
Distribuciones empíricas versus teóricas
Un método gráfico para comparar si un conjunto de datos se ajusta a una distribución teórica es comparar los valores esperados de la distribución teórica en el eje $x$ y en el eje $y$ los valores de los datos ordenados de menor a mayor. Si la distribución empírica fuese exactamente igual a la teórica los puntos caerían sobre la linea recta a $45^{\circ}$, es decir la linea donde $y = x$.
Can you write Python code for it?

muestra = np.random.normal(0, 1, 100)
dist = stats.norm(0, 1), stats.laplace(scale=0.7)
x = np.linspace(-4, 4, 100)
dist_pdf = dist[0].pdf(x), dist[1].pdf(x)

_, ax = plt.subplots(2, 2, figsize=(8, 8))
for i in range(2):
    osm, osr = stats.probplot(muestra, fit=False, dist=dist[i])
    ax[0,i].plot(osm, osm)
    ax[0,i].plot(osm, osr, 'o')
    ax[0,i].set_xlabel('Valores esperados')
    ax[0,i].set_ylabel('Valores observados')
    ax[1, i].plot(x, dist_pdf[i], lw=3)
    ax[1, i].hist(muestra, density=True)
    ax[1, i].set_ylim(0, np.max(dist_pdf) * 1.1)


I am working on the file "02_distribuciones_de_probabilidad.ipynb".
The first task is:
Probabilidades
La matemática es la lógica de la certeza mientras que la probabilidad es la lógica de la incerteza, dice Joseph K. Blitzstein condensando el pensamiento de cientos de personas antes que el. Entender como pensar en presencia de incertezas es central en Ciencia de Datos. Esta incerteza proviene de diversas fuentes, incluyendo datos incompletos, errores de medición, límites de los diseños experimentales, dificultad de observar ciertos eventos, aproximaciones, etc.
En este capítulo veremos una introducción breve a algunos conceptos centrales en probabilidad que nos dará el lenguaje para comprender mejor los fundamentos de varios métodos y procedimientos que veremos más adelante, para quienes tengan interés en profundizar en el tema recomiendo leer el libro Introduction to Probability de Joseph K. Blitzstein y Jessica Hwang.
Empecemos por el concepto de probabilidad, existen al menos tres grandes definiciones de probabilidad:

Decimos que una moneda tiene probabilidad 0,5 (o 50%) de caer cara, por que asumimos que ninguno de los dos eventos, {cara, ceca}, tiene preferencia sobre el otro. Es decir, pensamos que ambos eventos son equi-probables. Esto se conoce como definición clásica o naíf. Es la misma que usamos para decir que la probabilidad de obtener 3 al arrojar un dado es de $\frac{1}{6}$, o que la probabilidad de tener una hija es de 0,5. Esta definición se lleva a las patadas con preguntas como ¿Cuál es la probabilidad de existencia de vida en Marte?, claramente 0,5 es una sobreestimación, ya que el evento vida y el evento no-vida no son igualmente probables.

Otra forma de ver a una probabilidad es bajo el prisma frecuentista. En esta concepción de probabilidad, en vez de asumir que los eventos son igualmente probables, diseñamos un experimento (en el sentido amplio de la palabra) y contamos cuantas veces observamos el evento que nos interesa $x$ respecto del total de intentos $n$. Entonces podemos aproximar la probabilidad mediante la frecuencia relativa $\frac{n_x}{n}$, según este...
Can you write Python code for it?

distri = stats.randint(1, 7)  # límite inferior, límite superior + 1
x = np.arange(0, 8)
x_pmf = distri.pmf(x)  # la pmf evaluada para todos los "x"
media, varianza = distri.stats(moments='mv')
plt.vlines(x,  0, x_pmf, colors='C0', lw=5,
           label='$\mu$  = {:3.1f}\n$\sigma$ = {:3.1f}'.format(float(media),
                                                                 float(varianza)**0.5))
plt.xlabel('x')
plt.ylabel('$p(x)$')
plt.legend(frameon=True);


I am working on the file "02_dx_models.ipynb".
The first task is:
Again, the instantiation and usage of this model class are essentially the same.
Can you write Python code for it?


I am working on the file "02_dx_models.ipynb".
The first task is:
As with the stochastic_volatility class, this class generates simulated paths for both the risk factor and variance process.
Can you write Python code for it?

paths = svjd.get_instrument_values()

pdf = pd.DataFrame(paths, index=svjd.time_grid)

# index level paths
pdf[pdf.columns[:no_paths]].plot(colormap=colormap, lw=lw,
                           figsize=figsize, legend=legend)

vols = svjd.get_volatility_values()

pdf = pd.DataFrame(vols, index=svjd.time_grid)

# volatility paths
pdf[pdf.columns[:no_paths]].plot(colormap=colormap, lw=lw,
                           figsize=figsize, legend=legend)


I am working on the file "02_dx_models.ipynb".
The first task is:
Every model class needs a discounting object since this defines the risk-neutral drift of the risk factor.
Can you write Python code for it?


I am working on the file "02_dx_models.ipynb".
The first task is:
For easy plotting, we put the data with the time_grid information into a pandas DataFrame object.
Can you write Python code for it?

pdf = pd.DataFrame(paths, index=gbm.time_grid)


I am working on the file "02_dx_models.ipynb".
The first task is:
For the geometric Browniam motion class, the minimum set is as follows with regard to the constant parameter values. Here, we simply make assumptions, in practice the single values would, for example be retrieved from a data service provider like Thomson Reuters or Bloomberg. The frequency parameter is according to the pandas frequency conventions (cf. http://pandas.pydata.org/pandas-docs/stable/timeseries.html).
Can you write Python code for it?

me.add_constant('initial_value', 36.)
me.add_constant('volatility', 0.2)
me.add_constant('final_date', dt.datetime(2015, 12, 31))
  # time horizon for the simulation
me.add_constant('currency', 'EUR')
me.add_constant('frequency', 'M')
  # monthly frequency; paramter accorind to pandas convention
me.add_constant('paths', 10000)
  # number of paths for simulation


I am working on the file "02_dx_models.ipynb".
The first task is:
The effect might not be immediately visible in the paths plot, however, the mean of the simulated values in this case is higher by about 1 point compared to the square_root_jump_diffusion simulation without deterministic shift.
Can you write Python code for it?


I am working on the file "02_dx_models.ipynb".
The first task is:
The following visualizes 10 simulated paths for the risk factor process.
Can you write Python code for it?

paths = sv.get_instrument_values()

pdf = pd.DataFrame(paths, index=sv.time_grid)

# index level paths
pdf[pdf.columns[:no_paths]].plot(colormap=colormap, lw=lw,
                           figsize=figsize, legend=legend)


I am working on the file "02_dx_models.ipynb".
The first task is:
The following visualizes the first 10 paths of the simulation.
Can you write Python code for it?

pdf[pdf.columns[:no_paths]].plot(colormap=colormap, lw=lw,
                           figsize=figsize, legend=legend)


I am working on the file "02_dx_models.ipynb".
The first task is:
The generate_time_grid method generates a ndarray objet of datetime objects given the specifications in the market environment. This represents the discretization of the time interval between the pricing_date and the final_date. This method does not need to be called actively.
Can you write Python code for it?


I am working on the file "02_dx_models.ipynb".
The first task is:
The instantiation of a model class is then accomplished by providing a name as a string object and the respective market_environment object.
Can you write Python code for it?

gbm = geometric_brownian_motion('gbm', me)


I am working on the file "02_dx_models.ipynb".
The first task is:
The instantiation of the model class and usage then is the same as before.
Can you write Python code for it?

jd = jump_diffusion('jd', me)

paths = jd.get_instrument_values()

pdf = pd.DataFrame(paths, index=jd.time_grid)

pdf[pdf.columns[:no_paths]].plot(colormap=colormap, lw=lw,
                           figsize=figsize, legend=legend)


I am working on the file "02_dx_models.ipynb".
The first task is:
The method update_shift_values then calculates deterministic shift values for the relevant time grid by interpolation of the shift_base values.
Can you write Python code for it?

srjdp.update_shift_values()

srjdp.shift_values
  # shift values to apply to simulation scheme
  # given the shift base values


I am working on the file "02_dx_models.ipynb".
The first task is:
The simulated paths then are including the deterministic shift.
Can you write Python code for it?

paths = srjdp.get_instrument_values()

pdf = pd.DataFrame(paths, index=srjdp.time_grid)

pdf[pdf.columns[:no_paths]].plot(colormap=colormap, lw=lw,
                           figsize=figsize, legend=legend)


I am working on the file "02_dx_models.ipynb".
The first task is:
The simulation itself is initiated by a call of the method get_instrument_values. It returns an ndarray object containing the simulated paths for the risk factor.
Can you write Python code for it?

paths = gbm.get_instrument_values()

paths[:, :2]


I am working on the file "02_dx_models.ipynb".
The first task is:
This model class has a second set of simulated paths, namely for the variance process.
Can you write Python code for it?

vols = sv.get_volatility_values()

pdf = pd.DataFrame(vols, index=sv.time_grid)

# volatility paths
pdf[pdf.columns[:no_paths]].plot(colormap=colormap, lw=lw,
                           figsize=figsize, legend=legend)


I am working on the file "02_dx_models.ipynb".
The first task is:
Throughout this section we fix a constant_short_rate discounting object.
Can you write Python code for it?


I am working on the file "02_dx_models.ipynb".
The first task is:
geometric_brownian_motion
To instantiate any kind of model class, you need a market_environment object conataining a minimum set of data (depending on the specific model class).
Can you write Python code for it?

me = market_environment(name='me', pricing_date=dt.datetime(2015, 1, 1))


I am working on the file "02_dx_models.ipynb".
The first task is:
jump_diffusion
The next model is the jump diffusion model from Merton (1976) adding a log-normally distributed jump component to the geometric Brownian motion. Three more parameter values are needed:
Can you write Python code for it?

me.add_constant('lambda', 0.7)
  # probability for a jump p.a.
me.add_constant('mu', -0.8)
  # expected relative jump size
me.add_constant('delta', 0.1)
  # standard deviation of relative jump


I am working on the file "02_dx_models.ipynb".
The first task is:
square_root_diffusion
The square_root_diffusion model class is based on the square-root diffusion according to Cox-Ingersoll-Ross (1985). This class is often used to model stochastic short rates or a volatility process (eg like the VSTOXX volatility index). The model needs the following set of parameters:
Can you write Python code for it?

# short rate like parameters
me.add_constant('initial_value', 0.05)
  # starting value (eg inital short rate)
me.add_constant('volatility', 0.1)
  # volatility factor
me.add_constant('kappa', 2.5)
  # mean reversion factor
me.add_constant('theta', 0.01)
  # long-term mean


I am working on the file "02_dx_models.ipynb".
The first task is:
square_root_jump_diffusion
Experimental Status
Building on the square-root diffusion, there is a square-root jump diffusion adding a log-normally distributed jump component. The following parmeters might be for a volatility index, for example. In this case, the major risk might be a large positive jump in the index. The following model parameters are needed:
Can you write Python code for it?

# volatility index like parameters
me.add_constant('initial_value', 25.)
  # starting values
me.add_constant('kappa', 2.)
  # mean-reversion factor
me.add_constant('theta', 20.)
  # long-term mean
me.add_constant('volatility', 1.)
  # volatility of diffusion
me.add_constant('lambda', 0.3)
  # probability for jump p.a.
me.add_constant('mu', 0.4)
  # expected jump size
me.add_constant('delta', 0.2)
  # standard deviation of jump


I am working on the file "02_dx_models.ipynb".
The first task is:
stoch_vol_jump_diffusion
The next model class, i.e. stoch_vol_jump_diffusion, combines stochastic volatility with a jump diffusion according to Bates (1996). Our market environment object me contains already all parameters needed for the instantiation of this model.
Can you write Python code for it?

svjd = stoch_vol_jump_diffusion('svjd', me)


I am working on the file "02_dx_models.ipynb".
The first task is:
stochastic_volatility
Another important financial model is the  stochastic volatility model according to Heston (1993). Compared to the geometric Brownian motion, this model class need four more parameter values.
Can you write Python code for it?

me.add_constant('rho', -.5)
  # correlation between risk factor process (eg index)
  # and variance process
me.add_constant('kappa', 2.5)
  # mean reversion factor
me.add_constant('theta', 0.1)
  # long-term variance level
me.add_constant('vol_vol', 0.1)
  # volatility factor for variance process


I am working on the file "02_files.ipynb".
The first task is:
!!! There must be a better way !!!!
Can you write Python code for it?

with open('data/example_write.txt') as fobj:
    for line in fobj:
        print(line, end='')


I am working on the file "02_files.ipynb".
The first task is:
A few word on zip
Can you write Python code for it?


I am working on the file "02_files.ipynb".
The first task is:
Close the file
Can you write Python code for it?


I am working on the file "02_files.ipynb".
The first task is:
Feel free to format the date as you like. For a complete list of formatting directives, see strftime() and strptime() Behavior.
Can you write Python code for it?

print('{:%a %H:%M %d.%m.%Y}'.format(now))
print('{:%A %H:%M %d %B %Y %z}'.format(now))
print('{:%A %I %P:%M %d.%b.%y}'.format(now))
print('The day is {0:%d}, the month is {0:%B}.'.format(now))


I am working on the file "02_files.ipynb".
The first task is:
How NumPy script kiddies would do it
Can you write Python code for it?

import numpy as np

data = {}
data['phi'], data['sin'], data['cos'] = np.genfromtxt('data/data_write.txt', skip_header=8, delimiter=';').T

plt.figure(figsize=(12, 4))
plt.plot(data['phi'], data['sin'], color='red', marker='x')
plt.plot(data['phi'], data['cos'], 'g.')


I am working on the file "02_files.ipynb".
The first task is:
Or read all lines into a python list first
Can you write Python code for it?

with open('data/example_write.txt') as fobj:
    lines = fobj.readlines()

for line in reversed(lines):
    print(line, end='')


I am working on the file "02_files.ipynb".
The first task is:
Read content as str
Can you write Python code for it?


I am working on the file "02_files.ipynb".
The first task is:
Reading again will not show any content, because the current position is at the end of the file
Can you write Python code for it?


I am working on the file "02_files.ipynb".
The first task is:
Reading data from file
Can you write Python code for it?

# Read data in lines from file
with open('data/data_write.txt') as fobj:
    lines = fobj.readlines()

# Organize everythin nicly in a dict
data = {}
data['title'] = lines[0].strip('#').strip().rstrip()
data['time'] = lines[2].strip('#').strip().rstrip()
data['comment'] = lines[4].strip('#').strip().rstrip()
data['phi'] = []
data['sin'] = []
data['cos'] = []

for dataline in lines[8:]:

    # Remove linebreaks '\n' and split the line
    phi, sin, cos = dataline.strip().split(';')

    data['phi'].append(float(phi))
    data['sin'].append(float(sin))
    data['cos'].append(float(cos))

fig, axs = plt.subplots(2, 1, figsize=(12, 6))
axs[0].plot(data['phi'], data['sin'], color='red', marker='x')
axs[1].plot(data['phi'], data['cos'], 'g.')

for ax in axs:
    ax.set_xlim(0, 2 * math.pi)
    ax.set_ylim(-1.8, 1.8)


I am working on the file "02_files.ipynb".
The first task is:
Reading line by line
Can you write Python code for it?

with open('data/example_write.txt') as fobj:

    print(fobj.readline(), end='')
    print(fobj.readline(), end='')
    print(fobj.readline(), end='')
    print(fobj.readline(), end='')
    print(fobj.readline(), end='')
    print(fobj.readline(), end='')
    print(fobj.readline(), end='')
    print(fobj.readline(), end='')
    print(fobj.readline(), end='')
    print(fobj.readline(), end='')
    print(fobj.readline(), end='')
    print(fobj.readline(), end='')
    print(fobj.readline(), end='')


I am working on the file "02_files.ipynb".
The first task is:
Reading txt Files
Step by step
Open File
Can you write Python code for it?

filename = 'data/example_read.txt'
fobj = open(filename)


I am working on the file "02_files.ipynb".
The first task is:
So we have to seek it back to the start
Can you write Python code for it?

fobj.seek(0)

content = fobj.read()
print(content)


I am working on the file "02_files.ipynb".
The first task is:
Some words on datetime objects
Can you write Python code for it?

now = datetime.datetime.now()
now

print(now)

print('Year:', datetime.datetime.now().year)
print('Month:', datetime.datetime.now().month)
print('Day:', datetime.datetime.now().day)
print('Hour:', datetime.datetime.now().hour)
print('Minute:', datetime.datetime.now().minute)
print('Second:', datetime.datetime.now().second)
print('Microsecond: ',datetime.datetime.now().microsecond)


I am working on the file "02_files.ipynb".
The first task is:
There are more little helper products product(), chain(), cycle(), repeat(), ......  in iterttools
Now let us add some head data
Can you write Python code for it?

import datetime

with open('data/data_write.txt', 'w') as fobj:

    # Write header
    fobj.write('#Python in the Lab\n')
    fobj.write('#\n')
    fobj.write('#{} \n'.format(datetime.datetime.now()))
    fobj.write('#\n')
    fobj.write('#Writing some noisy sin and cos data to a file.\n')
    fobj.write('\n')

    fobj.write('\n')
    fobj.write('phi (2pi); sin (a.u.); cos (a.u.)\n')

    # Write data
    for phi, sin, cos in zip(phi_data, sin_data, cos_data):
        fstr = '{:.3f}; {:.3f}; {:.3f} \n'
        fobj.write(fstr.format(phi, sin, cos))


I am working on the file "02_files.ipynb".
The first task is:
Writing data to file
For practice we will only use modules of the Python standart libary in this example. Later we make everything easier with numpy.
Can you write Python code for it?

import math    # Python math module
import random  # Python random module


I am working on the file "02_files.ipynb".
The first task is:
Writing txt Files
The default mode is read only 'r', we have to enable the writing mode 'w'.
Can you write Python code for it?

with open('data/example_write.txt', mode='w') as fobj:
    fobj.write('\t\tPython in the Lab\n')
    fobj.write(content)


I am working on the file "02_geovisualization.ipynb".
The first task is:
There are three elements that can be adjusted in this interactive visualization:
1. First, you can select the metric that results want to be displayed from the 'metric' dropdown list.
2. Second, you can select the top n links with highest criticality score to be highlighted by adjusting the 'rank' slider.
3. Third, you can select whether you want to also display Bangladesh's administrative boundary by turning on the 'show_division' toggle button.
The red links represent Bangladesh's road network while the blue links represent Bangladesh's waterway network.
Can you write Python code for it?

widgets.interact_manual(net_v.plot_interactive, rank=widgets.IntSlider(min=50, max=500, step=10, value=50),
                        metric=widgets.Dropdown(options=all_metric, value='m1_01'),
                        show_division=widgets.Checkbox(value=False), result_gdf=fixed(result_gdf),
                        cmaps=fixed([new_cmap1, new_cmap2]), district_gdf=fixed(district_gdf));


I am working on the file "02_iDigBio_and_GBIF_Specimens_By_Continent_Over_Time.ipynb".
The first task is:
And we'll use the exact same code for graphing as well. We just need to remove the "australia" and it's color from our continent list because GBIF doesn't use it.
Can you write Python code for it?

# GBIF doesn't have "australia"
try:
    continents_list.remove("australia")
    colors.remove("#1bcda7")
except:
    pass

plots = stack_bar_plot(pandas_yc_cross_gbif, continents_list, colors)

# Start of WWI
plt.axvline(x=1914)

# Start of WWII
plt.axvline(x=1939)

# "1988 - October 31: President Reagan signs the NSF Authorization Act of 1988, thereby authorizing the doubling of the NSF budget over the next five years."
plt.axvline(x=1988)

plt.legend(plots, continents_list, loc=2)
plt.title("Specimens in GBIF by Collection Year and Continent")
plt.ylabel("Number of Specimen Records")
plt.xlabel("Year")
axes = plt.gca()
axes.set_xlim([1815, 2020])
axes.set_ylim([0, 1200000])
fig = plt.gcf()
fig.set_size_inches(12, 4)


I am working on the file "02_iDigBio_and_GBIF_Specimens_By_Continent_Over_Time.ipynb".
The first task is:
Loading the data set
Can you write Python code for it?

df = sqlContext.read.load("/guoda/data/idigbio-20190612T171757.parquet")

df.count()


I am working on the file "02_iDigBio_and_GBIF_Specimens_By_Continent_Over_Time.ipynb".
The first task is:
That looks pretty intuitive and we made the whole thing with one Spark statement. You can tell it's a Pandas dataframe now because the display is nice and tabular.
Making the graph
Plotting isn't the focus of this notebook. Briefly, we'll make a function that stacks up multiple variables (continent counts) on top of each other and shades them different colors.
Can you write Python code for it?

import pandas as pd
import numpy as np

def stack_bar_plot(pdf, columns, colors):
    """
    Matplotlib needs to have the bars stacked on top of each other by hand.
    First, we plot one column of data, then we make a ...
    """
    plots = []
    bottoms = pd.DataFrame(np.zeros((len(pdf['yearcollected_fixed']), 1)))
    for c in columns:
        plots.append(
          plt.bar(pdf['yearcollected_fixed'], pdf[c],
                  color=colors[len(plots)], edgecolor='none',
                  width=1.0, bottom=bottoms[0])
        )
        bottoms[0] += pdf[c]
    return plots


I am working on the file "02_iDigBio_and_GBIF_Specimens_By_Continent_Over_Time.ipynb".
The first task is:
There are a lot of things that are not continents there! While iDigBio cleans stuff up a bit before indexing it, not everything is obvious to do. For the purposes of our graph, we will make an "other" continent and group things that are not one of the 7 continents (plus "oceania" which GBIF uses and we'll look at GBIF data lower down) into that category.
Fixing continent with a user defined function
To create a new column that has a fixed continent field, we will write a user defined function in Python and apply that function to every row in the data frame. This is exactly how the year() function from the pyspark library works. But, since we are writing the function in Python, we need to convert it into a data frame aware function with the pyspark function udf(). Once we do that, we can use our custom function just the same way we used year() above.
So let's write our Python function:
Can you write Python code for it?

# allowed continents
continents = set(["africa", "australia", "asia", "oceania", "europe", "north america", "south america"])

def fix_continent(c):
    """
    If the given string contains one of the continents, return which one. If not
    then return "other". Lower case the string and also replace "_" with " " to
    be able to clean up GBIF's representation of continents that contain spaces.
    """
    for continent in continents:
        if (c) and (continent in c.lower().replace("_", " ")):
            return continent
    return "other"

# Some example calls to this function
print(fix_continent(None))
print(fix_continent("europe"))
print(fix_continent("oceania"))
print(fix_continent("NORTH_AMERICA"))
print(fix_continent("North America, Canada, Manitoba, Churchill"))
print(fix_continent("East Indies, Indonesia: Pulo Pandjang, off Sumatra"))
print(fix_continent("Asia; Thailand; Pathum Thani"))


I am working on the file "02_iDigBio_and_GBIF_Specimens_By_Continent_Over_Time.ipynb".
The first task is:
There it is, the number of specimen records in iDigBio by collection date and continent. We chose to add some vertical lines to the graph at particularly interesting dates. You can read the code comments to see what those are.
Doing the same with GBIF data
GBIF periocdically produces a similar graph to the above on their analytics page titled "Records by year of occurrence" but it's not very comparable because it include observation records as well. It would be great if we could run the same process on their data after filtering it for only specimens so we can look at how the contents of different occurrance record aggregators differs. (This is an expected and healthy thing. One of the goals of the GUODA collaboration is to begin to characterize the relationships between sets of aggregated data across domains without that work being done by one entity. Notice too that this copy of GBIF is pretty old. Again this is GUODA's fault as we have not been running our import regularly while we worked on improving other parts of our system.)
We can start by reading in the GBIF data that is stored in GUODA.
Can you write Python code for it?


I am working on the file "02_iDigBio_and_GBIF_Specimens_By_Continent_Over_Time.ipynb".
The first task is:
Year collected by continent
To get a feel for what a summary of year and continent look like, we will again group by the year column but also add the continent column. The result will be a row for each unique combination of year and continent with the count of how many times that occurs.
Can you write Python code for it?

yc_sum = (df
          .groupBy(year("datecollected").cast("integer").alias("yearcollected"),
                   "continent")
          .count()
          .orderBy("yearcollected")
          )

yc_sum.head(20)


I am working on the file "02_intro_to_python.ipynb".
The first task is:
%% boom %%   Mind.  Blown.
So, I'm dying to see what this looks like:
Can you write Python code for it?

p2 = figure()
p2.line(x, f(x))
p2_nbh = show(p2)


I am working on the file "02_intro_to_python.ipynb".
The first task is:
Another several examples: I'll use the "print" statement to print out the result for each calculation (if I didn't do this, it would just output the result of the last expression):
Can you write Python code for it?

print 2*3
print (4+6)*(2+9)  # should calculate to 110
print 12.0/11.0


I am working on the file "02_intro_to_python.ipynb".
The first task is:
Let's look at another example of float math:
Can you write Python code for it?


I am working on the file "02_intro_to_python.ipynb".
The first task is:
One major difference between using a calculator and doing calculations on the computer is that there are a couple of types of numbers -- integers and floating point values.  You can think of integers as whole numbers and floats (as floating point values are called) as supporting a decimal or fractional part of the value.  This shows up sometimes is odd behavior with division.
Can you write Python code for it?

print(5/3)     # Integer division gives a 'floor' value (rounding down, basically).
print(5.0/3.0) # Dividing floats (usually) gives the expected answer.
print(5.0/3)   # The interpreter uses the more complex type to infer the type for the result.
print(5/3.0)   # The order for type "upcasting" doesn't matter


I am working on the file "02_intro_to_python.ipynb".
The first task is:
So earlier, we assigned the value [0.0, 0.01, 0.02 ...] to the symbol x.  (I'm starting to use the more official names for things here.)  What happens if we pass that symbol into the function f?
Can you write Python code for it?

print x[:5]  # just to be clear that in the scope of this notebook, the symbol x is defined

print f(x)[:5]  # only show the first 5 entries


I am working on the file "02_intro_to_python.ipynb".
The first task is:
This is pretty math-y, what else can functions do?
Well, they can do anything you tell them...
Can you write Python code for it?

import random
def headache(name, number_of_repeats=5):
    """ a pretty useless function """
    namelist = list(name)
    for i in range(0, number_of_repeats):
        random.shuffle(namelist)
        for letter in namelist:
            print letter,

headache("travis", 40)


I am working on the file "02_intro_to_python.ipynb".
The first task is:
This may not seem very exciting at first, but variables are an important part of programming.  It's good to know that you can use them in Python.
Can you write Python code for it?

y = x**2 - 3*x + 12  # just like in algebra, right?


I am working on the file "02_intro_to_python.ipynb".
The first task is:
Wat ?!?! There are occasionally precision issues because of the way floating point values work.  It's actually an interesting abstraction (feel free to study a more detailed explanation of how IEEE Floats work). This is a good example of how abstractions can 'leak' (more on this later).  A good explanation of how this affects Python is here.  For our purposes, this really doesn't matter.  Just a curiosity, so, moving right along...
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
But if you want to make your strings compatible, you should do it the other way round:
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
Coding style
“The best programs are written so that computing machines can perform them quickly and so that human beings can understand them clearly." - Donald E. Knuth, Selected Papers on Computer Science
PEP8
Style guide for Python code
A style guide is about consistency. Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is most important.
A few rules

never use tabs, always 4 spaces
try to limit lines to 79 characters
use whitespace to make your code more readable
Can you write Python code for it?

spam(ham[1], {eggs: 2})        # YES!
spam( ham[ 1 ], { eggs: 2 } )  # NO!!

x, y = y, x     # YES!
x , y = y , x   # NO!!

counter = counter + 1  # YES!
counter=counter+1      # NO!!

result = add(x+1, 3)   # YES!
result = add(x + 1, 3) # YES!

def complex(real, imag=0.0):         # YES!
    return magic(r=real, i=imag)

def complex(real, imag = 0.0):       # NO!!
    return magic(r = real, i = imag)


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
Dict
A dictionary contains key / value pairs. Instead of the position you use the key as the key.
Can you write Python code for it?

phonebook = {'Max': 8389, 'Jannis': 8389, 'Thomas': 8325}
phonebook['Max']

phonebook['Katharina'] = 8592
phonebook

phonebook.keys()


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
Exceptions come in different types, which are given in the message. The type of this exception is IndexError.
So, let's handle that exception!
Can you write Python code for it?

def handle_index_error(i):
    try:
        print(my_list[i])
    except IndexError:
        print(my_list[-1])

handle_index_error(20)

handle_index_error(5)

handle_index_error('some useless string')


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
Follow these naming conventions:
lower_case_under for variables and functions and methods
WordCap for classes
ALL_CAPS for constants

And of course, there is more: https://www.python.org/dev/peps/pep-0008/
The Zen of Python
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
Functions
Definition Syntax
Can you write Python code for it?

def fib(n):    # write Fibonacci series up to n
    """Print a Fibonacci series up to n.""" # docstring
    a, b = 0, 1  # multiple assignement
    while a < n:
        print a,  # prevents the newline
        a, b = b, a+b  # another multiple assignement

fib(500)


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
If the boolean statement is True, the code below is executed.
Can you write Python code for it?

x = int(raw_input("Please enter an integer: "))

if x < 0:
    x = 0
    print('Negative changed to zero')
elif x == 0:
    print('Zero')
elif x == 1:
    print('Single')
else:
    print('More')


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
List
There are many compound data types in Python.
The most versatile one is a list. This is the closest to 'arrays' from other programming languages.
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
Lists can be indexed and sliced.
Can you write Python code for it?

my_list[0]

my_list[-1]

my_list = my_list + ['more', 'and', 'more', 'words', 'plus', 'a', 'number', 4]
my_list

my_list[2:6]

my_list[0:2]

my_list[:2]

my_list[4:-2]

my_list[4:-2:3]


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
More important datatypes
Bool
Values: True, False
    - []         False
    - [a, b]     True
    - 0          False
    - all other  True
None
Values: None
- frequently used to represent the absence of a value
Comments
Can you write Python code for it?

print("Hello, world!")  # Python 3, also works in Python 2


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
Naming Rules

Variables can only contain letters, numbers, and underscores. Variable names can start with a letter or an underscore, but can not start with a number.
Spaces are not allowed in variable names, so use underscores instead of spaces.
You cannot use Python keywords as variable names. If you absolutely need to, add an underscore to the end.
Variable names should be descriptive, without being too long. For example mc_wheels is better than just wheels or number_of_wheels_on_a_motorycle.
Be careful about using the lowercase letter l and the uppercase letters O and I in places where they could be confused with the numbers 1 and 0. And never use them as single character variable names.

Datatypes
Int & Float
(Integers and Floating point numbers)
You'll have to be careful about the datatype (sometimes)
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
Python 2 or Python 3?
Short version: Python 2.x is legacy, Python 3.x is the present and future of the language
- final 2.x version 2.7 was released 2010
- 3.0 was released in 2008
- 3.4 was released 2014
- recent standard libraby improvements are only available in 3.x
BUT:
- we are a bit lazy
- some third party packages are not (yet) available in Python 3
Software
We will use:
    - Python (https://www.python.org/)
    - IPython (http://ipython.org/)
    - IPython Notebook (http://ipython.org/notebook.html)
    - an editor of your choice
You could use:
    - an IDE (PyCharm, Eclipse)
    - Python debuggers (pdb)
    - Code checkers (pylint)
Interactive mode
Start the python command line interpreter by typing python, or the Python interactive shell by typing ipython into your terminal.
You can use this a calculator:
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
Set
A set will not store duplicate entries.
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
Some unicode strings can not be cast to strings.
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
The continue statement, also borrowed from C, continues with the next iteration of the loop:
Can you write Python code for it?

for num in range(2, 10):
    if num % 2 == 0:
        print "Found an even number", num
        continue
    print "Found a number", num


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
The indexing and slicing works for all built-in sequence types. (Also strings!)
In a list, you can change elements or slices. (Lists are mutable, strings for example are immutable.)
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
The keyword def introduces a function definition. It has to be followed by the function name and the paranthesized list of formal parameters.
Documentation strings
You should put a triple quoted string into the first line after the function definition, containing a description of the function. This is called docstring, and can be used to automatically produce documentation.
The return statement
It is simple to write a function that returns a list of the numbers of the Fibonacci series, instead of printing it:
Can you write Python code for it?

def fib2(n): # return Fibonacci series up to n
    """Return a list containing the Fibonacci series up to n."""
    result = []
    a, b = 0, 1
    while a < n:
        result.append(a)
        a, b = b, a+b
    return result

f100 = fib2(100)    # call it
f100                # write the result


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
The range() Function
Can you write Python code for it?

range?

range(4)

range(4,9)

range(4, 9, 2)


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
The return statement returns with a value from a function. return without an expression argument returns None. Falling off the end of a function also returns None.
Default Argument Values and Keyword Arguments
Can you write Python code for it?

def parrot(voltage, state='a stiff', action='voom', type_='Norwegian Blue'):
    print "-- This parrot wouldn't", action,
    print "if you put", voltage, "volts through it."
    print "-- Lovely plumage, the", type_
    print "-- It's", state, "!"

parrot(1000)  # 1 positional argument


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
The sentences are actually from a Monty Pythons episode. You can watch it later: https://www.youtube.com/watch?v=4vuW6tQ0218
Handling exceptions
You have seen error messages by now. Errors detected during excution are called exceptions and can be handled. Just as a reminder, error messages look like this:
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
This kind of type casting works for most datatypes in Python!
String
Strings are sets of characters and are either contained in single or double quotes.
Can you write Python code for it?

bla = 'bla bla bla'
bla

print(bla)

blub = "bla blub"
print(blub)


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
Unicode string
Like strings, but with more characters!
Can you write Python code for it?

my_unicode = u'Hellö World!'
my_unicode

print(my_unicode)


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
You can add new items to the end of the list with the append() method:
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
You can call the function using keyword arguments, positional arguments, or mix both.
Can you write Python code for it?

parrot('a lot of', 'bereft of life')  # 2 positional arguments

parrot('a lot of', type_='Lory', state='pushing up the daisies')  # 1 positional and 1 keyword argument


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
You can concatenate strings:
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
You can convert an int to a float:
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
You should use 4 spaces, not a tab!!
If you are presenting your code, be sure the indentation is right!
while Statements
Can you write Python code for it?

i = 5
while i > 0:
    print(i**2)
    i = i - 1


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
break and continue Statements, and else Clauses on Loops
The break statement, like in C,  breaks out of the smallest enclosing for or while loop.
Loop statements may have an else clause; it is executed when the loop terminates through exhaustion of the list (with for) or when the condition becomes false (with while), but not when the loop is terminated by a break statement.
Can you write Python code for it?

for n in range(2, 10):
    for x in range(2, n):
        if n % x == 0:
            print n, 'equals', x, '*', n/x
            break
    else:
        # loop fell through without finding a factor
        print n, 'is a prime number'


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
if Statements
Can you write Python code for it?


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
range() is often used in for-loops
Can you write Python code for it?

for i in xrange(len(my_list)):
    print i, my_list[i]


I am working on the file "02_introduction/intro_to_python.ipynb".
The first task is:
state, action, and type_ have default values, so they can be omitted from the function call.
Can you write Python code for it?

parrot(voltage=1000, action='jump')  # 2 keyword arguments


I am working on the file "02_linear_algebra.ipynb".
The first task is:
As you can see, we can easily access all this information. One-dimensional arrays in NumPy can be used to represent vectors, while two-dimensional arrays can be used to represent matrices.
a = np.array([1, 2, 3]) is equivalent to the vector:
\begin{equation}
  \mathbf{a} = \begin{pmatrix}
                1 \
                2 \
                3
               \end{pmatrix}
\end{equation}
The components of the vector can be accessed using so-called subscript notation, as in the following code snippet:
Can you write Python code for it?

print(a[0], a[1], a[2])   # Prints "1 2 3"
a[0] = 5                 # Change an element of the array
print(a)                  # Prints "[5, 2, 3]"


I am working on the file "02_linear_algebra.ipynb".
The first task is:
As you can see, we can quite easily select just parts of an array and either create new, smaller arrays or just obtain the value at a certain position.
Algebraic operations and functions are also defined on arrays. The definitions for addition and subtraction are obvious. Those for multiplication, division and other mathematical functions are less obvious: in NumPy they are always performed element-wise.
The following example will clarify.
Can you write Python code for it?

a = np.full(10, 7.)
b = np.full(10, 4.)
print(a)
print(b)

print('Sum of a and b')
print(a + b)

print('Difference of a and b')
print(a - b)

print('Elementwise product of a and b')
print(a * b)

print('Elementwise division of a and b')
print(a / b)

print('Elementwise square root of b')
print(np.sqrt(b))


I am working on the file "02_numpy/02_actividad.ipynb".
The first task is:
Paso 2)
Can you write Python code for it?

_data = np.copy(data)
mask = _data<rms
_data[mask] = 0.

image_plot(_data)


I am working on the file "02_numpy/02_actividad.ipynb".
The first task is:
Paso 3)
Can you write Python code for it?

def apply_filter(data, mask, kernel_filter):
    m,n = data.shape
    _data = data.copy()
    for i in range(m):
        for j in range(n):
            if not mask[i,j]: continue
            data[i,j] = np.sum(_data[i-1:i+2,j-1:j+2]*kernel_filter)

apply_filter(data, ~mask, kernel_filter)

image_plot(data)


I am working on the file "02_numpy/02_numpy.ipynb".
The first task is:
<div id='basic_numpy' />
Basic NumPy operations
The reasons of why you should use NumPy instead of any other iterable object_ in Python are:
* NumPy provides an ndarray structure for storing numerical data __in a contiguous way.
* Also implements fast mathematical operations on ndarrays, that exploit this contiguity.
* Brevity of the syntax for array operations. A language like C or Java would require us to write a loop for a matrix operation as simple as C=A+B.
Creating Arrays
There are several NumPy functions for creating common types of arrays. Below is a list of the most common used:
Can you write Python code for it?

# Arrays of zeros: np.zeros(shape)
print("Zeros:")
print( np.zeros((3,3)) )

# Arrays of ones: np.ones(shape)
print("\nOnes:")
print( np.ones((3,3)) )

# Empty array: np.empty(shape)
print("\nEmpty:")
print( np.empty((3,3)) )

# Range of values: np.range(start, stop, step)
print("\nRange:")
print( np.arange(0., 10., 1.) )

# Regular grid: np.linspace(start, end, n_values)
print("\nRegular grid:")
print( np.linspace(0., 1., 9) )

# Random secuences: np.random
print("\nRandom sequences:")
print( np.random.uniform(10, size=6) )

# Array constructor: np.array( python_iterable )
print("\nArray constructor")
print( np.array([2, 3, 5, 10, -1]) )

print( 10*np.random.random((5,5)) )


I am working on the file "02_numpy/02_numpy.ipynb".
The first task is:
<div id='bonus' />
Bonus: Useful libraries based in NumPy
numpy.linalg (Numpy's submodule)
Can you write Python code for it?

A = np.random.random((100,100))
b = np.random.random(100)
c = np.random.random(100)

### Matrix power ###
np.linalg.matrix_power(A,3)

### Cholesky decomposition ###
#np.linalg.cholesky(A) #A must be positive definite

### QR decomposition ###
np.linalg.qr(A, mode='reduced')

### SVD decomposition ###
np.linalg.svd(A, full_matrices=False)

### Eigenvectors ###
np.linalg.eig(A)

### Eigevalues ###
np.linalg.eigvals(A)

### Matrix or vector norm ###
np.linalg.norm(A, ord='fro')

### Condition number ###
np.linalg.cond(A, p=np.inf)

### Determinant ###
np.linalg.det(A)

### Linear solver Ax=b ###
np.linalg.solve(A,b)

### Least Squares Ax=b (over-determined) ###
np.linalg.lstsq(A,b)

### Inverse ###
np.linalg.inv(A)

### Pseudo-Inverse ###
np.linalg.pinv(A)

### and many more...
del A,b,c


I am working on the file "02_numpy/02_numpy.ipynb".
The first task is:

Vectorized Algorithms with NumPy
Vectorization: The process of converting a scalar implementation, which processes a single pair of operands at a time, to a vector implementation, which processes one operation on multiple pairs of operands at once.
Example: Gram-Shmidt Orthogonalization
The problem: Given a matrix $Q_{m\times n}$ with ($m>n$), find a vector $v$ orthogonal to the column space of $Q$.

Can you write Python code for it?

def naive_orth(Q,v):
    m,n = Q.shape
    for j in range(n):
        v -= np.dot(Q[:,j],v)*Q[:,j]
    return v

def vectorized_orth(Q,v):
    proy = np.dot(Q.transpose(),v)
    # v -= (proy*Q).sum(axis=1)
    v -= np.dot(Q,proy)
    return v

# Let's generate a random unitary matrix
# Q unitary matrix, dimensions 100000 x1000
m,n = 10000,100
A = 10 * np.random.random((m,n))
Q,R = np.linalg.qr(A, mode='reduced')
del R

# v will be the starting vector for orthogonalization
v = np.random.random(m)
v1 = v.copy()
v2 = v.copy()

%timeit naive_orth(Q,v1)

%timeit vectorized_orth(Q,v2)


I am working on the file "02_numpy/02_numpy.ipynb".
The first task is:
As you can see, some interesting things have happened:

The naive NumPy implementation, which iterates over data is actually slower than simply using a list. This is because the array stores a  very low-level representation of the numbers it stores , and this must be converted into Python-compatible objects before being returned to user, causing this extra overhead each time you index an array.
norm_square_numpy is slower than the clever NumPy implementation because two reasons:
There is time spend in allocating memory for storing the temporary result (vector x vector) and
This creates two implied loops, one to do the multiplication and one to do the sum.

The clever implementation uses np.dot() NumPy function which has no need to store intermediate results, and iterates just one time (but at C speed).

<div id='efficient' />
Eficient programming with NumPy
In-place and implicit copy operations
Prefer in-place over implicit-copy operations whenever possible. This will save memory (less work to garbage collector) and performs faster.
Can you write Python code for it?

def id(x):
    """
    This function returns the memory
    block address of an array.
    """
    return x.__array_interface__['data'][0]


I am working on the file "02_numpy/02_numpy.ipynb".
The first task is:
Basic Mathematical operations
Most of the operations performed in NumPy are handled element-wise, i.e, computing C = A + B will translates into $C[i,j] = A[i,j] + B[i,j]$. (The exception is broadcasting, and will be explained soon).
Below is a list with the most common used mathematical operations. For a comprehensive list se here: NumPy mathematical functions.
Can you write Python code for it?

# first we create two random arrays:
A = np.random.random((5,5))
B = np.random.random((5,5))

# sum
print("Sum:")
print( A+B )

# subtraction
print("\nSubtraction")
print( A-B )

# product
print("\nProduct")
print( A*B )

# matricial product
print("\nMatricial Product")
print( np.dot(A,B) )

# power
print("\n Power")
print( A**2 )

# Some common mathematical functions
print("\n np.exp()")
print( np.exp(A) )
print("\n np.sin()")
print( np.sin(A) )
print("\n np.cos()")
print( np.cos(A))
print("\n np.tan()")
print( np.tan(A) )


I am working on the file "02_numpy/02_numpy.ipynb".
The first task is:
Broadcasting
There is no need to always reshape arrays before operate on them. This useful feature implemented in NumPy arrays is called Broadcasting rules. In the visualization below, the extra memory indicated by the dotted boxes is never allocated, but it can be convenient to think about the operations as if it is.
<img src='data/broadcasting.png' style="width: 600px;">
How it works: The two arrays to be operated must match in at least one dimension. Then the array with less dimensions will logically extended to match the dimensions of the other
Can you write Python code for it?


I am working on the file "02_numpy/02_numpy.ipynb".
The first task is:
Efficient memory access
We have basically three alternatives to access arrays without loops:

array slicing
boolean masks
fancy indexing.

Note. If you find yourself looping over indices to select items on which operation is performed, it can probably be done more efficiently with one of these techniques!
Can you write Python code for it?

m, n = 1000000, 100
a = np.random.random_sample((m, n))
index = np.arange(0, m, 10)

#fancy indexing - indexing with lists
%timeit a[index,:]

#memory slice - memory views
%timeit a[::10]


I am working on the file "02_numpy/02_numpy.ipynb".
The first task is:
How an ndarray is stored in memory?
When there is more than one dimension, there are two ways of storing the elements in the memory block:
1. Elements can be stored in row-major order (also known as C-order) or,
2. In column-major order (also known as Fortran-order).
<img src='data/ndarray_storage.png' style="width: 800px;">
What are the strides?
NumPy uses the notion of strides to convert between a multidimensional index and the memory location of the underlying (1D) sequence of elements.
For example, the mapping between array[i,j] and the corresponding address the byte is:
* offset = array.strides[0] * i1 + array.strides[1] * i2
* address = base + offset
where base is the address of the first byte (array[0,0]).
Can you write Python code for it?

# Lets create a random array in C-order
A = np.random.random((5,2))
print("C strides:")
print(A.strides)

# Lets create a random array in F-order
B = np.asfortranarray(A)
print("\nF strides:")
print(B.strides)


I am working on the file "02_numpy/02_numpy.ipynb".
The first task is:
Table of Contents

1.- Useful Magics
2.- Basic NumPy Operations
3.- Internals of NumPy
4.- Efficient Programming in NumPy
5.- Vectorized Algorithms in NumPy
6.- Bonus

<div id='magics' />
Useful magic built-in functions for time-profiling

time: See how long it takes a code to run.
Can you write Python code for it?


I am working on the file "02_numpy/02_numpy.ipynb".
The first task is:
prun: See how long it took each function in a script to run.
Can you write Python code for it?

from time import sleep

def foo(): sleep(1)

def bar(): sleep(2)

def baz(): foo(),bar()

%prun baz()


I am working on the file "02_numpy/02_numpy.ipynb".
The first task is:
timeit:
See how long a code takes to run averaged over multiple runs.
It will limit the number of runs depending on how long the script takes to execute.
Provide an accurate time calculation by reducing the impact of startup or shutdown costs on the time calculation by executing the code repeatedly.
Can you write Python code for it?

%timeit [1+2 for i in range(10000)]

%timeit -n 100 [1+2 for i in range(10000)]

%%timeit
for i in range(10000):
    1+2


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
A new dedicated Infix operator for Matrix Multiplication
Python 3.5 (released on the 24th Aug.) introduces a new binary operator to be used for matrix multiplication, called @ . (Mnemonic: @ is * for mATrices.)
Some useful hints:

PEP465 Description: https://www.python.org/dev/peps/pep-0465/#abstract
Motivations
Rationale for Specification Details
Can you write Python code for it?


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
Alternatively, we can use the condition (mask) array directly within brackets to index the array
Can you write Python code for it?


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
And we also have another vector adj of values that contains some adjusting factors that we might want to apply to each sample (row) of data in D
Can you write Python code for it?


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
Array slices are mutable: if they are assigned a new value the original array from which the slice was extracted is modified:
Can you write Python code for it?


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
Excercise
Avoiding Vectorize

Implement the function to accept vector input from the beginning
This requires "more effort" but might give better performance

Vectorisation and Broadcasting
Vectorizing code is the key to writing efficient numerical calculation with Python/Numpy.
That means that as much as possible of a program should be formulated in terms of matrix and vector operations, like matrix-matrix multiplication.
However, sometimes it may happen that we need to perform operations between ndarray objects of different size and shape so we might be tempted of performing this operation explicitly (i.e. explicit for loop).
Although we will get correct results (assumed we iterated arrays in the right way), this will produce very inefficient code.
For this reason, NumPy has a very important concept called Broadcasting.
Broadcasting is Numpy's terminology for performing mathematical operations between arrays with different shapes (assuming scalars being array with 0-dim).
Working example
Assume we have some data in a matrix D:
Can you write Python code for it?


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
Exercises on Indexing
Index slicing is the technical name for the syntax M[lower:upper:step] to extract part of an array
Ex 2.1
Generate a three-dimensional array of any size containing random numbers taken from an uniform distribution (guess the numpy function in np.random). Then print out separately the first entry along the three axis (i.e. x, y, z)

hint: Slicing with numpy arrays works quite like Python lists

Ex 2.2
Create a vector and print out elements in reverse order
Hint: Use slicing for this exercise
Ex 2.3
Generate a $7 \times 7$ matrix and replace all the elements in odd rows and even columns with 1.
Hint: Use slicing to solve this exercise!
Note: Take a look at the original matrix, then.
Use fancy indexing to get all the elements of the previous matrix that are equals to 1
Ex 2.4
Generate a 10 x 10 matrix of numbers A. Then, generate a numpy array of integers in range 1-9. Pick 5 random values (with no repetition) from this array and use these values to extract rows from the original matrix A.
Ex 2.5
Repeat the previous exercise but this time extract columns from A
Ex 2.6
Generate an array of numbers from 0 to 20 with step 0.5.
Extract all the values greater than a randomly generated number in the same range.

hint: Try to write the condition as an expression and save it to a variable. Then, use this variable in square brackets to index.... this is when the magic happens!

Stacking and repeating arrays
Using function repeat, tile, vstack, hstack, and concatenate we can create larger vectors and matrices from smaller ones:
np.tile and np.repeat
Can you write Python code for it?


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
If we multiply arrays with compatible shapes, we get an element-wise multiplication of each row:
Can you write Python code for it?

A.shape, v1.shape

A * v1  #Broadcasting


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
If we omit an index of a multidimensional array it returns the whole row (or, in general, a N-1 dimensional array)
Can you write Python code for it?


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
Iterating over array elements
Generally, we want to avoid iterating over the elements of arrays whenever we can (at all costs). The reason is that in a interpreted language like Python (or MATLAB), iterations are really slow compared to vectorized operations.
However, sometimes iterations are unavoidable. For such cases, the Python for loop is the most convenient way to iterate over an array:
Can you write Python code for it?

v = np.array([1,2,3,4])

for element in v:
    print(element)

M = np.array([[1,2], [3,4]])

for row in M:
    print("row", row)

    for element in row:
        print(element)


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
Matrix algebra
What about matrix mutiplication?
There are two ways.
We can either use the np.dot function, which applies a matrix-matrix, matrix-vector, or inner vector multiplication to its two arguments:
Can you write Python code for it?

np.dot(A, A)

np.dot(A, v1)

np.dot(v1, v1)


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
Naive Solution
Can you write Python code for it?

%%timeit

# Create a new array filled with zeros, of the same shape as macros.
result = np.zeros_like(D)

# Now multiply each row of macros by cal_per_macro. In Numpy, `*` is
# element-wise multiplication between two arrays.
for i in range(D.shape[0]):
    result[i, :] = D[i, :] * adj

result


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
Negative indices counts from the end of the array (positive index from the begining):
Can you write Python code for it?

a = np.array([1,2,3,4,5])

a[-1] # the last element in the array

a[-3:] # the last three elements


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
Note: no changes in meaning for indexing operations
Can you write Python code for it?


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
OK, that didn't work because we didn't write the Theta function so that it can handle with vector input...
To get a vectorized version of Theta we can use the Numpy function np.vectorize. In many cases it can automatically vectorize a function:
np.vectorize
Can you write Python code for it?

Theta_vec = np.vectorize(Theta)

Theta_vec(np.array([-3,-2,-1,0,1,2,3]))


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
The loop-in-Python method takes ~1.5 seconds, the stretching method takes ~48 milliseconds, a ~75x speedup.
And now, finally, comes the interesting part.
You see, the operation we just did - stretching one array so that its shape matches that of another and then applying some element-wise operation between them - is actually pretty common.
This often happens when we want to take a lower-dimensional array and use it to perform a computation along some axis of a higher-dimensional array.
In fact, when taken to the extreme this is exactly what happens when we perform an operation between an array and a scalar - the scalar is stretched across the whole array so that the element-wise operation gets the same scalar value for each element it computes.
Numpy generalizes this concept into broadcasting - a set of rules that permit element-wise computations between arrays of different shapes, as long as some constraints apply.
Incidentally, this lets Numpy achieve two separate goals - usefulness as well as more consistent and general semantics.
Binary operators like * are element-wise, but what happens when we apply them between arrays of different shapes? Should it work or should it be rejected? If it works, how should it work?
Broadcasting defines the semantics of these operations.
Can you write Python code for it?

%%timeit
## Back to our example
D * adj  # Broadcasting


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
The same thing can be achieved with using : instead of an index:
Can you write Python code for it?


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
This feature is very useful to conditionally select elements from an array, using for example comparison operators:
Can you write Python code for it?

x = np.arange(0, 10, 0.5)
x

mask = (5 < x)

mask

x[mask]


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
This is a reasonable approach when coding in a low-level programming language: allocate the output, loop over input performing some operation, write result into output.
In Numpy, however, this is fairly bad for performance because the looping is done in (slow) Python code instead of internally by Numpy in (fast) C code
(we've proven this already when we compared ndarray vs list)
Using np.tile
Idea: Leverage on np.tile function to replicate adj over the number
of rows D has (i.e. D.shape[0]).
Can you write Python code for it?

%%timeit

adj_stretch = np.tile(adj, (D.shape[0], 1))

D * adj_stretch

adj_stretch


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
We can also index masks:

If the index mask is an Numpy array of with data type bool, then an element is selected (True) or not (False) depending on the value of the index mask at the position each element:
Can you write Python code for it?

b = np.array([n for n in range(5)])
b

row_mask = np.array([True, False, True, False, False])
b[row_mask]


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
We can assign new values to elements in an array using indexing:
Can you write Python code for it?

M[0,0] = 1

M

# also works for rows and columns
M[1,:] = 0
M[:,1] = -1

M


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
We can index elements in an array using the square bracket and indices:
Can you write Python code for it?

# v is a vector, and has only one dimension, taking one index
v[0]

# M is a matrix, or a 2 dimensional array, taking two indices
M[1,1]


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
We can omit any of the three parameters in M[lower:upper:step]:
Can you write Python code for it?

a[::] # lower, upper, step all take the default values

a[::2] # step is 2, lower and upper defaults to the beginning and end of the array

a[:3] # first three elements

a[3:] # elements from index 3


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
np.concatenate
Can you write Python code for it?

b = np.array([[5, 6]])

np.concatenate((a, b), axis=0)

np.concatenate((a, b.T), axis=1)


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
np.frompyfunc
Universal functions (Ufuncs) work on arrays, element-by-element, or on scalars.
Ufuncs accept a set of scalars as input, and produce a set of scalars as output.
Can you write Python code for it?

Theta_ufunc = np.frompyfunc(Theta, 1, 1)
print("Result: ", Theta_ufunc(np.arange(4)))


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
np.hstack and np.vstack
Can you write Python code for it?


I am working on the file "02_numpy_indexing.ipynb".
The first task is:
np.tile
Can you write Python code for it?


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
Basic Operations
Lets say we want to calculate the vector pointing from atom1 to atom2. In pure python we would need a loop iterarting over the two vectors. For example like this:
Can you write Python code for it?

vec = []
for i in range(len(atom1_xyz)):
    vec.append(atom2_xyz[i] - atom1_xyz[i])

print(vec)


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
But this is not all. We can also do more fancy indexing. If we compare two arrays or an array with something that can be broadcasted to the same shape, we get an array of booleans returned. This array is the result of an elementwise comparison. Such boolean arrays can also be used for indexing. Lets see how this works:
Can you write Python code for it?


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
Create an array filled with zeros and one filled with random numberers in the interval [0,1), each with a shape (5, 2).
Can you write Python code for it?

my_zeros_array = # your code goes here
my_random_array = # your code goes here

print(my_zeros_array)
print(my_random_array)


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
For higherdimensional arrays idexes for the different dimensions are separated by kommas. For example:
Can you write Python code for it?

a = np.array([[1, 2, 3],
              [4, 5, 6],
              [7, 8, 9]])

print(a[0, 1]) # the element in the row 0 and column 1
print(a[0, :]) # all elements in row 0
print(a[:, :-1]) # all elements in all columns except for the last culumn


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
Indexing
Now that we have caclulated some things we also want to access specific numbers in our result arrays. Numpy arrays can be indexed or sliced much like python lists. For example:
Can you write Python code for it?

a = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

print(a[3])     # a at the position 3 (keep in mind python counts from 0!)
print(a[2:])    # all numbers from position 2 to the end
print(a[2:8])   # all numbers from position 2 to (not including) position 8
print(a[2:8:2]) # every second number in the same range as above
print(a[:-2])   # all numbers except for the last two


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
Lets see how we use such boolean arrays for indexing:
Can you write Python code for it?

a = np.array([[2, 3, 5],
              [5, 2, 3],
              [3, 5, 2]])

a[a > 2] += 1 # here we add one to each value larger then 2

print(a)


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
Set all numbers in a which equal 2, to 0.
Can you write Python code for it?

a[?] =  # your code goes in here
print(a)


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
The array b has the shape (1, 2). In the first dimension one of them has the size 1. In the second dimension they both have the same size (2). Because all dimensions have eighter equal size or one of them is 1 numpy will be able to braodcast b to the shape of a. That is why we can for example do:
Can you write Python code for it?


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
Try to slice out the following block from the array a!
Can you write Python code for it?

block = np.array([[5, 6],
                  [8, 9]])

slice_of_a = a[# your code goes in here]

if np.array_equal(block, slice_of_a):
    print('That is correct!')
else:
    print('Sorry you have sliced out\n', slice_of_a, 'instead')


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
Use broadcasting to caculate all the vectors from atom1 to all other atoms! other_atoms is a 2d array where the first dimension represents different atoms and the second dimension represents the x, y and z coordinates.
Can you write Python code for it?

atom1 = np.array([5, 2, 8])
other_atoms = np.array([[5, 1, 4],
                        [8, 4, 6],
                        [3, 2, 4],
                        [9, 7, 3]])

vectors_from_atom1_to_all_others = # your code goes here
print(vectors_from_atom1_to_all_others)


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
Using numpy arrays this is a lot easier and faster. For two arrays of the same shape "+" "-" "/" "*" operations will be applied elementwise to the whole arrays. Calculate the vector pointing from atom1 to atom2 using numpy!
Can you write Python code for it?


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
We can not only use these operations for two arrays of the same shape. Whenever each dimension of the two arrays has either the same size or one of them has size 1, numpy broadcasts the smaller array to the shape of the larger array. Lets see how this works in an example:
Can you write Python code for it?


I am working on the file "02_numpy_md/numpy_md.ipynb".
The first task is:
Your first own numpy MD code
Now you have learned enough to test your numpy knowledge in an exciting example. You will write your own basic MD code. For a start it will be so basic it won't even be able to simulate 256 Argon atoms. For visualization purposes and to keep it simple we will performe our simulation in a 2D square box.
Task 1: Add atoms!
When we create an object from our simulation class the init function will be called. So far init adds the attributes n_atoms (number of atoms) and box_size. The simulation object also needs all the atom positions and velocities.
self.positions and self.velocities should both be 2D numpy arrays. The first dimension should have one element per atom and the second dimension should have 2 elements representing an x and y values. To distribute atoms randomly in the whole box the self.positons should contain random numbers between 0 and box_size. Velocities should contain random numbers between -0.005 and +0.005. Add positions and velocities to the class definition below and execute the cell. Hint: If you restart the cell while it was still runnig you wont see any output. Use the stop button first.
(If you have done everything correctly you should see randomly distributed atoms in the appearing plot.)
Task 2: Make them move!
So far our atoms are not moving. We need to fix that in our run function. The velocities array contains velocities in the unit distance/step. All you need to do each step is to add the velocities to the current positions to get the new positions and to overwrite self.positions with these new postions. Make the atoms move!
(If you have done everything correctly you should see the atoms move. However they wont stop at the boarders of the box.)
Task 3: Keep them in the box!
Obviously we don't want our atoms to go outside of the box. We want to keep them in the box by enforcing periodic boundary conditions. That is where our keep_in_box function comes into play. Use fancy indexing with boolean arrays to detect which atoms are outside the box and...
Can you write Python code for it?

import matplotlib.pyplot as plt
from IPython.display import clear_output, display
%matplotlib inline
import numpy as np

class Simulation:
    def __init__(self, n_atoms, box_size):
        self.n_atoms = n_atoms
        self.box_size = box_size
        self.positions = # Task 1
        self.velocities =  # Task 1
        self.fig, self.ax = plt.subplots()

    def run(self, n_steps):
        for step in range(n_steps):
            self.positions = self.positions  # Task 2
            self.keep_in_box()

            self.show(step)

    def keep_in_box(self):
        self.positions[...] += 0 # Task 3
        self.positions[...] -= 0 # Task 3

    def show(self, step):
        self.ax.scatter(self.positions[:, 0], self.positions[:, 1])
        self.ax.set_ylim((0, self.box_size))
        self.ax.set_xlim((0, self.box_size))
        self.ax.text(0.4, 0.9, 'step ' + str(step),
                     bbox={'facecolor':'white', 'alpha':0.5, 'pad':7})
        self.ax.set_aspect('equal', adjustable='box')
        clear_output(True)
        display(self.fig)
        self.ax.cla()

sim = Simulation(n_atoms=100, box_size=1)
sim.run(n_steps=500)


I am working on the file "02_numpy_md/recall_last_session.ipynb".
The first task is:
Analytical solution
Initialisation

We have first to define the starting position and velocity of our ball as in the numerical example.

Analytical solution

Then we can iterate over the saved points in time and calculate the analytical solution for this point and add it to the trajectory.

 Here you have to fill in the equations.

Error calculation

To be able to better compare the numeric and analytic solution we calculate the squared error between them and then plot it.

Can you write Python code for it?

# Initialisation
start_position=[0, 1] # start position of the ball (as in the numeric example)
start_velocity=[0.2, 0] # start velocity of the ball (as in the numeric example)

# Analytical solution
trajectory_analytical = [] # an empty list to store the analytic results
for point_in_time in trajectory_time: # iterate of the timepoints
    position_x = # fill in here
    position_y = # fill in here
    trajectory_analytical.append([position_x,position_y]) # update trajectory

# Error calculation
error = []
error_x_sq=error_y_sq=0
for i in range(len(trajectory_analytical)):
    diff_x=trajectory_analytical[i][0]-trajectory_numerical[i][0]
    diff_y=trajectory_analytical[i][1]-trajectory_numerical[i][1]
    error_x_sq+=diff_x**2
    error_y_sq+=diff_y**2
    error.append([error_x_sq,error_y_sq])

# Plotting
from animation import plot
%matplotlib inline

plot(trajectory_time, trajectory_numerical,trajectory_analytical,error)