I am working on the file "Part_2.ipynb".
The first task is:
3.3) Bokeh
Bokeh is a Python interactive visualization library that targets modern web browsers for presentation. Its goal is to provide elegant, concise construction of novel graphics in the style of D3.js, and to extend this capability with high-performance interactivity over very large or streaming datasets.
http://bokeh.pydata.org/en/latest/
Can you write Python code for it?

from bokeh.charts import Bar, Histogram, show, output_notebook
output_notebook()


I am working on the file "Part_2.ipynb".
The first task is:
Bar Chart
Can you write Python code for it?

summary = df[['ext price', 'name']].groupby('name').sum()
summary.plot(kind='bar', title="Total Sales by Account")


I am working on the file "Part_2.ipynb".
The first task is:
Histogram
Can you write Python code for it?

purchase_patterns = df[['ext price','date']]
purchase_patterns.head()
purchase_plot = purchase_patterns['ext price'].hist(bins=40)
purchase_plot.set_title("Purchase Patterns")
purchase_plot.set_xlabel("Order Amount($)")
purchase_plot.set_ylabel("Number of orders")


I am working on the file "Part_2.ipynb".
The first task is:
Stacked Bar Chart
Can you write Python code for it?

sales_by_sku=df[['name', 'sku', 'ext price']].groupby(['name', 'sku']).sum()
sales_by_sku.head()

sales_by_sku.unstack().head()

my_plot = sales_by_sku.unstack().plot(kind='bar',stacked=True, title="Total Sales by Customer and SKU", legend=None)
my_plot.set_xlabel("Customers")
my_plot.set_ylabel("Sales")


I am working on the file "Part_3.ipynb".
The first task is:
Load up some test data to play with
Can you write Python code for it?

tips = pd.read_csv('input/tips.csv')

tips['tip_percent'] = (tips['tip'] / tips['total_bill'] * 100)

tips.head()

tips.describe()


I am working on the file "Part_3.ipynb".
The first task is:
Plotting linear regression
http://web.stanford.edu/~mwaskom/software/seaborn/tutorial/regression.html
Can you write Python code for it?

sns.jointplot("total_bill", "tip_percent", tips, kind='reg');

sns.lmplot(x="total_bill", y="tip_percent", hue="ordered_alc_bev", data=tips)

sns.lmplot(x="total_bill", y="tip_percent", col="day", data=tips, aspect=.5)

sns.lmplot(x="total_bill", y="tip_percent", hue='ordered_alc_bev', col="time", row='gender', size=6, data=tips);


I am working on the file "Part_3.ipynb".
The first task is:
Plotting logistic regression
http://web.stanford.edu/~mwaskom/software/seaborn/tutorial/regression.html
Can you write Python code for it?

# Let's add some calculated columns
tips['tip_above_avg'] = np.where(tips['tip_percent'] >= tips['tip_percent'].mean(), 1, 0)
tips.replace({'Yes': 1, 'No': 0}, inplace=True)

tips.head()

sns.lmplot(x="tip_percent", y="ordered_alc_bev", col='gender', data=tips, logistic=True)

sns.lmplot(x="ordered_alc_bev", y="tip_above_avg", col='gender', data=tips, logistic=True)


I am working on the file "Part_4.ipynb".
The first task is:
Load up some data
Can you write Python code for it?

tips = pd.read_csv('input/tips.csv')
tips['tip_percent'] = (tips['tip'] / tips['total_bill'] * 100)
tips['tip_above_avg'] = np.where(tips['tip_percent'] >= tips['tip_percent'].mean(), 1, 0)
tips.replace({'Yes': 1, 'No': 0}, inplace=True)


I am working on the file "Part_4.ipynb".
The first task is:
WTH Does that stuff mean?
http://www.dummies.com/education/math/statistics/how-to-interpret-a-correlation-coefficient-r/
Can you write Python code for it?

# Describe what we want using R-style formulas
formula = 'tip_percent ~ total_bill + party_size + C(ordered_alc_bev) + C(gender) + C(day) + C(time)'
model = sm.ols(formula, data=tips)    # Describe model
results = model.fit()                 # Fit model
results.summary()                     # Summarize model

columns = ['total_bill', 'gender', 'ordered_alc_bev', 'day', 'time', 'party_size']
data = [[15.52, 'Female', 0, 'Sun', 'Lunch', 1]]

df = pd.DataFrame(data, columns=columns)

predictions = results.predict(df).tolist()  # Convert to simple list
predictions


I am working on the file "Part_4.ipynb".
The first task is:
http://blog.yhat.com/posts/logistic-regression-python-rodeo.html
Can you write Python code for it?

columns = ['total_bill', 'party_size', 'ordered_alc_bev']
training_columns = tips[columns].columns
logit = sm.Logit(tips['tip_above_avg'], tips[training_columns])  # Describe model
results = logit.fit()                                            # Fit model
results.summary()                                                # Summarize model

data = [[10.52, 4, 0]]
df = pd.DataFrame(data, columns=columns)
predictions = results.predict(df).tolist()  # Convert to simple list

predictions

from requests import get, post
request_data = [
	{"total_bill": 22.50, "party_size": 3, "ordered_alc_bev": 1, "gender": "Female", "day": "Sat", "time": "Dinner"},
	{"total_bill": 18.62, "party_size": 3, "ordered_alc_bev": 1, "gender": "Female", "day": "Sat", "time": "Dinner"},
	{"total_bill": 17.14, "party_size": 3, "ordered_alc_bev": 0, "gender": "Female", "day": "Sat", "time": "Dinner"},
]
post('http://localhost:5000/', json=request_data).json()


I am working on the file "Parte 1 - CUDA C/03 - Multiplicacion de vectores.ipynb".
The first task is:
El resultado final tendría que ser igual a 333300.0. Si tu resultado es correcto, ¡felicidades! has escrito correctamente tu primer código en CUDA C.
En caso de que no logres encontrar el resultado correcto y ya estés harto de intentar, recuerda que nos puedes contactar a nuestros correos (que están en el primer notebook).
Can you write Python code for it?

suma = 0.
for i in xrange(100):
    suma += i*(i+1)
suma

# Estas cuatro lineas hacen lo mismo que todo el código que hicimos anteriormente
# No se desanimen :(, los verdaderos resultados vendrán muy prontamente.


I am working on the file "Parte 2 - PyCUDA y aplicaciones/01 - Introduccion a PyCUDA.ipynb".
The first task is:
Ejecutando kernels
Durante este capítulo nos centraremos en un ejemplo muy simple. Escribir un código para duplicar cada una de las entradas en un arreglo, en seguida escribimos el kernel en CUDA C, y se lo otorgamos al constructor de pycuda.compiler.SourceModule
Can you write Python code for it?

mod = SourceModule("""
  __global__ void duplicar(float *a)
  {
    int idx = threadIdx.x + threadIdx.y*4;
    a[idx] *= 2;
  }
  """)


I am working on the file "Parte 2 - PyCUDA y aplicaciones/01 - Introduccion a PyCUDA.ipynb".
The first task is:
Finalmente recogemos la información del GPU y la mostramos con el a original
Can you write Python code for it?

a_duplicado = numpy.empty_like(a)
cuda.memcpy_dtoh(a_duplicado, a_gpu)
print a_duplicado
print a

print(type(a))
print(type(a_gpu))
print(type(a_duplicado))

a_duplicado


I am working on the file "Parte 2 - PyCUDA y aplicaciones/01 - Introduccion a PyCUDA.ipynb".
The first task is:
Si no hay errores, el código ahora ha sido compilado y cargado en el dispositivo. Encontramos una referencia a nuestra pycuda.driver.Function y la llamamos, especificando a_gpu como el argumento, y un tamaño de bloque de $4\times 4$:
Can you write Python code for it?

mod

func = mod.get_function("duplicar")
func(a_gpu, block=(4,4,1))

func


I am working on the file "Parte 2 - PyCUDA y aplicaciones/01 - Introduccion a PyCUDA.ipynb".
The first task is:
Transferir datos
El siguiente paso es transferir datos al GPU. Principalmente arreglos de numpy. Por ejemplo, tomemos un arreglo de números aleatorios de $4 \times 4$
Can you write Python code for it?

import numpy
a = numpy.random.randn(4,4)


I am working on the file "Parte 2 - PyCUDA y aplicaciones/01 - Introduccion a PyCUDA.ipynb".
The first task is:
como último paso, necesitamos tranferir los datos al GPU
Can you write Python code for it?


I am working on the file "Parte 2 - PyCUDA y aplicaciones/01 - Introduccion a PyCUDA.ipynb".
The first task is:
finalmente, necesitmos un arreglo hacia el cuál transferir la información, así que deberíamos guardar la memoria en el dispositivo:
Can you write Python code for it?


I am working on the file "Parte 2 - PyCUDA y aplicaciones/06 - Impresiones y tiempos en PyCUDA.ipynb".
The first task is:
En efecto se ejecutaron uno a la vez.
Rendimiento
A continuación agregamos un código para comparar el rendimiento de varias formas de operar. Cada una es clarificada en el código.
Can you write Python code for it?

import pycuda.driver as drv
import pycuda.tools
import pycuda.autoinit
import numpy
from pycuda.compiler import SourceModule
import pycuda.gpuarray as gpuarray
import pycuda.cumath
from pycuda.elementwise import ElementwiseKernel

blocks = 64
block_size = 128
valores = blocks * block_size

print "Usando", valores, "valores"

# Número de iteraciones para los cálculos
n_iter = 100000
print "Calculando %d iteraciones" % (n_iter)

# Crear dos timers
inicio = drv.Event()
fin = drv.Event()

# SourceModele

mod = SourceModule("""
__global__ void gpusin(float *dest, float *a, int n_iter)
{
  const int i = blockDim.x*blockIdx.x + threadIdx.x;
  for(int n = 0; n < n_iter; n++) {
    a[i] = sin(a[i]);
  }
  dest[i] = a[i];
}
""")

gpusin = mod.get_function("gpusin")

# creamos un arreglo 1s
a = numpy.ones(valores).astype(numpy.float32)
# creamos un arreglo para guardar el resultado
dest = numpy.zeros_like(a)

inicio.record() # comenzamos a tomar el tiempo
gpusin(drv.Out(dest), drv.In(a), numpy.int32(n_iter), grid=(blocks,1), block=(block_size,1,1) )
fin.record() # terminamos de tomar el tiempo
# calculamos cuánto duró la corrida
fin.synchronize()
segs = inicio.time_till(fin)*1e-3
print "Tiempo con SourceModule y primeros 3 resultados:"
print "%fs, %s" % (segs, str(dest[:3]))

# Usando ElementwiseKernel con sin en un ciclo for en C

kernel = ElementwiseKernel(
   "float *a, int n_iter",
   "for(int n = 0; n < n_iter; n++) { a[i] = sin(a[i]);}",
   "gpusin")

a = numpy.ones(valores).astype(numpy.float32)
a_gpu = gpuarray.to_gpu(a)
inicio.record() # comenzamos a tomar el tiempo
kernel(a_gpu, numpy.int(n_iter))
fin.record() # terminamos de tomar el tiempo
# calculamos cuánto duró la corrida
fin.synchronize()
segs = inicio.time_till(fin)*1e-3
print "Tiempo con Elementwise y primeros 3 resultados:"
print "%fs, %s" % (segs, str(a_gpu.get()[:3]))

# Elementwise haciendo el loop en Python

kernel = ElementwiseKernel(
   "float *a",
   "a[i] = sin(a[i]);",
   "gpusin")

a = numpy.ones(valores).astype(numpy.float32)
a_gpu =...


I am working on the file "Parte 2 - PyCUDA y aplicaciones/06 - Impresiones y tiempos en PyCUDA.ipynb".
The first task is:
Por último hagamos un kernel que imprima más datos acerca del thread, como su índice de bloque, la dimensión del bloque, y sus índices dentro del bloque.
Can you write Python code for it?

%%writefile ./Programas/saludar_bloques.py
import pycuda.driver as cuda
import pycuda.autoinit
from pycuda.compiler import SourceModule

mod = SourceModule("""
    #include <stdio.h>

    __global__ void say_hi()
    {
      printf("Soy el thread numero %d en threadIdx.x:%d.threadIdx.y:%d  blockIdx.:%d blockIdx.y:%d blockDim.x:%d blockDim.y:%d\\n",(threadIdx.x+threadIdx.y*blockDim.x+(blockIdx.x*blockDim.x*blockDim.y)+(blockIdx.y*blockDim.x*blockDim.y)),threadIdx.x, threadIdx.y,blockIdx.x,blockIdx.y,blockDim.x,blockDim.y);
    }
    """)

func = mod.get_function("say_hi")
func(block=(4,4,1),grid=(2,2,1))


I am working on the file "Partial Fraction Expansion.ipynb".
The first task is:
Complex roots
Here's another example with complex roots to see what happens
$$
Y(s) = U(s) G(s) = \frac{1}{s} \frac{2s+10}{s^2 +2s+10}
$$
Can you write Python code for it?


I am working on the file "Participants/JCarlos/02_Parsing.ipynb".
The first task is:
Plot the interaction matrix and print a summary of the interactions:
Cis interactions
Can you write Python code for it?

from pytadbit.mapping.analyze import insert_sizes

insert_sizes('results/HindIII/03_filtering/reads12.tsv',show=True,nreads=100000)

from pytadbit.mapping.filter import filter_reads

filter_reads('results/HindIII/03_filtering/reads12.tsv',max_molecule_length=750,min_dist_to_re=500)


I am working on the file "Participants/JCarlos/02_Parsing.ipynb".
The first task is:
The output are the tables with the reads as ID, chrm, pos, lenght, and start-end fragment
Can you write Python code for it?

! head -n 50 results/HindIII/02_parsing/reads1.tsv

from pytadbit.mapping import get_intersection


I am working on the file "Participants/JCarlos/02_Parsing.ipynb".
The first task is:
This function will filter all reads that are not mapped in both sides
Can you write Python code for it?

! mkdir -p results/HindIII/03_filtering

get_intersection('results/HindIII/02_parsing/reads1.tsv',
                'results/HindIII/02_parsing/reads2.tsv',
                'results/HindIII/03_filtering/reads12.tsv',
                verbose=True)

! head -n50 results/HindIII/03_filtering/reads12.tsv

from pytadbit.mapping.analyze import plot_distance_vs_interactions


I am working on the file "Participants/JCarlos/Finding Compartments.ipynb".
The first task is:
We don´t have normalized data at 300Kb resolution!
then normalize again before finding compartments as follow
(*all previuos steps were done at 1Mb resolution)
Can you write Python code for it?

hic_data1 = load_hic_data_from_reads(
'results/HindIII/03_filtering/valid_reads12_HindIII.tsv',
resolution= reso)

hic_data1.bias = load(open(
    'results/HindIII/04_normalizing/biases_1000000_HindIII.pick'))

hic_data1.bads = load(open(
    'results/HindIII/04_normalizing/bad_columns_1000000_HindIII.pick'))

hic_data2 = load_hic_data_from_reads(
'results/MboI/03_filtering/valid_reads12_MboI.tsv',
resolution= reso)

hic_data2.bias = load(open(
    'results/MboI/04_normalizing/biases_1000000_MboI.pick'))

hic_data2.bads = load(open(
    'results/MboI/04_normalizing/bad_columns_1000000_MboI.pick'))

hic_data1.normalize_hic()

hic_data2.normalize_hic()

hic_data = load_hic_data_from_reads(
'results/both/03_filtering/valid_reads12.tsv',
resolution=reso)

hic_data.normalize_hic()


I am working on the file "Participants/Nazaret/02_parsing.ipynb".
The first task is:
In Y we have number of reads per 500kb bins
It reflects different number of copies, for example in chromosome 2 the last part of the chromosome has less copies than the rest, therefore the lower number of reads.
Big peaks correspond to PCR duplicates
Can you write Python code for it?

from pytadbit.mapping.analyze import hic_map

hic_map("results/HindIII/03_filtering/reads12.tsv", resolution=1000000, show=True)

from pytadbit.mapping.analyze import insert_sizes

insert_sizes("results/HindIII/03_filtering/reads12.tsv", show=True, nreads=100000)

from pytadbit.mapping.filter import filter_reads

filter_reads("results/HindIII/03_filtering/reads12.tsv", max_molecule_length=750, min_dist_to_re=500)

masked = {1: {'fnam': 'results/HindIII/03_filtering/reads12.tsv_self-circle.tsv',
  'name': 'self-circle',
  'reads': 37383},
 2: {'fnam': 'results/HindIII/03_filtering/reads12.tsv_dangling-end.tsv',
  'name': 'dangling-end',
  'reads': 660146},
 3: {'fnam': 'results/HindIII/03_filtering/reads12.tsv_error.tsv',
  'name': 'error',
  'reads': 37395},
 4: {'fnam': 'results/HindIII/03_filtering/reads12.tsv_extra_dangling-end.tsv',
  'name': 'extra dangling-end',
  'reads': 3773498},
 5: {'fnam': 'results/HindIII/03_filtering/reads12.tsv_too_close_from_RES.tsv',
  'name': 'too close from RES',
  'reads': 3277369},
 6: {'fnam': 'results/HindIII/03_filtering/reads12.tsv_too_short.tsv',
  'name': 'too short',
  'reads': 296853},
 7: {'fnam': 'results/HindIII/03_filtering/reads12.tsv_too_large.tsv',
  'name': 'too large',
  'reads': 1843},
 8: {'fnam': 'results/HindIII/03_filtering/reads12.tsv_over-represented.tsv',
  'name': 'over-represented',
  'reads': 411157},
 9: {'fnam': 'results/HindIII/03_filtering/reads12.tsv_duplicated.tsv',
  'name': 'duplicated',
  'reads': 324490},
 10: {'fnam': 'results/HindIII/03_filtering/reads12.tsv_random_breaks.tsv',
  'name': 'random breaks',
  'reads': 968492}}

from pytadbit.mapping.filter import apply_filter

apply_filter ("results/HindIII/03_filtering/reads12.tsv",
              "results/HindIII/03_filtering/reads12_valid.tsv",
              masked, filters=[1, 2, 3, 4, 9, 10])


I am working on the file "Participants/Nazaret/02_parsing.ipynb".
The first task is:
the orange slope is the most important and you expect a slope of around -1 unless you work with cells badly synchronized or with weird karyotype
resolution= size of bins
Can you write Python code for it?

from pytadbit.mapping.analyze import plot_genomic_distribution

plot_genomic_distribution("results/HindIII/03_filtering/reads12.tsv", resolution=500000, show=True)


I am working on the file "Particle Distribution Visualization.ipynb".
The first task is:
Alternatively one can ask for a pandas dataframe where each column is one of the above properties of a macroparticle per row.
Can you write Python code for it?

panda_data = data.DistFrame()
panda_data[0:5]


I am working on the file "Particle Distribution Visualization.ipynb".
The first task is:
And finally there is the FEL_Approximations which calculate simple FEL properties per slice. This is a subclass of statistics and as such every method described above is callable.
This class conatins the 'undulator' function that calculates planar undulator parameters given a period and either a peak magnetic field or K value.
The data must be sliced and most statistics have to be run before the other calculations can take place.
These are 'pierce' which calculates the pierce parameter and 1D gain length for a given slice, 'gain length' which calculates the Ming Xie gain and returns three entries in the dict 'MX_gain', '1D_gain', 'pierce', which hold an array for these values per slice.
'FELFrame' returns a pandas dataframe with these and 'z_pos' for reference.
To make this easier, the class ProcessedData takes a filepath, number of slcies, undulator period, magnetic field or K and performs all the necessary steps automatically. As this is a subclass of FEL_Approximations all the values written above are accessible from here.
Can you write Python code for it?

FEL = pt.ProcessedData(filepath,num_slices=100,undulator_period=0.00275,k_fact=2.7)

panda_FEL = FEL.FELFrame()
panda_stats= FEL.StatsFrame()
panda_FEL[0:5]


I am working on the file "Particle Distribution Visualization.ipynb".
The first task is:
If it is important to plot the statistical data alongside the FEL data, that can be easily achieved by concatinating the two sets as shown below
Can you write Python code for it?

import pandas as pd

cat = pd.concat([panda_FEL,panda_stats], axis=1, join_axes=[panda_FEL.index]) #joins the two if you need to plot
#FEL parameters as well as slicel statistics on the same plot
cat['1D_gain']=cat['1D_gain']*40000000000 #one can scale to allow for visual comparison if needed
az = cat.plot(x='z_pos',y='1D_gain')
cat.plot(ax=az, x='z_pos',y='MX_gain',c='b')
plt.show()


I am working on the file "Particle Distribution Visualization.ipynb".
The first task is:
The module consists of a class 'ParticleDistribution' that initializes to a dictionary containing the following entries given a filepath:
|key | value     |
|----|-----------|
|'x' | x position|
|'y' | y position|
|'z' | z position|
|'px'| x momentum|
|'py'| y momentum|
|'pz'| z momentum|
|'NE'| number of electrons per macroparticle|
The units are in line with the Standard Unit specifications, but can be converted to SI by calling the class method SU2SI
Values can then be called by calling the 'dict':
Can you write Python code for it?

filepath = './example/example.h5'

data = pt.ParticleDistribution(filepath)
data.su2si
data.dict['x']


I am working on the file "Particle Distribution Visualization.ipynb".
The first task is:
This allows for quick plotting using the inbuilt pandas methods
Can you write Python code for it?

import matplotlib.pyplot as plt
matplotlib.style.use('ggplot') #optional

x_axis = 'py'
y_axis = 'px'

plot = panda_data.plot(kind='scatter',x=x_axis,y=y_axis)
#sets axis limits
plot.set_xlim([panda_data[x_axis].min(),panda_data[x_axis].max()])
plot.set_ylim([panda_data[y_axis].min(),panda_data[y_axis].max()])
plt.show(plot)


I am working on the file "Partie5_Controle_PID.ipynb".
The first task is:
Comparé à la commande en boucle ouverte, on constate qu'avec la commande boucle fermée avec terme proportionnel, on peut atteindre des valeurs $S^\ast$ de $S$ qui étaient non atteignables avec la boucle ouverte.
Testons maintenant si cette commande est robuste aux perturbations.
Test: Loi de commande en BOUCLE FERMEE avec action proportionnelle (P), AVEC PERTURBATION
Can you write Python code for it?

# tracé interactif de l'évolution de la dynamique boucle fermée en fonction de la valeur de Sast, kprop et disturb
# dans le cas d'une commande en BOUCLE FERMEE PROPORTIONNELLE AVEC PERTURBATION
def test(Sast,kprop,disturb):
    return culture_cont(Sast,'P',kprop,disturb)
interact(test,Sast=(0,4,0.1),kprop=(0,0.1,0.01),disturb=(0,0.4,0.01))


I am working on the file "Partie5_Controle_PID.ipynb".
The first task is:
Limites de la boucle ouverte
On touche ici aux limites de la commande boucle ouverte.

la commande boucle ouverte ne peut être utilisée que pour certaines valeurs de consignes
cette commande n'est pas robuste aux perturbations

Action proportionnelle
On va maintenant tester une loi de commande boucle fermée constituée d'un terme proportionnel à l'erreur, c'est à dire une loi de commande de la forme:
$$ u(t)=u^\ast+K_p(y^\ast-y^m(t))$$
où $u^\ast$ est la valeur de la loi de commande constante boucle ouverte qui permet de stabiliser le système à $S^\ast$
Dans le cas de notre exemple, cela correspondant à la loi de commande:
$$ Q(t)=V\mu(S^\ast)+K_p(S^\ast-S^m(t))$$
Can you write Python code for it?

# Contrôle boucle fermée de la culture bactérienne dans un réacteur continu :  commande proportionnelle
# -----------------------------------------------------------------------------------------------------

# Loi de commande
def fonction_u(t,x,Sast,k,muast,KI,KS,V,S0,control_type,coeffcontrol):
    if control_type == 'BO': # BOUCLE OUVERTE
        # loi donnée par Qast=mu(Sast)*V
        Qast = muast*Sast/(KS+Sast+Sast**2/KI)*V
        if type(t)==float:  # cas où t est scalaire
            valu = Qast
        else: # cas où t est un vecteur
            valu = np.ones(len(t))*Qast
    elif control_type == 'P': # BOUCLE FERMEE action PROPORTIONNELLE
        # récupération des paramètres de la loi de commande
        kprop = coeffcontrol
        # et de la valeur de S
        if type(t)==float: # cas où t est scalaire
            valS = x[1]
        else: # cas où t est un vecteur
            valS = x[:,1]
        # loi donnée par mu(Sast)*V+kprop*(Sast-S)
        valu = muast*Sast/(KS+Sast+Sast**2/KI)*V+kprop*(Sast-valS)
    return valu


I am working on the file "Partie5_Controle_PID.ipynb".
The first task is:
On remarque que, en ajoutant le terme intégral dans la commande, cela permet d'annuler l'erreur que l'on faisait avec les lois de commande boucle ouverte ou boucle fermée proportionnelle.
Les lois de commande Proportionnelle Intégrale sont donc robustes aux perturbations!
Action Dérivée
On peut également rajouter un terme dérivée dans la loi de commande boucle fermée. On obtient alors la loi de commande proportionnelle intégrale dérivée (PID) avec un terme proportionnel à l'erreur, un terme proportionnel à l'intégrale de l'erreur et un terme proportionnel à la dérivée de l'erreur, c'est à dire une loi de commande de la forme:
$$ u(t)=uK_0+K_p(y^\ast-y^m(t))+K_i\int_0^t(y^\ast-y^m(s))ds+K_d\frac{d(y^\ast-y^m)}{dt}$$
Dans le cas de notre exemple, cela correspondant à la loi de commande:
$$ Q(t)=K_0+K_p(S^\ast-S^m(t))+K_i\int_0^t(S^\ast-S^m(s))ds+K_d \frac{d(S^\ast-S^m)}{dt}$$
Application pratique
Numériquement, pour calculer cette loi de commande on peut remplacer $\frac{d(S^\ast-S^m(s))}{dt}=-\frac{S^m}{dt}$ (car $S^\ast$ est une constante) par le second membre de l'équation en $S$ du modèle, c'est à dire par $-k\mu(S^m)B^m+\frac{Q}{V}(S_0-S^m)$. On a alors:
$$ Q(t)=K_0+K_p(S^\ast-S^m(t))+K_i\int_0^t(S^\ast-S^m(s))ds-K_d \left(-k\mu(S^m(t))B^m(t)+\frac{Q(t)}{V}(S_0-S^m(t))\right)$$
Comme la commande $Q$ se retrouve alors dans les deux termes de l'égalité, il faut résoudre cette équation. En passant tous les termes dépendant de $Q$ dans le terme de gauche, et en multipliant par le bon coefficient on a:
$$ Q(t)=\left[K_0+K_p(S^\ast-S^m(t))+K_i\int_0^t(S^\ast-S^m(s))ds+K_d k\mu(S^m(t))B^m(t)\right] \frac{1}{1+\frac{K_d}{V}(S_0-S^m(t))}$$
Can you write Python code for it?

# Contrôle boucle fermée de la culture bactérienne dans un réacteur continu: commande proportionnelle intégrale dérivée
# ---------------------------------------------------------------------------------------------------------------------

# Loi de commande
def fonction_u(t,x,Sast,k,muast,KI,KS,V,S0,control_type,coeffcontrol):
    if control_type == 'BO': # BOUCLE OUVERTE
        # loi donnée par Qast=mu(Sast)*V
        Qast = muast*Sast/(KS+Sast+Sast**2/KI)*V
        if type(t)==float:  # cas où t est scalaire
            valu = Qast
        else: # cas où t est un vecteur
            valu = np.ones(len(t))*Qast
    elif control_type == 'P': # BOUCLE FERMEE action PROPORTIONNELLE
        # récupération des paramètres de la loi de commande
        kprop = coeffcontrol
        # et de la valeur de S
        if type(t)==float: # cas où t est scalaire
            valS = x[1]
        else: # cas où t est un vecteur
            valS = x[:,1]
        # loi donnée par mu(Sast)*V+kprop*(Sast-S)
        valu = muast*Sast/(KS+Sast+Sast**2/KI)*V+kprop*(Sast-valS)
    elif control_type == 'PI': # BOUCLE FERMEE action PROPORTIONNELLE INTEGRALE
        # récupération des paramètres de la loi de commande
        k0 = coeffcontrol[0]
        kprop = coeffcontrol[1]
        kint = coeffcontrol[2]
        # et de la valeur de valint, qui est l'intégrale entre 0 et t de Sast-S
        if type(t)==float: # cas où t est scalaire
            valint = x[2]; valS = x[1]
        else: # cas où t est un vecteur
            valint = x[:,2]; valS = x[:,1]
        # loi donnée par k0+kprop*(Sast-S) + kint*valint
        # où valint est l'intégrale entre 0 et t de Sast-S
        valu = k0+kprop*(Sast-valS)+kint*valint
    elif control_type == 'PID': # BOUCLE FERMEE action PROPORTIONNELLE INTEGRALE DERIVEE
        # récupération des paramètres de la loi de commande
        k0 = coeffcontrol[0]
        kprop = coeffcontrol[1]
        kint = coeffcontrol[2]
        kderiv = coeffcontrol[3]
        # et des valeurs de valint...


I am working on the file "Partie5_Controle_PID.ipynb".
The first task is:
Test 1: Loi de commande en BOUCLE OUVERTE, SANS PERTURBATION
Ici $k = 0.6$; $\mu^\ast = 2.3$; $K_S = 10$; $K_I = 0.1$; $S^{in} = 3.2$; $V = 0.5$;
donc les valeurs de $S$ atteignables sont $\left[0,\sqrt{K_SK_I}\right]\cup \left{S^{in} \right}=[0,1]\cup\left{3.2  \right}$
Can you write Python code for it?

# tracé interactif de l'évolution de la dynamique boucle fermée en fonction de la valeur de Sast
# dans le cas d'une commande en BOUCLE OUVERTE SANS PERTURBATION
def test(Sast):
    return culture_cont(Sast,'BO',0,0)
interact(test,Sast=(0,2,0.1))


I am working on the file "Partie5_Controle_PID.ipynb".
The first task is:
Test: Loi de commande en BOUCLE FERMEE avec action proportionnelle (P), SANS PERTURBATION
Can you write Python code for it?

# tracé interactif de l'évolution de la dynamique boucle fermée en fonction de la valeur de Sast et kprop
# dans le cas d'une commande en BOUCLE FERMEE PROPORTIONNELLE SANS PERTURBATION
def test(Sast,kprop):
    return culture_cont(Sast,'P',kprop,0)
interact(test,Sast=(0,4,0.1),kprop=(0,0.1,0.01))


I am working on the file "Partie5_Controle_PID.ipynb".
The first task is:
Test: Loi de commande en BOUCLE FERMEE avec actions proportionnelle et intégrale (PI), AVEC PERTURBATION
Can you write Python code for it?

# tracé interactif de l'évolution de la dynamique boucle fermée en fonction de la valeur de Sast, kprop, kint et disturb
# dans le cas d'une commande en BOUCLE FERMEE PROPORTIONNELLE INTEGRALE AVEC PERTURBATION
def test(Sast,k0,kprop,kint,disturb):
    return culture_cont(Sast,'PI',np.array([k0,kprop,kint]),disturb)
interact(test,Sast=(0,4,0.1),k0=(0,0.1,0.01),kprop=(0,0.1,0.01),kint=(0,0.02,0.001),disturb=(0,0.4,0.01))


I am working on the file "Partie5_Controle_PID.ipynb".
The first task is:
Test: Loi de commande en BOUCLE FERMEE avec actions proportionnelle et intégrale (PI), SANS PERTURBATION
Can you write Python code for it?

# tracé interactif de l'évolution de la dynamique boucle fermée en fonction de la valeur de Sast, kprop et kint
# dans le cas d'une commande en BOUCLE FERMEE PROPORTIONNELLE INTEGRALE SANS PERTURBATION
def test(Sast,k0,kprop,kint):
    return culture_cont(Sast,'PI',np.array([k0,kprop,kint]),0)
interact(test,Sast=(0,4,0.1),k0=(0,0.1,0.01),kprop=(0,0.1,0.01),kint=(0,0.02,0.001))


I am working on the file "Partie5_Controle_PID.ipynb".
The first task is:
Test: Loi de commande en BOUCLE FERMEE avec actions proportionnelle intégrale et dérivée (PID), AVEC PERTURBATION
Can you write Python code for it?

# tracé interactif de l'évolution de la dynamique boucle fermée en fonction de la valeur de Sast, kprop, kint, kderiv
# et disturb dans le cas d'une commande en BOUCLE FERMEE PROPORTIONNELLE INTEGRALE DERIVEE AVEC PERTURBATION
def test(Sast,k0,kprop,kint,kderiv,disturb):
    return culture_cont(Sast,'PID',np.array([k0,kprop,kint,kderiv]),disturb)
interact(test,Sast=(0,4,0.1),k0=(0,0.1,0.01),kprop=(0,0.1,0.01),kint=(0,0.02,0.001),kderiv=(0,0.5,0.1),disturb=(0,0.4,0.01))


I am working on the file "Partie5_TP_Controle_PID_sujet.ipynb".
The first task is:
2. Commande boucle ouverte
On note $S^\ast$ la valeur de $S$ que l'on souhaite atteindre (la consigne).
Trouver la commande (constante) en boucle ouverte revient à chercher la valeur de $Q$ telle que la valeur de $S$ à l'équilibre vaille $S^\ast$.
A COMPLETER
Can you write Python code for it?

# --------------------------
# Commande en boucle ouverte
# --------------------------
# modèle de fermentation en réacteur continu:
# on modifie légèrement la fonction qui code le modèle de fermentation en réacteur continu
# pour pouvoir tester des lois de commande
# On rajoute les variables suivantes:
#  Sast : la valeur de S qu'on veut atteindre (consigne)
# control_type : le type de loi de commande à appliquer
# coeffcontrol : les paramètres de la loi de commande
# disturb: une valeur représentant une perturbation de la commande.
# La commande réellement appliquée n'est pas exactement égal à celle calculée.
# On a: Qreal = Qcalc*(1+disturb)

def cont_fermenter(x,t,k1,k2,mu1max,mu2max,KN,KE,KS,Qin,V,Nin,Sin, \
                  Sast,control_type,coeffcontrol,disturb):

    # variables d'état du systèeme
    B = x[0] # biomasse
    N = x[1] # azote
    E = x[2] # ethanol
    S = x[3] # sucre

    # initialisation du vecteur second membre du système
    dx = np.zeros(4) # dérivées de B, N, E et S

    # loi de commande
    Q = fonction_u(t,x,Sast,k1,k2,mu1max,mu2max,KN,KE,KS,Qin,V,Nin,Sin, \
                   control_type,coeffcontrol)*(1+disturb)

    # calcul de mu1(S)
    mu1 = mu1max*N/(KN+N)
    # calcul de mu2(E,S)
    mu2 = mu2max*S/(KS+S)*KE/(KE+E)

    # second membre de l'équation de B
    dx[0] = mu1*B-Q/V*B
    # second membre de l'équation de N
    dx[1] = -k1*mu1*B+Q/V*(Nin-N)
    # second membre de l'équation de E
    dx[2] = mu2*B-Q/V*E
    # second membre de l'équation de S
    dx[3] = -k2*mu2*B+Q/V*(Sin-S)

    return dx

def fonction_u(t,x,Sast,k1,k2,mu1max,mu2max,KN,KE,KS,Qin,V,Nin,Sin, \
               control_type,coeffcontrol):

    if control_type == 'BO': # Controle BOUCLE OUVERTE
        Qast = coeffcontrol
        valu = np.ones(np.size(t))*Qast
    return valu

def culture_cont(Sast,control_type,coeffcontrol,disturb):
    # vecteur temps
    tmax = 150
    temps = np.linspace(0,tmax,2000) # vecteur temps

    #...


I am working on the file "Partie5_TP_Controle_PID_sujet.ipynb".
The first task is:
1. Calcul des points d'équilibre
Rappelons d'abord que le modèle du fermenteur continu s'écrit:
$$
\left{
\begin{array}{crl}
\frac{dX}{dt}= & \mu_1(N)X &-DX\
\frac{dN}{dt}= & -k_1\mu_1(N)X&+D(N_{in}-N)  \
\frac{dE}{dt}= & \mu_2(E,S)X &-DE \
\frac{dS}{dt}= & -k_2\mu_2(E,S)X&+D(S_{in}-S)
\end{array}
\right.
\text{ avec }
\left{
\begin{array}{lcr}
 \mu_1(N)&=&\mu_1^{max}\frac{N}{K_N+N}\
 \mu_2(E,S)&=&\mu_2^{max}\frac{S}{K_S+S}\frac{K_E}{K_E+E}.
\end{array}
\right.
$$
On utilisera comme valeurs de paramètres les valeurs identifiées à l'étape précédente.
On peut montrer (admis) que le domaine $\mathcal{D}=\left{ (X,N,E,S)\in\mathbb{R}^4\text{ tel que }0\leqslant X \leqslant \frac{N_{in}-N}{k_1},\,0\leqslant N \leqslant N_{in},\,0\leqslant E \leqslant \frac{S_{in}-S}{k_2},\,0\leqslant S \leqslant S_{in},  \right}$ est un invariant positif du système. Cela veut dire que si la condition initiale est dans $\mathcal{D}$ alors la trajectoire solution du système restera dans $\mathcal{D}$.
Can you write Python code for it?

# modèle de fermentation en réacteur continu
def cont_fermenter(x,t,k1,k2,mu1max,mu2max,KN,KE,KS,Qin,V,Nin,Sin):
    # variables d'état du systèeme
    B = x[0] # biomasse
    N = x[1] # azote
    E = x[2] # ethanol
    S = x[3] # sucre

    # initialisation du vecteur second membre du système
    dx = np.zeros(4) # dérivées de B, N, E et S

    # calcul de mu1(S)
    mu1 = mu1max*N/(KN+N)
    # calcul de mu2(E,S)
    mu2 = mu2max*S/(KS+S)*KE/(KE+E)

    # second membre de l'équation de B
    dx[0] = mu1*B-Qin/V*B
    # second membre de l'équation de N
    dx[1] = -k1*mu1*B+Qin/V*(Nin-N)
    # second membre de l'équation de E
    dx[2] = mu2*B-Qin/V*E
    # second membre de l'équation de S
    dx[3] = -k2*mu2*B+Qin/V*(Sin-S)

    return dx

# paramètres du modèle de réacteur batch: valeurs identifiées à l'étape précédente
k2 = 2.08478746168
k1 = 0.0280270885286
mu1max = 2.79969755302
mu2max = 4.38785020243
KN = 0.965992617646
KE = 3.13132366747
KS = 0.0396396754502
B0 = 0.000239863343873
N0 = 0.1994142379
E0 = 0.216807301097
S0 = 193.763999683
# paramètres supplémentaires pour le modèle de réacteur continu
Qin = 0.1
V = 0.5
Nin = 0.425
Sin = 200


I am working on the file "Partie5_TP_Controle_PID_sujet.ipynb".
The first task is:
Pour claculer les points d'équilibre du système, il faut résoudre le système:
$$
\left{
\begin{array}{r}
\frac{dX}{dt} =  0\
\frac{dN}{dt} =  0  \
\frac{dE}{dt} =  0 \
\frac{dS}{dt} =  0
\end{array}
\right.
\Leftrightarrow
\left{
\begin{array}{rl}
\mu_1(N)X -DX& =0\
-k_1\mu_1(N)X+D(N_{in}-N)& =0  \
\mu_2(E,S)X -DE & =0\
-k_2\mu_2(E,S)X+D(S_{in}-S) & =0
\end{array}
\right.
$$
Le résolution de ce système mène à deux cas:

Cas n°1: $X^{eq}=0$

A COMPLETER
On note ce point d'équilibre $E_0$:
$$\boxed{E_0=\left(X^{eq}= ,  N^{eq}=, E^{eq}=, S^{eq}= \right)}$$

Cas n°2: $X^{eq}\neq 0$

A COMPLETER
On note $E_1$ ce nouveau point d'équilibre donné par:
$$\boxed{E_1=\left(X^{eq}= ,  N^{eq}=, E^{eq}=, S^{eq}=\right) }$$
Can you write Python code for it?

# Calcul des points d'équilibre
# -----------------------------
# A COMPLETER
# valeur de N à l'équilibre
Neq = 0
# valeur de X à l'équilibre
Beq = 0
# valeur de S à l'équilibre:
Seq = 0
# Valeur de E à l'équilibre
Eeq = 0

# Simulation du modèle
# --------------------
# conditions initiales
x0 = np.array([B0, N0, E0, S0])

# vecteur temps
t = np.arange(0,300,0.1)

# integration numerique de l'EDO
y = scint.odeint(cont_fermenter,x0,t,args=(k1,k2,mu1max,mu2max,KN,KE,KS,Qin,V,Nin,Sin))

# Tracé
# -----
# tracé des variables simulées et des points d'équilibre pour vérifier que les trajectoires convergent bien vers le point
# d'équilibre calculé
f2, ax2 = plt.subplots(3,1,figsize = (10, 9))

ax2[0].plot(t,y[:,0],'r',label='Biomass')
ax2[0].plot([t[0],t[-1]],Beq*np.array([1,1]),'r--',label='Biomass at eq')
ax2[0].plot(t,y[:,1]*10,'g',label='Nitrogen')
ax2[0].plot([t[0],t[-1]],Neq*np.array([10,10]),'g--',label='Nitrogen at eq')
ax2[0].legend()
ax2[0].set_xlabel('time (h)')

ax2[1].plot(t,y[:,2],'b',label='Ethanol')
ax2[1].plot([t[0],t[-1]],Eeq*np.array([1,1]),'b--',label='Ethanol at eq')
ax2[1].plot(t,y[:,3],'k',label='Sugar')
ax2[1].plot([t[0],t[-1]],Seq*np.array([1,1]),'k--',label='Sugar at eq')
ax2[1].legend()
ax2[1].set_xlabel('time (h)')

ax2[2].plot(t,mu2max*y[:,3]/(KS+y[:,3])*KE/(KE+y[:,2])*y[:,0],'m',label=r'$\frac{dCO_2}{dt}$ (simulated)')
ax2[2].legend()
ax2[2].set_xlabel('time (h)')

plt.show()


I am working on the file "Partie6_Control_Obs_Sat.ipynb".
The first task is:
<a name="Sat"> Saturations </a>
Dans le paragraphe précédent, on a vu que, lorsque certaines variables du système n'étaient pas mesurées, il était possible de les estimer grâce à un observateur.
Un autre problème que l'on peut rencontrer en pratique vient des saturations sur les commandes. En effet, pour des raisons souvent physiques, on ne peut pas appliquer n'importe quelle commande à notre système. Il y a en effet des limites physiques aux valeurs de commande, que l'on peut exprimer de la manière suivante:
$$ u_{min} ≤ u ≤ u_{max}$$

Exemple: Croissance d'une population dans un réacteur continu (suite)
Dans cet exemple la commande étant un débit d'entrée, celle-ci ne pourra pas être négative. Les limites physiques de la pompe impose également une valeur de débit maximale, que l'on notera $Q_{max}$ et que l'on prendra égale à $0.1 L/h$.
On a donc:
$$ 0 ≤ Q ≤ Q_{max}$$
Or, on voit sur la figure précédente que la loi de commande que l'on a utilisée propose des valeurs de $Q$ négatives et supérieures à $Q_{max}$. C'est donc une loi de commande que l'on ne pourra pas appliquer telle qu'elle au système.

Un première solution à ce problème consiste à appliquer une saturation à la commande calculée $u_{calc}$.
La valeur $u_{réelle}$ que l'on appliquera sera déterminée à partir de $u_{calc}$ de la manière suivante:
$$ u_{réelle} = \left{ \begin{array}{ll}
u_{min} & \text{si }u_{calc} ≤ u_{min}\
u_{calc} & \text{si }u_{min} < u_{calc} < u_{max}\
u_{max} & \text{si }u_{max} ≤ u_{calc}\
\end{array}\right.$$
Can you write Python code for it?

def reacteur_obs(x,t,k,muast,KS,KI,Qin,V,S0,Qmax,Sast,control_type,coeffcontrol,obs,Cobs,sat,coeffsat,disturb):
    B = x[0] #biomass
    S = x[1] #substrat

    if control_type in ['PI','PID']:
        indObs = 3
        Shat = x[indObs+1]
        dx = np.zeros(6)
        dx[2] = Sast-S
        dx[5] = Sast-Shat
    else:
        dx = np.zeros(4)
        indObs = 2

    if obs == 0:
        xest = x[0:indObs]
    elif obs == 1:
        xest = x[indObs:]

    Q = fonction_u(t,xest,Sast,k,muast,KI,KS,V,S0,control_type,coeffcontrol)*(1+disturb)

    if sat == 1:
        if Q<0: Q=0
        elif Q>Qmax: Q=Qmax

    mu = muast*S/(KS+S+S**2/KI)
    dx[0] = mu*B-Q/V*B
    dx[1] = -k*mu*B+Q/V*(S0-S)

    Bhat = x[indObs]
    Shat = x[indObs+1]
    muhat = muast*Shat/(KS+Shat+Shat**2/KI)
    dx[indObs] = muhat*Bhat-Q/V*Bhat+Cobs[0]*(muhat*Bhat-mu*B)
    dx[indObs+1] = -k*muhat*Bhat+Q/V*(S0-Shat)+Cobs[1]*(muhat*Bhat-mu*B)
    return dx

def culture_cont3(Sast,control_type,coeffcontrol,obs,coeffobs,sat,coeffsat,disturb):
    tmax = 100
    temps = np.linspace(0,tmax,2000) # vecteur temps

    k = 0.6; muast = 2.3; KS = 10; KI = 0.1; S0 = 3.2; B0 = 9; Qin = 0.01; V = 0.5; Qmax = 0.1;

    B0obs = coeffobs[0]
    S0obs = coeffobs[1]
    Cobs = coeffobs[2:]

    # integration numerique de l'EDO
    if control_type in ['PI','PID']:
        x0 = np.array([B0,S0,0,B0obs,S0obs,0])
        indObs = 3
    else:
        x0 = np.array([B0,S0,B0obs,S0obs])
        indObs = 2
    x = scint.odeint(reacteur_obs,x0,temps,args=(k,muast,KS,KI,Qin,V,S0,Qmax,Sast, \
                                                 control_type,coeffcontrol,obs,Cobs,sat,coeffsat,disturb))
    if obs == 0:
        u = fonction_u(temps,x[:,0:indObs],Sast,k,muast,KI,KS,V,S0,control_type,coeffcontrol)
    elif obs == 1:
        u = fonction_u(temps,x[:,indObs:],Sast,k,muast,KI,KS,V,S0,control_type,coeffcontrol)

    if sat > 0: u = (u<=0)*0+(0<u)*(u<Qmax)*u+(u>Qmax)*Qmax

    plt.figure(figsize = (10, 3))
   ...


I am working on the file "Partie6_Control_Obs_Sat.ipynb".
The first task is:
Imposer une saturation sur une commande peut néanmoins dégrader fortement la dynamique du système en boucle fermée, voire carrément déstabiliser le système.
Cela est essentiellement dû au fait que lorsque la commande est saturée, le terme intégral continu de grossir car il continue d'intégrer l'erreur: on parle d'effet "wind-up".
 Méthode anti-windup
Pour pallier ce problème, on utilise des techniques d'"anti-wind-up" qui consistent en fait à vider le terme intégral (c'est à dire le diminuer) lorsque la commande est saturée. Concrètement, au lieu de calculer le terme intégral
$$I=K_i\int_0^t(y^\ast-y^m(s))ds$$
comme présenté précédemment, c'est à dire en résolvant:
$$ \frac{dI}{dt}=K_i(y^\ast-y^m),$$
on va résoudre l'équation:
$$ \frac{dI}{dt}=K_i(y^\ast-y^m)+K_{aw}(u_{réel}-u_{calc})$$.
Can you write Python code for it?

def reacteur_obs(x,t,k,muast,KS,KI,Qin,V,S0,Qmax,Sast,control_type,coeffcontrol,obs,Cobs,sat,coeffsat,disturb):
    B = x[0] #biomass
    S = x[1] #substrat

    if control_type in ['PI','PID']:
        indObs = 3
        Shat = x[indObs+1]
        dx = np.zeros(6)
        dx[2] = Sast-S
        dx[5] = Sast-Shat
    else:
        dx = np.zeros(4)
        indObs = 2

    if obs == 0:
        xest = x[0:indObs]
    elif obs == 1:
        xest = x[indObs:]

    Q = fonction_u(t,xest,Sast,k,muast,KI,KS,V,S0,control_type,coeffcontrol)*(1+disturb)

    if sat == 1:
        if Q<0: Q=0
        elif Q>Qmax: Q=Qmax
    elif sat == 2:
        if Q<0: Qr=0
        elif Q>Qmax: Qr=Qmax
        else:
            Qr = Q
        if control_type in ['PI','PID']:
            dx[2] = Sast-S+coeffsat*(Qr-Q)
            dx[5] = Sast-Shat+coeffsat*(Qr-Q)
        Q = Qr

    mu = muast*S/(KS+S+S**2/KI)
    dx[0] = mu*B-Q/V*B
    dx[1] = -k*mu*B+Q/V*(S0-S)

    Bhat = x[indObs]
    Shat = x[indObs+1]
    muhat = muast*Shat/(KS+Shat+Shat**2/KI)
    dx[indObs] = muhat*Bhat-Q/V*Bhat+Cobs[0]*(muhat*Bhat-mu*B)
    dx[indObs+1] = -k*muhat*Bhat+Q/V*(S0-Shat)+Cobs[1]*(muhat*Bhat-mu*B)
    return dx

# Simulation avec loi de commande PI, observateur et saturation et anti-windup
culture_cont3(2,'PI',np.array([0.07,0.01]),1,np.array([7,4,-2,0.1]),2,100,0.2)


I am working on the file "Parámetros de ecuaciones de estado (SRK, PR, RKPR)-Red Neuronal-Copy1.ipynb".
The first task is:
Como se observa, los resultados obtenidos son organizados en un DataFrame permitiendo agilizar la manipulación de los datos de una serie de sustancias puras.
Can you write Python code for it?


I am working on the file "Parámetros de ecuaciones de estado (SRK, PR, RKPR)-Red Neuronal-Copy1.ipynb".
The first task is:
De esta forma se observa el calculo simple de los parámetros para la sustancia pura 3-METHYLHEPTANE_RKPR
A continuación se realiza el mismo tipo de calculo pero tomando una serie de 9 sustancias puras, que se pueden extender facilmente a n sustancias, para obtener sus parámetros de nuevo con la ecuación de estado RKPR.
Can you write Python code for it?

properties_data = pt.Data_parse()

components = ["ISOBUTANE", "CARBON DIOXIDE", 'METHANE', "ETHANE", "3-METHYLHEPTANE", "n-PENTACOSANE",
              "NAPHTHALENE", "m-ETHYLTOLUENE", "2-METHYL-1-HEXENE"]

NMODEL = "RKPR"
ICALC = "constants_eps"
component_eos_list = np.zeros((len(components),4))

for index, component in enumerate(components):

    properties_component = properties_data.selec_component(component)
    pt.print_properties_component(component, properties_component)
    dinputs = np.array([properties_component[1]['Tc'], properties_component[1]['Pc'],
                        properties_component[1]['Omega'], properties_component[1]['Vc']])

    component_eos = pt.models_eos_cal(NMODEL, ICALC, dinputs)
    component_eos_list[index] = component_eos

components_table = pd.DataFrame(component_eos_list, index=components, columns=['ac', 'b', 'rm', 'del1'])

print(components_table)


I am working on the file "Parámetros de ecuaciones de estado (SRK, PR, RKPR)-Red Neuronal-Copy1.ipynb".
The first task is:
En el siguiente ejemplo se utiliza la ecuación RKPR pero esta vez con la especificación de la temperatura y densidad de líquido saturado para el CARBON DIOXIDE y de esta forma encontrar el valor del parámetro delta que verifica la especificación realizada para la densidad de líquido saturado.
Can you write Python code for it?

properties_data = pt.Data_parse()

dppr_file = "PureFull.xls"
component = "CARBON DIOXIDE"

NMODEL = "RKPR"
ICALC = "density"

properties_component = properties_data.selec_component(dppr_file, component)
pt.print_properties_component(component, properties_component)
#dinputs = np.array([properties_component[1]['Tc'], properties_component[1]['Pc'],
#                    properties_component[1]['Omega'], properties_component[1]['Vc']])

T_especific = 270.0
RHOLSat_esp = 21.4626
# valor initial of delta_1
delta_1 = 1.5

dinputs = np.array([properties_component[1]['Tc'], properties_component[1]['Pc'],
                    properties_component[1]['Omega'], delta_1, T_especific, RHOLSat_esp])

component_eos = pt.models_eos_cal(NMODEL, ICALC, dinputs)

print(component_eos)


I am working on the file "Parámetros de ecuaciones de estado (SRK, PR, RKPR)-Red Neuronal-Copy1.ipynb".
The first task is:
importar las linrerías requeridas, en este caso se trata de las librerías numpy, pandas junto con pyther
Can you write Python code for it?

properties_data = pt.Data_parse()

component = "3-METHYLHEPTANE"
component = "METHANE"
component = "ETHANE"
component = "PROPANE"
component = "n-HEXATRIACONTANE"

NMODEL = "RKPR"
NMODEL = "PR"
ICALC = "constants_eps"

properties_component = properties_data.selec_component(component)
pt.print_properties_component(component, properties_component)
dinputs = np.array([properties_component[1]['Tc'], properties_component[1]['Pc'],
                    properties_component[1]['Omega'], properties_component[1]['Vc']])

component_eos = pt.models_eos_cal(NMODEL, ICALC, dinputs)

#ac = component_eos[0]
print(component_eos)

components = ["ISOBUTANE", "CARBON DIOXIDE", 'METHANE', "ETHANE", "3-METHYLHEPTANE", "n-PENTACOSANE",
              "NAPHTHALENE", "m-ETHYLTOLUENE", "2-METHYL-1-HEXENE"]

NMODEL = "PR"
ICALC = "constants_eps"
component_eos_list = np.zeros((len(components),4))

for index, component in enumerate(components):

    properties_component = properties_data.selec_component(component)
    #pt.print_properties_component(component, properties_component)
    dinputs = np.array([properties_component[1]['Tc'], properties_component[1]['Pc'],
                        properties_component[1]['Omega'], properties_component[1]['Vc']])

    component_eos = pt.models_eos_cal(NMODEL, ICALC, dinputs)
    component_eos_list[index] = component_eos

#components_table_PR = pd.DataFrame(component_eos_list, index=components, columns=['ac', 'b', 'rm', 'del1'])
#print(components_table_PR)

components_table_PR = pd.DataFrame(component_eos_list, index=components, columns=['ac', 'b', 'rm', 'del1'])
print(components_table_PR)

components_table_PR

name = properties_data.read_dppr()[0]
name


I am working on the file "Parámetros de ecuaciones de estado (SRK, PR, RKPR).ipynb".
The first task is:
importar las linrerías requeridas, en este caso se trata de las librerías numpy, pandas junto con pyther
Can you write Python code for it?

properties_data = pt.Data_parse()

component = "3-METHYLHEPTANE"
component = "METHANE"
component = "ETHANE"
component = "PROPANE"
component = "n-HEXATRIACONTANE"

NMODEL = "RKPR"
NMODEL = "PR"
ICALC = "constants_eps"

properties_component = properties_data.selec_component(component)
pt.print_properties_component(component, properties_component)
dinputs = np.array([properties_component[1]['Tc'], properties_component[1]['Pc'],
                    properties_component[1]['Omega'], properties_component[1]['Vc']])

component_eos = pt.models_eos_cal(NMODEL, ICALC, dinputs)

#ac = component_eos[0]
print(component_eos)


I am working on the file "PatentsByLocation.ipynb".
The first task is:
Above, we used the getScore function to assign a score to a word based on its frequency in the region compared to its frequency in the overall data. A simple scoring function is just the ratio of the occurrance in the region to the occurrance in the overall data. Other scoring functions will, of course, identify different patterns in word usage.
Can you write Python code for it?

def getScore(singleFreq, overallFreq):
    return (singleFreq / overallFreq)


I am working on the file "PatentsByLocation.ipynb".
The first task is:
Additionally, we can look at patents by city. Granted, this may not be as accurate as analyzing by state for a few reasons. (1) There are likely fewer patents from a city than from a state. (2) The location is based on only one inventor when there are often multiple inventors listed per patent. (3) The location is based on the inventor's residence address and not necessarily the city where the work was done.
But analyzing by city can still be interesting. Let's look at some cities in the Bay Area.
Can you write Python code for it?

region_cities = ['San Jose', 'Santa Clara', 'Sunnyvale', 'Cupertino', 'Mountain View', 'Palo Alto',
                 'Los Altos', 'Menlo Park', 'Redwood City', 'San Mateo', 'South San Francisco',
                 'San Francisco', 'Berkeley', 'Oakland']
bestWordInRegion(region_cities, city=True)


I am working on the file "PatentsByLocation.ipynb".
The first task is:
Compare Regional Word Frequencies to Overall Word Frequencies
Now that we can generate a list of words and their occurrances in the patent abstracts, we need to be able to compare a list created from a region with the list created from the overall data.
Can you write Python code for it?

def getHighestScoredWordInRegion(regionalUsage, overallUsage):
    highestScore = 0
    wordWithHighestScore = None
    for wordAndFreq in regionalUsage:
        overallFreq = getOverallFreq(wordAndFreq[0], overallUsage)
        score = getScore(wordAndFreq[1], overallFreq)
        if score > highestScore:
            wordWithHighestScore = wordAndFreq[0]
            highestScore = score
    return wordWithHighestScore

def getOverallFreq(word, overallUsage):
    for wordAndFreq in overallUsage:
        if word == wordAndFreq[0]:
            return wordAndFreq[1]
    return 1


I am working on the file "PatentsByLocation.ipynb".
The first task is:
Let's take a look at the dataframe we've constructed.
Can you write Python code for it?


I am working on the file "PatentsByLocation.ipynb".
The first task is:
We can put this all together into a final function:
Can you write Python code for it?

def bestWordInRegion(region, city=False, state=False):
    overallUsage = getOverallWordFrequency(patent_df, 100000)
    output = []
    for location in region:
        if city:
            regionalUsage = getOverallWordFrequency(patent_df[patent_df['city'] == location], 500)
            output.append((location,
                           len(patent_df[patent_df['city'] == location].index),
                           getHighestScoredWordInRegion(regionalUsage, overallUsage)))
        if state:
            regionalUsage = getOverallWordFrequency(patent_df[patent_df['state'] == location], 200)
            output.append((location,
                           len(patent_df[patent_df['state'] == location].index),
                           getHighestScoredWordInRegion(regionalUsage, overallUsage)))
    output_df = pd.DataFrame.from_records(output, columns=['location', 'patents analyzed', 'frequent word'])
    print output_df
    return


I am working on the file "Path-demo-4.ipynb".
The first task is:
Calculate the revenue by checkout:
Can you write Python code for it?

%%sql
SELECT madlib.path(
     'eventlog',                -- Name of input table
     'path_output',             -- Table name to store path results
     'session_id',              -- Partition input table by session
     'event_timestamp ASC',     -- Order partitions in input table by time
     'buy:=page=''CHECKOUT''',  -- Define a symbol for checkout events
     '(buy)',                   -- Pattern search: purchase
     'sum(revenue) as checkout_rev',    -- Aggregate:  sum revenue by checkout
     TRUE                       -- Persist matches
     );
SELECT * FROM path_output ORDER BY session_id, match_id;


I am working on the file "Path-demo-4.ipynb".
The first task is:
For instances where a purchase is made within 4 pages of entering a site, compute the elasped time to checkout:
Can you write Python code for it?

%%sql
DROP TABLE IF EXISTS path_output, path_output_tuples;
SELECT madlib.path(
     'eventlog',                -- Name of input table
     'path_output',             -- Table name to store path results
     'session_id',              -- Partition input table by session
     'event_timestamp ASC',     -- Order partitions in input table by time
      $$ land:=page='LANDING',
        wine:=page='WINE',
        beer:=page='BEER',
        buy:=page='CHECKOUT',
        other:=page<>'LANDING' AND page<>'WINE' AND page<>'BEER' AND  page<>'CHECKOUT'
        $$,                     -- Symbols for page types
      '(land)[^(land)(buy)]{0,2}(buy)', -- Purchase within 4 pages entering site
     '(max(event_timestamp)-min(event_timestamp)) as elapsed_time',    -- Aggregate: elapsed time
     TRUE                       -- Persist matches
     );
SELECT * FROM path_output ORDER BY session_id, match_id;


I am working on the file "Path-demo-4.ipynb".
The first task is:
Note that there are 2 checkouts within session 102, which is apparent from the 'match_id' column. This serves to illustrate that the 'aggregate_func' operates on a per pattern match basis, not on a per partition basis. If in fact we wanted revenue by partition ('session_id' in this example), then we could do:
Can you write Python code for it?

%%sql
SELECT session_id, sum(checkout_rev) FROM path_output GROUP BY session_id ORDER BY session_id;


I am working on the file "Path-demo-4.ipynb".
The first task is:
Notice that the 'symbol' and 'match_id' columns are added to the right of the matched rows.
We are interested in sessions with an order placed within 4 pages of entering the shopping site via the landing page. We represent this by the regular expression: '(land)[^(land)(buy)]{0,2}(buy)'. In other words, visit to the landing page followed by from 0 to 2 non-entry, non-sale pages, followed by a purchase. The SQL is as follows:
Can you write Python code for it?

%%sql
DROP TABLE IF EXISTS path_output, path_output_tuples;
SELECT madlib.path(
     'eventlog',                -- Name of input table
     'path_output',             -- Table name to store path results
     'session_id',              -- Partition input table by session
     'event_timestamp ASC',     -- Order partitions in input table by time
      $$ land:=page='LANDING',
        wine:=page='WINE',
        beer:=page='BEER',
        buy:=page='CHECKOUT',
        other:=page<>'LANDING' AND page<>'WINE' AND page<>'BEER' AND  page<>'CHECKOUT'
        $$,                     -- Symbols for page types

      '(land)[^(land)(buy)]{0,2}(buy)', -- Purchase within 4 pages entering site
     'sum(revenue) as checkout_rev',    -- Aggregate:  sum revenue by checkout
     TRUE                       -- Persist matches
     );
SELECT * FROM path_output ORDER BY session_id, match_id;


I am working on the file "Path-demo-4.ipynb".
The first task is:
Since we set TRUE for 'persist_rows', we can view the associated pattern matches:
Can you write Python code for it?

%%sql
SELECT * FROM path_output_tuples ORDER BY session_id ASC, event_timestamp ASC;


I am working on the file "Path-demo-4.ipynb".
The first task is:
The data set describes shopper behavior on a notional web site that sells beer and wine. A beacon fires an event to a log file when the shopper visits different pages on the site: landing page, beer selection page, wine selection page, and checkout. Other pages on the site like help pages show up in the logs as well. Let’s assume that the log has been sessionized.
Can you write Python code for it?


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
2a.2. Broad ExAc
Now we retrieve frequency data from the Broad ExAc [http://exac.broadinstitute.org]:
Can you write Python code for it?

os.system('annotate_variation.pl -downdb -webfrom annovar -build hg19 exac03 humandb/')

infile_name = 'flat_clinvar_pathogenic_SNPs.tsv'
outfile_name = 'ExAc'
os.system('annotate_variation.pl -filter -build hg19 -dbtype exac03 -out %s %s humandb/' % (outfile_name,infile_name))

ExAc_data = pd.read_table('ExAc.hg19_exac03_dropped',
                     header=None)
ExAc_data = ExAc_data.drop(0,1)
ExAc_data.columns = ['ExAc_Overall_Frequency','Chromosome','Start','Stop','Ref','Alt',
                'Assertion','Disease','Accession']
print "There are %s variant-disease assertions with ExAc frequency data available." %ExAc_data.shape[0]
ExAc_data.head(5)


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
2b. Pulling Pubmed Publications
The following function, using the Python module BeautifulSoup, retrieves PubMed IDs directly from the ClinVar website:
Can you write Python code for it?

from bs4 import BeautifulSoup
import urllib2

def getPubMedIDs(RCV):
    try:
        response = urllib2.urlopen("http://www.ncbi.nlm.nih.gov/clinvar/" + RCV)
    except urllib2.HTTPError:
        print "Page not found"
    else:
        soup = BeautifulSoup(response.read())
        tag = soup.body.find("div", id="clinvar_rec_pubmed_ids1")
        if tag:
            return tag.text.split(", ")
    return []


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
4.2 Uncertainty in the disease risk conveyed by pathogenic variation
Here we perform a sensitivity analysis for penetrance using the disease hypertrophic cardiomyopathy (HCM). Making our assumptions explicit:

The prevalence of HCM is $1/500$ [Maron et al., Circulation 1995]

There are more than a thousand causal variants for HCM [Maron et al., J Am. Coll. Cardiol., 2012]. A conservative estimate for $h$, the heterogeneity parameter, is therefore $h=0.1$. A more realistic estimate is $h=0.001$

HCM is autosomal dominant: a single copy of a highly-penetrant causal allele is sufficient to cause disease

Can you write Python code for it?

# parameters above as Python variables
import numpy as np

prev = 1.0/500
het_range = np.linspace((0.001),(0.1),10)
pen = {} # to be computed below
genetic_model = 'AD'


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
Let's deduplicate:
Can you write Python code for it?

flat_clinvar = flat_clinvar.drop_duplicates()
print "There are %s distinct assertions in this version of Clinvar." %flat_clinvar.shape[0]


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
Let's retrieve all PubMed IDs for these accessions:
Can you write Python code for it?

HCM_PMIDs = {}
for acc in set(HCM.Accession):
    HCM_PMIDs[acc] = getPubMedIDs(acc)
HCM_PMIDs


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
Let's test this function on a few examples:
Can you write Python code for it?

print "Expecting 5 PubMedIDs for RCV000019428:", getPubMedIDs("RCV000019428")
print "Expecting 2 PubMedIDs for RCV000019429:", getPubMedIDs("RCV000019429")
print "Expecting 0 PubMedIDs for RCV000116253.2:", getPubMedIDs("RCV000116253.2")


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
Notice that some of variants contain multiple assertions, separated by the ',' or '|' characters. We can extract just these rows as follows:
Can you write Python code for it?

import re

accession = clinvar.accession.map(lambda x: re.split(r'[,|]+',x) if ('|' in x or ',' in x) else [x])
multi_rows = [i for i, e in enumerate(accession) if len(e) > 1]
multi = clinvar.ix[multi_rows,:]
multi.head(5)


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
Recall:
\begin{eqnarray}
\text{Penetrance} = P(D|G=j) = \frac{k \times P(G=j|D)}{P(G=j)} \
= \frac{k \times P(G=j|D)}{P(G=j|D)\times k + P(G=j|\overline{D})\times (1-k)}
\end{eqnarray}
Thus, grabbing all HCM variants and computing penetrance can be accomplished by:
Can you write Python code for it?

HCM = merged[merged.Disease.str.contains('hypertrophic_cardiomyopathy')]
HCM = HCM.drop_duplicates(subset = ['Chromosome','Start','Stop','Ref','Alt']) # 81 distinct variants
raw_names = zip(HCM.Chromosome,HCM.Start,HCM.Ref,HCM.Alt)
clean_names = [str(chrom)+':'+str(int(pos))+str(ref)+'>'+alt for (chrom,pos,ref,alt) in raw_names]
HCM[['Chromosome','Start','Ref','Alt','ExAc_Overall_Frequency','ESP_Overall_Frequency','Accession','Disease']].head(5)

import numpy as np

max_freqs = [np.nanmax(np.array([x,y])) for (x,y) in zip(HCM.ESP_Overall_Frequency,HCM.ExAc_Overall_Frequency)]
max_freqs = [x**2 + 2*x*(1-x) for x in max_freqs]

for i,var in enumerate(clean_names):
    these_pens = []
    for h in het_range:
        f = max_freqs[i]
        these_pens += [min(prev*h/f,1)]
    pen[var] = these_pens

# now export to R
import csv
outfile = open('sens_analysis','w')
for k,v in pen.items():
    outfile.write(k)
    for i in v:
        outfile.write(','+str(i))
    outfile.write('\n')
outfile.close()


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
Taking a closer look at raw_clinvar, we see that the clinical significance data is squished together in one field (ClinSig):
Can you write Python code for it?


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
We can flatten the clinvar data so there is one <b>assertion</b> per line:
Can you write Python code for it?

assertion = clinvar.assertion.map(lambda x: re.split(r'[,|]+',x) if ('|' in x or ',' in x) else [x])
disease = clinvar.disease.map(lambda x: re.split(r'[,|]+',x) if ('|' in x or ',' in x) else [x])
accession = clinvar.accession.map(lambda x: re.split(r'[,|]+',x) if ('|' in x or ',' in x) else [x])
v0 = clinvar.Chromosome.tolist()
v1 = clinvar.Start.tolist()
v2 = clinvar.Stop.tolist()
v3 = clinvar.Ref.tolist()
v4 = clinvar.Alt.tolist()
variants = zip(v0,v1,v2,v3,v4)
flat = []
for i, v in enumerate(variants):
    num_assertions = len(accession[i])
    v = list(v)
    if num_assertions > 1:
        flat += [v+[assertion[i][j],disease[i][j],accession[i][j]] for j in range(num_assertions)]
    else:
        flat += [v+assertion[i]+disease[i]+accession[i]]
flat_clinvar = pd.DataFrame(flat)
flat_clinvar.columns = ['Chromosome','Start','Stop','Ref','Alt','Assertion','Disease','Accession']
flat_clinvar.head(5)


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
We can load this file more cleanly with the pandas library (http://pandas.pydata.org/):
Can you write Python code for it?

import pandas as pd
raw_clinvar = pd.read_table('humandb/hg19_clinvar_20150629.txt', header=None)
raw_clinvar.columns = ['Chromosome','Start','Stop','Ref','Alt','ClinSig']
print 'There are %s rows in this version of ClinVar.' %raw_clinvar.shape[0]

distinct_variants = set(zip(raw_clinvar.Chromosome,raw_clinvar.Start,raw_clinvar.Stop,raw_clinvar.Ref,raw_clinvar.Alt))
num_distinct_variants = len(distinct_variants)
print 'There are %s distinct variants in this version of ClinVar.' %num_distinct_variants


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
We can use these data to filter clinical variants:
Can you write Python code for it?

infile_name = 'flat_clinvar_pathogenic_SNPs.tsv'
outfile_name = 'ESP'
os.system('annotate_variation.pl -filter -dbtype esp6500si_all -build hg19 -out %s %s humandb/' % (outfile_name,infile_name))

ESP_data = pd.read_table('ESP.hg19_esp6500si_all_dropped',
                     header=None)
ESP_data = ESP_data.drop(0,1)
ESP_data.columns = ['ESP_Overall_Frequency','Chromosome','Start','Stop','Ref','Alt',
                'Assertion','Disease','Accession']
print "There are %s variant-disease assertions with ESP frequency data available." %ESP_data.shape[0]
ESP_data.head(5)


I am working on the file "Pathogenicity_Notebook.ipynb".
The first task is:
and output to file:
Can you write Python code for it?

flat_clinvar.to_csv(path_or_buf='flat_clinvar.tsv',sep='\t',index=False,
                   columns=['Chromosome','Start','Stop','Ref','Alt','Assertion',
                               'Disease','Accession'])


I am working on the file "Patristic Distance Script Prototype.ipynb".
The first task is:
Aaaaand here, let's show what patristic distances we're capturing!
Can you write Python code for it?

plt.hist(graph_pds)
plt.xlabel('Patristic Distance')
plt.ylabel('Counts')
plt.title('Histogram of Patristic \n Distances Captured by the Network')
plt.show()


I am working on the file "Patristic Distance Script Prototype.ipynb".
The first task is:
Amongst the H3N8 isolates, here is the full range of patristic distances present in the PB2 gene tree.
Can you write Python code for it?

patristic_distances = dp.treecalc.PatristicDistanceMatrix(tree=tree).distances()
plt.hist(patristic_distances)
plt.xlabel('Patristic Distance')
plt.ylabel('Counts')
plt.title('Histogram of Patristic \n Distances in the PB2 Tree')

transmission_graph = nx.read_gpickle('Minto Flats.pkl')
transmission_graph.nodes(data=True)
for node in transmission_graph.nodes(data=True):
    if node[1]['subtype'] != 'H3N8':
        transmission_graph.remove_node(node[0])
        # transmission_graph.nodes(data=True)

len(transmission_graph.nodes())


I am working on the file "Patristic Distance Script Prototype.ipynb".
The first task is:
Because I don't have the taxa names identical between the tree and the transmission graph, therefore, I will use a
dictionary mapping to map the taxa name in the graph to the taxa name in the tree.
Can you write Python code for it?

taxadict = {str(t).split("'")[0].split("|")[0].replace("_", " "):t for t in tree.taxon_set}
# taxadict

graph_pds = []
for (taxa1, taxa2) in transmission_graph.edges():
    # Note here that some of the taxa names present in the network are absent from the trees.
    # As such, I will have to ask Kyle to re-run the H3N8 trees. Might take the chance to show
    # him some Python.
    if taxa1 in taxadict.keys() and taxa2 in taxadict.keys():
        t1 = taxadict[taxa1]
        t2 = taxadict[taxa2]
        pd = dp.treecalc.patristic_distance(tree, t1, t2)
        graph_pds.append(pd)


I am working on the file "Pauli with Geometric Algebra 4,0 over the Reals.ipynb".
The first task is:
ALGEBRA & DEFINITIONS
Clifford algebra is $$Cl_{4,0}(\mathbb{R})$$
Flat space, no metric, just signature
All constants are equal to 1
Can you write Python code for it?

from sympy import *
variables = (x, y, z, w) = symbols('x y z w', real=True)
print(variables)

metric=[ 1
        ,1
        ,1
        ,1]

myBasis='e_1 e_2 e_3 e_4'

sp4d = Ga(myBasis, g=metric, coords=variables,norm=True)
(e_1, e_2, e_3, e_4) = sp4d.mv()


I am working on the file "Pauli with Geometric Algebra 4,0 over the Reals.ipynb".
The first task is:
PHYSICS
The following symbols are defined :
Charge $$q \in \mathbb{R}$$
Mass $$m \in \mathbb{R}$$
Can you write Python code for it?

m, q = symbols('m q', real=True)
rquat = [sigma_1w, sigma_2w, sigma_3w]

B_x, B_y, B_z = symbols('B_x B_y B_z', real=True)
Bv =[B_x, B_y, B_z]
B = S(0)
for (dim, var) in zip(Bv, rquat):
    B += var * dim
B.texLabel='\\mathbf{B}'

display(Latex('Magnetic field $'+B.texLabel+'$ is defined with $B_x, B_y, B_z \\in \\mathbb{R}$'))
display(Math(B.texLabel+'=B_x'+sigma_1w.texLabel+'+B_y'+sigma_2w.texLabel+'+B_z'+sigma_3w.texLabel))
displayWithTitle(B, title=B.texLabel)

a, b, c, d = symbols('a b c d', real=True)

psiv =[b, c, d]
psi = S(a)
for (dim, var) in zip(psiv, rquat):
    psi += var * dim
psi.texLabel='\\psi'

display(Latex('Pauli spinor $'+psi.texLabel+'$ is defined with $a, b, c, d\\in \\mathbb{R}$'))
display(Math(psi.texLabel+'=a+b'+sigma_1w.texLabel+'+c'+sigma_2w.texLabel+'+d'+sigma_3w.texLabel))
displayWithTitle(psi, title=psi.texLabel)

psiv_dagger =[-b, -c, -d]
psi_dagger = S(a)
for (dim, var) in zip(psiv_dagger, rquat):
    psi_dagger += var * dim
psi_dagger.texLabel='\\psi^\\dagger'
display(Math(psi_dagger.texLabel+'=a-b'+sigma_1w.texLabel+'-c'+sigma_2w.texLabel+'-d'+sigma_3w.texLabel))
displayWithTitle(psi_dagger, title=psi_dagger.texLabel)

rho=psi*psi_dagger
rho.texLabel = '\\rho'
display(Math(rho.texLabel+'='+psi.texLabel+psi_dagger.texLabel))
displayWithTitle(rho, title=rho.texLabel)

spin_vector_w = psi*sigma_3w*psi_dagger
spin_vector_w.texLabel = '\\boldsymbol{s}_w'
display(Math(spin_vector_w.texLabel+'='+psi.texLabel+sigma_3w.texLabel+psi_dagger.texLabel))
displayWithTitle(spin_vector_w, title=spin_vector_w.texLabel)

up=1/2*(1+e_3)
up.texLabel='\\uparrow'
displayWithTitle(up, title=up.texLabel)
displayWithTitle(up*up, title=up.texLabel+'^2')

down=1/2*(1-e_3)
down.texLabel='\\downarrow'
displayWithTitle(down, title=down.texLabel)
displayWithTitle(down*down, title=down.texLabel+'^2')

gamma_5 = e_1*e_2*e_3*e_4
gamma_5.texLabel = '\\gamma_5'

left=1/2*(1-gamma_5)
left.texLabel='['
displayWithTitle(left, title=left.texLabel)
displayWithTitle(left*left,...


I am working on the file "Pauli with Geometric Algebra 4,0 over the Reals.ipynb".
The first task is:
Quaternions - Pauli matrices
http://en.wikipedia.org/wiki/Quaternion
https://en.wikipedia.org/wiki/Pauli_matrices
Can you write Python code for it?

sigma_1w=e_2*e_3
sigma_2w=e_3*e_1
sigma_3w=-e_1*e_2
sigma_1w.texLabel='\\sigma_{1w}'
sigma_2w.texLabel='\\sigma_{2w}'
sigma_3w.texLabel='\\sigma_{3w}'

display(Math('(1,'+sigma_1w.texLabel+','+sigma_2w.texLabel+','+sigma_3w.texLabel+')'))
CheckProperties(sigma_1w,sigma_2w,sigma_3w)

sigma_1x=e_2*e_3
sigma_2x=e_3*e_4
sigma_3x=-e_4*e_2
sigma_1x.texLabel='\\sigma_{1x}'
sigma_2x.texLabel='\\sigma_{2x}'
sigma_3x.texLabel='\\sigma_{3x}'

display(Math('(1,'+sigma_1x.texLabel+','+sigma_2x.texLabel+','+sigma_3x.texLabel+')'))
CheckProperties(sigma_1x,sigma_2x,sigma_3x)

sigma_1y=e_1*e_3
sigma_2y=e_3*e_4
sigma_3y=-e_4*e_1
sigma_1y.texLabel='\\sigma_{1y}'
sigma_2y.texLabel='\\sigma_{2y}'
sigma_3y.texLabel='\\sigma_{3y}'

display(Math('(1,'+sigma_1y.texLabel+','+sigma_2y.texLabel+','+sigma_3y.texLabel+')'))
CheckProperties(sigma_1y,sigma_2y,sigma_3y)

sigma_1z=e_1*e_2
sigma_2z=e_2*e_4
sigma_3z=-e_4*e_1
sigma_1z.texLabel='\\sigma_{1z}'
sigma_2z.texLabel='\\sigma_{2z}'
sigma_3z.texLabel='\\sigma_{3z}'

display(Math('(1,'+sigma_1z.texLabel+','+sigma_2z.texLabel+','+sigma_3z.texLabel+')'))
CheckProperties(sigma_1z,sigma_2z,sigma_3z)

gamma_4=e_1*e_2*e_3
gamma_1=e_4*e_2*e_3
gamma_2=e_1*e_4*e_3
gamma_3=e_1*e_2*e_4
displayWithTitle(gamma_4,title='\\gamma_4')
displayWithTitle(gamma_4*gamma_4,title='\\gamma_4^2')

displayWithTitle(gamma_1,title='\\gamma_1')
displayWithTitle(gamma_1*gamma_1,title='\\gamma_1^2')

displayWithTitle(gamma_2,title='\\gamma_2')
displayWithTitle(gamma_2*gamma_2,title='\\gamma_2^2')

displayWithTitle(gamma_3,title='\\gamma_3')
displayWithTitle(gamma_3*gamma_3,title='\\gamma_3^2')


I am working on the file "PearsonCorrelation.ipynb".
The first task is:
Does the Pearson test confirms that conclusion?
Can you write Python code for it?

correlation, pvalue = stats.pearsonr(data_clean['unemployrate'], data_clean['suicideper100th'])

display(Markdown("The correlation coefficient is {:.3g} and the associated p-value is {:.3g}.".format(correlation, pvalue)))


I am working on the file "PearsonCorrelation.ipynb".
The first task is:
The correlation coefficient is negative but really small and the p-value is large. So we can safetly conclude that there is no relationship between the unemployment rate and the suicide per 100,000 people.
Another test case
In order to look at the coefficient of determination, an another relationship that is significant will be analyzed below: Is the residential electricity consumption (response variable) related to the income per person (explanatory variable)?
Can you write Python code for it?

subdata2 = (data[['incomeperperson', 'relectricperperson']]
                .assign(income=lambda x: pd.to_numeric(data['incomeperperson'], errors='coerce'),
                        electricity=lambda x: pd.to_numeric(data['relectricperperson'], errors='coerce'))
                .dropna())

sns.regplot(x='income', y='electricity', data=subdata2)
plt.xlabel('Income per person (2000 US$)')
plt.ylabel('Residential electricity consumption (kWh)')
_ = plt.title('Scatterplot for the association between the income and the residential electricity consumption')

correlation, pvalue = stats.pearsonr(subdata2['income'], subdata2['electricity'])

display(Markdown("The correlation coefficient is {:.3g} and the associated p-value is {:.3g}.".format(correlation, pvalue)))
display(Markdown("And the coefficient of determination is {:.3g}.".format(correlation**2)))


I am working on the file "PearsonCorrelation.ipynb".
The first task is:
The first records of the data restricted to the three analyzed variables are:
Can you write Python code for it?

subdata = data[['internetuserate', 'suicideper100th', 'unemployrate']]
subdata.tail(10)


I am working on the file "PeelMetadata.ipynb".
The first task is:
This code provides a path for the location of the XML files. When running this code on your own machine, the only thing you will need to update is this information. The path to the file directory will depend entirely on where you saved the files.
Can you write Python code for it?


I am working on the file "PeelMetadata.ipynb".
The first task is:
We want to make sure that we have identified and parsed all of the files. Here we display the number of filenames saved to the variable dirs, and double check with the number of files saved on our machine.
Here's a screenshot of the file directory on my machine. If you look on the bottom of the image you can see the file count matches the number below the image in the dirs variable. This type of error checking is essential.
<img src="filecount.png" alt="file count" style="width: 400px;"/>
Can you write Python code for it?

print("There are", len(dirs), "XML files in the directory")


I am working on the file "Pendulum-Wave.ipynb".
The first task is:
<img src="animTopDown.gif">
Visualizing the Device from Head-On, Using Python
Below, a visualization of the device can be seen (from head-on). This simulates the position the center of mass of each golf ball, as if the pendulum was started with the balls all 10º from their equilibrium position.
Can you write Python code for it?

# Set up the figure, the axis, and the plot we want to animate
fig, ax = plt.subplots()
ax.set_xlim([-0.15, 0.15])
ax.set_ylim([-0.4, 0])
line, = ax.plot([], [], linestyle='', marker='o')
plt.xlabel("Horizonal displacement from equilibrium, meters")
plt.ylabel("Vertical displacement from equilibrium, meters")
fig.suptitle("Top View at t = 0 seconds", fontsize=12)
ax.spines['left'].set_position('zero')
ax.spines['right'].set_color('none')
ax.spines['bottom'].set_position('zero')
ax.spines['top'].set_color('none')
ax.xaxis.labelpad = 220
ax.yaxis.labelpad = 150

def init2():
    line.set_data([], [])
    return (line,)

def animate2(t):
    xPos = []
    yPos = []
    for n in range(12):
        xPos.append(funcDispX(lengthsM[n], funcTheta(math.radians(10), n, t)))
        yPos.append(funcDispY(lengthsM[n], funcTheta(math.radians(10), n, t)))
    fig.suptitle("Head-On View at t = {:.3f} seconds".format(round(t, 3)), fontsize=12)
    line.set_data(np.array(xPos), np.array(yPos))
    return (line,)

fps = 60
anim = animation.FuncAnimation(fig, animate2, init_func=init2, frames=np.linspace(0, 60, 60*fps//2), blit=True)
anim.save('animHeadOn.mp4', writer='ffmpeg', fps=str(fps), extra_args=['-vcodec', 'libx264'])
plt.close()


I am working on the file "Pendulum-Wave.ipynb".
The first task is:
From this point, the pendulum was built. Note that the lengths listed above are from the pivot point of the pendulums to the centers of mass of the hanging golf balls, so the actual strings will be shorter than the numbers given above.
The pendulums were spaced so that there was 0.5in between the golf balls, meaning that strings were hung about 2.2in, or 5.6cm, apart.
Viewing the Device from Above, Using Python
Below, a visualization of the device can be seen (from above). This simulates the position the center of mass of each golf ball, as if the pendulum was started with the balls all 10º from their equilibrium position. A dashed line has been drawn in to show the approximate pattern of the pendulums.
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
from matplotlib import animation
%matplotlib inline

location = np.linspace(0, 0.056*11, 12)

funcTheta = lambda theta_i, n, t: theta_i * math.cos((2 * pi * t) / (60 / (51 + n)))
funcDispX = lambda l, theta: l * math.sin(theta)
funcDispY = lambda l, theta: -1 * l * math.cos(theta)

# Set up the figure, the axis, and the plot we want to animate
fig, ax = plt.subplots()
ax.set_xlim([0, location[11]])
ax.set_ylim([-0.06, 0.06])
line, = ax.plot([], [], linestyle='--', marker='o')
plt.xlabel("Position along the machine, meters")
plt.ylabel("Horizonal displacement from equilibrium, meters")
fig.suptitle("Top View at t = 0 seconds", fontsize=12)
ax.spines['left'].set_position('zero')
ax.spines['right'].set_color('none')
ax.spines['bottom'].set_position('zero')
ax.spines['top'].set_color('none')
ax.xaxis.labelpad = 100

def init1():
    line.set_data([], [])
    return (line,)

def animate1(t):
    displacementArr = []
    for n in range(12):
        displacementArr.append(funcDispX(lengthsM[n], funcTheta(math.radians(10), n, t)))
    fig.suptitle("Top View at t = {:.3f} seconds".format(round(t, 3)), fontsize=12)
    line.set_data(location, np.array(displacementArr))
    return (line,)

fps = 60
anim = animation.FuncAnimation(fig, animate1, init_func=init1, frames=np.linspace(0, 60, 60*fps//2), blit=True)
anim.save('animTopDown.mp4', writer='ffmpeg', fps=str(fps), extra_args=['-vcodec', 'libx264'])
plt.close()


I am working on the file "People Watching.ipynb".
The first task is:
Import the data. You could do this from your own postgres instance if you ran your own wiki import, but if not, you can also just download the (zipped) csv from my website:
Can you write Python code for it?

df = pd.read_csv('https://douwe.com/static/projects/wiki_import/wikipeople.zip', compression='zip')
df['life_span'] = df['year_died'] - df['year_born']
df


I am working on the file "People Watching.ipynb".
The first task is:
Let's explore this bias by plotting the life expectancy by year born. As you can see, life expectancy increases over time (as expected) until roughly 1924. At that point it starts to drop sharply because more and more people born in those years are still alive making the set of those who are not a more and more biased set
Can you write Python code for it?

df.groupby('year_born').mean().reset_index().plot.line(x='year_born', y='life_span',
                    figsize=(12,8))


I am working on the file "People Watching.ipynb".
The first task is:
Let's have a look at how the year born and the year died correlate by plotting them against each other.
Can you write Python code for it?

fig, ax = plt.subplots()
df.plot.scatter(x='year_born',
                y='year_died',
                figsize=(12,12),
                ax=ax)


I am working on the file "People Watching.ipynb".
The first task is:
Looks reasonable, other than that there are some unlikely outliers of people either living to an unrealistically mature age or worse, dying before they were born. When processing lots of data from Wikipedia you have to expect that you end up with a small number of parsing errors.
By occupation
Now we'll group the data by occupation to get a rough idea of which occupations have the best life expectancy.
Can you write Python code for it?

top_jobs = [*df.groupby('occupation').count().sort_values('life_span').reset_index()[['occupation', 'life_span']].tail(60)['occupation']]
life_expentancy = df.dropna().groupby('occupation')['life_span'].mean().sort_values()
life_expentancy


I am working on the file "People Watching.ipynb".
The first task is:
Too get a cleaner dataset, let's only look at the top 60 occupations by number of people so we root out the noise and let's look only at people born before 1924. This will remove rappers and billionaires from our dataset since, but will make things a lot clearer.
Can you write Python code for it?

top_jobs = [*df.groupby('occupation').count().sort_values('life_span').reset_index()[['occupation', 'life_span']].tail(100)['occupation']]

life_expentancy = df[df['year_born'] < 1924].dropna().groupby('occupation')['life_span'].mean()
life_expentancy[top_jobs].reset_index().sort_values('life_span')


I am working on the file "People Watching.ipynb".
The first task is:
Turns out that aviators born before 1924 have a low life expectancy. Given the state of airplanes in those days, that is no real surprise of course. Nobility and soldiers also don't do very well, which again seems intuitive. Actresses live 4 years longer than actors, which roughly corresponds to the difference in life expectancy between males and females. Finally to live the longest, stay indoors and do something academic.
Feel free to play around with this. You can easily sample specific occupations to compare
Can you write Python code for it?

life_expentancy[['singers', 'poets', 'painters', 'philosophers']]


I am working on the file "People most likely to go missing in Mission District between 10am & 11am on Fridays.ipynb".
The first task is:
Changing the column labels of the data set
Can you write Python code for it?

incidents.columns = ['Id'
                    ,'Category'
                    ,'Description'
                    ,'DayOfWeek'
                    ,'Date'
                    ,'Time'
                    ,'District'
                    ,'Resolution'
                    ,'Address'
                    ,'Longitude'
                    ,'Latitude'
                    ,'Location'
                    ,'PdId']


I am working on the file "People most likely to go missing in Mission District between 10am & 11am on Fridays.ipynb".
The first task is:
Histogram plot of the "Missing Person" incidents across districts
The plot clearly shows that the Missing Person incidents are almost double of the same incidents in Southern district, which has the next highest "Missing Person" rate.
Can you write Python code for it?

count_by_category = pd.DataFrame({'count' : filtered.groupby( ['District'] ).size()}).reset_index()
count_by_category.sort_values(by='count',ascending= True,inplace=True)
index = count_by_category['District']

graph = vincent.Bar(count_by_category,columns=['count'], key_on='District')
graph.legend(title='District')
graph.axis_titles(x='District', y='Incident Count')
ax = AxisProperties(
    labels=PropertySet(
        angle=ValueRef(value=270),
        align=ValueRef(value='right')
        )
    )
graph.axes[0].properties = ax
graph.display()


I am working on the file "People most likely to go missing in Mission District between 10am & 11am on Fridays.ipynb".
The first task is:
Histogram plot of the incidents by category
The plot shows that Larceny/Theft is the most reported crime across all districts, second being Assault and 9th being Missing Person.
Can you write Python code for it?

count_by_category = pd.DataFrame({'count' : incidents.groupby( ['Category'] ).size()}).reset_index()
count_by_category.sort_values(by='count',ascending= True,inplace=True)
index = count_by_category['Category']

graph = vincent.Bar(count_by_category,columns=['count'], key_on='Category')
graph.legend(title='Category')
graph.axis_titles(x='Category', y='Incident Count')
ax = AxisProperties(
    labels=PropertySet(
        angle=ValueRef(value=270),
        align=ValueRef(value='right')
        )
    )
graph.axes[0].properties = ax
graph.display()
#ax = count_by_category.plot(kind="barh",x='Category', y='count',sort_columns=True)

by_year = incidents.pivot_table('Id'
                                , aggfunc='count'
                                , index='Weekday'
                                , columns='District')

graph = vincent.Line(by_year)
graph.legend(title='District')
graph.axis_titles(x='Weekday', y='Incident Count')

graph.display()


I am working on the file "People most likely to go missing in Mission District between 10am & 11am on Fridays.ipynb".
The first task is:
Missing Person incidents peak at 11 am in Mission district
The plot shows another surprising peak in the Missing Person incidents at 11 am.
Can you write Python code for it?

filter_by_district = 'MISSION'
filter_by_category = 'MISSING PERSON'
filtered = incidents[incidents['Category'] == filter_by_category]
filtered = filtered[filtered['District'] == filter_by_district]

by_hour = filtered.pivot_table('Id'
                              , aggfunc='count'
                              , index='Hour'
                              , columns='Category')

graph = vincent.Line(by_hour)
graph.legend(title='Category')
graph.axis_titles(x='Hour', y='Incident Count')

graph.display()


I am working on the file "People most likely to go missing in Mission District between 10am & 11am on Fridays.ipynb".
The first task is:
Most missing person cases seem to be around 10 to 12 am with the peak being at 11am
Can you write Python code for it?

by_hour = filtered.pivot_table('Id'
                                , aggfunc='count'
                                , index='Hour'
                                , columns='District')

graph = vincent.Bar(by_hour)
graph.legend(title='District')
graph.axis_titles(x='Hour', y='Incident Count')

graph.display()


I am working on the file "People most likely to go missing in Mission District between 10am & 11am on Fridays.ipynb".
The first task is:
The date and time of incident are in two separate columns. Combining them into a DateTime column
Can you write Python code for it?

# the date and time of incident are in two separate columns
# combining them into a date_time column
incidents['DateTime'] = pd.to_datetime(incidents['Date'] + ' ' + incidents['Time'])
date_idx = pd.DatetimeIndex(incidents['DateTime'])
incidents['Date'] = date_idx.date.astype('datetime64')
incidents['Hour'] = date_idx.hour
incidents['Year'] = date_idx.year
incidents['Month'] = date_idx.month
incidents['Weekday'] = date_idx.weekday


I am working on the file "People most likely to go missing in Mission District between 10am & 11am on Fridays.ipynb".
The first task is:
The graph result is surprising. "Missing Person" incidents are fourth highest compared to overall SFO
Comparing this trend across other districts
The comparison shows that the trend is clearly something that is unique to Mission District and the number of missing person incidents is rather high compared to the other districts
Can you write Python code for it?

filter_by_category = 'MISSING PERSON'

filtered = incidents[incidents['Category'] == filter_by_category]
by_hour = filtered.pivot_table('Id'
                                , aggfunc='count'
                                , index='Hour'
                                , columns='District')

graph = vincent.Line(by_hour) #,columns=['count'],key_on='District')
graph.legend(title='District')
graph.axis_titles(x='Hour', y='Incident Count')

graph.display()


I am working on the file "People most likely to go missing in Mission District between 10am & 11am on Fridays.ipynb".
The first task is:
Top incident category in Mission district
Limiting the incidents to Mission District and excluding the incident categories 'LARCENY/THEFT','NON-CRIMINAL','OTHER OFFENSES','WARRANTS'.
This shows that the Missing Person is the third highest incident category in Mission district.
Can you write Python code for it?

filtered = incidents[incidents['District'] == 'MISSION']
filtered = filtered[~filtered['Category'].isin(['LARCENY/THEFT'
                                               ,'NON-CRIMINAL'
                                               ,'OTHER OFFENSES'
                                               ,'WARRANTS'])]

count_by_category = pd.DataFrame({'count' : filtered.groupby( ['Category'] ).size()}).reset_index()
count_by_category.sort_values(by='count',ascending= True,inplace=True)
index = count_by_category['Category']

graph = vincent.Bar(count_by_category,columns=['count'], key_on='Category')
graph.legend(title='Category')
graph.axis_titles(x='Category', y='Incident Count')
ax = AxisProperties(
    labels=PropertySet(
        angle=ValueRef(value=270),
        align=ValueRef(value='right')
        )
    )
graph.axes[0].properties = ax
graph.display()


I am working on the file "Perceptron Binary Classification Learning Algorithm Tutorial.ipynb".
The first task is:
PLA 在測試資料的預測錯誤率也是完美的 0，當然這某種程度是因為我們的 Demo 資料有設計過，不過理論上測試資料的預測錯誤率應該不會和訓練資料的預測錯誤率差太多，只要實驗過程是一個客觀的過程、沒有經過人為的污染，機器學習的演算法的確可以做到正確的預測。
以上，你大概已經學會使用 FukuML 提供的 PLA 做訓練，然後使用訓練完成的 w 來進行未知資料的預測了，真的五個步驟就可以做完了！很簡單吧！
使用自己的訓練資料集和測試資料集
前面的教學我們是使用 FukuML 所提供的訓練資料集和測試資料集，真實情況你當然使用自己的資料，那要怎麼做呢？FukuML 提供了很簡易的方法可以讓大家載入自己的資料：
```
your_training_data_file = '/path/to/your/training_data/file'
pla_bc.load_train_data(your_training_data_file)
your_testing_data_file = '/path/to/your/testing_data/file'
pla_bc.load_test_data(your_testing_data_file)
```
就是這麼簡單，讓我們來實際演示一下：
Can you write Python code for it?


I am working on the file "Perceptron Binary Classification Learning Algorithm Tutorial.ipynb".
The first task is:
PLA 在訓練資料的預測錯誤率是完美的 0！這是當然的，因為 PLA 在線性可分的資料裡，一定會調整到沒有錯誤為止。
現在我們來看看 PLA 在整個測試資料集的預測錯誤率（$E_{out}$），在此之前，我們必須先載入測試資料集，一樣 FukuML 有提供 Demo 版本的測試資料集:
Can you write Python code for it?


I am working on the file "Perceptron Binary Classification Learning Algorithm Tutorial.ipynb".
The first task is:
PLA 會忠實的觀察資料給出答案，它認為這筆資料的答案也是 1。（事實上真的是）
當然，如果只是看一、兩筆資料猜對，大家可能會認為這只是運氣好，所以我們必須計算 PLA 在整個訓練資料集及整個測試資料集的預測表現如何。我們提供了很簡易的方法可以計算整體的錯誤率，如果要看 PLA 在整個訓練資料集的預測錯誤率（$E_{in}$）：
Can you write Python code for it?


I am working on the file "Perceptron Binary Classification Learning Algorithm Tutorial.ipynb".
The first task is:
prediction 這個方法會把預測結果回傳成一個 dictionary，預測結果的 key 是 prediction，value 是 1，測試資料的答案也是 1，所以 PLA 正確預測了結果！
假設我們現在要預測的是未知的資料、一些我們還沒有分好類的資料，那我們就是把資料特徵值向量丟進去 prediction 方法，並設定 ｀mode='future_data'，代表是做未知資料的預測，就可以進行預測了，比如丟進去 0.29634 0.4012 0.40266 0.67864 這筆特徵資料試試看：
Can you write Python code for it?

future_data = '0.29634 0.4012 0.40266 0.67864'
prediction = pla_bc.prediction(future_data, mode='future_data')


I am working on the file "Perceptron Binary Classification Learning Algorithm Tutorial.ipynb".
The first task is:
將預測結果印出來看看：
Can you write Python code for it?


I am working on the file "Perceptron Binary Classification Learning Algorithm Tutorial.ipynb".
The first task is:
我希望 FukuML 能儘量簡單易用，因此大家只要牢記 1. 載入訓練資料 -> 2. 設定參數 -> 3. 初始化 -> 4. 訓練 -> 5. 預測 這五個步驟就可以完成機器學習了～
現在第一個步驟要先載入訓練資料，但如果現在要讓大家生出一筆訓練資料應該會有困難，所以 FukuML 每個機器學習演算法都會有一個 Demo 用的內建資料，讓我們先用 Demo 用的內建資料來試試看。
Can you write Python code for it?


I am working on the file "Perceptron Binary Classification Learning Algorithm Tutorial.ipynb".
The first task is:
果然不是個 0 向量了呢！
有了這個 w，我們就可以用它來預測未來的資料，讓我拿一筆測試資料 0.97959 0.40402 0.96303 0.28133 1 來預測看看，前面 4 個值是這筆測試資料的特徵值，後面的 1 代表這筆測試資料的答案，我們來看看預測結果：
Can you write Python code for it?

test_data = '0.97959 0.40402 0.96303 0.28133 1'
prediction = pla_bc.prediction(test_data)


I am working on the file "Perceptron Binary Classification Learning Algorithm Tutorial.ipynb".
The first task is:
然後建構一個 PLA 二元分類物件：
Can you write Python code for it?


I am working on the file "Perceptron Binary Classification Learning Algorithm Tutorial.ipynb".
The first task is:
然後訓練資料的答案就存在 train_Y 中，也就代表每筆訓練資料的答案是什麼，正分類就是 1，負分類就是 -1。
接下來讓我們進行下一步，設定參數：
Can you write Python code for it?


I am working on the file "Perceptron Binary Classification Learning Algorithm Tutorial.ipynb".
The first task is:
訓練資料的特徵資料就存在 train_X 中，矩陣的每一個列就代表一筆資料，然後每一個行就代表一個特徵值，請注意矩陣的第一行都是 1，這是我們演算法自己補上的 $x_0$，並不是原本訓練資料就會有的特徵值，以這個 Demo 資料來說，每筆資料只有 4 個特徵值（feature），像第一筆資料的 4 個特徵值就是 0.97681 0.10723 0.64385 0.29556，演算法將前面補上 $x_0 = 1$，就變成了現在看到的樣子。
Can you write Python code for it?


I am working on the file "Perceptron Binary Classification Learning Algorithm Tutorial.ipynb".
The first task is:
載入測試資料之後，我們就可以計算 PLA 在測試資料集的預測錯誤率（$E_{out}$）：
Can you write Python code for it?


I am working on the file "Perceptron Binary Classification Learning Algorithm Tutorial.ipynb".
The first task is:
這樣就載入了 PLA 的 Demo 訓練資料，不信的話大家可以使用 pla_bc.train_X 及 pla_bc.train_Y 印出來看看：
Can you write Python code for it?


I am working on the file "Perceptron.ipynb".
The first task is:
Introdução
O tipo mais básico de Rede Neural Artificial é formada por apenas um neurônio, o Perceptron. Inicialmente, o Perceptron foi projetado para ser um classificador binário linear responsável por mapear uma ou mais entradas em uma saída desejada. Porém, também podemos utilizá-lo para resolver problemas de regressão linear. Ele foi projetado em 1957 por Frank Rosenblatt.
O perceptron é formado por:
<img src='images/perceptron.png' width='350'>

entradas $x_1,...,x_D$: representam os atributos dos seus dados com dimensionalidade $D$. O Perceptron aceita qualquer tamanho de entrada, porém a saída é sempre apenas um valor.
junção aditiva $\sum$: também chamada de função agregadora, nada mais é que a soma ponderada das entradas com os pesos ($w_1,...,w_D)$. Em geral, o resultado é somado com um bias $b$, responsável por deslocar o resultado do somatório. A junção aditiva é descrita pela seguinte fórmula:

$$\sum_i^D{x_iw_i} + b$$

função de ativação $f$: utilizada para mapear o resultado da junção aditiva em uma saída esperada. Mais detalhes abaixo.

Logo, o Perceptron é representado pela seguinte fórmula matemática:
$$\widehat{y}_i = f(\sum_i^D{x_iw_i} + b)$$
Onde:

$D$: representa a dimensionalidade das amostras, ou seja, a quantidade de atributos de cada amostra.
$x_i$: representam os atributos de uma amostra que servem de entrada para o Perceptron.
$w_i$: representam os pesos sinápticos que ponderam as entradas.
$b$: representa o bias, responsável por deslocar a fronteira de decisão além da origem e não depende de nenhum valor de entrada. Repare que o bias encontra-se fora do somatório.
$f$: função de ativação. Quando a função de ativação é linear, ou seja, nenhuma transformação é aplicada no resultado da junção aditiva, o Perceptron atua como um Regressor Linear. Se precisamos efetuar uma Classificação binária, devemos utilizar a função step (também conhecida como função degrau) para mapear a saída em um valor discreto (0 ou 1):

$$f = \begin{cases}1 & se \ wx+b > 0\0 & caso \ contr\acute...
Can you write Python code for it?


I am working on the file "Perceptron.ipynb".
The first task is:
Numpy
Can you write Python code for it?

D = x.shape[1]
w = 2*np.random.random(size=D)-1
b = 2*np.random.random()-1

for step in range(10001):
    cost = 0
    for x_n, y_n in zip(x, y):
        # qual linha devemos remover para transformar o Perceptron num regressor?
        y_pred = np.dot(x_n, w) + b
        y_pred = np.where(y_pred > 0, 1, 0)
        error = y_n - y_pred
        w = w + 1.0*np.dot(error, x_n)
        b = b + 1.0*error
        cost += error**2

    if step%1000 == 0:
        print('step {0}: {1}'.format(step, cost))

print('w: ', w)
print('b: ', b)


I am working on the file "Perceptron.ipynb".
The first task is:
Numpy com Pré-processamento
Can you write Python code for it?

minmax = MinMaxScaler(feature_range=(-1,1))
x = minmax.fit_transform(x.astype(np.float64))

print(x.min(), x.max())

reg = LinearRegression()
reg.fit(x,y)

print('w: ', reg.coef_)
print('b: ', reg.intercept_)

D = x.shape[1]
w = 2*np.random.random(size=D)-1
b = 2*np.random.random()-1

learning_rate = 1.0 # <- tente estimar a learning_rate

for step in range(1001):
    cost = 0
    for x_n, y_n in zip(x, y):
        y_pred = np.dot(x_n, w) + b
        error = y_n - y_pred
        w = w + learning_rate*np.dot(error, x_n)
        b = b + learning_rate*error
        cost += error**2

    if step%100 == 0:
        print('step {0}: {1}'.format(step, cost))

print('w: ', w)
print('b: ', b)


I am working on the file "Perceptron.ipynb".
The first task is:
Python
Can you write Python code for it?

D = x.shape[1]
w = [2*random() - 1 for i in range(D)]
b = 2*random() - 1

learning_rate = 1.0 # <- tente estimar a learning_rate

for step in range(1): # <- tente estimar o número de passos
    cost = 0
    for x_n, y_n in zip(x, y):
        y_pred = sum([x_i*w_i for x_i, w_i in zip(x_n, w)]) + b
        error = y_n - y_pred
        w = [w_i + learning_rate*error*x_i for x_i, w_i in zip(x_n, w)]
        b = b + learning_rate*error
        cost += error**2

    if step%200 == 0:
        print('step {0}: {1}'.format(step, cost))

print('w: ', w)
print('b: ', b)


I am working on the file "Perceptron.ipynb".
The first task is:
Python
Exercício: tentar estimar as learning_rates de w e b. Elas são diferentes por que nossos dados não estão na mesma escala!
Can you write Python code for it?

D = x.shape[1]
w = [2*random() - 1 for i in range(D)]
b = 2*random() - 1

for step in range(10001):
    cost = 0
    for x_n, y_n in zip(x, y):
        # qual linha devemos remover para transformar o Perceptron num regressor?
        y_pred = sum([x_i*w_i for x_i, w_i in zip(x_n, w)]) + b
        y_pred = 1 if y_pred > 0 else 0
        error = y_n - y_pred
        w = [w_i + 1.0*error*x_i for x_i, w_i in zip(x_n, w)]
        b = b + 1.0*error
        cost += error**2

    if step%1000 == 0:
        print('step {0}: {1}'.format(step, cost))

print('w: ', w)
print('b: ', b)


I am working on the file "Perceptron.ipynb".
The first task is:
Python
Numpy
Exercício de Classificação
Can you write Python code for it?

x, y = make_blobs(n_samples=100, n_features=2, centers=2, random_state=1234)

print(x.shape, y.shape)
plt.scatter(x[:,0], x[:,1], c=y.ravel(), cmap='bwr')

def plot_linear_classifier(x, y, w, b):
    x1_min, x1_max = x[:,0].min(), x[:,0].max()
    x2_min, x2_max = x[:,1].min(), x[:,1].max()

    x1, x2 = np.meshgrid(np.linspace(x1_min-1, x1_max+1,100), np.linspace(x2_min-1, x2_max+1, 100))
    x_mesh = np.array([x1.ravel(), x2.ravel()]).T

    plt.scatter(x[:,0], x[:,1], c=y.ravel(), cmap='bwr')

    y_mesh = np.dot(x_mesh, np.array(w).reshape(1, -1).T) + b
    y_mesh = np.where(y_mesh <= 0, 0, 1)

    plt.contourf(x1, x2, y_mesh.reshape(x1.shape), cmap='bwr', alpha=0.5)
    plt.xlim(x1_min-1, x1_max+1)
    plt.ylim(x2_min-1, x2_max+1)