I am working on the file "Inference_Big_data/Hopfield/Hopfield.ipynb".
The first task is:
Here we have use a list of structured patterns, the letters. Each letter is represented in a 10 by 10 grid. In this example we try to recover the letter $A$ from a noisy pattern. We can observe that $A$ is a stable attractor of the dynamic. Like in the case of uncorellated pattern the system can store the information. However is the network able to store as many information as in the case of uncorellated patterns ?
Can you write Python code for it?

letter_list = ['A', 'B', 'C', 'R','S', 'X', 'Y', 'Z']
hopfield_net = network.HopfieldNetwork(nr_neurons= pattern_shape[0]*pattern_shape[1])
pattern_list = [abc_dictionary[key] for key in letter_list ]
plot_tools.plot_pattern_list(pattern_list)

# store the patterns
hopfield_net.store_patterns(pattern_list)

# # create a noisy version of a pattern and use that to initialize the network
noisy_init_state = pattern_tools.get_noisy_copy(abc_dictionary['A'], noise_level=0.2)
hopfield_net.set_state_from_pattern(noisy_init_state)
overlap_matrix = pattern_tools.compute_overlap_matrix(pattern_list)
plot_tools.plot_overlap_matrix(overlap_matrix)
# from this initial state, let the network dynamics evolve.
states = hopfield_net.run_with_monitoring(nr_steps=5)
energy_list = []
for p in range(len(states)) :

    energy_list.append(- np.sum(np.dot(np.dot(hopfield_net.weights,states[p]),states[p])))
# each network state is a vector. reshape it to the same shape used to create the patterns.
states_as_patterns = pattern_tools.reshape_patterns(states, pattern_list[0].shape)

# plot the states of the network
plot_tools.plot_state_sequence_and_overlap(
    states_as_patterns, pattern_list, reference_idx=0, suptitle="Network dynamics")

plt.figure()
plt.plot(energy_list,'o')
plt.title('Energy of the network')


I am working on the file "Inference_Big_data/Hopfield/Hopfield.ipynb".
The first task is:
On this figure we saw that starting with a noisy pattern, the network is capable of finding the right pattern corresponding. We can observe that the correlation $m$ between the right pattern and the output is increasing with time.
But the thing we can ask is whether the network can retrieve a infinite number of patterns, with a strong noise ?
The following simulations will give a first answer to this question.
Can you write Python code for it?

pattern_size = 4
# create an instance of the class HopfieldNetwork
hopfield_net = network.HopfieldNetwork(nr_neurons= pattern_size**2)
# instantiate a pattern factory
factory = pattern_tools.PatternFactory(pattern_size, pattern_size)
# create a checkerboard pattern and add it to the pattern list
checkerboard = factory.create_checkerboard()
pattern_list = [checkerboard]
pattern_list.extend(factory.create_random_pattern_list(nr_patterns=6, on_probability=0.5))
plot_tools.plot_pattern_list(pattern_list)
hopfield_net.store_patterns(pattern_list)

# create a noisy version of a pattern and use that to initialize the network
noisy_init_state = pattern_tools.flip_n(checkerboard, nr_of_flips=3)
hopfield_net.set_state_from_pattern(noisy_init_state)

# from this initial state, let the network dynamics evolve.
states = hopfield_net.run_with_monitoring(nr_steps=6)

# each network state is a vector. reshape it to the same shape used to create the patterns.
states_as_patterns = factory.reshape_patterns(states)
# plot the states of the network

plt.rcParams["figure.figsize"] = [12,9]
plot_tools.plot_state_sequence_and_overlap(states_as_patterns, pattern_list, reference_idx=0, suptitle="Network dynamics with 7 patterns stored")

pattern_size = 4
# create an instance of the class HopfieldNetwork
hopfield_net = network.HopfieldNetwork(nr_neurons= pattern_size**2)
# instantiate a pattern factory
factory = pattern_tools.PatternFactory(pattern_size, pattern_size)
# create a checkerboard pattern and add it to the pattern list
checkerboard = factory.create_checkerboard()
pattern_list = [checkerboard]
pattern_list.extend(factory.create_random_pattern_list(nr_patterns=3, on_probability=0.5))
plot_tools.plot_pattern_list(pattern_list)
hopfield_net.store_patterns(pattern_list)

# create a noisy version of a pattern and use that to initialize the network
noisy_init_state = pattern_tools.flip_n(checkerboard, nr_of_flips=10)
hopfield_net.set_state_from_pattern(noisy_init_state)

# from this initial state, let the network dynamics...


I am working on the file "Inference_Big_data/Hopfield/Hopfield.ipynb".
The first task is:
This phase diagram shows that there is roughly a triangular region where the network is a good memory device. Outside this region (on the right) the device is not useful as a memory device; $m$ is 0. At the obundary, $m$ jumps discontinuously down to $0$, except on the $T$ axis. There is a critical temperature $T=1$ which sets the limit on the $T$ axis. In the terminology ofp hase transitions, this transition is of first order.
Like we could have propose, an increase in the Temperature leads to a decrease in the storage capacity. At high temperature it is harder to fix the state of the network.
Correlated patterns :
Right now we only have talked about uncorellated patterns. But in reality the patterns we see each day are strongly correlated. Hence we can ask if this algorithm works as well for correlated patterns.
Let's take the alphabet. We are going to observe the influence of the letter on the dynamics of the network, and on the storage capacity.
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
from neurodynex.hopfield_network import network, pattern_tools, plot_tools
import numpy

# the letters we want to store in the hopfield network
letter_list = ['A', 'B', 'C', 'S', 'X', 'Y', 'Z']

# set a seed to reproduce the same noise in the next run
# numpy.random.seed(123)

abc_dictionary =pattern_tools.load_alphabet()
print("the alphabet is stored in an object of type: {}".format(type(abc_dictionary)))
# access the first element and get it's size (they are all of same size)
pattern_shape = abc_dictionary['A'].shape
print("letters are patterns of size: {}. Create a network of corresponding size".format(pattern_shape))
# create an instance of the class HopfieldNetwork
hopfield_net = network.HopfieldNetwork(nr_neurons= pattern_shape[0]*pattern_shape[1])

# create a list using Pythons List Comprehension syntax:
pattern_list = [abc_dictionary[key] for key in letter_list ]
plot_tools.plot_pattern_list(pattern_list)

# store the patterns
hopfield_net.store_patterns(pattern_list)

# # create a noisy version of a pattern and use that to initialize the network
noisy_init_state = pattern_tools.get_noisy_copy(abc_dictionary['A'], noise_level=0.2)
hopfield_net.set_state_from_pattern(noisy_init_state)

# from this initial state, let the network dynamics evolve.
states = hopfield_net.run_with_monitoring(nr_steps=4)

# each network state is a vector. reshape it to the same shape used to create the patterns.
states_as_patterns = pattern_tools.reshape_patterns(states, pattern_list[0].shape)

# plot the states of the network
plot_tools.plot_state_sequence_and_overlap(
    states_as_patterns, pattern_list, reference_idx=0, suptitle="Network dynamics for letter A")


I am working on the file "Inference_Big_data/Hopfield/Hopfield.ipynb".
The first task is:
Those two pictures show the limits of out model. In the situation wherem ore patterns are stored, the netowrk is not able to correctly recover the ones he stored. Moreover, in a situation where he was able to recover the correct pattern, if we increase the noise, the network does not recover the pattern (this seems logical).
Weights distribution
Let's just have a quick look at the weight's distribution of the Hopfield network before looking at the phase transition. We are going to try to understand how the weights are instancied in the Hopfield network. For that we will create a checkerboard pattern, and a L-pattern and observe the behavior of the system. Each time we will plot the weight matrice and the weight distribution.
Can you write Python code for it?

pattern_size = 4

# create an instance of the class HopfieldNetwork
hopfield_net1 = network.HopfieldNetwork(nr_neurons= pattern_size**2)
%matplotlib inline
factory = pattern_tools.PatternFactory(pattern_size, pattern_size)
# create a checkerboard pattern and add it to the pattern list
checkerboard = factory.create_checkerboard()
pattern_list = [checkerboard]
hopfield_net1.store_patterns(pattern_list)
#plot_tools.plot_pattern_list(pattern_list)
plot_tools.plot_nework_weights(hopfield_net1)
plt.figure()
plt.hist(hopfield_net1.weights.flatten())
plt.title('Histogram of the weights distribution checkerboard pattern')
plt.show()

hopfield_net2 = network.HopfieldNetwork(nr_neurons= pattern_size**2)
%matplotlib inline
L_pattern = factory.create_L_pattern()
pattern_list= [L_pattern]
hopfield_net2.store_patterns(pattern_list)
#plot_tools.plot_pattern_list(pattern_list)
plot_tools.plot_nework_weights(hopfield_net2)
plt.figure()
plt.hist(hopfield_net2.weights.flatten())
plt.title('Histogram of the weights distribution for L pattern')
plt.show()

hopfield_net3 = network.HopfieldNetwork(nr_neurons= pattern_size**2)

L_pattern = factory.create_L_pattern()
checkboard = factory.create_checkerboard()
pattern_list= [L_pattern,checkboard]
plot_tools.plot_pattern_list(pattern_list)
hopfield_net3.store_patterns(pattern_list)
plot_tools.plot_nework_weights(hopfield_net3)
plt.figure()
plt.hist(hopfield_net3.weights.flatten())
plt.title('Histogram of the weights distribution for both patterns')
plt.show()


I am working on the file "Inference_Big_data/Hopfield/Hopfield.ipynb".
The first task is:
We will now study how a network stores and retrieves patterns. Using a small network allows us to have a close look at the network weights and dynamics.
Can you write Python code for it?

#%matplotlib inline
#%matplotlib qt
import matplotlib.pyplot as plt
from neurodynex.hopfield_network import network, pattern_tools, plot_tools

pattern_size = 5

# create an instance of the class HopfieldNetwork
hopfield_net = network.HopfieldNetwork(nr_neurons= pattern_size**2)
# instantiate a pattern factory
factory = pattern_tools.PatternFactory(pattern_size, pattern_size)
# create a checkerboard pattern and add it to the pattern list
checkerboard = factory.create_checkerboard()
pattern_list = [checkerboard]

# add random patterns to the list
pattern_list.extend(factory.create_random_pattern_list(nr_patterns=3, on_probability=0.5))
plot_tools.plot_pattern_list(pattern_list)
# how similar are the random patterns and the checkerboard? Check the overlaps
overlap_matrix = pattern_tools.compute_overlap_matrix(pattern_list)
#plot_tools.plot_overlap_matrix(overlap_matrix)

# let the hopfield network "learn" the patterns. Note: they are not stored
# explicitly but only network weights are updated !
hopfield_net.store_patterns(pattern_list)

# create a noisy version of a pattern and use that to initialize the network
noisy_init_state = pattern_tools.flip_n(checkerboard, nr_of_flips=3)
hopfield_net.set_state_from_pattern(noisy_init_state)

# from this initial state, let the network dynamics evolve.
states = hopfield_net.run_with_monitoring(nr_steps=4)

# each network state is a vector. reshape it to the same shape used to create the patterns.
states_as_patterns = factory.reshape_patterns(states)
# plot the states of the network

plt.rcParams["figure.figsize"] = [12,9]
plot_tools.plot_state_sequence_and_overlap(states_as_patterns, pattern_list, reference_idx=0, suptitle="Network dynamics")


I am working on the file "Inferential Statistics.ipynb".
The first task is:
Create a new distribution called fitted, and tweak its parameters so that it fits the sample distribution.
Discussion

What are your parameter values?
What's happening here? Isn't the sample drawn from the same distribution as the population?
How could the sample parameters and population parameters be different?

The key point here is that the samples are random samples of the population. The computed mean from the sample may not necessarily match with the actual population mean.
Central Limit Theorem
The central limit theorem is central to statistics.
Quoting from Wikipedia:

In probability theory, the central limit theorem (CLT) states that, given certain conditions, the arithmetic mean of a sufficiently large number of iterates of independent random variables, each with a well-defined expected value and well-defined variance, will be approximately normally distributed, regardless of the underlying distribution.

We're going to experimentally look at what this statement means.
A crazy dice
Let's say that we have a crazy dice with an underlying distribution that we'd never expect to encounter.
Can you write Python code for it?

xs = range(1,7)  # numbers 1 through 6
ps = [0.4, 0.1, 0, 0.2, 0.3, 0]
crazy_dice = rv_discrete(name='crazy_dice', values=[xs, ps])


I am working on the file "Inferential Statistics.ipynb".
The first task is:
Exercise 2
In your average bag of M&Ms, there are on average 30 brown, 20 yellow, 20 red, 10 orange, 10 green, and 10 blue out of every 100 M&M candies. I draw out 50 candies, and find that I have 22 brown. Is this bag an "average" bag?
Can you write Python code for it?

null_hypergeom = hypergeom.rvs(M=100, n=30, N=50, size=10000)
sum(null_hypergeom >= 22) / 10000


I am working on the file "Inferential Statistics.ipynb".
The first task is:
Exercise 3
Systolic blood pressure in healthy patients is Normally distributed with mean 122 mmHg and std. deviation 9 mmHg. In hypertensive patients, it is Normally with mean 140 and std. deviation 6 mmHg. (I pulled out these numbers from a hat - http://www.cdc.gov/nchs/data/nhsr/nhsr035.pdf)
A patient comes in and measures 130 mmHg.
What is the p-value of this patient's blood pressure, under:

the hypothesis that this patient is healthy?
the hypothesis that this patient is hypertensive?

Plot the distributions and the patient's systolic blood pressure.
Can you write Python code for it?

null_healthy = norm.rvs(122, 9, size=100000)
null_hyper = norm.rvs(140, 6, size=100000)

sns.kdeplot(null_healthy)
sns.kdeplot(null_hyper)
plt.axvline(130)

# P-value calculation under healthy hypothesis
np.sum(null_healthy >= 130) / len(null_healthy)

# P-value calculation under hypertension hypothesis
np.sum(null_hyper <= 130) / len(null_hyper)


I am working on the file "Inferential Statistics.ipynb".
The first task is:
Finally, let's calculate the probability of seeing this number by random chance.
Can you write Python code for it?

p_value = num_heads / len(null_flips)
p_value


I am working on the file "Inferential Statistics.ipynb".
The first task is:
Once again, plot this below.
Can you write Python code for it?


I am working on the file "Inferential Statistics.ipynb".
The first task is:
Our test here tells us whether it is possible for us to see X number of heads or more under randomness. Let us count up the number of times we see 16 or more heads under the null model.
Can you write Python code for it?

num_heads = sum(null_flips >= 16)
num_heads


I am working on the file "Inferential Statistics.ipynb".
The first task is:
Roll the dice 4 times, but repeat 1000 times now. What's the distribution of the means?
Can you write Python code for it?

# Class activity
means = []

for i in range(1000):
    mean = np.mean(crazy_dice.rvs(size=4))
    means.append(mean)

# plt.hist(np.array(means), color='green')
sns.kdeplot(np.array(means), color='blue')  # plot the density plot
sns.kdeplot(norm.rvs(2.9,0.6,size=100000), color='red')  # plot an approximation of the actual distribution


I am working on the file "Inferential Statistics.ipynb".
The first task is:
The probability of seeing this under a null model of p=0.5 is very low (approximately 1 in 200). Therefore, it is likely the case that the coin is biased.
Exercise 1
Epidemiologists have been longitudinally measuring the rate of cancer incidence in a population of humans. Over the first 10 years, cancer rates were 10 per 10000 by the age of 30. In the 11th and 12th years, cancer rates went up to 14 per 10000. Calculate the p-value of the 11th year and the 12th year separately.
Hint: Model this as a Poisson process.
What are you assuming about the null model?
Can you write Python code for it?

null_poisson = poisson.rvs(10, size=100000)
sum(null_poisson >= 14) / 100000


I am working on the file "Inferential Statistics.ipynb".
The first task is:
Visualize this using the plot below.
Can you write Python code for it?


I am working on the file "Inferential Statistics.ipynb".
The first task is:
With this dice, we are essentially saying that we expect the dice to throw 1s (40% of the time), 5s (30% of the time), 4s (20% of the time), and 2s (10% of the time), with no expectation of getting 3s and 6s.
Now, let's roll the dice 4 times. What's the mean?
Can you write Python code for it?

# Class activity
sample1 = crazy_dice.rvs(size=50)
sns.kdeplot(sample1)
np.mean(sample1)


I am working on the file "Inferential_Statistics/Inferential_Statistics.ipynb".
The first task is:
Independent variable: Treatment condition consisting of congruent and incongruent words
Dependent variable: Response time
Hypothesis
$H_0 :  \mu_C = \mu_I $ There is no difference in mean response time between the congruent and incongruent word conditions
$H_a :   \mu_C \neq \mu_I $ There is a difference in mean response time between the congruent and incongruent word conditions
$\mu_C$ and $\mu_I$ denote the population means for the congruent and incongruent groups respectively.

Statistical test: Dependent t-test for paired samples is the statistical test that will be used.

This is a within-subject design, where the same subjects are being presented with two test conditions.

The reasons for choosing this test are as follows:
1) The sample size is less than 30
2) The population standard deviation is unknown
3) It is assumed that the distributions are Gaussian
Data Exploration and Visualization
Can you write Python code for it?

stroop_data.describe()

print "Median:\n", stroop_data.median()
print "\nVariance:\n", stroop_data.var()

fig, axs = plt.subplots(figsize=(18, 5), ncols = 3, sharey=True)
plt.figure(figsize=(8, 6))
sns.set_palette("Set2")

# Fig 1 - Congruent Words - Response Time
sns.boxplot(y="Congruent", data=stroop_data,
            ax=axs[0]).set_title("Fig 1: Congruent Words - Response Time (in seconds)")

# Fig 2 - Incongruent Words - Response Time
sns.boxplot(y="Incongruent", data=stroop_data, color="coral",
            ax=axs[1]).set_title("Fig 2: Incongruent Words - Response Time (in seconds)")

# Fig 3 - Congruence vs. Incongruence
sns.regplot(x="Congruent", y="Incongruent", data=stroop_data, color="m", fit_reg=False,
            ax=axs[2]).set_title("Fig 3: Congruence vs. Incongruence (in seconds)")


I am working on the file "Inferential_Statistics/Inferential_Statistics.ipynb".
The first task is:
Variables
Can you write Python code for it?

stroop_data = pd.read_csv('./stroopdata.csv')
stroop_data.head()


I am working on the file "InfoRetri/VectorSpacePractice.ipynb".
The first task is:
Now let's try the second vectorization method
Can you write Python code for it?

tfidfVectorizer = TfidfVectorizer()
d1_tf, d2_tf, q_tf = tfidfVectorizer.fit_transform(col)
for i in (d1_tf, d2_tf, q_tf):
    print(i.shape)

# again, cosine similarities
cs_docs_tf = cosine_similarity(d1_tf, d2_tf)
print(content(cs_docs_tf))
cs_d1_q_tf = cosine_similarity(d1_tf, q_tf)
print(content(cs_d1_q_tf))
cs_d2_q_tf = cosine_similarity(d2_tf, q_tf)
print(content(cs_d2_q_tf))


I am working on the file "InfoRetri/VectorSpacePractice.ipynb".
The first task is:
There are at least two ways of vectorizing the collection, one is by simple count, the other is use the term frequency * inversed document frequency (to reduce the weight imposed by the very common words but meaningless words such as a, the, and, etc.). Let's start with the first one.
Can you write Python code for it?

# create a collection matrix (using the count vectorizer)
countVectorizer = CountVectorizer()
# The CountVectorizer will return a document-term sparse matrix
# the rows represent the documents, and the columns represent terms
# since we have only 2 documents, I use 2 variables to represent the 2 vectors returned
d1_count, d2_count, q_count = countVectorizer.fit_transform(col)
print(d1_count.shape)
print(d2_count.shape)
print(q_count.shape)

def content(name):
    '''
    print out the cosine similarity result

    params:
    -----------------
    name: the variable name for the cosine similarity function output (a 1X1 matrix in our case)
    '''
    return 'The cosine similarity between the two docs is {name:.4}.'.format(name=name[0][0])

# let's see the cosine similarity of the two documents first
cs_docs = cosine_similarity(d1_count, d2_count)
print(content(cs_docs))

# create a query matrix
d1_q = cosine_similarity(d1_count, q_count)
print(content(d1_q))
d2_q = cosine_similarity(d2_count, q_count)
print(content(d2_q))
# so the query will probably return the first file for us


I am working on the file "InformationRetrieval.ipynb".
The first task is:
Agenda

Abgrenzung zum Booleschen Retrievalmodell
Nutzer-zentrierte Herleitung des Vektorraummodells
Auswirkungen der natürlichen Sprache auf das Modell
Vor- und Nachteile des Vektorraummodells
Praktische Anwendungen im Text und Data Mining

Boolesches Retrieval

Relevanz wird anhand der Existenz des Anfrageterms im Dokument bestimmt
Score = 1 für relevante Dokumente
Score = 0 für nichtrelevante Dokumente
Kein Ranking möglich, ein Dokument ist relevant oder nicht
Extended Boolean Retrieval bringt Ranking
Auch in relationalen Datenbanken verwendet, In Bibliotheken: OPAC

Vorteile

Simple Anfragen sind leicht zu verstehen

Nachteile

Schwierig, genaue Anfragen zu spezifizieren
Logische Junktoren (UND, ODER) entsprechen nicht immer der natürlichen Sprache
Ergebnismengen werden schnell zu klein oder zu groß, je nach Junktoren
Ergebnis ist nur sortier- bzw. gruppierbar

Nur ist der Nutzer immer so präzise?
Nutzer sind sich im Unklaren...

über den Informationsbedarf
wie der Informationsbedarf formuliert werden kann
was in der Kollektion enthalten ist
was "gute" Ergebnisse sind, d.h.,
ob relevante Ergebnisse gefunden wurden
ob alle wichtigen Ergebnisse gefunden wurden
Dokumente, die einer Anfrage „ähnlich“ sind, sind wahrscheinlich relevant für diese Anfrage

[Kuhlthau 1991, Ingwersen 1996, Ellis/Haugan 1997, Reiterer et al. 2000] und viele mehr...
Wie lässt sich Ähnlichkeit ausdrücken?
Vom Dokument zum Vektor

Dokumente und Anfrage repräsentiert durch t-dimensionale Vektoren
t ist Termanzahl des Indexvokabulars
i-te Vektorkomponente ist Termgewicht $w_{dk}$ (reelle Zahl)
Hohes Termgewicht ⇒ entsprechender Term repräsentiert Dokument/Anfrage gut

[Salton et al. 1975]
Can you write Python code for it?

# create sample data
d = {'doc01': (1,2), 'doc02': (10,10),'doc03':(60,40),'doc04':(100,80),'doc05':(99,81),'doc06':(1,1),'doc07':(45,55),'doc08':(9,10),'doc09':(11,11),'doc10':(1,11)}
# create a data frame from our sample data
sampleDF=pd.DataFrame(data=d)
# transpose the data frame for later use
sampleDF=sampleDF.transpose()
sampleDF.columns = ['term 2 (z.B. Retrieval)', 'term 1 (z.B. Information)']
sampleDF.head(10)

# plot the sample data

sampleDF.plot(x=0,y=1,kind='scatter',alpha=0.75,s=70) # we have to define explicitly which data is used for the x and y axes


I am working on the file "InformationRetrieval.ipynb".
The first task is:
Clusterhypothese
“This hypothesis may be simply stated as follows: closely associated documents tend to be relevant to the same requests.”
[van Rijsbergen 1979, Ch. 3]
Das Skalarprodukt als Ähnlichkeitsmaß
$$
similarity(V_Q,V_D)=<V_Q,V_D>=\sum_{k=1}^tw_{qk}w_{dk}=|V_Q||V_D|\cos(\alpha)
$$
Vorkommenshäufigkeit des Terms im Dokument

Vorkommenshäufigkeit $tf_{dk}$ des Terms im Dokument, desto besser beschreibt der Term das Dokument
Achtung: absolute Vorkommenshäufigkeit bei langen Dokumenten tendenziell höher
Konsequenz: lange Dokumente werden bei der Suche bevorzugt
Can you write Python code for it?

vq=[0,1,1,0]
vd1=[50,5,0,0]
vd2=[0,2,2,0]
print "similarity between Vq and Vd1: %i"%np.inner(vq,vd1) # inner product is another name for the scalar/dot product
print "similarity between Vq and Vd2: %i"%np.inner(vq,vd2)


I am working on the file "InformationRetrieval.ipynb".
The first task is:
Vorkommenshäufigkeit des Terms im Dokument
Unabhängigkeit von Dokumentlänge durch Normierung der Dokumentvektoren:
$$
w_{dk}=\frac{tf_{dk}}{\sqrt{\sum_{i=1}^t}tf_{di}^2}
$$
Zipf's Law
Ordnet man Worte eines Textes nach ihrer Häufigkeit, so ist ihre Auftrittswahrscheinlichkeit p(n) umgekehrt proportional zu ihrer Position n innerhalb der Ordnung ⇒ Hyperbel
$$
p(n)\simeq \frac{1}{n}
$$

Zipf‘s Law als Motivation für Stoppworte

Stoppworte: Worte, die aus dem Indexvokabular entfernt werden, weil sie...
häufig (oder zu selten) sind
"ohne Bedeutung" sind
Wie wirken sich Stoppworte etc. auf Dokumentvektoren aus?
Stoppwort-Eliminierung vermindert Möglichkeiten der Phrasensuche
Can you write Python code for it?

# with stopword in first dimension
vq=[100,0,0,1,2]
b=[99,1,2,0,0]
c=[80,0,0,2,2]

print "Before stopword elimination"
print "\tsimilarity between Vq and b: %i"%np.inner(vq,b)
print "\tsimilarity between Vq and c: %i"%np.inner(a,c)
# after stopword elimination
vq2=[0,0,1,2]
b2=[1,2,0,0]
c2=[0,0,2,2]

print "\nAfter stopword elimination"
print "\tsimilarity between Vq and b: %i"%np.inner(vq2,b2)
print "\tsimilarity between Vq and c: %i"%np.inner(a2,c2)


I am working on the file "IngresosGastosGobierno.ipynb".
The first task is:
Ejemplito
Can you write Python code for it?

cobre = spend_gob.query("nivel3 == 'Cobre bruto'")

alt.Chart(cobre).mark_bar().encode(
    x='periodo:O',
    y='sum(real_amount):Q',
    color='nivel3'
)

alt.Chart(spend_gob).mark_area().encode(
    x='periodo:O',
    y='sum(real_amount):Q',
    color='nivel3',
    column='nivel2'
)


I am working on the file "Initiation.ipynb".
The first task is:
De nombreuses possibilités d'indexation
Can you write Python code for it?


I am working on the file "Initiation.ipynb".
The first task is:
Il est possible d'avoir des arguments par défaut.
Can you write Python code for it?

def mafonction(a, b=2):
    """Division de a par b"""
    return a/b
print(mafonction(5))
print(mafonction(5,10))

mafonction?


I am working on the file "Initiation.ipynb".
The first task is:
Il est possible de modifier les éléments d'une liste et même de changer leur type.
Can you write Python code for it?


I am working on the file "Initiation.ipynb".
The first task is:
La syntaxe maléfique de l'import de tout ce qui est dans une librairie.
python
from os import *
Numpy: tableaux numériques
Can you write Python code for it?

import numpy as np #abbréviation du nom de la librairie pour taper 2 lettres au lieu de 5


I am working on the file "Initiation.ipynb".
The first task is:
Librairies
Import
Toute la librairie
Can you write Python code for it?


I am working on the file "Initiation.ipynb".
The first task is:
Listes
Can you write Python code for it?

l = [a, b, 127, 3.14, ["James", "Bond"], c]
print(l)
print(l[0])
print(l[0][1])
print(l[0]+l[1].format(l[-2][0], l[-2][1])) # pour "déplier" tous les éléments de la liste l[-2] en tant qu'arguments de la fonction format
#possible d'écrire directement format(*l[-2])
print(l[-1].format(l[2]) + " Mr {}.".format(l[-2][-1]))


I am working on the file "Initiation.ipynb".
The first task is:
Mais on ne peut plus sélectionner les données directement, pas très pratique pour l'analyse de données. On revient donc en mode normal (fenêtre externe).
Can you write Python code for it?

%matplotlib

plt.figure('Evolution de b') # création ou sélection d'une figure. La figure est nommée, ce qui est plus explicite qu'une numérotation
plt.clf() # on efface la figure courante
plt.plot(b)
plt.yscale('log')
plt.ylabel('b')
plt.xlabel(r'$\gamma_c$') # pas de caractères spéciaux, donc on a le droit de mettre des antislash pour les commandes LaTeX
plt.savefig('mafig.pdf')

print('a\tb')


I am working on the file "Initiation.ipynb".
The first task is:
On peut documenter ses propres fonctions.
Can you write Python code for it?

def mafonction(a, b):
    """Division de a par b"""
    return a/b

mafonction?


I am working on the file "Initiation.ipynb".
The first task is:
Phrases à trous
Can you write Python code for it?

b = "Je m'appelle {}."
print(b)
print(b.format("Mathieu"))

print(a.format("Mathieu"))

b = "My name is {1}, {0} {1}."
print(b.format("James", "Bond"))

b2 = "My name is {nom}, {prenom} {nom}."
print(b2.format(prenom="James", nom="Bond"))

c = "Veuillez charger le fichier acquisition_t{:03d}.txt" #nombre décimal sur 3 chiffres, éventuellement complétés de zéros
print(c.format(5))
print(c.format(23))
print(c.format(328))

print(c.format(0.1))
print(c.format("hi"))


I am working on the file "Initiation.ipynb".
The first task is:
Pour un document à montrer, on préfère avoir des figures directement insérée dedans en tant qu'image.
Can you write Python code for it?


I am working on the file "Initiation.ipynb".
The first task is:
Quelques fonctions pour créer des tableaux standards.
Can you write Python code for it?

np.zeros(3), np.ones(2, complex), np.arange(4)

l = [1,5,7] #liste numérique
print(l)
a = np.array(l) #transformée en tableau numpy
print(a)
b = np.array([1.2, 5, 6]) #les 2 étapes d'un coup
print(b)
print("a est un tableau à {} dimensions de taille {} contenant uniquement des {}".format(a.ndim, a.shape, a.dtype))
print("b est un tableau à {} dimensions de taille {} contenant uniquement des {}".format(b.ndim, b.shape, b.dtype))


I am working on the file "Initiation.ipynb".
The first task is:
Syntaxe complète des instructions de formattage, si un jour vous avez envie d'afficher un nombre hexadécimal sur 7 charactères minimum, éventuellement complété avec des symboles livre sterling, ou un nombre flottant avec 2 chiffres significatifs.
On peut aussi obtenir de l'aide sur une méthode d'un objet.
Can you write Python code for it?

c.format?

print("Veuillez charger le fichier acquisition_t%03d.txt"%5)


I am working on the file "Initiation.ipynb".
The first task is:
Tableaux en 2 ou plus dimensions en imbriquant des listes
Can you write Python code for it?

a = np.array([[1,2,3],[7,4,5]])
print("a est un tableau à {} dimensions de taille {} contenant uniquement des {}".format(a.ndim, a.shape, a.dtype))

print(a)
print(a.T)


I am working on the file "Initiation.ipynb".
The first task is:
Toutes les opérations se font élément par élément, même la multiplication. Pas besoin de point comme en MATLAB. La multiplication matricielle se fait avec np.matmul.
Can you write Python code for it?

print(2 * a)
print(a + b)
print(a * b)
print(a / b)
print(a>2)
print(np.exp(a))


I am working on the file "Initiation.ipynb".
The first task is:
Un peu plus complexe : on imbrique deux indentations et on a 2 arguments avec des valeurs par défaut.
Can you write Python code for it?

def mafonction(a, b=2, diviser=True):
    """Division de a par b.
    Si on ne divise pas, on retourne a et b""" #on a le droit de passer à la ligne si on a ouvert 3 guillemets
    if diviser:
        return a/b
    else:
        return a,b
print(mafonction(5))
print(mafonction(5, 10))
print(mafonction(5, diviser=False))


I am working on the file "Input/notebooks/kafkaReceiveAndSaveToCassandraPy.ipynb".
The first task is:
Create streaming task

Receive data from Kafka 'test' topic every five seconds
Get stream content, and add receiving time to each message
Save each RDD in the DStream to Cassandra. Also print on screen
Can you write Python code for it?

ssc = StreamingContext(sc, 5)
kvs = KafkaUtils.createStream(ssc, "127.0.0.1:2181", "spark-streaming-consumer", {'test': 1})
data = kvs.map(lambda x: x[1])
rows= data.map(lambda x:Row(time_sent=x,time_received=time.strftime("%Y-%m-%d %H:%M:%S")))
rows.foreachRDD(saveToCassandra)
rows.pprint()


I am working on the file "Input/notebooks/kafkaReceiveAndSaveToCassandraPy.ipynb".
The first task is:
Get Cassandra table content
Can you write Python code for it?

data=sqlContext.read\
    .format("org.apache.spark.sql.cassandra")\
    .options(table="sent_received", keyspace="test_time")\
    .load()
data.show()


I am working on the file "Input/notebooks/kafkaReceiveAndSaveToCassandraPy.ipynb".
The first task is:
Get Cassandra table content using SQL
Can you write Python code for it?

data.registerTempTable("sent_received");
data.printSchema()
data=sqlContext.sql("select * from sent_received")
data.show()


I am working on the file "Input/notebooks/kafkaReceiveAndSaveToCassandraPy.ipynb".
The first task is:
Load modules and start SparkContext
Note that SparkContext must be started to effectively load the package dependencies. Two cores are used, since one is needed for running the Kafka receiver.
Can you write Python code for it?

from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext, Row
conf = SparkConf() \
    .setAppName("Streaming test") \
    .setMaster("local[2]") \
    .set("spark.cassandra.connection.host", "127.0.0.1")
sc = SparkContext(conf=conf)
sqlContext=SQLContext(sc)
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils


I am working on the file "Input/notebooks/kafkaReceiveAndSaveToCassandraPy.ipynb".
The first task is:
Start streaming
Can you write Python code for it?


I am working on the file "Input/notebooks/kafkaReceiveAndSaveToCassandraPy.ipynb".
The first task is:
Stop streaming
Can you write Python code for it?


I am working on the file "Input/notebooks/kafkaSendDataPy.ipynb".
The first task is:
Start Kafka producer
One message giving current time is sent every second to the topic test
Can you write Python code for it?

producer = KafkaProducer(bootstrap_servers='localhost:9092')
while True:
    message=time.strftime("%Y-%m-%d %H:%M:%S")
    producer.send('test', message)
    time.sleep(1)


I am working on the file "Input_utils.ipynb".
The first task is:
Accumulator inputs  -  Verguts& Fias##
Numerosity from 1 to 5, where unity is represented by 3 repeated ones. (e.g. 2 is represented as
[1,1,1,1,1,1,0,0,0,0,0,0,0,0,0]).
No zero vector.
Can you write Python code for it?

inputs =  accumulatorMatrix(5, times=2).outputs
labels = find_labels(inputs, multiple=2, one_hot=True)

Dataset = namedtuple('Dataset', ['data', 'labels'])
verguts2004 = Dataset(inputs, labels)
pickle_test(verguts2004, "verguts_accumulator")

verguts2004.labels


I am working on the file "Input_utils.ipynb".
The first task is:
Binary vector generator
Version 2 - via Itertool
Can you write Python code for it?

# def binary_vector_2(rows_n = [2,4,6,8,10], columns_n = 10):
#     rows = how_many(rows_n, 10)
#     index = 0
#     locations = np.zeros((rows, columns_n))

#     for i in rows_n:
#         for bin_string in kbits(10,i):
#             bin_array = np.fromstring(bin_string,'u1') - ord('0')
#             locations[index,:] = bin_array
#             index = index+1
#     return locations

# inputs = binary_vector_2()
# labels = find_labels(inputs, one_hot=True)
# #dataset_ver = Dataset(inputs, labels)
# #pickle_test(dataset_ver)
# inputs.shape

import numpy as np
import itertools
from scipy.special import comb

def kbits(n, k):
    """ Generate a list of ordered binary strings representing all the possibile
    way n chooses k.
    Args:
        n (int): set cardinality
        k (int): subset cardinality

    Returns:
        result (string): list of binary strings
    """
    result = []
    for bits in itertools.combinations(range(n), k):
        s = ['0'] * n
        for bit in bits:
            s[bit] = '1'
        result.append(''.join(s))
    return result

def binary_vector_2(rows_n = [2,4,6,8,10], distribution=[45], columns_n = 10):
    """ Matrix of binary vectors from distribution.
    Args:
        rows_n (int, ndarray): nx1
        distribution (int, ndarray): nx1

    Returns:
        ndarray of dimension rows_n * distribution, columns_n

    TODO: check inputs, here given as list, but should it be a ndarray?
    remove index accumulator and rewrite via len(kbit)

    Examples:
        Should be written in doctest format and should illustrate how
        to use the function.
        distribution=comb(columns_n, row)
        returns all possible combinations: in reality not, should remove randomness: or better set flag
        replacement = False
    """

    rows_n = np.array(rows_n)
    distribution = np.array(distribution)

    assert np.all(rows_n >0)
    assert np.all(distribution >0), "Distribution values must be positive. {}...


I am working on the file "Input_utils.ipynb".
The first task is:
Label the data
Can you write Python code for it?

def find_labels(inputs, multiple=1, one_hot=False):
    """
    Generate the labels corresponding to binary vectors. If one_hot = true, the label are
    on hot encoded, otherwise integers.

    Args:
        inputs (int, ndarray): ndarray row samples
        multiple (int): lenght of unity representation
        one_hot (bool): False for integer labels, True for one hot encoded labels

    Returns:
        labels (int): integer or one hot encoded labels

    """
    labels = (np.sum(inputs, axis=1)/multiple).astype(int)
    if one_hot:
        size = np.max(labels)
        label_matrix = np.zeros((labels.shape[0], size+1))
        label_matrix[np.arange(labels.shape[0]), labels] = 1
        labels = label_matrix
    return labels


I am working on the file "Input_utils.ipynb".
The first task is:
We now pickle the named_tuple
cfr. When to pickle
See http://localhost:8888/notebooks/Dropbox/Programming/Jupyter/Competitive-Unsupervised/NNTf.ipynb
for creating a panda dataframe out of the namedtuple
http://stackoverflow.com/questions/16377215/how-to-pickle-a-namedtuple-instance-correctly
https://blog.hartleybrody.com/python-serialize/
Simon and Petersons 2000, Input Dataset
The dataset consist of vecors of lenght 16 and vector of lenght 6 as label, one hot encoded.
50.000 inputs pattern are generated
A numerosities in range(6) is picked randomly.
Then locations are randomly selected.
Verguts and Fias: Inputs ##
Uniformly distributed input
The outlier 5 is represented only 10 times, this to allow the net to see it a reasonable numbers of times, but not too much, considering that it can only have one shape.
Can you write Python code for it?

rows_n = [2,4,6,8,10]
#comb(10, rows_n)
inputs = binary_vector_2(distribution = comb(10, rows_n))
labels = find_labels(inputs, multiple=2, one_hot=True)

count = 0
for i in inputs:
    print(count, i, int(np.sum(i)/2), labels[count])
    count +=1


Ac = A
Bc = B
Cc = C
Dc = D
Ec = E
Fc = F
Gc = G
Hc = H
Amplac = Ampla

print(A,B,C,D,E,F,G,H,Ampla)

counter_vagas = 0
counter_vagas_nao_cota = 0
counter_vagas_ouvintes = 0
counter_vagas_ouvintes_nao_cota = 0


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
CHECK LISTS SPLIT
Can you write Python code for it?

print ("CHECK - Total de Candidatos = ",(len(candidatos_ouvintes) + len(candidatos_surdos)))

for i in candidatos_ouvintes:
    if i["surdo"]:
        print ("ERROR = ", i)

for i in candidatos_surdos:
    if not i["surdo"]:
        print ("ERROR = ",i)


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
CHECK SPLIT COTAS
Can you write Python code for it?

print ((len(cota_A)) +\
(len(cota_B))+\
(len(cota_C))+\
(len(cota_D))+\
(len(cota_E))+\
(len(cota_F))+\
(len(cota_G))+\
(len(cota_H))+\
(len(ampla_con)))

print("COTA A =",len(cota_A),\
      "COTA B =",len(cota_B),\
      "COTA C =",len(cota_C),\
      "COTA D =",len(cota_D),\
      "COTA E =",len(cota_E),\
      "COTA F =",len(cota_F),\
      "COTA G =",len(cota_G),\
      "COTA H =",len(cota_H),\
      "Ampla =",len(ampla_con))


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
Criação dos aprovados finais
Can you write Python code for it?


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
DELETAR
GERADOR DE PONTUAÇÃO
Can you write Python code for it?

#def make_random_int(data):
 #   if data =="":
  #      return random.randint(0 , 1000)
   # else:
    #    return float(data)

for i in Unsorted_allCandidates:
    i["nascimento"] = make_date(i["nascimento"])
    i["surdo"] = make_boolean(i["surdo"])
    i["Cota"] = make_boolean(i["Cota"])
    i["Tipo cota"] = tipo_cota_clean(i["Tipo cota"])
    i["Pont"] = make_float(i["Pont"])
    i["Ano ENEM"] = make_int(i["Ano ENEM"])
    i["\ufeffNome"] = make_str(i["\ufeffNome"])
    i["\ufeffNome"] = make_upper(i["\ufeffNome"])
    #i["Pont"] = make_random_int(i["Pont"])

len(Unsorted_allCandidates)


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
Definição dos surdos aprovados
Can you write Python code for it?

surdos_aprovados = candidatos_surdos[:14]
print (len(surdos_aprovados))
print (len(allCandidates))


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
Divisão da Lista
Can you write Python code for it?

ouvintes_aprovados_lista_1 = list()

def list_divide(data_in , data_out, field):
    for i in data_in[:]:
        if i[field] == False:
            data_out.append(i)
    return data_out

list_divide(Alunos_Aprovados_Finais_Ordenado , ouvintes_aprovados_lista_1, "surdo")

Display_Ouvintes_aprovados = pd.DataFrame(ouvintes_aprovados_lista_1)
#Display_Ouvintes_aprovados

#Display_surdos_aprovados

Display_Alunos_Aprovados_Finais

#export_Alunos_Aprovados = pd.ExcelWriter("Lista_01_POLO_INES.xlsx" , engine = 'xlsxwriter')
#Display_Alunos_Aprovados_Finais.to_excel(export_Alunos_Aprovados, index=False, sheet_name="Lista_01_INES")
#export_Alunos_Aprovados.save()


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
Divisão das cotas
a)  Candidatos com renda familiar bruta per capita igual ou inferior a 1,5 salário mínimo, que tenham cursado integralmente o ensino médio em escolas públicas.
b) Candidatos com deficiência que tenha renda familiar bruta per capita igual ou inferior a 1,5 salário mínimo, que tenham cursado integralmente o ensino médio em escolas públicas.
c) Candidatos autodeclarados pretos, pardos ou indígenas, com renda familiar bruta per capita igual ou inferior a 1,5 salário mínimo e que tenham cursado integralmente o ensino médio em escola pública.
d) Candidatos com deficiência autodeclarados pretos, pardos ou indígenas, com renda familiar bruta per capita igual ou inferior a 1,5 salário mínimo e que tenham cursado integralmente o ensino médio em escola pública.
e) Candidatos que, independente da renda, tenham cursado integralmente o ensino médio em escolas públicas.
f) Candidatos com deficiência que, independente da renda, tenham cursado integralmente o ensino médio em escolas públicas.
g) Candidatos autodeclarados pretos, pardos ou indígenas que, independente da renda, tenham cursado integralmente o ensino médio em escolas públicas.
h) Candidatos com deficiência autodeclarados pretos, pardos ou indígenas que, independente da renda, tenham cursado integralmente o ensino médio em escolas públicas.
Can you write Python code for it?

ampla_con , cota_A , cota_B , cota_C , cota_D , cota_E , cota_F , cota_G , cota_H = list(),list(),list(),list(),list(),list(),list(),list(),list()


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
Exclui surdos aprovados da lista geral
Can you write Python code for it?

clear_aprovados(allCandidates , surdos_aprovados)

len(allCandidates)

Display_allCandidatas[:30]

Display_surdos_aprovados = pd.DataFrame(surdos_aprovados)
Display_surdos_aprovados


def polo_selector(polo , data):
    for i in data[:]:
        if i["Polo desejado"] != polo:
            data.remove(i)

    return data

polo_selector(polo , Unsorted_allCandidates)


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
Get CPFs
cpfs = list()
for i in allCandidates:
    cpfs.append(i["CPF"])
outF = open("myOutFile.txt", "w")
for line in cpfs:
  outF.write(line)
  outF.write("\n")
outF.close()
DATA CLEANUP
Can you write Python code for it?

def make_date(date):
    if date == '':
        return None
    else:
        return dt.strptime(date, '%d/%m/%Y')

def make_boolean(data):
    if data == "SIM":
        return True
    elif data =="NÃO":
        return False

def make_float(data):
    if data =="":
        return 0
    else:
        return float(data)

def make_int(data):
    if data =="":
        return 0
    else:
        return int(data)

def tipo_cota_clean(data):
    if data =="":
        return None
    else:
        return data

def make_str(data):
    if data =="":
        return data
    else:
        return str(data)

def make_upper(data):
    return data.upper()

Unsorted_allCandidates[0]


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
LIMPAR APROVADOS DA LISTA GERAL
Can you write Python code for it?

def clear_aprovados(data, check):
    counter = 0
    for i in data[:]:
        if i in check:
            data.remove(i)
            counter += 1
    print(counter)


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
LISTA PARA MSTECH
Can you write Python code for it?

def write_csv(outputname , listname):
    with open(outputname, "w") as output:
        fieldnames = ["\ufeffNome" , "CPF" , "Classificação"]
        writer = csv.DictWriter(output, fieldnames=fieldnames , delimiter = ";")
        return writer.writerows(listname)

Alunos_aprovados_finais_MSTECH = Alunos_Aprovados_Finais_Ordenado[:]

for i in Alunos_aprovados_finais_MSTECH[:]:
    del i["RG"]
    del i["nascimento"]
    del i["Ano ENEM"]
    del i["Pont"]
    del i["surdo"]
    del i["Cota"]
    del i["Tipo cota"]
    del i["ENEM"]
    del i["Polo desejado"]

for index, value in enumerate(Alunos_aprovados_finais_MSTECH[:],1):
    value["Classificação"] = index

Alunos_Aprovados_TI = copy.deepcopy(Alunos_aprovados_finais_MSTECH)

for i in Alunos_Aprovados_TI:
    list_str =list(i["CPF"])
    list_str[0],list_str[1],list_str[2],list_str[3],list_str[4],list_str[5] = "X","X","X","X","X","X"
    i["CPF"] = "".join(list_str)

Display_LISTA_TI = pd.DataFrame(Alunos_Aprovados_TI)

cols = Display_LISTA_TI.columns.tolist()
cols

cols = cols[-1:] + cols[:-1]
cols

Display_LISTA_TI = Display_LISTA_TI[cols]

Display_Alunos_Aprovados_Finais

Alunos_aprovados_finais_MSTECH

Display_LISTA_TI

name01 = "#ALUNOS_01_" + polo +".xlsx"
name02 = "#MS_" + polo + ".csv"
name03 = "#DIVULGACAO_" + polo +".xlsx"
print (name01)
print (name02)
print (name03)

export_Alunos_Aprovados = pd.ExcelWriter(name01 , engine = 'xlsxwriter')
Display_Alunos_Aprovados_Finais.to_excel(export_Alunos_Aprovados, index=False, sheet_name="Lista_01")
export_Alunos_Aprovados.save()

write_csv(name02, Alunos_aprovados_finais_MSTECH)

export_Alunos_Aprovados = pd.ExcelWriter(name03 , engine = 'xlsxwriter')
Display_LISTA_TI.to_excel(export_Alunos_Aprovados, index=False, sheet_name="Lista_01")
export_Alunos_Aprovados.save()


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
NUMERO DE COTISTAS
Can you write Python code for it?

def cotas(data , field):
    counter_AMPLA , counter_A , counter_B , counter_C , counter_D , counter_E ,\
    counter_F , counter_G , counter_H, counter_error = 0,0,0,0,0,0,0,0,0,0
    for i in data:
        if i[field] == None:
            counter_AMPLA += 1
            ampla_con.append(i)
        elif i[field] == "A":
            counter_A += 1
            cota_A.append(i)
        elif i[field] == "B":
            counter_B += 1
            cota_B.append(i)
        elif i[field] == "C":
            counter_C += 1
            cota_C.append(i)
        elif i[field] == "D":
            counter_D += 1
            cota_D.append(i)
        elif i[field] == "E":
            counter_E += 1
            cota_E.append(i)
        elif i[field] == "F":
            counter_F += 1
            cota_F.append(i)
        elif i[field] == "G":
            counter_G += 1
            cota_G.append(i)
        elif i[field] == "H":
            counter_H += 1
            cota_H.append(i)
        else:
            counter_error += 1

    return counter_AMPLA , counter_A , counter_B , counter_C , counter_D , counter_E ,\
    counter_F , counter_G , counter_H, counter_error

cotas(allCandidates , "Tipo cota")


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
Notebook Execute
Can you write Python code for it?


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
Notebook Import
Can you write Python code for it?

def execute_notebook(nbfile):

    with io.open(nbfile) as f:
        nb = current.read(f, 'json')

    ip = get_ipython()

    for cell in nb.worksheets[0].cells:
        if cell.cell_type != 'code':
            continue
        ip.run_cell(cell.input)


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
ORDENA POR NOTA
Can you write Python code for it?

Alunos_Aprovados_Finais_Ordenado = sorted(Alunos_Aprovados_Finais, key = lambda element: element["Pont"], reverse=True)

Display_Alunos_Aprovados_Finais = pd.DataFrame(Alunos_Aprovados_Finais_Ordenado)
Display_Alunos_Aprovados_Finais


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
SELECIONADOR DE VAGAS DOS OUVINTES
Can you write Python code for it?

# FIRST FASE - Pega todos os OUVINTES e aplica a regra de cotas e ampla concorrência

for i in candidatos_ouvintes[:]:
    if i in cota_A[:]:
        if A > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_A.remove(i)
            A -= 1
            counter_vagas_ouvintes+=1
            print ("A = ", A)

for i in candidatos_ouvintes[:]:
    if i in cota_B[:]:
        if B > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_B.remove(i)
            B -= 1
            counter_vagas_ouvintes+=1
            print ("B = ", B)

for i in candidatos_ouvintes[:]:
    if i in cota_C[:]:
        if C > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_C.remove(i)
            C -= 1
            counter_vagas_ouvintes+=1
            print ("C = ", C)

for i in candidatos_ouvintes[:]:
    if i in cota_D[:]:
        if D > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_D.remove(i)
            D -= 1
            counter_vagas_ouvintes+=1
            print ("D = ", D)

for i in candidatos_ouvintes[:]:
    if i in cota_E[:]:
        if E > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_E.remove(i)
            E -= 1
            counter_vagas_ouvintes+=1
            print ("E = ", E)

for i in candidatos_ouvintes[:]:
    if i in cota_F[:]:
        if F > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_F.remove(i)
            F -= 1
            counter_vagas_ouvintes+=1
            print ("F = ", F)

for i in candidatos_ouvintes[:]:
    if i in cota_G[:]:
        if G > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_G.remove(i)
            G -= 1
            counter_vagas_ouvintes+=1
            print ("G = ", G)

for i in candidatos_ouvintes[:]:
    if i in cota_H[:]:
        if H > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_H.remove(i)
            H -= 1
           ...


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
SORT BY SCORE
Can you write Python code for it?

allCandidates = sorted(Unsorted_allCandidates, key = lambda element: element["Pont"], reverse=True)

Display_allCandidatas = pd.DataFrame(allCandidates)
Display_allCandidatas[:50]


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
Selecionador de vagas para surdos
Can you write Python code for it?

# FIRST FASE - Pega todos os surdos e aplica a regra de cotas e ampla concorrência

for i in surdos_aprovados[:]:
    if i in cota_A[:]:
        if A > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_A.remove(i)
            A -= 1
            counter_vagas+=1
            print ("A = ", A)

for i in surdos_aprovados[:]:
    if i in cota_B[:]:
        if B > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_B.remove(i)
            B -= 1
            counter_vagas+=1
            print ("B = ", B)

for i in surdos_aprovados[:]:
    if i in cota_C[:]:
        if C > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_C.remove(i)
            C -= 1
            counter_vagas+=1
            print ("C = ", C)

for i in surdos_aprovados[:]:
    if i in cota_D[:]:
        if D > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_D.remove(i)
            D -= 1
            counter_vagas+=1
            print ("D = ", D)

for i in surdos_aprovados[:]:
    if i in cota_E[:]:
        if E > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_E.remove(i)
            E -= 1
            counter_vagas+=1
            print ("E = ", E)

for i in surdos_aprovados[:]:
    if i in cota_F[:]:
        if F > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_F.remove(i)
            F -= 1
            counter_vagas+=1
            print ("F = ", F)

for i in surdos_aprovados[:]:
    if i in cota_G[:]:
        if G > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_G.remove(i)
            G -= 1
            counter_vagas+=1
            print ("G = ", G)

for i in surdos_aprovados[:]:
    if i in cota_H[:]:
        if H > 0:
            Alunos_Aprovados_Finais.append(i)
            cota_H.remove(i)
            H -= 1
            counter_vagas+=1
            print ("H = ", H)

for...


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
plot_graph(Cotistas , "Pretos Pardos e Indígenas")
Read CSV
Can you write Python code for it?

def read_csv2(filename):
    with open(filename, "rb") as f:
        lines = [l.decode('utf8', 'ignore') for l in f.readlines()]
        reader = csv.DictReader(lines , delimiter=';')
    return list(reader)


I am working on the file "Inscritos ARQUIVO FINAL + MSTECH + TI.ipynb".
The first task is:
separacao = list()
for i in allCandidates:
    if i["Tipo cota"] =="A":
        separacao.append(i)
Display_separacao = pd.DataFrame(separacao)
Display_separacao
SPLIT SURDOS E OUVINTES
Can you write Python code for it?

def is_surdo(data, field, surdo, ouvinte ):
    if data[field]:
        return surdo.append(i)
    else:
        return ouvinte.append(i)

candidatos_surdos = list()
candidatos_ouvintes = list()

for i in allCandidates:
    is_surdo(i , "surdo" , candidatos_surdos, candidatos_ouvintes)

len(candidatos_ouvintes)

len(candidatos_surdos)


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
And here is our full data set in all its glory
70 features, no NaN's, and no completely meaningless features (std dev != 0)
Can you write Python code for it?

# Write it to file for use in the next part.
full_df.to_csv('insight_demo_roadset62.csv')


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Assemble Cleaned, Processed Data
With data and labels, so that it's well-suited to be fed to our RFC
Can you write Python code for it?

# Let's prepare the lables, or, our target positions
targpos_df = data_df[['spillID','targetPos']].drop_duplicates().sort('spillID')
targpos_df.head()


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Can we do better with K-Neighbor or SVM?
Can you write Python code for it?

from sklearn import neighbors
from sklearn import svm
from sklearn import preprocessing

C = 1.0  # SVM regularization parameter
k_neighbors = 15

svc = svm.SVC(kernel='linear', C=C).fit(d_train, l_train)
rbf_svc = svm.SVC(kernel='rbf', gamma=0.7, C=C).fit(d_train, l_train)
poly_svc = svm.SVC(kernel='poly', degree=3, C=C).fit(d_train, l_train)
lin_svc = svm.LinearSVC(C=C).fit(d_train, l_train)
knclf = neighbors.KNeighborsClassifier(n_neighbors, weights=weights).fit(d_train, l_train)
rfc = RandomForestClassifier(n_estimators=100, max_depth=None, max_features='sqrt',
                             min_samples_split=1, random_state=2).fit(d_train, l_train)

svc_result =      svc.predict(d_test)
rbf_svc_result =  rbf_svc.predict(d_test)
poly_svc_result = poly_svc.predict(d_test)
lin_svc_result =  lin_svc.predict(d_test)
knclf_result =    knclf.predict(d_test)
rfc_result =      rfc.predict(d_test)

print("SVC prediction accuracy = {0:5.1f}%".format(100.0 * svc.score(d_test, l_test)))
print("RBF SVC prediction accuracy = {0:5.1f}%".format(100.0 * rbf_svc.score(d_test, l_test)))
print("Polynomial SVC prediction accuracy = {0:5.1f}%".format(100.0 * poly_svc.score(d_test, l_test)))
print("Linear SVC prediction accuracy = {0:5.1f}%".format(100.0 * lin_svc.score(d_test, l_test)))
print("K-Neighbor prediction accuracy = {0:5.1f}%".format(100.0 * knclf.score(d_test, l_test)))
print("RFC prediction accuracy = {0:5.1f}%".format(100.0 * rfc.score(d_test, l_test)))


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Clean the Data by removing entries that correspond to Bad Spills
There are several criteria by which we classify a 5-second spill of data to be un-analyzable or irregular. Let's exclude those.
Can you write Python code for it?

try:
    # See if this has already been populated
    bad_spill_set
except:
    # Get the set of bad (non-sense) spills for this data
    bad_spill_set = get_bad_spills(server, schema)

# Get rid of entries that correspond to bad spills
data_df = data_df.query('spillID not in bad_spill_set')


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Clean the Data some more
Feature Exclusion

Pivot the dataframe
Eliminate rows with missing data
Inspect the means and standard deviations
Identify the zero standard deviation features, as they're not helpful
Remove them from our pivoted dataframe and our scaler_df dataset
Can you write Python code for it?

# We want to see our scalerNames as column indexes
pivoted_df = data_df.pivot('spillID', 'name', 'value'); pivoted_df.head()


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Create a Confusion histogram to examine prediction performance
Can you write Python code for it?

def confusion(labels, results, names):
    plt.figure(figsize=(10, 10))

    # Make a 2D histogram from the test and result arrays
    pts, xe, ye = np.histogram2d(labels.astype(int), results.astype(int), bins=len(names))

    # For simplicity we create a new DataFrame
    pd_pts = pd.DataFrame(np.flipud(pts.astype(int)), index=np.flipud(names), columns=names )

    # Display heatmap and add decorations
    hm = sns.heatmap(pd_pts, annot=True, fmt="d", cbar=False)

    _, ylabels = plt.xticks()
    _, xlabels = plt.yticks()
    plt.setp(xlabels, rotation=45)
    plt.setp(ylabels, rotation=45)
    plt.xlabel("Actual", size=22)
    plt.ylabel("Prediction", size=22)

    return pts

def per_target_accuracy(hist2d_pts, names):

    for i in range(len(names)):
        rowsum = np.sum(hist2d_pts.T[i])
        if rowsum>0:
            print names[i] + ":   \t" + str(round((hist2d_pts[i][i] / np.sum(hist2d_pts.T[i]))*100,2)) + "%"
        else:
            print names[i] + ":   \tN/A"

# Define names for the target positions
names = ['Hydrogen','Empty','Deuterium','None','Carbon','Iron','Tungsten']

pts = confusion(l_test, result, names)
per_target_accuracy(pts, names)


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Cross-Validation

Let us exhaustively check how robust the RFC is
We do this by cross-validation using the whole data set
Randomly split the data into 10 folds using KFold
Can you write Python code for it?

kf_total = cross_validation.StratifiedKFold(labels_revised, n_folds=10, shuffle=True, random_state=4)
rfc_scores = cross_validation.cross_val_score(rfc, data_reduced_scaled, labels_revised, cv=kf_total)

print "RFC Scores: ", rfc_scores
print "Accuracy: %0.2f (+/- %0.2f)" % (rfc_scores.mean(), rfc_scores.std() * 2)


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Domain knowledge: "Empty" and "None" Similarity

"Empty" is basically a hollow tin can
"None" is no target at all
Used to study background signals
Will be naturally very, very similar -- and we see above that they get mistaken for each other
We can collapse these into one category
Can you write Python code for it?

def relabel(label_array):
    # Collapse target position 4 and 2 both into category 2
    # Then shift over the rest
    label_array_revised = label_array.copy()
    label_array_revised[label_array_revised == 4] = 2
    label_array_revised[label_array_revised == 5] = 4
    label_array_revised[label_array_revised == 6] = 5
    label_array_revised[label_array_revised == 7] = 6

    return label_array_revised

# Call the new re-labelling function
#    and modify the names array
labels_revised = relabel(labels)
names = ['Hydrogen','Empty/None','Deuterium','Carbon','Iron','Tungsten']

d_train, d_test, l_train, l_test \
    = cross_validation.train_test_split(data_scaled, labels_revised, test_size=0.33, random_state=5)

rfc = RandomForestClassifier(n_estimators=100, max_depth=None, max_features='sqrt',
                             min_samples_split=1, random_state=2).fit(d_train, l_train)

result = rfc.predict(d_test)
print("RF prediction accuracy = {0:5.1f}%\n".format(100.0 * rfc.score(d_test, l_test)))

pts = confusion(l_test, result, names)   # Show confusion matrix
per_target_accuracy(pts, names)          # Print per-target accuracy


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Explore Visually
Can you write Python code for it?

# How many features are we working with here?
value_names = data_df.name.unique()
print len(value_names), "unique features"

# Take a look at the first eight
fig, axes = plt.subplots(nrows=4, ncols=2, figsize=(12,20))
index = 0
for value_name in value_names[:8]:
    subset_df = data_df[(data_df.name == value_name)]
    axis = axes[floor(index/2),index%2]
    sns.violinplot(subset_df.value,              # We want to inspect the feature
                   subset_df.targetPos,          #   the distributions, and how they
                   color="Paired",               #   differ for each target position.
                   bw=0.7,                       # The side-by-side, normalized nature
                   ax=axis)                      #   of violin plots are ideal for this.
    axis.set_title(value_name)
    axis.set_xlabel('Target Position', fontsize=20)
    axis.set_ylabel('Value', fontsize=20)
    index = index + 1

fig.subplots_adjust(hspace=0.4, wspace=0.4)
plt.show()


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Explore the Data
Can you write Python code for it?

data_df.head()                   # Peek at the data

data_df.info()                # ...and investigate data types.


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Feature Scaling

Over time, these features may drift a little due to many, many effects
Hopefully, the relative differences are key to differentiating targets
Can you write Python code for it?

data = engineered_df.values

scale = StandardScaler().fit(data)

data_scaled = scale.transform(data)


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Identify zero standard deviation features
Can you write Python code for it?

# It's sufficient to say that if the standard deviation is 0, then it's certainly not useful
zero_std_series = (pivoted_df.describe().ix['std'] == 0)
# Get an array of all the features with zero standard deviations
zero_std_features = zero_std_series[zero_std_series == True].index.values; zero_std_features


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Most Valuable Features

Rank our top most valuable features
See if any make obvious sense
Perhaps use only useful features in future iterations
Can you write Python code for it?

features = full_df.drop(['S:G2SEM','targetPos'], axis=1).columns.values
importances = rfc.feature_importances_
indices = np.argsort(importances)[::-1]

# Print the feature ranking
print("Feature ranking:")

useful_feature_list = []
for f in range(20):
    print("%d. Feature '%s' (%f)" % (f + 1, features[indices[f]], importances[indices[f]]))
    useful_feature_list.append(features[indices[f]])


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Pickle our trained RFC for use in the field
Can you write Python code for it?

from sklearn.externals import joblib

# Train using our full data set
rfc_final = RandomForestClassifier(n_estimators=100, max_depth=None, max_features=None,
                             min_samples_split=1, random_state=2).fit(data_reduced_scaled, labels_revised)

data_reduced[:5]

test_df = pd.read_csv('testrun.csv', index_col='spillID')

test_labels = test_df.values[:,-1]
test_labels = relabel(test_labels)

test_data = test_df[useful_feature_list]

result = rfc_final.predict(test_data)
print("RF prediction accuracy = {0:5.1f}%".format(100.0 * rfc_final.score(test_data, test_labels)))

pts = confusion(test_labels, result, names)
per_target_accuracy(pts, names)

from sklearn.metrics import confusion_matrix

print test_labels
print result

confusion_matrix(test_labels, result)

# Write our pickled classifier to file`
rfc_pickle_name = "models/target_rfc_roadset62.pkl"

if os.path.exists(rfc_pickle_name):
    os.remove(rfc_pickle_name)

joblib.dump(rfc_final, rfc_pickle_name)

data_reduced.shape
reduced_df.columns

useful_feature_list


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Plotting, Graphics
Can you write Python code for it?

import matplotlib.pyplot as plt     # For plotting some distributions
import seaborn as sns               # For easy, pretty plotting
sns.set_style("darkgrid")
sns.set_context("talk", font_scale=1.4)

sys.path.append('./modules')
from spill import get_bad_spills


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Remove these from our analysis
Can you write Python code for it?

# Get rid of these features
_ = pivoted_df.drop(zero_std_features, axis=1, inplace=True)


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Split our data up into training and test sets
Can you write Python code for it?

d_train, d_test, l_train, l_test \
    = cross_validation.train_test_split(data_scaled, labels, test_size=0.33, random_state=2)


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Tools used
Can you write Python code for it?

import MySQLdb as mdb               # Raw data source is MySQL
import pandas as pd                 # Workhorse data management tool
import numpy as np                  # For matrices, arrays, matrix math, and nan's
from math import floor


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Train on training data
Can you write Python code for it?

rfc.fit(d_train, l_train)

result = rfc.predict(d_test)

print("RF prediction accuracy = {0:5.1f}%".format(100.0 * rfc.score(d_test, l_test)))


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Train on useful, vetted, approved features

Time-invariant (always been read out, always will be read out)for the experiment
Ensures that our RFC can be used on past and future data
Can you write Python code for it?

reduced_df = pd.DataFrame( (full_df[useful_feature_list].values / full_df[['S:G2SEM']].values) * 5000000000000.0,
                              columns=useful_feature_list )
data_reduced = reduced_df.values
scale_reduced = StandardScaler().fit(data_reduced)
data_reduced_scaled = scale_reduced.transform(data_reduced)

# We've reduced the number of features and normalized them all to beam intensity
# Let's see how that's affected our predictor

d_train, d_test, l_train, l_test \
    = cross_validation.train_test_split(data_reduced_scaled, labels_revised, test_size=0.33, random_state=6)
rfc = RandomForestClassifier(n_estimators=100, max_depth=None, max_features='sqrt',
                             min_samples_split=1, random_state=2).fit(d_train, l_train)
result = rfc.predict(d_test)
print("RF prediction accuracy = {0:5.1f}%".format(100.0 * rfc_reduced.score(d_test, l_test)))

pts = confusion(l_test, result, names)
per_target_accuracy(pts, names)


I am working on the file "Insight-Demo-Roadset62.ipynb".
The first task is:
Wrangling the data into shape
Reading Data from MySQL
Can you write Python code for it?

server = 'e906-db3.fnal.gov'                         # Source MySQL server
schema = 'merged_roadset62_R004_V005'                # Source MySQL schema
analysis_schema = 'user_dannowitz_test1'   # A schema name for temporary storage
analysis_table = 'target_analysis'                   # A table name for that schema

# Aggregate data into our analysis schema and table.
# Table defined here:
create_query = """
               CREATE TABLE IF NOT EXISTS %s.%s
               (
                   spillID MEDIUMINT NOT NULL,
                   name VARCHAR(64),
                   value DOUBLE NOT NULL,
                   targetPos INT NOT NULL
                )"""

# Here is one MySQL query that will grab all the data we want
#     Most of this requires a bit of domain expertise to understand
#     as it's specific to our experiment's data structure
scaler_query =  """
                INSERT INTO %s.%s
                ### Get data from our Scaler table, along with the target position.
                ### This contains features from our triggering systems (data taking rates)

                SELECT s.spillID, scalerName AS `name`, value, targetPos
                FROM Scaler
                INNER JOIN Spill s              # Source of targetPos
                    USING(spillID)
                WHERE scalerName IS NOT NULL AND
                      s.spillID NOT BETWEEN 409000 AND 430000 AND
                      s.spillID NOT BETWEEN 416709 AND 424180 AND
                      s.spillID NOT BETWEEN 482574 AND 484924 AND
                      spillType='EOS'
                """

beam_query = """
             INSERT INTO %s.%s
             ### Get data from our Beam table, along with the target position
             ### This contains features from our proton beam and radiation monitors

             SELECT s.spillID, name, value, targetPos
             FROM Beam
             INNER JOIN Spill s              # Source of targetPos
                 USING(spillID)
             WHERE name IS NOT NULL AND
  ...


I am working on the file "Inspection - Regression.ipynb".
The first task is:
A zoom-in on the training pan range for my talk's slides.
Can you write Python code for it?

fig, ax = plt.subplots(1, 1, figsize=(16,8))
ax.hist(np.rad2deg(Ptr), bins=32)
ax.set_xlim(-100,100)
ax.axes.xaxis.set_major_formatter(mpl.ticker.FuncFormatter(lambda x, p: '{:n}°'.format(x)))
fig.suptitle('Tosato IDIAP Pan', fontsize=20, y=1.0)
lbplt.zoomtext(fig)
lbplt.savefig(fig, 'idiap_pan_hist', transparent=False)


I am working on the file "Inspection - Regression.ipynb".
The first task is:
Also, look at how ridiculously small the original images are on average:
Can you write Python code for it?


I am working on the file "Inspection - Regression.ipynb".
The first task is:
Clearly, either they have been mirrored already, or subsampled in order to have a symmetric distribution. Thus, I won't apply any flipping augmentation here to be on the safe side.
A closer look at the covered angle ranges
Can you write Python code for it?

def minmax(x): return np.min(x), np.max(x)
rangefmt = '[{:7.2f}°,{:6.2f}°]'

lbu.printnow('Training: Pan range: '+rangefmt+', Tilt range: '+rangefmt+', Roll range: '+rangefmt,
    *minmax(np.rad2deg(Ptr)) + minmax(np.rad2deg(Ttr)) + minmax(np.rad2deg(Rtr))
)
lbu.printnow('Testing:  Pan range: '+rangefmt+', Tilt range: '+rangefmt+', Roll range: '+rangefmt,
    *minmax(np.rad2deg(Pte)) + minmax(np.rad2deg(Tte)) + minmax(np.rad2deg(Rte))
)

def inrange(y, lim, name):
    lbu.printnow('{:.2%} of {} in [-{lim:n}°, {lim:n}°]', np.mean((-lim < y) & (y < lim)), name, lim=round(np.rad2deg(lim)))
def inrange2(y, lim1, lim2, name):
    lbu.printnow('{:.2%} of {} in [{:n}°, {:n}°]', np.mean((lim1 < y) & (y < lim2)), name, round(np.rad2deg(lim1)), round(np.rad2deg(lim2)))

inrange(Ptr, np.deg2rad(60), 'pan')

inrange(Ptr, np.deg2rad(56), 'pan')

inrange2(Ttr, np.deg2rad(-60), np.deg2rad(15), 'tilt')

inrange2(Ttr, np.deg2rad(-30), np.deg2rad(10), 'tilt')

inrange2(Rtr, np.deg2rad(-30), np.deg2rad(30), 'roll')

inrange2(Rtr, -0.2, 0.5, 'roll')


I am working on the file "Inspection - Regression.ipynb".
The first task is:
Looks very similarly useless, so no need to investigate further.
TownCentre Head Images (Benfold)
Can you write Python code for it?

def load_towncentre(where):
    panre = recompile('pan = ([+-]?\d+\.\d+)\n')
    valre = recompile('valid = ([01])\n')
    angles = []
    images = []
    names = []
    for father in listdir(where):
      try:
        for son in listdir(pjoin(where, father)):
            if not son.endswith('.txt'):
                continue

            lpan, lval = open(pjoin(where, father, son)).readlines()
            if int(valre.match(lval).group(1)) == 0:
                continue

            angles.append(float(panre.match(lpan).group(1)))
            # Now search for the corresponding filename, unfortunately, it has more numbers encoded...
            fnames = [f for f in listdir(pjoin(where, father)) if f.startswith(son.split('.')[0]) and not f.endswith('.txt')]
            assert len(fnames) == 1, "lolwut"
            names.append(fnames[0])
            images.append(cv2.imread(pjoin(where, father, fnames[0]), flags=cv2.IMREAD_COLOR))
      except NotADirectoryError:
        pass

    return images, angles, names

bbtc_img, bbtc_a, bbtc_n = load_towncentre('data/TownCentreHeadImages')
bbtc_a_n = [(a + 360*2) % 360 for a in bbtc_a]

ws = [i.shape[1] for i in bbtc_img]
hs = [i.shape[0] for i in bbtc_img]
ar = [h/w for w,h in zip(ws, hs)]
fig, axes = plt.subplots(2, 2, figsize=(16,8))
axes[0,0].hist(bbtc_a_n, bins=36)
axes[0,1].hist(ar, bins=32)
axes[1,0].hist(ws, bins=40)
axes[1,1].hist(hs, bins=40)
axes[0,0].set_title('Trainset gaze angles')
axes[0,1].set_title('Trainset aspect ratio [{:.2f}, {:.2f}]'.format(min(ar), max(ar)))
axes[1,0].set_title('Trainset width [{}, {}]'.format(min(ws), max(ws)))
axes[1,1].set_title('Trainset height [{}, {}]'.format(min(hs), max(hs)))
axes[0,0].set_xlim(0, 360)
fig.suptitle('B.Benfold TownCentre', fontsize=20)

fig, ax = plt.subplots(1, 1, figsize=(16,8))
ax.hist(bbtc_a_n, bins=36*2)
ax.set_xlim(0,360)
ax.axes.xaxis.set_major_formatter(mpl.ticker.FuncFormatter(lambda x, p: '{:n}°'.format(x)))
fig.suptitle('TownCentre Pan', fontsize=20, y=1.0)
lbplt.zoomtext(fig)
lbplt.savefig(fig,...


I am working on the file "Inspection - Regression.ipynb".
The first task is:
Shopping Full
Can you write Python code for it?

def extract_vector(arr, col, dtype=np.float32):
    N = len(arr)
    vec = np.empty(N, dtype=dtype)  # NOTE: DO NOT TRUST AUTO DTYPE EXTRACTION!
    for i in range(N):
        vec[i] = arr[i][col][0,0]
    return vec

def read_tosato_shopping(folder):
    mat = spio.loadmat(pjoin(folder, 'or_label.mat'))

    def loadall(traintest):
        gaze = extract_vector(mat['or_label_' + traintest][0], 0)
        xc = extract_vector(mat['or_label_' + traintest][0], 1)
        yc = extract_vector(mat['or_label_' + traintest][0], 2)
        size = extract_vector(mat['or_label_' + traintest][0], 3)
        return gaze, xc, yc, size

    return loadall('train'), loadall('test')

tosato_shopping = read_tosato_shopping('data/CAVIARShoppingCenterFull')
(ystr, xc_tr, yc_tr, size_tr), (yste, xc_te, yc_te, size_te) = tosato_shopping

lbu.printnow('{} files in Training.', len(ystr))
lbu.printnow('{} files in Testing.', len(yste))


I am working on the file "Instacart_EDA_DataStory_Assortment.ipynb".
The first task is:
Extra EDA

Below are a few macro-level analyses of order behavior
Can you write Python code for it?

# Total Number of Instances of each Category in Prior Order Set
# e.g. order_id 2 has 5 'pantry' items, order_id 3 has 0 'pantry' items, etc.

fig, ax = plt.subplots(figsize=(12, 5))
market_basket_prior.groupby(['department']).size().sort_values(ascending=False
                                                   ).plot(kind='bar',
                                                          fontsize=15)
ax.ticklabel_format(axis='y', style='plain')
plt.xticks(ha='right', rotation=55)
plt.title("Categories by Total Instances in Orders", fontsize=18);

dept_totals = pd.DataFrame(market_basket_prior.groupby(['department']
                                                      ).size().sort_values(ascending=False))
dept_totals.columns = ['Totals']
dept_totals.head(10)

# Popular products by total order count

fig, ax = plt.subplots(figsize=(12, 5))
market_basket_prior.groupby(['product_name']).size().sort_values(ascending=False
                                                   )[:20].plot(kind='bar',
                                                               fontsize=15)
ax.ticklabel_format(axis='y', style='plain')
plt.xticks(ha='right', rotation=55)
plt.title("Products by Total Instances in Orders\n Long live Bananas", fontsize=18);

product_totals = pd.DataFrame(market_basket_prior.groupby(['product_name']).size().sort_values(ascending=False))
product_totals.columns = ['Totals']
product_totals.head(10)


I am working on the file "Instacart_EDA_DataStory_Assortment.ipynb".
The first task is:
Market Basket vs. Order Meta Data
Now that we have centrally located product data, let's create a master 'market basket' dataframe that includes:

order assortment
orde contents

I am focusing on the composition of orders, and am setting aside order 'meta data', i.e.

order_dow,
order_hour_of_day,
etc.

*----
Can you write Python code for it?

market_basket_prior = prior_set.merge(products_df, on='product_id')
market_basket_prior = market_basket_prior.sort_values(by=['order_id', 'add_to_cart_order'])
market_basket_prior.head(10)

reordered     = market_basket_prior['reordered'].value_counts()[1]
not_reordered = market_basket_prior['reordered'].value_counts()[0]
pct_reordered = str(round(reordered / float(reordered + not_reordered)*100, 2)) + '%'
print 'Instances of Reordered Items:      ' + str(market_basket_prior.reordered.sum())
print 'Total Number of Items Purchased:   ' + str(market_basket_prior.shape[0])
print '\nUnique instances of a previously ordered product being ordered:' + '\nReorder Rate: ' + pct_reordered + '\n'

# Let's refine by excluding the first order, which arguably dilutes the reorder rate

excluding_first_order = orders[(orders['order_number'] > 1) & (orders['eval_set'] == 'prior')].sort_values('user_id')
excluding_first_order = excluding_first_order.merge(prior_set, on='order_id')
excluding_first_order

reordered     = excluding_first_order['reordered'].value_counts()[1]
not_reordered = excluding_first_order['reordered'].value_counts()[0]
pct_reordered = str(round(reordered / float(reordered + not_reordered)*100, 2)) + '%'
print 'Instances of Reordered Items:      ' + str(excluding_first_order.reordered.sum())
print 'Total Number of Items Purchased:   ' + str(excluding_first_order.shape[0])
print '\nUnique instances of a previously ordered product being ordered:' + '\nReorder Rate: ' + pct_reordered + '\n'


I am working on the file "Instacart_EDA_DataStory_Engagement.ipynb".
The first task is:
As expected, the reorder rate of 32.94% for orders 2 & 3 is significantly lower than the overall reorder rate of 58.97%
Can you write Python code for it?

# Let's compare against the other segments

pct_list = []
for segment in segments:
    reordered         = segments[segment]['reordered'].value_counts()[1]
    not_reordered     = segments[segment]['reordered'].value_counts()[0]
    pct_reordered     = str(round(reordered / float(reordered + not_reordered)*100, 2)) + '%'
    pct_list.append(round(reordered / float(reordered + not_reordered)*100, 2))

pct_name = []
for f in segments.keys():
    pct_name.append(f)
pct_list.sort()
pct_name.sort()
i = 0

for f in pct_list:
    print str(pct_name[i]), '\nUnique instances of a previously ordered product being ordered:\
                            ' + '\nReorder Rate: ' + str(f) + '%\n'
    i+=1

#Let's visualize the differences
f, ax = plt.subplots(figsize=(14,5))

ax.bar(range(6), pct_list)
plt.ylabel('Reorder Rate', fontsize=15)
plt.xlabel('Segment', fontsize=15)
plt.xticks(range(6), pct_name, ha='right', rotation=55, fontsize=15)
plt.title("New vs Seasoned \nCustomer Reorder Rate", fontsize=18);


I am working on the file "Instacart_EDA_DataStory_Engagement.ipynb".
The first task is:
Now let's look at the distribution of Product Count per Order
Can you write Python code for it?

# Plotting distribution of product counts for different segments.
first_orders    = segments['orders_02_03_new']['order_id'].value_counts()
intermediate    = segments['orders_07_08_intermediate']['order_id'].value_counts()
almost_seasoned = segments['orders_10_11_almostseasoned']['order_id'].value_counts()
seasoned        = segments['orders_20_21_seasoned']['order_id'].value_counts()
pro             = segments['orders_40_41_pro']['order_id'].value_counts()
superpro        = segments['orders_80_81_superpro']['order_id'].value_counts()

# Let's plot product count distribution for 'new' customers

f, ax = plt.subplots(figsize=(14,3))
plt.hist(first_orders, bins=np.arange(128))
plt.ylabel('Frequency', fontsize=15)
plt.xlabel('Product Count', fontsize=15)
plt.xticks(np.arange(0,130,5), rotation=45)
plt.title("Frequency: Product Count\nFirst Orders", fontsize=18);

plt.figure(figsize=(14,3))
intermediate.value_counts()[:20].plot(kind='bar')
plt.ylabel('Frequency', fontsize=15)
plt.xlabel('Order Count', fontsize=15)
plt.title("Frequency: Product Count\nIntermediate", fontsize=18);

plt.figure(figsize=(14,3))
pro.value_counts()[:20].plot(kind='bar')
plt.ylabel('Frequency', fontsize=15)
plt.xlabel('Order Count', fontsize=15)
plt.title("Frequency: Product Count\nPro", fontsize=18);

# Let's plot the top 20 for a closer look
plt.figure(figsize=(14,3))
superpro.value_counts()[:20].plot(kind='bar')
plt.ylabel('Frequency', fontsize=15)
plt.xlabel('Order Count', fontsize=15)
plt.title("Frequency: Product Count\nSuperpro", fontsize=18);


I am working on the file "Instacart_EDA_DataStory_Engagement.ipynb".
The first task is:
Of Note:

There's actually a remarkable consistency from order_number 2 to order_number 3
Can you write Python code for it?

# Customer Order Count, segmented by eval_set

order_count_bySet = orders.loc[:,('user_id', 'eval_set')].groupby(['user_id', 'eval_set']).size()
order_count_bySet.unstack().fillna('--')[:20]


I am working on the file "Instacart_EDA_DataStory_Engagement.ipynb".
The first task is:
Of Note:

cyclicality
correlation between product_id count (bars) and days since prior order (line)
Can you write Python code for it?

data2.sort_index() # order_number and corresponding products count


I am working on the file "Instalando_python.ipynb".
The first task is:
Once descargado, Instala Python usando un terminal Linux vía el comando:
$ bash FILENAME
Tendrás que acetar los términos de la licencia escribiendo yes y presionando ENTER/RETURN. Luego tendrás que presionar ENTER/RETURN nuevamente para instalar el programa en el directorio que se muestra por omisión. No obstante, i así lo deseas, puedes elegir otro directorio de instalación
LOS SIGUIENTES VIDEOS MUESTRAN EL PROCESO
Can you write Python code for it?

from IPython.display import YouTubeVideo


I am working on the file "Instalando_python.ipynb".
The first task is:
Instalando Anaconda en Linux
Can you write Python code for it?


I am working on the file "Instalando_python.ipynb".
The first task is:
Instalando Canopy en Linux
Can you write Python code for it?


I am working on the file "Instalando_python.ipynb".
The first task is:
Opción 2: Descarga Enthought Canopy siguiendo el enlace que se muestra.
<b>Alternativamente, puedes seguir el enlace que se muestra a continuación. En caso que nada se muestre, entonces sigue el enlace mencionado anteriormente.</b>
Can you write Python code for it?

HTML('<iframe src=https://store.enthought.com/?useformat=mobile width=700 height=350></iframe>')


I am working on the file "Installation - setuptools.ipynb".
The first task is:
Now if the user doesn't have the right version of six installed, it should be installed for them via PyPi when they run python setup.py develop.
Script/Command Generation
Sometimes you have a standalone application that you want a user to be able to use from their commandline. Regardless of the OS, SetupTools has got you covered here too with the entry_points argument.
Although entry_points can also be used to extend your application with plugins, this section will only focus on creating shell scripts, like the Nengo GUI, with another argument to setup().
Can you write Python code for it?

from setuptools import setup, find_packages

setup(
    name='bughandler',
    packages=find_packages(),
    entry_points={
        'console_scripts': [
            'spray_bug = killer.kill:spray',
            'shoot_bug = killer.kill:shoot',
        ]
    }
)


I am working on the file "Installation - setuptools.ipynb".
The first task is:
This should have the same effect. Obviously, if you really want your import statement to do something useful, you're going to need to add some content to kill.py and access it via import killer.kill.
Note: Usually your name and your root package have the same string (for example, in the Nengo repository), but I avoided that case here to reduce confusion.
Requirements
Let's say bughandler depends the PyPi package six any version after 1.9. How can you make sure that the user has that package installed in the least instrusive way? SetupTools has you covered with the install_requires argument to the setup function:
Can you write Python code for it?

from setuptools import setup, find_packages

setup(
    name='bughandler',
    packages=find_packages(),
    install_requires=[
        "six>=1.9",
    ],
)


I am working on the file "InstructorNotebooks/Iris_DataSet_Instructor.ipynb".
The first task is:
Import the dataset
Import the dataset and store it to a variable called iris. This dataset is similar to a python dictionary, with the keys: ['DESCR', 'target_names', 'target', 'data', 'feature_names']
The data features are stored in iris.data, where each row is an example from a single flow, and each column is a single feature. The feature names are stored in iris.feature_names. Labels are stored as the numbers 0, 1, or 2 in iris.target, and the names of these labels are in iris.target_names.
Can you write Python code for it?

# Import some data to play with
iris = datasets.load_iris()

# List the data keys
print('Keys: ' + str(iris.keys()))
print('Label names: ' + str(iris.target_names))
print('Feature names: ' + str(iris.feature_names))
print('')

# Store the labels (y), label names, features (X), and feature names
y = iris.target       # Labels are stored in y as numbers
labelNames = iris.target_names # Species names corresponding to labels 0, 1, and 2
X = iris.data
featureNames = iris.feature_names

# Show the first five examples
print(iris.data[1:5,:])


I am working on the file "InstructorNotebooks/Iris_DataSet_Instructor.ipynb".
The first task is:
Make your own plot
Below, try making your own plots. First, modify the previous code to create a similar plot, showing the petal width vs the petal length. You can start by copying and pasting the previous block of code to the cell below, and modifying it to work.
How is the data arranged differently? Do you think these additional features would be helpful in determining to which species of iris a new plant should be categorized?
What about plotting other feature combinations, like petal length vs sepal length?
Once you've plotted the data several different ways, think about how you would predict the species of a new iris plant, given only the length and width of its sepals and petals.
Can you write Python code for it?

# Put your code here!

# Plot the data

# Petal length and width
X_petal = X[:,2:]
# Get the minimum and maximum values with an additional 0.5 border
x_min, x_max = X_petal[:, 0].min() - .5, X_petal[:, 0].max() + .5
y_min, y_max = X_petal[:, 1].min() - .5, X_petal[:, 1].max() + .5

plt.figure(figsize=(8, 6))

# Plot the training points
plt.scatter(X_petal[:, 0], X_petal[:, 1], c=y, cmap=cmap_bold)
plt.xlabel('Petal length (cm)')
plt.ylabel('Petal width (cm)')
plt.title('Petal width vs length')

# Set the plot limits
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)

plt.legend(labelList, labelNames)

plt.show()


I am working on the file "InstructorNotebooks/Iris_DataSet_Instructor.ipynb".
The first task is:
Training and Testing Sets
In order to evaluate our data properly, we need to divide our dataset into training and testing sets.
Training Set
A portion of the data, usually a majority, used to train a machine learning classifier. These are the examples that the computer will learn in order to try to predict data labels.
Testing Set
A portion of the data, smaller than the training set (usually about 30%), used to test the accuracy of the machine learning classifier. The computer does not "see" this data while learning, but tries to guess the data labels. We can then determine the accuracy of our method by determining how many examples it got correct.
Creating training and testing sets
Below, we create a training and testing set from the iris dataset using using the train_test_split() function.
Can you write Python code for it?

from sklearn import cross_validation

X_train, X_test, y_train, y_test = cross_validation.train_test_split(X, y, test_size=0.3)
print('Original dataset size: ' + str(X.shape))
print('Training dataset size: ' + str(X_train.shape))
print('Test dataset size: ' + str(X_test.shape))


I am working on the file "InstructorNotebooks/Iris_DataSet_Instructor.ipynb".
The first task is:
Visualizing the data
Visualizing the data can help us better understand the data and make use of it. The following block of code will create a plot of sepal length (x-axis) vs sepal width (y-axis). The colors of the datapoints correspond to the labeled species of iris for that example.
After plotting, look at the data. What do you notice about the way it is arranged?
Can you write Python code for it?

# Plot the data

# Sepal length and width
X_sepal = X[:,:2]
# Get the minimum and maximum values with an additional 0.5 border
x_min, x_max = X_sepal[:, 0].min() - .5, X_sepal[:, 0].max() + .5
y_min, y_max = X_sepal[:, 1].min() - .5, X_sepal[:, 1].max() + .5

plt.figure(figsize=(8, 6))

# Plot the training points
plt.scatter(X_sepal[:, 0], X_sepal[:, 1], c=y, cmap=cmap_bold)
plt.xlabel('Sepal length (cm)')
plt.ylabel('Sepal width (cm)')
plt.title('Sepal width vs length')

# Set the plot limits
plt.xlim(x_min, x_max)
plt.ylim(y_min, y_max)

plt.legend(labelList, labelNames)

plt.show()


I am working on the file "Integracion numérica.ipynb".
The first task is:
Integraciónd de ODEs de segundo orden
La idea es convertir una ODE de segundo orden en una ODE de primer orden, en un espacio de mayor dimensión. Para esto, reescribimos la ecuación
$$
y''(t)= F(t,y(t))
$$
como un sistema de ecuaciones acopladas
$$
y'(t)= v(t)
$$
$$
v'(t)= F(t,y(t))
$$
introduciendo la "variable auxiliar" $v(t)$. Nótese que ahora se requieren dos condiciones iniciales, una para $y(t_0)$ y otra para $y'(t_0)=v(t_0)$.
Resolvamos la ecuación para el caso lineal
$$
y''(t)=- w^2 y(t)
$$
que se reformula como
$$
y'(t)=v(t)    \;\; v(t)=- w^2 y(t)
$$
Las soluciones son de la forma $y(t)=A \cos(w t) + B \sin(w t)$, $v(t)=-A \sin(w t) + B \cos(w t)$, con $A$ y $B$ dos parámetros a fijar a partir de las condiciones iniciales. Si fijamos $y(0)=1$, $y'(0)=0\Rightarrow A=1$, $B=0$ y por lo tanto,
$$
y(t)=\cos(w t)
$$
Can you write Python code for it?

# ODE de segundo  orden

# Resuelve la ecuación  y''= -w^2 y   con factor de decaimiento alpha=1.

# Resuelve la ecuación  y''= -w^2 y   con factor de decaimiento w^2=1.

res = integr.solve_ivp(lambda t,y: [y[1],-y[0]], np.array([0.,12.]),np.array([1.,0.]))

plt.plot(res.t,np.cos(res.t),label="exacto, w^2=1",c="red")
plt.scatter(res.t,res.y[0],label="numerico, w^2=1",c="red")

# Resuelve la ecuación  y''= -w^2 y   con factor de decaimiento w^2=2.

res = integr.solve_ivp(lambda t,y: [y[1],-2.*y[0]], np.array([0.,12.]),np.array([1.,0.]))

plt.plot(res.t,np.cos(np.sqrt(2)*res.t),label="exacto, w^2=2",c="blue")
plt.scatter(res.t,res.y[0],label="numerico, w^2=2",c="blue")

# Resuelve la ecuación  y''= -w^2 y   con factor de decaimiento w^2=16.

res = integr.solve_ivp(lambda t,y: [y[1],-16.*y[0]], np.array([0.,12.]),np.array([1.,0.]))

plt.plot(res.t,np.cos(np.sqrt(16)*res.t),label="exacto, w^2=16",c="green")
plt.scatter(res.t,res.y[0],label="numerico, w^2=16",c="green")

plt.legend()
plt.show()

# Para mejorar el resultado, definir la integración sobre más puntos vía el parámetro t_eval:

# Resuelve la ecuación  y''= -w^2 y   con factor de decaimiento w^2=1.

res = integr.solve_ivp(lambda t,y: [y[1],-y[0]], np.array([0.,12.]),np.array([1.,0.]),t_eval=np.linspace(0,12,100))

plt.plot(res.t,np.cos(res.t),label="exacto, w^2=1",c="red")
plt.scatter(res.t,res.y[0],label="numerico, w^2=1",c="red")

# Resuelve la ecuación  y''= -w^2 y   con factor de decaimiento w^2=2.

res = integr.solve_ivp(lambda t,y: [y[1],-2.*y[0]], np.array([0.,12.]),np.array([1.,0.]),t_eval=np.linspace(0,12,100))

plt.plot(res.t,np.cos(np.sqrt(2)*res.t),label="exacto, w^2=2",c="blue")
plt.scatter(res.t,res.y[0],label="numerico, w^2=2",c="blue")

# Resuelve la ecuación  y''= -w^2 y   con factor de decaimiento w^2=16.

res = integr.solve_ivp(lambda t,y: [y[1],-16.*y[0]], np.array([0.,12.]),np.array([1.,0.]),t_eval=np.linspace(0,12,100))

plt.plot(res.t,np.cos(np.sqrt(16)*res.t),label="exacto,...


I am working on the file "Integracion numérica.ipynb".
The first task is:
Integrales impropias
A veces, nos interesa evaluar integrales con límites en $\pm\infty$. Para esto, usamos la constantes de numpy
$np.inf$:
integr.quad(lambda x:np.exp(-x**2)/np.sqrt(np.pi),-np.inf,np.inf)
que devuelve
(1.0, 8.012979937383924e-09)
Integrales multidimensionales
Para evaluar integrales en más de una variable, podemos trabajar (usando el teorema de Fubini) con integrales iteradas:
integr.quad(lambda y:integr.quad(lambda x,y:np.exp(-x**2-y**2)/np.pi,-1,1, args=(y,))[0],-1,1)
(0.7101446264380782, 7.88418915085385e-15)
Sin embargo, esto tiene algunos problemas tanto en eficiencia y convergencia, como en la estimación del error de integración. Una mejor opción es usar la rutina "dblquad"
integrate.dblquad(lambda x,y:np.exp(-x**2-y**2)/np.pi, -1, 1, lambda x: -1, lambda x: 1)
Otra alternativa, útil en casos de integrales múltiples es usar "nquad"
integr.nquad(lambda x,y:np.exp(-x**2-y**2)/np.pi, ranges=[[-1.,1.],[-1.,1.]])
Can you write Python code for it?

res1 = integr.quad(lambda y:integr.quad(lambda x,y:np.exp(-x**2-y**2)/np.pi,-1,1, args=(y,))[0],-1,1)

res2 = integr.dblquad(lambda x,y:np.exp(-x**2-y**2)/np.pi, -1, 1, lambda x: -1, lambda x: 1)

res3 = integr.nquad(lambda x,y:np.exp(-x**2-y**2)/np.pi, ranges=[[-1.,1.],[-1.,1.]])
print(res1)
print(res2)
print(res3)


I am working on the file "Integration.ipynb".
The first task is:
Evidently we created a fourth order accurate method by extrapolating a second order accurate method.
This process can even be applied recursively!
Gauss Quadrature
Recall the Legendre polynomials from the Interpolation unit.  These polynomials were orthogonal, so if we have a polynomial represented in the basis of Legendre polynomials, i.e.,
$$ p_n(x) = \sum_i c_i P_i(x) $$
where $P_i(x)$ is the $i$th Legendre polynomial, then
$$ \int_{-1}^1 p_n(x) = 2 c_0 . $$

Given a set of points ${x_i}$, how can we determine $c_0$?
Can you write Python code for it?

def vander_legendre(x, n=None):
    if n is None:
        n = len(x)
    P = numpy.ones((len(x), n))
    if n > 1:
        P[:,1] = x
    for k in range(1,n-1):
        P[:,k+1] = ((2*k+1) * x * P[:,k] - k * P[:,k-1]) / (k + 1)
    return P

def fint_legendre_lin(f, a, b, n):
    x = numpy.linspace(a, b, n)
    fx = f(x)
    P = vander_legendre(numpy.linspace(-1,1,n))
    c = numpy.linalg.solve(P, fx)
    return c[0] * (b-a)

plot_accuracy(fint_legendre_lin, tests[:-1], range(3,25), ref=[2,3,4])


I am working on the file "Integration.ipynb".
The first task is:
Extrapolation
Looking above, we see that once we enter the asymptotic regime, errors are monotonic.  That is, the error is either all positive or all negative, and the magnitude is cut by a factor of 4 on each doubling of the number of points $n$.
Can you write Python code for it?

pyplot.figure()
for t in tests[:-1]:
    h = 1/(npoints - 1)
    pyplot.loglog(h, [numpy.abs(fint_trapezoid(t.f, a, b, n) - (t.F(b) - t.F(a)))
                        for n in npoints], '*', label=t.__name__)
pyplot.loglog(h, h**2, label='$h^2$')
pyplot.xlabel('h')
pyplot.ylabel('I')
pyplot.legend(loc='upper left')