I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
Optimization
Why do we care about gradients? As mentioned above, being able to compute them so easily opens up the possibility of automatically implementing gradient descent, an algorithm that follows derivatives downhill to find a local minimum of some function. In machine learning, we typically want to find the model parameters that minimize some loss, or error, function that measures how bad our model is.
TensorFlow provides amazing tools for doing just that, but to make sure we understand how they work, we'll implement a very simple gradient descent algorithm ourselves. We will be doing something very simple: finding the $x$ that minimizes $x^2$. (Yes, you can do this analytically, or by thinking about it for a second. But this example will bring together some of the ideas we've been discussing about variables and gradients.)
Let's start building our simple computational graph. It will consist of a variable, $x$, which is initialized to 5. It will also contain a "loss function," $x^2$.
Can you write Python code for it?

x = tf.Variable(5.0)
loss = x * x
g.get_operations()


I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
Sessions are more powerful than this, but to understand why, we need to talk about placeholders and variables.
Placeholders
Our program above is a bit rigid. If we wanted to perform the exact same addition calculation on different inputs, we would need to create a whole new trio of operations; we have no power to abstract. Placeholders fix this. A placeholder operation, just like a constant, has 0 inputs and 1 output. However, instead of fixing the output value when you define the operation, we pass the placeholder's value to sess.run when executing the graph. This allows us to execute the same graph multiple times with different placeholder values.
To add a placeholder operation to the default graph, we use tf.placeholder, passing in the type of the value we'd like the placeholder to hold. Valid types include tf.int8, tf.bool, tf.float32, and a whole lot more. See the documentation for a complete list. Like tf.constant, tf.placeholder returns a tensor (the output of the placeholder). We'll make a graph for adding two floats.
Can you write Python code for it?

a = tf.placeholder(tf.float32)
a

b = tf.placeholder(tf.float32)
flexible_sum_tensor = tf.add(a, b)

g.get_operations()


I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
The TensorFlow Tutorial I wish I had
This notebook is adapted from notes that I took when learning TensorFlow. It provides a slow, thorough introduction to the fundamentals of TensorFlow, answering questions like: What exactly is TensorFlow? Why do we need it? How does the computation graph work? What are operations, variables, placeholders, and sessions?
It assumes you know a good amount of Python. It starts from scratch with TensorFlow, but does not teach numpy or linear algebra; you won't necessarily need those skills to follow this tutorial, but understanding numpy basics, and how to "vectorize" your code, is an important skill for data scientists. Feel free to ask about resources for this if you're curious!
What exactly does TensorFlow do?
What is TensorFlow and what exactly does it do? Why do we need it? Why can't we do deep learning with pure Python or numpy? To answer these questions, you need to understand the problems TensorFlow trying to solve.
Python's limitations
Python is what's known as a dynamically typed programming language. This means that while your program is running, it stores in memory (alongside each variable) information about a variable's type. When you run x + y in Python, for example, Python's runtime looks up x's type, y's type, then figures out how to add those two things together, or throws an error if it doesn't know how. (For example, try adding a list and a string.) This is called a runtime type check.
This automatic bookkeeping makes Python a pleasure to use, but also leads to inefficiencies. If we store a long list of numbers, for instance, we must allocate memory not just for the data itself but for each number's metadata (type information). If we then want to sum the list, using a loop, for instance, we need to do type checks for every addition operation we perform. This makes pure Python nearly unuseable for processing large datasets.
Numpy for fast arithmetic
That's where numpy comes in. In pure Python, a list is a Python object that holds other Python objects, each...
Can you write Python code for it?


I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
The algorithm we have used here is vanilla gradient descent. TensorFlow actually comes with a whole family of optimizers that we can just plug into our model. We do not need to call tf.gradients or manually assign a variable at all; TensorFlow can create that whole portion of our computational graph.
To do this, we use one of TensorFlow's premade Optimizer classes. An Optimizer object keeps track of some parameters of the optimization algorithm (like step size, also called learning rate), and has a minimize method that can be used to automatically construct the portions of the computation graph that perform gradient computation and gradient descent. Here's how our example from above looks with TensorFlow's GradientDescentOptimizer:
Can you write Python code for it?

tf.reset_default_graph()
g = tf.get_default_graph()

x = tf.Variable(5.0)
loss = x * x

# Create the optimizer
optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.1)
# Add optimize nodes to the graph
minimize_loss = optimizer.minimize(loss)


I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
The operations are all still there! That's because reset_default_graph doesn't delete operations, it just creates a new graph and makes it the default. g still refers to the old graph. We can fix this (and let the old graph be garbage-collected) by reassigning g:
Can you write Python code for it?

g = tf.get_default_graph()
g.get_operations()


I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
The session is closed automatically at the end of the with block.
Note that trying to call sess.run(distance_from_origin) without feeding in a and b will result in an error. Furthermore, placeholder values are not persistent across multiple calls to sess.run, so even if you've previously provided values for a and b in prior calls to sess.run, these are no longer in memory anywhere by the time you make your next sess.run call.
One final trick for convenience. Since our graph is getting a bit crowded, we may want to clear it:
Can you write Python code for it?


I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
There are other optimizers that implement clever variations of gradient descent. For example, here's the popular Adam optimizer:
Can you write Python code for it?

adam_optimizer = tf.train.AdamOptimizer(learning_rate=0.1)
minimize_loss = adam_optimizer.minimize(loss)
g.get_operations()


I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
Variables
Like constants and placeholders, variable operations take 0 inputs and produce 1 output; the big difference is that a variable is mutable and persistent across runs of your graph (within a session). Whereas a constant's value is fixed when creating the graph, and a placeholder's value is set anew each time you call sess.run, a variable's value is set or changed while the graph is running, by side-effect-ful "assign" operations, and remembered even after sess.run is finished. (That memory is in the Session object, which manages stateful components like variables. Calling sess.close is necessary to free that memory.)
Let's use tf.Variable to add a variable to our new graph, initialized to 42.
Can you write Python code for it?

x = tf.Variable(42)

# summarize the operations now in the graph:
[(op, "{} inputs and {} output".format(len(op.inputs), len(op.outputs))) for op in g.get_operations()]


I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
We can now execute the graph:
Can you write Python code for it?

sess = tf.Session(config=config)
sess.run(grad)


I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
We then set up the major quantities in our model as follows:
1. Our data, the temperatures X and the hospital visit numbers y, are represented with placeholders. This is so we can fill these values with our actual data at execution time. (You may wonder: why not just put all the data in as a constant? Typically, rather than use Gradient Descent, we use Stochastic Gradient Descent, which means that instead of taking gradients of the loss computed on all our data every iteration, we feed in a small "batch" of data to the graph each iteration of training. This is more efficient, lets us handle large datasets, and provably converges to a local minimum just like normal Gradient Descent. To use this technique, we need placeholders: each time we call sess.run, we'll pass in different data.)
2. The parameters of our model, which we hope to learn, are represented as TensorFlow variables. This is so that as we run the training operation repeatedly, their current values can change.
3. We then use TensorFlow operations like addition and multiplication to create predicted y values based on the X values, according to our model. The loss will be computed based on this prediction and its divergence from the true y.
Can you write Python code for it?

X = tf.placeholder(tf.float32)
y = tf.placeholder(tf.float32)
m = tf.Variable(0.)
b = tf.Variable(0.)
predicted_y = X * m + b


I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
We will now create an operation that changes $x$ based on the current derivative of the loss. We can do this using the tf.assign function, which creates an operation that assigns a variable to a new value. (The Assign operation has two inputs, the reference to the variable's mutable storage, and the new value we wish to assign. As mentioned in the Variables section, we can pass a Variable object as the first input and TensorFlow will substitute the correct tensor. The Assign operation has one output, the tensor for which is returned by tf.assign, which is the new value of the variable after assignment.)
Because the derivative points in the direction in which our loss is growing fastest, we will subtract $x$'s derivative from $x$. (After all, we want the loss to shrink, not grow.) But rather than subtract the entire derivative, we will subtract a fraction of it. (We generally try not to take large steps in gradient descent; just because the derivative points us in some direction now, doesn't mean that going arbitrarily far in that direction will keep bringing us down the hill. $x^2$ provides a good example: if we are currently at $x=5$, the derivative is $10$, but going $-10$ steps to $-5$ completely steps over the valley we are trying to reach at $x=0$! By multiplying our derivative by a small step size of, say, 0.1, we can avoid this fate and progress slowly but almost-surely to our goal.)
Can you write Python code for it?

step_size = 0.1
new_x_value = x - dloss_dx * step_size
assign_new_x = tf.assign(x, new_x_value)


I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
Wow -- four operations were just added to the graph! Let's go through them one by one:
1. The first should look familiar. Although it's called Variable/initial_value, it is actually just a constant operation. Its output is a tensor that will evaluate to 42.
2. The second is a "Variable" operation. Like placeholder and constant, it has no inputs and one output. But its output type is int32_ref, a mutable int32. You can use an int32_ref basically anywhere you can use an int32. But you can also use it as an input to an assign operation, which is up next.
3. The third operation is an assign op. It takes two inputs, a ref to a mutable value (almost always the output of a Variable op), and a value, then assigns the variable to that value. This assign op takes its input from the Variable/initial_value constant and the Variable ref just discussed. When executed, it has the effect of initializing our variable to to 42. The output is another int32_ref tensor, referring to the same mutable storage as the output from (2).
4. Finally, we have an identity operation. It takes the variable ref as input and outputs a non-ref tf.int32. This isn't necessary in most cases, where it is ok to use the tf.int32_ref directly.
$x$, the return value of the tf.Variable constructor, is a Variable object, not a tensor, but in practice, you can use it as if it were a tensor. It will either be interpreted as the tensor output of operation (2) above -- the int32_ref pointing to the variable's current value -- or of operation (4), depending on context.
Let's start a session and play with our variable.
Can you write Python code for it?


I am working on the file "01B_TensorFlow_Fundamentals.ipynb".
The first task is:
g now has a Const operation! Note that tf.constant affected the graph g, even though we didn't explicitly say we wanted the constant operation to be added to g. It is possible to add operations to a specific, non-default graph, but most of the time, we add directly to the default graph, using functions like tf.constant. In fact, we generally don't even call get_default_graph to give g a name; we just use it implicitly.
Let's examine the constant operation we created. We can use the inputs and outputs attributes of the operation to confirm that there really are zero inputs and one output.
Can you write Python code for it?

const_operation = g.get_operations()[0]
len(const_operation.inputs), len(const_operation.outputs)


I am working on the file "01C_MLP_CNN_Assignment.ipynb".
The first task is:
Comparison
How do the sigmoid and rectified linear unit (ReLU) compare?

[Your answer here]

Convolutional Neural Network
Build a simple 2-layer CNN for MNIST digit classfication. Feel free to play around with the model architecture and see how the training time/performance changes, but to begin, try the following:
Image -> CNN (32 5x5 filters) -> nonlinearity (ReLU) ->  (2x2 max pool) -> CNN (64 5x5 filters) -> nonlinearity (ReLU) -> (2x2 max pool) -> fully connected (1024 hidden units) -> nonlinearity (ReLU) -> fully connected (10 hidden units) -> softmax
Some additional functions that you might find helpful:
Can you write Python code for it?

# Convolutional neural network functions
def conv2d(x, W):
    """conv2d returns a 2d convolution layer with full stride."""
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

def max_pool_2x2(x):
    """max_pool_2x2 downsamples a feature map by 2X."""
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')

# Tensorflow Function that might also be of interest:
# tf.reshape()


I am working on the file "01C_MLP_CNN_Assignment.ipynb".
The first task is:
Multilayer Perceptron
Build a multilayer perceptron for MNIST digit classfication. Feel free to play around with the model architecture and see how the training time/performance changes, but to begin, try the following:
Image -> fully connected (500 hidden units) -> nonlinearity (Sigmoid/ReLU) -> fully connected (10 hidden units) -> softmax
Skeleton framework for you to fill in (Code you need to provide is marked by ###):
Can you write Python code for it?

# Model Inputs
x = ### MNIST images enter graph here ###
y_ = ### MNIST labels enter graph here ###

# Define the graph

### Create your MLP here##
### Make sure to name your MLP output as y_mlp ###

# Loss
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_mlp))

# Optimizer
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

# Evaluation
correct_prediction = tf.equal(tf.argmax(y_mlp, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

with tf.Session() as sess:
    # Initialize all variables
    sess.run(tf.global_variables_initializer())

    # Training regimen
    for i in range(4000):
        # Validate every 250th batch
        if i % 250 == 0:
            validation_accuracy = 0
            for v in range(10):
                batch = mnist.validation.next_batch(100)
                validation_accuracy += (1/10) * accuracy.eval(feed_dict={x: batch[0], y_: batch[1]})
            print('step %d, validation accuracy %g' % (i, validation_accuracy))

        # Train
        batch = mnist.train.next_batch(50)
        train_step.run(feed_dict={x: batch[0], y_: batch[1]})

    print('test accuracy %g' % accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels}))


I am working on the file "01C_MLP_CNN_Assignment.ipynb".
The first task is:
Skeleton framework for you to fill in (Code you need to provide is marked by ###):
Hint: Convolutional Neural Networks are spatial models. You'll want to transform the flattened MNIST vectors into images for the CNN. Similarly, you might want to flatten it again sometime before you do a softmax. You also might want to look into the  conv2d() documentation to see what shape kernel/filter it's expecting.
Can you write Python code for it?

# Model Inputs
x = ### MNIST images enter graph here ###
y_ = ### MNIST labels enter graph here ###

# Define the graph

### Create your CNN here##
### Make sure to name your CNN output as y_conv ###

# Loss
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_, logits=y_conv))

# Optimizer
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)

# Evaluation
correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

with tf.Session() as sess:
    # Initialize all variables
    sess.run(tf.global_variables_initializer())

    # Training regimen
    for i in range(10000):
        # Validate every 250th batch
        if i % 250 == 0:
            validation_accuracy = 0
            for v in range(10):
                batch = mnist.validation.next_batch(50)
                validation_accuracy += (1/10) * accuracy.eval(feed_dict={x: batch[0], y_: batch[1]})
            print('step %d, validation accuracy %g' % (i, validation_accuracy))

        # Train
        batch = mnist.train.next_batch(50)
        train_step.run(feed_dict={x: batch[0], y_: batch[1]})

    print('test accuracy %g' % accuracy.eval(feed_dict={x: mnist.test.images, y_: mnist.test.labels}))


I am working on the file "01_00_numerical_differentiation.ipynb".
The first task is:
A more in-depth discussion about round-off erros in numerical differentiation can be found here
Special functions in numpy
numpy provides a simple method diff() to calculate the numerical derivatives of a dataset stored in an array by forward differences. The function gradient() will calculate the derivatives by midpoint (or central) difference, that provides a more accurate result.
Can you write Python code for it?

%matplotlib inline
import numpy as np
from matplotlib import pyplot

y = lambda x: x*x

x1 = np.arange(0,10,1)
x2 = np.arange(0,10,0.1)

pyplot.plot(x1,np.gradient(y(x1),1.),'r--o');
pyplot.plot(x1[:x1.size-1],np.diff(y(x1))/np.diff(x1),'b--x');


I am working on the file "01_00_numerical_differentiation.ipynb".
The first task is:
In addition, one could consider the midpoint difference, defined as:
$$ dy \approx \Delta y = y(x_0+\frac{\Delta_x}{2})-y(x_0-\frac{\Delta_x}{2}).$$
For a more complex function we need to import it from math. For instance, let's calculate the derivative of $sin(x)$ at $x=\pi/4$, including both the forward and midpoint differences.
Can you write Python code for it?


I am working on the file "01_00_numerical_differentiation.ipynb".
The first task is:
Let's try using powers of 1/2
Can you write Python code for it?


I am working on the file "01_00_numerical_differentiation.ipynb".
The first task is:
More discussion about numerical differenciation, including higher order methods with error extrapolation can be found here.
The module scipy also includes methods to accurately calculate derivatives:
Can you write Python code for it?

from scipy.misc import derivative

y = lambda x: x**2

dx = 1.
x = 1.

while(dx > 1.e-10):
    d = derivative(f, x, dx, n=1, order=3)
    print("%6.0e %20.16f %20.16f" % (dx, d, d-2.))
    dx = dx / 10.


from scipy.misc import derivative

y = lambda x: x**2

dx = 1.
x = 1.

while(dx > 1.e-10):
    d = derivative(y, x, dx, n=1, order=3)
    print("%6.0e %20.16f %20.16f" % (dx, d, d-2.))
    dx = dx / 10.


from scipy.misc import derivative
from math import *

f = lambda x: x**2

delta = 1.
while(delta > 1.e-10):
    x = 1.
    d = derivative(f, x, delta, n=1, order=3)
    print("%6.0e %20.16f %20.16f" % (delta, d, d-2.))
    delta = delta / 10.


I am working on the file "01_00_numerical_differentiation.ipynb".
The first task is:
Notice above that gradient() uses forward and backward differences at the two ends.
Can you write Python code for it?


I am working on the file "01_00_numerical_differentiation.ipynb".
The first task is:
One way to improve the roundoff errors is by simply using the decimal package
Can you write Python code for it?

from decimal import Decimal

delta = Decimal("1.")
while(delta >= Decimal("1.e-10")):
    x = Decimal("1.")
    d = (x+delta)*(x+delta)-x*x
    d = d / delta
    print("%6.0e %20.16f %20.16f" % (delta, d, d-Decimal("2.")))
    delta = delta / Decimal("10.")


from decimal import Decimal

dx = Decimal("1.")
while(dx >= Decimal("1.e-10")):
    x = Decimal("1.")
    dy = (x+dx)*(x+dx)-x*x
    d = dy / dx
    print("%6.0e %20.16f %20.16f" % (dx, d, d-Decimal("2.")))
    dx = dx / Decimal("10.")


I am working on the file "01_00_numerical_differentiation.ipynb".
The first task is:
Why is it that the sequence does not converge? This is due to the round-off errors in the representation of the floating point numbers. To see this, we can simply type:
Can you write Python code for it?


I am working on the file "01_01_euler.ipynb".
The first task is:
Actually, for this particularly simple case, calling a function may introduce unecessary overhead, but it is a an example that we will find useful for future applications. For a simple function like this we could have used a "lambda" function (more about lambda functions here).
Can you write Python code for it?


I am working on the file "01_01_euler.ipynb".
The first task is:
Alternatively, and in order to re use code in future problems, we could have created a function.
Can you write Python code for it?

def euler(y, f, dx):
    """Computes y_new = y + f*dx

    Parameters
    ----------
    y  : float
        old value of y_n at x_n
    f  : float
        first derivative f(x,y) evaluated at (x_n,y_n)
    dx : float
        x step
    """

    return y + f*dx

T = T0
for i in range(1,nsteps):
    T = euler(T, -r*(T-Ts), dt)
    my_temp[i] = T


def euler(y, f, dx):
    """Computes y_new = y + f*dx

    Parameters
    ----------
    y  : float
        old value of y_n at x_n
    f  : float
        first derivative f(x,y) evaluated at (x_n,y_n)
    dx : float
        x step
    """

    return y + f*dx

t = t0
for i in range(1,nsteps):
    t = euler(t, -r*(t-ts), dt)
    my_temp[i] = t


I am working on the file "01_01_euler.ipynb".
The first task is:
Let's try plotting the results. We first need to import the required libraries and methods
Can you write Python code for it?

%matplotlib inline
import numpy as np
from matplotlib import pyplot


I am working on the file "01_01_euler.ipynb".
The first task is:
Next, we create numpy arrays to store the (x,y) values
Can you write Python code for it?

my_time = np.zeros(nsteps)
my_temp = np.zeros(nsteps)


I am working on the file "01_01_euler.ipynb".
The first task is:
Now, let's study the effects of different time steps on the convergence:
Can you write Python code for it?

dt = 1.
#my_color = ['#003366','#663300','#660033','#330066']
my_color = ['red', 'green', 'blue', 'black']
for j in range(0,4):
    nsteps = int(tmax/dt)    #the arrays will have different size for different time steps
    my_time = np.linspace(dt,tmax,nsteps)
    my_temp = np.zeros(nsteps)
    T = T0
    for i in range(1,nsteps):
        T = euler(T, -r*(T-Ts), dt)
        my_temp[i] = T

    pyplot.plot(my_time, my_temp, color=my_color[j], ls='-', lw=3)
    dt = dt/2.

pyplot.xlabel('time');
pyplot.ylabel('temperature');
pyplot.xlim(8,10);
pyplot.ylim(48,58);


dt = 1.
#my_color = ['#003366','#663300','#660033','#330066']
my_color = ['red', 'green', 'blue', 'black']
for j in range(0,4):
    nsteps = int(tmax/dt)    #the arrays will have different size for different time steps
    my_time = np.linspace(dt,tmax,nsteps)
    my_temp = np.zeros(nsteps)
    T = T0
    for i in range(1,nsteps):
        T = euler(T, -r*(T-Ts), dt)
        my_temp[i] = T

    pyplot.plot(my_time, my_temp, color=my_color[j], ls='-', lw=3)
    dt = dt/2.

pyplot.xlabel('time');
pyplot.ylabel('temperature');
pyplot.xlim(8,10)
pyplot.ylim(48,58);


dt = 1.
#my_color = ['#003366','#663300','#660033','#330066']
my_color = ['red', 'green', 'blue', 'black']
for j in range(0,4):
    nsteps = int(tmax/dt)    #the arrays will have different size for different time steps
    my_time = np.linspace(dt,tmax,nsteps)
    my_temp = np.zeros(nsteps)
    T = T0
    for i in range(0,nsteps):
        T = euler(T, -r*(T-Ts), dt)
        my_temp[i] = T

    pyplot.plot(my_time, my_temp, color=my_color[j], ls='-', lw=3, label="dt="+str(dt))
    dt = dt/2.

pyplot.xlabel('time');
pyplot.ylabel('temperature');
pyplot.legend()
pyplot.xlim(8,10)
pyplot.ylim(48,58);


I am working on the file "01_01_euler.ipynb".
The first task is:
We could have saved effort by defining
Can you write Python code for it?

my_time = np.linspace(0.,tmax,nsteps)

pyplot.plot(my_time, my_temp, color='#003366', ls='-', lw=3)
pyplot.xlabel('time')
pyplot.ylabel('temperature');


I am working on the file "01_03_eqs_of_motion.ipynb".
The first task is:
Challenge 1.2:
Modify the previous code to include the effect of drag resistance, and solve Exercise 1.7, part 2.
An introduction to animations with matplotlib
Can you write Python code for it?

%matplotlib inline
import numpy as np
from matplotlib import pyplot
import matplotlib.animation as animation
from JSAnimation.IPython_display import display_animation

g = 9.8            # g acceleration
dt = 0.1           # time step

x = np.array([0.])
y = np.array([0.])

fig = pyplot.figure()
ax = pyplot.axes(xlim=(0, 20), ylim=(0, 6), xlabel='distance x(m)', ylabel='distance y(m)')
points, = ax.plot([], [], marker='o', linestyle='None')

pyplot.legend()
p = particle2(1., 0., 0., 10., 10.)

def init():
    points.set_data(x, y)
    return points

def animate(i):
    if p.y >= 0 and i > 0:
        p.euler(0., -g, dt)
        x[0] = p.x
        y[0] = p.y
    points.set_data(x, y)
    return points

anim = animation.FuncAnimation(fig, animate, frames = 20, interval=100, blit=True)

display_animation(anim, default_mode='once')


I am working on the file "01_03_eqs_of_motion.ipynb".
The first task is:
Exercise 1.5: Gravitational force
According to Newton’s law of gravitation, the action of the
gravitational field of the earth on a particle is given by
$$F=\frac{GMm}{(R+y)^2}=\frac{gm}{(1+y/R)^2},$$ where $y$ is measured
from the earth’s surface, $R$ is the earth’s radius, $G$ is the
gravitational constant, $M$ is the mass of the earth, and $g=GM/R^2$.
There is not simple analytical solution for this problem. Modify your
code to simulate the fall of a particle from an altitude $y_0$ with zero
initial velocity, and compute its speed when it hits the ground.
Determine the value of $y_0$ for which this impact velocity differs by
one percent from its value under a constant acceleration $g=9.8m/s^2$.
Take the radius of the earth to be $6.37\times 10^6m$.
Challenge 1.2:
Modify the previous code to introduce the a gravitational force that depends on the position, and solve Exercise 1.5
Exercise 1.6: Harmonic oscillator
The two coupled first order equations
$$\frac{dy}{dt}=p; \frac{dp}{dt}=-4\pi ^2 y$$ define a harmonic
oscillator with period $T=1$. Compute the position $y$ and momentum $p$
as a function of time using a generalization of the previous code. Plot
the results for $y_0=1$ and $p_0$=0. Investigate the accuracy with which
the system returns to the initial state at integral values of $t$.
Can you write Python code for it?

import math

y0 = 1
v0 = 0

p = particle(1., y0, v0)  # if the mass is m=1 then p = v (except for the units!)

tmax = 4.
dt = 0.02
nsteps = int(tmax/dt)    #the arrays will have different size for different time steps

t = np.linspace(0.,tmax,nsteps)
v = np.zeros(nsteps)
y = np.zeros(nsteps)
y[0] = y0

for i in range(1,nsteps):
    p.euler(-4*math.pi**2*p.y, dt)
    t[i] = t[i-1]+dt
    v[i] = p.v
    y[i] = p.y

pyplot.plot(t, v, color="#FF0000", ls='-', lw=3)
pyplot.xlabel('time(s)')
pyplot.ylabel('velocity(m/s)');

pyplot.plot(t, v, color="#0000FF", ls='-', lw=3)
pyplot.xlabel('time(s)')
pyplot.ylabel('position(m)');


I am working on the file "01_03_eqs_of_motion.ipynb".
The first task is:
Two dimensional trajectories
In a 2d trajectory, the direction of the drag force is opposite to the
speed vector ${\bf v}$. Newton’s equations of motion for $x$ and $y$
components are written $$\begin{aligned}
&& m\frac{dv_x}{dt}=-F_dx; \
&& m\frac{dv_y}{dt}=-mg-F_dy;\end{aligned}$$ Using $F_d=kv^2$,
$v_x=v\cos{\theta}$ and $v_y=v\sin{\theta}$, we find $$\begin{aligned}
&& \frac{dv_x}{dt}=-\frac{k}{m}vv_v, \
&& \frac{dv_x}{dt}=-g-\frac{k}{m}vv_v, \end{aligned}$$ where
$v^2=v_x^2+v_y^2$. Hence, we cannot calculate the vertical motion of the
object without reference to the horizontal component.

Object in a two dimensional trajectory under the effect of gravitational and dragging forces}
Exercise 1.7: Trajectory of a shot
Modify your code so that the 2d trajectory of an object can be computed,
and graphs of $y$ as a function of $x$ can be made.

As a check on your program, first neglect the effect of air
    resistance so that you an compare to known results. Supouse that the
    object is thrown and $t_0$ with an angle $\theta _0$ with an initial
    velocity $v_0=15$m/s. Vary $\theta_0$ and show that the maximum
    range occurs at $\theta_0=45^{\circ}$ Compare your result with the
    exact value $v_0^2/g$

Consider the effects of air resistance. Compute the maximum range,
    and the corresponding angle using $k/m=0.1$, $v_0=30$m/s.
Can you write Python code for it?

class particle2(object):

    def __init__(self, mass=1., x=0., y=0., vx=0., vy=0.):
        self.mass = mass
        self.x = x
        self.y = y
        self.vx = vx
        self.vy = vy

    def euler(self, fx, fy, dt):
        self.vx = self.vx + fx/self.mass*dt
        self.vy = self.vy + fy/self.mass*dt
        self.x = self.x + self.vx*dt
        self.y = self.y + self.vy*dt

%matplotlib inline
import numpy as np
from matplotlib import pyplot
from matplotlib.colors import ColorConverter as cc
import math

g = 9.8            # g acceleration
v0 = 30.           # initial velocity

dt = 0.1           # time step

colors = ['red','orange','yellow','green','magenta','cyan','blue','purple','black']

for angle in range(1,9):
    x = [0]                                  # we need to initialize the arrays for each value of the angle
    y = [0]
    vx = [math.cos(angle*0.1*math.pi/2.)*v0]
    vy = [math.sin(angle*0.1*math.pi/2.)*v0]
    t = [0.]

    p = particle2(1., 0., 0., vx[0], vy[0])
    while p.y >= 0.:
        fy = -g
        p.euler(0., fy, dt)
        x.append(p.x)
        y.append(p.y)
        vx.append(p.vx)
        vy.append(p.vy)
        t.append(t[-1]+dt)

    t_data = np.array(t) # we convert the list into a numpy array for plotting
    x_data = np.array(x)
    y_data = np.array(y)
    vx_data = np.array(vx)
    vy_data = np.array(vy)

    my_plot = pyplot.plot(x_data, y_data, color=(colors[angle]), ls='-', lw=3, label = str(angle*0.1))
    pyplot.legend()

pyplot.xlabel('position x(m)')
pyplot.ylabel('position y(m)');


I am working on the file "01_About_Python.ipynb".
The first task is:
Other Useful Statistics Libraries:

statsmodel - various statistical routines
scikit-learn - machine learning in Python
pyMC - for bayesian data analysis
pystan - Bayesian analysis based on stan

Python has many libraries for studying graphs, One well-known example is NetworkX
Here’s some example code that generates and plots a random graph, with node color determined by shortest path length from a central node
Can you write Python code for it?

import networkx as nx
import matplotlib.pyplot as plt
import numpy as np

G = nx.random_geometric_graph(200,0.12) # Generate random graph
pos = nx.get_node_attributes(G, 'pos') # Get positions of nodes
# Find node nearest the center point (0.5, 0.5)
dists = [(x - 0.5)**2 + (y - 0.5)**2 for x,y in list(pos.values())]
ncenter = np.argmin(dists)

# Plot graph, coloring by path length from central node
p = nx.single_source_shortest_path_length(G, ncenter)
plt.figure()
nx.draw_networkx_edges(G, pos, alpha=0.4)
nx.draw_networkx_nodes(G, pos, nodelist=list(p.keys()), node_size=120, alpha=0.5,
                          node_color=list(p.values()), cmap=plt.cm.jet_r)

plt.show()


I am working on the file "01_About_Python.ipynb".
The first task is:
Python’s data manipulation and statistics libraries have improved rapidly over the last few years. Pandas one of the most popular libraries for working with data. Pandas is fast, efficient, and well designed.
Can you write Python code for it?

import pandas as pd
import scipy as sp
data = sp.randn(5, 2) # Create 5x2 matrix of random numbers
dates = pd.date_range('28/12/2010', periods=5)
df = pd.DataFrame(data, columns=('price', 'weight'), index=dates)
print(df)

df.mean()


I am working on the file "01_About_Python.ipynb".
The first task is:
SciPy includes many of the standard routines used in:

linear algebra
integration
interpolation
optimization
distributios and random number generation
signal processing

The most popular and comprehensive Python library for creating figures and graphs is Matplotlib.

Plots, histograms, contour images, 3D, bar charts, etc., etc.
Output in many formats (PDF, PNG, EPS, etc.)
LaTeX integration

To see more plots you can see here
Other graphics libraries include:

Plotly
Bokeh
VPython -- 3D graphics and animations

Symbolic Algebra, it's useful to be able to manipulate symbolic expressions, as in Mathematica or Maple. The SymPy library provides this functionality from within the Python shell
Can you write Python code for it?

from sympy import Symbol
x, y = Symbol('x'), Symbol('y') # Treat 'x' and 'y' as algebraic symbols
print(x + x + x + y)

expression = (x+y)**2
expression.expand()

from sympy import solve
# Solving polynomials
solve(x**2 + x + 2)

from sympy import limit, sin, diff
limit(1 / x, x, 0)

limit(sin(x)/x , x, 0)

diff(sin(x),x)


I am working on the file "01_About_Python.ipynb".
The first task is:
The SciPy library is built on top of NumPy and provides additional functionality.
Can you write Python code for it?

from scipy.stats import norm
from scipy.integrate import quad
phi = norm()
value, error = quad(phi.pdf, -2,2) # Integrate using Gaussian quadrature
value


I am working on the file "01_Agua/.ipynb_checkpoints/agua-checkpoint.ipynb".
The first task is:
Aquellas ciudades cuya proporción es igual a uno serán seleccionadas.
Can you write Python code for it?

# Municipios por ciudad con información completa entre el total de los municipios que existen en esa ciudad.
zm = zm_sinNaN / zm
# Filtar aquellas ciudades cuyos municipios cuentan con toda la información disponible.
zm = zm[zm['_id'] == 1]
zm.head()


I am working on the file "01_Agua/.ipynb_checkpoints/agua-checkpoint.ipynb".
The first task is:
Datos
La base de datos está en formato delimitado por comas (csv). Este archivo es el resultado de la extracción de datos a los archivos que el INEGI distribuye en sus anuarios estadísticos.
Can you write Python code for it?

data = pd.read_csv("test.csv")
Markdown('La base de datos tiene {} filas y {} columnas. Cada columna es una variable:'.format(data.shape[0], data.shape[1]))


I am working on the file "01_Agua/.ipynb_checkpoints/agua-checkpoint.ipynb".
The first task is:
El campo "entubada_total" es el porcentaje de viviendas dentro del municipio que cuentan con agua potable entubada mientras que "entubada_dentro_de_vivienda" y "entubada_fuera_de_vivienda_dentro_de_terreno" muestran la proporción de estas viviendas que cuentan con tubería hasta la vivienda o cuya tubería entra al terreno sin llegar a la vivienda, respectivamente. Para este indicador basta utilizar los datos de entubada_total. El acarreo de agua potable no se considera una fuente mejorada de agua potable, toda vez que no es trazable la salubridad de la fuente.
Can you write Python code for it?

print('Los datos disponibles en la base de datos acerca de agua entubada son los siguientes:')
print('\tTotal de municipios en la base de datos: {}'.format(len(data['entubada_total'])))
print('\tMunicipios que cuentan con registros de agua entubada: {}'.format(len(data['entubada_total'].dropna())))

# Dataset de agua entubada
c_disponibilidad = vdesc.tolist()
c_disponibilidad.append('entubada_total'); c_disponibilidad.remove('collection')
data_disponibilidad = data[c_disponibilidad]
data_disponibilidad.to_csv('disponibilidad.csv')
data_disponibilidad.head()


I am working on the file "01_Agua/.ipynb_checkpoints/agua-checkpoint.ipynb".
The first task is:
El siguiente mapa muestra los municipios pertenecientes a Zonas Urbanas que cuentan con información completa:
Can you write Python code for it?


I am working on the file "01_Agua/.ipynb_checkpoints/agua-checkpoint.ipynb".
The first task is:
Indicadores
01 -  Disponibilidad y acceso al agua
AG01.1 - PROPORCIÓN DE LA POBLACIÓN QUE UTILIZA SERVICIOS DE AGUA POTABLE MANEJADOS DE MANERA SALUDABLE
Los servicios de agua potable manejados de manera saludable se definen como aquéllas fuentes mejoradas de agua potable que se encuentran dentro de las instalaciones y disponibles cuando son necesarias, libres de contaminación por materia fecal o química.
Una fuente mejorada de agua potable se define como aquélla que por la naturaleza de su construcción o a través de intervención activa, está protegida de la contaminación externa.
Este indicador está diseñado en base al indicador 6.1.1 de los Objetivos de Desarrollo Sostenible de la ONU.
Metodología:
Como se mencionó anteriormente, no todas las ciudades del SUN cuentan con información completa de indicadores de agua. Si bien para generar un indicador integral de sustentabilidad del agua es necesario que una ciudad cuente con la información completa para todos sus municipios, esto no nos impide analizar de manera individual cada indicador para las ciudades que cuenten con la información.
En la base de datos hay 12 columnas relacionadas al acceso a agua potable:
Can you write Python code for it?

dispyacc = list(); dispyacc.append(data.columns[8])
dispyacc = dispyacc + data.columns[13:24].tolist(); dispyacc


I am working on the file "01_Agua/.ipynb_checkpoints/agua-checkpoint.ipynb".
The first task is:
La informacion de este análisis puede ser exportada en formato CSV para integrarla al sistema de información Geográfica de la Plataforma de Conocimiento sobre Ciudades Sustentables.
Can you write Python code for it?


I am working on the file "01_Agua/.ipynb_checkpoints/agua-checkpoint.ipynb".
The first task is:
Las 38 variables que contiene la base de datos son:
Can you write Python code for it?

vdesc = data.columns[0:8]
for i,a in enumerate(data.columns):
    print(i,a)


I am working on the file "01_Agua/.ipynb_checkpoints/agua-checkpoint.ipynb".
The first task is:
Los primeros 3 cuartiles están por debajo del 99%, mientras que es de extrañar que el valor máximo sea 157884. Muy probablemente los valores que superan el 100 están capturados de manera errónea, por lo que serán excluidos del análisis.
Can you write Python code for it?

# Casos que serán excluidos del análisis
data_disponibilidad[data_disponibilidad['entubada_total']>100]

# Declaracion de variable con datos para el análisis
analisis_disponibilidad = data_disponibilidad[data_disponibilidad['entubada_total']<100]
analisis_disponibilidad = analisis_disponibilidad.set_index('entidad')


I am working on the file "01_Agua/.ipynb_checkpoints/agua-checkpoint.ipynb".
The first task is:
Quitando los municipios que no tienen información disponible, las estadísticas básicas de la información disponible son las siguientes:
Can you write Python code for it?


I am working on the file "01_Agua/.ipynb_checkpoints/agua-checkpoint.ipynb".
The first task is:
Selección de datos para el estudio
Para la creación de indicadores es necesario contar con información completa. En esta sección se seleccionarán de la base de datos, las ciudades del SUN con información completa. Python identifica las celdas sin información disponible con el indicador NaN. Por lo tanto, para identificar los municipios que cuentan con información completa será necesario filtrar todos los municipios que no tengan ningún NaN en sus variables.
Can you write Python code for it?

# Agrupación según clave SUN con todas las variables con valor diferente a "NaN".
zm_sinNaN = data.dropna().groupby("cve_sun").count()
print("Zonas metropolitanas sin valores 'NaN': ",zm_sinNaN.shape[0])


I am working on the file "01_Agua/.ipynb_checkpoints/agua-checkpoint.ipynb".
The first task is:
Tal y como muestran las líneas de código anteriores, exiten 59 zonas metropolitanas; cada una está compuesta por cierto número de municipios. Sin embargo, la tabla de datos original tiene muchos valores vacíos (NaN) por lo que, al filtar todos los municipios que no tienen valores vacíos y al agruparlos según su clave SUN, en realidad, de acuerdo con los datos, existen 38 ciudades cuyos municipios cuentan con todas las variables disponibles.
Esto no quiere decir que se cuenta con toda la información para todos los municipios que conforman estas 38 ciudades; la variable zm_sinNaN enlista el número de municipios por ciudad que no presentan valores NaN. Por ende, el siguiente paso es observar cuántas ciudades tienen toda la información para todos sus municipios.
Can you write Python code for it?

# Lista de claves SUN de las ciudades sin 'NaNs' para filtrar.
cves_sinNaN = zm_sinNaN.index.tolist()

# Filtro de ciudades con la lista anterior.
zm = zm.loc[zm.index.isin(cves_sinNaN)]

# ¿Qué ciudades tienen la información completa para todos sus municipios?()
(zm_sinNaN / zm)[:10]


I am working on the file "01_Agua/.ipynb_checkpoints/agua-checkpoint.ipynb".
The first task is:
Ya que se tienen las claves SUN de las ciudades sin datos faltantes hay que filtrar los datos de los municipios y declarar una nueva variable con esta información.
Can you write Python code for it?

muns = data.loc[data['cve_sun'].isin(zm.index.tolist())]
muns.head()


I am working on the file "01_Analytical_solutions/5_Greens_function_acoustic_1-3D.ipynb".
The first task is:
2D Green's function
Using the same approach as for the 1D medium, we can calculate the 2D Green's function $\mathbf{G_2(x,t;x_s,t_s)}$, which is governed by
\begin{equation}
\frac{\partial^2 G_2}{\partial t^2} - V_{p0}^2 \biggl(\frac{\partial^2 G_2}{\partial x^2} + \frac{\partial^2 G_2}{\partial z^2} \biggl)= \delta(t-t_s) \delta(x-x_s) \delta(z-z_s) \nonumber
\end{equation}
as
\begin{equation}
G_2(x,z,t) = \dfrac{1}{2\pi V_{p0}^2}\dfrac{H\biggl((t-t_s)-\dfrac{|r|}{V_{p0}}\biggr)}{\sqrt{(t-t_s)^2-\dfrac{r^2}{V_{p0}^2}}} \nonumber
\end{equation}
with the source-receiver distance (offset)
$r = \sqrt{(x-x_s)^2+(z-z_s)^2}$
Compared to the 1D Green's function, we have a damped Heaviside function due to the radiation characteristic of the infinite line source, introduced by the 2D approximation. Let's also plot the 2D Green's function.
Can you write Python code for it?

# Calculation of 2D Green's function

G2=np.zeros(nt)                    # initialization G with zeros
r=np.sqrt((xs-xr)**2+(ys-yr)**2)

for i in range (nt):
    if (((time[i]-ts)-abs(r)/vp0)>0):
        G2[i]=(1./(2*np.pi*vp0**2))*(1./np.sqrt((time[i]-ts)**2-(r**2/vp0**2)))
    else:
        G2[i]=0

# Plotting Green's function in 2D
plt.plot(time, G2)
plt.title("Green's function for hom. 2D acoustic medium" )
plt.xlabel("Time, s")
plt.ylabel("Amplitude")
plt.xlim((0, tmax))
plt.grid()
plt.show()


I am working on the file "01_Analytical_solutions/5_Greens_function_acoustic_1-3D.ipynb".
The first task is:
3D Green's function
Finally, we can calculate the 3D Green's function $\mathbf{G_3(x,t;x_s,t_s)}$, which is governed by
\begin{equation}
\frac{\partial^2 G_3}{\partial t^2} - V_{p0}^2 \biggl(\frac{\partial^2 G_3}{\partial x^2} + \frac{\partial^2 G_3}{\partial y^2} +\frac{\partial^2 G_3}{\partial z^2} \biggl)= \delta(t-t_s) \delta(x-x_s) \delta(y-y_s) \delta(z-z_s) \nonumber
\end{equation}
as
\begin{equation}
G_3(x,y,z,t) = \dfrac{1}{4 \pi V_{p0}^2 r}\delta\biggl((t-t_s)-\frac{r}{V_{p0}}\biggr) \nonumber
\end{equation}
with the source-receiver distance (offset)
$r = \sqrt{(x-x_s)^2+(y-y_s)^2+(z-z_s)^2}$
So the 3D Green's function for the homogeneous acoustic medium is a Delta distribution delayed by the traveltime between source and receiver. For the computation of the 3D Green's function, we have to approximate the  $\delta-$function. An example is the boxcar function
$$
\delta_{bc}(x) = \left{
\begin{array}{ll}
    1/dx &|x|\leq dx/2 \
    0 &\text{elsewhere}
\end{array}
\right.
$$
fulfilling the properties of the $\delta$ function as  $dx \rightarrow\; 0$. This function is used to properly scale the source term to obtain correct absolute amplitudes.
Can you write Python code for it?

# Calculation of 3D Green's function

G3=np.zeros(nt)                                # initialization G with zeros

r=np.sqrt((xs-xr)**2+(ys-yr)**2+(zs-zr)**2)    # defining offset
amp=1./(4*np.pi*(vp0**2)*r)                    # defining amplitudes
t_arr=ts+(r/vp0)                               # time arrival
i_arr=t_arr/dt
b=int(i_arr)
G3[b]= amp/dt

# Plotting Green's function in 3D
plt.plot(time, G3)
plt.title("Green's function for hom. 3D acoustic medium" )
plt.xlabel("Time, s")
plt.ylabel("Amplitude")
plt.xlim((0, tmax))
plt.grid()
plt.show()


I am working on the file "01_Analytical_solutions/5_Greens_function_acoustic_1-3D.ipynb".
The first task is:
Computation of Green's functions and seismograms for the acoustic wave equation
In the previous lessons we derived the equations of motion to describe wave propagation in 3D, 2D and 1D elastic and acoustic media. Before solving the underlying partial differential equations numerically using finite-differences (FD), we should estimate some analytical solutions.
This is not only useful to check if the FD codes contain any bugs, but also to get an idea of the accuracy of the numerical solution compared to the analytical.
1D Green's function
Let's start with a simple problem, like the equations of motion for a 1D acoustic medium assuming a constant density model, which we derived in Lesson 3 and 4:
\begin{equation}
\frac{\partial^2 P}{\partial t^2} - V_p^2\frac{\partial^2 P}{\partial x^2} = f\nonumber
\end{equation}
If we introduce the Dirac delta function
$$
\delta(x) = \left{
\begin{array}{ll}
    \infty &x=0 \
    0 &x\neq 0
\end{array}
\right.
$$
with the normalization condition
$$
\int_{-\infty}^{\infty} \delta(x)\; dx = 1
$$
as a source term of the acoustic wave equation:
\begin{equation}
f = \delta(t-t_s) \delta(x-x_s) \nonumber
\end{equation}
we call the solution of the 1D wave equation
\begin{equation}
\frac{\partial^2 G_1}{\partial t^2} - V_p^2 \frac{\partial^2 G_1}{\partial x^2} = \delta(t-t_s) \delta(x-x_s)
\end{equation}
Green's function $\mathbf{G_1(x,t;x_s,t_s)}$. This means that we place a source at $x = x_s$. The source time function describes the time-dependent behaviour of the source. In this case the source time function has an amplitude of $1\; \frac{Pa}{s^2}$ only at time $t = t_s$, otherwise the amplitude is zero. The pressure wavefield is recored at the receiver position x and time t.
In the following derivation of the Green's function, we assume the special case of a source located at $x_s = 0\; m$ and a source time $t_s = 0\; s$, which simplifies eq. (1) to
\begin{equation}
\frac{\partial^2 G_1}{\partial t^2} - V_p^2 \frac{\partial^2 G_1}{\partial x^2} = \delta(t)...
Can you write Python code for it?

# Import Libraries
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

# Define parameters
vp0 = 1.        # velocity m/s
r = 2.          # distance from source
tmax = 5.       # length of seismogram (s)
nt = 3000       # number of time samples
dt = tmax/nt    # time increment
ts = 0          # source time

# Acquisition geometry
xs=0            # coordinates of source
ys=0
zs=0

xr=r            # coordinates of receiver
yr=0
zr=0

# Define time vector
time = np.arange(0,tmax,dt)

# Calculating Green's function in 1D

G1=np.zeros(nt)      # initialization G with zeros

for i in range (nt):
    if (((time[i]-ts)-abs(xr-xs)/vp0)>=0):
        G1[i]=1./(2*vp0)
    else:
        G1[i]=0

# Plotting Green's function in 1D
plt.plot(time, G1)
plt.title("Green's function for hom. 1D acoustic medium" )
plt.xlabel("Time, s")
plt.ylabel("Amplitude")
plt.grid()
plt.show()


I am working on the file "01_Analytical_solutions/5_Greens_function_acoustic_1-3D.ipynb".
The first task is:
Excerise
Compute seismograms $G_{seis}(x,t)$ for the 1D, 2D and 3D acoustic media, by a convolution of the Green's function $G(x,t;x_s,t_s)$ with the source wavelet $s(t)$:
$$G_{seis}(x,t) = G(x,t;x_s,t_s) * s(t)$$
Plot the resulting seismograms together with the Green's function solutions.
Hints:
* Use the NumPy function np.convolve.
* How could you check if your implemented convolution is correct?
Can you write Python code for it?

# Computation of 1D seismogram

# Convolution of Green's function with the 1st derivative of a Gaussian
# COMPUTE YOUR SEISMOGRAM HERE!
#G1_seis=

# Plotting Green's function in 1D
plt.plot(time, G1)
plt.title("Green's function for hom. 1D acoustic medium" )
plt.xlabel("Time, s")
plt.ylabel("Amplitude")
plt.grid()
plt.show()

# Plotting convolved Green's function in 1D
# PLOT YOUR SEISMOGRAM HERE!
# plt.plot()
plt.title('After convolution')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.xlim (0, tmax)
plt.grid()
plt.show()

# Convolution of Green's function with the 1st derivative of a Gaussian
# COMPUTE YOUR SEISMOGRAM HERE!
#G2_seis=

# Plotting Green's function in 2D
plt.plot(time, G2)
plt.title("Green's function in 2D" )
plt.xlabel("Time, s")
plt.ylabel("Amplitude")
plt.xlim((0, tmax))
plt.grid()
plt.show()

# Plotting convolved Green's function in 1D
# PLOT YOUR SEISMOGRAM HERE!
# plt.plot()
plt.title('After convolution')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.xlim((0, tmax))
plt.grid()

# Convolution of Green's function with the 1st derivative of a Gaussian
# COMPUTE YOUR SEISMOGRAM HERE!
#G3_seis =

# Plotting Green's function in 3D
plt.plot(time, G3)
plt.title("Green's function in 3D" )
plt.xlabel("Time, s")
plt.ylabel("Amplitude")
plt.xlim((0, tmax))
plt.grid()
plt.show()

# Plotting convolved Green's function in 1D
# PLOT YOUR SEISMOGRAM HERE!
# plt.plot()
plt.title('After convolution')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.xlim (0, tmax)
plt.grid()
plt.show()


I am working on the file "01_Analytical_solutions/lecture_notebooks/5_Greens_function_acoustic_1-3D.ipynb".
The first task is:
2D Green's function
Using the same approach as for the 1D medium, we can calculate the 2D Green's function $\mathbf{G_2(x,t;x_s,t_s)}$, which is governed by
\begin{equation}
\frac{\partial^2 G_2}{\partial t^2} - V_{p0}^2 \biggl(\frac{\partial^2 G_2}{\partial x^2} + \frac{\partial^2 G_2}{\partial z^2} \biggl)= \delta(t-t_s) \delta(x-x_s) \delta(z-z_s) \nonumber
\end{equation}
as
\begin{equation}
G_2(x,z,t) = \dfrac{1}{2\pi V_{p0}^2}\dfrac{H\biggl((t-t_s)-\dfrac{|r|}{V_{p0}}\biggr)}{\sqrt{(t-t_s)^2-\dfrac{r^2}{V_{p0}^2}}} \nonumber
\end{equation}
with the source-receiver distance (offset)
$r = \sqrt{(x-x_s)^2+(z-z_s)^2}$
Compared to the 1D Green's function, we have a damped Heaviside function due to the radiation characteristic of the infinite line source, introduced by the 2D approximation. Let's also plot the 2D Green's function.
Can you write Python code for it?

# Calculation of 2D Green's function

                    # initialization G with zeros

# Plotting Green's function in 2D
plt.plot(time, G2)
plt.title("Green's function for hom. 2D acoustic medium" )
plt.xlabel("Time, s")
plt.ylabel("Amplitude")
plt.xlim((0, tmax))
plt.grid()
plt.show()


I am working on the file "01_Analytical_solutions/lecture_notebooks/5_Greens_function_acoustic_1-3D.ipynb".
The first task is:
Computation of Green's functions and seismograms for the acoustic wave equation
In the previous lessons we derived the equations of motion to describe wave propagation in 3D, 2D and 1D elastic and acoustic media. Before solving the underlying partial differential equations numerically using finite-differences (FD), we should estimate some analytical solutions.
This is not only useful to check if the FD codes contain any bugs, but also to get an idea of the accuracy of the numerical solution compared to the analytical.
1D Green's function
Let's start with a simple problem, like the equations of motion for a 1D acoustic medium assuming a constant density model, which we derived in Lesson 3 and 4:
\begin{equation}
\frac{\partial^2 P}{\partial t^2} - V_p^2\frac{\partial^2 P}{\partial x^2} = f\nonumber
\end{equation}
If we introduce the Dirac delta function
$$
\delta(x) = \left{
\begin{array}{ll}
    \infty &x=0 \
    0 &x\neq 0
\end{array}
\right.
$$
with the normalization condition
$$
\int_{-\infty}^{\infty} \delta(x)\; dx = 1
$$
as a source term of the acoustic wave equation:
\begin{equation}
f = \delta(t-t_s) \delta(x-x_s) \nonumber
\end{equation}
we call the solution of the 1D wave equation
\begin{equation}
\frac{\partial^2 G_1}{\partial t^2} - V_p^2 \frac{\partial^2 G_1}{\partial x^2} = \delta(t-t_s) \delta(x-x_s)
\end{equation}
Green's function $\mathbf{G_1(x,t;x_s,t_s)}$. This means that we place a source at $x = x_s$. The source time function describes the time-dependent behaviour of the source. In this case the source time function has an amplitude of $1\; \frac{Pa}{s^2}$ only at time $t = t_s$, otherwise the amplitude is zero. The pressure wavefield is recored at the receiver position x and time t.
In the following derivation of the Green's function, we assume the special case of a source located at $x_s = 0\; m$ and a source time $t_s = 0\; s$, which simplifies eq. (1) to
\begin{equation}
\frac{\partial^2 G_1}{\partial t^2} - V_p^2 \frac{\partial^2 G_1}{\partial x^2} = \delta(t)...
Can you write Python code for it?

# Import Libraries
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

# Define parameters
        # velocity m/s
        # distance from source
        # length of seismogram (s)
        # number of time samples
        # time increment
        # source time

# Acquisition geometry
            # coordinates of source

            # coordinates of receiver

# Define time vector

# Calculating Green's function in 1D

      # initialization G with zeros

# Plotting Green's function in 1D
plt.plot(time, G1)
plt.title("Green's function for hom. 1D acoustic medium" )
plt.xlabel("Time, s")
plt.ylabel("Amplitude")
plt.grid()
plt.show()


I am working on the file "01_Analytical_solutions/lecture_notebooks/5_Greens_function_acoustic_1-3D.ipynb".
The first task is:
Exercise
Derive the Green's function solutions of the 2D elastic SH problem:
\begin{align}
\rho\frac{\partial^2 u_y}{\partial t^2} - \frac{\partial}{\partial x} \mu \frac{\partial u_y}{\partial x} - \frac{\partial}{\partial z} \mu \frac{\partial u_y}{\partial z} &= f_y\nonumber \
\end{align}
and 1D elastic SH problem:
\begin{align}
\rho\frac{\partial^2 u_y}{\partial t^2} - \frac{\partial}{\partial x} \mu \frac{\partial u_y}{\partial x} &= f_y.\nonumber \
\end{align}
Assume a constant distribution of the shear modulus $\mu(x,z) = \mu_0 =  const. \neq 0\; Pa$ and density $\rho(x,z) = \rho_0 = const. \neq 0\; \frac{kg}{m^3}$ in the subsurface.
Hint: To solve this problem, you do not have to apply any Fourier transform or integrate along contours around poles
Computation of seismograms
In field data applications we can not excitate a source time function like a delta distribution, which would have a perfect white spectrum (all frequencies are excitated at once). As we will see later, some numerical problems arise when explicitly calculating Green's function with the time-domain finite-difference approach.
Instead we have to rely on band-limited source signals. Seismograms for an arbritary source wavelet can be computed from the the Green's function. In the following example, the source wavelet consists of the first derivative of the Gaussian.
Can you write Python code for it?

# Defining source time function
           # Frequency (Hz)
           # period
           # defining t0

# Initialization of source-time function

# Initialization of first derivative of gaussian

# Plotting of source time function
plt.plot(time, src)
plt.title('Source time function')
plt.xlabel('Time, s')
plt.ylabel('Amplitude')
plt.grid()
plt.show()


I am working on the file "01_Collecting_tweets_using_Twitter_API.ipynb".
The first task is:
Collecting tweets from the Streaming API
source : http://tweepy.readthedocs.io/en/v3.5.0/streaming_how_to.html
Step 1: Creating a StreamListener
This simple stream listener prints status text. The on_data method of Tweepy’s StreamListener conveniently passes data from statuses to the on_status method.
Create class MyStreamListener inheriting from StreamListener and overriding on_status.:
Can you write Python code for it?

#override tweepy.StreamListener to make it print tweet content when new data arrives
class MyStreamListener(tweepy.StreamListener):

    def on_status(self, status):
        print(status.text)


I am working on the file "01_Collecting_tweets_using_Twitter_API.ipynb".
The first task is:
Create a Twitter app and find your consumer token and secret

go to https://apps.twitter.com/
click Create New App
fill in the details
click on manage keys and access tokens
copy paste your Consumer Key (API Key) and Consumer Secret (API Secret) below:
click create my access token
Can you write Python code for it?

consumer_key = 'xxx'
consumer_secret = 'xxx'
access_token = 'xxx'
access_token_secret = 'xxx'


I am working on the file "01_Collecting_tweets_using_Twitter_API.ipynb".
The first task is:
Saving the stream to a file
Lets' define a new StreamListener that will save the collected data to a file
Can you write Python code for it?

#override tweepy.StreamListener to make it save data to a file
class StreamSaver(tweepy.StreamListener):
    def __init__(self, filename, max_num_tweets=2000, api=None):
        self.filename = filename

        self.num_tweets = 0

        self.max_num_tweets = max_num_tweets

        tweepy.StreamListener.__init__(self, api=api)

    def on_data(self, data):
        #print json directly to file

        with open(self.filename,'a') as tf:
            tf.write(data)

        self.num_tweets += 1

        print(self.num_tweets)

        if self.num_tweets >= self.max_num_tweets:
            return True

    def on_error(self, status):
        print(status)

# create the new StreamListener and stream object that will save collected tweets to a file
saveStream = StreamSaver(filename='testTweets.txt')
mySaveStream = tweepy.Stream(auth = api.auth, listener=saveStream)

mySaveStream.filter(track=['realdonaldtrump,trump'], languages=['en'])

mySaveStream.disconnect()


I am working on the file "01_Collecting_tweets_using_Twitter_API.ipynb".
The first task is:
Step 2: Creating a Stream
Using the api object we created and the StreamListener we can create a Stream Object:
Can you write Python code for it?

myStreamListener = MyStreamListener()
myStream = tweepy.Stream(auth = api.auth, listener=myStreamListener)


I am working on the file "01_Collecting_tweets_using_Twitter_API.ipynb".
The first task is:
Step 3: Starting a Stream
A number of twitter streams are available through Tweepy. Most cases will use filter, the user_stream, or the sitestream. For more information on the capabilities and limitations of the different streams see Twitter Streaming API Documentation
In this example we will use filter to stream all tweets containing the word python. The track parameter is an array of search terms to stream.
Can you write Python code for it?

myStream.filter(track=['new york'])

myStream.disconnect()

myStream.filter(track=['realdonaldtrump,trump'], languages=['en'])

myStream.disconnect()

# streaming tweets from a given location
# we need to provide a comma-separated list of longitude,latitude pairs specifying a set of bounding boxes
# for example for New York
myStream.filter(locations=[-74,40,-73,41])

myStream.disconnect()


I am working on the file "01_CreateDataForPrediction.ipynb".
The first task is:
Convert Folia Files
Because the folia files in corpus big (additional folia files) are not as nicely structured into acts as the texts for annotation, a custom script must be used.
Can you write Python code for it?

# convert additional folia files to input for prediction
# usage: !./batch_do_python.sh embem/machinelearningdata/folia2dataset_for_prediction.py <directory with input folia files> <directory for output files>

!./batch_do_python.sh embem/machinelearningdata/folia2dataset_for_prediction.py /home/jvdzwaan/data/embem_corpus_big/ /home/jvdzwaan/data/embem/txt/corpus_big-for_prediction

# convert annotation folia files to input for prediction
# usage: !./batch_do_python.sh embem/machinelearningdata/folia2dataset_for_prediction.py <directory with input folia files> <directory for output files>

!./batch_do_python.sh embem/machinelearningdata/folia2dataset_for_prediction.py /home/jvdzwaan/data/folia-annotatie/ /home/jvdzwaan/data/embem/txt/annotation-for_prediction


I am working on the file "01_CreateDataForPrediction.ipynb".
The first task is:
Create normalized data for prediction
Once text files for prediction have been created, these should be normalized
(the best performing classifiers are trained on normalized data).
Can you write Python code for it?

# normalize text data
# usage: %run embem/spellingnormalization/normalize_dataset.py <directory with input text files> <directory for output files>

# corpus big (nederlab texts)
#%run embem/spellingnormalization/normalize_dataset.py /home/jvdzwaan/data/embem/txt/corpus_big-for_prediction /home/jvdzwaan/data/embem/txt/corpus_big-for_prediction-normalized

# ceneton texts
%run embem/spellingnormalization/normalize_dataset.py /home/jvdzwaan/data/embem/txt/ceneton-for_prediction /home/jvdzwaan/data/embem/txt/ceneton-for_prediction-normalized

# edbo texts
#%run embem/spellingnormalization/normalize_dataset.py /home/jvdzwaan/data/embem/txt/edbo-for_prediction /home/jvdzwaan/data/embem/txt/edbo-for_prediction-normalized


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
2. Prepare data
Concat prior and train
To be used for final predictions
Can you write Python code for it?

prior_train = pd.concat([priors, train_all], axis = 0)
print prior_train.shape

if ( priors.shape[0] + train.shape[0] ) ==  prior_train.shape[0]:
    print "concat successful"


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
3.  Feature Generation
*Features built *

Order features

Current Order -  u'order_dow', u'order_hour_of_day', u'days_since_prior_order'
Prev Order - u'prev_order_dow',u'prev_order_hour_of_day', u'prev_days_since_prior_order'

User Order Features

u'avg_days_since_prior_order

Product Features

u'aisle_id', u'department_id'
u'p_orders', u'p_reorders', u'p_reorder_rate'

User Features

u'tot_orders', u'tot_prods', u'avg_basket', u'avg_reorder', u'std_basket'

User Product Features

u'up_orders', u'up_reorder', u'up_reorder_rate'

User Order features
How much time on an average does a user leave between orders
Can you write Python code for it?

## using prior stats

user_order_prior = pd.DataFrame()
user_order_prior['avg_days_since_prior_order'] = priors.groupby('user_id')['days_since_prior_order'].agg('mean')

## using prior train stats
user_order_all = pd.DataFrame()
user_order_all['avg_days_since_prior_order'] = prior_train.groupby('user_id')['days_since_prior_order'].agg('mean')

user_order_prior.reset_index(drop = False, inplace = True)
user_order_all.reset_index(drop = False, inplace = True)

print user_order_prior.head()
print
print user_order_all.head()


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
Create train eval
Keep 25% of the train data outside for cross validation
Can you write Python code for it?

"""
import random
train.reset_index(drop = True, inplace = True)
order_ids = list(train['order_id'].unique())
sample_size = int(0.25 * len(order_ids))
sample_orders = random.sample(order_ids, sample_size)

train_eval = train[train['order_id'].isin(sample_orders)]
train_new = train[~train['order_id'].isin(sample_orders)]

train_eval.to_csv('./data/train_eval.csv', index = False)
train_new.to_csv('./data/train_new.csv', index = False)

del train
train = train_new
del train_new
train.head()

"""


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
Join with orders table to get the user id
This gets us the user_id and other order features
Can you write Python code for it?

## set the index, for the join
orders.set_index('order_id', inplace=True, drop=False)

## Join with prior_train
print "Joining orders with prior_train"
prior_train = prior_train.join(orders, on = 'order_id', how = 'left', lsuffix = '_')
prior_train.drop('order_id_', inplace = True, axis = 1)

## Repeat the same only for prior
print "Joining orders with priors"
priors = priors.join(orders, on = 'order_id', how = 'left', lsuffix = '_')
priors.drop('order_id_', inplace = True, axis = 1)

## Joining orders with train
print "Joining orders with train"
train = train.join(orders, on = 'order_id', how = 'left', lsuffix = '_')
train.drop('order_id_', inplace = True, axis = 1)

## Joining orders with train_eval
print "Joining orders with train_eval"
train_eval = train_eval.join(orders, on = 'order_id', how = 'left', lsuffix = '_')
train_eval.drop('order_id_', inplace = True, axis = 1)

## reset the order table index
orders.reset_index(inplace=True, drop=True)
orders.head()


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
Load Products
Can you write Python code for it?

print('loading products')

products = pd.read_csv(IDIR + 'products.csv')

products.drop(['product_name'], axis=1, inplace=True)
products.aisle_id = products.aisle_id.astype(np.int8)
products.department_id = products.department_id.astype(np.int8)
products.product_id = products.product_id.astype(np.int32)


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
Make a data frame of user and previous product list
Can you write Python code for it?

## Using prior and train data
users_prior_all = pd.DataFrame()
users_prior_all['prod_list'] = prior_train.groupby('user_id')['product_id'].apply(set)
users_prior_all.reset_index(inplace = True, drop = False)

## Using only prior data
users_prior = pd.DataFrame()
users_prior['prod_list'] = priors.groupby('user_id')['product_id'].apply(set)
users_prior.reset_index(inplace = True, drop = False)
users_prior.head()


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
Order Feature
Can you write Python code for it?

## Order features

print "Order features"

test_df.set_index('order_id',inplace = True, drop = False)
orders.set_index('order_id', inplace = True, drop = False)

test_df = pd.merge(test_df, orders, left_on = 'order_id', right_on = 'order_id')

test_df.drop('eval_set', inplace = True, axis =1)
test_df.drop('user_id_y', inplace = True, axis =1)
test_df.drop('order_number', inplace = True, axis =1)

test_df = test_df.rename(columns={"user_id_x":"user_id"})

test_df.reset_index(drop = True, inplace= True)
train.reset_index(drop = True, inplace = True)

test_df.head()

test_df = join_features(feature_dict_all, test_df)
test_df.head()

test_df.to_csv('./features/features_test.csv',index = False)


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
Order Features
Attributes of the order we are about to predict
Can you write Python code for it?

print "Order features"

features.set_index('order_id',inplace = True, drop = False)
orders.set_index('order_id', inplace = True, drop = False)

features = pd.merge(features, orders, left_on = 'order_id', right_on = 'order_id')

#features.drop('order_id', inplace = True, axis =1)
features.drop('eval_set', inplace = True, axis =1)
features.drop('user_id_y', inplace = True, axis =1)
features.drop('order_number', inplace = True, axis =1)

features = features.rename(columns={"user_id_x":"user_id"})

features.reset_index(drop = True, inplace= True)
train.reset_index(drop = True, inplace = True)

features.head()

features = join_features(feature_dict_prior, features)
features.head()

features.to_csv('./features/features_train.csv', index = False)


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
Prepare Y Variable from train

Here we use the train table as our y variable. We create a dataset with user_id, product_id, is_product_in_train columns.

We can then use this to merge with user, product and user/product features.

This will serve as our testing data before use actually run our model on real data.
Can you write Python code for it?

## This block needs to be run only once
## Later the output of this block is stored in ./data/features.csv file
## The next block reads the file, it enough to run the next block subsequently

## We could have got this from order_id
## However since we have separted our train into two sets
## we need to interate the train_new aka train data we have created.

train.reset_index(inplace = True, drop = True)

train_list = pd.DataFrame()
train_list['ignore'] = train.groupby(['user_id','order_id'], group_keys = True).size()
train_list.reset_index(inplace = True, drop = False)

train.set_index(['order_id', 'product_id'], inplace = True, drop = False)

print "features"
count = 0

order_list = []
product_list  = []
user_list = []
labels = []

for user_record  in train_list.itertuples():
    count+=1
    if count%10000 == 0:
        print "Finished {} users".format(count)

    user_id  = user_record.user_id
    order_id = user_record.order_id
    prev_products = list(users_prior[users_prior.user_id == user_id]['prod_list'].values.tolist()[0])
    product_list+= prev_products
    order_list+=[order_id] * len(prev_products)
    user_list+=[user_id] * len(prev_products)
    labels+=[(order_id, product) in train.index for product in prev_products]

feature_df = pd.DataFrame({'user_id':user_list,'product_id':product_list,'order_id':order_list,'in_next_order':labels}, dtype=np.int32)

print feature_df.head()

feature_df.to_csv('./features/features.csv', index = False)

features = pd.read_csv('./features/features.csv')
features.head()


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
Prepare Y Varible from eval
Can you write Python code for it?

train_eval.reset_index(inplace = True, drop = True)

train_eval_list = pd.DataFrame()
train_eval_list['ignore'] = train_eval.groupby(['user_id','order_id'], group_keys = True).size()
train_eval_list.reset_index(inplace = True, drop = False)

test_df = get_y(train_eval_list, users_prior, orders)
test_df.head()

test_df = join_features(feature_dict_prior, test_df)
test_df.head()

test_df.to_csv('./features/features_eval.csv',index = False)


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
Prepare y variable
Can you write Python code for it?

def get_y(test_list, users_prior):

    feature = []
    count = 0
    for user_record in (test_list.itertuples()):
        count+=1
        if count%10000 == 0:
            print "Finished {} users".format(count)
        user_id = user_record.user_id
        order_id = user_record.order_id
        prev_products = list(users_prior[users_prior.user_id == user_id]['prod_list'].values.tolist()[0])

        for p_p in prev_products:
            feature.append((order_id, user_id ,p_p))

    test_df = pd.DataFrame(data = feature, columns =['order_id','user_id','product_id'])

    return test_df

train_eval.head()


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
Product Features
Features of the product, including the number of orders in which the product has appeared, number of re-orders where it has appeared
and reorder rate
Can you write Python code for it?

def product_features(priors):

    prods = pd.DataFrame()
    prods['p_orders'] = priors.groupby(priors.product_id).size().astype(np.float32)
    prods['p_reorders'] = priors['reordered'].groupby(priors.product_id).sum().astype(np.float32)

    prods['p_reorder_rate'] = (prods.p_reorders / prods.p_orders).astype(np.float32)

    ## set the index for products
    products.set_index('product_id', inplace = True, drop = False)
    products_prior = products.join(prods, rsuffix="_")
    ## Reset the index
    products_prior.reset_index(inplace = True, drop = True)
    del prods
    return products_prior

### Stats from prior and train
products_all = product_features(prior_train)
print products_all.head()
print
## Stats from prior
products_prior = product_features(priors)
print products_prior.head()


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
Product days since last ordered
Create a feature dict
Can you write Python code for it?

feature_dict_prior = {}

feature_dict_prior[1] = {"name":"user_order_feature","obj":user_order_prior,"index":['user_id']}
feature_dict_prior[2] = {"name":"last_order_feature","obj":max_order,"index":['user_id']}
feature_dict_prior[3] = {"name":"product_feature","obj":products_prior,"index":['product_id']}
feature_dict_prior[4] = {"name":"user_feature","obj":prod_count_prior,"index":['user_id']}
feature_dict_prior[5] = {"name":"user_pro_feature","obj":user_prod_prior,"index":['user_id','product_id']}

feature_dict_all = {}

feature_dict_all[1] = {"name":"user_order_feature","obj":user_order_all,"index":['user_id']}
feature_dict_all[2] = {"name":"last_order_feature","obj":max_order_all,"index":['user_id']}
feature_dict_all[3] = {"name":"product_feature","obj":products_all,"index":['product_id']}
feature_dict_all[4] = {"name":"user_feature","obj":prod_count_all,"index":['user_id']}
feature_dict_all[5] = {"name":"user_prod_feature","obj":user_prod_all,"index":['user_id','product_id']}


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
User Product Features
Can you write Python code for it?

def user_prod_features(priors):
    user_prod_prior = pd.DataFrame()

    ## Number of user's order where product id is present
    user_prod_prior['up_orders'] = priors.groupby(['user_id','product_id'])['order_id'].size()

    user_prod_prior.reset_index(inplace = True, drop = False)
    user_prod_prior.set_index(['user_id', 'product_id'], inplace = True, drop = False)

    ## Number of times the product was re-ordered by the user
    user_prod_prior['up_reorder']      = priors.groupby(['user_id', 'product_id'])['reordered'].agg(['sum'])
    user_prod_prior['up_reorder_rate'] = user_prod_prior.up_reorder / user_prod_prior.up_orders

    user_prod_prior.reset_index(drop = True, inplace= True)
    return user_prod_prior

## Stats from prior

user_prod_prior = user_prod_features(priors)
user_prod_all   = user_prod_features(prior_train)

print user_prod_prior.head()
print
print user_prod_all.head()


I am working on the file "01_Feature_Generation.ipynb".
The first task is:
Vowpalwabbit - Experimental
Can you write Python code for it?

"""
## Numpy savetxt is extremely slow
#VW_train = np.column_stack((y_train, X_train))
#print "Save"
#np.savetxt('./data/vw_train.csv', VW_train)
#print "done"

VW_train = pd.concat([Y, X],axis =1 )
print VW_train.shape
VW_train.head()

print "VW_train"
VW_train.to_csv('./data/vw_train.csv', index = False)

#python csv2vw.py ./data/vw_train.csv ./data/vw_train.txt 0 1
#python csv2vw.py ./data/vw_test.csv ./data/vw_test.txt 0 1

### Vowpal wabbit baseline model

#time vw ./data/vw_train.txt --predictions vwpred_train.out

vw_pred_train = pd.read_csv('vwpred_train.out', names=['y_p'])
vw_pred_train['y_pp']= vw_pred_train['y_p'].apply(lambda x: 1.0 if x > 0.35 else 0.0)
y_p3 = vw_pred_train['y_pp'].values
print "Vowpal wabbit accuracy {0:.2f}, precision {0:.2f}, recall {0:.2f}, f1-score {0:.2f}".format(
             accuracy_score(y_train, y_p3),
             precision_score(y_train, y_p3),
             recall_score(y_train, y_p3),
             f1_score(y_train, y_p3))

print confusion_matrix(y_train, y_p3)
"""


I am working on the file "01_Image_Basics.ipynb".
The first task is:
Accessing Attributes
If you are familliar with ITK, then these methods will follow your expectations:
Can you write Python code for it?

print image.GetSize()
print image.GetOrigin()
print image.GetSpacing()
print image.GetDirection()
print image.GetNumberOfComponentsPerPixel()


I am working on the file "01_Image_Basics.ipynb".
The first task is:
Accessing Pixels
There are the member functions GetPixel and SetPixel which provides an ITK-like interface for pixel access.
Can you write Python code for it?

help(image.GetPixel)

print image.GetPixel(0, 0, 0)
image.SetPixel(0, 0, 0, 1)
print image.GetPixel(0, 0, 0)

print image[0,0,0]
image[0,0,0] = 10
print image[0,0,0]


I am working on the file "01_Image_Basics.ipynb".
The first task is:
Are we still dealing with Image, because I haven't seen one yet...
While SimpleITK does not do visualization, it does contain a built in Show method. This function writes the image out to disk and than launches a program for visualization. By default it is configured to use ImageJ, because it is readily supports all the image types which SimpleITK has and load very quickly. However, it's easily customizable by setting enviroment variables.
Can you write Python code for it?


I am working on the file "01_Image_Basics.ipynb".
The first task is:
Conversion between numpy and SimpleITK
Can you write Python code for it?

nda = sitk.GetArrayFromImage(image)
print nda

help(sitk.GetArrayFromImage)

nda = sitk.GetArrayFromImage(image_RGB)
img = sitk.GetImageFromArray(nda)
img.GetSize()

help(sitk.GetImageFromArray)

img = sitk.GetImageFromArray(nda, isVector=True)
print img


I am working on the file "01_Image_Basics.ipynb".
The first task is:
For certain file types such as DICOM, additional information about the image is contained in the meta-data dicitonary.
Can you write Python code for it?

for key in image.GetMetaDataKeys():
        print "\"{0}\":\"{1}\"".format(key, image.GetMetaData(key))


I am working on the file "01_Image_Basics.ipynb".
The first task is:
Image Construction
There are a variety of ways to create an image. All images' initial value is well defined as zero.
Can you write Python code for it?

image = sitk.Image(256, 128, 64, sitk.sitkInt16)
image_2D = sitk.Image(64, 64, sitk.sitkFloat32)
image_2D = sitk.Image([32,32], sitk.sitkUInt32)
image_RGB = sitk.Image([128,128], sitk.sitkVectorUInt8, 3)


I am working on the file "01_Image_Basics.ipynb".
The first task is:
Since the dimension  and pixel type of a SimpleITK image is determined at run-time accessors are needed.
Can you write Python code for it?

print image.GetDimension()
print image.GetPixelIDValue()
print image.GetPixelIDTypeAsString()


I am working on the file "01_Image_Basics.ipynb".
The first task is:
The order of index and dimensions need careful attention during conversion
ITK's Image class does not have a bracket operator. It has a GetPixel which takes an ITK Index object as an argument, which is an array ordered as (x,y,z). This is the convention that SimpleITK's Image class uses for the GetPixel method as well.
While in numpy, an array is indexed in the opposite order (z,y,x).
Can you write Python code for it?

print img.GetSize()
print nda.shape
print nda.shape[::-1]


I am working on the file "01_Image_Basics.ipynb".
The first task is:
What is the dimension and size of a Vector image?
Can you write Python code for it?

print image_RGB.GetDimension()
print image_RGB.GetSize()

print image_RGB.GetNumberOfComponentsPerPixel()


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:

Tip de IPython:
`%run` es un _comando mágico_ del notebook que te permite ejecutar un archivo.

Si quieres hacerlo desde una línea de comandos podrías hacer:

`$ python3 static/mi_primer_script.py`

El método más simple es usar un editor (tu preferido) y ejecutar el script desde la línea de comandos. Pero existen también __IDE__s (integrated development environment pensados para facilitar la escritura de código y tener al alcance de la mano otras herramientas como profilers, debuggers, explorador de variables... Entre los más adecuados para la programación científica se encuentran IEP y Spyder (instalado con Anaconda).

¿Qué es IPython?
IPython no es más que un [intérprete](http://es.wikipedia.org/wiki/Int%C3%A9rprete_(inform%C3%A1tica) de Python con algunas mejoras sustanciales, pero además su interfaz notebook es más cómoda de manejar que la línea de comandos y nos da un poco más de flexibilidad.
Notebook de Jupyter
Será nuestra herramienta de trabajo durante el curso. Esto que estás leyendo ahora no es más que un notebook de IPython, que como diremos luego además de código puede contener texto e imágenes. Pero veamos primero cómo funciona.
Al iniciar el notebook de Jupyter, en la pantalla principal podemos ver una ruta y una lista de notebooks. Cada notebook es un archivo que está almacenado en el ordenador en la ruta que aparece. Si en esa carpeta no hay notebooks, veremos un mensaje indicando que la lista de notebooks está vacía.
Al crear un notebook o al abrir uno nuevo se abre la interfaz de Jupyter propiamente dicha donde ya podemos empezar a trabajar. Es similar a un intérprete, pero está dividida en celdas. Las celdas pueden contener, código, texto, imágenes...
Cada celda de código está marcada por la palabra In [<n>] y están numeradas. Tan solo tenemos que escribir el código en ella y hacer click arriba en Cell -> Run, el triángulo ("Run cell") o usar el atajo shift + Enter....
Can you write Python code for it?

from IPython.display import Image
Image(url="static/markdown_cell.gif")
# Fuente Practical Numerical Methods with Python
# http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:

Tip de IPython: podemos recuperar resultados pasados usando `_<n>`. Por ejemplo, para recuperar el resultado correspondiente a `Out [7]`, usaríamos `_7`. Esta variable guarda ese valor para toda la sesión.

Can you write Python code for it?


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:

Tip:
`ii += 1` equivale a `ii = ii + 1`. En el segundo Python, realiza la operación ii + 1 creando un nuevo objeto con ese valor y luego lo asigna a la variable ii; es decir, existe una reasignación. En el primero, sin embargo, el incremento se produce sobre la propia variable. Esto puede conducirnos a mejoras en velocidad.

Otros operadores 'in-place' son: `-=`, `*=`, `/=`

Se puede interrumpir el bucle a la mitad con la sentencia break:
Can you write Python code for it?

ii = 0
while ii < 5:
    print(ii)
    ii += 1
    if ii == 3:
        break


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:

Podemos realizar **asignación múltiple**, que hemos hecho en la celda anterior con las variables `x` e `y` para intercambiar valores de manera intuitiva:

Can you write Python code for it?


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
El Notebook tiene además numerosos atajos que irás aprendiendo sobre la marcha, puedes consultarlos en Help > Keyboard Shortcourts
Introducción a la sintaxis de Python
Tipos numéricos
Python dispone de los tipos numéricos y las operaciones más habituales:
Can you write Python code for it?


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
En Python la asignación no imprime el resultado por pantalla, al contrario de como sucede en MATLAB y Octave (salvo que se incluya el punto y coma al final). La mejor manera de visualizar la variable que acabamos de asignar es esta:
Can you write Python code for it?


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
En los dos tipos podemos:

Comprobar si un elemento está en la secuencia con el operador in:
Can you write Python code for it?


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
En una celda podemos escribir código que ocupe varias líneas. Si la última de ellas devuelve un resultado, este se imprimirá.
Can you write Python code for it?


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Esto nos será de gran ayuda en el futuro para construir arrays.
Estructuras de control (I): Condicionales
if <condition>:
    <do something>
elif <condition>:
    <do other thing>
else:
    <do other thing>

Importante: En Python los bloques se delimitan por sangrado, utilizando siempre cuatro espacios. Cuando ponemos los dos puntos al final de la primera línea del condicional, todo lo que vaya a continuación con *un* nivel de sangrado superior se considera dentro del condicional. En cuanto escribimos la primera línea con un nivel de sangrado inferior, hemos cerrado el condicional. Si no seguimos esto a rajatabla Python nos dará errores; es una forma de forzar a que el código sea legible.

Can you write Python code for it?

print(x,y)
if x > y:
    print("x es mayor que y")
    print("x sigue siendo mayor que y")

if 1 < 0:
    print("1 es menor que 0")
print("1 sigue siendo menor que 0")  # <-- ¡Mal!

if 1 < 0:
    print("1 es menor que 0")
     print("1 sigue siendo menor que 0")


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Los operadores de comparación son:

== igual a
!= distinto de
< menor que
<= menor o igual que

Devolverán un booleano: True o False
Can you write Python code for it?

x == y

print(x != y)

print(x < y)
print(x <= y)
print(x > y)
print(x >= y)

# incluso:
x = 5.
6. < x < 8.


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Otro tipo que nos resultará muy útil son los complejos:
Can you write Python code for it?

2 + 3j

1j

# Valor absoluto
abs(2 + 3j)


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Otros tipos de datos
Otro tipo de datos muy importante que vamos a usar son las secuencias: las tuplas y las listas. Ambos son conjuntos ordenados de elementos: las tuplas se demarcan con paréntesis y las listas con corchetes.
Can you write Python code for it?

una_lista = [1, 2, 3.0, 4 + 0j, "5"]
una_tupla = (1, 2, 3.0, 4 + 0j, "5")
print(una_lista)
print(una_tupla)
print(una_lista == una_tupla)


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
PEP 8
La guía de estilo:

Usa sangrado de 4 espacios, no tabuladores [IPython o tu editor se encargan de ello].
Acota las líneas a 79 caracteres.
Usa líneas en blanco para separar funciones y bloques de código dentro de ellas.
Pon los comentarios en líneas aparte si es posible.
Usa cadenas de documentación (docstrings).
Pon espacios alrededor de los operadores y después de coma.
Usa la convención minuscula_con_guiones_bajos para los nombres de las funciones y las variables.
Aunque Python 3 te lo permite, no uses caracteres especiales para los identificadores.

(Traducido de http://docs.python.org/3/tutorial/controlflow.html#intermezzo-coding-style)
Utilizando el módulo pep8
https://pypi.python.org/pypi/pep8
Y la extensión pep8magic
https://gist.github.com/Juanlu001/9082229/
Podemos comprobar si una celda de código cumple con las reglas del PEP8:

Hemos visto como la sintaxis de Python nos facilita escribir código legible así como aprendido algunas buenas prácticas al programar. Características como el tipado dinámico (no hace falta declarar variables) y ser lenguaje interpretado (no hace falta compilarlo) hacen que el tiempo que pasamos escrbiendo código sea menos que en otro tipo de lenguajes.
Se han presentado los tipos de variables, así como las estructuras de control básicas. En la siguiente clase practicaremos con algunos ejercicios para que te familiarices con ellas
Esperamos también que poco a poco te sientas cada vez más a gusto con el Notebook de IPython y puedas sacarle todo el partido
Referencias

Tutorial de Python oficial actualizado y traducido al español http://docs.python.org.ar/tutorial/
Vídeo de 5 minutos de IPython http://youtu.be/C0D9KQdigGk
Introducción a la programación con Python, Universitat Jaume I http://www.uji.es/bin/publ/edicions/ippython.pdf
PEP8 http://www.python.org/dev/peps/pep-0008/

Si te ha gustado esta clase:
<a href="https://twitter.com/share" class="twitter-share-button" data-url="https://github.com/CAChemE/curso-python-cientifico" data-text="Aprendiendo Python con"...
Can you write Python code for it?

%%html
Follow @Pybonacci

# Esta celda da el estilo al notebook
from IPython.core.display import HTML
css_file = 'static/styles/style.css'
HTML(open(css_file, "r").read())


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Podemos comprobar el tipo de una variable:
Can you write Python code for it?


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Podemos convertir variables a int, float, complex, str...
Can you write Python code for it?


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Podemos indexar las secuencias, utilizando la sintaxis [<inicio>:<final>:<salto>]:
Can you write Python code for it?

print(una_lista[0])  # Primer elemento, 1
print(una_tupla[1])  # Segundo elemento, 2
print(una_lista[0:2])  # Desde el primero hasta el tercero, excluyendo este: 1, 2
print(una_tupla[:3])  # Desde el primero hasta el cuarto, excluyendo este: 1, 2, 3.0
print(una_lista[-1])  # El último: 4 + 0j
print(una_tupla[:])  # Desde el primero hasta el último
print(una_lista[::2])  # Desde el primero hasta el último, saltando 2: 1, 3.0


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Saber cuandos elementos tienen con la función len:
Can you write Python code for it?


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Se puede elevar un número a otro con el operador **:
Can you write Python code for it?


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Si la ordenación no tiene sentido nos devolverá un error:
Can you write Python code for it?

1 + 1j < 0 + 1j

# En las cadenas de texto sí existe un orden
'aaab' > 'ba'


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Si queremos añadir ramas adicionales al condicional, podemos emplear la sentencia elif (abreviatura de else if). Para la parte final, que debe ejecutarse si ninguna de las condiciones anteriores se ha cumplido, usamos la sentencia else:
Can you write Python code for it?

print(x,y)
if x > y:
    print("x es mayor que y")
else:
    print("x es menor que y")

print(x, y)
if x < y:
    print("x es menor que y")
elif x == y:
    print("x es igual a y")
else:
    print("x no es ni menor ni igual que y")


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
También ecuaciones en latex y mucho más. Esto es una herramienta muy potente para explicar a alguien o a ti mismo lo que tu código hace, para hacer un informe, un trabajo, escribir en un blog...
Markdown es un lenguaje aparte, no te preocupes por él demasiado ahora, irás aprendiendo sobre la marcha... Para cuando lo vayas necesitando, aquí tienes una chuleta.
Can you write Python code for it?

Image(url="static/markdown_math.gif")
# Fuente Practical Numerical Methods with Python
# http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Un bloque else justo después del bucle se ejecuta si este no ha sido interrumpido por nosotros:
Can you write Python code for it?

ii = 0
while ii < 5:
    print(ii)
    ii += 1
    if ii == 3:
        break
else:
    print("El bucle ha terminado")

ii = 0
while ii < 5:
    print(ii)
    ii += 1
    #if ii == 3:
        #break
else:
    print("El bucle ha terminado")


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
Veremos más cosas acerca de indexación en NumPy, así que de momento no te preocupes. Sólo recuerda una cosa:
¡En Python, la indexación empieza por CERO!
Podemos complicarlo un poco más y hacer cosas como una lista de listas:
Can you write Python code for it?

mis_asignaturas = [
['Álgebra', 'Cálculo', 'Física'],
['Mecánica', 'Termodinámica'],
['Sólidos', 'Electrónica']
]
mis_asignaturas


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
for
El otro bucle en Python es el bucle for, y funciona de manera un que puede resultar chocante al principio. La idea es recorrer un conjunto de elementos:
for <element> in <iterable_object>:
    <do whatever...>
Can you write Python code for it?

for ii in (1,2,3,4,5):
    print(ii)

for nombre in "Juan", "Luis", "Carlos":
    print(nombre)

for ii in range(3):
    print(ii)

for jj in range(2, 5):
    print(jj)


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
¡Acabas de utilizar funciones! Como ves es una manera bastante estándar: los argumentos se encierran entre paréntesis y se separan por comas. Se hace de esta manera en otros lenguajes de programación y no requiere mayor explicación, de momento.

La función print es la que se usa para imprimir resultados por pantalla. Por si lo ves en algún sitio, en Python 2 era una sentencia y funcionaba de manera distinta, sin paréntesis y sin posibilidad de pasar argumentos adicionales.

Asignación y operadores de comparación
La asignación se realiza con el operador =. Los nombres de las variables en Python pueden contener caracteres alfanuméricos (empezando con una letra) a-z, A-Z, 0-9 y otros símbolos como la _.
Por cuestiones de estilo, las variables suelen empezar con minúscula, reservando la mayúcula para clases.
Algunos nombres no pueden ser usados porque son usados por python:
and, as, assert, break, class, continue, def, del, elif, else, except, exec, finally, for, from, global, if, import, in, is, lambda, not, or, pass, print, raise, return, try, while, with, yield
Can you write Python code for it?


I am working on the file "01_Intro-Python-IPython.ipynb".
The first task is:
¿Qué pinta tiene un programa en Python y cómo lo ejecuto?
Vamos a ver mi_primer_script.py que está en la carpeta static. De momento no te preocupes por el código, ya habrá tiempo para eso...
Can you write Python code for it?


I am working on the file "01_Mine/Block Berechnungen.ipynb".
The first task is:
Parzelle 346 "Lorena"
An der Hauptstrasse
Can you write Python code for it?

members = 4
area = 1853 # m^2
price_per_m2 = 140 # Fr/m^2
soil_calc(area,price_per_m2,members)


I am working on the file "01_Mine/Block Berechnungen.ipynb".
The first task is:
Parzelle 7337 & 7338 "Kössler"
Can you write Python code for it?

members = 4

area = 1069 + 567 # m^2
price_per_m2 = 200 # Fr/m^2
soil_calc(area,price_per_m2,members)

price_per_m2 = 160 # Fr/m^2
soil_calc(area,price_per_m2,members)


I am working on the file "01_Mine/Block Berechnungen.ipynb".
The first task is:
Vergleich Susten - Salgesch Musterhaus
Vergleich Susten vs Salgesch von Gervas
Can you write Python code for it?

members = 4
# Susten
area = 1100 # m^2
price_per_m2 = 200 # Fr/m^2
soil_calc(area,price_per_m2,members)

# Salgesch
area = 1100 # m^2
price_per_m2 = 160 # Fr/m^2
soil_calc(area,price_per_m2,members)


I am working on the file "01_Mine/MachineLearning/NeuroEvolution-Flappy-Bird-master/Jupyter Notebook/Flappy.ipynb".
The first task is:
DEEP LEARNING
This cell will train a MLP with the data generated playing with the network generated by the NEAT to play the game.
Can you write Python code for it?

import pandas as pd
import numpy as np
from sklearn import linear_model, datasets, metrics
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier

dat=pd.read_csv('flapyData.csv', sep=',',header=None)

inputs,outputs = np.column_stack((dat[0],dat[1],dat[2],dat[3],dat[4])),np.column_stack((dat[5],dat[6]))
X_train, X_test, Y_train, Y_test = train_test_split(inputs, outputs, test_size = 0.2, random_state=0)

mlp_classifier = MLPClassifier(hidden_layer_sizes=(10,4), max_iter=1000, tol=0.001, random_state=1, verbose=True)

mlp_classifier.fit(X_train,Y_train)