I am working on the file "4_Visibility_Space/4_5_2_uv_coverage_improving_your_coverage.ipynb".
The first task is:
4.5.2 $uv$ coverage : Improving your $uv$ coverage
In $\S$ 4.5.1 ➞, we have explored the ways in which the visibility function is sampled. Depending on the interferometer's location (in latitude), the projected baseline w.r.t. the observed source, and the time and frequency of our observation, the $uv$ plane is sampled along tracks which can be derived from the projected baseline in the ($u$,$v$,$w$) reference frame. These $uv$ tracks are portions of ellipses. Over the course of an observation, samples are accumulated along these tracks and form the $uv$ coverage.
To accurately measure the sky, we need a good knowledge of its attendant visibility function: we therefore need to sample the $uv$-plane as fully as possible.
TLG:GM: Check if the italic words are in the glossary.
We will see, in this section, how the $uv$ coverage can be improved using multiple element interferometers ($\S$ 4.5.2.1 ⤵)  and using time ($\S$ 4.5.2.2 ⤵) and frequency ($\S$ 4.5.2.3 ⤵) integration to our advantage.

4.5.2.1 Configuration of an $N$-element interferometer
Until now, we only considered 2-element interferometers. In practice, interferometers are built from $N>2$  elements. We then compute the corresponding $\frac{N(N-1)}{2}$ independent cross-correlations (doing this for each frequency channel, each time step and each polarization). For an $N$-element interferometer, the sampling in the $uv$ plane is thus proportional to $\sim N^2$.
In the following, we will show the characteristic $uv$ coverage of existing, well-known radio interferometers, observing at a single frequency and for a single time step (i.e. a snapshot observation). You may select the array of your choice from the list below:
Can you write Python code for it?

config = widgets.Dropdown(
    options={'VLAa':'configs/vlaa.enu.txt',
        'VLAb':'configs/vlab.enu.txt',
        'VLAc':'configs/vlac.enu.txt',
        'VLAd':'configs/vlad.enu.txt',
        'WSRT':'configs/wsrt.enu.txt',
        'kat7':'configs/kat-7.enu.txt',
        'meerkat':'configs/meerkat.enu.txt'},
    value="configs/vlaa.enu.txt",
    Description="Antennas:")

display(config)


I am working on the file "4_Visibility_Space/4_5_2_uv_coverage_improving_your_coverage.ipynb".
The first task is:
4.5.2.1.2 The snapshot $\boldsymbol{uv}$ coverage
Can you write Python code for it?

%matplotlib inline
Ntimes=3
plotBL.plotuv(antennaPosition,L,dec,h,Ntimes,lam)


I am working on the file "4_Visibility_Space/4_5_2_uv_coverage_improving_your_coverage.ipynb".
The first task is:

Figure 4.5.14: Distribution of the antennas in a $ENU$ reference frame. Each dot represents an antenna which will be part of $N$-1 baselines.
As for the previous simulations, we will simulate an observation using the VLA as the default case.
4.5.2.1.1 Observation configuration
Can you write Python code for it?

# Observation parameters
c=3e8                                         # Speed of light
f=1420e6                                      # Frequency
lam = c/f                                     # Wavelength

time_steps = 1200                              # time steps
h = np.linspace(-6,6,num=time_steps)*np.pi/12 # Hour angle window

# declination convert in radian

L = np.radians(34.0790)      # Latitude of the VLA
dec = np.radians(34.)


I am working on the file "4_Visibility_Space/4_5_2_uv_coverage_improving_your_coverage.ipynb".
The first task is:

Figure 4.5.16: Sampled $uv$ coverage of the interferometer for various values of time samples. Red and blue points corresponds to symmetric $uv$ points, since each baseline give us the measurement of $V_\nu$ at ($u$,$v$) and its complex conjugate $V^*_\nu$ at ($-u$,$-v$).
With an increasing number of time steps, you can start to see each individual baselines' elliptical $uv$ track in the $uv$ plane. It is easy to see that the longer the observation, the better our sampling of the visibility function.

4.5.2.3 Integration in frequency: Frequency Synthesis
We have seen in $\S$ 4.3 ➞ that the $u$, $v$, $w$ coordinates are usually normalized by $\lambda$. As a consequence, observing with the same interferometer at a different frequency gives a different set of spatial frequencies. Sampling at a different ($u,v$)-coordinates results in a rescaling of the uv coverage at the original frequency TLG:RC: Please rewrite last sentence. Indeed, for the same snapshot $uv$ coverage at wavelength $\lambda_1$, the $uv$ coverage at $\lambda_2 < \lambda_1$ will be a shrunk version of that a $\lambda_1$.
If the distribution of our sampling frequencies is (quasi) continuous, e.g. if the observing system operates in a bandwidth [$f_\text{min}$,$f_\text{max}$], with $N_\text{freqs}$ channels, we can contiguously sample portions of the $uv$ plane with $\frac{N(N-1)}{2}\times N_\text{freqs}$ independent measurements. For each baseline, a radial track will be generated due to the frequency scaling effect of $u$ and $v$.

Warning: It is only possible to do this up to a point: see *bandwidth smearing* in [$\S$ 9.3 ➞](../9_Practical/9_3_observing_smearing.ipynb)) for more details on the limits of this technique.

TLG:GM: Check if the italic words are in the glossary.
In the next block, we will plot a snapshot observation. You can...
Can you write Python code for it?

df=10e6 # frequency step
f0=c/lam # starting frequency
lamb0=lam # starting wavelength

def Interactplot(key,Nfreqs):
    print "Nfreqs="+str(Nfreqs)
    plotBL.plotuv_freq(antennaPosition,L,dec,h,Nfreqs,lamb0,df)

slider=IntSlider(description="Nfreqs",min=1,max=200,step=1,continuous_update=False)

slider.on_trait_change(Interactplot,'value')
display(slider)
Interactplot("",1)


I am working on the file "4_Visibility_Space/4_5_2_uv_coverage_improving_your_coverage.ipynb".
The first task is:

Figure 4.5.16: Sampled $uv$ coverage of the interferometer for various values of time samples. Red and blue points corresponds to symmetric $uv$ points, since each baseline give us the measurement of $V_\nu$ at ($u$,$v$) and its complex conjugate $V^*_\nu$ at ($-u$,$-v$).
With an increasing number of time steps, you can start to see each individual baselines' elliptical $uv$ track in the $uv$ plane. It is easy to see that the longer the observation, the better our sampling of the visibility function.

4.5.2.3 Integration in frequency: Frequency Synthesis
We have seen in $\S$ 4.3 ➞ that the $u$, $v$, $w$ coordinates are usually normalized by $\lambda$. As a consequence, observing with the same interferometer at a different frequency gives a different set of spatial frequencies. Sampling the ($u,v$)-plane with the same baseline and two diffe frequencies is equivalent to rescaling the projected baseline. Indeed, for the same snapshot $uv$ coverage at wavelength $\lambda_1$, the $uv$ coverage at $\lambda_2 < \lambda_1$ will be a shrunk version of that a $\lambda_1$.
If the distribution of our sampling frequencies is (quasi) continuous, e.g. if the observing system operates in a bandwidth [$f_\text{min}$,$f_\text{max}$], with $N_\text{freqs}$ channels, we can contiguously sample portions of the $uv$ plane with $\frac{N(N-1)}{2}\times N_\text{freqs}$ independent measurements. For each baseline, a radial track will be generated due to the frequency scaling effect of $u$ and $v$.

Warning: It is only possible to do this up to a point: see *bandwidth smearing* in [$\S$ 9.3 ➞](../9_Practical/9_3_observing_smearing.ipynb)) for more details on the limits of this technique.

TLG:GM: Check if the italic words are in the glossary.
In the next block, we will plot a snapshot observation. You can increase the number of continuous frequency channels  (at 10 MHz...
Can you write Python code for it?

df=10e6 # frequency step
f0=c/lam # starting frequency
lamb0=lam # starting wavelength

def Interactplot(key,Nfreqs):
    print("Nfreqs="+str(Nfreqs))
    plotBL.plotuv_freq(antennaPosition,L,dec,h,Nfreqs,lamb0,df)

slider=IntSlider(description="Nfreqs",min=1,max=200,step=1,continuous_update=False)

slider.on_trait_change(Interactplot,'value')
display(slider)
Interactplot("",1)


I am working on the file "4_Visibility_Space/4_5_2_uv_coverage_improving_your_coverage.ipynb".
The first task is:

Figure 4.5.15: Snapshot $uv$ coverage of the interferometer. Red and blue points correspond to symmetric $uv$ points: indeed, each baseline gives us the measurement of $V_\nu$ at ($u$,$v$) and its complex conjugate $V^*_\nu$ at ($-u$,$-v$).
Depending on the number of elements and their relative distribution on the ground, the shape of the snapshot $uv$ coverage can vary dramatically from one interferometer to another. One may prefer an antenna distribution which maximizes an instantaneous coverage which spans the entire $uv$ plane (e.g. VLA in Fig. 4.5.15 ⤵) rather than a compact array which samples a smaller portion of the $uv$ plane. With the raw increase of total sensitivity, improving the instantaneous $uv$ coverage is the main motivation for increasing the number of antennas and optimising their relative positions. A good $uv$ coverage is obtained if the $uv$ plane is sufficiently - and smoothly! - sampled. In the next section, we will see how the Earth contributes to improve the $uv$ coverage.

4.5.2.3 Time integration: Earth Rotation Synthesis
Observing a source for several hours requires tracking this source with fringe (or delay) tracking. As the source moves with the sky, the projected baselines - as seen from the source - will continuously vary along elliptical $uv$ tracks.
With good enough time sampling, it is possible to sample each $uv$ track along the observation and accumulate a number of $\frac{N(N-1)}{2}\times N_\text{times}$ independent measurements.
The following experiment will allow you to change the number of time steps collected during an observation, from a snapshot observation up to a 12 hour observation.
The next block will let you plot a snapshot observation. You can increase the duration of the observation with the slider which defines the total number of time steps for which the $uv$ coverage is computed. When using the slider, the new resulting plot will be...
Can you write Python code for it?

from ipywidgets import *
from IPython.display import display

def Interactplot(key,Ntimes):
    print "Ntimes="+str(Ntimes)
    plotBL.plotuv(antennaPosition,L,dec,h,Ntimes,lam)

slider=IntSlider(description="Ntimes",min=2,max=1200,step=100,continuous_update=False)

slider.on_trait_change(Interactplot,'value')
display(slider)
Interactplot("",2)


I am working on the file "4_Visibility_Space/4_5_2_uv_coverage_improving_your_coverage.ipynb".
The first task is:
From the list above, you can select different configurations corresponding to real instrumental layouts.
However, if you want to specify the relative positions of the antennas in an $ENU$ reference frame yourself, you can edit the following block and set the variable "custom" to "1".
Can you write Python code for it?

# you need to re-evaluate this box if you modify the array.
antennaPosition=np.genfromtxt(config.value)

# custom antenna distribution
custom=0
if (custom):
    antennaPosition = np.zeros((10, 2), dtype=float)
    antennaPosition[0,:] = [0,0]
    antennaPosition[1,:] = [-4, 5]
    antennaPosition[2,:] = [4, 5]
    antennaPosition[3,:] = [-10,0]
    antennaPosition[4,:] = [-8,-3]
    antennaPosition[5,:] = [-4,-5]
    antennaPosition[6,:] = [0,-6]
    antennaPosition[7,:] = [4,-5]
    antennaPosition[8,:] = [8,-3]
    antennaPosition[9,:] = [10,0]


I am working on the file "4_Visibility_Space/4_5_2_uv_coverage_improving_your_coverage.ipynb".
The first task is:
Import section specific modules:
Can you write Python code for it?

from IPython.display import display
from ipywidgets import *
from mpl_toolkits.mplot3d import Axes3D
import plotBL

HTML('../style/code_toggle.html')


I am working on the file "4_Visibility_Space/4_5_2_uv_coverage_improving_your_coverage.ipynb".
The first task is:
Let's plot the distribution of the antennas from the selected (or customized) interferometer:
Can you write Python code for it?

%matplotlib inline
mxabs = np.max(abs(antennaPosition[:]))*1.1;
# make use of pylab librery to plot
fig=plt.figure(figsize=(6,6))
plt.plot((antennaPosition[:,0]-np.mean(antennaPosition[:,0]))/1e3, \
         (antennaPosition[:,1]-np.mean(antennaPosition[:,1]))/1e3, 'o')
plt.axes().set_aspect('equal')
plt.xlim(-mxabs/1e3, mxabs/1e3)
plt.ylim(-mxabs/1e3, (mxabs+5)/1e3)
plt.xlabel("E (km)")
plt.ylabel("N (km)")
plt.title("Antenna positions")


I am working on the file "4_Visibility_Space/4_problem_set.ipynb".
The first task is:
(1) Calculate the ENU baseline difference vector $\boldsymbol{b}{12}^{xyz}$ of baseline 12? [1] Hint: $\boldsymbol{b}{12}^{xyz} = (x_2-x_1,y_2-y_1,z_2-z_1)$.
(2) Calculate the length $D_{12}$ of $\boldsymbol{b}_{12}^{xyz}$. [1]
(3) Calculate the azimuth angle $\mathcal{A}{12}$ and the elevation (altitude) angle $\mathcal{E}{12}$ of $\boldsymbol{b}_{12}^{xyz}$? [2] Hint: Remember the azimuth angle is measured from the north towards the east.
(4) Calculate $\boldsymbol{b}{12}^{XYZ}=\begin{bmatrix}X{12}\Y_{12}\Z_{12}\end{bmatrix}$? [3] Hint: Recall that
\begin{equation}
\begin{bmatrix}X_{12}\Y_{12}\Z_{12}\end{bmatrix}=D_{12}
 \begin{bmatrix}
 \cos L_a \sin \mathcal{E}{12} - \sin L_a \cos \mathcal{E}{12} \cos \mathcal{A}{12}\nonumber\
 \cos \mathcal{E}{12} \sin \mathcal{A}{12} \nonumber\
 \sin L_a \sin \mathcal{E}{12} + \cos L_a \cos \mathcal{E}{12} \cos \mathcal{A}{12}
 \end{bmatrix}
 \end{equation}
(5) Calculate the observational wavelength $\lambda$? [1] Hint: $\lambda f=c$.
(6) Calculate $\sqrt{X_{12}^2 + Y_{12}^2}\lambda^{-1}$, $|\sin \delta_0| \sqrt{X_{12}^2 + Y_{12}^2}\lambda^{-1}$ and $\cos\delta_0 Z_{12}\lambda^{-1}$ [3].
(7) Draw the uv-tracks of baseline 12 and 21 that are generated; during a 24h observation? [2] Hint: The baseline vector of an interferometer trace out an elliptical locus (after 24 hours it will complete one entire revolution). The values calculated in the previous question determine the shape of the elliptical locus. Moreover, $\boldsymbol{b}{12}^{xyz}=-\boldsymbol{b}{21}^{xyz}$.
(8) Generate the coordinate pair (u,v) associated with baseline 12 and $H_0=$ -4h. [2] Hint: Recall that
 \begin{equation}
 \begin{bmatrix}
 u_{12}\
 v_{12}
 \end{bmatrix} =\lambda^{-1}
 \begin{bmatrix}
 \sin H & \cos H & 0\
 -\sin \delta_0 \cos H & \sin \delta_0 \sin H & \cos \delta_0
 \end{bmatrix}
 \begin{bmatrix}
 X_{12}\
 Y_{12}\
 Z_{12}
\end{bmatrix}
\end{equation}
(9)  Why is u and v unit-less? Why then do we measure u and v in rad$^{-1}$? [1] Hint: In what unit do we measure length and wavelength?...
Can you write Python code for it?


I am working on the file "4_Visibility_Space/4_problem_set.ipynb".
The first task is:
Import section specific modules:
Can you write Python code for it?

from IPython.display import Image

HTML('../style/code_toggle.html')


I am working on the file "4_Visibility_Space/4_problem_set.ipynb".
The first task is:
Visibility Space Problem Set
1. Position of Orion
The positions of the stars in the Orion constellation can be found in the figure below.
<img src='figures/orion_fig.png' width=300>
Figure: Schematic sky map of the Orion consellation.
Betelgeuse and Rigel are the brightest stars in this constellation. The equatorial coordinates of the constellation's centre, Betelgeuse, and Rigel are:
| Name | Right Ascension ($\alpha$) | Declination ($\delta$) |
|:---:|:---:|:---:|
| Center | 5h 30m $(\alpha_0)$ | $0^{\circ}$ $(\delta_0)$ |
| Betelgeuse | 5h 55m 10.3053s | $7^{\circ}~24'~25.42''$ |
| Rigel | 5h 14m 32.272s | $-8^{\circ}~12'~5.898''$ |
(1) Calculate the $l$ and $m$ coordinates of Orion's center, Betelgeuse and Rigel? Assume that your field center was chosen to coincide with Orion's center. [specify your answers in degrees ($^{\circ}$)]. [5] Hint:
\begin{eqnarray}
    \Delta \alpha &=& \alpha-\alpha_0\nonumber\
    l &=& \cos \delta \sin \Delta \alpha \nonumber\
    m &=& \sin \delta \cos \delta_0 - \cos \delta \sin \delta_0\cos \Delta \alpha\nonumber
\end{eqnarray}
(2) What is the distance from the field center to Betelgeuse in the projected $lm$-plane? [1] Hint: $l^2+m^2 = d^2$.
(3) What is the angular distance from the field center to Betelgeuse on the celestial sphere? [1] Hint: $l^2+m^2 = \sin^2 \theta$.
(4) Verify the previous question by using the equatorial coordinates directly (stay on the celestial sphere)? [1] Hint: Use the spherical Pythagorean theorem.
(5) Why do we measure $l$ and $m$ in $^{\circ}$ if they are direction cosines and therefore by definition unit-less? [1] Hint: Use the previous three results to answer this question.
(6) What will the hour angle of Orion's center be when it appear above the horizon? In which direction will Orion's center appear? [1] Hint: Remember Orion's center is at $\delta = 0^{\circ}$. It is the same declination the sun has when it lies on one of the equinoxes.
Subtotal 10
2. Papino and Paperino
We will be using a fictitious piece of sky (containing only two...
Can you write Python code for it?


I am working on the file "4_Visualising_Time_Series_Data.ipynb".
The first task is:
Now you can plot the average sales across years and months.
Can you write Python code for it?

# grouping by year and month
df_time = df.groupby(["year", "month"]).Sales.mean()
df_time.head()

plt.figure(figsize=(8, 6))
# time series plot
sns.tsplot(df_time)
plt.xlabel("Time")
plt.ylabel("Sales")
plt.show()


I am working on the file "4_Visualising_Time_Series_Data.ipynb".
The first task is:
Now, since on each day, multiple orders were placed, we need to aggregate Sales using a metric such as mean, median etc., and then create a time series plot.
We will group by Order_Date and compute the sum of Sales on each day.
Can you write Python code for it?

# aggregating total sales on each day
time_df = df.groupby('Order_Date')['Sales'].sum()
print(time_df.head())

print(type(time_df))


I am working on the file "4_Visualising_Time_Series_Data.ipynb".
The first task is:
Since Order_Date is a string, we need to convert it into a datetime object. You can do that using pd.to_datetime().
Can you write Python code for it?

# Convert Order_Date to datetime type
df['Order_Date'] = pd.to_datetime(df['Order_Date'])

# Order_Date is now datetime type
df.info()


I am working on the file "4_Visualising_Time_Series_Data.ipynb".
The first task is:
There is another way to visualise numeric variables, such as Sales, across the year and month. We can pivot the month column to create a wide-format dataframe, and then plot a heatmap.
Can you write Python code for it?

# Pivoting the data using 'month'
year_month = pd.pivot_table(df, values='Sales', index='year', columns='month', aggfunc='mean')
year_month.head()


I am working on the file "4_Visualising_Time_Series_Data.ipynb".
The first task is:
Using Derived Date Metrics for Visualisation
It is often helpful to use derived variables from date such as month and year and using them to identify hidden patterns.
Can you write Python code for it?

# extracting month and year from date

# extract month
df['month'] = df['Order_Date'].dt.month

# extract year
df['year'] = df['Order_Date'].dt.year

df.head()


I am working on the file "4_Visualising_Time_Series_Data.ipynb".
The first task is:
We can now create a time-series plot using sns.tsplot().
Can you write Python code for it?

# time series plot

# figure size
plt.figure(figsize=(16, 8))

# tsplot
sns.tsplot(data=time_df)
plt.show()


I am working on the file "4_Visualising_Time_Series_Data.ipynb".
The first task is:
You can now create a heatmap using sns.heatmap().
Can you write Python code for it?

# figure size
plt.figure(figsize=(12, 8))

# heatmap with a color map of choice
sns.heatmap(year_month, cmap="YlGnBu")
plt.show()


I am working on the file "4_convolutions.ipynb".
The first task is:
Let's build a small network with two convolutional layers, followed by one fully connected layer. Convolutional networks are more expensive computationally, so we'll limit its depth and number of fully connected nodes.
Can you write Python code for it?

batch_size = 16
patch_size = 5
depth = 16
num_hidden = 64

graph = tf.Graph()

with graph.as_default():

  # Input data.
  tf_train_dataset = tf.placeholder(
    tf.float32, shape=(batch_size, image_size, image_size, num_channels))
  tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
  tf_valid_dataset = tf.constant(valid_dataset)
  tf_test_dataset = tf.constant(test_dataset)

  # Variables.
  layer1_weights = tf.Variable(tf.truncated_normal(
      [patch_size, patch_size, num_channels, depth], stddev=0.1))
  layer1_biases = tf.Variable(tf.zeros([depth]))
  layer2_weights = tf.Variable(tf.truncated_normal(
      [patch_size, patch_size, depth, depth], stddev=0.1))
  layer2_biases = tf.Variable(tf.constant(1.0, shape=[depth]))
  layer3_weights = tf.Variable(tf.truncated_normal(
      [image_size / 4 * image_size / 4 * depth, num_hidden], stddev=0.1))
  layer3_biases = tf.Variable(tf.constant(1.0, shape=[num_hidden]))
  layer4_weights = tf.Variable(tf.truncated_normal(
      [num_hidden, num_labels], stddev=0.1))
  layer4_biases = tf.Variable(tf.constant(1.0, shape=[num_labels]))

  # Model.
  def model(data):
    conv = tf.nn.conv2d(data, layer1_weights, [1, 2, 2, 1], padding='SAME')
    hidden = tf.nn.relu(conv + layer1_biases)
    conv = tf.nn.conv2d(hidden, layer2_weights, [1, 2, 2, 1], padding='SAME')
    hidden = tf.nn.relu(conv + layer2_biases)
    shape = hidden.get_shape().as_list()
    reshape = tf.reshape(hidden, [shape[0], shape[1] * shape[2] * shape[3]])
    hidden = tf.nn.relu(tf.matmul(reshape, layer3_weights) + layer3_biases)
    return tf.matmul(hidden, layer4_weights) + layer4_biases

  # Training computation.
  logits = model(tf_train_dataset)
  loss = tf.reduce_mean(
    tf.nn.softmax_cross_entropy_with_logits(logits, tf_train_labels))

  # Optimizer.
  optimizer = tf.train.GradientDescentOptimizer(0.05).minimize(loss)

  # Predictions for the training, validation, and test data.
  train_prediction = tf.nn.softmax(logits)
  valid_prediction =...


I am working on the file "4_convolutions.ipynb".
The first task is:
Reformat into a TensorFlow-friendly shape:
- convolutions need the image data formatted as a cube (width by height by #channels)
- labels as float 1-hot encodings.
Can you write Python code for it?

image_size = 28
num_labels = 10
num_channels = 1 # grayscale

import numpy as np

def reformat(dataset, labels):
  dataset = dataset.reshape(
    (-1, image_size, image_size, num_channels)).astype(np.float32)
  labels = (np.arange(num_labels) == labels[:,None]).astype(np.float32)
  return dataset, labels
train_dataset, train_labels = reformat(train_dataset, train_labels)
valid_dataset, valid_labels = reformat(valid_dataset, valid_labels)
test_dataset, test_labels = reformat(test_dataset, test_labels)
print 'Training set', train_dataset.shape, train_labels.shape
print 'Validation set', valid_dataset.shape, valid_labels.shape
print 'Test set', test_dataset.shape, test_labels.shape

def accuracy(predictions, labels):
  return (100.0 * np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
          / predictions.shape[0])


I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Build the Decoding Layer
Implement decoding_layer() to create a Decoder RNN layer.

Create RNN cell for decoding using rnn_size and num_layers.
Create the output fuction using lambda to transform it's input, logits, to class logits.
Use the your decoding_layer_train(encoder_state, dec_cell, dec_embed_input, sequence_length, decoding_scope, output_fn, keep_prob) function to get the training logits.
Use your decoding_layer_infer(encoder_state, dec_cell, dec_embeddings, start_of_sequence_id, end_of_sequence_id, maximum_length, vocab_size, decoding_scope, output_fn, keep_prob) function to get the inference logits.

Note: You'll need to use tf.variable_scope to share variables between training and inference.
Can you write Python code for it?

from tensorflow.contrib.rnn import BasicLSTMCell, MultiRNNCell
from tensorflow.contrib.layers import linear

def decoding_layer(dec_embed_input, dec_embeddings, encoder_state,
                   vocab_size, sequence_length, rnn_size, num_layers,
                   target_vocab_to_int, keep_prob):
    """
    Create decoding layer
    :param dec_embed_input: Decoder embedded input
    :param dec_embeddings: Decoder embeddings
    :param encoder_state: The encoded state
    :param vocab_size: Size of vocabulary
    :param sequence_length: Sequence Length
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param keep_prob: Dropout keep probability
    :return: Tuple of (Training Logits, Inference Logits)
    """
    lstm = BasicLSTMCell( rnn_size )
    dec_cell = MultiRNNCell( [lstm] * num_layers )

    output_fn = lambda x: linear(x, vocab_size)

    with tf.variable_scope("decoder") as decoding_scope:
        train_logits = decoding_layer_train(encoder_state, dec_cell, dec_embed_input,
                                        sequence_length, decoding_scope, output_fn, keep_prob)

        start_of_sequence_id = target_vocab_to_int['<GO>']
        end_of_sequence_id = target_vocab_to_int['<EOS>']
        maximum_length = sequence_length+1

    with tf.variable_scope(decoding_scope):
        decoding_scope.reuse_variables()
        infer_logits = decoding_layer_infer(encoder_state, dec_cell, dec_embeddings,
                                                start_of_sequence_id, end_of_sequence_id, maximum_length,
                                                vocab_size, decoding_scope, output_fn, keep_prob)

    return train_logits, infer_logits

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer(decoding_layer)


I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Build the Graph
Build the graph using the neural network you implemented.
Can you write Python code for it?

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
save_path = 'checkpoints/dev'
(source_int_text, target_int_text), (source_vocab_to_int, target_vocab_to_int), _ = helper.load_preprocess()
max_target_sentence_length = max([len(sentence) for sentence in source_int_text])

train_graph = tf.Graph()
with train_graph.as_default():
    input_data, targets, lr, keep_prob = model_inputs()
    sequence_length = tf.placeholder_with_default(max_target_sentence_length, shape=None, name='sequence_length')
    input_shape = tf.shape(input_data)

    train_logits, inference_logits = seq2seq_model(
        tf.reverse(input_data, [-1]), targets, keep_prob, batch_size, sequence_length, len(source_vocab_to_int), len(target_vocab_to_int),
        encoding_embedding_size, decoding_embedding_size, rnn_size, num_layers, target_vocab_to_int)

    tf.identity(inference_logits, 'logits')
    with tf.name_scope("optimization"):
        # Loss function
        cost = tf.contrib.seq2seq.sequence_loss(
            train_logits,
            targets,
            tf.ones([input_shape[0], sequence_length]))

        # Optimizer
        optimizer = tf.train.AdamOptimizer(lr)

        # Gradient Clipping
        gradients = optimizer.compute_gradients(cost)
        capped_gradients = [(tf.clip_by_value(grad, -1., 1.), var) for grad, var in gradients if grad is not None]
        train_op = optimizer.apply_gradients(capped_gradients)


I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Build the Neural Network
Apply the functions you implemented above to:

Apply embedding to the input data for the encoder.
Encode the input using your encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob).
Process target data using your process_decoding_input(target_data, target_vocab_to_int, batch_size) function.
Apply embedding to the target data for the decoder.
Decode the encoded input using your decoding_layer(dec_embed_input, dec_embeddings, encoder_state, vocab_size, sequence_length, rnn_size, num_layers, target_vocab_to_int, keep_prob).
Can you write Python code for it?

def seq2seq_model(input_data, target_data, keep_prob, batch_size, sequence_length,
                  source_vocab_size, target_vocab_size, enc_embedding_size,
                  dec_embedding_size, rnn_size, num_layers, target_vocab_to_int):
    """
    Build the Sequence-to-Sequence part of the neural network
    :param input_data: Input placeholder
    :param target_data: Target placeholder
    :param keep_prob: Dropout keep probability placeholder
    :param batch_size: Batch Size
    :param sequence_length: Sequence Length
    :param source_vocab_size: Source vocabulary size
    :param target_vocab_size: Target vocabulary size
    :param enc_embedding_size: Decoder embedding size
    :param dec_embedding_size: Encoder embedding size
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :return: Tuple of (Training Logits, Inference Logits)
    """

    # 1: Embedd input data
    embedded_input = tf.contrib.layers.embed_sequence(
        input_data, source_vocab_size, enc_embedding_size)

    # 2: Encode input
    encoder_state = encoding_layer(embedded_input, rnn_size, num_layers, keep_prob)

    # 3: Process target data
    preprocessed_target = process_decoding_input(target_data, target_vocab_to_int,
                                                 batch_size)

    # 4: Embedd target data
    dec_embeddings = tf.Variable(tf.truncated_normal(shape=(target_vocab_size, dec_embedding_size), dtype=tf.float32))
    dec_embed_input = tf.nn.embedding_lookup(dec_embeddings, preprocessed_target)

    # 5: Decode encoded input
    train_logits, infer_logits = decoding_layer(dec_embed_input, dec_embeddings,
                                                encoder_state, target_vocab_size, sequence_length,
                                                rnn_size, num_layers, target_vocab_to_int, keep_prob)

    return train_logits, infer_logits

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS...


I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Build the Neural Network
You'll build the components necessary to build a Sequence-to-Sequence model by implementing the following functions below:
- model_inputs
- process_decoding_input
- encoding_layer
- decoding_layer_train
- decoding_layer_infer
- decoding_layer
- seq2seq_model
Input
Implement the model_inputs() function to create TF Placeholders for the Neural Network. It should create the following placeholders:

Input text placeholder named "input" using the TF Placeholder name parameter with rank 2.
Targets placeholder with rank 2.
Learning rate placeholder with rank 0.
Keep probability placeholder named "keep_prob" using the TF Placeholder name parameter with rank 0.

Return the placeholders in the following the tuple (Input, Targets, Learing Rate, Keep Probability)
Can you write Python code for it?

def model_inputs():
    """
    Create TF Placeholders for input, targets, and learning rate.
    :return: Tuple (input, targets, learning rate, keep probability)
    """
    input_ = tf.placeholder(shape=(None, None), dtype=tf.int32, name='input')
    targets = tf.placeholder(shape=(None, None), dtype=tf.int32, name='targets')
    learning_rate = tf.placeholder(dtype=tf.float32, name='lr')
    keep_prob = tf.placeholder(dtype=tf.float32, name='keep_prob')
    return input_, targets, learning_rate, keep_prob

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_model_inputs(model_inputs)


I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Decoding - Inference
Create inference logits using tf.contrib.seq2seq.simple_decoder_fn_inference() and tf.contrib.seq2seq.dynamic_rnn_decoder().
Can you write Python code for it?

from tensorflow.contrib.seq2seq import simple_decoder_fn_inference, \
                                                                dynamic_rnn_decoder

def decoding_layer_infer(encoder_state, dec_cell, dec_embeddings,
                         start_of_sequence_id, end_of_sequence_id,
                         maximum_length, vocab_size, decoding_scope,
                         output_fn, keep_prob):
    """
    Create a decoding layer for inference
    :param encoder_state: Encoder state
    :param dec_cell: Decoder RNN Cell
    :param dec_embeddings: Decoder embeddings
    :param start_of_sequence_id: GO ID
    :param end_of_sequence_id: EOS Id
    :param maximum_length: Maximum length of
    :param vocab_size: Size of vocabulary
    :param decoding_scope: TensorFlow Variable Scope for decoding
    :param output_fn: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: Inference Logits
    """

    decoder_fn = simple_decoder_fn_inference(
        output_fn=output_fn,
        encoder_state=encoder_state,
        embeddings=dec_embeddings,
        start_of_sequence_id=start_of_sequence_id,
        end_of_sequence_id=end_of_sequence_id,
        maximum_length=maximum_length,
        num_decoder_symbols=vocab_size)

    outputs, final_state, final_context_state = dynamic_rnn_decoder(
            cell=dec_cell,
            decoder_fn=decoder_fn,
            scope=decoding_scope)

    return outputs

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_infer(decoding_layer_infer)


I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Decoding - Training
Create training logits using tf.contrib.seq2seq.simple_decoder_fn_train() and tf.contrib.seq2seq.dynamic_rnn_decoder().  Apply the output_fn to the tf.contrib.seq2seq.dynamic_rnn_decoder() outputs.
Can you write Python code for it?

def decoding_layer_train(encoder_state, dec_cell, dec_embed_input,
                         sequence_length, decoding_scope, output_fn, keep_prob):
    """
    Create a decoding layer for training
    :param encoder_state: Encoder State
    :param dec_cell: Decoder RNN Cell
    :param dec_embed_input: Decoder embedded input
    :param sequence_length: Sequence Length
    :param decoding_scope: TenorFlow Variable Scope for decoding
    :param output_fn: Function to apply the output layer
    :param keep_prob: Dropout keep probability
    :return: Train Logits
    """

    dec_cell = tf.contrib.rnn.DropoutWrapper(dec_cell, keep_prob)

    decoder_fn = tf.contrib.seq2seq.simple_decoder_fn_train(encoder_state)
    outputs, final_state, final_context_state = \
        tf.contrib.seq2seq.dynamic_rnn_decoder(
                                          cell=dec_cell,
                                          decoder_fn=decoder_fn,
                                          inputs=dec_embed_input,
                                          sequence_length=sequence_length,
                                          scope=decoding_scope)

    return output_fn(outputs)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_decoding_layer_train(decoding_layer_train)


I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Encoding
Implement encoding_layer() to create a Encoder RNN layer using tf.nn.dynamic_rnn().
Can you write Python code for it?

def encoding_layer(rnn_inputs, rnn_size, num_layers, keep_prob):
    """
    Create encoding layer
    :param rnn_inputs: Inputs for the RNN
    :param rnn_size: RNN Size
    :param num_layers: Number of layers
    :param keep_prob: Dropout keep probability
    :return: RNN state
    """
    lstm = tf.contrib.rnn.BasicLSTMCell(rnn_size)
    lstm_layers = tf.contrib.rnn.MultiRNNCell([lstm] * num_layers)
    lstm_layers = tf.contrib.rnn.DropoutWrapper(lstm_layers, keep_prob)
    _, state = tf.nn.dynamic_rnn(lstm_layers, rnn_inputs, dtype=tf.float32)
    return state

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_encoding_layer(encoding_layer)


I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Neural Network Training
Hyperparameters
Tune the following parameters:

Set epochs to the number of epochs.
Set batch_size to the batch size.
Set rnn_size to the size of the RNNs.
Set num_layers to the number of layers.
Set encoding_embedding_size to the size of the embedding for the encoder.
Set decoding_embedding_size to the size of the embedding for the decoder.
Set learning_rate to the learning rate.
Set keep_probability to the Dropout keep probability
Can you write Python code for it?

# Number of Epochs
epochs = 10
# Batch Size
batch_size = 256
# RNN Size
rnn_size = 256
# Number of Layers
num_layers = 2
# Embedding Size
encoding_embedding_size = 200
decoding_embedding_size = 200
# Learning Rate
learning_rate = 0.001
# Dropout Keep Probability
keep_probability = 0.5


I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Process Decoding Input
Implement process_decoding_input using TensorFlow to remove the last word id from each batch in target_data and concat the GO ID to the begining of each batch.
Can you write Python code for it?

def process_decoding_input(target_data, target_vocab_to_int, batch_size):
    """
    Preprocess target data for dencoding
    :param target_data: Target Placehoder
    :param target_vocab_to_int: Dictionary to go from the target words to an id
    :param batch_size: Batch Size
    :return: Preprocessed target data
    """

    go_id = target_vocab_to_int.get('<GO>')
    gos = tf.fill(dims=(batch_size,1), value=go_id)
    return tf.concat((gos, target_data[:,:-1]), axis=1)

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_process_decoding_input(process_decoding_input)


I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Sentence to Sequence
To feed a sentence into the model for translation, you first need to preprocess it.  Implement the function sentence_to_seq() to preprocess new sentences.

Convert the sentence to lowercase
Convert words into ids using vocab_to_int
Convert words not in the vocabulary, to the <UNK> word id.
Can you write Python code for it?

def sentence_to_seq(sentence, vocab_to_int):
    """
    Convert a sentence to a sequence of ids
    :param sentence: String
    :param vocab_to_int: Dictionary to go from the words to an id
    :return: List of word ids
    """
    sentences = sentence.lower()
    words = sentences.split()

    unk_id = vocab_to_int['<UNK>']
    return [vocab_to_int.get(word, unk_id) for word in words]

"""
DON'T MODIFY ANYTHING IN THIS CELL THAT IS BELOW THIS LINE
"""
tests.test_sentence_to_seq(sentence_to_seq)


I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Train
Train the neural network on the preprocessed data. If you have a hard time getting a good loss, check the forms to see if anyone is having the same problem.
Can you write Python code for it?

%pdb off
"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
import time

def get_accuracy(target, logits):
    """
    Calculate accuracy
    """
    max_seq = max(target.shape[1], logits.shape[1])
    if max_seq - target.shape[1]:
        target = np.pad(
            target_batch,
            [(0,0),(0,max_seq - target_batch.shape[1]),(0,0)],
            'constant')
    if max_seq - batch_train_logits.shape[1]:
        logits = np.pad(
            logits,
            [(0,0),(0,max_seq - logits.shape[1]), (0,0)],
            'constant')

    return np.mean(np.equal(target, np.argmax(logits, 2)))

train_source = source_int_text[batch_size:]
train_target = target_int_text[batch_size:]

valid_source = helper.pad_sentence_batch(source_int_text[:batch_size])
valid_target = helper.pad_sentence_batch(target_int_text[:batch_size])

with tf.Session(graph=train_graph) as sess:
    sess.run(tf.global_variables_initializer())

    for epoch_i in range(epochs):
        for batch_i, (source_batch, target_batch) in enumerate(
                helper.batch_data(train_source, train_target, batch_size)):
            start_time = time.time()

            _, loss = sess.run(
                [train_op, cost], {input_data: source_batch,
                 targets: target_batch,
                 lr: learning_rate,
                 sequence_length: target_batch.shape[1],
                 keep_prob: keep_probability}
                )

            batch_train_logits = sess.run(
                inference_logits,
                {input_data: source_batch, keep_prob: 1.0})
            batch_valid_logits = sess.run(
                inference_logits,
                {input_data: valid_source, keep_prob: 1.0})

            train_acc = get_accuracy(target_batch, batch_train_logits)
            valid_acc = get_accuracy(np.array(valid_target), batch_valid_logits)
            end_time = time.time()
            print('Epoch {:>3} Batch {:>4}/{} - Train Accuracy: {:>6.3f}, Validation Accuracy: {:>6.3f}, Loss:...


I am working on the file "4_dlnd_language_translation/dlnd_language_translation.ipynb".
The first task is:
Translate
This will translate translate_sentence from English to French.
Can you write Python code for it?

translate_sentence = 'he saw a old yellow truck .'

"""
DON'T MODIFY ANYTHING IN THIS CELL
"""
translate_sentence = sentence_to_seq(translate_sentence, source_vocab_to_int)

loaded_graph = tf.Graph()
with tf.Session(graph=loaded_graph) as sess:
    # Load saved model
    loader = tf.train.import_meta_graph(load_path + '.meta')
    loader.restore(sess, load_path)

    input_data = loaded_graph.get_tensor_by_name('input:0')
    logits = loaded_graph.get_tensor_by_name('logits:0')
    keep_prob = loaded_graph.get_tensor_by_name('keep_prob:0')

    translate_logits = sess.run(logits, {input_data: [translate_sentence], keep_prob: 1.0})[0]

print('Input')
print('  Word Ids:      {}'.format([i for i in translate_sentence]))
print('  English Words: {}'.format([source_int_to_vocab[i] for i in translate_sentence]))

print('\nPrediction')
print('  Word Ids:      {}'.format([i for i in np.argmax(translate_logits, 1)]))
print('  French Words: {}'.format([target_int_to_vocab[i] for i in np.argmax(translate_logits, 1)]))


I am working on the file "4_em-with-text-data_blank.ipynb".
The first task is:
Initializing cluster weights
We will initialize each cluster weight to be the proportion of documents assigned to that cluster by k-means above.
Can you write Python code for it?

num_docs = tf_idf.shape[0]
weights = []
for i in xrange(num_clusters):
    # Compute the number of data points assigned to cluster i:
    num_assigned = sum(cluster_assignment == i) # YOUR CODE HERE
    w = float(num_assigned) / num_docs
    weights.append(w)

cluster_assignment


I am working on the file "4_em-with-text-data_blank.ipynb".
The first task is:
Initializing covariances
To initialize our covariance parameters, we compute $\hat{\sigma}{k, j}^2 = \sum{i=1}^{N}(x_{i,j} - \hat{\mu}_{k, j})^2$ for each feature $j$.  For features with really tiny variances, we assign 1e-8 instead to prevent numerical instability. We do this computation in a vectorized fashion in the following code block.
Can you write Python code for it?

covs = []
for i in xrange(num_clusters):
    member_rows = tf_idf[cluster_assignment==i]
    cov = (member_rows.multiply(member_rows) - 2*member_rows.dot(diag(means[i]))).sum(axis=0).A1 / member_rows.shape[0] \
          + means[i]**2
    cov[cov < 1e-8] = 1e-8
    covs.append(cov)


I am working on the file "4_em-with-text-data_blank.ipynb".
The first task is:
Quiz Question. Select all the topics that have a cluster in the model created above. [multiple choice]
Comparing to random initialization
Create variables for randomly initializing the EM algorithm. Complete the following code block.
Can you write Python code for it?

np.random.seed(5) # See the note below to see why we set seed=5.
num_clusters = len(means)
num_docs, num_words = tf_idf.shape

random_means = []
random_covs = []
random_weights = []

for k in range(num_clusters):

    # Create a numpy array of length num_words with random normally distributed values.
    # Use the standard univariate normal distribution (mean 0, variance 1).
    # YOUR CODE HERE
    mean = np.random.normal(loc=0, scale=1, size=num_words)

    # Create a numpy array of length num_words with random values uniformly distributed between 1 and 5.
    # YOUR CODE HERE
    cov = np.random.uniform(low = 1.0, high = 5.0, size = num_words)

    # Initially give each cluster equal weight.
    # YOUR CODE HERE
    weight = 1.0 / num_clusters

    random_means.append(mean)
    random_covs.append(cov)
    random_weights.append(weight)


I am working on the file "4_em-with-text-data_blank.ipynb".
The first task is:
Quiz Question: For the above model, out_random_init, use the visualize_EM_clusters method you created above. Are the clusters more or less interpretable than the ones found after initializing using k-means?
Can you write Python code for it?

# YOUR CODE HERE. Use visualize_EM_clusters, which will require you to pass in tf_idf and map_index_to_word.
visualize_EM_clusters(tf_idf, out_random_init['means'], out_random_init['covs'],
                      map_index_to_word)


I am working on the file "4_em-with-text-data_blank.ipynb".
The first task is:
Quiz Question: Is the final loglikelihood larger or smaller than the final loglikelihood we obtained above when initializing EM with the results from running k-means?
Can you write Python code for it?

print out_random_init['loglik'][-1] - out['loglik'][-1]


I am working on the file "4_em-with-text-data_blank.ipynb".
The first task is:
Quiz Question: Try fitting EM with the random initial parameters you created above. (Use cov_smoothing=1e-5.) Store the result to out_random_init. What is the final loglikelihood that the algorithm converges to?
Can you write Python code for it?

out_random_init = EM_for_high_dimension(tf_idf, random_means, random_covs, random_weights, cov_smoothing=1e-5)

print out_random_init['loglik']


I am working on the file "4_em-with-text-data_blank.ipynb".
The first task is:
Running EM
Now that we have initialized all of our parameters, run EM.
Can you write Python code for it?

out = EM_for_high_dimension(tf_idf, means, covs, weights, cov_smoothing=1e-10)

out['loglik']


I am working on the file "4_em-with-text-data_blank.ipynb".
The first task is:
Using a utility we provide, we will create a sparse matrix representation of the documents. This is the same utility function you used during the previous assignment on k-means with text data.
Can you write Python code for it?

tf_idf, map_index_to_word = sframe_to_scipy(wiki, 'tf_idf')


I am working on the file "4_error.ipynb".
The first task is:
Example 2
Evaluate $e^x$ with its taylor series.
$$e^x = \sum^\infty_{n=0} \frac{x^n}{n!}$$
Can we pick $N < \infty$ that can approximate $e^x$ over a give range $x \in [a,b]$ such that the relative error $E$ satisfies $E < 8 \cdot \varepsilon_{\text{machine}}$?
What might be a better way than simply evaluating the Taylor polynomial directly for various $N$?
Can you write Python code for it?

import scipy.misc

def my_exp(x, N=10):
    value = 0.0
    for n in xrange(N + 1):
        value += x**n / scipy.misc.factorial(n)

    return value

x = numpy.linspace(-2, 2, 100)
for N in range(1, 50):
    error = numpy.abs((numpy.exp(x) - my_exp(x, N=N)) / numpy.exp(x))
    if numpy.all(error < 100.0 * numpy.finfo(float).eps):
        break

print N

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.plot(x, error)
axes.set_xlabel("x")
axes.set_ylabel("Relative Error")
plt.show()


I am working on the file "4_error.ipynb".
The first task is:
For more examples see the "100-digit Challenge" book.
Combination of Error
In general we need to concern ourselves with the combination of both truncation error and floating point error.
Example
Consider the finite difference approximation where $f(x) = e^x$ and we are evaluating at $x=1$?
$$f'(x) \approx \frac{f(x + \Delta x) - f(x)}{\Delta x}$$
Compare the error between decreasing $\Delta x$ and the true solution $f'(1) = e$.
Can you write Python code for it?

delta_x = numpy.linspace(1e-20, 5.0, 1000)
x = 1.0
f_hat = (numpy.exp(x + delta_x) - numpy.exp(x)) / (delta_x)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.loglog(delta_x, numpy.abs(f_hat - numpy.exp(1)) / numpy.exp(1))
axes.set_xlabel("$\Delta x$")
axes.set_ylabel("Relative Error")
plt.show()


I am working on the file "4_error.ipynb".
The first task is:
Horner's Method for Evaluating Polynomials
Given
$$P_N(x) = a_0 + a_1 x + a_2 x^2 + \ldots + a_N x^N$$
$$P_N(x) = p_1 x^N + p_2 x^{N-1} + p_3 x^{N-2} + \ldots + p_{N+1}$$
want to find best way to evaluate $P_N(x)$.  First consider two ways to write $P_3$:
$$ P_3(x) = p_1 x^3 + p_2 x^2 + p_3 x + p_4$$
and using nested multiplication:
$$ P_3(x) = ((p_1 x + p_2) x + p_3) x + p_4$$
Consider how many operations it takes for each...
$$ P_3(x) = p_1 x^3 + p_2 x^2 + p_3 x + p_4$$
$$P_3(x) = \overbrace{p_1 \cdot x \cdot x \cdot x}^3 + \overbrace{p_2 \cdot x \cdot x}^2 + \overbrace{p_3 \cdot x}^1 + p_4$$
Adding up all the operations we can in general think of this as a pyramid

We can estimate this way that the algorithm written this way will take approximately $O(N^2 / 2)$ operations to complete.
Looking at our other means of evaluation:
$$ P_3(x) = ((p_1 x + p_2) x + p_3) x + p_4$$
Here we find that the method is $O(N)$ (the 2 is usually ignored in these cases).  The important thing is that the first evaluation is $O(N^2)$ and the second $O(N)$!
Algorithm
Fill in the function and implement Horner's method:
```python
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x
Function to evaluate a polynomial in order N operations.  The polynomial is defined as

P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

The value x should be a float.
"""
pass

```
```python
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x
Function to evaluate a polynomial in order N operations.  The polynomial is defined as

P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

The value x should be a float.
"""

y = p[0]
for coefficient in p[1:]:
    y = y * x + coefficient

return y

```
or an alternative version that allows x to be a vector of values:
```python
def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x
Function to evaluate a polynomial in order N operations.  The polynomial is defined as

P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

The...
Can you write Python code for it?

def eval_poly(p, x):
    """Evaluates polynomial given coefficients p at x

    Function to evaluate a polynomial in order N operations.  The polynomial is defined as

    P(x) = p[0] x**n + p[1] x**(n-1) + ... + p[n-1] x + p[n]

    The value x can by a NumPy ndarray.
    """

    y = numpy.ones(x.shape) * p[0]
    for coefficient in p[1:]:
        y = y * x + coefficient

    return y

p = [1, -3, 10, 4, 5, 5]
x = numpy.linspace(-10, 10, 100)
plt.plot(x, eval_poly(p, x))
plt.show()


I am working on the file "4_error.ipynb".
The first task is:
Lets plot this numerically for a section of $x$.
Can you write Python code for it?


I am working on the file "4_error.ipynb".
The first task is:
Matrix vector multiplication
$$ \begin{bmatrix} \epsilon & 1 \ 1 & 1 \end{bmatrix} \cdot \begin{bmatrix} 1 \ 1 \end{bmatrix}
= \begin{bmatrix} \epsilon + 1 \ 1 + 1 \end{bmatrix} = \begin{bmatrix} 1 \ 2 \end{bmatrix}$$
Solve $A x = b$ where
$$A = \begin{bmatrix} 10 & 1 \ 3 & 0.3 \end{bmatrix}~~~~~~\text{and}~~~~~~b=\begin{bmatrix} 11 \ 3.3 \end{bmatrix}$$
and we know that
$$x = \begin{bmatrix} 1 \ 1 \end{bmatrix}$$
When used in a solver we get
$$x = \begin{bmatrix}-0.5 \ 16 \end{bmatrix}$$
Evaluation of a polynomial
$$f(x) = x^7 - 7x^6 + 21 x^5 - 35 x^4 + 35x^3-21x^2 + 7x - 1$$
Can you write Python code for it?


I am working on the file "4_error.ipynb".
The first task is:
Smallest number that can be represented is the underflow:  $1.0 \times 2^{-1} = 0.25$
Largest number that can be represented is the overflow:  $1.1 \times 2^1 = 2.2$
Real Systems - IEEE 754 Binary Floating Point Systems
Single Precision

Total storage alloted is 32 bits
Exponent is 8 bits $\Rightarrow E \in [-126, 127]$
Fraction 23 bits ($p = 24$)

s EEEEEEEE FFFFFFFFFFFFFFFFFFFFFFF
0 1      8 9                     31
Overflow $= 2^{127} \approx 3.4 \times 10^{38}$
Underflow $= 2^{-126} \approx 1.2 \times 10^{-38}$
$\epsilon_{\text{machine}} = 2^{-23} \approx 1.2 \times 10^7$
Double Precision

Total storage alloted is 64 bits
Exponent is 11 bits $\Rightarrow E \in [-1022, 1024]$
Fraction 52 bits ($p = 53$)

s EEEEEEEEEE FFFFFFFFFF FFFFFFFFFF FFFFFFFFFF FFFFFFFFFF FFFFFFFFFF FF
0 1       11 12                                                      63
Overflow $= 2^{1024} \approx 1.8 \times 10^{308}$
Underflow $= 2^{-1022} \approx 2.2 \times 10^{-308}$
$\epsilon_{\text{machine}} = 2^{-52} \approx 2.2 \times 10^{-16}$
Python Access to IEEE Numbers
Access many important parameters, such as machine epsilon:
python
import numpy
numpy.finfo(float).eps
Can you write Python code for it?

print numpy.finfo(float).eps
print numpy.finfo(float).max
print numpy.finfo(float).min
print numpy.finfo(float).nmant
print numpy.finfo(float).precision
print numpy.finfo(float).tiny


I am working on the file "4_error.ipynb".
The first task is:
Sources of Error
Error can come from many sources when using applying a numerical method:
 - Model/Data Error
 - Truncation Error
 - Floating Point Error
Goal: Categorize and understand each type of error and explore some simple approaches to analyzing error.
Model and Data Error
Errors in fundamental formulation
 - Lotka-Volterra - fractional rabbits, no extinctions, etc.
 - Data Error - Inaccuracy in measurement or uncertainties in parameters
Unfortunatley we cannot control model and data error directly but we can use methods that may be more robust in the presense of these types of errors.
Truncation Error
Errors arising from approximating a function with a simpler function (e.g. $sin(x) \approx x$ for $|x| \approx 0$.
Floating Point Error
Errors arising from approximating real numbers with finite-precision numbers and arithmetic.
Basic Definitions
Given a true value of a function $f$ and an approximate solution $\hat{f}$ define:
Absolute Error:  $e = |f - \hat{f}|$
Relative Error:  $r = \frac{e}{|f|} = \frac{|f - \hat{f}|}{|f|}$
Decimal precision p (number of significant digits) - $p$ is the precision if and only if$$
$$x = \text{round}(10^{-n} \cdot x) \cdot 10^n$$
is true where
$$n = \text{floor}(\log_{10} x) + 1 - p$$
Can you write Python code for it?

f = numpy.exp(1)
f_hat = 2.71

# Error
print "Absolute Error = ", numpy.abs(f - f_hat)
print "Relative Error = ", numpy.abs(f - f_hat) / numpy.abs(f)

# Precision
p = 3
n = numpy.floor(numpy.log10(f_hat)) + 1 - p
print "%s = %s" % (f_hat, numpy.round(10**(-n) * f_hat) * 10**n)


I am working on the file "4_error.ipynb".
The first task is:
Sources of Error
Error can come from many sources when using applying a numerical method:
 - Model/Data Error
 - Truncation Error
 - Floating Point Error
Goal: Categorize and understand each type of error and explore some simple approaches to analyzing error.
Model and Data Error
Errors in fundamental formulation
 - Lotka-Volterra - fractional rabbits, no extinctions, etc.
 - Data Error - Inaccuracy in measurement or uncertainties in parameters
Unfortunatley we cannot control model and data error directly but we can use methods that may be more robust in the presense of these types of errors.
Truncation Error
Errors arising from approximating a function with a simpler function (e.g. $sin(x) \approx x$ for $|x| \approx 0$.
Floating Point Error
Errors arising from approximating real numbers with finite-precision numbers and arithmetic.
Basic Definitions
Given a true value of a function $f$ and an approximate solution $\hat{f}$ define:
Absolute Error:  $e = |f - \hat{f}|$
Relative Error:  $r = \frac{e}{|f|} = \frac{|f - \hat{f}|}{|f|}$
Decimal precision p (number of significant digits): $\text{round}(10^{-n} \cdot x) \cdot 10^n$
with $n = \text{floor}(\log_{10} x) + 1 - p$.
Can you write Python code for it?


I am working on the file "4_error.ipynb".
The first task is:
Symbols and Definitions
Big-O notation: $f(x) = \text{O}(g(x))$ as $x \rightarrow a$ if and only if $|f(x)| \leq M |g(x)|$ as $|x - a| < \delta$ for $M$ and $a$ positive.
In practice we use Big-O notation to say something about how the terms we may have left out of a series might behave.  We saw an example earlier of this with the Taylor's series approximations:
Example:
$f(x) = \sin x$ with $x_0 = 0$ then
$$T_N(x) = \sum^N_{n=0} (-1)^{n} \frac{x^{2n+1}}{(2n+1)!}$$
We can actually write $f(x)$ then as
$$f(x) = x - \frac{x^3}{6} + \frac{x^5}{120} + O(x^7)$$
This becomes more useful when we look at this as we did before with $\Delta x$:
$$f(x) = \Delta x - \frac{\Delta x^3}{6} + \frac{\Delta x^5}{120} + O(\Delta x^7)$$
We can also develop rules for error propagation based on Big-O notation:
In general, there are two theorems that do not need proof and hold when the value of x is large:
Let
$$f(x) = p(x) + O(x^n)$$
$$g(x) = q(x) + O(x^m)$$
$$k = \max(n, m)$$ then
$$f+g = p + q + O(x^k)$$
$$f \cdot g = p \cdot q + O(x^{n\cdot m})$$
On the other hand, if we are interested in small values of x, say Δx, the above expressions can be modified as follows:
$$f(\Delta x) = p(\Delta x) + O(\Delta x^n)$$
$$g(\Delta x) = q(\Delta x) + O(\Delta x^m)$$
$$r = \min(n, m)$$ then
$$f+g = p + q + O(\Delta x^r)$$
$$f \cdot g = p \cdot q + p \cdot O(\Delta x^m) + q \cdot O(\Delta x^n) + O(\Delta x^{n+m}) = p \cdot q + O(\Delta x^r)$$
Note 1: In this case we suppose that at least the polynomial with $k = \max(n, m)$ has the following form:
$$p(\Delta x) = 1 + p_1x + p_2x^2 + ...$$
or $$q(\Delta x) = 1 + q_1x + q_2x^2 + ...$$
so that there is an O(1) term that guarantees the existence of $O(\Delta x^r)$ in the final product.
To get a sense of why we care most about the power on $\Delta x$ when considering convergence the following figure shows how different powers on the convergence rate can effect how quickly we converge to our solution.  Note that here we are plotting the same data two different ways.  Plotting the error as a...
Can you write Python code for it?

dx = numpy.linspace(1.0, 1e-4, 100)

fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2.0)
axes = []
axes.append(fig.add_subplot(1, 2, 1))
axes.append(fig.add_subplot(1, 2, 2))

for n in xrange(1, 5):
    axes[0].plot(dx, dx**n, label="$\Delta x^%s$" % n)
    axes[1].loglog(dx, dx**n, label="$\Delta x^%s$" % n)

axes[0].legend(loc=2)
axes[1].set_xticks([10.0**(-n) for n in xrange(5)])
axes[1].set_yticks([10.0**(-n) for n in xrange(16)])
axes[1].legend(loc=4)
for n in xrange(2):
    axes[n].set_title("Growth of Error vs. $\Delta x^n$")
    axes[n].set_xlabel("$\Delta x$")
    axes[n].set_ylabel("Estimated Error")
    axes[n].set_title("Growth of different")
    axes[n].set_xlabel("$\Delta x$")
    axes[n].set_ylabel("Estimated Error")

plt.show()


I am working on the file "4_error.ipynb".
The first task is:
Symbols and Definitions
Big-O notation: $f(x) = \text{O}(g(x))$ as $x \rightarrow a$ if and only if $|f(x)| \leq M |g(x)|$ as $|x - a| < \delta$ for $M$ and $a$ positive.
In practice we use Big-O notation to say something about how the terms we may have left out of a series might behave.  We saw an example earlier of this with the Taylor's series approximations:
Example:
$f(x) = \sin x$ with $x_0 = 0$ then
$$T_N(x) = \sum^N_{n=0} (-1)^{n} \frac{x^{2n+1}}{(2n+1)!}$$
We can actually write $f(x)$ then as
$$f(x) = x - \frac{x^3}{6} + \frac{x^5}{120} + O(x^7)$$
This becomes more useful when we look at this as we did before with $\Delta x$:
$$f(x) = \Delta x - \frac{\Delta x^3}{6} + \frac{\Delta x^5}{120} + O(\Delta x^7)$$
We can also develop rules for error propagation based on Big-O notation:
Let
$$f(\Delta x) = p(\Delta x) + O(\Delta x^n)$$
$$g(\Delta x) = q(\Delta x) + O(\Delta x^m)$$
$$r = \min(n, m)$$ then
$$f+g = p + q + O(\Delta x^r)$$
$$f \cdot g = p \cdot q + p \cdot O(\Delta x^m) + q \cdot O(\Delta x^n) + O(\Delta x^{n+m}) = p \cdot q + O(\Delta x^r)$$
Horner's Method for Evaluating Polynomials
Given
$$P_N(x) = a_0 + a_1 x + a_2 x^2 + \ldots + a_N x^N$$
$$P_N(x) = p_1 x^N + p_2 x^{N-1} + p_3 x^{N-2} + \ldots + p_{N+1}$$
want to find best way to evaluate $P_N(x)$.  First consider two ways to write $P_3$:
$$ P_3(x) = p_1 x^3 + p_2 x^2 + p_3 x + p_4$$
and using nested multiplication:
$$ P_3(x) = ((p_1 x + p_2) x + p_3) x + p_4$$
Consider how many operations it takes for each...
$$ P_3(x) = p_1 x^3 + p_2 x^2 + p_3 x + p_4$$
$$P_3(x) = \overbrace{p_1 \cdot x \cdot x \cdot x}^3 + \overbrace{p_2 \cdot x \cdot x}^2 + \overbrace{p_3 \cdot x}^1 + p_4$$
Adding up all the operations we can in general think of this as a pyramid

We can estimate this way that the algorithm written this way will take approximately $O(N^2 / 2)$ operations to complete.
Looking at our other means of evaluation:
$$ P_3(x) = ((p_1 x + p_2) x + p_3) x + p_4$$
Here we find that the method is $O(N)$ (the 2 is usually ignored in these...
Can you write Python code for it?


I am working on the file "4_error.ipynb".
The first task is:
Truncation Error and Taylor's Theorem
Taylor's Theorem:  Let $f(x) \in C^{m+1}[a,b]$ and $x_0 \in [a,b]$, then for all $x \in (a,b)$ there exists a number $c = c(x)$ that lies between $x_0$ and $x$ such that
$$ f(x) = T_N(x) + R_N(x)$$
where $T_N(x)$ is the Taylor polynomial approximation
$$T_N(x) = \sum^N_{n=0} \frac{f^{(n)}(x_0)\cdot(x-x_0)^n}{n!}$$
and $R_N(x)$ is the residual (the part of the series we left off)
$$R_N(x) = \frac{f^{(n+1)}(c) \cdot (x - x_0)^{n+1}}{(n+1)!}$$
Another way to think about these results involves replacing $x - x_0$ with $\Delta x$.  The primary idea here is that the residual $R_N(x)$ becomes smaller as $\Delta x \rightarrow 0$.
$$T_N(x) = \sum^N_{n=0} \frac{f^{(n)}(x_0)\cdot\Delta x^n}{n!}$$
and $R_N(x)$ is the residual (the part of the series we left off)
$$R_N(x) = \frac{f^{(n+1)}(c) \cdot \Delta x^{n+1}}{(n+1)!} \leq M \Delta x^{n+1} = O(\Delta x^{n+1})$$
Example 1
$f(x) = e^x$ with $x_0 = 0$
Using this we can find expressions for the relative and absolute error as a function of $x$ assuming $N=2$.
$$f'(x) = e^x, ~~~ f''(x) = e^x ~~~ f^{(n)}(x) = e^x$$
$$T_N(x) = \sum^N_{n=0} e^0 \frac{x^n}{n!} ~~~~\Rightarrow ~~~~ T_2(x) = 1 + x + \frac{x^2}{2}$$
$$R_N(x) = e^c \frac{x^{n+1}}{(n+1)!} = e^c \cdot \frac{x^3}{6} ~~~~ \Rightarrow ~~~~ R_2(x) \leq \frac{e^1}{6} \approx 0.5$$
$$e^1 = 2.718\ldots$$
$$T_2(1) = 2.5 \Rightarrow e \approx 0.2 ~~ r \approx 0.1$$
We can also use the package sympy which has the ability to calculate Taylor polynomials built-in!
Can you write Python code for it?

import sympy
x = sympy.symbols('x')
f = sympy.symbols('f', cls=sympy.Function)

f = sympy.exp(x)
f.series(x0=0, n=6)


I am working on the file "4_error.ipynb".
The first task is:
Truncation Error and Taylor's Theorem
Taylor's Theorem:  Let $f(x) \in C^{m+1}[a,b]$ and $x_0 \in [a,b]$, then for all $x \in (a,b)$ there exists a number $c = c(x)$ that lies between $x_0$ and $x$ such that
$$ f(x) = T_N(x) + R_N(x)$$
where $T_N(x)$ is the Taylor polynomial approximation
$$T_N(x) = \sum^N_{n=0} \frac{f^{(n)}(x_0)\cdot(x-x_0)^n}{n!}$$
and $R_N(x)$ is the residual (the part of the series we left off)
$$R_N(x) = \frac{f^{(n+1)}(c) \cdot (x - x_0)^{n+1}}{(n+1)!}$$
Example 1
$f(x) = e^x$ with $x_0 = 0$
Here we will use the package sympy which has the ability to calculate Taylor polynomials built-in!
Can you write Python code for it?

import sympy
x = sympy.symbols('x')
f = sympy.symbols('f', cls=Function)

f = exp(x)
f.series(x0=0, n=3)


I am working on the file "4_error.ipynb".
The first task is:
Truncation Error vs. Floating Point Error
Truncation error:  Errors arising from approximation of a function, truncation of a series...
$$\sin x \approx x - \frac{x^3}{3!} + \frac{x^5}{5!} + O(x^7)$$
Floating-point Error:  Errors arising from approximating real numbers with finite-precision numbers
$$\pi \approx 3.14$$
or $\frac{1}{3} \approx 0.333333333$ in decimal, results form finitely number of registers to represent each number.
Floating Point Systems
Numbers in floating point systems are represented as a series of bits that represent different pieces of a number.  In nomalized floating point systems there are some standar conventions for what these bits are used for.  In general the numbers are stored by breaking them down into the form
$$\hat{f} = \pm d_1 . d_2 d_3 d_4 \ldots d_p \times \beta^E$$
where
1. $\pm$ is a single bit and of course represents the sign of the number
2. $d_1 . d_2 d_3 d_4 \ldots d_p$ is called the mantissa.  Note that technically the decimal could be moved but generally, using scientific notation, the decimal can always be placed at this location.  The digits $d_2 d_3 d_4 \ldots d_p$ are called the fraction with $p$ digits of precision.  Normalized systems specifically put the decimal point in the front like we have and assume $d_1 \neq 0$ unless the number is exactly $0$.
3. $\beta$ is the base.  For binary $\beta = 2$, for decimal $\beta = 10$, etc.
4. $E$ is the exponent, an integer in the range $[E_{\min}, E_{\max}]$
The important points on any floating point system is that
1. There exist a discrete and finite set of representable numbers
2. These representable numbers are not evenly distributed on the real line
3. Airthmetic in floating point systems yield different results from infinite precision arithmetic (i.e. "real" math)
Example:  Toy System
Consider the toy 2-digit precision decimal system (normalized)
$$f = \pm d_1 . d_2 \times 10^E$$
with $E \in [-2, 0]$.
Number and distribution of numbers

How many numbers can we represent with this system?

What is the...
Can you write Python code for it?

d_1_values = [1, 2, 3, 4, 5, 6, 7, 8, 9]
d_2_values = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
E_values = [0, -1, -2]

fig = plt.figure(figsize=(10.0, 1.0))
axes = fig.add_subplot(1, 1, 1)

for E in E_values:
    for d1 in d_1_values:
        for d2 in d_2_values:
            axes.plot( (d1 + d2 * 0.1) * 10**E, 0.0, 'r+', markersize=20)
            axes.plot(-(d1 + d2 * 0.1) * 10**E, 0.0, 'r+', markersize=20)

axes.plot(0.0, 0.0, '+', markersize=20)
axes.plot([-10.0, 10.0], [0.0, 0.0], 'k')

axes.set_title("Distribution of Values")
axes.set_yticks([])
axes.set_xlabel("x")
axes.set_ylabel("")
axes.set_xlim([-0.1, 0.1])
plt.show()


I am working on the file "4_error.ipynb".
The first task is:
Using this we can find expressions for the relative and absolute error as a function of $x$ assuming $N=2$.
$$T_N(x) = \sum^N_n=0 \frac{x^n}{n!} = 1 + x + \frac{x^2}{2}$$
$$R_N(x) = e^c \frac{x^{n+1}}{(n+1)!} = e^c \cdot \frac{x^3}{6}$$
$$e^1 = 2.718\ldots$$
$$T_2(1) = 2.5 \Rightarrow e \approx 0.2 ~~ r \approx 0.1$$
Lets do this numerically for a section of $x$.
Can you write Python code for it?


I am working on the file "4_error.ipynb".
The first task is:
What is the underflow and overflow limits?
Smallest number that can be represented is the underflow:  $1.0 \times 10^{-2} = 0.01$
Largest number that can be represented is the overflow:  $9.9 \times 10^0 = 9.9$
Properties of Floating Point Systems
All floating-point systems are characterized by several important numbers
 - Smalled normalized number (underflow if below)
 - Largest normalized number (overflow if above)
 - Zero
 - Machine $\epsilon$ or $\epsilon_{\text{machine}}$
 - inf and nan, infinity and Not a Number respectively
 - Subnormal numbers
Binary Systems
Consider the 2-digit precision base 2 system:
$$f=\pm d_1 . d_2 \times 2^E ~~~~ \text{with} ~~~~ E \in [-1, 1]$$
Number and distribution of numbers

How many numbers can we represent with this system?

What is the distribution on the real line?

What is the underflow and overflow limits?

How many numbers can we represent with this system?
$$f=\pm d_1 . d_2 \times 2^E ~~~~ \text{with} ~~~~ E \in [-1, 1]$$
$$ 2 \times 1 \times 2 \times 3 + 1 = 13$$
What is the distribution on the real line?
Can you write Python code for it?

d_1_values = [1]
d_2_values = [0, 1]
E_values = [1, 0, -1]

fig = plt.figure(figsize=(10.0, 1.0))
axes = fig.add_subplot(1, 1, 1)

for E in E_values:
    for d1 in d_1_values:
        for d2 in d_2_values:
            axes.plot( (d1 + d2 * 0.1) * 2**E, 0.0, 'r+', markersize=20)
            axes.plot(-(d1 + d2 * 0.1) * 2**E, 0.0, 'r+', markersize=20)

axes.plot(0.0, 0.0, 'r+', markersize=20)
axes.plot([-4.5, 4.5], [0.0, 0.0], 'k')

axes.set_title("Distribution of Values")
axes.set_yticks([])
axes.set_xlabel("x")
axes.set_ylabel("")
axes.set_xlim([-3, 3])
plt.show()


I am working on the file "4_error.ipynb".
The first task is:
Why should we care about this?

Floating point arithmetic is not commutative or associative
Floating point errors compound, do not assume even double precision is enough!
Mixing precision is very dangerous

### Examples
Simple arithmetic $\delta < \epsilon_{\text{machine}}$
$$(1+\delta) - 1 = 1 - 1 = 0$$
$$1 - 1 + \delta = \delta$$
Catastrophic cancelation (assume $x + y \neq 0$):
$$x(1 + \epsilon_x) + y (1 + \epsilon_y) = x + y + x \epsilon_x + y \epsilon_y =(x+y) \left( 1 + \frac{x \epsilon_x + y \epsilon_y}{x+y} \right ) \Rightarrow$$
$$\epsilon_{x+y} = \frac{x}{x + y} \epsilon_x  + \frac{y}{x+y} \epsilon_y$$
This implies that the error made from doing the actual sum can be arbitrarily large!  This is not too bad unless $x$ and $y$ have opposite sign, then if $x+y ~ 0$ then $\frac{1}{x+y}$ can be huge.  This is refered to as catastrophic cancelation.
Can you write Python code for it?


I am working on the file "4_error.ipynb".
The first task is:
Why should we care about this?

Floating point arithmetic is not commutative or associative
Floating point errors compound, do not assume even double precision is enough!
Mixing precision is very dangerous

### Examples
Simple arithmetic $\delta < \epsilon_{\text{machine}}$
$$(1+\delta) - 1 = 1 - 1 = 0$$
$$1 - 1 + \delta = \delta$$
Catastrophic cancelation (assume $x + y \neq 0$):
$$x(1 + \epsilon_x) + y (1 + \epsilon_y) = x + y + x \epsilon_x + y \epsilon_y =(x+y) \left( 1 + \frac{x \epsilon_x + y \epsilon_y}{x+y} \right ) \Rightarrow$$
$$\epsilon_{x+y} = \frac{x}{x + y} \epsilon_x  + \frac{y}{x+y} \epsilon_y$$
This implies that the error made from doing the actual sum can be arbitrarily large!  This is not too bad unless $x$ and $y$ have opposite sign, then if $x+y ~ 0$ then $\frac{1}{x+y}$ can be huge.  This is refered to as catastrophic cancelation.
Matrix vector multiplication
$$ \begin{bmatrix} \epsilon & 1 \ 1 & 1 \end{bmatrix} \cdot \begin{bmatrix} 1 \ 1 \end{bmatrix}
= \begin{bmatrix} \epsilon + 1 \ 1 + 1 \end{bmatrix} = \begin{bmatrix} 1 \ 2 \end{bmatrix}$$
Solve $A x = b$ where
$$A = \begin{bmatrix} 10 & 1 \ 3 & 0.3 \end{bmatrix}~~~~~~\text{and}~~~~~~b=\begin{bmatrix} 11 \ 3.3 \end{bmatrix}$$
and we know that
$$x = \begin{bmatrix} 1 \ 1 \end{bmatrix}$$
When used in a solver we get
$$x = \begin{bmatrix}-0.5 \ 16 \end{bmatrix}$$
Evaluation of a polynomial
$$f(x) = x^7 - 7x^6 + 21 x^5 - 35 x^4 + 35x^3-21x^2 + 7x - 1$$
Can you write Python code for it?


I am working on the file "4_error.ipynb".
The first task is:
Why should we care about this?

Floating point arithmetic is not commutative or associative
Floating point errors compound, do not assume even double precision is enough!
Mixing precision is very dangerous

### Examples
Simple arithmetic $\delta < \epsilon_{\text{machine}}$
$$(1+\delta) - 1 = 1 - 1 = 0$$
$$1 - 1 + \delta = \delta$$
Matrix vector multiplication
$$ \begin{bmatrix} \epsilon & 1 \ 1 & 1 \end{bmatrix} \cdot \begin{bmatrix} 1 \ 1 \end{bmatrix}
= \begin{bmatrix} \epsilon + 1 \ 1 + 1 \end{bmatrix} = \begin{bmatrix} 1 \ 2 \end{bmatrix}$$
Solve $A x = b$ where
$$A = \begin{bmatrix} 10 & 1 \ 3 & 0.3 \end{bmatrix}~~~~~~\text{and}~~~~~~b=\begin{bmatrix} 11 \ 3.3 \end{bmatrix}$$
and we know that
$$x = \begin{bmatrix} 1 \ 1 \end{bmatrix}$$
When used in a solver we get
$$x = \begin{bmatrix}-0.5 \ 16 \end{bmatrix}$$
Evaluation of a polynomial
$$f(x) = x^7 - 7x^6 + 21 x^5 - 35 x^4 + 35x^3-21x^2 + 7x - 1$$
Can you write Python code for it?


I am working on the file "4_scraping/4_2_scraping.ipynb".
The first task is:
RequestsでWebページを取得
Can you write Python code for it?

# Requestsでgihyo.jpのページのデータを取得
import requests
r = requests.get('http://gihyo.jp/lifestyle/clip/01/everyday-cat')
r.status_code # ステータスコードを取得

r.text[:50] # 先頭50文字を取得


I am working on the file "4_scraping/4_2_scraping.ipynb".
The first task is:
Requestsを使いこなす

connpass APIリファレンス https://connpass.com/about/api/
Can you write Python code for it?

# JSON形式のAPIレスポンスを取得
r = requests.get('https://connpass.com/api/v1/event/?keyword=python')
data = r.json() # JSONをデコードしたデータを取得
for event in data['events']:
    print(event['title'])

# 各種HTTPメソッドに対応
payload = {'key1': 'value1', 'key2': 'value2'}
r = requests.post('http://httpbin.org/post', data=payload)
r = requests.put('http://httpbin.org/put', data=payload)
r = requests.delete('http://httpbin.org/delete')
r = requests.head('http://httpbin.org/get')
r = requests.options('http://httpbin.org/get')

# Requestsの便利な使い方
r = requests.get('http://httpbin.org/get', params=payload)
r.url

r = requests.get('https://httpbin.org/basic-auth/user/passwd', auth=('user', 'passwd'))
r.status_code


I am working on the file "4_wake.ipynb".
The first task is:
Add the wakes in the lattice
Navigator defines step (dz) of tracking and which, if it exists, physical process will be applied on each step.
In order to add collective effects (Space charge, CSR or wake) method add_physics_proc() must be run.
Method:
* Navigator.add_physics_proc(physics_proc, elem1, elem2)
    - physics_proc - physics process, can be CSR, SpaceCharge or Wake,
    - elem1 and elem2 - first and last elements between which the physics process will be applied.
Also must be define unit_step in [m] (by default 1 m). unit_step is minimal step of tracking for any collective effect.
For each collective effect must be define number of unit_steps so step of applying physics process will be
dz = unit_step*step [m]
Can you write Python code for it?

navi = Navigator(lat)

# add physics proccesses
navi.add_physics_proc(wake_v1, w1_start, w1_stop)
navi.add_physics_proc(wake_h1, w2_start, w2_stop)
navi.add_physics_proc(wake_v2, w3_start, w3_stop)
navi.add_physics_proc(wake_h2, w4_start, w4_stop)
navi.add_physics_proc(wake_v3, w5_start, w5_stop)
navi.add_physics_proc(wake_h3, w6_start, w6_stop)

# definiing unit step in [m]
navi.unit_step = 0.2

# deep copy of the initial beam distribution
p_array = deepcopy(p_array_init)
tws_track, p_array = track(lat, p_array, navi)


I am working on the file "4_wake.ipynb".
The first task is:
Beam distribution
Can you write Python code for it?

tau = np.array([p.tau for p in p_array])
dp = np.array([p.p for p in p_array])
x = np.array([p.x for p in p_array])
y = np.array([p.y for p in p_array])

ax1 = plt.subplot(311)
ax1.plot(-tau*1000, x*1000, 'r.')
plt.setp(ax1.get_xticklabels(), visible=False)
plt.ylabel("x, mm")
plt.grid(True)

ax2 = plt.subplot(312, sharex=ax1)
ax2.plot(-tau*1000, y*1000, 'r.')
plt.setp(ax2.get_xticklabels(), visible=False)
plt.ylabel("y, mm")
plt.grid(True)

ax3 = plt.subplot(313, sharex=ax1)
ax3.plot(-tau*1000, dp, 'r.')
plt.ylabel("dp/p")
plt.xlabel("s, mm")
plt.grid(True)

# plotting twiss parameters.
plot_opt_func(lat, tws_track, top_plot=["Dx"], fig_name="i1", legend=False)
plt.show()


I am working on the file "4_wake.ipynb".
The first task is:
Initialization of the wakes and the places of their applying
Can you write Python code for it?

from ocelot.cpbd.wake3D import *

# load wake tables of corrugated structures
wk_vert = WakeTable('wake_vert_1m.txt')
wk_hor = WakeTable('wake_hor_1m.txt')

# creation of wake object with parameters
wake_v1 = Wake()
# w_sampling - defines the number of the equidistant sampling points for the one-dimensional
# wake coefficients in the Taylor expansion of the 3D wake function.
wake_v1.w_sampling = 500
wake_v1.wake_table = wk_vert
wake_v1.step = 1 # step in Navigator.unit_step, dz = Navigator.unit_step * wake.step [m]

wake_h1 = Wake()
wake_h1.w_sampling = 500
wake_h1.wake_table = wk_hor
wake_h1.step = 1

wake_v2 = deepcopy(wake_v1)

wake_h2 = deepcopy(wake_h1)

wake_v3 = deepcopy(wake_v1)

wake_h3 = deepcopy(wake_h1)


I am working on the file "4_wake.ipynb".
The first task is:
Layout of the corrugated structure insertion. Create Ocelot lattice <img src="4_layout.png" />
Can you write Python code for it?

D00m25 = Drift(l = 0.25)
D01m = Drift(l = 1)
D02m = Drift(l = 2)

# Create markers for defining places of the wakes applying
w1_start = Marker()
w1_stop = Marker()

w2_start = Marker()
w2_stop = Marker()

w3_start = Marker()
w3_stop = Marker()

w4_start = Marker()
w4_stop = Marker()

w5_start = Marker()
w5_stop = Marker()

w6_start = Marker()
w6_stop = Marker()
# quadrupoles
Q1 = Quadrupole(l = 0.5, k1 = 0.215)

# lattice
lattice = (D01m, w1_start, D02m, w1_stop, w2_start, D02m, w2_stop, w3_start, D02m, w3_stop, D00m25, Q1,
           D00m25, w4_start, D02m, w4_stop, w5_start, D02m, w5_stop, w6_start, D02m, w6_stop, D01m)

# creation MagneticLattice
method = MethodTM()
method.global_method = SecondTM
lat = MagneticLattice(lattice, method=method)

# calculate twiss functions with initial twiss parameters
tws0 = Twiss()
tws0.E = 14  # in GeV
tws0.beta_x = 22.5995
tws0.beta_y = 22.5995
tws0.alpha_x = -1.4285
tws0.alpha_y = 1.4285
tws = twiss(lat, tws0, nPoints=None)
# ploting twiss paramentrs.
plot_opt_func(lat, tws, top_plot=["Dx"], fig_name="i1", legend=False)
plt.show()


I am working on the file "4_wake.ipynb".
The first task is:
Load beam file
Can you write Python code for it?

p_array_init = astraBeam2particleArray(filename='beam_chirper.ast')

plt.plot(-p_array_init.tau()*1000, p_array_init.p(), "r.")
plt.grid(True)
plt.xlabel(r"$\tau$, mm")
plt.ylabel(r"$\frac{\Delta E}{E}$")
plt.show()


I am working on the file "4_wake.ipynb".
The first task is:
Longitudinal beam distribution
Can you write Python code for it?

tau0 = p_array_init.tau()
p0 = p_array_init.p()

tau1 = p_array.tau()
p1 = p_array.p()

plt.figure(1)
plt.plot(-tau0*1000, p0, "r.", -tau1*1000, p1, "b.")
plt.legend(["before", "after"], loc=4)
plt.grid(True)
plt.xlabel(r"$\tau$, mm")
plt.ylabel(r"$\frac{\Delta E}{E}$")
plt.show()


I am working on the file "4notebooks/ISTD Assessment.ipynb".
The first task is:
After specifying an experiment above, run the next block to get files for that experiment.
Can you write Python code for it?

mtools = reload(mtools)
files = mtools.get_files_for_experiment(experiment.value)


I am working on the file "4notebooks/ISTD Assessment.ipynb".
The first task is:
Check that everything is correct by running the next cell
Can you write Python code for it?

print "Method = ",method.value
print "Experiment = ",experiment.value
print len(files), " files queued for assessment"
print "filter strings are: ", qc_str.value, blank_str.value, pos_str.value, neg_str.value

# print qc_str.value,blank_str.value
import pandas as pd
sys.path.insert(0,'/global/project/projectdirs/metatlas/anaconda/lib/python2.7/site-packages' )
#sys.path.append('/project/projectdirs/metatlas/projects/ms_monitor_tools')
import metatlas_get_data_helper_fun as ma_data

from metatlas import metatlas_objects as metob
from metatlas import h5_query as h5q

from datetime import datetime
df = pd.DataFrame()
counter = 0
for my_file in files:
#     finfo = h5q.get_info(my_file.hdf5_file)
#     num_pos_data = finfo['ms1_pos']['nrows'] + finfo['ms2_pos']['nrows']
#     num_neg_data = finfo['ms1_neg']['nrows'] + finfo['ms2_neg']['nrows']
#     do_polarity = []
#     if num_pos_data > 0:
#         do_polarity.append('positive')
#     if num_neg_data > 0:
#         do_polarity.append('negative')
    df.loc[counter, 'name has blank'] = blank_str.value in my_file.name.upper()
    df.loc[counter, 'name has QC'] = qc_str.value in my_file.name.upper()
    df.loc[counter, 'name has pos'] = pos_str.value in my_file.name.upper()
    df.loc[counter, 'name has neg'] = neg_str.value in my_file.name.upper()
    df.loc[counter, 'experiment'] = my_file.experiment
    df.loc[counter, 'filename'] = my_file.name
    df.loc[counter, 'datestamp'] = my_file.creation_time
    df.loc[counter, 'utc time'] = datetime.utcfromtimestamp(my_file.creation_time)
    df.loc[counter, 'lcms method'] = my_file.method #TODO: get instrument and lcms from the method object
    df.loc[counter, 'sample'] = my_file.sample
    df.loc[counter, 'username'] = my_file.username
#     df.loc[counter, 'num positive data'] = num_pos_data
#     df.loc[counter, 'num negative data'] = num_neg_data
    counter = counter + 1
    print counter
#     for compound in atlas.compound_identifications:
#         if compound.mz_references[0].detected_polarity in...


I am working on the file "4notebooks/ISTD Assessment.ipynb".
The first task is:
Get Data from Reference
Can you write Python code for it?

#TODO: have another sheet that creates an Atlas from the google sheet.  Get the values from teh atlas instead.
qc_hilic_vals, common_hilic_vals, istd_hilic_vals = mtools.filter_istd_qc_by_method(method.value,0.3,5)
print "ok"


I am working on the file "4notebooks/ISTD Assessment.ipynb".
The first task is:
Run the next block to specify:

m/z tolerance (ppm)

Retention time tolerance (minutes)

20 ppm and 0.3 minutes are good for most runs
Can you write Python code for it?

# mz_tolerance, rt_tolerance = mtools.get_rt_mz_tolerance_from_user()

method = mtools.get_method_dropdown()


I am working on the file "4notebooks/ISTD Assessment.ipynb".
The first task is:
Specify the blank, qc and pos-neg string used in your file naming
Can you write Python code for it?


I am working on the file "4notebooks/ISTD Assessment.ipynb".
The first task is:
The "FutureWarning" above is normal.  Run the next block to select your experiment.
Can you write Python code for it?

num_days = raw_input('How many days back to search: ')
experiment = mtools.get_recent_experiments(num_days = int(num_days))


I am working on the file "4notebooks/ISTD Assessment.ipynb".
The first task is:
Uncomment and run the lines below to print the values in the reference table:
You can also view the source of these references here.
Can you write Python code for it?

# optional block to view the atlases selected

# istd_hilic_vals
# qc_hilic_vals
# common_hilic_vals


I am working on the file "4notebooks/old/examplenotebooks/Specify information about the experiment methods samples and files.ipynb".
The first task is:

Convert All Your Files Manually

This is typically not performed because the "load_files" command above has already taken care of it

Can you write Python code for it?

# myH5Files = []
# for f in myFiles:
#     metatlas.mzml_to_hdf('%s'%(f))
#     myH5Files.append(f.replace('.mzML','.h5'))
#     print f

print len(myExperiment.finfos)


I am working on the file "4notebooks/old/examplenotebooks/Specify information about the experiment methods samples and files.ipynb".
The first task is:

Create an experiment

Can you write Python code for it?

myExperiment = metatlas_objects.Experiment(name = 'QExactive_Hilic_Pos_Actinobacteria_Phylogeny')


I am working on the file "4notebooks/old/examplenotebooks/Specify information about the experiment methods samples and files.ipynb".
The first task is:

Get a list of mzML files that you uploaded and assign them to a group

Can you write Python code for it?

myPath = '/global/homes/b/bpb/ExoMetabolomic_Example_Data/'
myPath = '/project/projectdirs/metatlas/data_for_metatlas_2/20150324_LPSilva_BHedlund_chloroflexi_POS_rerun/'
myFiles = glob.glob('%s*.mzML'%myPath)
myFiles.sort()
groupID = []
for f in myFiles:
    groupID.append('')
i = 0
while i < len(myFiles):
    a,b = os.path.split(myFiles[i])
    j = raw_input('enter group id for %s [number, "x" to go back]:'%b)
    if j == 'x':
        i = i - 1
    else:
        groupID[i] = j
        i = i + 1

print groupID
uGroupID = sorted(set(groupID))
print uGroupID


I am working on the file "4notebooks/old/examplenotebooks/Specify information about the experiment methods samples and files.ipynb".
The first task is:

Specify the descriptive names for each group

Can you write Python code for it?

uGroupName = []
for u in uGroupID:
    j = raw_input('enter group name for Group #%s: '%u)
    uGroupName.append(j)


I am working on the file "4notebooks/old/examplenotebooks/Specify information about the experiment methods samples and files.ipynb".
The first task is:

Steps in the file description and conversion process

upload mzml files

glob to get list of mzml files

for a homogenous set of mzml files make a single filespec object with

metatlas_objects.FileSpec(polarity = ,group = inclus = )

Call an experiment, e = metatlas_objects.Experiment(name = 'Test_20150722')

e.load_files(mzmlfiles,sp)

repeat this process for each homogeneous set of files

Alternative, you can specify your own filespec object for each file

Can you write Python code for it?

fsList = []
for i,g in enumerate(groupID):
    for j,u in enumerate(uGroupID):
        if g == u:
            fs = metatlas_objects.FileSpec(polarity = 1,
                                          group = uGroupName[j],
                                          inclusion_order = i)
            fsList.append(fs)
            myExperiment.load_files([myFiles[i]],fs)

myExperiment.save()

print myExperiment.finfos[0].hdf_file
print myExperiment.finfos[0].group
print myExperiment.finfos[0].polarity


I am working on the file "5 - Dicts and files.ipynb".
The first task is:
A for loop on a dictionary iterates over its keys by default. The keys will appear in an arbitrary order. The methods dict.keys() and dict.values() return lists of the keys or values explicitly. There's also an items() which returns a list of (key, value) tuples, which is the most efficient way to examine all the key value data in the dictionary. All of these lists can be passed to the sorted() function.
Can you write Python code for it?

## By default, iterating over a dict iterates over its keys.
## Note that the keys are in a random order.
for key in dict: print key

## Exactly the same as above
for key in dict.keys(): print key

## Get the .keys() list:
print dict.keys()

## Likewise, there's a .values() list of values
print dict.values()  ## ['alpha', 'omega', 'gamma']

## Common case -- loop over the keys in sorted order,
## accessing each key/value
for key in sorted(dict.keys()):
    print key, dict[key]

## .items() is the dict expressed as (key, value) tuples
print dict.items()

## This loop syntax accesses the whole dict by looping
## over the .items() tuple list, accessing one (key, value)
## pair on each iteration.
for k, v in dict.items(): print k, '>', v


I am working on the file "5 - Dicts and files.ipynb".
The first task is:
Del
The "del" operator does deletions. In the simplest case, it can remove the definition of a variable, as if that variable had not been defined. Del can also be used on list elements or slices to delete that part of the list and to delete entries from a dictionary.
Can you write Python code for it?

var = 6
del var  # var no more!
var

list = ['a', 'b', 'c', 'd']
del list[0]     ## Delete first element
del list[-2:]   ## Delete last two elements
print list

dict = {'a':1, 'b':2, 'c':3}
del dict['b']   ## Delete 'b' entry
print dict


I am working on the file "5 - Dicts and files.ipynb".
The first task is:
Files
The open() function opens and returns a file handle that can be used to read or write a file in the usual way. The code f = open('name', 'r') opens the file into the variable f, ready for reading operations, and use f.close() when finished. Instead of 'r', use 'w' for writing, and 'a' for append. The special mode 'rU' is the "Universal" option for text files where it's smart about converting different line-endings so they always come through as a simple '\n'. The standard for-loop works for text files, iterating through the lines of the file (this works only for text files, not binary files). The for-loop technique is a simple and efficient way to look at all the lines in a text file:
Can you write Python code for it?

  # Echo the contents of a file
  f = open('data/poem.txt', 'rU')
  for line in f:   ## iterates over the lines of the file
    print line,    ## trailing , so print does not add an end-of-line char
                   ## since 'line' already includes the end-of line.
  f.close()


I am working on the file "5 - Dicts and files.ipynb".
The first task is:
There are "iter" variants of these methods called iterkeys(), itervalues() and iteritems() which avoid the cost of constructing the whole list -- a performance win if the data is huge. However, I generally prefer the plain keys() and values() methods with their sensible names.
Strategy note: from a performance point of view, the dictionary is one of your greatest tools, and you should use where you can as an easy way to organize data. For example, you might read a log file where each line begins with an ip address, and store the data into a dict using the ip address as the key, and the list of lines where it appears as the value. Once you've read in the whole file, you can look up any ip address and instantly see its list of lines. The dictionary takes in scattered data and make it into something coherent.
Dict Formatting
The % operator works conveniently to substitute values from a dict into a string by name:
Can you write Python code for it?

hash = {}
hash['word'] = 'garfield'
hash['count'] = 42
s = 'I want %(count)d copies of %(word)s' % hash
s


I am working on the file "5 - Generate coupon probabilities - part 3.ipynb".
The first task is:
It looks like using  random variates will make the sim run a little slower, but not by a crazy amount if we use numpy's variate generation.  Stay away from pure python and scipy's version though, they'll kill the speed.
Now that we are drawing coupons using random variates instead of generating a cdf, we can simulate the ccp with a single function:
Can you write Python code for it?

def single_run_looped(n, dist, alpha, beta):
    """
    This is a single run of the CCP.
    n = number of unique coupons
    m_max = max number of draws to simulate (should be much greater than n)
    dist = how are the coupon probabilities distributed (uniform, normal, exponential)
    norm_scale = how much to scale the normal distribution standard deviation (0<= norm_scale <1)
    """
    m = 0 #start at zero draws
    cdf = (arange(n)+1.0)/n #create the draw probability distribution
    draws = [] #create our draw array
    uniques = [] #create our unique array (deque is faster but may break DB inserts - research)
    unique = 0
    while True:
        m+=1 #increment our draw counter
        rv = npbeta(alpha, beta) #randomness that decides which coupon to draw
        draw = (cdf>rv).sum()
        if draw not in draws:
            draws.append(draw)
            unique+=1
        uniques.append(unique) #store the info
        if unique==n:#we'll stop once we have drawn all our coupons
            return m #this line returns the number of draws; for testing
            #return uniques #this line returns the full unique draws list; the actual data we want to record

vectorized_single_run_looped = vectorize(single_run_looped)

start = timeit.default_timer()
#test our sim with known results on uniform probs
trials = 200000
n = 10
records = vectorized_single_run_looped([n]*trials, ['beta']*trials, [1.0]*trials, [1.0]*trials)
num_fails = np.where(records==0)
average = np.mean(records)
std_error = np.std(records)/np.sqrt(trials)
z_crit = 1.96
low_ci = average - z_crit*std_error
high_ci = average + z_crit*std_error
expectation = np.asarray([1/(n+1) for n in np.arange(n)]).sum()*n
et = timeit.default_timer()-start
print ("num_fails: ", len(num_fails[0]))
print("low_ci: ", low_ci, "point_est: ", average, "high_ci: ", high_ci)
print("expected value: ", expectation)
print("elapsed_time: ", et)


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
Alright we have our dataset in a hierarchical index. but that's not what we want to work with right now which will likely come up when you're analyzing data - you'll want to completely reset your index. Well have no fear, we can do that with the reset index command.
You may find yourself using this often just to get back to square one and start over when performing analysis.
Can you write Python code for it?


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
But when we use the head method to see the first 5 we see things are a little different and that’s because we now have multiple indices or a hierarchical or multi index. Now hierarchical indexes are extremely powerful but they’re beyond the scope of this current video - I'll touch on them a bit later in this section.
What you need to know right now is that there are levels that are stacked on one another and those can be queried.
Can you write Python code for it?


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
However let’s touch on the basis because you may come across them.
We've got a lot of the basic methods like shape.
Can you write Python code for it?


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
In this section we will be analyzing some financial data. Now pandas gives us access to some data through pandas.io.data
This is basically pandas remote data access:
http://pandas.pydata.org/pandas-docs/stable/remote_data.html
Functions from pandas.io.data and pandas.io.ga extract data from various Internet sources into a DataFrame. Currently the following sources are supported:

Yahoo! Finance
Google Finance
St.Louis FED (FRED)
Kenneth French’s data library
World Bank
Google Analytics

So let’s grab some  stocks from yahoo data with pandas.io.data. I've seen this list actively change so it's a good idea to see what is available to you - there's likely some really cool plugins that will continue to be built.
Can you write Python code for it?

import pandas.io.data

?pandas.io.data # <tab>


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
Now there’s a lot of volatility in oil right now. It’s been rough for producers to say the least. So let’s check out some stocks that are involved in that market.
First we’ll set start and end dates these are just date times. Now I can do this with date times in python but I can also just get a datetime with pandas which can parse a string to pull out a date time. This ends up being pretty handy.
Can you write Python code for it?


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
Now we can get these one by one in a for loop...
Can you write Python code for it?

for symbol in ticker_symbols:
    print(symbol)
    df = pd.io.data.get_data_yahoo(symbol,start=start,end=end)


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
Now we've reset our index. Now I don't want to work with this data set as I'm not an expert on financial data however we'll be working with a really cool data set in our next video. We'll be working with an airplane data set that has flights across the country. This is going to give us the opportunity to work on a ton of cool problems.
Can you write Python code for it?


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
Some summary statistics are available to us like mean on the panel.
Can you write Python code for it?


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
The major and minor axises are done differently. with the major_xs and minor_xs commands.
Can you write Python code for it?


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
We can perform different kinds of selections and transposition using the major and minor axes however as I said above I'm not going to cover this material. However I’m going to convert this panel to a data frame to show you how to do that and in the process and we’re going to cover a new topic!
Now when we convert the panel to a data frame with the to_frame command - we can see it looks a bit different.
Can you write Python code for it?


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
We can see they’ve got an items axis, a major axis and a minor axis. Panels are a core part of pandas however they are much less used in pandas and therefore are a bit neglected as of now. That’s not just me trying to avoid the topic - which would be my suspicion if I heard that. But that ’s stated almost verbatim in the docs.
http://pandas-docs.github.io/pandas-docs-travis/dsintro.html#from-dataframe-using-to-panel-method
Can you write Python code for it?


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
We’ve got these three axes so we want to query data in them. We've got to do that a bit differently.
Since we know these axis values we can query them.
Now items are done like standard DataFrame columns with dot syntax.
Can you write Python code for it?


I am working on the file "5 - pandas Advanced/5-1 Pandas IO Data, Different Ways of Indexing Data, Hierarchical Indexing and Panels.ipynb".
The first task is:
or we can just get them all by passing in the list....
Can you write Python code for it?


I am working on the file "5 - pandas Advanced/5-2 pandas DataFrame advaced read_csv, read_html, binning, categorical data, more grouping, Apply, Transforming, Aggregate functions.ipynb".
The first task is:
Now categorical data is different that what we're used to, we can see that it has a hierarchy. There's things that are less than other things and therefore is ordered. This can come in handy in certain types of analysis.
Really what this allows us to do is describe the data in terms of those categories
Can you write Python code for it?