I am working on the file "Session3/code/04 Unsupervised and supervised Learning.ipynb".
The first task is:
This new classification is a vast improvement over the previous one !
We are now going to do a visualisation of what has exactly happened :

2D plot of the data projected by tsne
location of centroids in the data (how well our k-MEANS algorithm picks up the new structure in the data).
We will see that while not perfects, the centroid detection works very well for some digits.
Can you write Python code for it?

# color map, one color per digit
list_color = ["r", "g", "b", "k", "plum", "pink", "lightseagreen", "blueviolet", "darkgray", "sandybrown"]

# dictionnary of color to be used in the plot
d_color = {}
for i in range(10) :
    d_color[i] = list_color[i]

fig = plt.figure()
ax = fig.add_subplot(111)

# Plot the data
for i in range(2000):
    ax.text(data_subset_tsne[i,0], data_subset_tsne[i,1], str(class_subset[i]), color=d_color[class_subset[i]], fontsize=12)

# Also plot the cluster centers
for c in km.cluster_centers_ :
    ax.plot(c[0], c[1], "x", color = "k", markersize = 15, markeredgewidth=4)

# choose the boundaries of the plot for an ideal view
ax.set_xlim([-2 + min(data_subset_tsne[:,0]),2 + max(data_subset_tsne[:,0])])
ax.set_ylim([-2 + min(data_subset_tsne[:,1]),2 + max(data_subset_tsne[:,1])])
plt.show()


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Advanced: Visualizing two categorical (also discrete) features
Heat Map
A Heat Map is a two-dimensional representation of data in which values are represented by colors. A simple heat map provides an immediate visual summary of information.
In this exercise, we will use "Pregnancy" and "Class" features:
1. Use only "Pregnancy" and "Class" columns: you may create a new dataframe on which we will work
2. We will use groupby to group the new two column dataframe based on both "Pregnancy" and "Class" features
3. On top of that use size() function to get the count of every possible pair of values of "Pregnancy" and "Class"
4. Then the new dataframe is reindexed using reset_index() with name="count" argument to set a name for the count column
5. A dataframe pivot table is generated using all three columns in the following order: "Pregnancy", "Class", "count"
6. Finally, this pivot table is used to generate the seaborn.heatmap
Note: This is an advanced piece of code. So you may require to consult different places before you get this right. Do not hesitate to ask for help.
Can you write Python code for it?

# Write code to create a Heat Map using the above steps
df2 = df[["Pregnancy", "Class"]].groupby(["Pregnancy", "Class"]).size().reset_index(name="count") # TO DELETE
sns.heatmap(df2.pivot("Pregnancy", "Class", "count"))


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
As you can discover from PCA documention (link above), PCA returns principal components as a numpy array. For the ease of plotting with seaborn, we will create a pandas DataFrame from the principal components.

use pandas.DataFrame() to convert numpy array
mark the columns as "PC1" and "PC2"
update the dateframe by adding the "Class" column of our original dataframe using DataFrame.join()
Can you write Python code for it?

# Write code to convert array2PC to a DataFrame with columns "PC1" and "PC2"
df2PC = pd.DataFrame(array2PC, columns=["PC1", "PC2"])

# Write code to update df2PC by appending "Class" column from orginal dataframe df
# Hint: using "join" on df2PC
df2PC = df2PC.join(df["Class"])


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Density Plot
The Density Plot gives interesting informations w.r.t. the type of probability distribution that you can assume for the feature.
Can you write Python code for it?

# Write code to draw a Density Plot of "BadyMass" feature
# Hint: use "DataFrame.plot.density" from pandas on the dataframe df
df['BodyMass'].plot.density()


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Let us now get some other information from the "Pregnancy" feature. We will now visualize the distribution of number of females for different "Pregnancy" values.  For that:
1. First count the number of samples for each possible "Pregnancy" value (hint: DataFrame.value_counts())
2. Plot the distribution using Bar Chart
Can you write Python code for it?

# Step 1: Write code to generate the count distrubution
df["Pregnancy"].value_counts()

# Step 2: Write code to create Bar Chart for the count distrubution
df["Pregnancy"].value_counts().plot.bar()


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Now, we will create a scatter plot to visualize our 7D data transformed into 2 principal components.
For creating scatter plots using seaborn we will use lmplot module with fit_reg=False.
Can you write Python code for it?

# For displaying the plots inside Notebook
%matplotlib inline

# Write code to create scatter plot for 2 PCs.
# Hint 1; use seaborn.lmplot and set fit_reg=False
# Hint 2: use hue option to visualize the "Class" labels in the plot
sns.lmplot("PC1", "PC2", df2PC, hue="Class", fit_reg=False)


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Now, you have to do PCA on the data as before but for 3 principal components. Then plot 3 principal components in a 3D scatter plot.
Hint: The hue equivalent for 3D scatter plot is c, and you have to pass the entire "Class" column, not just the name.
Can you write Python code for it?

# Write code to do PCA with 3 principal components on the dataframe with columns_pca
pca3 = PCA(n_components = 3)
array3PC = pca3.fit_transform(df[columns_pca])

df3PC = pd.DataFrame(array3PC, columns=["PC1", "PC2", "PC3"]) # TO DELETE
df3PC = df3PC.join(df["Class"])

fig_pca = plt.figure()
ax = fig_pca.add_subplot(111, projection='3d')
ax.scatter(df3PC["PC1"], df3PC["PC2"], df3PC["PC3"], c=df3PC["Class"])


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Question: Draw box plots of "BodyMass" and "Glucose" together. Why is the visualization is misleading?
Can you write Python code for it?

# Write code to draw Box Plot for "BodyMass" and "Glucose" together
# Hint: you can pass a list of features for "column" parameter
df.boxplot(column = ['BodyMass','Glucose'], grid='off')


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Question: Draw density plot for all continuous features. Why this visualization do not have above problem?
Can you write Python code for it?

# Write code to draw Density Plot for all 4 continuous features together
# Hint: you can filter dataframe by a list of columns and then use plot.density
df[['BodyMass','Glucose','Fold']].plot.density()


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Question: for each of the 8 features and the class, define their nature

Continuous:
Discrete:
Categorical:

Visualizing 1D Data
1D plots are a good way to detect outliers, level of variability in the features etc.
Here is a non-exhaustive list of possible methods:

For continuous features: Box Plot, Histogram, Density Representation
For discrete features: Bar Chart, Dot Chart
For categorical features: Pie Chart is most common. There exist a lot of newer representations, but in the end they all provide the exact same information.

Visualizing continuous features
Box Plot
The Box Plot is an interesting tool for data visualisation as it contains multiple statistics informations about the feature: the minimum and maximum values, the first and third quartiles (bottom and top line of the box),the median value (middle line in the box) and the range (dispersion).
We will use "BodyMass" as the example here.
Can you write Python code for it?

# Write code to draw a Box Plot of "BadyMass" feature
# Hint 1: use "DataFrame.boxplot" from pandas on the dataframe df
# Hint 2: choose the column properly
# Hint 3: you can control "grid" parameter as True or False
df.boxplot(column = 'BodyMass', grid=False)


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Questions:
1. Can you detect values?
2. Can you visualize any correlation between "BodyMass" and "Fold?
Remark: Pandas plot module is very useful for basic visualization techniques (more details here).
Now we will use explore seaborn library to create more advanced visualizations.
First, we will see seaborn jointplot(). It shows bivariate scatter plots and univariate histograms in the same figure.
Can you write Python code for it?

# Write code to create a jointplot using "BodyMass" and "Fold" features
# Hint 1: mention the "x" and "y" axis features, and out dataframe df as "data"
# Hint 2: "size" parameter controls the size of the plot. Here try size=6
sns.jointplot(x="BodyMass", y="Fold", data=df, size=6)


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Remark: Pie charts are very effective to visualize distribution of classes on the training data. It helps to discover if there exist a stong imbalance of classes in the data.
Warning: Pie charts cannot show more than a few values as the slices become too small. This makes them unsuitable for use with categorical features with a larger number of possible values.
Visualizing 2D Data
2D plots (or in multi-D plots in general) are important to detect potential dependencies in the data (colinearity, linearity etc.).
Again, the nature of the features will guide you to choose the good representation.
* 1 continuous vs. 1 continuous: Scatter Plot, Pair Plot
* 1 continous vs. 1 categorical: Box Plot (yes again!), Violin Plot (very similar to boxplot)
* 1 categorical vs. 1 categorical: Heatmap (to visualize counts per category or mean per category)
Visualizing continuous vs. continuous features
Scatter Plot
Scatter Plot is used to display values for typically two variables for a set of data on Cartesian coordinates. The data are displayed as a collection of points, each having the value of one variable determining the position on the horizontal axis and the value of the other variable determining the position on the vertical axis. A scatter plot can suggest various kinds of correlations between features.
Let's use "BodyMass" and "Fold" features together.
Can you write Python code for it?

# Write code to create Scatter Plot between "BodyMass" and "Fold" features
# Hint: use "DataFrame.plot.scatter()" on our dataframe df, and mention the "x" and "y" axis features
df.plot.scatter(x="BodyMass", y="Fold")


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Remark: The box plot highlight the presence of outliers (circles on the figure). One of the individuals has a BodyMass of 0, which is impossible. In this case, 0 is the code for missing values.
Histogram
The histogram groups data into intervals and is good to look at the distribution of the values.
Can you write Python code for it?

# Write code to draw a Histogram of "BadyMass" feature
# Hint 1: use "DataFrame.hist" from pandas on the dataframe df
# Hint 2: choose the column properly
# Hint 3: you can control "grid" parameter as True or False
# Hint 4: for this plot choose "bins" as 10, "alpha" as 0.5 and "ec" as "black"
df.hist(column = 'BodyMass', grid='off', bins=10, alpha=0.5, ec='black')


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Subplots
For the box plot and histogram, to visualize each feature in its own scale, it is better to draw one plot per feature. All these plots can be arranged in a grid.
Task: See the usage of plt.subplot(). Then draw:
1. One figure containing box plots of all 4 continuous features
2. One figure containing histograms of all 4 continuous features
Note: You can also play with basic customization for each figure (labeling, title, colors etc.)
Can you write Python code for it?

# Write code to create subplots to vizualize four continous features
# Hint: use plt.pyplot() in a 2 by 2 grid. You can adjust these using "nrows" and "ncols"
fig,axes = plt.subplots(nrows = 2,ncols = 2) # TO DELETE
df.hist('BodyMass', bins=10, grid='off', alpha=0.5, ec='black', ax = axes[0, 0])
df.hist('Glucose', bins=10, grid='off', alpha=0.5, ec='black', ax = axes[0, 1])
df.hist('Fold', bins=10, grid='off', alpha=0.5, ec='black', ax = axes[1, 0])
df.hist('BloodPress', bins=10, grid='off', alpha=0.5, ec='black', ax = axes[1, 1])

# For a neat display of plots
plt.tight_layout()


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Visualizing 3D Data
3D plots lets you visualize 3 features together. Like, 2D plots, 3D plots are used to analyze potential dependencies in the data (colinearity, linearity etc.).
Here also, based on the nature of the features you can choose the type of visualization. However, in this exercise we will explore to visualize 3 continuous features together.
In this part, we will use Axes3D module from matplotlib library. We will explore 3D scatter plot. For other kinds of 3D plots you can refer here.
Import Libraries
Can you write Python code for it?

from mpl_toolkits.mplot3d import Axes3D

# For interactive 3D plots
%matplotlib notebook


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Visualizing categorical features
Most common way to visualize the distribution of categorical feature is Pie Chart. It is a circular statistical graphic which is divided into slices to illustrate numerical proportion of different posible values of the categorical feature. The arc length of each slice is proportional to the quantity it represents.
As we are visualizing the count distribution, the first step is to use DataFrame.value_counts()
Can you write Python code for it?

# Write code to create a Pie Chart of the "Class" feature.
# Hint 1: plot the count distrubution, NOT the data itself
# Hint 2: use plot.pie() on the count distrubution
# Hint 3: use autopct="%1.1f%%" to display percentage values and following colors
colors = ['gold', 'lightcoral']
df['Class'].value_counts().plot.pie(autopct='%1.1f%%', colors=colors)

# For a neat display of the plot
_ = plt.axis('equal')


I am working on the file "Session4/DataViz_keys.ipynb".
The first task is:
Visualizing discrete features
Bar Chart or Bar Graph presents discrete data with rectangular bars with heights proportional to the values that they represent.
In this exerise, we will visualize "Pregnancy" feature.
Can you write Python code for it?

# Write code to create a Bar Chart of the "Pregnancy" feature.
df.Pregnancy.plot.bar() # TO DELETE

# For a neat display of plots: plating with the X-tick labels
_ = plt.xticks(list(df.index)[::100], list(df.index)[::100])


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
Error Estimates
This maximum likelihood value gives our best estimate of the parameters $\mu$ and $\sigma$ governing our model of the source. But this is only half the answer: we need to determine how confident we are in this answer, that is, we need to compute the error bars on $\mu$ and $\sigma$.
To see how this is done in the frequentist paradigm, see the sub-slides.
There are several approaches to determining errors in a frequentist paradigm.  We could:
* as above, fit a normal approximation to the maximum likelihood and report the covariance matrix (here we'd have to do this numerically rather than analytically).
* Alternatively, we can compute statistics like $\chi^2$ and $\chi^2_{\rm dof}$ to and use standard tests to determine confidence limits, which also depends on strong assumptions about the Gaussianity of the likelihood.
* We might alternatively use randomized sampling approaches such as "Jackknife"...
* or "Bootstrap", which maximize the likelihood for randomized samples of the input data in order to explore the degree of certainty in the result.
All of these would be valid techniques to use, but each comes with its own assumptions and subtleties.  Here, for simplicity, we'll use the basic bootstrap resampler found in the astroML package:
Can you write Python code for it?

from astroML.resample import bootstrap

def fit_samples(sample):
    # sample is an array of size [n_bootstraps, n_samples]
    # compute the maximum likelihood for each bootstrap.
    return np.array([optimize.fmin(neg_log_likelihood, theta_guess,
                                   args=(F, np.sqrt(F)), disp=0)
                     for F in sample])

samples = bootstrap(F, 1000, fit_samples)  # 1000 bootstrap resamplings


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
I should note that there is a huge literature on the details of bootstrap resampling, and there are definitely some subtleties of the approach that I am glossing over here. One obvious piece is that there is potential for errors to be correlated or non-Gaussian, neither of which is reflected by simply finding the mean and standard deviation of each model parameter. Nevertheless, I trust that this gives the basic idea of the frequentist approach to this problem.
Varying Photon Counts: The Bayesian Approach
The Bayesian approach to this problem is almost exactly the same as it was in the previous problem, and we can set it up by slightly modifying the above code.
Can you write Python code for it?

def log_prior(theta):
    # sigma needs to be positive.
    if theta[1] <= 0:
        return -np.inf
    else:
        return 0

def log_posterior(theta, F, e):
    return log_prior(theta) + log_likelihood(theta, F, e)

# same setup as above:
ndim, nwalkers = 2, 50
nsteps, nburn = 2000, 1000

starting_guesses = np.random.rand(nwalkers, ndim)
starting_guesses[:, 0] *= 2000  # start mu between 0 and 2000
starting_guesses[:, 1] *= 20    # start sigma between 0 and 20

sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior, args=[F, e])
sampler.run_mcmc(starting_guesses, nsteps)

sample = sampler.chain  # shape = (nwalkers, nsteps, ndim)
sample = sampler.chain[:, nburn:, :].reshape(-1, 2)


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
If this all worked correctly, the array sample should contain a series of 50000 points drawn from the posterior. Let's plot them and check:
Can you write Python code for it?

# plot a histogram of the sample
plt.hist(sample, bins=50, histtype="stepfilled", alpha=0.3, normed=True)

# plot a best-fit Gaussian
F_fit = np.linspace(975, 1025)
pdf = stats.norm(np.mean(sample), np.std(sample)).pdf(F_fit)

plt.plot(F_fit, pdf, '-k')
plt.xlabel("F"); plt.ylabel("P(F)")


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
Now let's make a simple visualization of the "measured" data:
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

fig, ax = plt.subplots()
ax.errorbar(F, np.arange(N), xerr=e, fmt='ok', ecolor='gray', alpha=0.5)
ax.vlines([F_true], 0, N, linewidth=5, alpha=0.2)
ax.set_xlabel("Flux");ax.set_ylabel("measurement number");


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
Now that we have the samples, we'll use a convenience routine from astroML to plot the traces and the contours representing one and two standard deviations:
Can you write Python code for it?

from astroML.plotting import plot_mcmc
fig = plt.figure()
ax = plot_mcmc(sample.T, fig=fig, labels=[r'$\mu$', r'$\sigma$'], colors='k')
ax[0].plot(sample[:, 0], sample[:, 1], ',k', alpha=0.1)
ax[0].plot([mu_true], [sigma_true], 'o', color='red', ms=10);


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
Varying Photon Counts: The Frequentist Approach
The resulting likelihood is the convolution of the intrinsic distribution with the error distribution, so we have
$$\mathcal{L}(D~|~\theta) = \prod_{i=1}^N \frac{1}{\sqrt{2\pi(\sigma^2 + e_i^2)}}\exp\left[\frac{-(F_i - \mu)^2}{2(\sigma^2 + e_i^2)}\right]$$
Analogously to above, we can analytically maximize this likelihood to find the best estimate for $\mu$:
$$\mu_{est} =  \frac{\sum w_i F_i}{\sum w_i};~~w_i = \frac{1}{\sigma^2 + e_i^2} $$
And here we have a problem: the optimal value of $\mu$ depends on the optimal value of $\sigma$.  The results are correlated, so we can no longer use straightforward analytic methods to arrive at the frequentist result.
Nevertheless, we can use numerical optimization techniques to determine the maximum likelihood value. Here we'll use the optimization routines available within Scipy's optimize submodule:
Can you write Python code for it?

def log_likelihood(theta, F, e):
    return -0.5 * np.sum(np.log(2 * np.pi * (theta[1] ** 2 + e ** 2))
                         + (F - theta[0]) ** 2 / (theta[1] ** 2 + e ** 2))

# maximize likelihood <--> minimize negative likelihood
def neg_log_likelihood(theta, F, e):
    return -log_likelihood(theta, F, e)

from scipy import optimize
theta_guess = [900, 5]
theta_est = optimize.fmin(neg_log_likelihood, theta_guess, args=(F, e))
print("""
      Maximum likelihood estimate for {0} data points:
          mu={theta[0]:.0f}, sigma={theta[1]:.0f}
      """.format(N, theta=theta_est))


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
We find that for 50 measurements of the flux, our estimate has an error of about 0.4% and is consistent with the input value.
Bayesian Approach to Simple Photon Counts
The Bayesian approach, as you might expect, begins and ends with probabilities.  It recognizes that what we fundamentally want to compute is our knowledge of the parameters in question, i.e. in this case,
$$ P(F_{\rm true}~|~D) $$
N.b.: that this formulation of the problem is fundamentally contrary to the frequentist philosophy, which says that probabilities have no meaning for model parameters like $F_{\rm true}$. Within the Bayesian interpretation of probability, this is perfectly acceptable.
To compute this result, Bayesians next apply Bayes' Theorem, a fundamental law of probability:
$$ P(F_{\rm true}~|~D) = \frac{P(D~|~F_{\rm true})~P(F_{\rm true})}{P(D)} $$
Though Bayes' theorem is where Bayesians get their name, it is not this law itself that is controversial, but the Bayesian interpretation of probability implied by the term $P(F_{\rm true}~|~D)$.
Let's take a look at each of the terms in this expression:

$P(F_{\rm true}~|~D)$: The posterior, or the probability of the model parameters given the data: this is the result we want to compute.
$P(D~|~F_{\rm true})$: The likelihood, which is proportional to the $\mathcal{L}(D~|~F_{\rm true})$ in the frequentist approach, above.
$P(F_{\rm true})$: The model prior, which encodes what we knew about the model prior to the application of the data $D$.
$P(D)$: The data probability, which in practice amounts to simply a normalization term.

If we set the prior $P(F_{\rm true}) \propto 1$ (a flat prior), we find
$$P(F_{\rm true}|D) \propto \mathcal{L}(D|F_{\rm true})$$
and the Bayesian probability is maximized at precisely the same value as the frequentist result! So despite the philosophical differences, we see that (for this simple problem at least) the Bayesian and frequentist point estimates are equivalent.
But What About the Prior?
We glossed over something here: the prior, $P(F_{\rm...
Can you write Python code for it?

def log_prior(theta):
    return 1  # flat prior

def log_likelihood(theta, F, e):
    return -0.5 * np.sum(np.log(2 * np.pi * e ** 2)
                         + (F - theta[0]) ** 2 / e ** 2)

def log_posterior(theta, F, e):
    return log_prior(theta) + log_likelihood(theta, F, e)


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
By eye, this seems to have worked as desired: the fit is much closer to our intuition!
However a Bayesian might point out that the motivation for this new loss function is a bit suspect: as we showed, the squared-loss can be straightforwardly derived from a Gaussian likelihood. The Huber loss seems a bit ad hoc: where does it come from? How should we decide what value of $c$ to use? Is there any good motivation for using a linear loss on outliers, or should we simply remove them instead? How might this choice affect our resulting model?
A Bayesian Approach to Outliers: Nuisance Parameters
The Bayesian approach to accounting for outliers generally involves modifying the model so that the outliers are accounted for. For this data, it is abundantly clear that a simple straight line is not a good fit to our data. So let's propose a more complicated model that has the flexibility to account for outliers.
One option is to choose a mixture between a signal and a background:
$$
\begin{array}{ll}
p({x_i}, {y_i},{e_i}~|~\theta,{g_i},\sigma,\sigma_b) = & \frac{g_i}{\sqrt{2\pi e_i^2}}\exp\left[\frac{-\left(\hat{y}(x_i~|~\theta) - y_i\right)^2}{2e_i^2}\right] \
&+ \frac{1 - g_i}{\sqrt{2\pi \sigma_B^2}}\exp\left[\frac{-\left(\hat{y}(x_i~|~\theta) - y_i\right)^2}{2\sigma_B^2}\right]
\end{array}
$$
What we've done is expanded our model with some nuisance parameters: ${g_i}$ is a series of weights which range from 0 to 1 and encode for each point $i$ the degree to which it fits the model. $g_i=0$ indicates an outlier, in which case a Gaussian of width $\sigma_B$ is used in the computation of the likelihood. This $\sigma_B$ can also be a nuisance parameter, or its value can be set at a sufficiently high number, say 50.
Our model is much more complicated now: it has 22 parameters rather than 2, but the majority of these can be considered nuisance parameters, which can be marginalized-out in the end, just as we marginalized (integrated) over $p$ in the Billiard example.  Let's construct a function which implements this...
Can you write Python code for it?

# theta will be an array of length 2 + N, where N is the number of points
# theta[0] is the intercept, theta[1] is the slope,
# and theta[2 + i] is the weight g_i

def log_prior(theta):
    #g_i needs to be between 0 and 1
    if (all(theta[2:] > 0) and all(theta[2:] < 1)):
        return 0
    else:
        return -np.inf  # recall log(0) = -inf

def log_likelihood(theta, x, y, e, sigma_B):
    dy = y - theta[0] - theta[1] * x
    g = np.clip(theta[2:], 0, 1)  # g<0 or g>1 leads to NaNs in logarithm
    logL1 = np.log(g) - 0.5 * np.log(2 * np.pi * e ** 2) - 0.5 * (dy / e) ** 2
    logL2 = np.log(1 - g) - 0.5 * np.log(2 * np.pi * sigma_B ** 2) - 0.5 * (dy / sigma_B) ** 2
    return np.sum(np.logaddexp(logL1, logL2))

def log_posterior(theta, x, y, e, sigma_B):
    return log_prior(theta) + log_likelihood(theta, x, y, e, sigma_B)


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
In other words, we'd give Bob the following odds of winning:
Can you write Python code for it?

print("Odds against Bob winning: {0:.0f} to 1".format((1. - freq_prob) / freq_prob))


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
It's clear on examination that the outliers are exerting a disproportionate influence on the fit. This is due to the nature of the squared loss function. If you have a single outlier that is, say 10 standard deviations away from the fit, its contribution to the loss will out-weigh that of 25 points which are 2 standard deviations away!
Clearly the squared loss is overly sensitive to outliers, and this is causing issues with our fit. One way to address this within the frequentist paradigm is to simply adjust the loss function to be more robust.
Frequentist Correction for Outliers: Huber Loss
The variety of possible loss functions is quite literally infinite, but one relatively well-motivated option is the Huber loss. The Huber loss defines a critical value at which the loss curve transitions from quadratic to linear. Let's create a plot which compares the Huber loss to the standard squared loss for several critical values $c$:
Can you write Python code for it?

t = np.linspace(-20, 20)

def huber_loss(t, c=3):
    return ((abs(t) < c) * 0.5 * t ** 2
            + (abs(t) >= c) * -c * (0.5 * c - abs(t)))

plt.plot(t, 0.5 * t ** 2, label="squared loss", lw=2)
for c in (10, 5, 3):
    plt.plot(t, huber_loss(t, c), label="Huber loss, c={0}".format(c), lw=2)
plt.ylabel('loss')
plt.xlabel('standard deviations')
plt.legend(loc='best', frameon=False);


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
Now we'll run the MCMC samples to explore the parameter space:
Can you write Python code for it?

# Note that this step will take a few minutes to run!

ndim = 2 + len(x)  # number of parameters in the model
nwalkers = 50  # number of MCMC walkers
nburn = 10000  # "burn-in" period to let chains stabilize
nsteps = 15000  # number of MCMC steps to take

# set theta near the maximum likelihood, with
np.random.seed(10)
starting_guesses = np.zeros((nwalkers, ndim))
starting_guesses[:, :2] = np.random.normal(theta1, 1, (nwalkers, 2))
starting_guesses[:, 2:] = np.random.normal(0.5, 0.1, (nwalkers, ndim - 2))

import emcee
sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior, args=[x, y, e, 50])
sampler.run_mcmc(starting_guesses, nsteps, rstate0=np.random.get_state())

sample = sampler.chain  # shape = (nwalkers, nsteps, ndim)
sample = sampler.chain[:, nburn:, :].reshape(-1, ndim)


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
Once we have these samples, we can exploit a very nice property of the Markov chains. Because their distribution models the posterior, we can integrate out (i.e. marginalize) over nuisance parameters simply by ignoring them!
We can look at the (marginalized) distribution of slopes and intercepts by examining the first two columns of the sample:
Can you write Python code for it?

plt.plot(sample[:, 0], sample[:, 1], ',k', alpha=0.1)
plt.xlabel('intercept')
plt.ylabel('slope');


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
So we see that the Bayesian result gives us 10 to 1 odds, which is quite different than the 18 to 1 odds found using the frequentist approach. So which one is correct?
A Brute Force/Monte Carlo Approach
For this type of well-defined and simple setup, it is actually relatively easy to use a Monte Carlo simulation to determine the correct answer. This is essentially a brute-force tabulation of possible outcomes: we generate a large number of random games, and simply count the fraction of relevant games that Bob goes on to win. The current problem is especially simple because so many of the random variables involved are uniformly distributed.
We can use the numpy package to do this as follows:
Can you write Python code for it?

import numpy as np
np.random.seed(0)

# play 100000 games with randomly-drawn p, between 0 and 1
p = np.random.random(100000)

# each game needs at most 11 rolls for one player to reach 6 wins
rolls = np.random.random((11, len(p)))

# count the cumulative wins for Alice and Bob at each roll
Alice_count = np.cumsum(rolls < p, 0)
Bob_count = np.cumsum(rolls >= p, 0)

# sanity check: total number of wins should equal number of rolls
total_wins = Alice_count + Bob_count
assert np.all(total_wins.T == np.arange(1, 12))
print("(Sanity check passed)")

# determine number of games which meet our criterion of (A wins, B wins)=(5, 3)
# this means Bob's win count at eight rolls must equal 3
good_games = Bob_count[7] == 3
print("Number of suitable games: {0}".format(good_games.sum()))

# truncate our results to consider only these games
Alice_count = Alice_count[:, good_games]
Bob_count = Bob_count[:, good_games]

# determine which of these games Bob won.
# to win, he must reach six wins after 11 rolls.
bob_won = np.sum(Bob_count[10] == 6)
print("Number of these games Bob won: {0}".format(bob_won.sum()))

# compute the probability
mc_prob = bob_won.sum() * 1. / good_games.sum()
print("Monte Carlo Probability of Bob winning: {0:.2f}".format(mc_prob))
print("MC Odds against Bob winning: {0:.0f} to 1".format((1. - mc_prob) / mc_prob))


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
There is not an extremely strong constraint on either of these, but we do see that $(g_1, g_2) = (1, 0)$ is slightly favored: the means of $g_1$ and $g_2$ are greater than and less than 0.5, respecively. If we choose a cutoff at $g=0.5$, our algorithm has identified $g_2$ as an outlier.
Let's make use of all this information, and plot the marginalized best model over the original data. As a bonus, we'll draw red circles to indicate which points the model detects as outliers:
Can you write Python code for it?

theta3 = np.mean(sample[:, :2], 0)
g = np.mean(sample[:, 2:], 0)
outliers = (g < 0.5)

plt.errorbar(x, y, e, fmt='.k', ecolor='gray')
plt.plot(xfit, theta1[0] + theta1[1] * xfit, color='lightgray')
plt.plot(xfit, theta2[0] + theta2[1] * xfit, color='lightgray')
plt.plot(xfit, theta3[0] + theta3[1] * xfit, color='black')
plt.plot(x[outliers], y[outliers], 'ro', ms=20, mfc='none', mec='red')
plt.title('Maximum Likelihood fit: Bayesian Marginalization');


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
We see a distribution of points near a slope of $\sim 0.45$, and an intercept of $\sim 31$. We'll plot this model over the data below, but first let's see what other information we can extract from this trace.
One nice feature of analyzing MCMC samples is that the choice of nuisance parameters is completely symmetric: just as we can treat the ${g_i}$ as nuisance parameters, we can also treat the slope and intercept as nuisance parameters! Let's do this, and check the posterior for $g_1$ and $g_2$, the outlier flag for the first two points:
Can you write Python code for it?

plt.plot(sample[:, 2], sample[:, 3], ',k', alpha=0.1)
plt.xlabel('$g_1$')
plt.ylabel('$g_2$')

print("g1 mean: {0:.2f}".format(sample[:, 2].mean()))
print("g2 mean: {0:.2f}".format(sample[:, 3].mean()))


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
We'll visualize this data below:
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

plt.errorbar(x, y, e, fmt='.k', ecolor='gray');


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
As a sanity check, let's make sure that the exact and approximate confidence intervals match for a large number of points:
Can you write Python code for it?

np.random.seed(0)
Dlarge = 10 + np.random.random(500)
print "approx: ({0:.3f}, {1:.3f})".format(*approx_CI(Dlarge))
print "exact: ({0:.3f}, {1:.3f})".format(*exact_CI(Dlarge))


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
As is promised by frequentism, 95% of the computed confidence intervals contain the true value. The procedure we used to compute the confidence intervals is, in fact, correct: our data just happened to be among the 5% where the method breaks down. But here's the thing: we know from the data themselves that we are in the 5% where the CI fails. The fact that the standard frequentist confidence interval ignores this common-sense information should give you pause about blind reliance on the confidence interval for any nontrivial problem.
For good measure, let's check that the Bayesian credible region also passes its test:
Can you write Python code for it?

np.random.seed(42)
N = int(1E7)
eps = 0.1

theta = 9 + 2 * np.random.random(N)
data = (theta + expon().rvs((3, N))).T
data.sort(1)
D.sort()
i_good = np.all(abs(data - D) < eps, 1)

print("Number of good samples: {0}".format(i_good.sum()))

theta_good = theta[i_good]
theta1, theta2 = bayes_CR(D)

within_CR = (theta1 < theta_good) & (theta_good < theta2)
print("Fraction of thetas in Credible Region: {0:.3f}".format(within_CR.sum() * 1. / within_CR.size))


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
Imagine now that we've observed some data, $D = {10, 12, 15}$, and we want to infer the value of $\theta$ from this data. We'll explore four approaches to this below.
1. Common Sense Approach
One general tip that I'd always recommend: in any problem, before computing anything, think about what you're computing and guess what a reasonable solution might be. We'll start with that here. Thinking about the problem, the hard cutoff in the probability distribution leads to one simple observation: $\theta$ must be smaller than the smallest observed value.
This is immediately obvious on examination: the probability of seeing a value less than $\theta$ is zero.  Thus, a model with $\theta$ greater than any observed value is impossible, assuming our model specification is correct. Our fundamental assumption in both Bayesianism and frequentism is that the model is correct, so in this case, we can immediately write our common sense condition:
$$
\theta < \min(D)
$$
or, in the particular case of $D = {10, 12, 15}$,
$$
\theta < 10
$$
Any reasonable constraint on $\theta$ given this data should meet this criterion. With this in mind, let's go on to some quantitative approaches based on Frequentism and Bayesianism.
2. Frequentist approach #1: Sampling Distribution via the Normal Approximation
In the frequentist paradigm, we'd like to compute a confidence interval on the value of $\theta$. We can start by observing that the population mean is given by
$$
\begin{array}{ll}
E(x) &= \int_0^\infty xp(x)dx\
     &= \theta + 1
     \end{array}
$$
So, using the sample mean as the point estimate of $E(x)$, we have an unbiased estimator for $\theta$ given by
$$
\hat{\theta} = \frac{1}{N} \sum_{i=1}^N x_i - 1
$$
The exponential distribution has a standard deviation of 1, so in the limit of large $N$, we can use the standard error of the mean (as above) to show that the sampling distribution of $\hat{\theta}$ will approach normal with variance $\sigma^2 = 1 / N$. Given this, we can write our 95% (i.e. 2$\sigma$) confidence interval...
Can you write Python code for it?

from scipy.special import erfinv

def approx_CI(D, sig=0.95):
    """Approximate truncated exponential confidence interval"""
    # use erfinv to convert percentage to number of sigma
    Nsigma = np.sqrt(2) * erfinv(sig)
    D = np.asarray(D)
    N = D.size
    theta_hat = np.mean(D) - 1
    return [theta_hat - Nsigma / np.sqrt(N),
            theta_hat + Nsigma / np.sqrt(N)]

D = [10, 12, 15]
print("approximate CI: ({0:.1f}, {1:.1f})".format(*approx_CI(D)))


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
Next let's create a function which will compute the confidence interval:
Can you write Python code for it?

from scipy.special import erfinv

def freq_CI_mu(D, sigma, frac=0.95):
    """Compute the confidence interval on the mean"""
    # we'll compute Nsigma from the desired percentage
    Nsigma = np.sqrt(2) * erfinv(frac)
    mu = D.mean()
    sigma_mu = sigma * D.size ** -0.5
    return mu - Nsigma * sigma_mu, mu + Nsigma * sigma_mu

print("95% Confidence Interval: [{0:.0f}, {1:.0f}]".format(*freq_CI_mu(D, 10)))


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
Note here that we've assumed $\sigma_x$ is a known quantity; this could also be estimated from the data along with $\mu$, but here we kept things simple for sake of example.
2. The Bayesian Approach
For the Bayesian approach, we start with Bayes' theorem:
$$
P(\mu~|~D) = \frac{P(D~|~\mu)P(\mu)}{P(D)}
$$
We'll use a flat prior on $\mu$ (i.e. $P(\mu) \propto 1$ over the region of interest) and use the likelihood
$$
P(D~|~\mu) = \prod_{i=1}^N \frac{1}{\sqrt{2\pi\sigma_x^2}}\exp\left[\frac{(\mu - x_i)^2}{2\sigma_x^2}\right]
$$
Computing this product and manipulating the terms, it's straightforward to show that this gives
$$
P(\mu~|~D) \propto \exp\left[\frac{-(\mu - \bar{x})^2}{2\sigma_\mu^2}\right]
$$
which is recognizable as a normal distribution with mean $\bar{x}$ and standard deviation $\sigma_\mu$.
That is, the Bayesian posterior on $\mu$ in this case is exactly equal to the frequentist sampling distribution for $\mu$.
From this posterior, we can compute the Bayesian credible region, which is the shortest interval that contains 95% of the probability. Here, it looks exactly like the frequentist confidence interval:
$$
CR_{\mu} =  \left(\bar{x} - 2\sigma_\mu,~\bar{x} + 2\sigma_\mu\right)
$$
For completeness, we'll also create a function to compute the Bayesian credible region:
Can you write Python code for it?

def bayes_CR_mu(D, sigma, frac=0.95):
    """Compute the credible region on the mean"""
    Nsigma = np.sqrt(2) * erfinv(frac)
    mu = D.mean()
    sigma_mu = sigma * D.size ** -0.5
    return mu - Nsigma * sigma_mu, mu + Nsigma * sigma_mu

print("95% Credible Region: [{0:.0f}, {1:.0f}]".format(*bayes_CR_mu(D, 10)))


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
Now that we have this Bayesian method, we can compare the results of the four methods:
Can you write Python code for it?

print("common sense:         theta < {0:.1f}".format(np.min(D)))
print("frequentism (approx): 95% CI = ({0:.1f}, {1:.1f})".format(*approx_CI(D)))
print("frequentism (exact):  95% CI = ({0:.1f}, {1:.1f})".format(*exact_CI(D)))
print("Bayesian:             95% CR = ({0:.1f}, {1:.1f})".format(*bayes_CR(D)))


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
So What's the Difference?
The above derivation is one reason why the frequentist confidence interval and the Bayesian credible region are so often confused. In many simple problems, they correspond exactly. But we must be clear that even though the two are numerically equivalent, their interpretation is very different.
Recall that in Bayesianism, the probability distributions reflect our degree of belief. So when we computed the credible region above, it's equivalent to saying

"Given our observed data, there is a 95% probability that the true value of $\mu$ falls within $CR_\mu$" - Bayesians

In frequentism, on the other hand, $\mu$ is considered a fixed value and the data (and all quantities derived from the data, including the bounds of the confidence interval) are random variables. So the frequentist confidence interval is equivalent to saying

"There is a 95% probability that when I compute $CI_\mu$ from data of this sort, the true mean will fall within $CI_\mu$." - Frequentists

Note the difference: the Bayesian solution is a statement of probability about the parameter value given fixed bounds. The frequentist solution is a probability about the bounds given a fixed parameter value. This follows directly from the philosophical definitions of probability that the two approaches are based on.
The difference is subtle, but, as I'll discuss below, it has drastic consequences. First, let's further clarify these notions by running some simulations to confirm the interpretation.
Confirming the Bayesian Credible Region
To confirm what the Bayesian credible region is claiming, we must do the following:

sample random $\mu$ values from the prior
sample random sets of points given each $\mu$
select the sets of points which match our observed data
ask what fraction of these $\mu$ values are within the credible region we've constructed.

In code, that looks like this:
Can you write Python code for it?

# first define some quantities that we need
Nsamples = int(2E7)
N = len(D)
sigma_x = 10

# if someone changes N, this could easily cause a memory error
if N * Nsamples > 1E8:
    raise ValueError("Are you sure you want this many samples?")

# eps tells us how close to D we need to be to consider
# it a matching sample. The value encodes the tradeoff
# between bias and variance of our simulation
eps = 0.5

# Generate some mean values from the (flat) prior in a reasonable range
np.random.seed(0)
mu = 80 + 40 * np.random.random(Nsamples)

# Generate data for each of these mean values
x = np.random.normal(mu, sigma_x, (N, Nsamples)).T

# find data which matches matches our "observed" data
x.sort(1)
D.sort()
i = np.all(abs(x - D) < eps, 1)
print("number of suitable samples: {0}".format(i.sum()))

# Now we ask how many of these mu values fall in our credible region
mu_good = mu[i]
CR = bayes_CR_mu(D, 10)
within_CR = (CR[0] < mu_good) & (mu_good < CR[1])
print "Fraction of means in Credible Region: {0:.3f}".format(within_CR.sum() * 1. / within_CR.size)


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
The exact confidence interval is slightly different than the approximate one, but still reflects the same problem: we know from common-sense reasoning that $\theta$ can't be greater than 10, yet the 95% confidence interval is entirely in this forbidden region! The confidence interval seems to be giving us unreliable results.
We'll discuss this in more depth further below, but first let's see if Bayes can do better.
4. Bayesian Credibility Interval
For the Bayesian solution, we start by writing Bayes' rule:
$$
p(\theta~|~D) = \frac{p(D~|~\theta)p(\theta)}{P(D)}
$$
Using a constant prior $p(\theta)$, and with the likelihood
$$
p(D~|~\theta) = \prod_{i=1}^N p(x~|~\theta)
$$
we find
$$
p(\theta~|~D) \propto \left{
\begin{array}{lll}
N\exp\left[N(\theta - \min(D))\right] &,& \theta < \min(D)\
0                &,& \theta > \min(D)
\end{array}
\right}
$$
where $\min(D)$ is the smallest value in the data $D$, which enters because of the truncation of $p(x~|~\theta)$.
Because $p(\theta~|~D)$ increases exponentially up to the cutoff, the shortest 95% credibility interval $(\theta_1, \theta_2)$ will be given by
$$
\theta_2 = \min(D)
$$
and $\theta_1$ given by the solution to the equation
$$
\int_{\theta_1}^{\theta_2} N\exp[N(\theta - \theta_2)]d\theta = f
$$
this can be solved analytically by evaluating the integral, which gives
$$
\theta_1 = \theta_2 + \frac{\log(1 - f)}{N}
$$
Let's write a function which computes this:
Can you write Python code for it?

def bayes_CR(D, frac=0.95):
    """Bayesian Credibility Region"""
    D = np.asarray(D)
    N = float(D.size)
    theta2 = D.min()
    theta1 = theta2 + np.log(1. - frac) / N
    return theta1, theta2


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
We see that, as predicted, 95% of the confidence intervals contain the true value of $\mu$.
Again, the important thing to note here is which of the variables is random. We use a single value of $\mu$, and consider it in relation to multiple confidence intervals constructed from multiple random data samples.
Discussion
We should remind ourselves again of the difference between the two types of constraints:

The Bayesian approach fixes the credible region, and guarantees 95% of possible values of $\mu$ will fall within it.
The frequentist approach fixes the parameter, and guarantees that 95% of possible confidence intervals will contain it.

Comparing the frequentist confirmation and the Bayesian confirmation above, we see that the distinctions which stem from the very definition of probability mentioned above:

Bayesianism treats parameters (e.g. $\mu$) as random variables, while frequentism treats parameters as fixed.
Bayesianism treats observed data (e.g. $D$) as fixed, while frequentism treats data as random variables.
Bayesianism treats its parameter constraints (e.g. $CR_\mu$) as fixed, while frequentism treats its constraints (e.g. $CI_\mu$) as random variables.

In the above example, as in many simple problems, the confidence interval and the credibility region overlap exactly, so the distinction is not especially important. But scientific analysis is rarely this simple; next we'll consider an example in which the choice of approach makes a big difference.
Example 2: Jaynes' Truncated Exponential
For an example of a situation in which the frequentist confidence interval and the Bayesian credibility region do not overlap, I'm going to turn to an example given by E.T. Jaynes, a 20th century physicist who wrote extensively on statistical inference in Physics.  In the fifth example of his Confidence Intervals vs. Bayesian Intervals (pdf), he considers a truncated exponential model. Here is the problem, in his words:

A device will operate without failure for a time $\theta$ because of a protective chemical...
Can you write Python code for it?

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

def p(x, theta):
    return (x > theta) * np.exp(theta - x)

x = np.linspace(5, 18, 1000)
plt.fill(x, p(x, 10), alpha=0.3)
plt.ylim(0, 1.2)
plt.xlabel('x')
plt.ylabel('p(x)');


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
We see that, as predicted, roughly 95% of $\mu$ values with data matching ours lie in the Credible Region.
The important thing to note here is which of the variables is random, and which are fixed. In the Bayesian approach, we compute a single credible region from our observed data, and we consider it in terms of multiple random draws of $\mu$.
Confirming the frequentist Confidence Interval
Confirmation of the interpretation of the frequentist confidence interval is a bit less involved. We do the following:

draw sets of values from the distribution defined by the single true value of $\mu$.
for each set of values, compute a new confidence interval.
determine what fraction of these confidence intervals contain $\mu$.

In code, it looks like this:
Can you write Python code for it?

# define some quantities we need
N = len(D)
Nsamples = int(1E4)
mu = 100
sigma_x = 10

# Draw datasets from the true distribution
np.random.seed(0)
x = np.random.normal(mu, sigma_x, (Nsamples, N))

# Compute a confidence interval from each dataset
CIs = np.array([freq_CI_mu(Di, sigma_x) for Di in x])

# find which confidence intervals contain the mean
contains_mu = (CIs[:, 0] < mu) & (mu < CIs[:, 1])
print "Fraction of Confidence Intervals containing the mean: {0:.3f}".format(contains_mu.sum() * 1. / contains_mu.size)


I am working on the file "Session4/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
What we find is that the Bayesian result agrees with our common sense, while the frequentist approach does not. The problem is that frequentism is answering the wrong question.
Numerical Confirmation
To try to quell any doubts about the math here, I want to repeat the exercise we did above and show that the confidence interval derived above is, in fact, correct.  We'll use the same approach as before, assuming a "true" value for $\theta$ and sampling data from the associated distribution:
Can you write Python code for it?

from scipy.stats import expon

Nsamples = 1000
N = 3
theta = 10

np.random.seed(42)
data = expon(theta).rvs((Nsamples, N))
CIs = np.array([exact_CI(Di) for Di in data])

# find which confidence intervals contain the mean
contains_theta = (CIs[:, 0] < theta) & (theta < CIs[:, 1])
print "Fraction of Confidence Intervals containing theta: {0:.3f}".format(contains_theta.sum() * 1. / contains_theta.size)


I am working on the file "Session4/code/00 Recap - Decision Tree Classifier.ipynb".
The first task is:
Recap - Decision Tree Classifier
Can you write Python code for it?

lfeat = iris.feature_names
df_iris = pd.DataFrame(iris.data, columns = lfeat)
model = DecisionTreeClassifier()
data = df_iris[lfeat].values
df_iris["Species"] = iris.target
target = df_iris["Species"].values
model.fit(data, target)
expected = target
predicted = model.predict(data)
print(metrics.classification_report(expected, predicted))
print(metrics.confusion_matrix(expected, predicted))


I am working on the file "Session4/code/01 Loading EPOS Category Data for modelling.ipynb".
The first task is:
We fail here because the description column is a string.
Lets try again without the description.
Can you write Python code for it?

# features
feature_names_integers = ['Barcode','UnitRRP']

# Extra features from panda (without description)
training_data_integers = df_training[feature_names_integers].values
training_data_integers[:3]

# train model again
model_dtc.fit(training_data_integers, target)

# Extract test data and test the model
test_data_integers = df_test[feature_names_integers].values
test_target = df_test["CategoryID"].values
expected = test_target
predicted_dtc = model_dtc.predict(test_data_integers)

print(metrics.classification_report(expected, predicted_dtc,    target_names=target_categories))

print(metrics.confusion_matrix(expected, predicted_dtc))

metrics.accuracy_score(expected, predicted, normalize=True, sample_weight=None)

predicted[:5]


I am working on the file "Session4/code/02 Decision Tree Classifier - random_state.ipynb".
The first task is:
We see that the results are not the same. This is because the Decision Tree Classifier chooses a feature at random in order to try to avoid overfitting. As we are about to start trying to improve the results by trying different strategies of preparing and loading data having varying will be unhelpful.
To aviod this we can manually set the random_state.
Can you write Python code for it?

model_dtc = DecisionTreeClassifier(random_state=511)
model_dtc.fit(training_data_integers, target)
predicted_dtc = model_dtc.predict(test_data_integers)
metrics.accuracy_score(expected, predicted_dtc, normalize=True, sample_weight=None)

model_dtc = DecisionTreeClassifier(random_state=511)
model_dtc.fit(training_data_integers, target)
predicted_dtc = model_dtc.predict(test_data_integers)
metrics.accuracy_score(expected, predicted_dtc, normalize=True, sample_weight=None)


I am working on the file "Session4/code/02 Decision Tree Classifier - random_state.ipynb".
The first task is:
and again.
Can you write Python code for it?

model_dtc = DecisionTreeClassifier()
model_dtc.fit(training_data_integers, target)
predicted_dtc = model_dtc.predict(test_data_integers)
metrics.accuracy_score(expected, predicted_dtc, normalize=True, sample_weight=None)


I am working on the file "Session4/code/02 Decision Tree Classifier - random_state.ipynb".
The first task is:
one more time :)
Can you write Python code for it?


I am working on the file "Session5/1_MachineLearning_Regression_keys.ipynb".
The first task is:
Homework
Can you write Python code for it?

# Write code to create a 3D scatter plot for "Area", "Bedroom" and actual "Price"
# Then add visualization of "Area" and "Bedroom" against the predicted price.
from mpl_toolkits.mplot3d import Axes3D

y_pred = lin_reg.predict(X)
fig_scatter = plt.figure()
ax = fig_scatter.add_subplot(111, projection='3d')
ax.scatter(housing["Area"], housing["Bedrooms"], housing["Price"])
ax.scatter(housing["Area"], housing["Bedrooms"], y_pred)

ax.set_xlabel("Area")
ax.set_ylabel("Bedrooms")
ax.set_zlabel("Price")


I am working on the file "Session5/1_MachineLearning_Regression_keys.ipynb".
The first task is:
LinearRegression
Train a LinearRegression model.
1. Create an instance of LinearRegression of sklearn
1. Use training set X_train and y_train to fit.
2. Use test set X_test and y_test to measure $R^2$ using score method.
Can you write Python code for it?

# Write code to train a Linear Regression model and to test its performance on the test set
lin_reg = LinearRegression()
lin_reg.fit(X_train, y_train)
lin_reg_R2 = lin_reg.score(X_test, y_test)

print("Linear Regression R2 = ", lin_reg_R2)


I am working on the file "Session5/1_MachineLearning_Regression_keys.ipynb".
The first task is:
Question: Why such a huge MSE?
Use LinearRegression.score() method to get coefficient of determination $R^2$ of the prediction. It is calculated based on MSE. The best possible score is $1.0$ and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of $y$, disregarding the input features, would get a $R^2$ score of $0.0$.
Warning: You need to pass input matrix X and original output y to score() method. This method first performs testing with the trained model and then calculates the $R^2$ score.
Can you write Python code for it?

# Write code to get coefficient of determination using "score()"
# Hint: you have to pass both X and original y to score()
R2 = lin_reg.score(X, y)
print("R2 = ", R2)


I am working on the file "Session5/1_MachineLearning_Regression_keys.ipynb".
The first task is:
Random Forest Regressor
Use RandomForestRegressor from sklearn library.
Can you write Python code for it?

# Write code to import necessary module for RandomForestRegressor
from sklearn.ensemble import RandomForestRegressor

# Write code to train a RandomForestRegressor model and to test its performance on the test set
rfr = RandomForestRegressor()
rfr.fit(X_train, y_train)
rfr_R2 = rfr.score(X_test, y_test)

print("Random Forest Regressor R2 = ", rfr_R2)


I am working on the file "Session5/1_MachineLearning_Regression_keys.ipynb".
The first task is:
Regression Models on Energy Efficiency Dataset
In this section, we will train various regression models on the Energy Efficiency Dataset. We will measure the performances in terms of $R^2$ and compare their performances.
Inputs and Outputs
First, identify the inputs and outputs, and convert them to numpy matrices.
Can you write Python code for it?

# Write code to convert desired dataframe columns into numpy arrays
X = energy.as_matrix(columns=["Compactness","Surface","Wall", "Roof", "Heiht",
                            "Orientation","Glazing","GlazingDist"])
y = energy.as_matrix(columns=["Heating"])


I am working on the file "Session5/1_MachineLearning_Regression_keys.ipynb".
The first task is:
Supplementary Material
This portion of the notebook is for your own practice. Do not hesitate to contact me for any question.
Energy Efficiency Dataset
This dataset poses the regression problem of predicting heating load of different buildings. More details of this dataset can be found in this link. The buildings are described using following 8 features:
1. Relative Compactness
2. Surface Area
3. Wall Area
4. Roof Area
5. Overall Height
6. Orientation
7. Glazing Area
8. Glazing Area Distribution
Import the data
The data is provided in the file energy_efficiency.csv. Like before, use read_csv() module from pandas to import the data.
Can you write Python code for it?

# Write code to import the data using pandas
# Hint: note that comma (",") is the field separator, and we have no "header"
energy = pd.read_csv("energy_efficiency.csv", sep=",", header=None)

# We name the columns based on above features
energy.columns = ["Compactness","Surface","Wall", "Roof", "Heiht",
              "Orientation","Glazing","GlazingDist", "Heating"]

# We sneak peek into the data
# Hint: use dataframe "head" method with "n" parameter
energy.head(n=5)


I am working on the file "Session5/1_MachineLearning_Regression_keys.ipynb".
The first task is:
Training a Multivariate Linear Regression Model
You will now train a Linear Regression model using both "Area" and "Bedrooms" features to predict "Price".
Question: What is your input X here? What is your output y here?
Note that only your input changes. Nothing else changes in your implmentation.
Can you write Python code for it?

# Write code to convert desired dataframe columns into numpy arrays
X = housing.as_matrix(columns=["Area","Bedrooms"])
y = housing.as_matrix(columns=["Price"])

# Write code to train a LinearRegression model
lin_reg = LinearRegression()
lin_reg.fit(X, y)

# Write code to calculate and print the MSE
R2 = lin_reg.score(X, y)
print("R2 = ", R2)


I am working on the file "Session5/1_MachineLearning_Regression_keys.ipynb".
The first task is:
Use your trained LinearRegression model to predict on the same dataset (i.e. "Area" features stored as numpy array X). Also calculate Mean Squared Error using the mean_squared_error() method from sklearn library.
Can you write Python code for it?

# Write code to predict prices using the trained LinearRegression
y_predicted = lin_reg.predict(X)

# Importing modules to calculate MSE
from sklearn.metrics import mean_squared_error

# Write code to calculate and print the MSE on the predicted values.
# Hint 1: use "mean_squared_error()" method
# Hint 2: you have to pass both original y and predicted y to compute the MSE.
mse = mean_squared_error(y, y_predicted)
print("MSE = ", mse)


I am working on the file "Session5/1_MachineLearning_Regression_keys.ipynb".
The first task is:
Using the following step, train a Linear Regression model on housing price dataset:
1. import LinearRegression module from sklearn
2. create an instance of LinearRegression and fit with the X and y arrays you created
Can you write Python code for it?

# Write code to learn a linear regression model on housing price dataset
from sklearn.linear_model import LinearRegression # TO DELETE

lin_reg = LinearRegression()
lin_reg.fit(X, y)


I am working on the file "Session5/2_TimeSeriesForecasting_keys.ipynb".
The first task is:
Checkpoint: Make sure you understand how shift() works.
Creating the Regression Matrix
Now you will use shift() multiple times to create a regression matrix from the temperature dataframe.
Step 1: Create a new empty DataFrame to store the regression matrix.
Can you write Python code for it?

# Write code to create an empty DataFrame
# Hint: "pandas.DataFrame()" without any arguments creates an empty DataFrame
reg_mat = pd.DataFrame()


I am working on the file "Session5/2_TimeSeriesForecasting_keys.ipynb".
The first task is:
Handling time series with Python Pandas
Python Pandas library is an efficient tool to work with time series data. Please see this link for all the detailed functionalities.
Pandas shift() method
A key method to help transform time series data into a supervised learning problem is the Pandas DataFrame.shift() method. Given a DataFrame, the shift() method can be used to create copies of columns that are pushed forward or pulled back.
This is the behavior required to create columns of lag observations as well as columns of forecast observations for a time series dataset in a supervised learning format.
Notes:
1. shift() takes a positive integer $k$ to push forward the data $k$ steps (rows of NaN values added to the front)
2. shift() takes a positive integer $-k$ to pull back the data $k$ steps (rows of NaN values added to the end)
Can you write Python code for it?

# Write code to try "shift()" method to push forward the data 1 step
# Hint: use head() at the end of youe code to see first lines
temperature.shift(1).head()

# Write code to try "shift()" method to pull back the data 1 step
# Hint: use head() at the end of youe code to see first lines
temperature.shift(-1).head()


I am working on the file "Session5/2_TimeSeriesForecasting_keys.ipynb".
The first task is:
LinearRegression
Train a LinearRegression model.
1. Create an instance of LinearRegression of sklearn.
1. Use training set X_train and y_train to fit.
2. Use test set X_test and y_test to measure $R^2$ using score method.
Can you write Python code for it?

# Write code to import LinearRegression from sklearn
from sklearn.linear_model import LinearRegression

lin_reg = LinearRegression()
lin_reg.fit(X_train, y_train)

R2 = lin_reg.score(X_test, y_test)
print("Linear Regression R2 = ", R2)


I am working on the file "Session5/2_TimeSeriesForecasting_keys.ipynb".
The first task is:
Multi target regression using RandomForestRegressor
Can you write Python code for it?

# Write code to import necessary module for RandomForestRegressor
from sklearn.ensemble import RandomForestRegressor

# Write code to train a MultiOutputRegressor model using RandomForestRegressor.
#    Test its performance on the test set
rfr = MultiOutputRegressor(RandomForestRegressor())
rfr.fit(X_train, y_train)
rfr_R2 = rfr.score(X_test, y_test)

print("Random Forest Regressor R2 = ", rfr_R2)


I am working on the file "Session5/2_TimeSeriesForecasting_keys.ipynb".
The first task is:
Multistepahead Forecasting: $T' > 1$
Let's repeat the above procedure for forecast horizon $4$, i.e. to predict values till next $4$ time stamps.
Python Scikit-Learn provides a wrapper module called MultiOutputRegressor for multi target regression. You can pass it a standard sklearn regressor instance directly, which will be used as a base.
We will use it with LinearRegression and RandomForestRegressor
Can you write Python code for it?

# Write code to generate X and y matrices.
#     This time set history to 20 and horizon to 4
history, horizon = 20, 4
X, y = regression_matrix(temperature, history, horizon)

# Write code for splitting the data into train and test sets.# TO DELETE


I am working on the file "Session5/2_TimeSeriesForecasting_keys.ipynb".
The first task is:
Our goal is to model time series forecasting as a machine learning regression task. It is time to identify the inputs and outputs
Question: What is your input X here? What is your output y here?
Can you write Python code for it?

# Write code to cerate input matrix X by selecting correct columns of reg_mat_numpy
# Hint: first k columns of a numpy matrix M can be selected by M[:,:k]
X = reg_mat_numpy[:, :history]

# Write code to cerate output matrix y by selecting correct columns of reg_mat_numpy
# Hint: last k columns of a numpy matrix M can be selected by M[:,-k:]
y = reg_mat_numpy[:, -horizon:]


I am working on the file "Session5/2_TimeSeriesForecasting_keys.ipynb".
The first task is:
Step 2: Assume the history $T = 5$. So, you have to use shift() five times. Each such shift will generate a new column of the regression matrix.
Note that you have to maintain the order as shown in the above equations. If we assume that "t" is the current time, the history columns of the regression matrix should be in the following order:
$$t-4, t-3, t-2, t-1, t$$
To get the "t-i" column, the time series needs to get pushed forward i steps. For each shift, you should store the newly generated column in the reg_mat dataframe with column name "t-i"
Can you write Python code for it?

# Write code to generate columns t-4, t-3, t-2, t-1, t IN THIS ORDER for reg_mat
# Hint: you do not need any shift to store the column "t"
reg_mat["t-4"] = temperature.shift(4)
reg_mat["t-3"] = temperature.shift(3)
reg_mat["t-2"] = temperature.shift(2)
reg_mat["t-1"] = temperature.shift(1)
reg_mat["t"] = temperature

# To check if all is good
reg_mat.head(10)


I am working on the file "Session5/2_TimeSeriesForecasting_keys.ipynb".
The first task is:
To numpy matrices
All machine learning algorithm implementations work efficiently with numpy matrices and arrays. The current format of our data is panda dataframes. Fortunately, pandas provides DataFrame.as_matrix() method to convert dataframes to numpy arrays.
Can you write Python code for it?

# Write code to convert entire reg_mat into a numpy matrix
reg_mat_numpy = reg_mat.as_matrix()


I am working on the file "Session5/2_TimeSeriesForecasting_keys.ipynb".
The first task is:
Training Set and Test Set
A portion of the dataset is to be set aside to be used only for testing. Fortunately sklearn provides a train_test_split() module to do that. You can specify a ratio for test_size parameter.
In this exercise, we will retain $20\%$ data for testing.
Can you write Python code for it?

# Importing the module
from sklearn.model_selection import train_test_split

# Write code for splitting the data into train and test sets.
# Hint: use "train_test_split" on X and y, and test size should be 0.2 (20%)
X_train,X_test,y_train,y_test = train_test_split(X, y, test_size=0.2)


I am working on the file "Session5/2_TimeSeriesForecasting_keys.ipynb".
The first task is:
Univariate Time Series Forecasting with Linear Regression
Finally, we have all the modules necessary to train a regression model.
Forecast horizon is to be fixed in the beginning based on the requirement. We will first try to predict only 1 step in the future, that is we set horizon = 1. We will set history=20.
Can you write Python code for it?

# Write code to generate X and y matrices from dataframe temperature
#     Set history to 20 and horizon to 1
# Hint: use the method "regression_matrix" you just wrote.
history, horizon = 20, 1
X, y = regression_matrix(temperature, history, horizon)


I am working on the file "Session5/2_TimeSeriesForecasting_keys.ipynb".
The first task is:
Univariate time series forecasting
For the sake of simplicity, we will use only MaxTemp variable to perform time series forecasting.
Also, we are not interested in the exact time here, but the sequence itself. We ignore the "Date", still the indices of the dataframe preserves the order. So, the index values will be considered as time stamps. This will satisfy our requirement and simplify the problem.
For this exercise create a new dataframe from the "MaxTemp" column.
Can you write Python code for it?

# Write code to create a new dataframe from weather
# Hint: call "pandas.DataFrame" and pass the desired column of weather
temperature = pd.DataFrame(weather["MaxTemp"])

# To check if all is good
temperature.head()


I am working on the file "Session5/2_TimeSeriesForecasting_keys.ipynb".
The first task is:
Visualization
A time series is always dependent on time. So, implicitly here we have two variables, time and the "MaxTemp". You can use DataFrame.plot.line() to create a line plot of the time series. Indices will serve as time stamps.
Can you write Python code for it?

# Write code to generate a line plot of temperature
# Hint: use "DataFrame.plot.line()" on our dataframe temperature
temperature.plot.line()


I am working on the file "Session5/code/09 Pandas - Part 1.ipynb".
The first task is:
We can use standard deviation to normalize data.
Here we generate a random set of data that creates a dataset that follows a Standard deviation from the mean.
https://en.wikipedia.org/wiki/Standard_deviation
Can you write Python code for it?

from numpy.random import normal
gaussian_numbers = normal(size=5000)
plt.hist(gaussian_numbers, bins=np.linspace(-5.0, 5.0, num=20)) # Set bin bounds
plt.show()


I am working on the file "Session5/code/09 Pandas - Part 1.ipynb".
The first task is:
Working with Data
Can you write Python code for it?

# Let's load data from a csv
df = pd.read_csv("../data/diabetes.csv")
df.info()

# Examine data
df.head()


I am working on the file "Session5/code/09 Pandas - Part 1.ipynb".
The first task is:
with numpy array we can do simple vectorized operations
so if i do
arr = arr - c
it subtracts c to all elements in arr if i do
arr = arr/c
it divides all elements in arr by c
Can you write Python code for it?

mean_age = np.mean(arr_age)
std_age = np.std(arr_age)
print ('Age Mean: {} Std:{}'.format(mean_age, std_age))

# So to compute the standardized array, I write :
arr_age_new = (arr_age - mean_age)/std_age
arr_age_new[:10]

# I can now apply the same idea to a pandas dataframe
# using some built in pandas functions :
df_new = (df - df.mean()) / df.std()
df_new.head()

df.head()

# Histogram
bins=np.linspace(-5.0, 5.0, num=20)

plt.hist(df_new["Age"].values, bins, alpha=0.5, label='age')
plt.hist(df_new["BMI"].values, bins, alpha=0.5, label='BMI')
plt.show()


I am working on the file "Session5/code/09 Pandas - Part 2.ipynb".
The first task is:
Correlations
We will now look at an interesting way to see interactions between time-series data : running correlations.
Below is the for the formula of the sample correlation coefficient $r$.

The running correlation has an extra twist to it. We select a window of points (let's say 30) on which we compute the correlation coefficient. And then we slide this window in time.
So the first value of $r$ is given by points $1, 2, 3, ... 30$.
The second value of $r$ is given by points $2, 3, 4, ... 31$.
And so on
This is where the phrase "running" comes from. This running correlation gives us information of how the correlation of two time series evolves with time.
Can you write Python code for it?

# Clear plotting
plt.clf()
plt.close()

#Let's plot the data
fig = plt.figure(figsize=(15, 15))
gs = GridSpec(2, 1, bottom=0.18, left=0.18, right=0.88)

ax7 = fig.add_subplot(gs[0])
ax30 = fig.add_subplot(gs[1])

# Get time axis
x_date = df_temp["Date"].values

# Get the y values

r_sol7 = df_temp["Ozone_interp"].rolling(window=7).corr(df_temp["Solar.R_interp"])
r_sol30 = df_temp["Ozone_interp"].rolling(window=30).corr(df_temp["Solar.R_interp"])

r_wind7 = df_temp["Ozone_interp"].rolling(window=7).corr(df_temp["Wind"])
r_wind30 = df_temp["Ozone_interp"].rolling(window=30).corr(df_temp["Wind"])

# On the first few points, no correlation can be computed (because there are not enough previous points)
# This gives a NaN value
# We arbitrarily set this to 0.
r_sol7 = r_sol7.fillna(0)
r_sol30 = r_sol30.fillna(0)
r_wind7 = r_wind7.fillna(0)
r_wind30 = r_wind30.fillna(0)

# Plot
ax7.plot(x_date, r_sol7, label = "7d corr Ozone/Solar", linewidth=1.5, color = "r")
ax7.plot(x_date, r_wind7, label = "7d corr Ozone/Wind", linewidth=1.5, color = "k")
ax30.plot(x_date, r_sol30, label = "30d corr Ozone/Solar", linewidth=1.5, color = "r")
ax30.plot(x_date, r_wind30, label = "30d corr Ozone/Wind", linewidth=1.5, color = "k")

#####################
# Figure cosmetics
#####################

# Axis labels, legend and formatting
for ax in [ax7, ax30]:
    ax.set_xlabel("time", fontsize=22)
    ax.set_ylabel("Correlation coefficient", fontsize=22)
    ax.legend(loc="best", fontsize=22)

# improve plot layout
gs.tight_layout(fig, h_pad=5)

#plt.savefig("visual2.png")
plt.show() #uncomment to plot


I am working on the file "Session5/code/09 Pandas - Part 2.ipynb".
The first task is:
Interpolation
As you can see from the printed values, there are a few NaN in our array.
It probably means that the data was not recorded at this date.
We have several ways of dealing with that :

We can ignore the relevant data line
Or we can infer the missing data with, say, an average of a few previous points.

We are going to infer the missing data through interpolation.
Say we have 1,NaN,2.
We infer the missing value with $(1+2)/2 = 1.5$
Can you write Python code for it?

# Let's see how to do that with pandas

# Get the data to interpolate
ozone = df_temp["Ozone"].values
solar = df_temp["Solar.R"].values
# Get a series of timestamps
timestamps = pd.to_datetime(df_temp["Date"].values)

# Create a new Series with the timestamp as index
#(or we could have set the index as timestamp in our df_temp dataframe)
s_ozone = pd.Series(ozone, index=timestamps)
s_solar = pd.Series(solar, index=timestamps)

oz_interp = s_ozone.interpolate(method = "time")
sol_interp = s_solar.interpolate(method = "time")

df_temp["Ozone_interp"] = oz_interp.values
df_temp["Solar.R_interp"] = sol_interp.values

df_temp.head(n=10) # use 10 rows to see how interpolation works

#Let's plot the data
import matplotlib.pylab as plt
from matplotlib.gridspec import GridSpec

%matplotlib inline
fig = plt.figure(figsize=(15, 15))
gs = GridSpec(2, 2, bottom=0.18, left=0.18, right=0.88)

axOz = fig.add_subplot(gs[0])
axSol = fig.add_subplot(gs[1])
axWind = fig.add_subplot(gs[2])
axTemp = fig.add_subplot(gs[3])

# Get time axis
x_date = df_temp["Date"].values

# Get the y values
y_oz = df_temp["Ozone_interp"].values
y_sol = df_temp["Solar.R_interp"].values
y_temp = df_temp["Temp"].values
y_wind = df_temp["Wind"].values

# Plot
axOz.plot(x_date, y_oz, label = "Ozone in ppm", linewidth=1.5)
axSol.plot(x_date, y_sol, label = "Solar Radiation", linewidth=1.5)
axWind.plot(x_date, y_temp, label = "Temperature in F", linewidth=1.5)
axTemp.plot(x_date, y_wind, label = "Wind", linewidth=1.5)

#####################
# Figure cosmetics
#####################

# Axis labels, legend and formatting
for ax in [axOz, axSol, axWind, axTemp]:
    ax.set_xlabel("time", fontsize=22)
    ax.legend(loc="best", fontsize=22)

# improve plot layout
gs.tight_layout(fig, h_pad=3)

#plt.savefig("visual.png") #uncomment to save plot
plt.show()


I am working on the file "Session5/code/10 Supervised Machine Learning - scikit learn.ipynb".
The first task is:
Digging deeper using metrics
Accuracy score, Classification report & Confusion matix
Here we will use a simple example to show metrics you can use: accuracy, classification reports and confusion matrices.

y_true is the test data
y_pred is the prediction
Can you write Python code for it?

y_true = ["cat", "ant", "cat", "cat", "ant", "bird", "bird"]
y_pred = ["ant", "ant", "cat", "cat", "ant", "cat", "bird"]

metrics.accuracy_score(y_true, y_pred, normalize=True, sample_weight=None)


I am working on the file "Session5/code/10 Supervised Machine Learning - scikit learn.ipynb".
The first task is:
Here 0 equates to setosa the first entry in the 'target_names' array
Can you write Python code for it?

# Now we create our model
model = LogisticRegression()
# We train it by passing in the test data and the actual results
model.fit(dataset.data, dataset.target)

# We use the model to create predictions
expected = dataset.target
predicted = model.predict(dataset.data)
# Using the metrics module we see the results of the model
metrics.accuracy_score(expected, predicted, normalize=True, sample_weight=None)


I am working on the file "Session5/code/10 Supervised Machine Learning - scikit learn.ipynb".
The first task is:
In the confusion_matrix the labels give the order of the rows.

ant was correctly categorised twice and was never miss categorised
bird was correctly categorised once and was categorised as cat once
cat was correctly categorised twice and was categorised as an ant once

Back to Iris predictions
Can you write Python code for it?

print(metrics.classification_report(expected, predicted,target_names=dataset['target_names']))

print (metrics.confusion_matrix(expected, predicted))


I am working on the file "Session5/code/10 Supervised Machine Learning - scikit learn.ipynb".
The first task is:
Iris Setosa

Iris Versicolor

Iris Virginica
Can you write Python code for it?

# Names of the type of information recorded about an Iris - called features
dataset['feature_names']

# First 10 sets of Iris data
dataset['data'][:10]

# The classification of each of the first 10 sets of Iris data - the target
dataset['target'][:10]


I am working on the file "Session_1/1_Introduction.ipynb".
The first task is:
REPL
What does this stand for?
READ EVAL PRINT LOOP

NOTE: Whitespace
Ever used a different language with semicolons and curly braces?
Python does not have that. EVERYTHING revolves around whitespace. Indentation us key and enforces clean code
We will cover this as we go on but be careful about spacing errors
Do not worry about the code for now, but take a look at how this works:
Can you write Python code for it?

for i in range(0,100):
print("bloop")

for i in range(0,3):
    print("bloop")


I am working on the file "Session_1/1_Introduction.ipynb".
The first task is:
Side note: Using the interpreter for errors
The example in the previous slide told you the error and the line it occured on, like this:

Always use this before you ask/search for the problem. The error text is what you should search for as well. We cannot answer questions like: "Why is my code not working?" any more than you can.
"I am trying to print a string at line 11 and it is giving me an indentation error" Saying something like this out loud (or in your head) may actually help you resolve the problem. That being said: we are here to help and never be afraid to ask for it, it is what you are paying for
Python REPL
When you type python in to your command prompt, this is the Python interpreter and it is an example of a REPL. Whatever you type is evaluated immediately and run, allowing you to write a new line.
What are the benefits of this? How else can this be done?
REPL Benefits

Rapid prototyping
Check it works on your system
Command line scripting
Slides, talks and demos!

But we don't always want to be writing one liners in the command line, do we?
Exercise: Python Scripts

Open up a text editor (atom)
Type print("hello, world")
Save the file as hello.py
In your command prompt, run it by typing python hello.py in the directory it was saved in

What happens?
REPL vs Scripts
What happens when you remove the print and the brackets?
A REPL is designed to show you the state of things with each command. So, typing:
"print me"
will print that string.
Scripts (or files) will not do this unless you tell it to.
This is partly because printing to the command prompt (also called: stdout) is one of the slowest operations you can perform in Python!
Learning Python
First port of call: This course and recommended reading
Second port of call: the interpreter (Learn by making mistakes. Fixing bugs and errors will increase your knowledge exponentially)
Third port of call: The Python docs
Fourth port of call: The Python docs
Fifth port of call: Us. Stop us and ask and we will do our best to help!
Asking for Help

Do you...
Can you write Python code for it?

language = "" #insert the language we are coding in here
print(language)


I am working on the file "Session_1/1_Introduction.ipynb".
The first task is:
Why Python?

Open Source (Free as in free speech, not free beer)
Widely used (bear with me)
MASSIVE standard library
Cross-platform
Used in Academia and Industry
Powerful yet easy to learn

What is this Python?

Made by Guido Van Rossum in 1991
Continues to be overseen by him
Interpreted
Object-oriented
Strongly Typed
Dynamically Typed

Benevolent Dictator for Life
Who Uses Python?

Academia (all disciplines)
Google (search)
Battlefield (Web APIs)
Disney (animation)
NASA (shuttle launch planning)
Instagram
Mozilla
Dropbox (EVERYTHING)
Government
You?

ya-di-ya-da
Python 2 vs. Python 3
Python 2

Legacy
Largest number of libraries
Coming to end of life soon

Python 3

Like the Xbox One versus the 360, it is partially backwards compatible
Almost a complete rewrite
Now the de facto version
Libraries in the process of being rewritten for support

How Does It Work?

Interpreted (not compiled)

Ever written in Java or C++?

Java files compiled to Class files before they are run

Python (and others) are not compiled, they are interpreted
What other languages are interpreted?

Interpeted Languages
HIGH LEVEL OVERVIEW Files are run line by line at runtime (no need to compile). Some checks are done before the file runs (spacing, module imports etc), but errors are typically not caught until that line of the file is executed.
Benefits:

(Generally) faster development time
Platform independent (source)
Dynamic types

Drawbacks:

Speed (kind of, we will look at that in this course)

What Can Python Do?

Websites (Django, Flask, Bottle)
Math (algorithms, optimisation and simulations)
Desktop apps (Kivy, Tkinter, WxPython)
Mobile apps (kivy)
Command line tools

Python Interpreter

Open up a command prompt (terminal) and type python
POW, you are in the python interpreter. Try this:
Can you write Python code for it?

"Hello, Python" #press enter here
"My name is <your name here>"


I am working on the file "Session_10/10_OOPython.ipynb".
The first task is:
Exercise
Create a class for a car and create a constructor (__init__) that sets the make, model and engine size.
Add some class variables that are general across all cars.
Create methods to get information about each instance.
Create static methods to do things with the car that would be common across all cars.
This is barely the surface to OO programming. You cover this in detail in the next semester.
Extra Credit - Overriding
When we create a class, we are given built in Python methods to run on it already (such as: print, add, multiply, type etc)
We can override these methods in order to do our own thing when we call it.
Prove it? Use on of your classes from earlier and print it. What is it printing? We can make that more detailed:
Can you write Python code for it?

class Bella:
    def __init__(self): #here is the constructor being overridden
        pass
    def __str__(self): #called when we call `print` on an object
        return "I want you, and I want you forever. One lifetime is simply not enough for me."

b = Bella()
print(b)


I am working on the file "Session_2/2_Coding.ipynb".
The first task is:
Coding in Python
Dr. Chris Gwilliams
gwilliamsc@cardiff.ac.uk
Writing in Python: PEP
Python Enhancement Proposals
Unsure how your code should be written? PEP is a style guide for Python and provides details on what is expected.

Use 4 spaces instead of tabs
Lines should be 79 characters long
Variables should follow snake_case
All lower case words, separated by underscores (_)

Classes should be Capitalised Words (MyClassExample)

PEP
Comments
Sometimes, you need to describe your code and the logic may be a bit complicated, or it took you a while to figure it out and you want to make a note.
You can't just write some text in the file or you will get errors, this is where comments come in! Comments are descriptions that the Python interpreter ignores.
Just type a # amd what ever you want to write and voíla!
It is ALWAYS a good idea to comment your code!
Can you write Python code for it?

# Does this make sense without comments?
with open('myfile.csv', 'rb') as opened_csv:
    spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
    for row in spamreader:
        print (', '.join(row))

# How about this?

#open csv file in readable format
with open('myfile.csv', 'rbU') as opened_csv:
    # read opened csv file with spaces as delimiters
    spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
    # loop through and print each line
    for row in spamreader:
        print (', '.join(row))


I am working on the file "Session_2/2_Coding.ipynb".
The first task is:
Don't Call It That:
These are keywords reserved in Python, so do not name any of your variables after these! You will learn about what many of these do throughout this course.
| False    | class  | finally  | is    | return |
|----------|--------|----------|-------|--------|
| continue | for    | lambda   | try   | True   |
| def      | from   | nonlocal | while | and    |
| del      | global | not      | with  | as     |
| elif     | if     | or       | yield | assert |
| else     | import | pass     | break | except |
| in       | raise  | None     |       |        |
Useful links on Python variables
http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html#other-languages-have-variables
http://anh.cs.luc.edu/python/hands-on/3.1/handsonHtml/variables.html
http://foobarnbaz.com/2012/07/08/understanding-python-variables/
http://www.diveintopython.net/native_data_types/declaring_variables.html
Types II
Python is Strongly Typed - The Python interpreter keeps track of all the variables and their associated types.
AND
Python is Dynamically Typed - Variables can be reassigned from their types. A variable is simply a value bound to a name, the variable does not hold a type, only the value does.
Can you write Python code for it?

print("Hello " + "World") #ok
print("hello" + 5) #strongly typed means this cannot happen!
name = "Chris"
name = "Pi"
"pi" + 6 #Strongly typed means no adding different types together!
name = 3.14 #dynamically typed means yes to changing the type of a variable!


I am working on the file "Session_2/2_Coding.ipynb".
The first task is:
Exercise
Try this with all the operators you know.
Built in Functions
A function is a block of code that:
- receives an input(s) (also known as arguments)
- performs an operation (or operations)
- Optionally, returns an output
Python has some built-in functions and we have used one already. What was it?
Functions - Print
print("Hello")
Format:
- function_name
- Open brackets
- inputs (separated, by, commas)
- Close brackets
Sometimes, inputs are optional. Not always. We will get to this.
Other Built in Functions

str()
len()
type()
int()

Exercise
Find out what the above functions do and use them in a script
What happens when you don't give each an argument?
Look up and write up definitions for the id and isinstance functions
str - converts an object to a string type
len - prints the length of an object
type - tells you the type of a literal or a variable
int - converts a type to an integer
id - a unique id that relates to where the item is stored in memory
isinstance - checks if an object if of the supplied type
Functions to Help You
dir
Can you write Python code for it?


I am working on the file "Session_2/2_Coding.ipynb".
The first task is:
Exercise
Try this with different types, how many can you find?
What happens when you do type([])?
type is an example of a built-in function, we will come back to these in a few sessions.
Strings
Strings in Python are unlike many languages you have seen before.
They can be:
Can you write Python code for it?


I am working on the file "Session_2/2_Coding.ipynb".
The first task is:
Multiplying Across Types
While Python is not happy to add/divide/subtract numbers from strings, it is more than happy to multiply
Can you write Python code for it?

float_type = 3.0
int_type = 5
print(int_type * float_type)

string_type = "hello"
print(string_type * int_type)

bool_type = True
print(string_type * bool_type) #why does this work?

print(int_type * bool_type)


I am working on the file "Session_2/2_Coding.ipynb".
The first task is:
Operating and Assigning
You may have a variable and want to change the value, this is reassigning, right?
python
year = 1998
year = 1999
There has to be an easier way. THERE IS. What is it?
Can you write Python code for it?


I am working on the file "Session_2/2_Coding.ipynb".
The first task is:
Recap Questions

What is a REPL?
What does strongly typed mean?
What does dynamically typed mean?
What is a literal?

Variable Recap

Give me 3 types in Python
What would you use each type for?

How do you declare multiple variables at the same time?
What is wrong with the code below?

name, age, height = 12, 'Terry'

Read, Eval, Print, Loop
Strongly typed is that type errors are caught as errors and Python keeps track of types of variables
Dynamically typed means that variables do not need a type declared at their declaration and the type can be changed at runtime
A literal is literally a value
Boolean - Store flags, String - hold text, Float - decimal numbers
Separate them with a comma
The sides do not match up, there is a missing literal

Finding Types
Got some code and don't know what the types are?
Python has some functions to help with this.
Can you write Python code for it?


I am working on the file "Session_2/2_Coding.ipynb".
The first task is:
Strings II
If you do not want to escape every special character, maybe there is a better way?
python
"I am a string and I don't care what is written inside me"
"""I am a string with triple double quotes and I can
run across multiple lines"""
Double quotes is generally better as you do not have to escape these special characters.
Exercise
Declare a variable to store a boolean literal
Declare and instantiate a new variable that stores your age
The first one was a trick! Declaring a new variable means giving it no value!
The second one would be: age = 20
Reassigning Variables
It would not be fun to have to create a new variable for every thing you want to store, right?
As well as being a huge inconvenience, it is actually really inefficient.
```python
age = 40
1 year passes
age = 41
```
Easy, right? By using the same variable name, it is now associated with your new value and the old value will be cleared up.
Printing
We have seen this print keyword thrown around alot, right? This is the best way to show some information.
Especially useful if your script takes a long time to run!
E.g. print("stuff")
Can you write Python code for it?

print("Got something to say?")
print("Use the print statement")
print("to print a string literal")
print("float literal")
print(3.14)
more_string = "or variable"
print(more_string)


I am working on the file "Session_2/2_Coding.ipynb".
The first task is:
What do single quotes usually mean in most languages?
Strings wrapped in 'single quotes' are typically chars (single characters). Python does not have this type.
char yes = 'Y' //char (Python does not have this)
string no = "no" //string
What are chars used for? What does Python have instead?
Chars are typically used as single character flags, like a 'Y' or an 'N' as an answer to a question, or to hold an initial.
Anything text based can be stored in a string but flags can be represented as a 0 or 1 or even using a Boolean value,, which is easiest to check against.
What happens if you use a single quote for strings and you write the word don't in the string? Try it out now!
How do we get around that?
Escaping Strings
When you want to include special characters (like ') then it is always good to escape them!
Ever seen a new line written as \n? That is an example of escaping.
Escape Character
An escape character is a character which invokes an alternative interpretation on subsequent characters in a character sequence.

This is pretty much always \
Can you write Python code for it?

'isn't # not gonna work

'isn\'t' # works a charm!


I am working on the file "Session_2/2_Coding.ipynb".
The first task is:
help
Can you write Python code for it?


I am working on the file "Session_2/2_Coding.ipynb".
The first task is:
id and Multi Variable Assignment
The id function is built into Python and returns the memory address of variables provided to it.
This is easier to see when we use multi-variable assignment in Python:
Can you write Python code for it?

a,b,c = 1,1,1
name, age, yob = "chris", 26, 1989
print(name, age, yob)
print(id(name), id(age), id(yob))
a,b,c = "Name",12,"6ft"
print(a,b,c) #NOTE: always balance the left and the right. 5 variables must have 5 values!


I am working on the file "Session_2/2_Homework.ipynb".
The first task is:
Look at the code below and fix the string in order to ensure it can be printed.
Can you write Python code for it?

my_string = "'Who's been eating my porridge?' said Papa Bear";
print(my_string)

assert_equal(my_string, "Who\'s been eating my porridge? said Papa Bear")


I am working on the file "Session_2/2_Homework.ipynb".
The first task is:
Rewrite the above code to Multi-Variable assignment
Can you write Python code for it?

assert_equal(credits, 20)
assert_equal(code, "CM6111")
assert_equal(name, "Computational Thinking")


I am working on the file "Session_2/2_Homework.ipynb".
The first task is:
Using operators, change the string below to print it 400 times.
Can you write Python code for it?

to_print = "Me, but 400 more times"
print(to_print) #do not change or repeat this line

assert_equal(to_print, "Me, but 400 more times"*400)


I am working on the file "Session_2/2_Homework.ipynb".
The first task is:
Why is the code below valid?
From your GCSE Maths knowledge, why are the 2 answers different?
What is the output of int(false)?
Can you write Python code for it?

print(True + 3 * 100)
print((True + 3) * 100)


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
But there must be a better way! What do you think?
AND/OR
There is a better way!
python
age > 15 and age < 25 # only true if both are true
age > 15 or age < 10 # true if either are true
The and keywords allow you to chain multiple conditionals together and react only if all of them are true
OR
We can use the or to allow multiple different scenarios.
Exercise
What is the output of these conditionals? Try it in your head first.
Can you write Python code for it?

make = "Tesla"
capacity = 80000
year = 2016
zero_to_sixty = 3

print(make == "Tesla" or make == "tesla")
print(make == "Tesla" and make == "tesla")
print(capacity < 80000 and capacity > 60000)
print(capacity > 60000 or capactity < 100000 and year == 2014 or  year == 2016)


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
Chained Comparison
So, we can write:
python
age > 15 and age < 25
Can you think of a way this could be shorter?
Can you write Python code for it?


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
Combining Comparisons
What if we want to know more than one thing about a value? What if I want to know if someone is more than 18 but also under 25?
We could do this:
Can you write Python code for it?


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
Comparison Operators
We can combine equality operators with comparison operators. What benefit is this?
Less Than or Equal To (<=)
2 <= 5
Greater Than or Equal To (>=)
2 >= 5
Exercise
What operator(s) could I have used to get these results?
python
2 ?? 5 #false
"a" ?? "b" #True
3.33 ?? 7 #False
400 ?? 1000 #True
1000 ?? 1000 #True
Can you write Python code for it?


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
Conditionals
What do you think a conditional is? Let's use these comparisons for good!
When we check if something is less than something else or for equality, just printing True or False is not always helpful. This is where conditionals come in.
If statements to the rescue!
Can you write Python code for it?

no_hours_slept = 3
if no_hours_slept < 5:
    print("Good Lord, turn Netflix off")
print("This print statement does not care how many hours you slept")


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
Elif (else if)
We can use this to check multiple statements if something is in a range of options (age is a good example).
Else then serves as our default state, the block of code that will
Exercise
Create a variable to store a pet type and write an if/else if/else statement that prints:
- four legged friend if the pet is a dog or cat
- reptilian rapport if the pet is a lizard or dragon
- aquatic amigo if the pet is a fish
- what the smeg in any other situation
Can you write Python code for it?

pet = "cat"
if pet == "fish":
    print("Aquatic amigo")
elif pet == "cat" or pet == "dog":
    print("four legged friend")
elif pet == "lizard" or pet == "dragon":
    print("reptilian rapport")
else:
    print("what the smeg")


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
Else
So, we can print when something is true, what about when it isn't?
Consider the situation of a nightclub app.
Can you write Python code for it?

age = 12
if age >= 18:
    print "come on in"


I am working on the file "Session_3/3_Conditionals_Flow_Control.ipynb".
The first task is:
Exercise
Add in brackets to make these statements give the answer printed:

4 * 5 / 2 - 1 (Answer: 20)
5 + 6 / 2 * 8 (Answer: 0.6875)
4 /1 * 2 + 1 (Answer: 1.333333)
Can you write Python code for it?