I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
In this representation, each "block" has a height of 0.25. The bandwidth is too narrow to provide any overlap between the blocks. This choice of kernel and bandwidth produces an estimate that is essentially a histogram with a large number of bins. It gives no sense of continuity for the distribution. Now, we examine the difference (relative to histograms) upon changing the the width (i.e. kernel) of the blocks.
Problem 1f
Plot the KDE of the PDF for the number of chinups middle aged men can do using bandwidths of 1 and 5 and a tophat kernel. How do the results differ from the histogram plots above?
Can you write Python code for it?

PDFtophat1 = # complete

# complete
# complete
# complete


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
Introduction to Visualization:
Density Estimation and Data Exploration
Version 0.1
There are many flavors of data analysis that fall under the "visualization" umbrella in astronomy. Today, by way of example, we will focus on 2 basic problems.

By AA Miller
16 September 2017
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
It turns out blocks are not an ideal representation for continuous data (see discussion on histograms above). Now we will explore the resulting PDF from other kernels.
Problem 1g Plot the KDE of the PDF for the number of chinups middle aged men can do using a gaussian and Epanechnikov kernel. How do the results differ from the histogram plots above?
Hint - you will need to select the bandwidth. The examples above should provide insight into the useful range for bandwidth selection. You may need to adjust the values to get an answer you "like."
Can you write Python code for it?

PDFgaussian = # complete
PDFepanechnikov = # complete


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
Now that we have a sense of the data structure, it is useful to examine the distribution of features. Above, we went to great pains to produce histograms, KDEs, and rug plots. seaborn handles all of that effortlessly with the distplot function.
Problem 3b
Plot the distribution of petal lengths for the Iris data set.
Can you write Python code for it?

# note - hist, kde, and rug all set to True, set to False to turn them off
with sns.axes_style("dark"):
    sns.distplot(iris['petal_length'], bins=20, hist=True, kde=True, rug=True)


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
Of course, when there are many many data points, scatter plots become difficult to interpret. As in the example below:
Can you write Python code for it?

with sns.axes_style("darkgrid"):
    xexample = np.random.normal(loc = 0.2, scale = 1.1, size = 10000)
    yexample = np.random.normal(loc = -0.1, scale = 0.9, size = 10000)

    plt.scatter(xexample, yexample)


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
Problem 1e
Plot the KDE of the PDF for the number of chinups middle aged men can do using a bandwidth of 0.1 and a tophat kernel.
Hint - as a general rule, the grid should be smaller than the bandwidth when plotting the PDF.
Can you write Python code for it?

grid = # complete
PDFtophat = kde_sklearn( # complete
plt.plot( # complete


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
That is much nicer than what was presented above. However - we still have a problem in that our data live in 4D, but we are (mostly) limited to 2D projections of that data. One way around this is via the seaborn version of a pairplot, which plots the distribution of every variable in the data set against each other. (Here is where the integration with pandas DataFrames becomes so powerful.)
Can you write Python code for it?

sns.pairplot(iris[["sepal_length", "sepal_width", "petal_length", "petal_width"]])


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
The folks behind seaborn have thought a lot about color palettes, which is a good thing. Remember - the choice of color for plots is one of the most essential aspects of visualization. A poor choice of colors can easily mask interesting patterns or suggest structure that is not real. To learn more about what is available, see the seaborn color tutorial.
Here we load the default:
Can you write Python code for it?

# default color palette

current_palette = sns.color_palette()
sns.palplot(current_palette)


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
These small changes significantly change the output PDF. With fewer bins we get something closer to a continuous distribution, while shifting the bin centers reduces the probability to zero at 9 chinups.
What if we instead allow the bin width to vary and require the same number of points in each bin? You can determine the bin edges for bins with 5 sources using the following command:
bins = np.append(np.sort(chinups)[::5], np.max(chinups))

Problem 1c
Plot a histogram with variable width bins, each with the same number of points.
Hint - setting normed = True will normalize the bin heights so that the PDF integrates to 1.
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
This plot is much more appealing (and informative) than the previous two. For the first time we can clearly see that the distribution is not actually centered on the origin. Now we will move back to the Iris data set.
Suppose we want to see univariate distributions in addition to the scatter plot? This is certainly possible with matplotlib and you can find examples on the web, however, with seaborn this is really easy.
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
We can see that the plot has significantly changed styles.
seaborn has 5 style presets: darkgrid, whitegrid, dark, white, and ticks. You can change the preset using the following:
sns.set_style("whitegrid")

which will change the output for all subsequent plots. Note - if you want to change the style for only a single plot, that can be accomplished with the following:
with sns.axes_style("dark"):

with all ploting commands inside the with statement.
Problem 3a
Re-plot the sine curve using each seaborn preset to see which you like best - then adopt this for the remainder of the notebook.
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefVisualization.ipynb".
The first task is:
While the above plot provides a significant improvement over the scatter plot by providing a better sense of the density near the center of the distribution, the binedge effects are clearly present. An even better solution, like before, is a density estimate, which is easily built into seaborn via the kdeplot function.
Can you write Python code for it?

with sns.axes_style("darkgrid"):
    sns.kdeplot(xexample, yexample,shade=False)


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Already with this simple plot we see a problem - the choice of bin centers and number of bins suggest that there is a 0% probability that middle aged men can do 10 chinups. Intuitively this seems incorrect, so lets examine how the histogram changes if we change the number of bins or the bin centers.
Problem 1b
Using the same data make 2 new histograms: (i) one with 5 bins (bins = 5), and (ii) one with the bars centered on the left bin edges (align = "left").
Hint - if overplotting the results, you may find it helpful to use the histtype = "step" option
Can you write Python code for it?

plt.hist(chinups, bins = 5, histtype="step", lw = 3)
plt.hist(chinups, align = "left", histtype="step", lw = 3)


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Even better - there is an option to create a PairGrid which allows fine tuned control of the data as displayed above, below, and along the diagonal. In this way it becomes possible to avoid having symmetric redundancy, which is not all that informative. In the example below, we will show scatter plots and contour plots simultaneously.
Can you write Python code for it?

g = sns.PairGrid(iris, vars = ["sepal_length", "sepal_width", "petal_length", "petal_width"],
                 hue = "species", diag_sharey=False)
g.map_lower(sns.kdeplot)
g.map_upper(plt.scatter, edgecolor='white')
g.map_diag(sns.kdeplot, lw=3)


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
For data sets where we have classification labels, we can even color the various points using the hue option, and produce KDEs along the diagonal with diag_type = 'kde'.
Can you write Python code for it?

sns.pairplot(iris, vars = ["sepal_length", "sepal_width", "petal_length", "petal_width"],
             hue = "species", diag_kind = 'kde')


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Here, we see that there are many points, clustered about the origin, but we have no sense of the underlying density of the distribution. 2D histograms, such as plt.hist2d(), can alleviate this problem. I prefer to use plt.hexbin() which is a little easier on the eyes (though note - these histograms are just as subject to the same issues discussed above).
Can you write Python code for it?

# hexbin w/ bins = "log" returns the log of counts/bin
# mincnt = 1 displays only hexpix with at least 1 source present
with sns.axes_style("darkgrid"):
    plt.hexbin(xexample, yexample, bins = "log", cmap = "viridis", mincnt = 1)
    plt.colorbar()


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
In this representation, each "block" has a height of 0.25. The bandwidth is too narrow to provide any overlap between the blocks. This choice of kernel and bandwidth produces an estimate that is essentially a histogram with a large number of bins. It gives no sense of continuity for the distribution. Now, we examine the difference (relative to histograms) upon changing the the width (i.e. kernel) of the blocks.
Problem 1f
Plot the KDE of the PDF for the number of chinups middle aged men can do using bandwidths of 0.5 and 2.5 and a tophat kernel. How do the results differ from the histogram plots above?
Can you write Python code for it?

PDFtophat1 = kde_sklearn(chinups, grid, bandwidth = 1, kernel = 'tophat')
plt.plot(grid, PDFtophat1, 'MediumAquaMarine', lw = 3, label = "bw = 1")

PDFtophat5 = kde_sklearn(chinups, grid, bandwidth = 5, kernel = 'tophat')
plt.plot(grid, PDFtophat5, 'Tomato', lw = 3, label = "bw = 5")
plt.legend()


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Introduction to Visualization:
Density Estimation and Data Exploration
Version 0.1
There are many flavors of data analysis that fall under the "visualization" umbrella in astronomy. Today, by way of example, we will focus on 2 basic problems.

By AA Miller
16 September 2017
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
It turns out blocks are not an ideal representation for continuous data (see discussion on histograms above). Now we will explore the resulting PDF from other kernels.
Problem 1g Plot the KDE of the PDF for the number of chinups middle aged men can do using a gaussian and Epanechnikov kernel. How do the results differ from the histogram plots above?
Hint - you will need to select the bandwidth. The examples above should provide insight into the useful range for bandwidth selection. You may need to adjust the values to get an answer you "like."
Can you write Python code for it?

PDFgaussian = kde_sklearn(chinups, grid, bandwidth = 1, kernel = 'gaussian')
plt.plot(grid, PDFgaussian, 'DarkOrange', lw = 3, label = "gaussian")

PDFepanechnikov = kde_sklearn(chinups, grid, bandwidth = 2, kernel = 'epanechnikov')
plt.plot(grid, PDFepanechnikov, 'SlateGrey', lw = 3, label = "epanechnikov")
plt.legend(loc = 2)


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Of course, when there are many many data points, scatter plots become difficult to interpret. As in the example below:
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Problem 1a
Plot the histogram for the number of chinups using the default settings in pyplot.
Can you write Python code for it?

plt.hist(chinups, histtype = "step", lw = 3)


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Seaborn
Seaborn is a plotting package that enables many useful features for exploration. In fact, a lot of the functionality that we developed above can readily be handled with seaborn.
Aside - aesthetics vary greatly from person to person. The mere act of importing seaborn drastically changes the appearance of all plots made thereafter. Some really like these changes, others do not. I've heard it said that seaborn is no good as it forces an import MakeUgly under the hood. The choice is up to you.
To begin, we will make the same plot that we created in matplotlib.
Can you write Python code for it?

import seaborn as sns

fig = plt.figure()
ax = fig.add_subplot(111)

ax.plot(x,y, lw = 2)
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_xlim(0, 6*np.pi)


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
So, what is the optimal choice of bandwidth and kernel? Unfortunately, there is no hard and fast rule, as every problem will likely have a different optimization. Typically, the choice of bandwidth is far more important than the choice of kernel. In the case where the PDF is likely to be gaussian (or close to gaussian), then Silverman's rule of thumb can be used:
$$h = 1.059 \sigma n^{-1/5}$$
where $h$ is the bandwidth, $\sigma$ is the standard deviation of the samples, and $n$ is the total number of samples. Note - in situations with bimodal or more complicated distributions, this rule of thumb can lead to woefully inaccurate PDF estimates. The most general way to estimate the choice of bandwidth is via cross validation (we will cover cross-validation later today).
What about multidimensional PDFs? It is possible using many of the Python implementations of KDE to estimate multidimensional PDFs, though it is very very important to beware the curse of dimensionality in these circumstances.
Problem 2) Data Exploration
Now a more open ended topic: data exploration. In brief, data exploration encompases a large suite of tools (including those discussed above) to examine data that live in large dimensional spaces. There is no single best method or optimal direction for data exploration. Instead, today we will introduce some of the tools available via python.
As an example we will start with a basic line plot - and examine tools beyond matplotlib.
Can you write Python code for it?

x = np.arange(0, 6*np.pi, 0.1)
y = np.cos(x)

plt.plot(x,y, lw = 2)
plt.xlabel('X')
plt.ylabel('Y')
plt.xlim(0, 6*np.pi)


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
That is much nicer than what was presented above. However - we still have a problem in that our data live in 4D, but we are (mostly) limited to 2D projections of that data. One way around this is via the seaborn version of a pairplot, which plots the distribution of every variable in the data set against each other. (Here is where the integration with pandas DataFrames becomes so powerful.)
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
The folks behind seaborn have thought a lot about color palettes, which is a good thing. Remember - the choice of color for plots is one of the most essential aspects of visualization. A poor choice of colors can easily mask interesting patterns or suggest structure that is not real. To learn more about what is available, see the seaborn color tutorial.
Here we load the default:
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
These small changes significantly change the output PDF. With fewer bins we get something closer to a continuous distribution, while shifting the bin centers reduces the probability to zero at 9 chinups.
What if we instead allow the bin width to vary and require the same number of points in each bin? You can determine the bin edges for bins with 5 sources using the following command:
bins = np.append(np.sort(chinups)[::5], np.max(chinups))

Problem 1c
Plot a histogram with variable width bins, each with the same number of points.
Hint - setting normed = True will normalize the bin heights so that the PDF integrates to 1.
Can you write Python code for it?

bins = np.append(np.sort(chinups)[::5], np.max(chinups))
plt.hist(chinups, bins = bins, histtype = "step", normed = True, lw = 3)


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
This plot is much more appealing (and informative) than the previous two. For the first time we can clearly see that the distribution is not actually centered on the origin. Now we will move back to the Iris data set.
Suppose we want to see univariate distributions in addition to the scatter plot? This is certainly possible with matplotlib and you can find examples on the web, however, with seaborn this is really easy.
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day0/TooBriefVizSolutions.ipynb".
The first task is:
We can see that the plot has significantly changed styles.
seaborn has 5 style presets: darkgrid, whitegrid, dark, white, and ticks. You can change the preset using the following:
sns.set_style("whitegrid")

which will change the output for all subsequent plots. Note - if you want to change the style for only a single plot, that can be accomplished with the following:
with sns.axes_style("dark"):

with all ploting commands inside the with statement.
Problem 3a
Re-plot the sine curve using each seaborn preset to see which you like best - then adopt this for the remainder of the notebook.
Can you write Python code for it?

sns.set_style("ticks")

fig = plt.figure()
ax = fig.add_subplot(111)

ax.plot(x,y, lw = 2)
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_xlim(0, 6*np.pi)


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
If this all worked correctly, the array sample should contain a series of 50000 points drawn from the posterior. Let's plot them and check:
Can you write Python code for it?

# plot a histogram of the sample
plt.hist(sample, bins=50, histtype="stepfilled", alpha=0.3, normed=True)

# plot a best-fit Gaussian
F_fit = np.linspace(975, 1025)
pdf = stats.norm(np.mean(sample), np.std(sample)).pdf(F_fit)

plt.plot(F_fit, pdf, '-k')
plt.xlabel("F"); plt.ylabel("P(F)")


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
Now in a similar manner to what we did above for the MCMC Bayesian posterior, we'll compute the sample mean and standard deviation to determine the errors on the parameters.
Can you write Python code for it?

mu_samp = samples[:, 0]
sig_samp = abs(samples[:, 1])

print " mu    = {0:.0f} +/- {1:.0f}".format(mu_samp.mean(), mu_samp.std())
print " sigma = {0:.0f} +/- {1:.0f}".format(sig_samp.mean(), sig_samp.std())


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
Now let's make a simple visualization of the "measured" data:
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

fig, ax = plt.subplots()
ax.errorbar(F, np.arange(N), xerr=e, fmt='ok', ecolor='gray', alpha=0.5)
ax.vlines([F_true], 0, N, linewidth=5, alpha=0.2)
ax.set_xlabel("Flux");ax.set_ylabel("measurement number");


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
Now that we have the samples, we'll use a convenience routine from astroML to plot the traces and the contours representing one and two standard deviations:
Can you write Python code for it?

from astroML.plotting import plot_mcmc
fig = plt.figure()
ax = plot_mcmc(sample.T, fig=fig, labels=[r'$\mu$', r'$\sigma$'], colors='k')
ax[0].plot(sample[:, 0], sample[:, 1], ',k', alpha=0.1)
ax[0].plot([mu_true], [sigma_true], 'o', color='red', ms=10);


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
Now we set up the problem, including generating some random starting guesses for the multiple chains of points.
Can you write Python code for it?

ndim = 1  # number of parameters in the model
nwalkers = 50  # number of MCMC walkers
nburn = 1000  # "burn-in" period to let chains stabilize
nsteps = 2000  # number of MCMC steps to take

# we'll start at random locations between 0 and 2000
starting_guesses = 2000 * np.random.rand(nwalkers, ndim)

import emcee
sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior, args=[F, e])
sampler.run_mcmc(starting_guesses, nsteps)

sample = sampler.chain  # shape = (nwalkers, nsteps, ndim)
sample = sampler.chain[:, nburn:, :].ravel()  # discard burn-in points


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
Varying Photon Counts: The Frequentist Approach
The resulting likelihood is the convolution of the intrinsic distribution with the error distribution, so we have
$$\mathcal{L}(D~|~\theta) = \prod_{i=1}^N \frac{1}{\sqrt{2\pi(\sigma^2 + e_i^2)}}\exp\left[\frac{-(F_i - \mu)^2}{2(\sigma^2 + e_i^2)}\right]$$
Analogously to above, we can analytically maximize this likelihood to find the best estimate for $\mu$:
$$\mu_{est} =  \frac{\sum w_i F_i}{\sum w_i};~~w_i = \frac{1}{\sigma^2 + e_i^2} $$
And here we have a problem: the optimal value of $\mu$ depends on the optimal value of $\sigma$.  The results are correlated, so we can no longer use straightforward analytic methods to arrive at the frequentist result.
Nevertheless, we can use numerical optimization techniques to determine the maximum likelihood value. Here we'll use the optimization routines available within Scipy's optimize submodule:
Can you write Python code for it?

def log_likelihood(theta, F, e):
    return -0.5 * np.sum(np.log(2 * np.pi * (theta[1] ** 2 + e ** 2))
                         + (F - theta[0]) ** 2 / (theta[1] ** 2 + e ** 2))

# maximize likelihood <--> minimize negative likelihood
def neg_log_likelihood(theta, F, e):
    return -log_likelihood(theta, F, e)

from scipy import optimize
theta_guess = [900, 5]
theta_est = optimize.fmin(neg_log_likelihood, theta_guess, args=(F, e))
print("""
      Maximum likelihood estimate for {0} data points:
          mu={theta[0]:.0f}, sigma={theta[1]:.0f}
      """.format(N, theta=theta_est))


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
We end up with a sample of points drawn from the (normal) posterior distribution.  The mean and standard deviation of this posterior are the corollary of the frequentist maximum likelihood estimate above:
Can you write Python code for it?

print("""
      F_true = {0}
      F_est  = {1:.0f} +/- {2:.0f} (based on {3} measurements)
      """.format(F_true, np.mean(sample), np.std(sample), N))


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
We find that for 50 measurements of the flux, our estimate has an error of about 0.4% and is consistent with the input value.
Bayesian Approach to Simple Photon Counts
The Bayesian approach, as you might expect, begins and ends with probabilities.  It recognizes that what we fundamentally want to compute is our knowledge of the parameters in question, i.e. in this case,
$$ P(F_{\rm true}~|~D) $$
N.b.: that this formulation of the problem is fundamentally contrary to the frequentist philosophy, which says that probabilities have no meaning for model parameters like $F_{\rm true}$. Within the Bayesian interpretation of probability, this is perfectly acceptable.
To compute this result, Bayesians next apply Bayes' Theorem, a fundamental law of probability:
$$ P(F_{\rm true}~|~D) = \frac{P(D~|~F_{\rm true})~P(F_{\rm true})}{P(D)} $$
Though Bayes' theorem is where Bayesians get their name, it is not this law itself that is controversial, but the Bayesian interpretation of probability implied by the term $P(F_{\rm true}~|~D)$.
Let's take a look at each of the terms in this expression:

$P(F_{\rm true}~|~D)$: The posterior, or the probability of the model parameters given the data: this is the result we want to compute.
$P(D~|~F_{\rm true})$: The likelihood, which is proportional to the $\mathcal{L}(D~|~F_{\rm true})$ in the frequentist approach, above.
$P(F_{\rm true})$: The model prior, which encodes what we knew about the model prior to the application of the data $D$.
$P(D)$: The data probability, which in practice amounts to simply a normalization term.

If we set the prior $P(F_{\rm true}) \propto 1$ (a flat prior), we find
$$P(F_{\rm true}|D) \propto \mathcal{L}(D|F_{\rm true})$$
and the Bayesian probability is maximized at precisely the same value as the frequentist result! So despite the philosophical differences, we see that (for this simple problem at least) the Bayesian and frequentist point estimates are equivalent.
But What About the Prior?
We glossed over something here: the prior, $P(F_{\rm...
Can you write Python code for it?

def log_prior(theta):
    return 1  # flat prior

def log_likelihood(theta, F, e):
    return -0.5 * np.sum(np.log(2 * np.pi * e ** 2)
                         + (F - theta[0]) ** 2 / e ** 2)

def log_posterior(theta, F, e):
    return log_prior(theta) + log_likelihood(theta, F, e)


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-01-Probability.ipynb".
The first task is:
We see that as expected for this simple problem, the Bayesian approach yields the same result as the frequentist approach!
Discussion
Now, you might come away with the impression that the Bayesian method is unnecessarily complicated, and in this case it certainly is. Using an Affine Invariant Markov Chain Monte Carlo Ensemble sampler to characterize a one-dimensional normal distribution is a bit like using the Death Star to destroy a beach ball.
But we did it here because it demonstrates an approach that can scale to complicated posteriors in many, many dimensions, and can provide nice results in more complicated situations where an analytic likelihood approach is not possible.
As a side note, you might also have noticed one little sleight of hand: at the end, we use a frequentist approach to characterize our posterior samples!  When we computed the sample mean and standard deviation above, we were employing a distinctly frequentist technique to characterize the posterior distribution. The pure Bayesian result for a problem like this would be to report the posterior distribution itself (i.e. its representative sample), and leave it at that. That is, in pure Bayesianism the answer to a question is not a single number with error bars; the answer is the posterior distribution over the model parameters!
Adding a Dimension: Exploring a more sophisticated model
Let's briefly take a look at a more complicated situation, and compare the frequentist and Bayesian results yet again.  Above we assumed that the star was static: now let's assume that we're looking at an object which we suspect has some stochastic variation — that is, it varies with time, but in an unpredictable way (a Quasar is a good example of such an object).
We'll propose a simple 2-parameter Gaussian model for this object: $\theta = [\mu, \sigma]$ where $\mu$ is the mean value, and $\sigma$ is the standard deviation of the variability intrinsic to the object.  Thus our model for the probability of the true flux at the time of each observation...
Can you write Python code for it?

np.random.seed(42)  # for reproducibility
N = 100  # we'll use more samples for the more complicated model
mu_true, sigma_true = 1000, 15  # stochastic flux model

F_true = stats.norm(mu_true, sigma_true).rvs(N)  # (unknown) true flux
F = stats.poisson(F_true).rvs()  # observed flux: true flux plus Poisson errors.
e = np.sqrt(F)  # root-N error, as above


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
By eye, this seems to have worked as desired: the fit is much closer to our intuition!
However a Bayesian might point out that the motivation for this new loss function is a bit suspect: as we showed, the squared-loss can be straightforwardly derived from a Gaussian likelihood. The Huber loss seems a bit ad hoc: where does it come from? How should we decide what value of $c$ to use? Is there any good motivation for using a linear loss on outliers, or should we simply remove them instead? How might this choice affect our resulting model?
A Bayesian Approach to Outliers: Nuisance Parameters
The Bayesian approach to accounting for outliers generally involves modifying the model so that the outliers are accounted for. For this data, it is abundantly clear that a simple straight line is not a good fit to our data. So let's propose a more complicated model that has the flexibility to account for outliers.
One option is to choose a mixture between a signal and a background:
$$
\begin{array}{ll}
p({x_i}, {y_i},{e_i}~|~\theta,{g_i},\sigma,\sigma_b) = & \frac{g_i}{\sqrt{2\pi e_i^2}}\exp\left[\frac{-\left(\hat{y}(x_i~|~\theta) - y_i\right)^2}{2e_i^2}\right] \
&+ \frac{1 - g_i}{\sqrt{2\pi \sigma_B^2}}\exp\left[\frac{-\left(\hat{y}(x_i~|~\theta) - y_i\right)^2}{2\sigma_B^2}\right]
\end{array}
$$
What we've done is expanded our model with some nuisance parameters: ${g_i}$ is a series of weights which range from 0 to 1 and encode for each point $i$ the degree to which it fits the model. $g_i=0$ indicates an outlier, in which case a Gaussian of width $\sigma_B$ is used in the computation of the likelihood. This $\sigma_B$ can also be a nuisance parameter, or its value can be set at a sufficiently high number, say 50.
Our model is much more complicated now: it has 22 parameters rather than 2, but the majority of these can be considered nuisance parameters, which can be marginalized-out in the end, just as we marginalized (integrated) over $p$ in the Billiard example.  Let's construct a function which implements this...
Can you write Python code for it?

# theta will be an array of length 2 + N, where N is the number of points
# theta[0] is the intercept, theta[1] is the slope,
# and theta[2 + i] is the weight g_i

def log_prior(theta):
    #g_i needs to be between 0 and 1
    if (all(theta[2:] > 0) and all(theta[2:] < 1)):
        return 0
    else:
        return -np.inf  # recall log(0) = -inf

def log_likelihood(theta, x, y, e, sigma_B):
    dy = y - theta[0] - theta[1] * x
    g = np.clip(theta[2:], 0, 1)  # g<0 or g>1 leads to NaNs in logarithm
    logL1 = np.log(g) - 0.5 * np.log(2 * np.pi * e ** 2) - 0.5 * (dy / e) ** 2
    logL2 = np.log(1 - g) - 0.5 * np.log(2 * np.pi * sigma_B ** 2) - 0.5 * (dy / sigma_B) ** 2
    return np.sum(np.logaddexp(logL1, logL2))

def log_posterior(theta, x, y, e, sigma_B):
    return log_prior(theta) + log_likelihood(theta, x, y, e, sigma_B)


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
In other words, we'd give Bob the following odds of winning:
Can you write Python code for it?

print("Odds against Bob winning: {0:.0f} to 1".format((1. - freq_prob) / freq_prob))


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
It's clear on examination that the outliers are exerting a disproportionate influence on the fit. This is due to the nature of the squared loss function. If you have a single outlier that is, say 10 standard deviations away from the fit, its contribution to the loss will out-weigh that of 25 points which are 2 standard deviations away!
Clearly the squared loss is overly sensitive to outliers, and this is causing issues with our fit. One way to address this within the frequentist paradigm is to simply adjust the loss function to be more robust.
Frequentist Correction for Outliers: Huber Loss
The variety of possible loss functions is quite literally infinite, but one relatively well-motivated option is the Huber loss. The Huber loss defines a critical value at which the loss curve transitions from quadratic to linear. Let's create a plot which compares the Huber loss to the standard squared loss for several critical values $c$:
Can you write Python code for it?

t = np.linspace(-20, 20)

def huber_loss(t, c=3):
    return ((abs(t) < c) * 0.5 * t ** 2
            + (abs(t) >= c) * -c * (0.5 * c - abs(t)))

plt.plot(t, 0.5 * t ** 2, label="squared loss", lw=2)
for c in (10, 5, 3):
    plt.plot(t, huber_loss(t, c), label="Huber loss, c={0}".format(c), lw=2)
plt.ylabel('loss')
plt.xlabel('standard deviations')
plt.legend(loc='best', frameon=False);


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
Now we'll run the MCMC samples to explore the parameter space:
Can you write Python code for it?

# Note that this step will take a few minutes to run!

ndim = 2 + len(x)  # number of parameters in the model
nwalkers = 50  # number of MCMC walkers
nburn = 10000  # "burn-in" period to let chains stabilize
nsteps = 15000  # number of MCMC steps to take

# set theta near the maximum likelihood, with
np.random.seed(10)
starting_guesses = np.zeros((nwalkers, ndim))
starting_guesses[:, :2] = np.random.normal(theta1, 1, (nwalkers, 2))
starting_guesses[:, 2:] = np.random.normal(0.5, 0.1, (nwalkers, ndim - 2))

import emcee
sampler = emcee.EnsembleSampler(nwalkers, ndim, log_posterior, args=[x, y, e, 50])
sampler.run_mcmc(starting_guesses, nsteps, rstate0=np.random.get_state())

sample = sampler.chain  # shape = (nwalkers, nsteps, ndim)
sample = sampler.chain[:, nburn:, :].reshape(-1, ndim)


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
Once we have these samples, we can exploit a very nice property of the Markov chains. Because their distribution models the posterior, we can integrate out (i.e. marginalize) over nuisance parameters simply by ignoring them!
We can look at the (marginalized) distribution of slopes and intercepts by examining the first two columns of the sample:
Can you write Python code for it?

plt.plot(sample[:, 0], sample[:, 1], ',k', alpha=0.1)
plt.xlabel('intercept')
plt.ylabel('slope');


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
Our task is to find a line of best-fit to the data. It's clear upon visual inspection that there are some outliers among these points, but let's start with a simple non-robust maximum likelihood approach. Like we saw in the previous post, the following simple maximum likelihood result can be considered to be either frequentist or Bayesian (with uniform priors): in this sort of simple problem, the approaches are essentially equivalent.
We'll propose a simple linear model, which has a slope and an intercept encoded in a parameter vector $\theta$. The model is defined as follows:
$$
\hat{y}(x~|~\theta) = \theta_0 + \theta_1 x
$$
Given this model, we can compute a Gaussian likelihood for each point:
$$
p(x_i,y_i,e_i~|~\theta) \propto \exp\left[-\frac{1}{2e_i^2}\left(y_i - \hat{y}(x_i~|~\theta)\right)^2\right]
$$
The total likelihood is the product of all the individual likelihoods. Computing this and taking the log, we have:
$$
\log \mathcal{L}(D~|~\theta) = \mathrm{const} - \sum_i \frac{1}{2e_i^2}\left(y_i - \hat{y}(x_i~|~\theta)\right)^2
$$
This should all look pretty familiar. This final expression is the log-likelihood of the data given the model, which can be maximized to find the $\theta$ corresponding to the maximum-likelihood model. Equivalently, we can minimize the summation term, which is known as the loss:
$$
\mathrm{loss} = \sum_i \frac{1}{2e_i^2}\left(y_i - \hat{y}(x_i~|~\theta)\right)^2
$$
This loss expression is known as a squared loss; here we've simply shown that the squared loss can be derived from the Gaussian log likelihood.
Standard Likelihood Approach
Following the logic of the previous post, we can maximize the likelihood (or, equivalently, minimize the loss) to find $\theta$ within a frequentist paradigm. For a flat prior in $\theta$, the maximum of the Bayesian posterior will yield the same result.
(note that there are good arguments based on the principle of maximum entropy that a flat prior is not the best choice here; we'll ignore that detail for now, as it's a very small effect for...
Can you write Python code for it?

from scipy import optimize

def squared_loss(theta, x=x, y=y, e=e):
    dy = y - theta[0] - theta[1] * x
    return np.sum(0.5 * (dy / e) ** 2)

theta1 = optimize.fmin(squared_loss, [0, 0], disp=False)

xfit = np.linspace(0, 100)
plt.errorbar(x, y, e, fmt='.k', ecolor='gray')
plt.plot(xfit, theta1[0] + theta1[1] * xfit, '-k')
plt.title('Maximum Likelihood fit: Squared Loss');


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
So we've estimated using frequentist ideas that Alice will win about 18 times for each time Bob wins. Let's try a Bayesian approach next.
Bayesian Approach
We can also approach this problem from a Bayesian standpoint. This is slightly more involved, and requires us to first define some notation.
We'll consider the following random variables:

$B$ = Bob Wins
$D$ = observed data, i.e. $D = (n_A, n_B) = (5, 3)$
$p$ = unknown probability that a ball lands on Alice's side during the current game

We want to compute $P(B~|~D)$; that is, the probability that Bob wins given our observation that Alice currently has five points to Bob's three.
The general Bayesian method of treating nuisance parameters is marginalization, or integrating the joint probability over the entire range of the nuisance parameter. In this case, that means that we will first calculate the joint distribution
$$
P(B,p~|~D)
$$
and then marginalize over $p$ using the following identity:
$$
P(B~|~D) \equiv \int_{-\infty}^\infty P(B,p~|~D) {\mathrm d}p
$$
This identity follows from the definition of conditional probability, and the law of total probability: that is, it is a fundamental consequence of probability axioms and will always be true. Even a frequentist would recognize this; they would simply disagree with our interpretation of $P(p)$ as being a measure of uncertainty of our own knowledge.
Building our Bayesian Expression
To compute this result, we will manipulate the above expression for $P(B~|~D)$ until we can express it in terms of other quantities that we can compute.
We'll start by applying the following definition of conditional probability to expand the term $P(B,p~|~D)$:
$$
P(B~|~D) = \int P(B~|~p, D) P(p~|~D) dp
$$
Next we use Bayes' rule to rewrite $P(p~|~D)$:
$$
P(B~|~D) = \int P(B~|~p, D) \frac{P(D~|~p)P(p)}{P(D)} dp
$$
Finally, using the same probability identity we started with, we can expand $P(D)$ in the denominator to find:
$$
P(B~|~D) = \frac{\int P(B~|~p,D) P(D~|~p) P(p) dp}{\int P(D~|~p)P(p) dp}
$$
Now the desired...
Can you write Python code for it?

from scipy.special import beta
bayes_prob = beta(6 + 1, 5 + 1) / beta(3 + 1, 5 + 1)

print("P(B|D) = {0:.2f}".format(bayes_prob))


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
The associated odds are the following:
Can you write Python code for it?

print("Bayesian odds against Bob winning: {0:.0f} to 1".format((1. - bayes_prob) / bayes_prob))


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
There is not an extremely strong constraint on either of these, but we do see that $(g_1, g_2) = (1, 0)$ is slightly favored: the means of $g_1$ and $g_2$ are greater than and less than 0.5, respecively. If we choose a cutoff at $g=0.5$, our algorithm has identified $g_2$ as an outlier.
Let's make use of all this information, and plot the marginalized best model over the original data. As a bonus, we'll draw red circles to indicate which points the model detects as outliers:
Can you write Python code for it?

theta3 = np.mean(sample[:, :2], 0)
g = np.mean(sample[:, 2:], 0)
outliers = (g < 0.5)

plt.errorbar(x, y, e, fmt='.k', ecolor='gray')
plt.plot(xfit, theta1[0] + theta1[1] * xfit, color='lightgray')
plt.plot(xfit, theta2[0] + theta2[1] * xfit, color='lightgray')
plt.plot(xfit, theta3[0] + theta3[1] * xfit, color='black')
plt.plot(x[outliers], y[outliers], 'ro', ms=20, mfc='none', mec='red')
plt.title('Maximum Likelihood fit: Bayesian Marginalization');


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-02-Nuisance.ipynb".
The first task is:
We'll visualize this data below:
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt

plt.errorbar(x, y, e, fmt='.k', ecolor='gray');


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
As expected, the approximate solution is very close to the exact solution for large $N$, which gives us confidence that we're computing the right thing.
Let's return to our 3-point dataset and see the results:
Can you write Python code for it?

print("approximate CI: ({0:.1f}, {1:.1f})".format(*approx_CI(D)))
print("exact CI:       ({0:.1f}, {1:.1f})".format(*exact_CI(D)))


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
As is promised by frequentism, 95% of the computed confidence intervals contain the true value. The procedure we used to compute the confidence intervals is, in fact, correct: our data just happened to be among the 5% where the method breaks down. But here's the thing: we know from the data themselves that we are in the 5% where the CI fails. The fact that the standard frequentist confidence interval ignores this common-sense information should give you pause about blind reliance on the confidence interval for any nontrivial problem.
For good measure, let's check that the Bayesian credible region also passes its test:
Can you write Python code for it?

np.random.seed(42)
N = int(1E7)
eps = 0.1

theta = 9 + 2 * np.random.random(N)
data = (theta + expon().rvs((3, N))).T
data.sort(1)
D.sort()
i_good = np.all(abs(data - D) < eps, 1)

print("Number of good samples: {0}".format(i_good.sum()))

theta_good = theta[i_good]
theta1, theta2 = bayes_CR(D)

within_CR = (theta1 < theta_good) & (theta_good < theta2)
print("Fraction of thetas in Credible Region: {0:.3f}".format(within_CR.sum() * 1. / within_CR.size))


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
Next let's create a function which will compute the confidence interval:
Can you write Python code for it?

from scipy.special import erfinv

def freq_CI_mu(D, sigma, frac=0.95):
    """Compute the confidence interval on the mean"""
    # we'll compute Nsigma from the desired percentage
    Nsigma = np.sqrt(2) * erfinv(frac)
    mu = D.mean()
    sigma_mu = sigma * D.size ** -0.5
    return mu - Nsigma * sigma_mu, mu + Nsigma * sigma_mu

print("95% Confidence Interval: [{0:.0f}, {1:.0f}]".format(*freq_CI_mu(D, 10)))


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
Note here that we've assumed $\sigma_x$ is a known quantity; this could also be estimated from the data along with $\mu$, but here we kept things simple for sake of example.
2. The Bayesian Approach
For the Bayesian approach, we start with Bayes' theorem:
$$
P(\mu~|~D) = \frac{P(D~|~\mu)P(\mu)}{P(D)}
$$
We'll use a flat prior on $\mu$ (i.e. $P(\mu) \propto 1$ over the region of interest) and use the likelihood
$$
P(D~|~\mu) = \prod_{i=1}^N \frac{1}{\sqrt{2\pi\sigma_x^2}}\exp\left[\frac{(\mu - x_i)^2}{2\sigma_x^2}\right]
$$
Computing this product and manipulating the terms, it's straightforward to show that this gives
$$
P(\mu~|~D) \propto \exp\left[\frac{-(\mu - \bar{x})^2}{2\sigma_\mu^2}\right]
$$
which is recognizable as a normal distribution with mean $\bar{x}$ and standard deviation $\sigma_\mu$.
That is, the Bayesian posterior on $\mu$ in this case is exactly equal to the frequentist sampling distribution for $\mu$.
From this posterior, we can compute the Bayesian credible region, which is the shortest interval that contains 95% of the probability. Here, it looks exactly like the frequentist confidence interval:
$$
CR_{\mu} =  \left(\bar{x} - 2\sigma_\mu,~\bar{x} + 2\sigma_\mu\right)
$$
For completeness, we'll also create a function to compute the Bayesian credible region:
Can you write Python code for it?

def bayes_CR_mu(D, sigma, frac=0.95):
    """Compute the credible region on the mean"""
    Nsigma = np.sqrt(2) * erfinv(frac)
    mu = D.mean()
    sigma_mu = sigma * D.size ** -0.5
    return mu - Nsigma * sigma_mu, mu + Nsigma * sigma_mu

print("95% Credible Region: [{0:.0f}, {1:.0f}]".format(*bayes_CR_mu(D, 10)))


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
So What's the Difference?
The above derivation is one reason why the frequentist confidence interval and the Bayesian credible region are so often confused. In many simple problems, they correspond exactly. But we must be clear that even though the two are numerically equivalent, their interpretation is very different.
Recall that in Bayesianism, the probability distributions reflect our degree of belief. So when we computed the credible region above, it's equivalent to saying

"Given our observed data, there is a 95% probability that the true value of $\mu$ falls within $CR_\mu$" - Bayesians

In frequentism, on the other hand, $\mu$ is considered a fixed value and the data (and all quantities derived from the data, including the bounds of the confidence interval) are random variables. So the frequentist confidence interval is equivalent to saying

"There is a 95% probability that when I compute $CI_\mu$ from data of this sort, the true mean will fall within $CI_\mu$." - Frequentists

Note the difference: the Bayesian solution is a statement of probability about the parameter value given fixed bounds. The frequentist solution is a probability about the bounds given a fixed parameter value. This follows directly from the philosophical definitions of probability that the two approaches are based on.
The difference is subtle, but, as I'll discuss below, it has drastic consequences. First, let's further clarify these notions by running some simulations to confirm the interpretation.
Confirming the Bayesian Credible Region
To confirm what the Bayesian credible region is claiming, we must do the following:

sample random $\mu$ values from the prior
sample random sets of points given each $\mu$
select the sets of points which match our observed data
ask what fraction of these $\mu$ values are within the credible region we've constructed.

In code, that looks like this:
Can you write Python code for it?

# first define some quantities that we need
Nsamples = int(2E7)
N = len(D)
sigma_x = 10

# if someone changes N, this could easily cause a memory error
if N * Nsamples > 1E8:
    raise ValueError("Are you sure you want this many samples?")

# eps tells us how close to D we need to be to consider
# it a matching sample. The value encodes the tradeoff
# between bias and variance of our simulation
eps = 0.5

# Generate some mean values from the (flat) prior in a reasonable range
np.random.seed(0)
mu = 80 + 40 * np.random.random(Nsamples)

# Generate data for each of these mean values
x = np.random.normal(mu, sigma_x, (N, Nsamples)).T

# find data which matches matches our "observed" data
x.sort(1)
D.sort()
i = np.all(abs(x - D) < eps, 1)
print("number of suitable samples: {0}".format(i.sum()))

# Now we ask how many of these mu values fall in our credible region
mu_good = mu[i]
CR = bayes_CR_mu(D, 10)
within_CR = (CR[0] < mu_good) & (mu_good < CR[1])
print "Fraction of means in Credible Region: {0:.3f}".format(within_CR.sum() * 1. / within_CR.size)


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
We immediately see an issue. By our simple common sense argument, we've determined that it is impossible for $\theta$ to be greater than 10, yet the entirety of the 95% confidence interval is above this range!  Perhaps this issue is due to the small sample size: the above computation is based on a large-$N$ approximation, and we have a relatively paltry $N = 3$.
Maybe this will be improved if we do the more computationally intensive exact approach?
The answer is no. If we compute the confidence interval without relying on large-$N$ Gaussian eapproximation, the result is $(10.2, 12.2)$.
Note: you can verify yourself by evaluating the code in the sub-slides.
3. Frequentist approach #2: Exact Sampling Distribution
Computing the confidence interval from the exact sampling distribution takes a bit more work.
For small $N$, the normal approximation will not apply, and we must instead compute the confidence integral from the actual sampling distribution, which is the distribution of the mean of $N$ variables each distributed according to $p(\theta)$. The sum of random variables is distributed according to the convolution of the distributions for individual variables, so we can exploit the convolution theorem and use the method of characteristic functions to find the following sampling distribution for the sum of $N$ variables distributed according to our particular $p(x~|~\theta)$:
$$
f(\theta~|~D) \propto
\left{
\begin{array}{lll}
z^{N - 1}\exp(-z) &,& z > 0\
0 &,& z < 0
\end{array}
\right}
;~ z = N(\hat{\theta} + 1 - \theta)
$$
To compute the 95% confidence interval, we can start by computing the cumulative distribution: we integrate $f(\theta~|~D)$ from $0$ to $\theta$ (note that we are not actually integrating over the parameter $\theta$, but over the estimate of $\theta$. Frequentists cannot integrate over parameters).
This integral is relatively painless if we make use of the expression for the incomplete gamma function:
$$
\Gamma(a, x) = \int_x^\infty t^{a - 1}e^{-t} dt
$$
which looks strikingly similar to...
Can you write Python code for it?

from scipy.special import gammaincc
from scipy import optimize

def exact_CI(D, frac=0.95):
    """Exact truncated exponential confidence interval"""
    D = np.asarray(D)
    N = D.size
    theta_hat = np.mean(D) - 1

    def f(theta, D):
        z = theta_hat + 1 - theta
        return (z > 0) * z ** (N - 1) * np.exp(-N * z)

    def F(theta, D):
        return gammaincc(N, np.maximum(0, N * (theta_hat + 1 - theta))) - gammaincc(N, N * (theta_hat + 1))

    def eqns(CI, D):
        """Equations which should be equal to zero"""
        theta1, theta2 = CI
        return (F(theta2, D) - F(theta1, D) - frac,
                f(theta2, D) - f(theta1, D))

    guess = approx_CI(D, 0.68) # use 1-sigma interval as a guess
    result = optimize.root(eqns, guess, args=(D,))
    if not result.success:
        print "warning: CI result did not converge!"
    return result.x


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
We see that, as predicted, 95% of the confidence intervals contain the true value of $\mu$.
Again, the important thing to note here is which of the variables is random. We use a single value of $\mu$, and consider it in relation to multiple confidence intervals constructed from multiple random data samples.
Discussion
We should remind ourselves again of the difference between the two types of constraints:

The Bayesian approach fixes the credible region, and guarantees 95% of possible values of $\mu$ will fall within it.
The frequentist approach fixes the parameter, and guarantees that 95% of possible confidence intervals will contain it.

Comparing the frequentist confirmation and the Bayesian confirmation above, we see that the distinctions which stem from the very definition of probability mentioned above:

Bayesianism treats parameters (e.g. $\mu$) as random variables, while frequentism treats parameters as fixed.
Bayesianism treats observed data (e.g. $D$) as fixed, while frequentism treats data as random variables.
Bayesianism treats its parameter constraints (e.g. $CR_\mu$) as fixed, while frequentism treats its constraints (e.g. $CI_\mu$) as random variables.

In the above example, as in many simple problems, the confidence interval and the credibility region overlap exactly, so the distinction is not especially important. But scientific analysis is rarely this simple; next we'll consider an example in which the choice of approach makes a big difference.
Example 2: Jaynes' Truncated Exponential
For an example of a situation in which the frequentist confidence interval and the Bayesian credibility region do not overlap, I'm going to turn to an example given by E.T. Jaynes, a 20th century physicist who wrote extensively on statistical inference in Physics.  In the fifth example of his Confidence Intervals vs. Bayesian Intervals (pdf), he considers a truncated exponential model. Here is the problem, in his words:

A device will operate without failure for a time $\theta$ because of a protective chemical...
Can you write Python code for it?

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

def p(x, theta):
    return (x > theta) * np.exp(theta - x)

x = np.linspace(5, 18, 1000)
plt.fill(x, p(x, 10), alpha=0.3)
plt.ylim(0, 1.2)
plt.xlabel('x')
plt.ylabel('p(x)');


I am working on the file "Sessions/Session04/Day1/LSSTC-DSFP4-Juric-FrequentistAndBayes-03-Credibility.ipynb".
The first task is:
We see that, as predicted, roughly 95% of $\mu$ values with data matching ours lie in the Credible Region.
The important thing to note here is which of the variables is random, and which are fixed. In the Bayesian approach, we compute a single credible region from our observed data, and we consider it in terms of multiple random draws of $\mu$.
Confirming the frequentist Confidence Interval
Confirmation of the interpretation of the frequentist confidence interval is a bit less involved. We do the following:

draw sets of values from the distribution defined by the single true value of $\mu$.
for each set of values, compute a new confidence interval.
determine what fraction of these confidence intervals contain $\mu$.

In code, it looks like this:
Can you write Python code for it?

# define some quantities we need
N = len(D)
Nsamples = int(1E4)
mu = 100
sigma_x = 10

# Draw datasets from the true distribution
np.random.seed(0)
x = np.random.normal(mu, sigma_x, (Nsamples, N))

# Compute a confidence interval from each dataset
CIs = np.array([freq_CI_mu(Di, sigma_x) for Di in x])

# find which confidence intervals contain the mean
contains_mu = (CIs[:, 0] < mu) & (mu < CIs[:, 1])
print "Fraction of Confidence Intervals containing the mean: {0:.3f}".format(contains_mu.sum() * 1. / contains_mu.size)


I am working on the file "Sessions/Session04/Day1/StatisticsAperitif.ipynb".
The first task is:
By eye (a metric that is hardly quantitative, but nevertheless worth developing because talks never provide all of the details), the quadratic fit appears "better" than the linear fit.
But, there are still "outliers" and in the realm of polynomial fitting, it is always possible to get a better fit by adding more degrees to the polynomial. Should we keep going here, or should we stop? (Again - we will discuss model selection with Adam on Friday)
[As a reminder - in machine learning we'd call this low training error, but the generalization error is likely huge]
How should we deal with these potential outliers?
(and to re-iterate, we cannot be certain that these points are, in fact, outliers)
Amazingly, if you scroll through the literature you can find solutions like the following: "We do not believe the data point at (x, y) for reasons A, B, C. Thus, we exclude that point from the fit."
This, obviously, lacks any sort of rigor. If data is going to be removed, and it is worth asking if data should ever be removed, it should not be subject to the "beliefs" of an individual person or group.
A more common approach that you might encounter is known as $k\sigma$ clipping, which is an iterative procedure to identify and remove outliers from a data set. The procedure is as follows:

Fit the model to the data
Identify any data points that are $k\sigma$ discrepant from the best-fit model
Remove the discrepant points, repeat steps 1 & 2 until there are no data beyond $k\sigma$

The motivation for this procedure is the following: in a small data set (such as the one above, 20 points), the likelihood of having large $\sigma$ deviations (let's say $k = 5$) is vanishingly small. Thus, it "makes sense" to remove those points.
Of course, this only makes sense if the uncertainties are truly gaussian with low variance. So again, specific, and likely untrue, assumptions have to be made.
Problem 3c
Develop a $k\sigma$ clipping procedure to fit a line to the data set. Set $k = 5$ and determine the best-fit line to the data....
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day1/StatisticsAperitif.ipynb".
The first task is:
By eye, the results above are not that satisfying. Several of those points do look like outliers, but there are also 2 points being rejected that are well within the other cluster of data.
Furthermore, the point at $(x, y) \approx (60, 170)$ was clipped in an early iteration of the algorithm, but now with the final model this point is actually within $k\sigma$ of the best-fit line. What should one do with points like this?
This is one of the great problems with $k\sigma$ clipping: how does one select the appropriate value for $k$?
Even if there was a good heuristic argument for selecting $k$, is there fundamentally any difference between an observation that is $k\sigma + \epsilon$ away from the model versus a point at $k\sigma - \epsilon$, where $\epsilon \ll 1$? Any choice of $k$ will automatically remove one of these points and not the other, which seems somewhat arbitrary...
Problem 3d
Perform the $k\sigma$ procedure on the data, but this time set $k = 7$. Plot the results as above.
How do these results compare to our previous fits?
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day1/StatisticsAperitif.ipynb".
The first task is:
By eye, this appears superior to the previous fit. At the same time, we have not actually optimized anything to definitively show that this is the case.
If there are outliers in the data, then it stands to reason that the uncertainties are likely not correctly estimated.
Problem 3e
Perform the $k\sigma$ procedure on the data, with $k = 5$, but the variance increased by a factor 4 (i.e. sigma_y increased by a factor of 2). Plot the results as above.
How do these results compare to our previous fits?
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day1/StatisticsAperitif.ipynb".
The first task is:
Introduction to Statistics:
An Aperitif for DSFP Session 4
Version 0.1

By AA Miller 2017 Sep 14
An Introduction to Statistics was covered during Session 1 of the DSFP. Typically, this initial lecture is used as a means of providing a brief overview of the Session 1 material, but that does not make sense in the context of Session 4, as half of you have not seen this lecture.
Instead, today we will focus on a relatively simple problem, while highlighting several challenges for the standard astronomical workflow, as a way of setting up the various lectures that will happen over the course of this week.
A lot of the lessons in this lecture are inspired by the paper Data Analysis Recipes: Fitting a Model to Data by Hogg, Bovy, & Lang. [This paper has been mentioned previously in the DSFP, though today we will only be able to scratch the surface of its content.]
In some sense - the goal right now is to make you really nervous about the work that you've previously done.
(Though this lecture should not be met with too much consternation, as you will have a toolkit to deal with all the issues that we raise by the end of the week)
Problem 1) Data
At the core of everything we hope to accomplish with the DSFP stands a single common connection: data.
There are many things we (may) want to do with these data: reduce them, visualize them, model them, develop predictions from them, use them to infer fundamental properties of the universe (!).
Before we dive into that really fun stuff, we should start with some basics:
Problem 1a
What is data?
Take a few min to discuss this with your partner
Solution 1a
write your answer here
Problem 1b
Below, I provide some data (in the form of numpy arrays). As good data scientists, what is the first thing you should do with this data?
Feel free to create a new cell if necessary.
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day1/StatisticsAperitif.ipynb".
The first task is:
Probelm 2) Fitting a Line to Data
There is a very good chance, though I am not specifically assuming anything, that upon making the previous plot you had a thought along the lines of "these points fall on a line" or "these data represent a linear relationship."
Problem 2a
Is the assumption of linearity valid for the above data?
Is it convenient?
Take a few min to discuss this with your partner
Solution 2a
write your answer here
Let us proceed with convenience and assume the data represent a linear relationship. In that case, in order to make predictions for future observations, we need to fit a line to the data.
The "standard" proceedure for doing so is least-squares fitting. In brief, least-squares minimizes the sum of the squared value of the residuals between the data and the fitting function.
I've often joked that all you need to be a good data scientist is google and stack overflow. Via those two tools, we can quickly deduce that the easiest way to perform a linear least-squares fit to the above data is with np.polyfit, which performs a least-squares polynomial fit to two numpy arrays.
Problem 2b
Use np.polyfit() to fit a line to the data. Overplot the best-fit line on the data.
Can you write Python code for it?

p = np.polyfit( # complete

plt.scatter( # complete
# complete


I am working on the file "Sessions/Session04/Day1/StatisticsAperitif.ipynb".
The first task is:
Problem 1c
[You may have already done this] Now that we understand the origin of the data, make a scatter plot showing their distribution.
Can you write Python code for it?

plt.scatter( # complete
# complete
# complete


I am working on the file "Sessions/Session04/Day1/StatisticsAperitif.ipynb".
The first task is:
Problem 2f
Confirm the results of this fit are the same as those from np.polyfit.
Hint - be sure to include the uncertainties.
Can you write Python code for it?

p = np.polyfit( # complete
print("The best-fit value for the slope and intercept are: {:.4f} and {:.4f}".format( # complete


I am working on the file "Sessions/Session04/Day1/StatisticsAperitif.ipynb".
The first task is:
Problem 3a
Using the least-squares methodology developed in Problem 2, determine the best-fit slope and intercept for a line fit to the data above.
Make a scatter plot of the data, and overplot the best-fit line. What if anything, do you notice about the data and the fit?
Can you write Python code for it?

Y = # complete
# complete

# complete

plt.errorbar( # complete
plt.plot( # complete

print("The best-fit value for the slope and intercept are: {:.4f} and {:.4f}".format( # complete


I am working on the file "Sessions/Session04/Day1/StatisticsAperitif.ipynb".
The first task is:
So we have now uncovered one of the peculiariaties of least-squares. Fitting y vs. x is not the same as fitting x vs. y.
There are a couple essential assumptions that go into standard least-squares fitting:

There is one dimension along which the data have negligible uncertainties
Along the other dimension all of the uncertainties can be described via Gaussians of known variance

These two conditions are rarely met for astronomical data. While condition 1 can be satisfied (e.g., time series data where there is essentially no uncertainty on the time of the observations), I contend that condition 2 is rarely, if ever, satisfied.
Speaking of uncertainties(1), we have not utilized any thus far. [I hope this has raised some warning bells.]
We will now re-organize our data to match what is originally in Hogg, Bovy, & Lang (previously x and y were swapped).
(1) There is an amazing footnote in Hogg, Bovy, & Lang about "errors" vs. "uncertainties" - I suggest everyone read this.
Problem 2d
Re-plot the data including the uncertatines.
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day1/StatisticsAperitif.ipynb".
The first task is:
There is a very good chance, though, again, I am not specifically assuming anything, that for the previous plots that you plotted x along the abscissa and y along the ordinate.
[Honestly, there's no one to blame if this is the case, this has essentially been drilled into all of us from the moment we started making plots. In fact, in matplotlib we cannot change the name of the abscissa label without adjusting the xlabel.]
This leads us to an important question, however. What if y does not depend on x and instead x depends on y? Does that in any way change the results for the fit?
Problem 2c
Perform a linear least-squares fit to x vs. y (or if you already fit this, then reverse the axes). As above, plot the data and the best-fit model.
To test if the relation is the same between the two fits, compare the predicted y value for both models corresponding to x = 300.
Can you write Python code for it?

p_yx = np.polyfit( # complete

plt.scatter( # complete
# complete

print("For y vs. x, then x=50 would predict y={:.2f}".format( # complete
print("For x vs. y, then x=50 would predict y={:.2f}".format( # complete


I am working on the file "Sessions/Session04/Day1/StatisticsAperitif.ipynb".
The first task is:
We are now assuming that x has negligible uncertainties and that y has uncertainties that can be perfectly described by Gaussians of known variance.
A portion of the appeal of least-squares is that it provides a deterministic method for determining the best fit. To understand that we now need to do a little linear algebra.
We can arrange the data in the following matricies:
$$ \mathbf{Y} = \left[ {\begin{array}{c}
            y_1 \
            y_2 \
            \dots \
            y_N
            \end{array}
           }
            \right] , $$
$$ \mathbf{A} = \left[ {\begin{array}{cc}
            1 & x_1 \
            1 & x_2 \
            \dots & \dots \
            1 & x_N
            \end{array}
           }
           \right] ,
           $$
$$ \mathbf{C} = \left[ {\begin{array}{cccc}
            \sigma_{y_1}^2 & 0 & \dots & 0 \
            0 & \sigma_{y_2}^2 & \dots & 0 \
            \vdots & \vdots & \ddots & \vdots \
            0 & 0 & \dots & \sigma_{y_1}^2 \
            \end{array}
           }
           \right] ,
           $$
where $\mathbf{Y}$ is a vector, and $\mathbf{C}$ is the covariance matrix.
Ultimately, we need to solve the equation
$$\mathbf{Y} = \mathbf{A}\mathbf{X}.$$
I am skipping the derivation, but the solution to this equations is:
$$ \left[ {\begin{array}{c}
            b \
            m \
            \end{array}
           }
            \right] = \mathbf{X} = \left[ \mathbf{A}^T \mathbf{C}^{-1} \mathbf{A}\right]^{-1} \left[ \mathbf{A}^T \mathbf{C}^{-1} \mathbf{Y}\right].$$
As noted in Hogg, Bovy, & Lang, this procedure minimizes the $\chi^2$ function, which is the total squared error, after appropriately scaling by the uncertainties:
$$ \chi^2 = \Sigma_{i = 1}^{N} \frac{[y_i - f(x_i)]^2}{\sigma_{y_i}^2} = \left[ \mathbf{Y}  - \mathbf{A}\mathbf{X}\right]^{T} \mathbf{C}^{-1} \left[ \mathbf{Y} - \mathbf{A} \mathbf{X}\right].$$
Problem 2e
Using the linear algebra equations above (i.e. avoid np.polyfit or any other similar functions), determine the weighted least-squares best-fit...
Can you write Python code for it?

Y = # complete
A = # complete
C = # complete

X = # complete

best_fit = # complete

plt.errorbar( # complete
plt.plot( # complete

print("The best-fit value for the slope and intercept are: {:.4f} and {:.4f}".format( # complete


I am working on the file "Sessions/Session04/Day2/GPLecture1.ipynb".
The first task is:
Combining kernels
Any affine tranform, sum or product of valid kernels is a valid kernel.
For example, a quasi-periodic kernel can be constructed by multiplying a periodic kernel with a non-periodic one. The following is frequently used to model stellar light curves:
$$
k_{\mathrm{QP}}(x,x') = A \exp \left[  -\Gamma_1 \sin^2\left(\frac{\pi r}{P}\right)   -\Gamma_2 r^2 \right].
$$
Example: Mauna Kea CO$_2$ dataset
(From Rasmussen & Williams textbook)
<img height="700" src="images/RW_mauna_kea.png">
2 or more dimensions
So far we assumed the inputs were 1-D but that doesn't have to be the case. For example, the SE kernel can be extended to D dimensions...
using a single length scale, giving the Radial Basis Function (RBF) kernel:
$$
k_{\rm RBF}(\mathbf{x},\mathbf{x'}) = A \exp \left[ - \Gamma \sum_{j=1}^{D}(x_j-x'_j)^2 \right],
$$
where $\mathbf{x}=(x_1,x_2,\ldots, x_j,\ldots,x_D)^{\mathrm{T}}$ represents a single, multi-dimensional input.
or using separate length scales for each dimension, giving the Automatic Relevance Determination (ARD) kernel:
$$
k_{\rm ARD}(\mathbf{x},\mathbf{x'}) = A \exp \left[ - \sum_{j=1}^{D} \Gamma_j (x_j-x'_j)^2 \right].
$$
Can you write Python code for it?

import george
x2d, y2d = np.mgrid[-3:3:0.1,-3:3:0.1]
x = x2d.ravel()
y = y2d.ravel()
N = len(x)
X = np.zeros((N,2))
X[:,0] = x
X[:,1] = y

k1 = george.kernels.ExpSquaredKernel(1.0,ndim=2)
s1 = george.GP(k1).sample(X).reshape(x2d.shape)

k2 = george.kernels.ExpSquaredKernel(1.0,ndim=2,axes=1) + george.kernels.ExpSquaredKernel(0.2,ndim=2,axes=0)
s2 = george.GP(k2).sample(X).reshape(x2d.shape)

pl.figure(figsize=(10,5))
pl.subplot(121)
pl.contourf(x2d,y2d,s1)
pl.xlim(x.min(),x.max())
pl.ylim(y.min(),y.max())
pl.xlabel(r'$x$')
pl.ylabel(r'$y$')
pl.title('RBF')
pl.subplot(122)
pl.contourf(x2d,y2d,s2)
pl.xlim(x.min(),x.max())
pl.ylim(y.min(),y.max())
pl.xlabel(r'$x$')
pl.title('ARD');

# Function to plot samples from kernel
def pltsamples3(par2=0.5,par3=0.5, kernel_shortname='SE'):
    x = np.r_[-5:5:201j]
    X = np.matrix([x]).T # scipy.spatial.distance expects matrices
    kernel = get_kernel(kernel_shortname)
    K = kernel(X,X,[0.0,0.0,par2,par3])
    fig=pl.figure(figsize=(10,4))
    ax1 = pl.subplot2grid((1,3), (0, 0), aspect='equal')
    pl.imshow(np.sqrt(K),interpolation='nearest',vmin=0,vmax=10)
    pl.title('Covariance matrix')
    ax2 = pl.subplot2grid((1,3), (0,1),colspan=2)
    np.random.seed(0)
    for i in range(5):
        y = np.random.multivariate_normal(np.zeros(len(x)),K)
        pl.plot(x,y)
    pl.xlim(-5,5)
    pl.ylim(-5,5)
    pl.xlabel('x')
    pl.ylabel('y')
    pl.title('Samples from %s prior' % kernel_shortname)
    pl.tight_layout()
interact(pltsamples3,
         par2=widgets.FloatSlider(min=1,max=3,step=1,description=r'$P$',value=2),
         par3=widgets.FloatSlider(min=-2,max=0,step=1,description=r'$\log_{10}\Gamma_2$',value=-1.),
         kernel_shortname=widgets.RadioButtons(options=['Per','QP'], value='Per',description='kernel')
        );


I am working on the file "Sessions/Session04/Day2/GPLecture1.ipynb".
The first task is:
Gaussian process regression
Lecture 1
Suzanne Aigrain, University of Oxford
LSST DSFP Session 4, Seattle, Sept 2017

Lecture 1: Introduction and basics

Tutorial 1: Write your own GP code

Lecture 2: Examples and practical considerations

Tutorial 3: Useful GP modules

Lecture 3: Advanced applications

Why GPs?

flexible, robust probabilistic regression and classification tools.

applied across a wide range of fields, from finance to zoology.

useful for data containing non-trivial stochastic signals or noise.

time-series data: causation implies correlation, so noise always correlated.

increasingly popular in astronomy [mainly time-domain, but not just].

Spitzer exoplanet transits and eclipses (Evans et al. 2015)

GPz photometric redshifts (Almosallam, Jarvis & Roberts 2016)

What is a GP?
A Gaussian process is a collection of random variables, any
finite number of which have a joint Gaussian distribution.
Consider a scalar variable $y$, drawn from a Gaussian distribution with mean $\mu$ and variance $\sigma^2$:
$$
p(y) = \frac{1}{\sqrt{2 \pi} \sigma} \exp \left[ - \frac{(y-\mu)^2}{2 \sigma^2} \right].
$$
As a short hand, we write: $y \sim \mathcal{N}(\mu,\sigma^2)$.
Can you write Python code for it?

def gauss1d(x,mu,sig):
    return np.exp(-(x-mu)**2/sig*2/2.)/np.sqrt(2*np.pi)/sig
def pltgauss1d(sig=1):
    mu=0
    x = np.r_[-4:4:101j]
    pl.figure(figsize=(10,7))
    pl.plot(x, gauss1d(x,mu,sig),'k-');
    pl.axvline(mu,c='k',ls='-');
    pl.axvline(mu+sig,c='k',ls='--');
    pl.axvline(mu-sig,c='k',ls='--');
    pl.axvline(mu+2*sig,c='k',ls=':');
    pl.axvline(mu-2*sig,c='k',ls=':');
    pl.xlim(x.min(),x.max());
    pl.ylim(0,1);
    pl.xlabel(r'$y$');
    pl.ylabel(r'$p(y)$');
    return
interact(pltgauss1d,
         sig=widgets.FloatSlider(value=1.0,
                                 min=0.5,
                                 max=2.0,
                                 step=0.25,
                                 description=r'$\sigma$',
                                 readout_format='.2f'));


I am working on the file "Sessions/Session04/Day2/GPLecture1.ipynb".
The first task is:
Now let us consider a pair of variables $y_1$ and $y_2$, drawn from a bivariate Gaussian distribution. The joint probability density for $y_1$ and $y_2$ is:
$$
\left[ \begin{array}{l} y_1 \ y_2 \end{array} \right] \sim \mathcal{N} \left(
\left[ \begin{array}{l} \mu_1 \ \mu_2 \end{array}  \right] ,
\left[ \begin{array}{ll}
\sigma_1^2 & C \
C & \sigma_2^2
\end{array}  \right]
\right),
$$
where $C = {\rm cov}(y_1,y_2)$ is the covariance between $y_1$ and $y_2$.
The second term on the right hand side is the covariance matrix, $K$.
We now use two powerful identities of Gaussian distributions to elucidate the relationship between $y_1$ and $y_2$.
The marginal distribution of $y_1$ describes what we know about $y_1$ in the absence of any other information about $y_2$, and is simply:
$$
p(y_1)= \mathcal{N} (\mu_1,\sigma_1^2).
$$
If we know the value of $y_2$, the probability density for $y_1$ collapses to the the conditional distribution of $y_1$ given $y_2$:
$$
p(y_1 \mid y_2) = \mathcal{N} \left( \mu_1 + C (y_2-\mu_2)/\sigma_2^2, \sigma_1^2-C^2\sigma_2^2 \right).
$$
If $K$ is diagonal, i.e. if $C=0$, $p(y_1 \mid y_2) = p(y_1)$. Measuring $y_2$ doesn't teach us anything about $y_1$. The two variables are uncorrelated.
If the variables are correlated ($C \neq 0$), measuring $y_2$ does alter our knowledge of $y_1$: it modifies the mean and reduces the variance.
Can you write Python code for it?

def gauss2d(x1,x2,mu1,mu2,sig1,sig2,rho):
    z = (x1-mu1)**2/sig1**2 + (x2-mu2)**2/sig2**2 - 2*rho*(x1-mu1)*(x2-mu2)/sig1/sig2
    e = np.exp(-z/2/(1-rho**2))
    return e/(2*np.pi*sig1*sig2*np.sqrt(1-rho**2))
def pltgauss2d(rho=0,show_cond=0):
    mu1, sig1 = 0,1
    mu2, sig2 = 0,1
    y2o = -1
    x1 = np.r_[-4:4:101j]
    x2 = np.r_[-4:4:101j]
    x22d,x12d = np.mgrid[-4:4:101j,-4:4:101j]
    y = gauss2d(x12d,x22d,mu1,mu2,sig1,sig2,rho)
    y1 = gauss1d(x1,mu1,sig1)
    y2 = gauss1d(x2,mu2,sig2)
    mu12 = mu1+rho*(y2o-mu2)/sig2**2
    sig12 = np.sqrt(sig1**2-rho**2*sig2**2)
    y12 = gauss1d(x1,mu12,sig12)
    pl.figure(figsize=(10,10))
    ax1 = pl.subplot2grid((3,3),(1,0),colspan=2,rowspan=2,aspect='equal')
    v = np.array([0.02,0.1,0.3,0.6]) * y.max()
    CS = pl.contour(x1,x2,y,v,colors='k')
    if show_cond: pl.axhline(y2o,c='r')
    pl.xlabel(r'$y_1$');
    pl.ylabel(r'$y_2$');
    pl.xlim(x1.min(),x1.max())
    ax1.xaxis.set_major_locator(pl.MaxNLocator(5, prune = 'both'))
    ax1.yaxis.set_major_locator(pl.MaxNLocator(5, prune = 'both'))
    ax2 = pl.subplot2grid((3,3),(0,0),colspan=2,sharex=ax1)
    pl.plot(x1,y1,'k-')
    if show_cond: pl.plot(x1,y12,'r-')
    pl.ylim(0,0.8)
    pl.ylabel(r'$p(y_1)$')
    pl.setp(ax2.get_xticklabels(), visible=False)
    ax2.xaxis.set_major_locator(pl.MaxNLocator(5, prune = 'both'))
    ax2.yaxis.set_major_locator(pl.MaxNLocator(4, prune = 'upper'))
    pl.xlim(x1.min(),x1.max())
    ax3 = pl.subplot2grid((3,3),(1,2),rowspan=2,sharey=ax1)
    pl.plot(y2,x2,'k-')
    if show_cond: pl.axhline(y2o,c='r')
    pl.ylim(x2.min(),x2.max());
    pl.xlim(0,0.8);
    pl.xlabel(r'$p(y_2)$')
    pl.setp(ax3.get_yticklabels(), visible=False)
    ax3.xaxis.set_major_locator(pl.MaxNLocator(4, prune = 'upper'))
    ax3.yaxis.set_major_locator(pl.MaxNLocator(5, prune = 'both'))
    pl.subplots_adjust(hspace=0,wspace=0)
    return
interact(pltgauss2d,
         rho=widgets.FloatSlider(min=-0.8,max=0.8,step=0.4,description=r'$\rho$',value=0),
        ...


I am working on the file "Sessions/Session04/Day2/GPLecture1.ipynb".
The first task is:
Periodic kernels...
...can be constructed by replacing $r$ in any of the above by a periodic function of $r$. For example, the cosine kernel:
$$
k_{\cos}(x,x') = A \cos\left(\frac{2\pi r}{P}\right),
$$
[which follows the dynamics of a simple harmonic oscillator], or...
...the "exponential sine squared" kernel, obtained by mapping the 1-D variable $x$ to the 2-D variable $\mathbf{u}(x)=(\cos(x),\sin(x))$, and then applying a squared exponential in $\boldsymbol{u}$-space:
$$
k_{\sin^2 {\rm SE}}(x,x') = A \exp \left[ -\Gamma \sin^2\left(\frac{\pi r}{P}\right) \right],
$$
which allows for non-harmonic functions.
Can you write Python code for it?

# Function to plot samples from kernel
def pltsamples3(par2=2.0, par3=2.0,kernel_shortname='Per'):
    x = np.r_[-5:5:201j]
    X = np.matrix([x]).T # scipy.spatial.distance expects matrices
    kernel = get_kernel(kernel_shortname)
    K = kernel(X,X,[0.0,0.0,par2,par3])
    fig=pl.figure(figsize=(10,4))
    ax1 = pl.subplot2grid((1,3), (0, 0), aspect='equal')
    pl.imshow(np.sqrt(K),interpolation='nearest',vmin=0,vmax=10)
    pl.title('Covariance matrix')
    ax2 = pl.subplot2grid((1,3), (0,1),colspan=2)
    np.random.seed(0)
    for i in range(3):
        y = np.random.multivariate_normal(np.zeros(len(x)),K)
        pl.plot(x,y-i*2)
    pl.xlim(-5,5)
    pl.ylim(-8,5)
    pl.xlabel('x')
    pl.ylabel('y')
    pl.title('Samples from %s prior' % kernel_shortname)
    pl.tight_layout()
interact(pltsamples3,
         par2=widgets.FloatSlider(min=1,max=3,step=1,description=r'$P$',value=2),
         par3=widgets.FloatSlider(min=-2,max=0,step=1,description=r'$\log\Gamma_2$',value=-1),
         kernel_shortname=widgets.RadioButtons(options=['Per','QP'], value='QP',description='kernel')
        );


I am working on the file "Sessions/Session04/Day2/GPLecture1.ipynb".
The first task is:
Textbooks
A good, detailed reference is Gaussian Processes for Machine Learning by C. E. Rasmussen & C. Williams, MIT Press, 2006.
The examples in the book are generated using the Matlab package GPML.
A more formal definition
A Gaussian process is completely specified by its mean function and covariance function.
We define the mean function $m(x)$ and the covariance function $k(x,x)$ of a real process $y(x)$ as
$$
\begin{array}{rcl}
m(x) & = & \mathbb{E}[y(x)], \
k(x,x') & = & \mathrm{cov}(y(x),y(x'))=\mathbb{E}[(y(x) − m(x))(y(x') − m(x'))].
\end{array}
$$
A very common covariance function is the squared exponential, or radial basis function (RBF) kernel
$$
K_{ij}=k(x_i,x_j)=A \exp\left[ - \Gamma (x_i-x_j)^2 \right],
$$
which has 2 parameters: $A$ and $\Gamma$.
We then write the Gaussian process as
$$
y(x) \sim \mathcal{GP}(m(x), k(x,x'))
$$
Here we are implicitly assuming the inputs $x$ are one-dimensional, e.g. $x$ might represent time. However, the input space can have more than one dimension. We will see an example of a GP with multi-dimensional inputs later.
The prior
Now consider a finite set of inputs $\boldsymbol{x}$, with corresponding outputs $\boldsymbol{y}$.
The joint distribution of $\boldsymbol{y}$ given $\boldsymbol{x}$, $m$ and $k$ is
$$
\mathrm{p}(\boldsymbol{y} \mid \boldsymbol{x},m,k) = \mathcal{N}( \boldsymbol{m},K),
$$
where $\boldsymbol{m}=m(\boldsymbol{x})$ is the mean vector,
and $K$ is the covariance matrix, with elements $K_{ij} = k(x_i,x_j)$.
Test and training sets
Suppose we have an (observed) training set $(\boldsymbol{x},\boldsymbol{y})$.
We are interested in some other test set of inputs $\boldsymbol{x}_*$.
The joint distribution over the training and test sets is
$$
\mathrm{p} \left( \left[ \begin{array}{l} \boldsymbol{y} \ \boldsymbol{y} \end{array} \right] \right)
= \mathcal{N} \left( \left[ \begin{array}{l} \boldsymbol{m} \ \boldsymbol{m}_ \end{array} \right],
\left[ \begin{array}{ll} K & K \ K_^T & K_{**} \end{array} \right] \right),
$$
where $\boldsymbol{m} =...
Can you write Python code for it?

def kernel_SE(X1,X2,par):
    p0 = 10.0**par[0]
    p1 = 10.0**par[1]
    D2 = cdist(X1,X2,'sqeuclidean')
    K = p0 * np.exp(- p1 * D2)
    return np.matrix(K)
def kernel_Mat32(X1,X2,par):
    p0 = 10.0**par[0]
    p1 = 10.0**par[1]
    DD = cdist(X1, X2, 'euclidean')
    arg = np.sqrt(3) * abs(DD) / p1
    K = p0 * (1 + arg) * np.exp(- arg)
    return np.matrix(K)
def kernel_RQ(X1,X2,par):
    p0 = 10.0**par[0]
    p1 = 10.0**par[1]
    alpha = par[2]
    D2 = cdist(X1, X2, 'sqeuclidean')
    K = p0 * (1 + D2 / (2*alpha*p1))**(-alpha)
    return np.matrix(K)
def kernel_Per(X1,X2,par):
    p0 = 10.0**par[0]
    p1 = 10.0**par[1]
    period = par[2]
    DD = cdist(X1, X2, 'euclidean')
    K = p0 * np.exp(- p1*(np.sin(np.pi * DD / period))**2)
    return np.matrix(K)
def kernel_QP(X1,X2,par):
    p0 = 10.0**par[0]
    p1 = 10.0**par[1]
    period = par[2]
    p3 = 10.0**par[3]
    DD = cdist(X1, X2, 'euclidean')
    D2 = cdist(X1, X2, 'sqeuclidean')
    K = p0 * np.exp(- p1*(np.sin(np.pi * DD / period))**2 - p3 * D2)
    return np.matrix(K)
def add_wn(K,lsig):
    sigma=10.0**lsig
    N = K.shape[0]
    return K + sigma**2 * np.identity(N)
def get_kernel(name):
    if name == 'SE': return kernel_SE
    elif name == 'RQ': return kernel_RQ
    elif name == 'M32': return kernel_Mat32
    elif name == 'Per': return kernel_Per
    elif name == 'QP': return kernel_QP
    else:
        print 'No kernel called %s - using SE' % name
        return kernel_SE

def pltsamples1(par0=0.0, par1=0.0, wn = 0.0):
    x = np.r_[-5:5:201j]
    X = np.matrix([x]).T # scipy.spatial.distance expects matrices
    kernel = get_kernel('SE')
    K = kernel(X,X,[par0,par1])
    K = add_wn(K,wn)
    fig=pl.figure(figsize=(10,4))
    ax1 = pl.subplot2grid((1,3), (0, 0), aspect='equal')
    pl.imshow(np.sqrt(K),interpolation='nearest',vmin=0,vmax=10)
    pl.title('Covariance matrix')
    ax2 = pl.subplot2grid((1,3), (0,1),colspan=2)
    np.random.seed(0)
    for i in range(3):
        y = np.random.multivariate_normal(np.zeros(len(x)),K)
...


I am working on the file "Sessions/Session04/Day2/GPLecture1.ipynb".
The first task is:
The Matern family
The Matern 3/2 kernel
$$
k_{3/2}(x,x')= A \left( 1 + \frac{\sqrt{3}r}{l} \right) \exp \left( - \frac{\sqrt{3}r}{l} \right),
$$
where $r =|x-x'|$.
It produces somewhat rougher behaviour, because it is only differentiable once w.r.t. $r$ (whereas the SE kernel is infinitely differentiable). There is a whole family of Matern kernels with varying degrees of roughness.
The rational quadratic kernel
is equivalent to a squared exponential with a powerlaw distribution of input scales
$$
k_{\rm RQ}(x,x') = A^2 \left(1 + \frac{r^2}{2 \alpha l} \right)^{-\alpha},
$$
where $\alpha$ is the index of the power law.
This is useful to model data containing variations on a range of timescales with just one extra parameter.
Can you write Python code for it?

# Function to plot samples from kernel
def pltsamples2(par2=0.5, kernel_shortname='SE'):
    x = np.r_[-5:5:201j]
    X = np.matrix([x]).T # scipy.spatial.distance expects matrices
    kernel = get_kernel(kernel_shortname)
    K = kernel(X,X,[0.0,0.0,par2])
    fig=pl.figure(figsize=(10,4))
    ax1 = pl.subplot2grid((1,3), (0, 0), aspect='equal')
    pl.imshow(np.sqrt(K),interpolation='nearest',vmin=0,vmax=10)
    pl.title('Covariance matrix')
    ax2 = pl.subplot2grid((1,3), (0,1),colspan=2)
    np.random.seed(0)
    for i in range(3):
        y = np.random.multivariate_normal(np.zeros(len(x)),K)
        pl.plot(x,y-i*2)
    pl.xlim(-5,5)
    pl.ylim(-8,5)
    pl.xlabel('x')
    pl.ylabel('y')
    pl.title('Samples from %s prior' % kernel_shortname)
    pl.tight_layout()
interact(pltsamples2,
         par2=widgets.FloatSlider(min=0.25,max=1,step=0.25,description=r'$\alpha$',value=0.5),
         kernel_shortname=widgets.RadioButtons(options=['SE','M32','RQ'], value='SE',description='kernel')
        );


I am working on the file "Sessions/Session04/Day2/GPLecture1.ipynb".
The first task is:
To make the relation to time-series data a bit more obvious, let's plot the two variables side by side, then see what happens to one variable when we observe (fix) the other.
Can you write Python code for it?

def SEKernel(par, x1, x2):
    A, Gamma = par
    D2 = cdist(x1.reshape(len(x1),1), x2.reshape(len(x2),1),
               metric = 'sqeuclidean')
    return A * np.exp(-Gamma*D2)
A = 1.0
Gamma = 0.01
x = np.array([-1,1])
K = SEKernel([A,Gamma],x,x)
m = np.zeros(len(x))
sig = np.sqrt(np.diag(K))
pl.figure(figsize=(15,7))
pl.subplot(121)
for i in range(len(x)):
    pl.plot([x[i]-0.1,x[i]+0.1],[m[i],m[i]],'k-')
    pl.fill_between([x[i]-0.1,x[i]+0.1],
                    [m[i]+sig[i],m[i]+sig[i]],
                    [m[i]-sig[i],m[i]-sig[i]],color='k',alpha=0.2)
pl.xlim(-2,2)
pl.ylim(-2,2)
pl.xlabel(r'$x$')
pl.ylabel(r'$y$');

def Pred_GP(CovFunc, CovPar, xobs, yobs, eobs, xtest):
    # evaluate the covariance matrix for pairs of observed inputs
    K = CovFunc(CovPar, xobs, xobs)
    # add white noise
    K += np.identity(xobs.shape[0]) * eobs**2
    # evaluate the covariance matrix for pairs of test inputs
    Kss = CovFunc(CovPar, xtest, xtest)
    # evaluate the cross-term
    Ks = CovFunc(CovPar, xtest, xobs)
    # invert K
    Ki = inv(K)
    # evaluate the predictive mean
    m = np.dot(Ks, np.dot(Ki, yobs))
    # evaluate the covariance
    cov = Kss - np.dot(Ks, np.dot(Ki, Ks.T))
    return m, cov
xobs = np.array([-1])
yobs = np.array([1.0])
eobs = 0.0001
pl.subplot(122)
pl.errorbar(xobs,yobs,yerr=eobs,capsize=0,fmt='k.')
x = np.array([1])
m,C=Pred_GP(SEKernel,[A,Gamma],xobs,yobs,eobs,x)
sig = np.sqrt(np.diag(C))
for i in range(len(x)):
    pl.plot([x[i]-0.1,x[i]+0.1],[m[i],m[i]],'k-')
    pl.fill_between([x[i]-0.1,x[i]+0.1],
                    [m[i]+sig[i],m[i]+sig[i]],
                    [m[i]-sig[i],m[i]-sig[i]],color='k',alpha=0.2)
pl.xlim(-2,2)
pl.ylim(-2,2)
pl.xlabel(r'$x$')
pl.ylabel(r'$y$');


I am working on the file "Sessions/Session04/Day2/GPTutorial1.ipynb".
The first task is:
Generate a set of $50$ one-dimensional inputs regularly spaced between -5 and 5 and store them in a variable called x, then compute the covariance matrix for these inputs, for $A=\Gamma=1$, store the results in a variable called K, and display it using matplotlib's imshow function.
Can you write Python code for it?

x = np.linspace(# complete
K = # complete
plt.imshow(K,interpolation='none');


I am working on the file "Sessions/Session04/Day2/GPTutorial1.ipynb".
The first task is:
How does the best fit likelihood compare to what you obtained using the SEKernel? Which kernel would you adopt if you had to chose between the two. Write your answer in the cell below.
The maximum log likelihood in the case of the SEKernel was -93.25, compared to -89.55 for the M32Kernel, so the SEKernel is preferred, as one would expect, though the difference is not very large as far as these things go.
Problem 4b: Mean, or no mean?
Now let us try comparing models with and without non-zero mean function. This time we are comparing models with different numbers of parameters, so we will need to evaluate the BIC rather than simply compare the likelihoods. The BIC is defined as:
$$
\mathrm {BIC} ={\ln(N)K-2\ln({\hat {L}})},
$$
where $N$ is the number of observations, $J$ is the number of parameters, and $\hat{L}$ refers to the likelihood maximised with respect to the parameters of the model.
Start by fitting the simulated dataset with a squared exponential kernel, with and without no mean function. Evaluate the maximum likelihood in each case and store it in variables L_mean and L_no_mean, respectively.
Can you write Python code for it?

# Copy and paste your answer to the previous problem and modify it as needed


I am working on the file "Sessions/Session04/Day2/GPTutorial1.ipynb".
The first task is:
NB: See how the predictive distribution continues the linear trend outside the range of hte data. If we had tried to model the same dataset without using a mean function, we might well have got a visually acceptable fit, but the predictive distribution would have returned to the prior rapidly outside the range of the data.
Problem 4: Model comparison
One question that is frequently asked about GPs is how to chose the covariance function. There is no simple answer to this. The best starting point is domain knowledge, things you know about your dataset a priori. But if you are looking for an empirical way to compare different kernels for a given dataset, this is a standard model comparison problem.
Similarly, if you are using GPs as part of a detection problem, where you are asking whether the data contain a particular signal which is represented via the mean function, you might want to compare models with the same covariance function, but with and without a (non-zero) mean function.
In both of the above examples, the number of hyper-parameters can vary between the models one is comparing, so model comparison is not straight forward. Ideally one would evaluate and compare the evidence for different kernels (i.e. the likelihood marginalised over the hyper-parameters), but doing this is outside the scope of the present tutorial. Therefore, here we will use a simple alternative, the Bayesian Information Criterion (BIC).
Problem 4a: Comparing two covariance functions
Consider the dataset we simulated earlier, with the linear trend and the correlated noise. The correlated noise was generated using a squared exponential GP. If we didn't know this, would we be able to distinguish between a squared exponential and a different kind of covariance function?
To test this, let's try a changing the covariance function. We will use the Matern 3/2 kernel:
$$
k_{3/2} (x,x') = A \left[ 1 + \sqrt{3r^2} \right] \exp \left[ - \sqrt{3r^2} \right]
$$
where $r^2=\Gamma(x-x')^2$. I picked this one for two reaons:
- This kernel gives...
Can you write Python code for it?

def M32Kernel(par, x1, x2):
    A, Gamma = par
    R = cdist(# complete
    return # complete


I am working on the file "Sessions/Session04/Day2/GPTutorial1.ipynb".
The first task is:
NB: The fit can be quite sensitive to the initial guesses for the parameters. Therefore, you may find the fit converges to unexpected values unless you start fairly close to the "correct" ones. In the next tutorial, we will look at more robust ways of exploring the parameter space.
Problem 3c: Including the mean function in the predictions
The simplest way to do this is to use the LinearMean function to evaluate the mean vector, and the Pred_GP function we wrote earlier to evaluate the stochastic component, and add the two together.
Plot the data together with the mean and 95% confidence interval of the predictive distribution, using the best-fit hyper-parameters.
Hint: you'll need to generate a new set of test inputs (the values at which you want to evaluate the predictive distribution) as our new dataset spans a wider range than the old one.
Can you write Python code for it?

# Generate test inputs (values at which we ant to evaluate the predictive distribution)
x = np.linspace(# complete
# Evaluate mean function at observed inputs, and compute residuals
mobs = # complete
robs = yobs-mobs
# Evaluate stochastic component at test inputs
m,C = Pred_GP(# complete
# Evaluate mean function at test inputs
m += # complete
sig = # complete
plt.errorbar(xobs,yobs,yerr=2*eobs,capsize=0,fmt='k.')
plt.plot(x,m,'k-')
plt.fill_between(x,m+2*sig,m-2*sig,color='k',alpha=0.2)
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.title('Maximum likelihood distribution');


I am working on the file "Sessions/Session04/Day2/GPTutorial1.ipynb".
The first task is:
Now draw 5 samples from the distribution and plot them.
Hint: You may find the function multivariate_normal from the module numpy.random useful.
Can you write Python code for it?

samples = multivariate_normal(# complete
plt.plot(x,samples.T)
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.title('Samples from prior distribution');


I am working on the file "Sessions/Session04/Day2/GPTutorial1.ipynb".
The first task is:
Now you are ready to fit for all the hyper-parameters simultaneously: those of the covariance function, and those of the mean function.
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day2/GPTutorial1.ipynb".
The first task is:
Plot the data and the predictive distribution and samples for the best-fit hyper-parameters
Can you write Python code for it?

# You can reuse code from Problem 1c almost exactly here...


I am working on the file "Sessions/Session04/Day2/GPTutorial1.ipynb".
The first task is:
Problem 3a: Including a mean function
This dataset contains a linear trend as well as correlated noise. We want leanr the trend at the same time as the noise, so we need to define a mean function
Complete the definition of the mean function below, so that it evaluates abd return $f = p[0] x + p[1]$. Check that it works by plotting its output on top of the data above, taking the parameter values from the code that generated the data.
Can you write Python code for it?

def LinearMean(p,x):
    return # complete
pm0 = [0.03, -0.3]
m = # complete
plt.errorbar(xobs,yobs,yerr=eobs,fmt='k.',capsize=0)
plt.plot(xobs,m,'r-')
plt.xlabel(r'$x$')
plt.ylabel(r'$y$');


I am working on the file "Sessions/Session04/Day2/GPTutorial1.ipynb".
The first task is:
Problem 3b: Likelihood with a mean function
Evaluating the likelihood of a GP with a non-zero mean function is easy: simply evaluate the mean vector, subtract it from the data, and compute the likelihood as before, but using the residuals rather than the original data.
Modify the likelihood function you defined earlier so that it does this. Check that it runs ok by calling it once on the dataset, using guesses for the values of the parameters.
Hint: use optional keyword arguments so that your likelihood function still works without a mean function. You will also need to tell the likelihood function how many of the parameters belong to the mean function.
Can you write Python code for it?

def NLL_GP2(p,CovFunc,x,y,e, MeanFunc=None, nmp = 0):
    if MeanFunc:
        pc = p[# complete
        pm = p[# complete
        r = y - # complete
    else:
        pc = p[:]
        r = y[:]
    # Evaluate the covariance matrix
    K = # complete
    # Add the white noise term
    K += # complete
    # invert it
    Ki = inv(K)
    # evaluate each of the three terms in the NLL
    term1 = # complete
    term2 = # complete
    term3 = # complete
    # return the total
    return term1 + term2 + term3

p0 = [0.005,2.0,0.03,-0.3]
print(NLL_GP2# complete