I am working on the file "Sessions/Session04/Day2/GPTutorial1.ipynb".
The first task is:
Some things to note:
- while the prior distribution is stationary (the mean and variance are constant over the input range) the predictive distribution is not;
- far away grom observations, the predictive distribution returns to the prior
- where observations are close together compared to the lenght scale of the GP, the predictive ability is good: the 95% confidence interval is narrow and the samples from the predictive distribution all behave similarly
- the predictive mean doesn't have the same behaviour as the samples from the predictive distribution
Problem 1d: Changing the hyper-parameters
Try changing the covariance function parameters $A$ and $\Gamma$ in the last cell, and then running it again. Make some notes on what happens in the cell below.
Provide your answer to Problem 1d here
So how do we know what values of the covariance paramters to use? To fit for them, we to evaluate the likelihood of the model. This process is known as training the GP, and is the subject of problem 2.
Problem 1e: Changing the uncertainties
Try changing the observational uncertainties by altering the value of eobs in the last code cell and then running it again. Also try variable uncertainties (heteroskedastic noise) by assigning an array of $N$ values to eobs. Make some notes on what happens as you change the errors in the cell below.
Hint: If you set the uncertainties to zero, you'll get an error, because the covariance matrix becomes ill-conditioned. Try setting them to a small non-zero value (e.g. $10^{-3}$) instead.
Provide your answer to Problem 1e here
Problem 2: Training the GP
In this problem we will learn how to compute the likelihood of a set of observations given a model (i.e. given a covariance function and parameters thereof) and how to optimize it relative to the hyper-parameters.
Problem 2a: The likelihood function
Under a GP model, the likelihood is simply a multivariate Gaussian with mean vector $\mathbf{m}$ and covariance matrix $K$:
$$
p(\mathbf{y} \, | \, \mathbf{m}, K) =...
Can you write Python code for it?

def NLL_GP(p,CovFunc,x,y,e):
    # Evaluate the covariance matrix
    K = # complete
    # Add the white noise term
    K += # complete
    # invert it
    Ki = inv(K)
    # evaluate each of the three terms in the NLL
    term1 = # complete
    term2 = # complete
    term3 = # complete
    # return the total
    return term1 + term2 + term3


I am working on the file "Sessions/Session04/Day2/GPTutorial1.ipynb".
The first task is:
Try evaluating the likelihood of the model given the observations you defined in problem 1 by executing the cell below. Hopefully it will run without errors...
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
As you can see, the predictive distribution are essentially indistinguishable in regions where we have lots of data, but the predictive ability of the model without mean function is much poorer away from the data. Of course, this is as expected.
Challenge problem: Kepler light curve of a spotted star
In this problem you will model the light curve of a spotted star observed by Kepler using a GP with a quasi-periodic covariance function.
McQuillan et al. (2014) used a method based on the auto-correlation function of the light curve to measure the rotation periods of over 30000 Kepler main-sequence stars. I have selected one, KIC2157356, which shows a very clear modulation, and for which McQuillan et al. measured a period of $13.61 \pm 0.05$ days.
You will find one year's worth of Kepler data for this star in this file. The 4 columns are: time in days, relative flux, flux error, and quarter number. The Kepler satellite rotates about its pointing axis every 3 months to keep its solar panels oriented towards the Sun, and this leads to breaks in the observations, as well as baseline shifts.
Your task is to read in the data and model it using the quasi-periodic covariance function:
$$
k_{\mathrm{QP}}(t,t') = A \exp \left[ - \Gamma_1 \sin^2 \left( \frac{\pi (t-t')}{P} \right) - \Gamma_2 (t-t')^2 \right]
$$
in order to evaluate the rotation period $P$. You should start by modelling each quarter separately, checking the consistency of your results on the different quarters.
Then introduce a mean function to account for the baseline shifts between quarters, and fit for the 4 quarters simultaneously. This will also enable you to get a better estimate of the evolutionary timescale for the star spots that cause the flux variation, which is controlled by the parameter $\Gamma_2$.
Can you think of a way to evaluate uncertainties on the period $P$?
NB: This is a much bigger dataset, even when considering only a single quarter. For the GP regression to run in reasonable time, you will probably need to sub-sample your data.
Can you write Python code for it?

from astropy.table import Table
tab = Table.read('KIC2157356.txt',format='ascii')
qs = np.unique(tab['quarter'])
for q in qs:
    t = tab[tab['quarter']==q]
    plt.plot(t['time'],t['flux'],'.')

def QPKernel(par,x1,x2):
    A, P, Gamma1, Gamma2 = par
    D = cdist(x1.reshape(len(x1),1), x2.reshape(len(x2),1),
              metric = 'euclidean')
    D2 = cdist(x1.reshape(len(x1),1), x2.reshape(len(x2),1),
               metric = 'sqeuclidean')
    return A * np.exp(-Gamma1*np.sin(D*np.pi/P)**2-Gamma2*D2)

for q in qs:
    # select data for the quarter
    t = tab[tab['quarter']==q]
    xobs = t['time']
    yobs = t['flux']
    eobs = t['error']
    # normalise
    m = np.median(yobs)
    yobs = yobs / m - 1
    eobs /= m
    # subsample to keep computing time reasonable
    xtrain = xobs[::10]
    ytrain = yobs[::10]
    etrain = eobs[::10]
    # fit for hyper-parameters
    p0 = [np.var(yobs), 13.61, 1.0, 1e-4]
    p1 = fmin(NLL_GP,p0,args=(QPKernel,xtrain,ytrain,etrain))
    print(p1)
    # test outputs for plots
    x = np.linspace(xobs.min()-10,xobs.max()+10,1000)
    plt.figure()
    plt.plot(xobs,yobs,'k,')
    plt.plot(xtrain,ytrain,'k.')
    m,C = Pred_GP(QPKernel,p0,xtrain,ytrain,etrain,x)
    sig = np.sqrt(np.diag(C))
    plt.plot(x,m,'k-')
    plt.fill_between(x,m+2*sig,m-2*sig,color='k',alpha=0.2)
    plt.xlabel(r'$t$')
    plt.ylabel(r'$y$')
    plt.title('Quarter {}'.format(q));


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
Evaluate and plot the mean and 95% confidence interval of the resulting posterior distribution, as well as a few samples, for a squared exponential GP with $A=\Gamma=1$, assuming the measurement uncertainty on each observation was 0.1
Can you write Python code for it?

eobs = 0.1
m,C=Pred_GP(SEKernel,[1.0,1.0],xobs,yobs,eobs,x)
sig = np.sqrt(np.diag(C))
samples = multivariate_normal(m,C,5)
plt.errorbar(xobs,yobs,yerr=2*eobs,capsize=0,fmt='k.')
plt.plot(x,m,'k-')
plt.fill_between(x,m+2*sig,m-2*sig,color='k',alpha=0.2)
plt.plot(x,samples.T,alpha=0.5)
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.title('Predictive distribution');


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
Gaussian Process regression tutorial 1: Solutions
Introduction
In this tutorial we will create a very basic, native python GP regression code and apply it to a very simple simulated example dataset.

By S Aigrain (University of Oxford)
Can you write Python code for it?

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from scipy.spatial.distance import cdist
from numpy.random import multivariate_normal
from numpy.linalg import inv
from numpy.linalg import slogdet
from scipy.optimize import fmin


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
Generate a set of $50$ one-dimensional inputs regularly spaced between -5 and 5 and store them in a variable called x, then compute the covariance matrix for these inputs, for $A=\Gamma=1$, store the results in a variable called K, and display it using matplotlib's imshow function.
Can you write Python code for it?

x = np.linspace(-5,5,50)
K = SEKernel([1.0,1.0],x,x)
plt.imshow(K,interpolation='none');


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
How does the best fit likelihood compare to what you obtained using the SEKernel? Which kernel would you adopt if you had to chose between the two. Write your answer in the cell below.
The maximum log likelihood in the case of the SEKernel was -93.25, compared to -89.55 for the M32Kernel, so the SEKernel is preferred, as one would expect, though the difference is not very large as far as these things go.
Problem 4b: Mean, or no mean?
Now let us try comparing models with and without non-zero mean function. This time we are comparing models with different numbers of parameters, so we will need to evaluate the BIC rather than simply compare the likelihoods. The BIC is defined as:
$$
\mathrm {BIC} ={\ln(N)K-2\ln({\hat {L}})},
$$
where $N$ is the number of observations, $J$ is the number of parameters, and $\hat{L}$ refers to the likelihood maximised with respect to the parameters of the model.
Start by fitting the simulated dataset with a squared exponential kernel, with and without no mean function. Evaluate the maximum likelihood in each case and store it in variables L_mean and L_no_mean, respectively.
Can you write Python code for it?

p0_mean = [0.005,2.0,0.03,-0.3]
p1_mean = fmin(NLL_GP2,p0_mean,args=(SEKernel,xobs,yobs,eobs,LinearMean,2))
NLL_mean = NLL_GP2(p1_mean,SEKernel,xobs,yobs,eobs,MeanFunc=LinearMean,nmp=2)
print(NLL_mean)
p0_no_mean = [0.005,2.0]
p1_no_mean = fmin(NLL_GP2,p0_no_mean,args=(SEKernel,xobs,yobs,eobs))
NLL_no_mean = NLL_GP2(p1_no_mean,SEKernel,xobs,yobs,eobs)
print(NLL_no_mean)


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
Now try changing the covariance parameters and the observational uncertainties, and see how that affects the likelihood. Does it behave as you would expect, given the way these parameters affected the predictive distribution?

Making $A$ too big or too small decreases the likelihood (increases the NLL), as does making $\Gamma$ too small. This is as one would expect - either the model becomes an obviously bad match, or the model uncertainty becomes huge (and that is penalised by the determinant term).
On the other hand, for such a sparse dataset, the likelihood asymptotes to a constant as one increases $\Gamma$ (decreases the length scale): essentially, the data doesn't contain information on the behaviour on length scales smaller than the minimum separation between observations.

Problem 2b: Fitting for the covariance parameters
We are now ready to find the best covariance hyper-parameters, i.e. those that minimize the NLL, for a given dataset and covariance function.
Hint: For simplicity we will do this using the fmin function from scipy.optimize. This is just a downhill simplex optimizer, once again it's not particularly stable or efficient, but it should do the trick for now. We will look at better ways of optimising and sampling the likelihood in Tutorial 2.
Can you write Python code for it?

p0 = [1.0,1.0]
p1 = fmin(NLL_GP,p0,args=(SEKernel,xobs,yobs,eobs))
print(p1)


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
Now you are ready to fit for all the hyper-parameters simultaneously: those of the covariance function, and those of the mean function.
Can you write Python code for it?

p1 = fmin(NLL_GP2,p0,args=(SEKernel,xobs,yobs,eobs,LinearMean,2))
print(p1)


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
Plot the data and the predictive distribution and samples for the best-fit hyper-parameters
Can you write Python code for it?

m,C=Pred_GP(SEKernel,p1,xobs,yobs,eobs,x)
sig = np.sqrt(np.diag(C))
samples = multivariate_normal(m,C,5)
plt.errorbar(xobs,yobs,yerr=2*eobs,capsize=0,fmt='k.')
plt.plot(x,m,'k-')
plt.fill_between(x,m+2*sig,m-2*sig,color='k',alpha=0.2)
plt.plot(x,samples.T,alpha=0.5)
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.title('Maximum likelihood distribution');


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
Problem 1: A probability distribution over functions
We saw in the lectures that a Gaussian Process enables us to set up a probability distribution over functions, but that probably sounds a little abstract. This problem aims to give you more of a feel for what that means, in practice.
You will start by defining a covariance function and using it to generate a covariance matrix. You will draw samples from the GP prior (i.e. draws from the probability distribution over functions, evaluated at a finite number of input locations), and then learn how to condition the prior on some data, and to draw samples from the resulting predictive distribution. Finally, you will explore the effect of altering the covariance matrix, specifically changing the hyper-parameters (the parameters of the covariance function) and the observational uncertainties.
Problem 1a: Your first covariance function
First we need to write a function that will generate a covariance matrix for a given covariance function, or kernel. Let's start with the squared exponential kernel, which is one of the simplest and most widely used.
$$
k_{\rm SE}(x,x') = A \exp \left[ \Gamma (x-x')^2 \right]
$$
where $A$ is the variance and $\Gamma$ the inverse length scale. This kernel gives rise to smoothly varying, infinitely differentiable functions.
Define a function SEKernel that computes the covariance matrix for the above kernel function. The function should take three mandatory arguments: an array containing the hyper-parameters of the covariance function ($A$ and $\Gamma$ in the equation above), and two arrays of input values. The function should return a 2-D array with shape $(N, M)$, where $N$ and $M$ are the numbers of inputs in each of the input arrays.
*Hint: You may find the function cdist from the module scipy.spatial.distance useful, but note it expects input arrays of shape $(N,D)$ where $D$ is the number of dimensions of the inputs.
Can you write Python code for it?

def SEKernel(par, x1, x2):
    A, Gamma = par
    D2 = cdist(x1.reshape(len(x1),1), x2.reshape(len(x2),1),
               metric = 'sqeuclidean')
    return A * np.exp(-Gamma*D2)


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
Problem 1b: The prior distribution: mean, confidence intervals and samples
The joint prior distribution over the outputs $\mathbf{y}$, evaluated at a given collection of inputs $\mathbf{x}$, is a multi-variate Gaussian distribution with zero mean vector and covariance matrix $K$:
$$
p(\mathbf{y}|\mathbf{x})=\mathcal(N)(\mathbf{0},K).
$$
(NB: we will consider non-zero mean functions later.)
Plot the mean and 95% confidence interval of this distribution for the x and K evaluated in the previous cell.
Hint: the variance $\sigma^2$ of the distribution is given by the diagonal elements of the covariance matrix, and the 95% confidence interval is bounded by the mean plus or minus 2 $\sigma$.
Can you write Python code for it?

m = np.zeros(len(x))
sig = np.sqrt(np.diag(K))
plt.plot(x,m,'k-')
plt.fill_between(x,m+2*sig,m-2*sig,color='k',alpha=0.2)
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.title('Prior distribution');


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
Problem 1c: The predictive distribution
If we have some observations $(\mathbf{x},\mathbf{y})$, only a subset of the functions included in our prior distribution will be compatible with them. As we saw in the lecture, the posterior distribution (also called the conditional or predictive distribution) for the test outputs $\mathbf{y}$, evaluated at test inputs $\mathbf{x}_$, given the observations, is a multivariate Gaussian:
$$
p(\mathbf{y} \, | \, \mathbf{x}_, \mathbf{x}, \mathbf{y}) = \mathcal{N}(\overline{\mathbf{y}}, \mathrm{cov}(\mathbf{y}_))
$$
with mean
$$
\overline{\mathbf{y}} = K(\mathbf{x}_,\mathbf{x}) K(\mathbf{x},\mathbf{x})^{-1} \mathbf{y}
$$
and covariance
$$
\mathrm{cov}(\mathbf{y}) = K(\mathbf{x}_,\mathbf{x}) - K(\mathbf{x}_,\mathbf{x}) K(\mathbf{x},\mathbf{x})^{-1} K(\mathbf{x},\mathbf{x}*),
$$
where $K(\mathbf{x},\mathbf{x'}){ij} = k(x_i,x'_j)$.
If the observations are noisy, the white noise variance should be added to the diagonal of the covariance (for pairs of observed data points only).
Complete the definition of the function Predict below. This function computes and returns the mean and covariance of the predictive distribution for a given covariance function, with associated parameters, a given set of observations $(\mathbf{x},\mathbf{y},\mathbf{\sigma})$, where $\mathbf{\sigma}$ are the uncertainties associated with each observation, at a given set of test inputs $\mathbf{x}_*$.
Hint: Use numpy's dot function to do matrix multiplication of two numpy arrays, or convert your arrays to numpy matrix objects before multiplying them together.
Hint: Use the inv function from numpy.linalg to invert the covariance matrix of the observations. It's not particularly stable or efficient, but code optimisation is not the point of this tutorial. In tutorial 2 we will look at some ready-made GP packages, which use much more optimized matrix inversion techniques.
Can you write Python code for it?

def Pred_GP(CovFunc, CovPar, xobs, yobs, eobs, xtest):
    # evaluate the covariance matrix for pairs of observed inputs
    K = CovFunc(CovPar, xobs, xobs)
    # add white noise
    K += np.identity(xobs.shape[0]) * eobs**2
    # evaluate the covariance matrix for pairs of test inputs
    Kss = CovFunc(CovPar, xtest, xtest)
    # evaluate the cross-term
    Ks = CovFunc(CovPar, xtest, xobs)
    # invert K
    Ki = inv(K)
    # evaluate the predictive mean
    m = np.dot(Ks, np.dot(Ki, yobs))
    # evaluate the covariance
    cov = Kss - np.dot(Ks, np.dot(Ki, Ks.T))
    return m, cov


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
Problem 3a: Including a mean function
This dataset contains a linear trend as well as correlated noise. We want leanr the trend at the same time as the noise, so we need to define a mean function
Complete the definition of the mean function below, so that it evaluates abd return $f = p[0] x + p[1]$. Check that it works by plotting its output on top of the data above, taking the parameter values from the code that generated the data.
Can you write Python code for it?

def LinearMean(p,x):
    return p[0] * x + p[1]
pm0 = [0.03, -0.3]
m = LinearMean(pm0,xobs)
plt.errorbar(xobs,yobs,yerr=eobs,fmt='k.',capsize=0)
plt.plot(xobs,m,'r-')
plt.xlabel(r'$x$')
plt.ylabel(r'$y$');


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
Some things to note:
- while the prior distribution is stationary (the mean and variance are constant over the input range) the predictive distribution is not;
- far away grom observations, the predictive distribution returns to the prior
- where observations are close together compared to the lenght scale of the GP, the predictive ability is good: the 95% confidence interval is narrow and the samples from the predictive distribution all behave similarly
- the predictive mean doesn't have the same behaviour as the samples from the predictive distribution
Problem 1d: Changing the hyper-parameters
Try changing the covariance function parameters $A$ and $\Gamma$ in the last cell, and then running it again. Make some notes on what happens in the cell below.

smaller $\gamma$ (longer length scale): The model cannot vary fast enough to reproduce the data. The predictive distribution is very narrow but doesn't overlap with the observations.
larger $\gamma$ (shorter length scale): The model has no trouble reproducing the data, but returns to the prior and loses predictive ability much faster away from the observations.
smaller $A$: The model cannot vary with enough amplitude to reproduce the data. The predictive distribution is very narrow but doesn't overlap with the observations.
larger $A$:  The model has no trouble reproducing the data, but the 95% confidence interval is huge away from the observations.

So how do we know what values of the covariance paramters to use? To fit for them, we to evaluate the likelihood of the model. This process is known as training the GP, and is the subject of problem 2.
Problem 1e: Changing the uncertainties
Try changing the observational uncertainties by altering the value of eobs in the last code cell and then running it again. Also try variable uncertainties (heteroskedastic noise) by assigning an array of $N$ values to eobs. Make some notes on what happens as you change the errors in the cell below.
Hint: If you set the uncertainties to zero, you'll get an error, because the...
Can you write Python code for it?

def NLL_GP(p,CovFunc,x,y,e):
    # Evaluate the covariance matrix
    K = CovFunc(p,x,x)
    # Add the white noise term
    K += np.identity(x.shape[0]) * e**2
    # invert it
    Ki = inv(K)
    # evaluate each of the three terms in the NLL
    term1 = 0.5 * np.dot(y,np.dot(Ki,y))
    term2 = 0.5 * slogdet(K)[1]
    term3 = 0.5 * len(y) * np.log(2*np.pi)
    # return the total
    return term1 + term2 + term3


I am working on the file "Sessions/Session04/Day2/GPTutorial1_WithSolutions.ipynb".
The first task is:
Thus the model with a non-zero mean function is strongly preferred (BIC differences $> 10$ are generally considered to represent very strong support for one model over the other).
How different would the predictive distributions and samples be? Try plotting them in each case.
Can you write Python code for it?

# Plot the data
plt.errorbar(xobs,yobs,yerr=2*eobs,capsize=0,fmt='k.')
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
plt.title('Model comparison')
# Evaluate and plot the predictive distribution with a mean function
mobs = LinearMean(p1_mean[-2:],xobs)
robs = yobs-mobs
m,C = Pred_GP(SEKernel,p1_mean[:2],xobs,robs,eobs,x)
m += LinearMean(p1[-2:],x)
sig = np.sqrt(np.diag(C))
plt.plot(x,m,'b-')
plt.fill_between(x,m+2*sig,m-2*sig,color='b',alpha=0.2)
# Now do the same for the model without mean function
m,C = Pred_GP(SEKernel,p1_no_mean,xobs,yobs,eobs,x)
sig = np.sqrt(np.diag(C))
plt.plot(x,m,'r-')
plt.fill_between(x,m+2*sig,m-2*sig,color='r',alpha=0.2)


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
As you can see, the delays estimated in this way aren't too far off.
To get initial guesses for the GP hyper-parameters, we can take the mean of the best-fit values from the three individual time-series. Do this in the cell below.
Can you write Python code for it?

gppar_0 = # complete
print('Estimated GP HPs')
print(gppar_0)
print('True GP HPs')
print(gppar_true)


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
As you can see, the optimization further improved our estimates of the time delays and the GP HPs. But how much can we trust these? Let's evaluate posterior uncertainties using MCMC.
Hyper-parameter marginalisation.
We now use MCMC to obtain uncertainty estimates, or confidence intervals, for the model hyper-parameters.
First we need to define the posterior function to pass to the emcee sampler. We will use improper, flat priors over all the parameters, so the posterior probability is just a trivial wrapper around our neg_ln_like_delays function. Complete the definition below:
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Execute the cell below to plot contours of the  predictive mean alongside the data.
Can you write Python code for it?

plt.scatter(xobs, yobs, c=zobs, s=20, marker='.')
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
cb = plt.colorbar()
cb.set_label(r'$z$');
plt.contour(X2D,Y2D,Z2D);


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Hopefully the distributions should look reasonable and be consistent with the true values.
We need to extract confidence intervals for the parameters from the MCMC chain, which we can access through sampler.flatchain
Can you write Python code for it?

samples = # complete
# The GP parameters were explored in log space, return them to linear space
#samples[:, -2:] = np.exp(samples[:, -2:])
# This handy bit of code will extract median and +/- 1 sigma intervals for each parameter
pv = map(lambda v: (v[1], v[2]-v[1], v[1]-v[0]), zip(*np.percentile(samples, [16, 50, 84], axis=0)))
# Print the results
for i in range(ndim):
    pval = # complete
    print("Param {}: {:5.2f} +{:4.2f} -{:4.2f} (true: {:5.2f})".format(i+1,pval[0], pval[1], pval[2], ptrue[i]))


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Hopefully, the MCMC estimates should be consistent with the true values...
Challenge problem: Active scheduling
Imagine you are monitoring a particular variable, you want to know its value to a given precision at anyone time, but each observation is costly, so you don't want to take any more than you have to. You can train a GP on the first few observations, then use the predictive distribution to work out when your uncertainty about the current value of the variable is so large that you need to take a new observation. Use the new observation to update the GP hyper parameters and the predictive distribution, and repeat the process...
First we generate a tightly sampled time series over 100 days. This will represent the "true" value of the variable. We will include some periodic behaviour as that makes the problem more interesting. Then we will "observe" 1 point per day for the first 20 days.
Can you write Python code for it?

xtrue = np.linspace(0,100,1000)
k = george.kernels.CosineKernel(np.log(12.3)) * george.kernels.ExpSquaredKernel(1000.0)
ytrue = george.GP(k).sample(xtrue)
xobs = xtrue[:200:10]
eobs = 10.0**(np.random.uniform(-1.5,-1,20))
yobs = ytrue[:200:10] + np.random.normal(0,1,20) * eobs
plt.plot(xtrue,ytrue)
plt.errorbar(xobs,yobs,yerr=eobs,fmt='.',capsize=0);


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Next we use the corner function from the corner module to plot the posterior distributions over the parameters. Complete the code below.
Can you write Python code for it?

labels = [r"$\Delta_1$", r"$\Delta_2$", r"$\ln A$", r"$\ln\l$"]
truths = ptrue
corner.corner(# complete


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Now assign those best-fit values to the parameter vector
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Now execute the cell below to plot the results.
Can you write Python code for it?

plt.figure(figsize=(12,5))
plt.subplot(121)
plt.plot(xobs,zobs,'.',c='grey')
plt.plot(xobs,zobs-fy,'k.')
s = np.argsort(xobs)
plt.plot(xobs[s],fx[s],'r-')
plt.subplot(122)
plt.plot(yobs,zobs,'.',c='grey')
plt.plot(yobs,zobs-fx,'k.')
s = np.argsort(yobs)
plt.plot(yobs[s],fy[s],'r-');


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Now let's plot the predictive distribution to check it worked ok. You can just copy and paste code from Problem 1.
Can you write Python code for it?

# copy plotting commands from problem 1c


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Now we are ready to cross-correlate the interpolated time-series. The easiest way to do this is using the function xcorr from matplotlib.pyplot. This function returns a tuple of 4 variables, the first two of which are the lags and corresponding cross-correlation values.
Can you write Python code for it?

dt = treg[1] - treg[0]
# Array to hold estimates of the time-delays
delays_0 = np.zeros(M-1)
for i in range(M-1):
    # use pyplot's xcorr function to cross-correlate yreg[i+1] with yreg[0]
    lags, corr, _, _ = # complete
    # find the lag that maximises the CCF, convert it to time delay, save in delays_0 array
    lmax = # complete
    plt.axvline(lmax,color=c[i+1])
    delays_0[i] = # complete
plt.xlabel('lag')
plt.ylabel('x-correlation');
# Compare estimated to true delays
print('Estimated time delays from cross-correlation')
print(delays_0)
print('True delays')
print(delays_true)


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Now we need to fit each time-series in turn, and compute the mean of the predictive distribution over a tightly sampled, regular grid of time values. If you take care to name our variables right, you can reuse the neg log likelihood and associated gradient functions from Problem 1.
Complete the code below and run it
Can you write Python code for it?

p0 = gp.get_parameter_vector()
# 2-D array to hold the best-fit GP HPs for each time-series
p1 = np.tile(# complete
# Regularly sampled time array
treg = # complete
# 2-D array to hold the interpolated time-series
yreg = # complete
c = ['r','g','b']
for i in range(M):
    # Compute the gp on the relevant subset of the 2-D time array t2d
    # complete
    # Assign the corresponding y values to the variable zobs
    # (this is the one that neg_ln_like uses to condition the GP)
    zobs = # complete
    # Optimize the likelihood using minimize
    result = minimize(# complete
    # Save the best-fit GP HPs in p1
    p1[i,:] = # complete
    # update the GP parameter vector with the best fit values
    gp.# complete
    # evaluate the predictive mean conditioned on zobs at locations treg and save in yreg
    yreg[i,:] = # complete
    # you might want to plot the results to check it worked
    plt.plot(t2d[i,:],y2d[i,:],'.',c=c[i])
    plt.plot(treg,yreg[i,:],'-',c=c[i])
# And let's print the GP HPs to see if they were sensible.
print('Individual GP fits: best-fit HPs')
print(p1)


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Now you will need to tell the GP object what inputs the covariance matrix is to be evaluated at. This is done using the compute method. 2-D inputs need to be passed as an $N \times 2$ array, which you will need to construct from the two 1-D arrays of $x$- and $y$-values we generated earlier. The second argument of compute should be the white noise standard deviation.
Can you write Python code for it?

Xobs = np.concatenate([[xobs],[yobs]]).T
gp.compute(# complete


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Problem 0: working through the george introductory tutorials
The george documentation includes some nice tutorials, which you'll need to run through before being able to tackle the problems below. Download and run the notebooks, making sure you understand what's going on at each step, and don't hesitate to ask questions!

A gentle introduction to Gaussian Process regression: essentially does the same thing as problem 3 from Tutorial 1, but without a mean function.
Model fitting with correlated noise: includes a mean function, and uses MCMC to explore the dependence of the posterior on the hyper-parameters. The same dataset is also analysed using a model with white noise only, to show how ignoring the correlations in the noise leads to over-confident estimates of the mean function parameters.

Now you should have an idea of how to set up a basic GP model using george, how to make predictions, and how to evaluate the likelihood, optimize it, and explore the posterior using MCMC. I would also encourage you to try out the other tutorials, but they are not pre-requisites for this one.
Problem 1: A simple 2-D problem
So far we have looked only at 1-D inputs, like time. Let's introduce a simple 2-d input case. We will generate some data using a 2-D polynomial and model it using a squared exponential GP.
Run the cell below to generate and plot the data.
Can you write Python code for it?

N = 100
xobs = np.random.uniform(-5,5,N)
yobs = np.random.uniform(-5,5,N)
zobs = - 0.05 * xobs**2 + 0.03 * yobs**2 - 0.02 * xobs * yobs
eobs = 0.01
zobs += np.random.normal(0,eobs,len(xobs))
plt.scatter(xobs, yobs, c=zobs, s=20, marker='.')
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
cb = plt.colorbar()
cb.set_label(r'$z$');


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Problem 1a: setting up the GP
Now we will construct the GP model using george. We will use a with different length scales in each of the two dimensions. To set this up in george, you have to multiply two individual kernels together, like that:
k = a * KernelName(b, ndim = 2, axes = 0) * KernelName(c, ndim = 2, axes = 1)
Here KernelName stands for the name of the kernel used (in george, the squared exponential kernel is called ExpSquaredKernel), a is the output variance, b is the metric, or length scale, applied to the first input dimension, and c to the second.
Note this is equivalent to the parametrisation used in the lectures:
$$
k(x,x') = A \exp \left[ - \Gamma (x-x')^2\right] = A \exp \left[ - (x-x')^2/m^2\right]
$$
with $\Gamma=1/m^2$.
Go ahead and define the kernel in the cell below, with some ball park values for the hyper-parameters (by ball-park, I mean not too many orders of magnitudes off). Then create a GP object using that kernel.
Can you write Python code for it?

k = 1.0 * george.kernels. # ...complete
gp = george.GP(# complete


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Problem 2a: Joint model
We start, once again, by defining the GP object. The kernel will consist of a sum of 2 squared exponentials, one applied to each dimension. It will be useful to be able to access each of the kernel objects separately later, so start by defining each of the component kernel, assigning them to variables k1 and k2, and then define the overal kernel k as the sum of the two. Then define the GP object itself.
Can you write Python code for it?

k1 = # complete
k2 = # complete
k = # complete
gp = # complete
Xobs = # complete


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Problem 3a: Initial guesses
Because the function goes up an down, you can probably guess that the likelihood surface is going to be highly multi-modal. So it's important to have a decent initial guess for the time delays.
A simple way to do obtain one would be by cross-correlation, but since the time-series are not regularly sampled (because of the small random term we added to each of the time arrays), we need to interpolate them onto a regular grid first. What better way to do this than with a GP? This will have the added advantage of giving us an initial estimate of the GP hyper-parameters too (we're assuming we don't know them either, though we will assume we know the white noise standard deviation).
First we need to define a GP object, based on a Matern 3/2 kernel with variable input scale and variance. Do this in the cell below.
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
The GP HPs aren't too far off either.
Problem 3b: Optimization
Now we have some initial guesses for the time-delays and the GP hyper-parameters, we're ready to model the time-series simultaneously, using a single GP. We need to write a new likelihood function to do this. The function will need to apply the delays to the times, before passing these times to george to evaluate the likelihood itself.
First let's define a function apply_delays, which will take the delays and the time array t as inputs, and return an $M \times N$ array of delayed times. This function will be called by the likelihood function, but it might be useful later for plotting the results too. It would also be useful for this function to warn us if the time-delays are such that one of the time-series no longer overlaps with the others at all, for example by returning a boolean variable that is true if all is well, but false if not.
Complete the definition below.
Can you write Python code for it?

def apply_delays(delays,t2d):
    t_delayed = np.copy(t2d)
    for i, delay in enumerate(delays):
        t_delayed[i+1,:] # complete
    ok = True
    M = len(delays) + 1
    for i in range(M):
        # complete
    return t_delayed, ok


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
There is no simple analytical way to evaluate the gradient of the log likelihood with respect to the time delays, so we will not define a grad_neg_log_like function for this problem. The gradient descent optimizer will be slower, since it will have to evaluate the gradients numerically, but for such a small dataset it doesn't matter.
Ok, now we are ready to run the optimizer. Like before, we can use the minimize function from scipy.optimize.
Can you write Python code for it?

# concatenated array of true time delays and GP HPs
ptrue = # complete
# same for the initial guesses
p0 = # complete
print('Initial guesses')
print(p0)
result = minimize(# complete
# save ML values in p1
p1 = # complete
print('ML parameters')
print(p1)
print('True parameters')
print(ptrue)


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Visualising the confidence intervals is a bit tricky in 3-D so we'll skip that. We could use emcee to explore the posterior distribution of the hyper-parameters, but we will leave that for a more realistic example.
Problem 2: Separable functions
In the above problem we were modelling a non-separable function of $x$ and $y$ (because of the cross-term in the polynomial). Now we will model a separable function, and use a GP with a sum rather than a product of kernels to separate the dependence on each of the input variable.
This exploits the fact that GPs preserve additivity. In other words, a GP with a sum of kernels, each depending on a disjoint subset of the inputs, sets up a probability distribution over functions that are sums of functions of the individual subsets of inputs. This is how the K2SC pipeline (for removing pointing systematics in K2 data) discussed in the lectures works.
As ever, we start by simulating a dataset. Execute the cell below.
Can you write Python code for it?

N = 100
xobs = np.random.uniform(-5,5,N)
yobs = np.random.uniform(-5,5,N)
zobs = -0.05 * xobs**2 + np.sin(yobs)
eobs = 0.01
zobs += np.random.normal(0,eobs,len(xobs))
plt.scatter(xobs, yobs, c=zobs, s=20, marker='.')
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
cb = plt.colorbar()
cb.set_label(r'$z$');


I am working on the file "Sessions/Session04/Day3/GPTutorial2.ipynb".
The first task is:
Your task is to devise and implement an algorithm that will schedule observations, based on the data to date, so as to ensure the uncertainty on the value of the function at any one time never exceeds 0.1. At each step, the aglorithm should:
- train a GP on the data acquired so far. You may assume the form of the covariance function is known, as is the output variance, so there are only two hyper-parameters to fit (the log period of the cosine kernel and the metric of the squared exponential term).
- make predictions for future values. If you're being clever, you can do this sequentially so you only look ahead a small time interval at a time, and stop as soon as the uncertainty exceeds the desired bound.
- use this to decide when to take the next observation
- add the next observation (by sampling the "true" values at the appropriate time and adding noise with the same distribution as above)
- repeat till the end time is reached.
Of course you will need to test your algorithm by comparing the predictions to the true values.
Can you write Python code for it?

gp = george.GP(k)
gp.set_parameter_vector([np.log(10),np.log(1000)])
gp.compute(xobs,yerr=eobs)
def nll(p):
    gp.set_parameter_vector(p)
    return -gp.log_likelihood(yobs)
def gnll(p):
    gp.set_parameter_vector(p)
    return -gp.grad_log_likelihood(yobs)
result = minimize(nll, gp.get_parameter_vector(), jac=gnll)
print(result)
gp.set_parameter_vector(result.x)
ypred, epred = gp.predict(yobs, xtrue, return_var=True)
plt.plot(xtrue,ytrue)
plt.errorbar(xobs,yobs,yerr=eobs,fmt='.',capsize=0);
plt.fill_between(xtrue,ypred + epred, ypred-epred,alpha=0.2,edgecolor='none')


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
As you can see, the delays estimated in this way aren't too far off.
To get initial guesses for the GP hyper-parameters, we can take the mean of the best-fit values from the three individual time-series. Do this in the cell below.
Can you write Python code for it?

gppar_0 = np.mean(p1,axis=0)
print('Estimated GP HPs')
print(gppar_0)
print('True GP HPs')
print(gppar_true)


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
As you can see, the optimization further improved our estimates of the time delays and the GP HPs. But how much can we trust these? Let's evaluate posterior uncertainties using MCMC.
Hyper-parameter marginalisation.
We now use MCMC to obtain uncertainty estimates, or confidence intervals, for the model hyper-parameters.
First we need to define the posterior function to pass to the emcee sampler. We will use improper, flat priors over all the parameters, so the posterior probability is just a trivial wrapper around our neg_ln_like_delays function. Complete the definition below:
Can you write Python code for it?

def lnprob(p):
    return -neg_ln_like_delays(p)


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Execute the cell below to plot contours of the  predictive mean alongside the data.
Can you write Python code for it?

plt.scatter(xobs, yobs, c=zobs, s=20, marker='.')
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
cb = plt.colorbar()
cb.set_label(r'$z$');
plt.contour(X2D,Y2D,Z2D);


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Gaussian Process regression tutorial 2: Solutions
In this tutorial, we are to explore some slightly more realistic applications of GPs to astrophysical (or at least, astronomy-like) datasets. We will do this using the popular george package by Daniel Foreman-Mackey.

By S Aigrain (University of Oxford)
In this tutorial, we are to explore some slightly more realistic applications of GPs to astrophysical (or at least, astronomy-like) datasets.
We will do this using the popular george package by Daniel Foreman-Mackey. george doesn't have all the functionality of more general packages such as GPy and scikit-learn, but it still has a nice modelling interface, is easy to use, and is faster than either of the other two.
We will also use another of Dan's packages, emcee to explore posterior probabilities using MCMC, and his corner.py module to plot the resulting parameter samples.
Required packages
Why george? george doesn't have all the functionality of GPy, but it is easy to use, and is faster than either of the other two. And I'm more familiar with it.
We will also use another of Dan's packages, emcee to explore posterior probabilities using MCMC, and his corner.py module to plot the resulting parameter samples.
Before you start, make sure you have the latest stable version of these packages installed. If you used george before, note the API has changed significantly between versions 0.2.x and 0.3.0.
The easiest way to install all three packages is with pip:
pip install emcee
pip install george
pip install corner
Full documentation is available here:
- https://george.readthedocs.io/
- https://emcee.readthedocs.io/
- https://corner.readthedocs.io/
Can you write Python code for it?

%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
import george, emcee, corner
from scipy.optimize import minimize


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Hopefully the distributions should look reasonable and be consistent with the true values.
We need to extract confidence intervals for the parameters from the MCMC chain, which we can access through sampler.flatchain
Can you write Python code for it?

samples = sampler.flatchain[:]
# The GP parameters were explored in log space, return them to linear space
#samples[:, -2:] = np.exp(samples[:, -2:])
# This handy bit of code will extract median and +/- 1 sigma intervals for each parameter
pv = map(lambda v: (v[1], v[2]-v[1], v[1]-v[0]), zip(*np.percentile(samples, [16, 50, 84], axis=0)))
# Print the results
for i in range(ndim):
    pval = pv[i]
    print("Param {}: {:5.2f} +{:4.2f} -{:4.2f} (true: {:5.2f})".format(i+1,pval[0], pval[1], pval[2], ptrue[i]))


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Hopefully, the MCMC estimates should be consistent with the true values...
Challenge problem: Active scheduling
Imagine you are monitoring a particular variable, you want to know its value to a given precision at anyone time, but each observation is costly, so you don't want to take any more than you have to. You can train a GP on the first few observations, then use the predictive distribution to work out when your uncertainty about the current value of the variable is so large that you need to take a new observation. Use the new observation to update the GP hyper parameters and the predictive distribution, and repeat the process...
First we generate a tightly sampled time series over 100 days. This will represent the "true" value of the variable. We will include some periodic behaviour as that makes the problem more interesting. Then we will "observe" 1 point per day for the first 20 days.
Can you write Python code for it?

xtrue = np.linspace(0,100,1000)
k = george.kernels.CosineKernel(np.log(12.3)) * george.kernels.ExpSquaredKernel(1000.0)
ytrue = george.GP(k).sample(xtrue)
xobs = xtrue[:200:10]
eobs = 10.0**(np.random.uniform(-1.5,-1,20))
yobs = ytrue[:200:10] + np.random.normal(0,1,20) * eobs
plt.plot(xtrue,ytrue)
plt.errorbar(xobs,yobs,yerr=eobs,fmt='.',capsize=0);


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Now assign those best-fit values to the parameter vector
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Now let's plot the predictive distribution to check it worked ok. You can just copy and paste code from Problem 1.
Can you write Python code for it?

zpred = gp.predict(zobs, Xpred, return_var=False, return_cov=False)
Z2D = zpred.reshape(X2D.shape)
plt.scatter(xobs, yobs, c=zobs, s=20, marker='.')
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
cb = plt.colorbar()
cb.set_label(r'$z$');
plt.contour(X2D,Y2D,Z2D);


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Now we are ready to cross-correlate the interpolated time-series. The easiest way to do this is using the function xcorr from matplotlib.pyplot. This function returns a tuple of 4 variables, the first two of which are the lags and corresponding cross-correlation values.
Can you write Python code for it?

dt = treg[1] - treg[0]
# Array to hold estimates of the time-delays
delays_0 = np.zeros(M-1)
for i in range(M-1):
    # use pyplot's xcorr function to cross-correlate yreg[i+1] with yreg[0]
    lags, corr, _, _ = plt.xcorr(yreg[0,:],yreg[i+1,:],maxlags=49,usevlines=False,marker='.',color=c[i+1])
    # find the lag that maximises the CCF, convert it to time delay, save in delays_0 array
    lmax = lags[np.argmax(corr)]
    plt.axvline(lmax,color=c[i+1])
    delays_0[i] = dt * lmax
plt.xlabel('lag')
plt.ylabel('x-correlation');
# Compare estimated to true delays
print('Estimated time delays from cross-correlation')
print(delays_0)
print('True delays')
print(delays_true)


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Now we are ready to define the likelihood function itself. The likelihood should accept a parameter array consisting of the shifts first, and then the GP hyper-parameters, and make use of the output of apply_delays to return a very high number if the time delays are unreasonable. Complete the definition below.
Can you write Python code for it?

def neg_ln_like_delays(p):
    delays = p[:-2]
    t_delayed, ok = apply_delays(delays,t2d)
    if not ok:
        return 1e25
    gp.set_parameter_vector(p[-2:])
    gp.compute(t_delayed.flatten(), yerr=wn)
    return -gp.log_likelihood(y2d.flatten())


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Now we are ready to run the MCMC, starting with a burn-in chain of 500 steps, after which we reset the sampler, and run the sampler again for 100 iterations. Complete the code below.
Can you write Python code for it?

print("Running burn-in...")
p2, _, _ = sampler.run_mcmc(p2, 50)
sampler.reset()

print("Running production...")
sampler.run_mcmc(p2, 500);


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Now we need to fit each time-series in turn, and compute the mean of the predictive distribution over a tightly sampled, regular grid of time values. If you take care to name our variables right, you can reuse the neg log likelihood and associated gradient functions from Problem 1.
Complete the code below and run it
Can you write Python code for it?

p0 = gp.get_parameter_vector()
# 2-D array to hold the best-fit GP HPs for each time-series
p1 = np.tile(p0,(3,1))
# Regularly sampled time array
treg = np.linspace(0,10,100)
# 2-D array to hold the interpolated time-series
yreg = np.zeros((3,100))
c = ['r','g','b']
for i in range(M):
    # Compute the gp on the relevant subset of the 2-D time array t2d
    gp.compute(t2d[i,:].flatten(),yerr=wn)
    # Assign the corresponding y values to the variable zobs
    # (this is the one that neg_ln_like uses to condition the GP)
    zobs = y2d[i,:].flatten()
    # Optimize the likelihood using minimize
    result = minimize(neg_ln_like, p0, jac=grad_neg_ln_like)
    # Save the best-fit GP HPs in p1
    p1[i,:] = result.x
    # update the GP parameter vector with the best fit values
    gp.set_parameter_vector(result.x)
    # evaluate the predictive mean conditioned on zobs at locations treg and save in yreg
    yreg[i,:] = gp.predict(zobs,treg,return_var=False,return_cov=False)
    # you might want to plot the results to check it worked
    plt.plot(t2d[i,:],y2d[i,:],'.',c=c[i])
    plt.plot(treg,yreg[i,:],'-',c=c[i])
# And let's print the GP HPs to see if they were sensible.
print('Individual GP fits: best-fit HPs')
print(p1)


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Now you will need to tell the GP object what inputs the covariance matrix is to be evaluated at. This is done using the compute method. 2-D inputs need to be passed as an $N \times 2$ array, which you will need to construct from the two 1-D arrays of $x$- and $y$-values we generated earlier. The second argument of compute should be the white noise standard deviation.
Can you write Python code for it?

Xobs = np.concatenate([[xobs],[yobs]]).T
gp.compute(Xobs, yerr=eobs)


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Problem 1a: setting up the GP
Now we will construct the GP model using george. We will use a with different length scales in each of the two dimensions. To set this up in george, you have to multiply two individual kernels together, like that:
k = a * KernelName(b, ndim = 2, axes = 0) * KernelName(c, ndim = 2, axes = 1)
Here KernelName stands for the name of the kernel used (in george, the squared exponential kernel is called ExpSquaredKernel), a is the output variance, b is the metric, or length scale, applied to the first input dimension, and c to the second.
Note this is equivalent to the parametrisation used in the lectures:
$$
k(x,x') = A \exp \left[ - \Gamma (x-x')^2\right] = A \exp \left[ - (x-x')^2/m^2\right]
$$
with $\Gamma=1/m^2$.
Go ahead and define the kernel in the cell below, with some ball park values for the hyper-parameters (by ball-park, I mean not too many orders of magnitudes off). Then create a GP object using that kernel.
Can you write Python code for it?

k = 1.0 * george.kernels.ExpSquaredKernel(1.0, ndim = 2, axes = 0) * george.kernels.ExpSquaredKernel(1.0, ndim = 2, axes = 1)
gp = george.GP(k)


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Problem 1b: Optimizing the likelihood
Following the example in the first george tutorial, define a simple neg log likelihood function, and a function to evaluate its gradient.
Can you write Python code for it?

def neg_ln_like(p):
    gp.set_parameter_vector(p)
    return -gp.log_likelihood(zobs)

def grad_neg_ln_like(p):
    gp.set_parameter_vector(p)
    return -gp.grad_log_likelihood(zobs)


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Problem 2a: Joint model
We start, once again, by defining the GP object. The kernel will consist of a sum of 2 squared exponentials, one applied to each dimension. It will be useful to be able to access each of the kernel objects separately later, so start by defining each of the component kernel, assigning them to variables k1 and k2, and then define the overal kernel k as the sum of the two. Then define the GP object itself.
Can you write Python code for it?

k1 = 1.0 * george.kernels.ExpSquaredKernel(1.0, ndim = 2, axes = 0)
k2 = 1.0 * george.kernels.ExpSquaredKernel(1.0, ndim = 2, axes = 1)
k = k1 + k2
gp = george.GP(k)
Xobs = np.concatenate([[xobs],[yobs]]).T


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Problem 2b: Separating the components
We now come to evaluating the predictive means for the individual components. The standard expression for the predictive mean is:
$$
\overline{\boldsymbol{y}} = K(\boldsymbol{x}_,\boldsymbol{x}) K(\boldsymbol{x},\boldsymbol{x})^{-1} \boldsymbol{y}
$$
The predictive mean for a given component of the kernel is obtained simply by replacing the first instance of the covariance matrix between test and training points, $K(\boldsymbol{x},\boldsymbol{x})$, by the corresponding matrix for the component in question only:
$$
\overline{\boldsymbol{y}}_{1,} = K_1(\boldsymbol{x}_*,\boldsymbol{x}) K(\boldsymbol{x},\boldsymbol{x})^{-1} \boldsymbol{y}.
$$
george doesn't provide a built-in function to do this, but
- the GP object has a method apply_inverse, which evaluates and returns the product $K(\boldsymbol{x},\boldsymbol{x})^{-1} \boldsymbol{y}$ for a given vector of training set outputs $\boldsymbol{y}$,
- the kernel object has a method get_value, which evaluates the covariance matrix for a given set of inputs.
Use these two functions to evaluate the two components of the best-fit GP model in our problem. Store the $x$- and $y$ components in variables fx and fy, respectively.
Hint: The apply_inverse method does what it says in the name, i.e. it modifies its argument by pre-multiplying it by the inverse of the covariance matrix. Therefore, you need to pass it a copy of the vector of obserced outputs, not the original.
Can you write Python code for it?

b = np.copy(zobs)
gp.apply_inverse(b)
K1 = k1.get_value(Xobs)
fx = np.dot(K1,b)
K2 = k2.get_value(Xobs)
fy = np.dot(K2,b)


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Problem 3a: Initial guesses
Because the function goes up an down, you can probably guess that the likelihood surface is going to be highly multi-modal. So it's important to have a decent initial guess for the time delays.
A simple way to do obtain one would be by cross-correlation, but since the time-series are not regularly sampled (because of the small random term we added to each of the time arrays), we need to interpolate them onto a regular grid first. What better way to do this than with a GP? This will have the added advantage of giving us an initial estimate of the GP hyper-parameters too (we're assuming we don't know them either, though we will assume we know the white noise standard deviation).
First we need to define a GP object, based on a Matern 3/2 kernel with variable input scale and variance. Do this in the cell below.
Can you write Python code for it?

k = 1.0 * george.kernels.Matern52Kernel(3.0)
gp = george.GP(k)


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
The GP HPs aren't too far off either.
Problem 3b: Optimization
Now we have some initial guesses for the time-delays and the GP hyper-parameters, we're ready to model the time-series simultaneously, using a single GP. We need to write a new likelihood function to do this. The function will need to apply the delays to the times, before passing these times to george to evaluate the likelihood itself.
First let's define a function apply_delays, which will take the delays and the time array t as inputs, and return an $M \times N$ array of delayed times. This function will be called by the likelihood function, but it might be useful later for plotting the results too. It would also be useful for this function to warn us if the time-delays are such that one of the time-series no longer overlaps with the others at all, for example by returning a boolean variable that is true if all is well, but false if not.
Complete the definition below.
Can you write Python code for it?

def apply_delays(delays,t2d):
    t_delayed = np.copy(t2d)
    for i, delay in enumerate(delays):
        t_delayed[i+1,:] += delay
    ok = True
    M = len(delays) + 1
    for i in range(M):
        tc = t_delayed[i,:]
        to = t_delayed[np.arange(M)!=i,:]
        if (tc.min() > to.max()) + (tc.max() < to.min()):
            ok = False
    return t_delayed, ok


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
There is no simple analytical way to evaluate the gradient of the log likelihood with respect to the time delays, so we will not define a grad_neg_log_like function for this problem. The gradient descent optimizer will be slower, since it will have to evaluate the gradients numerically, but for such a small dataset it doesn't matter.
Ok, now we are ready to run the optimizer. Like before, we can use the minimize function from scipy.optimize.
Can you write Python code for it?

ptrue = np.concatenate([delays_true,gppar_true])
p0 = np.concatenate([delays_0,gppar_0])
print('Initial guesses')
print(p0)
result = minimize(neg_ln_like_delays, p0)
p1 = np.array(result.x)
print('ML parameters')
print(p1)
print('True parameters')
print(ptrue)


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Using the best-fit hyper-parameters, evaluate the mean of the predictive distribution at the grid locations. The output will be a 1-D array, which you will need to reshape so it has the same shape as X2D and Y2D for plotting.
Can you write Python code for it?

zpred = gp.predict(zobs, Xpred, return_var=False, return_cov=False)
Z2D = zpred.reshape(X2D.shape)


I am working on the file "Sessions/Session04/Day3/GPTutorial2_WithSolutions.ipynb".
The first task is:
Your task is to devise and implement an algorithm that will schedule observations, based on the data to date, so as to ensure the uncertainty on the value of the function at any one time never exceeds 0.1. At each step, the aglorithm should:
- train a GP on the data acquired so far. You may assume the form of the covariance function is known, as is the output variance, so there are only two hyper-parameters to fit (the log period of the cosine kernel and the metric of the squared exponential term).
- make predictions for future values. If you're being clever, you can do this sequentially so you only look ahead a small time interval at a time, and stop as soon as the uncertainty exceeds the desired bound.
- use this to decide when to take the next observation
- add the next observation (by sampling the "true" values at the appropriate time and adding noise with the same distribution as above)
- repeat till the end time is reached.
Of course you will need to test your algorithm by comparing the predictions to the true values.
Can you write Python code for it?

gp = george.GP(k)
gp.set_parameter_vector([np.log(10),np.log(1000)])
gp.compute(xobs,yerr=eobs)
def nll(p):
    gp.set_parameter_vector(p)
    return -gp.log_likelihood(yobs)
def gnll(p):
    gp.set_parameter_vector(p)
    return -gp.grad_log_likelihood(yobs)
result = minimize(nll, gp.get_parameter_vector(), jac=gnll)
print(result)
gp.set_parameter_vector(result.x)
ypred, epred = gp.predict(yobs, xtrue, return_var=True)
plt.plot(xtrue,ytrue)
plt.errorbar(xobs,yobs,yerr=eobs,fmt='.',capsize=0);
plt.fill_between(xtrue,ypred + epred, ypred-epred,alpha=0.2,edgecolor='none')


I am working on the file "Sessions/Session04/Day4/1. HBM Truncated Gaussian Population Model.ipynb".
The first task is:

Data wrangling notes:

The pyjags output is a dictionary type (above, this is asigned to the "samples" variable). Here, the keys are e_sigma, h, and k (from the one-component analysis model used above). Furthermore,

 e_sigma has shape (1,10000,4),

h has shape (25,10000,4), and

The way to acces the latent variable marginal posteriors before updating the samples dictionary as we do below, you would use:

h1 = samples['h'][0,:,:], h2 = samples['h'][1,:,:],  ... hn = samples['h'][n,:,:] and,

for n population constituents.

Now, we need to add keys for each latent variable into the dictionary.
The keys we want here are:

 e_sigma (the population parameter here), and

h1, ..., h25 for the latent variables.

Our resulting dictionary elements will have the following shapes:

 e_sigma will have shape (10000,4)

h1 will have shape (10000,4)

.

.

.

h25 will have shape (10000,4)

for a data set with 25 population constituents.

Can you write Python code for it?

#print(samples)
#print(samples.items())

## Print and check the shape of the resultant samples dictionary:
print(samples_JAGS['e_sigma'].shape)
print(samples_JAGS['e_sigma'].squeeze(0).shape)
print(samples_JAGS['h'].shape)
print(samples_JAGS['h'][0,:,:].shape)
print('-----')

## Update the samples dictionary so that it includes keys for the latent variables
## Also, we will use LaTeX formatting to help make legible plots ahead.
samples_Nm1_JAGS = {}

## adjust the thin varible to only look at every 10th population element by setting it to 10
thin = 1
## Need to enter the number of hyperparameter variables here:
numHyperParams = 1
## Specify the dimension we want for our plot below, for legibility.
dim = (Ndata/thin)*2 + numHyperParams
print(dim)

for i in np.arange(0,Ndata,thin):
    #samples_Nm1({hval: samples['h'][i,:,:]})
    samples_Nm1_JAGS.update({'$h_{'+str(i)+'}$': samples_JAGS['h'][i,:,:]})
#print(samples_2['h11'].shape)

## Add the hyperparameter marginal posterior back in:
samples_Nm1_JAGS.update({'$e_{\sigma}$': samples_JAGS['e_sigma'].squeeze(0)})

## Below, examine the updated and reformatted sample dictionary to include keys for
## latent variables
for j, i in samples_Nm1_JAGS.items():
    print(j)
    print(i)
samples_Nm1_JAGS['$h_{5}$'][0]


I am working on the file "Sessions/Session04/Day4/1. HBM Truncated Gaussian Population Model.ipynb".
The first task is:

Below is code to look at summary statistics of the marginal posterior distributions  (the probabilistic parameter estimates)
for the hyperparameter and the latent variables
(each population constituent), in this case h and k (a.k.a projected eccentricity here), of the exoplanet systems we are simulating).

Can you write Python code for it?

## equal tailed 95% credible intervals, and posterior distribution means:
def summary(samples_JAGS, varname, p=95):
    values = samples_JAGS[varname][0]
    ci = np.percentile(values, [100-p, p])
    print('{:<6} mean = {:>5.1f}, {}% credible interval [{:>4.1f} {:>4.1f}]'.format(
      varname, np.mean(values), p, *ci))

for varname in samples_Nm1_JAGS:
    summary(samples_Nm1_JAGS, varname)

## Use pandas three dimensional Panel to represent the trace:

trace = pd.Panel({k: v for k, v in samples_Nm1_JAGS.items()})
trace.axes[0].name = 'Variable'
trace.axes[1].name = 'Iteration'
trace.axes[2].name = 'Chain'

## Point estimates:
print(trace.to_frame().mean())

## Bayesian equal-tailed 95% credible intervals:
print(trace.to_frame().quantile([0.05, 0.95]))
  ## ^ entering the values here could be a good question part

def plot(trace, var):
    fig, axes = plt.subplots(1, 3, figsize=(9, 3))
    fig.suptitle(var, y=0.95, fontsize='xx-large')

    ## Marginal posterior density estimate:
    trace[var].plot.density(ax=axes[0])
    axes[0].set_xlabel('Parameter value')
    axes[0].locator_params(tight=True)

    ## Autocorrelation for each chain:
    axes[1].set_xlim(0, 100)
    for chain in trace[var].columns:
        autocorrelation_plot(trace[var,:,chain], axes[1], label=chain)

    ## Trace plot:
    axes[2].set_ylabel('Parameter value')
    trace[var].plot(ax=axes[2])

    ## Save figure
    filename = var.replace("\\", "")
    filename = filename.replace("$", "")
    filename = filename.replace("}", "")
    filename = filename.replace("{", "")
    plt.tight_layout(pad=3)
    fig.savefig('Nm1_JAGS_'+'{}.png'.format(filename))

# Display diagnostic plots
for var in trace:
    plot(trace, var)

## Scatter matrix plot:

## Redefine the trace so that we only vizualize every 10th latent variable element in
## the scatter_matrix plot below. Vizualizing all 50 is too cumbersome for the scatter
## matrix.

samples_Nm1_for_scatter_matrix = {}
start = int(iters-1000)

## adjust the thin varible to...


I am working on the file "Sessions/Session04/Day4/1. HBM Truncated Gaussian Population Model.ipynb".
The first task is:

 1. Generate simulated data set from a truncated Gaussian generative model

Below we designate the population values of our generative model. These are the truths that we should recover if our hiararchical Bayesian model is properly specified and diagnostics have indicated that the simulation has "not-not converged". "You can't
prove convergence, at best you can fail to prove a failure to converge".

Set your one-component truncated Gaussian mixture simulated data hyperparameters:

Can you write Python code for it?

## In this simulated data set, there are "Ndata" planetary systems (with one planet each)
Ndata = 25

## Here we asign the dispersion of the simulated population to be 0.3, this is
## the truth we wish to recover when we run our diagnostics. In other words, this is the
## spread of projected eccentricities about the mean (In this case the values are symteric
## about zero, so the mean of the population is zero, and we can treat this paramter as
## known, simplyfying our model based on our science question!)
sigmae = 0.3

## We approximate the uncertainty for each measurement as normally distributed about a
## reported measurement point estimate, or a summary statistics for a posterior estimate.
sigmahobs = 0.04


I am working on the file "Sessions/Session04/Day4/1. HBM Truncated Gaussian Population Model.ipynb".
The first task is:
<h3 align="left"> 2. Create PyStan truncated Gaussian hierarchical Bayesian model. </h3>

Run the code block below to pre-compile the Stan model.
Can you write Python code for it?

eccmodel = """
data {
    int<lower=1> Ndata;
    real<lower=-1,upper=1> hhat[Ndata];
    real<lower=0,upper=1> hhat_sigma[Ndata];
}
parameters {
    real<lower=0> e_sigma;
    real<lower=-1,upper=1> h[Ndata];
}
model {

    e_sigma ~ uniform(0, 1.0);

    for (n in 1:Ndata)
      hhat[n] ~ normal(h[n], hhat_sigma[n]);

    for (n in 1:Ndata)
      increment_log_prob(normal_log(h[n], 0.0, e_sigma));
}
"""
# Compiled Stan Model
sm = pystan.StanModel(model_code=eccmodel)


I am working on the file "Sessions/Session04/Day4/1. HBM Truncated Gaussian Population Model.ipynb".
The first task is:
<h3 align="left"> 3. Perform diagnostics to asses if the model shows signs of having not converged. </h3>
Can you write Python code for it?


I am working on the file "Sessions/Session04/Day4/1. HBM Truncated Gaussian Population Model.ipynb".
The first task is:

Next we generate our true values and our simulated measurements:

Can you write Python code for it?

h = np.repeat(0.,Ndata) ## Simuated true values of h = e cos w
hhat = np.repeat(0.,Ndata) ## Simulated measurements of h
hhat_sigma  = np.repeat(sigmahobs,Ndata) ## measurement uncertainty summary statics

## Note: in this simulated data set, the measurement uncertainties for each projected eccentricity are
## set to be the same.  You can use the real heteroscadastic uncertainties from your
## real data set when testing with simulated data in the future.

for i in range(0,Ndata):
    h[i] = rnorm_bound(1,0,sigmae,lower_bound=-1,upper_bound=1)
    hhat[i] = rnorm_bound(1,h[i],sigmahobs,lower_bound=-1,upper_bound=1)

## Vizualize the true data values, and the simulated measurements:
print(h, hhat)
plt.hist( h, label='h')
#plt.hist( ##Complete, label='h')
plt.hist(hhat, label='hhat')
#plt.hist( ##Complete, label='hhat')
plt.legend()


I am working on the file "Sessions/Session04/Day4/1. HBM Truncated Gaussian Population Model.ipynb".
The first task is:
How is the truncation specified in the JAGS model above?
Can you write Python code for it?

## Load additional JAGS module
pyjags.load_module('glm')
pyjags.load_module('dic')

## See blog post for origination of the adapted analysis tools used here and below:
## https://martynplummer.wordpress.com/2016/01/11/pyjags/

num_chains = 4
iterations = 10000

## data list include only variables in the model
model = pyjags.Model(code, data=dict( Ndata=Ndata, hhat=hhat,
                                     hhat_sigma=hhat_sigma),
                     chains=num_chains, adapt=1000)

## Code to speed up compute time. This feature might not be
## well tested in pyjags at this time.
## threads=4, chains_per_thread=1

## 500 warmup / burn-in iterations, not used for inference.
model.sample(500, vars=[])

## Run model for desired steps, monitoring hyperparameter variables, and latent variables
## for hierarchical Bayesian model.
## Returns a dictionary with numpy array for each monitored variable.
## Shapes of returned arrays are (... shape of variable ..., iterations, chains).
## samples = model.sample(#iterations per chain here, vars=['e_sigma', 'h'])
samples_JAGS = model.sample(iterations, vars=['e_sigma', 'h'])

## Code to save, open and use pickled dictionary of samples:
## -- Pickle the data --
#with open('ecc_1_test.pkl', 'wb') as handle:
#   pickle.dump(samples, handle)
## -- Retrieve pickled data --
#with open('ecc_1_test.pkl', 'rb') as handle:
#   retrieved_results = pickle.load(handle)


I am working on the file "Sessions/Session04/Day4/1. HBM Truncated Gaussian Population Model.ipynb".
The first task is:
Overplot the true value and the measurement uncertainy with the marginal posterior.
Can you write Python code for it?

def plot_2(Ndata):
    for i in np.arange(0,Ndata):
        fig = plt.figure(figsize=(8,5))
        x = np.arange(-1,1, 0.001)
        fit = stats.norm.pdf(x, h[i], sigmahobs)
        plt.plot([hhat[i],hhat[i]], [0,10],label=r'$\hat{h_{'+str(i)+'}}$ (measured)')
        plt.plot([h[i],h[i]], [0,10],label=r'$h_{'+str(i)+'}$ (true value)')
        plt.plot(x,fit,'-',label=r'$N(h_{'+str(i)+'},\sigma_{\hat{h_{'+str(i)+'}}})$')
        plt.hist(samples_Stan['h'][:,i],histtype='step', bins=50, normed=True,label=r'Marginal Posterior of $h_{'+str(i)+'}$')
        plt.xlim(-0.7,0.7)
        plt.ylim(0,13)
        plt.legend()
        plt.text(-.49, 12,'variance of posterior  = '+str(np.var(samples_Stan['h'][:,i])))
        plt.text(-.49, 11.5,'variance of observation = '+str(sigmahobs**2))
        plt.tight_layout(pad=3)
        fig.savefig('Nm1_latent_var_h_Stan'+str(i)+'.png')

plot_2(Ndata)


I am working on the file "Sessions/Session04/Day4/1. HBM Truncated Gaussian Population Model.ipynb".
The first task is:
Plot draws from the posterior to summarize the population distribution:
Can you write Python code for it?

## take some some values of e_sigma from the posterior of e_sigma, at plot the population
## distribution for these values. These should be normal distributions centered at 0.0 with dispersions that are drawn from e_sigma.


I am working on the file "Sessions/Session04/Day4/1. HBM Truncated Gaussian Population Model.ipynb".
The first task is:
Where in the Stan model above where the truncation is indicated?:
Run the code cell below to carry out the HMC numerical simulation of the Stan model that was compiled above.
Can you write Python code for it?

ecc_dat = {'Ndata': len(hhat), 'hhat': hhat, 'hhat_sigma': hhat_sigma}

iters = 10000
fit = sm.sampling(data=ecc_dat, iter=iters, chains=4, seed=48389, refresh=1000, n_jobs=-1)


I am working on the file "Sessions/Session05/Day1/NoiseProblem.ipynb".
The first task is:
I've provided you with two datasets ("bias.npz" and "bias2.npz") which are the trimmed image of a CCD from the PFS project.  One has had a bias frame subtracted, one has not.
You can read the data into an Image, and the Image object is also able to return a view of the data from a single
amplifier (in the range 0..7).  For example, you can say:
Can you write Python code for it?

bias = Image(os.path.join(dataDir, "bias.npz"))

aim = bias.getAmpImage(3)

plt.clf()
imshow(aim)     # or you can use plt.imshow if you'd prefer;  imshow is imported from rhlUtils
plt.show();


I am working on the file "Sessions/Session05/Day1/PSFProblem.ipynb".
The first task is:
Just to make sure that we're all on the same page, here's code to display the image and stars (imshow is a simple utility imported from rhlUtils -- feel free to use plt.imshow if you'd rather)
Can you write Python code for it?

plt.figure(1)
plt.clf()
imshow(image, vmin=0, vmax=1000)
plt.title("Data")

plt.plot(calibs[:, 0], calibs[:, 1], '+')   # calibs[:, 2] contains the object's magnitude (not flux)

plt.show()


I am working on the file "Sessions/Session05/Day1/PSFProblem.ipynb".
The first task is:
OK, now use your PSF model to create an image of the residuals created by subtracting the scaled PSF from the stars.
How does it look?  Do you see what's going on?  Remember, I said that the PSF isn't a function of position, but CCDs aren't perfect.
Can you write Python code for it?

image = image0.copy()

# ...

plt.figure(3)
plt.clf()
imshow(image, vmin=image.min(), vmax=image.max()) # , vmin=0, vmax=100)
plt.title("Residuals")
plt.show();


I am working on the file "Sessions/Session05/Day1/PSFProblem.ipynb".
The first task is:
Time for you to do some work.  Write some code to estimate a PSF model by simply averaging all the objects, giving each equal weight.  You should add the option to use only some subset of the stars (e.g. the faintest 25%).
Your model should simply be an image of the PSF, normalised to have a maximum value of 1.0
You could use the calibs object to find all the stars, but in the real world there are stars in the data that are
not in the catalogue so you'll have to write a simple object finder (i.e. don't use the calibs!).  It's sufficient to find pixels that are larger than any neighbours to the left, right, top, or bottom; no, this isn't quite how you'd do it in the real world but it's not far off.
I told you that there were three data sets, taken in different places on the sky and with CCDs with different properties.  Choose one set (e.g. "stars_0.npz") to carry out the following activities, then go back and look at the other two.
Can you write Python code for it?

image = image0.copy()

#...

plt.figure(2)
plt.clf()
imshow(psfIm, vmin=0, vmax=1.1)
plt.title("PSF model")
plt.show();


I am working on the file "Sessions/Session05/Day1/PSFProblem.ipynb".
The first task is:
We have provided three images containing stars, taken with 3 different CCDs, in "stars_X.npz" where X = 0, 1, 2.
The true magnitudes of all the stars are also known, taken from a reliable reference catalogue.
OK, I'll be honest; these are simulations.  The stars' centers are all at centres of pixels
make your lives easier; there is no spatial structure in the PSF; and there is also no noise added of any kind.  You may buy me a drink later.
To read the image data and the calibs say something like:
Can you write Python code for it?

data = np.load(os.path.join(dataDir, "stars_0.npz"))
image, calibs = data["image"], data["calibs"]

image0 = image.copy()    # Keep a copy


I am working on the file "Sessions/Session05/Day1/PSFProblemSolutions.ipynb".
The first task is:
A powerful diagnostic is to measure the fluxes of the objects in different ways, and compare with the truth (i.e. the
magnitudes given in the calibs data structure).
For the purpose of this exercise you should use:
- aperture fluxes (the sum of all the pixels in a square centred on the star is good enough)
- psf fluxes (the flux in the best-fit PSF model, $\phi$.  The amplitude is simply $A = \frac{\sum \phi I}{\sum \phi^2}$, and the flux is $A \sum \phi$)
Convert them to magnitudes using the known zero-point mag0 (the magnitude corresponding to one count), and make suitable plots.
What do you think is going on?  Are there other measurements that you can make on the data to test your hypotheses?  Are there other observations that you'd like (I might be able to make them for you).
Can you write Python code for it?

def plotMags(image, calibs, yLabels=True):
    apFlux =   np.empty_like(xc, dtype='float')
    calibMag = np.empty_like(apFlux)
    psfFlux =  np.empty_like(apFlux)

    for i, (x, y) in enumerate(zip(xc, yc)):
        sub = image[y - psfSize//2:y + psfSize//2 + 1, x - psfSize//2:x + psfSize//2 + 1]

        apFlux[i] = sub.sum()
        psfFlux[i] = psfIm.sum()*np.sum(psfIm*sub)/np.sum(psfIm**2)

        delta = np.hypot(x - calibs[:, 0], y - calibs[:, 1])
        iDelta = np.argmin(delta)
        calibMag[i] = calibs[iDelta, 2]

    apMag = mag0 - 2.5*np.log10(apFlux)
    psfMag = mag0 - 2.5*np.log10(psfFlux)

    plt.plot(apMag, psfMag - apMag, 'o', label="psf")
    plt.plot(apMag, calibMag - apMag, 'o', label="ap")

    plt.ylim(-0.11, 0.11)
    if yLabels:
        plt.ylabel('xxMag - apMag');
    else:
        plt.gca().set_yticklabels([])

    plt.xlabel('apMag')

plotMags(image0, calibs0)
plt.legend();


I am working on the file "Sessions/Session05/Day1/PSFProblemSolutions.ipynb".
The first task is:
OK, now use your PSF model to create an image of the residuals created by subtracting the scaled PSF from the stars.
How does it look?  Do you see what's going on?  Remember, I said that the PSF isn't a function of position, but CCDs aren't perfect.
Can you write Python code for it?

image = image0.copy()

psfSize = psfIm.shape[0]
for x, y in zip(xc, yc):
    sub = image[y - psfSize//2:y + psfSize//2 + 1, x - psfSize//2:x + psfSize//2 + 1]

    sub -= psfIm*sub.sum()/psfIm.sum()

imshow(image, vmin=image.min(), vmax=image.max()) # , vmin=0, vmax=100)
plt.title("Residuals");


I am working on the file "Sessions/Session05/Day1/PSFProblemSolutions.ipynb".
The first task is:
Time for you to do some work.  Write some code to estimate a PSF model by simply averaging all the objects, giving each equal weight.  You should add the option to use only some subset of the stars (e.g. the faintest 25%).
Your model should simply be an image of the PSF, normalised to have a maximum value of 1.0
You could use the calibs object to find all the stars, but in the real world there are stars in the data that are
not in the catalogue so you'll have to write a simple object finder (i.e. don't use the calibs!).  It's sufficient to find pixels that are larger than any neighbours to the left, right, top, or bottom; no, this isn't quite how you'd do it in the real world but it's not far off.
I told you that there were three data sets, taken in different places on the sky and with CCDs with different properties.  Choose one set (e.g. "stars_0.npz") to carry out the following activities, then go back and look at the other two.

Here's my version of code do estimate the PSF.  I wrote my own naive object finder for DSFP;  in reality I'd call the one in the LSST stack (which has heritage stretching back to SDSS via PanSTARRS).
Can you write Python code for it?

def makePsfModel(image, pmin=0, pmax=100):
    """Return an image which is a model of the PSF

    Only use stars which lie within the pmin and pmax percentiles (inclusive)
    """
    yc, xc = np.where(np.logical_and.reduce([image > np.roll(image.copy(), (1,  0), (0, 1)),
                                             image > np.roll(image.copy(), (-1, 0), (0, 1)),
                                             image > np.roll(image.copy(), (0,  1), (0, 1)),
                                             image > np.roll(image.copy(), (0, -1), (0, 1)),
                                ]))
    I0 = image[yc, xc]

    psfSize = 15
    psfIm = np.zeros((psfSize, psfSize))
    for x, y, I in zip(xc, yc, I0):
        if I >= np.percentile(I0, [pmin]) and I <= np.percentile(I0, [pmax]):
            dpsf = image[y - psfSize//2:y + psfSize//2 + 1, x - psfSize//2:x + psfSize//2 + 1].copy()
            dpsf /= dpsf.max()

            psfIm += dpsf

    psfIm /= psfIm.max()

    return psfIm, xc, yc

psfIm, xc, yc = makePsfModel(image)
imshow(psfIm, vmin=0, vmax=1.1)
plt.title("PSF model");


I am working on the file "Sessions/Session05/Day1/PSFProblemSolutions.ipynb".
The first task is:
We have provided three images containing stars, taken with 3 different CCDs, in "stars_X.npz" where X = 0, 1, 2.
The true magnitudes of all the stars are also known, taken from a reliable reference catalogue.
OK, I'll be honest; these are simulations.  The stars' centers are all at centres of pixels
make your lives easier; there is no spatial structure in the PSF; and there is also no noise added of any kind.  You may buy me a drink later.
To read the image data and the calibs say something like:
Can you write Python code for it?

data = np.load(os.path.join(dataDir, "stars_0.npz"))
image, calibs = data["image"], data["calibs"]

image0, calibs0 = image.copy(), calibs    # Keep copies


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessing.ipynb".
The first task is:
Problem 1b
Plot the noise-free PSF for a star with $\mu = 10$ and $\mathrm{FWHM} = 3$. What is the flux of this star?
Can you write Python code for it?

x = # complete

plt.plot( # complete

print("The flux of the star is: {:.3f}".format( # complete


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessing.ipynb".
The first task is:
Problem 1c
Add Sky noise (a constant in this case) to your model. Define the sky as S, with total stellar flux F.
Plot the model for S = 100 and F = 500.
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessing.ipynb".
The first task is:
Problem 2) Add Noise
We will add noise to this simulation assuming that photon counting contributes the only source of uncertainty (this assumption is far from sufficient in real life). Within each pixel, $n$ photons are detected with an uncertainty that follows a Poisson distribution, which has the property that the mean $\mu$ is equal to the variance $\mu$. If $n \gg 1$ then $P(\mu) \approx N(\mu, \mu)$ [you can safely assume we will be in this regime for the remainder of this problem].
Problem 2a
Calculate the noisy flux for the simulated star in Problem 1c.
Hint - you may find the function np.random.normal() helpful.
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessing.ipynb".
The first task is:
Problem 3) Flux Measurement
We will now attempt to measure the flux from a simulated star.
Problem 3a
Write a function simulate() to simulate the noisy flux measurements of a star with centroid mu, FWHM fwhm, sky background S, and flux F.
Hint - it may be helpful to plot the output of your function.
Can you write Python code for it?

def simulate(# complete
    # complete
    # complete


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessing.ipynb".
The first task is:
Problem 4) PSF Flux measurement
In this problem we are going to use our knowledge of the PSF to estimate the flux of the star. We will compare these measurements to the aperture flux measurements above.
Problem 4a
Create the psf model, psf, which is equivalent to a noise-free star with fwhm = 5.
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessing.ipynb".
The first task is:
Problem 4b
Using the same parameters as problem 3, simulate a star and measure it's PSF flux.
Can you write Python code for it?

sim_star = simulate( # complete

psf_flux = # complete

print("The PSF flux is {:.3f}".format( # complete


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessing.ipynb".
The first task is:
Problem 4c
As before, write a Monte Carlo simulator to estimate the PSF flux of the star. How do your results compare to above?
Can you write Python code for it?

sim_fluxes = # complete
for # complete

print("The mean flux = {:.3f} with variance = {:.3f}".format( # complete


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessingSolutions.ipynb".
The first task is:
Problem 1) An (oversimplified) 1-D Model
For this introductory problem we are going to simulate a 1 dimensional detector (the more complex issues associated will real stars on 2D detectors will be covered tomorrow by Dora). We will generate stars as Gaussians $N(\mu, \sigma^2)$, with mean $\mu$ and variance $\sigma^2$.
As observed by LSST, all stars are point sources that reflect the point spread function (PSF), which is produced by a combination of the atmosphere, telescope, and detector. A standard measure of the PSF's width is the Full Width Half Maximum (FWHM).
There is also a smooth background of light from several sources that I previously mentioned (the atmosphere, the detector, etc). We will refer to this background simply as "The Sky".
Problem 1a
Write a function phi() to simulate a (noise-free) 1D Gaussian PSF. The function should take mu and fwhm as arguments, and evaluate the PSF along a user-supplied array x.
Hint - for a Gaussian $N(0, \sigma^2)$, the FWHM is $2\sqrt{2\ln(2)}\,\sigma \approx 2.3548\sigma$.
Can you write Python code for it?

def phi(x, mu, fwhm):
    """Evalute the 1d PSF N(mu, sigma^2) along x"""
    sigma = fwhm/2.3548
    flux = 1/np.sqrt(2*np.pi*sigma**2)*np.exp(-(x - mu)**2/(2*sigma**2))
    return flux


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessingSolutions.ipynb".
The first task is:
Problem 1b
Plot the noise-free PSF for a star with $\mu = 10$ and $\mathrm{FWHM} = 3$. What is the flux of this star?
Can you write Python code for it?

x = np.linspace(0,20,21)

plt.plot(x, phi(x, 10, 3))

print("The flux of the star is: {:.3f}".format(sum(phi(x, 10, 3))))


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessingSolutions.ipynb".
The first task is:
Problem 1c
Add Sky noise (a constant in this case) to your model. Define the sky as S, with total stellar flux F.
Plot the model for S = 100 and F = 500.
Can you write Python code for it?

S = 100 * np.ones_like(x)
F = 500

plt.plot(x, S + F*phi(x, 10, 3))


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessingSolutions.ipynb".
The first task is:
Problem 2b
Overplot the noisy signal, with the associated uncertainties, on top of the noise-free signal.
Can you write Python code for it?

plt.plot(x, no_noise)
plt.errorbar(x, noisy_flux, noisy_flux - no_noise, fmt = 'o')


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessingSolutions.ipynb".
The first task is:
Problem 3) Flux Measurement
We will now attempt to measure the flux from a simulated star.
Problem 3a
Write a function simulate() to simulate the noisy flux measurements of a star with centroid mu, FWHM fwhm, sky background S, and flux F.
Hint - it may be helpful to plot the output of your function.
Can you write Python code for it?

def simulate(x, mu, fwhm, S, F):
    source = F * phi(x, mu, fwhm)
    sky_plus_source = S * np.ones_like(x) + source
    noisy_flux = np.random.normal(sky_plus_source, np.sqrt(sky_plus_source))

    return noisy_flux


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessingSolutions.ipynb".
The first task is:
Problem 3c
Write a Monte Carlo simulator to estimate the mean and standard deviation of the flux from the simulated star.
Food for thought - what do you notice if you run your simulator many times?
Can you write Python code for it?

sim_fluxes = np.empty(1000)
for sim_num, dummy in enumerate(sim_fluxes):
    sim_star = simulate(x, 0, 5, 100, 1000)
    ap_flux = np.sum((sim_star - 100)[np.where(np.abs(x) <= 5)])

    sim_fluxes[sim_num] = ap_flux

print("The mean flux = {:.3f} with variance = {:.3f}".format(np.mean(sim_fluxes),
                                                             np.var(sim_fluxes, ddof=1)))


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessingSolutions.ipynb".
The first task is:
Problem 4b
Using the same parameters as problem 3, simulate a star and measure it's PSF flux.
Can you write Python code for it?

sim_star = simulate(x, 0, 5, 100, 1000)

psf_flux = np.sum((sim_star-100)*psf)

print("The PSF flux is {:.3f}".format(psf_flux))


I am working on the file "Sessions/Session05/Day1/ReIntroductionToImageProcessingSolutions.ipynb".
The first task is:
Problem 4c
As before, write a Monte Carlo simulator to estimate the PSF flux of the star. How do your results compare to above?
Can you write Python code for it?

sim_fluxes = np.empty(1000)
for sim_num, dummy in enumerate(sim_fluxes):
    sim_star = simulate(x, 0, 5, 100, 1000)
    psf_flux = np.sum((sim_star-100)*psf)

    sim_fluxes[sim_num] = psf_flux

print("The mean flux = {:.3f} with variance = {:.3f}".format(np.mean(sim_fluxes),
                                                             np.var(sim_fluxes, ddof=1)))


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Aha!  It's a Local Group galaxy, because otherwise you wouldn't be able to see individual stars like that. Lets see if we can identify it.  Lets see where the telescope was pointed:
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
CCDData has several other features like the ability to store (and propogate) uncertainties, or to mark certain pixels as defective, but we will not use that here, as we are focused on getting something out quickly. So lets just try visualizing it using our function from above:
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Conveniently, these apertures can even plot themselves:
Can you write Python code for it?

show_image(ccd_data_g_bkgsub, 12, 99.9, figsize=(6, 10))
apertures.plot(color='red')
plt.xlim(600, 800)
plt.ylim(2530, 2920)


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
D'Oh! That didn't work.  Why not?  The error message, while a bit cryptic, provides a critical clue: the overscan region found is one pixel shorter than the actual image.  This is because the "BIASSEC" keyword in this image doesn't follow quite the convention expected for the second (vertical) dimension. So we will just manually fill in a fits_section with the correct overscan region.
Can you write Python code for it?

im1.header['BIASSEC']

subed = ccdproc.subtract_overscan(im1, fits_section='[2049:2080,:]', overscan_axis=1)
show_image(subed)
subed.shape


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Exercise
Both bias images and overscan regions contain read noise (since they are read with the same electronics as everything else). See if you can determine the read noise (in counts) of this chip using the images we've examined thus far.  Compare to the LFC spec page and see if you get the same answer as they do (see the table in section 3 - this image is from CCD #0).
Combining and applying a "master bias"
The overscan is good at removing the bias along its scan direction, but will do nothing about pixel-specific bias variations or readout phenomena that are constant per-readout but spatially varying (like the "glow" on the left edge of the image that we saw above).  To correct that we will use a set of bias frames and subtract them from the images.  Before we can do this, however, we need to combine the biases together into a "master bias".  Otherwise we may simply be adding more noise than we remove, as a single bias exposure has the same noise as a single science exposure.
Look at the observing_log.csv file.  It shows you which files are biases.  The code below conveniently grabs them all into a single python list:
Can you write Python code for it?

biasfns = glob('python_imred_data/ccd.00?.0.fits.gz')
biasfns


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Getting the data
We need to first get the image data we are looking at for this notebook. This is actual data from an observing run by the author using the Palomar 200" Hale Telescope, using the Large Format Camera (LFC) instrument.
The cell below will do just this from the web (or you can try downloading from here: https://northwestern.box.com/s/4mact3c5xu9wcveofek55if8j1mxptqd), and un-tar it locally to create a directory with the image files. This is an ~200MB download, so might take a bit. If the wifi has gotten bad, try asking a neighbor or your instructor if they have it on a key drive or similar.
Can you write Python code for it?

!wget http://www.stsci.edu/\~etollerud/python_imred_data.tar
!tar xf python_imred_data.tar


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Hark!  The bias level is gone, as is the glowing strip on the left.  Two artifacts down.  It is now also quite evident that there is some kind of interesting astronomical object to the center-left...
Exercise
Opinions vary on the best way to combine biases. Try comparing the statistics of a bias made using the above procedure and one using a simple average (which you should be able to do with bias_combiner).  Which one is better?  Discuss with your neighbor when you might like one over the other.
Flats
The final image correction we will apply is a flat. Dividing by a flat removes pixel-by-pixel sensitivity variations, as well as some imperfections in instrument optics, dust on filters, etc. Different flats are taken with each filter, so lets look in the observing_log.csv to see which images are flats for the g-band filter:
Can you write Python code for it?

flat_g_fns = glob('python_imred_data/ccd.01[4-6].0.fits.gz')
flat_g_fns


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Hmm, not very useful, as it turns out.
In fact, astronomical data tend to have dynamic ranges that require a bit more care when visualizing.  To assist in this, the astropy.visualization packages has some helper utilities for re-scaling an image look more interpretable (to learn more about these see the astropy.visualization docs).
Can you write Python code for it?

from astropy import visualization as aviz

image = data_g[0].data
norm = aviz.ImageNormalize(image,
                           interval=aviz.PercentileInterval(95),
                           stretch=aviz.LogStretch())

fig, ax = plt.subplots(1,1, figsize=(6,10))
aim = ax.imshow(image, norm=norm, origin='lower')
plt.colorbar(aim)


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Inspecting this bias shows that it has far more counts than the science image, which is good because that means we have I higher S/N than the science.  It clearly shows the cross-shaped imprint in the middle that is also present in the science image, as well as vignetting near the bottom).
Now we combine the flats like we did with the biases to get a single flat to apply to science images.
Can you write Python code for it?

flat_g_combiner = ccdproc.Combiner(flats_g)
# feel free to choose a different combine algorithm if you developed a preference in the last exercise
combined_flat_g = flat_g_combiner.median_combine()

show_image(combined_flat_g, 90)


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
It appears as though there is a second overscan region along the columns.  We could conceivably choose to use that overscan region as well, but we should be suspicious of this given that it is not part of the BIASSEC mentioned in the header, and is included in DATASEC.  So it seems like it might be safer to just trim this region off and trust in the bias to remove the column-wise variations.
We know we need to apply this correction to every image, so to make the operations above repeatable easily, we move them into a single function to perform the overscan correction, which we will call later in the notebook:
Can you write Python code for it?

def overscan_correct(image):
    """
    Subtract the row-wise overscan and trim the non-data regions.

    Parameters
    ----------
    image : CCDData object
        The image to apply the corrections to

    Returns
    -------
    CCDData object
        the overscan-corrected image
    """
    subed = ccdproc.subtract_overscan(image, fits_section='[2049:2080,:]', overscan_axis=1)
    trimmed = ccdproc.trim_image(subed, fits_section='[1:2048,1:4096]')
    return trimmed


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Looks just the same, so looks like it accomplished the same aim but with less dependence on the FITS format.  So we will continue on with this data structure.
Overscan and Bias
Our first set of corrections center around removing the "bias" of the image - usually due to the voltage offset necessary to make a CCD work - along with associated pixel-by-pixel offsets due to the electronics.
Understanding Bias and Overscan areas
Examine the observing_log.csv file.  It shows that the first several images are called "bias".  That seems like a good place to start! Lets try examining one - both the image itself and its header.
Can you write Python code for it?

im1 = CCDData.read('python_imred_data/ccd.001.0.fits.gz', unit=u.count)
show_image(im1)
im1.header