I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
These plots look identical, but it is possible to change the style with seaborn.
seaborn has 5 style presets: darkgrid, whitegrid, dark, white, and ticks. You can change the preset using the following:
sns.set_style("whitegrid")

which will change the output for all subsequent plots. Note - if you want to change the style for only a single plot, that can be accomplished with the following:
with sns.axes_style("dark"):

with all ploting commands inside the with statement.
Problem 3a
Re-plot the sine curve using each seaborn preset to see which you like best - then adopt this for the remainder of the notebook.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
This plot is much more appealing (and informative) than the previous two. For the first time we can clearly see that the distribution is not actually centered on the origin. Now we will move back to the Iris data set.
Suppose we want to see univariate distributions in addition to the scatter plot? This is certainly possible with matplotlib and you can find examples on the web, however, with seaborn this is really easy.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
While the above plot provides a significant improvement over the scatter plot by providing a better sense of the density near the center of the distribution, the binedge effects are clearly present. An even better solution, like before, is a density estimate, which is easily built into seaborn via the kdeplot function.
Can you write Python code for it?

with sns.axes_style("darkgrid"):
    sns.kdeplot(xexample, yexample,shade=False)


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
which we will now change to colorblind, which is clearer to those that are colorblind.
Can you write Python code for it?

# set palette to colorblind
sns.set_palette("colorblind")

current_palette = sns.color_palette()
sns.palplot(current_palette)


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
Awesome!
You are now tooled up to solve any Bayesian problem that you might encounter.
Problem 4) Optimizing MH
While you are now, no doubt, wildly excited about what you have accomplished in $\sim{90}$ min time, there should be a few things that bother you. In particular, there were several "hard coded" options in the previous problem.
Weird Choice #1
Why 500 steps?
It's clear from 3i that ~10-20 steps is no where near enough, as the chain does not sample the maximum of the posterior. But is 500 steps too many? Or is it not enough?
(There are mathematical ways to judge this, which we will cover tomorrow)
Weird Choice #2
Why (0.2, 10) for the widths of the multivariate normal proposal function?
Previously we argued that it's possible to take steps that are too large, but is it also possible the steps are too small?
(Technically, as we will see tomorrow, the step size does not matter as all MCMC chains will eventually eplore the full posterior within an infinite number of steps, but who amoung us can wait infinity seconds to learn the slope of this line?)
Weird Choice #3
Why did we start the chain at (2.7,-70)?
(Again, technically this does not matter, but do you want to wait for an infinite number of steps before getting the results?)
Tommorrow we will formalize all of these choices, for now we will try to develop some intuition via some graphical examples.
Problem 4a
Write a function plot_post that recreates the plots from 3i and 3j side by side given inputs
theta_0, cov, nsteps, y, x, and y_unc.
Can you write Python code for it?

def plot_post(theta_0, cov, nsteps, y, x, y_unc):
    '''
    Plot posterior trace from MH MCMC

    Parameters
    ----------
    theta_0 : 1-D array_like of shape N
        starting position for the MCMC chain

    cov : 1-D or 2-D array_like, of shape (N,) or (N, N)
        Covariance matrix of the distribution. It must be symmetric
        and positive-semidefinite for proper sampling.

        1-D inputs for cov require the standard deviation along
        each axis of the N-dimensional Gaussian.

    nsteps : int
        Number of steps to take in the MCMC chain

    y : arr-like, shape (n_samples)
        Array of observational measurements

    x : arr-like, shape (n_samples)
        Array of positions where y is measured

    y_unc : arr-like, shape (n_samples)
        Array of uncertainties on y
    '''
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete

    return


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
Challenge 1c
Based on your solutions above, what do you estimate is the value of y at x = 50?
What is the 90% credible region for this prediction?
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
Challenge Problem) Inference & Forecasting
Challenge 1a
Based on your solution to the previous problem, what do you estimate is the value of $m$ and $b$?
What are the marginalized 90% credible regions for these paramters? How does this compare to the true answer?
Can you write Python code for it?

# complete

print('m = {0[0]:.3f} with a 90% credible region from {0[1]:.3f} to {0[2]:.3f}'.format( # complete
print('b = {0[0]:.3f} with a 90% credible region from {0[1]:.3f} to {0[2]:.3f}'.format( # complete


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
Problem 1b
Plot the observations and the corresponding uncertainties. Overplot a line showing the model from which the data were drawn.
Can you write Python code for it?

fig, ax = plt.subplots()

ax.errorbar( # complete
ax.plot([0,100], b_true + m_true*np.array([0,100]))
ax.set_xlabel('x')
ax.set_ylabel('y')
fig.tight_layout()


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
Problem 2) The Posterior
In order to approximate the posterior we need to be able to calculate it at any point within the multidimensional parameter space.
Assuming gaussian distributed scatter in the observations, we can describe the probability of any individual observation as a function of the model parameters:
$$p(y_i \mid m, b, x_i, \sigma_i) = \frac{1}{\sqrt{2\pi\sigma_{y_i}^2}}\exp{- \frac{(y_i - m\,x_i - b)^2}{2\sigma_{y_i}^2}},$$
where $y_i$ is the $i^\mathrm{th}$ observation, $x_i$ is the corresponding indpendent position at which $y_i$ is measured, $\sigma_{y_i}$ is the uncertainty on $y_i$ (constant in this case, but that doesn't have to be the case), and $m$ and $b$ are the slope and intercept for  the line.
Assuming the observations are independent, then the probability of all the observations is the product of the individual probabilities (note – this is the likelihood):
$$\mathcal{L} = \prod_i p(y_i \mid m, b, x_i, \sigma_i) = \prod_i \frac{1}{\sqrt{2\pi\sigma_{y_i}^2}}\exp{- \frac{(y_i - m\,x_i - b)^2}{2\sigma_{y_i}^2}}.$$
This product produces really small numbers (individual observations have typical probabilities  < 0.3), which can become numerically unstable, especially for a poor choice of model parameters.
A trick to alleviate this issue is to work with the $\log \mathcal{L}$, which using a logarithmic identity allows us to trade a product for a sum (computationally much easier!):
$$\log \mathcal{L} = \mathcal{K} - \sum_i \frac{(y_i - m\,x_i - b)^2}{2\sigma_{y_i}^2},$$
where $\mathcal{K}$ is a constant equal to $-\frac{n}{2}\log{2\pi} - \sum_i \log{\sigma_{y_i}}$.
Mathematically, maximizing the likelihood and the log likelihood are identical, and we will leverage this for the remainder of the problem.
Problem 2a
Write a function lnlikelihood to calculate the log likelihood given input parameters theta, y, x, and y_unc.
Hint 1 – the constant $\mathcal{K}$ is the same for any point in the parameter space. Thus it can be ignored as we only care about relative differences in the...
Can you write Python code for it?

def lnlikelihood(theta, y, x, y_unc):
    '''
    Calculate the log likelihood for a model assuming Gaussian uncertainties

    Parameters
    ----------
    theta : tuple
        Model parameters

    y : arr-like, shape (n_samples)
        Array of observational measurements

    x : arr-like, shape (n_samples)
        Array of positions where y is measured

    y_unc : arr-like, shape (n_samples)
        Array of uncertainties on y

    Returns
    -------
    lnl : float
        The log-likelihood of the observations given the model parameters
    '''
    model_preds = get_model_predictions( # complete

    lnl = # complete

    return lnl


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
Problem 2b
Confirm your function works by estimating the log likelihood for $m = 1.5$ and $b = 50$.
Is the log likelihood higher or lower if you use the true model parameters (see 1a)?
Does this make sense?
Can you write Python code for it?

print('ln L for m=1.5 and b=50 is: {:.4f}'.format(lnlikelihood( # complete
print('ln L for m=2.3 and b=15 is: {:.4f}'.format(lnlikelihood( # complete


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
Problem 3b
Calculate the Hastings ratio to move from $(2.1, 50)$ to $(2.2,30)$.
How often would the algorithm accept this step?
Can you write Python code for it?

print('The Hastings ratio is: {:.4f}'.format( # complete


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
Problem 3i
Plot the positions of the chain in the $(m, b)$ plane. Overplot the true answer as a five-pointed star.
Does your plot "make sense"?
Hint – connect the points with line segments so you can visualize its evolution.
Hint 2 – set alpha = 0.4 to get a sense of positions where multiple proposals were required for the chain to move.
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot( # complete
ax.plot(# complete,
        '*', ms=30,
        mfc='Crimson', mec='0.8', mew=2,
        alpha=0.7)
ax.set_xlabel('m', fontsize=14)
ax.set_ylabel('b', fontsize=14)
fig.tight_layout()


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
Problem 3j
Recreate the above plot, but this time color code the individual points by the value of the log posterior at each position.
Hint –– you will want to use pyplot.scatter
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot(pos[:,0], pos[:,1], 'o-', alpha=0.3)
ax.scatter( # complete
ax.plot(# complete,
        '*', ms=30,
        mfc='Crimson', mec='0.8', mew=2,
        alpha=0.7, zorder=20)
ax.set_xlabel('m', fontsize=14)
ax.set_ylabel('b', fontsize=14)
cbar = fig.colorbar(cax)
cbar.ax.set_ylabel(r'$\log \; \pi (\theta)$', fontsize=12)
fig.tight_layout()


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
Problem 4b
Plot the MH chain using theta_0 = (2.7, -70), cov = (0.25, 5), for 50 steps.
Do the same for a chain with 5000 steps.
What changes do you notice as you vary the number of steps?
Can you write Python code for it?

# 50 step plot
plot_post( # complete

# 5000 step plot
plot_post( # complete


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
Problem 4d
Create a summary plot using our "default" parameters of theta_0 = (2.7, -70), cov = (0.25, 5), and 500 steps.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
Without proof, I will state that we are looking for the following as we diagnose our MCMC:

dense sampling of the high probability portion of the posterior
draws from the posterior that are mostly consistent with the data
trace plots for m and b that are roughly i.i.d. (and do not show significant structure)
log posterior values that oscillate around the maximum with variations no larger than a few
a stable acceptance ratio of ~0.5 (note for simple problems 0.5 is good, for complex posteriors 0.25 is more likely)

Problem 4e
Vary the MH MCMC parameters to "better" sample the posterior according to the criteria given above.
Hint – this will likely take a lot of trial and error.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
write your answer here
As we will show below, the difference between an MCMC chain that takes forever and one that samples the posterior "quickly" is largely a function of the proposal distribution.
We want a symmetric proposal to ensure the transition ratio is equal to 1. The easiest way to accomplish this is with a normal distribution centered at the current position in the posterior, with a user-defined covariance matrix. (Today we will assume the covariance matrix is  diagonal).
Problem 3d
Write a function propose_jump that creates new proposal positions for the MCMC chain, based on its current position, theta, and a covariance matrix, cov, for the multivariate normal distribution centered on theta.
Hint 1 - allow cov to be either 1-D with standard deviations as the input, in which case the entries are converted to a diagonal matrix, or 2-D, where the full covariance matrix is specified.
Hint 2 - you may find numpy.random.multivariate_normal helpful.
Can you write Python code for it?

def propose_jump(theta, cov):
    '''
    Generate a proposed new position for MCMC chain

    Parameters
    ----------
    theta : 1-D array_like, of length N
        current position of the MCMC chain

    cov : 1-D or 2-D array_like, of shape (N,) or (N, N)
        Covariance matrix of the distribution. It must be symmetric
        and positive-semidefinite for proper sampling.

        1-D inputs for cov require the standard deviation along
        each axis of the N-dimensional Gaussian.

    Returns
    -------
    proposed_position : 1-D array_like, of length N
    '''
    if np.shape(theta) == np.shape(cov):
        cov = np.diag(np.array(cov)**2)

    proposed_position = # complete

    return proposed_position


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
write your answer here
In this case $H > 1$, and thus this proposed move  would be accepted every time.
Problem 3c
What about the reverse proposal? Calculate the Hastings ratio to move from (2.2, 30) to (2.1, 50).
On average, how many times would such a proposal need to be made to be accepted by the MH algortihm?
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
write your answer here
Problem 3) The Metropolis-Hastings Algorithm
The Metropolis-Hastings Algorithm (MH) is the most simple Markov Chain Monte Carlo (MCMC) procedure for estimating Bayesian posteriors.
We will cover the nitty gritty (i.e., mathematics) of how MCMC works tomorrow. For now, we will focus on Metropolis-Hastings as an algorithm, and assume it works.
The pseudo-code for the MH algorithm is:

pick some position $\theta_0$ in the parameter space and calculate the posterior $P(\theta_{0}\mid \mathbf{x})$

begin the chain

"propose" a move from the current position $\theta_{i}$ to a new position $\theta_{i+1}$
calculate the posterior at $\theta_{i+1}$, $P(\theta_{i+1}\mid \mathbf{x})$
draw a random number, $R \sim \mathcal{U}(0,1)$
if the ratio $P(\theta_{i+1}\mid \mathbf{x})/P(\theta_{i}\mid \mathbf{x})$ is $> R$, "accept" the proposed move and advance the chain to $\theta_{i+1}$
else "reject" the proposal and set $\theta_{i+1} = \theta_{i}$

repeat until chain is "finished"

A few subtleties worth noting: the use of the random number is essential for allowing the MH algorithm to "explore" the full posterior. It's actually possible to add links to the chain that are less probable than the current position. This is an important distinction relative to optimization routines that simply identify local maxima, but only progressing in the direction of increased probability.
Also "finished" isn't really defined - we will cover this tomorrow.
An extra subtle point: technically you need to calculate the full Hastings ratio, $H$, when evaluating whether or not to accept or reject a proposal:
$$H = \frac{\pi(\theta_{i+1})}{\pi(\theta_{i})} \frac{q(\theta_i \mid \theta_{i+1})}{q(\theta_{i+1} \mid \theta_{i})},$$
where $q(\theta_i \mid \theta_{i+1})/q(\theta_{i+1} \mid \theta_{i})$ is the ratio of transition probabilities from $\theta_{i+1}$ to $\theta_{i}$ and back again.
Fortunately, most proposals are symmetric, including what we will implement, meaning the ratio of transition probabilities is 1, and...
Can you write Python code for it?

def hastings_ratio(theta_1, theta_0, y, x, y_unc):
    '''
    Calculate the Hastings ratio

    Parameters
    ----------
    theta_1 : tuple
        proposed new posterior position

    theta_0 : tuple
        current posterior position

    y : arr-like, shape (n_samples)
        Array of observational measurements

    x : arr-like, shape (n_samples)
        Array of positions where y is measured

    y_unc : arr-like, shape (n_samples)
        Array of uncertainties on y

    Returns
    -------
    h_ratio : float
        The Hastings ratio
    '''
    lnpost1 = # complete
    lnpost0 = # complete

    h_ratio = # complete

    return h_ratio


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
write your answer here
Problem 3f
Repeat 3e but this time use a standard deviation of 0.1 for the slope and 10 for intercept.
Are the results substantially different?
Can you write Python code for it?

theta_0 = (2,45)
for _ in range(5):
    new_pos = propose_jump(theta_0, (0.1,10))
    H = hastings_ratio( # complete
    print('Jumping to {}, gives H = {:.4f}'.format(new_pos, H))


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
write your answer here
Problem 4b
Plot the MH chain using theta_0 = (2.7, -70), cov = (.025, .5), for 500 steps.
Do the same for cov = (2.5, 50).
What changes do you notice as you vary the size of the proposal distribution?
Can you write Python code for it?

# cov = (0.025, .5) plot
plot_post( # complete

# cov = (2.5, 50) plot
plot_post( # complete


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
write your answer here
Problem 4c
Plot the MH chain using theta_0 = (27, -100), cov = (.25, 5), for 500 steps.
Do the same for theta_0 = (.27, -7).
What changes do you notice as you vary the starting position of the chain?
Can you write Python code for it?

# theta_0 = (27, -100) plot
plot_post( # complete

# theta_0 = (.27, -7) plot
plot_post( # complete


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
write your answer here
The Hastings ratios are now much larger. It is clear that the width of the multivariate normal along each axis will affect the efficiency of the MH sampler.
Referring back to the pseudo code above, we now have all the ingredients necessary to put together a MH sampler. We just need to stitch everything together so that we actually form a chain.
Problem 3g
Write a function mh_mcmc that takes an initial position theta_0, covariance matrix for the multivariate normal proposal distribution cov, total number of steps nsteps, and observations y, x, and y_unc as inputs and returns the Metropolis-Hastings chain for the given inputs.
The output "chain" should include the position of the chain within the parameter space, the log of the posterior at that position, and the "acceptance ratio" of the chain until that point. The acceptance ratio is the fraction of previously proposed steps (including the current proposal) that have been accepted. We will come back to this later.
Can you write Python code for it?

def mh_mcmc(theta_0, cov, nsteps, y, x, y_unc):
    '''
    Metropolis-Hastings MCMC algorithm

    Parameters
    ----------
    theta_0 : 1-D array_like of shape N
        starting position for the MCMC chain

    cov : 1-D or 2-D array_like, of shape (N,) or (N, N)
        Covariance matrix of the distribution. It must be symmetric
        and positive-semidefinite for proper sampling.

        1-D inputs for cov require the standard deviation along
        each axis of the N-dimensional Gaussian.

    nsteps : int
        Number of steps to take in the MCMC chain

    y : arr-like, shape (n_samples)
        Array of observational measurements

    x : arr-like, shape (n_samples)
        Array of positions where y is measured

    y_unc : arr-like, shape (n_samples)
        Array of uncertainties on y

    Returns
    -------
    (positions, lnpost_at_pos, acceptance_ratio) : tuple
        positions : 2-D array_like of shape (nsteps+1, N)
            Position of the MCMC chain at every step

        lnpost_at_pos : 1-D array_like of shape nsteps+1
            log-posterior value at the position of the MCMC chain

        acceptance_ratio : 1-D array_like of shape nsteps+1
            acceptance ratio of all previous steps in the chain
    '''

    positions = np.zeros((nsteps+1, len(theta_0)))
    lnpost_at_pos = -np.inf*np.ones(nsteps+1)
    acceptance_ratio = np.zeros_like(lnpost_at_pos)
    accepted = 0

    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete

    return (positions, lnpost_at_pos, acceptance_ratio)


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
write your answer here
The posterior requires an estimate of not just the likelihood, but also the prior.
For this problem, how in the world do we choose the prior?
(In practice, this can be very complicated and we will discuss this in more detail on Wednesday and Friday)
When you have little to no knowledge of the actual model parameters, it is often "safe" to assume a wide and flat prior (sometimes referred to as an uniformative prior). A wide and flat prior essentially says that any value within the bounds of the prior is equally likely, while values outside the bounds are unacceptable.
While we know the true answer, let's pretend that we don't. From the data (technically, using the data to estimate the prior is cheating, as we'll discuss on Friday) we can see that the slope is definitely positive, and also definitely less than 300 (the largest y value in the data set). Thus, we will set a uniform prior on $m$ from 0 to 300:
$$P(m) \sim \mathcal{U}(0,300).$$
From similar arguments the intercept is likely somewhere between -100 and 100:
$$P(b) \sim \mathcal{U}(-100,100).$$
Problem 2c
Write a function lnprior to calculate the log of the prior from an input tuple of the model parameters theta.
Hint - while the $\log{0}$ is undefined, have the function return -np.inf for values outside the bounds of the prior.
Can you write Python code for it?

def lnprior(theta):
    '''
    Calculate the log of the prior for the model parameters

    Parameters
    ----------
    theta : tuple
        Model parameters

    Returns
    -------
    lnp : float
        The log-prior of the model parameters
    '''
    m, b = theta
    if 0 <= m <= 300 and -100 <= b <= 100:
        lnp = np.log(1/300) + np.log(1/200) # assumes P(m) and P(y) are completely independent
    else:
        lnp = # complete

    return lnp


I am working on the file "Sessions/Session10/Day1/MetropolisHastings.ipynb".
The first task is:
write your answer here
When evaluating MCMC chains, there are many different diagnostic plots that are helpful for that purpose.
Below is a helper function that overplots draws from the posterior on top of the observations, an estimate of the posterior, and the 1-d chains for $m$, $b$, $\log \pi$, and the acceptance rate.
Can you write Python code for it?

def plot_mh_summary(theta_0, cov, nsteps, y, x, y_unc):
    '''
    Plot the posterior, draws from the posterior, and 1-d chains

    Parameters
    ----------
    theta_0 : 1-D array_like of shape N
        starting position for the MCMC chain

    cov : 1-D or 2-D array_like, of shape (N,) or (N, N)
        Covariance matrix of the distribution. It must be symmetric
        and positive-semidefinite for proper sampling.

        1-D inputs for cov require the standard deviation along
        each axis of the N-dimensional Gaussian.

    nsteps : int
        Number of steps to take in the MCMC chain

    y : arr-like, shape (n_samples)
        Array of observational measurements

    x : arr-like, shape (n_samples)
        Array of positions where y is measured

    y_unc : arr-like, shape (n_samples)
        Array of uncertainties on y
    '''
    pos, lnpost, acc = mh_mcmc(theta_0, cov, nsteps, y_obs, x, y_unc)

    fig = plt.figure(figsize=(7.5,6))
    ax1 = plt.subplot2grid((4,5), (0, 0), colspan=2, rowspan=2)
    ax2 = plt.subplot2grid((4,5), (2, 0), colspan=2, rowspan=2)
    ax3 = plt.subplot2grid((4,5), (0, 2), colspan=3)
    ax4 = plt.subplot2grid((4,5), (1, 2), colspan=3, sharex=ax3)
    ax5 = plt.subplot2grid((4,5), (2, 2), colspan=3, sharex=ax3)
    ax6 = plt.subplot2grid((4,5), (3, 2), colspan=3, sharex=ax3)

    # posterior
    ax1.hexbin(pos[:,0], pos[:,1], gridsize=50, mincnt=1, bins='log')
    ax1.plot(2.3, 15, '*', ms=30,
        mfc='Crimson', mec='0.8', mew=2,
        alpha=0.7)
    ylims = ax1.get_ylim()
    xlims = ax1.get_xlim()
    ax1.plot([2.3, 2.3], ylims, 'Crimson', alpha=0.3)
    ax1.plot(xlims, [15, 15], 'Crimson', alpha=0.3)
    ax1.set_ylim(ylims)
    ax1.set_xlim(xlims)
    ax1.set_xlabel('m')
    ax1.set_ylabel('b')
    ax1.xaxis.set_ticks_position('top')
    ax1.xaxis.set_label_position('top')
    ax1.tick_params(top=True, bottom=False)

    # posterior draws
    ax2.errorbar(x, y_obs, y_unc, fmt='o')
#     ax2.plot([0,100],
#        ...


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
However, there should still be something that is nagging at you a little bit – how do you choose the value for $c$?
The truth is, there is no specific optimal choice of $c$, and thus, using the Huber loss function also requires an arbitrary decision.
There has to be a better way!
(Indeed, there is, as we will see throughout the week...)
Problem 5) Probability Theory
Suppose you have some event, which we will call $A$. We define the probability of event $A$ occurring as:
$$P(A).$$
Now suppose we want to know the probability that both event $A$ and event $B$ occur: $P(A \cap B)$. At first glance, it seems like this ought to be the product of the probability of $A$ and the probability of $B$:
$$P(A \cap B) = P(A)\,P(B).$$
However, this can only be true if $A$ and $B$ are independent.
To see why this is true, imagine a single coin. If event $A$ is a flip landing in heads, and event $B$ is a flip landing in tails, then $P(A)\,P(B) = 1/4$. However, the chances that a single coin flip results in both heads and tails is clearly 0.
The probability of $A$ and $B$ therefore requires a statement about conditional probability:
$$P(A \cap B) = P(A\mid{B})\,P(B),$$
which should be read as "the probability of $A$ and $B$ is equal to the probability of $A$ given $B$ multiplied by the probability of $B$."
The probability of $A$ and $B$ must be equal to the probability of $B$ and $A$, which leads to:
$$P(A\mid{B})\,P(B) = P(B\mid{A})\,P(A),$$
which we can rearrange as:
$$P(A\mid{B}) = \frac{P(B\mid{A})\,P(A)}{P(B)}.$$
This is Bayes' Law. It is 100% percent true.
At times you may encounter arguments or opinions that are "anti-Bayes." I want to emphasize that these sentiments have to do with the interpretation of Bayes' Law, and not Bayes' Law itself.
We will cover the interpretation of Bayes' Law in  much greater detail this afternoon...
Problem 6) Forecasting
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
If we apply the Huber loss to this data set, we find a fit that looks much better:
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
If you also suppose that $y$ varies linearly with $x$ (perhaps you have some theory that guides you towards this intuition), but that these data are produced via some noisy process, then you could write down the probability for any individual observation $y_i$ as:
$$P(y_i) = \frac{1}{\sqrt{2\pi\sigma_i^2}} \exp{-\frac{(y_i - m x_i - b)^2}{2\sigma_i^2}}$$
where $m$ is the slope of the line, $b$ is the intercept, and $\sigma_i$ is the unknown uncertainty on the $i^\mathrm{th}$ observation.
(note - I am being a little sloppy in my notation)
The probability of all of the observations is then the product of the probability for each observation:
$$P = \prod_i P(y_i)$$
and the model for the data now has 20 parameters ($m$, $b$, and 1 $\sigma_i$ for each of the 18 observations) and only 18 observations.
Surely this is hopeless; we cannot possibly constrain $m$ and $b$, right? ...
right?
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
In this case we see that increasing the order of the polynomial fit to the data lowers the "reduced" $\chi^2$, getting a value that is much much closer to 1.
However, a "chi-by-eye" examination of the two models presented here suggests that there is absolutely no way that the $14^\mathrm{th}$-order polynomial provides a better explanation of the relationship between $y$ and $x$. Certainly, none of us would use this model for forecasting.
This should make you very weary of any $\chi^2$-minimization techniques as a method to identify the "best-fit" model.
(You will nevertheless find this in all sorts of papers throughout the literature.)
(Full disclosure - I have been guilty of this in the past)
This raises an important and critical question: how does one decide which model is "best"?
(This is well beyond the scope of this lecture, but we will talk about this on Thursday and Friday)
Lie #3
You cannot fit a model with more free parameters than there are data points.
Suppose you have the following data:
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
Problem 4a
These data "seem like" they have been drawn from a linear relationship, and you may want to know the slope and intercept of that line, but there are some outliers.
What do you do?
Take a minute to discuss with your partner
Previously, we used least-squares fitting to determine the model parameters. Let's try that again, with one minor modification, we are going to assign a weight to each observation equal to the inverse of the variance for that observation.
(Note - np.polyfit() wants $1/\sigma_i$ as the weight, not $1/\sigma_i^2$)
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
Solution 2a
I intentionally mislead with the previous question.
The most important thing to do with any new data is understand where the data came from and what they represent. While the data are constants, they represent measurements of some kind. Thus, I would argue the most important thing to do with this data is understand where they came from (others may disagree).
In this case, we have "toy" data that were randomly generated for today's lecture. In that sense, there are no units or specific measurements that otherwise need to be understood.
Problem 2b
Make a scatter plot of the data to understand the relationship between $x$ and $y$.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
There is a very good chance, though, again, I am not specifically assuming anything, that for the previous plots that you plotted x along the abscissa and y along the ordinate.
[Honestly, there's no one to blame if this is the case, this has essentially been drilled into all of us from the moment we started making plots. In fact, in matplotlib we cannot change the abscissa label without adjusting the xlabel.]
This leads us to an important question, however. What if we flip y and x?
Does that change the results for the fit?
Problem 2e
Perform a linear least-squares fit to x vs. y (or if you already fit this, then reverse the axes). As above, plot the data and the best-fit model.
To test if the relation is the same between the two fits, compare the predicted y value for both models corresponding to x = 50.
Can you write Python code for it?

p_yx = np.polyfit(y, x, 1)
p_yx_eval = np.poly1d(p_yx)

fig = plt.figure(figsize=(6,5))
ax = plt.subplot2grid((3,1), (0, 0), rowspan=2)
ax_res = plt.subplot2grid((3,1), (2, 0), sharex=ax)

ax.plot(y, x, 'o')
ax.plot([-150,2000], p_yx_eval([-150,2000]))
ax.set_ylabel('x', fontsize=14)

ax_res.plot(y, x - p_yx_eval(y), 'o')
ax_res.axhline(color='C1')
ax_res.set_ylabel('residuals', fontsize=14)
ax_res.set_xlabel('y', fontsize=14)
plt.setp(ax.get_xticklabels(), visible=False)

fig.tight_layout()

print("For y vs. x, we find y = {:.4f}x + {:.4f}".format(p[0], p[1]))
print("\t for x=50 we would forecast y={:.2f}".format(p_eval(50)))
print("For x vs. y, we find x = {:.4f}y + {:.4f}".format(p_yx[0], p_yx[1]))
print("\t for x=50 we would forecast y={:.2f}".format((50 - p_yx[1])/p_yx[0]))


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
This "lie" can only be true if the data are singularly valued at every value of the dependent variable. Polynomials of the form $y = \sum_n a_n x^n$ cannot take on multiple values of $y$ at a single value for $x$.
Lie #3
The model that minimizes the $\chi^2$ is best.
(or a slightly more nuanced version of this statement – the model with $\chi^2$ per degree of freedom is closest to 1 is best)
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
This is somewhat amazing.
We have some data, we have no idea what the uncertainties are on the individual observations, and yet(!), we are able to recover the correct slope and intercept of the line.
(Note - we will discuss the particulars of the above model in greater detail later this week. I would rarely recommend the adopted model in practice!)
I hope at this stage I've convinced you that some common tropes regarding least-squares fitting do not always apply.
I also hope that I've raised some concerns on your end about the best methods for determining the slope and intercept of linearly related data.
Now on to another major problem for least-squares fitting...
Problem 4) Outliers
Suppose you have the following data set (and that you have reason to believe that the data are drawn from a linear relationship):
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
Understanding Data
(or, lies I've been told about statistics)
Version 0.1

By AA Miller (Northwestern/Adler Planetarium)
5 Sep 2019
According to the DSFP curriculum, this entire week is going to be devoted to statistics (an entire field unto itself).
In practice, however, statistics is too big a topic to fully tackle in a week.
A more apt description for this week's theme is models, and this is a good thing, because everything in astronomy is a model (more on this in just a bit).
Broadly speaking, we build models for two purposes:
forecasting
and
inference.
Forecasting is about predicting the outcome of some future, as of yet, unobserved event.
This can be extremely useful if you have measured the velocity dispersion of some galaxy and you would like to know the mass of the black hole at the center of that galaxy.
(aside – machine learning is essentially all about forecasting)
Inference is about understanding the fundamental relationship between different parameters. Inference can be used for forecasting (the otherway around isn't always true).
Inference is necessary for understanding physics. While the following statement can be incredibly useful, "based on 10 measurements of a force applied to this ball, I predict that a newly applied force, $F_\mathrm{new}$, will produce acceleration $F_\mathrm{new}/m_\mathrm{ball}$," it is far more useful to say $F = ma$ for any object with mass $m$.
(aside – machine learning is not useful for inference)
Before we start any discussion of models, we need to back up and zoom out a bit, because models require...
Problem 1) Data
At the core of everything we hope to accomplish with the DSFP stands a single common connection: data.
Using a vast array of telescopes across the planet and in outter space we are able to collect data that allow us to unravel the mysteries of the universe!
This naturally leads to the question:
Problem 1a
What is data?
Take a few min to discuss this with your partner
Solution 1a
While we just discussed several different ideas about the nature of...
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
We can immediately see that weighted least-squares fitting does not work well in this case.
We need to try another approach.
Can we just throw away the outliers?
The challenge then is definitively identifying what is and what is not an outlier. In this case it seems like there are 4 outliers, but what about the point below the line at $x \sim{23}$?
One can use an iterative procedure, often referred to as $k\sigma$ clipping to remove outliers. The psuedo code is:

select some $k$ for clipping
fit model to data
identify all observations that are $> k\sigma$ away from the best-fit model
remove the outliers
repeat the fit and remove process until no outliers remain

There are still two major problems: how to choose $k$?
(Is an observation that is $5.00001 \sigma$ off really all that worse than an observation that is $4.99999 \sigma$ off?)
The "outliers" are present in the initial fit, and as a result, they bias the model towards their location (for least-squares) and reduce the probability that they will be rejected as outliers.
Ultimately, the challenge with a least-squares approach and outliers is the use of a distance-squared "loss function." Skipping the derivation for now, the "best" model from least-squares comes from "minimizing the $\chi^2$" or "maximizing the likelihood" (these are the same thing):
$$\mathrm{best} = \min{\sum_i \frac{1}{2\sigma_i^2}(y - \mathrm{model})^2}$$
The term being minimized in this case is the "loss," and with a squared loss (as above) outliers play a far more significant role than observations that are close to the underlying relationship.
In practice, any loss function can be adopted. An alternative approach to "rejecting" outliers is to instead, demphasize them, which removes the decision about what to throw away and what to keep.
The Huber loss function is  a good way to accomplish this goal. Briefly, the Huber loss is quadratic, but it transitions to linear at some (typically large) critical value  $c$.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
We have now uncovered lie #1! The relationship between $x$ and $y$ is not the same as the relationship between $y$ and $x$ (as far as least-squares fitting is concerned).
Wait?! This is sorta like saying $F = ma$, but $a \neq F/m$.
How can this be?
There are a couple essential assumptions that go into standard least-squares fitting:

There is one dimension along which the data have negligible uncertainties
Along the other dimension all of the uncertainties can be described via Gaussians of known variance

These two conditions are rarely met for astronomical data. While condition 1 can be satisfied (e.g., time series data where there is essentially no uncertainty on the time of the observations), I contend that condition 2 is rarely, if ever, satisfied.
Problem 3) Lies, Damned Lies, and Statistics
While I may have stretched the definition of lie above, there are several misconceptions associated with model fitting (some of which are a direct consequence of using least-squares) that we should address.
Lie #2
For a 2-dimensional data set with one dependent variable (i.e. $y$ depends on $x$) and $n$ observations, a polynomial of the $n^\mathrm{th}$ degree can "perfectly" fit the data?
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day1/UnderstandingData.ipynb".
The first task is:
You now have some data $x$ and $y$.
Problem 2a
As a good data scientist, what is the first thing you should do with this data?
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day2/AdvancedSampling.ipynb".
The first task is:
After running an MCMC, it's always a good idea to run some convergence diagnostics.
With emcee, the recommended convergence diagnostic is the integrated autocorrelation time.
To get an estimate for the autocorrelation time for the chain for each parameter (Note: there isn't just one autocorrelation time per sampling, there's one for each scalar that you want to compute from the chain), we can use the get_autocorr_time method:
Can you write Python code for it?

tau = emcee_sampler.get_autocorr_time()
print("Integrated autocorrelation times:")
for i, k in enumerate(["bperp", "theta", "logs"]):
    print("{0}: {1}".format(k, tau[i]))


I am working on the file "Sessions/Session10/Day2/AdvancedSampling.ipynb".
The first task is:
An implementation in dynesty
Now let's update our model to use it with dynesty.
The only new function that we need to implement is a function called dynesty_prior_transform that takes samples on the unit cube and converts them to the parameter space that we're sampling.
The dynesty docs for this function are here.
This particular example is pretty simple, we need to convert 3 numbers in the range [0, 1] to three numbers in the ranges [-10, 10], [-pi/2, pi/2], and [-10, 10] respectively.
As above, implement your prior transform function below.
It should take a numpy array with three elements in the range [0, 1] and it should return a numpy array with entries (bperp, theta, logs).
Once your implementation is correct, the unit test should pass.
Can you write Python code for it?

def dynesty_prior_transform(u):
    #
    # YOUR CODE GOES HERE
    #

def dynesty_get_params(params):
    # This function is the same as the one above for emcee...
    return emcee_get_params(params)

def dynesty_loglike(params):
    # So is this one!
    return emcee_loglike(params)

# Now we run the unit tests to make sure that the functions are implemented correctly
from advanced_sampling_utils import test_dynesty_functions
test_dynesty_functions(dynesty_get_params, dynesty_prior_transform, dynesty_loglike, x, y)


I am working on the file "Sessions/Session10/Day2/AdvancedSampling.ipynb".
The first task is:
As usual, after running our sampler, we should check some convergence diagnostics.
With PyMC3, we'll use the pm.summary function:
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day2/AdvancedSampling.ipynb".
The first task is:
Challenge problem: uncertain x coordinates
A reasonable generalization of the above example is to include support for uncertainties on the $x$ coordinates.
There has been quite a bit written on this topic - including a blog post that I wrote - and there is cleverness that you can use to simplify the problem.
But in this problem, you should choose one of the three samplers above and generalize your implementation to include $N$ new parameters $\hat{x}_n$ (where $N$ is the number of datapoints) and $\hat{x}_n$ is the "true" value of $x_n$ that you're fitting.
You can use any of the three sampling libraries, but this is a case where PyMC3 really shines (so that's what I'd suggest trying!).
Here's a new simulated dataset where the $x$ coordinates are given by xobs with uncertainty xerr:
Can you write Python code for it?

np.random.seed(69302)
xerr = 0.5
xobs = x + xerr * np.random.randn(len(x))
plt.errorbar(xobs, y, xerr=xerr, fmt=".k")
plt.xlabel("x")
plt.ylabel("y");

#
# YOUR CODE GOES HERE
#


I am working on the file "Sessions/Session10/Day2/AdvancedSampling.ipynb".
The first task is:
If any errors appeared, let me know and we'll work it out.
If not, we're good to go!
Example 1: Fitting a line to data using emcee, dynesty, and PyMC3
In this example, we're going to fit a line to data, but we'll make it very slightly non-trivial by fitting for the amplitude of the noise as well as the slope and intercept of the line.
This problem has been written about all over the place, and you'll hear a few other discussions of it this week, but we'll start with a quick summary of the model so that we have consistent notation.
The model that we'll fit has three parameters: the slope $m$, the intercept $b$, and the logarithm of the size of errorbar $\log \sigma$.
Note: here and thoughout this tutorial, $\log$ means the natural logarithm, not $\log_{10}$.
For some setting of these parameters, let's simulate some fake data:
Can you write Python code for it?

np.random.seed(502489)

m_true = 3.1
b_true = -0.01
logs_true = np.log(3.5)
x = np.random.uniform(-5, 5, 25)
y = m_true * x + b_true + np.exp(logs_true)*np.random.randn(len(x))

plt.plot(x, y, ".k")
plt.xlabel("x")
plt.ylabel("y");


I am working on the file "Sessions/Session10/Day2/AdvancedSampling.ipynb".
The first task is:
In the following cell, we'll go through the steps to initialize and run our sampling.
This includes the following steps (and tuning parameter choices):

First, we need to initialize our sampler. For emcee, we need to select a point in parameter space for each "walker" in the ensemble. In this case, we'll use 100 walkers (I chose that number pretty arbitrarily), but you might find that you get better performance using a different choice. We'll initialize the walker coordinates by choosing a valid point in the parameter space and then add a small Gaussian random number to each walker coordinate so that they're not right on top of each other.

Next, we run our MCMC starting from the initial coordinates. For now, we'll just run 4000 steps, but you might want to use an automated convergence test when you use this in your research.
Can you write Python code for it?

np.random.seed(9281)
init = np.array([0.0, 0.0, 0.0])
init = init + 0.1 * np.random.randn(100, 3)

emcee_sampler = emcee.EnsembleSampler(len(init), 3, emcee_logprob)
emcee_sampler.run_mcmc(init, 4000, progress=True);


I am working on the file "Sessions/Session10/Day2/AdvancedSampling.ipynb".
The first task is:
Next, let's discard approximately 5 autocorrelation times worth of samples as "burn-in" (the phase where the sampler "forgets" about its initialization) and then estimate the number of effective samples.
Can you write Python code for it?

tau_max = int(tau.max())
emcee_samples = emcee_sampler.get_chain(discard=5*tau_max, flat=True)
emcee_samples = np.array(emcee_get_params(emcee_samples.T)).T

print("Effective number of samples for each parameter:")
print(len(emcee_samples) / emcee_sampler.get_autocorr_time(discard=5*tau_max))


I am working on the file "Sessions/Session10/Day2/AdvancedSampling.ipynb".
The first task is:
Once all your tests pass, let's run the nested sampling algorithm.
Unlike emcee, dynesty doesn't require an initialization step. Instead, it will start by sampling from the prior.
Also unlike emcee, dynesty uses a convergence diagnostic as a stopping criterion so you won't choose the number of model evaluations that the sampler will perform.
Instead, you can tune the convergence criteria following the documentation here.
After the sampler converges, we re-sample the results using the resample_equal function in order to get effectively independent samples from the posterior density.
Can you write Python code for it?

# Run the sampler
np.random.seed(3028)
dynesty_sampler = dynesty.NestedSampler(dynesty_loglike, dynesty_prior_transform, 3)
dynesty_sampler.run_nested()

# Resample the results to get samples from the posterior
dynesty_results = dynesty_sampler.results
dynesty_samples = dynesty.utils.resample_equal(
    dynesty_results.samples, np.exp(dynesty_results.logwt - dynesty_results.logz[-1])
)
dynesty_samples = np.array(dynesty_get_params(dynesty_samples.T)).T

print("Effective number of samples:")
print(np.any(np.abs(np.diff(np.sort(dynesty_samples, axis=0), axis=0)) > 1e-10, axis=1).sum())


I am working on the file "Sessions/Session10/Day2/AdvancedSampling.ipynb".
The first task is:
That's a lot of samples! (Probably more than we need...)
Then it can also be useful to plot the corner plot for the sampling to visualize the covariances between the parameters.
Since this fit was based on simulated data, we can also overplot the true values of the parameters; it looks like we recovered the truth!
Can you write Python code for it?

corner.corner(emcee_samples, labels=["m", "b", "logs"],
              truths=[m_true, b_true, logs_true]);


I am working on the file "Sessions/Session10/Day2/AdvancedSampling.ipynb".
The first task is:
The likelihood function in this case is:
$$
p({v_n}\,|\,{t_n},\,{A_j},\,{P_j},\,{\phi_j}) = \prod_{n=1}^N \mathrm{Normal}(v_n\,|\,\mu_n,\,\sigma_n)
$$
where
$$
\mu_n = \sum_{j=1}^J A_j\,\sin\left(\frac{2\,\pi\,t_n}{P_j} + \phi_j\right)
$$
and the parameters are the amplitudes $A_j$, periods $P_j$, and phases $\phi_j$ for each planet.
You should also select reasonable priors for each of these parameters (perhaps uniform or uniform in the log).
Implementation
Then in the following cells, implement this fit using at least one of the above samplers (or another if you have a favorite).
If you have time, it is worth implementing the fit using dynesty and one of the other two samplers to compare your results.
In all cases, some care should be taken to implement periodic boundary conditions for $\phi$. One option is to reparameterize using $u_j = \sqrt{A_j}\,\sin\phi_j$ and $w_j = \sqrt{A_j}\,\cos\phi_j$ instead of $A_j$ and $\phi_j$ directly. (Can you convince yourself that this will have the right prior on $A_j$ and $\phi_j$?) Another option is to parameterize in terms of $A_j$, $u_j \sim \mathrm{Normal}(0,\,1)$, and $w_j \sim \mathrm{Normal}(0,\,1)$ and then define $\phi_j = \arctan(u_j / w_j)$. This is the parameterization that I would use with PyMC3 and it is implemented as the Angle distribution in the advanced_sampling_utils module. If you're using dynesty, you can use the periodic parameter when initializing the NestedSampler.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Another way to see if we're on the right track is to plot the data phased to the orbital period that we found. Do that and optionally overplot the phased model as well.
Can you write Python code for it?

period, amplitude, t0, rv0 = res['x']

fig, ax = plt.subplots()
phased_t = (data['t'] - t0) % period
# complete


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Challenge Problem 1h
Try some different values of cov[0] (the step size for the orbital period). Make a plot of the acceptance fraction as a function of step size. Does this make sense?
Challenge Problem 1i
For different values of cov[0], plot the correlation length. Does this make sense?
Problem 2: Fitting a Keplerian to Data
In the previous example, the orbit we were fitting had negligible eccentricity, so we were able to fit it with a sinusoid. In this example, we'll look at the high-eccentricity planet HD 80606b and fit a full Keplerian model to its RV data. This requires introducing some new free parameters to the model, which as we will see are not always straightforward to sample!
Can you write Python code for it?

datafile = 'https://exoplanetarchive.ipac.caltech.edu/data/ExoData/0045/0045982/data/UID_0045982_RVC_006.tbl'
data = pd.read_fwf(datafile, header=0, names=['t', 'rv', 'rv_err'], skiprows=21)
data['t'] -= data['t'][0]


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Do a pairs plot for the first two parameters. Does the behavior of this chain seem efficient?
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot( # complete
ax.plot(theta_0[0], theta_0[1], '*', ms=30,
        mfc='Crimson', mec='0.8', mew=2,
        alpha=0.7)
ax.set_xlabel('Period', fontsize=14)
ax.set_ylabel(r'K (m s$^{-1}$)', fontsize=14)
fig.tight_layout()


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
From these results, what can we say about the true value of K? What is the probability that K > 84 m/s? 85 m/s? 90 m/s? Are these numbers a reliable estimator of the true probability, in your opinion?
Can you write Python code for it?

print('The probability that K > 84 m/s is: {0:.2f}'.format( # complete
print('The probability that K > 85 m/s is: {0:.2f}'.format( # complete
print('The probability that K > 90 m/s is: {0:.2f}'.format( # complete


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Multi-Dimensional Integration with MCMC

By Megan Bedell (Flatiron Institute)
10 September 2019
Problem 1: Fitting a Sinusoid to Data
In this example, we will download a time series of radial velocities for the star HD209458. This star hosts a Hot Jupiter exoplanet. In fact, this planet was the first to be seen in transit and was discovered 20 years ago yesterday!
Because the eccentricity is low for this planet, we can fit its orbit in the radial velocities with a relatively simple model: a sinusoid.
Below is a snippet of code that will download the time-series data from NASA Exoplanet Archive:
Can you write Python code for it?

datafile = 'https://exoplanetarchive.ipac.caltech.edu/data/ExoData/0108/0108859/data/UID_0108859_RVC_001.tbl'
data = pd.read_fwf(datafile, header=0, names=['t', 'rv', 'rv_err'], skiprows=22)
data['t'] -= data['t'][0]


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Now run the MCMC for 5000 steps. I'll give you (the diagonal of a) covariance matrix for the multi-dimensional Gaussian proposal function to start with. As you saw yesterday afternoon, this cov parameter sets the step sizes that the M-H algorithm will use when it proposes new values.
Can you write Python code for it?

cov = [0.01, 1, 0.05, 0.01]
pos, lnpost, acc = mh_mcmc( # complete


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Plot the data points and your best-fit model. Does the fit look reasonable? (You may need to zoom into a small time range to tell.)
Can you write Python code for it?

# complete
plt.xlabel('Time (days)')
plt.ylabel(r'RV (m s$^{-1}$)');


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Problem 1c
Run the MCMC.
Let's start with initialization values.
To save some time, I will assert that if we made a Lomb-Scargle periodogram of the RVs, there would be a peak near period = 3.53 days, so start with that guess and let's figure out what the best values might be for the other parameters.
(If you finish early and are up for a bonus problem, you can double-check my assertion using astropy timeseries!)
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Problem 1d
There were a couple of issues with the previous MCMC run. Let's start with this one: we started the chains running at a place that was not very close to the best-fit solution.
Find a better set of initialization values by optimizing before we run the MCMC.
We'll use scipy.optimize.minimize to get best-fit parameters. Remember that the lnlikelihood function needs to be maximized not minimized, so we'll need a new function that works the same way, but negative.
Can you write Python code for it?

def nll(*par):
  #complete

res = minimize(nll, theta_0,
               args=(data['rv'], data['t'], data['rv_err']),
               method='Powell')
print('Optimizer finished with message "{0}" and \n\
      best-fit parameters {1}'.format(res['message'], res['x']))


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Problem 1e
Now let's tackle another issues: chain efficiency. Calculate the auto-correlation length of your chain.
First, let's just plot the sequence of orbital period values in the chain in a trace plot. From eyeballing this sequence, about how many steps do you think are needed to reach a sample that is independent from the previous one(s)?
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Problem 1f
Change the step size of the MCMC. What does this do to the auto-correlation length? Does this seem better or worse, and why?
Can you write Python code for it?

cov = [0.0001, 0.1, 0.01, 0.1]
pos, lnpost, acc = mh_mcmc( # complete

plt.plot( # complete

acf = # complete

act = # complete


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Problem 1g
Using the step sizes and starting conditions that you deem best, run your MCMC for at least 500x the auto-correlation length to get a large number of independent samples. Plot the posterior distribution of radial velocity semi-amplitude K. This parameter is arguably the most important output of an RV fit, because it is a measurement of the mass of the planet.
Can you write Python code for it?

pos, lnpost, acc = mh_mcmc(# complete

plt.hist( # complete
plt.xlabel(r'K (m s$^{-1}$)');


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Problem 2a
Again, let's start by plotting the data. Make plots of the time series and the time series phased to a period of 111.4 days.
Can you write Python code for it?

fig, ax = plt.subplots()
ax.errorbar( # complete
ax.set_xlabel('Time (days)')
ax.set_ylabel(r'RV (m s$^{-1}$)');

# phased plot goes here


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Problem 2d
Run the MCMC for 1000 steps and plot a trace of the eccentricity parameter. How efficiently is it running?
Optional challenge: if you wrote a Gibbs sampler yesterday, use that instead of Metropolis-Hastings here!
Can you write Python code for it?

cov = [0.1, 100, 0.01, 0.1, 0.1, 100]

pos, lnpost, acc = mh_mcmc( # complete

#complete
plt.ylabel('Eccentricity')
plt.xlabel('Step');


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Problem 2e
Make a corner plot of the results. Which parameters seem most correlated? Which are most and least well-constrained by the data?
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
The following functions can be reused as-is from the previous day's Metropolis-Hastings exercise, so just copy-and-paste or import them:
lnlikelihood, lnposterior, hastings_ratio, propose_jump, mh_mcmc
Can you write Python code for it?

def lnlikelihood(theta, y, x, y_unc):
    model_preds = get_model_predictions(theta, x)

    lnl = -np.sum((y-model_preds)**2/(2*y_unc**2))

    return lnl

def lnposterior(theta, y, x, y_unc):
    lnp = lnprior(theta)
    if not np.isfinite(lnp):
        return -np.inf
    lnl = lnlikelihood(theta, y, x, y_unc)
    lnpost = lnl + lnp

    return lnpost

def hastings_ratio(theta_1, theta_0, y, x, y_unc):
    lnpost1 = lnposterior(theta_1, y, x, y_unc)
    lnpost0 = lnposterior(theta_0, y, x, y_unc)

    h_ratio = np.exp(lnpost1 - lnpost0)

    return h_ratio

def propose_jump(theta, cov):
    if np.shape(theta) == np.shape(cov):
        cov = np.diag(np.array(cov)**2)

    proposed_position = np.random.multivariate_normal(theta, cov)

    return proposed_position

def mh_mcmc(theta_0, cov, nsteps, y, x, y_unc):

    positions = np.zeros((nsteps+1, len(theta_0)))
    lnpost_at_pos = -np.inf*np.ones(nsteps+1)
    acceptance_ratio = np.zeros_like(lnpost_at_pos)
    accepted = 0

    positions[0] = theta_0
    lnpost_at_pos[0] = lnposterior(theta_0, y, x, y_unc)

    for step_num in np.arange(1, nsteps+1):
        proposal = propose_jump(positions[step_num-1], cov)
        H = hastings_ratio(proposal, positions[step_num-1], y, x, y_unc)
        R = np.random.uniform()

        if H > R:
            accepted += 1
            positions[step_num] = proposal
            lnpost_at_pos[step_num] = lnposterior(proposal, y, x, y_unc)
            acceptance_ratio[step_num] = float(accepted)/step_num
        else:
            positions[step_num] = positions[step_num-1]
            lnpost_at_pos[step_num] = lnpost_at_pos[step_num-1]
            acceptance_ratio[step_num] = float(accepted)/step_num

    return (positions, lnpost_at_pos, acceptance_ratio)


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMC.ipynb".
The first task is:
Write a lnprior function with flat priors on all parameters - again, this will be similar to yesterday's function, but with different values.
Hint: some of the bounds on these parameters will be physically motivated (i.e. orbital period cannot be negative). For others, you'll need to guess something reasonable but generous - i.e., a Hot Jupiter planet probably does not have an orbital period above a year or so.
Can you write Python code for it?

def lnprior(theta):
    period, amplitude, t0, rv0 = theta
    if 0 < period <= 1e4 and # complete
        lnp = np.log(1e-4) + # complete
    else:
        return -np.inf
    return lnp


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
From these results, what can we say about the true value of K? What is the probability that K > 84 m/s? 85 m/s? 90 m/s? Are these numbers a reliable estimator of the true probability, in your opinion?
Can you write Python code for it?

N_tot = len(pos[:,1])

print('The probability that K > 84 m/s is: {0:.2f}'.format(np.sum(pos[:,1] > 84.)/N_tot))
print('The probability that K > 85 m/s is: {0:.2f}'.format(np.sum(pos[:,1] > 85.)/N_tot))
print('The probability that K > 90 m/s is: {0:.2f}'.format(np.sum(pos[:,1] > 90.)/N_tot))


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
Multi-Dimensional Integration with MCMC

By Megan Bedell (Flatiron Institute)
10 September 2019
Problem 1: Fitting a Sinusoid to Data
In this example, we will download a time series of radial velocities for the star HD209458. This star hosts a Hot Jupiter exoplanet. In fact, this planet was the first to be seen in transit and was discovered 20 years ago yesterday!
Because the eccentricity is low for this planet, we can fit its orbit in the radial velocities with a relatively simple model: a sinusoid.
Below is a snippet of code that will download the time-series data from NASA Exoplanet Archive:
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
Note: we have not actually sampled parameter space around K > 90 m/s, so take this estimate with a grain of salt -- we can certainly conclude that the probability of K > 90 is low, but we'd need to actually calculate posterior values around K = 90 before we'd have a reliable estimate of the PDF there.
Challenge Problem 1h
Try some different values of cov[0] (the step size for the orbital period). Make a plot of the acceptance fraction as a function of step size. Does this make sense?
Challenge Problem 1i
For different values of cov[0], plot the correlation length. Does this make sense?
Problem 2: Fitting a Keplerian to Data
In the previous example, the orbit we were fitting had negligible eccentricity, so we were able to fit it with a sinusoid. In this example, we'll look at the high-eccentricity planet HD 80606b and fit a full Keplerian model to its RV data. This requires introducing some new free parameters to the model, which as we will see are not always straightforward to sample!
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
Now run the MCMC for 5000 steps. I'll give you (the diagonal of a) covariance matrix to start with. As you saw yesterday afternoon, this cov parameter sets the step sizes that the M-H algorithm will use when it proposes new values.
Can you write Python code for it?

cov = [0.01, 1, 0.05, 0.01]
pos, lnpost, acc = mh_mcmc(theta_0, cov, 5000,
                           data['rv'], data['t'], data['rv_err'])


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
Problem 1a
Plot the data. Let's take a look at what we're working with!
Can you write Python code for it?

fig, ax = plt.subplots()
ax.errorbar(data['t'], data['rv'], data['rv_err'], fmt='o', ms=4)
ax.set_xlabel('Time (days)')
ax.set_ylabel(r'RV (m s$^{-1}$)');


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
Problem 2a
Again, let's start by plotting the data. Make plots of the time series and the time series phased to a period of 111.4 days.
Can you write Python code for it?

fig, ax = plt.subplots()
ax.errorbar(data['t'], data['rv'], data['rv_err'], fmt='o', ms=4)
ax.set_xlabel('Time (days)')
ax.set_ylabel(r'RV (m s$^{-1}$)');

plt.errorbar(data['t'] % 111.4, data['rv'], data['rv_err'], fmt='o', ms=4)
plt.xlabel('Phase (days)')
plt.ylabel(r'RV (m s$^{-1}$)');


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
Problem 2c
Play around with the starting parameters until you're convinced that you have a reasonable fit.
Can you write Python code for it?

theta_0 = [111.4, 480, 0.95, 2.0, 89, -200] # P, K, ecc, omega, tp, rv0

plt.errorbar(data['t'], data['rv'], data['rv_err'],
             fmt='o', ms=4)
xs = np.linspace(900, 1050, 1000)
plt.plot(xs, get_model_predictions(theta_0, xs), c='DarkOrange')
plt.xlim([900,1050])
plt.xlabel('Time (days)')
plt.ylabel(r'RV (m s$^{-1}$)');


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
Problem 2d
Run the MCMC for 1000 steps and plot a trace of the eccentricity parameter. How efficiently is it running?
Optional challenge: if you wrote a Gibbs sampler yesterday, use that instead of Metropolis-Hastings here!
Can you write Python code for it?

cov = [0.1, 100, 0.01, 0.1, 0.1, 100]

pos, lnpost, acc = mh_mcmc(theta_0, cov, 10000,
                           data['rv'], data['t'], data['rv_err'])

plt.plot(pos[:,2])
plt.ylabel('Eccentricity')
plt.xlabel('Step');


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
Problem 2e
Make a corner plot of the results. Which parameters seem most correlated? Which are most and least well-constrained by the data?
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
The chain is now staying relatively stationary, which is good! However, it's still spending a long time at each point.
Problem 1e
Now let's tackle another issues: chain efficiency. Calculate the auto-correlation length of your chain.
First, let's just plot the sequence of orbital period values in the chain in a trace plot. From eyeballing this sequence, about how many steps do you think are needed to reach a sample that is independent from the previous one(s)?
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
This is worrying - it means we have achieved very few actual independent draws from the posterior in our chain.
Problem 1f
Change the step size of the MCMC. What does this do to the auto-correlation length? Does this seem better or worse, and why?
Can you write Python code for it?

cov = [0.0001, 0.1, 0.01, 0.1]
pos, lnpost, acc = mh_mcmc(theta_bestfit, cov, 5000,
                           data['rv'], data['t'], data['rv_err'])

plt.plot(pos[:,0]);

acf = emcee.autocorr.function_1d(pos[:,0])
plt.plot(acf)
plt.xlabel('Lag')
plt.ylabel('Normalized ACF');

act = emcee.autocorr.integrated_time(pos[:,0])
print('The integrated autocorrelation time is estimated as: {0}'.format(act))


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
This planet's orbit should look pretty different from a sine wave!
Problem 2b
Remake the get_model_predictions and lnprior functions to fit a Keplerian.
Since this is a bit in the weeds of astronomy for the purposes of this workshop, I've gone ahead and written a solver for Kepler's equation and a get_model_predictions function that will deliver RVs for you. Read over the docstring and use the information given there to write a lnprior function for theta.
Can you write Python code for it?

def calc_ea(ma, ecc):
    # Kepler solver - calculates eccentric anomaly
    tolerance = 1e-3
    ea = np.copy(ma)
    while True:
        diff = ea - ecc * np.sin(ea) - ma
        ea -= diff / (1. - ecc * np.cos(ea))
        if abs(diff).all() <= tolerance:
            break
    return ea

def get_model_predictions(theta, t):
    '''
    Calculate Keplerian orbital RVs

    Input
    -----
    theta : list
      A list of values for the following parameters:
       Orbital period,
       RV semi-amplitude,
       eccentricity (between 0-1),
       omega (argument of periastron; an angle in radians
              denoting the orbital phase where the planet
              passes closest to the host star)
       Tp (time of periastron; reference timestamp for the above)
       RV0 (constant RV offset)

    t : list or array
      Timestamps at which to calculate the RV

    Returns
    -------
    rvs : list or array
      Predicted RVs at the input times.
    '''
    P, K, ecc, omega, tp, rv0 = theta

    ma = 2. * np.pi / P * (t - tp)  # mean anomaly
    ea = calc_ea(ma, ecc)  # eccentric anomaly

    f = 2.0 * np.arctan2(np.sqrt(1+ecc)*np.sin(ea/2.0),
                         np.sqrt(1-ecc)*np.cos(ea/2.0)) # true anomaly
    rvs = - K * (np.cos(omega + f) + ecc*np.cos(omega))
    return rvs + rv0

def lnprior(theta):
    period, amplitude, ecc, omega, tp, rv0 = theta
    if 0 < period <= 1e5 and 0 <= amplitude <= 1e4: # physical priors
        lnp = np.log(1e-5) + np.log(1e-4)
    else:
        return -np.inf
    if np.abs(tp) <= 1e4 and np.abs(rv0) <= 1e4: # generous flat priors
        lnp += 2 * np.log(1/2e4)
    else:
        return -np.inf
    if 0 <= ecc < 1 and 0 < omega < 2*np.pi: # more physical priors
        lnp += np.log(1) + np.log(1/(2*np.pi))
    else:
        return -np.inf
    return lnp


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
Write a lnprior function with flat priors on all parameters - again, this will be similar to yesterday's function, but with different values.
Hint: some of the bounds on these parameters will be physically motivated (i.e. orbital period cannot be negative). For others, you'll need to guess something reasonable but generous - i.e., a Hot Jupiter planet probably does not have an orbital period above a year or so.
Can you write Python code for it?

def lnprior(theta):
    period, amplitude, t0, rv0 = theta
    if 0 < period <= 1e4 and 0 <= amplitude <= 1e3: # physical priors
        lnp = np.log(1e-4) + np.log(1e-3)
    else:
        return -np.inf
    if np.abs(t0) <= 1e3 and np.abs(rv0) <= 1e3: # generous flat priors
        lnp += 2 * np.log(1/2e3)
    else:
        return -np.inf
    return lnp


I am working on the file "Sessions/Session10/Day2/MultiDimensionalMCMCSolutions.ipynb".
The first task is:
Writing an autocorrelation function for this purpose actually gets a bit tricky, so we'll use the built-in functionality of emcee.
For the documentation on these functions, check the emcee user guide.
For a more in-depth look at how this is calculated and why it's tricky, check out this tutorial.
Can you write Python code for it?

acf = emcee.autocorr.function_1d(pos[:,0])
plt.plot(acf)
plt.xlabel('Lag')
plt.ylabel('Normalized ACF');

act = emcee.autocorr.integrated_time(pos[:,0], quiet=True)
print('The integrated autocorrelation time is estimated as: {0}'.format(act))


I am working on the file "Sessions/Session10/Day4/missing_information/howto.ipynb".
The first task is:
Go ahead and try to run it. You'll get an error traceback, the end of which points out that you've neglected to provide a solution to the posed problem. This is our preferred method of providing incomplete code, since an alternative like
python
x = # set x equal to something
will throw a different and less informative error if you accidentally run the cell before completing it.
The try-except wrapper is there so that we, the developers, can easily verify that the entire notebook runs if provided with a correct solution. There is no need for you to write solutions for each cell in separate files, and doing so will just make this notebook harder for you to use later. Instead, we suggest removing the try-except construction entirely, so your completed notebook cell would look like
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day4/missing_information/missingdata_tutorial.ipynb".
The first task is:
1. Defining a model
Before worrying about missing data, let's define a model that we might want to fit to these data. We're interested in whether the probability of having zero O-ring incidents (or non-zero incidents, conversely) is a function of temperature. One possible parametrization that allows this is the logistic function, which squeezes the real line onto the range (0,1).
For reasons that may be clear later, I suggest defining the model in terms of the probability of success (zero incidents)
$P_\mathrm{success}(T|T_0,T_s) = \frac{1}{1 + e^{-(T-T_0)/T_s}}$,
with parameters $T_0$ and $T_s$ respectively determining the center and width of the logistic function.

Ed. note: This turns out to be a spectacularly stupid choice of model, for reasons that hopefully will be clear once you visualize the fits below. If you'd like to propose something different, feel free; otherwise hold your nose and carry on.

1a. Implement this function and have a look
Can you write Python code for it?

try:
    exec(open('solutions/psuccess.py').read())
except IOError:
    REMOVE_THIS_LINE()
    def P_success(T, T0, Ts):
        REPLACE_WITH_YOUR_SOLUTION()


I am working on the file "Sessions/Session10/Day4/missing_information/missingdata_tutorial.ipynb".
The first task is:
1b. PGM and priors
Given the definition of the data and model above, draw the PGM for this problem, and write down an expression for the likelihood.
Choose priors for any parameters that need them. (As always, these should not be chosen based on looking at the data.) Complete the function below implementing the (log-)priors you've chosen.
Can you write Python code for it?

try:
    exec(open('solutions/priors.py').read())
except IOError:
    REMOVE_THIS_LINE()
    def ln_prior(T0, Ts):
        REPLACE_WITH_YOUR_SOLUTION()


I am working on the file "Sessions/Session10/Day4/missing_information/missingdata_tutorial.ipynb".
The first task is:
2a. Censored model definition
Work out how to adjust your PGM and expression for the likelihood to reflect our ignorance of the temperatures of successful launches.
Implement the (log)-likelihood for the censored model. Uncomment the cells below to see some hints if you want.
Can you write Python code for it?

#print(open('hints/hint1.txt').read())

#print(open('hints/hint2.txt').read())

#print(open('hints/hint3.txt').read())

try:
    exec(open('solutions/likelihood.py').read())
except IOError:
    REMOVE_THIS_LINE()
    def ln_like(REPLACE_WITH_YOUR_ARGUMENT_LIST_OF_CHOICE):
        REPLACE_WITH_YOUR_SOLUTION()


I am working on the file "Sessions/Session10/Day4/missing_information/missingdata_tutorial.ipynb".
The first task is:
For this notebook, we will simplify the data for each launch from integer (how many incidents of O-ring damage) to boolean (was there any damage, or not).
Can you write Python code for it?

failure_temps = oring_temps[np.where(oring_incidents > 0)[0]]
Nfailure = len(failure_temps)
success_temps = oring_temps[np.where(oring_incidents == 0)[0]]
Nsuccess = len(success_temps)
print('temperatures corresponding to failures:', failure_temps)
print('temperatures corresponding to successes:', success_temps)


I am working on the file "Sessions/Session10/Day4/missing_information/missingdata_tutorial.ipynb".
The first task is:
Make some plots summarizing the 2D and 1D posteriors of the parameters. The 2D plot shows approximate 68.3% and 95.4% credible regions based on the central limit theorem (i.e. assuming they are defined by threshold values of $-2\Delta\log p$). This should be fine for qualitatively evaluating the constraints, but feel free to do a better job if you want.
Can you write Python code for it?

# 2D parameter constraints
plt.contour(post_T0, post_Ts, post_lnp, -0.5*np.array([6.18, 2.3]), colors='blue', linestyles='solid');
plt.xlabel(r'$T_0$');
plt.ylabel(r'$T_s$');
plt.title(r'$p(T_0,T_s|\mathrm{data})$');

# 1D posterior
plt.plot(post_T0, np.sum(post_p, axis=0));
plt.xlabel(r'$T_0$');
plt.ylabel(r'$p(T_0|\mathrm{data})$');

# 1D posterior
plt.plot(post_Ts, np.sum(post_p, axis=1));
plt.xlabel(r'$T_s$');
plt.ylabel(r'$p(T_s|\mathrm{data})$');


I am working on the file "Sessions/Session10/Day4/missing_information/missingdata_tutorial.ipynb".
The first task is:
Now we go through the same evaluations of the posterior as above, and compare the two sets results.
Can you write Python code for it?

# Again, just fill in the call to your version of ln_like_complete
try:
    exec(open('solutions/getpostcomplete.py').read())
except IOError:
    post_lnp_complete = np.array( [ [ln_prior(T0,Ts)+REPLACE_WITH_YOUR_SOLUTION() for T0 in post_T0] for Ts in post_Ts] )

post_lnp_complete -= np.max(post_lnp_complete)
post_p_complete = np.exp(post_lnp_complete)

post_fail_prob_complete = [ posterior_prediction_Pfailure(T, post_p_complete) for T in post_fail_temp ]

# 2D parameter constraints
plt.contour(post_T0, post_Ts, post_lnp, -0.5*np.array([6.18, 2.3]), colors='blue', linestyles='solid');
plt.contour(post_T0, post_Ts, post_lnp_complete, -0.5*np.array([6.18, 2.3]), colors='red', linestyles='dashed');
plt.xlabel(r'$T_0$');
plt.ylabel(r'$T_s$');
plt.title(r'$p(T_0,T_s|\mathrm{data})$');

# 1D posterior
plt.plot(post_T0, np.sum(post_p, axis=0), 'b-');
plt.plot(post_T0, np.sum(post_p_complete, axis=0), 'r--');
plt.xlabel(r'$T_0$');
plt.ylabel(r'$p(T_0|\mathrm{data})$');

# 1D posterior
plt.plot(post_Ts, np.sum(post_p, axis=1), 'b-');
plt.plot(post_Ts, np.sum(post_p_complete, axis=1), 'r--');
plt.xlabel(r'$T_s$');
plt.ylabel(r'$p(T_s|\mathrm{data})$');

plt.plot(post_fail_temp, post_fail_prob, 'b-');
plt.plot(post_fail_temp, post_fail_prob_complete, 'r--');
plt.xlabel(r'$T$');
plt.ylabel(r'$P_\mathrm{failure}(T)$');


I am working on the file "Sessions/Session10/Day4/missing_information/missingdata_tutorial.ipynb".
The first task is:
Plot the function for a few different parameter values.
Can you write Python code for it?

T_axis = np.arange(32., 100.)
plt.plot(T_axis, P_success(T_axis, 70., 3.));
plt.plot(T_axis, P_success(T_axis, 70., 8.));
plt.plot(T_axis, P_success(T_axis, 50., 3.));
plt.xlabel('temperature');
plt.ylabel('Probability of a clean launch');


I am working on the file "Sessions/Session10/Day4/missing_information/missingdata_tutorial.ipynb".
The first task is:
Prove that we did that right (cf above).
Can you write Python code for it?

plt.plot(oring_temps, oring_incidents, 'bo');
plt.xlabel('temperature');
plt.ylabel('Number of incidents');


I am working on the file "Sessions/Session10/Day4/missing_information/missingdata_tutorial.ipynb".
The first task is:
Read in the data.
Can you write Python code for it?

oring_data = np.loadtxt('data/oring.dat', skiprows=1)
oring_temps = oring_data[:,0]
oring_incidents = oring_data[:,1]


I am working on the file "Sessions/Session10/Day4/missing_information/missingdata_tutorial.ipynb".
The first task is:
Something to think about: how would you go about quantifying the significance of a temperature dependent effect, given this model?
3. Complete data
For comparison, let's see how the constraints differ if we use the complete (uncensored) data set.
Implement the likelihood for the original, uncensored model.
Can you write Python code for it?

try:
    exec(open('solutions/likelihoodcomplete.py').read())
except IOError:
    REMOVE_THIS_LINE()
    def ln_like_complete(REPLACE_WITH_YOUR_ARGUMENT_LIST_OF_CHOICE):
        REPLACE_WITH_YOUR_SOLUTION()


I am working on the file "Sessions/Session10/Day4/missing_information/missingdata_tutorial.ipynb".
The first task is:
Tutorial: Coping with Missing Information
O-ring failure rates prior to the Challenger shuttle loss
In this tutorial, we will use a real data set where unwise data selection had serious consequences to illustrate how such selection effects could be modeled.
Background
On January 28, 1986, the Space Shuttle Challenger was destroyed in an explosion during launch. The cause was eventually found to be the failure of an O-ring seal that normally prevents hot gas from leaking between two segments of the solid rocket motors during their burn. The ambient atmospheric temperature of just 36 degrees Fahrenheit,  significantly colder than any previous launch, was determined to be a significant factor in the failure.
A relevant excerpt from the Report of the Presidential Commission on the Space Shuttle Challenger Accident reads:

Temperature Effects
The record of the fateful series of NASA and Thiokol meetings, telephone conferences, notes, and facsimile transmissions on January 27th, the night before the launch of flight 51L, shows that only limited consideration was given to the past history of O-ring damage in terms of temperature. The managers compared as a function of temperature the flights for which thermal distress of O-rings had been observed-not the frequency of occurrence based on all flights (Figure 6). In such a comparison, there is nothing irregular in the distribution of O-ring "distress" over the spectrum of joint temperatures at launch between 53 degrees Fahrenheit and 75 degrees Fahrenheit. When the entire history of flight experience is considered, including"normal" flights with no erosion or blow-by, the comparison is substantially different (Figure 7).
This comparison of flight history indicates that only three incidents of O-ring thermal distress occurred out of twenty flights with O-ring temperatures at 66 degrees Fahrenheit or above, whereas, all four flights with O-ring temperatures at 63 degrees Fahrenheit or below experienced O-ring thermal distress.
Consideration of the entire launch...
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

plt.rcParams.update({'font.size': 16});


I am working on the file "Sessions/Session10/Day5/model_selection/model_comparison_tutorial.ipynb".
The first task is:
2. Evaluating a Simple Model
First, let's assume a simple model $H_1$, that the sampling distribution is an exponential:
Model 1: $P(y|a_1, H_1) = \frac{1}{a_1}e^{-y/a_1}$; $y\geq0$
Our single parameter is $a_1$, the mean of the exponential distribution.
2a. Implementation in code
Complete the implementation of this model as a derived class of Model, below. Note than an ExponentialModel object still has all the methods defined for Model, in particular the ones that we don't need to redefine here.
Note that this includes choosing a reasonable prior for $a_1$. It should be a proper (normalizable) distribution. We don't want to deal with improper distributions when calculating the evidence later on.
Make sure you understand the workings of even the functions that are completely given.
Can you write Python code for it?

try:
    exec(open('solutions/exponentialmodel.py').read())
except IOError:
    REMOVE_THIS_LINE()

    class ExponentialModel(Model):
        """
        Simple exponential model for mis-centering.
        """
        def __init__(self):
            # Define any hyperparameters for the a1 prior here.
            # E.g., for uniform, something like "self.min_a1 = value" and "self.max_a1 = value"
            # More sophisticatedly, you could make these values arguments of __init__.
            REPLACE_WITH_YOUR_SOLUTION()
            # The next line finishes initialization by calling the parent class' __init__
            Model.__init__(self)

        def log_prior(self, a1):
            """
            Evaluate the log prior PDF P(a1|H)
            """
            REPLACE_WITH_YOUR_SOLUTION()

        def draw_samples_from_prior(self, N):
            """
            Return N samples of a1 from the prior PDF P(a1|H)
            """
            REPLACE_WITH_YOUR_SOLUTION()

        def log_likelihood(self, a1):
            """
            Evaluate the log of the likelihood function L(a1) = P(y|a1,H)
            Argument a1 is scalar.
            """
            return np.sum(st.expon.logpdf(y, scale=a1))

        def sampling_distribution(self, yy, a1):
            """
            Evaluate the sampling distribution P(yy|a,H) at a point in data space yy given parameter value a1
            We expect a vector input yy, and return the corresponding probabilities.

            Note: This is useful for making plots of "the model" overlaid on the histogram of the data
            """
            return st.expon.pdf(yy, scale=a1)

        def generate_replica_dataset(self, a1):
            """
            Draw a replica data set y_rep from the sampling distribution P(y_rep|a1,H).
            y_rep should have the same length as the true data set.
            Argument a1 is a scalar.
            """
            REPLACE_WITH_YOUR_SOLUTION()


I am working on the file "Sessions/Session10/Day5/model_selection/model_comparison_tutorial.ipynb".
The first task is:
2b. Fit the model to the data
The draw_samples_from_posterior method carries out a parameter inference with emcee, displaying its Markov chains, removing burn-in, thinning, and concatenating the chains. Since this step isn't really the point of this problem, the code is given to you, but you'll still need to experiment with the keyword argument ("kwarg") inputs (and read the code to see what they do) in order to get good results. (The suggestions in the cell below are pretty terrible.)
As a rule, you should start with burn=0 and thinby=1, and set these appropriately for a final run once you know what the time to convergence and autocorrelation roughly are.
The MCMC samples are stored in the Model.samples array.
Can you write Python code for it?

try:
    exec(open('solutions/fit.py').read())
except IOError:
    # This will execute out of the box, but will not work well. The arguments should be fiddled with.
    Model1.draw_samples_from_posterior(guess=[1000.0], nwalkers=8, nsteps=10, burn=0, thinby=1)


I am working on the file "Sessions/Session10/Day5/model_selection/model_comparison_tutorial.ipynb".
The first task is:
Check out a quick histogram of the data.
Can you write Python code for it?

plt.rcParams['figure.figsize'] = (8.0, 6.0)
bins = np.linspace(0,1000,20)
plt.hist(y, bins=bins, color="skyblue");
plt.xlabel("Measured distance $y$");
plt.ylabel("Frequency");


I am working on the file "Sessions/Session10/Day5/model_selection/model_comparison_tutorial.ipynb".
The first task is:
Compute the test statistic of the real data to verify that it works.
Can you write Python code for it?

T_true = T(y)
print("The test statistic of the real data is T =", T_true)


I am working on the file "Sessions/Session10/Day5/model_selection/model_comparison_tutorial.ipynb".
The first task is:
Roughly how precisely do we need to know the log Evidence, to be able to compare models? Run log_evidence with different values of N (the number of prior samples in the average) to until you're satisfied that you're getting a usefully accurate result.
Can you write Python code for it?

for Nevidence in [1, 10, 100]: # You *will* want to change these values
    %time logE1 = log_evidence(Model1, N=Nevidence)
    print("From", Nevidence, "samples, the log-evidence is", logE1, "\n")


I am working on the file "Sessions/Session10/Day5/model_selection/model_comparison_tutorial.ipynb".
The first task is:
This kind of plot should be familiar: it's often a good idea to evaluate model adequacy in data space. You should already be able to see telling differences between the a well-fitting model's sampling distribution, and the data histogram.
Now, let's compare a random predicted ("replica") data set, drawn from the posterior predictive distribution, with the data. To do this we first draw a parameter value from the posterior PDF, and then generate a dataset from the sampling distribution conditioned on that value. The result is a sample from $P(y_{rep}|y)$.
Can you write Python code for it?

plt.rcParams['figure.figsize'] = (8.0, 6.0)

# First the histogram of observed data, as backdrop:
plt.hist(y, bins=bins, color="skyblue", density=True, label="Observed")

# Choose a posterior sample at random and generate a replica dataset, and show its histogram
j = np.random.randint(0, len(Model1.samples))
mock = Model1.generate_replica_dataset(Model1.samples[j])
plt.hist(mock, bins=bins, alpha=1.0, histtype="step", color="red", density=True, label="Sample posterior prediction")

plt.xlabel("Measured distance $y$")
plt.ylabel("Normalized Frequency")
plt.legend();


I am working on the file "Sessions/Session10/Day5/model_selection/model_comparison_tutorial.ipynb".
The first task is:
This plot is nice because it is comparing apples with apples: do mock datasets drawn from our model sampling distribution with any plausible parameter value "look like" the real data?
To best evaluate this, we want to visualize the posterior predictive distribution of replica datasets. We can do this by plotting many replica datasets drawn from the posterior predictive PDF, e.g. one for each of our posterior samples. Let's put this plot in a function, so we can re-use it later.
Can you write Python code for it?

def visual_check(Model, Nreps=None):
    plt.rcParams['figure.figsize'] = (8.0, 6.0)

    # First the histogram of observed data, as backdrop:
    plt.hist(y, bins=bins, color="skyblue", density=True, label="Observed")

    # Compute the posterior mean parameter (vector)
    pm = np.mean(Model.samples, axis=0)

    # Make a large number of replica datasets, and overlay histograms of them all
    if Nreps is None: Nreps = len(Model.samples)
    alpha = 5.0 / Nreps
    for jj in np.round(np.linspace(0, len(Model.samples), num=Nreps, endpoint=False)):
        j = int(jj)
        if j==0:
            # Plot a dataset drawn using a = the posterior mean a, to give a good legend
            mock = Model.generate_replica_dataset(pm)
            plt.hist(mock, bins=bins, histtype="step", alpha=1.0, color="red", density=True, label="Sample posterior predictions")
        else:
            # Take the next posterior sample a and generate a replica dataset
            mock = Model1.generate_replica_dataset(Model.samples[j])
            plt.hist(mock, bins=bins, histtype="step", alpha=alpha, color="red", density=True)

    # Include the posterior mean model for comparison
    pp = Model1.sampling_distribution(bins, pm)
    plt.plot(bins, pp, linestyle="dashed", color="red", label="Posterior mean model")

    plt.xlabel("Measured distance $y$")
    plt.ylabel("Normalized Frequency")
    plt.legend();

visual_check(Model1, Nreps=100)


I am working on the file "Sessions/Session10/Day5/model_selection/model_comparison_tutorial.ipynb".
The first task is:
Tutorial: Checking and Comparing Models
Goodness of fit, information criteria, and Bayesian evidence
Introduction
In this tutorial we'll look at some simple, realistic, simulated data, and do some model evaluation, including

fitting a simple model, and then do a posterior predictive model check of the adequacy of the fit
quantifying the generalized predictive accuracy of the model with the Deviance Information Criterion (DIC)
calculating the Bayesian evidence for the model

Then you'll get to do it all again with a more complex model and determine which is preferred!
The Dataset
Our data is just a list of numbers. Each one represents a measured distance, $y$, between two different estimates of the center of a galaxy cluster: the location of the presumed central galaxy and a centroid of the diffuse, emissive gas. The context here is that automated algorithms sometimes fail to chose the central galaxy correctly (because of image artifacts or other problems), whereas the gas centroid is more reliable but also more expensive to measure. Therefore, we'd like to use this data set to characterize the distribution of mis-centerings so that the galaxy-based centers can be used for large sample, with the resulting errors propagated forward through future processing, e.g., weak lensing estimates.
Let's load up the data and have a look.
Can you write Python code for it?

import numpy as np
import scipy.stats as st
from scipy.special import logsumexp
import emcee
import matplotlib.pyplot as plt
%matplotlib inline

plt.rcParams.update({'font.size': 16});


I am working on the file "Sessions/Session10/Day5/model_selection/model_comparison_tutorial.ipynb".
The first task is:
We can now do the following:
* plot a histogram of $T(\mathrm{mock~data})$
* compare that distribution with $T(\mathrm{real~data})$
* compute and report the p-value for $T(\mathrm{real~data})$
And we want all of that in packaged in functions of the model, so that we can re-use it later (on different models!).
First let's write a function to compute the p-value, $P(T > T(y)|y,H)$:
Can you write Python code for it?

try:
    exec(open('solutions/pvalue.py').read())
except IOError:
    REMOVE_THIS_LINE()
    def pvalue(Model):
        """
        Compute the posterior predictive p-value, P(T > T(y)|y,H):
        """
        REPLACE_WITH_YOUR_SOLUTION()


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometry.ipynb".
The first task is:
Problem 1) An (oversimplified) 1-D Model
For this introductory problem we are going to simulate a 1 dimensional detector. Simulated stars that are detected by said detector will have Gaussian profiles $\mathcal{N}(\mu, \sigma^2)$, with mean position $\mu$ and variance $\sigma^2$.
As observed by LSST, all stars are point sources that reflect the point spread function (PSF), which is produced by a combination of the atmosphere, telescope, and detector. A standard measure of the PSF's width is the Full Width Half Maximum (FWHM). For a Gaussian the FWHM = $2\sqrt{2 \ln (2)} \sigma \approx 2.3548\,\sigma$.
In addition to the signal from the stars, the 1D detector also detects a smooth background of light from several sources (the atmosphere, the detector, etc). We will refer to this background simply as "The Sky."
Problem 1a
Write a function phi() to simulate a (noise-free) 1D Gaussian PSF. The function should take mu and fwhm as arguments, and evaluate the PSF along a user-supplied array x.
Can you write Python code for it?

def phi(x, mu, fwhm):
    """Evalute the 1d PSF N(mu, sigma^2) along x

    Parameters
    ----------
    x : array-like of shape (n_pixels,)
        detector pixel number
    mu : float
        mean position of the 1D star
    fwhm : float
        Full-width half-maximum of the stellar profile on the detector

    Returns
    -------
    flux : array-like of shape (n_pixels,)
        Flux in each pixel of the input array
    """
    sigma = # complete
    flux = # complete
    return # complete


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometry.ipynb".
The first task is:
Problem 1b
Plot the noise-free PSF for a star with a profile defined by $\mu = 10$ and $\mathrm{FWHM} = 3$.
Estimate the total flux of this star by "integrating" over the counts measured by the detector. If you are clever in the definition of your pixels, this integration reduces to a sum.
Hint - think about your input grid of pixels. Can you have a non-integer number of pixels? Also - the flux should be evaluated at the center of the pixel.
Can you write Python code for it?

x = # complete

pixel_plot( # complete

print("The flux of the star is: {:.3f}".format( # complete


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometry.ipynb".
The first task is:
Problem 4c
Build successively larger apertures with sizes increasing from a radius of 1 to 10 pixels. Measure the mean and variance for each aperture size. Plot the results.
Which aperture size has the smallest variance? Is this aperture best?
Do these results make sense?
Can you write Python code for it?

ap_array = # complete
# complete
# complete

for ap_num, ap_rad in enumerate(ap_array):
    # complete

fig, ax = plt.subplots()
ax.errorbar( # complete
# complete
# complete
# complete

print('aperture radius = {} has the smallest variance'.format( # complete


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometry.ipynb".
The first task is:
Problem 4e
Plot the uncertainty on the flux estimate (i.e., the square root of the variance) as a function of aperture radius.
Now which aperture size do you think is best?
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot( # complete
ax.set_xlabel('aperture radius (pix)')
ax.set_ylabel('uncertainty in F')
fig.tight_layout()


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometry.ipynb".
The first task is:
Problem 4g
Leaving all other variables the same, estimate the optimal aperture size (i.e. maximize the signal-to-noise ratio) for a star with a flux of 10.
What is the optimal aperture size?
Can you even measure the flux of this star?
Can you write Python code for it?

# complete
# complete
# complete

# complete
# complete
# complete

print('The optimal aperture is {} pix'.format( # complete


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometry.ipynb".
The first task is:
Problem 5b
Using the same parameters as problem 3, simulate a star and measure it's PSF flux.
Hint - you may find the minimize function from scipy.optimize helpful.
Can you write Python code for it?

# complete
# complete
# complete

sim_star = simulate(x, mu, fwhm, S, F)
psf_flux = # complete

print("The PSF flux is {:.3f}".format( # complete