I am working on the file "Sessions/Session09/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 3) Feature Engineering
It should now be clear why this is not a machine learning problem. We have, in this case, provided all the machine learning code that you will need.
Instead, this is a feature engineering problem. Feature engineering requires the utilization of domain knowledge to create new features for a data set to improve the performance of machine learning algorithms.
Add new features - if necessary
  - Utilize domain knowledge to create/compute new features
  - Combine features or represent them in an alternative fashion
Remove noisy/uniformative features - if necessary
  - Determine feature importance (RF)
  - Forward/backward selection to iteratively remove features
Below I have provided a partial function calc_feature which you can alter to calculate new features for the data set.
Can you write Python code for it?

def calc_feature(df, train=True):

    if train==True:
        lc_dir = "./training_lcs/"
    else:
        lc_dir = "./test_lcs/"

    feature = np.empty(len(df))
    for source_num, asas_id in enumerate(df["ASAS_ID"]):
        t, m, mu = read_lc(lc_dir+asas_id)
        # feature calculations
        # feature calculations
        # feature calculations

        feature[source_num] = feat_val

    return feature


I am working on the file "Sessions/Session09/Day2/FeatureEngineering.ipynb".
The first task is:
The provided training set comes with 3 features: i) the mean magnitude of the observations, ii) the total number of observations obtained, and iii) the duration of the observations.
Problem 2b
Using the helper function provided below, calculate the 10-fold cross-validation accuracy of a random forest machine learning model using the 3 features provided above.
Note - do not adjust any part of calc_cv_score throughout this exercise.
Can you write Python code for it?

def calc_cv_score(X, y):
    rf_clf = RandomForestClassifier(n_estimators=150, min_samples_leaf=1)

    cv_score = cross_val_score(rf_clf, X, y, cv=10, n_jobs=-1)

    print("These features have CV accuracy = {:.4f} +/- {:.4f}".format(np.mean(cv_score), np.std(cv_score, ddof=1)))

calc_cv_score( # complete


I am working on the file "Sessions/Session09/Day2/gps/03-FastGPs.ipynb".
The first task is:
Benchmarking our implementation
Let's do some timing tests and compare them to what we get with two handy GP packages: george and celerite. We'll learn how to use both along the way.
Below is the code we wrote in the Day 1 tutorials to sample from and compute the likelihood of a GP.
Can you write Python code for it?

import numpy as np
from scipy.linalg import cho_factor

def ExpSquaredKernel(t1, t2=None, A=1.0, l=1.0):
    """
    Return the ``N x M`` exponential squared
    covariance matrix between time vectors `t1`
    and `t2`. The kernel has amplitude `A` and
    lengthscale `l`.

    """
    if t2 is None:
        t2 = t1
    T2, T1 = np.meshgrid(t2, t1)
    return A ** 2 * np.exp(-0.5 * (T1 - T2) ** 2 / l ** 2)

def ln_gp_likelihood(t, y, sigma=0, A=1.0, l=1.0):
    """
    Return the log of the GP likelihood of the
    data `y(t)` given uncertainty `sigma` and
    an Exponential Squared Kernel with amplitude `A`
    and length scale `sigma`.

    """
    # The covariance and its determinant
    npts = len(t)
    kernel = ExpSquaredKernel
    K = kernel(t, A=A, l=l) + sigma ** 2 * np.eye(npts)

    # The marginal log likelihood
    log_like = -0.5 * np.dot(y.T, np.linalg.solve(K, y))
    log_like -= 0.5 * np.linalg.slogdet(K)[1]
    log_like -= 0.5 * npts * np.log(2 * np.pi)

    return log_like

def draw_from_gaussian(mu, S, ndraws=1, eps=1e-12):
    """
    Generate samples from a multivariate gaussian
    specified by covariance ``S`` and mean ``mu``.

    (We derived these equations in Day 1, Notebook 01, Exercise 7.)
    """
    npts = S.shape[0]
    L, _ = cho_factor(S + eps * np.eye(npts), lower=True)
    L = np.tril(L)
    u = np.random.randn(npts, ndraws)
    x = np.dot(L, u) + mu[:, None]
    return x.T

def compute_gp(t_train, y_train, t_test, sigma=0, A=1.0, l=1.0):
    """
    Compute the mean vector and covariance matrix of a GP
    at times `t_test` given training points `y_train(t_train)`.
    The training points have uncertainty `sigma` and the
    kernel is assumed to be an Exponential Squared Kernel
    with amplitude `A` and lengthscale `l`.

    """
    # Compute the required matrices
    kernel = ExpSquaredKernel
    Stt = kernel(t_train, A=1.0, l=1.0)
    Stt += sigma ** 2 * np.eye(Stt.shape[0])
    Spp = kernel(t_test, A=1.0, l=1.0)
    Spt = kernel(t_test,...


I am working on the file "Sessions/Session09/Day2/gps/answers/03-FastGPs.ipynb".
The first task is:

Exercise 1b

Let's time how long it takes to do the same operations using the george package (pip install george).
The kernel we'll use is
python
kernel = amp ** 2 * george.kernels.ExpSquaredKernel(tau ** 2)
where amp = 1 and tau = 1 in this case.
To instantiate a GP using george, simply run
python
gp = george.GP(kernel)
The george package pre-computes a lot of matrices that are re-used in different operations, so before anything else, ask it to compute the GP model for your timeseries:
python
gp.compute(t, sigma)
Note that we've only given it the time array and the uncertainties, so as long as those remain the same, you don't have to re-compute anything. This will save you a lot of time in the long run!
Finally, the log likelihood is given by gp.log_likelihood(y) and a sample can be drawn by calling gp.sample().
How do the speeds compare? Did you get the same value of the likelihood (assuming you computed it for the same sample in both cases)?
Can you write Python code for it?

import george

%%time
kernel = george.kernels.ExpSquaredKernel(1.0)
gp = george.GP(kernel)
gp.compute(t, sigma)

%%time
print(gp.log_likelihood(y))

%%time
gp.sample()


I am working on the file "Sessions/Session09/Day2/gps/answers/03-FastGPs.ipynb".
The first task is:

Exercise 1c

george offers a fancy GP solver called the HODLR solver, which makes some approximations that dramatically speed up the matrix algebra. Instantiate the GP object again by passing the keyword solver=george.HODLRSolver and re-compute the log likelihood. How long did that take?
(I wasn't able to draw samples using the HODLR solver; unfortunately this may not be implemented.)
Can you write Python code for it?

%%time
gp = george.GP(kernel, solver=george.HODLRSolver)
gp.compute(t, sigma)

%%time
gp.log_likelihood(y)


I am working on the file "Sessions/Session09/Day2/gps/answers/03-FastGPs.ipynb".
The first task is:

Exercise 2

The george package is super useful for GP modeling, and I recommend you read over the docs and examples. It implements several different kernels that come in handy in different situations, and it has support for multi-dimensional GPs. But if all you care about are GPs in one dimension (in this case, we're only doing GPs in the time domain, so we're good), then celerite is what it's all about:
bash
pip install celerite
Check out the docs here, as well as several tutorials. There is also a paper that discusses the math behind celerite. The basic idea is that for certain families of kernels, there exist extremely efficient methods of factorizing the covariance matrices. Whereas GP fitting typically scales with the number of datapoints $N$ as $N^3$, celerite is able to do everything in order $N$ (!!!) This is a huge advantage, especially for datasets with tens or hundreds of thousands of data points. Using george or any homebuilt GP model for datasets larger than about 10,000 points is simply intractable, but with celerite you can do it in a breeze.
Repeat the timing tests, but this time using celerite. Note that the Exponential Squared Kernel is not available in celerite, because it doesn't have the special form needed to make its factorization fast. Instead, use the Matern 3/2 kernel, which is qualitatively similar, and which can be approximated quite well in terms of the celerite basis functions:
python
kernel = celerite.terms.Matern32Term(np.log(1), np.log(1))
Note that celerite accepts the log of the amplitude and the log of the timescale. Other than this, you should be able to compute the likelihood and draw a sample with the same syntax as george.
How much faster did it run?
Can you write Python code for it?

import celerite
from celerite import terms

%%time
kernel = terms.Matern32Term(np.log(1), np.log(1))
gp = celerite.GP(kernel)
gp.compute(t, sigma)

%%time
gp.log_likelihood(y)

%%time
gp.sample()


I am working on the file "Sessions/Session09/Day2/gps/answers/03-FastGPs.ipynb".
The first task is:

Exercise 3

Let's use celerite for a real application: fitting an exoplanet transit model in the presence of correlated noise.
Below is a (fictitious) light curve for a star with a transiting planet. There is a transit visible to the eye at $t = 0$, which (say) is when you'd expect the planet to transit if its orbit were perfectly periodic. However, a recent paper claims that the planet shows transit timing variations, which are indicative of a second, perturbing planet in the system, and that a transit at $t = 0$ can be ruled out at 3 $\sigma$. Your task is to verify this claim.
Assume you have no prior information on the planet other than the transit occurs in the observation window, the depth of the transit is somewhere in the range $(0, 1)$, and the transit duration is somewhere between $0.1$ and $1$ day. You don't know the exact process generating the noise, but you are certain that there's correlated noise in the dataset, so you'll have to pick a reasonable kernel and estimate its hyperparameters.
Fit the transit with a simple inverted Gaussian with three free parameters:
python
def transit_shape(depth, t0, dur):
    return -depth * np.exp(-0.5 * (t - t0) ** 2 / (0.2 * dur) ** 2)
Read the celerite docs to figure out how to solve this problem efficiently.
HINT: I borrowed heavily from this tutorial, so you might want to take a look at it...
Can you write Python code for it?

import matplotlib.pyplot as plt
from celerite.modeling import Model
import os

# Define the model
class MeanModel(Model):
    parameter_names = ("depth", "t0", "dur")
    def get_value(self, t):
        return -self.depth * np.exp(-0.5 * (t - self.t0) ** 2 / (0.2 * self.dur) ** 2)

mean_model = MeanModel(depth=0.5, t0=0.05, dur=0.7)
mean_model.parameter_bounds = [(0, 1.0), (-0.1, 0.4), (0.1, 1.0)]
true_params = mean_model.get_parameter_vector()

# Simuate the data
np.random.seed(71)
x = np.sort(np.random.uniform(-1, 1, 70))
yerr = np.random.uniform(0.075, 0.1, len(x))
K = 0.2 * np.exp(-0.5 * (x[:, None] - x[None, :]) ** 2 / 10.5)
K[np.diag_indices(len(x))] += yerr ** 2
y = np.random.multivariate_normal(mean_model.get_value(x), K)
y -= np.nanmedian(y)

# Plot the data
plt.errorbar(x, y, yerr=yerr, fmt=".k", capsize=0)
t = np.linspace(-1, 1, 1000)
plt.plot(t, mean_model.get_value(t))
plt.ylabel(r"$y$")
plt.xlabel(r"$t$")
plt.xlim(-1, 1)
plt.gca().yaxis.set_major_locator(plt.MaxNLocator(5))
plt.title("simulated data");

# Save it
X = np.hstack((x.reshape(-1, 1), y.reshape(-1, 1), yerr.reshape(-1, 1)))
if not (os.path.exists("data")):
    os.mkdir("data")
np.savetxt("data/sample_transit.txt", X)

import matplotlib.pyplot as plt
t, y, yerr = np.loadtxt("data/sample_transit.txt", unpack=True)
plt.errorbar(x, y, yerr=yerr, fmt=".k", capsize=0)
plt.xlabel("time")
plt.ylabel("relative flux");

from celerite.modeling import Model
from scipy.optimize import minimize

# Define the transit model as a celerite `Model`
class MeanModel(Model):
    parameter_names = ("depth", "t0", "dur")
    def get_value(self, t):
        return -self.depth * np.exp(-0.5 * (t - self.t0) ** 2 / (0.2 * self.dur) ** 2)

# Instantiate it with some guesses (which are actually the true values in this case!)
mean_model = MeanModel(depth=0.5, t0=0.05, dur=0.7)
mean_model.parameter_bounds = [(0, 1.0), (-0.1, 0.4), (0.1, 1.0)]
true_params = mean_model.get_parameter_vector()

# Set up the GP model
kernel = terms.RealTerm(log_a=np.log(np.var(y)),...


I am working on the file "Sessions/Session09/Day2/gps/answers/03-FastGPs.ipynb".
The first task is:
Benchmarking our implementation
Let's do some timing tests and compare them to what we get with two handy GP packages: george and celerite. We'll learn how to use both along the way.

Exercise 1a

Let's time how long our custom implementation of a GP takes for a rather long dataset. Create a time array of 10,000 points between 0 and 10 and time how long it takes to sample the prior of the GP for the default kernel parameters (unit amplitude and timescale). Add a bit of noise to the sample and then time how long it takes to evaluate the log likelihood for the dataset. Make sure to store the value of the log likelihood for later.
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Jess notes: the following was produced using a python3.7 kernel. All code dependencies should be installed via the installation instructions above except for python; careful with your python paths here.
Learning goals
With this tutorial, you will learn how to:

Use basic GWOSC tools to query for LIGO-Virgo observing run times and GW event times
Download public LIGO (and Virgo) data from the GWOSC with GWpy
Plot a LIGO h(t) time series with GWpy
Make a spectral density plot of a time series
Design a filter in the frequency domain
Apply and characterize a whitening filter
Visualize LIGO data with spectrograms

This tutorial borrows from the excellent work of Duncan Macleod, Jonah Kanner, Alex Nitz, and others involved in the 2018 GWOSC webcourse - you can find many more great examples there.
Let's get started!

Using GWOSC tools
Here we'll use tools from the Gravitational Wave Open Science Center (GWOSC), namely the gwosc python module, which we should have already installed via our GWpy install.
First, let's see what's in GWOSC open datasets:
Can you write Python code for it?

from gwosc.datasets import find_datasets
find_datasets()


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
<span style="color:green">
Exercise
There's an error here! Can you find a way to fix the cell above and print the start and end times for the second LIGO-Virgo observing run (O2)?
</span>
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
<span style="color:green">
Exercises for using the GWOSC tools

How many events were detected during O2?
Which O2 event releases include data for the Virgo detector?
</span>
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
And O2:
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Apply and characterize a whitening filter
It's much easier to spot excess power in the data if we can weight it proportionally by the consistent frequency contributions from the noise its embedded in. (Whitening is a critical step for matched filtering and gravitational wave search algorithms that look for coherent excess power across a gravitational wave detector network.)
We can use GWpy to whiten our data for the time of GW150914 with an FFT length of 5 seconds and an overlap of 2 seconds.
Can you write Python code for it?

whitened_gw150914 = data.whiten(5,2)
plot = whitened_gw150914.plot(
    title='Whitened LIGO Hanford Observatory data for GW150914',
    ylabel='Strain amplitude',
    xlim=(GW150914gps-0.2, GW150914gps+0.2)
)


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Challenge 2
Do the same thing for GW170817, but this time using data from LIGO-Hanford. Can you estimate the time delay between the two detectors after running this procedure?
</span>
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Hmm, that looks like it still has some high frequency noise left.
What does the whitened data look like in the frequency domain?
Can you write Python code for it?

whitened_gw150914_asd = whitened_gw150914.asd()
plot = whitened_gw150914_asd.plot()
ax = plot.gca()
ax.set_xlim(5, 2000)


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Importing open LIGO data
We can use GWpy to download a LIGO time series from the GWOSC with fetch_open_data. Let's try to grab some data around a detected event, say GW150914 (the very first direct gravitational wave detection).
<span style="color:gray"> Note: The first time you import gwpy.timeseries, matplotlib may try to import some extra fonts and that can take a couple minutes. </span>
Can you write Python code for it?

# Import
from gwpy.timeseries import TimeSeries

# Grab the event gps time with GWOSC tools
GW150914gps = event_gps('GW150914')
print(GW150914gps)

# Set a start and end time around the event time (in units of seconds) to download LIGO data
window = 15
start = GW150914gps - window
end = GW150914gps + window

# Check that this looks sane
print('start time GPS = '+str(start))
print('end time GPS = '+str(end))


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Nice.

Make a spectral density plot
What does the data containing the signal look like in the frequency domain?
We can then call the .asd() method to calculate the amplitude spectral density and tranform our TimeSeries into a FrequencySeries.
Syntax:
asd(FFT_length_in_seconds, FFT_overlap_in_seconds, default_time_window='hann', default_FFT_method='welch')
Can you write Python code for it?

gw140914asd = data.asd(5, 2)
print(gw140914asd)


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Now that we've identified a time range of interest, we can grab some data. Let's choose the LIGO-Hanford detector (H1).
Can you write Python code for it?

# Grab H1 data around the time of interest from the GWOSC
data = TimeSeries.fetch_open_data('H1', start, end, verbose=True)
print(data)


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Querying event GPS times
If we want to know the GPS time of a particular event, we can grab that with event_gps. Let's try that for the first binary neutron star detection, GW170817.
Can you write Python code for it?

from gwosc.datasets import event_gps
GW170817gps = event_gps('GW170817')
print(GW170817gps)


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Querying for data around detected events
We can also query for event releases (data on the order of minutes around the event time) specifically by using find_datasets and specifying the event type.
Can you write Python code for it?

from gwosc.datasets import find_datasets
events = find_datasets(type='event')
print(events)


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Requiring verbose=True gives us details on the data download (and helps with debugging if needed).
The downloaded file is not stored permanently! If you run this cell again it will be downloaded again. However, you can use cache=True to store the file on your computer if you like.

Plot a LIGO time series
We can use GWpy to plot the data we downloaded from the GWOSC by calling the plot() method of our data object.

Jess note: This may take 2-3 minutes to render the first time.
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Starting to look promising, but check out that strong unrelated sinusoid. What the heck is that?
To the frequency domain! (Read: let's make a spectrum and check it out.)
Make an ASD of the data that has been passed through the highpass filter. Use an FFT length of 5 seconds and an overlap of 2 seconds.
Can you write Python code for it?

gw140914bpasd = gw140914bp.asd( # complete

## make a plot of that ASD
asdplot = gw140914bpasd.plot()
ax = asdplot.gca()
ax.set_xlim(5, 2000)


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Still no. What's left? What does our highpassed data look like in the frequency domain?
Make an ASD of the data that has been passed through the highpass filter. Use an FFT length of 5 seconds and an overlap of 2 seconds.
Can you write Python code for it?

gw140914hpasd = gw140914hp.asd( # complete

## make a plot of that ASD!
asdplot = gw140914hpasd.plot()
ax = asdplot.gca()
ax.set_xlim(5, 2000)


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
There is something there! It looks like it's sweeping up in frequency over time, but it's still hard to make out the details.

The Q transform
To zoom in even further, we can employ a muli-resolution technique called the Q transform.
Can you write Python code for it?

qspecgram = data.q_transform(outseg=(GW150914gps-0.2, GW150914gps+0.2))
plot = qspecgram.plot()
ax = plot.gca()
ax.set_xscale('seconds')
ax.set_yscale('log')
ax.set_ylim(20, 500)
ax.set_ylabel('Frequency [Hz]')
ax.grid(True, axis='y', which='both')
ax.colorbar(cmap='viridis', label='Normalized energy')


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
This is the real LIGO data (one of two detectors) used to make the first direct detection of GWs. Do you spot the signal yet?
Let's add a title and y-axis labels to make it more clear what we're looking at.
Can you write Python code for it?

plot = data.plot(
    title='LIGO-Hanford Observatory data for GW150914',
    ylabel='Strain amplitude'
)


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
We've really supressed that low frequency noise! But there's still too much noise at other frequencies.
Apply a bandpass filter
Because we've already identified our signal as a binary black hole system (two equal-ish mass black holes of roughly 30 solar masses each), we know our signal has frequency content (within our detector's sensitive range) between 50 and 250 Hz.
Let's apply a bandpass filter to look for excess power in that critical frequency range.
Use the data.bandpass method to apply a bandpass filter with corner frequencies of 50 and 250 Hz.
Can you write Python code for it?

## apply a bandpass filter to the highpassed data with a corner frequencies of 50 and 250 Hz
gw140914bp = gw140914hp.bandpass( # complete

## make a plot of the resulting time series
tsplot = gw140914bp.plot(
    title='LIGO-Hanford Observatory data for GW150914',
    ylabel='Strain amplitude'
)
ax = tsplot.gca()
ax.set_xlim(GW150914gps-0.2, GW150914gps+0.2)
ax.set_ylim(-0.2e-20, 0.2e-20)


I am working on the file "Sessions/Session09/Day3/LIGO_data_visualization.ipynb".
The first task is:
Wow, check out that strong line; this is the 60 Hz AC power line!
Let's get rid of it with a notch filter.
Apply a notch filter
Use the data.notch method to apply a notch filter that will supress the signal at 60 Hz from nearby AC power lines.
Can you write Python code for it?

gw140914n = gw140914bp.notch( # complete

## make a plot of the resulting time series
tsplot = gw140914n.plot(
    title='LIGO-Hanford Observatory data for GW150914',
    ylabel='Strain amplitude'
)
ax = tsplot.gca()
ax.set_xlim(GW150914gps-0.2, GW150914gps+0.2)
ax.set_ylim(-0.1e-20, 0.1e-20)


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
<span style="color:gray">Jess notes: this notebook was made with a PyCBC 1.8.0 kernel. </span>
Learning goals
With this tutorial, you learn how to:

Generate source waveforms detectable by LIGO, Virgo, KAGRA
Use PyCBC to run a matched filter search on gravitational wave detector data
Estimate the significance of a trigger given a background distribution
Challenge: Code up a trigger coincidence algorithm

This tutorial borrows heavily from tutorials made for the LIGO-Virgo Open Data Workshop by Alex Nitz. You can find PyCBC documentation and additional examples here.
Let's get started!

Generate a gravitational wave signal waveform
We'll use a popular waveform approximant (SOEBNRv4) to generate waveforms that would be detectable by LIGO, Virgo, or KAGRA.
First we import the packages we'll need.
Can you write Python code for it?

from pycbc.waveform import get_td_waveform
import matplotlib.pyplot as plt


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
Calculate an SNR time series
Now that we've pre-conditioned our data and defined a signal model, we can compute the output of our matched filter search.
Can you write Python code for it?

from pycbc.filter import matched_filter
import numpy

snr = matched_filter(template, conditioned,
                     psd=psd, low_frequency_cutoff=20)

plt.figure(figsize=[10, 4])
plt.plot(snr.sample_times, abs(snr))
plt.xlabel('Time (s)')
plt.ylabel('SNR')


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
Challenge
Our match filter analysis assumes the noise is stationary and Gaussian, which is not a good assumption, and this short data set isn't representative of all the various things that can go bump in the detector (remember the phone?).
The simple significance estimate above won't work as soon as we encounter a glitch! We need a better noise background estimate, and we can leverage our detector network to help make our signals stand out from our background.
Observing a gravitational wave signal between detectors is an important cross-check to minimize the impact of transient detector noise. Our strategy:

We look for loud triggers within a time window to identify foreground events that occur within the gravitational wave travel time (v=c) between detectors, but could come from any sky position.
We use time slides to estimate the noise background for a network of detectors.

If you still have time, try coding up an algorithm that checks for time coincidence between triggers in different detectors. Remember that the maximum gravitational wave travel time between LIGO detectors is ~10 ms. Check your code with the GPS times for the H1 and L1 triggers you identified for GW150914.
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
Define a signal model
Recall that matched filtering is essentially integrating the inner product between your data and your signal model in frequency or time (after weighting frequencies correctly) as you slide your signal model over your data in time.
If there is a signal in the data that matches your 'template', we will see a large value of this inner product (the SNR, or 'signal to noise ratio') at that time.
In a full search, we would grid over the parameters and calculate the SNR time series for each template in our template bank
Here we'll define just one template. Let's assume equal masses (which is within the posterior probability of GW150914). Because we want to match our signal model with each time sample in our data, let's also rescale our signal model vector to match the same number of time samples as our data vector (<- very important!).
Let's also plot the output to see what it looks like.
Can you write Python code for it?

m = 36 # Solar masses
hp, hc = get_td_waveform(approximant="SEOBNRv4_opt",
                     mass1=m,
                     mass2=m,
                     delta_t=conditioned.delta_t,
                     f_lower=20)

# We should resize the vector of our template to match our data
hp.resize(len(conditioned))
plt.plot(hp)
plt.xlabel('Time samples')


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
Estimate the single-detector significance of an event candidate
Great, we found a large spike in SNR! What are the chances this is a real astrophysical signal? How often would detector noise produce this by chance?
Let's plot a histogram of SNR values output by our matched filtering analysis for this time and see how much this trigger stands out.
Can you write Python code for it?

# import what we need
from scipy.stats import norm
from math import pi
from math import exp

# make a histogram of SNR values
background = (abs(snr))

# plot the histogram to check out any other outliers
plt.hist(background, bins=50)
plt.xlabel('SNR')
plt.semilogy()

# use norm.fit to fit a normal (Gaussian) distribution
(mu, sigma) = norm.fit(background)

# print out the mean and standard deviation of the fit
print('The fit mean = %f and the fit std dev = %f' )%(mu, sigma)


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
Exercise 1
What happens to the waveform when the total mass (let's say 20 Msol) stays the same, but the mass ratio between the component masses changes?
Compare the waveforms for a m1 = m2 = 10 Msol system, a m1 = 5 Msol, m2 = 15 Msol, and  a m1 = 2 Msol, m2 = 18 Msol system. What do you notice?
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
Exercise 4
Network SNR is the quadrature sum of the single-detector SNR from each contributing detector. GW150914 was detected by H1 and L1. Try calculating the network SNR (you'll need to estimate the SNR in L1 first), and compare your answer to the network PyCBC SNR as reported in the GWTC-1 catalog.
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
Exercise 5
At what single-detector SNR is the significance of a trigger > 5 sigma?
Remember that sigma is constant for a normal distribution (read: this should be simple multiplication now that we have estimated what 1 sigma is).
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
Let's see what these waveforms look like for different component masses. We'll assume the two compact object have masses equal to each other, and we'll set a lower frequency bound of 30 Hz (determined by the sensitivity of our detectors).
We can also set a time sample rate with get_td_waveform. Let's try a rate of 4096 Hz.
Let's make a plot of the plus polarization (hp) to get a feel for what the waveforms look like.
Hint –– you may want to zoom in on the plot to see the waveforms in detail.
Can you write Python code for it?

for m in [5, 10, 30, 100]:
    hp, hc = get_td_waveform(approximant="SEOBNRv4_opt",
                         mass1=m,
                         mass2=m,
                         delta_t=1.0/4096,
                         f_lower=30)
    plt.plot(hp.sample_times, hp, label='$M_{\odot 1,2}=%s$' % m)
plt.legend(loc='upper left')
plt.ylabel('GW strain (plus polarization)')
plt.grid()
plt.xlabel('Time (s)')
plt.show()


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
Note that the waveform template currently begins at the start of the vector. However, we want our SNR time series (the inner product between our data and our template) to track with the approximate merger time. To do this, we need to shift our template so that the merger is approximately at the first bin of the data.
For this reason, waveforms returned from get_td_waveform have their merger stamped with time zero, so we can easily shift the merger into the right position to compute our SNR time series.
Let's try shifting our template time and plot the output.
Can you write Python code for it?

template = hp.cyclic_time_shift(hp.start_time)
plt.plot(template)
plt.xlabel('Time samples')


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
Now let's see what happens if we decrease the lower frequency bound from 30 Hz to 15 Hz.
Can you write Python code for it?

for m in [5, 10, 30, 100]:
    hp, hc = get_td_waveform(approximant="SEOBNRv4_opt",
                         mass1=m,
                         mass2=m,
                         delta_t=1.0/4096,
                         f_lower= # complete

    pylab.plot(hp.sample_times, hp, label='$M_{\odot 1,2}=%s$' % m)
pylab.legend(loc='upper left')
pylab.ylabel('GW strain (plus polarization)')
pylab.grid()
pylab.xlabel('Time (s)')
pylab.show()


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
Run a matched filter search on gravitational wave detector data
PyCBC also maintains a catalog of open data as PyCBC time series objects, easy to manipulate with PyCBC tools. Let's try using that and importing the data around the first detection, GW150914.
Can you write Python code for it?

from pycbc.catalog import Merger
from pycbc.filter import resample_to_delta_t, highpass

merger = Merger("GW150914")

# Get the data from the Hanford detector
strain = merger.strain('H1')


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
That's better.
Calculating the spectral density of the data
Optimal matched filtering requires whitening; weighting the frequency components of the potential signal and data by the estimated noise amplitude.
Let's compute the power spectral density (PSD) of our conditioned data.
Can you write Python code for it?

from pycbc.psd import interpolate, inverse_spectrum_truncation
# Estimate the power spectral density

# We use 4 second samles of our time series in Welch method.
psd = conditioned.psd(4)

# Now that we have the psd we need to interpolate it to match our data
# and then limit the filter length of 1 / PSD. After this, we can
# directly use this PSD to filter the data in a controlled manner

psd = interpolate(psd, conditioned.delta_f)

# 1/PSD will now act as a filter with an effective length of 4 seconds
# Since the data has been highpassed above 15 Hz, and will have low values
# below this we need to informat the function to not include frequencies
# below this frequency.
psd = inverse_spectrum_truncation(psd, 4 * conditioned.sample_rate,
                                  low_frequency_cutoff=15)


I am working on the file "Sessions/Session09/Day4/Matched_filter_tutorial.ipynb".
The first task is:
You found the first gravitational wave detection in LIGO Hanford data! Nice work.

Exercise 3
How does the SNR change if you re-compute the matched filter result using a signal model with compenent masses that are closer to the current estimates for GW150914, say m1 = 36 Msol and m2 = 31 Msol?
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day4/workbook_QPO.ipynb".
The first task is:
1a. Compute the time steps and a cosine harmonic with the above-defined properties.
1b. Compute four exponentially damped versions of the harmonic oscillation.
$$D(t_i) = e^{-\zeta t_i}H(t_i)$$
Pick your own four $\zeta$s! I recommend values between 0.01 and 1.
1c. Plot them all on top of each other.
1d. Take the power spectrum of the harmonic and 4 damped harmonic time series.
The power $P$ at each frequency $\nu_i$, for the Fourier transform $F$, is $$P(\nu_i)=|F(\nu_i)|^2$$
Computing the Fourier frequencies and the index of the Nyquist frequency.
Can you write Python code for it?

freq = fftpack.fftfreq(len(a), d=a.meta['dt'])
nyq_ind = int(len(a)/2.)  # the index of the last positive Fourier frequency


I am working on the file "Sessions/Session09/Day4/workbook_QPO.ipynb".
The first task is:
1e. Plot them!
Notice the trend between the width of the peak in the power spectrum, and the strength of the damping factor.
For bonus points, put in a key/legend with the corresponding $\zeta$ value for each curve.
Problem 2: Analyzing NICER data of the black hole X-ray binary MAXI J1535-571
Import it with astropy tables from the file "J1535_evt.fits".
The data have come to us as an 'event list', meaning that it's a list of the time at which a photon was detected (in seconds, in spacecraft clock time)  and what the energy of the photon was (a channel number; channel/100=photon energy in keV).
2a. Turn this rag-tag list of photons into an evenly-spaced light curve
2a.i. First, clean it up a little by only keeping photons with energies greater than 1 keV and less than 12 keV, using a mask.
2a.ii. Then, make an evenly-spaced light curve array with np.histogram. Pick a light curve time resolution of dt=1/8seconds to start with. To put your light curve in units of count rate, divide the histogram (counts per bin) by dt (seconds per bin); to avoid typecasting errors, do this by multiplying by int(1/dt).
(yes, it takes a second or two; you're using half a million time bins in your light curve!)
2b. Let's try taking the power spectrum of it.
$$P(\nu_i)=|F(\nu_i)|^2$$
where $F$ is the FFT of your light curve.
Can you write Python code for it?

freq = fftpack.fftfreq(len(psd), d=dt)
nyq_ind = int(len(psd)/2.)  # the index of the last positive Fourier frequency


I am working on the file "Sessions/Session09/Day4/workbook_QPO.ipynb".
The first task is:
Apply this to the data (using JUST the frequency, power, and error at positive Fourier frequencies). Start with a rebin factor of 1.03.
Play around with a few different values of rebin_factor to see how it changes the plotted power spectrum. 1 should give back exactly what you put in, and 1.1 tends to bin things up quite a lot.
Congratulations! You can make great-looking power spectra! Now, go back to part 2d. and try 4 or 5 different combinations of dt and seg_length. What happens when you pick too big of a dt to see the QPO frequency? What if your seg_length is really short?
One of the most important things to notice is that for a real astrophysical signal, the QPO (and low-frequency noise) are present for a variety of different dt and seg_length parameters.
2g. Normalization
The final thing standing between us and a publication-ready power spectrum plot is the normalization of the power along the y-axis. The normalization that's commonly used is fractional rms-squared normalization, sometimes just called the rms normalization. For a power spectrum created from counts/second unit light curves, the equation is:
$$P_{frac.rms2} = P \times \frac{2*dt}{N * mean^2}$$
P is the power we already have,
dt is the time step of the light curve,
N is n_bins, the number of bins in one segment, and
mean is the mean count rate (in counts/s) of the light curve (so, you will need to go back to 2d. and re-run keeping a running sum-then-average of the mean count rate).
Don't forget to apply the normalization to the error, and re-bin after!
2h. Poisson noise level
Notice that the Poisson noise is a power law with slope 0 at high frequencies. With this normalization, we can predict the power of the Poisson noise level from the mean counts/s rate of the light curve!
$$P_{noise} = 2/mean$$
Compute this noise level, and plot it with the power spectrum.
Your horizontal Poisson noise line should be really close to the power at and above ~10 Hz.
2i.  For plotting purposes, we sometimes subtract the Poisson noise level from the power...
Can you write Python code for it?

seg_length = 64. # seconds
dt = 1./8.# seconds
n_bins = int(seg_length/dt) # Number of time bins in a segment of light curve


I am working on the file "Sessions/Session09/Day4/workbook_QPO.ipynb".
The first task is:
Plot it! Use similar code I gave you above to make the array of Fourier frequencies and get the index of the Nyquist frequency.
2d. Mean-subtracted
So, you can see something just to the left 10 Hz much clearer, but there's this giant whopping signal at the lowest frequency bin. This is what I've heard called the 'DC component', which arises from the mean count rate of the light curve. To get rid of it, subtract the mean from your light curve segment before taking the Fourier transform. Otherwise, keep the same code as above for 2c.iii. You may want to put some of this in a function for future use in this notebook.
2e. Error on average power
The average power at a particular frequency has a chi-squared distribution with two degrees of freedom about the true underlying power spectrum. So, error is the power value divided by the square root of the number of segments. A big reason why we love power spectra(/periodograms) is because this is so straight forward!
One way to intuitively check if your errors are way-overestimated or way-underestimated is whether the size of the error bar looks commeasurate with the amount of bin-to-bin scatter of power at neighbouring frequencies.
Plotting, this time with ax.errorbar instead of ax.plot.
The thing at ~8 Hz is a low-frequency QPO, and the hump at-and-below 1 Hz is broadband noise (which we'll discuss in detail this afternoon)!! Now that you've got the basic analysis step complete, we'll focus on plotting the data in a meaningful way so you can easily extract information about the QPO and noise.
2f. Re-binning
We often plot power spectra on log-log scaled axes (so, log on both the X and Y), and you'll notice that there's a big noisy part above 10 Hz. It is common practice to bin up the power spectrum geometrically (which is like making it equal-spaced in when log-plotted).
For this example, I'll use a re-binning factor of 1.03. If new bin 1 has the width of one old bin, new bin 2 will be some 1.03 bins wider. New bin 3 will be 1.03 times wider than that (the width of...
Can you write Python code for it?

def rebin(freq, power, err_power, rebin_factor=1.05):
    """
    Re-bin the power spectrum in frequency space by some re-binning factor
    (rebin_factor > 1). This is sometimes called 'geometric re-binning' or
    'logarithmic re-binning', as opposed to linear re-binning
    (e.g., grouping by 2)

    Parameters
    ----------
    freq : np.array of floats
        1-D array of the Fourier frequencies.

    power : np.array of floats
        1-D array of the power at each Fourier frequency, with any/arbitrary
        normalization.

    err_power : np.array of floats
        1-D array of the error on the power at each Fourier frequency, with the
        same normalization as the power.

    rebin_factor : float
        The factor by which the data are geometrically re-binned.

    Returns
    -------
    rb_freq : np.array of floats
        1-D array of the re-binned Fourier frequencies.

    rb_power : np.array of floats
        1-D array of the power at the re-binned Fourier frequencies, with the
        same normalization as the input power array.

    rb_err : np.array of floats
        1-D array of the error on the power at the re-binned Fourier
        frequencies, with the same normalization as the input error on power.
    """
    assert rebin_factor >= 1.0

    rb_power = np.asarray([])  # Array of re-binned power
    rb_freq = np.asarray([])   # Array of re-binned frequencies
    rb_err = np.asarray([])    # Array of error in re-binned power
    real_index = 1.0           # The unrounded next index in power
    int_index = 1              # The int of real_index, added to current_m every iteration
    current_m = 1              # Current index in power
    prev_m = 0                 # Previous index m

    ## Loop through the length of the array power, new bin by new bin, to
    ## compute the average power and frequency of that new geometric bin.
    while current_m < len(power):

    return rb_freq, rb_power, rb_err


I am working on the file "Sessions/Session09/Day4/workbook_QPO.ipynb".
The first task is:
Using Fourier Analysis to Analyze Quasi-Periodic Oscillations
By Abigail Stevens
Problem 1: damped harmonic oscillator example
Generating a light curve
Can you write Python code for it?

a = Table()
a.meta['dt'] = 0.0001  # time step, in seconds
a.meta['duration'] = 200 # length of time, in seconds
a.meta['omega'] = 2*np.pi  # angular frequency, in radians
a.meta['phi'] = 0.0  # offset angle, in radians


I am working on the file "Sessions/Session09/Day4/workbook_globalsignals.ipynb".
The first task is:
2f. Re-do 2b through the plot above but slightly changing the power spectrum shape in each segment.
Maybe you change the centroid frequency of the QPO, or the normalizing factors between the two components, or the slope of the power-law.
Bonus problems:
1. Use a different definition of the Lorentzian (below) to make a power spectrum.
Follow the same procedure. Start off with just one segment. Use the rms as the normalizing factor.
2. Using what you learned about data visualization earlier this week, turn the plots in this notebook (and the QPO one, if you're ambitious) into clear and easy-to-digest, publication-ready plots.
Can you write Python code for it?

def lorentz_q(v, v_peak, q, rms):
    """
    Form of the Lorentzian function defined in terms of
    peak frequency v_peak and quality factor q
    q = v_peak / fwhm
    with the integrated rms of the QPO as the normalizing factor.
    e.g. see Pottschmidt et al. 2003, A&A, 407, 1039 for more info
    """
    f_res = v_peak / np.sqrt(1.0+(1.0/(4.0*q**2)))
    r = rms / np.sqrt(0.5-np.arctan(-2.0*q)/np.pi)
    lorentz = ((1/np.pi)*2*r**2*q*f_res) / (f_res**2+(4*q**2*(v-f_res)**2))
    return lorentz


I am working on the file "Sessions/Session09/Day4/workbook_globalsignals.ipynb".
The first task is:
Global Signals in Time Series Data
By Abigail Stevens
Problem 1: Timmer and Koenig algorithm
The algorithm outlined in Timmer & Koenig 1995 lets you define the shape of your power spectrum (a power law with some slope, a Lorentzian, a sum of a couple Lorentzians and a power law, etc.) then generate the random phases and amplitudes of the Fourier transform to simulate light curves defined by the power spectral shape. This is a great simulation tool to have in your back pocket (or, "maybe useful someday" github repo).
Define some basic parameters for the power spectrum and resultant light curve
Can you write Python code for it?

n_bins = 8192  ## number of total frequency bins in a FT segment; same as number of time bins in the light curve
dt = 1./16.  # time resolution of the output light curve
df = 1. / dt / n_bins


I am working on the file "Sessions/Session09/Day4/workbook_globalsignals_solutions.ipynb".
The first task is:
1a. Make an array of Fourier frequencies
Yes you can do this with scipy, but the order of frequencies in a T&K power spectrum is different than what you'd get by default from a standard FFT of a light curve.
You want the zero frequency to be in the middle (at index n_bins/2) of the frequency array. The positive frequencies should have two more indices than the negative frequencies, because of the zero frequency and nyquist frequency. You can either do this with np.arange or with special options in fftpack.fftfreq.
Can you write Python code for it?

freqs = np.arange(float(-n_bins/2)+1, float(n_bins/2)+1) * df
pos_freq = freqs[np.where(freqs >= 0)]
## Positive should have 2 more than negative,
## because of the 0 freq and the nyquist freq
neg_freq = freqs[np.where(freqs < 0)]
nyquist = pos_freq[-1]
len_pos = len(pos_freq)


I am working on the file "Sessions/Session09/Day4/workbook_globalsignals_solutions.ipynb".
The first task is:
1b. Define a Lorentzian function and power law function for the shape of the power spectrum
Can you write Python code for it?

def lorentzian(v, v_0, gamma):
    """ Gives a Lorentzian centered on v_0 with a FWHM of gamma """
    numerator = gamma / (np.pi * 2.0)
    denominator = (v - v_0) ** 2 + (1.0/2.0 * gamma) ** 2
    L = numerator / denominator
    return L

def powerlaw(v, beta):
    """Gives a powerlaw of (1/v)^-beta """
    pl = np.zeros(len(v))
    pl[1:] = v[1:] ** (-beta)
    pl[0] = np.inf
    return pl


I am working on the file "Sessions/Session09/Day4/workbook_globalsignals_solutions.ipynb".
The first task is:
1e. To obtain a real valued time series, choose the Fourier components for the negative frequencies according to $F(-\nu_i)=F*(\nu_i)$ where the asterisk denotes complex conjugation.
Append to make one fourier transform array. Check that your T&K fourier transform has length n_bins. Again, for this algorithm, the zero Fourier frequency is in the middle of the array, the negative Fourier frequencies are in the first half, and the positive Fourier frequencies are in the second half.
Can you write Python code for it?

FT_pos = r_values + i_values*1j
FT_neg = np.conj(FT_pos[1:-1])
FT_neg = FT_neg[::-1]  ## Need to flip direction of the negative frequency FT values so that they match up correctly
FT = np.append(FT_pos, FT_neg)


I am working on the file "Sessions/Session09/Day4/workbook_globalsignals_solutions.ipynb".
The first task is:
1f. Obtain the time series by backward Fourier transformation of $F(\nu)$ from the frequency domain to the time domain.
Note: I usually use .real after an iFFT to get rid of any lingering 1e-10 imaginary factors.
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day4/workbook_globalsignals_solutions.ipynb".
The first task is:
2. More realistic simulation with T&K
Now you're able to simulate the power spectrum of a single segment of a light curve. However, as you learned this morning, we usually use multiple (~50+) segments of a light curve, take the power spectrum of each segment, and average them together.
2a. Turn the code from 1d to 1e into a function make_TK_seg
Make it so that you can give a different random seed to each segment.
Can you write Python code for it?

def make_TK_seg(pos_freq, power_shape, seed_num=0):
    np.random.seed(seed_num)
    rand_r = np.random.standard_normal(len_pos)
    rand_i = np.random.standard_normal(len_pos-1)
    rand_i = np.append(rand_i, 0.0) # because the nyquist frequency should only have a real value

    ## Creating the real and imaginary values from the lists of random numbers and the frequencies
    r_values = rand_r * np.sqrt(0.5 * power_shape)
    i_values = rand_i * np.sqrt(0.5 * power_shape)
    r_values[np.where(pos_freq == 0)] = 0
    i_values[np.where(pos_freq == 0)] = 0

    ## Combining to make the Fourier transform
    FT_pos = r_values + i_values*1j
    FT_neg = np.conj(FT_pos[1:-1])
    FT_neg = FT_neg[::-1]  ## Need to flip direction of the negative frequency FT values so that they match up correctly
    FT = np.append(FT_pos, FT_neg)
    return FT


I am working on the file "Sessions/Session09/Day4/workbook_globalsignals_solutions.ipynb".
The first task is:
Congratulations!
1g. Plot the power spectrum of your FT (only the positive frequencies) next to the light curve it makes.
Remember: $$P(\nu_i)=|F(\nu_i)|^2$$
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day4/workbook_globalsignals_solutions.ipynb".
The first task is:
Now the T&K algorithm. I've transcribed the 'recipe' section of the T&K95 paper, which you will convert to lines of code.
1c. Choose a power spectrum $S(\nu)$.
We will use a sum of one Lorentzians (a QPO with a centroid frequency of 0.5 Hz and a FWHM of 0.01 Hz), and a Poisson-noise power law. The QPO should be 100 times larger amplitude than the power-law.
Can you write Python code for it?

## For a Lorentzian-shaped QPO
qpo_v0 = 0.5  ## Centroid frequency of QPO
qpo_fwhm = 0.01  ## FWHM of QPO
## Relative scale factors
qpo_scale = 100.
noise_scale = 1
## Putting it together
power_shape = qpo_scale * lorentzian(pos_freq, qpo_v0, qpo_fwhm) + \
              noise_scale * powerlaw(pos_freq, 0)


I am working on the file "Sessions/Session09/Day4/workbook_globalsignals_solutions.ipynb".
The first task is:
Plot
Can you write Python code for it?

fig, ax = plt.subplots(1,1, figsize=(8,5))
ax.plot(rb_freq, rb_pow, linewidth=2.0)
ax.set_xscale('log')
ax.set_yscale('log')
ax.set_xlabel(r'Frequency (Hz)', fontproperties=font_prop)
ax.tick_params(axis='x', labelsize=16, bottom=True, top=True,
               labelbottom=True, labeltop=False)
ax.tick_params(axis='y', labelsize=16, left=True, right=True,
               labelleft=True, labelright=False)
plt.show()


I am working on the file "Sessions/Session09/Day4/workbook_globalsignals_solutions.ipynb".
The first task is:
Problem 1: Timmer and Koenig algorithm
The algorithm outlined in Timmer & Koenig 1995 lets you define the shape of your power spectrum (a power law with some slope, a Lorentzian, a sum of a couple Lorentzians and a power law, etc.) then generate the random phases and amplitudes of the Fourier transform to simulate light curves defined by the power spectral shape. This is a great simulation tool to have in your back pocket (or, "maybe useful someday" github repo).
Define some basic parameters for the power spectrum and resultant light curve
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day4/workbook_globalsignals_solutions.ipynb".
The first task is:
You'll want to change the x scale of your light curve plot to be like 20 seconds in length, and only use the positive Fourier frequencies when plotting the power spectrum.
Can you write Python code for it?

fig, (ax1, ax2) = plt.subplots(1,2, figsize=(18,6))
ax1.plot(pos_freq, pos_power, lw=2)
ax1.set_xscale('log')
ax1.set_yscale('log')
# ax1.set_xlim(pos_freq[1],2)
ax1.set_ylim(1e-3, 3e4)
ax1.set_xlabel("Frequency (Hz)", fontproperties=font_prop)
ax1.set_ylabel("Amplitude", fontproperties=font_prop)
ax1.set_title("Power spectral density", fontproperties=font_prop)
ax1.tick_params(axis='both', which='major', labelsize=16,
               top=True, right=True, bottom=True, left=True)
ax2.plot(np.arange(0,n_bins*dt, dt), lc, lw=2)
# ax2.set_xlim(0,256)
ax2.set_xlim(50,80)
ax2.set_ylim(-0.9,0.9)
ax2.set_title("Light curve", fontproperties=font_prop)
ax2.set_xlabel("Time (s)", fontproperties=font_prop)
ax2.set_ylabel("Amplitude", fontproperties=font_prop)
ax2.tick_params(axis='both', which='major', labelsize=16,
               top=True, right=True, bottom=True, left=True)
plt.show()


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Building Histograms with Bayesian Priors
An Introduction to Bayesian Blocks
========
Version 0.1

By LM Walkowicz 2019 June 14
This notebook makes heavy use of Bayesian block implementations by Jeff Scargle, Jake VanderPlas, Jan Florjanczyk, and the Astropy team.
Before you begin, please download the dataset for this notebook.
Problem 1) Histograms Lie!
One of the most common and useful tools for data visualization can be incredibly misleading. Let's revisit how.

Problem 1a
First, let's make some histograms! Below, I provide some data; please make a histogram of it.
Can you write Python code for it?

# execute this cell
np.random.seed(0)
x = np.concatenate([stats.cauchy(-5, 1.8).rvs(500),
                    stats.cauchy(-4, 0.8).rvs(2000),
                    stats.cauchy(-1, 0.3).rvs(500),
                    stats.cauchy(2, 0.8).rvs(1000),
                    stats.cauchy(4, 1.5).rvs(500)])

# truncate values to a reasonable range
x = x[(x > -15) & (x < 15)]

# complete
# plt.hist(


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Cool, we have loaded in the FITS file. Let's look at what's in it:
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Hey, nice histogram!
But how do we know we have visualized all the relevant structure in our data?
Play around with the binning and consider:
What features do you see in this data? Which of these features are important?
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Problem 1b
What are some issues with histograms?
Take a few min to discuss this with your partner
Solution 1b
write your solution here
Problem 1c
We have previously covered a few ways to make histograms better. What are some ways you could improve your histogram?
Take a few min to discuss this with your partner
Problem 1d
There are lots of ways to improve the previous histogram-- let's implement a KDE representation instead! As you have seen in previous sessions, we will borrow a bit of code from Jake VanderPlas to estimate the KDE.
As a reminder, you have a number of choices of kernel in your KDE-- some we have used in the past: tophat, Epanechnikov, Gaussian. Please plot your original histogram, and then overplot a few example KDEs on top of it.
Can you write Python code for it?

# execute this cell
from sklearn.neighbors import KernelDensity

def kde_sklearn(data, grid, bandwidth = 1.0, **kwargs):
    kde_skl = KernelDensity(bandwidth = bandwidth, **kwargs)
    kde_skl.fit(data[:, np.newaxis])
    log_pdf = kde_skl.score_samples(grid[:, np.newaxis]) # sklearn returns log(density)

    return np.exp(log_pdf)

# complete
# plt.hist(
# grid =
# PDF =
# plt.plot(


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Problem 1d
Which parameters most affected the shape of the final distribution?
What are some possible issues with using a KDE representation of the data?
Discuss with your partner
Solution 1d
Write your response here
Problem 2) Histograms Episode IV: A New Hope
How can we create representations of our data that are robust against the known issues with histograms and KDEs?

Introducing: Bayesian Blocks
We want to represent our data in the most general possible way, a method that
* avoids assumptions about smoothness or shape of the signal (which might place limitations on scales and resolution)
* is nonparametric (doesn't fit some model)
* finds and characterizes local structure in our time series (in contrast to periodicities)
(continued)

handles arbitrary sampling (i.e. doesn't require evenly spaced samples, doesn't care about sparse samples)
is as hands-off as possible-- user interventions should be minimal or non-existent
is applicable to multivariate data
can both analyze data after they are collected, and in real time

Bayesian Blocks works by creating a super-simple representation of the data, essentially a piecewise fit that segments our time series.
In the implementations we will use today, the model is a piecewise linear fit in time across each individual bin, or "block".
one modeling the signal as linear in time across the block:
$$x(t) = λ(1 + a(t − t_{fid}))$$
where $\lambda$ is the signal strength at the fiducial time $t_{fid}$, and the coefficient $a$ determines the rate of change over the block.
Scargle et al. (2012) point out that using a linear fit is good because it makes calculating the fit really easy, but you could potentially use something more complicated (they provide some details for using an exponential model, $x(t) = λe^{a(t−t_{fid}})$, in their Appendix C.
The Fitness Function

The insight in Bayesian Blocks is that you can use a Bayesian likelihood framework to compute a "fitness function" that depends only on the number and size of the blocks.
In every block, you are...
Can you write Python code for it?

# execute this cell

def bayesian_blocks(t):
    """Bayesian Blocks Implementation

    By Jake Vanderplas.  License: BSD
    Based on algorithm outlined in http://adsabs.harvard.edu/abs/2012arXiv1207.5578S

    Parameters
    ----------
    t : ndarray, length N
        data to be histogrammed

    Returns
    -------
    bins : ndarray
        array containing the (N+1) bin edges

    Notes
    -----
    This is an incomplete implementation: it may fail for some
    datasets.  Alternate fitness functions and prior forms can
    be found in the paper listed above.
    """
    # copy and sort the array
    t = np.sort(t)
    N = t.size

    # create length-(N + 1) array of cell edges
    edges = np.concatenate([t[:1],
                            0.5 * (t[1:] + t[:-1]),
                            t[-1:]])
    block_length = t[-1] - edges

    # arrays needed for the iteration
    nn_vec = np.ones(N)
    best = np.zeros(N, dtype=float)
    last = np.zeros(N, dtype=int)

    #-----------------------------------------------------------------
    # Start with first data cell; add one cell at each iteration
    #-----------------------------------------------------------------
    for K in range(N):
        # Compute the width and count of the final bin for all possible
        # locations of the K^th changepoint
        width = block_length[:K + 1] - block_length[K + 1]
        count_vec = np.cumsum(nn_vec[:K + 1][::-1])[::-1]

        # evaluate fitness function for these possibilities
        fit_vec = count_vec * (np.log(count_vec) - np.log(width))
        fit_vec -= 4  # 4 comes from the prior on the number of changepoints
        fit_vec[1:] += best[:K]

        # find the max of the fitness: this is the K^th changepoint
        i_max = np.argmax(fit_vec)
        last[K] = i_max
        best[K] = fit_vec[i_max]

    #-----------------------------------------------------------------
    # Recover changepoints by iteratively peeling off the last block
   ...


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Problem 2a
Let's visualize our data again, but this time we will use Bayesian Blocks.
Plot a standard histogram (as above), but now plot the Bayesian Blocks representation of the distribution over it.
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Problem 2b
How is the Bayesian Blocks representation different or similar?
How might your choice of representation affect your scientific conclusions about your data?
Take a few min to discuss this with your partner
If you are using histograms for analysis, you might infer physical meaning from the presence or absence of features in these distributions. As it happens, histograms of time-tagged event data are often used to characterize physical events in time domain astronomy, for example gamma ray bursts or stellar flares.
Problem 3) Bayesian Blocks in the wild
Now we'll apply Bayesian Blocks to some real astronomical data, and explore how our visualization choices may affect our scientific conclusions.
First, let's get some data!
All data from NASA missions is hosted on the Mikulski Archive for Space Telescopes (aka MAST). As an aside, the M in MAST used to stand for "Multimission", but was changed to honor Sen. Barbara Mikulski (D-MD) for her tireless support of science.
Some MAST data (mostly the original data products) can be directly accessed using astroquery (there's an extensive guide to interacting with MAST via astroquery here: https://astroquery.readthedocs.io/en/latest/mast/mast.html).
In addition, MAST also hosts what are called "Higher Level Science Products", or HLSPs, which are data derived by science teams in the course of doing their analyses. You can see a full list of HLSPs here: https://archive.stsci.edu/hlsp/hlsp-table
These data tend to be more heterogeneous, and so are not currently accessible through astroquery (for the most part). They will be added in the future. But never fear! You can also submit SQL queries via MAST's CasJobs interface.
Go to the MAST CasJobs http://mastweb.stsci.edu/mcasjobs/home.aspx
If I have properly remembered to tell you to create a MAST CasJobs login, you can login now (and if not, just go ahead and sign up now, it's fast)!
We will be working with the table of new planet radii by Berger et al. (2019).
If you like, you can check out the paper here!...
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Problem 3c
Let's look at the distribution of the small planet radii in this table, which are given in units of Earth radii. Select the planets whose radii are Neptune-sized or smaller.
Select the planet radii for all planets smaller than Neptune in the table, and visualize the distribution of planet radii using a standard histogram.
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Problem 3d
What features do you see in the histogram of planet radii? Which of these features are important?
Discuss with your partner
Solution 3d
Write your answer here
Problem 3e
Now let's try visualizing these data using Bayesian Blocks. Please recreate the histogram you plotted above, and then plot the Bayesian Blocks version over it.
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Problem 4c
Let's take a moment to reflect on the differences between these two representations of our data.
Please discuss with your partner:
* How many bursts are present in these two representations?
* How accurately would you be able to identify the time of the burst(s) from these representations? What about other quantities?
For the groups who worked with Channel 3:
You may have noticed very sharp features in your blocks representation of the data. Are they real?
To quote Jake VdP:

Simply put, there are spikes because the piecewise constant likelihood model says that spikes are favored. By saying that the spikes seem unphysical, you are effectively adding a prior on the model based on your intuition of what it should look like.

To quote Jeff Scargle:

Trust the algorithm!

Problem 5: Finding flares
As we have just seen, Bayesian Blocks can be very useful for finding transient events-- it worked great on our photon counts from BATSE! Let's try it on some slightly more complicated data: lightcurves from NASA's Kepler mission. Kepler's data consists primarily of point measures (rather than events)-- a Kepler lightcurve is just the change in the brightness of the star over time (with associated uncertainties).

Problem 5a
People often speak of transients (like the GRB we worked with above) and variables (like RR Lyrae or Cepheid stars) as being two completely different categories of changeable astronomical objects. However, some objects exhibit both variability and transient events. Magnetically active stars are one example of these objects: many of them have starspots that rotate into and out of view, creating periodic (or semi-periodic) variability, but they also have flares, magnetic reconnection events that create sudden, rapid changes in the stellar brightness.
A challenge in identifying flares is that they often appear against a background that is itself variable. While their are many approaches to fitting both quiescent and flare variability (Gaussian processes, which you saw earlier this week,...
Can you write Python code for it?

# execute this cell
kplr_hdul = fits.open('./data/kplr009726699-2011271113734_llc.fits')


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Problem 5b
Use a scatter plot to visualize the Kepler lightcurve. I strongly suggest you try displaying it at different scales by zooming in (or playing with the axis limits), so that you can get a better sense of the shape of the lightcurve.
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Problem 5c
These data consist of a variable background, with occasional bright points caused by stellar flares.
Brainstorm possible approaches to find the flare events in this data. Write down your ideas, and discuss their potential advantages, disadvantages, and any pitfalls that you think might arise.
Discuss with your partner
Solution 5c
Write your notes here
There are lots of possible ways to approach this problem. In the literature, a very common traditional approach has been to fit the background variability while ignoring the outliers, then to subtract the background fit, and flag any point beyond some threshold value as belonging to a flare. More sophisticated approaches also exist, but they are often quite time consuming (and in many cases, detailed fits require a good starting estimate for the locations of flare events).
Recall, however, that Bayesian Blocks is particularly effective at identifying change points in our data. Let's see if it can help us in this case!
Problem 5d
Use Bayesian Blocks to visualize the Kepler lightcurve. Note that you are now using data that consists of point measurements, rather than event data (as for the BATSE example).
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
Putting these results in context
Both standard histograms and KDEs can be useful for quickly visualizing data, and in some cases, getting an intuition for the underlying PDF of your data.
However, keep in mind that they both involve making parameter choices that are largely not motivated in any quantitative way. These choices can create wildly misleading representations of your data.
In particular, your choices may lead you to make a physical interpretation that may or may not be correct (in our example, bear in mind that the observed distribution of exoplanetary radii informs models of planet formation).
Bayesian Blocks is more than just a variable-width histogram
While KDEs often do a better job of visualizing data than standard histograms do, they also create a loss of information. Philosophically speaking, what Bayesian Blocks do is posit that the "change points", also known as the bin edges, contain information that is interesting. When you apply a KDE, you are smoothing your data by creating an approximation, and that can mean you are losing potential insights by removing information.
While Bayesian Blocks are useful as a replacement for histograms in general, their ability to identify change points makes them especially useful for time series analysis.
Problem 4) Bayesian Blocks for Time Series Analysis
While Bayesian Blocks can be very useful as a simple replacement for histograms, one of its great strengths is in finding "change points" in time series data. Finding these change points can be useful for discovering interesting events in time series data you already have, and it can be used in real-time to detect changes that might trigger another action (for example, follow up observations for LSST).

Let's take a look at a few examples of using Bayesian Blocks in the time series context.
First and foremost, it's important to understand the different between various kinds of time series data.
Event data come from photon counting instruments. In these data, the time series typically consists of...
Can you write Python code for it?


I am working on the file "Sessions/Session09/Day5/BayesianBlocksDSFP_Problems.ipynb".
The first task is:
We want the light curve, so let's check out what's in that part of the file!
Can you write Python code for it?

# execute this cell
lcdata = kplr_hdul[1].data
lcdata.columns

# execute this cell

t = lcdata['TIME']
f = lcdata['PDCSAP_FLUX']
e = lcdata['PDCSAP_FLUX_ERR']
t = t[~np.isnan(f)]
e = e[~np.isnan(f)]
f = f[~np.isnan(f)]
nf = f / np.median(f)
ne = e / np.median(f)


I am working on the file "Sessions/Session10/Day0/IntroToMachLearnSolutions.ipynb".
The first task is:
Finally, as a baseline for the exercises that follow, we will now make a simple 2D plot showing the separation of the 3 classes in the iris dataset. This plot will serve as the reference for examining the quality of the clustering algorithms.
Problem 1e Make a scatter plot showing sepal length vs. sepal width for the iris data set. Color the points according to their respective classes.
Can you write Python code for it?

print(iris.feature_names)  # shows that sepal length is first feature and sepal width is second feature

fig, ax = plt.subplots()
ax.scatter(iris.data[:,0], iris.data[:,1], c = iris.target, s = 30, edgecolor = "None", cmap = "viridis")
ax.set_xlabel('sepal length', fontsize=14)
ax.set_ylabel('sepal width', fontsize=14)
fig.tight_layout()


I am working on the file "Sessions/Session10/Day0/IntroToMachLearnSolutions.ipynb".
The first task is:
From this representation, we see right away that most of the virginica that are being misclassifed are being scattered into the versicolor class. However, this representation could still be improved: it'd be helpful to normalize each value relative to the total number of sources in each class, and better still, it'd be good to have a visual representation of the confusion matrix. This visual representation will be readily digestible. Now let's normalize the confusion matrix.
Problem 2e Calculate the normalized confusion matrix. Be careful, you have to sum along one axis, and then divide along the other.
Anti-hint: This operation is actually straightforward using some array manipulation that we have not covered up to this point. Thus, we have performed the necessary operations for you below. If you have extra time, you should try to develop an alternate way to arrive at the same normalization.
Can you write Python code for it?

normalized_cm = cm.astype('float')/cm.sum(axis = 1)[:,np.newaxis]

normalized_cm


I am working on the file "Sessions/Session10/Day0/IntroToMachLearnSolutions.ipynb".
The first task is:
Most importantly, iris contains data and target values. These are all you need for scikit-learn, though the feature and target names and description are useful.
Problem 1c What is the shape and content of the iris data?
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/IntroToMachLearnSolutions.ipynb".
The first task is:
Problem 1) Introduction to scikit-learn
At the most basic level, scikit-learn makes machine learning extremely easy within Python. By way of example, here is a short piece of code that builds a complex, non-linear model to classify sources in the Iris data set that we learned about yesterday:
from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
iris = datasets.load_iris()
RFclf = RandomForestClassifier().fit(iris.data, iris.target)

Those 4 lines of code have constructed a model that is superior to any system of hard cuts that we could have encoded while looking at the multidimensional space. This can be fast as well: execute the dummy code in the cell below to see how "easy" machine-learning is with scikit-learn.
Can you write Python code for it?

# execute dummy code here

from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
iris = datasets.load_iris()
RFclf = RandomForestClassifier().fit(iris.data, iris.target)


I am working on the file "Sessions/Session10/Day0/IntroToMachLearnSolutions.ipynb".
The first task is:
The normalization makes it easier to compare the classes, since each class has a different number of sources. Now we can procede with a visual representation of the confusion matrix. This is best done using imshow() within pyplot. You will also need to plot a colorbar, and labeling the axes will also be helpful.
Problem 2f Plot the confusion matrix. Be sure to label each of the axeses.
Hint - you might find the sklearn confusion matrix tutorial helpful for making a nice plot.
Can you write Python code for it?

plt.imshow(normalized_cm, interpolation = 'nearest', cmap = 'bone_r')# complete

tick_marks = np.arange(len(iris.target_names))
plt.xticks(tick_marks, iris.target_names, rotation=45)
plt.yticks(tick_marks, iris.target_names)

plt.ylabel( 'True')# complete
plt.xlabel( 'Predicted' )# complete
plt.colorbar()
plt.tight_layout()


I am working on the file "Sessions/Session10/Day0/IntroToMachineLearning.ipynb".
The first task is:
Finally, as a baseline for the exercises that follow, we will now make a simple 2D plot showing the separation of the 3 classes in the iris dataset. This plot will serve as the reference for examining the quality of the clustering algorithms.
Problem 1e Make a scatter plot showing sepal length vs. sepal width for the iris data set. Color the points according to their respective classes.
Hint - determine which columns of data correspond to sepal length and sepal width.
Can you write Python code for it?

plt.scatter( # complete
plt.xlabel('sepal length')
plt.ylabel('sepal width')


I am working on the file "Sessions/Session10/Day0/IntroToMachineLearning.ipynb".
The first task is:
From this representation, we see right away that most of the virginica that are being misclassifed are being scattered into the versicolor class. However, this representation could still be improved: it'd be helpful to normalize each value relative to the total number of sources in each class, and better still, it'd be good to have a visual representation of the confusion matrix. This visual representation will be readily digestible. Now let's normalize the confusion matrix.
Problem 2e Calculate the normalized confusion matrix. Be careful, you have to sum along one axis, and then divide along the other.
Anti-hint: This operation is actually straightforward using some array manipulation that we have not covered up to this point. Thus, we have performed the necessary operations for you below. If you have extra time, you should try to develop an alternate way to arrive at the same normalization.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/IntroToMachineLearning.ipynb".
The first task is:
Generally speaking, the procedure for scikit-learn is uniform across all machine-learning algorithms. Models are accessed via the various modules (ensemble, SVM, neighbors, etc), with user-defined tuning parameters. The features (or data) for the models are stored in a 2D array, X, with rows representing individual sources and columns representing the corresponding feature values. [In a minority of cases, X, represents a similarity or distance matrix where each entry represents the distance to every other source in the data set.] In cases where there is a known classification or scalar value (typically supervised methods), this information is stored in a 1D array y.
Unsupervised models are fit by calling .fit(X) and supervised models are fit by calling .fit(X, y). In both cases, predictions for new observations, Xnew, can be obtained by calling .predict(Xnew). Those are the basics and beyond that, the details are algorithm specific, but the documentation for essentially everything within scikit-learn is excellent, so read the docs.
To further develop our intuition, we will now explore the Iris dataset a little further.
Problem 1a What is the pythonic type of iris?
You likely haven't encountered a scikit-learn Bunch before. It's functionality is essentially the same as a dictionary.
Problem 1b What are the keys of iris?
Most importantly, iris contains data and target values. These are all you need for scikit-learn, though the feature and target names and description are useful.
Problem 1c What is the shape and content of the iris data?
Can you write Python code for it?

print(np.shape( # complete
print( # complete


I am working on the file "Sessions/Session10/Day0/IntroToMachineLearning.ipynb".
The first task is:
Problem 1) Introduction to scikit-learn
At the most basic level, scikit-learn makes machine learning extremely easy within Python. By way of example, here is a short piece of code that builds a complex, non-linear model to classify sources in the Iris data set that we learned about yesterday:
from sklearn import datasets
from sklearn.ensemble import RandomForestClassifier
iris = datasets.load_iris()
RFclf = RandomForestClassifier().fit(iris.data, iris.target)

Those 4 lines of code have constructed a model that is superior to any system of hard cuts that we could have encoded while looking at the multidimensional space. This can be fast as well: execute the dummy code in the cell below to see how "easy" machine-learning is with scikit-learn.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/IntroToMachineLearning.ipynb".
The first task is:
Problem 1d What is the shape and content of the iris target?
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/IntroToMachineLearning.ipynb".
The first task is:
Problem 2) Supervised Machine Learning
Supervised machine learning, on the other hand, aims to predict a target class or produce a regression result based on the location of labelled sources (i.e. the training set) in the multidimensional feature space. The "supervised" comes from the fact that we are specifying the allowed outputs from the model. As there are labels available for the training set, it is possible to estimate the accuracy of the model (though there are generally important caveats about generalization, which we will explore in further detail later).
The details and machinations of supervised learning will be explored further during the following break-out session. Here, we will simply introduce some of the basics as a point of comparison to unsupervised machine learning.
We will begin with a simple, but nevertheless, elegant algorithm for classification and regression: $k$-nearest-neighbors ($k$NN). In brief, the classification or regression output is determined by examining the $k$ nearest neighbors in the training set, where $k$ is a user defined number. Typically, though not always, distances between sources are Euclidean, and the final classification is assigned to whichever class has a plurality within the $k$ nearest neighbors (in the case of regression, the average of the $k$ neighbors is the output from the model). We will experiment with the steps necessary to optimize $k$, and other tuning parameters, in the detailed break-out problem.
In scikit-learn the KNeighborsClassifer algorithm is implemented as part of the sklearn.neighbors module.
Problem 2a Fit two different $k$NN models to the iris data, one with 3 neighbors and one with 10 neighbors. Plot the resulting class predictions in the sepal length-sepal width plane (same plot as above). How do the results compare to the true classifications? Is there any reason to be suspect of this procedure?
Hint - after you have constructed the model, it is possible to obtain model predictions using the .predict() method, which requires a...
Can you write Python code for it?

from sklearn.neighbors import KNeighborsClassifier

KNNclf = KNeighborsClassifier( # complete
preds = # complete
plt.figure()
plt.scatter( # complete

KNNclf = KNeighborsClassifier(# complete
preds = # complete
plt.figure()
plt.scatter( # complete


I am working on the file "Sessions/Session10/Day0/IntroToMachineLearning.ipynb".
The first task is:
The normalization makes it easier to compare the classes, since each class has a different number of sources. Now we can procede with a visual representation of the confusion matrix. This is best done using imshow() within pyplot. You will also need to plot a colorbar, and labeling the axes will also be helpful.
Problem 2f Plot the confusion matrix. Be sure to label each of the axeses.
Hint - you might find the sklearn confusion matrix tutorial helpful for making a nice plot.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/IntroToMachineLearning.ipynb".
The first task is:
These results are almost identical to the training classifications. However, we have cheated! In this case we are evaluating the accuracy of the model (98% in this case) using the same data that defines the model. Thus, what we have really evaluated here is the training error. The relevant parameter, however, is the generalization error: how accurate are the model predictions on new data?
Without going into too much detail, we will test this using cross validation (CV), which will be explored in more detail later. In brief, CV provides predictions on the training set using a subset of the data to generate a model that predicts the class of the remaining sources. Using cross_val_predict, we can get a better sense of the model accuracy. Predictions from cross_val_predict are produced in the following manner:
from sklearn.cross_validation import cross_val_predict
CVpreds = cross_val_predict(sklearn.model(), X, y)

where sklearn.model() is the desired model, X is the feature array, and y is the label array.
Problem 2b Produce cross-validation predictions for the iris dataset and a $k$NN with 5 neighbors. Plot the resulting classifications, as above, and estimate the accuracy of the model as applied to new data. How does this accuracy compare to a $k$NN with 50 neighbors?
Can you write Python code for it?

from sklearn.cross_validation import cross_val_predict

CVpreds = cross_val_predict( # complete
plt.figure()
plt.scatter( # complete
print("The accuracy of the kNN = 5 model is ~{:.4}".format(  # complete

CVpreds50 = cross_val_predict( # complete
print("The accuracy of the kNN = 50 model is ~{:.4}".format(  # complete


I am working on the file "Sessions/Session10/Day0/IntroToMachineLearning.ipynb".
The first task is:
We just found that the classifier does a much better job classifying setosa and versicolor than it does for virginica. The main reason for this is some viginica flowers lie far outside the main virginica locus, and within predominantly versicolor "neighborhoods". In addition to knowing the accuracy for the individual classes, it is also useful to know class predictions for the misclassified sources, or in other words where there is "confusion" for the classifier. The best way to summarize this information is with a confusion matrix. In a confusion matrix, one axis shows the true class and the other shows the predicted class. For a perfect classifier all of the power will be along the diagonal, while confusion is represented by off-diagonal signal.
Like almost everything else we have encountered during this exercise, scikit-learn makes it easy to compute a confusion matrix. This can be accomplished with the following:
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_test, y_prep)

Problem 2d Calculate the confusion matrix for the iris training set and the $k$NN = 50 model.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix
cm = confusion_matrix( # complete
print(cm)


I am working on the file "Sessions/Session10/Day0/IntroToMachineLearning.ipynb".
The first task is:
While it is useful to understand the overall accuracy of the model, it is even more useful to understand the nature of the misclassifications that occur.
Problem 2c Calculate the accuracy for each class in the iris set, as determined via CV for the $k$NN = 50 model.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
Already with this simple plot we see a problem - the choice of bin centers and number of bins suggest that there is a 0% probability that middle aged men can do 10 chinups. Intuitively this seems incorrect, so lets examine how the histogram changes if we change the number of bins or the bin centers.
Problem 1b
Using the same data make 2 new histograms: (i) one with 5 bins (bins = 5), and (ii) one with the bars centered on the left bin edges (align = "left").
Hint - if overplotting the results, you may find it helpful to use the histtype = "step" option
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
But! Histograms and scatter plots can be problematic as we have discussed many times before.
Problem 3d
Re-create the plot above but set kind='kde' to produce density estimates of the distributions.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
Ending the lie
Earlier I stated that histograms lie. One simple way to combat this lie: show all the data. Displaying the original data points allows viewers to somewhat intuit the effects of the particular bin choices that have been made (though this can also be cumbersome for very large data sets, which these days is essentially all data sets).  The standard for showing individual observations relative to a histogram is a "rug plot," which shows a vertical tick (or other symbol) at the location of each source used to estimate the PDF.
Problem 1d Execute the cell below to see an example of a rug plot.
Can you write Python code for it?

plt.hist(chinups, histtype = 'step')

# this is the code for the rug plot
plt.plot(chinups, np.zeros_like(chinups), '|', color='k', ms = 25, mew = 4)


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
Even better - there is an option to create a PairGrid which allows fine tuned control of the data as displayed above, below, and along the diagonal. In this way it becomes possible to avoid having symmetric redundancy, which is not all that informative. In the example below, we will show scatter plots and contour plots simultaneously.
Can you write Python code for it?

g = sns.PairGrid(iris, vars = ["sepal_length", "sepal_width", "petal_length", "petal_width"],
                 hue = "species", diag_sharey=False)
g.map_lower(sns.kdeplot)
g.map_upper(plt.scatter, edgecolor='white')
g.map_diag(sns.kdeplot, lw=3)


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
For data sets where we have classification labels, we can even color the various points using the hue option, and produce KDEs along the diagonal with diag_type = 'kde'.
Can you write Python code for it?

sns.pairplot(iris, vars = ["sepal_length", "sepal_width", "petal_length", "petal_width"],
             hue = "species", diag_kind = 'kde')


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
In this representation, each "block" has a height of 0.25. The bandwidth is too narrow to provide any overlap between the blocks. This choice of kernel and bandwidth produces an estimate that is essentially a histogram with a large number of bins. It gives no sense of continuity for the distribution. Now, we examine the difference (relative to histograms) upon changing the the width (i.e. kernel) of the blocks.
Problem 1f
Plot the KDE of the PDF for the number of chinups middle aged men can do using bandwidths of 1 and 5 and a tophat kernel. How do the results differ from the histogram plots above?
Can you write Python code for it?

PDFtophat1 = # complete

# complete
# complete
# complete


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
Introduction to Visualization:
Density Estimation and Data Exploration
Version 0.1
There are many flavors of data analysis that fall under the "visualization" umbrella in astronomy. Today, by way of example, we will focus on 2 basic problems.

By AA Miller
16 September 2017
Problem 1) Density Estimation
Starting with 2MASS and SDSS and extending through LSST, we are firmly in an era where data and large statistical samples are cheap. With this explosion in data volume comes a problem: we do not know the underlying probability density function (PDF) of the random variables measured via our observations. Hence - density estimation: an attempt to recover the unknown PDF from observations. In some cases theory can guide us to a parametric form for the PDF, but more often than not such guidance is not available.
There is a common, simple, and very familiar tool for density estimation: histograms.
But there is also a problem:
HISTOGRAMS LIE!
We will "prove" this to be the case in a series of examples. For this exercise, we will load the famous Linnerud data set, which tested 20 middle aged men by measuring the number of chinups, situps, and jumps they could do in order to compare these numbers to their weight, pulse, and waist size. To load the data (just chinups for now) we will run the following:
from sklearn.datasets import load_linnerud
linnerud = load_linnerud()
chinups = linnerud.data[:,0]
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
Now that we have a sense of the data structure, it is useful to examine the distribution of features. Above, we went to great pains to produce histograms, KDEs, and rug plots. seaborn handles all of that effortlessly with the distplot function.
Problem 3b
Plot the distribution of petal lengths for the Iris data set.
Can you write Python code for it?

# note - hist, kde, and rug all set to True, set to False to turn them off
with sns.axes_style("dark"):
    sns.distplot(iris['petal_length'], bins=20, hist=True, kde=True, rug=True)


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
Now that we have covered the basics of seaborn (and the above examples truly only scratch the surface of what is possible), we will explore the power of seaborn for higher dimension data sets. We will load the famous Iris data set, which measures 4 different features of 3 different types of Iris flowers. There are 150 different flowers in the data set.
Note - for those familiar with pandas seaborn is designed to integrate easily and directly with pandas DataFrame objects. In the example below the Iris data are loaded into a DataFrame. iPython notebooks also display the DataFrame data in a nice readable format.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
Of course, even rug plots are not a perfect solution. Many of the chinup measurements are repeated, and those instances cannot be easily isolated above. One (slightly) better solution is to vary the transparency of the rug "whiskers" using alpha = 0.3 in the whiskers plot call. But this too is far from perfect.
To recap, histograms are not ideal for density estimation for the following reasons:

They introduce discontinuities that are not present in the data
They are strongly sensitive to user choices ($N_\mathrm{bins}$, bin centering, bin grouping), without any mathematical guidance to what these choices should be
They are difficult to visualize in higher dimensions

Histograms are useful for generating a quick representation of univariate data, but for the reasons listed above they should never be used for analysis. Most especially, functions should not be fit to histograms given how greatly the number of bins and bin centering affects the output histogram.
Okay - so if we are going to rail on histograms this much, there must be a better option. There is: Kernel Density Estimation (KDE), a nonparametric form of density estimation whereby a normalized kernel function is convolved with the discrete data to obtain a continuous estimate of the underlying PDF. As a rule, the kernel must integrate to 1 over the interval $-\infty$ to $\infty$ and be symmetric. There are many possible kernels (gaussian is highly popular, though Epanechnikov, an inverted parabola, produces the minimal mean square error).
KDE is not completely free of the problems we illustrated for histograms above (in particular, both a kernel and the width of the kernel need to be selected), but it does manage to correct a number of the ills. We will now demonstrate this via a few examples using the scikit-learn implementation of KDE: KernelDensity, which is part of the sklearn.neighbors module.
Note There are many implementations of KDE in Python, and Jake VanderPlas has put together an excellent description of the strengths and weaknesses of...
Can you write Python code for it?

# execute this cell
from sklearn.neighbors import KernelDensity
def kde_sklearn(data, grid, bandwidth = 1.0, **kwargs):
    kde_skl = KernelDensity(bandwidth = bandwidth, **kwargs)
    kde_skl.fit(data[:, np.newaxis])
    log_pdf = kde_skl.score_samples(grid[:, np.newaxis]) # sklearn returns log(density)

    return np.exp(log_pdf)


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
Of course, this data set lives in a 4D space, so plotting more than univariate distributions is important (and as we will see tomorrow this is particularly useful for visualizing classification results). Fortunately, seaborn makes it very easy to produce handy summary plots.
At this point, we are familiar with basic scatter plots in matplotlib.
Problem 3c
Make a matplotlib scatter plot showing the Iris petal length against the Iris petal width.
Can you write Python code for it?


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
Of course, when there are many many data points, scatter plots become difficult to interpret. As in the example below:
Can you write Python code for it?

with sns.axes_style("darkgrid"):
    xexample = np.random.normal(loc = 0.2, scale = 1.1, size = 10000)
    yexample = np.random.normal(loc = -0.1, scale = 0.9, size = 10000)

    plt.scatter(xexample, yexample)


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
Seaborn
Seaborn is a plotting package that enables many useful features for exploration. In fact, a lot of the functionality that we developed above can readily be handled with seaborn.
To begin, we will make the same plot that we created in matplotlib.
Can you write Python code for it?

import seaborn as sns

fig = plt.figure()
ax = fig.add_subplot(111)

ax.plot(x,y, lw = 2)
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_xlim(0, 6*np.pi)


I am working on the file "Sessions/Session10/Day0/TooBriefVisualization.ipynb".
The first task is:
So, what is the optimal choice of bandwidth and kernel? Unfortunately, there is no hard and fast rule, as every problem will likely have a different optimization. Typically, the choice of bandwidth is far more important than the choice of kernel. In the case where the PDF is likely to be gaussian (or close to gaussian), then Silverman's rule of thumb can be used:
$$h = 1.059 \sigma n^{-1/5}$$
where $h$ is the bandwidth, $\sigma$ is the standard deviation of the samples, and $n$ is the total number of samples. Note - in situations with bimodal or more complicated distributions, this rule of thumb can lead to woefully inaccurate PDF estimates. The most general way to estimate the choice of bandwidth is via cross validation (we will cover cross-validation later today).
What about multidimensional PDFs? It is possible using many of the Python implementations of KDE to estimate multidimensional PDFs, though it is very very important to beware the curse of dimensionality in these circumstances.
Problem 2) Data Exploration
Now a more open ended topic: data exploration. In brief, data exploration encompases a large suite of tools (including those discussed above) to examine data that live in large dimensional spaces. There is no single best method or optimal direction for data exploration. Instead, today we will introduce some of the tools available via python.
As an example we will start with a basic line plot - and examine tools beyond matplotlib.
Can you write Python code for it?

x = np.arange(0, 6*np.pi, 0.1)
y = np.cos(x)

plt.plot(x,y, lw = 2)
plt.xlabel('X')
plt.ylabel('Y')
plt.xlim(0, 6*np.pi)