I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Given the lecture order this is a little late, but we have now identified the fundamental challenge in identifying periodic signals in astrophysical observations:
periodic models are highly non-linear!
This can easily be seen in the LS periodograms from the previous lecture: period estimates essentially need to be perfect to properly identify the signal. Take for instance the previous example, where we adjusted the initial guess for the period by less than 1% and it made the difference between correct estimates catastrophic errors.
This also means that classic optimization procedures (e.g., gradient decent) are helpless for this problem. If you guess the wrong period there is no obvious way to know whether the subsequent guess should use a larger or smaller period.
Problem 3) Sampling Techniques
Given our lack of success with maximum likelihood techniques, we will now attempt a Bayesian approach. As a brief reminder, Bayes theorem tells us that:
$$P(\theta|X) \propto P(X|\theta) P(\theta).$$
In words, the posterior probability is proportional to the likelihood multiplied by the prior. We will use sampling techniques, MCMC, to estimate the posterior.
Remember - we already calculated the likelihood above.
Problem 3a
Write a function lnprior1 to calculate the log of the prior on $\theta$. Use a reasonable, wide and flat prior for all the model parameters.
Hint - for emcee the log prior should return 0 within the prior and $-\infty$ otherwise.
Can you write Python code for it?

def lnprior1( # complete
    a, b, c = # complete
    if # complete
        return 0.0
    return -np.inf


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
If you run the cell below, you will see random samples from the posterior overplotted on the data. Do the posterior samples seem reasonable in this case?
Can you write Python code for it?

fig, ax = plt.subplots()
ax.errorbar(x, y, dy, fmt='o')
ax.set_xlabel('x')
ax.set_ylabel('y')

for s in samples[np.random.randint(len(samples), size=5)]:
    # Set up the GP for this sample.
    lnper, lna = s[:2]
    gp = george.GP(np.exp(lna) * kernels.CosineKernel(lnper))
    gp.compute(x, dy)
    # Compute the prediction conditioned on the observations and plot it.
    m = gp.sample_conditional(y - model2(s, x), x_grid) + model2(s, x_grid)

    ax.plot(x_grid, m, color="0.2", alpha=0.3)
fig.tight_layout()


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 1c
Later, we will be using MCMC. Execute the following cell which will plot the chains from emcee to follow the MCMC walkers.
Can you write Python code for it?

def plot_chains(sampler, nburn, paramsNames):
    Nparams = len(paramsNames)  # + 1
    fig, ax = plt.subplots(Nparams,1, figsize = (8,2*Nparams), sharex = True)
    fig.subplots_adjust(hspace = 0)
    ax[0].set_title('Chains')
    xplot = range(len(sampler.chain[0,:,0]))

    for i,p in enumerate(paramsNames):
        for w in range(sampler.chain.shape[0]):
            ax[i].plot(xplot[:nburn], sampler.chain[w,:nburn,i], color="0.5", alpha = 0.4, lw = 0.7, zorder = 1)
            ax[i].plot(xplot[nburn:], sampler.chain[w,nburn:,i], color="k", alpha = 0.4, lw = 0.7, zorder = 1)

            ax[i].set_ylabel(p)
    fig.tight_layout()
    return ax


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 2) Maximum-Likelihood Optimization
A common approach$^\dagger$ in the literature for problems where there is good reason to place a strong prior on the signal (i.e. to only try and fit a single model) is maximum likelihood optimization [this is sometimes also called $\chi^2$ minimization].
$^\dagger$The fact that this approach is commonly used, does not mean it should be commonly used.
In this case, where we are fitting for a known signal in simulated data, we are justified in assuming an extremely strong prior and fitting a sinusoidal model to the data.
Problem 2a
Write a function, correct_model, that returns the expected signal for our data given input time $t$:
$$f(t) = a + b\cos\left(\frac{2\pi t}{c}\right)$$
where $a, b, c$ are model parameters.
Hint - store the model parameters in a single variable (this will make things easier later).
Can you write Python code for it?

def correct_model( # complete
    # complete
    return # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 2d
Plot the input model, the noisy data, and the maximum likelihood model.
How does the model fit look?
Can you write Python code for it?

fig, ax = plt.subplots()
ax.errorbar( # complete
ax.plot( # complete
ax.plot( # complete
ax.set_xlabel('x')
ax.set_ylabel('y')
fig.tight_layout()


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 2e
Repeat the maximum likelihood optimization, but this time use an initial guess of 10 for the offset, 1 for the amplitude of variations, and 0.393 for the period.
Can you write Python code for it?

initial_theta =  # complete
res = minimize( # complete

print("The ML estimate for a, b, c is: {:.5f}, {:.5f}, {:.5f}".format( # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 3b
Write a function lnprob1 to calculate the log of the posterior probability. This function should take $\theta$ and x, y, dy as inputs.
Can you write Python code for it?

def lnprob1( # complete
    lp = lnprior1(theta)
    if np.isfinite(lp):
        return # complete
    return -np.inf


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 3c
Initialize the walkers for emcee, which we will use to draw samples from the posterior. Like before, we need to include an initial guess (the parameters of which don't matter much beyond the period). Start with a guess of 0.6 for the period.
As a quick reminder, emcee is a pure python implementation of Goodman & Weare's affine Invariant Markov Chain Monte Carlo (MCMC) Ensemble sampler. emcee seeds several "walkers" which are members of the ensemble. You can think of each walker as its own Metropolis-Hastings chain, but the key detail is that the chains are not independent. Thus, the proposal distribution for each new step in the chain is dependent upon the position of all the other walkers in the chain.
Choosing the initial position for each of the walkers does not significantly affect the final results (though it will affect the burn in time). Standard procedure is to create several walkers in a small ball around a reasonable guess [the samplers will quickly explore beyond the extent of the initial ball].
Can you write Python code for it?

guess = [10, 1, 0.6]
ndim = len(guess)
nwalkers = 100

p0 = [np.array(guess) + 1e-8 * np.random.randn(ndim)
      for i in range(nwalkers)]
sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob1, args=(x, y, dy), threads = ncores)


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 3f
Make a corner plot (use corner) to examine the post burn-in samples from the MCMC chains.
Can you write Python code for it?

samples = sampler.chain[:, nburn:, :].reshape((-1, ndim))
fig = # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 4c
Write a function lnprior2 to calculte $\ln P(\theta)$, the log prior for the model parameters. Use a wide flat prior for the parameters.
Note - a flat prior in log space is not flat in the parameters.
Can you write Python code for it?

def lnprior2( # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 4d
Write a function lnprob2 to calculate the log posterior given the model parameters and data.
Can you write Python code for it?

def lnprob2(# complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 4e
Intialize 100 walkers in an emcee.EnsembleSampler variable called sampler. For you initial guess at the parameter values set $\ln a = 1$, $\ln P = 1$, and $b = 8$.
Note - this is very similar to what you did previously.
Can you write Python code for it?

initial = # complete
ndim = len(initial)
p0 = [np.array(initial) + 1e-4 * np.random.randn(ndim)
      for i in range(nwalkers)]
sampler = emcee.EnsembleSampler( # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 4f
Run the chains for 200 steps.
Hint - you'll notice these are shorter chains than we previously used. That is because the computational time is longer, as will be the case for this and all the remaining problems.
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 4i
Reinitialize the walkers in a ball around the maximum log posterior value from the walkers in the previous burn in. Then run the MCMC sampler for 200 steps.
Hint - you'll want to run sampler.reset() prior to the running the MCMC, but after selecting the new starting point for the walkers.
Can you write Python code for it?

p = # complete
sampler.reset()

p0 = # complete
p0, _, _ = sampler.run_mcmc( # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 4j
Plot the chains. Have they converged?
Can you write Python code for it?

paramsNames = ['ln(P)', 'ln(a)', 'b']
nburn = # complete
plot_chains( # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 4k
Make a corner plot of the samples. Does the marginalized distribution on $P$ make sense?
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 4l
What is the marginalized best period estimate, including uncertainties?
Can you write Python code for it?

# complete

print('ln(P) = {:.6f} +{:.6f} -{:.6f}'.format( # complete
print('True period = 0.4, GP Period = {:.4f}'.format( # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 5c
Plot the chains from the MCMC. Did the chains converge?
Can you write Python code for it?

paramsNames = ['ln(P)', 'ln(a)', 'b', '$ln(\gamma)$']
nburn = # complete
plot_chains( # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 5d
Plot the final $\ln P$ vs. log posterior for each of the walkers. Do you notice anything interesting?
Hint - recall that you are plotting the log posterior, and not the posterior.
Can you write Python code for it?

# complete
# complete
# complete
# complete
# complete
# complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 5e
Re-initialize the walkers around the chain with the maximum log posterior value.
Run the MCMC for 500 steps.
Can you write Python code for it?

p = # complete
sampler.reset()
# complete
sampler.run_mcmc( # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 6b
Adjust the prior from problem 5 to be appropriate for this data set.
Can you write Python code for it?

def lnprior3( # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 6d
Initialize one third of your 100 walkers around each of the periods identified in the previous problem (note - the total number of walkers must be an even number, so use 34 walkers around one of the top 3 frequency peaks).
Run the MCMC for 500 steps following this initialization.
Can you write Python code for it?

initial1 = # complete
# complete
# complete

initial2 = # complete
# complete
# complete

initial3 = # complete
# complete
# complete

# complete
sampler = emcee.EnsembleSampler( # complete

p0, _, _ = sampler.run_mcmc( # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 6f
Plot $\ln P$ vs. log posterior.
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Problem 6g
Reinitialize the walkers around the previous walker with the maximum posterior value.
Run the MCMC for 500 steps. Plot the chains. Have they converged?
Can you write Python code for it?

# complete
sampler.reset()
# complete
# complete
sampler.run_mcmc( # complete

paramsNames = ['ln(P)', 'ln(a)', 'b', '$ln(\gamma)$']
nburn = # complete
plot_chains( # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
The Neverending Search for Periodicity: Techniques Beyond Lomb-Scargle
Version 0.1

By AA Miller 28 Apr 2018
In this lecture we will examine alternative methods to search for periodic signals in astronomical time series. The problems will provide a particular focus on a relatively new technique, which is to model the periodic behavior as a Gaussian Process, and then sample the posterior to identify the optimal period via Markov Chain Monte Carlo analysis. A lot of this work has been pioneered by previous DSFP lecturer Suzanne Aigrain.
For a refresher on GPs, see Suzanne's previous lectures: part 1 & part 2. For a refresher on MCMC, see Andy Connolly's previous lectures: part 1, part 2, & part 3.
An Incomplete Whirlwind Tour
In addition to LS, the following techniques are employed to search for periodic signals:
String Length
The string length method (Dworetsky 1983), phase folds the data at trial periods and then minimizes the distance to connect the phase-ordered observations.

 (credit: Gaveen Freer - http://slideplayer.com/slide/4212629/#)

Phase Dispersion Minimization
Phase Dispersion Minimization (PDM; Jurkevich 1971, Stellingwerth 1978), like LS, folds the data at a large number of trial frequencies $f$.
The phased data are then binned, and the variance is calculated in each bin, combined, and compared to the overall variance of the signal. No functional form of the signal is assumed, and thus, non-sinusoidal signals can be found.
Challenge: how to select the number of bins?

 (credit: Gaveen Freer - http://slideplayer.com/slide/4212629/#)

Analysis of Variance
Analysis of Variance (AOV; Schwarzenberg-Czerny 1989) is similar to PDM. Optimal periods are defined via hypothesis testing, and...
Can you write Python code for it?

ncores =          # adjust to number of CPUs on your machine
np.random.seed(23)


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
The cell below shows marginalized samples overplotted on the actual data. How well does the model perform?
Can you write Python code for it?

fig, ax = plt.subplots()
ax.errorbar(lc['hjd'], lc['mag'], lc['mag_unc'], fmt='o')
ax.set_xlabel('HJD (d)')
ax.set_ylabel('mag')

hjd_grid = np.linspace(2800, 3000,3000)

for s in samples[np.random.randint(len(samples), size=5)]:
    # Set up the GP for this sample.
    lnper, lna, b, lngamma = s
    gp = george.GP(np.exp(lna) * kernels.ExpSine2Kernel(np.exp(lngamma), lnper))
    gp.compute(lc['hjd'], lc['mag_unc'])
    # Compute the prediction conditioned on the observations and plot it.
    m = gp.sample_conditional(lc['mag'] - model3(s, lc['hjd']), hjd_grid) + model3(s, hjd_grid)

    ax.plot(hjd_grid, m, color="0.2", alpha=0.3)
fig.tight_layout()


I am working on the file "Sessions/Session06/Day2/FeatureEngineering.ipynb".
The first task is:
Features as a Representation of Time Series for Classification
version 0.1

By AA Miller (Northwestern CIERA/Adler Planetarium)
01 May 2018
This lecture is about machine learning...
But honestly, this lecture isn't really about machine learning...
This lecture is about the classification of variable sources in astronomical survey data. There are many different ways to approach such a classification problem, and today we will use a machine leaning approach to accomplish this task.
As a(n incredibly) brief reminder, machine learning algorithms use a training set with known labels$^1$ to develop a mapping between the data and the labels. You can, and should, think of this mapping as a black box. The mapping can occur between the raw data and the labels (e.g., neural net classification of images) or between representative features$^2$ and the labels.
$^1$ Labels are the parameters of interest to be estimated (a variable star classification in this case).
$^2$Features = measured properties of the sources.
Once the mapping between the data and the labels has been learned from the training set, new classifications can be obtained by applying the machine learning model to sources where the labels are unknown.
Break Out Problem 1
Why would it be useful to measure features from astronomical light curves in order to classify them in an automated fashion?
Solution to Break Out 1
Write your answer here
The peculiarities of astronomical light curves (observational gaps, heteroskedastic uncertainties, etc) makes it difficult to compare any 2 random sources. For example, the cadence of observations in one portion of the sky will ultimately be very different from any other point on the sky separated by an appreciable distance ($\sim 100^\circ$ for LSST).
The use of features allows us to place all sources on the same basis. In this way it then becomes possible to make 1 to 1 comparisons between sources with different observing sequences.
Problem 1) The ML Training Set
Here we are going to define some helper functions that...
Can you write Python code for it?

def lc_plot(t, m, m_unc, period=0.0):
    if period == 0.0:
        fig, ax = plt.subplots()
        ax.errorbar(t, m, m_unc,
                    fmt='o', color='MediumAquaMarine',
                    mec="0.2",mew=0.5)
        ax.set_xlabel('HJD (d)')
        ax.set_ylabel(r'$V_\mathrm{ASAS}\;(\mathrm{mag})$')
        fig.gca().invert_yaxis()
    elif period != 0.0:
        fig = plt.figure()
        gs = GridSpec.GridSpec(5, 1)
        ax_full = plt.subplot(gs[:2, :])
        ax_full.errorbar(t, m, m_unc,
                         fmt='o', color='MediumAquaMarine',
                         mec="0.2",mew=0.5)
        ax_full.set_xlabel('HJD (d)')
        ax_full.set_ylabel(r'$V_\mathrm{ASAS}\;(\mathrm{mag})$')
        plt.gca().invert_yaxis()

        ax_phase = plt.subplot(gs[2:, :])
        for repeat in [-1, 0, 1]:
            ax_phase.errorbar(t/period % 1 + repeat, m, m_unc,
                             fmt='o', color='MediumAquaMarine',
                             mec="0.2",mew=0.5)
        ax_phase.axvline(x=0, ls='--', color='0.8', lw=1, zorder=3)
        ax_phase.axvline(x=1, ls='--', color='0.8', lw=1, zorder=3)
        ax_phase.set_xlim(-0.2, 1.2)
        ax_phase.set_xlabel('Phase')
        ax_phase.set_ylabel(r'$V_\mathrm{ASAS}\;(\mathrm{mag})$')
        plt.gca().invert_yaxis()

    plt.tight_layout()

def read_lc(filename):
    hjd, mag, mag_unc = np.loadtxt(filename, unpack=True)
    return hjd, mag, mag_unc


I am working on the file "Sessions/Session06/Day2/FeatureEngineering.ipynb".
The first task is:
If you did not already have the training set, download and unpack the tarball.
We will be working with data from the ASAS survey, and I have already curated a training set that only includes stars in 1 of 7 classes: Mira variables, RR Lyrae stars, detatched eclipsing binaries, semi-detatched eclipsing binaries, W UMa binaries, Cepheids, and R Cor Bor stars.
[If you don't know what any of these things are, don't worry, we have examples below.]
Problem 1a
Plot an example Mira light curve.
Hint - just execute the cell.
Can you write Python code for it?

# Mira example
t, m, m_unc = read_lc("./training_lcs/181637+0341.6")
lc_plot(t, m, m_unc, period=150.461188)


I am working on the file "Sessions/Session06/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 1c
Plot an example detatched eclipsing binary (EB) light curve.
Can you write Python code for it?

# dEB example
t, m, m_unc = read_lc("./training_lcs/153835-6727.8")
lc_plot(t, m, m_unc, period=2*1.107174)


I am working on the file "Sessions/Session06/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 1d
Plot an example semi-detatched EB light curve.
Can you write Python code for it?

# aEB example
t, m, m_unc = read_lc("./training_lcs/141748-5311.2")
lc_plot(t, m, m_unc, period=1.514158)


I am working on the file "Sessions/Session06/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 1f
Plot an example Cepheid light curve.
Can you write Python code for it?

# Cepheid example
t, m, m_unc = read_lc("./training_lcs/065640+0011.4")
lc_plot(t, m, m_unc, period=4.022837)


I am working on the file "Sessions/Session06/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 1g
Plot an example R Cor Bor star light curve.
Can you write Python code for it?

# R Cor Bor example
t, m, m_unc = read_lc("./training_lcs/163242-5315.6")
lc_plot(t, m, m_unc, period=0.0)


I am working on the file "Sessions/Session06/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 2) Machine Learning Classification
To classify newly observed light curves we need a machine learning model.
Previously I said this is not a machine learning problem, and that is because we will all use the same pre-specified model. I have provided a file training_sources.csv which includes the name of the sources, along with some features, and their classification.
Problem 2a
Read in the training set file, and create a feature vector X and label array y.
Can you write Python code for it?

train_df = pd.read_csv("training_sources.csv")

X_train = np.array(train_df[["mean", "nobs", "duration"]])
y_train = np.array(train_df["Class"])


I am working on the file "Sessions/Session06/Day2/FeatureEngineering.ipynb".
The first task is:
Problem 3) Feature Engineering
It should now be clear why this is not a machine learning problem. We have, in this case, provided all the machine learning code that you will need.
Instead, this is a feature engineering problem. Feature engineering requires the utilization of domain knowledge to create new features for a data set to improve the performance of machine learning algorithms.
Add new features - if necessary
  - Utilize domain knowledge to create/compute new features
  - Combine features or represent in an alternative fashion
Remove noisy/uniformative features - if necessary
  - Determine feature importance (RF)
  - Forward/backward selection to iteratively remove features
Below I have provided a partial function calc_feature which you can alter to calculate new features for the data set.
Can you write Python code for it?

def calc_feature(df, train=True):

    if train==True:
        lc_dir = "./training_lcs/"
    else:
        lc_dir = "./test_lcs/"

    feature = np.empty(len(df))
    for source_num, asas_id in enumerate(df["ASAS_ID"]):
        t, m, mu = read_lc(lc_dir+asas_id)
        # feature calculations
        # feature calculations
        # feature calculations

        feature[source_num] = feat_val

    return feature


I am working on the file "Sessions/Session06/Day2/FeatureEngineering.ipynb".
The first task is:
The provided training set comes with 3 features: i) the mean magnitude of the observations, ii) the total number of observations obtained, and iii) the duration of the observations.
Problem 2b
Using the helper function provided below, calculate the 10-fold cross-validation accuracy of a random forest machine learning model using the 3 features provided above.
Note - do not adjust any part of calc_cv_score throughout this exercise.
Can you write Python code for it?

def calc_cv_score(X, y):
    rf_clf = RandomForestClassifier(n_estimators=150, min_samples_leaf=1)

    cv_score = cross_val_score(rf_clf, X, y, cv=10, n_jobs=-1)

    print("These features have CV accuracy = {:.4f} +/- {:.4f}".format(np.mean(cv_score), np.std(cv_score, ddof=1)))

calc_cv_score( # complete


I am working on the file "Sessions/Session06/Day2/IntroToParaView.ipynb".
The first task is:
First and foremost, you should have already installed ParaView on your machine. If not, you can download the package from paraview.org.
An Introduction to ParaView
version 0.1

Adam A Miller (Northwestern CIERA/Adler Planetarium)
01 May 2018
Our data do not live in 2 dimensions...
[As the third grade version of me would say, "Ummmm, Mr. Miller? Duh!"]
And yet, a fundamental limitation we that consisently place upon ourselves is to insist that we represent the data in 2D when communicating with the public and our colleagues.
In a sense, we are doing science with 1 arm tied behind our backs...
Our data also do not live in 3 dimensions.
But I can think of a distinct advantage to moving to 3D representations: our spatial representation of the Universe can be fully described in 3 dimensions.
Thus, if we want to describe the positions of stars in a cluster (Gaia?), or show variations in density within a Giant Molecular Cloud, or examine the surface of a minor planet or asteroid, these tasks are all acomplished much better in 3D.
(ignoring special relativity for now...)
Why interactive?
It would not be unreasonable to take the following stance: the heavy lifting in exploring a data set should be done by the researchers. In this sense, "final" plots or renditions can be presented to an audience via a slide or paper that summarize all of the salient features.
What if, however, you wish to enable science, or discovery, for amatuers (e.g., the Zooniverse, or even in the classroom (see James's talk from yesterday)?
It is unfair to ask this audience to calculate integrals, or to develop even a small fraction of the domain expertise that you and your colleagues (PhDs and PhD candidates) have. Interactivity provides a natural way for this audience to explore the data.
Furthermore, it may even help you in your quest for discovery and understaning.
So, today –– ParaView
ParaView is an open source platform that was specifically designed for data analysis and visualization. Briefly, ParaView is interesting (for us) for...
Can you write Python code for it?

from pyevtk.hl import gridToVTK, pointsToVTK


I am working on the file "Sessions/Session06/Day2/IntroToParaView.ipynb".
The first task is:
For simplicity, we have focused on the most ordered methods of producing vtk files. As we have demonstrated, pyevtk provides a simple interface to convert NumPy arrays into vtk binary files.
Problem 5) Example Data Analysis - K means
In addition to 3D rendering, ParaView provies utilities for performing (light) statistical analysis on data sets.
As a demonstration of this we will revisit one of our commonly used data sets (the famous Iris machine learning data).
Load the iris data into a pandas DataFrame via seaborn.
Can you write Python code for it?

import seaborn as sns
iris = sns.load_dataset("iris")


I am working on the file "Sessions/Session06/Day2/IntroToParaView.ipynb".
The first task is:
From here we can establish the coordinates of the cell and the point data.
Can you write Python code for it?

x = np.arange(0, lx + 0.1*dx, dx, dtype='float64')
y = np.arange(0, ly + 0.1*dy, dy, dtype='float64')
z = np.arange(0, lz + 0.1*dz, dz, dtype='float64')


I am working on the file "Sessions/Session06/Day2/IntroToParaView.ipynb".
The first task is:
Problem 4b
Execute the cell below to create a .vtu file that can be read by ParaView.
Open the file in ParaView. Represent the pressure data as Point Gaussian, and change the color to match that of the data.
From the ParaView render, how would you describe the pressure of the system? What about the temperature?
Can you write Python code for it?

pointsToVTK("./points", x, y, z, data = {"temp" : temp, "pressure" : pressure})


I am working on the file "Sessions/Session06/Day2/IntroToParaView.ipynb".
The first task is:
Problem 4c
Assign random values for the pressure and tempurature and export the data to a file called structure.vtr.
Open structure.vtr in ParaView. Represent the data as a surface with edges, and color the pressure data.
Note - pay attention to the shape of the pressure and temperature data.
Can you write Python code for it?

pressure = # complete

temp = # complete

gridToVTK("./structured", x, y, z, cellData = {"pressure" : pressure}, pointData = {"temp" : temp})


I am working on the file "Sessions/Session06/Day2/IntroToParaView.ipynb".
The first task is:
Problem 4e
Create a vtk file with the pressure and temperature data. Open the resuling file in ParaView.
Examine the volume rendering of the data. Does the visualization make sense given the input data?
Hint - the x, y, and z coordinates should all be specified as 3d arrays.
Can you write Python code for it?

gridToVTK("./structured", xt, yt, zt, cellData = {"pressure" : pressure}, pointData = {"temp" : temp})


I am working on the file "Sessions/Session06/Day2/IntroToParaView.ipynb".
The first task is:
Problem 5a
As a point of comparison, quickly visualize this data in 2D.
Hint - seaborn makes this easy and possible with a single line of code.
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day2/IntroToParaView.ipynb".
The first task is:
We will start with a relatively straightforward example of creating a collection of points in 3 dimentions
Problem 4a
Create 3 arrays, x, y, z, of 500 points each that consist of random draws from independent gaussian distributions with mean 0 and standard deviation 20.
Evalute the temperature and density on the grid provided below.
$$T = 2.4/x + 0.04y^2 + 5/z$$
$$\rho = 1.11x^2 + 3.1y + 0.6z^2$$
Can you write Python code for it?

np.random.seed(23)

x = # complete
y = # complete
z = # complete

temp = # complete
pressure = # complete


I am working on the file "Sessions/Session06/Day2/IntroToParaViewSolutions.ipynb".
The first task is:
As you can see: the iris data feature 3 classes in a four dimentional data set. We will now prepare this data for visualization in ParaView.
Problem 5b
Select 3 of the variables to serve as the (x, y, z) spatial coordinates for the data set. Note - the specific selection does not particularly matter, though 2 features are clearly better than the others.
Can you write Python code for it?

x = np.array(iris["petal_length"])
y = np.array(iris["petal_width"])
z = np.array(iris["sepal_length"])


I am working on the file "Sessions/Session06/Day2/IntroToParaViewSolutions.ipynb".
The first task is:
First and foremost, you should have already installed ParaView on your machine. If not, you can download the package from paraview.org.
An Introduction to ParaView
version 0.1

Adam A Miller (Northwestern CIERA/Adler Planetarium)
01 May 2018
Our data do not live in 2 dimensions...
[As the third grade version of me would say, "Ummmm, Mr. Miller? Duh!"]
And yet, a fundamental limitation we that consisently place upon ourselves is to insist that we represent the data in 2D when communicating with the public and our colleagues.
In a sense, we are doing science with 1 arm tied behind our backs...
Our data also do not live in 3 dimensions.
But I can think of a distinct advantage to moving to 3D representations: our spatial representation of the Universe can be fully described in 3 dimensions.
Thus, if we want to describe the positions of stars in a cluster (Gaia?), or show variations in density within a Giant Molecular Cloud, or examine the surface of a minor planet or asteroid, these tasks are all acomplished much better in 3D.
(ignoring special relativity for now...)
Why interactive?
It would not be unreasonable to take the following stance: the heavy lifting in exploring a data set should be done by the researchers. In this sense, "final" plots or renditions can be presented to an audience via a slide or paper that summarize all of the salient features.
What if, however, you wish to enable science, or discovery, for amatuers (e.g., the Zooniverse, or even in the classroom (see James's talk from yesterday)?
It is unfair to ask this audience to calculate integrals, or to develop even a small fraction of the domain expertise that you and your colleagues (PhDs and PhD candidates) have. Interactivity provides a natural way for this audience to explore the data.
Furthermore, it may even help you in your quest for discovery and understaning.
So, today –– ParaView
ParaView is an open source platform that was specifically designed for data analysis and visualization. Briefly, ParaView is interesting (for us) for...
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day2/IntroToParaViewSolutions.ipynb".
The first task is:
For simplicity, we have focused on the most ordered methods of producing vtk files. As we have demonstrated, pyevtk provides a simple interface to convert NumPy arrays into vtk binary files.
Problem 5) Example Data Analysis - K means
In addition to 3D rendering, ParaView provies utilities for performing (light) statistical analysis on data sets.
As a demonstration of this we will revisit one of our commonly used data sets (the famous Iris machine learning data).
Load the iris data into a pandas DataFrame via seaborn.
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day2/IntroToParaViewSolutions.ipynb".
The first task is:
From here we can establish the coordinates of the cell and the point data.
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day2/IntroToParaViewSolutions.ipynb".
The first task is:
Problem 4b
Execute the cell below to create a .vtu file that can be read by ParaView.
Open the file in ParaView. Represent the pressure data as Point Gaussian, and change the color to match that of the data.
From the ParaView render, how would you describe the pressure of the system? What about the temperature?
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day2/IntroToParaViewSolutions.ipynb".
The first task is:
Problem 4c
Assign random values for the pressure and tempurature and export the data to a file called structure.vtr.
Open structure.vtr in ParaView. Represent the data as a surface with edges, and color the pressure data.
Note - pay attention to the shape of the pressure and temperature data.
Can you write Python code for it?

pressure = np.random.rand(ncells).reshape( (nx, ny, nz))

temp = np.random.rand(npoints).reshape( (nx + 1, ny + 1, nz + 1))

gridToVTK("./structured", x, y, z, cellData = {"pressure" : pressure}, pointData = {"temp" : temp})


I am working on the file "Sessions/Session06/Day2/IntroToParaViewSolutions.ipynb".
The first task is:
Problem 5a
As a point of comparison, quickly visualize this data in 2D.
Hint - seaborn makes this easy and possible with a single line of code.
Can you write Python code for it?

g = sns.PairGrid(iris, hue="species", hue_kws={"cmap": ["Blues", "Oranges", "Greens"]})
g = g.map_diag(plt.hist, lw=3)
g = g.map_lower(sns.kdeplot, lw=1)
g = g.map_upper(plt.scatter)


I am working on the file "Sessions/Session06/Day2/IntroToParaViewSolutions.ipynb".
The first task is:
Problem 5c
Export the iris data as a vtk file for ParaView. Include both the species and the four features as data for the vtk file. Finally, be sure to randomize the order of the data prior to writing the vtk file.
Hint 1 - convert the species to a numerical representation.
Hint 2 - you may find np.random.choice helpful.
Can you write Python code for it?

species = np.array(iris["species"].map({"setosa":1,
                                        "versicolor":2,
                                        "virginica":3}))
petal_width = np.array(iris["petal_width"])
petal_length = np.array(iris["petal_length"])
sepal_width = np.array(iris["sepal_width"])
sepal_length = np.array(iris["sepal_length"])

shuffle = np.random.choice(range(len(x)), size=len(x), replace=False)

pointsToVTK("./iris", x[shuffle], y[shuffle], z[shuffle],
            data = {"species" : species[shuffle],
                    "sepal_length": sepal_length[shuffle],
                    "sepal_width": sepal_width[shuffle],
                    "petal_length": petal_length[shuffle],
                    "petal_width": petal_width[shuffle]})


I am working on the file "Sessions/Session07/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
By eye (a metric that is hardly quantitative, but nevertheless worth developing because talks never provide all of the details), the quadratic fit appears "better" than the linear fit.
But, there are still "outliers" and in the realm of polynomial fitting, it is always possible to get a better fit by adding more degrees to the polynomial. Should we keep going here, or should we stop? (Again - we will discuss model selection with Adam on Friday)
[As a reminder - in machine learning we'd call this low training error, but the generalization error is likely huge]
How should we deal with these potential outliers?
(and to re-iterate, we cannot be certain that these points are, in fact, outliers)
Amazingly, if you scroll through the literature you can find solutions like the following: "We do not believe the data point at (x, y) for reasons A, B, C. Thus, we exclude that point from the fit."
This, obviously, lacks any sort of rigor. If data is going to be removed, and it is worth asking if data should ever be removed, it should not be subject to the "beliefs" of an individual person or group.
A more common approach that you might encounter is known as $k\sigma$ clipping, which is an iterative procedure to identify and remove outliers from a data set. The procedure is as follows:

Fit the model to the data
Identify any data points that are $k\sigma$ discrepant from the best-fit model
Remove the discrepant points, repeat steps 1 & 2 until there are no data beyond $k\sigma$

The motivation for this procedure is the following: in a small data set (such as the one above, 20 points), the likelihood of having large $\sigma$ deviations (let's say $k = 5$) is vanishingly small. Thus, it "makes sense" to remove those points.
Of course, this only makes sense if the uncertainties are truly gaussian with low variance. So again, specific, and likely untrue, assumptions have to be made.
Problem 3c
Develop a $k\sigma$ clipping procedure to fit a line to the data set. Set $k = 5$ and determine the best-fit line to the data....
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
By eye, this appears superior to the previous fit. At the same time, we have not actually optimized anything to definitively show that this is the case.
If there are outliers in the data, then it stands to reason that the uncertainties are likely not correctly estimated.
Problem 3e
Perform the $k\sigma$ procedure on the data, with $k = 5$, but the variance increased by a factor 4 (i.e. sigma_y increased by a factor of 2). Plot the results as above.
How do these results compare to our previous fits?
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
Problem 1c
[You may have already done this] Now that we understand the origin of the data, make a scatter plot showing their distribution.
Can you write Python code for it?

plt.scatter( # complete
# complete
# complete


I am working on the file "Sessions/Session07/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
Problem 2f
Confirm the results of this fit are the same as those from np.polyfit.
Hint - be sure to include the uncertainties.
Can you write Python code for it?

p = np.polyfit( # complete
print("The best-fit value for the slope and intercept are: {:.4f} and {:.4f}".format( # complete


I am working on the file "Sessions/Session07/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
Problem 3) Are the Uncertainties Actually Gaussian?
Previously we noted that there are two essential assumptions that are required for least-squares fitting to be correct. We are now going to examine the latter requirement, namely, that the uncertainties can be perfectly described as Gaussians with known variance.
Earlier I stated this assumption is rarely satisfied. Why might this be the case?
In my experience (meaning this is hardly universal), if it's astro, it's got systematics. While I cannot prove this, I contend that systematic uncertainties are rarely Gaussian. If you are lucky enough to be in a regime where you can be confident that the systematics are Gaussian, I further contend that it is extremely difficult to be certain that the variance of that Gaussian is known.
Then there's another (astro-specific) challenge: in many circumstances, we aren't actually working with data, but rather with the results of other models applied to the data.
Let's take an optical astronomy (biased, but this is LSST after all) example. What are the data? In many cases inference is being based on measurements of brightness, but the true data in this case is simply a bunch of electron counts in a CCD. The brightness (or mag) is based on the application of a model (e.g., PSF, aperture, Kron) that is applied to the data. Thus, to assume that a flux (or mag) measurement has Gaussian uncertainties with known variance is to assume that whatever flux-measurement model has been applied always produces perfectly Gaussian uncertainties (and a lot of different assumptions go into flux-measurement models...)
As a demonstration associated with the challenges of these assumptions, we will examine the data set presented in Hogg, Bovy, & Lang.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
Problem 3a
Using the least-squares methodology developed in Problem 2, determine the best-fit slope and intercept for a line fit to the data above.
Make a scatter plot of the data, and overplot the best-fit line. What if anything, do you notice about the data and the fit?
Can you write Python code for it?

Y = # complete
# complete

# complete

plt.errorbar( # complete
plt.plot( # complete

print("The best-fit value for the slope and intercept are: {:.4f} and {:.4f}".format( # complete


I am working on the file "Sessions/Session07/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
So we have now uncovered one of the peculiariaties of least-squares. Fitting y vs. x is not the same as fitting x vs. y.
There are a couple essential assumptions that go into standard least-squares fitting:

There is one dimension along which the data have negligible uncertainties
Along the other dimension all of the uncertainties can be described via Gaussians of known variance

These two conditions are rarely met for astronomical data. While condition 1 can be satisfied (e.g., time series data where there is essentially no uncertainty on the time of the observations), I contend that condition 2 is rarely, if ever, satisfied.
Speaking of uncertainties(1), we have not utilized any thus far. [I hope this has raised some warning bells.]
We will now re-organize our data to match what is originally in Hogg, Bovy, & Lang (previously x and y were swapped).
(1) There is an amazing footnote in Hogg, Bovy, & Lang about "errors" vs. "uncertainties" - I suggest everyone read this.
Problem 2d
Re-plot the data including the uncertatines.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
The Assumptions of Least Squares
========
Version 0.1

By AA Miller 2018 Nov 04
Today we will focus on a relatively simple problem, while highlighting several challenges for the standard astronomical workflow, as a way of setting up the various lectures that will happen over the course of this week.
A lot of the lessons in this lecture are inspired by the paper Data Analysis Recipes: Fitting a Model to Data by Hogg, Bovy, & Lang. [This paper has been mentioned previously in the DSFP, though today we will only be able to scratch the surface of its content.]
In some sense - the goal right now is to make you really nervous about the work that you've previously done.
(Though this lecture should not be met with too much consternation –– we will discuss new approaches to fitting a line)
Problem 1) Data
At the core of everything we hope to accomplish with the DSFP stands a single common connection: data.
There are many things we (may) want to do with these data: reduce them, visualize them, model them, develop predictions from them, use them to infer fundamental properties of the universe (!).
Before we dive into that really fun stuff, we should start with some basics:
Problem 1a
What is data?
Take a few min to discuss this with your partner
Solution 1a
write your answer here
Problem 1b
Below, I provide some data (in the form of numpy arrays). As good data scientists, what is the first thing you should do with this data?
Feel free to create a new cell if necessary.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
We are now assuming that x has negligible uncertainties and that y has uncertainties that can be perfectly described by Gaussians of known variance.
A portion of the appeal of least-squares is that it provides a deterministic method for determining the best fit. To understand that we now need to do a little linear algebra.
We can arrange the data in the following matricies:
$$ \mathbf{Y} = \left[ {\begin{array}{c}
            y_1 \
            y_2 \
            \dots \
            y_N
            \end{array}
           }
            \right] , $$
$$ \mathbf{A} = \left[ {\begin{array}{cc}
            1 & x_1 \
            1 & x_2 \
            \dots & \dots \
            1 & x_N
            \end{array}
           }
           \right] ,
           $$
$$ \mathbf{C} = \left[ {\begin{array}{cccc}
            \sigma_{y_1}^2 & 0 & \dots & 0 \
            0 & \sigma_{y_2}^2 & \dots & 0 \
            \vdots & \vdots & \ddots & \vdots \
            0 & 0 & \dots & \sigma_{y_1}^2 \
            \end{array}
           }
           \right] ,
           $$
where $\mathbf{Y}$ is a vector, and $\mathbf{C}$ is the covariance matrix.
Ultimately, we need to solve the equation
$$\mathbf{Y} = \mathbf{A}\mathbf{X}.$$
I am skipping the derivation, but the solution to this equations is:
$$ \left[ {\begin{array}{c}
            b \
            m \
            \end{array}
           }
            \right] = \mathbf{X} = \left[ \mathbf{A}^T \mathbf{C}^{-1} \mathbf{A}\right]^{-1} \left[ \mathbf{A}^T \mathbf{C}^{-1} \mathbf{Y}\right].$$
As noted in Hogg, Bovy, & Lang, this procedure minimizes the $\chi^2$ function, which is the total squared error, after appropriately scaling by the uncertainties:
$$ \chi^2 = \Sigma_{i = 1}^{N} \frac{[y_i - f(x_i)]^2}{\sigma_{y_i}^2} = \left[ \mathbf{Y}  - \mathbf{A}\mathbf{X}\right]^{T} \mathbf{C}^{-1} \left[ \mathbf{Y} - \mathbf{A} \mathbf{X}\right].$$
Problem 2e
Using the linear algebra equations above (i.e. avoid np.polyfit or any other similar functions), determine the weighted least-squares best-fit...
Can you write Python code for it?

Y = # complete
A = # complete
C = # complete

X = # complete

best_fit = # complete

plt.errorbar( # complete
plt.plot( # complete

print("The best-fit value for the slope and intercept are: {:.4f} and {:.4f}".format( # complete


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
But! Histograms and scatter plots can be problematic as we have discussed many times before.
Problem 3d
Re-create the plot above but set kind='kde' to produce density estimates of the distributions.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
Even better - there is an option to create a PairGrid which allows fine tuned control of the data as displayed above, below, and along the diagonal. In this way it becomes possible to avoid having symmetric redundancy, which is not all that informative. In the example below, we will show scatter plots and contour plots simultaneously.
Can you write Python code for it?

g = sns.PairGrid(iris, vars = ["sepal_length", "sepal_width", "petal_length", "petal_width"],
                 hue = "species", diag_sharey=False)
g.map_lower(sns.kdeplot)
g.map_upper(plt.scatter, edgecolor='white')
g.map_diag(sns.kdeplot, lw=3)


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
Here, we see that there are many points, clustered about the origin, but we have no sense of the underlying density of the distribution. 2D histograms, such as plt.hist2d(), can alleviate this problem. I prefer to use plt.hexbin() which is a little easier on the eyes (though note - these histograms are just as subject to the same issues discussed above).
Can you write Python code for it?

# hexbin w/ bins = "log" returns the log of counts/bin
# mincnt = 1 displays only hexpix with at least 1 source present
with sns.axes_style("darkgrid"):
    plt.hexbin(xexample, yexample, bins = "log", cmap = "viridis", mincnt = 1)
    plt.colorbar()


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
In this representation, each "block" has a height of 0.25. The bandwidth is too narrow to provide any overlap between the blocks. This choice of kernel and bandwidth produces an estimate that is essentially a histogram with a large number of bins. It gives no sense of continuity for the distribution. Now, we examine the difference (relative to histograms) upon changing the the width (i.e. kernel) of the blocks.
Problem 1f
Plot the KDE of the PDF for the number of chinups middle aged men can do using bandwidths of 1 and 5 and a tophat kernel. How do the results differ from the histogram plots above?
Can you write Python code for it?

PDFtophat1 = # complete

# complete
# complete
# complete


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
Introduction to Visualization:
Density Estimation and Data Exploration
Version 0.1
There are many flavors of data analysis that fall under the "visualization" umbrella in astronomy. Today, by way of example, we will focus on 2 basic problems.

By AA Miller
16 September 2017
Can you write Python code for it?

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
Now that we have a sense of the data structure, it is useful to examine the distribution of features. Above, we went to great pains to produce histograms, KDEs, and rug plots. seaborn handles all of that effortlessly with the distplot function.
Problem 3b
Plot the distribution of petal lengths for the Iris data set.
Can you write Python code for it?

# note - hist, kde, and rug all set to True, set to False to turn them off
with sns.axes_style("dark"):
    sns.distplot(iris['petal_length'], bins=20, hist=True, kde=True, rug=True)


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
Now that we have covered the basics of seaborn (and the above examples truly only scratch the surface of what is possible), we will explore the power of seaborn for higher dimension data sets. We will load the famous Iris data set, which measures 4 different features of 3 different types of Iris flowers. There are 150 different flowers in the data set.
Note - for those familiar with pandas seaborn is designed to integrate easily and directly with pandas DataFrame objects. In the example below the Iris data are loaded into a DataFrame. iPython notebooks also display the DataFrame data in a nice readable format.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
Of course, even rug plots are not a perfect solution. Many of the chinup measurements are repeated, and those instances cannot be easily isolated above. One (slightly) better solution is to vary the transparency of the rug "whiskers" using alpha = 0.3 in the whiskers plot call. But this too is far from perfect.
To recap, histograms are not ideal for density estimation for the following reasons:

They introduce discontinuities that are not present in the data
They are strongly sensitive to user choices ($N_\mathrm{bins}$, bin centering, bin grouping), without any mathematical guidance to what these choices should be
They are difficult to visualize in higher dimensions

Histograms are useful for generating a quick representation of univariate data, but for the reasons listed above they should never be used for analysis. Most especially, functions should not be fit to histograms given how greatly the number of bins and bin centering affects the output histogram.
Okay - so if we are going to rail on histograms this much, there must be a better option. There is: Kernel Density Estimation (KDE), a nonparametric form of density estimation whereby a normalized kernel function is convolved with the discrete data to obtain a continuous estimate of the underlying PDF. As a rule, the kernel must integrate to 1 over the interval $-\infty$ to $\infty$ and be symmetric. There are many possible kernels (gaussian is highly popular, though Epanechnikov, an inverted parabola, produces the minimal mean square error).
KDE is not completely free of the problems we illustrated for histograms above (in particular, both a kernel and the width of the kernel need to be selected), but it does manage to correct a number of the ills. We will now demonstrate this via a few examples using the scikit-learn implementation of KDE: KernelDensity, which is part of the sklearn.neighbors module.
Note There are many implementations of KDE in Python, and Jake VanderPlas has put together an excellent description of the strengths and weaknesses of...
Can you write Python code for it?

# execute this cell
from sklearn.neighbors import KernelDensity
def kde_sklearn(data, grid, bandwidth = 1.0, **kwargs):
    kde_skl = KernelDensity(bandwidth = bandwidth, **kwargs)
    kde_skl.fit(data[:, np.newaxis])
    log_pdf = kde_skl.score_samples(grid[:, np.newaxis]) # sklearn returns log(density)

    return np.exp(log_pdf)


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
Of course, this data set lives in a 4D space, so plotting more than univariate distributions is important (and as we will see tomorrow this is particularly useful for visualizing classification results). Fortunately, seaborn makes it very easy to produce handy summary plots.
At this point, we are familiar with basic scatter plots in matplotlib.
Problem 3c
Make a matplotlib scatter plot showing the Iris petal length against the Iris petal width.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
Seaborn
Seaborn is a plotting package that enables many useful features for exploration. In fact, a lot of the functionality that we developed above can readily be handled with seaborn.
To begin, we will make the same plot that we created in matplotlib.
Can you write Python code for it?

import seaborn as sns

fig = plt.figure()
ax = fig.add_subplot(111)

ax.plot(x,y, lw = 2)
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_xlim(0, 6*np.pi)


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
That is much nicer than what was presented above. However - we still have a problem in that our data live in 4D, but we are (mostly) limited to 2D projections of that data. One way around this is via the seaborn version of a pairplot, which plots the distribution of every variable in the data set against each other. (Here is where the integration with pandas DataFrames becomes so powerful.)
Can you write Python code for it?

sns.pairplot(iris[["sepal_length", "sepal_width", "petal_length", "petal_width"]])


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
The folks behind seaborn have thought a lot about color palettes, which is a good thing. Remember - the choice of color for plots is one of the most essential aspects of visualization. A poor choice of colors can easily mask interesting patterns or suggest structure that is not real. To learn more about what is available, see the seaborn color tutorial.
Here we load the default:
Can you write Python code for it?

# default color palette

current_palette = sns.color_palette()
sns.palplot(current_palette)


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
These plots look identical, but it is possible to change the style with seaborn.
seaborn has 5 style presets: darkgrid, whitegrid, dark, white, and ticks. You can change the preset using the following:
sns.set_style("whitegrid")

which will change the output for all subsequent plots. Note - if you want to change the style for only a single plot, that can be accomplished with the following:
with sns.axes_style("dark"):

with all ploting commands inside the with statement.
Problem 3a
Re-plot the sine curve using each seaborn preset to see which you like best - then adopt this for the remainder of the notebook.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
This plot is much more appealing (and informative) than the previous two. For the first time we can clearly see that the distribution is not actually centered on the origin. Now we will move back to the Iris data set.
Suppose we want to see univariate distributions in addition to the scatter plot? This is certainly possible with matplotlib and you can find examples on the web, however, with seaborn this is really easy.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
While the above plot provides a significant improvement over the scatter plot by providing a better sense of the density near the center of the distribution, the binedge effects are clearly present. An even better solution, like before, is a density estimate, which is easily built into seaborn via the kdeplot function.
Can you write Python code for it?

with sns.axes_style("darkgrid"):
    sns.kdeplot(xexample, yexample,shade=False)


I am working on the file "Sessions/Session07/Day0/TooBriefVisualization.ipynb".
The first task is:
which we will now change to colorblind, which is clearer to those that are colorblind.
Can you write Python code for it?

# set palette to colorblind
sns.set_palette("colorblind")

current_palette = sns.color_palette()
sns.palplot(current_palette)


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Already with this simple plot we see a problem - the choice of bin centers and number of bins suggest that there is a 0% probability that middle aged men can do 10 chinups. Intuitively this seems incorrect, so lets examine how the histogram changes if we change the number of bins or the bin centers.
Problem 1b
Using the same data make 2 new histograms: (i) one with 5 bins (bins = 5), and (ii) one with the bars centered on the left bin edges (align = "left").
Hint - if overplotting the results, you may find it helpful to use the histtype = "step" option
Can you write Python code for it?

plt.hist(chinups, bins = 5, histtype="step", lw = 3)
plt.hist(chinups, align = "left", histtype="step", lw = 3)


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Ending the lie
Earlier I stated that histograms lie. One simple way to combat this lie: show all the data. Displaying the original data points allows viewers to somewhat intuit the effects of the particular bin choices that have been made (though this can also be cumbersome for very large data sets, which these days is essentially all data sets).  The standard for showing individual observations relative to a histogram is a "rug plot," which shows a vertical tick (or other symbol) at the location of each source used to estimate the PDF.
Problem 1d Execute the cell below to see an example of a rug plot.
Can you write Python code for it?

plt.hist(chinups, histtype = 'step')

# this is the code for the rug plot
plt.plot(chinups, np.zeros_like(chinups), '|', color='k', ms = 25, mew = 4)


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Even better - there is an option to create a PairGrid which allows fine tuned control of the data as displayed above, below, and along the diagonal. In this way it becomes possible to avoid having symmetric redundancy, which is not all that informative. In the example below, we will show scatter plots and contour plots simultaneously.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
For data sets where we have classification labels, we can even color the various points using the hue option, and produce KDEs along the diagonal with diag_type = 'kde'.
Can you write Python code for it?

sns.pairplot(iris, vars = ["sepal_length", "sepal_width", "petal_length", "petal_width"],
             hue = "species", diag_kind = 'kde')


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
In this representation, each "block" has a height of 0.25. The bandwidth is too narrow to provide any overlap between the blocks. This choice of kernel and bandwidth produces an estimate that is essentially a histogram with a large number of bins. It gives no sense of continuity for the distribution. Now, we examine the difference (relative to histograms) upon changing the the width (i.e. kernel) of the blocks.
Problem 1f
Plot the KDE of the PDF for the number of chinups middle aged men can do using bandwidths of 0.5 and 2.5 and a tophat kernel. How do the results differ from the histogram plots above?
Can you write Python code for it?

PDFtophat1 = kde_sklearn(chinups, grid, bandwidth = 1, kernel = 'tophat')
plt.plot(grid, PDFtophat1, 'MediumAquaMarine', lw = 3, label = "bw = 1")

PDFtophat5 = kde_sklearn(chinups, grid, bandwidth = 5, kernel = 'tophat')
plt.plot(grid, PDFtophat5, 'Tomato', lw = 3, label = "bw = 5")
plt.legend()


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Introduction to Visualization:
Density Estimation and Data Exploration
Version 0.1
There are many flavors of data analysis that fall under the "visualization" umbrella in astronomy. Today, by way of example, we will focus on 2 basic problems.

By AA Miller
16 September 2017
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Of course, this data set lives in a 4D space, so plotting more than univariate distributions is important (and as we will see tomorrow this is particularly useful for visualizing classification results). Fortunately, seaborn makes it very easy to produce handy summary plots.
At this point, we are familiar with basic scatter plots in matplotlib.
Problem 3c
Make a matplotlib scatter plot showing the Iris petal length against the Iris petal width.
Can you write Python code for it?

plt.scatter(iris['petal_length'], iris['petal_width'])
plt.xlabel("petal length (cm)")
plt.ylabel("petal width (cm)")


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Problem 1e
Plot the KDE of the PDF for the number of chinups middle aged men can do using a bandwidth of 0.1 and a tophat kernel.
Hint - as a general rule, the grid should be smaller than the bandwidth when plotting the PDF.
Can you write Python code for it?

grid = np.arange(0 + 1e-4,20,0.01)
PDFtophat = kde_sklearn(chinups, grid, bandwidth = 0.1, kernel = 'tophat')

plt.plot(grid, PDFtophat)


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
Seaborn
Seaborn is a plotting package that enables many useful features for exploration. In fact, a lot of the functionality that we developed above can readily be handled with seaborn.
To begin, we will make the same plot that we created in matplotlib.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
So, what is the optimal choice of bandwidth and kernel? Unfortunately, there is no hard and fast rule, as every problem will likely have a different optimization. Typically, the choice of bandwidth is far more important than the choice of kernel. In the case where the PDF is likely to be gaussian (or close to gaussian), then Silverman's rule of thumb can be used:
$$h = 1.059 \sigma n^{-1/5}$$
where $h$ is the bandwidth, $\sigma$ is the standard deviation of the samples, and $n$ is the total number of samples. Note - in situations with bimodal or more complicated distributions, this rule of thumb can lead to woefully inaccurate PDF estimates. The most general way to estimate the choice of bandwidth is via cross validation (we will cover cross-validation later today).
What about multidimensional PDFs? It is possible using many of the Python implementations of KDE to estimate multidimensional PDFs, though it is very very important to beware the curse of dimensionality in these circumstances.
Problem 2) Data Exploration
Now a more open ended topic: data exploration. In brief, data exploration encompases a large suite of tools (including those discussed above) to examine data that live in large dimensional spaces. There is no single best method or optimal direction for data exploration. Instead, today we will introduce some of the tools available via python.
As an example we will start with a basic line plot - and examine tools beyond matplotlib.
Can you write Python code for it?

x = np.arange(0, 6*np.pi, 0.1)
y = np.cos(x)

plt.plot(x,y, lw = 2)
plt.xlabel('X')
plt.ylabel('Y')
plt.xlim(0, 6*np.pi)


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
These plots look identical, but it is possible to change the style with seaborn.
seaborn has 5 style presets: darkgrid, whitegrid, dark, white, and ticks. You can change the preset using the following:
sns.set_style("whitegrid")

which will change the output for all subsequent plots. Note - if you want to change the style for only a single plot, that can be accomplished with the following:
with sns.axes_style("dark"):

with all ploting commands inside the with statement.
Problem 3a
Re-plot the sine curve using each seaborn preset to see which you like best - then adopt this for the remainder of the notebook.
Can you write Python code for it?

sns.set_style("ticks")

fig = plt.figure()
ax = fig.add_subplot(111)

ax.plot(x,y, lw = 2)
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_xlim(0, 6*np.pi)


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
While the above plot provides a significant improvement over the scatter plot by providing a better sense of the density near the center of the distribution, the binedge effects are clearly present. An even better solution, like before, is a density estimate, which is easily built into seaborn via the kdeplot function.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day0/TooBriefVizSolutions.ipynb".
The first task is:
which we will now change to colorblind, which is clearer to those that are colorblind.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
0b: Try a multi-line shell command
IPython magics often support "cell" magics by having %%<command> at the top of a cell.  Use that to cd into the directory below this one ("..") and then  ls inside that directory.
Hint: if you need syntax tips, run the magic() function and look for the ! or !! commands
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
0c: Create a new directory from Jupyter
While you can do this almost as easily with os.mkdir in Python, for this case try to do it using shell magics instead. Make a new directory in the directory you are currently in.  Use your system file browser to ensure you were sucessful.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
1b: Convert the directory into a git repo
Make that code into a git repository by doing git init in the directory you created,  then git add and git commit.
Can you write Python code for it?

%cd # complete

!git init

!git add code.py
!git commit -m #complete


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
1c: Create a repository for your code in Github
Go to github's web site in your web browser.   If you do not have a github account, you'll need to create one (follow the prompts on the github site).
Once you've got an account, you'll need to make sure your git client can authenticate with github.  If you're using a GUI, you'll have to figure it out (usually it's pretty easy). On the command line you have two options:
* The simplest way is to connect to github using HTTPS.  This requires no initial setup, but git will prompt you for your github username and password every so often.
* If you find that annoying, you can set up your system to use SSH to talk to github. Look for the "SSH and GPG keys" section of your settings on github's site, or if you're not sure how to work with SSH keys, check out github's help on the subject.
Once you've got github set up to talk to your computer, you'll need to create a new repository for the code you created.  Hit the "+" in the upper-right, create a "new repository" and fill out the appropriate details (don't create a README just yet).
To be consistent, it's recommended using the same name for your repository as the local directory name you used. But that is not a requirement, just a recommendation.
Once you've created the repository, connect your local repository to github and push your changes up to github.
Can you write Python code for it?

!git remote add <yourgithubusername> <the url github shows you on the repo web page> #complete

!git push <yourgithubusername> master -u


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
2d: Issue a pull request
Now use the github interface to create a new "pull request".  Once you've pushed your new branch up, you'll see a prompt to do this automatically appear on your fork's web page.  But if you don't, use the "branches" drop-down to navigate to the new branch, and then hit the "pull request" button. That should show you an interface that you can use to leave a title and description (in github markdown), and then submit the PR.  Go ahead and do this.
2e: Have them review the PR
Tell your neighbor that you've issued the PR.  They should be able to go to their repo, and see that a new pull request has been created.  There they'll review the PR, possibly leaving comments for you to change. If so, go to 2f, but if not, they should hit the "Merge" button, and you can jump to 2g.
2f: (If necessary) make changes and update the code
If they left you some comments that require changing prior to merging, you'll need to make those changes in your local copy, commit those changes, and then push them up to your branch on your fork.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
3b: Test your package
You should now be able to import your package and the code inside it as though it were some installed package like numpy, astropy, pandas,  etc.
Can you write Python code for it?

from <yourpkgname> import code#complete

#if your code.py has a function called `do_something` as in the example above, you can now run it like:
code.do_something()


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
3c: Apply packaging tricks
One of the nice things about packages is that they let you hide the implementation of some part of your code in one place while exposing a "cleaner" namespace to the users of your package.  To see a (trivial) example, of this, lets pull a function from your code.py into the base namespace of the package.  In the below make the __init__.py have one line: from .code import do_something.  That places the do_something() function into the package's root namespace.
Can you write Python code for it?

%%file <yourpkgname>/__init__.py

#complete


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
3d: Create a setup.py file
Ok, that's great in a pinch, but what if you want your package to be available from other directories?  If you open a new terminal somewhere else and try to import <yourpkgname> you'll see that it will fail, because Python doesn't know where to find your package.  Fortunately, Python (both the language and the larger ecosystem) provide built-in tools to install packages. These are built around creating a setup.py script that controls installation of a python packages into a shared location on your machine.  Essentially all Python packages are installed this way, even if it happens silently behind-the-scenes.
Below is a template bare-bones setup.py file.  Fill it in with the relevant details for your package.
Can you write Python code for it?

%%file setup.py
#!/usr/bin/env python

from distutils.core import setup

setup(name='<yourpkgname>',
      version='0.1dev',
      description='<a description>',
      author='<your name>',
      author_email='<youremail>',
      packages=['<yourpkgname>'],
     ) #complete


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
3e: Build the package
Now you should be able to "build" the package.  In complex packages this will involve more involved steps like linking against C or FORTRAN code, but for pure-python packages like yours, it simply involves filtering out some extraneous files and copying the essential pieces into a build directory.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
3f: Install the package
Alright, now that it looks like it's all working as expected, we can install the package.  Note that if we do this willy-nilly, we'll end up with lots of packages, perhaps with the wrong versions, and it's easy to get confused about what's installed (there's no reliable uninstall command...)  So before installing we first create a virtual environment using Anaconda, and install into that.  If you don't have anaconda or a similar virtual environment scheme, you can just do python setup.py install. But just remember that this will be difficult to back out (hence the reason for Python environments in the first place!)
Can you write Python code for it?

%%sh

conda create -n test_<yourpkgname> anaconda #complete
source activate test_<yourpkgname> #complete
python setup.py install


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
3g: Update the package on github
OK, it's now installable.  You'll now want to make sure to update the github version to reflect these improvements.  You'll need to add and commit all the files.  You'll also want to update the README to instruct users that they should use python setup.py install to install the package.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
4d: Install your package with pip
The pip tool is a convenient way to install packages on PyPI.  Again, we use Anaconda to create a testing environment to make sure everything worked correctly.
(Normally the -i wouldn't be necessary - we're using it here only because we're using the "testing" PyPI)
Can you write Python code for it?

%%sh

conda create -n test_pypi_<yourpkgname> anaconda #complete
source activate test_pypi_<yourpkgname> #complete
pip install -i https://testpypi.python.org/pypi <yourpkgname>

%%sh

cd $HOME
source activate test_pypi_<yourpkgname> #complete
python -c "import <yourpkgname>;<yourpkgname>.do_something()" #complete


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
Hopefully they are now satisfied and are willing to hit the merge button.
2g: Get the updated version
Now you should get the up-to-date version from the original owner of the repo, because that way you'll have both your changes and any other changes they might have made in the meantime.  To do this you'll need to connect your local copy to your nieghbor's github repo (not your fork).
Can you write Python code for it?

!git remote add <neighbors-username> <url-from-neighbors-github-repo> #complete
!git fetch <neighbors-username> #complete
!git branch --set-upstream-to=<neighbors-username>/master master
!git checkout master
!git pull


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
Now add it to the repository via git commit, and push up to github...
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
Now we can try running the package from anywhere (not just the source code directory), as long as we're in the same environment that we installed the package in.
Can you write Python code for it?

%%sh

cd $HOME
source activate test_<yourpkgname> #complete
python -c "import <yourpkgname>;<yourpkgname>.do_something()" #complete


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
Problem 2: Collaborating with others' repos
One very important advantages of working in repositories is that sharing the code becomes much easier, others (and your future self) can have a look at it, use it, and contribute to it.  So now we'll have you try modify your neighbors' project using github's Pull Request feature.
2a: Get (git?) your neighbor's code repo
Find someone sitting near you who has gotten through Problem 1.  Ask them their github user name and the name of their repository.
Once you've got the name of their repo, navigate to it on github.  The URL pattern is always "https://www.github.com/username/reponame". Use the github interface to "fork" that repo, yielding a "yourusername/reponame" repository.  Go to that one, take note of the URL needed to clone it (you'll need to grab it from the repo web page, either in "HTTPS" or "SSH" form, depending on your choice in 1a).  Then clone that onto your local machine.
Can you write Python code for it?

# Don't forget to do this cd or something like it... otherwise you'll clone *inside* your repo
%cd -0

!git clone <url from github>#complete
%cd <reponame>#complete


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
The -u is a convenience that means from then on you can use just git push and git pull to send your code to and from github.
1e: Modify the code and send it back up to github
Proper documentation is important.  But for now make sure to add a README to your code repository.  Always add a README with basic documentation.  Always.  Even if only you are going to use this code, trust me, your future self will be very happy you did it.
You can just call it README, but to get it to get rendered nicely on the github repository, you can call it README.md and write it using markdown syntax, REAMDE.rst in ReST or various other similar markup languages github understands.  If you don't know/care, just use README.md, as that's pretty standard at this point.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
Verify that there is a <yourpkg>-<version>.tar.gz file in the dist directory.  It should have all of the source code necessary for your package.
4c: Upload your package to PyPI
Once you have an account on PyPI (or testPyPI in our case) you can upload your distributions to PyPI using twine. If this is your first time uploading a distribution for a new project, twine will handle registering the project automatically filling out the details you provided in your setup.py.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/Code repositories.ipynb".
The first task is:
and push it up (to a branch on your github fork).
Can you write Python code for it?

!git push origin <name-of-branch>#complete


I am working on the file "Sessions/Session07/Day1/Code testing and CI.ipynb".
The first task is:
1e: Run the tests with py.test
Once you have an example test, you can try invoking py.test, which is how you should run the tests in the future.  This should yield a report that shows a dot for each test. If all you see are dots, the tests ran sucessfully.  But if there's a failure, you'll see the error, and the traceback showing where the error happened.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/Code testing and CI.ipynb".
The first task is:
1g: Check coverage
The coverage plugin we installed will let you check which lines of your code are actually run by the testing suite.
Can you write Python code for it?

!py.test --cov=<yourproject> tests/ #complete


I am working on the file "Sessions/Session07/Day1/Code testing and CI.ipynb".
The first task is:
2c
There are (at least) two significant bugs in this code (one fairly apparent, one much more subtle).  Try to catch them both, and write a regression test that covers those cases once you've found them.
One note about this function: in real code you're probably better off just using the Angle object from astropy.coordinates.  But this example demonstrates one of the reasons why that was created, as it's very easy to write a buggy version of this code.
Hint: you might find it useful to use astropy.coordinates.Angle to create test cases...
Can you write Python code for it?

#%%file <yourproject>/<filename>.py #complete, or just use your editor

import math

# know that to not have to worry about this, you should just use `astropy.coordinates`.
def angle_to_sexigesimal(angle_in_degrees, decimals=3):
    """
    Convert the given angle to a sexigesimal string of hours of RA.

    Parameters
    ----------
    angle_in_degrees : float
        A scalar angle, expressed in degrees

    Returns
    -------
    hms_str : str
        The sexigesimal string giving the hours, minutes, and seconds of RA for the given `angle_in_degrees`

    """
    if math.floor(decimals) != decimals:
        raise ValueError('decimals should be an integer!')

    hours_num = angle_in_degrees*24/180
    hours = math.floor(hours_num)

    min_num = (hours_num - hours)*60
    minutes = math.floor(min_num)

    seconds = (min_num - minutes)*60

    format_string = '{}:{}:{:.' + str(decimals) + 'f}'
    return format_string.format(hours, minutes, seconds)


I am working on the file "Sessions/Session07/Day1/Code testing and CI.ipynb".
The first task is:
2d
Hint: numpy has some useful functions in numpy.testing for comparing arrays.
Can you write Python code for it?

#%%file <yourproject>/<filename>.py #complete, or just use your editor

import numpy as np

def function_d(array1=np.arange(10)*2, array2=np.arange(10), operation='-'):
    """
    Makes a matrix where the [i,j]th element is array1[i] <operation> array2[j]
    """
    if operation == '+':
        return array1[:, np.newaxis] + array2
    elif operation == '-':
        return array1[:, np.newaxis] - array2
    elif operation == '*':
        return array1[:, np.newaxis] * array2
    elif operation == '/':
        return array1[:, np.newaxis] / array2
    else:
        raise ValueError('Unrecognized operation "{}"'.format(operation))


I am working on the file "Sessions/Session07/Day1/Code testing and CI.ipynb".
The first task is:
3b: Set up an account on travis
This turns out to be quite convenient.  If you go to the Travis web site, you'll see a "Sign in with GitHub" button.  You'll need to authorize Travis, but once you've done so it will automatically log you in and know which repositories are yours.
3c: Create a minimal .travis.yml file.
Before we can activate travis on our repo, we need to tell travis a variety of metadata about what's in the repository and how to run it.  The template below should be sufficient for the simplest needs.
Can you write Python code for it?

%%file .travis.yml

language: python
python:
  - "3.6"
# command to install dependencies
#install: "pip install numpy"  #uncomment this if your code depends on numpy or similar
# command to run tests
script: pytest


I am working on the file "Sessions/Session07/Day1/Code testing and CI.ipynb".
The first task is:
Problem 3: Set up travis to run your tests whenever a change is made
Now that you have a testing suite set up, you can try to turn on a continuous integration service to constantly check that any update you might send doesn't create a bug.  We will the Travis-CI service for this purpose, as it has one of the lowest barriers to entry from Github.
3a: Ensure the test suite is passing locally
Seems obvious, but it's easy to forget to check this and only later realize that all the trouble you thought you had setting up the CI service was because the tests were actually broken...
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/Code testing and CI.ipynb".
The first task is:
This should yield a report, which you can use to decide if you need to add more tests to acheive complete coverage.  Check out the command line arguments to see if you can get a more detailed line-by-line report.
Problem 2: Implement some unit tests
The sub-problems below each contain different unit testing complications.  Place the code from the snippets in your repository (either using an editor or the %%file trick), and write tests to ensure the correctness of the functions.  Try to achieve 100% coverage for all of them (especially to catch some hidden bugs!).
Also, note that some of these examples are not really practical - that is, you wouldn't want to do this in real code because there's better ways to do it.  But because of that, they are good examples of where something can go subtly wrong... and therefore where you want to make tests!
2a
When you have a function with a default, it's wise to test both the with-default call (function_b()), and when you give a value (function_b(1.2))
Hint: Beware of numbers that come close to 0... write your tests to accomodate floating-point errors!
Can you write Python code for it?

#%%file <yourproject>/<filename>.py #complete, or just use your editor

# `math` here is for *scalar* math... normally you'd use numpy but this makes it a bit simpler to debug

import math

inf = float('inf')  # this is a quick-and-easy way to get the "infinity" value

def function_a(angle=180):
    anglerad = math.radians(angle)
    return math.sin(anglerad/2)/math.sin(anglerad)


I am working on the file "Sessions/Session07/Day1/MachLearnAstroData.ipynb".
The first task is:
Problem 1) Examine the Training Data
For this problem the training set, i.e. sources with known labels, includes stars and galaxies that have been confirmed with spectroscopic observations. The machine learning model is needed because there are $\gg 10^8$ sources with photometric observations in SDSS, and only $4 \times 10^6$ sources with spectroscopic observations. The model will allow us to translate our knowledge from the spectroscopic observations to the entire data set. The features include each $r$-band magnitude measurement made by SDSS (don't worry if you don't know what this means...). This yields 8 features to train the models (significantly fewer than the 454 properties measured for each source in SDSS).
If you are curious (and it is fine if you are not) this training set was constructed by running the following query on the SDSS database:
SELECT TOP 20000
p.psfMag_r, p.fiberMag_r, p.fiber2Mag_r, p.petroMag_r,
p.deVMag_r, p.expMag_r, p.modelMag_r, p.cModelMag_r,
s.class
FROM PhotoObjAll AS p JOIN specObjAll s ON s.bestobjid = p.objid
WHERE p.mode = 1 AND s.sciencePrimary = 1 AND p.clean = 1 AND s.class != 'QSO'
ORDER BY p.objid ASC

First download the training set and the blind test set for this problem.
Problem 1a
Visualize the training set data. The data have 8 features ['psfMag_r', 'fiberMag_r', 'fiber2Mag_r', 'petroMag_r', 'deVMag_r', 'expMag_r', 'modelMag_r', 'cModelMag_r'], and a 9th column ['class'] corresponding to the labels ('STAR' or 'GALAXY' in this case).
Hint - just execute the cell below.
Can you write Python code for it?

sdss_df = pd.read_hdf("sdss_training_set.h5")
sns.pairplot(sdss_df, hue = 'class', diag_kind = 'hist')


I am working on the file "Sessions/Session07/Day1/MachLearnAstroData.ipynb".
The first task is:
Problem 1b
Based on your plots of the data, which feature do you think will be the most important for separating stars and galaxies? Why?
write your answer here - do not change it after later completing the problem
The final data preparation step it to create an independent test set to evalute the generalization error of the final tuned model. Independent test sets are generated by witholding a fraction of the training set. No hard and fast rules apply for the fraction to be withheld, though typical choices vary between $\sim{0.2}-0.5$.
sklearn.model_selection has a useful helper function train_test_split.
Problem 1c Split the 20k spectroscopic sources 70-30 into training and test sets. Save the results in arrays called: train_X, train_y, test_X, test_y, respectively. Use rs for the random_state in train_test_split.
Hint - recall that sklearn utilizes X, a 2D np.array(), and y as the features and labels arrays, respecitively.
Can you write Python code for it?

from sklearn.model_selection import train_test_split
rs = 1851

# complete

X = # complete
y = # complete

train_X, test_X, train_y, test_y = # complete


I am working on the file "Sessions/Session07/Day1/MachLearnAstroData.ipynb".
The first task is:
Problem 3b
Use 10-fold cross validation to estimate the FoM for the $k$NN model. Take the mean value across all folds as the FoM estimate.
Hint - the cross_val_score function from the sklearn.model_selection module performs the necessary calculations.
Can you write Python code for it?

from sklearn.model_selection import cross_val_score

knn_cv = cross_val_score( # complete

print('The kNN model FoM = {:.4f} +/- {:.4f}'.format( # complete


I am working on the file "Sessions/Session07/Day1/MachLearnAstroData.ipynb".
The first task is:
Problem 3d
Do the machine-learning models outperform the SDSS photometric classifier?
write your answer here
Problem 4) Model Optimization
While the "off-the-shelf" model provides an improvement over the SDSS photometric classifier, we can further refine and improve the performance of the machine learning model by adjusting the model tuning parameters. A process known as model optimization.
All machine-learning models have tuning parameters. In brief, these parameters capture the smoothness of the model in the multidimentional-feature space. Whether the model is smooth or coarse is application dependent -- be weary of over-fitting or under-fitting the data. Generally speaking, RF (and most tree-based methods) have 3 flavors of tuning parameter:

$N_\mathrm{tree}$ - the number of trees in the forest n_estimators (default: 10) in sklearn
$m_\mathrm{try}$ - the number of (random) features to explore as splitting criteria at each node max_features (default: sqrt(n_features)) in sklearn
Pruning criteria - defined stopping criteria for ending continued growth of the tree, there are many choices for this in sklearn (My preference is min_samples_leaf (default: 1) which sets the minimum number of sources allowed in a terminal node, or leaf, of the tree)

Just as we previously evaluated the model using CV, we must optimize the tuning paramters via CV. Until we "finalize" the model by fixing all the input parameters, we cannot evalute the accuracy of the model with the test set as that would be "snooping."
Before globally optimizing the model, let's develop some intuition for how the tuning parameters affect the final model predictions.
Problem 4a
Determine the 10-fold cross validation accuracy for $k$NN models with $k$ = 1, 10, 100.
How do you expect changing the number of neighbors to affect the results?
Can you write Python code for it?

for k in [1,10,100]:
    # complete

    print('With k = {:d}, the kNN FoM = {:.4f} +/- {:.4f}'.format( # complete


I am working on the file "Sessions/Session07/Day1/MachLearnAstroData.ipynb".
The first task is:
Problem 5c
Calculate the confusion matrix for the test set. Is there symmetry to the misclassifications?
Hint - the confusion_matrix function in sklearn.metrics will help.
Can you write Python code for it?

from sklearn.metrics import confusion_matrix

print(confusion_matrix( # complete


I am working on the file "Sessions/Session07/Day1/MachLearnAstroData.ipynb".
The first task is:
Problem 6) Classify New Data
Run the cell below to load in some new data (which in this case happens to have known labels, but in practice this will almost never be the case...)
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/MachLearnAstroData.ipynb".
The first task is:
Problem 6a
Create a feature and label array for the new data.
Hint - copy the code you developed above in Problem 2.
Can you write Python code for it?


I am working on the file "Sessions/Session07/Day1/MachLearnAstroData.ipynb".
The first task is:
Problem 6b
Calculate the accuracy of the model predictions on the new data.
Can you write Python code for it?

new_preds = # complete
print("The model has an accuracy of {:.4f}".format( # complete


I am working on the file "Sessions/Session07/Day1/MachLearnAstroData.ipynb".
The first task is:
Problem 6c
Can you explain why the accuracy for the new data is significantly lower than what you calculated previously?
If you can build and train a better model (using the trianing data) for classifying the new data - I will be extremely impressed.
write your answer here
Challenge Problem) Full RF Optimization
Now we will optimize the model over all tuning parameters. How does one actually determine the optimal set of tuning parameters? Brute force.
We will optimize the model via a grid search that performs CV at each point in the 3D grid. The final model will adopt the point with the highest accuracy.
It is important to remember two general rules of thumb: (i) if the model is optimized at the edge of the grid, refit a new grid centered on that point, and (ii) the results should be stable in the vicinity of the grid maximum. If this is not the case the model is likely overfit.
Use GridSearchCV to perform a 3-fold CV grid search to optimize the RF star-galaxy model. Remember the rules of thumb.
What are the optimal tuning parameters for the model?
Hint 1 - think about the computational runtime based on the number of points in the grid. Do not start with a very dense or large grid.
Hint 2 - if the runtime is long, don't repeat the grid search even if the optimal model is on an edge of the grid
Can you write Python code for it?

from sklearn.model_selection import GridSearchCV

# complete

print('The best model has {}'.format( # complete


I am working on the file "Sessions/Session07/Day1/MachLearnAstroData.ipynb".
The first task is:
write your answer here
Problem 4b
Determine the 10-fold cross validation accuracy for RF models with $N_\mathrm{tree}$ = 1, 10, 30, 100, and 300.
How do you expect changing the number of trees to affect the results?
Can you write Python code for it?

for ntree in [1,10,30,100,300]:
    # complete

    print('With {:d} trees the FoM = {:.4f} +/- {:.4f}'.format( # complete


I am working on the file "Sessions/Session07/Day1/MachLearnAstroData.ipynb".
The first task is:
write your answer here
Problem 5d
Calculate (and plot the region of interest) the ROC curve assumming that stars are the positive class.
Hint 1 - you will need to calculate probabilistic classifications for the test set using the predict_proba() method.
Hint 2 - the roc_curve function in the sklearn.metrics module will be useful.
Can you write Python code for it?

from sklearn.metrics import roc_curve

test_y_int = # complete
# complete

test_preds_proba = rf_clf.predict_proba( # complete

fpr, tpr, thresh = roc_curve( # complete

fig, ax = plt.subplots()
ax.plot( # complete
ax.set_xlabel('FPR')
ax.set_ylabel('TPR')


I am working on the file "Sessions/Session07/Day1/MachLearnAstroDataSolutions.ipynb".
The first task is:
A nice property of RF, relative to $k$NN, is that RF naturally provides an estimate of the most important features in a model.
RF feature importance is measured by randomly shuffling the values of a particular feature, and measuring the decrease in the model's overall accuracy. The relative feature importances can be accessed using the .feature_importances_ attribute associated with the RandomForestClassifer() object. The higher the value, the more important feature.
Problem 2c
Calculate the relative importance of each feature.
Which feature is most important? Does this match your answer from 1c?
Can you write Python code for it?

feat_str = ',\n'.join(['{}'.format(feat) for feat in np.array(feats)[np.argsort(rf_clf.feature_importances_)[::-1]]])

print('From most to least important: \n{}'.format(feat_str))