I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtraction.ipynb".
The first task is:
Problem 2) PSFs and Image weights in coadds
Problem (1) pretends that the input images are identical in quality, however this is never the case in practice. In practice, adding another image does not necessarily increase the SNR. For example, imagine you have two exposures, but in one the dome light was accidentally left on. A coadd with these two images weighted equally will have a worse SNR than the first image alone. Therefore the images should be aggregated with a weighted mean, so that images of poor quality don't degrade the quality of the coadd. What weights to we pick?
Weights can be chosen to either minimize the variance on the coadd or maximize the SNR of point sources on the coadd.
Some background:
Assuming that all noise sources are independent, the SNR of the measurement of flux from a star is:
\begin{equation}
SNR \propto {{N_{\rm photons}}\over{ \sigma_{\rm sky}} \sqrt{A} },
\end{equation}
where $N_{\rm photons}$ is the number of photons detected from the star,
$A$ is the area in pixels covered by the star.
The per-pixel sky noise  $\sigma_{sky}$ includes all sources of noise: dark current, read noise and sky-background, and it coded in the variance plane of the image. For the epoch $i$,  $\sigma^2_{i, {\rm sky}}$ is the average of the variance plane.
The $N_{\rm photons}$ is proportional to transparency $T$, and the area that the stellar photons cover is determined by the seeing: $A \propto {\rm FWHM}^2$.
Therefore, a coadd optimized for point-source detection would, to weight each image by the SNR$^2$, use the following as weights which prefers good-seeing epochs taken when the sky is transparent and dark.
\begin{align}
w_i & = {\rm SNR}^2 \propto {{T_i^2}\over{{\rm FWHM_i}^2 \sigma_i^2}}.
\end{align}
The usual inverse-variance weighting which produces the minimum-variance co-add, is given by,
\begin{align}
w_i & =T_i^2/\sigma_i^2
\end{align}
In practice, the factor of $T_i^2$ is incorporated into the variance when flux-scaling the single-epoch images to a common zeropoint. This step...
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtraction.ipynb".
The first task is:
Problem 2.2  Weighting images in Variable Seeing
Simulate 50 observations with FWHM's ranging from 2-10 pixels. Keep the flux amplitude F, and sky noise S both fixed.
Generate two coadds, (1) with the weights that minimize variance and (2) with the weights that maximize point source SNR. Weights should add up to 1. Plot both coadds.
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtraction.ipynb".
The first task is:
Problem 2.3 Image variance vs per pixel variance (Challenge Problem)
Why do we use per image variances instead of per pixel variances? Let's see! Start tracking the per-pixel variance when you simulate the star. Make a coadd of 200 observations with FWHM's ranging from 2 to 20 pixels.  Make a coadd weighted by the per-pixel inverse variance. How does the profile of the star look in this coadd compared to an unweighted coadd and compared to the coadd with the $w_i  = \frac{1}{{\rm FWHM_i}^2 \sigma_i^2}$? (You may have to plot the difference to see the change).
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtraction.ipynb".
The first task is:
Problem 3.1) Plot the difference of these two profiles:
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtraction.ipynb".
The first task is:
Problem 3.2)  What if we have amazing astrometric registration
and shrink the astrometric offset by a factor of a thousand. Is there a star sufficiently bright to produce the same dipole? What is its FLUX SCALE?
Can you write Python code for it?

ASTROM_OFFSET = 0.0001
PSF = 1.

# complete

# Plot both dipoles (for the offset=0.1 and the offset=0.0001 in the same figure.
# Same or different subplots up to you.


I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtraction.ipynb".
The first task is:
Problem 3.3) Distance between peaks.
Does the distance between the dipole's positive and negative peaks depend on the astrometric offset? If not, what does it depend on?  You can answer this by vizualizing the dipoles vs offsets. But for a challenge, measure the distance between peaks and plot them as a function of astrometric offset or another factor.
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtractionSolutions.ipynb".
The first task is:
Problem 1.2)  SNR vs N
Now compute the observed SNR of the simulated star on each coadd and compare to the expected SNR in the idealized case. The often repeated mnemonic for SNR inscrease as a function of number of images, $N$, is that noise decreses like $\sqrt{N}$. This is of course idealized case where the noise in each observation is identical.
Using your simulateN function, simulate a series of mean coadds with increasing N.

First, plot the empirical noise/uncertainty/stdev as a function of N. and overplot the expected uncertainty given the input sky level. You can use an area you know isn't touched by the star.
Next, plot the empirical SNR of the star (measured flux/fluxErr) as a function of N. Overplot the expected SNR. You can assume you know the sky level.

Your expected scaling with N should roughly track your empirical estimate.
Can you write Python code for it?

# complete

MU = 35
S = 100
F = 100
FWHM = 5

# hint. One way to start this
std = []
flux = []
fluxErrExpect = []
fluxErrObs = []
Ns = np.arange(1, 1000, 1)
for N in Ns:
    y = simulateN(x=x, mu=MU, F=F, fwhm=FWHM, S=S, N=N)
    # now coadd them:
    coadd = np.mean(y, axis=0)
    errEst = np.std(coadd[50:100])
    std.append(errEst)
    flux.append(np.sum(phi(x=x, mu=MU, fwhm=FWHM)*(coadd-100))/np.sum(phi(x=x, mu=MU, fwhm=FWHM)**2))
    fluxErrExpect.append(np.sum(phi(x=x, mu=MU, fwhm=FWHM)*(10/np.sqrt(N)))/np.sum(phi(x=x, mu=MU, fwhm=FWHM)**2))
    fluxErrObs.append(np.sum(phi(x=x, mu=MU, fwhm=FWHM)*(errEst))/np.sum(phi(x=x, mu=MU, fwhm=FWHM)**2))

plt.title('Flux estimate in coadd of N images')
plt.plot(Ns, flux, '.', label="coadd")
#plt.plot(Ns, 100, label="expected")
plt.xlabel('N')
plt.ylabel('Coadd PSF Flux')

plt.plot(Ns, np.array(std), '.', label="coadd", alpha=0.5)
plt.plot(Ns, np.sqrt(S)/np.sqrt(Ns), label="expected")
plt.xlabel('N')
plt.ylabel('pixel noise')
plt.legend()

plt.plot(Ns, np.array(flux)/np.array(fluxErrObs), '.', label="coadd", alpha=0.5)
plt.plot(Ns, F/np.array(fluxErrExpect), label="expected")
plt.xlabel('N')
plt.ylabel('PSF Flux SNR')
plt.legend()


I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtractionSolutions.ipynb".
The first task is:
Problem 2) PSFs and Image weights in coadds
Problem (1) pretends that the input images are identical in quality, however this is never the case in practice. In practice, adding another image does not necessarily increase the SNR. For example, imagine you have two exposures, but in one the dome light was accidentally left on. A coadd with these two images weighted equally will have a worse SNR than the first image alone. Therefore the images should be aggregated with a weighted mean, so that images of poor quality don't degrade the quality of the coadd. What weights to we pick?
Weights can be chosen to either minimize the variance on the coadd or maximize the SNR of point sources on the coadd.
Some background:
Assuming that all noise sources are independent, the SNR of the measurement of flux from a star is:
\begin{equation}
SNR \propto {{N_{\rm photons}}\over{ \sigma_{\rm sky}} \sqrt{A} },
\end{equation}
where $N_{\rm photons}$ is the number of photons detected from the star,
$A$ is the area in pixels covered by the star.
The per-pixel sky noise  $\sigma_{sky}$ includes all sources of noise: dark current, read noise and sky-background, and it coded in the variance plane of the image. For the epoch $i$,  $\sigma^2_{i, {\rm sky}}$ is the average of the variance plane.
The $N_{\rm photons}$ is proportional to transparency $T$, and the area that the stellar photons cover is determined by the seeing: $A \propto {\rm FWHM}^2$.
Therefore, a coadd optimized for point-source detection would, to weight each image by the SNR$^2$, use the following as weights which prefers good-seeing epochs taken when the sky is transparent and dark.
\begin{align}
w_i & = {\rm SNR}^2 \propto {{T_i^2}\over{{\rm FWHM_i}^2 \sigma_i^2}}.
\end{align}
The usual inverse-variance weighting which produces the minimum-variance co-add, is given by,
\begin{align}
w_i & =T_i^2/\sigma_i^2
\end{align}
In practice, the factor of $T_i^2$ is incorporated into the variance when flux-scaling the single-epoch images to a common zeropoint. This step...
Can you write Python code for it?

N = 50
skyArr = np.linspace(1, 1000, N)
stack = np.zeros((N, 100))
stackErr = np.zeros((N, 100))
inputErr = np.zeros((N, 100))

for i, sky in enumerate(skyArr):
    y = simulate(x=x, mu=MU, F=F, fwhm=FWHM, S=sky)
    stack[i] = y - np.repeat(sky, 100)
    stackErr[i] = np.sqrt(y)
    inputErr[i] = np.repeat(np.sqrt(sky), 100)

plt.plot(np.average(stack, weights=1/inputErr**2, axis=0), 'b-.', label='Minimum Variance')
plt.plot(np.average(stack, axis=0), 'r-.', label='Mean')
plt.legend()


I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtractionSolutions.ipynb".
The first task is:
Problem 2.3 Image variance vs per pixel variance (Challenge Problem)
Why do we use per image variances instead of per pixel variances? Let's see! Start tracking the per-pixel variance when you simulate the star. Make a coadd of 200 observations with FWHM's ranging from 2 to 20 pixels.  Make a coadd weighted by the per-pixel inverse variance. How does the profile of the star look in this coadd compared to an unweighted coadd and compared to the coadd with the $w_i  = \frac{1}{{\rm FWHM_i}^2 \sigma_i^2}$? (You may have to plot the difference to see the change).
Can you write Python code for it?

N = 5000
S = 100
FWHMs = np.linspace(2, 20, N)
stack = np.zeros((N, 100))
stackErr = np.zeros((N,100))
fwhmArr = np.zeros((N, 100))
inputArr = np.zeros((N, 100))

for i, fwhm in enumerate(FWHMs):
    y = simulate(x=x, mu=MU, F=F, fwhm=fwhm, S=S)
    stack[i] = y - S
    stackErr[i] = np.sqrt(y)
    fwhmArr[i] = np.repeat(fwhm, 100)

plt.plot(np.average(stack, weights=1/stackErr**2, axis=0), 'r-.', label='per-pixel')
plt.plot(np.average(stack, weights=1/FWHMs**2, axis=0), 'b--', label='per-image')
plt.plot(np.average(stack, axis=0), 'g--', label='unweighted')
plt.legend()


I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtractionSolutions.ipynb".
The first task is:
Problem 3) Dipoles in Image Subtraction
In the lesson, we said that just because you see a dipole in a difference image, does not mean that the astrometric registration is terrible. For this problem, we'll forgo the pixelated simulated star from the previous problems and operate with Gaussian profiles.
Can you write Python code for it?

# Create two Gaussian 1-D profiles with

ASTROM_OFFSET = 0.1  # units of e.g. pixels
FLUX_SCALE = 1. # units of e.g nanojansky
PSF = 1  # rms pixels

x = np.linspace(-5, 5)
y1 = FLUX_SCALE * norm.pdf(x, ASTROM_OFFSET, PSF)
y2 = FLUX_SCALE * norm.pdf(x, -ASTROM_OFFSET, PSF)
plt.plot(x, y1, label='profile 1')
plt.plot(x, y2, label='profile 2')
plt.xlabel('x (pixel)')
plt.ylabel('y (flux)')
plt.legend()


I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtractionSolutions.ipynb".
The first task is:
Problem 3.1) Plot the difference of these two profiles:
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
And just like that we have the best looking image (by far!) with a relatively minimal amount of effort. In this representation the ring nebula is a little saturated/washed out. We can correct that with a softening parameter (more below).
From Problem 1 it is clear: the inverse hyperbolic sine is the best transform to use when displaying astronomical images.
Problem 2) Color
As we have covered previously and extensively color is not a particularly useful tool for conveying information when visualizing data.
And yet...
When it comes to 2D CCD images, simply put, color wins.*

* With the important and glaring exception that the standard RGB format is not color-blind friendly, and therefore color images are not always inclusive.
For example, this  image with intensity information shows some stars and galaxies (and not much more).

 (credit: Zolt Levay/STScI)

Whereas the RGB version immediately reveals a high-$z$ quasar. (So this isn't just about "pretty pictures" we are using color to better communicate what we are looking at.)

 (credit: Zolt Levay/STScI)

Suppose you have data in 3 filters (e.g., g, r, i) and you wish to make a false color image. DSFP "best-practices" would suggest optimizing the luminance display for each filter and then showing them side by side (this is the colorblind-friendly solution):

 (credit: Zolt Levay/STScI)

You could even play with the color scheme slightly to be more subjective (while remaining colorblind-friendly):
<img style="display: block; margin-left: auto; margin-right: auto"...
Can you write Python code for it?

g_data = fits.getdata('./data/ztf_g_band.fits.gz')
r_data = fits.getdata('./data/ztf_r_band.fits.gz')


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
As before, there are few pixels bright enough over the full scale of the image to see any real structure in the data (though you can make out hints of the ring nebula in this otherwise awful image).
There are two ways to address this and we will do both: (i) use a different scaling, such as $\log$, and (ii) limit the full plotting range.
Problem 2d
Write a new (but very similar to 2c) function called scaled_log_intensity() that returns the log of the flux (to better highlight faint features), scaled from 0 to 1 (to allow output as an RGB image). Plot the newly scaled rgb array.
Can you write Python code for it?

def scaled_log_intensity( # complete
    '''
    Log scale input RGB arrays from 0 ---> 1

    Parameters
    ----------
    R_array : array-like of shape (M columns, N rows)
        R channel data array

    G_array : array-like of shape (M columns, N rows)
        G channel data array

    B_array : array-like of shape (M columns, N rows)
        B channel data array

    Returns
    -------
    rgb_scaled : array-like of shape (M columns, N rows, 3)
        RGB array for display with matplotlib
    '''

    # complete
    # complete
    # complete

fig, ax = plt.subplots( # complete
# complete
# complete


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
If you haven't already, please download the images that will be necessary for this exercise. Download the entire data directory and keep it in the same folder as this notebook.
Some quick background: these images are taken from the Zwicky Transient Facilty.* ZTF has a 47 $\mathrm{deg}^2$ camera, with $\sim$1 arcsec pixels and $\sim$2 arcsec seeing. It has 3 filters (g, r, i) and has mapped most of the Northern hemisphere. Today we are working with ZTF stacks, roughly 16 individual images has gone into each of these stacks.
* Yes, I am biased as a member of ZTF. However, I chose these images because they aren't all that nice. SDSS, DES, HSC, PS1, all would have been suitable replacements, typically with better seeing and as a result nicer results.
Problem 1) Luminance
We begin by examining how we want to represent the intensity of the light as detected by the ZTF CCD.
Problem 1a
Load the data from data\ztf_i_band.fits.gz and display a 2D "heatmap" using imshow().
Note - CCDs are linear detectors, so use a linear scaling (the default for imshow()) of the data in order to "see" what the detector sees.
Can you write Python code for it?

i_data = fits.getdata('./data/ztf_i_band.fits.gz')

# complete


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
Now the ring nebular stands out, (AND the color is telling us something, the strong green color reflects strong emission from $\mathrm{H}\alpha$ in the outer layers of the nebular, while the blue center is the result of [O III] emission), however, there is very little information being transmitted by the rest of the image. We can improve this by adding limits to the plotting range.
Problem 2e
Modify scaled_log_intensity() to "clip" each array at a maximum and minimum value. Replot your RGB array after clipping.
Hint – in principle this means six new variables should be added to the function (r_min, r_max, g_min, g_max, b_min, b_max). For now, use the same lower and upper bound for all 3 filters for simplicity. np.percentile() is a decent way to do this uniformly across all 3 filters.
Can you write Python code for it?

def scaled_log_intensity( # complete
    '''
    Log scale input RGB arrays from 0 ---> 1

    Parameters
    ----------
    R_array : array-like of shape (M columns, N rows)
        R channel data array

    G_array : array-like of shape (M columns, N rows)
        G channel data array

    B_array : array-like of shape (M columns, N rows)
        B channel data array

    # complete : # complete
        # complete

    # complete : # complete
        # complete

    Returns
    -------
    rgb_scaled : array-like of shape (M columns, N rows, 3)
        RGB array for display with matplotlib
    '''

    # complete
    # complete
    # complete

fig, ax = plt.subplots(# complete
# complete
# complete


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
Off the shelf that looks pretty good. However we are missing some of the fainter structures. We can "re-center" the data so that the low-level shadows are closer to the $\sim$linear portion of the $\operatorname{arsinh}$ response curve.
Problem 1g
Subtract the median from the data and re-plot using an inverse hyperbolic sine scaling.
Hint – do not forget about the nan values in the array.
Can you write Python code for it?

fig, ax = plt.subplots(# complete

# complete


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
Problem 3b
Create a new function tmp_lupton_2 that does everything in tmp_lupton_1 but also mutliplies the individual arrays by $f(I)$ as defined above. There will be 3 new parameters in the model now: $m$, $Q$, and $\alpha$.
Can you write Python code for it?

def tmp_lupton_2( # complete
    '''
    arsinh intensities scaled to RGB arrays from 0 ---> 1

    Parameters
    ----------
    R_array : array-like of shape (M columns, N rows)
        R channel data array

    G_array : array-like of shape (M columns, N rows)
        G channel data array

    B_array : array-like of shape (M columns, N rows)
        B channel data array

    # complete : # complete
        # complete

    # complete : # complete
        # complete

    # complete : # complete
        # complete

    Returns
    -------
    rgb_scaled : array-like of shape (M columns, N rows, 3)
        intensity scaled RGB array for display with matplotlib
    '''

    # complete
    # complete
    # complete


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
Problem 3c
Use tmp_lupton_2 to rescale the ZTF data and plot the results.
Hint – if the results appear a little underwhelming, do not forget to subtract the median value from the input arrays.
Can you write Python code for it?

fig, ax = plt.subplots( # complete
# complete


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
That looks terrible!
As the warning states, RGB data needs to be scaled from 0 to 1 (or 0 to 255 for integer inputs) in order to properly display the relative intensities in each channel.
Problem 2c
Build a helper function scaled_intensity() that takes as input 3 MxN arrays, and returns an MxNx3 array in which the input arrays have all been rescaled between 0 and 1.
Use your function to display a false color image of the ring nebula.
Hint –– here (and below) it may be helpful to write a solution in multiple cells.
Can you write Python code for it?

def scaled_intensity( # complete
    '''
    Linearly scale input RGB arrays from 0 ---> 1

    Parameters
    ----------
    R_array : array-like of shape (M columns, N rows)
        R channel data array

    G_array : array-like of shape (M columns, N rows)
        G channel data array

    B_array : array-like of shape (M columns, N rows)
        B channel data array

    Returns
    -------
    rgb_scaled : array-like of shape (M columns, N rows, 3)
        RGB array for display with matplotlib
    '''

    r_scaled = # complete

    g_scaled = # complete

    b_scaled = # complete

    rgb_scaled = # complete

    return # complete

fig, ax = plt.subplots( # complete

# complete


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
The first thing that jumps out of this image is all the white pixels. They are $\equiv$ nan because they have been flagged in the ZTF image processing pipeline as unreliable/no good.
Note - from an aesthic standpoint, you can make these disappear by inputting the following:
np.where(np.isnan(i_data), np.nanmedian(i_data), i_data)
to imshow(), which replaces the nan values with the median value from the image. While this is aesthetically more pleasing, it is also misleading. Most of the masked pixels are saturated.
We can quickly see the problem with a linear scaling. The dynamic range is too large to reveal any significant structure in the data. We can see the brightest stars, but that is it.
We can adjust the bounds on the plotted distribution to better highlight the salient features in the data.
Problem 1b
Make a histogram of the counts in the image. If necessary, adjust the default settings to get a better sense of the distribution.
Hint - be sure to input a 1D array, i_data is 2D.
Can you write Python code for it?

fig, ax = plt.subplots( # complete

# complete


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
This is an extremely similar solution to what we derived.
The stars all more or less look the same, though the outer outer halo of the ring nebular is now more clear (at the cost of saturating the inner $\mathrm{H}\alpha$ ring).
Problem 4b
This is optional In the cell below images from SDSS are loaded. Use make_lupton_rgb to create a false color RGB image. The star formation rate in the three main galaxies in the image are actually somewhat different, see if you can highlight that.
Can you write Python code for it?

from astropy.utils.data import get_pkg_data_filename

# Read in the three images downloaded from here:
g_name = get_pkg_data_filename('visualization/reprojected_sdss_g.fits.bz2')
r_name = get_pkg_data_filename('visualization/reprojected_sdss_r.fits.bz2')
i_name = get_pkg_data_filename('visualization/reprojected_sdss_i.fits.bz2')
g_dat = fits.open(g_name)[0].data
r_dat = fits.open(r_name)[0].data
i_dat = fits.open(i_name)[0].data

# complete
# complete
# complete


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
We can see a nearly logarithmic distribution, with nearly all the pixels near the "noise floor" of 175 or so, and a large tail to larger values.
Based on this histogram, we will clip the range we show to create better constrast in the visualization.
Problem 1c
Replot the i-band image while limiting the range that is displayed to extend from $m$ the minimum pixel value to $M$ the maximum pixel value. While it is possible to do this directly in imshow(), use numpy to limit the range of what is plotted (the utility of this will become clear later).
Can you write Python code for it?

m =
M =

fig, ax = plt.subplots(# complete

# complete


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
We have a false color image! The problem, as highlighted in Lupton et al. 2004, is that the "standard" log scaling of each channel leads to the cores of each of the individual stars appearing as a whitish color. While we have a false color image, we have actually lost all the color information.
Problem 3) Lupton et al. (2004)
As the Sloan Digital Sky Survey began producing multi-filter wide-field images at an "industrial" scale, it was quickly realized that superior solutions were needed for visualizing the 5 filter images (in particular, the use of "log" scaling dramatically reduced the information content).
Lupton et al. proposed a solution that has now been widely adopted. Half of the solution is to use $\operatorname{arsinh}$, and the other half is to be clever about defining the relative color in each pixel (perhaps the most useful insight from Lupton et al.).
We start by defining the intensity in each pixel $I \equiv (r + g + b)/3$, where the $r$, $g$, and $b$ values are the value in each pixel of the R(ed), G(reen), and B(lue) images, respectively. Then the output false color image can be defined by:
$$R = r f(I)/I, \ G = g f(I)/I, \ B = b f(I)/I,$$
where $f(I)$ is some scaling function, such as $\log$ or $\operatorname{arsinh}$.
As a final step the intensity in each filter is also clipped (which allows the preservation of color information even in pixels that have an intensity = 1; this is a hugely important insight). If
$$\mathrm{max_{RGB}} \equiv \mathrm{max(R,G,B)} > 1$$
then we need to set $R \equiv R/\mathrm{max_{RGB}}, G \equiv G/\mathrm{max_{RGB}},$ and $B \equiv B/\mathrm{max_{RGB}}$, similarly if $\mathrm{min_{RGB}} \leq 0$ then set $R \equiv G \equiv B \equiv 0$.
Finally, Lupton et al. point out that a useful parameterization of $f(I)$ is: $$f(I) = \operatorname{arsinh}(\alpha Q [I-m])/Q,$$ where $Q$ and $\alpha$ are both "softening" parameters that enable the user to highlight specific regions within the image with a linear stretch (and $m$ is the minimum for the plotting range as...
Can you write Python code for it?

def tmp_lupton_1( # complete
    '''
    normalize intensity and scale input RGB arrays from 0 ---> 1

    Parameters
    ----------
    R_array : array-like of shape (M columns, N rows)
        R channel data array

    G_array : array-like of shape (M columns, N rows)
        G channel data array

    B_array : array-like of shape (M columns, N rows)
        B channel data array

    Returns
    -------
    rgb_scaled : array-like of shape (M columns, N rows, 3)
        intensity scaled RGB array for display with matplotlib
    '''

    # complete
    # complete
    # complete


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
With the log scaling, we finally see the beautiful barred spiral galaxy to the northwest of the ring nebula. However, as was the case with the linear scaling, in order to see the spiral arms in the galaxy all the structure in the ring nebula gets blown out.
(There are also issues with log scaling when making false color images - we will come back to this.)
It is now generally accepted that the best scaling for CCD images is the inverse hyperbolic sine. This non-linear transformation $$\operatorname{arsinh} x = \ln (x + \sqrt{x^2 + 1})$$ has the really nice property that for $|x| \ll 1$ $\operatorname{arsinh} x \approx x$ and for $|x| \gg 1$ $\operatorname{arsinh} x \approx \ln (2x)$. Therefore, with a single transformation we can get the best properties of each of the stretches condisered above.
Problem 1f
Plot the i-band data with an inverse hyperbolic sine scaling.
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
Wow!
This image of the Ring nebular and the nearby spiral is really nice. The background is black. The stars are not saturated (in the sense that it is extremely easy to tell the difference between the red stars and blue stars). Structure in the ring nebular can be seen (though the outermost halo is missing, see e.g., 3c). And the nearby barred-spiral looks especially good. The nucleus is somewhat orange, suggesting an older stellar population, while the arms are clearly blue (and therefore forming stars at a higher rate than the nucleus).
This is a very very nice depiction of the ring nebula.
Problem 4) Astropy
As is often the case, the good people at astropy have already created a helper function that essentially does everything that we just developed in this notebook. In this case, with the astropy.visualization library there is a function called make_lupton_rgb that takes as input 3 MxN arrays, and returns a scaled MxNx3 rgb array for display purposes.
Problem 4a
Load make_lupton_rgb and make a plot of the results. How does it compare to your own solution?
Hint –– the make_lupton_rgb function uses a parameter called stretch instead of $\alpha$. As you tune the output note that $\mathrm{stretch} \approx 1/\alpha$.
Can you write Python code for it?

from astropy.visualization import make_lupton_rgb

rgb = make_lupton_rgb( # complete
fig, ax = plt.subplots( # complete


I am working on the file "Sessions/Session11/Day4/VisualizingImages.ipynb".
The first task is:
You should already see a huge improvement relative to the $\log$ scaled false color image that was produced in Problem 2. Further improvements can still be made, however. For instance, the background is a little "muddy" in the sense that everything looks a little brown instead of black.
Problem 3d
Create a new function lupton that does everything in tmp_lupton_2 but also clips and normalizes data with $I \equiv 0$ or $\mathrm{max_{RGB}} > 1$.
Can you write Python code for it?

def lupton( # complete
    '''
    arsinh intensities scaled to RGB arrays from 0 ---> 1

    Parameters
    ----------
    R_array : array-like of shape (M columns, N rows)
        R channel data array

    G_array : array-like of shape (M columns, N rows)
        G channel data array

    B_array : array-like of shape (M columns, N rows)
        B channel data array

    # complete : # complete
        # complete

    # complete : # complete
        # complete

    # complete : # complete
        # complete

    Returns
    -------
    rgb_scaled : array-like of shape (M columns, N rows, 3)
        intensity scaled RGB array for display with matplotlib
    '''

    # complete
    # complete
    # complete

    # complete
    # complete
    # complete

    # complete
    # complete
    # complete

fig, ax = plt.subplots( # complete

# complete


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
Appendix
The cells below show the process used to select the data for Problem 5
Can you write Python code for it?

from sklearn.datasets import load_digits

mnist = load_digits()
np.save('im.npy', mnist.images[16])


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
From the histogram we quickly see the problem with ticks that point "in" – they can be obscured by the data. Furthermore, while it is not the case here ticks that go "in" can be confused with data. For these reasons, ticks that point out are almost always superior.
What about the top and right axes? I often prefer these ticks because they make it easier to estimate data values far from the lower-left corner of the figure, but ticks all around can give a porcupine appearance, and lead to confusing grid plot layouts (see e.g., 3e).
Sometimes a better solution, though one that is rarely employed in journal publications, is grid lines. Grid lines eliminate the need to carefully trace ticklines to estimate data values.
Problem 4b
Remake the basic plot (1d) with gridlines.
Hint - See pyplot.grid() for documentation.
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
I contend that you are familiar with the content of the above image. I also contend that, in this case, the use of viridis obscures what is being shown in the image.
Problem 5b
Remake the above plot using the binary colormap. Do you recognize the image now?
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
If the grid of plots is very small on your screen – be sure to consider each panel in detail (either zoom in, or remake the plot at a larger size, etc).
What do you notice?
Write your response here
The open circle is the most flexible symbol. Overlapping circles do not create new circles thereby providing a more unique sense of the underlying density.
(Quick aside - as N grows very large, plotting the individual points is not informative. Consider alternatives, e.g., contours, 2d histogram, etc.)
Problem 3) Strong Visual Boundaries
We have, until now, ignored the different classes in our data set. We will now consider visualizations that feature $> 1$ symbol.
Problem 3a
Start with the matplotlib defaults and remake the "basic plot" from 1d with different symbols for class 0 and class 1 (ignore class 2 for now).
Can you write Python code for it?

class0 = # complete
class1 = # complete

fig, axes = plt.subplots( # complete

# complete
# complete
# complete
# complete


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
In some applications grid lines are useful to support rapid inference or comparisons.
Grid lines can also look like data. A visualization with 4 axes, grids, and other lines can quickly become confusing...
Speaking of axes, they are probably the least considered default in matplotlib. They are, however, like everything else in your visualizations, customizable.
Problem 4d
Remake the basic plot (1d) and remove the top and right axes.
Has the visualization significantly changed?
Hint – pyplot.spines() makes it easy to turn axes on or off.
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
In the vast majority of cases the default axes parameters should be fine. Removing the top and right axis encourages the eye to wander out towards infinity (though in a paper the placement of text will prevent that). The bold axes draw attention away from the data, which is rarely what you want the viewer to focus on (one possible exception – on slides a bold axis can draw attention towards the figure should there be competing text or titles that the audience might otherwise look at).
Problem 5) Color
There are two important default colors to know about in matplotlib: the default colormap is viridis and the default color sequence is the 10 color palette developed by Tableau.
In short, viridis is much better than the previous default (the dreaded jet/rainbow colormap). The Tableau palette was designed to be colorblind friendly when displaying categorical data (it isn't perfect, but it is better than picking random "well-known" colors – green, blue, red, yellow, etc).
Problem 5a
Download this (incredibly small) image data file, im.npy.$^\dagger$ Load the data into a variable im, and use pyplot.imshow() to display the image data.
What is this an image of?
$^\dagger$ The appendix at the end of this notebook shows how the file was created. Please don't skip ahead to learn the answer.
Can you write Python code for it?

im = np.load('im.npy')

# complete
# complete
# complete


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
In total there are 27 different style sheets, and they provide a wide range of initial plotting possibilities. Options include changes in the background colors (dark backgrounds, which are typically avoided because they look bad in papers, can often look really nice on projected slides), grids (seaborn and fivethirtyeight both provide grids that are not too intrusive, and colormaps and color cycles that vary significantly.
Problem 7) Salience
The lecture and this notebook are supposed to be about building salience, and yet we have spent the last 6 problems only talking about default settings in matplotlib – what's your deal Adam?
Problem 7a
Build a visualization showing all three classes created in problem 1. Class 2 should be the "salient feature" for this visualization. Thinking about the defaults, discussed to an annoying degree above, and things like visual boundaries, highlight the importance of class 2.
Hint - this is intentionally open ended. You will need to explore a bit before arriving at your final answer.
Can you write Python code for it?

# complete
# complete
# complete

# complete
# complete
# complete


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
Let's unpack what we just did: we created a data set with n_samples=367 sources, separated across n_classes=3 different classes, in a n_features=4 dimensional space. The classes are not balanced, execute the cell below to show that class 2 is quite rare.
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
Problem 1c) A histogram function
Create a function basic_histogram() that takes as arguments a maplotlib axis instance ax, a scikit-learn data array X, and optional **kwargs. The function should add a histogram to the axis instance using the first feature in X, and then return ax. Again, this is silly, but it will be useful later.
Can you write Python code for it?

def basic_histogram( # complete
    """Add simple histogram to axis object

    ### Write your own docstrings ###
    """

# complete
# complete
# complete
# complete


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
Problem 1d) The basic figure
Execute the cell below to create a side-by-side visualization showing a scatter plot of the first two features in the X array ($\mathtt{X1}$ vs. $\mathtt{X0}$) in the left panel, and a histrogram of the first feature $\mathtt{X0}$ in the right panel.
Hint - if there are only a few data points, you have indexed the X array incorrectly. If you do not have a scatter plot, you have either not specified a marker style and/or set ls = 'None' in the call to basic_scatter.
Can you write Python code for it?

fig, axes = plt.subplots(1, 2, sharex=True,
                         figsize=(8,4))
basic_scatter(axes[0], X)
basic_histogram(axes[1], X)
fig.tight_layout()


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
Problem 2d
Repeat the same grid using open symbols (i.e., only the outline of the symbol should be displayed).
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
Problem 6c
Remake the basic multiclass plot using the seaborn matplotlib style.
Do you see improvements relative to what we made previously?
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
Problem 6d
Remake the basic multiclass plot using the fivethirtyeight matplotlib style.
Do you see improvements relative to what we made previously?
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
Problem 6e
Remake the basic multiclass plot using the dark_background matplotlib style.
Do you see improvements relative to what we made previously?
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
Recall that scikit-learn feature arrays (typically labeled X) are 2D numpy arrays where each row corresponds to an individual source and each column corresponds to a different feature. We will mostly focus on features (i.e., columns) 1 and 2 and classes 0 and 1 today (the other class will become important in the end).
Problem 1b) A scatter plot function
Create a function basic_scatter() that takes as arguments a maplotlib axis instance ax, a scikit-learn data array X, and optional **kwargs. The function should add a scatter plot to the axis instance using the first two features in X, and then return ax. Note that this construction is a little silly, but it will allow us to easily loop and examine several different options in subsequent problems.
Hint - use the plt.plot() call to create the scatter plot. This will require "turning off" the lines ls = 'None', and specifying a marker m = 'o'. We will start with a circle but examine this choice later.
Can you write Python code for it?

def basic_scatter( # complete
    """Add simple scatter plot to axis object

    Parameters
    ----------
    ax : matplotlib.pyplot axis instance
        The axis object that will have/display the scatter plot

    X : {array-like, sparse matrix} of shape (n_samples, n_features)
        The input samples.

    m : matplotlib marker, optional (default = 'o')
        A matplotlib marker, see:
        https://matplotlib.org/api/_as_gen/matplotlib.lines.Line2D.html#matplotlib.lines.Line2D.set_marker

    Returns
    -------

    ax : matplotlib.pyplot axis instance
        The axis object with an associated scatter plot
    """

# complete
# complete
# complete
# complete


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
Salience in Matplotlib: Considering the Defaults For Basic Plotting
Version 0.1

By AA Miller
5 Jan 2021
As discussed in today's lecture, and previous lectures, there is no limit on the number of choices you have when developing visual communications. This ranges from the relatively mundane – should the plot axis be 1 mm or 1.3 mm thick? – to the subtle and vitally complex – should the isolated 7-$\sigma$ outlier be highlighted or just referenced in a caption?
All of these choices (yes, even something as mundane as the width of the axis) affect the salience of the visualization.
To better understand how to build salience in matplotlib today's problems will first focus on the default settings. After examining the defaults, and identifying situations when they simply will not do, we will apply what we have learned to improve the salience of an existing image.
Important disclaimer Many DSFP notebooks are designed with a goal to do the thing. While this notebook also has a do the thing component (namely to examine plotting choices in matplotlib), it is more essential to consider the output. Rather than rushing to "level up" each problem, consider the visual output and how it affects what you are communicating to your audience.
Problem 1) Matplotlib Defaults & A Basic Plot
The single most influential factor in the final appearance of the visualizations that you create is the default settings for your preferred plotting software,$^\dagger$ whether you typically use matplotlib or something else.
Therefore it is important to understand the defaults, and how they are secretly (or not so secretly) affecting your output.
$^\dagger$ There are some exceptions like D3.js where literally every element of the visualization has to be specified by the user.
The default settings in matplotlib changed dramatically between versions 1.0 and 2.0. These changes were all for the better, but, as the docs say "it is impossible to select the best default for all cases."
Problem 1a
Create a "toy" data set using the...
Can you write Python code for it?

np.random.seed(2003)
X, y = make_classification(n_samples=367, n_classes=3,
                           n_features=4, n_redundant=0, n_informative=4,
                           flip_y=0.0, weights=[0.575,0.42, 0.005])


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
Strong visual boundaries are essential for efficient communication.
Problem 3e
Execute the cell below to see 5 different scatter plots, where only the symbol is being varied.
Which combinations lead to strong visual boundaries? Which do not? (You may need to zoom to answer these questions)
Note - I have reduced the size of each class by a factor of 4 to reduce the crowding in these plots
Can you write Python code for it?

fig, axes = plt.subplots(1, 5, sharex=True,
                         figsize=(20,4))

for ax, sym_pair in zip(axes, [['v', '^'], # triangle up and triangle down
                               ['o', 'x'], # circle and x
                               ['h', '8'], # hexagon and octogon
                               ['s', 'p'], # square and pentagon
                               ['*', 'd']  # diamond and star
                              ]):
    for class_num, (this_class, m,  mec) in enumerate(zip([class0[0][::4], class1[0][::4]],
                                                             sym_pair,
                                                             ['0.25', 'k'])):
        basic_scatter(ax, X[this_class],
                      m=m, mfc='None', mec=mec, ms=7,
                      label='class {}'.format(class_num))
    ax.legend()
fig.tight_layout()


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
The above plot shows that class 0 and 1 have overlapping distributions, though class 0 tends to have larger (smaller) values of $\mathtt{X0}$ ($\mathtt{X1}$) than class 1.
We also see that the filled circle performs worse as the number of classes increases, while filled histograms obscure the data in regions that exhibit significant overlap.
While minor changes to the above code can rectify these changes, recall that we are showing the defaults for matplotlib.
Problem 3b
Following 2d re-make the plot from 3a using open circles and open histograms.
Hint - use the kwarg histtype='step' in the call to basic_histogram()
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
The image shows a pixelated scan of a handwritten "6" - which is more clear in greyscale than viridis.
Color can be very powerful when building visualizations, but it is often unnecessary, and in some cases worse than a simple greyscale. If you must use color – make specific choices that improve what is being communicated by the final product.
Problem 6) Defaults
The developers of matplotlib are all much smarter than I. Their choices for default parameters work really well. But in papers, talks, posters and other presentations your aim should be to highlight and communicate very specific ideas. To do so you cannot rely solely on the defaults.
Avoiding defaults can also be daunting, the previous 5 problems only begin to scratch the surface of what is possible.  Fortunately, it is possible to "pre-load" different styles than the "standard" matplotlib default using matplotlib style sheets.
Problem 6a
Remake the basic multiclass plot (3a) using the classic (i.e., v1.0) matplotlib style.
Do you see improvements relative to what we made previously?
Hint - set the plot style using plt.style.context("classic")
Can you write Python code for it?

with plt.style.context("classic"):
    # complete
    # complete
    # complete


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
The points no longer overlap, but they are also nearly illegible.
Problem 2c
Make a grid of 5 side-by-side plots showing the following common symbols: circle 'o', square 's', triangle '^', plus sign '+', and hexagon 'h'.
Can you write Python code for it?

fig, axes = plt.subplots( # complete
# complete
# complete
# Complete


I am working on the file "Sessions/Session12/Day1/SalienceInMatplotlib.ipynb".
The first task is:
Write your responses here
Problem 4) Axes & Grids
By default matplotlib has ticks pointing "out" and only located on the left and bottom axes (see any plot above to confirm).
Problem 4a
Remake the basic plot (1d) with the ticks pointing "in" from the axes.
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day2/TellingStoriesWithData.ipynb".
The first task is:
Here is a code sample showing how to read the data and draw a colored plot.
Can you write Python code for it?

# Load the csv with pandas
df = pd.read_csv('hipgalv.LSR.csv', index_col=0)
#print(df)

# Use matplotlib's default "Reds" colormap. More colormaps and information here:
# https://matplotlib.org/3.1.0/tutorials/colors/colormaps.html
cmap = cm.get_cmap('Reds')

# Make the plot
ax = df.plot.scatter(x='z',
                     y='Mv',
                     c='colorb_v',
                     cmap=cmap,
                     title="Sample Visualization")

# Set the x-axis label
ax.set_xlabel("Altitude")

# Set the y-axis label
ax.set_ylabel("Intrinsic Luminosity")

# Set the colormap label
f = plt.gcf()
cax = f.get_axes()[1]
cax.set_ylabel('B-V Color')


I am working on the file "Sessions/Session12/Day3/NotebookIII_part1_intro_to_3d.ipynb".
The first task is:
Here, we can access them like you might expect, but it's harder to do row-based reductions.  And yet, this is the way you'll probably see it most of the time.
Regular
Other times, you will find that the data you have access to comes in regularly shaped, multi-dimensional arrays.
Whereas before you were usually using some form of a set of columns, in this case you'll be using either one or multiple arrays with three dimensions.
In most of these cases, the data points will be assumed to be spaced out evenly; in some cases, however, you might find that there is a supplemental array for each dimension that describes the distance between cell values.
Note: It's important when dealing with data provided in 3D arrays like this to recognize whether the data comes from a regular sampling (and thus represents sample points) or if it is assumed to be part of a volumetric average over the volume of each individual element.
We can create some fake data in this way with numpy functions; for instance, let's create a field that is $1/r^2$ from the point (-0.05, -0.05, -0.05).
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day3/NotebookIII_part1_intro_to_3d.ipynb".
The first task is:
Note that you can imagine having record or dtype-ful arrays in this situation too, or you might be in a situation where there's either an extra axis at the beginning or at the end for the "field" name.
When dealing with data in 3D, there's one additional thing to be aware of, particularly when the data is really big -- which data values are next to which.  This won't always be important, but if you're using really big datasets, it can have a non-negligible impact on the speed with which you're manipulating data.  We can see that by looking at the ordering of the array.  If it's C-ordered, then arr[0,0,0] is right next to arr[0,0,1] in memory.  If it's Fortran ordered, it'll be the opposite -- arr[0,0,0] will be right next to arr[1,0,0].  (There are other ways to "stride" data, but these are the two most common.)
You might also find yourself with data that isn't evenly spread out.  We can (with a little effort!) generate some, of the same type as before.
Can you write Python code for it?

x = np.add.accumulate(np.random.random(128))
x /= x.max()

y = np.add.accumulate(np.random.random(128))
y /= y.max()

z = np.add.accumulate(np.random.random(128))
z /= z.max()

ir2 = (((x - center[0])**2)[:, None, None]
      +((y - center[1])**2)[None, :, None]
      +((z - center[2])**2)[None, None, :])**-1
print(ir2.shape)


I am working on the file "Sessions/Session12/Day3/NotebookIII_part1_intro_to_3d.ipynb".
The first task is:
Now, in general, this isn't necessarily the best way to access data.  But, if you're using numpy to simply load data off disk, and that data is organized in this way, it does pose some advantages, especially if it's a mix of different types -- for instance, if you have an ID field that's an integer.
The final way you might see things stored in memory using just numpy -- and in some ways, this is similar to how pandas provides access to data -- is through a dictionary of arrays.
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day3/NotebookIII_part1_intro_to_3d.ipynb".
The first task is:
This is the simplest way of representing it.  Looking at it, you'll see that since we've organized it in the way we did, it comes out to a shape of (3, 4), which indicates that our first axis is the row and the second is the column.  We don't know the names, so that makes it a bit tricky unless we knew the names of the columns beforehand!
Let's try a different way.  This way we will use a dtype in numpy.
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day3/NotebookIII_part1_intro_to_3d.ipynb".
The first task is:
What makes 3D data special?

3D data is special -- it's just like 2D data, but more.

We're mostly going to talk about data that has some degree of spatial organization in three dimensions.  We will return to how to look at data that does not have a spatial organization in all three, but for now, let's start with that.
The key thing here is that we have a mechanism of measuring the distance between the first three columns ($x, y, z$).  In fact, we know that the distance can be computed:
$ \sqrt{(x_1 - x_0)^2 + (y_1 - y_0)^2 + (z_1 - z_0)^2}$
This is really cool!  It means that -- built in! -- you already have an idea of how to represent these points, and how to measure the distance between them.  Not only that, it also means that there's a prescription for how to visualize the points from a certain viewpoint.
Data!  And its storage ...
We're going to talk about basically two different ways of storing data for 3D systems.  There are more, and I'll allude to them a little bit, but let's start with the two you're the most likely to see and deal with.
Discrete
Discretely sampled 3D data is typically ordered as columns (but, this is not always the case!) where three of the columns can be assigned to dimensions corresponding to spatial positions -- for instance, $x, y, z$ or maybe $r, \theta, \phi$.  It should also be noted that in many cases, you'll also be presented with a column that corresponds to some form of a "radius of influence," or a set of columns that describe influences along different directions (either axially aligned, or axes themsevles.)
So for instance, you might have a situation where your data comes to you looking something like this:
| x | y | z | temperature |
|-|-|-|-|
| 0.5 | 0.5 | 0.5 | 100 |
| 0.25 | 0.1 | 0.4 | 150 |
| 0.71 | 0.9 | 0.9 | 200 |
In this particular case, you now have three rows, where the x, y, z variables and a field variable are defined.  Now, usually, you'll have lots more data -- this is just a pretty simple set, because I was trying to come up with a set of random...
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day3/NotebookIII_part2_overview_regular_3d.ipynb".
The first task is:
Now we have some data!  And, we can use matplotlib to visualize it in reduced form.  Let's try this out:
Can you write Python code for it?

plt.imshow(data["sph_n4_m4"][:,:,N//4], norm=LogNorm())
plt.colorbar()

phi.min(), phi.max()

plt.imshow(data["sph_n1_m0"].max(axis=0), norm=LogNorm())
plt.colorbar()


I am working on the file "Sessions/Session12/Day3/NotebookIII_part2_overview_regular_3d.ipynb".
The first task is:
One thing I've run into with plotly while making this notebook has been that in many cases, the 3D plots strain a bit under large data sizes.  This is to be expected, and is completely understandable!  One of the really nice things about regular mesh data like this is that you can usually cut it down quite effectively with slices.  Unfortunately, what I have found -- and I may have done something completely wrong! -- is that plotly some times appears to almost work, and then doesn't quite make it when I throw too much data at it.  I've found that it seems to work best in the neighborhood of $64^3$ zones, maybe a bit more.
Other Summary Techniques
There are, of course, other ways you can take a look at a set of values!  Given a regular mesh, it's straightforward with numpy to apply any of the reduction operations along one of the axes.  For instance, you might take the min, the max, the sum, the mean and so forth.  If we do this with our spherical harmonics data:
Can you write Python code for it?


I am working on the file "Sessions/Session12/Day3/NotebookIII_part2_overview_regular_3d.ipynb".
The first task is:
This is getting a bit cumbersome, though!  Let's try using the ipywidgets library to speed this up just a bit.
We're going to use the ipywidgets.interact decorator around our function to add some inputs.  This is a pretty powerful decorator, as it sets up new widgets based on the info that you feed it, and then re-executes the function every time those inputs change.
Can you write Python code for it?

import ipywidgets

@ipywidgets.interact(dataset = list(sorted(data.keys())), slice_position = (0, N, 1))
def make_plots(dataset, slice_position):
    plt.imshow(data[dataset][slice_position,:,:], norm=LogNorm())
    plt.colorbar()


I am working on the file "Sessions/Session12/Day3/NotebookIII_part2_overview_regular_3d.ipynb".
The first task is:
We'll use the scipy spherical harmonics function to make some data, but first we need a reference coordinate system.  We'll start with $x, y, z$ and then transform them into spherical coordinates.
Note: we'll be using the convention that $\theta \in [0, \pi]$ and $\phi \in[0,2\pi)$, which is reverse from what SciPy expects.  So if you compare to the docstring for sph_harm, keep that in mind.  Feel free to switch the definitions if you like!
Can you write Python code for it?

N = 64
x = np.mgrid[-1.0:1.0:N*1j][:,None,None]
y = np.mgrid[-1.0:1.0:N*1j][None,:,None]
z = np.mgrid[-1.0:1.0:N*1j][None,None,:]

r = np.sqrt(x*x + y*y + z*z)
theta = np.arctan2(np.sqrt(x*x + y*y), z)
phi = np.arctan2(y, x)

np.abs(x - r * np.sin(theta)*np.cos(phi)).max()

np.abs(y - r * np.sin(theta)*np.sin(phi)).max()

np.abs(z - r * np.cos(theta)).max()

data = {}
for n in [1, 4]:
    for m in range(n + 1):
        data[f"sph_n{n}_m{m}"] = np.absolute(scipy.special.sph_harm(m, n, phi, theta))


I am working on the file "Sessions/Session12/Day3/NotebookIII_part3_overview_discrete_3d.ipynb".
The first task is:
And we're left with what seems to be a rather servicable distribution of projected mass.  There are a few artifacts we need to clean up, especially including our units (which I will forego) but also the slight overlapping of the hexagonal bins.  We can turn that off by making our edge lines a bit smaller.
Can you write Python code for it?

plt.hexbin(part_gas["Coordinates"][:,0], part_gas["Coordinates"][:,1], part_gas["Masses"], reduce_C_function=np.sum, norm=mcm.LogNorm(), linewidth=0.2)
plt.colorbar()


I am working on the file "Sessions/Session12/Day3/NotebookIII_part3_overview_discrete_3d.ipynb".
The first task is:
Doesn't look like much yet, but we can probably fix that!  Let's move our geometry into a scene, so we can control it a bit more.
Can you write Python code for it?

scene = pythreejs.Scene(children = [pp, pythreejs.AmbientLight(color='#FFFFFF')], background=None)
cam = pythreejs.PerspectiveCamera(position = [0,0,1], up = [1,0,0])
renderer = pythreejs.Renderer(width = 800, height = 800, camera = cam, scene=scene, alpha=True, controls=[pythreejs.OrbitControls(controlling=cam)])
display(renderer)


I am working on the file "Sessions/Session12/Day3/NotebookIII_part3_overview_discrete_3d.ipynb".
The first task is:
Now let's rotate this around.  We'll use an ipywidgets.interact like we did before, but this time we'll also allow for subsetting so that we can see our updates a little faster.
Can you write Python code for it?

@ipywidgets.interact(phi = (-np.pi, np.pi, 0.01), theta = (-np.pi, np.pi, 0.01), psi = (-np.pi, np.pi, 0.01), subset = [1, 10, 100, 1000])
def make_plot(phi, theta, psi, subset = 1):
    rotated = recentered[::subset] @ rotation_matrix(phi, theta, psi)
    plt.hexbin(rotated[:,0], rotated[:,1], part_gas["Masses"], reduce_C_function=np.sum, norm=mcm.LogNorm(), linewidth=0.2)
    plt.colorbar()


I am working on the file "Sessions/Session12/Day3/NotebookIII_part3_overview_discrete_3d.ipynb".
The first task is:
Now, we're going to open it using h5py and load in a handful of fields.
Can you write Python code for it?

part_dm = {}
part_gas = {}
with h5py.File("GadgetDiskGalaxy/snapshot_200.hdf5", "r") as f:
    part_gas["Coordinates"] = f["/PartType0/Coordinates"][()]
    part_gas["Masses"] = f["/PartType0/Masses"][()]
    part_gas["Density"] = f["/PartType0/Density"][()]
    part_gas["Velocities"] = f["/PartType0/Velocities"][()]
    part_gas["SmoothingLength"] = f["/PartType0/SmoothingLength"][()]
    part_dm["Coordinates"] = f["/PartType1/Coordinates"][()]
    part_dm["Velocities"] = f["/PartType1/Velocities"][()]


I am working on the file "Sessions/Session12/Day3/NotebookIII_part3_overview_discrete_3d.ipynb".
The first task is:
Our first order of business: let's try making a density plot!  There are, as you can probably tell, far too many particles to simply make a scatter plot.  So let's try hexbinning the data.
Can you write Python code for it?

plt.hexbin(part_gas["Coordinates"][:,0], part_gas["Coordinates"][:,1], part_gas["Masses"], reduce_C_function=np.sum, norm=mcm.LogNorm())
plt.colorbar()


I am working on the file "Sessions/Session12/Day3/NotebookIII_part3_overview_discrete_3d.ipynb".
The first task is:
So far it doesn't look like much!  But, during the problems section, we'll dig deeper into how we can change this.
For now, let's just satisfy ourselves by making it so that our coordinates are all not only centered, but scaled.  It's a fair bit easier to maneuver pythreejs if the values span roughly unitary.
Can you write Python code for it?

max_extent = (recentered.max(axis=0) - recentered.min(axis=0)).max()
rescaled = recentered / max_extent


I am working on the file "Sessions/Session12/Day3/NotebookIII_part3_overview_discrete_3d.ipynb".
The first task is:
ThreeJS
Let's try plotting some of this in pythreejs.
Can you write Python code for it?

pg = pythreejs.BufferGeometry(
            attributes = dict(
                    position = pythreejs.BufferAttribute(array=recentered, normalized=False),
                    value = pythreejs.BufferAttribute(array=part_gas["Masses"], normalized=False)
            )
)
pm = pythreejs.PointsMaterial()

pp = pythreejs.Points(geometry=pg, material=pm)

pp


I am working on the file "Sessions/Session12/Day3/NotebookIII_part3_overview_discrete_3d.ipynb".
The first task is:
We can update our pythreejs values inline by setting values on the properties.  But, we have to do it in a particular way to make it notice and update our display.  Note that we should also probably rescale our sizes, since they're an awful lot bigger (relatively) now!
Can you write Python code for it?

pg.attributes["position"].array = rescaled
pm.size = 1.0/max_extent


I am working on the file "Sessions/Session12/Day3/NotebookIII_part3_overview_discrete_3d.ipynb".
The first task is:
We won't duplicate here the process of making these images from different axis-aligned angles, like we did with the regular mesh data.  Instead, let's take a look at how we might apply some changes thaUnfortunately, getting the edges exactly right is kind of a hard thing so it's going to require a bit of work and fiddling.
With discrete data, unfortunately, it can be quite difficult to get the visualizations of extended particles exactly right.  There's a fair bit of formalism that goes into Lagrangian hydrodynamics, and often it is formulated with Smoothed Particle Hydrodynamics (SPH).  In SPH, the particles are sampling points of a function, but they have extended influence
$${\displaystyle A({\boldsymbol {r}})=\int A\left({\boldsymbol {r^{\prime }}}\right)W(|{\boldsymbol {r}}-{\boldsymbol {r^{\prime }}}|,h)\mathrm {d} V\left({\boldsymbol {r^{\prime }}}\right)}$$
Usually this is done with a kernel that cuts off at some point, to avoid convolving over the entire domain.  The mechanisms by which smoothing kernels are defined take a couple different forms, including strict cutoffs at a given number of particles, an iterative calculation of the number of nearest neighbors to compute, and with different radius-dependent functions to define contributions.  You'll usually see it written in some form of:
$${\displaystyle A({\boldsymbol {r}})=\sum {j}V{j}A_{j}W(|{\boldsymbol {r}}-{\boldsymbol {r}}_{j}|,h)}$$
In this case, $V_{j}$ is the volume, which we usually rewrite as the mass divided by the density.  (You might have noticed above that we loaded both the masses and the densities for our gas particles.)
A proper course on SPH would go into considerable detail about the choices of smoothing kernels, but for our purposes we will use the smoothing length loaded from the file, along with a cubic spline kernel; keep in mind that different locations define the smoothing length differently, as half-length or full-length.  You can find a good amount of detail on formalism about visualizing SPH particles in Daniel...
Can you write Python code for it?

def rotation_matrix(phi, theta, psi):
    s_phi, c_phi = np.sin(phi), np.cos(phi)
    s_theta, c_theta = np.sin(theta), np.cos(theta)
    s_psi, c_psi = np.sin(psi), np.cos(psi)
    Rx = np.array([[1, 0, 0], [0, c_phi, -s_phi], [0, s_phi, c_phi]])
    Ry = np.array([[c_theta, 0, s_theta], [0, 1, 0], [-s_theta, 0, c_theta]])
    Rz = np.array([[c_psi, -s_psi, 0], [s_psi, c_psi, 0], [0, 0, 1]])
    return Rz@Ry@Rx


I am working on the file "Sessions/Session13/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
By eye (a metric that is hardly quantitative, but nevertheless worth developing because talks never provide all of the details), the quadratic fit appears "better" than the linear fit.
But, there are still "outliers" and in the realm of polynomial fitting, it is always possible to get a better fit by adding more degrees to the polynomial. Should we keep going here, or should we stop? (Again - we will discuss model selection with Adam on Friday)
[As a reminder - in machine learning we'd call this low training error, but the generalization error is likely huge]
How should we deal with these potential outliers?
(and to re-iterate, we cannot be certain that these points are, in fact, outliers)
Amazingly, if you scroll through the literature you can find solutions like the following: "We do not believe the data point at (x, y) for reasons A, B, C. Thus, we exclude that point from the fit."
This, obviously, lacks any sort of rigor. If data is going to be removed, and it is worth asking if data should ever be removed, it should not be subject to the "beliefs" of an individual person or group.
A more common approach that you might encounter is known as $k\sigma$ clipping, which is an iterative procedure to identify and remove outliers from a data set. The procedure is as follows:

Fit the model to the data
Identify any data points that are $k\sigma$ discrepant from the best-fit model
Remove the discrepant points, repeat steps 1 & 2 until there are no data beyond $k\sigma$

The motivation for this procedure is the following: in a small data set (such as the one above, 20 points), the likelihood of having large $\sigma$ deviations (let's say $k = 5$) is vanishingly small. Thus, it "makes sense" to remove those points.
Of course, this only makes sense if the uncertainties are truly gaussian with low variance. So again, specific, and likely untrue, assumptions have to be made.
Problem 3c
Develop a $k\sigma$ clipping procedure to fit a line to the data set. Set $k = 5$ and determine the best-fit line to the data....
Can you write Python code for it?


I am working on the file "Sessions/Session13/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
Probelm 2) Fitting a Line to Data
There is a very good chance, though I am not specifically assuming anything, that upon making the previous plot you had a thought along the lines of "these points fall on a line" or "these data represent a linear relationship."
Problem 2a
Is the assumption of linearity valid for the above data?
Is it convenient?
Take a few min to discuss this with your partner
Solution 2a
write your answer here
Let us proceed with convenience and assume the data represent a linear relationship. In that case, in order to make predictions for future observations, we need to fit a line to the data.
The "standard" proceedure for doing so is least-squares fitting. In brief, least-squares minimizes the sum of the squared value of the residuals between the data and the fitting function.
I've often joked that all you need to be a good data scientist is google and stack overflow. Via those two tools, we can quickly deduce that the easiest way to perform a linear least-squares fit to the above data is with np.polyfit, which performs a least-squares polynomial fit to two numpy arrays.
Problem 2b
Use np.polyfit() to fit a line to the data. Overplot the best-fit line on the data.
Can you write Python code for it?

p = np.polyfit( # complete

plt.scatter( # complete
# complete


I am working on the file "Sessions/Session13/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
Problem 1c
[You may have already done this] Now that we understand the origin of the data, make a scatter plot showing their distribution.
Can you write Python code for it?

plt.scatter( # complete
# complete
# complete


I am working on the file "Sessions/Session13/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
Problem 2f
Confirm the results of this fit are the same as those from np.polyfit.
Hint - be sure to include the uncertainties.
Can you write Python code for it?

p = np.polyfit( # complete
print("The best-fit value for the slope and intercept are: {:.4f} and {:.4f}".format( # complete


I am working on the file "Sessions/Session13/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
Problem 3) Are the Uncertainties Actually Gaussian?
Previously we noted that there are two essential assumptions that are required for least-squares fitting to be correct. We are now going to examine the latter requirement, namely, that the uncertainties can be perfectly described as Gaussians with known variance.
Earlier I stated this assumption is rarely satisfied. Why might this be the case?
In my experience (meaning this is hardly universal), if it's astro, it's got systematics. While I cannot prove this, I contend that systematic uncertainties are rarely Gaussian. If you are lucky enough to be in a regime where you can be confident that the systematics are Gaussian, I further contend that it is extremely difficult to be certain that the variance of that Gaussian is known.
Then there's another (astro-specific) challenge: in many circumstances, we aren't actually working with data, but rather with the results of other models applied to the data.
Let's take an optical astronomy (biased, but this is LSST after all) example. What are the data? In many cases inference is being based on measurements of brightness, but the true data in this case is simply a bunch of electron counts in a CCD. The brightness (or mag) is based on the application of a model (e.g., PSF, aperture, Kron) that is applied to the data. Thus, to assume that a flux (or mag) measurement has Gaussian uncertainties with known variance is to assume that whatever flux-measurement model has been applied always produces perfectly Gaussian uncertainties (and a lot of different assumptions go into flux-measurement models...)
As a demonstration associated with the challenges of these assumptions, we will examine the data set presented in Hogg, Bovy, & Lang.
Can you write Python code for it?


I am working on the file "Sessions/Session13/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
Problem 3a
Using the least-squares methodology developed in Problem 2, determine the best-fit slope and intercept for a line fit to the data above.
Make a scatter plot of the data, and overplot the best-fit line. What if anything, do you notice about the data and the fit?
Can you write Python code for it?

Y = # complete
# complete

# complete

plt.errorbar( # complete
plt.plot( # complete

print("The best-fit value for the slope and intercept are: {:.4f} and {:.4f}".format( # complete


I am working on the file "Sessions/Session13/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
So we have now uncovered one of the peculiariaties of least-squares. Fitting y vs. x is not the same as fitting x vs. y.
There are a couple essential assumptions that go into standard least-squares fitting:

There is one dimension along which the data have negligible uncertainties
Along the other dimension all of the uncertainties can be described via Gaussians of known variance

These two conditions are rarely met for astronomical data. While condition 1 can be satisfied (e.g., time series data where there is essentially no uncertainty on the time of the observations), I contend that condition 2 is rarely, if ever, satisfied.
Speaking of uncertainties(1), we have not utilized any thus far. [I hope this has raised some warning bells.]
We will now re-organize our data to match what is originally in Hogg, Bovy, & Lang (previously x and y were swapped).
(1) There is an amazing footnote in Hogg, Bovy, & Lang about "errors" vs. "uncertainties" - I suggest everyone read this.
Problem 2d
Re-plot the data including the uncertatines.
Can you write Python code for it?


I am working on the file "Sessions/Session13/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
The Assumptions of Least Squares
========
Version 0.1

By AA Miller (Northwestern/CIERA)
2018 Nov 04
Today we will focus on a relatively simple problem, while highlighting several challenges for the standard astronomical workflow, as a way of setting up the various lectures that will happen over the course of this week.
A lot of the lessons in this lecture are inspired by the paper Data Analysis Recipes: Fitting a Model to Data by Hogg, Bovy, & Lang. [This paper has been mentioned previously in the DSFP, though today we will only be able to scratch the surface of its content.]
In some sense - the goal right now is to make you really nervous about the work that you've previously done.
(Though this lecture should not be met with too much consternation –– we will discuss new approaches to fitting a line)
Problem 1) Data
At the core of everything we hope to accomplish with the DSFP stands a single common connection: data.
There are many things we (may) want to do with these data: reduce them, visualize them, model them, develop predictions from them, use them to infer fundamental properties of the universe (!).
Before we dive into that really fun stuff, we should start with some basics:
Problem 1a
What is data?
Take a few min to discuss this with your partner
Solution 1a
write your answer here
Problem 1b
Below, I provide some data (in the form of numpy arrays). As good data scientists, what is the first thing you should do with this data?
Feel free to create a new cell if necessary.
Can you write Python code for it?


I am working on the file "Sessions/Session13/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
There is a very good chance, though, again, I am not specifically assuming anything, that for the previous plots that you plotted x along the abscissa and y along the ordinate.
[Honestly, there's no one to blame if this is the case, this has essentially been drilled into all of us from the moment we started making plots. In fact, in matplotlib we cannot change the name of the abscissa label without adjusting the xlabel.]
This leads us to an important question, however. What if y does not depend on x and instead x depends on y? Does that in any way change the results for the fit?
Problem 2c
Perform a linear least-squares fit to x vs. y (or if you already fit this, then reverse the axes). As above, plot the data and the best-fit model.
To test if the relation is the same between the two fits, compare the predicted y value for both models corresponding to x = 50.
Can you write Python code for it?

p_yx = np.polyfit( # complete

plt.scatter( # complete
# complete

print("For y vs. x, then x=50 would predict y={:.2f}".format( # complete
print("For x vs. y, then x=50 would predict y={:.2f}".format( # complete


I am working on the file "Sessions/Session13/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
Unlike the data in Problems 1 and 2, there appear to be some significant outliers (of course - this appearance of outliers is entirely dependent upon the assumption of linearity, there may actually be no outliers and a complex relation between x and y). As such, it does not appear (to me) as though the best-fit line provides a good model for the data.
Problem 3b
Perform a least-squares 2nd order polynomial fit to the data. Overplot the bestfit curve.
How does this compare to the linear model fit?
Can you write Python code for it?

Y = # complete
# complete

# complete

plt.errorbar( # complete
plt.plot( # complete


I am working on the file "Sessions/Session13/Day0/LeastSquaresAssumptions.ipynb".
The first task is:
We are now assuming that x has negligible uncertainties and that y has uncertainties that can be perfectly described by Gaussians of known variance.
A portion of the appeal of least-squares is that it provides a deterministic method for determining the best fit. To understand that we now need to do a little linear algebra.
We can arrange the data in the following matricies:
$$ \mathbf{Y} = \left[ {\begin{array}{c}
            y_1 \
            y_2 \
            \dots \
            y_N
            \end{array}
           }
            \right] , $$
$$ \mathbf{A} = \left[ {\begin{array}{cc}
            1 & x_1 \
            1 & x_2 \
            \dots & \dots \
            1 & x_N
            \end{array}
           }
           \right] ,
           $$
$$ \mathbf{C} = \left[ {\begin{array}{cccc}
            \sigma_{y_1}^2 & 0 & \dots & 0 \
            0 & \sigma_{y_2}^2 & \dots & 0 \
            \vdots & \vdots & \ddots & \vdots \
            0 & 0 & \dots & \sigma_{y_1}^2 \
            \end{array}
           }
           \right] ,
           $$
where $\mathbf{Y}$ is a vector, and $\mathbf{C}$ is the covariance matrix.
Ultimately, we need to solve the equation
$$\mathbf{Y} = \mathbf{A}\mathbf{X}.$$
I am skipping the derivation, but the solution to this equations is:
$$ \left[ {\begin{array}{c}
            b \
            m \
            \end{array}
           }
            \right] = \mathbf{X} = \left[ \mathbf{A}^T \mathbf{C}^{-1} \mathbf{A}\right]^{-1} \left[ \mathbf{A}^T \mathbf{C}^{-1} \mathbf{Y}\right].$$
As noted in Hogg, Bovy, & Lang, this procedure minimizes the $\chi^2$ function, which is the total squared error, after appropriately scaling by the uncertainties:
$$ \chi^2 = \Sigma_{i = 1}^{N} \frac{[y_i - f(x_i)]^2}{\sigma_{y_i}^2} = \left[ \mathbf{Y}  - \mathbf{A}\mathbf{X}\right]^{T} \mathbf{C}^{-1} \left[ \mathbf{Y} - \mathbf{A} \mathbf{X}\right].$$
Problem 2e
Using the linear algebra equations above (i.e. avoid np.polyfit or any other similar functions), determine the weighted least-squares best-fit...
Can you write Python code for it?

Y = # complete
A = # complete
C = # complete

X = # complete

best_fit = # complete

plt.errorbar( # complete
plt.plot( # complete

print("The best-fit value for the slope and intercept are: {:.4f} and {:.4f}".format( # complete


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
Even better - there is an option to create a PairGrid which allows fine tuned control of the data as displayed above, below, and along the diagonal. In this way it becomes possible to avoid having symmetric redundancy, which is not all that informative. In the example below, we will show scatter plots and contour plots simultaneously.
Can you write Python code for it?

g = sns.PairGrid(iris, vars = ["sepal_length", "sepal_width", "petal_length", "petal_width"],
                 hue = "species", diag_sharey=False)
g.map_lower(sns.kdeplot)
g.map_upper(plt.scatter, edgecolor='white')
g.map_diag(sns.kdeplot, lw=3)
g.add_legend()


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
Here, we see that there are many points, clustered about the origin, but we have no sense of the underlying density of the distribution. 2D histograms, such as plt.hist2d(), can alleviate this problem. I prefer to use plt.hexbin() which is a little easier on the eyes (though note - these histograms are just as subject to the same issues discussed above).
Can you write Python code for it?

# hexbin w/ bins = "log" returns the log of counts/bin
# mincnt = 1 displays only hexpix with at least 1 source present
fig, ax = plt.subplots()
cax = ax.hexbin(xexample, yexample, bins = "log", cmap = "viridis", mincnt = 1)
ax.set_xlabel('X', fontsize=14)
ax.set_ylabel('Y', fontsize=14)
fig.tight_layout()
plt.colorbar(cax)


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
It turns out blocks are not an ideal representation for continuous data (see discussion on histograms above). Now we will explore the resulting PDF from other kernels.
Problem 1g Plot the KDE of the PDF for the number of chinups middle aged men can do using a gaussian and Epanechnikov kernel. How do the results differ from the histogram plots above?
Hint - you will need to select the bandwidth. The examples above should provide insight into the useful range for bandwidth selection. You may need to adjust the values to get an answer you "like."
Can you write Python code for it?

PDFgaussian = # complete
PDFepanechnikov = # complete

fig, ax = plt.subplots()
ax.plot(# complete
ax.plot(# complete
ax.legend(loc = 2)
ax.set_xlabel('chinups', fontsize=14)
ax.set_ylabel('PDF', fontsize=14)
fig.tight_layout()


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
Now that we have a sense of the data structure, it is useful to examine the distribution of features. Above, we went to great pains to produce histograms, KDEs, and rug plots. seaborn handles all of that effortlessly with the displot function.
Problem 2b
Plot the distribution of petal lengths for the Iris data set.
Can you write Python code for it?

# note - kde, and rug all set to True, set to False to turn them off
with sns.axes_style("dark"):
    sns.displot(iris['petal_length'], bins=20,
                kde=True, rug=True)
plt.tight_layout()


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
Now that we have covered the basics of seaborn (and the above examples truly only scratch the surface of what is possible), we will explore the power of seaborn for higher dimension data sets. We will load the famous Iris data set, which measures 4 different features of 3 different types of Iris flowers. There are 150 different flowers in the data set.
Note - for those familiar with pandas seaborn is designed to integrate easily and directly with pandas DataFrame objects. In the example below the Iris data are loaded into a DataFrame. iPython notebooks also display the DataFrame data in a nice readable format.
Can you write Python code for it?


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
Of course, when there are many many data points, scatter plots become difficult to interpret. As in the example below:
Can you write Python code for it?

np.random.seed(2016)
xexample = np.random.normal(loc = 0.2, scale = 1.1, size = 10000)
yexample = np.random.normal(loc = -0.2, scale = 0.9, size = 10000)

fig, ax = plt.subplots()
ax.scatter(xexample, yexample)
ax.set_xlabel('X', fontsize=14)
ax.set_ylabel('Y', fontsize=14)
fig.tight_layout()


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
Problem 1a
Plot the histogram for the number of chinups using the default settings in pyplot.
Can you write Python code for it?

fig, ax = plt.subplots()
ax.hist( # complete
ax.set_xlabel('chinups', fontsize=14)
ax.set_ylabel('N', fontsize=14)
fig.tight_layout()


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
Seaborn
Seaborn is a plotting package that enables many useful features for exploration. In fact, a lot of the functionality that we developed above can readily be handled with seaborn.
To begin, we will make the same plot that we created in matplotlib.
Can you write Python code for it?

import seaborn as sns

fig, ax = plt.subplots()

ax.plot(x,y, lw = 2)
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_xlim(0, 6*np.pi)
fig.tight_layout()


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
So, what is the optimal choice of bandwidth and kernel? Unfortunately, there is no hard and fast rule, as every problem will likely have a different optimization. Typically, the choice of bandwidth is far more important than the choice of kernel. In the case where the PDF is likely to be gaussian (or close to gaussian), then Silverman's rule of thumb can be used:
$$h = 1.059 \sigma n^{-1/5}$$
where $h$ is the bandwidth, $\sigma$ is the standard deviation of the samples, and $n$ is the total number of samples. Note - in situations with bimodal or more complicated distributions, this rule of thumb can lead to woefully inaccurate PDF estimates. The most general way to estimate the choice of bandwidth is via cross validation (we will cover cross-validation during the session on Machine Learning).
What about multidimensional PDFs? It is possible using many of the Python implementations of KDE to estimate multidimensional PDFs, though it is very very important to beware the curse of dimensionality in these circumstances.
Problem 2) Data Exploration
Now a more open ended topic: data exploration. In brief, data exploration encompases a large suite of tools (including those discussed above) to examine data that live in large dimensional spaces. There is no single best method or optimal direction for data exploration. Instead, today we will introduce some of the tools available via python.
As an example we will start with a basic line plot - and examine tools beyond matplotlib.
Can you write Python code for it?

x = np.arange(0, 6*np.pi, 0.1)
y = np.cos(x)

fig, ax=plt.subplots()
ax.plot(x,y, lw = 2)
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_xlim(0, 6*np.pi)
fig.tight_layout()


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
Something is wrong here - the choice of bin centers and number of bins suggest that there is a 0% probability that middle aged men can do 10 chinups. This is intuitively incorrect; we will now adjust the bins in the histogram.
Problem 1b
Using the same data make 2 new histograms: (i) one with 5 bins (bins = 5), and (ii) one with the bars centered on the left bin edges (align = "left").
Hint - if overplotting the results, you may find it helpful to use the histtype = "step" option
Can you write Python code for it?

fig, ax = plt.subplots()
ax.hist(# complete
ax.hist(# complete
ax.set_xlabel('chinups', fontsize=14)
ax.set_ylabel('N', fontsize=14)
fig.tight_layout()


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
That is much nicer than what was presented above. However - we still have a problem in that our data live in 4D, but we are (mostly) limited to 2D projections of that data. One way around this is via the seaborn version of a pairplot, which plots the distribution of every variable in the data set against each other. (Here is where the integration with pandas DataFrames becomes so powerful.)
Can you write Python code for it?

sns.pairplot(iris[["sepal_length", "sepal_width",
                   "petal_length", "petal_width"]])
plt.tight_layout()


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
The folks behind seaborn have thought a lot about color palettes, which is a good thing. Remember - the choice of color for plots is one of the most essential aspects of visualization. A poor choice of colors can easily mask interesting patterns or suggest structure that is not real. To learn more about what is available, see the seaborn color tutorial.
Here we load the default:
Can you write Python code for it?

# default color palette

current_palette = sns.color_palette()
sns.palplot(current_palette)


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
These small changes significantly change the estimator for the PDF. With fewer bins we get something closer to a continuous distribution, while shifting the bin centers reduces the probability to zero at 9 chinups.
What if we instead allow the bin width to vary and require the same number of points in each bin? You can determine the bin edges for bins with 5 sources using the following command:
bins = np.append(np.sort(chinups)[::5],
                  np.max(chinups))
Problem 1c
Plot a histogram with variable width bins, each with the same number of points.
Hint - setting density = True will normalize the bin heights so that the PDF integrates to 1.
Can you write Python code for it?

bins = np.append(# complete
fig, ax = plt.subplots()
ax.hist( # complete
ax.set_xlabel('chinups', fontsize=14)
ax.set_ylabel('N', fontsize=14)
fig.tight_layout()


I am working on the file "Sessions/Session13/Day0/TooBriefVisualization.ipynb".
The first task is:
which we will now change to colorblind, which is clearer to those that are colorblind.
Can you write Python code for it?

# set palette to colorblind
sns.set_palette("colorblind")

current_palette = sns.color_palette()
sns.palplot(current_palette)


I am working on the file "Sessions/Session13/Day2/01-Introduction-to-GPs.ipynb".
The first task is:

Exercise 2

Use the function above to draw 10,000 samples from a zero-mean Gaussian with covariance
$$
\mathbf{\Sigma} =
\begin{pmatrix}
 1 & 0.5 \
 0.5 & 1
\end{pmatrix}
$$
Plot the "corner" plot for your samples using the corner package (!pip install corner):
python
from corner import corner
fig = corner(samples);
Vary the terms in the covariance matrix (recalling that it must be symmetric!) to get a sense of how they affect the joint distribution of samples.
Parametrizing the covariance
As the number of dimensions increases, and in particular as it becomes infinite (which it will, when we get to actual Gaussian Processes), it's no longer convenient to describe the covariance in terms of every single entry in the covariance matrix. Instead, it's useful to introduce the notion of a kernel, a function of just a few parameters that describes the overall structure of the covariance matrix.
In general, the covariance matrix can have any structure, but quite often in timeseries analysis data points close to each other in time are strongly correlated. This is true whether your timeseries contains photometric variability from a rotating star, the rise and fall of a supernova light curve, a gravitational wave signal, or PSF shape changes due to temperature fluctuations on the detector. This is because each of these processes have characteristic timescales over which they operate, and on timescales shorter than that, all measurements you make are likely to be close to relatively each other. But if you wait long enough, what your system will do down the line will be pretty decoupled from what it is doing right now.
So we can imagine defining a kernel that looks something like this:

where the covariance peaks at zero time lag and drops smoothly to zero as the time lag increases. This particular kernel is extremely useful and has a name: the...
Can you write Python code for it?

def ExpSquaredCovariance(t, A=1.0, l=1.0, tprime=None):
    """
    Return the ``N x M`` exponential squared
    covariance matrix.

    """
    if tprime is None:
        tprime = t
    TPrime, T = np.meshgrid(tprime, t)
    return A ** 2 * np.exp(-0.5 * (T - TPrime) ** 2 / l ** 2)


I am working on the file "Sessions/Session13/Day2/01-Introduction-to-GPs.ipynb".
The first task is:
As I mentioned in my lecture, in a single dimension, the probability density function for a Gaussian with mean $\mu$ and variance $\sigma^2$ is
$$
p(y | \mu, \sigma^2) =
\frac{1}{\sqrt{2\pi\sigma^2}}
\exp\left[
-\frac{1}{2}\left(\frac{y - \mu}{\sigma}\right)^2
\right]
$$
In Exercise 1 below, we'll derive the expression for a Gaussian in $N$ dimensions. We'll do it for the case where there isn't any covariance across the $N$ dimensions, in which case the probability density is obtained by multiplying the expression above $N$ times.

Exercise 1

Show (numerically) that the probability function for an uncorrelated Gaussian in $N$ dimensions
$$
p(\mathbf{y} | \boldsymbol{\mu}, \mathbf{\Sigma}) =
\prod_{n=0}^{N-1}
\frac{1}{\sqrt{2\pi\sigma_n^2}}
\exp\left[
-\frac{1}{2}\left(\frac{y_n - \mu_n}{\sigma_n}\right)^2
\right]
$$
can be written in vector form as
$$
p(\mathbf{y} | \boldsymbol{\mu}, \mathbf{\Sigma}) =
\frac{1}{\sqrt{(2\pi)^N \mathrm{det}(\mathbf{\Sigma})}}
\exp\left[
-\frac{1}{2}
(\mathbf{y} - \boldsymbol{\mu})^\top
\mathbf{\Sigma}^{-1}
(\mathbf{y} - \boldsymbol{\mu})
\right]
$$
where
$$
\mathbf{\Sigma} =
\begin{pmatrix}
 \sigma_0^2 & 0 & 0 \
 0 & \sigma_1^2 & 0 \
 0 & 0 & \ddots
\end{pmatrix}
$$
is a diagonal covariance matrix. You can use the following random inputs to test the equivalence of the two expressions:
Can you write Python code for it?


I am working on the file "Sessions/Session13/Day2/01-Introduction-to-GPs.ipynb".
The first task is:
Don't worry about sigma yet -- we'll discuss this below.

Exercise 7

Given
python
t_train = np.array([2.5, 7.5])
y_train = np.array([1.0, -1.0])
t_test = np.linspace(0, 10, 1000)
compute the mean and covariance of the GP. Draw 30 samples from it as we did above. Plot the mean function and shade the 1- and 2-$\sigma$ levels. Comment on the behavior of the GP. Does the GP correctly predict the training points? How does the variance behave close to and far from the training points? How does the GP behave very far from the training points?
The last thing we'll do regarding drawing from a GP is to account for noise in the observations. If the measurement error on the data points is Gaussian and uncorrelated (white), as we usually assume it to be, we simply add a term to the joint distribution of the training and test data:
$$
\begin{align}
\begin{pmatrix}
 y \
 y_\star
\end{pmatrix}
&=
\mathcal{N}
\left[
\mathbf{0},
\begin{pmatrix}
 \mathbf{\Sigma}(t, t) + \sigma_n^2 \mathbf{I} & \mathbf{\Sigma}(t, t_\star)\
 \mathbf{\Sigma}(t_\star, t) & \mathbf{\Sigma}(t_\star, t_\star)
\end{pmatrix}
\right]
\end{align}
$$
where $\sigma_n$ is the standard deviation for the $n^\mathrm{th}$ data point in the training set and $\mathbf{I}$ is the identiy matrix. If you look at our function compute_gp above, you'll see that it accepts a sigma keyword, and that it adds its square to the $\mathbf{\Sigma}(t, t)$ term in the covariance, as shown above.

Exercise 8

Re-plot the figure from Exercise 7, this time given an observational uncertainty $\sigma = 0.25$ at both training points. Comment on the new behavior of the GP.
GP optimization
In the previous section, we learned how to construct and sample from a simple GP. This is useful for making predictions, i.e.,...
Can you write Python code for it?

def ln_gp_likelihood(t, y, sigma=0, A=1.0, l=1.0):
    """
    Return the log of the GP likelihood for a datatset y(t)
    with uncertainties sigma, modeled with a Squared Exponential
    Kernel with amplitude A and lengthscale l.

    """
    # The covariance and its determinant
    npts = len(t)
    K = ExpSquaredCovariance(t, A=A, l=l) + sigma ** 2 * np.eye(npts)

    # The log marginal likelihood
    log_like = -0.5 * np.dot(y.T, np.linalg.solve(K, y))
    log_like -= 0.5 * np.linalg.slogdet(K)[1]
    log_like -= 0.5 * npts * np.log(2 * np.pi)

    return log_like


I am working on the file "Sessions/Session13/Day2/01-Introduction-to-GPs.ipynb".
The first task is:
Don't worry about the tprime keyword for now -- we'll use it later in Exercise 7.

Exercise 3

For t = np.linspace(0, 2, 11), draw 10,000 samples from the Gaussian described by the covariance matrix above and plot the corresponding corner plot, labeling each subplot with the time point it represents. Comment on the structure of the plot and the correlations among the different dimensions.
From Gaussians to Gaussian Processes
So far we've been taking about Gaussians distributions and how they can jointly model a few random variables at a time. But what happens as the dimensionality of the Gaussian increases and eventually becomes infinite? The Gaussian no longer represents a collection of random variables, but instead the behaviour of a continuous function. And that's exactly what a Gaussian Process is: a distribution over functions with infinitely many points. We can't technically model a function with infinitely many points on a computer, but we can investigate what happens as the number of points becomes very large.

Exercise 4

Construct a 1000 x 1000 covariance matrix using the squared exponential kernel for a timeseries spanning 10 time units, and plot an image of it using plt.imshow. Comment on the structure of the matrix.
Below, we draw 30 samples from the zero-mean Gaussian described by the covariance matrix you plotted in the previous exercise. These are shown as the transluscent black curves. The mean of the process is the solid blue line, and the 1 and 2 sigma levels are indicated with the blue shading.
Can you write Python code for it?

t = np.linspace(0, 10, 1000)
mu = np.zeros_like(t)
S = ExpSquaredCovariance(t, A=1.0, l=1.0)
np.random.seed(1)
for i in range(30):
    samples = draw_from_gaussian(mu, S)
    plt.plot(t, samples.T, color="k", alpha=0.1)

mu_ = np.zeros_like(t)
plt.plot(t, mu_, color="C0")
plt.xlabel("time")
plt.ylabel("GP")
std_ = np.sqrt(S.diagonal())
plt.fill_between(t, mu_ - std_, mu_ + std_, color="C0", alpha=0.3);
plt.fill_between(t, mu_ - 2 * std_, mu_ + 2 * std_, color="C0", alpha=0.15);