I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Looks to be the same file... But not a few differences: it's immediately an image, with no need to do [0].  Also note that you had to specify a unit.  Some fits files come with their units specified (allowing fits fulls of e.g. calibrated images to know what their units are), but this one is raw data, so we had to specify the unit.
Note also that CCDDatathe object knows all the header information, and has a copy of the data just as in the astropy.io.fits interface:
Can you write Python code for it?

ccddata_g.meta  # ccddata_g.header is the exact same thing

ccddata_g.data


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Not too much exciting there to the eye... but it should have better noise properties than any of the biases by themselves.  Now lets do the final step in the bias process and try subtracting the bias from the science image.  Lets look at the docstring for ccdproc.subtract_bias
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Now it is clear that the overscan region is quite different from the rest of the image.
Subtracting overscan
Fortunately, subtracting the overscan region is fairly trivial with the ccdproc package.  It has a function to do just this task.  Take a look at the docs for the function by executing the cell below.
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Now sanity-check that they are actually the correct objects:
Can you write Python code for it?

plt.figure(figsize=(8, 16))
plt.imshow(srcs, cmap=srcs.cmap('#222222'), origin='lower')

plt.scatter(x[msk], y[msk], c='r', s=150, marker='*')

plt.xlim(600, 800)
plt.ylim(330+2200, 720+2200)


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Now we both load these images into memory and apply the overscan correction in a single step.
Can you write Python code for it?

biases = [overscan_correct(CCDData.read(fn, unit=u.count)) for fn in biasfns]

# The above cell uses Python's "list comprehensions", which are faster and more
# compact than a regular for-loop.  But if you have not seen these before,
# it's useful to know they are exactly equivalent to this:

# biases = []
# for fn in biasfns:
#     im = overscan_correct(CCDData.read(fn, unit=u.count))
#     biases.append(im)


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Now we load these files like we did the biases, applying both the overscan and bias corrections:
Can you write Python code for it?

# These steps could all be one single list comprehension. However,
# breaking them into several lines makes it much clearer which steps
# are being applied. In practice the performance difference is
#  ~microseconds, far less than the actual execution time for even a
# single image

flats_g = [CCDData.read(fn, unit=u.count) for fn in flat_g_fns]
flats_g = [overscan_correct(flat) for flat in flats_g]
flats_g = [ccdproc.subtract_bias(flat, master_bias) for flat in flats_g]
show_image(flats_g[0], 90)


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Now we're in business!  This now looks like an astronomical image of a galaxy and various objects around it, aside from some residual amp glow (discussed below in the "Advanced Exercise").
Exercise
We still haven't gotten quite rid of the glow.  It seems there is a slight residual glow in the strip on the left, and a quite prominent glow to the lower-left.
The lower-left glow is apparently time-dependent - that is, it is absent in the biases (0 sec exposure), weak in the flats (70 sec exposure), but much stronger in the science images (300 sec).  This means it is probably due to amplifier electronics that emit a continuous glow even when not reading out. That's an annoying thing about semiconductors: while they are great light absorbers but also great emmitters!  Oh well, at least in means our electricity bills are getting cheaper...
In any event, this means the only way to correct for this glow is to use a "dark" in place of a bias.  A dark is an exposure of the same time as the target exposure, but with the camera shutter closed. This exposure should then capture the full amplifier glow for a given exposure time.  You may have noticed the data files included a darks directory. If you look there you'll see several images, including dark exposures of times appropriate for our images. They have overlapping exposure  numbers and no log, because they were taken on a different night as the science data, but around the same time.  So you will have to do some sleuthing to figure out which ones to use.
Once you've figured this out, try applying the darks to the images in the same way as the biases and see if you can get rid of the remaining glow.  If you get this working, you can use those images instead of the ones derived above for the "Photometry" section.
Advanced Exercise
Due to time constraints, the above discussion has said little about uncertainties. But there is enough information in the images above to compute running per-pixel uncertainties. See if you can do this, and attach them to the final file as the...
Can you write Python code for it?

from astropy.stats import SigmaClip

bkg_estimator = photutils.ModeEstimatorBackground(sigma_clip=SigmaClip(sigma=3.))
# Note: for some versions of numpy you may need to do ``ccd_data_g_flattened.data`` in the line below
bkg_val = bkg_estimator.calc_background(ccd_data_g_flattened)
bkg_val


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Of particular note is the fits_section keyword.  This seems like it might be useful because the header already includes a FITS-style "BIASSEC" keyword.  Lets try using that:
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Several things to note here.  First, it says it has an exposure time of zero, as expected for a bias.  There is also a big glowing strip on the left that is a few hunderd counts. This is probably an example of "amplifier glow", where the readout electronics emit light (or electronic noise) that is picked up by the CCD as it reads out.  Forrunately, this pattern, along with the other patterns evident in this image, are mostly static, so they can be removed by simply subtracting the bias.
Another thing that might not be obvious to the untrained eye is that the image dimensions are sllightly larger than a power-of-two.  This is not the size of the actual detector.  Another hint is in the header in the section called BIASSEC: this reveals the section of the chip that is used for "overscan" - the readout process is repeated several times past the end of the chip to characterize the average bias for that row.  Lets take a closer look at the overscan area.  Note that we do this in one of the science exposures because it's not visible in the bias (which is in some sense entirely overscan):
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Simply eye-balling this histogram reveals that the background fluctuations are at the level of ~20 counts.  So if we want a 3-sigma threshold, we use 60.  We also, somewhat arbitrary, require at last 5 pixels for a source to be included:
Can you write Python code for it?

ccd_data_g_bkgsub.shape

# as above, for some numpy versions you might need a `.data` to get this to work
srcs = photutils.detect_sources(ccd_data_g_bkgsub, 60, 5)


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
The above creates a directory called  "python_imred_data" - lets examine it:
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
The header contains all the metadata about the image, while the data stores "counts" from the CCD (in the form of 16-bit integers). Seems sensible enough.  Now lets try plotting up those counts.
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
The results given are fluxes in units of counts, which we can convert to "instrumental" magnitudes and put into the table:
Can you write Python code for it?

apphot['aperture_mags'] = u.Magnitude(apphot['aperture_sum'])
apphot


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
This shows that this file is a relatively simple image - a single "HDU" (Header + Data Unit). Lets take a look at what that HDU contains:
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
We can clearly see our two objects from before standing out as distinct sources. Lets have a closer look:
Can you write Python code for it?

plt.figure(figsize=(6, 10))
plt.imshow(srcs, cmap=srcs.cmap('#222222'), origin='lower')
apertures.plot(color='red')
plt.xlim(600, 800)
plt.ylim(2530, 2920)


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
Well that looks better. It's now clear this is an astronomical image.  However, it is not a very good looking one.  It is full of various kinds of artifacts that need removing before any science can be done.  We will address how to correct these in the rest of this notebook.
Exercise
Try playing with the parameters in the ImageNormalize class.  See if you can get the image to look clearer.  The image stretching part of the docs are your friend here!
To simplify this plotting task in the future, we will make a helperfunction that takes in an image and scales it to some settings that seem to look promising.  Feel free to adjust this function to your preferences base on your results in the exercise, though!
Can you write Python code for it?

def show_image(image, percl=99, percu=None, figsize=(6, 10)):
    """
    Show an image in matplotlib with some basic astronomically-appropriat stretching.

    Parameters
    ----------
    image
        The image to show
    percl : number
        The percentile for the lower edge of the stretch (or both edges if ``percu`` is None)
    percu : number or None
        The percentile for the upper edge of the stretch (or None to use ``percl`` for both)
    figsize : 2-tuple
        The size of the matplotlib figure in inches
    """
    if percu is None:
        percu = percl
        percl = 100-percl

    norm = aviz.ImageNormalize(image, interval=aviz.AsymmetricPercentileInterval(percl, percu),
                                      stretch=aviz.LogStretch())

    fig, ax = plt.subplots(1,1, figsize=figsize)
    plt.colorbar(ax.imshow(image, norm=norm, origin='lower'))


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
While it has many options, the defaults are probably sufficient for our needs right now:
Can you write Python code for it?

apphot = photutils.aperture_photometry(ccd_data_g_bkgsub, apertures)
apphot


I am working on the file "Sessions/Session05/Day1/python_imred.ipynb".
The first task is:
You should get the SDSS and your computed difference to be within ~0.001 mags if all went well in the previous steps. Guess we're on the right track!
Exercise
The astroquery package contains a sub-package astroquery.sdss that can be used to programatically search the SDSS for objects.  See if you can use that to automate the process we followed above to compare SDSS and our image.
Note: you may need to install astroquery the same way as you did ccdproc or photutils (see the top of this notebook).
To go one step further, try using the SDSS to calibrate (at least, roughly) our measurements.  This will require identifying matching objects in the field (ideally fairly bright stars) and using them to compute the instrumental-to-$g$-band offset.  See if you get magnitudes to match the SDSS on other objects.
Source Detection using Thresholding
As with photometry, source-finding as a complex subject.  Here we overview a straightforward algorithm that photutils provides to find heterogeneous objects in an image.  Note that this is not optimal if you are only looking for stars.  photutils provides several star-finders that are better-suited for that problem (but are not covered further here).
Have a look at the options to the relevant photutils function:
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day2/GalaxyPhotometry.ipynb".
The first task is:
For this and the remaining problems, assume that the galaxy is centered at (x, y) = (128, 138), that the ellipticity (1 - b/a) is fixed at 0.3, and that $\theta = 0.8377$. [In principle we could ask you to determine each of these values, but that is outside the scope of the current problem.]
Problem 2d
Measure the mean "radial" flux of the galaxy at $a = 5$ and $a = 20$. Do these answers make sense?
Can you write Python code for it?

a5_flux = # complete
a20_flux = # complete

print("The mean flux at 5 pixels is {:.6f}".format( # complete
print("The mean flux at 20 pixels is {:.6f}".format( # complete


I am working on the file "Sessions/Session05/Day2/GalaxyPhotometry.ipynb".
The first task is:
Problem 1) Visualize the Galaxy Image Data
The simulated galaxy for today's problem comes from the Illustris Simulation, which is a large cosmological simulation of galaxy formation.
Download the fits image for our simulated galaxy: https://northwestern.box.com/s/a8qp2afv6ed9r36v0y0vyc3vf82s1bw0
(Note - The Illustris team has made simulated images of many different galaxies available.)
Problem 1a
Using your favorite tool, display an image of the galaxy using a logarithmic stretch. Do you notice anything unusual about the image?
Can you write Python code for it?

gal_im = fits.getdata( # complete

# complete
# complete

plt.colorbar()
plt.tight_layout()


I am working on the file "Sessions/Session05/Day2/GalaxyPhotometry.ipynb".
The first task is:
Problem 2) Measuring Radial Intensities
Before we can measure the radial profile of this galaxy, we need to compute the flux intensity at a given radius. By convention, this is done by computing the mean flux along an ellipse centered on the galaxy (where the "radius" in this case is the semi-major axis).
An ellipse with arbitrary centroid and rotation angle can be described as
$$ \frac{((x-h)\cos{\theta} - (y-k)\sin{\theta})^2}{a^2} + \frac{((x-h)\sin{\theta} + (y-k)\cos{\theta})^2}{b^2} = 1,$$
where $h, k, a, b, \theta$ are the x centroid, y centroid, semi-major axis, semi-minor axes, and the rotation angle of the elipse, respectively.
This can be re-written as (it's worth checking this on the board or scratch paper):
$$\left(\frac{\sin^2{\theta}}{a^2} + \frac{\cos^2{\theta}}{b^2}\right)y^2 + \left(-2\frac{((x-h)\cos{\theta} + k\sin{\theta})\sin{\theta}}{a^2} + 2\frac{((x-h)\sin{\theta} - k\cos{\theta})\cos{\theta}}{b^2}\right)y + \left(\frac{((x-h)\cos{\theta} + k\sin{\theta})^2}{a^2} + \frac{((x-h)\sin{\theta} - k\cos{\theta})^2}{b^2} - 1\right) = 0$$
Problem 2a
Write a function to solve for $y$ given $x$ and the ellipse parameters.
Hint - the function np.roots may be helpful.
Can you write Python code for it?

def y_ellipse(x, h, k, a, b, theta):

    # complete
    # complete

    # complete
    # complete
    # complete
    # complete

    return # complete


I am working on the file "Sessions/Session05/Day2/GalaxyPhotometry.ipynb".
The first task is:
Problem 2b
Plot an elipse using your equation with $h = 0$, $k = 0$, $a = 4$, $b = 2$, and $\theta = \pi/6$.
Hint - most of this code has been provided for you.
Can you write Python code for it?

x_grid = np.linspace(-3.605, 3.605, 1000)
y_vals = np.vstack((np.empty_like(x_grid), np.empty_like(x_grid)))
for x_num, x in enumerate(x_grid):
    roots = y_ellipse( # complete

    y_vals[:,x_num] = np.sort(roots)

plt.plot(np.append(x_grid, x_grid[::-1]), np.append(y_vals[0], y_vals[1][::-1]))
plt.xlim(-4, 4)
plt.ylim(-4, 4)


I am working on the file "Sessions/Session05/Day2/GalaxyPhotometry.ipynb".
The first task is:
Problem 2e
Plot the "mean radial profile" of the galaxy.
Hint - when choosing a grid of radii to measure, recall that the function does not make sub-pixel estimates of the flux.
Can you write Python code for it?

r_grid = # complete
mean_rad_prof = # complete
for # complete

plt.plot( # complete


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
0.1 Directory Set up
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
0.2 Display images
Can you write Python code for it?

def plotfits(imno):
    img = fits.open(datadir+objname+'_{0:02d}.fits'.format(imno))[0].data

    f = plt.figure(figsize=(10,12))
    #im = plt.imshow(img, cmap='hot')
    im = plt.imshow(img[480:580, 460:600], cmap='hot')
    plt.clim(1800, 2800)
    plt.colorbar(im, fraction=0.034, pad=0.04)
    plt.savefig("figure{0}.png".format(imno))
    plt.show()

numb = 1
plotfits(numb)

numb = 2
plotfits(numb)


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
1. Photometry set up
Select part of the image for ease of display
Can you write Python code for it?

partimg = fits.open(datadir+objname+'_01.fits')[0].data[480:580, 460:600]


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
1.1 Centroiding: Center of Mass
Calculate Center of Mass (CoM) defined as: $\bar{x} = \frac{\sum A_i x_i}{\sum A_i }$, $\bar{y} = \frac{\sum A_i  y_i}{\sum A_i }$.
Can you write Python code for it?

def cent_weight(n):
    """
    Assigns centroid weights
    """
    wghts=np.zeros((n),np.float)
    for i in range(n):
        wghts[i]=float(i-n/2)+0.5
    return wghts

def calc_CoM(psf, weights):
    """
    Finds Center of Mass of image
    """
    cent=np.zeros((2),np.float)
    temp=sum(sum(psf) - min(sum(psf) ))
    print(temp)

    cent[1]=sum(( sum(psf) - min(sum(psf)) ) * weights)/temp
    cent[0]=sum(( sum(psf.T) - min(sum(psf.T)) ) *weights)/temp
    return cent


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
1.2 Aperture Photometry
Science Aperture
Can you write Python code for it?

def circle(npix, r1):
    """
    Builds a circle
    """
    pup=np.zeros((npix,npix),np.int)
    for i in range(npix):
        for j in range(npix):
            r=np.sqrt((float(i-npix/2)+0.5)**2+(float(j-npix/2)+0.5)**2)
            if r<=r1:
                pup[i,j]=1
    return pup


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
1.3 Tests
a. Display image with target and comparison centers before and after centroiding
Can you write Python code for it?

im = plt.imshow(partimg, cmap='hot')
plt.clim(1800, 2800)
plt.scatter(targcen[1], targcen[0], c='g', marker='x')
plt.scatter(compcen[1], compcen[0], c='g', marker='x')
plt.scatter(tcenter[1], tcenter[0], c='b', marker='x')
plt.scatter(ccenter[1], ccenter[0], c='b', marker='x')
plt.show()


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
2. Photometry
2.1 Calculate SNR
Calculate Signal-to-Noise Ratio. CCD noise = sqrt(signal + background + dark current + read noise)
Can you write Python code for it?

def calcsnr(target, bg):
    signal = target - bg
    noise = np.sqrt(signal + bg)
    snr = signal / noise
    return snr, noise


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
2.2 Optimize photometry aperture
Can you write Python code for it?

#Try a range of aperture sizes
apertures = np.arange(1, 10, 1)

snrold = 0
for aper in apertures:
    apermask = circle(searchr*2, aper)
    targc = np.sum(apermask*newtarg) / np.sum(apermask)
    snr, noise = calcsnr(targc, targbg)
    if snr > snrold:
        bestaper = aper
        snrold = snr

snr, noise = calcsnr(targc, targbg)


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
2.3 Calculate the target's magnitude
Can you write Python code for it?

targc = circle(searchr*2, ap_r)*newtarg
targskyc = annulus(searchr*2, sky_inner, sky_outer)*newtarg
compc = circle(searchr*2, ap_r)*newcomp
compskyc = annulus(searchr*2, sky_inner, sky_outer)*newcomp

ratio = np.sum(compc)/np.sum(targc)
sigmaratio = ratio*np.sqrt((np.sum(targc)/np.sum(targskyc))**2 + (np.sum(compc)/np.sum(compc))**2)

deltamag = -2.5*np.log10(ratio)
sigmamag = 2.5*sigmaratio/(ratio*np.log(10))

refmag = 19.4
mag = refmag - deltamag
print("Measured Magnitude = {:0.3f} ± {:0.3f}".format(mag, sigmamag))


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
Aperture photometry set up
Can you write Python code for it?

searchr = 6  ## search box size
ap_r    = 2   ## aperture radius

sky_inner = 3
sky_outer = 5


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
Define new regions where the target and comparison are centered.
Can you write Python code for it?

newtarg = partimg[int(round(tcenter[0]))-searchr : int(round(tcenter[0]))+searchr, int(round(tcenter[1]))-searchr : int(round(tcenter[1]))+searchr]
newcomp = partimg[int(round(ccenter[0]))-searchr : int(round(ccenter[0]))+searchr, int(round(ccenter[1]))-searchr : int(round(ccenter[1]))+searchr]


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
Define starting values
Can you write Python code for it?

targcen = np.array([22,42])  ## target center
compcen = np.array([75,125])  ## comparison center


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
Extract values from regions
Create mask
Can you write Python code for it?

circmask = circle(searchr*2, ap_r)
annmask = annulus(searchr*2, sky_inner, sky_outer)


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
Place mask on region
Can you write Python code for it?

targaper = newtarg * circmask
compaper = newcomp * circmask


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
Sky annulus
Can you write Python code for it?

def annulus(npix, r_inner,r_outer=-1.):
    """
    Builds an annulus
    """
    pup=np.zeros((npix,npix),np.int)
    for i in range(npix):
        for j in range(npix):
            r=np.sqrt((float(i-npix/2)+0.5)**2+(float(j-npix/2)+0.5)**2)
            if ((r<=r_outer)&(r>=r_inner)):
                pup[i,j]=1
    return pup


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
Use centroiding algorithm to find the actual centers of the targe and comparison.
Can you write Python code for it?

## Cut a box between search limits, centered around targcen
targbox = partimg[targcen[0]-searchr : targcen[0]+searchr, targcen[1]-searchr : targcen[1]+searchr]
weights = cent_weight(len(targbox))
tcenoffset = calc_CoM(targbox, weights)
print(tcenoffset)
tcenter = targcen + tcenoffset

plt.plot(sum(targbox))
plt.show()

compbox = partimg[compcen[0]-searchr : compcen[0]+searchr, compcen[1]-searchr : compcen[1]+searchr]
compw = cent_weight(len(compbox))
ccenoffset = calc_CoM(compbox,compw)
ccenter = compcen + ccenoffset

print(tcenter)


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry-Solutions.ipynb".
The first task is:
b. Disply image with aperture mask and sky annulus
Can you write Python code for it?

im = plt.imshow(targaper, cmap='hot')
plt.clim(1800, 2800)
plt.show()

im = plt.imshow(targann, cmap='hot')
plt.clim(1800, 2800)
plt.show()


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
0.1 Directory Set up
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
0.2 Display images
Can you write Python code for it?

def plotfits(imno):
    img = fits.open(datadir+objname+'_{0:02d}.fits'.format(numb))[0].data

    f = plt.figure(figsize=(10,12))
    #im = plt.imshow(img, cmap='hot')
    im = plt.imshow(img[480:580, 460:600], cmap='hot')
    plt.clim(1800, 2800)
    plt.colorbar(im, fraction=0.034, pad=0.04)
    plt.savefig("figure{0}.png".format(imno))
    plt.show()

numb = 1
plotfits(numb)

numb = 2
plotfits(numb)


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
1. Photometry set up
Select part of the image for ease of display.
Can you write Python code for it?

partimg = fits.open(datadir+objname+'_01.fits'.format(numb))[0].data[480:580, 460:600]


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
1.2 Aperture Photometry
Science Aperture
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
1.3 Tests
a. Display image with target and comparison centers before and after centroiding
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
2.2 Optimize photometry aperture
Can you write Python code for it?

## Write code here that tries a range of photometry apertures and finds the best SNR ##

print(bestaper)
print(snr)


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
2.3 Calculate the target's magnitude and uncertainty
Given the comparison is of known magnitude of 19.4
Can you write Python code for it?

targc = circle(searchr*2, ap_r)*newtarg
targskyc = annulus(searchr*2, sky_inner, sky_outer)*newtarg
compc = circle(searchr*2, ap_r)*newcomp
compskyc = annulus(searchr*2, sky_inner, sky_outer)*newcomp

ratio = np.sum(compc)/np.sum(targc)
### complete here ###
### complete here ###
### complete here ###

refmag = 19.4
### complete here ###
print("Measured Magnitude = {:0.3f} ± {:0.3f}".format(mag, sigmamag))


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
Define new regions where the target and comparison are centered.
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
Define starting values. Fill in values here:
Can you write Python code for it?

targcen = np.array([##,##])  ## target center
compcen = np.array([##,##])  ## comparison center


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
Inspect PSF to see whether shift makes sense
Can you write Python code for it?

plt.plot(sum(targbox))
plt.show()

compbox = partimg[compcen[0]-searchr : compcen[0]+searchr, compcen[1]-searchr : compcen[1]+searchr]
compw = cent_weight(len(compbox))
ccenoffset = calc_CoM(compbox,compw)
ccenter = compcen + ccenoffset

print(tcenter)

compw


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
Place mask on region
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
Place mask on sky annulus slice.
Can you write Python code for it?

targann = newtarg * annmask
compann = newcomp * annmask


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
Sky annulus
Can you write Python code for it?

def annulus(npix, r_inner,r_outer=-1.):
    """
    Builds an annulus
    """
    pup=np.zeros((npix,npix),np.int)
    for i in range(npix):
        for j in range(npix):
            #### Fill in annulus form here ####
            if ((r<=r_outer)&(r>=r_inner)):
                pup[i,j]=1
    return pup


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
Sum all flux inside target and comparison apertures and divide by number of pixels to get average count per pixel.
Can you write Python code for it?

targc = np.sum(targaper) / np.sum(circmask)
targbg= np.sum(targann) /  np.sum(annmask)
compc = np.sum(compaper) /  np.sum(circmask)
compbg= np.sum(compann) /  np.sum(annmask)

snr, noise = calcsnr(targc, targbg)
print(snr)

snr, noise = calcsnr(compc, compbg)
print(snr)


I am working on the file "Sessions/Session05/Day2/Introduction to Photometry.ipynb".
The first task is:
b. Disply image with aperture mask and sky annulus
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day3/Introduction to Astrometry.ipynb".
The first task is:
1. Centroiding on Images
Write a text file with image centers.
Write code to open each image and extract centroid position from previous exercise.
Save results in a text file.
Can you write Python code for it?

centers = np.array([[502,501], [502,501]])
np.savetxt('centers.txt', centers, fmt='%i')
centers = np.loadtxt('centers.txt', dtype='int')

searchr = 5


I am working on the file "Sessions/Session05/Day3/Introduction to Astrometry.ipynb".
The first task is:
1.1 Center of Mass
Can you write Python code for it?

def cent_weight(n):
    """
    Assigns centroid weights
    """
    wghts=np.zeros((n),np.float)
    for i in range(n):
        wghts[i]=float(i-n/2)+0.5
    return wghts

def calc_CoM(psf, weights):
    """
    Finds Center of Mass of image
    """
    cent=np.zeros((2),np.float)
    temp=sum(sum(psf) - min(sum(psf) ))
    cent[1]=sum(( sum(psf) - min(sum(psf)) ) * weights)/temp
    cent[0]=sum(( sum(psf.T) - min(sum(psf.T)) ) *weights)/temp
    return cent

centlist = []
for i, center in enumerate(centers):
    image = fits.open(datadir+objname+'_{0:02d}.fits'.format(i+1))[0].data
    searchbox = image[center[0]-searchr : center[0]+searchr, center[1]-searchr : center[1]+searchr]
    boxlen = len(searchbox)
    weights = cent_weight(boxlen)
    cen_offset = calc_CoM(searchbox, weights)
    centlist.append(center + cen_offset)

print(centlist[0])


I am working on the file "Sessions/Session05/Day3/Introduction to Astrometry.ipynb".
The first task is:
1.a. Create a new image using an mapping arc sinh that captures the full dynamic range effectively. Consider Gaussian smoothing to get rid of inhomogineties in the image.
Can you write Python code for it?

## Some functions you may want to use
import skimage.exposure as skie
from scipy.ndimage import gaussian_filter

### code here ###
### code here ###
### code here ###
### code here ###
### code here ###
### code here ###


I am working on the file "Sessions/Session05/Day3/Introduction to Astrometry.ipynb".
The first task is:
1.b. Create a new image that is scaled between the lower and upper limits for displaying the star map.
Search the arcsinh-stretched original image for local maxima and catalog those brighter than a threshold that is adjusted based on the image.
Can you write Python code for it?

## Consider using
import skimage.morphology as morph

### code here ###
### code here ###
### code here ###
### code here ###
### code here ###


I am working on the file "Sessions/Session05/Day3/Introduction to Astrometry.ipynb".
The first task is:
3. Matching
3.1 Get astrometric catalog
Can you write Python code for it?

job = Gaia.launch_job_async("SELECT * \
FROM gaiadr1.gaia_source \
WHERE CONTAINS(POINT('ICRS',gaiadr1.gaia_source.ra,gaiadr1.gaia_source.dec),CIRCLE('ICRS', 193.34, 33.86, 0.08))=1;" \
, dump_to_file=True)

print (job)

r = job.get_results()
print (r['source_id'], r['ra'], r['dec'])
print(type(r['ra']))


I am working on the file "Sessions/Session05/Day3/Introduction to Astrometry.ipynb".
The first task is:
3.2 Perform Match
Convert Gaia WCS coordinates to pixels
Can you write Python code for it?

ra  = np.array(r['ra'])
dec = np.array(r['dec'])

xpix, ypix = ### fill in one line here ###


I am working on the file "Sessions/Session05/Day3/Introduction to Astrometry.ipynb".
The first task is:
Convert shifted coordinate into WCS
Can you write Python code for it?

wparams, tscoords = load_wcs_from_file(datadir+objname+'_{0:02d}.fits'.format(1), np.array([targshifted[0][0][0]]), np.array([targshifted[0][0][1]]))


I am working on the file "Sessions/Session05/Day3/Introduction to Astrometry.ipynb".
The first task is:
Ex. 2 Find the amount of shift needed.
Match catalogue stars to identified stars and measure amount of shift needed to overlay FoV stars to catalogue.
E.g. Find closest star to one of the Gaia stars near the center of image. Find magnitude of shift. Shift all other Gaia stars and see whether resulting difference is small.
Can you write Python code for it?

### code here ###
### code here ###
### code here ###
### code here ###
### code here ###
### code here ###
### code here ###
### code here ###
### code here ###
### code here ###
### code here ###
### code here ###
### code here ###
### code here ###
### code here ###


I am working on the file "Sessions/Session05/Day3/Introduction to Astrometry.ipynb".
The first task is:
Find position of Asteroid in WCS
Can you write Python code for it?

wparams, scoords = load_wcs_from_file(datadir+objname+'_{0:02d}.fits'.format(1), x2, y2)

print(scoords)

wparams, tcoords = load_wcs_from_file(datadir+objname+'_{0:02d}.fits'.format(1), np.array([centlist[0][0]]), np.array([centlist[0][1]]))

print(tcoords)


I am working on the file "Sessions/Session05/Day3/Introduction to Astrometry.ipynb".
The first task is:
Plot image with identified stars and target
Can you write Python code for it?

f = plt.figure(figsize=(10,12))
plt.imshow(opt_img, cmap='hot')
plt.colorbar(fraction=0.046, pad=0.04)
plt.scatter(x2, y2, s=80, facecolors='none', edgecolors='r')
plt.scatter(502.01468185, 501.00082137, s=80, facecolors='none', edgecolors='y' )
plt.show()


I am working on the file "Sessions/Session05/Day4/WavelengthSolution solutions.ipynb".
The first task is:
Fit a curve to those residuals (it's generally better to fit to residuals, especially when you have a more informative model than our current linear one).
Use Chebyshev polynomials;  I recommend using np.polynomial.chebyshev.Chebyshev.fit.  You should set the domain of the fit so that it'll be usable over all the CCD's rows.
Experiment with a range of order of fitter, and look at the rms error in the wavelength solution.  You can look at $\chi^2/\nu$ too, if you like, but I think you'll find that the centroiding errors are wrong.
You probably want to look at the fit (to the residuals!) and at the residuals from the fit.

This cell answers both this question and the next.  If you set useLinearApproximation = True you'll get this solution; if it's False you'll do the fit relative to my instrument model.  Note that I'm using different orders for the two fits.
The variable plotResiduals controls whether you plot the fit or the residuals.  And note the if False ...: continue at the top of the loop; if you change that to if True ...: continue or remove it you'll only fit myFiberId
I've also added a floor to the centroiding errors.  This is often handy; not only does it allow you to include the systematic errors (quoted errors tend to be only statistical) and thus down-weight points with unrealistically low errors, but it also allows you to perform an unweighted fit by making the floor large enough.
Can you write Python code for it?

import numpy.polynomial.chebyshev
myFiberId = 315

pixelPosErrorFloor = 0.1

plt.clf()
plotResiduals = True
useLinearApproximation = True

fitOrder = 5 if useLinearApproximation else 2

for fiberId in arcLines:
    if False and fiberId != myFiberId:
        continue

    fiberData = arcLines[fiberId]
    #
    # Unpack data
    #
    pixelPos = fiberData["pixelPos"]
    pixelPosErr = fiberData["pixelPosErr"]
    refWavelength = fiberData["refWavelength"]
    modelFitWavelength = fiberData["modelFitWavelength"]
    status = fiberData["status"]

    good = np.logical_and.reduce([(status & statusFlags["FIT"]) != 0,
                                   status & (statusFlags["SATURATED"]) == 0])
    #
    # OK, on to work
    #
    linearWavelength = wavelength0 + nmPerPix*(pixelPos)
    if useLinearApproximation:
        wavelength = linearWavelength
    else:
        wavelength = modelFitWavelength

    nominalPixelPos = (refWavelength - wavelength0)/nmPerPix
    fitWavelengthErr = pixelPosErr*nmPerPix

    x = nominalPixelPos[good]
    y = (refWavelength - wavelength)[good]
    yerr = np.hypot(fitWavelengthErr, pixelPosErrorFloor*nmPerPix)[good]

    used = np.ones_like(good[good])

    wavelengthCorr = np.polynomial.chebyshev.Chebyshev.fit(
        x[used], y[used], fitOrder, domain=[pixels[0], pixels[-1]], w=1/yerr[used])
    arcLines[fiberId]['wavelengthCorr'] = wavelengthCorr
    yfit = wavelengthCorr(x)

    if plotResiduals:
        plt.plot(x, y - yfit, 'o', label=str(fiberId))
        plt.axhline(0, ls=':', color='black')
    else:
        ax = plt.plot(x, y, 'o', label=str(fiberId))[0]
        plt.plot(pixels, wavelengthCorr(pixels), color=ax.get_color())

    print("%-3d rms = %.2f mpix" %
          (fiberId, 1e3*np.sqrt(np.sum((y - yfit)**2)/(len(y) - fitOrder))))

plt.legend(loc=9, bbox_to_anchor=(1.0, 1.1));


I am working on the file "Sessions/Session05/Day4/WavelengthSolution solutions.ipynb".
The first task is:
Here's the plot for all the fibres:
Can you write Python code for it?

plt.clf()
for fiberId, fiberData in arcLines.items():
    pixelPos = fiberData["pixelPos"]
    pixelPosErr = fiberData["pixelPosErr"]
    refWavelength = fiberData["refWavelength"]
    modelFitWavelength = fiberData["modelFitWavelength"]
    status = fiberData["status"]

    wavelength = wavelength0 + nmPerPix*(pixelPos)

    good = np.logical_and.reduce([(status & statusFlags["FIT"]) != 0,
                                   status & (statusFlags["INTERPOLATED"]) == 0])

    plt.plot(pixelPos[good], (refWavelength - wavelength)[good], 'o', label=str(fiberId))

plt.xlabel("pixels")
plt.ylabel(r"$\lambda_{ref} - \lambda_{linear}$");
plt.legend(loc=9, bbox_to_anchor=(1.0, 1.1));


I am working on the file "Sessions/Session05/Day4/WavelengthSolution solutions.ipynb".
The first task is:
I think that the answer is, "No".  That's why we need a wavelength solution!
Fortunately I have measured the positions of the brighter arclines (the dict arcLines).  Choose a fibre and plot the measured lines on top of the spectrum.  You should find that I did a good job with the centroiding.
Can you write Python code for it?

fiberId = 5

plt.clf()
plt.plot(pixels, spectra[fiberId]["flux"], label=str(fiberId))

for p in arcLines[fiberId]["pixelPos"]:
    plt.axvline(p, ls=':', color='black')

plt.xlim(100, 600)
plt.ylim(-1e4, 2.5e5)
plt.legend()
plt.xlabel("pixels");


I am working on the file "Sessions/Session05/Day4/WavelengthSolution solutions.ipynb".
The first task is:
Let's start by looking at some extracted spectra
Read the data (and don't worry about the details of how this cell works!).

The reference lines (a dict arcLines indexed by fiberId containing numpy arrays):
pixelPos           Measured centroid of an arc line (pixels)
pixelPosErr        Error in pixelPos
refWavelength      Nominal wavelength (from NIST)
modelFitWavelength Wavelength corresponding to pixelPos, based on instrument model
status             bitwise OR of flags (see statusFlags)
A dict statusFlags giving the meaning of the status bits

wavelength0 and npPerPix  A couple of scalers giving an approximate wavelength solution

The arc spectra (a dict spectra indexed by fiberId containing numpy arrays):

flux     The flux level of each pixel
fluxVar  The error in flux
modelFitLambda  The wavelength at each pixel according to my instrument model
An array pixels giving the pixel values associated with the wavelengths, fluxes, and fluxVars arrays
Can you write Python code for it?

dataDump = np.load(os.path.join(dataDir, "spectro.npz"))

packed      = dataDump["packed"]
statusFlags = dataDump["statusFlags"].reshape(1)[0]
fiberIds    = dataDump["fiberIds"]
dataFields  = dataDump["dataFields"]
wavelength0 = dataDump["wavelength0"]
nmPerPix    = dataDump["nmPerPix"]
pixels      = dataDump["pixels"]
fluxes      = dataDump["fluxes"]
fluxVars    = dataDump["fluxVars"]
modelFitLambdas = dataDump["modelFitLambdas"]

arcLines = {}
spectra = {}
for i, fiberId in enumerate(fiberIds):
    arcLines[fiberId] = {}
    spectra[fiberId] = {}

    for j, n in enumerate(dataFields):
        arcLines[fiberId][n] = packed[i][j]
    arcLines[fiberId]['status'] = arcLines[fiberId]['status'].astype('int')

    spectra[fiberId]["modelFitLambda"] = modelFitLambdas[i]
    spectra[fiberId]["flux"] = fluxes[i]
    spectra[fiberId]["fluxVar"] = fluxVars[i]

del packed; del dataFields
del modelFitLambdas; del fluxes; del fluxVars


I am working on the file "Sessions/Session05/Day4/WavelengthSolution solutions.ipynb".
The first task is:
Plot the spectrum.  Do the lines from the different fibres line up?
Can you write Python code for it?

plt.clf()
for fiberId in fiberIds:
    plt.plot(pixels, spectra[fiberId]["flux"], label=str(fiberId))

plt.legend()
plt.xlabel("pixels");


I am working on the file "Sessions/Session05/Day4/WavelengthSolution solutions.ipynb".
The first task is:
Take a look at the statusFlags and the values of status from your fibre.  You probably want to ignore some of the data. The FIT bit is good;  the SATURATED, INTERPOLATED, and CR bits are bad (and the other bits aren't set in this dataset).
Remake your residual plot with the bad lines removed;  is that better?
Can you write Python code for it?

fiberId = 5
assert fiberId in arcLines, "Unknown fiberId: %d" % fiberId

pixelPos = arcLines[fiberId]['pixelPos']
refWavelength = arcLines[fiberId]['refWavelength']
status = arcLines[fiberId]["status"]

wavelength = wavelength0 + nmPerPix*pixelPos

good = np.logical_and.reduce([(status & statusFlags["FIT"]) != 0,
                               status & (statusFlags["INTERPOLATED"]) == 0])

plt.plot(pixelPos[good], (refWavelength - wavelength)[good], 'o', label=str(fiberId))

plt.legend()
plt.xlabel("pixels")
plt.ylabel(r"$\lambda_{ref} - \lambda_{linear}$");


I am working on the file "Sessions/Session05/Day4/WavelengthSolution solutions.ipynb".
The first task is:
We have good pixel centroids and we know the true wavelengths, so we can measure our wavelength solution.
Let's concentrate on just one fiber for now; choose a fibre, any fibre.
Plot some of the fitted lines.  A good place to start would be the pixel position (pixelPos) and the reference wavelength (refWavelength) for a fibre, then use wavelength0 and nmPerPix to construct an approximate (linear) wavelength solution and look at the residuals from the linear fit.
Can you write Python code for it?

fiberId = 5
assert fiberId in arcLines, "Unknown fiberId: %d" % fiberId
pixelPos = arcLines[fiberId]['pixelPos']
refWavelength = arcLines[fiberId]['refWavelength']

wavelength = wavelength0 + nmPerPix*pixelPos

plt.plot(pixelPos, refWavelength - wavelength, label=str(fiberId))

plt.legend()
plt.xlabel("pixels")
plt.ylabel(r"$\lambda_{ref} - \lambda_{linear}$");


I am working on the file "Sessions/Session05/Day4/WavelengthSolution.ipynb".
The first task is:
Fit a curve to those residuals (it's generally better to fit to residuals, especially when you have a more informative model than our current linear one).
Use Chebyshev polynomials;  I recommend using np.polynomial.chebyshev.Chebyshev.fit.  You should set the domain of the fit so that it'll be usable over all the CCD's rows.
Experiment with a range of order of fitter, and look at the rms error in the wavelength solution.  You can look at $\chi^2/\nu$ too, if you like, but I think you'll find that the centroiding errors are wrong.
You probably want to look at the fit (to the residuals!) and at the residuals from the fit.
Can you write Python code for it?

import numpy.polynomial.chebyshev
myFiberId = 315

pixelPosErrorFloor = 1 # 1e-4

plt.clf()
plotResiduals = True
useLinearApproximation = False

fitOrder = 5 if useLinearApproximation else 2

for fiberId in arcLines:
    if False and fiberId != myFiberId:
        continue

    fiberData = arcLines[fiberId]
    #
    # Unpack data
    #
    pixelPos = fiberData["pixelPos"]
    pixelPosErr = fiberData["pixelPosErr"]
    refWavelength = fiberData["refWavelength"]
    modelFitWavelength = fiberData["modelFitWavelength"]
    status = fiberData["status"]

    good = np.logical_and.reduce([(status & statusFlags["FIT"]) != 0,
                                   status & (statusFlags["SATURATED"]) == 0])
    #
    # OK, on to work
    #
    linearWavelength = wavelength0 + nmPerPix*(pixelPos)
    if useLinearApproximation:
        wavelength = linearWavelength
    else:
        wavelength = modelFitWavelength

    nominalPixelPos = (refWavelength - wavelength0)/nmPerPix
    fitWavelengthErr = pixelPosErr*nmPerPix

    x = nominalPixelPos[good]
    y = (refWavelength - wavelength)[good]
    yerr = np.hypot(fitWavelengthErr, pixelPosErrorFloor*nmPerPix)[good]

    used = np.ones_like(good[good])

    wavelengthCorr = np.polynomial.chebyshev.Chebyshev.fit(
        x[used], y[used], fitOrder, domain=[pixels[0], pixels[-1]], w=1/yerr[used])
    arcLines[fiberId]['wavelengthCorr'] = wavelengthCorr
    yfit = wavelengthCorr(x)

    if plotResiduals:
        plt.plot(x, y - yfit, 'o', label=str(fiberId))
        plt.axhline(0, ls=':', color='black')
    else:
        ax = plt.plot(x, y, 'o', label=str(fiberId))[0]
        plt.plot(pixels, wavelengthCorr(pixels), color=ax.get_color())

    print("%-3d rms = %.2f mpix" %
          (fiberId, 1e3*np.sqrt(np.sum((y - yfit)**2)/(len(y) - fitOrder))))

plt.legend(loc=9, bbox_to_anchor=(1.0, 1.1));


I am working on the file "Sessions/Session05/Day4/WavelengthSolution.ipynb".
The first task is:
I think that the answer is, "No".  That's why we need a wavelength solution!
Fortunately I have measured the positions of the brighter arclines (the dict arcLines).  Choose a fibre and plot the measured lines on top of the spectrum.  You should find that I did a good job with the centroiding.
We have good pixel centroids and we know the true wavelengths, so we can measure our wavelength solution.
Let's concentrate on just one fiber for now; choose a fibre, any fibre.
Plot some of the fitted lines.  A good place to start would be the pixel position (pixelPos) and the reference wavelength (refWavelength) for a fibre, then use wavelength0 and nmPerPix to construct an approximate (linear) wavelength solution and look at the residuals from the linear fit.
Can you write Python code for it?

fiberId = 5
assert fiberId in arcLines, "Unknown fiberId: %d" % fiberId
pixelPos = arcLines[fiberId]['pixelPos']
refWavelength = arcLines[fiberId]['refWavelength']

wavelength = wavelength0 + nmPerPix*pixelPos

#...


I am working on the file "Sessions/Session05/Day4/WavelengthSolution.ipynb".
The first task is:
Let's start by looking at some extracted spectra
Read the data (and don't worry about the details of how this cell works!).

The reference lines (a dict arcLines indexed by fiberId containing numpy arrays):
pixelPos           Measured centroid of an arc line (pixels)
pixelPosErr        Error in pixelPos
refWavelength      Nominal wavelength (from NIST)
modelFitWavelength Wavelength corresponding to pixelPos, based on instrument model
status             bitwise OR of flags (see statusFlags)
A dict statusFlags giving the meaning of the status bits

wavelength0 and npPerPix  A couple of scalers giving an approximate wavelength solution

The arc spectra (a dict spectra indexed by fiberId containing numpy arrays):

flux     The flux level of each pixel
fluxVar  The error in flux
modelFitLambda  The wavelength at each pixel according to my instrument model
An array pixels giving the pixel values associated with the wavelengths, fluxes, and fluxVars arrays
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/WavelengthSolution.ipynb".
The first task is:
Take a look at the statusFlags and the values of status from your fibre.  You probably want to ignore some of the data. The FIT bit is good;  the SATURATED, INTERPOLATED, and CR bits are bad (and the other bits aren't set in this dataset).
Remake your residual plot with the bad lines removed;  is that better?
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/WavelengthSolution.ipynb".
The first task is:
Use your wavelength solution to plot the arc spectra against wavelength (you started out by plotting against pixels).
Do the Ne lines for the different fibres agree now?
Can you write Python code for it?

plt.clf()
for fiberId in fiberIds:
    xlab = r"$\lambda$ (nm)"
    if useLinearApproximation:
        x = wavelength0 + nmPerPix*(pixels)
    else:
        x = spectra[fiberId]["modelFitLambda"]

    x += arcLines[fiberId]['wavelengthCorr'](pixels)

    plt.plot(x, spectra[fiberId]["flux"], label=str(fiberId))

plt.legend()
plt.xlabel(xlab)

if True:
    plt.xlim(625, 675)
    plt.ylim(-10000, 250000)


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/01_Registration/Register_images_exercise.ipynb".
The first task is:
Options you might want to look at:
--ra, --dec and --radius: Restrict the solution to some radius around RA and Dec. The regular telescope WCS should be plenty for an initial guess.
--scale-units, --scale-low and --scale-high: You might not know the exact pixel scale (and it's a function of where you are on the detector in any case, but you also set limits from this based on the existing CD1_2, CD2_1
-D, -N: Write to an output directory and write out a new FITS file with the solved WCS.
<u>Don't use out/ as the output directory.</u>
--parity: You can usually set this and get a speedup of 2x
To get info from the header, you can use astropy, or just use imhead from the WCSTools package at the command line
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/01_Registration/Register_images_exercise.ipynb".
The first task is:
Use the astrometry.net client (solve-field) to determine an accurate WCS solution for this field.
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/01_Registration/Register_images_exercise.ipynb".
The first task is:
While the images have been de-trended, they still have the original WCS from the telescope. They aren't aligned. You could use ds9 to check this trivially, but lets do it with astropy instead.
Can you write Python code for it?

import astropy.io.fits as afits
from astropy.wcs import WCS
from astropy.visualization import ZScaleInterval
import matplotlib

%matplotlib notebook
%pylab

##### I've given you some imports above. They're a big hint on what to try. You get to do this!!! #####


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/01_Registration/Register_images_exercise.ipynb".
The first task is:
or
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/01_Registration/Register_images_solutions.ipynb".
The first task is:
Options you might want to look at:
--ra, --dec and --radius: Restrict the solution to some radius around RA and Dec. The regular telescope WCS should be plenty for an initial guess.
--scale-units, --scale-low and --scale-high: You might not know the exact pixel scale (and it's a function of where you are on the detector in any case, but you also set limits from this based on the existing CD1_2, CD2_1
-D, -N: Write to an output directory and write out a new FITS file with the solved WCS.
<u>Don't use out/ as the output directory.</u>
--parity: You can usually set this and get a speedup of 2x
To get info from the header, you can use astropy, or just use imhead from the WCSTools package at the command line
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/01_Registration/Register_images_solutions.ipynb".
The first task is:
The commands I used are in Register_images_solution.txt and the output I generated in the out/ subdirectory.
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/01_Registration/Register_images_solutions.ipynb".
The first task is:
Use the above info to solve for the WCS for both images.
The problem with the distortion polynomial that astronometry.net uses is that routines like ds9 ignore it. Look at astrometry.net's wcs-to-tan routine and convert the solved WCS to the tangent projection.
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/01_Registration/Register_images_solutions.ipynb".
The first task is:
Use the astrometry.net client (solve-field) to determine an accurate WCS solution for this field.
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/01_Registration/Register_images_solutions.ipynb".
The first task is:
While the images have been de-trended, they still have the original WCS from the telescope. They aren't aligned. You could use ds9 to check this trivially, but lets do it with astropy instead.
Can you write Python code for it?

import astropy.io.fits as afits
from astropy.wcs import WCS
from astropy.visualization import ZScaleInterval
import matplotlib

%matplotlib notebook
%pylab

f1 = afits.open('wdd7.040920_0452.051_6.fits')
f2 = afits.open('wdd7.080104_0214.1025_6.fits')

f1wcs = WCS(f1[0].header)
f2wcs = WCS(f2[0].header)

zscaler = ZScaleInterval(nsamples=1000, contrast=0.25)

f1d = zscaler(f1[0].data)
f2d = zscaler(f2[0].data)

fig = figure(figsize=(10,4))
ax = fig.add_subplot(1,1,1)
ax.imshow(f1d.T, cmap='Reds')
ax.imshow(f2d.T, cmap='Blues', alpha=0.5)
tight_layout()
xlabel('x')
ylabel('y')
savefig('out/original_misalignment.pdf')


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/01_Registration/Register_images_solutions.ipynb".
The first task is:
or
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/02_Reprojection/Reproject_images_exercise.ipynb".
The first task is:
Create a figure instance, zscale the differences and see what the residuals look like
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/02_Reprojection/Reproject_images_exercise.ipynb".
The first task is:
Hopefully that's halfway reasonable looking compared to what we started from! For funsies, scale both images to have mean 1 and attempt to subtract them. Remember than there are NaNs in the reprojected image!
Can you write Python code for it?

#### You get to do this. You can look for NaNs in the image, or the reproject functions should return a footprint ####


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/02_Reprojection/Reproject_images_exercise.ipynb".
The first task is:
Now construct a simple difference image from the original data
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/02_Reprojection/Reproject_images_exercise.ipynb".
The first task is:
Now display the two images as you did in the last exercise - you should be able to copy and paste the same code even
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/03_Stacking/Stacking_images_exercise.ipynb".
The first task is:
Create the stacks - median, unweighted mean and using your weighting function
If you decided to use the masks, make sure you use the corresponding functions in numpy.ma
if you want to get fancy, you can even try astropy.stats.sigma_clip, and then you'll have tried the most commonly used stacking methods with SWarp
Can you write Python code for it?

#### You get to do this ####
#
# from astropy.stats import sigma_clip

# You'll probably want these to look at the results
from astropy.visualization import ZScaleInterval
zscaler = ZScaleInterval(nsamples=1000, contrast=0.25)


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/03_Stacking/Stacking_images_exercise.ipynb".
The first task is:
Now that the data is loaded, decide on how you'd like to weight the data. Normalize them in some sensible way. You'll be comparing this against a median stack and an unweighted simple average.
Can you write Python code for it?

#### You get to do this ####

# Here's an example for weights
#
# weights = 10.**-0.4*zptmag # I'm just using the flux zeropoint to set the weights
# wsum = np.sum(weights)
# weights /= wsum


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/03_Stacking/Stacking_images_exercise.ipynb".
The first task is:
Then, write a function that takes a filename and loads data from the image and the header that you think might be useful to weight the images by when producing a stack. (hint: SKYADU might be useful, maybe AIRMASS, ZPTMAG, EXPTIME - I'm tossing out suggestions - you can pick and examine what happens with different choices).
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/03_Stacking/Stacking_images_exercise.ipynb".
The first task is:
We used both of these extra images to generate a weight per image for image stacking or when doing image subtraction. Depending on how you proceed with this exercise, you could use one or both, or ignore them altogether.
Begin by taking a look at a single image's FITS header
Can you write Python code for it?


I am working on the file "Sessions/Session05/Day4/stackdiff_Narayan/03_Stacking/Stacking_images_exercise.ipynb".
The first task is:
You might also want to look at the data (it's ALWAYS a good idea to check) with ds9 and get a sense for it. Are they really aligned or do I just have no idea what I did back when I was a wee grad student (it's ALWAYS a good idea to check...).
For example, did I actually take out the background? Are the PSFs similar-ish? Are the exposure times the same. WCSTools gethead is your friend.
Can you write Python code for it?

%matplotlib notebook
%pylab
import astropy.io.fits as afits
### astropy can seamlessly handle the gzipped fits images


I am working on the file "Sessions/Session05/Day5/MultiwavelengthPhotometry.ipynb".
The first task is:
Problem 1) Download and Examine the Data
The images for this exercise can be downloaded from here: https://northwestern.box.com/s/x6nzuqtdys3jo1nufvswkx62o44ifa11. Be sure to place the images in the same directory as this notebook (but do not add them to your git repo!).
Before we dive in, here is some background information on the images we will be analyzing: the imaging data and the group information all come from the Galaxy And Mass Assembly (GAMA) survey; and more specifically, its panchromatic data release.
Many of the difficult steps associated with multiband galaxy photometry have already been done for you: GAMA constructs large mosaics of co-added FITS images in 20 bands to measure photometry. The images we will use today are from the g, r, and i mosaics that I (MA) built $\sim$7 years ago. They are built from SDSS observations in those bands, and have all been convolved to a seeing of approximately 2”, background subtracted, and renormalized to a common zeropoint of 30 magnitudes. The group catalogue was done by Aaron Robotham (see https://arxiv.org/abs/1106.1994).
In the downloaded directory there are g, r, and i images of 36 galaxies that all belong to the same cluster. These image cutouts have been centered on the galaxy position, are $\sim$80.7" on a side, and have a pixel scale of 0.339"/pix.
To begin we will focus on a single galaxy, before eventually working on the entire cluster.
Problem 1a
Display the $r$-band image of the galaxy 85698. Use a asinh stretch.
Can you write Python code for it?

r_filename = "galaxy_images/85698_sdss_r.fits"
r_data = fits.getdata( # complete

plt.imshow( # complete

plt.colorbar()
plt.tight_layout()


I am working on the file "Sessions/Session05/Day5/MultiwavelengthPhotometry.ipynb".
The first task is:
Problem 1b
Roughly how many sources are present in the image?
Hint - an exact count is not required here.
Solution 1b
Write your answer here
Problem 2) Source Detection
Prior to measuring any properties of sources in the image, we must first determine the number of sources present in the image. Source detection is challenging, and there are many different thresholding approaches.
Today, we will streamline this step in order to spend more time focusing on the issues associated with matching photometric measurements across different images. We will use the detect_sources function in photutils to identify objects in our image.
The simplest model assumes that the background is constant over the entire image. Once the background is determined, it can be subtracted from the image to determine high significance "peaks" corresponding to sources. After this week, we have learned that the background isn't so simple, nevertheless we will use the detect_threshold convenience function to estimate a constant background for our images. detect_threshold produces a "detection image" that can be used to estimate the significance of the flux detected in any individual pixel.
Problem 2a
Create a detection threshold image using the detect_threshold function, set the snr parameter to 3.
Can you write Python code for it?

threshold = detect_threshold( # complete


I am working on the file "Sessions/Session05/Day5/MultiwavelengthPhotometry.ipynb".
The first task is:
Problem 2d
Plot the segmentation image side-by-side with the actual image of the field.
Are you concerned or happy with the results?
Hint - no stretch should be applied to the segmentation image.
Can you write Python code for it?

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(7,4))

ax1.imshow(# complete
ax2.imshow(# complete
fig.tight_layout()


I am working on the file "Sessions/Session05/Day5/MultiwavelengthPhotometry.ipynb".
The first task is:
Problem 3) Source Centroids and Shapes
Now that we have defined all of the sources in the image, we must determine the centroid for each source (in order to ultimately make some form of photometric measurement). As Dora mentioned earlier in the week, there are many ways to determine the centroid of a given source (e.g., fitting a model, finding the max of the marginalized 1-d distribution, etc). Today we will use the centroid_com function, which calculates the "center of mass" of the 2d image moments to determine the source centroids.
To measure the centroid we want to isolate the source in question, thus we have generated a convenience function to return the extent of each source from its corresponding segmentation image.
Can you write Python code for it?

def get_source_extent(segm_data, source_num):
    """
    Determine extent of sources for centroid measurements

    Parameters
    ----------
    segm_data : array-like
        Segementation image produced by photutils.segmentation.detect_sources

    source_num : int
        The source number from the segmentation image

    Returns
    -------
    source_extent : list
        The minimum y, maximum y, minimum x, and maximum x pixel values
        over which a source is detected
    """
    source_pix = np.where(segm_data == source_num)
    source_extent = [np.min(source_pix[0]), np.max(source_pix[0]),
                     np.min(source_pix[1]), np.max(source_pix[1])]

    return source_extent


I am working on the file "Sessions/Session05/Day5/MultiwavelengthPhotometry.ipynb".
The first task is:
Problem 4b
Overplot your apertures on the sources that have been detected.
Hint - each aperture object has a plot() attribute that can be used to show the aperture for each source.
Can you write Python code for it?

fig, ax1 = plt.subplots()

ax1.imshow( # complete
# complete
# complete
fig.tight_layout()


I am working on the file "Sessions/Session05/Day5/MultiwavelengthPhotometry.ipynb".
The first task is:
Problem 4c
Measure the counts and uncertainty detected from each source within the apertures defined in 4a.
Hint - you will need to loop over each aperture as aperture_photometry does not take multiple apertures of different shapes as a single argument.
Can you write Python code for it?

source_cnts = # complete
source_cnts_unc = # complete
for source_num, ap in enumerate(apertures):
    phot = # complete
    source_cnts[source_num] = # complete
    source_cnts_unc[source_num] = # complete


I am working on the file "Sessions/Session05/Day5/MultiwavelengthPhotometry.ipynb".
The first task is:
Problem 5) Multiwavelength Photometry
Ultimately we want to measure colors for these galaxies. We now know the $r$-band magnitude for galaxy 85698, we need to measure the $g$ and $i$ band magnitudes as well.
Problem 5a Using the various pieces described above, write a function to measure the magnitude of the galaxy at the center of the image. You should create a new background image for every field.
Hint - creating an actual function is essential as we will eventually run this on every image.
Hint 2 - source_properties directly measures source centroids, use this it will be faster.
Can you write Python code for it?

def cluster_galaxy_photometry(data):
    '''
    Determine the magnitude of the galaxy at the center of the image

    Parameters
    ----------
    data : array-like
        Background subtracted 2D image centered on the galaxy
        of interest

    Returns
    -------
    mag : float
        Magnitude of the galaxy

    mag_unc : float
        Uncertainty of the magnitude measurement
    '''

    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete
    # complete

    return mag, mag_unc


I am working on the file "Sessions/Session05/Day5/MultiwavelengthPhotometry.ipynb".
The first task is:
Problem 5b
Confirm that the function calculates the same $r$-band mag that was calculated in Problem 4.
Can you write Python code for it?

# complete

print("""Previously, we found m = {:.3f} mag.
This new function finds m = {:.3f} mag.""".format( # complete


I am working on the file "Sessions/Session05/Day5/MultiwavelengthPhotometry.ipynb".
The first task is:
The images have been normalized to a zero point of 30. Thus, we can convert from counts to magnitudes via the following equation:
$$m = 30 - 2.5 \log (\mathrm{counts}).$$
Recall from Dora's talk that the uncertainty of the magnitude measurements can be calculated as:
$$\frac{2.5}{\ln(10)} \frac{\sigma_\mathrm{counts}}{\mathrm{counts}}.$$
Problem 4d
Calculate the magnitude of each source in the image.
Can you write Python code for it?

source_mag = # complete
source_mag_unc = # complete

for source_num, (mag, mag_unc) in enumerate(zip(source_mag, source_mag_unc)):
    print("Source {:d} has m = {:.3f} +/- {:.3f} mag".format( # complete


I am working on the file "Sessions/Session05/Day5/MultiwavelengthPhotometry.ipynb".
The first task is:
With apertures now defined, we can finally measure the flux of each source. The aperture_photometry function returns the flux (actually counts) in an image for the provided apertures. It takes the image, apertures, and bakground image as arguments.
Note - the background has already been subtracted from these images so we currently do not have an estimate of the full background for these sources.
We will create a background image that is approximately correct (we know this because we know the properties of the SDSS survey and detector). In this case what we are doing is not only incorrect, it's entirely made up and should not be repeated in your own work. Nevertheless, this (bad) approximation is necessary to produce uncertainty estimates.
Execute the cell below to create an uncertainty image to use with the aperture_photometry function.
Can you write Python code for it?

bkg = np.random.normal(100, 35, r_data.shape)
uncertainty_img = calc_total_error(r_data, bkg - np.mean(bkg), 1)


I am working on the file "Sessions/Session06/Day1/BuildingBetterModels.ipynb".
The first task is:
How good is the approximation?
 Problem 3b
Let's minimize using the basinhopping algorithm now, again not using our derivatives.
Can you write Python code for it?

times0 = np.array(times) - min(times)
results = scipy.optimize.basinhopping(  # complete
best_x = results.x

print('All done! Best score: `{}`, took `{}` function evaluations.'.format(results.fun, results.nfev))


I am working on the file "Sessions/Session06/Day1/BuildingBetterModels.ipynb".
The first task is:
Problem 1b
We know what the data looks like, and we've loaded a model that can be used to fit the data which computes a likelihood. Let's minimize the parameters of this model using various scipy.optimize routines. Note that since we are trying to maximize the likelihood, we have constructed a wrapper function around ln_likelihood, my_func, to reverse its sign, and to handle bad function evaluations.
Most optimize routines in scipy require a derivative. Since we don't have this available, scipy must construct an approximate one, unless the method doesn't require a gradient to be computed (like differential_evolution). For this first sub-problem, optimize my_func using differential_evolution.
Hints: Each variable is bounded to the range (0, 1), but problems can arise if an optimizer attempts to compute values outside or right at the boundaries. Therefore, it is recommended to use a bounded optimizer in scipy, where the bounds do not include 0 or 1.
Can you write Python code for it?

import scipy

def my_func(x):
    try:
        fx = -float(my_model.ln_likelihood(x))
    except:
        fx = np.inf
    return fx

eps = 0.00001
bounds = # complete
results = scipy.optimize.differential_evolution(  # complete
best_x = results.x

print('All done! Best score: `{}`.'.format(-results.fun))


I am working on the file "Sessions/Session06/Day1/BuildingBetterModels.ipynb".
The first task is:
Problem 2a
Compute the derivative for $\log p$ (above expression) with respect to $m_0$ (Mathematica might be helpful here). Below are the derivatives for the other five free parameters $\alpha$, $\beta$, $\tau$, $t_0$, and $\sigma$:
$$
\begin{align}
\frac{\partial\log p}{\partial \alpha} &= \sum_i -\frac{5 \log \left(1-e^{-\frac{t+t_0}{\tau }}\right) \left{\log (100) (\bar{m}-m_0)+5 \log \left[\left(1-e^{-\frac{t+t_0}{\tau }}\right)^{\alpha } \left(\frac{t+t_0}{\tau }\right)^{-\beta }\right]\right}}{4 \log ^2(10) \left(\sigma_i^2+\sigma^2\right)}\
\frac{\partial\log p}{\partial \beta} &= \sum_i \frac{5 \log \left(\frac{t+t_0}{\tau }\right) \left{\log (100) (\bar{m}-m_0)+5 \log \left[\left(1-e^{-\frac{t+t_0}{\tau }}\right)^{\alpha } \left(\frac{t+t_0}{\tau }\right)^{-\beta }\right]\right}}{4 \log ^2(10) \left(\sigma_i^2+\sigma^2\right)}\
\frac{\partial\log p}{\partial \tau} &= \sum_i \frac{5 \left(\alpha  (t+t_0)-\beta  \tau  \left(e^{\frac{t+t_0}{\tau }}-1\right)\right) \left(\log (100) (\bar{m}-m_0)+5 \log \left(\left(1-e^{-\frac{t+t_0}{\tau }}\right)^{\alpha } \left(\frac{t+t_0}{\tau }\right)^{-\beta }\right)\right)}{4 \tau ^2 \log ^2(10) \left(\sigma_i^2 + \sigma^2\right) \left(e^{\frac{t+t_0}{\tau }}-1\right)}\
\frac{\partial\log p}{\partial t_0} &= \sum_i \frac{5 \left(\alpha  (t+t_0)-\beta  \tau  \left(e^{\frac{t+t_0}{\tau }}-1\right)\right) \left(\log (100) (m_0-\bar{m})-5 \log \left(\left(1-e^{-\frac{t+t_0}{\tau }}\right)^{\alpha } \left(\frac{t+t_0}{\tau }\right)^{-\beta }\right)\right)}{4 \tau  \log ^2(10) \left(\sigma_i^2+\sigma^2\right) (t+t_0) \left(e^{\frac{t+t_0}{\tau }}-1\right)}\
\frac{\partial\log p}{\partial \sigma} &= \sum_i \frac{\sigma_i}{4 \log ^2(10) \left(\sigma_i^2+\sigma^2\right)^2} \left{5 \log \left[\left(1-e^{-\frac{t+t_0}{\tau }}\right)^{\alpha } \left(\frac{t+t_0}{\tau }\right)^{-\beta }\right]\right.\&\times\left.\left(4 \log (10) (\bar{m}-m_0)+5 \log \left[\left(1-e^{-\frac{t+t_0}{\tau }}\right)^{\alpha } \left(\frac{t+t_0}{\tau }\right)^{-\beta }\right]\right)+4 \log...
Can you write Python code for it?

def dlogp_dalpha(x, tt, mm, vv):
    m0, alpha, beta, tau, t0, sigma = tuple(x)
    t = np.array(tt)
    m = np.array(mm)
    v = np.array(vv)

    derivs = np.sum(-5.0 * np.log(1.0 - np.exp(-(t + t0) / tau)) * (np.log(100.0) * (m - m0) + 5.0 * lf(
        alpha, beta, tau, t0, t)) / (4.0 * np.log(10.0) ** 2 * (v ** 2 + sigma ** 2)))

    return derivs

def dlogp_dbeta(x, tt, mm, vv):
    m0, alpha, beta, tau, t0, sigma = tuple(x)
    t = np.array(tt)
    m = np.array(mm)
    v = np.array(vv)

    derivs = np.sum(5.0 * np.log((t + t0) / tau) * (np.log(100.0) * (m - m0) + 5.0 * lf(
        alpha, beta, tau, t0, t)) / (4.0 * np.log(10.0) ** 2 * (v ** 2 + sigma ** 2)))

    return derivs

def dlogp_dtau(x, tt, mm, vv):
    m0, alpha, beta, tau, t0, sigma = tuple(x)
    t = np.array(tt)
    m = np.array(mm)
    v = np.array(vv)

    derivs = np.sum(5.0 * (alpha * (t + t0) - beta * tau * (np.exp((t + t0)/tau) - 1.0)) * (
        np.log(100.0) * (m - m0) + 5.0 * lf(
            alpha, beta, tau, t0, t)) / (4.0 * tau ** 2 * np.log(10.0) ** 2 * (v ** 2 + sigma ** 2) * (
        np.exp((t + t0)/tau) - 1.0)))

    return derivs

def dlogp_dt0(x, tt, mm, vv):
    m0, alpha, beta, tau, t0, sigma = tuple(x)
    t = np.array(tt)
    m = np.array(mm)
    v = np.array(vv)

    derivs = np.sum(-5.0 * (alpha * (t + t0) - beta * tau * (np.exp((t + t0)/tau) - 1.0)) * (
        np.log(100.0) * (m - m0) + 5.0 * lf(
            alpha, beta, tau, t0, t)) / (4.0 * tau * (t + t0) * np.log(10.0) ** 2 * (v ** 2 + sigma ** 2) * (
        np.exp((t + t0)/tau) - 1.0)))

    return derivs

def dlogp_dsigma(x, tt, mm, vv):
    m0, alpha, beta, tau, t0, sigma = tuple(x)
    t = np.array(tt)
    m = np.array(mm)
    v = np.array(vv)

    derivs = np.sum(sigma/(4.0 * np.log(10.0) ** 2 * (v**2 + sigma**2)**2) * (5.0 * lf(
        alpha, beta, tau, t0, t) * (4.0 * np.log(10.0) * (m - m0) + 5.0 * lf(
        alpha, beta, tau, t0, t)) + 4.0 * np.log(10.0) ** 2 * ((m0 - m) ** 2 - v ** 2 - sigma ** 2)))

   ...


I am working on the file "Sessions/Session06/Day1/BuildingBetterModels.ipynb".
The first task is:
Problem 2c
Make sure the derivatives for all the above function are consistent with the finite differences of the objective function. How large is the error for an eps = 1e-8 (the default distance used when no Jacobian is provided)? Make a histogram for each derivative of these errors for 100 random parameter combinations drawn from the bounds (in other words, six plots with 100 samples each).
Hint: you will likely have to remove some nans.
Can you write Python code for it?

# Set up bounds/test parameters.
abounds = [
    [0.0, 30.0],
    [0.1, 50.0],
    [0.1, 10.0],
    [0.1, 200.0],
    [0.1, 200.0],
    [0.001, 1.0]
]

test_times = [1.0, 20.0]
test_mags = [23.0, 19.0]
test_errs = [0.1, 0.2]

# Draw a random parameter combo to test with.
n = 100
dm0_diff = np.zeros(n)
for p in range(n):
    test_x = [abounds[i][0] + x * (abounds[i][1] - abounds[i][0]) for i, x in enumerate(np.random.rand(6))]

    # Check that every derivative expression is close to finite difference.
    teps = 1e-10

    xp = list(test_x)
    xp[0] += teps
    exactd = dlogp_dm0(test_x, test_times, test_mags, test_errs)
    dm0_diff[p] = (exactd - (
        analytic_f(test_x, test_times, test_mags, test_errs) - analytic_f(
            xp, test_times, test_mags, test_errs)) / teps) / exactd

    # complete for rest of parameters

plt.subplot(321)
plt.hist(dm0_diff[~np.isnan(dm0_diff)]);
# complete for rest of parameters


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 1c
Create a function, phase_plot, that takes x, y, and $P$ as inputs to create a phase-folded light curve (i.e., plot the data at their respective phase values given the period $P$).
Include an optional argument, y_unc, to include uncertainties on the y values, when available.
Can you write Python code for it?

def phase_plot( # complete

    phases = # complete
    # complete
    # complete
    fig, ax = plt.subplots()
    ax.errorbar( # complete
    # complete
    # complete
    fig.tight_layout()


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 1d
Plot the phase folded data generated in 1b.
Does you plot match your expectations?
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 2) A Brief Review of Fourier Analysis
In astronomical time series, we crave the detection of periodic signals because they can often provide fundamental insight into the sources we are studying (e.g., masses in a binary, classifications for pulsating stars, etc).
The standard$^\dagger$ choice for most astronomers to identify such signals is the Lomb-Scargle (LS) periodogram (Lomb 1976; Scargle 1982).
$^\dagger$Standard does not mean best, fastest, or even correct depending on your specific application.
At the heart of understanding any periodic signals is Fourier analysis. Thus, to understand how to interpret the LS periodogram, we first need to consider Fourier transforms.
Note - here we aim to jog your memory regarding Fourier analysis. Detailed mathematical calculations can be found elsewhere.
Given a continuous signal, $g(t)$ the Fourier transform of that signal is defined as:
$$\hat{\mathrm{g}}(f) = \int_{-\infty}^{\infty} g(t) \,e^{-2\pi i f t} \,dt,$$
where $i$ is an imaginary number. The inverse of this equation is defined as:
$$ g(t) = \int_{-\infty}^{\infty} \hat{\mathrm{g}}(f) \,e^{-2\pi i f t} \,dt.$$
For convenience, we will use the Fourier transform operator $\mathcal{F}$, from which the above equations reduce to:
$$\mathcal{F}(g) = \hat g$$
$$\mathcal{F}^{-1}(\hat{g}) = g$$
There are many useful properties of the Fourier transform including that the Fourier transform is a linear operator. Additionally, a time shift imparts a phase shift. Perhaps most importantly for our present purposes, however, is that the squared amplitude of the resulting transform allows us to get rid of the imaginary component and measure the power spectral density or power spectrum:
$$ \mathcal{P}_g = \left|\mathcal{F}(g)\right|^2.$$
The power spectrum is a real-valued function that quantifies the contribution of each frequency $f$ to the total signal in $g$. The power spectrum thus provides a way to identify the dominant frequency in any given signal.
Next we consider some common Fourier pairs, that will prove...
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 3c
Write a function, ls_periodogram, to calculate the LS periodogram for observations $y$, $\sigma_y$, $t$ over a frequency grid f_grid.
Can you write Python code for it?

def ls_periodogram( # complete
    psd = np.empty_like(f_grid)
    chi2_0 = # complete
    for f_num, f in enumerate(f_grid):
        psd[f_num] = # complete

    return psd


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 3f
Plot the phase-folded data at the newly found "best" fit period.
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 4) Other Considerations and Faster Implementations
While ls_periodogram functions well, it would take a long time to evaluate $\sim4\times 10^5$ frequencies for $\sim2\times 10^7$ variable LSST sources. Fortunately, there are significantly faster implementations, including (as you may have guessed) one in astropy.
Problem 4a
LombScargle in astropy.stats is fast. Run it below to compare to ls_periodogram.
Can you write Python code for it?

from astropy.stats import LombScargle
frequency, power = LombScargle(x, y, y_unc).autopower()


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 4e
Calculate and plot the periodogram for the window function (i.e., set y = 1 in LombScargle) of the observations. Do you notice any significant power?
Hint - you may need to zoom in on the plot to see all the relevant features.
Can you write Python code for it?

ls = LombScargle( # complete

freq_window, power_window = # complete

fig, ax = plt.subplots()
ax.plot( # complete
ax.set_ylabel("Power")
ax.set_xlabel("Period (d)")
ax.set_xlim(0,500)

axins = plt.axes([.2, .65, .5, .2])
axins.plot( # complete
axins.set_xlim(0,5)


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 5b
Use LombScargle to measure the periodogram. Then plot the periodogram and the phase folded light curve at the best-fit period.
Hint - search periods longer than 2 hr.
Can you write Python code for it?

frequency, power = # complete

# complete

fig,ax = plt.subplots()

ax.plot(# complete
ax.set_ylabel("Power")
ax.set_xlabel("Period (d)")
ax.set_xlim(0, 800)

axins = plt.axes([.25, .55, .6, .3])
axins.plot( # complete
axins.set_xlim(0,5)
fig.tight_layout()

# plot the phase folded light curve
phase_plot( # complete


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Problem 5c
Now plot the light curve folded at twice the best LS period.
Which of these 2 is better?
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Sampling a signal directly at the Nyquist frequency results in a lack of any variability. But does this just mean that $f_\mathrm{Ny}$ is special? What happens at $f > f_\mathrm{Ny}$?
Problem 2b
As above, generate and plot a periodic signal with $f = 0.7$ on an even grid from 0 to 10. Overplot the underlying signal in addition to the observations.
Can you write Python code for it?

x = # complete
y = # complete
x_signal = # complete
y_signal = # complete
fig, ax = plt.subplots()
ax.scatter( # complete
ax.plot( # complete
fig.tight_layout()


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
The common Fourier pairs are especially useful in light of the convolution theorem. Fourier transforms convert convolutions into point-wise products. We define a convolution as:
$$ [f \ast g] (t) = \int_{-\infty}^{\infty} f(\tau) \,g(t - \tau) \,d\tau,$$
where $\ast$ is the convolution symbol. From the convolution theorem:
$$ \mathcal{F} {f \ast g} = \mathcal{F}(f)  \mathcal{F}(g) $$
Furthermore, the Fourier transform of a product is equal to the convolution of the Fourier transforms:
$$ \mathcal{F}{f \cdot g} = \mathcal{F}(f) \ast \mathcal{F}(g) $$
This property will be very important for understanding the Lomb-Scargle periodogram.
Fourier transforms are all well and good, but ultimately we desire a measure of periodicity in actual observations of astrophysical sources, which cannot be (a) continuous, or (b) infinite.
The first thing to understand with real world observations is the Nyquist frequency limit. If observations are obtained in a uniformly spaced manner at a rate of $f_0 = 1/T$ one can only recover the frequncy information if the signal is band-limited between frequencies $\pm f_0/2$. Put another way, the highest frequency that can be detected in such data is $f_0/2$.
This result can be (somewhat) intuited by looking at simulated data.
Problem 2a
Generate and plot a periodic signal with $f = f_\mathrm{Ny} = 1/2$ on a grid from 0 to 10, comprising of 10 even samples (i.e., 0, 1, 2, 3, ..., 10). Overplot the underlying signal in addition to the observations.
Can you write Python code for it?


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
The observations are identical! Here is what you need to remember about the Nyquist frequency:

If you are going to obtain observations at regular intervals, and there is a specific signal you wish to detect, then be sure to sample the data such that $f_\mathrm{Ny} > f_\mathrm{signal}$.
For all $f > f_\mathrm{Ny}$, $f$ will be aliased with $f \pm 2n f_\mathrm{Ny}$ signals, where $n$ is an integer. Practically speaking, this means it does not make sense to search for signals with $f > f_\mathrm{Ny}$.
Finally, (and this is something that is often wrong in the literature) there is no Nyquist limit for unevenly sampled data (see VanderPlas 2017 for further details). Thus, for (virtually all) ground-based observing one need not worry about the Nyquist limit.

Staying on the topic of non-continuous observations, I present without derivation the discrete Fourier transform:
$$ \hat g_\mathrm{obs}(f) = \sum_{n = 0}^N g_n\,e^{-2\pi i f n\Delta t}$$
where $g_n = g(n\Delta t)$, and $\Delta t$ is the sampling interval. Our discussion of the Nyquist frequency tells us that we cannot detect frequencies $f > 1/2\Delta T$. Thus, the relevant frequencies to search for power given $\Delta t$ are between 0 and $f_\mathrm{Ny}$, which we can sample on a grid $\Delta f = 1/(N \Delta t)$. From there:
$$\hat g_k = \sum_{n = 0}^N g_n\,e^{-2\pi i f knN}$$
where $\hat g_k = \hat g_\mathrm{obs} (k\Delta f)$. This is the discrete Fourier transform.
I said a full derivation will not be provided, and that is true. To understand how we went from a continuous integral to the summation above, recall that regular observations of a continous signal over a finite interval is equivalent to multiplying the continuous signal by a Dirac comb function and a window function. The delta functions from the Dirac comb function collapse the integral to a sum, while the window function limits that sum from $0$ to $N$.
From the discrete Fourier transform we can then calculate the periodogram (an estimator of the power spectrum):
$$\mathcal{P}(f) =...
Can you write Python code for it?

def chi2( # complete
    # complete
    # complete
    # complete


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
There are many choices regarding the calculation of the periodogram, so read the docs.
Floating Mean Periodogram
A basic assumption that we preivously made is that the data are "centered" - in other words, our model explicitly assumes that the signal oscillates about a mean of 0.
For astronomical applications, this assumption can be harmful. Instead, it is useful to fit for the mean of the signal in addition to the periodic component (as is the default in LombScargle):
$$y(t;f) = y_0(f) + A_f \sin(2\pi f(t - \phi_f).$$
To illustrate why this is important for astronomy, assume that any signal fainter than $-2$ in our simulated data cannot be detected.
Problem 4c
Remove the observations from x and y where $y \le -2$ and calculate the periodogram both with and without fitting the mean (fit_mean = False in the call to LombScargle). Plot the periodograms. Do both methods recover the correct period?
Can you write Python code for it?

# complete
freq_no_mean, power_no_mean = LombScargle( # complete
freq_fit_mean, power_fit_mean = LombScargle( # complete

fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True)
ax1.plot( # complete
ax2.plot( # complete
ax1.set_xlim(0,15)

fig.tight_layout()


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
This example also shows why it is somewhat strange to provide an uncertainty with a LS best-fit period. Errors tend to be catastrophic, and not some small fractional percentage, with the LS periodogram (we'll explore this a little further later).
In the case of the above EB, the "best-fit" period was off by a factor 2. This is not isolated to EBs, however, LS periodograms frequently identify a correct harmonic of the true period, but not the actual period of variability.
Conclusions
The Lomb-Scargle periodogram is a useful tool to search for sinusoidal signals in noisy, irregular data.
However, as highlighted throughout, there are many ways in which the methodology can run awry.
In closing, I will summarize some practical considerations from VanderPlas (2017):

Choose an appropriate frequency grid (defaults in LombScargle are not sufficient)
Calculate the LS periodogram for the observation times to search for dominant signals (e.g., 1 day in astro)
Compute LS periodogram for data (avoid multi-Fourier models if signal unknown)
Plot periodogram and various FAP levels (do not over-interpret FAP)
If window function shows strong aliasing, plot phased light curve at each peak (now add more Fourier terms if necessary)
If looking for a particular signal (e.g., detatched EBs), consider different methods that better match expected signal
Inject fake signals into data to understand systematics if using LS in a survey pipeline

Finally, Finally
As a very last note: know that there are many different ways to search for periodicity in astronomical data. Depending on your application (and computational resources), LS may be a poor choice (even though this is often the default choice by all astronomers!) Graham et al. (2013) provides a summary of several methods using actual astronomical data. The results of that study show that no single method is best. However, they also show that no single method performs particularly well: the detection efficiences in Graham et al. (2013) are disappointing given the importance of...
Can you write Python code for it?

def fourier_pairs_plot():

    fig, ax = plt.subplots(4, 2, figsize=(10, 6))
    fig.subplots_adjust(left=0.04, right=0.98, bottom=0.02, top=0.95,
                        hspace=0.3, wspace=0.2)

    x = np.linspace(-5, 5, 1000)

    for axi in ax.flat:
        axi.xaxis.set_major_formatter(plt.NullFormatter())
        axi.yaxis.set_major_formatter(plt.NullFormatter())

        # draw center line
        axi.axvline(0, linestyle='dotted', color='gray')
        axi.axhline(0, linestyle='dotted', color='gray')

    style_re = dict(linestyle='solid', color='k', linewidth=2)
    style_im = dict(linestyle='solid', color='gray', linewidth=2)
    text_style = dict(size=14, color='gray')

    # sine -> delta
    ax[0, 0].plot(x, np.cos(x),**style_re)
    ax[0, 0].set(xlim=(-5, 5), ylim=(-1.2, 1.2))
    ax[0, 0].annotate('', (-np.pi, 0), (np.pi, 0),
                      arrowprops=dict(arrowstyle='|-|', color='gray'))
    ax[0, 0].text(0, 0, '$1/f_0$', ha='center', va='bottom', **text_style)
    ax[0, 0].set_title('Sinusoid')

    ax[0, 1].plot([-5, 2, 2, 2, 5], [0, 0, 1, 0, 0], **style_re)
    ax[0, 1].plot([-5, -2, -2, -2, 5], [0, 0, 1, 0, 0], **style_re)
    ax[0, 1].set(xlim=(-5, 5), ylim=(-0.2, 1.2))
    ax[0, 1].annotate('', (0, 0.4), (2, 0.4), arrowprops=dict(arrowstyle='<-', color='gray'))
    ax[0, 1].annotate('', (0, 0.4), (-2, 0.4), arrowprops=dict(arrowstyle='<-', color='gray'))
    ax[0, 1].text(1, 0.45, '$+f_0$', ha='center', va='bottom', **text_style)
    ax[0, 1].text(-1, 0.45, '$-f_0$', ha='center', va='bottom', **text_style)
    ax[0, 1].set_title('Delta Functions')

    # gaussian -> gaussian
    ax[1, 0].plot(x, np.exp(-(2 * x) ** 2), **style_re)
    ax[1, 0].set(xlim=(-5, 5), ylim=(-0.2, 1.2))
    ax[1, 0].annotate('', (0, 0.35), (0.6, 0.35), arrowprops=dict(arrowstyle='<-', color='gray'))
    ax[1, 0].text(0, 0.4, '$\sigma$', ha='center', va='bottom', **text_style)
    ax[1, 0].set_title('Gaussian')

    ax[1, 1].plot(x, np.exp(-(x / 2) ** 2), **style_re)
    ax[1, 1].set(xlim=(-5, 5),...


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Uncertainty on the best-fit period
How do we report uncertainties on the best-fit period from LS? For example, for the previously simulated LSST light curve we would want to report something like $P = 102 \pm 4\,\mathrm{d}$. However, the uncertainty from LS periodograms cannot be determined in this way.
Naively, one could report the width of the peak in the periodogram as the uncertainty in the fit. However, we previously saw that the peak width $\propto 1/T$ (the peak width does not decrease as the number of observations or their S/N increases; see Vander Plas 2017). Reporting such an uncertainty is particularly ridiculous for long duration surveys, whereby the peaks become very very narrow.
An alternative approach is to report the False Alarm Probability (FAP), which estimates the probability that a dataset with no periodic signal could produce a peak of similar magnitude, due to random gaussian fluctuations, as the data.
There are a few different methods to calculate the FAP. Perhaps the most useful, however, is the bootstrap method. To obtain a bootstrap estimate of the LS FAP one leaves the observation times fixed, and then draws new observation values with replacement from the actual set of observations. This procedure is then repeated many times to determine the FAP.
One nice advantage of this procedure is that any effects due to the window function will be imprinted in each iteration of the bootstrap resampling.
The major disadvantage is that many many periodograms must be calculated. The rule of thumb is that to acieve a FAP $= p_\mathrm{false}$, one must run $n_\mathrm{boot} \approx 10/p_\mathrm{false}$ bootstrap periodogram calculations. Thus, an FAP $\approx 0.1\%$ requires an increase of 1000 in computational time.
LombScargle provides the false_alarm_probability method, including a bootstrap option. We skip that for now in the interest of time.
As a final note of caution - be weary of over-interpreting the FAP. The specific question answered by the FAP is, what is the probability that...
Can you write Python code for it?

data = # complete

fig, ax = plt.subplots()
ax.errorbar( # complete
ax.set_xlabel('HJD (d)')
ax.set_ylabel('V (mag)')
ax.set_ylim(ax.get_ylim()[::-1])
fig.tight_layout()


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Unlike ls_periodogram, LombScargle effectively takes no time to run on the simulated data.
Problem 4b
Plot the periodogram for the simulated data.
Can you write Python code for it?

fig, ax = plt.subplots()

ax.plot( # complete
# complete
# complete
# complete
fig.tight_layout()


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
We can see that the best fit model doesn't match the signal in the case where we do not allow a floating mean.
Can you write Python code for it?

fit_mean_model = LombScargle(x[bright], y[bright], y_unc[bright],
                             fit_mean=True).model(np.linspace(0,10,1000),
                                                  freq_fit_mean[np.argmax(power_fit_mean)])
no_mean_model = LombScargle(x[bright], y[bright], y_unc[bright],
                             fit_mean=False).model(np.linspace(0,10,1000),
                                                  freq_no_mean[np.argmax(power_no_mean)])

fig, ax = plt.subplots()
ax.errorbar(x[bright], y[bright], y_unc[bright], fmt='o', label='data')
ax.plot(np.linspace(0,10,1000), fit_mean_model, label='fit mean')
ax.plot(np.linspace(0,10,1000), no_mean_model, label='no mean')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.legend()
fig.tight_layout()


I am working on the file "Sessions/Session06/Day1/ExtractingPeriodicSignals.ipynb".
The first task is:
Window Functions
Recall that the convolution theorem tells us that:
$$\mathcal{F}[f\cdot g] = \mathcal{F}(f) \ast \mathcal{F}(g)$$
Telescope observations are effectively the product of a continous signal with several delta functions (corresponding to the times of observations). As a result, the convolution that produces the periodogram will retain signal from both the source and the observational cadence.
To illustrate this effect, let us simulate "realistic" observations for a 10 year telescope survey. We do this by assuming that a source is observed every 3 nights (the LSST cadence) within $\pm 4\,\mathrm{hr}$ of the same time, and that $\sim 30\%$ of the observations did not occur due to bad weather. We further assume that the source cannot be observed for 40% of the year because it is behind the sun.
Simulate a periodic signal with this cadence, a period = 220 days (typical for Miras), amplitude = 12.4, and noise = 1. Plot the simulated light curve.
Problem 4d
Simulate a periodic signal with 3 day cadence (and the observing conditions described above), a period = 220 days (typical for Miras), amplitude = 12.4, and variance of the noise = 1. Plot the simulated light curve.
Can you write Python code for it?

# set up simulated observations
t_obs = np.arange(0, 10*365, 3)      # 3d cadence
# complete
# complete
# complete

y = gen_periodic_data( # complete
# complete

fig, ax = plt.subplots()
ax.errorbar( # complete
ax.set_xlabel("Time (d)")

ax.set_ylabel("Flux (arbitrary units)")


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
As you can see - force feeding this problem into a Bayesian framework does not automatically generate more reasonable answers. While some of the chains appear to have identified periods close to the correct period most of them are suck in local minima.
There are sampling techniques designed to handle multimodal posteriors, but the non-linear nature of this problem makes it difficult for the various walkers to explore the full parameter space in the way that we would like.
Problem 4) GPs and MCMC to identify a best-fit period
We will now attempt to model the data via a Gaussian Process (GP). As a very brief reminder, a GP is a collection of random variables, in which any finite subset has a multivariate gaussian distribution.
A GP is fully specified by a mean function and a covariance matrix $K$. In this case, we wish to model the simulated data from problem 1. If we specify a cosine kernel for the covariance:
$$K_{ij} = k(x_i - x_j) = \cos\left(\frac{2\pi \left|x_i - x_j\right|}{P}\right)$$
then the mean function is simply the offset, b.
Problem 4a
Write a function model2 that returns the mean function for the GP given input parameters $\theta$.
Hint - no significant computation is required to complete this task.
Can you write Python code for it?

def model2( # complete
    # complete
    return # complete


I am working on the file "Sessions/Session06/Day1/GaussianProcessPeriodicity.ipynb".
The first task is:
Because we have no idea where to initialize our walkers in this case, we are going to use an ad hoc common sense + brute force approach.
Problem 6c
Run LombScarge on the data and determine the top three peaks in the periodogram. Set nterms = 2, and the maximum frequency to 5 (this is arbitrary but sufficient in this case).
Hint - you may need to search more than the top 3 periodogram values to find the 3 peaks.
Can you write Python code for it?

from astropy.stats import LombScargle

frequency, power = # complete

print('Top LS period is {}'.format(# complete
print( # complete