I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometry.ipynb".
The first task is:
Problem 5c
Following 4a write a function to simulate many realizations of the star and estimate  the flux and variance using the PSF model.
How does the PSF estimate compare to the aperture estimate?
Can you write Python code for it?

# complete
# complete
# complete
# complete
# complete

print("The mean flux = {:.3f} with variance = {:.3f}".format( # complete


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometry.ipynb".
The first task is:
Upshot
Dropping simple apertures on an image provides a fast and simple method to estimate the flux of a star.
This approach comes at a cost, however, as the aperture method employed here provides high variance estimates of the flux.
Fortunately, it is possible to do much better via PSF photometry (and in fact, the Cramer-Rao bound mathematically proves that PSF photometry is the lowest variance estimator of the flux of a star). This means that aperture photometry is never better than PSF photometry despite some claims to contrary in the literature. There are cases where the PSF is extremely difficult to estimate, in which case aperture photometry may be the only decent way to estimate the flux, but even then PSF photometry would be better.
(Technically speaking, aperture photometry is PSF photometry. The catch is that the PSF model (a 1D or circular top hat) is a terrible match to the actual aparition of the stars on the image. When the model of the PSF is good, and in the case of our simulated data set we know the PSF perfectly, then PSF flux estimates will be a minimum variance estimator.)
Problem 5/Challenge Problem) PSF Flux measurement
We are going to cover PSF modeling and PSF photometry in far greater detail later this week, but here we are going to quickly meausure the flux using a model of the PSF, which we will compare to the aperture results.
Problem 5a
Create the psf model, psf, which is equivalent to a noise-free star with fwhm = 5.
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometry.ipynb".
The first task is:
We will now perform aperture photometry. In optical astronomy, most apertures are a top-hat function, and the flux is estimated by multiplying the aperture by the (background-subtracted) signal and summing the resulting output. In 2D apertures are typically a circle, for the remainder of this 1D problem we will use a top-hat aperture. If the aperture contains partial pixels, then the counts in pixels with parial coverage are multiplied by the fractional coverage of the pixel.
Problem 3b
Using an aperture with a radius of 5 pixels centered on the source (i.e. the aperture is 10 pixels long), measure the flux from a star centered at mu = 20, with fwhm = 5, S = 100, and F = 1000. Assume you can perfectly measure the background, and subtract this prior to measuring the flux.
Extra long Hint - if you create your aperture using a single np.where() command (or similar) you are not going to get the correct answer. To quickly see why this is the case, imagine you'd been asked to use an aperture with a radius of 3.2 pixels. Thus, properly calculating the aperture requires a bit of thought. A for loop is a fairly intuitive way to handle this problem (though it can also be done with a series of where commands, and both possibilities will be presented in the solutions).
DO NOT SPEND TOO MUCH TIME ON THIS PROBLEM If you get stuck, use a single np.where() command. When you plot the results in the 3c you'll see how things are off, but this won't affect the general results in the remainder of the notebook.
Can you write Python code for it?

x = np.linspace(0,40,41)
mu = 20
S = 100
fwhm = 5
F = 1000
sim_star = # complete

ap_radius = # complete

# complete
# complete
# complete
# complete
# complete

# complete
# complete
# complete

print("The star has flux = {:.3f}".format( # complete


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometry.ipynb".
The first task is:
write your answer here
We will now examine alternative methods of estimating the flux.
Problem 3) Aperture Flux Measurements
In some sense, 2c provides the most simplistic method for estimating the flux of star (add up all the counts after subtracting th background). This solution, however, cannot generalize to any sort of real life problem because there are always multiple stars (and galaxies) detected in every individual image.
However, we can approximate the above procedure by "isolating" the individual stars in any image (say by taking a 10x10 cutout around each star), and then estimating the flux in a similar fashion. As we are about to show, the size of the box (or more appropriately a circle for 2D optical images) is critical when estimating the flux.
Problem 3a
Write a function simulate() to simulate the noisy count measurements of a star with centroid mu, FWHM fwhm, sky background S, and flux F.
Hint - it may be helpful to plot the output of your function.
Can you write Python code for it?

def simulate(x, mu, fwhm, S, F):
    '''simulate a noisy stellar signal

    Parameters
    ----------
    x : array-like
        detector pixel number
    mu : float
        mean position of the 1D star
    fwhm : float
        Full-width half-maximum of the stellar profile on the detector
    S : float
        Constant sky background in each pixel
    F : float
        Total stellar flux

    Returns
    -------
    noisy_counts : array-like (same shape as x)
        the (noisy) number of counts in each pixel
    '''
    # complete
    # complete
    # complete

    noisy_counts = # complete

    return noisy_counts


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometrySolutions.ipynb".
The first task is:
Problem 1b
Plot the noise-free PSF for a star with a profile defined by $\mu = 10$ and $\mathrm{FWHM} = 3$.
Estimate the total flux of this star by "integrating" over the counts measured by the detector. If you are clever in the definition of your pixels, this integration reduces to a sum.
Hint - think about your input grid of pixels. Can you have a non-integer number of pixels? Also - the flux should be evaluated at the center of the pixel.
Can you write Python code for it?

x = np.linspace(0,30,31)

pixel_plot(x, phi(x+.5, 10, 3))

print("The flux of the star is: {:.3f}".format(sum(phi(x+.5, 10, 3))))


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometrySolutions.ipynb".
The first task is:
Problem 3c
Plot the simulated counts from 3b and overplot your aperture. You may want to multiple the aperture by a factor of 100 to see it better.
Hint – after you have run pixel_plot() you will need to call matplotlib functions directly (e.g., plt.plot) to overplot on the pixel data. Also, if you created an aperture_mask in 3b it may help to plot that as well.
Can you write Python code for it?

pixel_plot(x, sim_star-S)
# add aperture
plt.plot([0,mu-ap_radius, mu-ap_radius, mu+ap_radius,mu+ap_radius, np.max(x)], [0,0,100,100,0,0])
# add aperture_mask -- only different with non-integer aperture radius
plt.step(x, aperture_mask*100, where='post')


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometrySolutions.ipynb".
The first task is:
Problem 3c
Write a Monte Carlo simulator to estimate the mean and standard deviation of the flux from the simulated star.
Food for thought - what do you notice if you run your simulator many times?
Can you write Python code for it?

sim_fluxes = np.empty(1000)
for sim_num, dummy in enumerate(sim_fluxes):
    sim_star = simulate(x, mu, 5, S, 1000)

    ap_radius = 5
    aperture_mask = np.zeros_like(x)
    # somewhat vectorized solution
    aperture_mask[np.where((np.abs(x + 1 - mu) <= ap_radius) &
                      (np.abs(x - mu) <= ap_radius))] = 1
    low_pix = np.where((np.abs(x + 1 - mu) <= ap_radius) &
                       (np.abs(x - mu) > ap_radius))
    aperture_mask[low_pix] = ap_radius - (mu - (x[low_pix] + 1))
    high_pix = np.where((np.abs(x + 1 - mu) > ap_radius) &
                       (np.abs(x - mu) <= ap_radius))
    aperture_mask[high_pix] = ap_radius - (x[high_pix] - mu)

    ap_flux = np.sum((sim_star - S)*aperture_mask)

    sim_fluxes[sim_num] = ap_flux

print("The mean flux = {:.3f} with variance = {:.3f}".format(np.mean(sim_fluxes),
                                                             np.var(sim_fluxes, ddof=1)))


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometrySolutions.ipynb".
The first task is:
Problem 4) Avoid Arbitrary Numbers -- the Curve of Growth
In Problem 3 we decided to use an aperture radius of 5. Why?
(In practice, an aperture radius equal to the FWHM is a pretty good choice, as we will show below. However, this is not optimal in all situations)
We will now try to optimize the choice of aperture for the star in question.
Problem 4a
Using your solution to Problem 3, write a function aperture_flux() that estimates the mean flux and it's variance in a given aperture of size ap_radius for a simulated star.
Hint - this function is going to have many inputs, ap_radius, the position of the star, the flux of the star, the FWHM of the star, a pixel grid, the value of the sky background, and the number of simulations per input radius.
Can you write Python code for it?

def aperture_flux(ap_radius,
                  x=np.linspace(0,40,41),
                  mu=20,
                  fwhm=5,
                  S=100,
                  F=1000,
                  n_sim=1000):
    '''Measure mean and variance of flux in a given aperture

    Parameters
    ----------
    ap_radius : float
        radius of the top hat aperture
    x : array-like (default: np.arange(0,40,41))
        grid of simulated pixels
    mu : float (default: 20)
        position of the star
    fhwm : float (default: 5)
        FWHM of the stellar profile
    S : float (default: 100)
        level of the sky background
    F : float (default: 1000)
        stellar flux
    n_sim : float (default: 1000)
        number of simulations

    Returns
    -------
    mean_flux, variance
    '''

    sim_fluxes = np.empty(n_sim)
    for sim_num, dummy in enumerate(sim_fluxes):
        sim_star = simulate(x, mu, fwhm, S, F)

        aperture_mask = np.zeros_like(x)
        # somewhat vectorized solution
        aperture_mask[np.where((np.abs(x + 1 - mu) <= ap_radius) &
                          (np.abs(x - mu) <= ap_radius))] = 1
        low_pix = np.where((np.abs(x + 1 - mu) <= ap_radius) &
                           (np.abs(x - mu) > ap_radius))
        aperture_mask[low_pix] = ap_radius - (mu - (x[low_pix] + 1))
        high_pix = np.where((np.abs(x + 1 - mu) > ap_radius) &
                           (np.abs(x - mu) <= ap_radius))
        aperture_mask[high_pix] = ap_radius - (x[high_pix] - mu)

        ap_flux = np.sum((sim_star - S)*aperture_mask)

        sim_fluxes[sim_num] = ap_flux
    return np.mean(sim_fluxes), np.var(sim_fluxes, ddof=1)


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometrySolutions.ipynb".
The first task is:
Problem 4b
Confirm your function works by calculating the mean and variance of the flux in a 5 pixel radius aperture.
Can you write Python code for it?

mean, var = aperture_flux(5)

print('The mean flux in a r = 5 pix aperture is {:.4f} +/- {:.4f}'.format(mean, np.sqrt(var)))


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometrySolutions.ipynb".
The first task is:
Problem 4e
Plot the uncertainty on the flux estimate (i.e., the square root of the variance) as a function of aperture radius.
Now which aperture size do you think is best?
Can you write Python code for it?

fig, ax = plt.subplots()
ax.plot(ap_array, np.sqrt(ap_var)/cog)
ax.set_xlabel('aperture radius (pix)')
ax.set_ylabel('uncertainty in F')
fig.tight_layout()


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometrySolutions.ipynb".
The first task is:
Problem 5b
Using the same parameters as problem 3, simulate a star and measure it's PSF flux.
Hint - you may find the minimize function from scipy.optimize helpful.
Can you write Python code for it?

from scipy.optimize import minimize

# minimize the square of the residuals to determine flux
def sum_res(A, flux=sim_star-S, model=psf):
    return sum((flux - A*model)**2)

sim_star = simulate(x, mu, fwhm, S, F)
psf_flux = minimize(sum_res, 980, args=(sim_star-S, psf))

print("The PSF flux is {:.3f}".format(psf_flux.x[0]))


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometrySolutions.ipynb".
The first task is:
Problem 5c
Following 4a write a function to simulate many realizations of the star and estimate  the flux and variance using the PSF model.
How does the PSF estimate compare to the aperture estimate?
Can you write Python code for it?

sim_fluxes = np.empty(5000)
for sim_num, dummy in enumerate(sim_fluxes):
    sim_star = simulate(x, mu, fwhm, S, F)
    psf_flux = minimize(sum_res, 980, args=(sim_star-S, psf))

    sim_fluxes[sim_num] = psf_flux.x[0]

print("The mean flux = {:.3f} with variance = {:.3f}".format(np.mean(sim_fluxes),
                                                             np.var(sim_fluxes, ddof=1)))


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometrySolutions.ipynb".
The first task is:
write your answer here
The estimated flux is much, much, much larger than the true flux of the star.
Problem 2c
The flux has been measured incorrectly because we are counting photons from the "sky."
Subtract the sky background S from the counts in the detector and once again estimate the flux using the same method as 1b.
Does your estimate match your expectation? (recall that $F = 500$)
Note - estimating the value of the "sky" background in real life is extremely difficult and an entire lecture will be devoted to this topic
Can you write Python code for it?

print('The total flux = {:.4f}'.format(np.sum(signal_plus_noise - S)))


I am working on the file "Sessions/Session11/Day1/IntroductionToBasicStellarPhotometrySolutions.ipynb".
The first task is:
write your answer here
The optimal radius appears to be ~4 pixels. (Note - this is really close to the FWHM as we discussed previously)
Here we have discovered a universal truth about aperture photometry: very small and very large apertures produce lower signal-to-noise estimates than something in between.
However, the optimal value of that something in between is different for every star (as you will show below).
Problem 4f
Leaving all other variables the same, estimate the optimal aperture size (i.e. maximize the signal-to-noise ratio) for a star with a flux of 10000.
What is the optimal aperture size?
Hint –– you only need to repeat 4c and 4e for this answer.
Can you write Python code for it?

ap_mean = np.zeros_like(ap_array)
ap_var = np.zeros_like(ap_array)

for ap_num, ap_rad in enumerate(ap_array):
    ap_mean[ap_num], ap_var[ap_num] = aperture_flux(ap_rad, F=1e4, n_sim=3000)

fig, ax = plt.subplots()
ax.plot(ap_array, np.sqrt(ap_var)/cog)
ax.set_xlabel('aperture radius (pix)')
ax.set_ylabel('uncertainty in F')
fig.tight_layout()

print('The optimal aperture is {} pix'.format(ap_array[np.argmin(np.sqrt(ap_var)/cog)]))


I am working on the file "Sessions/Session11/Day1/InvestigatingDetectors.ipynb".
The first task is:
5) Measuring the gain
We have just one more step before we can report the read noise. We need to measure the gains so we can convert the noise in ADU into electrons. To do that, we're going to use the trick we saw at the start of this notebook. We need to add two things though: we want to use pairs of images, to cancel out any fixed spatial patterns, and we need images with significant counts in them so that we're not just measuring read noise. The formula we want to implement is thus:
$$ \frac{1}{\textrm{gain}} =  \langle \frac{(I_1 - I_2)^2}{I_1 + I_2} \rangle $$
where $I_1$ and $I_2$ are the pixel values from each image, and the $\langle$ $\rangle$ brackets denote taking the mean of this ratio over all pixels.
We have some flat field images from those same sensors that we can use:
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day1/InvestigatingDetectors.ipynb".
The first task is:
Before diving in to programming, take a careful look at the components in the simulated image. What are the two noise sources, and why do they have that functional form? We're going to be looking a lot at the image "gain"; does it make sense how that is applied?
Let's make some simulations. What we want to do is loop over a set of input levels light levels, from zero to "full well" capacity (on order of 10,000 electrons). For each simulated image, we want to measure the mean signal level (because that's what we see as users of a CCD) and the standard deviation of that image. Save those in two lists, but at the end convert those back to numpy arrays to make downstream usage easier.
For right now, set the read noise to 5, and the gain to 0.8.
Can you write Python code for it?

# Question
noise_level_list = # complete
measured_signal_level_list = # complete
input_signal_levels = # complete

for # complete
    # complete
    # complete

noise_levels = # complete
measured_signal_levels = # complete


I am working on the file "Sessions/Session11/Day1/InvestigatingDetectors.ipynb".
The first task is:
Photon Transfer Curve
1) Simulated Images
The "Photon Transfer Curve" is the name given to the relationship between the signal level and the noise level in a sensor. We're going to do a few experiments to show how it works in principle, and then we'll look at some real images and make some diagnostic measurements.
First we need a model of the noise in CCD image. I'm going to give this to you so we all start out on the same page
Can you write Python code for it?

def simulated_image(signal_level, read_noise, gain):
    """
    Return a 1-D simulated "image" with the noise properties of
    a CCD sensor. The image is always 1000 pixels long.

    signal_level is the mean number of electrons in each pixel.
    read_noise is the noise of the readout amplifier, in electrons.
    gain is the number of electrons per ADU.

    """
    return (1/gain) * (read_noise*np.random.randn(1000) +  np.random.poisson(signal_level, size=1000))


I am working on the file "Sessions/Session11/Day1/InvestigatingDetectors.ipynb".
The first task is:
Plot the noise vs. the measured signal level, on a log-log plot.
Can you write Python code for it?

# Question

# Complete

plt.ylabel("RMS")
plt.xlabel("Measured Signal Level")


I am working on the file "Sessions/Session11/Day1/InvestigatingDetectors.ipynb".
The first task is:
Since each amplifier can have a slightly different gain, we want to run this per-HDU and output a table of values. Since we're looping over the HDUs, we can also print the finished read noise values at the same time. Note that those have a factor of $\sqrt{2}$ because we took the difference of two bias frames, so the noise is greater than a single image.
Can you write Python code for it?

# Question
for hdu in range(1,16):
    flat1_data = flat1_file[ # complete
    flat2_data = flat2_file[ # complete

    debiased_flat1 = # complete
    debiased_flat2 = # complete

    squared_noise = # complete
    summed_intensity = # complete

    # Some pixels with low counts are likely artifacts and can skew the measurement.
    # It helps to only keep pixels that have significant flux; you can experiment with this cutoff
    ok_values = (summed_intensity > # complete

    # Remember that as we defined gain above, the formula returns 1/gain.
    reciprocal_gain = # complete

    print(hdu, reciprocal_gain, 1/reciprocal_gain)


I am working on the file "Sessions/Session11/Day1/InvestigatingDetectorsSolutions.ipynb".
The first task is:
5) Measuring the gain
We have just one more step before we can report the read noise. We need to measure the gains so we can convert the noise in ADU into electrons. To do that, we're going to use the trick we saw at the start of this notebook. We need to add two things though: we want to use pairs of images, to cancel out any fixed spatial patterns, and we need images with significant counts in them so that we're not just measuring read noise. The formula we want to implement is thus:
$$ \frac{1}{\textrm{gain}} =  \langle \frac{(I_1 - I_2)^2}{I_1 + I_2} \rangle $$
where $I_1$ and $I_2$ are the pixel values from each image, and the $\langle$ $\rangle$ brackets denote taking the mean of this ratio over all pixels.
We have some flat field images from those same sensors that we can use:
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day1/InvestigatingDetectorsSolutions.ipynb".
The first task is:
Before diving in to programming, take a careful look at the components in the simulated image. What are the two noise sources, and why do they have that functional form? We're going to be looking a lot at the image "gain"; does it make sense how that is applied?
Let's make some simulations. What we want to do is loop over a set of input levels light levels, from zero to "full well" capacity (on order of 10,000 electrons). For each simulated image, we want to measure the mean signal level (because that's what we see as users of a CCD) and the standard deviation of that image. Save those in two lists, but at the end convert those back to numpy arrays to make downstream usage easier.
For right now, set the read noise to 5, and the gain to 0.8.
Can you write Python code for it?

# Answer
noise_levels = []
measured_signal_levels = []
input_signal_levels = np.logspace(0, 4, 30)

for input_signal_level in input_signal_levels:
    image = simulated_image(input_signal_level, 5, 0.8)
    noise_levels.append(np.std(image))
    measured_signal_levels.append(np.mean(image))

noise_levels = np.array(noise_levels)
measured_signal_levels = np.array(measured_signal_levels)


I am working on the file "Sessions/Session11/Day1/InvestigatingDetectorsSolutions.ipynb".
The first task is:
Now we're going to plot something slightly different. Plot the variance this time, and on a linear plot instead of log-log (again vs. measured signal level). Fit a straight line to the data (in linear space) and print the coefficients. Also print the reciprocal of the slope.
Where did this slope come from?
Can you write Python code for it?

# Answer
high_counts, = np.where(measured_signal_levels > 100)

plt.plot(measured_signal_levels[high_counts], (noise_levels[high_counts])**2, 'ko')
fit = np.polyfit(measured_signal_levels, noise_levels**2, 1)
print(1/fit)

plt.plot(x, fit[0] * x, '-')
plt.ylabel("Variance")
plt.xlabel("Measured Signal Level")


I am working on the file "Sessions/Session11/Day1/InvestigatingDetectorsSolutions.ipynb".
The first task is:
Plot the noise vs. the measured signal level, on a log-log plot.
What is the behavior you see? What are the two different noise regimes?
Fit a straight line to the "bright" portion of the data (high signal levels) and print the resulting coefficients. Remember that you're looking at a log-log plot, and so you want to fit the logs of the variables. You can add this to the plot in the cell above.
Why does the line have that value of the slope?
Can you write Python code for it?

# Answer
high_counts, = np.where(signal_levels > 100)
fit = np.polyfit(np.log10(measured_signal_levels[high_counts]), np.log10(noise_levels[high_counts]), 1)
print(fit)
x = np.logspace(2, 4, 10)
plt.loglog(measured_signal_levels, noise_levels, 'ko')
plt.plot(x, 10**(fit[0]*np.log10(x) ), '-')
plt.ylabel("Standard Deviation")
plt.xlabel("Measured Signal Level")


I am working on the file "Sessions/Session11/Day1/InvestigatingDetectorsSolutions.ipynb".
The first task is:
Since each amplifier can have a slightly different gain, we want to run this per-HDU and output a table of values. Since we're looping over the HDUs, we can also print the finished read noise values at the same time. Note that those have a factor of $\sqrt{2}$ because we took the difference of two bias frames, so the noise is greater than a single image.
Can you write Python code for it?

# Answer
for hdu in range(1,16):
    flat1_data = flat1_file[hdu].data
    flat2_data = flat2_file[hdu].data

    bias1 = bias1_file[hdu].data
    bias2 = bias2_file[hdu].data

    debiased_flat1 = flat1_data - bias1
    debiased_flat2 = flat2_data - bias2

    squared_noise = (debiased_flat1 - debiased_flat2 )**2
    summed_intensity = ((debiased_flat1) + (debiased_flat2))
    ok_values = (summed_intensity > 5000)
    reciprocal_gain = np.mean(squared_noise[ok_values]/(summed_intensity[ok_values]) )

    print(hdu, reciprocal_gain, 1/reciprocal_gain, measured_stddevs[hdu]/reciprocal_gain/np.sqrt(2))


I am working on the file "Sessions/Session11/Day1/InvestigatingDetectorsSolutions.ipynb".
The first task is:
The slope here is related to the gain (either proportionally or inversely, depending on how one chooses to define gain). This can be summarized as
$$ \frac{1}{\textrm{gain}} =  \langle \frac{\textrm{Variance}}{\textrm{Mean Signal Level}} \rangle $$
It's a clever and useful trick, or at least it seems like a trick, because the standard deviation plot wasn't affected by the gain at all. Go back and try varying the gain and re-run the plots, and you'll see what does and doesn't change.
One way to think of it is that the measured signal level is affected linearly by the gain, but the variance is affected by the square of the gain. Dividing these two gives you a linear relation back, but when dividing the square root of the variance, the gain cancels out.
2) Looking at real bias frames
Remember that a bias frame is an image that is exposed for zero seconds; it's just immediately read-out without being exposed to light. You might think that is a pretty boring image, particularly if you're at the telescope and getting ready for a night of observing. But to a camera engineer, bias frames hold lots of information about how the camera is operating.
We're going to look at images from one example LSST sensor; this was taken on a test stand and not the actual camera, so don't take it as representative of real camera performance.
Our first step is, as usual, to look at the image and make sure it seems reasonable.
Can you write Python code for it?

bias1_file = fits.open("00258334360-S10-det003.fits")
bias1_data = bias1_file[1].data

plt.imshow(bias1_data, cmap='gray',
           vmin=(np.median(data) - 0.2*np.std(data)),
           vmax=(np.median(data) + 0.2*np.std(data)))
plt.colorbar()


I am working on the file "Sessions/Session11/Day2/FindingSources.ipynb".
The first task is:
1.3.1) Bin the image. Plot the bin centers. What bin size did you pick?
Can you write Python code for it?

bin_centers = # complete

bin_values = # complete

# overplot the binned esimtates:
ax.plot(bin_centers, bin_values, 'o')


I am working on the file "Sessions/Session11/Day2/FindingSources.ipynb".
The first task is:
1.3.2) Spatially model the binned estimates (bin_values vs bin_centers) as a chebyshev polynomial.
Evaluate your model on the image grid and overplot.(what degree/order did you pick?)
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day2/FindingSources.ipynb".
The first task is:
2.2) Correlate the image with the PSF kernel,
and plot the result.
What are the tradeoffs when choosing the size of your PSF kernel? What happens if its too big? What happens if it's too small.?
hint: scipy.signal.convolve
Can you write Python code for it?

# detection_image = # complete

# plot your detection image
# Note: pay attention to how scipy.signal.convolve handles the edges.


I am working on the file "Sessions/Session11/Day2/FindingSources.ipynb".
The first task is:
Problem 1) Simple 1-D Background Estimation
Problem 1.1) Estimate a the background as a constant offset (order = 0)
For this problem we will use a simulated star with a constant background offset of $S=100$.
Background estimation is typically done by inspecting the distribution of counts in pixel bins. First inspect the distribution of counts, and pick an estimator for the background that is robust to the star (reduces biased fromthe star). Remember that we haven't done detection yet and don't know where the sources are.
Can you write Python code for it?

# simulate the star
x = np.linspace(0, 100)
mu = 35
S = 100
fwhm = 5
F = 500

fig = plt.figure(figsize=(8,4))
ax = plt.subplot()
sim_star = # complete
pixel_plot(x, sim_star, fig=fig, ax=ax)

# plot and inspect histogram

fig = plt.figure(figsize=(6,4))
# complete

S_estimate = # complete

print('My background estimate = {:.4f}'.format(S_estimate)
print('The mean pixel count = {:.4f}'.format( # complete

# plot your background model over the "image"

fig, ax = pixel_plot(x, sim_star)
pixel_plot(x, # complete, fig=fig, ax=ax)


I am working on the file "Sessions/Session11/Day2/FindingSources.ipynb".
The first task is:
Problem 1.2) Estimate a the background as a ramp/line (order = 1)
Now let's simulate a slightly more complicated background a linear ramp: $y = 3x + 100$. First simulate the same star with the new background. Then we're going to fit it using the following steps:
* Bin the image
* Use your robust estimator to estimate the background value per bin center
* Fit these bin center with a model

A common simple model that astronomers use are Chebyshev polynomials. Chebyshevs have some very nice properties that prevent ringing at the edges of the fit window. Another popular way to "model" the bin centers is non-parametrically via interpolation.
Can you write Python code for it?

# Double check that your simulate function can take S optionally as array-like

# Create and plot the image with S = 3*x + 100

sim_star = # complete
pixel_plot(x, sim_star)

# bin the image in 20-pixel bins

# complete
bin_centers = # complete
bin_values = # complete

# Fit the bin_values vs bin_centers with a 1st-order chebyshev polynomial
# Evaluate your model for the full image
# hint: look up np.polynomial.chebyshev.chebfit and np.polynomial.chebyshev.chebeval

# complete

# Replot the image:
fig, ax = pixel_plot(x, sim_star)
# binned values
ax.plot(bin_centers, bin_values, 'o')

# Overplot your background model:
# complete

# Finally plot your background subtracted image:

# complete


I am working on the file "Sessions/Session11/Day2/FindingSources.ipynb".
The first task is:
Problem 1.3) Estimate a more realistic background (still in 1D)
Now repeat the the exercise in problem 1.2 with a more complex background.
Can you write Python code for it?

SIGMA_PER_FWHM = 2*np.sqrt(2*np.log(2))

fwhm = 5
x = np.linspace(0, 100)
background = 1000*norm.pdf(x, 50, 18) + 100*norm.pdf(x, 20, fwhm/SIGMA_PER_FWHM) + 100*norm.pdf(x, 60, fwhm/SIGMA_PER_FWHM)

sim_star3 = simulate(x=x, mu=35, fwhm=fwhm, S=background, F=200)
fig, ax = pixel_plot(x, sim_star3)


I am working on the file "Sessions/Session11/Day2/FindingSources.ipynb".
The first task is:
Problem 2) Finding Sources
Now that we have subtracted background image,  let’s look for sources.  In the lecture we focused on the matched filter interpretation.  Here we will go into the hypothesis testing and maximum likelihood interpretations.
Maximum likelihood interpretation:
Assume that we know there is a point source somewhere in this image.   We want to find  to pixel that has the maximum likelihood of having a point source centered on it.  recall from session 10, the probability for an  individual observation $I_i$ is:
$$P(X_i) = \frac{1}{\sqrt{2\pi\sigma_i^2}} \exp{-\frac{(X_i - y_i)^2}{2\sigma_i^2}}$$
Here: $X_i$ is the pixel value of pixel $i$ in the image and $y_i$ is the model prediction for that pixel.
The model in this case is your simulate() function from the IntroductionToBasicStellarPhotometry.ipynb notebook: the PSF evaluated at a distance from the center multiplied by the flux amplitude: $F * \phi(x - x_{center}) + S$ Where $F$ is the flux amplitude $\phi$ is the PSF profile (a function of position), and $S$ is the background.
Plug it in:
$$P(X_i) = \frac{1}{\sqrt{2\pi\sigma_i^2}} \exp{-\frac{(X_i - (F * \phi_i(x_{center}) + S))^2}{2\sigma_i^2}}$$
Hypothesis test interpretation:
If I were teaching source detection to my non-scientist, college stats 101 students, I'd frame the problem like this:
Pretend you have an infinitely large population of pixels Say I know definitively, that the arbitrarily large population of pixels is drawn from $N(0,100)$ (has a variance of 10). I have another sample of 13 pixels. I want to tesst the hypothesis that those 13 pixels were drawn from the $N(0,100)$ pop too.
Test the hypothesis that your subsample of 13 pixels were drawn from the larger sample.
* $H_0$: $\mu = 0$
* $H_A$: $\mu > 0$
$$z = \frac{\bar{x} - \mu}{\sigma / \sqrt{n}} $$
$$z = \frac{\sum{x}/13 - 0}{10 /\sqrt{13}} $$
OK, if this is coming back now, let's replace this with our real estimator for PSF flux, which is a weighted mean of the pixels where the weights are the PSF $\phi_i$....
Can you write Python code for it?

# set up simulation
x = np.linspace(0, 100)
mu = 35
S = 100
fwhm = 5
F = 300

fig = plt.figure(figsize=(8,4))
ax = plt.subplot()
sim_star = simulate(x, mu=mu, fwhm=fwhm, S=S, F=F)

# To simplify this pretend \sqrt{\sum \phi_i^2 \sigma_i^2we know for sure the background = 100
# Plots the backround subtracted image
image = sim_star - 100
pixel_plot(x, image, fig=fig, ax=ax)


I am working on the file "Sessions/Session11/Day2/FindingSourcesSolutions.ipynb".
The first task is:
1.3.2) Spatially model the binned estimates (bin_values vs bin_centers) as a chebyshev polynomial.
Evaluate your model on the image grid and overplot.(what degree/order did you pick?)
Can you write Python code for it?

fig, ax = pixel_plot(x, sim_star3)
ax.plot(bin_centers, bin_values, 'o')

coefficients = np.polynomial.chebyshev.chebfit(bin_centers, bin_values, 2)
bg = np.polynomial.chebyshev.chebval(x, coefficients)
ax.plot(x, bg, '-', label='order=2')

coefficients = np.polynomial.chebyshev.chebfit(bin_centers, bin_values, 3)
bg = np.polynomial.chebyshev.chebval(x, coefficients)
ax.plot(x, bg, '-', label='order=3')

ax.legend()


I am working on the file "Sessions/Session11/Day2/FindingSourcesSolutions.ipynb".
The first task is:
1.3.3) Subtract off the model and plot the "background-subtracted image."
Can you write Python code for it?

# Plot the background subtracted image
fig, ax = pixel_plot(x, sim_star3 - bg)


I am working on the file "Sessions/Session11/Day2/FindingSourcesSolutions.ipynb".
The first task is:
2.1) Make a kernel for the PSF.
Properties of kernels: They're centered at x=0 (which also means that they have an odd number of pixels) and sum up to 1. You can use your phi().
Can you write Python code for it?

xx = np.arange(-7, 8)
kernel = phi(xx, mu=0, fwhm=5)
pixel_plot(xx, kernel)
print(xx)
print(kernel)


I am working on the file "Sessions/Session11/Day2/FindingSourcesSolutions.ipynb".
The first task is:
2.2) Correlate the image with the PSF kernel,
and plot the result.
What are the tradeoffs when choosing the size of your PSF kernel? What happens if its too big? What happens if it's too small.?
hint: scipy.signal.convolve
Can you write Python code for it?

import scipy.signal

size = len(kernel)//2
detection_image = scipy.signal.convolve(image, kernel, mode='same')
#  mode='same' pads then clips the padding This is the same as:
#         size = len(kernel)//2
#         scipy.signal.convolve(image, kernel)[size:-size]

# Note: pay attention to how scipy.signal.convolve handles the edges.
pixel_plot(x, detection_image)

print(len(scipy.signal.convolve(image, kernel, mode='full')))
print(len(scipy.signal.convolve(image, kernel, mode='same')))
print(len(scipy.signal.convolve(image, kernel, mode='valid')))


I am working on the file "Sessions/Session11/Day2/FindingSourcesSolutions.ipynb".
The first task is:
2.4) Dilate footprint to provide a window or region for the point source.
We will use this window to compute the centroid and total flux of the star in the next two lessons. In the meantime, compute the flux like we did in introductionToStellarPhotometry assuming the input center.
Can you write Python code for it?

# complete
growBy = fwhm
mask = detection_image > threshold_value
print(np.count_nonzero(mask))
dilated_mask = scipy.ndimage.binary_dilation(mask, iterations=growBy)
print(np.count_nonzero(dilated_mask))

fig, ax = pixel_plot(x[dilated_mask], image[dilated_mask])

# easy aperture flux:
np.sum(image[dilated_mask])

# Copied from solutions of previous notebook

from scipy.optimize import minimize

psf = phi(x[dilated_mask], mu=35, fwhm=5)
im = image[dilated_mask]

# minimize the square of the residuals to determine flux
def sum_res(A, flux=im, model=psf):
    return sum((flux - A*model)**2)

sim_star = simulate(x, mu, fwhm, S, F)
psf_flux = minimize(sum_res, 300, args=(im, psf))

print("The PSF flux is {:.3f}".format(psf_flux.x[0]))


I am working on the file "Sessions/Session11/Day2/FindingSourcesSolutions.ipynb".
The first task is:
And now you can see that this problem is fairly unrealistic as far as background subtraction goes and should probably be treated with a deblender. Typically in images,
* For chebyshev polynomials we use bin sizes of at least 128 pixels and orders of no more than 6. The spatial scale is controlled by the order, and bin size is less important. In fact you could probably use bin size of 1 if you really wanted to.
* For interpolation, the spatial scale is controlled by the bin size. We usually choose bins >= 256 pixels.
Problem 2) Finding Sources
Now that we have subtracted background image,  let’s look for sources.  In the lecture we focused on the matched filter interpretation.  Here we will go into the hypothesis testing and maximum likelihood interpretations.
Maximum likelihood interpretation:
Assume that we know there is a point source somewhere in this image.   We want to find  to pixel that has the maximum likelihood of having a point source centered on it.  recall from session 10, the probability for an  individual observation $I_i$ is:
$$P(X_i) = \frac{1}{\sqrt{2\pi\sigma_i^2}} \exp{-\frac{(X_i - y_i)^2}{2\sigma_i^2}}$$
Here: $X_i$ is the pixel value of pixel $i$ in the image and $y_i$ is the model prediction for that pixel.
The model in this case is your simulate() function from the IntroductionToBasicStellarPhotometry.ipynb notebook: the PSF evaluated at a distance from the center multiplied by the flux amplitude: $F * \phi(x - x_{center}) + S$ Where $F$ is the flux amplitude $\phi$ is the PSF profile (a function of position), and $S$ is the background.
Plug it in:
$$P(X_i) = \frac{1}{\sqrt{2\pi\sigma_i^2}} \exp{-\frac{(X_i - (F * \phi_i(x_{center}) + S))^2}{2\sigma_i^2}}$$
Hypothesis test interpretation:
If I were teaching source detection to my non-scientist, college stats 101 students, I'd frame the problem like this:
Pretend you have an infinitely large population of pixels Say I know definitively, that the arbitrarily large population of pixels is drawn from $N(0,100)$ (has a variance of...
Can you write Python code for it?

# set up simulation
x = np.arange(100)
mu = 35
S = 100
fwhm = 5
F = 300

fig = plt.figure(figsize=(8,4))
ax = plt.subplot()
sim_star = simulate(x, mu=mu, fwhm=fwhm, S=S, F=F)

# To simplify this pretend we know for sure the background = 100
# Plots the backround subtracted image
image = sim_star - 100
pixel_plot(x, image, fig=fig, ax=ax)


I am working on the file "Sessions/Session11/Day2/FindingSourcesSolutions.ipynb".
The first task is:
Answer to the question: Bigger PSF kernels =  more accurate convolution, more pixels lost on the edges, and more expensive computation. Smaller kernels don't get close enough to zero on the edges and resulting correlated image can look "boxy"
2.3) Detect pixels
for which the null hypothesis that there's no source centered there is ruled out at the 5$\sigma$ level.
Can you write Python code for it?

# Using a robust estimator for the detection image standard deviation,
# Compute the 5 sigma threshold
N_SIGMA = 5
q1, q2 = np.percentile(detection_image, (30.85, 69.15))
std = q2 - q1

threshold_value = std * N_SIGMA
print('5 sigma threshold value is = {:.4f}'.format(threshold_value))


I am working on the file "Sessions/Session11/Day2/FindingSourcesSolutions.ipynb".
The first task is:
Problem 1.3) Estimate a more realistic background (still in 1D)
Now repeat the the exercise in problem 1.2 with a more complex background.
Can you write Python code for it?

SIGMA_PER_FWHM = 2*np.sqrt(2*np.log(2))

fwhm = 5
x = np.arange(100)
background = 1000*norm.pdf(x, 50, 18) + 100*norm.pdf(x, 20, fwhm/SIGMA_PER_FWHM) + 100*norm.pdf(x, 60, fwhm/SIGMA_PER_FWHM)

sim_star3 = simulate(x=x, mu=35, fwhm=fwhm, S=background, F=200)
fig, ax = pixel_plot(x, sim_star3)


I am working on the file "Sessions/Session11/Day2/FindingSourcesSolutions.ipynb".
The first task is:
The noise estimate is a little high, but not bad for the first iteration. In future interations we will mask the fotoprints detected in the initial round, before recomputing.
In practice, we use a sigma-clipped rms as the estimate of the standard deviation, in which we iteratively clip outliers until we have what looks like a normal distribution and compute a plain std.
Can you write Python code for it?

qq = scipy.stats.probplot(detection_image, dist="norm")
plt.plot(qq[0][0], qq[0][1])
plt.ylabel('value')
plt.xlabel('Normal quantiles')

# Just for fun to see what's going on:

fig, ax = pixel_plot(x, detection_image)
plt.axhline(threshold_value)


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotion.ipynb".
The first task is:
(a) Is this stellar image background-limited or source-limited?  To answer, consider which one contributes more noise within the half-light radius of the star.
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotion.ipynb".
The first task is:
(b) Now create a MoffatPSF instance with a true center of your choice (keep the default half_light_radius=4, which fits nicely into 32x32 images).  Make a loop that will create photon lists for each of 10,000 observations of a star with an average flux n_photon=1e3.  Use your function from (a) to create estimates $(\hat x_0, \hat y_0)$ of each star.  Then produce a histogram of these 10,000 estimates (one for $x$ and one for $y$), and figure out the standard deviations $\sigma_x$ and $\sigma_y$ of your measurements.
Tip: Draw your photon lists 1 at a time, then measure each and throw it away before you draw the next one.  We don't want to use up our memory keeping them all around at once.
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotion.ipynb".
The first task is:
(b) Now read the FITS catalogs into tables.  We will use the columns XWIN_IMAGE and YWIN_IMAGE to give us the pixel coordinates, with ERRAWIN_IMAGE giving the rough uncertainty on each (in units of pixels, 1 pixel = 0.264 arcsec).
By whatever means you choose, figure out which row in each catalog corresponds to the fast-moving star at the coordinates given above.  Then use the WCS's to obtain the ICRS RA & Dec measured for this star on these two images.
Can you write Python code for it?

# I'll start you off by giving the command to read in the catalog to a table:
cat_old = pf.getdata('old_catalog.fits',1)

# Your work here...


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotion.ipynb".
The first task is:
(b) Write a function aperture_centroid(img,R) that will return the values of $(y0,x0)$ that satisfy
$$ M_x = \sum_{xy} I_{xy} W(x-x0,y-y0) (x-x0)=0$$
(and the same for y0), where $W$ is a weight function. I provide for you below a class GaussAp which you can use to create a Gaussian-aperture weight function with some $\sigma_w$:
$$ W(x,y) = \exp{\left(-\frac{x^2+y^2}{2\sigma_w^2}\right)}$$
To do this, you'll need to start with an initial guess for $(y0,x0),$ and then calculate the values of $M_f, M_x$ and $M_y$.  Then move the aperture's $x_0,y_0$ by $(M_x/M_f, M_y/M_f)$ and try again.  Iterate until the centroid doesn't move any more (or until you have done 10 iterations and need to quit!).
Can you write Python code for it?

# Here's my weight function class
# Remember, to use a class, you first create an "instance"
# of it, e.g.:  weight = GaussAp(sigma=2.6)
# and then you can call this instance like a function of dx,dy

class GaussAp:
    # A tophat weight function of radius sigma
    def __init__(self,sigma):
        self.sigma = sigma
        return
    def __call__(self,dx,dy):
        # Given equal-shaped arrays dx=x-x0, dy=y-y0,
        # returns an array of the same shape giving weight function.

        # Calculate distance of a pixel from the center
        rsq = np.square(dx) + np.square(dy)

        # Now return the weight
        return np.exp(-rsq/(2.*self.sigma*self.sigma))

# Your work here...


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotion.ipynb".
The first task is:
(c) Now see whether our formula for the accuracy of source-limiting centroiding is accurate:
$$ \sigma_x = \frac{\sigma_{\rm PSF}}{\sqrt{N_\gamma}}$$
in this case.  You'll need to use your MoffatPSF instance to estimate its standard deviation width $\sigma_{\rm PSF}$.
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotion.ipynb".
The first task is:
(c) Using these two sky coordinates, estimate the rate of proper motion (in mas/yr) of the star between the two exposures.  Note that proper motions are traditionally given in real angular motions, so that (RA PM) = (difference in RA) * cos(dec) / (time interval). [You can ignore the fact that there might be parallax motion mixed in here.]
Give an uncertainty for our measurement as well.  Here we have to be careful, because ERRAWIN_IMAGE includes only the shot-noise errors.  It does not include the contribution from atmospheric turbulence, which we know is dominant for brighter stars.  So let's add another 10 mas of uncertainty in quadrature to each coordinate of each measurement as a guess of the size of the turbulence.
Does the DES measurement agree with Gaia's estimates?  Are there any reasons it might disagree?
pmra  =  326.06  +- 0.94    mas/yr
pmdec = -124.12 +- 0.81 mas/yr
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotion.ipynb".
The first task is:
Exercise 2: aperture centroids for background-limited stars
Now we're going to work with pixelized images and add background noise (say, $n=4000$). We know that simple centroiding will not be a good idea so we're going to need to use some apertures.  Before proceeding, let's just draw a background-noisy star:
Can you write Python code for it?

x0,y0 = 15.22,16.17
hlr = 4.
flux = 4e4
n_bg = 4e3
star = MoffatPSF(half_light_radius=4.,x0=x0, y0=y0)
img = star.realizeImage(n_pix=32,n_photon=flux)
addBackground(img, variance=n_bg)
pl.imshow(img,origin='lower',interpolation='nearest')
pl.colorbar()


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotion.ipynb".
The first task is:
Exercise 3: Using a WCS
This exercise requires four fits files: old_image.fits, new_image.fits, old_catalog.fits, and new_catalog.fits. Please download these files, and place them in the same directory as this notebook.
The 2 FITS images are from the Dark Energy Survey (DES), named old_image.fits and new_image.fits, and they overlap on the sky. According to Gaia DR2, there is a fast-moving star located at roughly $(\alpha,\delta)=$(29.91148,-8.212267). If you display these two images using DS9, then align them using the Frame->Match->Frame->WCS option, you will be able to move your cursor to those coordinates actually see the star move.
We also have catalogs (FITS tables) named old_catalog.fits and new_catalog.fits that contain the precise pixel centroids measured by SExtractor on these frames.  Your exercise is to extract the WCS from each image's header using astropy, apply it to the $(x,y)$ coordinates to obtain $(\alpha,\delta)$ for this star in each image, and then estimate its proper motion.
(a) Get a WCS and the MJD-OBS out of headers. [MJD is Modified Julian Date.  Basically it gives the time of the exposure in units of days since some reference moment.]
Can you write Python code for it?

# I'll start you off by constructing an astropy WCS from the first image
h = pf.getheader('old_image.fits')
wcs_old = WCS(h)
mjd_old = h['MJD-OBS']

# And here's an example of using the WCS to map to the sky:
radec = wcs_old.pixel_to_world(1834,1620)
print(radec)
print(radec.ra.degree)

# Your work here...


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotion.ipynb".
The first task is:
Useful tools
First I'm going to give you one class and one function that will be needed for the first couple of problems.  The class called MoffatPSF describes a PSF with the following circular profile:
$$ I(r) \propto \left[1 + (r/r_0)^2\right]^{-\beta}.$$
Many ground-based PSFs look more like this Moffat function than they do like a Gaussian, so let's use this for our astrometry experiments.  We'll stick with $\beta=2.5$ When you create a MoffatPSF, you give it the  half_light_radius $r_{1/2}$ which encloses half of the light, rather than giving it $r_0.$  You also give the coordinates $(x_0,y_0)$ of the center of the star.  All of these arguments we'll assume to be in units of pixels.
Once you have made a MoffatPSF, you can do a few things with it:
* draw a picture of the PSF as a 2d image (such that it sums to unity)
* realizeXY will return you the (x,y) pixel locations of arrival for one observation of the star - including Poisson noise.
* realizeImage will return you a 2d array that is the pixelized image of the arriving photons.
If you're curious you can look inside the class to see how it's done - I've used a few non-obvious tricks to make this fast since we want to create a lot of star images. There is a common trick of random-number generation when we are picking photon locations, where we transform a uniform deviate (which computers are good at) using a function that creates some desired distribution instead.  But you don't need to understand this to do the problems.
Then there is a function addBackground which will add background noise of a chosen level (denoted as $n$ in the lecture notes) to any image.
Can you write Python code for it?

# Make a Moffat class to draw
class MoffatPSF:
    def __init__(self,half_light_radius=4., x0=15.75,y0=15.25):
        # Create a Moffat PSF model that is centered at [y0,x0]
        # with the specified half-light radius (given in pixels)
        self.beta = 2.5
        # Calculate r0 from half-light radius
        tmp = np.power(0.5,1./(1-self.beta)) - 1
        self.r0 = half_light_radius / np.sqrt(tmp)
        self.x0 = x0
        self.y0 = y0
        # Factor that makes the integral of PSF be unity:
        self.norm = (self.beta-1) / (np.pi * self.r0**2)
    def draw(self,n_pix=32):
        # Return an array of shape (n_pix,n_pix) drawing the PSF
        # (using numpy convention
        # of zero-indexing and fastest index, x, last)

        # First make arrays holding the x and y coordinates
        xy=np.indices( (n_pix,n_pix),dtype=float)
        x = xy[1].copy()
        y = xy[0].copy()
        rsq = np.square(x-self.x0)+np.square(y-self.y0)

        # Here's the basic formla for Moffat:
        result = self.norm * np.power(1+rsq/(self.r0**2), -self.beta)
        # Tune up the normalization to unity:
        result /= np.sum(result)
        return result
    def realizeXY(self,n_photon=1e3):
        # Produce an array of the arrival positions of
        # photons, assuming that they are Poisson sample
        # of a mean of n_photon arrivals.
        # The returned array has shape (2,n_detected)
        # where mean of n_detected is n_photon

        # First pick total number of photons:
        n_detected = np.random.poisson(n_photon)

        # Then distribute them in radius using the trick
        # of inverse cumulative distribution of uniform distribution.
        r_uniform = np.random.random(n_detected)
        rsq = np.power(r_uniform, 1./(1-self.beta))-1.
        r = self.r0 * np.sqrt(rsq)

        # Draw position angles at random and produce output
        theta = np.random.random(n_detected) * (2*np.pi)
        return...


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotionSolutions.ipynb".
The first task is:
(b) Now create a MoffatPSF instance with a true center of your choice (keep the default half_light_radius=4, which fits nicely into 32x32 images).  Make a loop that will create photon lists for each of 10,000 observations of a star with an average flux n_photon=1e3.  Use your function from (a) to create estimates $(\hat x_0, \hat y_0)$ of each star.  Then produce a histogram of these 10,000 estimates (one for $x$ and one for $y$), and figure out the standard deviations $\sigma_x$ and $\sigma_y$ of your measurements.
Tip: Draw your photon lists 1 at a time, then measure each and throw it away before you draw the next one.  We don't want to use up our memory keeping them all around at once.
Can you write Python code for it?

xy = []
x0,y0 = 15.22,16.17
star = MoffatPSF(x0=x0, y0=y0)
# Run my loop of realizations
for i in range(10000):
    xy_i = centroid(star.realizeXY(n_photon=1000.))
    xy.append(xy_i)

# Make the xy estimates into a 2xN numpy array
xy = np.stack(xy).T

# Draw our histograms
h = pl.hist(xy[1]-x0,bins=100,range=(-1,1),histtype='step',label='X')
h = pl.hist(xy[0]-y0,bins=100,range=(-1,1),histtype='step',label='Y')
pl.xlabel('Position error')
pl.ylabel('Frequency')
pl.legend()

# Calculate the standard deviations and means of the estimates
print('X mean, standard deviation:',np.mean(xy[1,:]),np.std(xy[1,:]))
print('X mean, standard deviation:',np.mean(xy[0,:]),np.std(xy[0,:]))


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotionSolutions.ipynb".
The first task is:
(b) Now read the FITS catalogs into tables.  We will use the columns XWIN_IMAGE and YWIN_IMAGE to give us the pixel coordinates, with ERRAWIN_IMAGE giving the rough uncertainty on each (in units of pixels, 1 pixel = 0.264 arcsec).
By whatever means you choose, figure out which row in each catalog corresponds to the fast-moving star at the coordinates given above.  Then use the WCS's to obtain the ICRS RA & Dec measured for this star on these two images.
Can you write Python code for it?

# I'll start you off by giving the command to read in the catalog to a table:
cat_old = pf.getdata('old_catalog.fits',1)

# Let's locate the desired star by mapping the whole catalog to sky coordinates, and using
# astropy's ability to give distances between SkyCoords and find the one that's close
# to the Gaia position.
radec = wcs_old.pixel_to_world(cat_old['XWIN_IMAGE'],cat_old['YWIN_IMAGE'])

# Find the mininum distance between a catalog object and our Gaia target
target = co.SkyCoord(ra=29.91148,dec=-8.212267,frame='icrs',unit='deg')
dist = radec.separation(target)
index_old = np.argmin(dist)

# Save the coordinats of this star
coords_old = radec[index_old]
sigma_old = cat_old['ERRAWIN_IMAGE'][index_old] * 0.264 # Change into arcseconds
print("Star is index",index_old,"in old image at",coords_old)

# Now repeat for the new catalog
cat_new = pf.getdata('new_catalog.fits',1)
radec = wcs_new.pixel_to_world(cat_new['XWIN_IMAGE'],cat_new['YWIN_IMAGE'])
dist = radec.separation(target)
index_new = np.argmin(dist)
print(dist[index_new]) ###

# Save the coordinats of this star in the new image
coords_new = radec[index_new]
sigma_new = cat_new['ERRAWIN_IMAGE'][index_new] * 0.264 # Change into arcseconds
print("Star is index",index_new,"in new image at",coords_new)


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotionSolutions.ipynb".
The first task is:
(b) Write a function aperture_centroid(img,R) that will return the values of $(y0,x0)$ that satisfy
$$ M_x = \sum_{xy} I_{xy} W(x-x0,y-y0) (x-x0)=0$$
(and the same for y0), where $W$ is a weight function. I provide for you below a class GaussAp which you can use to create a Gaussian-aperture weight function with some $\sigma_w$:
$$ W(x,y) = \exp{\left(-\frac{x^2+y^2}{2\sigma_w^2}\right)}$$
To do this, you'll need to start with an initial guess for $(y0,x0),$ and then calculate the values of $M_f, M_x$ and $M_y$.  Then move the aperture's $x_0,y_0$ by $(M_x/M_f, M_y/M_f)$ and try again.  Iterate until the centroid doesn't move any more (or until you have done 10 iterations and need to quit!).
Can you write Python code for it?

# Here's my weight function class
# Remember, to use a class, you first create an "instance"
# of it, e.g.:  weight = GaussAp(sigma=2.6)
# and then you can call this instance like a function of dx,dy

class GaussAp:
    # A tophat weight function of radius sigma
    def __init__(self,sigma):
        self.sigma = sigma
        return
    def __call__(self,dx,dy):
        # Given equal-shaped arrays dx=x-x0, dy=y-y0,
        # returns an array of the same shape giving weight function.

        # Calculate distance of a pixel from the center
        rsq = np.square(dx) + np.square(dy)

        # Now return the weight
        return np.exp(-rsq/(2.*self.sigma*self.sigma))

# Now a function that calculates the weighted moments
def weighted_moments(img, weight, x, y, x0, y0):
    # img is the input image
    # weight is a weight function
    # x,y are the coordinates of the image
    # x0, y0 are the centers for the weight function
    w = weight(x-x0,y-y0)
    mf = np.sum(img*w)
    mx = np.sum(img*w*(x-x0))
    my = np.sum(img*w*(y-y0))
    return mf,mx,my

# Now my function that will iterate over x0,y0 to converge
def aperture_centroid(img, x, y, weight):
    # Iteratively apply the weighted centroiding to the
    # image (with coordinate arrays x,y given) until
    # convergence, where the Mx and My moments are nulled.

    # Start at the center of the image
    x0 = 0.5*(img.shape[1]-1)
    y0 = 0.5*(img.shape[0]-1)

    iteration = 0
    MAX_ITERATIONS = 20
    tolerance = 0.001 # Quit when centroid moves less than this

    while iteration < MAX_ITERATIONS:
        mf,mx,my = weighted_moments(img,weight,x,y,x0,y0)
        dx = mx / mf
        dy = my / mf
        if abs(dx)<tolerance and abs(dy)<tolerance:
            # Our moments are now close to zero - done!
            return np.array((y0,x0))

        # otherwise update centroid and try again
        x0 = x0 + dx
        y0 = y0 + dy
        iteration = iteration+1

    # If we get here, we've exceeded our iteration...


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotionSolutions.ipynb".
The first task is:
(c) Now see whether our formula for the accuracy of source-limiting centroiding is accurate:
$$ \sigma_x = \frac{\sigma_{\rm PSF}}{\sqrt{N_\gamma}}$$
in this case.  You'll need to use your MoffatPSF instance to estimate its standard deviation width $\sigma_{\rm PSF}$.
Can you write Python code for it?

# Draw a PSF with a lot of photons to measure its std in each axis (they should be the same)
lots_of_photons = star.realizeXY(1e6)
sigma_psf = np.std(lots_of_photons,axis=1)
print("PSF width:",sigma_psf)

# And divide by sqrt(N)
print("Estimated centroid error:",sigma_psf / np.sqrt(1e3))


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotionSolutions.ipynb".
The first task is:
(c) Now your job is to find the weight size radius $\sigma_w$ that yields the best accuracy on the centroid.  To do so, use the Moffat star as created above, and measure the centroid of 4,000 realizations contaminated with background noise at the specified n_bg.
Calculate the resultant $\sigma_x,\sigma_y$ for these 4,000 stars, trying values of $\sigma_w=2,2.5,\ldots,4.5.$  Plot the astrometric accuracy vs $\sigma_w$.
Which $\sigma_w$ is best?  See if you can make a rough comparison to the rule-of-thumb $\sigma_x\approx \sigma_{\rm PSF} / \nu.$
Can you write Python code for it?

weight_list = np.arange(2.0,5.,0.5)
x0,y0 = 15.22,15.88
nTrials = 4000
star = MoffatPSF(x0=x0, y0=y0)
sigma_x = []
sigma_y = []

for sigma in weight_list:
    weight = GaussAp(sigma)
    xy = []
     # Run my loop of realizations
    for i in range(nTrials):
        img = star.realizeImage(n_pix=32,n_photon=flux)
        addBackground(img, n_bg)
        xy_i = aperture_centroid(img,x,y,weight)
        xy.append(xy_i)
    # Put the results into a 2d array
    xy = np.vstack(xy).T
    # And get the centroid error
    sigma_x.append( np.std(xy[1]))
    sigma_y.append( np.std(xy[0]))
    # print out the mean positions, as a check
    print("At sigma",sigma,"mean x,y are ({:.3f},{:.3f})".format(np.mean(xy[1]),
                                                                np.mean(xy[0])))

# Now we can look at the results:
pl.plot(weight_list,sigma_x,'ro',label='X')
pl.plot(weight_list,sigma_y,'bs',label='Y')
pl.grid()
pl.xlabel('Aperture sigma (pixels)')
pl.ylabel('Centroid error (pixels)')
pl.legend(framealpha=0.5)


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotionSolutions.ipynb".
The first task is:
(c) Using these two sky coordinates, estimate the rate of proper motion (in mas/yr) of the star between the two exposures.  Note that proper motions are traditionally given in real angular motions, so that (RA PM) = (difference in RA) * cos(dec) / (time interval). [You can ignore the fact that there might be parallax motion mixed in here.]
Give an uncertainty for our measurement as well.  Here we have to be careful, because ERRAWIN_IMAGE includes only the shot-noise errors.  It does not include the contribution from atmospheric turbulence, which we know is dominant for brighter stars.  So let's add another 10 mas of uncertainty in quadrature to each coordinate of each measurement as a guess of the size of the turbulence.
Does the DES measurement agree with Gaia's estimates?  Are there any reasons it might disagree?
pmra  =  326.06  +- 0.94    mas/yr
pmdec = -124.12 +- 0.81 mas/yr
Can you write Python code for it?

# First get the time interval, in years:
dt = (mjd_new - mjd_old) / 365.25 #(number of days in a Julian year)
print("Years between observations:",dt)

# Now get the change in dec and RA, which we convert to milliarcsec
dRA = (coords_new.ra.deg - coords_old.ra.deg) * 3600 * 1000. * np.cos(coords_new.dec.rad)
dDec = (coords_new.dec.deg - coords_old.dec.deg) * 3600 * 1000.

# The uncertainty in the difference is the quadrature sum of the individual errors
sigma_diff = np.sqrt( sigma_new**2 + sigma_old**2) * 1000. # convert to mas.
# Add 2 exposures' worth of 10 mas turbulence to the sigma
sigma_diff = np.sqrt(sigma_diff**2 + 2 * 10.**2)

# Now divide by time to get the rates of motion.
print("PMRA:  {:.2f} +- {:.2f}".format(dRA/dt, sigma_diff/dt))
print("PMDec: {:.2f} +- {:.2f}".format(dDec/dt, sigma_diff/dt))


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotionSolutions.ipynb".
The first task is:
Exercise 1: source-dominated centroiding
(a) Write a function centroid(photon_list) which will estimate the position of a star by taking the average of its x and y photon arrival positions, if it's give a (2,N) array of photon positions such as produced by MoffatPSF.realizeXY.
Can you write Python code for it?

def centroid(photon_list):
    # Returns (y,x) centroids in an array of length 2
    return np.mean(photon_list, axis=1)


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotionSolutions.ipynb".
The first task is:
Exercise 2: aperture centroids for background-limited stars
Now we're going to work with pixelized images and add background noise (say, $n=4000$). We know that simple centroiding will not be a good idea so we're going to need to use some apertures.  Before proceeding, let's just draw a background-noisy star:
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotionSolutions.ipynb".
The first task is:
Exercise 3: Using a WCS
This exercise requires four fits files: old_image.fits, new_image.fits, old_catalog.fits, and new_catalog.fits. Please download these files, and place them in the same directory as this notebook.
The 2 FITS images are from the Dark Energy Survey (DES), named old_image.fits and new_image.fits, and they overlap on the sky. According to Gaia DR2, there is a fast-moving star located at roughly $(\alpha,\delta)=$(29.91148,-8.212267). If you display these two images using DS9, then align them using the Frame->Match->Frame->WCS option, you will be able to move your cursor to those coordinates actually see the star move.
We also have catalogs (FITS tables) named old_catalog.fits and new_catalog.fits that contain the precise pixel centroids measured by SExtractor on these frames.  Your exercise is to extract the WCS from each image's header using astropy, apply it to the $(x,y)$ coordinates to obtain $(\alpha,\delta)$ for this star in each image, and then estimate its proper motion.
(a) Get a WCS and the MJD-OBS out of headers. [MJD is Modified Julian Date.  Basically it gives the time of the exposure in units of days since some reference moment.]
Can you write Python code for it?

# I'll start you off by constructing an astropy WCS from the first image
h = pf.getheader('old_image.fits')
wcs_old = WCS(h)
mjd_old = h['MJD-OBS']

# And here's an example of using the WCS to map to the sky:
radec = wcs_old.pixel_to_world(1834,1620)
print(radec)
print(radec.ra.degree)

# Here's the second image:
h = pf.getheader('new_image.fits')
wcs_new = WCS(h)
mjd_new = h['MJD-OBS']


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotionSolutions.ipynb".
The first task is:
Here is one more very helpful hint.  A lot of our activities require us to do sums like $\sum_{x,y} x I_{xy}.$  To do that it'll be very helpful to have numpy arrays that hold the x and y coordinate values of each pixel in an image.  Here I'll make them for you for the case of a 32x32 square image.
Note that the numpy indexing convention places the y coordinate first in the [y,x] indexing.  Also the first pixel is number 0.  This differs from the convention used in FITS images, which is that the first pixel is number 1, and the first axis is the x axis.
Can you write Python code for it?

xy=np.indices( (32,32),dtype=float)
x = xy[1].copy()
y = xy[0].copy()
pl.imshow(x,origin='lower',interpolation='nearest')
pl.colorbar()


I am working on the file "Sessions/Session11/Day2/MeasuringCentroidsAndProperMotionSolutions.ipynb".
The first task is:
Useful tools
First I'm going to give you one class and one function that will be needed for the first couple of problems.  The class called MoffatPSF describes a PSF with the following circular profile:
$$ I(r) \propto \left[1 + (r/r_0)^2\right]^{-\beta}.$$
Many ground-based PSFs look more like this Moffat function than they do like a Gaussian, so let's use this for our astrometry experiments.  We'll stick with $\beta=2.5$ When you create a MoffatPSF, you give it the  half_light_radius $r_{1/2}$ which encloses half of the light, rather than giving it $r_0.$  You also give the coordinates $(x_0,y_0)$ of the center of the star.  All of these arguments we'll assume to be in units of pixels.
Once you have made a MoffatPSF, you can do a few things with it:
* draw a picture of the PSF as a 2d image (such that it sums to unity)
* realizeXY will return you the (x,y) pixel locations of arrival for one observation of the star - including Poisson noise.
* realizeImage will return you a 2d array that is the pixelized image of the arriving photons.
If you're curious you can look inside the class to see how it's done - I've used a few non-obvious tricks to make this fast since we want to create a lot of star images. There is a common trick of random-number generation when we are picking photon locations, where we transform a uniform deviate (which computers are good at) using a function that creates some desired distribution instead.  But you don't need to understand this to do the problems.
Then there is a function addBackground which will add background noise of a chosen level (denoted as $n$ in the lecture notes) to any image.
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day3/GalaxyPhotometryAndShapes.ipynb".
The first task is:
(b) Use this to calculate the best achievable measurement accuracy on $e$ for our standard image.
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day3/GalaxyPhotometryAndShapes.ipynb".
The first task is:
(c) Make a graph showing how the optimal $\sigma_e$ varies as the size $\sigma_{\rm PSF}$ of the Gaussian PSF varies from being $0.2\times r_0$ to being $3\times r_0.$.  What's the lesson here?
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day3/GalaxyPhotometryAndShapes.ipynb".
The first task is:
(c) Now create a plot of the S/N level of the flux measurement vs the radius R of the aperture.  Here the signal is the mean, and the noise the std deviation, of the tophat_flux of many noisy measurements of this galaxy.  You can use either an analytic or numeric estimate of these quantities.  Report what the optimal tophat S/N is, and what R achieves it.
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day3/GalaxyPhotometryAndShapes.ipynb".
The first task is:
(d) Repeat part (c), but this time use a Gaussian aperture whose width $\sigma_w$ you vary to optimize the S/N ratio of the aperture flux, i.e. a function gaussian_flux(img,sigma_w) is needed.  Which performs better, the optimized tophat or the optimized Gaussian?
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day3/GalaxyPhotometryAndShapes.ipynb".
The first task is:
Exercise 1: Aperture photometry
Here we'll try out a few forms of aperture photometry and see how they compare in terms of the S/N ratios they provide on the galaxy flux.
(a) Write a function tophat_flux(img,R) which implements a simple tophat aperture sum of flux in all pixels within radius R of the center of the galaxy.  We will keep the center of our galaxy fixed at pixel [32,32] so you don't have to worry about iterating to find the centroid.
Draw a noiseless version of a circular galaxy with the characteristics in the cell below.  Then use your tophat_flux function to plot the "curve of growth" for this image, with R on the x axis going from 5 to 30 pixels, and the y axis showing the fraction of the total flux that falls in your aperture.
How many scale radii do we need the aperture to be to miss <1% of the flux?
Can you write Python code for it?

r0 = 4.
e = 0.
flux = 1e4
sigma_psf = 2.

# your work here...


I am working on the file "Sessions/Session11/Day3/GalaxyPhotometryAndShapes.ipynb".
The first task is:
Exercise 2: Spurious color
This time let's consider that we want to measure an accurate $g-r$ color for our galaxy, but the seeing is $\sigma_{\rm PSF}=2$ pixels in the $r$ image but $\sigma_{\rm PSF}=2.5$ pixels in the $g$ image.  Let's see how the size of our aperture biases our color measurement.
(a) Draw a noiseless $g$-band and a noiseless $r$-band image of our galaxy.  Let's assume that the true color $g-r \equiv 2.5\log_10(f_r/f_g) = 0,$ i.e. that the $g$ and $r$ fluxes of the galaxy are both equal to our nominal flux.  Plot the difference between the two images: are they the same?
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day3/GalaxyPhotometryAndShapesSolutions.ipynb".
The first task is:
(b) Use this to calculate the best achievable measurement accuracy on $e$ for our standard image.
Can you write Python code for it?

# We take the Fisher sum
Fee = np.sum(dI_de * dI_de) / n_bg
# ...and then the Cramer-Rao bound on sigma_e is
print("Lower bound on sigma_e:",1./np.sqrt(Fee))


I am working on the file "Sessions/Session11/Day3/GalaxyPhotometryAndShapesSolutions.ipynb".
The first task is:
(c) Make a graph showing how the optimal $\sigma_e$ varies as the size $\sigma_{\rm PSF}$ of the Gaussian PSF varies from being $0.2\times r_0$ to being $3\times r_0.$.  What's the lesson here?
Can you write Python code for it?

# I want to wrap all of the above into a function of sigma_psf:
def sigma_opt(sigma_psf):
    de = 0.01
    img1 = drawDisk(r0=r0,flux=flux,sigma_psf=sigma_psf,e=+de)
    img2 = drawDisk(r0=r0,flux=flux,sigma_psf=sigma_psf,e=-de)
    dI_de = (img1-img2)/(2*de)
    Fee = np.sum(dI_de * dI_de) / n_bg
    return 1./np.sqrt(Fee)

# Now make our plot
psfs = np.linspace(0.2*r0,3*r0,40)

# use our function to get results
sigma_e = [sigma_opt(sigma_psf) for sigma_psf in psfs]

# And plot
pl.plot(psfs,sigma_e,'r-',lw=3)
pl.xlabel("PSF sigma (pixels)")
pl.ylabel('Optimal ellipticity error')
pl.ylim(0)
pl.xlim(0)
pl.grid()
pl.annotate('Scale length',(r0,0.15),xytext=(r0,0.25),ha='center',
           arrowprops={'arrowstyle':'->'})


I am working on the file "Sessions/Session11/Day3/GalaxyPhotometryAndShapesSolutions.ipynb".
The first task is:
(d) Repeat part (c), but this time use a Gaussian aperture whose width $\sigma_w$ you vary to optimize the S/N ratio of the aperture flux, i.e. a function gaussian_flux(img,sigma_w) is needed.  Which performs better, the optimized tophat or the optimized Gaussian?
Can you write Python code for it?

# I can do this quickly by making a Gaussian Aperture class and plugging it into
# my aperture_flux code
class GaussAp:
    # A tophat weight function of radius sigma
    def __init__(self,sigma):
        self.sigma = sigma
        return
    def __call__(self,dx,dy):
        # Given equal-shaped arrays dx=x-x0, dy=y-y0,
        # returns an array of the same shape giving weight function.

        # Calculate distance of a pixel from the center
        rsq = np.square(dx) + np.square(dy)

        # Now return the weight
        return np.exp(-rsq/(2.*self.sigma*self.sigma))

sigma_list = np.arange(1.0,10.,0.25)
sn_ratio = []
for s in sigma_list:
    weight = GaussAp(s)
    gauss_flux = aperture_flux(noiseless,weight)
    expected_variance = n_bg * np.sum(np.square(weight(x,y)))
    sn_ratio.append( gauss_flux / np.sqrt(expected_variance))

# Now plot the results
pl.plot(sigma_list, sn_ratio,'ro')
pl.xlabel('Gaussian aperture sigma (pixels)')
pl.ylabel('Flux S/N ratio')
pl.grid()

# Where's the best?
i = np.argmax(sn_ratio)
print("Best S/N is",sn_ratio[i],"at aperture sigma",sigma_list[i])

# The Gaussian aperture yields about 15% higher S/N, equivalent to 30% more exposure time!


I am working on the file "Sessions/Session11/Day3/GalaxyPhotometryAndShapesSolutions.ipynb".
The first task is:
Exercise 2: Spurious color
This time let's consider that we want to measure an accurate $g-r$ color for our galaxy, but the seeing is $\sigma_{\rm PSF}=2$ pixels in the $r$ image but $\sigma_{\rm PSF}=2.5$ pixels in the $g$ image.  Let's see how the size of our aperture biases our color measurement.
(a) Draw a noiseless $g$-band and a noiseless $r$-band image of our galaxy.  Let's assume that the true color $g-r \equiv 2.5\log_10(f_r/f_g) = 0,$ i.e. that the $g$ and $r$ fluxes of the galaxy are both equal to our nominal flux.  Plot the difference between the two images: are they the same?
Can you write Python code for it?

img_g = drawDisk(r0=r0,e=e,flux=flux,sigma_psf=2.5)
img_r = drawDisk(r0=r0,e=e,flux=flux,sigma_psf=2.0)
pl.imshow(img_g-img_r,origin='lower',interpolation='nearest')
pl.colorbar()


I am working on the file "Sessions/Session11/Day3/GalaxyPhotometryAndShapesSolutions.ipynb".
The first task is:
We can see here that a naive use of "matched" apertures can cause significant spurious color, even when the aperture has a sigma that is many times that of the galaxy and PSF. But the tophat does better.  So without any kind of PSF matching, we have to use algorithms with non-optimal S/N in order to approach true colors.
Exercise 3: Degradation of ellipticity measurements by seeing
It's hard to measure the shape of a galaxy that is not resolved by the PSF.  That means that poorly-resolved galaxies are less useful for detecting weak-lensing (WL) shear.  Let's see if we can quantify this by using the Fisher matrix to determine the best possible measurement accuracy on the parameter $e$ of our model (we'll make things easy by holding all other parameters of the galaxy model as fixed).
Remember how the Fisher matrix works: for an image signal $I_{xy}$ and noise $\sigma_{xy}$ in each pixel, the Fisher information for a parameter $\theta$ is
$$ F_{\theta\theta} = \sum_{xy} \frac{1}{\sigma^2_{xy}} \left(\frac{\partial I_{xy}}{\partial\theta}\right)^2.$$
Here we're interested in $\theta=e$.
(a) Draw two versions of our standard galaxy, with $e = \pm0.01.$  Use these to calculate and plot the quantity we need, $\frac{\partial I_{xy}}{\partial e}.$  Comment on how this picture relates to the fact that we like to measure WL shear using the moment of $x^2-y^2$.
Can you write Python code for it?

de = 0.01
img1 = drawDisk(r0=r0,flux=flux,sigma_psf=sigma_psf,e=+de)
img2 = drawDisk(r0=r0,flux=flux,sigma_psf=sigma_psf,e=-de)
dI_de = (img1-img2)/(2*de)
pl.imshow(dI_de,origin='lower',interpolation='nearest')
pl.colorbar()
pl.title('dI/de')

# Look at that! The signature of WL shear is a quadrupole pattern, which is matched
# and therefore picked out of an image by the (x^2-y^2) moment!


I am working on the file "Sessions/Session11/Day3/PSFphotometry.ipynb".
The first task is:
1) Centroiding
Now that we have a test case to work on, let's find its position on the CCD.
To do that, we're going to need two arrays: one which has the same shape as cutout, but where each value is the X coordinate of the pixel, and another where each value is the Y coordinate of the pixel. Numpy has a function called meshgrid() that will give us this; we just need to supply an iterator for the X values, and an iterator for the Y values. It looks like this:
Can you write Python code for it?

xx, yy = np.meshgrid(range(2, 10), range(20, 30))
print("xx: ", xx)
print("yy: ", yy)


I am working on the file "Sessions/Session11/Day3/PSFphotometry.ipynb".
The first task is:
2) PSF Photometry
We needed the centroid first, because we're going to use that position to place our "PSF" model. Since we have not yet fit a real PSF model to the sources in the image, we'll use a Gaussian as an approximation.
I'll give you the function for a normalized 2D Gaussian:
Can you write Python code for it?

def gaussian2D(radius, mu):
    return 1/(mu**2*2*np.pi)*np.exp(-0.5*((radius)/mu)**2)


I am working on the file "Sessions/Session11/Day3/PSFphotometry.ipynb".
The first task is:
Do the results look reasonable, compared to the image of the cutout you made above? Note that this is the Gaussian width, not the full-width at half-max that is typically quoted for PSF sizes.
If those look good, now we just need to run the second moment estimator over all the sources in your catalog. Our goal is to find if there's one particular size that fits lots of objects; that's likely to be our PSF size and the objects are likely to be stars.
Can you write Python code for it?

# Question
%%time

# We will put the x and y moments in these lists
moments_x = []
moments_y = []

for peak_x, peak_y in # complete
    image_cutout = image[ # complete
    start_x = int( # complete
    start_y = int( # complete

    centroid_x, centroid_y = # complete

    moment_x, moment_y = second_moment( # complete
    moments_x.append( # complete
    moments_y.append( # complete


I am working on the file "Sessions/Session11/Day3/PSFphotometry.ipynb".
The first task is:
Double check that the PSF flux you get matches (approximately) the flux you get from aperture photometry. If your cutout image is small enough that there are no other sources in it, you can just sum the cutout itself. No need to apply a more restrictive aperture for a debugging check like this.
Can you write Python code for it?

# Question
aperture_flux = # complete
print(aperture_flux)


I am working on the file "Sessions/Session11/Day3/PSFphotometry.ipynb".
The first task is:
First just make an image of an example PSF, on the same grid as the cutout.
Note that the Gaussian is parameterized in terms of a radius, which means you will need to compute that radius from the position of every pixel in your image. meshgrid is again the tool for this.
You can either use your centroid() function here, or for debugging it's fine to manually set x_center and y_center to specific values.
Can you write Python code for it?

xx, yy = np.meshgrid( # complete
x_center, y_center = # complete
radius = np.sqrt(( # complete
                 + ( # complete
                 )

psf_size_pixels = 2.5
psf_image = gaussian2D( # complete
plt.imshow( # Complete


I am working on the file "Sessions/Session11/Day3/PSFphotometry.ipynb".
The first task is:
If your psf_flux reasonably matches your aperture_flux, well done! You have a working PSF photometry measurement, now it just needs to get wrapped up in a convenient function for later use.
Can you write Python code for it?

# Question

# We need to pass both the centroid x and y, and the image cutout start x,y because the star
# isn't necessarily at the very center of the cutout.

def psf_flux_gaussian(image_cutout, centroid_x, centroid_y, radius, x_start, y_start):

    x_size, y_size = # complete
    xx, yy = # complete

    r = # complete
    psf_image = # complete
    psf_flux = # complete
    return psf_flux


I am working on the file "Sessions/Session11/Day3/PSFphotometry.ipynb".
The first task is:
Just to be sure, we should check that the PSF image is normalized (approximately) by summing the pixel values.
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day3/PSFphotometry.ipynb".
The first task is:
Let's plot the positions of the peaks on the image, to make sure they look reasonable
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day3/PSFphotometry.ipynb".
The first task is:
Let's run the second moment estimator on one of the cutouts you made above.
Can you write Python code for it?

# Question
second_moment(cutout, # complete


I am working on the file "Sessions/Session11/Day3/PSFphotometry.ipynb".
The first task is:
Now let's select a smaller region around something that looks like a good, isolated star. Remember to update the extent so we know which pixels we're looking at.
Can you write Python code for it?

# Question
plt.imshow(image[ # complete
          # complete
    )


I am working on the file "Sessions/Session11/Day3/PSFphotometry.ipynb".
The first task is:
Ok, now we can compute the actual PSF flux. Remember the formula from the lecture is:
$$ f_{\rm ML}(x, y) = \frac{\sum_i \hat{f}_i p_i(x,y)}{\sum_i p_i^2(x, y)}$$
where $\hat{f_i}$ are your image values, and $p_i$ are are your PSF model values.
Can you write Python code for it?

# Question
psf_flux = # complete
print(psf_flux)


I am working on the file "Sessions/Session11/Day3/PSFphotometry.ipynb".
The first task is:
To use the peak-finder, we need to create a "detection image" by convolving the real image with the PSF. Of course, we don't know the PSF yet, so you can substitute a guess: try a Gaussian kernel, with a 2.5 pixel width.
The %%time "magic" will show us how long the convolution and peak-finding took.
Can you write Python code for it?

%%time

# Question

convolved_image = astropy.convolution.convolve( # complete

peak_x_values, peak_y_values = # complete


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
0) Finding an example star
I think a good way to work on a problem like this is to start with the core of the algorithm, working on just a single test case. After we have that working and tested, we can build out the infrastructure around it to run on the entire image.
Let's display a small subset of the image, say 400x400 pixels. By default, imshow() will scale the colorbar to the minimum and maximum pixel values, so let's also set some more reasonable limits so we can see some stars.
We also need to use the extent= keyword argument to imshow() so that the labels on the X and Y axes correspond to the pixel coordinates that we've selected.
You can also open the images in ds9 if you like, for easier browsing.
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
2) PSF Photometry
We needed the centroid first, because we're going to use that position to place our "PSF" model. Since we have not yet fit a real PSF model to the sources in the image, we'll use a Gaussian as an approximation.
I'll give you the function for a normalized 2D Gaussian:
Can you write Python code for it?


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
A good debugging check is to look at a few cutouts centered on your newly-found detections. You can flip through a few of these by changing the value of n.
Can you write Python code for it?

# answer
n = 50
peak_x = peak_x_values[n]
peak_y = peak_y_values[n]
cutout = image[(peak_x - 5):(peak_x + 5), (peak_y - 5):(peak_y + 5)]

plt.imshow(cutout)


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
Did you get a table full of photometry? If so, great! If it's not working well, it's likely to be a problem with getting the right inputs to the different functions you're calling. You've tested all the steps separately, so they should be working. Getting the right indices on your image cutout is always a tricky part.
If you have extra time, try adding an aperture photometry function to the processing.  You can plot the size (from the second moment) against flux to find what objects might be galaxies, and generate the cutout image to see if they're really galaxies.
Extra Credit: Measuring the PSF
Once we have sources identified in an image, we want to identify which would be good for PSF determination, and then we want to measure their PSFs. In our case we're going to do both of these at once, we're going to measure sizes for all sources, and then use the mean size of those which we think are stars as our PSF model. In a more sophisticated pipeline, the object sizes might be used as a cut before passing to some more complicated PSF determination process.
To obtain object sizes, we're going to measure the "second moment".
This will look a lot like the centroid algorithm. The formula we want to implement is:
$$I_{xx}^2 =  \frac{\sum_i (\hat{f_i} (x_i - x_{\rm center}))^2}{\sum_i \hat{f_i}^2} $$
Let's try building it directly in the function this time; if it gives you trouble, feel free to try it out in some notebook cells directly (so you can see the intermediate variables better) before putting it back in the function.
Can you write Python code for it?

# answer
def second_moment(image_cutout, centroid_x, centroid_y, start_x, start_y):
    x_size, y_size = image_cutout.shape
    xx, yy = np.meshgrid(np.arange(start_x, start_x + x_size),
                         np.arange(start_y, start_y + y_size))
    x_width = np.sqrt(np.sum((image_cutout*(xx - centroid_x))**2))/np.sqrt(np.sum(image_cutout**2))
    y_width = np.sqrt(np.sum((image_cutout*(yy - centroid_y))**2))/np.sqrt(np.sum(image_cutout**2))
    return (x_width, y_width)


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
Do the results look reasonable, compared to the image of the cutout you made above? Note that this is the Gaussian width, not the full-width at half-max that is typically quoted for PSF sizes.
If those look good, now we just need to run the second moment estimator over all the sources in your catalog. Our goal is to find if there's one particular size that fits lots of objects; that's likely to be our PSF size and the objects are likely to be stars.
Can you write Python code for it?

# Answer
%%time

# We will put the x and y moments in these lists
moments_x = []
moments_y = []

for peak_x, peak_y in # complete
    image_cutout = image[(peak_x - 5):(peak_x + 5), (peak_y - 5):(peak_y + 5)]
    start_x = int(peak_x - 5)
    start_y = int(peak_y - 5)

    moment_x, moment_y = second_moment(image_cutout, peak_x, peak_y, start_x, start_y)
    moments_x.append(moment_x)
    moments_y.append(moment_y)


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
Do the values you got make sense? Are they within the range of x and y coordinate values of the cutout? Does it roughly match where the star is? If not, are they possibly swapped, x-for-y and y-for-x? (It's very easy to get confused with the ordering of x and y indicies in Numpy, I make that mistake all the time).
If they make sense, try overplotting the coordinates on one of your larger cutout images.
Can you write Python code for it?

# Answer
plt.imshow(image[:50, 150:200], vmin=-10, vmax=70,
          extent=(150, 200, 50, 0))
plt.axvline(x_center, color='r')
plt.axhline(y_center, color='r')


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
First just make an image of an example PSF, on the same grid as the cutout.
Note that the Gaussian is parameterized in terms of a radius, which means you will need to compute that radius from the position of every pixel in your image. meshgrid is again the tool for this.
You can either use your centroid() function here, or for debugging it's fine to manually set x_center and y_center to specific values.
Can you write Python code for it?

xx, yy = np.meshgrid(np.arange(155, 185), np.arange(15, 35))
x_center, y_center = (164.08, 25.24)
radius = np.sqrt((xx - x_center)**2 + (yy - y_center)**2)

psf_size_pixels = 2.5
psf_image = gaussian2D(radius, psf_size_pixels)
plt.imshow(psf_image)


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
If your lines cross on your chosen star, great! You've completed the first step of doing photometry, centroiding the object.
Let's take the code you prototyped in the notebook cells, and wrap it into a nice function we can use later. When we call this function, we need to tell it about the coordinates of the image we're providing, so we'll add the x_start and y_start parameters to convey that. We don't need to know the other two corners, because we can figure that out from the size of image_cutout.
Can you write Python code for it?

# Answer
def centroid(image_cutout, x_start, y_start):
    x_size, y_size = image_cutout.shape
    xx, yy = np.meshgrid(np.arange(start_x, x_start + x_size),
                         np.arange(start_y, y_start + y_size))
    x_center = np.sum(image_cutout*xx)/np.sum(image_cutout)
    y_center = np.sum(image_cutout*yy)/np.sum(image_cutout)
    return (x_center, y_center)


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
If your psf_flux reasonably matches your aperture_flux, well done! You have a working PSF photometry measurement, now it just needs to get wrapped up in a convenient function for later use.
Can you write Python code for it?

# Answer

# We need to pass both the centroid x and y, and the image cutout start x,y because the star
# isn't necessarily at the very center of the cutout.

def psf_flux_gaussian(image_cutout, centroid_x, centroid_y, radius, x_start, y_start):

    x_size, y_size = image_cutout.shape
    xx, yy = np.meshgrid(np.arange(start_x, x_start + x_size),
                         np.arange(start_y, y_start + y_size))

    r = np.sqrt((xx - centroid_x)**2 + (yy - centroid_y)**2)
    psf_image = gaussian2D(r, radius)
    psf_flux = np.sum(psf_image * cutout)/np.sum(psf_image**2)
    return psf_flux


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
Let's plot the positions of the peaks on the image, to make sure they look reasonable
Can you write Python code for it?

# Answer

plt.plot(peak_x_values, peak_y_values, '.')


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
Let's run the second moment estimator on one of the cutouts you made above.
Can you write Python code for it?

# Answer
second_moment(cutout, 5.0, 5.0, 0, 0)


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
Note how the values in a column are the same in xx, and all the values in a row are the same in yy.
Let's make an xx and yy with the values corresponding to the pixel coordinates in your cutout image.
Can you write Python code for it?

# Answer
xx, yy = np.meshgrid(np.arange(155, 185), np.arange(15, 35))


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
Ok, now we can compute the actual PSF flux. Remember the formula from the lecture is:
$$ f_{\rm ML}(x, y) = \frac{\sum_i \hat{f}_i p_i(x,y)}{\sum_i p_i^2(x, y)}$$
where $\hat{f_i}$ are your image values, and $p_i$ are are your PSF model values.
Can you write Python code for it?

# Answer
psf_flux = np.sum(psf_image * subset)/np.sum(psf_image**2)
print(psf_flux)


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
Ok, we need to cut down the image one more time, this time to give us a "cutout" image of a single star-like object. The cutout should only be about 20x20 pixels.
Can you write Python code for it?

cutout = image[15:35, 155:185]
plt.imshow(cutout)


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
To use the peak-finder, we need to create a "detection image" by convolving the real image with the PSF. Of course, we don't know the PSF yet, so you can substitute a guess: try a Gaussian kernel, with a 2.5 pixel width.
The %%time "magic" will show us how long the convolution and peak-finding took.
Can you write Python code for it?

%%time

# Answer

convolved_image = astropy.convolution.convolve(image, astropy.convolution.Gaussian2DKernel(2.6))

peak_x_values, peak_y_values = find_peaks(convolved_image, 100)


I am working on the file "Sessions/Session11/Day3/PSFphotometrySolutions.ipynb".
The first task is:
With that function all filled in, let's run it on the image!
Can you write Python code for it?

%%time
# Answer

photometry_table = run_photometry(image, 50, median_moment)
print(photometry_table[:20])

print(photometry_table[:20])


I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtraction.ipynb".
The first task is:
Problem 1.1) Make a simple mean coadd
Simulate N observations of a star, and coadd them by taking the mean of the N observations. (We can only do this because they are already astrometrically and photometrically aligned and have the same background value.)
Can you write Python code for it?

MU = 35
S = 100
F = 100
FWHM = 5

x = np.arange(100)

# simulate a single observation of the star and plot:
y = # complete
pixel_plot(x, y)

# Write a simulateN function that returns an array of size (N, x)
# representing N realizations of your simulated star
# This will stand in as a stack of multiple observations of one star

def simulateN(x, mu, fwhm, S, F, N):
    """simulate a noisy stellar signal

    Parameters
    ----------
    x : array-like
        detector pixel number
    mu : float
        mean position of the 1D star
    fwhm : float
        Full-width half-maximum of the stellar profile on the detector
    S : float or array-like of len(x)
        Sky background for each pixel
    F : float
        Total stellar flux
    N: int
        Number of images to simulate

    Returns
    -------
    noisy_counts : array-like of shape (N, x)
        the (noisy) number of counts in each pixel
    """
    # complete
    return noisy_counts

# simulate N=50 images with the same star
x = np.arange(100)
stack = # complete
# where stack is an array of size (50, 100) representing a pile of 50 images with 100 pixels

# coadd by taking the mean and plot the result
coadd = # complete
pixel_plot(x, coadd)

# Try a few different N to see how it affects the S/N of your result

# Plot the coadds of N=[1, 10, and 100] on the same plot:
# complete


I am working on the file "Sessions/Session11/Day4/CoadditionAndSubtraction.ipynb".
The first task is:
Problem 1.2)  SNR vs N
Now compute the observed SNR of the simulated star on each coadd and compare to the expected SNR in the idealized case. The often repeated mnemonic for SNR inscrease as a function of number of images, $N$, is that noise decreses like $\sqrt{N}$. This is of course idealized case where the noise in each observation is identical.
Using your simulateN function, simulate a series of mean coadds with increasing N.

First, plot the empirical noise/uncertainty/stdev as a function of N. and overplot the expected uncertainty given the input sky level. You can use an area you know isn't touched by the star.
Next, plot the empirical SNR of the star (measured flux/fluxErr) as a function of N. Overplot the expected SNR. You can assume you know the sky level.

Your expected scaling with N should roughly track your empirical estimate.
Can you write Python code for it?

# complete

# hint. One way to start this
# std = []
# flux = []
# Ns = np.arange(1, 1000, 5)
# for N in Ns:
    # y = simulateN(...)
    # complete

# plt.plot(Ns, ...,  label="coadd")
# plt.plot(Ns, ..., label="expected")
# plt.xlabel('N')
# plt.ylabel('pixel noise')
# plt.legend()

# complete

# plt.plot(Ns, ...,  label="coadd")
# plt.plot(Ns, ..., label="expected")
# plt.xlabel('N')
# plt.ylabel('PSF Flux SNR')
# plt.legend()