I am working on the file "66_Registration_Demons.ipynb".
The first task is:
SimpleITK also includes a set of Demons filters which are independent of the ImageRegistrationMethod. These include:
1. DemonsRegistrationFilter
2. DiffeomorphicDemonsRegistrationFilter
3. FastSymmetricForcesDemonsRegistrationFilter
4. SymmetricForcesDemonsRegistrationFilter
As these filters are independent of the ImageRegistrationMethod we do not have access to the multiscale framework. Luckily it is easy to implement our own multiscale framework in SimpleITK, which is what we do in the next cell.
Can you write Python code for it?

def smooth_and_resample(image, shrink_factor, smoothing_sigma):
    """
    Args:
        image: The image we want to resample.
        shrink_factor: A number greater than one, such that the new image's size is original_size/shrink_factor.
        smoothing_sigma: Sigma for Gaussian smoothing, this is in physical (image spacing) units, not pixels.
    Return:
        Image which is a result of smoothing the input and then resampling it using the given sigma and shrink factor.
    """
    smoothed_image = sitk.SmoothingRecursiveGaussian(image, smoothing_sigma)

    original_spacing = image.GetSpacing()
    original_size = image.GetSize()
    new_size = [int(sz/float(shrink_factor) + 0.5) for sz in original_size]
    new_spacing = [((original_sz-1)*original_spc)/(new_sz-1)
                   for original_sz, original_spc, new_sz in zip(original_size, original_spacing, new_size)]
    return sitk.Resample(smoothed_image, new_size, sitk.Transform(),
                         sitk.sitkLinear, image.GetOrigin(),
                         new_spacing, image.GetDirection(), 0.0,
                         image.GetPixelIDValue())

def multiscale_demons(registration_algorithm,
                      fixed_image, moving_image, initial_transform = None,
                      shrink_factors=None, smoothing_sigmas=None):
    """
    Run the given registration algorithm in a multiscale fashion. The original scale should not be given as input as the
    original images are implicitly incorporated as the base of the pyramid.
    Args:
        registration_algorithm: Any registration algorithm that has an Execute(fixed_image, moving_image, displacement_field_image)
                                method.
        fixed_image: Resulting transformation maps points from this image's spatial domain to the moving image spatial domain.
        moving_image: Resulting transformation maps points from the fixed_image's spatial domain to this image's spatial domain.
        initial_transform: Any SimpleITK transform, used to...


I am working on the file "66_Registration_Demons.ipynb".
The first task is:
Utilities
Load utilities that are specific to the POPI data, functions for loading ground truth data, display and the labels for masks.
Can you write Python code for it?


I am working on the file "67_Registration_Semiautomatic_Homework.ipynb".
The first task is:
Homework: semiautomatic landmark localization
You will now improve the localization of the fixed landmarks in the moving image using intensity based registration. This registration is performed independetly for each landmark.
The output of the following cell is expected to be a list of tuples (3D) called updated_moving_image_points.
Hint: You need to initialize the intensity based registration in a way that takes into account that the images may have a significant rotation between them (up to 180$^o$).
Registration (semiautomatic landmark localization)
Evaluate the quality of the semiautomatic localization by registering the two images, and then comparing the registration errors using the known reference data.
Can you write Python code for it?

updated_moving_image_points_flat = [c for p in updated_moving_image_points for c in p]
semi_automatic_transform = sitk.VersorRigid3DTransform(sitk.LandmarkBasedTransformInitializer(sitk.VersorRigid3DTransform(),
                                                                         fixed_image_points_flat,
                                                                         updated_moving_image_points_flat))

semi_automatic_errors_mean, semi_automatic_errors_std, _, semi_automatic_errors_max,_ = ru.registration_errors(semi_automatic_transform,
                                                                                fixed_points,
                                                                                moving_points,
                                                                                display_errors=True,
                                                                                min_err=manual_errors_min,
                                                                                max_err = manual_errors_max)
print('After registration (semiautomatic point localization), errors (TRE) in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(semi_automatic_errors_mean,
                                                                                semi_automatic_errors_std,
                                                                                semi_automatic_errors_max))

gui.RegistrationPointDataAquisition(fixed_image, moving_image, fixed_window_level=(215,50), known_transformation=semi_automatic_transform)


I am working on the file "67_Registration_Semiautomatic_Homework.ipynb".
The first task is:
Manual Landmark Localization
We now localize N(>=3) landmarks in the two images. Note that you can zoom and pan the images, just remember to change the interaction mode from "edit" to "view".
NOTE: In edit mode, the GUI will force you to enter corresponding points by disabling the option for consecutively localizing multiple (>2) points in the same image. In view mode, point localization is disabled which is useful for zoooming/panning (in edit mode zooming/panning will also localize points due to the mouse button click).
Can you write Python code for it?

point_acquisition_interface = gui.RegistrationPointDataAquisition(fixed_image, moving_image, fixed_window_level=(215,50))


I am working on the file "67_Registration_Semiautomatic_Homework.ipynb".
The first task is:
Registration (manual landmark localization)
Evaluate the quality of the manual localization by registering the two images, and then comparing the registration errors using the known reference data.
Can you write Python code for it?

fixed_image_points, moving_image_points = point_acquisition_interface.get_points()

fixed_image_points_flat = [c for p in fixed_image_points for c in p]
moving_image_points_flat = [c for p in moving_image_points for c in p]
manual_localized_transformation = sitk.VersorRigid3DTransform(sitk.LandmarkBasedTransformInitializer(sitk.VersorRigid3DTransform(),
                                                                         fixed_image_points_flat,
                                                                         moving_image_points_flat))

manual_errors_mean, manual_errors_std, manual_errors_min, manual_errors_max,_ = \
    ru.registration_errors(manual_localized_transformation,
                           fixed_points,
                           moving_points,
                           display_errors=True)
print('After registration (manual point localization), errors (TRE) in millimeters, mean(std): {:.2f}({:.2f}), max: {:.2f}'.format(manual_errors_mean, manual_errors_std, manual_errors_max))


I am working on the file "67_Registration_Semiautomatic_Homework.ipynb".
The first task is:
We can also evaluate the registration qualitatively by using a linked cursor approach via the same GUI we used to localize corresponding points. This time the points will be added in pairs.
Can you write Python code for it?


I am working on the file "6_Deconvolution/6_1_sky_models.ipynb".
The first task is:
6.1 Sky Models
Before we dive into deconvolution methods we need to introduce the concept of a sky model. Since we are making an incomplete sampling of the visibilities with limited resolution we do not recover the 'true' sky from an observation. The dirty image is the 'true' sky convolved (effectively blurred) out by the array point spread function (PSF). As discussed in the previous chapter, the PSF acts as a type of low-pass spatial filter limiting our resolution of the sky. We would like to some how recover a model for the true sky. At the end of deconvolution one of the outputs is the sky model.
We can look at the deconvolution process backwards by taking an ideal sky, shown in the left figure below (it may be difficult to see but there are various pixels with different intensities), and convolving it with the PSF response of the KAT-7 array, shown on the right, this is a point-source based sky model which will be discussed below. The sky model looks like a mostly empty image with a few non-zero pixels. The PSF is the same as the one shown in the previous chapter.
Can you write Python code for it?

fig = plt.figure(figsize=(16, 7))

gc1 = aplpy.FITSFigure('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-model.fits', \
                       figure=fig, subplot=[0.0,0.1,0.35,0.8])
gc1.show_colorscale(vmin=-0.1, vmax=1.0, cmap='viridis')
gc1.hide_axis_labels()
gc1.hide_tick_labels()
plt.title('Sky Model')
gc1.add_colorbar()

gc2 = aplpy.FITSFigure('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-psf.fits', \
                       figure=fig, subplot=[0.5,0.1,0.35,0.8])
gc2.show_colorscale(cmap='viridis')
gc2.hide_axis_labels()
gc2.hide_tick_labels()
plt.title('KAT-7 PSF')
gc2.add_colorbar()

fig.canvas.draw()


I am working on the file "6_Deconvolution/6_1_sky_models.ipynb".
The first task is:
Import section specific modules:
Can you write Python code for it?

import matplotlib.image as mpimg
from IPython.display import Image
from astropy.io import fits
import aplpy

#Disable astropy/aplpy logging
import logging
logger0 = logging.getLogger('astropy')
logger0.setLevel(logging.CRITICAL)
logger1 = logging.getLogger('aplpy')
logger1.setLevel(logging.CRITICAL)

from IPython.display import HTML
HTML('../style/code_toggle.html')


I am working on the file "6_Deconvolution/6_1_sky_models.ipynb".
The first task is:
Left: a point-source sky model of a field of sources with various intensities. Right: PSF response of KAT-7 for a 6 hour observation at a declination of $-30^{\circ}$.
By convolving the ideal sky with the array PSF we effectively are recreating the dirty image. The figure on the left below shows the sky model convolved with the KAT-7 PSF. The centre image is the original dirty image created using uniform weighting in the previous chapter. The figure on the right is the difference between the two figures. The negative offset is an effect of the imager producing an absolute value PSF image. The main point to note is that the difference image shows the bright sources removed resulting in a fairly noise-like image.
Can you write Python code for it?

fig = plt.figure(figsize=(16, 5))

fh = fits.open('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-model.fits')
skyModel = fh[0].data
fh = fits.open('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-psf.fits')
psf = fh[0].data
fh = fits.open('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-dirty.fits')
dirtyImg = fh[0].data
fh = fits.open('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-residual.fits')
residualImg = fh[0].data

#convolve the sky model with the PSF
sampFunc = np.fft.fft2(psf) #sampling function
skyModelVis = np.fft.fft2(skyModel) #sky model visibilities
sampModelVis = sampFunc * skyModelVis #sampled sky model visibilities
convImg = np.fft.fftshift(np.fft.ifft2(sampModelVis)).real + residualImg #sky model convolved with PSF

gc1 = aplpy.FITSFigure(convImg, figure=fig, subplot=[0,0.0,0.30,1])
gc1.show_colorscale(vmin=-1., vmax=3.0, cmap='viridis')
gc1.hide_axis_labels()
gc1.hide_tick_labels()
plt.title('PSF convolved with Sky Model')
gc1.add_colorbar()

gc2 = aplpy.FITSFigure(dirtyImg, figure=fig, subplot=[0.33,0.0,0.30,1])
gc2.show_colorscale(vmin=-1., vmax=3.0, cmap='viridis')
gc2.hide_axis_labels()
gc2.hide_tick_labels()
plt.title('Dirty')
gc2.add_colorbar()

gc3 = aplpy.FITSFigure(dirtyImg - convImg, figure=fig, subplot=[0.67,0.0,0.30,1])
gc3.show_colorscale(cmap='viridis')
gc3.hide_axis_labels()
gc3.hide_tick_labels()
plt.title('Difference')
gc3.add_colorbar()

fig.canvas.draw()


I am working on the file "6_Deconvolution/6_1_sky_models.ipynb".
The first task is:
Left: residual image after running a CLEAN deconvolution. Right: restored image constructed from convolving the point-source sky model with an 'ideal' PSF and adding the residual image.
Deconvolution, as can be seen in the figure on the left, builds a sky model by subtracts sources from the dirty image and adding them to a sky model. The resulting image shows the residual flux which was not added to the sky model. The restored image is a reconstruction of the field by convolving the sky model with an 'ideal' PSF and adding the residual image. This process will be discussed in the sections that follow.
The assumption that most sources are point source-like or can be represented by a set of point sources is the basis for the standard deconvolution process in radio interferometry, CLEAN. Though there are many new methods to perform deconvolution, CLEAN is the standard method and continues to dominate the field.
6.1.2 Resolved Sources
We need to consider what it means for a source to be 'resolved'. Each baseline in an array measures a particular spatial resolution. If the angular scale of a source is smaller than the spatial resolution of the longest baseline then the source is unresolved on every baseline. If the angular scale of a source is larger than the shortest baseline spatial resolution then the source is resolved on all baselines. In between these two extremes a source is resolved on longer baselines and unresolved on shorter baselines, thus the source is said to be partially resolved. The term extended source is often used as a synonym for fully- and partially-resolved sources.
Simple Gaussian extended sources are shown in the figure below. All sources have been normalized to have the same integrated flux. On the left is a very extended source, moving right is progressively smaller extended sources until the right most source which is a nearly point-source like object. Transforming each source into the visibility space, via Fourier transform, we plot the flux of each source as a function of baseline...
Can you write Python code for it?

def gauss2d(sigma):
    """Return a normalized 2d Gaussian function, sigma: size in pixels"""
    return lambda x,y: (1./(2.*np.pi*(sigma**2.))) * np.exp(-1. * ((xpos**2. + ypos**2.) / (2. * sigma**2.)))

imgSize = 512
xpos, ypos = np.mgrid[0:imgSize, 0:imgSize].astype(float)
xpos -= imgSize/2.
ypos -= imgSize/2.
sigmas = [64., 16., 4., 1.]

fig = plt.figure(figsize=(16, 7))

#Gaussian image-domain source
ax1 = plt.subplot2grid((2, 4), (0, 0))
gauss1 = gauss2d(sigmas[0])
ax1.imshow(gauss1(xpos, ypos))
ax1.axis('off')
plt.title('Sigma: %i'%int(sigmas[0]))

#Gaussian image-domain source
ax2 = plt.subplot2grid((2, 4), (0, 1))
gauss2 = gauss2d(sigmas[1])
ax2.imshow(gauss2(xpos, ypos))
ax2.axis('off')
plt.title('Sigma: %i'%int(sigmas[1]))

#Gaussian image-domain source
ax3 = plt.subplot2grid((2, 4), (0, 2))
gauss3 = gauss2d(sigmas[2])
ax3.imshow(gauss3(xpos, ypos))
ax3.axis('off')
plt.title('Sigma: %i'%int(sigmas[2]))

#Gaussian image-domain source
ax4 = plt.subplot2grid((2, 4), (0, 3))
gauss4 = gauss2d(sigmas[3])
ax4.imshow(gauss4(xpos, ypos))
ax4.axis('off')
plt.title('Sigma: %i'%int(sigmas[3]))

#plot the visibility flux distribution as a function of baseline length
ax5 = plt.subplot2grid((2, 4), (1, 0), colspan=4)
visGauss1 = np.abs( np.fft.fftshift( np.fft.fft2(gauss1(xpos, ypos))))
visGauss2 = np.abs( np.fft.fftshift( np.fft.fft2(gauss2(xpos, ypos))))
visGauss3 = np.abs( np.fft.fftshift( np.fft.fft2(gauss3(xpos, ypos))))
visGauss4 = np.abs( np.fft.fftshift( np.fft.fft2(gauss4(xpos, ypos))))
ax5.plot(visGauss1[int(imgSize/2),int(imgSize/2):], label='%i'%int(sigmas[0]))
ax5.plot(visGauss2[int(imgSize/2),int(imgSize/2):], label='%i'%int(sigmas[1]))
ax5.plot(visGauss3[int(imgSize/2),int(imgSize/2):], label='%i'%int(sigmas[2]))
ax5.plot(visGauss4[int(imgSize/2),int(imgSize/2):], label='%i'%int(sigmas[3]))
ax5.hlines(1., xmin=0, xmax=int(imgSize/2)-1, linestyles='dashed')
plt.legend()
plt.ylabel('Flux')
plt.xlabel('Baseline Length')
plt.xlim(0, int(imgSize/8)-1)
ax5.set_xticks([])
ax5.set_yticks([])


I am working on the file "6_Deconvolution/6_1_sky_models.ipynb".
The first task is:
Left: the point-source sky model convolved with the KAT-7 PSF with the residual image added. Centre: the original dirty image. Right: the difference between the PSF-convoled sky model and the dirty image.
Now that we see we can recreate the dirty image from a sky model and the array PSF, we just need to learn how to do the opposite operation, deconvolution. In order to simplify the process we incorporate knowledge about the sky primarily by assuming a simple model for the sources.
6.1.1 The Point Source Assumption
We can use some prior information about the sources in the sky and the array as a priori information in our deconvolution attempt. The array and observation configuration results in a PSF which has a primary lobe of a particular scale, this is the effective resolution of the array. As we have seen in $\S$ 5.4 ➞ the choice of weighting functions can result in different PSF resolutions. But, no matter the array there is a limit to the resolution, so any source which has a smaller angular scale than the PSF resolution appears to be a point source. A point source is an idealized source which has no angular scale and is represented by a spatial Dirac $\delta$-function. Though all sources in the sky have a angular scale, many are much smaller than the angular resolution of the array PSF, so they can be approximated as a simple point source.
A nice features of the point source model of a source is that the Fourier transform of a Dirac $\delta$-function, by the Fourier shift theorem, is a simple complex phase function and the constant flux
$$
\mathscr{F} { C(\nu) \cdot \delta\,(l- l_0, m - m_0)}(u, v) = C(\nu) \cdot\iint \limits_{-\infty}^{\infty} \delta\,(l- l_0, m - m_0) \, e^{-2 \pi i (ul + vm)}\,dl\,dm = C(\nu) \cdot e^{-2 \pi i (ul_0 + vm_0)} \quad (6.1.1)
$$
where $C(\nu)$ is the flux of the source (which can include a dependence on observing frequency $\nu$), and $(l_0, m_0)$ is the phase centre. At the phase centre $(l_0, m_0)$, $\mathscr{F} { C \cdot \delta\,(0, 0)} = C$. A $\delta$-function...
Can you write Python code for it?

fig = plt.figure(figsize=(16, 7))

gc1 = aplpy.FITSFigure('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-residual.fits', \
                       figure=fig, subplot=[0.1,0.1,0.35,0.8])
gc1.show_colorscale(vmin=-0.8, vmax=3., cmap='viridis')
gc1.hide_axis_labels()
gc1.hide_tick_labels()
plt.title('Residual')
gc1.add_colorbar()

gc2 = aplpy.FITSFigure('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-image.fits', \
                       figure=fig, subplot=[0.5,0.1,0.35,0.8])
gc2.show_colorscale(vmin=-0.8, vmax=3., cmap='viridis')
gc2.hide_axis_labels()
gc2.hide_tick_labels()
plt.title('Restored')
gc2.add_colorbar()

fig.canvas.draw()


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
6.2 Interative Deconvolution with Point Sources (CLEAN)
If we carry on with the logic from the previous section ➞ that we can use point sources as a generic unit for building sky models then we are ready to introduce the standard deconvolution method in radio interfereometry, CLEAN. Deconvolution is a broad term, that is defined by different, discipline-specific techniques to separate out two functions which have previously been (approximately) convolved together. Such as in the case of the array point spread function and the sky.
This problem of deconvolution seems fairly simple at first glance. We know how to convolve two functions $f$ and $g$ to produce an new function $h$ where
$$h = f \circ g \qquad.$$
Given, $h$ and one of the other functions, say $g$ then $f$ can be deconvolved by using the convolution theorem
$$\mathscr{F}{f} = \frac{\mathscr{F}{h}}{\mathscr{F}{g}}$$
Easy, we have solved deconvolution, unfortunately this is only true for a noise-less system. And, as we will see in the next chapter ➞, our interferometric observation definitely contains noise in the visibility measurements. When we are measuring the sky visibility function we are convolving the sky with the array response and some instrumental noise $\epsilon$.
$$I^D = \text{PSF} \circ I_{true} + \epsilon$$
If this $\epsilon$ is small compared to the sky signal then we can approximate $\epsilon$ as 0 and do this simple deconvolution method, which is called inverse filtering. Unfortunately, in radio astronomy, and most cases where deconvolution is required, $\epsilon$ is not small. Applying an inverse filter to a noisy signal will result in a poor estimate of the original signal. As an example of why inverse filtering fails in synthesis imaging let us apply the method to the simple KAT-7 observation we have been using in previous sections. As this is simulated data we know the exact sky model and produce a dirty image which is free of any calibration issues, i.e. the observation is full known.
Can you write Python code for it?

fig = plt.figure(figsize=(16, 7))

gc1 = aplpy.FITSFigure('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-dirty.fits', \
                       figure=fig, subplot=[0.0,0.1,0.35,0.8])
gc1.show_colorscale(vmin=-1., vmax=3.0, cmap='viridis')
gc1.hide_axis_labels()
gc1.hide_tick_labels()
plt.title('Dirty Image')
gc1.add_colorbar()

gc2 = aplpy.FITSFigure('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-psf.fits', \
                       figure=fig, subplot=[0.5,0.1,0.35,0.8])
gc2.show_colorscale(cmap='viridis')
gc2.hide_axis_labels()
gc2.hide_tick_labels()
plt.title('KAT-7 PSF')
gc2.add_colorbar()

fig.canvas.draw()


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
After a few iterations the residual image is beginning to look noise like, with some PSF structure remaining. As the remaining pixel value approach the noise level we need to halt deconvolution. This is a tricky decision and many issues occur when we 'over' or 'under' deconvolve an image. The resulting artefacts are further exasperated by poor calibration, but that will come later. For now though, the process of halting an iterative deconvolution requires setting either a mimimum flux threshold which halts when all pixels in the residual image are at or below the threshold, or we keep track of the number of iterations and halt after a specified number of cycles.
As for the final sky model, we can print the components.
Can you write Python code for it?

# now sum the accumulated point source model ("clean components") into a model image
print 'Clean components:'
print 'x  y  flux'

I_model = np.zeros([imsize,imsize])
for x, y, f in model:
    print x, y, f
    I_model[x,y] += f


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
As we can see, many of the components are at or near the same pixel values. We should expect this as we are not removing all the flux from a pixel during an iteration, only a portion of the flux which is determined by the gain $g$ scale factor. These are not separate sources, but in the sky model they are presented as such. An additional step of source finding ($\S$ 6.5 ➞) needs to be applied to the model in order to combine these sources.
Plotting the deconvolved sky model image next to the original true sky model image (below) we can see that the three sources are in the new sky model, though the recovery is not perfect. Perhaps we did under-deconvolved, perhaps we are including some noise into the sky model, there are a number of issues that cause deconvolution to be imperfect. But, what we can see is that iterative deconvolution produces more realistic results compared to the inverse filtering methods we started the section discussing.
Can you write Python code for it?

plotmax = np.max(I)
plotmin = np.min(I)
fig, axes = plt.subplots(figsize=(16,6))

plt.subplot(121)
plt.title('True Sky')
plt.imshow(I, cmap=cm.jet, vmax=plotmax, vmin=plotmin, interpolation='nearest')
plt.colorbar()

plt.subplot(122)
plt.title('Deconvolved Sky')
plt.imshow(I_model, cmap=cm.jet, vmax=plotmax, vmin=plotmin, interpolation='nearest')
plt.colorbar();


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
Figure: uv coverage from a declination $-30^{\circ}$, 6 hour, 10 MHz observation with KAT-7.
Conceptually, we can think of this partial $uv$ coverage as the product of the complete visibility function with full $uv$ coverage filling the UV-plane $V(u,v)$, with a sampling function $S(u,v)$ that is unity (for uniform weighting) where we have $uv$ samples and zero where we don't:
$$ S(u,v) \cdot V_{true}(u,v) $$
From the van Cittert-Zernike Theorem we know our image on the sky is the Fourier transform of the conceptual complete visibility function:
$$ I_{true}(l,m) = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} V_{true}(u,v) \; e^{-2\pi i(ul+vm)} \; du \, dv$$
But what we get from the interferometer isn't $V_{true}(u,v)$, it is the incomplete $uv$ coverage product $S(u,v) \cdot V_{true}(u,v)$:
$$ \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} S(u,v) \cdot V_{true}(u,v) \; e^{-2\pi i(ul+vm)} \; du \, dv \;=\; ?? $$
How do we deal with this? The Convolution Theorem states that multiplication in one domain is equivalent to convolution in the Fourier domain:
$$ h \cdot g \;\; \rightleftharpoons \;\; H \; \circ \; G $$
In our incomplete $uv$-coverage situation, we are multiplying the conceptual complete visibility function $V_{true}(u,v)$ by a sampling function $S(u,v)$, which equates to a convolution in the image domain:
$$ S(u,v) \cdot V_{true}(u,v) \;\; \rightleftharpoons \;\; \textrm{PSF}(l,m) \; \circ \; I_{true}(l,m) $$
where $I_{true}(l,m)$ is the true sky image, the Fourier Transform of the complete visibility $V(u,v)$, and $\textrm{PSF}(l,m)$ is the Fourier transform of the sampling function $S(u,v)$, the point spread functions. In older literature the term dirty beam is often used instead of the PSF, they are equivalent. We term the product $\textrm{PSF}(l,m) \; \circ \; I_{true}(l,m)$ the "Dirty Image" $I^D(l,m)$.

  Summary

•
The Fourier transform of our sampled visibility $V(u,v)$ data gives us a dirty image $I^D(l,m)$.

•...
Can you write Python code for it?

imsize = 50

# create noise background
noise_rms = 0.1
I = noise_rms*(np.random.random([imsize,imsize])-0.5)

# add three point sources with different flux values
I[20,20] += 1
I[32,15] += 1.45
I[30,34] += 1.12

plt.imshow(I, cmap=cm.jet, interpolation='nearest')
plt.colorbar()
plt.title('$I_{true}(l,m)$');


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
Import section specific modules:
Can you write Python code for it?

from IPython.display import Image
from IPython.core.display import clear_output, display

from scipy.signal import convolve2d

from copy import copy
import time

import matplotlib
matplotlib.rcParams['figure.figsize'] = (10.0, 6.0)
import matplotlib.cm as cm

from astropy.io import fits
import aplpy

#Disable astropy/aplpy logging
import logging
logger0 = logging.getLogger('astropy')
logger0.setLevel(logging.CRITICAL)
logger1 = logging.getLogger('aplpy')
logger1.setLevel(logging.CRITICAL)

HTML('../style/code_toggle.html')


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
Left: Dirty image of a point source field observed by KAT-7. Right: PSF response of KAT-7 for a 6 hour observation at a declination of $-30^{\circ}$.
Can you write Python code for it?

fh = fits.open('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-psf.fits') #KAT-7 PSF
psf = fh[0].data[0,0]
fh = fits.open('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-dirty.fits') #Dirty image
dirtyImg = fh[0].data[0,0]

#apply inverse filter
sampFunc = np.fft.fft2(psf)
obsVis = np.fft.fft2(dirtyImg)
trueVis = obsVis / sampFunc
trueImg = np.abs(np.fft.ifft2(trueVis))

fig = plt.figure(figsize=(16, 7))

gc1 = aplpy.FITSFigure(trueImg, figure=fig, subplot=[0.0,0.1,0.35,0.8])
gc1.show_colorscale(cmap='viridis')
gc1.hide_axis_labels()
gc1.hide_tick_labels()
plt.title('Model (Inverse Filtered)')
gc1.add_colorbar()

gc2 = aplpy.FITSFigure('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-model.fits', \
                       figure=fig, subplot=[0.5,0.1,0.35,0.8])
gc2.show_colorscale(vmin=-0.1, vmax=1.0, cmap='viridis')
gc2.hide_axis_labels()
gc2.hide_tick_labels()
plt.title('True Sky Model')
gc2.add_colorbar()

fig.canvas.draw()


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
Left: Dirty image thresholded to a minimum of 2. Right: The sky model recovered by applying inverse filtering to the thresholded dirty image.
By applying a threshold to the dirty image (figure above on the left) we are effectively selecting out only signal so then we should be able to recover a sky model of at least the brightest sources. Unfortunately, we do not, as can be seen in the figure above on the right. Instead we see rings around the bright sources. This is because we are removing the PSF structure around the sources in the dirty image. When we apply the inverse PSF filter those missing sidelobes result in source edge effects, i.e. rings. Now, this is not terrible as we could go through the image detecting circles with a Hough transform and come up with a sky model of positions (but not necessarily flux). But, surely we could do something even better.
First, we know the PSF response of the array. Second, we know that every source in the sky is convolved with (approximately) that PSF. We just need to find sources, which are not due to noise, and subtract the PSF multiplied by a factor (to account for different flux values) from that position in the image, and then add that position and flux multiplier to a sky model. Further, since each pixel contains some amount of noise or possible flux due to the PSF from some other source, we do not want to subtract all the flux in a single pixel in one go. Instead, we could iteratively subtract a portion of the flux in a pixel until some noise threshold is reached. This is called iterative deconvolution, a type of greedy algorithm, which is the basis what we call the class of CLEAN deconvoloution algorithms in synthesis imaging.
To understand the details of CLEAN lets do a swift review of the visibility sampling function and the array PSF.
6.2.1 Sampling, the PSF and the dirty image
In $\S$ 5.2 ➞ we discussed the effects of sampling in imaging. When we observe with a radio interferometric array we collect visibility data on...
Can you write Python code for it?


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
Left: a noise sky model derived by applying an inverse filter. Right: The true point source sky model.
We compute the inverse filtered deconvolved sky model using the array PSF and dirty image:
$$I_{true} = \mathscr{F}^{-1} \left { \frac{\mathscr{F}{I^D}}{\mathscr{F}{\textrm{PSF}}} \right } $$
The resulting sky model in the left figure above is just noise with some bright structure along the edges due to the edge effects. As we can see, we do not recover any sources in the true sky model (right figure above). We need a different method to deal with the noise.
Perhaps we can modify this inverse filter method by only selecting a part of the image which is clearly made up of sources and not noise. We can do this by setting a minimum threshold on the dirty image and applying the inverse filter to the thresholded image.
Can you write Python code for it?

#clip the dirty image to only select pixels above a minimum threshold, set all other pixels to zero
thresh = 2.
clippedDirtyImg = np.clip(dirtyImg, thresh, np.max(dirtyImg))

#apply inverse filter
sampFunc = np.fft.fft2(psf)
obsVis = np.fft.fft2(clippedDirtyImg)
trueVis = obsVis / sampFunc
trueImg = np.abs(np.fft.ifft2(trueVis))

fig = plt.figure(figsize=(16, 7))

gc1 = aplpy.FITSFigure(clippedDirtyImg, figure=fig, subplot=[0.0,0.1,0.35,0.8])
gc1.show_colorscale(vmin=thresh, vmax=10.0, cmap='viridis')
gc1.hide_axis_labels()
gc1.hide_tick_labels()
plt.title('Thresholded Dirty Image')
gc1.add_colorbar()

gc2 = aplpy.FITSFigure(trueImg, figure=fig, subplot=[0.5,0.1,0.35,0.8])
gc2.show_colorscale(cmap='viridis')
gc2.hide_axis_labels()
gc2.hide_tick_labels()
plt.title('Model (Inverse Filtered)')
gc2.add_colorbar()

fig.canvas.draw()


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
Let us plot the original dirty image and the final residual image scaled to see the residuals:
Can you write Python code for it?

plotmax = np.max(I_residual)
plotmin = np.min(I_residual)

fig, axes = plt.subplots(figsize=(16,6))

plt.subplot(121)
plt.title('$I^D(l,m)$')
plt.imshow(I_dirty, cmap=cm.jet, vmax=plotmax, vmin=plotmin, interpolation='nearest')
plt.colorbar()

plt.subplot(122)
plt.title('$I^R(l,m)$')
plt.colorbar()
plt.imshow(I_residual, cmap=cm.jet, vmax=plotmax, vmin=plotmin, interpolation='nearest');


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
Now let us plot the results of the CLEAN:
Can you write Python code for it?

plotmax = np.max(I_dirty)
plotmin = np.min(I_dirty)

fig, axes = plt.subplots(figsize=(16,12))

plt.subplot(221)
plt.imshow(I, cmap=cm.jet, vmax=plotmax, vmin=plotmin, interpolation='nearest')
plt.title('$I_{true}(l,m)$')
plt.colorbar()

plt.subplot(222)
plt.colorbar()
plt.title('$I^D(l,m)$')
plt.imshow(I_dirty, cmap=cm.jet, vmax=plotmax, vmin=plotmin, interpolation='nearest');

plt.subplot(223)
plt.title('I_model(l,m)')
plt.imshow(I_model, cmap=cm.jet, vmax=plotmax, vmin=plotmin, interpolation='nearest')
plt.colorbar()

plt.subplot(224)
plt.colorbar()
plt.title('I_restored(l,m)');
plt.imshow(I_restored, cmap=cm.jet, vmax=plotmax, vmin=plotmin, interpolation='nearest');


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
Now set up a fake PSF. We will just make up a star shape to be the PSF:
Can you write Python code for it?

PSFsize = 13
PSF = np.zeros([PSFsize,PSFsize])
PSFmid = (PSFsize - 1)/2
PSF[:,PSFmid] = 0.5
PSF[PSFmid,:] = 0.5
d1, d2 = np.diag_indices_from(PSF)
PSF[d1,d2] = 0.5
PSF[d1,d2[::-1]] = 0.5
PSF[PSFmid-2:PSFmid+3,PSFmid-2:PSFmid+3] = 0
PSF[PSFmid-1:PSFmid+2,PSFmid-1:PSFmid+2] = 0.75
PSF[PSFmid,PSFmid] = 1.0

plt.imshow(PSF, cmap = cm.jet, interpolation='nearest')
plt.colorbar()
plt.title('PSF(l,m)');


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
Now we convolve our true sky image $I_{true}(l,m)$ with the PSF, to get the dirty image $I^D(l,m)$
Can you write Python code for it?


I am working on the file "6_Deconvolution/6_2_clean.ipynb".
The first task is:
The last steps of CLEAN are somewhat optional, they are performed to produce a 'restored' image, this can be thought of an idealized image produced by an interferometric array with the same resolution as the observing array but with all spatial modes (up to the maximum resolution) fully sampled. That is, a restoring beam, which is usually taken to be a 2-D Gaussian of the same width as the main lobe of the PSF, is convoled with the point source sky model. This is done to reintroduce the the resolution limits of the array without including the PSF sidelobe structure. The restored image is the 'pretty' image we like to show off.
In our toy example here it is hardly worth doing a proper Gaussian fit, but let's do it anyway:
Can you write Python code for it?

# first get just the main lobe of the star shaped PSF
main_lobe = np.zeros([PSFsize,PSFsize])
main_lobe[PSFmid-1:PSFmid+2,PSFmid-1:PSFmid+2] = 0.75
main_lobe[PSFmid,PSFmid] = 1.0

fig, axes = plt.subplots(figsize=(16,6))
plt.subplot(121)
plt.imshow(PSF, cmap=cm.jet, interpolation='nearest')
plt.colorbar()
plt.title('PSF(l,m)');

plt.subplot(122)
plt.imshow(main_lobe, cmap=cm.jet, interpolation='nearest')
plt.colorbar()
plt.title('main lobe(l,m)');

# now fit a symmetric 2D gaussian to the main lobe
import scipy.optimize as opt

def gaussian2dsymmetric((x,y),A,x0,y0,sigma):
    gauss2d = A*np.exp(-((x-x0)**2.0 + (y-y0)**2.0)/(2.*sigma**2.0))
    return gauss2d.ravel()

x,y = np.meshgrid(range(PSFsize),range(PSFsize))
popt, pcov = opt.curve_fit(gaussian2dsymmetric,(x, y),main_lobe.ravel(), p0=[1.0,6.5,6.5,2.])
A, x0, y0, sigma = popt
print "Fit results:"
print "A: {0},  x0: {1}  y0: {2}  sigma: {3}".format(A,x0,y0,sigma)

# use fitted values to create CLEAN beam (or restoring beam)
#   normalise by dividing through by A
clean_beam = gaussian2dsymmetric((x,y),A,x0,y0,sigma).reshape(PSFsize,PSFsize)/A

# plot the CLEAN beam
plt.imshow(clean_beam, cmap=cm.jet, interpolation='nearest')
plt.colorbar()
plt.title('CLEAN beam(l,m)');


I am working on the file "6_Deconvolution/6_4_residuals_and_iqa.ipynb".
The first task is:
6.4 Residuals and Image Quality
Using CLEAN or another deconvolution methods produces 'nicer' images than the dirty image (except when deconvolution gets out of control). What it means for an image to be 'nicer' is not a well defined metric, in fact it is almost completely undefined. When we talk of the quality of an image in synthesis imaging we rarely use a quantitative metric, but instead rely on the subjective opinion of the people looking at the image. I know, this is not very scientific. The field of computer vision has been around for decades, this is a field which has developed the objective metrics and techniques for image quality assessment. At some point in the future these methods will need to be incorporated into radio astronomy. This is bound to happen as we have moved to using automated calibration, imaging, and deconvolution pipelines.
We have two some what related questions we need to answer when we are reducing visibilities into a final image:

When should you halt the deconvolution process?
What makes a good image?

In $\S$ 6.2 ➞ we covered how we can separate out a sky model from noise using an iterative CLEAN deconvolution process. But, we did not discuss at what point we halt the process. There is no well-defined point in which to stop the process. Typically and ad hoc decision is made to run deconvolution for a fixed number of iterations or down to a certain flux level. These halting limits are set by adjusting the CLEAN parameters until a 'nice' image is produced. Or, if the visibilities have been flux calibrated, which is possible with some arrays, the signal is fixed to some real flux scale. Having knowledge about the array and observation a theoretical noise floor can be computed, then CLEAN can be ran to a known noise level. One could imagine there is a more automated way to decide when to halt CLEAN, perhaps keeping track of the iterations and deciding if there is a convergence?
As a thought experiment we can think about a observation with perfect...
Can you write Python code for it?

def generalGauss2d(x0, y0, sigmax, sigmay, amp=1., theta=0.):
    """Return a normalized general 2-D Gaussian function
    x0,y0: centre position
    sigmax, sigmay: standard deviation
    amp: amplitude
    theta: rotation angle (deg)"""
    #norm = amp * (1./(2.*np.pi*(sigmax*sigmay))) #normalization factor
    norm = amp
    rtheta = theta * 180. / np.pi #convert to radians

    #general function parameters (https://en.wikipedia.org/wiki/Gaussian_function)
    a = (np.cos(rtheta)**2.)/(2.*(sigmax**2.)) + (np.sin(rtheta)**2.)/(2.*(sigmay**2.))
    b = -1.*(np.sin(2.*rtheta))/(4.*(sigmax**2.)) + (np.sin(2.*rtheta))/(4.*(sigmay**2.))
    c = (np.sin(rtheta)**2.)/(2.*(sigmax**2.)) + (np.cos(rtheta)**2.)/(2.*(sigmay**2.))
    return lambda x,y: norm * np.exp(-1. * (a * ((x - x0)**2.) - 2.*b*(x-x0)*(y-y0) + c * ((y-y0)**2.)))

def genRstoredBeamImg(fitsImg):
    """Generate an image of the restored PSF beam based on the FITS header and image size"""
    fh = fits.open(fitsImg)

    #get the restoring beam information from the FITS header
    bmin = fh[0].header['BMIN'] #restored beam minor axis (deg)
    bmaj = fh[0].header['BMAJ'] #restored beam major axis (deg)
    bpa = fh[0].header['BPA'] #restored beam angle (deg)
    dRA = fh[0].header['CDELT1'] #pixel size in RA direction (deg)
    ra0 = fh[0].header['CRPIX1'] #centre RA pixel
    dDec = fh[0].header['CDELT2'] #pixel size in Dec direction (deg)
    dec0 = fh[0].header['CRPIX2'] #centre Dec pixel

    #construct 2-D ellipitcal Gaussian function
    gFunc = generalGauss2d(0., 0., bmin/2., bmaj/2., theta=bpa)

    #produce an restored PSF beam image
    imgSize = 2.*(ra0-1) #assumes a square image
    xpos, ypos = np.mgrid[0:imgSize, 0:imgSize].astype(float) #make a grid of pixel indicies
    xpos -= ra0 #recentre
    ypos -= dec0 #recentre
    xpos *= dRA #convert pixel number to degrees
    ypos *= dDec #convert pixel number to degrees
    return gFunc(xpos, ypos) #restored PSF beam image

def convolveBeamSky(beamImg, skyModel):
    """Convolve...


I am working on the file "6_Deconvolution/6_4_residuals_and_iqa.ipynb".
The first task is:
6.4 Residuals and Image Quality
Using CLEAN or other deconvolution methods produces 'nicer' images than the dirty image (except when deconvolution gets out of control). What it means for an image to be 'nicer' is not a well defined metric, in fact it is almost completely undefined. When we talk of the quality of an image in synthesis imaging we rarely use a quantitative metric, but instead rely on the subjective opinion of the people looking at the image. I know, this is not very scientific. The field of computer vision has been around for decades, this is a field which has developed the objective metrics and techniques for image quality assessment. At some point in the future these methods will need to be incorporated into radio astronomy. This is bound to happen as we have moved to using automated calibration, imaging, and deconvolution pipelines.
We have two some what related questions we need to answer when we are reducing visibilities into a final image:

When should you halt the deconvolution process?
What makes a good image?

In $\S$ 6.2 ➞ we covered how we can separate out a sky model from noise using an iterative CLEAN deconvolution process. But, we did not discuss at what point we halt the process. There is no well-defined point in which to stop the process. Typically and ad hoc decision is made to run deconvolution for a fixed number of iterations or down to a certain flux level. These halting limits are set by adjusting the CLEAN parameters until a 'nice' image is produced. Or, if the visibilities have been flux calibrated, which is possible with some arrays, the signal is fixed to some real flux scale. Having knowledge about the array and observation a theoretical noise floor can be computed, then CLEAN can be run to a known noise level. One could imagine there is a more automated way to decide when to halt CLEAN, perhaps keeping track of the iterations and deciding if there is a convergence?
As a thought experiment we can think about a observation with perfect...
Can you write Python code for it?


I am working on the file "6_Deconvolution/6_4_residuals_and_iqa.ipynb".
The first task is:
Figure: residual image and sky model after 1000 deconvolution iterations. The residual image has been over-deconvolved leading to noise components being added to the sky model.
The second question of what makes a good image is why we still use subjective opinion. If we consider the realistic case of imaging and deconvolving a real set of visibilities then we have the added problem that there will be always be, at some level, calibration errors. These errors, and cause of these errors, can be identified by a trained eye whether it is poor gain calibration, interference, strong source sidelobes, or any number of other issues. Errors can cause a deconvolution process to diverge resulting in an unrealistic sky model. Humans are very good at looking at images and deciding if they make sense, but we can not easily describe how we do our image processing, thus we find it hard to implement algorithms to do the same. Looking at the dirty image and deconvolved image of the same field below most people would say the deconvoled image is objectively 'better' than the dirty image. Yet we do not know exactly why that is the case.
Can you write Python code for it?

fig = plt.figure(figsize=(16, 7))

gc1 = aplpy.FITSFigure('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-dirty.fits', \
                       figure=fig, subplot=[0.1,0.1,0.35,0.8])
gc1.show_colorscale(vmin=-1.5, vmax=3., cmap='viridis')
gc1.hide_axis_labels()
gc1.hide_tick_labels()
plt.title('Dirty Image')
gc1.add_colorbar()

gc2 = aplpy.FITSFigure('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-image.fits', \
                       figure=fig, subplot=[0.5,0.1,0.35,0.8])
gc2.show_colorscale(vmin=-1.5, vmax=3., cmap='viridis')
gc2.hide_axis_labels()
gc2.hide_tick_labels()
plt.title('Deconvolved Image')
gc2.add_colorbar()

fig.canvas.draw()


I am working on the file "6_Deconvolution/6_4_residuals_and_iqa.ipynb".
The first task is:
Left: dirty image from a 6 hour KAT-7 observation at a declination of $-30^{\circ}$. Right: deconvolved image.
The deconvolved image does not have the same noisy PSF structures around the sources that the dirty image does. We could say that these imaging artefacts are localized and related to the PSF response to bright sources. The aim of deconvolution is to remove these PSF like structures and replace them with a simple sky model which is decoupled fro the observing system. Most of difficult work in radio interferometry is the attempt to understand and remove the instrumental effects in order to recover the sky signal. Thus, we have some context for why the deconvolved image is 'better' than the dirty image. The challenge in automatically answering what makes a good image is some how encoding both the context and human intuition. Indeed, a challenge left to the reader.
6.4.1 Dynamic Range and Signal-to-Noise Ratio
Dynamic range is the standard metric, which has been used for decades, to describe the quality of an interferometric image. The dynamic range (DR) is defined as the ratio of the peak flux $I_{\textrm{peak}}$ to the standard deviation of the noise in the image $\sigma_I$. The dynamic range can be computed for either a dirty or deconvolved image.
$$\textrm{DR} = \frac{I_{\textrm{peak}}}{\sigma_I}$$
Now this definition of the dynamic range is not well defined. First, how is the peak flux defined? Typically, the peak pixel value anywhere in the image is taken to be the peak flux. But, be careful, changing the resolution of the image will result in different flux values. Decreasing the resolution can result in more flux being included in a single pixel, likewise by increasing the resolution the flux will be spread across more pixels. The second issue is how the noise of the image is computed, possible options are:

Use the entire image
Use the entire residual image
Randomly sample the image
Choose a 'relatively' empty region

This is not an exhaustive list of methods, but the typical method is option 4....
Can you write Python code for it?

#load deconvolved image
fh = fits.open('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-image.fits')
deconvImg = fh[0].data
#load residual image
fh = fits.open('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-residual.fits')
residImg = fh[0].data

peakI = np.max(deconvImg)
print 'Peak Flux: %f Jy'%(peakI)

print 'Dynamic Range:'
#method 1
noise = np.std(deconvImg)
print '\tMethod 1:', peakI/noise

#method 2
noise = np.std(residImg)
print '\tMethod 2:', peakI/noise

#method 3
noise = np.std(np.random.choice(deconvImg.flatten(), int(deconvImg.size*.01))) #randomly sample 1% of pixels
print '\tMethod 3:', peakI/noise

#method 4, region 1
noise = np.std(deconvImg[0,0,0:128,0:128]) #corner of image
print '\tMethod 4a:', peakI/noise

#method 4, region 2
noise = np.std(deconvImg[0,0,192:320,192:320]) #centre of image
print '\tMethod 4b:', peakI/noise


I am working on the file "6_Deconvolution/6_4_residuals_and_iqa.ipynb".
The first task is:
Left: dirty image from a 6 hour KAT-7 observation at a declination of $-30^{\circ}$. Right: deconvolved image.
The deconvolved image does not have the same noisy PSF structures around the sources that the dirty image does. We could say that these imaging artefacts are localized and related to the PSF response to bright sources. The aim of deconvolution is to remove these PSF like structures and replace them with a simple sky model which is decoupled fro the observing system. Most of difficult work in radio interferometry is the attempt to understand and remove the instrumentational effects in order to recover the sky signal. Thus, we have some context for why the deconvolved image is 'better' than the dirty image. The challenge in automatically answering what makes a good image is some how encoding both the context and human intution. Indeed, a challenge left to the reader.
6.4.1 Dynamic Range and Signal-to-Noise Ratio
Dynamic range is the standard metric, which has been used for decades, to describe the quality of an interferometric image. The dynamic range (DR) is defined as the ratio of the peak flux $I_{\textrm{peak}}$ to the standard deviation of the noise in the image $\sigma_I$. The dynamic range can be computed for either a dirty or deconvolved image.
$$\textrm{DR} = \frac{I_{\textrm{peak}}}{\sigma_I}$$
Now this definition of the dynamic range is not well defined. First, how is the peak flux defined? Typically, the peak pixel value anywhere in the image is taken to be the peak flux. But, be careful, changing the resolution of the image will result in different flux values. Decreasing the resolution can result in more flux being included in a single pixel, likewise by increasing the resolution the flux will be spread across more pixels. The second issue is how the noise of the image is computed, possible options are:

Use the entire image
Use the entire residual image
Randomly sample the image
Choose a 'relatively' empty region

This is not an exhaustive list of methods, but the typical method is...
Can you write Python code for it?


I am working on the file "6_Deconvolution/6_4_residuals_and_iqa.ipynb".
The first task is:
Method 1 will always result in a lower dynamic range than Method 2 as the deconvoled image includes the sources where method 2 only uses the residuals. Method 3 will result in a dynamic range which varies depending on the number of pixels sampled and which pixels are sampled. One could imagine an unlucky sampling where every pixel chosen is part of a source, resulting in a large standard deviation. Method 4 depends on the region used to compute the noise. In the Method 4a result a corner of the image, where there are essentially no sources, results in a high dynamic range. On the other hand, choosing the centre region to compute the noise standard deviation results in a low dynamic range. This variation between methods can lead to people playing 'the dynamic range game' where someone can pick the result that best fits what they want to say about the image. Be careful, and make sure your dynamic range metric is well defined and unbiased.
There is a qualitative explanation for computing the image noise and the dynamic range by human interaction. Humans are very good at image processing, so we can quickly select regions which are 'noise-like', so it is easier to just look at an image then to try to come up with a complicated algorithm to find these regions. The dynamic range has a number of issues, but it is correlated with image quality. For a fixed visibility set, improving the dynamic range of an image usually results in a improvement in the quality of the image, as determined by a human.
A significant disadvantage to using dynamic range is that it is a global metric which reduced an image down to a single number. It provides no information about local artefacts. This is becoming an important issue in modern synthesis imaging as we push into imaging significant portions of the primary beam and need to account for direction-dependent effects. These topics are discussed in Chapter 7. But, as is noted in <cite data-cite='taylor1999synthesis'>Synthesis Imaging in Radio Astronomy II (Lecture 13) </cite> ⤴...
Can you write Python code for it?

fig = plt.figure(figsize=(8, 7))

gc1 = aplpy.FITSFigure('../data/fits/deconv/KAT-7_6h60s_dec-30_10MHz_10chans_uniform_n100-residual.fits', \
                       figure=fig)
gc1.show_colorscale(vmin=-1.5, vmax=3., cmap='viridis')
gc1.hide_axis_labels()
gc1.hide_tick_labels()
plt.title('Residual Image')
gc1.add_colorbar()

fig.canvas.draw()


I am working on the file "6_Deconvolution/6_5_source_finding.ipynb".
The first task is:
6.5 Source Finding
In radio astronomy, source finding is the process through which the attributes of radio sources -- such as flux density and mophorlogy -- are measured from data. In this section we will only cover source finding in the image plane.
Source finding techniques usually involve four steps, i) charecterizing the noise (or background estimation), ii) thresholding the data based on knowledge of the noise, iii) fidinding regions in the thresholded image which neighbouring pixels are "similar" (this is that same as blob detection in image processing), and iv) parameterizing these 'blobs' through a function (usually a 2D Gaussian). The source attributes are then estimated from the parameterization of the blobs.
6.5.1 Noise Charecterization
As mentioned before, the radio data we process with source finders is noisy. To charecterise this noise we need to make a few assumptions about its nature, namely we assume that the niose results from some stochastic process and that it can be described by a normal distribution
$$ G(x \, | \, \mu,\sigma^2) = \frac{1}{\sigma \sqrt{2\pi}}\text{exp}\left( \frac{-(x-\mu)^2}{2\sigma^2}\right) $$
where, $\mu$ is the mean (or expected value) of the varaible $x$, and $\sigma^2$ is the varience of the distribution; $\sigma$ is the standard deviation. Hence, the noise can be parameterized through the mean and the standard deviation. Let us illustrate this with an example. Bellow is a noise image from a MeerKAT simulation, along with a histogram of of the pixels (in log space).
Can you write Python code for it?

noise_image = "../data/fits/noise_image.fits"
with astropy.io.fits.open(noise_image) as hdu:
    data = hdu[0].data[0,0,...]

fig, (image, hist) = plt.subplots(1, 2, figsize=(18,6))
histogram, bins = np.histogram(data.flatten(), bins=401)

dmin = data.min()
dmax = data.max()
x = np.linspace(dmin, dmax, 401)

im = image.imshow(data)

mean = data.mean()
sigma = data.std()
peak = histogram.max()

gauss = lambda x, amp, mean, sigma: amp*np.exp( -(x-mean)**2/(2*sigma**2))

fitdata = gauss(x, peak, mean, sigma)

plt.plot(x, fitdata)
plt.plot(x, histogram, "o")
plt.yscale('log')
plt.ylim(1)


I am working on the file "6_Deconvolution/6_5_source_finding.ipynb".
The first task is:
6.5 Source Finding
In radio astronomy, source finding is the process through which the attributes of radio sources -- such as flux density and mophorlogy -- are measured from data. In this section we will only cover source finding in the image plane.
Source finding techniques usually involve four steps, i) charecterizing the noise (or background estimation), ii) thresholding the data based on knowledge of the noise, iii) finding regions in the thresholded image with "similar" neighbouring pixels (this is that same as blob detection in image processing), and iv) parameterizing these 'blobs' through a function (usually a 2D Gaussian). The source attributes are then estimated from the parameterization of the blobs.
6.5.1 Noise Charecterization
As mentioned before, the radio data we process with source finders is noisy. To charecterize this noise we need to make a few assumptions about its nature, namely we assume that the niose results from some stochastic process and that it can be described by a normal distribution
$$ G(x \, | \, \mu,\sigma^2) = \frac{1}{\sigma \sqrt{2\pi}}\text{exp}\left( \frac{-(x-\mu)^2}{2\sigma^2}\right) $$
where, $\mu$ is the mean (or expected value) of the variable $x$, and $\sigma^2$ is the variance of the distribution; $\sigma$ is the standard deviation. Hence, the noise can be parameterized through the mean and the standard deviation. Let us illustrate this with an example. Bellow is a noise image from a MeerKAT simulation, along with a histogram of of the pixels (in log space).
Can you write Python code for it?


I am working on the file "6_Deconvolution/6_5_source_finding.ipynb".
The first task is:
Now, in reality the noise has to measured in the presence of astrophysical emission. Furthermore, radio images are also contaminated by various instrumental effects which can manifest as spurious emission in the image domain. All these factors make it difficult to charercterize the noise in a synthesized image. Since the noise generally dominates the images, the mean and standard deviation of the entire image are still fairly good approximations of the noise. Let us now insert a few sources (image and flux distribution shown below) in the noise image from earlier and then try to estimate noise.
Can you write Python code for it?

noise_image = "../data/fits/star_model_image.fits"
with astropy.io.fits.open(noise_image) as hdu:
    data = hdu[0].data[0,0,...]

fig, (image, hist) = plt.subplots(1, 2, figsize=(18,6))
histogram, bins = np.histogram(data.flatten(), bins=101)

dmin = data.min()
dmax = data.max()
x = np.linspace(dmin, dmax, 101)

im = image.imshow(data)

mean = data.mean()
sigma_std = data.std()

peak = histogram.max()

gauss = lambda x, amp, mean, sigma: amp*np.exp( -(x-mean)**2/(2*sigma**2))

fitdata_std = gauss(x, peak, mean, sigma_std)

plt.plot(x, fitdata_std, label="STD DEV")

plt.plot(x, histogram, "o", label="Data")
plt.legend(loc=1)

plt.yscale('log')
plt.ylim(1)


I am working on the file "6_Deconvolution/6_5_source_finding.ipynb".
The first task is:
The MAD and negtive value standard deviation methods produce a better solution to the noise distribution in the presence of sources.
6.5.2 Blob Detection and Charercterization
Once the noise has been estimated, the next step is to find and charecterize sources in the image. Generically in image processing this is known as blob detection. In a simple case during synthesis imaging we define a blob as a group contiguous pixels whose spatial intensity profile can be modelled by a 2D Gaussian function. Of course, more advanced functions could be used. Generally, we would like to group together near by pixels, such as spatially 'close' sky model components from deconvolution, into a single complex source. Our interferometric array has finite spatial resolution, so we can further constrain our blobs not to be significantly smaller than the image resolution. We define two further constraints of a blob, the peak and boundary thresholds. The peak threshold, defined as
$$
    \sigma_\text{peak} = n * \sigma,
$$
is the minimum intensity the maximum pixel in a blob must have relative to the image noise. That is, all blobs with peak pixel lower than $\sigma_\text{peak}$ will be excluded from being considered sources. And the boundary threshold
$$
    \sigma_\text{boundary} = m * \sigma,
$$
defines the boundary of a blob, $m$ and $n$ are natural numbers with $m$ < $n$.
6.5.2.1 A simple source finder
We are now in a position to write a simple source finder. To do so we implement the following steps:

Estimate the image noise and set peak and boundary threshold values.
Blank out all pixel values below the boundary value.
Find Peaks in image.
For each peak, fit a 2D Gaussian and subtract the Gaussian fit from the image.
Repeat until the image has no pixels above the detection threshold.
Can you write Python code for it?

def gauss2D(x, y, amp, mean_x, mean_y, sigma_x, sigma_y):
    """ Generate a 2D Gaussian image"""
    gx = -(x - mean_x)**2/(2*sigma_x**2)
    gy = -(y - mean_y)**2/(2*sigma_y**2)

    return amp * np.exp( gx + gy)

def err(p, xx, yy, data):
    """2D Gaussian error function"""
    return gauss2D(xx.flatten(), yy.flatten(), *p) - data.flatten()

def fit_gaussian(data, psf_pix):
    """Fit a gaussian to a 2D data set"""

    width = data.shape[0]
    mean_x, mean_y = width/2, width/2
    amp = data.max()
    sigma_x, sigma_y = psf_pix, psf_pix
    params0 = amp, mean_x, mean_y, sigma_x,sigma_y

    npix_x, npix_y = data.shape
    x = np.linspace(0, npix_x, npix_x)
    y = np.linspace(0, npix_y, npix_y)
    xx, yy = np.meshgrid(x, y)

    params, pcov, infoDict, errmsg, sucess = optimize.leastsq(err,
                        params0, args=(xx.flatten(), yy.flatten(),
                        data.flatten()), full_output=1)

    perr = abs(np.diagonal(pcov))**0.5
    model = gauss2D(xx, yy, *params)

    return params, perr, model

def source_finder(data, peak, boundary, width, psf_pix):
    """A simple source finding tool"""

    # first we make an estimate of the noise. Lets use the MAD mean
    sigma_noise = mad_mean(data)

    # Use noise estimate to set peak and boundary thresholds
    peak_sigma = sigma_noise*peak
    boundary_sigma = sigma_noise*boundary

    # Pad the image to avoid hitting the edge of the image
    pad = width*2
    residual = np.pad(data, pad_width=((pad, pad), (pad, pad)), mode="constant")
    model = np.zeros(residual.shape)

    # Create slice to remove the padding later on
    imslice = [slice(pad, -pad), slice(pad,-pad)]

    catalog = []

    # We will need to convert the fitted sigma values to a width
    FWHM = 2*np.sqrt(2*np.log(2))

    while True:

        # Check if the brightest pixel is at least as bright as the sigma_peak
        # Otherwise stop.
        max_pix = residual.max()
        if...


I am working on the file "6_Deconvolution/6_5_source_finding.ipynb".
The first task is:
The pixel statistics of the image are no longer Gaussian as apparent from the long trail of the flux distribution. Constructing a Gaussian model from the mean and standard deviation results in a poor fit (blue line in the figure on the right). A better method to estimate the variance is to measure the dispersion of the data points about the mean (or median), this is the mean/median absolute deviation (MAD) technique. We will refer to the to median absolute deviation as the MAD Median, and the mean absolute deviation as the MAD Mean. A synthesis imaging specfic method to estimate the varience of the noise is to only consider the negative pixels. This works under the assumption that all the astrophysical emission (at least in Stokes I) has a positive flux density. The Figure below shows noise estimates from methods mentioned above.
Can you write Python code for it?

mean = data.mean()
sigma_std = data.std()
sigma_neg = data[data<0].std() * 2
mad_mean = lambda a: np.mean( abs(a - np.mean(a) ))
sigma_mad_median = np.median( abs(data - np.median(data) ))

mad_mean = lambda a: np.mean( abs(a - np.mean(a) ))
sigma_mad_mean = mad_mean(data)

peak = histogram.max()

gauss = lambda x, amp, mean, sigma: amp*np.exp( -(x-mean)**2/(2*sigma**2))

fitdata_std = gauss(x, peak, mean, sigma_std)
fitdata_mad_median = gauss(x, peak, mean, sigma_mad_median)
fitdata_mad_mean = gauss(x, peak, mean, sigma_mad_mean)
fitdata_neg = gauss(x, peak, mean, sigma_neg)

plt.plot(x, fitdata_std, label="STD DEV")
plt.plot(x, fitdata_mad_median, label="MAD Median")
plt.plot(x, fitdata_mad_mean, label="MAD Mean")
plt.plot(x, fitdata_neg, label="Negative STD DEV")
plt.plot(x, histogram, "o", label="Data")
plt.legend(loc=1)

plt.yscale('log')
plt.ylim(1)


I am working on the file "6_Deconvolution/6_5_source_finding.ipynb".
The first task is:
The pixel statistics of the image are no longer Gaussian as apparent from the long trail of the flux distribution. Constructing a Gaussian model from the mean and standard deviation results in a poor fit (blue line in the figure on the right). A better method to estimate the variance is to measure the dispersion of the data points about the mean (or median), this is the mean/median absolute deviation (MAD) technique. We will refer to the to median absolute deviation as the MAD Median, and the mean absolute deviation as the MAD Mean. A synthesis imaging specific method to estimate the variance of the noise is to only consider the negative pixels. This works under the assumption that all the astrophysical emission (at least in Stokes I) has a positive flux density. The Figure below shows noise estimates from methods mentioned above.
Can you write Python code for it?


I am working on the file "6_Deconvolution/6_5_source_finding.ipynb".
The first task is:
Using this source finder we can produce a sky model which contains all 17 sources in our test image from earlier in the section.
Can you write Python code for it?

test_image = "../data/fits/star_model_image.fits"
with astropy.io.fits.open(test_image) as hdu:
    data = hdu[0].data[0,0,...]

catalog, model, residual, sigma_noise = source_finder(data, 5, 2, 50, 10)

print "Peak_Flux     Pix_x     Pix_y     Size_x     Size_y"
for source in catalog:
    print "  %.4f     %.1f     %.1f     %.2f     %.2f"%source

fig, (img, mod, res) = plt.subplots(1, 3, figsize=(24,12))
vmin, vmax = sigma_noise, data.max()

im = img.imshow(data, vmin=vmin, vmax=vmax)
img.set_title("Data")

mod.imshow(model, vmin=vmin, vmax=vmax)
mod.set_title("Model")

res.imshow(residual, vmin=vmin, vmax=vmax)
res.set_title("Residual")

cbar_ax = fig.add_axes([0.92, 0.25, 0.02, 0.5])
fig.colorbar(im, cax=cbar_ax, format="%.2g")


I am working on the file "6_Error_Propagation.ipynb".
The first task is:
Compute predictions and errors for the test set
Steps:
1. compute predictions
\begin{equation}
h_(\textbf{x},\textbf{W}) =  h_{1}(\textbf{x},\textbf{W}),...,h_{j}(\textbf{x},\textbf{W}))
\end{equation}

    j = 3

compute jacobian matrix
\begin{equation}
Jac = \frac{\partial h_{j}(\textbf{x},\textbf{W})}{\partial \textbf{x}} =  (\frac{\partial h_{j}(\textbf{x},\textbf{W})}{\partial x_{1}},...,\frac{\partial h_{j}(\textbf{x},\textbf{W})}{\partial x_{n}})
\end{equation}
j = 1,...,3

n = 7214

compute covariance matrix
\begin{equation}
Cov = Jac \times \Delta \textbf{x}^2 \times Jac^T
\end{equation}

obtain propagated variance due to error spectrum from the diagonal of the covariance matrix
\begin{equation}
\sigma_{\mathrm{prop}}^2 \approx diag(Cov)
\end{equation}

determine which region of the label-space the labels are within to obtain the intrinsic scatter in the corresponding bin. These values have been predetermined from training StarNet on synthetic data and applying it to a test set of synthetic data
\begin{equation}
\sigma_{\mathrm{int}}
\end{equation}

combine propagated error with the intrinsic scatter term
\begin{equation}
\Delta h_{j} = \sqrt{\sigma_{\mathrm{prop}}^2  + \sigma_{\mathrm{int}}^2}
\end{equation}
Can you write Python code for it?

variance = np.zeros((len(test_labels),3))
predictions = np.zeros(test_labels.shape)
print('Making predictions and computing propagated variance for '+str(len(test_labels))+' spectra')
time_start = time.time()
for i in range(len(test_labels)):
    spectrum = test_spectra[i:i+1]
    err_spectrum = test_err_spectra[i:i+1]
    variance[i] = calc_variance(model,spectrum,err_spectrum,denormalize)
    predictions[i] = denormalize(model.predict(spectrum))
    if i%int(0.1*len(test_labels))==0:
        print('\n'+str(i+1)+' completed.\n'+str(time.time()-time_start)+' seconds elapsed.')
print('\nAll '+str(i+1)+' completed.\n'+str(time.time()-time_start)+' seconds elapsed.')
f.close()


I am working on the file "6_Error_Propagation.ipynb".
The first task is:
Create a denormalization function
Can you write Python code for it?

mean_and_std = np.load(denormalization_path)
mean_labels = mean_and_std[0]
std_labels = mean_and_std[1]
num_labels = mean_and_std.shape[1]

def denormalize(lb_norm):
    return ((lb_norm*std_labels)+mean_labels)


I am working on the file "6_Error_Propagation.ipynb".
The first task is:
Define functions to:

compute the jacobian matrix
compute the covariance
compute the variance

Note: these functions can be combined into one, but they are separated here to allow users to extract intermediate results for analysis
Can you write Python code for it?

def calc_jacobian(model,spectra,denormalize=None):

    if denormalize==None:
        y_list = tf.unstack(model.output)
    else:
        y_list = tf.unstack(denormalize(model.output[0]))

    J = [tf.gradients(y, model.input) for y in y_list]

    jacobian_func = [K.function([model.input, K.learning_phase()], j_) for j_ in J]

    jacobian = np.array([jf([spectra,False]) for jf in jacobian_func])[:,0,0,:]
    '''
    for i in range(len(spectra)):
        jacobian = np.array([jf([spectra,False]) for jf in jacobian_func])[:,:,0,:,0]
        np.save('temp/temp_jacobian_'+str(i)+'.npy',jacobian)
        if i%int(0.1*len(spectra))==0:
            print('Jacobians completed: '+str(i))

    for i in range(len(spectra)):
        if i==0:
            jacobian = np.load('temp/temp_jacobian_'+str(i)+'.npy')
        else:
            jacobian = np.concatenate((jacobian,np.load('temp/temp_jacobian_'+str(i)+'.npy')))
        subprocess.check_output(['rm','temp/temp_jacobian_'+str(i)+'.npy'])
    '''
    return jacobian

def calc_covariance(model,spectra,err_spectra,denormalize=None):
    jac_matrix = calc_jacobian(model,spectra,denormalize)
    err_spectra[err_spectra > 6] = 0
    jac_matrix = np.nan_to_num(jac_matrix)
    covariance = np.einsum('ij,jl->il',(jac_matrix*(err_spectra**2)),jac_matrix.T)
    return covariance

def calc_variance(model,spectra,err_spectra,denormalize=None):
    covariance = calc_covariance(model,spectra,err_spectra,denormalize)
    return np.diagonal(covariance, offset=0)


I am working on the file "6_Error_Propagation.ipynb".
The first task is:
Define path variables for your keras model, denormalization data, and test data
Can you write Python code for it?

model_path = datadir + 'starnet_cnn.h5'
denormalization_path = datadir + 'mean_and_std.npy'
test_data_path = datadir + 'test_data.h5'


I am working on the file "6_Error_Propagation.ipynb".
The first task is:
Load Test Data
The error propagation technique takes some time, so for the purpose of example, we will only use the first 100 spectra in the test set
Can you write Python code for it?

num_test = 300

f = h5py.File(test_data_path, 'r')
test_spectra = f['spectrum']
test_err_spectra = f['error_spectrum']
test_ap_ids = f['Ap_ID'][0:num_test]
test_labels = np.column_stack((f['TEFF'][0:num_test],f['LOGG'][0:num_test],f['FE_H'][0:num_test]))

print('Test set contains '  + str(len(test_ap_ids))+' stars')


I am working on the file "6_Genesis_postprocessor.ipynb".
The first task is:
Genesis Field file dfl
can be read with readRadiationFile function
Can you write Python code for it?

dfl=readRadiationFile(out_file+'.dfl', out.ncar)

gen_outplot_dfl(dfl, out)
plt.show()

gen_outplot_dfl(dfl, out, freq_domain=True,far_field=True)
plt.show()

dpa=read_particle_file(out_file+'.dpa',out('nbins'),out('npart'))
dist=dpa2dist(out,dpa)

plot_dist(dist, showfig=True)


I am working on the file "6_Grouping_and_Summarising.ipynb".
The first task is:
Notice that we have indexed the Profit column in the DataFrameGroupBy object exactly as we index a normal column in a dataframe. Alternatively, you could also use df_by_segment.Profit.
Can you write Python code for it?


I am working on the file "6_Grouping_and_Summarising.ipynb".
The first task is:
Say you want to understand how well or poorly the business is doing in various customer segments, regions, product categories etc. Specifically, you want to identify areas of business where you are incurrring heavy losses, and want to take action accordingly.
To do that, we will answer questions such as:
* Which customer segments are the least profitable?
* Which product categories and sub-categories are the least profitable?
* Customers in which geographic region cause the most losses?
* Etc.
First, we will merge all the dataframes, so we have all the data in one master_df.
Can you write Python code for it?

# Merging the dataframes one by one
df_1 = pd.merge(market_df, customer_df, how='inner', on='Cust_id')
df_2 = pd.merge(df_1, product_df, how='inner', on='Prod_id')
df_3 = pd.merge(df_2, shipping_df, how='inner', on='Ship_id')
master_df = pd.merge(df_3, orders_df, how='inner', on='Ord_id')

master_df.head()


I am working on the file "6_Grouping_and_Summarising.ipynb".
The first task is:
Thus, within FURNITURE, TABLES are the least profitable, followed by BOOKCASES.
Can you write Python code for it?

# Recall the df.describe() method?
# To apply multiple functions simultaneously, you can use the describe() function on the grouped df object
by_product_cat['Profit'].describe()

# Some other summary functions to apply on groups
by_product_cat['Profit'].count()

by_product_cat['Profit'].min()

# E.g. Customers in which geographic region are the least profitable?
master_df.groupby('Region').Profit.mean()

# Note that the resulting object is a Series, thus you can perform vectorised computations on them

# E.g. Calculate the Sales across each region as a percentage of total Sales
# You can divide the entire series by a number (total sales) easily
(master_df.groupby('Region').Sales.sum() / sum(master_df['Sales']))*100


I am working on the file "6_Linear_algebra.ipynb".
The first task is:
Q11. Calculate the determinant of x.
Can you write Python code for it?


I am working on the file "6_Linear_algebra.ipynb".
The first task is:
Q12. Calculate the rank of x.
Can you write Python code for it?


I am working on the file "6_Linear_algebra.ipynb".
The first task is:
Q13. Compute the sign and natural logarithm of the determinant of x.
Can you write Python code for it?


I am working on the file "6_Linear_algebra.ipynb".
The first task is:
Q14. Return the sum along the diagonal of x.
Can you write Python code for it?


I am working on the file "6_Linear_algebra.ipynb".
The first task is:
Q3. Predict the results of the following code.
Can you write Python code for it?

x = np.array([[1, 4], [5, 6]])
y = np.array([[4, 1], [2, 2]])
#print np.vdot(x, y)
#print np.vdot(y, x)
#print np.dot(x.flatten(), y.flatten())
#print np.inner(x.flatten(), y.flatten())
#print (x*y).sum()


I am working on the file "6_Linear_algebra.ipynb".
The first task is:
Q4. Predict the results of the following code.
Can you write Python code for it?

x = np.array(['a', 'b'], dtype=object)
y = np.array([1, 2])
#print np.inner(x, y)
#print np.inner(y, x)
#print np.outer(x, y)
#print np.outer(y, x)


I am working on the file "6_Linear_algebra.ipynb".
The first task is:
Q6. Compute the qr factorization of x and verify it.
Can you write Python code for it?


I am working on the file "6_Linear_algebra.ipynb".
The first task is:
Q9. Predict the results of the following code.
Can you write Python code for it?

#print np.array_equal(np.dot(x, eigenvecs), eigenvals * eigenvecs)


I am working on the file "6_Linear_algebra_Solutions.ipynb".
The first task is:
Decompositions
Q5. Get the lower-trianglular L in the Cholesky decomposition of x and verify it.
Can you write Python code for it?

x = np.array([[4, 12, -16], [12, 37, -43], [-16, -43, 98]], dtype=np.int32)
L = np.linalg.cholesky(x)
print L
assert np.array_equal(np.dot(L, L.T.conjugate()), x)


I am working on the file "6_Linear_algebra_Solutions.ipynb".
The first task is:
Matrix and vector products
Q1. Predict the results of the following code.
Can you write Python code for it?

x = [1,2]
y = [[4, 1], [2, 2]]
print np.dot(x, y)
print np.dot(y, x)
print np.matmul(x, y)
print np.inner(x, y)
print np.inner(y, x)


I am working on the file "6_Linear_algebra_Solutions.ipynb".
The first task is:
Q12. Calculate the rank of x.
Can you write Python code for it?

x = np.eye(4)
out1 = np.linalg.matrix_rank(x)
out2 = np.linalg.svd(x)[1].size
assert out1 == out2
print out1


I am working on the file "6_Linear_algebra_Solutions.ipynb".
The first task is:
Q14. Return the sum along the diagonal of x.
Can you write Python code for it?

x = np.eye(4)
out1 = np.trace(x)
out2 = x.diagonal().sum()
assert out1 == out2
print out1


I am working on the file "6_Linear_algebra_Solutions.ipynb".
The first task is:
Q3. Predict the results of the following code.
Can you write Python code for it?

x = np.array([[1, 4], [5, 6]])
y = np.array([[4, 1], [2, 2]])
print np.vdot(x, y)
print np.vdot(y, x)
print np.dot(x.flatten(), y.flatten())
print np.inner(x.flatten(), y.flatten())
print (x*y).sum()


I am working on the file "6_Linear_algebra_Solutions.ipynb".
The first task is:
Q6. Compute the qr factorization of x and verify it.
Can you write Python code for it?


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
As you can see, there is a column of solutions for each variable (X and Y, in this case).
So, we can plot these and observe the dynamics of our system.
Can you write Python code for it?

plt.figure()
plt.plot(t, sol[:, 0])
plt.plot(t, sol[:, 1])
plt.xlabel('Time (a.u.)')
plt.ylabel('Concentration (a.u.)')
plt.legend(['[X]', '[Y]'])


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
Dopri5 Solver
Can you write Python code for it?

# Define parameters as before
a = 1.0
b = 2.5

# Define initial conditions
y0 = [0.5, 0.5]
t0 = 0

# Final time
t1 = 25

# Define the ODE solver object
solver = ode(bruss_ode).set_integrator('dopri5', nsteps=10000)

# System/solver parameters are set as properties on the object
solver.set_initial_value(y0, t0).set_f_params(a, b)

# Dopri5 allows for a `solout` function to be attached.
# This function is run at the end of every complete timestep.
# We use it here just to append solutions to the full trajectory.
traj = []
def solout(t, y):
    traj.append([t, *y])

solver.set_solout(solout)

# Integrate until the full time
# (Also possible to take smaller sub-steps in a loop, as in the next section)
solver.integrate(t1)

# Form an array for easier manipulation
traj = np.asarray(traj)
make_plot(traj)


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
Hmm... This is clearly quite wrong!
It looks like although the function is quite powerful, maybe it needs a little help.
We can sugegst an initial value for the unknown parameter to see if that helps:
Can you write Python code for it?

# Curve fit function: fits fixedSinus to measured data over x
# Now we add the keyword argument p0, which specifies an initial guess
popt, pcov = curve_fit(fixed_sinus, x, y_measured, p0=1.5)

# Success?
print('*** Fitted value of wavelength was: ', popt, '\n')

plt.plot(x, y_measured, '*g')
plt.plot(x, fixed_sinus(x, popt), '--k')
plt.legend(['Experimental data', 'Fitted model'])


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
If we're interested in seeing approximately when the transition from a stable steady state to a limit cycle occurs, we might instead plot a number of trajectories, using different values of $B$ rather than different intial conditions.
_Aside: Numerical methods ('continuation') for finding the actual value of $B$ at which the transition occurs (the 'bifurcation point') are beyond the scope of this tutorial, but are discussed briefly at the end of the section. _
2. Using scipy.integrate.ode
In many cases we will actually want more control over the numerical integration process.
Helpfully, we have an alternative interface to various integration routines called simply ode (documentation here).
This is sold as a more object-oriented part of the module, but in fact it also provides more flexibility in terms of available solvers relative to odeint.
Can you write Python code for it?


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
It is also necessary to define the initial conditions ($X(t_0)$ and $Y(t_0)$), and the time points over which we will integrate:
Can you write Python code for it?

# Initial conditions
y0 = [0.5, 0.5]
# Time from 0 to 25
t = np.linspace(0, 25, 101)


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
It is quite obvious here that the sampling is dense in the regions where changes in $[X]$ are steep, and spreads out significantly in the regions where the gradient is small.
Closing Comments
Finally, in reality you will probably want to do some more in-depth experiments and analysis with your system (and your system may be somewhat more complex than those presented today).
As an alternative to coding up your own solutions using the building blocks presented today, you could consider using the PyDSTool package (its source code repository is here.
It provides quite a nice interface for modelling systems of ODEs, and additionally it does some quite clever stuff for you, saving time, effort, and testing, e.g.,
- it will compile your RHS functions in order to make simulations run faster
- integrators support adaptive timestepping
- automatic generation of Jacobians (and other numerical specifics) under the hood
- automatic generation of large ODE systems (e.g., networks of coupled systems)
and much more.
PyDSTool is partially integrated with the extremely powerful numerical continuation package AUTO which allows you to perform bifurcation/continuation analysis of your system directly from PyDSTool.
Somewhat tangentially, if you are eventually interested in numerical continuation you'll find that modern versions of AUTO include their own Python API which allows you to build your own analysis using the full feature set of AUTO.
Scipy.optimize
At some point, you may find that you need to fit a model to some data.
Here we refer to a model in quite a generic sense: this could be a regression or curve fit, the outcome of an ODE or PDE model, a complex black-box system such as a multi-scale finite element simulation of heat transfer in a nuclear reactor...
The important thing to consider is that the model receives a set of parameters (is parametrised) as input, and provides some data as output.
These simulated data may be directly comparable to experimentally acquired data, and in this case we may use an...
Can you write Python code for it?


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
Much better!
Obviously this is not great as in general you may not know a good starting parameter value (although you should have an idea of bounds and constraints especially if your model is not too abstract).
One way of getting around this is to initialise your fitting function from a bounded range of random initial values and choosing some that are markedly better.
Some optimisation routines will do (basically) this internally.
Another option is to choose a more robust optimisation method.
More general minimization
scipy.optimize contains a fairly wide range of optimization routines.
In general these aim to minimize some generic function, rather than to 'fit a curve' as we did a moment ago.
Therefore, in order to fit a model to some data using these methods, we must formulate an objective function.
This function can simply compute the distance, or error, between our desired outcome (the measured data) and the current output of the model.
Can you write Python code for it?

def objective_function(par):
    """Compute the sum of squares difference between the measured data and model output."""
    # The current model prediction, based on the current parameters' values
    y_model = sinus(x, *par)
    # Sum-of-squares difference between the model and 'measured' data
    return np.sum(np.power(y_measured - y_model, 2.0))


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
Not bad, considering it is more generic.
If we wanted further control over the process, we can also provide bounds as an optional keyword argument to the minimizer:
Can you write Python code for it?

# Here we specify that the parameters must all be positive
bopt = minimize(objective_function, np.random.rand(3), bounds=[(0, None)]*3)
print('Optimal parameters: ')
print('Amplitude: ', bopt.x[0])
print('Offset: ', bopt.x[1])
print('Frequency: ', bopt.x[2])

plt.plot(x, y_measured, '*g')
plt.plot(x, sinus(x, *bopt.x), '--k')
plt.legend(['Experimental data', 'Fitted model'])


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
Some Performance Improvements: Using the Jacobian
We can specify the Jacobian, if known, to help the solver along a little bit. This will generally make the solution faster, and potentially allow the solver to perform fewer steps by providing knowledge about the system's derivatives a priori.
More advanced packages, such as PyDSTool (mentioned below), may automatically compute a numerical or symbolic representation of the Jacobian in order to optimise simulation efficiency.
Can you write Python code for it?

def bruss_jac(t, y, a, b):
    """Jacobian function of the Brusselator."""
    j = np.empty((2, 2))
    j[0, 0] = 2*y[0]*y[1] - (b + 1)
    j[0, 1] = y[0]**2
    j[1, 0] = b - 2*y[0]*y[1]
    j[1, 1] = - y[0]**2
    return j

# Previous trajectory
old_trajectory = traj.copy()

# Define parameters as before
a = 1.0
b = 10.0

# Define initial conditions
y0 = [0.5, 0.5]
t0 = 0

# Final time
t1 = 100

# Define the ODE solver object
solver = ode(bruss_ode, jac=bruss_jac).set_integrator('vode', method='bdf')

# System/solver parameters are set as properties on the object
solver.set_initial_value(y0, t0).set_f_params(a, b).set_jac_params(a, b)

# We can't define a `solout` function, but we can automatically use
# the internal automatic step of the solver
traj = []
while solver.successful() and solver.t < t1:
    s = solver.integrate(t1, step = True)
    traj.append([solver.t, *s])

# Integrate until the full time
# (Also possible to take smaller sub-steps in a loop, as in the next section)
solver.integrate(t1)

traj = np.asarray(traj)
make_plot(traj)

print('Original trajectory had dimensions', old_trajectory.shape)
print('New trajectory (w/ Jacobian) has dimensions', traj.shape)

# Sampling density of original trajectory
plt.figure(figsize=(12, 4))
plt.hist(old_trajectory[:, 0], bins=50)
plt.xlabel('Time')
plt.ylabel('Sampling density')
plt.title('Sampling: no Jacobian')
plt.ylim([0, 1200])

# Sampling density of original trajectory
plt.figure(figsize=(12, 4))
plt.hist(traj[:, 0], bins=50)
plt.xlabel('Time')
plt.ylabel('Sampling density')
plt.title('Sampling: WITH Jacobian')
plt.ylim([0, 1200])


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
The basic form of the right hand side function is the same as before.
Can you write Python code for it?

# Solve with Euler
t0 = 0
t1 = 10
h = 0.5

y0 = 1.0

T, Y = euler_solve(t0, t1, h, y0, rhs_function)

# Solve with odeint
sol = odeint(rhs_function, y0, T)

tFine = np.linspace(T[0], T[-1], 1000)

plt.plot(T, Y, 'k-o')
plt.plot(T, sol, 'g-o')
plt.plot(t_fine, real_solution(y0, tFine), 'r--')
plt.xlabel('Time')
plt.legend(['Euler method', 'odeint', 'Exact solution'])


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
The scipy.optimize.minimize function provides a general interface for minimization.
Can you write Python code for it?

from scipy.optimize import minimize

# We attempt to minimize our objective
# This function now contains the full (3 parameter) sin function,
# and so we need to provide 3 starting points; we choose [0.5, 0.5, 0.5]
opt = minimize(objective_function, [0.5]*3)
print('Optimal parameters: ')
print('Amplitude: ', opt.x[0])
print('Offset: ', opt.x[1])
print('Frequency: ', opt.x[2])

plt.plot(x, y_measured, '*g')
plt.plot(x, sinus(x, *opt.x), '--k')
plt.legend(['Experimental data', 'Fitted model'])


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
The variable sol (first returned value) contains the solution over the desired time points.
Can you write Python code for it?

print('The number of time points is', len(t))
print('The solution matrix contains', sol.shape, 'entries')


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
To make our task a bit easier for this illustrative example we fix a couple of parameters and define a new function with these ($A = 1.0$ and $t = 0.0$), with only the frequency varying.
Can you write Python code for it?

def fixed_sinus(x, p):
    """Sinusoidal curve with all parameters fixed apart from frequency.
        p: frequency
    """
    A = 1.0
    t = 0.0
    return sinus(x, A, t, p)


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
Vode Solver
Can you write Python code for it?

# Define parameters as before
a = 1.0
b = 10

# Define initial conditions
y0 = [0.5, 0.5]
t0 = 0

# Final time
t1 = 100

# Define the ODE solver object
solver = ode(bruss_ode).set_integrator('vode', method='bdf')

# System/solver parameters are set as properties on the object
solver.set_initial_value(y0, t0).set_f_params(a, b)

# We can't define a `solout` function, but we can automatically use
# the internal automatic step of the solver
traj = []
while solver.successful() and solver.t < t1:
    s = solver.integrate(t1, step = True)
    traj.append([solver.t, *s])

# Integrate until the full time
# (Also possible to take smaller sub-steps in a loop, as in the next section)
solver.integrate(t1)

traj = np.asarray(traj)
make_plot(traj)


I am working on the file "6_Optimization_Integration_ODEs.ipynb".
The first task is:
⚠ Warning ⚠ the arguments of the right-hand-side function $f(t, y)$, that is passed to ode(), are reversed relative to odeint()!
I.e.,
-  odeint  expects a RHS function of the form func(y, t, ...)
-  ode  expects a RHS function of the form func(t, y, ...)
Consequently, we define a new function for the Brusselator, where these two arguments are now in the order expected by ode().
Can you write Python code for it?

def bruss_ode(t, y, a, b):
    """The brusselator system, defined for scipy.integrate.ode"""
    # Unpack both species
    X, Y = y
    # Derivatives
    dydt = [
        a + Y*(X**2) - (b + 1)*X, # d[X]/dt
        b*X - Y*(X**2),           # d[Y]/dt
    ]
    return dydt


I am working on the file "6_Ordinary_Differential_Equations.ipynb".
The first task is:
6.1 Initial Value Problem
Euler's Method
Can you write Python code for it?

def euler_method(f, a, b, y0, step=10):
    t = a
    w = y0
    ws = np.zeros(step + 1)
    ws[0] = y0
    h = (b - a) / step
    for i in range(step):
        w += h * f(t, w)
        t += h
        ws[i + 1] = w

    return w, ws


I am working on the file "6_Ordinary_Differential_Equations.ipynb".
The first task is:
6.4 Runge-Kutta Methods And Applications
Midpoint Method
$$
\begin{align}
w_0 &= y_0 \
w_{i+1} &= w_i + hf(t_i + \frac{h}{2},w_i + \frac{h}{2}f(t_i,w_i))
\end{align}
$$
Can you write Python code for it?

def midpoint_method(f, a, b, y0, step = 10):
    t = a
    w = y0
    ws = np.zeros(step + 1)
    ws[0] = y0
    h = (b - a) / step
    for i in range(step):
        w += h * f(t + h / 2, w + h / 2 * f(t, w))
        t += h
        ws[i + 1] = w

    return ws


I am working on the file "6_Ordinary_Differential_Equations.ipynb".
The first task is:
6.7 Multistep Methods
Adams-Bashforth Two-Step Method
$w_{i + 1} = w_i + h [\frac{3}{2}f(t_i, w_i) - \frac{1}{2}f(t_{i - 1}, w_{i - 1})]$
Can you write Python code for it?

def adams_bashforth(f, a, b, y0, step = 10):
    h = (b - a) / step
    w = y0
    w_n = explicit_trapezoid_method(f, a, b, y0)[0]
    t = a
    t_n = a + h
    for _ in range(step - 1):
        tmp_w_n = w_n
        w_n += h * (1.5 * f(t_n, w_n) - 0.5 * f(t, w))
        w = tmp_w_n
        t_n += h
        t += h

    return w_n


f = lambda t, y : t * y + np.power(t, 3)
ans = lambda t : - pow(t, 2) - 2 + 3 * math.exp(pow(t, 2) / 2)
print('%.15f' %ans(1) )
print('%.15f' %ode_rkf45(f, 0, 1, 1, tol=1e-13) )

f = lambda t, y : t * y + np.power(t, 3)
r = ode(f).set_integrator('dopri5')
r.set_initial_value(1, 0)
terminate = 0.9
dt = 0.1
while r.successful() and r.t <= terminate:
    print(r.t + dt, r.integrate(r.t + dt))


I am working on the file "6_Ordinary_Differential_Equations.ipynb".
The first task is:
Example
Apply Euler's Method to initial value problem
$
\begin{cases}
 & y' = ty + t^3\
 & y(0) = 1\
 & t\:in\:[0,1]
\end{cases}
$
Can you write Python code for it?

f = lambda t, y : t * y + np.power(t, 3)
w = euler_method(f, 0, 1, 1)
print(w[0])


I am working on the file "6_Ordinary_Differential_Equations.ipynb".
The first task is:
Example
Apply Runge-Kutta of order four to the initial value problem
$$
\left{\begin{matrix}\begin{align}
 & y' = ty + t^3 \
 & y(0) = 1
\end{align}\end{matrix}\right.
$$
Can you write Python code for it?

f = lambda t, y : t * y + np.power(t, 3)
ans = lambda t : - pow(t, 2) - 2 + 3 * math.exp(pow(t, 2) / 2)
w_data = runge_kutta_method(f, 0, 1, 1, step = 10)
print(' answer:%.15f' %ans(1) )
print('predict:%.15f' %w_data[-1] )


I am working on the file "6_Ordinary_Differential_Equations.ipynb".
The first task is:
Example
Apply strongly stable method, weakly stable method, and unstable method to the initial value problem
$$
\left{\begin{matrix}\begin{align}
 & y' = -3y \
 & y(0) = 1 \
 &t\:in\:[0,2]
\end{align}\end{matrix}\right.
$$
Can you write Python code for it?

f = lambda t, w : -3 * w
w = adams_bashforth(f, 0, 2, 1, step = 20)
print(w)


I am working on the file "6_Ordinary_Differential_Equations.ipynb".
The first task is:
Runge-Kutta Method of order four (RK4)
Can you write Python code for it?

def runge_kutta_method(f, a, b, y0, step = 10):
    t = a
    h = (b - a) / step
    w_data = np.zeros(step + 1)
    w = w_data[0] = y0
    for i in range(step):
        s1 = f(t, w)
        s2 = f(t + h / 2, w + s1 * h / 2)
        s3 = f(t + h / 2, w + s2 * h / 2)
        s4 = f(t + h, w + h * s3)
        t += h
        w += h / 6 * (s1 + 2 * s2 + 2 * s3 + s4)
        w_data[i + 1] = w

    return w_data


I am working on the file "6_Ordinary_Differential_Equations.ipynb".
The first task is:
Taylor Method for order k
$w_0 = y_0$
$w_{i+1} = w_i + hf(t_i,w_i) + \frac{h^2}{2}f'(t_i,w_i) + \cdots + \frac{h^k}{k!}f^{(k-1)}(t_i,w_i)$
6.3 Systems of ordinary differential equations
Example
Apply Euler's Method to the first-order system of two equations
$$
\left{\begin{matrix}\begin{align}
y_1' &= y_2^2 - 2y_1 \
y_2' &= y_1 - y_2 - ty_2^2 \
y_1(0) &= 0 \
y_2(0) &= 1
\end{align}\end{matrix}\right.
$$
Can you write Python code for it?

def euler_method_vec(f1, f2, a, b, y0, step=10):
    t = a
    ws1 = np.zeros(step + 1)
    ws2 = np.zeros(step + 1)
    ws1[0] = y0[0]
    ws2[0] = y0[1]
    h = (b - a) / step
    for i in range(step):
        ws1[i + 1] = ws1[i] + h * f1(ws1[i], ws2[i])
        ws2[i + 1] = ws2[i] + h * f2(t, ws1[i], ws2[i])
        t += h

    return ws1, ws2

f1 = lambda y1, y2 : np.power(y2, 2) - 2 * y1
f2 = lambda t, y1, y2 : y1 - y2 - t * np.power(y2, 2)
euler_method_vec(f1, f2, 0, 1, np.array([0, 1]), 10)


I am working on the file "6_Principal_Component_Analysis.ipynb".
The first task is:
Now that the data has been compressed, we can check the ratianed variance.
Can you write Python code for it?


I am working on the file "6_Principal_Component_Analysis.ipynb".
The first task is:
Now we can use a demo data set to show dimensionality reduction and data visualization.
We will use the Iris Data set as always.
Can you write Python code for it?

from sklearn.datasets import load_iris
iris = load_iris()

X = iris['data']
y = iris['target']

print X.shape


I am working on the file "6_Principal_Component_Analysis.ipynb".
The first task is:
We can find that the dimension of the original $X$ matrix is 4. We can then compress it to 2 using PCA technique with the PCA() class that we defined above.
Can you write Python code for it?

pca = PCA(num_components=2)
pca.fit(X)

X_reduce = pca.project(X)
print X_reduce.shape


I am working on the file "6_Principal_Component_Analysis.ipynb".
The first task is:
We have 97.76% of variance retained. This is okay for data visulization purposes. But if we used PCA in supervised learning pipelines, we might want to add more dimension to keep more than 99% of the variation from the original data.
Finally, with the compressed dimension, we can plot to see the distribution of iris dataset.
Can you write Python code for it?

%pylab inline
pylab.rcParams['figure.figsize'] = (10, 6)

from matplotlib import pyplot as plt

for c, marker, class_num in zip(['green', 'r', 'cyan'], ['o', '^', 's'], np.unique(y)):

    plt.scatter(x=X_reduce[:, 0][y == class_num], y=X_reduce[:, 1][y == class_num], c=c, marker=marker,
                label="Class {}".format(class_num), alpha=0.7, s=30)

plt.xlabel("Component 1")
plt.ylabel("Component 2")
plt.legend()

plt.show()


I am working on the file "6_VisualizationSchoolGrouping.ipynb".
The first task is:
Apply the filtering logic that we used in the main notebook.
Can you write Python code for it?

import math
print 'Total number of unique school districts: ' + str(len(np.unique(districts['Agency ID - NCES Assigned [District] Latest available year'])))
districts = districts[districts['Highest Grade Offered [District] 2009-10']=='12th Grade']
print 'Total number of school districts that have high schools: ' + str(len(districts))
districts = districts[districts['SURVYEAR']!='–']
print 'Total number of school districts that have a row on raw graduation data: ' + str(len(districts))
districts = districts[districts['AFGR']>=0]
print 'Total number of school districts with valid graduation data: ' + str(len(districts))

districts=districts.replace([np.inf, -np.inf], np.nan)


I am working on the file "6_VisualizationSchoolGrouping.ipynb".
The first task is:
Calculate the ratio of free and reduced lunch students to total students.
Can you write Python code for it?

districts['r_lunch_free_reduced'] = districts['Total Free and Reduced Lunch Students [Public School] 2009-10']/districts['Total Students [Public School] 2009-10']