I am working on the file "5_Imaging/5_2_sampling_functions_and_psfs.ipynb".
The first task is:
Row 1: Square sampling function (left) and the resulting PSF (right).
Row 2: Cross sampling function (left) and the resulting PSF (right).
Row 3: Set of circles sampling function (left) and the resulting PSF (right)
Row 4: Gaussian sampling function (left) and the resulting PSF (right)
Different shapes have a dramatic effect on the resulting PSF and hence the observed image. A square sampling function results in a strong primary lobe but underlying, repeated sidelobe structures. The cross sampling limits the samples to two orientations but at different spatial frequency scales, which results a strong cross pattern and high sidelobes. A collection of small circles at different positions results in a similar Airy disc of a single circle seen earlier. A Gaussian sampling function, which we will return to later in the chapter, has the nice feature that it's Fourier transform is another Gaussian. It is often desirable to taper, or modify, the sampling function to be more Gaussian-like. But, that is to come later.
Returning to the Hubble image which we started the section with we can create a sampling function which approximates the obscuration pattern of the mirror.
Can you write Python code for it?

#Approximate Hubble PSF
#http://www.stsci.edu/institute/org/telescopes/Reports/krist_tinytim_spie.pdf

#ring sampling
sampling = circularSamplingMap(512, 230, 70)

#mask circle sections
radius = 15
smallCirc = circularSamplingMap(2*radius+1, radius, 0)
sampling[60-radius:60+radius+1, 300-radius:300+radius+1] -= smallCirc
sampling[300-radius:300+radius+1, 60-radius:60+radius+1] -= smallCirc
sampling[400-radius:400+radius+1, 400-radius:400+radius+1] -= smallCirc

#mask diagonals
xpos, ypos = np.mgrid[0:512, 0:512]
idx = np.argwhere(np.abs(xpos-ypos) < 5)
sampling[idx[:,0], idx[:,1]] = 0.
sampling[idx[:,0], 511 - idx[:,1]] = 0.

fig, axes = plt.subplots(figsize=(16,8))
fig.tight_layout()

plt.subplot(1,2,1)
samplingPlot = plt.imshow(sampling)
plt.title('Hubble Sampling Function')
samplingPlot.set_cmap('gray')

plt.subplot(1,2,2)
psfPlot = plt.imshow(10. * np.log10(np.abs( np.fft.fftshift( np.fft.fft2(sampling)))))
plt.title('Hubble PSF (dB)')


I am working on the file "5_Imaging/5_2_sampling_functions_and_psfs.ipynb".
The first task is:
The most simple telescope we can imagine is a circular aperture which refracts light to a measurement plane, this is a refracting telescope in the style of Galileo's original design. The aperture, which has a certain circumference defines the maximum resolution of the telescope and the 2-D cross section of the aperture, a disk, is the 'sampling' function. That is, all light within the circumference of the lense is seen, and all light beyond the lense is excluded. The sampling function of this telescope is a binary disk.
Can you write Python code for it?

def circularSamplingMap(imgSize, outer, inner=0):
    """Return a circular sampling map of size [imgSize, imgSize]
    imgSize: image size in pixels
    outer: outer radius (in pixels) to exclude sampling above
    inner: inner radius (in pixels) to exclude sampling below"""
    zeros = np.zeros((imgSize,imgSize), dtype='float')
    ones = np.ones((imgSize,imgSize), dtype='float')
    xpos, ypos = np.mgrid[0:imgSize,0:imgSize]
    radius = np.sqrt((xpos - imgSize/2)**2. + (ypos - imgSize/2)**2.)
    sampling = np.where((outer >= radius) & (radius >= inner), ones, zeros)
    return sampling

sampling = circularSamplingMap(512, 20, 0)
fig, axes = plt.subplots(figsize=(8,8))
samplingPlot = plt.imshow(sampling)
plt.title('Sampling Function')
samplingPlot.set_cmap('gray')


I am working on the file "5_Imaging/5_2_sampling_functions_and_psfs.ipynb".
The first task is:
This ring sampling effectively acts as a bandpass filter which removes spatial frequency components that in the radius below 5 pixels and above 20 pixels. Creating another ring sampling function with a larger radius produces a different bandpass filter of the true image.
Can you write Python code for it?

fig, axes = plt.subplots(figsize=(16,16))
fig.tight_layout()

plt.subplot(2,2,1)
sampling = circularSamplingMap(512, 100, 30)
samplingPlot = plt.imshow(sampling)
samplingPlot.set_cmap('gray')
plt.title('Sampling Function')

plt.subplot(2,2,2)
psfPlot = plt.imshow(10. * np.log10(np.abs( np.fft.fftshift( np.fft.fft2(sampling)))))
plt.title('PSF')

plt.subplot(2,2,3)
samplingPlot = plt.imshow(np.abs(sampling * fftDuck))
plt.title('Sampled Visibilities')

plt.subplot(2,2,4)
obsDuck = np.abs(np.fft.ifft2(np.fft.fftshift(sampling * fftDuck)))
obsPlot = plt.imshow(obsDuck)
obsPlot.set_cmap('gray')
plt.title('Observed Image')


I am working on the file "5_Imaging/5_3_gridding_and_degridding.ipynb".
The first task is:
5.3 Gridding and Degridding for using the FFT
In the previous section several sampling functions were presented. There the sampling functions were already neatly discretized and displayed as images. Each image was a grid of pixels (all with the same size). Fourier inverting such regularly sampled data is done with a fast Fourier Transform (FFT) algorithm. This is called the "fast" Fourier Transform because it is computationally more efficient that the Direct Fourier Transform (DFT). To give an idea of how fast this algorithm is, if there are $N^2$ pixels in the image, the FFT takes roughly $2N^2\log(N)$ computational operations. In contrast, the complexity of the DFT also depends on the number of visibilities $M$ and takes $N^2M$ steps. Here $M\approx N^2$ and for each pixel we must take $M$ complex exponentiations and multiplications. Specifically, the DFT calculates the intensity of each pixel:
\begin{equation}
 I(l,m) = \sum_{k=0}^{M-1}V_k(u,v)e^{2\pi i (lu+mv)}\text{, }V_k\text{ are the M measurements taken by the telescope}
\end{equation}
From this it should be clear that as the number of baselines or observation time is increased, the FFT approach would be far less time-consuming than the the direct approach. Unfortunately radio interferometers don't take measurements at regular intervals, and thus an FFT cannot be used on the observation data directly. Instead the data has to be resampled onto a grid with points spaced at regular intervals before taking the FFT. This resampling process (called gridding) and its inverse (called degridding) is the topic of this section. The big idea here is that we do gridding and degridding because it enables us to create an image faster than if we did the DFT.
As you will see later some u,v space image deconvolution algorithms such as the Cotton-Schwab ➞ major-minor cycle algorithm require that sources in image space are reconverted back into the non-regular measurement space. Here an accurate degridding operation is required to...
Can you write Python code for it?


I am working on the file "5_Imaging/5_3_gridding_and_degridding.ipynb".
The first task is:
Figure: Each observed visibility is centered at some sampling coordinate
in continuous u,v space and is weighted with some function $C(u,v)$, which extends only to finite “full support” region as illustrated. The result is either binned in a regularly spaced grid when gridding or gathered from this grid when degridding. After all of the observed visibilities have been gridded an Inverse Fast Fourier Transform is performed to create an image of the sky. The reverse operations are done when simulating a set of visibility measurements from a model sky.
The value at each grid point, then, is a weighted accumulation of all the nearby visibilties. In one dimension this can be stated as (visually illustrated below):
\begin{equation}
 (\forall a \in {1,2,\dots,N}) \mathscr{V}(a\Delta{x}) = \sum_{\substack{
                                                                     i | x_i \geq a\Delta{x}-\text{half support}, \
                                                                     x_i \leq a\Delta{x}+\text{half support}}
                                                                 }{\mathscr{V}(x_i) \, C(a\Delta{x}-x_i)}
\end{equation}
Can you write Python code for it?

N = 30
dx = 1
a = np.arange(N)
M = 75
tap_pos = (N//2)*dx
conv_hsup = 5*dx
conv_x = np.linspace(-conv_hsup,conv_hsup,1000)
vis_x = np.sort(np.random.rand(M)*N*dx)
vis = 1.5+np.random.rand(M)*0.7 + 0.3
plt.figure(figsize=(13,5))
ax1 = plt.axes()
ax1.axes.get_yaxis().set_visible(False)
for x in a*dx:
    plt.plot([x,x],[-1.5,-1.0],'k')
plt.plot(vis_x,vis,'b.')
plt.plot(vis_x,np.ones([M])*-1.25,'g.')
plt.plot([vis_x[0],vis_x[0]],[-1.25,vis[0]],'b--')
plt.plot(tap_pos+conv_x,np.sinc(conv_x),'r')

plt.arrow(tap_pos-conv_hsup, -0.25, 0, 2.5+0.5, head_width=0.0, head_length=0.0, fc='m', ec='m')
plt.arrow(tap_pos+conv_hsup, -0.25, 0, 2.5+0.5, head_width=0.0, head_length=0.0, fc='m', ec='m')
plt.arrow(tap_pos-conv_hsup, -0.25, conv_hsup*2, 0, head_width=0.0, head_length=0.0, fc='m', ec='m')
plt.arrow(tap_pos-conv_hsup, 2.5+0.25, conv_hsup*2, 0, head_width=0.0, head_length=0.0, fc='m', ec='m')

plt.text(tap_pos+0.75, -0.70, "$\sum{\mathscr{V}(x_i)C(a\Delta{x}-x_i)}$", fontsize=11,color='m')
plt.text(tap_pos+conv_hsup+0.15, 0, "$C$", fontsize=13,color='r')
plt.text(0.5, 2.5, "$\mathscr{V}(x_i),x_i\in\mathbb{R}$", fontsize=16,color='b')
plt.arrow(tap_pos, -0.25, 0, -0.45, head_width=0.75, head_length=0.3, fc='m', ec='m')
plt.ylim(-1.75,3.0)
plt.xlim(-0.5*dx,N*dx)
plt.xlabel("Grid position ($a\Delta{x}$)",fontsize=15)
plt.show()


I am working on the file "5_Imaging/5_3_gridding_and_degridding.ipynb".
The first task is:
Figure: Here we have illustrated the gridding process in one dimension. Given a continuous visibility function sampled at some, non-regular, points in $x$ and a convolution function, $C$, the points stored at regular intervals (black bars) is approximately a convolution between the visibility function and the convolution function. The coordinates the visibility function is sampled at, is plotted with green dots in-between the regularly sampled grid positions.
The weighting function, $C$ can be any number of functions proposed in the literature. These include linear, Lagrange, sinc (including one of the many window functions), Gaussian, modified B-spline, etc.
You may have noticed that the interpolating function above is remarkably close to that of a discrete convolution. If the resampling was done on data that was regularly sampled and the convolution function evaluated at these regular discrete steps then the function would just the ordinary discrete convolution. However, the function as it stands is not quite a convolution by the strictest definition of the word. Gridding and degridding should be thought of as approximations to the discrete convolution. Nevertheless we will use the regular convolution notation in our discussion.
For those coming from a signal processing background it is useful to think of the convolutional gridding and degridding operations in terms of the ordinary upsampling and downsampling operations. In gridding, as with traditional upsampling, the space in-between samples are filled with zero values. The only difference is that with gridding the original measurements are not regularly-spaced, as would be the case with upsampling. Just as with upsampling it is then necessary to assign values to these new zero values in-between the measured values. With gridding the values are smeared out over the grid points within a some area of support.
This is a very important point. During the gridding process, because the uv plane is not fully sampled, many of the grid points are assigned a value...
Can you write Python code for it?


I am working on the file "5_Imaging/5_3_gridding_and_degridding.ipynb".
The first task is:
Figure: The magnitudes of the Fourier transforms of various functions. It is desirable that most of the energy of these functions fall within some central region and that the response drops off sharply at the edge of this central region
After Fourier transformation the effects of the convolution function on the image can be mitigated by point-wise dividing the image through by the Fourier transform of the convolution function, $c(l,m)$. This has the effect of flattening the response of the passband, by removing the tapering towards the edges of the image, but raises the amplitude of any aliased sources at the edge of the image.
In practice the proloid spheroidal functions are used in imaging programs such as lwimager, but the definition of these functions are beyond the scope of the introductory discussion here and the reader is referred to the work of Donald Rhodes, On the Spheroidal Functions ⤴ for a detailed discussion of their definition and proof of their aliasing reduction properties.
It is also worth noting that the convolution functions used in gridding and degridding need not be the same function. In degridding the focus is solidly on the accuracy of the predicted visibility. Here it can be advantageous to minimize the difference between a direct transformation approach and a Fast Fourier Transform approach with degridding, see for instance the discussion by Sze Tan, Aperture-synthesis mapping and parameter estimation ⤴ for further detail.
5.3.3 Example Simulator and Imager
We conclude this section with some sample code to illustrate prediction and imaging using resampling and the FFT. To start off let's set up a uv tracks as would be seen by the JVLA in D configuration. Recall that image resolution is given by the longest track in u and v and that the baseline is always measured in wavelengths. You can play around with the nyquist rate and image sizes. Notice how subnyquist sampling adversely affect...
Can you write Python code for it?

%run jvla_d_constants
RA = 0.0
DECLINATION = 90.0
global Nx, Ny, uvw, scaled_uv, model_sky, model_regular, max_uv, cell_size_l, cell_size_m
cellsize_slider=FloatSlider(min=0.25, #subnyquist
                            max=2.5,
                            value=1.0,
                            step=0.1,
                            continuous_update=False)
sampling_field = HBox([Label("Nyquist rate scaling factor (1x=critical sampling)"), cellsize_slider])
imsize_slider=FloatSlider(min=1,
                          max=3,
                          value=2.0,
                          step=0.1,
                          continuous_update=False)
imsize_field = HBox([Label("Image size scaling factor (1x=1deg)"), imsize_slider])

def interact_plot(key):
    global Nx, Ny, uvw, scaled_uv, model_sky, model_regular, max_uv, cell_size_l, cell_size_m
    #User defined scaling factors:
    clear_output(wait=True)
    sampling_tweek_factor = cellsize_slider.value
    im_size_tweek_factor = imsize_slider.value

    #Set up interferometer sampling pattern
    uvw = track_simulator.sim_uv(RA, DECLINATION, 1.5, 60/3600.0, ENU, ARRAY_LATITUDE)

    #Work out the required sampling rate (let's just do a square grid for simplicity)
    max_uv = np.max(np.abs(uvw[:,0:2]/CENTRE_CHANNEL))
    print "Maximum extent in uv: %f" % (max_uv)
    cell_size_l = cell_size_m = np.rad2deg((1 / (2 * max_uv)) / sampling_tweek_factor)
    print "Cell sizes in l,m: (%f,%f) arcsecs" % (cell_size_l*3600.0, cell_size_m*3600.0)

    #arbitrarily choose the field size to be 1.0 square degrees
    FIELD_SIZE = 1.0 * im_size_tweek_factor

    #Then work out the number of pixels required to create a field of this size, given the nyquist image resolution
    Nx = int(np.round(FIELD_SIZE / cell_size_l))
    Ny = int(np.round(FIELD_SIZE / cell_size_m))
    print "Image size in pixels to cover %.3f square degrees: (%d,%d)" % (FIELD_SIZE, Nx, Ny)

    #Setup a random model sky consisting of point sources:
    model_sky = np.zeros([Nx,Ny])
    for i in...


I am working on the file "5_Imaging/5_3_gridding_and_degridding.ipynb".
The first task is:
Figure: The simulated model sky (l,m space) and its fourier transform in the visibility space (u,v space). Overlayed on top are the uv tracks for JVLA D. In the bottom plots the scaled tracks are overlayed on the grid predicted from the model. As you can see if subnyquist rates are chosen angular resolution is lost because the long baselines fall outside the grid and must be discarded during imaging
To complete the prediction (also known as the "forward" step) the measurements are resampled onto the u,v tracks of the interferometer using the degridding algorithm discussed above. Measurements are gathered and weighted from the vacinity of each of the points along the sampling track in order to "predict" a value at the u,v coordinate.
Can you write Python code for it?

# %load convolutional_degridder.py
import numpy as np

def fft_degrid(model_image, uvw, ref_lda, Nx, Ny, convolution_filter):
    """
    Convolutional gridder (continuum)

    Keyword arguments:
    model_image --- Model image
    uvw --- interferometer's scaled uvw coordinates
            (Prerequisite: these uv points are already scaled by the similarity
            theorem, such that -N_x*Cell_l*0.5 <= theta_l <= N_x*Cell_l*0.5 and
            -N_y*Cell_m*0.5 <= theta_m <= N_y*Cell_m*0.5)
    ref_lda --- array of reference lambdas (size of vis channels)
    Nx,Ny --- size of image in pixels
    convolution_filter --- pre-instantiated AA_filter anti-aliasing
                           filter object
    """
    assert model_image.ndim == 3
    filter_index = \
        np.arange(-convolution_filter.half_sup,convolution_filter.half_sup+1)
    model_vis_regular = np.zeros(model_image.shape, dtype=np.complex64)
    for p in xrange(model_image.shape[0]):
        model_vis_regular[p, :, :] = \
            np.fft.fftshift(np.fft.fft2(np.fft.ifftshift(model_image[p, :, :])))
    vis = \
        np.zeros([uvw.shape[0],
                  ref_lda.shape[0],
                  model_image.shape[0]],
                 dtype=np.complex)

    for r in xrange(uvw.shape[0]):
        for c in xrange(vis.shape[1]):
            scaled_uv = uvw[r,:] / ref_lda[c]
            disc_u = int(np.round(scaled_uv[0]))
            disc_v = int(np.round(scaled_uv[1]))
            frac_u_offset = int((1 + convolution_filter.half_sup +
                                 (-scaled_uv[0] + disc_u)) *
                                convolution_filter.oversample)
            frac_v_offset = int((1 + convolution_filter.half_sup +
                                 (-scaled_uv[1] + disc_v)) *
                                convolution_filter.oversample)

            if (disc_v + Ny // 2 + convolution_filter.half_sup >= Ny or
                disc_u + Nx // 2 + convolution_filter.half_sup >= Nx or
                disc_v + Ny // 2 -...


I am working on the file "5_Imaging/5_3_gridding_and_degridding.ipynb".
The first task is:
Import section specific modules:
Can you write Python code for it?

from IPython.display import Image
import track_simulator
import AA_filter


I am working on the file "5_Imaging/5_3_gridding_and_degridding.ipynb".
The first task is:
Next comes a simplified imaging step using the FFT and gridding. The visibilities on the irregularly-spaced u,v tracks are resampled onto regular coordinates. This is done by weighting and smearing each measured visibility out onto the regular coordinates in the vacinity of its u,v coordinate. After resampling the inverse FFT is used to transform the measurements in the spatial frequency domain to those in the spacial domain, thereby approximately reconstructing the model sky we started with.
Can you write Python code for it?

# %load convolutional_gridder.py
import numpy as np

def grid_ifft(vis, uvw, ref_lda, Nx, Ny, convolution_filter):
    """
    Convolutional gridder (continuum)

    Keyword arguments:
    vis --- Visibilities as sampled by the interferometer
    uvw --- interferometer's scaled uvw coordinates
            (Prerequisite: these uv points are already scaled by the similarity
            theorem, such that -N_x*Cell_l*0.5 <= theta_l <= N_x*Cell_l*0.5 and
            -N_y*Cell_m*0.5 <= theta_m <= N_y*Cell_m*0.5)
    ref_lda --- array of reference lambdas (size of vis channels)
    Nx,Ny --- size of image in pixels
    convolution_filter --- pre-instantiated AA_filter anti-aliasing
                           filter object
    """
    assert vis.shape[1] == ref_lda.shape[0], (vis.shape[1], ref_lda.shape[0])
    filter_index = \
        np.arange(-convolution_filter.half_sup,convolution_filter.half_sup+1)
    # one grid for the resampled visibilities per correlation:
    measurement_regular = \
        np.zeros([vis.shape[2],Ny,Nx],dtype=np.complex)
    # for deconvolution the PSF should be 2x size of the image (see
    # Hogbom CLEAN for details), one grid for the sampling function:
    sampling_regular = \
        np.zeros([2*Ny,2*Nx],dtype=np.complex)
    for r in xrange(uvw.shape[0]):
        for c in xrange(vis.shape[1]):
            scaled_uv = uvw[r,:] / ref_lda[c]
            disc_u = int(np.round(scaled_uv[0]))
            disc_v = int(np.round(scaled_uv[1]))
            frac_u_offset = int((1 + convolution_filter.half_sup +
                                 (-scaled_uv[0] + disc_u)) *
                                convolution_filter.oversample)
            frac_v_offset = int((1 + convolution_filter.half_sup +
                                 (-scaled_uv[1] + disc_v)) *
                                convolution_filter.oversample)
            disc_u_psf = int(np.round(scaled_uv[0]*2))
            disc_v_psf = int(np.round(scaled_uv[1]*2))
            frac_u_offset_psf = int((1 +...


I am working on the file "5_Imaging/5_5_widefield_effect.ipynb".
The first task is:
5.5 The Break Down of the Small Angle Approximation and the W-Term
Up to this point we used a resampling step and the Fast Fourier Transform to move between the image and visibility domains. Recall that we used the following simplified Fourier relationship to justify this synthesis process:
\begin{equation}
 \begin{split}
     V(u,v) &= \int_\text{sky}{I(l,m)e^{-2\pi i/\lambda(\vec{b}\dot(\vec{s}-\vec{s}_0))}}dS\
            &= \int\int{I(l,m)e^{-2\pi i/\lambda(ul+vm+w(\sqrt{1-l^2-m^2}-1))}}\frac{dldm}{\sqrt{1-l^2-m^2}}\
            &\approx\int\int{I(l,m)e^{-2\pi i/\lambda(ul+vm)}}dldm\
 \end{split}
\end{equation}
The last approximation to the model is just a Fourier transform by definition and is the one used when we were imaging up to this point. However, the more accurate version that relates the measurement to the brightness distribution along the celestial sphere is not the classical Fourier transform. The approximation is only valid when $n - 1 = \sqrt{1-l^2-m^2} - 1 \ll 1$ (ie. images of small regions of the sky) and/or $w \approx 0$ (the array is coplanar). Here $(n-1)$ is the projection height difference between the planar approximation tangent to the celestial sphere and a source's true position on the sphere, see the illustration below.
<img src="figures/orthogonal_projection_difference.png" alt="Smiley face" width="512">
Figure: The direction cosines (here $l$ is plotted against $n$) lie along the unit celestial sphere. $n$ is given by $n=\sqrt{1-l^2-m^2}$. If the projection pole (tangent point of the image) is at the same point as the phase reference centre, $n_0 = 1$. The total error between the orthogonal (SIN) projection of the source onto the tangent image plane and the source position on the celestial sphere is given as $\epsilon=(n-n_0)=(\sqrt{1-l^2-m^2} - 1)$.
Under the assumptions of a narrow field of view and coplanar measurements it is valid to use the FFT to construct a planar approximation to the sky. This section discusses the problem of wide-field imaging using non-coplanar...
Can you write Python code for it?

NO_ANTENNA = 4
NO_BASELINES = NO_ANTENNA * (NO_ANTENNA - 1) / 2 + NO_ANTENNA
CENTRE_CHANNEL = 1e9 / 299792458 #Wavelength of 1 GHz
#Create a perfectly planar array with both a perfectly East-West baseline and 2 2D baselines
ENU_2d = np.array([[5,0,0],
                [-5,0,0],
                [10,0,0],
                [0,23,0]]);
ENU_ew = np.array([[5,0,0],
                 [-5,0,0],
                 [10,0,0],
                 [0,0,0]]);
ARRAY_LATITUDE = 30 #Equator->North
ARRAY_LONGITUDE = 0 #Greenwitch->East, prime -> local meridian

fig = plt.figure(figsize=(10, 5))
ax=fig.add_subplot(121)
ax.set_title("2D Array")
ax.plot(ENU_2d[:,0],ENU_2d[:,1],"ko")
ax.set_xlabel("East")
ax.set_ylabel("North")
ax.set_xlim(-30,30)
ax.set_ylim(-30,30)
ax=fig.add_subplot(122)
ax.set_title("East-west array")
ax.plot(ENU_ew[:,0],ENU_ew[:,1],"ko")
ax.set_xlabel("East")
ax.set_ylabel("North")
ax.set_xlim(-30,30)
ax.set_ylim(-30,30)
plt.show()


I am working on the file "5_Imaging/5_5_widefield_effect.ipynb".
The first task is:
Figure: $u,v,w$ tracks and their projections onto ($u,v,w=0$) for a 2D and east-west interferometer
When the measurement domain is sampled along a single plane, as is true for the east-west interferometer, then all $w$ can be written as the same linear combination of u and v: $w = \alpha{u}+\beta{v}$. Although this introduces a slight distortion of the u,v coordinates in the Fourier relationship between the sky and the measurements, the distorted relationship remains a valid two-dimensional Fourier transform. It can be stated as:
\begin{equation}
    \begin{split}
        V(u,v,w) &= \int\int{I(l,m)e^{-2\pi i/\lambda(ul' + vm')}\frac{dldm}{\sqrt{1-l^2-m^2}}}\
        l' &= l + \alpha(\sqrt{1-l^2-m^2} - 1)\
        m' &= m + \beta(\sqrt{1-l^2-m^2} - 1)\
    \end{split}
\end{equation}
5.5.2 Non-coplanar Sampling
The same can not be said for two-dimensional arrays. There is no fixed relationship between $w$ and $u,v$. Instead the relationship depends both on the time-variant zenithal and parallactic angles, and the $u,v$ coverage only remains co-planar for instantaneous observations, provided the array layout is approximately flat.
Neglecting the $w(n-1)$ term by synthesizing wide-field images with two-dimensional arrays, using a planar approximation, introduces a direction-dependent error in the measurement. This phase error depends on the height-difference between antennae, as is illustrated by the tilted interferometer below.
Can you write Python code for it?


I am working on the file "5_Imaging/5_5_widefield_effect.ipynb".
The first task is:
Figure: A quadrant of an image, not w-projection corrected, from a MeerKAT simulated observation, the phase centre is at the top right corner. Sources further from the phase centre have a larger amount of smearing compared to closer in.
Can you write Python code for it?

gc1 = aplpy.FITSFigure('../data/fits/wterm/MeerKAT_6h60s_dec-30_10MHz_10chans_uniform_n3000-image.fits')
cpx = gc1.pixel2world(256, 256)
gc1.recenter(cpx[0], cpx[1], radius=0.2)
gc1.show_colorscale(vmin=-0.2, vmax=1., cmap='viridis')
gc1.hide_axis_labels()
gc1.hide_tick_labels()
plt.title('MeerKAT Observation (W-Corrected)')
gc1.add_colorbar()


I am working on the file "5_Imaging/5_5_widefield_effect.ipynb".
The first task is:
Figure: A quadrant of an image, w-projection corrected, from a MeerKAT simulated observation, the phase centre is at the top right corner. Sources far from the phase centre remain point-like when the correction is accounted for as compared to the un-corrected image above.
5.3.3 Correcting Non-coplanar Baselines Effects
There are various ways the delay error introduced when discarding the $w(n-1)$ term during resampling and 2D Fast Fourier Transform can be corrected for, these include:
Full 3D transform: Similar to the 2D Direct Fourier Transform the Fourier transform can be computed for every element in a cube of $l,m,n$ values. The sky lies along the unit sphere within this cube. See Perley's discussion in <cite data-cite='taylor1999synthesis'>Synthesis Imaging in Radio Astronomy II</cite> ⤴ for a full derivation of this usually computationally and memory prohibitive technique.
Snapshot imaging: As alluded to earlier the visibility measurements taken during very short observations are co-planar, assuming the physical array lies on a flat plane. During each observation the $l,m$ coordnates are slightly distorted and the images have to be interpolated to the same coordinates before the images can be averaged into a single map of the sky.
Facet imaging: In facet imaging the goal is to drive the $(n-1)$ factor down to 0; satisfying the narrow-field assumption that makes the 2D Fourier inversion valid. There are a few ways in which the sky can be split into smaller images, but the classical faceting approach is to tile the celestial sphere with many small tangent images, approximating the sky by a polyhedron.
The algorithm behind tangent (polyhedron) facet imaging is simple to implement. First the sky is recentred at the image centres $l_i,m_i$ of each of the narrow-field facets, by phase rotating the measured visibilities. Each of the facet-images is then rotated to be tangent to the sky sphere. As the Fourier transform preserves rotations, the facets can be tilted by rotating the u,v coordinates of the...
Can you write Python code for it?


I am working on the file "5_Imaging/5_5_widefield_effect.ipynb".
The first task is:
Figure: As the two figures show the projection of the source vector onto the two baselines are different for the coplanar and tilted interferometers. The phase for signals taken by co-planar interferometer baselines along some line of sight, $\vec{s}$, is given as $\phi = \frac{-2\pi i}{\lambda}(ul + vm)$, as opposed to tilted baselines that measure this same phase as $\phi_\text{tilt} = \frac{-2\pi i}{\lambda}{[ul + vm + w(n-1)]}$. The signal propagation delay is worse on the longest baselines and along the direction of sources far away from the phase centre of the interferometer.
It is important to realize that this phase term is purely geometric in origin; inserting a delay to correct $\Delta{w}$ for non-coplanar measurements only serves to correct the error in a single line of sight. In other words only the phase centre of the interferometer is changed by such a correction.
The small angle approximation $\sqrt{1+x} \approx 1+ \frac{x}{2}$ gives some intuition on how this phase error effects the brightness of sources away from the phase centre. It can be shown that:
\begin{equation}
    V(u,v,w) \approx {\int\int{I(l,m)(e^{2\pi i/\lambda wl^2/2}e^{2\pi i /\lambda wm^2/2})e^{-2\pi i /\lambda(ul+vm)}\frac{dldm}{n}}}
\end{equation}
Since $w$ can be rewritten as a complex relationship of $u,v$ and time-variant elevation and azimuth angles we expect to see a time- and baseline-variant shift in source position. This relative position shift also grows roughly quadratically with the source offsets in l and m. The images below show how sources are smeared over large areas during long observations first on the data captured with the JVLA and then on a simulated MeerKAT observation.
Can you write Python code for it?


I am working on the file "5_Imaging/5_5_widefield_effect.ipynb".
The first task is:
Figure: ENU coordinates for two hypothetical flat arrays: a 2D array and an east-west array
The two-dimensional interferometer has two major advantages over its one-dimensional east-west counterpart:
1. Improved u,v coverage at lower declinations, as plotted below.
2. Recall that the interferometer response is maximum when the phase-reference centre is orthogonal to the baseline. At lower observation angles it is desirable to have baseline components that are not aligned from east-to-west.
Can you write Python code for it?

DECLINATION = 0
T_OBS = 12
T_INT = 1/60.0
uw_2hr_2d = track_simulator.sim_uv(0.0,DECLINATION,T_OBS,T_INT,ENU_2d,ARRAY_LATITUDE,False)/CENTRE_CHANNEL
uv_2hr_ew = track_simulator.sim_uv(0.0,DECLINATION,T_OBS,T_INT,ENU_ew,ARRAY_LATITUDE,False)/CENTRE_CHANNEL
fig = plt.figure(figsize=(10, 5))
ax=fig.add_subplot(121)
ax.set_title("2D Array")
ax.plot(uw_2hr_2d[:,0],uw_2hr_2d[:,1],'k.')
ax.set_xlabel("u")
ax.set_ylabel("v")
ax.set_xlim(-10,10)
ax.set_ylim(-10,10)
ax=fig.add_subplot(122)
ax.set_title("East-west Array")
ax.plot(uv_2hr_ew[:,0],uv_2hr_ew[:,1],'k.')
ax.set_xlabel("u")
ax.set_ylabel("v")
ax.set_xlim(-10,10)
ax.set_ylim(-10,10)
plt.show()


I am working on the file "5_Imaging/5_5_widefield_effect.ipynb".
The first task is:
Figure: Only phase steering the visibilities to new phase centres without tilting the u,v,w coordinates to
correspond to the new phase tracking centre significantly reduces the achievable field of view. Here instead
each facet is parallel to the original tangent plane. As the new centre is taken further away from the original
phase tracking centre the effective facet size must be shrunk down to achieve a comparable projection error
at the edge of the synthesized facets.
Can you write Python code for it?


I am working on the file "5_Imaging/5_5_widefield_effect.ipynb".
The first task is:
Figure: Uncorrected image of the 8 hour observation of the supernova reminant G55.7+3.4 on the JVLA in D-configuration. Notice the eliptical smearing around the point source.
Can you write Python code for it?


I am working on the file "5_Imaging/5_5_widefield_effect.ipynb".
The first task is:
Figure: W-projection image of the 8 hour observation of the supernova reminant G55.7+3.4 on the JVLA in D-configuration.
Can you write Python code for it?

gc1 = aplpy.FITSFigure('../data/fits/wterm/MeerKAT_6h60s_dec-30_10MHz_10chans_uniform_n3000_w0-image.fits')
cpx = gc1.pixel2world(256, 256)
gc1.recenter(cpx[0], cpx[1], radius=0.2)
gc1.show_colorscale(vmin=-0.2, vmax=1., cmap='viridis')
gc1.hide_axis_labels()
gc1.hide_tick_labels()
plt.title('MeerKAT Observation (Not Corrected)')
gc1.add_colorbar()


I am working on the file "5_Imaging/5_5_widefield_effect.ipynb".
The first task is:
Figure: u,v coverage at declinatin $\delta=0$ for both a 2D and east-west array
The one drawback to using these two-dimensional layouts is that the measurements taken over the duration of the observation do not remain co-planar, even though the array layout is perfectly flat. The uvw tracks and their projections are plotted in 3-space below to illustrate this. This is opposed to the tracks created by the east-west interferometer which all remain in the same plane parallel to the Earth's equator. Alos, if an observation is sufficiently short, called a snapshot observation, then the rotation of the Earth is short enough to approximate the measurements as coplanar.
Can you write Python code for it?

DECLINATION = 45
T_INT = 1/60.0
T_OBS = 12
uvw_2d = track_simulator.sim_uv(0.0,DECLINATION,T_OBS,T_INT,ENU_2d,ARRAY_LATITUDE,False)/CENTRE_CHANNEL
uvw_ew = track_simulator.sim_uv(0.0,DECLINATION,T_OBS,T_INT,ENU_ew,ARRAY_LATITUDE,False)/CENTRE_CHANNEL
fig=plt.figure(figsize=(10, 5))
ax=fig.add_subplot(121,projection='3d')
ax.set_title("2D Array")
ax.view_init(elev=10, azim=160)
ax.plot(uvw_2d[:,0],uvw_2d[:,1],uvw_2d[:,2],'k.')
ax.set_xlabel("u")
ax.set_ylabel("v")
ax.set_zlabel("w")
ax=fig.add_subplot(122,projection='3d')
ax.set_title("East-west array")
ax.view_init(elev=10, azim=160)
ax.plot(uvw_ew[:,0],uvw_ew[:,1],uvw_ew[:,2],'k.')
ax.set_xlabel("u")
ax.set_ylabel("v")
ax.set_zlabel("w")
plt.show()

fig = plt.figure(figsize=(10, 5))
ax=fig.add_subplot(121)
ax.set_title("2D Array")
ax.plot(uvw_2d[:,0],uvw_2d[:,1],'k.')
ax.set_xlabel("u")
ax.set_ylabel("v")
ax=fig.add_subplot(122)
ax.set_title("East-west array")
ax.plot(uvw_ew[:,0],uvw_ew[:,1],'k.')
ax.set_xlabel("u")
ax.set_ylabel("v")
plt.show()


I am working on the file "5_Imaging/5_5_widefield_effect.ipynb".
The first task is:
Figure: u,v coverage at declinatin $\delta=0$ for both a 2D and east-west array
The one drawback to using these two-dimensional layouts is that the measurements taken over the duration of the observation do not remain coplanar, even though the array layout is perfectly flat. The uvw tracks and their projections are plotted in 3-space below to illustrate this. This is opposed to the tracks created by the east-west interferometer which all remain in the same plane parallel to the Earth's equator. Alas, if an observation is sufficiently short, called a snapshot observation, then the rotation of the Earth is short enough to approximate the measurements of a two-dimensional interferometer as coplanar.
Can you write Python code for it?


I am working on the file "5_Imaging/5_5_widefield_effect.ipynb".
The first task is:
Import section specific modules:
Can you write Python code for it?

from IPython.display import Image
from mpl_toolkits.mplot3d import Axes3D
import track_simulator

from astropy.io import fits
import aplpy

#Disable astropy/aplpy logging
import logging
logger0 = logging.getLogger('astropy')
logger0.setLevel(logging.CRITICAL)
logger1 = logging.getLogger('aplpy')
logger1.setLevel(logging.CRITICAL)

HTML('../style/code_toggle.html')


I am working on the file "5_Numerical_Differentiation_And_Integration.ipynb".
The first task is:
5.1 Numerical Differentiation
Two-point forward-difference formula
$f'(x) = \frac{f(x+h) - f(x)}{h} - \frac{h}{2}f''(c)$
where $c$ is between $x$ and $x+h$
Example
Use the two-point forward-difference formula with $h = 0.1$ to approximate the derivative of $f(x) = 1/x$ at $x = 2$
Can you write Python code for it?

# Parameters
x = 2
h = 0.1

# Symbolic computation
sym_x = sym.Symbol('x')
sym_deri_x1 = sym.diff(1 / sym_x, sym_x)
sym_deri_x1_num = sym_deri_x1.subs(sym_x, x).evalf()

# Approximation
f = lambda x : 1 / x
deri_x1 = (f(x + h) - f(x)) / h

# Comparison
print('approximate = %f, real value = %f, backward error = %f' %(deri_x1, sym_deri_x1_num, abs(deri_x1 - sym_deri_x1_num)) )


I am working on the file "5_Numerical_Differentiation_And_Integration.ipynb".
The first task is:
5.2 Newton-Cotes Formulas For Numerical Integration
Trapezoid Rule
$\int_{x_0}^{x_1} f(x) dx = \frac{h}{2}(y_0 + y_1) - \frac{h^3}{12}f''(c)$
where $h = x_1 - x_0$ and $c$ is between $x_0$ and $x_1$
Simpson's Rule
$\int_{x_0}^{x_2} f(x) dx = \frac{h}{3}(y_0 + 4y_1 + y_2) - \frac{h^5}{90}f^{(iv)}(c)$
where $h = x_2 - x_1 = x_1 - x_0$ and $c$ is between $x_0$ and $x_2$
Example
Apply the Trapezoid Rule and Simpson's Rule to approximate $\int_{1}^{2} \ln(x) dx$ and find an upper bound for the error in your approximations
Can you write Python code for it?

# Apply Trapezoid Rule
trapz = scipy.integrate.trapz([np.log(1), np.log(2)], [1, 2])

# Evaluate the error term of Trapezoid Rule
sym_x = sym.Symbol('x')
expr = sym.diff(sym.log(sym_x), sym_x, 2)
trapz_err = abs(expr.subs(sym_x, 1).evalf() / 12)

# Print out results
print('Trapezoid rule : %f and upper bound error : %f' %(trapz, trapz_err) )

# Apply Simpson's Rule
area = scipy.integrate.simps([np.log(1), np.log(1.5), np.log(2)], [1, 1.5, 2])

# Evaluate the error term
sym_x = sym.Symbol('x')
expr = sym.diff(sym.log(sym_x), sym_x, 4)
simps_err = abs( pow(0.5, 5) / 90 * expr.subs(sym_x, 1).evalf() )

# Print out results
print('Simpson\'s rule : %f and upper bound error : %f' %(area, simps_err) )


I am working on the file "5_Numerical_Differentiation_And_Integration.ipynb".
The first task is:
5.3 Romberg Integration
Can you write Python code for it?

def romberg(f, a, b, step):
    R = np.zeros(step * step).reshape(step, step)
    R[0][0] = (b - a) * (f(a) + f(b)) / 2
    for j in range(1, step):
        h = (b - a) / pow(2, j)
        summ = 0
        for i in range(1, pow(2, j - 1) + 1):
            summ += h * f(a + (2 * i - 1) * h)
        R[j][0] = 0.5 * R[j - 1][0] + summ

        for k in range(1, j + 1):
            R[j][k] = ( pow(4, k) * R[j][k - 1] - R[j - 1][k - 1] ) / ( pow(4, k) - 1 )

    return R[step - 1][step - 1]


I am working on the file "5_Numerical_Differentiation_And_Integration.ipynb".
The first task is:
5.4 Adaptive Quadrature
Can you write Python code for it?

''' Use Trapezoid Rule '''

def adaptive_quadrature(f, a, b, tol):
    return adaptive_quadrature_recursively(f, a, b, tol, a, b, 0)

def adaptive_quadrature_recursively(f, a, b, tol, orig_a, orig_b, deep):
    c = (a + b) / 2
    S = lambda x, y : (y - x) * (f(x) + f(y)) / 2
    if abs( S(a, b) - S(a, c) - S(c, b) ) < 3 * tol * (b - a) / (orig_b - orig_a) or deep > 20 :
        return S(a, c) + S(c, b)
    else:
        return adaptive_quadrature_recursively(f, a, c, tol / 2, orig_a, orig_b, deep + 1) + adaptive_quadrature_recursively(f, c, b, tol / 2, orig_a, orig_b, deep + 1)

''' Use Simpon's Rule '''

def adaptive_quadrature(f, a, b, tol):
    return adaptive_quadrature_recursively(f, a, b, tol, a, b, 0)

def adaptive_quadrature_recursively(f, a, b, tol, orig_a, orig_b, deep):
    c = (a + b) / 2
    S = lambda x, y : (y - x) * ( f(x) + 4 * f((x + y) / 2) + f(y) ) / 6
    if abs( S(a, b) - S(a, c) - S(c, b) ) < 15 * tol  or deep > 20 :
        return S(a, c) + S(c, b)
    else:
        return adaptive_quadrature_recursively(f, a, c, tol / 2, orig_a, orig_b, deep + 1) + adaptive_quadrature_recursively(f, c, b, tol / 2, orig_a, orig_b, deep + 1)


I am working on the file "5_Numerical_Differentiation_And_Integration.ipynb".
The first task is:
Example
Apply Romberg Integration to approximate $\int_{1}^{2} \ln{x}dx$
Can you write Python code for it?

f = lambda x : np.log(x)
result = romberg(f, 1, 2, 4)
print('Romberg Integration : %f' %(result) )

f = lambda x : np.log(x)
result = scipy.integrate.romberg(f, 1, 2, show=True)
print('Romberg Integration : %f' %(result) )


I am working on the file "5_Numerical_Differentiation_And_Integration.ipynb".
The first task is:
Example
Use Adaptive Quadrature to approximate the integral $ \int_{-1}^{1} (1 + \sin{e^{3x}}) dx $
Can you write Python code for it?

f = lambda x : 1 + np.sin(np.exp(3 * x))
val = adaptive_quadrature(f, -1, 1, tol=1e-12)
print(val)


I am working on the file "5_Numerical_Differentiation_And_Integration.ipynb".
The first task is:
Midpoint Rule
$ \int_{x_0}^{x_1} f(x)dx = hf(\omega) + \frac{h^3}{24}f''(c) $
where $ h = (x_1 - x_0) $, $\omega$ is the midpoint $ x_0 + h / 2 $, and $c$ is between $x_0$ and $x_1$
Composite Midpoint Rule
$ \int_{a}^{b} f(x) dx = h \sum_{i=1}^{m}f(\omega_{i}) + \frac{(b - a)h^2}{24} f''(c) $
where $h = (b - a) / m$ and $c$ is between $a$ and $b$. The $\omega_{i}$ are the midpoints of the $m$ equal subintervals of $[a,b]$
Example
Approximate $\int_{0}^{1} \frac{\sin x}{x} dx$ by using the composite Midpoint Rule with $m = 10$ panels
Can you write Python code for it?

# Parameters
m = 10
h = (1 - 0) / m
f = lambda x : np.sin(x) / x
mids = np.arange(0 + h/2, 1, h)

# Apply composite midpoint rule
area = h * np.sum(f(mids))

# Error term
sym_x = sym.Symbol('x')
expr = sym.diff(sym.sin(sym_x) / sym_x, sym_x, 2)
mid_err = abs( (1 - 0) * pow(h, 2) / 24 * expr.subs(sym_x, 1).evalf() )

# Print out
print('Composite Midpoint Rule : %.8f, error = %.8f' %(area, mid_err) )


I am working on the file "5_Numerical_Differentiation_And_Integration.ipynb".
The first task is:
Three-point centered-difference formula for second derivative
$f''(x) = \frac{f(x - h) - 2f(x) + f(x + h)}{h^2} - \frac{h^2}{12}f^{(iv)}(c)$
for some $c$ between $x - h$ and $x + h$
Rounding error
Example
Approximate the derivative of $f(x) = e^x$ at $x = 0$
Can you write Python code for it?

# Parameters
f = lambda x : math.exp(x)
real_value = 1
h_msg = "$10^{-%d}$"
twp_deri_x1 = lambda x, h : ( f(x + h) - f(x) ) / h
thp_deri_x1 = lambda x, h : ( f(x + h) - f(x - h) ) / (2 * h)

data = [
    ["h",
     "$f'(x) \\approx \\frac{e^{x+h} - e^x}{h}$",
     "error",
     "$f'(x) \\approx \\frac{e^{x+h} - e^{x-h}}{2h}$",
     "error"],
]

for i in range(1,10):
    h = pow(10, -i)
    twp_deri_x1_value = twp_deri_x1(0, h)
    thp_deri_x1_value = thp_deri_x1(0, h)
    row = ["", "", "", "", ""]
    row[0] = h_msg %i
    row[1] = '%.14f' %twp_deri_x1_value
    row[2] = '%.14f' %abs(twp_deri_x1_value - real_value)
    row[3] = '%.14f' %thp_deri_x1_value
    row[4] = '%.14f' %abs(thp_deri_x1_value - real_value)
    data.append(row)

table = ply_ff.create_table(data)
plotly.offline.iplot(table, show_link=False)


I am working on the file "5_PDE/basic_finitediff.ipynb".
The first task is:
Luego debemos construir una matriz $A$ y un vector $b$ bajo esta nueva numeración tal que el sistema $Av=b$ sea resoluble y el resultado podamos trasladarlo de vuelta al sistema de $w_{ij}$. Esta matriz naturalmente será de tamaño $mn \times mn$ y cada punto de la grilla tendrá su propia ecuación, como uno podría pensar.
La entrada $A_{pq}$ corresponde al $q$-ésimo coeficiente lineal de la $p$-ésima ecuación del sistema $Av =b$. Por ejemplo la ecuación
$$\frac{w_{i-1,j} -2w_{ij} + w_{i+1,j}}{h^2} + \frac{w_{i,j-1}-2w_{ij}+ w_{i,j+1}}{k^2} = f(x_i, y_j)$$
Corresponde a la ecuación para el punto $(i,j)$ de la grilla, y será la ecuación $p = i + (j-1)m$. Entonces, la entrada $A_{pq}$ no es más que, dada la conversión definida:
\begin{align}
A_{i + (j-1)m, i + (j-1)m} &= -\frac{2}{h^2}- \frac{2}{k^2}\
A_{i + (j-1)m, i+1 + (j-1)m} &= \frac{1}{h^2}\
A_{i + (j-1)m, i-1 + (j-1)m} &= \frac{1}{h^2}\
A_{i + (j-1)m, i + jm} &= \frac{1}{k^2}\
A_{i + (j-1)m, i + (j-2)m} &= \frac{1}{k^2}\
\end{align}
Análogamente los $b$ del lado derecho del sistema son, naturalmente, la función dada en el punto $(x_i, y_j)$
$$b_{i+(j-1)m} =  f(x_i, y_j)$$
Como las condiciones de borde son conocidas estas ecuaciones excluyen dichos puntos, los índices $i,j$ van desde $1 < i < m$ y $1< j<n$. Las ecuaciones correspondientes a los bordes también consisten en evaluar las funciones dadas en los puntos $(x_i, y_j)$ e introducen $1$'s en la matriz $A$, mientras que para $b$ esto se traduce a colocar $g_s(z)$ donde corresponda utilizando la misma convención lineal (No vale la pena repasar estas ecuaciones).
Al final del día este sistema sigue siendo lineal, y con métodos básicos podemos resolverlos sin problemas!
Ejemplo
1) Aplique diferencias finitas para una grilla de 25 puntos (5 por lado) para aproximar la solución de la ecuación de Laplace en el rectángulo $[0,1]\times[1,2]$ dadas las siguientes condiciones de borde:
\begin{align}
u(x,1) &= \ln(x^2+1)\
u(x,2) &= \ln(x^2+4)\
u(0,y) &= 2\ln(y)\
u(1,y) &= \ln(y^2+1)
\end{align}
2) Verifique que...
Can you write Python code for it?

%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
from scipy import linalg
from mpl_toolkits.mplot3d import Axes3D

def f(x,y):
    return 0.0

# Condiciones de borde
def g1(x):
    return np.log(x**2 + 1)

def g2(x):
    return np.log(x**2 + 4)

def g3(y):
    return 2*np.log(y)

def g4(y):
    return np.log(y**2 + 1)

# Puntos de la grilla
m, n = 30, 30
# Precálculo de m*n
mn = m * n
# Cantidad de steps
M = m - 1
N = n - 1
# Limites del dominio, x_left, x_right, y_bottom, y_top
xl, xr = (0.0, 1.0)
yb, yt = (1.0, 2.0)
# Tamaño de stepsize por dimensión
h = (xr - xl) / float(M)
k = (yt - yb) / float(N)
# Precálculo de h**2 y k**2
h2 = h**2.0
k2 = k**2.0

# Generar arreglos para dimension...
x = [xl + (i - 1)*h for i in range(1, m+1)]
y = [yb + (i - 1)*k for i in range(1, n+1)]

A = np.zeros((mn, mn))
b = np.zeros((mn))

for i in range(1, m-1):
    for j in range(1, n-1):
        A[i+(j-1)*m, i-1+(j-1)*m] = 1.0/h2
        A[i+(j-1)*m, i+1+(j-1)*m] = 1.0/h2

        A[i+(j-1)*m, i+(j-1)*m] = -2.0/h2 -2.0/k2

        A[i+(j-1)*m, i+(j-2)*m] = 1.0/k2
        A[i+(j-1)*m, i+j*m] = 1.0/k2

        b[i+(j-1)*m] = f(x[i], y[j])

for i in range(0,m):
    j = 0
    A[i+(j-1)*m, i+(j-1)*m] = 1.0
    b[i+(j-1)*m] = g1(x[i])
    j = n-1
    A[i+(j-1)*m, i+(j-1)*m] = 1.0
    b[i+(j-1)*m] = g2(x[i])

for j in range(1, n-1):
    i = 0
    A[i+(j-1)*m, i+(j-1)*m] = 1.0
    b[i+(j-1)*m] = g3(y[j])
    i = m-1
    A[i+(j-1)*m, i+(j-1)*m] = 1.0
    b[i+(j-1)*m] = g4(y[j])

v = linalg.solve(A, b)
w = np.reshape(v, (m,n))

fig = plt.figure(figsize=(10,7))
ax = fig.add_subplot(111, projection='3d')
xv, yv = np.meshgrid(x, y)
ax.plot_surface(xv, yv, w, rstride=1, cstride=1)
plt.xlabel("x")
plt.ylabel("y")
plt.show()

fig = plt.figure(figsize=(10,7))
ax = fig.add_subplot(111, projection='3d')
xv, yv = np.meshgrid(x, y)
zv = np.log(xv**2 + yv**2)
ax.plot_surface(xv, yv, zv, rstride=1, cstride=1)
plt.show()


I am working on the file "5_PDE/basic_finitediff.ipynb".
The first task is:
Para simplificar el sistema a resolver, cambiaremos los índices dobles por indices lineales mediante la conversión
$$v_{i+(j-1)m} = w_{ij}$$
Se puede pensar también como una operación de stack donde las filas de la grilla son colocadas una al lado de otra en orden.
Can you write Python code for it?

plt.figure(figsize=(10,5))
plt.title("Mesh para coordenadas lineales")
for y in yy:
    plt.plot(xx, [y for _x in xx], 'co')
plt.xlim(xl-0.1, xr+0.1)
plt.ylim(yb-0.1, yt+0.1)
plt.xticks([xl, xr], ['$x_l$', '$x_r$'], fontsize=20)
plt.yticks([yb, yt], ['$y_b$', '$y_t$'], fontsize=20)
plt.text(xl, yb, "$v_{1}$", fontsize=20)
plt.text(xl+h, yb, "$v_{2}$", fontsize=20)
plt.text(xl, yb+k, "$v_{m+1}$", fontsize=20)
plt.text(xr, yb+k, "$v_{2m}$", fontsize=20)
plt.text(xl, yt, "$v_{(n-1)m+1}$", fontsize=20)
plt.text(xr, yt, "$v_{mn}$", fontsize=20)
plt.text(xr, yb, "$v_{m}$", fontsize=20)
plt.title("Mesh para coordenadas en dos dimensiones")
plt.show()
plt.show()


I am working on the file "5_PDE/schroedinger.ipynb".
The first task is:
1. Gráfico de Condición inicial
Can you write Python code for it?

# Initial conditions, outside for plotting.
f_re = lambda x: amp*np.exp(-(x-xc)**2.0/s)*np.cos(2.0*np.pi*(x-xc)/wl)
f_im = lambda x: amp*np.exp(-(x-xc)**2.0/s)*np.sin(2.0*np.pi*(x-xc)/wl)

# Center of wave
xc = 0
# Wavelength
wl = 4
# Exponential decay coeficient
s = 3
# Amplitude
amp = 0.5

xx = np.linspace(-3, 3, 400)
yy_re = f_re(xx)
yy_im = f_im(xx)

plt.figure(figsize=(12,6))
plt.ylim(-1, 1)
plt.plot(xx, yy_re, 'b-', lw=2.0, label=r"$\Re(\Psi_0)$")
plt.plot(xx, yy_im, 'r-', lw=2.0, label=r"$\Im(\Psi_0)$")
plt.grid(True)
plt.legend(loc='best')
plt.draw()


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
And now we can, for instance, plot surfaces
Can you write Python code for it?

X = np.linspace(-5, 5, 40)
Y = np.arange(-5, 5, 0.25)  # Defining an array of equi-spaced numbers
X, Y = np.meshgrid(X, Y)
R = np.sqrt(X**2 + Y**2)
Z = np.sin(R)

fig = plt.figure()
ax = fig.gca(projection='3d')

surf = ax.plot_surface(X, Y, Z, cmap=matplotlib.cm.coolwarm,
                       linewidth=0, antialiased=False)

ax.set_zlim(-1.01, 1.01)

fig.colorbar(surf, shrink=0.5)

plt.show()


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
Bokeh contains some tools to interact with the plots. These include zooming, selecting som subset of data, panning... etc. You can check all the tools available in the documentation
Can you write Python code for it?


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
Error bars can be added in both the $x$ and $y$ axes. This is achieved using the errorbar function
Can you write Python code for it?

xs = np.linspace(1, 5, 5)
ys = xs ** 2
xerr = 1 / np.sqrt(xs)
yerr = xs ** (1 / 3) + [0.2 for _ in xs]

fig, ax = plt.subplots()
ax.errorbar(xs, ys, xerr=xerr, yerr=yerr, ls='none', fmt='o')    # ls is for line style, fmt for format
ax.set_yscale('log')
plt.show()


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
Exercise 1. Create a three by three grid. Put the BIST logo in the center. Surround him with aesthetically pleasing functions in the remaining subplots. Hint: matplotlib.image.imread("http://bist.eu/wp-content/uploads/2016/05/BIST-7-centres-GRISOS.png") will get you the BIST logo, and the function imshow will plot it.
Can you write Python code for it?

# All the data has been loaded already, except for the image
bist = matplotlib.image.imread('http://bist.eu/wp-content/uploads/2016/05/BIST-7-centres-GRISOS.png')

# The actual plot
fig = plt.figure(figsize=(12, 12))
axes = [[],[],[]]

axes[0].append(fig.add_subplot(3, 3, 1))
axes[0][0].scatter(x, x ** 2 + 0.25 * np.random.randn(len(x)))
axes[0][0].set_title('Scatter')

axes[0].append(fig.add_subplot(3, 3, 2))
axes[0][1].step(x, y1, lw=2)
axes[0][1].set_title('Step')

axes[0].append(fig.add_subplot(3, 3, 3))
axes[0][2].bar(x, y1, align='center', width=0.5, alpha=0.5)
axes[0][2].set_title('Bar')

axes[1].append(fig.add_subplot(3, 3, 4))
axes[1][0].fill_between(x, y1, y2, color='green', alpha=0.5);
axes[1][0].set_title('Fill between');

axes[1].append(fig.add_subplot(3, 3, 5))
axes[1][1].imshow(bist)
axes[1][1].set_title('BIST')

axes[1].append(fig.add_subplot(3, 3, 6))
axes[1][2].hist(n, bins=100)
axes[1][2].set_title('Histogram')
axes[1][2].set_xlim((min(n), max(n)))

axes[2].append(fig.add_subplot(3, 3, 7, projection='3d'))
axes[2][0].plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=matplotlib.cm.hot, linewidth=0, antialiased=False)
axes[2][0].set_title('Surface')

axes[2].append(fig.add_subplot(3, 3, 8))
p = axes[2][1].pcolor(X / (2 * np.pi), Y / (2 * np.pi), Z, cmap=matplotlib.cm.RdBu, vmin=abs(Z).min(), vmax=abs(Z).max())
axes[2][1].set_title('Color map')
fig.colorbar(p, ax=axes[2][1])

axes[2].append(fig.add_subplot(3, 3, 9, polar=True))
axes[2][2].plot(t, t, color='blue', lw=3);
axes[2][2].set_title('Polar coordinates')

fig.tight_layout()
plt.show()


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
Exercise 3. Plot the marginal distributions of petal length versus sepal length for each of the three families of  flowers. First plot the three of them in the same plot, then do it on separate subplots. Note that when you are doing the three on the same plot, you will have to modify your axes length to see all the data.
Can you write Python code for it?

# First three together
plt.xlim(0.7, 7)
plt.ylim(3, 8.5)
sns.regplot(x='petal length (cm)', y='sepal length (cm)', data=iris.loc[iris['target']=='Iris-setosa'], label='Setosa')
sns.regplot(x='petal length (cm)', y='sepal length (cm)', data=iris.loc[iris['target']=='Iris-versicolor'], label='Versicolor')
sns.regplot(x='petal length (cm)', y='sepal length (cm)', data=iris.loc[iris['target']=='Iris-virginica'], label='Virginica')
plt.legend()
plt.show()

# Now separate
fig, ax = plt.subplots(ncols=3, figsize=(12,4))
for i, type_ in enumerate(np.unique(iris['target'])):
    sns.regplot(x='petal length (cm)', y='sepal length (cm)', data=iris.loc[iris['target']==type_], ax=ax[i])
    ax[i].set_title=(type_)

plt.tight_layout()
plt.show()


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
Interactive Plotting
Now we turn to review some packages very useful not only for visualizing data, but also interacting with it and obtaining individual information about the points studied. In particular we will review briefly the interactive capabilities of Matplotlib, after which we will move on to the dedicated package Bokeh.
Matplotlib
Matplotlib contains some interacting capabilities, although in general it is heavier to code and most of the features have to be coded by hand (and, as you will see, some of the features are not flawlessly implemented with Jupyter). Let us analyze an example where clicking on points give you the information about them.
Can you write Python code for it?

%matplotlib
fig = plt.figure()
ax = fig.add_subplot(111)
ax.set_title('Click on points')

line, = ax.plot(np.random.rand(100), 'o', picker=5)  # 5 points tolerance

def onpick(event):
    thisline = event.artist
    xdata = thisline.get_xdata()
    ydata = thisline.get_ydata()
    ind = event.ind
    points = tuple(zip(xdata[ind], ydata[ind]))
    print('onpick points:', points)

fig.canvas.mpl_connect('pick_event', onpick)

plt.show()


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
Let us define an array with all the names of the features and plot their correlations.
Can you write Python code for it?

features = iris.columns.values[:-1]
sns.pairplot(iris, vars=features, hue='target', size=3);


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
Matplotlib can also handle 3-dimensional plots. For that we need to import the 3-dimensional axes submodule
Can you write Python code for it?

from mpl_toolkits.mplot3d.axes3d import Axes3D


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
Notice the difference with this case:
Can you write Python code for it?

plt.plot(x, y1)
plt.xlabel('x')
plt.ylabel('y')
plt.show()
plt.xlabel('same x')
plt.plot(x, y2)
plt.show()


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
Plotting with Pandas can be as easy as calling the plot command of the dataset.
Can you write Python code for it?


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
The curve you see in the image is the kernel density estimation, the density function that better approximates the observed distribution
You might have noticed by now than Seaborn not only produces nicer plots, but also sometimes performs some data analysis on its own. Recall the jointplot a few steps above, where we were getting the Pearson's $r$ coefficient and the corresponding $p$ value. There is a special argument that you can pass to jointplot to plot a more complete analysis with the regression line, confidence intervals and kernel density estimation, the kind option:
Can you write Python code for it?

sns.jointplot(x='sepal length (cm)', y='petal width (cm)', data=iris, size=5, kind='reg');


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
We imported the module matplotlib.plot as plt, and we call a function of it called plot to plot the square function. You always plot discrete points: x is a numpy array containing five points as a linear approximation between zero and five. On closer inspection, the curve is not smooth: this is because five points are not enough for the illusion of smoothness. Let us add some more points, labels for the axes, and a title for the figure:
Can you write Python code for it?

x = np.linspace(0, 5, 100)
y = x**2
plt.plot(x, y)
plt.xlabel('x')
plt.ylabel('y')
plt.title('The most exciting function ever, full stop.');


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
We will however use seaborn for some still basic but nicer and more complete visualization
Can you write Python code for it?

sns.jointplot(x='sepal length (cm)', y='sepal width (cm)', data=iris, size=5);


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
You see it is relatively simple to create these interactive plots. However, Bokeh makes it everything much much simpler, and has a lot of interactive tools already built in.
Bokeh
Bokeh is a Python interactive visualization library. Not only it allows for creating aesthetically-pleasant plots (we already have Seaborn for that), but it also gives a lot of power to interact with the plot. You can zoom, pan, focus on a subset of data..., and obtain live information of each data point as you scroll the mouse over it. As we will see, one of the peculiarities of Bokeh is that it does not operate on the notebook, but every plot is opened in a browser tab. Indeed, Bokeh "translates" the plot into HTML code that can then be shared easily without losing the interactivity.
Let us begin with a simple example with randomly-generated points
Can you write Python code for it?

N = 4000
x = np.random.random(size=N) * 100
y = np.random.random(size=N) * 100
radii = np.random.random(size=N) * 1.5

# Colors for the points. Each color is written in RGB format #RRGGBB, where each of the intensities is specified by
# a 2-digit hexadecimal number. Each of these numbers is specified by a string %02x. Note that we keep the
# intensity of B constant, while R and G depend on the specific point
colors = ['#%02x%02x%02x' % (int(r), int(g), 150) for r, g in zip(50 + 2 * x, 30 + 2 * y)]


I am working on the file "5_Plotting_DataVis_Exploration.ipynb".
The first task is:
You should find the plot right in the directory of the notebook.
Object-oriented Paradigm
The image that you see displayed is composed of a hierarchical structure of components. On the top level, it is an instance of the Figure class. This is what plt.plot() creates for you, with all the other underlying structures within; this function is for your convenience to avoid dealing with classes if you want a simple plot. The structures in the hierarchy include the area where you draw, which is technically called the Axes class. You may have more than one Axes if you have subplots or embedded plots. Axes than have x and y axes, which in turn have a scale, ticks, labels, and so on. If you have a single Axes class instantiated, like in the examples below, you can access and change most parts of the hierarchy like you did above with the x and y labels and the figure title. If you want to do anything non-trivial, you have to compose the figure and its components yourself. The examples in this section are mainly from this tutorial. A more detailed version of the plot of the square function is:
Can you write Python code for it?

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(x, y)
ax.set_xlabel('x')
ax.set_ylabel('y')
plt.show()


I am working on the file "5_RecommenderSystems/Recommender systems - Part 1-students.ipynb".
The first task is:
Advance work: you can try to modify the function compute_Pearson_correlation() to force than a minimum number of common items have been rated (for instance, more than 4 items) to consider that two users can be similar. Check again the MAE and RMSE, have they been reduced? Is the method scalability improved?
Recommendation analysis
Once we have design a recommender system, we can analyze the movies which should be recommended for a given user. For this purpose, let's start loading the movies information and, then, analyze which movies should be recommended.
In the file 'u.item' we have the information associated to each item. So, let's start loading this file...
Can you write Python code for it?

moviesFilename = 'u.item'
f = open(moviesFilename)
rawMovies = f.readlines()
print rawMovies[:5]


I am working on the file "5_RecommenderSystems/Recommender systems - Part 1-students.ipynb".
The first task is:
Alternative Least Squares algorithm (optional exercise)
Finally, let's implement the ALS algorithm. As you, know this method tries to approximate the ratings matrix by factorizing it as the product of two matrices:
$$ R = X * Y $$
where $X$ describes properties of each user, and $Y$ describes properties of each item. These two matrices are known as latent factors, since they are a low-dimension representation of users and items.
These matrices are computed such that the error for the users/items pairs over the known ratings is minimized:
$$ \min_{X, Y} \sum_{(u, i) \in P} \left( r_{u,i} - X_u Y_i \right)^2 + \lambda \left( ||X||^2 + ||Y||^2  \right)
$$
where P is the set known rated items.
The ALS algorithm does this by first randomly selecting the latent factors matrices $X$ and $Y$ and then, given the values of $Y$, it optimizes the values of $X$ such that the error is minimized:
$$ \min_{X} \sum_{(u, i) \in P} \left( r_{u,i} - X_u  Y_i \right)^2 + \lambda ||X||^2
 $$
Then, by the computed users' factors ($X$), it optimizes the value of the item's matrix solving:
$$ \min_{Y} \sum_{(u, i) \in P} \left( r_{u,i} - X_u Y_i \right)^2 + \lambda ||Y||^2 $$
This alternation between which matrix to optimize is the reason for the "alternating" in the name.
To solve this optimization problem, either for users or items factors, you can use the ridge regression implementation of scikit-learn.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

from sklearn.linear_model import Ridge

def train_ALS(ratings, lambda_, n_factors):
    """ Compute the latent factors of the ALS algorithm
    Args:
        ratings: crs matrix with the ratings of all the users to the rated items
        lambda_: regularization parameter
        n_factors: number of latent factors
    Returns:
        X, Y: latent factor matrices of users and items
    """

    # Parameters
    n_iterations = 20

    # Get the number of users and items
    n_users, n_items = # <FILL IN>

    # Random initialization of latent factors
    np.random.seed(0)
    X = 5 * np.random.rand(n_users, n_factors)
    Y = 5 * np.random.rand(n_factors, n_items)

    # Define the classifier
    clf = Ridge(alpha=lambda_, fit_intercept=False, max_iter=100,tol=0.01)
    for ii in range(n_iterations):
        for u in range(n_users):
            # From ratings matrix get the rated items by user u (use toarray() method)
            # Use np.squeeze to remove useless dimensions of r_u
            r_u = # <FILL IN>
            # Let's create an index matrix indicating the positions where there is or there isn't a rating
            w_u = # <FILL IN>
            # Solve the optimization problem
            # Find X_u to minimize (w_u*(r_u-X[u,:]*Y)^2)
            clf.fit(Y.T, r_u.T,w_u.T)
            # Get the coefficients computed by the model and add it to the latent factor matrix
            X[u,:] = # <FILL IN>
        for i in range(n_items):
            # From ratings matrix get the rating corresponding to item i (use toarray() method)
            # Use np.squeeze to remove useless dimensions of r_i
            r_i = # <FILL IN>
            # Let's create an index matrix indicating the positions where there is or there isn't a rating
            w_i = # <FILL IN>
            # Solve the optimization problem
            #...


I am working on the file "5_RecommenderSystems/Recommender systems - Part 1-students.ipynb".
The first task is:
Baseline recommenders
In this section we are going to build two baseline approaches. In particular, let's consider:
1. Mean based baseline which predicts new ratings as the average value of the ratings given by this user to previous rated items.
2. Mode based baseline, similar the mean baseline, but using the most frequent value (mode) among the known ratings.
So, complete the following cells to implement the functions corresponding to these two approaches.
Hint: When you are working with a csr matrix R, you can get the i-th row of the matrix as  R[i,:]. And, if you need their non-zero values, you can use its attribute .data to extract them; i.e., R[i,:].data will return the non-zero values of the i-th of a crs matrix.
1. Mean based baseline
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

def mean_recomender(user_id, item_ids, ratings):
    """ Calculate the output of a mean based recommender
    Args:
        user_id: id of the user to compute its output
        item_ids: numpy array with the ids of the items whose rating has to be estimated
        ratings: crs matrix with the ratings of all the users to the rated items
    Returns:
        pred_items: numpy array of dimensions 1 x number of items with the computed predictions for each item.
        This prediction is computed as the mean value of the items which the user_id has already rated.
    """
    # Compute the number of items in the rating matrix
    n_items = #<FILL IN>
    # Select the values of the items rated by the user_id
    ratings_u = #<FILL IN>
    # Compute the mean value of the ratings_u
    if ratings_u.shape[0]>0:
        mean_rating = #<FILL IN>
    else: # Cold start problem (average rating)
        mean_rating = 3
    # Generate a numpy vector of dimensions 1 x n_items with all their values equal to mean_rating
    pred_items = #<FILL IN>
    return pred_items

# Testing function mean_recomender()
user_id = 20
item_ids = np.array([8, 0, 100])
pred_mean = mean_recomender(user_id, item_ids, rating_train)
print pred_mean

###########################################################
# TEST CELL
###########################################################
Test.assertEquals(pred_mean.shape, (1,3), 'incorrect result: pred_mean shape is incorrect')
Test.assertEquals(np.round(np.sum(pred_mean),2), 8.3, 'incorrect result: pred_mean values are incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 1-students.ipynb".
The first task is:
Creating training and test rating matrices
Now, to be able to train and evaluate the different methods, let's divide the rating matrix into two different matrix:
* one of them with the 75% of the ratings for training the different recommenders;
* other one, with the remaining 25%, for testing purposes.
Hint: you may need the following functions:
* sparse.find()
* np.random.permutation()
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

# From the reduced rating matrix extract the users, items and ratings. Use the sparse.find() method
[users, items, ratings] = #<FILL IN>

# Get the number of ratings
number_ratings = #<FILL IN>

# Compute the number of training ratings as the 75% of the total number of ratings
np.random.seed(0)
n_tr = #<FILL IN>

# Create a permuted range from 0 to the number of ratings
random_pos = #<FILL IN>

# Select the first n_tr positions of random_pos as the training positions,
# and the remaining random_pos indexes and the positions of the testing ratings
pos_tr = #<FILL IN>
pos_test = #<FILL IN>

# Build the training and testing rating matrices
# Create a coo_matrix and, then, convert it to csr format
# Note that the number of users and items has not changed
rating_train = #<FILL IN>
rating_test = #<FILL IN>

###########################################################
# TEST CELL
###########################################################
Test.assertEquals(n_tr, 75000, 'incorrect result: n_tr value is incorrect')
Test.assertEquals(np.sum(random_pos[:5]), 142525, 'incorrect result: random_pos values are incorrect')
Test.assertEquals(np.round(rating_train.mean(),2), 0.17, 'incorrect result: the values of the training rating matrix are incorrect')
Test.assertEquals(np.round(rating_test.mean(),2), 0.06, 'incorrect result: the values of the testing rating matrix are incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 1-students.ipynb".
The first task is:
Each line in the dataset is formatted as:
    >>   MovieID | Title | Date | url | Genres codification\n
Now, let's format each line to create a list of [MovieID, Title]. The remaining variables will be dropped because we do not need them for this exercise.
Can you write Python code for it?


I am working on the file "5_RecommenderSystems/Recommender systems - Part 1-students.ipynb".
The first task is:
Finally, let's transform the rating matrix into a   Compressed Sparse Row (CSR) format, since we will have to acces to the users values (read rows), so this format facilitates this type of access.
Can you write Python code for it?


I am working on the file "5_RecommenderSystems/Recommender systems - Part 1-students.ipynb".
The first task is:
Formatting the data
As you can checked, each line is formatted as:
UserID \t MovieID \t Rating \t Timestamp \n
So, let's convert each line to a list with the fields [UserID, MovieID, Rating] (we drop the timestamp because we do not need it for this exercise). Note that the result of this conversion will provide a list of lists.
Tip: Check the Python function split() to convert each line into a list of items splitted by a given character.
Can you write Python code for it?


I am working on the file "5_RecommenderSystems/Recommender systems - Part 1-students.ipynb".
The first task is:
Now, let's evaluate the performance of the mean and mode based baselines.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

# Compute the number of users and items
n_users, n_items = # <FILL IN>
# Create two empty prediction matrix in crs format
pred_mean_ratings = sparse.lil_matrix((n_users, n_items))
pred_mode_ratings = sparse.lil_matrix((n_users, n_items))

# Work user by user
for u in range(n_users):
    # Get, form the test matrix, the item id to be predicted for this user (check .indices attribute of crs matrix)
    item_ids = # <FILL IN>
    # Get predictions with the mean based baseline for user u
    pred_mean_u = # <FILL IN>
    # Get predictions with the mode based baseline for user u
    pred_mode_u = # <FILL IN>
    # Build the prediction matrices
    pred_mean_ratings[u,item_ids] = # <FILL IN>
    pred_mode_ratings[u,item_ids] = # <FILL IN>

#Compute the error (MAE and RMSE) for each baseline method over the test_users
MAE_mean = # <FILL IN>
RMSE_mean = # <FILL IN>

MAE_mode = # <FILL IN>
RMSE_mode = # <FILL IN>

print 'Mean model ... MAE: %2.2f , RMSE: %2.2f ' % (MAE_mean, RMSE_mean)
print 'Mode model ... MAE: %2.2f , RMSE: %2.2f ' % (MAE_mode, RMSE_mode)

###########################################################
# TEST CELL
###########################################################

Test.assertEquals(np.round(MAE_mean,2), 0.84, 'incorrect result: MAE value of mean recommeder is incorrect')
Test.assertEquals(np.round(RMSE_mean,2), 1.04, 'incorrect result: RMSE value of mean recommeder is incorrect')
Test.assertEquals(np.round(MAE_mode,2), 0.86, 'incorrect result: MAE value of mode recommeder is incorrect')
Test.assertEquals(np.round(RMSE_mode,2), 1.19, 'incorrect result: RMSE value of mode recommeder is incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 1-students.ipynb".
The first task is:
Now, let's evaluate the performance of this recommender over all the users
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

print 'Please, be patient, this computation takes a while... '

# Compute the number of users and items
n_users, n_items = # <FILL IN>
# Create an empty prediction matrix in crs format
pred_ratings = sparse.lil_matrix((n_users, n_items))

# Work user by user
for u in test_users:
    # Get, form the test matrix, the item id to be predicted for this user (check .indices attribute of crs matrix)
    item_ids = # <FILL IN>
    # Get predictions with the used based CF method for user u
    pred_u = # <FILL IN>
    # Build the prediction matrix
    pred_ratings[u,item_ids] = # <FILL IN>

# Compute the error (MAE and RMSE) over test_users
MAE = # <FILL IN>
RMSE = # <FILL IN>

print 'MAE: %2.2f , RMSE: %2.2f ' %(MAE, RMSE)

###########################################################
# TEST CELL
###########################################################

Test.assertEquals(np.round(MAE,2), 0.82, 'incorrect result: MAE value is incorrect')
Test.assertEquals(np.round(RMSE,2), 1.02, 'incorrect result: RMSE value is incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 1-students.ipynb".
The first task is:
Performance evaluation
To evaluate the performance of the different recommenders, we are going to use two measurements:
* The Mean Average Error:
$$MAE  = \frac{1}{N} \sum_{i=1}^N |p_i -r_i|$$
* The Root Mean Square Error:
$$ RMSE = \sqrt{\frac{1}{N} \sum_{i=1}^N (p_i -r_i)^2}$$
Thus, complete the following cells two build two functions able to compute the MAE and RMSE values.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

def get_MAE(pred_rating, real_rating):
    """ Calculate the MAE
    Args:
        pred_rating: crs matrix, with dimensions n_users x n_items, with the predicted ratings.
        real_rating: crs matrix, with dimensions n_users x n_items, with the real ratings.
    Returns:
        MAE: Mean Absolute Error computed over the non-zero entries of real_rating.
    """

    # Extract the non-zero positions of real_rating and their values (use sparse.find() method)
    [pos_users, pos_items, real_values] = # <FILL IN>
    # Extract the predicted values of the non-zero positions
    pred_values = # <FILL IN>
    # Compute the MAE (check np.absolute method)
    MAE = # <FILL IN>
    return MAE

###########################################################
# TEST CELL
###########################################################

matrix_1 = sparse.eye(10).tocsr()
matrix_2 = (1.2*sparse.eye(10)).tocsr()
matrix_2[0,0]= 0.4
Test.assertEquals(np.round(get_MAE(matrix_1, matrix_2),2), 0.24, 'incorrect result: MAE value is incorrect')

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

def get_RMSE(pred_rating, real_rating):
    """ Calculate the RMSE
    Args:
        pred_rating: crs matrix, with dimensions n_users x n_items, with the predicted ratings.
        real_rating: crs matrix, with dimensions n_users x n_items, with the real ratings.
    Returns:
        RMSE: Root Mean Square Error computed over the non-zero entries of real_rating.
    """

    # Extract the non-zero positions of real_rating and their values (use sparse.find() method)
    [pos_users, pos_items, real_values] = # <FILL IN>
    # Extract the predicted values of the non-zero positions
    pred_values = # <FILL IN>
    # Compute the RMSE (check np.sqrt and...


I am working on the file "5_RecommenderSystems/Recommender systems - Part 1-students.ipynb".
The first task is:
To speed up the evaluations of the recommender systems, let's create a reduced set with 50 testing users. Of course, you can later extend the method evaluation to all the users, although the evaluation of some methods can take several minutes.
Can you write Python code for it?

np.random.seed(0)
all_users = np.random.permutation(number_users)
test_users = all_users[:50]


I am working on the file "5_RecommenderSystems/Recommender systems - Part 1-students.ipynb".
The first task is:
User based recommendations
In this section let's build a user-based collaborative filter by finding users who are similar to each other and user their rated items to predict the ratings over item which haven't been rated by other users.
The general algorithm will be (in pseudocode) as follows:
1. For each user, rank every other user in terms of closeness. For this step, we will use the rating matrix and use correlation as the measure of closeness.
2. Then, to estimate the ratings of unrated items for a given user, we will average the ratings of the neighbors of this user (using the correlation measurement computed in Step 1).
Step 1: Finding similar users
The similarity between different users is determined by comparing each user with every other user and computing a similarity score. Here, we are going to use the Pearson correlation coefficient:
$$ sim(user_a, user_b) = \frac{\sum_{p \in P} (r_{a,p} -\bar{r}a)(r{b,p} -\bar{r}b)}
{\sqrt{ \sum{p \in P} (r_{a,p} -\bar{r}a)^2} \sqrt{ \sum{p \in P} (r_{b,p} -\bar{r}_b)^2}}$$
where $P$ is set of items rated for both users a and b, $r_{u,p}$ is the rating of the user u to item p, and $\bar{r}_u$ is the mean value of the all the ratings of the user u.
Complete the next code so that it is able to obtain the correlation coefficient between to given users.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

def compute_Pearson_correlation(ratings, id_u1, id_u2):
    """ Calculate correlation coefficient
    Args:
        ratings: crs matrix, with dimensions n_users x n_items, with the ratings used to measure similarities.
        id_u1: id user 1
        id_u2: id user 2
    Returns:
        corr_value: correlation coefficient
    """
    # Get the indexes and values of the items rated by user 1 (use sparse.find() function)
    [pos_u1, items_u1, values_u1] = # <FILL IN>
    # Get the indexes and values of the items rated by user 2 (use sparse.find() function)
    [pos_u2, items_u2, values_u2] = # <FILL IN>
    # Get the set of items rated by both users (you can use np.intersect1d() method)
    items_intersect = # <FILL IN>

    if items_intersect is not None: # If the are common rated items...
        # Compute the mean values of all the items rated by user 1 and user 2
        m_1 = # <FILL IN>
        m_2 = # <FILL IN>
        # Get the ratings of users 1 and 2 in items_intersect (you can use .toarray() method)
        r_u1 = # <FILL IN>
        r_u2 = # <FILL IN>
        # Remove their means
        r_u1 = # <FILL IN>
        r_u2 = # <FILL IN>
        # Compute the correlation coefficient
        corr_value = # <FILL IN>
        # Remove useless dimensions
        corr_value =np.squeeze(corr_value)

    else: # Else correlation is 0
        corr_value = 0

    # Checking that the correlation is not NaN (this would happen if the denominatior is 0),
    # in this case, set the corrlation coefficient to 0
    if math.isnan(corr_value):
        corr_value = 0

    return corr_value

###########################################################
# TEST CELL
###########################################################

Test.assertEquals(np.round(compute_Pearson_correlation(rating_train, 5, 12),2), 0.36, 'incorrect...


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
2. Compute the average rating of each user
For the next step, let's use the getAverages( ) function to compute the average rating of all the users in a distributed way.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################
# From ratingsRDD with tuples of (UserID, MovieID, Rating) create an RDD with tuples of
# the (UserID, Rating), i.e, remove the MovieID field.
RDD_users_ratings = trainingRDD.# FILL IN

# From the RDD of (UserID, Rating) create an RDD with tuples of
# (UserID, iterable of Ratings for that UserID), where iterable of Ratings for that UserID has
# all the rated items of UserID. Review groupByKey() method of RDD elements.
RDD_users_allratings = RDD_users_ratings.# FILL IN

# Using getAverages(), compute the average rating of each user.
RDD_users_mean  = RDD_users_allratings.# FILL IN

###########################################################
# TEST CELL
###########################################################
Test.assertEquals(RDD_users_ratings.first(), (186, 3.0), 'incorrect result: RDD_users_ratings is incorrect')
Test.assertEquals(list(RDD_users_allratings.first()[1])[:5], [4.0, 5.0, 4.0, 3.0, 3.0], 'incorrect result: RDD_users_allratings is incorrect')
Test.assertEquals(np.round(RDD_users_mean.first()[1],2), 3.69, 'incorrect result: RDD_users_mean is incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
2. Create a list of ratings
Here, you should create a new RDD with one element for each item, where each element is given by (item_id, list_ratings), where list of ratings has a set of tuples (user_id, rating) with  the id of the user who has rated item_id and the assigned rating.
Besides, the ratings of the list have to be normalized (subtracting its corresponding user average rating).
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

# 2.1. Create an RDD with training ratings subtracting the users' mean
# Create an RDD with elements (user, (item, rating))
trainingRDD_aux = # FILL IN

# Combine it with the users_mean -> (user, ((item, rating), user_mean))
# Hint: leftOuterJoin()
trainingRDD_mean = # FILL IN
# Create a new RDD subtracting the mean of each rating and reorganize it -> (user, item, rating_norm)
trainingRDD_norm = # FILL IN

# 2.2. Create an RDD with normalized training ratings with the form (item, list((user, rating)))
RDD_tratings_item = # FILL IN

###########################################################
# TEST CELL
###########################################################

id_item = 22
ratings_item = RDD_ratings_item.filter(lambda x : x[0]==id_item).first()
ratings_check = sc.parallelize(list(ratings_item[1]))

Test.assertEquals(np.round(ratings_check.filter(lambda x: x[0] == 608).first()[1],2), 0.26, 'incorrect result: rating value is incorrect')
Test.assertEquals(np.round(ratings_check.filter(lambda x: x[0] == 184).first()[1],2), -0.66, 'incorrect result: rating value is incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
2. Format your data
Convert RDD rawRatings into a new RRD where each line has been transformed with the function format_ratings().
Can you write Python code for it?


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
3. Combine previous RDDs
Until now, we have these RDDs:
* RDD_sim_users
* RDD_users_mean
* RDD_ratings_item
To make the predictions over the test data we need for each pair (userID, itemID) of the test ratings, we need to build an element containing:
- the average rating of userID
- the similar users of userID
- the list of rating of itemID
so that we can call the compute_predictions function with the corresponding input parameters.
Then, here, we are going to combine the above RDD to create a new RDD with elements given by:
((userID, itemID), average_rating_userID, list_similar_users_to_userID, list_ratings_itemID)
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

# 3.1 Create an input RDD, testForPredictingRDD, consisting of (UserID, MovieID) pairs
# that you extract from testRDD (i.e., remove the field rating)
RDD_test_ids = # FILL IN

# 3.2 Combine RDD_test_ids with RDD_users_mean to create an RDD (user, (item, mean_user))
# Hint: leftOuterJoin()
RDD_test_ids_mean = # FILL IN

# 3.3 Combine RDD_test_ids_mean with RDD_sim_users to create an RDD with elements
# (user, ((item, mean_user), list_sim_user)). Hint: leftOuterJoin()
# Next, reformat it to obtain elements (item, (user, mean_user, list_sim_user))
RDD_test_ids_sim = # FILL IN

# 3.4 Combine RDD_test_ids_sim with RDD_ratings_item to create an RDD with elements
# (item, ((user, mean_user , list_sim_user), list_item_rating)). Hint: leftOuterJoin()
# Next, reformat it to obatian elements ((user, item), mean_user, list_sim_user, list_item_rating)
RDD_test_ids_sim_rat = # FILL IN

###########################################################
# TEST CELL
###########################################################

check_out = RDD_test_ids_sim_rat.filter(lambda x: x[0]==(218, 516)).first()
Test.assertEquals(np.round(check_out[1],2), 3.62, 'incorrect result: mean value of the RDD is incorrect')

sim_check = sc.parallelize(list(check_out[2]))
Test.assertEquals(np.round(sim_check.filter(lambda x: x[0] == 24).first()[1],2), 0.31, 'incorrect result: similarity value is incorrect')

rating_check = sc.parallelize(list(check_out[3]))
Test.assertEquals(np.round(rating_check.filter(lambda x: x[0] == 308).first()[1],2), 0.23, 'incorrect result: rating value is incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
4. Performance evaluation
Finally, let's evaluate the goodness of the computed predictions over the test data. To evaluate it, we are going to use two measurements:
* The Mean Average Error:
$$MAE  = \frac{1}{N} \sum_{i=1}^N |p_i -r_i|$$
* The Root Mean Square Error:
$$ RMSE = \sqrt{\frac{1}{N} \sum_{i=1}^N (p_i -r_i)^2}$$
The next cell contains a function that given two RDDs, the first with the predicted ratings and the second with the real rating values, is able to compute the RMSE value. Use it as example to create a new function able to calculate the MAE value.
Can you write Python code for it?

def get_RMSE(predictedRDD, actualRDD):
    """ Compute the root mean squared error between two RDD with the predicted and actual ratings
    Args:
        predictedRDD: predicted ratings for each movie and each user where each entry is in the form
                      (UserID, MovieID, Rating)
        actualRDD: actual ratings where each entry is in the form (UserID, MovieID, Rating)
    Returns:
        RSME (float): computed RSME value
    """
    # Transform predictedRDD into the tuples of the form ((UserID, MovieID), Rating)
    predictedReformattedRDD = predictedRDD.map(lambda x: ((x[0],x[1]),x[2]))

    # Transform actualRDD into the tuples of the form ((UserID, MovieID), Rating)
    actualReformattedRDD = actualRDD.map(lambda x: ((x[0],x[1]),x[2]))

    # Compute the squared error for each matching entry (i.e., the same (User ID, Movie ID) in each
    # RDD) in the reformatted RDDs using RDD transformtions - do not use collect()
    squaredErrorsRDD = (predictedReformattedRDD.join(actualReformattedRDD).map(lambda x: pow(x[1][0]-x[1][1],2)))

    # Compute the total squared error - do not use collect()
    totalError = squaredErrorsRDD.reduce(lambda a,b: a+b)

    # Count the number of entries for which you computed the total squared error
    numRatings = squaredErrorsRDD.count()

    # Using the total squared error and the number of entries, compute the RSME
    return math.sqrt(float(totalError)/numRatings )

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

# Create a function to compute the MAE error
def get_MAE(predictedRDD, actualRDD):
    """ Compute the mean absolute error between predicted and actual
    Args:
        predictedRDD: predicted ratings for each movie and each user where each entry is in the form
                      (UserID, MovieID, Rating)
        actualRDD: actual ratings where each entry is in the form (UserID, MovieID, Rating)
    Returns:
        MAE...


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
5. Evaluate performance
Can you write Python code for it?

# Compute the error MAE
MAE = get_MAE(RDD_predictions, testRDD)
# Compute the error RMSE
RMSE = get_RMSE(RDD_predictions, testRDD)

print 'User based model ... MAE: %2.2f , RMSE: %2.2f ' % (MAE, RMSE)

###########################################################
# TEST CELL
###########################################################

Test.assertEquals(np.round(MAE,2), 0.80, 'incorrect result: MAE value is incorrect')
Test.assertEquals(np.round(RMSE,2), 1.02, 'incorrect result: RMSE value is incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
Alternative Least Squares algorithm
Now, let's work with the ALS algorithm. As you know, this method tries to approximate the ratings matrix by factorizing it as the product of two matrices:
$$ R = X * Y $$
where $X$ describes properties of each user, and $Y$ describes properties of each item. These two matrices are known as latent factors, since they are a low-dimension representation of users and items.
If we examine the utilities of the MLLib, we can find a implementation of the ALS algorithm. So, in this section, we will learn to use this module of the MLlib.
 Training a ALS model
This library includes the method ALS.train( ) which directly allows us training an ALS model. Use this function to train a recommender system.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

from pyspark.mllib.recommendation import ALS

# Define parameters
n_latent_factors = 5
numIterations = 15

# Train the model (set seed=0L)
sc.setCheckpointDir('checkpoint/')
model = # FILL IN  , seed=0L)


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
Baseline recommender
In this section we are going to build a mean based baseline; that is, the recommender will predict new ratings as the average value of the ratings given by this user to previous rated items.
To design this approach, let's start building a function that, given a user_id and all its associated ratings, is able to compute the average value of all the ratings.
1. Build function getAverages( )
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

def getAverages(IDandRatingsTuple):
    """ Calculate average rating
    Args:
        IDandRatingsTuple: a single tuple of (ID_user, (Rating1, Rating2, Rating3, ...))
    Returns:
        tuple: a tuple of (ID_user, averageRating)
    """
    id_user = # FILL IN
    mean_value = # FILL IN
    return (id_user, mean_value)

###########################################################
# TEST CELL
###########################################################
check_ratings = (0, iter(2, 5, 3, 1, 2))
check_output = getAverages(check_ratings)
Test.assertEquals(check_output, (0, 2.6), 'incorrect result: check_output is incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
Computing predictions
Once the model has been trained, let's make the recommendations. For this purpose, the ALS model has a method model.predictAll(testdata) which estimates the ratings over an RDD of ID pairs (userID, itemID).
So, complete the next cell to estimate the rating over the pairs of (users, items) of our test data.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################

# Create a new RDD, RDD_test_ids, consisting of (UserID, MovieID) pairs
# that you extract from testRDD. That is, remove the filed Rating from testRDD.
RDD_test_ids = # FILL IN

# Estimate their ratings with model.predictAll( )
predictions = # FILL IN

# Print the first 10 predictions
predictions.take(10)

###########################################################
# TEST CELL
###########################################################
check_predictions = predictions.filter(lambda x: (x[0]==621) & (x[1]==68)).first()
Test.assertEquals(np.round(check_predictions[2],1), 3.7, 'incorrect result: predicted value is incorrect')
check_predictions = predictions.filter(lambda x: (x[0]==880) & (x[1]==8)).first()
Test.assertEquals(np.round(check_predictions[2],1), 4, 'incorrect result: predicted value is incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
Creating training and test rating matrices
Now, to be able to train and evaluate the different methods, let's divide the rating matrix into two different matrix:
* one of them with the 75% of the ratings for training the different recommenders;
* other one, with the remaining 25%, for testing purposes.
Hint: you can apply the randomSplit() method of the RDD to divide it at random.
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################
trainingRDD, testRDD = ratingsRDD.randomSplit(#FILL IN, seed=0L)

print 'Training: %s,  test: %s\n' % (trainingRDD.count(), testRDD.count())

###########################################################
# TEST CELL
###########################################################
Test.assertEquals(trainingRDD.count(), 75008, 'incorrect result: number of training ratings is incorrect')
Test.assertEquals(testRDD.count(), 24992, 'incorrect result: number of test ratings is incorrect')
Test.assertEquals(trainingRDD.first(), (186, 302, 3.0), 'incorrect result: the values of the training RDD are incorrect')
Test.assertEquals(testRDD.first(), (196, 242, 3.0), 'incorrect result: the values of the testing RDD are incorrect')


I am working on the file "5_RecommenderSystems/Recommender systems - Part 2-Students.ipynb".
The first task is:
Formatting the data
As you can checked, each line is formatted as:
UserID \t MovieID \t Rating \t Timestamp \n. So, let's convert each line to a list with the fields [UserID, MovieID, Rating] (we drop the timestamp because we do not need it for this exercise).
In order to work in a distributed way, let's start implementing a function (format_rating) that let's convert each line into the desired format. Then, we can call this function from our RDD with a map method to apply it over each line.
Tip: Check the Python function split() to convert each line into a list of items split by a given character.
1. Create function format_ratings( )
Can you write Python code for it?

###########################################################
# TODO: Replace <FILL IN> with appropriate code
###########################################################
def format_ratings(line):
    """ Parse a line in the ratings dataset
    Args:
        line (str): a line in the ratings dataset in the form of UserID \t MovieID \t Rating \t Timestamp \n
    Returns:
        tuple: (UserID, MovieID, Rating)
    """
    # Divide each line with the character '\t'
    items = # FILL
    # Get UserID and convert it to int
    user_id = # FILL
    # Get ItemID and convert it to int
    item_id = # FILL
    # Get Reating and convert it to float
    rating_id = # FILL
    # Return UserID, ItemID and Rating.
    return # FILL

###########################################################
# TEST CELL
###########################################################
check_line = u'196\t242\t3\t881250949'
check_tuple = format_ratings(check_line)
Test.assertEquals(check_tuple, (196, 242, 3), 'incorrect result: data are incorrectly formatted')