I am working on the file "Python/63_Registration_Initialization.ipynb".
The first task is:
Why loop if you can process in parallel
As the metric evaluations are independent of each other, we can easily replace looping with parallelization.
Can you write Python code for it?

from multiprocessing.pool import ThreadPool
from functools import partial

# This function evaluates the metric value in a thread safe manner
def evaluate_metric(current_rotation, tx, f_image, m_image):
    registration_method = sitk.ImageRegistrationMethod()
    registration_method.SetMetricAsMattesMutualInformation(numberOfHistogramBins=50)
    registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
    registration_method.SetMetricSamplingPercentage(0.01)
    registration_method.SetInterpolator(sitk.sitkLinear)
    current_transform = sitk.Euler3DTransform(tx)
    current_transform.SetRotation(*current_rotation)
    registration_method.SetInitialTransform(current_transform)
    res = registration_method.MetricEvaluate(f_image, m_image)
    return res

p = ThreadPool(len(all_orientations) + 1)
orientations_list = [(0, 0, 0)] + list(all_orientations.values())
all_metric_values = p.map(
    partial(
        evaluate_metric, tx=initial_transform, f_image=fixed_image, m_image=moving_image
    ),
    orientations_list,
)
best_orientation = orientations_list[np.argmin(all_metric_values)]
print("best orientation is: " + str(best_orientation))

initial_transform.SetRotation(*best_orientation)
final_transform, _ = multires_registration(fixed_image, moving_image, initial_transform)


I am working on the file "Python/64_Registration_Memory_Time_Tradeoff.ipynb".
The first task is:
Compare the error distributions
To fairly compare the two registration above we look at their running times (see results above) and their
error distributions (plotted below).
Can you write Python code for it?

plt.hist(
    original_resolution_errors,
    bins=20,
    alpha=0.5,
    label="original resolution",
    color="blue",
)
plt.hist(
    resampled_resolution_errors,
    bins=20,
    alpha=0.5,
    label="higher resolution",
    color="green",
)
plt.legend()
plt.title("TRE histogram");


I am working on the file "Python/64_Registration_Memory_Time_Tradeoff.ipynb".
The first task is:
Higher Resolution
For this registration we use the higher resolution image and nearest neighbor interpolation.
Can you write Python code for it?

%%timeit -r1 -n1
# The arguments to the timeit magic specify that this cell should only be run once.

# We define this variable as global so that it is accessible outside of the cell (timeit wraps the code in the cell
# making all variables local, unless explicitly declared global).
global resampled_resolution_errors

final_transform, optimizer_termination = register_images(
    fixed_image, resampled_moving_image, initial_transform, sitk.sitkNearestNeighbor
)
(
    final_errors_mean,
    final_errors_std,
    _,
    final_errors_max,
    resampled_resolution_errors,
) = ru.registration_errors(final_transform, fixed_points, moving_points)

print(optimizer_termination)
print(
    f"After registration, errors in millimeters, mean(std): {final_errors_mean:.2f}({final_errors_std:.2f}), max: {final_errors_max:.2f}"
)


I am working on the file "Python/64_Registration_Memory_Time_Tradeoff.ipynb".
The first task is:
Original Resolution
For this registration we use the original resolution and linear interpolation.
Can you write Python code for it?

%%timeit -r1 -n1
# The arguments to the timeit magic specify that this cell should only be run once.

# We define this variable as global so that it is accessible outside of the cell (timeit wraps the code in the cell
# making all variables local, unless explicitly declared global).
global original_resolution_errors

final_transform, optimizer_termination = register_images(
    fixed_image, moving_image, initial_transform, sitk.sitkLinear
)
(
    final_errors_mean,
    final_errors_std,
    _,
    final_errors_max,
    original_resolution_errors,
) = ru.registration_errors(final_transform, fixed_points, moving_points)

print(optimizer_termination)
print(
    f"After registration, errors in millimeters, mean(std): {final_errors_mean:.2f}({final_errors_std:.2f}), max: {final_errors_max:.2f}"
)


I am working on the file "Python/65_Registration_FFD.ipynb".
The first task is:
Free Form Deformation
This function will align the fixed and moving images using a FFD. If given a mask, the similarity metric will be evaluated using points sampled inside the mask. If given fixed and moving points the similarity metric value and the target registration errors will be displayed during registration.
As this notebook performs intra-modal registration, we use the MeanSquares similarity metric (simple to compute and appropriate for the task).
Can you write Python code for it?

def bspline_intra_modal_registration(
    fixed_image,
    moving_image,
    fixed_image_mask=None,
    fixed_points=None,
    moving_points=None,
):

    registration_method = sitk.ImageRegistrationMethod()

    # Determine the number of BSpline control points using the physical spacing we want for the control grid.
    grid_physical_spacing = [50.0, 50.0, 50.0]  # A control point every 50mm
    image_physical_size = [
        size * spacing
        for size, spacing in zip(fixed_image.GetSize(), fixed_image.GetSpacing())
    ]
    mesh_size = [
        int(image_size / grid_spacing + 0.5)
        for image_size, grid_spacing in zip(image_physical_size, grid_physical_spacing)
    ]

    initial_transform = sitk.BSplineTransformInitializer(
        image1=fixed_image, transformDomainMeshSize=mesh_size, order=3
    )
    registration_method.SetInitialTransform(initial_transform)

    registration_method.SetMetricAsMeanSquares()
    # Settings for metric sampling, usage of a mask is optional. When given a mask the sample points will be
    # generated inside that region. Also, this implicitly speeds things up as the mask is smaller than the
    # whole image.
    registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
    registration_method.SetMetricSamplingPercentage(0.01)
    if fixed_image_mask:
        registration_method.SetMetricFixedMask(fixed_image_mask)

    # Multi-resolution framework.
    registration_method.SetShrinkFactorsPerLevel(shrinkFactors=[4, 2, 1])
    registration_method.SetSmoothingSigmasPerLevel(smoothingSigmas=[2, 1, 0])
    registration_method.SmoothingSigmasAreSpecifiedInPhysicalUnitsOn()

    registration_method.SetInterpolator(sitk.sitkLinear)
    registration_method.SetOptimizerAsLBFGSB(
        gradientConvergenceTolerance=1e-5, numberOfIterations=100
    )

    # If corresponding points in the fixed and moving image are given then we display the similarity metric
    # and the TRE during the registration.
    if fixed_points and moving_points:
       ...


I am working on the file "Python/65_Registration_FFD.ipynb".
The first task is:
Multi-resolution control point grid
In the example above we used the standard image registration framework. This implies the same transformation model at all image resolutions. For global transformations (e.g. rigid, affine...) the number of transformation parameters has no relationship to the changing resolution. For the BSpline transformation we can potentially use fewer control points for images with lower frequencies, higher levels of the image pyramid, increasing the number of control points as we go down the pyramid. With the standard framework we use the same number of control points for all pyramid levels.
To use a multi-resolution control point grid we have a specific initializer for the BSpline transformation, SetInitialTransformAsBSpline.
The following code solves the same registration task as above, just with a multi-resolution control point grid.
Can you write Python code for it?

def bspline_intra_modal_registration2(
    fixed_image,
    moving_image,
    fixed_image_mask=None,
    fixed_points=None,
    moving_points=None,
):

    registration_method = sitk.ImageRegistrationMethod()

    # Determine the number of BSpline control points using the physical spacing we
    # want for the finest resolution control grid.
    grid_physical_spacing = [50.0, 50.0, 50.0]  # A control point every 50mm
    image_physical_size = [
        size * spacing
        for size, spacing in zip(fixed_image.GetSize(), fixed_image.GetSpacing())
    ]
    mesh_size = [
        int(image_size / grid_spacing + 0.5)
        for image_size, grid_spacing in zip(image_physical_size, grid_physical_spacing)
    ]

    # The starting mesh size will be 1/4 of the original, it will be refined by
    # the multi-resolution framework.
    mesh_size = [int(sz / 4 + 0.5) for sz in mesh_size]

    initial_transform = sitk.BSplineTransformInitializer(
        image1=fixed_image, transformDomainMeshSize=mesh_size, order=3
    )
    # Instead of the standard SetInitialTransform we use the BSpline specific method which also
    # accepts the scaleFactors parameter to refine the BSpline mesh. In this case we start with
    # the given mesh_size at the highest pyramid level then we double it in the next lower level and
    # in the full resolution image we use a mesh that is four times the original size.
    registration_method.SetInitialTransformAsBSpline(
        initial_transform, inPlace=True, scaleFactors=[1, 2, 4]
    )
    registration_method.SetMetricAsMeanSquares()
    # Settings for metric sampling, usage of a mask is optional. When given a mask the sample points will be
    # generated inside that region. Also, this implicitly speeds things up as the mask is smaller than the
    # whole image.
    registration_method.SetMetricSamplingStrategy(registration_method.RANDOM)
    registration_method.SetMetricSamplingPercentage(0.01)
    if fixed_image_mask:
        registration_method.SetMetricFixedMask(fixed_image_mask)

 ...


I am working on the file "Python/65_Registration_FFD.ipynb".
The first task is:
Perform Registration
The following cell allows you to select the images used for registration, runs the registration, and afterwards computes statistics comparing the target registration errors before and after registration and displays a histogram of the TREs.
To time the registration, uncomment the timeit magic.
<b>Note</b>: this creates a separate scope for the cell. Variables set inside the cell, specifically tx, will become local variables and thus their value is not available in other cells.
Can you write Python code for it?

#%%timeit -r1 -n1

# Select the fixed and moving images, valid entries are in [0,9].
fixed_image_index = 0
moving_image_index = 7

tx = bspline_intra_modal_registration(
    fixed_image=images[fixed_image_index],
    moving_image=images[moving_image_index],
    fixed_image_mask=(masks[fixed_image_index] == lung_label),
    fixed_points=points[fixed_image_index],
    moving_points=points[moving_image_index],
)
(
    initial_errors_mean,
    initial_errors_std,
    _,
    initial_errors_max,
    initial_errors,
) = ru.registration_errors(
    sitk.Euler3DTransform(), points[fixed_image_index], points[moving_image_index]
)
(
    final_errors_mean,
    final_errors_std,
    _,
    final_errors_max,
    final_errors,
) = ru.registration_errors(tx, points[fixed_image_index], points[moving_image_index])

plt.hist(initial_errors, bins=20, alpha=0.5, label="before registration", color="blue")
plt.hist(final_errors, bins=20, alpha=0.5, label="after registration", color="green")
plt.legend()
plt.title("TRE histogram")
print(
    f"Initial alignment errors in millimeters, mean(std): {initial_errors_mean:.2f}({initial_errors_std:.2f}), max: {initial_errors_max:.2f}"
)
print(
    f"Final alignment errors in millimeters, mean(std): {final_errors_mean:.2f}({final_errors_std:.2f}), max: {final_errors_max:.2f}"
)


I am working on the file "Python/65_Registration_FFD.ipynb".
The first task is:
Utilities
Load utilities that are specific to the POPI data, functions for loading ground truth data, display and the labels for masks.
Can you write Python code for it?


I am working on the file "Python/66_Registration_Demons.ipynb".
The first task is:
Now we will use our newly minted multiscale framework to perform registration with the Demons filters. Some things you can easily try out by editing the code below:
1. Is there really a need for multiscale - just call the multiscale_demons method without the shrink_factors and smoothing_sigmas parameters.
2. Which Demons filter should you use - configure the other filters and see if our selection is the best choice (accuracy/time).
Can you write Python code for it?

# Define a simple callback which allows us to monitor the Demons filter's progress.
def iteration_callback(filter):
    print(f"\r{filter.GetElapsedIterations()}: {filter.GetMetric():.2f}", end="")

fixed_image_index = 0
moving_image_index = 7

# Select a Demons filter and configure it.
demons_filter = sitk.FastSymmetricForcesDemonsRegistrationFilter()
demons_filter.SetNumberOfIterations(20)
# Regularization (update field - viscous, total field - elastic).
demons_filter.SetSmoothDisplacementField(True)
demons_filter.SetStandardDeviations(2.0)

# Add our simple callback to the registration filter.
demons_filter.AddCommand(
    sitk.sitkIterationEvent, lambda: iteration_callback(demons_filter)
)

# Run the registration.
tx = multiscale_demons(
    registration_algorithm=demons_filter,
    fixed_image=images[fixed_image_index],
    moving_image=images[moving_image_index],
    shrink_factors=[4, 2],
    smoothing_sigmas=[8, 4],
)

# Compare the initial and final TREs.
(
    initial_errors_mean,
    initial_errors_std,
    _,
    initial_errors_max,
    initial_errors,
) = ru.registration_errors(
    sitk.Euler3DTransform(), points[fixed_image_index], points[moving_image_index]
)
(
    final_errors_mean,
    final_errors_std,
    _,
    final_errors_max,
    final_errors,
) = ru.registration_errors(tx, points[fixed_image_index], points[moving_image_index])

plt.hist(initial_errors, bins=20, alpha=0.5, label="before registration", color="blue")
plt.hist(final_errors, bins=20, alpha=0.5, label="after registration", color="green")
plt.legend()
plt.title("TRE histogram")
print(
    f"\nInitial alignment errors in millimeters, mean(std): {initial_errors_mean:.2f}({initial_errors_std:.2f}), max: {initial_errors_max:.2f}"
)
print(
    f"Final alignment errors in millimeters, mean(std): {final_errors_mean:.2f}({final_errors_std:.2f}), max: {final_errors_max:.2f}"
)


I am working on the file "Python/66_Registration_Demons.ipynb".
The first task is:
Register and transfer the segmentation.
Can you write Python code for it?

# Select a Demons filter and configure it.
demons_filter = sitk.DiffeomorphicDemonsRegistrationFilter()
demons_filter.SetNumberOfIterations(20)
# Regularization (update field - viscous, total field - elastic).
demons_filter.SetSmoothDisplacementField(True)
demons_filter.SetStandardDeviations(0.8)

# create initial transform
initial_tfm = initial_transform = sitk.CenteredTransformInitializer(
    segmented_img,
    new_img,
    sitk.Euler2DTransform(),
    sitk.CenteredTransformInitializerFilter.GEOMETRY,
)
# Run the registration.
final_tfm = multiscale_demons(
    registration_algorithm=demons_filter,
    fixed_image=segmented_img,
    moving_image=new_img,
    initial_transform=initial_tfm,
    shrink_factors=[6, 4, 2],
    smoothing_sigmas=[6, 4, 2],
)

# Display the transformed segmentation.
fig, axes = multi_image_display2D([segmented_img, new_img])
for contour in contours_list:
    # Plot on segmented image.
    axes[0].plot(contour[0], contour[1], linewidth=5)
    # Transform the contour points from segmented image to new image (requires the use of points in physical space)
    transformed_contour = [
        new_img.TransformPhysicalPointToContinuousIndex(
            final_tfm.TransformPoint(
                segmented_img.TransformContinuousIndexToPhysicalPoint(p)
            )
        )
        for p in zip(contour[0], contour[1])
    ]
    x_coords, y_coords = zip(*transformed_contour)
    axes[1].plot(x_coords, y_coords, linewidth=5)


I am working on the file "Python/66_Registration_Demons.ipynb".
The first task is:
Transferring Segmentation
The following example illustrates the use of the Demons registration method to transfer a segmentation from one dataset to another. Note that we use the same multi scale demons framework we used above for registering 3D images to register 2D images.
Data provided courtesy of T. Sorensen, Signal Analysis and Interpretation Laboratory University of Southern California. Data acquisition protocol described in:
S.G. Lingala, A. Toutios, J. Toger, Y. Lim, Y. Zhu, Y-C. Kim, C. Vaz, S. Narayanan, K.S. Nayak "State of the art MRI protocol for comprehensive assessment of vocal tract structure and function" Proceedings of Interspeech, San Francisco, 2016.
NOTE: Segmented contour points are given in pixel coordinates. This is useful for plotting. To transfer the points to the second image we need to (1) use their physical coordinates, (2) transform them, and then (3) obtain the pixel coordinates for plotting.
Load Data
Can you write Python code for it?

import glob
import pandas as pd
from gui import multi_image_display2D

# Fetch all of the data associated with this example.
data_directory = os.path.dirname(fdata("mr_slice_atlas/readme.txt"))

segmented_img = sitk.ReadImage(os.path.join(data_directory, "segmented_image.mha"))
new_img = sitk.ReadImage(os.path.join(data_directory, "new_image.mha"))

contours_list = []
for file_name in glob.glob(os.path.join(data_directory, "*.csv")):
    df = pd.read_csv(file_name)
    contours_list.append((list(df["X"]), list(df["Y"])))

# Display the images and overlay the contours onto the segmented image.
fig, axes = multi_image_display2D([segmented_img, new_img])
for contour in contours_list:
    axes[0].plot(contour[0], contour[1], linewidth=5)


I am working on the file "Python/66_Registration_Demons.ipynb".
The first task is:
Utilities
Load utilities that are specific to the POPI data, functions for loading ground truth data, display and the labels for masks.
Can you write Python code for it?


I am working on the file "Python/67_Registration_Semiautomatic_Homework.ipynb".
The first task is:
Homework: semiautomatic landmark localization
You will now improve the localization of the fixed landmarks in the moving image using intensity based registration. This registration is performed independently for each landmark.
The output of the following cell is expected to be a list of tuples (3D) called updated_moving_image_points.
Hint: You need to initialize the intensity based registration in a way that takes into account that the images may have a significant rotation between them (up to 180$^o$).
Can you write Python code for it?


I am working on the file "Python/67_Registration_Semiautomatic_Homework.ipynb".
The first task is:
Manual Landmark Localization
We now localize N(>=3) landmarks in the two images. Note that you can zoom and pan the images, just remember to change the interaction mode from "edit" to "view".
NOTE: In edit mode, the GUI will force you to enter corresponding points by disabling the option for consecutively localizing multiple (>2) points in the same image. In view mode, point localization is disabled which is useful for zooming/panning (in edit mode zooming/panning will also localize points due to the mouse button click).
Can you write Python code for it?

point_acquisition_interface = gui.RegistrationPointDataAquisition(
    fixed_image, moving_image, fixed_window_level=(215, 50)
)


I am working on the file "Python/67_Registration_Semiautomatic_Homework.ipynb".
The first task is:
Registration (manual landmark localization)
Evaluate the quality of the manual localization by registering the two images, and then comparing the registration errors using the known reference data.
Can you write Python code for it?

# fixed_image_points, moving_image_points = point_acquisition_interface.get_points()
fixed_image_points = [
    (156.48434676356158, 201.92274575468412, 68.0),
    (194.25413436597393, 98.55771047484492, 32.0),
    (128.94523819661913, 96.18284152323203, 32.0),
]
moving_image_points = [
    (141.46826904042848, 156.97653126727528, 48.0),
    (113.70102381552435, 251.76553994455645, 8.0),
    (180.69457220262115, 251.76553994455645, 8.0),
]

fixed_image_points_flat = [c for p in fixed_image_points for c in p]
moving_image_points_flat = [c for p in moving_image_points for c in p]
manual_localized_transformation = sitk.VersorRigid3DTransform(
    sitk.LandmarkBasedTransformInitializer(
        sitk.VersorRigid3DTransform(), fixed_image_points_flat, moving_image_points_flat
    )
)

(
    manual_errors_mean,
    manual_errors_std,
    manual_errors_min,
    manual_errors_max,
    _,
) = ru.registration_errors(
    manual_localized_transformation, fixed_points, moving_points, display_errors=True
)
print(
    f"After registration (manual point localization), errors (TRE) in millimeters, mean(std): {manual_errors_mean:.2f}({manual_errors_std:.2f}), max: {manual_errors_max:.2f}"
)


I am working on the file "Python/68_Registration_Errors.ipynb".
The first task is:
FRE is not a surrogate for TRE
In the next code cell we illustrate that FRE and TRE are not correlated. We first add the same fixed bias to
all of the moving fiducials. This results in a large TRE, but the FRE remains zero. We then add a fixed bias to half of the moving fiducials and the negative of that bias to the other half. This results in a small TRE, but the FRE is large.
Can you write Python code for it?

# Registration with same bias added to all points
fixed_fiducials = copy.deepcopy(ideal_fixed_fiducials)
fixed_targets = copy.deepcopy(ideal_fixed_targets)
moving_fiducials = copy.deepcopy(ideal_moving_fiducials)
bias_vector = [4.5, 4.5]
bias_fle = np.sqrt(bias_vector[0] ** 2 + bias_vector[1] ** 2)
for fiducial in moving_fiducials:
    fiducial[0] += bias_vector[0]
    fiducial[1] += bias_vector[1]
FLE_values = [bias_fle] * len(moving_fiducials)
moving_targets = copy.deepcopy(ideal_moving_targets)

# Flatten the point lists, SimpleITK expects a single list/tuple with coordinates (x1,y1,...xn,yn)
fixed_fiducials_flat = [c for p in fixed_fiducials for c in p]
moving_fiducials_flat = [c for p in moving_fiducials for c in p]

transform = sitk.LandmarkBasedTransformInitializer(
    sitk.Euler2DTransform(), fixed_fiducials_flat, moving_fiducials_flat
)

FRE_information = registration_errors(transform, fixed_fiducials, moving_fiducials)
TRE_information = registration_errors(transform, fixed_targets, moving_targets)
FLE_information = (
    np.mean(FLE_values),
    np.std(FLE_values),
    np.min(FLE_values),
    np.max(FLE_values),
    FLE_values,
)
display_errors(
    fixed_fiducials,
    fixed_targets,
    FLE_information,
    FRE_information,
    TRE_information,
    title="FRE<TRE",
)

# Registration with bias in one direction for half the fiducials and in the opposite direction for the other half
moving_fiducials = copy.deepcopy(ideal_moving_fiducials)
pol = 1
for fiducial in moving_fiducials:
    fiducial[0] += bias_vector[0] * pol
    fiducial[1] += bias_vector[1] * pol
    pol *= -1.0
FLE_values = [bias_fle] * len(moving_fiducials)
moving_targets = copy.deepcopy(ideal_moving_targets)

# Flatten the point lists, SimpleITK expects a single list/tuple with coordinates (x1,y1,...xn,yn)
fixed_fiducials_flat = [c for p in fixed_fiducials for c in p]
moving_fiducials_flat = [c for p in moving_fiducials for c in p]

transform = sitk.LandmarkBasedTransformInitializer(
    sitk.Euler2DTransform(), fixed_fiducials_flat,...


I am working on the file "Python/68_Registration_Errors.ipynb".
The first task is:
FRE-TRE, and Occam's razor
When we perform registration, our goal is to minimize the Target Registration Error. In practice it needs to be below a problem specific threshold for the registration to be useful.
The target point(s) can be a single point or a region in space, and we want to minimize our registration error for this target. We go about this task by minimizing another quantity, in paired-point registration this is the FRE, in the case of intensity based registration we minimize an appropriate similarity metric. In both cases we expect that TRE is minimized indirectly.
This can easily lead us astray, down the path of overfitting. In our 2D case, instead of using a rigid transformation with three degrees of freedom we may be tempted to use an affine transformation with six degrees of freedom. By introducing these additional degrees of freedom we will likely improve the FRE, but what about TRE?
In the cell below  you can qualitatively evaluate the effects of overfitting. Start by adding noise with no rotation or translation and then register. Switch to an affine transformation model and see how registration effects the fiducials and targets. You can then repeat this qualitative evaluation incorporating translation/rotation and noise.
In this notebook we are working in an ideal setting, we know the appropriate transformation model is rigid. Unfortunately, this is often not the case. So which transformation model should you use? When presented with multiple competing hypotheses we select one using the principle of parsimony, often referred to as Occam's razor. Our choice is to select the simplest model that can explain our observations.
In the case of registration, the transformation model with the least degrees of freedom that reduces the TRE below the problem specific threshold.
Can you write Python code for it?


I am working on the file "Python/68_Registration_Errors.ipynb".
The first task is:
Fiducial Configuration
Even when our model of the world is correct, no outliers and FLE is isotropic and homogeneous, the fiducial configuration
has a significant effect on the TRE. Ideally you want the targets to be at the centroid of your fiducial configuration.
This is illustrated in the code cell below. Translate, rotate and add noise to the fiducials, then register. The targets that are near the fiducials should have a better alignment than those far from the fiducials.
Now, reset the setup. Where would you add two fiducials to improve the overall TRE? Experiment with various fiducial configurations.
Can you write Python code for it?


I am working on the file "Python/7 Neural Networks/Autograd-Intro.ipynb".
The first task is:
In the lecture we have seen that the following identity holds for the derivative of the sigmoid function:
$$ S'(x) = S(x) \cdot \bigl(1 - S(x)\bigr) $$
Let's test this identity.
Can you write Python code for it?


I am working on the file "Python/7 Neural Networks/Autograd-Intro.ipynb".
The first task is:
The function sigmoid implements the sigmoid function, which is defined as
$$ \texttt{S}(x) = \frac{1}{1 + \mathrm{e}^{-x}}. $$
Can you write Python code for it?

def S(x):
    return 1.0 / (1.0 + np.exp(-x))

def Q(x):
    return np.multiply(x, x)

Q_grad = autograd.grad(Q)

Q_grad(1.0)


I am working on the file "Python/7 Neural Networks/Digit-Recognition-NN.ipynb".
The first task is:
As our data is stored as a tuple of numpy arrays, we have to import numpy.
Can you write Python code for it?


I am working on the file "Python/7 Neural Networks/Digit-Recognition-NN.ipynb".
The first task is:
Given a neural network $n$ and an input vector $x$ for this neural network, the function $n.\texttt{feedforward}(x)$ compute the output of the neural network.
The code is a straightforward implementation of the feedforward equations.
These equations are repeated here for convenience:
- $\mathbf{a}^{(1)}(\mathbf{x}) = \mathbf{x}$
- $\mathbf{a}^{(l)}(\mathbf{x}) = S\Bigl( W^{(l)} \cdot \mathbf{a}^{(l-1)}(\mathbf{x}) + \mathbf{b}^{(l)}\Bigr)$
  for all $l \in {2, 3}$.
The input x is the activation of the input layer and therefore is equal to $\mathbf{a}^{(1)}(\mathbf{x})$.
AH is the activation of the hidden layer and hence equal to $\mathbf{a}^{(2)}(\mathbf{x})$, while
AO is the activation of the output layer and therefore equal to $\mathbf{a}^{(3)}(\mathbf{x})$.
Can you write Python code for it?

def feedforward(self, x):
    AH = sigmoid(self.mWeightsH @ x  + self.mBiasesH) # hidden layer
    AO = sigmoid(self.mWeightsO @ AH + self.mBiasesO) # output layer
    return AO

Network.feedforward = feedforward


I am working on the file "Python/7 Neural Networks/Digit-Recognition-NN.ipynb".
The first task is:
Given a neural network $n$, the method sgd(training_data, epochs, mbs, eta, test_data) uses stochastic gradient descent to train the network.  The parameters are as follows:
* training_data is a list of tuples of the form $(x, y)$ where $x$ is an
  input of the neural net and $y$ is a vector of length 10 representing the desired output.
* epochs is the number of epochs to train,
* mbs is the size of the minibatches,
* eta is the learning rate,
* test_data is a list of tuples of the form $(x, y)$ where $x$ is an
   input and $y$ is the desired output digit.
Can you write Python code for it?

def sgd(self, training_data, epochs, mbs, eta, test_data):
    n_test = len(test_data)
    n      = len(training_data)
    for j in range(epochs):
        random.shuffle(training_data)
        mini_batches = [training_data[k:k+mbs] for k in range(0, n, mbs)]
        for mini_batch in mini_batches:
            self.update_mini_batch(mini_batch, eta)
        print('Epoch %2d: %d / %d' % (j, self.evaluate(test_data), n_test))

Network.sgd = sgd


I am working on the file "Python/7 Neural Networks/Digit-Recognition-NN.ipynb".
The first task is:
Given a neural network n, the method n.evaluate(test_data) uses the test data to compute  the number of examples that are predicted correctly by the neural network n.
Can you write Python code for it?

def evaluate(self, test_data):
    test_results = [(np.argmax(self.feedforward(x)), y) for x, y in test_data]
    return sum(y1 == y2 for y1, y2 in test_results)

Network.evaluate = evaluate

%%time
np.random.seed(1)
net = Network(50)
net.sgd(training_data, 30, 10, 0.25, test_data)


I am working on the file "Python/7 Neural Networks/Digit-Recognition-NN.ipynb".
The first task is:
In order to be able to show the images of the handwritten digits, we use matplotlib.
Can you write Python code for it?


I am working on the file "Python/7 Neural Networks/Digit-Recognition-NN.ipynb".
The first task is:
The class Network is used to represent a feed-forward neural network with one hidden layer.
The constructor is called with the argument hiddenSize.  This parameter specifies the number of neurons in the hidden layer.  The network has $28 \cdot 28 = 784$ input nodes.  Each of the input nodes corresponds to the gray scale value of a single pixel in a $28 \cdot 28$ gray scale image of the digit that is to be recognized.  The number of output neurons is 10.  For $i \in {0,\cdots,9}$, the $i$th output neuron tries to recognize the digit $i$.
Can you write Python code for it?

class Network(object):
    def __init__(self, hiddenSize):
        self.mInputSize  = 28 * 28
        self.mHiddenSize = hiddenSize
        self.mOutputSize = 10
        self.mBiasesH    = np.zeros((self.mHiddenSize, 1))   # biases hidden layer
        self.mBiasesO    = np.zeros((self.mOutputSize, 1))   # biases output layer
        self.mWeightsH   = rndMatrix(self.mHiddenSize, self.mInputSize)  # weights hidden layer
        self.mWeightsO   = rndMatrix(self.mOutputSize, self.mHiddenSize) # weights output layer


I am working on the file "Python/7 Neural Networks/Digit-Recognition-NN.ipynb".
The first task is:
The file mnist.pkl.gz contains a triple of the form
train, validate, test
Here train is a pair of the form (X, y) where
- X is a numpy array of shape (50000, 784),
- y is a numpy array of shape (50000, ).
For every $i \in {0,\cdots, 49,999}$ we have that $\textbf{X}[i]$ is an image of a handwritten digit and $\textbf{y}[i]$ is a digit, i.e. an element of the set
${0,\cdots,9}$.
The structure of validate and test is similar, but these contain only $10,000$ images each.
The function $\texttt{load_data}()$ returns a pair of the form
$$ (\texttt{training_data}, \texttt{test_data}) $$
where
- $\texttt{training_data}$ is a list containing 50,000 pairs $(\textbf{x}, \textbf{y})$       s.t.
  - $\textbf{x}$ is a 784-dimensional numpy.ndarray containing the input image, and
  - $\textbf{y}$ is a 10-dimensional numpy.ndarray corresponding to the correct digit for
    $\textbf{x}$.
- To keep things simple, we do not use the validation data.
- $\texttt{test_data}$ is a list containing 10,000 pairs $(\textbf{x}, y)$.  In each case,
  $\textbf{x}$ is a 784-dimensional numpy.ndarray containing the input image,
  and $y \in {0,\cdots,9}$ is the corresponding digit value.
Note that the formats for training data and test data are different.  For the training data $\textbf{y}$ is a vector, but for the test data $y$ is a digit.
Can you write Python code for it?

def load_data():
    with gzip.open('../mnist.pkl.gz', 'rb') as f:
        train, validate, test = pickle.load(f, encoding="latin1")
    print(f'shape of training data: {(train[0].shape, train[1].shape)}')
    training_inputs    = [np.reshape(x, (784, 1)) for x in train[0]]
    training_results   = [vectorized_result(y) for y in train[1]]
    training_data      = list(zip(training_inputs, training_results))
    test_inputs        = [np.reshape(x, (784, 1)) for x in test[0]]
    test_data          = list(zip(test_inputs, test[1]))
    return training_data, test_data

list(zip([1, 2], ['a', 'b']))


I am working on the file "Python/7 Neural Networks/Digit-Recognition-NN.ipynb".
The first task is:
The function $\texttt{sigmoid}(x)$ computes the sigmoid of $x$, which is defined as
$$ \texttt{sigmoid}(x) = S(x) := \frac{1}{1 + \texttt{exp}(-x)}. $$
Since we are using NumPy to compute the exponential function, this function also works when $x$ is a vector.
Can you write Python code for it?

def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))

sigmoid(np.array([-1, 0, 1]))


I am working on the file "Python/7 Neural Networks/Digit-Recognition-NN.ipynb".
The first task is:
The function $\texttt{vectorized_result}(d)$ converts a digit $d \in {0,\cdots,9}$ into a numpy array $\mathbf{y}$ of shape $(10, 1)$ such that we have
$$
\mathbf{y}[i] =
\left{
  \begin{array}{ll}
     1 & \mbox{if $i = d$;} \
     0 & \mbox{otherwise}
  \end{array}
\right.
$$
for all $i \in {0,\cdots,9}$.
This function is used to convert a digit $d$ into the expected output of a neural network that has an output unit for every digit.
Can you write Python code for it?

def vectorized_result(d):
    e    = np.zeros((10, 1), dtype=np.float32)
    e[d] = 1.0
    return e

vectorized_result(2)


I am working on the file "Python/7 Neural Networks/NN-Architecture.ipynb".
The first task is:
The function $\texttt{generateNN}(\texttt{Topology})$ takes a
network topology Topology as its argument and draws a graph of the
resulting fully connected feed-forward neural net.  A network topology is a list of numbers specifying the number of neurons of each layer.
For example,  the network topology [3, 8, 6, 2] specifies a neural network with three layers of neurons.
The network has $3$ input nodes, the first hidden layer has $8$ neurons, the second hidden layer has $6$ neurons, and
the output layer has $2$ neurons.
Can you write Python code for it?

def generateNN(Topology):
    L = len(Topology)
    input_layer   = ['i' + str(i) for i in range(1, Topology[0]+1)]
    hidden_layers = [['h' + str(k+1) + ',' + str(i) for i in range(1, s+1)]
                                                    for (k, s) in enumerate(Topology[1:-1])]
    output_layer  = ['o' + str(i) for i in range(1, Topology[-1]+1)]
    nng = gv.Graph()
    nng.attr(rankdir='LR', splines='false')
    # create nodes for input layer
    for n in input_layer:
        nng.node(n, label='', shape='point', width='0.05')
    # create nodes for hidden layers
    for NodeList in hidden_layers:
        for n in NodeList:
            nng.node(n, label='', shape='circle', width='0.1')
    # create nodes for output layer
    for n in output_layer:
        nng.node(n, label='', shape='circle', width='0.1')
    # connect input layer to first hidden layer
    for n1 in input_layer:
        for n2 in hidden_layers[0]:
            nng.edge(n1, n2)
    # connect hidden layers d to hidden layer d+1
    for d in range(0, L-3):
        for n1 in hidden_layers[d]:
            for n2 in hidden_layers[d+1]:
                nng.edge(n1, n2)
    # connect output layer
    for n1 in hidden_layers[L-3]:
        for n2 in output_layer:
            nng.edge(n1, n2)
    return nng

Topology = [3, 6, 4, 2]

nn1 = generateNN(Topology)
nn1

Topology = [8, 12, 8, 6, 3]

nn2 = generateNN(Topology)
nn2

Topology = [12, 9, 10, 8, 7, 8, 6, 5, 4, 8, 5, 6, 7, 5, 4, 4, 4, 7, 8, 9]
nn3 = generateNN(Topology)
nn3


I am working on the file "Python/7 Neural Networks/NN-Architecture.ipynb".
The first task is:
Visualize the Architecture of a Neural Network
Can you write Python code for it?


I am working on the file "Python/7 Neural Networks/Neural-Network-Keras.ipynb".
The first task is:
Below, we create a neural network with two hidden layers.
- The first hidden layer has 60 nodes and uses the ReLU function
  as activation function.
- The second hidden layer uses 30 nodes and also uses the ReLu function.
- The output layer uses the softmax function as
  activation function.  This function is defined as follows:
  $$ \sigma(\mathbf{z})i := \frac{e^{z_i}}{\sum\limits{d=0}^{9} e^{z_d}}  $$
  Here, $N$ is the number of output nodes and $z_i$ is the sum of the inputs of the $i$-th output neuron.
  This function guarantees that the outputs of the 10 output nodes can be interpreted as probabilities, since
  there sum is equal to $1$.
- The loss function used is the cross-entropy.
  If a neuron outputs the value $a$, when it should output the value $y \in {0,1}$, the cross entropy cost of
  this neuron is defined as
  $$ C(a, y) := - y \cdot \ln(a) - (1-y)\cdot \ln(1-a). $$
- The cost function is minimized using stochastic gradient descent with a learning rate of $0.3$.
Can you write Python code for it?

model = keras.models.Sequential()
model.add(keras.layers.Dense( 80, activation='relu', input_dim=784))
model.add(keras.layers.Dense( 40, activation='relu'               ))
model.add(keras.layers.Dense( 40, activation='relu'               ))
model.add(keras.layers.Dense( 10, activation='softmax'            ))
model.compile(loss       = 'categorical_crossentropy',
              optimizer  = tf.keras.optimizers.SGD(lr=0.3),
              metrics    = ['accuracy'])
model.summary()

%%time
history = model.fit(X_train, Y_train, validation_data=(X_test, Y_test), epochs=30, batch_size=100, verbose=1)


I am working on the file "Python/7 Neural Networks/Neural-Network-Keras.ipynb".
The first task is:
Building a Neural Network with Keras
Can you write Python code for it?

import gzip
import pickle
import numpy  as np
import keras
import tensorflow as tf


I am working on the file "Python/7 Neural Networks/Neural-Network-Keras.ipynb".
The first task is:
Let us see what we have read:
Can you write Python code for it?

X_train.shape, X_test.shape, Y_train.shape, Y_test.shape


I am working on the file "Python/7 Neural Networks/Neural-Network-Keras.ipynb".
The first task is:
The function $\texttt{vectorized_result}(d)$ converts the digit $d \in {0,\cdots,9}$ and returns a NumPy vector $\mathbf{x}$ of shape $(10, 1)$ such that
$$
\mathbf{x}[i] =
\left{
  \begin{array}{ll}
     1 & \mbox{if $i = j$;} \
     0 & \mbox{otherwise.}
  \end{array}
\right.
$$
This function is used to convert a digit $d$ into the expected output of a neural network that has an output unit for every digit.
Can you write Python code for it?

def vectorized_result(d):
    e    = np.zeros((10, ), dtype=np.float32)
    e[d] = 1.0
    return e


I am working on the file "Python/7 Neural Networks/Reverse-Mode-AD.ipynb".
The first task is:
Given a computational graph CG, the function parents returns a dictionary Parents such that
for every node name n occurring in CG we have that Parents[n] is the set of nodes that are parents
of the node labeled with n.
Can you write Python code for it?

def parents(CG):
    Parents = {}
    for node in CG:
        match node:
            case (p, _, a):
                add_to_dictionary(Parents, a, p)
            case (p, _, a1, a2):
                add_to_dictionary(Parents, a1, p)
                add_to_dictionary(Parents, a2, p)
    return Parents

parents(CG)

def node_dictionary(CG):
    D = {}
    for node in CG:
        name    = node[0]
        D[name] = node
    return D

node_dictionary(CG)


I am working on the file "Python/7 Neural Networks/Reverse-Mode-AD.ipynb".
The first task is:
Reverse Mode Automatic Differentiation
We demonstrate reverse mode AD with the function
$$ f(x_1, x_2) = \sin(x_1 + x_2) \cdot \cos(x_1 - x_2) + (x_1 + x_2) \cdot (x_1 - x_2) $$
To compute the function step by step, we introduce the following auxiliary variables:
* $v_1 := x_1 + x_2$,
* $v_2 := x_1 - x_2$,
* $v_3 := \sin(v_1)$,
* $v_4 := \cos(v_2)$,
* $v_5 := v_3 \cdot v_4$,
* $v_6 := v_1 \cdot v_2$,
* $y   := v_5 + v_6$.
Can you write Python code for it?


I am working on the file "Python/7 Neural Networks/Reverse-Mode-AD.ipynb".
The first task is:
The computational graph GC defined below implements the function f that is defined above.
Can you write Python code for it?


I am working on the file "Python/7 Neural Networks/Reverse-Mode-AD.ipynb".
The first task is:
The function eval_graph takes two arguments:
* CG is a computational graph,
* Values is a dictionary assigning values to variable names.
Can you write Python code for it?

def eval_graph(CG, Values):
    for node in CG:
        match node:
            case (v, ):
                pass
            case (v, r):
                Values[v] = r
            case (v, '+', a1, a2):
                Values[v] = Values[a1] + Values[a2]
            case (v, '-', a1, a2):
                Values[v] = Values[a1] - Values[a2]
            case (v, '*', a1, a2):
                Values[v] = Values[a1] * Values[a2]
            case (v, '/', a1, a2):
                Values[v] = Values[a1] / Values[a2]
            case (v, 'sqrt', a):
                Values[v] = math.sqrt(Values[a])
            case (v, 'exp', a):
                Values[v] = math.exp(Values[a])
            case (v, 'log', a):
                Values[v] = math.log(Values[a])
            case (v, 'sin', a):
                Values[v] = math.sin(Values[a])
            case (v, 'cos', a):
                Values[v] = math.cos(Values[a])
            case (v, 'atan', a):
                Values[v] = math.atan(Values[a])
    return Values['y']

eval_graph(CG, { 'x1': math.pi/4, 'x2': math.pi/4 })

def add_to_dictionary(D, key, value):
    if key in D:
        D[key] |= { value }
    else:
        D[key]  = { value }


I am working on the file "Python/7 Neural Networks/Reverse-Mode-AD.ipynb".
The first task is:
The function partial_derivative takes three arguments:
* Node is a computational node,
* arg is the name of a node occurring as argument in Node,
* Values is a dictionary that stores a value for every node name.
The function computes the partial derivative of Node w.r.t. arg.
Can you write Python code for it?

def partial_derivative(Node, arg, Values):
    match Node:
        case n, '+', a1, a2:
            if arg == a1 == a2:
                return 2
            if arg == a1 or arg == a2:
                return 1
            else:
                assert False, f'partial_derivative({Node}, {arg})'
        case n, '-', a1, a2:
            if arg == a1 == a2:
                return 0
            if arg == a1:
                return 1
            if arg == a2:
                return -1
            else:
                assert False, f'partial_derivative({Node}, {arg})'
        case n, '*', a1, a2:
            if arg == a1 == a2:
                return 2 * Values[a1]
            if arg == a1:
                return Values[a2]
            if arg == a2:
                return Values[a1]
            else:
                assert False, f'partial_derivative({Node}, {arg})'
        case n, '/', a1, a2:
            if arg == a1 == a2:
                return 0
            if arg == a1:
                return 1 / Values[a2]
            if arg == a2:
                return -Values[a1] / Values[a2] ** 2
            else:
                assert False, f'partial_derivative({Node}, {arg})'
        case n, 'sqrt', a:
            return 0.5 / math.sqrt(Values[a])
        case n, 'exp', a:
            return math.exp(Values[a])
        case n, 'log', a:
            return 1 / Values[a]
        case n, 'sin', a:
            return math.cos(Values[a])
        case n, 'cos', a:
            return -math.sin(Values[a])
        case n, 'atan', a:
            return 1 / (1 + Values[a]**2)

def adjoints(CG, Values):
    eval_graph(CG, Values)
    NodeDict = node_dictionary(CG)
    Parents  = parents(CG)
    n        = len(CG)
    Adjoints = {}
    Adjoints['y'] = 1
    for Node in reversed(CG[:-1]):
        name   = Node[0]
        result = 0
        for parent_name in Parents[name]:
            parent_node = NodeDict[parent_name]
            result += Adjoints[parent_name] * partial_derivative(parent_node, name, Values)
       ...


I am working on the file "Python/70_Data_Augmentation.ipynb".
The first task is:
At this point we select the images we want to work with, skip the following cell if you want to work with the original data.
Can you write Python code for it?


I am working on the file "Python/70_Data_Augmentation.ipynb".
The first task is:
Augmentation using spatial transformations
We next illustrate the generation of images by specifying a list of transformation parameter values representing a sampling of the transformation's parameter space.
The code below is agnostic to the specific transformation and it is up to the user to specify a valid list of transformation parameters (correct number of parameters and correct order). To learn more about the spatial transformations supported by SimpleITK you can explore the Transforms notebook.
In most cases we can easily specify a regular grid in parameter space by specifying ranges of values for each of the parameters. In some cases specifying parameter values may be less intuitive (i.e. versor representation of rotation).
Utility methods
Utilities for sampling a parameter space using a regular grid in a convenient manner (special care for 3D similarity).
Can you write Python code for it?

def parameter_space_regular_grid_sampling(*transformation_parameters):
    """
    Create a list representing a regular sampling of the parameter space.
    Args:
        *transformation_paramters : two or more numpy ndarrays representing parameter values. The order
                                    of the arrays should match the ordering of the SimpleITK transformation
                                    parametrization (e.g. Similarity2DTransform: scaling, rotation, tx, ty)
    Return:
        List of lists representing the regular grid sampling.

    Examples:
        #parametrization for 2D translation transform (tx,ty): [[1.0,1.0], [1.5,1.0], [2.0,1.0]]
        >>>> parameter_space_regular_grid_sampling(np.linspace(1.0,2.0,3), np.linspace(1.0,1.0,1))
    """
    return [
        [p.item() for p in parameter_values]
        for parameter_values in np.nditer(np.meshgrid(*transformation_parameters))
    ]

def similarity3D_parameter_space_regular_sampling(
    thetaX, thetaY, thetaZ, tx, ty, tz, scale
):
    """
    Create a list representing a regular sampling of the 3D similarity transformation parameter space. As the
    SimpleITK rotation parametrization uses the vector portion of a versor we don't have an
    intuitive way of specifying rotations. We therefor use the ZYX Euler angle parametrization and convert to
    versor.
    Args:
        thetaX, thetaY, thetaZ: numpy ndarrays with the Euler angle values to use, in radians.
        tx, ty, tz: numpy ndarrays with the translation values to use in mm.
        scale: numpy array with the scale values to use.
    Return:
        List of lists representing the parameter space sampling (vx,vy,vz,tx,ty,tz,s).
    """
    return [
        list(eul2quat(parameter_values[0], parameter_values[1], parameter_values[2]))
        + [p.item() for p in parameter_values[3:]]
        for parameter_values in np.nditer(
            np.meshgrid(thetaX, thetaY, thetaZ, tx, ty, tz, scale)
        )
    ]

def similarity3D_parameter_space_random_sampling(
    thetaX,...


I am working on the file "Python/70_Data_Augmentation.ipynb".
The first task is:
Create reference domain
All input images will be resampled onto the reference domain.
This domain is defined by two constraints: the number of pixels per dimension and the physical size we want the reference domain to occupy. The former is associated with the computational constraints of deep learning where using a small number of pixels is desired. The later is associated with the SimpleITK concept of an image, it occupies a  region in physical space which should be large enough to encompass the object of interest.
Can you write Python code for it?

dimension = data[0].GetDimension()

# Physical image size corresponds to the largest physical size in the training set, or any other arbitrary size.
reference_physical_size = np.zeros(dimension)
for img in data:
    reference_physical_size[:] = [
        (sz - 1) * spc if sz * spc > mx else mx
        for sz, spc, mx in zip(img.GetSize(), img.GetSpacing(), reference_physical_size)
    ]

# Create the reference image with a zero origin, identity direction cosine matrix and dimension
reference_origin = np.zeros(dimension)
reference_direction = np.identity(dimension).flatten()

# Select arbitrary number of pixels per dimension, smallest size that yields desired results
# or the required size of a pretrained network (e.g. VGG-16 224x224), transfer learning. This will
# often result in non-isotropic pixel spacing.
reference_size = [128] * dimension
reference_spacing = [
    phys_sz / (sz - 1) for sz, phys_sz in zip(reference_size, reference_physical_size)
]

# Another possibility is that you want isotropic pixels, then you can specify the image size for one of
# the axes and the others are determined by this choice. Below we choose to set the x axis to 128 and the
# spacing set accordingly.
# Uncomment the following lines to use this strategy.
# reference_size_x = 128
# reference_spacing = [reference_physical_size[0]/(reference_size_x-1)]*dimension
# reference_size = [int(phys_sz/(spc) + 1) for phys_sz,spc in zip(reference_physical_size, reference_spacing)]

reference_image = sitk.Image(reference_size, data[0].GetPixelIDValue())
reference_image.SetOrigin(reference_origin)
reference_image.SetSpacing(reference_spacing)
reference_image.SetDirection(reference_direction)

# Always use the TransformContinuousIndexToPhysicalPoint to compute an indexed point's physical coordinates as
# this takes into account size, spacing and direction cosines. For the vast majority of images the direction
# cosines are the identity matrix, but when this isn't the case simply multiplying the central index by the
# spacing will not yield...


I am working on the file "Python/70_Data_Augmentation.ipynb".
The first task is:
Data generation
Once we have a reference domain we can augment the data using any of the SimpleITK global domain transformations. In this notebook we use a similarity transformation (the generate_images function is agnostic to this specific choice).
Note that you also need to create the labels for your augmented images. If these are just classes then your processing is minimal. If you are dealing with segmentation you will also need to transform the segmentation labels so that they match the transformed image. The following function easily accommodates for this, just provide the labeled image as input and use the sitk.sitkNearestNeighbor interpolator so that you do not introduce labels that were not in the original segmentation.
Can you write Python code for it?

def augment_images_spatial(
    original_image,
    reference_image,
    T0,
    T_aug,
    transformation_parameters,
    output_prefix,
    output_suffix,
    interpolator=sitk.sitkLinear,
    default_intensity_value=0.0,
    additional_image_information=None,
):
    """
    Generate the resampled images based on the given transformations.
    Note: Images are written to disk with the useCompression flag
          set to true. This uses the default compression level for the user selected file
          format (via output_suffix).
    Args:
        original_image (SimpleITK image): The image which we will resample and transform.
        reference_image (SimpleITK image): The image onto which we will resample.
        T0 (SimpleITK transform): Transformation which maps points from the reference image coordinate system
            to the original_image coordinate system.
        T_aug (SimpleITK transform): Map points from the reference_image coordinate system back onto itself using the
               given transformation_parameters. The reason we use this transformation as a parameter
               is to allow the user to set its center of rotation to something other than zero.
        transformation_parameters (List of lists): parameter values which we use T_aug.SetParameters().
        output_prefix (string): output file name prefix (file name: output_prefix_p1_p2_..pn_.output_suffix).
        output_suffix (string): output file name suffix (file name: output_prefix_p1_p2_..pn_.output_suffix), also determines
                                the file formt usef for writing to disk.
        interpolator: One of the SimpleITK interpolators.
        default_intensity_value: The value to return if a point is mapped outside the original_image domain.
        additional_image_information: (Iterable([SimpleITK.Image, Interpolator, default_intensity_value])): Apply the same
                                     transformations to this set of images using the given interpolators and intensity values.
                ...


I am working on the file "Python/70_Data_Augmentation.ipynb".
The first task is:
Histogram equalization, increasing the entropy, of images prior to using deep learning is a common preprocessing step. Unfortunately, ITK and consequentially SimpleITK do not have a histogram equalization filter.
The following cell illustrates this functionality for all integer scalar SimpleITK images (2D,3D).
Can you write Python code for it?

def histogram_equalization(
    image, min_target_range=None, max_target_range=None, use_target_range=True
):
    """
    Histogram equalization of scalar images whose single channel has an integer
    type. The goal is to map the original intensities so that resulting
    histogram is more uniform (increasing the image's entropy).
    Args:
        image (SimpleITK.Image): A SimpleITK scalar image whose pixel type
                                 is an integer (sitkUInt8,sitkInt8...
                                 sitkUInt64, sitkInt64).
        min_target_range (scalar): Minimal value for the target range. If None
                                   then use the minimal value for the scalar pixel
                                   type (e.g. 0 for sitkUInt8).
        max_target_range (scalar): Maximal value for the target range. If None
                                   then use the maximal value for the scalar pixel
                                   type (e.g. 255 for sitkUInt8).
        use_target_range (bool): If true, the resulting image has values in the
                                 target range, otherwise the resulting values
                                 are in [0,1].
    Returns:
        SimpleITK.Image: A scalar image with the same pixel type as the input image
                         or a sitkFloat64 (depending on the use_target_range value).
    """
    arr = sitk.GetArrayViewFromImage(image)

    i_info = np.iinfo(arr.dtype)
    if min_target_range is None:
        min_target_range = i_info.min
    else:
        min_target_range = np.max([i_info.min, min_target_range])
    if max_target_range is None:
        max_target_range = i_info.max
    else:
        max_target_range = np.min([i_info.max, max_target_range])

    min_val = arr.min()
    number_of_bins = arr.max() - min_val + 1
    # using ravel, not flatten, as it does not involve memory copy
    hist = np.bincount((arr - min_val).ravel(), minlength=number_of_bins)
    cdf = np.cumsum(hist)
    cdf = (cdf - cdf[0]) / (cdf[-1] -...


I am working on the file "Python/70_Data_Augmentation.ipynb".
The first task is:
Optional: create corresponding segmentation reference data
Mimic the data available when performing semantic segmentation, the following cell creates corresponding "reference binary segmentation" data for each of the images.
Can you write Python code for it?

reference_segmentations = [sitk.OtsuThreshold(image, 0, 1) for image in data]


I am working on the file "Python/70_Data_Augmentation.ipynb".
The first task is:
SimpleITK has a sigmoid filter that allows us to map intensities via this nonlinear function to our desired range. Unlike the standard sigmoid settings that are applied when used as an activation function, the sigmoid filter is not necessarily centered on zero and the minimum and maximum output values are not necessarily 0 and 1.
The filter itself is defined as:
$$f(I) = (max_{output} - min_{output}) \frac{1}{1+ e^{-\frac{I-\beta}{\alpha}}} + min_{output}$$
Where $\alpha$ is the curve steepness (the larger the $\alpha$ the steeper the slope, the smaller the $\alpha$ the closer we get to a linear mapping in the output range) and $\beta$ is the intensity value for the sigmoid midpoint.
Can you write Python code for it?

def sigmoid_mapping(
    image, curve_steepness, output_min=0, output_max=1.0, intensity_midpoint=None
):
    """
    Map the image using a sigmoid function.
    Args:
        image (SimpleITK image): scalar input image.
        curve_steepness: Control the sigmoid steepness, the larger the number the steeper the curve.
        output_min: Minimum value for output image, default 0.0 .
        output_max: Maximum value for output image, default 1.0 .
        intensity_midpoint: intensity value defining the sigmoid midpoint (x coordinate), default is the
                            median image intensity.
    Return:
        SimpleITK image with float pixel type.
    """
    if intensity_midpoint is None:
        intensity_midpoint = np.median(sitk.GetArrayViewFromImage(image))

    sig_filter = sitk.SigmoidImageFilter()
    sig_filter.SetOutputMinimum(output_min)
    sig_filter.SetOutputMaximum(output_max)
    sig_filter.SetAlpha(1.0 / curve_steepness)
    sig_filter.SetBeta(float(intensity_midpoint))
    return sig_filter.Execute(sitk.Cast(image, sitk.sitkFloat64))

# Change the order of magnitude of curve steepness [1.0,0.1,0.01] to see the effect of this parameter.
# Also change it from positive to negative.
disp_images(
    [sigmoid_mapping(img, curve_steepness=0.01) for img in data], fig_size=(6, 2)
)


I am working on the file "Python/70_Data_Augmentation.ipynb".
The first task is:
Transferring deformations - exercise for the interested reader
Using SimpleITK we can readily transfer deformations from a spatio-temporal data set to another spatial data set to simulate temporal behavior. Case in point, using a 4D (3D+time) CT of the thorax we can estimate the respiratory motion using non-rigid registration and Free Form Deformation or displacement field transformations. We can then register a new spatial data set to the original spatial CT (non-rigidly) followed by application of the temporal deformations.
Note that unlike the arbitrary spatial transformations we used for data-augmentation above this approach is more computationally expensive as it involves multiple non-rigid registrations. Also note that as the goal is to use the estimated transformations to create plausible deformations you may be able to relax the required registration accuracy.
Augmentation using intensity modifications
SimpleITK has many filters that are potentially relevant for data augmentation via modification of intensities. For example:
* Image smoothing, always read the documentation carefully, similar filters use use different parametrization $\sigma$ vs. variance ($\sigma^2$):
  * Discrete Gaussian
  * Recursive Gaussian
  * Smoothing Recursive Gaussian

Edge preserving image smoothing:
Bilateral image filtering, edge preserving smoothing.

Median filtering

Adding noise to your images:

Additive Gaussian
Salt and Pepper / Impulse
Shot/Poisson

Speckle/multiplicative

Adaptive Histogram Equalization
Can you write Python code for it?

def augment_images_intensity(image_list, output_prefix, output_suffix):
    """
    Generate intensity modified images from the originals.
    Args:
        image_list (iterable containing SimpleITK images): The images which we whose intensities we modify.
        output_prefix (string): output file name prefix (file name: output_prefixi_FilterName.output_suffix).
        output_suffix (string): output file name suffix (file name: output_prefixi_FilterName.output_suffix).
    """

    # Create a list of intensity modifying filters, which we apply to the given images
    filter_list = []

    # Smoothing filters

    filter_list.append(sitk.SmoothingRecursiveGaussianImageFilter())
    filter_list[-1].SetSigma(2.0)

    filter_list.append(sitk.DiscreteGaussianImageFilter())
    filter_list[-1].SetVariance(4.0)

    filter_list.append(sitk.BilateralImageFilter())
    filter_list[-1].SetDomainSigma(4.0)
    filter_list[-1].SetRangeSigma(8.0)

    filter_list.append(sitk.MedianImageFilter())
    filter_list[-1].SetRadius(8)

    # Noise filters using default settings

    # Filter control via SetMean, SetStandardDeviation.
    filter_list.append(sitk.AdditiveGaussianNoiseImageFilter())

    # Filter control via SetProbability
    filter_list.append(sitk.SaltAndPepperNoiseImageFilter())

    # Filter control via SetScale
    filter_list.append(sitk.ShotNoiseImageFilter())

    # Filter control via SetStandardDeviation
    filter_list.append(sitk.SpeckleNoiseImageFilter())

    filter_list.append(sitk.AdaptiveHistogramEqualizationImageFilter())
    filter_list[-1].SetAlpha(1.0)
    filter_list[-1].SetBeta(0.0)

    filter_list.append(sitk.AdaptiveHistogramEqualizationImageFilter())
    filter_list[-1].SetAlpha(0.0)
    filter_list[-1].SetBeta(1.0)

    aug_image_lists = []  # Used only for display purposes in this notebook.
    for i, img in enumerate(image_list):
        aug_image_lists.append([f.Execute(img) for f in filter_list])
        for aug_image, f in zip(aug_image_lists[-1], filter_list):
           ...


I am working on the file "Python/70_Data_Augmentation.ipynb".
The first task is:
While the sigmoid mapping visually appears to work as expected, it is always good to "trust but verify". In the next cell we create a 1D image and plot the resulting sigmoid mapped values, ensuring that what we expect is indeed what is happening. This also allows us to see the effects in a more controlled manner.
To see the effects of various settings combinations try:
Setting the curve_steepness to [1.0,0.1,0.01, -0.01, -0.1, -1.0]
Setting the intensity_midpoint to [-50, 0, 50].
Can you write Python code for it?

import matplotlib.pyplot as plt

# Create a 1D image with values in [-100,100].
arr_x = np.array([list(range(-100, 101))])
image1D = sitk.GetImageFromArray(arr_x)
plt.figure()
plt.plot(
    arr_x.ravel(),
    sitk.GetArrayViewFromImage(
        sigmoid_mapping(image1D, curve_steepness=1.0, intensity_midpoint=0)
    ).ravel(),
);


I am working on the file "Python/Advancd Python.ipynb".
The first task is:

How decorators work?

    When @my_decorator is called it goes to the __init__ function and executes it. We can also run the original function inside the init as function declaration is complete before executing the __init__. After the init function the original function is decorated that is replaced with the __call__ function so when we run original function the code of __call__ is executed.

Can you write Python code for it?

def foo():
    pass
foo = staticmethod(foo)

# You can replace the above code using decorator as
@staticmethod
def foo():
    pass


I am working on the file "Python/Advancd Python.ipynb".
The first task is:
Decorators
Simple alternative to metaclasses. The '@' indicates the application of decorator.
Can you write Python code for it?

@myDecorator
def aFunction():
    print('Inside aFunction')

class my_decorator(object):
    def __init__(self, f):
        print("inside my_decorator.__init__()")
        f()
    def __call__(self):
        print("inside my_decorator.__call__()")

@my_decorator
def aFunction():
    print("inside aFunction()")

# The part before it would run before going to the next statement
# as we used decorators in which we defined the __init__ function
print("Finished decorating aFunction()")

aFunction()
# Decorators replace the original function object with the __call__


I am working on the file "Python/Bsignedrank.ipynb".
The first task is:
... and on the right
Can you write Python code for it?

samples = bt.signrank_MC(scores, rope=0.01,  prior_strength=0.6, prior_place=bt.RIGHT)
fig = bt.plot_posterior(samples,names)
plt.show()


I am working on the file "Python/Bsignedrank.ipynb".
The first task is:
Checking sensitivity to the prior
To check the effect of the prior, let us a put a greater prior on the left.
Can you write Python code for it?

samples = bt.signrank_MC(scores, rope=0.01,  prior_strength=0.6, prior_place=bt.LEFT)
fig = bt.plot_posterior(samples,names)
plt.show()


I am working on the file "Python/Bsignedrank.ipynb".
The first task is:
The first value (left) is the probability that the first classifier (the left column of x) has a higher score than the second (or that the differences are negative, if x is given as a vector).
In the above case, the right (AODE) performs worse than naive Bayes with a probability of 0.88, and they are practically equivalent with a probability of 0.12.
If we add arguments verbose and names, the function also prints out the probabilities.
Can you write Python code for it?

left, within, right = bt.signrank(scores, rope=0.01, verbose=True, names=names)


I am working on the file "Python/Bsignedrank.ipynb".
The first task is:
The prior with a strength of 1 has negligible effect. Only a much stronger prior on the left would shift the probabilities toward NBC:
Can you write Python code for it?

samples = bt.signrank_MC(scores, rope=0.01,  prior_strength=6, prior_place=bt.LEFT)
fig = bt.plot_posterior(samples,names)
plt.show()


I am working on the file "Python/Bsigntest.ipynb".
The first task is:
... and on the right
Can you write Python code for it?

samples = bt.signtest_MC(scores, rope=0.01,  prior_strength=1, prior_place=bt.RIGHT)
fig = bt.plot_posterior(samples,names)
plt.show()


I am working on the file "Python/Bsigntest.ipynb".
The first task is:
Checking sensitivity to the prior
To check the effect of the prior, let us a put a greater prior on the left.
Can you write Python code for it?

samples = bt.signtest_MC(scores, rope=0.01,  prior_strength=1, prior_place=bt.LEFT)
fig = bt.plot_posterior(samples,names)
plt.show()


I am working on the file "Python/Bsigntest.ipynb".
The first task is:
Functions in the module accept the following arguments.

x: a 2-d array with scores of two models (each row corresponding to a data set) or a vector of differences.
rope: the region of practical equivalence. We consider two classifiers equivalent if the difference in their performance is smaller than rope.
prior_strength: the prior strength for the Dirichlet distribution. Default is 1.
prior_place: the region into which the prior is placed. Default is bayesiantests.ROPE, the other options are bayesiantests.LEFT and bayesiantests.RIGHT.
nsamples: the number of Monte Carlo samples used to approximate the posterior.
names: the names of the two classifiers; if x is a vector of differences, positive values mean that the second (right) model had a higher score.

Summarizing probabilities
Function signtest(x, rope, prior_strength=1, prior_place=ROPE, nsamples=50000, verbose=False, names=('C1', 'C2')) computes the Bayesian sign test and returns the probabilities that the difference (the score of the first classifier minus the score of the first) is negative, within rope or positive.
Can you write Python code for it?

import bayesiantests as bt
left, within, right = bt.signtest(scores, rope=0.01)
print(left, within, right)


I am working on the file "Python/Bsigntest.ipynb".
The first task is:
The posterior distribution can be plotted out:
1. using the function signtest_MC(x, rope, prior_strength=1, prior_place=ROPE, nsamples=50000) we generate the samples of the posterior
2. using the function plot_posterior(samples,names=('C1', 'C2')) we then plot the posterior in the probability simplex
Can you write Python code for it?

%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams['figure.facecolor'] = 'black'

samples = bt.signtest_MC(scores, rope=0.01)

fig = bt.plot_posterior(samples,names)
plt.savefig('triangle.png',facecolor="black")
plt.show()


I am working on the file "Python/Chapter-02/Fibonacci-Numbers.ipynb".
The first task is:
It looks like the Fibonacci numbers grow exponentially.  Let us confirm this hypothesis by plotting the logarithm of these numbers.
Can you write Python code for it?

import math

X = X[1:]
Y = Y[1:]
logY = [math.log(y) for y in Y]
sns.set_style('darkgrid')
plt.figure(figsize=(15, 10))
plt.plot(X, logY, 'bo')
plt.xticks(X)
plt.yticks(list(range(17)))
plt.xlabel('n')
plt.ylabel('ln(F(n))')
plt.title('The Logarithms of the Fibonacci Numbers')
plt.show()


I am working on the file "Python/Chapter-02/Fibonacci-Numbers.ipynb".
The first task is:
It seems that the Fibonacci numbers grow pretty fast. Let us plot these numbers to get a better understanding of their growth.
Can you write Python code for it?

import matplotlib.pyplot as plt
import seaborn           as sns

m = 34
X = []
Y = []
for n in range(m):
    X.append(n)
    Y.append(fibonacci(n))
sns.set_style('darkgrid')
plt.figure(figsize=(15, 10))
plt.plot(X, Y, 'bo')
plt.xticks(X)
plt.yticks([y * 2e5 for y in range(19)])
plt.xlabel('n')
plt.ylabel('F(n)')
plt.title('The Fibonacci Numbers')
plt.show()


I am working on the file "Python/Chapter-02/Fibonacci-Numbers.ipynb".
The first task is:
Now it is even possible to compute the $100,000^\mbox{th}$ Fibonnaci number.
Can you write Python code for it?


I am working on the file "Python/Chapter-02/Fibonacci-Numbers.ipynb".
The first task is:
The Fibonacci Numbers
The Fibonacci numbers $F_n$ are defined by induction for all $n\in\mathbb{N}$:
  - $F_0 := 0$,
  - $F_1 := 1$,
  - $F_{n+2} = F_{n+1} + F_n$ for all $n \in \mathbb{N}$.
Given a natural number n, the function fibonacci(n) computes the number $F_n$.
Can you write Python code for it?

def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n-1) + fibonacci(n-2)

[ (n,fibonacci(n)) for n in range(19) ]


I am working on the file "Python/Chapter-02/Fibonacci-Numbers.ipynb".
The first task is:
The computation tree for the computation of fibonacci(6) shows the reason for the inefficiency:
* fibonacci(5) is computed once,
* fibonacci(4) is computed 2 times,
* fibonacci(3) is computed 3 times,
* fibonacci(2) is computed 5 times,
* fibonacci(1) is computed 8 times, and
* fibonacci(0) is computed 5 times.
If we want to compute the Fibonacci numbers efficiently, we must not compute the value fibonacci(n) for a given n more than once.  The easiest way to achieve this is by storing the Fibnacci numbers in a list L.  In the implementation below, L[n] stores the $n$-th Fibonacci number.
Can you write Python code for it?

def fibonacci_mem(n):
    if n <= 1:
        return n
    L    = [0 for k in range(n+1)]
    L[0] = 0
    L[1] = 1
    for k in range(2, n+1):
        L[k] = L[k-1] + L[k-2]
    return L[n]


I am working on the file "Python/Chapter-02/Fibonacci-Numbers.ipynb".
The first task is:
The logarithmic plot confirms this.  In order to investigate the reason for this exponential growth, we compute the computation tree.  This tree shows the recursive invocations of the function.
Can you write Python code for it?

import graphviz as gv

class ComputationTree:
    def __init__(self, arg, value, left=None, right=None):
        self.mArg   = arg
        self.mValue = value
        self.mLeft  = left
        self.mRight = right

def isLeaf(self):
    return self.mLeft == None and self.mRight == None

ComputationTree.isLeaf = isLeaf
del isLeaf

def toDot(self):
    ComputationTree.sCounter = 0 # static variable of the class ComputationTree
    dot = gv.Digraph(node_attr={'shape': 'record', 'style': 'rounded'})
    NodeDict = {}
    self._assignIDs(NodeDict)
    for n, t in NodeDict.items():
        dot.node(str(n), label='{' + str(t.mArg) + '|' + str(t.mValue) + '}')
        if t.mLeft != None and t.mRight != None:
            dot.edge(str(n), str(t.mLeft .mID))
            dot.edge(str(n), str(t.mRight.mID))
    return dot

ComputationTree.toDot = toDot
del toDot

def _assignIDs(self, NodeDict):
    ComputationTree.sCounter += 1
    self.mID = ComputationTree.sCounter
    NodeDict[self.mID] = self
    if self.isLeaf():
        return
    self.mLeft ._assignIDs(NodeDict)
    self.mRight._assignIDs(NodeDict)

ComputationTree._assignIDs = _assignIDs
del _assignIDs

def fibonacci_tree(n):
    if n <= 1:
        return ComputationTree(n, n)
    C1 = fibonacci_tree(n-1)
    C2 = fibonacci_tree(n-2)
    return ComputationTree(n, C1.mValue + C2.mValue, C1, C2)

t = fibonacci_tree(6)
t.toDot()


I am working on the file "Python/Chapter-02/Fibonacci-Numbers.ipynb".
The first task is:
This plot looks linear and confirms our hypothesis that these numbers grow exponentially.
Computing the Fibonacci numbers took quite long.  Lets measure these times and plot them.
Can you write Python code for it?

import time

m = 36
Y = []
X = list(range(m))
for n in range(m):
    start = time.time()
    print(f'fib({n}) = {fibonacci(n)}')
    stop  = time.time()
    print(stop - start)
    Y.append(stop - start)

sns.set_style('darkgrid')
plt.figure(figsize=(15, 10))
plt.plot(X, Y, 'bo')
plt.xticks(X)
plt.xlabel('n')
plt.ylabel('time in seconds')
plt.title('Time to Compute the Fibonacci Numbers')
plt.show()


I am working on the file "Python/Chapter-02/Power.ipynb".
The first task is:
Efficient Computation of Powers
The function power takes two natural numbers $m$ and $n$ and computes $m^n$.  Our first implementation is inefficient and takes $n-1$ multiplication to compute $m^n$.
Can you write Python code for it?

def power(m, n):
    r = 1
    for i in range(n):
        r *= m
    return r

power(2, 3), power(3, 2)

%%time
p = power(3, 500000)

p


I am working on the file "Python/Chapter-02/Power.ipynb".
The first task is:
Next, we try a recursive implementation that is based on the following two equations:
1. $m^0 = 1$
2. $m^n = \left{\begin{array}{ll}
         m^{n//2} \cdot m^{n//2}          & \mbox{if $n$ is even};    \
         m^{n//2} \cdot m^{n//2} \cdot m  & \mbox{if $n$ is odd}.
         \end{array}
         \right.
         $
Can you write Python code for it?

def power(m, n):
    if n == 0:
        return 1
    p = power(m, n // 2)
    if n % 2 == 0:
        return p * p
    else:
        return p * p * m

%%time
p = power(3, 500000)


I am working on the file "Python/Chapter-02/Power.ipynb".
The first task is:
Next, we try a recursive implementation that is based on the following two equations:
1. $m^0 = 1$
2. $m^n = \left{\begin{array}{ll}
         m^{n/2} \cdot m^{n/2}          & \mbox{if $n$ is even};    \
         m^{n/2} \cdot m^{n/2} \cdot m  & \mbox{if $n$ is odd}.
         \end{array}
         \right.
         $
Can you write Python code for it?


I am working on the file "Python/Chapter-04/Digit-Recognition.ipynb".
The first task is:
For example, the distance between the first two images of the training set is computed as follows:
Can you write Python code for it?


I am working on the file "Python/Chapter-04/Digit-Recognition.ipynb".
The first task is:
Given a list $L$ of digits, the function $\texttt{maxCounts}(L)$ returns a pair $(d, p)$ where $d$ is the digit that occurs most frequently in $L$
and $p$ is the percentage of occurrences of $d$ in $L$.  For example, we have
$$ \texttt{maxCounts}([5,2,3,5,2,5,6,5,7,8]) = (5, 0.4)  $$
because the digit $5$ is the most frequent digit in the list $[5,2,3,5,2,5,6,5,7,8]$ and $40$% of the digits in this list are fives.
Can you write Python code for it?

def maxCount(L):
    Frequencies         = {}    # number of occurrences for each digit
    most_frequent       = L[0]  # most frequent digit so far
    most_frequent_count = 1     # number of occurrences of most frequent digit
    for d in L:
        if d in Frequencies:
            Frequencies[d] += 1
        else:
            Frequencies[d]  = 1
        if Frequencies[d] > most_frequent_count:
            most_frequent       = d
            most_frequent_count = Frequencies[d]
    return most_frequent, most_frequent_count / len(L)

maxCount([3, 3, 4, 2, 1, 2, 3, 2, 5, 3])


I am working on the file "Python/Chapter-04/Digit-Recognition.ipynb".
The first task is:
Given an image of a digit stored in the vector $\mathbf{x}$ and a number of neighbours $k$, the function $\texttt{digit}(\mathbf{x}, k)$ computes those
$k$ images in the training set X_train that are closest to the image $\mathbf{x}$.  Here
closeness of images is defined in terms of the Euclidean distance of the vectors that store the
images.  From these $k$ images of the training set the function chooses the digit that occurs most frequently.  It returns a pair $(d, p)$ where $d$ is the digit that is most frequently occurring in the list of $k$ neighbours and $p$ is the percentage of images in the $k$ neighbours of $\mathbf{x}$ that show
the digit $d$.
Can you write Python code for it?

def digit(x, k):
    n          = X_train.shape[0]  # number of all training images
    Distances  = [ (distance(X_train[i, :], x), i) for i in range(n)]
    Neighbours = [ Y_train[i] for _, i in sorted(Distances)]
    return maxCount(Neighbours[:k])


I am working on the file "Python/Chapter-04/Digit-Recognition.ipynb".
The first task is:
Given two arrays $\mathbf{x}$ and $\mathbf{y}$ of the same dimension $n$, the function $\texttt{distance}(\mathbf{x}, \mathbf{y})$ computes the
Euclidean distance between $\mathbf{x}$ and $\mathbf{y}$.  This distance is defined as follows:
$$ \sqrt{\sum\limits_{i=1}^n (x_i - y_i)^2} $$
Can you write Python code for it?

def distance(x, y):
    return np.sqrt(np.sum((x - y)**2))


I am working on the file "Python/Chapter-04/Digit-Recognition.ipynb".
The first task is:
Let us check how much memory is used at the moment.  The function call below prints all the memory of our machine that is currently used.
Can you write Python code for it?

def check(n, k):
    d, p = digit(X_test[n, :], k)
    if d == Y_test[n]:
        return 0
    else:
        print(f"\nImage number {n} wrongly identified: I guessed a {d}, but it's a {Y_test[n]}.")
        show_image(n)
        return 1

%%time
errors = 0
for n in range(10_000):
    errors += check(n, 7)
print(f'There were {errors} errors out of 10000 images.')


I am working on the file "Python/Chapter-04/Digit-Recognition.ipynb".
The first task is:
Let us check what we have read:
Can you write Python code for it?

X_train.shape, X_test.shape, Y_train.shape, Y_test.shape


I am working on the file "Python/Chapter-04/Digit-Recognition.ipynb".
The first task is:
Let us inspect the first hand written image of a digit.
Can you write Python code for it?


I am working on the file "Python/Chapter-04/Digit-Recognition.ipynb".
The first task is:
The distance between the 9th and the 15th image should be smaller, because both of these images show the digit $1$ and hence these images are quite similar.
This similarity results in a smaller distance between these images.
Can you write Python code for it?


I am working on the file "Python/Chapter-04/Digit-Recognition.ipynb".
The first task is:
The function $\texttt{show_digit}(\texttt{row}, \texttt{columns}, \texttt{offset})$
shows $\texttt{row} \cdot \texttt{columns}$ images of the training data.  The first image shown is the image at index $\texttt{offset}$.
Can you write Python code for it?

def show_digits(rows, columns, offset=0):
    f, axarr = plt.subplots(rows, columns)
    for r in range(rows):
        for c in range(columns):
            i     = r * columns + c + offset
            image = 1 - X_train[i, :]
            image = np.reshape(image, (28, 28))
            axarr[r, c].imshow(image, cmap="gray")
            axarr[r, c].axis('off')
    plt.savefig("digits.pdf")
    plt.show()


I am working on the file "Python/Chapter-04/Digit-Recognition.ipynb".
The first task is:
This function performs $k$-nearest neighbour classification for the $n$-th image of the test set.  It also prints the image.
Can you write Python code for it?

def test(n, k):
    print(f'Testing image {n}:')
    show_image(n)
    d, p = digit(X_test[n, :], k)
    print(f'I believe with a certainty of {p * 100}% that the image shows the digit {d}.')

test(2, 13)


I am working on the file "Python/Chapter-04/Digit-Recognition.ipynb".
The first task is:
We take a look at the first 24 images.
Can you write Python code for it?


I am working on the file "Python/Chapter-04/Insertion-Sort.ipynb".
The first task is:
The auxiliary function insert is specified as follows:

$\mathtt{insert}(x,[]) = [x]$
$x \preceq y \rightarrow \mathtt{insert}\bigl(x, [y] + R\bigr) = [x,y] + R$
$\neg x \preceq y \rightarrow
      \mathtt{insert}\bigl(x, [y] + R\bigr) = [y] + \mathtt{insert}(x,R)$

Again, a recursive implementation is straightforward.
Can you write Python code for it?

def insert(x, L):
    if L == []:
        return [x]
    y, *R = L
    if x <= y:
        return [x, y] + R
    else:
        return [y] + insert(x, R)

insert(5, [1, 3, 4, 7, 9])

sort([7, 8, 11, 12, 2, 5, 3, 7, 9])


I am working on the file "Python/Chapter-04/Merge-Sort-Array.ipynb".
The first task is:
The function isOrdered(L) checks that the list L is sorted in ascending order.
Can you write Python code for it?

def isOrdered(L):
    for i in range(len(L) - 1):
        assert L[i] <= L[i+1], f'{L} not sorted at index {i}'


I am working on the file "Python/Chapter-04/Merge-Sort-Array.ipynb".
The first task is:
The function merge takes five arguments.
  - L      is a list,
  - start  is an integer such that $\texttt{start}  \in {0, \cdots, \texttt{len}(L)-1 }$,
  - middle is an integer such that $\texttt{middle} \in {0, \cdots, \texttt{len}(L)-1 }$,
  - end    is an integer such that $\texttt{end}    \in {0, \cdots, \texttt{len}(L)-1 }$,
  - A      is a list of the same length as L.
Furthermore, the indices start, middle, and end have to satisfy the following:
$$ 0 \leq \texttt{start} < \texttt{middle} < \texttt{end} \leq \texttt{len}(L) $$
The function assumes that the sublists L[start:middle] and L[middle:end] are already sorted.
The function merges these sublists so that when the call returns the sublist L[start:end]
is sorted.  The last argument A is used as auxiliary memory.
Can you write Python code for it?

def merge(L, start, middle, end, A):
    A[start:end] = L[start:end]
    idx1 = start
    idx2 = middle
    i    = start
    while idx1 < middle and idx2 < end:
        if A[idx1] <= A[idx2]:
            L[i]  = A[idx1]
            idx1 += 1
        else:
            L[i]  = A[idx2]
            idx2 += 1
        i += 1
    if idx1 < middle:
        L[i:end] = A[idx1:middle]
    if idx2 < end:
        L[i:end] = A[idx2:end]

L = [7, 8, 11, 12, 2, 5, 3, 7, 9, 3, 2]
sort(L)
L


I am working on the file "Python/Chapter-04/Merge-Sort-Array.ipynb".
The first task is:
The function mergeSort is called with 4 arguments.
  - The first parameter $\texttt{L}$ is the list that is to be sorted.
    However, the task of $\texttt{mergeSort}$ is not to sort the entire list $\texttt{L}$ but only
    the part of $\texttt{L}$ that is given as
    $$\texttt{L[start:end]}$$
  - Hence, the parameters $\texttt{start}$ and $\texttt{end}$ are indices specifying the
    subarray that needs to be sorted.
  - The final parameter $\texttt{A}$ is used as an auxiliary array.  This array is needed
    as temporary storage and is required to have the same size as the list $\texttt{L}$.
Can you write Python code for it?

def mergeSort(L, start, end, A):
    if end - start < 2:
        return
    middle = (start + end) // 2
    mergeSort(L, start,  middle, A)
    mergeSort(L, middle, end   , A)
    merge(L, start, middle, end, A)


I am working on the file "Python/Chapter-04/Merge-Sort-Array.ipynb".
The first task is:
The predefined function sorted is a lot quicker:
Can you write Python code for it?

%%timeit
k = 1_000_000
L = [ rnd.randrange(2*k) for x in range(k) ]
S = sorted(L)


I am working on the file "Python/Chapter-04/Merge-Sort-Array.ipynb".
The first task is:
We import the class Counter from the module collections.  This module provides us with a dictionary that keeps count
how many times an item occurs in a list.
Can you write Python code for it?

from collections import Counter

Counter(['a', 'b', 'a', 'b', 'c', 'a'])

def demo():
    L = [ rnd.randrange(1, 99+1) for n in range(1, 19+1) ]
    print("L = ", L)
    S = L[:]
    sort(S)
    print("S = ", S)
    print(Counter(L))
    print(Counter(S))
    print(Counter(L) == Counter(S))

demo()


I am working on the file "Python/Chapter-04/Merge-Sort-Iterative.ipynb".
The first task is:
An Iterative Implementation of Merge Sort
The function $\texttt{sort}(L)$ sorts the list $L$ in place using merge sort.
It takes advantage of the fact that, in Python, lists are stored internally as arrays.
The function sort is a wrapper for the function merge_sort.  Its sole purpose is to allocate the auxiliary array A,
which has the same size as the array storing L.
Can you write Python code for it?

def sort(L):
    A = L[:]  # A is a copy of L
    mergeSort(L, A)


I am working on the file "Python/Chapter-04/Merge-Sort-Iterative.ipynb".
The first task is:
Python offers a predefined function sorted that can be used to sort a list of numbers.
Let us see how it compares to our implementation.
Can you write Python code for it?

%%timeit
k = 1_000_000
L = [ rnd.randrange(2*k) for x in range(k) ]
S = sorted(L)

help(sorted)


I am working on the file "Python/Chapter-04/Merge-Sort-Iterative.ipynb".
The first task is:
Testing
Can you write Python code for it?

import random as rnd
from collections import Counter

def demo():
    L = [ rnd.randrange(1, 100) for n in range(1, 20) ]
    print("L = ", L)
    S = L[:]
    sort(S)
    print("S = ", S)
    print(Counter(L))
    print(Counter(S))
    print(Counter(L) == Counter(S))

demo()


I am working on the file "Python/Chapter-04/Merge-Sort-Iterative.ipynb".
The first task is:
The function $\texttt{testSort}(n, k)$ generates $n$ random lists of length $k$, sorts them, and checks whether the output is sorted and contains the same elements as the input.
Can you write Python code for it?

def testSort(n, k):
    for i in range(n):
        L = [ rnd.randrange(2*k) for x in range(k) ]
        oldL = L[:]
        sort(L)
        isOrdered(L)
        sameElements(oldL, L)
        print('.', end='')
    print()
    print("All tests successful!")

%%time
testSort(100, 20000)

%%timeit
k = 1_000_000
L = [ rnd.randrange(2*k) for x in range(k) ]
sort(L)


I am working on the file "Python/Chapter-04/Merge-Sort-Iterative.ipynb".
The first task is:
The function merge takes five arguments.
  - L      is a list,
  - start  is an integer such that $\texttt{start}  \in {0, \cdots, \texttt{len}(L)-1 }$,
  - middle is an integer such that $\texttt{middle} \in {0, \cdots, \texttt{len}(L)-1 }$,
  - end    is an integer such that $\texttt{end}    \in {0, \cdots, \texttt{len}(L)-1 }$,
  - A      is a list of the same length as L.
Furthermore, the indices start, middle and end have to satisfy the following inequations:
$$ 0 \leq \texttt{start} < \texttt{middle} < \texttt{end} \leq \texttt{len}(L) $$
The function assumes that the sublists L[start:middle] and L[middle:end] are already sorted.
The function merges these sublists so that when the call returns the sublist L[start:end]
is sorted.  The last argument A is used as auxiliary memory.
Can you write Python code for it?

def merge(L, start, middle, end, A):
    A[start:end] = L[start:end]
    idx1 = start
    idx2 = middle
    i    = start
    while idx1 < middle and idx2 < end:
        if A[idx1] <= A[idx2]:
            L[i]  = A[idx1]
            idx1 += 1
        else:
            L[i]  = A[idx2]
            idx2 += 1
        i += 1
    if idx1 < middle:
        L[i:end] = A[idx1:middle]
    if idx2 < end:
        L[i:end] = A[idx2:end]


I am working on the file "Python/Chapter-04/Merge-Sort-Iterative.ipynb".
The first task is:
The function mergeSort is called with 2 arguments.
  - The first parameter $\texttt{L}$ is the list that is to be sorted.
  - The second parameter $\texttt{A}$ is used as an auxiliary array.  This array is needed
    as temporary storage and is required to have the same size as the list $\texttt{L}$.
The implementation uses two loops:
* The outer while loop sorts sublists of length n. Before the $\texttt{n}^{\mbox{th}}$ iteration of the outer while
  loop, all sublists of the form L[n*k:n*(k+1)] are sorted.  After the $\texttt{n}^{\mbox{th}}$ iteration, all
  sublists of the form L[2*n*k:2*n*(k+1)] are sorted.
* The inner while loop merges the sublists L[n*k:n*(k+1)] and L[n*(k+1):n*(k+2)] for even values of k.
Can you write Python code for it?

def mergeSort(L, A):
    n = 1
    while n < len(L):
        k = 0
        while n * k + n < len(L):
            top = min(n * k + 2 * n, len(L))
            merge(L, n * k, n * k + n, top, A)
            k += 2
        n *= 2


I am working on the file "Python/Chapter-04/Merge-Sort-Iterative.ipynb".
The first task is:
The function sameElements(L, S) returns Trueif the lists L and S contain the same elements and, furthermore, each
element $x$ occurring in L occurs in S the same number of times it occurs in L.
Can you write Python code for it?

def sameElements(L, S):
    assert Counter(L) == Counter(S)


I am working on the file "Python/Chapter-04/Merge-Sort.ipynb".
The first task is:
Due to the recursion limit in Python we can only sort lists of length 2000.
Can you write Python code for it?


I am working on the file "Python/Chapter-04/Merge-Sort.ipynb".
The first task is:
The function $\texttt{testSort}(n, k)$ generates $n$ random lists of length $k$, sorts them, and checks whether the output is sorted and contains the same elements as the input.
Can you write Python code for it?

def testSort(n, k):
    for i in range(n):
        L = [ rnd.randrange(2*k) for x in range(k) ]
        S = sort(L)
        isOrdered(S)
        sameElements(L, S)
        print('.', end='')
    print()
    print("All tests successful!")


I am working on the file "Python/Chapter-04/Merge-Sort.ipynb".
The first task is:
The function sameElements(L, S) returns Trueif the lists L and S contain the same elements and, furthermore, each
element $x$ occurring in L occurs in S the same number of times it occurs in L.
Can you write Python code for it?


I am working on the file "Python/Chapter-04/Quick-Sort-Array.ipynb".
The first task is:
An Array-Based Implementation of Quick-Sort
The function $\texttt{sort}(L)$ sorts the list $L$ in place.
Can you write Python code for it?

def sort(L):
    quickSort(0, len(L) - 1, L)


I am working on the file "Python/Chapter-04/Quick-Sort-Array.ipynb".
The first task is:
Next, we sort a hundred thousand integers.  This time, many of the integers have the same value.
Can you write Python code for it?

L = [ rnd.randrange(100) for x in range(100_000) ]

%%time
sort(L)


I am working on the file "Python/Chapter-04/Quick-Sort-Array.ipynb".
The first task is:
Next, we sort a million random integers.
Can you write Python code for it?

%%timeit
k = 1_000_000
L = [ rnd.randrange(1000 * k) for x in range(k) ]
sort(L)


I am working on the file "Python/Chapter-04/Quick-Sort-Array.ipynb".
The first task is:
The function $\texttt{partition}(\texttt{start}, \texttt{end}, L)$ returns an index $m$ into the list $L$ and
regroups the elements of $L$ such that after the function returns the following holds:

$\forall i \in {\texttt{start}, \cdots, m-1} : L[i] \leq L[m]$,
$\forall i \in { m+1, \cdots, \texttt{end} }  : L[m] <    L[i]$,
$L[m] = \texttt{pivot}$.

Here, pivot is the element that is at the index end at the time of the invocation
of the function, i.e. we have

$L[\texttt{end}] = \texttt{pivot}$

at invocation time.
The for-loop of partition maintains the following invariants:

$\forall i \in {\texttt{start}, \cdots, \texttt{left} } : L[i] \leq \texttt{pivot}$,
$\forall i \in {\texttt{left}+1, \cdots, \texttt{idx}-1} : \texttt{pivot} < L[i]$,
$L[\texttt{end}] = \texttt{pivot}$.

These invariants are depicted below:

This algorithm has been suggested by Nico Lomuto.  It is not the most efficient implementation of partition, but
it is easier to understand than the algorithm given by Tony Hoare that uses two separate loops.
Can you write Python code for it?

#@logging
def partition(start, end, L):
    pivot = L[end]
    left  = start - 1
    for idx in range(start, end):
        if L[idx] <= pivot:
            left += 1
            swap(left, idx, L)
    swap(left + 1, end, L)
    return left + 1


I am working on the file "Python/Chapter-04/Quick-Sort-Array.ipynb".
The first task is:
The function $\texttt{swap}(x, y, L)$ swaps the elements at index $x$ and $y$ in $L$.
Can you write Python code for it?

def swap(x, y, L):
    L[x], L[y] = L[y], L[x]


I am working on the file "Python/Chapter-04/Quick-Sort-Array.ipynb".
The first task is:
The function $\texttt{testSort}(n, k)$ generates $n$ random lists of length $k$, sorts them, and checks whether the output is sorted and contains the same elements as the input.
Can you write Python code for it?

def testSort(n, k):
    for i in range(n):
        L = [ rnd.randrange(2*k) for x in range(k) ]
        oldL = L[:]
        sort(L)
        isOrdered(L)
        sameElements(oldL, L)
        assert len(L) == len(oldL)
        print('.', end='')
    print()
    print("All tests successful!")

%%time
testSort(100, 20000)


I am working on the file "Python/Chapter-04/Quick-Sort-Array.ipynb".
The first task is:
The function logging is used as a decorator.  It takes a function
f as its argument and returns a new function logged_f that returns
the same result as the function f, but additionally it prints its arguments
before the function is called and when the function returns, both the function
call and the result is printed.
The decorator logging is useful for debugging.
Can you write Python code for it?

def logging(f):
    def logged_f(*a):
        print(f'{f.__name__}{a}')
        r = f(*a)
        print(f'{f.__name__}{a} = {r}')
        return r
    return logged_f


I am working on the file "Python/Chapter-04/Quick-Sort-Array.ipynb".
The first task is:
The function quickSort(start, end, L) sorts the sublist L[start:end+1] in place.
Can you write Python code for it?

def quickSort(start, end, L):
    if end <= start:
        return  # at most one element, nothing to do
    m = partition(start, end, L)  # m is the split index
    quickSort(start, m - 1, L)
    quickSort(m + 1, end  , L)


I am working on the file "Python/Chapter-04/Radix-Sort.ipynb".
The first task is:
Radix Sort
As radix sort is based on counting sort, we have to start our implementation of radix sort by defining the function countingSort that we have already discussed previously.  The easiest way to do this is by using the %run magic that Juypter notebooks provide.
Can you write Python code for it?


I am working on the file "Python/Chapter-04/Radix-Sort.ipynb".
The first task is:
The function $\texttt{extractByte}(n, k)$ takes a natural number $n < 2^{32}$ and a number $k\in {1,2,3,4}$ as arguments.  It returns the $k$-th byte of $n$.
Can you write Python code for it?


I am working on the file "Python/Chapter-04/Radix-Sort.ipynb".
The first task is:
The function $\texttt{radixSort}(L)$ sorts a list $L$ of unsigned 32 bit integers and returns the sorted list.
The idea is to sort these numbers by first sorting them with respect to their last byte, then to sort the list with respect to the second byte, then with respect to the third byte, and finally with respect to the most important byte.
These four sorts are done using counting sort.
The fact that counting sort is stable guarantees that when we sort with respect to the second byte, numbers that have the same second byte will still be sorted with respect to the first byte.
Can you write Python code for it?

def radixSort(L):
    L = [(n, 0) for n in L]
    for k in range(1, 4+1):
        L = [(n, extractByte(n, k)) for (n, _) in L]
        L = countingSort(L)
    return [n for (n, _) in L]


I am working on the file "Python/Chapter-04/Radix-Sort.ipynb".
The first task is:
The function $\texttt{testSort}(n, k)$ generates $n$ random lists of length $k$, sorts them, and checks whether the output is sorted and contains the same elements as the input.
Can you write Python code for it?

def testSort(n, k):
    for i in range(n):
        L = [ rnd.randrange(2**31) for x in range(k) ]
        oldL = L[:]
        L = radixSort(L)
        isOrdered(L)
        sameElements(oldL, L)
        print('.', end='')
    print()
    print("All tests successful!")

%%time
testSort(100, 20000)

%%timeit
k = 1_000_000
L = [ rnd.randrange(2*k) for x in range(k) ]
S = radixSort(L)


I am working on the file "Python/Chapter-05/Calculator-Frame.ipynb".
The first task is:
The Shunting Yard Algorithm (Operator Precedence Parsing)
Can you write Python code for it?


I am working on the file "Python/Chapter-05/Calculator-Frame.ipynb".
The first task is:
The function $\texttt{evalBefore}(o_1, o_2)$ receives to strings representing arithmetical operators.  It returns True if the operator $o_1$ should be evaluated before the operator $o_2$ in an arithmetical expression of the form $a \;\texttt{o}_1\; b \;\texttt{o}_2\; c$.  In order to determine whether $o_1$ should be evaluated before $o_2$ it uses the
precedence and the associativity of the operators.
Its behavior is specified by the following rules:
- $\texttt{precedence}(o_1) > \texttt{precedence}(o_2) \rightarrow \texttt{evalBefore}(\texttt{o}_1, \texttt{o}_2) = \texttt{True}$,
- $o_1 = o_2 \rightarrow \texttt{evalBefore}(\texttt{o}_1, \texttt{o}_2) = \texttt{isLeftAssociative}(o_1)$,
- $\texttt{precedence}(o_1) = \texttt{precedence}(o_2) \wedge o_1 \not= o_2 \rightarrow \texttt{evalBefore}(\texttt{o}_1, \texttt{o}_2) = \texttt{True}$,
- $\texttt{precedence}(o_1) < \texttt{precedence}(o_2) \rightarrow \texttt{evalBefore}(\texttt{o}_1, \texttt{o}_2) = \texttt{False}$.
Can you write Python code for it?

def evalBefore(stackOp, nextOp):
    "your code here"

import stack