Image Geometric Transformation In Numpy and OpenCV | by Daryl Tan | Towards Data Science

Geometric transformation is an essential image processing techniques that have wide applications. For example, a simple use case would be in computer graphics to simply rescale the graphics content when displaying it on a desktop vs mobile.
It could also be applied to projectively warp an image to another image plane. For instance, instead of looking at a scene straight ahead, we wish to look at it from another viewpoint, perspective transformation is applied in this scenario to achieve that.
One other exciting application is in training deep neural networks. Training deep model requires vast amount of data. And in almost all cases, models benefit from higher generalisation performance as training data increases. One way to artificially generate more data is to randomly apply an affine transformation to the input data. A technique also known as augmentation.
In this article, I would like to walk you through some of the transformation and how we can perform them in Numpy first to understand the concept from first principles. Then how it could be easily achieved using OpenCV. if you are like me who likes to understand concepts from basic theories, this post would be of interest to you!
In particular, I will focus on 2D affine transformation. What you need is some basic knowledge of Linear Algebra and you should be able to follow. The accompanying code can be found here if you prefer to tingle with it yourself!
Without going too much into the mathematical details, the behaviour of the transform is controlled by some parameters in the affine A.
x’ = Ax
where A = [[a_11, a_12, a_13],           [a_21, a_22, a_23],           [  0 ,   0 ,   1 ]]
is a 2x3 matrix or 3x3 in homogenous coordinate, and x is a vector of the form [x, y] or [x, y, 1] in homogeneous coordinate. The formula above says that A takes any vector x and maps it to another vector x’.
Generally, an affine transformation has 6 degrees of freedom, warping any image to another location after matrix multiplication pixel by pixel. The transformed image preserved both parallel and straight line in the original image (think of shearing). Any matrix A that satisfies these 2 conditions is considered an affine transformation matrix.
To narrow our discussion, there are some specialized forms of A and this is what we are interested in. This includes the Rotation, Translation and Scaling matrices as shown in the figure below.
One very useful property of the above affine transformations is they are linear functions. They preserve the operation of multiplication and addition and obey the superposition principle.
In other words, we can composite 2 or more transformations: vector addition to represent translation and matrix multiplication to represent the linear mapping as long as we represent them in homogenous coordinate. For example, we could represent a rotation followed by a translation as
A = array([[cos(angle),  -sin(angle), tx],            [sin(angle), cos(angle),  ty],            [0,          0,           1]])
In Python and OpenCV, the origin of a 2D matrix is located at the top left corner starting at x, y= (0, 0). The coordinate system is left-handed where x-axis points positive to the right and y-axis points positive downwards.
But most transformation matrix you find in textbooks and literature including the 3 matrices shown above follows the right-hand coordinate system. So some minor adjustments must be made to align the axis direction.
Before we experiment with the transformations on images, let’s look at how we could do it on point coordinates. Because they are essentially the same with images being an array of 2D coordinates in a grid.
Utilising from what we have learnt above, the following code below can be used to transform the points [0, 0], [0, 1], [1, 0], [1,1] . The blue dots in figure 3.
Python provides a useful shorthand operator, @ to represent matrix multiplication.
# Points generatordef get_grid(x, y, homogenous=False):    coords = np.indices((x, y)).reshape(2, -1)    return np.vstack((coords, np.ones(coords.shape[1]))) if homogenous else coords# Define Transformationsdef get_rotation(angle):    angle = np.radians(angle)    return np.array([        [np.cos(angle), -np.sin(angle), 0],        [np.sin(angle),  np.cos(angle), 0],        [0, 0, 1]    ])def get_translation(tx, ty):    return np.array([        [1, 0, tx],        [0, 1, ty],        [0, 0, 1]    ])def get_scale(s):    return np.array([        [s, 0, 0],        [0, s, 0],        [0, 0, 1]    ])R1 = get_rotation(135)T1 = get_translation(-2, 2)S1 = get_scale(2)# Apply transformation x' = Axcoords_rot = R1 @ coordscoords_trans = T1 @ coordscoords_scale = S1 @ coordscoords_composite1 = R1 @ T1 @ coordscoords_composite2 = T1 @ R1 @ coords
It is important to take note that, with a few exceptions, matrices generally do not commute. i.e.
A1 @ A2 != A2 @ A1
Therefore, for the transformations
# Translation and then rotationcoords_composite1 = R1 @ T1 @ coords# Rotation and then translationcoords_composite2 = T1 @ R1 @ coords
You will observe in figure 3 that they do not result in the same mapping and that order matters. How the function is applied can be understood from right to left.
Now for images, there are several things to take note. Firstly, as mention before, we must realign the vertical axis. Secondly, the transformed points must be projected onto an image plane.
In essence, the steps that need to be taken are:
Create a new image I’(x, y) to output the transform pointsApply the transformation AProject the points onto a new image plane, only considering those that lie within image boundary.
Create a new image I’(x, y) to output the transform points
Apply the transformation A
Project the points onto a new image plane, only considering those that lie within image boundary.
Let's look at a transformation where we wish to zoom in by 2x and rotate an image by 45 degrees about its centre position.
This could be done by applying the following composite matrix.
height, width = image.shape[:2]tx, ty = np.array((width // 2, height // 2))angle = np.radians(45)scale = 2.0R = np.array([    [np.cos(angle), np.sin(angle), 0],    [-np.sin(angle), np.cos(angle), 0],    [0, 0, 1]])T = np.array([    [1, 0, tx],    [0, 1, ty],    [0, 0, 1]])S = np.array([    [scale, 0, 0],    [0, scale, 0],    [0, 0, 1]])A = T @ R @ S @ np.linalg.inv(T)
Applying to image
# Grid to represent image coordinatecoords = get_grid(width, height, True)x_ori, y_ori = coords[0], coords[1] # Apply transformationwarp_coords = np.round(A@coords).astype(np.int)xcoord2, ycoord2 = warp_coords[0, :], warp_coords[1, :]# Get pixels within image boundaryindices = np.where((xcoord >= 0) & (xcoord < width) &                   (ycoord >= 0) & (ycoord < height))xpix2, ypix2 = xcoord2[indices], ycoord2[indices]xpix, ypix = x_ori[indices], y_ori[indices]# Map the pixel RGB data to new location in another arraycanvas = np.zeros_like(image)canvas[ypix, xpix] = image[yy, xx]
Few points to take note in the 2 code snippets above.
Left-handed coordinate system rotation is accounted for by swapping the sign.Since points are rotated about the origin, we first translate the centre to the origin before doing rotation and scaling.Points are then translated back to the image planeThe transforms points are rounded to integers to represent discrete pixel value.Next, we only consider the pixels that lie within the image boundaryMap correspondence I(x, y)and I’(x, y)
Left-handed coordinate system rotation is accounted for by swapping the sign.
Since points are rotated about the origin, we first translate the centre to the origin before doing rotation and scaling.
Points are then translated back to the image plane
The transforms points are rounded to integers to represent discrete pixel value.
Next, we only consider the pixels that lie within the image boundary
Map correspondence I(x, y)and I’(x, y)
As you can see, due to step 4, the resulting image (figure 4) will have several aliasing and holes. To eliminate this, open-source libraries use interpolation techniques to fill the gaps after transformation.
Inverse Warping
Another approach to prevent aliasing is to formulate the warping as that of resampling from the source image I(x, y) given the warped points X’. This can be done by multiplying X’ by the inverse of A. As a caveat, the transformation has to be invertible.
Apply the inverse of the transformation to X’.
Apply the inverse of the transformation to X’.
X = np.linalg.inv(A) @ X'
Note: for images, the inverse warping of X’ is simply reprojecting I’(x, y) onto I(x, y). So we simply apply the inverse transformation to I’(x, y) pixel coordinates as you will see below.
2. Determine where it lands in the original image plane
3. Resample from I(x, y) the RGB pixels and map it back to I’(x, y)
code
# set up pixel coordinate I'(x, y)coords = get_grid(width, height, True)x2, y2 = coords[0], coords[1]# Apply inverse transform and round it (nearest neighbour interpolation)warp_coords = (Ainv@coords).astype(np.int)x1, y1 = warp_coords[0, :], warp_coords[1, :]# Get pixels within image boundariesindices = np.where((x1 >= 0) & (x1 < width) &                   (y1 >= 0) & (y1 < height))xpix1, ypix1 = x2[indices], y2[indices]xpix2, ypix2 = x1[indices], y1[indices]# Map Correspondencecanvas = np.zeros_like(image)canvas[ypix1, xpix1] = image[ypix2,xpix2]
Running the above code should give you a dense, hole-free image :) Feel free to download the code and play around with the parameters to apply other transformations.
Now that you have a better understanding of geometric transformation, most developers and researchers usually save themselves the hassle of writing all those transformations and simply rely on optimised libraries to perform the task. Doing affine transformation in OpenCV is very simple.
There are a few ways to do it.
Write the affine transformation yourself and call cv2.warpAffine(image, A, output_shape)
Write the affine transformation yourself and call cv2.warpAffine(image, A, output_shape)
The code below shows the overall affine matrix that would give the same results as above. A good exercise would be to derive the formulation yourself!
def get_affine_cv(t, r, s):    sin_theta = np.sin(r)    cos_theta = np.cos(r)        a_11 = s * cos_theta    a_21 = -s * sin_theta        a_12 = s * sin_theta    a_22 = s * cos_theta            a_13 = t[0] * (1 - s * cos_theta) - s * sin_theta * t[1]    a_23 = t[1] * (1 - s * cos_theta) + s * sin_theta * t[0]return np.array([[a_11, a_12, a_13],                 [a_21, a_22, a_23]])A2 = get_affine_cv((tx, ty), angle, scale)warped = cv2.warpAffine(image, A2, (width, height))
2. Rely on OpenCV to return the affine transformation matric using cv2.getRotationMatrix2D(center, angle, scale).
This function rotates the image about the point center with angle and scale it with scale
A3 = cv2.getRotationMatrix2D((tx, ty), np.rad2deg(angle), scale)warped = cv2.warpAffine(image, b3, (width, height), flags=cv2.INTER_LINEAR, borderMode=cv2.BORDER_CONSTANT, borderValue=0)
In this article, I have covered the basic concepts of geometric transformation and how you can apply it on images. Many advanced computer vision such as slam using visual odometry and multiview view synthesis relies on first understanding transformations.
I believe it is most certainly beneficial as a computer vision practitioner to understand how these transformations work under the hood when we use powerful libraries such as imgaug and albumentation.
Thanks for reading! And I hope you have gained a better understanding of how the formulas are written and used in libraries. Follow to see more post on computer vision and machine learning:) Do sound out in the comments if you spot any mistake or anything is unclear!