Face Morphing — A Step-by-Step Tutorial with Code
By: Azmarie Wang
Tags: Computer Vision, Morph, Triangulation

In addition to the 68 facial feature points, we also need to select some points outside of the face to include the background into the morphing frames. To make sure the background is included, we need to select at least 4 points (4 corners), here we will use 8 points (4 corners + 4 center points from each edge) to make the transition more smooth.

Example use of dlib’s facial landmarks (68 points) for the face + 8 points for the background

Warping Triangular Mesh

Now, how do we go from feature points to pixels? Enters the idea of the triangular mesh.

With the paired landmarks, we could define the same triangular mesh over two sets of points, which will give us triangle-to-triangle correspondences between two sets of facial features. Why would we use a triangle over other primitives? My understanding is that they are easy to work with and tessellate well so that we won’t end up with cracks in the mesh where two edges join.

Now we can break down the problem of defining the intermediate shape into many sub-problems — finding each intermediate triangles and then cross-dissolve the values into colors in the warped shape. Before we dive into how to assign the points into triangles, we will first show the feasibility of warping one triangle to another.

In fact, with only (x,y) coordinates initially, we are unable to capture the degree of freedom that represents such translation. In such cases, it is often convenient to work with homogenous coordinates that introduce a third w coordinate, where each (x, y) becomes (x, y, w) . Since two points are equivalent iff x'/w' = x/w and y'/w'= y/w , we can just set w=1 .

With this system, we can represent the 2D transformations with a matrix multiplication:

There are 6 unknows (a, b, c, d, e, f) in the matrix system above. If we vectorize transformation parameters, each pair of points (x, y) and (x’, y’) can determine two parameters.

Thus, using three pairs of vertices from the source and target triangles, we can solve the 6 unknowns with 6 equations, which uniquely identify the affine transformation matrix. With that, we can now solve any transformation of two triangles!

Forward Warping

Intuitively, we want to calculate the transformation T that goes from source A to destination B such that T(A) = B, and then we can apply T to every pixel bounded by the triangle in A to find its corresponding coordinate in B. This process is called forward warping.

However, since T could be a matrix of floats from solving the matrix system, the calculated (x’,y’) coordinates could very likely land in between pixels. In this case, many source pixels can map to the same destination pixel, yet some destination pixels may not be covered.

Inverse Warping

To combat this, we use inverse warping instead, which solves the inverse of T. By solving for the inverse of T and applying the inverse affine transform to every coordinate in B, our resulting image can avoid “holes” (See figure below). When the pixel lands in between coordinates in the source triangle, we can then interpolate within the source neighbours.

Forward and backward image warping. In the case of forward warping (A), holes can occur in the warped image, marked in gray. Backward warping (B) eliminates this problem since intensities at locations that do not coincide with pixel coordinates can be obtained from the original image using an interpolation scheme. Image Credit: Non-rigid Registration Using Free-form Deformations

By iterating through each pair of triangles, and calculating for each pixel in the source/ destination images, we are finally able to produce the morphed faces in the intermediate frames.

Let’s recap the morphing procedure with more details this time:

Given the corresponding features pairs, for each frame at time t:

Compute the intermediate shape by linear interpolation of each pair For each triangle T in the intermediate shape

A is from I₀ (t = 0), T is from the intermediate frame (assume t = 0.5), and B is from I₁ (t = 1)

Calculate the affine transforms from corresponding triangle A in I₀ and B in I₁ to T from the intermediate frame

Warp A and B to shape of T

Warp A and B towards T using results from the previous step

Cross-dissolve color: Blue + Yellow = Green!

For each pixel in the triangle, find the corresponding points in A and B, and cross-dissolve the color by interpolating

Delaunay Triangulation

So far, we talked about choosing the facial landmarks (points) and generating the intermediate frames with triangular mesh. There’s only one link missing now — how can we organize the points into such triangles?

Given a set of points in a plane, triangulation refers to the subdivision of the plane into triangles, with the points as vertices. There is an exponential number of triangulations of a point set. The triangulation we will use for morphing is Delaunay triangulation — what makes Delaunay triangulation special?

Let α ( T) = ( α1, α2 ,.., αn) be the vector of angles in the triangulation T in increasing order. A triangulation T1 is “better” than T2 if the smallest angle of T1 is larger than the smallest angle of T2. With this, Delaunay triangulation is the “best” triangulation for maximizing the minimal angles, and they tend to avoid long skinny acute angles. Thus, we will use Delaunay triangulation in our implementation.

Each frame of the animation shows a Delaunay triangulation of the four points. Halfway through, the triangulating edge flips showing that the Delaunay triangulation maximizes the minimum angle, not the edge-length of the triangles. Figure credit: Delaunay triangulation Wikipedia

Here, I don’t intend to dwell on the Delaunay Triangulation algorithm, if you are interested, I found this book chapter very thorough.

In OpenCV, Delaunay Triangulation can be handled easily with Subdiv2D — the function creates an empty Delaunay subdivision where 2D points can be added using the function insert().