iOS Image Face Centering Using Apple’s Vision Framework
By: Kirill Kudaev
Tags: Swift, Xcode, iOS, Mobile, Software Development

Problem

During the development of multiple Ancestry app features, our team ran into a similar issue. We often had to display images of different aspect ratios within an ImageView for which the aspect ratio was fixed.

The image below illustrates the problem this creates. On the left, you can see the initial uncropped image we get from our server. In the middle, you can see what happens if we try to display that image within an ImageView that has a fixed aspect ratio. The boundaries of the ImageView are marked in red.

Initial image (left). Trying to set that image to a horizontal ImageView (middle). Desired result (right).

This current ImageView has an aspectFill ratio so it crops the top and the bottom parts of our vertical image. The user only sees the content inside of the red border. As a result, our users don’t see the faces of people we’re trying to show them. This significantly hurts the experience of our product. Not good.

So how we can fix this? Wouldn’t it be nice to somehow identify the face position on our image and then crop the image accordingly? The desired result is illustrated on the right in green.

Okay, great… But how can we achieve this desired behavior? The good news is that with iOS 11 Apple released a new Vision Framework that has face detection capabilities built-in!

Solution

Let’s utilize Apple’s Vision Framework to detect and center faces in our images. First of all, we should understand what the Vision API has to offer in our case:

VNImageRequestHandler allows you to process image analysis requests by calling its perform(_:) function and passing an array of requests to conform to VNRequest protocol.

VNDetectFaceRectanglesRequest is one of those requests. This specific request allows us to find faces and their coordinates within an image. Exactly what we need in our case.

VNFaceObservation is a type of observation that results from a VNDetectFaceRectanglesRequest . This object contains the necessary facial-feature information.

Step One. Detecting the facial-feature information.

Let’s start with getting a VNFaceObservation for each of the faces on our image and not worry about the image cropping logic yet. We are going to add the following extension to CGImage :

Note: Vision API is only available since iOS 11. As you can see in the code snippet, we have to mark our functions with @available(iOS 11.0, *)

Here’s what’s going on in the above code:

First of all, we create a VNDetectFaceRectanglesRequest . If for some reason our request returns an error, let’s return .failure . If no faces were found, return .notFound result. Now we can finally iterate through all of the results and add them to our faces array. We’ll need this array in Step Two. For now, let’s just print the total number of faces we found and return the original uncropped image in our .success . Finally, we call our VNDetectFaceRectanglesRequest using perform(_:) function of VNImageRequestHandler .

Step Two. Cropping the initial image.

Now let’s add some more logic to crop the initial image based on the facial-feature information we got from Step One:

Let’s walk through the above code:

The responsibility of this function is to return a CGRect that tells us how to crop our initial image. Let’s look into how this function works. First of all, we initialize our total... variables. These are important for cases when more than one face was detected in our image. minX and minY variables are used to keep track of the farthest bottom-left face in our image. We’ll need those coordinates later. Let’s now iterate through all the faces we were able to identify. P Why do we multiply face.boundingBox.width by CGFloat(width) ? face.boundingBox.width returns a number between 0 and 1 that represents the face width in proportion to the full width of the image. We multiply that number by CGFloat(width) to get the absolute width of the face. Same issue with face.boundingBox.height and face.boundingBox.origin.x . This coordinate space transformation might also seem a bit confusing. First, we do 1 — face.boundingBox.origin.y to get the relative y position from the top instead of the bottom of our image. We then multiply that number by CGFloat(height) and subtract h to correctly represent the absolute y coordinate in the flipped-coordinate space for our CGRect initialization later on. In the flipped-coordinate space, the origin is in the upper-left corner and the rectangle extends towards the lower-right corner. Calculate average width, height, x, and y coordinates of faces by dividing our total... variables by the number of faces in the image. This line might seem a bit tricky. Here we calculate how much extra space we want to add around the face(s) that were found. For that, we calculate the distance between our avgX (which represents the average x coordinate of all faces in the photo) and minX (left-most face coordinate). We then add a custom margin parameter that we will talk about later in this article. We can now use our offset and averages to create a CGRect that represents how we need to crop our initial image. Now let’s use that CGRect to crop the initial image. Return .notFound if cropping(to rect: CGRect) returned nil Return .success with our new face-centered image!