Wink! Or How your Users can interact with your iOS App by changing their Face Expressions
By: César Vargas Casaseca
Tags: Swift, Mobile App Development, AR, iOS, iOS App Development

Behind the Scenes

In this section we are going to see how Wink uses ARKit to to detect the user face expressions. If you wanna start using Wink right away in your app, go directly to the next passage.

As in any other AR experience with SceneKit , we need to create an ARSCNView instance, that will be added to our FacialExpressionDetectorViewController view.

After that we set the delegate to our class, that will receive the view’s AR scene information with SceneKit content. Once we have added the view to our hierarchy and set the delegate, we call to run the session with a face tracking configuration in viewWillAppear :

In the same way, we should not forget to pause it when the view will disappear.

Once we have setup the ARSCNView that will display the AR experience and run the session, it is time to start detecting the face expressions and pass them to the client. We do this through the ARSCNViewDelegate methods:

Firstly, we have to return a new node with the representation of face topology in the our scene view for each 3D anchor, that is, the position and orientation of something of interest in the user’s face. If these concepts are too advanced for you or you need to refresh them, this tutorial is perfect to get familiar with the ARKit basics.

Now that we have a node for each anchor, ARKit will let us know whenever the node was updated, that is, when new information was obtained from the scene view:

Here we refresh our nodes with the face anchors, and proceed to detect the facial expressions from the ARFaceAnchor :

Not a new Avenger, but a face with AR Nodes. See how ARKit is able to detect them in 3D thanks to the True Depth Camera

To better understand this, we need to know what is a FacialExpressionAnalyzer in Wink. A FacialExpressionAnalyzer is a struct encapsulating the data needed to detect one specific face expression:

It contains the Wink FacialExpression object that represents the gestures in the user’s face (mouthSmileLeft, mouthSmileRight …), the AR object that it corresponds to, and the minimum valid coefficient to accept a facial expression as actually happening. As pointed before, Wink comes with a default set of analyzers:

that is created and assigned in FacialExpressionDetectorViewController :

As we will see later, it is very easy to extend this functionality with our own with having to modify the code, thus following the Open-Closed Principle:

“Software entities should be open for extension, but closed for modification”.

Going back to detectFacialExpression(from anchor: ARFaceAnchor) , we see how we compactMap the analyzers into face expressions whose current value is higher than the minimumValidCoefficient , meaning that the user is actually doing that gesture as much as our requirements accepts them. Once we have the array with the current Face Expressions, we send it through the Combine's PassthroughSubject object.

Let’s Play!

Photo by Jeremiah Lawrence on Unsplash

This was all very interesting, but to the developer who wants to roll fast it will be probably not so useful. After all, the purpose of Wink is to ease the process of face expression detections without the need of dealing with the compelling but arcane world of ARKit.

To start using Wink in your screen’s UIViewController , create a new instance of FacialExpressionDetectorViewController and add it to the former through the usual process of view controller containment:

You might then want to place it as you wish adjusting the constraints, or hide it if you don’t want the camera view to be shown. In this case we add it to the upper left corner of our view:

After these steps, it comes the climax of the movie; we have to subscribe to the User Face Expression changes, sinking the Combine publisher that come with the FacialExpressionDetectorViewController :

In our sample app we are just describing the current face expressions with the help of a FacialExpression extension, in your case here is where you react to the user face expression changes to trigger the desired action:

My expressions repertoire with which I will conquer Hollywood

In this debug sample you can see how Wink detects my face expressions, and how the client react to them displaying a description of the retrieved gestures. The statistics and the web are part of the Wink debug mode, they can be of course be disabled in production.

That’s it! The process of using Wink is very straightforward, just four steps: