Using ARKit To Display Video In Augmented Reality
By: Bilguun Batbold
Tags: iOS, Arkit, Swift, Programming, AR

Image taken from Unsplash

What we will be doing

As a kid growing up, I was really fascinated by the animated Daily Prophet prints in the Harry Potter series. If you have watched them before, the news articles come up ‘live’ just like what you see below.

In this tutorial, we will be recreating this using ARKit. We will first detect an image of Sirius Black, and then overlay a video to be played on top of the image. Let’s get started!

Tutorial

Before we being this tutorial, let’s gather all the materials we will need for this tutorial to work.

Gathering the resources

I will be using this particular photo

First what I have done is to browse Pinterest to retrieve this particular print. Next, I have cropped the image that we will be detecting and overlaying the actual video.

Cropped image

And finally, we will be playing a video over this detected image.

I will be using this particular video.

Creating a new project

Create a new AR Project

Language: Swift, Content Tech: SceneKit

Now that you have created a project, let’s create a new AR Resource Group in our Assets Folder, Assets.xcassets

Add our cropped image (mine is named black ) and give it a size. The size is our actual cropped image size on our printed photo. From my measurements, my printed photo was around 14x15cm. Once we key in 0.14 meters in width, Xcode automatically generates the height.

Next, drag and drop the video file that we will be using to play over the detected image.

Video file black.mp4 added

Make sure when we add the video, we add it to our Project Target.

Adding in code

Delete all the code in ViewController.swift and replace it with the above. What we have done here is set up the AR Configuration to track images and set what images our app should track.

Next we will use one of the delegate methods to add our video over the detected image. The delegate method informs us as it is about to add a node to the scene. This node will be empty and we can customise is accordingly.

But before we enter new code, this will be the logic flow:

First detect when our app has added a new node in an anchor Check if our anchor matches ARImageAnchor , if it matches then it means that our image was detected Create a video node containing our video file and add it to a video scene. Adjust all the positions necessary Create a new plane that has the same orientation and size as our detected image Add our video node to the plane Create a new node containing the plane Add this new node as a child node

This is what we will end up with. Let’s run the app and see the outcome!

Nice! Our AR App is working as expected! However, there is a small issue. The original Daily Prophet news, has a loop effect, so let’s loop our video over the image.

Looping the video

Edit our code to be as above. What we have changed is:

We use AVPlayer to play item Initialise our video node using AVPlayer Add an observer such that when the player finishes its current item, reset the video and play again

Go ahead and run the application! The final output should be very similar to what we have set out to achieve!