SnapML in Lens Studio—Apply AR Effects to Segmented Objects
By: Austin Kodra
Tags: Lens Studio, Heartbeat, Snapchat, Augmented Reality, Machine Learning

Step One: Preparing the Template

Our template projects come loaded and configured with simple, demonstrative AR effects already attached. For the Christmas Tree project, we use a color slider that allows you to dynamically change the color of the tree (the fully-segmented area). But it strikes me that nobody is going to want to change the color of their tree wholesale.

So first things first, let’s disable the color slider, which we can do by unchecking the UI Color Picker script under the Slider Orthographic Camera object in the Objects panel (top left).

We don’t need to delete or remove anything for the purposes of this tutorial, but if you’ve got bigger plans for your Lens, it might be good to clean this up.

Step Two: Choosing an Animated Asset

For simplicity’s sake, I’ll only add one decorative element to my tree—this glitter animated gif:

I initially wanted to anchor an individual ornament to a specific part of the tree, but I realized that’s perhaps a bit out of my element (my guess is that would take some scripting). So instead, I’ve decided to add the above glittery global effect to the segmented tree.

Note that this is a gif with a transparent background — that’s because we don’t want the gif background to block out the tree fronds behind it. Of course, this asset can be anything you want, whether it’s a static image or something more complex.

Drag and drop this asset into the Resources panel in Lens Studio

Step Three: Create a New 2D Texture Parameter Node

Now that we have our AR decorative effect ready to go, we need to figure out a way to attach it to the tree’s segmentation mask (the output of the ML Component).

Because we’re using a 2D asset, we’ll need to add a new 2D Texture Parameter node to our primary segmentation Material ( Fritz Segmentation Texture ). This is how we’ll bring in our new texture (in this case the animated gif) to layer on top of the segmented tree.

We do this by selecting the Fritz Segmentation Texture Material, opening up the Material Graph Editor, and creating a new Texture Parameter node. I’ve decided to change the name of this new parameter to Mask for simplicity’s sake, but you can use whatever naming conventions you prefer.

Next, we’ll need to hook this up to (ultimately) our Shader.

Step Four: Connecting the 2D Texture Parameter Node

Currently, our new node is sitting off on its own, so we need to hook it up in the Material Graph so that it’s ultimately connected to the Shader.

If you look at the current state of the Graph, you’ll notice that there’s already a 2D parameter, Mask Color , connected to the Mixer , which in turn hooks everything up to the Shader .

This is the parameter that was helping us apply the solid color effect to the segmented tree when we first opened up our template. With this info, we know that this is where we need to connect our new 2D Texture node.

Because all I want to do is add my animated gif as a layer on top of the tree, I can remove the Mask Color parameter entirely, and simply connect the new parameter to the Mixer .

If you want to add more than one effect to your tree, you’ll need to (at minimum) add a Multiply node, hook up each unique parameter to it, and then connect that Multiply node to the Mixer .

You final Material Graph for the Fritz Segmentation Texture should look like this:

Step Five: Adding the gif to the 2D Texture Parameter

In the Material Inspector, you’ll now have a new Graph parameter called Mask . Set this parameter to the asset we imported earlier (in my case, the gif)

From here, we can play with (among other things) different blend modes to see what works best for a given effect. As you can see above, I’ve chosen Add — just is visually what I preferred for this example.

If we head on over to the Preview panel, we’ll see our effect layered on top of the segmented tree, both on live video and in a still image.

And here’s a look at the effect on a still image of an undecorated tree (image source):

Next Steps?

That’s all for this tutorial—it was exciting for me to jump into the Material Graph Editor, as I hadn’t gotten a chance to do that before, and it was a handy way to experiment with adding AR effects to a segmented object.

Now that you have a first look at creating on top of a segmentation ML Component, I’d encourage you to jump in and see what you can do. I’m 100% sure you’ll come up with something better than my glittery tree.

Share what you come up with by Tweeting us @fritzlabs—we’re excited to see what you’ll create!