SnapML in Lens Studio: ML Component Breakdown
By: Austin Kodra
Tags: Lens Studio, Heartbeat, Snapchat, Augmented Reality, Machine Learning

Model Compatibility

First things first, in order to get an ML model inside Lens Studio that you’ll be use creatively, you’ll need to ensure that the model is compatible with SnapML. But what does this mean, how can you tell if your model is incompatible, and what can you do to fix it?

In essence, there are two parts of model compatibility that you need to be aware of: model file format, and model layer compatibility.

**An additional, but important note: ML Assets (Lens Studio’s term for model files) must be under 10MB to use in Lens Studio

Model File Format

Machine learning models can take many different forms. There are model file formats that are broadly compatible across server-side deployment targets (cloud servers, on-premise servers, etc), and then others that are designed for mobile or embedded devices (i.e. smartphones, dev boards like Raspberry Pi’s, other AI-capable “smart” devices).

These file formats all depend on what the target platform or device accepts. For instance, for iOS native applications, Core ML (.mlmodel) is the model framework and type you’d need to work with. For native Android, you’d use a TensorFlow Lite model type (.tflite).

For SnapML, there are two ML model types that are compatible: Protobuf files (.pb) and ONNX files (.onnx). These model types can be created by converting from TensorFlow and/or PyTorch, which are the two primary deep learning frameworks/libraries for ML engineers and researchers.

While the Lens Studio team might add new model types to support in the future (fingers crossed for .tflite), currently, your model will need to be successfully converted to either .pb or .onnx.

The Lens Studio team has recently updated their SnapML docs to include instructions for exporting your ML models from both TensorFlow and PyTorch into formats compatible with SnapML.

Export from TensorFlow

Export from PyTorch

Model Layer Compatibility

Though model conversion can be a bit fraught, there are clear and known methods for doing so. A more ongoing and iterative concern that Lens Creators will need to grapple with is the compatibility of the “layers” that make up their models.

Broadly speaking, an ML model layer (or more specifically, a neural network model layer) is a set of mathematical operations that take the data you feed into said model and help produce the intended output. For instance, many computer vision neural networks contain “convolutional layers”, which can be visualized in the image below:

While the technical specifics are a bit outside the scope of this post, you can imagine neural network layers as mathematical layers through which data passes and is processed, all in order to produce the desired output (i.e. an accurate prediction).

So if you’re building your own models, whether from scratch in TensorFlow or PyTorch or from other existing pre-trained models, you’ll need to regularly consult with this layer compatibility chart in the SnapML docs:

What’s more, this compatibility is broken down by mobile device platform (iOS, Android) and by processor. To get the most out of your ML-powered Lenses, and to ensure they work properly, you’ll want to work with model layers that are compatible with at least the CPU, and, if possible, the GPU/NPU (the use of advanced hardware will generally lead to better model performance).

This is something to pay particular attention to when working with pre-trained models and Notebooks you pull from external sources or from locations where you can’t immediately verify compatibility — some of these pre-trained models, though optimized for mobile platforms, might need to have some layers removed/adjusted if they aren’t supported by Lens Studio. This can then lead to the need to retrain models, as removing network layers can significantly (and adversely) affect performance.

Model Import

Once you have a model built and exported with both supported layers and in a compatible format, you’ll need to import it into Lens Studio. If you import a model into a completely blank project, you’ll need to do quite a bit of manual work (including material graphing, scripting, etc).

So if possible, we’d recommend starting with a template project that most closely matches your use case. We have a number of [specific pre-trained templates] at Fritz AI, and any custom models trained will already be configured. Lens Studio also offers a number of helpful templates that make it easier to import custom ML models built with their provided Python Notebooks (typically iPython or Jupyter Notebooks, to be precise).

When you create an ML Component Object (from the top left Objects panel), you’ll automatically be prompted to import your trained model file — in my case, as you can see below, through Finder on my MacBook.

Because I trained my model using Fritz AI, I didn’t have to worry about configuring the import — as mentioned above, Fritz AI does all that work under the hood.

But if you’re bringing in your own custom model, the best thing you can do to ensure that your model makes it into Lens Studio unscathed is to make sure everything is compatible up front as you’re training it — with supported layers, a model architecture supported by and optimized for SnapML, the expected number of channels, etc.