6 Takeaways from Snapchat’s Lens Fest
By: Austin Kodra
Tags: Lens Studio, Heartbeat, Snapchat, Augmented Reality, Machine Learning

2. Lens Studio 3.3: Visual Scripting, Texture Compression, & more

A big part of Lens Fest was also the official unveiling of Lens Studio 3.3. I’ve been impressed by the features the Lens Studio team has been apple to pack into iterative 3.x updates since 3.0 launched back in the early summer. And 3.3, from a Creator’s perspective, has to be pretty exciting.

Here’s a quick rundown of what seem like the most impactful new additions:

Texture Compression. This new feature allows you to compress texture assets directly inside Lens Studio. From what I could tell in the comments when this announcements, this is going to be a really important improvement workflow, especially for those who have to commonly bounce back and forth between graphics engines and Lens Studio for this task.

This new feature allows you to compress texture assets directly inside Lens Studio. From what I could tell in the comments when this announcements, this is going to be a really important improvement workflow, especially for those who have to commonly bounce back and forth between graphics engines and Lens Studio for this task. Visual Scripting: As someone who isn’t a programmer but is code-curious, this was the announcement that most intrigued me. It allows you, instead of programming scripts in JavaScript, to graph that same custom Lens logic with a node system similar to (or maybe the same as?) material graphs. When I hop back into Lens Studio, this is certainly something I want to try.

As someone who isn’t a programmer but is code-curious, this was the announcement that most intrigued me. It allows you, instead of programming scripts in JavaScript, to graph that same custom Lens logic with a node system similar to (or maybe the same as?) material graphs. When I hop back into Lens Studio, this is certainly something I want to try. New Templates: In addition to a template getting you up and running with visual scripting, the Lens Studio team also added a Face Morph template, that allows you to morph faces custom 3D meshes; a Tween template that lets you use a dropdown menu to set up animations; and a Configuration template that leverages various UI widgets to help you create adjustable Lenses.

In addition to these new features, there are a number of performance enhancements, bug fixes, and other improvements, with quite a few of those focused on SnapML. You can check those out in the release notes.

3. LiDAR moves Lens Studio closer to enabling full occlusion

For the first time, with the iPhone 12 series, LiDAR has made its way to a mobile device. What is LiDAR? Simply put, it’s a sensor that sends bursts of light (in the form of a laser) and measures the time it takes for that light ray to reflect off a surface back onto the scanner. In practice, what this does is allow the camera to create a 3D topological depth map of a scene.

In one Q&A session with some folks from Apple, their team used an analogy that I really liked. Imagine your entire living room covered tightly in plastic wrap — this sealed wrap basically creates a transparent overlay over all the textures in the space. LiDAR in the iPhone 12 allows the Snapchat camera to create this kind of overlay of any environment, complete with hyperrealistic 3D depth information about it.

In the demo Snapchat and Apple co-produced, the environmental textures and graphics (grass, rocks, vines, starry sky) grow dynamically with the camera scene as it changes—vines only grow on walls, the starry sky only blankets ceilings, etc.

Essentially, this is a huge step towards fine-grained scene understanding, more complete control over occlusion, and more realistic AR experiences more broadly. This tech is in its infancy (at least in terms of mobile experiences), so be on the lookout for some incredible experiments and experiences on this next generation of smartphones powered at least in part by LiDAR.

4. Snapchat announces $3.5M fund for Creators

There’s already been quite a bit of coverage on this item from other awesome outlets, so I won’t go into too much detail, but this feels like, along with the ongoing residency series, to be a first step in Snapchat’s deeper investment in Lenses as (at least in part) commercial products and Lens Creators as the pioneers of the next era of immersive AR experiences. Will be exciting to see how this fund impacts the space.

5. Snapchat Lenses are more than just face filters

This was certainly already true before this week, but I think it really hit home for me that the Lens ecosystem extends far beyond face filters — as I used to call them back in the day when the Big Mouth Lens was a personal revelation (to be honest, it still is).

Snapchat Lenses truly are lenses through which to view the world, not simply layers to place on top of the world. It’s a subtle but powerful difference when you consider features like World Lenses, Landmarkers, and now, SnapML. Combine these features with incredible new tech like LiDAR and Spectacles (more on Snapchat Spectacles here — not something I knew much about before this week), and we’re just scratching the surface of this new era in social AR — and mixed reality experiences more broadly.

I especially appreciated this in-depth conversation from 3 of the engineers that work on the Lens Studio team:

6. Shoutout to the Lens Creator Community

For my last takeaway, I’d like to give a quick shoutout to the Lens Creator community. I and our team have had the privilege of talking to a whole bunch of Creators as we’ve built out support for SnapML in Fritz AI, and though we really enjoyed those 1:1 conversations, seeing this community come together in the most difficult and trying of circumstances truly was special.

I repeatedly saw such healthy, helpful, and encouraging interactions in session chats, networking events, and in other public squares. No negativity, super inclusive, and just a really interesting, diverse group of people.

Additionally, kudos to the Snap team who put this event on, and who clearly cares deeply about their Creator community, partners, and others who make all this magic happen. Thank you for including us, and we hope this is the beginning of a long journey together.