The 3D Ecosystem
By: Gideon Mann
Tags: 3D Printing, Metaverse

Facebook’s change in name to “Meta” has focused a lot of attention on the idea of a “metaverse”, but co-existing with the metaverse is a set of technologies around “3D”. These 3D technologies contribute to the metaverse but also have a distinct set of purposes and use.

3D in the metaverse

One of the key aspects of the metaverse is that it is a “digital twin” of the real world, existing in a 3D world, or perhaps VR or AR. No matter where, these 3D worlds require a set of technology needed for these experiences. Chief among these technologies is computer graphics (e.g. ray-tracing) which is required to render 3D objects. For AR, computer vision is also critical in order to be register and track surfaces in the real-world, since in order to understand how to embed objects in the world, it is critical to understand the underlying geometries of the world. This is true for complex 3D AR realizations, but it’s also true for a simple snap-chat filter that puts a dog tongue on someone’s face: you have to be able to find and track the face in real-time. Holograms are another technology that also relies on 3D objects. Looking glass factory is one example of a desktop holographic interface (the entry level device is $300). It’s just another end-point for 3D images and render them on a holographic display on your desk. Google’s telepresence project Starline is another example of how light field technology can realize immersive 3D spaces.

Another part of the metaverse concept is that objects can be transferred between the spaces. What is interesting about 3D is that they already are able to commute between existing VR and AR spaces. To give a small, but illustrative example: a child can create a 3D design in TinkerCAD (a 3D platform for children), render it as a Minecraft object, import it into that platform and play inside of that design. In more sophisticated platforms, this happens in larger ways. The Unreal rending engine enables creation of incredibly realistic 3D human models that can then be imported into the Epic platform. Sites like kitbash3d.com stock inventories of 3d objects that can then be used across movies and games. Of course, these objects are lack dynamics, they’re static constructs (3D geometries, color etc) where you’d expect more from a realized metaverse object (e.g. physical dynamics — like the movement of an opening door).

3D Object Creation is getting easier, but it’s still pretty hard!

One of the main roadblocks to compelling digital environments is the cost to create 3D content. Authoring a 3D model from scratch requires artistic talent, computer expertise (to manipulate the in-progress piece of art), and compute power, though the availability of powerful graphics cards and the decreasing cost of compute has dramatically decreased the cost of building 3D models. As one example, Blender (for 3D illustration) is a massively popular open-source 3D authoring tool which can run on a mid-price desktop. As a consequence, what used to take a render farm as recently as 20 years ago, can now be done locally and freely. Unity’s acquisition of Weta (the digital effects studio) speaks to still how central authoring 3D content is. Here is Unity SVP Marc Whitten explaining the rationale behind the acquisition (link):

“Think back 10 years ago,” he told me. “Think about 2D, and just how many photos were being created. There was probably a lot! But project forward 10 years… and it’s just extraordinarily greater now. I think we’re in the same place for 3D.” “The one difference, though” he notes, “is that 10 years ago, I … kind of thought I could take a photo. I can sort of do something with 2D. But that’s actually almost not-true — because every time I click the shutter button on my iPhone, it’s probably running 5 million lines of super advanced code to turn me into [a photographer]. But in 3D, I actually personally can’t model something today. What we need to do over the next 10 years is take that same approach; to take this incredibly deep technology and make sure that it works effortlessly.”

The other side of this is 3D objects created from an engineering (and less artistic) point of view. Premium CAD (computer-aided design) software can thousands of dollar per year, but there are emerging low-cost or free 3D authoring tools enabled by the emergence of powerful and inexpensive graphics cards and cheap cloud compute has lowered the barrier to authoring 3D objects. Mid-tier authoring tool, like Fusion 360 from AutoCAD, costs $500/yr. But there is now the emergence of freemium software. OnShape, as an example allows you to create content as long as it is public. SketchUp allows you to create within some limitations. These low-cost tools are getting rapidly better, and its getting easier to create new 3D content.

Of course, there is another way to create 3D objects: scanning which builds a 3D representation of the world by photographs. Long range LiDAR, as used in autonomous vehicles, is still expensive, but the iPhone 12 has a short range LiDAR cameras and as a consequence it is very inexpensive to generate fairly faithful 3d scans, as suggested in this tweet. Even ignoring LiDAR, depth imaging software, enabled by machine learning, has improved greatly so that you can estimate a near-real time depth map from an embedded piece of software (e.g the Luxonis open-source AI drivern camera package). Already these scanners are being used, e.g. in real estate where 3D scans are being generated for homes that enable someone to virtually walk through a home (Matterport as one example).

An Introvert’s 3D

The metaverse presupposes a social world that is the center for 3D artifacts, but 3D technology is also used to create objects in the physical world, outside of a social context.

Venkatesh Rao (@vgr) has been writing and talking about the emerging REALIST stack for engineering, which describes a set of technologies/disciplines from tensor programming to lithium batteries that form a new stack for building machines. The “A” in the stack is “Additive” for additive manufacturing technologies (e.g. 3D printing FFM).

Additive manufacturing technologies are powerful — fast to deliver custom prototypes and the technology changes the way design happens, as they enable cheap and fast realization of 3D parts. The innovation in the space has been amazing. The RepRap printer platform is an open hardware platform for building 3D printers. 3d printer parts are now cheap and ubiquitous (e.g. stepper motors). There is wild experimentation with various filament and extruder types like metal, ceramic, wood, and chocolate. Even “slicers” (e.g. the PrusaSlicer) that convert a 3D representation into “gcode” (the physical movement of an extruder feed or CNC cutter) have been massively developed. The ease of use of the REALIST stack has gone down so much, that it is no surprise that former artists are picking up the space (e.g. odd_jayy) and able to produce novel pieces. In fact, it feels like the skill set needed now to do interesting work in robotics isn’t engineering, but art & design.

It’s not just additive technology though, it’s also CNC machines, laser cutters. Once you have a 3D model, you can realize or instantiate the objects this in virtual worlds (VR, AR, hologram) and physical worlds (FFM, cnc, laser cutters).

What may be surprising is that in order to build anything interesting, you still need to go beyond the 3D printer paradigm. Typically, you need some small hardware pieces (e.g. a flexible thread), active components (motors, servos), and compute (Arduino, rasperry pi). At that point, you can build an incredible variety of small robotic applications, like a 3D printer itself or a bionic arm. It feels like 3D printing hasn’t reached its final form.

As a side point, usually VR and AR are thought of as ends in of themself, but the ability to use AR to make a visual product that is consumed in another way is another (asocial) angle. While Filmmakers have used computer graphics for a long time, film makers can now utilize augmented reality environments to film (e.g. ILM’s Mandalorian stage). This has the potential to radically decrease filming costs — see lighttwist.io as a company working on bringing this technology to the masses. Here the AR environment is being used not as a place to interact socially, but as a tool to create another 2D visual object.

Asocial replication has some missing pieces

Social (e.g. metaverse) is also not the only way that 3D objects can be distributed. The acquisition of Sketchfab, a 3D content viewing and sharing repository, by Epic Games is one example of the importance of distribution of 3D models. Beyond that, are repositories of 3D images like Thingiverse and grabcad which host millions of CAD designs for free.

These repositories are great because there was nothing like them before, but from a utility perspective, they are bare bones. It’s impossible to find anything — you can only search by the keywords authors assign. It’s not even clear what search would mean for 3d shapes. The selection of content is also random, according to what amateurs have chosen to scan and upload. On top of this, it is incredibly challenging to take existing 3D objects and repurpose them, especially for CAD applications. It should be straight-forward to take an existing 3D cad drawing of a hinge, embed it into a 3D image and allow for physical dynamics of a swinging door, but there is no straight-forward way to do this no.

It feels, in a very real sense, like what the web was like in the late 1990s — full of amazing content but predominantly inaccessible and incomplete. A social metaverse is an imagined future for 3D from Facebook. I’m curious to see what the future for 3D from Google would look like.

Altogether

Altogether, 3D is a significant emerging technology theme that is starting to have significant commercial impact. Hard to imagine it won’t have less in the coming years. Already technology like computer graphics (for rendering), computer vision (for registering existing 3D surfaces), additive manufacturing (3D printers, slicers), LiDAR (for 3D scanning), lightfield (for holograms) and tools like for authoring (TinkerCAD, Blender) and browsing (SketchFab, thingiverse, grabcad) have made a connected ecosystem to support 3D objects.

When there is technology for searching among 3D models beyond SEO keywords and for adaptation of existing models, we’ll have reached yet another enormous set of opportunities for 3D that will lift all of the pieces of the ecosystem.