Adventures of Photogrammetry in Babylon.js
Tags: Photogrammetry, Babylonjs, Graphics, 3d Modeling

Turns out, the smaller the object is, the higher fidelity image you are going to need. Unless you have a tripod or something to keep your hand steady, its hard to get a non-shaky video. You also want to make sure the object is in the center frame for most of your video which can be a little tricky as you're manually circling around the object, trying to capture both high and low images. I heard capturing in slow mode can help counter this too. Make sure to not capture a ton of the background. I found that also led problems with small objects. Best angles came from when the camera was looking more downward at the object.

That’s when I thought I’d give it another try, perhaps with a larger object. I personally found going outside was the perfect solution. Lots of natural light from the sun and space to walk around a larger object at a distance. This was key because my camera didn't have to focus on the tiny details and could just capture the object as a whole. This created a smooth, consistent video that became perfect for the algorithm.

Once I had a solid data set that was properly generating the mesh, things were pretty simple from there. With a couple quick buttons clicks I was able to generate a textured mesh and boom, its looked just like the real thing.

From here we export it to a FBX and turn into a GLB for Babylon. For this set I simply used Blender to export again. Voila we have a GLB we can just drag into the sandbox to take a took. Wow how stunning. From here we can import into any Babylon.js project. For my demo I just made a playground with 2 photogrammetry meshes.

Well here's another catch. If we took a look at the number of polygons, we would see there are thousands and thousands…almost a million polygons. Now Babylon.js does a great job at rendering these at 60 fps. But there is a bit of overhead when it comes to loading. Granted being able to load such a gorgeous asset in only seven seconds is still remarkable, but this can feel slow if you’re used to really fast playground loads. I would definitely suggest using an async function for loading your models. Still, in the playground this can lead to a bit of slowdown for development every time you are refreshing. If you really want that high polygon model for the final production, I’d suggest using something stand in while you are developing. Another thing we can do however is lower the count by using a decimation modifier in Blender. This can reduce the count by a fraction, just keep an eye on the quality as you go lower and lower. In the end it’s a choice for what you want to do in your scene, how fast you want to run, and how many models want to have.

Another thing to think about is lighting. When creating the textured mesh in 3DF Zephyr, lighting from your photos will already be taken into account. Therefore, it’s important to remember if you using lights on other parts of your scene you should either disable the lighting on the mesh, or at least think about what you trying to achieve. Double lighting can look weird if you’re going for realism. Definitely something to keep in mind.

Finally, don’t forget about mobile! Surprisingly most phones within the Babylon team were able to render these scenes, but do keep in mind it can be very taxing in the hardware. Perhaps a lower and higher polygon version for mobile vs desktop.

So there you have it. A couple tips and tricks for dealing with photogrammetry in Babylon.js. I would 100% recommend trying it out sometime if you want to create a visually impressive model, especially if don’t have the modeling skills like myself ;)

Babylon.js can remarkably render even the expensive models and still run at 60 fps no sweat. Just be cautious of those load times by using other software to downscale it, especially if load time is important. I know that’s something I will probably do on my next demo for sure. Until then, have fun taking those videos/photos and can’t wait to see what you all create!

Photo by Ailbhe Flynn on Unsplash

Seen you real soon,

Pamela Wolf — Babylon.js Team

https://twitter.com/ms_Destiny14