Messed up 3D Meshes: A case study in the limits of integer floating point
Tags: Math, Debugging, Babylonjs, 3d

Ok, no noticeable visual change. We can reasonably rule out the optimization algorithm causing this change in geometry…well somewhat at least. However, the fact that we have the same general shape at different polycounts is still incredibly suspicious. Looking at the optimization logic we can then get the general gist of what it does:

Our unoptimized geometry optimizer https://github.com/BabylonJS/Exporters/blob/992d3d5025e9b10b6d929abded0733fe974fd1dc/3ds%20Max/Max2Babylon/Exporter/BabylonExporter.Mesh.cs#L894-L912

Essentially, we search all other vertices that we have exported to the current vertex that we are extracting from Max, and replace it if we already have an equivalent vertex with matching position, UV coordinates, and other relevant attributes:

Now why is all this relevant? It will all make sense soon, I promise! The key here is in the datatypes:

See Autodesk’s documentation for the IPoint3 type. Key point being that the coordinates of these points for our vertices are stored as integers, rather than the IEEE-754 floating point while we are working with them. The key benefit of using integer floating point opposed to the standard floating point format is that its error is consistent within its range, as opposed to floating point precision loss, where one encounters greater loss in precision the farther from 0 your value is:

Imagine this square as the set of all real numbers, while the intersection points are the ones that we can represent as IEEE-754 floating point. (Shamelessly borrowed from: https://www.pathengine.com/Contents/Overview/FundamentalConcepts/WhyIntegerCoordinates/page.php)

The relative precision of an integer floating point space

As you can see, integer floating point is great for representing incredibly large values, not so great for representing incredibly small values compared to your range.

Again, why is this relevant? If we go back to our original issue, I originally hinted at one crucial detail!:

The answer was in front of us the whole time

The affected meshes are only the ones scaled to be virtually invisible before export! This is a common technique for animating visibility within glTF’s current capabilities, and as a result, it exposed some flaw with how we were retrieving geometry:

Raise your hand when you see it… https://github.com/BabylonJS/Exporters/blob/992d3d5025e9b10b6d929abded0733fe974fd1dc/3ds%20Max/Max2Babylon/Exporter/BabylonExporter.Mesh.cs#L739-L753

We were retrieving the mesh geometry in world space, then converting the geometry to local space… Yikes.

Luckily enough the fix was simple enough, and the investigation was very fun, at least when it didn’t nearly bring me to doubt my understanding of digital math. Instead of retrieving all our geometry at 0.001x scale, then scaling it up to local space, we instead should have grabbed it at the appropriate frame of reference:

Use local space, dummy. https://github.com/BabylonJS/Exporters/pull/621/files

You can always check out the latest and greatest version of the exporters here:

https://github.com/BabylonJS/Exporters/releases