Repository: zeux/meshoptimizer (C++)

Issue:
Vertex remapping/reindexing with approximate equality
As the README states,
> meshopt_generateVertexRemap uses binary equivalence of vertex data, which is generally a reasonable default; however, in some cases some attributes may have floating point drift causing extra vertices to be generated. For such cases, it may be necessary to quantize some attributes (most importantly, normals and tangents) before generating the remap, or use a custom weld algorithm that supports per-attribute tolerance instead.

Since rolling a custom version of the algorithm for a single-line change also comes with a maintenance cost, would it be possible to provide a variant of the algorithm that allows the specification of custom distance functions?

For instance by extending the `meshopt_Stream` struct or by passing a function pointer to the signature, which then gets the stream and element?


Discussion: Yeah I plan to add support for tolerance based remapping eventually. It would be helpful to have a generic algorithm for this use case.

This is not a single-line change, and exposing a callback isn't quite enough; this requires a different algorithm, and perhaps an interface where the implementation has direct access to floating point values (or not; for normals it might be better to compare the dot product for example).



Fix (patch):
diff --git a/README.md b/README.md
index 7cd858e43..d9d0488a9 100644
--- a/README.md
+++ b/README.md
@@ -56,7 +56,8 @@ First, generate a remap table from your existing vertex (and, optionally, index)
 size_t index_count = face_count * 3;
 size_t unindexed_vertex_count = face_count * 3;
 std::vector<unsigned int> remap(unindexed_vertex_count); // temporary remap table
-size_t vertex_count = meshopt_generateVertexRemap(&remap[0], NULL, index_count, &unindexed_vertices[0], unindexed_vertex_count, sizeof(Vertex));
+size_t vertex_count = meshopt_generateVertexRemap(&remap[0], NULL, index_count,
+    &unindexed_vertices[0], unindexed_vertex_count, sizeof(Vertex));
 ```
 
 Note that in this case we only have an unindexed vertex buffer; when input mesh has an index buffer, it will need to be passed to `meshopt_generateVertexRemap` instead of `NULL`, along with the correct source vertex count. In either case, the remap table is generated based on binary equivalence of the input vertices, so the resulting mesh will render the same way. Binary equivalence considers all input bytes, including padding which should be zero-initialized if the vertex structure has gaps.
@@ -70,7 +71,18 @@ meshopt_remapVertexBuffer(vertices, &unindexed_vertices[0], unindexed_vertex_cou
 
 You can then further optimize the resulting buffers by calling the other functions on them in-place.
 
-`meshopt_generateVertexRemap` uses binary equivalence of vertex data, which is generally a reasonable default; however, in some cases some attributes may have floating point drift causing extra vertices to be generated. For such cases, it may be necessary to quantize some attributes (most importantly, normals and tangents) before generating the remap, or use a custom weld algorithm that supports per-attribute tolerance instead.
+`meshopt_generateVertexRemap` uses binary equivalence of vertex data, which is generally a reasonable default; however, in some cases some attributes may have floating point drift causing extra vertices to be generated. For such cases, it may be necessary to quantize some attributes (most importantly, normals and tangents) before generating the remap, or use `meshopt_generateVertexRemapCustom` algorithm that allows comparing individual attributes with tolerance by providing a custom comparison function:
+
+```c++
+size_t vertex_count = meshopt_generateVertexRemapCustom(&remap[0], NULL, index_count,
+    &unindexed_vertices[0].px, unindexed_vertex_count, sizeof(Vertex),
+    [&](unsigned int lhs, unsigned int rhs) -> bool {
+        const Vertex& lv = unindexed_vertices[lhs];
+        const Vertex& rv = unindexed_vertices[rhs];
+
+        return fabsf(lv.tx - rv.tx) < 1e-3f && fabsf(lv.ty - rv.ty) < 1e-3f;
+    });
+```
 
 ## Vertex cache optimization
 
@@ -573,6 +585,7 @@ Currently, the following APIs are experimental, with the functions marked with `
 - `meshopt_computeSphereBounds`*
 - `meshopt_encodeVertexBufferLevel`*
 - `meshopt_generateProvokingIndexBuffer`*
+- `meshopt_generateVertexRemapCustom`*
 - `meshopt_partitionClusters`
 - `meshopt_simplifySloppy`
 - `meshopt_spatialSortTriangles`
diff --git a/demo/main.cpp b/demo/main.cpp
index 1e3e9ea2e..012a8524d 100644
--- a/demo/main.cpp
+++ b/demo/main.cpp
@@ -1287,6 +1287,44 @@ void provoking(const Mesh& mesh)
 	    int(mesh.indices.size() / 3), int(pcount), double(pcount) / double(bestv) * 100.0 - 100.0, (end - start) * 1000);
 }
 
+static int reindexCompare(void* context, unsigned int lhs, unsigned int rhs)
+{
+	const Vertex* vertices = static_cast<Vertex*>(context);
+	const Vertex& lv = vertices[lhs];
+	const Vertex& rv = vertices[rhs];
+
+	float ln = lv.nx * lv.nx + lv.ny * lv.ny + lv.nz * lv.nz;
+	float rn = rv.nx * rv.nx + rv.ny * rv.ny + rv.nz * rv.nz;
+
+	// 1/1024px UV tolerance, 3 degree normal tolerance
+	return fabsf(lv.tx - rv.tx) < 1e-3f &&
+	       fabsf(lv.ty - rv.ty) < 1e-3f &&
+	       (lv.nx * rv.nx + lv.ny * rv.ny + lv.nz * rv.nz >= 0.9986f * sqrtf(ln * rn));
+}
+
+void reindexFuzzy(const Mesh& mesh)
+{
+	std::vector<PackedVertex> pv(mesh.vertices.size());
+	packMesh(pv, mesh.vertices);
+
+	std::vector<unsigned int> remap(mesh.vertices.size());
+
+	double start = timestamp();
+
+	size_t up = meshopt_generateVertexRemap(&remap[0], &mesh.indices[0], mesh.indices.size(), &pv[0], mesh.vertices.size(), sizeof(PackedVertex));
+
+	double middle = timestamp();
+
+	size_t uf = meshopt_generateVertexRemapCustom(&remap[0], &mesh.indices[0], mesh.indices.size(), &mesh.vertices[0].px, mesh.vertices.size(), sizeof(Vertex), reindexCompare, const_cast<Vertex*>(&mesh.vertices[0]));
+
+	double end = timestamp();
+
+	printf("ReindexQ : %d vertices => %d unique vertices in %.2f msec\n",
+	    int(mesh.vertices.size()), int(up), (middle - start) * 1000);
+	printf("ReindexF : %d vertices => %d unique vertices in %.2f msec\n",
+	    int(mesh.vertices.size()), int(uf), (end - middle) * 1000);
+}
+
 void nanite(const std::vector<Vertex>& vertices, const std::vector<unsigned int>& indices); // nanite.cpp
 
 bool loadMesh(Mesh& mesh, const char* path)
@@ -1468,6 +1506,8 @@ void process(const char* path)
 	spatialSort(mesh);
 	spatialSortTriangles(mesh);
 
+	reindexFuzzy(mesh);
+
 	if (path)
 		processDeinterleaved(path);
 }
diff --git a/gltf/mesh.cpp b/gltf/mesh.cpp
index d5c744a59..bf231e32a 100644
--- a/gltf/mesh.cpp
+++ b/gltf/mesh.cpp
@@ -66,6 +66,15 @@ static void transformNormal(float* res, const float* ptr, const float* transform
 	res[2] = z * s;
 }
 
+static Stream* getStream(Mesh& mesh, cgltf_attribute_type type, int index = 0)
+{
+	for (size_t i = 0; i < mesh.streams.size(); ++i)
+		if (mesh.streams[i].type == type && mesh.streams[i].index == index)
+			return &mesh.streams[i];
+
+	return NULL;
+}
+
 // assumes mesh & target are structurally identical
 static void transformMesh(Mesh& target, const Mesh& mesh, const cgltf_node* node)
 {
@@ -722,15 +731,6 @@ static void filterTriangles(Mesh& mesh)
 	mesh.indices.resize(write);
 }
 
-static Stream* getStream(Mesh& mesh, cgltf_attribute_type type, int index = 0)
-{
-	for (size_t i = 0; i < mesh.streams.size(); ++i)
-		if (mesh.streams[i].type == type && mesh.streams[i].index == index)
-			return &mesh.streams[i];
-
-	return NULL;
-}
-
 static void simplifyAttributes(std::vector<float>& attrs, float* attrw, size_t stride, Mesh& mesh)
 {
 	assert(stride >= 6); // normal + color
diff --git a/src/indexgenerator.cpp b/src/indexgenerator.cpp
index 0d53020e3..5f36d2463 100644
--- a/src/indexgenerator.cpp
+++ b/src/indexgenerator.cpp
@@ -5,6 +5,7 @@
 #include <string.h>
 
 // This work is based on:
+// Matthias Teschner, Bruno Heidelberger, Matthias Mueller, Danat Pomeranets, Markus Gross. Optimized Spatial Hashing for Collision Detection of Deformable Objects. 2003
 // John McDonald, Mark Kilgard. Crack-Free Point-Normal Triangles using Adjacent Edge Normals. 2010
 // John Hable. Variable Rate Shading with Visibility Buffer Rendering. 2024
 namespace meshopt
@@ -86,6 +87,46 @@ struct VertexStreamHasher
 	}
 };
 
+struct VertexCustomHasher
+{
+	const float* vertex_positions;
+	size_t vertex_stride_float;
+
+	int (*callback)(void*, unsigned int, unsigned int);
+	void* context;
+
+	size_t hash(unsigned int index) const
+	{
+		const unsigned int* key = reinterpret_cast<const unsigned int*>(vertex_positions + index * vertex_stride_float);
+
+		unsigned int x = key[0], y = key[1], z = key[2];
+
+		// replace negative zero with zero
+		x = (x == 0x80000000) ? 0 : x;
+		y = (y == 0x80000000) ? 0 : y;
+		z = (z == 0x80000000) ? 0 : z;
+
+		// scramble bits to make sure that integer coordinates have entropy in lower bits
+		x ^= x >> 17;
+		y ^= y >> 17;
+		z ^= z >> 17;
+
+		// Optimized Spatial Hashing for Collision Detection of Deformable Objects
+		return (x * 73856093) ^ (y * 19349663) ^ (z * 83492791);
+	}
+
+	bool equal(unsigned int lhs, unsigned int rhs) const
+	{
+		const float* lp = vertex_positions + lhs * vertex_stride_float;
+		const float* rp = vertex_positions + rhs * vertex_stride_float;
+
+		if (lp[0] != rp[0] || lp[1] != rp[1] || lp[2] != rp[2])
+			return false;
+
+		return callback ? callback(context, lhs, rhs) : true;
+	}
+};
+
 struct EdgeHasher
 {
 	const unsigned int* remap;
@@ -183,6 +224,43 @@ static void buildPositionRemap(unsigned int* remap, const float* vertex_position
 	allocator.deallocate(vertex_table);
 }
 
+template <typename Hash>
+static size_t generateVertexRemap(unsigned int* remap, const unsigned int* indices, size_t index_count, size_t vertex_count, const Hash& hash, meshopt_Allocator& allocator)
+{
+	memset(remap, -1, vertex_count * sizeof(unsigned int));
+
+	size_t table_size = hashBuckets(vertex_count);
+	unsigned int* table = allocator.allocate<unsigned int>(table_size);
+	memset(table, -1, table_size * sizeof(unsigned int));
+
+	unsigned int next_vertex = 0;
+
+	for (size_t i = 0; i < index_count; ++i)
+	{
+		unsigned int index = indices ? indices[i] : unsigned(i);
+		assert(index < vertex_count);
+
+		if (remap[index] != ~0u)
+			continue;
+
+		unsigned int* entry = hashLookup(table, table_size, hash, index, ~0u);
+
+		if (*entry == ~0u)
+		{
+			*entry = index;
+			remap[index] = next_vertex++;
+		}
+		else
+		{
+			assert(remap[*entry] != ~0u);
+			remap[index] = remap[*entry];
+		}
+	}
+
+	assert(next_vertex <= vertex_count);
+	return next_vertex;
+}
+
 template <size_t BlockSize>
 static void remapVertices(void* destination, const void* vertices, size_t vertex_count, size_t vertex_size, const unsigned int* remap)
 {
@@ -197,55 +275,49 @@ static void remapVertices(void* destination, const void* vertices, size_t vertex
 		}
 }
 
-} // namespace meshopt
-
-size_t meshopt_generateVertexRemap(unsigned int* destination, const unsigned int* indices, size_t index_count, const void* vertices, size_t vertex_count, size_t vertex_size)
+template <typename Hash>
+static void generateShadowBuffer(unsigned int* destination, const unsigned int* indices, size_t index_count, size_t vertex_count, const Hash& hash, meshopt_Allocator& allocator)
 {
-	using namespace meshopt;
-
-	assert(indices || index_count == vertex_count);
-	assert(!indices || index_count % 3 == 0);
-	assert(vertex_size > 0 && vertex_size <= 256);
-
-	meshopt_Allocator allocator;
-
-	memset(destination, -1, vertex_count * sizeof(unsigned int));
-
-	VertexHasher hasher = {static_cast<const unsigned char*>(vertices), vertex_size, vertex_size};
+	unsigned int* remap = allocator.allocate<unsigned int>(vertex_count);
+	memset(remap, -1, vertex_count * sizeof(unsigned int));
 
 	size_t table_size = hashBuckets(vertex_count);
 	unsigned int* table = allocator.allocate<unsigned int>(table_size);
 	memset(table, -1, table_size * sizeof(unsigned int));
 
-	unsigned int next_vertex = 0;
-
 	for (size_t i = 0; i < index_count; ++i)
 	{
-		unsigned int index = indices ? indices[i] : unsigned(i);
+		unsigned int index = indices[i];
 		assert(index < vertex_count);
 
-		if (destination[index] == ~0u)
+		if (remap[index] == ~0u)
 		{
-			unsigned int* entry = hashLookup(table, table_size, hasher, index, ~0u);
+			unsigned int* entry = hashLookup(table, table_size, hash, index, ~0u);
 
 			if (*entry == ~0u)
-			{
 				*entry = index;
 
-				destination[index] = next_vertex++;
-			}
-			else
-			{
-				assert(destination[*entry] != ~0u);
-
-				destination[index] = destination[*entry];
-			}
+			remap[index] = *entry;
 		}
+
+		destination[i] = remap[index];
 	}
+}
 
-	assert(next_vertex <= vertex_count);
+} // namespace meshopt
 
-	return next_vertex;
+size_t meshopt_generateVertexRemap(unsigned int* destination, const unsigned int* indices, size_t index_count, const void* vertices, size_t vertex_count, size_t vertex_size)
+{
+	using namespace meshopt;
+
+	assert(indices || index_count == vertex_count);
+	assert(!indices || index_count % 3 == 0);
+	assert(vertex_size > 0 && vertex_size <= 256);
+
+	meshopt_Allocator allocator;
+	VertexHasher hasher = {static_cast<const unsigned char*>(vertices), vertex_size, vertex_size};
+
+	return generateVertexRemap(destination, indices, index_count, vertex_count, hasher, allocator);
 }
 
 size_t meshopt_generateVertexRemapMulti(unsigned int* destination, const unsigned int* indices, size_t index_count, size_t vertex_count, const struct meshopt_Stream* streams, size_t stream_count)
@@ -263,44 +335,24 @@ size_t meshopt_generateVertexRemapMulti(unsigned int* destination, const unsigne
 	}
 
 	meshopt_Allocator allocator;
-
-	memset(destination, -1, vertex_count * sizeof(unsigned int));
-
 	VertexStreamHasher hasher = {streams, stream_count};
 
-	size_t table_size = hashBuckets(vertex_count);
-	unsigned int* table = allocator.allocate<unsigned int>(table_size);
-	memset(table, -1, table_size * sizeof(unsigned int));
-
-	unsigned int next_vertex = 0;
-
-	for (size_t i = 0; i < index_count; ++i)
-	{
-		unsigned int index = indices ? indices[i] : unsigned(i);
-		assert(index < vertex_count);
-
-		if (destination[index] == ~0u)
-		{
-			unsigned int* entry = hashLookup(table, table_size, hasher, index, ~0u);
-
-			if (*entry == ~0u)
-			{
-				*entry = index;
+	return generateVertexRemap(destination, indices, index_count, vertex_count, hasher, allocator);
+}
 
-				destination[index] = next_vertex++;
-			}
-			else
-			{
-				assert(destination[*entry] != ~0u);
+size_t meshopt_generateVertexRemapCustom(unsigned int* destination, const unsigned int* indices, size_t index_count, const float* vertex_positions, size_t vertex_count, size_t vertex_positions_stride, int (*callback)(void*, unsigned int, unsigned int), void* context)
+{
+	using namespace meshopt;
 
-				destination[index] = destination[*entry];
-			}
-		}
-	}
+	assert(indices || index_count == vertex_count);
+	assert(!indices || index_count % 3 == 0);
+	assert(vertex_positions_stride >= 12 && vertex_positions_stride <= 256);
+	assert(vertex_positions_stride % sizeof(float) == 0);
 
-	assert(next_vertex <= vertex_count);
+	meshopt_Allocator allocator;
+	VertexCustomHasher hasher = {vertex_positions, vertex_positions_stride / sizeof(float), callback, context};
 
-	return next_vertex;
+	return generateVertexRemap(destination, indices, index_count, vertex_count, hasher, allocator);
 }
 
 void meshopt_remapVertexBuffer(void* destination, const void* vertices, size_t vertex_count, size_t vertex_size, const unsigned int* remap)
@@ -362,33 +414,9 @@ void meshopt_generateShadowIndexBuffer(unsigned int* destination, const unsigned
 	assert(vertex_size <= vertex_stride);
 
 	meshopt_Allocator allocator;
-
-	unsigned int* remap = allocator.allocate<unsigned int>(vertex_count);
-	memset(remap, -1, vertex_count * sizeof(unsigned int));
-
 	VertexHasher hasher = {static_cast<const unsigned char*>(vertices), vertex_size, vertex_stride};
 
-	size_t table_size = hashBuckets(vertex_count);
-	unsigned int* table = allocator.allocate<unsigned int>(table_size);
-	memset(table, -1, table_size * sizeof(unsigned int));
-
-	for (size_t i = 0; i < index_count; ++i)
-	{
-		unsigned int index = indices[i];
-		assert(index < vertex_count);
-
-		if (remap[index] == ~0u)
-		{
-			unsigned int* entry = hashLookup(table, table_size, hasher, index, ~0u);
-
-			if (*entry == ~0u)
-				*entry = index;
-
-			remap[index] = *entry;
-		}
-
-		destination[i] = remap[index];
-	}
+	generateShadowBuffer(destination, indices, index_count, vertex_count, hasher, allocator);
 }
 
 void meshopt_generateShadowIndexBufferMulti(unsigned int* destination, const unsigned int* indices, size_t index_count, size_t vertex_count, const struct meshopt_Stream* streams, size_t stream_count)
@@ -406,33 +434,9 @@ void meshopt_generateShadowIndexBufferMulti(unsigned int* destination, const uns
 	}
 
 	meshopt_Allocator allocator;
-
-	unsigned int* remap = allocator.allocate<unsigned int>(vertex_count);
-	memset(remap, -1, vertex_count * sizeof(unsigned int));
-
 	VertexStreamHasher hasher = {streams, stream_count};
 
-	size_t table_size = hashBuckets(vertex_count);
-	unsigned int* table = allocator.allocate<unsigned int>(table_size);
-	memset(table, -1, table_size * sizeof(unsigned int));
-
-	for (size_t i = 0; i < index_count; ++i)
-	{
-		unsigned int index = indices[i];
-		assert(index < vertex_count);
-
-		if (remap[index] == ~0u)
-		{
-			unsigned int* entry = hashLookup(table, table_size, hasher, index, ~0u);
-
-			if (*entry == ~0u)
-				*entry = index;
-
-			remap[index] = *entry;
-		}
-
-		destination[i] = remap[index];
-	}
+	generateShadowBuffer(destination, indices, index_count, vertex_count, hasher, allocator);
 }
 
 void meshopt_generateAdjacencyIndexBuffer(unsigned int* destination, const unsigned int* indices, size_t index_count, const float* vertex_positions, size_t vertex_count, size_t vertex_positions_stride)
diff --git a/src/meshoptimizer.h b/src/meshoptimizer.h
index 295324c78..54320c178 100644
--- a/src/meshoptimizer.h
+++ b/src/meshoptimizer.h
@@ -74,6 +74,19 @@ MESHOPTIMIZER_API size_t meshopt_generateVertexRemap(unsigned int* destination,
  */
 MESHOPTIMIZER_API size_t meshopt_generateVertexRemapMulti(unsigned int* destination, const unsigned int* indices, size_t index_count, size_t vertex_count, const struct meshopt_Stream* streams, size_t stream_count);
 
+/**
+ * Experimental: Generates a vertex remap table from the vertex buffer and an optional index buffer and returns number of unique vertices
+ * As a result, all vertices that are equivalent map to the same (new) location, with no gaps in the resulting sequence.
+ * Equivalence is checked in two steps: vertex positions are compared for equality, and then the user-specified equality function is called (if provided).
+ * Resulting remap table maps old vertices to new vertices and can be used in meshopt_remapVertexBuffer/meshopt_remapIndexBuffer.
+ *
+ * destination must contain enough space for the resulting remap table (vertex_count elements)
+ * indices can be NULL if the input is unindexed
+ * vertex_positions should have float3 position in the first 12 bytes of each vertex
+ * callback can be NULL if no additional equality check is needed; otherwise, it should return 1 if vertices with specified indices are equivalent and 0 if they are not
+ */
+MESHOPTIMIZER_EXPERIMENTAL size_t meshopt_generateVertexRemapCustom(unsigned int* destination, const unsigned int* indices, size_t index_count, const float* vertex_positions, size_t vertex_count, size_t vertex_positions_stride, int (*callback)(void*, unsigned int, unsigned int), void* context);
+
 /**
  * Generates vertex buffer from the source vertex buffer and remap table generated by meshopt_generateVertexRemap
  *
@@ -722,6 +735,8 @@ template <typename T>
 inline size_t meshopt_generateVertexRemap(unsigned int* destination, const T* indices, size_t index_count, const void* vertices, size_t vertex_count, size_t vertex_size);
 template <typename T>
 inline size_t meshopt_generateVertexRemapMulti(unsigned int* destination, const T* indices, size_t index_count, size_t vertex_count, const meshopt_Stream* streams, size_t stream_count);
+template <typename T, typename F>
+inline size_t meshopt_generateVertexRemapCustom(unsigned int* destination, const T* indices, size_t index_count, const float* vertex_positions, size_t vertex_count, size_t vertex_positions_stride, F callback);
 template <typename T>
 inline void meshopt_remapIndexBuffer(T* destination, const T* indices, size_t index_count, const unsigned int* remap);
 template <typename T>
@@ -930,6 +945,19 @@ inline size_t meshopt_generateVertexRemapMulti(unsigned int* destination, const
 	return meshopt_generateVertexRemapMulti(destination, indices ? in.data : NULL, index_count, vertex_count, streams, stream_count);
 }
 
+template <typename T, typename F>
+inline size_t meshopt_generateVertexRemapCustom(unsigned int* destination, const T* indices, size_t index_count, const float* vertex_positions, size_t vertex_count, size_t vertex_positions_stride, F callback)
+{
+	struct Call
+	{
+		static int compare(void* context, unsigned int lhs, unsigned int rhs) { return (*static_cast<F*>(context))(lhs, rhs) ? 1 : 0; }
+	};
+
+	meshopt_IndexAdapter<T> in(NULL, indices, indices ? index_count : 0);
+
+	return meshopt_generateVertexRemapCustom(destination, indices ? in.data : NULL, index_count, vertex_positions, vertex_count, vertex_positions_stride, &Call::compare, &callback);
+}
+
 template <typename T>
 inline void meshopt_remapIndexBuffer(T* destination, const T* indices, size_t index_count, const unsigned int* remap)
 {