Pytorch 3D Mesh Fitting Symmetry and Feature Matching

jajajakob: <p><em>if this is the wrong place to post about this please let me know. I wasn’t sure which category would fit best.</em></p>
<p>Hey everyone,</p>
<p>(Disclaimer: I don’t really have any background in machine learning and am approaching this from a visual effects TD perspective… so bear with me ^^)</p>
<p>I’m currently trying to develop a mesh fitting algorithm to be able to morph between two 3d meshes with different topologies using the chamfer function in pytorch3d.</p>
<p>I followed the <a href="https://github.com/facebookresearch/pytorch3d/blob/main/docs/tutorials/deform_source_mesh_to_target_mesh.ipynb" rel="noopener nofollow ugc">tutorials</a> provided and managed to set everything up in my 3d software (Houdini). I was able to successfully morph between different meshes.</p>
<p>I now want to define certain key points / features which appear in the source as well as target mesh to be able to create a cleaner transition. (e.g. match eyes to eyes, nose to nose when fitting two different heads)<br>
I found the <a href="https://pytorch3d.readthedocs.io/en/latest/modules/loss.html" rel="noopener nofollow ugc">“loss.chamfer_distance”</a> function takes weights as an argument but I don"t understand how to use them or whether this is even useful for my problem. Can someone explain what exactly this is supposed to do:</p>
<p>Explanation from pytorch3d docs:</p>
<blockquote>
<p>weights – Optional FloatTensor of shape (N,) giving weights for batch elements for reduction operation.</p>
</blockquote>
<p>I also wondered if it was possible to fit the mesh while maintaining topological symmetry of the source mesh or how I would go about implementing this.</p>
<p>I’m very thankful for any pointers on how to approach any of my two problems!</p>
<p>Thanks in advance!</p>
<p>Best,</p>
<p>Jakob</p>

ptrblck: <p>Take this explanation with a grain of salt, as I’m neither deeply familiar with your use case nor with PyTorch3D.</p>
<p>The description:</p>
<blockquote>
<p><strong>weights</strong> – Optional FloatTensor of shape (N,) giving weights for batch elements for reduction operation.</p>
</blockquote>
<p>sounds ambiguous, as apparently two reductions are used:</p>
<blockquote>
<p><strong>batch_reduction</strong> – Reduction operation to apply for the loss across the batch, can be one of [“mean”, “sum”] or None.</p>
</blockquote>
<blockquote>
<p><strong>point_reduction</strong> – Reduction operation to apply for the loss across the points, can be one of [“mean”, “sum”].</p>
</blockquote>
<p>Based on the shape of <code>weigths</code> I would assume that it’s used in the <code>batch_reduction</code> since each sample is supposed to get a weight value.</p>
<p>Based on your description I understand that you would rather want to add weights to specific points instead. If so, I would have tried to apply the weighting to the unreduced loss, but it doesn’t seem that the reduction option provide the <code>'none'</code> option.<br>
In that case you could try to implement your custom weighted <code>chamfer_distance</code> by copying the implementation from <a href="https://github.com/facebookresearch/pytorch3d/blob/a6508ac3dfaaf59d8bdce176bfbafad94c1d0604/pytorch3d/loss/chamfer.py#L113-L219">here</a> and add the point weighting manually.</p>

jajajakob: <p>Thanks a lot for the explanation!<br>
I believe you are right considering the weight use for batching.<br>
While this isn’t the best case outcome i’m happy to have understood it a little more and will give implementing it a shot in the next few days.</p>
<p>One more thing:<br>
could you elaborate on what you mean by “unreduced loss”?</p>
<p>Thanks again! =)</p>

ptrblck: <p>The native PyTorch loss functions, such as <code>nn.CrossEntropyLoss</code>, allow you to use <code>reduction='none'</code>, which will calculate the loss for each sample/pixel etc. without calculating the <code>sum</code> or <code>mean</code> (default) of it. The result would thus not be a scalar value but a tensor which you could reduce afterwards (e.g. by calling <code>mean</code> on it). This allows for more flexibility as you would have the chance to add an elementwise weighting etc. before the final reduction.</p>

jajajakob: <p>understood! awesome thank you for the detailed explanation</p>