# Review of “Photometric DIC: A Unified Framework for Global Stereo Digital Image Correlation based on the Construction of Textured Digital Twins”

Raphaël Fouque, Robin Bouclier, Jean-Charles Passieux, Jean-Noël Périé, Jean-François F Witz, Julien Réthoré

► To cite this version:

Raphaël Fouque, Robin Bouclier, Jean-Charles Passieux, Jean-Noël Périé, Jean-François F Witz, et al.. Review of “Photometric DIC: A Unified Framework for Global Stereo Digital Image Correlation based on the Construction of Textured Digital Twins”. 2022. hal-03704301

HAL Id: hal-03704301

https://hal.science/hal-03704301

Submitted on 7 Jul 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Identifiers

Open Review
OAI hal-03704301
Reviewed Article
DOI 10.46298/jtcam.7467

History

Review 7 Sep, 2021
Rebuttal 20 Feb, 2022

Licence

CC BY 4.0
©The Authors

# Review of “Photometric DIC: A Unified Framework for Global Stereo Digital Image Correlation based on the Construction of Textured Digital Twins”

Raphaël FOUQUE<fnref n="1" />,<fnref n="2" />, ROBIN BOUCLIER<fnref n="1" />,<fnref n="3" />, JEAN-CHARLES PASSIEUX<fnref n="1" />, JEAN-NOËL PÉRIÉ<fnref n="1" />, JEAN-FRANÇOIS WITZ<fnref n="4" />,R, and JULIEN RÉTHORÉ<fnref n="5" />,E

## Review of version 1

Permalink: hal-03218388v1

**Author** We are pleased to submit a corrected version of our manuscript to be published in the Journal of Theoretical, Computational and Applied Mechanics. We would like to thank the reviewers for their positive reviews and large list of valuable comments and suggestions. We apologize for the quite long delay we needed to address the large number of question and comments. We did our best to address all of them. We hope that the manuscript is now to the standard of the journal and are looking forward to hearing from you.

## Reviewer 1 (Jean-François WITZ)

**Reviewer** This work proposes an adaptation of the work of Computer Vision (CV) around the reconstruction of shapes by taking images under different angles, under the assumption of having several cameras observing the structure under different angles with a simultaneous image taking and an identical response. Instead of the classical conservation of gray levels which leads to the optical flow, the authors propose to separate the representation into two states, the texture called here albedo and the shape. The process consists in reconstructing the albedo and the shape by a numerical twin using the classical equations of radiometry, simplified via the postulate of the isotropy of the reflection (Lambertian lighting) and a distant point source of light, which allows them to simplify the radiometric equations. They start with an initialization of the shape positioning in space using traditional methods, and then calculate the albedo directly from the rotations of the stage. They then use a Gauss-Newton algorithm on the extrinsic parameters, as well as on the shape-related DoFs using a fixed point, following the traditional multi-scale FE-DIC process. The process is used on two sets of images containing the same amount of information and angular deviations to verify that the method is not biased.

**Author** We would like to thank the reviewer for carefully reading our work and for the quite long list of valuable comments and questions that we tried to consider with the greatest care in the revised

<footnote n="1">¹ Institut Clément Ader, CNRS UMR 5312, Université de Toulouse, INSA/ISAE/Mines Albi/UPS, 3 rue Caroline Aigle, 31400 Toulouse, France</footnote>

<footnote n="2">² DGA Aeronautical Systems, 47 rue Saint Jean, BP 93123, 31131 Balma Cedex, France</footnote>

<footnote n="3">³ Institut de Mathématiques de Toulouse, CNRS UMR 5219, Université de Toulouse, INSA/UT1/UT2/UPS, 135 avenue de Rangueil, 31077 Toulouse Cedex 04, France</footnote>

<footnote n="4">⁴ University of Lille, CNRS, Centrale Lille, Laboratoire de Mécanique, Multiphysique, Multiéchelle (LaMcube), UMR CNRS 9013, 59000, Lille, France</footnote>

<footnote n="5">⁵ Research Institute in Civil and Mechanical Engineering (Gem), CNRS UMR 6183 CNRS/Ecole Centrale de Nantes/Université de Nantes, F-44 321 Nantes, France</footnote>

<footnote>R Reviewer</footnote>

<footnote>E Editor</footnote>
manuscript.

We believe that in this accurate synthesis and in the remaining of the review, the Referee missed a substantial part of what makes, to us, the novelty of the manuscript. We not only suggest a way to measure a shape thanks to optical measurements based on an intrinsic texture identification, we also apply this method to track displacements in a general case and apply it to the measurement of a large rigid-body rotation. A so-called intrinsic texture, or albedo, had never been used before for the latter in the DIC community. In computer vision, based on the reviewed literature, we have not read about tracking displacements based on such a method. Admittedly, the corresponding part is quite short and not enough emphasis had been put on displacement measurements in the introduction and conclusion. For this reason, we modified them to make our contributions appear clearly.

**Reviewer** Overall opinion: There is no doubt that this method is very innovative in the field of mechanics, and the work to include it in the traditional DIC formalism fully justifies considering this work as very original and therefore completely acceptable for publication in this journal. I find, however, that a few questions need to be answered:

* What does your method bring to the table compared to free software like meshroom?

* Do they use similar methods?

* How do they differ from your approach?

* Is it possible to include your contributions in such a framework (meshroom)?

Personally, I don't have the time to go through all the available publications to know if your method really brings an added value, so it seems to me very important to specify it well.

I am sure that your work is not just rewriting a work already done in CV in the field of mechanics, but I think that you will be able to justify it much better than me, it does not seem to be done here.

**Author** As we mentioned above, from a mechanical point of view, our objective is, starting from an analysis-suitable FE mesh, to measure the shape, the albedo and above all the displacement in order to bridge simulation and experiments. Our objective is therefore very different from what is usually encountered in computer vision. We added a sentence in the introduction to stress this point. Regarding the particular case of Meshroom, and in a few words, it is a very powerful tool in the sense that no prior knowledge on the object shape is needed, and, thanks to an algorithm similar to SIFT, no target is needed either, as keypoints are extracted from pictures to allow the cameras to be calibrated. However, it is based on standard optical flow conservation (no albedo) and, here again, only enables the experimenter to measure a shape (no displacement measurement).

**Reviewer** The scientific quality is very good and the explanatory approach extremely clear. I find that overall the emphasis is not necessarily put enough on the existing alternatives in terms of parts metrology, indeed, the traditional methods of metrology are not mentioned at all (Coordinate-measuring machine, 3D-scanner, ...). I think that it would be important to explain very synthetically the different existing methods to measure parts, without contact, with contact, in order to be able to position the method, with what traditionally exists in metrology. A simple example is the use of 3D scanners (methods of projection of patterns such as patterns of fringe or clouds of points) which allow measurements of shapes very quickly. What does the method bring in comparison to these tools, which allow the reconstruction but also the plating of a texture on the model? It is obvious that to have a Lagrangian transformation in all directions and to be able to follow parts that are straining, is a perfectly acceptable answer. This allows you to situate your method in relation to the existing one and quickly highlight a very strong added value.

**Author** We fully agree with these two last sentences. Indeed, we differ from dimensional metrology approaches in this ability to track displacements/strains of the object in addition to its shape and albedo. Reading the reviewer's comments, we understand that this point, however fundamental, was not clear enough in the original version of the paper. We tried to make this point clearer in the introduction and in the conclusion of the revised manuscript. We think that the body of the paper was already quite clear, especially thanks to the organization of the paper: Section 3 calibration (shape + albedo) and Section 4 (displacement)
**Reviewer** Personally, I think that adding the DoF of the hole position on the plate, because it has not been measured, is really not necessary and shows more a methodological bias of the experimental process (it is quite simple to machine a part with a perfectly controlled hole position with a 3-axis numerical control and even with a conventional milling machine), than a real added value. I think that imposing the right value at the beginning is quite acceptable and avoids putting your validation process at fault, as a simple vertical profile projector will most probably give much better results when it comes to hole positioning and is most certainly available in your metrology structures, knowing that if I think about the process, then I imagine that it consists of a remeshing step from the CAD, or a local strain of the mesh with RBF type functions on the contour of the circle from the rigid body displacements in any case, all this is not explained enough as it is and does not seem to me to be of major interest for the method, if you still want to keep it, it would be better to explain in detail the method you used, because for the mesh, it is not a rigid body movement, but a strain, so it is debatable to consider it as an extrinsic parameter and not a change of shape in the plane, for me extrinsic parameter, it is the rigid body movements, but the position of the hole in relation to the parts, is not one.

**Author** As mentioned by the reviewer himself, we consider this point to be a bit of a detail. We mentioned this detail for the sake of transparency. Since it seems to create more confusion than clarity from the reviewer, we decided to remove this detail from the revised version of the paper.

**Reviewer** Detailed recommendations: The transition from Equation (12) to (13) is a bit rough, and would require some transitional steps, or at least an explicit reminder of the change in variable of integration involving the need for the calculation of the Jacobian.

**Author** The transition form Eq. (12) to (13) may appear a bit complicated because we wanted to get a similar form as in CV, where the weighting defined in the images is introduced, i.e. $J_i^c$. However, note that $J_i^c \circ P_i^c$ could be simply substituted by $\det(\nabla P_i^c)$ in Eq. (13), thus going from (12) to (13) is a simple change of variable through mapping $P_i^c$. A remark on this point has been added below Eq. (13) in the revised version of the paper.

**Reviewer** Why not use composition for $L_e$ and $\Phi_D$ in Equation (18)? Does $r_i^c$ not have to be composed? It is not clear to me if this term is updated automatically when the basis is changed or if it does not have to be involved. It seems strange to me that it is the same in both reference frames, it should be specified.

**Author** The reviewer is right: $r_i^c$ is defined over the true (deformed) geometry so it needs to be composed with $\Phi_D$ in Eq. (18). Actually, the mistake emanates from Eq. (12) where $r_i^c$ needs to be composed with $\beta_i^c$. Everything has been corrected in the revised version of the paper.

**Reviewer** Equation (19) is quite difficult to read, the first line contains compositions and the second line replaces the compositions by the definition of $\Phi_D$, I think it would be nice to choose either the composition or the expression of $\Phi_D$ in the functions, note that I have a preference for the composition.

**Author** The reviewer is right. Thanks. Eq. (18) has been modified as suggested. For the sake of standardization, Eqs. (20) and (21) have also been modified accordingly.

**Reviewer** I also find it a bit strange that the visibility function $V_i^c$ is positioned in the middle of the equation, as it is a term that comes to simplify the integration, it seems to me more coherent to put it before the Jacobian, as it is a regularization term and it is not really from the formulation.

**Author** We understand the comment. However, we need here to compose $V_i^c$ with $\Phi_D$ since $V_i^c$ is defined over the true (deformed) geometry. In other words, the visibility function changes when the specimen deforms. Hence, this is for the sake of compactness that we did not put $V_i^c$ before the Jacobian. That being said, we take into account, in the revised paper, the reviewers' comment in Eq. (15) where $V_i^c$ is introduced for the first time.

**Reviewer** A lot is said about discretization and integration, it is not easy to figure out, I think a simplified diagram with two positions of the same T3 element on an object, with different angles, projected on a camera would illustrate this effectively. I think that's what Figure 14 is trying to do, but it
didn't really enlighten me. You could take a cube with two triangles per side, rotate it and display
the visibility function, as well as the integration domain in the camera frame of reference.

**Author** The integration is specified in Section 3.3. In contrast, Fig. 14 is in the appendix. Fig. 14 does not concern the integration. It illustrates the physical interpretation of the different terms in *J*. The integration strategy is quite simple and is strongly based on previous developments (see [Pierré et al. (2017)]). We take an homogeneous number of points per unit length over the specimen. Thus the number of integration points for each element simply depends on its dimensions in the mesh coordinate system. We do not think a figure would add much. In addition, the paper is already quite long so we decided not to add such a figure. We recall that the object of the paper is the theoretical derivation of a new formulation for SDIC.

**Reviewer** I find that the point made about using the visibility function would need a little more discussion. I think indeed that it is a key point of the performance of the algorithm, it could be very interesting to quantify it (performance with and without).

**Author** Not sure to fully understand this question. Which point are you talking about and where is it in the paper? Eq. (16)? Page 12? Page 14? Page 21? Anyway, let us remind at this stage that the visibility function $V_i^c$, as defined in the most accurate manner from a theoretical point of view in Eq. (16), naturally appears when making the choice of integrating over the whole deformed ROI $\tilde{\Omega}$ instead of the corresponding visible part $\tilde{\Omega}_{vis}^c$ (indicator function). In other words, we do not integrate the residual in the parts of the specimen that are not seen by the camera. As a consequence, it would not be possible to run the algorithm without it. The delicate task is only to compute it in practice. Here, since the specimen is convex, we propose to rely on the sign of $\langle n(\bar{X}), r_i(\bar{X}) \rangle$. Then, we add a threshold to this criterion to remove some points that are seen with an angle larger than 66 degrees, because, as illustrated in Computer Vision (Birkbeck2006), results are expected to be more accurate. The maximum value is defined here so that we do not end up with a singular Hessian matrix. This is consistent with well-known developments in experimental mechanics regarding the angles between the cameras for a good stereo-based measurement (see, e.g., balcaen17dicuncer). Some comments along with reference (balcaen17dicuncer) below have been added in pages 13 and 14 to precise this point.

**Reviewer** On the shape part, although the k-mean clustering is a simple and general way to find the right normals at the nodes, I find it a pity not to use the information directly from the CAD.

**Author** It is true. However, although the developed method is very generic to incorporate any geometrical description (CAD, FE-based, etc), we underline that the geometrical input in the applications conducted in this work is a FE mesh. As a result, we do not have in hand the CAD data and thus, we needed to resort to a k-means clustering algorithm. We actually adopt the point of view of the experimenter in this work who starts with a FE mesh provided by the analysts and who needs to perform a measurement using this FE mesh to communicate with simulation. A remark regarding this positioning has been added in the revised paper at the end of Section 3.1.

**Reviewer** I also find that the DoFs to be identified are not clearly stated. If I understood correctly, the node normals, calculated using an average of the normals of the neighboring elements, then classified by k-mean, are used as out-of-plane translation directions of the target surface at each point of the mesh. Is this normal updated? Doesn't having so many DoF generate spurious strains that should be quantified?

**Author** That is not what is written: node normals are *usually* obtained as an average of the normals of neighbouring elements and we pointed out that this definition is not completely satisfactory, especially for nodes on edges or corners of the mesh. For a given node, the k-means algorithm was run directly over all the available normals of neighbouring elements. Overall, there are three types of nodes: those that lie within the face and have one dof (normal direction), those that are on an edge to which we attribute two dofs in the two perpendicular directions of the edge, and those that are located on a vertex that have 3 dofs (so in all directions). We did not encounter instabilities in our numerical tests with such a parameterisation. We recall that we have a large amount of data which may explain the good results that we obtained. As for the update of the
normal, it is clearly explained in Section 3.2 that we start with Ω close to $\tilde{\Omega}$ so we do not need to update the normal (see equation in blue at the beginning of Section 3.2). We have added complements in the paragraph “shape” in Section 3.4.1 to make this point clearer.

**Reviewer** I think it is a pity to go directly to a mesh without mentioning the possibility to keep the definition of surfaces by NURBS, knowing that R. Bouclier has worked a lot on this subject (as well as the possibility to make mechanical computations in IGA), mentioning the possibility to use a parametric representation with much less DoF seems to me necessary. I am well aware that this is not the main point of this article which presents a generic method of shape registration from 2D images, but explaining why you made this choice, when you are one of the few teams who have the skills not to do so, seems to me relevant, because in ill-posed problems, the reduction of the solution search space is probably one of the most important points.

**Author** Thanks for your consideration of our team! Indeed, it would be a good idea to involve a spline model for the geometrical and kinematic description, which would have the interest to (i) establish a full link between the geometrical description in CAD, measurement, and simulation using IsoGeometric Analysis (IGA), and (ii) naturally regularise the underlying measurement problems since it involves reduced design spaces. However, we decided to consider a FE mesh as the geometrical input in this work since it remains the common practice in global-DIC. We actually adopt the point of view of the experimenter who starts with a FE mesh provided by the analysts and who needs to perform a measurement using this FE mesh to communicate with standard FE-based simulation. Finally, as for regularization, let us notice that our framework increases the number of data with the multiview set-up so it can be used with a rather fine FE mesh while alleviating the problem ill-posedness. A remark along with suitable references have been added at the end of Section 3.1 to precise this point.

**Reviewer** I didn't necessarily understand what happens to the integration on the half-sphere, I have the impression that it is taken into account by the visibility function and the projector, but as this one is never explained, I'm not sure I really understood everything, regarding its construction. I would really appreciate if the construction of the projectors and the integration could be explained, it would allow a better understanding.

**Author** Again, not sure to fully understand this question. If the reviewer refers to equation (4), the quadrature on the half-sphere is done analytically based on two main assumption: the Lambertian model and the distant point light source. It neither depends on the visibility function nor on the camera model.

**Reviewer** I also think it is a pity that the only real error measurement is at the pixel, because if this is the place where things are measured, an object can be measured with many other tools and so it should not be difficult to calculate distances to the measured shape or even distances to the analytical starting shape.

**Author** The first track, which would consist in comparing our measurements to reference ones (for example to those provided by a coordinate-measuring machine) is natural, but debatable. If it would allow us to be reassured, it brings a new set of questions (is the reference reliable, which reference system to adopt, how to compare measurements of different resolutions and spatial resolutions, etc...). On the other hand, it requires the development/use of tools for estimating the distance fields between point clouds, which introduces new uncertainties. The second track, that of image synthesis from known references, seems more attractive. One could indeed generate several synthetic images from a predefined CAD and texture. The shape and texture reconstructed from the proposed approach could then be compared to previous ground truth data. However, here again, many ingredients will have to be integrated to ensure that no bias is introduced in the analysis. In both cases the work required is well beyond the scope of this paper. What we wanted to do here was simply to estimate the bias resulting from the use of different batches of images.

**Reviewer** One can also ask the question of the field of view. One can consider that the camera is placed very far from the object and the plate with the hole does not cover the whole image very well. This choice clearly limits the resolution, but also greatly limits the impact of radial distortions
that you chose not to model. I think this could be discussed.

**Author** We needed to see the calibration target on which the sample is placed. This is essentially what constrained the field of view. It is true that this also limits the effect of distortions that would not be well taken into account with the pinhole model used herein. However, as mentioned earlier, the pinhole camera model without distortions was practically true with the set-up used, as the calibration reprojection errors did not change much with or without distortions.

**Reviewer** In fact, there is no way to know if the errors are due to the algorithm, to the difference between the physical part made and its digital twin which must be updated, to take into account reality, or to the identification of the albedo. This is for me the major problem of this article. A simple way to involve only the algorithm, is to generate a virtual scene from the chosen mesh. This task may seem too big for a review, but it can be done quite easily with Blender and the BlenDIC plugin from eikosim. I think that it would be a great added value to add this analysis to your work, because you will be able to use the ad-hoc lighting models, and thus not have any hypothesis on the identification of the albedo in relation to your experimental set-up, nor on the uncertainty of measurement of the grey levels related to the noise of the camera.

**Author** Clearly, this is an interesting avenue, but as mentioned by the reviewer, it is far beyond the scope of this article. Blender is mainly about making the direct problem of the inverse problem we are trying to solve. We have to keep in mind that there are other sources of uncertainty that we will not be taking into account anyway. We should not risk falling into the inverse crime. We consider that this article already brings a significant contribution to the experimental mechanics community. We recall that no stereo software exists that is capable to follow even the simple rigid body motion of the test considered in this paper.

**Reviewer** At first reading one can find figure 12 a bit worrying, because if it is true that there is an indeterminacy on the local positioning of the albedo in the plane of the object, this position should not be so sensitive because the free edges should regularize and as the shape is, from what I think to have understood, only modified from an evolution according to the normals of the faces calculated at the nodes, there should not be this problem. But, there may be something I am missing. It should be relativized, because almost all the errors, even in this direction is less than 0.5Px, so it is very correct, it should be pointed out! On the other hand, this reinforces for me the need to calculate the transformation and strains of the identified shape in relation to a reference shape (synthetic or measured) in order to see what their order of magnitude is in relation to what would be a mechanical test on this structure. In the same way, since you have implemented Garcia's stereo method, you could quite easily compare the performances using the strategy proposed in Figure 1-a, this would allow you to situate your method in relation to the classical mechanical method. It is obvious that you can use a FEM grid instead of subsets. This would allow for example to demonstrate that there is an angular limit for this method, which is not the case for yours.

**Author** First, it is important to understand that moving a node in the middle of a surface of a mesh along a tangent direction to this surface does not actually change the shape. In other words, a given surface can be exactly represented using many different meshes (no uniqueness). Then, yes, the shape is only modified along the normal of the faces except, precisely, for edges and corners (see paragraph Shape in Section 3.4.1).

**Reviewer** Moreover, it is said that the backward projection is not used, but this one enters in the construction of the visibility function, which does not seem to me sufficiently described. As this function is used in the definition of the integration domain, it should not be complicated to transport it to the image repositories to illustrate what it represents on the image, as you have chosen the image as a tool for evaluating the results.

**Author** As already said to answer a previous comment of the reviewer, the backward projection is used in (16) to define the visibility function in the most accurate manner from a theoretical point of view. The visibility function is only introduced to say that we do not integrate the residual in the parts of the specimen that are not seen by the camera. In practice, we perform as explained in Section
3.2. to compute the visibility function. The backward projection is not used. We tried to make this point clearer in Section 3.2 in the revised paper.

**Reviewer** I also find it quite strange to increase the visibility perimeter by setting a threshold > 0 while you reduce the integration domain to avoid oscillations, this calls for a comment.

**Author** As already said to answer a previous comment of the reviewer, we add a threshold to the criterion to remove some points that are seen with an angle larger than 66 degrees, to increase the accuracy on the results. This is consistent with well-known developments in experimental mechanics regarding the angles between the cameras for a good stereo-based measurement (see, e.g., (balcaen17dicuncer)). Some comments along with reference (balcaen17dicuncer) have been added in pages 13 and 14 to precise this point.

**Reviewer** I think that in view of your conclusions on the usefulness of this visibility function, taking some time to better explain its construction would be beneficial to the manuscript, even if it is in the appendix.

**Author** We hope that the modifications mentioned above allows a better understanding regarding the visibility function. The appendix is not related to the visibility function. It is related to the physical interpretation of different terms in the weighting $\mathcal{J}$.

**Reviewer** As a research engineer, I regret that there is no access to the source codes of this work, I would have really appreciated to be able to redo these calculations, or to go and see how the operators that I did not perfectly understand are constructed. I imagine that this work is based on Pyxel, it would be really a good thing to include these algorithms with this example in Pyxel, I think it would show a real willingness to deploy this tool in our community, something that will be quite difficult without it, because I estimate at least that it would take a full time thesis on this subject for anyone who would try this method, which is quite prohibitive to allow the dissemination of this work. To be able to leave this article with a working source code would therefore be a real plus, and could undoubtedly be the object of another valorization in a journal with a review committee. I am fully aware that this is very time consuming, I think it would be nice to put it in perspective, if you feel like it.

**Author** We totally agree with the reviewer on this question. Open-source is in our culture, since, as the reviewer pointed out, we are the developers of the open-source FE-DIC library pyxel. However, this work is at a very low level of maturity and deserves more work and robustness before being made available to the community. Let us recall that this manuscript is our very first article on a new stereo method, and it is not published yet.

**Reviewer** Moreover you say you have reimplemented Garcia's work, of which there is no open implementation, it would be really a good thing for all those who want to get into stereo, to have access to this kind of tools, because CV tools like OpenCV for example, work like a black box, not allowing to modify anything, it is for example not possible to implement another model of camera, without going to make C++ of very high flight, which makes that today, there is only the toolbox of Matlab to my knowledge to make 3D reconstruction by mastering all the assumptions, and it is a pity.

**Author** It's definitely on the todo list, but, to us, it is a bit of a side issue.

**Reviewer** I also think that this work is a very good opportunity to set up a new way of doing shape reconstruction where the geometrical model would be at the heart of the strategy. I am thinking, for example, of augmented reality for surgery, where strain models are identified in 3D and then recalibrated in stereo during surgery. There is a large bibliography on this subject, the most advanced work that I know of on this subject is that of S. Cottin's team at INRIA. Moreover it would allow you to talk about what exists in surface reconstruction from 3D measurements, such as the work of our team directly modeling geometry in B-Splines, this is only a suggestion, don't see any constraint. I think however, that the prospect of having an accurate in-situ measurement for surgery based on data assimilation using the target shape model with different rendering implementations depending on the observed modalities is extremely attractive! Knowing
in addition that your team is very competent in model reduction, this would open up great perspectives!

**Author** This perspective, although not specific to the method proposed here, does indeed seem quite interesting. We do not believe that it calls for a change in the article.

**Reviewer 2 (Anonymous)**

**Reviewer** In this contribution the authors present a novel photometric DIC approach introducing concept from the computer vision community into the experimental mechanics one. Hereby, the central idea is not to rely on the general known grey level conservation assumption but to design an algorithm that is based on irradiance equations. The objective of the paper is clear and the novelty for the experimental mechanics world is direct, but before the paper can be fully accepted for publication I would like some key issues to be tackled.

**Author** We would like to thank the reviewer for such a positive feedback and for providing us with insightful comments and suggestions. All of them have been taken into account in the revised manuscript.

**Reviewer** The introduction does not really read fluently and in fact I could only understand the considerations of small deformations after going through the entire paper. My confusion relates to the usage of the wording deformation whereby in my opinion the aim is to measure displacements when the object is subject to large rigid body motion (eventually on top of internal non-rigid deformation). Moreover, whereas incremental standard DIC procedures can deal with large rigid body motions when occurring in the field of view, the use case here is out of plane displacements within a multiple camera setup. Standard DIC will fail because there is no link between images possible in the same camera, whereas the novel approach does not rely on this. This can i.m.o. be stated more clearly.

**Author** The reviewer is totally right. The aim is not only to measure displacements when the object is subject to large rigid body motion. We considered this situation as "a very simple (but illustrative) example". The goal is actually to be able to perform displacement/strain fields measurement using DIC on mechanical experiments where classic DIC cannot be applied because the link between images is not possible in the same camera. The added value of this article concerns more the theoretical developments. The example of rigid rotation, at the end of the article, simply allowed us to make a first simple illustration of a kinematics which cannot be instrumented by classic DIC. But obviously, in future works, we will apply the approach to instrument a more representative mechanical test with strains. We thank the reviewer. This point was not sufficiently clear in the introduction in particular. We have improved the introduction and the conclusion in this sense.

**Reviewer** In the introduction, also the comment is made the FE-SDIC has the same kinematic basis as an FE analysis, making it straightforward to compare simulation and measurement. I do not agree with this comment since a converged FEA will generally involve elements which dimensions are too small to solve from a DIC perspective. Secondly, an FEA simulation involves calculations based on stress equilibrium whereas DIC solves based on grey level conservation. Notwithstanding this is not a key issue for the current paper, the authors should be careful in making these kind of hard statements.

**Author** First of all, we are afraid not to fully agree with the reviewer that an optimal (or converged) kinematic description for simulation is generally suboptimal for measurement. Indeed, if the mesh needs to be locally refined for the simulation, it is because it must represent a locally heterogeneous field, which (if the model is predictive) should also occur in the measured field. If this same mesh proves to be sub-optimal with respect to the definition of the images, it is because the instrumentation is not completely adapted to observe the localized phenomenon. In such a situation, we consider it would be necessary either to zoom in (to improve image resolution) or to rely on a multiscale instrumentation to adapt the images to the mesh and not the opposite (Passieux2015). One of the perspectives of the formalism presented in this article is precisely to offer a clean framework to assimilate measurements at different resolutions. In cases where it is not possible to adapt the resolution of the image to the mesh size, it is possible to rely
on weak regularization (Rethore2013; Pierre2017) or strong regularization (Chapelier2021).

**Reviewer** The envisaged use case is not fully clear to me. Can the authors give a couple of applications that involve these kind of large out-of-plane rotations that require a full-field approach? This will facilitate the applicability interpretation for the experimental mechanists, which is one of the objectives of this paper.

**Author** The authors thank the reviewer for pointing this out. The method we propose allows us to consider tests that conventional stereo methods can hardly or simply not instrument. We think for example of folded tape springs (kwok12), slender elastic rods (lazarus13; miller14), flapping wings (wu11), large torsional deformation of flexible parts (sicard14) or elastic ribbons (charrondiere20) to name a few. We could also be able to track large rigid or elastic body translation and rotations of projectiles (Passieux14). This finally would allow us to consider experiments in which the cameras can move, which could definitely offer new opportunities such as Stereo-DIC with camera mounted on drones (kalaitzakis21) or on robotic arms (khrenov18). These references were added in the perspective Section of the revised manuscript.

**Reviewer** It is not clear to me why a pinhole camera model is practically true for the current work. In particular when an object undergoes large rigid body motion lens distortions do become relevant.

**Author** The answer to this question is twofold.

* First, for the computation of the Jacobian $J_i^c$ we assumed a pinhole camera model without distortions in order to build a closed form approximation. Distorsions would add a lot of complexity and we guess it would certainly bring the same first order approximation.

* Second, in the example Section, for the camera model used for the calibration, more complex non-linear models could be used without theoretical difficulties. We have chosen to keep a simple pinhole model mainly for two reasons: (i) using the same model for the theoretical developments and numerical illustration was better to perserve the homogeneity/consistency of the article and (ii) a pinhole model was considered “practically true” since the reprojection errors estimated after calibration with and without non-linear distortions were approximately the same. A comment was added to the paper.

**Reviewer** The shape optimization relies in the example on a “free” object, i.e. all faces are visible and can be fully incorporated within the adjusting of the dimensioning of the CAD geometry. How robust will the approach be if one of the sides would be not visible or clamped. Is this still the case?

**Author** The limitation that the reviewer mentions is rather a limitation of classical stereo. It is partly due to the fact that in classical stereo, the cameras cannot move between the calibration and displacement measurement phases. One of the objective of the methodology that accompanies the new formalism is precisely to decouple the proposed generalisation of the calibration procedure (instrinsics, extrinsics, shape and albedo measurements, Section 3) from the mechanical experiment itself (displacement/strain measurement, Section 4): (a) the calibration where the object (or the camera) moves to acquire images from many points of view (regularisation), possibly outside the testing machine ; (b) the displacement measurement phase, with few well positioned cameras. In the formalism proposed here, it is no longer necessary to have the cameras fixed between the shape phase and displacement phase (or even between two displacement phases). The displacement measurement can thus be monitored from different viewpoints than those used in the calibration phase. In the calibration phase, it is clear that all faces must be visible by more than one image. We can imagine generalising the learning phase by positioning the object in different configurations so that all faces could be seen by at least two cameras each. We agree with the reviewer that there will always exist complex experimental configurations where it will hardly be possible to move the object or the camera around the object (heavy or very large or small scales). In this case, the measurement will only be possible on the sides on which the calibration could be done.

Once again, the major contribution of this paper is on the theoretical side. The application for the instrumentation of a real mechanical experiment is part of the perspectives and will be the subject of future works.

An effort has been made to clearly distinguish the calibration phase from the displacement
measurement phase throughout the article (especially in the introduction and conclusion).

**Reviewer** Error metric in camera calibration: What is reported here – reprojection error of input points for calibration or epipolar distance?

**Author** It is the target points reprojection error. It was mentioned in the title and caption of Figure 9.

**Reviewer** It is assumed the FEA model allows to consider small corrections. What is small? Suppose the object is a hanging beam subject to its eigen weight, does this fall under this category?

**Author** $\underline{D}$ is small provided that we may approximate $\underline{n}(\underline{X} + \underline{D}(\underline{X})) \sim \underline{n}(\underline{X})$ and $\det(\nabla\underline{\phi}_D) \sim 1$. That is $\underline{D}$ should be small compared to the specimen characteristic length which is $\overline{\text{close}}$ to the typical mesh size. Thank you for this excellent question that could be the subject of many other studies, such as the metrology of flexible parts (and where the effect of gravity can no longer be neglected: for example in the case of large structures such as wings or rotor blades), particularly thanks to the formalism proposed here. The interest of a numerical twin is that it can be enriched with a mechanical model, including elastic properties. One could then integrate in the same formalism the deformations due to the gravity and, thus, to reach shape errors unbiased by the specimen own weight.

## Review of version 2

Permalink: hal-03218388v2

**Author** Dear Editor, we are pleased to learn that our paper has been accepted for publication in the Journal of Theoretical, Computational and Applied Mechanics. Again, we would like to thank the reviewers for their help in improving the paper. You will find below our answer to their last comments.

### Reviewer 1 (Jean-François Witz)

**Reviewer** All the remarks were discussed, most of them were taken into account in a satisfactory way. I am not really satisfied by the evaluation of the bias which does not allow to make the partition between algorithm and error of the real part, I think that it would have been really important to work on a synthetic case.

**Author** We are sorry to leave some parts of the reviewer's comments unsatisfied. We still agree that this is an attractive perspective.

**Reviewer** In this regard, I have found: "Clearly, this is an interesting avenue, but as mentioned by the reviewer, it is far beyond the scope of this article. Blender is mainly about making the direct problem of the inverse problem we are trying to solve. We have to keep in mind that there are other sources of uncertainty that we will not be taking into account anyway. We should not risk falling into the inverse crime".

Blender would indeed be a way to create the direct model, with shots for all angles, without having all the problems related to cameras and defects of a real part, having a perfect knowledge of the albedo. As blender rendering engine for direct synthetic methods are not the one you use in your method, there wouldn't be any "inverse crime". I have no problem with the fact that you have neither the desire nor the time to do so, but the summoning of a "inverse crime" seems rather inappropriate here.

**Author** The reviewer is perfectly right, this last sentence should have been omitted but this should not be taken as a summoning. We should rather have put it this way: we made our algorithm operate on actual pictures, with all the complexity that it involves compared to synthetic ones that would be "perfect" in some sense. One may think of depth of field, lighting, BRDF for instance. We totally agree with the reviewer that generating synthetic images is the best way to assess the performance of the proposed methodology. However we would like to follow a different path compared to the one suggested by the reviewer, that is relying on Blender. We would like to be able to generate synthetic pictures on our own and be able to fully control the image formation process. We are fully aware that this will take some time and that is the main reason why we do
not wish to spend even more on taking up Blender.

**Reviewer** "First, it is important to understand that moving a node in the middle of a surface of a mesh along a tangent direction to this surface does not actually change the shape." is inappropriate, unless you think I am not aware of this fact?

**Author** This sentence is the prerequisite for introducing the solution non-uniqueness. We are sorry if it felt inappropriate.

**Reviewer** These points should not constitute a barrier for the publication of the article, and I am happy to give my consent for the publication of this article.

**Author** We would like to thank again the reviewer for their remarks and the possibility to publish.

**Reviewer 2 (Anonymous)**

**Reviewer** Dear all, I am happy with all the changes the authors have made in view of my remarks and comments.

**Author** Thank you for your help in correcting and refining our paper.

**Editor's assessment (Julien RÉTHORÉ)**

In their first report, both reviewers agreed on the high level of novelty of the approach described in the document. The reviewers suggested changes that would improve the clarity of the paper as well as its potential impact. The authors carefully considered the reviewers' requests and/or unambiguously justified the reasons for modifying/not modifying their paper in a detailed response from which a rich scientific discussion was conducted. For these reasons, the paper was accepted in its revised form.

**Open Access** This review is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the authors—the copyright holder. To view a copy of this license, visit creativecommons.org/licenses/by/4.0.

<figure><img src="image_2.png" /></figure>