# Stereo digital image correlation: formulations and perspectives

Raphaël Fouque, Robin Bouclier, Jean-Charles Passieux, Jean-Noël Périé

► To cite this version:

Raphaël Fouque, Robin Bouclier, Jean-Charles Passieux, Jean-Noël Périé. Stereo digital image correlation: formulations and perspectives. Comptes Rendus. Mécanique, 2021, 349 (3), pp.453-463. 10.5802/crmeca.93. hal-03260895v2

HAL Id: hal-03260895

https://hal.science/hal-03260895v2

Submitted on 10 Sep 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Comptes Rendus
## Mécanique

Raphaël Fouque, Robin Bouclier, Jean-Charles Passieux and Jean-Noël Périé

Stereo digital image correlation: formulations and perspectives

Volume 349, issue 3 (2021), p. 453-463

<https://doi.org/10.5802/crmeca.93>

© Académie des sciences, Paris and the authors, 2021.

Some rights reserved.

<figure><img src="image_2.png" /></figure>

This article is licensed under the

CREATIVE COMMONS ATTRIBUTION 4.0 INTERNATIONAL LICENSE.

http://creativecommons.org/licenses/by/4.0/
Short paper / *Note*

# Stereo digital image correlation: formulations and perspectives

Raphaël Fouque*<sup>a, b</sup>, Robin Bouclier<sup>© a, c</sup>, Jean-Charles Passieux<sup>© a</sup>
and Jean-Noël Périé<sup>© a</sup>

<sup>a</sup> Institut Clément Ader, CNRS UMR 5312, Université de Toulouse, INSA/ISAE/Mines Albi/UPS, 3 rue Caroline Aigle, 31400 Toulouse, France.

<sup>b</sup> DGA Aeronautical Systems, 47 rue Saint Jean - BP 93123-31131 Balma Cedex, France.

<sup>c</sup> Institut de Mathématiques de Toulouse, CNRS UMR 5219, Université de Toulouse, INSA/UT1/UT2/UPS, 135 avenue de Rangueil, 31077 Toulouse Cedex 04, France.

*E-mails*: fouque@insa-toulouse.fr (R. Fouque), robin.bouclier@math.univ-toulouse.fr (R. Bouclier), passieux@insa-toulouse.fr (J.-C. Passieux), jean-noel.perie@iut-tlse3.fr (J.-N. Périé)

**Abstract.** A literature review regarding functionals used in stereo digital image correlation (SDIC) is presented. A suitable functional for performing data assimilation, in the sense that all available information at all times is wisely taken into account, is also introduced. It is based on the comparison between a substitute image and actual ones, together with an associated weight. An interpretation of this weight is proposed. Eventually, a link between the functional and global SDIC ones is established. It clearly shows that the former encompasses the latter and provides the consistent weighting scheme to use in usual SDIC instead of ad hoc schemes.

**Keywords.** Data assimilation, Multi-view functional, Global stereo DIC, Photometric DIC, Large deformations.

**Note.** The first author is an employee of the DGA (Direction Générale de l'Armement - Ministry of Armed Forces), France.

*Manuscript received 9th June 2021, revised and accepted 26th July 2021.*

## 1. Introduction

Stereo digital image correlation (SDIC) is a full-field measurement technique that allows to retrieve a three-dimensional (3D) displacement field on a (possibly) non-planar surface, the region of interest (ROI) [1]. In a test-simulation dialogue perspective, the global methods for SDIC show the advantage of facilitating comparisons between measurements and simulations. As the same kinematic basis can be chosen, it is possible to define an error between, for instance,

<footnote>* Corresponding author.</footnote>
finite element (FE) DIC measurements and FE simulations simply by subtracting the associated displacement field degrees of freedom. For this reason, important studies have been carried out recently to extend the scope of existing methods and be able to perform measurements on increasingly complex geometries [2–5]. When following this path, the issue of visibility and surface curvature is raised. We wish to carefully address this topic here from a theoretical viewpoint and relate it to a close one: the weighting scheme in the SDIC functionals.

For these purposes, in Section 2, the usual SDIC frameworks are introduced together with the proposed data assimilation suitable functional. Then, we establish the link between this functional and the usual SDIC formulations. In Section 3, we consider shape measurements, while Section 4 tackles displacement measurements. First, when considering only two time steps, and then with an arbitrary number of time steps. This leads us to Sections 5 and 6, where discussions regarding the perspectives are proposed and concluding remarks are drawn.

## 2. State of the art

In this section, we introduce the usual global SDIC frameworks. That is, we begin this state of the art by focusing on shape measurements and then on displacement measurements. Eventually, the proposed functional is introduced.

To write the different functionals, we consider a set of $N_c$ cameras, each of which took a reference state image of the ROI $\Omega$. The associated set of reference state images is denoted by $(I_c^0)_c$. We also introduce the camera models $(\underline{p}_c)_c$. For all $c$, $\underline{p}_c$ maps a 3D point from the world reference frame to a two-dimensional (2D) point in the image reference frame associated with camera $c$. A set of camera parameters $\underline{p}_c$ is associated with this mapping. It encompasses extrinsic (relative position of the camera with respect to the specimen) and intrinsic parameters (focal/sampling parameters, camera centre coordinates in the pictures and possibly camera distortions).

### 2.1. Shape measurements

When considering planar (or near-planar) surfaces, it is possible to ensure that every single point of the ROI $\Omega$ remains visible by all cameras at all times. In this case, the functional associated with the shape measurement step reads [3, 6]:

$$F_1(\underline{U}_0, (\underline{p}_c)_c) = \sum_{c=1}^{N_c} \sum_{i=1}^{c-1} \int_{\Omega} \left( I_i^0 \circ \underline{P}_i \circ \phi_{\underline{U}_0}(\underline{X}) - I_c^0 \circ \underline{P}_c \circ \phi_{\underline{U}_0}(\underline{X}) \right)^2 d\underline{X}, \quad (1)$$

where $\underline{U}_0$ is the shape correction field and $\forall \underline{X} \in \Omega$, $\phi_{\underline{U}_0}(\underline{X}) = \underline{X} + \underline{U}_0(\underline{X})$. This functional ensures in the least-squares sense that the grey level corresponding to a physical point is the same for all cameras, and drives the shape correction field $\underline{U}_0$ accordingly. Note that the sum is written over all camera pairs. There are two main drawbacks to this formulation. First, computational costs associated with the problem scale as $N_c^2$ which is not ideal in a multi-camera setup. Second, the problem is extremely ill-posed [3, Figure 3].

**Remark 1.** Often raw data stemming from pictures are not compared directly as might seem suggested by (1). Instead, some corrections are made, for instance, by using a zero-mean normalised sum of squared differences (ZNSSD) correlation criterion [6].

In order to cope with the aforementioned problem ill-posedness but also to account for the surface sampling performed by each camera sensor, a shape measurement functional based on
a residual thought as the difference between a substitute image and actual images was built [7]. In this case, Equation (1) becomes

$$F'_{1}(\underline{U}_{0}, (\underline{p}_{c})_{c}, \hat{I}) = \sum_{c=1}^{N_{c}} \int_{\Omega} (I_{c}^{0} \circ \underline{P}_{c} \circ \underline{\phi}_{\underline{U}_{0}}(\underline{X}) - \hat{I}(\underline{X}))^{2} \underline{dX}. \quad (2)$$

It shows the benefit to tackle the two main issues identified previously. The problem scales as $N_c$ (like the displacement measurement one, see Section 2.2), and the optimisation procedure relies on an alternating optimisation (fixed-point algorithm) between the shape correction field, extrinsic, and the substitute image. It makes indeed the formulation much less ill-posed than in [3, 6] as it discards the functional kernel directions such as local and global slidings (illustrated in [3, Figure 3]).

Then, authors explicitly dealt with visibility issues by relying on a weighting term based on visibility [8]:

$$F'_{1}(\underline{U}_{0}, (\underline{p}_{c})_{c}) = \sum_{c=1}^{N_c} \sum_{i=1}^{c-1} \int_{\Omega} V_c(\underline{X}) V_i(\underline{X}) (I_i^0 \circ \underline{P}_i \circ \underline{\phi}_{\underline{U}_0}(\underline{X}) - I_c^0 \circ \underline{P}_c \circ \underline{\phi}_{\underline{U}_0}(\underline{X}))^2 \underline{dX}, \quad (3)$$

where $\forall c \in [[1, N_c]]$, $V_c$ is the visibility function associated with camera c such that

$$V_c: \begin{array}{l} \Omega \to \{0, 1\} \\ \underline{X} \to \begin{cases} 1 & \text{if } \underline{X} \text{ is visible by camera } c \\ 0 & \text{otherwise.} \end{cases} \end{array}$$

## 2.2. Displacement measurements

To engage in displacement measurements, we consider a deformed state of which the same $N_c$ cameras shoot the associated pictures $(I_c^1)_c$. The displacement measurement functional is then given by [3]:

$$F_2(\underline{U}_1) = \sum_{c=1}^{N_c} \int_{\Omega} (I_c^1 \circ \underline{P}_c \circ \underline{\phi}_{\underline{U}_1}(\underline{X}) - I_c^0 \circ \underline{P}_c \circ \underline{\phi}_{\underline{U}_0}(\underline{X}))^2 \underline{dX}. \quad (4)$$

**Remark 2.** With such notations, the displacement field associated with the deformation of the ROI $\Omega$ between the reference state and the deformed state is $\underline{U}_1 - \underline{U}_0$.

Visibility issues in the displacement measurement step were addressed by resorting on the assumption that a point visible by a camera in the reference state remains visible by this camera at all times, [4]:

$$F'_{2}(\underline{U}_{1}) = \sum_{c=1}^{N_{c}} \int_{\Omega} V_{c}(\underline{X}) (I_{c}^{1} \circ \underline{P}_{c} \circ \underline{\phi}_{\underline{U}_{1}}(\underline{X}) - I_{c}^{0} \circ \underline{P}_{c} \circ \underline{\phi}_{\underline{U}_{0}}(\underline{X}))^{2} \underline{dX}. \quad (5)$$

However, no clear justification for the weighting terms associated with the visibility is given, neither for (3) [8] nor for (5) [4]. Also, the general case, where the displacement field is such that a part of the structure may disappear from view, is not tackled.

## 2.3. Proposed functional

In this work, we wish to thoroughly establish weighting schemes for both shape and displacement measurements in global SDIC frameworks. To this end, we propose a unified formulation which, as will be shown in the remainder of this paper, encompasses all usual ones. The main idea is to define the weighting scheme in the pictures first as in [9]. Besides, relying on the framework described in [9] allows to overcome limitations associated with the assumption that a point visible by a camera in the reference state remains visible by the same camera at all times.
We consider $N_t$ time steps associated with $N_c$ cameras. The corresponding pictures are denoted by $(I_c^t)_{(c,t)}$. With these notations, the weighting scheme defined here as $\alpha_c^t(X)$ stands for the level of confidence associated with the corresponding observation $I_c^t \circ P_c \circ \phi_{\underline{U}_t}(X)$ and thus to the residual $(I_c^t \circ P_c \circ \phi_{\underline{U}_t}(X) - \hat{I}(X))$. As $\alpha_c^t$ is defined on the whole ROI $\Omega$, we expect it to meet some desirable properties. First, $\alpha_c^t$ should be equal to 0 in regions of $\Omega$ that are not seen in $I_c^t$. Second, it should somehow take into account the mapping performed by the optical system between the surface and the image. $\alpha_c^t(X)$ should, therefore, depend on the optical system characteristics and to the distance together with the local orientation of the surface with respect to the imager (surface foreshortening). Eventually, camera noise plays an important role in the level of confidence associated with an observation and should be accounted for in $\alpha_c^t$ (e.g., as defined by [4]). Let us assume that we can assign, at all times, the same level of confidence to two pixels coming from a same picture $c$ and, without loss of generality, to two pixels coming from different pictures. Hence, we may assign a unit weight in every image plane (if our last assumption is not fulfilled, we may assign the weight $1/\sigma_c^2$ to the image plane corresponding to camera $c$ by assuming a Gaussian white uniform noise of variance $\sigma_c$ in images shot by $c$ as in [4]). Then, by following the developments in [10] and [9], we build a functional, suitable for data assimilation, based on every piece of available information [11]:

$$F(\underline{U}_t)_t, (\underline{p}_c)_c, \hat{I} = \sum_{t=0}^{N_t-1} \sum_{c=1}^{N_c} \int_{\Omega} \alpha_c^t(X) (I_c^t \circ P_c \circ \phi_{\underline{U}_t}(X) - \hat{I}(X))^2 dX. \quad (6)$$

Here, data assimilation should be understood as a general method to take advantage of all available observations to evaluate quantities of interest. In our case (DIC), these quantities are typically displacements, camera parameters, or the substitute image, for instance. With previous assumptions, $\alpha_c^t$ is given by

$$\alpha_c^t(X) = \left| \det \left( \nabla \phi \right) \right| (\mathcal{J}_c V_c) \circ \phi_{\underline{U}_t}(X), \quad (7)$$

where $\mathcal{J}_c = \left| \det \nabla P_c \right|$. Note that $\alpha_c^t$ naturally meets the desirable properties listed above: the visibility function $\bar{V}_c$ naturally appears when integrating by substitution, and the surface sampling performed by the imager is taken into account, thanks to $\mathcal{J}_c$. On top of that, the contribution of the displacement field $\underline{U}_t$ is taken into account.

**Remark 3.** $\forall (\underline{X}, c, t), \alpha_c^t(\underline{X}) \ge 0$.

**Remark 4.** Let us stress once again that with such notations, the displacement field associated with time $t$ is $\underline{U}_t - \underline{U}_0$. Usually, after the shape measurement step, $\Omega$ is updated such that $\tilde{\Omega} = \phi_{\underline{U}_0}(\Omega)$. However, if we want to be able to efficiently perform a minimisation with respect to all arguments of $F$ (i.e., $(\underline{U}_t)_t, (\underline{p}_c)_c$ and $\hat{I}$), constantly updating the integration domain of all integrals may not be the most effective minimisation strategy. Keeping that in mind, defining the displacement on the nominal shape $\Omega$ as $\tilde{\underline{U}}_t = \underline{U}_t - \underline{U}_0$ is a small price to pay.

### 3. Shape measurements

In this section, we develop Functional $F$ from (6) so as to establish the link between this formulation and the usual SDIC shape measurement functionals (see (1) and (3)). For that, we consider $N_t = 1$, that is, only reference pictures $I_c^0$ are available. Note that with such considerations, Functional $F$ (6) is very close to $F'_1$ in (2):

$$F(\underline{U}_0, (\underline{p}_c)_c, \hat{I}) = \sum_{c=1}^{N_c} \int_{\Omega} \alpha_c^0(X) (I_c^0 \circ P_c \circ \phi_{\underline{U}_0}(X) - \hat{I}(X))^2 dX. \quad (8)$$
As in [7], $\hat{I}$ is obtained by minimising Functional F (8), that is directly (solution of a linear least-squares problem)

$$ \forall \underline{X} \in \Omega, \quad \sum_{c=1}^{N_c} \alpha_c^0(\underline{X}) \neq 0, \quad \hat{I}(\underline{X}) = \frac{\sum_{c=1}^{N_c} \alpha_c^0 I_c^0 \circ \underline{P}_c \circ \underline{\phi}_{\underline{U}_0}(\underline{X})}{\sum_{c'=1}^{N_c} \alpha_{c'}^0(\underline{X})} \quad (9) $$

**Remark 5.** If $\exists \underline{X} \in \Omega, \sum_{c=1}^{N_c} \alpha_c^0(\underline{X}) = 0$, then $\forall c \in [[1, N_c]], \alpha_c^0(\underline{X}) = 0$ as $\forall (\underline{X}, c), \alpha_c^0(\underline{X}) \ge 0$. Practically, it means that the point $\underline{X}$ cannot be seen by any camera. Hence $\hat{I}(\underline{X})$ can be set to any arbitrary real number without affecting the value of Functional F. Thus, in what follows, we do not consider this case any longer.

**Remark 6.** As there is no substitute image in usual SDIC frameworks, we will constantly rely on Functional F minimisation with respect to $\hat{I}$ to establish the link between Functional F and the functionals commonly used in SDIC.

To reduce the amount of notation, we denote $f_c = I_c^0 \circ \underline{P}_c \circ \underline{\phi}_{\underline{U}_0}$. And for any function h defined over $\Omega$, in the following, we will simply write $\int_{\Omega} h$ instead of $\int_{\Omega} h(\underline{X})d\underline{X}$. Now, let us develop (8) ((6) with $N_t = 1$):

$$ F = \int_{\Omega} \left( \sum_{c=1}^{N_c} \alpha_c^0 f_c^2 - 2 \hat{I} \sum_{c=1}^{N_c} \alpha_c^0 f_c + \hat{I}^2 \sum_{c=1}^{N_c} \alpha_c^0 \right), $$

using the identity (9), it follows

$$ F = \int_{\Omega} \left( \sum_{c=1}^{N_c} \alpha_c^0 f_c^2 - 2 \frac{\sum_{i=1}^{N_c} \alpha_i^0 f_i}{\sum_{k=1}^{N_c} \alpha_k^0} \sum_{c=1}^{N_c} \alpha_c^0 f_c + \frac{\sum_{i=1}^{N_c} \alpha_i^0 f_i \sum_{c=1}^{N_c} \alpha_c^0 f_c}{\sum_{k=1}^{N_c} \alpha_k^0} \right). $$

This previous expression can be simplified and factored as

$$ F = \int_{\Omega} \frac{1}{\sum_k \alpha_k^0} \left( \sum_i \sum_c \alpha_i^0 \alpha_c^0 f_c^2 - \sum_i \sum_c \alpha_i^0 \alpha_c^0 f_i f_c \right). $$

As $\sum_i \sum_c \alpha_i^0 \alpha_c^0 f_c^2 = \sum_i \sum_c \alpha_i^0 \alpha_c^0 f_i^2$:

$$ F = \frac{1}{2} \int_{\Omega} \frac{1}{\sum_k \alpha_k^0} \left( \sum_i \sum_c \alpha_i^0 \alpha_c^0 f_c^2 - 2 \sum_i \sum_c \alpha_i^0 \alpha_c^0 f_i f_c + \sum_i \sum_c \alpha_i^0 \alpha_c^0 f_i^2 \right). $$

We can factor this expression as

$$ F = \frac{1}{2} \sum_i \sum_c \int_{\Omega} \frac{\alpha_i^0 \alpha_c^0}{\sum_k \alpha_k^0} (f_c^2 - 2f_i f_c + f_i^2), $$

which can finally be rewritten as

$$ F = \sum_c \sum_{i<c} \int_{\Omega} \frac{\alpha_c^0 \alpha_i^0}{\sum_k \alpha_k^0} (f_c - f_i)^2. $$

This last equation is very close to Functionals $F_1$ (1) and $F_1''$ (3) used in a standard global SDIC framework for the shape measurement step (see Section 2.1). These developments allow to establish a link between a weight assigned to each observation $f_c$, namely $\alpha_c^0$, and the associated weight in the usual framework, which should be $\alpha_c^0 \alpha_i^0 / \sum_k \alpha_k^0$ when comparing $f_c$ to $f_i$.

In $F_1$ (1) [3], it is (implicitly) assumed that

$$ \alpha_c^0 = \left| \det \left( \nabla \underline{\phi} \right) \right| (\mathcal{J}_c V_c) \circ \underline{\phi}_{\underline{U}_0} \sim 1, $$
because (near) planar surfaces are considered. In this case, considering the correct weighting scheme does not change much the functional expression as $\alpha_c^0 \alpha_i^0 / \sum_k \alpha_k^0 \sim 1/N_c$. Note that we just showed that according to previous assumptions

$$F_1/N_c = F'_1.$$

However, in $F_1''$ (3) [8], $\alpha_c^0 \sim V_c$ is assumed. Hence, when considering more complex geometries, the correct weight when comparing $f_c$ to $f_i$ should be $V_c V_i / \sum_k V_k$, instead of $V_c V_i$.

# 4. Displacement measurements

Let us now consider the displacement measurement step. Before getting into the general case $N_t > 2$, we establish the link between Functional F (6) and usual frameworks considering only two time steps ($N_t = 2$). It is done first by relying on a substitute image $\hat{I}$ given by (9) (only reference state images are used to build $\hat{I}$), and then by updating $\hat{I}$, thanks to data provided by deformed state images.

## 4.1. Incremental displacement measurements

Since $N_t = 2$, Functional F (6) writes as follows:

$$ 
\begin{aligned}
F(\underline{U}_0, \underline{U}_1, (\underline{p}_c)_c, \hat{I}) &= \sum_{c=1}^{N_c} \int_{\Omega} \alpha_c^0(\underline{X}) (I_c^0 \circ \underline{P}_c \circ \underline{\phi}_{\underline{U}_0}(\underline{X}) - \hat{I}(\underline{X}))^2 + \alpha_c^1(\underline{X}) (I_c^1 \circ \underline{P}_c \circ \underline{\phi}_{\underline{U}_1}(\underline{X}) - \hat{I}(\underline{X}))^2 d\underline{X} \\
&= \sum_{c=1}^{N_c} \int_{\Omega} \alpha_c^0(f_c - \hat{I})^2 + \alpha_c^1(g_c - \hat{I})^2,
\end{aligned} 
\quad (10) $$

where $g_c = I_c^1 \circ \underline{P}_c \circ \underline{\phi}_{\underline{U}_1}$.

### 4.1.1. Substitute image based on reference state images only

Here, as in [7] we keep on using the same substitute image based on (9). In this case, Functional F (10) is minimised with respect to $\underline{U}_1$ only and reads:

$$ F = \underbrace{\sum_{c=1}^{N_c} \int_{\Omega} \alpha_c^0 (f_c - \hat{I})^2}_{\text{Constant} = F_0} + \sum_{c=1}^{N_c} \int_{\Omega} \alpha_c^1 (g_c - \hat{I})^2 $$

$$ F = F_0 + \sum_{c=1}^{N_c} \int_{\Omega} \alpha_c^1 (g_c^2 - 2\hat{I}g_c + \hat{I}^2) $$

We can then make use of (9):

$$ 
\begin{aligned}
F &= F_0 + \int_{\Omega} \frac{1}{\sum_{k=1}^{N_c} \alpha_k^0} \left( \sum_{c=1}^{N_c} \sum_{i=1}^{N_c} \alpha_i^0 \alpha_c^1 g_c^2 - 2 \sum_{c=1}^{N_c} \sum_{i=1}^{N_c} \alpha_i^0 \alpha_c^1 g_c f_i + \frac{\sum_{c=1}^{N_c} \alpha_c^1}{\sum_{k=1}^{N_c} \alpha_k^0} \sum_{i=1}^{N_c} \sum_{j=1}^{N_c} \alpha_j^0 \alpha_i^0 f_i f_j \right) \\
&= F_0 + \sum_i \sum_c \int_{\Omega} \frac{\alpha_i^0 \alpha_c^1}{\sum_k \alpha_k^0} \left( g_c^2 - 2g_c f_i + \frac{\sum_j \alpha_j^0 f_i f_j}{\sum_k \alpha_k^0} \right) \\
&= F_0 + \sum_i \sum_c \int_{\Omega} \frac{\alpha_i^0 \alpha_c^1}{\sum_k \alpha_k^0} \left( (g_c - f_i)^2 - f_i^2 + \frac{\sum_j \alpha_j^0 f_i f_j}{\sum_k \alpha_k^0} \right) \\
&= F_0 + \sum_i \sum_c \int_{\Omega} \frac{\alpha_i^0 \alpha_c^1}{\sum_k \alpha_k^0} \left( (g_c - f_i)^2 - f_i^2 + f_i \hat{I} \right)
\end{aligned}
$$
$$
\begin{align*}
&= F_0 + \sum_i \sum_c \int_{\Omega} \frac{\alpha_i^0 \alpha_c^1}{\sum_k \alpha_k^0} \left( (g_c - f_i)^2 - (f_i - \hat{I})^2 + \hat{I}(\hat{I} - f_i) \right) \\
&= F_0 + \sum_i \sum_c \int_{\Omega} \frac{\alpha_i^0 \alpha_c^1}{\sum_k \alpha_k^0} \left( (g_c - f_i)^2 - (f_i - \hat{I})^2 \right) + \sum_c \int_{\Omega} \alpha_c^1 \hat{I} \left( \hat{I} - \frac{\sum_i \alpha_i^0 f_i}{\sum_k \alpha_k^0} \right).
\end{align*}
$$

The last sum equals 0 by definition of $\hat{I}$ (9):

$$
\begin{align}
F &= F_0 + \sum_i \sum_c \int_{\Omega} \frac{\alpha_i^0 \alpha_c^1}{\sum_k \alpha_k^0} \left( (g_c - f_i)^2 - (f_i - \hat{I})^2 \right) \nonumber \\
&= \sum_c \int_{\Omega} \alpha_c^0 (f_c - \hat{I})^2 + \sum_i \sum_c \int_{\Omega} \frac{\alpha_i^0 \alpha_c^1}{\sum_k \alpha_k^0} (g_c - f_i)^2 - \sum_c \sum_j \int_{\Omega} \frac{\alpha_c^0 \alpha_j^1}{\sum_k \alpha_k^0} (f_c - \hat{I})^2 \nonumber \\
&= \sum_i \sum_c \int_{\Omega} \frac{\alpha_i^0 \alpha_c^1}{\sum_k \alpha_k^0} (g_c - f_i)^2 + \sum_c \int_{\Omega} \alpha_c^0 (f_c - \hat{I})^2 \left( 1 - \frac{\sum_j \alpha_j^1}{\sum_k \alpha_k^0} \right) \nonumber \\
&= \sum_c \int_{\Omega} \frac{\alpha_c^0 \alpha_c^1}{\sum_k \alpha_k^0} (g_c - f_c)^2 + \sum_c \sum_{i \neq c} \int_{\Omega} \frac{\alpha_i^0 \alpha_c^1}{\sum_k \alpha_k^0} (g_c - f_i)^2 + \sum_c \int_{\Omega} \alpha_c^0 (f_c - \hat{I})^2 \left( 1 - \frac{\sum_j \alpha_j^1}{\sum_k \alpha_k^0} \right). \tag{11}
\end{align}
$$

At this point, let us point out that relying on a substitute image for the displacement mea-
surement step exhibits some interesting properties. First, it encompasses usual formulations of
the shape measurement step (Equations (4) or (5)), thanks to the first sum in (11), but also every
spatio-temporal cross-correlations $g_c - f_i$, with cameras $i \neq c$, which are usually not included.
Also, as (2) compared to (1), (6) scales linearly with the number of cameras, unlike (11) which
scales quadratically. Finally, note that the last term in (11) may be neglected if for every point $\underline{X}$
of $\Omega$, $\sum_j \alpha_j^1(\underline{X}) \approx \sum_k \alpha_k^0(\underline{X})$, that is, if every point $\underline{X}$ is equally well observed in the pictures ($I_c^0$)$_c$
and in the pictures ($I_c^1$)$_c$.

Again, these developments establish a link between a weight associated to a given observation and the consistent weighting scheme that should be adopted in the usual framework. In $F_2$ (4) [3], it is assumed that $\alpha_c^0 \sim \alpha_c^1 \sim 1$ and again, considering a consistent weighting scheme $\alpha_c^0 \alpha_c^1 / \sum_k \alpha_k^0$ when comparing $f_c$ to $g_c$ only scales $F_2$ by a constant factor $1/N_c$. However, when introducing a visibility function such that $\alpha_c^0 \sim \alpha_c^1 \sim V_c$ as in $F_2'$ (5) [4], the consistent weight when comparing $f_c$ to $g_c$ should be $\alpha_c^0 \alpha_c^1 / \sum_k \alpha_k^0 \sim V_c / \sum_k V_k$ instead of $V_c$.

4.1.2. *Substitute image updating*

Here, we perform data assimilation in the sense that Functional F from (10) (Equation (6) with $N_t = 2$) is minimised with respect to all arguments (i.e., $\underline{U}_0, \underline{U}_1, (\underline{p}_c)_c$ and $\hat{I}$), unlike the previous section. For this reason, $\hat{I}$ is updated by minimising Functional F (10):

$$
\hat{I} = \frac{\sum_{c=1}^{N_c} \alpha_c^0 f_c + \alpha_c^1 g_c}{\sum_{k=1}^{N_c} \alpha_k^0 + \alpha_k^1}. \qquad (12)
$$

Hence, we can develop

$$
\begin{align*}
F &= \int_{\Omega} \left( \sum_{c=1}^{N_c} \alpha_c^0 f_c^2 - 2\hat{I} \sum_{c=1}^{N_c} \alpha_c^0 f_c + \hat{I}^2 \sum_{c=1}^{N_c} \alpha_c^0 \right) + \left( \sum_{c=1}^{N_c} \alpha_c^1 g_c^2 - 2\hat{I} \sum_{c=1}^{N_c} \alpha_c^1 g_c + \hat{I}^2 \sum_{c=1}^{N_c} \alpha_c^1 \right) \\
&= \int_{\Omega} \left( \sum_c (\alpha_c^0 f_c^2 + \alpha_c^1 g_c^2) - 2\hat{I} \sum_c (\alpha_c^0 f_c + \alpha_c^1 g_c) + \hat{I}^2 \sum_c (\alpha_c^0 + \alpha_c^1) \right).
\end{align*}
$$
Making use of the expression of $\hat{I}$ (12) a first time

$$F = \int_{\Omega} \left( \sum_{c} (\alpha_{c}^{0} f_{c}^{2} + \alpha_{c}^{1} g_{c}^{2}) - \hat{I}^{2} \sum_{c} (\alpha_{c}^{0} + \alpha_{c}^{1}) \right),$$

and a second time after factoring by $1/\sum_k (\alpha_k^0 + \alpha_k^1)$ in the integral

$$F = \int_{\Omega} \frac{1}{\sum_{k} (\alpha_{k}^{0} + \alpha_{k}^{1})} \left[ \sum_{i} \sum_{c} (\alpha_{i}^{0} + \alpha_{i}^{1}) (\alpha_{c}^{0} f_{c}^{2} + \alpha_{c}^{1} g_{c}^{2}) - \sum_{i} \sum_{c} (\alpha_{c}^{0} f_{c} + \alpha_{c}^{1} g_{c}) (\alpha_{i}^{0} f_{i} + \alpha_{i}^{1} g_{i}) \right] \\ = \sum_{i} \sum_{c} \int_{\Omega} \frac{\alpha_{i}^{0} \alpha_{c}^{0} (f_{c}^{2} - f_{i} f_{c}) + \alpha_{i}^{0} \alpha_{c}^{1} g_{c}^{2} + \alpha_{c}^{0} \alpha_{i}^{1} f_{c}^{2} + \alpha_{i}^{1} \alpha_{c}^{1} (g_{c}^{2} - g_{i} g_{c}) - 2 \alpha_{i}^{0} \alpha_{c}^{1} f_{i} g_{c}}{\sum_{k} (\alpha_{k}^{0} + \alpha_{k}^{1})}.$$

Again as, for instance, $\sum_i \sum_c \alpha_c^0 \alpha_i^1 f_c^2 = \sum_i \sum_c \alpha_i^0 \alpha_c^1 f_i^2$:

$$F = \sum_{i} \sum_{c} \int_{\Omega} \frac{1}{\sum_{k} (\alpha_{k}^{0} + \alpha_{k}^{1})} \left[ \alpha_{i}^{0} \alpha_{c}^{1} (g_{c} - f_{i})^{2} + \frac{1}{2} \alpha_{i}^{0} \alpha_{c}^{0} (f_{c} - f_{i})^{2} + \frac{1}{2} \alpha_{i}^{1} \alpha_{c}^{1} (g_{c} - g_{i})^{2} \right] \\ = \sum_{c} \int_{\Omega} \frac{\alpha_{c}^{0} \alpha_{c}^{1}}{\sum_{k} (\alpha_{k}^{0} + \alpha_{k}^{1})} (g_{c} - f_{c})^{2} + \sum_{c} \sum_{i \neq c} \int_{\Omega} \frac{\alpha_{c}^{1} \alpha_{i}^{0}}{\sum_{k} (\alpha_{k}^{0} + \alpha_{k}^{1})} (g_{c} - f_{i})^{2} \\ + \sum_{c} \sum_{i < c} \int_{\Omega} \frac{\alpha_{i}^{0} \alpha_{c}^{0}}{\sum_{k} (\alpha_{k}^{0} + \alpha_{k}^{1})} (f_{c} - f_{i})^{2} + \sum_{c} \sum_{i < c} \int_{\Omega} \frac{\alpha_{i}^{1} \alpha_{c}^{1}}{\sum_{k} (\alpha_{k}^{0} + \alpha_{k}^{1})} (g_{c} - g_{i})^{2}. \quad (13)$$

Considering only the terms such that $i = c$ in the previous expression of Functional F (first term) allows to retrieve a functional similar to the one used in the usual frameworks for the displacement measurement step (see (4) and (5)).

Let us stress again that relying on a substitute image $\hat{I}$ in a displacement measurement perspective shows the benefit to have a much richer functional than the usual ones. In (13), there are indeed terms proportional to $(f_c - f_i)^2$ and $(g_c - g_i)^2$ which are similar to a shape measurement (see Section 3). The stereo correspondence is thus preserved. There are also terms proportional to $(g_c - f_i)^2$, $i \neq c$ (spatio-temporal cross-correlations) which have no counterparts in the usual frameworks.

## 4.2. Data assimilation displacement measurements

In this section, we investigate the possibility to minimise Functional F defined in (6) with respect to every argument (i.e., $(\underline{U}_t)_{0 \le t \le N_t-1}$, $(\underline{P}_c)_{1 \le c \le N_c}$, $\hat{I}$) and show to which extent this functional is suitable for performing data assimilation by, once again, establishing the link with usual frameworks. Here, by data assimilation, we mean benefitting from all available observations to evaluate quantities of interest (i.e., for instance in DIC, displacements, camera parameters, substitute image). A key element in such an approach is the level of confidence associated with observations that we have already discussed.

In what follows, to reduce the amount of notation, we will simply write $I_c^t$ instead of $I_c^t \circ \underline{P}_c \circ \underline{\phi}_{\underline{U}_t}$. With such notations, the expression of $\hat{I}$ is simply

$$\hat{I} = \frac{\sum_{t=0}^{N_t-1} \sum_{c=1}^{N_c} \alpha_c^t I_c^t}{\sum_{s=0}^{N_t-1} \sum_{i=1}^{N_c} \alpha_i^s}.$$

Note that this expression for $\hat{I}$, stemming from the minimisation of Functional F (6), is very close to the heuristic approach used in [12]. In the context of heat haze effects, relying on a substitute
image based on all available pictures is essential as the confidence associated with the reference picture is low. Then we can develop Functional F from (6) (similar treatment as in Section 4.1.2):

$$
\begin{align*}
F &= \sum_{t=0}^{N_t-1} \sum_{c=1}^{N_c} \int_{\Omega} \alpha_c^t \left( (I_c^t)^2 - 2I_c^t \hat{I} + \hat{I}^2 \right) \\
&= \int_{\Omega} \sum_t \sum_c \alpha_c^t (I_c^t)^2 - \hat{I}^2 \sum_t \sum_c \alpha_c^t \\
&= \int_{\Omega} \frac{1}{\sum_r \sum_j \alpha_j^r} \left( \sum_t \sum_c \sum_s \alpha_i^s \alpha_c^t (I_c^t)^2 - \sum_t \sum_c \sum_s \alpha_i^s \alpha_c^t I_c^t I_i^s \right) \\
&= \frac{1}{2} \sum_t \sum_c \sum_s \sum_i \int_{\Omega} \frac{\alpha_i^s \alpha_c^t}{\sum_r \sum_j \alpha_j^r} (I_c^t - I_i^s)^2.
\end{align*}
$$

Finally, Functional F can be split in different parts ($s \neq t$ and $s = t$):

$$
F = \sum_t \left[ \sum_{s<t} \underbrace{\left( \sum_c \int_{\Omega} \frac{\alpha_c^s \alpha_c^t}{\sum_r \sum_j \alpha_j^r} (I_c^t - I_i^s)^2 + \sum_{c \neq c} \int_{\Omega} \frac{\alpha_i^s \alpha_c^t}{\sum_r \sum_j \alpha_j^r} (I_c^t - I_i^s)^2 \right)}_{\text{Similar to a displacement measurement functional}} + \underbrace{\left( \sum_c \int_{\Omega} \frac{\alpha_i^t \alpha_c^t}{\sum_r \sum_j \alpha_j^r} (I_c^t - I_i^t)^2 + \sum_{c \neq c} \int_{\Omega} \frac{\alpha_i^t \alpha_c^t}{\sum_r \sum_j \alpha_j^r} (I_c^t - I_i^t)^2 \right)}_{\text{Spatio-temporal cross-correlations}} \right]. \quad (14)
$$

This final expression for Functional F clearly establishes the link with usual frameworks. We can
see that it includes usual shape measurements (see (1)) at all times, together with terms similar
to displacement measurements (see (4) or (5)) for all pairs of times, as well as spatio-temporal
cross-correlations (comparing $I_c^t$ to $I_i^s$, with cameras $i \neq c$, (spatial) and times $s \neq t$ (temporal)).
Once again, it is much richer than the usual functionals.

5. Discussions

In order to establish the links above between Functional F (6) and usual frameworks, we had to adopt the same experimental setups. That is, at all times, the number of cameras N_c is the same and the cameras are assumed to remain in a fixed position all along the experiment. Let us stress that it does not have to be the case, and that the formulation proposed (6) is easily extended to arbitrary number of pictures at each time (N_c^t), with moving cameras (p_c^t). This would allow to consider experimental setups with cameras supported by robotic arms or even drones, for instance. For these reasons and others that we wish to illustrate in what follows, the functional proposed (6) opens up new perspectives in terms of experimental setups. It offers much more flexibility to the experimenter, while providing a much greater robustness, as it increases the amount of data for each problem (camera calibration, shape, displacement).

First, as already evoked, when considering reference state images $f_c$ and deformed ones $g_c$ that see totally disjoint regions of the ROI, the first term in (11), equivalent to usual SDIC frameworks, becomes zero, as the product $\alpha_c^0 \alpha_c^1$ equals zero. This kind of situation totally incapacitates all DIC software (including SDIC and 2D-DIC). This may arise in the case of large rotations as described in [9]. Yet, Functional F (6) allows to naturally address this issue, thanks to the cross-correlation terms.

Also, when considering large strains, relying on all available pictures with a weight $\alpha_c^t$ depending on the displacement field $U_t$, as the one that naturally arises in [9] and given here in (7), would be particularly helpful to perform a finer sampling of the substitute image [10, 11]. For large positive strains, $|\det \nabla\phi_{U_t}| = |\det(I + \nabla U_t)| > 1$. This assigns a greater level of confidence to the image
$I_c^t$, which is consistent with the better sampling achieved by the pixels in $I_c^f$ of the ROI. In other words, in the case of large (positive) strains, it is unfortunate, in the current frameworks, to identify the substitute image in the reference state images only, as the information in deformed state ones is much more reliable.

On top of that, note that the weighting scheme together with the construction of Functional $F$ (6) naturally provides a way to merge results from different times and different viewpoints in order to perform multiscale substitute image identification and, most importantly, multiscale displacement measurements. That is, cameras with different resolutions imaging the ROI [13]. Currently, the dialogue between measurements performed at two different resolutions is still an open problem.

Then, regarding camera calibration, some research works identify projection parameters on the sole basis of reference state images [7]. This camera calibration process, while convenient from an experimenter perspective, has the major drawback not to calibrate the whole volume spanned by the object, which can result in a stereo correspondence loss. Identifying camera parameters based on the minimisation of (6) would allow to calibrate the whole volume spanned by the ROI, precisely because the minimisation would be performed on all positions occupied by the object. Also, this would allow to avoid calibrating the stereo rig at different times (based on targets), as done to prevent temporal drift during long experiments. Regarding this matter, the last terms in (14), similar to shape measurement functionals, turns out to be useful.

Finally, this formulation is particularly suitable for spatio-temporal regularisation and one could imagine making use of it to perform SDIC measurements during tests with a single moving camera, or a rotating object (in a tomograph, for instance) in front of the fixed camera, relying on similar techniques as in [14].

## 6. Conclusion

There are two main results associated with the developments presented herein. First, we established a link between functionals on which global SDIC usually relies and a functional based on the sum of errors between a substitute image and observations from all cameras, at all times: Functional $F$ (6). We showed that the latter is much richer than the formers in a displacement measurement context. Based on the consideration of large displacements, camera calibration, and stereo correspondence issues, we illustrated that all the terms usually discarded in SDIC frameworks are actually extremely useful. For this reason, Functional $F$ (6) appears to (a) be well-suited to perform data assimilation in SDIC, as expected from the construction of this functional based on all available data, (b) stand for an interesting perspective in the formulation of the SDIC problem, as it can be seen as a *dense* counterpart of bundle adjustment methods [1, 10].

Second, we proposed an interpretation of the weighting scheme used in Functional $F$ (6). This weight can be thought of as the level of confidence associated with each pixel. From this interpretation, we gave properties that this weight should satisfy regarding visibility, camera noise, and surface curvature. Thanks to the link established between this functional and usual frameworks, we provided a weighting scheme for usual SDIC frameworks consistent with our previous interpretation.

We did not provide a framework allowing to perform global SDIC based on Functional $F$ (6). A natural outlook of these theoretical developments is thus to propose a numerical implementation allowing to minimise efficiently this functional with actual data.

## Conflicts of interest

Raphaël Fouque is an employee of the DGA (Direction Générale de l'Armement—Ministry of Armed Forces), France. The submission of this article was subjected to prior approval by the DGA.
## Dedication

The manuscript was written through contributions of all authors. All authors have given approval to the final version of the manuscript.

## Acknowledgments

The authors of this paper gratefully acknowledge the financial support of the Direction Générale de l'Armement (DGA)—Ministry of Armed Forces, France. The first author is an employee of the DGA (Direction Générale de l'Armement—Ministry of Armed Forces), France.

## References

[1] M. A. Sutton, J.-J. Orteu, H. W. Schreier, Image Correlation for Shape, Motion and Deformation Measurements: Basic Concepts, Theory and Applications, Springer, Boston, MA, 2009, 321 pages.

[2] J.-E. Dufour, S. Leclercq, J. Schneider, S. Roux, F. Hild, “3D surface measurements with isogeometric stereocorrelation—Application to complex shapes”, Opt. Lasers Eng. 87 (2016), p. 146-155.

[3] J.-E. Pierré, J.-C. Passieux, J.-N. Périé, “Finite element stereo digital image correlation: framework and mechanical regularization”, Exp. Mech. 57 (2017), no. 3, p. 443-456.

[4] F. Hild, S. Roux, “Displacement uncertainties with multiview correlation schemes”, J. Strain Anal. Eng. Des. 55 (2020), no. 7-8, p. 199-211.

[5] M. Vitse, M. Poncelet, A. E. Iskef, J.-E. Dufour, R. Gras, A. Bouterf, B. Raka, C. Giry, F. Gatuingt, F. Hild, F. Ragueneau, S. Roux, “Toward virtual design and optimization of a structural test monitored by a multi-view system”, J. Strain Anal. Eng. Des. 56 (2021), no. 2, p. 112-128.

[6] G. Colantonio, M. Chapelier, R. Bouclier, J.-C. Passieux, E. Marenić, “Noninvasive multilevel geometric regularization of mesh-based three-dimensional shape measurement”, Int. J. Numer. Methods Eng. 121 (2020), no. 9, p. 1877-1897.

[7] J.-E. Dufour, F. Hild, S. Roux, “Shape, displacement and mechanical properties from isogeometric multiview stereo-correlation”, J. Strain Anal. Eng. Des. 50 (2015), no. 7, p. 470-487.

[8] M. Chapelier, R. Bouclier, J.-C. Passieux, “Free-form deformation digital image correlation (FFD-DIC): a non-invasive spline regularisation for arbitrary finite element measurements”, Comput. Methods Appl. Mech. Eng. 384 (2021), article no. 113992.

[9] R. Fouque, R. Bouclier, J.-C. Passieux, J.-N. Périé, “Photometric DIC: a unified framework for global stereo digital image correlation based on the construction of textured digital twins”, preprint, 2021, https://hal.archives-ouvertes.fr/hal-03218388.

[10] B. Goldlücke, M. Aubry, K. Kolev, D. Cremers, “A super-resolution framework for high-accuracy multiview reconstruction”, Int. J. Comput. Vis. 106 (2014), no. 2, p. 172-191.

[11] V. Tsiminaki, W. Dong, M. R. Oswald, M. Pollefeys, “Joint multi-view texture super-resolution and intrinsic decomposition”, in British Machine Vision Conference, BMVC, 2019, p. 1-14.

[12] M. Berny, T. Archer, A. Mavel, P. Beauchêne, S. Roux, F. Hild, “On the analysis of heat haze effects with spacetime DIC”, Opt. Lasers Eng. 111 (2018), p. 135-153.

[13] J.-C. Passieux, F. Bugarin, C. David, J.-N. Périé, L. Robert, “Multiscale displacement field measurement using digital image correlation: application to the identification of elastic properties”, Exp. Mech. 55 (2015), no. 1, p. 121-137.

[14] C. Jailin, A. Bouterf, M. Poncelet, S. Roux, “In situ µ CT-scan mechanical tests: fast 4D mechanical identification”, Exp. Mech. 57 (2017), no. 8, p. 1327-1340.