# Motion-Induced Phase Error Compensation Using Three-Stream Neural Networks
## 2 Principle
### Three-Stream Neural Networks-Based Motion-Induced Error Compensation Method  
As discussed before, it is still problematic to accurately retrieve the phase from phase-shift fringe patterns of moving objects, especially for those with complex surfaces and different types of motion. To address this issue, deep learning technique is introduced to suppress the periodical phase error caused by motion. The diagram of the proposed temporal three-stream neural networks-based method is shown in Figure 1. Firstly, image sequences of three-step fringe patterns, which contain three different kinds of phase shift orders (order1 = (0, 2\(\pi\)/3, 4\(\pi\)/3), order2 = (2\(\pi\)/3, 4\(\pi\)/3, 0), and order3 = (4\(\pi\)/3, 0, 2\(\pi\)/3)) are cyclically projected onto the object over time. Secondly, every three adjacent images of the captured fringe patterns modulated by the object are passed into CNN 1-3 according to their orders. Thirdly, the numerator \(M(x\),\(y)\) and denominator \(D(x\),\(y)\) of the arctangent function in Equation (3) are predicted for accurate phase calculation and phase unwrapping. Lastly, the 3D shape is reconstructed using the traditional triangulation method.  
The design principles of the proposed framework are follows: during the projection of fringe patterns, the phase information and the position of the object change continuously from frame to frame, which make it difficult to directly learn the features from image sequence using a single neural network. Therefore, these fringe patterns are artificially divided into three groups according to their orders of phase shift values and passed to different subnets of three-stream neural networks. Since the inputs of each subnet have similar distribution in spatial dimension after grouping operation, it will be easier for networks to focus on the elimination of motion-induced error.  
All the CNN 1-3 adopt the architecture of U-Net [23], whose effectivity in phase prediction of non-sinusoidal fringe patterns has been verified [22]. The detail of the network is shown in bottom of Figure 1. The whole structure consists of an encoder, decoder, and skip connection. For the encoder, the input images are firstly processed by two convolutional blocks, which is implemented by a combination of a convolutional layer (Conv), batch normalization layer (BN) [26], and linear rectification function (ReLU) [27]. The max pooling followed by two convolution blocks are applied to down-sample the tensors by 1/2 in width and height, and the channel dimension is doubled. The same operations are used four times to increase the receptive field gradually. For the symmetrical  
Figure 1: Proposed temporal three-stream neural networks-based method for motion-induced error compensation (**top**) and the inner structure of CNN1–3 (**bottom**).  
decoder, the down-sampling operations are replaced by up-sampling. The multi-level features from skip connection are concatenated with tensors of doubled resolution to enhance the detail information. The output layer only has one single convolution operation without ReLU, since both positive and negative terms existed in \(M\)(\(x\),\(y\)) and \(D\)(\(x\),\(y\)). The kernel of all convolutional layer is 3 \(\times\) 3 with padding 1. The start features in the input layer is 64. Taking the fringe patterns modulated by moving object as input, the CNN 1-3 in three-stream neural networks will output the \(M\)(\(x\),\(y\)) and \(D\)(\(x\),\(y\)), whose motion-induced error has been suppressed.  
A simple L2-norm function is used to optimize the neural network and can be written as:  
\[Loss(\theta)=\big{[}M_{p}(\theta)-M_{g}\big{]}^{2}+\big{[}D_{p}(\theta)-D_{g} \big{]}^{2}. \tag{10}\]  
where \(\theta\) is the parameter of the network trained to minimize the loss function. The subscript \(p\) and \(g\) of \(M\) and \(D\) represent the prediction and ground truth, respectively.