## 2 The Algorithm
### Expansion
#### 2.2.2 Optimization
The relation between pixel coordinates (\(s+i\), \(t+j\)) and (\(s+i\), \(t+j\))\({}^{K}\) can be found based on the fact that they correspond to the same 3D point \(X(s+i\), \(t+j)\). Along the line from the optical center \(O_{R}\) of the reference image through the (\(s+i\), \(t+j\)) pixel in the reference image, one can find the world coordinate of a 3D point \(X(s+i\), \(t+j)\) using Equation (7)  
\[X(s+i,t+j)=(O_{R}+\mathbf{r}_{R}(s+i,t+j)\times(h+ih_{s}+jh_{t})) \tag{7}\]  
where \(\mathbf{r}_{R}(s+i\), \(t+j)\) is the normalized vector from the optical center \(O_{R}\) of reference image to (\(s+i\), \(t+j\)) pixel in the reference image. The term (\(h+ih_{s}+jh_{t}\)) represents the depth at the (\(s+i\), \(t+j\)) pixel in the reference image, where \(h\) is the depth of the (\(s\), \(t\)) pixel in the reference image, \(h_{s}\) and \(h_{t}\) are the rates of depth-change per pixel along row and column directions in the reference image. Projecting 3D point \(X(s+i\), \(t+j)\) back on an auxiliary image \(K\), we get  
\[(s+i,t+j)^{K}=\mathbf{P}_{K}\times X(s+i,t+j) \tag{8}\]  
where \(\mathbf{P}_{K}\) is the camera matrix of image \(K\). Substituting Equations (7) and (8) into Equation (6) and expand it in [PERSON]'s series with only linear terms, we get  
\[ \begin{array}{ll}I_{R}(s+i,t+j)&=c_{K}\times I_{K}(\mathbf{P}_{K}\times(O_{R}+ \mathbf{r}_{R}(s+i,t+j)\times(h+ih_{s}+jh_{t})))\\ &+c_{k}\times(dh+idh_{s}+jdh_{t})\times\partial I_{K}(s+i,t+j)/\partial h(s+ i,t+j)\end{array}  \tag{9}\]  
where \(dh\), \(dh_{s}\) and \(dh_{t}\) are increments of \(h\), \(h_{s}\) and \(h_{t}\). As the initial values of \(h\), \(h_{s}\) and \(h_{t}\) are not accurate, the values of \(h\), \(h_{s}\) and \(h_{t}\) are updated with \(dh\), \(dh_{s}\) and \(dh_{t}\) and solved iteratively. When \(dh\), \(dh_{s}\) and \(dh_{t}\) approach zero, Equation (9) approximates Equation (6) well and \(h\), \(h_{s}\) and \(h_{t}\) attain their final accurate values. Substituting the final \(h\), \(h_{s}\) and \(h_{t}\) in Equation (7), the final 3D position and normal (encoded by \(h_{s}\) and \(h_{t}\)) can be determined. In the above optimization, some parameters are adjusted automatically to cope with various situations. For example, in the case of texture weak areas, the window size should be increased following expression (10) to include enough textures to improve accuracy. \[{\sigma_{2}}^{2}\geq th_{r2} \tag{10}\]In Expression (10), \({\sigma_{2}}^{2}\) is the intensity variance of the \(m\times m\) window in the reference image (\(m\) should not be large than 21) and \(th_{\sigma 2}\) is a threshold value. In case of a smaller number of images and/or big view angles, the required number of auxiliary images and the similarity for convergence are decreased to make the reconstruction as successful as possible. In any case the view angle of the reference image should be within 80 degrees after reconstruction, or else the reconstructed point might be inaccurate and should be deleted. It was discovered in experiments that some points, which had failed to optimize based on the original high-resolution image, might succeed based on the same image at a reduced resolution. So, a multi-level image pyramid is established, and the optimization of a point commences at a lower-level image, until it ends successfully or images of all levels have been tested.