## 3 Method
### Architecture Overview  
The VUDNet architecture integrates a Variational Auto Encoder (VAE) and a modified U-Net, both of which are provided with an input 2D image \(I\), the reflection of pattern \(\mathcal{P}\) on some unknown surface \(\mathcal{S}\). The VAE is a generative model that aims to learn the underlying distribution of the data, providing a coarse estimation of the depth map. In contrast, the U-Net is a discriminative model designed to refine this estimation, yielding a finer and more precise depth map. The hybrid nature of our architecture leverages the strengths of both generative and discriminative approaches, creating a comprehensive depth estimation model.  
The VAE component of our architecture consists of an encoder function \(E_{\text{enc}}:\mathbb{R}^{H\times W\times c}\to\mathbb{R}^{Z}\) that compresses the input image into a latent space representation \(z=E_{\text{enc}}(I)\)  
Figure 2: General architecture of the proposed VUDNet for end-to-end 3D reconstruction of specular free-form surfaces. The network integrates a Variational Autoencoder (VAE, **bottom**) for coarse depth estimation and a modified U-Net (**top**) for detail refinement. The ensemble approach leverages both generative and discriminative components, combining their outputs (**right**) to produce accurate depth maps from single-shot 2D images.  
followed by a decoder function \(E_{\text{dec}}:\mathbb{R}^{Z}\rightarrow\mathbb{R}^{H\times W}\) that reconstructs the depth map from this latent representation, producing depth estimate \(\hat{D}_{k}^{\text{V}}=E_{\text{dec}}(z)\). This process allows the VAE to capture and model the complex variations in specular surfaces. On the other hand, the modified U-Net architecture, tailored for depth estimation, employs a series of downsampling and upsampling layers specifically structured to refine the depth map, producing estimate \(\hat{D}_{k}^{\text{U}}\). Unlike the traditional U-Net, our modified version diminishes the effect of skip connections to prevent the direct transfer of the texture information present in the patterning of input image \(I\), thereby ensuring that the output is a detailed depth map rather than a combination of depth and texture features.  
The output depth maps from both the VAE and the modified U-Net are then fed into a fully connected Multi-Layer Perceptron (MLP) that serves as an ensemble method. We let \(f_{\text{MLP}}:\mathbb{R}^{H\times W}\times\mathbb{R}^{H\times W}\rightarrow \mathbb{R}^{H\times W}\) denote the MLP function that combines the depth maps from the two networks:  
\[\hat{D}_{k}=f_{\text{MLP}}(\hat{D}_{k}^{\text{V}},\hat{D}_{k}^{\text{U}}) \approx\psi(\mathcal{F}^{-1}(I)) \tag{1}\]  
Such an ensemble approach is beneficial for several reasons:  
* **Reduced Overfitting:** Ensemble methods inherently reduce the risk of overfitting [34]. They are known for their ability to improve generalization by leveraging the strengths of multiple models and mitigating individual model biases [34]. In our work, this is especially true as we combine one discriminative model and one generative model. Each model captures different aspects of the data distribution, and their combination leads to a more generalizable solution.
* **Complementary Strengths:** The VAE's ability to model complex data distributions complements the U-Net's strength in preserving spatial details, making the ensemble approach particularly effective for depth estimation tasks.
* **Improved Performance:** Hybrid models that incorporate both generative and discriminative components have been shown to outperform single-method models in various tasks [35; 36; 37]. This combination harnesses the power of both methodologies, leading to improved performance in reconstructing 3D surfaces from 2D images.