# Gaussian-UDSR: Real-Time Unbounded Dynamic Scene Reconstruction with 3D Gaussian Splitting
## 3 Method
### Gaussian Color Feature Prediction Network  
Illumination affects the expression of object color, i.e., the spherical harmonic coefficient. In contrast, the default ambient illumination in the original 3 DGS is constant, so the object's color will be optimized incorrectly once the illumination changes. Hence, the spherical harmonic coefficient in the original 3 DGS only applies to static scenes. Therefore, we use a neural network to predict the color features instead of the traditional spherical harmonic coefficients. The color prediction consists of five parts: feature extractor, dynamic mask generator, dynamic feature sampler, feature fusion network and color decoder. The feature extractor and dynamic feature mask generator model are based on the U-Net [39] architecture. As shown in Figure 3, ResNet [40] is used as the basis to capture the multi-level feature information of the image, introduces residual connections to solve the gradient disappearance and explosion problems in the network training, and extracts multiple intermediate layer features from ResNet, and it fuses them in the decoder to recover finer-grained spatial information in the scene and ensure the combination of detail information in the lower layer, local semantic information in the middle layer, and global semantic information in the higher layer, to improve the segmentation accuracy between dynamic objects and static background. The input to the dynamic mask generator includes multi-scale feature maps extracted by a U-Net from the input image, as well as the 2D projections of each Gaussian point. The purpose of this module is to identify which areas of the image correspond to dynamic objects and to generate a binary dynamic mask. For each Gaussian point, the corresponding feature responses are sampled from multiple feature map layers based on its image-plane position. These responses are then fused using a residual network and semantic decoder to determine whether the point lies within a dynamic region. The output is a binary mask aligned with the image space, highlighting the areas associated with dynamic elements. This mask helps the feature prediction network to focus on dynamic-specific cues, improving the accuracy of color and motion estimation and enhancing the overall fidelity of dynamic scene reconstruction. The dynamic feature sampler enables Gaussian points to dynamically capture global and local dynamic appearance feature information on feature map slices by introducing Gaussian point position attributes, focusing on meaningful regions on the feature map. Gaussian points are sampled on multiple feature maps denoted as:  
\[(P\textsubscript{f}_{i}^{1},P\textsubscript{f}_{i}^{2},\cdots,P\textsubscript{f }_{i}^{k})=\sum_{m=1}^{2}\sum_{n=1}^{2}\omega_{x}(p_{i}^{m},x_{m})\cdot\omega_{ y}(p_{i}^{n},y_{n})\cdot F^{m}(x_{m},y_{n}) \tag{10}\]  
where \(P\textsubscript{f}_{i}^{k}\) denotes the predicted feature of the \(i\)-th sampled point on the \(k\)-th feature map. \(p_{i}^{m}\) denotes the coordinates of the learnable sampling point on the \(m\)-th feature map obtained by the camera transform and learning mechanism. \(\omega_{x}(p_{i}^{m},x_{m})\) and \(\omega_{y}(p_{i}^{n},y_{n})\) are weight functions for weighting according to the \(x_{m}\) and \(y_{n}\) positions. \(x_{m}\) and \(y_{n}\), are the horizontal and vertical coordinates of the sampled point on the \(m\)-th feature map. \(m,n\) are indexed from 1 to 2, which are used to calculate the weighted average of the neighborhood around the sampled points. The predicted features \((P\textsubscript{f}_{i}^{1},P\textsubscript{f}_{i}^{2},\cdots,P\textsubscript{f }_{i}^{k})\) of the \(i\)-th Gaussian point sampled from the \(k\) feature maps are stitched together to represent the dynamic appearance of the \(i\)-th Gaussian point:  
\[Pf_{d}=Pf_{i}^{1}\oplus Pf_{i}^{2}\oplus\cdots\oplus Pf_{i}^{k} \tag{11}\]  
The Gaussian color feature prediction network uses an encoder-decoder structure that outputs a 64-dimensional Gaussian appearance feature vector during the color prediction phase.