SpectroMotion: Dynamic 3D Reconstruction of Specular Scenes

## Abstract

Abstract We present SpectroMotion, a novel approach that combines 3D Gaussian Splatting (3DGS) with physically-based rendering (PBR) and deformation fields to reconstruct dynamic specular scenes. Previous methods extending 3DGS to model dynamic scenes have struggled to accurately represent specular surfaces. Our method addresses this limitation by introducing a residual correction technique for accurate surface normal computation during deformation, complemented by a deformable environment map that adapts to time-varying lighting conditions. We implement a coarse-to-fine training strategy that significantly enhances both scene geometry and specular color prediction. We demonstrate that our model outperforms prior methods for view synthesis of scenes containing dynamic specular objects and that it is the only existing 3DGS method capable of synthesizing photorealistic real-world dynamic specular scenes, outperforming state-of-the-art methods in rendering complex, dynamic, and specular scenes.

## 1 Introduction

Figure: Figure 1: Our method, SpectroMotion, recovers and renders dynamic scenes with higher-quality reflections compared to prior work. It introduces physical normal estimation, deformable environment maps, and a coarse-to-fine training strategy to achieve superior results in rendering dynamic scenes with reflections. Here we present a rendered test image along with its corresponding normal maps and a ground-truth image. For Deformable 3DGS, we use the shortest axes of the deformed 3D Gaussians as the normals. We have highlighted the specular regions for a scene from the NeRF-DS dataset (Yan et al., ) to demonstrate the effectiveness of our approach.
Refer to caption: x1.png

3D Gaussian Splatting (3DGS) (Kerbl et al., ) has emerged as a groundbreaking technique in 3D scene reconstruction, offering fast training and real-time rendering capabilities. By representing 3D space using a collection of 3D Gaussians and employing a point-based rendering approach, 3DGS has significantly improved efficiency in novel-view synthesis. However, extending 3DGS to accurately model dynamic scenes, especially those containing specular surfaces, has remained a significant challenge.

Existing extensions of 3DGS have made progress in either dynamic scene reconstruction or specular object rendering, but none have successfully combined both aspects. Methods tackling dynamic scenes often struggle with accurate representation of specular surfaces, while those focusing on specular rendering are limited to static scenes. This gap in capabilities has hindered the application of 3DGS to real-world scenarios where both motion and specular reflections are present.

We present SpectroMotion, a novel approach that addresses these limitations by combining 3D Gaussian Splatting with physically-based rendering (PBR) and deformation fields. Our method introduces three key innovations: a residual correction technique for accurate surface normal computation during deformation, a deformable environment map that adapts to time-varying lighting conditions, and a coarse-to-fine training strategy that significantly enhances both scene geometry and specular color prediction.

Our evaluations demonstrate that SpectroMotion outperforms prior methods in view synthesis of scenes containing dynamic specular objects, as illustrated in Figure 1. It is the only existing 3DGS method capable of synthesizing photorealistic real-world dynamic specular scenes, surpassing state-of-the-art techniques in rendering complex, dynamic, and specular content. This advancement represents a significant leap in 3D scene reconstruction, particularly for challenging scenarios involving moving specular objects.

In summary, we make the following contributions:

- •
We propose SpectroMotion, a physically-based rendering (PBR) approach combining deformation fields and 3D Gaussian Splatting for real-world dynamic specular scenes.
- •
We introduce a residual correction method for accurate surface normals during deformation, coupled with a deformable environment map to handle time-varying lighting conditions in dynamic scenes.
- •
We develop a coarse-to-fine training strategy enhancing scene geometry and specular color prediction, outperforming state-of-the-art methods.

## 2 Related Work

##### Dynamic Scene Reconstruction.

Recent works have leveraged NeRF representations to jointly solve for canonical space and deformation fields in dynamic scenes using RGB supervision (Guo et al., ; Li et al., ; Park et al., ; ; Pumarola et al., ; Tretschk et al., ; Xian et al., ; Liu et al., ). Further advancements in dynamic neural rendering include object segmentation (Song et al., ), incorporation of depth information (Attal et al., ), utilization of 2D CNNs for scene priors (Lin et al., ; Peng et al., ), and multi-view video compression (Li et al., ). However, these NeRF-based methods are computationally intensive. To address this, recent research has adapted 3D Gaussians for dynamic scenes (Yang et al., ; Wu et al., ; Huang et al., ; Liang et al., ; Wang et al., ; Mihajlovic et al., ; Stearns et al., ), primarily focusing on deforming spatial coordinates through deformation fields. Nevertheless, these approaches do not explicitly account for changes in object surface during the deformation process. Our work extends this line of research by combining specular object rendering based on normal estimation with a deformation field, enabling each 3D Gaussian to effectively model dynamic specular scenes.

##### Reflective Object Rendering.

While significant progress has been made in rendering reflective objects, challenges arising from complex light interactions persist. Recent years have seen numerous studies addressing these issues, primarily by decomposing appearance into lighting and material properties  (Bi et al., ; Boss et al., ; Li & Li, ; Srinivasan et al., ; Zhang et al., ; Munkberg et al., ; Zhang et al., ; Verbin et al., ; Zhao et al., ). Building on this foundation, some research has focused on improving the capture and reproduction of specular reflections  (Verbin et al., ; Ma et al., ; Verbin et al., ), while others have leveraged signed distance functions (SDFs) to enhance normal estimation  (Ge et al., ; Liang et al., ; ; Liu et al., ; Zhang et al., ).
The emergence of 3D Gaussian Splatting (3DGS) has sparked a new wave of techniques  (Jiang et al., ; Liang et al., ; Yang et al., ; Ye et al., ; Zhu et al., ; Shi et al., ) that integrate Gaussian splatting with physically-based rendering. Nevertheless, accurately modeling dynamic environments and time-varying specular reflections remains a significant challenge. To address this limitation, our work introduces a novel approach incorporating a deformable environment map and additional explicit Gaussian attributes, specifically designed to capture specular color changes over time.

Figure: Figure 2: Method Overview. Our method stabilizes the scene geometry through three stages. In the static stage, we stabilize the geometry of the static scene by minimizing photometric loss $\mathcal{L}_{\text{color}}$ between vanilla 3DGS renders and ground truth images. The dynamic stage combines canonical 3D Gaussians G with a deformable Gaussian MLP to model dynamic scenes while simultaneously minimizing normal loss $\mathcal{L}_{\text{normal}}$ between rendered normal map $\mathbf{N}^{t}$ and gradient normal map from depth map ${\mathbf{D}^{t}}$, thus further enhancing the overall scene geometry. Finally, the specular stage introduces a deformable reflection MLP to handle changing environment lighting, deforming reflection directions $\omega^{t}_{r}$ to query a canonical environment map for specular color $\mathbf{c}_{s}^{t}$. It is then combined with diffuse color $\mathbf{c_{d}}$ (using zero-order spherical harmonics) and learnable specular tint $\mathbf{s_{\mathbf{tint}}}$ per 3D Gaussian to obtain the final color $\mathbf{c}_{\mathbf{final}}^{t}$. This approach enables the modeling of dynamic specular scenes and high-quality novel view rendering.
Refer to caption: x2.png

## 3 Method

##### Overview of the approach.

The overview of our method is illustrated in Fig. 2. Given an input monocular video sequence of frames and corresponding camera poses, we design a three-stage approach to reconstruct the dynamic specular scene, as detailed in Section 3.2. Accurate specular reflection requires precise normal estimation, so Section 3.3 elaborates on how we estimate normals in dynamic scenes. Finally, we introduce the losses used throughout the training process in Section 3.4.

### 3.1 Preliminary

##### 3D Gaussian Splatting.

Each 3D Gaussian is defined by a center position $\bm{x}\in\mathbb{R}^{3}$ and a covariance matrix $\bm{\Sigma}$. 3D Gaussian Splatting (Kerbl et al., ) optimizes the covariance matrix using scaling factors $\bm{s}\in\mathbb{R}^{3}$ and rotation unit quaternion $\bm{r}\in\mathbb{R}^{4}$. For novel-view rendering, 3D Gaussians are projected onto 2D camera planes using differentiable splatting (Yifan et al., ):

$$ $\mathbf{\Sigma}^{\prime}\mathbf{=}\mathbf{J}\mathbf{W}\mathbf{\Sigma}\mathbf{W }^{T}\mathbf{J}^{T}.$ (1) $$

Pixel colors are computed using point-based volumetric rendering:

$$ $C=\sum_{i\in{N}}{T}_{i}\mathbf{\alpha}_{i}{c}_{i},\quad\mathbf{\alpha}_{i}= \mathbf{\sigma}_{i}e^{-\frac{1}{2}(\bm{x})^{T}\mathbf{\Sigma}^{\prime}(\bm{x})},$ (2) $$

where ${T}_{i}=\prod_{j=1}^{i-1}(1-\alpha_{j})$ is the transmittance, $\sigma_{i}$ is the opacity, and $\mathbf{c}_{i}$ is the color of each 3D Gaussian.

### 3.2 Specular Rendering

##### Limitations of existing methods.

The current Dynamic 3DGS-based methods (Wu et al., ; Yang et al., ; ) encounter limitations in accurately modeling environments that include specular objects. This issue arises from the inherent low-frequency characteristics of low-order spherical harmonics (SH), which are inadequate for capturing complex visual effects such as specular highlights. In contrast, other specialized 3DGS-based methods for static specular object scenes (Jiang et al., ; Liang et al., ) often incorporate environment maps to model lighting, which is then combined with BRDF to simulate the entire scene. However, vanilla environment maps are not suitable for modeling lighting scenarios that involve time-variant elements. This results in the existing 3DGS-based methods being insufficient for effectively modeling dynamic specular object scenes.

##### Proposed solution overview.

To address these challenges, we introduce physical normal estimation (Section 3.3) and deformable environment maps to model the specular color of real-world dynamic scenes. However, this approach alone is insufficient, as precise scene geometry is crucial for accurate reflections. Therefore, we introduce our coarse-to-fine training strategy, which helps stabilize scene geometry while simultaneously predicting accurate specular color. Our coarse-to-fine training strategy is divided into three stages: the static stage, the dynamic stage, and the specular stage. In the following paragraphs, we will introduce each of these stages in detail.

#### 3.2.1 Coarse-to-Fine Training Strategy

##### Static stage.

In the static stage, we employ vanilla 3DGS (Kerbl et al., ) for static scene reconstruction to stabilize the geometry of the static scene. Specifically, we optimize the position $\bm{x}$, scaling $\bm{s}$, rotation $\bm{r}$, opacity $\alpha$, and coefficients of spherical harmonics (SH) of the 3D Gaussians by minimizing the photometric loss $\mathcal{L}_{\text{color}}$ between the rendered image and the corresponding image:

$$ $\mathcal{L}_{\text{color}}=(1-\lambda_{\text{D-SSIM}})\mathcal{L}_{\text{1}}+ \lambda_{\text{D-SSIM}}\mathcal{L}_{\text{D-SSIM}}.$ (3) $$

##### Dynamic stage.

Following the static stage, we address dynamic objects using Deformable 3DGS (Yang et al., ). For each 3D Gaussian in canonical 3D Gaussians G, we input its position $\bm{x}$ and time $t$ into a deformable Gaussian MLP with parameters $\theta_{\textit{G}}$ to predict position, rotation, and scaling residuals:

$$ $(\Delta\bm{x}^{t},\Delta\bm{r}^{t},\Delta\bm{s}^{t})=F_{\theta_{G}}(\gamma(x), \gamma(t)),$ (4) $$

where $\gamma$ denotes positional encoding.
Attributes of the corresponding 3D Gaussian in deformed 3D Gaussians $\textbf{G}^{t}$ at time $t$ is obtained by:

$$ $(\bm{x}^{t},\bm{r}^{t},\bm{s}^{t})=(\Delta\bm{x}^{t},\Delta\bm{r}^{t},\Delta \bm{s}^{t})+(\bm{x},\bm{r},\bm{s}).$ (5) $$

This approach separates motion and geometric structural learning, allowing accurate simulation of dynamic behaviors while maintaining a stable geometric reference.
To further enhance scene geometry, we estimate normals of deformed 3D Gaussians and optimize them using:

$$ $\mathcal{L}_{\text{normal}}=1-{\mathbf{N}^{t}}\cdot\hat{\mathbf{N}^{t}},$ (6) $$

where ${\mathbf{N}^{t}}$ is the rendered normal map and $\hat{\mathbf{N}^{t}}$ is the normal map derived from the rendered depth map ${\mathbf{D}^{t}}$. This process improves local associations among 3D Gaussians and optimizes both depth and normal information across the entire scene.

##### Specular stage.

We adopt an image-based lighting (IBL) model with a learnable cube map. Inspired by the rendering equation (Kajiya, ), split-sum approximation (Karis & Games, ; Munkberg et al., ), and Cook-Torrance reflectance model (Cook & Torrance, ), we formulate the outgoing radiance of the specular component $L_{s}$ as:

$$ $L_{s}=\int_{\Omega}\frac{DGF}{4(\omega^{t}_{o}\cdot\mathbf{n}^{t})(\omega_{i} \cdot\mathbf{n}^{t})}(\omega_{i}\cdot\mathbf{n}^{t})d\omega_{i}\int_{\Omega}L_ {i}(\omega_{i})D(\omega_{i},\omega^{t}_{o})(\omega_{i}\cdot\mathbf{n}^{t})d \omega_{i},$ (7) $$

where $\Omega$ is the hemisphere around the surface normal $\mathbf{n}^{t}$. $D$, $G$, and $F$ represent the GGX normal distribution function (Walter et al., ), geometric attenuation, and Fresnel term, respectively. $\omega^{t}_{o}$ is the view direction, and $L_{i}(\omega_{i})$ is the incident radiance.
The first term, representing the specular BSDF with a solid white environment light, is precomputed and stored in a look-up table. The second term is pre-integrated in a filtered cubemap, where each mip-level corresponds to a specific roughness value. Roughness $\rho\in[0,1]$ is a learnable parameter for each 3D Gaussian.
After the static and dynamic stages, the geometry is well-defined. This allows us to accurately calculate reflection directions $\omega^{t}_{r}$:

$$ $\omega^{t}_{r}=2(\omega^{t}_{o}\cdot\mathbf{n}^{t})\mathbf{n}^{t}-\omega^{t}_{ o}.$ (8) $$

Reflection directions can query the environment map for the specular color of static environment light. To handle time-varying lighting in dynamic scenes, we introduce a deformable environment map, detailed in the following section.

#### 3.2.2 Deformable Environment Map for Dynamic Lighting.

The concept of a deformable environment map involves treating the vanilla environment map as a canonical environment map and combining it with a deformation field. This approach enables us to model time-varying lighting conditions effectively. To implement this, we first apply positional encoding to the reflection direction $\omega^{t}_{r}$ and time $t$. These encoded values are then input into a deformable reflection MLP with parameters $\theta_{\textit{R}}$. This process allows us to obtain the deformed reflection residual $\Delta\bar{\omega}^{t}_{r}$ for each specified time $t$:

$$ $\Delta\bar{\omega}^{t}_{r}=F_{\theta_{\textit{R}}}(\gamma(\omega^{t}_{r}), \gamma(t)).$ (9) $$

Subsequently, we add the deformed reflection residual $\Delta\bar{\omega}^{t}_{r}$ to the reflection direction $\omega^{t}_{r}$, yielding the deformed reflection direction $\bar{\omega}^{t}_{r}$. This can be expressed as:

$$ $\bar{\omega}^{t}_{r}=\Delta\bar{\omega}^{t}_{r}+\omega^{t}_{r}$ (10) $$

We can then use this deformed reflection direction $\bar{\omega}^{t}_{r}$ to query the canonical environment map, allowing us to obtain time-varying specular color $\mathbf{c}_{s}^{t}$. This approach effectively captures the dynamic nature of lighting in the scene while maintaining a stable canonical reference.

#### 3.2.3 Color Decomposition and Staged Training Strategy.

We decompose the final color $\mathbf{c}_{\mathbf{final}}^{t}$ into diffuse and specular components to better distinguish between high and low-frequency information:

$$ $\mathbf{c}_{\mathbf{final}}^{t}=\mathbf{c}_{d}+\mathbf{c}_{s}^{t}\odot\mathbf{ s_{\mathbf{tint}}},$ (11) $$

where $\mathbf{c_{d}}$ is the diffuse color (using zero-order spherical harmonics as view-independent color), $\mathbf{s_{\mathbf{tint}}}\in[0,1]^{3}$ is the learnable specular tint stored in each 3D Gaussian, and $\mathbf{c_{s}}^{t}$ is the view-dependent color component.
To manage the transition from spherical harmonics to $\mathbf{c}_{\mathbf{final}}^{t}$ and mitigate potential geometry disruptions, in the early specular stage, we fix the deformable Gaussian MLP and most 3D Gaussian attributes, optimizing only zero-order SH, specular tint, and roughness. We temporarily suspend densification during this phase. As $\mathbf{c}_{\mathbf{final}}^{t}$ becomes more complete, we gradually resume optimization of all parameters and reinstate the densification process.

We further split the specular stage into two parts, applying a coarse-to-fine strategy to the environment map. In the first part, we focus on optimizing the canonical environment map for time-invariant lighting. This establishes a stable foundation for the overall lighting structure. In the second part, we proceed to optimize the deformable reflection MLP for dynamic elements. This approach ensures a more robust learning process, allowing us to capture the static lighting conditions before introducing the complexities of dynamic components.

### 3.3 Physical Normal Estimation

##### Challenges in normal estimation for 3D Gaussians.

Normal estimation is crucial for modeling specular objects, as it directly affects surface reflections. However, the discrete nature of 3D Gaussians makes this process challenging, as it typically requires a continuous surface. GaussianShader (Jiang et al., ) observed that 3D Gaussians tend to flatten during training, leading to the use of the shortest axis as an initial approximation of the surface normal. To improve accuracy, they introduced a residual normal $\Delta\mathbf{n}$ for each 3D Gaussian to compensate for errors in this approximation.
However, this method alone is insufficient for deformed 3D Gaussians, as the residual should vary at each time step. A straightforward approach of rotating the residual based on the quaternion difference between canonical and deformed Gaussians proves inadequate, as it fails to account for shape changes during deformation. If the shortest axis of the canonical 3D Gaussian is no longer the shortest after deformation, this method results in incorrect rotation.
Consequently, a more sophisticated approach is needed to accurately model the normals of deformed 3D Gaussians. This approach must consider both the rotation and the change in shape during the deformation process, ensuring accurate normal estimation for dynamic specular objects.

##### Improved rotation calculation for deformed 3D Gaussians.

To overcome the limitations of naive methods and accurately model the normal of deformed 3D Gaussians, we propose using both the shortest and longest axes of canonical and deformed Gaussians to compute the rotation matrix. This approach accounts for both rotation and shape changes during deformation.
We first align the deformed Gaussian’s axes with those of the canonical Gaussian using the following method:

$$ $\mathbf{v}_{s}^{t}=\begin{cases}\mathbf{v}_{s}^{t}&\text{if }\mathbf{v}_{s} \cdot\mathbf{v}_{s}^{t}>0,\\ -\mathbf{v}_{s}^{t}&\text{otherwise.}\end{cases},\quad\mathbf{v}_{l}^{t}= \begin{cases}\mathbf{v}_{l}^{t}&\text{if }\mathbf{v}_{l}\cdot\mathbf{v}_{l}^{t }>0,\\ -\mathbf{v}_{l}^{t}&\text{otherwise.}\end{cases},$ (12) $$

where $\mathbf{v}_{s}$ and $\mathbf{v}_{l}$ represent the shortest and longest axes of canonical 3D Gaussians, while $\mathbf{v}_{s}^{t}$ and $\mathbf{v}_{l}^{t}$ denote the same for deformed 3D Gaussians. We then construct orthogonal matrices using these aligned axes and their cross products:

$$ $\mathbf{U}=\begin{bmatrix}\mathbf{v}_{s}&\mathbf{v}_{l}&\mathbf{v}_{s}\times \mathbf{v}_{l}\end{bmatrix},\quad\mathbf{V}^{t}=\begin{bmatrix}\mathbf{v}_{s}^ {t}&\mathbf{v}_{l}^{t}&\mathbf{v}_{s}^{t}\times\mathbf{v}_{l}^{t}\end{bmatrix}.$ (13) $$

Finally, we derive the rotation matrix:

$$ $\mathbf{R}^{t}=\mathbf{V}^{t}\mathbf{U}^{\top}.$ (14) $$

This method provides a robust solution for calculating the rotation of deformation process, ensuring accurate normal estimation for dynamic specular objects.

Figure: Figure 3: Normal estimation. (a) shows that flatter 3D Gaussians align better with scene surfaces, their shortest axis closely matching the surface normal. In contrast, less flat 3D Gaussians fit less accurately, with their shortest axis diverging from the surface normal. (b) shows that when the deformed 3D Gaussian becomes flatter ($t=t_{1}$), normal residual $\Delta\mathbf{n}$ is rotated by $\mathbf{R}^{t}_{1}$ and scaled down by $\frac{\beta}{\beta^{t}_{1}}$, as flatter Gaussians require smaller normal residuals. Conversely, when the deformation results in a less flat shape ($t=t_{2}$), $\Delta\mathbf{n}$ is rotated by $\mathbf{R}^{t}_{2}$ and amplified by $\frac{\beta}{\beta^{t}_{2}}$, requiring a larger correction to align the shortest axis with the surface normal. (c) shows how $\gamma^{k}$ changes with $w$ (where $w=\frac{|\mathbf{v}_{s}^{t}|}{|\mathbf{v}_{l}^{t}|}$) for $k=1$, $k=5$, and $k=50$. Larger $w$ indicates less flat Gaussians, while smaller $w$ represents flatter Gaussians. As $k$ increases, $\gamma^{k}$ decreases more steeply as $w$ rises. For $k=5$, we observe a balanced behavior: $\gamma^{k}$ approaches 1 for low $w$ and 0 for high $w$, providing a nuanced penalty adjustment across different Gaussian shapes.
Refer to caption: x3.png

##### Adjusting normal residuals and ensuring accuracy.

To account for shape changes during deformation, we scale the normal residual based on the ratio of oblateness $\frac{\beta}{\beta^{t}}$ between canonical and deformed 3D Gaussians.

$$ $\beta=\frac{\lvert\mathbf{v}_{l}\rvert-\lvert\mathbf{v}_{s}\rvert}{\lvert \mathbf{v}_{l}\rvert},\quad\beta^{t}=\frac{\lvert\mathbf{v}_{l}^{t}\rvert- \lvert\mathbf{v}_{s}^{t}\rvert}{\lvert\mathbf{v}_{l}^{t}\rvert},$ (15) $$

where $\beta$ and $\beta^{t}$ represent the oblateness of canonical and deformed 3D Gaussians, respectively. This is because flatter 3D Gaussians tend to align more closely with the surface, meaning their shortest axis becomes more aligned with the surface normal, as shown in Fig. 3(a). In such cases, less compensation from the normal residual is needed. Conversely, less flat Gaussians require more compensation, as illustrated in Fig. 3(b). We then obtain deformed normal residuals:

$$ $\Delta\mathbf{n}^{t}=\frac{\beta}{\beta^{t}}\mathbf{R}^{t}\Delta\mathbf{n}.$ (16) $$

The final normal $\mathbf{n}^{t}$ is computed by adding this residual to the shortest axis and ensuring outward orientation:

$$ $\mathbf{n}^{t}=\Delta\mathbf{n}^{t}+\mathbf{v}_{s}^{t},\quad\mathbf{n}^{t}= \begin{cases}\mathbf{n}^{t}&\text{if }\mathbf{n}^{t}\cdot\omega^{t}_{o}>0,\\ -\mathbf{n}^{t}&\text{otherwise.}\end{cases}$ (17) $$

This approach adjusts for Gaussian flatness and ensures accurate normal estimation.

### 3.4 Loss Functions

##### Normal regularization.

To allow the normal residual to correct the normal while not excessively influencing the optimization of the shortest axis towards the surface normal, we introduce a penalty term for the normal residual:

$$ $\mathcal{L}_{\text{reg}}=\gamma^{k}\|\Delta\mathbf{n}\|_{2}^{2}\quad\text{ where}\quad\gamma=\sqrt{1-\frac{\lvert\mathbf{v}_{s}^{t}\rvert^{2}}{\lvert \mathbf{v}_{l}^{t}\rvert^{2}}}.$ (18) $$

In our experiments, we set $k=5$. When $k=5$, less flatter 3D Gaussians have $\gamma^{k}$ approaching 0. Their shortest axis aligns poorly with the surface normal, requiring more normal residual correction and smaller penalties. Conversely, flatter Gaussians have $\gamma^{k}$ near 1. Their shortest axis aligns better with the surface normal, needing less normal residual correction and allowing larger penalties, as shown in Fig. 3(c).

**Table 1: Quantitative comparison on the NeRF-DS (Yan et al., ) dataset. We report the average PSNR, SSIM, and LPIPS (VGG) of several previous models on test images. The best, the second best, and third best results are denoted by red, orange, yellow.**
|  | As |  | Basin |  | Bell |  | Cup |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |
| Deformable 3DGS (Yang et al., ) | 26.04 | 0.8805 | 0.1850 |  | 19.53 | 0.7855 | 0.1924 |  | 23.96 | 0.7945 | 0.2767 |  | 24.49 | 0.8822 | 0.1658 |
| 4DGS (Wu et al., ) | 24.85 | 0.8632 | 0.2038 |  | 19.26 | 0.7670 | 0.2196 |  | 22.86 | 0.8015 | 0.2061 |  | 23.82 | 0.8695 | 0.1792 |
| GaussianShader (Jiang et al., ) | 21.89 | 0.7739 | 0.3620 |  | 17.79 | 0.6670 | 0.4187 |  | 20.69 | 0.8169 | 0.3024 |  | 20.40 | 0.7437 | 0.3385 |
| GS-IR (Liang et al., ) | 21.58 | 0.8033 | 0.3033 |  | 18.06 | 0.7248 | 0.3135 |  | 20.66 | 0.7829 | 0.2603 |  | 20.34 | 0.8193 | 0.2719 |
| NeRF-DS (Yan et al., ) | 25.34 | 0.8803 | 0.2150 |  | 20.23 | 0.8053 | 0.2508 |  | 22.57 | 0.7811 | 0.2921 |  | 24.51 | 0.8802 | 0.1707 |
| HyperNeRF (Park et al., ) | 17.59 | 0.8518 | 0.2390 |  | 22.58 | 0.8156 | 0.2497 |  | 19.80 | 0.7650 | 0.2999 |  | 15.45 | 0.8295 | 0.2302 |
| Ours | 26.80 | 0.8851 | 0.1761 |  | 19.75 | 0.7922 | 0.1896 |  | 25.46 | 0.8497 | 0.1600 |  | 24.65 | 0.8879 | 0.1588 |
|  | Plate |  | Press |  | Sieve |  | Mean |  |  |  |  |  |  |  |  |
| Method | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |
| Deformable 3DGS (Yang et al., ) | 19.07 | 0.7352 | 0.3599 |  | 25.52 | 0.8594 | 0.1964 |  | 25.37 | 0.8616 | 0.1643 |  | 23.43 | 0.8284 | 0.2201 |
| 4DGS (Wu et al., ) | 18.77 | 0.7709 | 0.2721 |  | 24.82 | 0.8355 | 0.2255 |  | 25.16 | 0.8566 | 0.1745 |  | 22.79 | 0.8235 | 0.2115 |
| GaussianShader (Jiang et al., ) | 14.55 | 0.6423 | 0.4955 |  | 19.97 | 0.7244 | 0.4507 |  | 22.58 | 0.7862 | 0.3057 |  | 19.70 | 0.7363 | 0.3819 |
| GS-IR (Liang et al., ) | 15.98 | 0.6969 | 0.4200 |  | 22.28 | 0.8088 | 0.3067 |  | 22.84 | 0.8212 | 0.2236 |  | 20.25 | 0.7796 | 0.2999 |
| NeRF-DS (Yan et al., ) | 19.70 | 0.7813 | 0.2974 |  | 25.35 | 0.8703 | 0.2552 |  | 24.99 | 0.8705 | 0.2001 |  | 23.24 | 0.8384 | 0.2402 |
| HyperNeRF (Park et al., ) | 21.22 | 0.7829 | 0.3166 |  | 16.54 | 0.8200 | 0.2810 |  | 19.92 | 0.8521 | 0.2142 |  | 19.01 | 0.8167 | 0.2615 |
| Ours | 20.84 | 0.8180 | 0.2198 |  | 26.49 | 0.8665 | 0.1889 |  | 25.22 | 0.8712 | 0.1513 |  | 24.17 | 0.8529 | 0.1778 |

Figure: Figure 4: Qualitative comparison on the NeRF-DS Yan et al. () dataset.
Refer to caption: x4.png

##### Total training loss.

To refine all parameters in the dynamic and specular stages, we employ the total training loss:

$$ $\mathcal{L}=\mathcal{L}_{\text{color}}+\lambda_{\text{normal}}\mathcal{L}_{ \text{normal}}+\mathcal{L}_{\text{reg}},$ (19) $$

where $\mathcal{L}_{\text{color}}$ and $\mathcal{L}_{\text{normal}}$ are obtained as described in Section 3.2.1.
In our experiments, we set $\lambda_{\text{normal}}=0.01$.

**Table 2: Quantitative comparison on the NeRF-DS (Yan et al., ) dataset with our labeled dynamic specular masks. We report PSNR, SSIM, and LPIPS (VGG) of previous methods on dynamic specular objects using the dynamic specular objects mask generated by Track Anything (Yang et al., ). The best, the second best, and third best results are denoted by red, orange, yellow.**
|  | As |  | Basin |  | Bell |  | Cup |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |
| Deformable 3DGS (Yang et al., ) | 24.14 | 0.7432 | 0.2957 |  | 17.45 | 0.5530 | 0.3138 |  | 19.42 | 0.5516 | 0.2940 |  | 20.10 | 0.5446 | 0.3312 |
| 4DGS (Wu et al., ) | 22.70 | 0.6993 | 0.3517 |  | 16.61 | 0.4797 | 0.4084 |  | 14.64 | 0.2596 | 0.4467 |  | 18.90 | 0.4132 | 0.4032 |
| GaussianShader (Jiang et al., ) | 19.27 | 0.5652 | 0.5232 |  | 15.71 | 0.4163 | 0.5941 |  | 12.10 | 0.1676 | 0.6764 |  | 14.90 | 0.3634 | 0.6146 |
| GS-IR (Liang et al., ) | 19.32 | 0.5857 | 0.4782 |  | 15.21 | 0.4009 | 0.5644 |  | 12.09 | 0.1757 | 0.6722 |  | 14.80 | 0.3445 | 0.6046 |
| NeRF-DS (Yan et al., ) | 23.67 | 0.7478 | 0.3635 |  | 17.98 | 0.5537 | 0.4211 |  | 14.73 | 0.2439 | 0.5931 |  | 19.95 | 0.5079 | 0.3494 |
| HyperNeRF (Park et al., ) | 17.37 | 0.6934 | 0.3834 |  | 18.75 | 0.5671 | 0.4125 |  | 13.93 | 0.2292 | 0.6051 |  | 15.07 | 0.4860 | 0.4183 |
| Ours | 24.51 | 0.7534 | 0.2896 |  | 17.71 | 0.5675 | 0.3048 |  | 19.60 | 0.5680 | 0.2862 |  | 20.28 | 0.5473 | 0.3176 |
|  | Plate |  | Press |  | Sieve |  | Mean |  |  |  |  |  |  |  |  |
| Method | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |
| Deformable 3DGS (Yang et al., ) | 16.12 | 0.5192 | 0.3544 |  | 19.64 | 0.6384 | 0.3268 |  | 20.74 | 0.5283 | 0.3109 |  | 19.66 | 0.5826 | 0.3181 |
| 4DGS (Wu et al., ) | 13.93 | 0.4095 | 0.4229 |  | 20.17 | 0.5434 | 0.4339 |  | 19.70 | 0.4498 | 0.3879 |  | 18.09 | 0.4649 | 0.4078 |
| GaussianShader (Jiang et al., ) | 9.87 | 0.2992 | 0.6812 |  | 16.84 | 0.4408 | 0.6093 |  | 16.19 | 0.3241 | 0.5862 |  | 14.98 | 0.3681 | 0.6121 |
| GS-IR (Liang et al., ) | 11.09 | 0.3254 | 0.6270 |  | 16.43 | 0.4083 | 0.5776 |  | 16.42 | 0.3339 | 0.5749 |  | 15.05 | 0.3678 | 0.5856 |
| NeRF-DS (Yan et al., ) | 14.80 | 0.4518 | 0.3987 |  | 19.77 | 0.5835 | 0.5035 |  | 20.28 | 0.5173 | 0.4067 |  | 18.74 | 0.5151 | 0.4337 |
| HyperNeRF (Park et al., ) | 16.03 | 0.4629 | 0.3775 |  | 14.10 | 0.5365 | 0.5023 |  | 18.39 | 0.5296 | 0.3949 |  | 16.23 | 0.5007 | 0.4420 |
| Ours | 16.53 | 0.5369 | 0.3041 |  | 21.70 | 0.6630 | 0.3252 |  | 20.36 | 0.5089 | 0.3190 |  | 20.10 | 0.5921 | 0.3066 |

Figure: Figure 5: Qualitative comparison on NeRF-DS dataset with labeled dynamic specular masks.
Refer to caption: x5.png

**Table 3: Quantitative comparison on the HyperNeRF (Park et al., ) dataset. We report the average PSNR, SSIM, and LPIPS (VGG) of several previous models. The best, the second best, and third best results are denoted by red, orange, yellow.**
|  | Broom |  | 3D printer |  | Chicken |  | Peel Banana |  | Mean |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |  | PSNR↑ | SSIM↑ | LPIPS↓ |
| Deformable 3DGS (Yang et al., ) | 22.35 | 0.4952 | 0.5148 |  | 21.47 | 0.6921 | 0.2147 |  | 23.55 | 0.6747 | 0.2334 |  | 21.28 | 0.5302 | 0.4472 |  | 22.16 | 0.5981 | 0.3525 |
| 4DGS (Wu et al., ) | 21.21 | 0.3555 | 0.5669 |  | 21.90 | 0.6993 | 0.3198 |  | 28.69 | 0.8143 | 0.2772 |  | 27.77 | 0.8431 | 0.2049 |  | 24.89 | 0.6781 | 0.3422 |
| GaussianShader (Jiang et al., ) | 17.21 | 0.2263 | 0.5812 |  | 17.31 | 0.5926 | 0.5054 |  | 19.70 | 0.6520 | 0.5004 |  | 19.99 | 0.7097 | 0.3308 |  | 18.55 | 0.5452 | 0.4795 |
| GS-IR (Liang et al., ) | 20.46 | 0.3420 | 0.5229 |  | 18.24 | 0.5745 | 0.5204 |  | 20.64 | 0.6592 | 0.4536 |  | 20.15 | 0.7159 | 0.3021 |  | 19.87 | 0.5729 | 0.4498 |
| NeRF-DS (Yan et al., ) | 22.37 | 0.4371 | 0.5694 |  | 22.16 | 0.6973 | 0.3134 |  | 27.32 | 0.7949 | 0.3139 |  | 22.75 | 0.6328 | 0.3919 |  | 23.65 | 0.6405 | 0.3972 |
| HyperNeRF (Park et al., ) | 20.72 | 0.4276 | 0.5773 |  | 21.94 | 0.7003 | 0.3090 |  | 27.40 | 0.8013 | 0.3052 |  | 22.36 | 0.6257 | 0.3956 |  | 23.11 | 0.6387 | 0.3968 |
| Ours | 22.04 | 0.5145 | 0.4494 |  | 19.96 | 0.6444 | 0.2397 |  | 22.20 | 0.6203 | 0.1970 |  | 27.34 | 0.8895 | 0.1290 |  | 22.89 | 0.6672 | 0.2538 |

## 4 Experiments

### 4.1 Evaluation Results

We evaluate our method on two real-world datasets: NeRF-DS dataset (Yan et al., ) and HyperNeRF dataset (Park et al., ).

##### Entire scene of the NeRF-DS dataset.

The NeRF-DS dataset (Yan et al., ) is a monocular video dataset comprising seven real-world scenes from daily life, featuring various types of moving or deforming specular objects. We compare our method with the most relevant state-of-the-art approaches. As shown in Tab. 1 and Fig. 4, the quantitative results demonstrate that our method decisively outperforms baselines in reconstructing and rendering real-world highly reflective dynamic specular scenes.

##### Dynamic specular object of NeRF-DS dataset.

Since each scene in the NeRF-DS dataset (Yan et al., ) contains not only dynamic specular objects but also static background objects, we use Track Anything (Yang et al., ) to obtain masks for the dynamic specular objects. This allows us to evaluate only the dynamic specular objects. As shown in Tab. 2 and Fig. 5, our method outperforms baselines when evaluating the dynamic specular objects in these monocular sequences.

##### HyperNeRF dataset.

The HyperNeRF dataset, while also containing real-world dynamic scenes, does not include specular objects. As shown in Tab. 3 and appendix Fig. 14, the results demonstrate that we are on par with state-of-the-art techniques for rendering novel views and our method’s performance is not limited to shiny scenes.

This strong performance across different types of real-world datasets further confirms the effectiveness of our approach in handling a wide range of scene characteristics. The success can be attributed to our key innovations: physical normal estimation, deformable environment map, and coarse-to-fine training strategy, which together enable robust reconstruction and rendering of diverse scenes. Notably, unlike NeRF-DS, our approach does not require mask supervision to clearly distinguish between static and dynamic objects, as illustrated in Fig. 6. Additionally, Fig. 7 illustrates our method’s decomposition results. As shown, our approach consistently achieves a realistic separation of specular and diffuse components across different scenes in the NeRF-DS dataset.

**Table 4: Ablation studies on different coarse to fine training strategy stages.**
| Stage | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
| --- | --- | --- | --- |
| Static | 20.26 | 0.7785 | 0.2953 |
| St. + Dynamic | 24.02 | 0.8508 | 0.1831 |
| St. + Dy. + Specular | 24.17 | 0.8529 | 0.1778 |

Figure: Figure 6: Visualization our deformation magnitudes. (a) The left side shows the ground truth of the dynamic object, while (b) on the right side, we render the magnitude of the output of the position residual by our deformable Gaussian MLP. The brighter areas indicate greater movement of the 3D Gaussians. The figure shows that even without mask supervision, our method can still effectively distinguish which objects are dynamic.
Refer to caption: x6.png

Figure: Figure 7: Visualization our specular and diffuse color. Specular regions are emphasized while non-specular areas are dimmed to highlight the results of specular region color decomposition.
Refer to caption: x7.png

Figure: Figure 8: Qualitative comparison of each training stage in our coarse-to-fine approach.
Refer to caption: x8.png

Figure: Figure 9: Qualitative comparison of ablation study without different components.
Refer to caption: x9.png

Figure: Figure 10: Qualitative comparison of ablation study on SH, Static environment map, and Deformable enviorment map.
Refer to caption: x10.png

### 4.2 Ablation Study

##### Different coarse to fine training strategy stages.

As shown in the Tab. 6 and Fig. 8 , each stage contributes effectively to the model’s performance. The Dynamic stage enhances object stability compared to the Static stage alone, while the Specular stage improves reflection clarity beyond the combined Static and Dynamic stages. This coarse-to-fine approach establishes a stable geometric foundation before addressing complex specular effects. Note that the total iterations for each row in the Tab. 6 are 40,000.

##### Ablation study on coarse-to-fine, and loss function.

The model’s performance was evaluated without key components: the coarse-to-fine training strategy, normal loss $\mathcal{L}_{\text{normal}}$, normal regularization $\mathcal{L}_{\text{reg}}$, and $\gamma^{k}$. Fig. 9 and Tab. 6 illustrate the effects of these omissions.
Without the coarse-to-fine approach, the model, trained directly at the specular stage, produces incomplete scene geometry, affecting environment map queries for specular color. Omitting normal loss $\mathcal{L}_{\text{normal}}$ removes direct supervision on normals, impeding geometric refinement and reducing rendering quality. This also leads to inaccurate reflection directions and less precise specular colors. Removing normal regularization $\mathcal{L}_{\text{reg}}$ allows the normal residual to dominate normal optimization, resulting in insufficient optimization of the 3D Gaussians’ shortest axis towards the correct normal , which in turn reduces the rendering quality. Without $\gamma^{k}$ in normal regularization, the normal residual decreases for both non-flattened and flat Gaussians. This particularly affects less flat 3D Gaussians whose shortest axis significantly deviates from the surface normal. The insufficient normal residual correction causes these 3D Gaussians’ shortest axes to deviate greatly from their original direction in an attempt to align with the surface normal, ultimately reducing rendering quality.

##### Ablation study on SH, Static environment map, and Deformable enviorment map.

Fig. 10 and Tab. 6 demonstrate the superiority of the deformable environment map over the static environment map, which in turn outperforms Spherical Harmonics (SH). SH struggles to accurately model high-frequency specular colors. While the static environment map can model high-frequency colors, it is best suited for static lighting conditions. In contrast, the deformable environment map excels in modeling time-varying lighting, offering superior performance for dynamic scenes.

## 5 Conclusion

SpectroMotion enhances 3D Gaussian Splatting for dynamic specular scenes by combining specular rendering with deformation fields. Using normal residual correction, coarse-to-fine training, and deformable environment map, it achieves superior accuracy and visual quality in novel view synthesis, outperforming existing methods while maintaining geometric consistency.

##### Limitations.

Though we stabilize the entire scene’s geometry using a coarse-to-fine training strategy, some failure cases still occur. Please refer to the appendix for visual results of failure cases.

## Appendix A Appendix / supplemental material

### A.1 Implementation Details

We use PyTorch as our framework and 3DGS (Kerbl et al., ) as our codebase. Our coarse-to-fine approach is divided into three sequential stages: static, dynamic, and specular. In the static stage, we train the vanilla 3D Gaussian Splatting (3DGS) for 3000 iterations to stabilize the static geometry. The dynamic stage then introduces a deformable Gaussian MLP to model dynamic objects. We first optimize both the canonical Gaussians and the deformable Gaussian MLP for 3000 iterations until the scene becomes relatively stable. Subsequently, we introduce $\mathcal{L}_{\text{normal}}$, enabling simultaneous optimization of the scene’s normal and depth, further refining the geometry for another 3000 iterations. After the dynamic stage concludes, we transition to the specular stage, which involves changing the color representation from complete spherical harmonics to $\mathbf{c}_{\mathbf{final}}$. To mitigate potential geometry disruptions due to the initially incomplete $\mathbf{c}_{\mathbf{final}}$, we fix the deformable Gaussian MLP and all 3D Gaussian attributes except for zero-order SH, specular tint, and roughness, while temporarily suspending densification. After 6000 iterations, once $\mathbf{c}_{\mathbf{final}}$ becomes more complete, we resume optimization of all parameters and reinstate the densification process. Concurrently, during the first 2000 iterations of the specular stage, we optimize only the canonical environment map to learn time-invariant lighting. For the canonical environment, we use $6\times 128\times 128$ learnable parameters. Subsequently, we begin optimizing the deformable reflection MLP to capture time-varying lighting effects. We train total iterations 40,000.
All training and rendering are conducted on an NVIDIA RTX 4090 GPU.

#### A.1.1 Network Architecture of the Deformable Gaussian MLP and Deformable reflection MLP

We use deformable Gaussian MLP to predict each coordinate of 3D Gaussians and time to their corresponding deviations in position, rotation, and scaling. As shown in Fig.  11, the MLP initially processes the input through eight fully connected layers that employ ReLU activations, featuring 256-dimensional hidden layers, and outputs a 256-dimensional feature vector. This vector is then passed through three additional fully connected layers combined with ReLU activation to separately output the offsets over time for position, rotation, and scaling. Notably, similar to NeRF, the feature vector and the input are concatenated in the fourth layer. For the deformable reflection MLP, we utilize the same network architecture, as shown in Fig. 12.

Figure: Figure 11: Architecture of the deformable Gaussian MLP
Refer to caption: x11.png

Figure: Figure 12: Architecture of the deformable reflection MLP
Refer to caption: x12.png

### A.2 Additional Experiment Results

We provide an HTML interface in the supplementary material zip file for browser-rendered video results of all compared methods. This includes qualitative comparisons on the NeRF-DS dataset for each scene, as shown in Fig. 13, as well as qualitative comparisons on the NeRF-DS dataset for each scene with labeled dynamic specular masks, as shown in Fig. 15. Additionally, failure cases are presented in Fig. 16.

Figure: Figure 13: Qualitative comparison on NeRF-DS dataset per-scene.
Refer to caption: x13.png

Figure: Figure 14: Qualitative comparison on the HyperNeRF Park et al. () dataset.
Refer to caption: x14.png

Figure: Figure 15: Qualitative comparison on NeRF-DS dataset per-scene with labeled dynamic specular masks.
Refer to caption: x15.png

Figure: Figure 16: Failure cases. In some dramatic scenes, relying solely on the Deformable Gaussian MLP is insufficient, such as when an arm enters or exits the scene, leading to many floaters occurring.
Refer to caption: x16.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Given my concerns listed in Weaknesses, I'm currently leaning towards the negative side about this submission. Besides, I believe the writeup could be further polished to make it easier to follow, like changing the order of certain sections. For example, $n^t$ is introduced in Section 3.2.1, but its formal computation is not explained until Section 3.3. Moving the explanation of $n^t$ computation earlier could facilitate better comprehension. I look forward to the authors' rebuttal to see if these concerns can be adequately addressed.
soundness: 2
strengths: This work introduces an interesting solution for modeling both dynamic and specular objects, incorporating an improved normal estimation technique alongside normal regularization, which are particularly noteworthy contributions. According to the reported results, this method demonstrates enhanced performance in novel view synthesis across multiple datasets.
summary: This paper presents an extension of Gaussian Splatting to effectively model scenes containing dynamic and specular objects. Building upon the classic 3DGS pipeline, the authors introduce separete MLPs to model the deformation fields of Gaussians and environment maps. A novel technique for normal estimation is also proposed, which ensures improved accuracy in normal estimation, subsequently enhancing the computation of specular colors. The approach involves joint optimization of Gaussian parameters, MLP parameters, and a trainable environment map, organized into distinct training stages to promote efficient convergence. Experimental results demonstrate superior performance on popular datasets across different metrics.
weaknesses: 1. While the method claims to model both specular and dynamic objects, the results presented from NeRF-DS appear relatively static to me, despite the significant camera motion in these data. I would be particularly interested in seeing results that exhibit greater dynamics, similar to those from D-NeRF Synthetic (e.g., Lego and Jumpingjacks) or HyperNeRF (e.g., cut-lemon and expresso).
2. The paper introduces a deformable environment map to account for dynamic lighting; however, the visualizations provided still seem quite static. Although the ablation study demonstrates the effectiveness of this deformable environment map, I would appreciate additional visualizations where the dynamic lighting effect is more obvious, such as scenes where the light source is moving. Additionally, based on my understanding, this formulation does not seem to handle changes in light color, such as a transition from red to blue light. Please correct me if I am mistaken.
3. To validate the accuracy of normal estimation, it would be beneficial to see more examples of rendered normal maps from various scenes. Currently, these are only available in Figures 1 and 2. Some synthetic data (e.g., the Shiny Blender dataset) could also be used to quantatively evaluate the normal estimation accuracy.
4. Given the explicit decomposition of lighting and material in the proposed method, it has potential applications in areas like material editing and relighting. I believe including examples of these applications would further strengthen the claims made in the paper, particularly by illustrating the variations in specular patterns under different lighting conditions. For example, could the roughness of objects be adjusted to make them appear more specular? Additionally, using different environment maps (e.g., indoor and outdoor HDRI maps) could illustrate variations in rendering.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The static stage. Does the pipeline directly optimize a static 3DGS from the input images, even the scene behind the images is dynamic? This will cause significant artifacts in the dynamic parts (in Fig. 8), so why not directly start from the dynamic stage?
2. In Fig. 7, seems like the decomposition between diffuse color and specular color does not work very well.
3. GaussianShader and GS-IR are designed for static scenes, not for dynamic scenes. Is it fair to compare with them?
soundness: 3
strengths: 1. The paper is generally well-written, easy to follow and understand.
2. The proposed deformable environment map and physical normal estimation is reasonable and meaningful.
3. The experimental results show improvements over baselines.
summary: The paper proposes SpectroMotion, a 3-stage pipeline to reconstruct specular dynamic scenes using 3D Gaussian Splatting representation and physically-based rendering. The paper leverages deformable environment map to model the time-varying lighting, and proposes a more physically-correct normal estimation for deformed Gaussians using rotation and residual adjustments. Experimental results show that the proposed method outperforms state-of-the-art baselines.
weaknesses: 1. Complex pipeline. The proposed pipeline consists of 3 stages, which may limit its performance and scalability.
2. The proposed physical normal estimation from 3D Gaussians is good, but recent methods (e.g. 2D Gaussian Splatting) have proposed a more general and intuitive solution for well-defined Gaussian normals. With these more simple and effective representations, I doubt the necessity of designing complex corrections for the ill-defined 3D Gaussian normal estimation problem.
3. Missing ablation study. The physical normal estimation is a key design of the method, but the paper doesn't present an ablation study on the effectiveness of this design.
4. Limited novelty. The major parts of the pipeline, including the 3DGS representation, the deformable MLP, and the physically-based rendering model, are not new in this research field. Despite some task-specific designs, I'm not sure if the novelty and innovation can reach the acceptance level of ICLR.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Please justify the use of deformable environment map based on the doubts I listed in the weakness section.
2. Please explain the PBR process in more detail, specifcally how the integration is done, how the BRDF is estimated, and how this affects the inference time.
3. Please explain the weird performance difference on HyperNeRF dataset.
soundness: 2
strengths: **Motivation**
* This paper correctly identifies the challenging moving specular scene reconstruction using the existing methods. It extends the NeRF-DS problem settings using the latest deformable 3DGS methods to improve the performance.

**Method**
* The proposed method utilizes the PBR to improve the modeling of specular colors under motion. This is very challenging because of the difficulty of accurately reconstructing the surface's normal direction and environment lighting under a monocular dynamic setting. 
* The paper uses surface normal self-consistency and proposes surface normal residual correction algorithms to improve the surface normal prediction. 
* To tackle the environment lighting estimation, this paper proposes a novel dynamic environment mapping method to extend the idea of the deformation field to the reflected direction when querying the environment map.

**Experiment**
* The paper demonstrates a minor improvement over the existing dynamic 3DGS methods and NeRF-based methods designed for dynamic specular objects. Both qualitative and quantitative results support this improvement to a certain extent.
summary: This paper proposes an extension to the existing deformable 3DGS methods to improve the performance for dynamic scenes with moving specular objects. It identifies the limited capabilities of existing dynamic 3DGS on specular objects and introduces a 3-stage training pipeline. First it trains the static part of the scene, and then adds in the dynamic part using deformation MLP. Lastly, it adds in the specular color estimation using the reflected ray direction to query a deformable environment map. To ensure the quality of surface normal estimation, it uses a consistency loss between estimated gs normal with the depth gradient. To ensure surface normal correctness under deformation, it introduces a normal correction term based on the flatness of the Gaussian. To model changing environment lighting, a deformation field for reflection direction is introduced to change the reflected direction when querying the environment map. As a result, the performance on scenes with move specular objects demonstrates a minor quality improvement.
weaknesses: **Method**
* My main concern regarding the proposed method is the theoretical validity of the deformable environment map. 
* First, an important assumption of using an environment map to model a specular surface is that the environment should be far away from the surface. However, the reflective surface present in the datasets is mainly reflecting objects close to it. This assumption is further violated when this specular surface starts moving. This is all because the query of environment mapping considers only direction but not the position of the surface point. So I am not very sure whether the use of an environment map for such scenes is a good choice.
* Second, the deformable environment map lacks theoretical support. The paper justifies it as a model for the changing environment lighting, but the scenes used in the paper are of static environment lighting. Although the ablation study shows a minor improvement in the performance when using this deformable environment map, but it could be because of the following three reasons: 1. compensating the inaccurate surface normal estimated, 2. compensating the relative motion of the object with its surrounding due to the incorrect environment mapping assumption mentioned above, 3. MLP carries more modeling capacity to fit the specular lighting. 
* Third, the paper claims the use of the Cook-Torrance reflectance model for PBR, but this model requires an expensive integration in the hemisphere based on the BRDF. This raises a few issue: 1. the paper seems to miss out how this BRDF is estimated, 2. the paper seems to miss out how this integration is done, 3. the paper did not report the inference time which could be a concern due to this expensive integration. 

**Experiments**
* The performance improvement on the NeRF-DS dataset is not very significant.
* The performance improvement on the NeRF-DS dataset specular region is less significant than the overall improvement, which indicates that the performance improvement on the non-specular regions is contributing more to the overall performance.
* The PSNR for HyperNeRF dataset is lower than the 4DGS by 2dB, but the LPIPS is better. This is not impossible, but some explanation would be helpful.

I believe this paper is a useful extension to the dynamic specular object reconstruction, but the methods introduced are not very well justified with theoretical reasoning. The small performance improvement also decreases the contribution of this paper.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In Table 1, why are the metrics so low for all methods except SpectroMotion? This is inconsistent with the results of Deformable-GS, especially for HyperNeRF and NeRF-DS. Did the authors use different resolutions or evaluation metrics?
2. In the paper, I couldn’t find a clear explanation of how $c_s$ is calculated. It’s unclear whether the authors followed the approach from GaussianShader, where $c_s$ is derived using Eq. 7, or if they adopted the Gaussian-DR method, where the specular color is directly obtained through querying the environment map.
soundness: 4
strengths: - This paper is well-written and easy to understand.
- Using zero-order spherical harmonics (SH) and Cook-Torrance BRDF to model diffuse and specular respectively makes sense.
- Using normal residuals to flatten the Gaussian, thereby improving the accuracy of normal estimation in dynamic scenes, is insightful for the community.
- The visual results have improved on the NeRF-DS and HyperNeRF datasets.
summary: This paper aims to address the modeling of dynamic scenes with specular highlights and reflections. Building on Deformable-GS, the authors introduce the Cook-Torrance BRDF to compute the high-frequency specular components. Additionally, they incorporate techniques such as robust rotation calculation and normal residuals to improve geometric accuracy, which in turn enhances the accuracy of reflective view directions and overall model fitting capability. The experimental results show that SpectroMotion achieves impressive results on NeRF-DS dataset. The robust results on HyperNeRF also demonstrate that the approach can be applied to more general scenarios.
weaknesses: 1. Inappropriate reference:
   - In Line 103, strictly speaking, Gaussian-DR is not a PBR method, because its approach does not incorporate the rendering equation.
   - Missing reference: Relightable 3D Gaussians: Realistic Point Cloud Relighting with BRDF Decomposition and Ray Tracing, by Gao et al., ECCV 2024.
2. This method, in principle, lacks novelty and can be seen as a combination of Deformable-GS and Gaussian-Shader to some extent.
3. The introduction of BRDF calculations and the deformation reflection MLP will further increase the rendering burden, making real-time rendering difficult to achieve.
4. The quantitative comparisons on the HyperNeRF dataset are not convincing. Inaccuracies in HyperNeRF's camera poses lead to slight misalignments between rendered images and the ground truth, as noted in the original HyperNeRF paper. This results in abnormal outcomes for metrics sensitive to minor shifts—such as PSNR, SSIM, and LPIPS—where sharp but misaligned images score lower than blurry ones. In 3DGS, these inaccurate poses further cause an excessive increase in the number of Gaussians, leading to out-of-memory issues. Therefore, the quantitative metrics for HyperNeRF are not very meaningful, and greater emphasis should be placed on visual quality.