Kati8/ICLR_2023_2025 huggingface dataset:
authors: Meng YOU
authors: Zhiyu Zhu
authors: Hui LIU
authors: Junhui Hou
reviews: 
Comments: The paper presents a training-free sampling method that generates image sequences conditioned on input camera trajectories. The input view is warped onto the target views, serving as pseudo-GT variables. To condition the reverse process, the paper re-computes the predicted mean of a reverse step with an interpolation between the predicted mean of the current noisy latent and the warped samples. It further explores calculating the optimal interpolation weight and two guidance methods (replacement and gradient). Both quantitative result and qualitative results show the proposed method can generate smooth, high-fidelity image sequences.
Strengths: 1. The paper presents a training-free method that enables a pre-trained diffusion model to generate image sequences based on given camera trajectories.
2. The qualitative videos demonstrate that the proposed method generates plausible outputs.
Weaknesses: 1. The paper relies on an off-the-shelf depth estimation network to warp the input image(s), which may be prone to scale inconsistencies across frames. Since the reverse process is largely dependent on the warped images, depth estimation errors could propagate to the final outputs.
2. The paper does not explain how occluded regions are handled in depth-warping. Handling the occluded regions can be particularly challenging when the camera pose variation is large, as $\tilde{\mathbf{\mu}}_{t, \mathbf{p}_i}$ in Eq.12 may produce degenerate outputs. 
3. The camera trajectories in most qualitative results are limited to relatively small variations or 360-degree circular poses, where scale ambiguity from the depth estimation network can largely be ignored. This raises robustness of the proposed method when the camera variations are large. The paper could benefit from including more examples with larger camera trajectories.
4. The toy experiment in Fig.2 (a) shows that the $\mathcal{E}_D$ decreases with the diffusion reverse process. This brings another question on how the predicted means converge to the GT image (loss=0) where it is very unlikely to have sampled $\mathbf{X}_t$ that would have led to the desired GT. I believe more details of the experiment can help understanding the paper better.
5. The paper makes some assumptions to compute the optimal interpolation weight $\lambda(t, \mathbf{p}_i)$ in Sec.4.2. However, the paper does not present ablation study on the choice of the weight (except when $\lambda(t, \mathbf{p}_i) = \infty$, which also shows comparable results). To validate the choice of the weight schedule, the paper could present comparisons to other naive techniques (e.g., linear, constant, exponential). Additionally, the interpolation weight depends on a set of hyperparameters $\{ v_1, v_2, v_3 \}$ which may require engineering effort to tune on new scenes. 
6. While the paper shows promising quantitative and qualitative results, the number of scenes used for evaluation is insufficient to validate the method's effectiveness and differs from previous state-of-the-art methods.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 2
Clarity: 2
reviews: 
Comments: The authors propose a training free novel view synthesis paradigm based on video diffusion model. Specificcally, warped depth maps are utilized and certain sampling methods are proposed to ensure high quality NVS.
Strengths: 1. The work is complete and results seem to be good quantitively.
Weaknesses: 1. The work is complete, but the novelty and the contribution are not strong enough to meet the quality of ICLR. A good training-free is not that appealing after previous works like MotionCtrl, and it highly relys on the capability of SVD. It is hard to know about it generalizability around various video generation methods. 
2. It is recommended to show "Directly guided sampling" and "Posterior sampling" clearly in the view of practical implementation. Some illustration would be appreciated.
3. Some equations should be displayed more clearly. For instance in eq.8, I(P_0) should not be related to the pose index i, but in eq.9 it is related to i. Maybe in eq.9 the I(P_0) should be revised as I(P_i)?
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: 1. This paper proposes a novel view synthesis pipeline without any training. The pipeline can take single or multiple views of static scenes or monocular videos of dynamic scenes as input.
2. This paper modulates the score function with the warped input views to control the video diffusion process and generate visually pleasing results. They achieve the modulation in an adaptive fashion based on the view pose and the number of diffusion steps.
3. They conduct extensive results on both static and dynamic scenes and show promising results with both evaluation numbers and visualizations.
Strengths: 1. The proposed adaptive modulation of the score function in the diffusion process is novel.
2. The proposed method achieves better results in various scenarios compared to baselines.
3. The authors provide the code with an anonymous link, ensuring the applicability of the results.
Weaknesses: My primary concerns are with the references and experimental details:

1. Some key references on diffusion-based NVS are missing [1,2,3,4,5,6,7]. Among these, [3] specifically focuses on scenes and has released its code. Is there a particular reason it was not included in the comparison?
2. How is the synthesized view pose calculated in this paper? In Line 364, it states that 'current depth estimation algorithms struggle to derive absolute depth from a single view or monocular video, resulting in a scale gap between the synthesized and ground truth images.' When calculating pose error, does the proposed method account for this scale gap?

**Minor Points:**

1. In Table 1, it is stated that MotionCtrl [Wang et al., 2023b] and 3D-aware [Xiang et al., 2023] do not require training. However, as I understand, they do require fine-tuning.

[1] Wu, Rundi, et al. "Reconfusion: 3d reconstruction with diffusion priors." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2024.

[2] Watson, Daniel, et al. "Novel View Synthesis with Diffusion Models." *The Eleventh International Conference on Learning Representations*.

[3] Yu, Jason J., et al. "Long-term photometric consistent novel view synthesis with diffusion models." *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2023.

[4] Cai, Shengqu, et al. "Diffdreamer: Towards consistent unsupervised single-view scene extrapolation with conditional diffusion models." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[5] Tseng, Hung-Yu, et al. "Consistent view synthesis with pose-guided diffusion models." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2023.

[6] Chan, Eric R., et al. "Generative novel view synthesis with 3d-aware diffusion models." *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2023.
Recommendation: 6
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: This work introduces a diffusion model-based approach to achieve novel view synthesis. In particular, it leverages the depth-warped views as guidance to achieve adaptative modulation. Experiments on single-view images, multi-view images, and monocular video input-based novel view synthesis showcase the efficacy of the introduced methods.
Strengths: * The idea of using depth-warped images as guidance for novel view synthesis is reasonable.
* It is interesting to see that the temporal consistent video diffusion model can be effectively reformulated to achieve geometrical consistent NVS in a training-free manner.
* Experiments on several challenging settings, including 360-degree NVS from a single view, verify the significance of the introduced method.
Weaknesses: * Accessing the geometry accuracy. For the 360-degree case, e.g., the truck, it would be better to apply mesh reconstruction on the rendered views, similar to Fig. 5(b) in latentSplat [Wewer et al. ECCV 2024]. The reconstructed mesh will provide a clearer understanding of how well the rendered views maintain correct geometry.

* Pixel-aligned metrics. For the NVS task, it would be better to report comparisons with state-of-the-part methods regarding pixel-aligned metrics, e.g., PSNR and SSIM. 

* Discussion with feed-forward 3DGS models. It might be interesting to see comparisons with detailed analysis between the introduced methods and those feed-forward 3DGS models, e.g., pixelSplat [Charatan et al., CVPR 2024], MVSplat [Chen et al., ECCV 2024]. And it would be better to consider adding these methods to the related work for better coverage of recent NVS works.
Recommendation: 6
Reviewer confidence: 2
Soundness/correctness: 3
Clarity: 2