Title: Fin3R: Fine-tuning Feed-forward 3D Reconstruction Models via Monocular Knowledge Distillation
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Reconsturction

Knowledge Distillation

Fine-Tuning
Abstract: We present Fin3R, a simple, effective, and general fine-tuning method for feed-forward 3D reconstruction models. The family of feed-forward reconstruction model regresses pointmap of all input images to a reference frame coordinate system, along with other auxiliary outputs, in a single forward pass. However, we find that current models struggle with fine geometry and robustness due to (\textit{i}) the scarcity of high-fidelity depth and pose supervision and (\textit{ii}) the inherent geometric misalignment from multi-view pointmap regression. Fin3R jointly tackles two issues with an extra lightweight fine-tuning step. We freeze the decoder, which handles view matching, and fine-tune only the image encoder—the component dedicated to feature extraction. The encoder is enriched with fine geometric details distilled from a strong monocular teacher model on large, unlabeled datasets, using a custom, lightweight LoRA adapter. We validate our method on a wide range of models, including DUSt3R, MASt3R, CUT3R, and VGGT. The fine-tuned models consistently deliver sharper boundaries, recover complex structures, and achieve higher geometric accuracy in both single- and multi-view settings, while adding only the tiny LoRA weights, which leave test-time memory and latency virtually unchanged. Project page: \href{http://visual-ai.github.io/fin3r}{https://visual-ai.github.io/fin3r}


Peer reviews:
comments: SUMMARY:
This paper proposed a new fine-tuning strategy to improve the performance of the current feed-forward reconstruction models. The authors involve MoGe as the teacher network and finetune the model on SA-1B dataset. Besides, they also involve the LoRA model, and meanwhile rescale the norm of the feature to satisfy the requirements of the frozen decoders.

STRENGTHS AND WEAKNESSES:
**Strengths**
1. The authors made a thorough analysis of the limitations of the current feed-forward reconstruction methods.
2. The authors proposed a new fine-tuning strategy.

**Weaknesses**
1. Instead of distilling the knowledge from MoGe, why not directly integrate it into the proposed model? Since in Table 1, none of the finetuned method has superior performance than MoGe.
2. In Line 132-133, the authors claimed, 'the limitations in detail recovery primarily originate from the encoder'. However, I did not see any theoretical or empirical evidence to support the statement. Finetuning the decoder with a frozen encoder could also improve the performance. 
3. What motivates the authors to integrate LoRA instead of others?
4. The improvement in Figure 5 and 6 is tiny, in the 6th line of Figure 4, it seems such finetune degrade the original performance.
5. In Table 3, why the proposed method improved the static scene reconstruction methods, but somehow make the dynamic scene reconstruction method worse?
6. How about finetuning the encoder and decoder on the new dataset without any proposed so-called strategy?

QUESTIONS:
Please refer to the weaknesses.

LIMITATIONS:
Please refer to weaknesses.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 4
DATE: 2025-06-25

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an encoder-only fine-tuning method for the *3R family, where monocular depth information from MoGe (single-view DUST3R) is distilled into the backbone encoder feature. When fine-tuning, they apply re-normalization LoRA on the encoder to address the over-smoothed geometry caused by multi-view pointmap regression.\n\nThe goal of this approach is to produce sharper and more robust single-view depth, while performing comparable multi-view reconstruction.\n\nThey compare their method to the backbones in *3R family, such as DUSt3R, CUT3R, and VGGT, evaluating on benchmarks for monocular depth, pose estimation and 3D reconstruction.\n\nSTRENGTHS AND WEAKNESSES:\n[Strengths]\n\nThis work is insightful, offering valuable insights for fine-tuning the *3R family to adapt them for producing fine-grained details. Instead of fine-tuning the decoder or using noisy depth labels, they chose to apply a re-normalized LoRA to fine-tune the encoder using MoGe. This approach can be generalizable to subsequent *3R methods. \n\n[Major Weaknesses]\n\n1) Missing video (multi-view) depth evaluation: The goal of the paper is improving monocular depth while preserving multi-view consistency. While the method is evaluated on pose estimation and 3D reconstruction, improved pose can lead to better 3D reconstruction. It is very important to evaluate multi-view depth, because it can help better factor out the impact of this fine-tuning method on multi-view consistency. And these datasets (e.g., DTU, ScanNet, KITTI, ETH3D, T&T, Bonn, TUM Dyn) are also widely used in previous work like CUT3R/VGGT.\n\n[Minor Weaknesses]\n\n1) Monocular depth distillation in the encoder may harm inherent multi-view consistency in feature space except increasing feature norms. It could be verified whether the features after adaption still enable fine-grained, multi-view consistent reconstruction using Feat2GS benchmark. There seems to be an overclaim in L132-135.\n\n2) CUT3R is not only a feed-forward 3D reconstruction model but also a 4D foundation model. It would be interesting if the authors could show the proposed adaptation’s performance on 4D fine details reconstruction.\n\n3) L108-111: The motivation for 'Scale Uncertainty' paragraph is unclear. There’s a common issue in *3R models—incorrect depth or scale leads to multi-layered walls in reconstruction. It would be great to demonstrate this improvement by providing some visual examples where the baseline predicts layered walls, but this proposed method predicts a single layer wall, \n\n4) L116-117 lacks evidence: It would be helpful to cite VGGT(in the Table 3 of the original paper), as it supports this point.\n\n5) Lack of discussion on LORA3D and TEST3R. Why not use Visual Prompt Tuning insdead of LORA?\n\nQUESTIONS:\nRefer to [Major Weaknesses] 1) and [Minor Weaknesses] 1) 2)\nFor now, I am not yet convinced by the performance of the method due to the mentioned points around evaluation. However, I would be willing to raise my score if the authors address the concerns raised in my review and provide clarification on the points mentioned above.\n\nLIMITATIONS:\nNo, there is a lack of discussion on the limitations of monocular depth distillation's impact on the inherent multi-view consistency of the backbone feature.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors fine-tune 3D estimators (outputting point clouds) that are trained on videos (using structure from motion as a means to get GT/surrogate 3D) with pre-trained single-view estimators that have higher detail fidelity. First attempts that did not work, due to feature shift, are reported, showing the difficulty of the task. The main contribution is to prevent this shift with only training the encoder and with weight normalization.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n* An effective method to prevent drift by normalizing weights after an update.  \n* Keeping the same network architecture and even pre-trained decoder keeps inference efficient and provides a drop-in replacement\n* Section 4.1 Monocular Depth Estimation shows clear and consistent improvement across several backbones\n\nWeaknesses:\n* Results for MultiView Pose Estimation on RealEstate10k are a bit mixed, fine tuning sometimes makes things worse\n* Results in Table 5 are also a bit mixed. As no std across multiple runs are reported, it is hard to judge how significant the improvements are. Could you explain when the method degrades?\n* Figure 6 is only for VGGT, it would be nice to see side views for all methods. In general, depth maps are hard to read, in particular if the scale is not normalized as in Figure 7. It would be much better to show 3D side views for analyzing depth map quality.\n\n* In Table 7, ablation, I am missing an experiment on fine-tuning just the encoder vs. the whole model (please explicitly state whether any of the existing models are fine-tuning all). Also, some of the initial exploratory experiments were (unless I am mistaken) done on fine-tuning the whole model. Does this also hold when only fine-tuning the encoder?\n\nSummary:\nA simple yet effective method. My main concern is the significance of the improvements on real data for which repeated experiments with std reported would help.\n\nQUESTIONS:\n- Line 168 \" we use 2-view data with distillation supervision applied exclusively to the view1 pointmap head for distillation loss\" - why not also to the second? Is it a good thing to only use one? Would drift be prevented if two are used?\n- Line 218 \" The only exception is CUT3R—likely due to its training includes the RealEstate10k dataset\" What does this dataset bring to the table that prevents improvement?\n\n- Line 171 \"During each epoch, we sample 20,000 images from SA-1B [23], 1,000 from Hypersim [40], and 1,000 from TartainAir [61]\" I wonder how this differs from the training of baselines and whether this could also have an effect on the improvements?\n- Was there a validation set used to determine stopping, would there be overfitting if training for more than 10 epochs?\n\nLIMITATIONS:\n* Limitations are not clearly stated. In particular, given the mixed results on real images, it would be important to discuss and exemplify where things are getting worse.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Fin3R, a lightweight and effective fine-tuning method aimed at improving feed-forward 3D reconstruction networks such as DUSt3R, CUT3R, and VGGT. These models predict 3D pointmaps of input images in a single pass but frequently struggle with capturing fine geometric details due to challenges like limited high-fidelity supervision and geometric misalignments across views. Fin3R addresses these issues by fine-tuning only the image encoder, while freezing the decoder of the model. By introducing a monocular teacher model and a custom LoRA adapter, Fin3R distills high-quality geometry from large unlabeled data, enhancing the encoder's ability to capture complex structures and sharp boundaries. Experiments on the DUSt3R, CUT3R, and VGGT validate that Fin3R consistently improves geometry prediction in both single-view and multi-view settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThis paper is well-written and easy to follow.\n\n2.\tThe improvement across several tasks with different datasets is promising, especially considering that the proposed method only requires fine-tuning the target model with a small amount of unseen data.\n\nWeaknesses:\n\n1.\tThe paper lacks experiments on dense multi-view stereo (MVS) estimation using the high-quality DTU dataset, which could effectively reflect both pose and depth improvements. Additionally, according to Sec. 3.3, only monocular distillation loss and pointmap loss are used for fine-tuning. It is unclear why the proposed method improves relative pose estimation for models like VGGT and CUT3R, which directly regress camera parameters.\n\n2.\tThere is insufficient discussion and comparison with similar methods that also use LoRA to fine-tune feed-forward 3D reconstruction networks, such as LoRA3D (ICLR 2025).\n\n3.\tCan the proposed method be used to improve Mast3R, which uses an architecture similar to DUSt3R but regresses metric depth?\n\n4.\tConfidence maps are widely used in feed-forward 3D reconstruction methods to filter unreliable predictions. It would be interesting to discuss whether the proposed method can improve the accuracy of pointmaps or depths while increasing confidence in predictions.\n\n6.\tCould the use of larger-scale datasets and more diverse scenes further enhance the fine-tuning results? Additionally, what was the rationale behind selecting the SA-1B dataset for fine-tuning? What specific criteria informed this decision?\n\nQUESTIONS:\n1. What are the criteria for selecting training data for fine-tuning?\n\n2. Please provide more results on large-scale outdoor scenes, such as the Tank and Temples dataset.\n\n3. Can the proposed method be used to fine-tune models for metric depth/pointmap prediction, such as Mast3R?\n\nLIMITATIONS:\nAlthough the method proposed in this paper improves the performance of most models, it negatively impacts some models that already perform well, which is often undesirable in practical applications.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}