Multiview Equivariance Improves 3D Correspondence Understanding with Minimal Feature Finetuning
Authors: Yang You
Yixin Li
Congyue Deng
Yue Wang
Leonidas Guibas
Venue: ICLR 2025 Poster (2025)
Keywords: Vision Foundation Models; 3D Representation Learning; Fine-tuning; 3D Equivariance
Primary area: foundation or frontier models, including LLMs
TL;DR: This work evaluates and improves the 3D awareness of ViT-based models by enhancing their 3D equivariance through a simple finetuning strategy using 3D correspondences.

Abstract: Vision foundation models, particularly the ViT family, have revolutionized image understanding by providing rich semantic features. However, despite their success in 2D comprehension, their abilities on grasping 3D spatial relationships are still unclear.
In this work, we evaluate and enhance the 3D awareness of ViT-based models. We begin by systematically assessing their ability to learn 3D equivariant features, specifically examining the consistency of semantic embeddings across different viewpoints. Our findings indicate that improved 3D equivariance leads to better performance on various downstream tasks, including pose estimation, tracking, and semantic transfer. Building on this insight, we propose a simple yet effective finetuning strategy based on 3D correspondences, which significantly enhances the 3D understanding of existing vision models. Remarkably, even finetuning on a single object for just one iteration results in substantial performance gains. Code is available on https://github.com/qq456cvb/3DCorrEnhance.

Published as a conference paper at ICLR 2025
MULTIVIEW EQUIVARIANCE IMPROVES 3D CORRE- SPONDENCE UNDERSTANDING WITH MINIMAL FEA- TURE FINETUNING
Yang You', Yixin Li', Congyue Deng', Yue Wang", Leonidas Guibas P Department of Computer Science, Stanford University, U.S.A.
Department of Computer Science, University of Southern California, U.S.A.
K guibas @ aee.ssstie www.sstaforr.ee.e edu
ht tps::p///o githbb. com http:////5oobb///rrreoeeeo http://hhw....cooooo////////nnaa
Prke Fdt
Pretrained VIT
Feature equivariance finetuning with one synthetic object
Setural
BUTINN
Finetuned
WORLS
POO41
Figure 1: Improving 3D correspondence understanding through finetuning on feature equiv- ariance. Left: finetuning feature equivariance one synthetic object can already enhance the vi- sion transformer's ability to generate better 3D feature correspondences on general objects. Right: This improvement further leads to superior performance across multiple 3D tasks, including pose estimation, video tracking, and semantic correspondence.
ABSTRACT
Vision foundation models, partieularly the ViT family, have revolutionized image understanding by providing rich semantic features. However, despite their success in 2D comprehension. their abilities on grasping 3D spatial relationships are still unclear. In this work, we evaluate and enhance the 3D awareness of ViT-based models. We begin by systematically assessing their ability learn 3D equivariant features, specifically examining the consistency of semantic embeddings across different viewpoints. Our findings indicate that improved 3D equivariance leads to better performance on various downstream tasks, including pose estimation, tracking, and semantic transfer. Building on this insight, we propose a simple yet effective finetuning strategy based on 3D correspondences, which significantly en- hances the 3D correspondence understanding cxisting vision models, Remark- ably, finetuning single object for one iteration results in substantial gains.
INTRODUCTION
Common camera imaging systems struggle to depict the 3D world due to the limitation of capturing only a single perspective at any given moment. In contrast, human perceptual capabilities exhibit remarkable trait known view equivariance Köhler (1967); Koffka (2013); Wilson & Farah (2003), allowing us to robustly understand 3D spatial relationships, as seen in tasks ranging from basic object recognition Vetter al. (1995); DiCarlo & Cox (2007) more complex processes like mental rotation and simulation Stewart et al. (2022).
Current large vision models, however, are primarily trained on 2D images, owing to the ease of data acquisition and annotation in 2D. Consequently, their performance is typically evaluated on 2D tasks

---

Published as a conference paper at ICLR 2025
Amir et al. (2021); Hedlin et al. (2023); Tang et al. (2023); Zhang et al. (2023). This raises critical questions: To what extent do these models possess an inherent awareness of 3D structures? How does this awareness impact their performance image-based 3D vision tasks? And, can we further enhance the 3D awareness of these vision foundation models?
Many image-based 3D scene understanding and content generation tasks depend heavily on large 2D vision models, underscoring the importance of investigating these questions. Existing works have begun to explore this area in task-specific contexts. For example, DietNeRF Jain et al. (2021) finds that CLIP Radford et al. (2021) demonstrates higher feature similarities between views from the same scene than from different scenes, which aids 3D reconstruction. LeRF Kerr et al. (2023)
shows that regularizing CLIP with DINO Caron al. (2021) features improves 3D feature distilla- tion from multiple views. However, these studies are tied to specific tasks such as feature distillation.
El Banani et al. (2024) probes the multi-view consistency of ViTs on the NAVI Jampani et al. (2023) and ScanNet Dai et al. (2017) datasets. However, the limited size of these datasets makes it chal- lenging to draw comprehensive conclusions.
To address the first question, how well do vision modely understand 3D structures, we present comprehensive study of the 3D) awareness of large 2D vision models. Specifically, we investigate the view equivariance of latent features-iee., the consistency of multi-view 2D image features rep- resenting the same 3D point across different views. Using off-the-shelf multiview correspondences rendered from Objaverse Deitke al. (2023) (synthetic) and MVImgNet Yu et al. (2023) (real- world), we find that current large vision models do exhibit some degree of view-consistent feature generation, with DINOv2 demonstrating the strongest performance.
To answer the second question, how does this awareness influence performance in image-based 3D vision tasks, we find that the quality of 3D equivariance is strongly correlated with performance on three downstream tasks requiring 3D correspondence understanding: pose estimation, video track- ing, and semantic correspondence. Consistent with previous findings Omek et al. (2023); Tumanyan et al. (2024); Zhang et al. (2023), DINOV2 Oquab et al. (2023) excels in these tasks.
Finally, to address the third question, can we improve the 3D awareness of vision foundation models, we propose a simple yet effective method to enhance the view equivariance of 2D foundation mod- els. thereby significantly improving their 3D correspondence understanding. During training, we randomly select two different views of the same object from Objaverse and sample corresponding pixels. We apply the SmoothAP Brown et al. (2020) loss to enforce feature similarity between these corresponding pixels. This finetuning process, requiring only 10K iterations with LoRA and an ad- ditional convolutional layer of a Vision Transformer (ViT), significantly improves the performance of all tested models on 3D tasks. For instance, DINOv2 gains improvements of 9.58 (3em-3deg in pose estimation), 5.0 (Average Jaccard in tracking), and 5.06 (PCK @0.05 in semantic correspon- dence). Surprisingly. even finetuning one iteration yields notable gains in 3D correspondence understanding. In such cases, DINOv2's performance improves by 4.85, 3.55, and 3.47 for 3em-3deg (pose estimation), Average Jaccard (tracking), and PCK @0.05 (semantic correspondence), respectively.
single multi-view pair sampled from one object for just
To summarize, our key contributions are: (i) We conduct a comprehensive evaluation of 3D equivari- ance capabilities in 2D vision foundation models. (ii) We demonstrate that the quality of 3D equiv- ariance is closely tied to performance on three downstream tasks that require 3D correspondence understanding: pose estimation, video tracking, and semantic correspondence. (iii) We propose simple but effective finetuning method that improves the 3D correspondence understanding of 2D foundation models, leading marked performance gains across all evaluated tasks.
2 EVALUATION OF MULTIVIEW FEATURE EQUIVARIANCE
To assess how effectively current vision transformers capture 3D correspondence understanding, we introduce a 3D equivariance evaluation benchmark focused on the quality of correspondences between 2D points across different views for the same object. Additionally, we present three well- established application tasks that rely on 3D correspondence. demonstrating a strong correlation between the quality of 3D equivariance and downstream task performance. We evaluate five state- oh-the-art vision transformers: DINOv2 Oquab et al. (2023), DINOv2-Reg Darcet et al. (2023), MAE He et al. (2022a). CLIP Radford et al. (2021) and DeiT Touvron et al. (2022), extracting their final-layer features with L2 normalization. For DINOv2, we use the base model; results for other variants provided in the supplementary material.
2

---

Published as a conference paper at ICLR 2025
To evaluate 3D equivariance, utilize rendered or annotated multiview correspondences from Objaverse Deitke et al. (2023) and MVImgNet Yu et al. (2023), covering both synthetic and real images. For Objaverse, we randomly select 1,000 objects from the Objaverse repository, rendered across 42 uniformly distributed camera views, producing 42,000 images. Dense correspondences computed for each object across every unique ordered pair of views, resulting in 1.8 billion correspondence pairs for evaluation. Similarly, 1,000 objects are randomly drawn from MVImgNet, yielding 33.3 million annotated correspondence pairs for evaluation. Since MVImgNet employs COLMAP to reconstruct 3D points, it provides sparser correspondences compared to Objaverse.
Ref. View
View 1
View 2
Imput
DINOv2
DINOv2-Reg
MAE
CLIP
DEIT
Figure 2: Feature visualizations of different models. The sample image rendered from Obje- verse. Colors are computed from the high-dimensional features using PCA. We can see that MAE struggles to distinguish different parts of the content (e.g (e.g.similar features between head and body). Both CLIP and DeiT produce inconsistent features for the chest region between View and View 2. DINOv2 gives the best correspondence.
Metric and Results We propose the Average Pixel Error (APE), a metric that quantifies the av- erage distance between predicted and ground-truth pixel correspondences, normalized by the length of the shortest image edge. The predicted correspondence is determined by identifying the nearest neighbor in the second view, given a reference point feature in the first view. APE for Objaverse is shown in Figure 3, where APE is plotted on the x-axis, meaning lower values (towards the left) indicate better performance. APE and PCDP for MVImgNet are plotted on Figure 5's y-axis with hollow circle O and striped bar IZ representing the evaluted pretrained models (fine-tuning results will be discussed later). Percentage of Correct Dense Points (PCDP) is a metric designed to evaluate dense correspondences, similar to Percentage of Correct Keypoints& (PCK). It is reported at various thresholds (5%, 10%, and 20% of the shortest image edge). We can see that DINOv2 and its registered version outperform other vision transformers, highlighting DINOv2's superior capabil- ity for 3D equivariance. In Figure 2, we provide feature visualizations using PCA, where DINOv2 again demonstrates the best multiview feature consistency.
2.1 FEATURE EQUIVARIANCE CORRELATES TO CERTAIN TASK PERFORMANCES
3D Equivariance itself is not interesting unless it can be used. Below, we will talk about three mature downstream applications that require 3D equivariance capability, and show a correlation between the quality of 3D equivariance and the downstream applications.
2.1.1 TASK DEFINITIONS
One-Shot Object Pose Estimation In one-shot pose estimation, we assume access to video sequence or 3D mesh the target object and aim to estimate its pose in arbitrary environments. During onboarding, a database. At inference, we compute correspondences between the input image and the stored features to match 2D keypoints the image to their 3D counterparts. Pose estimation from these 2D-3D correspondences is achieved using RANSAC Fischler & Bolles (1981) PnP (Perspective- n-Point). Points are uniformly sampled using stratified sampling (stride 4) on 512 512 resized images. RANSAC PnP runs for 10,000 iterations with a threshold of 8.
store dense 2D image features from all rendered
annotated views in
He et al. (2022b) includes 40 low-textured household items captured in two videos: for reference and
We evaluate on the Texxxure Texture and VCB-Video datasets.
3

---

Published as a conference paper at ICLR 2025
1-shot pose estimation (OnePose-LowTex)
1-shot pose estimation (YCB-Video)
70-
lcm-ldeg
Posa accuracy 1%)
3cm-3deg
Average Rocall (AR)
DINOv2-reg
5cm 5deg
50-
DINOv2-reg
40-
DINOv2
30-
20-
DelT
CLIP
CLIP
10
MAE
10.4
19.6
10.8
20.0
20.2
20.4
20.6
10.2
19.4
20.0 20.2
20.4
20.6
Festure invariance (APE)
Fecture invarience (APE)
Video tracking (TAP-VID-DAVIS)
Semantic correspondence (PF-PASCAL)
Keypain matching accuracy (%)
Average Jaccerd Incex (AI)
40
DINOV
diff
view
DINOv2
PCK0.05
PCKO.05
DINOv2-reg
NON22-e
PCK0.10
80
PCK0.15
PCK0.15
34-
MAE
40
DefT
CLIP
MAE
20
DelT
CLIP
20
19:2
19.b 19.8
Feature invariance (APE)
Feature invariance (APF)
Figure Correlation between multiview feature equivariance and the task performances. Along the horizontal axis, lower APE indicates better feature equivariance, while the vertical axis reflects higher task performance across all four plots. The data points align roughly along the diag- onal from the top left to the bottom right, suggesting strong correlation between improved feature equivariance and better task performance.
one for testing, simulating a one-shot scenario. We evaluate on every 10 frames in the video. Fol- lowing He et al. (2022b), pose accuracy is evaluated using lem-ldeg, 3em-3deg, and 5cm-5deg thresholds. The YCB-Vides dataset Xiang et al. (2017) comprises 21 objects and 92 RGB-D video sequences with pose annotations and CAD models for one-shot generalization. A database is created by rendering objects from 96 icospherical viewpoints. We report Average Recall (AR) for Visible Surface Discrepancy (VSD), Maximum Symmetry-Award Surface Distance (MSSD), and Maximum Symmetry-Ave Aware Projection Distance (MSPD) following Hodañ et al. (2020).
Video Tracking For video tracking, given the reference frame, we identify corresponding points in other frames by computing cosine similarities between the dense features of the target object. To improve robustness and accuracy, we follow the process in DINO-Tracker Tumanyan et (2024), which applies softmax operation within the neighborhood of the location with highest similarity.
We evaluate the models on the TAP-Vid-DAVIS Doersch et al. (2022) dataset, benchmark designed for testing video tracking in complex, real- worldworld scenarios. Performance is measured using com- monly applied metrics Tumanyan et al. (2024), including the Average Jaccard Index (AJ), Position Accuracy (8%..)) and Occlusion Accuracy (OA).
Semantic Correspondence In the semantic correspondence task, utilize feature correspon- dences to establish precise keypoint matches between images captured from different instances from the same category. Following the method in Zhang et al. (2023), for given reference keypoint, identify the best match by selecting the location with the highest cosine feature similarity.
We use the PF-PASCAL Ham et al. (2017) dataset as our evaluation benchmark. This dataset typ- ically consists of image pairs taken from the same viewpoint, but additionally report the result by shuffling the image pairs to include different viewpoints, thereby increasing the challenge. We follow standard practice use PCK@0.05, PCK @0.10 10. and PCK @0.15 as evaluation metrics.
The pipelines for all three tasks are illustrated in the figures provided in the supplementary material.
2.1.2 ON THE CHOICE OF THREE TASKS
Correspondence estimation is a fundamental component of 3D vision understanding, underlying key tasks such as epipolar geometry, stereo vision for 3D reconstruction, and optical flow tracking
4

---

Published as a conference paper at ICLR 2025
Pase Estimation
Tracking
Semantic Transfer
Figure 4: Illustration of different types of correspondence tasks evaluated in work.
describe the motion of perceived 3D world. Stereo cameras, and even human perception, rely on disparity maps- effectively correspondences between projected 3D parts understand depth and spatial relationships.
The three tasks we evaluated-pose estimation, video tracking, and semantic correspondence intentionally selected cover diverse aspects of correspondence estimation, ranging from simpler to more complex scenarios: 1. Pose Estimation examines correspondences within the same in- stance under rigid transformations (3): 2. Video Tracking extends this correspondences for the same instance under potential non-rigid or articulated transformations, such as humans or ani- mals in motion; 3. Semantic Correspondence requires correspondences across different instances with similar semantics, often under arbitrary viewpoint changes. An qualitative illustration of these correspondence types is shown in Figure 4.
2.1.3 RESULTS AND FINDINGS
Quantitative results are presented in Figure 3, where the y-axis in each graph shows the performance of the vision models. DINOv2 consistently outperforms all other models across all three tasks, in alignment with the rankings for 3D equivariance on the x-axis. There a clear correlation between the quality of 3D equivariance and performance the downstream tasks: methods with lower APE tend to perform better across all tasks, clustering towards the top-left of the graphs.
FEATURE FINETUNING WITH MULTIVIEW EQUIVARIANCE
Given the correlation between the multíview equivariance of network features and task perfor- mances, we naturally come up with question: Can we finetune the networks on feature equiv- ariance to improve their 3D correspondence understanding and achieve better task performances?
Finetuning method The high-level intuition of improving the multiview equivariance of the net- work features is to enforce the similarity between features of corresponding pixels in 3D) space. We experiment with multiple strategies including different training objectives and network architectures.
For the training loss, rather than employing conventional contrastive loss, opted for the SmoothAP Brown et al. (2020) loss, which demonstrated superior performance. While contrastive loss help align the features of corresponding pixels, it relies predefined fixed margin for positive and negative samples, which ad hoc and often suboptimal. In contrast, SmoothAP opti- mizes ranking loss directly, leading to improved average precision for feature retrieval between corresponding pixels. We also experimented with the differentiable Procrustes alignment loss Li al. (2022), but did not outperform. Detailed ablation results are given in Section 4.3.
In terms of architecture, we apply LoRA in the last four blocks to finetune large foundation models, we introduced a single convolutional layer with a kernel size of 3 and a stride of 1. The motiva- tion behind this addition is rooted in the observation that ViT-family models process image tokens much lower-resolution feature maps (e.g.. 14x smaller in DINOv2). The
as patches, resulting standard approach to obtain high-resolution per-pixel features is to apply linear interpolation. Con- sequently, it is beneficial to explicitly exchange information between neighboring patches before interpolation to achieve more accurate results. More ablation results are given Section 4.1.
During training, randomly select views of the same object from 10K subset of Objaverse at each iteration and sample corresponding pixels. The model is trained for 10K iterations using the AdamW W optimizer with learning rate of le-5 and weight decay of le-4. In the supplementary, show that our finetuning method is robust to the choice of leamning rate.
3.1 IMPROVED FEATURE EQUIVARIANCE WITH GENERALIZATION
Figure 5 illustrates the performance of various models before and after finetuning. After fine- tuning on Objaverse, all models show improved 3D equivariance both Objaverse (synthetic) and
5

---

Published as a conference paper at ICLR 2025
Original Finetuned
Feature inverience on MVimntte (ARE)
CLIP
80-
PCDP on Mylingtles (N)
CLIP
DelT
MAE
DINOv2 DINOVZ
Original model Finetuned
DeiT
DINOv2 reg
MAE
PCDPO.10
PCDPO.
PCDPO
PCDPO.20
18.0
18.5 19.0
19.5
20.0 20.5
C1N0c2-wg
Festure rnveriance Objaverse (APE)
Objaverse
Figure 5: Generalization from synthetic images (Objaverse) to real images (MVImg.Net), Left: Data points roughly around the diagonal from the bottom left to the upper right indicate the correla- tion between the APE tested on the two datasets. The next to the model name means it is finetuned. All finetuning done on Objaverse with only synthetic data. Right: Finetuned on Objaverse, the feature equivariance of the model (measured in PCDP) improves on MVImgNet.
Irput Images
Before fimettriing
Figure 6: Feature visualization of DINOv2 before and after finetuning on MVImgNet objects (left two) and TAP-VID-DAVIS scenes (right one). For each example, we select three different views. The first column provides a reference color produced by PCA, while the second and third columns show the predicted feature correspondences. Our finetuned model demonstrates reduced noise and smoother feature boundaries, particularly noticeable in the reduction of jagged edges.
MVImgNet (real-world). This demonstrates the capacity of vision foundation models to perform sim-to-real transfer, finetuning synthetic Objaverse objects results in enhanced performance on the real-world MVImgNet dataset. Additionally, the performance on the two datasets corre- lated, with data points roughly aligning along the diagonal, indicating that improvements in synthetic environments translate well to real-world settings. DINOv2 stands out as the best model. We also compare the feature visualizations before and after finetuning in Figure 6, from which we can that after finetuning the model produces more consistent features with less noise.
3.2 IMPROVED TASK PERFORMANCES
One-shot Object Pose Estimation Figure shows the performance of pose estimation on the OnePose-LowIH Tex and YCB- Video datasets before and after fine-tuning. As illustrated, all Vision Transformers (ViTs) exhibit noticeable improvements after being fine-tuned on synthetic Objaverse data. For instance, the best-performing model, DINOv2-Reg, improves by 3.46, 6.67, and 6.92 for the fem-ldege 3cm-3deg, and 5em-5deg thresholds, respectively. Additionally, models that performed weaker before fine-tuning show larger gains. For example, DeiT improves by 4.65, 16.39, and 17.76. Similar trends are observed for the YCB- Video dataset, where models like MAF, initially the weakest, show substantial improvement after fine-tuning.
Video Tracking Similarly, the video tracking task, we observe consistent improvements across all ViTs after fine-tuning, as shown in Figure 8. The top-performing model, DINOv2, achieves improvements of 6.45, 5.73, and 2.69 AJ, 8% and OA, respectively.
Semantic Correspondence In the semantic correspondence task, shown in Figure 9, DINOv2 exhibits improvements of 5.06, 3.86, and 1.98 for PCK @0.05, PCK @0.10. 10, and PCK @0. 15, respec-

---

Published as a conference paper at ICLR 2025
1-shot pose estimation (OnePose-LowTex)
1-shot pose estimation (YCB-Video)
80-
Griginal model
Rostured (ours)
Rretuned
Orighnl
Faveuneed
s 1cm-ldeg
1cm-1deg
70
3om-3deg
3om-3dey
Som-Sdeg
Scm-5deg
60
20
Pase sccuracy PM
50-
G2
Del
DINOVE mg
MSPD
Original
Bretunes
AR
Orighnl
Pnstuned
40
25
0
Gos mg
MB6
Del
129
De07
DINOV2
DINOV2 reg
MAE
CUI
Delf
Figure 7: One-shot pose estimation results before and after feature equivariance finetuning.
ise Original
Finetuned
sset Original
Finetuned
I000 Original
Finetuned
Avarage jocard Insox
40-
Rpphiin ton Accuracy
60
Occission Accuracy
80
30-
80-
60
20-
80
DRRO22
DINO2--reg
MIE
CUP
DelT
DINOV2
DINOV2-1e9
CLPP
Del
DMN02
DINO22-reg
MAE
CLPP
DerT
Figure 8: Video tracking results before and after feature equivariance finetuning.
tively. Notably, we find that fine-tuned models show enhanced understanding of keypoint semantics across different instances, even from the same viewpoint. This suggests that 3D equivariance con- tributes to a better understanding of fine-grained semantics, despite not finetuned for that purpose.
We also compared with FiT Yue et (2024) and DUS:3 Wang et (2024), while their perfor- mance are much worse than ours. Detailed quantitative results including FiT and DUSt3R on all these tasks are available the supplementary materials.
3.3 EXTREMELY FEW-SHOT FINETUNING
Training with Only One Object We plot the performance relative to the number of training ob- jects used, as shown in Figure 10, keeping the total number of iterations fixed at 10K. Surprisingly, fine-tuning on just one object already provides significant performance improvements. Addition- ally, the object was randomly selected from Objaverse. We tested six different objects, all of which yielded similar results. The results are shown in Figure 11. Notably, even simple shapes like an un- textured hemisphere can enhance the 3D correspondence understanding of the ViTs in these tasks.
Convergence Within a Few Iterations Figure 12 plots the performance of downstream tasks versus the number of training iterations on single object. Interestingly, our experiments reveal that training with just a single multi-view pair of one object for a single iteration significantly boosts the model's 3D equivariance, as shown by the sharp improvement at the first elbow of Figure 12. This finding is remarkable, indicating that fine-tuning for 3D correspondence vision transformers is highly efficient in capturing essential 3D spatial relationships with minimal data. Even with such a minimal training setup. the model effectively learns the desired 3D properties, substantially improving performance across tasks without requiring extensive training or large datasets.
3.4 FINETUNING VIT ENIIANCES 3D TASKS THE WILD
key advantage of VTT features studied here are highly generalizable across diverse datasets and tasks, supporting a even wider range of applications. For example, SparseDFF Wang et al. (2023) uses DINO to aggregate and fine-tune consistent features across views for few-shot transfer manip- ulation policy learning; LERF Kerr et al. (2023) employs dense DINO features for regularization; and Wild Gaussians Kulhanek et al. (2024) utilizes off- off-the-shelf DINO features as priors to estimate occlusions and reconstruct 3D scenes in complex settings. To demonstrate the effectiveness of our fine-tuned features, we conducted experiments on Wild Gaussians (W-G) and found that replacing the original features with our fine-tuned DINO features improved novel view synthesis quality in
7

---

Published as a conference paper at ICLR 2025
Different viewpoint
Same viewpoint
70
Koypaint motching accuacy (8))
model
Finctuned lours)
Keypoine morching acturacy (1)
Original model
Finctuned tours)
60-
PEX0.05
PCKO
80
PCX0.0r
FCKO.05
PCX0.10
PCKO 10
PCX9.10
PCK0.10
50-
PCK0.15
PCK0.15
PCK0.15
PCK0.15
60-
40-
D2002
DINOxt-reg
MAE
LUP
DelT
DO0
GINOV2
LUP
Delf
Figure Semantic correspondence results before and after feature equivariance finetuneing.
1-shat pose estimation (OnePose-LowTeN
Scm Seeg
3om 40n---55
1cm
10.0
70.0
20
20
Video tracking (TAP-VID-DAVIS)
45.0
05.0
42.5
60.3
20
40
60
80
100
40
100
20
10
80
100
Semantic correspondence (PF-PASCAL, different view)
60.0
2000
45.0
Figure 10: Finetuned performances w.r.f. #training objects. We evaluate the performances of the DINOv2 model finetuned with 0, 5, 10, 20, 50, 100 objects on the three tasks.
the wild, as shown in Table Additionally, in the supplementary we show that substituting LERF's DINO regularizer with fine-tune tine-tuned version enhances language- eebeede field performance, with detailed results and analysis provided therein.
Frestin
Palr
PSNR
LIGS
PSNK.1
PSSRTT
SH0E
LPPS
PSNK
SS0M
LMPS
PSNR
5500
LPUSS
LMUS
W-4:
30.02
unek
0.209
21:0
IL213
21.51
0.00
0152
21.11
B1324
23.05
0.77
0.16
6334
Oaz
21.01
0.72
0.234
2035
8,62
0,213
23744
0800
8.51
21.35
0.02
0333
24.1
8,778
0.063
22.11
0.734
0,301
Table 1: Quantitative comparison of novel view synthesis quality across different scenes.
Our fine-tuned DINO features consistently improve performance over the original Wild-Gaussians method, showing higher PSNR and SSIM scores, and lower LPIPS values.
DESIGN CHOICES FOR FINETUNING
In this section, we ablate and verify the design choices of our finetuning strategy and share some findings. We use the best DINOv2 base model for all our ablations.
4.1 ADDITIONAL CONVOLUTION LAYER HEAD
We append a single convolution layer to the original model architecture and find that gives sur- prisingly good performance. Adding single convolutional layer to the finetuning architecture was motivated by the need to improve the resolution and consistency of the dense feature maps produced by Vision Transformer (ViT) models. The typical ViT models process images low-resolution patches, and while global attention mechanisms facilitate communication between patches, they are not optimized for generating dense per-pixel features during interpolation. By incorporating volutional layer with kernel size of 3 and a stride of 1, we can explicitly exchange information between neighboring patches, allowing the model generate more accurate and high-resolution feature maps before interpolation. We ablate the number of convolutional layers and table 2 shows that one conv layer gives the best performance.
4.2 TRAINING DATA
MVImgNet v.S. Objaverse Our results indicate that finetuning on MVImgNet is slightly worse compared finetuning Objaverse, likely due to the denser correspondences provided by OB-
8

---

Published as a conference paper at ICLR 2025
1-shot estimation (OnePose Low]ex)
Video (TAP-VID-DAVIS)
Jenrcc Becoent Podeo
62
-2
Scm 1dng
Semantic (PF-PASCAL)
Finetuning with different objects. All results are tested with finetuned DINOv2. Dashed lines indicate the performances of the original pretrained model. The feature finetuning method is effective with as few as one single object. also shows insensitivity to the specific choice of the object, even if the object has limited textures uncommon in daily life.
Figure 11:
1-shot pose estimation (OnePose-LowTen
22.0
3cm 350
3cm
10.0
52.0
100
4200
6000
8000
10000
10000
2000
6000
8000
10000
Video tracking (TAP-VID-DAVIS)
62.5
4 42.5
82.0
2000
4090
5000
8000
10000
2000
4000
6000
3000
10000
2000
4000
6000
8000
10000
Semantic correspondence (PF-PASCAL, different view)
PC0.05
PCC0.10
63.0
53.0
02.5
8009
2000
10000
Figure 12: Finetuned DINOv2 performances w.r.f #training iterations, trained with only one object over 0, 5, 10, 20, 50, 100, 1000, 10000 training iterations.
javerse. Both datasets provide a similar object-centric multi-view setup. Although Objaverse is synthetic dataset and MVImgNet consists of real-world captures, large foundation models tend to be largely agnostic to the distinction between simulated and real images.
Object-centric datasets v.s. scene-centric datasets interesting result, as shown in Table 3, that finetuning scene-centric datasets (e.g. RealEstate Zhou et al. (2018), Spaces Flynn al. (2019), and LLFF Mildenhall et al. (2019), which contain diverse real-world scenes with complex backgrounds, does not necessarily improve the performance but sometimes make it worse (e.g. PF- PASCAL). This may indicate that 3D objects themselves have already encoded enough 3D spatial reasoning information. And scene-centrie dataset does include much more background clutter that may distract the network, leading to less accurate feature representations.
4.3 Loss FUNCTIONS
We start with naive contrastive loss and found that it does not perform well. This is because contrastive loss does not directly optimize for the correspondence. In contrast, SmoothAP optimizes ranking loss directly, leading responding pixels. We also experimented with the differentiable Procrustes alignment loss from Li al. (2022), but did not outperform SmoothAP, Detailed comparisons are given in Table
improved average precision for feature retrieval between
5 RELATED WORKS
Vision Transformers Dosovitskiy (2020) (ViTs) have made significant strides in image understand- ing by employing self-attention mechanisms to capture global contextual information, outperform- ing traditional convolutional neural networks (CNNs) in tasks such image classification and ob- ject detection. However, despite their success 2D applications, adapting these models to grasp 3D spatial relationships remains challenging and relatively unexplored area.

---

Published as a conference paper at ICLR 2025
OnePose-LowTes Tex
TAP-VID-DAVIS
PF-PASCAL (Dift. View)
ViT models
Icm ldeg
3cm 3deg
Sem Sdle
AJ
OA
PCK0.05
PCK0.10
PCK0.15
DINOv2-FT (Cony
11.69
53.85
72.83
60.79
84.08
57.14
65.26
DINOv2-FT (Conv 1)
13.58
$8.03
77.35
46.85
63.84
84.15
47.25
60.76
67.57
DINOv2-FT (Cony 2)
13.12
56.14
75.45
47.42
63
84.
46.32
58.05
64.90
DINOv2-FT (Conv 3)
12.15
53.63
74.46
46.84
62.14
82.90
41.60
53.97
60.22
Table 2: Ablation on the number of appended conv layers.
OuePose-LowTex Tex
TAP-VID-DAVIS
PF-PASCAL (Diff. View)
VIT models
lem-Ideg
Jem-3dog
5em-5deg
8ang
0A
PCK0.05
PCKO.10
PCK0.15
DINOv2-FT (Objaverse) 13.58
58.03
77.35
46.85
63.84
84,15
47.25
60.76
67.57
DINOv2-FT (MVIngNet)
13.65
56.98
74.61
41.53
58.89
82.67
45.13
57.93
65.40
DINOv2-FT (Scene-Centric)
15.95
60.79
76.35
47.36
63.07
80.27
41.73
52.33
60.33
Table 3: Ablation on the dataset used for finetuning.
OnePose-LooW/e
TAP-VIDDDAVIS
PF-PASCAL (Diff. View)
VIT models
lem-Idog
3em-3deg
5cm-5deg
AJ
OA
PCK0.05
PCK0.10
PCK0.15
DINOv2-FT (SmoothAP)
13,58
58.03
77.35
46.85
63.84
84.15
47.25
60.76
67.57
DINOv2-FT (Contrastive)
13.28
55.57
75.68
43.79
62,20
81.84
46.70
58.08
66.21
DINOv2-FT (DiffProc) 12.92
55.00
74.86
43.60
61,32
82.74
43.89
57.22
64.66
Table 4: Ablation on the loss function used. SmoothAP delivers the best overall performance.
There is growing interest in assessing the 3D comprehension of vision models. While some studies have investigated how well generative models capture geometric information from a single image Bhattad et al. (2024); Du et al. (2023); Sarkar et al. (2024), these efforts are generally specific to generative models, limiting their applicability to broader vision tasks. More closely aligned with our work is El Banani et al. (2024), which evaluated the 3D awareness of visual foundation models through task-specific probes and zero-shot inference using frozen features. In contrast, we delve deeper and introduce a simple yet effective method for finetuning 3D awareness in ViTs.
Several researchers have also explored applying large-scale models to 3D tasks. For instance, some approaches utilize features from pre-trained models for tasks such as correspondence match- ing Zhang et al. (2023); Cheng et al. (2024) and pose estimation Ömek et al. (2023). Ima- geNet3D Ma et al. (2024) investigates how global tokens from ViT vary across views to aid pose estimation. While their work focuses on view-dependent global features, ours emphasizes dense, pixel-level features invariant to viewpoint changes. Their top-down pose estimation approach classi- fies poses using pretrained features with a domain-specific linear layer, which limits its applicability across diverse datasets. In contrast, we argue that finding correspondences, or learning equivariant representations, a more effective strategy for general unseen tasks and datasets.
Recent works, such as FiT Yue al. (2024) and DVT Yang et al. (2024), attempt finetune pre- trained features. FiT lifts 2D features into 3D space and then projects them back into 2D to enforce 3D consistency. DVT, on the other hand, implements denoising process reduce periodic noise artifacts in images, method that is orthogonal to our approach. Additionally, DUSt3R Wang et al. (2024) directly predicts 3D coordinates for each 2D pixel, but it lacks shared consistent feature space and forfeits the rich semantic information provided by large vision models.
6 CONCLUSION
In this work, we systematically evaluated the 3D awareness of large vision models, with a specific focus on their ability to maintain view equivariance. Our comprehensive study demonstrates that current vision transformers, particularly DINOv2, exhibit strong 3D equivariant properties, which significantly correlate with performance on downstream tasks such as pose estimation, video track- ing, and semantic transfer. Building on these insights, we introduced simple yet effective finetuning method that enhances the 3D) correspondence understanding of 2D ViT's By leveraging multiview correspondences and applying loss function that enforces feature consistency across views, our ap- proach yields substantial improvements in task performance with minimal computational overhead. Remarkably, even a single iteration of finetuning can lead notable performance gains.
Our findings highlight the importance of 3D equivariance in vision models and provide a practi- cal path to improving 3D correspondence understanding in existing models. We believe this work opens up new opportunities for enhancing the 3D capabilities of vision transformers. All code and resources will be made publicly available to support further research in this direction.
10

---

Published as a conference paper at ICLR 2025
7 STATEMENTS
Ethics Statement. Our method leverages open-sourced simulation data and real data whose data collection process follows strict ethical guidelines. In using these data, we follow the same ethical considerations to protect sensitive information. There no ethical concerns detected of the proposed method to our knowledge, and we will strive to adhere to ICLR code of conduct for future use of the proposed method.
Reproducibility Statement. We provide extensive details for ease of re-implementation. We strive to ensure our method is reproducible and the findings in this paper are generalizalble. We will release code, results, and scripts for reproduction promote future research in 3D deep learning.
8 ACKNOWLEDGEMENTS
Yang You is supported in part by the Outstanding Doctoral Graduates Development Scholarship of Shanghai Jiao Tong University. Congyue Deng, Yang You, and Leonidas Guibas acknowledge sup- port from the Toyota Research Institute University 2.0 Program, ARL grant W911NN-21-2-0104,, Vannevar Bush Faculty Fellowship, and gift from the Flexiv corporation. Yue Wang acknowledges funding supports from Toyota Research Institute, Dolby, and Google DeepMind. Yue Wang is also supported by Powell Research Award.
REFERENCES
Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual de- scriptors, arXiv preprint arXiv:21 2(3):4, 2021.
Anand Bhattad, Daniel McKee, Derek Hoiem, and David Forsyth. Stylegan knows normal, depth, albedo, and more. Advances in Neural Information Processing Systems, 36, 2024.
Irving Biederman. Recognttion-hy-componnntt a theory of human image understanding. Psycho- logical review, 94(2):115, 1987.
Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman. Smooth-ap: Smoothing the path towards large-scale image retrieval. In European conference computer vision, pp- 677-694. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pp. 1660-9660, 2021.
Xinle Cheng, Congyue Deng, Adam Harley, Yixin Zhu, and Leonidas Guibas. Zero-shot image feature consensus with deep functional maps. arXiv preprint arXiv:2403. 12038. 2024.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotate 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309, 16588, 2023.
Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of anno- tated 3d objects. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13142-13153, 2023.
James J DiCarlo and David D Cox. Untangling invariant object recognition. Trends in cognitive sciences, 11(8):333-341, 2007.
Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point a video. Advances in Neural Information Processing Systems,
2022.

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1192) 2020.
Xiaodan Du, Nicholas Kolkin, Greg Shakhnarovich, and Anand Bhattad. Generative models: What do they know? do they know things? let's find out! arXiv preprint arXiv:2311.171377 2023.
Mohamed EI Banani, Amit Raj, Kevis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Ru- binstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3d awareness of visual foundation models. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 21795-21 806, 2024.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381-395, 1981.
John Flynn, Michael Broxton, Paul Debevec, Matthew Vulal,, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
2367-2376, 2019.
Bumsub Ham, Minsu Cho, Cordelia Schmid, and Jean Ponce. Proposal flow: dences from object proposals. IEEE transactions on paltern analysis and machine intelligence, 10(7):1711-1725, 2017.
Semantic correspon-
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022a.
Xingyi He, Jiaming Sun, Yuang Wang, Di Huang. Hujun Bao, and Xiaowei Zhou. Onepose++: Keypoint-free one-shot object pose estimation without cad models. Advances in Neural Informa- lion Processing Systems,
2022b.
Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion. arXiv preprint arXiv:2305. 15581, 2023.
Tomáš Hodañ, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother, and Jiř Matas. Bop challenge 2020 on 6d object localization. In European Conference on Computer Vision, pp. 577-594. Springer, 2020.
Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf diet: Semantically consistent few- shot view synthesis. In Proceedings of the IEFF/CVE International Conference on Computer Vision, pp. 5885-5894, 2021.
Varun Jampani, Kevis-Kokitsi Maninis, Andreas Engelhardt, Arjum Karpur, Karen Truong, Kyle Sar- gent, Stefan Popov, Andre Araujo, Ricardo Martin-Brualle Kaushal Patel, Daniel Vlasic, Vittorio Ferrari, Ameesh Makadia, Ce Liu, Yuanzhen Li, and Howard Zhou. Navi: Category-agnostic image collections with high-quality 3d shape and pose annotations. In NeurlPS, 2023. URI. https: http///avviiaaaet....e g ioo.b.io/.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. arXiv preprint arXiv:2307.07635, 2023.
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Lan- guage embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19729 197399, 2023.
Kurt Koffka. Principles of Gestalt psychology, volume 44. Routledge, 2013.
Wolfgang Köhler. Gestalt psychology. Psychologische Forschung, xXXX)XXVII-XXX, 1967.
12

---

Published as a conference paper at ICLR 2025
Jonas Kulhanek, Songyou Peng, Zuzana Kukelova, Mare Pollefeys, and Torsten Sattler. Wildgaus- sians: 3d gaussian splatting in the wild. arXiv preprint arXiv:2407.08447, 2024.
Yann Labbé, Lucas Manuelli, Arsalan Mousavian, Stephen Tyree, Stan Birchfield, Jonathan Trem- blay. Justin Carpentier, Mathicu Aubry, Dieter Fox, and Josef Sivic. Megapose: 6d pose estima- tion of novel objects via render & compare. arXiv preprint arXiv:2212.068700 2022.
Lei Li, Hongho Fu, and Maks Ovsjanikov. Wsdesc: Weakly supervised 3d local descriptor learning for point cloud registration. IEEE Transactions on Visualization and Computer Graphics, 29(7): 3368-3379, 2022.
Wufei Ma, Guamning Zeng. Guofeng Zhang, Qihao Liu, Letian Zhang, Adam Kortylewski, Yaoyao Liu, and Alan Yuille. Imagenet3d: Towards preprint arXiv:2406.09611 2024.
object-level 3d understanding. arXiv
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (ToG), 38(4):1-14, 2019.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Evin Pmnar Örnek, Yann Labbé, Bugra Tekin, Lingni Ma, Cem Keskin, Christian Forster, and Tomas Hodan. Foundpose: Unseen object pose estimation with foundation features. arXiv preprint arXiv:2311.1800, 2023.
Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondfej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5706-5715, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
Ayush Sarkar, Hanlin Mai, Amitabh Mahapatra, Svetlana Lazebnik, David A Forsyth, and Anand Bhattad, Shadows don't lie and lines can't bend! generative models don't know projective ge- ometry... for now. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 28120-28111111 2024.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from rgbd images. In Computer Vision-ECC 2012: 12th European Conference Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pp. 746-760. Springer, 2012.
Emma EM Stewart, Frieder Hartmann, Yaniv Morgenstern, Katherine R Storrs, Guido Maiello, and Roland W Fleming. Mental object rotation based on two-dimensional visual representations. Current Biology, 32(21):R122) 1224 R R1225, 2022.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. arXiv preprint arXiv:2306.0388 03881, 2023.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In European confer- ence on computer vision, pp. 516-533. Springer, 2022.
Narek Tumanyan, Assaf Singer, Shai Bagon, and Tali Dekel. Dino-tracker: Taming dino for self- supervised point tracking in a single video. arXiv preprint arXiv:2403. 14548, 2024.
Thomas Vetter, Anya Hurlbert, and Tomaso Poggio. View-based models of 3d object recognition: invariance to imaging transformations. Cerebral Cortex, 5(3):261-269) 1995.
13

---

Published as a conference paper at ICLR 2025
Qianxu Wang. Haotong Zhang, Congyue Deng, Yang You, Hao Dong, Yixin Zhu, and Leonidas Guibas. Sparsedff:: Sparse-view view feature distillation for one-shot dexterous manipulation. arXiv preprint arXiv:2310.16836 2023.
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20697-20709, 2024.
Kevin Wilson and Martha J Farah. When does the visual system use viewpoint-invarian repre- sentations during recognition? Cognitive Brain Research, 16(3):399-415, 2003.
Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neu- ral network for 6d object pose estimation cluttered scenes. arXiv preprint 1.0019, 2017.
Jiawei Yang, Katie Luo, Jiefeng Li, Kilian Q Weinberger, Yonglong Tian, and Yue Wang. Denois- ing vision transformers. arXiv preprint arXiv:2401.028 022.0000.997,.. 2024.
Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Chenming Zhu, Zhangyang Xiong, Tianyou Liang, al. Mvimgnet: large-scale dataset of multi-view images. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 9150-9161, 2023.
Yuanwen Yue, Anurag Das, Francis Engelmann, Siyu Tang, and Jan Eric Lenssen. Improving 2d feature representations by 3d-aware fine-tuning. arXiv preprint arXiv:2407.20229, 2024.
Junyi Zhang. Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. arXiv preprint arXiv:2300. 15347, 2023.
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 1,Xiv:1805.09817, 2018.
14


Reviews:
[{"review_id": "o7YSsLOg4C", "reviewer": "Reviewer_f5uK", "summary": "This paper evaluates 2D ViT-based foundation models' abilities to learn 3D equivariant features, shows the significance of 3D equivariant on 3D downstream tasks (pose estimation, video tracking, and semantic correspondence), and proposes a very simple finetuning strategy that boosts the 3D understanding abilities of these existed 2D foundation models by introducing 3D information from either synthetic data (Objaverse) or real data (MVImgNet).", "questions": "Please refer to the weakness part for my questions. \n\nOverall, this works presents an interesting yet very simple method that is easily reproducible to make 2D vision model generates better 3D equivariant features, and I believe it can benefit the community.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed finetuning strategy is simple to adopt and easily reproducible. Finetuning shows performance boost with even only one additional sample.\n\n2. The experiments are comprehensive, covering three main downstream tasks of 3D equivariant features (one-shot object pose estimation, video tracking and semantic correspondence), highlighting the significance of 3D equivariant features. \n\n3. The paper discusses finetuning using different types of data (synthetic data, real data and scenes), and conducts good ablations of the model's design (added conv head).", "weaknesses": "Minor concerns:\n1. Table 1 ablates on number of added conv layers to a given ViT, and one additional conv head gives best performance boost instead of two or three. Some analysis of why this is happening will be nice. \n\n2. The whole paper lacks some mathematical formulation and explanation. For example, there is no formula for two evaluation metrics define in the paper: APE and PCDP. Also, it does not have math for the loss (SmoothAP). Some additional math can be more followable than text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "52FXCbKCRq", "reviewer": "Reviewer_XtFz", "summary": "In this work the authors studied the importance of multiview equivariance for the tasks of pose estimation, video tracking, and semantic correspondence. Results show that vision models with better multiview equivariance also achieve better performance for the three downstream tasks. Moreover, by finetuning the model on synthetic multi-view images, models with better equivariance perform better on various tasks.", "questions": "The authors could provide some clarifications on **[W2]** and **[W3]**.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors studied the multiview equivariance property of vision foundation models, and associate it with the performance of three downstream tasks. This enable a more systematic way to analyze the part correspondence of vision models and help to understand the limitations of models on downstream tasks.\n2. The authors proposed to finetune the model with multi-view synthetic images, improving multiview equivariance and downstream tasks. This proposed approach is straightforward but demonstrated effective on downstream tasks.", "weaknesses": "1. The three tasks considered in this paper, (keypoint-based) pose estimation, video tracking, and semantic correspondence, are all ultimately part correspondence problem, which benefits from multiview equivariance. The title and introduction gives the impression that multiview equivariance improves 3D understanding in general, but truly the experiments only focused on very specific tasks.\n2. I understand that pose estimation is a 3D understanding problem, but I don't think video tracking and semantic correspondence falls into the picture of 3D understanding, given the title of the paper. Specifically how this paper fits into the analysis of 3D awareness considering previous works [A,B].\n3. The authors argued the importance of multiview equivariance on tasks such as pose estimation, video tracking, and semantic correspondence. This is only partially true as it also depends on the nature of the algorithm, bottom-up or top-down. For instance, [B] studied the 3D awareness of vision foundation models for pose estimation. Vision-language models often learns a top-down representation for 2D/3D recognition so view equivariance could hurt the performance in such cases. The authors should address these points to reinforce the integrity of the paper.\n\n[A] Probing the 3d awareness of visual foundation models.\n[B] ImageNet3D: Towards General-Purpose Object-Level 3D Understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tT8BReZOVe", "reviewer": "Reviewer_A2uw", "summary": "This paper evaluates the 3D awareness of ViT-based models and later proposes a strategy to improve 3D equivariance with minimum feature finetuning. The tasks for evaluating 3D awareness of ViT-based models are one-shot object pose estimation, video tracking, and semantic correspondence, and features from DINOv2, DINOv2-Reg, MAE, CLIP and DeiT are evaluated. Experimental results show that with some simple strategy of finetuning the foundation models, their 3D awareness can be improved with an obvious margin and the foundation model features can have better multi-view equivariance.", "questions": "-- This paper mainly studies ViT-based foundation models. Are there any reasons not to study foundation models with other architectures, like ConvNeXt [1]? Is it because in current days ViT-based models are the most commonly used ones? And do the authors expect non-ViT-based models to have similar behaviour as ViT-based models?\n\n-- In Lines 363-364, the paper mentions that \"even simple shapes like an untextured hemisphere can enhance 3D understanding\". However, I do not find this point reasonable (although this might be supported by experimental evidence). In principle, an untextured hemisphere would be rotation-invariant when the viewing angles rotate in certain directions, due to the symmetry of the shape, which makes the features at the points on the same radius to be consistent. How could the model actually learn 3D understanding from such a hemisphere shape? I am not sure whether this can be shown from some visualizations of the learned feature maps of a hemisphere shape.\n\n[1] Liu et al. A ConvNet for the 2020s. CVPR 2022.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "++ This paper is very well motivated and super clearly written. The paper first starts from evaluating the capability of vision foundation models on understanding 3D structures. Then, the paper shows the strong correlation between the multi-view equivariance and the performance of the chosen downstream tasks, demonstrating the reason for improving multi-view equivariance. Finally, the paper proposes a solution to improve the multi-view equivariance and therefore on the downstream tasks. The workflow of this paper is very natural and easy to understand.\n\n++ The proposed solution to improve multi-view equivariance is simple but effective. With simply learning the equivariance on two views from the objects in Objaverse, the multi-view equivariance can be improved, so as their performance on downstream tasks.\n\n++ The experimental results are extensive and clearly presented mainly in the forms of figures (e.g., Figures 3-4, Figures 6-10), to clearly show the improvement from employing the proposed feature finetuning method. \n\n++ It is a very interesting and inspirational finding in Section 3.3 that only tuning the model with a single multi-view pair of one object for a single iteration can significantly boost the multi-view equivariance of foundation models.", "weaknesses": "-- I think the biggest weakness is that there is no recent methods for comparison on the chosen downstream tasks for reference. I did not mean that the performance of foundation models need to beat the current state-of-the-art on these specific tasks, but it is necessary to provide these comparisons to give readers a sense of how good foundation models can achieve in performance. If performance from foundation models are far away behind the current state-of-the-arts, then there will be less need or motivation for future research to work on employing foundation models for 3D tasks.\n\n-- This paper only studied the final-layer features from the vision transformer models. However, in other works that study the representation capability of features like LPIPS [1] or DVT [2], features from multiple layers are studied. Are there any reasons for studying the final-layer features? Otherwise this study would lose generalizability.\n\n[1] Zhang et al. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. CVPR 2018.\n\n[2] Yang et al. Denoising Vision Transformers. ECCV 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LQALOlswS8", "reviewer": "Reviewer_unL9", "summary": "This paper answers 3 questions: 1) To what extent do these models possess an inherent awareness of 3D structures? 2) How\ndoes this awareness impact their performance on image-based 3D vision tasks? 3) Can we further\nenhance the 3D awareness of these vision foundation models?\nTo answer the first question, the authors evaluate the pixel error of multi-view correspondence by using 5 pretrained vision models, and show DINOv2 has the strongest multiview equivalence. For the second question, the authors evaluate the performance of these models on 3 downstream tasks (pose estimation, video tracking, and semantic correspondence). Finally, the author proposes a simple finetuning strategy by enforcing feature similarity between corresponding views of a rendered synthetic object. Consistent improvements on downstream tasks are shown compared to the model without finetuning.", "questions": "1. The authors compared the FiT baseline in Tab 3~7 in the supplement. I have some trouble understanding the result. 1) What is the difference between FiT and FiT-Reg? 2) The FiT results are significantly worse than the DINOv2 baseline. Since FiT is also based on DINO, this result is counterintuitive. The authors are basically claiming that the FiT fine-tuning is very harmful for all the tasks. Could the authors provide more evidence (e.g., visualizations) and discussion on this? I think FiT is an important baseline so this comparison will greatly affect my judgement of this paper.\n2. The 3D correspondence is sometimes ambiguous, for example when the object is symmetric, or when self-occlusion occurs and the corresponding point is occluded. Have the authors deal with these cases explicitly? Would this ambiguity harm your model's performance?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly organized and easy to follow. The figures and tables are also very clear and easy to understand.\n2. The experiments are pretty comprehensive, covering 5 popular pretrained vision models, and 3 downstream tasks. Some interesting settings such as fine-tuning with only one object and with only 1 iteration are covered.", "weaknesses": "1. My main concern regarding this paper is about the limited practical use case of the finetuned DINO features. On the one hand, the fine-tuned feature alone is not very useful, as its performance on downstream tasks is much worse than the SOTA models (see Tab4 and Tab5 in the supplement, e.g., for point tracking, AJ=46.85 compared to Co-Tracker=65.6). On the other hand, there is no evidence showing that the fine-tuned feature this way will benefit downstream tasks if task-specific training/finetuning is performed. For example, the paper will be much more convincing to me if the fine-tuned DINO, as pre-trained weight, could be used to achieve SOTA performance on any specific vision task.\n2. The conclusion that the authors draw to the first question, i.e., \"To what extent do these models possess an inherent awareness of 3D structures\", is not convincing. Having feature equivariance doesn't imply 3D awareness: SIFT could also match the same keypoint across views, even more accurately, but can you say SIFT has 3D awareness? The correspondence could be just coming from 2D local patch statistics, and no 3D reasoning is needed.\n3. The evaluation protocol for the downstream tasks is not explained very clearly. For example, L160, you mentioned \"during training\" and \"during inference\", what is the training target and how long has the model been trained for? L214 in supp. How do you do monocular depth estimation with pretrained features? Do you need any additional training (e.g., using linear probing)? I think it would be beneficial if the authors could demonstrate the input/output of each task with figures.\n4. There is no comparison with baselines on 3D-aware fine-tuning in the main body of the paper. However, there are comparisons with FiT in the supp, which I think is important and should be moved to the main paper. I have questions regarding the details of this comparison, see the next section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)