Foundation Vision Models are Unsupervised Image Canonicalizers

Authors: Utkarsh Singhal, Ryan Feng, Stella X. Yu, Atul Prakash
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Invariance, Canonicalization, Foundation Models, CLIP, SD, Augmentation, Vision, Robustness

Abstract:
One of the most significant and longstanding problems in computer vision is invariance - the ability to robustly handle changes in real-world transformations such as rotation, viewpoint, and lighting. Unfortunately, popular foundation models remain brittle under such transformations. While existing solutions towards invariance have shown promise, they all fundamentally require some model training, limiting their ability to adapt broadly to new tasks, transformations, and datasets. Our key insight is that foundation model priors can be used to reason about transformations. We thus propose Foundation Model Canonicalization (FMC), an approach that can undo nuisance transformations in images without any model training. With a single core approach, FMC can make models like CLIP and SAM invariant to different transformations without any training or fine-tuning. Our approach FMC flexibly adapts to new foundation models and tasks, making it significantly easier for newer and larger models to achieve invariance.

Decision: reject
Reviewer scores: 3, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1729981409493

thread_type:
official_review

<thread_object>
creator: reviewer_wapx

<content>
summary:
The paper proposes a training-free method for image canonicalization based on pre-trained foundation models. The method is based on the hypothesis that the canonical image will have the minimal energy evaluated using the foundation models. The authors permute the possible augmentation and use the proposed method to evaluate their energy.

strengths:
- This paper proposes a training-free method for image canonicalization using pre-trained foundation model, which is interesting.
- The paper is easy to follow and the idea is easy to understand.

weaknesses:
- This method formulates the image canonicalization problem as an optimization problem, which needs to permutes all possible augmentation/transformation, which is slow and is sample-inefficient.
- Moreover, the method is dependent on human-designed transformations, e.g. color, viewpoint, rotation, etc. It may have limited potential to generalize real-world transformation that are actually very complex.
- No reason or intuition on why picking the mentioned three vision foundation models. Why using them, not other foundation models?
- No ablation on figuring out which foundation model is useful for recovering which transformation. And no ablation on whether it is necessary to use all three foundation models.
- Besides, I am actually not sure whether this method can generalize to any vision foundation models. For example, DINO/DINOv2, which uses contrastive learning for representation learning might not suitable for this task, as they might not be sensitive to transformations/augmentations.
- The authors should provide a deeper understanding on how to choose these foundation models and how to balance the energy they contribute to the final energy function.

questions:
Please see comments above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730095044598

thread_type:
official_review

<thread_object>
creator: reviewer_cgaq

<content>
summary:
FMC introduces a training-free approach to equip foundation models, like CLIP and SAM, with canonicalization-based invariance, enhancing their adaptability across various models and a wide range of downstream transformations.

strengths:
1. This energy function-based training-free method is technically sound and novel. 
2. The experiments are convincing and sufficient, validating the proposed methods across multiple dimensions, such as datasets (Sec. 5.1), models (Sec. 5.2), and transforms (Sec. 5.3).
3. The paper is well-motivated (a training-free general-invariance method) and easy to understand.

weaknesses:
Please see the "Questions" section.

questions:
For Takeaway #9, 

1. The performance gains in Fig. 6 (a) and Fig. 6 (c) are not monotonous and I would like to see more analysis regarding it. 

2.  Zero123 is trained on Objaverse, how about the performance comparison on other datasets, such as OmniObject3D [1] and Co3D.

3.  How about changing the Zero123 baseline to the advanced image-to-3D methods, like One-2-3-45 [2] and Unique3D [3].

[1] OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generation. CVPR 2023

[2] One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization. NeurIPS 2024

[3] Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image. NeurIPS 2024
</content>
</thread_object>

</thread>
<thread>
creation date:
1730549438193

thread_type:
official_review

<thread_object>
creator: reviewer_dvf9

<content>
summary:
The paper studies to what extent large pre-trained models can be used to canonicalize images with respect to transformations such as rotation and lighting. The proposed approach is to define an energy $E(I)$ for every image $I$ by combining pre-trained CLIP, stable diffusion and SAM and to use brute-force search or bayesian optimization to find the transformation $t$ that minimizes $E(t(I))$. The finding is that this works quite well, i.e. that the proposed energy can be used to canonize images to get improved downstream performance.

strengths:
1. The approach is simple and novel.
2. The paper demonstrates that CLIP, Stable diffusion and SAM (at least when combined) have a strong knowledge of the distribution of internet images under common image transformations.

weaknesses:
1. The approach needs to apply three large pre-trained models (including 500 steps of stable diffusion) to many transformations of the input image. It must be quite computationally expensive, but this is not commented on.
2. There is no comparison to test-time-augmentation, which would be the most common approach to get invariance/equivariance from a non-invariant/equivariant model. Since the proposed approach requires evaluating pre-trained models on several input transformations it seems not to have a computational advantage over test-time-augmentation, which previous work on canonicalization might have had.
3. The approach is claimed to be training-free, but the energy hyperparameters need to be tuned using bayesian optimization (Appendix A.3).

Post-rebuttal:
- The paper is now more complete, with detailed discussion on computational tradeoffs and comparison with TTA.
- I am not convinced that the approach is practically useful. But the paper can be viewed as a reasonable step towards a practically useful method for canonicalization.

questions:
1. How large is the computational cost? In particular compared to using test-time-augmentation.
2. Is the performance better than using test-time-augmentation?
3. How computationally expensive is the tuning of the energy hyperparameters?
4. What is the typical number of transformations used to find the canonical one? How much does the performance improve for, say, each doubling of the number of transformations?
5. What is the advantage of the proposed approach over using the downstream model for the task at hand for canonicalization? For instance, in image classification, the classification model itself could be used to define an energy similar to the CLIP-energy. (This, again, would be a sort of test-time-augmentation.)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730959468876

thread_type:
official_review

<thread_object>
creator: reviewer_hfht

<content>
summary:
This paper propose a training-free adaptation approach for vision foundation models, e.g. CLIP and SAM, to adapt on unseen 2D image rotation. They conduct experiments on small datasets e.g. CIFAR and STL10 with C4/C8 rotations. Results show some improvement vs baseline w/o such adaptation.

strengths:
Improvement for robustness of foundation models are useful.  Training free adaptation reduce the limitation of usage of those adaptation.

weaknesses:
In the abstract the author talk about rotation, viewpoint, and lighting. However, in the experiment, they only conduct on 2D image rotation.  The 2D in-plane rotation is far too simple, and easy to solve. One can easy train a additional small estimation via self-supervision (data augmentation in training). Or use some existing approach to detect the rotation first [1]. Unless the author can show some results also for other type of perturbation, otherwise, it is hard to convince me the effectiveness of the approach. Additionally, CIFAR10 and STL10 is consider too small and far from realworld usage, I would command the author to conduct more experiment on larger dataset.

[1] Maji, S., & Bose, S. (2020). Deep image orientation angle detection. arXiv preprint arXiv:2007.06709

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1733292566144

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Summary

comment:
We thank the reviewers for their engagement and guidance. Here, we provide a summary of the paper’s main contributions, new results from the discussion, and improvements to the paper we will make for the camera-ready version. We request each reviewer to read this summary and consider the changes in the whole context in their recommendations during reviewer AC discussions. In particular, we believe the new and updated results in 3D, day-night transformations, and active vision make the paper much stronger.

**Main Contribution**: Our paper provides an unsupervised approach to achieving invariance. Rather than imposing specific transformations or ranges on the training data or architecture, we build a system of modules to reason about the data distribution. Specifically, we combine the knowledge from foundation models (which, by themselves, do not have invariance) together in a way that enables us to perform canonicalization through unsupervised energy functions. This flexible design enables us to achieve superior levels of invariance and generalize across tasks, models, and transformations without any training.

**Key points**: 

**1.** In our initial draft, we already show the capabilities of our approach in more settings than popular NeurIPS and ICML papers that study canonicalization  (e.g., Learned Canonicalization and Prior-Regularized Learned Canonicalization [1, 2]):

   **1a:** We extend our evaluations to ImageNet, whereas prior papers traditionally evaluate on smaller datasets like CIFAR10/STL.

   **1b:** We extend our evaluations on 3D viewpoint changes, a much more complicated transformation that cannot be handled by classic methods like equivariant networks.

**2.** Even so, we further extended our approach to work on day-night transformations and active vision during the rebuttal period, two challenging settings that prior invariance work has not studied (to our knowledge).

**3.** These results show the generalizability of our approach. By extracting the knowledge from foundation models, we generalize (without training) across models, tasks, and transformations.



**Discussion Summary:**

**Reviewer HfHT:** We corrected factual misunderstandings in the review (our paper does look at transformations beyond 2D, and our paper does evaluate on datasets like ImageNet). Unfortunately, HfHT has failed to respond.

**Reviewer dVF9:** The main concerns of this review were (1) runtime, (2) comparison against TTA, and (3) hyperparameter tuning. We provided TTA comparisons showing our advantage, promised to add a subsection detailing runtime cost, and hyperparameter sensitivity (more details below). 

dVF9 agrees that our paper is a reasonable step towards a practical canonicalization method in the future and has increased their score in response. 

Overall, we believe this discussion has made the paper better and more complete. We are especially grateful to dVF9 for their productive engagement and useful suggestions.

**Reviewer CgAq**: The main concerns of this review were the 3D experiments. We explained the noise in our plots and promised to add error bars. We also showed the potential of our ranking function to perform extremely well with a perfect generator. We also ran preliminary results on CO3D as well as an active vision scenario.


**Reviewer wapX**: The main concerns of this review were (1) the generality of our method, (2) sample complexity, and (3) ablations and discussion about which energy function to use. We fixed  point 3 in the current revision and strongly disagree with their assessment for 1 and 2:

*Generality*: We pointed out that the 3D viewpoint example is already extremely difficult, and traditional approaches like equivariant nets cannot handle it. This means our paper already goes beyond the current invariance methods . Since wapX did not specify which transforms they would like to see, we added several new results on real-world transformations (see results below). 

*Sample complexity*: We clarified that we are not using brute-force search and added a sample complexity plot showing significant advantages against the brute-force approach.

To our knowledge, they have not responded yet to these arguments/results.

[1] https://arxiv.org/pdf/2211.06489 
[2] https://arxiv.org/pdf/2310.01647
</content>
</thread_object>

</thread>


Metareview: