Test-Time Canonicalization by Foundation Models for Robust Perception
Authors: Utkarsh Singhal
Ryan Feng
Stella X. Yu
Atul Prakash
Venue: ICML 2025 poster (2025)
Keywords: equivariance
invariance
CLIP
diffusion
canonicalization
ecological
unsupervised
test-time scaling
Primary area: deep_learning->robustness
TL;DR: Invariance to diverse and complex transformations by leveraging knowledge in vision foundation models.

Abstract: Perception in the real world requires robustness to diverse viewing conditions. Existing approaches often rely on specialized architectures or training with predefined data augmentations, limiting adaptability.  Taking inspiration from mental rotation in human vision, we propose FoCal, a test-time robustness framework that transforms the input into the most typical view. At inference time, FoCal explores a set of transformed images and chooses the one with the highest likelihood under foundation model priors. This test-time optimization boosts robustness while requiring no retraining or architectural changes.
Applied to models like CLIP and SAM, it significantly boosts robustness across a wide range of transformations, including 2D and 3D rotations, contrast and lighting shifts, and day-night changes. We also explore potential applications in active vision. By reframing invariance as a test-time optimization problem, FoCal offers a general and scalable approach to robustness. Our code is available at: https://github.com/sutkarsh/focal .

Test- Time Canonicalization by Foundation Models for Robust Perception
Utkarsh Singhal
Ryan Feng
Stella X. Yu 12
Atul Prakash"
Abstract
Dut-oo-Diisributtion
Viewpoint
Lighting
Environment
Rotation
Perception in the real world requires robustness to diverse viewing conditions. Existing approaches often rely on specialized architectures or train- ing with predefined data augmentations, limiting adaptability. Taking inspiration from mental rota- tion in human vision, we propose FOCAL, a test- time robustness framework that transforms the in- put into the most typical view. At inference-time, FOCAL explores a set of transformed images and chooses the one with the highest likelihood under foundation model priors. This test-time optimiza- tion boosts robustness while requiring retrain- ing architectural changes. Applied to models like CLIP and SAM, it significantly boosts ro- bustness across wide range of transformations, including 2D and 3D rotations, contrast and light- ing shifts, and day-night changes. We also ex- plore potential applications in active vision. By reframing invariance as test-time optimization problem, FOCAL offers general and scalable approach to robustness. Our code is available at: https//giihuu..oomssuuaaatt
Foundation Model Guided Canonicalization (FoCal)
In-Distribution
"Bench" Recognition (CLIP g))
"Chair"
X
Segmentation (SAM *))
Figure I. Introducing FOCAL: test-time framework for approx- imate invariance to complex transformations at scale. (top) Given variations in viewpoint, illumination, environment, and rotation, FoCAL use foundation model priors select risually-typical version the input. (bottom) out-of-distributiin inputs can lead to incorrect predictions on CLIIP ("bench") and incorrect segmen- tations on box-prompted SAM (missing body parts in the segmen- tation mask). Using FOCAL canonicalized versions fixes these errors. Thus, FOCAL offers a scalable, data-driver method for robust perception across diverse transforms.
1. Introduction
A robot navigating a cluttered home or a self-driving car on a street must reliably recognize objects despite changes in viewpoint, orientation, and lighting. Such robustness is essential for embodied perception in dynamic environments.
In contrast, even leading foundation models exhibit brittle- (Figure 1): CLIP (Radford al., 2021) misclassifies images from uncommon viewpoints, and SAM (Kirillov et al., 2023) incorrectly segments objects presented side- ways. This vulnerability stems partly from the overrepresen- tation of upright poses and ideal lighting in internet training data a phenomenon known as photographer 's bias. Con- sequently, these models are unprepared for diverse visual inputs seen embodied settings (Madan al., 2024).
Human vision has evolved to excel this task: We readily recognize familiar objects from diverse viewpoints while mentally rotating unfamiliar objects to typical views to fa- cilitate recognition (Shepard & Metzler, 1971). Through natural visual exposure, we develop invariances that adapt each object and context, generalizing to novel ones while preserving necessary distinctions. That is, human perception develops invariances shaped by ecological demands.
Data augmentation (DA) and equivariant networks are two common approaches to improve neural network robustness to specific transformations. DA exposes models to care- fully selected transformations during training (e.g., rotations, crops), but struggles with rare classes (Zhou et al., 2022) and hurts some classes through over-regularization (Xiao et al.,
"quual contribution 'UC Berkeley Uniierriiyy Michigan. Correspondence to: Utkarsh Singhal <s.utkarsh www.karsh@bbkkkly...oo berkeley.edu>>
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Test-Time Canonicalization by Foundation Models for Robust Perception
2021). Equivariant networks (Cohen & Welling, 2016) build mathematical symmetries (e.g., 2D rotations) directly into the architecture, but this approach does not extend to com- plex real-world transformations like 3D viewpoint changes.
Input
Canonical
Critically, both approaches hard-wire invariances into the model during training through either curated training data or the model architecture. This strategy is inherently rigid: it assumes prior knowledge of relevant transformations and often fails under distribution shifts such novel viewpoints or lighting conditions beyond those seen during training.
Step 1: Vary
Step 2: Rank (CLIP+SD energy)
(spply transform)
Figure 2. Foundation Model Guided Canonicalization: Our method works in two steps: Vary and Rank. We first produce transformed variations of the input image and then rank them using energy functions derived from CLIP and Stable Diffusion. The minimizer of this energy function the most visually typical version, which we refer to as the "canonical". This approach is training-free, cenofformmtiinnnnootttei and highly generalizable.
We propose a fundamentally different approach: Instead of hard-wiring invariance during training, our Foundation- model guided Canonicalization (FOCAL) achieves invari- ance by reasoning over transformations at inference time (Figure 1). Like human vision, which mentally rotates unfa- miliar objects to canonical views for recognition (Shepard & Metzler, 1971), FOCAT. leverages foundation models' visual priors to transform inputs toward visually typical views.
approximate invariance method that scales to complex trans- formations, including 3D viewpoint shifts, lighting changes, and environmental variations. (3) We demonstrate the effec- tiveness of FoCAL through evaluations on modern models such as CLIP, OV-Seg, and SAM, across diverse datasets including ImageNet, COCO, Objaverse-LVV and CO3D.
FOCAL uses a "vary and rank" strategy (Figure 2): It gen- erates candidate transforms (e-g.. via generative models) and selects the most visually typical instance by minimiz- ing an energy function based on Stable Diffusion (SD) and CLIP. The optimized instance is then passed to downstream models for inference. This perspective connects to recent test-time scaling work where learned rankers select among outputs (Snell et al., 2024). However, FOCAL distinguishes itself by grounding ranking in data-driven visual typicality.
Our results challenge the prevailing assumption that achiev- ing invariance requires specialized training architectural design, offering path toward robust perception for embod- ied agents. By bridging insights from human mental rota- tion and modern test-time compute scaling (Zaremba et al., 2025), FOCAl demonstrates that test-time optimization can serve as scalable approach to robust visual perception under real-world variation.
Our approach enables robust perception across diverse con- ditions, including changes in viewpoint, lighting, and envi- ronment, without any additional training or architectural changes. Invariance in our method does not arise from hard- wired constraints, but is instead an emergent property of our test-time optimization. Unlike prior canonicalization work that requires transform-specific training (Mondal et al., 2023), FOCAL works for wide range of datasets and trans- forms. By reframing invariance test-fime optimization, FOCAL offers a data-driven method for robustness at scale.
2. Robust Perception by Minimizing Foundation Model Energy Functions
We begin with a motivating example: rotated image. When presented with upside-down photograph, humans natu- rally mentally rotate it to understand its contents (Shepard & Metzler, 1971). How can we automatically find such "canonical" (or visually-typical representations based on the priors in foundation models?
We evaluate FOCAL across a range of challenging transfor- mations, including 3i-viewpoint, illumination, day-night changes, and 2D rotations. We find that FOCAL improves out-of-distribution performance of foundation models such as CLIP (Radford et al., 2021) on ImageNet (Deng et al., 2009) scale datasets. To our knowledge, this is significant improvement in dataset size and transformation complexity compared to previous canonicalization work. Remarkably, FOCAL consistently matches or outperforms task-specific canonicalizers like PRLC (Mondal et al., 2023) even in their trained settings, despite operating entirely at test time.
2.1. Problem setting
We denote the input image by I E X, where X' is the space of images. Images may undergo transformations : X' + X. such as rotations. We denote the set of such transformations (e.g., the group of image rotations) by T.
Given an image, we may want to process it using an existing neural network (c.g., for recognition or segmentation). We denote this operation with a function f : X - y with inputs x E X' and outputs y E Y, where y is the space of outputs (e.g.. classes or masks).
Contributions. (1) We introduce FOCAL, a test-time, data- driven framework that leverages the visual priors of foun- dation models to achieve invariance. (2) We propose
2

---

Test-Time Canonicalization by Foundation Models for Robust Perception
FOCAL
Model Energies
Total Energy
CLIP
71
SD
X 72
rotation
color
viewpoint
Figure 3. Transformation distributions define a slice through the distribution of natural images, enabling us to use foundation propose common
models to canonicalize. Given the complex distribution of natural data, which spans many transformations, solution that applies across arbitrary transformation based slices of the distribution (left). Given a particular slice (red curve in the example above), simulate different versions of the input image along this slice of the distribution (e.g.. color variations), using energy functions built on CLIP (Radford et al., 2021) and Stable Diffusion (Rombach et al., 2021) models for each sample (model energies in the figure above). We minimize the total energy predict the canonical version (right, with the optimum shown by star). In summary. our method uses internet-scale priors Iearned by foundation models to canonicalize diverse transformations.
For recognition, want the function f to be invariant to the transformations in T, meaning that it should produce the same output regardless of how the input is transformed. For example, if we rotate an image, the recognition function should still recognize the object in the image:
But how do construct such a canonicalizer h? Under mild conditions, Kaba et al. (2022) show that if the canoni- calizer h is defined as an energy minimizer over transforma- tions, then it achieves the desired invariance and equivari- ance propertics. Specifically, they show that the canonical- izer can be defined as:
Invariance: f((()) = f() VIET
For segmentation, we want the function to be equivariant to the transformations in T, meaning that the output should change predictably when the input is transformed. For ex- ample, if we rotate an image, the segmentation mask should also be rotated:
h(z)= = arg min E(t(x)) IET
(1)
where F. X' + R is a real-valued function referred to as the "energy function". Strikingly, this result does not require E or ff be equivariant or invariant. Instead, the canonicalizer h emerges from the minimization process.
Equivariance: f(t(z))= =f(((x))) TT
Caveat: This framework guarantees invariance for invert- ible transformations, but many important transformations are non-invertible (e.g.. viewpoint shifts). However, in prac- tice, we find that generative models can still provide apprex- imate invariance for some easier cases (Figure 5).
where e' is the corresponding transformation in the output space y, like the segmentation mask rotating with the input.
We next explain how canonicalization achieves these desiderata in the context of recent work by Kaba et al. (2022) with simplified notation and more general transformations.
Secondly, invariant doesn't imply correct: canonical can be out- out-of-distribution for the downstream model, leading to low accuracy. Mondal et al. (2023) at- tempt to fix this by training the canonicalizer on the target dataset. We instead pick the most visually typical (likely in in-distribution) version with foundation model priors.
The chosen
Canonicalization refers to the process of transforming an input into a fixed "canonical" version regardless of the transformations applied. For rotations, this might look like uprighting image so that it is always presented in standard orientation.
Canonicalization:
Kaba et al. (2022) formalize this with canonicalizer func- tion h: : X + T that maps the input to the transformation needed to "canonicalize" it. The core idea is that the canon- ical image (a)) is the same regardless of the input's transform, i.c., h ((t())) = h(=) (x) for all e T.
2.2. Key Insight: Slices of Natural Image Distribution
Building on this foundation, we make a crucial observation: an image æ and all its transformed versions t(x) Vt E T define a slice through the natural image distribution Paata (az) (Figure 3). Within this slice, some versions appear more frequently in real-world data than others. We find the most likely version with visual priors in foundation models.
Thus, for invariance, it is sufficient apply the canonicalizer to the input before processing it with the function f:
let ã : = t(x: (<(())=)))= =f
If we define an energy function E R - log Pdata then min- imizing E over the transformed versions of x effectively finds the most probable version of the image within that slice. Denoting the 'best" transform t = arg min,er F((((z)),
Here, h(z) effectively undoes the transformation on x, achieving invariant output.
3

---

Test-Time Canonicalization by Foundation Models for Robust Perception
Candidates
Energy Function
for random variable RDD can be written
Po(x) = Z e Ea(x)
X
red/ red/green ratio
where Z(0) is the normalizing constant and Eg RR" + R is the energy function. Small values of Egg (x) correspond to more likely x.
CLIP Energy: Following Grathwohl et al. (2020), any clas- sifier can be seen as an energy-based model. The classifier energy for fo with input æ and output y is the negative logit:
X
x
Eo(x, v) = - fo
blue/green ratio
For unconditional energy, Grathwohl et al. (2020) use log- sum-exp over labels, but we simplify this using combi- nation of mean and max logits:
Figure Foundation Model Guided Canonicalization in contin- uous and multi-dimensional transform spaces: Given an input image, we generate different transformed versions the image (candidates). Each is ranked by a combined energy function shown in Figure 3, with the minimum of this grid representing the canonical form. However, brute- force search is infeasible for con- tinuous or multi-dimensional transform spaces such as color shift. Thus, apply Bayesian Optimization for efficient optimization in such spaces. Black crosses show the locations of BO guesses, and the yellow shows the location of the found optimum. BO finds the optimum without using hundreds of evaluations. Thus, combining energy functions and Bayesian optimization provides general approach to canonicalization.
ECLIP a, (a: mcan - max (c)
2E1,2..../CII
where a, I E E are hyperparameters and fe [c] is the class c logit. We use CLIP ViT-H-14 14 (Ilharco et al., 2021) with cosine similarity of image & text embeddings as logits.
Diffusion Energy: Following Graikos et al. (2022), dif- fusion models provide effective image priors through free energy minimization:
Exi((
1 T BeNN(on)) [[le (xx
Σ
(2)
the canonical image x=* = t" (az) is likely the most visually typical or informative version of the image. For example, if æ is a rotated image, the canonical form x* is likely the upright version of that image. For a color-shifted image, the canonical form is likely the version in typical lighting.
where x, v + VT- āre is the noisy imput, Eg is the pre-trained diffusion model, and =I Bi) with denoising schedule 3r. We use SD-2-base (Rombach et al., 2021), finding that 5 - 10 steps are usually sufficient.
Since modern foundation models like CLIP and Stable Dif- fusion are trained on vast datasets of natural images, they implicitly learn visual priors about natural data. We can therefore extract energy functions E s Pog from these models to guide canonicalization.
2.4. Foundation Model Guided Canonicalization
'Vry'' and 'Rank": Our method consists of two steps: Vary, where we produce variations of an input image, and Rank, where we rank the variations (Figure 2). We optimize the FOCAL energy function over transforms of the input a::
Note that in our instance, the "canonicalized" views do not necessarily need to be exactly "upright" - the idea simply that we try to map all inputs to the same, optimized view for invariance. As long as such view is in-distribution, it is suitable for downstream recognizers and segmenters.
t' arg min EPOCAL (t(x))
EET
y f(t (x))
(4)
Next, we formalize energy functions and describe how to extract them from foundation models.
where T is the set of transformations, EFOCAL is the energy function defined below, and y is the invariant output.
2.3. Energy Functions from Foundation Models
Combining energy functions: We minimize the combined energy EFOCAL. (I)) over all transformations E T to find the canonical version of the input image æ. This done by solving the following optimization problem:
We extract energy functions from pre-trained foundation models, which can be readily combined with each other and with hand-designed priors.
Energy-based models: Any probability distribution (x
EFOCAL (1(x)) = 71 ECLPP(((2)) + (am(((())))
(5)

---

Test-Time Canonicalization by Foundation Models for Robust Perception
Original Canonicalized
Energy
Filtered Objaverse LVIS
1.0
Examples of View Difficulty
-60
Elevation
(30
0.8
(33)
Accuracy
0.6
+50%
(66)
-180
170-120
-00
-60
-80
6
69
90
120
150
0.4
Azimuth
0.2
(100)
Finctuned CLIP
0.0
0
25
50
75
100
Viewpoint Difficulty Percentile
Figure 5. Viewpoint canonicalization improves accuracy by improving viewpoint robustness: (Left) Example renderings from Objaverse-LVIS before (original) and after (canonicalized) applying FOCAL. The canonicalized views provide more informative perspectives for object recognition. (Middle) Energy plot showing different clevations (y axis) and azimuths (rr axis), Locations with lower energy (darker purple) correspond with more canonical views that can be more casily classified correctly (green box). In contrast, locations with high energy (yellow) tend to be incorrect (red box). Star denotes the optimal, selected view. (Right) Accuracy on quality-filtered Objaverse-LVIS across different viewpoints ranked by difficulty, with easier viewpoints on the left and harder viewpoints on the right. The finetuned CL.IP model (OV-Seg) (purple line) exhibits steep accuracy drop for difficult viewpoints, whereas FOCAL. (orange line) maintains more stable performance, achieving 50% higher accuracy for difficult views. Examples on the right depict truck render difficulty percentiles ranging from 0 (easiest) 100 (hardest). In summary, we find that FOCAL identifies more visually typical versions of input images, enabling higher performance on bad input 3D viewpoints.
where at 1,3,7, 71, 72 IR are hyperparameters.
objective function based on a few evaluations. We utilize Bayesian Optimization (BO) with Gaussian Process (GP) using RBF kernel and the Expected Improvement (EI) acquisition function (Jones et al., 1998) to balance explo- ration and exploitation. This approach balances exploring uncertain regions with exploiting promising areas, typically finding good solutions in 50-100 evaluations.
Intuitively, CLIP energy focuses semantics, selecting the image that most closely resembles a predefined category, while SD energy acts as a general appearance-based prior.
Assumptions:
For invariance, we assume that the transfor- mation invertible. Under mild conditions used in Kaba al. (2022), the energy optimization produces invariant out- puts. For high downstream accuracy, we assume: (1) there is at least one in-distribution image in the set of transformed images, (2) the foundation models can be used as a prior where in-distribution images have lower energy than out-of- distribution ones, and (3) the downstream model performs best in-distribution. If the first two assumptions hold for given sample, the energy minimization scheme returns image that is in-distribution. If the third assumption holds, this process is likely to result in higher accuracy.
BO commonly used for optimization problems like hy- perparameter search and has been successfully applied to high-dimensional problems such as optimization in the la- tent space of generative models (Maus et al.. 2022; Gómez- Bombarelli et al., 2018; Castro et al., 2022; Tripp et al., 2020). For even higher-dimensional problems, gradient- based optimization another potential alternative. We leave the exploration of gradient-based methods to future work.
3. Experiments
Bayesian Optimization for Efficient Search: While ex- haustive search can be used to minimize the energy function in Section 2.4 for a small number of transformations (e.g., rotations around the circle), it is infeasible for continu- ous and nigher-dimensional transformations. This critical since many common transformations high-dimensional: color transformation (Figure 7) is 2D, "active vision" setting is 6D, and combining transforms is even higher-dimonn dingenr-dimessoo.
This section outlines our experimental settings and results. We pick classification and segmentation as standard tasks and test FOCAL on diverse transformations (viewpoint shift, color, contrast, day-night, active vision). We find that Fo- CAL generalizes across many different settings and even beats PRLC (Mondal et al., 2023) on 2D rotations with their jointly-trained classifiers. These results show the generaliz- ability and wide applicability of FOCAL as a canonicalizer.
Fortunately, this problem can be handled by Bayesian Opti- mization (BO) (Nogueira, 2014; Frazier, 2018) (Figure 4), well-established method for efficient optimization in both low and mid-dimensional spaces. BO utilizes a probabilistic model, such as Gaussian Process (GP), to approximate the
3.1. 3D Viewpoints
We evaluate FOCAL as a method for approximate invariance to viewpoint transforms on Objaverse-LVIS (Deitke et al.,
5

---

Test-Time Canonicalization by Foundation Models for Robust Perception
Original
Canonicalized
Filtered CO3D
Color Shifts (CIFAR100)
Contrast (ImageNet- 1k)
0.7
GT Prob. 0.3
Finetuned CLIP
0.6
Ours
Accuracy (%)
40
0.5
23
Accuracy
70
Default
0.4
Naive CLIP
65
0.3
Ours
GT Prob. 0.5
60
02
05
U8
10
0.0
033
13
0.2
log-chrominance distance
log-contrast distance
Color Contrast
0.1
0.0
GT Probability Threshold
Figure Viewpoint canonicalization improves accuracy on CO3D. For each CO3D video. randomly sample frame with ground truth label probability below two different difficulty thresh- olds (0.3, 0.5). (Left) We show examples the left of input image that is incorrectly classified at first but is correctly classified chair after running through FOCAL. (Right) Accuracy com- parison. FOCAL (orange) outperforms finetuned CLIP (OV-Seg) (purple), highlighting its ability (approximately) canonicalize viewpoints in realistic settings.
Figure 7. FOCAL canonicalizes illumination (color & contrast) We evaluate FOCAL using CLIP two different datasets (CI- FAR100, ImageNet-IK3 and transformations (color, contrast), (a) FoCAL improves CLIP's accuracy color-shifted images by 9.9% on average and nearly 15% for larger shifts. (b) FOCAL improves CLIP's accuracy under contrast changes by 4.1% on average and nearly 12% for larger contrast shifts. These results highlight FOCAL's ability to handle lighting transformations.
2023) and CO3D (Reizenstein et al., 2021). Objaverse-LVIS dataset contains 46K 3D assets and CO3D contains 19K real multiview video sequences with object segmentations.
This result shows the generalizability of FOCAL to 3D view- point transformations. To our knowledge, this result rep- resents a significant step forward in invariance methods applied to viewpoint transformations. We include further comparisons against test-time augmentation (TTA) and in- teresting findings that suggest that TRELLIS (Xiang et al., 2024) itself can canonicalize to an extent on inputs near its training distribution Appendix A.2.
We begin by filtering datasets to ensure high-quality objects and viewpoints. For Objaverse-LVIS, we noticed cases of misleading and overlapping labels and thus filtered out such objects. For CO3D, we use sequences with sufficient variation viewpoint quality. Given the filtered videos, we then sample one of the poor viewpoints (i.e., ground truth probability of < for some threshold !) and filter images with blurriness segmentation entors (Appendix B.I).
CO3D: We then test FOCAL's 's ability to improve the accu- racy of our selected CO3D frames. We show in Fig. 6 for both threshold of f 0.3 and a threshold of 0.5 that FOCAL improves the accuracy of our frames by 43.8% and 26.8%, respectively. We also show example renders and additional comparisons in the Appendix (Figure 11 and A.7). This result further shows our capability to canonicalize over 3D viewpoints on real-world images.
We use TRELLIS (Xiang et al., 2024) to produce view- point variations for each input. Specifically, we render views on the sphere at 30 intervals for the "vary" stage (Appendix 1. We rank these renders using FOCAL en- ergy and pick the view with the minimum energy. Since TRELLIS generates background-remover images, we use OV-Seg (Liang et al., 2023), version of CLIP fine-tuned on bookkround-removeed images, as our classifier.
3.2. Lighting (color and contrast)
We evaluate chrominance (color) and contrast transforma- tions on CIFAR100 (Krizhevsky et al., 2010) and Ima- geNet (Deng et al., 2009) with CLIP (Radford et al., 2021). Unlike classic methods, FOCAL achieves invariance with- out requiring specialized architectures trained on curated ground truth (Hernandezz-uurez et al., 2020).
Objaverse-LVIS:
We include example renders and accuracy plots over viewpoints in Figure 5. find that FoCAL often canonicalizes input images from didistiiibtttion views to canonical forms, increasing accuracy. We demonstrate this by ranking viewpoints by ground truth classification probability (best to worst) and computing average accuracy per rank. Figure 5 shows that FOCAL significantly improves classification accuracy for difficult viewpoints (12.0% to 62.0% on the worst viewpoints) and improves the overall stability (max accuracy min accuracy).
As shown in Figure 7, FOCAL improves accuracy by 9.9% for color shifts and 4.1% for contrast shifts over vanilla CLIP, with gains reaching 15% and 12% for extreme trans- formations. While not surpassing supervised approaches
6

---

Test-Time Canonicalization by Foundation Models for Robust Perception
Classification Accuracy
ImageNet-IK
(Relative to Rotation Aug)
75
Canonical Histogram
PRLC*
Random Rotation (C8) Acc. (%)
Accuracy Gain (%)
7,5
Ours
Energy
70
5.0
2.5
65
0.0
Interpolation Factor in Latent Space
PRIC
Ours
Night
Day
60
Night Image
Optimized Image
Day Image
CIFAR10
CIFAR100
ResNet
55
STL10
VIT
R-50
VIT
CLIP
Figure FOCAL beats PRLC on their image classification eval- uations: We evaluate FOCAL on 2D rotations following the setting of Mondal et al. (2023). (a) Left plot shows classification accuracy gains relative rotation augmentation baseline across multiple datasets (CIFARIO, CIFAR10,, STL10) using both ResNet-50 and VIT architectures. Each paired line represents different evaluation setting. with FOCAL (Ours) consistently matching or outperform- ing PRLC in each setting. (b) We evaluate each of PRIC's trained canonicalizers on an unseen dataset (ImageNet- 1 K) and pick the best canonicalizer for each architecture. We find that FOCAL still beats them as PRLC's canonicalizers struggle to generalize out- side their training setting. In short, FOCAL outperforms trained canonicalizers of Mondal et al. (2023) on seen as well as unseen datasets, highlighting FOCAL'S generalizability.
Figure 9. FOCAL can canonicalize day-night transformations. We evaluate the CLIP energy function as we interpolate between the day-night image pairs in the latent space of diffusion model, finding that FOCAL prefers day images with 91% accuracy. also plot the energy function (purple) and the histogram (yellow) of the energy optimum for 5000 pairs, showing that the chosen images are close to daytime. Thus, FOCAL can choose more visually typical and informative daytime images in the day-night transformation setting in the latent space of diffusion model.
with DA. We train a ResNet-32 on CIFARIO for 100 epochs with LR 10-3 The C8 rotated accuracy is 72% with just DA, 72.1% with just FOCAL, and 73.2% with FOCAL+DA. We leave a rigorous study of this setting to future work.
like (Barron & Tsai, 2017; Hernandez-Juarez et al., 2020) (Appendix A,9), FOCAL maintains stable accuracy across variations (details in Appendix B.2), demonstrating lighting canonicalization without specific training or architecture.
Segmentation: We also evaluate PRLC's segmentation set- ting with SAM (Kirillov et al., 2023) using the pre-trained SAM canonicalizer supplied by the authors. Here, we find that our method matches their pre-traine pre-trained canonicalizer in mAP (65.9) while achieving +2.1% better pose accuracy.
3.3. 2D Rotation (Comparison against PRLC)
Classification: We compare against PRLC (Mondal et al., 2023) on their 2D rotation settings (C8) using PRI-C-trained ViT (Dosovitskiy et al., 2021) and PRIC- trained ResNet50 (He et al., 2016) models across CI- FARIO (Krizhevsky et al., 2010), CIFAR100 (Krizhevsky al., 2010), and STL.10 (Coates al., 2011).
These results show that FOCAL not only matches PRLC's performance on their trained datasets without requiring any training but also outperforms PRLC in novel settings be- yond their original training scope. Further cross-dataset and cross-classifier generalization results are in Appendix A.
We report accuracy on upright images, rotated images (on C'8 rotations), pose accuracy (i.e., did the canonicalizer pick the correct C8 rotation). and pose error (i.c., the average error in degrees) (details in Appendix, Table 3, summary in Figure 8). We find that FOCAL matches or beats PRLC's specially trained canonicalizers on every one of their evalu- ated settings zero-shot (Figure 8).
3.4. Day-Night transformation
We apply FOCAL on day-night transformations modeled by UrbanIR (Lin et al., 2024), a state-of-the-art relighting model. We use their pre-trained KITTI (Geiger al., 2012) checkpoints and relighting code to render 2000 pairs of day- night frames. Since it is unclear what classes should be used for CLIP energy, we simply use single class "street." This energy picks day images 91% of the time.
We extend this setting further to CLIP and ImageNet (Deng et al., 2009), finding that FOCAL generalizes much better to this setting (by to +11% than PRLC, which struggles to generalize beyond its training scope (Figure and Table 5).
To further study FOCAL's energy function, we create more variations of each scene that lie between day and night im-
We also benchmark how our method helps in combination

---

Test-Time Canonicalization by Foundation Models for Robust Perception
flowers
garden
treehill
bicycle
bonsai
room
stump
kitchen
Optimized Random Pose Initialization
Figure 10. Applying FOCAL to active vision: We optimize a 6-DoF camera pose (x, y, z, yaw, pitch, roll) in 3D virtual environment. We use Bayesian optimization to minimize FOCAL energy. and plot the results above along with randomly chosen poses for comparison. The first row shows the initialization, second row shows a random pose, and the third row shows the result from FOCAL. We find that the camera naturally focuses on salient objects and tends maintain upright angles, reflecting behaviors observed in simpler settings (green underline). Although some failure cases remain (red underline), these results illustrate FOCAL 's ability to generalize to significantly more complex transformations and viewing conditions a zero-shot manner.
ages by interpolating their latent vectors in the latent space of Stable Diffusion 2 (Rombach et al., 2021). This pro- cess creates a restricted latent space containing variations of the scene. We plot the energy function over this space (Figure 9). We then use FOCAL canonicalize this trans- form by optimizing the energy function in the latent space, finding that the canonicals concentrate around day images.
viewpoints, distances, and with varying reflections. While this experiment is preliminary and the process is not yet fully reliable, it shows FOCAL's generalizability.
4. Related Work
Training-time data augmentation: Data augmentation during training is the simplest and most popular way to achieve invariance. Recent work, such as VIAT (Ruan et al., 2023), ViewFool (Dong et al., 2022), and Ruan et al. (2024), use adversarial viewpoints as augmentations during training/fine-tungg
3.5. Active vision
We also show exploratory application of FOCAL's energy function to more realistic 3D setting: optimizing camera parameters in a virtual environment (modeled by a Gaussian splat (Kerbl et al., 2023)). This setting models a robotic agent navigating the scene. has degrees-of offrreddoof translational axes (x.y.z), and rotations (yaw, pitch, roll). The camera then moves around the virtual scene, searching for the view that optimizes FOCAL energy.
However, it requires fixing the transfor- mations ahead-of-time, and thus adapting existing model (e.g., CLIP (Radford et al., 2021)) new transformations incurs expensive re-training or fine-tuning costs. Addition- ally, the range of augmentations (c.g., rotation degrees) is unknown and artificially chosen, which can hurt accuracy for some classes (Bouchacourt et al., 2021; Kirichenko et al., 2024). Furthermore, the resulting model is not as robust for classes with fewer training examples (Zhou et al., 2022), making this approach unfit for imbalanced datasets (i.e., most modern datasets like LAION-5B (Schuhmann et al., 2022)). In contrast, test-time approaches like ours provide reliable invariance regardless of the training dataset.
We pick 8 scenes (Figure 10) and use the same translation ([-3, 3)) and rotation range (full rotations) for each scene. We use ImageNet classes to compute CL.IP energy and mini- mize it using 150 iterations of BO. We also found that using 450 initial random points can help exploration but is unnec- essary for most scenes. We hope to study the optimization schedules more systematically in the future.
Equivariant networks: Architectures like CNNs (LeCun et al., 1999; Fukushima, 1988) and group-equivariant net- works (Cohen & Welling, 2016; Cohen al., 2019; Kondor & Trivedi, 2018; Bronstein et al.. 2021) hardcode symme- try priors. While effective for fixed groups (e.g., 2D ro- tations) or 3D point clouds (Deng et al., 2021), they fail
We find that this process leads the camera focus salient objects in typical angles (like in our previous experiments) in Figure 10. We note that this setting is a significantly more complex "transformation" than classic examples like 2D rotation. Each camera pose sees objects from different
8

---

Test-Time Canonicalization by Foundation Models for Robust Perception
for complex transformations lacking group structure (e-g., viewpoint changes). Our approach imposes no architectural constraints, enabling broader applicability.
5. Limitations and Future Work
FoCAL demonstrates strong performance and broad ap- plicability across variety of transformations, including viewpoint, lighting, and environmental changes. FOCAL draws inspiration from mental rotation and test-time scal- ing, extracting priors contained in internet-scale foundation models to perform test-time alignment to visually-typical version. FOCAL is thus a significant step forward towards robust perception, with two main limitations and directions for future work that we discuss here: speeding up FOCAL and automatically selecting the transformation.
Learned invariance:
Recent methods aim to learn the in- variances from the data by learning augmentation ranges per transformation (Augerino (Benton et al., 2020)), per instance (InstaAug (Miao al., 2022)), or via normalizing flows (Singhal al., 2023; Allingham et al., 2024). How- ever, they remain tied to training data and cannot adapt post-hoc. In contrast, our method extracts invariance from foundation models without dataset-specific training.
Learned canonicalization: Learned canonicalization has roots in mental rotation (Shepard Metzler, 1971; Hock & Tromley, 1978). Tarr& & Pinker (1989) drew ties between mental rotation and invariant object recognition. These works suggest that canonicalizing can robustly align neural networks to the adaptable nature of the human brain.
Speeding up FOCAL. Evaluating the energy function for many candidates is computationally expensive (Ap- pendix A.10). Specifically, the complexity is N x (Cransffomm + Cevergy + Cinference), where N is the number of transforms. This limitation is similar to recent LLMs that use test-time search for better reasoning and robustness. Fu- ture work could explore a System- 1/2 scheme where canon- icalization is only used when necessary, enabling users to gain the robustness benefits of FOCAL without incurring as much of cost in runtime efficiency. As a preliminary exper- iment, in 2D, we were able to detect upright vs. non-upright images with 95% accuracy by comparing the input against + -090 and 900 and thresholding the energy difference.
These developments have inspired progress in deep learning architectures (Jaderberg et al., 2015; Boominathan et al., 2016). More recently, Kaba et al. (2022) propose a learned canonicalization (LC) approach via minimizing a learned energy function. At test time, it minimizes this function to canonicalize the inputs. PRLC (Mondal et al., 2023) adds regularization prior to align the canonical with the train set distribution for better accuracy.
Selecting the Transformation. While FOCAl shows great progress in achieving robust visual perception across a wide variety of transformations at test-time, it remains an open question on how to select which transformation generator(s) to use. Future work could explore how to automatically determine which generator(s) to use at test-time, making FOCAl even more easily applicable to real-world settings.
However, LC and PRLC still require training specific to the dataset and transform, and do not generalize well beyond their trained settings (Section 3). In contrast, our approach makes no such assumptions; instead, it leverages pre-trained foundation models. Furthermore, we show promising results on significantly more complex transformations.
3D robustness and pose estimation: Existing approaches for 3D robustness pool features across multiple views (Fan et al., 2024; Su et al., 2015; Yang & Wang, 2019; Wei et al., 2020; 2022; Hamdi et al., 2021: Kanezaki al., 2018), whereas our technique only requires one view at test time. Chen et al. (2020) learns category-level pose estimation using by-ythhss...sttssss.... This approach is related to our approach; however, it is category-specific, whereas our model is sategory-agnostic ImageNet3D (Ma et al., 2024) annotates a large dataset of 3D objects with poses for open- set pose estimation, whereas our method unsupervised.
6. Conclusion
FOCAL offers test-time, scalable, data-driven strategy for robust visual perception. FOCAL takes inspiration from canonicalization, projecting out-of-distribution inputs to the most visually-typical view. By leveraging the priors learned by foundation models trained on internet-scale im- age datasets, FOCAL'S "vary and rank" scheme enables us to optimize for visually-typical views that are suitable for downstream models. By challenging the prevailing as- sumption that invariance requires dataset-specific training or architectural compromises, FOCAL offers a new, test-time path toward robust perception for embodied agents facing a wide variety of transformations.
Out-of-distributio detection: based models have previously been used for OOD detection (Hendryeks & Gimpel, 2017; Liu et al., 2020; Lee et al., 2018; Liang et al., 2018; Graham et al., 2023), but this capability has yet to be harnessed for invariance. To our knowledge, this work is the first to leverage large-scale generative models in conjunction with Equation (1) to create provably invariant models.
A Acknowledgements. This project was supported, in part, by Beyster Fellowship to R. Feng, by NSF 2215542, NSF 2313151 and Bosch gift funds to S. Yu at UC Berkeley and the University of Michigan.
9

---

Test-Time Canonicalization by Foundation Models for Robust Perception
Impact Statement
European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVI 16, pp. 139-156. Springer, 2020.
This paper presents work whose goal is to advance the field of machine learning. Specifically, this paper aims to have positive impact on the robustness of machine learning mod- els. Our test-time approach may make models more reliable and frustworthy on
Coates, A., Ng, A., and Lee, H. An analysis of single- layer networks in unsupervised feature learning, In Pro- ceedings of the fourteenth international conference artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
transformations,
References
Cohen, T. and Welling, M. Group equivariant convolutional networks. In International conference on machine learn- ing, pp. 2990-2999. PMLR, 2016.
Allingham, J.. Mlodozeniec, B., Padhy, S., Antorán, J., Krueger, D., Turner, R., Nalismick, E., and Hernández- Lobato, J. M. A generative model of symmetry trans- formations. Advances in Neural Information Processing Systems, 37:91091-999 1110099111000 2024.
Cohen, S., Geiger, M., and Weiler, M. A general theory of equivariant cnns on homogeneous spaces. In Advances in Neural Information Processing Systems, pp. 9142-9153, 2019.
Barron, J. T. and Tsai, Y.-T. Fast fourier color constancy. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 886 894, 2017.
Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L.., Ehsani, K., Kembhavi, A., and Farhadi, A. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 13142- 13153, 2023.
Benton, G., Finzi, M., Izmailov, P., and Wilson, A. G. Learn- ing invariances in neural networks from training data. Advances in neural information processing systems, 33: 17605-17616, 2020.
Deng, C., Litany, O., Duan, Y., Poulenard, A., Tagliasacchi, A., and Guibas, L. J. Vector neurons: A general frame- work for so -3)eeqivarr networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1 12200-12209, 2021.
Blender Foundation. Blender A 3D Modeling and Ren- dering Package. Blender Foundation, 2022. URL https:/ / /wwwwww.... www.blenderr. org. Version 3.2.2.
Boominathan, L., Srinivas. S., and Babu, R. V. Compen- sating for large in-plane rotations in natural images. In Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing, pp. 1-8, 2016.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255.. Ieee. 2009.
Bouchacourt, D., Ibrahim, M., and Morcos, A. Grounding inductive biases in natural images: invariance stems from variations in data. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 19566-19579. Curran Associates, Inc., 2021. URL https: /ptp/eeddcceee.....eee/// neurips.cc/pen paarrpps.c//papee/2222//111 22ff800887776778820044440000 pdf.
Dong, Y., Ruan, S., Su, H., Kang, C., Wei, X., and Zhu, J. Viewfool: Evaluating the robustness of visual recog- nition to adversarial viewpoints. Advances in Neural Information Processing Systems, 35:36789-36800, 2022.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Paper.
Bronstein, M. M., Bruna, J., Cohen, T., and Veličković, P. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021. URL https://arxii org/abs/ 2204 13478.
Fan, L., Huang, Y., Ge, Y., Li, W., and Duan, L. Beyond viewpoint: Robust 3d object recognition under arbitrary views through joint multi-part representation. In Eu- ropean Conference on Computer Vision, pp. 291-309. Springer, 2024.
Castro, E., Godavarthi, A., Rubinfien, J., Givechian, K., Bhaskar, D., and Krishnaswamy, S. Transformer-based protein generation with regularized latent space optimiza- tion. Nature Machine Intelligence, 4(10):840-85)) 2022.
Frazier, P. I. A tutorial on bayesian optimization. arXiv preprint arxiv: 2018.
Chen, X., Dong, Z., Song, J., Geiger, A., and Hilliges, O. Category level object pose estimation via neural analysis- by-synthesis. In Computer Vision-ECCV 2020: 16th
Fukushima, K. Neocognitron: A hierarchical neural network capable of visual pattern recognition. Neural networks, 1 12):119-130, 1988.
10

---

Test-Time Canonicalization by Foundation Models for Robust Perception
Geiger, A., Lenz, P., and Urtasun, R. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp. 1354-33611 IEEE, 2012.
Ilharco, G., Wortsman, M., Wightman, R., Gordon, C., Car-
lini, N., Taori, R., Dave, A., Shankar, V., Namkoong. H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L. Openclip, July 2021. URL https: /doii org/ 10.
5281 zenodo. 51 43773. If you use this software, please cite it as below.
Gómez-Bombarelli, R., Wei, J. N., Duvenaud, D., Hernández-Lobato, J. M., Sánchez-Lengeling B., She- berla, D., Aguilera-Iparraguinn J.. Hirzel, T. D., Adams, R. P., and Aspuru-Guzia sign using data-driven continuous representation of molecules. ACS central science, 4(2):268-277) 2018.
Jaderberg, M., Simonyan, K., Zisserman, A., and kavukcuoglu, k. Spatial transformer networks. In Ad- vances Neural Information Processing Systems, vol- ume 28, 2015.
Automatic chemical de-
Graham, M. S., Pinaya, W. H., Tudosiu, P.-.,, Nachev, P., Ourselin, S., and Cardoso, J. Denoising diffusion models for out-of-distribution detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2948-2957, 2023.
Jones, D. R., Schonlau, M., and Welch, W. J. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13:455-4925 1998.
Kaba, S.-O., Mondal, A. K., Zhang, Y., Bengio, Y., and Ra- vanbakhsh, S. Equivariance with learned canonicalization functions. In NeurlPS 2022 Workshop on Symmetry and Geomelry in Neural Representations, 2022. URL https:
Graikos, A., Malkin, N., Jojic, N., and Samaras, D. Diffu- sion models as plug-and-play priors. Advances in Neural Information Processing Systems, 35:14715-14723 2022.
/ //openreview.o net forum
Grathwohl, W., Wang. K.-C., Jacobsen, J.-H., Duvenaud, D., Norouzi, M., and Swersky, K. Your classifier is secretly an energy based model and you should treat it like one. International Conference on Learning Representations, 2020.
Kanezaki, A., Matsushita, Y., and Nishida, Y. Rotationnet:
Joint object categorization and pose estimation using mul- tiviews from unsupervised viewpoints. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5010-5019, 2018.
Gupta, A., Dollar, P., and Girshick, R. Lvis: dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5356-5364, 2019.
Kerbl, B., Kopanas, G., Leimkühler, T., and Drettakis, G. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1) 1,23,
Hamdi, A., Giancola, S., and Ghanem, B. Mvtn: Multi- view transformation network for 3d shape recogmition. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 1-11, 2021.
Kirichenko, P., Ibrahim, M., Balestriero, R., Bouchacourt, D., Vedantam, S. R., Firooz, H., and Wilson, A. G. Un- derstanding the detrimental class-level effects of data aug- mentation. Advances in Neural Information Processing Systems, 36, 2024.
Hc, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Patern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pp. 770-778, 2016. doi: 10.1109/CVPR.20) 2016.90. URL https:// doi. org/ 10. 1109/1199/00 1109/CVPR. 20.6.90.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C.,, Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., al. Segment anything. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 4015 2023.
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and nut-of-distributian examples in neural networks. International Conference on Learning Repre- sentations, 2017.
Kondor, R. and Trivedi, S. On the generalization of equivari- ance and convolution in neural networks to the action of compact groups. In International Conference on Machine Learning, ICML, 2018.
Hemnandez-Juarez, D., Parisot, S., Busam, B., Leonardis, A., Slabaugh, G., and McDonagh, S. A poothss approach to color constancy. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2270-2280, 2020.
KRIES, V. Influence of adaptation on the effects produced by luminous stimuli. handbuch der Physiologie des Men- schen., 3:109-282, 1905.
Hock, H. S. and Tromley, C. L.. Mental rotation and per- ceptual uprightness. Perception & Psychophysics, 24(6): 529-533, 1978.
Krizhevsky, A., Hinton, G., et al. Convolutional deep belief networks on cifar-10. Unpublished manuscript, 40(7): 1-9, 2010.

---

Test-Time Canonicalization by Foundation Models for Robust Perception
LeCun, Y., Haffner, P., Bottou, L., and Bengio, Y. Ob- ject recognition with gradient-based learning. In Shape, contour and grouping in computer vision, pp. 319-345. Springer, 1999.
Mondal, A. K., Panigrahi, S. S., Kaba, O., Mudumba, S. R., and Ravanbakhsh, S. Equivariant adaptation of large pretrained models. Advances in Neural Information Pro- cessing Systems, 36:50293-5000005 2023.
Lee, K., Lee, K., Lee, H., and Shin, J. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information pro- cessing systems, 31, 2018.
Nogueira, F. Bayesian Optimization: Open source constrained global
optimization
tool for Python, 2014.
URL https://
sat.aaa.con/aaeeelanoooinniiiiio Bayesiandptimmiztiion
Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang, H., Zhang, P., Vajda, P., and Marculescu, D. semantic segmentation with mask adapted clip. Pro- ceedings of the IEEF/CVF Conference Computer Vi- sion and Pattern Recognition, pp. 7061-7070, 2023.
OpenAI. Gpt-4o mini api, 2025. URL https:// piatform. openai com/
www/docs/cccccoccoooo/c//cccc.cc docs soaddelsspppttt Accessed: 2025-05-21.
Radford, A., Kim, J. W., Hallacy, C.,, Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 9748-8763. PMLR, 2021.
Liang, S., Li, Y., and Srikant, R. Enhancing the reliability of nut-of-distribution image detection in neural networks. International Conference on Learning Representations, 2018.
Reizenstein, J.. Shapovalov, R., Henzler, P., Sbordone, L., Labatut, P., and Novotny, D. Common objects in 3d: Large-scale learning and evaluation of real-life 3d cat- egory reconstruction. In International Conference on Computer Vision, 2021.
Lin, T.-Y., Maire, M., Belongie. S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. I.. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models, 2021.
Lin, Z.-H., Liu, B., Chen, Y.-T., Forsyth, D., Huang, J.-B., Bhattad, A., and Wang, S. Urbanir: Large-scale urban scene inverse rendering from a single video. In Synthetic Data for Computer Vision Workshop @ CVPR 2024, 2024.
Ruan, S., Dong, Y., Su, H., Peng, J., Chen, N., and Wei, X. Towards viewpoint-invariant visual recognition via adversarial training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4709- 4719, 2023.
Liu, W., Wang, X., Owens, J., and Li, Y. Energy-based out- of-distribution detection. Advances in neural information processing systems, 33:21464 21475, 2020.
Ruan, S., Dong. Y., Liu, H., Huang, Y., Su, H., and Wei, X. Omniview-tuning: Boosting viewpoint invariance of vision-language pre-training models. arXiv preprint arXiv:2404. 12139, 2024.
Ma, W., Zhang, G., Liu, Q. Zeng, G., Kortylewski, A., Liu, Y., and Yuille, A. Imagenet3d: Towards general-purposs purpose object-level 3d understanding. Advances in Neural Infor- mation Processing Systems, 37:96127-961 96149,, 2024.
Schuhmann, C., Beaumont, R., Vencu, R. Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35: 25278-255 25294, 2022.
Madan, S., Li, Y., Zhang, M., Pfister, H., and Kreiman, G. Improving -nttt--disttiuttion generalization by mimick- ing the human visual diet. In The First Workshop NeuroAI@ NeurIPS2024, 2024.
Maus, N., Jones, H., Moore, J., Kusner, M. J., Bradshaw, J., and Gardner, J. Local latent space bayesian optimization over structured inputs. Advances in neural information processing systems, 55:34505-34518, 2022.
Shepard, R. N. and Metzler, J. Mental rotation of three- dimensional objects. Science, 171(3972):01-7703 1971,
Singhal, U., Esteves, C., Makadia, A., and Yu, S. X. Learn- ing to transform for generalizable instance-wise invari- ance. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 1221,, October 2023.
Miao, N., Rainforth, T., Mathieu, E., Dubois, Y., Teh, Y. W., Foster, A., and Kim, H. Instance-specin aug- mentation: Capturing local invariances, 2022. URL wtgg/bs/////oo///////gg///// https://arxee org/ abs /206 00051.
12

---

Test-Time Canonicalization by Foundation Models for Robust Perception
Snell, C., Lee, Xu, K., and Kumar, A. Scaling IIm test- time compute optimally can be more effective than scal- ing model parameters. arXiv preprint arXiv:2408.0311, 2024.
Su, H., Maji, S., Kalogerakis, E., and Leamnd--Mlller, E. Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pp. 945-953, 2015.
Tarr, M. J. and Pinker, S. Mental rotation and orientation- dependence in shape recognition. Cognitive psychology, 21(2):233-28) 1989.
Tripp, A., Daxberger, E., and Hernández-Lobato, M. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. Advances in Neural Information Processing Systems, 33:11 11272, 2020.
Wei, X., Yu, R., and Sun, View-gen: View-based graph convolutional network for 3d shape analysis. In Proceed- ings of the IEEE/CVF Conference Computer Vision and Pultern Recognition, pp. 1850-1859, 2020.
Wei, X., Yu, R., and Sun, J. Learning view-based graph convolutional network for multi-view 3d shape analy- sis. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(6):7525-7541, 2022.
Xiang. Lv, Z., Xu, S., Deng, Y., Wang, R., Zhang, B., Chen, D., Tong, X., and Yang. J. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:2412.01500 2024.
Xiao, T., Wang, X., Efros, A. A., and Darrell, What should not be contrastive in contrastive learning. International Conference Learning Representations, 2021. URL https:://// http://pppeeeeeeeeeeeeeeeeeeeeee id=C2BY3NzuV20,
Yang, Z. and Wang, L. Learning relationships for multi-view 3d object recognition. In Proceedings of the IEEE/CVF international conference computer vision, pp. 7505- 7514, 2019.
Zaremba, W.. Nitishinskaya, E., Barak, B., Lin, S., Toyer, S., Yu, Y., Dias, R., Wallace, E., Xiao, K., Heidecke, J.. and Glaese, A. Trading inference-time compute for adversarial robustness, 2025. URL httph:t/////// //axii. org/aba /ttpaab../.............
Zhou, A., Tajwar, F., Robey, A., Knowles, T., Pappas, G. J., Hassani, H., and Finn, C. Do deep networks transfer invariances across classes? In International Conference on Learning Representations, 2022.
13

---

Test-Time Canonicalization by Foundation Models for Robust Perception
Appendix Summary
This appendix provides extended results, analyses, and implementation details supporting the main paper. The content orgamized as follows:
Additional Results and Figures
3D CO3D examples: Shows examples canonicalized frames in CO3D (Figure 11)
Additional results showing while TRELLIS can canonicalize to extent on inputs near its training distribution, FOCAL still outperforms it (Figure 12, Table Table 2)
3D Results:
2D Classification and Pose Estimation: Demonstrates improved accuracy and pose estimation across CI-
FAR10/100/STLIO (Figure 13, Table 3)
Segmentation Results: Matches PRLC's COCO mAP while achieving +2.1% pose accuracy (Table 4)
ImageNei Generalization: Outperforms PRLC by up to +1 1..4% on rotated inputs (Table 5)
Cross-Dataset Analysis: Shows < 3% variance in pose error PRLC's s 12-18% drops (Figure 14)
Ablations: Validates energy function design and compares against TTA (Table 6, Table 7).
DINOv2 Contrast results: Evaluating FOCAL with DINOv2 shows similar trends to CLIP (Figure 16)
- Color Correction Results: Shows direct comparison against stronger supervised color correction baselines (Appendix A.9) Computational Complexity Analysis: Discussion on FLOPs and Runtime (Table 8)
Experimental Setup
3D protocols with Co3DiObjaverse-LLE
Color transformation formalization
Bayesian hyperparameter optimization details
A. Additional Results and Figures
A.1. CO3D Examples
Figure 11, we show some examples of CO3D (Reizenstein al., 2021) frames being corrected through FOCAL, comparing the original input views that were incorrectly classified by OV-Seg (Liang et al., 2023) and their canonicalized views that are correctly classified by OV-Seg.
Original
Original
Canonicalized
Canonicalized
0.3
0.5
Figure 11. Viewpoint canonicalization improves accuracy by improving viewpoint robustness CO3D: Example renderings from filtered CO3D hefore (original) and after (canonicalized) applying FOCAL with an original ground-truth probability threshold of 0.3 and 0.5. The canonicalized views provide more informative perspectives for object recognition.
14

---

Test-Time Canonicalization by Foundation Models for Robust Perception
Filtered Objaverse-LVIS
1.0
0.8
Accuracy
0.6
0.4
0.2
Finetuned CLIP
Trellis
Ours
0.0
20
40
60
80
GT Probability
Figure 12. Accuracy over GT probability. For each object in Objaverse, we take random render and evaluate FOCAL vs. OV-Seg and TRELLIS baseline along bins GT label probability. We evaluate FOCAL with pretilts at intervals degrees. We find that FOCAL still consistently outperforms the TRELLIS baseline.
Threshold OV-Seg (Finetuned CLIP)
Random Rotation
TRELLIS
Ours
0.3
5.71%
38.8%
45.9%
49.5% (+3.6%)
0.5
28.5%
46.6%
53.4%
55.3% (+1.9%)
Table 1. Comparison of FOCAL against TRELLIS as a canonicalizer and random rotations. Accuracy is reported for each setting on quality-filtered CO3D frames, where for each video, we randomly sample frame with ground truth label probability below two different difficulty thresholds (0.3, 0.5) and filter as described in Appendix B.1 for image and segmentation quality. While surprisingly, random rotation and TRELLIS perform reasonably well, FOCAL outperforms both. One possible explanation that CO3D primarly contains horizontal orbits from above the object, limiting tilt variation in its viewpoints.
A.2. 3D Results
We also found that TRELLIS (Xiang et al., 2024) itself can used as a canonicalizer in limited settings, where the input images are near its training distribution. TRELLIS was trained on Objaverse (Deitke al., 2023) views with varying elevation and azimuth (no tilt) and implicitly uprights objects when creating the 3D assets it generates. We can then select particular view and stick with it everytime, creating new point of comparison with TRELLIS. We found that 30 degrees elevation and 150 degrees azimuth works the best.
To better understand FOCAL'' performance vs. TRELLIS (Xiang et al., 2024), we take each filtered Objaverse-L (Deitke et al., 2023) object we evaluated in Section 3 and select a random view. We then randomly tilt the image by a random 45 degree angle to ensure some distributional shift from TRELLIS' training data. To adapt FOCAL to handle the additional tilt axis, we perform 2D canonicalization in Section and then attempt angles 45 degree intervals centered the selected 2D canonical angle, and take the minimum energy viewpoint of good images tend to be at elevation 30, we explore 12 azimuths an interval of 30 degrees that this elevation.
all the examined images. Finding that majority
We plot the accuracy of the three methods (plain OV-Seg (Liang et al., 2023), TRELLIS, and FOCAL) binned by the GT probability the input view in Figure 12. We find that our approach consistently outperforms TRELLIS at all GT probability bins. Like Figure 5, FOCAL outperforms OV-Seg on bad viewpoints with some decrease on good viewpoints.
Next, we further amalyze our results on CO3D (Reizenstein al., 2021), including using TRELLIS (Xiang al., 2024). We also compare the performance against selecting randomly rotated (in 2D) image. The results the filtered and thresholded data splits from Section 3 are shown in Table 1.
We find that FOCAL outperforms random rotations and TRELLIS as a canonicalizer, but random rotations and TRELLIS still provide surprisingly strong performance. FOCAL provides a 3.6% gain over TRELLIS on the 0.3 threshold and a 1 1.9% gain over TRELLIS on the 0.5 threshold. One possible explanation for this that CO3D primarily contains horizontal orbits from above the object, limiting variation in its viewpoints. Because of this, there also likely less tilt variation, keeping the
15

---

Test-Time Canonicalization by Foundation Models for Robust Perception
CLIP Accuracy Cs Rotsted CIFAR1O, CIFAR100, and STL10
Accuracy C8 Rotations
100
100
PRLC VIT
Aceuracy
80
PRLC
90-
Ouni
60
40
CIF AR10
CIFAR100
STL10
Accuracy
70-
Pose Brror on Rotations
Error [in degreess)
THE
40-
50-
Orucle
PRLC Upright
Ours Upright
40
Vauilla Upright
PRLC Rotated
Ours Rotated
Vanilla Rotated
CIFÁR
CIFARI00
STL10
30
CIF ARIO
CIFAR100
ST
Figure 13. FOCAL beats PRLC dataset specialist canonicalizers rotated accuracy and pose estimation. (a) We show that FOCAL achieves better pose accuracy than PRIC on their datasets. (b) We demonstrate that FOCAI outperforms PRIC dataset specialist models terms pose error. (c) result of our superior pose estimation, we generalize better to new models like CLIP. Dashed lines represent oracle performance, i.c., perfectly undoing the rotation except for any loss of corner information from cropping. This result highlights FOCAL's strong canonicalization ability.
poses to in an-distribution poses. As for random rotation, depending on the sharpness of the angle above 2D rotations may effectively mimic moving around the orbit. Thresholding objects for hardness may also lead towards "adversarial" examples that lie in an extreme local extrema. We leave further investigation to future work.
Finally, we compare test-time augmentation (TTA) against FOCAL on 3D viewpoints. We compare against TTA strategy that averages over 1, 5, and 10 random transformations out of the 60 generated TRELLIS (Xiang et al., 2024) renders. To match FOCAL, we also blur the logits before applying TTA (Appendix B.1). We also compare FOCAL with a variant of FOCAL that averages the logits over the top 5 (and top 10) ranked viewpoints. We evaluate on Objaverse-L.VIS (Deitke al., 2023) and report the results in Table We find that FOCAL with averaging over the top 10 ranks performs the best on average 84.5% best view performance of 93.4%, and worst view performance of 71.3%, where best and worst view defined as the first and last ranks of the quality-filterrs Objaverse-LVV dataset in Figure 5. FOCAL performs especially well on the best view. TTA also performs well, achieving some good performance on the worst view (although less than ours at equivalent numbers of views), perhaps owing to the nature of 3D consisting of more viable in-distribution viewpoints than a setting like 2D rotation.
Method # Views
Mean
Best View
Worst View
Ours
79.5%
93.3%
61.6%
Ours
84.1 1%
93.9%
70.4
Ours
10
84.5%
93.4%
71.3%
TTA
67.8%
75.3%
56.2%
TTA
75.2%
83.9% 85.1%
62.6%
TTA
10
76.4%
63.7%
Table Comparison of FOCAL vs. TTA when averaging over multiple views. For the same number of views, our method achicves strictly better accuracy in each category (best, worst, mean).
A.3. 2D Classification and Pose Estimation
FOCAL demonstrates superior canonicalization capabilities across three key 2D benchmarks. On CIFAR10/100 (Krizhevsky et al., 2010) and STL10 (Coates et al., 2011), our method achieves consistent improvements over PRLC's (Mondal et al., 2023) dataset-specific canonicalizers: +0.3-2.3% upright accuracy gains (Table 3), +0.4-2. 1% robustness random C8
16

---

Test-Time Canonicalization by Foundation Models for Robust Perception
ResNet50
VIT
Pretrained Network
(PRLC-Trained)
PRLC-Traiine
Datasets
Canoncalizer
Accuracy
Random Rotation (C8)
Accuracy
Random Rotation (C8)
Rotation Aug.
94.9
90.1
96.3
93.7
PRLC
96.1
95.1
95.8
94.8
CIFARIO
Ours
96.5 (+0.4%)
95.6 (+0.5%)
97.4 (+1.1 %)
96.0 (+1.2%)
Oracle
96.6
95.9
97.6
96.7
Rotation Aug. 80.2
74.1
82.6
78.4
PRLC 83.1
81.8
83.9
82.2
CIFAR100
Ours
83.8 (+0.7%)
82.2 (+0.4%)
86.3 (+2.4%)
84.4 (+2.2%)
Oracle
84.4
83.4
87.1
85.5
Rotation Aug. 98.1
95.0
97.9
94.1
PRLC 95.2
94.1
95.7
93.9
STL10
Ours 96.2 (-1.9%)
95.6 (+0.6%)
96.0
(-1.9%)
95.2
(+1.1 %)
Oracle
97.4
96.7
97.3
96.4
Table 3. FOCAL beats PRLC dataset specialist canonicalizers and models rotated accuracy. We find that FOCAL outperforms PRLC, without any training, across all PRLC specific model and dataset pairs on both upright inputs and randomly rotated inputs. We compare against just upright images in the Accuracy columns. Oracic refers system where the exact angle to upright known, and thus only measures the change accuracy due to loss of information due to rotating, cropping. and re-rotating. Random Rotation (C8) applies random C8 transform to the input before passing it the aligner / model. Best non-oracle numbers taken from (Mondal et al., 2023), and the rest are reproduced using their provided code and default hyperparameters. This result highlights that beat PRLC
bolded. Rotation Augmentation
their best settings.
Pretrained Network
SAM
Dataset
Canonizalizer
mAP Random Rotation (C4) (%)
Pose Accuracy (%)
Naive
62.1
PRLC
65.9
86.8
COCO
Ours
65.9
88.9 (+2.1 %)
Oracle
66.3
Table FOCAL matches PRLC on segmentation: We first report FOCAL's performance on mAP on COCO, PRLC without any training (left). then show that C4 pose accuracy is higher than PRLC by 2.1%. This shows FOCAL's ability to generalize segmentation without supervision. All numbers are reproduced using Mondal et al. (2023)'s pre-trained checkpoint.
rotations (Table 3), lower pose error and higher pose accuracy (Figure 13).
Notably, FOCAL approaches oracle performance gaps within 0.9-2. % across all metrics, suggesting our energy minimization framework effectively approximates ideal canonicalization despite unknown rotation angles. Our method achieves this without any supervision or dataset/rask-speeifee training.
A.4. Segmentation Results
FOCAL'S geometric canonicalization transfers seamlessly to segmentation tasks without vegmentation-specinn training. On COCO (Lin et al., 2014), FOCAL: (1) Matches PRIC's 65.9 mAP despite PRLC being trained specifically on segmentation data whereas FOCAL is zero-shot; (2) Achieves 88.9% pose accuracy (+2.1 1 % over PRLC); (3) Nears oracle mAP (66.3 vs 66.9). This demonstrates that our approach learns fundamental viewpoint normalization rather than task-specific artifacts. The pose accuracy gains directly translate to better segmentation consistency across rotations, as evidenced by the mAP parity despite PRLC's segmentation-awee -ware training (Table 4).
17

---

Test-Time Canonicalization by Foundation Models for Robust Perception
ResNet50
VIT
CLIP Vannila-Trrinned
ImageNet
(Vanilla-Trained)
(raiila--Traned)
Random Rot.
Random Rot.
Random Rot. (C8)
Canonicalizer
Accuracy
Accuracy
Accuracy
(C8)
(C8)
None
73.3
49.0
79.4
58.8
77.1
67.0
PRLC*
63.1
59.2
63.7
60.5
72.1
69.6
Ours
64.4 (+1.1)
62.4 (+3.2)
72.7
(+9.0)
71.2 (+11.4)
75.2 (+3.1)
73.9
(+4.3)
Oracle
73.3
70.8
79.4
77.4
77.1
75.3
Table FoCAL generalizes better ImageNet outperforms PRLC's canonicalizers. find that FOCAL outperforms PRLC on ImageNet, without any training, on both upright inputs and randomly rotated inputs. We compare against just upright images in the Accuracy columns. Oracle refers a system where the exact angle to upright is known, and thus only measures the change accuracy due to loss of information due to rotating, cropping, and re-rotating. Random Rot. (C8) applies random C8 transform to the input before passing the aligner/ model. Best non-oracle rows on rotated performance are bolded. For PRIC, the canonicalizers were the best performing ones from other datasets (STL10 for both ResNet50 and VIT). Thus, they were trained specifically for ImageNet.
A.5. ImageNet Generalization
FOCAL shows remarkable cross-dataset generalization on ImageNet (Deng et al., 2009): +11.4% rotated accuracy for ViT (Dosovitskiy et al., 2021) vs PRLC (Mondal et al., 2023) (71.9% vs 60.5%); +4.3% absolute gain for ResNet50 (He al., 2016) under rotation; CLIP (Radford et al., 2021) performance improves +4.4% under rotation.
Notably, PRLC canonicalizers trained on small datasets (STLI0CIFAR) degrade significantly ImageNet (-16.7% ResNet50 upright accuracy ours), failing generalize outside their training setting. FOCAL, however, maintains strong performance through its dataset-agnostic energy formulation.
A.6. Cross-Datasee and Model Analysis
FOCAL's 's unified framework enables superior cross-domain transfer compared to PRLC's (Mondal al., 2023) specialized canonicalizers. When transferring across datasets, PRLC suffers 12- 18% drops in pose accuracy (Figure 14c), while our method maintains < 3% variance in pose error. Transferring to CLIP (Radford et al., 2021) yields particularly strong results, with +15% relative accuracy gains over PRLC (Figure 15). These results underscore a key advantage: by avoiding dataset- specific training, FOCAL develops general canonicalization strategies that transfer seamlessly across both architectures (ResSNe500ViT//CLII)) and data distributions (CIFAR/STL.10/magee))
Pose Accuracy VII
Pose Error VI
Powe Accuracy R50
Pose Ervor R50
1200
69.4
57.6
46.6
6300
37.5
51.0
59.2
72.5
66.0
48.9
36.7
46.2
56.8
c200
51.3
51.3
33.9
1:100
59.1
56.5
69.9
C200
54.2
58.8
36.1
58.00
52.2
67.7
$1a.0:
S1L30
$11,00
CtTEE
21.4
20.3
54.6
72.1
73.8
43.0
47.3
37.0
59.8
50.6
59.3
39.2
C10 Åltgn. C100 Altzz STL Align.
C10 Align. C100 Allgm STL
Aliza
C10 Alizz. C100 Alig. STL. Alig.
C1O Alig. C100 Aligo.SI1. Allgm
(a) VIT Accuracy (1)
(b) VIT Error (L)
(c) R50 Accuracy (1)
(d) R50 Error (()
Figure FOCAL generalizes better across datasets mixing up aligners downstream models. PRLC performance on pose estimation drops significantly when using canonicalizer trained from different dataset compared to FOCAL, which applies one technique across all settings. This result highlights generalizability across datasets of an unsupervised approach.
A.7. Ablation Studies
Component ablations validate critical design choices in our energy formulation. Combining CLIP and diffusion energies reduces pose error by 64% compared to CLIP alone (13.5 vs 37.19) (Table 6), while test-time augmentation (TTA) underperforms by 10-14% on CIFAR benchmarks (Table 7). These experiments confirm that naive augmentation cannot substitute learned canonicalization and that multi-energy fusion provides complementary benefits.
18

---

Test-Time Canonicalization by Foundation Models for Robust Perception
Acc. va. Angle, CIFARIO CLIP
Acc. vs. Angle, CIFARIO0 CLIP
Acc. va. Angle. STL CLIP
1.0
1.00
0.8
0.8-
0.95
Accurcy
Accuracy
Aceuray
Naive
Naive
Naive
0.6
0.6
0.90
PRLC
PRLC
Ours
Ours
Ours
0.4
Oracle
Oracle
0.85
Oracle
180-135-90
90 135 180
-180-135
90 -15
90
-800
135-03
-15
90
135 180
Augle
Angle
Augle
CIFARIO
(b) CIFARIO0
(c) STLIO
Figure 15. Accuracy vs. C8 angle on CLIP. Like on ResNet50, we find that using FOCAL leads to invariant predictions over angles, outperforming PRLC. The contrast particularly clear for CLIP on CIFARIO and CIFARIO0, where our accuracy over angle consistently above PRLC.
Method
Pose Accuracy
Avg. Pose Error (degrees)
Only CLIP 68.9%
37.1
Only diff
82.7%
22.6"
diff+clip
89.5%
13.5
Table Ablation study on energy function components pose estimation. The combination diffusion (diff) CLIP achieves the highest accuracy and lowest pose enror.
CIFAR10
CIFAR100
STL10
No uprighting
65.4
50.6
93.4
Ours
93.7
76.2
97.5
TTA
82.8 (-10.9)
61.7 (-14.5)
96.6 (-0.9)
Table Comparison of CLIP's accuracy across different datasets using No Uprighting, FOCAL, and Time Augmentation (TTA), Our method significantly outperforms TTA, especially for larger invariance ranges like CS.
A.8. Contrast Canonicalization for DINOv2
We also provide contrast results for FOCAL on DINOv2 in Figure 16 (similar to CLIP in the main paper Figure 7). It follows the same trend, achieving significant accuracy gains on the largest transformations.
Accuracy over contrast (inageNet-1K)
Accuracy (96)
80
Defult
(Nuiv)
70
Canonioalied (ouns
0.01
0.5
1.0
1.5
Distance in log-contrast space (binned)
Figure 16. Contrast results for DINOv2
A.9. Comparison against Color Correction baselines
We benchmark our method on the RCC dataset and compare it against Barron & Tsai (2017) and the gray world method. We search for the two log- corommimmeeeee values in the range [-0.7, -0.3], and used both classifier and diffusion energies with the default values for classifier energy, and coefficient -10 for diffusion energy, with the diffusion energy evaluated on
19

---

Test-Time Canonicalization by Foundation Models for Robust Perception
single step / 50. We used BO optimize the color using 10 random initial points and 50 BO iterations. Our method only produces visually typical (in-distribution) images, necessarily the most color-neutral version, and thus, performs poorly on this test compared to baselines designed to achieve high color accuracy this task (Figure 17): Specifically, we achieve a median angular error of 6.4" compared to Barron & Tsai (2017)'s s 1.3%, though we do beat gray-world's 9.97". Our method is nowhere near the specialized state-of-the-an methods on this task, but ours is a more general algorithm designed to improve invariance. We leave it to future work to improve the base performance of FOCAL against supervised color approaches.
Original Image
Ours
Ground Truth
Figure 17. Example image of color correction from the RCC Dataset. Our method (center) produces image with typical lighting, which is not necessarily the same as completely neutral white colors (right).
A.10. Computational Complexity Analysis
While our method primarily focuses on improving generalization through equivariant learning, we provide a detailed analysis of its computational costs. The overall complexity can be decomposed as:
Naive Cost Cinference
TTA Cost N X Cinference
Our Cost = N x (Crransform + Cenergy + Cinference)
where represents the number of evaluated transformations, Cransform denotes transformation cost, Cevalaation encompasses CLIP and diffusion forward passes, and Cinference the final prediction cost. In principle, inference only has to be done once (after uprighting), the Cinterence does not need to be multiplied by N. We use the current cost as upper bound to compare the cost more easily against TTA. We also assume that system-1/2 1/2 methods skip unnecessary canonicalization are not being used. This setting represents the worst-case scenario for our method from computational cost perspective.
Example calculation
For 2D rotation with N = 8 orientations and K 5 diffusion steps, and CLIP as classifier:
Standard inference
(CLIP)
TTA:
(Rotation CLIP)
Our method
(Rotation CLIP 5 x Diffusion)
Assuming image rotation uses negligible FLOPs, we get: 1x for naive, Rs 8x for TTA, RJ 56 for ours.
Latency reduction through parallel inference Despite increased FLOPs, our method is parallelizable since the energy functions can be evaluated independently on different points, and even different diffusion steps can be evaluated in parallel (for energy computation only). Thus, the theoretical latency under parallelism only:
Theoretical Latency (parallel) max (Classifier, Diffusion step, CLIP)
FLOPs and runtimes for our experiments Here, we detail the FLOPs and runtime costs for key experimental settings used this paper. We use the DeepSpeed profiler and note the FLOPS, total cost, and runtime/latency (unoptimized). All
20

---

Test-Time Canonicalization by Foundation Models for Robust Perception
experiments were done on an RTX 2080Ti GPU except 3D viewpoint, which was done on RTX 6000 Ada Generation GPU. FLOPs for 31D are omitted due compatibility issues with DeepSpeed and TRELLIS. 45% of the average 3D runtime was spent generating the 3D asset in TRELLIS.
Table 8. Per-Experiment approximate FLOP Costs and Runtime latency (unoptimized)
Experiment
#Transforms
Diffusion Steps
Baseline FLOPs (T)
TTA
FOCAL FLOPs (T)
Runtime
(N)
FLOPs (T)
(s/it)
2D Rotation (C10/C100/IN)
0
0.33
2.6
2.6
0.47
2D Rotation (COCO)
4
10
0.33
1.29
19.1
0.94
Color (C100/IN)
35
0
0.33
11.6
11.6
5.2
Color (RCC)
60
1
0.33
19.8
46.2
10.3
Contrast (C100/IN)
12
0
0.33
3.97
3.97
1.22
Day-Night
0
0.66
0.16
Active vision
500
49.5
238.1
3D Viewpoint
61
5
13.3
B. Experimental Setup
B.1. Experimental Setup- 3D 3DD
Dataset Details: The Objaverse-LVVe LVIS (Deitke et al., 2023) dataset contains 46207 3D assets labeled with one of 1156 categories from LVIS (Gupta et al., 2019). From these 3D assets, multiview images can then be generated with software such Blender (Blender Foundation, 2022). The CO3D dataset contains collection of 18619 real multiview video sequences of 51 MS-COCO (Lin et al., 2014) objects with each video recording the object in circular orbit, along with segmentation masks for each frame.
Rendering: For Objaverse-LVIS (Deitke et al., 2023), we generate our base input renders at viewpoints in the upper viewing hemisphere. We sample at an interval of 30 degrees and radius of 2.2. We offset the views by 10 degrees to avoid perfectly aligned views that eliminate critical 3D context of the object. This leads to 36 renders in total.
For Objaverse-LVVI (Deitke al., 2023) in the "vvay" stage sample 60 viewpoints. These
every 30 degrees of ele-
vation and azimuth, for grid of [-60, -30, 0, 30, 60] [-180, -150,- 120,, -90, -60, -30, 0, 30, 60, 90, 120, 150, 180]. For CO3D (Reizenstein et al., 2021), we sample the 12 viewpoints at the elevation 30 and azimuths of [-180, - 150, - 120, 90, -60, -30, 30, 60, 90, 120, 150, 180].
Filtering: Due to Objaverse-LVIS (Deitke et al., 2023) label quality concerns primarily stemming from the existence of multiple similar labels (e.g., orange vs. mandarin orange vs. tangerine, ring vs. wedding ring, etc.) we filter out any objects that: 1) had fewer than 10% of its renders classified correctly 2) lacked clear winner class that was predicted least 33% more than the 2nd most common class. This results in test set of 14789 objects.
For Objaverse-LVIS in Fig. 5, we evaluate FOCAL on the Oth, 12th, 24th, and 35th ranked viewpoints, which corresponds to the Oth, 33rd, 66th, and 100th percentile.
For CO3D (Reizenstein et al., 2021), we focus on videos which contain sufficient viewpoint variation to induce errors in classification. Specifically, we filter for videos that have at least one frame with a probability greater than 80% for the ground truth class and least one frame with a probability less than threshold We run with two thresholds: t 0.3 and t 0.5. We then pick random frame of probability less than for each video. For t = 0.3, this gives us 11157 frames, and for 0.5, this gives 12186 frames.
With these frames, we then filter out the frames for segmentation and image quality. To do this, we crop the objects based on their segmentation mask's 's bounding box, multiply the size of the crop by 1.2, and resize the crop to 518 x 518 (following the preprocessing of TRELLIS (Xiang et al., 2024). We then pass the crop and the cropped segmentation to 1ili--mmn--222--- (OpenAI, 2025) with the following prompt:
21

---

Test-Time Canonicalization by Foundation Models for Robust Perception
Prompt
You are evaluating an image (left) and its segmentation overlay (right)
Criteria:
1) There is a single, clearly visible object that fits completely in the frame.
2) The main object should not have any parts outside the frame. There should be clear margin all sides.
3) The main object should be easily distinguishable and not blurry.
4) The segmentation mask should accurately outline the main object.
all criteria are met, respond 'PASS' plus the main object. Otherwise, respond 'FAIL' plus a short reason. Your answer must start with 'PASS' or 'FAIL' and use fewer than 10 words.
This filtering results in 1504 frames (13.5%) for 0.3, and 1865 frames (15.3%) for 0.5.
Calculating Energy: For both Objaverse (Deitke al., 2023) and CO3D (Reizenstein al., 2021), we use 1, 0.5 following the 2D experiments (B.3). We also used the diffusion energy (steps 500 to 1000 with stride 100) with factor of 5, We also normalized the CLIP (Radford et al., 2021) energy with normalizing prompt of "a photo of an object on bright white backdrop." and temperature of 0.5 to adjust CLIP to the background removed images used in these experiments.
B.2. Experimental Setup Color and Contrast
We define the color shift transformation using the popular von Kries model (KRIES, 1905) where an illuminant vector with the RGB values L [LR, LG, La] E R is multiplied element-wise with every pixel in the image. We then generate this illuminant vector by sampling in the log-chrominance space (Barron Tsai, 2017). Specifically,
Iu Top ~ 1[
(6)
RR La, LB] = exp(- 1 exp(-L
where z = /exp(-LL.)= exp(- + normalizing constant and Lus Ly are the log-chroma values sampled from the uniform distribution with range [-1, 1]. Intuitively, the log-chroma space defines the R/G and B/G ratios in log-space. range of [-1, 1] corresponds roughly 7x change in the ratio between the minimum and maximum points of the range.
We define the contrast as gamma transformation x7, where the log of the gamma is sampled randomly at uniform log(>) ~ U[-2, 2). This means gamma lies between e -2 and e².
We optimize the energy functions using Bayesian optimization. For initialization, we use random as well as a grid of initial samples. Color uses a uniform grid points, and iterations.
random points, and 20 iterations. Contrast uses grid points, random
B.3. Hyperparameters for the energy functions
For experiments on ImageNet, CIFARIO, CIFARIO0, and STL10, we only used the classification energy for computational efficiency. We used a = 1, 3 0.5 for all these settings. In practice, a wide range of 33 works well. The same is true for active vision experiments, but the figures shown in the paper used 3 0.33.
For segmentation, used the diffusion energy (steps 50 to 150 with stride 10) with factor of 0.67 along with CLIP energy factor of 0.54 and 3 0.2.
All these hyperparameters were found using Bayesian Optimization with the same kernel and acquisition function mentioned in Section 2.4 and performed using the Bayesian Optimization Toolbox (Nogueira, 2014) for 300 time steps. Each energy hyperparameter was tuned on small training or validation set by recording logits and finding the combination of energy functions that maximized accuracy.
22


Reviews:
[{"review_id": "zb5cCCLuTV", "reviewer": "Reviewer_XwyL", "summary": "This paper introduces Foundation-model guided Canonicalization (FOCAL), a novel zero-shot framework designed to enhance the invariance of vision models to various transformations by test-time optimization. The method generates candidate transformations (e.g., 3D novel views) and selects a canonical version by optimizing an energy function derived from the foundation models like CLIP and Stable Diffusion. The authors demonstrate the effectiveness of FOCAL across 2D/3D rotations, illumination shifts, and day-night changes, showing improved robustness for models like CLIP and SAM without requiring any training or architectural modifications. The work also explores applications in active vision.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We are glad you found our use of foundation models for optimizing the energy function clever and we hope FOCAL can be a key step towards a zero-shot approach towards a wide range of visual invariance. We address your concerns and questions below:\n\n* **“‘The proposed approach is fully complementary to training-time approaches like DA.’ To make this claim, don’t they need to show that FOCAL + DA outperforms FOCAL alone? But I couldn’t find any such experiment.”**\n\n   - Good point. We find that FOCAL+DA indeed outperforms either FOCAL or DA by themselves, and will add these results to the main paper. As an example, we find that on a ResNet32 trained on CIFAR10, the C8 rotated accuracy is 72% with just DA, 72.1% with just FOCAL, and 73.2% with FOCAL+DA.\n\n* **“How fast does the proposed method run? The paper states that ‘evaluating the energy function for many candidates is computationally expensive,’ but I think providing a more concrete runtime would be more helpful for the audience.”**\n\n    - Good question. Our paper focuses on generalization rather than efficiency, but we agree that computational cost is important, and will include it in the paper. Our method’s cost is:\n\n       (# of transforms evaluated) X (Cost of transforming + Cost of evaluating [CLIP + Diffusion model] + Cost of inference).\n\n       **Example**: Consider 2D rotation (with 8 rotations around the circle). When using CLIP as the downstream model and 5 diffusion steps, we must evaluate the energy for each of the 8 rotations. Thus, our method requires roughly 56x more FLOPs compared to standard inferencing. As for latency, our procedure is highly parallelizable, and the latency can be close to standard inference latency.\n\n         As another point of comparison, consider a test-time-augmentation (TTA) strategy that just averages the outputs. Such a strategy would use 8x FLOPs than the naive classifier. Thus, in comparison to TTA, our approach is 7x more expensive in FLOPs.\n\n         We will add a more detailed runtime analysis in the camera-ready paper, and a table detailing the FLOPs and runtime for each experiment (in seconds).\n\n* **“Could you elaborate on potential strategies for improving the computational efficiency of FOCAL?”**\n\n     - **Only using canonicalization when necessary**: Similar to the “mental rotation” [1] phenomenon in humans, where we classify familiar poses quickly but go through a slow mental uprighting process to classify unfamiliar poses, one approach is to apply canonicalization selectively.\n\n         We have preliminary results for an approach that may significantly reduce amortized computational complexity by skipping the canonicalization when unnecessary.\n\n         For example, for 2D rotations, we can skip the canonicalization by comparing the image’s CLIP energy against +90 and -90 degree rotations, and thresholding the energy difference. We can use this simple classifier to detect upright vs. non-upright with 95% accuracy.\n         In summary, with only 3 CLIP inferences, we can detect whether to do canonicalization with 95% accuracy. If the extreme rotations are rare (as is typical in the real world), this can bring significant computational savings and, on average, be even cheaper than TTA.\n\n          [1] https://psycnet.apa.org/record/1971-28060-001\n\n\n     - **Fewer diffusion steps**: The majority of our computational cost comes from the diffusion energy, especially because it uses multiple steps. Diffusion classifier [2] has done an extensive analysis of efficient diffusion step schedules that can be used to rank inputs (in the context of classification). We didn’t explore these schedules since our focus was to show that CLIP and SD can be used for invariance to diverse and complex transformations, but leveraging efficient diffusion step schedules can be a promising future direction.\n\n         [2] https://arxiv.org/pdf/2303.16203\n\n     - **Distilling FOCAL energy into a cheaper (or one-shot) EBM**: Since the desired output is only a scalar energy value, it might be possible to distill CLIP and SD’s energy function into a smaller single-shot EBM like [3]. This is yet another promising future research direction.\n\n          [3]: https://implicit-pdf.github.io/", "claims_and_evidence": "- FOCAL improves robustness to various transformations in a zero-shot manner.\n\t- The experiments presented in Section 4 provide empirical support for this claim, where they show improved performance on viewpoint shifts, illumination changes, 2D rotations, and day-night transformations.\n- FOCAL outperforms or matches task-specific canonicalizers like PRLC in their trained settings, despite being zero-shot.\n\t- The 2D rotation experiments in Section 4.3 and Figure 7 support this claim. FOCAL matches or surpasses PRLC's performance on 2D rotation tasks and demonstrates better generalization on ImageNet.\n- The proposed approach is fully complementary to training-time approaches like DA.\n\t- To make this claim, don’t they need to show that FOCAL + DA outperforms FOCAL alone? But I couldn’t find any such experiment.", "methods_and_evaluation_criteria": "- The proposed method is well-motivated and clearly explained. The \"vary and rank\" scheme is intuitive. The energy functions derived from CLIP and Stable Diffusion seem appropriate.\n- The evaluation criteria are suitable for assessing the method's effectiveness. The choice of datasets (ImageNet, CIFAR, Objaverse-LVIS, CO3D) and transformations (viewpoint shifts, illumination changes, 2D rotations, day-night) is comprehensive and relevant.", "theoretical_claims": "- No proof provided.", "experimental_designs_or_analyses": "- The experiments described in Section 4 are comprehensive (e.g viewpoint invariance, Color and Contrast, 2D rotation, day-night transformation, active vision), and for each task, they use multiple datasets to verify the results. \n- Hyperparameters for the energy functions are explained in Section B.3.", "relation_to_broader_scientific_literature": "- The authors clearly discuss the limitations of data augmentation and equivariant networks, highlighting the challenges they face in open-ended scenarios. \n- The paper builds upon the theoretical foundations laid by Kaba et al. (2022) and leverages the energy-based model perspective from Grathwohl et al. (2019) and the use of diffusion models as priors from Graikos et al. (2022).", "essential_references_not_discussed": "- No major concerns here", "other_strengths_and_weaknesses": "- I think their idea of using foundation models to optimize the energy function for guiding canonicalization is clever, and the experiments demonstrate its strong performance compared to previous approaches.", "other_comments_or_suggestions": "- No major typos.", "questions_for_authors": "- How fast does the proposed method run? The paper states that \"evaluating the energy function for many candidates is computationally expensive,\" but I think providing a more concrete runtime would be more helpful for the audience.\n- On a related note, could you elaborate on potential strategies for improving the computational efficiency of FOCAL?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GgJXH0TZ62", "reviewer": "Reviewer_tdnV", "summary": "The paper introduces FOCAL, a zero-shot framework designed to achieve invariant perception at test-time using pre-trained foundation models like CLIP and Stable Diffusion. FOCAL generates transformed versions of input images and selects a canonical version by minimizing an energy function derived from these models, requiring no additional training or architectural changes.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We are glad you recognize our empirical strengths in canonicalizing images on a range of transformations, and we hope FOCAL can be a key step towards a zero-shot approach. We respond to your concerns and questions below:\n\n- **“The paper lacks an analytical explanation regarding why foundation models, despite their reduced direct classification accuracy under transformations, possess robust and accurate ranking abilities, even without finetuning on downstream tasks.”**\n\n     - That’s a good point, and we will add a more detailed explanation about this in section 4 as outlined below.\n\n     - Specifically, we assume: (1) there is at least one in-distribution image in the set of transformed images, and (2) the foundation models can be used as a prior where ID images have lower energy than OOD. This has been shown/used previously by [1, 2].\n\n       [1] https://arxiv.org/abs/2206.09012 \n\n       [2] https://arxiv.org/abs/1912.03263\n\n    - If these two assumptions hold for a given sample, the energy minimization scheme returns an image that is in-distribution. Importantly, this scheme only requires the foundation models to distinguish between ID vs OOD images. Even if the classification and energy values for OOD images are not well-behaved, our scheme still works as long as in-distribution images have lower energy than out-of-distribution images. \n\n       We acknowledge that the limits of foundation models to serve as image priors are still not fully understood. There may be transformations or OOD images for which assumption 2 may not hold. A rigorous understanding of exactly when foundation models work well as image priors is a great future direction.\n\n\n\n- **“While evaluating single transformations separately is understandable for methodological clarity, this raises concerns that employing Bayesian optimization to search through a more complex and realistic transformation space could become increasingly infeasible.”**\n\n    - This is a fair question. In general, handling multiple and complex transformations remains a difficult open problem. For our approach, we agree that more complex transformations (resulting in a higher-dimensional optimization problem) would likely lead to higher sample complexity, which would make the optimization more difficult.\n\n    - However, Bayesian optimization and gradient descent for latent space optimization have been successful in other mid-to-high dimensional optimization problems like protein design [3,4,5,6]. We are still figuring out how to best use these techniques for invariance, and we think it could be a great future research direction. We hope our paper serves as a strong foundation to explore this direction and hope to solve it in future work.  \n      [3]: https://arxiv.org/abs/2006.09191  \n      [4]: https://arxiv.org/abs/2201.11872  \n\n      [5]: https://arxiv.org/abs/1610.02415  \n\n      [6]: https://www.nature.com/articles/s42256-022-00532-1\n\n- **“Additionally, Figure 5 shows that finetuned CLIP performs better for samples with high viewpoint rankings (easy samples). How can we determine, given a specific sample image, whether naive finetuned CLIP or FoCAL would yield better performance?”**\n\n    - Great question! Usually, the more in-distribution/upright an image is, the more likely it is that finetuned CLIP will do better. As a heuristic: if the given image is in a local minimum of the energy function, FOCAL will likely not be helpful. We used this to create a decision rule for 2D rotations that allows us to skip canonicalization where it is unnecessary with 95% accuracy (see “only using canonicalization when necessary” in our response to Xwyl above). We will add these results to the appendix.\n\n- **“Could the authors clarify under which circumstances one energy term (diffusion or CLIP) is more critical than the other, and what specific roles each model plays in different canonicalization scenarios?”**\n\n    - Also a great question. We don’t have a general theoretical answer, but we have run some ablations (e.g., Table 4 in the appendix). In our experience, CLIP is more important when there is a clearly defined object in the image with an easily defined caption (e.g., Objaverse classification). Diffusion helps much more for scenes with many objects (e.g., in segmentation) and visual structures (e.g., edges). We currently find the best combination via hyperparameter optimization, but hope to have a more general approach in future work.", "claims_and_evidence": "The authors claim that vision foundation models, such as CLIP, perform poorly on direct classification tasks when faced with viewpoint, color, or lighting changes. But they exhibit strong performance in ranking (**the ability to select the most plausible (canonical) image from among multiple candidate images** via energy-based forms) multiple transformed images by their probability of belonging to the dataset distribution. They propose leveraging this capability to canonicalize images, improving classification accuracy.\n\nHowever, the paper lacks an analytical explanation regarding why foundation models, despite their reduced direct classification accuracy under transformations, possess robust and accurate ranking abilities, even without finetuning on downstream tasks. For instance, Figure 5 suggests CLIP inherently identifies canonical viewpoints even without task-specific fine-tuning, outperforming a fine-tuned version in terms of viewpoint robustness. The paper provides strong experimental results but does not sufficiently explore or explain the underlying mechanisms or reasons behind this key capability.", "methods_and_evaluation_criteria": "As acknowledged by the authors, estimating the canonical form for every image involves multiple transformations, necessitating the use of Gaussian processes for efficient search. Despite using GP, the proposed approach still likely demands significantly more computational resources. Furthermore, in realistic scenarios, images typically undergo multiple combined transformations rather than isolated ones (e.g., viewpoint + color + contrast + active vision). While evaluating single transformations separately is understandable for methodological clarity, this raises concerns that employing Bayesian optimization to search through a more complex and realistic transformation space could become increasingly infeasible.\n\nAdditionally, Figure 5 shows that finetuned CLIP performs better for samples with high viewpoint rankings (easy samples). How can we determine, given a specific sample image, whether naive finetuned CLIP or FoCAL would yield better performance?", "theoretical_claims": "There are no theoretical claims to discuss.", "experimental_designs_or_analyses": "What is the inference time of the proposed method compared to baselines such as finetuned CLIP, TTA, and PRLC?", "relation_to_broader_scientific_literature": "Proposed model could be used in the future to enhance the test-time performance of foundation models.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "In section B.3, the weighting between the diffusion and CLIP energy terms is currently set via hyperparameters. Could the authors clarify under which circumstances one energy term (diffusion or CLIP) is more critical than the other, and what specific roles each model plays in different canonicalization scenarios?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AZGhW2wwI1", "reviewer": "Reviewer_PWqm", "summary": "The paper proposes to construct an energy measure from outputs of pretrained foundation models. This energy can be minimized over transformations of the input image to obtain a canonicalization of the image. The minimization can be done through bayesian optimization. The experimental results are good.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. We are glad you found our experimental results to be good and we hope FOCAL can be a key step towards a zero-shot approach towards a wide range of visual invariance. We respond to your concerns and questions below:\n\n- **“If the downstream model is trained to be invariant, canonicalization is not required.”**\n\n    - The difficulty lies in practically achieving a general form of invariance just with data augmentation. Data augmentation still has limitations on long-tailed data [1], generalizing to new transformations, or generalizing outside the trained range. Thus, even if a downstream model is trained with data augmentation, our method can still complement training-time data augmentation by improving invariance in such cases. As an example, in our 2D rotation experiments, FOCAL increases the model’s invariance despite it being trained with data augmentation: even on a balanced CIFAR10 dataset setting, we find that the EKLD with FOCAL+DA is over 10x lower than just DA (0.0053 vs. 0.056).  \n      [1] https://arxiv.org/abs/2203.09739 \n\n- **“but a large part of the argument is the alleviation of having to train a canonicalization network. In other words, this is an argument that is based on reduced computational requirements … paper should include explicit numbers on compute to give the reader a proper sense of the trade-offs involved.”**\n\n    - We fully agree that it is important to discuss runtime costs, and will add that analysis to the paper. Please see our “How fast does the proposed method run?...” response to XwyL for details.\n    - To clarify, our biggest claimed benefit is not runtime but rather generalization. \n    - Consequently, our method outperforms PRLC’s specialized canonicalizers not only on their trained settings (Figure 7, Table 1) but also on new datasets (e.g, ImageNet) and downstream models (e.g., CLIP). In addition, our approach also works for more complex transformations than previously explored with canonicalization (lighting, 3D, day-night, active vision).\n    - If runtime is a priority and training a canonicalizer is acceptable, our method can still be useful as a source of supervision. Please see our “Distilling FOCAL energy into a cheaper (or one-shot) EBM” response to XwyL.\n\n- **“It contains comparisons to TTA. It is unclear if those comparisons are at an equal compute budget.”**\n\n    - Yes, this comparison uses an equal compute budget.\n\n- **“Boominathan, Lokesh, Suraj Srinivas, and R. Venkatesh Babu. ‘Compensating for large in-plane rotations in natural images.’ Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing. 2016.”**\n\n    - Thanks! We agree this is a useful citation and will add it to the paper.\n\n- **“What is the computational cost of the method?”**\n\n    - See “How fast does the proposed method run?...” response to XwyL above\n\n- **“In Figure 1(b), is the left-most example an output from the model? I.e. does it produce a background-free teddy bear from an image with background?”**\n\n    - The left-most example is an output from TRELLIS [2], the 3D generator we used. TRELLIS removes the background as part of its pipeline, as TRELLIS is trained on background removed images.   \n      [2] https://arxiv.org/abs/2412.01506", "claims_and_evidence": "The authors write \n> As a bonus, this approach is fully\ncomplementary to training-time approaches like DA.\n\nIf the foundation models were trained to be invariant to the considered transformations, they couldn't be used to determine which transformation is the canonical. If the downstream model is trained to be invariant, canonicalization is not required.\n\n# Post-rebuttal:\nRegarding the compatibility with data augmentation – it may be true that the model becomes more invariant by further adding a canonicalization, but it will not in the balanced case become better on average since the downstream model does not have a preferred orientation if it is trained with data augmentation.", "methods_and_evaluation_criteria": "The evaluations make sense.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The paper makes a strong argument for using foundation models for canonicalization, but a large part of the argument is the alleviation of having to train a canonicalization network. In other words, this is an argument that is based on reduced computational requirements. However, since foundation models are quite heavy to run and several transformations have to be tested in a Bayesian optimisation scheme, it seems like the method is heavier than prior work at inference time. This is mentioned in Limitation (1), but the paper should include explicit numbers on compute to give the reader a proper sense of the trade-offs involved.\n\n# Post-rebuttal:\nI am satisfied with the addition of computational requirements to the paper.", "relation_to_broader_scientific_literature": "The paper shows that the proposed method works. This has not been demonstrated earlier.", "essential_references_not_discussed": "Kaba et al. were not the first to propose canonicalization by minimizing an energy score, see for instance \n\nBoominathan, Lokesh, Suraj Srinivas, and R. Venkatesh Babu. \"Compensating for large in-plane rotations in natural images.\" Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing. 2016.", "other_strengths_and_weaknesses": "The paper is generally well-written and clear.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. What is the computational cost of the method?\n2. In Figure 1(b), is the left-most example an output from the model? I.e. does it produce a background-free teddy bear from an image with background?\n\nIncluding the computational cost in the paper would lead me to raise my score if no critical concerns are shown in other reviews.\n\n# Post-rebuttal:\nI am satisfied with the addition of computational requirements to the paper. Thus I will raise my score to Accept as indicated.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mOi40PRq1s", "reviewer": "Reviewer_snMu", "summary": "The paper introduces FoCAL, a zero-shot framework for achieving approximate invariance to complex transformations at scale without requiring additional training. FoCAL operates in two steps: (1) generating multiple transformed variations of an input and (2) ranking them using energy functions derived from CLIP and Stable Diffusion to select the most \"canonical\" version. The method is evaluated on transformations such as 3D viewpoint shifts, lighting changes, and environmental variations across multiple datasets, including ImageNet, COCO, Objaverse-LVIS, and CO3D.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We are glad you found value in our idea and we hope FOCAL can be a key step towards a zero-shot approach towards a wide range of visual invariance. We address your concerns and questions below:\n\n- **Clarification question**: By DA, are you referring to data augmentation or domain adaptation? \n\n  We use DA for data augmentation in our paper (line 50, right column). Apologies for the confusion. We are not sure what an appropriate comparison against domain adaptation would be. Thus, the rest of this response assumes data augmentation, but if you meant domain adaptation, please clarify what you have in mind.\n\n- **Comparisons on 2D rotation**: Our PRLC [1] experiments (Figure 7 in the main paper, Table 1 in the appendix) indeed show a comparison against [1] as well as against data augmentation. \n\n  - The experimental setting in Table 1 is ideal for data augmentation because it uses balanced datasets (e.g., CIFAR10) with known augmentations (2D rotation). On these datasets (CIFAR10, CIFAR100, STL10), we beat both data augmentation and PRLC [1] baselines.\n\n  - We don’t show explicit comparison against equivariant architectures because we could not find ViT/R50 scale baselines that could be compared fairly, but we would be happy to evaluate any baselines that you suggest. \n\n  - We do have an indirect comparison against equivariant nets through [1] because it uses an equivariant net to canonicalize the images.  \n     [1] https://proceedings.neurips.cc/paper_files/paper/2023/file/9d5856318032ef3630cb580f4e24f823-Paper-Conference.pdf \n- **“Figure 4 is never cited in the paper.”**\n\n   - Thanks! Will fix.\n- **“In lines 317-318, it is mentioned that FoCAL does not surpass supervised approaches (e.g., Barron & Tsai, 2017; Hernandez-Juarez et al., 2020), but no results are provided. Could the authors provide a comparison with these methods?”**\n\n  - That’s a fair point. On the RCC (Recommended Color Checker) dataset, Barron & Tsai (2017) achieve a median angle error of 1.3 degrees. The Gray World baseline achieves a median angle error of 9.97 degrees. Ours achieves a median angle error of 7.3 degrees. When it comes to classification, we find that the classification accuracy differences are minute (<0.5%). \n\n  - We will add a more detailed comparison in the paper. We will also try to find supervised baselines for the contrast transformation (or train such baselines ourselves) and add those to the paper as well.", "claims_and_evidence": "The claims made in the submission are generally supported by convincing experimental results. However, as mentioned in the *Methods And Evaluation Criteria* section, additional results could better demonstrate the advantages or disadvantages of their method in comparison.", "methods_and_evaluation_criteria": "There are some limitations in the evaluation. For example, the paper does not include comparisons with domain adaptation (DA) methods or equivariant architectures for transformations like rotation, which would have provided a more comprehensive understanding of the method's performance. For example, in Section 4.2: Lighting (color and contrast), the only comparison made is with vanilla CLIP.  Additionally, while the authors mention that their method is not superior to supervised approaches, they do not provide quantitative results or scores to explicitly assess the difference between their method and the supervised approaches.", "theoretical_claims": "There are no formal proofs in this paper, but the approach is grounded in empirical design choices with inspiration from prior studies.", "experimental_designs_or_analyses": "As reported in the *Methods And Evaluation Criteria* section, additional comparison would improve the experimental design of the paper.", "relation_to_broader_scientific_literature": "The key contribution of this paper is to provide a zero-shot framework for achieving approximate invariance to complex transformations such as lighting changes and 3D viewpoint shifts. This approach builds upon existing work, distinguishing itself by using energy functions derived from CLIP and Stable Diffusion to evaluate transformations, without requiring additional training or task-specific fine-tuning. Additionally, the paper relates to the broader work on equivariant architectures, providing evidence that FoCAL can handle complex transformations. FoCAL leverages pre-trained models and applies a ranking mechanism to select the canonical version of transformed inputs, thus offering a more generalizable solution, making the method particularly useful for large-scale, real-world applications where transformation invariance is needed without extensive retraining.", "essential_references_not_discussed": "I don't know any additional essential references.", "other_strengths_and_weaknesses": "The paper is well-written and easy to follow, with clear explanations of the key ideas. The figures are self-explanatory and complement the text, helping to further clarify the proposed approach. Additionally, the authors provide a thoughtful discussion of the method’s limitations and potential future work, which adds transparency to the study. However, one weakness is the lack of comparisons with existing data augmentation methods and equivariant architectures for transformations like rotation. A direct comparison with these methods would help highlight the advantages and limitations of the proposed approach and could strengthen the paper’s effectiveness.", "other_comments_or_suggestions": "Figure 4 is never cited in the paper.", "questions_for_authors": "* Q1: I understand that FoCAL addresses problems that other methods struggle with (e.g., rare classes for data augmentation and complex real-world transformations for equivariant architectures), but how does it perform in settings where these methods are typically used and perform well?\n\n* Q2: How does the method compare with the one proposed in [1]?\n\n* Q3: In lines 317-318, it is mentioned that FoCAL does not surpass supervised approaches (e.g., Barron & Tsai, 2017; Hernandez-Juarez et al., 2020), but no results are provided. Could the authors provide a comparison with these methods?\n\n-----\n[1] Mondal, Arnab Kumar, et al. \"Equivariant adaptation of large pretrained models.\" Advances in Neural Information Processing Systems 36 (2023): 50293-50309.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper initially received ratings of (4, 3, 2, 3) from reviewers. After the rebuttal, the ratings were raised to (4, 3, 4, 4), all of which are positive.

Reviewers noted that "the paper is well-written and easy to follow." They also found the proposed idea "of using foundation models to optimize the energy function for guiding canonicalization is clever, and the experiments demonstrate its strong performance compared to previous approaches." Reviewers expressed concerns about the computational issues of the proposed method, but the rebuttal effectively addressed these concerns.

As a result, the area chair agrees with the reviewers' assessments and recommends the acceptance of the paper. Furthermore, the area chair encourages the authors to revise the paper as promised, in particular, to add "a more detailed runtime analysis in the camera-ready paper, and a table detailing the FLOPs and runtime for each experiment (in seconds)."
Decision: Accept (poster)