LR0.FM: Low-Res Benchmark and  Improving robustness for Zero-Shot Classification in Foundation Models
Authors: Priyank Pathak
Shyam Marjit
Shruti Vyas
Yogesh S Rawat
Venue: ICLR 2025 Poster (2025)
Keywords: Benchmark
Robustness
Zeroshot
Foundation model
low resolution
low quality
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Benchmarking Foundation model for low resolution zero shot image classification, with a novel proposal for improving model robustness without effecting pre-trained weights

Abstract: Visual-language foundation Models (FMs) exhibit remarkable zero-shot generalization across diverse tasks, largely attributed to extensive pre-training on largescale datasets. However, their robustness on low-resolution/pixelated (LR) images, a common challenge in real-world scenarios, remains underexplored. We introduce LR0.FM, a comprehensive benchmark evaluating the impact of low resolution on the zero-shot classification performance of 10 FM(s) across 66 backbones and 15 datasets. We propose a novel metric, Weighted Aggregated Robustness, to address the limitations of existing metrics and better evaluate model performance across resolutions and datasets. Our key findings show that: (i) model size positively correlates with robustness to resolution degradation, (ii) pre-training dataset quality is more important than its size, and (iii) fine-tuned and higher resolution models are less robust against LR. Our analysis further reveals that the model makes semantically reasonable predictions at LR, and the lack of fine-grained details in input adversely impacts the model’s initial layers more than the deeper layers. We use these insights and introduce a simple strategy, LR-TK0, to enhance the robustness of models without compromising their pre-trained weights. We demonstrate the effectiveness of LR-TK0 for robustness against low-resolution across several datasets and its generalization capability across backbones and other approaches. Code is available at this : https://github.com/shyammarjit/LR0.FM

Published as a conference paper at ICLR 2025
LRO.FM: LOW-RES BENCHMARK AND IMPROVING ROBUSTNESS FOR ZERO-SHOT CLASSIFICATION IN FOUNDATION MODELS
Priyank Pathak', Shyam Marjit', Shruti Vyas' & Yogesh S Rawat!
'University of Central Florida, UIIT Guwahati priy griyank@ucf.o edu,
shyam. http::/huef-rreeggihubbiiiittt
ac. in, {shruti, yogeshJ&ucf. edu
ALBEF(s) MetaCLIP(s)
M2-Encoder(s) OpenCLIP(s)
CLIP(s) BLIP(s)
CoCa(s) CLIPA(s)
SigLIP(s) EVA(S)
224 128 64 32
16 224
128
64 32
15 224
120 62
32 EuroSAT
26
224
129
64
32
224
128
16 224
128 64
22 16
Caltech101
Flower102
ImageNet
Oxford Pets
SUN397
Figure 1: Top-1 zero-shot classification accuracy (y-axis) resolution (x-axis): Backbones for foundation models are merged as shade, with average performance across backbones in the dark.
ABSTRACT
Visual-language foundation Models (FMs) exhibit remarkable zero-shot general- ization across diverse tasks, largely attributed to extensive pre-training on large- scale datasets. However, their robustness on cov-resolutioo/pixelated (LR) im- ages, common challenge in real-world scenarios, remains underexplored. We introduce LRO.FM, a comprehensive benchmark evaluating the impact of low res- olution on the zero-shot classification performance of 10 FM(s) across 66 back- bones and 15 datasets. We propose novel metric, Weighted Aggregated Ro- bustness, address the limitations of existing metries and better evaluate model performance across resolutions and datasets. Our key findings show that: (i) model size positively correlates with robustness to resolution degradation, (ii) pre-training dataset quality is more important than its size, and (iii) fine-tuned and higher resolution models are less robust against LR. Our analysis further re- veals that the model makes semantically reasonable predictions at LR, and the lack of fine-gra fine-grained details in input adversely impacts the model's initial layers more than the deeper layers. We use these insights and introduce simple strat- egy, LR-TKO, to enhance the robustness of models without compromising their pre-trained weights. We demonstrate the effectiveness of LR-TKO for robust- ness against low resolution across several datasets and its generalization capabil- ity across backbones and other approaches. Code is available at this link.
INTRODUCTION
Vision-Language Foundation Models (FMs), such CLIP (Radford et al., 2021), LLaMA (Tou- vron et al., 2023), and other variants, have shown extraordinary generalization capabilities across wide range of downstream tasks, including image classification (Ilharco et al., 2021), 1), object detec- tion (Zhong et al., 2022), and semantic segmentation (Xu et al., 2023). These models benefit from large-scale, multi-modal pre-training on diverse datasets like DataComp- IB (Gadre et al., 2023) and LAION-5B (Schuhmann et al., 2022), enabling them with zero-shot capabilities. Although these models excel on high-resolution benchmarks, their performance with low-resolution (LR) pixelated images, common real-world challenge, remains adequately underexplored.
Low-resolution images frequently arise in various practical scenarios, such as surveillance footage (Davila et al., 2023), satellite imagery (Patil et al., 2017), and privacy-protected pixelated

---

Published as a conference paper at ICLR 2025
224 224
128 x 128
224 224
64 x 64
224 x 224
32
224 224
16 16
Wild Boar
Pig
Corn
Hot Pot
Vulture
Balid eagle
Bubble
Balloon
Matchstick
Lighter
Volcano
Church
Banana
Orange
Sandbar
Lighthouse
Figure 2: Zero-Shot misclassifications: EVA-CLIP [2023a] correct classification at 224x224 (green)
& misclassification lower resolution (red). However, ImageNet tbbels-based mispredictions semantically reasonable (humans), indicating viability of pre-trained weights at low resolution.
data (Zhou et al., 2020) etc. In these cases, details crucial for accurate classification may be ob- scured by artifacts like pixelation and compression, leading to substantial performance degradation. For instance, small objects (faces) within larger images (Cheng et al., 2019) pose unique challenges, often requiring models to rely on limited visual cues. Given the widespread presence of LR images in real-world applications, it crucial understand how robust FMs are in these settings.
Motivated by this, we present an in-depth benchmarking study of FMs, focusing on their zero-shot classification performance under LR conditions. We introduce LRO.FM, comprehensive bench- mark that evaluates 10 foundation models across 66 backbones and 15 diverse image classification datasets, ranging from large-scale datasets like ImageNet (Deng et al., 2009) to ame-grained and texture-specific datasets like Oxford Pets (Parkhi et al., 2012) and DTD (Cimpoi et al., 2014). Our study systematically examines the effects of resolution degradation, revealing key insights into how model size, pre-training dataset quality, and fine-tuning impact robustness in LR scenarios.
Metrics for measuring robustness y (Schiappa et al. (2024b)) and its averaging across datasets (SAR) have some limitations: 1) They can produce misleadingly high scores when models perform poorly on challenging datasets, and 2) They tend to ignore certain datasets, skewing the overall comparison. To address these, propose a new metric, Weighted Aggregated Robustness (WAR), which provides a more balanced evaluation by considering performance drops across datasets more fairly.
Our analysis reveals several interesting insights. Larger models tend to maintain robustness better when faced with LR inputs, while the quality of the pre-training dataset is more crucial than its size in preserving performance. Furthermore, fine-tuned models and those with higher-resolutton inputs significantly underperform against resolution drop. We also observe that although models struggle at low-resolution (fig. 1) and loss of fine-grained details (fig. 2: e.g. Vulture Bald Fagle, Bubble vs Balloon etc.), their predictions often remain semantically reasonable, even at extreme resolutions (fig. 2: e.g. Orange Banana, Church Volcano etc.). Supplementary demonstrates more examples (including real-world) where such mispredictions are made. This suggests solution for low-resolution does not require extensive modifications to the model and its pre-trained weights.
Based on these insights, we propose a simple yet effective solution, LR-TKO: LR-Zero-Shot To- kens, which introduces low-resolution-spedee specific tokens enhance robustness without altering the pre-trained model weights. Our method preserves the model's 's semantic reasoning capabilities while compensating for the loss of fine-grained detail, offering a feature super-resolution- (Chen et al., 2024). By training synthetic diffusion-based high-resolution images, LR-TKO im- proves performance low-resolution zero-shot classification tasks, making FMs more robust for practical, real-world applications.
approach
summary, we make the following contributions this work,
We present LRO. FM, comprehensive benchmarking of Visual-Language Foundation Models (FMs) on zero-shot classification of low-resolution images, providing several key insights. To the best of our knowledge, no prior work has explored this aspect of FMs.
2. We introduce simple and effective method, LR-TKO, to enhance model robustness against low-resolution inputs without altering the pre-trained weights.
3. We introduce Weighted Aggregated Robustness (WAR), novel robustness metric for eval- uating models under challenging conditions, overcoming the limitations of existing metrics.
2

---

Published as a conference paper at ICLR 2025
Table 1: Benchmark Models (66 Backbones): Pre-training is image-text pairs from datasets like
DataComp- 1B (DC-1B) (Gadre et al., 2023), Conceptual Captions (CC) (Sharma et al., 2018), Con- ceptual 12M (C- 12-12M (Changpinyo et al., 2021). Text Encoders are mostly modified vanilla trans-
formers (Tran. (( (Vaswani et al., 2017). Vision backbones use (modified) ViTs (Dosovitskiy, 2021).
Models
#Backbones
Pre-training (Dataset Size Billion:B & Million:M)
Text Encoder
CLIP [2021]
4 ViTs & 5 ResNets
WTT-400M (2021)
400M
Tran. [2019)
OpenCLIP [2021]
ViTs
DC-IB, LAION-2B (022), DFN-5B 2023)
1B-5B
[ran.|2021]
MetaCLIP [2024a]
ViTs
Self
400M-2.5B
OpenCLIP
CLIPA (v1&v2)
ViTs
DC-IB, LAION-2B (2022)
IB-2B
Autoregressive Tran. [2017]
[2023b: 2023c]
SigLIP (2023)
ViTs
WebLI [2023b]
10B
Tran.
CoCa (2022)
3 ViTs
LAION-2B (2022), COCO [2014]
2B
Tran. Decoder
M 2ncoder(2024)
3M Encoder
BM-6B [2024]
6B
Magneto [2023]
ALBEF [2021]
ALBEF
COCO [2014], Visual Genome (2017),
4M-14M
BERT [2119]
(VIT)
CC, SBU Captions [201 1, C-12M
BLIIP [2022b]
8 ViTs
ALBEF (2021), LAION- [2021]
14M-129M)
BERT [019
EVA-CLIP(&INT)
8 EVA(s)
LAION-400M 2021), 1, LAION-2B 2022),
400M-2B
OpenCIIP
(2023a, 2023b]
(ViT(s))
Merged-2B (2023a)
2 RELATED WORKS
Foundation Models (FM): Large-scale models (Kirillov et al., 2023; Girdhar et al., 2023), pre- trained on massive datasets, demonstrate generalization across numerous downstream tasks. For ex- ample, CLIP (Radford et al., 2021) embeds 400 million image-tex pairs in shared feature space for zero-shot image classification and image-text retrieval. It is also effective in other domains like video-text retrieval (Luo et al., 2022), and video and audio understanding (Lin al., 2022; Guzhov et al., 2022). Joint vision-text learning has also succeeded in tasks such as self-supervision (Miech et al., 2020), few-shot (Alayrac al., 2022), multi-modal retrieval (Yu et al., 2022) etc. However, the robustness of these models against real-world challenges e.g. harmful images (Qu et al., 2024), image quality (Wu al., 2024), text quality (Xu al., 2024b), etc. requires further exploration.
Zero Shoo/Open-set/In-ttee wild image classification predicts an unseen class by match- ing the image with labels (Sun et al., 2023a). In the past, traditional models have been tested for their zero-shot capabilities (Chao et al., 2016; Xian et al., 2017), however, FMs are better suited for this task. Benchmarking their zero-shot capabilities is relatively newer area of research (Schiappa et al., 2024a; Schulter et al., 2023). To assess the performance comprehensively, we have expanded the pool of models from traditional 10-11 FM backbones e.g. 4 backbones (Li et al., 2022a), 9 back- bones (Liu et al., 2024), 6 backbones ((Zhang et al., 2024)) etc. to 66 backbones.
Zero Shot:
Low Resolution (LR): LR images are captured in various practical scenarios and are sometimes used intentionally for computational cost reduction (RECLIP (Li et al., 2023a)). LR benchmarks mostly focus face recognition (Luevano et al., 2021; Li et al., 2018; Tirupattur et al., 2021), with some work in zero-shootuncconstrannee recognition (Li et al., 2019; Cheng et al., 2019). Super Res- olution (Ohtani et al., 2024; Gao et al., 2023) are often domain-specttn sppaii- or restores only 2 644 x 64. However, there is a lack of study on the robustness of FM(s) against real-world challenges (Xu et al., 2024b; Schiappa et al., 2023), with no previous work on very LR. We benchmark FM(s) against LR images and propose a lightweight solution for improving robustness, without training on any of the target datasets (Chen al., 2024).
BENCHMARKING SETUP
Model: Table I lists all 10 Foundation models used in our benchmarking'. CLIP, OpenCLIP, Meta- CLIP, CLIPA, and SigLIP use the same ViT model with different pre-training datasets and slight architectural modifications (e.g. layer norm position, token masking etc.). M (built on top of CoCa), ALBEF, and BLIP use modified cross attention between text and vision transformers. EVA-CLIP is family of models equipped with recent advancements egg. architectural modifica-
EVVA-CIIP (Sun al., 2023a) & EVA-CLIP-188 (Sun et al., 2023b) merged into one.
3

---

Published as a conference paper at ICLR 2025
Oxford PePts
Imagen e
ImageNet TereeNet-AOktan
ImaeNNtt
AOxford-
Flowers100
102 DTD
UCF10R
C1:Sushi
C2:0ysters
"aphoto of J. a type food
Text Encoder
ImageNet- SUN397
C3:Apple Class Labels
food101
Template
Caltech
Fct
fo
fc3
I ImageNett -eeNet-Sketch
Image
fimy
frng- Set
fim fea
fimg fea
EuroSAT ImageNet FGVC-Aircraft - Stanford-Cam Carrs
Enzoder
21 X
Figure 3: Left: Dataset: Size ix log # test images, and color gradient xx # of test classes orange is 10 & black is 1000 classes). Right: Zero Shot Evaluation: Food-101 image (32x32) generates image embeddings fimg. while class labels are filled in templates (1 shown) generating text embeddings (averaged across templates). The dot product of fi fing with text features gives classification logits.
tions, token dropping, training via distillation etc. surpassing all existing works. Backbones are re- ferred to using their publicly available pre-trained weights, e.g. CLIP-ViT I. (400M), which means: CLIPP model ViT-I. architecture, pre-trained on 400 million datasets. 'B' would indicate a billion.
Dataset: Figure (left) highlights benchmarking datasets size and the number of classes for: Im-
agenet [2009], ImageNet-A [2021b], ImageNet-V2 [2019], ImageNet-R [2021a], ImageNet- Sketch (ImageNet-SK) [2019), Caltech101 [2007), DTD split-1 (DTD) [2014], Food101 [2014), SUN397 [2014] Stanford Cars (Cars) [2020], FGVC Aircraft (Aircraft) [2013), Oxford Pets (Pets) [2012], Ox- ford Flowers 102 (Flowers 102) [2016), EuroSAT (2019), UCF101 [2012). Details in Supplementary.
Zero-Shot Image Classification We adopt CLIP (Radford et al., 2021) evaluation protocol for all the models as shown in fig. 3 (right). Image encoder generates embeddings for images, while test labels are used with dataset-specific templates (multiple templates, Supplementary) e.g. "a photo of a [label]". Model's Text encoder generates final text embeddings (averaged across all templates) for the class label. The dot product visual and text embeddings produces class logits, with the highest logit score determining the predicted class. Accuracy is computed using Top-1 match.
Low Resolution: Models are evaluated on their pre- pre-trained resolution, namely x24x224 256 x 256, 378x378 etc. Low resolution simulated by downsampling HR images 16x 16, 32 32. 64 x x4,,, and 128 x 128 using bicubic interpolation, followed by model specific preprocessing similar to their HR counterparts, e.g. resizing to 224 x 224, center crop, etc. Performance degradation starts below 64 64 (fig. 1), so we focus mainly on 16 x 16 and 32 x 32. This downsampling mimics pixelation as seen in low-resolution cameras (e.g. self-driving cars) and distant images (e.g. CCTV), etc.
Evaluation Metrics: We represent top-1 accuracy on the dataset 'D" with a resolution x as Ao E [0, 1]. egg. HR accuracy A A (LR accuracy), where HR is model specific E {224, 256, 372, 384, 512]. Top-1 scores averaged across datasets is ACC-n. Robustness against arti- facts (Schiappa et al., 2024b) is measured by relative robustness (Y/2 In is dataset-specific, and it is common to average scores across datasets for model comparison, denoted by Simple Aggregated Robustness (SAR-n). Higher number indicates more robustness. However, there are two significant issues with yp and SAR-n:
(A / A
Problem A) Misleading high robustness: If the model performs poorly on challenging dataset i.e. performance close random predictions, then downsampling will likely maintain this random prediction with minimal drop in accuracy, giving abnormally high robustness score. Fx. 'ALBEF (4M)' for Aireraft dataset, (A (Auircraft = 1%). A 2.7%, Asireraft 1%. aireraft 37%, i.e. random predictions yields~440' robustness (40% robustness is among the highest, more in Supplementary).
Solution:
Improved Relative Robustness r:: A naive solution is calculate relative robustness only for correct predictions at the HR resolution. However, tracking predictions for each model across all datasets might not be scalable, especially if the dataset contains millions of images. We propose zero-ing out robustness near random predictions. We first define accuracy gap for the model on dataset with 'C' classes as ED A Apand with ED [0, 1], and A = 1//C represents random prediction accuracy². If A >> A ED will be high. Conversely, if A prediction, En +. Using εD, we compute improved relative robustness 1' as
HR ~ random
D
x - ((E)))))))) a(E)))
a a>>1 & 0 0 5
(I)
Randomm guessing one of the 'C' class yields 1/C accuracy, referred A in this work.
4

---

Published as a conference paper at ICLR 2025
Vn
D
1.0
Imagenet
Yic
ImageNet Flowers102
Robustness
Spearman Correlation
DTD
Aircraft
EuroSAT
ImageNet-R
T2
Imagenet-A
Pets
ImageNet-Se Food,
0.5
DTD
EuroSAT
Caltech101
Cars
ImageNet-A
0 0.2
0.5 D
0.0
SUN397
ImageNet-v2
Figure 4: Left: Improved r vs traditional DD: D 707 except near random predictions (ED 0). Mid: Correlation between the ordering of models after averaging of robustness (SAR) across datasets (716 & 1%) with dataset's true ordering. SAR final ranking ignores datasets like EuroSAT (0.26). Right: Optimized dataset weights for WAR-16. Supplementary contains numeric value.
WAR
ImageNet-R
ALBEF(s)
M2-Encoder(s)
CLIP|s]
CoCa(s)
SigLIP(s)
ImageNet
FloworlDZ
MetaCLIP(s)
BLIP(s)
OpenCLIP(s)
CLIPA(s)
EVA(s)
Strong
ImageNettVV
0.42
DTD
Mederata (6.1..))
Aircraft
SUN397
Weak/N es 6.3f
WAR
WAR-16
UCF101
EuroSAT
0.77
Food101
Caltech101
I
Imageeeet-5k SK
Cars
i) Model Size (log)
Pre-training dataset
Figure 5: Evaluations at 16x16 Left: SAR vs WAR: WAR improves the correlation (between the ordering of models after aggregation with individual datasets) for EuroSAT (0.26 0.49 and ImageNet-A (0.56 + 0.68), both computed via T Right: i) Model Size & ii) Pre-training dataset size positively impacts robustness. (i) Dot size GFLOPs, no impact robustness (ii) Dot size x Model Size, positively impact robustness. ResNets (+), and transformers (O).
when En 0 i.e. near random predictions, T 0, otherwise r & Tn D. shown in fig. 4 (left). Hyperparameter a the rate at which rD declines as accuracy approaches random prediction. We chose = 200 as middle between 100 (the drop at & ~ 0.2) and 500 (the drop at &D ~ 0).
Problem B) SAR overlooks datasets: When comparing models, their robustness scores are averaged across datasets (SAR). Ideally, model ranking, after averaging robustness across datasets, should stay consistent with their rankings on individual datasets. However, fig. 4 (mid) shows the rankings of 66 models after averaging correlate (Spearman Rank) highly with ImageNet (0.99) and DTD (0.88), but only moderately with ImageNet-A (0.56) and weakly not with EuroSAT (0.26). Most datasets follow the ImageNet trend, influencing the final model rankings and minimizing the impact of datasets like ImageNet-A and EuroSAT (behave differently) as these datasets aren't 't present.
Solution: Weighted Aggregated Robustness: Averaging the robustness scores gives each dataset score of 1. We propose adjusting the dataset weights so that the model rankings after aggregation reflect each dataset fairly (fig. 4 (right)). Weights are optimized such that the correlation (Spearman) between the model rankings after the weighted average and individual dataset rankings are maxi- mized. The weighted sum of robustness is: WAR-n= Ed I x ww/// E
Datasets |wg]] where w is dataset weight, and I'd is dataset-specific improved robustness score for the resolution n x..
Datasets
We use Ax tool (Bakshy et al., 2018) for optimizing the weights of the dataset w E [0.1, 1] such that the Spearman correlation (SC) between the final model ranking obtained after the weighted av- craging and individual dataset ranking is maximized on empirically found (more in Supplementary):
0.95 x (SC(Imagenet) + SC(IIaCeeee--VV V2) (Cc + SC(ImageNet-A) SC(EuroSAT) SCCUuee (2)
Optimizing wh, may give minimal weights to some datasets, thus WAR-n may not reflect the true robustness and is more apt for model comparisons, representing all the datasets. Hence we use both Weighted Aggregated Robustness (WAR) using improved relative robustness r (1)) and sim- ple averaging (SAR) using traditional robustness n for evaluating models. Note, a.12 and r mea- sure dataset robustness while SAR and WAR measure averaged robustness across the datasets.
5

---

Published as a conference paper at ICLR 2025
Model Size: 149M
SB
Meta CLIP(s)
OpenCLIP(s) EVA(s)
Vanilla
FLICKR MSCDCO
CLIPA(s)
COCO
WAR-16
BUP VIT-8/15 (129M)
WAR-16
WAR-15
ALBEF BLIP VIT L/16 (14M)
CoCa VIT-L/14 (100n22__1330_ bsok)
DataComp-18 18 (1.48) LAION-2B (2.4B)
150M
430M
960M 2.5B
(129M)
Figure 6: Left: DataComp- vs LAION-2B: Smaller DataComp- IB pre-training helps robustness.
Models are ordered via size. Mid: Model comparison w/o Size: Models binned into size buckets (+30M). Right: Fine-tuning degrades robustness. (left & midy: bigger models are more robust.
Layer of 224 x 224
64
ImageNNttA
CLINA VIT-G/14
Aircraft
CLIAA VIT-
Netngeett-Sktthh
CVP-02-CUP--24
ImageNet-V2
VIT-LF44
OpenCUP VIT-1/14 quickgelu
DTD
SUNE977
of
135
SiLLIP VIT S040DM
Food101
ImageNet
IntageNittRR
16x 16
32x
100
CUP VIT-U16
SigtiP VIT-1/16
UCF101
EuroSAT
035
Size
Flowers102
SIUP VIT-8/16
Caltech101
224
256
336
378 3344
512
Average
Datasel
Pots
Model Variants with different Input Resolution
64x 64
128x 128
Figure 7: Left: High Input Resolution Model are less robust. Mid: t-SNE of Dataset robustness Dataset represented via 66 models robustness (T indicates clusters. Right: Layers-wise fea- tures L2 similarity: n n model layers similarity w/ 224 x 224 ones, for EVA02-B-16. For a given heatmap (e.g. 16x16), the lower right indicates the similarity of deeper layers (brighter means more similar), while the upper left represents non-similar shallow layers (dull means less similar).
BENCHMARKING ANALYSIS
Proposed WAR Metries:
Spearman correlation between the rankings of 66 models, calculated us-
ing SAR and WAR averaging of relative robustness T (across all datasets), and the individual dataset rankings is shown in fig. 5(left). WAR shows slight decrease in avg. correlation (SAR-16 16 0.89 vs WAR-16 16 0.87). but it also improves the representation of EuroSAT & ImageNet-A. The correlation score for EuroSAT increased from a weak/no correlation of 0.26 to a moderate 0.49.
Model Architecture/ / Pretraining: Figure 5 (right, (i) shows, on average, larger model (x-axis) are more robust. Among the models, CeiP-ResNets (stars) are the least robust (compared to trans- formers (dots)) while EVA, MetaCLIP, CLIPA, and OpenCLIP exhibit the highest robustness against the LR. Higher GFLOP (size of dots) weakly impacts robustness with too many exceptions.
Pretraining QQuaiity over Quantity": Figure (right (ii) shows pre-training dataset size weakly correlates with robustness, with exceptions like SigLIP (1OB), and M2-encoder (6B) performing worse. Models pre-trained on DataComp- generally outperform those pre-trained on LAION-2B, despite having over 500M fewer image-text pairs (fig. 6 (left)). This suggests that the model and quality of pre-training have greater impact robustness than the quantity of pre-training.
We remove architectural size advantages by categorizing oor-perfogning models into parameter buckets as shown in fig. (mid). For smaller models (150M and 430M parameters), OpenCLIP matches EVA and outperforms MetaCLIP and CLIPA, despite these two being built on top of OpenCLIP. However, for larger models, this trend reverses, with EVA-CLIP remaining su- perior for comparable sizes. Two factors contribute to performance discrepancies within models of the same parameter size: (1) Fine-tuning: ALBEF and BLIP fine-tuned variants are less robust on FuroSAT and Aireraft, reducing their overall robustness (fig. (right))
Model Specific:
(2) Higher input resolution: Models with higher input resolutions (e.g. 336 x 336) are generally less robust than their 224 x 224 counterparts, likely due to increased interpolation from 16 x 16 to higher resolutions (fig. 7 (left)).
Dataset Specific: Relative robustness of 66 models on each dataset forms its robustness vector rep- resentations. Representing these vectors using t-SNE (fig. 7 (mid)), reveal three major dataset clus- ters: high-robustness (long bars) (e.g. Caltech101), weakly robust (medium bars) (e.g. ImageNet), and least robust (smallest bar) (e.g. ImageNet-A), This indicates that low-resolution performance varies by dataset, which warrants a deeper dive into dataset-speciffe robustness, left as futuree work.

---

Published as a conference paper at ICLR 2025
224 x 224
128 x 128
64 x 64
32 x
16 16
Figure 8: Feats t-SNE: EVA-O2-CLIP-B/IC test features for Food- colored using class labels.
With low resolutions (16x 16, and 32 32), features become indistinguishable, thereby overlapping.
HR: 224 x 224
AddSR
BSRGAN
ESRGAN
HR: 224 224'
AddSR
BSRGAN
ESRGAN
LR: 16 16
IDM
Inf-DiT
Swinir
LR: x
IDM
Inf-DiT
Swinir
Figure 9: Super resolution at 16x 16: Image from Pets (left) and Food102 (right). Models include AddSR (2024), BSRGAN [2021], ESRGAN [2018), IDM [2023], Inf-DiT [2024), and Swinir [2021].
Inside Model: Figure shows the accuracy of all models first drops at 64 x 64, with a more signifi- cant decline after 32 x2. EVA-B/I6 features t-SNE (fig. 8) shows features become indistinguish- able resolution decreases. Inside the model, Figure 7 (right) shows the pairwise similarity (L2 distance) (Kornblith al., 2019) between layers of models trained at different resolutions with the 224 224. Diagonal elements (ith layer x n model similarity with jth layer of a model trained at x24x 224), is more similar towards the deeper end (lower right, the similarity is brighter), than the initial layers (upper left, the similarity is dull). Additionally, model similarity increases with resolution, while layers remain differentiable at all resolutions (dull non-diagonal values).
PROPOSED METHOD: LR-TKO
Figure 2 reveals key insights: i) LR lacks fine-grained details ii) FM(s) make semantically reasonable predictions even at 16x 16, highlighting the importance of preserving semantic capabili- ties (pre-training). While super-resolution (SR) methods could restore lost details without affecting models, zero-shot SR for very low resolutions (S 64 64), doesn't 't work well in practice, as shown fig. 9, where SR models fail to reconstruct out-of-domain images at 16. To enhance model robustness against low resolution, our solution LR-TKO adds trainable LR tokens on top of frozen transformers (preserving the pre-trained weights). These LR tokens learn to bridge the gap be- tween the high-resolution (HR) and low-resolution (LR) domains, via self-supervised distillation (section 5. We train these tokens on synthetically generated diffusion- based images (Section 5.2) in task- setting, ensuring the model is not exposed to any of the 15 target datasets.
5.1 LR TOKENS
To preserve the zero-shot capabilities of the model: pre-trained weights the model are frozen. In- stead, additional trainable tokens, referred to as "LR Tokens". are added on top of the spatial tokens after RGB to patch tokens conversion (patchification) and before each transformer block. As shown in fig. 10 (left) # LR tokens # Spatial tokens x (N+1) blocks. These tokens aim to compensate for the loss of details in low resolution, thereby enhancing the model's interpretability of LR im- ages. Contrary to prompt learning (Jia et al., 2022), where task-specific tokens concatenated to the spatial tokens, ours are added/merged.. Figure 7 (right) indicates LR feature at the initial layer deviates more than the later ones, thus LR tokens added at every block.
LR-TKO Technique: We adopt the multi-scale paradigm (Chen et al., 2019) i.e. training multiple low resolutions per HR image, given its success in the LR domain. Model without LR tokens (frozen pre-trained weights) acts as a teacher generating feature representations for HR images, true embedding /T In contrast, LR tokens (& pre-trained model) act student, generating

---

Published as a conference paper at ICLR 2025
Synthetic Dataset
Frozen Parameters
Trainable LR Tokens
Teacher Model
Transformer Block
Cunl 5501
Vanilaa
Spatial
Transformer Block
HR
N Block
(frozen)
CiP Loss
Tokens
(frozen)
Class Token
Student Model
Patch Generation
[Block Takens) X
pistton
LR
Figure 10: Fire (& ice) icons represent trainable (& frozen) parameters. Left: LR tokens are added to the frozen spatial patches (white) after patch generation, before each frozen transformer block, and class token as a final feature. Right: LR-TK0: Multi-scale training (only I shown for simplicity). Teacher (w/o LR tokens) generates HHR (HR), Student (w/ LR tokens) generates both f fīR
grNsx logo
"wedding under
Synthetic Images: (Left) Images generated using PIXART-a [2023a] using randomly sampled captions from Conceptual Captions (2018). (Right) Multiple images per caption.
Figure I1:
embeddings for both HR (Si and LR image(s) (SLRR) shown in fig. 10 (right). Pu (s( matched with f using contrastive loss (Radford et al., 2021), similar to text and image alignment. Anchoring HR-LR features around frozen teacher avoids direct matching of HR-LR embeddings, preventing pulling the HR features towards LR ones (converging into one) (Khalid et al., 2020). This also ensures features w/ and w/o spatial tokens remain similar (regularization). Feature matching doesn't require any labels for these synthetic images, aka unsupervised. It also task agnostic, i.e. doesn't 't involve any model task-related characteristics (classification in this case).
5.2 SYNTHETIC HR DATASET
We use the diffusion model PIXART-a (Chen et al., 2023a) to generate synthetic HR images, via 7,000 randomly sampled captions from Conceptual Captions (Sharma al., 2018). We expand our training set by creating multiple images (variations, human observation) per caption as shown in fig. 11. Conceptual Captions are commonly used pretraining many zero-shot models (table 1), and using synthetic diffusion-basee images helps LR tokens capture wide range of domains, ensuring generalized training. Random captions avoid targeting any specific dataset. To our knowledge, work is the first train model on synthetic diffusion images for zero-shot evaluation, contrary to training on a subset of target datasets (Chen et al., 2024). Following the multi-scale paradigm, we randomly sampled spatial resolution (height width) from three LR resolution buckets [16,3]], [32, 64), [64, 128], forming HR-LR image pairs.
downsample HR images
Zero-Shot: If 7,000 (or fewer) concepts/captions can consistently enhance model performance across 15 datasets, it suggests that the model is likely learning the relationship between HR and LR features rather than exploiting shortcuts. This is supported by greater improvements at LR (16x 16) compared HR (128 x 128). If the model somehow cheats the zero-shot evaluation using diffusiin-generaaeed images, we would expect similar or better performance improvements at HRs.
PROPOSED METHOD: EXPERIMENTATION & ABLATION
Implementation Details Models are trained with 7K captions (& 30 images/captions) in a multi- scale paradigm. EVA is trained for 200 epochs, while MetaCLIP and OpenCLIP are for 10 epochs. Evaluation metrics (section 3): SAR (simple averaging of -),, WAR (weighted averaging of re), and Acc (average top-1). 1). Higher number means better performance. Vanilla model's HR accuracy computes the accuracy gap ED and dataset weights derived for 16x 16 used for all resolutions (more in Supplementary). 'EVA-02-CA CLIPC (EVA-B/16), is used for all our model-leve analysis.
8

---

Published as a conference paper at ICLR 2025
Table 2: LR-TKO improvement on Foundation models:
'Meta-B/16': MetaCLIP-viT-B/16
(2.5B), OC-B//6': OpenCLIP-ViT-BB16 Higher number & better performance.
#
16x 16
32
x2
64 64
128x
128
224 x 224
Model
Param
SAR
WAR
Acc
SAR
WAR
Acc
SAR
WAR
Acc
SAR
WAR
Acc
SAR
WAR
Acc
EVA-B/16
149.7M
38.0
30.7
28.1
74,4
64,8
53.5
92,4
85.8
65.2
98.4
96.1
68.8
100
100
69.6
+LR-TK0
155.2M
42.4
35.4
31.3
75.3
66.4
91.8
85.9
95.5
68.3
99.1
69.0
Meta-B/16
149.6M
32.1
27.2
23.4
65.3
54.4
47.0
89.5
83.6
62.9
98.5
96.7
68.5
100
100.0
69.4
+LR-TK0
151.6M
41.9
38.9
30.2
71.7
66.0
51.0
89.3
85.4
62.6
96.7
95.4
67.3
97.6
97.4
67.9
OC-B/16
149.6M
33.4
26.5
24.8
68.6
59.5
49.8
84.1
63.6
68.3
70.4
+LR-TK0
151.6M
37.4
34.4
27.4
69.0
63.0
49.9
88.8
84.2
63.4
96.8
95.1
68.4
99.0
99.0
69.8
72.4
EVA-02-CLIP--BI BL16
EVA-O2-CLIP-B/EE + LR-TKO
+00.7
+6.1
Top-1.
48.3
+4.0
+3.2
+5.1
+4.9
+2.8
+3.3
+6.2
+1.0
1.44
+1.1
ImageNet
Aircraft Pets
ImageNet--V
DTD
ImageNet-R
Cars
SUN397
ImageNet-SK
EUroSAT
UCF101
ImageNet-A
Food101
Caltech101
Flawer102
Figure 12: Baseline vs LR-TKO: Top-1 accuracy for EVA-B/16 on 16x16. (more in Supplementary)
Table 3: Comparison with SR methods:
EVA-B/16 results, SR-specific pre-processing.
Table 4: Generalization of LR-TKO with other
Zero-Shot Techniques: Visual prompt Tuning (VPT) [2022] concatenates 50 learnable tokens to spatial tokens. RobustSAM [2024] is an image segmentation model modified for classification.
16x 16
32 x 32
Method
SAR
WAR
Acc
SAR
WAR
Acc
Baseline
34.1
26.8
25.0
71.8
59.0
51.2
BSRGAN
12.4
12.2
8.8
37.3
28.7
26.9
WAR
SAR
LR-TKO
ESRGAN
14.2
15.1
10.0
40.3
32.6
28.9
16
32
64
128
224
16
Swinir
17.9
17.6
12.7
47.7
38.3
34.3
Baseline
30.7
64.8
85.8
100
38.0
+VPT
35.5
64.1
84.6
94.5
97.8
42.6
AddSR
20.5
16.8
15.0
48.3
36.0
35.2
+RobustSAM
32.2
61.5
82.7
92.4
93.0
37.8
Inf-DIT
29.0
25.3
20.9
67.7
58.6
48.0
+LR Tokens
35.4 66.4
85.9
95.5
98.7
42.4
Our
38.9
29.5
28.4
73.1
62.0
52.0
6.1
RESULTS
Table shows our LR tokens consistently enhance robustness low resolutions (16x16 & 32x32),
particularly for MetaCLIP. While the low resolution is often seen as domain shift problem (Ge
al., 2020), leading to potential declines in HR performance, our multi-scale training and HR teacher distillation minimize accuracy drops at higher resolutions (1-2% accuracy drop). Also, LR tokens have a minimal parameter gain (+3%). Figure 12 shows Top-1 accuracy for EVA-B/I6 with and without LR-TKO, at 16 16, with max improvement Flower- 102 (6.2%). Table compares EVA-B/16 with super-resolution (SR) methods, with SR methods performing poorly in approach is better suited for
zero-shot settings for very low resolutions (fig. 9). In contrast, zero-shot scenarios. Diffusion-based SR method IDM is too computationally expensive to evaluate on large datasets like ImageNet (results in Supplementary). Table 4 applies our LR-TKO technique to visual prompt tuning which concatenates tokens (instead of adding) only before the first block. RobustSAM (segmentation models) modified for image classification (Supplementary).
6.2 ABLATION STUDY
Design Choices: Table 5 shows not freezing the pre-trained weights (i.e. fine-tuning the last blocks at 1/100 of the default learning rate) with and without LR tokens (first two rows) degrades the performance, indicating the necessity of preserving pre-trained weights. Our design choice is task agnostic i.e. model's classification plays no role in learning the HR-LR relationship but classifying LR images into captions (as class labels, task-oriented) has more or less the same performance. Table 6 shows benefit of multi-scale training (3 buckets, faster to train).

---

Published as a conference paper at ICLR 2025
Table Ablation: EVA-B/16 trained with 7K captions and 50 images/caption. 'CL': use of classifier. Not frozen means fine-tuning end-to-end.
Table 6: Multi-Scale (MS) Buckets:
'+" indicates Cumulative addition. E.g. [64,128 has [16,32] and [32,64] buckets.
Frozen LR Tk. CL
3AR-16
WAR-16
SAR-32
WAR-32
MS Buckets
WAR-16
WAR-32
Bascline (frozen)
38.0
30.7
74.4
64.8
Baseline
30.74
64.81
31.1
24.5
67.2
56.6
[16, 32]
34.01
64.77
32.8
27.8
68.1
58.3
[32, 64]
35.28
66.10
42.3
35.2
75.3
66.4
[64, 128]
35.45
66.40
42.0
34.7
75.2
65.9
[128, 224]
35.73
65.91
7K Captions
Baseline +LR-TK0
+13.6
44
WAR-16
LR-TKO
SAR-16
2K Captions
+7.9
SAR-16
42
+4.7
+4.9
40
Baseline
Baseline
37
[0]
[4] [8]
[11]
10
30
50
70.
90
of Images/ / Caption
8/16 L/14 L@336 6/14 G/14+
Position of LR Token Introduction
#lmages/Cap- tion: Robustness vs. Size of diffusion generated dataset.
Figure 13:
Figure 14: LR-TKO improves all EVA backbones: L@336
Figure 15: [i] LR tokens intro- duced starting from 7th block (& none after patchification).
is L/14 with 336 input
Imaga Net-SK
HR RGB
Baseline@224 Baseline@16
LR-TKO @ 16
HR RGB
Baseline@224 Baseline @16 LR-TKO @ 16
Food101
Figure 16: LR token Grad-CAM: Baseline (EVA-B/16) attention is scattered at 16x16 (compared to 224x224). LR-TKO focuses the object, likely capturing fine-grained details. @: input resolution.
Imagge/Captiin: Figure 13 shows multiple images per caption & even 2000 captions consistently
improve performance across 15 datasets, hinting at bridging the gap between HR-LR domains.
EVA backbones: Figure 14 shows LR tokens enhance various EVA backbones, namely, Base (B/16), Large (L/14 & I.3336) and (G/14 & G/14+). Larger backbones, B<IKG benefit from more tokens (via more layers). Model with 336 input underperforms (validation, fig. (left)).
Position of LR Tokens: Figure 15 shows introducing tokens in the earlier layer (starting from [i]-th block, and subsequent layers) is more helpful than later. This helps validate the observation fig. (right), i.e. initial layers suffer more at low resolution than deeper ones, validating the choice of fixing (introducing tokens) at initial layers than just final features.
Grad-CAM results: On low resolutions 16 16, vanilla model attention is dispersed and not as concentrated as 224 224 (fig. 16). However, our method (w/ LR tokens) shows focus on the object which helps to learn better representations at low resolution.
CONCLUSION
Our extensive evaluation of Lisgal-Language Foundation Models through the LR0.FM benchmark has highlighted critical limitations in their ability to generalize under low-resolution conditions, prevalent issue in real-world scenarios. While larger models and higher-quality pre-training datasets offer increased robustness, our findings underscore the significant impact of fine-tuning and input resolution on performance. Importantly, we observed that low-resolution inputs primarily disrupt the early layers of these models, leading to degraded performance. To address these challenges, we introduced the L.-TKO strategy, which improves model robustness to low-resolution inputs without altering pre-trained weights, offering a practical solution for real-world applications. Additionally, our proposed Weighted Aggregated Robustness metric provides a more comprehensive evaluation of model resilience, addressing the limitations of existing metrics.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGMENT
The authors thank Steven Dick (UCF High-Performance Computing) and Rohit Gupta (UCF CRCV) for their help in generating the synthetic data set.
REFERENCES
Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Bart, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Eytan Bakshy, Lili Dworkin, Brian Karrer, Konstantin Kashin, Ben Letham, Ashwin Murthy, and Shaun Singh. Ae: A domain-agnostic platform for adaptive experimentation. In NeurlPS Systems for ML Workshop, 2018. URL http: / Learningsys org nips18 httpp/ss//sssssseeeeeeseeeesssss passets/papee. pdd.
8 67CameraReaae Submi ssionAE$20 Beee:a200022NNeNe
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. food-101-mining discriminative compo- nents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 446-461. Springer, 2014.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-texi pre-training to recognize long-tail visual concepts. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 3558-3568, 2021.
Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In Computer Vision-ECCY 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceed- ings, Part II 14, pp. 52-68. Springer, 2016.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis, 2023a.
Wei-Ting Chen, Yu-Jiet Vong, Sy-Yen Yen Kuo, Sizhou Ma, and Jian Wang. Robustsam: Segment anything robustly degraded images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4081-4091, 2024.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Se- bastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong. Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish V Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Bureu Karagol Ayan, Carlos Riquelme Ruiz, Andreas Peter Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, and Radu Soricut. PaLI: A jointly-scaled multilingual language-iman model. In The Eleventh International Conference on Learning Representations, 2023b. URL https: // //openrevii eww net./ forum? d=mWVoRZ4400..
Yun-Chun Chen, Yu-Jhe Li, Xiaofei Du, and Yu-Chiang Frank Wang. Learning resolution-invariant deep representations for person re-identification. In Proceedings of the AAAI conference on arti- ficial intelligence, volume 33, pp. 8215-8222, 2019.
Zhiyi Cheng, Xiatian Zhu, and Shaogang Gong. Low-resolution face recognition. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part III 14, pp. 605-621. Springer, 2019.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3606-3613, 2014.
Daniel Davila, Dawei Du, Bryon Lewis, Christopher Funk, Joseph Van Pelt, Roderic Collins, Kellie Corona, Matt Brown, Scott McCloskey, Anthony Hoogs, et al. Mevid: Multi-view extended videos with identities for video person re-identification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1634-1643, 2023.

---

Published as a conference paper at ICLR 2025
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A aaaee-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associ- ation Computational Linguistics, 2019. URL https: /ttpss///pppp /tppi.ppppa//// wwweemnnniccchooooooo
orr/
CorpusID: 52967399.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference Learning Representations, 2021.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander Toshev, and Vaishaal Shankar. Data filtering networks. arXiv preprint arXiv:2309. 0t..22399.7722... 17425, 2023.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah M Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Vencu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Dat- acomp: In search of the next generation of multimodal datasets. In Thirty-seventh Confer- ence Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: h////ppeevvvi
.et/ / forum? dd=dVaWCDMBc......
Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xi- antong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 10021-10030, 2023.
Shiming Ge, Shengwei Zhao, Chenyu Li, Yu Zhang, and Jia Li. Efficient low-resolution face recog- nition via bridge distillation. IEEE Transactions on Image Processing, 29:6898-6999 6908, 2020. doi: 10.1 1..11099TP2222222995544
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15180-15190, 2023.
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech-256 object category dataset. Caltech
Technical Report, 2007.
Qingpei Guo, Furong Xu, Hanxiao Zhang, Wang Ren, Ziping Ma, Lin Ju, Jian Wang, Jingdong Chen, and Ming Yang. M2-encoder: Advancing bilingual image-text understanding by large- scale efficient pretraining. 2024. URL https:t///. arxiv, org/ abs /2401 15896.
Andrey Guzhov, Federico Raue, Jöm Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 976-980. IEEE, 2022.
2022-220022200
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2266, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of -ut-tf-disttibuttin generalization. ICCV, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. CVPR, 2021b.
12

---

Published as a conference paper at ICLR 2025
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. URL http:tttttt..oogg////oo 5281/ zenodo. 51 43773.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Syed Safwan Khalid, Muhammad Awais, Zhen-Hua Feng, Chi-Ho Chan, Ammarah Farooq, Ali Akbari, and Josef Kittler Resolution invariant face recognition using a distillation approach. IEEE Transactions on Biometrics, Behavior, and Identity Science, 2(4):410-420, 2020. doi: 10. 1100/TB10M.20003307733
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4022 2023.
Simon Kornblith, Mohammad Norouzi. Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMLR, 2019.
Tin Kramberger and Božidar Potočnik. Lsun-stanford car dataset: enhancing lcrge-scale car image datasets using deep learning for usage in gan training. Applied Sciences, 10(14):4913, 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International of computer vision, 123:32-73, 2017.
Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, and Jianfeng Gao. Elevater: A benchmark and toolkit Koyejo, S. Mohamed, A. Agarwal,
aaguagge-augenteed visual models. D. Belgrave, K. Cho. and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 9287-9301. Curran Associates, Inc., 2022a.
for evaluating
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NeurlPS, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-
training for unified vision-language understanding and generation. In ICML, 2022b.
Pei Li, Loreto Prieto, Domingo Mery, and Patrick Flynn. Face recognition in low quality images: A survey, arXiv preprint arXiv: 1tt.:1805.1111.... 2018.
Pei Li, Loreto Prieto, Domingo Mery, and Patrick Flynn. On low-resolution face recognition in the wild: Comparisons and new techniques. IEEE Transactions on Information Forensics and Security, 14(8):2000-2122 2019. doi: 1LL.1/99TFF.221888890000
Runze Li, Dahun Kim, Bir Bhanu, and Weicheng Kuo. RECLIP: Resource-efficient CLIP by train- ing with small images. Transactions Machine Learning Research, 2023a. ISSN 2835-8856. URL https:/ http:///ppeeeevveeee net forum? id=Uffc
Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training. In NeurIPS, 2023b.
Xianhang Li, Zeyu Wang, and Cihang Xie. C2ipa-v2: Scaling clip training with 81.1% zero-shot imagenet accuracy within $10,000 budget; an extra $4,000 unlocks 81.8% accuracy. arXiv preprint arXiv:2306. /5658, 2023c.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Lue Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international confer- on computer vision, pp. 1833-1844, 2021.
13

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard De Melo, Xiaogang Wang, Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In European Con- ference Computer Vision, pp. 388-404. Springer, 2022.
Fan Liu, Tianshu Zhang, Wenwen Dai, Chuanyi Zhang, Wenwen Cai, Xiaocong Zhou, and Delong Chen. Few-shot adaptation of multi-modal foundation models: A survey, Artificial Intelligence Review, 57(10):268, 2024.
Yuanyuan Liu, Fan Tang, Dengwen Zhou, Yiping Meng, and Weiming Dong. Flower classifica- tion via convolutional neural network. In 2016 IEEE International Conference on Functional- Structural Plant Growth Modeling, Simulation, Visualization and Applications (FSPMA), pp- 110-116. IEEE, 2016.
Luis S. Luevano, Leonardo Chang. Heydi Méndez-Vázque, Vázquez, Yoanna Martínez-Díaz, and Miguel González-Mendoza. A study on the performance of unconstrained very low resolution face recog- nition: Analyzing current trends and new research directions. IEEE Access, 9.75470-75493, 2021. doi: 10. 10.1009/ACESS.0000 20..CCCEEESS....8....1 33080712.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning, Neurocomputing. 508: 293-304, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.511 2013.
Antoine Miech, Bean-Baptiste Alayrac, Lueas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zis- serman. End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9879- 9889, 2029.
Go Ohtani, Ryu Tadokoro, Ryosuke Yamada, Yuki M Asano, Iro Laina, Christian Rupprecht, Naka- masa Inoue, Rio Yokota, Hirokatsu Kataoka, and Yoshimitsu Aoki. Rethinking image super- resolution from training data perspectives. European Conference on Computer Vision, pp. 19-36. Springer, 2024.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using million captioned photographs. Advances in neural information processing systems, 24, 2011.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 1498-35055. IEEE, 2012.
Jyoti S. Patil, R. S. Pawase, and Yogesh H. Dandawate. Classification of low resolution astronom- ical images using convolutional neural networks. 2017 2nd IEEE International Conference on Recent Trendy in Electronics, Information & Communication Technology (RTEICT), pp. 1168-
1172, 2017. URL fattps:/aape
Yiting Qu, Xinyue Shen, Yixin Wu, Michael Backes, Savvas Zannettou, and Yang Zhang. Un- safebench: Benchmarking image safety classifiers on real-world and ai-generated images. arXiv preprint arXiv:2405. N.X::2205.034866 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021.
14

---

Published as a conference paper at ICLR 2025
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 2019.
PMLR,
Madeline Schiappa, Raiyaan Abdullah, Shehreen Azad, Jared Claypoole, Michael Cogswell, Ajay Divakaran, and Yogesh Rawat. Probing conceptual understanding of large visual-language mod- cls. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1797-1807, 2024a.
Madeline Chantry Schiappa, Naman Biyani, Prudvi Kamtam, Shruti Vyas, Hamid Palangi. Vibhav Vineet, and Yogesh Rawat. large-scale robustness analysis of video action recognition mod- els. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
pp.
Madeline Chantry Schiappa, Shehreen Azad, Sachidanand Vs. Yunhao Ge. Ondrej Miksik, Yo- gesh S Rawat, and Vibhav Vineet. Robustness analysis on foundational segmentation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1786-1796, 2024b.
Christoph
Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 252944, 2022.
Samuel Schulter, Vijay Kumar B G. Yumin Suh, Konstantinos M. Dafnis, Zhixing Zhang, Shiyu Zhao, and Dimitris Metaxas. Omnilabel: A challenging benchmark for language-based object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11953-119966 October 2023.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hyperyymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018 8.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization vision-language models. Advances in Neural Information Processing Systems, 35: 15:14444444289 2022.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset 101 human actions classes from videos in the wild. arXiv preprint arXiv: / /2.0402, 2012.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip scale. arXiv preprint arXiv:2303.15388 2023a.
Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, and Xinlong Wang. Eva-clip-18b: Scaling clip to 18 billion parameters. arXiv preprint arX1v:2402.042252 2023b.
Praveen Tirupattur, Aayush J Rana, Tushar Sangam, Shruti Vyas, Yogesh S Rawat, and Mubarak Shah. Tinyaction challenge: Recognizing real-world low-resolution activities in videos. arXiv preprint arXiv:2107. 1494, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv preprint urXiv:2302. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, volume 30, pp. 5998-6008, 2017.
15

---

Published as a conference paper at ICLR 2025
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. In Advances in Neural Information Processing Sys- tems, pp. 10506-10518, 2019.
Hongyu Wang, Shuming Ma, Shaohan Huang, Li Dong, Wenhui Wang, Zhiliang Peng, Yu Wu, Payal Bajaj. Saksham Singhal, Alon Benhaim, Barun Patra, Zhun Liu, Vishrav Chaudhary, Xia Song, and Furu Wei. Magneto: A foundation transformer. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 36077-36092 PMLR, 23-29 Jul 2023.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference computer vision (ECCV) workshops, pp. 0-0, 2018.
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin. Q-bench: A benchmark for general- purpose foundation models on low-level vision. In ICLR, 2024.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the the good, the bad and the ugly. In Proceedings of the IEEE conference computer vision and pultern recognition, pp- 4582 4592, 2017.
Rui Xie, Ying Tai, Kai Zhang, Zhenyu Zhang, Jun Zhou, and Jian Yang. Addsr: Accelerat- ing diffusion-baad based blind super-resolution with adversarial diffusion distillation. arXiv preprint 2444:22004.1717,, 2024.
Hu Xu, Saining Xie, Xiaoqing Tan, Po-Yao Huang, Russell Howes, Vasu Sharma, Shang- Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying CLIP data. In The Twelfth International Conference on Learning Representations, 2024a. URL https://
openreview, n forum? 1d=5BcFInfE1..
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2955-2966, 2023,
Zhenlin Xu, Yi Zhu, Tiffany Deng, Abhay Mittal, Yanbei Chen, Manchen Wang. Paolo Favaro, Joe Tighe, and Davide Modolo. Benchmarking zero-shot recognition with vision-language models: Challenges on granularity and specificity. In CVPR 2024 Workshop on "What is Next in Multi- modal Foundation Models?". 2024b.
Zhuoyi Yang, Heyang Jiang. Wenyi Hong. Jiayan Teng, Wendi Zheng, Yuxiao Dong, Ming Ding, and Jie Tang. Inf-dit: Upsampling any-resolution image with memory-effiiint diffusion trans- former. In European Conference on Computer Vision, pp. 141-156. Springer, 2024.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. Transactions Machine Learn- ing Research, 2022. ISSN 2835-8856. URL https://////oeneee /wwpnneveeeeeeeecooooeee net forum? id= Ee277P3AYC.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 11975-11986, October 2023.
Kai Zhang, Jingyun Liang, Lue Van Gool, and Radu Timofte. Designing practical degradation model for deep blind image super-resolution. In IEEE International Conference Computer Vision, pp. 4791-4800, 2021.
Wenbo Zhang, Yifan Zhang, Jianfeng Lin, Binqiang Huang, Jinlu Zhang, and Wenhao Yu. A progressive framework vision-lante languge knowledge distillation and alignment for multilingual scene. arXiv preprint arXiv:2404. /249, 2024.
16

---

Published as a conference paper at ICLR 2025
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region- based language-ima -mage pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pp. 16793-16803, 2022.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. Advances in neural information processing systems, 27, 2014.
Jizhe Zhou, Chi-Man Pun, and Yu Tong. Privacy-sensitive objects pixelation for live video stream- ing. Proceedings of the 28th ACM International Conference on Multimedia, MM '20, pp. 3025-3033, New York, NY, USA, 2020. Association for Computing Machinery. ISBN
9781450379885. doi: 10.1145//99417134419922 URL https: 3394 3394171.4 3413972.
17


Reviews:
[{"review_id": "l283qiaEMh", "reviewer": "Reviewer_HMAF", "summary": "This paper tackles the problem of low-resolution zero-shot classification with vision-language foundation models. The authors observe that zero-shot classification accuracy drops when the resolution of input images decreases. First, they identified limitations in the existing evaluation metrics. They propose an improved relative robustness measure, which avoids misleading high scores when the model behaves closely to random predictions. Additionally, they propose weighted aggregated robustness (WAR) instead of simply averaging scores over datasets, in order to fairly reflect all the datasets. Second, they benchmark 10 foundation models (spanning 66 backbones) using 15 datasets on low-resolution zero-shot classification. Analysis on the dimensions of pretraining data and pretrained model reveals different observations: 1- Robustness to low-resolution correlates positively with pretraining data quality and model size, 2- models with higher resolution input data are less robust to low-resolution, 3- Initial layers are specifically affected when resolution changes. Finally, the authors propose generating high-resolution data using a diffusion model and learning tokens appended to frozen model weight in a teacher-student framework, where the teacher sees high resolution and the student sees both high and low resolutions. Extensive experiments and ablations show the effectiveness of the proposed framework in improving zero-shot classification accuracy, specifically in extreme low-resolution scenarios (e.g., 16x16, 32x32).", "questions": "I am not sure to have understood how results in Figure 7 (right) affect the choice of LR tokens position. Additionally, in Figure 15, what does the x-axis mean? For instance, does [5] mean that LR tokens are introduced starting from the 5th block?\n\nPotential typos:\n* Figure 3 (right): $f_{c1}$ $\\rightarrow$ $f_{c2}$, $f_{c3}$.\n* Figure 4 caption: tradational $\\rightarrow$ traditional. \n* Figure 5 caption: Pertaining $\\rightarrow$ pretraining.\n* Line 476: Table 3 $\\rightarrow$ Table 4.\n\n.................................................................................................................................................................................................................\n\nPost-rebuttal\n\n.................................................................................................................................................................................................................\n\nI would like to thank the authors for addressing my concerns. I vote with a score of 8 for the paper being the first to benchmark a variety of VLM backbones on low-resolution (LR) images, providing extensive experiments offering insights about the influence of factors like model size and pretraining data quality on the robustness to LR. The identified flaws in existing robustness score and SAR (which measures robustness of a model across datasets) are clear and the proposed solutions are sound (improved relative robustness and WAR, respectively). On the other hand, LR-TK0 as a framework for learning  tokens using pairs of HR-LR images (generated with a diffusion model) is sound and effective especially for extremely low-resolution. \nReading other reviewers' concerns (whom i would like to thank for bringing out some points that helped removing ambiguity in figure captions and details about the proposed metrics), and the corresponding authors' replies, I believe there is no critical weakness in the paper and therefore increase my score.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper starts with a good introduction motivating the study of low-resolution (LR) zero-shot classification from a practical angle. The contributions (benchmarking, adjusted metrics, method to improve LR image classification) are clearly conveyed. The potential viability of a solution for LR mispredictions is nicely motivated by Figure 2 in the introduction, showing that the mispredictions in LR case are still semantically decent.\n\nThe proposed improvement of the relative robustness score [1] which makes it close to 0 when the predictions are close to random predictions is sound and interesting. Moreover, the proposed weighted aggregated robustness score is well-motivated and shown to better reflect each dataset fairly. \n\nFor the benchmarking part, the paper is rich in figures and results showing the problem (degraded LR classification) and insights about correlated factors (model size and pretraining data quality). These results being observed on a wide variety of backbones are interesting and each dimension (for example the pretraining data quality) could be investigated in a specific line of future research. Additionally, pretraining research can benefit from these observations to better account for factors that are expected to improve LR downstream performance. \n\nFinally, proposing a solution based on freezing model weights and appending tokens that are learned in a teacher-student framework is sound and practical. Generating data with a diffusion model to train these tokens is straightforward and shown experimentally to be effective. \n\n[1] Schiappa et al., Robustness Analysis on Foundational Segmentation Models. CVPRW 2024", "weaknesses": "The authors mention that low-resolution images are simulated by downsampling high-resolution images using bicubic interpolation. I'm wondering whether using \"real\" low-resolution images can lead to different observations in all the experimental results. Would the degradation in zero-shot classification accuracy in that case start to be significant from 64x64 or from a higher resolution? Would all the correlations with model size, pretraining dataset size and quality, resolution of pretraining images ...  be the same in this case? Same question for the proposed token learning process.\n\nThe phenomenon shown in Figure 2 is interesting: misclassified low-resolution images are still assigned reasonable semantic predictions. Only few examples are shown in this figure. How often does this happen when measured on a full dataset? Is it resulting from the downsampling of the images instead of using images directly captured at low-resolution (relating to first point)?\n\nThe LR-TK0 framework is shown in Table 2 to be effective for 16x16 and 32x32. However, for higher resolutions, the metrics (WAR and SAR) and classification accuracy are still on par or lower than the original model. Isn't this compromising the initial model for ''intermediate'' resolutions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AEA6AmzKZ2", "reviewer": "Reviewer_pEiF", "summary": "This paper has explored the zero-shot ability of the visual-langrage foundation model on low-resolution images. The first contribution is that a new robust metric is proposed, named weighted aggregated robustness, which takes into account the cues of the order of the datasets and the gap between the best and random performance on the individual dataset. With abundant experiments considering different kinds of foundation models, backbones, and datasets, the paper reveals that the model is more robust with a larger size and better training data quality. Based on these analyses, a new strategy is designed to improve the zero-shot performance of the foundation model on low-resolution images by adding low-resolution tokens. \nThis research work is interesting and the paper gives abundant experiments and analyses. However, the illustration and description of the figures, the structure of the paper, and the clarity of some core concepts make the paper hard to follow.", "questions": "My main concern is the weakness 1 and 4 in the weaknesses part. I think it could not be clearly solved due to the page limitation. I suggest the authors choose the core and important parts, then make them clearly introduced and put them in the main body.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The new robustness metric is reasonable.\n2. The experiment analysis is sufficient.\n3. The effectiveness of the proposed LR-TK0 is verified. \n4. The viewpoint of this paper is interesting.", "weaknesses": "1. The \"misleading high robustness\" of SAR is clear. The \"SAR overlooks dataset\" is hard to understand as: 1) What does the ordering of the model mean? 2) What is the object function to optimize the model weight? 3) From the middle subfigure of Fig 4, the difference between SAR and WAR is small. how does the figure prove the SAR overlooks datasets? 4) From the last sentence of page 5, both SAR and WAR are used. SAR is for the individual dataset and WAR is used across the datasets. Does it mean that it does matter that SAR has  \"misleading high robustness\" problem? 5) How to choose the hyperparameter alpha in Eq. 1?\n2. The figures are hard to read as:1) Too many factors are shown with the area of the circle (Fig. 5), the length of the bar (Fig. 7), the color of the text (Fig. 3), etc. 2) The subfigures has no relation in some figure, such as Fig. 3 to Fig. 5.\n3. The paper claims that 66 backbones are used. What are they?\n4. The LR-TK0 technique is not clear on page 7 as 1) it lacks the introduction of LR token, 2) What's \"Path generation\" in Fig. 10? 3)How to compute the contrastive loss as there are three numbers in Fig. 10?\n5. The experiment analyses of Table 2 and Table 3 are not clear. What's the experimental setting when the SR method is used in Table 2? It lacks a summary of Table 3.\n6. Some minor typos: 1)The caption of Table 2. 2)The format of the figure references is not consistent. 3) \"The naming convention ...\" in Line 178 is hard to understand. 4) There's an extra period after \"Problem B) SAR overlooks datasets\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HGYsxFhK2e", "reviewer": "Reviewer_d8wv", "summary": "This paper proposes a benchmark LR0.FM for foundation models on LR. The authors propose the metric WAR-n to measure the robustness of foundation models. Additionally, the authors propose a method LR-TK0 to enhance performance on LR.", "questions": "Please respond to the questions in the weakness.\n\n\n____________________\npost-rebuttal message:\n\nThe authors have promised the suggested revisions, which may facilitate the readers' understanding.\n\nTherefore I have raised my rating from 3 to 6, accordingly.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. I believe this benchmark for LR is important.\n\n2. Extensive benchmark experiments analyses about LR0.FM.", "weaknesses": "1. In Problem B) SAR overlooks datasets, it is unclear what Spearman Rank correlation and AX optimization are, as well as how to obtain weights through AX optimization, making it difficult to understand why WAR is proposed.\n\n2. In Table 2, Ours has performance improvement over EVA-B/16, which should be attributed to fine-tuning using the synthetic images&LR images and it certainly improves the performance. Therefore, the experiments comparison are not fair and please explain what the appealing novelty is. Additionally, in Table 3, ours performs better than the SR method, but it is unclear what the differences are between ours and SR methods and why ours is more effective.\n\n3. In the introduction, for \"We observe that existing robustness metrics have some limitations\", you should introduce what the existing metrics are.\n\n4. In Figure 1, it's not clear what the y-axis is.\n\n5. In section 3, in \"The naming convention would represent the publicly available backbone pretrained weight names. Pre-training Sizes are indicated like CLIP-ViT H(400M), where M denotes million image-text pretraining.\", \"CLIP-ViT H\" is not the weight names and it should be clear that M is the size of dataset.\n\n6. In evaluation metrics, in $\\gamma_n^D$, only 224 or each HR is used? May be $\\gamma_n^D = 1 - \\left(\\frac{A_{m} - A_n^D}{A_{m}}\\right), m=224,256,378...$ is better.\n\n7. There are some wrong details. Here, only the obvious mistakes are pointed out: In figure 10, \"Fire (& ice) icons represent trainable (& frozen) parameters.\" is not a suitable caption and there are no (a) and (b) in the figure\n\n8. In table 5, what is the difference between Baseline row and the next below row?\n\n9. In Grad-CAM results, it's not clear what the vanilla model is.\n\n10. The figure caption should explain the figure content rather than the meaning of the axes. Several figures have this issue, such as Figure 13 and Figure 14.\n\n11. In some figures, it's not needed to use arrow, like in figure 4 and figure 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "D76nm0ueUu", "reviewer": "Reviewer_TJWB", "summary": "The paper proposes a low-resolution benchmark named LR0.FM and a simple training method, LR-TK0, can improve low-resolution performance. The paper measures the low-resolution performance of vision-language foundation models using LR0.FM and presents valuable findings based on performance analysis. LR-TK0 suggests adding a visual prompt to each spatial token of every layer to enhance the low-resolution performance of VL models. Experiments exhibit the effect of LR-TK0.", "questions": "- Is the low-resolution performance important for vision-language models? Do you have any practical use case on it?\n\n- Compared to another robustness benchmark in VL models, does low-resolution robustness show a different pattern, or is it similar to others?\n\n- Problem A and Problem B seem to be practical issues for benchmarking. But I'm not sure how significant these problems are. Could you provide a quantitative report on the significance of this problem?\n\n- In Section 4, the description points to Figure 5 (right, (a)) and (right (b)). But, only (i) and (ii) exist in Figure 5 (right).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- LR0.FM benchmark includes diverse VL foundation models on various datasets. I believe the low-resolution reports on 10 foundation models across 66 backbones for 15 datasets might be helpful to a lot of researchers in VL foundation models.\n- The paper includes meaningful revision on evaluation metrics, i.e., `improved relative robustness` and `weighted aggregation for resolution.` Even though these metrics are not perfect, I think raising issues on metrics and trying to solve them can provide positive guidance for the benchmark field.\n- The paper provides various aspects of benchmark results in Figures 5 - 7, which are more impressive and insightful than simple ranking.", "weaknesses": "- Although I agree with the efficiency of the low-resolution benchmark, I'm not sure of the value of low-resolution performance in the vision-language model domain. Generally, images in `16x16` or `32x32` are not the main target of VL models, which limits the contribution and impacts of LR0.FM benchmark.\n\n- There is no comparison with other benchmark results. It would be more informative if the paper provided whether the low-resolution robustness aligns with other robustness benchmarks or has a unique pattern.\n\n- LR-TK0 is a valid way to improve low-resolution performance. But, the connection between LR0.FM and LR-TK0 is not strong. It feels like reading two related papers rather than enhancing the contribution of LR0.FM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)