Adaptive Camera Sensor for Vision Models
Authors: Eunsu Baek
Sung-hwan Han
Taesik Gong
Hyung-Sin Kim
Venue: ICLR 2025 Poster (2025)
Keywords: domain-adaptation
sensor-control
domain-shift
Out-of-Distribution
data-centric
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Domain shift remains a persistent challenge in deep-learning-based computer vision, often requiring extensive model modifications or large labeled datasets to address. Inspired by human visual perception, which adjusts input quality through corrective lenses rather than over-training the brain, we propose Lens, a novel camera sensor control method that enhances model performance by capturing high-quality images from the model’s perspective, rather than relying on traditional human-centric sensor control. Lens is lightweight and adapts sensor parameters to specific models and scenes in real-time (i.e., test-time input adaptation). At its core, Lens utilizes VisiT, a training-free, model-specific quality indicator that evaluates individual unlabeled samples at test time using confidence scores, without additional adaptation costs. To validate Lens, we introduce ImageNet-ES Diverse, a new benchmark dataset capturing natural perturbations from varying sensor and lighting conditions. Extensive experiments on both ImageNet-ES and our new ImageNet-ES Diverse show that Lens significantly improves model accuracy across various baseline schemes for sensor control and model modification, while maintaining low latency in image captures. Lens effectively compensates for large model size differences and integrates synergistically with model improvement techniques. Our code and dataset are available at github.com/Edw2n/Lens.git.

Published as a conference paper at ICLR 2025
ADAPTIVE CAMERA SENSOR FOR VISION MODELS
Eunsu Baek
Sung-Hwan Han
Graduate School of Data Science
Department of Computer Science & ngineering
Seoul National University Scoul, Republic of Korea beshu 0 ac.ke kr
Seogang University
Scoul, Republic of Korea sunghwan NaN meaNigmaill mmaai.com com
Taesik Gongi
Hyung-Sin Kimt
Department of Computer Science Engineering
Graduate School of Data Science Scoul National University Scoul, Republic of Korea hyungk snu. acuaaac.k .r
Ulsan National Institute of Science and Technology Ulsan, Republic of Korea
taesik. gonggunist..cnm
ABSTRACT
Domain shift remains a persistent challenge in deep-learning-basee computer vision, often requiring extensive model modifications large labeled datasets address. Inspired by human visual perception, which adjusts input quality through corrective lenses rather than over- over-training the brain, propose Lens, novel camera sensor control method that enhances model performance by capturing high-quality images from the model's perspective, rather than relying traditional cluman-centryy sensor control. Lens is lightweight and adapts sensor parameters specific models and scenes in real-time (i.e., test-time input adaptation). At its core, Lens utilizes VisiT, a training-free. model-specific quality indicator that evaluates individual unlabeled samples test time using confidence scores, without additional adaptation costs. To validate Lens, we introduce ImageNet-E3. Diverse, a new benchmark dataset capturing natural perturbations from varying sensor and lighting conditions. Extensive experiments on both ImageNet-ES and our new ImageNet- ES Diverse show that Lens significantly improves model accuracy across various baseline schemes for sensor control and model modification, while maintaining low latency in image captures. Lens effectively compensates for large model size differences and integrates synergistically with model improvement techniques. Our code and dataset are available at ctthub..oom/dww2//2 wwt//////////....e/emmmmmmmm
INTRODUCTION
Domain shift, the distribution gap between training and test data, is well-known challenge that degrades the performance of deep-learning-basee computer vision models. Existing solutions mainly focus either model generalization (Hendrycks al., 2021; 2019; Sohn et al., 2020; Zhou et al., 2023; Ganin et al., 2016; Cherti et al., 2023; Liu et al., 2021b; 2022: Oquab et al., 2023) or model adaptation (French al., 2017; Sun & Saenko, 2016; Gong al., 2024; Yuan al., 2023; Wang et al., 2022b; Youn et al., 2022), which require modifying the model itself. However, they typically necessitate significant changes to the model and access large, labeled target datasets, making them costly, time-consuming, and impractical for real-time applications on resource-constrainge devices.
In contrast, human visual perception operates through fine-tuned interplay between the eyes (sensors) and the brain (model). The eyes function as precise sensors, capturing visual data, while the brain processes and interprets When visual input is compromised, whether by blurriness or glare, the typical response is to improve the quality of the input through corrective lenses, sunglasses, or magnifying lenses, rather than retraining the brain interpret flawed images better. This analogy highlights that the model is not all you need; acquiring high-quality images through camera sensors is essential to mitigate covariate shifts and improve visual perception (i.c., test-time input adaptation).
* edw2n. github.i.......
t Corresponding authors.

---

Published as a conference paper at ICLR 2025
Score
Sarsing
Acquiied
Quality Extrmaton Prodiction
Scase
Sersing
Aequired
Quality Extrmation Predetten
Horss?
Quilyty
Fre Buder
Dog?
Model
Iow Wiyl
Low viliiiity
Horsa
Delleyy Fre
Fumman
Model
Dest Faramnter Settin
The concept of Lens: Lens mimics the human vision system, where eyesight quality can be improved through visual sensor control, such as glasses. It leverages sensor parameter adjustments to acquire higher-quality images, thereby enhancing model accuracy.
Figure I:
Despite existing sensor controls like auto-exposure, which are optimized for human perception, we argue that camera sensor control designed for high-quality image acquisition to improve model perception requires fundamentally different approach. Furthermore, in dynamic environments and on resource-constrainee scenes. To address these issues, we introduce Lens (Figure 1), novel adaptive sensor control system that captures high-quality images robust to real-world perturbations. The core idea of Lens is to identify optimal sensor parameters that allow the target neural network better discriminate between objects, akin to adjusting a pair of glasses for clear vision. Lens achieves this by leveraging VisiT (Vision Test for neural networks), a training-free, model-specific quality indicator that operates on individual unlabeled samples at test time without additional adaptation costs. VisiT assesses data quality based on confidence scores tailored the target model, ensuring high-quality data without the need for extensive retraining or data collection. By acquiring the most discriminative images for the target model, Lens significantly boosts model accuracy without requiring model modification.
devices, sensor control mechanisms must be able to quickly adapt to varying
To demonstrate the effectiveness of Lens in realistic sensor control environments, we construct a testbed ES-Studio Diverse, where images are captured using a physical camera with varying sensor parameters and light conditions. Using this setup, we create new dataset called Image Diverse, including 192,000 images that capture diverse natural covariate shifts via variations in sensor and light settings, based on 1,000 samples from TinylmageNet (Le & Yang, 2015).
As the first in-depth study on model-centric sensor control, we evaluate Lens across two benchmarks Im (Baek et al., 2024) and our newly created ImageNet-ES Diverse using multiple model architectures. We compare Lens against various baselines, including human-targeted or random sensor control methods, domain generalization techniques, and lightweight test-time adaptation (TTA) methods. Our results show that Lens with VisiT significantly outperforms these methods, improving accuracy by up to 47.58% while effectively reducing image capture time only 0.16 seconds, making it fast enough for real-time operation. The effect of sensor control even compensates for a model size difference of up to 50x. Additionally. an ablation study on the quality estimator shows that VisiT' outperforms state-of of-e--of-the-at out-of-distriben distribbutoon (OOD) scoring methods, validating confidence scores as an effective quality proxy. Our qualitative analysis further supports these findings with visual insights. The results verify the potential of the new regime: test-time input adaptation.
Our key contributions are as follows:
Lens: We introduce Lens, simple yet effective adaptive sensor control method that evaluates image quality from the model's perspective and optimizes camera parameters to improve accuracy. VISIT (Vision Test for neural networks): Lens adopts VisiT, training- free, model-aware quality indicator that operates on individual unlabeled samples at test time. As the first attempt of its kind, VisiT estimates data quality based on confidence scores for generalizablilit, and simplicty.
CSAs (Candidate Selection Algorithms): We propose CSAs to balance real-time adaptation and accuracy improvements, enabling Lens's efficient operation under practical constraints.
ImageNet- Diverse: We release ImageNet-ES Diverse, a new benchmark dataset containing 192,000 images that capture natural covariate shifts through varying sensor and lighting conditions. Insights and Findings: Our extensive experiments not only highlight the superiority of Lens but also reveal valuable insights for future research: (1) Sensor control significantly improves accuracy without model modification.
(2) Sensor control synergistically integrates with model improvement techniques. (3) Sensor control must be tailored in a model- and scene-specifie manner. (4) High-quality images for model perception differ from those optimized for human vision.
2

---

Published as a conference paper at ICLR 2025
Sonee D1
Lens
image
VisIT
?
Vaif
Torget NN +
Dut Irage
Caneta
Viif
R:
Targe: aige
Wait
Qc
X..P
(b) VisiT.
Hall Candlllls
If Pu of
Arges
Whif
R4
F arg x
Takan Image
ViaIT
Quality
Re,
Scone
Best selection
WaiT
Xs.
Wail aomes for INc gatin rough aasheiiiee
Ou.
SceneC
Kr My Neand Network
(a) Lens.
(c) Model-specific design of Visi7' and Lens.
Figure 2: Workflow of Lens. Lens is a post-hoc, adaptive, and camera-agnostic sensor control system that dynamically responds to scene characteristics while accounting for model- and scene-specific manners based on VisiT scores to provide optimal image quality for neural networks.
RELATED WORK
2.1 MODEL IMPROVEMENT: HANDLING DOMAIN-SHIFTER INPUT DATA
Frequent domain shifts pose a significant challenge when deploying neural networks in dynamic real- world environments. Although traditional studies have aimed improve model's generalizability or adaptability, these methods place a computational burden, particularly for resource-constrain devices operating in real-time applications. Domain generalization techniques (Hendrycks et al., 2021; 2019; Sohn et al., 2020; Zhou et al., 2023; Cherti et al., 2023; Liu et al., 2021b; 2022: Oquab et al., 2023) aim train models handle diverse data distributions, but typically results in significantly larger and more complex models. Domain adaptation approaches (Ganin et al., 2016; French et al., 2017: Sun & Sacnko, 2016) adapt models to specific target domain, which necessitates frequent retraining and the collection of substantial amounts of labeled target data.
To address the need for lightweight, real-time adaptation without the cost of labeling, test-time adaptation (TTA) (Nado et al., 2020; Schneider et al., 2020; Wang et al., 2021) methods have been developed, allowing models adjust to new domains using small amount of unlabeled target data with unsupervised objectives. However, these lightweight TTA methods can lead to model collapse when faced with rapidly changing environments.
Lastly, fundamental limitation of these model-centric techniques is their inability to address the data acquisition process itself. They struggle to cope with severe domain shifts that stem from low-quality data, such images captured in over-exposed or low-light conditions (Baek al., 2024).
2.2 INPUT DATA IMPROVEMENT: MITIGATING DOMAIN SHIFTS
To address domain shifts through improved data quality, camera sensor control has recently gained attention. Unlike traditional camera auto-exposure methods designed for human perception (Kuno et al., 1998; Liang et al., 2007), this new research focuses optimizing sensor inputs specifically for deep-learning models. However, the absence of suitable benehmark datasets led early work rely on camera sensor simulation (Paul et al., 2023), which falls short in generalizing to real-world domain shifts. Although some research has explored the control of physical camera sensors (Odinaev et al., 2023; Onzon al., 2021), these efforts have been limited to highly const-coont environments with only a narrow range of exposure options.
To overcome these shortcomings, the ImageNet-ES dataset (Baek al., 2024) was introduced, capturing domain shifts in real-world conditions by employing a physical camera with varying sensor parameters, such ISO, shutter speed, and aperture. While the ImageNet-ES dataset demonstrates the potential of sensor control in addressing covariate shifts, identifying the optimal sensor parameters for specific models remains an open challenge. Furthermore, additional benchmark datasets are needed to enhance the generalizability of emerging control mechanisms. To the best of our knowledge, this work offers the first comprehensive exploration on camera sensor control using realistic benchmarks, including ImageNet-ES and our newly introduced ImageNet-Ba ES Diverse dataset.
3

---

Published as a conference paper at ICLR 2025
3 Lens: ADAPTIVE GLASSES FOR VISION MODELS
We introduce Lens, post-hoc, adaptive, and camera-agnostic sensor control system for neural networks, designed to adaptively respond to dynamic scene characteristics. The key idea behind Lens to identify the optimal sensor control parameters that capture images in a way that enhances the target model's ability to discriminate features-both in a model-specific and scene-specific manner- akin adjusting pair of prescription glasses to provide clear vision tailored to an individual's needs and environment. By focusing solely on sensor parameter adjustments and avoiding any modifications to the model itself, Lens prevents model collapse and catastrophic forgetting, ensuring reliable performance across varying domains. Moreover, it is lightweight and efficient in terms of both computation and memory. To achieve this, we propose VisiT (Vision Test for Neural Networks), a lightweight vision tester integrated into Lens that evaluates whether the images captured by the camera sensor are optimally suited for the target model and scene. Visi7' operates during test time on individual unlabeled samples without modifying the target model.
3.1 OVERALL FRAMEWORK
Figure 2a illustrates the overall framework of Lens, which operates with a target neural network M that supports batch inference and a camera sensor equipped with a set of N available parameter options, P = {pI. P2, ................................ PN }. Let z represent the image captured by the camera from target scene s using a sensor parameter option p. The goal of Lens á such that the captured image Zx,p maximizes the accuracy of the target model's ' interpretation of the scene 8. Let (i M) denote the quality estimate for image Ts,p in the context of model M. The optimal parameter option P. as selected by Lens, can be represented as:
select the optimal sensor parameter
arg max (Z p
PEP
Lens adaptively selects the optimal sensor parameter ò for each model and scene in real-time, rather than relying on a globally fixed parameter determined through offline training for all models and/or scenes. The key insight that different models have distinct ways of extracting and prioritizing features for scene interpretation. As shown in Figure 2c, two different models can perceive the same captured image differently (left side of the figure), leading to different optimal parameters for each model, even for the same scene (right side of the figure). Similarly, even with a fixed model, each scene contains unique features that are crucial for accurate prediction (further discussed in Appendix D). As a result, the optimal sensor parameter is likely to vary for each specific combination of model and scene (Baek et al. (2024)).
Model- and Scene-Specific Sensor Control.
VisiT (Lightweight Vision Test for neural networks). Lens incorporates VisiT' (Figure 2b) to esti- mate Q(=.,p; M). which represents the quality of an unlabeled captured image Ta,p when interpreted by the target model M. VisiT is designed for real-time applications, operating as lightweight and training-free free module at test time, providing model-specific quality estimates for unlabeled images. To achieve the design goal, it is essential to determine an appropriate metric quality. Specifically, we utilize the model's confidence score for its prediction on the image x., as a simple yet effective proxy for image quality, which will be further discussed in Section 3.2.
proxy for image
CSA (Candidate Selection Algorithm). The latency of Lens in selecting the optimal parameter highly depends on the camera sensor's s latency to capture multiple images for different candidate parameter options. While capturing and evaluating images for all N available parameter options would provide the highest accuracy, introduces significant latency for a single scene prediction, which is undesirable for real-time operation. To address the issue, Lens uses CSA to select a subset of the full parameter set P, denoted as P' {r/_... .p...... p'x ). as the candidate options. The number of candidate options, ((< N). can be determined based on the system's 's need to balance temporal overhead with accuracy. Note that, since Lens operates with batch inference, capturing multiple images doesn't 't incur additional inference costs.
crucial aspect of CSA is minimizing capture latency without sacrificing accuracy when selecting K candidate options. For example, sensor parameters like shutter speed significantly impact the capture time. Therefore, within the same time budget, it may (or may not) be more beneficial to prioritize multiple high-shutter-speed options over single low-shutter-speed option, depending the specific target scene and model. We explore this trade-off by implementing and evaluating several simple CSAs their performance in Section 5 and discuss their camera-agnostic properties in Appendix A.
4

---

Published as a conference paper at ICLR 2025
Entcientte
correct incorrect
Samples (%)
Reshet28 Swiet
0.5
1.5
6D
6.5
18 6.0
6.5
1.0 0.
0.5
1.0 0.0
0.5
Confidence Score
Ash Score
Score
Score
(a) Confidence based.
(b) OOD -uutrof-Distriuutiin)) score based.
Figure 3: Quality indicators proxies for image quality assessment: Each score is normalized
between to 1.
3.2 VisiT: LIGHTWEIGHT VISION TEST FOR NEURAL NETWORKS
In this subsection, provide detailed description of VisiT, the real-time image quality estimator for unlabeled test-time data. The key requirements for VisiT design are: (1) Alignment with correctness: The quality estimator must reliably indicate whether the model can accurately predict the sample. (2) Label-free operation: It must function with unlabeled data provided during test time (3) Single-sample assessment: The estimator should be capable of evaluating each image sample independently and immediately. (4) Lightweight operation: It should involve minimal computational overhead, ensuring seamless integration into sensor control pipeline.
Confidence as a Proxy for Image Quality Assessment. We propose using the confidence score as simple yet effective proxy, For sample image and target model M, defined as:
Con fidence (2;; M) max Softmax (x)).
cEC
where C is the of all possible classes, and (x) represents the output logits of the model M before applying the softmax function. The confidence score reflects how certain a model about its predictions and has been widely used in tasks such as pseudo-labeling consistency regularization, and high-quality image selection in semi- and self-supervised learning (Oliver et al., 2018; Sajjadi al., 2016; Sohn al., 2020; Lee al., 2013; Cui al., 2022; Chen al., 2020; Xie et al., 2020). particularly well-suited for real-time applications, as it requires only inference on sample without incurring additional computational overhead, such as training.
Correlation between Proxies and Image Quality. We conducted an experiment to evaluate the correlation between various proxies and image quality under real-world covariate shifts, using the ImageNet-ES validation dataset (Baek et al., 2024) (details in Appendix E.2). We compared our confidence score with caat-f-dditributiinn (OOD) scores, commonly used identify OOD samples, across three models: EfficientNet (Tan & Le. 2019b), Swin-T (Liu et al., 2021b), and ResNet18 (He al., 2016). The OOD scores were sourced from four state-of-the-ar methods: ViM (Wang et al., 2022a), ASH (Djurisic et al., 2023), ReAct (Sun et al., 2021), and KNN (Sun et al., 2022)).
As shown in Figure 3, OOD scores tend to overlap between correct and incorrect samples across all OOD techniques and models, suggesting that OOD scores quality. This discrepancy arises because OOD scores are primarily designed to detect semantic shifts (high-level features), but are less effective in identifying covariate shifts, which reflect variations in low-level features. In contrast, samples with higher confidence scores have a greater likelihood of being correct, while those with lower confidence scores more likely to be incorrect. These results underscore the effectiveness of confidence scores as a reliable proxy for image quality.
always reliable indicators of image
4 ImageNet-ES Diverse: A NEW REAL- WORLD BENCHMARK
Lens improves image quality by dynamically controlling camera sensor settings, such as ISO, shutter speed, and aperture, to optimize environmental light for each scene. The quality of image is significantly influenced by the amount and distribution of light within a scene, which depends on both the characteristics of the objects and the surrounding environment. Therefore, it is essential to evaluate the robustness Lens across various scene characteristics. While the recent ImageNet-ES. (Back et al., 2024) dataset captures real-world scenes with varying sensor parameters, it is limited to only two lighting conditions. Furthermore, as features images displayed on screen representing light-emittnng objects (e.g., traffic lights) the impact of ambient light conditions can be restricted.
5

---

Published as a conference paper at ICLR 2025
Remote camera control (sensor parameters,
Manual Altemation
(b) Specifics of data collection scheme.
(c) Light options.
Remote light control
Conoiiaation
Dosriptiuna
Light Options
Left
Right
Light 16 options)
Original somples
Datzeet
Test
1.1
255
255
optinns, L1-17
L2
127
127
Camur Asto Fapeour
Manuol
Num. shate
27
150
Asto
250/ 2003 16000
Shuter speed
Auoo
(L/4") (1/00) (L.1
Apertane
AlRo
2:0//99/110 /16
Captzred images
300000
102,000
(encluded)
Dark room
Cihhr collection settings
the
ImageNet-
150 on!
Printed image
ImageNet- ES Back (2024), except, for the lighing settings
(Bark et al., 2024).
(a) ES-Studio Diverse.
Figure Environment and sensor specifics of ImageNet-E5 Diverse.
Light
AE
Manual Parameter Control
L4
Figure 5: Representative examples of our ImageNet-ES Diverse dataset.
To rigorously evaluate Lens, new benchmark dataset is necessary complement ImageNet-ES and effectively capture the impact of diverse environmental perturbations. To this end, we developed ImageNer-ES Diverse, a more versatile dataset with 192,000 samples of non-illuminous objects taken with physical camera on a customized testbed called ES-Studio Diverse (Figure 4a). This dataset includes various sensor parameter settings (Figure 4b) and a broader range of lighting conditions (Figure 4c). As illustrated in Figure 5, ImageNet-ES Diverse unveils how sensor control interacts with diverse scene characteristics, valuable not only for Lens evaluation but also for future research exploring the effects of sensor settings and light conditions. Further details are in Appendix C.
5 EXPERIMENTS
We design experiments evaluate the impact of Lens, which is the first approach to introduce model- and scene-specific sensor control, in comparison to traditional model-adjustment solutions that completely overlook image capture pipelines and focus solely on over-training for optimizing predic- tion accuracy under real-world perturbations. Our experiments are conducted across various model architectures, including widely used methods for domain generalization and test time adaptation.
Datasets. We utilize the test sets of ImageNet-ES (Back al., 2024) and our ImageNettEES Diverse, both derived from Tiny ImageNet (Le & Yang. 2015) (TIN). These datasets encompass extensive natural perturbations in both environmental and sensor domains including 27 manual controls and 5 auto-exposure shots. ImageNet-ES focuses on Luminous objects, while ImageNet-KS Diverse features non-luminous objects, allowing them complement each other effectively. This diversity allows us to validate our approach across a wide range of real-world covariate shifts. More details about each dataset are in Appendix E.
Baselines and Oracles in the Image Acquisition Pipeline. For performance comparison, we consider two baselines and two oracles within the data acquisition pipeline. The first baseline, Auto-Exposure (AE), is commonly used sensor control designed to optimize images for human perception, though not necessarily for computer vision models. The second baseline, called Random, randomly selects parameter settings, and we calculate its performance as the average over all available options. To explore the potential of model- and scene-specific parameter control, we introduce two oracles: Oracle-Specific (Oracle-S) and Oracle-Fixes (Oracle-F), Oracle-S ideally selects the best sensor parameter for each sample and model, representing the upper bound for Lens. Oracle-F on the other hand, serves the upper bound for fixed parameter settings, without considering model-scene interactions. The best global parameter option for Oracle-F is selected based on the average accuracy across all models in Table and all scenes in both datasets.

---

Published as a conference paper at ICLR 2025
Table Accuracy comparison among the baselines and Lens with various models.
ImageNet ES (Baek al., 2024)
ImageNet ES Diverse (new)
Num. Params
Pretraining Dataset
Model
DG method
IN
Oracle
Naive control
Lens
Oracle
Naive control Lens
8
F
AB Random (Ours)
S
F
AE Random (Ours)
ResNet-50
IN-IK
86.0 92.4 49.5
32.1
50.4
78.6
63.1
38.2
17.6
12.0
43.3
26M
DeepAngment +AugMixt
(He et 2016)
IN-21K
87.0 93.0 66.9
53.2
61.4
83.1
80.2
64.1
36.2
23.6
65.1
ResNet-152
(He et al,, 2016)
60M
IN-IK
87.8 93.6 58.2 41.1
81.1
69.2
43.4
21.9
48.8
EfticietNet-BOO
SM
IN-IK
88.2 94.1 68.2 51.8
58.3
81.2
66.9
42.2
21.8
14.0
45.9
(Tan & Le, 2019b)
EfficientNet-B3
(Tan & Le. 2019b)
12M
IN-IK
88.1 94.8 73.9 62.0
66.3
83.5
75.8
57.2
33.6
21.4
55.7
SwinV2-T
28M
IN-IK
90.6 95.1 70.4 54.1
63.1
82.6
71.5
50.8
26.5
16.9
50.3
(Liu et al., 2022)
SwinV2-S
(Liu et al., 2022)
50M
IN-IK
91.7 95.4 74.4 59.9
65.5
84.5
75.3
53.9
30.8
18.9
55.6
SwinV2-B
88M
IN-IK
91.9 95.3 75.3 60.0
65.5
85.3
74.6
53.6
30.8
18.5
55.3
(Liu al., 2022)
OpenCLLIP-b
CCerti et al., 2023)
87M
LAION-2B
Text-guided
94.3 97.4 81.4 66.1
71.3
90.9
82.7
66.4
38.8
24.5
67.6
(Cherti al., 2023)
OpenCLIP-h
632M
LAION-2B
peetrain
94.9 98.5 87.1 79.0
77.6
93.0
87.9
74.6
45.5
29.3
74.4
DINOv2-h
(Oquab al., 2023)
90M LVD-142M
Dataset
93.6 97.6 85.1 74.5
13.9
90.6
87.5
72.6
44.5
28.3
72.9
DINOv2-g
1.IB LVD-142M
curation
94.7 98.0 90.7 84.3
79.8
93.1
92.8
82.5
62.8
35.3
82.9
(Oquah et al., 2023)
All models
90.7 95.4 73.4 59.8
65.6
85.6
77.3
58.3 34,2
21.4
59.8
c: (llendrycks al., 202)), (lendryck et 2019), IN: ImageNet (Le Yang, 2015), St Specific, Fixed, Auto exposure, Randum: Rundom selection
5.1 GENERALIZABILITY OF Lens
We investigate the effectiveness of Lens across various models, including representative (He et al., 2016; Liu et al., 2022), lightweight (Tan & Le, 2019b), and foundation (Cherti et al., 2023; Oquab al., 2023) models. Furthermore, we examine whether Lens can be constructively integrated with domain generalization (DG) techniques. Detailed model setups are in Appendix 1.
Table 1 summarizes the results. While Oracle-F selects the best fixed parameter maximize average accuracy, still suffers performance drops in many cases, revealing the limitations of using fixed parameters- - no single parameter optimally supports all scenarios. In contrast, Oracle-S consistently outperforms Oracle-F by large margins and even matches or exceeds performance on ImageNet (IN), the training domain. This highlights the potential of scene- and model sseeific sensor control. More importantly. Lens consistently boosts model performance compared to AE and Random across both benchmarks and all models, by large margins ranging from 8.71 % to 47.58%. Lens also delivers significantly better worst-case performance than Oracle-F, with gains of 29.1% % in ImageNet-ES and 5.43% in ImageNet-BB Diverse, demonstrating the robustness of adaptive sensor control. These results show the importance of targeting sensor control to the model, rather than human perception, and demonstrate that Lens effectively unlocks the potential of model-specific adaptive sensor control.
Moreover, Lens, without additional pretraining or extra data collection, outperforms the baseline methods even when they are combined with complex DG techniques like DeepAugment (Hendrycks et al., 2021) and AugMix (Hendrycks et al., 2019)), and applied significantly larger models. For instance, Lens on Res (He et al., 2016) outperforms baseline controls on DG-applied ResNet-50, and even those the larger Res (He et al., 2016), with gains ranging from 7.09% % to 37.48%. Furthermore, Lens on EfficientNet-B33 (Tan & Le, 2019b), with only 12M parameters, surpasses the DG-enhanced OpenCLIP-h (Cherti et al., 2023), model with 632M parameters, delivering 7.34% higher accuracy; Lens can compensate for 50x model size difference through real-time sensor control. Lastly, when combined with DG techniques and larger models, Lens's 'ss performance improves further, highlighting its synergistic nature. These findings emphasize the importance of optimizing data acquisition process, rather than focusing solely on model improvements.
5.2 REAL-TIME ADAPTATION PERFORMANCE
To assess the real-time adaptability of Lens, we compare its performance with lightweight Test-Time Adaptation (TTA) methods, which are designed for real-time model adaptation. Additionally, analyze the adaptation cost of Lens, focusing on the image capturing overhead associated with selected sensor parameter candidates, demonstrating its efficiency in real-time scenarios.
7

---

Published as a conference paper at ICLR 2025
Table 2: Real-time adaptation performance analysis of Lens against TTA methods.
control
Tes--Tmee Aduepaaiin
Leus (Ours)
Model
TIN
Eiwiromments
Rundom
BN2
TENT
(k=27)
CSA1
(k=6)
(k-6)
(k-18)
CSA3
ResNee-18 al., 2016)
$0
(Back al., 2024)
ImageNet-ES
87.9
54.1
39.2
46.6
30.7
34.2
32.0
(2.41sec)
13.8
(0.53ezc)
73.4
(0.53sec)
72.6
(0. (6se))))
73.7
ImageNet-ES Diverse
9.2
15.8
20.0
16.0
(2.41seo)
(0.33ce)
(0.53
27,4
(0. (6.16))c)
25.1
EffiieetNee-B BO (Tan & Lo, 2019b)
84.9
IetgeNet-E3 (Back al., 2024)
92.3
61.2
42.6
51.2
31.9
41.7
42.6
(2.41se3)
17.8
(0.53ecc)
76.2
(0.53rec)
77.4
(6se))
78.6
ImageNett-Be ES Diverse
60.5
38.5
19.9
11.7
15.0
17.5
15.6
(2.4lsec
39.6
(3wc))
32.4
(0.53vec
32,9
(0, 16soc)
31.4
TIN: Tiny ImageNet (Le & Yang, 2015), Al: Autoo Expusure, Random: Random Selection. Specific, F: Fixed BNI: (Nulo al., 2020), BN2: (Schneider 2020), TENT: (Wang et 2021)
CSAI: Randoes Selection, CSA2: Grid Random Selection, CSA3: Con Based Selection
TTA Baselines and Target Models. We establish three representative TTA basclines: BNI (Prediction-time batch normalization, Nado et al. (2020)), BN2 (Batch Normalization Adaption, Schneider et al. (2020)), and TENT (Wang et al., 2021). These methods are applied to the batch normalization layer and offer minimal computational and memory overhead. We apply these TTA baselines to two lightweight models ResNet-18 (He et al., 2016) and EfficientNet-Bi (Tan & Le, 2019b) - using data acquired via the traditional auto-exposure (AE) method. We then compare their performance against the same models when used with data acquired through Lens. Detailed explanations of each TTA method and the deployed models are provided in Appendix E.4.
Table 3: Ablations on the Quality Estimator
Common
Taktn Time 150
100-
CSA1
Models
C1
CSA2
Tiny ImageNet
80.4
89.3
CSA3
Oracle
F
87.9
54.1
61.2
96.3
81.5
95.0
74.8
Accurscy Accursc (8)
100
ImageNet- ES
ImageNett
Nuive control
Random
AR
46.6
39.2
42.6
51.2
70.4
71.4
62.0
66.5
ES
VIM
53.4
60.6
81.3
74.8
Lene with
ReAc
53.2
60.1
81.5
74,0
OOD techniques
ASH
47.2
3,2
ImageNet- ES Diverse
53.2
100
Lous (ours)
73.8
77.8
Tiny ImugeNatt Yong, 201 ImtgeNNet-ES (Rock al., 2014)
S: S-peecific, F: Fixed
K (# of photos taken)
10
35
20
25
Taken Time (%)
20
40
60
80 100
AE: Auso exposure, Random: Random selection
Cl: Re (He et al, 2016), C2: FMfeiesNNes-O (Tan La, 20196) Swin (Liu et al., 2021b), V2: DelT (Teuvion al., 2022)
Figure 6: Cost analysis of CSAs (EfficientNet-Bel BO) on ImageNet-ES & ImageNet-ES Diverse.
Candidate Selection Algorithms (CSAs) for Lens. Capturing images for all available parameter options for a scene introduces high latency, so we develop three candidate selection algorithms (CSAs) for Lens enable lightweight, real-time operation. These CSA algorithms consider two key factors: the number of image captures (K) and the overall capture time per scene.
CSA1: A simple method that randomly selects K options from the available options.
CSA2: A grid-based random selection leveraging spatial locality. Observing that parameter settings closer in parameter space often yield similar image qualities, CSA2 divides the parameter space into grids and randomly selects K options from these grids. With 27 available options in our benchmarks (i.e., three options per each of the three parameters), the number of grids becomes 1
for K = 1-7, 2 for K = 8-26, and 3 for K = 27.
CSA3: This method selects K options with the lowest capture costs, prioritizing settings with shorter shutter speeds, which are the primary contributors to capture latency. If multiple options share the same capture cost, the selection is made randomly.
Results. Table 2 shows that Lens with full options (K = 27) significantly outperforms all TTA baselines across all models and both benchmarks, with gains from 14.6% 45.9% This underscores the superiority of sensor adaptation to model adaptation. Furthermore, the three CSAs for Lens drastically reduce capturing time by 93.4% (to only 0.16 seconds) or require as few as 6 image captures while maintaining accuracy. Figure 6 shows detailed interactions between capture time, K, and accuracy for EfficientNet-BB BO (Tan & Le, 2019b) across both benchmarks, using five random seeds. Note that the correlation between capture time and K is consistent across both benchmarks (marked as "common") because the CSAs are neither model- nor scene-specific, relying instead on non-deterministic selection at the time of capture. While each CSA has different trade-off between
8

---

Published as a conference paper at ICLR 2025
L4
L6
Lens (65.5%)
Leas
S15 (36.18)
AE (37.2%)
Lenss
S15 [48.6%)
AE
Lenss
$15
AE
(63.9%)
(20.8%)
AE (36.5%)
(a) Image quality comparisons based on human-level perception. (815: one of 27 sensor parameter options which can be taken by 'Random" )
2.0
Random (23.8%)
Activation
2.5
05
00
Feature representation indices
(b) Feature (model-level perception) activations on 'L2' samples.
(c) Feature embeddings.
Figure 7: Sensing for human vs. sensing for DNN (ResNet-50 (He et al., 2016) augmented with AugMix (Hendrycks et al., 2019) and DeepAug (Hendrycks al., 2021)) ImageNet-ESS Diverse.
K and taken time, all CSAs maintain high accuracy until taken time significantly decreases. These results verify Lens's ability balance accuracy and efficiency in real-time scenarios.
5.3 ABLATION STUDY ON THE QUALITY ESTIMATOR
To investigate the effectiveness of VisiT, replace VisiT with four state-of-the-art OOD scoring methods, as introduced in Section 3.2. We evaluate these approaches on the ImageNet-ES dataset across four models: ResNet-18 (He al., 2016), EfficientNet (Tan Le, 2019b). Swin-T T (Liu et al., 2021b), and DeiT (Touvron et al., 2022). As shown in Table 3, Lens integrated with VisiT consistently outperforms Lens paired with all OOD scoring baselines across every model, achieving an average gain of 20.7% %. This verifies that confidence scores are more reliable than OOD scores for evaluating the image quality from the model's perspective in the face of real-world perturbations.
5.4 QUALITATIVE ANALYSIS
Sensing for Human Vision vs. Model Vision. Figure 7a highlights the fundamental difference in how humans and neural networks perceive images, using examples from ImageNet-ES Diverse. While humans may struggle to discern details in dark or bright images (those selected by Lens in L2, L4, and L6), these images lead to better model accuracy (63.9-6655%) In contrast, models perform poorly (20-48.64 6%) on images captured using auto-exposure (AE) settings or human-centered settings (S15 in L2. L4, and L6). Figure 7b further emphasizes this perceptual mismatch by showcasing distinct feature activation distributions for sample images under different sensor control methods. Specifically, the images provided by AE and Random settings cause the model to heavily activate certain features (those far from the average) that are treated as marginal for images acquired by Lens, which can degrade prediction performance. Moreover, Figure 7c demonstrates that, although Lens-acquired images may seem unintuitive from human perspective, they enable the model to generate feature embeddings cooudiins-coossstinng of 1,000 points with 5 points per label and color-coded accordingly that clearly distinguishable between classes compared to those captured with AE and Random settings. These findings highlight the critical need understand perception differences between humans and neural networks when designing effective sensor control strategies.

---

Published as a conference paper at ICLR 2025
(a) S1: Diverse-LI.
(c) S2: Diverse-LI.
(e) S3: Diverse-Ll.
NoMMet S
Seird
DIMON2
Restet S
Swird
DEMONE
Replet SE-A
S
(b) S1: Luminous-LL
(d) S2: Luminous-L.l
(f) S3: Luminous-LL.
chrostmas siscerd
coal
bu buts
Trartt
Varnor
dessma Jocking
COST
Canter
VSSIT
(g) Class wise Analysis for ResNet50-A. (Luminous) (h) Class- wise Analysis for ResNet50-A. (Diverse)
Figure 8: Model- and scene- specific solution spaces of parameter control in real perturba- tions. ResNet50-A: ResNet50 (He et al., 2016) Augmix (Hendrycks et al., 2019) DeepAug- ment (Hendrycks et al., 2021), Swin-B (Liu et al., 2022), and DINOv2 (Oquab et al., 2023).
Solution Space Analysis. Figure 8 shows the necessity of model- and scene-specific sensor control to handle real-world perturbations. Each grid point is one of the 27 parameter options from ImageNet-FSS and ImageNet-ESS Diverse, color-coded by the VisiT score of the image captured with that option. Each subfigure shows the results from three different models, showing that the same parameter setting for an identical sample can yield significantly different quality scores when the model is changed. For example, an optimal parameter for Swin-B (Liu et al., 2022)) may perform poorly for DINOv2 (Oquab et al., 2023) or ResNet18 (He et al., 2016), verifying the need for model-specific control. The figure pairs (8a and 8b), (8c and 8d), and (Se and 8f) represent the same class sample captured under an identical lighting condition but with different object characteristics from "Diverse" scenes in ImageNer-ES Diverse and "Luminous" scenes in ImageNet-ES. The column-wise differences between the two datasets emphasize the importance of scene-specific control. With the same sample and LI setting, fast shutter speeds yield low-quality images in "Diverse" scenes but high-quality images in "Luminous" scenes. Finally, Figures 8g and 8h show that under the same model, lighting conditions, and object characteristics, optimal parameters can vary across different classes. Overall, sensor parameters must be dynamically adjusted based on both model and scene characteristics.
CONCLUSION
Lens is the first method to introduce model- and scenc-specific camera sensor control inspired by human visual perception; by capturing high-quality images from the model's perspective, Lens improves model performance. Lens employs VisiT. a lightweight, training-frees model-specific quality indicator based on model confidence, which operates on individual unlabeled samples at test time. Evaluations two benchmarks of real perturbations, including our new dataset ImageNet-EES Diverse collected to address previously missing but notable perturbations, demonstrate that Lens with VisiT improves accuracy by up to 47.58% outperforming representative TTA baselines and DG techniques based on naive control. Furthermore, Lens shows generalizability across various architectures and can be synergistically combined with all DG methods. By ensuring efficiency in adaptation costs while maintaining performance, Lens has the potential for real-time applications (e.g., upcoming embodied sensor control's 's importance, showing its
AI). Qualitative analysis validates model/scene-speciie significant impact over DG/TTA, and offering a promising approach for real-world Al adaptability.
Limitations and Future Work. While Lens presents a novel paradigm of sensing for deep neural networks (test-time input adaptation) with significant potential for adoption in challenging scenarios across various tasks, such as autonomous driving, surveillance, and real-time 3D vision applications, it also opens avenues for further exploration. In this work, model confidence serves as a simple yet effective proxy for image quality assessment, but this can lead to overconfidence, especially in poorly calibrated models. Future work could explore robust quality estimators, synergies with TTA, and improved Candidate Selection Algorithms (CSAs) using model/scene factors and reinforcement learning for resource scheduling. Further discussions detailed in Appendix A.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We include the source code in the supplementary material, along with instructions. Detailed informa- tion on the experiments, including datasets, scenarios, and hyperparameters, are in Appendix.
ACKNOWLEDGMENTS
This work was supported in part by Institute of Information communications Technology Planning Evaluation (IIT)) grant funded by the Korea government (MSIT) (No. RS-2023-00223530) in part by the National Research Foundation (NRF) of Korea grant funded by the Korea government (MSIT) (No. RS-2023-00222663), and in part by the Creative-Pioneering Researchers Program through Seoul National University.
REFERENCES
Eunsu Back, Keondo Park, Jiyoon Kim, and Hyung-Sin Kim. Unexplored faces of robustness and out-of-distribution: Covariate shifts in environment and sensor domains. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arxiv. arXiv preprint 17Xx::700.00587,,, 5, 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.
1597-1607. PMLR, 2020.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818 18882,82,, 2023.
Peng Cui, Yang Yue, Zhijie Deng, and Jun Zhu. Confidence- based reliable learning under dual noises. Advances in Neural Information Processing Systems, 35:35116 192116-3333 35129, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Andrija Djurisic, Nebojsa Bozanic, Arjun Ashok, and Rosanne Liu. Extremely simple activation shaping for out-of-distribution detection. In The Eleventh International Conference on Learning Representations, 2023.
Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. arXiv preprint arXiv: /706./7700.03 052008 2017.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarill training of neural networks. Journal of machine learming research, 17(59):1-35. 2016.
Taesik Gong, Yewon Kim, Taeckyung Lee, Sorn Chottananurak, and Sung-Ju Lee, Sotta: Robust test-time adaptation on noisy data streams. Advances Neural Information Processing Systems, 36, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and dit-of-dissributtin examples in neural networks. In International Conference on Learning Representations, 2017. URL https:
net htt.htttss//pperrviieeee hwtp///wwe//eeeeeeeeeee/ee wete:/iwwwweeeeeeeeeeeeeeeeeeeee wwt.:orrntttteeeee http://rrmmmmioomom fttt

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. 2019.
arXiv preprint
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEF/CVF international conference on computer vision, pp. (340,8349, 2021.
Andrew Howard, Mark Sandler, Grace Chu, Liang Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1314-1324, 2019.
Tetsuya Kuno, Hiroaki Sugiura, and Narihiro Matoba. A new automatic exposure system for digital still cameras. IEEE Transactions Consumer Electronics, 44(1):192-199, 1998.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896. Atlanta, 2013.
JiaYi Liang. YaJic Qin, and ZhiLiang Hong. An auto-cxposure algorithm for detecting high contrast lighting conditions. In 2007 7th International Conference ASIC, pp. 725-728. IEEE, 2007.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pp. 21-37. Springer, 2016.
7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10012-10022, October 2021a.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021b.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp.
2022.
Jonathan Long. Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 31.1-3333444 2015.
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference Multimedia, pp. 1485-1488, 2010.
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
Ismoil Odinaev, Jing Wei Chin, Kin Ho Luo, Zhang Ke, Richard H.Y. SO, and Kwan Long Wong. Optimizing camera exposure control settings for remote vital sign measurements in low-light environments. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 6086-6093, 2023.
12

---

Published as a conference paper at ICLR 2025
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-super learning algorithms. Advances in neural information processing systems. 31, 2018.
Emmanuel Onzon, Fahim Mannan, and Felix Heide. Neural auto-exposure for high-dynamic range object detection. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7706-7716, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:230..07199, 2023.
Sibendu Paul, Kunal Rao, Giuseppe Coviello, Murugan Sankaradas, Oliver Po, Charlie Hu, and Srimat Chakradhar. Enhancing video analytics accuracy via real-time automated camera parameter tuning. In Proceedings of the 20th ACM Conference on Embedded Networked Sensor Systems, SenSys '22, pp. 291-304, 2023.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real saal-imme object detection with region proposal networks. IEEE transactions pattern analysis and machine intelligence, 39(6):1 137-1149, 2016.
Mchdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-superrised learning. Advances in neural information processing systems, 29, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Advances 2020.
in neural information processing systems, 33:
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying supervise learning with consistency and confidence. Advances in neural information processing systems, 33:596-600 -088, 2020.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16. 2016, Proceedings, Part III 14, pp. 443 450. Springer, 2016.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34: 144-151, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. International Conference on Machine Learning, pp. 20827-20840. PMLR, 2022.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114 14. PMLR, 2019a.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pp. 6105-61 14, PMLR, 2019b.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. CoRR, abs/2012. 12877, 2020. URL https: http::///wwwww//.//.eee www.iioooggaaaaoo 2.....
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit iii: Revenge of the vit. In European conference computer vision, pp. 516-533. Springer, 2022.
13

---

Published as a conference paper at ICLR 2025
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test- time adaptation by entropy minimization. International Conference on Learning Representations, 2021.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual- logit matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4921-4930, 2022a.
Qin Wang. Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp.
7201-7211, 2022b.
Ross Wightman. Pytorch image models. 2019.
pytorch-image-men
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in neural information processing systems, 33:6256-6268, 2020.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, al. Openood: Benchmarking generalized out-of- distribution detection. Advances in Neural Information Processing Systems, 35:32598-3266 2022.
Jiseok Youn, Jaehun Song, Hyung-Sin Kim, and Saewoong Bahk, Bitwidth-adaapiie quantization- neural network training: meta-learning approach. In European Conference on Computer Vision, pp. 208-224. Springer, 2022.
Longhui Yuan, Binhui Xie, and Shuang Li. Robust dest-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp- 15922-15932, 2023.
Kaiyang Zhou, Chen Change Loy, and Ziwei Liu. Semi-supervised domain generalization with stochastic stylematch. International Journal of Computer Vision, (L(9):2377-23877 2023.
14

---

Published as a conference paper at ICLR 2025
Appendix
Adaptive Camera Sensor for Vision Models
A FURTHER DISCUSSION
In this section, we discuss future directions of this work, as outlined in Section 6.
A.I TOWARDS MORE REALISTIC SCENARIOS
In this study, we utilized ImageNet-ES and ImageNet-ES Diverse as real-world perturbations, state- of-the-art Environmental and Sensor (ES) perturbation datasets. These datasets are pioneering in enabling effective evaluation of the impact of sensor control on environmental changes. By leveraging these resources, our work lays robust foundation for addressing domain shift challenges in more complex and realistic scenarios through sensor control.
A.1.1 POTENTIAL OF Lens FOR ADAPTATION IN VARIOUS SETTINGS
More Realistic Datasets. Extending Lens from classification tasks to advanced vision tasks such as semantic segmentation and object detection, and further into applications like autonomous driving or surveillance systems, presents promising research direction. However, existing datasets lack both sensor control information and the labeled data necessary for these tasks. While ImageNet-FH and ImageNet-ES Diverse have facilitated the evaluation of Lens for classification, similar datasets tailored to other vision tasks are required. Therefore, the creation and implementation of sensor- controlled datasets for these advanced tasks crucial for future research on Lens. Additionally, to encompass a broader range of realistic scenarios, we intend to collect and integrate more dynamic datasets, including multiple objects and dynamically changing scenes, as well as advanced tasks that incorporate ES perturbations similar to those in ImageNet-ESS and ImageNet-ES Diverse. This will enable us to validate and enhance the robustness of our methodology against various domain shifts encountered in real-world applications, thereby providing a comprehensive evaluation of our method's resilience and effectiveness across diverse environments.
Potential to Adaptation on Advanced Vision Tasks, To showcase Lens's versatility in various vision tasks and its value in collecting dataset containing sensor control factors, we performed a qualitative analysis focusing on two key applications: Semantic Segmentation and Object Detection. We compared Lens with AE (Auto Exposure), baseline camera sensor control method described in Section 5. The evaluation involved two semantic segmentation models (FCN Long et al. (2015) and DeepLab v3 Chen et (2017)) and two object detection models (Faster R-CNN Ren al. (2016) and SSDLite300 (Liu et al., 2016; Sandler et al., 2018)), representing standard or lightweight architectures. The analysis focused on the 'dog' class, a commonly used category in the training datasets of target models and superclass in the evaluation datasets. Since these tasks generate multiple outputs, unlike the classification tasks for which Lens was initially designed, we adapted Lens by modifying the Visi7' score for each specific task. Detailed experimental setups including the VisīT' adaptations, are provided in Table 4. As shown in Figures 9 and 10, Lens achieves results that closely approximate, and sometimes outperform, those of the original sample (source domain) in most cases for both tasks and all targeted models. In contrast, AE failed to recognize the target class ('dog') ) in corresponding results. This suggests that Lens has significant potential for adaptation to other vision tasks using similar approaches. Furthermore, given that the large models evaluated in Section 5.1 have been consistently improved by our system and share the backbone and datasets of representative Vision-I Language Models (VLMs) or iiration-based models, we can expect that Lens has substantial potential enhance other VLM models' performance through adaptation. However, the performance of Lens varies depending the customization of the VisiT score for each target task, indicating that further elaboration on this aspect represents a promising avenue for future research.
Generalizability in Heterogencous Camera Devices As outlined in the methodology section, while performance values can vary with camera devices, Lens operates in a camera-agnostic manner, allowing it to be applied regardless of the camera model. Lens employs a strategy that selects sensor options to achieve the highest image quality. This strategy remains effective even when camera equipment varies. Specifically, the main modules (Visil and CSAs) used for assessing image quality
15

---

Published as a conference paper at ICLR 2025
Source
Luminous
Diverse
Source
Luminous
Diverse
Sample
IN
AE
Lens
AE
Lens
Sample
IN
Leaa
AE
Lens
(a) FCN Long et al. (2015)
(b) DeepLabV3 Chen et al. (2017)
Figure Qualitative Analysis on both Benchmark (Semantic Segmentation)
are camera-agnostic: 1) VisiT ensures camera-agnostic functionality by assessing image quality through the confidence scores of images selected by the Camera Selection Algorithm (CSA). and 2) Proposed CSA algorithms in our work are inherently agneri- because they select camera parameter candidates based solely on the provided sensor parameter information, independent of specific camera models. As long as the necessary information for each CSA algorithm is supplied, they operate regardless of the camera type. The required information for each proposed CSA algorithm as follows: i) Random Selection (CSA/): Supported ranges or available sensor parameter options from deployed camera models. ii) Grid Random Selection (CSA2): Grid information of camera parameter ranges based on a specified value of K, derived from camera control specifications. iii) Cost-Based Selection (CSA3): Cost associated with each parameter option across deployed camera models. However, performance may vary depending on specific camera hardware and environmental conditions. Additionally, while it is important explore methods for more precisely identifying optimal solutions within continuous parameter spaces, it is equally crucial to consider factors such as system latency and adaptability, including training and inference times. To address this, future research should focus on balancing these aspects to facilitate the development of practical and efficient solutions.
A.1.2 POTENTIAL OF Lens FOR MORE CHALLENGING SCENARIOS.
Addressing Overconfidence. Although Lens has achieved already significant improvements by utilizing confidence scores as quality estimators for sensor control compared to existing baselines, these scores may not be optimal in all scenarios. As highlighted in Section 6, the issue of overconfi-
Table 4: Detailed Settings of Experiments on Other Vision Tasks
Taskss
Semantic Segmentation
Object Detection
Description
Identify and highlight pixels corresponding 'dog'.
(if the maximum confidence score indicates 'dog')
Detect objects draw bounding boxes,
(only for confidence scores >0.6)
Lens Adaptution (VisT Score)
Average the confidence scores the highlighted pixels.
Average the confidence scores the valid bounding boxes.
FCN Long et al. (2015) (ResNe Deeplaab v3 Chen al. (201 (MohileNet v3)
Faster RCNN Rea al. (2016) (ResNet50).
Models (backhone)
StDLit (MobileNet v3)
[Trainin]] COCO vI Lin et (2014) ImageNet-1k Deng al. (2009) (backbonc)
Datasets
[Evaluation] Luminus (ImageNet-ES Back et al. (2024), Diverse (maggeNrr ES Diverse)
ResNe150 He al. (2016), MobileNes v3 Howard at al. (2019), SSDLise00 (Liu et all, 2016; Sundler et all, 2018) All modill in these experiments were implemented using the pretrained models provied by the Tocchvision Marcol & Rodriguez (2010) liereyy
16

---

Published as a conference paper at ICLR 2025
Luminous
Source
Luminous
Diverse
Sample
IN
AB
Lens
AB
Lens
Sample
Lews
AE
Zens
(a) FasterRCNN Ren et al. (2016)
(h) SSDLitee300 (Liu al., 2016; Sandler et al., 2018)
Figure 10: Qualitative Analysis on both Benchmark (Object Detection)
dence is evident in the performance gap between Oracle-S and Lens, suggesting that mitigating overconfidence could further enhance Lens. As an initial attempt at sensor control for vision models, Lens leverages confidence scores, building on its generalizability and simplicity. This approach demonstrates substantial potential in two key areas for addressing domain shift problems: real-time applications and ensuring compatibility with diverse camera devices and models. Mov- ing forward, while maintaining the design principles of Lens, our research will focus on reducing overconfidence by refining our methodologies and evaluating the approach's 's adaptability in various real-world environments to improve Lens's 's reliability and performance. Additionally, as indicated in the ablation study in Section 5.3, existing OOD (Out-of-Disrribution) scores address overconfidence stemming from semantic shifts but fail to handle covariate shifts caused by real perturbations (e.g., ImageNet-ESS Luminous and Diverse). Therefore, addressing overconfidence for sensor control requires innovative approaches beyond classical OOD studies, emphasizing the analysis of inter- mediate model layers related to low-level features rather than solely focusing on activations in the final layers.
Addressing Time-constrained Scenarios. In real-world applications such as autonomous driving and surveillance systems, rapid environmental shifts present significant challenges, and responsiveness is critical for delivering high-quality service. The responsiveness of Lens, which integrates our developed CSA algorithms, depends on the rate of environmental changes. However, by implementing Lens within a batch inference system, it can adapt to changes within 0.2 to 0.5 seconds. To achieve more rapid responses. it is necessary to develop CSA algorithms that select minimal number of options (possibly one or two) with reduced capture times. This represents a promising direction for future research on Lens. Successfully adapting sensing systems to cime-constrained scenarios requires careful consideration of several additional factors, which can provide potential avenues for future research in this field. In these contexts, it essential to account for limited available resources and ensure effective scheduling within specified timeframes. This involves balancing trade-offs between accuracy, the number of images captured, and system latency. Furthermore, the latency of each module-sude model inference and image capture architecture and must be meticulously managed to maintain overall system performance. Considering these factors, optimizing CSA algorithms emerges promising direction for Lens.
vary depending the deployed system
17

---

Published as a conference paper at ICLR 2025
A.2 POTENTIAL OF Lens ON NEW FACES
Addressing Radical Distortion Problems. In our current study, we did not evaluate radial distortion because it arises independently from light changes caused by environmental factors and sensor control. These factors posed critical issues in real domain shifts, but existing works related to robustness couldn't 't handle them effectively, making them the primary focus of our investigation. Despite not evaluating radial distortion directly, our methodology has the potential to address by controlling framing parameters such PTZ (pan, tilt, and zoom). Given two key points, 1) Adjusting pan, tilt, and zoom can minimize radial distortion effects. 2) Our policy algorithm selects the highest-quality images based on camera parameters. Therefore, incorporating framing parameters as control options is expected to effectively manage radial distortion. As a result, jointly applying sensor and framing control could enable the handling of broader spectrum of domain shifts more effectively. Future research will explore integrating advanced PTZ control algorithms and real-time image quality assessments to further enhance our methodology's robustness against diverse domain shifts.
Lens for Representation Learning. Our method was specifically designed to capture high-quality images scenarios that utilize model inference results and did not initially consider the high iug-quulitty image acquisition processes required for the training stages of representation learning, suggested in the review. Given that most representation learning pipelines predominantly rely on fine-tuning pre-trained models for downstream tasks, we recognize the possibility of integrating Lens during the training stage. This integration could generate customized high-quality images tailored for both pre-trained models and target tasks, potentially reducing data collection costs and enhancing model performance.
B MORE ANALYSIS
Label-wise Analysis. To validate the performance of Lens for individual labels in the source domain (ImageNet), we assessed the label-wise accuracy of the target models in Section 5.1 (Experiment 1) for both the representative baseline (AE: Auto Exposure setting) and Lens. As illustrated in Figure 11, Lens consistently outperforms the baseline, regardless of the performance of individual labels in the source domain. While there are limitations to the improvements when the accuracy in the original sample is excessively low, in most cases, the accuracy enhancements approach those observed in the sampled data (ImageNet Deng et al. (2009): source domain). This pattern is consistent across all datasets and models utilized in our experiments.
Ablation Study on the Quality Estimator: Confidence (C) vs. Entropy of Logits (E). The confidence score and the entropy of logits are interchangeable approaches, on logits. As shown in Table 5, replacing the VisiT score with the entropy of logits yields performance comparable that of VisiT using the confidence score; however, it does not exceed this performance. Therefore, we opted to introduce confidence as a simpler and more representative metric for use in VisiT' for Lens.
both metrics are based
C DETAILS ON ImageNet-ES Diverse AND ES-Studio Diverse IMPLEMENTATIONS
This section provides details on how AS-Sttdio Diverse is constructed and how ImageNet-ES Diverse collected within the ES-Studio Diverse environment. our previous research (Baek et al., 2024), we developed ES-Studio, enabling individual control over environmental and sensor parameters involved in image acquisition. Utilizing ES-Studio, we compiled ImageNet-ESS, a novel dataset comprising 202,000 samples of perturbed data from the environment and camera sensor domains (referred to as "Luminous").
As we mentioned in section 4, to effectively capture the impact of diverse environmental((gght) perturbations, we construct ES-Studio Diverse and testset of ImageNet-ES Diverse (referred to as "Diverse") based on the designs of ES-Studio and ImageNet-ES. We described the common and different settings between Luminous and Diverse on designs for testbed construction and collection configurations in Table 6 and 7.
18

---

Published as a conference paper at ICLR 2025
Luminous
Diverse
ResNet50
80
20
100
EffiientNee EftilenNeer be ResNet152 Resiee50
80
60
40
100
an
100
an
20
1004
an
Accuracy (%)
Swin-T
20
100
Swin
60
Swin-8
100
OpenCUPpp
60
20
100
DINOV26 OpenCLiAh
a0
a0
100
60
20
100
DINOV2
80
All [baselnn]
ImageNet (Source Domain) Label-wise Accuracy (%)
Figure 11: Generalizability of Lens Based on Label-wise Performance in the Source Domain.
The key changes in Diverse compared to Luminous are: 1) The display medium is changed from a screen to a 1000 banners; and 2) The lighting options are expanded from two to six options. Detailed changes are described below.
19

---

Published as a conference paper at ICLR 2025
Table Ablation study on VisiT: Confidence (C) vs. Entropy (E) of Logits.
ImageNet-ES Luminous
ImageNet-EE Diverse
Num. Params
Pretraining Dataset
Model
method
IN
Naive control
Lens
Naive control
Lens
AE Random
C
E
AE Random
C E
IN-IK
86.0 32.1
50.4
78.6
78.8
17.6
12.0
43.3 42.9
ResNet-50
26M
DeepAugment" MixMMixi
(He et al., 2016)
IN-21K
87.0 53.2
61.4
83.1 83.2 36.2
23.6
65.1 65.2
ResNet-152
60M
IN-IK
87.8 41.1
54.3
81.1
81.6
21.9
14.2
48.8 49.3
(He et al., 2016)
EffciemtNet-B00
5M
IN-IK
88.2 51.8
58.3
81.2
80.5
21.8
14.0
45.9
46.3
Tan & Le, 2019b)
EfficiemNNet-33
12M
IN-IK
88.1 62.0
66.3
83.5
82.9
33.6
21.4
55.7 55.6
(Tan Le, 2019b)
SwinV2-T
28M
IN-IK
90.6 54.1
63.1
82.6
82.1
26.5
50.3 50.0
(Liu al., 2022)
SwinV2-S
50M
IN-IK
91.7 59.9
65.5
84.5
84.9
30.8
55.6 55.8
(Liu al., 2022)
SwinV2-B
88M
IN-IK
91.9 60.0
65.5
85.1
30.8
55.3
(Liu ul., 2022)
OpenCLIP-b
87M
LAION-2B
94.3 66.1
71.3
90.9
90.4
38.8
24.5
67.6 67.1
Cherti et al., 2023)
Text-guided
OpenCLIP-h
pretrain
632M
LAION-2B
94.9 79.0
77.6
93.0
92.9
45.5
29.3
74.4 74.5
(Cherti et al., 2023)
DINOv2-b
90M LVD-142M
93.6 74.5
73.9
90.6
91.1
44.5
28.3
72.9 73.9
(Oquab et al., 2023)
Dataset curation
DINOv2-g
1.1B
LVD-142M
94.7
84.3
79.8
93.1
93.4
62.8
35.3
82.9
83.5
(Oquab et al., 2023)
All models
90.7
59.8
65.6
85.6
85.5
34.2
21.4
59.8
59,9
Lominous: Buok et all (2024), and Tendrycks al., 2021), 1: (elldddryc et ol., 20199, IN: ImugeNet (Le & Yong, 2015), AE: Auto exposure
Ceiling
Camera
Ventilation outlet
Banner
Ventilattoon outlet
5m
(a) External.
(b) Internal.
Figure 12: Actual appearance of ES-Studio Diverse.
Banner:
Print & Material: As shown in Figure 13a, each sample image from the Tiny-ImageNet (Le & Yang, 2015) subset was printed on a separate banner with A4-size (210 mm x 297 mm) with 600 DPI, precisely centered while maintaining the original image's 's aspect ratio. A total of 1,000 banners were produced, each containing single sample image. To prevent image quality degradation due to the printing process and paper material properties, tested various DPIs (72, 300, 600) and paper types. We ultimately selected 600 DPI because it provided the best image quality and minimized distortions and noise caused by reduced
carefully
20

---

Published as a conference paper at ICLR 2025
(a) Sample banner.
(b) Alignment banner.
Figure 13: Banner examples of ImageNet-ES Diverse.
Table 6: Comparison on Testhed between Luminous and Diverse
Comparison
Components
ES IS-Suudio
St-dio Diverse
Specifications
Different
Display for reference
(Alieraioo)
(by controller)
Screen
Bunner (by humans)
Screen: S5-inch ED 4K UHD TV (LG OF OL EDSSIAFNA) Banner: 1000 sizeee PVC with 600 DPI bannerssmple)
Darkmeen
Compleetll enclosed
blackout fabric Brightness controllable ceiling lamps
1.5
Ewironment
Philips
Ambiance
lamps)
Sensor factor
Sensar controllable camera
"Canon EOS RP body 'RP 24-105mm P4-7.1 IS STM less
Contel system
Destrop Computer Will Network
Apple Mac Stadio M2 Max OCAPI" + Philips Huc API
4CCAPI: Canon camera control AP]
expressiveness during printing, among the three DPIs tested. We also chose a PVC banner, as it is resistant to light reflection, humidity, and creasing.
Placement: In a darkroom of ES- ES-Studio Diverse, the banner is fixed in place (Figure 12). To prevent image distortion, the camera's height carefully adjusted, ensuring it's positioned 28 away from the banner in straight line. Furthermore, the banner is securely attached to fixed position on a magnetic blackboard using six magnets. To minimize framing- induced distortion, use "Alignment banner (Figure 13b)" adjust the camera angle, keeping the camera grid and the banner's support line parallel.
Cropping Process: Aside from the cropping process, the preprocessing steps are identical to those in Luminous preprocessing Baek al. (2024), labeling the images using the data collection log. The cropping process extracts the valid area from the collected images using the ROI coordinates. Since the banner and camera framing are fixed, the ROI is identical for captured images with the same reference banner. We developed and use an interactive tool obtain the ROI coordinates: 1) Zoom in on a captured image (taken with AE) for each sample. 2) Using drag interaction, select the valid area, ensuring it includes the sample but has minimal surrounding area. 3) Crop the images for all settings environments, 27 parameter options, and 5 AE shots) using the ROI coordinates.
Lighting Options: As shown in Table 4c, five additional lighting options, varying in magnitude and direction, are defined. 1.5 (Turn off, included Luminous) option is excluded from Diverse because it produces blacked-out, uninformative images in non-luminous scenes.
Except for these two aspects, all other components and configuration settings the same in ImageNet-ES. Moreover, the validation process is the same as that used for ImageNet-ES. and Diverse validated by five individuals to assess the integrity of the collected data. Details for the common
21

---

Published as a conference paper at ICLR 2025
Table 7: Comparison on Configurations of testset collection between Luminous and Diverse
Comparison
Collection Configurations
ImageNet-ES
ImageNet-ES Diverse
Light Options
I.1 (on) 1.5 (off)
L.-I.7 w/o L,
Different
Caputured images
64,000
19,2000
1000 samples
Reference Samples
(Randomly selected samples/ctass in TnnyimageNet validation set) 6240 x 4160 pixels. AF (Auto focus) mode (w/ metering mode)
Common
Frame Setting
AE: Shots, M: 27 parameter options (controlled parameters: Shutter speed, Aperture, ISO)
Sensor Options
AEE Auta Exposure, M. Mannil Coutrll (Details in Table 8)
Table 8: Manual camera sensor parameter setting of test set of Luminous and Diverse
16
198
100
190
1087
100
158
1000
108
100
150
170
100
speed, A: Apetture
design aspects of both testbeds and datasets are described in the supplementary material of our previous research (Baek et al., 2024).
D DETAILS ON SCENE SPECIFIC CAMERA CONTROL CONCEPTS
This section details the Model-Scene specific camera control concepts within our framework. As illustrated in Figure 14, the design of VisiT is scene-specific, acknowledging that optimal parameters can vary significantly across different scenes, even when utilizing the same underlying model. The core principle of scene-specific sensor control that universally optimal, fixed parameter set does not exist. Our solution space analysis (Figures 8g and 8h) demonstrably shows that optimal parameters differ drastically between, for instance, the "Luminous" and "Diverse" scenes. This highlights the necessity of dynamically adapting control strategies to the specific characteristics of each scene.
E DETAILS ON EXPERIMENTAL. SETUPS AND RESULTS
E.
ANOTIER BENCIIMRRK: IMAGENET-ES
Table 9: Environment and Sensor specifics of ImageNet-ES (Back et al., 2024).
Dataset
Original samples
Light
Camera sensor
ISO
Sbutter speed
Aperture
Cuptured images
Test
sampls/Cias))
1,000
On/Off
Auto expostre shons)
Manual (27 options)
25 16000000
Auto
G/4"V12/00 R0/ WIL/ (11/1000")
Auto
LS 13.0/9901116
Auto
$4,000
10,000
Validaation
samples/clas))
1,000
On/Off
Auto exposure shons)
Manual (64 options)
200/809110012200 2800
Auto
(U"4)1 (K1/////vL. (1011"0"11120
Auto
Auto
128,000
10,000
In this paper, two test datasets were used, both of which include extensive natural perturbations in environmental and sensor domains, incorporating 27 manual controls and auto-exposure shots.
For ImageNet-ES (Baek et al., 2024), a subset of Tiny-ImageNet (Le & Yang, 2015) was displayed on monitor and captured using camera. During the process, lighting conditions were varied by turning the lights on and off, and camera parameters were adjusted. Details of the environment and sensor specifications are provided in Table 9. Five images were randomly selected from each of the 200 classes in the Tiny- ImageNet validation set (Le & Yang, 2015). To ensure visual fidelity, each sampled image had a resolution greater than 375 x 500 pixels, avoiding distortion when displayed on the TV screen. In total, 1,000 samples were collected for the test dataset and these samples are used identically in constructing the test set of ImageNet-BB ES Diverse. For the validation set, we employed the same process on non-overlapping set of 10,000 samples, utilizing 64 distinct parameter options to create the ImageNet-ESs validation set. Further details can be found in our previous research (Back et al., 2024). As potential future work, the ImageNet-Be ES Diverse validation set could be constructed by applying the six light options of ImageNet-Ei ES Diverse to the same 10,000 samples and 64 parameter options used in the ImageNet-FS validation set.
22

---

Published as a conference paper at ICLR 2025
pl
p'j
x
K options, Pj fp'm InP 12, 1'
Camera Control
Qs
scenel $1
67
Qsyprrs
Peypp
scene2 S2
G1
K options, P2
Camera Control
2'
2x
X P
Xs2
M Target Neural Network
VisiT
p'i
p'1
Figure 14: Scene-specific design of VisïT.
E.2 EXPERIMENT SETUPS OF
(OOD) DETECTION.
E.2.1 OOD DUUT-OF-DISTRIBUTION) DETECTION FRAMEWORK SETTINGS AND TECHNIQUES
OOD Detection Framework and Datasets. To leverage OOD detection techniques proxy, we employed the Semantic-Centric framework (Yang et al., 2022), a prevalent and widely-adopted approach in OOD research. Existing studies predominantly focus classifying samples either belonging to classes encountered during training or not. Consequently, we structured the dataset configuration for OOD detection training and validation detailed in Table 10. For evaluating the efficacy of OOD detection methods as proxies for validation (quality indicator) and testing (ablation on Visi?',, partitioned the ImageemaeeNet (Le & Yang. 2015) (ImageNet-ES reference dataset) validation set into three distinct subsets: S1, S2, and S3. The validation and test splits of ImageNeetESS were assigned the same images S1 and S2, respectively. The remaining images, comprising 40 images per class, were designated as S3. Given that ImageNee images are provided in resized 64x64 format, corresponding images from the original ImageNet dataset were utilized to maintain the native resolution. This partitioning strategy for Imng-ImageNet is further described in Table 11.
OOD Detection methods. We validate OOD detection techniques on validation set of ImageNet- including ViM (Wang et al., 2022a), ReAct (Sun et al., 2021), ASH (Djurisic et al., 2023) and MSP (Hendrycks Gimpel, 2017). These methods demonstrate state-of-the-an performance and serve as baselines in recent OOD research. To validate current OOD detection methods, we leverage results and APIs provided by OpenOOD (Yang al., 2022). All implementations are based on the OpenOOD package.
E.2.2 MODELS
We selected four models with diverse architectures, widely adopted in OOD detection research, as underlying models to adapt OOD detection methods. Details of these models are provided in Table 12. All model weights used in the OOD detection experiments were obtained from the timm library (Wightman, 2019). As the pre-trained weights produce predictions for 1,000 classes, we fine-tur tuned the classifier of each model to output 200 classes, corresponding to Inng-ImaeNNe. used for fine-tuning, with the feature
Non-resized images from the ImageNt training
23

---

Published as a conference paper at ICLR 2025
Table 10: Datasets Used OOD Detection Experiments
Table 11: Description of partitions of Tiny-
ImageNet validation set (10K samples)
Esperimens
Vahilaiar
Sering
Quality
Ablatinn Study
Semuniis
OperlmngoO (rai))
Teatinos
Pariiia
Ref
(valyee M (82)
cont::
Itet
Table 12: Description of underlying models for OOD detection experiments. (Optimizer: SGD,
Scheduler: Reducel.ROnPlateal, Batch size: 128)
Model
of params
Pretrained
Acc. Valria
Training configuration
EfficientNet-BO Tan & Le (2019a)
4.3M
86.2%
Ir: 5 x 10
epochs: 20
ResNet18 He al. (2016)
11.3M
ImageNet- IK
82.4%
Ir: 5 x 10
epochs: 15
DeïT Touvron et al. (2020)
86M
91.2%
Ir: 5 10
epochs: 20
Swin-B Liu al. (2021a)
86.9M
94.2%
Ir: 5 10-3,
epoccs:: 20
extractor of each model frozen during this process. The specific training configuration and final accuracy also presented in Table 12.
E.2.3 FURTHER ANALYSIS
Figure 15 illustrates the correlation between proxy candidates and image quality on the light Vision Transformer model (DeiT Touvron et al. (2020)), following the methodology outlined in Section 3.2. Consistent with the findings presented in Figure 3, higher confidence score exhibits a strong positive correlation with classification accuracy.
E.3 EXPERIMENT SETUPS OF GENERALIZABILITY
E.3.1 TARGET MODELS SETUP
For all generalization experiments, model weights were sourced from PyTorch. Exceptions to this OpenCLIP-bil and the DG variant (DeepAuument Hendrycks al. (2021) + Augmix Hendrycks et al. (2019)) of ResNet-50, which were obtained from timm Wightman (2019) and Hendrycks et al. (2019), respectively.
E.4 EXPERIMENT SETUPS OF REAL-TIME ADAPTATIONS
E.4.1 TEST-TIME ADAPTATION (TTA) TECINIQUES
In this section, we describe details of the test-time adaptation techniques employed in the real-time adaptation experiments section 5.2.
Prediction-time Batch Normalization (BN1)(Nado et al., 2020): Improves model robustness to covariate shifts by updating Batch Normalization statistics during prediction. computationally efficient as it does not require backward propagation.
Batch Normalization Adaptation (BN2) (Schneider et al., 2020): Dynamically updates the running statisties (running mean and variance) of Batch Normalization during inference. Instead of using test batch statistics in isolation, it continuously updates the running mean and variance as test data is encountered, making adaptation more flexible under varying test batches.
Fully Test-time Entropy Minimization (Tent) (Wang et al., 2021): Adapts to domain shifts during test time by minimizing the output entropy predictions. It updates not only BN statistics but also the entire model's parameters, providing greater robustness across a broader range of test samples, but increasing computational costs.
E.4.2 MODELS
Of the models used in Section 3, we selected two representative lightweight architectures, ResNet- 18 (He et al., 2016) and EfficientNet-BO (Tan & Le, 2019b), for this experiment. These models were chosen due to their compatibility with our three TTA methods, as they include Batch Normalization (BN) layers required for the application of these techniques.
24

---

Published as a conference paper at ICLR 2025
Samile
DST
0.0
1.0
1.0
Confidence Score
VIM Score
Ash Score
ReAct Score
KNN Score
(a) Confidence based.
(b) OOD -Out-of-Distribun score based.
Figure 13: Proxy for image quality assessment on DeiT: Each score normalized between 0 to 1.
Table 13: Detailed results of real-tiime time adaptation for ResNet18 He et al. (2016)
Eas.
18
25
24
27
Lievives
C33
418
618
68.3
788
12.3
136
116
718
116
131
724
733
73
736
73.
73.
734
758
738
13.
118
110
116
131
738
739
C31
33
153
157
227
248
260
361
238
328
3LD
312
316
124
323
33.2
35.
38.2
363
357
352
348
164
248
342
343
344
348
C33
66
11
19
31
25
84
124
16
193
301
201
25.3
363
981
208
366
319
331
113
340
143
346
E.5
DETAILED RESULTS FOR EXPERIMENTS ON REAL-TIME ADAPTATIONS.
Tables 13 and 14 detail the performance real-time adaptation results for the Lens system as the number of input images, k, varies across the complete range from to 27.
Table 14: Detailed results of real-time adaptation for EfficientNet-B Tan & Le (2019b)
Fin.
C332
300
612
15.
114
11.
713
161
715
715
713
716
113
712
118
11.
116
118
113
11
661
127
164
710
725
766
764
768
764
76A
763
79.1
755
113
162
26.5
384
10a
314
318
34.
350
361
364
315
31.
364
364
36.
36.8
362
381
362
36A
362
362
312
36.5
333
316
033
18
31
33.2
36.1
396
F ADDITIONAL REPRESENTATIVE EXAMPLES FROM ImageNet-ES Diverse.
More ImageNet-Bi ES Diverse examples are provided in Figure 16.
25

---

Published as a conference paper at ICLR 2025
255 2000 (EE0)
Image
AE
Manual Parameter Settings
PeSEIIpAR VNDE
(a)
Original Image
Manual Parameter Settings
Euzterppent 1OT VOE: V1003" Anstunn A.
(b)
50 393 208L HEE0
Manual Parameter Settings
Apertnn: E.A
Original Image
AE
Manual Parameter Settings
Redti geat NET SIRT SIEEST
Original Image
7000 TOD0)
Settings
ADutaNn/SAAA SUA TH
Figure 16: ImageNet-ES Diverse samples (a)-(e): Each subfigure demonstrates how image characteris- tics (e.g., brightness, color, sharpness) change based on the sensor parameter settings. Even in manual mode, the image varies significantly depending on environmental conditions (L.1-.7, excluding L5) and parameter adjustments. Although auto exposure fails to produce high-quality images for neural networks, it provides images of consistent quality for humans across various environments. This indicates that sensor parameters and environmental conditions significantly influence the image quality for both neural networks and humans.
26


Reviews:
[{"review_id": "j4VxrnE9D9", "reviewer": "Reviewer_y23X", "summary": "The paper proposes Lens to address domain shift in vision tasks by adapting camera sensor parameters in real-time to optimize images. , Lens utilizes VisiT, a model-specific quality indicator based on confidence scores, to dynamically assess image quality during test time without the need for additional data or retraining. The authors also introduce a new benchmark ImageNet-ES Diverse with varied sensor and lighting conditions.", "questions": "Please see the weakness part. I'm not an expert in this area so I may adjust my score according to other reviewers' reviews and the author's rebuttal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- I'm not an expert in this area but I love the idea of this work. This work improves model accuracy by selecting the optimal sensor settings, enabling it to achieve substantial accuracy improvements even under varying lighting and sensor conditions.\n- Easy to follow. Clear writing.\n- A new benchmark ImageNet-ES Diverse is introduced, which may benefit future research. The author also provided details about data collection and hardware in Appendix.\n- The method is efficient without retraining and can be run in real-time with low latency.\n- Experiments show notable performance improvement compared to baselines.", "weaknesses": "- Authors utilize the model’s confidence score for its prediction on the image as a simple yet effective proxy for image quality. How do you make sure the confidence score is good enough. Is there a case of overconfidence?\n- Could the method deal with rapid environmental shifts where the optimal parameter can change frequently? Possibly leading to unstable.  \n- The performance of Lens is closely tied to the capabilities of the camera sensor. Is it generalizable to new devices, e.g., varying ISO and  aperture ranges?\n- How might Lens perform with other types of deep vision models, such as detection, segmentation, or even VLM models, that require different kinds of spatial or temporal context?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2bem8HrIhn", "reviewer": "Reviewer_Mps5", "summary": "This research paper proposes an adaptive camera sensor control method with the goal to enhance model performance by enabling the capture of high-quality images optimized for vision models. Unlike traditional approaches that rely on static settings, this method dynamically adjusts camera sensor parameters specific to the model requirements, ensuring that images are tailored to the model. Central to this approach is the use of VisiT that provides a confidence and quality indicator for each parameter set, ensuring that the chosen settings maximize image quality for model accuracy and robustness. Additionally, the paper introduces a new dataset, ImageNet-ES Diverse, specifically designed for this purpose, featuring images captured under a variety of sensor and lighting conditions.", "questions": "Experiments and analysis on domain shifts in the real world data would be helpful.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) Addressing model robustness at the sensor level is an interesting approach that is likely to enhance models' resilience to environmental variations.\n2)  ImageNet-ES Diverse is a large dataset. Releasing the data for the community would be useful.", "weaknesses": "1) It would be useful to test this approach in more real-world scenarios. For instance, capturing images in diverse environments, such as indoor scenes (e.g., similar to NYU dataset), outdoor settings (e.g., similar to CityScapes or Waymo datasets), or general scenarios with multiple objects in complex settings (e.g., similar to COCO or ADE20K datasets), would provide a more comprehensive evaluation of the method's robustness.\n\n2) The method leverages VisiT’s ability to assess image quality based on the model's confidence score for a given image. However, it raises the question of whether this approach remains effective when the model encounters object classes or contexts in which its performance is notably weak. These kinds of insights and analysis would be very helpful.\n\n3) Including results from larger, more general multimodal models that utilize extensive training datasets and demonstrate robustness to varied environmental settings would be valuable. For instance, adding evaluations from models like LLaVA, GPT-4, or Claude could provide useful insights into the method's broader applicability.\n\n4) Could you provide more details about Fig. 7c? It appears that the features are not well-clustered for the Lens as well, Isn’t it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gQMVLx9UG5", "reviewer": "Reviewer_v7Km", "summary": "The authors propose  Lens, a camera sensor control method that enhances model performance by capturing\nhigh-quality images from the model’s perspective, rather than relying on traditional human-centric sensor control.\n\nThought is well received.\n\nLens is lightweight and adapts sensor parameters to specific models and scenes in real-time training-free, Lens uses a model-specific quality indicator (VisiT) that evaluates individual unlabeled samples at test time using confidence scores, without additional adaptation costs.\nA new benchmark ImageNet-ES Diverse is introduced to validate Lens. \n\nLens is claimed to be a novel adaptive sensor control system that captures high-quality images robust to real-world perturbations. \nThe core idea of Lens is to identify optimal sensor parameters that allow the target neural network to better discriminate between\nobjects, akin to adjusting a pair of glasses for clear vision.\n\nDetailed experiments have been done, including real physical time analysis. \n\n\nThe limitations are well received. \n\nI just have one thought, would your method be robust against perspective and radial distortion both.", "questions": "Please follow weaknesses and summary.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Paper is well written and easy to follow and understand .\nDetailed experiments are there. \nTwo datasets have been investigated including one that was introduced by the authors.", "weaknesses": "I do not see much weakness. I am simply thinking about radial distortion. \nAnother thought is how easy it is to incorporate the proposed methods in SSL, multi, and Meta-learning. \nPerspective distortion is evident in real-world applications, as indicated in the following papers \na) Möbius Transform for Mitigating Perspective Distortions in Representation Learning\nb) LCM: Log Conformal Maps for Robust Representation Learning to Mitigate Perspective Distortion\nRadial distortion is often evident as well when using cameras like fisheye view. \nBoth distortions distort the visual concepts, and therefore, the models do not perform well.\n\nAuthors are not forced to do experiments. However, they are encouraged to comment on such situations and whether 2'Lens' would be robust against them as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rwW4WjFYuP", "reviewer": "Reviewer_5LQ1", "summary": "This paper proposes a camera sensor control algorithm that actively collects high-quality images based on the model’s perspective. The authors build a training-free, model-specific quality indicator called VisiT to evaluate unlabeled samples at test time without adaptation cost. They also propose a new benchmark based on ImageNet-ES that captures the natural perturbations from changes of sensor and lighting conditions. The experimental results show the effectiveness of the proposed method.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ This work provides a new perspective for TTA by improving the sensor parameters, which is very interesting to strengthen the performance of foundation models.\n\n+ The proposed benchmark provides a fair and diverse comparison for the generalization ability of the TTA in more scenarios.\n\n+ The experimental results show the TTA can increase the accuracy by a sizable margin with different vision models.", "weaknesses": "- The novelty of this paper is limited. Leveraging the logits as a confidence score is widely adopted in many uncertainty-based methods for many applications such as classification and detection. Besides that, I don’t see other technical contribution from the proposed method.\n\n- More details about the used techniques should be given for readers to better understand the intuition of this method. For example, how do you generate the camera parameter candidates? And how do you guarantee the discrete candidate can cover the optimal solution in the continuous parameter space?\n\n- As evaluating the confidence score requires one forward pass of the image, deploying the Lens method seems to significantly increase the inference latency of the original model. Although subset selection method is used in CSA, it may miss the optimal solution. How do you deal with the trade-off?\n\n- Some visualization could be added to show the best camera parameters of different input images and different backbones. Analysis to the results of selection can also be added to give readers more intuition.\n\n- Have you ever tried other proxies that are often used in uncertainty learning such as entropy of logits?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)