# Can less precise be more reliable? A systematic evaluation of quantization's impact on CLIP beyond accuracy

Aymen Bouguerra, Daniel Montoya, Alexandra Gomez-Villa, Fabio Arnez, Chokri Mraidha

► To cite this version:

Aymen Bouguerra, Daniel Montoya, Alexandra Gomez-Villa, Fabio Arnez, Chokri Mraidha. Can less precise be more reliable? A systematic evaluation of quantization's impact on CLIP beyond accuracy. 2025. hal-05327748

HAL Id: hal-05327748

https://hal.science/hal-05327748v1

Preprint submitted on 23 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
# Can Less Precise Be More Reliable? A Systematic Evaluation of Quantization's Impact on CLIP Beyond Accuracy

Aymen Bouguerra,¹ Daniel Montoya,¹ Alexandra Gomez-Villa,² Fabio Arnez¹ Chokri Mraidha¹

¹ Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France

² Computer Vision Center, Barcelona, Spain

{aymen.bouguerra, daniel-alfonso.montoyavasquez, fabio.arnez}@cea.fr, agomezvi@cvc.uab.es

## Abstract

The powerful zero-shot generalization capabilities of vision-language models (VLMs) like CLIP have enabled new paradigms for safety-related tasks such as out-of-distribution (OOD) detection. However, additional aspects crucial for the computationally efficient and reliable deployment of CLIP are still overlooked. In particular, the impact of quantization on CLIP's performance beyond accuracy remains underexplored. This work presents a large-scale evaluation of quantization on CLIP models, assessing not only in-distribution accuracy but a comprehensive suite of reliability metrics and revealing counterintuitive results driven by pre-training source. We demonstrate that quantization consistently improves calibration for typically underconfident pre-trained models, while often degrading it for overconfident variants. Intriguingly, this degradation in calibration does not preclude gains in other reliability metrics; we find that OOD detection can still improve for these same poorly calibrated models. Furthermore, we identify specific quantization-aware training (QAT) methods that yield simultaneous gains in zero-shot accuracy, calibration, and OOD robustness, challenging the view of a strict efficiency-performance trade-off. These findings offer critical insights for navigating the multi-objective problem of deploying efficient, reliable, and robust VLMs by utilizing quantization beyond its conventional role.

<figure><img src="image_2.png" /><figcaption>Figure 1: The dichotomous impact of quantization on zero-shot Performance. WIT models (blue) consistently improve in calibration (left), with several QAT methods achieving simultaneous accuracy gains. In contrast, LAION models (orange) show systematic degradation in calibration (right). The lack of points near the origin suggests that quantization is always impactful.</figcaption></figure>

## 1 Introduction

Vision-Language Models (VLMs), particularly CLIP (Radford et al. 2021), have revolutionized computer vision through their remarkable generalization capabilities. Their powerful zero-shot performance has made them a go-to model for safety-related tasks, particularly out-of-distribution (OOD) detection. Consequently, a significant body of work has emerged to rigorously benchmark their reliability, with large-scale evaluations like OpenOOD and ImageNet-X specifically designed to probe the OOD robustness of foundation models (Zhang et al. 2024; Noda et al. 2025; Miyai et al. 2025; Mayilvahanan et al. 2023; Tu, Deng, and Gedeon 2023).

However, the substantial computational requirements of these models pose a significant barrier to real-world deployment. To mitigate this, model quantization has become the

standard compression method, dramatically reducing memory and computational overhead by lowering the precision of the model's weights (Courbariaux, Bengio, and David 2015; Esser et al. 2020). These two domains (rigorous reliability benchmarking and efficiency-driven quantization) have evolved in parallel, creating a critical blind spot. While the community has extensively characterized the intrinsic robustness of full-precision CLIP, it remains largely unknown if these safety-critical properties survive the aggressive, lossy compression of quantization. Our work directly addresses this gap.

Despite quantization's popularity, current evaluation practices suffer from a critical limitation: an overwhelming focus on accuracy-based metrics while neglecting essential reliability considerations. This narrow paradigm is particularly problematic for safety-critical applications, where model trustworthiness extends far beyond classification accuracy. The oversight becomes more concerning when compressed
models are deployed in real-time systems, where reliability
failures would have immediate consequences.

To address this gap, we move beyond conventional
accuracy-focused evaluation and ask:

How does quantization impact VLMs' reliability?

To answer this question, we systematically evaluate quan-
tization's impact across four critical reliability dimensions:
(1) Robustness to quantization noise, examining stability
across bit-widths and methods; (2) Uncertainty quality and
calibration, evaluating whether models provide reliable con-
fidence estimates; (3) Out-of-distribution detection, evaluat-
ing whether the model can distinguish between two to more
semantic distribution of samples; and (4) Distribution shift
robustness, assessing performance under realistic data vari-
ations.

We evaluate these four dimensions by applying a suite of
quantization techniques to CLIP models pre-trained on WIT
and LAION, and testing them on rigorous benchmarks de-
signed to assess calibration, OOD detection, and data-shift
robustness. Our analysis reveals interesting findings that are
partially illustrated in Figure 19, showing that quantization
always has a measurable impact beyond accuracy, affect-
ing all reliability attributes, indicating a complex interplay
between pre-training source and compression strategy. Our
findings challenge the view of quantization as a simple com-
pression or regularization tool, and suggest it can be used
as a complex operator that could predictably improve both
model efficiency and reliability. In summary, our contribu-
tions are threefold:

1. We reveal a fundamental dichotomy where the direct impact of quantization is critically dependent on the model's pre-training source. The same quantization procedure affects the same VLMs very differently depending on their training data. We also highlight a post-quantization adaptation that drastically improves calibration.

2. We demonstrate a surprising decoupling of reliability metrics, showing that quantization can degrade a model's accuracy and calibration while simultaneously preserving or even improving its out-of-distribution detection capabilities.

3. We uncover how quantization, and more specifically, quantization-aware training, enables the model to retain coarse-grained features while suppressing the fine-grained ones, increasing robustness to detail-suppressing noise like blur while decreasing it to covariate shift and spurious correlations.

## 2 Related Work

**Quantization of Large Pre-trained Models.** Quantiz-
ing large models involves a trade-off between data-free
Post-Training Quantization (PTQ) (Jacob et al. 2018) and
the more accurate but data-dependent Quantization-Aware
Training (QAT) (Courbariaux, Bengio, and David 2015).
Applying QAT to foundational models requires fine-tuning
on training or proxy datasets, creating a significant risk of
catastrophic forgetting. While parameter-efficient methods

like LoRA (Hu et al. 2021) help, the core tension between
adaptation and forgetting remains an open problem.

**Quantization as a Regularizer.** Quantization is increas-
ingly understood as an implicit regularizer that can improve
generalization by finding flatter, more robust minima in the
loss landscape (Hochreiter and Schmidhuber 1997; Tallec,
Blier, and Ollivier 2023; Saqib, Hieu, and Mathieu 2025).
However, this effect has only been studied on weight-only
quantization and exclusively through the lens of accuracy
and domain generalization. Its impact on a wider range of
reliability metrics, and how it interacts with a model's pre-
training source (e.g., WIT (Radford et al. 2021) vs. LAION
(Schuhmann et al. 2022)), remains largely unexplored. In
this work, we will consider quantization as a complex op-
eration that can act as a regularizer, and study its impact on
several reliability detentions, where each benchmark aims to
answer two critical questions: what does quantizing a model
do to its attributes, and how can these changes help us un-
derstand quantization to the fullest

**Benchmarking VLM Reliability.** The evaluation of VLM robustness and reliability has matured, with sophisticated benchmarks for OOD detection (OpenOOD) (Yang, Zhou, and Liu 2022; Zhang et al. 2024; Wang et al. 2024) and spurious correlations (CounterAnimal) (Hochlehnert et al. 2025). While these benchmarks have been instrumental in characterizing full-precision models, their application to systematically study the impact of architectural interventions like quantization is nascent. Closely related to our work, Tu, Deng, and Gedeon (2023) evaluate CLIP's robustness to distribution shifts, OOD detection, and predictive uncertainty. In contrast, in this work, we go one step further, considering the impact of CLIP quantization, a key requirement for computationally efficient deployment.

## 3 Experiments and Results

Trustworthy AI systems require deep learning models to
be developed with a focus on robustness and resilience as
a means to ensure a reliable and safe deployment (Euro-
pean Parliament and Council of the European Union 2024;
Arnez Yagualca 2023; Hendrycks et al. 2021b). Conse-
quently, we assess the impact of quantization on CLIP's
reliability in zero-shot settings and across four key dimen-
sions: (1) Robustness to quantization noise, (2) Impact on
predictive uncertainty, (3) OoD detection, and (4) Robust-
ness to distribution shift. In the first dimension, we exam-
ine performance stability across bit-widths and quantization
schedules. Next, we assess the CLIP's capacity to provide
reliable confidence estimates by observing the calibration of
predicted probabilities. Then we evaluate CLIP's capacity to
detect distribution shifts using its outputs and joint vision-
language features. Finally, we examine accuracy and shift-
detection under realistic data variations & perturbations.

In our experiments, we quantize only the visual encoder,
using the frozen text encoder as a stable semantic anchor to
mitigate semantic drift during adaptation (Li et al. 2022b).
This allows us to reduce the computational requirements as
the text embeddings are only computed once. Crucially, all
downstream classification and OOD evaluation datasets are
used exclusively for testing. They are never seen or used dur-
ing any quantization or adaptation phase, ensuring a strict
zero-shot evaluation of all quantization and discrimination
methods. We considered two pretrained sources, two vi-
sual backbones, 10 quantization methods, and 4 types of
weight/activation bit-width across 10 datasets. In total, we
conducted 4560 evaluations.

3.1 Experimental Setup

**Model Architectures.** We conduct our experiments on various pre-trained Open-CLIP models (e.g., ViT-B/32, ViT-L/14) stemming from different pre-training sources (e.g., WIT, LAION).

**Quantization strategies.** Our evaluation covers multiple quantization techniques from data-free post-training quantization (PTQ) (Jacob et al. 2018; Xiao et al. 2023) to quantization-aware training (QAT) (Courbariaux, Bengio, and David 2015). QAT is performed on the CC3M proxy dataset (Sharma et al. 2018)—distinct from the models’ original WIT or LAION pre-training data—using advanced methods like LSQ (Esser et al. 2020) and LoRA-accelerated distillation (Polino, Pascanu, and Alistarh 2018; Hu et al. 2021). Our primary focus is Light QAT, which uses minimal iterations and samples to adapt to the quantization grid while mitigating overfitting to the proxy data (Kirkpatrick et al. 2017) or forgetting previously acquired knowledge.

**Out-of-distribution Detection Methods.** We employ a diverse set of OOD scoring functions, categorized into two groups. The first includes classic, output-based methods that operate on the final logits: maximum softmax probability (MSP) (Hendrycks and Gimpel 2017) and the Energy Score (Liu et al. 2020). The second, more sophisticated group consists of VLM-specific methods that operate directly within the joint embedding space. This includes maximum concept matching (MCM), which calculates the maximum similarity score between an image and the set of text embeddings for all in-distribution class names; a low score indicates the image is semantically distant from all known concepts and thus likely OOD (Ming and Li 2022). We also use methods based on negative text prompts, such as a Negative Label (Li et al. 2023) score and a Generic Negative score. These methods gauge novelty by measuring an image's similarity to explicitly out-of-distribution or generic negative concepts (e.g., “*a photo of something*”), a technique explored for its robustness in large-scale models

**Evaluation Metrics.** We employ a suite of standard met-
rics to assess model performance and reliability:

*   **Accuracy:** Standard top-1 classification accuracy.

*   **ECE:** Expected Calibration Error, to measure the alignment of confidence with accuracy (Guo et al. 2017).

*   **NLL:** Negative Log-Likelihood evaluates the probabilis-
    tic correctness of a model's predictions, heavily penal-
    izing overconfident errors (Bishop 2006). Please refer to
    the appendix for NLL results.

<figure><img src="image_3.png" /><figcaption>Figure 2: Average In-distribution accuracy change for WIT (blue) and LAION (orange) sources for both ViT/B-32 and ViT/L-14 backbones under various quantization methods, relative to the FP32 baseline (0%).</figcaption></figure>

*   **AUROC:** Area Under the ROC Curve, for threshold-independent OOD detection performance (Fawcett 2006).

*   **FPR@95TPR:** False positive rate at 95% true positive rate, to evaluate a practical OOD detection operating point (Hendrycks and Gimpel 2017).

**Benchmark Datasets** Our benchmark datasets, inspired by recent surveys on OOD detection (Yang et al. 2022; Zhang et al. 2024) in VLMs (Yang, Zhou, and Liu 2022; Miyai et al. 2025), categorize datasets in 4 groups along two axes: *covariate Shift* and *semantic Shift*. Please refer to the appendix for an illustration of our experimental protocol.

1.  **In-Distribution (ID):** Standard sets including ImageNet-1k (Deng et al. 2009) and CIFAR (Krizhevsky and Hinton 2009).

2.  **Data-shifted ID (Covariate Shift):** Datasets with in-
input perturbations, including CIFAR100-C, ImageNet-C
(Hendrycks and Dietterich 2019), -R, -A (Hendrycks
et al. 2021a), -Sketch (Wang et al. 2019), and -V2 (Recht
et al. 2019).

3.  **Near-OOD (Semantic Shift):** Datasets with moderate semantic shifts and subsets of Imagenet1k, including ImageNet-X, a challenging benchmark for real-world OOD evaluation (Noda et al. 2025).

4.  **Far-OOD (Semantic Shift):** Datasets with distinct categories, including SUN-397 (Xiao et al. 2010), Places365 (Zhou et al. 2017), iNaturalist (Van Horn et al. 2018), and DTD (Cimpoi et al. 2014).

3.2 Robustness to Quantization Noise

**Impact on Zero-Shot Accuracy: Model scale inverts the relationship between pre-training noise and quantization robustness.** Our results reveal a complex, scale-dependent interplay between the regularization from noisy pre-training data and the subsequent pressure of quantization, a phenomenon reminiscent of the “double descent” behavior where more capacity can paradoxically improve generalization (Nakkiran et al. 2021). At the scale of ViT-B/32, as shown in Figure 2, the LAION-pretrained model (orange) is more fragile to quantization than its WIT counterpart (blue).
We hypothesize that for this smaller model, the regularization from noisy data consumes its limited parameter redundancy. The subsequent application of quantization (Courbariaux et al. 2016) then acts as a second injection of noise, forcing the model past a critical point in the information bottleneck and causing it to lose essential, task-relevant information (Tishby, Pereira, and Bialek 2000).

However, this trend is not universal and, counterintuitively, reverses in larger models. For a ViT-L/14 model, we observe that the LAION pre-training yields a model **more** robust to quantization. In this high-capacity regime, large neural networks are known to be highly overparameterized, containing significant redundancy in their weights (Frankle and Carbin 2019). Here, the noisy data acts as a beneficial regularizer, and there remains ample superfluous capacity for quantization to compress without harming the core learned representations. Therefore, the impact of pre-training data on a model’s "quantizability" is not a fixed characteristic but is critically mediated by model capacity.

**Less precise can also be more accurate.** As illustrated in Figure 2, several quantized models have surpassed their full precision counterparts, improving accuracy beyond the baseline by acting as an implicit sharpness-aware optimizer. Please refer to the appendix for further details about this phenomenon.

**Robustness to Quantization Aggressiveness: Quantization methods are not made equal.** While our previous results show that accuracy can be recovered by most methods with ease for 8-bit quantization, we find it extremely hard to replicate using lower precisions. As shown in Figure 3, all models degrade as precision decreases, but methods that optimize the quantized representation range are highly more resilient to extremely low precision. Simpler QAT techniques, like basic 'QAT-LoRA', are brittle and fall off a "quantization cliff" at 4-bit precision, with performance collapsing to near-zero. The performance inversion when using advanced methods is puzzling: the LAION model, which was less robust at 8-bits, now significantly outperforms the WIT model when quantized to 4-bits. This suggests that while the WIT model is easier to compress naively, the LAION model's representations, when guided by a powerful regularizer like distillation, can be molded into a more fundamentally robust low-bit state, overcoming its initial fragility.

**Robustness to Catastrophic Forgetting during QAT: The optimal QAT duration is a trade-off between adapting to quantization and catastrophic forgetting on the proxy dataset.** A core challenge in deploying large pre-trained models is that the original training data is often inaccessible, forcing any adaptation to be performed on a smaller proxy dataset. This turns QAT into a high-stakes balancing act. As shown in Figure 4, the optimal number of training iterations is dictated by the tension between two opposing forces: adapting to quantization noise and forgetting the model's original general-purpose knowledge by overfitting to the narrow proxy dataset. Although we achieve better performance on 'w8a8' using light QAT, adaptation using more diverse data and for longer iterations becomes required as

<figure><img src="image_4.png" /><figcaption>Figure 3: Robustness to Decreasing Quantization Precision. Average ID accuracy vs. bit-width. While simpler QAT methods collapse at 4-bit precision, advanced methods are more robust.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Figure 4: Accuracy evolution of quantized model accuracy relative to iteration steps; higher precision models exhibit catastrophic forgetting behavior while lower precision models struggle to recuperate.</figcaption></figure>

we reduce bit-width precision.

At higher precision (e.g., 'w8a8'), the quantization noise is minimal. The model requires only a short QAT schedule to adapt its weights. Further training provides diminishing returns and increases the risk of catastrophic forgetting, explaining the quick performance plateau, then degradation.

In contrast, at ultra-low precision (e.g., 'w4a6'), the aggressive quantization effectively destroys the information encoded in the weights, inducing a state of precision-based amnesia. The model has insufficient parameter redundancy to function. In this regime, a long QAT schedule is mandatory for recovery. The model must use the proxy dataset to re-learn its function from scratch within the highly constrained low-bit space. This difficult search through a complex optimization landscape requires thousands of iterations to succeed (Gong et al. 2019). The upward curve shows that recovering from precision-induced forgetting is the dominant and necessary effect, even at the cost of specializing to the proxy dataset.

## 3.3 Uncertainty quality and calibration

**Direct quantization's impact on calibration is directly tied to its initial, pre-quantization calibration state** Our findings reveal a stark dichotomy in how quantization impacts model calibration, a key measure of reliability. As shown in Figure 5, QAT consistently **improves** calibration for WIT-trained models (15% ECE improvement on average when using basic QAT), where its regularizing effect (Talec, Blier, and Ollivier 2023) appears to correct for underconfidence. Conversely, the same methods systematically **degrade** calibration for LAION-trained models, suggesting
<figure><img src="image_6.png" /><figcaption>Figure 5: Impact of QAT Methods on CLIP Model Calibration on ViT-B/32. Violin plots show Relative ECE Change (%), comparing LAION (left, blue) and WIT (right, orange) pre-training. Negative values signify calibration improvement. Black crosses represent runs with less than 2% accuracy degradation. Please refer to our appendix for ViT-B/16 and ViT-L/14 results.</figcaption></figure>

the added quantization noise exacerbates poor priors learned from web-scale data. This effect holds even for models with no accuracy loss (black crosses), exposing a critical reliability trade-off for safety-critical applications where predictive uncertainty is paramount.

**QAT's optimization in the quantized domain directly re-shapes the final logit distribution, but the outcome is fundamentally dependent on the model's initial state.** TQAT minimizes task loss under quantization noise, directly altering the final logits that determine confidence. For under-confident WIT models, this optimization beneficially sharpens the output distribution, increasing confidence to match empirical accuracy, as seen in Figure 6.

Conversely, this can be detrimental for LAION models. While quantization correctly regulates the highest-confidence logits, it perversely boosts the logits of already overconfident low-to-mid certainty predictions. This confidence distortion is not an artifact of accuracy loss, as it persists even when accuracy is maintained at larger scales. This demonstrates that the regularization from quantization is not a simple smoothing but a complex reshaping of the logit structure, contingent on the model's initial properties.

A standard reliability diagram is insufficient to reveal these dynamics, as it compares bin populations that are not composed of the same samples before and after QAT. The bin-wise evolution in Figure 6(b) is thus essential, as it explicitly follows the same initial groups of samples to deconstruct these internal, underlying transformations.

**A re-adaptation of the logit scale is crucial to correct the logit magnitude distortion introduced by QAT.** While QAT adapts the weights, it simultaneously alters the dynamic range of the final logit vectors, rendering the pre-trained logit scale suboptimal. This mismatch is a known issue in large pre-trained models, which often require re-calibration before deployment (Desai and Durrett 2020). A second recalibration step is therefore necessary. This step is particularly effective for the LAION model. As QAT distorts

<figure><img src="image_7.png" /><figcaption>Figure 6: Direct Impact of QAT on Calibration. These plots show the state after only QAT has been applied. The process performs a coarse correction, improving calibration by squeezing the confidences towards the center axis. Please refer to our appendix for the dataset-specific bin-wise shift.</figcaption></figure>

their logit magnitudes, exacerbating overconfidence without necessarily altering the 'argmax' or the relative ordering of the logits. Logit scale adaptation directly targets and corrects this magnitude distortion. By learning a single temperature parameter (Kull et al. 2019), we drastically reduce both the inherent overconfidence as well as the one introduced by QAT and pull the bins back towards the diagonal. Figure 7 shows the final, superior state after this adaptation, demonstrating its necessity for achieving a well-calibrated model. Note that we only use ID data or classification datasets for evaluations.

**QAT squeezes confidence, while logit scale tuning adapts them to the quantized model's accuracy** The aggregated plots in Figure 6.b summarize the two-phase re-calibration. QAT's regularization acts as a structural operator on the logit vector. The optimization under quantization noise penalizes extreme logit values; large positive logits, which create high confidence, are brittle to weight perturbations, while near-zero logits, creating low confidence, are unstable. QAT finds a robust middle ground by attenuating large logits and amplying small ones, resulting in the observed confidence-
<figure><img src="image_8.png" /><figcaption>(a) Reliability Curves</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>(b) Aggregated Bin-Wise Shift</figcaption></figure>

<caption>Figure 7: Final State after QAT & Logit Scale Re-Adaptation. After adapting the logit scale to the new quan-tized model, calibration is further improved. Please refer to our appendix for the dataset-specific bin-wise shift.</caption>

squeezing effect towards the dashed line in Figure 6(b).

However, this mechanical confidence-squeezing is agnos-
tic to the model's actual accuracy, leading to a new, often
mismatched, calibration state as seen in Figure 6(b). This is
where logit scale adaptation becomes critical. It is this fi-
nal, global correction that aligns confidence with accuracy,
moving the model to the superior state shown in Figure 7(b).

## 3.4 Impact on Out-of-Distribution Detection

**ID Accuracy does not necessarily correlate with better OOD detection.** The apparent decoupling of OOD performance from accuracy is explained by where in the model the OOD score is calculated. As shown in Figure 9, methods that rely on the final output confidence, like MSP and Energy (square marker), fail catastrophically for the quantized LAION model. This is a direct consequence of our earlier findings: because QAT severely distorts the LAION model's output logits, the core assumption of these methods—that OOD samples will have lower confidences (Hendrycks and Gimpel 2017).

In stark contrast, methods that operate on the model's

<figure><img src="image_10.png" /><figcaption>Figure 8: Divergent Impact of QAT on Robustness to Co-variate Shift. QAT improves robustness on low-level shifts but degrades it on complex ones like ImageNet-V2, -S, -R and -R.</figcaption></figure>

deeper representations remain effective. The resilience of
the VLM methods strongly suggests that even if quantization
degrades the quality of the max logit magnitudes, it largely
preserves the information in deeper feature space. We also
note that quantization can improve out-of-distribution de-
tection, illustrated by the colored vertical lines exceeding
the full precision baseline. Please refer to the appendix for
FPR@95 results.

3.5 Robustness to Covariate Shift

QAT enhances robustness to low-level, information-suppressing corruptions but degrades generalization on more complex covariate shifts. To evaluate how quantization affects robustness to covariate shifts. The results, shown in Figure 8, vary based on the nature of the distribution shift.

For common, low-level corruptions like blur and noise
(top three rows) on CIFAR, QAT acts as a powerful regu-
larizer. It not only improves the model's accuracy on these
corrupted images (Figure 8) but also substantially boosts its
ability to detect them as out-of-distribution (please refer to
the appendix for the data-shift detection results). This "rare
double win" as these goals are contradictory. These results
suggest that the QAT process makes the model's represen-
tations less sensitive to simple input perturbations, a known
benefit of certain regularization techniques.

However, this trend reverses for more complex and chal-
lenging datasets like ImageNet, and on semantic-style co-
variate shifts like ImageNet-R (renditions) and ImageNet-
A (adversarial examples). For these datasets (bottom four
rows), QAT consistently slightly degrades the model's accu-
racy. This can be explained by its tendency to ignore fine-
grained detail, whether they are meaningful or not, and fo-
cus on coarse-grained features that could penalize general-
ization, which is not a universal robustness enhancer, and its
benefits are specific to the type of distribution shift encoun-
tered.

**Robustness to Spurious Correlations: Quantization am-
plifies reliance on spurious correlations by degrading
fine-grained shape features, forcing the model to rely on
coarse-grained textures.** This occurs because quantiza-
tion's limited bit-depth creates an information bottleneck.
<figure><img src="image_11.png" /><figcaption>Figure 9: Impact of quantization on OOD Detection (AUROC). Average AUROC across quantization methods (higher is better). QAT methods (center, right) maintain OOD performance for the LAION model, despite this model suffering from significant accuracy and calibration degradation. VLM-specific OOD methods consistently outperform classic methods. Vertical lines represent the maximum improvement and degradation relative to the full precision baseline of that experiment.</figcaption></figure>

It disproportionately degrades the high-frequency signals corresponding to fine-grained object shapes while preserving the more robust, low-frequency signals of coarse background textures. When QAT optimizes the model under this constraint, it finds the path of least resistance: it learns to rely more heavily on the now-dominant texture cues to minimize loss on the proxy dataset. This active re-weighting of feature importance towards texture bias explains why the model’s performance collapses when the spurious background cue is removed (Teney, Abbasi, and van den Hengel 2022). We test this vulnerability using datasets with spurious animal-background correlations, and then with uncommon backgrounds (e.g., a polar bear in the jungle), CounterAnimals (Hochlehnert et al. 2025). As shown in Table 1, QAT consistently increases the model’s vulnerability, making it more easily fooled.

To verify these claims, we analyze the 2D Fourier spectrum of the model’s feature maps. As shown in Figure 10, the full-precision (FP32) spectrum contains significant energy in both the low-frequency (coarse-grained, center) and high-frequency (fine-grained, outer) bands. Both PTQ and QAT act as low-pass filters, suppressing the magnitude of the high-frequency components, which correspond to fine-grained textures and sharp edges. The relative error plots confirm this, showing that the quantization error is highest in these high-frequency bands. This provides an explanation for our earlier finding on spurious correlations and data-shift robustness: Quantization forces the model to rely on the more general, less-changing features.

## 4 Conclusion

This work reframes quantization not as a simple compression tool, but as a complex operator with profound and often multidimensional impacts on model reliability. Our primary finding is that a model’s pre-training source is a fundamental determinant of its response to quantization, creating a stark dichotomy where the same technique can improve the reliability of one model while severely degrading another.

<table><caption>Table 1: Quantization exacerbates vulnerability to spurious correlations. The final column isolates the additional accuracy drop due to QAT, a direct measure of increased vulnerability.</caption><thead><tr><th>Model</th><th>FP32 Vuln.</th><th>QAT Vuln.</th><th>Added Vuln.</th></tr></thead><tbody><tr><td>WIT<br>(Normal → Counter)</td><td>16.7%<br>(83.1 → 66.4)</td><td>17.9%<br>(82.5 → 64.6)</td><td>+1.2%</td></tr><tr><td>LAION<br>(Normal → Counter)</td><td>23.1%<br>(84.0 → 60.9)</td><td>24.8%<br>(82.5 → 57.7)</td><td>+1.7%</td></tr></tbody></table>

<figure><img src="image_12.png" /><figcaption>Figure 10: Frequency-domain impact of quantization. Top: The Fourier spectrum of feature maps for FP32, PTQ, and QAT. Bottom: The relative error introduced by PTQ and QAT, respectively.</figcaption></figure>

We demonstrate that this manifests as a critical trade-off: by depressing fine-grained learned features, quantization can improve robustness to high-frequency noise while simultaneously amplifying reliance on spurious correlations and degrading generalization on complex covariate shifts. Furthermore, we reveal a surprising decoupling of reliability metrics, where a model’s calibration can be severely degraded even as its OOD detection capabilities remain robust.
This work encourages the community to advance quantization beyond a simple compression method and toward a multi-objective optimization tool. Our findings reveal that quantization's impact is not a fixed trade-off but a malleable outcome dependent on the method and model. This presents an opportunity to develop new quantization strategies that are not merely designed to preserve accuracy but are explicitly optimized to simultaneously enhance both model efficiency and reliability.

## References

Arnez Yagualca, F. A. 2023. *Deep neural network uncertainty runtime monitoring for robust and safe AI-based automated navigation*. Theses, Université Paris-Saclay.

Bishop, C. M. 2006. *Pattern recognition and machine learning*. Springer.

Bondarenko, Y.; Chiaro, R. D.; and Nagel, M. 2024. Low-Rank Quantization-Aware Training for LLMs. arXiv:2406.06385.

Cimpoi, M.; Maji, S.; Kokkinos, I.; Mohamed, S.; and Vedaldi, A. 2014. Describing textures in the wild. In *Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)*, 3606–3613.

Courbariaux, M.; Bengio, Y.; and David, J.-P. 2015. Binaryconnect: Training deep neural networks with binary weights during propagations. In *Advances in neural information processing systems (NIPS)*, volume 28.

Courbariaux, M.; Hubara, I.; Soudry, D.; El-Yaniv, R.; and Bengio, Y. 2016. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. In *Advances in neural information processing systems (NIPS)*, volume 29.

Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In *2009 IEEE conference on computer vision and pattern recognition*, 248–255. IEEE.

Desai, S.; and Durrett, G. 2020. Calibration of Pre-trained Transformers. In *Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, 3979–3991.

Esser, S. K.; McKinstry, J. L.; Bablani, D.; Mallya, A.; Appuswamy, R.; and Rath, D. 2020. Learned step size quantization. In *International Conference on Learning Representations (ICLR)*.

European Parliament and Council of the European Union. 2024. Artificial Intelligence Act. https://artificialintelligenceact.eu/fr/article/15/. Regulation (EU) 2024/1689. Specifically referencing Article 15 on 'Accuracy, robustness and cybersecurity'. Accessed: 2025-08-01.

Fawcett, T. 2006. An introduction to ROC analysis. *Pattern recognition letters*, 27(8): 861–874.

Frankle, J.; and Carbin, M. 2019. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In *International Conference on Learning Representations (ICLR)*.

Gong, R.; Liu, X.; Jiang, S.; Li, T.; Fua, P.; and Yan, S. 2019. Differentiable soft quantization: Bridging full-precision and

low-bit neural networks. In *Proceedings of the IEEE/CVF international conference on computer vision (ICCV)*, 4852–4861.

Guo, C.; Pleiss, G.; Sun, Y.; and Weinberger, K. Q. 2017. On calibration of modern neural networks. In *International conference on machine learning (ICML)*, 1321–1330. PMLR.

Hendrycks, D.; Basart, S.; Mu, N.; Kadavath, S.; Wang, F.; Dorundo, E.; Desai, R.; Zhu, T.; Parajuli, S.; Hvilshoj, M.; et al. 2021a. The many faces of robustness: A critical analysis of out-of-distribution generalization. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 8340–8349.

Hendrycks, D.; Carlini, N.; Schulman, J.; and Steinhardt, J. 2021b. Unsolved problems in ml safety. *arXiv preprint arXiv:2109.13916*.

Hendrycks, D.; and Dietterich, T. 2019. Benchmarking neural network robustness to common corruptions and perturbations. In *International Conference on Learning Representations (ICLR)*.

Hendrycks, D.; and Gimpel, K. 2017. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In *International Conference on Learning Representations (ICLR)*.

Hochlehnert, A.; Bhatnagar, H.; Udandarao, V.; Albanie, S.; Prabhu, A.; and Bethge, M. 2025. A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility. *arXiv preprint arXiv:2504.07086*.

Hochreiter, S.; and Schmidhuber, J. 1997. Flat minima. *Neural computation*, 9(1): 1–42.

Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. LoRA: Low-Rank Adaptation of Large Language Models. *arXiv preprint arXiv:2106.09685*. Published at the International Conference on Learning Representations (ICLR) 2022.

Jacob, B.; Kligys, S.; Chen, B.; Zhu, M.; Tang, M.; Howard, A.; Adam, H.; and Kalenichenko, D. 2018. Quantization and training of neural networks for efficient integer-arithmetic-only inference. In *Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)*, 2704–2713.

Kar, P.; Arık, S. O.; Choi, D.; Bhattacharjee, B.; Lien, A.-T.; and Pfister, T. 2023. LoCoOp: Few-Shot Out-of-Distribution Detection via Prompt Learning. In *The Eleventh International Conference on Learning Representations (ICLR)*.

Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; et al. 2017. Overcoming catastrophic forgetting in neural networks. *Proceedings of the national academy of sciences*, 114(13): 3521–3526.

Krizhevsky, A.; and Hinton, G. 2009. Learning multiple layers of features from tiny images. Technical report, University of Toronto.

Kull, M.; Perello-Nieto, M.; Käng, M.; Filho, T. M.; Song, H.; and Flach, P. 2019. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. In *Advances in Neural Information Processing Systems (NeurIPS)*, volume 32.
Lee, K.; Lee, K.; Lee, H.; and Shin, J. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In *Advances in neural information processing systems (NIPS)*, volume 31.

Li, Y.; Xu, S.; Zhang, B.; Cao, X.; Gao, P.; and Guo, G. 2022a. Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer. In *Advances in Neural Information Processing Systems (NeurIPS)*.

Li, Z.; Cui, C.; Liu, X.; Zhang, Y.; Chang, S.; Cheng, H.; Cheng, Y.; and Chen, J. 2022b. CLIP-Q: Turning full-precision CLIP into a 4-bit model. In *Advances in Neural Information Processing Systems (NeurIPS)*, volume 35, 24031–24043.

Li, Z.; Wang, F.; Zhang, Z.; and Li, F. 2023. NegCLIP: A Negative-Prompt-based Method for OOD Detection in Vision-Language Models. *arXiv preprint arXiv:2310.03114*.

Liu, W.; Wang, X.; Owens, J.; and Li, Y. 2020. Energy-based out-of-distribution detection. In *Advances in Neural Information Processing Systems (NeurIPS)*, volume 33, 21464–21475.

Mayilvahanan, P.; Wiedemer, T.; Rusak, E.; Bethge, M.; and Brendel, W. 2023. Does CLIP's Generalization Performance Mainly Stem from High Train-Test Similarity? *arXiv preprint arXiv:2310.09562*.

Ming, Y.; and Li, Y. 2022. Delving into the Open-Set World: A Framework for Unsupervised Out-of-Distribution Detection. In *European Conference on Computer Vision (ECCV)*.

Miyai, A.; Yang, J.; Zhang, J.; Ming, Y.; Lin, Y.; Yu, Q.; Irie, G.; Joty, S.; Li, Y.; Li, H. H.; Liu, Z.; Yamasaki, T.; and Aizawa, K. 2025. Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey. In *Transactions on Machine Learning Research (TMLR)*.

Nakkiran, P.; Kaplun, G.; Bansal, Y.; Yang, T.; Barak, B.; and Sutskever, I. 2021. Deep double descent: Where bigger models and more data hurt. *Journal of Statistical Mechanics: Theory and Experiment*, 2021(12): 124003.

Noda, S.; Miyai, A.; Yu, Q.; Irie, G.; and Aizawa, K. 2025. A Benchmark and Evaluation for Real-World Out-of-Distribution Detection using Vision-Language Models. *arXiv preprint arXiv:2501.18463v1*.

Polino, A.; Pascanu, R.; and Alistarh, D. 2018. Quantization-aware knowledge distillation. In *International Conference on Learning Representations (ICLR) Workshop*.

Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In *International conference on machine learning (ICML)*, 8748–8763. PMLR.

Recht, B.; Roelofs, R.; Schmidt, L.; and Shankar, V. 2019. Do ImageNet Classifiers Generalize to ImageNet? In *International Conference on Machine Learning (ICML)*, 5389–5400. PMLR.

Saqib, J.; Hieu, L.; and Mathieu, S. 2025. QT-DoG: Quantization-aware Training for Domain Generalization.

In *International Conference on Learning Representations (ICLR)*.

Schuhmann, C.; Beaumont, R.; Vencu, R.; Gordon, C.; Wightman, R.; Cherti, M.; Coombes, T.; Katta, A.; Mullis, C.; Wortsman, M.; et al. 2022. LAION-5B: An open large-scale dataset for training next generation image-text models. In *Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track*.

Shao, W.; Zhao, L.; He, Z.; Jiao, Z.; Chen, P.; and Ng, K.-T. 2023. OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models. In *The Eleventh International Conference on Learning Representations (ICLR)*.

Sharma, P.; Ding, N.; Goodman, S.; and Soricut, R. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset. In *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, 2556–2565.

Tallec, C.; Blier, L.; and Ollivier, Y. 2023. Revisiting the Regularization Effect of Quantization. *arXiv preprint arXiv:2310.03113*.

Teney, D.; Abbasi, E.; and van den Hengel, A. 2022. On the Pitfalls of Spurious Correlations for OOD Generalization. In *International Conference on Learning Representations (ICLR)*.

Tishby, N.; Pereira, F. C.; and Bialek, W. 2000. The information bottleneck method. In *Proceedings of the 37th Annual Allerton Conference on Communication, Control, and Computing*, 368–377.

Tu, W.; Deng, W.; and Gedeon, T. 2023. A closer look at the robustness of contrastive language-image pre-training (clip). *Advances in Neural Information Processing Systems*, 36: 13678–13691.

Van Horn, G.; Mac Aodha, O.; Song, Y.; Cui, Y.; Sun, C.; Shepard, A.; Adam, H.; Perona, P.; and Belongie, S. 2018. The iNaturalist species classification and detection dataset. In *Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)*, 8769–8778.

Wang, H.; Ge, S.; Lipton, Z.; and Xing, E. P. 2019. Learning robust global representations by penalizing local predictive power. In *Advances in Neural Information Processing Systems (NeurIPS)*, volume 32.

Wang, Q.; Lin, Y.; Chen, Y.; Schmidt, L.; Han, B.; and Zhang, T. 2024. A Sober Look at the Robustness of CLIPs to Spurious Features. In *Advances in Neural Information Processing Systems (NeurIPS)*.

Xiao, G.; Lin, J.; Seznec, M.; Wu, H.; Demouth, J.; and Han, S. 2023. Smoothquant: Accurate and efficient post-training quantization for large language models. In *International Conference on Machine Learning (ICML)*, 38087–38101. PMLR.

Xiao, J.; Hays, J.; Ehinger, K. A.; Oliva, A.; and Torralba, A. 2010. Sun database: Large-scale scene recognition from abbey to zoo. In *2010 IEEE computer society conference on computer vision and pattern recognition*, 3485–3492. IEEE.

Yang, J.; Wang, P.; Zou, D.; Zhou, Z.; Ding, K.; Peng, W.; Wang, H.; Chen, G.; Li, B.; Sun, Y.; Du, X.; Zhou,
K.; Zhang, W.; Hendrycks, D.; Li, Y.; and Liu, Z. 2022. OpenOOD: Benchmarking Generalized Out-of-Distribution Detection. In *Advances in Neural Information Processing Systems (NeurIPS)*, volume 35, 30150–30164.

Yang, J.; Zhou, K.; and Liu, Z. 2022. Full-Spectrum Out-of-Distribution Detection. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 16293–16302.

Yin, D.; Gontareva, A.; Gontarev, I.; Kornblith, S.; Gu, S.; and Le, Q. V. 2019. A Fourier perspective on the generalization of deep neural networks. In *International Conference on Machine Learning (ICML)*, 7133–7142. PMLR.

Zhang, J.; Yang, J.; Wang, P.; Wang, H.; Lin, Y.; Zhang, H.; Sun, Y.; Du, X.; Li, Y.; Liu, Z.; Chen, Y.; and Li, H. 2024. OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection. *Journal of Data-centric Machine Learning Research*.

Zhou, B.; Lapedriza, A.; Khosla, A.; Oliva, A.; and Torralba, A. 2017. Places: A 10 million image database for scene recognition. In *IEEE transactions on pattern analysis and machine intelligence*, volume 40, 1452–1464. IEEE.
## Abstract

This appendix provides a comprehensive extension to our main paper, focusing on explanations, reproducibility, and exhaustive results. We begin with a transparent discussion of our study's limitations. The centerpiece is a novel frequency-domain analysis using the 2D Fourier spectrum, which reveals how Quantization-Aware Training (QAT) reshapes a model's feature representation toward more robust, low-frequency concepts. To ensure full reproducibility, we meticulously document our experimental methodology, including a detailed justification for using simulated quantization and a full breakdown of model architectures, datasets, and hyperparameters. We also add the plots that did not fit in our main paper and that are very relevant. We then present additional analyses and visualizations, including a conceptual illustration of QAT's preference for flat minima and detailed reliability diagrams. Finally, we provide comprehensive tables containing the complete numerical results for all experiments, covering in-distribution accuracy, calibration, out-of-distribution detection, and robustness for every model and method evaluated.

## 4.1 Limitations

Our study, while comprehensive, has certain limitations.

* **Visual Encoder Only:** Our primary limitation is that we only quantize the visual encoder. Quantizing the text encoder, or both encoders simultaneously, introduces additional challenges related to maintaining the alignment of the joint embedding space and represents an important avenue for future work.

* **Proxy Dataset:** All QAT methods were fine-tuned on CC3M as a proxy dataset. The choice of proxy data could influence the final performance, especially regarding catastrophic forgetting.

* **Dynamic Quantization:** In our work, we quantize all linear layers without prior analysis. Research has shown that for LLM, selective layers quantization is essential for quantization as some layers gain massive rounding errors when quantized (Li et al. 2022a). We did not include results using more specialized and complex quantization methods to avoid adding additional noise to our study. Including such complex methods would have made it difficult to separate the true impact of quantization from whatever additional methods were used.

* **Pure Zero-shot:** Some important baselines are missing from this work, some due to the complexity of implementation, and others due to relying on sample data e.g. Mahalanobis(Lee et al. 2018), LoCoOp(Kar et al. 2023), OmniQuant(Shao et al. 2023).

# 5 Frequency-Domain Analysis of Quantization Effects

To investigate the mechanistic underpinnings of how quantization affects feature representations, we analyze the 2D

Fourier spectrum of the Vision Transformer's internal feature maps (Yin et al. 2019). Our analysis is grounded in the foundational principles of Fourier theory, which establishes a direct correspondence between a signal's frequency content and its spatial rate of change. We extend this principle to the feature maps of a ViT, where low-frequency components at the spectrum's origin represent coarse-grained visual concepts (e.g., broad shapes), while high-frequency components in the outer bands encode fine-grained details (e.g., sharp edges, textures). This spectral analysis thus allows for a quantitative measurement of the model's capacity to represent features of varying granularity. The feature representations for our analysis were extracted from the output of the final residual block within the Vision Transformer encoder. Specifically, we capture the sequence of output tokens from the vision_model transformer.resblocks module, immediately prior to the application of the terminal layer normalization (ln_post) and any subsequent projection layers. This layer provides the richest per-patch semantic representation while preserving the spatial topology necessary for our 2D spectral analysis, and we also argue the significance of our selection by only including the attention section of the ViT (which is known to be more sensitive to quantization) rather than the MLP section of the model (which is more quantize-friendly(Li et al. 2022a))

**Relative Spectral Error (RSE)** To precisely quantify the deviation of quantized models from the full-precision baseline, we define the Relative Spectral Error (RSE). The RSE at each 2D frequency coordinate $(u, v)$ is calculated as the normalized absolute difference between the quantized spectrum, $S_{quant}$, and the baseline spectrum, $S_{FP32}$:

$$RSE(u, v) = \frac{|S_{quant}(u, v) - S_{FP32}(u, v)|}{S_{FP32}(u, v) + \epsilon} \quad (1)$$

Here, $S_{quant}$ represents either $S_{PTQ}$ or $S_{QAT}$, and $\epsilon$ is a small constant (e.g., $10^{-9}$) to ensure numerical stability where $S_{FP32}(u, v)$ is close to zero. The resulting 2D RSE map provides a spatial visualization of the error distribution across the frequency domain.

## 5.1 Initial Findings with Aggressive Quantization

We first investigate the effects of aggressive 6-bit quantization on a ViT-B/32 model. Figure 11 shows that while the full-precision (FP32) spectrum exhibits significant energy across all bands, Post-Training Quantization (PTQ) severely attenuates the high-frequency components, indicating a catastrophic loss of fine-grained representational capacity. Subsequent Quantization-Aware Training (QAT) only partially restores these components. The Relative Spectral Error (RSE) maps confirm that quantization error is maximized in these high-frequency bands. These findings suggest that aggressive quantization acts as a potent low-pass filter, forcing the model to rely on robust, low-frequency features, which may explain observed improvements in robustness to certain data shifts.
<figure><img src="image_13.png" /><figcaption>Figure 11: Spectral analysis of a ViT-B/32 model with 6-bit quantization. From left to right: the FP32 baseline spectrum, the PTQ spectrum showing severe high-frequency attenuation, and the partially restored QAT spectrum. Bottom row shows corresponding RSE maps.</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Figure 12: Spectral analysis of a ViT-L/14 model with 8-bit quantization. Top row: FP32, PTQ, and QAT spectra. Bottom row: RSE maps, where PTQ induces a mid-frequency "error halo" (left), and QAT eliminates this halo in favor of a low-magnitude, global error floor (right), indicating a complex adaptation.</figcaption></figure>

## 5.2 Refined Analysis with Standard Quantization

While informative, these initial results arise from conditions—aggressive 6-bit precision and a low-resolution 7x7 feature grid—that may not reflect standard use cases. To disentangle these factors and study the adaptive nature of QAT with greater fidelity, we designed a refined experiment with standard w8a8 precision and a higher-resolution ViT-L/14 model, providing a 16x16 feature grid.

Our refined analysis, presented in Figure 12, reveals a more nuanced mechanism. The 8-bit PTQ induces a targeted, rather than catastrophic, degradation, manifesting as a distinct "error halo" in the mid-frequency bands of the RSE map. This confirms that even standard quantization preferentially degrades fine-grained features. The most striking result emerges from the QAT model. While it successfully eliminates the localized error halo, it does not converge to the original FP32 state. Instead, it finds a new equilibrium characterized by a low-magnitude, but globally distributed, error floor across the entire spectrum. This demonstrates that QAT is a sophisticated re-optimization process, as it reduces the coarse-grained features errors, compromising the fine-grained ones as an effective strategy to reduce global loss; the model finds a novel, stable representation within the 8-bit constraint, trading the targeted error for a minimal, systemic deviation. This suggests the robustness of quantized models arises not from a simple forgetting of fine-grained features, but from an active convergence to a new, more generalized internal representation that leverages coarse-grained, general features. Code for reproducibility of both plots is available.

## 5.3 Extended Experimental Setup

This section provides the full details of our experimental methodology to ensure clarity and reproducibility. Code will be provided for reproducibility, including all results (in CSV format), plotting notebooks, Quantization simulation, quantization simulation verification, and Fourier. We used an H100 cluster for the evaluations.

**Model Architectures** We used publicly available pre-trained Open-CLIP models. WIT pretrained denotes OpenAI weights. For all experiments, quantization and fine-tuning were applied **only to the visual encoder**, while the text encoder was kept frozen at FP32 precision. The primary models evaluated are detailed in Table 2.

<table><caption>Table 2: Details of the primary CLIP models used in our evaluation.</caption><thead><tr><td>Architecture</td><td>Pre-training Source</td><td>OpenCLIP Model Tag</td></tr></thead><tbody><tr><td>ViT-B/32</td><td>WIT (OpenAI)</td><td>openai</td></tr><tr><td>ViT-B/32</td><td>LAION</td><td>laion400m.e32</td></tr><tr><td>ViT-L/14</td><td>WIT (OpenAI)</td><td>openai</td></tr><tr><td>ViT-L/14</td><td>LAION</td><td>laion400m.e32</td></tr><tr><td>ViT-B/16</td><td>WIT (OpenAI)</td><td>openai</td></tr><tr><td>ViT-B/16</td><td>LAION</td><td>laion400m.e32</td></tr></tbody></table>

**Benchmark Dataset Taxonomy** Our benchmark suite systematically covers different forms of distribution shift, and we use the same datasets that are recognised and used by the community. For more details on dataset preparation for OOD detection, please refer to citep. They are categorized along two dimensions: *Covariate Shift* and *Semantic Shift*, as shown in Figure 13. We also use a specialized dataset for spurious correlations (Hochlehnert et al. 2025).

**Quantization and Training Configurations** Our experiments involved two main families of QAT, plus PTQ methods referenced from the literature. All QAT was performed on a subset of the CC3M(Sharma et al. 2018) dataset, as its higher quality and better curation proved slightly
<figure><img src="image_15.png" /><figcaption>Figure 13: Systematic benchmark suite categorized by Co-variate and Semantic Shift. Arrows indicate the ID reference dataset for OOD detection.</figcaption></figure>

more effective for short adaptation schedules compared to LAION400m (Schuhmann et al. 2022).

A crucial aspect of our methodology is the use of **simulated or "fake" quantization**. This approach was necessary because, at the time of our experiments, standard deep learning frameworks (e.g., PyTorch) and GPU hardware (e.g., NVIDIA H100 & RTX 3090) lack native support for performing computations with INT8, INT6 or INT4 activations or for models like Vision Transformers. While libraries like 'bitsandbytes' offer efficient kernels for weight-only quantization (e.g., for QLoRA), a full pipeline with quantized activations is not readily available. Therefore, simulation is the standard and necessary method in research to evaluate the potential performance of such models before dedicated hardware or software support exists.

**The Role of Fake Quantization** Fake quantization is a simulation technique that models the error introduced by quantization and de-quantization within a standard full-precision (FP32/FP16) training and inference loop. The process for a tensor $x$ is as follows:

1. **Quantize:** The full-precision input tensor $x$ is scaled, shifted, and rounded to the nearest integer value within the target bit-width's range (e.g., [-128, 127] for INT8).

$$x_{quant} = \text{round} \left( \frac{x}{\text{scale}} + \text{zero\_point} \right)$$

2. **Clamp:** The integer values are clamped to the representable range of the target bit-width.

3. **De-quantize:** The clamped integer tensor is immediately converted back to a full-precision floating-point tensor.

$$x_{dequant} = (x_{quant} - \text{zero\_point}) \times \text{scale}$$

The resulting tensor, $x_{dequant}$, has the same data type as the input but contains the precision loss that *would have occurred* in a true low-bit system. For the backward pass, a **Straight-Through Estimator (STE)** bypasses the non-differentiable 'round' function, enabling the model to learn weights that are robust to the simulated quantization noise.

**Verification of Simulation Correctness.** To ensure our simulation was valid, we implemented a verification utility. For any quantized model, this utility counts the number of unique values in both the weight and activation tensors (via forward hooks). For a successful INT8 simulation, this count must be $\le 2^8 = 256$, which we confirmed for our implementations.

**Motivation: Why Not Use Existing Kernels like QLoRA for Inference?** While methods like QLoRA enable the training of models with 4-bit weights, they are primarily designed to reduce memory usage during the training phase, not to accelerate inference. As shown in our direct performance benchmark in Table 3, using these kernels for inference can lead to a significant slowdown compared to the FP32 baseline. The overhead of de-quantizing the weights on-the-fly for each computation outweighs the benefits of reduced memory bandwidth. True inference acceleration requires dedicated hardware and software support for low-precision matrix multiplications for multi-head attention, which is what our simulation-based study aims to evaluate in terms of potential accuracy and reliability before such support becomes widespread.

**Post-Training Quantization (PTQ)** PTQ methods ('PTQ', 'SmoothQuant', 'RaanA') use a small calibration set to determine fixed quantization parameters. Our evaluation of these methods relies on the fake quantization simulation for inference to measure their impact on accuracy and reliability.

**Full Quantization-Aware Training (QAT)** This approach ('QAT-LORA' variants) involves fine-tuning using the fake quantization simulation. Although our QAT-LORA implementation is original, we suggest practitioners refer to more recent methods like LR-QAT (Bondarenko, Chiaro, and Nagel 2024), which are designed with inference efficiency in mind by re-fusing the LoRA matrices post-training.

**Quantization Primitive:** We implemented a custom Learnable Scale Quantization (LSQ) function with a gradient scaling factor of $\frac{1}{\sqrt{N \cdot Q_{max}}}$ to stabilize training.

**Model Architecture:** We apply LoRA adapters to the vision encoder's linear layers. Our custom 'EnhancedFake-Quantizer' modules are then injected such that the base model's path is fake-quantized while the parallel LoRA path operates on the full-precision input.

**Training Hyperparameters:** These models are trained with the parameters in Table 4. An optional knowledge distillation loss (MSE on normalized features) is used for 'Dissill' variants.
<table><caption>Table 3: Performance and memory benchmark of OpenAI CLIP models using non-simulated, kernel-based quantization. Inference time is the average latency per batch on an RTX 3090 GPU. Note the significant slowdown for 8-bit and 4-bit inference, which motivates our use of simulation to evaluate accuracy/reliability trade-offs, as current kernels are not optimized for inference speed. Failures for the L/14 model are attributed to a known bug in the tested library version.</caption><thead><tr><th>Model</th><th>Quantization</th><th>Memory (MB)</th><th>Time (ms)</th><th>Speedup (vs. FP32)</th><th>Memory Gain (vs. FP32)</th></tr></thead><tbody><tr><td rowspan="4">B/32</td><td>32-bit (FP32)</td><td>358.27</td><td>19.61</td><td>1.00x</td><td>0.0%</td></tr><tr><td>16-bit (FP16)</td><td>297.79</td><td>17.67</td><td>1.11x</td><td>16.9%</td></tr><tr><td>8-bit (INT8)</td><td>179.82</td><td>81.12</td><td>0.24x</td><td>49.8%</td></tr><tr><td>4-bit (NF4)</td><td>123.80</td><td>47.70</td><td>0.41x</td><td>65.4%</td></tr><tr><td rowspan="4">B/16</td><td>32-bit (FP32)</td><td>579.51</td><td>12.33</td><td>1.00x</td><td>0.0%</td></tr><tr><td>16-bit (FP16)</td><td>294.63</td><td>14.16</td><td>0.87x</td><td>49.2%</td></tr><tr><td>8-bit (INT8)</td><td>177.66</td><td>90.90</td><td>0.14x</td><td>69.3%</td></tr><tr><td>4-bit (NF4)</td><td>119.65</td><td>49.83</td><td>0.25x</td><td>79.3%</td></tr><tr><td rowspan="4">L/14</td><td>32-bit (FP32)</td><td>1312.47</td><td>27.28</td><td>1.00x</td><td>0.0%</td></tr><tr><td>16-bit (FP16)</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>8-bit (INT8)</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>4-bit (NF4)</td><td>278.31</td><td>70.20</td><td>0.39x</td><td>78.8%</td></tr></tbody></table>

<table><caption>Table 4: Hyperparameters for Full QAT methods.</caption><thead><tr><th>Hyperparameter</th><th>Value</th></tr></thead><tbody><tr><td>LoRA Rank (r)</td><td>8</td></tr><tr><td>LoRA Alpha (α)</td><td>16</td></tr><tr><td>Base Learning Rate</td><td>5 × 10<sup>-5</sup></td></tr><tr><td>LSQ Scale Learning Rate</td><td>1 × 10<sup>-7</sup></td></tr><tr><td>Optimizer</td><td>AdamW</td></tr><tr><td>Training Steps</td><td>100-8100</td></tr><tr><td>Batch size</td><td>100</td></tr><tr><td>Precision</td><td>W8A8 to W4A6</td></tr><tr><td>Distillation α</td><td>0.5 (if applicable)</td></tr></tbody></table>

<figure><img src="image_16.png" /><figcaption>Figure 14: Evolution of ViT/B-32 Quantized model accuracy on Cifar10 dataset, relative to the amount of unique CC3M samples used during QAT adaptation.</figcaption></figure>

**Light Quantization-Aware Training (Light QAT)** This approach ('Basic-QAT', 'QAT-LSQ') employs a unified 'FakeQuantize' module and a short training schedule (100 batches) to quickly adapt the model to quantization noise. 'Distill' variants use a KL-Divergence loss. Light Qat uses a smaller learning rate and a limited amount of adaptation samples, as QAT is an adaptation procedure and not a training one, it doesn't follow the same general tools as for training, we wish to avoid catastrophic forgetting by reducing the amount of new semantic information introduced during the adaptation, we loop back the data and limit the number of unique samples, the experiment below shows that a single sample is enough to adapt the model to quantization, and increasing the number of unique samples doesn't show a clear trend of improvement in this use-case.

<table><caption>Table 5: Hyperparameters for Light QAT methods.</caption><thead><tr><th>Hyperparameter</th><th>Value</th></tr></thead><tbody><tr><td>Learning Rate</td><td>1 × 10<sup>-6</sup></td></tr><tr><td>Optimizer</td><td>AdamW</td></tr><tr><td>Training Steps</td><td>100</td></tr><tr><td>Batch size</td><td>100</td></tr><tr><td>Unique samples used</td><td>100</td></tr><tr><td>Distillation α</td><td>0.5 (if applicable)</td></tr></tbody></table>

## 5.4 Additional Analyses and Results

**On QAT's Preference for Flat Minima** A significant body of work establishes that neural networks converging to "flat" minima in the loss landscape generalize better (Hochreiter and Schmidhuber 1997). As illustrated in Figure 15, QAT inherently penalizes sharp regions because the

weight perturbation from quantization ($Q(w) - w$) causes a large increase in loss. This forces the optimizer to seek out flatter, more robust regions, which can sometimes lead to better generalization and accuracy than the original FP32 model.
<figure><img src="image_17.png" /><figcaption>Figure 15: A conceptual illustration of how QAT forces the optimizer to abandon a sharp minimum in favor of a flatter, more robust solution.</figcaption></figure>

**Detailed Calibration Analysis** Here we provide the detailed visualizations of the calibration process, fulfilling the promises from the main paper. Figure 16 shows the direct impact of QAT (Phase 1), illustrating how quantization alone squeezes confidences, often pulling them away from the ideal diagonal. Figure 17 shows the final state after our full two-phase re-calibration process (QAT followed by logit scale tuning). The points move much closer to the diagonal, demonstrating the necessity of the second tuning phase to correct for the distortions introduced by QAT and achieve a well-calibrated model.

**Accuracy vs. NLL Trade-off** The main paper's teasier plot shows the trade-off between accuracy and ECE. Figure 18 provides another view, plotting accuracy change against Negative Log-Likelihood (NLL) change. NLL is a stricter metric that penalizes overconfident errors more heavily than ECE. The ideal quadrant (green) represents a simultaneous improvement in accuracy and a reduction in NLL. This plot further highlights the complex, multi-objective nature of deploying quantized models, where gains in one metric can often come at the cost of another.

**OOD Detection Performance** This section provides more detailed results on OOD detection to complement the main paper. Figure 21 shows the False Positive Rate at 95% True Positive Rate (FPR@95), a key metric for evaluating performance at a practical operating point. Lower is better. This plot reinforces the findings from the AUROC analysis, showing that VLM-specific methods are generally more robust to quantization and that the impact of quantization is highly dependent on the pre-training source.

**Robustness to Covariate Shift** Here we expand on the covariate shift results from the main paper. Figure 22 shows how Light QAT methods affect the model's ability to detect corrupted data as OOD, measured by the change in AUROC. For low-level corruptions like noise and blur, QAT significantly improves detection ability (a large positive change). However, for more complex, semantic-style shifts like ImageNet-R and -Sketch, the effect is neutral or slightly negative. This reveals a nuanced trade-off: QAT makes the

model robust to simple noise at the cost of sensitivity to fine-grained stylistic changes.

**Covariate Shift Data Generation Pipeline** For experiments on datasets like ImageNet-C, it is crucial to apply corruptions in a way that mimics real-world sensor noise. Our 'ResizeThenCorruptDataset' class ensures this by applying corruptions *after* the standard resizing and cropping but *before* the final 'ToTensor' and normalization steps. This "Resize-then-Corrupt" order prevents artifacts from the corruption process being altered by subsequent resizing operations.

## 5.5 Comprehensive Tables

Below we present the detailed numerical results for some experiments (PTQ ViT/B32, B16, L14 and Light QAT ViT/B32). The data is grouped by experimental configuration, predictive quality, and OOD detection (Model, Pre-training, and ID/OOD datasets) to facilitate direct comparison between the different quantization methods within each scenario. Please note that we cannot add all tables due to time constraints, please refer to the CSVs in the code.
<figure><img src="image_18.png" /><figcaption>Figure 16: Dataset-specific reliability diagrams showing the direct impact of Quantization-Aware Training (Phase 1). This plot shows the calibration state after QAT has been applied but before logit scale tuning. Note how the confidences for both WIT (blue/purple) and LAION (red/orange) models are shifted away from their initial FP32 state, often towards the center, but are not yet aligned with the ideal diagonal.</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>Figure 17: Dataset-specific reliability diagrams showing the final calibration state after the full two-phase process (QAT + Logit Scale Tuning). The "Final (Tuned)" points (hollow markers) show a significant improvement over the intermediate "Adapted (Mid)" state, moving much closer to the perfect calibration line (dashed diagonal). This demonstrates the effectiveness and necessity of the second logit-tuning phase.</figcaption></figure>
Trade-offs in CLIP Quantization: Accuracy and Uncertainty

Impact of Quantization on Zero-Shot Accuracy vs. NLL

<figure><img src="image_20.png" /><figcaption>Figure 18: The trade-off between zero-shot accuracy and Negative Log-Likelihood (NLL). The ideal outcome (green, top-left) is a gain in accuracy and a reduction in NLL. Most methods fall into the trade-off or degradation quadrants, highlighting the challenge of improving both metrics simultaneously. Note the logarithmic scale for the NLL axis.</figcaption></figure>
<figure><img src="image_21.png" /><figcaption>Figure 19: Full-size teaser figure: The dichotomous impact of quantization on zero-shot Performance. WIT models (blue) consistently improve in calibration (left), with several QAT methods achieving simultaneous accuracy gains. In contrast, LAION models (orange) show systematic degradation in calibration (right). The lack of falloffs near the origin suggests that quantization is always impactful.</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>Figure 20: Average FPR@95 across all Far-OOD datasets. Lower values are better. This plot complements the AUROC results from the main paper, showing OOD detection performance at a practical operating point. The VLM-specific methods (teal and pink) consistently outperform classic methods (blue and orange), and their performance is more resilient to quantization.</figcaption></figure>
<figure><img src="image_23.png" /><figcaption>Figure 21: Confidence bins shift after QAT and Logit Adaptation, we can clearly see the trend where the overconfident LAION pretrained model gets its confidence corrected down, while the underconfident WIT pretrained gets its confidence increased, showing the uncertainty awareness introduced by the quantized-adapted model.</figcaption></figure>

<figure><img src="image_24.png" /><figcaption>Figure 22: The change in OOD detection (AUROC) performance on various covariate shift datasets after applying Light QAT methods. Positive values indicate improved detection. QAT substantially improves the ability to detect low-level corruptions (top rows) but has a mixed-to-negative impact on detecting complex semantic shifts (bottom rows).</figcaption></figure>
<table><caption>Table 6: PTQ Results: Out-of-Distribution (OOD) Detection for OpenAI pre-trained ViT-L-14. A/F denote AUROC/FPR95.</caption><thead><tr><th rowspan="3">Scenario</th><th rowspan="3">Method</th><th colspan="8">Out-of-Distribution Detection Performance</th></tr><tr><th colspan="4">Traditional Methods</th><th colspan="4">VLM-based Methods</th></tr><tr><th colspan="2">MSP</th><th colspan="2">Energy</th><th colspan="2">MCM</th><th colspan="2">Gen.Neg.</th><th colspan="2">NegLabel</th></tr><tr><th></th><th></th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th></tr></thead><tbody><tr><td rowspan="3">ID: CIFAR10<br/>OOD: CIFAR100</td><td>FP32</td><td>94.78</td><td>19.61</td><td>88.59</td><td>49.47</td><td>91.70</td><td>40.78</td><td>95.65</td><td>20.36</td><td>95.02</td><td>25.33</td></tr><tr><td>PTQ</td><td>78.15</td><td>53.23</td><td>78.44</td><td>59.54</td><td>79.72</td><td>56.58</td><td>79.72</td><td>56.88</td><td>80.10</td><td>59.19</td></tr><tr><td>SmoothQuant</td><td>84.66</td><td>41.02</td><td>81.06</td><td>54.51</td><td>83.67</td><td>48.36</td><td>86.93</td><td>41.79</td><td>86.65</td><td>44.49</td></tr><tr><td></td><td>RaanA</td><td>79.12</td><td>51.58</td><td>80.08</td><td>57.33</td><td>81.18</td><td>54.48</td><td>80.83</td><td>54.45</td><td>81.21</td><td>57.18</td></tr><tr><td rowspan="3">ID: CIFAR100<br/>OOD: CIFAR10</td><td>FP32</td><td>80.12</td><td>55.05</td><td>80.45</td><td>59.15</td><td>83.76</td><td>49.14</td><td>82.17</td><td>48.19</td><td>81.86</td><td>51.52</td></tr><tr><td>PTQ</td><td>55.25</td><td>82.64</td><td>54.26</td><td>83.67</td><td>54.00</td><td>81.30</td><td>52.49</td><td>82.18</td><td>52.39</td><td>82.36</td></tr><tr><td>SmoothQuant</td><td>60.41</td><td>77.82</td><td>60.62</td><td>79.97</td><td>61.42</td><td>74.85</td><td>58.26</td><td>76.53</td><td>59.50</td><td>77.24</td></tr><tr><td></td><td>RaanA</td><td>55.29</td><td>82.57</td><td>54.83</td><td>83.16</td><td>54.49</td><td>81.12</td><td>52.49</td><td>81.97</td><td>52.59</td><td>82.60</td></tr><tr><td rowspan="3">ID: ImageNet-X<br/>OOD: ImageNet-X OOD</td><td>FP32</td><td>79.00</td><td>63.18</td><td>74.20</td><td>72.46</td><td>76.15</td><td>70.30</td><td>77.35</td><td>70.65</td><td>78.44</td><td>68.74</td></tr><tr><td>PTQ</td><td>61.87</td><td>82.56</td><td>59.56</td><td>86.53</td><td>60.95</td><td>84.90</td><td>60.62</td><td>84.92</td><td>60.51</td><td>84.15</td></tr><tr><td>SmoothQuant</td><td>67.16</td><td>76.88</td><td>64.16</td><td>82.53</td><td>66.10</td><td>80.64</td><td>65.91</td><td>81.16</td><td>66.45</td><td>79.59</td></tr><tr><td></td><td>RaanA</td><td>62.91</td><td>81.54</td><td>60.45</td><td>85.87</td><td>61.87</td><td>84.34</td><td>61.66</td><td>84.17</td><td>61.79</td><td>82.94</td></tr><tr><td rowspan="3">ID: ImageNet10<br/>OOD: ImageNet20</td><td>FP32</td><td>96.95</td><td>10.60</td><td>96.09</td><td>26.40</td><td>96.18</td><td>25.80</td><td>95.36</td><td>28.80</td><td>96.60</td><td>21.40</td></tr><tr><td>PTQ</td><td>86.97</td><td>44.40</td><td>87.55</td><td>52.60</td><td>87.97</td><td>52.80</td><td>85.42</td><td>57.60</td><td>87.31</td><td>55.60</td></tr><tr><td>SmoothQuant</td><td>92.45</td><td>31.20</td><td>92.49</td><td>42.40</td><td>92.77</td><td>42.40</td><td>90.32</td><td>48.40</td><td>91.88</td><td>46.20</td></tr><tr><td></td><td>RaanA</td><td>88.53</td><td>43.40</td><td>88.86</td><td>52.40</td><td>89.27</td><td>52.40</td><td>86.53</td><td>56.80</td><td>88.21</td><td>55.20</td></tr><tr><td rowspan="3">ID: ImageNet20<br/>OOD: ImageNet10</td><td>FP32</td><td>95.23</td><td>23.80</td><td>92.97</td><td>53.00</td><td>93.32</td><td>52.80</td><td>92.02</td><td>64.30</td><td>94.43</td><td>42.80</td></tr><tr><td>PTQ</td><td>87.27</td><td>38.40</td><td>83.58</td><td>69.70</td><td>84.85</td><td>64.90</td><td>83.48</td><td>63.70</td><td>85.18</td><td>54.70</td></tr><tr><td>SmoothQuant</td><td>90.79</td><td>29.80</td><td>88.96</td><td>56.00</td><td>89.97</td><td>49.90</td><td>88.57</td><td>54.60</td><td>90.72</td><td>47.60</td></tr><tr><td></td><td>RaanA</td><td>87.67</td><td>40.50</td><td>84.15</td><td>66.40</td><td>85.49</td><td>62.20</td><td>84.06</td><td>63.40</td><td>85.86</td><td>53.90</td></tr><tr><td rowspan="3">ID: IN1k<br/>OOD: DTD</td><td>FP32</td><td>80.69</td><td>77.70</td><td>64.74</td><td>79.50</td><td>70.93</td><td>77.44</td><td>85.18</td><td>54.30</td><td>84.80</td><td>63.40</td></tr><tr><td>PTQ</td><td>73.94</td><td>78.27</td><td>63.10</td><td>76.21</td><td>69.55</td><td>73.52</td><td>75.52</td><td>63.93</td><td>77.36</td><td>64.39</td></tr><tr><td>SmoothQuant</td><td>75.62</td><td>78.69</td><td>61.83</td><td>81.66</td><td>69.11</td><td>80.48</td><td>78.98</td><td>62.96</td><td>80.57</td><td>64.78</td></tr><tr><td></td><td>RaanA</td><td>74.31</td><td>78.53</td><td>61.89</td><td>78.72</td><td>68.75</td><td>75.11</td><td>76.73</td><td>62.82</td><td>79.08</td><td>62.45</td></tr><tr><td rowspan="3">ID: IN1k<br/>OOD: iNaturalist</td><td>FP32</td><td>77.32</td><td>83.34</td><td>65.52</td><td>84.37</td><td>69.39</td><td>84.27</td><td>85.60</td><td>58.50</td><td>72.61</td><td>84.27</td></tr><tr><td>PTQ</td><td>68.91</td><td>77.57</td><td>60.91</td><td>88.69</td><td>65.09</td><td>86.82</td><td>78.34</td><td>69.80</td><td>71.10</td><td>79.30</td></tr><tr><td>SmoothQuant</td><td>72.66</td><td>76.23</td><td>61.38</td><td>90.12</td><td>66.43</td><td>87.71</td><td>81.51</td><td>66.43</td><td>73.69</td><td>81.04</td></tr><tr><td></td><td>RaanA</td><td>69.05</td><td>77.18</td><td>61.03</td><td>88.30</td><td>65.22</td><td>86.49</td><td>78.45</td><td>69.24</td><td>71.63</td><td>79.14</td></tr><tr><td rowspan="3">ID: IN1k<br/>OOD: Places365</td><td>FP32</td><td>79.71</td><td>70.10</td><td>78.85</td><td>63.32</td><td>81.38</td><td>62.17</td><td>88.43</td><td>47.62</td><td>85.92</td><td>54.65</td></tr><tr><td>PTQ</td><td>69.63</td><td>81.07</td><td>78.21</td><td>63.49</td><td>78.21</td><td>65.60</td><td>80.11</td><td>59.84</td><td>75.16</td><td>65.79</td></tr><tr><td>SmoothQuant</td><td>71.63</td><td>79.89</td><td>76.26</td><td>68.22</td><td>77.96</td><td>67.32</td><td>84.11</td><td>54.38</td><td>76.88</td><td>65.89</td></tr><tr><td></td><td>RaanA</td><td>68.17</td><td>82.54</td><td>76.17</td><td>66.03</td><td>75.97</td><td>67.23</td><td>81.11</td><td>58.45</td><td>72.83</td><td>68.11</td></tr><tr><td rowspan="3">ID: IN1k<br/>OOD: SUN397</td><td>FP32</td><td>80.34</td><td>70.28</td><td>82.60</td><td>54.64</td><td>84.54</td><td>55.22</td><td>86.49</td><td>53.34</td><td>87.34</td><td>51.63</td></tr><tr><td>PTQ</td><td>67.94</td><td>81.72</td><td>78.58</td><td>60.31</td><td>77.50</td><td>63.07</td><td>77.28</td><td>62.57</td><td>73.76</td><td>66.50</td></tr><tr><td>SmoothQuant</td><td>69.83</td><td>80.99</td><td>78.81</td><td>60.77</td><td>79.18</td><td>62.66</td><td>79.44</td><td>60.93</td><td>76.87</td><td>64.87</td></tr><tr><td></td><td>RaanA</td><td>69.01</td><td>82.17</td><td>79.56</td><td>58.77</td><td>78.72</td><td>62.46</td><td>78.18</td><td>61.71</td><td>75.07</td><td>65.44</td></tr></tbody></table>
<table><caption>Table 7: PTQ Results: Out-of-Distribution (OOD) Detection for LAION-400M pre-trained ViT-L-14. A/F denote AU-ROC/FPR95.</caption><thead><tr><th rowspan="3">Scenario</th><th rowspan="3">Method</th><th colspan="12">Out-of-Distribution Detection Performance</th></tr><tr><th colspan="4">Traditional Methods</th><th colspan="4">VLM-based Methods</th><th colspan="2"></th></tr><tr><th colspan="2">MSP</th><th colspan="2">Energy</th><th colspan="2">MCM</th><th colspan="2">Gen.Neg.</th><th colspan="2">NegLabel</th></tr><tr><th></th><th></th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th></tr></thead><tbody><tr><td rowspan="4">ID: CIFAR10<br/>OOD: CIFAR100</td><td>FP32</td><td>92.39</td><td>24.30</td><td>91.07</td><td>47.89</td><td>91.31</td><td>46.36</td><td>93.64</td><td>36.85</td><td>93.33</td><td>37.29</td></tr><tr><td>PTQ</td><td>91.17</td><td>26.52</td><td>89.22</td><td>53.65</td><td>89.53</td><td>52.32</td><td>92.59</td><td>40.48</td><td>92.58</td><td>40.93</td></tr><tr><td>SmoothQuant</td><td>92.28</td><td>24.82</td><td>90.73</td><td>48.24</td><td>91.00</td><td>47.42</td><td>93.51</td><td>38.48</td><td>93.32</td><td>37.45</td></tr><tr><td>RaanA</td><td>91.78</td><td>25.17</td><td>89.99</td><td>51.21</td><td>90.28</td><td>50.13</td><td>93.05</td><td>39.07</td><td>92.97</td><td>39.08</td></tr><tr><td rowspan="4">ID: CIFAR100<br/>OOD: CIFAR10</td><td>FP32</td><td>79.97</td><td>52.59</td><td>84.91</td><td>51.28</td><td>85.39</td><td>50.05</td><td>87.91</td><td>45.35</td><td>87.36</td><td>44.04</td></tr><tr><td>PTQ</td><td>78.98</td><td>53.37</td><td>84.43</td><td>52.26</td><td>84.95</td><td>51.14</td><td>87.15</td><td>47.66</td><td>85.66</td><td>48.12</td></tr><tr><td>SmoothQuant</td><td>79.69</td><td>53.03</td><td>84.95</td><td>51.28</td><td>85.47</td><td>50.06</td><td>87.18</td><td>47.00</td><td>86.61</td><td>45.82</td></tr><tr><td>RaanA</td><td>78.67</td><td>55.66</td><td>83.95</td><td>52.43</td><td>84.48</td><td>51.09</td><td>86.84</td><td>49.10</td><td>85.26</td><td>49.40</td></tr><tr><td rowspan="4">ID: ImageNet-X<br/>OOD: ImageNet-X OOD</td><td>FP32</td><td>75.49</td><td>65.81</td><td>75.12</td><td>73.71</td><td>75.59</td><td>72.86</td><td>76.68</td><td>71.67</td><td>78.52</td><td>69.09</td></tr><tr><td>PTQ</td><td>74.95</td><td>66.59</td><td>74.72</td><td>73.38</td><td>75.21</td><td>72.69</td><td>76.20</td><td>71.48</td><td>78.09</td><td>68.92</td></tr><tr><td>SmoothQuant</td><td>75.21</td><td>66.24</td><td>74.69</td><td>73.64</td><td>75.19</td><td>72.85</td><td>76.16</td><td>71.53</td><td>77.86</td><td>70.06</td></tr><tr><td>RaanA</td><td>75.02</td><td>66.68</td><td>74.72</td><td>73.94</td><td>75.20</td><td>73.57</td><td>76.15</td><td>71.82</td><td>77.95</td><td>68.85</td></tr><tr><td rowspan="4">ID: ImageNet10<br/>OOD: ImageNet20</td><td>FP32</td><td>96.30</td><td>17.40</td><td>96.49</td><td>19.80</td><td>96.52</td><td>19.80</td><td>97.21</td><td>16.80</td><td>97.43</td><td>12.40</td></tr><tr><td>PTQ</td><td>96.13</td><td>18.00</td><td>96.43</td><td>21.00</td><td>96.46</td><td>20.60</td><td>97.20</td><td>16.00</td><td>97.43</td><td>11.60</td></tr><tr><td>SmoothQuant</td><td>96.16</td><td>19.00</td><td>96.23</td><td>20.00</td><td>96.26</td><td>20.00</td><td>97.05</td><td>16.80</td><td>97.26</td><td>13.40</td></tr><tr><td>RaanA</td><td>96.21</td><td>19.00</td><td>96.41</td><td>19.80</td><td>96.44</td><td>19.80</td><td>97.17</td><td>16.20</td><td>97.42</td><td>10.80</td></tr><tr><td rowspan="4">ID: ImageNet20<br/>OOD: ImageNet10</td><td>FP32</td><td>93.59</td><td>23.90</td><td>93.13</td><td>44.50</td><td>93.27</td><td>44.40</td><td>94.05</td><td>37.80</td><td>94.88</td><td>34.10</td></tr><tr><td>PTQ</td><td>93.19</td><td>24.40</td><td>92.52</td><td>43.40</td><td>92.66</td><td>42.50</td><td>93.65</td><td>37.20</td><td>94.49</td><td>33.90</td></tr><tr><td>SmoothQuant</td><td>93.51</td><td>26.00</td><td>92.67</td><td>47.50</td><td>92.83</td><td>46.70</td><td>93.65</td><td>41.30</td><td>94.26</td><td>35.50</td></tr><tr><td>RaanA</td><td>93.35</td><td>23.80</td><td>92.53</td><td>43.80</td><td>92.68</td><td>43.20</td><td>93.73</td><td>37.70</td><td>94.55</td><td>33.70</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: DTD</td><td>FP32</td><td>71.78</td><td>81.36</td><td>84.45</td><td>61.26</td><td>84.63</td><td>61.58</td><td>80.21</td><td>68.01</td><td>68.03</td><td>87.42</td></tr><tr><td>PTQ</td><td>71.50</td><td>79.62</td><td>84.47</td><td>59.93</td><td>84.62</td><td>60.27</td><td>81.79</td><td>62.76</td><td>70.91</td><td>82.22</td></tr><tr><td>SmoothQuant</td><td>71.82</td><td>80.02</td><td>83.50</td><td>62.17</td><td>83.71</td><td>62.40</td><td>79.70</td><td>68.52</td><td>68.26</td><td>85.33</td></tr><tr><td>RaanA</td><td>71.55</td><td>80.29</td><td>84.52</td><td>60.30</td><td>84.66</td><td>60.66</td><td>82.07</td><td>61.29</td><td>71.33</td><td>82.55</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: iNaturalist</td><td>FP32</td><td>74.17</td><td>70.61</td><td>67.97</td><td>81.03</td><td>68.90</td><td>80.20</td><td>81.46</td><td>62.81</td><td>79.87</td><td>69.16</td></tr><tr><td>PTQ</td><td>74.18</td><td>70.95</td><td>67.14</td><td>82.30</td><td>68.12</td><td>81.53</td><td>79.19</td><td>67.91</td><td>77.17</td><td>74.16</td></tr><tr><td>SmoothQuant</td><td>74.18</td><td>70.54</td><td>67.30</td><td>81.66</td><td>68.27</td><td>81.19</td><td>80.39</td><td>65.27</td><td>78.02</td><td>73.06</td></tr><tr><td>RaanA</td><td>74.07</td><td>71.01</td><td>67.40</td><td>81.65</td><td>68.35</td><td>80.78</td><td>79.81</td><td>66.95</td><td>77.79</td><td>73.47</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: Places365</td><td>FP32</td><td>70.93</td><td>78.13</td><td>90.05</td><td>39.83</td><td>89.98</td><td>40.30</td><td>87.34</td><td>51.01</td><td>84.35</td><td>61.66</td></tr><tr><td>PTQ</td><td>70.32</td><td>78.76</td><td>90.40</td><td>39.83</td><td>90.32</td><td>39.97</td><td>88.62</td><td>45.55</td><td>86.14</td><td>55.21</td></tr><tr><td>SmoothQuant</td><td>70.13</td><td>78.61</td><td>89.98</td><td>39.99</td><td>89.90</td><td>40.38</td><td>87.21</td><td>52.01</td><td>84.99</td><td>57.90</td></tr><tr><td>RaanA</td><td>70.15</td><td>78.72</td><td>90.01</td><td>40.64</td><td>89.93</td><td>41.26</td><td>87.34</td><td>50.21</td><td>85.61</td><td>56.22</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: SUN397</td><td>FP32</td><td>70.67</td><td>79.87</td><td>88.36</td><td>41.08</td><td>88.34</td><td>41.60</td><td>87.17</td><td>47.93</td><td>84.56</td><td>62.05</td></tr><tr><td>PTQ</td><td>70.28</td><td>80.30</td><td>88.22</td><td>42.07</td><td>88.18</td><td>42.99</td><td>86.61</td><td>49.42</td><td>84.43</td><td>60.64</td></tr><tr><td>SmoothQuant</td><td>70.29</td><td>80.15</td><td>88.23</td><td>40.93</td><td>88.20</td><td>41.29</td><td>86.64</td><td>48.87</td><td>84.55</td><td>60.55</td></tr><tr><td>RaanA</td><td>70.05</td><td>79.99</td><td>88.19</td><td>41.52</td><td>88.14</td><td>42.13</td><td>86.22</td><td>50.46</td><td>83.81</td><td>62.48</td></tr></tbody></table>
<table><caption>Table 8: PTQ Results: Out-of-Distribution (OOD) Detection for OpenAI pre-trained ViT-B-16. A/F denote AUROC/FPR95.</caption><thead><tr><th rowspan="3">Scenario</th><th rowspan="3">Method</th><th colspan="10">Out-of-Distribution Detection Performance</th></tr><tr><th colspan="4">Traditional Methods</th><th colspan="4">VLM-based Methods</th></tr><tr><th colspan="2">MSP</th><th colspan="2">Energy</th><th colspan="2">MCM</th><th colspan="2">Gen.Neg.</th><th colspan="2">NegLabel</th></tr><tr><th></th><th></th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th></tr></thead><tbody><tr><td rowspan="4">ID: CIFAR10<br/>OOD: CIFAR100</td><td>FP32</td><td>90.57</td><td>33.60</td><td>83.49</td><td>67.22</td><td>87.39</td><td>58.47</td><td>92.00</td><td>36.77</td><td>91.09</td><td>37.54</td></tr><tr><td>PTQ</td><td>63.01</td><td>75.92</td><td>64.65</td><td>82.22</td><td>66.21</td><td>78.02</td><td>70.76</td><td>72.62</td><td>70.25</td><td>72.22</td></tr><tr><td>SmoothQuant</td><td>78.78</td><td>52.99</td><td>75.25</td><td>71.90</td><td>78.91</td><td>63.48</td><td>83.43</td><td>52.09</td><td>82.42</td><td>54.56</td></tr><tr><td>RaanA</td><td>63.26</td><td>76.51</td><td>66.46</td><td>81.41</td><td>67.74</td><td>77.49</td><td>70.93</td><td>72.85</td><td>70.69</td><td>72.03</td></tr><tr><td rowspan="4">ID: CIFAR100<br/>OOD: CIFAR10</td><td>FP32</td><td>68.74</td><td>70.47</td><td>71.52</td><td>75.02</td><td>73.88</td><td>66.99</td><td>71.53</td><td>64.23</td><td>68.91</td><td>65.49</td></tr><tr><td>PTQ</td><td>55.84</td><td>89.90</td><td>52.94</td><td>90.71</td><td>54.33</td><td>86.98</td><td>51.59</td><td>90.50</td><td>53.46</td><td>90.27</td></tr><tr><td>SmoothQuant</td><td>53.45</td><td>86.77</td><td>58.33</td><td>85.99</td><td>57.33</td><td>81.73</td><td>53.23</td><td>83.66</td><td>52.07</td><td>85.50</td></tr><tr><td>RaanA</td><td>55.24</td><td>90.19</td><td>53.23</td><td>90.47</td><td>54.37</td><td>86.76</td><td>51.37</td><td>90.37</td><td>53.31</td><td>89.46</td></tr><tr><td rowspan="4">ID: ImageNet-X<br/>OOD: ImageNet-X OOD</td><td>FP32</td><td>74.23</td><td>70.75</td><td>69.24</td><td>77.94</td><td>71.72</td><td>75.67</td><td>74.12</td><td>72.71</td><td>73.78</td><td>72.51</td></tr><tr><td>PTQ</td><td>54.01</td><td>90.00</td><td>52.65</td><td>91.78</td><td>53.54</td><td>90.54</td><td>53.65</td><td>90.25</td><td>52.96</td><td>90.63</td></tr><tr><td>SmoothQuant</td><td>58.23</td><td>85.50</td><td>56.67</td><td>88.85</td><td>58.09</td><td>86.82</td><td>58.34</td><td>86.48</td><td>57.40</td><td>86.43</td></tr><tr><td>RaanA</td><td>54.65</td><td>88.68</td><td>53.18</td><td>91.40</td><td>54.13</td><td>89.83</td><td>54.34</td><td>89.82</td><td>53.65</td><td>89.94</td></tr><tr><td rowspan="4">ID: ImageNet10<br/>OOD: ImageNet20</td><td>FP32</td><td>97.64</td><td>10.20</td><td>96.82</td><td>18.20</td><td>97.03</td><td>17.40</td><td>96.63</td><td>25.40</td><td>97.62</td><td>12.20</td></tr><tr><td>PTQ</td><td>72.46</td><td>62.00</td><td>76.41</td><td>65.40</td><td>76.84</td><td>64.20</td><td>77.96</td><td>64.40</td><td>80.26</td><td>64.40</td></tr><tr><td>SmoothQuant</td><td>82.08</td><td>47.40</td><td>84.44</td><td>57.40</td><td>85.01</td><td>55.60</td><td>85.75</td><td>57.00</td><td>86.20</td><td>49.80</td></tr><tr><td>RaanA</td><td>75.95</td><td>60.00</td><td>77.61</td><td>60.40</td><td>78.46</td><td>61.40</td><td>79.99</td><td>62.40</td><td>81.75</td><td>61.40</td></tr><tr><td rowspan="4">ID: ImageNet20<br/>OOD: ImageNet10</td><td>FP32</td><td>95.42</td><td>15.90</td><td>94.16</td><td>34.80</td><td>94.82</td><td>30.00</td><td>93.80</td><td>40.20</td><td>95.44</td><td>26.40</td></tr><tr><td>PTQ</td><td>72.39</td><td>62.90</td><td>76.76</td><td>71.90</td><td>77.87</td><td>67.80</td><td>77.13</td><td>66.90</td><td>75.70</td><td>64.10</td></tr><tr><td>SmoothQuant</td><td>81.85</td><td>47.50</td><td>82.61</td><td>62.90</td><td>84.02</td><td>58.30</td><td>82.55</td><td>64.70</td><td>83.24</td><td>56.20</td></tr><tr><td>RaanA</td><td>73.05</td><td>64.80</td><td>77.36</td><td>68.70</td><td>78.63</td><td>65.00</td><td>77.54</td><td>68.20</td><td>76.47</td><td>65.90</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: DTD</td><td>FP32</td><td>78.95</td><td>80.22</td><td>72.63</td><td>69.47</td><td>77.94</td><td>69.21</td><td>85.92</td><td>57.17</td><td>87.94</td><td>58.39</td></tr><tr><td>PTQ</td><td>63.10</td><td>86.87</td><td>50.85</td><td>89.83</td><td>57.41</td><td>86.78</td><td>65.86</td><td>78.81</td><td>68.68</td><td>78.14</td></tr><tr><td>SmoothQuant</td><td>68.84</td><td>82.87</td><td>61.05</td><td>81.81</td><td>67.07</td><td>80.42</td><td>74.68</td><td>71.92</td><td>77.81</td><td>68.83</td></tr><tr><td>RaanA</td><td>63.60</td><td>86.31</td><td>52.49</td><td>88.84</td><td>58.77</td><td>85.35</td><td>67.11</td><td>77.52</td><td>69.48</td><td>76.98</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: iNaturalist</td><td>FP32</td><td>73.36</td><td>78.80</td><td>64.24</td><td>86.71</td><td>68.15</td><td>85.35</td><td>85.89</td><td>57.70</td><td>77.04</td><td>77.64</td></tr><tr><td>PTQ</td><td>57.72</td><td>86.75</td><td>49.04</td><td>93.98</td><td>52.62</td><td>90.44</td><td>72.47</td><td>78.18</td><td>69.43</td><td>82.49</td></tr><tr><td>SmoothQuant</td><td>63.13</td><td>85.42</td><td>57.89</td><td>89.79</td><td>61.67</td><td>88.17</td><td>75.59</td><td>75.40</td><td>71.45</td><td>84.23</td></tr><tr><td>RaanA</td><td>61.80</td><td>84.75</td><td>50.92</td><td>93.61</td><td>56.21</td><td>89.96</td><td>74.71</td><td>77.97</td><td>72.71</td><td>81.99</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: Places365</td><td>FP32</td><td>75.82</td><td>76.96</td><td>83.45</td><td>54.51</td><td>84.57</td><td>55.13</td><td>87.45</td><td>51.27</td><td>82.86</td><td>62.82</td></tr><tr><td>PTQ</td><td>58.98</td><td>88.70</td><td>67.44</td><td>78.35</td><td>65.98</td><td>78.69</td><td>73.03</td><td>72.98</td><td>60.85</td><td>81.72</td></tr><tr><td>SmoothQuant</td><td>61.05</td><td>86.16</td><td>73.06</td><td>71.59</td><td>71.92</td><td>72.03</td><td>75.30</td><td>69.02</td><td>67.68</td><td>75.97</td></tr><tr><td>RaanA</td><td>58.51</td><td>87.60</td><td>69.75</td><td>76.57</td><td>69.00</td><td>78.04</td><td>74.28</td><td>73.73</td><td>63.05</td><td>81.48</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: SUN397</td><td>FP32</td><td>75.21</td><td>78.80</td><td>82.12</td><td>54.36</td><td>83.42</td><td>57.12</td><td>84.87</td><td>59.73</td><td>83.83</td><td>63.00</td></tr><tr><td>PTQ</td><td>56.03</td><td>88.69</td><td>66.56</td><td>79.00</td><td>64.77</td><td>80.45</td><td>69.86</td><td>75.11</td><td>63.30</td><td>80.30</td></tr><tr><td>SmoothQuant</td><td>60.69</td><td>86.16</td><td>72.11</td><td>71.98</td><td>71.04</td><td>73.23</td><td>70.64</td><td>73.95</td><td>67.27</td><td>77.99</td></tr><tr><td>RaanA</td><td>57.10</td><td>88.48</td><td>65.72</td><td>80.84</td><td>64.83</td><td>80.54</td><td>70.25</td><td>74.37</td><td>64.56</td><td>79.55</td></tr></tbody></table>
<table><caption>Table 9: PTQ Results: Out-of-Distribution (OOD) Detection for LAION-400M pre-trained ViT-B-16. A/F denote AU-ROC/FPR95.</caption><thead><tr><th rowspan="3">Scenario</th><th rowspan="3">Method</th><th colspan="12">Out-of-Distribution Detection Performance</th></tr><tr><th colspan="4">Traditional Methods</th><th colspan="4">VLM-based Methods</th><th colspan="2"></th></tr><tr><th colspan="2">MSP</th><th colspan="2">Energy</th><th colspan="2">MCM</th><th colspan="2">Gen.Neg.</th><th colspan="2">NegLabel</th></tr><tr><th></th><th></th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th></tr></thead><tbody><tr><td rowspan="4">ID: CIFAR10<br>OOD: CIFAR100</td><td>FP32</td><td>89.98</td><td>32.09</td><td>89.36</td><td>52.87</td><td>89.85</td><td>51.21</td><td>91.93</td><td>43.94</td><td>91.42</td><td>35.67</td></tr><tr><td>PTQ</td><td>66.55</td><td>76.38</td><td>73.97</td><td>76.40</td><td>74.44</td><td>75.49</td><td>75.86</td><td>70.91</td><td>69.87</td><td>72.94</td></tr><tr><td>SmoothQuant</td><td>88.31</td><td>36.06</td><td>88.20</td><td>57.18</td><td>88.84</td><td>55.73</td><td>90.21</td><td>52.53</td><td>89.58</td><td>42.18</td></tr><tr><td>RaanA</td><td>65.76</td><td>77.27</td><td>73.31</td><td>77.31</td><td>73.84</td><td>76.02</td><td>74.76</td><td>73.47</td><td>68.92</td><td>76.57</td></tr><tr><td rowspan="4">ID: CIFAR100<br>OOD: CIFAR10</td><td>FP32</td><td>73.87</td><td>66.03</td><td>81.29</td><td>63.22</td><td>81.95</td><td>61.82</td><td>78.60</td><td>64.67</td><td>79.79</td><td>58.63</td></tr><tr><td>PTQ</td><td>60.58</td><td>82.57</td><td>59.17</td><td>85.41</td><td>60.47</td><td>83.97</td><td>57.59</td><td>88.01</td><td>61.18</td><td>80.53</td></tr><tr><td>SmoothQuant</td><td>71.21</td><td>70.54</td><td>79.28</td><td>65.68</td><td>80.01</td><td>63.95</td><td>76.38</td><td>68.68</td><td>76.18</td><td>64.24</td></tr><tr><td>RaanA</td><td>60.68</td><td>82.22</td><td>59.56</td><td>84.83</td><td>60.74</td><td>83.43</td><td>57.88</td><td>88.02</td><td>61.34</td><td>80.25</td></tr><tr><td rowspan="4">ID: ImageNet-X<br>OOD: ImageNet-X OOD</td><td>FP32</td><td>72.54</td><td>71.62</td><td>72.45</td><td>77.36</td><td>73.14</td><td>76.54</td><td>71.82</td><td>77.36</td><td>72.14</td><td>75.16</td></tr><tr><td>PTQ</td><td>63.70</td><td>81.32</td><td>63.89</td><td>85.95</td><td>64.65</td><td>85.20</td><td>63.15</td><td>85.70</td><td>62.58</td><td>83.53</td></tr><tr><td>SmoothQuant</td><td>70.39</td><td>73.27</td><td>70.05</td><td>80.37</td><td>70.85</td><td>79.24</td><td>69.55</td><td>79.17</td><td>69.46</td><td>77.92</td></tr><tr><td>RaanA</td><td>62.58</td><td>81.64</td><td>62.86</td><td>86.76</td><td>63.52</td><td>85.93</td><td>61.62</td><td>86.64</td><td>61.21</td><td>83.81</td></tr><tr><td rowspan="4">ID: ImageNet10<br>OOD: ImageNet20</td><td>FP32</td><td>95.84</td><td>17.40</td><td>96.25</td><td>23.40</td><td>96.29</td><td>23.20</td><td>95.84</td><td>26.80</td><td>95.89</td><td>20.00</td></tr><tr><td>PTQ</td><td>87.23</td><td>43.00</td><td>91.14</td><td>46.80</td><td>91.20</td><td>45.60</td><td>90.04</td><td>50.40</td><td>89.99</td><td>44.00</td></tr><tr><td>SmoothQuant</td><td>94.38</td><td>20.80</td><td>95.60</td><td>26.00</td><td>95.64</td><td>25.60</td><td>95.01</td><td>30.80</td><td>95.11</td><td>25.60</td></tr><tr><td>RaanA</td><td>87.33</td><td>44.00</td><td>91.32</td><td>47.20</td><td>91.37</td><td>47.60</td><td>90.12</td><td>49.00</td><td>89.98</td><td>47.00</td></tr><tr><td rowspan="4">ID: ImageNet20<br>OOD: ImageNet10</td><td>FP32</td><td>93.60</td><td>21.20</td><td>94.54</td><td>28.20</td><td>94.69</td><td>26.70</td><td>94.26</td><td>30.60</td><td>95.20</td><td>26.30</td></tr><tr><td>PTQ</td><td>86.43</td><td>38.90</td><td>88.79</td><td>50.80</td><td>89.21</td><td>48.50</td><td>86.66</td><td>58.60</td><td>88.66</td><td>51.80</td></tr><tr><td>SmoothQuant</td><td>91.55</td><td>26.90</td><td>93.35</td><td>36.30</td><td>93.53</td><td>35.70</td><td>93.54</td><td>32.60</td><td>94.37</td><td>27.00</td></tr><tr><td>RaanA</td><td>86.32</td><td>39.80</td><td>89.06</td><td>52.20</td><td>89.43</td><td>50.10</td><td>86.98</td><td>57.10</td><td>88.54</td><td>49.30</td></tr><tr><td rowspan="4">ID: IN1k<br>OOD: DTD</td><td>FP32</td><td>69.85</td><td>83.21</td><td>82.99</td><td>61.78</td><td>83.24</td><td>62.93</td><td>82.18</td><td>61.24</td><td>72.11</td><td>76.08</td></tr><tr><td>PTQ</td><td>64.73</td><td>84.63</td><td>74.95</td><td>72.48</td><td>75.66</td><td>72.66</td><td>71.80</td><td>73.30</td><td>65.04</td><td>81.08</td></tr><tr><td>SmoothQuant</td><td>68.87</td><td>82.58</td><td>79.41</td><td>67.82</td><td>80.03</td><td>68.59</td><td>80.27</td><td>61.31</td><td>70.53</td><td>77.26</td></tr><tr><td>RaanA</td><td>62.64</td><td>85.79</td><td>74.20</td><td>74.44</td><td>74.69</td><td>75.60</td><td>70.81</td><td>73.57</td><td>61.22</td><td>85.12</td></tr><tr><td rowspan="4">ID: IN1k<br>OOD: iNaturalist</td><td>FP32</td><td>71.78</td><td>73.63</td><td>65.81</td><td>83.41</td><td>67.11</td><td>82.66</td><td>73.62</td><td>76.13</td><td>74.16</td><td>76.32</td></tr><tr><td>PTQ</td><td>64.24</td><td>81.93</td><td>59.36</td><td>88.24</td><td>60.69</td><td>87.83</td><td>64.07</td><td>81.61</td><td>60.95</td><td>84.51</td></tr><tr><td>SmoothQuant</td><td>70.41</td><td>74.73</td><td>64.00</td><td>85.02</td><td>65.48</td><td>84.12</td><td>71.63</td><td>77.49</td><td>70.87</td><td>81.17</td></tr><tr><td>RaanA</td><td>63.97</td><td>82.12</td><td>59.90</td><td>89.98</td><td>61.19</td><td>89.41</td><td>64.62</td><td>79.63</td><td>62.51</td><td>83.69</td></tr><tr><td rowspan="4">ID: IN1k<br>OOD: Places365</td><td>FP32</td><td>68.23</td><td>80.60</td><td>88.11</td><td>44.49</td><td>88.01</td><td>45.42</td><td>81.50</td><td>62.97</td><td>74.52</td><td>76.56</td></tr><tr><td>PTQ</td><td>61.32</td><td>86.05</td><td>84.30</td><td>52.29</td><td>83.98</td><td>54.21</td><td>71.36</td><td>77.68</td><td>64.82</td><td>83.67</td></tr><tr><td>SmoothQuant</td><td>66.65</td><td>81.52</td><td>87.48</td><td>47.37</td><td>87.37</td><td>48.23</td><td>80.79</td><td>65.40</td><td>72.31</td><td>78.32</td></tr><tr><td>RaanA</td><td>60.24</td><td>86.51</td><td>84.77</td><td>52.05</td><td>84.27</td><td>53.82</td><td>73.09</td><td>76.37</td><td>65.02</td><td>84.04</td></tr><tr><td rowspan="4">ID: IN1k<br>OOD: SUN397</td><td>FP32</td><td>65.92</td><td>83.41</td><td>86.35</td><td>45.78</td><td>86.19</td><td>47.16</td><td>85.92</td><td>51.25</td><td>81.65</td><td>63.22</td></tr><tr><td>PTQ</td><td>56.13</td><td>88.73</td><td>78.82</td><td>61.79</td><td>77.98</td><td>64.44</td><td>77.14</td><td>66.07</td><td>70.75</td><td>74.63</td></tr><tr><td>SmoothQuant</td><td>63.28</td><td>84.94</td><td>84.52</td><td>52.73</td><td>84.30</td><td>54.36</td><td>83.62</td><td>57.47</td><td>78.43</td><td>66.85</td></tr><tr><td>RaanA</td><td>55.80</td><td>89.12</td><td>76.80</td><td>64.46</td><td>76.11</td><td>67.06</td><td>77.05</td><td>67.17</td><td>70.81</td><td>73.91</td></tr></tbody></table>
<table><caption>Table 10: PTQ Results: Out-of-Distribution (OOD) Detection for OpenAI pre-trained ViT-B-32. A/F denote AUROC/FPR95.</caption><caption>Out-of-Distribution Detection Performance</caption><thead><tr><th rowspan="3">Scenario</th><th rowspan="3">Method</th><th colspan="4">Traditional Methods</th><th colspan="4">VLM-based Methods</th></tr><tr><th colspan="2">MSP</th><th colspan="2">Energy</th><th colspan="2">MCM</th><th colspan="2">Gen.Neg.</th><th colspan="2">NegLabel</th></tr><tr><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th></tr></thead><tbody><tr><td rowspan="4">ID: CIFAR10<br/>OOD: CIFAR100</td><td>FP32</td><td>90.56</td><td>34.04</td><td>86.00</td><td>61.30</td><td>88.36</td><td>55.10</td><td>91.88</td><td>39.47</td><td>91.00</td><td>37.72</td></tr><tr><td>PTQ</td><td>89.46</td><td>36.47</td><td>85.50</td><td>59.62</td><td>87.91</td><td>54.74</td><td>91.06</td><td>42.84</td><td>89.90</td><td>40.26</td></tr><tr><td>SmoothQuant</td><td>90.43</td><td>35.26</td><td>85.13</td><td>62.61</td><td>87.81</td><td>56.85</td><td>91.61</td><td>42.20</td><td>90.80</td><td>38.58</td></tr><tr><td>RaanA</td><td>89.29</td><td>36.89</td><td>85.73</td><td>59.09</td><td>88.06</td><td>53.73</td><td>90.84</td><td>44.01</td><td>89.77</td><td>41.34</td></tr><tr><td rowspan="4">ID: CIFAR100<br/>OOD: CIFAR10</td><td>FP32</td><td>72.39</td><td>70.08</td><td>73.32</td><td>70.56</td><td>76.21</td><td>66.38</td><td>75.59</td><td>65.61</td><td>71.32</td><td>69.38</td></tr><tr><td>PTQ</td><td>71.51</td><td>70.98</td><td>74.55</td><td>70.90</td><td>76.89</td><td>66.64</td><td>75.64</td><td>66.72</td><td>69.70</td><td>71.23</td></tr><tr><td>SmoothQuant</td><td>72.49</td><td>69.73</td><td>74.05</td><td>70.84</td><td>76.79</td><td>66.39</td><td>75.96</td><td>65.28</td><td>70.82</td><td>70.48</td></tr><tr><td>RaanA</td><td>71.46</td><td>71.42</td><td>73.86</td><td>71.33</td><td>76.28</td><td>67.58</td><td>75.12</td><td>66.25</td><td>69.54</td><td>71.89</td></tr><tr><td rowspan="4">ID: ImageNet-X<br/>OOD: ImageNet-X OOD</td><td>FP32</td><td>73.03</td><td>73.24</td><td>68.66</td><td>81.34</td><td>70.60</td><td>79.13</td><td>72.56</td><td>78.69</td><td>72.55</td><td>75.29</td></tr><tr><td>PTQ</td><td>71.49</td><td>75.12</td><td>67.04</td><td>83.03</td><td>69.09</td><td>81.16</td><td>71.15</td><td>80.11</td><td>70.95</td><td>77.63</td></tr><tr><td>SmoothQuant</td><td>72.42</td><td>74.44</td><td>68.13</td><td>81.75</td><td>70.13</td><td>79.69</td><td>72.00</td><td>78.66</td><td>71.76</td><td>76.13</td></tr><tr><td>RaanA</td><td>71.17</td><td>75.10</td><td>67.03</td><td>82.37</td><td>69.02</td><td>80.66</td><td>71.10</td><td>80.04</td><td>70.55</td><td>77.32</td></tr><tr><td rowspan="4">ID: ImageNet10<br/>OOD: ImageNet20</td><td>FP32</td><td>96.28</td><td>18.20</td><td>95.41</td><td>27.00</td><td>96.68</td><td>16.60</td><td>95.58</td><td>25.00</td><td>95.52</td><td>27.20</td></tr><tr><td>PTQ</td><td>96.27</td><td>19.00</td><td>95.31</td><td>27.20</td><td>96.27</td><td>20.20</td><td>95.46</td><td>26.80</td><td>95.44</td><td>26.40</td></tr><tr><td>SmoothQuant</td><td>96.17</td><td>19.80</td><td>95.46</td><td>24.60</td><td>96.68</td><td>17.20</td><td>95.61</td><td>23.80</td><td>95.59</td><td>24.80</td></tr><tr><td>RaanA</td><td>96.21</td><td>20.20</td><td>95.37</td><td>25.60</td><td>96.42</td><td>19.20</td><td>95.57</td><td>23.40</td><td>95.52</td><td>25.40</td></tr><tr><td rowspan="4">ID: ImageNet20<br/>OOD: ImageNet10</td><td>FP32</td><td>93.27</td><td>25.20</td><td>92.76</td><td>37.40</td><td>94.67</td><td>27.60</td><td>93.74</td><td>32.70</td><td>93.32</td><td>35.20</td></tr><tr><td>PTQ</td><td>92.43</td><td>27.80</td><td>92.50</td><td>39.90</td><td>93.80</td><td>28.80</td><td>93.29</td><td>37.70</td><td>93.05</td><td>38.00</td></tr><tr><td>SmoothQuant</td><td>92.67</td><td>29.80</td><td>92.19</td><td>39.70</td><td>94.17</td><td>26.90</td><td>93.36</td><td>35.30</td><td>94.17</td><td>38.10</td></tr><tr><td>RaanA</td><td>92.20</td><td>30.40</td><td>92.11</td><td>38.40</td><td>93.61</td><td>31.50</td><td>93.07</td><td>41.40</td><td>93.61</td><td>36.90</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: DTD</td><td>FP32</td><td>75.74</td><td>78.68</td><td>71.53</td><td>71.85</td><td>74.98</td><td>70.57</td><td>84.84</td><td>54.54</td><td>86.63</td><td>51.98</td></tr><tr><td>PTQ</td><td>74.94</td><td>80.37</td><td>68.14</td><td>77.23</td><td>72.14</td><td>76.59</td><td>83.20</td><td>59.92</td><td>84.53</td><td>61.04</td></tr><tr><td>SmoothQuant</td><td>76.14</td><td>78.63</td><td>70.34</td><td>73.56</td><td>74.27</td><td>72.44</td><td>84.51</td><td>55.66</td><td>86.53</td><td>53.55</td></tr><tr><td>RaanA</td><td>74.54</td><td>80.07</td><td>68.58</td><td>76.63</td><td>72.40</td><td>76.11</td><td>83.30</td><td>59.36</td><td>84.58</td><td>59.62</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: iNaturalist</td><td>FP32</td><td>71.34</td><td>76.14</td><td>59.91</td><td>86.56</td><td>63.39</td><td>85.54</td><td>77.41</td><td>72.99</td><td>73.56</td><td>78.67</td></tr><tr><td>PTQ</td><td>70.96</td><td>76.44</td><td>59.72</td><td>87.61</td><td>63.43</td><td>86.32</td><td>76.64</td><td>75.01</td><td>72.30</td><td>81.64</td></tr><tr><td>SmoothQuant</td><td>70.79</td><td>76.94</td><td>59.09</td><td>86.67</td><td>62.72</td><td>85.89</td><td>75.39</td><td>76.23</td><td>72.63</td><td>81.11</td></tr><tr><td>RaanA</td><td>70.84</td><td>76.76</td><td>58.89</td><td>87.92</td><td>62.64</td><td>86.70</td><td>76.24</td><td>75.46</td><td>73.12</td><td>80.72</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: Places365</td><td>FP32</td><td>75.00</td><td>76.47</td><td>86.43</td><td>47.10</td><td>86.90</td><td>48.23</td><td>82.50</td><td>59.85</td><td>78.99</td><td>66.33</td></tr><tr><td>PTQ</td><td>74.18</td><td>77.88</td><td>84.02</td><td>53.87</td><td>84.80</td><td>54.66</td><td>80.83</td><td>62.88</td><td>77.44</td><td>69.38</td></tr><tr><td>SmoothQuant</td><td>74.77</td><td>76.57</td><td>84.62</td><td>51.67</td><td>85.38</td><td>52.56</td><td>81.11</td><td>62.03</td><td>80.21</td><td>63.97</td></tr><tr><td>RaanA</td><td>73.67</td><td>78.01</td><td>83.56</td><td>53.97</td><td>84.36</td><td>54.64</td><td>80.83</td><td>63.33</td><td>77.24</td><td>70.43</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: SUN397</td><td>FP32</td><td>73.65</td><td>77.33</td><td>82.75</td><td>52.08</td><td>83.60</td><td>53.40</td><td>80.34</td><td>61.72</td><td>80.50</td><td>63.08</td></tr><tr><td>PTQ</td><td>72.23</td><td>78.85</td><td>81.02</td><td>57.21</td><td>81.89</td><td>58.64</td><td>78.09</td><td>65.94</td><td>78.42</td><td>66.65</td></tr><tr><td>SmoothQuant</td><td>73.35</td><td>77.49</td><td>80.76</td><td>56.54</td><td>81.91</td><td>57.47</td><td>78.79</td><td>64.99</td><td>79.49</td><td>65.34</td></tr><tr><td>RaanA</td><td>72.94</td><td>78.51</td><td>81.43</td><td>56.09</td><td>82.40</td><td>56.98</td><td>79.02</td><td>65.39</td><td>79.23</td><td>65.51</td></tr></tbody></table>
<table><caption>Table 11: PTQ Results: Out-of-Distribution (OOD) Detection for LAION-400M pre-trained ViT-B-32. A/F denote AU-ROC/FPR95.</caption><thead><tr><th rowspan="3">Scenario</th><th rowspan="3">Method</th><th colspan="12">Out-of-Distribution Detection Performance</th></tr><tr><th colspan="4">Traditional Methods</th><th colspan="4">VLM-based Methods</th></tr><tr><th colspan="2">MSP</th><th colspan="2">Energy</th><th colspan="2">MCM</th><th colspan="2">Gen.Neg.</th><th colspan="2">NegLabel</th></tr><tr><th></th><th></th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th></tr></thead><tbody><tr><td rowspan="4">ID: CIFAR10<br/>OOD: CIFAR100</td><td>FP32</td><td>88.65</td><td>37.58</td><td>88.38</td><td>59.87</td><td>88.90</td><td>58.14</td><td>90.40</td><td>48.46</td><td>89.25</td><td>46.25</td></tr><tr><td>PTQ</td><td>76.01</td><td>59.28</td><td>83.30</td><td>61.75</td><td>83.55</td><td>60.72</td><td>84.39</td><td>56.14</td><td>76.34</td><td>60.80</td></tr><tr><td>SmoothQuant</td><td>84.55</td><td>45.77</td><td>86.41</td><td>62.56</td><td>87.06</td><td>60.75</td><td>88.86</td><td>54.26</td><td>86.14</td><td>51.30</td></tr><tr><td>RaanA</td><td>76.73</td><td>58.29</td><td>83.45</td><td>62.00</td><td>83.80</td><td>60.36</td><td>84.69</td><td>55.94</td><td>77.19</td><td>59.68</td></tr><tr><td rowspan="4">ID: CIFAR100<br/>OOD: CIFAR10</td><td>FP32</td><td>73.91</td><td>69.47</td><td>79.34</td><td>67.00</td><td>80.18</td><td>64.95</td><td>78.60</td><td>71.44</td><td>76.42</td><td>64.60</td></tr><tr><td>PTQ</td><td>63.04</td><td>80.43</td><td>64.64</td><td>81.27</td><td>65.64</td><td>80.50</td><td>62.96</td><td>82.02</td><td>60.69</td><td>79.43</td></tr><tr><td>SmoothQuant</td><td>67.02</td><td>76.14</td><td>72.75</td><td>74.15</td><td>73.57</td><td>72.16</td><td>71.94</td><td>75.10</td><td>67.94</td><td>72.29</td></tr><tr><td>RaanA</td><td>63.87</td><td>79.42</td><td>66.69</td><td>79.63</td><td>67.65</td><td>78.45</td><td>64.37</td><td>79.90</td><td>61.05</td><td>79.39</td></tr><tr><td rowspan="4">ID: ImageNet-X<br/>OOD: ImageNet-X OOD</td><td>FP32</td><td>70.62</td><td>72.69</td><td>70.24</td><td>80.11</td><td>71.01</td><td>79.01</td><td>70.16</td><td>77.81</td><td>67.69</td><td>80.64</td></tr><tr><td>PTQ</td><td>64.40</td><td>79.80</td><td>64.38</td><td>85.00</td><td>65.21</td><td>84.13</td><td>64.25</td><td>84.09</td><td>60.63</td><td>86.68</td></tr><tr><td>SmoothQuant</td><td>66.25</td><td>77.82</td><td>65.99</td><td>84.33</td><td>66.81</td><td>83.44</td><td>65.99</td><td>82.50</td><td>62.68</td><td>85.78</td></tr><tr><td>RaanA</td><td>64.49</td><td>79.95</td><td>64.55</td><td>85.27</td><td>65.39</td><td>84.37</td><td>64.54</td><td>83.82</td><td>60.81</td><td>86.52</td></tr><tr><td rowspan="4">ID: ImageNet10<br/>OOD: ImageNet20</td><td>FP32</td><td>93.84</td><td>23.00</td><td>94.43</td><td>34.60</td><td>94.48</td><td>34.60</td><td>94.72</td><td>31.00</td><td>92.54</td><td>39.60</td></tr><tr><td>PTQ</td><td>88.75</td><td>35.80</td><td>90.85</td><td>45.20</td><td>91.01</td><td>44.80</td><td>90.28</td><td>42.40</td><td>86.95</td><td>48.20</td></tr><tr><td>SmoothQuant</td><td>91.10</td><td>33.40</td><td>91.80</td><td>43.60</td><td>91.93</td><td>43.60</td><td>92.24</td><td>37.80</td><td>89.37</td><td>45.20</td></tr><tr><td>RaanA</td><td>88.19</td><td>38.40</td><td>91.09</td><td>46.20</td><td>91.22</td><td>44.80</td><td>90.20</td><td>42.00</td><td>86.66</td><td>46.60</td></tr><tr><td rowspan="4">ID: ImageNet20<br/>OOD: ImageNet10</td><td>FP32</td><td>90.51</td><td>35.80</td><td>89.86</td><td>61.90</td><td>90.10</td><td>61.20</td><td>90.87</td><td>43.90</td><td>86.43</td><td>60.80</td></tr><tr><td>PTQ</td><td>84.90</td><td>42.70</td><td>88.11</td><td>55.60</td><td>88.51</td><td>52.50</td><td>88.43</td><td>47.70</td><td>81.53</td><td>62.50</td></tr><tr><td>SmoothQuant</td><td>86.28</td><td>41.20</td><td>87.79</td><td>54.60</td><td>88.17</td><td>52.30</td><td>87.85</td><td>48.60</td><td>81.68</td><td>63.50</td></tr><tr><td>RaanA</td><td>85.23</td><td>40.00</td><td>88.05</td><td>59.50</td><td>88.42</td><td>59.80</td><td>87.82</td><td>50.20</td><td>81.44</td><td>62.20</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: DTD</td><td>FP32</td><td>68.95</td><td>83.40</td><td>79.77</td><td>63.72</td><td>80.21</td><td>64.18</td><td>71.85</td><td>79.26</td><td>73.79</td><td>68.17</td></tr><tr><td>PTQ</td><td>64.80</td><td>85.20</td><td>74.43</td><td>67.54</td><td>74.93</td><td>69.28</td><td>68.93</td><td>79.56</td><td>68.96</td><td>72.72</td></tr><tr><td>SmoothQuant</td><td>66.36</td><td>84.69</td><td>76.86</td><td>66.04</td><td>77.41</td><td>67.11</td><td>69.58</td><td>79.54</td><td>70.43</td><td>71.44</td></tr><tr><td>RaanA</td><td>64.25</td><td>85.70</td><td>75.91</td><td>66.15</td><td>76.19</td><td>68.04</td><td>69.35</td><td>79.13</td><td>66.84</td><td>74.49</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: iNaturalist</td><td>FP32</td><td>70.75</td><td>74.67</td><td>60.90</td><td>85.31</td><td>62.60</td><td>84.32</td><td>69.25</td><td>79.83</td><td>67.49</td><td>80.24</td></tr><tr><td>PTQ</td><td>65.26</td><td>81.34</td><td>58.66</td><td>88.29</td><td>60.30</td><td>87.48</td><td>68.51</td><td>82.09</td><td>65.26</td><td>83.20</td></tr><tr><td>SmoothQuant</td><td>66.89</td><td>79.35</td><td>61.02</td><td>86.82</td><td>62.65</td><td>85.57</td><td>69.82</td><td>81.72</td><td>64.39</td><td>83.39</td></tr><tr><td>RaanA</td><td>66.14</td><td>80.49</td><td>58.72</td><td>88.02</td><td>60.45</td><td>87.32</td><td>67.70</td><td>83.32</td><td>63.55</td><td>84.60</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: Places365</td><td>FP32</td><td>66.73</td><td>81.39</td><td>85.48</td><td>48.52</td><td>85.42</td><td>50.42</td><td>81.56</td><td>63.67</td><td>80.81</td><td>59.51</td></tr><tr><td>PTQ</td><td>59.78</td><td>86.64</td><td>84.51</td><td>53.30</td><td>83.99</td><td>54.36</td><td>79.46</td><td>66.31</td><td>69.65</td><td>74.91</td></tr><tr><td>SmoothQuant</td><td>62.13</td><td>84.66</td><td>83.54</td><td>55.38</td><td>83.21</td><td>57.19</td><td>79.21</td><td>67.21</td><td>74.05</td><td>69.60</td></tr><tr><td>RaanA</td><td>58.90</td><td>87.04</td><td>83.16</td><td>57.52</td><td>81.86</td><td>59.79</td><td>77.66</td><td>68.77</td><td>68.15</td><td>75.90</td></tr><tr><td rowspan="4">ID: IN1k<br/>OOD: SUN397</td><td>FP32</td><td>64.96</td><td>83.53</td><td>84.67</td><td>48.12</td><td>84.51</td><td>50.12</td><td>81.93</td><td>60.14</td><td>81.72</td><td>54.58</td></tr><tr><td>PTQ</td><td>57.63</td><td>87.95</td><td>82.70</td><td>55.79</td><td>81.94</td><td>58.43</td><td>79.10</td><td>64.49</td><td>69.81</td><td>69.12</td></tr><tr><td>SmoothQuant</td><td>60.08</td><td>86.78</td><td>82.49</td><td>53.51</td><td>81.96</td><td>56.16</td><td>80.53</td><td>63.21</td><td>74.55</td><td>65.13</td></tr><tr><td>RaanA</td><td>56.18</td><td>88.86</td><td>82.44</td><td>54.16</td><td>81.49</td><td>57.11</td><td>78.63</td><td>65.76</td><td>67.93</td><td>70.22</td></tr></tbody></table>
<table><thead><tr><th>Scenario</th><th>Method</th><th>Acc (%)</th><th>ECE</th><th>NLL</th></tr></thead><tbody><tr><td rowspan="4">ID: CIFAR10</td><td>FP32</td><td>93.59</td><td>4.260</td><td>0.229</td></tr><tr><td>PTQ</td><td>66.20</td><td>8.663</td><td>1.054</td></tr><tr><td>SmoothQuant</td><td>79.09</td><td>8.274</td><td>0.674</td></tr><tr><td>RaanA</td><td>66.81</td><td>8.059</td><td>1.033</td></tr><tr><td rowspan="4">ID: CIFAR100</td><td>FP32</td><td>74.45</td><td>13.168</td><td>1.090</td></tr><tr><td>PTQ</td><td>33.56</td><td>8.895</td><td>3.013</td></tr><tr><td>SmoothQuant</td><td>47.42</td><td>12.591</td><td>2.292</td></tr><tr><td>RaanA</td><td>35.37</td><td>9.071</td><td>2.926</td></tr><tr><td rowspan="4">ID: ImageNet-X</td><td>FP32</td><td>81.48</td><td>2.185</td><td>0.682</td></tr><tr><td>PTQ</td><td>52.23</td><td>5.503</td><td>2.084</td></tr><tr><td>SmoothQuant</td><td>62.78</td><td>4.066</td><td>1.495</td></tr><tr><td>RaanA</td><td>54.74</td><td>4.745</td><td>1.937</td></tr></tbody></table>

(a) OpenAI pre-trained ViT-L-14.

<table><thead><tr><th>Scenario</th><th>Method</th><th>Acc (%)</th><th>ECE</th><th>NLL</th></tr></thead><tbody><tr><td rowspan="4">ID: CIFAR10</td><td>FP32</td><td>94.62</td><td>2.986</td><td>0.207</td></tr><tr><td>PTQ</td><td>93.17</td><td>3.709</td><td>0.272</td></tr><tr><td>SmoothQuant</td><td>94.50</td><td>2.952</td><td>0.207</td></tr><tr><td>RaanA</td><td>93.81</td><td>3.322</td><td>0.237</td></tr><tr><td rowspan="4">ID: CIFAR100</td><td>FP32</td><td>78.03</td><td>11.065</td><td>0.946</td></tr><tr><td>PTQ</td><td>76.42</td><td>11.357</td><td>0.997</td></tr><tr><td>SmoothQuant</td><td>77.52</td><td>10.929</td><td>0.957</td></tr><tr><td>RaanA</td><td>76.31</td><td>11.340</td><td>1.014</td></tr><tr><td rowspan="4">ID: ImageNet-X</td><td>FP32</td><td>80.38</td><td>9.394</td><td>0.902</td></tr><tr><td>PTQ</td><td>79.36</td><td>9.937</td><td>0.930</td></tr><tr><td>SmoothQuant</td><td>79.95</td><td>9.477</td><td>0.916</td></tr><tr><td>RaanA</td><td>79.44</td><td>9.854</td><td>0.930</td></tr></tbody></table>

(b) LAION-400M pre-trained ViT-L-14.

<table><caption>Table 12: In-Distribution PTQ Results for ViT-L-14 models (multiple scenarios shown).</caption><thead><tr><th>Scenario</th><th>Method</th><th>Acc (%)</th><th>ECE</th><th>NLL</th></tr></thead><tbody><tr><td rowspan="4">ID: CIFAR10</td><td>FP32</td><td>88.13</td><td>6.917</td><td>0.410</td></tr><tr><td>PTQ</td><td>46.80</td><td>5.957</td><td>1.449</td></tr><tr><td>SmoothQuant</td><td>72.17</td><td>10.087</td><td>0.850</td></tr><tr><td>RaanA</td><td>46.19</td><td>5.025</td><td>1.457</td></tr><tr><td rowspan="4">ID: CIFAR100</td><td>FP32</td><td>62.79</td><td>16.012</td><td>1.590</td></tr><tr><td>PTQ</td><td>19.72</td><td>6.513</td><td>3.600</td></tr><tr><td>SmoothQuant</td><td>37.31</td><td>15.698</td><td>2.841</td></tr><tr><td>RaanA</td><td>20.07</td><td>6.514</td><td>3.600</td></tr><tr><td rowspan="4">ID: ImageNet-X</td><td>FP32</td><td>74.23</td><td>5.427</td><td>0.954</td></tr><tr><td>PTQ</td><td>24.67</td><td>1.087</td><td>3.751</td></tr><tr><td>SmoothQuant</td><td>40.20</td><td>5.604</td><td>2.735</td></tr><tr><td>RaanA</td><td>26.62</td><td>1.809</td><td>3.629</td></tr></tbody></table>

(a) OpenAI pre-trained ViT-B-16.

<table><thead><tr><th>Scenario</th><th>Method</th><th>Acc (%)</th><th>ECE</th><th>NLL</th></tr></thead><tbody><tr><td rowspan="4">ID: CIFAR10</td><td>FP32</td><td>91.70</td><td>3.658</td><td>0.276</td></tr><tr><td>PTQ</td><td>64.41</td><td>9.813</td><td>1.052</td></tr><tr><td>SmoothQuant</td><td>90.38</td><td>3.443</td><td>0.313</td></tr><tr><td>RaanA</td><td>62.13</td><td>9.966</td><td>1.116</td></tr><tr><td rowspan="4">ID: CIFAR100</td><td>FP32</td><td>71.43</td><td>10.843</td><td>1.119</td></tr><tr><td>PTQ</td><td>35.24</td><td>15.717</td><td>2.810</td></tr><tr><td>SmoothQuant</td><td>68.02</td><td>10.652</td><td>1.233</td></tr><tr><td>RaanA</td><td>35.83</td><td>16.640</td><td>2.764</td></tr><tr><td rowspan="4">ID: ImageNet-X</td><td>FP32</td><td>75.68</td><td>9.526</td><td>1.030</td></tr><tr><td>PTQ</td><td>56.09</td><td>16.584</td><td>2.083</td></tr><tr><td>SmoothQuant</td><td>70.59</td><td>11.676</td><td>1.261</td></tr><tr><td>RaanA</td><td>52.44</td><td>18.272</td><td>2.302</td></tr></tbody></table>

(b) LAION-400M pre-trained ViT-B-16.

<table><caption>Table 13: In-Distribution PTQ Results for ViT-B-16 models (multiple scenarios shown).</caption><thead><tr><th>Scenario</th><th>Method</th><th>Acc (%)</th><th>ECE</th><th>NLL</th></tr></thead><tbody><tr><td rowspan="4">ID: CIFAR10</td><td>FP32</td><td>89.76</td><td>5.031</td><td>0.338</td></tr><tr><td>PTQ</td><td>87.71</td><td>3.722</td><td>0.385</td></tr><tr><td>SmoothQuant</td><td>89.36</td><td>4.703</td><td>0.347</td></tr><tr><td>RaanA</td><td>87.84</td><td>4.118</td><td>0.384</td></tr><tr><td rowspan="4">ID: CIFAR100</td><td>FP32</td><td>65.10</td><td>10.632</td><td>1.341</td></tr><tr><td>PTQ</td><td>63.35</td><td>9.529</td><td>1.406</td></tr><tr><td>SmoothQuant</td><td>64.91</td><td>10.303</td><td>1.345</td></tr><tr><td>RaanA</td><td>63.32</td><td>9.296</td><td>1.411</td></tr><tr><td rowspan="4">ID: ImageNet-X</td><td>FP32</td><td>73.23</td><td>1.687</td><td>0.983</td></tr><tr><td>PTQ</td><td>70.48</td><td>2.052</td><td>1.105</td></tr><tr><td>SmoothQuant</td><td>72.14</td><td>1.901</td><td>1.036</td></tr><tr><td>RaanA</td><td>69.76</td><td>2.003</td><td>1.132</td></tr></tbody></table>

(a) OpenAI pre-trained ViT-B-32.

<table><thead><tr><th>Scenario</th><th>Method</th><th>Acc (%)</th><th>ECE</th><th>NLL</th></tr></thead><tbody><tr><td rowspan="4">ID: CIFAR10</td><td>FP32</td><td>90.75</td><td>3.644</td><td>0.304</td></tr><tr><td>PTQ</td><td>72.15</td><td>6.395</td><td>0.840</td></tr><tr><td>SmoothQuant</td><td>86.59</td><td>3.862</td><td>0.411</td></tr><tr><td>RaanA</td><td>73.18</td><td>6.356</td><td>0.807</td></tr><tr><td rowspan="4">ID: CIFAR100</td><td>FP32</td><td>70.59</td><td>11.284</td><td>1.122</td></tr><tr><td>PTQ</td><td>49.51</td><td>8.121</td><td>1.977</td></tr><tr><td>SmoothQuant</td><td>60.69</td><td>10.403</td><td>1.503</td></tr><tr><td>RaanA</td><td>52.45</td><td>7.214</td><td>1.827</td></tr><tr><td rowspan="4">ID: ImageNet-X</td><td>FP32</td><td>71.91</td><td>10.641</td><td>1.207</td></tr><tr><td>PTQ</td><td>58.12</td><td>14.431</td><td>1.902</td></tr><tr><td>SmoothQuant</td><td>61.76</td><td>14.062</td><td>1.705</td></tr><tr><td>RaanA</td><td>58.30</td><td>14.379</td><td>1.897</td></tr></tbody></table>

(b) LAION-400M pre-trained ViT-B-32.

<caption>Table 14: In-Distribution PTQ Results for ViT-B-32 models (multiple scenarios shown).</caption>
<table><caption>Table 15: Light QAT Results: Out-of-Distribution (OOD) Detection for OpenAI pre-trained ViT-B-32. A/F denote AU-ROC/FPR95.</caption><thead><tr><th rowspan="3">Scenario</th><th rowspan="3">Method</th><th colspan="10">Out-of-Distribution Detection Performance</th></tr><tr><th colspan="4">Traditional Methods</th><th colspan="4">VLM-based Methods</th></tr><tr><th colspan="2">MSP</th><th colspan="2">Energy</th><th colspan="2">MCM</th><th colspan="2">Gen.Neg.</th><th colspan="2">NegLabel</th></tr><tr><th></th><th></th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th></tr></thead><tbody><tr><td rowspan="5">ID: CIFAR10<br>OOD: CIFAR100</td><td>FP32</td><td>90.56</td><td>34.04</td><td>85.99</td><td>61.30</td><td>88.36</td><td>55.10</td><td>91.88</td><td>39.47</td><td>90.97</td><td>37.57</td></tr><tr><td>Basic QAT</td><td>90.92</td><td>34.39</td><td>88.07</td><td>53.28</td><td>90.44</td><td>46.76</td><td>92.38</td><td>39.18</td><td>91.52</td><td>36.39</td></tr><tr><td>QAT-LSQ</td><td>90.83</td><td>33.92</td><td>87.05</td><td>57.61</td><td>89.22</td><td>51.65</td><td>92.25</td><td>41.33</td><td>91.18</td><td>38.79</td></tr><tr><td>QAT-LSQ (PC)</td><td>90.56</td><td>34.18</td><td>88.29</td><td>52.99</td><td>90.12</td><td>48.20</td><td>92.20</td><td>39.19</td><td>91.05</td><td>37.45</td></tr><tr><td>QAT-Distill</td><td>90.86</td><td>33.70</td><td>87.63</td><td>54.56</td><td>89.74</td><td>49.30</td><td>92.51</td><td>38.62</td><td>91.38</td><td>36.75</td></tr><tr><td rowspan="5">ID: CIFAR100<br>OOD: CIFAR10</td><td>FP32</td><td>72.39</td><td>70.08</td><td>73.32</td><td>70.56</td><td>76.21</td><td>66.38</td><td>75.59</td><td>65.61</td><td>70.98</td><td>68.86</td></tr><tr><td>Basic QAT</td><td>75.26</td><td>67.32</td><td>78.53</td><td>64.38</td><td>80.46</td><td>61.27</td><td>77.84</td><td>64.76</td><td>72.65</td><td>69.25</td></tr><tr><td>QAT-LSQ</td><td>75.08</td><td>67.31</td><td>78.26</td><td>66.03</td><td>80.16</td><td>63.19</td><td>77.55</td><td>65.12</td><td>72.09</td><td>69.12</td></tr><tr><td>QAT-LSQ (PC)</td><td>75.16</td><td>66.63</td><td>77.63</td><td>65.85</td><td>79.81</td><td>62.53</td><td>78.27</td><td>62.11</td><td>72.96</td><td>66.72</td></tr><tr><td>QAT-Distill</td><td>74.43</td><td>68.16</td><td>76.65</td><td>66.92</td><td>79.33</td><td>64.20</td><td>78.01</td><td>63.56</td><td>73.08</td><td>67.14</td></tr><tr><td rowspan="5">ID: ImageNet-X<br>OOD: ImageNet-X OOD</td><td>FP32</td><td>73.03</td><td>73.24</td><td>68.66</td><td>81.34</td><td>70.60</td><td>79.13</td><td>72.56</td><td>78.69</td><td>72.82</td><td>74.97</td></tr><tr><td>Basic QAT</td><td>72.67</td><td>73.35</td><td>69.25</td><td>80.50</td><td>71.01</td><td>78.42</td><td>72.45</td><td>78.39</td><td>72.63</td><td>75.14</td></tr><tr><td>QAT-LSQ</td><td>72.63</td><td>73.68</td><td>68.94</td><td>80.44</td><td>70.79</td><td>78.72</td><td>72.66</td><td>77.66</td><td>72.76</td><td>74.73</td></tr><tr><td>QAT-LSQ (PC)</td><td>72.69</td><td>72.98</td><td>69.21</td><td>80.26</td><td>71.03</td><td>78.19</td><td>72.59</td><td>77.94</td><td>72.83</td><td>74.66</td></tr><tr><td>QAT-Distill</td><td>72.74</td><td>73.65</td><td>68.96</td><td>81.00</td><td>70.87</td><td>78.97</td><td>72.27</td><td>78.96</td><td>72.47</td><td>75.40</td></tr><tr><td rowspan="5">ID: ImageNet10<br>OOD: ImageNet20</td><td>FP32</td><td>96.28</td><td>18.20</td><td>95.41</td><td>27.00</td><td>95.52</td><td>27.20</td><td>95.58</td><td>25.00</td><td>96.72</td><td>17.40</td></tr><tr><td>Basic QAT</td><td>96.38</td><td>18.40</td><td>95.94</td><td>22.00</td><td>96.07</td><td>20.60</td><td>95.97</td><td>22.00</td><td>96.95</td><td>14.40</td></tr><tr><td>QAT-LSQ</td><td>96.55</td><td>16.20</td><td>95.87</td><td>25.20</td><td>95.98</td><td>24.00</td><td>95.86</td><td>24.40</td><td>96.85</td><td>15.40</td></tr><tr><td>QAT-LSQ (PC)</td><td>96.25</td><td>18.60</td><td>95.83</td><td>24.80</td><td>95.93</td><td>24.00</td><td>95.80</td><td>25.40</td><td>96.83</td><td>14.00</td></tr><tr><td>QAT-Distill</td><td>95.90</td><td>20.60</td><td>95.38</td><td>26.80</td><td>95.56</td><td>26.80</td><td>95.47</td><td>25.80</td><td>96.61</td><td>17.00</td></tr><tr><td rowspan="5">ID: ImageNet20<br>OOD: ImageNet10</td><td>FP32</td><td>93.27</td><td>25.20</td><td>92.76</td><td>37.40</td><td>93.32</td><td>35.20</td><td>93.74</td><td>32.70</td><td>94.87</td><td>27.00</td></tr><tr><td>Basic QAT</td><td>93.08</td><td>26.10</td><td>93.17</td><td>34.60</td><td>93.54</td><td>36.90</td><td>93.73</td><td>37.60</td><td>94.73</td><td>25.80</td></tr><tr><td>QAT-LSQ</td><td>93.19</td><td>26.70</td><td>93.04</td><td>37.80</td><td>93.61</td><td>33.40</td><td>93.83</td><td>36.90</td><td>94.66</td><td>27.80</td></tr><tr><td>QAT-LSQ (PC)</td><td>92.69</td><td>29.60</td><td>93.08</td><td>35.80</td><td>93.58</td><td>33.80</td><td>93.98</td><td>31.60</td><td>94.84</td><td>24.90</td></tr><tr><td>QAT-Distill</td><td>93.03</td><td>26.10</td><td>92.93</td><td>40.10</td><td>93.71</td><td>33.00</td><td>93.76</td><td>35.20</td><td>94.73</td><td>26.80</td></tr><tr><td rowspan="5">ID: IN1k<br>OOD: DTD</td><td>FP32</td><td>75.74</td><td>78.68</td><td>71.53</td><td>71.85</td><td>74.98</td><td>70.57</td><td>84.84</td><td>54.54</td><td>86.16</td><td>53.69</td></tr><tr><td>Basic QAT</td><td>74.59</td><td>80.81</td><td>70.04</td><td>75.27</td><td>73.72</td><td>73.41</td><td>84.12</td><td>56.10</td><td>85.33</td><td>55.93</td></tr><tr><td>QAT-LSQ</td><td>74.46</td><td>80.96</td><td>70.40</td><td>74.13</td><td>73.69</td><td>73.45</td><td>84.24</td><td>57.03</td><td>85.14</td><td>57.73</td></tr><tr><td>QAT-LSQ (PC)</td><td>75.10</td><td>80.32</td><td>71.05</td><td>72.89</td><td>74.37</td><td>72.21</td><td>84.52</td><td>55.52</td><td>85.60</td><td>55.48</td></tr><tr><td>QAT-Distill</td><td>74.92</td><td>80.06</td><td>70.20</td><td>74.16</td><td>73.98</td><td>72.77</td><td>84.21</td><td>54.97</td><td>85.58</td><td>55.30</td></tr><tr><td rowspan="5">ID: IN1k<br>OOD: iNaturalist</td><td>FP32</td><td>71.34</td><td>76.14</td><td>59.91</td><td>86.56</td><td>63.39</td><td>85.54</td><td>77.41</td><td>72.99</td><td>74.70</td><td>77.01</td></tr><tr><td>Basic QAT</td><td>71.55</td><td>75.46</td><td>60.82</td><td>86.81</td><td>64.38</td><td>85.73</td><td>76.27</td><td>75.45</td><td>74.55</td><td>78.06</td></tr><tr><td>QAT-LSQ</td><td>71.67</td><td>76.25</td><td>62.53</td><td>85.48</td><td>65.69</td><td>84.53</td><td>77.50</td><td>73.73</td><td>74.85</td><td>77.72</td></tr><tr><td>QAT-LSQ (PC)</td><td>71.32</td><td>76.31</td><td>61.89</td><td>85.59</td><td>65.04</td><td>84.74</td><td>76.49</td><td>74.83</td><td>73.79</td><td>79.20</td></tr><tr><td>QAT-Distill</td><td>71.57</td><td>76.08</td><td>60.09</td><td>87.52</td><td>63.68</td><td>86.08</td><td>76.84</td><td>74.42</td><td>74.95</td><td>77.78</td></tr><tr><td rowspan="5">ID: IN1k<br>OOD: Places365</td><td>FP32</td><td>75.00</td><td>76.47</td><td>86.43</td><td>47.10</td><td>86.90</td><td>48.23</td><td>82.50</td><td>59.85</td><td>78.46</td><td>67.33</td></tr><tr><td>Basic QAT</td><td>74.15</td><td>77.18</td><td>84.50</td><td>52.40</td><td>85.21</td><td>53.55</td><td>80.78</td><td>63.49</td><td>78.89</td><td>67.91</td></tr><tr><td>QAT-LSQ</td><td>73.97</td><td>77.98</td><td>85.63</td><td>49.47</td><td>85.99</td><td>50.45</td><td>81.00</td><td>63.48</td><td>79.08</td><td>66.95</td></tr><tr><td>QAT-LSQ (PC)</td><td>74.26</td><td>76.95</td><td>85.98</td><td>48.77</td><td>86.34</td><td>50.35</td><td>81.03</td><td>62.30</td><td>78.96</td><td>66.92</td></tr><tr><td>QAT-Distill</td><td>74.35</td><td>77.15</td><td>85.51</td><td>50.41</td><td>86.20</td><td>51.47</td><td>81.42</td><td>61.55</td><td>77.84</td><td>68.21</td></tr><tr><td rowspan="5">ID: IN1k<br>OOD: SUN397</td><td>FP32</td><td>73.65</td><td>77.33</td><td>82.75</td><td>52.08</td><td>83.60</td><td>53.40</td><td>80.34</td><td>61.72</td><td>79.58</td><td>66.08</td></tr><tr><td>Basic QAT</td><td>72.36</td><td>79.03</td><td>81.30</td><td>55.59</td><td>82.21</td><td>57.13</td><td>78.74</td><td>64.39</td><td>78.05</td><td>68.67</td></tr><tr><td>QAT-LSQ</td><td>72.61</td><td>78.28</td><td>82.23</td><td>53.87</td><td>82.89</td><td>54.96</td><td>79.16</td><td>65.10</td><td>78.00</td><td>69.11</td></tr><tr><td>QAT-LSQ (PC)</td><td>72.83</td><td>78.49</td><td>81.88</td><td>53.66</td><td>82.62</td><td>55.65</td><td>79.11</td><td>64.15</td><td>78.65</td><td>68.26</td></tr><tr><td>QAT-Distill</td><td>73.09</td><td>78.12</td><td>82.15</td><td>54.15</td><td>83.06</td><td>55.25</td><td>79.52</td><td>63.00</td><td>78.86</td><td>67.28</td></tr></tbody></table>
<table><caption>Table 16: Light QAT Results: Out-of-Distribution (OOD) Detection for LAION-400M pre-trained ViT-B-32. A/F denote AU-ROC/FPR95.</caption><thead><tr><th rowspan="3">Scenario</th><th rowspan="3">Method</th><th colspan="10">Out-of-Distribution Detection Performance</th></tr><tr><th colspan="4">Traditional Methods</th><th colspan="4">VLM-based Methods</th></tr><tr><th colspan="2">MSP</th><th colspan="2">Energy</th><th colspan="2">MCM</th><th colspan="2">Gen.Neg.</th><th colspan="2">NegLabel</th></tr><tr><th></th><th></th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th><th>A</th><th>F</th></tr></thead><tbody><tr><td rowspan="5">ID: CIFAR10<br/>OOD: CIFAR100</td><td>FP32</td><td>88.65</td><td>37.58</td><td>88.38</td><td>59.87</td><td>88.90</td><td>58.14</td><td>90.40</td><td>48.46</td><td>88.30</td><td>48.87</td></tr><tr><td>Basic QAT</td><td>83.18</td><td>47.76</td><td>87.61</td><td>56.90</td><td>87.25</td><td>57.33</td><td>88.83</td><td>51.16</td><td>83.88</td><td>54.59</td></tr><tr><td>QAT-LSQ</td><td>83.95</td><td>47.10</td><td>87.51</td><td>59.40</td><td>88.00</td><td>57.35</td><td>89.03</td><td>50.91</td><td>84.07</td><td>54.74</td></tr><tr><td>QAT-LSQ (PC)</td><td>83.59</td><td>47.45</td><td>86.94</td><td>60.76</td><td>87.47</td><td>58.88</td><td>88.86</td><td>51.38</td><td>83.84</td><td>55.87</td></tr><tr><td>QAT-Distill</td><td>82.70</td><td>49.47</td><td>87.09</td><td>59.18</td><td>86.11</td><td>60.09</td><td>88.72</td><td>52.31</td><td>83.62</td><td>55.67</td></tr><tr><td rowspan="5">ID: CIFAR100<br/>OOD: CIFAR10</td><td>FP32</td><td>73.91</td><td>69.47</td><td>79.34</td><td>67.00</td><td>80.18</td><td>64.95</td><td>78.59</td><td>71.44</td><td>75.78</td><td>65.56</td></tr><tr><td>Basic QAT</td><td>65.36</td><td>76.02</td><td>73.07</td><td>72.58</td><td>73.34</td><td>72.82</td><td>71.60</td><td>76.47</td><td>65.36</td><td>75.43</td></tr><tr><td>QAT-LSQ</td><td>65.46</td><td>75.43</td><td>72.35</td><td>74.20</td><td>73.19</td><td>72.96</td><td>70.37</td><td>77.94</td><td>65.46</td><td>77.15</td></tr><tr><td>QAT-LSQ (PC)</td><td>66.33</td><td>74.50</td><td>73.10</td><td>72.90</td><td>73.97</td><td>71.72</td><td>71.70</td><td>75.51</td><td>66.33</td><td>75.58</td></tr><tr><td>QAT-Distill</td><td>66.28</td><td>75.04</td><td>73.41</td><td>72.19</td><td>73.36</td><td>71.35</td><td>72.16</td><td>75.73</td><td>66.28</td><td>75.13</td></tr><tr><td rowspan="5">ID: ImageNet-X<br/>OOD: ImageNet-X OOD</td><td>FP32</td><td>70.62</td><td>72.69</td><td>70.24</td><td>80.11</td><td>71.01</td><td>79.01</td><td>70.16</td><td>77.81</td><td>66.74</td><td>81.99</td></tr><tr><td>Basic QAT</td><td>67.08</td><td>77.14</td><td>67.27</td><td>82.44</td><td>67.77</td><td>81.74</td><td>66.78</td><td>80.94</td><td>62.79</td><td>85.22</td></tr><tr><td>QAT-LSQ</td><td>67.07</td><td>77.01</td><td>67.13</td><td>82.72</td><td>67.93</td><td>81.47</td><td>66.74</td><td>81.36</td><td>62.63</td><td>85.62</td></tr><tr><td>QAT-LSQ (PC)</td><td>67.46</td><td>76.80</td><td>67.23</td><td>82.52</td><td>68.06</td><td>81.18</td><td>67.01</td><td>80.83</td><td>63.00</td><td>84.93</td></tr><tr><td>QAT-Distill</td><td>67.51</td><td>76.75</td><td>67.33</td><td>82.26</td><td>68.30</td><td>81.15</td><td>67.04</td><td>81.06</td><td>62.91</td><td>85.45</td></tr><tr><td rowspan="5">ID: ImageNet10<br/>OOD: ImageNet20</td><td>FP32</td><td>93.84</td><td>23.00</td><td>94.43</td><td>34.60</td><td>94.48</td><td>34.60</td><td>94.72</td><td>31.00</td><td>93.39</td><td>37.00</td></tr><tr><td>Basic QAT</td><td>92.22</td><td>32.40</td><td>92.53</td><td>46.00</td><td>92.77</td><td>45.60</td><td>92.51</td><td>40.20</td><td>90.74</td><td>42.60</td></tr><tr><td>QAT-LSQ</td><td>92.15</td><td>32.60</td><td>92.74</td><td>40.00</td><td>92.89</td><td>39.40</td><td>92.46</td><td>35.80</td><td>90.63</td><td>42.20</td></tr><tr><td>QAT-LSQ (PC)</td><td>91.68</td><td>30.80</td><td>92.55</td><td>40.40</td><td>92.71</td><td>39.60</td><td>92.27</td><td>34.80</td><td>90.39</td><td>43.20</td></tr><tr><td>QAT-Distill</td><td>91.41</td><td>30.80</td><td>92.70</td><td>37.80</td><td>92.73</td><td>39.40</td><td>92.50</td><td>37.80</td><td>90.51</td><td>41.40</td></tr><tr><td rowspan="5">ID: ImageNet20<br/>OOD: ImageNet10</td><td>FP32</td><td>90.51</td><td>35.80</td><td>89.86</td><td>61.90</td><td>90.10</td><td>61.20</td><td>90.87</td><td>43.90</td><td>87.64</td><td>59.10</td></tr><tr><td>Basic QAT</td><td>88.77</td><td>39.30</td><td>88.69</td><td>57.90</td><td>89.39</td><td>55.20</td><td>88.96</td><td>48.40</td><td>84.91</td><td>61.90</td></tr><tr><td>QAT-LSQ</td><td>86.91</td><td>40.90</td><td>88.52</td><td>55.60</td><td>88.84</td><td>54.20</td><td>88.87</td><td>53.20</td><td>84.52</td><td>60.60</td></tr><tr><td>QAT-LSQ (PC)</td><td>86.95</td><td>41.00</td><td>88.32</td><td>60.80</td><td>88.63</td><td>59.60</td><td>88.76</td><td>50.30</td><td>84.63</td><td>62.90</td></tr><tr><td>QAT-Distill</td><td>86.92</td><td>39.10</td><td>88.74</td><td>55.80</td><td>88.95</td><td>55.10</td><td>88.95</td><td>49.70</td><td>84.65</td><td>64.70</td></tr><tr><td rowspan="5">ID: IN1k<br/>OOD: DTD</td><td>FP32</td><td>68.95</td><td>83.40</td><td>79.77</td><td>63.72</td><td>80.21</td><td>64.18</td><td>71.85</td><td>79.26</td><td>73.69</td><td>68.37</td></tr><tr><td>Basic QAT</td><td>66.51</td><td>85.35</td><td>78.42</td><td>62.83</td><td>78.57</td><td>64.53</td><td>72.52</td><td>76.19</td><td>71.89</td><td>69.61</td></tr><tr><td>QAT-LSQ</td><td>66.62</td><td>84.48</td><td>78.26</td><td>62.41</td><td>78.62</td><td>64.20</td><td>71.65</td><td>76.82</td><td>72.27</td><td>68.93</td></tr><tr><td>QAT-LSQ (PC)</td><td>66.61</td><td>85.18</td><td>79.37</td><td>61.03</td><td>79.67</td><td>63.09</td><td>72.48</td><td>75.86</td><td>71.90</td><td>69.30</td></tr><tr><td>QAT-Distill</td><td>66.80</td><td>85.16</td><td>78.21</td><td>63.07</td><td>78.29</td><td>65.10</td><td>72.78</td><td>76.23</td><td>72.62</td><td>68.49</td></tr><tr><td rowspan="5">ID: IN1k<br/>OOD: iNaturalist</td><td>FP32</td><td>70.75</td><td>74.67</td><td>60.90</td><td>85.31</td><td>62.60</td><td>84.32</td><td>69.25</td><td>79.83</td><td>66.28</td><td>80.94</td></tr><tr><td>Basic QAT</td><td>67.42</td><td>79.30</td><td>61.87</td><td>85.41</td><td>63.17</td><td>84.72</td><td>70.44</td><td>80.21</td><td>65.58</td><td>80.98</td></tr><tr><td>QAT-LSQ</td><td>68.28</td><td>77.59</td><td>60.63</td><td>86.29</td><td>62.33</td><td>85.30</td><td>69.74</td><td>80.10</td><td>65.77</td><td>81.12</td></tr><tr><td>QAT-LSQ (PC)</td><td>68.02</td><td>78.29</td><td>61.26</td><td>85.99</td><td>62.88</td><td>85.05</td><td>70.06</td><td>79.41</td><td>65.84</td><td>80.68</td></tr><tr><td>QAT-Distill</td><td>67.57</td><td>79.11</td><td>61.02</td><td>85.78</td><td>63.18</td><td>84.34</td><td>70.07</td><td>79.71</td><td>65.38</td><td>80.79</td></tr><tr><td rowspan="5">ID: IN1k<br/>OOD: Places365</td><td>FP32</td><td>66.73</td><td>81.39</td><td>85.48</td><td>48.52</td><td>85.42</td><td>50.42</td><td>81.56</td><td>63.67</td><td>79.58</td><td>61.00</td></tr><tr><td>Basic QAT</td><td>62.00</td><td>84.94</td><td>85.01</td><td>51.33</td><td>84.79</td><td>52.89</td><td>81.52</td><td>61.78</td><td>74.72</td><td>66.96</td></tr><tr><td>QAT-LSQ</td><td>62.12</td><td>85.08</td><td>84.83</td><td>52.28</td><td>84.43</td><td>53.95</td><td>81.76</td><td>61.54</td><td>75.40</td><td>65.61</td></tr><tr><td>QAT-LSQ (PC)</td><td>62.59</td><td>84.73</td><td>85.14</td><td>50.70</td><td>84.76</td><td>52.29</td><td>81.94</td><td>61.10</td><td>75.76</td><td>64.53</td></tr><tr><td>QAT-Distill</td><td>62.41</td><td>84.75</td><td>85.13</td><td>51.05</td><td>84.67</td><td>53.08</td><td>81.56</td><td>62.10</td><td>75.45</td><td>65.62</td></tr><tr><td rowspan="5">ID: IN1k<br/>OOD: SUN397</td><td>FP32</td><td>64.96</td><td>83.53</td><td>84.67</td><td>48.12</td><td>84.51</td><td>50.12</td><td>81.93</td><td>60.14</td><td>80.99</td><td>58.21</td></tr><tr><td>Basic QAT</td><td>60.18</td><td>87.18</td><td>84.21</td><td>49.84</td><td>83.51</td><td>52.21</td><td>80.58</td><td>61.65</td><td>73.40</td><td>67.46</td></tr><tr><td>QAT-LSQ</td><td>60.04</td><td>87.27</td><td>84.42</td><td>49.96</td><td>83.74</td><td>52.44</td><td>80.44</td><td>62.01</td><td>73.57</td><td>67.88</td></tr><tr><td>QAT-LSQ (PC)</td><td>60.90</td><td>87.06</td><td>84.32</td><td>50.23</td><td>83.76</td><td>52.74</td><td>81.23</td><td>61.67</td><td>74.28</td><td>67.50</td></tr><tr><td>QAT-Distill</td><td>59.96</td><td>87.27</td><td>84.14</td><td>49.93</td><td>83.64</td><td>52.17</td><td>80.43</td><td>62.40</td><td>72.89</td><td>68.94</td></tr></tbody></table>
<table><caption>Table 17: Consolidated In-Distribution Light QAT Results for OpenAI and LAION-400M pre-trained ViT-B-32 models.</caption><thead><tr><th rowspan="2">Scenario</th><th rowspan="2">Method</th><th colspan="3">OpenAI ViT-B-32</th><th colspan="3">LAION ViT-B-32</th></tr><tr><th>Acc(%)</th><th>ECE</th><th>NLL</th><th>Acc(%)</th><th>ECE</th><th>NLL</th></tr></thead><tbody><tr><td rowspan="5">CIFAR10</td><td>FP32</td><td>89.76</td><td>5.03</td><td>0.34</td><td>90.75</td><td>3.64</td><td>0.30</td></tr><tr><td>Basic QAT</td><td>90.25</td><td>2.91</td><td>0.31</td><td>83.90</td><td>4.36</td><td>0.48</td></tr><tr><td>QAT-LSQ</td><td>90.07</td><td>3.23</td><td>0.32</td><td>85.90</td><td>3.84</td><td>0.42</td></tr><tr><td>QAT-LSQ (PC)</td><td>89.49</td><td>3.21</td><td>0.33</td><td>85.60</td><td>3.92</td><td>0.43</td></tr><tr><td>QAT-Distill</td><td>90.09</td><td>3.49</td><td>0.31</td><td>81.78</td><td>4.76</td><td>0.54</td></tr><tr><td rowspan="5">CIFAR100</td><td>FP32</td><td>65.10</td><td>10.63</td><td>1.34</td><td>70.59</td><td>11.28</td><td>1.12</td></tr><tr><td>Basic QAT</td><td>67.27</td><td>6.72</td><td>1.23</td><td>61.95</td><td>8.62</td><td>1.43</td></tr><tr><td>QAT-LSQ</td><td>66.63</td><td>7.31</td><td>1.25</td><td>62.63</td><td>8.86</td><td>1.39</td></tr><tr><td>QAT-LSQ (PC)</td><td>67.08</td><td>7.74</td><td>1.24</td><td>63.72</td><td>8.73</td><td>1.34</td></tr><tr><td>QAT-Distill</td><td>66.83</td><td>8.40</td><td>1.25</td><td>63.45</td><td>8.86</td><td>1.37</td></tr><tr><td rowspan="5">ImageNet-X</td><td>FP32</td><td>73.23</td><td>1.69</td><td>0.98</td><td>71.91</td><td>10.64</td><td>1.21</td></tr><tr><td>Basic QAT</td><td>72.70</td><td>1.28</td><td>1.00</td><td>64.21</td><td>12.91</td><td>1.57</td></tr><tr><td>QAT-LSQ</td><td>72.93</td><td>1.77</td><td>0.99</td><td>64.31</td><td>12.73</td><td>1.56</td></tr><tr><td>QAT-LSQ (PC)</td><td>73.15</td><td>1.73</td><td>0.99</td><td>64.66</td><td>12.88</td><td>1.55</td></tr><tr><td>QAT-Distill</td><td>72.89</td><td>1.78</td><td>1.00</td><td>64.94</td><td>12.78</td><td>1.53</td></tr><tr><td rowspan="5">ImageNet10</td><td>FP32</td><td>98.80</td><td>0.74</td><td>0.03</td><td>97.60</td><td>1.37</td><td>0.06</td></tr><tr><td>Basic QAT</td><td>99.20</td><td>0.75</td><td>0.03</td><td>98.20</td><td>1.83</td><td>0.07</td></tr><tr><td>QAT-LSQ</td><td>99.20</td><td>0.84</td><td>0.03</td><td>97.20</td><td>1.79</td><td>0.09</td></tr><tr><td>QAT-LSQ (PC)</td><td>99.00</td><td>0.68</td><td>0.03</td><td>97.60</td><td>1.22</td><td>0.09</td></tr><tr><td>QAT-Distill</td><td>99.00</td><td>0.55</td><td>0.03</td><td>97.80</td><td>1.52</td><td>0.08</td></tr><tr><td rowspan="5">ImageNet20</td><td>FP32</td><td>94.90</td><td>1.60</td><td>0.16</td><td>95.80</td><td>1.68</td><td>0.16</td></tr><tr><td>Basic QAT</td><td>94.70</td><td>1.36</td><td>0.16</td><td>93.80</td><td>1.96</td><td>0.20</td></tr><tr><td>QAT-LSQ</td><td>94.60</td><td>1.47</td><td>0.17</td><td>92.20</td><td>2.84</td><td>0.24</td></tr><tr><td>QAT-LSQ (PC)</td><td>94.70</td><td>1.17</td><td>0.16</td><td>92.90</td><td>2.44</td><td>0.23</td></tr><tr><td>QAT-Distill</td><td>94.50</td><td>1.47</td><td>0.16</td><td>93.20</td><td>2.92</td><td>0.22</td></tr><tr><td rowspan="5">IN1k</td><td>FP32</td><td>63.36</td><td>1.86</td><td>1.40</td><td>62.89</td><td>13.72</td><td>1.66</td></tr><tr><td>Basic QAT</td><td>62.79</td><td>1.72</td><td>1.42</td><td>54.97</td><td>15.25</td><td>2.06</td></tr><tr><td>QAT-LSQ</td><td>62.74</td><td>1.67</td><td>1.43</td><td>54.82</td><td>14.98</td><td>2.07</td></tr><tr><td>QAT-LSQ (PC)</td><td>62.87</td><td>1.72</td><td>1.42</td><td>55.51</td><td>15.20</td><td>2.03</td></tr><tr><td>QAT-Distill</td><td>62.81</td><td>1.73</td><td>1.42</td><td>55.92</td><td>15.21</td><td>2.01</td></tr></tbody></table>