QT-DoG: Quantization-Aware Training for Domain Generalization
Authors: Saqib Javed
Hieu Le
Mathieu Salzmann
Venue: ICML 2025 poster (2025)
Keywords: Domain Generalization
Quantization
Ensemble
Network Compression
Flat Minima
Regularization
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: QT-DoG leverages weight quantization to promote flatter minima, enhancing generalization across unseen domains while reducing model size and computational costs.

Abstract: A key challenge in Domain Generalization (DG) is preventing overfitting to source domains, which can be mitigated by finding flatter minima in the loss landscape. In this work, we propose Quantization-aware Training for Domain Generalization (QT-DoG) and demonstrate that weight quantization effectively leads to flatter minima in the loss landscape, thereby enhancing domain generalization. Unlike traditional quantization methods focused on model compression, QT-DoG exploits quantization as an implicit regularizer by inducing noise in model weights, guiding the optimization process toward flatter minima that are less sensitive to perturbations and overfitting. We provide both an analytical perspective and empirical evidence demonstrating that quantization inherently encourages flatter minima, leading to better generalization across domains. Moreover, with the benefit of reducing the model size through quantization, we demonstrate that an ensemble of multiple quantized models further yields superior accuracy than the state-of-the-art DG approaches with no computational or memory overheads. Code is released at: https://saqibjaved1.github.io/QT_DoG/.

QT-DoG: Quantiration-aware Training for Domain Generalization
Saqib Javed Hieu Le
Mathieu Salzmann
Abstract
70
A key challenge in Domain Generalization (DG) is preventing overfitting to source domains, which can be mitigated by finding flatter minima in the loss landscape. In this work, we propose Quantization-avee -wartoeoooaaweeeeeeeeeeeeeeeee Training for Domain Gener- alization (QT-DoG) and demonstrate that weight quantization effectively leads to flatter minima in the loss landscape, thereby enhancing domain generalization. Unlike traditional quantization methods focused on model compression, QT-DoG exploits quantization as an implicit regularizer by inducing noise in model weights, guiding the optimization process toward flatter minima that are less sensitive to perturbations and over- fitting. We provide both analytical perspec- tive and empirical evidence demonstrating that quantization inherently encourages flatter min- ima, leading to better generalization across do- mains. Moreover, with benefit of reducing the model size through quantization, we demonstrate that ensemble of multiple quantized models further yields superior accuracy than the state-of- the-art DG approaches with no computational or memory overheads. Code is released at: ht https: / //sagibjavedi werll.gittee github. Lo/OT_DoCi
Test Domain Accuracy [%6
69
M-50
M
M+5
68
pus
One Model
62
ERM
MIRO
SWAD
DART
QT-DoG
DIWA
EOA
EOQ
Algorithms
Figure I. Performance Comparison on the Domainbed Bench- mark. We show the average accuracy on different datasets. One Model refers to methods training single ResNet-50 model. Mul- tiple Models refers to training models for averaging or ensem- bling, which affects the training cost. We compare QT-DoG and EoQ to other state-of-the- art methods. The marker size is propor- tional to the memory footprint. EoQ shows superior performance despite being 4 times smaller than its full-precision counterpart. Additionally. QT-DoG demonstrates comparable performance One Model methods, despite its significantly smaller size.
also in new, unseen (target) data distributions (Blanchard et al., 2011; Muandet al., 2013; Zhou et al., 2022).
In the broader context of generalization, with training and test data drawn from the same distribution, the literature has revealed a relationship between the flatness of the loss land- scape and the generalization ability of deep learning mod- els (Keskar et al., 2017; Dziugaite & Roy, 2017: Garipov et al., 2018; Izmailov et al., 2018; Jiang et al., 2020; Foret et al., 2021; Zhang et al., 2023). This relationship has then been leveraged by many recent works, demonstrating that flatter minimum also improves Out-of-Distributt (OOD) performance (Cha et al., 2021; Ramé et al., 2023; Arpit et al., 2022). At the heart of all these DG methods lies the idea of weight averaging (Izmailov et al., 2018), which involves averaging weights from several trained models or at various stages of the training process.
1. Introduction
Many works have shown that deep neural networks trained under the assumption that the training and test samples are drawn from the same distribution fail to generalize in the presence of large training-testing testing discrepancies, such as texture (Geirhos al., 2019; Bahng et al., 2020), back- ground (Xiao et al., 2020), or day-to-night (Dai & Van Gool, 2018; Michaelis et al., 2019) shifts. Domain Generalization (DG) addresses this problem and aims to learn models that perform well not only in the training (source) domains but
In this work, we demonstrate that flatter minima in the loss landscape be effectively achieved through weight quan- tization using Quantization-awan Training (QAT), making it an effective approach for DG. By restricting the possible weight values to a lower bit precision, quantization imposes constraints on the weight space, introducing quantization noise into the network parameters. This noise, as discussed
'CVLab, EPFL, Switzerland. SSwsss Data Science Center, Switzerland., Correspondence Mathicu Salzmann <math- Saalzmann@ccpl.cc..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

QT-DoG: Quantization-aware Training for Domain Generalization
in prior works (An, 1996; Murray& & Edwards, 1992: Good- fellow al., 2016; Hochreiter & Schmidhuber, 1994), acts as form of regularization that naturally encourages the opti- mization process to converge toward flatter minima. Further- more, our results show that models trained with quantization not only generalize better across domains but also reduce overfitting to source domains. To the best of our knowledge, this is the first work to explicitly explore the intersection of quantization and domain generalization. Through both analytical reasoning and empirical validation, we provide strong evidence that QAT promotes flatter minima, leading to enhanced generalization performance on unseen domains.
2.1.1. DOMAIN ALIGNMENT
The methods in this category focus on reducing the differ- ences among the source domains and learn domain- invariant features (Arjovsky et al., 2019; Krueger et al., 2021; Rame et al., 2022a; Sun et al., 2016; Sagawa et al., 2020; Ganin et al.. 2016; Li et al., 2023; Cheng et al., 2024). The core idea is that, if the learnt features are invariant across the different source domains, they will also be robust to the unseen target domain. For matching feature distributions across source domains, DANN (Ganin et al., 2016) uses an adversarial loss while CORAL (Sun & Saenko, 2016) and DICA (Muandet et al., 2013) seek to align latent statistics of different domains. Unfortunately, most of these methods fail to generalize well and were shown not to outperform ERM on various benchmarks (Gulrajani & Lopez-Paz, 2021; Ye et al., 2022; Koh et al., 2021).
The benefit of having fast and light-weight quantized mod- els then further allow us to even make an ensemble of them, termed Ensemble of Quantization (EoQ). EoQ achieves su- perior performance while maintaining the computational efficiency of single full-precision model. This stands in contrast to ensemble-based methods like (Ramé et al., 2023; Arpit et al., 2022), which require storing and running multiple full -uee-prrcision models. With our approach, quanti- zation not only improves generalization but also reduces the model's memory footprint and computational cost infer- ence. As shown in Figure 1, EoQ yields a model with a mem- ory footprint similar to the state-of-the approaches and much smaller than other ensemble-base methods, yet outperforms all its competitors in terms of accuracy.
2.1.2. REGULARIZATION
In the literature, various ways of regularizing models (im- plicit and explicit) have also been proposed to achieve bet- ter generalization. For example, invariant risk minimiza- tion (Arjovsky et al., 2019) relies on a regularization tech- nique such that the learned classifier is optimal even under a distribution shift. Moreover, (Huang et al., 2020) tries to suppress the dominant features learned from the source domain and pushes the network to use other features corre- lating with the labels. Furthermore, (Krueger et al., 2021) proposes risk extrapolation that uses regularization to mini- mize the variance between domain-wise losses, considering that is representative of the variance including the target domain.
single-model DG
Our contributions can be summarized as follows:
We are the first to demonstrate that quantization-aware training, traditionally used for model compression, can serve an implicit regularizer, with quantization noise enhancing domain generalization.
1.1.3. VISION TRANSFORMERS
We demonstrate that QAT promotes flatter minima in the loss landscape and provide an analytical perspective behind this effect. Additionally, we show that QAT stabilizes model behavior on OOD data during training.
Recent studies have increasingly utilized vision transformers for domain generalization (Sultana et al., 2022). Some ap- proaches enhance vision transformers by integrating knowl- edge distillation (Hinton et al., 2015) and leveraging text modality from CLIP (Radford et al., 2021) to learn more features (Moayeri et al., 2023; Chen et al., 2024; Huang et al., 2023; Liu et al., 2024; Yu al., 2024; Shu et al., 2023; Addepalli et al., 2024).
In contrast traditional DG methods that often in- crease model size or computational cost, QT-DoG not only improves generalization but also significantly re- duces the model size, enabling efficient deployment in real-world applications. EoQ, for instance, requires nearly 6 times less memory than Arpit et al. (2022) and 12 times less training compute compared to Ramé et al. (2023), which trains 60 models for diverse averaging.
2. 1.4. ENSEMBLING
Ensembling of deep networks (Lakshminarayanan et al., 2017: Hansen & Salamon, 1990; Krogh & Vedelsby, 1995) is foundational strategy and has consistently proven to be robust in the past. Many works have been proposed to train multiple diverse models and combine them obtain better in in-domain accuracy and robustness to domain shifts (Arpit et al., 2022; Thopalli et al., 2021; Mesbah et al., 2022; Li et al., 2022: Lee et al., 2022; Pagliardini et al., 2023). However, ensembles require multiple models to be stored
2. Related Work
2.1. Domain Generalization
Numerous multi-source domain generalization (DG) meth- ods have been proposed in the past. In this section, we review some of the recent approaches, categorizing them into different groups based on their methodologies.
2

---

QT-DoG: Quantization-aware Training for Domain Generalization
and separate forward pass for each model, which increases the computational cost and memory footprint, especially if the models are large.
network during the quantization process and thus better preserves the model's full-precision accuracy. Yang et al. (2023); Esser et al. (2020); Zhou et al. (2017); Bhalgat et al. (2020); Yamamoto (2021); Yao et al. (2020); Shin et al. (2023). In the next section, we provide some background on quantization and on the method we will use in our approach. Our goal in this work is not to introduce a new quantization strategy but rather demonstrate the impact of quantization generalization.
2.1.5. WEIGHT AVERAGING
Combining or averaging weights from different training stages or models has emerged as a robust approach to im- prove OOD generalization (Wortsman et al., 2022b; Matena & Raffel, 2022; Wortsman et al., 2022a: Gupta et al., 2020; Choshen et al., 2022; Wortsman et al., 2021; Maddox et al., 2019; Benton al., 2021; Cha et al., 2021; Jain al., 2023; Ramé al., 2023). Techniques like SWAD (Cha et al., 2021) leverage weight averaging to identify flat minima, reducing overfitting and enhancing generalization under distribution shifts. Similarly, DiWA (Rame et al., 2022b) combines weights from independently trained models to improve robustness through increased diversity.
3. Domain Generalization by Quantization
We build our method on the simple ERM approach to show- case the effects of quantization on the training process and on the generalization to unseen data from different do- main. Despite the simplicity of this approach, we will show in Section 4.1 that it yields significant accuracy boost on the test data from the unseen target domain. Furthermore, it stabilizes the behavior of the model on OOD data during training, making it similar to that on the in-domain data. In the remainder of this section, we focus on providing some insights on how quantization enhances DG.
Arpit al. (2022) integrates ensembling with weight av- eraging, yielding superior performance compared to either method alone, albeit with significant memory and computa- tional costs. To address these challenges, we demonstrate that quantization can improve generalization while reducing resource demands.
3.1. Quantization
Although flatter minima are not universally indicative of better domain generalization (Andriushchenko al., 2023), they remain valuable tool for improving robustness in many scenarios. Moreover, recent findings (Mueller et al., 2023) highlight that selective application of SAM (Foret et al., 2021), such as restricting it to normalization layers, can further refine its effectiveness. The consistent empirical success of SAM underscores its reliability as method for enhancing domain generalization, despite the nuanced rela- tionship between flatness and performance across different settings.
Let w be a single model weight to be quantized, s the quan- tizer step size, and QNv and Qr the number of negative and positive quantization levels, respectively. We define the quantization process that computes w, quantized and integer scaled representation of the weights,
T |clip(v/a, -QN Qp)l.
(1)
where the function clip(k; r2) is defined as
T2 [k] if < k
2.2. Model Quantization
clip(ks, r1,)2
ri if T2 2 12
(2)
Model quantization is used in deep learning reduce the memory footprint and computational requirements of deep network. In conventional neural network, the model pa- rameters and activations are usually stored as -igh-precision bhating-poinn numbers, typically 32-bit or 64-bit. The pro- cess of model quantization entails transforming these pa- rameters into lower bit-width representations, such as 8-bit integers or binary values. Existing techniques fall into two main categories. Post- post-Trainin Quantization (PTQ) quan- tizes a pre- trained network using a small calibration dataset and is thus relatively simple to implement (Nagel et al., 2020; Li et al., 2021; Frantar et al., 2022; Zhao et al., 2019; Cai et al., 2020; Nagel et al., 2019; Shao et al., 2024; Lin al., 2024; Chee et al., 2023; Li et al., 2025; Ramachan- dran et al., 2024; Shang al., 2024; Zhang& & Shrivastava, 2025). Quantization-Awer Training (QAT) retrains the
Here, [k] represents rounding & to nearest integer. If we quantize a weight to bits, for unsigned data Q2 = and Qr = 2 1, and for signed data QN 20-1 and Qp 2 2
Note that the quantization process described in Eq. 1 yields scaled value. A quantized representation of the data at the same scale as w can then be obtained as
Wip
Φ X 8.
(3)
This transformation results in a discretized weight space that inherently introduces noise. We demonstrate generalization ability of QT-DoG with different quantization methods in section 4. 1.3.
3

---

QT-DoG: Quantization-wwar Training for Domain Generalization
Lonn Arand metple Speneity
This has been shown to be prevented by introducing noise in the model weights during training (An, 1996; Murray & Edwards, 1992; Goodfellow et al., 2016; Hochreiter & Schmidhuber, 1994). Here, we argue that quantization inher- ently induces such noise and thus helps to find flatter minima. Here, we argue that quantization inherently induces noise, which aids in finding flatter minima. To support this, we use a second-order Taylor series expansion to analyze how quantization-induced perturbations affect the curvature of the loss landscape, leading to flatter minima.
Figure 2. Distribution of Quantization Noise and Weights. We plot the weights(left), quantization noise(middle) and symme- tryrright) of random layer in ResNet-50 model. We found KL- divergence to be 0.0009 between quantization noise and uniform distribution with same minimum and maximum value.
Let y: = f(x, w) represent the predicted output of the network f. which is parameterized by the weights w. A quantized network can then be represented as
3.2. Quantization as uniform noise
Quantization can be modeled as additive noise under certain assumptions (Gray& & Neuhoff, 1998). The noise or error in- troduced by the quantizer is bounded within - $. 5 where s represents the quantization step size, as described in the previous section. When is small relative to the dynamic range of the weight distribution, the quantization noise can be well approximated by a uniform distribution (Boncelet, 2009).
fax wg) fx, + A) 1).".
where We denotes the quantized weights and yi the cor- responding prediction. The quantized weights can thus be thought of as introducing perturbations (4) to the full- precision weights, akin to noise affecting the weights.
Such noise induced by the weight quantization can also be seen as a form of regularization, akin to more traditional methods. For small perturbations, (An, 1996; Murray & Edwards, 1992; Goodfellow et al., 2016) show that this type of regularization encourages the parameters to navigate towards regions of the parameter space where small pertur- bations of the weights have minimal impact the output, i.e., flatter minima.
This observation is empirically validated in Figure 2, where we analyze the weight distribution of randomly selected layer and apply 7-bit quantization. We then compute the Kullback-Leibler divergence (Kullback & Leibler, 1951) between the resulting quantization noise and a uniform dis- tribution with the same minimum and maximum values, find- ing to be very low. We extend this analysis in Appendix J across multiple layers, we observe that neural network weights generally tend to exhibit smooth and symmetric distributions, reinforcing the conclusion that quantization noise/error generally follows a uniform distribution. Prior works (Zhang et al., 2024; Murray& & Edwards, 1992) show that adding noise to network weights can improve gener- alization, and similarly. quantization- training with structured uniform noise also enhances generalization.
When noise is introduced via quantization, second-order Taylor series approximation of the loss function for the perturbed weights w + A can be expressed as
1 ATHA 2
L(w + + R ((w) VL(w) T
A
(4)
where L(w) is the loss at the original weights w, (L(w) is the gradient of the loss at w, and 7 = V2L(w) is the Hes-
sian matrix, which contains second-order partial derivatives of the loss function with respect to the weights, representing the curvature of the loss surface.
3.3. Quantization Leads to Flat Minima
In the literature (Rame et al., 2022b; Arpit et al., 2022:
Krueger et al., 2021; Cha et al., 2021; Rame al., 2022b; Foret et al., 2021), it has been established that a model's generalization ability can be increased by finding a flatter minimum during training. This is the principle we exploit in our work, but from the perspective of quantization, and pro- vide an analytical view into how it contributes to achieving flatter minima. In practice, ERM can have several solutions with similar training loss values but different generalization ability. Even when the training and test data are drawn from the same distribution, the standard optimizers, such as SGD and Adam (Kingma & Ba, 2015), often lead to sub-optimal generalization by finding sharp and narrow minima (Keskar et al., 2017; Dziugaite& & Roy, 2017; Garipov et al., 2018: Izmailov et al., 2018; Jiang et al., 2020; Foret et al., 2021).
Eq. 4 shows how the quantization noise A interacts with the curvature H of the loss function. In regions with large curvature (sharp minima), the Hessian H. has large eigenval- ues, and even small perturbations A result in large increases in the loss (Dinh et al., 2017). In contrast, in flat regions (small eigenvalues of H), the loss remains nearly unchanged for small perturbations. Quantization noise acts as an im- plicit regularizer by introducing perturbations A that disrupt the model's weight updates. In sharper minima, where the Hessian H eigenvalues are large, small noise significantly increases the loss, causing the model to "escape" these re- gions and search for flatter, more stable minima. In flatter regions, where the Hessian H. eigenvalues are small, the noise has less impact, helping the model settle into these
4

---

QT-DoG: Quantization-aware Training for Domain Generalization
ERM -X-
SAM
SWA
SWAD -X- QT-DOG
Average Train Flatness
Averuge Test Flatness
0.25
[F,(w))
0.30
Flatuess [P,(w)]
044
0.3
0.15
0.66
0.00
0.0
Radius [V]
Radius IM
Figure 3. Local Flatness Comparison: We plot the average training (left) and testing (right) local flatness Fr(w) (Eq. 5) for ERM (Gul- rajani & Lopez-Paz, 2021), SAM (Foret al., 2021), SWA (Izmailov al., 2018) and SWAD (Cha et al., 2021) by varying the radius y on different domains of PACS. We evaluate the training flatness Fi- 2 (w)) on the seen domains (left) and the test flatness F. (wv) on the unseen domains (right).
regions with lower loss. This encourages convergence to so- lutions that are less sensitive to small changes in the input or model parameters, which is beneficial for out-of-distribtiion (OOD) generalization.
cess. Similar to (Dinh et al., 2017; Cha et al., 2021), we interpret flat minima as "a large connected region in weight space where the error remains approximately constant," as defined by (Hochreiter & Schmidhuber, 1997). Our loss flatness analysis shows that QT-DoG can find flatter mini- mum in comparison to not only ERM but also SAM (Foret et al., 2021) and SWA (Izmailov et al., 2018).
In the case of quantization training, the induced noise A is influenced by the quantization bin width or the quan- tizer step size s, and thus ranges between - and + This 8 is directly dependent on the quantization levels or the bit-width chosen for weight quantization. As the number of bits per weight decreases, the amount of induced noise increases. Hence, the impact of the additional noise can be weighed by choosing an optimal bit-width. As will be shown in Section 4, certain bit-widths thus yield better and flatter minima that enhance generalization. However, if we induce too much noise(very low bit- bit-precision) it introduces over-regularization. This excessive noise can overly restrict the search space, preventing the model from reaching a good solution. Instead, the optimization process may focus on minimizing the loss in way that avoids sharp regions, but sacrifices the ability to find true minimum of the loss function. This is also evident in Table 10 in the appendix.
Following the approach in Cha et al. (2021), we quantify local flatness (w) by measuring the expected change in loss values between model with parameters w and a perturbed model with parameters /wo| = |w]| + 7 where w' lies sphere of radius y centered at w.. This is expressed as
Fww () E(w)],
where E(w) denotes the accumulated loss over the samples of potentially multiple domains.
For our analysis, we will evaluate flatness in both the source domains and the target domain, and thus E (ww) is evaluated using either source samples or target ones accordingly.
As Cha et al. (2021), we approximate F,, (2x) by Monte- Carlo sampling with 100 samples. In Figure 3, we compare the Fw) of QT-DoG to that of ERM (Gulrajani & Lopez- Paz, 1, SAM (Foret et al., 2021), SWA (Izmailov et al., 2018) 8) and SWAD (Cha et al., 2021) for different radii 7 QT-DoG not only finds a flatter minimum than ERM, SAM and SWA but also yields a comparable flatness to SWAD's despite being 75% smaller in model size.
Moreover, Rissanen (1978); Hochreiter & Schmidhuber (1997) show that a flatter minimum corresponds to a low complexity network and requires fewer bits of information per weight. More importantly, Hochreiter & Schmidhuber (1997) demonstrates the importance of the bit-precision of the network weights and adds regularization term in the loss function that seeks to lower the weight bit bit-precision to lead to flatter minima. Here, by using quantization, we are explicitly reducing bit-precision of the network weights, thus achieving the same goal.
3.5. Stable Training Process
Here, demonstrate the robustness of out-of-domain per- formance to model selection using the in-domain validation set. Specifically, we seek to show that accuracy on the in- domain validation data is a good measure to pick the best model for out-of -o-mann distribution. Therefore, we assume that during training, the model selection criterion based on
3.4. Empirical Analysis of Quantization-aware Training and Flatness
In this section, we demonstrate that a flatter minimum is reached when incorporating quantization in the ERM pro-
5

---

QT-DoG: Quantization-aware Training for Domain Generalization
with Quantization
Without Quantization
Quantization Step
location 100
location 46
109
Volidation AC:IN)
Unitdwtion Arr7%))
1933
7000
9030
1828
1460
1000
2303
5001
Peraaions
Iterations
location
50
Doman AcceM
Doman
23
Rerations
Iterations
Figure Model Quantization improves out-of-domain performance as well as training stability. The plots were computed using the Terrainc dataset with domain L100 (left) and L46 (right) as test domain, and the other domains as training/validation data. The top two plots illustrate in-domain validation accuracy, while the bottom two represent out-of-domain test accuracy. The network used for these plots was a ResNet-50. For quantized models, shown in blue in each plot, quantized the model after 2000 steps. Note that the model accuracy is not only better with quantization but also much more stable for out distribution data after the quantization step.
this validation data can select the best model for the OOD data even if the model starts to overfit. In other words, it is expected that the out-of-domain evaluation at each point of the training phase should improve or rather stay stable the model is close to overfitting to the in-domain data. For these experiments, we use the Terralncognita dataset (Beery et al., 2018) and consider the same number of iterations for the DomainBed protocol (Gulrajani & Lopez-Paz. 2021).
cess to obtain smaller quantized models. We refer to this as the Ensemble of Quantization (FoQ). As Breiman (1996), we use the bagging method to combine the multiple predic- tions. Therefore, the class predicted by EoQ for an input x is given by
ara max Softmax ( EE((;;
(6)
F.
where E is the fotal number of modèls' in the ensemblee, wi denotes the parameters of the ith quantized model, and the subscript & denotes the h element of the vector argument. Finally, we use the in-domain validation set performance to pick the best model state (weights) wi of the zth quantized model used in the ensemble.
As can be seen in Figure 4, vanilla ERM (without quanti- zation) quickly overfits to the in-domain validation/training dataset. That is, the OOD performance is highly unstable during the whole training process. By contrast, our quan- tized model much more stable. Specifically, we quantize our model at 2000 steps, and it can be seen that the model performance out-of-domain distribution is also unstable before that. Once the model weights are quantized, we see a regularization effect and the performance becomes much more stable on the OOD data. We provide training plots encompassing different domains as target settings for the sake of completeness. This inclusion serves to illustrate that quantization genuinely enhances stability in the training process. On the left, "te lo___ooatio____ 100" is considered 10tiiiooonO00""""
4. Experiments
We evaluate our approach on diverse datasets from Do- mainbed and WILDS (Koh et al., 2021) Benchmark. All implementation, datasets, metric details and various ablation studies are provided in the Appendix.
4.1. Results
In this section, we demonstrate the superior performance of our proposed approach by comparing it to recent state-of- the-art DG methods. We also present some visual evidence for the better performance of our quantization approach. Furthermore, show how quantization not only enhances model generalization but also yields better performance on in-domain data.
target domain while "te_ vre_location_46 46" is used as the target domain for the plot on the right. These experiments evidence that model selection based on the in-domain validation set much more reliable when introducing quantization into training.
3.6. Ensembles of Quantization
For our ensemble creation, we train multiple models inde- pendently from initialization, using random seeds to ensure diversity and, incorporate quantization into the training pro-
4. I. COMPARISON WITH DG METHODS
Table I reports ofomain performances on five DG benchmarks and compares our proposed approaches prior
6

---

QT-DoG: Quantization-awan aware Training for Domain Generalization
Table Comparison with domain generalization methods. Performance benchmarking 5 datasets of the DomainBed benchmark. Highest accuracy is shown in bold, while second best is underlined. 'o not report confidence interval and ensembles do not have confidence interval because an ensemble uses all the models make prediction. Our proposed method is colored Gray. Average accuracies and standard emrrr are reported from three trials. For all the reported results, use the same training -dmaann validation protocol as (Gulrajani & Lopez-Paz, 2021). Models indicate the number of models trained, and Size represents the relative network size.
Algorithm
Models
Size
PACS
VLCC
Office
Terralnc
DomainNet
Avg.
ResNet-50
(25M Parameters, Pre-trained ImageNet)
ERM
1x
84.7 0.5
77.4 + 0.3 67.5 0.5 46.2 + 0.4 41.2 0.2
63.8
IRM
1x
1. 1.1
78.1 0.0 66.6 1.0 47.9 1 0.7
1.9
62.5
Group DRO
1x
84.1 = 0.4
77.2
06.9:00.3
47.0
0.3
33.7
61.8
Mixup
1x
84.3 + 0.5
77.7 0.4
69.0
48,9 0.8
3..6
63.9
MLDG
lx
0.6
77.1
77.1
0.4
6.1
4..1
4.
63.6
CORAL
1x
86.0
77.7.:0.5
6..6.1
4.
41.8
64.1
MMD
1x
85.0+
76.7 0.9
67.7
4..3+
39.4 0.8
63.6
Fish
1x
0.3 77.8
68.6
4..1
42.7 0.2
63.9
Fishr
1x
85.5 1 0.4
77.8
0.1
47.4 1.6
41.7
65.7
SWAD
1x
88.1 0.4
79.1
0.4
70.6+
50.0
46.5 0.2
66.9
MIRO
1x
0.4
79.0 + 0.0
50.4
1.1
44.3 0.2
65.9
CCFP
1x
86.6 =0.2
78.9 +0.3
68.9 #0.1
48.6 +0.4
41.2 0.0
64.8
ARM
1x
85.1
77.6
64.8
45.5
35.5
61.7
VREx
1x 84.9
78.3
66.4
46.4
33.6
61.9
RSC
lx 85.2
77.1
65.5
46.6
38.9
62.7
Mixstyle
1x
85.2
77.9
60.4
44.0
34.0
60.3
SagNet
1x 86.3
77.8
68.1
48.6
40.3
64.2
QT-DoG (ours)
0.22x
87.8
78.4 0.4
68.9
50.8 0.2
45.1
66.2
ERM Ens.
6x
87.6
78.5
70.8
49.2
47.7
66.8
DiWA
1x
89.0
78.6
72.8
$1.9
47.7
68.0
BoAt
6x
88.6
79.1
72.5
52,3
47.4
68.0
DART
4-6
4x 6x
78.5 + 0.7
87.3 + 0.5
70.1 + 0.2
48.7
0.8
45.8 + 0.0
66.1
EoQ (ours)
1.1x
89.3
79.5
72.3
53.2
47.9
68.4
works. These results demonstrate the superiority of EoQ across five DomainBed datasets, with an average improve- ment of 0.4% over the ttate-of-the-an art EoA while reducing the memory footprint by approximately 75%. Compared to DiWA, we significantly reduce the computational bur- den and memory requirements for training, achieving 12-fold reduction, DiWA requires training 60 models for diverse averaging. EoQ achieves the most significant gain (7% improvement) on Terralncognita (Beery et al., 2018), with nonetheless substantial gains of 3-5% w.r.t. ERM on PACS (Li et al., 2017) and DomainNet (Peng et al., 2019).
Table 2. Combination with other methods. Results of PACS and Terra Incognita datasets incorporating QT-DoG with CORAL and MixStyle. C represents the compression factor of the model.
Algorithm
PACS
TerraIne
CORAL
85.5 0.6
47.1 0.2
CORAL QT-DoG
86.9 1 0.2
50.6 0.3
4.6x
MixStyle
85.2 2 0.3
44.0 0.4
MixStyle + QT-DeG
86.8 + 0.3
47.7 0.2
4.6x
Moreover, quantization not only reduces the memory foot- print but also the latency of the model. For example, run- ning a ResNet-50 model on AMD EPYC7300 processor yields a latency of 34.28mm for full-precision and 21. 11.02ms for our INT8 quantized model.
The results also demonstrate that simply introducing quanti- zation into the ERM-based approach (Gulrajani & Lopez- Paz, 2021) surpasses yields comparable accuracy many existing works, although the size and computational budget of our quantization basodd approach is significantly lower than that of the other methods. For our results Table 1 and Figure 1, we employed 7-bit quantization on the network. Therefore, as shown in Figure 1, the model size is drasti- cally reduced, becoming more than times smaller than the other methods. Being smaller in memory footprint, our quantization-basee approach can utilize ensembling without increasing the memory storage and computational resources.
4.1.2. COMBINATIONS WITH OTHER METHODS
Since QT-DoG requires modifications to training proce- dures or model architectures, it is universally applicable and can seamlessly integrate with other DG methods. As shown in Table 2, we integrate QT-DoG with CORAL (Sun et al., 2016) and MixStyle (Zhou et al., 2021). Both CORAL and MixStyle demonstrate improved performance when com-
7

---

QT-DoG: Quantization-aware Training for Domain Generalization
Table Comparison between ERM and QT-DoG the Amazon and Camelyon datasets. We report the in in-domain and out-of-domall
accuracy with respective metrics as shown. C represents the compression factor of the model.
Dataset
Method
In-dist
Out-dist
C
Metric
Amazon
ERM
71.9 = 0.1
53.8 + 0.8
10th percentile acc
Amazon
QT-DoG
79.2 0.5
55.9 0.6
4.6x
10th percentile acc
Camelyon
ERM
93.2 = 5.2
70.3 1 6.4
Average acc
Camelyon
QT-DoG
96.4 = 2.1
78.4 + 2.2
4.6x
Average acc
Table 4. Model quantization with different quantization algo- rithms. We report the average target domain accuracy and the average souree domain accuracy across all domains in PACS.
Table S. Quantization a Vision Transformer Comparison
performance PACS and Terralnc datasets with and without QT-DoG quantization of ERM_ ViT (Sultana et al., 2022) with
DeiT-Small backbone.
Algorithm
Type
In-domain
Out-domain
Algorithm
PACS
TerraInc
Compression
No quant
96.6 + 0.2
84.7 + 0,5
OBC
PTQ
96.8 + 0.2
83.7 + 0.4
ERM_ VIT
84.3 - 0.2
43.2 4: 0.2
INQ LSQ
QAT
97.1 0.2
87.4 + 0.3
ERM-SD_ VIT
86.3 + 0.2
44.3 + 0.2
QAT
97.3 + 0.2
87.8 + 0.3
ERM VIT + QT-DoG
86.2 = 0.3
45.6 + 0.4
4.6x
bined with QT-DoG, reinforcing our findings that QAT aids in identifying flat minima, thereby enhancing DG.
Test Domain Accuracy [%6]
90
2.00%
+3.10%
+1.80%
+0.60%
4.1.3. DIFFERENT QUANTIZATION METHODS
80
In this section, we perform an ablation study by replacing LSQ (Esser et al., 2020) with other quantization algorithms. We use INQ (Zhou et al., 2017) as another quantization- aware training method but also perform quantization using OBC (Frantar et al., 2022), that uses a more popular post- training quantization (PTQ) approach to quantize a network. We perform this ablation study on the PACS dataset, and the results are shown in Table 4. All the experiments are performed with 7-bit quantization. We observe that, while
70
PACS (ERM Baseline)
Terra Incognito (ERM Baseline)
60
+4.00%
50
+2.50%
+2.30%
+1.60%
40
Full Precision
8-bit
7-bit
6-bit
5-bit
the QAT approaches tend to enhance generalization, the PTQ approach fails to do so. This is due to the fact that there is no training involved after the quantization step in PTQ. That is, with PTQ, we do not train the network with quantization noise to find a flatter minimum.
Bit Width
Figure 5. Bit precision analysis for efficient quantization. We show results on out-of-domain test accuracy with two different datasets, ie.. PACS and TerraIncognita. For each bit precision, we report the increase in the test domain accuracy averaged across all domains. The 7-bit quantized model exhibits the maximum increase for both datasets. We quantize the model at 2000 steps.
4.1.4. RESULTS ON WILDS DATASET
We performed experiments with 7 bit quantization on two datasets from the WILDS benchmark (Koh et al., 2021).
baseline ERM-ViT VIT to its quantized counterpart on the PACS and Terra Incognita datasets, demonstrating QT-DoG'ss 's ef- fectiveness across different architectures. The results clearly show that QT-DoG also improves the performance of vision transformers.
We utilized the same experimental settings outlined in the WILDS benchmark repository and incorporated quan- tization into the training process. The results presented in Table confirm our findings on Domainbed benchmark. We used the same BERT model (Sanh et al., 2019) as in WILDS (Koh et al., 2021). These results highlight that QT- DoG generalizes well across both architectural variations and input modalities, including text.
4.1.6. BIT PRECISION ANALYSIS
Here, we empirically analyze the effect of different bit- precisions for quantization on the generalization of the model. We perform experiments with four different bit levels and present an analysis in Figure 5 on the PACS (Li et al., 2017) and Terralncognita (Beery et al., 2018) datasets. We report the test domain accuracy averaged across all do- mains. For both datasets, 7-bit precision was found to the
4. 1.5.. GENERALITY WITH VISION TRANSFOR MER
In Table 5, we present the results of quantizing a vision trans- former (ERM- VIT, DeiT-small) (Sultana et al., 2022) for domain generalization. We compare the performance of the
8

---

QT-DoG: Quantization-aware Training for Domain Generalization
Table Performance comparison with CLiP-based methods. We report accuracy on DomainNet, Terralncognita, and Office datasets,
as well as the average performance (AVG). QT-DoG achieves competitive accuracy while offering substantial compression.
Algorithm
Backhone
DomainNet
TerraInc
Office
Avg.
ERM
CLIP
59.9 + 0.1
60.9 + 0.2
83.0 + 0.1
67.9
CLIPood
CLIP
63.5 = 0.1
60.5 1 0.4
87.0 1 0.1
70.3
QT-DoG
CLIP
63.1 = 0.2
61.9 1 0.3
86.7 1 0.2
70.6
4.6x
of the models with and without quantization. These visual- izations evidence that quantization focuses on better regions than ERM, and with a much larger receptive field. In certain cases, ERM does not even focus on the correct image region. It is quite evident that quantization pushes the model to learn more generalized patterns, leading to a model that less sensitive to the specific details of the training set.
ERM
QT-DoG
Cartoom
Sketch
Photo
5. Conclusion
Figure GradCAM visualization
ERM (Gulrajani & Lopez-Paz, 2021) and QT-DoG. We show results the PACS dataset (Li et al., 2017) and consider a different domain as test domain in each run, indicated by the different columns.
We introduced QT-DoG,, a novel generalization strategy based on neural network quantization. Our approach lever- ages the insight that QAT can find flatter minima in the loss landscape, serving as an effective regularization method to reduce overfitting and enhance the generalization capabili- ties. We empirically demonstrated, supported by analytical insights, that quantization not only enhances generalization but also helps stabilize the training process. Our exten- sive experiments across diverse datasets show that incorpo- rating quantization with an optimal bit-width significantly enhances domain generalization, yielding performance com- parable existing methods while reducing the model size, Additionally, proposed EoQ, powerful ensembling strategy that addresses the challenges of memory footprint and computational load by creating ensembles of quantized models. EoQ outperforms tfate-of-the-ar art methods while be- ing approximately four times smaller than its full coll-precision ensembling counterparts.
optimal bit precision to have the best out-of-domain general- ization while maintaining in-domain accuracy. Nonetheless, 8 bits and bits also show improvements, albeit smaller than with 7-bit quantization. These results evidence that, even with 6 times smaller model, quantization still yields better
performance without sacrificing the in-domain accuracy.
4.7.7. GENERALITY WITH CLIP-BASED METHODS
We compare QT-DoG with existing CLIP-based domain generalization methods using the ViT-B/16 backbone to maintain consistency in architecture. As shown in the table below, QT-DoG achieves higher average accuracy than both ERM (Gulrajani & Lopez-Paz, 2021) and CLIPood (Shu et al., 2023) across the NetainNet, Terralncognita, and Of- fice datasets. provides 0.3 percent improvement over the best baseline, CLIPood. Importantly, this performance gain comes with 4.6 times reduction model size, showing that QT-DoG improves both generalization and efficiency.
Acknowledgements
We thank Soumava Kumar Roy, Chen Zhao, Ahmad Jarrar Khan and Muhammad Zakwan for their help. This project has received funding from the European Union's 's Horizon 2020 research and innovation programme under the Marie Sklodowska Curie grant agreement No. 945363. Moreover, this work was funded in part by the Swiss National Science Foundation and the Swiss Innovation Agency (Innosuisse) via the BRIDGE Discovery grant No. 194729.
4.2. Visualizations
In Figure 6, we present some of the examples' from the PACS dataset and show GradCAM (Gildenblat & contrib- utors, 2021) results in the target domain. We perform four different experiments by considering different target do- main for each run, while utilizing the other domains for training. We use the output from the last convolutional layer
Impact Statement
In this paper, we presented our research work aiming to highlight the relation of model quantization for the domain
More examples are provied the appendix.
9

---

QT-DoG: Quantization-aware Training for Domain Generalization
generalization problem setup. In considering the societal im- pact of our approach, few concerns emerge, such as perfor- mance degradation in extreme-level quantization. Although the proposed approach provides competitive performance compared to previous tate-of-thhe-rt models on domain gen- cralization benchmarks, we note that performance slightly varies extreme quantization cases. Our future work is aimed at a thorough analysis of this aspect safety-critical applications, such medical imaging, autonomous driving. or surveillance.
Breiman, L. Bagging predictors. Machine learning, 24(2):
123-140, 1996.
Cai, Y., Yao, Z., Dong, Z., Gholami, A., Mahoney, M. W. and Keutzer, K. Zeroq: A novel zero shot quantization framework. In CVPR, 2020.
Cha, J., Chun, S., Lee, K., Cho, H.-.., Park, S., Lee, Y., and Park, S. SWAD: Domain generalization by seeking flat minima. In NeurlPS, 2021.
Chee, J., Cai, Y., Kuleshov, V., and De Sa, C. QulP: 2-Bit Quantization of Large Language Models With Guaran- tees. In NeurIPS, 2023.
References
Addepalli, S., Asokan, A. R., Sharma, L., and Babu, R. V. Leveraging Vision-Language Models for Improving Do- main Generalization in Image Classification. In CVPR, June 2024.
Chen, Z., Wang, W. Zhao, Z.,, Su, F., Men, A., and Meng, H. PracticalDG: Perturbation Distillation on Vision- Language Models for Hybrid Domain Generalization. In CVPR, 2024.
An, G. The effects of adding noise during backpropaga- tion training on a generalization performance. Neural Comput,, 1996.
Cheng, D., Xu, Z., Jiang, X., Wang, N., Li, D., and Gao, X. Disentangled Prompt Representation for Domain Gener- alization. In CVPR, June 2024.
Andriushchenko, M., Croce, F., Müller, M., Hein, M., and Flammarion, N. A Modern Look at the Relationship between Sharpness and Generalization. ICML, 2023.
Choshen, L., Venezian, E., Slonim, N., and Katz, Y. Fusing finetuned models for better pretraining. arXiv preprint, 2022.
Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez- Paz, D. Invariant risk minimization. arXiv preprint arXiv: 1907.02893, 2019.
Dai, and Van Gool, L.. Dark model adaptation: Semantic image segmentation from daytime to nighttime. In ITCS, 2018
Arpit, D., Wang, H., Zhou, Y., and Xiong, C. Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), NeurlPS, 2022.
Dinh, L., Pascanu, R., Bengio, S., and Bengio, Y. Sharp minima can generalize for deep nets. In ICMI., 2017.
Dong, Z., Yao, Z., Gholami, A., Mahoney, M., and Keutzer, K. HAWQ: Hessian AWare Quantization of Neural Net- works with Mixed-Precision. In ICCV, 2019.
Bahng, H., Chun, S., Yun, S., Choo, J., and Oh, S. J. Learn ing De-biased Representations with Biased Representa- tions. In ICML, 2020.
Dziugaite, G. K. and Roy, D. M. Computing nonvacuous generalization bounds for deep (stochastic) neural net- works with many more parameters than training data. In UAI, 2017.
Beery, S., Van Horn, G., and Perona, P. Recognition in terra incognita. In ECCV, 2018.
Benton, G., Maddox, W., Lotfi, S., and Wilson, G. G. Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling. In ICML, 2021.
Esser, S. K., McKinstry, J. L., Bablani, D., Appuswamy, R., and Modha, D. S. Learned Step Size Quantization. ICLR, 2020.
Bhalgat, Y., Lee, J., Nagel, M., Blankevoort, T., and Kwak, N. LSQ+: Improving low-bit quantization through learn able offsets and better initialization. In CVPR, 2020.
Fang, C., Xu, Y., and Rockmore, D. N. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In ICCV, 2013.
Blanchard, G.,
Lee, G., and Scott, C. Generalizing from several related classification tasks a new unlabeled sample. NeurIPS, 2011.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness- -ware Minimization for Efficiently Improving Generalization. In ICLR, 2021.
Boncelet, C. Chapter 7 Image Noise Models. In The Essential Guide to Image Processing. Academic Press, 2009.
Frantar, E., Singh, S. P., and Alistarh, D. Optimal Brain Compression: A Framework for Accurate Post-Truie Bost-Training Quantization and Pruning. NeurlPS, 2022.
10

---

QT-DoG: Quantizationawwrre Training for Domain Generalization
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. Domain-adversarial training of neural networks. JMLR, 2016.
Jain, S., Addepalli, S., Sahu, P. K., Dey, and Babu, R. V. DART: Aggreeftttggggggeeeeeeeeee Training Improves Generalization of Neural Networks. In CVPR, 2023.
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. In ICLR, 2020.
Garipov, T., Izmailov, P., Podoprikhin, D., Vetrov, D.P., and Wilson, A. G. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. In NeurlPS, 2018.
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In ICLR, 2017.
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wich- mann, F. A., and Brendel, W. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019.
Kingma, D. P. and Ba, J. Adam: Method for Stochastic Optimization. In ICLR, 2015.
Gildenblat, J. and contributors. PyTorch library for CAM methods. https: httg//gtthbbbbt/// com/ jacobgil/ setorch-gradee cam, 2021.
Koh, P. W. Sagawa, S., Marklund, H., Xie, S. M., Zhang,
M., Balsubramani, A., Hu, W., Yasunaga, M., Phillips, R. L.., Gao, I., Lee, T., David, E., Stavness, I., Guo, W., Earnshaw, B. A., Haque, I. S., Beery, S., Leskovec, J., Kundaje, A., Pierson, E., Levine, S., Finn, C.. and Liang, P. WILDS: A benchmark of in-the-wild distri- bution shifts. In International Conference on Machine Learning (ICML), 2021.
Goodfellow, I,, Bengio, Y., and Courville, A. Deep learning. MIT press. 2016.
Gray, R. and Neuhoff, D. Quantization. IEEE Transactions on Information Theory, 44(6):2325-288,, 1998. doi:
(0.1109/18.720441
Gulrajani, I. and Lopez-Paz, D. In Search of Lost Domain Generalization. In ICLR, 2021.
Krogh, A. and Vedelsby, J. Neural network ensembles, cross validation, and active learning. In NeurIPS, 1995.
Krueger, D., Caballero, E., Jacobsen, J.-H., Zhang, A., Bi- nas, J., Zhang, D., Le Priol, R., and Courville, A. Out-of- distribution generalization via risk extrapolation (rex). In ICML, 2021.
Gupta, V., Serrano, S. A., and DeCoste, D. Stochastic Weight Averaging in Parallel: Large-Batch Training That Generalizes Well. In ICLR, 2020.
Hansen, L. K. and Salamon, P. Neural network ensembles. TPAMI, 1990.
Kullback, S. and Leibler, R. A. On information and suffi- ciency. Annals of Mathematical Statistics, 1951.
He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition. In CVPR, 2016.
Lakshninarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurlPS, 2017.
Hinton, G., Vinyals, O., and Dean, J. Distilling the Knowl- edge in Neural Network. In NeurlPS, 2015.
Lee, Y., Yao, H., and Finn, C. Diversify and Disambiguate: Learning From Underspecified Data. ICML Workshop, 2022.
Hochreiter, S. and Schmidhuber, J. Simplifying neural nets by discovering flat minima. NeurIPS, 1994.
Li, C., Zhang, D., Huang, W., and Zhang, Cross contrast- ing feature perturbation for domain generalization. In ICCV, 2023.
Hochreiter, S. and Schmidhuber, J. Flat minima. Neural Computation, 1997.
Huang. Z., Wang, H. Xing, E. P. and Huang, D. Self- challenging improves cross-domain generalization. In ECCV, 2020.
Li, D., Yang. Y., Song, Y.Z.., and Hospedales, T. M. Deeper, broader and artier domain generalization. In ICCV, 2017.
Li*, M., Lin Y., Zhang*. Z., Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y., and Han, S. SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models. In ICLR, 2025.
Huang, Z., Zhou, A., Ling, Z., Cai, M., Wang, H., and Lee, J. A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance. In ICCV, 2023.
Li, Y., Gong, R., Tan, Z., Yang. Y., Hu, P., Zhang, Q., Yu, F., Wang, W., and Gu. S. BRECQ: Pushing the Limit Post-Training Quantization by Block Reconstruction. ICLR, 2021.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. Averaging weights leads to wider optima and better generalization. In UAI, 2018.

---

QT-DoG: Quantization-aware Training for Domain Generalization
Li, Z.,, Ren, K., Jiang, X., Li, B., Zhang, H., and Li, D. Domain Generalization using Pretrained Models without Fine-tuning, arXiv preprint, 2022.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G.,
Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning Transferable Visual Models From Natural Language Supervision. In ICML, 2021.
Lin, J., Tang, J., Tang, H., Yang. S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. AWQ: Activation-aware Weight Quantization for LLM Com- pression and Accelcration. In MLSys, 2024.
Ramachandran, A., Kundu, S., and Krishna, T. Clamp-vit: Contrastive data-free learning for adaptive post-training quantization of vits. In Leonardis, A., Ricci, E.,, Roth, S., Russakovsky, O., Sattler, T., and Varol, G. (eds.), ECCV, 2024.
Liu, F., Kim, M., Ren, Z., and Liu, X. Distilling CLIP with Dual Guidance for Learning Discriminative Human Body Shape Representation. In CVPR, June 2024.
Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., and simple baseline for bayesian uncertainty in deep learning. In NeurlPS, 2019.
Rame, A., Dancette, C., and Cord, M. Fishr: Invariant Gradient Variances for Out-of-distribution Generalization. In ICML, 2022a.
Wilson,
Matena, M. and Raffel, C. Merging Models with Fisher- weighted averaging. In NewrlPS, 2022.
Rame, A., Kirchmeyer, M., Rahier, T., Rakotomamonjy, A., Gallinari, P., and Cord, M. Diverse weight averaging for out-of-distribution generalization. In NeurIPS, 2022b.
Mesbah, Y. Ibrahim, Y. Y., and Khan, A. M. Domain Generalization Using Ensemble Learning, In ISWA, 2022.
Ramé, A., Ahuja, K., Zhang. J., Cord, M., Bottou, L., and Lopez-Paz, Model Ratatouille: Recyeling Diverse Models for Out-of-Distribution Generalization. In /CML, 2023.
Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bring- mann. O., Ecker, A. S., Bethge, M., and Brendel, W. Benchmarking robustness in object detection: Au- tonomous driving when winter is coming, NeurIPS, 2019.
Rissanen, J. Modeling by shortest data description. Auto- matica, 1978.
Moayeri, M., Rezaei, K., Sanjabi, M., and Feizi, S. Text- To- Concept (and Back) via Cross-Model Alignment. In ICML, 2023.
Russakovsky, O., Deng, Su, H., Krause, J., Satheesh, S., Ma. S., Huang, Z., Karpathy, A., Khosla, A., Bern- stein, M., al. Imagenet large scale visual recognition challenge. IJCV, 2015.
Muandet, K., Balduzzi, D., and Schölkopf, B. Domain generalization via invariant feature representation. ICML,, 2013.
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Distributionally Robust Neural Networks. In ICLR, 2020.
Mueller, M., Vlaar, J., Rolnick, D., and Hein, M. Normal- ization layers are all that sharpness-awar minimization needs. In NeurlPS, 2023.
Sanh, V., Debut, L., Chaumond, J., and Wolf, T. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. NeurIPS Workshop. 2019.
Murray, A. and Edwards, P. Synaptic Weight Noise During Generalization
MLP Learning Enhances and Learning Trajectory. In NeurlPS, 1992.
Shang, Y., Liu, G., Kompella, R., and Yan, Y. Enhanc- ing Post- DostnTraining Quantization Calibration Through Con- trastive Learning. 2024.
Nagel, M., van Baalen, M., Blankevoort, T., and Welling, M. Data-Free Quantization Through Weight Equalization and Bias Correction. ICCV, 2019.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K. Z., Gao, P., Qiao, Y., and Luo, P. OmniQuant: Omnidirectionally Calibrated Quantization for Large Lan- guage Models. ICLR, 2024.
Nagel, M., Amjad, R. A., van Baalen, M., Louizos, C., and Blanevoort, T. Up Down? Adaptive Rounding for Post-Training Quantization. ICML, 2020.
Pagliardini, M., Jaggi, M., Fleuret, F., and Karimireddy, S. Agree to Disagree: Diversity through Disagreement for Better Transferability. In ICLR, 2023.
Shin, J., So, J., Park, S., Kang, S., Yoo, S., and Park, E. NIPQ: Noise Proxy- Froxy-Based Integrated Pseudo- Quantization. In CVPR, 2023.
Peng, X., Bai, Q. Xia, X., Huang, Z., Saenko, K., and Wang, B. Moment matching for multi multi-source domain adaptation. In ICCV, 2019.
Shu, Y., Guo, X., Wu, J., Wang, X., Wang, J., and Long, M. CLIPood: Generalizing CLIP to Distrr--Disttiuutiinns In ICMI., 2023.
12

---

QT-DoG: Quantization-aware Training for Domain Generalization
Sultana, M., Naseer, M., Khan, M. H., Khan, S., and Khan, F. S. Self-Distilled Vision Transformer for Domain Gen- eralization. In ACCV, 2022.
Yu, X., Yoo, S., and Lin, Y. CLIPCEIL: Domain General- ization through CLIP via Channel réfinement and Image- text aLignment. In Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., and Zhang, C. (eds.), NeurIPS, 2024.
Sun, B. and Saenko, K. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, 2016.
Zhang, H. R., Li, D., and Ju, H. Noise stability optimization for finding flat minima: A hessian-based regularization approach. Transactions on Machine Learning Research, 2024.
Sun, B., Feng, J., and Saenko, K. Return of Frustratingly Easy Domain Adaptation. In AAAI, 2016.
Thopalli, K., Katoch, S., Thiagarajan, J. J., Turaga, P. K., and Spanias, A. Multi-Domain Ensembles for Domain Generalization, In NeurlPS Workshop, 2021.
Zhang, T. and Shrivastava, A. LeanQuant: Accurate and Scalable Large Language Model Quantization with Loss- error-aware Grid. In ICLR, 2025.
Venkateswara, H., Eusebio, J., Chakraborty, S., and Pan- chanathan, S. Deep hashing network for unsupervised domain adaptation. In CVPR, 2017.
Zhang, X., Xu, R., Yu, H., Dong, Y., Tian, P., and Cui, P. Flatness-Awarc Minimization for Domain Generalization. In ICCV, October 2023.
Wortsman, M., Horton, M., Guestrin, C., Farhadi, A., and Rastegari, M. Learning Neural Network Subspaces. ICML, 2021.
Zhao, R., Hu, Y., Dotzel, J., De Sa, C., and Zhang, Im- proving Neural Network Quantization without Retraining using Outlier Channel Splitting. ICML, June 2019.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontijo-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., and Schmidt, L. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022a.
Zhou, A., Yao, A., Guo, Y., Xu, I.,, and Chen, Y. Incremen- tal Network Quantization: Towards Lossless CNNs with now-Precision Weights. ICLR, 2017.
Zhou, K., Yang, Y., Qiao, Y., and Xiang, T. Domain Gener- alization with MixStyle. In ICLR, 2021.
Wortsman, M., Ilharco, G., Kim, J. W., Li, M., Kornblith, S., Roelofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Robust fine-tuning of zero-shot models. In CVPR, 2022b.
Zhou, K., Liu, Z., Qiao, Y., Xiang, T., and Loy, C. C. Do- main Generalization: A Survey, TPAMI, 2022.
Namkoong, H.,
Xiao, K. Y., Engstrom, L., Ilyas, A., and Madry, A. Noise or Signal: The Role of Image Backgrounds in Object Recognition. In ICLR, 2020.
Yalniz, Z., Jégou, H., Chen, K., Paluri, M., and Maha- jan, D. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv: 1XX5:009500004:: 2019.
Yamamoto, K. Learnalee Companding Quantization for Accurate Low-bit Neural Networks. In CVPR, 2021.
Yang, J., Tang, C., Yu, C., and Lv, J. GWQ: Group-Wise Quantization Framework for Neural Networks. In Asian Conference on Machine Learning. 2023.
Yao, Z., Dong, Z., Zheng, Z., Gholami, A., Yu, J., Tan, E., Wang, L., Huang, Q. Wang, Y., Mahoney, M. W., and Keutzer, K. HAWQV3: Dyadic Neural Network Quantization. In ICML, 2020.
Ye, N., Li, K., Hong, L., Bai, H., Chen, Y., Zhou, F., and Li, Z. OoD-Bench: Benchmarking and Understanding Out- of-Distribution Generalization Datasets and Algorithms. In CVPR, 2022.
13

---

QT-DoG: Quantization-aware Training for Domain Generalization
A. Datasets and Metrics
We demonstrate the effectiveness of our proposed method on diverse classification datasets used for evaluating multi-source Domain Generalization:
PACS (Li al., 2017) a object classification challenge encompassing four domains, with total of 9,991 samples. serves to validate our method in smaller-scale settings. VLCS (Fang et 2013) poses a 5 object classification problem across four domains. With 10,729 samples, VLCS provides a good benchmark for close Out-of-Distribution (OOD), featuring subtle distribution shifts simulating real-life scenarios. OfficeHome (Venkateswara et al., 2017) comprises a total of 1 15,588 samples. It presents a 65-way classification challenge featuring everyday objects across four domains. Terralncognita (Beery et al., 2018) addresses a 10 object classification challenge of animals captured in wildlife cameras, with four domains representing different locations. The dataset contains 24,788 samples, illustrating a realistic use-case where generalization is crucial. DomainNet (Peng et al., 2019) provides a 345 object classification problem spanning six domains. With 586,575 samples, it is one of the largest datasets. Furthermore, present results WILDS benchmark datasets to demonstrate the effectiveness of our approach in real-world applications.
We report out-of-domain accuracies for each domain and their average, i.e., a model is trained and validated on training domains and evaluated on the unseen target domain. Each out-of-domain performance is an average of three different runs with different validaaiintion splits for the quantized models. We then combine the predictions of the different quantized models for our EoQ results.
B. Implementation Details
We use the same training procedure as DomainBed (Gulrajani & Lopez-Paz, 2021), incorporating additional components from quantization. Specifically, we adopt the default hyperparameters from DomainBed (Gulrajani & Lopez-Paz, 2021 I), including a batch size of 32 (per-domain). We employ a ResNet-50 (He et al., 2016) pre-trained on ImageNet (Russakovsky et al., 2015) as initial model and use a learning rate of 5e-5 along with the Adam optimizer, and no weight decay. Following SWAD(Cha et al., 2021), the models are trained for 15,000 steps on DomainNet and 5.000 steps on the other datasets. In the training process, we keep a specific domain the target domain, while the remaining domains are utilized source domains. During this training phase, 20% of the samples are used for validation and model selection. We validate the model every 300 steps using held-out data from the source domains, and assess the final performance on the excluded domain (target).
We use LSQ (Esser et al., 2020) and INQ (Zhou et al., 2017) for model quantization, with the same configuration as existing quantization methods (Esser et al., 2020; Bhalgat et al., 2020; Dong et al., 2019; Yao et al., 2020; Zhou et al., 2017), where all layers are quantized to lower bit precision except the last one. We quantize the models at 8,000 steps for DomainNet and 2,000 steps for the other datasets. Moreover, each channel in a layer has different scaling factor
C. Discussion and Limitations
Despite showing success and surpassing the tfate-of-the-e art methods in terms of performance, EoQ also has some limitations. First, it requires training multiple models like Rame et al. (2022b); Arpit et al. (2022), to create diversity and form an ensemble. This ensemble creation increases the training computational load. Nevertheless, our quantized ensembling models much smaller in size.
Another limitation of this work is the challenge of determining the optimal bit precision for achieving the best performance in OOD generalization. In our experiments on the DomainBed benchmark, we identified 7 bits as the optimal precision. However, this may not hold true for other datasets. A potential future direction is to utilize a small number of target images to identify the optimal bit precision, which would significantly reduce the computational overhead associated with this process.
Lastly, given our utilization of uniform quantization strategy, it would be interesting to investigate whether specific layers can be more effectively exploited than others through mixed-precision techniques to have even better domain generalization performance.
14

---

QT-DoG: Quantization-aware Training for Domain Generalization
D. Per-Domain Performance Improvement
also report per-domain performance improvement for PACS (Li al., 2017) and Terra Incognito (Beery et al., 2018) dataset. We choose the best model based on the validation and report the results and The results with quantization correspond to 7 bit-precision and we perform quantization after 2000 steps. Table and show that EoQ is consistently better than the current state-of-the-an methods across domains for different datasets.
Algorithm
Art
Cartoon
Painting
Sketch
Avg.
ERM (our runs)
89.8
79.7
96.8
72.5
84.7
SWAD
89.3
83.4
97.3
82.5
88.1
EoA
90.5
83.4
98.0
82.5
88.6
DiWA
90.6
83.4
98.2
83.8
89.0
QT-DoG
89.1
82.4
96.9
82.3
87.8
EoQ
90.7
83.7
98.2
84.8
89.3
Table 7. Per-Domain Accuracy Comparison for PACS. We report the accuracy for each domain of the PACS dataset along with the average across domains. Our proposed quantization shaded Gray.
Algorithm
L100
L38
L43
L46
Avg.
ERM (our runs)
58.2
38.3
57.1
35.1
47.2
SWAD
55,4
44.9
59.7
39.9
50.0
DiWA
57.2
50.1
60.3
39.8
51.9
EoA
57.8
46.5
61.3
43.5
52.3
QT-DoG
60.2
46.4
55.2
41.4
50.8
EoQ
61.8
48.2
59.2
43.7
53.2
Table 8, Per-Domain Accuracy Comparison for Terra Incognito. We report the accuracy for each domain of the Terra Incognito dataset along with the average across all domains, Our proposed quantization is shaded Gray.
E. Bit Precision Analysis Extended
In contrast to main manuscript, Table 9 provides all the results in tabular form. We show how quantization outperforms the vanilla ERM approach. This shows the superior performance of quantization over ERM despite being more than 6 times smaller in the case of bit-precision.
PACS
TerraInc
Algorithm
Compression
In-domain
Out-domain
In-domain
Out-domain
ERM (our runs)
96.9 + 0.1
84.7 + 0.5
91.7 + 0.2
47.2 + 0.4
QT-DoG(8)
4x
97.0 + 0.1
85.0 = 0.1
90.9 + 0.2
49.1 0.1
QT-Doo(77
4.6x
97.3 0.2
87.8 0.3
92.3 + 0.2
50.8 0.2
(6-DoG(6)
5.3x
97.1 + 0.1
86.5 + 0.1
91.1 + 0.0
49.0 + 0.3
QT-DoG(5)
6.4x
97.0 + 0.1
85.3 + 0.4
91.0 + 0.1
48.4 + 0.2
Table Model quantization with different bit-precisions vs vanilla ERM. We report the average target domain accuracy as well as the average source domain accuracy across all domains for the PACS (Li et al., 2017) and Terralncognita (Beery et al., 2018) datasets. Quantization not only enhances the generalization ability but also retains the source domain performance. QT-DoG(x) indicates a model quantized with bit-precision.
However, as shown in Table 10, decreasing bit- bit-precision through quantization does not always improve performance above the baseline; after a point, there is a tradeoff between compression and generalization. Specifically, our experiments with 4-bit precision and lower did not yield satisfactory results. Finding the sweet spot for balancing speed and performance can
15

---

QT-DoG: Quantization-awars Training for Domain Generalization
be interesting research direction. Our results evidence that there exist configurations that can improve both speed and performance.
Algorithm
Bit-Precision
PACS
ERM
32
84.7 0.5
7
87.8 + 0.3
6
86.5 + 0.1
5
85.3 + 0.4
QT-DoG
4
84.3 + 0.3
3
83.3 + 0.4
82.8 + 0.2
Table 10. Effect of aggressive quantization. Performance comparison between ERM and QT-DoG with varying hit-precision PACS.
F. Experiments with larger pre-training datasets
We also show experimental results with ResNeXt-50-32x4 in Table 11. Note that both ResNet-50 and ResNeXt-50-32x4d have 25M parameters. However, ResNeXt-50-32x4d is pre-trained larger dataset i.e Instagram IB images( Yalniz et al., 2019). It is evident from Table 11 that incorporating quantization into training consistenlty improve accuracy when network is pre-trained a larger dataset. Furthermore, EoQ again showed superior performance in comparison to other methods across five DomainBed datasets.
Algorithm
M
s
PACS
VLCC
Office
Terralne
DomainNet
Avg.
ResNeXt-50
32x4d (25M Parameters, Pre-trained 1B Images)
ERM
1x
88.7 + 0.3 79.0 + 0.1 70.9 = 0.5 51.4 + 1.2 48.1 = 0,2
67.7
SMA
1x
92.7 + 0.3 79.7 + 0.3 78.6 = 0.1 53.3 + 0.1 53.5 + 0.1
71.6
QT-DoG (ours)
1
1x
92.9 + 0.3 79.2 + 0.4 78.9 = 0.3 54.1 + 0.2 53.9 + 0.2
71.8
ERM Ens.
6
6x
91.2
80.3
77.8
53.5
52.8
71.1
BoAt
6
6x
93.2
80.4
80.2
55.2
54.6
72.7
EoQt (ours)
5
1.1x
93.5
80.3
80.3
55.6
54.8
72.9
Table 11. Comparison with other methods for ResNeXt-50, Performance benchmarking on datasets the DomainBed benehmark. Highest accuracy shown in bold, while second best is underlined. Ensembles do not have confidence interval because ensemble uses all the models to make a prediction. Our proposed method is colored in Gray. Average accuracies and standard errors are reported from three trials. For all the reported results, corresponds to the number of models trained during training and S corresponds to the relative network size.
the training-doman validation protocol (Gulrajani & Lopez-Paz. 202.). M
G. In-domain Performance Improvement using Quantization
We further study the in-domain test accuracy our quantization approach without ensembling on PACS and Terralncognita datasets. (Cha et al., 2021), we split the in-domain datasets into training (60%), validation (20%), and test (20%) sets. We choose the best model based on the validation set and report the results on the test set Table 12. The results with quantization correspond
bit- titeiiecision.
QT-DoG also enhances the in-domain performance. The regularization effect introduced by quantization prevents the model from overfitting to edge cases and pushes it to learn more meaningful and generalizable features, which we also demonstrate in Section 4.2. As the training data consists various domains and the quantization limits the range of weight values, discourages the model from becoming overly complex and overfitting to the noise in the training data. Therefore, the model more robust to minor input fluctuations.
16

---

QT-DoG: Quantizationaaware Training for Domain Generalization
Method
PACS
TerraIne
Compression
ERM
96.6 = 0.2
90.1 + 0.2
SAM
97.3 = 0.1
90.8 = 0.1
SWA
97.1 07.10.11
90.7 = 0.1
SMA
96.8 0.2
90.7 - 0.4
SWAD
97.7 = 0.2
90.8 + 0.3
QT-DoG
97.3 = 0.2
91.1 + 0.2
~4466x
Table 12. Comparison between generalization methods on PACS and TerraIne for IID settings. We report the accuracy averaged across all domains. Our proposed approach is shaded in Gray. Highest accuracy is shown in bold, while second best is underlined.
H. Ablation on Layerwise and Channelwise scale
We conducted an ablation study where we set s at the layer level, rather than on a per-channel basis. We see that Channelwise can lead to 1.5% accuracy as compared to layerwise s. The results of this experiment on the PACS dataset with 7 bit quantization are shown below:
Scale
OOD Accuracy
No quantization
84.7 + 0.5
Layerwise
86.3 = 0.4
Channelwise
87.8 + 0.3
Table 13. OOD Accuracy with channelwise layerwise Scaling factor for quantization.
I. Ablation on Quantization Steps
We conducted an ablation study on the PACS dataset to identify the optimal number steps after which quantization should be applied. We perform 7-bit quantization and the results are summarized below:
Quantization Step
OOD Accuracy
No quantization
84.7 0.5
1000
86.2 + 0.4
2000
87.8 + 0.3
3000
86.9 + 0.4
4000
85.1 + 0.3
Table 14. OOD Accuracy across different quantization steps.
J. Adding Uniform Noise to weights
Quantization noise and uniform weight noise share similarities in that both introduce perturbations to the model's parameters. However, guantization noise specifically arises from the discretization of the weights, which can lead to more structured form of regularization due to the rounding or truncation during the quantization process. In contrast, uniform weight noise typically adds random perturbations with a uniform distribution, which may not exhibit the same structured regularization properties.
In Table 15, provide the results of our ablation study the PACS dataset with uniform noise with different minimum and maximum value:
17

---

QT-DoG: Quantization-aware Training for Domain Generalization
Noise
OOD Accuracy
No noise
84.7 + 0.5
Uniform(-0.0001, 0.0001)
82.9 # 0.6
Uniform(-0.00006, 0.00005)
83.8 0.5
Uniform(-0.00001, 0.00001)
85.1 + 0.4
Uniform(-0.000005, 0.000005)
85.6 # 0.3
Table 15. OOD Accuracy under different noise levels
K. Visualization
More GradCAM Results
In Figure 9, 10, 11, 12, we present some of the examples from the Terra dataset and show GradCAM (Gildenblat & contributors, 2021) results on the target domain. We the output from the last convolutional layer of the models with and without quantization for GradCAM. Similar considering a different target domain for each run, while utilizing the other domains for training. Both models trained with the similar settings as (Gulrajani & Lopez-Paz, 2021). For quantization method, we quantized the model after 2000 iteration and employ 7 bit-precision as it provides the best out-of-domain performance. Moreover, we present some more examples for PACS dataset in Figure 7.
experiments PACS dataset, perform four different experiments by
These visualizations further proves that quantization pushes the model to be less sensitive to the specific details of the training
ERM
QT-DoG
ERM
QT-DoG
ERM
QT-DoG
Art
A
RAAA
Cartoon
Sketch
Photo
Figure 7. GradCAM visualization for ERM (Gulrajani & Lopez-Paz, 2021) and QT-DoG. We show results on the PACS datasct (Li al., 2017) and consider a different domain test domain in each run, indicated by the different rows in the figure.
18

---

QT-DoG: Quantization-wware Training for Domain Generalization
L. Quantization Noise follows Uniform Distribution
a well established fact in the literature that the distribution quantization error can be modeled as uniform distri- bution (Boncelet, 2009: Gray Neuhoff, 1998). For uniform data distribution, only needs to hold within bounds of quantization bins as in this case half of the samples will be floored while the other half will be ceiled. As the the distribution of data is uniform with flat pdf, the error introduced by the rounding operation will also have the same distribution.
In many scenarios, however, the distribution of network weights is not uniform. Therefore, we extend this argument to more general data distributions. If the data follows a symmetric distribution with a reasonably smooth pdf, we can approximate the pdf of this distribution by a piecewise linear function, interpolating between the boundaries of the quantization bins. This approximation becomes more accurate as we increase the bit-width, which in turn increases the number of quantization bins. Although these linear pdfs within the quantization bins are not flat, the symmetry of the distribution ensures that there exists another bin with similar but negatively sloped pdf. Thus, when considered whole, the quantization error becomes uniform, ie é U [- 2. 2.
In order to model the quantization error as uniform error, we need to have symmetric weight distribution. Deep neural networks are overparameterized models that can have multiple optimal solutions. To guide the models toward specific distributions, weight priors are typically applied in the form of regularizers. These priors lead to trained weights that correspond to the specified distribution. L.1 regularization encourages sparsity and promotes a Laplace distribution for the weights, which symmetric and has sharper peaks around zero. On the other hand, L2 regularization promotes Gaussian distribution, which is also symmetric but follows smooth bell-shaped curve with heavier tails compared to the Laplace distribution.
These regularization techniques result in weight distributions that are symmetric and smooth, satisfying the necessary condition for quantization error to be modeled as a uniform distribution, where the quantization noise/error exhibits a very low KL divergence with respect to a uniform distribution.
We plot various layers' weight distributions along with the quantization noise in Figure 8 and demonstrate that weights generally exhibit symmetric and smooth distribution. Additionally, the quantization noise/error shows very low Kullback- Leibler (KL) divergence (Kullback & Leibler, 1951) with respect to uniform distribution having the same minimum and maximum values. To assess symmetry, we employ reverse-order and forward-order medians, ensuring a balanced representation of the weight distribution. The reverse-order median captures the statistical properties of the lower half, while the forward-order median does the same for the upper half, providing a more robust symmetry evaluation.
M. Reproducibility
To guarantee reproducibility, we will provide the source code publicly along with the details of the environments and dependencies. We will also provide instructions to reproduce the main results of Table I in the main paper. Furthermore, we will also share instructions and code to plot the loss surfaces and GradCAM results.
Every experiment in our work was executed on a single NVIDIA A100, Python 3.8. 16, PyTorch 1.10.0, Torchvision 0.1 and CUDA 12.1.
19

---

QT-DoG: Quantization-wware Training for Domain Generalization
liyer_...orv3.weight Distribution
ayyer..l.cmv/ weight Quantizttio error
@yver....oov..wepppp Symmetry
0.0
2000
0.4
600
0.0
400
-0.4
20
200
-0.0
-0.4
-0.2
0.
-0.001
-0.002 -0 00a
0,000
0.002
0.002
0.001
-0.2
0.0
0.8
sakue
Reversed Mediln
layer2.2.comw weightt Distribution
layer2.2 weight Ouantization Error
2.yer Symmetry
8000
230
0.0
3003
2500
Provency
200
180
0.0
302
2000
-0.6
2.0
-044
-0.2
0/8
0.2
0.4
0.6
-0.000
-0 0c0
0.000 Value
0.001
0,007
-30
-04
-0.7
0.0
62
0.5
MB
Moversed Median
layerd.3.comm weight
layend wewgh..wwwwww....cco...........e Symmetry
06-
34000
Medban Chtleed
-2000
600
8000
0000
4000
2:0
Median
lyyer3.5.dm ayye33.5.oovv3 conv3. weight Distribution
layer3.5.comv. convo. we igght antiiaation Error
layer335.comv3 weight: Symmetry
30000
020
0.2
8000
0.0
430
4000
200
2000
-0.4
-0.2
2:
0.2
04
0.00000 00180 20100
www.boocc........ooooooo www.. 0oo1t.000 00150.0000
-0.4
-0.
0.0
02
0.4
Sarko
Valux
Soruseed
Median
Figure Distribution of Quantization Noise and Weights plot weights(left), quantization noise(middle) and symmetry(rightt various layers in ResNet-50 model. We found KL-divergence to be [Top [0.0061, 0,004, 00069, 0.00054) Bottom] between quantization noise uniform distribution with same minimum and maximum value.
20

---

QT-DoG: Quantization-awar Training for Domain Generalization
Figure 9. Visualization of GradCAM results the Terra Incognito dataset with L.38 as test domain. We show original image, GradCAM with ERM (Gulrajani& & Lopez-Paz, 2021) and GradCAM with QT-DoG [Left to Right].
21

---

QT-DoG: Buantization-aware Training for Domain Generalization
Figure 10. Visualization of GradCAM results the Terra Incognito dataset with L46 test domain. We show original image. GradCAM with ERM (Gulrajani & Lopez-Paz, 2021) and GradCAM with QT-DoG [Left to Right].
22

---

QT-DoG: Quantization-aware Training for Domain Generalization
Figure 11. Visualization of GradCAM results the Terra Incognito dataset with L43 test domain. We show original image, GradCAM with ERM (Gulrajani & Lopez-Paz, 2021) and GradCAM with QT-DoG [Left to Right].
23

---

QT-DoG: auaatization-aware Training for Domain Generalization
Figure 12. Visualization of GradCAM results on the Terra Incognito dataset with L.000 as test domain. We show original image, GradCAM with ERM (Gulrajani & Lopez-Paz, 2021) and GradCAM with QT-DoG [Left to Right].
24


Reviews:
[{"review_id": "sDlEwqSAuV", "reviewer": "Reviewer_Hjrn", "summary": "This paper proposes QT-DoG (Quantization-aware Training for Domain Generalization), which introduces weight quantization as an implicit regularizer by injecting noise during training, guiding the optimization toward flatter minima in the loss landscape to enhance generalization on unseen target domains. Quantization not only mitigates overfitting to source domains but also compresses model size, enabling efficient ensembling of multiple quantized models (EoQ). Experiments demonstrate that QT-DoG outperforms state-of-the-art methods on benchmarks (e.g., PACS, TerraIncognita) with a 0.4% average accuracy improvement and 75% model size reduction. Quantization stabilizes training dynamics, reducing performance fluctuations on out-of-distribution data. The ensemble of quantized models (EoQ) achieves superior accuracy while maintaining computational efficiency, surpassing traditional full-precision ensemble approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful and constructive feedback. We greatly appreciate your recognition of our key contributions, including the theoretical insights connecting quantization and flat minima,  and the practical benefits of model compression. We address your major questions and concerns below.\n\n**Smoother transitions and proper introduction of abbreviations:**\n\nIn the revised version, we will improve the overall clarity and readability of the paper by ensuring smoother transitions and clearer exposition of key concepts. Specifically,  at Line 213, we will add \"Here, we argue that quantization inherently induces noise, which aids in finding flatter minima. To support this, we use a second-order Taylor series expansion to analyze how quantization-induced perturbations affect the curvature of the loss landscape, leading to flatter minima.\". We will also make sure that all abbreviations are properly defined on their first appearance.\n \n**Comparison with recent CLIP-based approaches:**\n\nWe thank the reviewer for the suggestion to include CLIP-based approaches to strengthen the evaluation and will do so in the revised version based. Specifically, we have now compared QT-DoG against CLIPooD and ERM with a CLIP backbone. For fairness, we have adopted the same architecture (ViT-B/16) as used in CLIPooD. The results evidence that QT-DoG continues to improve domain generalization performance even when applied to CLIP-based models, yielding a 0.3 improvement in the average accuracy while compressing the model by 4.6 times, as shown below.\n\n| Algorithm  | Backbone | DomainNet        | TerraInc    | Office | AVG |  Compression |\n|-----------|----------|------------|------------|--------|--------|-------------|\n| ERM       | CLIP     | 59.9 ± 0.1 | 60.9 ± 0.2 | 83.0 ± 0.1      | 67.9      | None           |\n| CLIPood   | CLIP     | 63.5 ± 0.1 | 60.5 ± 0.4 | 87.0 ± 0.1     | 70.3      | None           |\n| QT-DoG    | CLIP     | 63.1 ± 0.2 | 61.9 ± 0.3 | 86.7 ± 0.2      | **70.6**      | **4.6x**          |", "claims_and_evidence": "This paper introduces QT-DoG, a domain generalization method leveraging quantization-aware training. By injecting weight quantization noise as implicit regularization during optimization, the approach guides models toward flatter minima in the loss landscape, thereby enhancing generalization capabilities on unseen target domains. The quantization mechanism not only mitigates source domain overfitting but also achieves substantial model compression, enabling efficient ensembling of lightweight models. Experimental results demonstrate that QT-DoG outperforms state-of-the-art methods by 0.4% average accuracy on benchmark datasets including PACS and TerraIncognita, while reducing model size to one-fourth of original dimensions. The quantization process also stabilizes training dynamics, significantly decreasing performance fluctuations on out-of-distribution data. The proposed ensemble strategy EoQ, built upon quantized models, surpasses traditional full-precision ensemble methods in accuracy while maintaining the computational efficiency of a single model.", "methods_and_evaluation_criteria": "The proposed method leverages quantization-aware training (QAT) to inject structured noise into model weights as an implicit regularization mechanism, effectively guiding the optimization toward flatter regions of the loss landscape and significantly enhancing generalization to unseen domains. By suppressing source domain overfitting and enabling lightweight multi-model ensembles (EoQ) through inherent model compression, the approach achieves performance breakthroughs via ensemble diversity while maintaining single-model inference efficiency. Experimental results demonstrate 0.4%-7% accuracy improvements across cross-domain benchmarks, 78% model size reduction, and superior deployment efficiency in resource-constrained scenarios, offering a practical solution for real-world applications like edge computing that demand both generalization capability and operational efficiency.", "theoretical_claims": "The authors theoretically establish, for the first time, the intrinsic connection between weight quantization and flat minima in loss landscapes. They formulate quantization noise as an implicit regularization mechanism through second-order Taylor expansion analysis, demonstrating how uniform perturbations interact with Hessian curvature to drive optimization toward flatter regions. Furthermore, they bridge low-bit quantization with model complexity reduction via Rissanen's minimum description length principle and Hochreiter's flat minimum theory, providing an information-theoretic foundation for improved generalization. This work redefines quantization—traditionally a compression tool—as a novel theoretical framework for domain generalization, fundamentally expanding its conceptual boundaries in machine learning theory.", "experimental_designs_or_analyses": "The authors theoretically establish, for the first time, the intrinsic connection between weight quantization and flat minima in loss landscapes. They formulate quantization noise as an implicit regularization mechanism through second-order Taylor expansion analysis, demonstrating how uniform perturbations interact with Hessian curvature to drive optimization toward flatter regions. Furthermore, they bridge low-bit quantization with model complexity reduction via Rissanen's minimum description length principle and Hochreiter's flat minimum theory, providing an information-theoretic foundation for improved generalization. This work redefines quantization—traditionally a compression tool—as a novel theoretical framework for domain generalization, fundamentally expanding its conceptual boundaries in machine learning theory.", "relation_to_broader_scientific_literature": "This work reshapes broader scientific research by establishing a novel link between weight quantization and flat minima in loss landscapes, offering fresh theoretical and practical insights for domain generalization (DG) and model optimization. It challenges the conventional perception of quantization as merely a compression tool, demonstrating that low-precision training inherently serves as a potent regularization mechanism—a finding likely to inspire exploration of synergies between other model compression techniques (e.g., pruning, distillation) and generalization capabilities. The innovative concept of efficient quantized model ensembles opens new avenues for robust AI deployment in resource-constrained scenarios, prompting cross-community rethinking of efficiency-generalization tradeoffs in fields like edge computing and federated learning. The methodological framework could extend to other distribution shift scenarios (e.g., domain adaptation, continual learning), advancing the development of more universal and lightweight adaptive systems.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths:\n\nThe INSIGHT provided by the authors is helpful and will inspire many subfields of research with practical applications.\n\nQuantization drastically reduces model size (7-bit models compressed to 22% of original) while enabling lightweight ensembles (EoQ) that achieve state-of-the-art performance with computational costs comparable to single models, addressing the high resource demands of traditional ensembling.\n\n Quantization noise effectively suppresses source-domain overfitting, with experiments showing reduced fluctuations in target-domain accuracy and more reliable model selection.\n\nWeakness:\n\nThe writing in this paper needs to be improved, such as how to transition the proof of the correlation between quantization and noise, and abbreviations should be provided in full when first given.\n\n Absence of comparisons with recent DG methods (e.g., post-2023 CLIP-based approaches) or advanced quantization techniques (e.g., QuIP for LLMs), potentially weakening relevance to current SOTA.", "other_comments_or_suggestions": "no", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ig3psYtTma", "reviewer": "Reviewer_zEDU", "summary": "The paper introduces QT-DoG, a quantization-aware training (QAT) method for domain generalization (DG), and is the first (it says) to demonstrate that QAT, traditionally used for model compression, can serve as an implicit regularizer, with quantization noise enhancing generalization. Theoretical and empirical analyses show that QAT promotes flatter minima in the loss landscape and stabilizes model behavior on out-of-distribution (OOD) data. Unlike traditional DG methods that often increase model size or computational cost, QT-DoG improves generalization and significantly reduces model size, enabling efficient deployment in real-world applications. Its ensemble of quantized models (EoQ) performs state-of-the-art DG while surpassing prior methods such as ERM and SWAD across multiple benchmarks, demonstrating superior accuracy-compression trade-offs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of our results' competitiveness and your acknowledgment of the broader relevance of our work to the scientific community. We address your major concerns below.\n\n**Theoretical proofs combined with the theory of model compression and domain adaptation should be offered.**\n\nWe appreciate the reviewer’s suggestion. We agree that building a rigorous theoretical bridge between quantization, model compression, and domain adaptation is a valuable direction. However, doing so is nontrivial, and thus our work focuses on highlighting an intuitive but underexplored connection, which we believe is of interest to the community: Quantization introduces structured noise that leads to flatter minima, which correlates with better generalization. While our current analysis based on second-order Taylor expansion does not constitute a full theoretical proof, we hope that it will provide the foundation for further investigation. We will clarify this in the revised manuscript.\n\n**More figure and analysis for EoQ should be offered. It also should be included in abstract.**\n\nWe appreciate the reviewer’s positive feedback on our proposed EoQ. Our goal with EoQ is to demonstrate that quantization can be effectively integrated into domain generalization frameworks, particularly ensemble models, offering significant benefits in terms of inference efficiency and model storage reduction. If the reviewer envisions specific analyses that could further illustrate the impact of EoQ, we would be happy to provide them. We indeed have included EoQ in the abstract. \n\n**Novelty is a problem since only a visual experiment and main results for generalization.**\n\nIt is unclear to us how novelty is related to the visual experiment and the results. Our novel contribution is to demonstrate that quantization-aware training, traditionally used for model compression, can serve as an implicit regularizer, with quantization noise\nenhancing DG. This novel insight has been positively acknowledged by all reviewers.\n\n**More details of the baseline choices? Can the authors compare with more SOTA methods**\n\nWe choose the baselines following prior works in the domain generalization literature, as also noted by the reviewer. Following the reviewer's suggestion, we have included a comparison with the state-of-the-art CLIP-based method, CLIPooD. For a fair evaluation, we use the same architecture (ViT-B/16). The results below demonstrate that QT-DoG not only improves domain generalization performance by 0.3% in average accuracy but also yields a 4.6x model compression.\n\n| Algorithm|Backbone|DomainNet|TerraInc|Office|AVG|Compression|\n|-|-|-|-|-|-|-|\n|ERM |CLIP |59.9 ± 0.1|60.9 ± 0.2|83.0 ± 0.1|67.9 |None|\n|CLIPood|CLIP|63.5 ± 0.1|60.5 ± 0.4|87.0 ± 0.1|70.3|None|\n|QT-DoG|CLIP|63.1 ± 0.2|61.9 ± 0.3|86.7 ± 0.2|**70.6**|**4.6x**\n\n**Why DG needs quantization methods; instead of a copy of the existing strategy to a new setting?.**\n\nOur work goes beyond merely applying an existing strategy to a new setting. It establishes a fundamental link between quantization and domain generalization, demonstrating how quantization can inherently improve generalization at minimal cost. This insight not only provides a practical and effective solution for DG but also introduces a new perspective and application for quantization methods, benefiting both research communities.\n\nMoreover, as shown in Table 2, QT-DoG can be combined with existing DG methods (e.g., CORAL, MixStyle) to further enhance performance, demonstrating that quantization is not just an independent technique but a complementary tool that can strengthen domain generalization approaches.\n\n**Computational costs are important but for most DG tasks, the benchmarks are only of limited-scale**\n\nThe main goal of our work is not to reduce computational cost via quantization but demonstrate a connection between quantization and domain generalization. The fact that quantization further reduces model size in an added benefit.  Note that, while we agree that current benchmarks are of limited scale, this is likely to change, as we see an increasing trend in the deployment of large models that often must operate under resource constraints. Our approach that jointly reduces size and improves generalization will thus be highly relevant in this setting.\n\n**Q1. Do you plan to test the QT-DoG on a larger Transformer model (e. g., ViT-Base, BERT) to verify that it also applies?**\n\nNote that we already include BERT results on the Amazon-WILDS dataset in Table 11 in the appendix. We will explicitly mention this in the main paper. Moreover, in the final version, we will include the results discussed above for ViT-Base with a CLIP pretrained backbone.\n\n**Q2. How about the model generalization ability under extremely low bit quantification?**\n\nWe have studied this in Table 8. After a certain point, there is a trade-off between compression and generalization; very aggressive quantization can lead to a performance drop.", "claims_and_evidence": "-C1: Quantifying noise can be used as a regularization means to promote the flat minimum of the loss function and thus improve the domain generalization ability. \n-E1: The paper demonstrates this with a part section of analysis (Taylor expansion analysis) and experiments (flatness measurements on different datasets). \n-Potential problem: More solid theoretical proofs can be offered here, combined with the theory of model compression and domain adaptation.\n\n-C2: QT-DoG outperforms the existing domain generalization methods, such as ERM, SWAD, MIRO, etc. \n-E2: Experimental results show that QT-DoG outperforms existing methods on multiple DG benchmark datasets, and the model is smaller and more computationally efficient. \n\n-C3: The EoQ method can provide a higher generalization performance than some classical methods (such as EoA and DiWA) while reducing the computational cost. \n-E3: Experiments show that EoQ reduces the training computational cost by 12 times compared with DiWA, and achieves better generalization performance.\n-Potential problem: Only a main table for this. More figure and detailed experimental analysis should be offered to present of the generalization ability of EoQ. Besides,  both QT-DoG and EoG should be included in abstract? it seems the abstract did not fit with some detailed information.", "methods_and_evaluation_criteria": "The method selection and evaluation indicators in this paper are reasonable: The DomainBed and WILDS benchmark datasets, which are the standard experimental environments for field generalization studies. Evaluation metrics using average classification accuracy met the criteria for the domain generalization task.", "theoretical_claims": "No explicit theoretical proofs. Cite from other papers with some natural language analysis. A more solid format would be better.\nI read the analysis, which is based on the Taylor expansion (second-order approximation) to show how quantized noise affects the curvature of the loss function, allowing the model to avoid steep minima. It seems to make sense but I cannot make the guarantee for its correctness.", "experimental_designs_or_analyses": "The main results follow the classic setting, which is valid. But I think more details of the baseline choices should be offered in main paper or related work. \nBesides,  quantification methods use QAT (LSQ, INQ, etc.) for ablation experiments to verify the impact of the different quantification strategies. GradCAM visualization shows how quantification affects the attention region of the model, which provides intuitive evidence for the effectiveness of the method. It seems that more ablation studies from different angles can be offered would be better, like the specialty of this method to apply the quantization method to DG instead of just a re-implementation in a different setting.", "relation_to_broader_scientific_literature": "This paper is relevant to the following research areas:\nDomain generalization (Domain Generalization): compared with ERM, SWAD, MIRO, DIWA and other methods, a new regularization method (quantifying noise) is proposed.\nModel quantization (Model Quantization): Different from the traditional quantization for compression, this paper first explores the impact of quantification on the generalization performance.\nThe relationship between loss flatness and generalization: SAM (Sharpness-Aware Minimization), SWAD, and the combined quantification provides a new perspective.", "essential_references_not_discussed": "More latest & SOTA works should be included. Take the DG field for example, ClipOOD, VL2V-SD. The main results are compared with classic methods but not with the latest ones. Besides, I think the related work should be re-organized. As the author mentioned this paper is not like traditional strategy while in related work, it said not a new method but demonstrates the impact of quantization on generalization. If like this, I think the novelty is a problem since only a visual experiment and main results for generalization.", "other_strengths_and_weaknesses": "Quantization methods are interesting. However, the motivation is more important, like why DG needs quantization methods, instead of a copy of the existing strategy to a new setting. Since the core problem of DG is not been touched by these quantization methods. Computational costs are important but for most DG tasks, only limited-scale benchmarks.", "other_comments_or_suggestions": "See the aforementioned sections.", "questions_for_authors": "Some are included in the previous sections if the authors willing to reply.\nBesides,\n1. Do you plan to test the QT-DoG on a larger Transformer model (e. g., ViT-Base, BERT) to verify that it also applies?\n2. How about the model generalization ability under extremely low bit quantification?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WuyDMqDD60", "reviewer": "Reviewer_N998", "summary": "The paper studies the use of quantization-aware training (QAT) for domain generalization and comes with the finding that QAT can be a valuable tool for improving generalization to out of domain settings. Strong results are obtained, showing clear differences compared to basic ERM. Combination with ensembling is studied further, showing additional benefits in improving DG abilities.\n\n## Update after rebuttal\nThanks for the additional explanations and experiments, I continue recommending acceptance for the paper.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s thoughtful feedback and recognition of our work’s contributions, especially in terms of its novelty and the various analyses provided in the appendix. Below, we provide detailed responses to each question and concern.\n\n**Performance Without Ensemble:**\n\nThank you for pointing out that SWAD slightly outperforms QT-DoG without the ensemble. However, a key distinction is that QT-DoG is 4.6× smaller than SWAD, making it significantly more efficient. Additionally, when comparing models of equal size (EoQ vs. SWAD), our EoQ method achieves significantly better performance. We will clarify this important trade-off in the revised version.\n \n**Q1: What model is used for the NLP experiments in the appendix?**\n\nWe used the same BERT model as in WILDS [1]. Different architectures and different modalities further demonstrate the effectiveness of our method. Thank you for appreciating this! \n\n[1] WILDS: A Benchmark of in-the-Wild Distribution Shifts\n\n**Q2: How would the approach perform for additional one or two models of different architecture? E.g. simple CNN, different transformer model**\n\nAs suggested by the reviewer, we conducted additional experiments using ViT-B/16 with a CLIP-based backbone, and our findings support the original results, demonstrating that QT-DoG effectively improves generalization across different architectures with different datasets. These results, shown in the table below, will be included in the revised version.\n\n| Algorithm  | Backbone | DomainNet        | TerraInc    | Office | AVG |  Compression |\n|-----------|----------|------------|------------|--------|--------|-------------|\n| ERM       | CLIP     | 59.9 ± 0.1 | 60.9 ± 0.2 | 83.0 ± 0.1      | 67.9      | None           |\n| CLIPood   | CLIP     | 63.5 ± 0.1 | 60.5 ± 0.4 | 87.0 ± 0.1     | 70.3      | None           |\n| QT-DoG    | CLIP     | 63.1 ± 0.2 | 61.9 ± 0.3 | 86.7 ± 0.2      | **70.6**      | **4.6x**   \n\nThank you for your valuable suggestion!\n\n**Q3: How well would SWAD work if used in a similar ensemble?**\n\nSWAD [1] and SMA [2] are both weight averaging methods, yielding similar accuracies to QT-DoG but for larger model sizes. An ensemble of SMA corresponds to EoA [2], and our results show that EoA performs slightly below our proposed EoQ. Moreover, our EoQ is 6× smaller than EoA, making it significantly more efficient. Even if an ensemble of SWAD were to achieve similar or slightly better accuracy, it would also result in a much larger model compared to EoQ, reducing its practical advantages in terms of efficiency and deployment.\n\n[1] SWAD: Domain generalization by seeking flat minima. Cha et al., NeurIPS 2021.\\\n[2] Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization, NeurIPS 2022.\n\n**Would be better to explain how QAT is used in more depth, even if it may be a rather direct application..**\n\nThank you for your suggestion. We will incorporate additional explanations of QAT in Section 3.1.", "claims_and_evidence": "There is clear evidence to show QAT indeed helps improve DG, using a number of commonly used datasets. There is also analytical discussion that supports the claim that QAT can help improve DG performance.", "methods_and_evaluation_criteria": "Standard methods and metrics are used for evaluating DG performance, these are suitable for the evaluation.", "theoretical_claims": "There is a discussion of how QAT contributes to better DG from an analytical perspective, the arguments appear correct and support the claim. It is a valuable aspect to have included in the paper.", "experimental_designs_or_analyses": "The designs and analyses are sensible, I appreciate that a commonly used DomainBed benchmark is used. Various analyses  e.g. on the quantization type are included and these are valuable to have. The evaluation is primarily performed with the ResNet50 model, but there are some results to show it also works well on ViT. Generally it could be useful to have it evaluated on a larger number of models, e.g. one or two more, but the current extent is reasonable to suggest it may be a generally worthwhile approach.", "relation_to_broader_scientific_literature": "The paper introduces a somewhat surprising yet intuitive observation that QAT helps improve DG. It does not introduce a particularly new method, but rather identifies a useful connection between what is used for other purposes and the area of DG. It also compares with a number of recent techniques for domain generalization.", "essential_references_not_discussed": "As far as I am aware all essential references are included.", "other_strengths_and_weaknesses": "The insight that QAT is helpful for DG is valuable and novel from my perspective. The paper is well-written and is easy to read. The appendix has an extensive number of additional results and also as part of the main paper various useful analyses are included.\n\nThe method relies on quantization aware training but it seems this part is not that much discussed in the paper, and more explanation is e.g. given to standard quantization. Would be better to explain how QAT is used in more depth, even if it may be a rather direct application..\n\nWithout the ensemble the model is not SoTA, SWAD gives a better performance. Nevertheless it is very interesting to see such a simple approach can work so well.", "other_comments_or_suggestions": "Editorial comments:\nL268 centered at w.. - two dots\nFig. 4 plot titles could be better written so that they are easier to understand", "questions_for_authors": "Q1: What model is used for the NLP experiments in the appendix?\n\nQ2: How would the approach perform for additional one or two models of different architecture? E.g. simple CNN, different transformer model\n\nQ3: How well would SWAD work if used in a similar ensemble?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UkrVeV3O9g", "reviewer": "Reviewer_6Vn5", "summary": "Domain generalization is a research field that pursues performance improvement against unseen domains of data. Widely used optimizers such as SGD(Stochastic Gradient Descent) tend to push an optimized point towards sharp and narrow minima. Thus, neural networks trained with those optimizers show low generalization ability. To resolve this problem, previous works add noise while training to achieve robustness against out-of-domain data.\nFrom this point of view, this paper draws attention to quantization for domain generalization for the first time. Quantization gives noise to full precision inputs and weight parameters, and it is essentially similar to domain generalization works that give noise while training. Based on this attribute, the paper claims that quantization can help enhance the generalization ability of target models. In addition, because quantization leads to reducing inference costs, ensembles of quantized models show a comparable memory footprint compared to a single full-precision model while achieving better accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s positive feedback and recognition of the novelty of our approach, as well as the various visual and theoretical analyses presented. We also value the reviewer’s constructive comments and thoughtful questions, which we address in detail below.\n\n**Quantization vs. Noise — what's the difference?**\n\nWe appreciate the reviewer’s insightful comment. While quantization and noise injection share the general idea of introducing perturbations during training to improve generalization, they differ in several key ways. Noise injection typically involves temporary, stochastic perturbations (e.g., Gaussian noise) applied to weights, activations, or gradients. In contrast, quantization imposes a persistent, structured constraint by discretizing weights and activations throughout training and inference. This constraint acts as a form of implicit regularization, consistently guiding the optimizer toward flatter minima — as supported by our loss landscape visualizations and flatness analyses. Unlike random noise, quantization changes the geometry of the optimization space in a deterministic way, further enhancing stability. Moreover, while many domain generalization methods rely on noise or data augmentation to promote robustness, our work is the first to show that quantization achieves similar regularization effects, with the added benefit of computational efficiency. We will revise the paper to make this distinction clearer. \n\n**Q1. Can you separate the positive effect? How much is the performance gain from DG, and how much is the performance loss from quantization?**\n\nWe address this in Table 8 through ablation studies on quantization bitwidths, showing that, whereas aggressive quantization (e.g., 2-bit) may degrade accuracy, moderate quantization (e.g., 7-bit) enhances generalization while maintaining efficient inference. Additionally, Table 7 demonstrates that moderate quantization incurs little to no in-domain accuracy loss while significantly improving out-of-domain performance. We will clarify this point and emphasize that the net generalization gain often outweighs precision loss, with certain configurations offering both performance improvements and resource savings.\n\n\n**Q2. In Figure 4, what is different between the two networks except for quantization? Why are the validation accuracies of two networks before the quantization step?**\n\nThe two models shown in Figure 4 are trained with the same architecture, training setup, optimizer, and random seed. In principle, their in-domain accuracies before the quantization step should be the same, but they nonetheless vary slightly because of slight GPU-induced randomness in the computation process, on which even the random seed has no impact. After quantization, however, we can see that the quantized model yields much more stable curves in the test domain (bottom plots). To mitigate this randomness, we followed the same process as SWAD and EoA, ensuring fixed random seeds and conducting multiple runs.", "claims_and_evidence": "- Quantization improves the flatness of loss surface while training neural networks.\n    - The paper compares the flatness of the proposed method with other works.\n    - With several experiments, the authors compare the proposal with other domain generalization works, and show how performance changes depending on quantization.\n- The paper exploits cost reduction by an ensemble of quantized models, pursuing additional performance gain compared to a single full precision counterpart.\n    - The paper compares the size, memory footprint, and performance of models.", "methods_and_evaluation_criteria": "The proposed method is based on quantization, and the paper clearly exhibits the effects of quantization in terms of loss flatness.", "theoretical_claims": "The argument of the paper is logically developed in that it considers quantization equivalent to noise for the purpose of enhancing generalization ability in domain generalization research. However, it would be better if the paper compares the fundamental difference between adding noise to full precision networks and quantization.", "experimental_designs_or_analyses": "With various datasets for domain generalization, the paper shows the effect of the proposal that quantization forces an optimized point on the loss surface to move towards a flat surface, which leads to performance improvement.", "relation_to_broader_scientific_literature": "There have been studies aimed at finding flat loss surfaces to improve quantization performance. However, it seems novel that quantization is good for domain generalization because quantized networks have a flatter loss surface compared to full precision counterparts.", "essential_references_not_discussed": "The reviewer is not an expert on domain generalization, thus it is hard to know other research that is not addressed in this paper.", "other_strengths_and_weaknesses": "The reviewer is not an expert on domain generalization. But, it seems a novel approach that claims the effect of quantization in terms of flattening loss surfaces compared to full precision networks. So, I’d like to recommend this paper for now. However, I will examine it again after reading the reviews of other experts.", "other_comments_or_suggestions": "In Line 218 of page 4, it is an equation but it isn’t marked with equation number.", "questions_for_authors": "- Even though quantization helps domain generalization of target models, quantization itself can harm the performance of models. It seems that performance gain from domain generalization exceeds performance loss from quantization. Can the authors analyze each effect separately? (How much is the performance gain from the domain generalization effect, and how much is the performance loss from quantization?)\n- In Figure 4, what is different between the two networks except for quantization? Why are the validation accuracies of two networks before the quantization step?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is an empirical study about quantization for domain generalization. Some concerns were raised regarding whether the experiments on domain generalization tasks are comprehensive and whether some claims made by the authors are well supported. In the rebuttal, the authors address most of the concerns, while more experiments about verifying whether quantization can provide better generalization ability for LLLMs would make the submission stronger.
Decision: Accept (poster)