Fisher-aware Quantization for DETR Detectors with Critical-category Objectives

Venue: ICML_cc_2024_Workshop_WANT
Published: 2024-05-26 02:26:54

Abstract: The impact of quantization on the overall performance of deep learning models is a well-studied problem. However, understanding and mitigating its effects on a more fine-grained level is still lacking, especially for harder tasks such as object detection with both classification and regression objectives. This work defines the performance for a subset of task-critical categories, i.e. the critical-category performance, as a crucial yet largely overlooked fine-grained objective for detection tasks. We analyze the impact of quantization at the category-level granularity, and propose methods to improve performance for the critical categories. Specifically, we find that certain critical categories have a higher sensitivity to quantization, and are prone to overfitting after quantization-aware training (QAT). To explain this, we provide theoretical and empirical links between their performance gaps and the corresponding loss landscapes with the Fisher information framework. Using this evidence, we apply a Fisher-aware mixed-precision quantization scheme, and a Fisher-trace regularization for the QAT on the critical-category loss landscape. The proposed methods improve critical-category metrics of the quantized transformer-based DETR detectors. They are even more significant in case of larger models and higher number of classes where the overfitting becomes more severe. For example, our methods lead to 10.4% and 14.5% mAP gains for, correspondingly, 4-bit DETR-R50 and Deformable DETR on the most impacted critical classes in the COCO Panoptic dataset.

Main paper: 
 introduction: 
 Object detection is a challenging core application in computer vision, which is crucial for practical tasks such as autonomous driving etc. Recent DEtection TRansformer (DETR) model (Carion et al., 2020) and its variants achieve state-of-the-art results on multiple detection benchmarks (Liu et al., 2022) . However, their performance comes at the cost of large model sizes and complicated architecture. Therefore, quantization (Choi et al., 2018; Dong et al., 2020; 2019; Polino et al., 2018; Yang et al., 2021) is typically applied in real-world applications to reduce the memory footprint and inference latency time on cloud and edge devices (Horowitz, 2014) . Inevitably, the perturbation of weights and activations introduced by the quantization process degrades the performance of floating-point models. Previous research mainly focuses on a trade-off between the model size and the overall performance (e.g. average accuracy for classification and mean average precision (mAP) for detection) of the quantized models (Dong et al., 2019; Yang et al., 2021; Xiao et al., 2022) .However, a fine-grained performance objectives are often more important than the overall performance in the real world (Barocas et al., 2019; Tran et al., 2022) . Suppose an autonomous vehicle is processing a scene containing people, vehicles, trees, light poles, and buildings, as illustrated in Fig. 1 (left) 1 . Some non-critical objects (light poles, trees, and buildings) only need to be localized to avoid collision, yet misclassification within this group of categories is not as critical if they are all considered as "other obstacles". On the other hand, critical classes such as a person or vehicle require both accurate classification and localization for safe operation. The overall performance cannot distinguish between an error within non-critical categories vs. a critical object error. So, it is missing the granularity to represent the true task-critical objectives of real-world applications. Yet to the best of our knowledge, for both post-training quantization (PTQ) and quantization-aware training (QAT), the analysis of the impact on such task-critical fine-grained objectives of object detection models is overlooked.Figure 1 : Overview. We investigate a practical setting with task-dependent critical-category objectives, which is formulated with label transformation in Sec. 3.2. We empirically observe disparate effects of quantization on the critical-category performance in Sec. 3.3 , where PTQ and QAT lead to performance gaps for critical categories. With the theoretical analysis using Fisher information, we propose Fisher-aware quantization scheme and regularization in Sec. 4 to improve critical performance for quantized DETR models as demonstrated in Sec. 5 experiments.In this paper, we follow this practical yet overlooked setting to formulate a set of task-critical objectives for DETR-based object detection models, accomplish a fine-grained quantization impact analysis, and propose techniques to improve the corresponding objectives. Specifically, we disentangle classification and localization objectives to define a fine-grained critical-category performance with non-critical label transformation, as shown in the updated bounding boxes in Fig. 1 (left) . With this formulation, we introduce a thorough analysis of the impact of quantization on the criticalcategory performance of DETR model. As illustrated in Fig. 1 (right), we find that quantization has a disparate effect on the category-wise performance, where some groups of classes are more sensitive to quantization with up to 1.7% additional mAP drop. Furthermore, unlike the almostguaranteed improvement of the overall performance after QAT, such training can increase performance gaps for some task-critical categories. We provide both theoretical and empirical analysis of such quantization effects using the loss surface landscape of the critical objectives using the Fisher information framework (Perronnin & Dance, 2007) .Based on this analysis, we propose two novel techniques: Fisher-aware mixed-precision quantization scheme, and Fisher-trace regularization. Both techniques optimize the landscape of critical objectives and, therefore, improve critical-category performance. Our experiments show consistent critical-category performance improvements for DETR object detectors with different backbones and architecture variants. The contributions of this paper are summarized as follows:• We formulate the critical-category performance for object detection applications, and observe disparate effects of quantization on the performance of task-critical categories. • We provide analytical explanations of the quantization effects on critical-category performance for DETR-based models using a theoretical link to the Fisher information matrix. • We propose Fisher-aware mixed-precision quantization scheme that considers the sensitivity of critical-category objectives and improves corresponding detection metrics. • We propose Fisher-trace regularization for the loss landscape of our objectives during quantization-aware training to further improve critical-category results. 
related work: 
 Object detection. Object detection is a core task for visual scene understanding. Conventional object detectors rely on a bounding box proposals (Girshick, 2015) , fixed-grid anchors (Redmon et al., 2016) or window centers (Tian et al., 2019) . However, the performance of these methods is largely affected by the bounding box priors and the post-processing steps (Carion et al., 2020) . The transformer-based DETR (Carion et al., 2020) provides a fully end-to-end detection pipeline without a surrogate tasks. Follow-up research further enhances DETR by introducing a deformable attention (Zhu et al., 2021) , query denoising (Li et al., 2022) , and learnable dynamic anchors as queries (Liu et al., 2022) . With the growing popularity of DETR-based architecture, we believe that understanding of quantization impact on DETR performance is an important topic, especially at the fine-grained level. Common object detection benchmarks evaluate fine-grained performance metrics that depend on the object size (Lin et al., 2014) or occlusion status (Geiger et al., 2012) . However in practical applications, object type i.e. its category is often more important than the object size. This motivates us to further investigate detectors with critical-category objectives.Efficiency-performance tradeoff. Multiple methods have been proposed to compress deep neural network (DNN) models, including pruning (Han et al., 2015; Wen et al., 2016; Yang et al., 2020b; 2023) , quantization (Polino et al., 2018; Dong et al., 2020; Yang et al., 2021; Guo et al., 2022) , factorization (Wen et al., 2017; Ding et al., 2019; Yang et al., 2020a) , and neural architecture search (Wu et al., 2019; Cai et al., 2020) etc. This work explores the impact of quantization that is widely supported by the hardware (Horowitz, 2014) and can be almost universally applied to DNN compression in architecture-agnostic fashion. Post-compression model performance is the key focus in previous research. However, the overall i.e. average performance hides the important fine-grained metrics e.g. the results for certain groups of categories. Recent works (Tran et al., 2022; Good et al., 2022) analyze the disparate impact of pruning on classification accuracy, which leads to the fairness concerns (Barocas et al., 2019) . Our work extends this direction and investigates quantization effects of DETR-based object detection at the critical-category performance granularity.Second-order information in deep learning. Unlike conventional optimization with the first-order gradients, recent research has found the importance of utilizing second-order information to increase the generalization and robustness of DNN models. Sharpness-aware minimization (Foret et al., 2021) links a loss landscape sharpness with the model ability to generalize. The latter can be improved using a regularized loss with the Hessian eigenvalues (Yang et al., 2022) computed w.r.t. parameter vector. Hessian eigenvalues are also used as importance estimates to guide the precision selection in mixed-precision quantization (Dong et al., 2019; 2020; Yao et al., 2021) . Given the difficulty of exact Hessian computation, Fisher information matrix is proposed as an approximation of the importance in pruning (Kwon et al., 2022) . In this work, we link the disparate impact of quantization to the critical objectives with the second-order Fisher information, and propose Fisher-aware quantization and regularization methods to overcome the quantization effects on critical categories in object detection. 
critical-category performance analysis: 
 In this section, we provide an introduction to the notations and the object detection training objectives for DETR model in Sec. 3.1; formulate our critical-category performance in Sec. 3.2; and finally empirically analyze the impact of quantization on such critical performance in Sec. 3.3.We recap the computation process of DETR-based object detectors, and provide relevant notations for the rest of this paper. We mainly focus our discussion on the output of DETR model rather than its architecture. Then, the following notation is applicable to both the original DETR (Carion et al., 2020) and its more advanced variants such as DAB-DETR (Liu et al., 2022) , Deformable DETR (Zhu et al., 2021) as well as any other detector with the end-to-end architecture.Given an input image x, the DETR-type model f (•) outputs a fixed-size set of N bounding box predictions f (x) = {[p i , bi ]} i=1...N . In each bounding box prediction, pi is the vector of classification logits and bi is the vector of bounding box coordinates. The former pi ∈ R n+1 contains logits for n classes and an empty-box class (∅). The predicted bounding box bi ∈ R 4 consists of 4 scalars that define the center coordinates as well as the height and the width relative to the image size.During the training, annotation is provided for each training image as a set of ground truth boxesy i = {[c i , b i ]}, where c i is the target class label and b i defines the bounding box. A Hungarian matching process is performed to find the closest one-to-one matching between ground truth boxes and predicted boxes including those with "no object" ∅ predictions. The training loss is computed between each pair of matched boxes, which is defined as a linear combination of a classification loss L cls (p i , c i ) for all boxes, and a box loss L box ( bi , b i ) for all non-empty boxes.As discussed in Sec. 1, the overall performance metric evaluated on the validation dataset can be not the most effective objective in some real-world scenarios. Category-level fine-grained performance for some specific task-critical categories can be more crucial than the averaged metrics. Here we provide a practical definition of the critical-category objectives for the detection task, and a corresponding evaluation method when applied to DETR-type detectors.In classification tasks, class-level performance is often defined as the accuracy or loss of the model on a subset of the validation dataset that contains objects belonging to a certain group of classes (Tran et al., 2022) . However, such definition is not practical for object detection task, as each input image in the dataset contains multiple objects from different categories. Instead, this work defines the critical objective based on the entire validation dataset, but with a transformed model outputs and annotations during the loss computation in order to focus it towards a certain group of critical object categories.Formally, suppose there are in total n categories in the dataset. Without loss of generality, suppose the first m categories are the critical ones for a certain task that requires both an accurate classification and localization. Then, the rest of categories (m + 1 to n) are non-critical and a misclassification between them is acceptable. This can be expressed by the transformed prediction p′ i ∈ R m+2 asp′ i [j] =    pi [j] j = 1 . . . m max pi [m + 1 : n] j = m + 1 pi [n + 1] j = m + 2.(1)The (m + 1)-th category in p′ i corresponds to "others", which represents non-critical categories. The max function is used to avoid a distinction when classifying non-critical categories. The (m + 2)-th category in p′ i is used for ∅ class, which is originally defined as the (n + 1)-th category in pi . The same transformation is also applied to the ground truth box label c i , where all c i ∈ {m+1, . . . , n} are set as the (m + 1)-th label in the transformed c ′ i . No change is applied to the bounding box coordinates for the predicted and ground truth bounding boxes as we only define critical performance at the classification granularity to have simplified yet practical and instructive setting.Note that the logit transformation can be applied directly to the output of a trained DETR model without any change to its architecture or weights W . With both p′ i and c ′ i being transformed, the Hungarian matching, loss computation, and mAP computation can be performed without modification. We name the loss computed with the original pi and c i as "Overall objective" and it is expressed asEQUATIONSimilarly, the "Critical objective" is defined with the transformed p′ i and c ′ i asEQUATIONEach objective corresponds to either the "Overall performance" or the "Critical performance" when evaluating the mAP detection metric with the original or transformed outputs and labels, respectively.With the defined objective, we analyze how quantization affects the critical performance of DETR model. Specifically, we start with the official pretrained checkpoint of DETR with ResNet-50 backbone 2 . We use a symmetric linear quantizer Q(•) (Dong et al., 2019) to quantize a weight tensor W to q bits, which can be expressed by We quantize all trainable weights in the DETR model with an exception of the final feed-forward (FFN) layers for the class and bounding box outputs. Quantization of these FFN layers leads to a catastrophic performance drop in the PTQ setting (Yuan et al., 2022) . A 4-bit quantization is applied uniformly to the weights of all layers for all experiments in this section.Q(W ) = Round W 2 q-1 -1 max(|W |) max(|W |) 2 q-1 -1 . (4)For critical performance, we define critical categories based on the "super category" labels in the COCO dataset (Lin et al., 2014) . In total, 12 super categories are available in the COCO, where each contains from 1 to 10 categories of similar objects. For each selected super category, we consider all the categories within it as critical categories, while the rest of categories as non-critical and transform their logits and labels accordingly. The mAP measured at the transformed output is denoted as the critical mAP of this super category. For example, when measuring the critical performance of "Indoor" super category, "book", "clock", "vase", "scissors", "teddy bear", "hair drier", and "toothbrush" are considered as critical categories (the first m categories in the Eq. ( 1) logit-label transformation), while others are set as non-critical. We perform such evaluation for all 12 super categories to understand the category-level impact of quantization on DETR.As shown in Tab. 1, quantization has a disparate impact on the critical performance of the DETR model. The mAP drop after quantization has an up to 1.7% gap. We further perform 50 epochs of QAT and report the critical performance in Tab. 2. The performance increases differently for each super category with a gap of up to 1.1% mAP. 
methods to overcome critical-category quantization effects: 
 In this section, we provide theoretical analysis on the cause of empirical performance gaps in Sec. 4.1. Then, we propose our methods to improve such performance from the aspect of quantization scheme design and quantization-aware training objective in Sec. 4.2 and Sec. 4.3, respectively.For a pretrained DETR model with weights W , we investigate how quantization affects the critical objective L F (W ). We obtain the following two insights:Insight 1: Larger Fisher trace of critical objectives results in a higher sensitivity to weight perturbation. The quantization process replaces the floating-point weights W of the pretrained DETR model with the quantized values Q(W ) as in Eq. ( 4). Effectively, this perturbs the weights away from their optimal values, which leads to an increase in the critical objective value. With the second-order Taylor expansion around W , the quantization-perturbed loss L F (Q(W )) can be approximated using the non-perturbed objective L F (W ) asEQUATIONwhere the gradient g = ∂L F (W )/∂W , the Hessian H = ∂ 2 L F (W )/∂W 2 , and the weight perturbation or, in other words, the quantization error ∆ = Q(W ) -W . We assume that the pretrained model converges to a local minimum. Then, the first-order term can be ignored because g is close to zero (LeCun et al., 1989) . Hence, Eq. ( 5) can be rewritten asEQUATION)For large models such as DETR, computation of the exact Hessian matrix H is practically infeasible. Previous research (Kwon et al., 2022) shows that the Hessian can be approximated by the empirical Fisher information matrix I computed as the expectation over the entire dataset byEQUATIONIn practice, we can also assume I to be a diagonal matrix. This further simplifies Eq. ( 6) asEQUATIONwhere the latter result represents a sum of Fisher trace elements (tr(I) = i I ii ) weighted by the squared quantization error and it is performed over each element w i of W .Eq. ( 8) provides a feasible yet effective sensitivity metric to estimate the impact of quantization noise on the critical objectives. It analytically connects the quantization-caused weight perturbation with the maximum likelihood estimator for critical objectives using Fisher information framework (Ly et al., 2017) . Hence, an objective with the larger sensitivity metric leads to inferior critical performance. To verify Eq. ( 8), we plot the Fisher trace for the critical objectives when estimated using the pretrained floating-point DETR in Fig. 2 (left). We can clearly see a correlation between a larger Fisher trace and a lower critical-category performance metric for the post-training quantization setting.Insight 2: Sharp loss landscape leads to a poor generalization for critical categories after quantization-aware training. During the conventional QAT process, weights of the DETR model are trained to minimize the overall objective L A (Q(W )). However, a convergence of L A does not guarantee good performance on all critical objectives L F . Foret et al. (2021) find a positive correlation between the validation performance for the objective L F and a sharpness S of the loss landscape around the local minima Q(W ) of the QAT. The minima sharpness S(Q(W )) of the quantized model is formulated asEQUATIONwhere ρ > 0 is a ℓ 2 norm bound for the worst-case weight perturbation.Finding the exact solution to the maximization in Eq. ( 9) can be computationally costly. With the detailed derivations in Appendix A, we can simplify this problem asEQUATIONHence, the trace of the diagonal Fisher information matrix from Eq. ( 7) approximates the sharpness of the critical loss landscape for the quantized model. The sharper loss landscape leads to inferior test-time critical performance after QAT. To verify this, we empirically measure the Fisher trace using Eq. ( 10) on different critical objectives of the DETR model after 50 epochs of QAT. As shown in Fig. 2 (right), lower Fisher trace leads to higher critical mAP after QAT. Hence, Fisher trace of the critical objective is a good indication of the model's post-QAT generalization.With the derived quantization impact on the loss in Eq. ( 8), we propose a mixed-precision quantization scheme that minimizes the quantization effects within a model-size budget. Specifically, we define the following minimization problem minEQUATIONwhere L denotes the number of layers in the model, w i is the i-th layer's weight vector, and ∆ i = Q(w i ) -w i is the quantization error of the i-th layer's weight when quantized to q i bits. B denotes the model size allowance of the quantized model. As the quantization precision q i takes discrete integer values, the optimization problem in Eq. ( 11) can be efficiently solved as an Integer Linear Programming (ILP) problem (Dong et al., 2020; Yao et al., 2021) .Note that in Eq. ( 11) we consider the Fisher information of both critical and overall objectives. This approach allows us to achieve good overall performance and increase the critical performance of interest for the quantized model. A hyperparameter α is used to balance L F and L A .Besides using Fisher information metric to design quantization scheme, we further apply regularization during the QAT process of the quantized DETR model to achieve better generalization performance on critical categories. Following the derivation in Eq. ( 10), we propose to minimize critical loss sharpness S(Q(W )) during the conventional QAT optimization. Specifically, for a critical objective L F , we add the Fisher trace regularization asEQUATIONwhere λ ≥ 0 is the strength of the regularization, and I F denotes the Fisher information matrix of the critical objective L F (Q(W )) w.r.t. weight W .In addition to the DETR training loss terms from Eqs. ( 2) and ( 3), we further add a distillation loss (Hinton et al., 2015) between the quantized model (student) and the pretrained full-precision model (teacher) following common QAT practice (Dong et al., 2020; Yang et al., 2021) . The distillation objective consists of a KL-divergence loss for class logits of the student and teacher models, and a ℓ 1 loss for the corresponding bounding box coordinates. Since we expect the student model to have the same behavior as the teacher model, the distillation loss uses a fixed one-to-one mapping between the predicted boxes of the two models without performing the Hungarian matching. 
experimental setup: 
 Datasets and metrics. We follow DETR (Carion et al., 2020) Model architectures. We conduct the majority of our experiments on the DETR model with ResNet-50 backbone (DETR-R50). To show the scalability, we also experiment with ResNet-101 backbone variant (DETR-R101), DAB-DETR (Liu et al., 2022) and Deformable DETR (Zhu et al., 2021) .Implementation details. We perform quantization of the pretrained models using their publicly available checkpoints. We apply symmetric layer-wise weight quantization using Eq. ( 4), where weights are scaled by the max of absolute values without clamping. We keep normalization and softmax operations at full precision. For sensitivity analysis, we compute Fisher trace for our method using all training set. Whereas for the implementation of HAWQ-V2 (Dong et al., 2020 ) baseline, we randomly sample 1,000 training images due to the high cost of Hessian computation. We solve mixed-precision quantization problem in Eq. ( 11) by the ILP with 3-8 bit budget for each layer. We perform QAT with the straight-through gradient estimator (Bengio et al., 2013) for 50 epochs with 1e-5 learning rate. Regularization strength λ in Eq. ( 12) grows linearly from 1e-3 to 5e-3 throughout the training epochs when Fisher regularization is applied. To mitigate the variance in training, in all experiments we report the mean and, if shown, ± standard error of the final 5 epochs of training.As proposed in Sec. 4.2, we use Fisher information as a sensitivity measurement for designing mixed-precision quantization scheme for object detection. We compare the proposed quantization scheme with linear uniform quantization (Polino et al., 2018) and HAWQ-V2 (Dong et al., 2020) on the COCO dataset task using its detection and panoptic variants. Additional CityScapes (Cordts et al., 2016) dataset results are reported in Appendix C. Tab. 3 reports the critical mAP of super category "Person", "Animal", and "Indoor" for the COCO detection dataset. We generate HAWQ-V2 and Fisher-Overall quantization schemes with only the overall objective, and evaluate the model after QAT on these super categories. For Fisher-Critical schemes, we apply the critical-category objective for the super category of interest to the ILP optimization in Eq. ( 11), and report the critical mAP of the quantization scheme corresponding to each super category. The overall mAP is not affected much by the choice of Fisher-Critical vs. Fisher-Overall objectives as additionally described in Appendix C.With the same average quantization precision, our Fisher-aware method consistently outperforms uniform quantization and the mixed-precision scheme derived from HAWQ-V2 on different models and datasets. We note that the improvement of HAWQ-V2 over the uniform quantization is not consistent on DETR-based models. This is caused by the instability of Hessian trace estimation for the complicated DETR architecture and the harder object detection task. Fisher-aware approach, on the other hand, is stable. In addition, we evaluate the time to estimate the Fisher trace w.r.t. the Hessian trace for a batch of images on a Tesla P100 GPU, where Fisher trace can be estimated with 200-300× less computational cost. This allows us to estimate Fisher trace with a larger amount of training data, which leads to a higher precision and stability.Comparing to Fisher-overall, applying the critical objective leads to consistent improvements in the corresponding critical performance. In particular, we improve critical mAP by up to 0.2% on DETR-R50, 0.5% on DETR-R101, 0.8% on DAB DETR-R50, and 0.4% on Deformable DETR-R50.Note that the originally poorly-performing critical categories such as "Person" result in significant improvements. We further extend our experiments to the COCO panoptic dataset with the same setup, where we report the mAP box of different quantization schemes in Tab. 4. With the increased number of categories in the panoptic dataset, the impact of quantization on each individual critical category becomes even more disparate. Fisher-aware quantization with the overall objective improves critical mAP by about 2× over uniform quantization. Further improvement on critical mAP is consistently achieved with the Fisher-critical quantization scheme. This further shows the importance of considering critical objectives when applying object detection models to real-world applications.In this section, we compare the post-QAT results when using the conventional overall loss L A only vs. our approach with Fisher trace regularization from Sec. 4.3 on the COCO detection dataset in Tab. 5. The experimental results show that applying the proposed Fisher-trace regularization further improves critical mAP. When combined with the mixed-precision quantization scheme from Sec. 4.2, our method leads to a 1.15% and 0.48% critical ("person" class) performance improvement on DETR-R50 model over the uniform quantization in Tab. 3 for 4-bit (34.6% → 35.75%) and 6-bit (37.3% → 37.78%) precision, respectively. Note that our regularization scheme has a negligible impact on the overall mAP: 37.07% → 36.97% for 4-bit and 39.67% → 39.70% for 6-bit precision, respectively.Tab. 6 reports post-QAT results for COCO panoptic dataset. The proposed regularization further increases critical performance by 0.11% and 0.34% mAP for, correspondingly, 4-bit and 5-bit precision settings when compared to our PTQ results in Tab. 4. Note that the uniform PTQ quantization significantly underperforms in this setting. In Appendix D ablation study, we analyze the impact of regularization strength. In addition, we compare model generalization abilities for a naive approach when the critical loss L F is added to the overall loss L A as a heuristic QAT objective, and our Fisher-trace regularization scheme that minimizes sharpness of the loss landscape. 
conclusions: 
 This work investigated the impact of quantization on the fine-grained critical-category performance of DETR-based object detectors. Motivated by the demand for practical applications, we formulated the critical performance via the logit-label transformation of the corresponding categories. We found that both the conventional PTQ and QAT cause disparate quantization effects on such critical performance. We theoretically linked the disparate quantization effects with the sensitivity of critical objectives to the quantization weight perturbation and the sharpness of the critical loss landscape in the QAT. We characterized both derivations with the trace of the Fisher information matrix of the critical objectives w.r.t. model weights. We proposed the Fisher-aware mixed-precision quantization scheme and Fisher-trace regularization to improve the critical performance of interest. We hope this work motivates future explorations on the fine-grained impacts of other compression methods in the computer vision area and a general machine learning research.In the following supplementary materials, we provide detailed derivation and additional visualizations and experimental results. Specifically, Appendix A provides detailed derivation on Eq. ( 10). Appendix B shows detailed visualizations of the layer-wise Fisher sensitivity and the corresponding quantization precision settings derived using the proposed Fisher-aware quantization scheme method. Appendix C further analyzes how the overall and critical-category objectives from Eq. ( 11) impact the assigned precision, and shows additional experimental results for the Fisher-aware quantization scheme. For Fisher-trace regularization, Appendix D performs ablation study on the impact of regularization strength λ, compares the critical Fisher trace of the converged model with or without Fisher-trace regularization, and verifies the design choice of using Fisher regularization instead of the sum of overall and critical-category objective in the optimization.A DETAILED DERIVATION OF EQ. ( 10)Starting from the formulation in Eq. ( 9), we perform the first-order Taylor expansion on the perturbed loss L F (Q(W ) + ϵ) asEQUATIONBy substituting Eq. ( 13) into Eq. ( 9), the solution to maximization problem can be simplified asEQUATIONNote that both ϵ and ∂L F (Q(W ))/∂W are vectors with the same dimensions as weight W . Then, their inner product can achieve the maximum when they are parallel vectors. Therefore, we can solve the maximization in Eq. ( 14) asEQUATIONwhich is the final approximation of the loss landscape sharpness in Eq. ( 10). 
 
 Appendix: This section illustrates the Fisher-aware sensitivity and the corresponding quantization schemes assigned to the Fisher-overall configurations from Tab. 3 and Tab. 4. We report four models, namely DETR-R50 (Fig. 3 ), DETR-R101 (Fig. 4 ), DAB DETR-R50 (Fig. 5 ), and Deformable DETR-R50 (Fig. 6 ), on the COCO detection dataset, and, additionally, report DETR-R50 (Fig. 7 ) and DETR-R101 (Fig. 8 ) on the COCO panoptic dataset.As shown in the figures, the backbone layers demonstrate a relatively stable sensitivity distribution, while the transformer encoder and decoder layers show sensitivity distribution with high variance. This is expected given the different functionalities of transformer layers within an attention block (Carion et al., 2020) . Correspondingly, the quantization precision is assigned to each layer via solving the ILP in Eq. ( 11), where higher precision is assigned to layers with higher sensitivity, while satisfying the constraint on the overall budget allowance. This section provides additional experimental results on utilizing critical-category objectives in the proposed Fisher-aware mixed-precision quantization scheme. To start with, Figs. 9 to 14 compares the quantization scheme of Fisher-overall configuration and Fisher-critical configuration for different models and critical categories reported in Tabs. 3 and 4. As shown in the figures, the inclusion of different critical objectives into the sensitivity analysis leads to significant change in the precision assigned to some of the layers. This further illustrates the diverse sensitivity distribution of various critical-category objectives across layers. By considering the proposed objectives of interest in the sensitivity analysis, we are able to improve the critical performance of quantized models. https://dl.fbaipublicfiles.com/detr/detr-r50-e632da11.pthhttps://github.com/encounter1997/DE-DETRs 


Decision: The paper proposes a new detection model that combines DETR-type architectures with low (4) bit quantization. The reviewers agree that the paper is interesting, with novel elements. The AC agrees and recommends for acceptance. Please try to address the feedback received by the camera ready.