Title: TexQ: Zero-shot Network Quantization with Texture Feature Distribution Calibration
Venue: NeurIPS 2023
Accepted: yes
Keywords: Zero-shot quantization

Texture feature calibration

Post-training quantization

low bit width

Neural network compression
Abstract: Quantization is an effective way to compress neural networks. By reducing the bit width of the parameters, the processing efficiency of neural network models at edge devices can be notably improved. Most conventional quantization methods utilize real datasets to optimize quantization parameters and fine-tune. Due to the inevitable privacy and security issues of real samples, the existing real-data-driven methods are no longer applicable. Thus, a natural method is to introduce synthetic samples for zero-shot quantization (ZSQ). However, the conventional synthetic samples fail to retain the detailed texture feature distributions, which severely limits the knowledge transfer and performance of the quantized model. In this paper, a novel ZSQ method, TexQ is proposed to address this issue. We first synthesize a calibration image and extract its calibration center for each class with a texture feature energy distribution calibration method. Then, the calibration centers are used to guide the generator to synthesize samples. Finally, we introduce the mixup knowledge distillation module to diversify synthetic samples for fine-tuning. Extensive experiments on CIFAR10/100 and ImageNet show that TexQ is observed to perform state-of-the-art in ultra-low bit width quantization. For example, when ResNet-18 is quantized to 3-bit, TexQ achieves a 12.18% top-1 accuracy increase on ImageNet compared to state-of-the-art methods. Code at https://github.com/dangsingrue/TexQ.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper starts from an interesting viewpoint that the synthetic of the previous ZSQ method usually fails to model the similar text feature like real data. As a result, the authors suggest retaining the texture feature. At first, they synthesize calibration images with a LAWS texture feature energy preserve loss. Then, the calibration images are used to provide mean and variance centers for each class, which is then used to guide the generator to synthesize samples. At last, they proposed using a mix-up strategy to further augment the synthetic data.\n\nSTRENGTHS:\n1, The viewpoint is interesting and persuasive. Also, the results do demonstrate the effectiveness of their method.\n2, Sec 5.1 provide many meaningful discussion, which makes this paper more convincing.\n\nWEAKNESSES:\n1, To be honest, the writing of this paper is not good enough and I get lost in many parts. \n\nFor example, Line 171, ‘Q always faces linear decisions when inferencing’ is very strange. I can't understand the meaning behind it.\n\nLine 166, where µ_l (\\bar(x)| \\bar(y) = k) and σ_l (\\bar(x)| \\bar(y) = k) should be the calibrating mean and variance center since calibration set C = {( \\bar(x), \\bar(y))} in Line 160.\n\n2, Why do the authors want to use a calibration center in Step2: Synthetic samples generation? The paper lacks some discussion.\n\n3, The principle behind Calibration set generation and Synthetic samples generation is not clear. Why do you still need Synthetic samples, even though you already have Calibration set. I know such a way could give a better performance. The authors should provide some ablation study when only using Calibration set.\n\n4, The role of Eq6. The K set only selects top-k texture elements that account for more than 80%. However, the images are randomly initialized from Gaussian noise. I am wondering if the distribution of the K set keeps the same for each synthetic image. For example, R5R5 is very fit for Gaussian noise and is selected for each image.\n\nQUESTIONS:\n1, Line 193, what is \\bar(D)?\n2, It actually uses synthetic data from the generator and from direct optimization. It may an unfair comparison.\n\nLIMITATIONS:\nThe authors do provide a discussion about the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
TexQ is a novel zero-shot quantization (ZSQ) method that addresses the limitations of conventional synthetic samples in retaining texture feature distributions. It achieves state-of-the-art results in ultra-low bit width quantization, with a significant accuracy increase compared to existing methods on ImageNet.

STRENGTHS:
Strengths are shown below:

1. This paper tried to exploit the important direction for inevitable privacy and security.
2. The performance on the benchmark datasets is promising.
3. The paper is well-organized.

WEAKNESSES:
1. Maybe the structures used are insufficient with ResNet-18 and MobileNet-V2. What about the most commonly used ResNet-50?
2. Maybe the datasets verified are insufficient with Cifar and ImageNet. What about the others tasks, such as CoCo of detection.

QUESTIONS:
The authors could refer to the Weaknesses. Additionally, I also have some open questions to discuss.

1. The authors could discuss how to ensure the generalization of the method since the calibration is conducted on a specific dataset.
2. Could the method be used in the Transformer structure?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes TexQ, which targets keeping the texture information of the synthetic samples of zero-shot quantization. The texture feature energy distribution calibration method is applied to the synthesized samples, and mixup knowledge distillation is introduced to improve the diversity of the synthetic samples. Extensive experiments of ResNet, MobileNet on CIFAR, and ImageNet datasets prove the effectiveness of this method.\n\nSTRENGTHS:\n* The paper is well-organized and easy to follow. Fig.3 is an excellent visualization of the proposed system architecture.\n\n* To measure the texture feature distribution for the input sample of quantization is novel. The comparison (Fig. 4/5) of synthetic samples and natural images is insightful and may help future research on the ZSQ.\n\n* The experiment results are convincing and high accuracy (especially for 3-bit) demonstrates the effectiveness of TexQ.\n\nWEAKNESSES:\n* The introduction of the concept \"LAWS texture feature energy\" needs to be improved. The example given in Eq.5 is not straightforward. Visualization of features extracted using $E_5, S_5, W_5, R_5$ would be better. Why these \"texture\" feature is important to retain should also be included more in the discussion.\n* There are too many artifact weighting coefficients $\\alpha_i, i \\in [1,5]$ to balance different loss function. The author claims in Sec. 4.1 that they \"empirically\" select the value. An ablation study or sensitivity analysis on these parameters would be better to prove that the proposed loss is solid and that better performance does not come from a grid search of the parameters.\n* The method is only validated on CNNs. The experiment results on Vision Transformer would be a plus.\n\nQUESTIONS:\n* Most previous work uses the term ''data-free quantization (DFQ)'' instead of \"zero-shot quantization (ZSQ).\" I want the author to make sure that these two terminologies are identical.\n* What is the actual (visual/physical/signal) meaning of the feature extracted using the filters $E_5, S_5, W_5, R_5$?\n* Can TexQ be transferred to language tasks without \"texture\" in the data?\n\nLIMITATIONS:\nThe author includes a discussion of the problems to be solved in Section 5.2, which is great. One crucial limitation I would like to raise is that this work is built upon the concept of \"texture\" and is **only meaningful for visual samples**. The application to other tasks with different modality (language, speech, etc.) would be limited.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper points out that there is a strong dependency between the performance of CNN and the texture feature of the dataset. For extending this concept to the quantization field, the paper adopts calibration samples which are trained with manually designed texture filters. In addition to synthetic samples which are generally used in ZSQ works (generated by a network that is trained with Batch Normalization layers’ statistics of a full precision model), the paper exploits calibration samples to quantize a model without the original dataset. Furthermore, the paper applies mixup data augmentation to improve the quantized model’s performance.\n\nSTRENGTHS:\n- This paper is well-motivated and easy to follow.\n\n- The paper adopts a concept of texture feature in Zero-shot quantization for the first time.\n\n- The paper demonstrates the proposed method well with several formulas and figures.\n\nWEAKNESSES:\n- It seems costly that generate calibration samples along with synthetic samples for quantizing a neural network.\n\ntypo:\n- In the 215th row, presnted -> presented\n\nQUESTIONS:\n- In equation 1, only a rounding operation is applied to obtain quantized integers without a clip operation?\n\n- The paper design texture filter manually. Is it possible to get texture filters with the training process?\n\n- Calibration samples are obtained without a generator. Is it possible to generate calibration samples with another generator, or the synthetic sample generator by giving the texture feature energy distribution calibration loss?\n\n- How expensive it is to generate both samples, compared to other works that generate synthetic samples only?\n\n- Qimera [1] executed several experiments with mixup and cutmix, similar with 'Mixup knowledge distillation module' described in 3.2.3, claiming that superposed latent embedding works better than mixup and cutmix. Can a comparison with those methods be provided?\n\n\n[1] Choi et al. Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples. In Conference on Neural Information Processing Systems. 2021\n\nLIMITATIONS:\nPrevious ZSQ works which exploit Batch Normalization layers' statistics for generating synthetic samples are hard to apply transformer-based models.\n\nIt is worth analyzing that the proposed method can be applied to those models.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
They suggested a zero-shot quantization method to retain the detailed texture feature distribution and introduced the mixup knowledge distillation module to diversify synthetic samples for finetuning

STRENGTHS:
They identified the new feature required when generating synthetic data for quantization.

WEAKNESSES:
They should compare their work with MixMix [1], Genie[2] and KW[3].

The authors only empirically showed their superiority. i.e. lt lacks explanations of intuitive or mathematic. The author should give more reasons.

The image they generated showed a little bit of poor quality to argue that it has captured the texture feature distributions. please see the synthetic images in [1], [2], [3].

[1] Li, Yuhang, et al. "Mixmix: All you need for data-free compression are feature and data mixing." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[2] Jeon, Yongkweon, Chungman Lee, and Ho-young Kim. "Genie: Show Me the Data for Quantization." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[3] Haroush, Matan, et al. "The knowledge within: Methods for data-free model compression." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

QUESTIONS:
1. please provide more detail on why the texture feature distribution calibration is important when generating synthetic data.

2. PTQ methods such as AdaRound, AdaQuant, and Brecq can employ synthetic data to quantize models. how about adapting these post-training quantization schemes for your methods?

3. I expect to see more evaluation on various models in order to convince the superiority of your work.

LIMITATIONS:
It lacks a literature survey.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-06