Efficient Low-Bit Quantization with Adaptive Scales for Multi-Task Co-Training

Authors: Boyu Liu, Haoyu Huang, Linlin Yang, Yanjing Li, Guodong Guo, Xianbin Cao, Baochang Zhang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Low-Bit Quantization, Multi-Task Learning, Co-Training, Quantization-Aware Training, Quantization Scale

Abstract:
Co-training can achieve parameter-efficient multi-task models but remains unexplored for quantization-aware training. Our investigation shows that directly introducing co-training into existing quantization-aware training (QAT) methods results in significant performance degradation. Our experimental study identifies that the primary issue with existing QAT methods stems from the inadequate activation quantization scales for the co-training framework. To address this issue, we propose Task-Specific Scales Quantization for Multi-Task Co-Training (TSQ-MTC) to tackle mismatched quantization scales. Specifically, a task-specific learnable multi-scale activation quantizer (TLMAQ) is incorporated to enrich the representational ability of shared features for different tasks. Additionally, we find that in the deeper layers of the Transformer model, the quantized network suffers from information distortion within the attention quantizer. A structure-based layer-by-layer distillation (SLLD) is then introduced to ensure that the quantized features effectively preserve the information from their full-precision counterparts. Our extensive experiments in two co-training scenarios demonstrate the effectiveness and versatility of TSQ-MTC. In particular, we successfully achieve a 4-bit quantized low-level visual foundation model based on IPT, which attains a PSNR comparable to the full-precision model while offering a $7.99\times$ compression ratio in the $\times4$ super-resolution task on the Set5 benchmark.

Decision: accept (poster)
Reviewer scores: 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730502870699

thread_type:
official_review

<thread_object>
creator: reviewer_icgh

<content>
summary:
This paper presents a novel method for multi-task network quantization-aware training, addressing performance degradation caused by model quantization. The approach incorporates two techniques: (1) Task-Specific Learnable Multi-Scale Activation Quantizer (TLMAQ): Addressing the scale conflict in quantizing diverse task features, improving the quantized model's representational capabilities across tasks. (2) Structure-based Layer-by-Layer Distillation (SLLD): Strengthening full-precision model supervision over quantized models, reducing information distortion from quantization.

strengths:
(1) The proposed method effectively addresses the scale discrepancies among multi-task features, aligning with intuitive expectations.
(2) The novelty is good, the introduction of task-aware quantization into multi-task learning represents an innovative approach.
(3) Method is architecture-agnostic, featuring a wide range of applicability.
(4) originality, quality, clarity, and significance: This paper demonstrates good originality, is well-written, and clearly communicates ideas. Furthermore, the application of quantization to enhance model efficiency holds significant importance for multi-task learning

weaknesses:
(1) There is a minor error in Section 2.1 - regarding the order of description for the "single-modal task-related data scenario" and the "multi-modal data scenario" are misaligned with the order of "former scenario" and "latter scenario". 
(2) The paper only considered the low-level multi-task scenarios; however, However, it lacks effective exploration of high-level visual tasks. Specifically, Can the proposed method still maintain effectiveness when processing high-level visual multitasking data such as NYUD-v2, Pascal Context and Cityscapes.
(3) The experiment involved a comparison of model quantization techniques, including LSQ (2019) and PAMS (2020). Nevertheless, a comparison with the most recent Quantization-Aware Training (QAT) methods is conspicuously absent.

questions:
Apart from the points (2) and (3) outlined in the "Weaknesses" section, the following are additional concerns:
(1) While the model design is good and interesting, the model seems a little complex. Would it be possible to provide some comparative analyses regarding the model's speed and complexity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730514522938

thread_type:
official_review

<thread_object>
creator: reviewer_gsjw

<content>
summary:
This paper proposes Task-Specific Scales Quantization for Multi-Task Co-Training (TSQ-MTC) to address the performance degradation issue of existing quantization-aware training (QAT) methods when integrated with co-training. The proposed method introduces a task-specific learnable multi-scale activation quantizer (TLMAQ) to enrich the representational ability of shared features across different tasks and a structure-based layer-by-layer distillation (SLLD) to ensure that the quantized features effectively preserve the information from their full-precision counterparts. Extensive experiments on two co-training data scenarios demonstrate the effectiveness of TSQ-MTC, which achieves a 4-bit quantized low-level visual foundation model based on IPT with a PSNR comparable to the full-precision model and a 7.99× compression ratio in the ×4 super-resolution task on the Set5 benchmark.

strengths:
1. The authors provide a comprehensive evaluation of the challenges of the task, i.e., directly integrating multi-task co-training with QAT. This helps clarify the bottleneck of existing QAT methods and motivates the proposed TSQ-MTC method to address the performance degradation issue.

2. The proposed TSQ-MTC method introduces two novel components, TLMAQ and SLLD, to enhance the representational ability of shared features across different tasks and preserve the information from full-precision features in deeper layers of the Transformer model. 

3. The experimental evaluation across the main text and appendices provides detailed insights into the effectiveness and versatility of the proposed TSQ-MTC method in two co-training data scenarios.

weaknesses:
1. This paper referred the challenge of computational and memory overhead of co-trained models (Lines 047-049), but the computational complexity and efficiency of the proposed TSQ-MTC method are not discussed in detail. 

2. The authors detailedly analysis the proposed SLLD in Table 4 and Figure B. However, according to the ablation studies provided in Table 3, Table D, Table E, the proposed SLLD has minor improvements in performance given the potential for minor fluctuations in experimental results.

questions:
As listed in Weaknesses, there are two questions:
1. Could the authors provide the computational cost of training and inference compared to existing QAT methods?
2. Could the authors report the variance of the results of the proposed SLLD method in the ablation studies?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730639148872

thread_type:
official_review

<thread_object>
creator: reviewer_lorg

<content>
summary:
The study finds that directly applying co-training to existing QAT methods significantly degrades performance. The main issue identified is the inadequacy of activation quantization scales in the co-training framework. To address this, the authors propose a Task-Specific Scales Quantization method suitable for multi-task co-training.

strengths:
1. This work effectively incorporates quantization-aware training into co-training and significantly reduces the performance gap between multi-task co-trained models and their 4-bit quantized counterparts.
2. The authors design task-specific learnable multi-scale activation quantizer and SLLD to solve the issues of naive integration.
3. From the experimental results, it appears that the author's techniques are effective.

weaknesses:
1. In table 1, quantitative results for super-resolution tasks are shown. But I am still curious about the results of deraining and denoising tasks.
2. How about the parameters change of your method?
3. Can you provide the some comparisons with some SOTA single task methods to further demonstrate the superiority of your method?

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732519223547

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Paper Revision

comment:
We sincerely thank all the reviewers for their constructive feedback and kind support. \
We have revised our submission based on the provided suggestions, with the updated content highlighted in blue.

Specifically, some key modifications are as follows:
- At the beginning of the experimental section, we have outlined the organization of experiments in the main paper and appendix to facilitate locating all key details (e.g., the parameter count and FLOPs comparison). (lines 397–402)
- We have referenced supplementary experiments from the appendix in the main paper (e.g., denoising and deraining experiments). (lines 429–431)
- We have corrected the writing error pointed out by Reviewer icGh. (lines 136–138)
- We have included the experimental results of Q-ViT in both the main paper and the appendix. (Table 1, Table A, and Table B)

We have adjusted the font size of the tables in the manuscript to comply with the maximum page limit while ensuring that all content remains unchanged, except for the revised sections. Additionally, we have polished some writing mistakes in the manuscript. We hope that the revised version meets your expectations.
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this paper, the authors presented a new low-bit quantization method for multi-task co-training. Specifically, performance degradation was observed from the study, which identified the primary issue with existing quantization-aware training (QAT) methods. Following that, the authors propose task-specific scales quantization for multi-task co-training (TSQ-MTC) to address the mismatched quantization scales. A task-specific learnable multi-scale activation quantizer and a structure-based layer-by-layer distillation were introduced. Experimental evaluation showed the effectiveness of the proposed method. The main strengths of this paper are:
- The proposed method effectively addresses the scale discrepancies among multi-task features, incorporates quantization-aware training into multi-task co-training, and significantly reduces the performance gap. The approach also intuitively makes sense.
- Novel approaches were introduced to address the studied problem. Specifically, the above-mentioned learnable quantizer and distillation components enhanced the representational ability of shared features across different tasks.
- Comprehensive experimental evaluation was provided, showing the effectiveness of the proposed method, and providing insights for following research in this direction.
- The paper is well-written, well-motivated, and easy to follow.

The main weaknesses of this paper include:
- Some concerns about the applicability of the proposed method, e.g. to other tasks such as draining and denoising, comparison to single task methods, and high-level visual tasks.
- The performance improvement of the proposed structure-based layer-by-layer distillation is incremental.
- Missing comparison to related work, e.g. most recent QAT methods.

During the rebuttal phase, most of the major concerns were well addressed. Considering the above contributions introduced by this paper, the AC is happy to recommend an Accept, but recommends the authors incorporate all the further provided evidence and clarifications into the final version.