Title: PMQ-VE: Progressive Multi-Frame Quantization for Video Enhancement
Venue: NeurIPS 2025
Accepted: yes
Keywords: quantization

deep learning
Abstract: Multi-frame video enhancement tasks aim to improve the spatial and temporal resolution and quality of video sequences by leveraging temporal information from multiple frames, which are widely used in streaming video processing, surveillance, and generation. Although numerous Transformer-based enhancement methods have achieved impressive performance, their computational and memory demands hinder deployment on edge devices. Quantization offers a practical solution by reducing the bit-width of weights and activations to improve efficiency. However, directly applying existing quantization methods to video enhancement tasks often leads to significant performance degradation and loss of fine details. This stems from two limitations: (a) inability to allocate varying representational capacity across frames, which results in suboptimal dynamic range adaptation; (b) over-reliance on full-precision teachers, which limits the learning of low-bit student models. To tackle these challenges, we propose a novel quantization method for video enhancement: Progressive Multi-Frame Quantization for Video Enhancement (PMQ-VE). This framework features a coarse-to-fine two-stage process: Backtracking-based Multi-Frame Quantization (BMFQ) and Progressive Multi-Teacher Distillation (PMTD). BMFQ utilizes a percentile-based initialization and iterative search with pruning and backtracking for robust clipping bounds. PMTD employs a progressive distillation strategy with both full-precision and multiple high-bit (INT) teachers to enhance low-bit models' capacity and quality. Extensive experiments demonstrate that our method outperforms existing approaches, achieving state-of-the-art performance across multiple tasks and benchmarks. The code will be made publicly available.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Backtracking-based Multi-Frame Quantization, a simple but effective framework that addresses the limitations of traditional global quantization in multi-frame video enhancement. Besides, the authors propose PMTD, a method that leverages knowledge from multi-level teacher models to guide a low-bit student model, thereby improving its mapping quality and overall performance. The structure of the paper is clear, the motivation is explicit, and the experimental results demonstrate the effectiveness of the two innovations proposed by the authors.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper proposes a two-stage PMQ-VE framework and progressive multi-teacher distillation (PMTD) to address inter-frame distribution differences and capacity gaps in multi-frame video enhancement.\n2. The percentile-based initialization and backtracking search efficiently optimize frame-specific bounds with negligible overhead.\n3. Extensive experiments across STVSR, VSR, and VFI demonstrate their effectiveness.\n\nWeaknesses:\n1. The study lacks analysis of computational trade-offs across different quantization levels (e.g., 2-bit vs. 4-bit) for practical edge deployment scenarios.\n\nQUESTIONS:\n1. Instead of searching for the optimal upper and lower bounds for each frame, searching every few frames (because the spatial information of adjacent frames of the video is extremely close) can we further reduce the computational complexity of the method?\n2. What is the impact of varying the number of intermediate-bit teachers (e.g., using 6-bit alongside 8-bit) in the PMTD strategy on low-bit model performance?\n3. How does the percentile-based initialization in BMFQ specifically suppress outliers, and what percentile ranges are experimentally validated as optimal?\n\nLIMITATIONS:\nNot applicable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes PMQ-VE, a novel two-stage quantization framework for multi-frame video enhancement tasks. PMQ-VE introduces Backtracking-based Multi-Frame Quantization to search frame-specific clipping bounds via percentile initialization and backtracking search, and Progressive Multi-Teacher Distillation to leverage full-precision and intermediate-bit teachers for knowledge transfer. Experiments show PMQ-VE outperforms SOTA methods across benchmarks under low-bit quantization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: 1. The novel BMFQ and PMTD strategies address the challenges in video enhancement quantization. 2. The authors conducted comprehensive experiments and ablation studies on different tasks and benchmarks, which demonstrates quantitative and qualitative improvements.\n\nWeakness\n1.The main problem is the clarity of BMFQ implementation. The motivation of BMFQ is also not well introduced. The paper lacks explanation on whether BMFQ is applied during training or testing, and how frame-specific bounds are generalized to unseen test frames (if BMFQ is applied to the training set). This ambiguity causes lots of confusion about the practical deployment workflow (e.g., if per-frame boundary search is required for each test input, it may introduce considerable inference computational cost; if the per-frame boundary search is done for each training input, the parameters would not be suitable for the test frames).\n2. Besides, the backtracking search in BMFQ is described as \"negligible overhead,\" but the paper does not provide concrete data (e.g., training/inference time increase, memory usage) to support this claim, which is critical for evaluating practical efficiency.\n\nQUESTIONS:\n1. The main problem is: Is BMFQ performed during the training phase (i.e., QAT, to learn frame-specific bounds for the training set) or during inference for each test frame (i.e., PTQ)? I suppose BMFQ might be a QAT strategy, but the authors have not explicitly pointed this out. If it is training-phase only, it seems that you are overfitting lb and ub parameters for each specific frame in the training set, then how do the learned bounds generalize to unseen test frames with different distributions? If it's done at test phase, it means you are quantizing a full-precision model during the test phase. Since this full-precision model, which can lead to better image quality than the quantized model, can be used for inference during testing, then what's the point and meaning of doing quantization? Additionally, if it's done at test phase, how much is the computational cost? Please clarify the workflow.\n\n2. The paper claims BTBI has \"negligible overhead,\" but please provide quantitative comparisons (e.g., training iterations, GPU memory usage, inference latency) between PMQ-VE and baseline methods to validate this. If overhead exists, how does it impact edge deployment scenarios where efficiency is critical?\n\n3. In PMTD, how are the intermediate-bit teachers trained? Are they quantized from the full-precision model using the same BMFQ strategy, or via other methods?\n\nI'm not sure if my understanding is entirely correct so far. If the authors can address my concerns well, I might increase my evaluation score; otherwise, I will still tend to reject the manuscript\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a multi-frame video enhancement method relying on two underlying concepts - a)Backtracking based quantization - involves frame based adaptive quantization where the quantization bounds are obtained adaptively for every frame using backtracking, b) Progressive multi-teacher distillation - this framework aims to reproduce representational capacity of full-precision teacher by employing a progressive training scheme with intermediate 8/4-bit teacher. Extensive experiments across 3 tasks : Video Frame Interpolation (VFI), Video Super-Resolution (VSR), and Spatio-Temporal Video Super-Resolution (STVSR) are performed. The results show improvement over existing state-of-the-art low-bit models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The idea of adaptively modifying quantization for each frame using backtracking is a novel idea and hasn't been explored earlier. Also from the ablation experiments it is evident that this provides the highest performance boost.\n\n2. Progressive quantization is a simple but powerful idea where distillation is done to very low-bits in a progressive manner. From the results this also leads to better performance. \n\n3. The paper is presented in an elegant manner and the algorithm is easy to comprehend. Overall the proposed technique improves upon the existing state-of-the-art and is generalizable across different video enhancement tasks as well as can be applied across different deep learning models.\n\n\nWeaknesses\n1. The quantization scheme proposed involves a recursive backtracking strategy performed on a frame level manner. This could be compute heavy for videos with large number of frames and/or with high resolution. Also authors haven't reported the computational complexity of their method, so hard to understand how different video characteristics impact on the compute.\n\n2. Although the authors propose the above method for video enhancement there isn't any specific information associated with enhancement (except it is applied for each frame) that is employed in the design. Perhaps the same design can be applicable to other tasks as well? For example in videos there is significant overlap between temporal information between successive frames. This could be employed to reduce the recursive search as doing backtracking based search for every video frame could be wasteful as well as expensive.\n\nQUESTIONS:\n1. Although the backtracking based quantization is proposed for frame based manner from the implementation details it appears to be done at a crop/patch level. Just wondering if spatial dimensions have any impact? Additionally if instead of doing backtracking in a patch-based manner instead of frame level are there any pros/cons associated with it?\n\n2. What is the average recursion depth observed in the training? Can that be reduced by leveraging certain assumptions such as the motion does not change significantly between consecutive frames?\n\nLIMITATIONS:\nAs the authors have pointed out, diffusion models aren't included in this work. It will interesting to know how they perform with this technique.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTransformer-based video enhancement models are too resource-intensive for edge deployment, while traditional quantization methods fail because they cannot handle varying data distributions across frames or the large representation gap between teacher and student models. To solve this, this paper proposes a novel two-stage method PMQ-VE, containing the Backtracking-based Multi-Frame Quantization (BMFQ) and the Progressive Multi-Teacher Distillation (PMTD). Experiments show that PMQ-VE achieves state-of-the-art performance on multiple benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. This paper investigates and solves the problem of quantizing multi-frame video enhancement models. This is a practical and important problem, as the efficient deployment of video models is a key bottleneck for many real-world applications. \n\n2. The proposed PMQ-VE framework has two core components, BMFQ and PMTD, effectively addressing the corresponding challenges.\n\n3. The results consistently demonstrate the superiority of the proposed method, with significant gains in PSNR/SSIM on multiple datasets.\n\n4. The paper is well-written and easy to understand. \n\nWeaknesses:\n\n1. The paper lacks detailed quantitative analysis and proof of the computational cost of proposed modules such as BTBI, and lacks comparison with other baseline methods.\n\n2. All the experimental results reported in this paper are the results of a single experiment, rather than the standard deviations and means of multiple experiments. While the performance gains are substantial, reporting statistics over multiple independent experiments can demonstrate the stability and reliability of the results.\n\n3. The proposed framework introduces new hyperparameters. However, the paper does not provide a sensitivity analysis for these parameters, making it difficult to assess the method's robustness or how to tune them for new scenarios.\n\nQUESTIONS:\n1. Could you further analyze and discuss the computational overhead of the BTBI module, particularly in comparison to baseline methods?\n\n2. Could you comment on the stability and reliability of the results, and elaborate on why the substantial performance gains are reported from a single run?\n\n3. Could you elaborate on the selection of the new hyperparameters and discuss the expected sensitivity of the framework's performance to their variations?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}