Group-aware Parameter-efficient Updating for Content-Adaptive Neural Video Compression

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-06 14:59:22

Abstract: Content-adaptive compression is crucial for enhancing the adaptability of the pre-trained neural codec for various contents. Although these methods have been very practical in neural image compression (NIC), their application in neural video compression (NVC) is still limited due to two main aspects: 1), video compression relies heavily on temporal redundancy, therefore updating just one or a few frames can lead to significant errors accumulating over time; 2), NVC frameworks are generally more complex, with many large components that are not easy to update quickly during encoding. To address the previously mentioned challenges, we have developed a content-adaptive NVC technique called Group-aware Parameter-Efficient Updating (GPU). Initially, to minimize error accumulation, we adopt a group-aware approach for updating encoder parameters. This involves adopting a patch-based Group of Pictures (GoP) training strategy to segment a video into patch-based GoPs, which will be updated to facilitate a globally optimized domain-transferable solution. Subsequently, we introduce a parameter-efficient delta-tuning strategy, which is achieved by integrating several light-weight adapters into each coding component of the encoding process by both serial and parallel configuration. Such architecture-agnostic modules stimulate the components with large parameters, thereby reducing both the update cost and the encoding time. We incorporate our GPU into the latest NVC framework and conduct comprehensive experiments, whose results showcase outstanding video compression efficiency across four video benchmarks and adaptability of one medical image benchmark.

Main paper: 
 introduction: 
 Video compression, a cornerstone of multimedia computing, has been an essential technology for decades. This importance is driven by the need to efficiently store and transmit an ever-growing variety of video content. However, conventional video compression standards (e.g., H.266/VVC [6] ) rely on manually designed modules that may no longer suffice for contemporary requirements. In recent times, Neural Video Compression (NVC) techniques [27] that are optimized end-to-end with advanced neural modules have reached performance levels comparable to those of traditional codecs.Despite achieving significant success, there are still some limitations to NVC in practical coding applications. The primary challenge is generalization, as most existing NVC models struggle to perform adaptively across different sequential contents. This issue arises because most pre-trained NVC models are optimized for a specific expected rate-distortion (RD) cost within a certain domain (e.g., online video resource [47] ), leading to a notable domain gap when applied to cases outside the training domain (e.g., medical volumetric images). To mitigate the issue of such domain gaps, several content-adaptive methodologies [7, 12, 22, 24, 35, 38, 42, 43, 45, 46, 49, 51, 54] have been introduced in the field of neural image compression (NIC). Many approaches involve updating the parameters of the encoder neural modules (i.e., online-updating strategy) to generate quantized compressed features with improved rate-distortion values. This enhancement significantly improves compression performance in testing image scenarios.Conversely, adopting the direct migration of update strategies from NIC to NVC frameworks still seems impractical. A primary obstacle lies in the evolution of NVC architectures, which now incorporate extensive and complex modules [9, [25] [26] [27] with millions of parameters on the encoding side. Some early studies [31] have explored online-updating all parameters on the encoding side mirroring strategies used in content-adaptive NICs. Though it initially yielded performance gains in early light-weight NVC frameworks [32] , applying such updates to the intricate architectures of contemporary NVCs will result in prohibitively long encoding time and immense computational demands. Recent investigations [41] have also explored updating solely the motion to circumvent the necessity of optimizing all encoding-side parameters. However, given that motion usually represents just a minor portion of the total bit-rates, confining updates to merely motion components substantially limits the scope for performance enhancements.Another challenge limiting the effectiveness of content-adaptive NVC is error accumulation. Video compression heavily relies on temporal redundancy. i.e., the compression efficiency of the current frame is greatly dependent on the reconstruction quality of the previously compressed frame. However, the online updating strategy, which aims to optimize the rate-distortion value by minimizing both bit-rate and reconstruction quality simultaneously, can negatively impact the quality of subsequent coding frames through increased error accumulation. This dynamic underlines the difficulty of achieving both high compression efficiency and maintaining quality across frames.To tackle the aforementioned challenges, we propose a novel content-adaptive NVC method, termed Group-aware Parameterefficient Updating (GPU). First, leveraging the breakthroughs in Parameter-Efficient Transfer Learning (PETL) [10, 14, 16] across various vision and language tasks, which demonstrates the efficiency of adapting a pre-trained model to new domains by fine-tuning a minimal set of parameters, our approach incorporates a low-rank adaptation style [17] module, namely encoding-side adapters. These adapters are ingeniously crafted to stimulate specific components within the comprehensive encoding modules. By focusing optimization efforts on these adapters, which constitute a significantly smaller portion of the encoding modules' parameters, we can then streamline optimization and reduce costs dramatically. Moreover, we delve into innovative ways to integrate these adapters into the NVC framework, exploring both serial and parallel configurations. The serial adapter is strategically placed immediately after a module, enhancing its processing, while the parallel adapter is woven into the module through a skip-connection layout, offering a blend of original and adapted processing pathways. This strategic placement of adapters allows for the modification of less than 10% of the parameters in our encoding module, markedly boosting compression efficiency with minimal modification.Second, to mitigate error accumulation, we introduce a groupaware optimization strategy. This method diverges from previous methods [3, 31, 41] , which concentrated on updating a narrow sequence of consecutive frames. Instead, our approach considers the entire Group of Pictures (GoP) during the updating process. Specifically, when compressing an individual frame, we optimize the entire GoP, which it belongs, to minimize the errors in individual reconstructed frames. Furthermore, to circumvent the significant memory demand that group-based updating could impose on the processing of high-resolution videos, we implement a patch-based GoP updating strategy. It spatially segments each GoP into multiple patch-based GoPs, which will be updated sequentially. In general, this approach ensures that the updates made to this framework do not adversely impact the predictive coding process of other frames within the GoP, thus maintaining the integrity and efficiency of the compression across the entire group.By integrating two above strategies, we incorporate our GPU into the latest NVC framework [27] , significantly improving its coding performance across four most representative video compression benchmarks, HEVC [40] Class B, C, D and E datasets. Moreover, to validate the adaptability of our content-adaptive method in various contents, we also test it on compressing medical volumetric image benchmarks ACDC [5] , which serves the most important cardiac dataset with multiple MRI slices. Our framework not only outperforms traditional video codecs, such as H.266/VVC [6] , but also surpasses previous content-adaptive NVC methods [31, 41] in performance on video benchmarks and adaptability on medical benchmark. Our contributions can be summarized as follows:• Introduction of encoding-side adapters inspired by Parameter-Efficient Transfer Learning, optimizing a minimal set of parameters by using low-rank adaptation strategy to adapt pre-trained models to new domains efficiently.• Implementation of a group-aware optimization strategy to mitigate error accumulation by considering an entire Group of Pictures (GoP) during online updating, ensuring consistent and improved predictive coding across frames. • Integration of our approach to enhance the coding performance of the latest NVC methods, demonstrating superior results over both traditional video codecs and existing content adaptive NVC methods across four key video and one medical volumetric image compression benchmarks. 
related works 2.1 neural video compression: 
 Current methods in neural video compression can be intuitively categorized into two main types: deep residual coding and deep contextual coding. The first-category approach is inspired by established video compression standards [6, 40] , which involves predictive coding (e.g., motion compensation) and the encoding of residual data. A notable advancement in this category is DVC [32] , which utilizes Convolutional Neural Networks (CNNs) throughout the traditional residual coding process. Following this, numerous works [4, 8, 19, 21, 29, 48] have enhanced this framework by incorporating more robust modules and sophisticated approaches. For instance, FVC [21] implements all coding operations in the feature space to improve compression efficiency.In the deep contextual coding category [9, 13, 25-28, 30, 34, 39] , research extends generative-based neural image compression (NIC) techniques by developing spatio-temporal conditional entropy models that leverage both spatial and temporal contexts. In the early studies, some works [13, 30] have introduced spatial-temporal conditional models by maintaining a latent temporal variable produced from all previous compressed frames. Unlike earlier approaches that rely on accumulating all available information, DCVC family [25] [26] [27] 39] introduces a series of methods that employ motion compensation from the consecutive compressed frame to derive temporal context directly, which has been further refined by progressive introducing new methods enhance such as multi-scale temporal context mining [39] or hybrid entropy model [26] . The latest iteration of DCVC_DC has shown to surpass the performance of the conventional video coding standard H.266/VVC [6] .Though above methods have achieved notable success, their performances are often constrained across diverse video content. This limitation stems primarily from the fact that many NVC frameworks are pre-trained on specific domains [47] , which restricts their generalizability and adaptability to a wide range of content. To address this challenge, this work introduces a content-adaptive NVC method named GPU, which is seamlessly integrated into the latest NVC framework, DCVC_DC. This integration significantly enhances its compression performance, demonstrating the potential of content-adaptive approaches in overcoming the generalizability limitations of existing NVC frameworks.Content-adaptive methods play essential roles in both NIC [7, 12, 22, 24, 35, 38, 42, 43, 45, 46, 49, 51, 54] and NVC [3, 31, 41] , which is to refine the latent representation produced by encoder side during encoding process. In this way, the model can adapt to current coding samples and reduce the domain shift between training and testing domains, which enhances the adaptability when compressing various types of testing data.Those approaches have yielded significant accomplishments in NIC. For instance, Campos et al. [7] refined latent variables through back-propagation, a technique that was further enhanced by Yang et al. [49] through the incorporation of a differentiable quantization module. Concurrently, some researchers have pursued decoderbased strategies; for example, Rozendaal et al. [43] opted to update all parameters within the decoder and the entropy model. Nonetheless, it's important to note that these decoder-focused methods typically demand higher bit-rates for the storage and transmission of parameters to the decoder.In the field of NVC, the development of content-adaptive strategies is still in its infancy. Lu et al. [33] introduced the first method of this kind by updating all parameters on the encoding side, drawing parallels with most content-adaptive approaches in NIC. This method was seamlessly integrated into early NVC framework [32] , which typically featured lightweight architectures. However, as more advanced modules(e.g., Transformers [44] ) are incorporated to enhance the performance, modern NVC frameworks [9, 20, [25] [26] [27] [28] have become significantly more complex. Updating the entire encoder is now impractical due to the substantial computational load and increased encoding time. To address this issue, Tang et al. [41] developed a novel methodology for online and offline updating, specifically targeting the compression of the motion. This approach requires motion information from H.266/VVC [6] as the auxiliary ground-truth. Yet, since motion data represents only a small fraction of the overall bit-rates, optimizing motion alone is insufficient for significant performance gains.Additionally, a key difference between NIC and NVC lies in video compression's dependence on the reconstruction quality of preceding frames. The compression of any current-coding frame relies heavily on the quality of its reference frame. Updating only a single frame or a small subset, as suggested in certain studies [3, 41] , may lead to cumulative errors in subsequent frames.In contrast, our method employs a group-aware and parameterefficient updating strategy to overcome the aforementioned challenges. First, we jointly optimize the entire GoP when generating the bitstream for a single frame, which ensures that updating a particular frame does not lead to significant error accumulation. Secondly, we implement an encoder-side adaptor, which allows us to avoid optimizing the entire encoder network with those comprehensive modules. This adaptation makes the updating process more efficient in terms of parameter use.PETL [10] was initially developed to enhance the training efficiency of natural language processing methods. It has since garnered significant attention due to the rapid development of vision and language task [14, 16, 52] . This approach has become increasingly popular as it offers a performance nearly on par with that of full fine-tuning, strategy making it a trendy strategy in contemporary research.Our PETL strategy is closely aligned with the concept of Low-Rank Adaptation [11, 14, 15, 17, 23, 36, 52] , which are recognized for their practicality and efficiency in delta-tuning methods by using adaptors. Various modifications to adaptors have been explored. These include adjustments in the placement of adaptors within the model [14, 52] , the application of targeted pruning techniques to enhance model efficiencys [15] , and the adoption of reparametrization strategies to optimize performance [11, 23] . Such innovations demonstrate the adaptability and versatility of adaptors in enhancing task-specific model performance.To the best of our knowledge, our work represents the pioneering effort to integrate the PETL strategy within the NVC framework. Specifically, we introduce two distinct adaptor configurations: parallel and series adaptors. The parallel adaptor is designed for use with large-parameter modules, such as the motion and contexture feature encoding network. Conversely, the series adaptor is tailored for light-weight modules, including motion estimation, motion encoding, motion hyper-prior encoding and contexture hyper-prior encoding modules. These dual-adaptor styles significantly enhance our ability to optimize the NVC process during encoding. 
methodology 3.1 gpu: 
 3.1.1 Patch-based GoP Updating. To compress a specified video sequence X = {𝑋 1 , 𝑋 2 , . . . , 𝑋 𝑡 -1 , 𝑋 𝑡 , . . . }, we initially divide it into 𝐾 non-overlap groups of pictures (GoP), each consisting of 𝑇 frames, denoted as G 𝑘 = {𝑋 𝑡 , 𝑋 𝑡 +1 , . . . , 𝑋 𝑡 +𝑇 -1 }. This process results in X being represented as X = {G 1 , G 2 , . . . , G 𝐾 }. In this work, we adopt the widely used predictive-frame (P-Frame) coding approach in NVC, as referenced in [9, 18, 20, 21, [25] [26] [27] 53] . We designate the first frame in each GoP as an intra-coding frame (I-Frame) and the subsequent frames as P-Frames, following a pattern of {𝐼𝑃𝑃𝑃𝑃 • • • } and only apply our updating mechanism upon P-frames.Require:Data X = {G 1 , G 2 , . . . , G 𝐾 }; Iteration 𝑀𝐴𝑋 ; Network 𝑓 (•); Updating Parameters 𝜃 𝑢 ; Frozen Parameters 𝜃 𝑓 return The Compressed Data X = { Ĝ1 , Ĝ2 , . . . , Ĝ𝐾 } for G 𝑘 in X do segment G 𝑘 into {G 1 𝑘 , G 2 𝑘 , • • • G 𝑁 𝑘 } for 𝑆𝑇 𝐸𝑃 in 𝑀𝐴𝑋 do for G 𝑖 𝑘 in G 𝑘 do 𝜃 𝑖 𝑢 = 𝑎𝑟𝑔𝑚𝑖𝑛 𝜃 𝑖 𝑢 L (𝑓 (G 𝑖 𝑘 , 𝜃 𝑢 , 𝜃 𝑓 ), G 𝑖 𝑘 ) 𝜃 𝑢 = 𝜃 𝑖 𝑢 end for end for Ĝ𝑘 = 𝑓 (G 𝑘 , 𝜃 𝑢 , 𝜃 𝑓 ) end forOur goal is to perform our online updating mechanism, GPU, for each GoP G 𝑘 to minimize error propagation. However, directly updating G 𝑘 poses significant challenges due to the potentially high resolution and large number of frames involved, leading to substantial memory and computational demands. To address this, we spatially partition G 𝑘 into 𝑁 patch-based GoPs G 𝑖 𝑘 , each located at a specific spatial position 𝑖. This allows us to apply the updating mechanism on each G 𝑖 𝑘 individually on the GPU, using the objective function L, which will be elaborated in Section 3.2.2. After updating all patch-based GoPs G 𝑖 𝑘 , we proceed to compress the entire GoP G 𝑘 into a bit-stream utilizing our NVC framework with the newly updated parameters on the encoder side. This process is iteratively carried out for all G 𝑘 s. We demonstrate this algorithm 1 and its implementation details in Fig. 1 .3.1.2 Encoder-side Adaptor. Existing content-adaptive NVC [32, 41] methods often update all parameters in the encoding modules directly. While this full-updating strategy may naturally improve performance, it also significantly increases computational costs. Such a strategy becomes impractical for modern NVC systems [27] , which typically involve a vast number of parameters.To enhance our updating process's efficiency, we have implemented the adaptor mechanism, a technique widely used in PETL methods [17] . As shown in Fig. 2 (c ), our approach 𝛿 (•) utilizes a streamlined architecture comprising merely three convolutional layers W and two LeakyReLU activation layers 𝜎 (•). This structure is designed to process input features, denoted as F 𝑖𝑛 ∈ R 𝐶 𝑖𝑛 ×𝐻 𝑖𝑛 ×𝑊 𝑖𝑛 and generate updated features F ∈ R 𝐶 ×𝐻 ×𝑊 using adaptors. Initially, we apply a 1 × 1 convolutional layer for pre-processing, characterized by weights W 𝑝𝑟𝑒 ∈ R 𝐶 𝑖𝑛 ×𝐶 𝑑𝑤 ×1×1 to reshape the feature into a channel-squeezed form F ∈ R 𝐶 𝑑𝑤 ×𝐻 ×𝑊 s.t., 𝐶 𝑑𝑤 < 𝐶 𝑖𝑛 & 𝐶 𝑑𝑤 < 𝐶. Following the strategy commonly employed in Low-Rank Adaptation methods [17, 38] , we then apply a depth-wise convolution, with weights W 𝑑𝑤 ∈ R 𝐶 𝑑𝑤 ×𝑀 ×𝑀 and a kernel size of 𝑀 to produce the feature F ∈ R 𝐶 𝑑𝑤 ×𝐻 ×𝑊 . Finally, we employ a 1 × 1 (i.e., point-wise) convolutional layer with zero-initialized weights, W 𝑧𝑒𝑟𝑜 ∈ R 𝐶 𝑑 𝑤 ×𝐶 ×1×1 inspired by [50] . Initially, this results in the delta feature 𝛿 (F 𝑖𝑛 ) having no impact on the feature F 𝑖𝑛 . Over the updating process, the impact of 𝛿 (F 𝑖𝑛 ) will incrementally increase from zero, optimizing the parameters to enhance the overall model performance. The entire updating can be concisely represented by the following equation, where × and ⊗ denote the convolutional and depth-wise convolutional operations:EQUATIONThe configuration of adaptors plays a crucial role in our approach. Drawing inspiration from previous studies that highlight the importance of strategical adaptors' placement [14, 38, 52] , we explore the implementation of adaptors within various coding components in two distinct configurations: parallel and serial placement. Specifically, for complex coding modules (e.g., the contextual feature encoder), that incorporate multiple sophisticated convolutional blocks (e.g., residual blocks) with a large number of parameters, we insert our adaptors in parallel within the coding component. These are placed next to convolutional blocks to generate a delta feature 𝛿 (F). This setup is illustrated in Fig. 2 (a) , where the adaptors work alongside convolutional blocks to enhance the feature representation. It can be summarized as:EQUATIONwhere W 𝑐𝑜𝑑𝑖𝑛𝑔 represents the weights with frozen parameters of coding components from the original NVC framework. Conversely, for lighter coding modules (e.g., the hyper-prior encoder), employing parallel insertion for each internal convolution block may not significantly reduce parameters. Therefore, we opt for a serial placement of the adaptor. This approach involves positioning the adaptor after all the smaller convolutional blocks and outside the coding component. Such a configuration is depicted in Fig. 2 (b) , where the adaptor acts sequentially, enhancing the module's output by processing the aggregated features from the preceding convolution blocks. It can be summarized as: Modules that are also relevant to the decoding phase (white) remain unaltered. The adaptors are integrated within these encoding-specific modules, allowing for the freezing of existing module parameters while updating those within themselves. Step 1. Motion Estimation and Compression. We first estimate the raw motion 𝑉 𝑡 between reference frame (i.e., previously reconstructed frame) X𝑡 -1 and current-coding frame 𝑋 𝑡 through the Motion Estimation operation, where we directly adopt a compact optical flow network SpyNet [37] as in [9, [25] [26] [27] 32] . Following this, we employ an auto-encoder-style module, Motion Encoder, to compress 𝑉 𝑡 into the motion feature M 𝑡 , followed by quantizing it to M𝑡 . To losslessly transmit M𝑡 , we need to adopt arithmetic encoder (resp., decoder) to losslessly encode to (resp., decode from) the bit-stream. To minimize the bit-rates, we employ a hyper-prior-based entropy model as in [27] , which enhances the estimation of M𝑡 's distribution and improves the efficiency of cross-entropy coding. Consequently, this phase involves generating a quantized motion hyper-prior feature by using a Motion Hyper-Prior Encoder. In this step, we can update the Motion Estimation, Motion Encoder and Motion Hyper-Prior Encoder online during the encoding procedure, aiming to compress the quantized motion feature M𝑡 in a better way. Specifically, we incorporate serial adaptors for those light-weight coding modules, Motion Estimation and Motion Hyper-Prior Encoder, while for more complex coding modules, Motion Encoder, parallel adaptors are employed to facilitate improvements.EQUATIONStep 2. Motion Reconstruction and Compensation. Using the transmitted quantized motion feature M𝑡 , we can proceed with motion compensation operation. This process begins with the reconstruction of M𝑡 into the reconstructed motion V𝑡 by employing Motion Reconstruction modules that consist of several standard convolutional blocks. Once we have V𝑡 , the motion compensation procedure is carried out to generate warped features F𝑡 , effectively reducing temporal redundancy. In this work, we apply a multi-scale temporal context extraction strategy, directly following [39] to obtain multiple scales of temporal context information F𝑡 . Since the modules in this phase are integral to both the encoding and decoding stages, we do not update them during the encoding process.Step EQUATIONwhere 𝐷 (•) represents the distortion, 𝑅(•) represents the bit-rate cost and we use 𝜆 as a hyper-parameter to control the trade-off between rate and distortion during optimimization. 
experiments 4.1 experimental protocols: 
 4.1.1 Datasets. To evaluate the effectiveness of our approach, we implemented it on video sequences from four of the most representative video compression datasets: HEVC [40] Class B, C, D, and E. These datasets are selected by the standardization committees ISO/IEC and ITU-T and are extensively utilized in the assessment of new video coding technologies. They encompass videos with resolutions ranging from 416×240 to 1920×1080 featuring a diverse array of video scenes. Furthermore, we assessed the adaptability of our method using the most widely used cardiac medical MRI sequence dataset, ACDC [5] , where we utilized 10 scans from Patient 101, each with a resolution of 256 × 232. We measured the bit-rates by bits-per-pixel (bpp) and assessed the quality of compression by using the peak signal-to-noise ratio (PSNR).Our evaluation aimed to assess the efficacy of our method relative to both traditional video codecs and a range of NVC methods, including standard and content-adaptive variants. For traditional codecs, we utilized the H.265/HEVC [40] and H.266/VVC [6] standards, employing their reference software versions HM-16.20 [1] and VTM-11.2 [2], respectively. Our analysis also included two leading NVCs as baselines: DCVC [25] and its enhanced version, DCVC_DC [27] , with the latter representing the latest advancement in NVC study to the best of our knowledge. It surpasses traditional video codecs in performance, sharing the high-level syntax with DCVC but featuring a more intricate architecture. Additionally, we examined two content-adaptive NVC methods: one by Lu et al. [33] and another by Tang et al. [41] with Tang's method being the most recent development in the field of content-adaptive NVC to the best of our knowledge. To ensure a fair comparison, we recalibrated the performance metrics for DCVC, DCVC_DC, H.266, and H.265 using publicly available codebases and pre-trained models. Our setup involved an I-Frame interval of 32 and the encoding of 96 frames from each video, consistent with [27] . For each MRI scan from the ACDC dataset, we adopted an I-Frame interval of 30 and encoded 30 slices. For the other codecs, due to the unavailability of public implementations, we relied on performance data as reported in their original studies.Effectiveness on Videos. In Fig. 4 and Table 1 , we present quantitative comparisons for rate-distortion and BD-Rate (%) performance of our proposed method compared to existing video compression techniques across four video compression datasets. Our approach demonstrates superior performance over two contemporary content-adaptive NVC frameworks proposed by Tang et al. [41] and Lu et al. [33] , achieving bit-rate reductions of 59.01% and 65.57% on the average. Moreover, our experimental findings reveal a notable average bit-rate savings of 30.69% when compared to the conventional codec H.266/VVC, and a 2.51% improvement over the most recent NVC method, DCVC_DC. Fig. 6 showcases qualitative comparisons among our method, DCVC_DC, and VTM. Notably, our approach achieves superior reconstruction quality, while also utilizing a lower bit-rates for the compression of specific frames (the 80 th and 57 th ) from the "Basket-ballPass" and "BlowingBubbles" sequences. Our method distinctly recovers more structural details, exemplified by the enhanced texture representation in complex areas such as the wall and the tissue box. This level of detail recovery is not observed in the outputs of either DCVC_DC or VTM. Additionally, it has been noted that DCVC_DC tends to introduce blur-related noise artifacts, whereas VTM exhibits ringing artifacts under similar conditions.In terms of MRI sequence compression, shown in Fig. 5 on the ACDC benchmark, our method exhibits a marked enhancement in compression performance. Compared to our baseline method, DCVC_DC, and the conventional video coding standard, H.266/VVC, our approach achieves bit-rate savings of 7.85% and 30.05%, respectively. This notable improvement underscores the adaptability of our method, highlighting its capacity to bridge the domain gap, which is a prevalent challenge in current NVC research, thereby allowing us to extend its applicability across varied domains.. We observed that our method achieves more substantial improvements in medical data than in standard videos. One plausible explanation is that our NVC was pre-trained based on a large-scale video source [47] , which closely resembles the content of our test videos, resulting in a smaller domain gap compared to the medical domain. Though performances in the video benchmarks are nearing saturation due to advancements in NVCs, our method demonstrated superior RD values and enhanced perceptual quality. Most notably, the significant improvements in medical data underscore the versatility and broad applicability of our content-adaptive NVC approach, which is crucial for facilitating compression research across diverse data modalities and applications. We undertook an ablation study utilizing the ACDC dataset on the machine with a single NVIDIA A100 GPU and AMD EPYC 7763 CPU. Here, we execute the online updating of all parameters associated with the encoder side (blue components in Fig. 3 ) within our content-adaptive NVC framework, designated as "Full Updating". Fig. 7 illustrates the RD performance achieved by this full-updating strategy, while Table 2 details the associated computational complexity. Our findings reveal that, although the proposed PETL method incurs a negligible additional bit-rate cost (less than 2%) compared to the full updating approach, they significantly reduce the optimization requirement to less than 10% of the parameters. Moreover, these methods only necessitate 82.1% the optimizing time to train 1 iteration of 1 GoP (i.e., per epoch) required by the full updating strategy. Looking ahead, as more sophisticated techniques are integrated into NVC, it is anticipated that both the number of parameters and the encoding time could further increase, which means our architecture-agnostic updating methods could become increasingly practical.Impact of frame number on online-updating efficiency. We also undertook a comparative analysis on the ACDC dataset to assess the impact of the number of frames updated during the online updating process. Specifically, our default configuration Ground-Truth GPU (Ours) -: 34.66@0.274bpps DCVC_DC -: 34.55@0.288bpps VTM -: 28.49@0.317bpps BlowingBubbles (057)Ground-Truth GPU (Ours) -: 34.07@0.023bpps DCVC_DC -: 33.94@0.025bpps VTM -: 30.25@0.023bpps Table 2 : The computational complexity on ACDC of our method using encoding-side adapters to delta-tune the encoder (i.e., GPU) against the variant directly optimizing all modules within the encoding network (i.e., Full Updating).Full Updating GPU (Ours) #Optimizing Parameters (M) 2.91 0.23 Optimizing Time (s/Epoch) 2.85 2.34 involves optimizing all frames (i.e., 32) within a GoP in one online updating session. This study extends our framework to include variations in the number of updated frames (i.e., 1, 10, and 20), each subset being smaller than the full GoP size. The results in Fig. 8 reveal that while the alternative strategies, which update only 10 and 20 frames, remain managing to improve upon our baseline method DCVC_DC, our default GPU updating the entire GoP outperforms these alternatives by achieving bit-rate savings of 7.98% and 3.76%, respectively. Notably, the variant only updating single frame experienced a considerable decline in performance, incurring an additional bit-rate cost of 21.86% over our GPU. This significant discrepancy underscores the fundamental differences between content-adaptive NIC and NVC, particularly highlighting the challenges associated with accumulating errors over time.  
conclusion: 
 In this research, we investigate innovative methods for contentadaptive NVC. We propose a novel strategy, termed GPU, notable for its patch-based GoP updating mechanism and encoder-side adaptor modules, tailored for diverse configurations. Our extensive experiments validate our GPU's effectiveness and adaptability, demonstrating its integration with contemporary NVC framework for the compression of both standard videos and specialized medical MRI sequences. Remarkably, it consistently surpasses state-of-the-art video compression algorithms, such as H.266/VVC and DCVC_DC, thereby establishing a formidable benchmark for both video and MRI compression. These achievements underscore our GPU approach as a robust and efficient baseline for content-adaptive NVC methods. As the landscape of NVC evolves to incorporate more intricate techniques, our architecture-agnostic updating strategy is anticipated to increase, offering a solution to reduce computational requirements while maximizing efficiency. Moreover, such efficient content-adaptive NVC is poised to broaden the scope of NVC technologies, enabling them to address more extensive applications of complex content types beyond standard video. This research heralds new avenues for the compression of more diverse content.  
 
 Appendix:  


Decision: Based on the reviews, the paper demonstrates a technically sound and innovative approach to Parameter-Efficient Transfer Learning for video compression, with substantial experimental evidence supporting its effectiveness. The strengths, such as detailed ablation studies and better performance on out-of-domain datasets, are significant. However, there were some limitations, including a lack of comprehensive comparisons with state-of-the-art methods and some missing experimental results, and the main important ones have been addressed in the rebuttal. Considering the overall contributions and the clear presentation of the methodology make a compelling case for acceptance, I recommend acceptance (Poster).

Reviews: review: paper_summary: paper_summary: This paper introduces a novel technique called Group-aware Parameter-Efficient Updating (GPU) to enhance neural video compression (NVC). The GPU method employs encoding-side adapters inspired by Parameter-Efficient Transfer Learning (PETL) to stimulate specific components within encoding modules, enabling efficient parameter optimization. These adapters allow the method to efficiently handle varying content types within videos. To mitigate error accumulation and maintain compression quality, the method updates entire Groups of Pictures (GoPs) rather than individual frames, using a patch-based strategy to manage memory demands. The GPU approach improves video compression efficiency and adaptability, outperforming traditional codecs like H.266/VVC and previous content-adaptive NVC methods in experiments on a medical dataset.
strength_weakness: strength_weakness: 1. The Paremeter-Efficient Transfer Learning on video compression tasks is quite interesting and effective.

2. The authors present a variety of experimental results, including qualitative outcomes and ablation studies.

3. The proposed method demonstrates better results on out-of-domain datasets compared to DCVC-DC.

4. The paper presentation is well-structured and easy to understand.
limitations: limitations: 1. Some recent NIC papers (cited by the authors) used decoder updates. The authors stated that decoder-focused methods require high bit rates. However, the previous work, e.g., [43] showed a promising result. I expect more justification as to why decoder-side adaptation does not work in NVC. 

2.  I hate to say that, but, there is no clear improvement compared to the state-of-the-art methods, such as (DCVC-DC) or DCVC-FM [1]. How does your model compare to DCVC-DC in terms of model size (# parameters), encoding, and decoding speed? 

3. For a fair comparison, the authors should have provided the experimental results using full updating variants for video compression tasks.  In Figure 4, from my understanding, the RD-curves of Lu et al., and Tang et al. are from the original paper, and the backbone architecture is quite different (model size is also very different). Hence, I do not think you can make the conclusion that the proposed fine-tuning scheme is more effective than full finetuning.

4. Experimental results on MCL-JCV dataset?

5. Why not apply the proposed technique to DCVC variants?

6. The authors tested the impact of updating the number of frames, including a single frame. They stated that they only utilized P-frame compression, which makes it unclear what is meant by updating a single frame (this could be interpreted as 'not updating P-frames' or 'updating only one P-frame'). Additionally, to demonstrate the strength of patch-based GoP updates, comparing them with a small GoP (e.g., 5) with full resolution would be beneficial.

7.  [43] is an NVC paper, but the authors introduced it as neural image compression (NIC).

[Missing references]
 [1] Li, Jiahao, Bin Li, and Yan Lu. "Neural Video Compression with Feature Modulation." arXiv preprint arXiv:2402.17414 (2024).
 [2] Lv, Yue, et al. "Dynamic Low-Rank Instance Adaptation for Universal Neural Image Compression." Proceedings of the 31st ACM International Conference on Multimedia. 2023.
paper_summary: paper_summary: This paper studies the task of content-adaptive video compression. The authors design a new method called GPU to minimize errors on temporal redundancy. Besides, GPU also uses some light-weight adapters to finetune the model efficiently. Extensive experiments show the advantages of the proposed technique.
strength_weakness: strength_weakness: - Overall, this paper is technical sound.
- The authors also conduct some experiments to show the advantages of their technique over existing baselines.
limitations: limitations: - The number of compared methods is too small. DCVC and DCVC-DC should be included.
- Compression latency, an important metric, should also be reported and compared. 
- The continuity of the two motivations in the author's paper does not seem consistent. I think the two novelties are relatively independent. The authors should express the connection between them more accurately.
- The writing and the presentation of the paper should also be advanced. The authors should include an illustration in the 1st or 2nd page to highlight the limitations of existing techniques and their design to address the limitations.

By taking these limitations into consideration, my initial score before rebuttal is a borderline.
paper_summary: paper_summary: The paper introduces Group-aware Parameter-Efficient Updating (GPU), a content-adaptive technique for neural video compression (NVC). It addresses challenges in NVC, such as error accumulation and complex frameworks, by using a patch-based Group of Pictures (GoP) training strategy and a parameter-efficient delta-tuning approach. GPU demonstrates good compression efficiency across various benchmarks.
strength_weakness: strength_weakness: (1) The method is a plug-and-play framework. Based on the theoretical description in the paper, I believe this method can be easily migrated to other neural network video compression frameworks.

(2) The description of GoP updating is very detailed, and I can clearly understand the authors' expression.

(3) Adequate ablation studies demonstrate the effectiveness of Group-aware Parameter-Efficient Updating.
limitations: limitations: (1) I notice that GPU demonstrates a significant performance improvement compared to other content adaptive methods. However, compared to DCVC_DC (baseline), the performance improvement seems less pronounced (30.68% vs. 27.44%). Does it suggest that the architecture of DCVC_DC contributes more to the performance?

(2) When comparing time complexity, I hope the author could include as many methods from Fig. 5 in the comparison as possible.

(3) The datasets tested by authors seem somewhat limited. I hope that testing results on datasets such as MCL-JCV [1] and UVG [2] can be added to demonstrate the generalizability of the method.

[1] Wang, Haiqiang, et al. "MCL-JCV: a JND-based H. 264/AVC video quality assessment dataset." 2016 IEEE international conference on image processing (ICIP). IEEE, 2016.

[2] Mercat, Alexandre, Marko Viitanen, and Jarno Vanne. "UVG dataset: 50/120fps 4K sequences for video codec analysis and development." Proceedings of the 11th ACM Multimedia Systems Conference. 2020.
paper_summary: paper_summary: This paper presents a novel method for neural video compression (NVC) that adapts to different video contents effectively. The proposed approach, named Group-aware Parameter-efficient Updating (GPU), addresses two main challenges in neural video compression: error accumulation and the complexity of updating large encoder parameters. The GPU method uses a group-aware strategy for parameter updates and introduces lightweight adapter modules for efficient optimization. The method segments videos into patch-based groups of pictures (GoPs) and updates them sequentially, reducing memory demands and improving compression efficiency. The approach is validated on HEVC test sequences, showing superior performance compared to existing methods.
strength_weakness: strength_weakness: 1. The use of group-aware parameter-efficient updating and lightweight adapters represents a significant advancement in content-adaptive NVC.
2. Group-aware parameter-efficient updating analyzes the video sequence and adjust GOP size dynamically.
3. The adapter is very light weight, only 3 conv layers and 2 LeakyRelu.
4. A patch-based GoP updating strategy significantly reduces computational complexity and memory requirements.
limitations: limitations: 1. The method involves complex segmentation and updating strategies, including online training (adapter) which might be challenging to implement and integrate into existing systems when 2-pass coding is disabled.
2. The process of fine-tuning and adapting the model to new content can be computationally intensive, limiting its use to Random Access mode, it cannot be use in LDB mode as future frames are not available in LDB mode.