Title: Neural B-frame Video Compression with Bi-directional Reference Harmonization
Venue: NeurIPS 2025
Accepted: yes
Keywords: neural video compression

video coding

B-frame compression
Abstract: Neural video compression (NVC) has made significant progress in recent years, while neural B-frame video compression (NBVC) remains underexplored compared to P-frame compression. NBVC can adopt bi-directional reference frames for better compression performance. However, NBVC's hierarchical coding may complicate continuous temporal prediction, especially at some hierarchical levels with a large frame span, which could cause the contribution of the two reference frames to be unbalanced. To optimize reference information utilization, we propose a novel NBVC method, termed Bi-directional Reference Harmonization Video Compression (BRHVC), with the proposed Bi-directional Motion Converge (BMC) and Bi-directional Contextual Fusion (BCF). BMC converges multiple optical flows in motion compression, leading to more accurate motion compensation on a larger scale. Then BCF explicitly models the weights of reference contexts under the guidance of motion compensation accuracy. With more efficient motions and contexts, BRHVC can effectively harmonize bi-directional references. Experimental results indicate that our BRHVC outperforms previous state-of-the-art NVC methods, even surpassing the traditional coding, VTM-RA (under random access configuration), on the HEVC datasets. The source code will be released. The source code is released at https://github.com/kwai/NVC.


Peer reviews:
comments: SUMMARY:
The authors analyse the effect of the Unbalanced Reference Contribution (URC) issue with a quantitative experiment in Section 3. Based on the motivation of coping with URC issue, they build a novel framework termed BRHVC, with proposed BMC and BCF. Experimental results show that their BRHVC outperforms previous SOTA nerual video compression methods, which is a remarkable progress.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The proposition and analysis of the URC issue are very interesting. The authors design a clever toy experiment to illustrate the significant impact of URC on bi-directional compression.
2. The motivation behind the design of BRHVC is reasonable. The BMC can provide more accurate optical flows in a large scale, while the BCF offers reference weights to better integrate reference information.
3. The progress of BRHVC is remarkable compared with the baseline. And the overall performance outperforms previous SOTA nerual video compression methods, which is comparable to VTM-RA. 
4. The paper is well-written with good readability and is easy to understand.

Weaknesses:
1. The description of the baseline is too brief. It seems that the baseline employs a similar architecture to DCVC-B [33], and its performance is also quite comparable. So why not directly use DCVC-B as the baseline? What are the main differences between the baseline and DCVC-B?
2. There is a lack of ablation study. Although, as mentioned above, the improvement of BRHVC over the baseline is reasonable, there is still a need for a model with similar complexity to be compared with BRHVC.

QUESTIONS:
See Weaknesses.

LIMITATIONS:
As mentioned in paper, BRHVC does not perform sufficiently well on high frame rate datasets, which may require further investigation in future work.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces BRHVC, a novel neural B-frame video compression method designed to tackle the challenge of unbalanced reference contributions in hierarchical coding. It achieves this through two proposed modules: Bi-directional Motion Converge (BMC) for improved motion handling and Bi-directional Contextual Fusion (BCF) for adaptively weighting bi-directional references. BRHVC demonstrates state-of-the-art performance, outperforming previous neural methods and surpassing VTM-RA on HEVC datasets.\n\nSTRENGTHS AND WEAKNESSES:\n** Pros\n - The paper effectively identifies and proposes solutions for the Unbalanced Reference Contribution problem inherent in hierarchical B-frame coding.\n - It achieves SOTA performance, notably surpassing the high-performance VTM-RA codec on standard datasets.\n - It introduces innovative BMC and BCF modules that enhance motion compensation and context fusion for better reference utilization.\n\n** Cons\n - The proposed framework integrates BMC and BCF modules, where BMC refines motion estimation and compensation, and BCF uses the resulting context features for fusion. This suggests a sequential flow of information. One may consider a mechanism for tighter integration, such as joint optimization or feedback loops, where the effectiveness of the context fusion (BCF output) could inform or adapt the motion processing (BMC). Exploring a more deeply intertwined design between motion and context modules might potentially enable further performance improvements by allowing them to learn to cooperate more effectively end-to-end.\n\n - As shown in the ablation study (Table 2), BRHVC introduces a substantial increase in encoding time (about 50% over baseline) and decoding time (about 30% over baseline) compared to the baseline model. While the authors state this overhead is \"relatively acceptable\" given the BD-rate improvement, such a significant increase in computational cost could be a major barrier for real-world deployment, especially in scenarios with strict real-time constraints or limited hardware resources (like mobile devices for decoding). The trade-off between performance and complexity might not be universally \"acceptable\" depending on the specific application.\n\n - The ablation study effectively demonstrates the contribution of BCF alone and the additional contribution of adding BMC after BCF. However, it lacks the result of using BMC alone (e.g., Baseline w/ BMC). Presenting this result would provide clearer insight into the independent contribution of the BMC module and its efficiency, as well as how its interaction with BCF leads to the final performance gain. \n\n - The paper primarily uses PSNR-based BD-rate to demonstrate performance gains. While PSNR is a standard objective metric in video compression, it does not always correlate perfectly with perceived visual quality by humans. Metrics like SSIM, MS-SSIM, or VMAF are often considered more representative of subjective user experience. The absence of results using these alternative metrics makes it difficult to fully assess the practical benefits of BRHVC in terms of visual perception.\n\n - The paper acknowledges that BRHVC's performance gains are less pronounced on high frame rate datasets such as MCL-JCV and UVG compared to HEVC datasets. While identifying this limitation is valuable, the paper does not provide a detailed analysis or hypothesis as to why this performance gap exists. Is the Unbalanced Reference Contribution problem less severe in high frame rate videos due to smaller frame spans? Are there other dominant challenges in high frame rate compression that BRHVC's current design doesn't adequately address? A deeper analysis of the causes would be crucial for guiding future research on improving NBVC for high frame rate content.\n\n- All primary experiments are conducted using an Intra Period (IP) of 32. The hierarchical coding structure of B-frames, and consequently the characteristics of reference spans and the severity of the URC problem, can change significantly with different IP lengths. While IP=32 is a common setting for RA configurations, evaluating BRHVC's performance across a range of IP values (e.g., shorter IPs like 8 or 16, or even longer ones) would provide a more comprehensive understanding of its robustness and the generalizability of its benefits across various hierarchical coding structures.\n\nQUESTIONS:\n- Could you please clarify the exact differences between the \"Baseline\" model used in the ablation study and the \"DCVC-B (re-trained)\"? What accounts for the small performance difference shown in Table 2?\n- The experiments focus on an Intra Period (IP) of 32. How does BRHVC's performance, and the severity of the Unbalanced Reference Contribution problem, vary with different IP settings? Would BRHVC maintain its advantage at shorter or longer IP values?\n- The performance gains are less significant on high frame rate datasets like MCL-JCV and UVG. What are the potential reasons or hypotheses for this observation? Are there specific characteristics of these datasets that limit BRHVC's effectiveness compared to HEVC datasets?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes BRHVC, a novel neural B-frame video compression method that addresses the inaccurate motion compensation and unbalanced reference contribution issue in bi-directional coding. The paper introduces two key modules(BMC and BCF) to enhance motion compensation and adaptively harmonizes reference information. Despite underperforming VTM-RA on several test datasets, the paper provides visualized analysis of module design and effect, and includes runtime efficiency evaluation, offering a well-rounded and transparent study.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. Multiscale optical flow extraction is employed in this paper, which is a novel approach that effectively mitigates inaccuracies in motion estimation. Its effectiveness is further supported by clear visualizations provided in the appendix.\n\n2. The BCF module is well-designed to leverage multiscale motion-compensated features through a weighted summation scheme to improve contextual reconstruction. The approach demonstrates a certain level of originality.\n\n3. The authors perform a quantitative evaluation of runtime efficiency, including encoding and decoding time, which contributes to assessing the system’s practical applicability.\n\n\nWeakness:\n\n1. The multiscale optical flow extraction process effectively requires three separate calls to SpyNet for each frame, which may incur redundant computation. As shown in Figure 13 of the appendix, when the motion between frames is small, extracting optical flow at the original resolution and directly downsample can yield comparable results, suggesting the current approach may be computationally excessive in such cases.\n\n2. The paper lacks detailed reporting on the overall computational complexity (e.g., in terms of kMACs/pixel), as well as comparisons with similar systems. In addition, it does not provide a breakdown of the computational cost and latency associated with each individual module.\n\n3. The proposed system underperforms the VTM-RA baseline on 4 out of 6 benchmark datasets, indicating that its overall compression performance still requires further improvement.\n\nQUESTIONS:\n1. The paper states that the Motion Compensation module adopts the Contextual-Feature Extraction (CFE) method as used in references [20] and [33]. However, this specific concept does not appear explicitly in the main texts of those references. Could the authors clarify what is meant by this term?\n\n2. Could the authors provide a theoretical analysis of why the proposed system performs less effectively on high-resolution datasets, such as UVG and MCL-JCV?\n\n3. How is the proposed method compared to recent multimodal-based video compression methods, such as M3-CVC?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a new video coding model, Bi-directional Reference Harmonization Video Compression (BRHVC). The model is based on the conditional coding framework; Unlike most existing work, which focuses on low-delay settings, BRHVC targets hierarchical B-frame coding. While existing learned B-frame coding models offer limited gains over low-delay methods, the authors tackle these limitations by (1) utilising multi-scale optical flow to warp features at different scales and jointly encoding these flows into the latent representation, which improves performance on sequences with large motions, and (2) explicitly modelling the weighting of different reference contexts, enabling better use of bi-directional information.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, although the proposed model demonstrates improvement over the baselines, its novelty is modest, and the new modules have not yet been validated comprehensively.\n\nStrengths\n- The proposed model focuses on hierarchical B-frame coding, addressing a major limitation of current neural video codecs.\n- The motivation is sound: balancing the use of multiple references is indeed important for video coding.\n\nWeaknesses\n- While the proposed techniques are verified by an ablation study showing BD-rate gains for each component, the experiments are not comprehensive. In particular, adding these modules increases both computational complexity and model size. It is unclear whether the added components are the best choice - could similar gains be achieved by simply stacking more layers at the same positions with comparable complexity?\n- The memory footprint (a limitation of conditional coding models) and complexity (e.g., MACs per pixel) are not provided. While encoding/decoding speeds are reported, memory usage and MACs are also critical for practical application, since the decoding speed can vary significantly across platforms.\n- Although the experiments show that there are different importance of references, the authors did not demonstrate how existing models fail to leverage bi-directional context. This omission weakens the motivation for the paper.\n\nQUESTIONS:\n- How exactly were the experiments performed when removing the BCF/BMC modules?\n- The authors should also validate the use of different motion vectors at each scale.\n- It may help to clarify the two main B-frame coding strategies - low-delay B and hierarchical B. The proposed model only address the latter category.\n\nLIMITATIONS:\nThe authors have discussed some limitations of the proposed methods. Please refer to the weaknesses listed above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}