Title: A Unified Solution to Video Fusion: From Multi-Frame Learning to Benchmarking
Venue: NeurIPS 2025
Accepted: yes
Keywords: image fusion

video fusion

low-level vision
Abstract: The real world is dynamic, yet most image fusion methods process static frames independently, ignoring temporal correlations in videos and leading to flickering and temporal inconsistency. To address this, we propose Unified Video Fusion (UniVF), a novel and unified framework for video fusion that leverages multi-frame learning and optical flow-based feature warping for informative, temporally coherent video fusion. To support its development, we also introduce Video Fusion Benchmark (VF-Bench), the first comprehensive benchmark covering four video fusion tasks: multi-exposure, multi-focus, infrared-visible, and medical fusion. VF-Bench provides high-quality, well-aligned video pairs obtained through synthetic data generation and rigorous curation from existing datasets, with a unified evaluation protocol that jointly assesses the spatial quality and temporal consistency of video fusion. Extensive experiments show that UniVF achieves state-of-the-art results across all tasks on VF-Bench. Project page: [vfbench.github.io](https://vfbench.github.io).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes UniVF, a unified framework for video fusion that incorporates multi-frame learning and optical flow-based feature warping to ensure temporal coherence and informative fusion. Alongside the method, the authors introduce VF-Bench, a comprehensive benchmark consisting of four video fusion tasks: multi-exposure, multi-focus, infrared-visible, and medical video fusion. The benchmark includes careful construction or curation of well-aligned video pairs and establishes a joint evaluation protocol combining spatial quality and temporal consistency metrics. Extensive empirical studies show UniVF achieves strong quantitative and qualitative performance across all four tasks, and ablation studies support the design components.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. VF-Bench covers a diverse set of video fusion scenarios with carefully constructed or curated and well-aligned video pairs.\n2. UniVF leverages multi-frame encoding/decoding and incorporates state-of-the-art optical flow for temporal alignment, with the additional use of temporal consistency loss supplementing the objective for smooth fusion.\n3. The paper adds two temporal consistency metrics, measuring intra-video smoothness and flow coherence, giving a more holistic evaluation of the fusion problem.\n4. The main technical details for model, dataset, loss, and evaluation metrics are sufficiently described. UniVF consistently outperforms state-of-the-art fusion methods, with results contextualized in the paper.\n\n## Weaknesses\n\n1. Although the paper emphasizes the benefit of video fusion, most of the visual results shown are still static image comparisons. It would be beneficial to see a comparison of the fused videos to demonstrate consistency over time.\n2. The physical or perceptual meaning of MS2R remains vague. Could the authors explain it in more intuitive terms?\n3. The transformer backbone uses four Restormer blocks, but there is no sensitivity analysis showing whether performance saturates or improves with more/less blocks. \n4. In UniVF, after calculating the optical flow on adjacent images, the corresponding features are warped. I suggest adding ablation experiments to see the effect of warping directly on the image and then feeding it into the encoder.\n5. There is no report on inference time, model size, or real-time capability. Since the framework includes optical flow and transformer operations, its computational burden needs to be discussed.\n6. RCVS [1] are mentioned in related work but missing in task evaluations. As a baseline for RGBT video stream fusion, additional it in comparison would provide a more convincing benchmark.\n\n*[1] Xie, Housheng, et al. \"RCVS: A Unified Registration and Fusion Framework for Video Streams.\" IEEE Transactions on Multimedia 2024*\n\nQUESTIONS:\nI hope the author will address the relevant issues raised in the Weaknesses in the rebuttal.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces UniVF, a unified video fusion framework that incorporates multi-frame learning and optical flow-based feature warping for temporally coherent video fusion. It is evaluated on a newly proposed VF-Bench, which contains datasets for four distinct fusion tasks. The authors also design a new temporal loss and evaluation metrics to support future research. Experiments show that UniVF outperforms a variety of baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \nThe paper addresses an important video fusion problem and proposes a reasonably well-designed method with clear motivations. The architecture is sound, and the benchmark is timely. Loss and evaluation metrics are novel and reasonable. \n\nWeaknesses:\n1. Methods like DCINN [1] are absent from the infrared-visible (IVF) comparison. \n2. The framework relies heavily on SEA-RAFT for flow estimation, but the impact of this choice is not analyzed. An ablation replacing SEA-RAFT with other classical models would clarify whether the gains come from the model itself or the flow estimator.\n3. Eq. (7) introduces a validity mask M to select reliable flow regions. However, there is no explanation of how this mask is computed or applied in practice.\n4. The paper omits practical information like inference speed, model parameters, etc. Given the use of multi-frame inputs and transformer blocks, this is crucial for evaluating deployment feasibility.\n5. The framework operates on 3-frame snippets, but it is unclear how performance and speed change if more frames are used. This limits the understanding of temporal modeling capabilities beyond short-term dependencies.\n6. The model architecture (Fig. 2) is well-illustrated but lacks the notation mapping to equations. Adding symbols would help make the flow of information clearer to readers.\n\n[1] A general paradigm with detail-preserving conditional invertible network for image fusion. International Journal of Computer Vision (2024)\n\nQUESTIONS:\nBased on the Weaknesses, there are several areas that require clarification or further experiments. The ablation studies are somewhat shallow, and there are missing comparisons to several key baselines (e.g., DCINN). More detailed architectural descriptions would also help clarity. Additionally, the paper does not analyze the choice of optical flow estimator or its impact on results.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose UniVF, a unified framework for fusing video inputs from different modalities and settings. It achieves temporally coherent results via optical flow-based feature warping. Additionally, VF-Bench is introduced, a benchmark for evaluating video fusion methods in four categories. The proposed unified fusion loss guarantees the temporally coherent fusion results, and the proposed benchmark’s unified evaluation protocol jointly assesses spatial and temporal video quality. Extensive experiments show that UniVF achieves state-of-the-art results across all tasks on VF-Bench.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: This is a very promising paper with a practical goal: designing a single model to solve various video fusion problems. The paper is ambitious and brings value via a large-scale benchmark and a clean video fusion framework, combined with newly proposed loss functions and evaluation protocols tailored for video fusion. \nWeaknesses: However, the verification of temporal consistency is insufficient—strip-based visualization would help. The sigma parameter in the synthetic multi-focus dataset is critical but underexplained. The motivations for the HDR data pipeline and color space conversion also lack clarity. Technical details of the warping process need to be specified. See the Questions section for more details.\n\nQUESTIONS:\n1.The proposed temporal consistency is mainly supported by metric tables. A common method for visualizing temporal coherence is showing time-slice (strip) images—these would give an intuitive picture of flicker or ghosting and are currently missing.\n2.When constructing the multi-focus dataset, the blur intensity factor $\\sigma$ is a key parameter that determines the degree of blurring in the near-focus plane and far-focus plane. How is this parameter determined? Additionally, could the authors provide a more detailed explanation of how the near-focus plane and far-focus plane are defined and separated in the proposed data generation process?\n3.For multi-exposure fusion dataset, I still have some confusion regarding the data generation pipeline. For example, it is unclear why exposure adjustment needs to be performed in the linear light domain, and why a conversion from 10-bit BT.2020 color space to 8-bit BT.709 is necessary after exposure adjustment. The description in the paper is somewhat brief, and the motivations behind these steps should be explained more clearly. \n4.It would be helpful if the authors could clarify how the feature warping operation is specifically performed — is it based on a deterministic affine transformation or implemented as a learnable module? \n5.The IVF branch should include C2RF [1]—a recent strong fusion model with cross-modal alignment.\n6.Using \"image domain metrics\" vs. \"temporal domain metrics\" is potentially misleading since both are computed in the image domain. A better term might be \"spatial domain metrics\" to distinguish them clearly.\n[1] C2RF: Bridging Multi-modal Image Registration and Fusion via Commonality Mining and Contrastive Learning. IJCV 2025.\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe proposed method addresses the issue of flickering artifacts in video fusion, which arise from neglecting temporal correlations. The authors propose UniVF, a unified framework for video fusion that explicitly models spatial-temporal dependencies through multi-frame learning and optical flow-based feature warping to ensure temporally coherent outputs. To facilitate research and evaluation, the authors introduce VF-Bench, the first comprehensive benchmark for video fusion, covering four distinct tasks. It is accompanied by a unified evaluation protocol that jointly assesses spatial quality and temporal consistency. The authors report that UniVF achieves state-of-the-art performance across all tasks on this new benchmark. This work provides a much-needed benchmark and a strong baseline for future research in temporally coherent video fusion.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1.VF-Bench Contribution.\nThe proposed VF-Bench is a valuable contribution, addressing the long-standing lack of high-quality, well-aligned, and diverse video pairs for fusion tasks. Its data synthesis strategy and unified evaluation protocol promote standardized and reproducible research.\n2.UniVF Framework Design.\nThe UniVF framework is well-motivated, employing optical flow-based feature warping to ensure temporal consistency. Combined with a temporal consistency loss, it presents a coherent and effective solution for video fusion.\nWeaknesses:\n1.Core operations like \"feature warping\" are mentioned but not thoroughly described. It would help readers if the paper explicitly stated how bilinear warping is used and applied.\n2.Figure 2 shows two encoders for the input streams. Are their weights shared or independent? This design choice could affect performance and needs clarification.\n3.The proposed temporal loss only uses immediate neighbors. How to balance video fusion performance with the size of the neighbors window?\n4.SEA-RAFT is used without justification. It's unclear whether this choice is optimal or arbitrary. An ablation comparing several flow methods (e.g., RAFT, Unimatch) would help isolate the contribution from the quality of the flow.\n5.A critical design decision-whether the flow network is frozen or trained end-to-end-is not discussed. Please clarify your choice.\nThese weaknesses need to be addressed with additional experiments and a deeper architectural discussion.\n\nQUESTIONS:\nThe technical content is solid, and the paper is novel and well-organized. However, as noted in the weaknesses:\n1.Foundational concepts like warping are assumed known. \n2.It is unclear whether the two encoders in Fig. 2 share weights\n3.Temporal loss only considers adjacent frames without modeling longer dependencies. \n4.The rationale for choosing SEA-RAFT is lacking. \n5.It's unclear if optical flow is frozen or jointly trained.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}