Title: ControlFusion: A Controllable Image Fusion Network with Language-Vision Degradation Prompts
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image fusion

multimodal images

degradation
Abstract: Current image fusion methods struggle with real-world composite degradations and lack the flexibility to accommodate user-specific needs. To address this, we propose ControlFusion, a controllable fusion network guided by language-vision prompts that adaptively mitigates composite degradations. On the one hand, we construct a degraded imaging model based on physical mechanisms, such as the Retinex theory and atmospheric scattering principle, to simulate composite degradations and provide a data foundation for addressing realistic degradations. On the other hand, we devise a prompt-modulated restoration and fusion network that dynamically enhances features according to degradation prompts, enabling adaptability to varying degradation levels. To support user-specific preferences in visual quality, a text encoder is incorporated to embed user-defined degradation types and levels as degradation prompts. Moreover, a spatial-frequency collaborative visual adapter is designed to autonomously perceive degradations from source images, thereby reducing complete reliance on user instructions. Extensive experiments demonstrate that ControlFusion outperforms SOTA fusion methods in fusion quality and degradation handling, particularly under real-world and compound degradations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ControlFusion, a novel and controllable framework for infrared-visible image fusion (IVIF) that simultaneously performs image restoration and fusion. The core problem is the poor performance of existing methods on real-world images suffering from complex, composite degradations (e.g., noise, low-light, haze, blur) and their lack of flexibility to accommodate user-specific needs. The authors presents a physics-driven degradation model to construct a realistic training dataset (DDL-12), and then use it for training controllable image fusion network. The training stage involves textual-visual prompt alignment and prompt-modulated restoration & fusion to achieve the controllable image fusion with prompts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper addresses a critical and practical limitation of current image fusion techniques: the inability to handle complex, real-world degradations and the lack of user control. The core idea of using language-vision prompts to control the fusion process is interesting and effectively implemented.\n\n2. The paper proposes a realistic imaging model based on physical principles like the Retinex theory and atmospheric scattering to synthesize a high-quality training dataset (DDL-12) that bridges the domain gap between synthetic and real-world data.\n\n3. Extensive experiments demonstrate that ControlFusion significantly outperforms state-of-the-art methods in both fusion quality and degradation handling, while also providing user-driven customizability.\n\n\nWeaknesses\n\n1. The introduction of multiple sophisticated modules for CLIP, SFVA, and PMM inevitably increases the model's parameter count and computational overhead compared to simpler fusion networks. This could be a potential drawback for deployment on resource-constrained edge devices where real-time performance is critical. This naturally leads to a question about the performance-efficiency trade-off. While the performance improvement is significant, it is important to consider whether this improvement mainly comes from using more computatioal cost.\n\n2. The paper's SFVA extracts features from two fundamentally different domains: spatial and frequency. The method to combine them is a simple concatenation followed by a linear layer. This leads to a question about its effectiveness, as spatial features (capturing local textures) and frequency features (representing global periodic patterns) have vastly different properties. A simple linear fusion might not be powerful enough to model their different characteristics. A comprehensive study on this (e.g., more sophisticated fusion mechanisms) may be necessary to strengthen the paper's contribution.\n\n3. While the DDL-12 dataset is a synthetic dataset, the model trained with it may not be generalizable to real-world scenarios. A \"domain gap\" inevitably exists between these simulated degradations and the unpredictable and complex degradations found in real-captured images. To fully validate the model's practical utility and its superiority in bridging the domain gap, it would be beneficial to include more experiments on realistic benchmarks and real-cpatured images.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a controllable image fusion network guided by language-vision prompts, named ControlFusion, to address the problem of complex degradations in real-world scenarios. It integrates a physics-driven degradation model, a spatial-frequency collaborative visual adapter (SFVA), and a prompt-modulated restoration and fusion network. The model is capable of handling compound degradations and supports both language and visual prompts for further control. Extensive experiments on multiple datasets demonstrate the model's fusion quality and robustness under various degradation conditions.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper introduces an imaging degradation model based on Retinex theory and atmospheric scattering, covering various degradation types for both infrared and visible modalities.\n2. ControlFusion can control the fusion process through textual and visual prompts, enabling the network to flexibly adapt to different types and levels of degradation.\n3. The method uses a spatial-frequency visual adapter (SFVA) to extract visual degradation prompts and align them with CLIP-based text embeddings, which is well-designed and addresses the limitations of purely text-driven methods.\n4. The authors evaluate the model across multiple datasets (MSRS, LLVIP, RoadScene, FMB), multiple metrics (EN, SD, VIF, Qabf, CLIP-IQA, MUSIQ, TReS), and both with/without pre-enhancement. Ablation studies and real-world generalization are also included.\n\n## Weaknesses\n\n### Major Weaknesses\n\n1. Although the visual prompt pathway alleviates dependence on textual input, the prompt-modulated module still hinges on CLIP-based representations. How does this ensure the effective extraction of degradation prompts from the source images?\n2. The model includes multi-stage Transformers, visual-text prompt alignment, and SFVA, which increases inference time and model complexity. The author should compare the parameters and time complexity of the model.\n3. The proposed method supports user-provided text prompts that include degradation severity. However, the paper does not provide sufficient details on the design of the prompts, such as the structure of text templates or how multiple degradations and their combinations are represented.\n4. In the experimental evaluation related to degradations, the dataset used appears to contain a mixture of real and synthetic degradations. Additionally, qualitative results in Figure 5 alone is not sufficiently convincing to demonstrate the method’s effectiveness under real-world scenarios. To better validate effectiveness and robustness, the authors are encouraged to provide both quantitative and qualitative comparisons on datasets only with real degradations, such as night-time scenes from MSRS.\n\n### Minor Weaknesses\n\n1. The notation in Equation (9) needs to be corrected. Specifically, $Q_{vi}$ and $K_{ir}$ appear twice in the equation, which is confusing and likely a typographical error.\n2. Some sentences in the paper have punctuation issues. For example, in Section 5.3, the first sentence: \"removing composite degradations in real scenes..\" contains an incorrect punctuation mark.\n\nQUESTIONS:\nAlthough Figure 8 demonstrates the superiority of visual prompts over textual prompts, I would encourage the authors to further clarify: \n\n1. Whether the visual prompt remains robust across different levels of degradation? \n2. Whether the visual prompt is capable of handling compound degradations? Besides, please address the concerns in the Weakness part.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a controllable fusion network guided by language-vision prompts, called ControlFusion. The main contributions include two aspects. First, they construct a degraded imaging model based on physical mechanisms, to simulate data with composite degradation for learning. Second, a prompt-modulated restoration and fusion network is designed to dynamically enhances features according to degradation prompts. As a result, this work can adapt to varying degradation levels, showing promising robust fusion performance. They conduct extensive experiments to prove the advantages of ControlFusion.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) Datasets involving composite degradation are highly valuable for advancing research in multimodal image fusion. This work customizes simulated imaging based on physical models, which facilitates the batch generation of the required data.\n2) This work uses text as a degradation prompt, which specifies not only the type but also the level of degradation. This provides a highly flexible interface for controlling the fusion process. It is also very meaningful for the deployment of multimodal image fusion models in real-world degraded environments.\n3) The proposed spatial-frequency visual adapter is interesting, as it enables the perception of degradation distribution within source images based on their frequency characteristics, which is crucial for subsequent information restoration and fusion.\n4) The conducted experiments are relatively comprehensive and effectively demonstrate the advantages and distinctive features of the proposed method.\n\nWeaknesses:\n1) One notable feature of the proposed method is its ability to specify the level of degradation. However, it is unclear how users are expected to have prior knowledge of the degradation severity. This could limit the practical utility of the interface. It would be helpful if the authors could provide guidance for determining appropriate degradation levels. In addition, the authors should provide a standardized paradigm for constructing textual prompts to facilitate the reproducibility of the proposed method.\n2) The imaging model presented in Section 3 is based on existing physical models, but its core innovation should be further emphasized. It is recommended that this part be integrated into the methodology section to strengthen its connection to the overall contribution of the proposed method.\n3) This work constructs degraded data based on existing datasets such as MSRS, RoadScene, and LLVIP, assuming that the original data are clean. However, if the original data are not clean, how is this issue addressed? Would the trained models be affected by the inherent noise or degradation in the base datasets?\n4) In the “Comparison without Pre-enhancement” Section, the qualitative comparison on the four mentioned datasets is lacking.\n5) In the “Comparison with Pre-enhancement” Section, it would be better to include full-reference fusion metrics, since relying solely on some perceptual metrics such as SD and EN cannot fully reflect fusion performance under degraded scenarios.\n6) To more comprehensively demonstrate the real-world reliability of the proposed method, please include real composite degradation data, such as the nighttime scenes from the MSRS dataset.\n7) Average running time is recommended to provide comparative evaluations of computational efficiency.\n\nQUESTIONS:\n1) In Figure 2, does \"composite degradation\" refer to the simulated data? The logical relationship among (I), (II), and (III) is somewhat unclear.\n2) In Figure 3, the authors provide the frequency spectrum of images containing various types of degradation; it would be better to additionally present the frequency spectrum of the images without degradation for comparison.\n3) Does training the model without the proposed DDL-12 dataset degrade its performance in real-world degradation scenarios?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors aim to fuse infrared and visible images under complex, multi-stage degradations. Specifically, these degradations stem from illumination, weather, and sensor effects. Based on this modeling process, they created a synthetic training dataset. The training also incorporates both degradation prompts and automatic degradation detection, so that the text prompt and the degraded image each provide cues about the degradation.\n\nSTRENGTHS AND WEAKNESSES:\n- Strength:\nThe consideration of multi-stage degradations appears relatively new for image fusion tasks, and the proposed method explicitly addresses this, improving robustness in real-world scenarios.\n\nAs reported, the proposed method is effective on both synthetic and real-world datasets.\n\n- Weakness\nThe source of its performance on real-world datasets is questionable, given the training data's limited diversity. Additionally, the real-world tests do not include weather-related degradations.\n\nDesign-wise, the method primarily leverages data priors by constructing a multi-degradation paired dataset, yet it completely overlooks potential priors inherent in the infrared data format, making the overall solution less insightful.\n\nQUESTIONS:\n- Have you evaluated how the number of degradation intensity levels used during training affects your model’s generalization on real‐world datasets (e.g., FMB, LLVIP)? For instance, if you change from 4 levels to 2, 3, or more, how does the performance vary?\n\n- How are those pre-enhancements applied? Line 227 just mentions the application of Instruct-IR and OneRestore. In Table 2, do you use only one of them, or different fusion methods has different pre-enhancement method? Meanwhile, have the authors tried applying degradation-specific SOTA methods sequentially as pre-enhancement?\n\n- How do the authors ensure that the training data domain covers the degradations present in real-world datasets? For example, the synthetic dataset simulates rain as streaks, but in reality, raindrops are much more common and challenging. Similarly, the haze model used here is homogeneous, yet varying the depth scale and hyperparameters can produce a wide range of haze effects. How can the proposed dataset and method handle such complex real-world data given the limited diversity in the training set?\n\n- It seems that the real-world testing does not include weather degradations. Can the authors justify this omission?\n\n- How can the infrared branch data provide insightful observations that can be implicitly or explicitly leveraged in this fusion task? It seems the authors simply enlarged the training dataset and helped the model learn everything without considering sensor-specific priors that could be exploited.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}