Title: Text-DiFuse: An Interactive Multi-Modal Image Fusion Framework based on Text-modulated Diffusion Model
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image fusion

multi-modal fusion

text

diffusion
Abstract: Existing multi-modal image fusion methods fail to address the compound degradations presented in source images, resulting in fusion images plagued by noise, color bias, improper exposure, etc. Additionally, these methods often overlook the specificity of foreground objects, weakening the salience of the objects of interest within the fused images. To address these challenges, this study proposes a novel interactive multi-modal image fusion framework based on the text-modulated diffusion model, called Text-DiFuse. First, this framework integrates feature-level information integration into the diffusion process, allowing adaptive degradation removal and multi-modal information fusion. This is the first attempt to deeply and explicitly embed information fusion within the diffusion process, effectively addressing compound degradation in image fusion. Second, by embedding the combination of the text and zero-shot location model into the diffusion fusion process, a text-controlled fusion re-modulation strategy is developed. This enables user-customized text control to improve fusion performance and highlight foreground objects in the fused images. Extensive experiments on diverse public datasets show that our Text-DiFuse achieves state-of-the-art fusion performance across various scenarios with complex degradation. Moreover, the semantic segmentation experiment validates the significant enhancement in semantic performance achieved by our text-controlled fusion re-modulation strategy. The code is publicly available at https://github.com/Leiii-Cao/Text-DiFuse.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA new paradigm of multi-modal image fusion named Text-DiFuse is introduced, based on the diffusion model. The paradigm embeds a mechanism for aggregating feature-level multi-modal image information into the diffusion process of degrading multi-modal images, addressing the optimization gap between \"degradation removal\" and \"multi-modal information fusion\". Additionally, a zero-shot model is introduced to modulate the fusion strategy based on user-input target text, enhancing the saliency of the target of interest. The conducted experiments suggest significant improvements in both human visual perception and advanced computer vision tasks.\n\nSTRENGTHS:\n1)\tEmbedding the mechanism of aggregating feature-level information into multiple diffusion processes to fuse multi-modal information is interesting. It is foreseeable that this diffusion paradigm produces fused images with better fidelity compared to methods based on likelihood-constrained diffusion models. \n2)\tThe coupled approach effectively resolves the issue of compound degradation in the process of multi-modal fusion, as evidenced by experimental results demonstrating significant advantages over the sequential approach.\n3)\tThe authors emphasize the importance of foreground targets in advanced visual tasks and propose enhancing target saliency through zero-shot assisted re-modulation. This approach diverges from traditional uniform fusion rules, demonstrating effectiveness.\n4)\tThis approach shows strong applicability. It demonstrates superior performance in multiple tasks including infrared and visible image fusion, medical image fusion, and polarization image fusion.\n\nWEAKNESSES:\n1)\tAfter the diffusion model is effectively trained, the sampling process can follow different step intervals. The information fusion in this method is integrated into the diffusion process, but the article does not seem to specify the sampling interval at which the results are obtained. Also, this article does not discuss the impact of the sampling interval on the fusion performance.\n2)\tThe presentation is slightly unclear. For example, from Equation 2 to Equation 6, both the features and the images carry the condition N that represents the degradation. Why does equation 7 no longer include N? Why can it be considered that the degradation has been removed at this point?\n3)\tIn Table 2 and Figure 4, some existing image restoration methods are cascaded in front of the fusion method to promote fairness in comparison, such as low-light enhancement (CLIP-LIT), denoising (SDAP), and white balance (AWB) algorithms. \nPlease explain the choice of the order in which they are connected in series, i.e. why low light enhancement first, then denoising, and finally white balance. \n4)\tModulating the salience of targets of interest in the fusion process through language is novel. Intuitively, I think the improvement in semantic properties brought about by this modulation is widespread. Currently, the effectiveness of language modulation has only been verified in the semantic segmentation scenario. It is recommended to provide an evaluation in the object detection scenario to further verify its role.\n\nQUESTIONS:\nPlease refer to the weaknesses part.\n\nLIMITATIONS:\nThe authors have analyzed the limitations and potential negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the topic of multi-modal image fusion. Two innovations enhance the performance of the fusion. One is the clever integration of information fusion into the diffusion process. This coupling way enables the fusion function to resist degradation. The other is the introduction of a text-based fusion remodulation strategy. This changes the limitation of previous fusion methods that could only use fixed mappings, allowing for the dynamic adjustment of the fused image based on specific requirements. This remodulation also enhances semantic attributes, improving the scores of the semantic segmentation task.\n\nSTRENGTHS:\n1. Integrating information fusion into the diffusion process is novel. Especially, each sampling step triggers an information fusion, which enhances the sufficiency of information fusion. This coupling can ensure the robustness of information fusion, addressing challenges such as low light, noise, and color cast. \n2. The introduction of multi-modal large models is interesting, particularly the ability to remodulate fused images using textual commands. This capability could potentially facilitate the flexible deployment of the proposed method across different application requirements. The demonstration of enhanced semantic attributes and improved semantic segmentation performance is good. \n3. Overall, the experiments are relatively sufficient. The comparative experiments include both baseline comparisons and pre-enhancement comparisons, which are important for ensuring fairness. \n4. The code is provided, which helps in reproducing the performance.\n\nWEAKNESSES:\n1. On page 5, line 174, the source data used for fusion contains degradation, [{Xb,Y}|N]. My question is, in Equations (9) and (10), where do the clean {Xb,Y} used to guide the fusion come from? Is there a multi-modal dataset that contains paired degraded and clean data? The paper seems to lack an explanation for this. \n2. The forward process of the diffusion model involves T steps of noise addition, while the reverse process consists of T steps of iterative sampling. Is the Z0 obtained in equation (8) a hypothetical Z0 derived from the diffusion relation at each sampling, or is it the Z0 after completing the full T steps of sampling? This determines the object of the constraints in the loss functions (9) and (10). It would be better to provide a detailed discussion on this. \n3. Only after the T steps of sampling can the data without degradation be obtained. So why can Z_{t-1}^b in equation (7) be considered free from degradation N? \n4. It's understandable that using textual modulation to control the desired targets of interest can enhance semantic attributes. My question is whether these enhanced semantic attributes can be generalized. In other words, can it also be effective in other high-level visual tasks besides semantic segmentation? \n5. Typo: The Zt on the left side of equation (8) seems to have a missing superscript b.\n\nQUESTIONS:\nPlease answer the question raised in Weaknesses.\n\nLIMITATIONS:\nYes, there are discussions about the limitations and potential negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses two primary challenges in multimodal image fusion: the mixed degradation of modalities and the insufficient salience of target objects. It proposes two methods to tackle these challenges: feature-level fusion diffusion and the re-modulation of fusion rules in target areas using a zero-shot segmentation model. They implement adequate experiments for evaluation, and the results demonstrate this method's advanced performance across various aspects, including the visual and semantic.\n\nSTRENGTHS:\n+ The mixed degradation of modalities and the insufficient salience of target objects are two interesting problems in multimodal image fusion. This paper’s discussion and solution of these two problems may promote the usability of fusion methods in real scenarios. \n+ The information fusion at the feature level is integrated into the diffusion process, which effectively realizes the degradation removal.\n+ The customized object highlighting strategy based on the zero-shot segmentation model is flexible. In particular, its gain in semantic attributes will increase the usability of the fused image in downstream tasks.\n+ This paper conducts lots of comparative experiments and ablation studies on the overall method. \n+ The narrative of this paper is comprehensive and clear. For me, it's easy to follow.\n\nWEAKNESSES:\n- This paper mentioned that the diffusion model is pre-trained to enable the denoising network to have the degradation removal function. However, details about the construction of the data used to train the diffusion model are missing. They need to describe this process to make the overall approach clearer.\n- This paper focuses on multimodal image fusion, being reflected in the title. In the main text, the proposed method is evaluated in two scenarios: infrared and visible image fusion and medical image fusion. In the supplementary materials, they further provide experiments on polarization image fusion. I am curious whether the applicable scenarios of the proposed method can be further expanded, such as the typical fusion of near-infrared and visible bands.\n- The experiments on polarization image fusion only provide visual results, and it would be better to add a quantitative evaluation.\n- I noticed that the proposed method separates the chrominance component and the brightness component, and then performs de-degradation on them separately. An explanation of why this operation is needed should be given. Perhaps an ablation experiment could more intuitively show the effect of this operation.\n- There are some minor typos, such as potential misspellings of dataset names in Tables 1 and 2. In addition, there seems to be a lack of underline on AG's second place.\n\nQUESTIONS:\n1. How were the degradation condition data constructed, were paired supervised datasets used or synthetic datasets?\n2. Has there been an attempt to evaluate the fusion on the fusion of near-infrared and visible bands?\n3. Could you provide the quantitative results of polarization image fusion?\n4. The separation of chrominance and brightness requires more explanation.\n\nLIMITATIONS:\nLimitations and broader impacts have been included.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an interactive framework that can exploit the intrinsic connection between image restoration and multi-modal image fusion.\nThe authors embed information fusion within the diffusion process and address the \"composite degradation challenge\" i.e., multi-modal information integration with\neffective information restoration from degradation like colour casts, noise, and improper lighting. Particularly, first, independent conditional diffusion models are applied\nto each modality with compound degradation -- the degradation removal priors are embedded into the encoder-decoder network. A fusion control module (FCM) sits in\nthe multi-step diffusion process to manage the integration of multi-modal features and remove degradation during T-step sampling. Next, to interactively enhance\nfocus on objects of interest during diffusion fusion, the authors designed a text-controlled fusion re-modulation strategy that incorporates a text and a zero-shot OWL-ViT to\nidentify the objects of interest. In other words, this step performs a secondary modulation with the built-in prior to enhance saliency.\n\nSTRENGTHS:\n- It is interesting to see the effect of combining image restoration and multi-modal image fusion in a single framework.\n - The proposed method is well-motivated and the authors provide a clear explanation of the method.\n - The Text-controlled fusion re-modulation strategy could be useful in many applications.\n - The authors provide the code in the supplementary material (although I have only dry run the code and not tested it).\n - Extensive experiments are conducted to validate the proposed method.\n - The authors provide ablation studies to show the effectiveness of each component of the proposed method.\n\nWEAKNESSES:\nFor now, I have minor concerns and mostly questions (as listed in the next section).\n - The authors should add a brief discussion on the competitors in supplementary material. For example, differences between TarDAL, DeFusion, LRRNet, DDFM, and MRFS.\n - Typo in Eq. 2: $\\Theta_{t}^{X^{B}}$ should be $\\Theta_{t}^{X^{b}}$.\n - Improve the caption of Figure 2. I had to read the entire paper to understand the figure (it should be self-explanatory).\n - Not much of a weakness, but the authors could improve the clarity of the paper if they added the tensor dimension of each variable in Figure 2.\n\nQUESTIONS:\n- In the proposed method, input visual image X is broken into brightness and chroma components. I wonder if this step is absolutely necessary -- or can we skip $\\eta^{c}_{\n    theta}$ and directly combine both $X$ and $Y$ as three-channel images $\\mathbb{R}^{H \\times W \\times 3}$.\n - What if I use InstructIR (for image restoration) followed by MaxFusion (for multi-modal fusion) -- how would it compare with the proposed method?\n    - (InstructIR) https://arxiv.org/pdf/2401.16468 | Github: https://github.com/mv-lab/InstructIR\n    - (MaxFusion) https://arxiv.org/pdf/2404.09977 | Github: https://github.com/Nithin-GK/MaxFusion\n - In the Limitation and Future work section, will a no-training approach be possible? For example, \"MaxFusion\"-like approach but with the proposed deep integration of image restoration and multi-modal fusion.\n\nLIMITATIONS:\nThe authors discussed limitation section in supplementary A.4 -- Particularly, Table S1 shows number of parameters and runtime. This is highly appreciated.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}