TeRF: Text-driven and Region-aware Flexible Visible and Infrared Image Fusion

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 05:19:06

Abstract: The fusion of visible and infrared images aims to produce high-quality fusion images with rich textures and salient target information. Existing methods lack interactivity and flexibility in the execution of fusion. It is unfeasible to express the requirements to modify the fusion effect, and the different regions in the source images are treated equally across the identical fusion model, which causes the fusion homogenization and low distinction. Besides, their pre-defined fusion strategies invariably lead to monotonous effects, which are insufficiently comprehensive. They fail to adequately consider data credibility, scene illumination, and noise degradation inherent in the source information. To address these issues, we propose the Text-driven and Region-aware Flexible visible and infrared image fusion, termed as TeRF. On the one hand, we propose a flexible image fusion framework with multiple large language and vision models, which facilitates the visual-text interaction. On the other hand, we aggregate comprehensive fine-tuning paradigms for the different fusion requirements to build a unified fine-tuning pipeline. It allows the linguistic selection of the regions and effects, yielding visually appealing fusion outcomes. Extensive experiments demonstrate the competitiveness of our method both qualitatively and quantitatively compared to existing state-of-the-art methods.

Main paper: 
 introduction: 
 Existing imaging technologies rely on specialized optical systems to capture and convert radiation information, which generates optical images with distinct characteristics. However, these images often provide a limited and partial modality of the diverse information presenting within the identical scene, thereby failing to adequately describe the entire scene. In this context, the image fusion technique has come into being with the objective of extracting pertinent and complementary information from diverse source images and amalgamating it into fused images. In this field, visible and infrared images fusion (VIF) stands out as one of the most widely adopted techniques [33, 35] . Visible images typically capture intricate spatial details and plentiful color in the scene, but are susceptible to environmental lighting conditions. Infrared images record the thermal radiation information to reflect the salient characteristics of targets, but barely carry scene textures and colors. Hence, VIF aims to generate high-quality images with pronounced saliency and intricate texture details, as depicted in Figure 1 . Harnessing these advantages, VIF technology holds promise for various applications, including target detection [18] , military surveillance [3] , autonomous driving [22] , and others [16] .To date, the process of visible and infrared image fusion can be delineated into three key stages: feature extraction, feature fusion, and image reconstruction. Initially, it is imperative to extract representative features from the source images by employing specific transformations or projections. Subsequently, the complementary features are seamlessly integrated by adopting appropriate fusion strategies. Finally, corresponding reverse projections are employed to transform the fused feature back to the common image domain, thereby facilitating the process of image reconstruction. Generally, traditional VIF methods rely on manually designed fusion strategies, which can be further divided into multi-scale transformation-based methods, sparse representation-based methods, subspace-based methods, saliency-based methods, and others, according to the applied transformation [32, 34] . However, these methods often fall short in addressing diverse working conditions, as their fusion performance is constrained by limited linear fitting capabilities.Over the past few years, deep learning has facilitated leaps and bounds in the visible and infrared fusion task owing to its powerful non-linear fitting ability. According to the structures of deep models, deep-learning-based methods can be categorized into convolution neural network (CNN)-based methods, auto-encoder (AE)-based methods, generative adversarial network (GAN)-based methods, transformer-based methods, and others. CNN-based methods [9, 14] leverage the remarkable local connectivity characteristics to explore the hierarchical patterns and features, which promotes the integration of complementary information to synthesize high-quality fused images. AE-based methods [36, 40] structure, where the front encoder seeks the efficient representation of the source images and the end decoder restores them with minimal loss. After that, the encoded representations of different source images are combined under the predefined fusion strategy before generating fused images through the decoder. GAN-based methods [15, 19, 23] utilize multiple generators and discriminators to explore the feature distributions of paired source images and perform appropriate constraints to manipulate the style and the content of the fused images. Transformer-based methods [17, 24, 27, 37] incorporate a novel attention mechanism that captures the long-range dependencies to focus on relevant parts of the source information while downplaying the irrelevant information. This effectively improves network efficiency and enhances convergence performance.Despite the effectiveness of deep-learning-based methods in integrating complementary information from heterogeneous modalities of visible and infrared images [11, 28, 43] , there still exist several limitations. On the one hand, interactivity and flexibility are absent in the execution of fusion. It is unavailable to express the requirements to modify the corresponding fusion effects since the models lack a command interface. Besides, existing methods uniformly process various regions of the source images, which leads to fusion homogenization and low distinction across the identical model. Essentially, it retains the generality but sacrifices the diversity of fusion results. On the other hand, the predefined fusion strategies are insufficiently comprehensive in the credibility of source information, refinement of lighting conditions, and noise removal. Specifically, current models struggle to effectively allocate the varying degrees of fusion tendency for different source images, presenting a singularity that cannot be applied to all scenarios. In addition, they lack adaptability in handling inappropriate light conditions, where texture and color fidelity deteriorate within the scene. Moreover, noise is inadequately taken into account as it inevitably exists and reduces the image quality. These issues lead to poor information reliability, weak information discovery ability, and low information fidelity, which seriously limits the scope of application and performance of existing fusion methods [7, 14, 18, 41] .To address these challenges, we integrate the advantages of pretrained large vision models, large language models, and multiple fine-tuning paradigms to achieve Text-driven and Region-aware Flexible visible and infrared image fusion, termed TeRF, as shown in Figure 2 . For interactivity and flexibility, we introduce the language model LLaMa [25] as a text analyzer, which leverages the in-context learning to accurately recognize the relevant prompts for segmentation targets and fine-tuning tasks. After that, we combine the vision models to achieve region-aware effects, where the GroundingDINO [13] and SAM [5] models cooperate for the finegrained semantic segmentation in a successive manner. Benefiting from the priors of the large models, text-driven interaction and region-aware flexibility can be effectively achieved. For the incompleteness of fusion strategies, a high-performance fusion backbone and a unified fine-tuning pipeline are devised. The former provides high-quality fusion results, serving as precursors with enhanced visual effects and improved detection performance. The latter leverages comprehensive fine-tuning paradigms to fully facilitate multi-task feasibility, which comprises the modification of fusion tendency, the adjustment of illumination, and the removal of noise. This effectively improves the completeness of the fusion strategies and the practicality in various scenarios. As illustrated in Figure 1 , our basic performance outperforms the representative methods, and we can linguistically modify the fusion effects on specified regions in a continuous manner(I-III). Our contributions are summarized below:• We propose a text-driven and region-aware framework for VIF with high interactivity and flexibility. It ensembles the large language and vision models for linguistically modifying the fusion effects of different regions. • A high-performance fusion backbone is devised to attain superior fusion precursors. Furthermore, a unified fine-tuning pipeline is constructed for the flexible fusion modification, which fully concerns the comprehensive fusion strategies.• Extensive experiments are conducted to demonstrate the superiority of our method in terms of interactivity, flexibility, and completeness both qualitatively and quantitatively. 
methodology: 
 For convenience and generality, the used notations are summarized here. Two heterologous source images including infrared image and visible image are denoted as 𝐼 𝑖𝑟 and 𝐼 𝑣𝑖 , respectively. The fused image generated by the pretrained fusion network F (•) is denoted as 𝐼 𝑓 , while the image acquired by fine-tuning pipeline T (•) is defined as 𝐼 𝑆 . The employed LLaMa for text parser is determined as 𝛹 (•). The applied vision model for text-assisted region segmentation is defined as R (•), which consists of GroundingDINO and SAM.The realization of interactivity and flexibility within TeRF is illustrated in Figure 3 . To start with, we define the instruction prompts with a few examples and demonstrations, thereby fine-tuning the LLaMa to accommodate entity recognition and text classification.After that, the predefined instruction prompts as well as input text commands 𝝋 are combined together to form the contextual information, which enables the large language model to generate the output in the specific format from the provided demonstration in the incontext learning manner. Subsequently, the segmentation prompts 𝝋 𝑠𝑒𝑔 and fine-tuning prompts 𝝋 𝑡𝑎𝑠𝑘 are split and simplified, which can be formulated as:EQUATIONMeanwhile, the fusion of visible and infrared images is also being implemented. The pre-trained fusion model is devoted to synthesizing the fused result 𝐼 𝑓 by integrating salient target information and rich texture information from the source images. In this way, the fused image potentially yields enhanced visual effects and improved detection performance. This process can be defined as:EQUATIONwhere 𝜃 F is the pretrained parameters of fusion network. Then the simplified prompts as well as fused images are served for both large vision models and fine-tuning pipeline. 𝝋 𝑠𝑒𝑔 is embedding into token by predefined bert-based encoder [2] at first. Subsequently, it assists the pre-trained vision models in locating the target and segmenting the pertinent regions, ultimately generating a set of text-related segmentation masks. Notably, this process is in a successive manner. At first, GroundingDINO simultaneously receives textual and visual signals, leveraging them to generate bounding boxes as visual cues. These cues provide a coarse localization of the target, facilitating subsequent fine-grained segmentation by the SAM model. Given that SAM lacks an interface for direct text-to-semantic segmentation, such a cascaded approach becomes necessary. The entire process can be denoted as:EQUATIONwhere 𝑀 𝑆 stands for the collected semantic masks. At the same time, 𝝋 𝑡𝑎𝑠𝑘 triggers the switch of the fine-tuning pipeline, where the task-related fine-tuning paradigm is activated for the modification of the fusion effect. It can be described as:EQUATIONwhere 𝐼 𝑖 𝑆 represents the 𝑖 𝑡ℎ fine-tuning image and 𝜃 𝑖 𝑆 are the learnable parameters for the specific tasks. At last, a set of fine-tuning images 𝐼 𝑆 along with the set of corresponding masks are aligned to merge the final outputs, as formulated below:EQUATIONwhere 𝑀 𝑖 𝑆 denotes the 𝑖 𝑡ℎ masks, 𝑁 represents the total number of masks, and the symbol ⊙ stands for the element-wise addition.As aforementioned, two major parts of designed networks are the predefined fusion network and fine-tuning pipeline.The former network designs are illustrated in Figure 4 . The infrared image 𝐼 𝑖𝑟 and visible image 𝐼 𝑣𝑖 are concatenated together and sent to the feature extraction module at first. It contains a convolutional layer with PReLU(•) as activation in the front and a two-layer convolutional layer with PReLU(•) in the end. Subsequently, three residual dense blocks are employed to perform hierarchical feature exploration. These features are then sent to the feature refinement module, which comprises three consecutive convolutional layers and a ReLU(•) activation function at the end. After that, the preceding features are expended in the channel dimension, which are packed into transformer-based attention blocks to enhance the relevant features and weaken the others. At last, features flow into the feature integration module, which contains a front convolutional layer, a pixel shuffle layer [10] , and an end convolutional layer.The latter network designs are represented in Figure 5 . The basic element of the fine-tuning pipeline is the lightweight fine-tuning unit, which is constituted by three repeated convolutional layers with a ReLU(•) activation at the head and a final convolutional layer at the tail. Depending on different fine-tuning tasks, a variable number of units are employed. In cases of a tendency bias towards the fusion effect, it suffices to unlock the predefined fusion network and perform minor optimization. In cases of illumination adjustment, three fine-tuning units are employed to improve the quality of exposure. As for denoising tasks, a single fine-tuning unit is introduced for noise estimation and removal.The purpose of VIF is to retain the ample spatial information and the salient information of the source images. Hence, we strip away the spatial and color attributes from RGB space to YCrCb space. Accordingly, we preserve the maximum intensity and gradient of the source images on the 𝑌 channel in order to fulfill the purpose, which can be expressed as below:EQUATIONwhere 𝑌 ( •) is the Y-channel of each image, ∇ stands for the Sobel operation, and ∥ • ∥ 1 represents 𝑙 1 -norm. To ensure minimal chromatic aberration in the fused images, we constrain them to align closely with visible images in the other two channels. It can be formulated as:EQUATIONwhere 𝐶𝑟 ( •) and 𝐶𝑏 ( •) are the Cr-channel and Cb-channel of the corresponding source image, respectively. Besides, we devise a structural consistency loss to constrain the structures of fused images and source images to be similar, which can be defined as:EQUATIONwhere SSIM(•) computes the structural similarity of the image pair.Concerning the above three points, the overall loss functions can be formulated as below:EQUATIONwhere 𝛼, 𝛽 and 𝜂 are the trade-off hyper-parameters. EQUATIONEQUATIONCorrespondingly, for the inclination of the infrared attribute, they are redefined as L 𝑖𝑟 * 𝑌 and L 𝑖𝑟 * ssim :EQUATIONEQUATION2.4.2 Enhance or Weaken The Illumination. Following the classical Retinex theory [6] , an image can be decomposed into the product of a reflectance map 𝑅 and an illuminance map 𝐿, with the addition of noise 𝑁 . Among them, 𝑅 describes the intrinsic property of the captured scene, and 𝐿 represents the illumination intensity emitted by the light source. Consequently, within the framework of RRDNet [42] , we employ three independent fine-tuning units to extract these components and appropriately adjust the illuminance map. The devised loss can be formulated as below:EQUATIONEQUATIONEQUATIONwhere 𝑆 0 is the maximum value of the RGB channel, 𝐺 • (•) is the gaussian filter. ∇ 𝑥 and ∇ 𝑦 are the Sobel operators in the horizon and vertical directions, respectively. 𝑤 𝑛 equals normalization of 1/ 𝐼 𝑓 ⊙ (∇ 𝑥 𝑅) 2 ⊙ (∇ 𝑦 𝑅) 2 . The total loss is denoted as:EQUATIONwhere 𝜆 𝑡 , 𝜆 𝑟 and 𝜆 𝑛 are the trade-off parameters. After the extraction of the key components, we perform gamma correction on the illumination map for the reconstruction of the enhanced image.As the pretrained fusion model cannot provide denoising priors, a fine-tuning unit is introduced additionally. Following the framework of ZSN2N [21] , the unit is optimized using the following loss functions to achieve noise removal:EQUATIONEQUATION)where 𝑥 is a noise image, 𝐷 1 and 𝐷 2 are the manually designed kernels for down-sample operation to derive the paired noise image, 𝜃 is the parameters of the fine-tuning unit 𝑓 . The total loss functions L 𝑁 2𝑁 can be expressed as:EQUATIONwhere L res yields the residual information constraints and L cons provides the consistency constraints for better performance as well as avoiding the over-fitting.When the text instructions involve different requirements for effects or regions, the fine-tuning units are reinitialized and optimized accordingly. However, when it is required to conduct identical fine-tuning tasks for the same region consecutively, it is imperative to guarantee that the effects get better as the increase of fine-tuning iterations.Regarding the inclination of the source attribute, the fusion network reloads the previous model updated in the last fine-tuning iteration and continues optimization for the fixed epochs. For illumination adjustment, as the reflectance map of the image is independent and stays unchanged, It simply necessitates performing corresponding transformations on the illumination map multiple times. Concerning the noise removal, the original noise prior remains unchanged, but more optimization epochs are performed to continuously improve the performance of the reloaded model. In summary, comprehensive fusion strategies are fully developed. FusionGAN [20] , DIDFuse [39] , Res2Fusion [29] , RFNNest [8] , SD-Net [31] , LRRNet [9] , TarDAL [12] . For quantitative assessment, five metrics are selected to objectively evaluate the fusion performance, including average gradient (AG) [1] , structural similarity index (SSIM) [26] , visual information fidelity (VIF) [4] , mutual information (MI), and gradient-based quality index 𝑄 𝐴𝐵/𝐹 [30] .  
experimental results and analysis 3.1 experiment settings: 
 33.2.1 Qualitative Comparisons. The visual comparisons over three datasets are depicted in Figure 6 . Evidently, FusionGAN emphasizes brightness overly at the expense of fine details. RFNNest fails to preserve sufficient spatial information and yields ambiguous effects. LRRNet and SDNet effectively retain the texture details albeit at the expense of color fidelity, which leads to diminished luminance. TarDAL and DDFM maintain the balance of color information and salient information better, but the contrast of the fusion result is not high enough. In particular, when visible information and infrared information are completely opposite, the fusion strategy adopted by DIDFuse leads to information cancellation and distortion. Res2Fusion insufficiently integrates the infrared information, which reduces the saliency of targets. In contrast, our method effectively retains the ample textures within the scenes while rationally integrating salient information from the source images. Moreover, the applied fine-tuning paradigms further enhance the visual effect of the fusion results with the assistance of texts, which showcases flexibility and interactivity. Consequently, our method achieves favorable results and outperforms others.To further validate the competitiveness of our approach, we conduct the quantitative evaluation of five metrics. For fairness, we exclusively compare the performance of our baseline with other methods, and the impacts of fine-tuning tasks are individually discussed in the next section. The quantitative results are reported over three datasets in Table 1 . Upon analysis of the results, our basic fusion model achieves the top scores of all five metrics on both MSRS and LLVIP datasets. It indicates that our method retains rich gradient information and maintains the highest fidelity of source information. However, our baseline attains the best performance across AF, VIF, MI, and 𝑄 𝐴𝐵/𝐹 but ranks second in SSIM on RoadScene datasets. This may be attributed to the significant disparity between visible and infrared image information, making it difficult for the fused image to simultaneously accommodate the structural similarities of both. In general, our method surpasses others and is the most competitive.We conducted a series of experiments to assess the impacts of finetuning tasks on the RoadScene datasets. The visual comparisons are illustrated in Figure 7 , while the metrics evaluation is presented in Table 2 . Each task is performed once across the entire region As for the inclination of source attributes, the gradient-related indicators (e.g. AG, SF, 𝑄 𝐴𝐵/𝐹 ) of (I) increase, while these of (II) decrease. It indicates that the brightness changes of infrared images are drastic on this dataset. In addition, EN and SD also keep the same change in (I) and (II). Since fine-tuning approaches are not for neutral fusion effects, the value of SSIM inevitably reduces. For the illumination adjustment, the two fine-tuning methods are completely opposites. All the metrics expect for SSIM, get lower in (III) while higher in (IV). Compared to the baseline, enhancing illumination leads to a decrease in these metrics as most scenes in the datasets may be overexposed. Conversely, reducing illumination leads to a more balanced distribution of information. Similarly, both approaches make the fused information deviate from the source information, inevitably leading to a decrease in SSIM. Regarding noise removal, it is observed that the introduction of noise significantly deteriorates the image quality as the value of most metrics gets worse. However, the increase in AG and SF can be attributed to the alteration of intensity coherence caused by noise. As fusion can Increase the brightness of the trees, raise the luminance of the trees, boost the brightness level of the trees.Lower the luminance of the trees, darken the trees area, reduce the brightness of the trees.Make people's fusion effect bias towards infrared perception, make person more salient, promote the infrared fusion effect of people.Bias the fusion effect of person towards visible light, enhance the person's detail, direct the visible attributes of person.Remove noise from the road path, Clean up the road path by reducing noise, eliminate noise from the road path. alter the priors of noise originating from the source image, fusing with the denoised source image is better than directly denoising the noisy fused image. Credibly, noise removal can restore most metrics and visual effects.Due to the excessive number of regions in the image, artificial oneto-one text modulation is not feasible. Therefore, we present qualitative results in Figures 8 and 9 for two different configurations. One involves consecutively employing the same fine-tuning effect for the same region with similar texts, the other is fine-tuning different areas randomly with different texts. It is shown that our approach can accurately analyze segmentation targets and fine-tuning tasks from different text representations. When performing the same finetuning tasks repeatedly, it can progressively enhance the effects. In addition, when dealing with multi-objective fine-tuning tasks, it can harmoniously integral different effects of different objectives.In general, the high-quality fine-tuning results verify that TeRF achieves flexible image fusion in handling diverse tasks and targets.Furthermore, we conduct the efficiency experiments to record test time consumption and model parameters for all methods, as detailed in Table 3 . The majority of deep methods exhibit high speed in processing test image samples. However, the diffusion-based method DDFM necessitates an amount of time for optimizing by performing multi-step noise inference. In comparison, our baseline demonstrates superior performance albeit at the expense of parameters and time, which can be attributed to the incorporation of multiple dense connections and the transformer-based attention mechanism. Hence, it is generally acceptable. In addition, we also provide the running time for each fine-tuning task. 𝐹 ×0 represents the modification of the fusion attributes without fine-tuning unit. 𝐹 ×1 refers to the noise removal task with single unit and 𝐹 ×3 employs three units to adjust the illumination. Obviously, the time cost for fine-tuning is relatively high. Because model training and testing are performed as a whole, the running time includes both of them. However, other deep learning methods require extensive pre-training on large datasets before conducting fast testing. Precision Recall mAP @0.50 @0.75 @[0. As aforementioned, we leverage the prior within large vision models to achieve region-aware fusion. It may potentially introduce unfairness in comparing the performance of high-level vision tasks using fine-tuning results. Therefore, We compare the detection performance of our baseline with other methods to verify the improvement of downstream fusion performance on high-level tasks.As source attribute inclination alters the characteristics of fused images, the quality of their detection performance largely depends on the original source images. We utilize general text prompts for the illumination adjustments of primary targets and corresponding background to verify priors from large models as well as fine-tuning can promote detection performance, which is denoted as Ours * . We adopted YOLOv8 1 as the object detection backbone and finetuning it on the LLVIP datasets. The qualitative comparisons are displayed in Figure 10 . It is obvious that our baseline has higher detection performance, while other methods miss detections or have lower confidence. In addition, the fine-tuning results improve the detection performance as well as visual appeal. Regarding the quantitative assessments, five metrics are provided, including precision rate, recall rate, and mean average precision (mAP) with different threshold values. The results are presented in Table 4 . Clearly, our method achieves the highest scores in all five provided metrics, which indicates that our method is the most competitive in terms of performance for object detection. In general, the fine-tuning as well as the priors from the large models improves the detection performance further. 
conclusion: 
 This work proposes a text-driven and region-aware flexible visible and infrared image fusion, termed as TeRF. Multiple large models including LLaMa, GroundingDINO, and SAM, are encompassed to improve the interactivity and flexibility of VIF. Besides, a highperformance fusion network and a fine-tuning pipeline are devised for comprehensive fusion strategies. The former component provides visually appealing fusion precursors with dense connections and the transformer-based attention mechanism. The latter part fully concerns the data credibility, scene illumination, and noise degradation inherent in the source information with unified structures, which improve the information reliability and fidelity. All in all, TeRF allows linguistically segmenting the specific region and modifying the fusion effect accordingly with language instructions.  
 
 Appendix:  


Decision: After rebuttal, all the 4 reviews are positive, so I recommend the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: A new method combining large language and visual models is designed to perform visible and infrared image fusion of specific areas with text prompts. It considers a complete fusion strategy of data credibility, scene lighting and noise degradation, and builds a unified fine-tuning pipeline. Extensive experiments have demonstrated the advantages of the proposed method over the latest SOTA methods.
strength_weakness: strength_weakness: + The paper presents a competitive VIF network with high performance that surpasses existing SOTA methods.
+ The paper combines multiple large models and coordinates the processing of multi-modal data, making the fusion effect and region linguistically controllable.
+ The overall fine-tuning framework is flexible, interactive and easy to expand, and has the potential to synthesize text-fusion image pairs.
+ The paper is easy to follow. The motivations and problem statement are explained well.
+ The work can be reproduced based on the provided code.
limitations: limitations: - It is not clear whether the fine-tuning pipeline contains the fusion network or not. In Figure 3 and Figure 5, it has some conflicts.
- More details about the evaluation metrics are helpful for the readers to properly understand the analysis.
- The paper uses three datasets for experiments, but does not cite related articles.
- Since the values of the same metrics are quite close, it might be helpful for the readers if the authors provide also the variance or a statistical test to prove that their method is truly better.
- In the experiment on high-level task, it is necessary to add the baseline experimental results, because the introduction of text may lead to unfair comparison.
- The font size in some of the figures, such as Figure 7 and Figure 10, is too small.
paper_summary: paper_summary: This paper presents a method for fusing visible and infrared images with high flexibility and interactivity, allowing for linguistic modification of the fusion effect in specific areas according to textual instructions. The authors employ the large language model LLaMa to analyze text prompts and utilize large vision models, GroundingDINO and SAM, to identify target regions. Their proposed pipeline incorporates unified fine-tuning to achieve various fusion effects. Overall, the idea of using the large language model and large vision model for image fusion is interesting. Besides, it achieves a relatively good performance along with flexible fusion results.
strength_weakness: strength_weakness: 1. The idea of applying multiple large models to assist the fusion process is interesting, as well as the use of text prompts in image fusion. The priors within the large models are fully explored to achieve flexible and interactive fusion.
2. This work addresses practical issues in the fusion of infrared and visible images, including the inability to specify targets/areas for customized fusion, incomplete fusion strategies, and the limitation of achieving a diverse range of fusion effects.
3. It is easy to follow and understand. The experimental results support the effectiveness of the presented method and show relatively good fusion outcomes.
limitations: limitations: 1. A comparison of the generalization performance of the fusion model is not provided. It would be better to carry out the experiment on other datasets, (e.g. FMB dataset [1]).
2. In Tab. 3, are the parameters in the large models being computed? And it would be better to provide the Flops.
3. The implementation details are absent when analyzing text prompts. For example, instruction prompts seem to play a big role in LLaMa, but the related information is lacking.
4. Comparing the power of the GroundingDINO and the YOLOv8 performance will be good. It can provide more inspiration for multimodal image fusion. 
5. There are some recent works about multi-modality image fusion that have not been surveyed. Their information is as [2-4].

[1] "Multi-interactive feature learning and a full-time multi-modality benchmark for image fusion and segmentation." ICCV, 2023.

[2] "Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration." IJCAI, 2022.

[3] "An interactively reinforced paradigm for joint infrared-visible image fusion and saliency object detection." Information Fusion, 2023.

[4] "ReCoNet: Recurrent Correction Network for Fast and Efficient Multi-modality Image Fusion." ECCV, 2022.
paper_summary: paper_summary: This manuscript proposes the Text-driven and Region-aware Flexible visible and infrared image fusion, termed as TeRF to address the issues lying in existing methods, such as the lack of interactivity and flexibility. Authors propose a flexible image fusion framework with multiple large language and vision models, which facilitates the visual-text interaction and aggregate comprehensive fine-tuning paradigms for the different fusion requirements to build a unified fine-tuning pipeline. On the whole, the problem mentioned in the paper is interesting, but the novelty of the devised method is weak.
strength_weakness: strength_weakness: 1. For interactivity and flexibility, authors introduce the language model LLaMa as a text analyzer, which leverages the in-context learning to accurately recognize the relevant prompts for segmentation targets and fine-tuning tasks. The proposed method combining both text and image information together is suitable for more task-oriented image fusion results.
limitations: limitations: 1. Authors define the instruction prompts with a few examples and demonstrations, thereby fine-tuning the LLaMa to accommodate entity recognition and text classification. Since text commands will be input to the LLaMa, the difference between the instruction prompts and the text commands should be clarified. Please give more detailed descriptions of the instruction prompts.
2. The total number of masks N needs a more explicit definition. Does each mask represent one specific object? If so, why the masks shown in Figure 3 are the same? The absence of clear explanation about the mask as well as its settings make readers confused, which significantly reduces the readability of the paper.
3. Experiments to validate the effectiveness of the proposed prompts are insufficient. Ablation study on \phi_seg and \phi_task should be shown to verify the idea of the paper.
4. In Figure 3, there two inputs \phi_task and I_f of the Fine-tuning Pipeline. However in Figure 5, there is only one input \phi_task shown. Please make them consistent.
paper_summary: paper_summary: This paper tries to use the text information to improve the flexibility of image fusion methods. According to the results, I think this purpose is achieved by employing the llama. The motivation sounds interesting and reasonable, and the performance and visualisations are also good.
However, there are some essential parts are missed. Please refer to the weakness.
strength_weakness: strength_weakness: 1. The motivation seems interesting，and also demonstrated effectively by employing the language model. 
2. The design is reasonable and from the visualisations, I think it can correctly achieve the purpose..
3. The performance is also good.
4. Most of the experiments are siginificant and can illustrate the purpose.
limitations: limitations: 1. The ablation study is missing, which makes the readers confused about the detail contributions of each component.
2. The related works introduced in the Introduction section are not comprehensive, some methods involving the text information into the image fusion task have already been proposed. Such as the TextFusion, which is currently publicly available on Arxiv.
3. The writing should be improved. I can find some mistakes like in line 96, 'feature' should be replaced by 'features', and in fig.2 'Source image' should be 'Source images' since there are two inputs.
4. Achieving controable image fusion is interesting, but it's still less meaningful if the discussion stops here. I suggest the authors to think more about how this helps downstream tasks, which will be more important in my view.