MMDRFuse: Distilled Mini-Model with Dynamic Refresh for Multi-Modality Image Fusion

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-06 05:01:16

Abstract: In recent years, Multi-Modality Image Fusion (MMIF) has been applied to many fields, which has attracted many scholars to endeavour to improve the fusion performance. However, the prevailing focus has predominantly been on the architecture design, rather than the training strategies. As a low-level vision task, image fusion is supposed to quickly deliver output images for observing and supporting downstream tasks. Thus, superfluous computational and storage overheads should be avoided. In this work, a lightweight Distilled Mini-Model with a Dynamic Refresh strategy (MMDRFuse) is proposed to achieve this objective. To pursue model parsimony, an extremely small convolutional network with a total of 113 trainable parameters (0.44 KB) is obtained by three carefully designed supervisions. First, digestible distillation is constructed by emphasising external spatial feature consistency, delivering soft supervision with balanced details and saliency for the target network. Second, we develop a comprehensive loss to balance the pixel, gradient, and perception clues from the source images. Third, an innovative dynamic refresh training strategy is used to collaborate history parameters and current supervision during training, together with an adaptive adjust function to optimise the fusion network. Extensive experiments on several public datasets demonstrate that our method exhibits promising advantages in terms of model efficiency and complexity, with superior performance in multiple image fusion tasks and downstream pedestrian detection application.

Main paper: 
 introduction: 
 Images captured by different physical sensors under diverse conditions contain unique attributes, which challenge the design of a general image processing system. Drawing on this, to unify the visual pixel distribution at the image level, image fusion provides a solution to combine source images into a single and comprehensive output image [11] . Considering the input configurations, image fusion tasks are ranging from digital image fusion [2, 46] , multi-modality image fusion (MMIF) [38] , to remote sensing image fusion [12, 40] . In particular, MMIF [19, 49, 50] has attracted wide attention in recent decades, which includes Infrared and Visible Image Fusion (IVIF) [21, 31] , and Medical Image Fusion (MIF) [43] .In terms of the IVIF task, the target sources are visible and infrared images [49] . Specifically, the visible modality, captured by optical devices such as digital cameras, is expert at preserving textural details and colours, while it is susceptible to illumination conditions. On the other hand, the infrared modality, derived from infrared imaging devices, captures the inherent heat radiation emitted by living beings or powered objects. This modality is obtained based on the variation in radiation intensity between the target and its surroundings, lacking the supportive colour and texture contained in the visible spectrum. The IVIF task aims to generate an informative image that leverages the strengths of both, enhancing the overall visual quality and serving various following processing demands [11, 50] . Specifically, the fused images can be applied to several downstream tasks, e.g., semantic segmentation [20, 35] , object tracking [4, 13] , object detection [8, 34] , and saliency detection [27] . As for the MIF task, by combing images obtained through different medical imaging devices (CT [3] , MRI, PET, SPETC e.t.c.), MIF can provide a clearer view of both the structure and functional information of the human tissues and organs, supporting more precious disease diagnosis [43, 50] .After decades of study, the performance of IVIF has been advanced. In contrast to traditional approaches that rely on multi-scale decomposition (MSD) [52] , sparse representation (SR) [45] , or lowrank representation (LRR) [14] , to extract handcrafted features, more robust representations can be obtained by deep solutions, such as convolutional neural networks (CNNs), Transformers, and their hybrid versions [11] . To pursue better visual effects, various functional blocks have been developed, e.g., aggregated residual dense block [21] and gradient residual dense block [31] . To facilitate smooth training, residual connections [15, 17] and skip connections [37] are adopted to prevent gradient vanishing or distortion in the generated images. Besides modifying network modules, more efforts have been explored to formulate fusion as a generation task. For instance, [19, 23] , utilise a generator to obtain the fused image, accompanied by two discriminators to guarantee its fidelity compared to the sources. Essentially, the generator contains multiple CNN layers, and this adversarial training scheme is easy to crash and hard to control. Such computation burden is also suffered by the diffusion solution [50] , which obtains stable and controllable high-quality fused images without discriminator but requires more computation resources than previous approaches.Despite the performance promotion achieved by the above attempts, all the involved network designs suffer from excessive complexity and redundancy. Therefore, striking a balance between model performance and resource requirements is a pressing issue that needs to be addressed. Motivated by knowledge distillation [9, 44] , it is straightforward to compress the teacher parameters into a relatively smaller student model. Essentially, we propose to establish a strong teacher model with powerful feature extraction and reconstruction ability. To deliver effective supervision, as shown in Figure 1 , we adopt three dedicated designs to integrate supervision signals from teacher, source images, and history records.Specifically, a digestible distillation strategy is proposed to relax the strict consistency constraints between the student network and the teacher network on the intermediate feature dimension. We construct groups of isomorphic transformation modules within both the teacher and student networks individually. Based on the output characteristics at each stage, we match them from the feature end and output end. To exploit the source input images, intensity details, edge gradients, and perception semantics are comprehensively reflected by our loss function. Furthermore, to reinforce the cues embedded in the historical parameters of the training process, we propose the dynamic refresh strategy. A set of dual evaluation metrics are devised to distinguish whether the current cues are useful or not. The useful one demonstrates that our training trajectory is correct at present and can be refreshed as internal supervision signals, while the inadequate one can be optimized by the above signals.In this paper, we design a Distilled Mini-Model with Dynamic Refresh for Multi-Modality Image Fusion (MMDRFuse). As shown in Figure 1 , our MMDRFuse achieves promising fusion performance, while it is significantly smaller than other SOTA models, with only 113 trainable parameters (0.44 KB). The contributions of our work can be summarised as follows:• An end-to-end fusion model with a total of 113 trainable parameters and 0.44KB size, which can efficiently facilitate fusion and support downstream tasks. This mini model unveils that boosting image fusion performance benefits more from providing suitable supervision rather than stacking complex networks. • A digestible distillation strategy to relax the feature-level consistency, softening the supervision from the teacher model. • A comprehensive loss function to preserve intrinsic clues from source inputs, collaborating pixels, gradients, and perceptions. • A dynamic refresh strategy to effectively manage the historical states of parameters during training, endowed with a dual evaluation metric system to adaptively refine supervision signals towards the correct direction. • Promising performance in terms of fusion quality and efficiency against SOTA methods both in IVIF and MIF tasks. The obtained 113 parameters can even support downstream pedestrian detection without any semantic information as input. 
related work 2.1 advanced mmif formulations: 
 Recent studies in MMIF can be broadly categorised into Auto-Encoder (AE) paradigm [6, 15, 17, 18, 49] , GAN-based paradigm [19, 24] , Transformer-based paradigm [22, 49] , downstream taskdriven paradigm [19, 31, 48] , and text-driven paradigm [6] . Among the involved techniques, AE exploits an encoder to extract features from source images and a decoder to obtain fused images by reconstructing the latent features. To get rid of the limitations of handcrafted fusion rules, [17] devises a residual fusion network to smooth the training stage. To alleviate computation, [49] uses Lite-Transformer to fuse base features and detail features come from different modalities. Besides network design, [37] directly conducts an end-to-end fusion network with a memory unit to allocate effective supervision signals.To better support downstream tasks, [19] proposes a bilevel optimisation formulation based on GANs and Object Detection (OD) network, forming a cooperative training scheme to yield optimal network parameters with fast inference for both tasks. Besides, [31] Figure 2 : Illusatration of our distillation process. TConv1, TConv2, ... , TConv13 represent the convolutional layers in the teacher network. SConv1, SConv2 represent the convolutional layers in the student network. TOutput and SOutput denote their outputs respectively. introduces a real-time segmentation model to grind the semantic information for the fused images. Although linking the downstream tasks with image fusion can obtain guided semantics, such a solution sacrifices too much in computation, running time, and storage space. To break away from existing modelling techniques, we investigate the possibility of designing internal and external supervision signals to serve an extremely tiny model. In particular, the internal and external supervision denote the historical state of the model itself and the guidance of a powerful teacher model, respectively. Such a combination is typically controlled by a dynamic refresh strategy to harmonise the training of our 0.44KB mini model.Knowledge Distillation is introduced by [9] to tackle the cumbersome of deploying computationally expensive models. According to the locations of processing knowledge, it can be categorised into three branches, i.e., response-level [9, 47] , feature-level [5, 29, 30, 44, 51] and relation-level [25, 42] settings. Typically, [9] distils knowledge by aligning the student's output with the teacher's output while [29] additionally focuses on the consistency of intermediate features between teacher and student. Following this setting, advanced studies are dedicated to paying attention to feature-based distillation. In terms of formulating feature consistency, [44] proves that activation-based attention transfer and gradient-based spatial attention transfer are more effective than full-activation transfer. Considering feature diversity, [5] proposes to gradually learn the low-level feature maps after the high-level consistency is obtained. However, this knowledge delivery solution still costs a lot of resources during the training process.Given that the structure of our student network is too simple (113 parameters) to directly learn from a powerful teacher, similar to feeding a baby, how to transfer digestible supervision from the teacher to the student poses significant challenges. Drawing on this, we feed the student with more finely processed knowledge by combining feature-based and response-based distillation within isomorphic transformation modules. This process can also be understood as a mother carefully preparing swallowable pureed food for her child. 
approach: mmdrfuse 3.1 digestible distillation: 
 In essence, the ultimate goal of our MMDRFuse is to obtain a mini model with satisfactory fusion performance and downstream supportive power. It is impossible to completely depend on directly training the mini model itself, which is greatly limited by its parameter volume. Accordingly, we first train a teacher model, which is utilized to learn considerable feature extraction and reconstruction ability. Furthermore, the key of teacher module is to as a transportation media, converting high-level abstract feature information into fragments that student can digest.Existing observation unveils that aligning knowledge at the output end is too violent for the student model. In particular, different from high-level visual tracks, such as classification and detection, the output of a fusion system is low-level pixels, which challenges the strictness of teacher guidance. Therefore, we propose to add a buffer before the final output to transfer the supervision that can be digested better. At the same time, feature maps of the middle layer is supposed to be consistently distributed as the teacher model [44] , which can help to produce the expected output.We adopt a relatively complex teacher network to obtain a model with robust fusion ability. This network incorporates a densely connected network (DCN) responsible for extracting deep image features comprising 64 channels. Additionally, it integrates two convolutional layers to generate four-channel feature maps, alongside a single-channel image. By contrast, our student model only comprises two convolutional layers. The first one is used to extract feature maps with four channels and the latter functions as a decoder for producing the fused image. We utilise each output of the student network to implement feature-based and response-based distillation with the last two outputs of the teacher, respectively. Figure 2 provides an illustration of the network architecture and distillation mechanism.To ensure that the student outputs closely resemble those of the teacher network, we encourage consistency between their outputs at the pixel level. Specifically, the feature-based distillation and the response-based distillation can be formulated as:EQUATIONwhere 𝑣𝑒𝑐 (•) denotes the vectorisation operation. 𝐹 (𝑥) = 𝐶 𝑖=1 𝑥 𝑖 is a spatial mapping function used to conduct attention mapping across channel dimensions. 𝐹 (𝑥) comprehensively considers all channels from 𝑥 by distributing average weights to each spatial region. On the other hand, for the feature-based distillation, 𝑆 1 ∈ 𝑅 𝐻 ×𝑊 ×4 and 𝑇 1 ∈ 𝑅 𝐻 ×𝑊 ×4 are the extracted features of the teacher network and student network, respectively. For the response-based distillation, 𝑆 2 ∈ 𝑅 𝐻 ×𝑊 ×1 and 𝑇 2 ∈ 𝑅 𝐻 ×𝑊 ×1 are the output images of these two networks. As these output feature maps and images share an identical number of channels, the teacher network can effectively impart knowledge to the student module at the same semantic level.In this section, we introduce our loss function for optimising the fusion network from the source inputs. Firstly, we expect the outputs to be similar to source images at the pixel level. Specifically, for the multimodal image fusion task, visible or MRI images usually contain information about the light condition with texture details, while infrared or PET images tend to present significant thermal and functional information. Thus, an intensity loss function [31] is designed to generate fused images that share the same pixel distribution with source inputs, i.e.EQUATIONwhere 𝐼 ir and 𝐼 vis denote the source infrared and visible images, 𝑂 represents the output of the fusion network. Notably, if source images are under normal light conditions, the intensity loss function can help to maintain detailed targets, clear structure, etc. However, if the light condition in texture areas is darker than the infrared image, the impact is dominated by the latter. Thus, it seems not enough to rely solely on the intensity loss function. This phenomenon can be observed by the examples of Figure 5 . Therefore, we add a maximum gradient loss to help transfer gradient information mainly from the visible image, it is defined as:EQUATIONwhere ∥ • ∥ 𝐹 represents the Frobenius norm,▽ represents gradient operator. Furthermore, to better fuse fine-grained features from source images, we employ a normalised VGG-19 network [21] to extract hierarchical features of input and output images, formulating a feature-based perception loss. As depicted in Figure 3 , the feature maps in the shallow layers exhibit clearer edge texture details and gradient information, while the feature maps in deep layers become blurred, encompassing higher-level semantic information, abstract contexts, and salient structure information. This information is a significant factor regarding the fusion performance [37] . Hence, the perception loss is formulated as:EQUATIONwhere 𝐷 𝑖 represents the number of channels of extraction layer 𝑖, 𝜙 𝑖,𝑗 represents 𝑗 -𝑡ℎ channel of accordingly 𝑖 -𝑡ℎ feature map. Finally, the above components collectively constitute the comprehensive loss function:EQUATIONBy adding the maximum gradient loss and maximum perception loss, our fusion network not only focuses on regions with high brightness but also pays attention to texture information in dark areas, which cannot be achieved solely by intensity loss. At the same time, without intensity loss, we can not retain salient targets only by gradient loss and perception loss. Hence, the combination of these loss functions can complement each other's weaknesses while keeping their respective strengths.Traditional methods often directly discard the states of parameters from intermediate iterations during the training process, which, we argue, can be properly utilised to serve as supervision signals.However, useful information can not be guaranteed in the intermediate iterations, sometimes including noises, artefacts and so on. Hence, we need to identify the effectiveness of them. We devise a dual evaluation system to identify them and further utilise the advantageous parameters. We call this process as dynamic refresh, Figure 4 illustrates the basic process. In particular, dynamic refresh is applied throughout the entire training process, and we adopt two metrics to measure whether it is helpful to fusion. Image quality assessment structural similarity (SSIM) [36] and Gradient Magnitude Similarity Deviation (GMSD) [41] are involved as the metrics. SSIM measures the previous iterations from the perspective of structure, illuminance, and contrast, while GMSD highlights texture details. Accordingly, we measure every output during the training process and keep the two best outputs according to two evaluation metrics, which correspond to the green database and the yellow database in Figure 4 , respectively. The evaluation process can be described as follows:EQUATIONEQUATIONwhere 𝑂 𝑐𝑢𝑟 symbols the output of current epoch, 𝑂 𝑏𝑠 symbols the historical output with the best SSIM value, 𝑂 𝑏𝑔 symbols the historical output with the best GMSD value. 𝑆 𝑏𝑠 ,𝑆 𝑐𝑢𝑟 represent the SSIM value of the best history output and the current output, respectively. 𝐺 𝑏𝑔 ,𝐺 𝑐𝑢𝑟 represent the GMSD value of the best history output and the current output, respectively. By calculating SSIM and GMSD values of the current output and the two best historic outputs, once the value of the current output is higher than the recorded outputs, they will be replaced by the current output, which achieves dynamic update. The refreshing process can be described as follows:EQUATIONOn the contrary, the current output can learn from historic outputs by following refresh loss:EQUATIONEQUATION) Motivated by the phenomenon presented in Figure 3 , we combine perception loss of the last two feature maps before max-pooling operation with intensity loss function, to collectively serve as supervision signals by the item of 𝐿 𝑠 . Similarly, we combine perception loss of the first three feature maps before the max-pooling operation with maximum gradient loss function, to collectively serve as supervision signals by the item of 𝐿 𝑔 . 𝐿 𝑠 can help to fuse context and spatial structure, brightness, and contrast information from 𝑂 𝑏𝑠 , while 𝐿 𝑔 can help to fuse texture detail and gradient information from 𝑂 𝑏𝑔 . Furthermore, we measure the gap between 𝑆 𝑏𝑠 and 𝑆 𝑐𝑢𝑟 , as well as 𝐺 𝑏𝑔 and 𝐺 𝑐𝑢𝑟 to serve as two self-adaptive coefficient:EQUATIONHence, our dynamic refresh loss can be described as follows:EQUATIONFinally, we use comprehensive loss and refresh loss to collectively train our teacher model and further combined with distillation loss to train our student model. The total loss function can be described as follows:EQUATION 
evaluation 4.1 setup: 
 4.1.1 Datasets and Metrics. We perform MMDRFuse on three types of fusion tasks: Infrared and Visible Image Fusion (IVIF), Medical Image Fusion (MIF), and Pedestrian Detection (PD). We select six SOTA methods to compare with our method, including TextFusion [6] , MUFusion [37] , LRRNet [16] , MetaFusion [48] , DeFusion [18] , and TarDAL [19] . To better prove the generalisation ability of our design, we only train our model (both teacher and student) on the IVIF task, and directly apply the trained model to other tasks. The training dataset is MSRS [32] , we select 1083 image pairs from it, and they are cropped into 128×128 image patches, ending with 16245 image pairs. For the MMIF task, we conduct test experiments on three datasets: MSRS, LLVIP [10] , and RoadScene [39] . The image numbers of them are 361, 250, and 50, respectively. For the MIF task, we directly test the model on MRI-SPECT and MRI-PET, which include 73 and 42 image pairs, respectively. For the PD task, we trained six SOTA methods and our method on 2000 image pairs from the MSRS dataset, and then test the detection effect on 250 image pairs from the LLVIP dataset. We perform PD experiments by using the Yolov5 as a detector to evaluate the pedestrian detection performance with the value of mAP@.5:.95, which provides a more comprehensive evaluation by calculating the average precision according to the IOU threshold from 0.5 to 0.95, taking into account the performance of the model throughout the entire retrieval process. The training epoch, batch size, and optimizer are set as 3, 16, and SGD optimiser, respectively.Besides, we adopt six metrics to fairly judge our method including entropy (EN) [1] , standard deviation (SD) [33] , mutual information (MI) [28] , visual information fidelity (VIF) [7] , structural similarity index measure (SSIM) [36] , and 𝑄 𝐴𝐵/𝐹 [26] . EN measures fused images from the perspective of information theory, which is primarily used to assess the amount of information and complexity contained in the fused image. SD is a statistical metric, that measures an image from its brightness and contrast, but sometimes it can be disturbed by noise. MI measures the amount of information that two images share in common. VIF not only considers the pixel-level similarity with source images, but also cares about human visual perception. SSIM measures fused images from structure, illuminance and contrast. 𝑄 𝐴𝐵/𝐹 not only focuses on the visual quality of the image but also takes into account the preservation of the image content and structure. Due to the dynamic refresh being impacted by the comprehensive loss items, we first verify the efficacy of comprehensive loss, including intensity loss, maximum gradient loss, and maximum perception loss, individually. Subsequently, we demonstrate the effectiveness of our dynamic refresh strategy and the distillation mechanism step by step.As we mentioned in the comprehensive loss function part, the intensity loss is mainly used to retain the pixel distribution, salient targets and general structural information. The maximum gradient item and maximum perception item are further utilised to retain texture, gradient and high-level semantic information. We initially explore their effects on the teacher model. From Figure 5 , we can see the fused image only trained by intensity loss (Stage I), for areas with lower brightness in visible light images like the blue box and red box, it indeed fails to preserve textural details and gradient information. After applying the maximum gradient loss and maximum perception loss function (Stage II), from the ground shadows within the red and blue boxes, it can be observed that the missing information becomes increasingly rich and clearly visible.Our dynamic refresh strategy is mainly adopted to sufficiently explore the vital information contained in intermediate outputs.With the combined effect of comprehensive loss functions, the extraction power becomes increasingly strong, fully absorbing the information contained in not only the source images but also the historical reference outputs. From the last image in Figure 5 (Stage III), We can see the information within the red and blue boxes displayed with unprecedented clarity, little artifact and noise. Besides, quantitative results displayed in Table 1 show that our comprehensive loss function and dynamic refresh strategy are effective. We design a student model with only two convolutional layers and 113 trainable parameters by utilising a powerful teacher. Specifically, distillation is performed at the feature and response ends. To verify the effectiveness of distillation, we conduct an ablation study including the student model without distillation, the student model with direct distillation, and the student model with our digestible distillation. From the first two rows on Table 2 , we can observe that students can only learn a little bit of the superficial aspects from the teacher through direct distillation, resulting in a decrease in SSIM. In contrast, through our digestible distillation, all four metrics have shown notable improvements. Positive evidence of these four metrics means the quality of the fused image itself is higher than before, sharing more detailed information with source images from both pixel level and visual perception perspectives.In fact, there are many volumes of targeted students available for us to choose from. During this process, we gradually compress the student model down to the minimum, where different model sizes correspond to different compression ways and varying effects. As displayed in Table 3 , from the perspective of metrics value, we should opt for the second student model. However, in pursuit of an extremely mini model size, we choose the third configuration, which achieves an extremely tiny size, while dose not compromising performance to a significant extent.Finally, we obtain the mini model with only 113 trainable parameters. Before we display the fusion performance of our experiments, we first compare the model efficiency and complexity with several SOTA approaches, as displayed in Table 4 . Among all the involved methods, our MMDRFuse exhibits the smallest model size, the least number of floating-point operations, and the fastest average running time. In the following experiments, we will further demonstrate that our MMDRFuse not only enjoys advantages in efficiency and complexity but also leads to promising model performance. 5 . Our method consistently ranks first in three metrics (MI, VIF, 𝑄 𝐴𝐵/𝐹 ) across the three datasets, and first or second in two metrics (EN, SSIM). Besides, in terms of the SD metric that is susceptible to noise, we rank first on MSRS, second on RoadScene, and do not perform particularly well on LLVIP, which is a common occurrence and does not detract from our excellent performance.The outstanding performance in full-reference (MI, VIF, 𝑄 𝐴𝐵/𝐹 , SSIM) and no-reference (EN, SD) indicators show that the fusion results not only closely resemble the source images at the visual and pixel levels, but also possess rich, high-quality information within each fused image itself. The above results also demonstrate that our mini model can adeptly handle a variety of scenes and lighting conditions, and it surpasses much larger models in both visual effects and evaluation metrics, perfectly achieving a balance between performance and various costs. To verify the generalisation ability of our method, we directly apply the above six SOTA methods and our design to MIF without retraining. We show the qualitative results of MRI-PET images in Figure 8 . From the green and red boxes, we can observe that our MMDRFuse not only preserves the detailed internal structural information displayed in MRI images To further validate the performance of our MMDRFuse on downstream detection tasks, we conduct experimental exploration on pedestrian detection. As can be seen from Figure 9 , although LRR-Net ( [16] ) achieves the best detection precision, it requires significantly more time and computational resources than our design. Furthermore, compared with MetaFuse ( [48] ), which introduces the detection task as additional supervision, our method performs better even without any prior settings in detecting pedestrians. This can be attributed to the introduction of perceptual loss, combined with the dynamic refresh strategy and distillation mechanism that jointly learns advanced semantic information. In contrast, our method can precisely detect pedestrians from significant targets within contrasting backgrounds without detecting semantic information as input and with less running time as well as a mini model size under 1 KB, achieving satisfactory detection results at a considerably marginal cost. 
conclusion: 
 An investigation into compressing the image fusion model is conducted in this work. We utilise the specially designed comprehensive loss function and the dynamic refresh strategy based on intermediate fusion results to first formulate a fusion model. Further combined with the digestible distillation strategy, we successfully train an extremely tiny (0.44 KB) student network from the teacher model. Experimental results on multiple tasks demonstrate that our mini model not only exhibits advantages in efficiency and complexity but also achieves promising results on the MMIF and downstream detection tasks.  
 
 Appendix:  


Decision: This work proposes a lightweight Distilled Mini-Model with a Dynamic Refresh strategy (MMDRFuse) for Multi-Modality Image Fusion, achieving model parsimony through careful supervisions and demonstrating advantages in experiments and downstream applications. With an average score of 5 and the unanimous positive reviews, the paper is competitive for this year's ACM MM conference. The AC agrees to accept the paper and strongly recommends incorporating the content from the rebuttal into the final version.

Reviews: review: paper_summary: paper_summary: This paper proposes a well-designed distillation strategy to transfer the knowledge from powerful teacher model to extremely small student model.
The authors adopt three dedicated designs to integrate supervision signals from teacher, source images, and history records.
Experiments demonstrate the efficiency and effectiveness of the proposed method, with superior performance in multiple image fusion tasks and downstream pedestrian detection application.
strength_weakness: strength_weakness: 1. This paper proposes an end-to-end extremely small fusion model with a total of 113 trainable parameters (0.44 KB) and promising advantages in terms of model efficiency and complexity, it greatly promots the development of multiple image fusion tasks.
2. The dynamic refresh training strategy identifies the useful information in the intermediate iterations, and refine the supervision signals adaptively. This method is innovative and proven effective in Table 1.
3. Efficiency and effectiveness of the proposed method is validated in multiple image fusion tasks and downstream pedestrian detection application.
limitations: limitations: 1. In Table 1, the performance of student1 is worse than that of student2, while the model size of student1 is 2 orders of magnitude larger than that of student2. The results look a bit strange, I'm not sure whether student1 is trained enough.
2. The third row in Table 2 appears to use a combination of digestible distillation loss, comprehensive loss, and dynamic refresh loss. Because the quantitative result in Table 2 is the same as that in Table 3 and Table 5 (b).
3. It lacks details about the average running time in Table 4, how do you get the time ? 
4. Will you plan to release the code ?
paper_summary: paper_summary: They utilise the specially designed comprehensive loss function and the dynamic refresh strategy based on intermediate fusion results to first formulate a fusion model. They successfully train an extremely tiny (0.44 KB) student network from the teacher model with the smallest model size, the fastest running time and the best performance.
strength_weakness: strength_weakness: Extensive experiments on several public datasets demonstrate that their method exhibits promising advantages in terms of model efficiency and complexity, with superior performance in multiple image fusion tasks and downstream pedestrian detection application, with the smallest model size, the fastest running time and the best performance.
limitations: limitations: Based on my limited expertise in the field, I find the paper to be impressive. The extensive experiments conducted on various public datasets demonstrate the method's promising advantages in terms of model efficiency, complexity, and performance in image fusion tasks and pedestrian detection applications. The authors' focus on the smallest model size, fastest running time, and superior overall performance adds significant value to the research field. Therefore, my rating is Weak Accept.
paper_summary: paper_summary: The paper primarily discusses the training of a minimal student model using distillation, a comprehensive loss function, and dynamic refresh strategies. It provides a detailed explanation of the Digestible Distillation technology, comprehensive loss function, and Dynamic Refresh Strategy techniques used in this paper. In terms of experiments, the model is trained on Visible Image Fusion (IVIF), Medical Image Fusion (MIF), and Pedestrian Detection (PD), achieving generalization from the IVIF task to the other two tasks. Ablation studies are conducted to compare the effects of distillation with and without distillation, direct distillation, and the new distillation method proposed in this paper. Comparisons are also made regarding the use of the comprehensive loss function and the Dynamic Refresh Strategy. Finally, experimental results are presented to show the performance of the IVIF, MIF, and PD tasks in comparison with other state-of-the-art (SOTA) methods.
strength_weakness: strength_weakness: Directly transferring knowledge from the teacher model to the student model is unstable, so the authors propose adding a buffer before the final output to enable the student model to digest it better. The teacher model uses a densely connected network (DCN) to extract image features, which is then followed by two convolutions to produce feature maps with 4 channels. To achieve lightness, the student model only adopts two convolution operations: the first convolution is used to extract feature maps from the teacher model, and the second convolution acts as a decoder to produce the fused image. The Comprehensive Loss introduces an intensity loss to ensure that the fused image has the same pixel distribution as the input images, a gradient loss is sued to help visible images transfer gradient information, and a perception loss is mainly used to eliminate the blurring of feature maps and abstract information in the fusion performance. The dynamic refresh strategy, where parameter changes during the training process are often neglected in traditional methods, utilizes these changes as supervisory signals. Sometimes supervisory signals may contain some useless information, such as noise, so an evaluation system is designed to determine usefulness, calculating SSIM and GMSD for intermediate results and taking the optimal values.
limitations: limitations: Achieving good performance across multiple task metrics with a small model is commendable, but the distillation strategy lacks innovation. Moreover, we should focus more on unsupervised or weakly supervised approaches, as the cost of obtaining supervised signals is relatively high. No ablation experiments are conducted on the three losses separately within the comprehensive Loss, so it is uncertain which loss truly has the effect. The details and the purpose of the dynamic refresh strategy are not clear, how should we interpret "to strengthen the cues embedded in the historical parameters of the training process, we propose the dynamic refresh strategy"? Although knowledge distillation based method proposed in this paper works well, it is not very innovative approaches in idea.
In addition, the SD in Figure 1 and the SSIM in the relevant description make me confused.
paper_summary: paper_summary: This paper proposes a lightweight Distilled Mini-Model, named MMDRFuse, for multi-modality image fusion. The MMDRFuse has only 113 parameters (0.44 KB) but achieves a good performance as well as other bigger models, which is achieved by three carefully designed supervisions. The first is a two-stage distillation, which aims to distill knowledge from the feature and output. The second is the Comprehensive Loss, which involves image-level and feature-level complementary. The third is a Dynamic Refresh Strategy, which applies Image quality as an assessment structural similarity (SSIM) and Gradient Magnitude Similarity Deviation (GMSD) as the metrics to measure whether the current model is the best one. Experiments are conducted on several public benchmark datasets and a pedestrian detection task.
strength_weakness: strength_weakness: The method is novelty, and the writing is good and easy to follow.
limitations: limitations: 1.	The granularity of ablation experiments should be more detailed, e.g., more ablation experiments should be performed where any one of the three items in the 𝐿_{𝑐𝑜𝑚𝑝} loss is missing.
2.	Figure 4 is confusing, more notations and comments should be added.
3.