E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection
Authors: Jiaqing Zhang
Mingxiang Cao
Weiying Xie
Jie Lei
DaixunLi
Wenbo Huang
Yunsong Li
Xue Yang
Venue: NeurIPS 2024 oral (2024)
Keywords: Multimodal Fusion
Object detection
Primary area: machine_vision
TL;DR: A novel end-to-end training algorithm for multimodal fusion detection

Abstract: Multimodal image fusion and object detection are crucial for autonomous driving. While current methods have advanced the fusion of texture details and semantic information, their complex training processes hinder broader applications. Addressing this challenge, we introduce E2E-MFD, a novel end-to-end algorithm for multimodal fusion detection. E2E-MFD streamlines the process, achieving high performance with a single training phase. It employs synchronous joint optimization across components to avoid suboptimal solutions associated to individual tasks. Furthermore, it implements a comprehensive optimization strategy in the gradient matrix for shared parameters, ensuring convergence to an optimal fusion detection configuration. Our extensive testing on multiple public datasets reveals E2E-MFD's superior capabilities, showcasing not only visually appealing image fusion but also impressive detection outcomes, such as a 3.9\% and  2.0\% $\text{mAP}_{50}$ increase on horizontal object detection dataset M3FD and oriented object detection dataset DroneVehicle, respectively, compared to state-of-the-art approaches.

E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection
Jiaqing Zhang', Mingxiang Cao', Weiying Xie Jie Lei", Daixun Li', Wenbo Huang,, Yunsong Li', Xue Yang**
1 The State Key Laboratory Integrated Services Networks, Xidian University
University of Technology Sydney SSotheeast University *Shanghai AI Laboratory
https://github. www.iceyy-aaanneeeeeeeeooe 2aaieeyyyaaann/EE-MMM E2E-MFD
Abstract
Multimodal image fusion and object detection are crucial for autonomous driving. While current methods have advanced the fusion of texture details and semantic information, their complex training processes hinder broader applications. Ad- dressing this challenge, we introduce E2E-MFD, a novel end-to-end algorithm for multimodal fusion detection. E2E-MFD streamlines the process, achieving high performance with a single training phase. It employs synchronous joint opti- mization across components to avoid suboptimal solutions associated to individual tasks. Furthermore, it implements comprehensive optimization strategy in the gradient matrix for shared parameters, ensuring convergence to an optimal fusion detection configuration. Our extensive testing multiple public datasets reveals E2E-MFD's s superior capabilities, showcasing not only visually appealing image fusion but also impressive detection outcomes, such a 3.9% and 2.0% mAP50 increase on horizontal object detection dataset M3FD and oriented object detection dataset Drone Vehicle, respectively, compared to state-of-the ofeee----te-rrtttt approaches.
Introduction
Precise and reliable object parsing is critical in fields such autonomous driving [l] and remote sensing monitoring [2]. Relying solely on visible sensors can lead inaccuracies in object recognition in challenging environments, like inclement weather conditions. Visible-infrare image fusion [3;4; 6] as a typical common multimodal fusion (MF) task addresses these challenges by leveraging complementary information from different modalities, leading the rapid development of various multimodal image fusion techniques [7; 8; 9; 10; 11]. Techniques like CDDFuse [12] and DIDFuse employ two-step process where MF network is trained initially, followed by training object detection (OD) network with the results from the MF network to assess fusion effectiveness separately. Although deep neural networks have significantly enhanced the ability learn representations across modalities, resulting in promising multimodal fusion outcomes, the focus has predominantly been on producing visually appealing images. This emphasis often overlooks the improvement of downstream high-level visual tasks, such as enhanced object parsing, which remains a substantial hurdle.
Recent studies have devoted into designing joint learning methods that integrate fusion networks with high-level tasks such as object detection [ [14] and segmentation [15; 16]. The synergy between MF and OD in Multimodal Fusion Detection (MFD) methods has emerged research. This partnership allows MF to produce richer, more informative images, enhancing OD performance, while OD contributes valuable object semantie insights MF, aiming to accurately locate and identify objects in a scene. Typically, MFD networks adopt a cascaded design where joint
vibrant area of
*Coresponniigg author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Stage1
(1 0,1)
Stage min (d. (we(a)
Stage minue" (u, gr Mr,..)) ( 2 (d. (a.y: S(0")
(3)
MF
OD
0D
Barkbbone
Stage unt 6" (u. (a. 9: of (dt V (oc02))
Backbone
Nodes
(d)
Stage]
sito (e Bull
MF
(b)
MF
Root
Barkhuuee
Stage 3 min (u, (i..y 6.). 2 id, (m (mx8a)) 6.61)
Branches
S
Stage
Stage 2 min (d.0 Bell
Mattewoel suges
Falesteag
MF
QU
Object Untestie ho beack
MF
Nulitaneul Paxas Netwerk
Figure I: Comparison of (d) E2E-MFD with existing MF-OD task paradigms (a) Two-Stage (Separate Cascaded), (b) Two-stage (Joint Cascaded) and (c) Multi-stage (Joint Cascaded).
optimization techniques [17] use the OD network guide the MF network toward creating images that facilitate easier object detection. Notably, Zhao et. al [18] introduced a joint learning method for multimodal fusion detection, incorporating meta-feature embedding from OD to improve fusion by generating semantic object features through meta-learning simulation. Despite these advancements, highlighted in Figure 1, significant challenges persist: 1) Current optimization approaches rely on a multi-step. progressively joint method, compromising efficiency: 2) These methods overly focus on leveraging OD information for fusion enhancement, leading to difficulty in parameter balancing and susceptibility to local optima of individual tasks. Therefore, the quest for unified feature set that simultaneously caters each task remains formidable.
In this paper, we introduce E2E-MFD, an end-to-end algorithm for multimodal fusion detection, designed to seamlessly blend detailed image fusion and object detection from coarse to fine levels. E2E-MFD facilitates the interaction of intrinsic features from both domains through synchronous joint optimization, allowing for streamlined, one-stage process. To reconcile fine-grained details with semantic information, we propose the novel concept of an (ORPPT) coupled with coarse-to-fine diffusion processing (CFDP) mechanism. This approach is inspired by the natural process of visual perception, tailored to meet the specific needs of MF and OD. Furthermore, introduce a Gradient Matrix Task-Aligmment (GMTA) technique to fine-tune the optimization of shared components, thereby minimizing the adverse impacts traditionally associated with inherent optimization challenges. This ensures an efficient convergence towards an optimal set of fusion detection weights, enhancing both the accuracy and efficacy of multimodal fusion detection.
Phylogenetic Tree
Our contributions in this paper highlighted as follows: (1) We present E2E-MFD, a pioneering approach to efficient synchronous joint learning, innovatively integrating image fusion and object detection into a single-stage, end-to-end framework. This methodology significantly enhances the outcomes of both tasks. (2) We introduce a novel GMTA technique, designed to evaluate and quantify the impacts of the image fusion and object detection tasks. This aids in optimizing the training process's stability and ensures convergence to an optimal configuration of fusion detection weights. (3) Through comprehensive experimentation on image fusion and object detection, we demonstrate the efficacy and robustness of our proposed method.
2 Related Work
2.1 Multimodal Fusion Object Detection
Due to the powerful nonlinear fitting capabilities of deep neural networks, deep learning has made significant progress in low-level vision tasks, particularly image fusion tasks [19; 20; 21; 22; 23; 24]. Early efforts [9; 25; 13; 26; 27; 28] tended to achieve excellent fusion results by adjusting network structures or loss functions, overlooking the fact that image fusion should aim improve the performance of downstream application tasks. Fusion images with good quality metrics may be suitable for human visual perception but may not be conducive to practical application tasks [16; 29]. Some research has acknowledged this issue, Yuan et al. [30] utilized various aligned modalities for improved oriented object detection [31; 32; 33; 34] to address the challenge cross-modal weakly misalignment in acrial inblerinfrrred images. Liu et al. [17] proposing joint learning method,

---

pioneered the exploration of the MF and OD combination methods. Then the optimization of the loss function for segmentation [16] and detection [14] has been validated to be effective in guiding the generation of fused images. They consider the downstream task network as an additional constraint to assist the MF network in generating fusion results with clearer objects. Zhao et al. [18] leveraged semantic information from OD features to aid MF and perform meta- embedding to generate meta-features from OD features, which are then used to guide the MF network in learning pixel- level semantic information. Liu et al. [15] proposed multiiinteractive feature learning architecture for image fusion and segmentation by enhancing fine-grained mapping of all the vital information between two tasks, so that the modality or semantic features can be fully mutual-interacee interacctiee. However, OD considers the semantic understanding of objects. while MF and segmentation primarily focus on the pixel-level relationship between image pairs. The optimization coupling of detection and fusion tasks becomes more challenging to investigate these complementary differences, from which both fusion and detection can benefit. It is worth mentioning that these visible-infrared multimodal fusion detection methods are usually designed in a cascaded structure with tedious training steps. Researchers lack the adoption of end-to-end architectures, which would enable one-step network inference to generate credible fused images and detection results through a set of network parameters.
2.2 Multi-task Learning
Multi-task learning (MTL.) [35; 36] involves the simultaneous learning of multiple tasks through parameter sharing. Prior approaches involve manually crafting the architecture, wherein the bottom layers of a model are shared across tasks [37; 38). Some approaches tailor the architecture based on task affinity [39], while others utilize techniques such as Neural Architecture Search [40; 37; 41] or routing networks [42] to autonomously discern sharing patterns and determine the architecture An alternative method typically combines task-specific objectives into a weighted sum [43; 44; 45]. In addition, most approaches (e.g. [46; 47; 48; 49; 50]) aim to mitigate the effects of conflicting or dominating gradients. The approach of explicit gradient modulation [48; 49; 50; 51], has demon- strated superior performance in resolving conflicts between task gradients by substituting conflicting gradients with modified, non-conflicting gradients. Inspired by the above multimodal learning meth- ods, we introduce Gradient Matrix Task-Alignment method to align the orthogonal components contained in image fusion and object detection tasks thereby effectively eliminating the inherent optimization barrier that exists between two tasks.
3
The Proposed Method
3.1 Problem Formulation
MF-OD task concentrates on generating an image that benefits emphasizing objects with superior visual perception capability. The goal of OD is to find the location and identify the class of each object in an image which can naturally provide rich semantic information along with object location information. Therefore, the motivation of OD-aware MF to construct a novel infrared and visible image fusion framework that can benefit from the semantic information and object location infor- mation contained in OD. For this purpose, we suppose a pair of visible image x E iI #XXCC and infrared image y E I8 HxWXC, The optimization model is formulated as:
min C (t.N (x, y: 0.)), 0,
(I)
where represents the output of the different task network N with the learnable parameters 0,. Z() is constraint term to optimize the network. Previous approaches solely design image fusion or object detection networks in a cascaded way, which can only achieve outstanding results for one task. To produce visually appealing fused images alongside accurate object detection results, we we jointly integrate the two tasks into a unified goal synchronously which can be rewritten as:
(2) where 0- = 0 0 defined as the shared parameters for MF and OD networks. and d denote the fused image and detection result, which are produced by the MF network ((-) and OD network (-) with the learnable parameters 0,, and Oa, w is a predefined weighting factor to balance the task training. S(.) is a constrained term to jointly optimize the two tasks. In this paper, we regard the S(.) as a feature learning constrained manner and achieve this goal by designing a Gradient Matrix Task- Align-mlemment training scheme.
0,, 0a = arg min wL" (u, 1 (a. y: 0.)) + (1 (1-w) c" (d, L (x, 0,)) + S
3

---

Multimodal Images
Backhone
Nodel C Coarse-to fine Diffusion Process (CFDP) 0,
Bos
Res
Synchroaous liint Optimization
H, M2 orgini of" (p.0(kx. (4- (slir)
libek
Node2 Object- Region-Piznl Phylogenetic Tree (O (ORPPT))
[x PFMM (Branch 03
RFRM (Brench
RFRM (Branch bg)
RFRM (Brench Lgg
Gradlent Matrix Task- Algument (GMTA)
Brthagradaeed
Fanaede Space
Parsenta Span
Gradient Dominance
and Conflict
Pbdl
Objec
Figure 2: An overview of the proposed E2E-MFD framework, which consists of a backbone, nodes, and branches. The backbone is utilized to extract multimodal image features. A fine-grained fusion network (ORPPT) and diffusion-basee object detection network (CFDP) are optimized by synchronous joint optimization (GMTA) in end-to-end manner.
3.2 Architecture
Our proposed E2E-MFD is designed with parallel principle, composited by image fusion and object detection sub-network. Details of the whole architecture are shown in Figure 2, A fusion network (ORPPT) and object detection network (CFDP) can sufficiently realize the granularity-aware detail information and semantic information extraction.
Object-Region-Pixel Phylogenetic Tree. The important fact of that humans pay more attention to different regions from coarse to fine for object detection in object scale and image fusion in pixel scale. Inspired by phylogenetic tree, to simulate humans to study the interactions of hierarchies under different granularity views, we construct an Reriect-Regio--PPP
Phylogenetic Tree (ORPPT) as (-) to extract different features multiple region scales. Given an image pair æ E R H R XHXCv firstly extract image features (x) and f by shared parallel backbone to save memory and computing resources. Then these features are added in channel dimension to obtain the final L multimodal image features 01, OI, OL E R W; xHi xC: The parameters in f(-) are shared with the OD network.
Although f(x) and fzy can describe the characteristics of visible and infrared modalities, they lack insights into the muulti-granulariyy perspective. Therefore, we utilize the branches including the one pixel feature mining module (PFMM) and I. region feature refine module (RFRM) to mine the multiple granularities from coarse fine. The PFMM Bg is set the same with feature fusion block in the MetaFusion [18] with the input pair y. We set 1, ... I.... ...... L to denote each region branch. For branch a CNN (-) is firstly utilized to extract the granularity-wis feature (pr()r) E RR H2 at the region level. A set of learnable region prompts R, {ri.m E RC introduced to define the M, different regions of granularity-
feature, where Ti,m denotes the th region prompt at branch Then the feature vector is mapped into the region mask A = aa,m E R H2 m=1 by conducting the dot product between the feature vector and region prompt followed by batch normalization and a ReLU activation:
A = ReLRLLBUN R; (p((o)))
Finally, the vector of the object-level feature is weighted by the region mask and further aggregated form region representation:
B 1.m
(o,) = a a (ot);
where (o.) denotes the m region representation and (i.j) denotes the spatial location. These region-level representations are further concatenated to form the observation Br(or) (or), (r,)) br,M, (or) of branch l. These multi-grained attentions concentrate opera- tion on the spatial location information and the extent of the regions which are similar to the task requirements. The B,, B2. .... Bt up-sampled keep the consistent spatial size with the

---

pixel-level fusion features Bg. Then, the region-level fusion features B1, B2....... Bn are assembled by addition operation followed by Convolution with 1 + reduce the channel numbers. Highlighted region-level features are extracted by a Convolution with 1 x 1 kernel + Sigmoid and then injected into pixel-level features by multiplication and addition. Finally, five Convolutions with 3 x 3 kernel ReI.U layers are constructed to reconstruct the fusion result u.
Coarse-to- Fine Diffusion Process. Diffusion models, inspired by nonequilibrium thermodynamies, are a class of likelihood--aat models. DiffusionDet [52] is the first neural network model to utilize the diffusion model for object detection, introducing a novel paradigm that achieves promising results compared to traditional object detection models. Combining diffusion simulation with the diffusion and recovery process of the object box, the Coarse-to to-Fine Diffusion Process (CFDP) introduces as an efficient detection head to assist fusion networks to focus more on object areas. CFDP model defines a Markovian chain of diffusion forward process by gradually adding noise to set of bounding boxes. The forward noise process is defined as:
(5) which transforms bounding boxes E N to latent noisy bounding boxes 2, for t E {0, (0. У} by adding noise to 2o. āg == IT= a₂ = IT=_ (1 - 3a) and 33 represents the noise variance schedule. During training stage, a neural network Vo, (ze, t) trained to predict 20 from 2t by minimizing the training objective with &g loss:
q (Zt =0) N (zx Vārzo. (1- I),
(6) At inference stage, bounding boxes 20 is reconstructed from noise 27 with the model Vo, and updating rule in an iterative way, i.e., Zr + 2T - -> Zn- In this work, we aim to solve the object detection task via the diffusion model. A neural network Ve, (z t, x, y) is trained predict 20 from noisy boxes 2,, conditioned on the corresponding image pair æ, y.
La = 2
v (za, t) - zoll2.
3.3 Loss Function
The total loss is combined with an image fusion loss function Li and object detection loss La. Li consists of three types of losses, i.e., structure loss LSSIM, pixel loss Lpisel and gradient loss Lerad For one fused image, it should preserve overall structures and maintain similar intensity distribution from source images. To this end, the structural similarity index (SSIM) is introduced in function:
ESSIM = (1 SSIM(u, x)) /2 (1 SSIM(u, y)) /2,
where LSSIM denotes structure similarity loss. In the fused image, we expect the object regions to have a more significant contrast compared the background region. Therefore, the object regions need to preserve the maximum pixel intensity and the background region needs to be slightly below the maximum pixel intensity to bring out the contrast between the object and background. The ground-truue bounding boxes of the objects in images are denoted as (Ze: Ve, w, h) for horizontal boxes and (Ie: Me; w, O) for rotated boxes, where (Iee Mc) is the center location, w and h are the width and height, 0 = angle pi/180 /180, respectively. Based on these ground-truth bounding boxes, we construct the object mask Im and the background mask denoted as J..- The object regions pixel loss pixsel and the background region pixel loss Lpixel are formulated as:
Spixel Im (n mar(x, y)) Criset
|l(1 Im) (u mean (a. y) 1))
where || - || stands for the L -noom. The operator o denotes the elementwise multiplication, max(-) denotes the element-wise maximization, and mean(-) denotes the element-wise average operation. Therefore, the object-aware pixel loss Episel is defined as:
Cpixel
Coisel + Ciisel
Besides, gradient information of images always characterizes texture details, thus, we use Lynd to constrain these textual factors to multi-scale manner:
Lgrad
Σ 3,7,5,7
| u - max (V* v
(10)
where denotes gradient operators that calculate by V u - -((u) with combination of different Gauss (G) kernel size k. Totally, we obtain L. 771 CSSIM + 72p 173 Cgnd.
5

---

Visible image
Infrared image
U2Fusion
Tardal
SwinFusion
PlATusioo
DIDFuse
CDDFuse
Metal'usion
E2E-MFD
Figure 3: Visual results of image fusion on M3FD.
3.4 Gradient Matrix
The MF and OD tasks have distinct optimization objectives. MF primarily emphasizes capturing the pixel-Ievel relationship between image pairs, while OD incorporates object semantics within the broader context of diverse scenes. An inherent optimization barrier exists between these two tasks. We observe that the prevailing challenges in multi-task learning are arguably task dominance and conflicting gradients. We introduce Gradient Matrix Task Aaisgmmetmment (GMTA) by presenting the condition number to mitigate the undesired effects of the optimization barrier in task-shared parameters O which are supposed to be balanced between the MF and OD tasks. The individual task gradients of MF and OD task are calculated by 9T, V L. and ga = Vo- Lo in the training optimization process. The gradient matrix can be defined as G {g 9a) In multi-task optimiza- tion, a cumulative gradient g = Gw is linear combination of task gradients and the stability of a linear system is measured by the condition number of its matrix in numerical analysis. Hence the stability of the gradient matrix is equal to the ratio of the maximum and minimum singular values (non-negative) of the corresponding matrix: x< x(G)
Learning from the Aligned-MTL [51],
Omun
the condition number is optimal (vs(G (G) 1) if and only if the gradients are orthogonal and equal in magnitude which means that the system of gradients has no dominance or conflicts:
k GG) 1 + In Id
(11)
The final linear system of gradients defined by G satisfies the optimal condition in terms a condition number. Thereby, we consider the feature learning constraint S(0") can be defined as the following optimization to eliminate instability in the training process:
min | G
s.t. x(G)= = + min G s.t. G &
(12)
The problem be treated as Procrustes problem and can be solved by performing singular value decomposition (SVD) G (G- UEV") and rescaling singular values corresponding to principal components so that they are equal to the smallest singular value:
G = oUV = =OGVE-
(13)
where,
(V, = eigh G).
(14)
Σ diagl 1//mmaz v 1/Amin);
(15)
eigh represents a function for finding cigenvectors V and eigenvalues and diag stands for diagonal matrix. Amar and Amin are maximum eigenvalues and minimum eigenvalues from A.The stability criterion, a condition number, defines a linear system to an arbitrary position scale. To alleviate this ambiguity, choose the largest scale that guarantees convergence the optimum: this is minimal singular value of an initial gradient matrix:
Omiu (G) V Vmmin-
(16)

---

Table 1: Quantitative results of different fusion methods on TNO, RoadScene, and M3FD datasets. The model training (Tr.) and test (Te.) time is counted on an NVIDIA GeForce RTX 3090. The best result is highlighted.
M3FD
TNO
RoadScene
Tr.
Te.
Task
Method
EN
MI
VIF
EN
MI
VIF
EN
MI
VIF
Time
Time
DIDFusel 13|
6.13
14.65
1.51
6.30
15.30
1.47
6.67
16.65
1.55
3h9m38s
0.096s
U2Fusion 26]
5.66
14.22
1.50
5.78
14.89
1.49
6.25
16.30
1.57
4h8m36s
2.091s
MF
PlAFusion [58]
5.75
13.92
1.59
5.05
13.61
1.36
6.37
16.22
1.58
5h35m20c
0.003s
SwinFusion [59]
5.80
13.83
1.58
6.09
14.28
1.55
6.30
15.93
1.60
3h38m5s
0.044s
CDDFuse [2]
5.77
13.82
1.58
6.21
15.03
1.49
6.54
16.54
1.57
5h59m594
0.096s
Tardal[ [7
5.72
14.68
1.47
5.87
14.99
1.43
6.72
16.98
1.54
5h36m28s
0.0935
MF-OD
Metafusion [8]
6.20
15.19
1.54
6.29
16.03
1.44
6.35
16.76
1.57
6h477388
0.002s
E2E-MPD
6.36
15.47
1.65
6.40
16.28
1.60
6.79
17.11
1.69
2h50m322s
0.014s
4 Experiments and Analysis
4.1 Dataset and Implementation Details
We conduct experiments on four widely-used visible-infrared image datasets: TNO [53], RoadScene [26]. M3FD [17] and Drone Vehicle [3]. TNO and RoadScene are just used to evaluate MF perfor- mance. M3FD is adopted to evaluate both MF and OD performance. RoadScene with 37 image pairs, TNO with 42 image pairs and M3FD with 300 pairs are only used for the MF task in the testing stage, and the MF network is trained by the M3FD dataset which is divided into training set (2,940 image pairs) and a testing set (1,260 image pairs). Besides, Drone Vehicle consists of 28,433 28,439 image pairs is utilized train and test MF and OD for oriented objects. We conduct all the experiments with one GeForce RTX 3090 GPU, and the code of M3FD is based on Detectron2 [54], while the code of DroneVehicla is based on MMDetection 2.26.0 [55] and MMRotate 0.3.4 [56]. On the M3FD dataset, the pretrained DiffusionDet is used for the initialization of the OD network. In the training phase, E2E-MFD is optimized by AdamW with a batch size of 1. We set the learning rate 2.5e and the weight decay as le- 4. The default training iteration is only 15,000. On the Vehicle dataset, the pretrained LSKNet [57] is used for the initialization of the object detection network, and we fine-tune it for 12 epochs with a batch size of 4. The E2E-MFD is optimized by Adam W and the learning rate and the weight decay set to 1e 4 and 0.05.
4.2 Main Results
Results on Multimodal Image Fusion. Qualitative results of different fusion methods are depicted in Figure 3. All the fusion methods can fuse the main features of the infrared and visible images to some extent and we can observe two remarkable advantages of our method. First, the significant characteristics of infrared images can be effectively highlighted by our method. Our M3FD fusion image captures the person riding a motorcycle. In comparison with other methods, our method demonstrates high contrast and recognition of the objects. Second, our method preserves rich details from the visible images, including color and texture Our advantages are evident in the fusion images across the M3FD dataset, such as the clear outline of the white car's rear and a man on motorcycle. While retaining a substantial amount of detail, our method maintains a high resolution without introducing blurriness. In contrast, other methods fail to achieve these two advantages simultaneously. Sequentially, we provide quantitative results of different fusion methods in Table 1. Our E2E-MFD generally achieves the best metric values. Specifically, the largest average value on MI proves that our method transfers more considerable information from both source images. Values of EN reveal that our results contain edged details and the highest contrast between objects and the background. The large VIF shows our fusion results have high-quality visual effects and small distortion compared with the source images. Moreover, our method achieves the fastest training time to finish the joint learning at one stage which means that faster iterative updates can be done on new datasets. The test time to generate fused image ranked third.
Results on Multimodal Object Detection. To more effectively evaluate the fusion images and observe their impact on downstream detection tasks, we conduct tests using the baseline detector YOLOv5s on all SOTA methods on the M3FD dataset. We follow the same parameter settings, and the visualization results are shown in Figure 4. The detection results are poor when using only single-modal image inputs, with instances of missed detection, such as the motorcycle and rider next to the car and people on the far right in the image. Almost all fusion methods reduce
7

---

Table 2: Quantitative results of object detection on M3FD dataset among all the image fusion methods
+ detector (i.e. YOLOv5s [60]). means using the fusion images generated by E2E-MFD for object detection training. The best result is highlighted.
Task
Method
People
Car
Bus
Motorcycle
Lamp
Truck
mAP
mAP
mAP550:9
50:95
Intrared
49.3
67.1
72.9
35.8
43.6
61.6
85.3
55.1
Visible
38.1
69.4
75.5
44.4
44.8
63.2
86.3
55.9
DIDFusion [3]
45.8
68.8
73.6
42.2
43.7
61.5
86.2
S6.2
U2Fusion
47.7
70.1
73.2
43.2
44.6
63.9
87.1
$7.1
MF
PIAFusion 58
46.5
69.6
75.1
45.4
44.8
61.7
87.3
57.2
SwinFusion [5]]
44.5
68.5
42.2
44.4
63.5
56.1
CDDFuse [2]
46.1
69.7
74.2
42.2
44.2
62.7
87.0
$6.5
E2E-OD
CF]
52.0
68.2
79.2
49.9
45.2
69.6
89.8
6U.7
ICAFusion[ [22]
48.8
68.5
72.3
45.5
43.6
64.7
87,4
$7.2
[
49.8
65.4
46.6
43.7
61.1
86.0
$6.0
MetaFusion[ [8]
48.4
66.7
70.5
49.1
46.4
59.9
86.7
56.8
MF-OD
Ours (YOLOv5s)*
51.0
67.9
69.4
50.2
48.7
61.6
87.9
58.1
Ours (DiffusionDett)
58.5
67.7
79.9
50.3
46.2
70.2
90.3
62.1
Ours (22--MFD)
60.1
69.5
81.4
52.2
47.6
72.2
91.8
63.8
Visible image
image
U2Fusion
Turdul
PIAFusion
DIDFuse
CDDFuse
MelaFusion
Ours (YOLOv5s)
E2B-MFD
Ground truth
Figure 4: Visual results of object detection on M3FD.
missed detection and improve confidence by fusing information from both modalities. Through the design of an end-to-end fusion detection synchronous optimization strategy, obtain fusion images that are visually and detection frriionn,,f especially for occluded and overlapping objects, as seen in the blue ellipse with the motorcycle and the overlapping people on the far right in the image. To further assess the quality of fusion images, we conduct a fair comparison between our method and the SOTA methods on YOLOv5s. shown Table the MF methods demonstrate performance improvement compared to single-modal detection, indicating that well-fused images can effectively assist downstream tasks. contrast, the fusion image we generate has achieved the best performance YOLOv5s. Additionally, the detection performance of fusion images DiffusionDet is also impressive, albeit slightly lower than when optimizing fusion and detection tasks
Table 3: Quantitative results of object detection on Drone Vehicle sets among all the SOTA methods, where means using the fusion images generated by E2E-MFD for object detection training and testing. The best result is highlighted.
Modality
Detectors
Truck
Freight Car
Bus
mAP30
RetinaNet-OBB
28,2
13.7
62.1
38.1
Faster R-CNN-OBB [64]
67.9
38.6
26.3
67.0
23.2
44.6
RGB
Gliding Vertex [65]
75.8
46.1
33.8
68.1
38.7
52.5
YOLOv5s-OBB [60]
89.0
53.6
41.9
84,8
32.6
60.4
LSKNet-OBB [57]
89.5
70.0
51.8
89.4
56.9
71.5
RetinaNet-OBB [63
79.9
32.8
28.1
67.3
16.4
44.9
Faster R-CNN-OBB [64]
42.5
35.2
77.9
$4.6
Gliding Vertex [65]
59.7
43.0
78.8
62.9
YOLOv5s-OBB [60]
95.6
57.2
47.5
89.4
35.2
65.0
LSKNet-OBB [57]
90.3
73.3
57.8
89.2
53.2
72.8
UA-CMDet [3]
87.5
60.7
46.8
87.1
38.0
64.0
TSFADet [30]
89.2
72.0
54.2
88.1
48.8
70.4
CALNet [66]
90.3
76.2
63.0
89.1
58.5
75.4
RGB+IR
Ours YOOLOvS--OBI
69.9
49.9
92.6
44.5
70.7
Ours (LSSKNetOBBB)'
90.3
77.0
63.5
89.5
59.0
75.9
Ours (E2E-MFD
90.3
79.3
64.6
89.8
63.1
77.4

---

simultaneously with E2E-MFD. Thanks to the collaborative optimization of both tasks, the detection performance further enhanced. Furthermore, even when compared to end-to-end object detection methods (E2F-OD), our approach demonstrates significant performance improvements, This better underscores the advantages of our training paradigm and the effectiveness of our method.
Results on Multimodal Oriented Object Detection. As shown in Table 3, our fusion detection synchronous optimization strategy achieves the highest accuracy. Furthermore, the outstanding detection performance on YOLOv5s-OBB [60] and LSKNet using the generated fusion images (with at least 5.7% and 3.1% higher AP values compared to single modalities) demonstrate the robustness of our method. This validates the superior quality of the fusion images. indicating that they are not only visually appealing but also provide rich information for the detection task.
4.3 Ablation Studies
Analysis of Gradient Matrix. As described in Section 3.4, the MF and OD tasks pursue different optimization goals. To visualize the task dominance and conflicting gradients, we plot the gradient matrix in the training stage illustrated by Figure 5. We perform GMTA operation every 1,000 iteration loss updates. Blue represents the gradients of shared parameters computed by the OD loss function, while yellow represents the gradients of shared parameters computed by the MF loss function. During the training process, it can be observed that the gradient values of the OD task are larger and dominant, while that of the MF task are smaller. This may affect the learning process of the fusion task during training. Conversely, the utilization of GMTA effectively mitigates this gradient dominance and conflict, facilitating a balance of shared parameters between MF and OD,
Effect of Gradient Matrix Task- Alignment. To verify the effectiveness
Table 4: The validation of GMTA on M3FD.
of GMTA, we compare separate opti- mizations for MF and OD, as well joint optimization, w/o and w indicating whether use GMTA. Specifically, MF represents using only Lu to optimize the fusion network, OD represents using only La to optimize the object detection network, and E2E-MFF represents simultancous optimization of the fusion and detection networks using both Cu and Ca loss functions. The results, as shown in Table 4, indicate that methods incorporating GMTA optimization constraints with shared weights achieve the best results for both MF and OD. This is because MF primarily emphasizes capturing the pixel-Ievel relationship between image pairs, while OD incorporates object semantics within the broader context of diverse scenes. Therefore, optimizing the entire network with shared loss functions may be influenced by local optimal solutions of individual tasks. The accuracy of E2E-MFD (w/o GMTA) shows slight decrease compared to separately training the detection network. In contrast, GMTA orthogonalizes the gradients of shared parameters corresponding to the two tasks, allowing the joint network to converge to an optimal point with fusion detection weights.
Task MF
EN
MI
VIF
mAPso
mAP
6.09
14.00
1.48
OD
1
90.28
62.75
E2E-MED (w/o GMTA)
6.12
14.70
1.39
90.05
62.60
E2E-MED (w GMETA)
6.36
15:47
1.65
91.80
63,83
To compare the effectiveness of different Multi-task learning methods on our algo- rithm, we selected three robust MTL op- timization methods. As shown in Table 5, all MTL methods addressed the conflict between the MF and OD tasks to varying degrees. By introducing the concept of GMTA, we achieved a better balance the gradient optimization process between the two tasks, resulting in the best performance.
Table 5: Ablation of different MTL methods on M3FD.
Method
EN
MI
VIF
mAP30
mAPan.s
E2E-MED (wlo GMTA)
6.12
14.70
1.39
90.05
62.60
PCGrad[50]
6.13
15.01
1.48
90.59
62.71
CAGiradl 47|
6.17
15.05
1.48
90.71
62.74
Nash-MTL[499
6.29
15.28
1.51
90.91
62.97
EIE-MED (w GMEAA
6.36
15.47
1.65
91.80
63,83
The GMTA process operates during the computation and updating stages of two gradients. The GMTA performed approximately every n iteration (gradi- ent update), focusing on balancing the independence and coherence of various tasks. Table presents the ablation analysis of the n parameter. Decreasing initially disrupts task optimization due to frequent alignment, while increasing becomes crucial when
Table 6: Ablation studies of the iteration pa- rameter on M3FD dataset.
EN
MI
VIF
MAPso
mAP50-05
500
5.93
14.78
1,58
90.93
62.73
1000
6.36
15.47
1.65
91.80
63.83
1500
6.24
15.08
1.62
91.10
62.96
2000
6.13
14.69
1.45
90.35
62.75

---

MP
100
We GMIA
GINDIA
GMIA
Figure 5: Visualization of task dominance and conflicting gradients in joint learning of OD and MF.
the network determines task optimization directions.
However, excessively large leads to significant deviations in task paths, making alignment more challenging and negatively impacting performance.
Study of Branches in the Object-Region Pixel Phy- logenetic Tree. We investigate the combination of the pixel feature mining module (0) and the region feature refinement module (1,2.3,4). The results are shown in Table 7. It can be observed that with the increase the number of branches, the fusion net- work achieves higher image fusion quality and object detection performance. Region features provide the fusion network with multi-level semantic features of objects. However, when higher-level semantic object information added, the performance of the fusion network declines. This is because the detailed information contained in the deeper layers of the backbone structure shared with the detection network decreases abruptly, which may affect the fusion network's 's absorption of these features, thereby inftuencing pixel-level fusion tasks.
Table 7: Ablation study of the number of
ORPPT branches on M3FD.
Branch
EN
MI
VIF
mAPso
mAP
0 6.10
15.19
1.54
91,51
63.20
0,1
6.10
15.29
1.54
91.65
62.96
0,1,2
6.19
15.28
1.58
91.73
63,46
0,1,2,3
6.36
15.47
1.65
91.80
63.83
0,1,2,3,4
14.31
1.40
91.73
63.55
We have implemented the Dbject-Region-PPe Pixell Phylogenetic Tree (ORPPT) to explore the hierarchical interactions under different granularity views and extract various features across multiple region scales, as introduced in Section 3.2. As shown in Figure 6, the detailed information will decrease as the backhone structure shared by the detection network deepens, resulting in decrease in detailed information. This may affect the absorption of these features by the fusion network of the shared backbone network, thereby affecting pixel-level fusion tasks. This provides evidence for analysis of the reasons for Section 4.3 ablation experiment for Object-Region-Pixx. Phylogenetic Tree. This illustrates the importance of balancing the semantic information provided between the OD and MF network with pixel-level information.
B
B
B
Figure 6: Feature map visualization of various branches in the ORPPT.
5
Conclusion
Within this paper, an end-to-end optimization is proposed formulate fusion and detection in harmonious one-stage training process. We introduce a object-region pixel phylogenetic tree structure and coarse-to- fine diffusion process simulate these two tasks in different visual perceptions needed for diverse task requirements. In addition, we align the orthogonal components of the fusion detection linear system of the gradients by gradient matrix task-alignment. By unrolling the model to a well- designed fusion network and diffusion-basee based detection network, we can generate visual-friendly result for fusion and object detection in efficient way without tedious training steps and inherent optimization barriers.
10

---

References
[1] Kai Jiang, Jiaxing Huang. Weiying Xie, Jie Lei, Yunsong Li, Ling Shao, and Shijian Lu. DA- BEV: Unsupervised Domain Adaptation for Bird's Eye View Perception, page 322-341. Springer Nature Switzerland, October 2024. ISBN 9783031730078 doi: 10000//977-8-001-77000000
19. URL httt://dx. doi. wtg//.../// 100.10007/9955
[2] Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, al. A survey on multimodal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 958-979, 2024.
[3] Yiming Sun, Bing Cao, Pengfei Zhu, and Qinghua Hu. Drone- rgb-infrared cross- tross-modality vehicle detection via uncertainty- aware aware learning. IEEE Transactions on Circuits and Systems for Video Technology, 32(10):6700-6) 6713, 2022.
[4] Wujie Zhou, Xinyang Lin, Jingsheng Lei, Lu Yu, and Jenq-Neng Hwang. Mffenet: Multiscale feature fusion and enhancement network for oh-thhrmma urban road scene parsing. IEEE Transactions on Multimedia, 24:2526-2538, 2021.
[5] Zixiang Zhao, Haowen Bai, Yuanzhi Zhu, Jiangshe Zhang, Shuang Xu, Yulun Zhang, Kai Zhang, Deyu Meng, Radu Timofte, and Lue Van Gool. Ddfm:: Denoising diffusion model for multi-modality image fusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8082-8093, October 2023.
[6] Wujie Zhou, Shaohua Dong, Caie Xu, and Yaguan Qian. Edge-aware guidance fusion network for thermal scene parsing. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 36, pages 3571-3579, 2022.
[7] Hui Li, Xiao-Jun Wu, and Josef Kittler. Rfn-nest: An end-to-end residual fusion network for infrared and visible images. Information Fusion, 73:72-86 86, 2021.
[8] Hui Li, Tianyang Xu, Xiao-Jun Wu, Jiwen Lu, and Josef Kittler. Lrmett: A novel representation learning guided fusion network for infrared and visible images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[9] Jinyuan Liu, Xin Fan, Ji Jiang, Risheng Liu, and Zhongxuan Luo. Learning a deep multi-scale feature ensemble and an edge-attention guidance for image fusion. IEEE Transactions on Circuits and Systems for Video Technology, 32(11:105-119)
2021. 2021.
[10] Zhanbo Huang, Jinyuan Liu, Xin Fan, Risheng Liu, Wei Zhong, and Zhongxuan Luo. Reconet: Recurrent correction network for fast and efficient multi-modality image fusion. European Conference on Computer Vision (ECCV), pages 339-555. Springer, 2022.
[11] Wang Di, Liu Jinyuan, Fan Xin, and Risheng Liu. Unsupervised misaligned infrared and visible image fusion via cross-modality image generation and registration. In International Joint Conference on Artificial Intelligence (IJCAI), 2022.
[12] Zixiang Zhao, Haowen Bai, Jiangshe Zhang. Yulun Zhang, Shuang Xu, Zudi Lin, Radu Timofte, and Lue Van Gool. Cddfuse: Correlation-driver dual-branch feature decomposition for multi- modality image fusion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5906-591 19665996.750 2023.
[13] Zixiang Zhao, Shuang Xu, Chunxia Zhang, Junmin Liu, Jiangshe Zhang, and Pengfei Li. Didfuse: Deep image decomposition for infrared and visible image fusion. In International Joint Conference on Artificial Intelligence (IJCAI), pages 970-976. ijcai.org, 2020.
[14] Yiming Sun, Bing Cao, Pengfei Zhu, and Qinghua Hu. Detfusion: detection-drivee infrared and visible image fusion network. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4003-4011, 2022.
[15] Jinyuan Liu, Zhu Liu, Guanyao Wu, Long Ma, Risheng Liu, Wei Zhong, Zhongxuan Luo, and Xin Fan. Multi-interactive feature learning and full-time modti-modality benchmark for image fusion and segmentation. Proceedings of the IEFF/CVE International Conference on Computer Vision (ICCV), pages 8115-8124 2023.

---

[16] Linfeng Tang. Jiteng Yuan, and Jiayi Ma. Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network. Information Fusion, 82: 28-42, 2022.
[17] Jinyuan Liu, Xin Fan, Zhanbo Huang, Guanyao Wu, Risheng Liu, Wei Zhong, and Zhongxuan Luo. Target-aware dual adversarial learning and multi-scenario multi-modality benchmark to fuse infrared and visible for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 5802-5811 2022.
[18] Wenda Zhao, Shigeng Xie, Fan Zhao, You He, and Huchuan Lu. Metafusion: Infrared and visible image fusion via meta-feature embedding from object detection. In CVPR, pages 13955-13965, 2023.
[19] Hui Li, Xiao-Jun Wu, and Tariq Durrani. Nestfuse: An infrared and visible image fusion architecture based on nest connection and spatial/channel attention models. IEEE Transactions on Instrumentation and Measurement, 69(12)9645-9656 2020.
[20] Risheng Liu, Jinyuan Liu, Zhiying Jiang, Xin Fan, and Zhongxuan Luo. A bilevel integrated model with data-driven layer ensemble for multi-modality image fusion. IEEE Transactions Image Processing. 30: 10:121111 2020.
[21] Jiayi Ma, Han Xu, Junjun Jiang, Xiaoguang Mei, and Xiao-Pins Zhang. Ddegan: A dual- discriminator conditional generative adversarial network for multi-resolution image fusion. IEEE Transactions on Image Processing, 29:4980-4995, 2020.
[22] Jayy Ma, Wei Yu, Pengwei Liang, Chang Li, and Junjun Jiang. Fusiongan: A generative adversarial network for infrared and visible image fusion. Information Fusion, 48:1 1-26, 2019.
[23] Jayy Ma, Hao Zhang, Zhenfeng Shao, Pengwei Liang, and Han Xu. Ganmec: A generative adversarial network with multiclassification constraints for infrared and visible image fusion. IEEE Transactions on Instrumentation and Measurement, 70:1-14, 2020.
[24] Nirmala Paramanandham and Kishore Rajendiran. Infrared and visible image fusion using discrete cosine transform and swarm intelligence for surveillance applications. Infrared Physics Technology, 88:13-22, 2018.
(25) Risheng Liu, Zhu Liu, Jinyuan Liu, and Xin Fan. Searching hierarchically aggregated fusion architecture for fast multi-modality image fusion. In Proceedings of the 29th ACM International Conference on Multimedia, pages 1600-1608, 2021.
[26] Han Xu, Jiayi Ma, Junjun Jiang, Xiaojje Guo, and Haibin Ling, U2fusion: A unified unsuper- vised image fusion network. IEEE Transactions on Paltern Analysis and Machine Intelligence, 44(1):502-518, 2020.
[27] Zhu Liu, Jinyuan Liu, Guanyao Wu, Long Ma, Xin Fan, and Risheng Liu. Bi-level dynamic learning for jointly multi-modaliy image fusion and beyond. arXiv preprint arXiv:2305.06670. 2023.
[28] Xin Zhang, Weiying Xie, Yunsong Li, Jie Lei, Kai Jiang, Leyuan Fang, and Qian Du. Block- wise partner learning for model compression. IEEE Transactions on Neural Networks and Learning Systems, pages 1-14, 2023. doi: 10.
[29] Kai Jiang, Jiaxing Huang, Weiying Xie, Jie Lei, Yunsong Li, Ling Shao, and Shijian Lu. Domain adaptation for 06969.
object detectors, 2024. URL https::// /ttps/////xxvvi//// wtgg/bs//////////o//
[30] Maoxun Yuan. Yinyan Wang. and Xingxing Wei. Translation, scale and rotation: Cross-modal alignment meets rgb-infrared vehicle detection. In European Conference on Computer Vision (ECCV), pages 509-525. Springer, 2022.
[31] Xue Yang, Jirui Yang, Junchi Yan, Yue Zhang, Tengfei Zhang, Zhi Guo, Xian Sun, and Kun Fu. Scrdet: Towards more robust detection for small, cluttered and rotated objects. In Proceedings of the IEEF/CVE International Conference on Computer Vision (ICCV), pages 8232-8241, 2019.
12

---

[32] Xue Yang. Junchi Yan, Ziming Feng, and Tao He. R3det: Refined single-stage detector with feature refinement for rotating object. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 35, pages 3163-3171, 2021.
[33] Xue Yang, Junchi Yan, Wenlong Liao, Xiaokang Yang, Jin Tang, and Tao He. Scrdet++: Detecting small, cluttered and rotated objects via instance-level feature denoising and rotation loss smoothing. IEEE Transactions on Paltern Analysis and Machine Intelligence, 45(2):
2384-2399, 2022.
[34] Xue Yang, Gefan Zhang, Xiaojiang Yang, Yue Zhou, Wentao Wang. Jin Tang, Tao He, and Junchi Yan. Detecting rotated objects as gaussian distributions and its 3-d generalization. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(4):4335-4333 2022.
[35] Caruana. Multitask learning: A knowledge-based source of inductive biasl. In International Conference on Machine Learning (ICML), pages 41 41-48. Citeseer, 1993.
[36] Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05099 2017.
[37] Felix JS Bragman, Ryutaro Tanno, Sebastien Ourselin, Daniel C Alexander, and Jorge Cardoso. Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1385-1394, 2019.
[38] lasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid- and high-level vision using diverse datasets and limited memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 6129-6138, 2017.
[39] Simon Vandenhende, Stamatios Georgoulis, Bert De Brabandere, and Luc Van Gool. Branched multi-task networks: deciding what layers to share. arXiv preprint 1xXiv:190.022920 2019.
[40] Chanho Ahn, Eunwoo Kim, and Songhwai Oh. Deep elastic networks with model selection for multi-task learning. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 6529 6529-6538, 2019.
[41] Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems (NeurIPS), 33:8728 8740, 2020.
[42] Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selec- tion of non-linear functions for multi-task learning. International Conference Learning Representations (ICLR), 2017.
[43] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi Must learning using uncertainty to weigh losses for scene geometry and semanties. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7482-7491, 2018.
[44] Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gra- dient normalization for adaptive loss balancing in deep multitask networks. In International Conference Machine Learning (ICML), pages 794-803 PMLR, 2018.
[45] Minh-Thang Luong, Quoe Le, lly Sutskever, Oriol Vinyals, and Lukasz Kaiser. Multi-task sequence to sequence learning. International Conference on Learning Representations (ICIR), 2015.
[46] Jean-Antoine Désidéri. Mutiple-gradient descent algorithm for multiobjective optimization. In European Congress on Computational Methods in Applied Sciences and Engineering (ECCO- MAS 2012), 2012.
[47] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems (NeurIPS), 34: 18878-18888 18890, 2021.
13

---

[48] Liyang Liu, Yi Li, Zhanghui Kuang, J Xue, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Towards impartial multi-task learning. In International Conference on Learning Representations (ICI.R), 2021.
[49] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game. arXiv preprint arXiv:2202.01017, 2022.
[50] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems (NeurIPS), 33:5824-5836, 2020.
[51] Dmitry Senushkin, Nikolay Patakin, Arseny Kuznetsov, and Anton Konushin. Independent component aligmment for multi-task learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20083-20093, 2023.
[52] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 19830-19843, 2023.
[53] Alexander Toet. The tno multiband image data collection. Data in Brief, 15:249-251, 2017.
[54] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
https: thub. com http:///ebboeeeeaaaeeeeeetttottt
[55] Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint Xrviv: /90.00900077155 2019.
[56) Yue Zhou, Xue Yang, Gefan Zhang, Jiabao Wang, Yanyi Liu, Liping Hou, Xue Jiang, Xingzhao Liu, Junchi Yan, Chengqi Lyu, Wenwei Zhang, and Kai Chen. Mmrotate: A rotated object detection benchmark using pytorch. In Proceedings of the 30th ACM International Conference on Multimedia, 2022.
[57] Yuxuan Li, Qibin Hou, Zhaohui Zheng, Ming-Ming Cheng, Jian Yang, and Xiang Li. Large selective kernel network for remote sensing object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 16794-16805, 2023.
[58] Linfeng Tang, Jiteng Yuan, Hao Zhang. Xingyu Jiang, and Jiayi Ma. Piafusion: A progressive infrared and visible image fusion network based on illumination aware. Information Fusion, 83:
79 92, 2022.
[59] Jiay Ma, Linfeng Tang, Fan Fan, Jun Huang, Xiaoguang Mei, and Yong Ma. Swinfusion: Cross-domain long-range learning for general image fusion via swin transformer. JEEE/CAA Journal of Automatica Sinica, 9(7):1200-1211 1217, 2022.
(60] ultralytics/yolov5:: v3.1 Bug Fixes and Performance Improvements. 5281/zenodo. 41544770, 2020.
[61] Fang Qingyun, Han Dapeng, and Wang Zhaokui. Cross-modality fusion transformer for multispectral object detection. arXiv preprint arXiv:2111.0022 2021.
[62] Jifeng Shen, Yifei Chen, Yue Liu, Xin Zuo, Heng Fan, and Wankou Yang. Icafusion: Iterative cross-attention guided feature fusion for multispectral object detection. Pattern Recognition, 145:109913, 2024.
[63] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings the IEEF/CVF International Conference on Computer Vision (ICCV), pages 2980-2988, 2017.
[64] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances Neural Information Processing Systems (NeurIPS), 28, 2015.
14

---

[65] Yongchao Xu, Mingtao Fu, Qimeng Wang, Yukang Wang, Kai Chen, Gui-Song Xia, and Xiang Bai. Gliding vertex on the horizontal bounding box for multi-oriented object detection. IEEE Transactions on Puttern Analysis and Machine Intelligence,
2020.
[66] Xiao He, Chang Tang, Xin Zou, and Wei Zhang. Multispectral object detection via cross- modal conflict-aware learning. In Proceedings of the 31st ACM International Conference on Multimedia, pages 1465-1474, 2023.
[67] J Wesley Roberts, Jan A Van Aardt, and Fethi Babikker Ahmed. Assessment of image fusion procedures using entropy, image quality, and multispectral classification. Journal of Applied Remote Sensing, 2(1 ):023522, 2008.
[68] Guihong Qu, Dali Zhang, and Pingfan Yan. Information measure for performance of image fusion. Electronics Letters, 38(7):1, 2002.
[69] Yu Han, Yunze Cai, Yin Cao, and Xiaoming Xu. A new image fusion performance metric based on visual information fidelity. Information Fusion, 14(2):127-135, 2013.
[70] Liqiang He and Simisa Todorovic. Destr: Object detection with split transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9377-9386, 2022.
15

---

A Appendix
A.1 Metric
Three metrics used for MF evaluation: entropy (EN) [67], mutual information (MI) [68] and visual information fidelity (VIF) [69]. EN evaluates the information richness in an image, and the higher EN means more information. MI evaluates the information similarity between the input images and fused images. The higher MI illustrates more information the input images is fused. VIF measures the ability to extract visible information from the input image, and larger VIF represents less visible distortion in the fused result. Here, we use the channel in the HSV space of the fusion results to calculate these metrics. Moreover, performance, where the average of mAPs sampling every 5 from 50 to AP is calculated. A higher mAP50095 means better OD effect.
mAPs0:95 [70] to comprehensively evaluate OD
A.2 Experiment setting on YOLOO55s
To more effectively evaluate the fusion images and observe their impact on downstream detection tasks, we conduct tests using the bascline detector YOLOv5s on all SOTA methods on the M3FD dataset. All images are resized to x024x 1024 and trained from scratch for 300 epochs with batch size of 64. In addition, the experiments for DroneVehicl dataset are conducted on YOLOv5s-OBD using the generated fusion images, all images resized to 640x640 640 and trained from scratch for 50 epochs with a batch size of 64.
A.3 Experiments on Vehicle
Drone comprises aerial RGB-IR images captured by drones, encompassing various scenes from an aerial perspective. It contains five categories of target objects. The dataset consists of 28,439 pairs of images, divided into training, validation, and test sets. We train the MF network on the training set and perform inference directly on the test set. Finally, fusion images are generated for both the training and test sets. detection accuracies are obtained on the test set. The visualization results of the fusion and detection of Drone DroneVehicle are shown in Figure 7 and 8. In the process of generating fusion images, we first obtain RGB images, then convert them to HSV format, and only take the V channel as the final fusion image. preserves the brightness information of the objects while containing rich details from visible images. As shown in Figure 7, the first row and the second row are images extracted from the training set and the test set, respectively. Although the style of the fusion images resembles that of the infrared images, by combining visible image information, the fusion images differentiate different parts of each object, reflecting varying degrees of brightness information. This is advantageous for fine-grate -raige classification. Additionally, as shown in Figure 8. the visually friendly fusion images also assist in the object detection task, allowing for sufficient detection even in cases of small and dense objects.
Visible image
Infrared image
E2E-MFD
Visible image
Infrared image
E2E-MFD
Visible image
Infrared image
EZE-MFD
Visible imagge
Infrured image
E2E-MFD
Figure Qualitative results image fusion on Vehicle.
A.4 Experiments on Coase-to-Fine Diffusion Process
We conducted ablation experiments on CFDP Table investigating its inclusion and the number of proposed boxes. In the setting without CFDP, we maintained the backbone network while substituting
16

---

Ground Trudis
E2E-MPD
Figure 8: Visual results of object detection on Drone Vehicle.
CFDP with RPN (Region Proposal Network), standard components in two-stage object detectors. Results indicate that CFDP enhances detailed information capture and precise box guidance, thereby enhancing fusion image quality and detection performance. For optimal balance between performance and efficiency, we selected 500 proposal boxes.
Table 8: Ablation studies of the CFDP M3FD dataset.
Settings
Proposal boxes
MI
VIF
mAPso
mAP5a.95
Tr.Time
w/o CFDP
500
5.71
14.39
1.45
90.13
61.98
2h52ml Is
300
6.01
14.57
1.53
90.89
63.29
2h23m45s
CFDP
500
6.36
15.47
1.65
91.80
63.83
2h50m32s
1000
6.37
15.34
1.63
92.05
63.75
3h32m30s
A.5
More Fusion Visualization Results
More comparisons of infrared-visible image fusion visualization results are depicted Figure 10 and 11. These fusion results demonstrate the advantages of synchronously optimizing fusion and detection tasks. With minimal training costs, we obtain fusion images that are visually and detection-friendly Specifically, the fusion images retain significant object information extracted from infrared images, while also preserving detailed information such as texture, color, and background from visible images. Our method effectively combines the strengths of both modalities to enhance the overall performance of the detection.
17

---

Visible
Infrared
U2Fusion
Tardal
SwinFusion
PtAFusion
use
CDDFuse
MetaFusion
E2E-MFD
Visible image
Infrared image
U2Fusion
Tardal
Sw in Swinnusion
PlAFusion
DIDFuse
CDDPuse
MetaFusion
E2E-MFD
Visible image
Infrared image
U2Fusion
Tardal
SwinFuusion
MetaFusicn
Figure 9: Qualitative results of image fusion on M3FD.
A.6 More Detection Visualization Results
The visualization of infrared-visible object detection demonstrated in Figure 12. Our fusion images noticeably superior due the effective combination of object information from infrared images and texture details from visible images. This integration enables the object detection network to distinguish between the background and the object clearly. Additionally, for small and occluded objects, the clear edge details assist the network in achieving improved detection results.
A.7 Limitations and Broader Impacts
Limitations. Our E2E-MFD approach is effective for the joint learning of the multimodal fusion and object detection task and has been validated on various datasets. However, the validations of the current model rely the visible and infrared modalities. Constrained by the scarcity of relevant datasets within the community, the paper lacks validation with additional datasets containing new modalities. Future research will focus on addressing this gap by exploring, constructing, and incorporating more diverse multimodal datasets serving multimodal fusion detection.
Broader Impacts. Our paper aims to broaden the applicability of joint learning of multimodal data and object detection various research domains. However, this broader scope may present challenges when using the model in domains that include harmful content. These challenges arise from the data itself, rather than from the model. Therefore, crucial have adequate data regularization to effectively address these concerns.
18

---

Visblle image
Infrared image
U2Fusier
Tardal
SwinFusion
PLATusior
DIDI iee
CDDI use
MataFission
C2E MFD
Visibl
Intiared
U2fossic:
Tardal
Swinf
PlAFusion
DIDFuee
CDDFuse
MetaFusion
EI5-MFD
Vieble image
Image
Figure 10: Qualitative results of image fusion on RoadScene.
Visible image
Infrured image
U2Fusion
Tandal
Swintusion
PIAFasion
DIDFuse
CDDPuse
MetaFusion
E2E-MFD
Visible image
Infirrod imago
U2Tusion
Tanbal
SwinFusiom
PlAFFusion
DIDFese
CDD
MetoEusion
E2E-MFD
Visible
Inttared
U'Fassion
Figure 11: Qualitative results of image fusion on TNO.
19

---

Visible image
Inftured
U/Fusioon
Tardal
SwinFusion
PlAFasion
DIDFuse
CDD
MetaFosion
Ours (YOLOs5))
E2E-MFD
Ground tnith
Visible image
Infrured image
U2Fusion
Tardal
SwinFusion
PIAFusion
MetaPusion
Visible
Infrured image
U2Fosion
Tardal
SwinFusion
PIA Fuasion
DIDFuse
CDDFuse
MetaFusion
Ours (YOLOv5s)
E2E-MPD
Ground trulh
Figure 12: Visual results of object detection on M3FD.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction accurately state the contributions of the E2E- MFD model, including its novel design of synchronous joint learning of the multimodal fusion and object detection.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of the the work is detailed discussed in the Appendix A.7.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the necessary information needed to reproduce the main experimental results are fully stated in the Section and Appendix A. This information ensures the understanding the results which support the main claims and conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The paper provide the open access to the data and code with sufficient instruc- tions for reproducing the main experimental results in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. wwtpccccppccccoecccc.cccccc.cttt g/guidg Cooeeuumiis wwwbbbbmisssons wwwiiisssssppliieeeiooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The training and test details including data splits, hyperparameters and type of optimizer are illustrated in Section 4.1 and Appendix A.3 to ensure the results of the paper can be fully understand and reproduced.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The appropriate information about the statistical significance of the experiments are carefully reported in Section ensuring that the findings are presented with an appropriate level of precision and confidence.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: The paper provide the sufficient information the computation computer resources including the type of compute works (GeForce RTX 3090 GPU), memory, and time of execution in the Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https://neee /ttee://neeeppeeeeeee http///uurrs..pppppprpeeeeeeeeee cc/ppblic///pcppttcccccccccccccc /Etthi csGuidelines?
Answer: [Yes]
Justification: The research conducted in the paper conforms with the NeurIPS Code Ethics which ensuring all aspects of our study, including data collection, analysis, and reporting.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader Impacts are discussed paper. We highlight the benefits of our research, such as advancing the field of multimodal fusion detection learning, which can lead improvements various applications, including autonomous vehicles and remote sensing, At the same time, the limitations of the method are analysed in Appendix A.7.

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: NA.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators original owners of assets are properly credited and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will open the code totally the paper is accepted with the new assets documented and documentation is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects..
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "MaRSRl6qu5", "reviewer": "Reviewer_bix9", "summary": "This paper proposed an end-to-end algorithm for multimodal fusion detection, experiments on fusion and detection tasks showed the better performance than some methods.", "questions": "See the weaknesses.", "limitations": "See the weaknesses.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper proposed an end-to-end algorithm with one-stage training process，for multimodal fusion detection, experiments on fusion and detection tasks showed the better performance than some methods.", "weaknesses": "1. Why use the V channel in the HSV space of the fusion results to calculate the metrics?\n2. The best result of car detection highlighted in table 2 is wrong. Additionally, add analysis of why the proposed algorithm couldn’t realize the best detection effect of car.\n3. Provide a detailed justification for the chosen datasets. Explaining why these specific datasets are representative or challenging.\n4. The details of GMTA should be added.\n5. It is mentioned that the GMTA operation is executed every 1000 iterations, but more specific implementation details, such as the specific setting and selection basis of parameters, are lacking.\n6. Ablation studies for CFDP should be added to verify how CFDP impacts the final results.\n7. The advantages of ORPPT and GMTA compared with existing techniques are not fully demonstrated. It needs to be more explicit about how these innovations solve existing problems or lead to performance improvements. \n8. More SOTA methods and metrics should be added for image fusion task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reviewer bix9\nThanks for your comments.\n\n**A1:** This is a common default setting in the field such as \"CVPR23 MetaFusion\". V (brightness) channel can effectively measure the algorithm’s ability to handle low-light environments.\n\n**A2:** We have made revision in Tab. 2. It's common to observe fluctuations in detection accuracy within a single category on M3FD dataset in Tab. R.1. Despite fluctuations, our algorithm significantly outperforms others in $\\text{mAP}$.\n\n**A3:** We utilize widely recognized datasets with diverse tasks, sufficient quantity, and complex environment. TNO and Roadscene are evaluation datasets for MF. TNO captures multispectral images day and night, while Roadscene comprises aligned image pairs from road scenes with vehicles and pedestrians. We also enrich experimental validation by integrating horizontal and oriented OD datasets. The M3FD dataset covers complex scenarios with diverse camera angles. Additionally, DroneVehicle dataset is considered for oriented OD with diverse scenes from an aerial viewpoint.\n\n**A4:** The process of GMTA is illustrated in Section 3.4 in the paper and we detailed this section to eliminate the confusion. GMTA mitigates the undesired effects of the optimization barrier in task-shared parameters which are supposed to be balanced between the MF and OD tasks. Conflicts in the gradient matrix $\\boldsymbol{G}$ are related to a negative cosine distance between gradient vectors ($<\\boldsymbol{g}_u,\\boldsymbol{g}_d><0$), while dominance is caused by an imbalance of their magnitudes ($\\Vert \\boldsymbol{g}_u \\Vert \\gg \\Vert \\boldsymbol{g}_d \\Vert$ or $\\Vert \\boldsymbol{g}_u \\Vert \\ll \\Vert \\boldsymbol{g}_d \\Vert$). \n\nLearning from the Aligned-MTL (multi-task learning), the condition number is optimal ($\\kappa(\\boldsymbol{G})=1$) if and only if the gradients are orthogonal and equal in magnitude which means that the system of gradients has no dominance or conflicts:\n$$\\kappa(\\boldsymbol{G})=1 \\iff <\\boldsymbol{g}_u,\\boldsymbol{g}_d>=1.$$\n\nThe final gradient linear system $\\hat{\\boldsymbol{G}}$ satisfies the optimal condition by the condition number. Thus, the feature learning constraint $\\mathcal{S}(\\boldsymbol{ \\theta}^{\\star})$ can be defined as the following optimization to eliminate training instability:\n$$ \\min _{\\hat{\\boldsymbol{G}}}\\|\\boldsymbol{G}-\\hat{\\boldsymbol{G}}\\|_F^2 \\quad \\text { s.t. }\\kappa(\\hat{\\boldsymbol{G}})=1 \\iff \\min _{\\hat{\\boldsymbol{G}}}\\|\\boldsymbol{G}-\\hat{\\boldsymbol{G}}\\|_F^2 \\quad \\text { s.t. }\\hat{\\boldsymbol{G}}^{\\top} \\hat{\\boldsymbol{G}}=\\boldsymbol{I}.$$\n\nThe problem can be treated as a Procrustes problem and can be solved by performing a singular value decomposition (SVD) to $\\boldsymbol{G}$ ($\\boldsymbol{G}=\\boldsymbol{U} \\boldsymbol{\\Sigma} \\boldsymbol{V}^\\top$) and rescaling singular values corresponding to principal components so that they are equal to the smallest singular value:\n$$\n\t\t\\hat{\\boldsymbol{G}}=\\sigma \\boldsymbol{U} \\boldsymbol{V}^{\\top}=\\sigma \\boldsymbol{G} \\boldsymbol{V} \\boldsymbol{\\Sigma}^{-1} \\boldsymbol{V}^\\top,\n$$\nwhere, \n$$(\\boldsymbol{V}, \\boldsymbol{\\lambda})= eigh(\\boldsymbol{G}^{\\top}\\boldsymbol{G}),$$\n$$\\boldsymbol{\\Sigma}^{-1} = diag(\\sqrt{1/\\lambda_{max}},\\sqrt{1/\\lambda_{min}}).$$\n$eigh$ finds eigenvectors $\\boldsymbol{V}$ and eigenvalues $\\boldsymbol{\\lambda}$, with $diag$ representing a diagonal matrix. $\\lambda_{max}$ and $\\lambda_{min}$ are the maximum and minimum eigenvalues of $\\boldsymbol{\\lambda}$. The stability criterion, defined by the condition number, sets a linear system to an arbitrary position scale. To resolve this ambiguity, we select the largest scale ensuring convergence to the optimum, corresponding to the minimal singular value of the initial gradient matrix:\n$$\\sigma=\\sigma_{\\min }(\\boldsymbol{G})=\\sqrt{\\lambda_{min}}.$$\n\n**A5:** The GMTA process operates during the computation and updating stages of two gradients, approximately every $n$ iteration (gradient update), to balance the independence and coherence of various tasks. Tab. R.2 presents the ablation analysis of the $n$ parameter. Detail analysis is in **Experiment Results part of Global Author Rebuttal**.\n\n**A6:** We have added ablation experiments on CFDP, involving whether to use CFDP and the number of proposed boxes, as shown in Tab. R.3. Detail analysis is in **Experiment Results part of Global Author Rebuttal**.\n\n**A7:** The paper designs the first end-to-end joint training paradigm in MFD. E2E-MFD enhances interaction between intrinsic features through synchronous joint optimization, streamlining the process. Our solution goes beyond task concatenation, incorporating dedicated module design. Inspired by a phylogenetic tree analogy, we employ an ORPPT to extract features across multiple region scales. By utilizing PFMM and $L$ RFRM branches, we capture various granularities from coarse to fine. Tab. 5 validates our approach, while Fig. 6 underscores the importance of effectively integrating pixel-level and object-level details. \n\nMF primarily focuses on pixel-level relationships between image pairs, while OD integrates object semantics within diverse scene contexts. This inherent optimization barrier between the tasks necessitates a solution. We propose GMTA, a gradient alignment concept within the multi-task learning framework, to optimize task dominance and resolve conflicting gradients in end-to-end joint training. Results in Tab. 4 demonstrate that GMTA, with shared weights, yields superior performance. Comparison with other methods in Tab. 6 further validates our approach. Fig. 5 illustrates how GMTA balances shared parameters between MF and OD, effectively mitigating gradient dominance and conflict.\n\n**A8:** In Tab. R.4, three recent fusion SOTA methods (CVPR2024 SHIP, PR2024 CFNet, and PR2024 DSFusion) and three evaluation metrics (Qabf, PSNR, and SSIM) are incorporated to valid effectiveness of our method. Detailed analysis is illustrated in **Author Rebuttal Experiment Results**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TkTQ4t26Nl", "reviewer": "Reviewer_G1Ro", "summary": "This paper proposes a joint learning diagram for multimodal fusion and object detection with task alignment module.\n\nThe suggested network achieves SOTA performance with affordable computational cost.", "questions": "n/a", "limitations": "See weakness", "rating": 9, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper presents a novel approach to learning image fusion and objection detection in a synchronous and joint way\n\nThe proposed network achieves the SOTA performance in both tasks.", "weaknesses": "Globally, I am ok with the significance of the work with the SOTA performance, despite the fact that it comes with additional computational cost.\n\nI am more concerned by other issues:\n\n1. After reviewing, this paper gives me the impression that the proposed method is more like a combination of existing works/modules/tricks to achieve the SOTA performance. In other words, I am concerned by the novelty.\n\n2. Secondly, this paper is hard to read and follow. The motivation of the proposed work is not strong enough compared to SOTA works. The diagrams are a little bit confusing. The writing needs to be improved. \n\n3. I am ok with all the proposed modules and blocks to be claimed as novel, such as the blocks in nodes 1 and 2. It seems that the main contribution that the author claimed is on task alignment. This seems to be a very generic learning strategy. The authors should further validate its effectiveness with other works such as MetaFusion or other applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reviewer G1Ro\n\nThank you for your feedback.\n\n**A1**: Reviewer 6qWk and reviewer cEzg both affirmed the novelty of our paper. As reviewer cEzg commented, \"The idea of simultaneously learning image fusion (MF) and object detection (OD) tasks to mutually benefit each other is intriguing and reasonable.\" **It is important to note that the contribution of this paper lies in introducing the first end-to-end joint training paradigm in the fusion detection field.** In current research, joint learning algorithms are an emerging hotspot, leveraging fusion networks and object detection synergies to enhance image informativeness and improve detection performance. However, **existing** optimization methods are typically **non-end-to-end**, relying on multiple steps, as shown in Figure 1. These approaches are cumbersome and reduce training efficiency. Therefore, we propose the **first end-to-end synchronous joint optimization** algorithm, E2E-MFD, which facilitates interaction between intrinsic features from both domains through synchronous joint optimization, enabling a **streamlined, one-stage process**.\n\nMoreover, our end-to-end solution does not merely concatenate two tasks (modules) but involves dedicated module design. To harmonize **fine-grained details** with **semantic information**, we introduce the novel concept of an Object-Region-Pixel Phylogenetic Tree (ORPPT) coupled with a coarse-to-fine diffusion processing (CFDP) mechanism. Additionally, we find that the MF and OD tasks have naturally distinct optimization objectives. MF primarily focuses on capturing pixel-level relationships between image pairs, while OD integrates object semantics within diverse scene contexts. Therefore, there exists an **inherent optimization barrier** between these two tasks. To address this, we especially introduce the concept of **gradient alignment** in the multi-task learning field, proposing GMTA to align gradients for o**ptimizing task dominance and resolving conflicting gradients** in the end-to-end joint training process. As you mentioned, we ultimately propose a novel approach to learning image fusion and objection detection synchronously and jointly. We are the first to innovatively **integrate image fusion and object detection** into a single-stage, end-to-end framework, achieving **SOTA results** on multiple datasets.\n\n**A2**: As pointed out by reviewer 6qWk, “This paper presents the first attempt to achieve simultaneous single-stage training of image fusion and object detection, and the results appear very promising. Inspired by multitask learning, the GMTA method is introduced to reasonably balance the loss functions, thereby converging the fusion detection weights to optimal points.” As discussed in A1, previous SOTA methods relied on non-end-to-end optimization approaches, dividing joint training into multiple steps, which led to complexity during training. These methods excessively emphasized leveraging OD information for MF enhancement, complicating parameter balancing and making them susceptible to local optima of individual tasks. **Therefore, achieving a unified feature set that simultaneously satisfies the characteristics of each task through end-to-end training remains a formidable challenge.** This paper introduces E2E-MFD, an end-to-end multimodal fusion detection algorithm. E2E-MFD aims to seamlessly integrate detailed image fusion and object detection from coarse to fine levels. We introduce the gradient alignment concept from the multi-task learning domain, aiming to eliminate conflicting gradients between object detection and multimodal fusion tasks through the design of GMTA optimization mode. **By facilitating synchronous joint optimization and fostering interaction between intrinsic features from both tasks, E2E-MFD achieves a streamlined single-stage process in an end-to-end manner.** In addition, reviewers bix9, 6qWk, and cEzg both acknowledged our writing presentation. We will strive to improve our writing skills to the best of our ability. Please help us identify which diagrams have caused you confusion. We will make every effort to revise the diagrams and provide comprehensive explanations.\n\n**A3**: Our goal is to explore an end-to-end joint training approach. Thank you for recognizing the **novelty** of the **design of node1 and node2**. In this framework, Node 1 serves the object detection task, while Node 2 acts as a module for MF tasks, with personalized settings harnessing the respective roles of the nodes. As you mentioned, one of our primary contributions is introducing **an end-to-end synchronous training paradigm** for multimodal fusion detection, where synchronized joint optimization allows both tasks to complement each other synergistically. This collaboration enables MF to generate richer, more informative images, enhancing the performance of OD, which in turn provides valuable semantic insights to MF for accurate localization and identification of objects within scenes. However, it is crucial to address the issue of **gradient conflicts** in the joint optimization of fusion and detection tasks, known as task consistency. This challenge, though a common concern in multitask learning, is effectively tackled for the **first time** in the realm of **multimodal fusion detection** through end-to-end training. As reviewer cEzg noted, “An end-to-end fusion detection algorithm is proposed, effectively avoiding the local optimum problem encountered in multi-stage training models.”\n\nAdditionally, it should be noted that the training approach of Metafusion is not end-to-end. Our algorithm essentially represents an advanced end-to-end version of “metafusion,” (i.e., a non-end-to-end image fusion method) taken to its ultimate level of refinement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NKSHjqHoQ9", "reviewer": "Reviewer_6qWk", "summary": "This paper focuses on the task of multimodal image fusion detection, combining texture details and target semantic information. An end-to-end multimodal fusion detection algorithm named E2E-MFD is proposed, which employs synchronous joint optimization, differing from existing independent or cascaded joint methods. The authors introduce a Gradient Matrix Task-Alignment method to help resolve gradient conflict issues in the image fusion and object detection tasks. Experiments on horizontal and oriented object detection datasets demonstrate the effectiveness of this method.", "questions": "See weaknesses above.", "limitations": "The authors note that current model validation relies on visible light and infrared modalities due to limited relevant datasets within the community. They express a need for new dataset guidelines and contributions to the open-source community to address multi-modal dataset validation challenges in the future.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper presents the first attempt to achieve simultaneous single-stage training of image fusion and object detection, and the results appear very promising. \n\n2. Inspired by multitask learning, the Gradient Matrix Task-Alignment method is introduced to reasonably balance the loss functions, thereby converging the fusion detection weights to optimal points. \n\n3. The multi-granularity strategy in the Object-Region-Pixel Phylogenetic Tree demonstrates its effectiveness in learning shared parameters, thereby enhancing object detection performance. \n\n4. The writing and figures in the paper are clear and easy to understand. Proper use of formulas enhances the comprehensibility of their method.\n\n5. The experiments and ablation studies comprehensively demonstrate the results.", "weaknesses": "The paper presents a thorough and well-executed series of experiments that significantly contribute to the strength and credibility of the research. However, I think some problems need to be addressed:\n1. In the experiments, YOLOv5s is compared. But why not comparing with the latest yolo?\n2. The three backbone networks involved in the Figure 1 of the paper are not specified in the text, which would limit other researchers to know the details.\n3. In the line 22, \"a MF network\" should be \"an MF network\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reviewer 6qWk\nThank you for your constructive insights. \n\n**A1**: We acknowledge your point regarding comparing YOLOv5s with the latest version of YOLO. We chose YOLOv5s as it represents a well-established baseline in the field, and our focus was on evaluating the results of the detection accuracy of different fusion methods under the same detector. It is consistent with papers \"CVPR2023 MetaFusion: Infrared and Visible Image Fusion via Meta-Feature Embedding from Object Detection\" and \"CVPR2022 Target-aware Dual Adversarial Learning and a Multi-scenario Multi-Modality Benchmark to Fuse Infrared and Visible for Object Detection. However, we will consider including a comparison with the latest YOLO version in future work to provide a more comprehensive evaluation.\n\n**A2**: Thank you for pointing out this omission. We apologize for the oversight. In Figure 1, the three backbone networks represent the same backbone. The parameters included are defined as the shared parameters for MF and OD networks. We will clarify this in the revised manuscript to ensure that other researchers have the necessary details for replication and comparison purposes.\n\n**A3**: Thank you for your corrections; we have made the necessary revisions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YGImEsDm2b", "reviewer": "Reviewer_cEzg", "summary": "This paper introduces a novel end-to-end algorithm named E2E-MFD for multimodal image fusion and object detection. Unlike existing joint learning methods, its key innovation lies in the synchronous joint optimization approach, simplifying the fusion detection process into a single training step and enhancing efficiency compared to traditional multi-step methods. To harmonize the losses between the image fusion and object detection networks, a Gradient Matrix Task-Alignment method is proposed. This method balances the gradients of shared parameters between the image fusion and object detection tasks, addressing the challenges of task dominance and conflicting gradients in multi-task learning. Additionally, an image fusion network with an Object-Region-Pixel Phylogenetic Tree is designed to perceive information at different granularity levels. Experimental results demonstrate the performance of the proposed method in both image fusion and object detection.", "questions": "1.In \"Study of branches in the Object-Region-Pixel Phylogenetic Tree\", the authors analyzed the reasons for performance degradation under settings 0, 1, 2, 3, and 4, and provided visual evidence. Could you elaborate on why the fusion performance only starts to decline after adding the fourth setting?\n2.For Figure 4, the targets are only circled. Enlarging the highlighted areas, similar to the other figures, would make them clearer.", "limitations": "Expanding new modal datasets or implementing modality conversion between multimodal data will become a solution to the single issue raised in the paper about publicly available multimodal object detection datasets.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The idea of learning image fusion and object detection tasks simultaneously to mutually benefit each other is intriguing and reasonable. \n\n- An end-to-end fusion detection algorithm is proposed, effectively avoiding the local optimum problem encountered in multi-stage training models. Specific modules such as Gradient Matrix Task-Alignment and Object-Region-Pixel Phylogenetic Tree are introduced to achieve this goal. \n\n- Sufficient Experiments demonstrate that these modules facilitate the learning process, and jointly optimizing these two tasks outperforms other existing pipelines. \n\n- The authors clearly describe their methods in the paper and enhance its comprehensibility through the judicious use of formulas and figures.", "weaknesses": "The overall idea is pretty interesting and reasonable. I can see the insight in the proposed method. However, there are some typos in paper:\n1.The L_SSIM in the line 179 seems not same with the Equation (7).\n2.I see a dashed arrow in the figure 1, what’s this mean?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reviewer cEzg\nThank you for your feedback. \n\n**A1**: Regarding the typo in Equation (7) and the inconsistency with $\\mathcal{L}_{\\text {SSIM }}$ on line 179, we apologize for the oversight. We have revised line 179 to ensure consistency with the description in Equation (7). \n\n**A2**: The dashed arrow in Figure 1 represents Dashed arrows indicating cutting off gradient flow. We have clarified this in the figure caption for better understanding.\n\n**A3**: In the study of branches in the Object-Region-Pixel Phylogenetic Tree, we analyzed the performance degradation under settings 0, 1, 2, 3, and 4, and provided visual evidence. The fusion performance begins to decline after adding the fourth setting primarily due to the increased complexity and interaction among multiple branches. As more branches are added, the network may struggle to effectively balance and integrate pixel-level and object-level information, leading to a gradual decline in fusion performance.\n\n**A4**: Regarding Figure 4, we acknowledge your suggestion to enlarge the highlighted areas where targets are circled. We have enhanced the clarity of these areas in Figure 4 to provide a clearer representation of the targets, similar to other figures in the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Author Rebuttal
We thank the reviewers for their comments. We are encouraged that the reviewers appreciate the **sound technology** (6qWk, cEzg), **well-organized writing** (bix9, 6qWk, cEzg), **certain influence** (bix9, 6qWk, cEzg), **clear motivation** (bix9, 6qWk, cEzg), and **excellent experimental performance** (bix9, G1Ro, 6qWk, cEzg). All suggestions were seriously considered and we will carefully revise the manuscript. We address each reviewer in individual comments.

## Motivation and Novelty.
The paper introduces the first end-to-end joint training paradigm in the MFD field, as acknowledged by reviewer comments: reviewer cEzg finds the idea of simultaneous learning intriguing and reasonable, reviewer 6qWk notes it’s the first attempt at simultaneous single-stage training, and reviewer G1Ro acknowledges the novel approach presented in the paper.

Current research aims to enhance image informativeness and detection performance through joint learning algorithms integrating MF and OD networks. However, existing non-end-to-end optimization methods involve cumbersome multiple steps that reduce training efficiency. Therefore, we propose E2E-MFD as the first end-to-end joint training paradigm in the MFD field. E2E-MFD facilitates interaction between intrinsic features from both domains through a streamlined, one-stage process. Our approach includes dedicated module design, such as the novel ORPPT and CFDP mechanism, to effectively balance and integrate fine-grained details with semantic information at pixel and object levels. Additionally, we found the gradient conflict problem in synchronous training for the first time and designed GMTA  to optimize task dominance and resolve conflicting gradients. Comprehensive experiments show that E2E-MFD is superior to the existing pipeline.

## Experimental Results.
According to the suggestions of **reviewer bix9**, we have added the ablation experiments and detailed information is available in the **pdf document**:

A study of fluctuations in single category detection accuracy M3FD dataset is shown **Table R.1**. We prove that it is common to observe fluctuations in detection accuracy within a single category on the M3FD dataset. Despite fluctuations, our algorithm significantly outperforms others in overall mAP. 

Table R.1: A study of fluctuations in single category detection accuracy.

| Method  | People | Car  | Bus  | Motorcycle | Lamp | Truck | $ \text{mAP} _ {50} $ | $ \text{mAP} _ {50:95} $ |
| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
| U2Fusion |    47.5    |   69.3   | 73.0     |      43.7      | 44.9     | 62.2      |          86.9           |            56.8            |
| U2Fusion |    47.7    |   **70.1**   | 73.2     |      43.2      | 44.6     | 63.9      |          87.1           |            57.1            |
| E2E-MFD |    **60.1**    |   69.5   |  **81.4**     |      **52.2**      |  **47.6**    |  **72.2**      |          **91.8**           |            **63.8**            |


The GMTA is performed approximately every $n$ iteration (gradient update), focusing on balancing the independence and coherence of various tasks. **Table R.2** presents the ablation analysis of the $n$ parameter of GMTA. Decreasing $n$ initially disrupts task optimization due to frequent alignment, while increasing $n$ becomes crucial when the network determines task optimization directions. However, excessively large $n$ leads to significant deviations in task paths, making alignment more challenging and negatively impacting performance. 

Table R.2: Ablation studies of the iteration parameter $n$.

| $n$  | EN | MI  | VIF | $\text{mAP} _ {50}$ | $\text{mAP} _ {50:95}$ |
|:-:|:-:|:-:|:-:|:-:|:-:|
| 500  | 6.17 | 15.05 | 1.58 | 90.93 | 62.93   |
| 1000 | **6.36**  | **15.47**  | **1.65**  | **91.80**  | **63.83** |
| 1500 | 6.24 | 15.08 | 1.62 | 91.10 | 63.16    |
| 2000 | 6.13 | 14.69 | 1.45 | 90.35 | 62.75  |

We conducted ablation experiments on CFDP in **Table R.3**, investigating its inclusion and the number of proposed boxes. In the setting without CFDP, we maintained the backbone network while substituting CFDP with RPN(Region Proposal Network), standard components in two-stage object detectors. Results indicate that CFDP enhances detailed information capture and precise box guidance, thereby enhancing fusion image quality and detection performance. For optimal balance between performance and efficiency, we selected 500 proposal boxes.

Table R.3: Ablation studies of the CFDP.

| Settings | Proposal boxes | EN | MI | VIF | $\text{mAP} _ {50}$ | $\text{mAP} _ {50:95}$ | Tr.Time |
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| w/o CFDP     | 500                | 6.07   | 14.78  | 1.58    | 90.13                 | 61.98                    | 2h52m11s    |
|    w CFDP         | 300                | 6.23   | 14.97  | 1.60    | 90.89                 | 63.29                    | 2h23m45s    |
|   w CFDP          | 500                | 6.36   | **15.47** | **1.65** | 91.80               | **63.83**               | 2h50m32s    |
|   w CFDP           | 1000               | **6.37** | 15.34  | 1.63    | **92.05**             | 63.75                    | 3h32m30s    |

In **Table R.4**, three recent fusion SOTA methods (CVPR2024 SHIP, PR2024 CFNet, and PR2024 DSFusion) and three evaluation metrics (Qabf, PSNR, and SSIM) on the three datasets (M3FD, TNO, and RoadScene) are incorporated to valid effectiveness of our method. Compared with other SOTA methods, E2E-MFD achieved superior performance across multiple metrics.

Table R.4: Quantitative results of different fusion methods.

| Method| EN ↑ | MI ↑ | VIF ↑ | Qabf ↑ | PSNR ↑ | SSIM ↑ |
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
| CFNet      | 5.64 | 13.97 | 1.54 | 0.44 | 27.91 | 1.24   |
| DSFusion   | 5.93 | 13.95 | 1.57 | 0.45 | 28.12 | 1.34   |
| SHIP       | 6.19 | 15.02 | 1.61 | 0.50 | 29.25 | 1.38   |
| E2E-MFD    | **6.36** | **15.47** | **1.65** | **0.51** | **30.01** | **1.42** |

Meta-review: This paper proposes a joint learning method for multimodal fusion and object detection. The proposed network achieves SOTA performance with affordable computational cost. This paper received one slight negative score and three very positive scores. All the positive reviewers were satisfied with the author response, while the negative reviewer did not reply to the rebuttal. The AC read the response and discussions and decided to accept this paper.
Decision: Accept (oral)