# End-To-End Multi-View Multi-Modal Detection-Driven Image Fusion: One Method to Fuse them all

Gwendal Bernardi, Godefroy Brisebarre, Sébastien Roman, Mohsen

Ardabilian, Emmanuel Dellendrea

► To cite this version:

Gwendal Bernardi, Godefroy Brisebarre, Sébastien Roman, Mohsen Ardabilian, Emmanuel Dellendrea. End-To-End Multi-View Multi-Modal Detection-Driven Image Fusion: One Method to Fuse them all. 2026. hal-05493938

HAL Id: hal-05493938

https://hal.science/hal-05493938v1

Preprint submitted on 4 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
End-To-End Multi-View Multi-Modal
Detection-Driven Image Fusion: One Method to
Fuse them all

Gwendal Bernardi*†, Godefroy Brisebarre*, Sébastien Roman*, Mohsen Ardabilian†, Emmanuel Dellendrea†

*Tiama, Saint-Genis-Laval, France

†École Centrale de Lyon, CNRS, Université Claude Bernard Lyon 1,

INSA Lyon, Université Lumière Lyon 2, LIRIS, UMR5205, 69130 Écully, France

Mail: g.bernardi@tiama.com, gwendal.bernardi@ec-lyon.fr

*Abstract*—We present EDIF, an end-to-end detection-driven framework designed to unify multi-modal and multi-view image fusion within a single architecture. While most existing fusion methods address either spectral complementarity (multi-modal) or viewpoint variability (multi-view) in isolation, real-world perception systems increasingly require both. EDIF formulates fusion as an object-level alignment problem: heterogeneous images are encoded as sets of keypoints, which are matched and aggregated through a graph attention mechanism to form object-centric representations directly optimized for detection. To stabilize training across heterogeneous components, we introduce a three-stage task-driven strategy that progressively aligns keypoint extraction, object localization, and cross-sensor grouping. In addition, we release the Multi-Modal and Multi-View Object Detection Dataset (MMDOD), a new benchmark designed to study detection-driven fusion under strong modality-view dependencies. MMDOD contains over 10,000 images of transparent objects captured under four complementary modalities (visible, NIR, low-contrast, polarization shift) and six viewpoints, with detailed object-level annotations. Experiments on RGB-thermal, multi-camera, and joint multi-modal multi-view benchmarks show that EDIF achieves performance competitive with recent specialized methods, while uniquely operating within a unified framework. On MMDOD, EDIF significantly outperforms adapted multi-modal multi-view baselines, highlighting the benefits of detection-driven, object-level fusion. The proposed MMDOD dataset is publicly available at https://datasets.liris.cnrs.fr/mmdod-version1.

***Index Terms***—Multi-View, Multi-Modal, Object Detection, Image Fusion, GNN

# I. INTRODUCTION

Image fusion is a central problem in computer vision, enabling robust perception in applications ranging from medical imaging to autonomous driving [1], [2]. By integrating complementary cues from heterogeneous sources, fusion improves reliability under challenging conditions such as occlusion, low illumination, or sensor degradation. Existing methods are typically specialized: multi-view fusion enforces geometric consistency across cameras, while multi-modal fusion integrates spectral or sensor-specific cues [3]. Real-world perception, however, often combines both dimensions, as in multi-modal multi-view medical imaging [4] or autonomous driving platforms with multiple sensors and viewpoints [5], [6]. Despite this need, unified architectures capable of jointly

handling multi-modal and multi-view fusion remain largely underexplored. Fusion approaches can be categorized into supervised, unsupervised, and task-driven paradigms [3]. Supervised methods rely on ground-truth fused images, which are often unavailable, while unsupervised strategies depend on heuristics weakly correlated with downstream tasks. Task-driven learning, in contrast, supervises fusion via an end objective such as detection, providing semantically grounded guidance. This motivates our detection-driven formulation of unified fusion. In this work, we introduce End-To-End Multi-View Multi-Modal Detection-Driven Image Fusion (EDIF), a generalist framework that unifies multi-modal and multi-view fusion within a single detection-oriented architecture. EDIF departs from dense feature aggregation strategies (e.g., MVMM-Net [4], GMMVF [7]) by formulating fusion as an object-level alignment problem. Input images are encoded as keypoints enriched with local descriptors, which are matched and refined through a graph attention network to reason jointly over spectral and viewpoint variations. Keypoints are then softly grouped into object-centric representations, optimized directly for bounding box regression and classification through a unified detection head. Stable end-to-end optimization is achieved via a three-stage training strategy that progressively aligns keypoint extraction, object localization, and cross-sensor grouping. To evaluate EDIF, we introduce the Multi-Modal and Multi-View Object Detection Dataset (MMDOD), a benchmark explicitly designed to study detection-driven fusion under strong modality-view dependencies. MMDOD contains over 10,000 annotated images of transparent objects captured under four complementary modalities and six viewpoints, covering both destructive and non-destructive artifacts. For fair comparison, we adapt MVMM-Net into a detection-oriented baseline (MVMM-DET). Our main contributions are summarized as follows:

* An end-to-end detection-driven fusion framework. We propose EDIF, the first architecture capable of performing unified multi-modal and multi-view fusion for object detection, bridging both domains within a single, task-driven pipeline.
<figure><img src="image_2.png" /><figcaption>Fig. 1. Overview of the proposed EDIF pipeline. The architecture consists of four modules: (1) a keypoint extractor that identifies interest points in each image, (2) a keypoint encoder that projects descriptors into a shared representation, (3) a graph attention-based feature matching block that establishes correspondences across views and modalities, and (4) a decoder with two branches for bounding box regression and object classification. Training proceeds in three stages: keypoint localization with supervision from ground-truth box centers, bounding box regression and confidence estimation, and joint training of feature matching and classification with full detection supervision.</figcaption></figure>

* A transformer-augmented keypoint extractor (Trans-Point). We enhance SuperPoint with transformer layers for improved multi-modal and multi-view keypoint reliability.
* A dual-head detection and grouping strategy. EDIF introduces a GNN-based grouping mechanism to aggregate keypoints into object clusters, followed by dual-head classification and bounding-box prediction.
* A three-stage progressive learning paradigm. Our training schedule aligns keypoint, detection, and semantic grouping optimization with a decreasing $\sigma$ for stable and efficient convergence.
* A new benchmark: MMDOD. We release the first publicly available dataset for detection-driven multi-modal and multi-view fusion, along with an adapted MVMM-DET baseline for fair comparison and reproducibility.

Extensive experiments on RGB-thermal, multi-camera, and MMDOD benchmarks show that EDIF achieves competitive performance with specialized models while uniquely unifying both fusion dimensions, demonstrating the benefits of detection-driven, object-level reasoning.

## II. RELATED WORK

Our work lies at the intersection of image fusion and feature matching. Image fusion aims to integrate heterogeneous sources, traditionally through either multi-modal or multi-view paradigms, while feature matching focuses on establishing explicit correspondences across inputs. EDIF unifies these two lines of research within a single detection-driven framework.

a) *Multi-modal and multi-view fusion:* Multi-modal fusion integrates complementary cues from different sensing modalities such as RGB, infrared, depth, or polarization.

Early approaches primarily targeted image reconstruction using encoder-decoder or generative architectures [8]–[10]. More recent methods adopt task-driven objectives, optimizing fusion for downstream tasks such as object detection, using attention mechanisms, uncertainty modeling, or meta-learning strategies [11]–[14]. In parallel, multi-view fusion has focused on enforcing geometric consistency across cameras, initially through depth-based or 3D reasoning pipelines [15], [16], and more recently via task-driven multi-view detection frameworks such as MVDet, MVDetr, and QMVDet [17], [18]. While these approaches achieve strong performance, multi-modal methods remain largely modality-specific, and multi-view methods are typically restricted to single-modality (RGB) inputs. The joint exploitation of multi-modal and multi-view information remains comparatively underexplored. MVMM-Net [4] constitutes an early attempt to combine both dimensions, but relies on naïve feature aggregation and is not designed for detection-oriented objectives. GMMVF [7] introduced graph reasoning for multi-modal multi-view fusion, yet remains task-specific and not directly applicable to object detection. In contrast, EDIF introduces the first end-to-end detection-driven architecture that explicitly models both spectral and angular dependencies through keypoint-based graph attention matching and unified object-level reasoning.

b) *Feature matching:* Feature matching methods aim to establish reliable correspondences between local features across images. SuperPoint [19] introduced self-supervised keypoint detection and description, while SuperGlue [20] incorporated graph neural networks for context-aware matching. Subsequent works such as LightGlue and OmniGlue improved
efficiency and generalization [21], and recent approaches extended matching to multi-view and dense settings [22], [23]. These advances demonstrate the effectiveness of graph-based attention for establishing consistent correspondences, motivating the matching-driven formulation adopted in EDIF.

c) Datasets: Numerous datasets support either multi-modal fusion (e.g., KAIST, LLVIP, M3FD, MSRS) or multi-view perception (e.g., Wildtrack, MultiViewX, nuScenes, KITTI) [24]–[29]. However, existing benchmarks typically isolate these two dimensions and do not provide consistent multi-modal, multi-view object-level annotations. This limitation hinders the development and evaluation of generalist multi-sensor fusion models. As emphasized in recent surveys [3], unified benchmarks remain a critical missing component, motivating the introduction of MMDOD.

### III. METHOD

We aim to unify multi-modal and multi-view fusion within a single detection-driven framework. Inspired by recent advances in feature matching and multi-view reasoning [20], [22], we formulate fusion as an object-level alignment problem. EDIF integrates three main components: (i) a keypoint extractor that converts each image into a compact set of local descriptors, (ii) a graph attention-based matching and grouping module that reasons jointly over spectral and viewpoint variations, and (iii) a detection head that predicts object locations and categories from object-centric representations. As illustrated in Fig. 1, heterogeneous images are independently encoded as keypoints, which are then matched and softly grouped into latent object hypotheses. These object-level embeddings are finally decoded into bounding boxes and class labels under unified detection supervision.

a) Keypoint Extractor: TransPoint.: Robust local representations are central to unified fusion. We propose TransPoint, a Transformer-enhanced extension of SuperPoint [19]. Given an input image, TransPoint outputs a fixed set of keypoints $\{(x_i, d_i, c_i)\}$, where $x_i$ denotes spatial location, $d_i$ a descriptor, and $c_i$ a confidence score. The architecture combines a convolutional backbone for local feature extraction with stacked Transformer encoder layers [30] to capture long-range dependencies and modality-specific context. Two lightweight heads predict interest point heatmaps and corresponding descriptors. A top-N selection retains the most confident points, yielding a compact and modality-agnostic representation suitable for heterogeneous inputs.

b) Graph Attention Matching and Grouping.: Keypoints extracted from all modalities and viewpoints are pooled into a single heterogeneous set and aligned using a graph attention neural network (GANN). Each keypoint is treated as a node whose initial embedding concatenates descriptor and confidence information. Multi-head attention layers propagate contextual information across nodes, enabling joint reasoning over appearance similarity, spatial relations, and cross-sensor consistency beyond pairwise matching. The refined keypoint embeddings are then softly assigned to a fixed number of object hypotheses through a differentiable grouping module

inspired by permutation-invariant set prediction [31]. Sinkhorn normalization [32] produces a soft assignment matrix, allowing each object hypothesis to aggregate complementary evidence from multiple views and modalities. Each group therefore represents a latent object supported by heterogeneous observations and serves as the fused object-level representation.

c) Decoder and Detection Head.: Object-level group embeddings are decoded into detections. A residual multi-layer perceptron regresses normalized bounding boxes and predicts an objectness confidence for each group. In parallel, aggregated group descriptors are processed by a lightweight classifier to output class logits. Together, these components form a unified detection pipeline: TransPoint provides compact local representations, the GANN aligns and fuses them across heterogeneous inputs, and the detection head directly predicts object instances from group-level embeddings. Unlike specialized fusion systems that operate at the pixel or feature-map level, EDIF performs detection-driven fusion at the object level, enabling a single architecture to handle both multi-modal and multi-view perception.

### IV. LOSS AND THREE-STAGE TRAINING

EDIF is trained under a task-driven paradigm in which object detection provides the supervisory signal for the entire architecture. Direct end-to-end optimization is unstable due to the interaction between heterogeneous components and the discrete nature of grouping. We therefore adopt a three-stage training strategy that progressively introduces structural constraints, from point-level geometry to object-level semantics and finally cross-sensor consistency.

a) Stage 1: Keypoint Pretraining with Descriptor Consistency.: The first stage pre-trains the keypoint extractor to produce spatially stable points and discriminative descriptors. Supervision is derived from ground-truth bounding box centers, encoded as Gaussian heatmaps:

$$T_i(x, y) = \sum_{j=1}^{M_i} \exp\left(-\frac{(x - x_j)^2 + (y - y_j)^2}{2\sigma^2}\right), \quad (1)$$

where $(x_j, y_j)$ denotes object centers. These centers are not intended to represent semantic landmarks, but act as weak geometric anchors that stabilize early learning. To avoid collapse under sparse supervision, we anneal the Gaussian variance during training, which smoothly transitions from coarse to fine localization. This curriculum-style supervision improves convergence and prevents early overfitting to background responses. The detector is optimized using a combination of positional L1 loss and weighted heatmap loss to balance localization accuracy and score calibration. To encourage view- and modality-invariant representations, descriptors are trained using a supervised contrastive loss [33]:

$$\mathcal{L}_{\text{desc}} = -\frac{1}{N} \sum_{i=1}^{N} \frac{1}{|P(i)|} \sum_{j \in P(i)} \log \frac{\exp(f_i^\top f_j / \tau)}{\sum_{k \neq i} \exp(f_i^\top f_k / \tau)}, \quad (2)$$
where positives $P(i)$ correspond to keypoints associated with the same object instance. This loss enforces semantic alignment across views and sensors, providing a robust initialization for downstream matching and grouping.

b) Stage 2: Object Localization and Confidence Estimation.: In the second stage, the keypoint encoder, grouping module, and decoder are jointly optimized to regress bounding boxes and predict objectness confidence. Predictions are matched to ground truth using Hungarian assignment with a cost combining IoU and L1 distance. This matching ensures a one-to-one correspondence and stabilizes supervision under variable object counts. For matched pairs, localization is optimized using a mixed regression objective:

$$ \mathcal{L}_{\text{box}} = \alpha \|B^{\text{pred}} - B^{\text{gt}}\|_{1} + (1 - \text{IoU}(B^{\text{pred}}, B^{\text{gt}})), \quad (3) $$

which balances precise coordinate regression with overlap quality. Each prediction additionally outputs a confidence score trained with binary cross-entropy, suppressing spurious detections induced by the fixed keypoint budget.

c) Stage 3: Detection and Cross-Sensor Grouping.: The final stage trains the full architecture end-to-end with a composite objective combining box regression, classification, objectness, and group-level consistency. Bounding boxes are refined using generalized IoU with additional center and size terms:

$$ \mathcal{L}_{\text{box}} = (1 - \text{IoU}) + \lambda_c \|\Delta c\|_1 + \lambda_s \|\Delta s\|_1, \quad (4) $$

which explicitly enforces correct object size and position. Semantic categories are supervised with Focal Loss and label smoothing, while objectness is guided by IoU-aware soft targets, assigning high scores only to well-overlapping predictions. Corss-image consistency is enforced via group coherence, encouraging keypoints of the same object to share similar assignments:

$$ \mathcal{L}_{\text{group}} = \frac{1}{|\mathcal{P}|} \sum_{(i,j) \in \mathcal{P}} (1 - \cosine(S_i, S_j)), \quad (5) $$

with a complementary sparsity loss to promote peaked assignments:

$$ \mathcal{L}_{\text{sparsity}} = \frac{1}{BN} \sum_{b,n} - \sum_{g} S_{b,n,g} \log S_{b,n,g}. \quad (6) $$

This loss unifies instance-level detection, confidence calibration, and cross-sensor grouping in a compact, end-to-end objective.

## V. MMDOD: A MULTI-VIEW AND MULTI-MODAL DATASET FOR DETECTION

We introduce the Multi-Modal and Multi-View Object Detection Dataset (MMDOD), designed for transparent object inspection and evaluation of unified fusion methods. Unlike existing datasets, MMDOD provides over 10,000 images captured from six calibrated viewpoints across four complementary modalities: visible, near-infrared (NIR), low-contrast, and polarization-shift imaging. These modalities highlight texture, transparency, low-contrast defects, and birefringence effects.

The dataset contains more than 21,000 annotated objects in eight categories, covering both destructive (e.g., scratches) and non-destructive artifacts (e.g., stickers, filters), with distractors such as reflections and shadows. Detectability strongly depends on modality-view combinations, making MMDOD a controlled benchmark for evaluating detection-driven multi-modal multi-view fusion. Baselines using MVMM-DET [4] and a stronger modality-aware variant demonstrate that EDIF outperforms these models, confirming the advantage of jointly learning object-level representations across spectral and angular cues. MMDOD is released to facilitate reproducible evaluation and research on unified multi-image perception.

## VI. EXPERIMENTAL RESULTS

We evaluate EDIF under three complementary scenarios: multi-modal fusion (RGB-thermal), multi-view fusion (multicamera pedestrian detection), and joint multi-modal multi-view fusion. This protocol allows us to assess whether a single detection-driven framework can remain effective across heterogeneous fusion settings typically addressed by specialized architectures. All experiments are conducted under the standard evaluation protocols of each benchmark.

a) Multi-Modal Fusion: We first evaluate EDIF on RGB-thermal benchmarks, which assess the ability of fusion models to exploit complementary information from visible and infrared modalities. On the M3FD dataset (Table I), EDIF achieves an mAP@50-95 of 55.8, which is comparable to recent task-driven fusion approaches. While MetaFusion [14] reports the highest score (56.5), EDIF outperforms several widely used fusion strategies such as TarDAL [34], U2Fusion [35], and FusionGAN [9]. These results indicate that EDIF can effectively leverage spectral complementarity, despite not being specifically optimized for single multi-modal fusion.

<table><caption>TABLE I OBJECT DETECTION PERFORMANCE ON M3FD AND FLIR DATASETS.</caption><caption>M3FD (mAP@50-95)</caption><thead><tr><th>Method (M3FD)</th><th>mAP@50-95</th><th>Method (FLIR)</th><th>mAP@50</th><th>mAP@75</th></tr></thead><tbody><tr><td>FusionGAN [9]</td><td>54.2</td><td>GAFF [36]</td><td>72.9</td><td>32.9</td></tr><tr><td>U2Fusion [35]</td><td>55.7</td><td>CFT [37]</td><td>78.7</td><td>35.5</td></tr><tr><td>TarDAL [34]</td><td>54.4</td><td>CSAA [38]</td><td>79.2</td><td>37.4</td></tr><tr><td>MetaFusion [14]</td><td><strong>56.5</strong></td><td>LRAF-Net [11]</td><td>80.5</td><td>-</td></tr><tr><td>EDIF (ours)</td><td>55.8</td><td>CBAM [12]</td><td><strong>86.2</strong></td><td><strong>43.0</strong></td></tr><tr><td></td><td></td><td>EDIF (ours)</td><td>84.2</td><td>41.2</td></tr></tbody></table>

On the FLIR dataset (Table I), which includes challenging outdoor and low-illumination scenarios, EDIF achieves 84.2 mAP@50 and 41.2 mAP@75. While remaining slightly below the strongest specialized model (CBAM [12]), EDIF clearly outperforms MFPT-based [39] and uncertainty-aware [13] baselines. This confirms that the proposed object-level fusion strategy generalizes well to standard multi-modal detection settings.

b) Multi-View Fusion: We next evaluate EDIF on multicamera pedestrian detection benchmarks, which emphasize geometric consistency and cross-view reasoning. On Wildtrack (Table II), EDIF achieves a MODA of 0.928 and a MODP
of 0.812, performing on par with recent methods such as QMVDet [18] and 3DROM [40], while outperforming earlier baselines such as MVDet [41]. EDIF also achieves one of the highest precision scores, indicating a low false-positive rate across views. On MultiViewX (Table III), EDIF reaches

<table><caption>TABLE II
PERFORMANCE ON WILDTRACK.</caption><thead><tr><th>Method</th><th>MODA</th><th>MODP</th><th>Precision</th><th>Recall</th></tr></thead><tbody><tr><td>MVDet [41]</td><td>0.882</td><td>0.757</td><td>0.947</td><td>0.936</td></tr><tr><td>MVDetr [17]</td><td>0.915</td><td>0.821</td><td><strong>0.974</strong></td><td>0.940</td></tr><tr><td>3DROM [40]</td><td><strong>0.935</strong></td><td>0.759</td><td>0.972</td><td>0.962</td></tr><tr><td>QMVDet [18]</td><td>0.931</td><td><strong>0.826</strong></td><td>0.966</td><td><strong>0.964</strong></td></tr><tr><td>EDIF (ours)</td><td>0.928</td><td>0.812</td><td>0.973</td><td>0.959</td></tr></tbody></table>

a MODA of 0.947 and a MODP of 0.908, remaining close to the strongest reported methods while surpassing several established baselines. These results show that EDIF retains strong multi-view reasoning capabilities, despite not relying on architectures specifically designed for ground-plane localization.

<table><caption>TABLE III
PERFORMANCE ON MULTIVIEWX.</caption><thead><tr><th>Method</th><th>MODA</th><th>MODP</th><th>Precision</th><th>Recall</th></tr></thead><tbody><tr><td>MVDet [41]</td><td>0.893</td><td>0.796</td><td>0.968</td><td>0.867</td></tr><tr><td>MVDetr [17]</td><td>0.937</td><td>0.913</td><td>0.995</td><td>0.942</td></tr><tr><td>3DROM [40]</td><td>0.950</td><td>0.849</td><td>0.990</td><td><strong>0.961</strong></td></tr><tr><td>QMVDet [18]</td><td><strong>0.951</strong></td><td><strong>0.922</strong></td><td><strong>0.996</strong></td><td>0.955</td></tr><tr><td>EDIF (ours)</td><td>0.947</td><td>0.908</td><td>0.993</td><td>0.949</td></tr></tbody></table>

c) *Multi-Modal and Multi-View Fusion:* We evaluate EDIF on the MMDOD benchmark, which requires joint exploitation of multi-modal and multi-view information. To assess the benefit of multi-modal and multi-view fusion, we compare EDIF with three baselines: a mono-image DETR [42], which processes single RGB images and represents a strong single-view detection architecture; MVMM-DET [4]; and a stronger adapted variant incorporating modality-aware encoders and a more powerful backbone. Results are reported in Table IV. The mono-image DETR achieves high localization

<table><caption>TABLE IV
RESULTS ON MMDOD (MULTI-MODAL AND MULTI-VIEW).</caption><thead><tr><th>Method</th><th>mAP@50</th><th>mAP@75</th><th>F1-Score</th></tr></thead><tbody><tr><td>Baseline Mono-View DETR [42]</td><td>0.855</td><td>0.780</td><td>0.313</td></tr><tr><td>MVMM-DET [4]</td><td>0.649</td><td>0.292</td><td>0.321</td></tr><tr><td>EDIF (ours)</td><td><strong>0.823</strong></td><td><strong>0.655</strong></td><td><strong>0.682</strong></td></tr></tbody></table>

scores (mAP@50 and mAP@75) due to its strong single-view detection capabilities, but performs poorly on F1-Score, reflecting its difficulty in correctly classifying objects when multiple modalities and viewpoints provide complementary cues. In contrast, EDIF leverages the full multi-modal multi-view context, resulting in significantly improved classification performance while maintaining competitive localization. These results demonstrate that jointly modeling spectral and angular information under detection supervision provides substantial gains over both single-view and adapted multi-modal multi-

view baselines, highlighting the value of unified object-level representations.

## VII. DISCUSSION AND LIMITATIONS

EDIF demonstrates that a single detection-driven framework can effectively integrate multi-modal and multi-view information. On standard benchmarks, it achieves competitive performance compared to specialized methods, and on MMDOD, which explicitly requires joint spectral and angular reasoning, EDIF yields substantial improvements over adapted fusion baselines. This confirms that learning object-centric representations under unified detection supervision is particularly beneficial when both modalities and viewpoints vary. However, EDIF does not always surpass task-specific state-of-the-art models in settings limited to a single fusion dimension, reflecting the inherent trade-off between specialization and generalization. The three-stage training stabilizes optimization but increases training complexity, and the architecture incurs computational overhead from keypoint extraction and graph reasoning, limiting real-time deployment. Additionally, MMDOD focuses on a specific industrial scenario, leaving open questions about generalization to more diverse or dynamic environments. Future work includes optimizing the architecture for computational efficiency, handling missing views or modalities, and improving model components to achieve state-of-the-art performance across all fusion scenarios. Extensive studies on module contributions, hyperparameter sensitivity, and trade-offs between accuracy and speed will further guide the design of generalist detection-driven fusion systems applicable to heterogeneous multi-sensor setups.

## VIII. CONCLUSION

We presented EDIF, a end-to-end multi-view multi-modal detection-driven framework for image fusion that unifies multi-modal and multi-view settings. By leveraging graph-based feature matching, EDIF captures multi-modal and multi-view dependencies within a single representation. Experiments on RGB-thermal, multi-camera, and joint datasets show that EDIF is competitive with specialized state-of-the-art methods while uniquely generalizing across heterogeneous inputs. This flexibility positions EDIF as a promising foundation for real-world multi-sensor detection systems. In future work, we plan to extend EDIF to handle missing modalities or viewpoints, a critical challenge in real-world deployments. We aim to investigate graph neural network-based mechanisms to dynamically adapt the fusion process when certain modalities or views are unavailable. These directions hold potential to further enhance the robustness and adaptability of EDIF in unconstrained multi-sensor environments.

## REFERENCES

[1] T. Baltrušaitis, C. Ahuja, and L.-P. Morency, "Multimodal machine learning: A survey and taxonomy," *IEEE transactions on pattern analysis and machine intelligence*, vol. 41, no. 2, pp. 423-443, 2018.

[2] S. Karim, G. Tong, J. Li, A. Qadir, U. Farooq, and Y. Yu, "Current advances and future perspectives of image fusion: A comprehensive review," *Information Fusion*, vol. 90, pp. 185-217, 2023.
[3] G. Bernardi, G. Brisebarre, S. Roman, M. Ardabilian, and E. Dellandrea, "A comprehensive survey on image fusion: Which approach fits which need," *Information Fusion*, vol. 126, p. 103594, 2026. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1566253525006669
[4] J. Song, Y. Zheng, M. Zakir Ullah, J. Wang, Y. Jiang, C. Xu, Z. Zou, and G. Ding, "Multiview multimodal network for breast cancer diagnosis in contrast-enhanced spectral mammography images," *International Journal of Computer Assisted Radiology and Surgery*, vol. 16, no. 6, pp. 979-988, 2021.
[5] V. Vs, J. M. J. Valanarasu, P. Oza, and V. M. Patel, "Image fusion transformer," *2022 IEEE International Conference on Image Processing (ICIP)*, pp. 3566-3570, 2022.
[6] J. Sun, L. Chen, Y. Xie, S. Zhang, Q. Jiang, X. Zhou, and H. Bao, "Disp r-cnn: Stereo 3d object detection via shape prior guided instance disparity estimation," in *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 2020, pp. 10548-10557.
[7] J. Chen and S. Dey, "Graph-based multi-modal multi-view fusion for facial action unit recognition," *IEEE Access*, vol. 12, pp. 69310-69324, 2024.
[8] H. Li, X.-J. Wu, and J. Kittler, "Rfn-nest: An end-to-end residual fusion network for infrared and visible images," *Information Fusion*, vol. 73, pp. 72-86, 2021.
[9] J. Ma, W. Yu, P. Liang, C. Li, and J. Jiang, "Fusionian: A generative adversarial network for infrared and visible image fusion," *Information fusion*, vol. 48, pp. 11-26, 2019.
[10] J. Ma, H. Xu, J. Jiang, X. Mei, and X.-P. Zhang, "Ddcgan: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion," *IEEE Transactions on Image Processing*, vol. 29, pp. 4980-4995, 2020.
[11] H. Fu, S. Wang, P. Duan, C. Xiao, R. Dian, S. Li, and Z. Li, "Lraf-net: Long-range attention fusion network for visible-infrared object detection," *IEEE Transactions on Neural Networks and Learning Systems*, 2023.
[12] S. A. Deevi, C. Lee, L. Gan, S. Nagesh, G. Pandey, and S.-J. Chung, "Rgb-x object detection via scene-specific fusion modules," in *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision*, 2024, pp. 7366-7375.
[13] Y. Sun, B. Cao, P. Zhu, and Q. Hu, "Drone-based rgb-infrared cross-modality vehicle detection via uncertainty-aware learning," *IEEE Transactions on Circuits and Systems for Video Technology*, vol. 32, no. 10, pp. 6700-6713, 2022.
[14] W. Zhao, S. Xie, F. Zhao, Y. He, and H. Lu, "Metafusion: Infrared and visible image fusion via meta-feature embedding from object detection," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2023, pp. 13955-13965.
[15] C. Shu, J. Deng, F. Yu, and Y. Liu, "3dppe: 3d point positional encoding for transformer-based multi-camera 3d object detection," in *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 2023, pp. 3580-3589.
[16] A. Rabby and C. Zhang, "Beyondpixels: A comprehensive review of the evolution of neural radiance fields," *arXiv preprint arXiv:2306.03000*, 2023.
[17] Y. Hou and L. Zheng, "Multiview detection with shadow transformer (and view-coherent data augmentation)," in *Proceedings of the 29th ACM International Conference on Multimedia*, 2021, pp. 1673-1682.
[18] H.-M. Hsu, X. Yuan, Y.-Y. Chuang, W. Sun, and R.-I. Chang, "Query-based multiview detection for multiple visual sensor networks," *Sensors*, vol. 24, no. 15, p. 4773, 2024.
[19] L. Hui, J. Yuan, M. Cheng, J. Xie, X. Zhang, and J. Yang, "Superpoint network for point cloud oversegmentation," in *Proceedings of the IEEE/CVF International Conference on Computer Vision*, 2021, pp. 5510-5519.
[20] P.-E. Sarlin, D. DeTone, T. Malisiewicz, and A. Rabinovich, "Superglue: Learning feature matching with graph neural networks," in *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 2020, pp. 4938-4947.
[21] H. Jiang, A. Karpur, B. Cao, Q. Huang, and A. Araujo, "Omniglue: Generalizable feature matching with foundation model guidance," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2024, pp. 19865-19875.
[22] B. Roessle and M. Nießner, "End2end multi-view feature matching with differentiable pose optimization," in *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, October 2023.

[23] J. Edstedt, Q. Sun, G. Bökman, M. Wadenbäck, and M. Felsberg, "Roma: Robust dense feature matching," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2024, pp. 19790-19800.
[24] S. Hwang, J. Park, N. Kim, Y. Choi, and I. S. Kweon, "Multispectral pedestrian detection: Benchmark dataset and baselines," *Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2015.
[25] X. Jia, C. Zhu, M. Li, W. Tang, and W. Zhou, "Llvip: A visible-infrared paired dataset for low-light vision," in *Proceedings of the IEEE/CVF international conference on computer vision*, 2021, pp. 3496-3504.
[26] J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong, and Z. Luo, "Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection," *IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 5802-5811, 2022.
[27] L. Tang, J. Yuan, H. Zhang, X. Jiang, and J. Ma, "Piafusion: A progressive infrared and visible image fusion network based on illumination aware," *Information Fusion*, 2022.
[28] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, "nuscenes: A multimodal dataset for autonomous driving," *arXiv preprint arXiv:1903.11027*, 2019.
[29] A. Geiger, P. Lenz, and R. Urtasun, "Are we ready for autonomous driving? the kitti vision benchmark suite," *Conference on Computer Vision and Pattern Recognition (CVPR)*, 2012.
[30] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, "Transformers in vision: A survey," *ACM computing surveys (CSUR)*, vol. 54, no. 10s, pp. 1-41, 2022.
[31] S. H. Rezatofighi, R. Kaskman, F. T. Motlagh, Q. Shi, D. Cremers, L. Leal-Taixé, and I. Reid, "Deep perm-set net: Learn to predict sets with unknown permutation and cardinality using deep neural networks," *arXiv preprint arXiv:1805.00613*, 2018.
[32] M. Cuturi, "Sinkhorn distances: Lightspeed computation of optimal transport," *Advances in neural information processing systems*, vol. 26, 2013.
[33] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, "Supervised contrastive learning," *Advances in neural information processing systems*, vol. 33, pp. 18661-18673, 2020.
[34] J. Liu, X. Fan, Z. Huang, G. Wu, R. Liu, W. Zhong, and Z. Luo, "Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection," in *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, 2022, pp. 5802-5811.
[35] H. Xu, J. Ma, J. Jiang, X. Guo, and H. Ling, "U2fusion: A unified unsupervised image fusion network," *IEEE transactions on pattern analysis and machine intelligence*, vol. 44, no. 1, pp. 502-518, 2020.
[36] H. Zhang, E. Fromont, S. Lefèvre, and B. Avignon, "Guided attentive feature fusion for multispectral pedestrian detection," in *Proceedings of the IEEE/CVF winter conference on applications of computer vision*, 2021, pp. 72-80.
[37] F. Qingyun, H. Dapeng, and W. Zhaokui, "Cross-modality fusion transformer for multispectral object detection," *arXiv preprint arXiv:2111.00273*, 2021.
[38] Y. Cao, J. Bin, J. Hamari, E. Blasch, and Z. Liu, "Multimodal object detection by channel switching and spatial attention," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2023, pp. 403-411.
[39] Y. Zhu, X. Sun, M. Wang, and H. Huang, "Multi-modal feature pyramid transformer for rgb-infrared object detection," *IEEE Transactions on Intelligent Transportation Systems*, vol. 24, no. 9, pp. 9984-9995, 2023.
[40] R. Qiu, M. Xu, Y. Yan, J. S. Smith, and X. Yang, "3d random occlusion and multi-layer projection for deep multi-camera pedestrian localization," in *European Conference on Computer Vision*. Springer, 2022, pp. 695-710.
[41] Y. Hou, L. Zheng, and S. Gould, "Multiview detection with feature perspective transformation," in *European Conference on Computer Vision*. Springer, 2020, pp. 1-18.
[42] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-end object detection with transformers," in *European Conference on Computer Vision*. Springer, 2020, pp. 213-229.